Home

Decorated Concurrency

Deco is a simplified parallel computing model for Python. Deco automatically parallelizes Python programs, and requires minimal modifications to existing serial programs.

Install using pip:

pip install deco

General Usage

Deco uses Python decorators to alter functions and make them execute concurrently or in parallel. It includes two decorators to facilitate concurrent programming. The first step in using deco is to identify a function that should execute in parallel and apply the @concurrent decorator to it. Take a look at the @concurrent page for more information on using this decorator on its own.

The second step in using deco is to mark any functions that call @concurrent functions with the second decorator @synchronized. Functions marked with @synchronized will be modified to make sure that all calls to @concurrent functions appropriately synchronized and have their results handled correctly. More information about how @synchronized works can be found on its page in the wiki.

The following is an example usage of these two decorators which would be the ideal use case of deco.

@concurrent # We add this for the concurrent function
def process_url(url, data):
  #Does some work which takes a while
  return result

@synchronized # And we add this for the function which calls the concurrent function
def process_data_set(data):
  results = {}
  for url in urls:
    results[url] = process_url(url, data)
  return results

Adding those two decorators is all that's necessary to make this program execute in parallel. Deco does however have some limitations, discussed below briefly and more in depth in the pages for each decorator.

Limitations

• The @concurrent decorator will only speed up functions that take longer than ~1ms
  • If they take less time your code will run slower!
• The @synchronized decorator only works on 'simple' functions, make sure the function meets the following criteria
  • Only calls, or assigns the result of @concurrent functions to indexable objects such as:
    • concurrent(...)
    • process(concurrent(...), ...)
    • result[key] = concurrent(...)
  • Never indirectly reads objects that get assigned to by calls of the @concurrent function

How it works

For an in depth discussion of the mechanisms at work, we wrote a paper for a class which can be found here.

As an overview, DECO is mainly just a smart wrapper for Python's multiprocessing.pool. When @concurrent is applied to a function it replaces it with calls to pool.apply_async. Additionally when arguments are passed to pool.apply_async, DECO replaces any index mutable objects with proxies, allowing it to detect and synchronize mutations of these objects. The results of these calls can then be obtained by calling wait() on the concurrent function, invoking a synchronization event. These events can be placed automatically in your code by using the @synchronized decorator on functions that call @concurrent functions. Additionally while using @synchronized, you can directly assign the result of concurrent function calls to index mutable objects. These assignments get refactored by DECO to automatically occur during the next synchronization event. All of this means that in many cases, parallel programming using DECO appears exactly the same as simpler serial programming.