taylorism

Framework for parallelisation of tasks.

Functions

taylorism.batch_main(common_instructions={}, individual_instructions={}, scheduler=None, verbose=False, maxlenreport=1024, print_report=<built-in function print>, sharedmemory_common_instructions={})

Run execution of the instructions as a batch process, waiting for all instructions are finished and finally printing report.

Args and kwargs are those of run_as_server() function.

taylorism.run_as_server(common_instructions={}, individual_instructions={}, scheduler=None, verbose=False, maxlenreport=1024, sharedmemory_common_instructions={})

Build a Boss instance, make him hire workers, run the workers, and returns the Boss instance.

Be aware that the Boss MUST be told when no more instructions will be appended, or the subprocess will continue to live alone (until destruction of the Boss instance).

Parameters:

• common_instructions (dict) – to be passed to the workers

• individual_instructions (dict) – to be passed to the workers

• scheduler – scheduler to rule scheduling of workers/threads

• verbose (bool) – is the Boss verbose or not.

• maxlenreport (int) – the maximum number of lines for the report (when running in verbose mode)

• sharedmemory_common_instructions (dict) – special “instructions”, whose memory allocation is shared among workers and from main process. Warning: these objects must inherit in some way from multiprocessing.Array or sharedctypes. For n-dimensional arrays, it is advised to be instances of the here-defined SharedNumpyArray.

Classes

class taylorism.BindedWorker(*kargs, **kwargs)

Bases: Worker

Workers binded to a cpu core (Linux only).

This class is deprecated. Instead, inherit from Worker and create a scheduler with binded=True.

Note

This class is managed by footprint.

• info: Not documented

• priority: PriorityLevel::DEFAULT (rank=1)

Automatic parameters from the footprint:

• expected_time (builtins.float) - rxx - How long the worker is expected to run (in s).

  • Optional. Default is 0.0.

• memory (builtins.float) - rxx - Memory that should be used by the worker (in MiB).

  • Optional. Default is 0.0.

• name (builtins.str) - rwx - Name of the worker.

  • Optional. Default is None.

• scheduler_hooks (footprints.stdtypes.FPList) - rxx - List of callbacks before starting effective task work.

  • Optional. Default is FPList::<<as_list:: []>>.

• scheduler_ticket (builtins.int) - rxx - The slot number given by the scheduler (optional).

  • Optional. Default is None.

expected_time

How long the worker is expected to run (in s) (see the documentation above for more details).

memory

Memory that should be used by the worker (in MiB) (see the documentation above for more details).

name

Name of the worker (see the documentation above for more details).

scheduler_hooks

List of callbacks before starting effective task work (see the documentation above for more details).

scheduler_ticket

The slot number given by the scheduler (optional) (see the documentation above for more details).

class taylorism.Boss(scheduler=None, name=None, verbose=False, maxlenreport=1024, sharedmemory_common_instructions={})

Bases: object

Template for bosses. A Boss is an object supposed to order tasks to a series of workers.

Optionally can be attributed to the Boss a name and a verbose-ity (to report in log, the workers reports).

Also, a scheduler can be assigned, to rule the ordering of tasks to workers. Custom schedulers can be used, they only need to inherit from .schedulers.BaseScheduler and implemented launchable() method.

end()

Ends the listening process once instructions are treated. MUST BE CALLED (or wait_till_finished) for each Boss to avoid zombies processes.

get_report(interim=True)

Get report of the work executed.

If interim, ask for an interim report if no report is available, i.e. containing the work done by the time of calling.

make_them_work(terminate=False, stop_listening=False)

Order the workers to work.

Parameters:

• terminate (bool) – if True, no other instructions could be appended later.

• stop_listening (bool) – if True, alive workers go on their jobs, but they are not listened to anymore; this is a bit tricky but might be useful ?

set_instructions(common_instructions={}, individual_instructions={}, fatal=True)

Set instructions to be distributed to workers.

Parameters:

• common_instructions (dict) – are a series of arguments shared by each worker, to be passed to the Worker factory.

• individual_instructions (dict) – are a series of arguments proper to each worker, hence all individual instructions must have the same length

• fatal (bool) – if True, an error in parsing instructions will stop the workers already running and the boss internal subprocess

stop_them_working()

Stop the workers.

wait_till_finished()

Block the calling tree until all instructions have been executed.

class taylorism.Worker(*args, **kwargs)

Bases: FootprintBase

Template for workers. A Worker is an object supposed to do a task, according to instructions. The instructions has to be added to footprint attributes in actual classes.

Note

This class is managed by footprint.

• info: Not documented

• priority: PriorityLevel::DEFAULT (rank=1)

Automatic parameters from the footprint:

• expected_time (builtins.float) - rxx - How long the worker is expected to run (in s).

  • Optional. Default is 0.0.

• memory (builtins.float) - rxx - Memory that should be used by the worker (in MiB).

  • Optional. Default is 0.0.

• name (builtins.str) - rwx - Name of the worker.

  • Optional. Default is None.

• scheduler_hooks (footprints.stdtypes.FPList) - rxx - List of callbacks before starting effective task work.

  • Optional. Default is FPList::<<as_list:: []>>.

• scheduler_ticket (builtins.int) - rxx - The slot number given by the scheduler (optional).

  • Optional. Default is None.

binding()

Return the actual physical binding of the current process to a cpu if available.

The cpus.CpusToolUnavailableError may be raised depending on the system.

bye()

Block the Boss until the worker has finished his job. THIS METHOD SHOULD NEVER BE CALLED BY THE OBJECT HIMSELF ! (WOULD CAUSE A DEADLOCK if called from inside the worker’s subprocess)

expected_time

How long the worker is expected to run (in s) (see the documentation above for more details).

memory

Memory that should be used by the worker (in MiB) (see the documentation above for more details).

property messenger

Return actual messenger for communication with the boss.

name

Name of the worker (see the documentation above for more details).

scheduler_hooks

List of callbacks before starting effective task work (see the documentation above for more details).

scheduler_ticket

The slot number given by the scheduler (optional) (see the documentation above for more details).

stop_working()

Make the worker stop working.

Since the worker process sets up a Signal handler, it should not stop abruptly when this method is called for the first time…

work()

Send the Worker to his job.

Sub-Modules

taylorism.examples	Basic examples of how to use the module.
taylorism.schedulers	Contains classes for Schedulers.
taylorism.util	Various utility classes and functions to be used with the taylorism package.