Welcome to taylorism’s documentation!

A framework for parallelisation of tasks.

The package works with three levels of subprocessing:

  • the main program, in which are defined each task “instructions”, and that communicates with the intermediate subprocess

  • the intermediate subprocess, that receive instructions sets from the main program and distributes them to a range of subprocesses

  • the working subprocesses, that are defined by their instructions, execute it and then return to the intermediate process.

This segmentation takes the image of a Boss distributing instructions to Workers, and getting their report when they finish.

The main program can:

  • initiate the Boss class

  • give instructions to the Boss, for them to be executed (by the Worker objects)

  • tell the Boss to send the Worker objects to work

  • tell the Boss to stop the Worker objects

  • inform the Boss that there will not be any more instruction to do

  • wait until all instructions have been executed

  • get report (interim or final) from the Boss (composed of Worker objects reports)

The Boss (in its subprocess loop):

  • receives either series of instructions or “control signals” from the main program. Control signals concern:

    • halt/resume execution of instructions

    • request for (interim) report

    • end of instructions information

    • stop signals (letting workers finish current tasks or not)

  • collects reports from the workers, with handling of their errors: if an error occurs in a subprocess, all other subprocesses are stopped and the error is returned to the main program.

  • schedule execution of instructions, asking a Scheduler with regards to:

    • pending instructions (work to do)

    • current workers (work being done)

    • previous reports (work already done)

    (cf. taylorism.schedulers doc for more details).

The Worker objects are defined BY their instructions (their attributes), and their task (what they do with their instructions). Each Worker is an independent subprocess, triggered by the Boss subprocess and supposed to return to it the report of its work when finished.

Use of the module

A proper use is to define a custom worker class inheriting from Worker, with all its necessary instructions as footprints attributes. Then implement the task to be done in the _task() method.

The so-implemented workers will then be “hired”=generated by the Boss as soon as the instructions given to the Boss meet those of the workers (footprints magics !).

Cf. taylorism.examples for a simple example.

Dependencies

footprints (MF package)

bronx (MF package)

Package auto-documentation

For more details, please dive in the documentation embedded in sub-modules:

taylorism

Framework for parallelisation of tasks.

Indices and tables