taylorism.schedulers

Contains classes for Schedulers.

Among a set of instructions to be passed to a Worker, and according to its own criteria, the Scheduler determine at the current moment the ones that can be launched right now simultaneously, and those that must be delayed.

A scheduler hence basically has one method: launchable(pending_instructions, workers, report).

Its parameters (constant among execution) can be attributed in its constructor. Other quantities, variables among execution, must be available within workers (work being done) and report (work done).

A set of basic schedulers is given.

Starting from version 1.0.7, schedulers should be created using the footprints package:

import footprints as fp
# In order to create a NewMaxThreadsScheduler scheduler:
mt_sched = fp.proxy.scheduler(limit='threads', max_threads=2)

Compatibility classes are still provided (see LaxistScheduler, MaxThreadsScheduler, MaxMemoryScheduler and SingleOpenFileScheduler) but they should not be used anymore.

Dependencies

footprints (MF package)

Functions

taylorism.schedulers.BindingAwareScheduler(cls)

A class decorator that wraps the original _workers_hooks method to add binding’s setup method to the list Worker’s hooks.

NB: The class’ footprint should include a ‘binded’ attribute.

taylorism.schedulers.binding_setup(worker)

Bind a worker to its scheduler_ticket compute core.

Classes

class taylorism.schedulers.BaseScheduler(*args, **kw)

Bases: FootprintBase

Abstract base class for schedulers.

identity

Scheduler identity.

launchable(pending_instructions, workers, report)

Split pending_instructions into “launchable” and “not_yet_launchable” instructions according to the scheduler own rules.

For that purpose and in a generic manner, the scheduler may need:

• pending_instructions: todo

• workers: being done

• report: done.

class taylorism.schedulers.LaxistScheduler

Bases: _AbstractOldSchedulerProxy

Deprecated class: should not be used from now and on.

class taylorism.schedulers.LongerFirstScheduler(*args, **kw)

Bases: NewMaxMemoryScheduler

A scheduler based on the NewMaxMemory Scheduler. It aims at launching as soon as possible the workers that are expected to have the longest run-time (whilst there is enough available memory).

Workers needs to have 2 attributes:

• expected_time representing the expected run time

• memory representing the expected memory consumed

This scheduler is typically used with Bateur workers, in vortex’s src/common/algo/odbtools.py (for parallel BATOR run).

Setup the maximum available memory.

binded

Binds the process to a single cpu.

identity

Scheduler identity.

launchable(pending_instructions, workers, report)

Limit strategy: only processes that fit in a given amount of memory could run.

max_memory

Amount of usable memroy (in MiB)

memory_max_percentage

Max memory level as a fraction of the totalsystem memory (used only if max_memroy is not provided).

memory_per_task

If a worker do not provide any information on memory, request at least memory_per_task MiB of memory.

class taylorism.schedulers.MaxMemoryScheduler(max_memory_percentage=0.75, total_system_memory=None)

Bases: _AbstractOldSchedulerProxy

Deprecated class: should not be used from now and on.

Parameters:

• max_memory_percentage (float) – Max memory level as a fraction of the total system memory

• total_system_memory (float) – Memory available on this system (in MiB)

class taylorism.schedulers.MaxThreadsScheduler(max_threads=0)

Bases: _AbstractOldSchedulerProxy

Deprecated class: should not be used from now and on.

class taylorism.schedulers.NewLaxistScheduler(*args, **kw)

Bases: BaseScheduler

No sorting is done !

identity

Scheduler identity.

launchable(pending_instructions, workers, report)

Very crude strategy: any pending instruction could be triggered.

class taylorism.schedulers.NewLimitedScheduler(*args, **kw)

Bases: BaseScheduler

A scheduler that dequeue the pending list as long as a maximum number of simultaneous tasks (max_threads) is not reached.

identity

Scheduler identity.

class taylorism.schedulers.NewMaxMemoryScheduler(*args, **kw)

Bases: NewLimitedScheduler

A basic scheduler that dequeue the pending list as long as a critical memory level (according to ‘memory’ element of workers instructions (in MB) and total system memory) is not reached.

Setup the maximum available memory.

binded

Binds the process to a single cpu.

identity

Scheduler identity.

launchable(pending_instructions, workers, report)

Limit strategy: only processes that fit in a given amount of memory could run.

max_memory

Amount of usable memroy (in MiB)

memory_max_percentage

Max memory level as a fraction of the totalsystem memory (used only if max_memroy is not provided).

memory_per_task

If a worker do not provide any information on memory, request at least memory_per_task MiB of memory.

class taylorism.schedulers.NewMaxThreadsScheduler(*args, **kw)

Bases: NewLimitedScheduler

A basic scheduler that dequeue the pending list as long as a maximum number of simultaneous tasks (max_threads) is not reached.

binded

Binds the process to a single cpu.

identity

Scheduler identity.

launchable(pending_instructions, workers, report)

Limit strategy: only max_threads processes could run simultaneously.

class taylorism.schedulers.NewSingleOpenFileScheduler(*args, **kw)

Bases: NewMaxThreadsScheduler

Ensure that files will not be open 2 times simultaneously by 2 workers. And with a maximum threads number.

binded

Binds the process to a single cpu.

identity

Scheduler identity.

launchable(pending_instructions, workers, report)

Limit strategy: deal with what looks like a file locking.

class taylorism.schedulers.SingleOpenFileScheduler(max_threads=0)

Bases: _AbstractOldSchedulerProxy

Deprecated class: should not be used from now and on.