cars.orchestrator.cluster.multiprocessing_cluster

Contains abstract function for multiprocessing Cluster

Module Contents

Classes

MultiprocessingCluster	MultiprocessingCluster
MpJob	Encapsulation of multiprocessing job Id (internal use for mp_local_cluster)
MpDelayedTask	Delayed task
MpDelayed	multiprocessing version of dask.delayed
MpFuture	Multiprocessing version of distributed.future
MpFutureTask	multiprocessing version of distributed.future
MpFutureIterator	iterator on multiprocessing.pool.AsyncResult, similar to as_completed

Functions

check_job_done(done_list, obj, current_can_run)	Check if obj is a delayed.
check_tbb_installed(→ bool)	Check if numba finds tbb correctly installed

Attributes

RUN
TERMINATE
REFRESH_TIME
job_counter

cars.orchestrator.cluster.multiprocessing_cluster.RUN = 0

cars.orchestrator.cluster.multiprocessing_cluster.TERMINATE = 1

cars.orchestrator.cluster.multiprocessing_cluster.REFRESH_TIME = 1

cars.orchestrator.cluster.multiprocessing_cluster.job_counter

class cars.orchestrator.cluster.multiprocessing_cluster.MultiprocessingCluster(conf_cluster, out_dir, launch_worker=True)

Bases: cars.orchestrator.cluster.abstract_cluster.AbstractCluster

MultiprocessingCluster

check_conf(conf)

Check configuration

Parameters

conf (dict) – configuration to check

Returns

overloaded configuration

Return type

dict

cleanup()

Cleanup cluster

scatter(data, broadcast=True)

Distribute data through workers

Parameters

data – task data

create_task(func, nout=1)

Create task

Parameters

• func – function

• nout – number of outputs

start_tasks(task_list)

Start all tasks

Parameters

task_list – task list

rec_start(delayed_object, memorize)

Record task

Parameters

• delayed_object (MpDelayed) – delayed object to record

• memorize – list of MpDelayed already recorded

static refresh_task_cache(pool, task_cache, in_queue, per_job_timeout)

Refresh task cache

Parameters

• task_cache – task cache list

• in_queue – queue

future_iterator(future_list)

Start all tasks

Parameters

future_list – future_list list

cars.orchestrator.cluster.multiprocessing_cluster.check_job_done(done_list, obj, current_can_run)

Check if obj is a delayed. If obj is a delayed and job is done, replace it

Parameters

• done_list – list of done tasks

• obj – object to process

• current_can_run – current global can_run

class cars.orchestrator.cluster.multiprocessing_cluster.MpJob(idx, return_index)

Encapsulation of multiprocessing job Id (internal use for mp_local_cluster)

__slots__ = ['task_id', 'r_idx']

class cars.orchestrator.cluster.multiprocessing_cluster.MpDelayedTask(func, args, kw_args)

Delayed task

class cars.orchestrator.cluster.multiprocessing_cluster.MpDelayed(delayed_task, return_index=0)

multiprocessing version of dask.delayed

class cars.orchestrator.cluster.multiprocessing_cluster.MpFuture(mp_future_task, return_index)

Multiprocessing version of distributed.future

cleanup()

Cleanup future

ready()

Check if future is ready

successful()

Check if future is successful

set(sucess, obj)

Set results to future

Parameters

• sucess (bool) – sucess of future

• obj – result

wait(timeout=None)

Wait

Parameters

timeout – timeout to apply

get(timeout=None)

Get result

Parameters

timeout – timeout to apply

class cars.orchestrator.cluster.multiprocessing_cluster.MpFutureTask(cluster)

multiprocessing version of distributed.future

set(obj)

Set result to associated delayed object, and clean cache

Parameters

obj (tuple(bool, Union(dataset, dataframe))) – result object

class cars.orchestrator.cluster.multiprocessing_cluster.MpFutureIterator(future_list, cluster)

iterator on multiprocessing.pool.AsyncResult, similar to as_completed Only returns the actual results, delete the future after usage

__iter__()

Iterate

__next__()

Next

cars.orchestrator.cluster.multiprocessing_cluster.check_tbb_installed() → bool

Check if numba finds tbb correctly installed :return: tbb found