cars.orchestrator.cluster.mp_cluster.mp_objects

Contains class objects used by multiprocessing cluster

Module Contents

Classes

MpJob	Encapsulation of multiprocessing job Id (internal use for mp_local_cluster)
MpDelayedTask	Delayed task
MpDelayed	multiprocessing version of dask.delayed
MpFuture	Multiprocessing version of distributed.future
MpFutureIterator	iterator on multiprocessing.pool.AsyncResult, similar to as_completed
PreviousData	Object used in FactorisedObject for args already computed
FactorizedObject	Object used as args of function factorised_func

Functions

transform_mp_delayed_to_previous_data(obj)	Replace MpDelayed by PreviousData object
transform_previous_data_to_results(obj, res)	Replace PreviousData object by real data

class cars.orchestrator.cluster.mp_cluster.mp_objects.MpJob(idx, return_index)

Encapsulation of multiprocessing job Id (internal use for mp_local_cluster)

__slots__ = ['task_id', 'r_idx']

class cars.orchestrator.cluster.mp_cluster.mp_objects.MpDelayedTask(func, args, kw_args)

Delayed task

__repr__()

Repr function :return: printable self CarsDataset

__str__()

Str function :return: printable self CarsDataset

custom_print()

Return string of self :return : printable delayed

modify_delayed_task(wrapper)

Modify delayed to add wrapper (disk, None)

Parameters

wrapper (fun) – wrapper function

class cars.orchestrator.cluster.mp_cluster.mp_objects.MpDelayed(delayed_task, return_index=0)

multiprocessing version of dask.delayed

__repr__()

Repr function :return: printable self CarsDataset

__str__()

Str function :return: printable self CarsDataset

custom_print()

Return string of self :return : printable delayed

get_depending_delayed()

Get all the delayed that current delayed depends on

:return list of depending delayed :rtype: list(MpDelayed)

class cars.orchestrator.cluster.mp_cluster.mp_objects.MpFuture(mp_future_task, return_index)

Multiprocessing version of distributed.future

cleanup()

Cleanup future

ready()

Check if future is ready

successful()

Check if future is successful

set(success, obj)

Set results to future

Parameters

• success (bool) – success of future

• obj – result

wait(timeout=None)

Wait

Parameters

timeout – timeout to apply

get(timeout=None)

Get result

Parameters

timeout – timeout to apply

class cars.orchestrator.cluster.mp_cluster.mp_objects.MpFutureIterator(future_list, cluster, timeout=None)

iterator on multiprocessing.pool.AsyncResult, similar to as_completed Only returns the actual results, delete the future after usage

__iter__()

Iterate

__next__()

Next

class cars.orchestrator.cluster.mp_cluster.mp_objects.PreviousData(delayed)

Object used in FactorisedObject for args already computed in factorized function

cars.orchestrator.cluster.mp_cluster.mp_objects.transform_mp_delayed_to_previous_data(obj)

Replace MpDelayed by PreviousData object

Parameters

data – data to replace if necessary

cars.orchestrator.cluster.mp_cluster.mp_objects.transform_previous_data_to_results(obj, res)

Replace PreviousData object by real data

Parameters

data – data to replace if necessary

class cars.orchestrator.cluster.mp_cluster.mp_objects.FactorizedObject(current_task, previous_task)

Object used as args of function factorised_func It contains several tasks that can be run within a single function

__str__()

Return str(self).

get_args()

Get args of first task to execute

set_args(args)

Set args of first task to execute

Parameters

args – arguments to set

get_kwargs()

Get kwargs of first task to execute

set_kwargs(kwargs)

Set kwargs of first task to execute

Parameters

args – keyword arguments to set

pop_next_task(previous_result=None)

Run the next task to execute, remove it from the list and return the result

Parameters

previous_result – output of previous task