cars.orchestrator.cluster.mp_cluster.mp_objects

Contains class objects used by multiprocessing cluster

Classes

`MpJob`	Encapsulation of multiprocessing job Id (internal use for mp_local_cluster)
`MpDelayedTask`	Delayed task
`MpDelayed`	multiprocessing version of dask.delayed
`MpFuture`	Multiprocessing version of distributed.future
`MpFutureIterator`	iterator on multiprocessing.pool.AsyncResult, similar to as_completed
`PreviousData`	Object used in FactorisedObject for args already computed
`FactorizedObject`	Object used as args of function factorised_func

Functions

`transform_mp_delayed_to_previous_data`(obj)	Replace MpDelayed by PreviousData object
`transform_previous_data_to_results`(obj, res)	Replace PreviousData object by real data

Module Contents

class cars.orchestrator.cluster.mp_cluster.mp_objects.MpJob(idx, return_index)[source]

Encapsulation of multiprocessing job Id (internal use for mp_local_cluster)

__slots__ = ['task_id', 'r_idx']

task_id

r_idx

class cars.orchestrator.cluster.mp_cluster.mp_objects.MpDelayedTask(func, args, kw_args)[source]

Delayed task

func

args

kw_args

associated_objects = []

__repr__()[source]: Repr function :return: printable self CarsDataset

__str__()[source]: Str function :return: printable self CarsDataset

custom_print()[source]: Return string of self :return : printable delayed

modify_delayed_task(wrapper)[source]

Modify delayed to add wrapper (disk, None)

Parameters:: wrapper (fun) – wrapper function

class cars.orchestrator.cluster.mp_cluster.mp_objects.MpDelayed(delayed_task, return_index=0)[source]

multiprocessing version of dask.delayed

delayed_task

return_index = 0

__repr__()[source]: Repr function :return: printable self CarsDataset

__str__()[source]: Str function :return: printable self CarsDataset

custom_print()[source]: Return string of self :return : printable delayed

get_depending_delayed()[source]

Get all the delayed that current delayed depends on

:return list of depending delayed :rtype: list(MpDelayed)

class cars.orchestrator.cluster.mp_cluster.mp_objects.MpFuture(mp_future_task, return_index)[source]

Multiprocessing version of distributed.future

mp_future_task

result = None

_success = None

return_index

event

cleanup()[source]: Cleanup future

ready()[source]: Check if future is ready

successful()[source]: Check if future is successful

set(success, obj)[source]

Set results to future

Parameters:

success (bool) – success of future
obj – result

wait(timeout=None)[source]

Wait

Parameters:: timeout – timeout to apply

get(timeout=None)[source]

Get result

Parameters:: timeout – timeout to apply

class cars.orchestrator.cluster.mp_cluster.mp_objects.MpFutureIterator(future_list, cluster, timeout=None)[source]

iterator on multiprocessing.pool.AsyncResult, similar to as_completed Only returns the actual results, delete the future after usage

future_list

cluster

was_killed = False

timeout = None

past_time

__iter__()[source]: Iterate

__next__()[source]: Next

class cars.orchestrator.cluster.mp_cluster.mp_objects.PreviousData(delayed)[source]

Object used in FactorisedObject for args already computed in factorized function

return_index

cars.orchestrator.cluster.mp_cluster.mp_objects.transform_mp_delayed_to_previous_data(obj)[source]

Replace MpDelayed by PreviousData object

Parameters:: data – data to replace if necessary

cars.orchestrator.cluster.mp_cluster.mp_objects.transform_previous_data_to_results(obj, res)[source]

Replace PreviousData object by real data

Parameters:: data – data to replace if necessary

class cars.orchestrator.cluster.mp_cluster.mp_objects.FactorizedObject(current_task, previous_task)[source]

Object used as args of function factorised_func It contains several tasks that can be run within a single function

__str__()[source]

get_args()[source]: Get args of first task to execute

set_args(args)[source]

Set args of first task to execute

Parameters:: args – arguments to set

get_kwargs()[source]: Get kwargs of first task to execute

set_kwargs(kwargs)[source]

Set kwargs of first task to execute

Parameters:: args – keyword arguments to set

pop_next_task(previous_result=None)[source]

Run the next task to execute, remove it from the list and return the result

Parameters:: previous_result – output of previous task