cars.orchestrator.cluster.mp_cluster.mp_objects
Contains class objects used by multiprocessing cluster
Module Contents
Classes
Encapsulation of multiprocessing job Id (internal use for mp_local_cluster) |
|
Delayed task |
|
multiprocessing version of dask.delayed |
|
Multiprocessing version of distributed.future |
|
iterator on multiprocessing.pool.AsyncResult, similar to as_completed |
|
Object used in FactorisedObject for args already computed |
|
Object used as args of function factorised_func |
Functions
Replace MpDelayed by PreviousData object |
|
|
Replace PreviousData object by real data |
- class cars.orchestrator.cluster.mp_cluster.mp_objects.MpJob(idx, return_index)
Encapsulation of multiprocessing job Id (internal use for mp_local_cluster)
- __slots__ = ['task_id', 'r_idx']
- class cars.orchestrator.cluster.mp_cluster.mp_objects.MpDelayedTask(func, args, kw_args)
Delayed task
- __repr__()
Repr function :return: printable self CarsDataset
- __str__()
Str function :return: printable self CarsDataset
- custom_print()
Return string of self :return : printable delayed
- modify_delayed_task(wrapper)
Modify delayed to add wrapper (disk, None)
- Parameters
wrapper (fun) – wrapper function
- class cars.orchestrator.cluster.mp_cluster.mp_objects.MpDelayed(delayed_task, return_index=0)
multiprocessing version of dask.delayed
- __repr__()
Repr function :return: printable self CarsDataset
- __str__()
Str function :return: printable self CarsDataset
- custom_print()
Return string of self :return : printable delayed
- get_depending_delayed()
Get all the delayed that current delayed depends on
:return list of depending delayed :rtype: list(MpDelayed)
- class cars.orchestrator.cluster.mp_cluster.mp_objects.MpFuture(mp_future_task, return_index)
Multiprocessing version of distributed.future
- cleanup()
Cleanup future
- ready()
Check if future is ready
- successful()
Check if future is successful
- set(success, obj)
Set results to future
- Parameters
success (bool) – success of future
obj – result
- wait(timeout=None)
Wait
- Parameters
timeout – timeout to apply
- get(timeout=None)
Get result
- Parameters
timeout – timeout to apply
- class cars.orchestrator.cluster.mp_cluster.mp_objects.MpFutureIterator(future_list, cluster, timeout=None)
iterator on multiprocessing.pool.AsyncResult, similar to as_completed Only returns the actual results, delete the future after usage
- __iter__()
Iterate
- __next__()
Next
- class cars.orchestrator.cluster.mp_cluster.mp_objects.PreviousData(delayed)
Object used in FactorisedObject for args already computed in factorized function
- cars.orchestrator.cluster.mp_cluster.mp_objects.transform_mp_delayed_to_previous_data(obj)
Replace MpDelayed by PreviousData object
- Parameters
data – data to replace if necessary
- cars.orchestrator.cluster.mp_cluster.mp_objects.transform_previous_data_to_results(obj, res)
Replace PreviousData object by real data
- Parameters
data – data to replace if necessary
- class cars.orchestrator.cluster.mp_cluster.mp_objects.FactorizedObject(current_task, previous_task)
Object used as args of function factorised_func It contains several tasks that can be run within a single function
- __str__()
Return str(self).
- get_args()
Get args of first task to execute
- set_args(args)
Set args of first task to execute
- Parameters
args – arguments to set
- get_kwargs()
Get kwargs of first task to execute
- set_kwargs(kwargs)
Set kwargs of first task to execute
- Parameters
args – keyword arguments to set
- pop_next_task(previous_result=None)
Run the next task to execute, remove it from the list and return the result
- Parameters
previous_result – output of previous task