cars.orchestrator.cluster.mp_objects

Contains class objects used by multiprocessing cluster

Module Contents

Classes

MpJob	Encapsulation of multiprocessing job Id (internal use for mp_local_cluster)
MpDelayedTask	Delayed task
MpDelayed	multiprocessing version of dask.delayed
MpFuture	Multiprocessing version of distributed.future
MpFutureIterator	iterator on multiprocessing.pool.AsyncResult, similar to as_completed

class cars.orchestrator.cluster.mp_objects.MpJob(idx, return_index)

Encapsulation of multiprocessing job Id (internal use for mp_local_cluster)

__slots__ = ['task_id', 'r_idx']

class cars.orchestrator.cluster.mp_objects.MpDelayedTask(func, args, kw_args)

Delayed task

__repr__()

Repr function :return: printable self CarsDataset

__str__()

Str function :return: printable self CarsDataset

custom_print()

Return string of self :return : printable delayed

modify_delayed_task(wrapper)

Modify delayed to add wrapper (disk, None)

Parameters

wrapper (fun) – wrapper function

class cars.orchestrator.cluster.mp_objects.MpDelayed(delayed_task, return_index=0)

multiprocessing version of dask.delayed

__repr__()

Repr function :return: printable self CarsDataset

__str__()

Str function :return: printable self CarsDataset

custom_print()

Return string of self :return : printable delayed

get_depending_delayed()

Get all the delayed that current delayed depends on

:return list of depending delayed :rtype: list(MpDelayed)

class cars.orchestrator.cluster.mp_objects.MpFuture(mp_future_task, return_index)

Multiprocessing version of distributed.future

cleanup()

Cleanup future

ready()

Check if future is ready

successful()

Check if future is successful

set(success, obj)

Set results to future

Parameters

• success (bool) – success of future

• obj – result

wait(timeout=None)

Wait

Parameters

timeout – timeout to apply

get(timeout=None)

Get result

Parameters

timeout – timeout to apply

class cars.orchestrator.cluster.mp_objects.MpFutureIterator(future_list, cluster)

iterator on multiprocessing.pool.AsyncResult, similar to as_completed Only returns the actual results, delete the future after usage

__iter__()

Iterate

__next__()

Next