cars.orchestrator.cluster.mp_cluster.mp_wrapper

Contains functions for wrapper disk

Module Contents

Classes

AbstractWrapper

AbstractWrapper

WrapperNone

AbstractWrapper

WrapperDisk

WrapperDisk

Functions

removing_disk_data(path)

Remove directory from disk

none_wrapper_fun(*argv, **kwargs)

Create a wrapper for functionn running it

disk_wrapper_fun(*argv, **kwargs)

Create a wrapper for function

load_args_or_kwargs(args_or_kwargs)

Load args or kwargs from disk to memory

is_dumped_object(obj)

Check if a given object is dumped

load(path)

Load object from disk

load_shared_data(path)

Load shared object from disk

dump_single_object(obj, path)

Dump object to disk

create_path(obj, tmp_dir, id_num)

Create path where to dump object

dump(res, tmp_dir, id_list)

Dump results to tmp_dir, according to ids

Attributes

DENSE_NAME

SPARSE_NAME

DICT_NAME

SHARED_NAME

cars.orchestrator.cluster.mp_cluster.mp_wrapper.DENSE_NAME = 'DenseDO'
cars.orchestrator.cluster.mp_cluster.mp_wrapper.SPARSE_NAME = 'SparseDO'
cars.orchestrator.cluster.mp_cluster.mp_wrapper.DICT_NAME = 'DictDO'
cars.orchestrator.cluster.mp_cluster.mp_wrapper.SHARED_NAME = 'SharedDO'
class cars.orchestrator.cluster.mp_cluster.mp_wrapper.AbstractWrapper[source]

AbstractWrapper

abstract get_obj(obj)[source]

Get Object

Parameters

obj – object to transform

Returns

object

abstract get_function_and_kwargs(func, kwargs, nout=1)[source]

Get function to apply and overloaded key arguments

Parameters

  • func – function to run

  • kwargs – key arguments of func

  • nout – number of outputs

Returns

function to apply, overloaded key arguments

abstract cleanup(**kwargs)[source]

Cleanup tmp_dir

abstract cleanup_future_res(future_res)[source]

Cleanup future result

Parameters

future_res – future result to clean

class cars.orchestrator.cluster.mp_cluster.mp_wrapper.WrapperNone(tmp_dir)[source]

Bases: AbstractWrapper

AbstractWrapper

get_obj(obj)[source]

Get Object

Parameters

obj – object to transform

Returns

object

get_function_and_kwargs(func, kwargs, nout=1)[source]

Get function to apply and overloaded key arguments

Parameters

  • func – function to run

  • kwargs – key arguments of func

  • nout – number of outputs

Returns

function to apply, overloaded key arguments

cleanup(**kwargs)[source]

Cleanup tmp_dir

cleanup_future_res(future_res)[source]

Cleanup future result

Parameters

future_res – future result to clean

class cars.orchestrator.cluster.mp_cluster.mp_wrapper.WrapperDisk(tmp_dir)[source]

Bases: AbstractWrapper

WrapperDisk

cleanup(keep_shared_dir=False)[source]

Cleanup tmp_dir :param keep_shared_dir: do not clean directory of shared objects

cleanup_future_res(future_res)[source]

Cleanup future result

Parameters

future_res – future result to clean

get_function_and_kwargs(func, kwargs, nout=1)[source]

Get function to apply and overloaded key arguments

Parameters

  • func – function to run

  • kwargs – key arguments of func

  • nout – number of outputs

Returns

function to apply, overloaded key arguments

get_obj(obj)[source]

Get Object

Parameters

obj – object to transform

Returns

object

scatter_obj(obj)[source]

Distribute data through workers

Parameters

obj – object to dump

cars.orchestrator.cluster.mp_cluster.mp_wrapper.removing_disk_data(path)[source]

Remove directory from disk

Parameters

path – path to delete

cars.orchestrator.cluster.mp_cluster.mp_wrapper.none_wrapper_fun(*argv, **kwargs)[source]

Create a wrapper for functionn running it

Parameters

  • argv – args of func

  • kwargs – kwargs of func

Returns

path to results

cars.orchestrator.cluster.mp_cluster.mp_wrapper.disk_wrapper_fun(*argv, **kwargs)[source]

Create a wrapper for function

Parameters

  • argv – args of func

  • kwargs – kwargs of func

Returns

path to results

cars.orchestrator.cluster.mp_cluster.mp_wrapper.load_args_or_kwargs(args_or_kwargs)[source]

Load args or kwargs from disk to memory

Parameters

args_or_kwargs – args or kwargs of func

Returns

new args

cars.orchestrator.cluster.mp_cluster.mp_wrapper.is_dumped_object(obj)[source]

Check if a given object is dumped

Parameters

obj – object

Returns

is dumped

Return type

bool

cars.orchestrator.cluster.mp_cluster.mp_wrapper.load(path)[source]

Load object from disk

Parameters

path (str) – path

Returns

object

cars.orchestrator.cluster.mp_cluster.mp_wrapper.load_shared_data(path)[source]

Load shared object from disk

Parameters

path (str) – path

Returns

object

cars.orchestrator.cluster.mp_cluster.mp_wrapper.dump_single_object(obj, path)[source]

Dump object to disk

Parameters

path (str) – path

cars.orchestrator.cluster.mp_cluster.mp_wrapper.create_path(obj, tmp_dir, id_num)[source]

Create path where to dump object

Parameters

  • tmp_dir – tmp_dir

  • id_num – id of object

Returns

path

cars.orchestrator.cluster.mp_cluster.mp_wrapper.dump(res, tmp_dir, id_list)[source]

Dump results to tmp_dir, according to ids

Parameters

  • res – objects to dump

  • tmp_dir – tmp_dir

  • id_list – list of ids of objects

Returns

path