cars.orchestrator.cluster.mp_factorizer

Contains functions needed to factorize delayed

Module Contents

Functions

factorize_delayed(task_list)

Factorize Task list

factorize_delayed_rec(delayed, graph_usages, ...)

Factorize Task list

compute_graph_delayed_usages(task_list)

Compute the number of times every delayed is used in graph

get_delayed_usage_rec(delayed, graph_usages, ...)

Get number of time input delayed is used

number_of_usage(delayed, graph_usages)

Compute the number of times a delayed is used

compute_nb_depending_task(depending_delayed_list)

Compute the number of different delayed task in list of delayed

factorized_fun(*args, **kwargs)

This function unpack multiple functions with their arguments,

get_number_of_steps(kwargs)

Get number of following function stored in kwargs

generate_args_kwargs_factorize(fun1, args1, kwargs1, ...)

Generate args for new delayed
cars.orchestrator.cluster.mp_factorizer.factorize_delayed(task_list)

Factorize Task list

Only factorize tasks that depends on a single task

Parameters

task_list (list(MpDelayed)) – list of delayed

cars.orchestrator.cluster.mp_factorizer.factorize_delayed_rec(delayed, graph_usages, already_seen_delayed_tasks)

Factorize Task list

Only factorize tasks that depends on a single task

Parameters

  • delayed (MpDelayed) – delayed to factorize

  • graph_usages (dict example: {delayed1: 3}) – number of usages of delayed

  • already_seen_delayed_tasks – list of MpDelayedTask already seen

cars.orchestrator.cluster.mp_factorizer.compute_graph_delayed_usages(task_list)

Compute the number of times every delayed is used in graph

Parameters

task_list (list(MpDelayed)) – list of delayed

Returns

number of usages of delayed

Return type

dict example: {delayed1: 3}

cars.orchestrator.cluster.mp_factorizer.get_delayed_usage_rec(delayed, graph_usages, already_seen_tasks)

Get number of time input delayed is used

Parameters

  • delayed (MpDelayed) – delayed to factorize

  • graph_usages (dict example: {delayed1: 3}) – number of usages of delayed

  • already_seen_tasks (list[MpDelayedTask]) – list of seen delayed task

cars.orchestrator.cluster.mp_factorizer.number_of_usage(delayed, graph_usages)

Compute the number of times a delayed is used

Parameters

  • delayed (MpDelayed) – delayed to factorize

  • graph_usages (dict example: {delayed1: 3}) – number of usages of delayed

Returns

number of usages of delayed

Return type

int

cars.orchestrator.cluster.mp_factorizer.compute_nb_depending_task(depending_delayed_list)

Compute the number of different delayed task in list of delayed

Parameters

depending_delayed_list (list[MpDelayed]) – list of delayed

Returns

number of depending task

Return type

int

cars.orchestrator.cluster.mp_factorizer.factorized_fun(*args, **kwargs)

This function unpack multiple functions with their arguments, and run them sequentialy

cars.orchestrator.cluster.mp_factorizer.get_number_of_steps(kwargs)

Get number of following function stored in kwargs

Parameters

kwargs (dict) – key arguments of function

Returns

number of steps

Return type

int

cars.orchestrator.cluster.mp_factorizer.generate_args_kwargs_factorize(fun1, args1, kwargs1, fun2, args2, kwargs2)

Generate args for new delayed

WARNING : modify inplace args and kwargs if you compute inbetween the function errors will occure

Parameters

  • fun1 (callable) – first function to be called

  • args1 (list) – list of arguments of function 1

  • kwargs1 (dict) – dict of keyword arguments of function 1

  • args2 (list) – list of arguments of function 2

  • kwargs2 (dict) – dict of keyword arguments of function 2

Pram fun2

second function to be called

Returns

new function, new arguments, new keyword arguments

Return type

function, list, dict