cars.orchestrator.cluster.log_wrapper

Contains functions for wrapper logs

Module Contents

Classes

AbstractLogWrapper	AbstractLogWrapper
LogWrapper	LogWrapper
CProfileWrapper	CProfileWrapper
MemrayWrapper	MemrayWrapper
CarsMemProf	CarsMemProf

Functions

log_function(*argv, **kwargs)	Create a wrapper for function running it
time_profiling_function(*argv, **kwargs)	Create a wrapper to profile the function elapse time
memory_profiling_function(*argv, **kwargs)	Create a wrapper to profile the function occupation memory
switch_messages(func, total_time[, max_memory])	create profile message with specific message
log_message(func, message)	log profiling message
loop_function(argv, kwargs, func[, nb_iteration])	generate a loop on each cluster function to eval possible leak
get_current_memory()	Get current memory of process
log_delta_memory(func, memory_start, memory_end)	Log memory infos
generate_summary(out_dir, used_conf[, clean_worker_logs])	Generate Profiling summary
filter_lists(names, data, cond)	Filter lists with condition on name
generate_boxplot(axis, names, data_full, title, data_type)	Generate boxplot
generate_histo(axis, names, data, title, data_type[, ...])	Generate histogram
generate_pie_chart(axis, names, data, title)	Generate pie chart, data in %
cars_profile([name, interval])	CARS profiling decorator

Attributes

THREAD_TIMEOUT

cars.orchestrator.cluster.log_wrapper.THREAD_TIMEOUT = 2

class cars.orchestrator.cluster.log_wrapper.AbstractLogWrapper(conf_profiling, out_dir)

AbstractLogWrapper

available_modes

classmethod register_subclass(short_name: str)

Allows to register the subclass with its short name :param short_name: the subclass to be registered :type short_name: string

abstract check_conf(conf)

Check configuration

Parameters

conf (dict) – configuration to check

Returns

overloaded configuration

Return type

dict

abstract get_func_args_plus(func)

getter for the args of the future function

Param

function to apply

Returns

function to apply, overloaded key arguments

class cars.orchestrator.cluster.log_wrapper.LogWrapper(conf_profiling, out_dir)

Bases: AbstractLogWrapper

LogWrapper

simple log wrapper doing nothing

check_conf(conf)

Check configuration

Parameters

conf (dict) – configuration to check

Returns

overloaded configuration

Return type

dict

get_func_args_plus(func)

getter for the args of the future function

Param

function to apply

Returns

function to apply, overloaded key arguments

class cars.orchestrator.cluster.log_wrapper.CProfileWrapper(conf_profiling, out_dir)

Bases: AbstractLogWrapper

CProfileWrapper

log wrapper to analyze the internal time consuming of the function. The wrapper use cprofile API.

check_conf(conf)

Check configuration

Parameters

conf (dict) – configuration to check

Returns

overloaded configuration

Return type

dict

get_func_args_plus(func)

getter for the args of the future function

Param

function to apply

Returns

function to apply, overloaded key arguments

class cars.orchestrator.cluster.log_wrapper.MemrayWrapper(conf_profiling, out_dir)

Bases: AbstractLogWrapper

MemrayWrapper

log wrapper to analyze the internal allocation memory consuming of the function. The wrapper use cprofile API.

check_conf(conf)

Check configuration

Parameters

conf (dict) – configuration to check

Returns

overloaded configuration

Return type

dict

get_func_args_plus(func)

getter for the args of the future function

Param

function to apply

Returns

function to apply, overloaded key arguments

cars.orchestrator.cluster.log_wrapper.log_function(*argv, **kwargs)

Create a wrapper for function running it

Parameters

• argv – args of func

• kwargs – kwargs of func

Returns

path to results

cars.orchestrator.cluster.log_wrapper.time_profiling_function(*argv, **kwargs)

Create a wrapper to profile the function elapse time

Parameters

• argv – args of func

• kwargs – kwargs of func

Returns

path to results

cars.orchestrator.cluster.log_wrapper.memory_profiling_function(*argv, **kwargs)

Create a wrapper to profile the function occupation memory

Parameters

• argv – args of func

• kwargs – kwargs of func

Returns

path to results

cars.orchestrator.cluster.log_wrapper.switch_messages(func, total_time, max_memory=None)

create profile message with specific message depends on elapsed time (LONG, FAST…).

:param func : profiled function :param total_time : elapsed time of the function

cars.orchestrator.cluster.log_wrapper.log_message(func, message)

log profiling message

:param func : logged function :param message : log message

cars.orchestrator.cluster.log_wrapper.loop_function(argv, kwargs, func, nb_iteration=5)

generate a loop on each cluster function to eval possible leak

:param argv : input argv :param kwargs : input kwargs :param func : function to evaluation :param nb_iteration (int, optional): number of the iteration loop. :param Defaults to 5.

Returns:

_type_: result of the function

cars.orchestrator.cluster.log_wrapper.get_current_memory()

Get current memory of process

Returns

memory

Return type

float

cars.orchestrator.cluster.log_wrapper.log_delta_memory(func, memory_start, memory_end)

Log memory infos

Parameters

• func – profiled function

• memory_start (float) – memory before the run of function

• memory_end (float) – memory after the run of function

cars.orchestrator.cluster.log_wrapper.generate_summary(out_dir, used_conf, clean_worker_logs=False)

Generate Profiling summary

cars.orchestrator.cluster.log_wrapper.filter_lists(names, data, cond)

Filter lists with condition on name

cars.orchestrator.cluster.log_wrapper.generate_boxplot(axis, names, data_full, title, data_type)

Generate boxplot

cars.orchestrator.cluster.log_wrapper.generate_histo(axis, names, data, title, data_type, data_min_err=None, data_max_err=None)

Generate histogram

cars.orchestrator.cluster.log_wrapper.generate_pie_chart(axis, names, data, title)

Generate pie chart, data in %

cars.orchestrator.cluster.log_wrapper.cars_profile(name=None, interval=0.1)

CARS profiling decorator

Param

func: function to monitor

class cars.orchestrator.cluster.log_wrapper.CarsMemProf(pid, pipe, interval=0.1)

Bases: threading.Thread

CarsMemProf

Profiling thread

run()

Run