cars.orchestrator.cluster.log_wrapper

Contains functions for wrapper logs

Module Contents

Classes

AbstractLogWrapper

AbstractLogWrapper

LogWrapper

LogWrapper

CProfileWrapper

CProfileWrapper

MemrayWrapper

MemrayWrapper

CarsMemProf

CarsMemProf

Functions

log_function(*argv, **kwargs)

Create a wrapper for function running it

time_profiling_function(*argv, **kwargs)

Create a wrapper to profile the function elapse time

memory_profiling_function(*argv, **kwargs)

Create a wrapper to profile the function occupation memory

switch_messages(func, total_time[, max_memory])

create profile message with specific message

log_message(func, message)

log profiling message

loop_function(argv, kwargs, func[, nb_iteration])

generate a loop on each cluster function to eval possible leak

get_current_memory()

Get current memory of process

log_delta_memory(func, memory_start, memory_end)

Log memory infos

generate_summary(out_dir, used_conf[, clean_worker_logs])

Generate Profiling summary

filter_lists(names, data, cond)

Filter lists with condition on name

generate_boxplot(axis, names, data_full, title, data_type)

Generate boxplot

generate_histo(axis, names, data, title, data_type[, ...])

Generate histogram

generate_pie_chart(axis, names, data, title)

Generate pie chart, data in %

cars_profile([name, interval])

CARS profiling decorator

Attributes

THREAD_TIMEOUT

cars.orchestrator.cluster.log_wrapper.THREAD_TIMEOUT = 2
class cars.orchestrator.cluster.log_wrapper.AbstractLogWrapper(conf_profiling, out_dir)

AbstractLogWrapper

available_modes
classmethod register_subclass(short_name: str)

Allows to register the subclass with its short name :param short_name: the subclass to be registered :type short_name: string

abstract check_conf(conf)

Check configuration

Parameters

conf (dict) – configuration to check

Returns

overloaded configuration

Return type

dict

abstract get_func_args_plus(func)

getter for the args of the future function

Param

function to apply

Returns

function to apply, overloaded key arguments

class cars.orchestrator.cluster.log_wrapper.LogWrapper(conf_profiling, out_dir)

Bases: AbstractLogWrapper

LogWrapper

simple log wrapper doing nothing

check_conf(conf)

Check configuration

Parameters

conf (dict) – configuration to check

Returns

overloaded configuration

Return type

dict

get_func_args_plus(func)

getter for the args of the future function

Param

function to apply

Returns

function to apply, overloaded key arguments

class cars.orchestrator.cluster.log_wrapper.CProfileWrapper(conf_profiling, out_dir)

Bases: AbstractLogWrapper

CProfileWrapper

log wrapper to analyze the internal time consuming of the function. The wrapper use cprofile API.

check_conf(conf)

Check configuration

Parameters

conf (dict) – configuration to check

Returns

overloaded configuration

Return type

dict

get_func_args_plus(func)

getter for the args of the future function

Param

function to apply

Returns

function to apply, overloaded key arguments

class cars.orchestrator.cluster.log_wrapper.MemrayWrapper(conf_profiling, out_dir)

Bases: AbstractLogWrapper

MemrayWrapper

log wrapper to analyze the internal allocation memory consuming of the function. The wrapper use cprofile API.

check_conf(conf)

Check configuration

Parameters

conf (dict) – configuration to check

Returns

overloaded configuration

Return type

dict

get_func_args_plus(func)

getter for the args of the future function

Param

function to apply

Returns

function to apply, overloaded key arguments

cars.orchestrator.cluster.log_wrapper.log_function(*argv, **kwargs)

Create a wrapper for function running it

Parameters
  • argv – args of func

  • kwargs – kwargs of func

Returns

path to results

cars.orchestrator.cluster.log_wrapper.time_profiling_function(*argv, **kwargs)

Create a wrapper to profile the function elapse time

Parameters
  • argv – args of func

  • kwargs – kwargs of func

Returns

path to results

cars.orchestrator.cluster.log_wrapper.memory_profiling_function(*argv, **kwargs)

Create a wrapper to profile the function occupation memory

Parameters
  • argv – args of func

  • kwargs – kwargs of func

Returns

path to results

cars.orchestrator.cluster.log_wrapper.switch_messages(func, total_time, max_memory=None)

create profile message with specific message depends on elapsed time (LONG, FAST…).

:param func : profiled function :param total_time : elapsed time of the function

cars.orchestrator.cluster.log_wrapper.log_message(func, message)

log profiling message

:param func : logged function :param message : log message

cars.orchestrator.cluster.log_wrapper.loop_function(argv, kwargs, func, nb_iteration=5)

generate a loop on each cluster function to eval possible leak

:param argv : input argv :param kwargs : input kwargs :param func : function to evaluation :param nb_iteration (int, optional): number of the iteration loop. :param Defaults to 5.

Returns:

_type_: result of the function

cars.orchestrator.cluster.log_wrapper.get_current_memory()

Get current memory of process

Returns

memory

Return type

float

cars.orchestrator.cluster.log_wrapper.log_delta_memory(func, memory_start, memory_end)

Log memory infos

Parameters
  • func – profiled function

  • memory_start (float) – memory before the run of function

  • memory_end (float) – memory after the run of function

cars.orchestrator.cluster.log_wrapper.generate_summary(out_dir, used_conf, clean_worker_logs=False)

Generate Profiling summary

cars.orchestrator.cluster.log_wrapper.filter_lists(names, data, cond)

Filter lists with condition on name

cars.orchestrator.cluster.log_wrapper.generate_boxplot(axis, names, data_full, title, data_type)

Generate boxplot

cars.orchestrator.cluster.log_wrapper.generate_histo(axis, names, data, title, data_type, data_min_err=None, data_max_err=None)

Generate histogram

cars.orchestrator.cluster.log_wrapper.generate_pie_chart(axis, names, data, title)

Generate pie chart, data in %

cars.orchestrator.cluster.log_wrapper.cars_profile(name=None, interval=0.1)

CARS profiling decorator

Param

func: function to monitor

class cars.orchestrator.cluster.log_wrapper.CarsMemProf(pid, pipe, interval=0.1)

Bases: threading.Thread

CarsMemProf

Profiling thread

run()

Run