cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler

Contains multiprocessing_profiler class

Attributes

`TERMINATE`
`RUN`
`RAM_PER_WORKER_CHECK_SLEEP_TIME`
`INTERVAL_CPU`
`SAVE_TIME`
`TIME`
`MAIN_MEMORY`
`MAX_PROCESS_MEMORY`
`MAIN_AND_PROCESS`
`TOTAL_PROCESS_MEMORY`
`AVAILABLE_RAM`
`TOTAL_RAM`
`MAIN_CPU_USAGE`
`TOTAL_PROCESS_CPU_USAGE`
`MAIN_AND_PROCESS_CPU`

Classes

`Timer`	Start time
`MultiprocessingProfiler`	MultiprocessingProfiler

Functions

`clean_thread`(thread)	Clean thread
`get_process_memory`(process)	Get process current memory
`get_cpu_usage`(process)	Get cpu usage
`save_figure_in_thread`(to_fill_dataframe, file_path)	Save data during compute
`check_pool_memory_usage`(main_process_id, pool, ...)	Check memory usage of each worker in pool
`save_data`(dataframe, file_path)	Save dataframe to disk

Module Contents

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.TERMINATE = 1

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.RUN = 0

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.RAM_PER_WORKER_CHECK_SLEEP_TIME = 2

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.INTERVAL_CPU = 0.2

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.SAVE_TIME = 120

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.TIME = 'time'

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.MAIN_MEMORY = 'main_memory'

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.MAX_PROCESS_MEMORY = 'max_process_memory'

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.MAIN_AND_PROCESS = 'main_and_processes'

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.TOTAL_PROCESS_MEMORY = 'total_process_memory'

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.AVAILABLE_RAM = 'available_ram'

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.TOTAL_RAM = 'total_ram'

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.MAIN_CPU_USAGE = 'main_cpu_usage'

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.TOTAL_PROCESS_CPU_USAGE = 'total Proces_cpu_usage'

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.MAIN_AND_PROCESS_CPU = 'main_and_process_cpu'

class cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.Timer[source]

Start time

timer

class cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.MultiprocessingProfiler(pool, out_dir, max_ram_per_worker, mp_dataframe=None, timer=None)[source]

MultiprocessingProfiler

Used to profile memory in processes

main_pid

pool

out_dir

file_plot

max_ram_per_worker

monitor_thread

saver_thread

cleanup()[source]: Cleanup

save_plot()[source]: Save plots

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.clean_thread(thread)[source]

Clean thread

Parameters:: thread – thread to clean

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.get_process_memory(process)[source]

Get process current memory

:param process

Returns:: memory Mb

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.get_cpu_usage(process)[source]

Get cpu usage

Parameters:: process – Process to monitor

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.save_figure_in_thread(to_fill_dataframe, file_path)[source]

Save data during compute

Parameters:

to_fill_dataframe – dataframe to fill
file_path – path to save path

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.check_pool_memory_usage(main_process_id, pool, max_ram_per_worker, to_fill_dataframe, timer)[source]

Check memory usage of each worker in pool

Parameters:

main_process_id – main process id
pool – pool of worker
max_ram_per_worker – max ram to use per worker
to_fill_dataframe – dataframe to fill
timer – timer

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.save_data(dataframe, file_path)[source]

Save dataframe to disk

Parameters:

dataframe – file
file_path