cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler

Contains multiprocessing_profiler class

Module Contents

Classes

Timer

Start time

MultiprocessingProfiler

MultiprocessingProfiler

Functions

clean_thread(thread)

Clean thread

get_process_memory(process)

Get process current memory

get_cpu_usage(process)

Get cpu usage

save_figure_in_thread(to_fill_dataframe, file_path)

Save data during compute

check_pool_memory_usage(main_process_id, pool, ...)

Check memory usage of each worker in pool

save_data(dataframe, file_path)

Save dataframe to disk

Attributes

TERMINATE

RUN

RAM_PER_WORKER_CHECK_SLEEP_TIME

INTERVAL_CPU

SAVE_TIME

TIME

MAIN_MEMORY

MAX_PROCESS_MEMORY

MAIN_AND_PROCESS

TOTAL_PROCESS_MEMORY

AVAILABLE_RAM

TOTAL_RAM

MAIN_CPU_USAGE

TOTAL_PROCESS_CPU_USAGE

MAIN_AND_PROCESS_CPU

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.TERMINATE = 1
cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.RUN = 0
cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.RAM_PER_WORKER_CHECK_SLEEP_TIME = 2
cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.INTERVAL_CPU = 0.2
cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.SAVE_TIME = 120
cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.TIME = 'time'
cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.MAIN_MEMORY = 'main_memory'
cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.MAX_PROCESS_MEMORY = 'max_process_memory'
cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.MAIN_AND_PROCESS = 'main_and_processes'
cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.TOTAL_PROCESS_MEMORY = 'total_process_memory'
cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.AVAILABLE_RAM = 'available_ram'
cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.TOTAL_RAM = 'total_ram'
cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.MAIN_CPU_USAGE = 'main_cpu_usage'
cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.TOTAL_PROCESS_CPU_USAGE = 'total Proces_cpu_usage'
cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.MAIN_AND_PROCESS_CPU = 'main_and_process_cpu'
class cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.Timer[source]

Start time

class cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.MultiprocessingProfiler(pool, out_dir, max_ram_per_worker, mp_dataframe=None, timer=None)[source]

MultiprocessingProfiler

Used to profile memory in processes

cleanup()[source]

Cleanup

save_plot()[source]

Save plots

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.clean_thread(thread)[source]

Clean thread

Parameters

thread – thread to clean

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.get_process_memory(process)[source]

Get process current memory

:param process

Returns

memory Mb

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.get_cpu_usage(process)[source]

Get cpu usage

Parameters

process – Process to monitor

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.save_figure_in_thread(to_fill_dataframe, file_path)[source]

Save data during compute

Parameters

  • to_fill_dataframe – dataframe to fill

  • file_path – path to save path

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.check_pool_memory_usage(main_process_id, pool, max_ram_per_worker, to_fill_dataframe, timer)[source]

Check memory usage of each worker in pool

Parameters

  • main_process_id – main process id

  • pool – pool of worker

  • max_ram_per_worker – max ram to use per worker

  • to_fill_dataframe – dataframe to fill

  • timer – timer

cars.orchestrator.cluster.mp_cluster.multiprocessing_profiler.save_data(dataframe, file_path)[source]

Save dataframe to disk

Parameters

  • dataframe – file

  • file_path –