cars.orchestrator.cluster.slurm_dask_cluster

Contains abstract function for SLURM dask Cluster

Classes

SlurmDaskCluster

SlurmDaskCluster

Functions

start_cluster(nb_workers, walltime, out_dir[, ...])

Create a Dask cluster.

Module Contents

class cars.orchestrator.cluster.slurm_dask_cluster.SlurmDaskCluster(conf_cluster, out_dir, log_dir, launch_worker=True, data_to_propagate=None)[source]

Bases: cars.orchestrator.cluster.abstract_dask_cluster.AbstractDaskCluster

SlurmDaskCluster

check_conf(conf)[source]

Check configuration

Parameters:

conf (dict) – configuration to check

Returns:

overloaded configuration

Return type:

dict

start_dask_cluster()[source]

Start dask cluster

cleanup()[source]

Cleanup cluster

cars.orchestrator.cluster.slurm_dask_cluster.start_cluster(nb_workers, walltime, out_dir, timeout=600, activate_dashboard=False, python=None, account=None, qos=None)[source]

Create a Dask cluster.

Each worker will be spawned in an independent job with a single CPU allocated to it, and will use a single process. This is done to maximize CPU utilization and minimize scheduling delay.

The CARS_SLURM_QUEUE environment variable, if defined, is used to specify the queue in which worker jobs are scheduled.

Parameters:

  • nb_workers (int) – Number of dask workers

  • walltime (string) – Walltime for each dask worker

  • out_dir (string) – Output directory

  • timeout (int) – timeout of the cluster client

  • activate_dashboard (bool) – option to activate the dashborad server mode

  • python (string) – specfic python path

  • account (string) – SLURM account

  • qos (string) – Quality of Service parameter for TREX cluster

Returns:

Dask cluster and dask client

Return type:

(dask_jobqueue.SLURMCluster, dask.distributed.Client) tuple