cars.orchestrator.cluster.slurm_dask_cluster
Contains abstract function for SLURM dask Cluster
Module Contents
Classes
SlurmDaskCluster |
Functions
|
Create a Dask cluster. |
- class cars.orchestrator.cluster.slurm_dask_cluster.SlurmDaskCluster(conf_cluster, out_dir, launch_worker=True)
Bases:
cars.orchestrator.cluster.abstract_dask_cluster.AbstractDaskClusterSlurmDaskCluster
- check_conf(conf)
Check configuration
- Parameters
conf (dict) – configuration to check
- Returns
overloaded configuration
- Return type
dict
- start_dask_cluster()
Start dask cluster
- cleanup()
Cleanup cluster
- cars.orchestrator.cluster.slurm_dask_cluster.start_cluster(nb_workers, walltime, out_dir, timeout=600, activate_dashboard=False, python=None, account=None, qos=None)
Create a Dask cluster.
Each worker will be spawned in an independent job with a single CPU allocated to it, and will use a single process. This is done to maximize CPU utilization and minimize scheduling delay.
The CARS_SLURM_QUEUE environment variable, if defined, is used to specify the queue in which worker jobs are scheduled.
- Parameters
nb_workers (int) – Number of dask workers
walltime (string) – Walltime for each dask worker
out_dir (string) – Output directory
timeout (int) – timeout of the cluster client
activate_dashboard (bool) – option to activate the dashborad server mode
python (string) – specfic python path
account (string) – SLURM account
qos (string) – Quality of Service parameter for TREX cluster
- Returns
Dask cluster and dask client
- Return type
(dask_jobqueue.SLURMCluster, dask.distributed.Client) tuple