cars.orchestrator.cluster.pbs_dask_cluster

Contains abstract function for PBS dask Cluster

Module Contents

Classes

PbsDaskCluster

PbsDaskCluster

Functions

start_cluster(nb_workers, walltime, out_dir[, ...])

Create a Dask cluster.

class cars.orchestrator.cluster.pbs_dask_cluster.PbsDaskCluster(conf_cluster, out_dir, launch_worker=True)

Bases: cars.orchestrator.cluster.abstract_dask_cluster.AbstractDaskCluster

PbsDaskCluster

check_conf(conf)

Check configuration

Parameters

conf (dict) – configuration to check

Returns

overloaded configuration

Return type

dict

start_dask_cluster()

Start dask cluster

cleanup()

Cleanup cluster

cars.orchestrator.cluster.pbs_dask_cluster.start_cluster(nb_workers, walltime, out_dir, timeout=600, activate_dashboard=False, python=None)

Create a Dask cluster.

Each worker will be spawned in an independent job with a single CPU allocated to it, and will use a single process. This is done to maximize CPU utilization and minimize scheduling delay.

The CARS_PBS_QUEUE environment variable, if defined, is used to specify the queue in which worker jobs are scheduled.

Parameters

• nb_workers (int) – Number of dask workers

• walltime (string) – Walltime for each dask worker

• out_dir (string) – Output directory

Returns

Dask cluster and dask client

Return type

(dask_jobqueue.PBSCluster, dask.distributed.Client) tuple