cars.orchestrator.cluster.pbs_dask_cluster

Contains abstract function for PBS dask Cluster

Module Contents

Classes

PbsDaskCluster

PbsDaskCluster

Functions

start_cluster(nb_workers, walltime, out_dir[, timeout])

This function create a dask cluster.

get_dashboard_link(cluster)

This function returns the dashboard address.

stop_cluster(cluster, client)

This function stops a dask cluster.
class cars.orchestrator.cluster.pbs_dask_cluster.PbsDaskCluster(conf_cluster, out_dir, launch_worker=True)

Bases: cars.orchestrator.cluster.abstract_dask_cluster.AbstractDaskCluster

PbsDaskCluster

start_dask_cluster()

Start dask cluster

cleanup()

Cleanup cluster

cars.orchestrator.cluster.pbs_dask_cluster.start_cluster(nb_workers, walltime, out_dir, timeout=600)

This function create a dask cluster. Each worker has nb_cpus cpus. Only one python process is started on each worker.

Threads number: start_cluster will use OMP_NUM_THREADS environment variable to determine how many threads might be used by a worker when running C/C++ code. (default to 1)

Workers number: start_cluster will use CARS_NB_WORKERS_PER_PBS_JOB environment variable to determine how many workers should be started by a single PBS job. (default to 1)

Queue worker: start_cluster will use CARS_PBS_QUEUE to determine in which queue worker jobs should be posted.

Parameters

  • nb_workers (int) – Number of dask workers

  • walltime (string) – Walltime for each dask worker

  • out_dir (string) – Output directory

Returns

Dask cluster and dask client

Return type

(dask_jobqueue.PBSCluster, dask.distributed.Client) tuple

This function returns the dashboard address.

Parameters

cluster (dask_jobqueue.PBSCluster) – Dask cluster

Returns

Link to the dashboard

Return type

string

cars.orchestrator.cluster.pbs_dask_cluster.stop_cluster(cluster, client)

This function stops a dask cluster.

Parameters

  • cluster (dask_jobqueue.PBSCluster) – Dask cluster

  • client (dask.distributed.Client) – Dask client