cars.orchestrator.cluster.abstract_dask_cluster

Contains abstract function for abstract dask Cluster

Module Contents

Classes

AbstractDaskCluster

AbstractDaskCluster

DaskFutureIterator

iterator on dask futures, similar to as_completed

ComputeDSMMemoryLogger

A subclass of WorkerPlugin dedicated to monitoring workers memory

Functions

set_config()

Set particular DASK config such as:

save_config(output_dir, file_name)

Save DASK global config

write_yaml_config(yaml_config, output_dir, file_name)

Writes a YAML config to disk.

register_xarray()

Add hook to dask so it correctly estimates memory used by xarray
class cars.orchestrator.cluster.abstract_dask_cluster.AbstractDaskCluster(conf_cluster, out_dir, launch_worker=True)

Bases: cars.orchestrator.cluster.abstract_cluster.AbstractCluster

AbstractDaskCluster

abstract check_conf(conf)

Check configuration

Parameters

conf (dict) – configuration to check

Returns

overloaded configuration

Return type

dict

abstract start_dask_cluster()

Start dask cluster

create_task_wrapped(func, nout=1)

Create task

Parameters

  • func – function

  • nout – number of outputs

get_delayed_type()

Get delayed type

start_tasks(task_list)

Start all tasks

Parameters

task_list – task list

scatter(data, broadcast=True)

Distribute data through workers

Parameters

data – task data

future_iterator(future_list, timeout=None)

Start all tasks

Parameters

future_list – future_list list

class cars.orchestrator.cluster.abstract_dask_cluster.DaskFutureIterator(future_list, timeout=None)

iterator on dask futures, similar to as_completed Only returns the actual results, delete the future after usage

__iter__()
__next__()
cars.orchestrator.cluster.abstract_dask_cluster.set_config()

Set particular DASK config such as: - scheduler

cars.orchestrator.cluster.abstract_dask_cluster.save_config(output_dir: str, file_name: str)

Save DASK global config

Parameters

  • output_dir – output directory path

  • file_name – output file name

cars.orchestrator.cluster.abstract_dask_cluster.write_yaml_config(yaml_config: dict, output_dir: str, file_name: str)

Writes a YAML config to disk. TODO: put in global file if needed elsewhere than DASK conf save.

Parameters

  • yaml_config – YAML config to write

  • output_dir – output directory path

  • file_name – output file name

cars.orchestrator.cluster.abstract_dask_cluster.register_xarray()

Add hook to dask so it correctly estimates memory used by xarray

class cars.orchestrator.cluster.abstract_dask_cluster.ComputeDSMMemoryLogger(outdir)

Bases: distributed.diagnostics.plugin.WorkerPlugin

A subclass of WorkerPlugin dedicated to monitoring workers memory

This plugin enables two things:

  • Additional dask log traces (for each worker internal state change):

    • amount of tasks

    • associated memory

  • A numpy data file with memory metrics and timing

setup(worker)

Associate plugin with a worker :param worker: The worker to associate the plugin with

transition(key, start, finish, **kwargs)

Callback when worker changes internal state