cars.orchestrator.orchestrator

this module contains the orchestrator class

Module Contents

Classes

Orchestrator

Orchestrator

Functions

flatten_object(cars_ds_list, delayed_type)

Flatten list of CarsDatasets to list of delayed

update_saving_infos(saving_info_left[, row, col])

Update saving infos dict with row and col arguments
class cars.orchestrator.orchestrator.Orchestrator(orchestrator_conf=None, out_dir=None, launch_worker=True, out_json_path=None)

Orchestrator

add_to_clean(tmp_dir)
get_conf()

Get orchestrator conf

Returns

orchestrator conf

add_to_save_lists(file_name, tag, cars_ds, dtype='float32', nodata=0, cars_ds_name=None, optional_data=False, save_points_cloud_by_pair=False)

Save file to list in order to be saved later

Parameters

  • file_name – file name

  • tag – tag

  • cars_ds – cars dataset to register

  • cars_ds_name – name corresponding to CarsDataset, for information during logging

  • optional_data (bool) – True if the data is optionnal

  • save_points_cloud_by_pair (bool) – True if data by pair

add_to_replace_lists(cars_ds, cars_ds_name=None)

Add CarsDataset to replacing Registry

Parameters

  • cars_ds (CarsDataset) – CarsDataset to replace

  • cars_ds_name – name corresponding to CarsDataset, for information during logging

add_to_compute_lists(cars_ds, cars_ds_name=None)

Add CarsDataset to compute Registry: computed, but not used in main process

Parameters

  • cars_ds (CarsDataset) – CarsDataset to comput

  • cars_ds_name – name corresponding to CarsDataset, for information during logging

save_out_json()

Check out_json and save it to file

update_out_info(new_dict)

Update self.out_json with new dict

Parameters

new_dict (dict) – dict to merge

get_saving_infos(cars_ds_list)

Get saving infos of given cars datasets

Parameters

cars_ds_list (list[CarsDataset]) – list of cars datasets

:return : list of saving infos :rtype: list[dict]

get_data(tag, future_object)

Get data already on disk corresponding to window of object

Parameters

  • tag (str) – tag

  • future_object (xarray Dataset) – object

Returns

data on disk corresponding to tag

Return type

np.ndarray

compute_futures(only_remaining_delayed=None)

Compute all futures from regitries

Parameters

only_remaining_delayed – list of delayed if second run

reset_cluster()

Reset Cluster

reset_registries()

Reset registries

breakpoint()

Breakpoint : compute all delayed, save and replace data in CarsDatasets

__enter__()

Function run on enter

__exit__(exc_type, exc_value, traceback_msg)

Function run on exit.

Compute cluster tasks, save futures to be saved, and cleanup cluster and files

cars.orchestrator.orchestrator.flatten_object(cars_ds_list, delayed_type)

Flatten list of CarsDatasets to list of delayed

Parameters

  • cars_ds_list (list[CarsDataset]) – list of cars datasets flatten

  • delayed_type – type of delayed

Returns

list of delayed

Return type

list[Delayed]

cars.orchestrator.orchestrator.update_saving_infos(saving_info_left, row=None, col=None)

Update saving infos dict with row and col arguments

Parameters

  • saving_info_left (dict) – saving infos

  • row (int) – row

  • col (int) – col

Returns

updated saving infos dict

Return type

dict