cars.orchestrator.orchestrator
this module contains the orchestrator class
Module Contents
Classes
Orchestrator |
Functions
|
Merge dict2 into dict 1 |
|
Flatten list of CarsDatasets to list of delayed |
|
Update saving infos dict with row and col arguments |
Attributes
- cars.orchestrator.orchestrator.SYS_PLATFORM
- cars.orchestrator.orchestrator.IS_WIN
- class cars.orchestrator.orchestrator.Orchestrator(orchestrator_conf=None, out_dir=None, launch_worker=True, out_json_path=None)
Orchestrator
- add_to_clean(tmp_dir)
- get_conf()
Get orchestrator conf
- Returns
orchestrator conf
- add_to_save_lists(file_name, tag, cars_ds, dtype='float32', nodata=0, cars_ds_name=None, optional_data=False, save_by_pair=False)
Save file to list in order to be saved later
- Parameters
file_name – file name
tag – tag
cars_ds – cars dataset to register
cars_ds_name – name corresponding to CarsDataset, for information during logging
optional_data (bool) – True if the data is optionnal
save_by_pair (bool) – True if data by pair
- add_to_replace_lists(cars_ds, cars_ds_name=None)
Add CarsDataset to replacing Registry
- Parameters
cars_ds (CarsDataset) – CarsDataset to replace
cars_ds_name – name corresponding to CarsDataset, for information during logging
- add_to_compute_lists(cars_ds, cars_ds_name=None)
Add CarsDataset to compute Registry: computed, but not used in main process
- Parameters
cars_ds (CarsDataset) – CarsDataset to comput
cars_ds_name – name corresponding to CarsDataset, for information during logging
- save_out_json()
Check out_json and save it to file
- save_index()
Save all product index files
- update_out_info(new_dict)
Update self.out_json with new dict
- Parameters
new_dict (dict) – dict to merge
- update_index(new_dict)
Update self.product_index with new dict
- Parameters
new_dict (dict) – dict to merge
- get_saving_infos(cars_ds_list)
Get saving infos of given cars datasets
- Parameters
cars_ds_list (list[CarsDataset]) – list of cars datasets
:return : list of saving infos :rtype: list[dict]
- get_data(tag, future_object)
Get data already on disk corresponding to window of object
- Parameters
tag (str) – tag
future_object (xarray Dataset) – object
- Returns
data on disk corresponding to tag
- Return type
np.ndarray
- compute_futures(only_remaining_delayed=None)
Compute all futures from regitries
- Parameters
only_remaining_delayed – list of delayed if second run
- reset_cluster()
Reset Cluster
- reset_registries()
Reset registries
- breakpoint()
Breakpoint : compute all delayed, save and replace data in CarsDatasets
- __enter__()
Function run on enter
- __exit__(exc_type, exc_value, traceback_msg)
Function run on exit.
Compute cluster tasks, save futures to be saved, and cleanup cluster and files
- cars.orchestrator.orchestrator.merge_dicts(dict1, dict2)
Merge dict2 into dict 1
- Parameters
dict1 (dict) – dict 1
dict2 (dict) – dict 2
- cars.orchestrator.orchestrator.flatten_object(cars_ds_list, delayed_type)
Flatten list of CarsDatasets to list of delayed
- Parameters
cars_ds_list (list[CarsDataset]) – list of cars datasets flatten
delayed_type – type of delayed
- Returns
list of delayed
- Return type
list[Delayed]
- cars.orchestrator.orchestrator.update_saving_infos(saving_info_left, row=None, col=None)
Update saving infos dict with row and col arguments
- Parameters
saving_info_left (dict) – saving infos
row (int) – row
col (int) – col
- Returns
updated saving infos dict
- Return type
dict