cars.orchestrator.orchestrator

this module contains the orchestrator class

Attributes

SYS_PLATFORM

IS_WIN

RAM_THRESHOLD_MB

RAM_CHECK_SLEEP_TIME

Classes

Orchestrator

Orchestrator

Functions

merge_dicts(dict1, dict2)

Merge dict2 into dict 1

flatten_object(cars_ds_list, delayed_type)

Flatten list of CarsDatasets to list of delayed

update_saving_infos(saving_info_left[, row, col])

Update saving infos dict with row and col arguments

get_disparity_range_cropped(obj)

Get CROPPED_DISPARITY_RANGE value in attributes

check_ram_usage()

Check RAM usage

Module Contents

cars.orchestrator.orchestrator.SYS_PLATFORM
cars.orchestrator.orchestrator.IS_WIN
cars.orchestrator.orchestrator.RAM_THRESHOLD_MB = 500
cars.orchestrator.orchestrator.RAM_CHECK_SLEEP_TIME = 5
class cars.orchestrator.orchestrator.Orchestrator(orchestrator_conf=None, out_dir=None, log_dir=None, launch_worker=True, out_yaml_path=None)[source]

Orchestrator

tmp_dir_list = []
launch_worker = True
orchestrator_conf = None
cluster
conf
task_timeout
id_generator
cars_ds_savers_registry
cars_ds_replacer_registry
cars_ds_compute_registry
achievement_tracker
dir_tile_profiling
tile_profiler
cars_ds_names_info = []
target_task_id = None
out_yaml_path = None
out_yaml
product_index
add_to_clean(tmp_dir)[source]
get_conf()[source]

Get orchestrator conf

Returns:

orchestrator conf

set_target_task(task_id)[source]

Set the target task ID for progress tracking. Called by pipelines to associate orchestrator tile processing with a ProgressTree task.

Parameters:

task_id (int) – Task ID from ProgressTree

add_to_save_lists(file_name, tag, cars_ds, dtype='float32', nodata=0, cars_ds_name=None, optional_data=False, save_by_pair=False, rio_tags=None, rio_band_description=None, nbits=None)

Save file to list in order to be saved later

Parameters:

  • file_name – file name

  • tag – tag

  • cars_ds – cars dataset to register

  • cars_ds_name – name corresponding to CarsDataset, for information during logging

  • optional_data (bool) – True if the data is optionnal

  • save_by_pair (bool) – True if data by pair

add_to_replace_lists(cars_ds, cars_ds_name=None)[source]

Add CarsDataset to replacing Registry

Parameters:

  • cars_ds (CarsDataset) – CarsDataset to replace

  • cars_ds_name – name corresponding to CarsDataset, for information during logging

add_to_compute_lists(cars_ds, cars_ds_name=None)[source]

Add CarsDataset to compute Registry: computed, but not used in main process

Parameters:

  • cars_ds (CarsDataset) – CarsDataset to comput

  • cars_ds_name – name corresponding to CarsDataset, for information during logging

save_out_yaml()[source]

Check out_file and save it to file

save_index()[source]

Save all product index files

update_out_info(new_dict)[source]

Update self.out_file with new dict

Parameters:

new_dict (dict) – dict to merge

update_index(new_dict)[source]

Update self.product_index with new dict

Parameters:

new_dict (dict) – dict to merge

get_saving_infos(cars_ds_list)[source]

Get saving infos of given cars datasets

Parameters:

cars_ds_list (list[CarsDataset]) – list of cars datasets

:return : list of saving infos :rtype: list[dict]

get_data(tag, future_object)[source]

Get data already on disk corresponding to window of object

Parameters:

  • tag (str) – tag

  • future_object (xarray Dataset) – object

Returns:

data on disk corresponding to tag

Return type:

np.ndarray

compute_futures(only_remaining_delayed=None)[source]

Compute all futures from regitries

Parameters:

only_remaining_delayed – list of delayed if second run

reset_cluster()[source]

Reset Cluster

reset_registries()[source]

Reset registries

breakpoint()

Breakpoint : compute all delayed, save and replace data in CarsDatasets

cleanup()[source]

Cleanup orchestrator

__enter__()[source]

Function run on enter

__exit__(exc_type, exc_value, traceback_msg)[source]

Function run on exit.

Compute cluster tasks, save futures to be saved, and cleanup cluster and files

cars.orchestrator.orchestrator.merge_dicts(dict1, dict2)[source]

Merge dict2 into dict 1

Parameters:

  • dict1 (dict) – dict 1

  • dict2 (dict) – dict 2

cars.orchestrator.orchestrator.flatten_object(cars_ds_list, delayed_type)[source]

Flatten list of CarsDatasets to list of delayed

Parameters:

  • cars_ds_list (list[CarsDataset]) – list of cars datasets flatten

  • delayed_type – type of delayed

Returns:

list of delayed

Return type:

list[Delayed]

cars.orchestrator.orchestrator.update_saving_infos(saving_info_left, row=None, col=None)[source]

Update saving infos dict with row and col arguments

Parameters:

  • saving_info_left (dict) – saving infos

  • row (int) – row

  • col (int) – col

Returns:

updated saving infos dict

Return type:

dict

cars.orchestrator.orchestrator.get_disparity_range_cropped(obj)[source]

Get CROPPED_DISPARITY_RANGE value in attributes

Parameters:

obj – object to look in

:rtype bool

cars.orchestrator.orchestrator.check_ram_usage()[source]

Check RAM usage