cars.data_structures.cars_dataset ================================= .. py:module:: cars.data_structures.cars_dataset .. autoapi-nested-parse:: cars_dataset module: Attributes ---------- .. autoapisummary:: cars.data_structures.cars_dataset.CARS_DS_TYPE_ARRAY cars.data_structures.cars_dataset.CARS_DS_TYPE_POINTS cars.data_structures.cars_dataset.CARS_DS_TYPE_DICT cars.data_structures.cars_dataset.TILES_INFO_FILE cars.data_structures.cars_dataset.OVERLAP_FILE cars.data_structures.cars_dataset.GRID_FILE cars.data_structures.cars_dataset.PROFILE_FILE cars.data_structures.cars_dataset.ATTRIBUTE_FILE cars.data_structures.cars_dataset.DATASET_FILE cars.data_structures.cars_dataset.DATAFRAME_FILE cars.data_structures.cars_dataset.CARSDICT_FILE cars.data_structures.cars_dataset.PROFILE cars.data_structures.cars_dataset.WINDOW cars.data_structures.cars_dataset.OVERLAPS cars.data_structures.cars_dataset.ATTRIBUTES cars.data_structures.cars_dataset.SAVING_INFO Classes ------- .. autoapisummary:: cars.data_structures.cars_dataset.CarsDataset Functions --------- .. autoapisummary:: cars.data_structures.cars_dataset.run_save_arrays cars.data_structures.cars_dataset.run_save_points cars.data_structures.cars_dataset.load_single_tile_array cars.data_structures.cars_dataset.load_single_tile_points cars.data_structures.cars_dataset.load_single_tile_dict cars.data_structures.cars_dataset.save_single_tile_array cars.data_structures.cars_dataset.save_single_tile_points cars.data_structures.cars_dataset.save_single_tile_dict cars.data_structures.cars_dataset.fill_dataset cars.data_structures.cars_dataset.fill_dataframe cars.data_structures.cars_dataset.fill_dict cars.data_structures.cars_dataset.save_all_dataframe cars.data_structures.cars_dataset.save_dataframe cars.data_structures.cars_dataset.save_dataset cars.data_structures.cars_dataset.create_tile_path cars.data_structures.cars_dataset.save_numpy_array cars.data_structures.cars_dataset.load_numpy_array cars.data_structures.cars_dataset.create_none cars.data_structures.cars_dataset.overlap_array_to_dict cars.data_structures.cars_dataset.window_array_to_dict cars.data_structures.cars_dataset.dict_profile_to_rio_profile cars.data_structures.cars_dataset.rio_profile_to_dict_profile cars.data_structures.cars_dataset.save_dict cars.data_structures.cars_dataset.make_yaml_safe cars.data_structures.cars_dataset.load_dict cars.data_structures.cars_dataset.separate_dicts cars.data_structures.cars_dataset.get_attributes cars.data_structures.cars_dataset.get_window_dataset cars.data_structures.cars_dataset.get_overlaps_dataset cars.data_structures.cars_dataset.get_profile_rasterio cars.data_structures.cars_dataset.get_profile_for_tag_dataset cars.data_structures.cars_dataset.generate_rasterio_window Module Contents --------------- .. py:data:: CARS_DS_TYPE_ARRAY :value: 'arrays' .. py:data:: CARS_DS_TYPE_POINTS :value: 'points' .. py:data:: CARS_DS_TYPE_DICT :value: 'dict' .. py:data:: TILES_INFO_FILE :value: 'tiles_info.yaml' .. py:data:: OVERLAP_FILE :value: 'overlaps.npy' .. py:data:: GRID_FILE :value: 'grid.npy' .. py:data:: PROFILE_FILE :value: 'profile.json' .. py:data:: ATTRIBUTE_FILE :value: 'attributes.yaml' .. py:data:: DATASET_FILE :value: 'dataset' .. py:data:: DATAFRAME_FILE :value: 'dataframe.csv' .. py:data:: CARSDICT_FILE :value: 'cars_dict' .. py:data:: PROFILE :value: 'profile' .. py:data:: WINDOW :value: 'window' .. py:data:: OVERLAPS :value: 'overlaps' .. py:data:: ATTRIBUTES :value: 'attributes' .. py:data:: SAVING_INFO :value: 'saving_info' .. py:class:: CarsDataset(dataset_type, load_from_disk=None, name='unknown') CarsDataset. Internal CARS structure for organazing tiles (xr.Datasets or pd.DataFrames). .. py:attribute:: dataset_type .. py:attribute:: tiles :value: None .. py:attribute:: tiles_info .. py:attribute:: _tiling_grid :value: None .. py:attribute:: overlaps :value: None .. py:attribute:: attributes .. py:attribute:: name :value: 'unknown' .. py:attribute:: final_function :value: None .. py:method:: __repr__() Repr function :return: printable self CarsDataset .. py:method:: __str__() Str function :return: printable self CarsDataset .. py:method:: custom_print() Return string of self :return: printable self .. py:property:: shape Return the shape of tiling grid (nb_row, nb_col) :return: shape of grid .. py:property:: tiling_grid Tiling grid, containing pixel windows of tiles :return: tiling grid, of shape [N, M, 4], containing [row_min, row_max, col_min, col_max] :rtype: np.ndarray .. py:method:: __getitem__(key) Get item : return the [row, col] dataset :param key: tuple index :return: tile :rtype: xr.Dataset or pd.DataFrame .. py:method:: __setitem__(key, newvalue) Set new tile :param key: tuple of row and col indexes :type key: tuple(int, int) :param newvalue: tile to set .. py:method:: load_single_tile(tile_path_name: str) Load a single tile :param tile_path_name: Path of tile to load :type tile_path_name: str :return: single tile :rtype: xarray Dataset or Panda dataframe to file .. py:method:: save_single_tile(tile, tile_path_name: str) Save xarray Dataset or Panda dataframe to file :param tile: tile to save :type tile: xr.Dataset or pd.DataFrame :param tile_path_name: Path of file to save in .. py:method:: run_save(future_result, file_name: str, **kwargs) Save future result when arrived :param future_result: xarray.Dataset received :param file_name: filename to save data to .. py:method:: get_window_as_dict(row, col, from_terrain=False, resolution=1) Get window in pixels for rasterio. Set from_terrain if tiling grid was defined in geographic coordinates. :param row: row :type row: int :param col: col :type col: int :param from_terrain: true if in terrain coordinates :type from_terrain: bool :param resolution: resolution :type resolution: float :return: New window : { "row_min" : row_min , "row_max" : row_max "col_min" : col_min "col_max" : col_max } :rtype: Dict .. py:method:: create_grid(nb_col: int, nb_row: int, row_split: int, col_split: int, row_overlap: int, col_overlap: int) Generate grid of positions by splitting [0, nb_row]x[0, nb_col] in splits of xsplit x ysplit size :param nb_col : number of columns :param nb_row : number of lines :param col_split: width of splits :param row_split: height of splits :param col_overlap: overlap to apply on rows :param row_overlap: overlap to apply on cols .. py:method:: generate_none_tiles() Generate the structure of data tiles, with Nones, according to grid shape. .. py:method:: create_empty_copy(cars_ds) Copy attributes, grid, overlaps, and create Nones. :param cars_ds: CarsDataset to copy :type cars_ds: CarsDataset .. py:method:: generate_descriptor(future_result, file_name, tag=None, dtype=None, nodata=None, rio_tags=None, rio_band_description=None, nbits=None) Generate de rasterio descriptor for the given future result Only works with pixelic tiling grid :param future_result: Future result :type future_result: xr.Dataset :param file_name: file name to save futures to :type file_name: str :param tag: tag to save :type tag: str :param dtype: dtype :type dtype: str :param nodata: no data value :type nodata: float .. py:method:: save_cars_dataset(directory) Save whole CarsDataset to given directory, including tiling grids, attributes, overlaps, and all the xr.Dataset or pd.DataFrames. :param directory: Path where to save self CarsDataset :type directory: str .. py:method:: load_cars_dataset_from_disk(directory) Load whole CarsDataset from given directory :param directory: Path where is saved CarsDataset to load :type directory: str .. py:function:: run_save_arrays(future_result, file_name, tag=None, descriptor=None, nbits=None) Save future when arrived :param future_result: xarray.Dataset received :type future_result: xarray.Dataset :param file_name: filename to save data to :type file_name: str :param tag: dataset tag to rasterize :type tag: str :param descriptor: rasterio descriptor .. py:function:: run_save_points(future_result, file_name, overwrite=False, save_by_pair=False, point_cloud_format='csv', overwrite_file_name=True, nbits=None) Save future result when arrived :param future_result: pandas Dataframe received :type future_result: pandas Dataframe :param file_name: filename to save data to :type file_name: str :param overwrite: overwrite file :type overwrite: bool :param point_cloud_format: output point cloud format :type point_cloud_format: str :param overwrite_file_name: generate a new filename from input filename matching input dataframe attributes :type overwrite_file_name: bool .. py:function:: load_single_tile_array(tile_path_name: str) -> xarray.Dataset Load a xarray tile :param tile_path_name: Path of tile to load :type tile_path_name: str :return: tile dataset :rtype: xr.Dataset .. py:function:: load_single_tile_points(tile_path_name: str) Load a panda dataframe :param tile_path_name: Path of tile to load :type tile_path_name: str :return: Tile dataframe :rtype: Panda dataframe .. py:function:: load_single_tile_dict(tile_path_name: str) Load a CarsDict :param tile_path_name: Path of tile to load :type tile_path_name: str :return: Tile dataframe :rtype: Panda dataframe .. py:function:: save_single_tile_array(dataset: xarray.Dataset, tile_path_name: str) Save xarray to directory, saving the data in a different file that the attributes (saved in a .json next to it). :param dataset: dataset to save :type dataset: xr.Dataset :param tile_path_name: Path of file to save in :type tile_path_name: str .. py:function:: save_single_tile_points(dataframe, tile_path_name: str) Save dataFrame to directory, saving the data in a different file that the attributes (saved in a .json next to it). :param dataframe: dataframe to save :type dataframe: pd.DataFrame :param tile_path_name: Path of file to save in :type tile_path_name: str .. py:function:: save_single_tile_dict(dict_cars, tile_path_name: str) Save cars_dict to directory, saving the data in a different file that the attributes (saved in a .json next to it). :param dict_cars: dataframe to save :type dict_cars: pd.DataFrame :param tile_path_name: Path of file to save in :type tile_path_name: str .. py:function:: fill_dataset(dataset, saving_info=None, window=None, profile=None, attributes=None, overlaps=None) From a full xarray dataset, fill info properly. User can fill with saving information (containing CarsDataset id), window of current tile and its overlaps, rasterio profile of full data, and attributes associated to data :param dataset: dataset to fill :type dataset: xarray_dataset :param saving_info: created by Orchestrator.get_saving_infos :type saving_info: dict :param window: :type window: dict :param profile: :type profile: dict :param attributes: :type attributes: dict .. py:function:: fill_dataframe(dataframe, saving_info=None, attributes=None) From a full pandas dataframe, fill info properly. User can fill with saving information (containing CarsDataset id), and attributes associated to data :param dataframe: dataframe to fill :type dataframe: pandas dataframe :param saving_info: created by Orchestrator.get_saving_infos :type saving_info: dict :param attributes: :type attributes: dict .. py:function:: fill_dict(data_dict, saving_info=None, attributes=None) From a fulldict, fill info properly. User can fill with saving information (containing CarsDataset id), and attributes associated to data :param data_dict: dictionnary to fill :type data_dict: Dict :param saving_info: created by Orchestrator.get_saving_infos :type saving_info: dict :param attributes: attributes associated to data :type attributes: dict .. py:function:: save_all_dataframe(dataframe, file_name, save_by_pair=False, overwrite=True, point_cloud_format='csv', overwrite_file_name=True) Save DataFrame to csv and laz format. The content of dataframe is merged to the content of existing saved Dataframe, if overwrite==False The option save_by_pair separate the dataframe by pair (one folder by pair) :param file_name: file name to save data to :type file_name: str :param overwrite: overwrite file if exists :type overwrite: bool :param point_cloud_format: point cloud format (csv or laz) :type point_cloud_format: str :param overwrite_file_name: generate a new filename from input filename matching input dataframe attributes, using only directory from input filename :type overwrite_file_name: bool .. py:function:: save_dataframe(dataframe, file_name, overwrite=True, point_cloud_format='csv') Save dataframe (csv, laz, attr file) .. py:function:: save_dataset(dataset, file_name, tag, use_windows_and_overlaps=False, descriptor=None, nbits=None) Reconstruct and save data. In order to save properly the dataset to corresponding tiff file, dataset must have been filled with saving info, profile, window, overlaps (if not 0), and rasterio descriptor if already created. See fill_dataset. :param dataset: dataset to save :type dataset: xr.Dataset :param file_name: file name to save data to :type file_name: str :param tag: tag to reconstruct :type tag: str :param use_windows_and_overlaps: use saved window and overlaps :type use_windows_and_overlaps: bool :param descriptor: descriptor to use with rasterio :type descriptor: rasterio dataset .. py:function:: create_tile_path(col: int, row: int, directory: str) -> str Create path of tile, according to its position in CarsDataset grid :param col: numero of column :type col: int :param row: numero of row :type row: int :param directory: path where to save tile :type directory: str :return: full path :rtype: str .. py:function:: save_numpy_array(array: numpy.ndarray, file_name: str) Save numpy array to file :param array: array to save :type array: np.ndarray :param file_name: numero of row :type file_name: str .. py:function:: load_numpy_array(file_name: str) -> numpy.ndarray Load numpy array from file :param file_name: numero of row :type file_name: str :return: array :rtype: np.ndarray .. py:function:: create_none(nb_row: int, nb_col: int) Create a grid filled with None. The created grid is a 2D list : ex: [[None, None], [None, None]] :param nb_row: number of rows :param nb_col: number of cols :return: Grid filled with None :rtype: list of list .. py:function:: overlap_array_to_dict(overlap) Convert matrix of overlaps, to dict format used in CarsDatasets. Input is : [o_up, o_down, o_left, o_right]. Output is : {"up": o_up, "down": o_down, "left": o_left, "right": o_right} :param overlap: overlaps :type overlap: List :return: New overlaps :rtype: Dict .. py:function:: window_array_to_dict(window, overlap=None) Convert matrix of windows, to dict format used in CarsDatasets. Use overlaps if you want to get window with overlaps inputs are : - window : [row_min, row_max, col_min, col_max], with pixel format - overlap (optional): [o_row_min, o_row_max, o_col_min, o_col_max] outputs are : { "row_min" : row_min - o_row_min, "row_max" : row_max + o_row_max, "col_min" : col_min - o_col_min, "col_max" : col_max - o_col_max, } :param window: window :type window: List :param overlap: overlaps :type overlap: List :return: New window :rtype: Dict .. py:function:: dict_profile_to_rio_profile(dict_profile: Dict) -> Dict Transform a rasterio Profile transformed into serializable Dict, into a rasterio profile. :param profile: rasterio Profile transformed into serializable Dict :type profile: Dict :return: Profile :rtype: Rasterio Profile .. py:function:: rio_profile_to_dict_profile(in_profile: Dict) -> Dict Transform a rasterio profile into a serializable Dict. :param in_profile: rasterio Profile transformed into serializable Dict :type in_profile: Dict :return: Profile :rtype: Dict .. py:function:: save_dict(dictionary, file_path: str) Save dict to json file :param dictionary: dictionary to save :type dictionary: Dict :param file_path: file path to use :type file_path: str .. py:function:: make_yaml_safe(obj) Converter .. py:function:: load_dict(file_path: str) -> Dict Load dict from json file :param file_path: file path to use :type file_path: str .. py:function:: separate_dicts(dictionary, list_tags) Separate a dict into two, the second one containing the given tags. For example, {key1: val1, key2: val2, key3: val3} with list_tags = [key2] will be split in : {key1: val1, key3: val3} and {key2: val2} .. py:function:: get_attributes(obj) Get attributes in .attrs of dataset or dataframe :param obj: dataset or dataframe :type obj: xr.Dataset or pandas.Dataframe .. py:function:: get_window_dataset(dataset) Get window in dataset :param dataset: dataset :type dataset: xr.Dataset .. py:function:: get_overlaps_dataset(dataset) Get overlaps in dataset :param dataset: dataset :type dataset: xr.Dataset .. py:function:: get_profile_rasterio(dataset) Get profile in dataset :param dataset: dataset :type dataset: xr.Dataset .. py:function:: get_profile_for_tag_dataset(dataset, tag: str) -> Dict Get profile according to layer to save. This function modify current rasterio dataset to fix the number of bands of the data associated to given tag. :param tag: tag to use :type tag: str :return: Profile :rtype: Rasterio Profile .. py:function:: generate_rasterio_window(window: Dict) -> rasterio.windows.Window Generate rasterio window to use. :param window: window to convert, containing 'row_min', 'row_max', 'col_min', 'col_max :type window: dict :return: rasterio window :rtype: rio.windows.Window