cars.data_structures.cars_dataset

cars_dataset module:

Attributes

`CARS_DS_TYPE_ARRAY`
`CARS_DS_TYPE_POINTS`
`CARS_DS_TYPE_DICT`
`TILES_INFO_FILE`
`OVERLAP_FILE`
`GRID_FILE`
`PROFILE_FILE`
`ATTRIBUTE_FILE`
`DATASET_FILE`
`DATAFRAME_FILE`
`CARSDICT_FILE`
`PROFILE`
`WINDOW`
`OVERLAPS`
`ATTRIBUTES`
`SAVING_INFO`

Classes

CarsDataset

CarsDataset.

Functions

`run_save_arrays`(future_result, file_name[, tag, ...])	Save future when arrived
`run_save_points`(future_result, file_name[, overwrite, ...])	Save future result when arrived
`load_single_tile_array`(→ xarray.Dataset)	Load a xarray tile
`load_single_tile_points`(tile_path_name)	Load a panda dataframe
`load_single_tile_dict`(tile_path_name)	Load a CarsDict
`save_single_tile_array`(dataset, tile_path_name)	Save xarray to directory, saving the data in a different file that
`save_single_tile_points`(dataframe, tile_path_name)	Save dataFrame to directory, saving the data in a different file that
`save_single_tile_dict`(dict_cars, tile_path_name)	Save cars_dict to directory, saving the data in a different file that
`fill_dataset`(dataset[, saving_info, window, profile, ...])	From a full xarray dataset, fill info properly.
`fill_dataframe`(dataframe[, saving_info, attributes])	From a full pandas dataframe, fill info properly.
`fill_dict`(data_dict[, saving_info, attributes])	From a fulldict, fill info properly.
`save_all_dataframe`(dataframe, file_name[, ...])	Save DataFrame to csv and laz format. The content of dataframe is merged to
`save_dataframe`(dataframe, file_name[, overwrite, ...])	Save dataframe (csv, laz, attr file)
`save_dataset`(dataset, file_name, tag[, ...])	Reconstruct and save data.
`create_tile_path`(→ str)	Create path of tile, according to its position in CarsDataset grid
`save_numpy_array`(array, file_name)	Save numpy array to file
`load_numpy_array`(→ numpy.ndarray)	Load numpy array from file
`create_none`(nb_row, nb_col)	Create a grid filled with None. The created grid is a 2D list :
`overlap_array_to_dict`(overlap)	Convert matrix of overlaps, to dict format used in CarsDatasets.
`window_array_to_dict`(window[, overlap])	Convert matrix of windows, to dict format used in CarsDatasets.
`dict_profile_to_rio_profile`(→ Dict)	Transform a rasterio Profile transformed into serializable Dict,
`rio_profile_to_dict_profile`(→ Dict)	Transform a rasterio profile into a serializable Dict.
`save_dict`(dictionary, file_path)	Save dict to json file
`make_yaml_safe`(obj)	Converter
`load_dict`(→ Dict)	Load dict from json file
`separate_dicts`(dictionary, list_tags)	Separate a dict into two, the second one containing the given tags.
`get_attributes`(obj)	Get attributes in .attrs of dataset or dataframe
`get_window_dataset`(dataset)	Get window in dataset
`get_overlaps_dataset`(dataset)	Get overlaps in dataset
`get_profile_rasterio`(dataset)	Get profile in dataset
`get_profile_for_tag_dataset`(→ Dict)	Get profile according to layer to save.
`generate_rasterio_window`(→ rasterio.windows.Window)	Generate rasterio window to use.

Module Contents

cars.data_structures.cars_dataset.CARS_DS_TYPE_ARRAY = 'arrays'

cars.data_structures.cars_dataset.CARS_DS_TYPE_POINTS = 'points'

cars.data_structures.cars_dataset.CARS_DS_TYPE_DICT = 'dict'

cars.data_structures.cars_dataset.TILES_INFO_FILE = 'tiles_info.yaml'

cars.data_structures.cars_dataset.OVERLAP_FILE = 'overlaps.npy'

cars.data_structures.cars_dataset.GRID_FILE = 'grid.npy'

cars.data_structures.cars_dataset.PROFILE_FILE = 'profile.json'

cars.data_structures.cars_dataset.ATTRIBUTE_FILE = 'attributes.yaml'

cars.data_structures.cars_dataset.DATASET_FILE = 'dataset'

cars.data_structures.cars_dataset.DATAFRAME_FILE = 'dataframe.csv'

cars.data_structures.cars_dataset.CARSDICT_FILE = 'cars_dict'

cars.data_structures.cars_dataset.PROFILE = 'profile'

cars.data_structures.cars_dataset.WINDOW = 'window'

cars.data_structures.cars_dataset.OVERLAPS = 'overlaps'

cars.data_structures.cars_dataset.ATTRIBUTES = 'attributes'

cars.data_structures.cars_dataset.SAVING_INFO = 'saving_info'

class cars.data_structures.cars_dataset.CarsDataset(dataset_type, load_from_disk=None, name='unknown')[source]

CarsDataset.

Internal CARS structure for organazing tiles (xr.Datasets or pd.DataFrames).

dataset_type

tiles = None

tiles_info

_tiling_grid = None

overlaps = None

attributes

name = 'unknown'

final_function = None

__repr__()[source]: Repr function :return: printable self CarsDataset

__str__()[source]: Str function :return: printable self CarsDataset

custom_print()[source]: Return string of self :return: printable self

property shape: Return the shape of tiling grid (nb_row, nb_col) :return: shape of grid

property tiling_grid

Tiling grid, containing pixel windows of tiles

Returns:: tiling grid, of shape [N, M, 4], containing [row_min, row_max, col_min, col_max]
Return type:: np.ndarray

__getitem__(key)[source]

Get item : return the [row, col] dataset

Parameters:: key – tuple index
Returns:: tile
Return type:: xr.Dataset or pd.DataFrame

__setitem__(key, newvalue)[source]

Set new tile

Parameters:

key (tuple(int, int)) – tuple of row and col indexes
newvalue – tile to set

load_single_tile(tile_path_name: str)[source]

Load a single tile

Parameters:: tile_path_name (str) – Path of tile to load
Returns:: single tile
Return type:: xarray Dataset or Panda dataframe to file

save_single_tile(tile, tile_path_name: str)[source]

Save xarray Dataset or Panda dataframe to file

Parameters:

tile (xr.Dataset or pd.DataFrame) – tile to save
tile_path_name – Path of file to save in

run_save(future_result, file_name: str, **kwargs)[source]

Save future result when arrived

Parameters:

future_result – xarray.Dataset received
file_name – filename to save data to

get_window_as_dict(row, col, from_terrain=False, resolution=1)[source]

Get window in pixels for rasterio. Set from_terrain if tiling grid was defined in geographic coordinates.

Parameters:

row (int) – row
col (int) – col
from_terrain (bool) – true if in terrain coordinates
resolution (float) – resolution

Returns:

New window : { “row_min” : row_min , “row_max” : row_max “col_min” : col_min “col_max” : col_max }

Return type:

Dict

create_grid(nb_col: int, nb_row: int, row_split: int, col_split: int, row_overlap: int, col_overlap: int)[source]

Generate grid of positions by splitting [0, nb_row]x[0, nb_col] in splits of xsplit x ysplit size

:param nb_col : number of columns :param nb_row : number of lines :param col_split: width of splits :param row_split: height of splits :param col_overlap: overlap to apply on rows :param row_overlap: overlap to apply on cols

generate_none_tiles()[source]

Generate the structure of data tiles, with Nones, according: to grid shape.

create_empty_copy(cars_ds)[source]

Copy attributes, grid, overlaps, and create Nones.

Parameters:: cars_ds (CarsDataset) – CarsDataset to copy

generate_descriptor(future_result, file_name, tag=None, dtype=None, nodata=None, rio_tags=None, rio_band_description=None, nbits=None)[source]

Generate de rasterio descriptor for the given future result

Only works with pixelic tiling grid

Parameters:

future_result (xr.Dataset) – Future result
file_name (str) – file name to save futures to
tag (str) – tag to save
dtype (str) – dtype
nodata (float) – no data value

save_cars_dataset(directory)[source]

Save whole CarsDataset to given directory, including tiling grids, attributes, overlaps, and all the xr.Dataset or pd.DataFrames.

Parameters:: directory (str) – Path where to save self CarsDataset

load_cars_dataset_from_disk(directory)[source]

Load whole CarsDataset from given directory

Parameters:: directory (str) – Path where is saved CarsDataset to load

cars.data_structures.cars_dataset.run_save_arrays(future_result, file_name, tag=None, descriptor=None, nbits=None)[source]

Save future when arrived

Parameters:

future_result (xarray.Dataset) – xarray.Dataset received
file_name (str) – filename to save data to
tag (str) – dataset tag to rasterize
descriptor – rasterio descriptor

cars.data_structures.cars_dataset.run_save_points(future_result, file_name, overwrite=False, save_by_pair=False, point_cloud_format='csv', overwrite_file_name=True, nbits=None)[source]

Save future result when arrived

Parameters:

future_result (pandas Dataframe) – pandas Dataframe received
file_name (str) – filename to save data to
overwrite (bool) – overwrite file
point_cloud_format (str) – output point cloud format
overwrite_file_name (bool) – generate a new filename from input filename matching input dataframe attributes

cars.data_structures.cars_dataset.load_single_tile_array(tile_path_name: str) → xarray.Dataset[source]

Load a xarray tile

Parameters:: tile_path_name (str) – Path of tile to load
Returns:: tile dataset
Return type:: xr.Dataset

cars.data_structures.cars_dataset.load_single_tile_points(tile_path_name: str)[source]

Load a panda dataframe

Parameters:: tile_path_name (str) – Path of tile to load
Returns:: Tile dataframe
Return type:: Panda dataframe

cars.data_structures.cars_dataset.load_single_tile_dict(tile_path_name: str)[source]

Load a CarsDict

Parameters:: tile_path_name (str) – Path of tile to load
Returns:: Tile dataframe
Return type:: Panda dataframe

cars.data_structures.cars_dataset.save_single_tile_array(dataset: xarray.Dataset, tile_path_name: str)[source]

Save xarray to directory, saving the data in a different file that the attributes (saved in a .json next to it).

Parameters:

dataset (xr.Dataset) – dataset to save
tile_path_name (str) – Path of file to save in

cars.data_structures.cars_dataset.save_single_tile_points(dataframe, tile_path_name: str)[source]

Save dataFrame to directory, saving the data in a different file that the attributes (saved in a .json next to it).

Parameters:

dataframe (pd.DataFrame) – dataframe to save
tile_path_name (str) – Path of file to save in

cars.data_structures.cars_dataset.save_single_tile_dict(dict_cars, tile_path_name: str)[source]

Save cars_dict to directory, saving the data in a different file that the attributes (saved in a .json next to it).

Parameters:

dict_cars (pd.DataFrame) – dataframe to save
tile_path_name (str) – Path of file to save in

cars.data_structures.cars_dataset.fill_dataset(dataset, saving_info=None, window=None, profile=None, attributes=None, overlaps=None)[source]

From a full xarray dataset, fill info properly. User can fill with saving information (containing CarsDataset id), window of current tile and its overlaps, rasterio profile of full data, and attributes associated to data

Parameters:

dataset (xarray_dataset) – dataset to fill
saving_info (dict) – created by Orchestrator.get_saving_infos
window (dict)
profile (dict)
attributes (dict)

cars.data_structures.cars_dataset.fill_dataframe(dataframe, saving_info=None, attributes=None)[source]

From a full pandas dataframe, fill info properly. User can fill with saving information (containing CarsDataset id), and attributes associated to data

Parameters:

dataframe (pandas dataframe) – dataframe to fill
saving_info (dict) – created by Orchestrator.get_saving_infos
attributes (dict)

cars.data_structures.cars_dataset.fill_dict(data_dict, saving_info=None, attributes=None)[source]

From a fulldict, fill info properly. User can fill with saving information (containing CarsDataset id), and attributes associated to data

Parameters:

data_dict (Dict) – dictionnary to fill
saving_info (dict) – created by Orchestrator.get_saving_infos
attributes (dict) – attributes associated to data

cars.data_structures.cars_dataset.save_all_dataframe(dataframe, file_name, save_by_pair=False, overwrite=True, point_cloud_format='csv', overwrite_file_name=True)[source]

Save DataFrame to csv and laz format. The content of dataframe is merged to the content of existing saved Dataframe, if overwrite==False The option save_by_pair separate the dataframe by pair (one folder by pair)

Parameters:

file_name (str) – file name to save data to
overwrite (bool) – overwrite file if exists
point_cloud_format (str) – point cloud format (csv or laz)
overwrite_file_name (bool) – generate a new filename from input filename matching input dataframe attributes, using only directory from input filename

cars.data_structures.cars_dataset.save_dataframe(dataframe, file_name, overwrite=True, point_cloud_format='csv')[source]: Save dataframe (csv, laz, attr file)

cars.data_structures.cars_dataset.save_dataset(dataset, file_name, tag, use_windows_and_overlaps=False, descriptor=None, nbits=None)[source]

Reconstruct and save data. In order to save properly the dataset to corresponding tiff file, dataset must have been filled with saving info, profile, window, overlaps (if not 0), and rasterio descriptor if already created. See fill_dataset.

Parameters:

dataset (xr.Dataset) – dataset to save
file_name (str) – file name to save data to
tag (str) – tag to reconstruct
use_windows_and_overlaps (bool) – use saved window and overlaps
descriptor (rasterio dataset) – descriptor to use with rasterio

cars.data_structures.cars_dataset.create_tile_path(col: int, row: int, directory: str) → str[source]

Create path of tile, according to its position in CarsDataset grid

Parameters:

col (int) – numero of column
row (int) – numero of row
directory (str) – path where to save tile

Returns:

full path

Return type:

str

cars.data_structures.cars_dataset.save_numpy_array(array: numpy.ndarray, file_name: str)[source]

Save numpy array to file

Parameters:

array (np.ndarray) – array to save
file_name (str) – numero of row

cars.data_structures.cars_dataset.load_numpy_array(file_name: str) → numpy.ndarray[source]

Load numpy array from file

Parameters:: file_name (str) – numero of row
Returns:: array
Return type:: np.ndarray

cars.data_structures.cars_dataset.create_none(nb_row: int, nb_col: int)[source]

Create a grid filled with None. The created grid is a 2D list : ex: [[None, None], [None, None]]

Parameters:

nb_row – number of rows
nb_col – number of cols

Returns:

Grid filled with None

Return type:

list of list

cars.data_structures.cars_dataset.overlap_array_to_dict(overlap)[source]

Convert matrix of overlaps, to dict format used in CarsDatasets. Input is : [o_up, o_down, o_left, o_right]. Output is : {“up”: o_up, “down”: o_down, “left”: o_left, “right”: o_right}

Parameters:: overlap (List) – overlaps
Returns:: New overlaps
Return type:: Dict

cars.data_structures.cars_dataset.window_array_to_dict(window, overlap=None)[source]

Convert matrix of windows, to dict format used in CarsDatasets. Use overlaps if you want to get window with overlaps inputs are :

window : [row_min, row_max, col_min, col_max], with pixel format

overlap (optional): [o_row_min, o_row_max, o_col_min, o_col_max]

outputs are :

{: “row_min” : row_min - o_row_min, “row_max” : row_max + o_row_max, “col_min” : col_min - o_col_min, “col_max” : col_max - o_col_max,

}

Parameters:

window (List) – window
overlap (List) – overlaps

Returns:

New window

Return type:

Dict

cars.data_structures.cars_dataset.dict_profile_to_rio_profile(dict_profile: Dict) → Dict[source]

Transform a rasterio Profile transformed into serializable Dict, into a rasterio profile.

Parameters:: profile (Dict) – rasterio Profile transformed into serializable Dict
Returns:: Profile
Return type:: Rasterio Profile

cars.data_structures.cars_dataset.rio_profile_to_dict_profile(in_profile: Dict) → Dict[source]

Transform a rasterio profile into a serializable Dict.

Parameters:: in_profile (Dict) – rasterio Profile transformed into serializable Dict
Returns:: Profile
Return type:: Dict

cars.data_structures.cars_dataset.save_dict(dictionary, file_path: str)[source]

Save dict to json file

Parameters:

dictionary (Dict) – dictionary to save
file_path (str) – file path to use

cars.data_structures.cars_dataset.make_yaml_safe(obj)[source]: Converter

cars.data_structures.cars_dataset.load_dict(file_path: str) → Dict[source]

Load dict from json file

Parameters:: file_path (str) – file path to use

cars.data_structures.cars_dataset.separate_dicts(dictionary, list_tags)[source]

Separate a dict into two, the second one containing the given tags.

For example, {key1: val1, key2: val2, key3: val3} with list_tags = [key2] will be split in : {key1: val1, key3: val3} and {key2: val2}

cars.data_structures.cars_dataset.get_attributes(obj)[source]

Get attributes in .attrs of dataset or dataframe

Parameters:: obj (xr.Dataset or pandas.Dataframe) – dataset or dataframe

cars.data_structures.cars_dataset.get_window_dataset(dataset)[source]

Get window in dataset

Parameters:: dataset (xr.Dataset) – dataset

cars.data_structures.cars_dataset.get_overlaps_dataset(dataset)[source]

Get overlaps in dataset

Parameters:: dataset (xr.Dataset) – dataset

cars.data_structures.cars_dataset.get_profile_rasterio(dataset)[source]

Get profile in dataset

Parameters:: dataset (xr.Dataset) – dataset

cars.data_structures.cars_dataset.get_profile_for_tag_dataset(dataset, tag: str) → Dict[source]

Get profile according to layer to save. This function modify current rasterio dataset to fix the number of bands of the data associated to given tag.

Parameters:: tag (str) – tag to use
Returns:: Profile
Return type:: Rasterio Profile

cars.data_structures.cars_dataset.generate_rasterio_window(window: Dict) → rasterio.windows.Window[source]

Generate rasterio window to use.

Parameters:: window (dict) – window to convert, containing ‘row_min’, ‘row_max’, ‘col_min’, ‘col_max
Returns:: rasterio window
Return type:: rio.windows.Window