cars.data_structures.cars_dataset
cars_dataset module:
Module Contents
Classes
CarsDataset. |
Functions
|
Save future when arrived |
|
Save future result when arrived |
|
Load a xarray tile |
|
Load a panda dataframe |
|
Load a CarsDict |
|
Save xarray to directory, saving the data in a different file that |
|
Save dataFrame to directory, saving the data in a different file that |
|
Save cars_dict to directory, saving the data in a different file that |
|
From a full xarray dataset, fill info properly. |
|
From a full pandas dataframe, fill info properly. |
|
From a fulldict, fill info properly. |
|
Save DataFrame to csv and laz format. The content of dataframe is merged to |
|
Save dataframe (csv, laz, attr file) |
|
Reconstruct and save data. |
|
Create path of tile, according to its position in CarsDataset grid |
|
Save numpy array to file |
|
Load numpy array from file |
|
Create a grid filled with None. The created grid is a 2D list : |
|
Convert matrix of overlaps, to dict format used in CarsDatasets. |
|
Convert matrix of windows, to dict format used in CarsDatasets. |
|
Transform a rasterio Profile transformed into serializable Dict, |
|
Transform a rasterio profile into a serializable Dict. |
|
Save dict to json file |
|
Load dict from json file |
|
Separate a dict into two, the second one containing the given tags. |
|
Get attributes field in .attr of dataframe |
|
Get window in dataset |
|
Get overlaps in dataset |
|
Get profile in dataset |
|
Get attributes in dataset |
|
Get profile according to layer to save. |
|
Generate rasterio window to use. |
Attributes
- cars.data_structures.cars_dataset.CARS_DS_TYPE_ARRAY = 'arrays'
- cars.data_structures.cars_dataset.CARS_DS_TYPE_POINTS = 'points'
- cars.data_structures.cars_dataset.CARS_DS_TYPE_DICT = 'dict'
- cars.data_structures.cars_dataset.TILES_INFO_FILE = 'tiles_info.json'
- cars.data_structures.cars_dataset.OVERLAP_FILE = 'overlaps.npy'
- cars.data_structures.cars_dataset.GRID_FILE = 'grid.npy'
- cars.data_structures.cars_dataset.PROFILE_FILE = 'profile.json'
- cars.data_structures.cars_dataset.ATTRIBUTE_FILE = 'attributes.json'
- cars.data_structures.cars_dataset.DATASET_FILE = 'dataset'
- cars.data_structures.cars_dataset.DATAFRAME_FILE = 'dataframe.csv'
- cars.data_structures.cars_dataset.CARSDICT_FILE = 'cars_dict'
- cars.data_structures.cars_dataset.PROFILE = 'profile'
- cars.data_structures.cars_dataset.WINDOW = 'window'
- cars.data_structures.cars_dataset.OVERLAPS = 'overlaps'
- cars.data_structures.cars_dataset.ATTRIBUTES = 'attributes'
- cars.data_structures.cars_dataset.SAVING_INFO = 'saving_info'
- class cars.data_structures.cars_dataset.CarsDataset(dataset_type, load_from_disk=None, name='unknown')
CarsDataset.
Internal CARS structure for organazing tiles (xr.Datasets or pd.DataFrames).
- property shape
Return the shape of tiling grid (nb_row, nb_col) :return: shape of grid
- property tiling_grid
Tiling grid, containing pixel windows of tiles
- Returns
tiling grid, of shape [N, M, 4], containing [row_min, row_max, col_min, col_max]
- Return type
np.ndarray
- __repr__()
Repr function :return: printable self CarsDataset
- __str__()
Str function :return: printable self CarsDataset
- custom_print()
Return string of self :return: printable self
- __getitem__(key)
Get item : return the [row, col] dataset
- Parameters
key – tuple index
- Returns
tile
- Return type
xr.Dataset or pd.DataFrame
- __setitem__(key, newvalue)
Set new tile
- Parameters
key (tuple(int, int)) – tuple of row and col indexes
newvalue – tile to set
- load_single_tile(tile_path_name: str)
Load a single tile
- Parameters
tile_path_name (str) – Path of tile to load
- Returns
single tile
- Return type
xarray Dataset or Panda dataframe to file
- save_single_tile(tile, tile_path_name: str)
Save xarray Dataset or Panda dataframe to file
- Parameters
tile (xr.Dataset or pd.DataFrame) – tile to save
tile_path_name – Path of file to save in
- run_save(future_result, file_name: str, **kwargs)
Save future result when arrived
- Parameters
future_result – xarray.Dataset received
file_name – filename to save data to
- get_window_as_dict(row, col, from_terrain=False, resolution=1)
Get window in pixels for rasterio. Set from_terrain if tiling grid was defined in geographic coordinates.
- Parameters
row (int) – row
col (int) – col
from_terrain (bool) – true if in terrain coordinates
resolution (float) – resolution
- Returns
New window : { “row_min” : row_min , “row_max” : row_max “col_min” : col_min “col_max” : col_max }
- Return type
Dict
- create_grid(nb_col: int, nb_row: int, row_split: int, col_split: int, row_overlap: int, col_overlap: int)
Generate grid of positions by splitting [0, nb_row]x[0, nb_col] in splits of xsplit x ysplit size
:param nb_col : number of columns :param nb_row : number of lines :param col_split: width of splits :param row_split: height of splits :param col_overlap: overlap to apply on rows :param row_overlap: overlap to apply on cols
- generate_none_tiles()
- Generate the structure of data tiles, with Nones, according
to grid shape.
- create_empty_copy(cars_ds)
Copy attributes, grid, overlaps, and create Nones.
- Parameters
cars_ds (CarsDataset) – CarsDataset to copy
- generate_descriptor(future_result, file_name, tag=None, dtype=None, nodata=None)
Generate de rasterio descriptor for the given future result
Only works with pixelic tiling grid
- Parameters
future_result (xr.Dataset) – Future result
file_name (str) – file name to save futures to
tag (str) – tag to save
dtype (str) – dtype
nodata (float) – no data value
- save_cars_dataset(directory)
Save whole CarsDataset to given directory, including tiling grids, attributes, overlaps, and all the xr.Dataset or pd.DataFrames.
- Parameters
directory (str) – Path where to save self CarsDataset
- load_cars_dataset_from_disk(directory)
Load whole CarsDataset from given directory
- Parameters
directory (str) – Path where is saved CarsDataset to load
- cars.data_structures.cars_dataset.run_save_arrays(future_result, file_name, tag=None, descriptor=None)
Save future when arrived
- Parameters
future_result (xarray.Dataset) – xarray.Dataset received
file_name (str) – filename to save data to
tag (str) – dataset tag to rasterize
descriptor – rasterio descriptor
- cars.data_structures.cars_dataset.run_save_points(future_result, file_name, overwrite=False, save_points_cloud_by_pair=False)
Save future result when arrived
- Parameters
future_result (pandas Dataframe) – pandas Dataframe received
file_name (str) – filename to save data to
overwrite (bool) – overwrite file
- cars.data_structures.cars_dataset.load_single_tile_array(tile_path_name: str) xarray.Dataset
Load a xarray tile
- Parameters
tile_path_name (str) – Path of tile to load
- Returns
tile dataset
- Return type
xr.Dataset
- cars.data_structures.cars_dataset.load_single_tile_points(tile_path_name: str)
Load a panda dataframe
- Parameters
tile_path_name (str) – Path of tile to load
- Returns
Tile dataframe
- Return type
Panda dataframe
- cars.data_structures.cars_dataset.load_single_tile_dict(tile_path_name: str)
Load a CarsDict
- Parameters
tile_path_name (str) – Path of tile to load
- Returns
Tile dataframe
- Return type
Panda dataframe
- cars.data_structures.cars_dataset.save_single_tile_array(dataset: xarray.Dataset, tile_path_name: str)
Save xarray to directory, saving the data in a different file that the attributes (saved in a .json next to it).
- Parameters
dataset (xr.Dataset) – dataset to save
tile_path_name (str) – Path of file to save in
- cars.data_structures.cars_dataset.save_single_tile_points(dataframe, tile_path_name: str)
Save dataFrame to directory, saving the data in a different file that the attributes (saved in a .json next to it).
- Parameters
dataframe (pd.DataFrame) – dataframe to save
tile_path_name (str) – Path of file to save in
- cars.data_structures.cars_dataset.save_single_tile_dict(dict_cars, tile_path_name: str)
Save cars_dict to directory, saving the data in a different file that the attributes (saved in a .json next to it).
- Parameters
dict_cars (pd.DataFrame) – dataframe to save
tile_path_name (str) – Path of file to save in
- cars.data_structures.cars_dataset.fill_dataset(dataset, saving_info=None, window=None, profile=None, attributes=None, overlaps=None)
From a full xarray dataset, fill info properly. User can fill with saving information (containing CarsDataset id), window of current tile and its overlaps, rasterio profile of full data, and attributes associated to data
- Parameters
dataset (xarray_dataset) – dataset to fill
saving_info (dict) – created by Orchestrator.get_saving_infos
window (dict) –
profile (dict) –
attributes (dict) –
- cars.data_structures.cars_dataset.fill_dataframe(dataframe, saving_info=None, attributes=None)
From a full pandas dataframe, fill info properly. User can fill with saving information (containing CarsDataset id), and attributes associated to data
- Parameters
dataframe (pandas dataframe) – dataframe to fill
saving_info (dict) – created by Orchestrator.get_saving_infos
attributes (dict) –
- cars.data_structures.cars_dataset.fill_dict(data_dict, saving_info=None, attributes=None)
From a fulldict, fill info properly. User can fill with saving information (containing CarsDataset id), and attributes associated to data
- Parameters
data_dict (Dict) – dictionnary to fill
saving_info (dict) – created by Orchestrator.get_saving_infos
attributes (dict) –
- cars.data_structures.cars_dataset.save_all_dataframe(dataframe, file_name, save_points_cloud_by_pair=False, overwrite=True)
Save DataFrame to csv and laz format. The content of dataframe is merged to the content of existing saved Dataframe, if overwrite==False The option save_points_cloud_by_pair separate the dataframe by pair :param file_name: file name to save data to :type file_name: str :param overwrite: overwrite file if exists :type overwrite: bool
- cars.data_structures.cars_dataset.save_dataframe(dataframe, file_name, overwrite=True)
Save dataframe (csv, laz, attr file)
- cars.data_structures.cars_dataset.save_dataset(dataset, file_name, tag, use_windows_and_overlaps=False, descriptor=None)
Reconstruct and save data. In order to save properly the dataset to corresponding tiff file, dataset must have been filled with saving info, profile, window, overlaps (if not 0), and rasterio descriptor if already created. See fill_dataset.
- Parameters
dataset (xr.Dataset) – dataset to save
file_name (str) – file name to save data to
tag (str) – tag to reconstruct
use_windows_and_overlaps (bool) – use saved window and overlaps
descriptor (rasterio dataset) – descriptor to use with rasterio
- cars.data_structures.cars_dataset.create_tile_path(col: int, row: int, directory: str) str
Create path of tile, according to its position in CarsDataset grid
- Parameters
col (int) – numero of column
row (int) – numero of row
directory (str) – path where to save tile
- Returns
full path
- Return type
str
- cars.data_structures.cars_dataset.save_numpy_array(array: numpy.ndarray, file_name: str)
Save numpy array to file
- Parameters
array (np.ndarray) – array to save
file_name (str) – numero of row
- cars.data_structures.cars_dataset.load_numpy_array(file_name: str) numpy.ndarray
Load numpy array from file
- Parameters
file_name (str) – numero of row
- Returns
array
- Return type
np.ndarray
- cars.data_structures.cars_dataset.create_none(nb_row: int, nb_col: int)
Create a grid filled with None. The created grid is a 2D list : ex: [[None, None], [None, None]]
- Parameters
nb_row – number of rows
nb_col – number of cols
- Returns
Grid filled with None
- Return type
list of list
- cars.data_structures.cars_dataset.overlap_array_to_dict(overlap)
Convert matrix of overlaps, to dict format used in CarsDatasets. Input is : [o_up, o_down, o_left, o_right]. Output is : {“up”: o_up, “down”: o_down, “left”: o_left, “right”: o_right}
- Parameters
overlap (List) – overlaps
- Returns
New overlaps
- Return type
Dict
- cars.data_structures.cars_dataset.window_array_to_dict(window, overlap=None)
Convert matrix of windows, to dict format used in CarsDatasets. Use overlaps if you want to get window with overlaps inputs are :
window : [row_min, row_max, col_min, col_max], with pixel format
overlap (optional): [o_row_min, o_row_max, o_col_min, o_col_max]
- outputs are :
- {
“row_min” : row_min - o_row_min, “row_max” : row_max + o_row_max, “col_min” : col_min - o_col_min, “col_max” : col_max - o_col_max,
}
- Parameters
window (List) – window
overlap (List) – overlaps
- Returns
New window
- Return type
Dict
- cars.data_structures.cars_dataset.dict_profile_to_rio_profile(dict_profile: Dict) Dict
Transform a rasterio Profile transformed into serializable Dict, into a rasterio profile.
- Parameters
profile (Dict) – rasterio Profile transformed into serializable Dict
- Returns
Profile
- Return type
Rasterio Profile
- cars.data_structures.cars_dataset.rio_profile_to_dict_profile(in_profile: Dict) Dict
Transform a rasterio profile into a serializable Dict.
- Parameters
in_profile (Dict) – rasterio Profile transformed into serializable Dict
- Returns
Profile
- Return type
Dict
- cars.data_structures.cars_dataset.save_dict(dictionary, file_path: str, safe_save=False)
Save dict to json file
- Parameters
dictionary (Dict) – dictionary to save
file_path (str) – file path to use
safe_save (bool) – if True, be robust to types
- cars.data_structures.cars_dataset.load_dict(file_path: str) Dict
Load dict from json file
- Parameters
file_path (str) – file path to use
- cars.data_structures.cars_dataset.separate_dicts(dictionary, list_tags)
Separate a dict into two, the second one containing the given tags.
For example, {key1: val1, key2: val2, key3: val3} with list_tags = [key2] will be split in : {key1: val1, key3: val3} and {key2: val2}
- cars.data_structures.cars_dataset.get_attributes_dataframe(dataframe)
Get attributes field in .attr of dataframe
- Parameters
dataframe (pandas dataframe) – dataframe
- cars.data_structures.cars_dataset.get_window_dataset(dataset)
Get window in dataset
- Parameters
dataset (xr.Dataset) – dataset
- cars.data_structures.cars_dataset.get_overlaps_dataset(dataset)
Get overlaps in dataset
- Parameters
dataset (xr.Dataset) – dataset
- cars.data_structures.cars_dataset.get_profile_rasterio(dataset)
Get profile in dataset
- Parameters
dataset (xr.Dataset) – dataset
- cars.data_structures.cars_dataset.get_attributes(dataset)
Get attributes in dataset
- Parameters
dataset (xr.Dataset) – dataset
- cars.data_structures.cars_dataset.get_profile_for_tag_dataset(dataset, tag: str) Dict
Get profile according to layer to save. This function modify current rasterio dataset to fix the number of bands of the data associated to given tag.
- Parameters
tag (str) – tag to use
- Returns
Profile
- Return type
Rasterio Profile
- cars.data_structures.cars_dataset.generate_rasterio_window(window: Dict) rasterio.windows.Window
Generate rasterio window to use.
- Parameters
window (dict) – window to convert, containing ‘row_min’, ‘row_max’, ‘col_min’, ‘col_max
- Returns
rasterio window
- Return type
rio.windows.Window