How to contribute
CARS is an open source software : don’t hesitate to hack it and contribute !
Please go to the GitHub repository for source code.
Read also CARS Contribution guide with LICENCE and Contributor Licence Agrements.
Contact: cars AT cnes.fr
Developer Install
Install procedure is globally followed but adapted to get CARS development environment. Obviously, we recommend to use a virtualenv environment, so that CARS do not interfere with other packages installed on your system.
Install OTB and VLFeat: see Dependencies.
Clone CARS repository from GitHub :
$ git clone https://github.com/CNES/cars.git
$ cd cars
Install CARS in a virtualenv in developer mode
$ make install-dev # CARS installed in ``venv`` virtualenv
Run CARS in virtualenv
$ source venv/bin/activate
$ source venv/bin/env_cars.sh
$ cars -h
The detailed development install method is described in Makefile
Particularly, it uses the following pip editable install:
pip install -e .[dev]
With this pip install mode, source code modifications directly impacts cars
command line.
Coding guide
Here are some rules to apply when developing a new functionality:
Comments: Include a comments ratio high enough and use explicit variables names. A comment by code block of several lines is necessary to explain a new functionality.
Test: Each new functionality shall have a corresponding test in its module’s test file. This test shall, if possible, check the function’s outputs and the corresponding degraded cases.
Documentation: All functions shall be documented (object, parameters, return values).
Use type hints: Use the type hints provided by the typing python module.
Use doctype: Follow sphinx default doctype for automatic API
Quality code: Correct project quality code errors with pre-commit automatic workflow (see below)
Factorization: Factorize the code as much as possible. The command line tools shall only include the main workflow and rely on the cars python modules.
Be careful with user interface upgrade: If major modifications of the user interface or of the tool’s behaviour are done, update the user documentation (and the notebooks if necessary).
Logging and no print: The usage of the print() function is forbidden: use the logging python standard module instead.
Limit classes: If possible, limit the use of classes at one or 2 levels and opt for a functional approach when possible. The classes are reserved for data modelling if it is impossible to do so using xarray and for the good level of modularity.
Limit new dependencies: Do not add new dependencies unless it is absolutely necessary, and only if it has a permissive license.
Pre-commit validation
A pre-commit validation is installed with code quality tools (see below). It is installed automatically by make install-dev command.
Here is the way to install it manually:
$ pre-commit install -t pre-commit # for commit rules
$ pre-commit install -t pre-push # for push rules
This installs the pre-commit hook in .git/hooks/pre-commit and .git/hooks/pre-push from .pre-commit-config.yaml file configuration.
It is possible to test pre-commit before committing:
$ pre-commit run --all-files # Run all hooks on all files
$ pre-commit run --files cars/__init__.py # Run all hooks on one file
$ pre-commit run pylint # Run only pylint hook
Documentation
CARS contains its Sphinx Documentation in the code in docs directory.
To generate documentation, use:
$ make docs
The documentation is then build in docs/build directory and can be consulted with a web browser.
Documentation can be edited in docs/source/ RST files.
Jupyter notebooks tutorials
CARS contains tutorials and quick start scripts in tutorials directory.
To generate a Jupyter kernel with CARS installation, use:
$ make notebook
Follow indications to run a jupyter notebook.
Kernel is created with following command (with cars-version updated):
$ python -m ipykernel install --sys-prefix --name=cars-version --display-name=cars-version
To run the jupyter notebook, use:
$ jupyter notebook
To generate slides of CARS main tutorial jupyter notebook :
$ cd tutorials
$ jupyter nbconvert main_tutorial.ipynb --to slides --post serve # for remote reveal.js lib
or
$ cd tutorials
$ git clone https://github.com/hakimel/reveal.js.git
$ jupyter nbconvert main_tutorial.ipynb --to slides --reveal-prefix reveal.js
$ jupyter nbconvert main_tutorial.ipynb --to html --template reveal --reveal-prefix reveal.js # Same
$ python -m server.http 8000
Slides can be then seen following http://localhost:8000/ URL.
Go to Jupyter Nbconvert documentation for details.
Code quality
CARS uses Isort, Black, Flake8 and Pylint quality code checking.
Use the following command in CARS virtualenv to check the code with these tools:
$ make lint
Use the following command to format the code with isort and black:
$ make format
Isort
Isort is a Python utility / library to sort imports alphabetically, and automatically separated into sections and by type.
CARS isort
configuration is done in pyproject.toml
Isort manual usage examples:
$ cd CARS_HOME
$ isort --check cars tests # Check code with isort, does nothing
$ isort --diff cars tests # Show isort diff modifications
$ isort cars tests # Apply modifications
Isort messages can be avoided when really needed with “# isort:skip” on the incriminated line.
Black
Black is a quick and deterministic code formatter to help focus on the content.
CARS black
configuration is done in pyproject.toml
If necessary, Black doesn’t reformat blocks that start with “# fmt: off” and end with # fmt: on, or lines that ends with “# fmt: skip”. “# fmt: on/off” have to be on the same level of indentation.
Black manual usage examples:
$ cd CARS_HOME
$ black --check cars tests # Check code with black with no modifications
$ black --diff cars tests # Show black diff modifications
$ black cars tests # Apply modifications
Flake8
Flake8 is a command-line utility for enforcing style consistency across Python projects. By default it includes lint checks provided by the PyFlakes project, PEP-0008 inspired style checks provided by the PyCodeStyle project, and McCabe complexity checking provided by the McCabe project. It will also run third-party extensions if they are found and installed.
CARS flake8
configuration is done in setup.cfg
Flake8 messages can be avoided (in particular cases !) adding “# noqa” in the file or line for all messages. It is better to choose filter message with “# noqa: E731” (with E371 example being the error number). Look at examples in source code.
Flake8 manual usage examples:
$ cd CARS_HOME
$ flake8 cars tests # Run all flake8 tests
Pylint
Pylint is a global linting tool which helps to have many information on source code.
CARS pylint
configuration is done in dedicated .pylintrc file.
Pylint messages can be avoided (in particular cases !) adding “# pylint: disable=error-message-name” in the file or line. Look at examples in source code.
Pylint manual usage examples:
$ cd CARS_HOME
$ pylint tests cars # Run all pylint tests
$ pylint --list-msgs # Get pylint detailed errors information
Tests
CARS includes a set of tests executed with pytest tool.
To launch tests:
make test
It launches only the unit_tests
and pbs_cluster_tests
test targets
Before the tests execution, the CARS_TEST_TEMPORARY_DIR
can be defined to indicate where to write the temporary data bound to the test procedure (if the variable is not set, cars will use /tmp
).
Several kinds of tests are identified by specific pytest markers:
the unit tests defined by the
unit_tests
marker:make test-unit
the PBS cluster tests defined by the
pbs_cluster_tests
marker:make test-pbs-cluster
the Jupyter notebooks test defined by the
notebook_tests
marker:make test-notebook
Advanced testing
To execute the tests manually, use pytest
at the CARS projects’s root (after initializing the environment):
$ python -m pytest
To run only the unit tests:
$ cd cars/
$ pytest -m unit_tests
To run only the PBS cluster tests:
$ cd cars/
$ pytest -m pbs_cluster_tests
To run only the Jupyter notebooks tests:
$ cd cars/
$ pytest -m notebook_tests
It is possible to obtain the code coverage level of the tests by installing the pytest-cov
module and use the --cov
option.
$ cd cars/
$ python -m pytest --cov=cars
It is also possible to execute only a specific part of the test, either by indicating the test file to run:
$ cd cars/
$ python -m pytest tests/test_tiling.py
Or by using the -k
option which will execute the tests which names contain the option’s value:
$ cd cars/
$ python -m pytest -k end2end
By default, pytest
does not display the traces generated by the tests but only the tests’ status (passed or failed). To get all traces, the following options have to be added to the command line (which can be combined with the previous options):
$ cd cars/
$ python -m pytest -s -o log_cli=true -o log_cli_level=INFO