gdal_warp()was partially rewritten: now reshaped rasters conserve square pixels also after warping operations. This change also affects
sen2r()products if reshaping features (reprojection or custom resolution) are required. This update was necessary due to changes performed in GDAL since version 3.2.0 (see OSGeo/gdal♯3294), which did not grant homogeneity between outputs produced with different GDAL versions. Note that, in case the user wants to update an existing sen2r archive produced with sen2r < 1.4.0, the old existing grid is maintained so that product grids are equal (in order to produce outputs with the new grid, users must create a new archive).
s2_download()is no more able to use existing products equivalent to found ones.
s2_download()no more uses existing SAFE products instead than downloading new equivalent ones (this in order to manage images split in two SAFE archives).
"dhus"API service instead of default
"apihub"(this could be useful in case of
s2_list()to be used for future implementations.
build_example_param_file()now generates more recent images, 2020-08-01 instead than 2019-07-23 (this because the previous ones required two SAFE which are now on LTA); the same was done for some tests.
WVP(see documentation for description). The GUI and internal functions were modified to support their selection and generation.
Bcc) and Excess Green (
shinyWidgets) are now suggested dependencies. In the case they are missing and the user tries to run the GUI, an error is returned with the command for installing them.
sen2rinstallation over Ubuntu Focal.
rgdalis now an explicit dependency (this because it is used by
rasterbut it is not a mandatory dependency).
gdalUtil()). This allows reducing the use of external runtime dependencies.
sysis installed (#257).
*_cloudlist.txt) were created in the directory of the parameter file (if existing) and used, during subsequent
sen2r()executions, in order not to try reprocessing images cloud covered or failed for some reason. Now these files were replaced with an hidden TOML file
.ignorelist.txtcontaining the list of base names of non-produced files and the dates of cloud covered images. This file is placed in the output folder, so to be linked with the produced archive. The usefulness of this edit is twofold:
rm_invalid_safe(): now the presence of the JP2 images required by the processing chain is checked by default. L2A products without all the three resolutions will not be used.
"res"to the default
safe_getMetadata()output, with the list of the available resolutions (“
all” is returned in case of L1C products).
sen2cor()to limit the maximum Sen2cor execution time (after that time, Sen2Cor is stopped). This is useful to avoid blocking processing chains due to internal Sen2Cor errors.
sys(it is used to call Sen2Cor using
system(), allowing exporting Sen2Cor output to log files).
rgdal(since it is implicitly called by internal functions), as requested by CRAN policy.
toolsto suggested dependencies.
leaflet*dependencies to suggested (this will be done in a future release).
RETRY(), so to avoid errors in case of temporary unavailability of services.
init_python()(this urgent bug fixing was the reason of the urgent release of the current version).
sen2r(..., rm_safe = "yes") and
max_mask < 100and
mask_typeis not specified, a warning is returned).
st_crs2()accepting also WKT strings / EPSG codes in the form
sen2r()a short report is returned to output summarising the status of the required processing (see new internal function
sen2r_process_report()). This should facilitate applying sen2r in a cycle until all “expected dates” are processed.
sen2r()(see below) and a new option was added in the GUI.
init_python()was deprecated (Python is no longer managed by
bigtifffor BigTIFF management in functions
sen2r()) to manage GIPP parameters in Sen2Cor.
sen2r()was deprecated (no longer needed).
safe_getMetadata()(from existing offline SAFE archives) and
s2_list()(from SciHub online metadata).
st_crs2()using class methods (see the function help for newly accepted inputs).
safe_getMetadata()now directly read XML SAFE files instead than calling GDAL through
reticulate(this avoids errors in particular OS conditions).
safelistclass for lists of SAFE Sentinel-2 archives (see safelist-class);
sen2r()execution now causes saving the used parameters in a json file located in
~/.sen2r/proc_par(this can be used to take trace of the executed processing chains);
safe_getMetadata()function was rewritten: now files are scanned only if the user requires metadata which must be retrieved from file content, otherwise only file names are analysed. Inputs can be also
safelistobjects. The support for oldname SAFE products is deprecated.
safe_getMetadata(): new arguments
allow_oldnames(see the function reference for details);
output_typewas deprecated (use
as.data.tableto obtain a data.table instead than the new default
multiple_nameswere deprecated, since they are not used with SAFE compact names (old names are no more supported);
Sys.setenv()effects now do not affect the R environment after exiting from sen2r() execution;
safe_shortname()does no more support oldname SAFE products;
Starting from this version sen2r supports ordering products from Long Term Archive (LTA) (see news at https://inthub.copernicus.eu/userguide/LongTermArchive). Now the user can automatically order SAFE products which are not available for direct download, and use them when made available. Some internal functions can be exploited to manually manage orders.
Here above the related changes:
safe_is_valid()to check if an order was processed;
s2_order()to order products from LTA.
sen2r was accepted on CRAN (2019-10-21, version 1.1.0). From now, it is possible to install the CRAN version with the following command:
The development version – which may contain updated features – can still be installed with the following command:
sen2r()is NULL instead of
'no'value for argument
'l2a'without selecting any L2A products instead).
sen2r()) instead than 10/20 m.
parallel = FALSEas default in
sen2r()(parallelisation must be explicitly required).
sen2rpackage directory is no more used to store files, but a subfolder
.sen2rof the user Home directory is created and used. In this way, reinstalling sen2r will not require anymore to reinstall runtime dependencies / to reconfigure the package. The permission to write on this folder is asked to the user when the package is loaded for the first time.
s2_download()now use internal methods instead of calling python function derived from s2download scripts by Hagolle (this substantially improve the speed of SAFE search), which were removed.
sen2ris now supported also on MacOS.
rgdalfrom explicit dependencies.
str_pad2(), equivalent to str_pad() without the needing to load
raster_metadata(), to get raster metadata, without rgdal dependency.
check_scihub_login()to check if SciHub credentials are right.
check_scihub_connection()to check for internet connectivity.
build_example_param_file()(not exported) to build an example parameter file.
check_gdal()was rewritten without using
gdalUtils, in order to be simpler and faster.
"by groups"instead than
"step by step": this ensures best computing speed and improves the management of disk usage, although it is not possible to see all the output messages in standard output (to see them, it is necessary to redirect them in a log file).
"cloud_low_proba"is not yet available in the GUI (if is still a valid argument value for compatibility), and
"cloud_and_shadow"now includes class “unclassified”. This in order to include “unclassified” only in “conservative” mask values, and to exclude it in non-conservative ones.
httr: a change in Wget available version for Windows (1.20.3 instead than 1.19.4) caused the old link to break; moreover, downloading Wget using the new link was not possible with
download.file()due to a redirect. So, httr dependency was introduced, and download of Wget, aria2 and sen2cor is now performed with GET function.
dplyr: the internal function
tiles_intersects()now perform dissolve operations over polygons of S2 tiles using
s2_calcindices(): now internal raster routines are used by default instead than
gdal_calc.py, allowing reducing RAM usage and computation time.
sen2r_getElements()was rewritten in order to: 1) avoid a RAM leak present in the function, which was causing a high RAM use in the first part of the execution of
sen2r(); 2) considerably speed up the execution (now the metadata of 100 files is retrieved in 4 milliseconds instead than 80).
s2_calcindices()uses internal routines; the legacy mode can be used with the argument
proc_mode = "gdal_calc"(default value is
list_indices()now returns only checked indices by default (specify
all = TRUEto return also non-checked ones).
sen2r_getElements()is now a data.table instead than a list (use
format = "list"to reproduce the old default).
First stable release of package sen2r! See the announcement here.
path_indicesis not yet necessary (if empty,
tiles_intersects(): when an extent is loaded, now only required tiles are automatically used, instead than all the overlapping ones.
sen2r()documentation. The 4 different modes allow to privilege stability, velocity or disk space. In this version, the legacy mode (“by_step”) is maintained as the default.
s2_rgb()to produce RGB images, integrating it in
sen2r()and in the GUI.
s2_gui()was improved by adding a modaldialog which explains how to apply buffer and smoothing to cloud masks)
safe_isvalid()to check for valid SAFE archive names, and use it to filter existing SAFE folders (#104)
This is an improvement of version 0.3.2, with several fixes and improvements.
s2_dop()to know the Dates Of Passage over orbits
sen2r()and the GUI
This is an improvement of version 0.3.1, with several fixes and improvements.
Major GUI improvements in the definition of the extent: now the main map is used only to show the extent and the tiles, while the definition of the extent (drawn, specified with a bounding box or taken from a loaded polygon) is done in separate maps (this allows to avoid many errors).
Add radiobutton for index datatype, to choose the datatype when an index is generated.
Add maximum cloud cover threshold.
Implement seasonal time period (#74).
Implement the download with aria2c.
Major commit: retrieve also existing filenames. Before this commit, if some products were partially retrieved (e.g. output BOA) and some depending on them are missing (e.g. an index) but SAFE were not required, the corresponding expected vectors were not retrieved (missing SAFE tile names nothing was generated). Now, existing names matching the parameters are attached to each "_exp" vector.
Improvement in integer / byte data types for generation of indices: Int16 is now the default data type; choosing an integer data type the output values are clipped on the valid format range, and the scaling factor can be chosen by the user; choosing Byte, interval -1 to 1 is coerced to 0-200 (with nodata=255).
Use standard nodata values to compute indices.
Return the list of created files after processing.
Accept custom mask combinations: now it is possible to define masks with a custom combination of SCL classes to be masked.
Allow argument “timewindow” in
sen2r() to be a single integer value, which is interpreted as the period between today and n days ago.
Uniform temporary directories (#73). Now the principal functions have the parameters tmpdir and rmtmp to set the temporary directory and the choice to delete /not to delete temporary files. Only two exceptions: create_indices_db() uses a fixed temporary directory (since it is a function intended not to be run by end users); sen2cor() launched from sen2r() uses a default temporary directory if
tmpdir is in a SAMBA mountpoint over Windows.
Update to sen2cor 2.5.5, and manage S2A L2A operational products (https://scihub.copernicus.eu/news/News00305).
Add a welcome message and a logo.
Change package name from SALTO to sen2r, and rename the main function consequently to
Remove s2download as dependence (now it is integrated within the package).
Remove sprawl dependencies.
Avoid blurs and artifices in thumbnails. stack2rgb() now works with GDAL routines; gdalwarp is used instead than gdal_translate for reducing the size of input images (to avoid averaging NA values, which produced blurs on borders of BOA thumbnails); intermediate GTiff files are used to bypass the presence of artifices in SCL PNG thumbnails and to produce JPEG multiband TOA/BOA thumbnails.
Load python modules correctly. Now on Windows all the modules are imported using import_from_path() in init_python(), which is called anywhere modules are needed.
Fix unrecognised osgeo in Windows. When running gdal_calc in Windows with multiple Python installations, the wrong one (i.e. Anaconda) is used, and “no module names osgeo” error is returned.
Other minor fixes.
This is an improvement of version 0.3.1, with several fixes and secondary improvements:
This version allows to perform all basic operations with Sentinel-2 data: finding and downloading the required tiles (on Linux), correcting them with sen2cor, processing them (merging adjacent tiles, clipping, reprojecting, rescaling, applying atmospheric mask) to create output multiband files (in a format managed by GDAL) and spectral indices.
This is an intermediate version, which does not allow yet to perform a complete workflow, but it allows to manually find (
s2_list()), download (
s2_download()), correct (
s2_sen2cor()), translate in a format managed by GDAL (
s2_translate()) and merge by orbit number (
s2_merge()) Sentinel-2 data.
An example of possible manual workflow is shown in the temporary script
WIP/tutorial.R. For a complete automatic workflow on Sentinel-2 data, wait version 0.3.0.