2. Detailed Processing Model Documentation (DPMD)¶
2.1. Introduction¶
This is the Detailed Processing Model Documentation (DPMD) for the Sen2Three application.
2.1.1. Purpose and Scope¶
This document is produced in the context of the development of the Sentinel 2 level 3 processor. Its purpose is to detail the algorithms of the software and to list and describe the individual modules classes and functions of the software.
2.1.2. Applicable Documents¶
Table 2-1: Applicable Documents
Reference | Code | Title | Issue |
---|---|---|---|
L3_DPMD | ATBD | Level 3 ATBD | 1.0 |
2.1.3. Reference Documents¶
Table 2-2: Reference Documents
Reference | Code | Title | Issue |
---|---|---|---|
L3_SUM | SUM | Software Installation and User Manual | 1.0 |
L3_PFS | PFS | Product Format Specification | 1.0 |
L3_IODD | IODD | Input Output Data Definition | 1.0 |
2.1.4. Acronyms and Abbreviations¶
All acronyms and abbreviations are listed in [L3-GLODEF]
2.1.5. Document Structure¶
This is the second part of a set of four documents describing the Sentinel 2 level 3 Processor for Spatio- Temporal synthesis, consisting of:
- Software Installation and User’s Manual (SUM);
- Detailed Processing Model Documentation (DPMD), this document;
- Product Format Specification (PFS);
- Input Output Data Definition (IODD).
Chapter 2.2, the High Level Processing Algorithm gives an overview on the processors logic in form of pseudocode and the corresponding function calls.
Chapter 2.3, the Module Description gives an overwiev on all Modules together with a short description on the functionality.
Chapter 2.4, the Detailed Module Listing will list for each Processor Module the dependency to the other modules, followed by a listing of all classes, their relevant member functions and their corresponding in and output parameters.
2.2. High Level Processing Algorithm¶
The high level processing algorithm acts as as follows:
- Input is a directory of users choice. This can contain a series of users products which all should belong to the same area of interest, but with different time stamps.
- The algorithm will perform a chronological sort and processes L2A products in a sequential way.
- All unprocessed L2A scenes will be processed. If no more unprocessed scenes are found, the algorithm will generate a L3 output product.
- The processor keeps a registry of already processed input data. The user can place new collected input data into the directory and restart the processing. The L3 product will be updated, containing the already processed scenes and the new collected ones.
- On users choice, the history of the processing can be deleted. The processor then will then generate a new L3 product and performs a new synthesis of all products contained in the input directory.
Step 1 - process L2A tiles:
for each unprocessed L2A product in source directory:
if product is within given time range:
import L2A product metadata;
if current L3 product not present:
create L3 product;
for each tile in L2A product:
import L2A tile metadata;
import L2A bands;
if current L3 tile not present:
create L3 tile;
else:
perform L3 synthesis;
update L3 tile metadata;
update L3 bands;
Step 2 - post processing:
update L3 product metadata;
for each tile in L3 product:
update L3 tile metadata;
export L3 bands;
2.2.1. Functional Groups¶
2.2.2. Functional Group 1: Product Initialization¶
From main:
create single object of class type L3_Config and initialize;
L3_Config.updateUserProduct:
if L3_Product.existL3_TargetProduct() == True:
for tile in tile list:
if L3_Tables.testBand == true:
tile already present:
L3_Product.reinitL3_TargetProduct;
L3_Product.appendTile;
else:
new tile found:
L3_Product.createL3_Tile;
else:
create L3 target product and initialize:
L3_Product.createL3_TargetProduct;
2.2.3. Functional Group 2: Table Initialization¶
From main:
create single object of class type L3_Tables;
L3_Tables.init:
if Database exist:
L3_Tables.importBandList('L2A'):
convert JPEG 2000 images to database;
else:
L3_Tables.initDatabase;
L3_Tables.importBandList('L3');
convert JPEG 2000 images to database;
L3_Tables.createPreviewImage('L3');
L3_Tables.testBand:
L3_Tables.verifyProductId;
L3_Tables.getBandNameFromIndex;
2.2.4. Functional Group 3: Pixel Mask Generation¶
From main:
create single object of class type L3_Synthesis;
L3_Sythesis.preProcessing:
L3_Synthesis.setPixelMasks:
1. initialize good and bad pixels;
2. create mosaic map, if not exist, else read from L3_Tables;
3. replace always bad pixels of previous classification map, if 'better' pixels exist;
4. update mosaic map and scene classification map according to selected algorithm:
if self.config.algorithm == 'MOST_RECENT':
previous classification map will always be replaced with good pixels of recent classification map;
else if self.config.algorithm == 'TEMP_HOMOGENEITY':
previous classification map will only be replaced if sum of current good pixels
is better than sum of good pixels of the best classification map in the past;
else if self.config.algorithm == 'RADIOMETRIC_QUALITY':
previous classification map will be replaced if either:
- the average of the current AOT or
- the average of the current Solar Zenith Angle
is better than the equivalent parameter of the best classification map in the past;
else if self.config.algorithm == 'AVERAGE':
images are an average of the current good pixels and the
good pixels of all previous scenes. Mosaic map is the per pixel sum
of all good pixels in the past and is used for calculating the average;
5. update good and bad pixel masks;
Subroutines:
L3_Tables.testBand;
L3_Tables.getBand;
L3_Tables.setBand;
2.2.5. Functional Group 4: Synthesis Algorithms¶
From main:
L3_Sythesis.forwardProcessing:
the default processing routine, removes clouds and dark features.
for all bands in tile:
L3_Sythesis.replaceBadPixels:
1. get the bands from previous and current scene;
2. get the good pixel masks from function L3_Synthesis.setPixelMasks;
3. replace always bad pixels of previous scene, if 'better' pixels exist in recent scene;
4. update bands according to selected algorithm:
if self.config.algorithm == 'MOST_RECENT':
previous pixels will always be replaced with good pixels of recent scene;
else if self.config.algorithm == 'TEMP_HOMOGENEITY':
previous pixels will only be replaced if sum of current good pixels
is better than sum of good pixels of the best scene in the past;
else if self.config.algorithm == 'RADIOMETRIC_QUALITY':
previous pixels will be replaced if either:
- the average of the current AOT or
- the average of the current Solar Zenith Angle
is better than the equivalent parameter of the best scene in the past;
else if self.config.algorithm == 'AVERAGE':
images are an average of the current good pixels and the
good pixels of all previous scenes. Mosaic map is the per pixel sum
of all good pixels in the past and is used for calculating the average;
L3_Synthesis.updateTileStatistics:
1. update metadata for mosaic map;
2. update metadata for scene classification map;
3. update metadata on tile base;
Subroutines:
L3_Tables.getBand;
L3_Tables.setBand;
2.2.6. Functional Group 5: Level 3 Product Export¶
From main:
L3_Process.postProcess:
L3_Sythesis.postProcess:
L3_Synthesis.updateProductStatistics:
validate the metadata;
L3_Tables.exportTile:
L3_Tables.exportBandList:
convert database to JPEG 2000 images;
L3_Product.postProcess:
copy and terminate the logging;
2.3. Module Description¶
The Sen2Three application is designed by the following essential modules (classes):
- L3_Process: main processor module, coordinates the interaction between all other modules.
- L3_Config: this class keeps the configuration parameters for an easy access for all other modules.
- L3_Product: this class provides the generation and access to the xml based metadata forming the level 2a and level 3 products. It has a reference to an L3_Config object.
- L3_Tables: this class manages the conversion of the JPEG-2000 based input data to an internal format (and vice versa). It provides a high performance access to the image data for all other modules. It has a reference to an L3_Product object.
- L3_Synthesis: this class performs the four level 3 synthesis algorithms: Most Recent, Temporal Homogeneity, Radiometric Quality and Average wich are detailed in the Algorithms section of the [SUM]. It has a reference to an L3_Tables module.
- L3_Display: this is an an optional class for the data display and plot. It can be activated via a configurable option.
- L3_XmlParser: this is a helper class, wrapping the access to the tree based xml metadata into the form of high level input output routines. It also provides a validator for the consistency check of the product metadata, using xsd schemes.
2.4. Detailled Module Listing¶
It follows a detailed listing of all Modules, Classes and Functions
2.4.1. L3_Process¶
-
class
sen2three.L3_Process.
L3_Process
(config)¶ The main processor module, which coordinates the interaction between the other modules.
Parameters: config (a reference to the config object) – the config object for the current tile (via __init__) -
postProcessing
()¶ Perform the L3 post processing, triggers the export of L3 product, tile metadata and bands
Returns: true if export succeeds, false else Return type: bool
-
preProcessing
()¶ Perform the L3 pre processing, currently empty.
-
-
sen2three.L3_Process.
doTheLoop
(config)¶ Initializes a product and processor object. Cycles through all input products and granules and calls the sequential processing of the individual tiles if the criteria for processing are fulfilled.
Parameters: config (a reference to the config object) – the config object Returns: the processor object, for doing the preprocessing, -1 if processing error occurred. Return type: processor object or -1
-
sen2three.L3_Process.
main
(args=None)¶ Processes command line, initializes the config and product modules and starts sequentially the L3 processing
2.4.2. L3_Config¶
-
class
sen2three.L3_Config.
L3_Config
(resolution, sourceDir=None, configFile='L3_GIPP')¶ Provide the configuration input of the GIPP (via __init__).
Parameters: -
appendTile
()¶ Append a tile to the product history.
Returns: true, if operation successful. Return type: boolean
-
checkTileConsistency
(tilePath, tileId)¶ Low level routine for determine if tile should be processed
Parameters: - tilePath – the tile path
- tileId – the current tile ID
Returns: true, if conditions are fulfilled
Return type: boolean
-
checkTimeRange
(userProduct)¶ Check if current user product is within configured time range.
Parameters: userProduct (str) – the user product ID. Returns: true, if within time range, false else. Return type: boolean
-
exitError
(reason=None)¶ Throw an error message if a fatal error occurred and and terminate the application.
-
getFloat
(label, key)¶ Low level routine for getting a flot32 value from configuration file.
Parameters: Returns: the parameter value.
Return type: float32
-
getFloatArray
(node)¶ Low level routine for getting a float32 array from configuration file.
Parameters: node (str) – the parameter node. Returns: the parameter array. Return type: numpy array
-
getInt
(label, key)¶ Low level routine for getting an int value from configuration file.
Parameters: Returns: the parameter value.
Return type:
-
getIntArray
(node)¶ Low level routine for getting an integer array from configuration file.
Parameters: node (str) – the parameter node. Returns: the parameter array. Return type: numpy array
-
getNrTilesProcessed
()¶ Get the number of processed tiles.
Returns: number of processed tiles. Return type: unsigned int
-
getNrTilesToProcess
()¶ Get the number of tiles to process.
Returns: number of processed tiles. Return type: unsigned int
-
getStr
(label, key)¶ Low level routine for getting a string value from configuration file.
Parameters: Returns: the parameter value.
Return type:
-
getUintArray
(node)¶ Low level routine for getting an unsigned integer array from configuration file.
Parameters: node (str) – the parameter node. Returns: the parameter array. Return type: numpy array
-
init
(processorVersion)¶ Perform the base initialization.
Parameters: processorVersion (str) – the version string.
-
initLogger
()¶ Initialises the logging system.
-
parNotFound
(parameter)¶ Throw a fatal error message if a configuration parameter is not found and terminate the application.
-
putArrayAsStr
(a, node)¶ Low level routine for setting a numpy array as string into configuration file. Can be one or two dimensional.
Parameters: - a (numpy array) – the numpy array
- node (str) – the parameter node.
Returns: false, if array exceeds two dimensions.
Return type: boolean
-
readGipp
()¶ Reads the GIPP file and performs initialisation of the configuration properties.
-
readTileMetadata
(tileID)¶ Read the metadata for the given tile ID.
Parameters: tileID (str) – the tile ID.
-
reinitTile
(L3_TILE_ID)¶ reinit tile after change has been performed :param L3_TILE_ID: the current L3 tile ID :type tile: str
Returns: true, if present. Return type: boolean
-
setProductVersion
()¶ Sets the product version from metadata.
Returns: true, if operation successful. Return type: boolean
-
setTimeEstimation
(resolution)¶ Set the time estimation for the current processing.
Parameters: resolution (unsigned int) – the selected resolution.
-
tileExists
(tileId)¶ Check if a tile is present in the product history.
Parameters: tileId (str) – the tile identifier. Returns: true, if present. Return type: boolean
-
tileIsSelected
(tile, tileFilter)¶ Low level routine for determine if tile should be processed
Parameters: - tile (str) – the current tile ID
- tileFilter (an array of strings) – list of accepted tiles.
Returns: false, if tile not in filter list
Return type: boolean
-
timestamp
(procedure)¶ Marks a time stamp for the duration of the previous procedure.
Parameters: procedure (str) – the previous procedure.
-
writeTimeEstimation
(resolution, tMeasure)¶ Store the time estimation for the current processing.
Parameters: - resolution (unsigned int) – the selected resolution.
- tMeasure (float 32) – the measured performance data in seconds + fractions.
-
2.4.3. L3_Product¶
-
class
sen2three.L3_Product.
L3_Product
(config)¶ Parameters: config (a reference to the L3_Config object) – the config object for the current tile (via __init__) -
checkCriteriaForTermination
()¶ Check if one of the two criteria for termination is reached:
Returns: true if reached Return type: bool
-
createTable
()¶ Create a HDF5 table
-
existL3_TargetProduct
(userProduct)¶ Check if an L3 target product already exists. If not, trigger the creation, else trigger the reinitialisation of the existing one
Returns: true if succesful Return type: bool
-
getTableVal
(key)¶ Check if one of the two criteria for termination is reached: :param key: the search key. :type key: str :return: the value :rtype: bool
-
insert
(tree, node)¶ A support class, inserting a new node at the correct location in the given tree.
Parameters: - tree (an Objectify Element.) – the QI subtree.
- node (an Objectify Element.) – the QI node to insert.
Returns: true if succesful
Return type:
-
postProcessing
()¶ update the user product and product metadata, copy the logfile to QI data as a report, Check if at target product is present.
-
setTableVal
(key, value)¶ Check if one of the two criteria for termination is reached: :param key: the search key. :type key: str :param value: the value to be passed. :type value: float :return: false if wrong key :rtype: bool
-
updateProductMetadata
()¶ Update the product metadata for each new synthesis.
-
updateTableRow
(classification)¶ Update the statistics of a row in the table.
Parameters: classification (an Objectify element) – the statistics
-
2.4.4. L3_Tables¶
-
class
sen2three.L3_Tables.
L3_Tables
(product)¶ A support class, managing the conversion of the JPEG-2000 based input data to an internal format based on HDF5 (pyTables). Provide a high performance access to the image data for all bands of all tiles to be processed.
Parameters: config (a reference to the L3_Config object.) – the config object for the current tile. -
createPreviewImage
(productLevel)¶ Create an RGB preview image from bands 2-4.
Parameters: productLevel (str) – [L1C | L2A | L3]. Returns: false if image cannot be created, else true. Return type: boolean
-
createTci
(productLevel)¶ Create an RGB TCI image from bands 2-4.
Parameters: productLevel (str) – [L1C | L2A | L3]. Returns: false if image cannot be created, else true. Return type: boolean
-
delBand
(productLevel, bandIndex)¶ Delete a single band from H5 database.
Parameters: - productLevel (str) – [L1C | L2A | L3].
- bandIndex (unsigned int) – the band index.
Returns: false if error occurred during deletion of band.
Return type: boolean
-
delBandList
(productLevel)¶ Delete the complete list of bands from H5 database.
Parameters: productLevel (str) – [L1C | L2A | L3]. Returns: false if error occurred during deletion of band. Return type: boolean
-
delDatabase
()¶ Delete the H5 database.
Returns: true if succeeds, false if database does not exist. Return type: boolean
-
exportBandList
(productLevel)¶ Export all bands of current tile. converts all bands from hdf5 to JPEG-2000.
Parameters: productLevel (str) – [L1C | L2A | L3]. Returns: false if error occurred during export. Return type: boolean
-
exportTile
(L3_TILE_ID)¶ Prepare the export of a synthesized tile.
Parameters: identifier (str) – the tile ID.
-
getBand
(productLevel, bandIndex, dataType=<type 'numpy.uint16'>)¶ Get a single band from database.
Parameters: - productLevel (str) – [L1C | L2A | L3].
- bandIndex (unsigned int) – the band index.
- dataType (default: unsigned int 16) – data type of band.
Returns: the pixel data.
Return type: a 2 dimensional numpy array (row x column) of type unsigned int 16
-
getBandSize
(productLevel, bandIndex)¶ Get size of image array.
Parameters: - productLevel (str) – [L1C | L2A | L3].
- bandIndex (unsigned int) – the band index.
Returns: image size (nrows x ncols)
Return type: data tuple (unsigned int).
-
getDataType
(productLevel, bandIndex)¶ Get data type of image array.
Parameters: - productLevel (str) – [L1C | L2A | L3].
- bandIndex (unsigned int) – the band index.
Returns: data type.
Return type:
-
importBand
(bandIndex, filename)¶ convert JPEG-2000 input file to internal H5 file format.
Parameters: - bandIndex (unsigned int) – the band index.
- filename (str) – file name of JPEG-2000 input image.
Returns: false if error occurred during import.
Return type: boolean
-
importBandList
(productLevel)¶ Import all bands of current tile.
Parameters: productLevel (str) – [L1C | L2A | L3]. Return type: none.
-
init
()¶ Checks the existence of a L3 target database for the processed tile. If the database exists, the given tile will be imported. If the database does not exist it will be created and the current tile will become the base for the subsequent processing.
-
initDatabase
()¶ Initialize H5 target database for usage.
-
scalePreview
(arr)¶ Scale image array for preview. Helper function used by createRgbImage().
Parameters: arr (2 dimensional numpy array (nrow x ncols).) – the image array. Returns: false if image cannot be created, else true. Return type: boolean
-
scaleTci
(arr)¶ Scale TCI array. Helper function used by createTci().
Parameters: arr (2 dimensional numpy array (nrow x ncols).) – the image array. Returns: false if image cannot be created, else true. Return type: boolean
-
setBand
(productLevel, bandIndex, arr)¶ Set a single band from numpy array to H5 database.
Parameters: - productLevel (str) – [L1C | L2A | L3].
- bandIndex (unsigned int) – the band index.
- arr (a 2 dimensional numpy array (row x column) of type unsigned int 16.) – the pixel data.
Returns: false if error occurred during setting of band.
Return type: boolean
-
testBand
(productLevel, bandIndex)¶ Test if band exists in database.
Parameters: - productLevel (str) – [L1C | L2A | L3].
- bandIndex (unsigned int) – the band index.
Returns: true if band exists, else false.
Return type: boolean
-
testDb
()¶ Test consistency of database.
Param: none. Returns: false if database is inconsistent, else true. Return type: boolean
-
2.4.5. L3_Synthesis¶
-
class
sen2three.L3_Synthesis.
L3_Synthesis
(config)¶ Performs the spatio temporal algorithms.
Parameters: config (a reference to the L3_Config object.) – the config object for the current tile (via __init__). -
aotIsLower
(GPM2A)¶ Check if current Aerosol Optical Thickness is lower than any AOT of the past: :return: true if lower :rtype: bool
-
forwardProcessing
()¶ This is the default processing routine: it removes clouds and dark features from an input image. Calls the routine replaceBadPixels and updates the statistic
-
isMoreRecent
()¶ Check if current timestamp is more recent than any timestamp of the past: :return: true if more recent :rtype: bool
-
postProcessing
()¶ Performs a validation of the metadada and prepares a statistics display, if activated
-
preProcessing
()¶ Performs the pre processing, updates the pixel mask
-
process
(tables)¶ Triggers pre -, processing and post processing
-
readSolarZenithAngle
(productStr)¶ Helper class for reading the mean solar zenith angle from metadata
Parameters: productStr (str) – [L1C | L2A | L3]. Returns: solar zenith angle Return type: unsigned float
-
replaceBadPixels
(bandIndex)¶ Replaces the bad pixels by good ones, according to following algorithm:
get the bands from previous and current scene;
get the good pixel masks from function L3_Synthesis.setPixelMasks;
set the bad pixels to saturated / defective;
update bands according to selected algorithm:
- if self.config.algorithm == ‘MOST_RECENT’:
previous pixels will always be replaced with good pixels of recent scene if the time stamp of the recent tile is more actual than of any scene in the past;
- else if self.config.algorithm == ‘TEMP_HOMOGENEITY’:
previous pixels will only be replaced if sum of current good pixels is better than sum of good pixels of the best scene in the past;
- else if self.config.algorithm == ‘RADIOMETRIC_QUALITY’:
previous pixels will be replaced if either: - the average of the current AOT is lower or - the average of the current Solar Zenith Angle is higher than the equivalent parameter of the best classification map in the past;
- else if self.config.algorithm == ‘AVERAGE’:
images are an average of the current good pixels and the good pixels of all previous scenes. Mosaic map is the per pixel sum of all good pixels in the past and is used for calculating the average;
Parameters: bandIndex (unsigned int) – the band index.
-
setPixelMasks
()¶ Sets the pixel masks according to following algorithm:
initialize good and bad pixels;
create mosaic map, if not exist, else read from L3_Tables;
update mosaic map and scene classification map according to selected algorithm:
- if self.config.algorithm == ‘MOST_RECENT’:
previous classification map will always be replaced with good pixels of recent classification map if the time stamp of the recent tile is more actual than of any scene in the past;
- else if self.config.algorithm == ‘TEMP_HOMOGENEITY’:
previous classification map will only be replaced if sum of current good pixels is better than sum of good pixels of the best classification map in the past;
- else if self.config.algorithm == ‘RADIOMETRIC_QUALITY’:
previous classification map will be replaced if either: - the average of the current AOT is lower or - the average of the current Solar Zenith Angle is higher than the equivalent parameter of the best classification map in the past;
- else if self.config.algorithm == ‘AVERAGE’:
images are an average of the current good pixels and the good pixels of all previous scenes. Mosaic map is the per pixel sum of all good pixels in the past and is used for calculating the average;
update good and bad pixel masks;
-
szaIsHigher
()¶ Check if current Solar Zenith Angle is higher than any SZA of the past: :return: true if higher :rtype: bool
-
updateL3ClassificationQI
()¶ Update the L3 classification QI after each new cycle
-
2.4.6. L3_Display¶
-
class
sen2three.L3_Display.
L3_Display
(config)¶ A support class, for display of the scene classification and the mosaic map using the Python Image Library (PIL). The display of the data is a configurable option.
param config: the config object for the current tile (via __init__). type config: a reference to the L3_Config object. -
displayData
(tables)¶ Performs the display of the scene classification and the mosaic map.
Parameters: tables (a reference to the L3_Tables object.) – the table object for the current tile (via __init__).
-
2.4.7. L3_XmlParser¶
-
class
sen2three.L3_XmlParser.
L3_XmlParser
(config, productStr)¶ A parser for the assignment of xml based metadata to python configuration objects and vice versa. Performs also a validation of the metadata against the corresponding scheme.
param productStr: the product string for the given metadata (via __init__). type productStr: a string. -
getRoot
(key=None)¶ Gets the root of an xml tree, addressed by the corresponding key.
Parameters: key (a string) – the search key Returns: the tree Return type: an element tree
-
getTree
(key, subkey)¶ Gets the subtree of an xml tree, addressed by the corresponding key and subkey.
Parameters: - key (a string) – the search key
- subkey (a string) – the search subkey
Returns: the tree
Return type: an element tree
-
setTree
(key, subkey)¶ Sets the subtree of an xml tree, addressed by the corresponding key and subkey.
Parameters: - key (a string) – the search key
- subkey (a string) – the search subkey
Returns: true if succesful
Return type:
-
validate
()¶ Validator for the metadata.
Returns: true, if metadata are valid. Return type: boolean
-
2.4.8. L3_Library¶
-
sen2three.L3_Library.
showImage
(arr)¶ Debug routine for data display
Parameters: arr (a 2dim numpy array of unsigned int (16 bit)) – the image to be displayed Returns: false, if no image Return type: boolean