Thematic Exploitation Platforms 1

The present ESA operations concept for the PDGS has evolved within the support of the exploitation scenarios of ENVISAT, ERS, and ESA’s Third Party Mission (TPM).  Today, the opportunities stemming from evolution of technology, shifts in user expectations and increase of data volumes and complexity, drive the emergence of a complementary operations concept, based on collaborative Exploitation Platforms, with the role of ESA as coordinator of multiple, synergistic European efforts.

An Exploitation Platform is a virtual workspace, providing a user community with access to (i) large volume of data, (ii) algorithm development and integration environment, (iii) processing software and services (e.g. toolboxes, visualization routines), (iv) computing resources, (v) collaboration tools (eg. forums, wiki, knowledge base, open publications, social networking…), (vi) general operation capabilities (e.g. user management and access control, accounting, etc…).

The platform thus provides a complete work environment for its’ users, enabling them to easily and effectively perform data-intensive research. The platform is a complementary service to the traditional data dissemination, bringing the user directly to the data instead of the data to the user. Processing software can be executed directly within the platform, avoiding moving large volumes of data and spending time on data sourcing, development of tools for basic data manipulation, etc. Moreover, the platform offers a collaboration environment, where the scientist can share his algorithm and knowhow with the community, publish results and perform collaborative development.

This presentation will focus on the Thematic Exploitation Platform (TEP) projects, which are Exploitation Platforms tailored to a particular user community and research field,  such as Forestry, Hydrology, Geohazards, Polar, Urban and Coastal, developed in a close collaboration between the ESA Science and Ground segment departments. In the presentation, the concept of the Thematic Exploitation Platform will be described and contextualized in the overall evolution of the ESA’s support to EO data exploitation.

The TEPs integrate data coming from multiple sources, such as the ENVISAT ASAR and ERS archives hosted by ESA, the Sentinel data hosted by the Copernicus Ground Segment and data from other space agencies members of the ESA TPM. Moreover, the TEP platforms will partner with external scientists networks, according to their particular theme, such as the European Plate Observing System (EPOS) project, providing the Geohazard TEP with tools to exploit EO data for solid Earth monitoring, geophysical and geological data analysis.

The Geohazards Thematic Exploitation Platform (GEP) is an ESA funded R&D activity to exploit the benefits of new techniques for large scale on demand and systematic processing of EO data in the Geohazards domain. It supports the geohazards community¹ by creating an Exploitation Platform with new models of collaboration where data providers, users and processors produce scientific and commercial exploitable results in the Cloud.

The Geohazards TEP is creating an ecosystem of partnerships for data, applications and ICT resources. With already 22 European early adopters in a validation initiated in March 2015 it defines a new partnership where users bring applications and processors that ran against the data to create value added products with a scientific and/or a commercial value. This partnership seeks a concrete balance where:

Data providers attract users that extract the value out of the data asset.

Platform provides key performance indicator (KPI) on data usage to the data providers.

Cloud providers' services provides commodities to support the ICT challenges created by the growing volume of environmental data from space.

Each project is concerned with either integrating an application or running on demand processing using an application available in the platform. GEP has already guaranteed funding to expand the user base to gradually reach a total of 60 separate users in 2017.

Within GEP, the Open Science stands on the reproducible, citable, discoverable and shareable of data, applications and results. The GEP community can make all of their applications and research outputs available in a citable, shareable and discoverable manner. The partnership with platforms like Github, Zenodo and Figshare provides enhanced object, code and document libraries, where the GEP community can upload files in a number of supported formats and items are attributed a Digital Object Identifier (DOI) a "digital identifier" used to uniquely identify a digital object.

Included activities in the developments and enhancements planned for 2016 are the integration of new accounting and credits management functions. This will pave the road for pay-as-you-go and subscription based services, creating revenue streams for the partners on the platform (technology partners, processing service partners, data provider partners and Cloud provider partners) and will ignite a huge change how the different communities interact. The Platform's capability for massive processing on multi-tenant computing resources is already addressing the challenges of monitoring the worldwide tectonic areas, and of studying the crust behaviour. Moreover, to exploit the generated results, the Platform leverages open APIs for the integration of interactive post-processing and visualization services.

In this presentation, we will describe the GEP consortium effort to support the Open Science  challenges and the contribution to the CEOS WG Disasters to support its Seismic Hazards Pilot and terrain deformation applications of its Volcano Pilot.

¹ The geohazards community objectives have been defined in the context of the International Forum on Satellite EO and Geohazards in Santorini in 2012

This paper presents the experience of ALTAMIRA INFORMATION uploading InSAR (Synthetic Aperture Radar Interferometry) applications in thematic platforms, such as TEP-QuickWin. This work has been supported by ESA (European Space Agency) in the scope of the Geohazards Thematic Exploitation Platform. Two different processing chains are presented jointly with ground motion maps obtained from the cloud computing one being DIAPASON for classical DInSAR and SPN (Stable Point Network) for PSI (Persistent Scatterer Interferometry) processing. The product obtained from DIAPASON is the interferometric phase related to ground motion (phase fringes from a SAR pair). SPN provides motion data (mean velocity and time series) on high-quality pixels from a stack of SAR images. Both processing chains have been implemented to be exploited with historical data coming from ERS-1/2 and ENVISAT satellites, and current acquisitions of SENTINEL-1 in SLC and TOPSAR modes.

The main challenge of this software is to reach an automation level to be used by non-expert users. The processing chains are able to run systematically without any action of the users, using only the minimum number of input parameters. The majority of parameters are automatically selected depending on the characteristics of the areas to be processed and the available SAR images. The user can easily select the stack of SAR images in the graphical environment of the geohazards platform and afterwards launch the processing chain, obtaining without any additional interaction the final ground motion maps.

The automatic selection of parameters is a key point, since it can significantly impact the quality of the final result by reducing the number of points and the signal-to-noise ratio of the measurements. In addition, the automatic processing of very large datasets (number of images and pixels) need to rely on adequate computing resources. For this reason, specific procedures are necessary to process algorithms in different computing nodes, in order to allow the simultaneous use of the platform by several users.

Some users from the geohazards community have been selected to test the functionalities of the platform, in order to generate different ground motion maps of areas affected by seismic, volcanology and other phenomena. The feedback coming from these users is the demonstration of the utility of the presented solution.

Finally, ALTAMIRA INFORMATION is also working on a cost model of the presented environment in order to explore its commercial exploitation. Future advanced users could be found in the geohazards community as well as in other commercial sectors.

The Sentinels Synergy Framework (www.sensyf.eu) is a Cloud-based data processing framework. The SenSyF FP7 project, started in 2012, has been pioneer in addressing Big Data issues from the Earth Observation point of view, and is a precursor of several of the technologies and methodologies that will be deployed in ESA's Thematic Exploitation Platforms and other related systems. The SenSyF system focuses on developing fully automated data management, together with access to a processing and exploitation facility and Earth Observation specific tool, such as ESA's Sentinlel Toolboxes deployed on a distributed environment. SenSyF is both a development and validation platform for data intensive applications using Earth Observation data. With SenSyF, scientific, institutional or commercial institutions developing EO-based applications and services can take advantage of distributed computational and storage resources, tailored for applications dependent on big Earth Observation data, and without resorting to deep infrastructure and technological investments.

In addition to the SenSyF Cloud Platform, leveraging the HADOOP Streaming programming model, SenSyF links to the Sentinels Scienfic Data Hub or the future ESA ngEO catalogue. For the integration phase, a Cloud Sandbox model furnishes an environment very similar to the space agencies operational environments, where the services are proof-tested against large EO dataset series. Built upon the processing infrastructure, a complete set of tools (SenSyF SDK - Service Development Kit) for data pre-processing is included, comprising the basic operations of data acquisition and dissemination, retrieval, atomic manipulation tools (re-gridding, image composition, etc.) as well as high level common processing algorithms (e.g. vegetation indexes calculation). The common tools (based on libraries such as GDAL, GRASS, Sentinel Toolboxes), are provided "ready to use" as part of the Cloud Sandbox, along with a set of tutorials and user manuals to support service developers. With those tools users can easily integrate and deploy an Earth Observation service based on Sentinel data sources and using Cloud Computing resources.

Throughout the last three years, the SenSyF team has tested the concept on seven of beta testing services, covering a wide spectrum of use cases and application areas within WO. The services considered range from the monitoring water quality and quantity, and monitoring of the agriculture irrigation process, to mapping the growing season in arctic and alpine areas. Also included as beta testers were the development of a spectro-temporal integration and land cover and land cover tools, which can eventually integrate the framework itself in the future.

During the last three years of FP7 SenSyF project, the development and integration of the services in the platform has provided an considerable amount of valuable information and insight on the advantages and disadvantages of a generic exploitation platform, as well as on the integration path and adoption barriers of such a system. With the closeout of the FP7 SenSyF project, a large effort is being devoted to assessing this issues and identifying future development paths, including the need to mitigate the technology and knowledge gaps typical of inter-communities projects.

For each service Use Case included in SenSyF (seven in total), a detailed analysis is performed on 1) the applicability and benefits to the service developers, including an assessment of the adoption of an EO-focused cloud-based platform; 2) the approach as deployed in SenSyF, including adopted standards, technologies and tools, and identification of future directions; 3) the strategy and approach to service integration and communication with service providers, as well as eventual training needs; 4) the sustainability of both the platform and services, including business and pricing models and competitive market analysis; and 5) the interaction with and role of large institutions and funding agencies, such as ESA and the EU. This assessment and lessons learned by the SenSyF consortium will provide valuable inputs for future exploitation platforms, designed to generate Value Added Services using Earth Observation data.

UNAVCO is a non-profit university-governed consortium that operates the National Science Foundation (NSF) Geodesy Advancing Geosciences and EarthScope (GAGE) facility and provides operational support to the Western North America InSAR Consortium (WInSAR).  The seamless synthetic aperture radar archive (SSARA) is a seamless distributed access system for SAR data and higher-level data products such as interferograms, time series, and other derived products.  A federated query service provides a single search interface for SAR and product metadata hosted at UNAVCO, the Alaska Satellite Facility (ASF), and the Group of Earth Observation’s  (GEO) Geohazards Supersites and Natural Laboratories (GSNL) data archives.   SSARA also provides a web service interface for accessing digital elevation models (DEMs) at OpenTopography, providing global 30 meter DEM data from SRTM and ASTER for terrain correction.  A key component of the SSARA infrastructure at UNAVCO is a user-contributed InSAR archive for interferograms, time series, and other derived data products. This archive leverages the hierarchical data format release 5 (HDF5) to create a homogeneous archive and provides a robust set of features for storing InSAR data.  The HDF5 product files can store either radar coordinate or geocoded interferograms with all relevant metadata and wrapped, unwrapped, coherence, and look vector data layers.  Digital object identifiers (DOI) for the InSAR products are available to encourage data citation and  reuse. Further development has led to the adoption of the HDF-EOS5 specification which provides standards for data and metadata storage within HDF5.  Use of HDF-EOS5 enables easier integration with GIS software packages and conversion to other data formats (i.e. NetCDF or GeoTIFF), and will enable easy exchange of InSAR products across the scientific user community.