SEBASTIEN SAUNIER (TELESPAZIO-FRANCE) DAVIDE GIUDICI (ARESYS) WILL MCCARTY (NASA) - VH-RODA 2021 WORKSHOP- 22ND APRIL 2021
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Sebastien Saunier (Telespazio-France)
Davide Giudici (ARESYS)
Will McCarty (NASA)
VH-RODA 2021 workshop- 22nd April 2021
European Space Agency / National Aeronautics and Space Administration
Introduction
The ESA Earthnet Data Assessment Pilot (EDAP) is a project that is responsible for
assessing the quality and suitability of candidate missions being considered for
Earthnet Third Party Missions.
ONE MAJOR OUTCOME OF THE MISSION QUALITY ASSESSMENT PROCESS IS THE MATURITY MATRIX
The maturity matrix proposes an objective quality assessment framework
The maturity matrix is designed to ensure that decisions on candidate
missions can be made fairly and with confidence.
ORGANIZATION
NPL in charge of quality assessment Guidelines
Expert teams in charge of mission evaluation (Assessment Methodology &
Accuracy Analysis)
QUALITY ASSESSMENT OF OPTICAL, RADAR AND ATMOSPHERIC MISSIONS ARE PERFORMED
VH-RODA 2021 workshop - 22nd April 2021 | Slide 2
Scope
• Review of Missions being evaluated by ESA
• Description of Maturity Matrix (Optical / Radar)
• Synthesis of the Maturity Matrix Results (Optical)
• First Maturity Matrix results for Radar missions
• On-going Radar Mission Activities
• NASA Approach for Maturity Matrix
• NASA Data Assessment Methods and Results
• Conclusions
VH-RODA 2021 workshop - 22nd April 2021 | Slide 3
Summary of the missions addressed
VHR, HR, MR Optical LR Optical SAR Atmospheric
Planet Dove OceanSat-2 ICEYE-X2 Spire
Planet Dove-R GOMX-4 (Hyperscout-1) SAOCOM-1A Tansat
Proba-1 OceanSat 3A PAZ (SEOSAR) GHGSat-D (Claire)
Skysat MOS Capella GCOM-C
Landsat 1-7 FSSCat (Hyperscout-2) ICEYE-X4, X6, X7 GOSAT
Landsat 8 SAOCOM-1B GOSAT-2
Blacksky GHGSat-C1 (Iris)
Superview
Vision-1
Superdove
HySIS
Maxar HD (15 cm)
Kompsat-3 / 3A
Vivid-X2
Today:
NEMO-HD
Jilin-1 SP03, GF02A, GXA, KF01,
Missions assessed (TN available on the website)
GF03A, GF03B, GP01 [TBC]
Missions under assessment
GaoFen-2, GaoFen-7 [TBC]
GaoFen-4 [TBC]
Missions to be assessed (before november 2021)
VH-RODA 2021 workshop - 22nd April 2021 | Slide 4
THE MATURITY MATRIX
VH-RODA 2021 workshop - 22nd April 2021 | Slide 5
A synthesis of Maturity Matrix Results
(Optical Missions)
• The content of next slides is a graphical representation of results get across all optical missions. All EDAP
assessments, shown previously, are covered, these are:
• Product Information
• Product Generation
• Ancillary Information
• Uncertainty Characterization
• EDAP Validation
This presentation does not give detail on the grading definition, so please refer to EDAP.REP.002 Optical Mission
Quality Assessment Guidelines, 1.0, 16 October 2019
https://earth.esa.int/documents/700255/3814580/EDAP.REP.002_1.0+-
+Optical+Mission+Quality+Assessment+Guidelines.pdf/b674a420-d160-48a3-a966-5637324d9904
VH-RODA 2021 workshop - 22nd April 2021 | Slide 6
Product Information
Product Details
Metrological Traceability
Availability & Accessibility
Documentation
User Product
Documentation Format
DOVE DOVE-R SuperDove SkySat BlackSky Vision-1
EDAP Grading
Not Not VH-RODA 2021 workshop - 22nd April 2021 | Slide 7
Basic Intermediate Good Excellent
Assessed Assessable
0 1 2 3 4 5
EDAP Grading
Not Not VH-RODA 2021 workshop - 22nd April 2021 | Slide 8
Basic Intermediate Good Excellent
Assessed Assessable
0 1 2 3 4 5
Ancillary Information
Product Flags Ancillary Data
DOVE DOVE-R SuperDove SkySat BlackSky Vision-1
EDAP Grading
Not Not VH-RODA 2021 workshop - 22nd April 2021 | Slide 9
Basic Intermediate Good Excellent
Assessed Assessable
0 1 2 3 4 5
EDAP Grading
Not Not VH-RODA 2021 workshop - 22nd April 2021 | Slide 10
Basic Intermediate Good Excellent
Assessed Assessable
0 1 2 3 4 5EDAP Grading
Not Not VH-RODA 2021 workshop - 22nd April 2021 | Slide 11
Basic Intermediate Good Excellent
Assessed Assessable
0 1 2 3 4 5Maturity Matrix –Trends & Perspectives
• The product information assessment results are generally “good”, for
some missions, the product format and content should be improved
(json), including a better traceability. Moreover, the algorithms are rarely
documented, and the metrological traceability documentation does not
exist.
• The product generation assessment shows that the pre flight calibration
& characterization is almost never addressed, situation is better for post
launch calibration. Key calibration parameters / processing are not
sufficiently discussed as geometry or Level 2 processing (when relevant).
• The ancillary information is shared; pixel based information (mask) is
systematically embedded in the user product, instrument parameters as
RSRs are given, there is a room of improvement with the provision of
VH-RODA 2021 workshop - 22 April 2021 | Slide 12
nd
quantitative per pixel information.Maturity Matrix – Noticeable Trends
• The Uncertainty Characterization assessment results contrasts sharply
depending on the data provider. The uncertainty sources are generally
not sufficiently discussed.
• The EDAP Validation assessment demonstrates that the quality of product
is generally in agreement with the accuracy specifications claimed by the
data provider. This statement remains applicable for a small EDAP test
data set.
• In some cases, the product design is steered by applications (military):
synthetic / uncalibrated data measurements are delivered.
• In general, beside main objectives, EDAP bring support in product
development to a larger extent. VH-RODA 2021 workshop - 22 April 2021 | Slide 13
ndSAR Missions Analysis Status
ICEYE
FULL ANALYSIS ON X2
WORK IN PROGRESS ON X4-X5, X6-X7
CAPELLA C-2 (FORMERLY KNOWN AS SEQUOIA)
PRELIMINARY SAMPLE DATASETS AVAILABLE
MILESTONE
MILESTONE
MILESTONE
Requirement
Requirement
Submit Data
Place Order
Define Data
Download
Technical
with Data
Approval
Provider
Analysis
Data
Note
ESA
Mission
1
2
3
s
s
ICEYE X2
Extended activity
Capella
ICEYE X4-X5
ICEYE X6-X7
VH-RODA 2021 workshop - 22nd April 2021 | Slide 14Quality assessment of ICEYE
ICEYE:
The world’s first small SAR satellite (< 100 kg) constellation
Customized for the high revisit rate and frequent delivery of high-resolution imagery
Evaluation of ICEYE performed primarily by FMI
The assessment is divided into two main parts:
Documentation review
Assessment of the test datasets (validation)
Data assessed:
Single Look Complex (SLC) images
Stripmap mode for X2
Stripmap and Spotlight modes for X4-X7
Software used in data analysis:
SAR Quality Toolbox developed by Aresys
SNAP (Sentinel Application Platform)
Test sites:
Low backscatter areas; Doldrums, desert
Corner reflector sites; Rosamond, Sodankylä
Homogeneous areas; rainforest, glacier
Urban environment; Helsinki X4 Spotlight High, Sodankylä corner
reflector site
VH-RODA 2021 workshop - 22nd April 2021 | Slide 15Data analysis and evaluation
X4 Stripmap: ENL in Amazon rainforest
X4 Stripmap: IRF-analysis for assessing spatial VH-RODA 2021 workshop - 22nd April 2021 | Slide 16
resolution, localization error and side lobes X2 Stripmap: Range profile for deriving NESZMaturity Matrix for ICEYE X2
• Assessment performed during summer 2020
• Report is available online (released in autumn 2020):
https://earth.esa.int/eogateway/activities/edap/sar-
missions
• The EDAP Grading is mostly ”good”
or ”intermediate”
• ICEYE’s documentation was found to be
overall in a good level.
• FMI’s validation results agree with
ICEYE’s specification
• Assessment of X4-X7 data is ongoing
VH-RODA 2021 workshop - 22nd April 2021 | Slide 17Capella C-2 data
Stripmap dataset
Imperial County, California, U.S.A.
Acquisition date: 19-Jan-2021
Product type: SLC + GEO
Resolution: 1 x 1 m
VH-RODA 2021 workshop - 22nd April 2021 | Slide 18Spotlight dataset
Kern County, California, U.S.A.
Acquisition date: 05-Jan-
2021Product type: GEO
Resolution : 50 x 50 cm
VH-RODA 2021 workshop - 22nd April 2021 | Slide 19NASA Efforts to Move Towards Matrix
• The Role of Commercial Data
• The goal of the CSDA is to evaluate the scientific role for commercial data in the
context of NASA’s earth science data portfolio
• The goal of the matrix is to better facilitate the transfer of common evaluation
parameters between NASA and ESA
• The evaluation matrix postdates the initial pilot assessments of Planet and Maxar
• Agreement to match sustained efforts to matrix
• Sustained efforts include radiometric and geometric calibration assessments
• Aim to incorporate matrix into future assessments for future vendors
VH-RODA 2021 workshop - 22nd April 2021 | Slide 20NASA Assessment on Commercial Data
• So far NASA’s assessments have focused on optical imagery and GNSS science
data
Planet Maxar Spire
PlanetScope Dove-Classic WorldView-1 GNSS-RO
PlanetScope Dove-R WorldView-2 GNSS-R
RapidEye WorldView-3
SkySat WorldView-4
QuickBird
GeoEye
IKONOS
VH-RODA 2021 workshop - 22nd April 2021 | Slide 21Geometric Assessment
• Assessment during the pilot showed regional variation in the accuracy and
precision of the geolocation
• Ongoing effort underway to further investgate geolocation accuracy
• Albuquerque, NM airport & Singapore sites
• Use panchromatic, high-resolution Maxar/WorldView data as reference for
Planet
R. Wolfe, NASA/GSFC
VH-RODA 2021 workshop - 22nd April 2021 | Slide 22Planet Radiometric Assessment
• Utilizing Benchmark Land Multi-Site Analysis and Intercomparison of Products
(BELMANIP) sites, assess radiometric performance of four planet bands
• Blue, Green, Red, Near-IR
• 9749 Planet scenes covering July 2019-June 2020
• ~73 scenes per site 6000
5000
TOA reflectance, x10000
4000
3000
2000
NIR
1000 Red
Green
Blue
0
2019-05-10 2019-07-09 2019-09-07 2019-11-06 2020-01-05 2020-03-05 2020-05-04 2020-07-03
BELMANIP Site 239 Timeseries of TOA Reflectance
E. Vermote, NASA/GSFC VH-RODA 2021 workshop - 22nd April 2021 | Slide 23Maxar Radiometric Assessment
• Adapted MAIAC-MODIS calibration
algorithm over deserts to Maxar
• Focus on Libya 4 Pseudo-Invariant
Calibration Site (PICS)
A Lyapustin, NASA/GSFC
• Characterized calibration trends for
WV2, WV3, GeoEye1, QuickBird2
• Statistically significant trends
seen in multiple bands
• Current effort to cross-calibrate
Maxar w/ MODIS and DESIS
VH-RODA 2021 workshop - 22nd April 2021 | Slide 24Get the Basics to Improve the Science
• Ultimately targeting how
commercial data complements
NASA observations
• Planet Dove fleet has the potential
with a temporal consistency closer
to kilometer-scale imagery
• Evaluation showed variance
2x as large as MODIS for
West Africa dry season
• Rigorous intercalibration of
fleet can bring this variance
down
C. Tucker, NASA/GSFC
VH-RODA 2021 workshop - 22nd April 2021 | Slide 25Get the Basics to Improve the Science
(cont’d)
• Planet data has the potential to provide
very high resolution aerosol retrievals
• 4x larger radiometric uncertainty
compared to MODIS, VIIRS
• Significant improvement in spatial
resolution to complement flagship
missions
• Shows reasonable comparison to
AERONET
• Further analysis needed to map
effects of cross-fleet calibration
differences to Aerosol retrievals
J. Lee, Univ. of MD
VH-RODA 2021 workshop - 22nd April 2021 | Slide 26CSDA Moving Forward with Matrix
• Adoption of the matix is a significant step towards consistency needed as a
sustained program
• Provides a baseline set of metrics to consistently assess vendors past,
present, and future
• Matrix serves as a step towards understanding strengths and shortcomings
to scientific applications
• First order – quick-look, qualitative assessment
• Second order – Deeper details/quantitative assessment
• Ultimately, this serves as a first step towards scientific application
• Allows investigators to readily link uncertainties to their science
VH-RODA 2021 workshop - 22nd April 2021 | Slide 27NASA CSDA Program Website
https://earthdata.nasa.gov/csda/
VH-RODA 2021 workshop - 22nd April 2021 | Slide 28VH-RODA 2021 workshop - 22nd April 2021 | Slide 29
Conclusions
MAIN THINGS ALREADY SAID.
STOP.
DROP.
SHUT ‘EM DOWN OPEN UP SHOP.
WHOAAA WHOAAAAA THIS IS HOW SPOONY ROLLS.
VH-RODA 2021 workshop - 22nd April 2021 | Slide 30ESA EDAP WEBSITE
CONCLUSIONS
CONCLUSION
CHALLENGES
LESSONS LEARNED
FUTURE WORK
https://earth.esa.int/eogateway/activities/edap
VH-RODA 2021 workshop - 22nd April 2021 | Slide 31ESA EDAP CONSORTIUM
https://earth.esa.int/eogateway/activities/edap
VH-RODA 2021 workshop - 22nd April 2021 | Slide 32You can also read
