Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Q2 2024 added - Quarterly Validation report #365

Open
wants to merge 19 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
118 changes: 118 additions & 0 deletions docs/validation/quarterly-report/2023-q3/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# 2023 Q3: DEA Quarterly Validation Report

:::{contents} In this report
:local:
:backlinks: none
:::

## Executive Summary

This Quarterly report summarises validation for DEA surface reflectance products for Quarter 3 (July-September) of 2023
and presents aggregate validation results to the end of this quarter.

* During this quarter, a total of 2 measurements were taken across 1 field site, to capture 2 overpasses.
* Validation of Landsat 8 and Sentinel-2A all improved in accuracy, taking into account the data from this quarter.
* No new data were captured durine Landsat 9 or Sentinel-2B overpasses
* On an averaged band-by-band basis, Landsat 8 is validated to 3.5%, Landsat 9 is validated to 10.7%, Sentinel-2A is validated to 2.4% and Sentinel-2B is validated to 2.5%.

## Introduction

This quarterly report presents a summary of results from Q3 2023 from the Digital Earth
Calibration/Validation team. The report is presented in the following sections:

* Background — this section outlines the context around this work, with particular attention paid ton historical work leading up to this quarter.
* Summary of Validation Work — this section provides an overall view of the field site measurements undertaken.
* Comments on Individual Sites of Interest — this section focuses on any sites where some aspect of the site or measurement was atypical.
* Summary of Band-by-Band Matching — this section presents comparison data for this quarter’s results, in the context of all previous results.
* Comments on How This Quarter’s Work Has Affected Combined Validation Results — this section discusses how the average results for each sensor have changed with the introduction of new validation data this quarter. All band data for each platform is combined to show averaged validation results.

## Background

The Digital Earth branch within Geoscience Australia offers a suite of Earth observation products, based on data from
both Landsat and Sentinel platforms. The core products are Landsat 8 and 9 and Sentinel-2A and -2B surface reflectance (SR).
To deliver these products with confidence, the Calibration/Validation team perform vicarious validation
by measuring field sites with hand-held equipment or an Unstaffed Aerial Vehicle (UAV; commonly known as drone) equipment
close to the time of an overpass. This work began with Phase 1, where measurements were performed by multiple groups
across continental Australia. Full details on the results and methodology can be found in the Phase 1 report.

Data for both SR products and from field site measurements are made freely available. For SR products, you can visualise
the data in [DEA Maps](https://maps.dea.ga.gov.au/), or for a more in-depth understanding and direct access to data, please visit the [DEA Knowledge Hub Data Products section](https://knowledge.dea.ga.gov.au/data/). Field measurement data are made available through the [National Spectral Database](https://www.ga.gov.au/scientific-topics/dea/dea-data-and-products/national-spectral-database).

As more field sites are measured and as newer measurements are made over the same field sites, the overall validation of
SR products becomes more accurate. The purpose of this report is to provide an up-to-date status of validation accuracy,
based on the most recent measurements.

## Summary of Validation Work

1 site was measured, with 2 individual field site captures. The Table below summarises these captures.

:::{csv-table} Summary of field site captures
:header-rows: 1

"Site capture (Date, Field site, Overpasses)","Latitude, Longitude (WGS84)","Instrument","Comments"
"<a href='/validation/site-report/2023-07-25-MUL/'>2023-07-25-MUL: S-2A</a>","-35.12279, 148.86261","Hand-held ASD FR-4","Excellent matchup"
"<a href='/validation/site-report/2023-09-22-MUL/'>2023-09-22-MUL: L8</a>","-35.12274, 148.86260","Hand-held ASD FR-4","Excellent matchup"
:::

:::{figure} ./2023Q3_Locations.png

The Figure shows the locations of the field sites measured in this quarter.
:::

## Comments on Individual Sites of Interest

No individual sites of interest.

## Summary of Band-by-Band Matching

:::{figure} ./2023Q3-Matchup.png

The Figure shows comparison data for each platform. Black dots represent data that were collected prior to this quarter.
Coloured symbols represent data that were collected in this quarter. The diagonal line in each panel shows the
one-to-one correspondence between field and satellite data. Note that this diagonal line does NOT show the line of best
fit. It is plotted this way to highlight any trends where the data may be biased away from the line of one-to-one
correspondence. The statistics in the bottom-right corner of each panel provide details for the line of best fit
through all points up to and including this quarter’s data.
:::

The Table below lists overall validation results. These are based on the standard deviation of the scatter that we find
for each band of each sensor. This is when taking all the validation results together, up to and including this quarter’s
results. The band-by-band scatter is representative of the validation performance of each band. Rather than providing
values for each individual band, we characterise all results by looking at the mean and maximum scatter for each
platform.

:::{csv-table} Validation Results
:header-rows: 1

"Satellite platform","Mean band-by-band scatter","Maximum band-by-band scatter"
"Landsat 8","2.7%","3.5%"
"Landsat 9","10.7%","31%"
"Sentinel-2A","2.4%","2.9%"
"Sentinel-2B","2.5%","4.7%"
:::

For example, the Table shows that each Landsat 8 band is typically validated to 2-3%, with the worst performance
of a band being 3.5%. Note that there is much larger scatter for Landsat 9, indicating higher uncertainty in validation.
This is because there have been fewer field site measurements to coincide with the relatively new Landsat 9 platform.

## Effect on Cumulative Validation Results

This section discusses the effect that this quarter’s validation results have had on the total validation
results over all time.

For Landsat 8, this quarter has seen an overall improvement in validation results. There was 1 field site comparison
measurement.

For Landsat 9, this quarter has seen no change in validation results. There were no field site comparison
measurements.

For Sentinel-2A, this quarter has seen an overall improvement in validation results. There was 1 field site comparison
measurement.

For Sentinel-2B, this quarter has seen no change in validation results. There was 1 field site comparison
measurement.

## Acknowledgments

The field validation data were collected by Geoscience Australia.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
120 changes: 120 additions & 0 deletions docs/validation/quarterly-report/2023-q4/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
# 2023 Q4: DEA Quarterly Validation Report

:::{contents} In this report
:local:
:backlinks: none
:::

## Executive Summary

This Quarterly report summarises validation for DEA surface reflectance products for Quarter 4 (October-December) of 2023
and presents aggregate validation results to the end of this quarter.

* During this quarter, a total of 2 measurements were taken across 2 field sites, to capture 3 overpasses.
* Validation of Landsat 8 and 9 and Sentinel-2B all improved in accuracy, taking into account the data from this quarter.
* On an averaged band-by-band basis, Landsat 8 is validated to 2.6%, Landsat 9 is validated to 9.9%, Sentinel-2A is validated to 2.4% and Sentinel-2B is validated to 2.5%.

## Introduction

This quarterly report presents a summary of results from Q4 2023 from the Digital Earth
Calibration/Validation team. The report is presented in the following sections:

* Background &mdash; this section outlines the context around this work, with particular attention paid ton historical work leading up to this quarter.
* Summary of Validation Work &mdash; this section provides an overall view of the field site measurements undertaken.
* Comments on Individual Sites of Interest &mdash; this section focuses on any sites where some aspect of the site or measurement was atypical.
* Summary of Band-by-Band Matching &mdash; this section presents comparison data for this quarter’s results, in the context of all previous results.
* Comments on How This Quarter’s Work Has Affected Combined Validation Results &mdash; this section discusses how the average results for each sensor have changed with the introduction of new validation data this quarter. All band data for each platform is combined to show averaged validation results.

The Q4 2023 validation report includes field site measurements that were captured as part of the first transect work
across South Australia and New South Wales. No other field site measurements were conducted during this quarter.

## Background

The Digital Earth branch within Geoscience Australia offers a suite of Earth observation products, based on data from
both Landsat and Sentinel platforms. The core products are Landsat 8 and 9 and Sentinel-2A and -2B surface reflectance (SR).
To deliver these products with confidence, the Calibration/Validation team perform vicarious validation
by measuring field sites with hand-held equipment or an Unstaffed Aerial Vehicle (UAV; commonly known as drone) equipment
close to the time of an overpass. This work began with Phase 1, where measurements were performed by multiple groups
across continental Australia. Full details on the results and methodology can be found in the Phase 1 report.

Data for both SR products and from field site measurements are made freely available. For SR products, you can visualise
the data in [DEA Maps](https://maps.dea.ga.gov.au/), or for a more in-depth understanding and direct access to data, please visit the [DEA Knowledge Hub Data Products section](https://knowledge.dea.ga.gov.au/data/). Field measurement data are made available through the [National Spectral Database](https://www.ga.gov.au/scientific-topics/dea/dea-data-and-products/national-spectral-database).

As more field sites are measured and as newer measurements are made over the same field sites, the overall validation of
SR products becomes more accurate. The purpose of this report is to provide an up-to-date status of validation accuracy,
based on the most recent measurements.

## Summary of Validation Work

2 sites were measured, with 2 individual field site captures. The Table below summarises these captures.

:::{csv-table} Summary of field site captures
:header-rows: 1

"Site capture (Date, Field site, Overpasses)","Latitude, Longitude (WGS84)","Instrument","Comments"
"<a href='/validation/site-report/2023-11-26-SA1/'>2023-11-26-SA1: L8</a>","-32.17493, 140.64202","Hand-held ASD FR-4","Excellent matchup"
"<a href='/validation/site-report/2023-11-27-NSW1/'>2023-11-27-NSW1: L9, S-2B</a>","-31.81354, 142.10370","Hand-held ASD FR-4","Good matchup. Some cloud nearby."
:::

:::{figure} ./2023Q4_Locations.png

The Figure shows the locations of the field sites measured in this quarter.
:::

## Comments on Individual Sites of Interest

No sites of particular interest.

## Summary of Band-by-Band Matching

:::{figure} ./2024Q3-Matchup.png

The Figure shows comparison data for each platform. Black dots represent data that were collected prior to this quarter.
Coloured symbols represent data that were collected in this quarter. The diagonal line in each panel shows the
one-to-one correspondence between field and satellite data. Note that this diagonal line does NOT show the line of best
fit. It is plotted this way to highlight any trends where the data may be biased away from the line of one-to-one
correspondence. The statistics in the bottom-right corner of each panel provide details for the line of best fit
through all points up to and including this quarter’s data.
:::

The Table below lists overall validation results. These are based on the standard deviation of the scatter that we find
for each band of each sensor. This is when taking all the validation results together, up to and including this quarter’s
results. The band-by-band scatter is representative of the validation performance of each band. Rather than providing
values for each individual band, we characterise all results by looking at the mean and maximum scatter for each
platform.

:::{csv-table} Validation Results
:header-rows: 1

"Satellite platform","Mean band-by-band scatter","Maximum band-by-band scatter"
"Landsat 8","2.6%","3.3%"
"Landsat 9","9.9%","27%"
"Sentinel-2A","2.4%","2.9%"
"Sentinel-2B","2.5%","4.6%"
:::

For example, the Table shows that each Landsat 8 band is typically validated to 2-3%, with the worst performance
of a band being 3.3%. Note that there is much larger scatter for Landsat 9, indicating higher uncertainty in validation.
This is because there have been fewer field site measurements to coincide with the relatively new Landsat 9 platform.

## Effect on Cumulative Validation Results

This section discusses the effect that this quarter’s validation results have had on the total validation
results over all time.

For Landsat 8, this quarter has seen an overall improvement in validation results. There was 1 field site comparison
measurement.

For Landsat 9, this quarter has seen an overall improvement in validation results. There were 1 field site comparison
measurement.

For Sentinel-2A, this quarter has seen no change in validation results. There were no field site comparison
measurements during this quarter.

For Sentinel-2B, this quarter has seen an overall improvement in validation results. There was 1 field site comparison
measurement.

## Acknowledgments

The field validation data were collected by Geoscience Australia.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
100 changes: 100 additions & 0 deletions docs/validation/quarterly-report/2024-q1/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# 2024 Q1: DEA Quarterly Validation Report

:::{contents} In this report
:local:
:backlinks: none
:::

## Executive Summary

This Quarterly report summarises validation for DEA surface reflectance products for Quarter 1 (January-March) of 2024
and presents aggregate validation results to the end of this quarter.

* During this quarter, no new measurements were taken.
* Validation of Landsat 8, 9 and Sentinel-2A, -2B all remained the same.
* On an averaged band-by-band basis, Landsat 8 is validated to 2.5%, Landsat 9 is validated to 15%, Sentinel-2A is validated to 2.4% and Sentinel-2B is validated to 2.5%.

## Introduction

This quarterly report presents a summary of results from Q1 2024 from the Digital Earth
Calibration/Validation team. The report is presented in the following sections:

* Background &mdash; this section outlines the context around this work, with particular attention paid ton historical work leading up to this quarter.
* Summary of Validation Work &mdash; this section provides an overall view of the field site measurements undertaken.
* Comments on Individual Sites of Interest &mdash; this section focuses on any sites where some aspect of the site or measurement was atypical.
* Summary of Band-by-Band Matching &mdash; this section presents comparison data for this quarter’s results, in the context of all previous results.
* Comments on How This Quarter’s Work Has Affected Combined Validation Results &mdash; this section discusses how the average results for each sensor have changed with the introduction of new validation data this quarter. All band data for each platform is combined to show averaged validation results.

## Background

The Digital Earth branch within Geoscience Australia offers a suite of Earth observation products, based on data from
both Landsat and Sentinel platforms. The core products are Landsat 8 and 9 and Sentinel-2A and -2B surface reflectance (SR).
To deliver these products with confidence, the Calibration/Validation team perform vicarious validation
by measuring field sites with hand-held equipment or an Unstaffed Aerial Vehicle (UAV; commonly known as drone) equipment
close to the time of an overpass. This work began with Phase 1, where measurements were performed by multiple groups
across continental Australia. Full details on the results and methodology can be found in the Phase 1 report.

Data for both SR products and from field site measurements are made freely available. For SR products, you can visualise
the data in [DEA Maps](https://maps.dea.ga.gov.au/), or for a more in-depth understanding and direct access to data, please visit the [DEA Knowledge Hub Data Products section](https://knowledge.dea.ga.gov.au/data/). Field measurement data are made available through the [National Spectral Database](https://www.ga.gov.au/scientific-topics/dea/dea-data-and-products/national-spectral-database).

As more field sites are measured and as newer measurements are made over the same field sites, the overall validation of
SR products becomes more accurate. The purpose of this report is to provide an up-to-date status of validation accuracy,
based on the most recent measurements.

## Summary of Validation Work

No new sites were measured.

:::{csv-table} Summary of field site captures
:header-rows: 1

"Site capture (Date, Field site, Overpasses)","Latitude, Longitude (WGS84)","Instrument","Comments"
:::

## Comments on Individual Sites of Interest

No sites of particular interest.

## Summary of Band-by-Band Matching

:::{figure} ./2024Q1-Matchup.png

The Figure shows comparison data for each platform. Black dots represent data that were collected prior to this quarter.
Coloured symbols represent data that were collected in this quarter. The diagonal line in each panel shows the
one-to-one correspondence between field and satellite data. Note that this diagonal line does NOT show the line of best
fit. It is plotted this way to highlight any trends where the data may be biased away from the line of one-to-one
correspondence. The statistics in the bottom-right corner of each panel provide details for the line of best fit
through all points up to and including this quarter’s data.
:::

The Table below lists overall validation results. These are based on the standard deviation of the scatter that we find
for each band of each sensor. This is when taking all the validation results together, up to and including this quarter’s
results. The band-by-band scatter is representative of the validation performance of each band. Rather than providing
values for each individual band, we characterise all results by looking at the mean and maximum scatter for each
platform.

:::{csv-table} Validation Results
:header-rows: 1

"Satellite platform","Mean band-by-band scatter","Maximum band-by-band scatter"
"Landsat 8","2.5%","3.1%"
"Landsat 9","15%","29%"
"Sentinel-2A","2.4%","2.9%"
"Sentinel-2B","2.5%","4.5%"
:::

For example, the Table shows that each Landsat 8 band is typically validated to 2-3%, with the worst performance
of a band being 3.1%. Note that there is much larger scatter for Landsat 9, indicating higher uncertainty in validation.
This is because there have been fewer field site measurements to coincide with the relatively new Landsat 9 platform.

## Effect on Cumulative Validation Results

This section discusses the effect that this quarter’s validation results have had on the total validation
results over all time.

Since no new measurements were taken during this quarter, validation performance remains the same.

## Acknowledgments

The field validation data were collected by Geoscience Australia.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
125 changes: 125 additions & 0 deletions docs/validation/quarterly-report/2024-q2/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
# 2024 Q2: DEA Quarterly Validation Report

:::{contents} In this report
:local:
:backlinks: none
:::

## Executive Summary

This Quarterly report summarises validation for DEA surface reflectance products for Quarter 2 (April-June) of 2024
and presents aggregate validation results to the end of this quarter.

* During this quarter, a total of 3 measurements were taken across 2 field sites, to capture 3 overpasses.
* Validation of Landsat 8, Sentinel-2A and 2B all improved in accuracy, taking into account the data this quarter. There were no Landsat 9 overpasses matched during this quarter.
* On an averaged band-by-band basis, Landsat 8 is validated to 2.5%, Landsat 9 is validated to 15% (no new data), Sentinel-2A is validated to 2.4% and Sentinel-2B is validated to 2.5%.

## Introduction

This quarterly report presents a summary of results from Q2 2024 from the Digital Earth
Calibration/Validation team. The report is presented in the following sections:

* Background &mdash; this section outlines the context around this work, with particular attention paid to historical work leading up to this quarter.
* Summary of Validation Work &mdash; this section provides an overall view of the field site measurements undertaken.
* Comments on Individual Sites of Interest &mdash; this section focuses on any sites where some aspect of the site or measurement was atypical.
* Summary of Band-by-Band Matching &mdash; this section presents comparison data for this quarter’s results, in the context of all previous results.
* Comments on How This Quarter’s Work Has Affected Combined Validation Results &mdash; this section discusses how the average results for each sensor have changed with the introduction of new validation data this quarter. All band data for each platform is combined to show averaged validation results.

The Q2 2024 validation report includes field site measurements that were captured as part of the winter transect work
across South Australia. Note that only one field site measurement for SA1 is part of this report, with other sites appearing in the Q3 2024 validation report.

## Background

The Digital Earth branch within Geoscience Australia offers a suite of Earth observation products, based on data from
both Landsat and Sentinel platforms. The core products are Landsat 8 and 9 and Sentinel-2A and -2B surface reflectance (SR).
To deliver these products with confidence, the Calibration/Validation team perform vicarious validation
by measuring field sites with hand-held equipment or an Unstaffed Aerial Vehicle (UAV; commonly known as drone) equipment
close to the time of an overpass. This work began with Phase 1, where measurements were performed by multiple groups
across continental Australia. Full details on the results and methodology can be found in the Phase 1 report.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a link to the Phase 1 report?


Data for both SR products and from field site measurements are made freely available. For SR products, you can visualise
the data in [DEA Maps](https://maps.dea.ga.gov.au/), or for a more in-depth understanding and direct access to data, please visit the [DEA Knowledge Hub Data Products section](https://knowledge.dea.ga.gov.au/data/). Field measurement data are made available through the [National Spectral Database](https://www.ga.gov.au/scientific-topics/dea/dea-data-and-products/national-spectral-database).

As more field sites are measured and as newer measurements are made over the same field sites, the overall validation of
SR products becomes more accurate. The purpose of this report is to provide an up-to-date status of validation accuracy,
based on the most recent measurements.

## Summary of Validation Work

2 sites were measured, with 3 individual field site captures. The Table below summarises these captures.

:::{csv-table} Summary of field site captures
:header-rows: 1

"Site capture (Date, Field site, Overpasses)","Latitude, Longitude (WGS84)","Instrument","Comments"
"<a href='/validation/site-report/2024-04-16-MUL/'>2024-04-16 MUL: S-2B</a>","-35.12280, 148.86258","Hand-held ASD FR-4","Excellent matchup"
"<a href='/validation/site-report/2024-05-21-MUL/'>2024-05-21 MUL: S-2A</a>","-35.12389, 148.86283","Drone mounted SR-3500","Poor matchup in CA and blue bands."
"<a href='/validation/site-report/2024-06-30-SA1/'>2024-06-30 SA1: L8</a>","-31.81348, 140.64083","Drone mounted SR-3500","Nearby cloud noted."
:::

:::{figure} ./2024Q2_Locations.png

The Figure shows the locations of the field sites measured in this quarter.
:::

## Comments on Individual Sites of Interest

No sites of particular interest.

## Summary of Band-by-Band Matching

:::{figure} ./2024Q2-Matchup.png

The Figure shows comparison data for each platform. Black dots represent data that were collected prior to this quarter.
Coloured symbols represent data that were collected in this quarter. The diagonal line in each panel shows the
one-to-one correspondence between field and satellite data. Note that this diagonal line does NOT show the line of best
fit. It is plotted this way to highlight any trends where the data may be biased away from the line of one-to-one
correspondence. The statistics in the bottom-right corner of each panel provide details for the line of best fit
through all points up to and including this quarter’s data.
:::

The Table below lists overall validation results. These are based on the standard deviation of the scatter that we find
for each band of each sensor. This is when taking all the validation results together, up to and including this quarter’s
results. The band-by-band scatter is representative of the validation performance of each band. Rather than providing
values for each individual band, we characterise all results by looking at the mean and maximum scatter for each
platform.

:::{csv-table} Validation Results
:header-rows: 1

"Satellite platform","Mean band-by-band scatter","Maximum band-by-band scatter"
"Landsat 8","2.5%","3.1%"
"Landsat 9","15%","29%"
"Sentinel-2A","2.4%","2.9%"
"Sentinel-2B","2.5%","4.5%"
:::

For example, the Table shows that each Landsat 8 band is typically validated to 2-3%, with the worst performance
of a band being 3.1%. Note that there is much larger scatter for Landsat 9, indicating higher uncertainty in validation.
This is because there have been fewer field site measurements to coincide with the relatively new Landsat 9 platform.

## Effect on Cumulative Validation Results

This section discusses the effect that this quarter’s validation results have had on the total validation
results over all time.

For Landsat 8, this quarter has seen an slight improvement in validation results. There was 1 field site comparison
measurement. Overall, the field data for Landsat 8 overpasses continue to improve the
validation reliability.

For Landsat 9, this quarter has not seen any change in validation results: there were no field site comparison
measurements. The larger uncertainty of Landsat 9, when compared to Landsat 8 above, is most likely due to few
overall field site comparisons with the newer Landsat 9 OLI2 sensor.

For Sentinel-2A, this quarter has seen a slight improvement in validation results. There was 1 field site comparison
measurement at Mullion on 21 May, 2024. This measurement shows an excellent match. Overall, the field data for Sentinel-2A
overpasses continue to improve the validation reliability.

For Sentinel-2B, this quarter has seen a slight improvement in validation results. There was 1 field site comparison
measurement at Mullion on 16 April, 2024.


## Acknowledgments

The field validation data were collected by Geoscience Australia.

Binary file not shown.
62 changes: 30 additions & 32 deletions docs/validation/quarterly-report/2024-q3/index.md
Original file line number Diff line number Diff line change
@@ -7,49 +7,48 @@

## Executive Summary

This Quarterly report summarises validation for DEA surface reflectance products for quarter 3 (July-September), 2024
This Quarterly report summarises validation for DEA surface reflectance products for Quarter 3 (July-September) of 2024
and presents aggregate validation results to the end of this quarter.

* During this quarter, 6 field sites were measured 9 times and can be matched to 11 overpasses.
* During this quarter, a total of 9 measurements were taken across 6 field sites, to capture 11 overpasses.
* Validation of Landsat 8 and 9 and Sentinel-2A all improved in accuracy, taking into account the data from this quarter.
* Validation of Sentinel-2B degraded slightly, which is likely because the only site measurement matched to a Sentinel-2B overpass was over a challenging site with complex terrain and surface morphology.
* On an averaged band-by-band basis, Landsat 8 is validated to 2.4%, Landsat 9 is validated to 13.4%, Sentinel-2A is validated to 2.2% and Sentinel-2B is validated to 2.6%.

## Introduction

This quarterly report presents a summary of results from Q3 2024 (July-September) from the Digital Earth
This quarterly report presents a summary of results from Q3 2024 from the Digital Earth
Calibration/Validation team. The report is presented in the following sections:

* Background outlines the context around this work, with particular attention on historical work leading up to this quarter.
* Summary of Validation Work presents an overall picture of the field site measurements undertaken in a table and map.
* Comments on Individual Sites of Interest focuses on any sites where some aspect of the site or measurement was atypical.
* Summary of Band-by-Band Matching presents comparison data for this quarter’s results, in the context of all previous results.
* Comments on How This Quarter’s Work Has Affected Combined Validation Results discusses how the average results for each sensor have changed with the introduction of new validation data from this quarter. This section combines all band data for each platform to show averaged validation results.
* Background &mdash; this section outlines the context around this work, with particular attention paid ton historical work leading up to this quarter.
* Summary of Validation Work &mdash; this section provides an overall view of the field site measurements undertaken.
* Comments on Individual Sites of Interest &mdash; this section focuses on any sites where some aspect of the site or measurement was atypical.
* Summary of Band-by-Band Matching &mdash; this section presents comparison data for this quarter’s results, in the context of all previous results.
* Comments on How This Quarter’s Work Has Affected Combined Validation Results &mdash; this section discusses how the average results for each sensor have changed with the introduction of new validation data this quarter. All band data for each platform is combined to show averaged validation results.

The Q3, 2024 validation report includes field site measurements that were captured as part of the winter transect work
The Q3 2024 validation report includes field site measurements that were captured as part of the winter transect work
across South Australia and New South Wales. Note that one field site measurement for SA1 is part of the Q2 2024
validation report and not shown here. No other field site measurements were conducted during this quarter.

## Background

The Digital Earth branch within Geoscience Australia offers a suite of Earth observation products, based on data from
both Landsat and Sentinel platforms. The core products are Landsat 8 and 9 and Sentinel-2A and -2B surface reflectance
(SR). To deliver these products with surety, the Calibration and Validation (Cal/Val) team perform vicarious validation
by measuring field sites with hand-held or Unstaffed Aerial Vehicle (UAV, commonly known as drones)-based equipment
both Landsat and Sentinel platforms. The core products are Landsat 8 and 9 and Sentinel-2A and -2B surface reflectance (SR).
To deliver these products with confidence, the Calibration/Validation team perform vicarious validation
by measuring field sites with hand-held equipment or an Unstaffed Aerial Vehicle (UAV; commonly known as drone) equipment
close to the time of an overpass. This work began with Phase 1, where measurements were performed by multiple groups
across continental Australia. Full details on the results and methodology can be found in the Phase 1 report.

Data for both SR products and from field site measurements are made freely available. For SR products, you can visualise
the data at DEA Maps, or for a more in-depth understanding and direct access to data, please visit the DEA Data and
Products page. Field measurement data are made available through the National Spectral Database.
the data in [DEA Maps](https://maps.dea.ga.gov.au/), or for a more in-depth understanding and direct access to data, please visit the [DEA Knowledge Hub Data Products section](https://knowledge.dea.ga.gov.au/data/). Field measurement data are made available through the [National Spectral Database](https://www.ga.gov.au/scientific-topics/dea/dea-data-and-products/national-spectral-database).

As more field sites are measured and as newer measurements are made over the same field sites, the overall validation of
SR products becomes more accurate. The purpose of this report is to provide an up-to-date status of validation accuracy,
based on the most recent measurements.

## Summary of Validation Work

6 sites were measured, with 9 individual field site captures. The table below summarises these captures:
6 sites were measured, with 9 individual field site captures. The Table below summarises these captures.

:::{csv-table} Summary of field site captures
:header-rows: 1
@@ -109,64 +108,63 @@ The Figure shows comparison data for each platform. Black dots represent data th
Coloured symbols represent data that were collected in this quarter. The diagonal line in each panel shows the
one-to-one correspondence between field and satellite data. Note that this diagonal line does NOT show the line of best
fit. It is plotted this way to highlight any trends where the data may be biased away from the line of one-to-one
correspondence. Statistics, given in the bottom right-hand corner of each panel, show details for the line of best fit
correspondence. The statistics in the bottom-right corner of each panel provide details for the line of best fit
through all points up to and including this quarter’s data.
:::

The table below lists overall validation results. These are based on the standard deviation of the scatter that we find
for each band, for each sensor, when taking all the validation results together, up to, and including, this quarter’s
The Table below lists overall validation results. These are based on the standard deviation of the scatter that we find
for each band of each sensor. This is when taking all the validation results together, up to and including this quarter’s
results. The band-by-band scatter is representative of the validation performance of each band. Rather than providing
values for each individual band, we characterise all results by looking at the mean and maximum scatter for each
platform.

:::{csv-table} Validation Results
:header-rows: 1

"Satellite Platform","Mean band-by-band scatter","Maximum band-by-band scatter"
"Satellite platform","Mean band-by-band scatter","Maximum band-by-band scatter"
"Landsat 8","2.4%","3.1%"
"Landsat 9","13.4%","35.9%"
"Sentinel-2A","2.2%","2.7%"
"Sentinel-2B","2.6%","4.4%"
:::

The Table indicates that, for example, each Landsat 8 band is validated to typically 2-3%, with the worst band
performance being 3.1%. Note that there is much larger scatter (ie. uncertainty in validation) for Landsat 9. This is
because there have been fewer field site measurements to coincide with the relatively new Landsat 9 platform.

For example, the Table shows that each Landsat 8 band is typically validated to 2-3%, with the worst performance
of a band being 3.1%. Note that there is much larger scatter for Landsat 9, indicating higher uncertainty in validation.
This is because there have been fewer field site measurements to coincide with the relatively new Landsat 9 platform.

## Effect on Cumulative Validation Results

This section discusses the effect that this quarter’s validation results have made on the total all-time validation
results.
This section discusses the effect that this quarter’s validation results have had on the total validation
results over all time.

For Landsat 8, this quarter has seen an overall improvement in validation results. There were 5 field site comparison
measurements. The NIR band for NSW6 on 6 July, 2024 stands out as below the line of equality. This is the Boggabri field
site, where there is a significant slope to the site, as well as strongly changing surface reflectance within the field
site, where there is a significant slope to the site, as well as strongly changing surface reflectance (SR) within the field
site. A BRDF correction to the slope has not yet been applied to the SR-3500 data (expected to be done in Q1, 2025),
which may also contribute to the discrepancy. Overall, the field data for Landsat 8 overpasses continue to improve the
validation reliability, where the statistics indicate that, with all bands taken together, Landsat 8 data can be relied
upon to around 1% of the surface reflectance.
upon to around 1% of the SR.

For Landsat 9, this quarter has seen an overall improvement in validation results. There were 4 field site comparison
measurements. Again, the NIR band for NSW6 on 5 July, 2024 stands out as below the line of equality. This is the
Boggabri field site, where there is a significant slope to the site, as well as strongly changing surface reflectance
Boggabri field site, where there is a significant slope to the site, as well as strongly changing SR
within the field site. A BRDF correction to the slope has not yet been applied to the data, which may also contribute to
the discrepancy. Overall, the field data for Landsat 9 overpasses continue to improve the validation reliability, where
the statistics indicate that, with all bands taken together, Landsat 9 data can be relied upon to around 2% of the
surface reflectance. The larger uncertainty of Landsat 9, when compared to Landsat 8 above, is most likely due to few
SR. The larger uncertainty of Landsat 9, when compared to Landsat 8 above, is most likely due to few
overall field site comparisons with the newer Landsat 9 OLI2 sensor.

For Sentinel-2A, this quarter has seen an overall improvement in validation results. There was 1 field site comparison
measurement at NSW3 on 3 July, 2024. This measurement shows an excellent match. Overall, the field data for Sentinel-2A
overpasses continue to improve the validation reliability, where the statistics indicate that, with all bands taken
together, Sentinel-2A data are consistently within around 1% of the surface reflectance.
together, Sentinel-2A data are consistently within around 1% of the SR.

For Sentinel-2B, this quarter has seen an overall degradation in validation results. There was 1 field site comparison
measurement at NSW6 on 5 July, 2024. This measurement is the Boggabri field site, where there is a significant slope to
the site, as well as strongly changing surface reflectance within the field site. A BRDF correction to the slope has not
the site, as well as strongly changing SR within the field site. A BRDF correction to the slope has not
yet been applied to the data, which may also contribute to the discrepancy. Overall, the field data for Sentinel-2B in
this quarter has slightly degraded validation performance, where the statistics indicate that, with all bands taken
together, Sentinel-2B data can be relied upon to around 1% of the surface reflectance.
together, Sentinel-2B data can be relied upon to around 1% of the SR.


## Acknowledgments