-
Notifications
You must be signed in to change notification settings - Fork 6
RTC‐S1 Gamma Acceptance Testing Instructions
This page contains instructions for performing Acceptance Testing for the RTC-S1 Gamma delivery from the OPERA-ADT team. These instructions pertain to the latest version of the Gamma release, currently v3.1. These instructions assume the user has access to the JPL FN-Artifactory, and has Docker installed on their local machine. Optionally, access to the opera-dev-pge AWS machine allows for significantly faster downloads from Artifactory.
The image is currently hosted on JPL FN-Artifactory, which requires JPL VPN access and JPL credentials. You may also need to be added to the gov.nasa.jpl.opera.adt organization.
Once you have access, the container tarball delivery is available under general/gov/nasa/jpl/opera/adt/rtc_s1/r3.1/gamma/dockerimg_rtc_gamma_0.3.1.tar. Sample inputs are available under general/gov/nasa/jpl/opera/adt/rtc_s1/r3/gamma/delivery_3_gamma_0.3.zip, but for the current release the expected outputs are in a separate archive under general/gov/nasa/jpl/opera/adt/rtc_s1/r3.1/gamma/expected_output_dir_3.1_gamma_0.3.1.zip.
Download all images/archives to a location on your local machine. This location will be referred to throughout this instructions as <RTC_DIR>
Note that the sample data is quite large, so the download from AF can take some time.
The first step in running the RTC-S1 image is to load it into Docker via the following command:
docker load -i <RTC_DIR>/dockerimg_rtc_gamma_0.3.1.tar
This should add the Docker image to your local repository with the name opera/rtc and the tag gamma_0.3.1.
Once the delivery_3_gamma_0.3.zip file is downloaded to your local machine, unpack it to <RTC_DIR>:
cd <RTC_DIR>; unzip delivery_3_gamma_0.3.zip
This will create a delivery_3_gamma_0.3 directory within <RTC_DIR> containing the following files/directories:
- input_dir/
|_ opera_burst_database_deploy_2022_1212.sqlite3
|_ dem.tif
|_ S1B_IW_SLC__1SDV_20180504T104507_20180504T104535_010770_013AEE_919F.zip
|_ S1B_OPER_AUX_POEORB_OPOD_20180524T110543_V20180503T225942_20180505T005942.EOF
- expected_output_dir/ # Ignore this version
- rtc_s1_gamma.yaml
- scratch_dir/
Once the expected_output_dir_3.1_gamma_0.3.1.zip file is downloaded to your local machine, unpack it to <RTC_DIR>:
cd <RTC_DIR>; unzip expected_output_dir_3.1_gamma_0.3.1.zip
This will create a expected_output_dir directory within <RTC_DIR> containing the following files/directories:
* expected_output_dir/
|_ log_docker.log
|_ t069_147169_iw3
|_ OPERA_L2_RTC-S1_069-147169-IW3_v0.3_incidence_angle.tif
|_ OPERA_L2_RTC-S1_069-147169-IW3_v0.3_layover_shadow_mask.tif
|_ OPERA_L2_RTC-S1_069-147169-IW3_v0.3_local_incidence_angle.tif
|_ OPERA_L2_RTC-S1_069-147169-IW3_v0.3_nlooks.tif
|_ OPERA_L2_RTC-S1_069-147169-IW3_v0.3_rtc_anf.tif
|_ OPERA_L2_RTC-S1_069-147169-IW3_v0.3_VH.tif
|_ OPERA_L2_RTC-S1_069-147169-IW3_v0.3_VV.tif
|_ OPERA_L2_RTC-S1_069-147169-IW3_v0.3.h5
|_ OPERA_L2_RTC-S1_069-147169-IW3_v0.3.png
|_ t069_147169_iw3.log_docker.log
|_ <additional subdirs and log files for each burst from the input SLC>
In order to execute the SAS, the input file directory, runconfig, scratch and output locations will be mounted into the container instance as Docker Volumes. To help streamline this process, we recommend making the following changes within the delivery_3_gamma_0.3 directory:
-
Create a directory named
runconfig, and copy the existing runconfig YAML file into it:mkdir -p <RTC_DIR>/delivery_3_gamma_0.3/runconfigcp <RTC_DIR>/delivery_3_gamma_0.3/rtc_s1_gamma.yaml <RTC_DIR>/delivery_3_gamma_0.3/runconfig/rtc_s1_gamma.yaml -
Modify the copied runconfig to point the
product_pathvalue underproduct_groupto "output_dir". This is required since we will invoking the SAS image using a different working directory than its default. Theproduct_pathsection of the RunConfig should be modified to match the following:
product_group:
product_path: output_dir # <- update to point at output_dir
scratch_path: scratch_dir
output_dir: output_dir
product_id: OPERA_L2_RTC-S1_{burst_id}
...
-
Move the
expected_output_dirextracted fromunzip expected_output_dir_3.1_gamma_0.3.1.zipinto thedelivery_3_gamma_0.3directory:mv <RTC_DIR>/expected_output_dir <RTC_DIR>/delivery_3_gamma_0.3/ -
Make sure the runconfig, output and scratch directories have write permissions set
chmod ga+w runconfig/rtc_s1_gamma.yaml output_dir/ scratch_dir/
We're now ready to execute the RTC-S1 Gamma. Run the following the command to kick off execution with the test assets:
docker run --rm -u $(id -u):$(id -g) \
-w /home/rtc_user \
-v <RTC_DIR>/delivery_3_gamma_0.3/runconfig:/home/rtc_user/runconfig \
-v <RTC_DIR>/delivery_3_gamma_0.3/input_dir:/home/rtc_user/input_dir:ro \
-v <RTC_DIR>/delivery_3_gamma_0.3/output_dir:/home/rtc_user/output_dir \
-v <RTC_DIR>/delivery_3_gamma_0.3/scratch_dir:/home/rtc_user/scratch_dir \
-i --tty opera/rtc:gamma_0.3.1 rtc_s1.py /home/rtc_user/runconfig/rtc_s1_gamma.yaml 2>&1 | tee <RTC_DIR>/rtc_at.log
Note that for the Gamma release, the mounted runconfig directory must not be configured with the read-only flag (:ro). This is because the SAS attempts to read the RunConfig file in r+ mode.
Running on the OPERA PGE Dev machine, execution on all bursts took roughly 115 minutes. Once execution is complete, you should see a number of directories within the <RTC_DIR>/delivery_3_gamma_0.3/output_dir/ directory, one for each burst ID processed. There should be 28 sub-directories, each containing several output products:
-
OPERA_L2_RTC-S1_<BURST-ID>_v0.3.h5: HDF5 file containing the metadata for the product(s) -
OPERA_L2_RTC-S1_<BURST-ID>_v0.3_VH.tif: GeoTIFF containing the RTC raster layer for the VH polarization channel -
OPERA_L2_RTC-S1_<BURST-ID>_v0.3_VV.tif: GeoTIFF containing the RTC raster layer for the VV polarization channel -
OPERA_L2_RTC-S1_<BURST-ID>_v0.3_incidence_angle.tif: GeoTIFF containing the "incidence angle" static layer -
OPERA_L2_RTC-S1_<BURST-ID>_v0.3_layover_shadow_mask.tif: GeoTIFF containing the "layover shadow mask" static layer -
OPERA_L2_RTC-S1_<BURST-ID>_v0.3_local_incidence_angle.tif: GeoTIFF containing the "local incidence angle" static layer -
OPERA_L2_RTC-S1_<BURST-ID>_v0.3_nlooks.tif: GeoTIFF containing the "nlooks" static layer -
OPERA_L2_RTC-S1_<BURST-ID>_v0.3_rtc_anf.tif: GeoTIFF containing the "rtc anf" static layer -
OPERA_L2_RTC-S1_<BURST-ID>_v0.3.png: PNG "browse" preview image for the dataset
Now that we've successfully executed the SAS container and generated outputs, the last step is to perform a QA check against the expected outputs.
For the current Gamma delivery, the comparison script (rtc_compare.py) must be downloaded to <RTC_DIR>, and then invoked with the following shell script to compare a single set of burst-based products at a time. rtc_compare.py expects the paths to the expected and output HD5 metadata product, from which it can find the GeoTIFF layers to compare automatically.
The following shell script can be used to automated the comparisons across all burst-based products:
#!/bin/bash
declare -a burst_ids=("t069_147169_iw3"
"t069_147170_iw1"
"t069_147170_iw2"
"t069_147170_iw3"
"t069_147171_iw1"
"t069_147171_iw2"
"t069_147171_iw3"
"t069_147172_iw1"
"t069_147172_iw2"
"t069_147172_iw3"
"t069_147173_iw1"
"t069_147173_iw2"
"t069_147173_iw3"
"t069_147174_iw1"
"t069_147174_iw2"
"t069_147174_iw3"
"t069_147175_iw1"
"t069_147175_iw2"
"t069_147175_iw3"
"t069_147176_iw1"
"t069_147176_iw2"
"t069_147176_iw3"
"t069_147177_iw1"
"t069_147177_iw2"
"t069_147177_iw3"
"t069_147178_iw1"
"t069_147178_iw2"
"t069_147178_iw3")
for burst_id in "${burst_ids[@]}"; do
echo "-------------------------------------"
echo "Comparing results for ${burst_id}"
burst_id_uppercase=${burst_id^^}
burst_id_replace_underscores=${burst_id_uppercase//_/-}
burst_id_replace_T=${burst_id_replace_underscores//T/}
burst_id_pattern="*_${burst_id_replace_T}_*.h5"
output_file=$(ls ./delivery_3_gamma_0.3/output_dir/${burst_id}/${burst_id_pattern})
expected_file=$(ls ./delivery_3_gamma_0.3/expected_output_dir/${burst_id}/${burst_id_pattern})
python3 rtc_compare.py ${expected_file} ${output_file}
done
After saving a copy of the script to <RTC_DIR> with a name such as rtc_compare_products_gamma_0.3.1.sh, and making it executable, you can compare all expected/actual products with the following command:
./rtc_compare_products_gamma_0.3.1.sh
Note that the local Python environment needs GDAL, NumPy and h5py installed in order to run rtc_compare.py.
The rtc_compare.py script performs a number of checks on both the image layers and metadata for each burst-based set of products. All tests should be marked as PASS, or WARNING for some instances of metadata where we expect a mismatch between input file paths, time stamps etc...
An example report for a single burst product is provided below.
-------------------------------------
Comparing results for t069_147169_iw3
*******************************************************
************ TESTING (HDF5 file) ************
*******************************************************
*** file 1: ./delivery_3_gamma_0.3/expected_output_dir/t069_147169_iw3/OPERA_L2_RTC-S1_069-147169-IW3_v0.3.h5
*** file 2: ./delivery_3_gamma_0.3/output_dir/t069_147169_iw3/OPERA_L2_RTC-S1_069-147169-IW3_v0.3.h5
-------------------------------------------------------
Checking the dataset.
[PASS] Dataset 1 of 51: //science/SENTINEL1/RTC/grids/frequencyA/slantRangeSpacing
[PASS] Dataset 2 of 51: //science/SENTINEL1/identification/boundingPolygon
[PASS] Dataset 3 of 51: //science/SENTINEL1/identification/absoluteOrbitNumber
[PASS] Dataset 4 of 51: //science/SENTINEL1/RTC/metadata/orbit/position
[PASS] Dataset 5 of 51: //science/SENTINEL1/identification/beamID
[PASS] Dataset 6 of 51: //science/SENTINEL1/identification/productType
[PASS] Dataset 7 of 51: //science/SENTINEL1/identification/project
[PASS] Dataset 9 of 51: //science/SENTINEL1/RTC/grids/frequencyA/xCoordinates
[PASS] Dataset 10 of 51: //science/SENTINEL1/RTC/grids/frequencyA/listOfPolarizations
[PASS] Dataset 12 of 51: //science/SENTINEL1/RTC/grids/frequencyA/rangeBandwidth
[PASS] Dataset 13 of 51: //science/SENTINEL1/RTC/grids/frequencyA/xCoordinateSpacing
[PASS] Dataset 14 of 51: //science/SENTINEL1/RTC/metadata/processingInformation/inputs/demSource
[PASS] Dataset 15 of 51: //science/SENTINEL1/identification/platform
[PASS] Dataset 16 of 51: //science/SENTINEL1/RTC/metadata/orbit/interpMethod
[PASS] Dataset 17 of 51: //science/SENTINEL1/identification/lookDirection
[PASS] Dataset 18 of 51: //science/SENTINEL1/identification/productID
[PASS] Dataset 19 of 51: //science/SENTINEL1/identification/sensor
[PASS] Dataset 20 of 51: //science/SENTINEL1/identification/productVersion
[PASS] Dataset 21 of 51: //science/SENTINEL1/identification/productLevel
[PASS] Dataset 22 of 51: //science/SENTINEL1/RTC/grids/frequencyA/projection
[PASS] Dataset 23 of 51: //science/SENTINEL1/RTC/metadata/orbit/orbitType
[PASS] Dataset 24 of 51: //science/SENTINEL1/RTC/metadata/processingInformation/algorithms/demInterpolation
[PASS] Dataset 25 of 51: //science/SENTINEL1/identification/isUrgentObservation
[PASS] Dataset 26 of 51: //science/SENTINEL1/identification/orbitPassDirection
[PASS] Dataset 28 of 51: //science/SENTINEL1/identification/trackNumber
[PASS] Dataset 29 of 51: //science/SENTINEL1/RTC/metadata/orbit/velocity
[PASS] Dataset 30 of 51: //science/SENTINEL1/identification/listOfFrequencies
[PASS] Dataset 31 of 51: //science/SENTINEL1/RTC/grids/frequencyA/yCoordinateSpacing
[PASS] Dataset 32 of 51: //science/SENTINEL1/RTC/grids/frequencyA/zeroDopplerTimeSpacing
[PASS] Dataset 34 of 51: //science/SENTINEL1/RTC/metadata/processingInformation/algorithms/geocoding
[PASS] Dataset 35 of 51: //science/SENTINEL1/RTC/metadata/processingInformation/inputs/l1SLCGranules
[PASS] Dataset 36 of 51: //science/SENTINEL1/identification/diagnosticModeFlag
[PASS] Dataset 38 of 51: //science/SENTINEL1/identification/zeroDopplerStartTime
[PASS] Dataset 39 of 51: //science/SENTINEL1/RTC/grids/frequencyA/yCoordinates
[PASS] Dataset 40 of 51: //science/SENTINEL1/RTC/grids/frequencyA/centerFrequency
[PASS] Dataset 41 of 51: //science/SENTINEL1/RTC/metadata/processingInformation/algorithms/radiometricTerrainCorrection
[PASS] Dataset 43 of 51: //science/SENTINEL1/identification/acquisitionMode
[PASS] Dataset 44 of 51: //science/SENTINEL1/identification/isGeocoded
[PASS] Dataset 45 of 51: //science/SENTINEL1/RTC/grids/frequencyA/radiometricTerrainCorrectionFlag
[PASS] Dataset 46 of 51: //science/SENTINEL1/RTC/metadata/orbit/time
[PASS] Dataset 47 of 51: //science/SENTINEL1/identification/softwareVersion
[PASS] Dataset 48 of 51: //science/SENTINEL1/identification/zeroDopplerEndTime
[PASS] Dataset 49 of 51: //science/SENTINEL1/RTC/metadata/orbit/referenceEpoch
[PASS] Dataset 50 of 51: //science/SENTINEL1/identification/burstID
[PASS] Dataset 51 of 51: //science/SENTINEL1/identification/processingType
Checking the attributes.
HDF5 test summary:
[PASS] Same dataset structure confirmed.
[PASS] Same attributes structure confirmed.
[PASS] The datasets of the two HDF files are the same within the tolerance.
Relative tolerance = 0.0001, Absolute tolerance = 1e-05
[PASS] The attributes of the two HDF files are the same within the tolerance
Relative tolerance = 0.0001, Absolute tolerance = 1e-05
*******************************************************
************ TESTING (VH polarization) ************
*******************************************************
*** file 1: ./delivery_3_gamma_0.3/expected_output_dir/t069_147169_iw3/OPERA_L2_RTC-S1_069-147169-IW3_v0.3_VH.tif
*** file 2: ./delivery_3_gamma_0.3/output_dir/t069_147169_iw3/OPERA_L2_RTC-S1_069-147169-IW3_v0.3_VH.tif
-------------------------------------------------------
[PASS] Comparing number of bands
Comparing RTC-S1 bands...
[PASS] Band 1 - RTC-S1 radar backscatter gamma0 (VH)"
[PASS] Comparing geotransform
[PASS] Comparing metadata
*******************************************************
************ TESTING (VV polarization) ************
*******************************************************
*** file 1: ./delivery_3_gamma_0.3/expected_output_dir/t069_147169_iw3/OPERA_L2_RTC-S1_069-147169-IW3_v0.3_VV.tif
*** file 2: ./delivery_3_gamma_0.3/output_dir/t069_147169_iw3/OPERA_L2_RTC-S1_069-147169-IW3_v0.3_VV.tif
-------------------------------------------------------
[PASS] Comparing number of bands
Comparing RTC-S1 bands...
[PASS] Band 1 - RTC-S1 radar backscatter gamma0 (VV)"
[PASS] Comparing geotransform
[PASS] Comparing metadata
*******************************************************
************ Overall results ************
*******************************************************
[PASS] HDF5 test
[PASS] VH polarization test
[PASS] VV polarization test
*******************************************************
For the Acceptance Test to pass, the "Overall results" section for each burst product should display PASS for all three test categories.
- When running the docker container, the local directory containing the runconfig must be mounted as docker volume without read-only permission set. This is because the SAS opens the runconfig file in
r+mode (reading and writing). Typically, the runconfig should not be modified during PGE/SAS execution. - The file names assigned to output products by the SAS do not include the upper-case "T" in the burst ID. This has been included with burst-based products in the past.
- The
--logargument to the SAS container does not support including any path components (i.e. joined with "/"), since it takes this path and appends it to the burst ID for each per-burst log. This confuses the operating system when the SAS attempts to create the log file and results in an error. This makes it impossible to specify that logs should be written to anywhere else than the current working directory, which causes complications with write permissions if thedocker runcommand tries to reassign the working directory (via-wflag). - The current User's Guide for this release does not contain any information on utilizing
rtc_compare.pyto perform the QA check. The existing version of the script still works for this release. But it might need to be updated to include comparisons of the static layers.