Skip to content

DSWx‐NI Interface Acceptance Test Instructions

Jump to bottom
Scott Collins edited this page Apr 2, 2024 · 5 revisions

This page contains instructions for performing Acceptance Testing for the DSWx-NI Interface delivery from the OPERA-ADT team. These instructions assume the user has access to the JPL FN-Artifactory, and Docker is installed on their local machine. Optionally, access to the opera-dev-pge AWS machine allows for significantly faster downloads from Artifactory.

Acquiring the DSWx-NI Interface Docker Image

The image is currently hosted on JPL FN-Artifactory, which requires JPL VPN access and JPL credentials. You may also need to be added to the gov.nasa.jpl.opera.adt organization.

Once you have access, the container tarball delivery is available under general/gov/nasa/jpl/opera/adt/dswx_ni/r1/interface/dockerimg_dswx_ni_interface_0.1.tar. Sample inputs and outputs are available under general/gov/nasa/jpl/opera/adt/dswx_ni/r1/interface/sample_data.zip. Useful background documentation is also available under general/gov/nasa/jpl/opera/adt/dswx_ni/r1/interface/documents/.

Download both images to a location on your local machine. This location will be referred to throughout this instructions as <DSWX-NI_DIR> Note that the sample data is quite large, so the download from Artifactory can take some time.

Loading the image into Docker

The first step in running the DSWx-NI image is to load it into Docker via the following command:

    docker load -i <DSWX-NI_DIR>/dockerimg_dswx_ni_interface_0.1.tar

Use the following command to verify the Docker image with the name opera/dswx-ni and the tag interface_0.1 has been added to your Docker repository:

    docker image ls

Preparing the test data

Once the sample_data.zip file is downloaded to your local machine, unpack it to <DSWX-NI_DIR>:

    cd <DSWX-NI_DIR>; unzip sample_data.zip

This will create a sample_data directory within <DSWX-NI_DIR> containing the following files/directories:

    expected_output/
    |_ OPERA_L3_DSWx-NI_T11SLS_20110226T061749Z_20240329T181033Z_LSAR_30_v0.1_B01_WTR.tif
    |_ <Additional expected output products>
    input_dir/
    |_ ancillary_data/
       |_ algorithm_parameter_ni.yaml
       |_ dem.tif
       |_ hand.tif
       |_ MGRS_collection_db_DSWx-NI_v0.1.sqlite
       |_ MGRS_tile.sqlite
       |_ reference_water.tif
       |_ worldcover.tif
    |_ RTC
       |_ ALPSRP271200660_gcov.h5
       |_ ALPSRP271200670_gcov.h5
    output_dir/
    scratch_dir/
    dswx_ni.yaml

In order to execute the SAS, the input file directory, runconfig, scratch and output locations will be mounted into the container instance as Docker Volumes. To help streamline this process, we recommend making the following changes within the sample_data directory:

  • Create a directory named runconfig, and copy the existing runconfig YAML file into it:

      `mkdir -p <DSWX-NI_DIR>/sample_data/runconfig`

  `cp <DSWX-NI_DIR>/sample_data/dswx_ni.yaml <DSWX-NI_DIR>/sample_data/runconfig/dswx_ni.yaml`

  • Modify the copied version of the dswx_ni.yaml runconfig such that all paths within are absolute with respect to the container's file system. For example, the path input_dir/RTC/ALPSRP271200660_gcov.h5 should become /home/dswx_user/input_dir/RTC/ALPSRP271200660_gcov.h5 and so on...

  • Make sure the output and scratch directories have write permissions set

      `chmod ga+w <DSWX-NI_DIR>/output_dir/ <DSWX-NI_DIR>/scratch_dir/`

Executing the DSWx-NI container on the sample datasets

We're now ready to execute the DSWx-NI container. Run the following the command to kick off execution with the test assets:

    docker run --rm -u $(id -u):$(id -g) \
      -w /home/dswx_user/scratch_dir \
      -v <DSWX-NI_DIR>/sample_data/runconfig:/home/dswx_user/runconfig:ro \
      -v <DSWX-NI_DIR>/sample_data/input_dir:/home/dswx_user/input_dir:ro \
      -v <DSWX-NI_DIR>/sample_data/output_dir:/home/dswx_user/output_dir \
      -v <DSWX-NI_DIR>/sample_data/scratch_dir:/home/dswx_user/scratch_dir \
      -i --tty opera/dswx-ni:interface_0.1 sh -ci "dswx_ni.py /home/dswx_user/runconfig/dswx_ni.yaml \
         --log /home/dswx_user/output_dir/dswx_ni_0.1.log"

Execution should complete in about 10 minutes. Once complete, you should see ~21 files in <DSWX-NI_DIR>/sample_data/output_dir. Files can be categorized into sets of four GeoTIFF files sharing the same time stamps, plus a .png "browse" image (for more information see OPERA_DSWX_NI_SAS_Design_Interface.docx in the 'documents' directory, optionally downloaded from Artifactory):

      OPERA_L3_DSWx-NI_T11SLS_20110226T061749Z_20240402T154233Z_LSAR_30_v0.1_B01_WTR.tif
      OPERA_L3_DSWx-NI_T11SLS_20110226T061749Z_20240402T154233Z_LSAR_30_v0.1_B02_BWTR.tif
      OPERA_L3_DSWx-NI_T11SLS_20110226T061749Z_20240402T154233Z_LSAR_30_v0.1_B03_CONF.tif
      OPERA_L3_DSWx-NI_T11SLS_20110226T061749Z_20240402T154233Z_LSAR_30_v0.1_B04_DIAG.tif
      OPERA_L3_DSWx-NI_T11SLS_20110226T061749Z_20240402T154233Z_LSAR_30_v0.1_BROWSE.png

With this version a log file (dswx_ni_0.1.log) is also written to the output_dir.

Running the Quality Assurance Test

There are comparison scripts available within the OPERA PGE repository. From a cloned copy of the repo, copy the following files to <DSWX-NI_DIR>:

cp <REPO DIR>/opera-sds-pge/.ci/scripts/dswx_s1/diff_dswx_files.py <DSWX-NI_DIR>
cp <REPO DIR>/opera-sds-pge/.ci/scripts/dswx_s1/dswx_comparison.py <DSWX-NI_DIR>

Where <REPO DIR> is the location containing the cloned copy of the OPERA PGE repository.

You can now perform the test by executing:

$ python3 <DSWX-NI_DIR>/diff_dswx_files.py <DSWX-NI_DIR>/sample_data/expected_output <DSWX-S1_DIR>/sample_data/output_dir

Or if you want the results written to a text file called dswx_ni_comparison.log:

$ python3 <DSWX-NI_DIR>/diff_dswx_files.py <DSWX-NI_DIR>/sample_data/expected_output <DSWX-S1_DIR>/sample_data/output_dir 2>&1 | tee dswx_ni_comparison.log

The QA output should look similar to the following, but for each pair of output products compared:

Comparing files:
    file 1: sample_data/output_dir/OPERA_L3_DSWx-NI_T11SLS_20110226T061749Z_20240329T181033Z_LSAR_30_v0.1_B01_WTR.tif
    file 2: sample_data/expected_output//OPERA_L3_DSWx-NI_T11SLS_20110226T061749Z_20240402T154233Z_LSAR_30_v0.1_B01_WTR.tif
[OK]Comparing number of bands
Comparing DSWx bands...
[OK]     Band 1 - Water classification (WTR)"
[OK]Comparing geotransform
[OK]Comparing projection
[OK]Comparing metadata
...

Known Issues with this version

  1. The Interface SAS software hardcodes the current directory ('.') as the scratch directory in the following lines:

This can cause permission issues as the default working directory (/home/dswx_user) does not have write permissions. A bug ticket has been opened on the DSWX-SAR repo here https://github.com/opera-adt/DSWX-SAR/issues/78

  1. On running the above tests on opera-pge-ci (c6i.2xlarge), the following FAILs were observed:
Comparing files:
    file 1: sample_data/expected_output/OPERA_L3_DSWx-NI_T11SLS_20110226T061749Z_20240329T181033Z_LSAR_30_v0.1_B01_WTR.tif
    file 2: sample_data/output_dir/OPERA_L3_DSWx-NI_T11SLS_20110226T061749Z_20240402T154233Z_LSAR_30_v0.1_B01_WTR.tif
[OK]Comparing number of bands
Comparing DSWx bands...
[FAIL]     Band 1 - Water classification (WTR)"
/data/home/collinss/OPERA/DSWx-NI/interface/0.1/dswx_comparison.py:59: RuntimeWarning: overflow encountered in ubyte_scalars
  if (abs(image_1[i, j] - image_2[i, j]) >=
            * input 1 has value "0" in position (x: 2615, y: 460) whereas input 2 has value "1" in the same position.

Comparing files:
    file 1: sample_data/expected_output/OPERA_L3_DSWx-NI_T11SLS_20110226T061749Z_20240329T181033Z_LSAR_30_v0.1_B02_BWTR.tif
    file 2: sample_data/output_dir/OPERA_L3_DSWx-NI_T11SLS_20110226T061749Z_20240402T154233Z_LSAR_30_v0.1_B02_BWTR.tif
[OK]Comparing number of bands
Comparing DSWx bands...
[FAIL]     Band 1 - Binary Water classification (BWTR)"
/data/home/collinss/OPERA/DSWx-NI/interface/0.1/dswx_comparison.py:59: RuntimeWarning: overflow encountered in ubyte_scalars
  if (abs(image_1[i, j] - image_2[i, j]) >=
            * input 1 has value "0" in position (x: 2615, y: 460) whereas input 2 has value "1" in the same position.

Comparing files:
    file 1: sample_data/expected_output/OPERA_L3_DSWx-NI_T11SLS_20110226T061749Z_20240329T181033Z_LSAR_30_v0.1_B03_CONF.tif
    file 2: sample_data/output_dir/OPERA_L3_DSWx-NI_T11SLS_20110226T061749Z_20240402T154233Z_LSAR_30_v0.1_B03_CONF.tif
[OK]Comparing number of bands
Comparing DSWx bands...
[FAIL]     Band 1 - Confidence values (CONF)"
            * input 1 has value "2" in position (x: 2610, y: 313) whereas input 2 has value "1" in the same position.

Comparing files:
    file 1: sample_data/expected_output/OPERA_L3_DSWx-NI_T11SLS_20110226T061749Z_20240329T181033Z_LSAR_30_v0.1_B04_DIAG.tif
    file 2: sample_data/output_dir/OPERA_L3_DSWx-NI_T11SLS_20110226T061749Z_20240402T154233Z_LSAR_30_v0.1_B04_DIAG.tif
[OK]Comparing number of bands
Comparing DSWx bands...
[FAIL]     Band 1 - Diagnostic layer (DIAG)"
/data/home/collinss/OPERA/DSWx-NI/interface/0.1/dswx_comparison.py:59: RuntimeWarning: overflow encountered in ubyte_scalars
  if (abs(image_1[i, j] - image_2[i, j]) >=
            * input 1 has value "99" in position (x: 2527, y: 0) whereas input 2 has value "100" in the same position.

Comparing files:
    file 1: sample_data/expected_output/OPERA_L3_DSWx-NI_T11SLT_20110226T061749Z_20240329T181033Z_LSAR_30_v0.1_B01_WTR.tif
    file 2: sample_data/output_dir/OPERA_L3_DSWx-NI_T11SLT_20110226T061749Z_20240402T154233Z_LSAR_30_v0.1_B01_WTR.tif
[OK]Comparing number of bands
Comparing DSWx bands...
[FAIL]     Band 1 - Water classification (WTR)"
            * input 1 has value "1" in position (x: 2260, y: 68) whereas input 2 has value "0" in the same position.

Comparing files:
    file 1: sample_data/expected_output/OPERA_L3_DSWx-NI_T11SLT_20110226T061749Z_20240329T181033Z_LSAR_30_v0.1_B02_BWTR.tif
    file 2: sample_data/output_dir/OPERA_L3_DSWx-NI_T11SLT_20110226T061749Z_20240402T154233Z_LSAR_30_v0.1_B02_BWTR.tif
[OK]Comparing number of bands
Comparing DSWx bands...
[FAIL]     Band 1 - Binary Water classification (BWTR)"
            * input 1 has value "1" in position (x: 2260, y: 68) whereas input 2 has value "0" in the same position.

Comparing files:
    file 1: sample_data/expected_output/OPERA_L3_DSWx-NI_T11SLT_20110226T061749Z_20240329T181033Z_LSAR_30_v0.1_B03_CONF.tif
    file 2: sample_data/output_dir/OPERA_L3_DSWx-NI_T11SLT_20110226T061749Z_20240402T154233Z_LSAR_30_v0.1_B03_CONF.tif
[OK]Comparing number of bands
Comparing DSWx bands...
[FAIL]     Band 1 - Confidence values (CONF)"
            * input 1 has value "6" in position (x: 3617, y: 16) whereas input 2 has value "0" in the same position.

Comparing files:
    file 1: sample_data/expected_output/OPERA_L3_DSWx-NI_T11SLT_20110226T061749Z_20240329T181033Z_LSAR_30_v0.1_B04_DIAG.tif
    file 2: sample_data/output_dir/OPERA_L3_DSWx-NI_T11SLT_20110226T061749Z_20240402T154233Z_LSAR_30_v0.1_B04_DIAG.tif
[OK]Comparing number of bands
Comparing DSWx bands...
[FAIL]     Band 1 - Diagnostic layer (DIAG)"
            * input 1 has value "54" in position (x: 2046, y: 0) whereas input 2 has value "51" in the same position.

Comparing files:
    file 1: sample_data/expected_output/OPERA_L3_DSWx-NI_T11SMS_20110226T061749Z_20240329T181033Z_LSAR_30_v0.1_B04_DIAG.tif
    file 2: sample_data/output_dir/OPERA_L3_DSWx-NI_T11SMS_20110226T061749Z_20240402T154233Z_LSAR_30_v0.1_B04_DIAG.tif
[OK]Comparing number of bands
Comparing DSWx bands...
[FAIL]     Band 1 - Diagnostic layer (DIAG)"
/data/home/collinss/OPERA/DSWx-NI/interface/0.1/dswx_comparison.py:59: RuntimeWarning: overflow encountered in ubyte_scalars
  if (abs(image_1[i, j] - image_2[i, j]) >=
            * input 1 has value "89" in position (x: 33, y: 0) whereas input 2 has value "90" in the same position.

Comparing files:
    file 1: sample_data/expected_output/OPERA_L3_DSWx-NI_T11SMT_20110226T061749Z_20240329T181033Z_LSAR_30_v0.1_B01_WTR.tif
    file 2: sample_data/output_dir/OPERA_L3_DSWx-NI_T11SMT_20110226T061749Z_20240402T154233Z_LSAR_30_v0.1_B01_WTR.tif
[OK]Comparing number of bands
Comparing DSWx bands...
[FAIL]     Band 1 - Water classification (WTR)"
            * input 1 has value "1" in position (x: 343, y: 338) whereas input 2 has value "0" in the same position.

Comparing files:
    file 1: sample_data/expected_output/OPERA_L3_DSWx-NI_T11SMT_20110226T061749Z_20240329T181033Z_LSAR_30_v0.1_B02_BWTR.tif
    file 2: sample_data/output_dir/OPERA_L3_DSWx-NI_T11SMT_20110226T061749Z_20240402T154233Z_LSAR_30_v0.1_B02_BWTR.tif
[OK]Comparing number of bands
Comparing DSWx bands...
[FAIL]     Band 1 - Binary Water classification (BWTR)"
            * input 1 has value "1" in position (x: 343, y: 338) whereas input 2 has value "0" in the same position.

Comparing files:
    file 1: sample_data/expected_output/OPERA_L3_DSWx-NI_T11SMT_20110226T061749Z_20240329T181033Z_LSAR_30_v0.1_B03_CONF.tif
    file 2: sample_data/output_dir/OPERA_L3_DSWx-NI_T11SMT_20110226T061749Z_20240402T154233Z_LSAR_30_v0.1_B03_CONF.tif
[OK]Comparing number of bands
Comparing DSWx bands...
[FAIL]     Band 1 - Confidence values (CONF)"
            * input 1 has value "6" in position (x: 285, y: 16) whereas input 2 has value "0" in the same position.

Comparing files:
    file 1: sample_data/expected_output/OPERA_L3_DSWx-NI_T11SMT_20110226T061749Z_20240329T181033Z_LSAR_30_v0.1_B04_DIAG.tif
    file 2: sample_data/output_dir/OPERA_L3_DSWx-NI_T11SMT_20110226T061749Z_20240402T154233Z_LSAR_30_v0.1_B04_DIAG.tif
[OK]Comparing number of bands
Comparing DSWx bands...
[FAIL]     Band 1 - Diagnostic layer (DIAG)"
            * input 1 has value "50" in position (x: 25, y: 0) whereas input 2 has value "47" in the same position.

Note that when running on a c6a.2xlarge instance (AMD), all comparisons pass cleanly, indicating machine precision/rounding errors when running on Intel vs AMD platforms.

A bug report with ADT has been opened to track this issue here https://github.com/opera-adt/DSWX-SAR/issues/79

Clone this wiki locally