Handling of missing pixels in s2p

[oleg-alexandrov]

It looks to me that when S2P aligns images it does not have the concept that sometimes in the left image there is valid intensity data, but the right image lacks data at that location, which happens because, obviously, the left and the right image do not overlap everywhere.

As I see it, at the alignment stage S2P is happy enough to write 0 or some value where there is no data. Then during the stereo correlation stage, those unreal pixels are used as if they are valid. Sometimes there is a bluring or some other operation as well before correlation, such as in the TV-L1 algorithm, when valid and invalid pixel values are further blended together.

This results in noise and inaccurate heights in the output DEM in my experiments. I saw this for both the TV-L1 and MGM algorithms. Am I missing something, or is this a valid issue with the current processing? If it is valid, the solution won't be simple, as there lots of steps in both pre-processing and correlation where pixels should be handled more carefully.

[mnhrdt]

Please, can you send an example input where this happens? (or at least some screenshot of the output showing the described artifact). The outputs from mgm and tvl1 are dense at the beginning, but then they are filtered immediately and the validity mask is used everywhere. I think I do not understand the issue...

[oleg-alexandrov]

I think the problems show up before even invoking mgm and tvl1, in the pre-processing of inputs. And then the filtering cannot solve them. But I could be wrong.

Here is all data, inputs and outputs:

https://byss.arc.nasa.gov/oalexan1/bug.tgz

The DEMs I obtained, with TV-L1 and MGM:

test_bug_tvl1_old/dsm.tif

test_bug_mgm_old/dsm.tif

You can see noise, especially for the MGM one, on the left. The input images don't have that noise.

Here are some intermediate images showing the noise before running the correlation step:

./test_bug_tvl1_old/tiles_row_2612_height_1304/col_3144_width_1571/pair_1/rectified_ref.tif
./test_bug_tvl1_old/tiles_row_2612_height_1304/col_3144_width_1571/pair_1/rectified_sec.tif
./test_bug_mgm_old/tiles_row_2612_height_1304/col_3144_width_1571/pair_1/rectified_ref.tif
./test_bug_mgm_old/tiles_row_2612_height_1304/col_3144_width_1571/pair_1/rectified_sec.tif

I think the problem is filling with incorrect values.

I attached all inputs, so you can reproduce this. There are two config files, that you can use:

config_mgm.json

config_tvl1.json

In my images, where there is no valid input, I use a no-data value, as it is standard with Tif images. I think S2P does not understand it and thinks that that data is valid. Does S2P have a way of providing a mask for the input data?

My own version of S2P is 3-4 months old. If you cannot reproduce the problem, maybe it is fixed.

Thank you.

[oleg-alexandrov]

Here is another testcase which should be much easier to debug:

https://byss.arc.nasa.gov/oalexan1/bug2.tgz

There is again a .json file containing all settings.

In this example, all input pixels are valid. Yet, when alignment takes place, images are rotated. Empty areas show up where there is no valid value. They are again are filled in, and then in callTVL1.sh the valid and fake values are mixed. Here is an example from this testcase illustrating that:

test_ice_tvl1_img23_undist_bilinear3_crop5_old/tiles_row_0_height_1811/col_0_width_1474/pair_1/rectified_ref.tif

test_ice_tvl1_img23_undist_bilinear3_crop5_old/tiles_row_0_height_1811/col_0_width_1474/pair_1/rectified_sec.tif

And the final DEM is incorrect, as you see here:

test_ice_tvl1_img23_undist_bilinear3_crop5_old/dsm.tif

I was able to get good results by running callTVL1.sh on images which were pre-aligned, while avoiding S2P's alignment part.

I ran MGM on this dataset as well. One tile of the 6 failed, and there was a lot of noise too.