Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Building ISCE3 0.4.0 from source code w/ CUDA capability #8

Open
seongsujeong opened this issue Nov 2, 2021 · 3 comments
Open

Building ISCE3 0.4.0 from source code w/ CUDA capability #8

seongsujeong opened this issue Nov 2, 2021 · 3 comments

Comments

@seongsujeong
Copy link

seongsujeong commented Nov 2, 2021

Hi, @vbrancat , @rtburns-jpl , and @gmgunter ,

Following the previous posting regarding ISCE3 conda package (Issue #7 ), here I would like to report the issue that I have experienced from building the software from source code (for CUDA capability).

I am using ISCE3 source code from public GitHub in branch "develop"

I have tried building the software on two linux-based systems as below:

Docker
Environment: Docker container from the image "nvidia/cuda:latest"
kernel: 5.10.47-linuxkit
gcc version 9.3.0
nvcc version 11.0.194
Host OS: MacOS ver 11.6 (Big Sur)
Docker version: 20.10.8, build 3967b7d

AWS EC2
kernel: 5.4.0-1045-aws
Instance type: p3.2xlarge
nvcc version: 11.5.50
gcc version 9.3.0
AMI: Ubuntu 20.04

What I did and saw:

  • I installed the dependency using conda as in the instruction page below:
    https://isce-framework.github.io/isce3/install_linux.html



  • cmake error message has helped me to identify additional packages not listed on the instruction page.
    Few examples are eigen, pyre, gmock, etc.

  • I had to use version-down few packages to make the building attempt successful.
    Few of them are hdf5 (to 1.10.6), pybind11 (to 2.6.2), and pyre (to 1.9.9).

  • ctest: Bunch of tests were failed as below:

The following tests FAILED:
	126 - test.python.pybind..config (Failed)
	127 - test.python.pybind.core.attitude (Failed)
	128 - test.python.pybind.core.basis (Failed)
	129 - test.python.pybind.core.constants (Failed)
	130 - test.python.pybind.core.datetime_ (Failed)
	131 - test.python.pybind.core.ellipsoid (Failed)
	132 - test.python.pybind.core.euler (Failed)
	133 - test.python.pybind.core.interp1d (Failed)
	134 - test.python.pybind.core.kernels (Failed)
	135 - test.python.pybind.core.linspace (Failed)
	136 - test.python.pybind.core.LUT1d (Failed)
	137 - test.python.pybind.core.LUT2d (Failed)
	138 - test.python.pybind.core.orbit (Failed)
	139 - test.python.pybind.core.projections (Failed)
	140 - test.python.pybind.core.quaternion (Failed)
	141 - test.python.pybind.core.statevector (Failed)
	142 - test.python.pybind.core.timedelta (Failed)
	143 - test.python.pybind.focus.backproject (Failed)
	144 - test.python.pybind.focus.chirp (Failed)
	145 - test.python.pybind.focus.presum (Failed)
	146 - test.python.pybind.focus.rangecomp (Failed)
	147 - test.python.pybind.geocode.geocodeSlc (Failed)
	148 - test.python.pybind.geocode.geocodeCov (Failed)
	149 - test.python.pybind.geocode.radar_grid_cube (Failed)
	150 - test.python.pybind.geometry.bbox (Failed)
	151 - test.python.pybind.geometry.dem (Failed)
	152 - test.python.pybind.geometry.geo2rdr (Failed)
	153 - test.python.pybind.geometry.rdr2geo (Failed)
	154 - test.python.pybind.geometry.rtc (Failed)
	155 - test.python.pybind.geometry.metadata_cubes (Failed)
	156 - test.python.pybind.image.resamp (Failed)
	157 - test.python.pybind.io.gdal.dataset (Failed)
	158 - test.python.pybind.io.gdal.raster (Failed)
	159 - test.python.pybind.io.raster (Failed)
	160 - test.python.pybind.polsar.symmetrize (Failed)
	161 - test.python.pybind.signal.convolve2D (Failed)
	162 - test.python.pybind.signal.covariance (Failed)
	163 - test.python.pybind.signal.crossmul (Failed)
	164 - test.python.pybind.signal.crossmultiply (Failed)
	165 - test.python.pybind.signal.filter2D (Failed)
	166 - test.python.pybind.product.generic_product (Failed)
	167 - test.python.pybind.product.radargridparameters (Failed)
	168 - test.python.pybind.product.swath (Failed)
	169 - test.python.pybind.unwrap.icu (Failed)
	170 - test.python.pybind.unwrap.phass (Failed)
	171 - test.python.pybind.geometry.ltpcoordinates (Failed)
	172 - test.python.pybind.geometry.pntintersect (Failed)
	173 - test.python.pybind.antenna.frame (Failed)
	174 - test.python.pybind.antenna.geometryfunc (Failed)
	175 - test.python.pybind.cuda.core.compute_capability (Failed)
	176 - test.python.pybind.cuda.core.device (Failed)
	177 - test.python.pybind.cuda.focus.backproject (Failed)
	178 - test.python.pybind.cuda.geocode.geocode (Failed)
	179 - test.python.pybind.cuda.geometry.geo2rdr (Failed)
	180 - test.python.pybind.cuda.geometry.rdr2geo (Failed)
	181 - test.python.pybind.cuda.image.resamp (Failed)
	182 - test.python.pybind.cuda.signal.crossmul (Failed)
	183 - test.python.pkg.pybind_nisar.products.readers.attitude (Failed)
	184 - test.python.pkg.pybind_nisar.products.readers.orbit (Failed)
	185 - test.python.pkg.pybind_nisar.products.readers.raw (Failed)
	186 - test.python.pkg.pybind_nisar.workflows.crossmul (Failed)
	187 - test.python.pkg.pybind_nisar.workflows.focus (Failed)
	188 - test.python.pkg.pybind_nisar.workflows.stage_dem (Failed)
	189 - test.python.pkg.pybind_nisar.workflows.gcov (Failed)
	190 - test.python.pkg.pybind_nisar.workflows.geo2rdr (Failed)
	191 - test.python.pkg.pybind_nisar.workflows.gslc (Failed)
	192 - test.python.pkg.pybind_nisar.workflows.gpu_check (Failed)
	193 - test.python.pkg.pybind_nisar.workflows.geocode_insar (Failed)
	194 - test.python.pkg.pybind_nisar.workflows.insar (Failed)
	196 - test.python.pkg.pybind_nisar.workflows.point_target_info (Failed)
	197 - test.python.pkg.pybind_nisar.workflows.rdr2geo (Failed)
	198 - test.python.pkg.pybind_nisar.workflows.resample_slc (Failed)
	199 - test.python.pkg.pybind_nisar.workflows.rubbersheet (Failed)
	200 - test.python.pkg.pybind_nisar.workflows.unwrap (Failed)
	201 - test.python.pkg.pybind_nisar.workflows.antenna_parser (Failed)
	202 - test.python.pkg.pybind_nisar.workflows.dense_offsets (Failed)
	203 - test.python.pkg.pybind_nisar.workflows.cuda_insar (Failed)

But interestingly manually importing some of the failed modules are actually working on python. For example, the code below was working;

from isce3.io import Raster as raster # failed ctest no. 159

Also I was able to process some test data using CUDA.

So Two things I suspect:

  • Building ISCE3 0.4.0 from source code is sensitive to the version of certain dependent packages
  • Some changes in the object names have prevented ctest to run smoothly, even though the build was successful. For example, I noticed that the tutorial package imports raster from isce3.io, but it actually seems like Raster in this version of ISCE3.

Thanks again for the opportunity to try out this software.
Best,

@gmgunter
Copy link
Member

gmgunter commented Nov 3, 2021

Hi @seongsujeong.

Sorry, the public installation instructions for isce3 have gotten somewhat outdated and are a bit misleading.

In particular, I suspect the reason all of the python unit tests are failing is because the instructions tell you to run ctest before installing the software with make install. In fact, the software must be installed before running the test suite. Otherwise, all of the python unit tests fail due to ImportErrors.

Let me look into updating the public docs. That should hopefully address the other issues like unlisted dependencies as well. @rtburns-jpl we should consider adding this as an additional step in our release procedure.

@rtburns-jpl
Copy link
Member

Thanks for the report, I'm looking into this and think we can resolve most of these issues.

One part that sticks out to me is that you needed to downgrade pybind11 to 2.6.2 - can you elaborate on the issue you encountered? I wasn't aware of any incompatibility with newer versions, so if there is one, I want to get that sorted out quickly.

@seongsujeong
Copy link
Author

Hi @gmgunter, Thanks for the feedback regarding the issue. Running ctest after make install did make them pass except for stage_dem, which I am considering as "okay" as mentioned in issue #5.

My apologies for the false alarm of the pybind11 version. I confirmed the installation goes okay with the recent version of pybind11 (i.e. 2.8.1). I think I was confused after a few times of trials and errors to troubleshoot the procedure.

LiangJYu pushed a commit to LiangJYu/isce3 that referenced this issue Apr 16, 2022
gmgunter pushed a commit that referenced this issue Oct 2, 2023
* add GUNW

* refactor and reformat

* change the h5dataset

* formate with black and refactor

* add GOFF

* fix typos

* condense the code

* add RUNW

* add ROFF RIFG RUNW

* reformat with black

* reformat

* rename the io to insar

* test the codes and add a test.py

* address VB comments

* add the georeference to the GUNW and GOFF products

* fix some parameters of pixelOffsets

* change the geolocation_grid

* fix the iono disable error

* fix the mixed mode

* revise the pixelOffsets dimension

* address VB 2rd review

* make insar new spec package

* fix the insar package issue

* test the insar workflow

* change the RUNW az and rg looks and fix VB comments

* change H5Dataset

* add the RIFG test

* refactor

* refactor the h5_prep

* fix product bug

* correlation surface peak

* fix h5_prep.run for insar

* run isort

* fix the geocode_insar bugs

* fix the unit tests

* fix the crossmul unit test

* attempt to fix the unit tests but failed

* change the productspec version from 1.0.0 to 0.9

* change the product.h5

* remove the odd looks checking

* remove prep_ds_insar

* some formats

* reformat the unwrap

* fix errors in unit test (#6)

* fix errors in unit test

* delete old files

Co-authored-by: Jungkyo Jung <[email protected]>

* address the crossmul unit test failure

* fix the crossmul and rubbersheet unit tests

* fix the hydrostatic

* fix minors

* product specification version

* minor change

* change 0.9 to 0.9.0

* change the prep_insar to prepare_insar_hdf5

* fix LY comments

* address LY comments

* reduce file size (#7)

Co-authored-by: Jungkyo Jung <[email protected]>

* fix geocode_insar bugs

* add the scenecenter parameters

* fix typos

* uploaded compressed files (#8)

Co-authored-by: Jungkyo Jung <[email protected]>

Co-authored-by: Xiaodong Huang <[email protected]>
Co-authored-by: Jungkyo Jung <[email protected]>
Co-authored-by: Jungkyo Jung <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants