Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add no_sign_request to s3 read test #778

Closed
wants to merge 3 commits into from

Conversation

gspowley
Copy link
Member

Set "vfs.s3.no_sign_request": True in the test_large_export_correctness pytest, which reads public s3 data.

Copy link
Member

@teo-tsirpanis teo-tsirpanis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@gspowley
Copy link
Member Author

For #777.

@@ -95,7 +95,7 @@ if (NOT HTSLIB_FOUND)
ExternalProject_Add(ep_htslib
PREFIX "externals"
URL https://github.com/TileDB-Inc/m2w64-htslib-build/releases/download/1.20-0/m2w64-htslib-1.20-0.tar.gz
URL_HASH SHA1=da39a3ee5e6b4b0d3255bfef95601890afd80709
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did the mismatch happen? We should investigate it before changing the hash.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unclear exactly how the mismatch was introduced. It was added in #746

But it's been known to be a problem for about a month now. I first saw it in #763, which I then also used in #766

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change shouldn't be needed for this PR though. That Windows CI has m2w64-htslib installed from conda. For some reason, in this PR, conda installs m2w64-htslib 1.20, which CMake can't find -- Could NOT find HTSlib (missing: HTSLIB_LIBRARIES HTSLIB_INCLUDE_DIR)

Whereas the last successful Windows CI build from last week installed m2w64-htslib 1.19 with conda, which CMake was able to find -- Found HTSlib: C:/Users/runneradmin/micromamba/envs/win-env/Library/lib/hts-3.lib

So 2 puzzles:

  1. Why is conda installing m2w64-1.20 this week but 1.19 just last week?
  2. Why can't CMake find htslib 1.20?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a workaround if we can't quickly figure this out, we could pin m2w64-htslib to 1.19

- m2w64-htslib

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The conda binary for 1.20 is smaller compared to 1.19:

image

However, I downloaded the conda binary, and it still has Library/lib/hts-3.lib, so I would think that CMake should be able to find it.

Copy link
Member Author

@gspowley gspowley Sep 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The existing hash for m2w64-htslib-1.20-0.tar.gz was wrong.

da39a3ee5e6b4b0d3255bfef95601890afd80709 is the hash from downloading the wrong URL.

$ wget -O- https://github.com/TileDB-Inc/m2w64-htslib-build/releases/download/foobar | sha1sum 
--2024-09-27 15:51:28--  https://github.com/TileDB-Inc/m2w64-htslib-build/releases/download/foobar
Resolving github.com (github.com)... 140.82.114.4
Connecting to github.com (github.com)|140.82.114.4|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-09-27 15:51:29 ERROR 404: Not Found.

da39a3ee5e6b4b0d3255bfef95601890afd80709  -

@jdblischak jdblischak linked an issue Sep 27, 2024 that may be closed by this pull request
Copy link
Member

@teo-tsirpanis teo-tsirpanis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On second thought the hash mismatch issue should not block this PR, considering that it has happened before, and the artifact is in a first-party repository.

@gspowley
Copy link
Member Author

gspowley commented Sep 27, 2024

@teo-tsirpanis any idea why the windows build is failing after the htslib hash was fixed?

@gspowley
Copy link
Member Author

Pinning m2w64-htslib=1.19 still fails. For reference, comparing a previous passing windows build to the failing build.

Passing

https://github.com/TileDB-Inc/TileDB-VCF/actions/runs/10905538566/job/30264621123

MSBuild version 17.11.2+c078802d4 for .NET Framework

  MSBuild version 17.11.2+c078802d4 for .NET Framework
  
    TILEDB_VCF_OBJECTS.vcxproj -> D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build\libtiledbvcf\src\TILEDB_VCF_OBJECTS.dir\Release\TILEDB_VCF_OBJECTS.lib
    tiledbvcf.vcxproj -> D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build\libtiledbvcf\src\Release\tiledbvcf.dll
    tiledbvcf-bin.vcxproj -> D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build\libtiledbvcf\src\Release\tiledbvcfcli.exe
    Auto build dll exports
    tiledbvcf4test.vcxproj -> D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build\libtiledbvcf\src\Release\tiledbvcf4test.dll
    1>
    -- Install configuration: "Release"
    -- Installing: C:/Users/runneradmin/micromamba/envs/win-env/Library/bin/tiledbvcf.exe
    -- Installing: C:/Users/runneradmin/micromamba/envs/win-env/Library/lib/tiledbvcf.lib
    -- Installing: C:/Users/runneradmin/micromamba/envs/win-env/Library/bin/tiledbvcf.dll
    -- Installing: C:/Users/runneradmin/micromamba/envs/win-env/Library/include/tiledbvcf/tiledbvcf.h
    -- Installing: C:/Users/runneradmin/micromamba/envs/win-env/Library/include/tiledbvcf/tiledbvcf_enum.h
    -- Installing: C:/Users/runneradmin/micromamba/envs/win-env/Library/include/tiledbvcf/tiledbvcf_export.h
    -- Installing: C:/Users/runneradmin/micromamba/envs/win-env/Library/lib/tiledbvcf4test.lib
    -- Installing: C:/Users/runneradmin/micromamba/envs/win-env/Library/bin/tiledbvcf4test.dll
  Building Custom Rule D:/a/TileDB-VCF/TileDB-VCF/libtiledbvcf/CMakeLists.txt

(win-env) D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build>if 0 NEQ 0 exit 1 

Failing

https://github.com/TileDB-Inc/TileDB-VCF/actions/runs/11077183199/job/30781889277

MSBuild version 17.11.9+a69bbaaf5 for .NET Framework
  MSBuild version 17.11.9+a69bbaaf5 for .NET Framework
  
    TILEDB_VCF_OBJECTS.vcxproj -> D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build\libtiledbvcf\src\TILEDB_VCF_OBJECTS.dir\Release\TILEDB_VCF_OBJECTS.lib
    tiledbvcf.vcxproj -> D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build\libtiledbvcf\src\Release\tiledbvcf.dll
    tiledbvcf-bin.vcxproj -> D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build\libtiledbvcf\src\Release\tiledbvcfcli.exe
    Auto build dll exports
    tiledbvcf4test.vcxproj -> D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build\libtiledbvcf\src\Release\tiledbvcf4test.dll
    1>
    -- Install configuration: "Release"
    -- Installing: /Library/lib/hts-3.lib
    -- Installing: /Library/lib/tiledb.lib
    -- Installing: /Library/bin/tiledbvcf.exe
    -- Installing: /Library/lib/tiledbvcf.lib
    -- Installing: /Library/bin/tiledbvcf.dll
    -- Installing: /Library/include/tiledbvcf/tiledbvcf.h
    -- Installing: /Library/include/tiledbvcf/tiledbvcf_enum.h
    -- Installing: /Library/include/tiledbvcf/tiledbvcf_export.h
    -- Installing: /Library/lib/tiledbvcf4test.lib
    -- Installing: /Library/bin/tiledbvcf4test.dll
    -- Up-to-date: /Library/lib/hts-3.lib
    -- Up-to-date: /Library/lib/tiledb.lib
  Building Custom Rule D:/a/TileDB-VCF/TileDB-VCF/libtiledbvcf/CMakeLists.txt
D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build>if 0 NEQ 0 exit 1 

@jdblischak
Copy link
Collaborator

Pinning m2w64-htslib=1.19 still fails.

@gspowley thanks for trying this. The whole situation is bizarre given that we didn't change anything that should have affected the Windows build.

Note that even though it still failed, CMake was able to find htslib when 1.19 was installed, unlike when 1.20 was installed:

Found HTSlib: D:/a/TileDB-VCF/TileDB-VCF/libtiledbvcf-build/externals/install/lib/hts-3.lib

More evidence that the build environment has shifted without us realizing it. Today I synced my fork, and the latest commit that passed two weeks ago, 2271db0, failed on my fork, jdblischak@2271db0

@jdblischak
Copy link
Collaborator

@teo-tsirpanis do we still need this PR now that this was fixed upstream in TileDB-Inc/TileDB#5324?

xref: #777 (comment)

I still plan to continue troubleshoot the broken Windows build, but that is orthogonal to this update to the tests.

@teo-tsirpanis
Copy link
Member

With that Core PR the vfs.s3.no_sign_request option becomes optional (or not, I'm not 100% sure that this is always the case). Since the tests in VCF pass, it's up to @gspowley on whether to merge this PR.

@gspowley
Copy link
Member Author

I'll close this PR. It looks like we'll hit a windows build issue in the future though.

@gspowley gspowley closed this Sep 30, 2024
@gspowley gspowley mentioned this pull request Oct 14, 2024
@ihnorton
Copy link
Member

For future reference, the underlying cause of the failure here is that cmake can't find our externally-provided htslib.

I don't see anything obvious in the cmake 3.30 release notes, but my first guess is that this is due to a change in the cmake version. From the builds in this comment:

  • passing
2024-09-17T14:51:20.8722101Z   cmake                       3.29.6              h400e5d1_0             conda-forge
  • failing
2024-09-27T20:20:34.5669799Z   + cmake                                      3.30.4  h400e5d1_0            conda-forge      14MB

([sc-46350] for follow-up time-permitting)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

The nightly build job failed on Friday (2024-09-27)
4 participants