Add no_sign_request to s3 read test

[gspowley]

Set "vfs.s3.no_sign_request": True in the test_large_export_correctness pytest, which reads public s3 data.

[teo-tsirpanis]

Thanks!

[gspowley]

For #777.

[teo-tsirpanis]

Why did the mismatch happen? We should investigate it before changing the hash.

[jdblischak]

Unclear exactly how the mismatch was introduced. It was added in #746

But it's been known to be a problem for about a month now. I first saw it in #763, which I then also used in #766

[jdblischak]

This change shouldn't be needed for this PR though. That Windows CI has m2w64-htslib installed from conda. For some reason, in this PR, conda installs m2w64-htslib 1.20, which CMake can't find -- Could NOT find HTSlib (missing: HTSLIB_LIBRARIES HTSLIB_INCLUDE_DIR)

Whereas the last successful Windows CI build from last week installed m2w64-htslib 1.19 with conda, which CMake was able to find -- Found HTSlib: C:/Users/runneradmin/micromamba/envs/win-env/Library/lib/hts-3.lib

So 2 puzzles:

1. Why is conda installing m2w64-1.20 this week but 1.19 just last week?
2. Why can't CMake find htslib 1.20?

[jdblischak]

As a workaround if we can't quickly figure this out, we could pin m2w64-htslib to 1.19

TileDB-VCF/ci/gha-win-env.yml

Line 10 in 2271db0

- m2w64-htslib

[jdblischak]

The conda binary for 1.20 is smaller compared to 1.19:

However, I downloaded the conda binary, and it still has Library/lib/hts-3.lib, so I would think that CMake should be able to find it.

[gspowley]

The existing hash for m2w64-htslib-1.20-0.tar.gz was wrong.

da39a3ee5e6b4b0d3255bfef95601890afd80709 is the hash from downloading the wrong URL.

$ wget -O- https://github.com/TileDB-Inc/m2w64-htslib-build/releases/download/foobar | sha1sum 
--2024-09-27 15:51:28--  https://github.com/TileDB-Inc/m2w64-htslib-build/releases/download/foobar
Resolving github.com (github.com)... 140.82.114.4
Connecting to github.com (github.com)|140.82.114.4|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-09-27 15:51:29 ERROR 404: Not Found.

da39a3ee5e6b4b0d3255bfef95601890afd80709  -

[teo-tsirpanis]

On second thought the hash mismatch issue should not block this PR, considering that it has happened before, and the artifact is in a first-party repository.

[gspowley]

@teo-tsirpanis any idea why the windows build is failing after the htslib hash was fixed?

[gspowley]

Pinning m2w64-htslib=1.19 still fails. For reference, comparing a previous passing windows build to the failing build.

Passing

https://github.com/TileDB-Inc/TileDB-VCF/actions/runs/10905538566/job/30264621123

MSBuild version 17.11.2+c078802d4 for .NET Framework

  MSBuild version 17.11.2+c078802d4 for .NET Framework
  
    TILEDB_VCF_OBJECTS.vcxproj -> D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build\libtiledbvcf\src\TILEDB_VCF_OBJECTS.dir\Release\TILEDB_VCF_OBJECTS.lib
    tiledbvcf.vcxproj -> D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build\libtiledbvcf\src\Release\tiledbvcf.dll
    tiledbvcf-bin.vcxproj -> D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build\libtiledbvcf\src\Release\tiledbvcfcli.exe
    Auto build dll exports
    tiledbvcf4test.vcxproj -> D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build\libtiledbvcf\src\Release\tiledbvcf4test.dll
    1>
    -- Install configuration: "Release"
    -- Installing: C:/Users/runneradmin/micromamba/envs/win-env/Library/bin/tiledbvcf.exe
    -- Installing: C:/Users/runneradmin/micromamba/envs/win-env/Library/lib/tiledbvcf.lib
    -- Installing: C:/Users/runneradmin/micromamba/envs/win-env/Library/bin/tiledbvcf.dll
    -- Installing: C:/Users/runneradmin/micromamba/envs/win-env/Library/include/tiledbvcf/tiledbvcf.h
    -- Installing: C:/Users/runneradmin/micromamba/envs/win-env/Library/include/tiledbvcf/tiledbvcf_enum.h
    -- Installing: C:/Users/runneradmin/micromamba/envs/win-env/Library/include/tiledbvcf/tiledbvcf_export.h
    -- Installing: C:/Users/runneradmin/micromamba/envs/win-env/Library/lib/tiledbvcf4test.lib
    -- Installing: C:/Users/runneradmin/micromamba/envs/win-env/Library/bin/tiledbvcf4test.dll
  Building Custom Rule D:/a/TileDB-VCF/TileDB-VCF/libtiledbvcf/CMakeLists.txt

(win-env) D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build>if 0 NEQ 0 exit 1 

Failing

https://github.com/TileDB-Inc/TileDB-VCF/actions/runs/11077183199/job/30781889277

MSBuild version 17.11.9+a69bbaaf5 for .NET Framework
  MSBuild version 17.11.9+a69bbaaf5 for .NET Framework
  
    TILEDB_VCF_OBJECTS.vcxproj -> D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build\libtiledbvcf\src\TILEDB_VCF_OBJECTS.dir\Release\TILEDB_VCF_OBJECTS.lib
    tiledbvcf.vcxproj -> D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build\libtiledbvcf\src\Release\tiledbvcf.dll
    tiledbvcf-bin.vcxproj -> D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build\libtiledbvcf\src\Release\tiledbvcfcli.exe
    Auto build dll exports
    tiledbvcf4test.vcxproj -> D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build\libtiledbvcf\src\Release\tiledbvcf4test.dll
    1>
    -- Install configuration: "Release"
    -- Installing: /Library/lib/hts-3.lib
    -- Installing: /Library/lib/tiledb.lib
    -- Installing: /Library/bin/tiledbvcf.exe
    -- Installing: /Library/lib/tiledbvcf.lib
    -- Installing: /Library/bin/tiledbvcf.dll
    -- Installing: /Library/include/tiledbvcf/tiledbvcf.h
    -- Installing: /Library/include/tiledbvcf/tiledbvcf_enum.h
    -- Installing: /Library/include/tiledbvcf/tiledbvcf_export.h
    -- Installing: /Library/lib/tiledbvcf4test.lib
    -- Installing: /Library/bin/tiledbvcf4test.dll
    -- Up-to-date: /Library/lib/hts-3.lib
    -- Up-to-date: /Library/lib/tiledb.lib
  Building Custom Rule D:/a/TileDB-VCF/TileDB-VCF/libtiledbvcf/CMakeLists.txt
D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build>if 0 NEQ 0 exit 1 

[jdblischak]

Pinning m2w64-htslib=1.19 still fails.

@gspowley thanks for trying this. The whole situation is bizarre given that we didn't change anything that should have affected the Windows build.

Note that even though it still failed, CMake was able to find htslib when 1.19 was installed, unlike when 1.20 was installed:

Found HTSlib: D:/a/TileDB-VCF/TileDB-VCF/libtiledbvcf-build/externals/install/lib/hts-3.lib

More evidence that the build environment has shifted without us realizing it. Today I synced my fork, and the latest commit that passed two weeks ago, 2271db0, failed on my fork, jdblischak@2271db0

[jdblischak]

@teo-tsirpanis do we still need this PR now that this was fixed upstream in TileDB-Inc/TileDB#5324?

xref: #777 (comment)

I still plan to continue troubleshoot the broken Windows build, but that is orthogonal to this update to the tests.

[teo-tsirpanis]

With that Core PR the vfs.s3.no_sign_request option becomes optional (or not, I'm not 100% sure that this is always the case). Since the tests in VCF pass, it's up to @gspowley on whether to merge this PR.

[gspowley]

I'll close this PR. It looks like we'll hit a windows build issue in the future though.

[ihnorton]

For future reference, the underlying cause of the failure here is that cmake can't find our externally-provided htslib.

I don't see anything obvious in the cmake 3.30 release notes, but my first guess is that this is due to a change in the cmake version. From the builds in this comment:

• passing

2024-09-17T14:51:20.8722101Z   cmake                       3.29.6              h400e5d1_0             conda-forge

• failing

2024-09-27T20:20:34.5669799Z   + cmake                                      3.30.4  h400e5d1_0            conda-forge      14MB

([sc-46350] for follow-up time-permitting)