Bump pyarrow version to 16.0.0

[JohnMoutafis]

Attempt to resolve the following issue:

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Reported by @jparismorgan and related to pyarrow version.
Related to this Vector Search PR

[JohnMoutafis]

@ihnorton @NikolaosPapailiou although the tests are passing, is there any edge case breakage that may be caused by this update?

[jparismorgan]

Thanks for updating this!

Though I think we may need to update to 16.0.0 instead?

• I see that it is the pyarrow 16.0 release which has [GH-39532](https://github.com/apache/arrow/issues/39532) - [Python] Compatibility with NumPy 2.0 in it
• And on the [Python] Compatibility with NumPy 2.0 issue it lists 16.0 as well:

(Unsure) Also, should it be >= rather than ==?

[JohnMoutafis]

Didn't knew that, as both versions are supposedly working with numpy v1.16.1 and above... Will update
About the >= vs ==, I prefer having pinned dependencies and update them manually if needed, to ensure build stability and environment reproducibility

[jparismorgan]

Does that mean that users of the package will need to have pyarrow==16.0.0 exactly? What would happen if they depend on another package which, for example, requires pyarrow>=16.1.0 (which seems reasonable for another package to set if they depend on a bug fix from a later release)?

Or am I misunderstanding this and it's a build-time only requirement? And at runtime they can have whatever version they want? (I didn't think so b/c when I looked at this for Vector Search I though you'd use [build-system] requires to specify build time requirements (like here)).

[JohnMoutafis]

That's a valid point and I am not sure about that.
I will read about it and update you here!

[JohnMoutafis]

@jparismorgan You were right about that, people using tiledb-cloud will need a minimum version of 16.0.0 but it will not block versions higher than that.
Nice catch!

[jparismorgan]

Thanks for the update! Will test with Vector Search once released.