feat(rust/drivers): adbc driver for datafusion

[tokoko]

Fixes #2263

An initial driver implementation for datafusion:

• bunch of todos everywhere
• options are not supported
• get_objects ignores filters
• get_objects takes depth into account, but only when returning results, not for info retrieval
• minimal testing

P.S. I'm not much of a rust person, so I'd appreciate any kind of pointers

[mbrobbel]

Very cool! I'll split my review over multiple chunks.

The failing test is known to be flaky.

[mbrobbel]

Instead of adding arrow we can add the arrow subcrates to reduce the size of our dependency tree.
It looks like we need to add:

• arrow-cast for https://docs.rs/arrow-cast/latest/arrow_cast/pretty/fn.print_batches.html (dev-dependency)
• arrow-select for https://docs.rs/arrow-select/latest/arrow_select/concat/fn.concat_batches.html

[tokoko]

thanks. makes sense, didn't realize most of these functions could be imported from multiple crates. Added both as a dev-dependency.

[mbrobbel]

To fix the failing CI job we need to install protoc (required by the substrait crate):

You could add

- name: Install Protoc
  uses: arduino/setup-protoc@v3
  with:
    repo-token: ${{ secrets.GITHUB_TOKEN }}

as a step after

arrow-adbc/.github/workflows/rust.yml

Lines 68 to 72 in 4639ca8

	- name: Use stable Rust
	id: rust
	run: |
	rustup toolchain install stable --no-self-update
	rustup default stable

.

[tokoko]

@mbrobbel thanks, got everything green. I had to temporarily add #![allow(refining_impl_trait)] to lib.rs, couldn't get unimplemented methods that return Result<impl RecordBatchReader + Send> to pass clippy checks otherwise.

[mbrobbel]

I think this looks good as an initial commit for this driver. The todo!()s and unwraps can be handled in follow-up PRs.

[lidavidm]

...Why does installing protoc need the repo token?

[lidavidm]

Hmm, supposedly to query Github for the latest version...

Is there some other source for protoc that doesn't rely on a third party action?

[tokoko]

• We can just do curl+unzip but it will require some logic to choose the right binary in order to work cross-platform.
• Another option is to install grpcio-tools from pypi and then alias protoc to python -m grpc_tools.protoc, but that obviously requires python to be installed.

[lidavidm]

I suppose system protoc would be too old? (And Conda too heavy?)

[tokoko]

pixi should be fine, I think. I can add pixi.toml in rust directory, we can even install rust from there instead of using rustup iirc. Not sure if just protoc justifies pixi adoption, but it would be the simplest alternative.

[tokoko]

(1) substrait crate depends on protoc
(2) I guess, but if we still need to write some logic to install the system protoc (check whether to use apt or brew dependending on the os), we might as well download the binaries instead.

[lidavidm]

Or we can use the action minus the token? (Even if it's from Arduino, I'm a bit hesitant about passing even a read-only token to an external action.)

[tokoko]

sure, that works as well.. I will drop the token then, if we see that the step becomes flaky as a result, we can always make the changes afterwards.

[tokoko]

Looks like this is a no-go 😆. judging from the action code, even if I pin an exact version, the action still tries to fetch all versions, presumably to validate.. seems like there's no way to make it work w/o a token.

[tokoko]

@lidavidm Finally got it working with pre-compiled binaries

[eitsupi]

How about these?

[lidavidm]

Let's merge this as a start. Thanks @tokoko @eitsupi!