Possibility of Migrating to Spark Connect

[MrPowers]

Newer language clients are being built with Spark Connect, see Spark Connect Rust for example.

Spark Connect should make it so language bindings are easier to maintain and more fully featured. Would you be willing to consider making sparkle work with Spark Connect?

[facundominguez]

Hello @MrPowers. We could consider merging contributions related to Spark Connect. It is unclear to me what support is needed on the sparkle side, but feel free to open issues to discuss the use cases and the technical obstacles.

[MrPowers]

@facundominguez - Spark Connect is a new architecture that decouples the client from the server. This would allow for Haskell users to run Spark computations without any Java/JVM dependencies on the client side. This page has a good overview.

Decoupling the Haskell language bindings and freeing this implementation from the JVM should provide a better experience for the Haskell community. When the server JVM/Java version upgrades, the client code shouldn't have to upgrade. That should also make the codebase more maintainable.

Looking forward to collaborate on this and thanks for making such a cool library.