New transport transport mechanisms: Lakefs

[112358fn]

Hi, I am currently working in a project were we use both Azure and LakeFS. I would like to contribute to this project adding the LakeFS transport. That is why I wanted to get in contact with you to know if someone is already working in a similar solution.

• If yes, can I help with something?
• if no, do you think it is interesting for the project? and do you have any advice/recommendations? (for now I have been taking inspirations of the azure and s3 transports)

lakeFS uses a specific format for path URIs. You can read more here. For example: lakefs://<REPO>/<REF>/<KEY>

[mpenkov]

Nobody is working on this as far as I know.

The concept looks interesting, but their price tag is a bit high (looks like $2500/month for business users). It looks like there's some free option, but there are probably caveats (there is no such things as a free lunch, after all). For me personally this is a turn-off for two reasons:

1. How many people would actually use smart_open with lakefs?
2. As an open source project with essentially no budget, do we really want to support and maintain an extension for someone who charges $2500/month? Keep in mind that this support/maintenance is for the duration of the smart_open project.

My gut feeling is "no, don't bother with integrating this into smart_open". If your project uses it, sure, build some glue code that allows you to achieve what you need.

Perhaps @piskvorky has a different opinion?

[arielshaqed]

Hi, I'm an engineer at Treeverse who works on lakeFS. I'd like to help -- if that works for you.

I think that there are two parts here: work that the smart_open project might absorb, and whether lakeFS is too expensive to achieve enough adoption to make this work reasonable. As a contributor to another OSS project, I obviously know that cannot decide either of these for you.

How much work would this require from you? I don't know. I can say that we have a tested OpenAPI REST interface with Python bindings published to PyPI. As a data lifecycle management too, we care a great deal about ensuring that tools that work continue to work. We have a significant vested interest in compute working with data on lakeFS.

Is lakeFS too expensive to get mass adoption? I suspect not. lakeFS OSS has an Apache 2 license. There is no price tag for using lakeFS. Indeed the vast majority of our users install and use lakeFS without paying. The same is true of our "business" users: almost all of our users manage a lot of data, and I suspect you would qualify them as "business". As an engineer I would never introduce a data storage product from a startup if that product were not open source: if Treeverse goes down, I don't want to lose my data.

Treeverse is a commercial company and some of our users do give us money. Our paying users pay for our cloud hosting of the service, and for support. You're absolutely right that they have enough data to justify paying the price. To be clear: most of our users don't pay.

Please let me know if I can help.

[112358fn]

@mpenkov I have been using and testing this implementation and it works. I move the PR from draft to ready for review. I left the comment in the PR regarding testing. Let me know what you think and thanks in advance!