Future for vips gem?

[ioquatix]

I've been maintaining the vips gem for a while, and having a self-contained gem was quite good. I'm still okay with doing that but it does require an investment of time. The gem is slow to install as it must compile vips every time. To be honest, this is kind of annoying. But it's also annoying when a system update can break the Ruby gems, so there is a bit of a trade off.

I'm wondering if we can make this better some how. I'd be happy for example to transfer the vips gem to you and just release the ruby-vips gem as vips.

On the other hand, maybe it makes sense for us to have a self-contained release - and we could work on that direction - perhaps bundling compiled files into the gem to avoid the overhead. I'd like to think there is some value in isolated gems which don't break if system dependencies aren't met/change. That being said, this was much worse (dependency/version hell) ~5 years ago when I made that fork. These days it mostly just works on all platforms.

I don't really have a strong opinion about this but I wanted to start a conversation about it to see if there was something worth investing time and energy into.

cc @jcupitt

[jcupitt]

Hi @ioquatix,

I think it's quite common now for python / JS / etc. packages with large native components to include pre-compiled binaries for the top few most common platforms. OpenCV, PIL, Sharp, NetVips etc. all do this.

Is this common in the ruby world? Or do people use separate gems (as you've done here)? I'm a bit unsure.

I'd be happy to look into automatically downloading a compatible binary somehow, and/or perhaps moving the vips gem into libvips.org.

Off the top of my head, the obvious questions to think about are:

1. Preparing a set of tested binaries for all relevant platforms is a colossal pain. Fortunately, sharp has done this already, perhaps we can leverage that somehow (pinging @lovell and @kleisauke for comment).
2. There are obvious and serious issues around security, but again I think this is mostly a solved problem for libvips. Sharp and netvips have a secure, complete and maintained set of curated binaries with checksums, all hosted on a secure server somewhere with plenty of bandwidth.
3. There's some choice about which binary you want (minimal, all loaders, even untested ones, a user install, the system install etc.) and this needs to be communicated to the gem somehow, I'm unsure of the best mechanism.
4. We'd need to be more active around updates. At the moment, ruby-vips can update independently from libvips, and this is a really nice feature I'd be sad to lose. Maybe a separate libvips gem which was tied directly to libvips updates and fixes would resolve this?

[Nakilon]

Is this common in the ruby world?

I remember some guy wanted to use ruby instead of numpy. I don't remember the exact gems but they were installing alone and when you require them they automatically find the "backend library" for large number arithmetics (installed via brew) available that could be one of a several. Just an example I'm aware of and it seemed neat.

[lovell]

libvips and a selection of its dependencies (mostly web-related en/decoders that are considered "safe" i.e. fuzz tested), is made available as a statically-linked shared library for the most common platforms+archs via the tarballs at https://github.com/lovell/sharp-libvips/releases

These can be adapted for use with other language runtimes. Kleis maintains a fork for use with NetVips - see https://github.com/kleisauke/libvips-packaging/releases

Every version of sharp published to npm uses one version of these binaries and includes the integrity hash values.
https://github.com/lovell/sharp/blob/e40a881ab4a5e7b0e37ba17e31b3b186aef8cbf6/package.json#L157-L169

Patch releases on the same version line of sharp use the same version of libvips. A new version of libvips will be introduced with a minor version increment of sharp (although sharp is still version 0 so technically it's a semver "major" bump).

The logic to select, download and unpack the relevant tarball is in https://github.com/lovell/sharp/blob/main/install/libvips.js

If we're looking for an example from the Ruby ecosystem that does something similar, the nokogiri gem might be a good place to start.

[jcupitt]

Thanks Lovell!

I suppose the next step would be for someone to make a quick proof of concept hack.