feat: allow searching for similar images by providing a path or URL to an image

[ramayer]

Sometimes it's more convenient to search for "a picture similar to the one I have in mind", instead of trying to describe it with words. Especially considering the extremely short phrases CLIP allows.

I'd like to be able to run:

 rclip https://i.redd.it/a7jf5rlghsz51.jpg

to return pictures of similar buildings like these.

It would also be nice to be able to search for pictures similar to a local file, like rclip /tmp/turtle.jpg and/or rclip file:///tmp/turtle.jpg.

This branch has a working work-in-progress for both of those features; but probably isn't ready for a pull request yet because it has a python3.9 dependency (str.removeprefix is a 3.9 feature). If you're interested, I'll remove that dependency and open a pull request.

[yurijmikhalevich]

I like the option to search for similar images.

This will affect optimizing querying by splitting the model into text and vision transformers, but we may only load the vision transformer when the user needs to search for similar images.

I also like an option to search for rclip /image.png -a night :)

Let's merge the feature for adding and subtracting queries first.

I left a few comments here: ramayer@ea09b08

[yurijmikhalevich]

Let's merge this whenever you are ready 😄

[ramayer]

I have some questions about the desired syntax/behavior. How do we want to decide when to treat something as an image?

One possibility would be to add two new command line parameters:

• --add-image
• --sub-image

Each of those parameters could take a reference to an image that could start with http://, https:// for images to be fetched through their respective protocols; or a path (relative or absolute) or file:// to refer to a local file. If we go this way, we'd want to make query optional; since I find the use case of just looking for similar images very useful.

.

A different possibility would be not add any new parameters, but rather use the same parameters (query, --add, --sub) that we currently use; but try to guess the user's intent based on the format of the string like:

• If a string starts with either https:// or http:// - assume the intent was an image to fetch over HTTP.
• If a string starts with / or c:\ or file:// or ./ - assume the intent was similarity to a local file
• If it starts with anything else - treat it like the current strings.

That's what my current local environment is doing. Kinda convenient to just go rclip https://i.redd.it/a7jf5rlghsz51.jpg or rclip /mnt/pics/2001-01-01/whatever.jpg; but it opens the door to complex and confusing parsing of parameters.

.

Yet another possibility would be a hybrid - where the default query parameter gets some magic parsing logic applied to it; but the --add and --sub continue to work as text strings; and we add --add-image and sub-image like the first option.

.

Or would we want even more parameters to separate out local files vs network access; or to distinguish between URIs and local paths?

I'm not sure which option I like best. I can imagine annoyances with each of them.

[yurijmikhalevich]

Hi @ramayer,

I like the idea of having a simple parser that treats everything that starts with https://, http://, file://, /, X:\, or ./ as a file path.

I would love to keep the number of parameters to a minimum unless we really need to add them. I can't imagine a situation when you'll want to include these prefixes in your regular text query. What do you think?

Let's start by implementing a simple parser. If we will encounter any issues in the future, we will adjust accordingly :)

[yurijmikhalevich]

Closed by: #21