Snapshots for manual/automated visual regression testing

[jamiebuilds]

What's up?

Snapshots are a really efficient way to tell if a fixture has changed from one commit to another. They are much faster than re-running an entire browser visual regression tool that requires diffing screenshots from real browsers.

However, looking at changes in snapshots is not a great experience. A lot of the time when snapshots have changed, it was intended. But that just makes it that much harder to catch when unintended changes have happened. Being able to actually visually see what the changes are is a much more reliable experience.

So what if we used snapshots for their performance, and the Cosmos UI + Browser-based visual regression tools for their feedback?

1. Take snapshots of every fixture in a project
2. Compare snapshots to discover which fixtures have "changed"
3. Highlight fixtures that have "changed" in sidebar in the the Cosmos UI
4. Allow the fixtures in the sidebar to be filtered down to just the ones that have "changed"
5. Have a button in the UI to write the updated snapshot to the file system
6. Allow the cosmos-export CLI to statically build just the fixtures that have "changed" between two commits (Useful for deploying on every PR)
7. Have a library API/CLI command for listing all the fixtures that have "changed" so that other tooling can be run on just the fixtures that have "changed".

Mkay, tell me more...

cosmos-snapshot

Re-run all of the snapshots and list the ones that have changed vs the snapshots written to the file system.

$ cosmos-snapshot
/path/to/__fixtures__/foo.tsx

cosmos-snapshot --write

Same as cosmos-snapshot but write the changed snapshots back to the file system (similar to how prettier --write or jest -u works).

$ cosmos-snapshot --write
/path/to/__fixtures__/foo.tsx (updated)

cosmos-snapshot --since <git-ref>

Instead of comparing the snapshots against the file system, test them against a git ref.

$ cosmos-snapshot --since master
/path/to/__fixtures__/foo.tsx
/path/to/__fixtures__/bar.tsx
/path/to/__fixtures__/baz.tsx

Filters

On top of being able to filter fixtures by their name or path, you should also be able to filter by changed fixtures. Here's an idea for the query syntax:

@changed
@changed(since: <git-ref>)

Note: I think it's important this has a syntax so you can use it for CLI options like below.

cosmos-export --filter <query>

Instead of exporting every fixture, export only the fixtures matching a particular filter.

$ cosmos-export --filter "@changed(since: master)"
/path/to/__fixtures__/foo.tsx
/path/to/__fixtures__/bar.tsx
/path/to/__fixtures__/baz.tsx

Implementation Note: This should not rebuild every single fixture to re-snapshot them. It should be comparing the snapshots on the file system against the ones in the git ref. This will make the export process much faster so it can be run on every commit in a CI environment in even the biggest projects.

Snapshotting Quickly

I can see that Jest Snapshots are being used today. However, running jest is a rather heavyweight way of getting the Cosmos snapshots. It would be much faster if you used some of the libraries of either Jest or AVA to take the snapshots.