Skip to content

v0.17-alpha: Introducing the -watch option, and waaaay more!

Compare
Choose a tag to compare
@ajbouh ajbouh released this 08 Sep 06:24
· 15 commits to master since this release
So this commit is massive. Sorry about that. I got carried
away adding -watch support and ended up improving a lot of
other things along the way.

Current qa -watch features include runtime dependency analysis
of any require, load, and require_relative. Editing a test
file will re-run all test cases in that file, but editing
a dependency of a single test will only re-run test that
use that functionality. Future work may introduce finer-
grained logic.

Debut (and rename) "qa auto" as "qa watch". This functionality is
available on any qa test run command. For example:
  qa minitest -watch

Globs used by the -watch option support {foo,bar} style
expressions (in addition to * and **), so you can do obscene things
like:
  qa test-unit -watch 'test/{unit,functional}/**/test*.rb'
Note the single quotes so the shell doesn't expand the pattern
and prevent the filesystem watcher from discovering new tests
that fit the pattern.

Split -warmup into -warmup and -eager-load. This way we can avoid
-eager-load when capturing runtime dependencies, which would
introduce noise. Make -warmup default to true again, so basic
rails tests can pass out of the box with a -jobs value > 1.

Make test filter for rspec deterministic across runs (other
frameworks already had this property). This was required for
-watch functionality.

Add -memprofile and -heapdump command line options to help
diagnose memory usage problems within qa itself. This was
important for getting heap usage low enough for -watch to
be useful on large rails projects like discourse.

Improve support for syntax and load errors during test runs. This
was also important for -watch to be useful in the real word.

Also add automated demo asciicast creation. A decent amount
of work in this behemoth of a change was thanks to iterating
on qa so the demo asciicasts would look reasonable. For example:
- Strip minitest paths from backtraces.
- Refine normal qa run output to be tighter and easier to scan.
- Tweak colors and ordering of snail report.
- Fix test seeding to actually generate different values each
  qa run, discovered while trying to generate interesting data for
  a `qa flaky` demo.
- `qa flaky` output has color and no more cruft.
- `qa flaky` shows error messages in overview.
- `qa flaky` shows relative frequency of each type of error.

Other changes that got sucked into this change and couldn't
escape its gravity well:
- Rename tapjio event types to have more regular, sane names.
- Fix a bug in minitest spec support that caused problems
  for tests that had different but identically named describe
  blocks. Thanks to @halostatue for reporting it!
- Add framework-specific commands, so tab completion works. So
  instead of:
    qa run minitest:test/**/test*.rb
      use
    qa minitest test/**/test*.rb
- Fix hang after ctrl-c of qa run command.
- Enable local variable capture by default on macOS, leave disabled by
  default on Linux (for stability reasons).

Finally, update README.md and include the most basic asciicast.