v0.18-alpha

Better flaky test debugging tools

The headline features for this release are focused around debugging flaky tests. See the asciicast for a demo of how to use it. Please file an issue if you have any trouble!

The other major change for this release is that qa is now only available as a rubygem. It's no longer trivially distributable as a tgz, as it has dependencies on other rubygems. For now we accept this limitation as it keeps the build process simple and these third-party dependencies provide a dramatically better user experience.

Other usability improvements:

• Improve messaging around seeds used for multiple
  test runs.
• Errors now always capture locals, since we're
  using the more stable binding_of_caller gem.
• Support for qa help
• Support for -runs argument, so qa can run your tests many times
• Use TEST_ENV_NUMBER instead of QA_WORKER, like parallel_tests gem
• Add support for QA_ARCHIVE environment variable to avoid needing to specify -archive for each command.
• Add qa flaky top subcommand.
• Add qa flaky repro subcommand to reproduce a specific flaky outcome and attach a pry-remote session to it.
• Make qa flaky an (experimental) REPL for working with flaky tests. The two subcommands you can use with it are repro and top.

Along with a host of fixes:

• Fix local variable capture in cases where bindings do not exactly match the exception backtrace.
• Move (experimental) ActiveRecord model eager loading to -eager-load option (which is disabled by default)
• Concat /lib, etc to $LOAD_PATH, rather than prepending it.
• Fix yet another hang-on-ctrl-c.

v0.17-alpha: Introducing the -watch option, and waaaay more!

So this commit is massive. Sorry about that. I got carried
away adding -watch support and ended up improving a lot of
other things along the way.

Current qa -watch features include runtime dependency analysis
of any require, load, and require_relative. Editing a test
file will re-run all test cases in that file, but editing
a dependency of a single test will only re-run test that
use that functionality. Future work may introduce finer-
grained logic.

Debut (and rename) "qa auto" as "qa watch". This functionality is
available on any qa test run command. For example:
  qa minitest -watch

Globs used by the -watch option support {foo,bar} style
expressions (in addition to * and **), so you can do obscene things
like:
  qa test-unit -watch 'test/{unit,functional}/**/test*.rb'
Note the single quotes so the shell doesn't expand the pattern
and prevent the filesystem watcher from discovering new tests
that fit the pattern.

Split -warmup into -warmup and -eager-load. This way we can avoid
-eager-load when capturing runtime dependencies, which would
introduce noise. Make -warmup default to true again, so basic
rails tests can pass out of the box with a -jobs value > 1.

Make test filter for rspec deterministic across runs (other
frameworks already had this property). This was required for
-watch functionality.

Add -memprofile and -heapdump command line options to help
diagnose memory usage problems within qa itself. This was
important for getting heap usage low enough for -watch to
be useful on large rails projects like discourse.

Improve support for syntax and load errors during test runs. This
was also important for -watch to be useful in the real word.

Also add automated demo asciicast creation. A decent amount
of work in this behemoth of a change was thanks to iterating
on qa so the demo asciicasts would look reasonable. For example:
- Strip minitest paths from backtraces.
- Refine normal qa run output to be tighter and easier to scan.
- Tweak colors and ordering of snail report.
- Fix test seeding to actually generate different values each
  qa run, discovered while trying to generate interesting data for
  a `qa flaky` demo.
- `qa flaky` output has color and no more cruft.
- `qa flaky` shows error messages in overview.
- `qa flaky` shows relative frequency of each type of error.

Other changes that got sucked into this change and couldn't
escape its gravity well:
- Rename tapjio event types to have more regular, sane names.
- Fix a bug in minitest spec support that caused problems
  for tests that had different but identically named describe
  blocks. Thanks to @halostatue for reporting it!
- Add framework-specific commands, so tab completion works. So
  instead of:
    qa run minitest:test/**/test*.rb
      use
    qa minitest test/**/test*.rb
- Fix hang after ctrl-c of qa run command.
- Enable local variable capture by default on macOS, leave disabled by
  default on Linux (for stability reasons).

Finally, update README.md and include the most basic asciicast.

Stability fixes and better flakiness detection logic

Changes since v0.15-alpha

• Use tcp listener during qa tests
• Don't die on syntax errors
• Only close test context if it started w/o error
• go fmt
• Fix error and TAPJ stream handling
• Add suite label and coderef options
• Properly generate watch expression for runner
• Fix typo in -eval-after-fork help text
• Add description of flaky test detection algorithm

v0.15-alpha: Consolidate and simplify ruby-specific logic

Drop emitter package in favor of moving more logic
into run and runner packages.

v0.14-alpha: Consolidate and simplify ruby-specific logic

Drop emitter package in favor of moving more logic
into run and runner packages.

v0.13-alpha

Update README.md

v0.12-alpha: Disable warmup logic by default

It's still pretty experimental and probably shouldn't
be on by default yet.

v0.9-alpha

Add default globs for -squashall, -pendantic

v0.8-alpha

FIX linux build

v0.11-alpha

Add test coverage for stdout, stderr capture