Publish metrics to Dockstore

[adamnovak]

Changelog Entry

To be copied to the draft changelog by merger:

• Toil now tracks workflow execution history in ~/.toil/histroy.sqlite
• Toil can now publish workflow execution metrics to Dockstore for workflows where it knows the TRS ID (i.e. workflows run by TRS ID or Dockstore URL).
• Toil will prompt the user to decide whether they want to publish workflow metrics, and, if it thinks it can get ahold of the user, will not proceed until they decide.

Reviewer Checklist

• Make sure it is coming from issues/XXXX-fix-the-thing in the Toil repo, or from an external repo.
  • If it is coming from an external repo, make sure to pull it in for CI with:

    contrib/admin/test-pr otheruser theirbranchname issues/XXXX-fix-the-thing
    

  • If there is no associated issue, create one.
• Read through the code changes. Make sure that it doesn't have:
  • Addition of trailing whitespace.
  • New variable or member names in camelCase that want to be in snake_case.
  • New functions without type hints.
  • New functions or classes without informative docstrings.
  • Changes to semantics not reflected in the relevant docstrings.
  • New or changed command line options for Toil workflows that are not reflected in docs/running/{cliOptions,cwl,wdl}.rst
  • New features without tests.
• Comment on the lines of code where problems exist with a review comment. You can shift-click the line numbers in the diff to select multiple lines.
• Finish the review with an overall description of your opinion.

Merger Checklist

• Make sure the PR passed tests, including the Gitlab tests, for the most recent commit in its branch.
• Make sure the PR has been reviewed. If not, review it. If it has been reviewed and any requested changes seem to have been addressed, proceed.
• Merge with the Github "Squash and merge" feature.
  • If there are multiple authors' commits, add Co-authored-by to give credit to all contributing authors.
• Copy its recommended changelog entry to the Draft Changelog.
• Append the issue number in parentheses to the changelog entry.

[adamnovak]

The history command I cut from this is in the toil-history branch.

There aren't really any unit tests for the history tracking machinery, or any fake Dockstores against which we test the submission machinery.

[adamnovak]

One missing piece here is locking/waiting for reporting when workflows finish simultaneously. Multiple Toil jobs can be trying to send the same workflows/jobs to Dockstore at the same time, and Dockstore will fail at least one of them with a duplicate ID complaint.

The publication will be retried by the next Toil workflow to come along though. Maybe that's better than a real locking/reservation system?

[adamnovak]

One thing we might need is connection keeping like in the caching system. Right now we just reconnect to the database for every job we have to add to it, which is possibly slow and might not do our job-recording throughput any favors.

[adamnovak]

Do we need a way to batch database writes from each Toil process, with some kind of atexit recording method, to avoid slowdowns caused by database lock contention?