GT Mode

Ground-Truthing mode enables you to assess the performance of your tracker on an unbiased set of APT-generated test frames.

Usage

• Create a project, add movies, label frames, and train a tracker iteratively as you normally would in APT.
• Select Evaluate>Ground-Truthing Mode. An orange "GT Mode" indicator should appear in the main UI, and the Manage Movies window should appear.
• Manage Movies is now tabbed, with the "GT Movie List" tab selected. The project now contains two sets of movies: i) "regular" movies used for training and parameter refinement, and ii) "GT" movies for testing tracker performance.
  • Add test movies to the GT movie list. If possible, it is best to use movies that are not also in the regular movie list, ie that the project has never seen before.
• When the movie list is complete, press the "GT Frames" button to bring up the Ground-Truthing window. The project can be saved at any time during this process.
• In the Ground-Truthing window, press the Suggest button to generate a new/fresh list of frames to label. At the moment, frames are sampled randomly from the available GT movies, with all frames equally weighted. Other options are available at the command-line (see below).
• Click on a row in the table to navigate to a frame, or use the "Next Unlabeled" button. The APT main axes should become highlighted, indicating that the current frame/target is a GT frame. Label this frame. These labels will be used as GT labels against which the tracker will be compared.
• When all GT frames are labeled, press "Compute GT Performance". APT will track the GT frames using the trained tracker, and compare the results to the manual GT labels. ** Along with various plots, the Labeler property .gtTblRes provides a results table for each GT row: manual labels, tracked positions, and L2 error.
• Save the project to preserve your GT movie list, the list of GT frames with their labels, and the GT results.

Open Issues

• See GTSuggestionType.m for additional GT-sampling methods that are currently only available at the command-line.
• There is currently no way to merge GT movies/labels back into Regular movies/labels for eg use in retraining/improving a tracker.
• The final reporting (graphs, output statistics) of GT performance is not finalized or polished yet.
• Many project actions are unsupported in GT mode, eg: track-and-export, importing labels, cross-validation, etc. Not all of these actions are explicitly prohibited however, so they may need to unusual errors.

---

Requirements/Questions

Suppose you are working on an APT project P and want to assess GT performance. How important is it that:

1. P's GT movies/labels are stored in the same project (and .lbl file) as P?
2. While working on GT, you are able to easily see/browse P's regular movielist, labels, etc? What about vice versa, ie seeing P's GT movies/labels/etc while working "normally" on P?
3. It is very easy to toggle between working on GT and working "normally" on P?
4. You be able to incorporate GT labels back into the normal labels for P for retraining? (Subsequently maybe you might create new GT labels to reassess?)
5. You be able to arbitrarily track/track-and-export parts of GT movies, as opposed to having eg a totally canned "Compute GT Performance" button/operation that tracks GT movies only where labeled and which allows no further interaction?

Any other thoughts, things I missed etc?

Functional

201709 GT Manager UI note