Retry extension has limitations when used in Azure DevOps pipelines: no flakiness UI and coverage publishing error. Should those be documented?

[turbobobbytraykov]

TL;DR:

1. Retry/Policy extension cannot provide "Flaky" tests in the Tests tab because that's a programmatic challenge imposed by Azure DevOps as a platform & tooling
2. When using the Retry/Policy extension with the PublishTestResults, the PublishTestResults fails because of an ignored issue in its implementation for .coverage files from retries. This is not your problem per se.

I'm bringing these things up as a discussion as I'm not sure wether they're worth your time as issues, but I'd like a public record of each, because at least for me, both of these things push me to using the VSTest task in Azure DevOps and not the MSTest runner.

Longer:

1. About reporting Flaky tests.
   Man, I wanted to have the "Flaky" flag for retried tests just like from the VSTest task in Azure DevOps, but as the official documentation states, you either use the VSTest task or gotta do custom shenanigans - you, as TRX producers and me - as Azure DevOps project admin (either system or custom flakiness detection - can't have both for reasons beyond my comprehension).

My request here is to perhaps note that the retries will produce multiple .trx files and flakiness is not recorded in the same way as the VSTest task.

When you have retries, and use the PublishTestResults plugin's mergeTestResults = true option, things looks like this:

Pros:

• test runs are summarized
  Cons:

• misleading number of tests reported due to the retries

If mergeTestResults = false, then the same results look like:

Runs are not summarized here, but split as per the separate TRX files, but at least the visibility is still there.

2. If you use the Retry/Policy extension, the PublishTestResults task errors out (but still publishes test results!).
   This issue is purely on the maintainers of the PublishTestResults task and I found 2 instance of it being reported & ignored (#12985 and #18471)
   My suggestion is to either change the paths to the .coverage files or just mention that this is a known issue in the PublishTestResults task.

[Evangelink]

Yes sadly we are aware of the limitations and we are having trouble getting help from AZ do so we can take over providing proper integration with the new extension. Previously, the retry was provided only in AzDO and fully handled by their team.