-
-
Notifications
You must be signed in to change notification settings - Fork 60
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Retry mechanism #62
Comments
I think that there is no such feature currently. Maybe in |
So far I've seen that Maven Surefire defines a rerun mechanism, Gradle does (or plan to do) the same, but both tools are not well suited when we use Cucumber on a device farm. In my case for example, I'd have to upload the apks (instrumentation+app under tests) as much as I want retries. And when using Bitrise, which generates only one Firebase test token by workflow, I'd have to define multiple workflows or recursively chaining the same workflow, clearly something that I want to avoid. apache/maven-surefire#249 There is a JUnit 5 rerunner project here https://github.com/artsok/rerunner-jupiter/ but so far Cucumber is locked on JUnit4 so I'm discarding this option for now as I'm not sure what you're planning on this upgrade. On JUnit4, there is a TestRule class that can be define next to your annotated [@]Test methods to define the retry behaviour: https://stackoverflow.com/a/8301639/1898596 or https://wiki.saucelabs.com/display/DOCS/How+to+Deal+with+Flaky+Java+Tests
Having the app fully restarted improves the test stability so it could be great to handle the retry mechanism in such a way it's Orchestrator/Spoon compatible. Eventually another cool thing could be to detect the retry count directly from a tag on the test itself, so that we can mark some tests as flaky with some details about how to retry them. Can Gherkin offer a proper way to pass data on a tag for this purpose? (Personally I'm great with a global settings for now) As my understanding of JUnit intricacies are very limited, feel free to push any information or ideas here 🙏 |
I hacked something this evening, looks promising so far (I mean it works in my simple case :) ) If you've some time to review it, I'd love some feedback on the approach before to go further. And if the approach with custom AndroidFeatureRunner + RunListener is viable, I'd love a list of stuff I'd have to support to make this feature mergeable (scenario, outline, ...). |
I'd love to have a mechanism that retries failed tests but not with the rerun file mechanism.
I'm interested to have a report that is OK if at least one of the X attempts is OK (retry a test only when it fails indeed), and generate a unique report files. (Better again if the report can flag the flaky tests.)
Is something already available for this use case?
If no, I think about forking this repository to be able to implement my own AndroidFeatureRunner + RunNotifier, to be able to call child.run(myNotifier) as much as I need. Is it a good way to approach the problem?
Thanks for your time and this great tool!
The text was updated successfully, but these errors were encountered: