Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry mechanism #62

Open
glureau-betclic opened this issue Dec 11, 2020 · 3 comments
Open

Retry mechanism #62

glureau-betclic opened this issue Dec 11, 2020 · 3 comments

Comments

@glureau-betclic
Copy link

I'd love to have a mechanism that retries failed tests but not with the rerun file mechanism.
I'm interested to have a report that is OK if at least one of the X attempts is OK (retry a test only when it fails indeed), and generate a unique report files. (Better again if the report can flag the flaky tests.)

Is something already available for this use case?

If no, I think about forking this repository to be able to implement my own AndroidFeatureRunner + RunNotifier, to be able to call child.run(myNotifier) as much as I need. Is it a good way to approach the problem?

Thanks for your time and this great tool!

@lsuski
Copy link
Contributor

lsuski commented Dec 11, 2020

I think that there is no such feature currently. Maybe in AndroidJunitRunner from which CucumberAndroidJUnitRunner inherits but I doubt. In my opinion this feature could be implemented independently of cucumber and based on junit. From perpective of junit and other tools relying on it Cucumber feature is treated as class and scenario as method so I would research if something similar is available in Android or junit world. If you plan to implement it then you can make PR and we could try to integrate it in this repo

@glureau-betclic
Copy link
Author

So far I've seen that Maven Surefire defines a rerun mechanism, Gradle does (or plan to do) the same, but both tools are not well suited when we use Cucumber on a device farm. In my case for example, I'd have to upload the apks (instrumentation+app under tests) as much as I want retries. And when using Bitrise, which generates only one Firebase test token by workflow, I'd have to define multiple workflows or recursively chaining the same workflow, clearly something that I want to avoid.

apache/maven-surefire#249
https://blog.gradle.org/gradle-flaky-test-retry-plugin

There is a JUnit 5 rerunner project here https://github.com/artsok/rerunner-jupiter/ but so far Cucumber is locked on JUnit4 so I'm discarding this option for now as I'm not sure what you're planning on this upgrade.

On JUnit4, there is a TestRule class that can be define next to your annotated [@]Test methods to define the retry behaviour: https://stackoverflow.com/a/8301639/1898596 or https://wiki.saucelabs.com/display/DOCS/How+to+Deal+with+Flaky+Java+Tests
It has some drawbacks as far as I understand:

  • not sure it's possible to replay the [@]Setup method, so I suppose the Background in Cucumber, and that's indeed quite problematic if we want a proper restart.
  • it pushes the question of the expected behaviour when used in combination with Orchestrator, as the instrumentation app is restarted, rerun could potentially run on another device (sharding).

Having the app fully restarted improves the test stability so it could be great to handle the retry mechanism in such a way it's Orchestrator/Spoon compatible.

Eventually another cool thing could be to detect the retry count directly from a tag on the test itself, so that we can mark some tests as flaky with some details about how to retry them. Can Gherkin offer a proper way to pass data on a tag for this purpose? (Personally I'm great with a global settings for now)

As my understanding of JUnit intricacies are very limited, feel free to push any information or ideas here 🙏

@glureau-betclic
Copy link
Author

I hacked something this evening, looks promising so far (I mean it works in my simple case :) )
glureau-betclic@5653931?w=1

If you've some time to review it, I'd love some feedback on the approach before to go further.

And if the approach with custom AndroidFeatureRunner + RunListener is viable, I'd love a list of stuff I'd have to support to make this feature mergeable (scenario, outline, ...).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants