Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancement: present final rerun test summary info section in the same way as the test session starts section #191

Open
jeffwright13 opened this issue May 18, 2022 · 0 comments

Comments

@jeffwright13
Copy link

The test session starts section is really nice in how it presents a running test suite's test outcomes: each test is shown in succession, is color-coded according to outcome, and the whole flow is easy to follow because you can follow along as individual tests get run, and either pass or fail, thereby triggering the plugin's rerun mechanism.

For example, it's easy to see in the following run:

  • that the first test passed without a rerun
  • the second test ran twice (1 rerun) but failed both times, leading to an overall failed test result
  • the third test ran three times (2 reruns) and eventually passed
  • the fourth test ran four times (3 reruns) and eventually failed
  • the fifth test ran three times (2 reruns) and eventually failed
  • the last test ran once and passed

Screen Shot 2022-05-17 at 11 49 05 PM

However, the rerun_test_summary_info section (which is obtained by invoking command line option -raR) leaves a bit to be desired:

  • not color-coded
  • only states which tests required a rerun (regardless of final outcome) instead of giving a blow-by-blow account of each test as reruns occur

Is it possible to make the final section more like the first?

Here is my testcase for above screenshot. I ran the test session as:
pytest -v -raR -k flaky

import pytest
import random

def flakey_pass_fail():
    random.seed(random.randint(0,100))
    return random.randint(0, 1) == 1

@pytest.mark.flaky(reruns=0)
def test_flaky_0_reruns():
    result = bool(flakey_pass_fail())
    assert random.randint(0,1)

@pytest.mark.flaky(reruns=1)
def test_flaky_1_reruns():
    result = bool(flakey_pass_fail())
    assert random.randint(0,1)

@pytest.mark.flaky(reruns=2)
def test_flaky_2_reruns():
    result = bool(flakey_pass_fail())
    assert random.randint(0,1)

@pytest.mark.flaky(reruns=3)
def test_flaky_3_reruns():
    result = bool(flakey_pass_fail())
    assert random.randint(0,1)

@pytest.mark.flaky(reruns=2)
def test_flaky_2_reruns_but_always_fail():
    assert False

@pytest.mark.flaky(reruns=2)
def test_flaky_2_reruns_and_always_pass():
    assert True
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant