Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Possible testing framework improvements #36

Open
AxelKrypton opened this issue Mar 14, 2024 · 0 comments
Open

Possible testing framework improvements #36

AxelKrypton opened this issue Mar 14, 2024 · 0 comments
Assignees

Comments

@AxelKrypton
Copy link
Member

AxelKrypton commented Mar 14, 2024

To keep the test log file "clean" we got the habit to suppress standard error of function calls which are expected to fail.

However, by now, I found myself many times uncommenting the suppression to see what that output was and, reviewing PR, I realised that some problems which were not noticed could have been if that output was not suppressed.

It might be cool and convenient to get rid of all these failures. Clearly, this would then clutter the log file so much, that a test failure would not be visible any more. One could do the following then:

  • Use a different logger level (like Print_Attention) to print tests failures, while waiting that the BashLogger developer (😅) adds a way to customise the logger with new levels.
  • Since going through all tests is already painful, this should be done only once and I'd delegate the standard pattern
    if [ $? -ne 0 ]]; then
        Print_Error '...'
        return 1
    fi
    to a function, which would then use a different level. Sometimes we test for unexpected success and this new function should have an interface like Expect_Success $? 'message to be printed' 'even split' 'over lines' with possible siblings Expect_Failure with the same interface and Expect_Given_Failure which also take the expected exit code. These functions in tests should be called with a trailing || return 1 to fail the test.

Another nice improvement would be to make the runner call the Clean_Tests_Environment_For_Following_Test__* only if the test passed. At the moment we kind of do not implement these functions so much because we want files in case of failure, but then we leave them also for those tests which pass. Another alternative approach would be to make the runner remove the test dedicated folder if the test pass (of course, unless the user gave -k). This makes again the Clean* functions rarely needed, which is actually good IMO as it is less work for the developer. I'd go this way.

@AxelKrypton AxelKrypton self-assigned this Mar 14, 2024
@AxelKrypton AxelKrypton transferred this issue from another repository Apr 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant