Refactor test for comparing output, do not show plots while testing, no default behavior modified

[bobleesj]

Context

No code was modified under src

• refactored tests/test_NMF_analysis_code.py which runs main(arg) with args being passed.
• prevented from displaying plots while running pytest (the original code does indeed support not showing plots)

Here is the discussion that requires your input: To make the tests pass locally (3 cases), I updated the expected .json outputs (since it's integrated with the cloud, didn't want to change anything under src)

Purpose of this PR

However, it still fails via GitHub Actions, most likely due to convergence problem as displayed in the warnings below or random seed. But, the merit of this PR could be refactoring the test function so that it's easier to debug down the road.

Local test:

========================================== warnings summary ===========================================
tests/test_NMF_analysis_code.py: 19 warnings
  /Users/imac/miniconda3/envs/diffpy.nmf_mapping_env/lib/python3.13/site-packages/sklearn/decomposition/_nmf.py:1759: ConvergenceWarning: Maximum number of iterations 1000 reached. Increase it to improve convergence.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================================== 5 passed, 19 warnings in 8.54s ===============================

[github-actions]

Warning! No news item is found for this PR. If this is a user-facing change/feature/fix,
please add a news item by copying the format from news/TEMPLATE.rst.

[bobleesj]

tl;dr

This main(args=args) creates 3 .json files under nmf_result at the top directly level, which is the default behavior. I then copy these outputs to tmpdir and then compare those three json files under tmpdir against the expected .json files under tests/output/output1, tests/output/output2, and tests/output/output3.

The three 3 json files are:

    json_files_to_check = [
        "component_index_vs_pratio_col.json",
        "component_index_vs_RE_value.json",
        "x_index_vs_y_col_components.json",
    ]

[bobleesj]

@sbillinge ready for your insight. Thank you.

[sbillinge]

In general, my impulse here is to junk this test. In the spirit of the discussion around the test for Qmin Qmax in diffpy.fourigui, I have a strong sense that this test just runs the code, pastes the output in a file and then makes sure the output is the same. The question then is "what behavior is this testing?" If the code is suddenly run with different precision on someone's platform and the test fails, is this desired? In other words, from a functional point of view, do we care about precision? The answer is no of course, we just want the code to work and do an nmf decomposition successfully.

This seems to fall into the trap that the test tests the code, not the behavior.

The right way to discuss this is to address the question of what behavior is this function being tested supposed to be implementing? Then let's write a test that captures the different behaviors we want this to have under different situations.

[bobleesj]

@sbillinge You are right. This was a refactoring task. So we are still just trying to satisfy our test functions that simply run the code, instead of writing tests to capture the behavior that WE want.

The right way to discuss this is to address the question of what behavior is this function being tested supposed to be implementing? Then let's write a test that captures the different behaviors we want this to have under different situations.

Yes, I probably need to re-visit this code after the Fall 2024 tasks. Or do we have a member who could take over this task?

[bobleesj]

Closing: We want to write tests that do not simply fix or refactor the existing test functions that run main and assert the output content. Instead, we aim to test the expected behavior that we want our code to produce