Add tests and docs for tests #25
Comments
liammulh
changed the title
The Vue Test Utils are "a set of utility functions aimed to simplify testing Vue.js components". This package allows us to unit test the Vue components of Numberscope. Also, adds complete examples of a Vue component test (SeqGetter.vue) and of a Vue view (Home.vue). The tests may be run with `npm run test:vue`. This works toward, but does not fully complete, a resolution of issue #25.
|
Unit tests are added in #123. |
|
The ability to add integration tests is also done in #123. |
|
Is there an example integration test? What are the relevant source files? Thanks. |
gwhitney
changed the title
|
As far as I can see, there do not actually exist any integration tests as of yet, and in addition, some of the unit tests are disabled because certain aspects of the testing are not working (as documented in the code). We need to get to the point that each source file has at least some tests properly running on its code. |
Yes, this is true. I believe integration tests can be written using Vitest without any further configuration.
I agree. A couple of points I think we need to discuss:
I'm not actively working on this, so I'm going to un-assign myself. |
Is there any guidance of how to go about this? In my mind, an "integration test" is something like "bring up the frontscope in a browser, click on certain buttons, and it should produce a picture" -- i.e., something that uses the whole system running end-to-end. I presume vitest doesn't allow this sort of thing (as I presume it does not incorporate a browser that allows prescribed clicks etc.) but perhaps there are some sorts of integration tests for a package like frontscope that do not involve the use of a browser and simulated user interaction. Any thoughts on what an example of such might be? I mean, can I put a test on the Ultimately, I think we will also need to do automated browser testing with simulated clicks as well -- but I think that will require something like https://www.browserstack.com/ and I think setting something like that up should be left to another issue. |
|
Okay, the definitions I have in my head are:
These definitions seem to comport with what is written in this blog post. (I think the author, Kent C. Dodds, is a fairly reliable source for info on testing JS.) As for E2E testing for Vue, I think the Vue people want us to use Cypress. See https://vuejs.org/guide/scaling-up/testing.html#recommendation-2. I believe Cypress can also be used for testing components. Would the following be helpful in the docs somewhere? Types of tests
|
I think we might be able to test some of the TypeScript files this way. I will look into this further after I get some dinner. |
|
OK, my apologies for confusing the terminology a bit between integration and end-to-end testing. It is a bit of a blurry line, though: take component testing, which is mentioned in the references you give as one important type of integration testing. How much does a component test for the Scope view differ from end-to-end testing? As far as I can see, pretty much only by not having navigation among the views, which in our case is a pretty minor aspect of the app. So I think at the moment we should go as far as we can with vitest testing and worry about Cypress end-to-end testing as another issue. In particular, the Scope and BundleCard tests are skipped for now, but we should have tests for both and between them (or possibly as an additional integration test) we should have a test that loads enough of the system so that a bundle can be created, I guess by function calls rather than events, and we can check that it is actually drawing on canvas. I guess that vitest can be used to do that, but I admit I am a bit unclear on how to set up such a test. |
Maybe? The issue I was running into when I tried to create tests for those components makes me skeptical that Vitest + Vue Test Utils is enough for this. I'm also not clear on how exactly we'd set up such a test. I think we should investigate if it would be easier to write this test using Cypress or Vitest. I don't want us to spend a ton of time trying to get this working using Vitest only to find out that it would have been a lot easier to do in Cypress. Also, I think Cypress is something we're going to want eventually, so I don't see the harm in investing some time into it. I'll sink some time into this tomorrow morning and report back here with my findings. |
|
Yes well the operative word was "guess" -- it was just that. If you have the inclination and time to investigate whether testing these sorts of things would be easier with Cypress, more power to you. I think it's good that we seem to be in agreement that testing them one way or the other is of significant value. |
|
Okay, I've just installed Cypress, and I think I see a solution to the problem I was having when trying to add tests for the components @gwhitney mentioned above:
If we add the p5 script in the wrapper, I think the issues I was having will go away. That being said, it's been a while since I was working on adding those tests, so maybe I'm mis-remembering the issue. |
|
I worked on my Cypress PR a bit last week. From the Cypress docs:
I think it makes sense to rewrite the existing tests in Cypress and just abandon Vitest in favor of Cypress.
Need I say more? |
|
As usual, my position on something like testing is that if you are willing to do the work and Cypress seems suitable (more so than vitest), by all means proceed. |
|
P.S. And indeed, if you are pursuing this then it would be great if you could assign yourself so that we can see that the last remaining issue in the first Project of the "roadmap" is being attended to :) |
liammulh
changed the title
liammulh
changed the title
|
This is an "epic" issue in the sense that it has multiple "sub"-issues: Right now, we have two testing frameworks: Vitest and Cypress. I think the plan was to rewrite Vitest tests in Cypress. However, I think Cypress is mostly used for testing code running in the browser, but we also need to be able to test generic code that doesn't run in the browser. I think we can use Vitest for this.
|
|
This should be brought to some satisfactory status with #420. Setting milestone. |
Also adds new tests for `src/shared/defineFeatured.ts` and corrects the documentation extraction facility for the package manager scripts. Resolves numberscope#25. Resolves numberscope#73. Resolves numberscope#246.
* test: Implement end-to-end testing with Playwright This is an in-progress cleaned version of the final PR (#361) of the Delft student user interface project, which it supersedes. Comments from the original PR that remain relevant: * The end-to-end tests run using Firefox and Chromium. * New tests are in the e2e folder. * The tests often depend on specific classes and IDs, so they may need to be updated upon changes to Numberscope. * The tests can be executed as the following npm script: `npm run test:e2e` * An interactive testing UI and debugger can be executed as the following npm script: `npm run test:e2e:ui` Caveats concerning trying this cleaned PR and its status: * Make certain to run `npm install` after pulling this PR. * Many of the tests do not yet pass, perhaps because of the "specific classes and IDs" point mentioned above and the fact that ui2 has diverged significantly from the Delft PR series. * Tests are not yet run automatically prior to commit. * I do not think there are any image tests yet, we need to try to add them. * Tests are not yet performed in the continuous integration checks to be run on GitHub; they should be. * fix: Repair gallery/click on featured item test * fix: Repair gallery/saving a specimen test * fix: Repair scope tests. * test: Run end-to-end tests prior to any commit * test: Add image tests to e2e The strong desire for image tests led to a cascade of changes in this commit, mostly driven by the need to have reproducible images: - Removes all use of `sketch.noLoop()` and `sketch.loop()` in favor of the previously existing `stop()` and `continue()` visualizer methods, to allow: - Adds a `frames=NNN` query parameter to URLs to set the maximum number of frames a visualization may draw - Switches from the "static" instance of mathjs to a "dynamic" one, to allow its random number configuration to be controlled. In conjunction with this, moves all math functions into a single math module, as extensions of mathjs. - Removes all use of `Math.random()` in favor of the mathjs random generator - Adds a `randomSeed=AAAA` query parameter to URLs to make the mathjs random generator reproducible. - Documents all of the above changes. * doc: describe running and creating code tests Also adds new tests for `src/shared/defineFeatured.ts` and corrects the documentation extraction facility for the package manager scripts. Resolves #25. Resolves #73. Resolves #246. * test: Test that the caching mechanism won't double-calculate In other words, it should never call calculate twice for the same index. This is tested by 10K random accesses to indices less than 1M, followed by accessing the first 10K entries, followed by accessing the last 10K entries. Hopefully that should suffice. Resolves #54. * fix: Prevent ModFill from hanging on extremely large input This is an initial pass at addressing #113. Note, however, that ModFill is not reporting to the person doing visualization that it is running with different parameter values than shown. So that still must be done, but for that part we will need a resolution to #112, which will be a sufficiently involve change that we should leave it to a spearate PR from this #420. * doc: Update PR checklist. Resolves #174. * maintenance: Remove and disallow trailing whitespace in code Resolves #219. * maintenance: Make TypeScript target ES2022 in all cases. Resolves #226. * maintenance: Run typecheck in CI Resolves #292. * maintenance: Run GitHub CI on pushes to main as well Resolves #217 * test: Add OEIS Sequence-Visualizer test grid Use a list of "stressful sequences" to perform at least two image tests on each visualizer. As this change uncovered several existing errors, there are numerous other changes in this commit to return to a state of all tests passing. Here is an enumeration of other changes, in no particular order: * Change husky pre-commit actions so that the end-to-end tests are not run if there is an existing successful test since any files in the project last changed. * On most image tests, snapshot only the visualizer canvas. * Make 'axios' a runtime dependency, rather than just development, as it is used when obtaining OEIS sequence values in the running frontend. * So many other changes are made to Histogram.ts that this PR also adds hatching to show the elements with unknown factorization. Adds the "p5.brush" package to draw the hatching. Since this necessitates WebGL, adds a new 'P5GLVisualizer' base class for visualizers that wish to use a WebGL canvas. * Updates several dependencies to latest to make sure their out-of-dateness was not contributing to test issues. * Switches to the (ES) "module" import system from "commonjs" to ensure that was not contributing to test issues. Resolves #294. * Specifies font locations in a way that vite understands how to relocate in both the 'dev' and 'build' versions of the app. * Properly marks both factor cache and value cache as empty at the initialization of an OEIS sequence. * Allows ±Infinity as valid values of INTEGER param fields for convenience. * Allows specimenQuery to take the output of parseSpecimenQuery to recreate the same specimen query as was parsed, for the sake of testing. * Turns off the browser default context menu on visualizers, which wasn't particularly useful and was interfering with the UI for some visualizers. * Fixes typos in Differences visualizer (this.first -> sequence.first) * Makes FactorHistogram visualizer a p5 WebGL-based visualizer, and adds hatching to the "0 factors" bar to indicate the terms with unknown factorizations. * Uses the WebGL default controls to implement panning, zooming, and (rather uselessly but somewhat spectacularly) rotating the plot in three dimensions for the FactorHistogram visualizer. * On FactorHistogram's first pass over the sequence data, collects the counts for each possible number of factors, as opposed to the number of factors for each entry, to avoid a possibly disastrously long loop on the second pass that accumulates bin counts. This also streamlines the computation of the largest number of factors in the data. * Factors out repeated code in FactorHistogram, for labeling bars and displaying text. * Fits the hover box in FactorHistogram to the text to be displayed. * FactorHistogram displays a temporary message if factoring is taking a long time. * Moves the bar labels of FactorHistogram just under their respective bars, to make sure they are visible. * Selects y-axis ticks at round numbers for FactorHistogram, and moves the tick labels closer to their ticks. * Prevents FactorHistogram from looping except when there is mouse activity. * Prevents ModFill from freezing up if too large a "mod dimension" is chosen. Note TODO: display a warning when a smaller mod dimension than requested is actually used. * Prevents Vue's reactivity system from attempting to modify the behavior of p5 sketches, using the Vue `markRaw` method. This change prevents some instances of infinite loops caused by cascading change notifications. * Tightens up the typing of P5Visualizer so that it is possible to derive another visualizer base class P5GLVisualizer from it. Also splits up the inhabit method so that P5GLVisualizer can modify it as needed. * Allow negative start indices for Show Factors visualizer. * Show at most 100 terms in Show Factors (no room on screen for more than that). * Updates TypeScript target versions of JavaScript Resolves #226. * refactor: better typing for param descriptions * feat: Add ExtendedBigint ParamType * refactor: Remove generic parameters from top levels of Paramable hierarchy * feat: Uniform sequence bounds controls All params for controlling which terms of a sequence will be used in the visualization are removed from individual visualizers. Instead, there are uniform params in the Sequence classes themselves. Resolves #411. In this implementation, several other changes are necessary and/or expedient to make. In particular, the type of sequence indices is changed to bigint and bounds to ExtendedBigint (the union of bigint and ±Infinity), in preparation for addressing #455. Additional changes include: * Removal of SequenceDefault class, as everything now derives from Cached. * More care in caching of OEIS sequences, to provide some help with #459. * Individual-parameter validation functions now take a validation status to update based on the finding. This refactor avoids a frequent need to merge status objects. * Individual-parameter validation functions are called in a context of `this` set to the Paramable object, so that other data from the paramable can be accessed. * New `math` functions for dealing with ExtendedBigints. * NumberGlyph visualizer now tries to display as many terms as are available and will fit on the screen, except in case there are infinitely many terms available, in which case it still defaults to 64 terms. This change is needed so that the general length parameter for Sequences would affect what NumberGlyph displayed. * Puts reactivation of "known" OEIS sequences into a function rather than at top level of sequences.ts so that the timing of when it occurs can be controlled. * Renames property `_size` of P5Visualizer to `size` as multiple derived classes seem to need this value (i.e., not just used internally in the base class). * Makes sure the p5 loop is restarted if need be every time `.show()` is called on a visualizer. * Temporarily disables A000521 transversal tests until we can get incremental OEIS loading to work. * Adds a test for starting a visualization deep into a sequence. * refactor: Apply OEIS modulus on the fly, so cache can point to communal one * test: First working e2e tests with text and transversal Unfortunately, due to the intricacies of end-to-end testing with image comparison, this is a very large commit. It introduces running tests in a Docker container for the sake of reproducibility. (So note that henceforth you must have docker installed and running on your machine to perform testing, and hence to make a commit.) However, it turns out not all tests can be run in a Docker container -- Firefox is not able to supply a WebGL context inside of Docker. Hence, some tests still need to be run directly on the host machine. To deal with all this, I felt it necessary to introduce the "make" tool. In particular, creating the necessary Docker image is slow, and I didn't want to have to repeat that except when truly necessary. That, in turn, means there are now many more configuration files, as well as auxiliary files created by the Makefile. The directory tree was becoming hopelessly cluttered. To keep things straight in my head, I felt it was really necessary to reorganize the directory structure. The biggest change is to put as many of the configuration files as possible in a new etc/ directory (the old-school unix name for where to put such things). I managed to get almost everything put in there. The one major holdout is the tsconfig files; I just couldn't find a way to get TypeScript to run without its config files at the top level of the project. Slightly annoying, but then, there are so many more annoying things about TypeScript. Anyhow, accomplishing the configuration file rearrangement resulted in updating eslint. Unfortunately, it went through a _major_ rewrite from version 8 to 9, and prettier-eslint has not been updated to work with the new version. I tried to work on that update myself, but prettier-eslint is incredibly intricate in the way that it handles configurations because it is trying to provide a great deal of flexibility and power, and it is trying to infer prettier configuration from eslint configuration (and maybe even vice-versa as well, I'm not sure). Since we were only using a tiny fraction of that power, it turned out to be significantly easier to just replace prettier-eslint with a custom script tools/prettiest.js that runs first prettier, then eslint. It also meant we could get rid of the lint-staged tool, so there is that. (While I was at it, I ran the "depcheck" tool and got rid of some other unneeded cruft that had accumulated.) But now it all works, and as a side benefit since just about every npm script now runs through make, you should never need to worry about running install before dev or build before preview, etc. Make keeps track of what depends on what and whether/how to redo the prerequisites before taking the requested action. Even better, if you just ran successful end-to-end tests, which take a while now, then `git commit` will not have to re-run them. Those are the major points. In addition, there are a number of minor changes resulting from errors uncovered in getting all the tests to work, etc: - Reordering of element attributes and other reformatting in vue components, because with the lint updates and reconfiguration, it now seems significantly stricter about vue templates. - First pass at an updated User Guide for ui2; it definitely needs more work. - Some other more minor documentation updates. - Updates tools/editor/autoformat.el so that emacs can use the new "prettiest" tool for formatting (even though I know I am likely the only one in the world who will ever use that). * test: Add skipped tests reflecting known shortcomings As per Numberscope discussion today, this is the last missing element of the end-to-end testing PR. With these skipped tests, we are flagging that there are known concerns we want to resolve at least by beta, if not alpha. Also fixes a small gap in the docs for running from source that Aaron noticed in the meeting. With this, the PR will be marked ready for final review. * fix: Generate docker image even with no existing test results * chore: Attempt to run e2e tests in CI workflow * test: No WebGL tests in CI :-( * fix: format of Playwright reporter parameter * chore: Try to grab GitHub CI actuals to make separate CI test file * fix: oops test must succeed to generate artifact * chore: Another try to grab ci snapshots * test: Add in the extracted CI snapshots * doc: Note that PR reviewers must run e2e tests themselves, too. * chore: Stop grabbing snapshots when we don't need them * fix: Changes as per review comments * doc: more info on Docker (and fix remaining typo) * test: Fuzz the pixel comparison in Firefox WebGL tests * test: Really fuzz the pixel comparison in Firefox WebGL tests * test: Check if docker tests passed before updating result directory * doc: Improvements per code review * doc: show the expected output of successful end-to-end test * chore: remove stray comment that no longer applies * fix: useful default camera controls for WebGL: left drag pans, wheel zooms * fix: correct Husky action inclusion and test it * fix: don't alter the URL just loaded, and reset frame limit on changes * fix: adjust dragging, detected mouse position, and text size for zoom/pan * fix: Keep the 'too many bins' message in a fixed absolute canvas position
|
Done in #420 merged into ui2. Closing. |
github-project-automation
bot
moved this from In Progress
to Done
in 01 Update frontscope
EDIT: We have Vitest set up and working. #245 adds the ability to create E2E tests with Cypress. The original comments in this issue are stale. See #25 (comment) for up-to-date status.
The text was updated successfully, but these errors were encountered: