Add documentation and licensing for e2e tests #1045 (#1046)

CONTRIBUTING.md

@@ -79,9 +79,13 @@ _You must test your code yourself before asking for review, like this_:
   that their fix is working, but we discover they only applied the fix in one of these two modes. Don't be **that** contributor!
 * Unit tests, which test for regressions with basic app functions, are run automatically with GitHub Actions on each PR and push to a PR. If one of these tests fails, you will want
   to debug. First, see if you can also see the failure by running the tests with `npm test`, whcih should run the tests in all your installed browsers. To address any issues
-  identified, see below under "Unit tests" so you can debug;
+  identified, see [TESTS.md](./TESTS.md) so you can debug;
+* End-to-end (e2e) tests are also run on GitHub Actions when you push to your PR. These test typical user actions in a headless browser. Tests are currently enabled for latest
+  Firefox, Edge, Chrome in Linux and Windows, and in IE Mode on Windows (this is the equivalent to testing on Internet Explorer 11). You can run these tests yourself in a
+  non-headless browser with `npm run tests-e2e-firefox`, `npm run tests-e2e-iemode`, etc. For more information, see [TESTS.md](./TESTS.md). 
 * As an alternative to the Vite server, we also provide [http-server](https://www.npmjs.com/package/http-server), which you can launch by running `npm run web-server` in the root of
-  this repository. This does not have Hot Module Replacement, and you will need to refresh the page yourself by doing `Ctrl-Shift-R` with DevTools open. Again, you will only see the latest version of your code if you turn on "Bypass AppCache" and turn off the browser's native caching (see above).
+  this repository. This does not have Hot Module Replacement, and you will need to refresh the page yourself by doing `Ctrl-Shift-R` with DevTools open. Again, you will only see the
+  latest version of your code if you turn on "Bypass AppCache" and turn off the browser's native caching (see above).
 
 If all the tests are working fine in your browsers, you **must finally test the extension versions**. Please note that we are using Manifest V3 for the Chromium extensions,
 and Manifest V2 for the Firefox extension, so there are different instructions for the two browser families:
@@ -91,21 +95,3 @@ and Manifest V2 for the Firefox extension, so there are different instructions f
 
 If your feature works and tests are passing, make a PR, describe the testing you have done, and ask for a code review. If you do not state what testing you have done, we reserve
 the right not to review your code until you have completed these manual tests!
-
-## Unit tests
-
-You can manually run and debug Unit tests simply by opening `tests/index.html` in Firefox, Edge, or Chromium/Chrome through a (local) web server, such as Vite or http-server (see above).
-Use DevTools (F12) to debug and find out what is failing. Note that this only tests the unbundled (source) code, and so it only works in browsers that support ES6 modules. You can't use
-these tests in IE11 or older Firefox/Chromium.
-
-You can run the UI tests with npm on all your installed browsers with `npm test` in your terminal. Before running the tests, if you didn't already, you will need to fetch development
-dependencies (see "Build system and setup" above). If testing this way, make sure that `http-server` is not already running, because another copy is launched for these tests, and the
-ports may conflict. If running tests in parallel like this produces unexpected results (some tests might be too slow and assert before they have completed correctly), then you can run
-individual tests in headless mode with `npm run test-unit-firefox`, `npm run test-unit-edge`, etc. (see `package.json` for full list of scripts). Note that browsers need to be available
-in standard locations for this to work: they won't be fetched or installed by the script.
-
-We use [Test_Café_](https://testcafe.io/) for testing. You can find out which browsers it knows about by running `npx testcafe --list-browsers` (it may take some time to discover local
-browsers).
-
-If you want to run individual tests visually, not headless, it's easiest simply to open `tests/index.html` in the respective browser, and this allows you to debug. If you really want to
-do it from the commandline, then you'll need, e.g., `npx testcafe chrome ./tests/initTestCafe.js --app "http-server --silent -p 8080 ."` (adapt the browser as necessary). 

TESTS.md

@@ -0,0 +1,49 @@
+# Automated tests
+
+We run two types of automated tests on each push and pull request. These are Unit tests and End-to-end (e2e) tests. Unit tests are designed to test small units of code, crucial
+functions that the rest of the app relies on. End-to-end tests are designed to test the functionality of the app as it might be used by a typical user.
+
+## Unit tests
+
+Unit tests are implemented, for historic reaons, with QUnit. When run in an automated way, these are currently run by using the browser testing framework Test_Café.
+
+You can manually run and debug Unit tests simply by opening `tests/index.html` in Firefox, Edge, or Chromium/Chrome through a (local) web server, such as Vite or http-server (see
+above). Use DevTools (F12) to debug and find out what is failing. Note that this only tests the unbundled (source) code, and so it only works in browsers that support ES6 modules.
+You *cannot* use these tests in IE11 or older Firefox/Chromium.
+
+You can run the unit tests with npm on all your installed browsers with `npm test` in your terminal. Before running the tests, if you didn't already, you will need to fetch
+development dependencies (see "Build system and setup" above). If testing this way, make sure that `http-server` is not already running, because another copy is launched for these
+tests, and the ports may conflict. If running tests in parallel like this produces unexpected results (some tests might be too slow and assert before they have completed correctly),
+then you can run individual tests in headless mode with `npm run test-unit-firefox`, `npm run test-unit-edge`, etc. (see `package.json` for full list of scripts). Note that browsers
+need to be available in standard locations for this to work: they won't be fetched or installed by the script.
+
+We currently use [Test_Café_](https://testcafe.io/) to run the unit tests in headless browsers in GitHub actions. If you want to run this locally, you can find out which browsers it
+knows about by running `npx testcafe --list-browsers` (it may take some time to discover local browsers).
+
+When you run `npm test`, it will run the tests visually, not headless. The individual browser tests (e.g. `npm run test-unit-chrome`) are run headless. If you want to run these
+individual tests visually, not headless, it's easiest simply to open `tests/index.html` in the respective browser, and this allows you to debug. If you really want to do it from the
+commandline, then you'll need, e.g., `npx testcafe chrome ./tests/initTestCafe.js --app "http-server --silent -p 8080 ."` (adapt the browser as necessary).
+
+## End-to-end tests
+
+End-to-end (e2e) tests are implemented with Selenium WebDriver, which in turn uses Mocha as the testing framework. The tests can be found in the `tests` directory, and are
+implemented as ES6 modules. Each test consists of a runner for a specified browser (e.g. `microsoftEdge.e2e.runner.js`) which in turn imports one or more specification test suites
+(e.g. `legacy-ray_charles.e2e.spec.js`). The test suites load a specific ZOM archive, and undertake tests in both JQuery and ServiceWorker modes.
+
+These tests are run automatically on every push and pull requests by the GitHub Actions runner `CI.yml` (in the `.github/workflows` directory). They can also be run locally with the
+following procedure:
+
+* Ensure you have installed the dependencies (`npm install` in the project root);
+* Ensure you have built the app with source code, so you can more easily debug (`npm build-src`);
+* Ensure that no copy of `http-server` is currently running, because this will conflict with the test server that is launched with each test;
+* Run `npm run test-e2e-firefox` (or edge or chrome) to check that it's working;
+* If you have Microsoft Edge installed on Windows, you can run the tests in IE Mode with `npm run test-e2e-iemode`;
+* You can run `npm run test-e2e` to run in all of Firefox, Chrome, Edge and IE Mode at once, but you will need to ensure all three browsers are installed in standard locations,
+  and the IE Mode test will only work on Windows (Edge for Linux does not include this mode).
+
+The ZIM archive that is tested is also found in `tests`. In the case of `legacy-ray_charles.e2e.spec.js`, this is a legacy split ZIM archive that has XZ compression, so a useful test
+of that type of ZIM. We are looking to expand the tests to run also on a modern small ZIM with ZSTD compression and dynamic content. If you wish to develop tests for a new archive,
+be sure to create a new `e2e.spc.js` file that corresponds to that archive. It will be easiest to duplicate the existing legacy rach_charles test and change the name of your copy.
+To luanch your new tests, you will need to add them to each browser's runner as an import.
+
+Please note that we are currently using **Selenium** WebDriver, *not* WebDriver.io, which is a different (but related) project with subtly different test syntax.

tests/chromium.e2e.runner.js

@@ -1,6 +1,6 @@
 import { Builder } from 'selenium-webdriver';
 import { Options } from 'selenium-webdriver/chrome.js';
-import e2e from './kiwix-js.e2e.spec.js';
+import e2e from './legacy-ray_charles.e2e.spec.js';
 
 /* eslint-disable camelcase */
 

tests/firefox.e2e.runner.js

@@ -1,6 +1,6 @@
 import { Builder } from 'selenium-webdriver';
 import firefox from 'selenium-webdriver/firefox.js';
-import e2e from './kiwix-js.e2e.spec.js';
+import e2e from './legacy-ray_charles.e2e.spec.js';
 
 /* eslint-disable camelcase */
 

tests/ieMode.e2e.runner.js

@@ -1,6 +1,6 @@
 import { Builder } from 'selenium-webdriver';
 import { Options } from 'selenium-webdriver/ie.js';
-import e2e from './kiwix-js.e2e.spec.js';
+import e2e from './legacy-ray_charles.e2e.spec.js';
 
 /* eslint-disable camelcase */
 

tests/kiwix-js.e2e.spec.js → tests/legacy-ray_charles.e2e.spec.js

@@ -1,5 +1,5 @@
 /**
- * kiwix-js.e2e.spec.js : End-to-end tests implemented with Selenium WebDriver and Mocha
+ * legacy-ray_charles.e2e.spec.js : End-to-end tests implemented with Selenium WebDriver and Mocha
  *
  * Copyright 2023 Jaifroid and contributors
  * Licence GPL v3:

tests/microsoftEdge.e2e.runner.js

@@ -1,6 +1,6 @@
 import { Builder } from 'selenium-webdriver';
 import { Options } from 'selenium-webdriver/edge.js';
-import e2e from './kiwix-js.e2e.spec.js';
+import e2e from './legacy-ray_charles.e2e.spec.js';
 
 /* eslint-disable camelcase */
 

www/index.html

@@ -430,6 +430,8 @@ <h3 id="licence">Licence information</h3>
                         <li>The <a href="https://rollupjs.org/" target="_blank">Rollup JavaScript Bundler</a>, copyright by the contributors, released under an <a href="https://github.com/rollup/rollup/blob/master/LICENSE.md" target="_blank">MIT Licence</a></li>
                         <li><a href="https://vitejs.dev/" target="_blank">Vite Frontend Tooling</a>, copyright Yuxi You and Vite contributors, released under an <a href="https://github.com/vitejs/vite/blob/main/LICENSE" target="_blank">MIT Licence</a></li>
                         <li>The <a href="https://testcafe.io/" target="_blank">Test<i>Café</i></a> suite, copyright Developer Express Inc., released under an <a href="https://github.com/DevExpress/testcafe/blob/master/LICENSE" target="_blank">MIT Licence</a></li>
+                        <li><a href="https://www.selenium.dev/documentation/webdriver/" target="_blank">Selenium WebDriver</a>, copyright The Software Freedom Conservancy, released under an <a href="https://github.com/SeleniumHQ/selenium/blob/trunk/LICENSE" target="_blank">Apache Lincence v2</a></li>
+                        <li><a href="https://mochajs.org/" target="_blank">Mocha Testing Framework</a>, copyright The OpenJS Foundation and Mocha contributors, released under a <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank"> Creative Commons Attribution 4.0 International Licence</a></li>
                     </ul>
                     <p style="text-align: right"><a href="#contents">↑ Back to Contents</a></p>