ReleaseNotes

Release notes for "Dtest" module

This file documents API, usage, portability etc. changes that have been
introduced in new versions of the "Dtest" module. This information should be
kept in mind when upgrading to newer versions of the module. This file may also
documment major bug fixes in so far as they may impact upgrade decisions. More
complete and detailed information on changes to the "Dtest" module can be found
in the ChangeLog file.

Release R1-15l (2024-10-08 08:27:09):

	https://dartsgitlab-internal.jpl.nasa.gov/darts-lab/jpl-internal/all-access/development/infrastructure/dtest/-/issues/11

Release R1-15k (2024-10-02 14:41:00):

	Updated GPU_MEM

Release R1-15j (2024-09-25 16:26:33):

	Let testing work when we can't import GpuMem (e.g. if we don't have
	cuda)

Release R1-15i (2024-09-17 15:49:08):

	Adding GPU memory to tests. See https://dartsgitlab-internal.jpl.nasa.gov/darts-lab/jpl-internal/all-access/development/infrastructure/dtest/-/issues/11.

Release R1-15h (2024-09-10 13:05:51):

	* Updating types in gpu_mem.cc
	* Bug fixes in locations we write GPU_MEM to.
	* See https://dartsgitlab-internal.jpl.nasa.gov/darts-lab/jpl-internal/all-access/development/infrastructure/dtest/-/issues/11

Release R1-15f (2024-09-09 11:38:49):

	https://dartsgitlab-internal.jpl.nasa.gov/darts-lab/jpl-internal/all-
	access/development/infrastructure/dtest/-/issues/11

Release R1-15e (2024-09-06 08:57:39):

	* Moving tags before RESOURCES.

Release R1-15a (2024-08-29 08:19:23):

	Tweak logic to avoid generating numpy bools from comparisons or
	returning a false positive on a numpy false value. See also:

	https://dartsgitlab-internal.jpl.nasa.gov/darts-lab/jpl-internal/all-access/development/infrastructure/dtest/-/issues/7

Release R1-15 (2024-08-17 08:41:59):

	* Now splitting with shlex and combinging args in sys.argv before
	parsing. Shlex is more robust and makes parsing args simpler. Combinging
	and parsing rather than parsing and combining means we don't have to try
	and sort out defaults before combining.

Release R1-14z (2024-08-13 12:34:20):

	* Calling logTee before we create the pool. We must modify values on
	Test before this happens, otherwise, it seems the workers don't pick the
	changes up.

Release R1-14y (2024-08-12 13:17:00):

	* Now using a shared variable to track GPU memory usage.
	* See https://dartsgitlab-internal.jpl.nasa.gov/darts-lab/jpl-internal/all-access/development/infrastructure/dtest/-/issues/9

	* Getting rid of quiet global variable. Cleaning up functions as a result.
	* Moving tests to Dtest args only.
	* See https://dartsgitlab-internal.jpl.nasa.gov/darts-lab/jpl-internal/all-access/development/infrastructure/dtest/-/issues/8

Release R1-14x (2024-08-10 13:38:59):

	* Bug fix. Should have been the intersection of two sets, not if both sets contain values.

Release R1-14w (2024-08-10 09:44:16):

	* Now processing extra arguments from SiteDefs that come through the Makefiles. This allows different exclude_tags, run_only_tags, and truth_suffix to each test.
	* Removing some global variables from Dtest that are now placed on args directly.
	* Fix the issue where tests were showing up relative to the sbox main directory rather than individual module directory.
	* See https://dartsgitlab-internal.jpl.nasa.gov/darts-lab/jpl-internal/all-access/development/infrastructure/dtest/-/issues/8

Release R1-14v (2024-08-09 09:44:27):

	* Improving exception handling so that dtest-sbox doesn't exit out completely if one module fails.
	* Updating tests.

Release R1-14u (2024-08-09 06:37:00):

	* Adding dtest-sbox.
	* Major refactor of dtest.
	* See https://dartsgitlab-internal.jpl.nasa.gov/darts-lab/jpl-internal/all-access/development/infrastructure/dtest/-/issues/8

Release R1-14t (2024-07-29 10:46:15):

	Apply patch from COMPASS. Removes some regtest .orig files because the
	comparison doesn't provide any info beyond whether running succeeded.
	Adds byte designation for python 3 in scripts/dtest-html.

Release R1-14s (2024-07-15 07:51:17):

	Adding CUI tag to test that require dlab3.

Release R1-14r (2024-07-01 11:32:57):

	Updating conf.py with new version of extensions. See https://dartsgitlab-internal.jpl.nasa.gov/darts-lab/jpl-internal/all-access/team/infrastructure/housekeeping/-/issues/22#note_22887/

Release R1-14n (2024-04-25 15:49:39):

	* Adding convenience method for renaming orig files to fc38 and creating
	new orig files. This will be used for the port to FC40.

Release R1-14g (2024-03-29 11:30:12):

	* Adding the ability to specify GPU_MEM in the RESOURCES field of a DTESTDEFS.cfg file.
	* See https://dartsgitlab-internal.jpl.nasa.gov/darts-lab/jpl-internal/all-access/development/infrastructure/dtest/-/issues/6

Release R1-14e (2024-02-27 11:48:29):

	Add the 'raises' context manager to check that a code block correctly
	raises an expected error:

	>>> arr = [1, 2, 3]
	>>> with raises(IndexError):
	...     arr[5]

	See the docstring for more details.

Release R1-14c (2024-01-08 14:14:51):

	* Updating to work with SOAEigen.

Release R1-14a (2023-11-07 14:15:55):

	Added critical failure symbol to last 10 string in emails.

Release R1-14 (2023-11-07 10:59:28):

	Added more info to emails and website.

Release R1-13w (2023-10-09 09:37:12):

	Emails no longer error out when only one entry is in HDF5 file.

Release R1-13u (2023-10-05 15:55:25):

	Last passed results only show for the same sandbox in emails.

Release R1-13c (2023-07-12 15:39:26):

	Made regtest HDF5 appends more robust.

Release R1-12x (2023-04-28 15:14:55):

	* Adding reg test for reg test webiste from Shay Ma's internship.

Release R1-12w (2023-02-23 07:09:17):

	* Adding missing return statement. Without this, the SERIAL tests get
	added to the serial and parallel groups and are run twice (once in
	serial and once in parallel).

Release R1-12v (2023-02-22 11:05:04):

	* Updating to include the logic for processing the SERIAL tag.
	* See https://dartsgitlab-internal.jpl.nasa.gov/darts-lab/jpl-internal/all-access/development/infrastructure/dtest/-/issues/5

Release R1-12t (2023-01-27 16:47:52):

	Added docker_regtest.py which can be used if `make regtest` does not
	work.

Release R1-12s (2022-07-24 08:48:23):

	Bugfix dtest-mail when total tests = 0

Release R1-12n (2022-05-27 09:02:50):

	Added Dtest version to reggtest email

Release R1-12m (2022-05-27 04:28:12):

	Fixed formatting in dtest-mail

Release R1-12k (2022-05-26 16:38:29):

	Updated genRegtestMail to reflect chhanges with previous regtest

Release R1-12j (2022-04-06 16:58:59):

	Add python_diff.py which is a simple diff written in python.

Release R1-12h (2021-11-17 13:18:05):

	regsummary now works correclty.

Release R1-12g (2021-07-28 11:42:52):

	Removed hard-coded path to tkdiff.

Release R1-11y (2020-05-30 18:13:47):

	Updated DCompareUtils.py to correctly compare nan values.

Release R1-11s (2020-01-30 17:18:30):

	Explictly runs tests in bash since test commands often assume bash. The
	old behavior was to use Python's default of sh. On fedora this is an
	alias to bash, so it still works, but other operating systems may break
	this assumption.

	Python3 compatability fixes

Release R1-11o (2019-12-02 12:38:35):

	Removed doxygen tag file that should not have been under version
	control.

Release R1-11m (2019-11-14 17:33:24):

	Slightly loosened up C++ kwstyle checking for class member variables.
	Acceptable member variables, may have a leading underscore, then the first
	character must be a lower case letter.  After that a combination of
	lower/upper-case letters, digits, and underscores may be used.

	Cleaned up code for some code checking regtests.

Release R1-11l (2019-11-04 09:06:43):

	Slightly relaxed member function name requirement; allow underscores in the
	function name.  Generally, member functions should be camelcase with the first
	letter lower case.  But in some cases, an underscore near the end can be
	helpful.

Release R1-11h (2019-09-11 08:46:06):

	Fix to unit test code to deal with unit tests improperly failing.

Release R1-11f (2019-08-26 11:46:38):

	Added codechecking tag to test_dtest_kwstyle.

Release R1-11d (2019-05-14 16:55:02):

	Updated array comparison to ignore extra spaces. Fixed bug in which
	arrays were not compared correctly.

Release R1-11c (2019-03-25 10:06:11):

	Updated DUnitTest regtest evaluation code to be more robust to variation in
	spaces between sites.

Release R1-11b (2019-03-11 17:56:27):

	Fixed regtests, some code cleanup.

Release R1-10y (2018-07-02 18:21:46):

	Added support for comparision utilities to ignore filenames with a list of
	site- specific list of prefixes.  The SiteDefs site-specific environment
	variable DTEST_IGNORE_PATH_PREFIXES is queried to get a list path prefixes
	that should be ignored by dtest file comparision utilities.  If this site var
	is not defined, it defaults to '/home'.

Release R1-10w (2018-05-14 17:41:40):

	Fixed pylint-no-sandbox. Reverted cmpCheckPoint file return value behavior for
	verbose mode.

Release R1-10v (2018-05-10 11:27:11):

	Updates to model comparision function to better handle paths, etc.
	Improved error messages for the checkpoint comparision functions.

Release R1-10u (2018-04-30 17:54:41):

	Added work-around for dtest hanging in Python 3.  This is a temporary fix that
	only affects running dtest in a Python3 sandbox.  The real issue still needs
	to be tracked down.

Release R1-10t (2018-04-20 14:03:28):

	Updated pylint test script to work on regtest server (dirac).

Release R1-10s (2018-04-19 16:08:21):

	Minor python3 compatability fix

Release R1-10r (2018-04-19 15:42:47):

	Minor changes to get PEP8 and pylint tests passing.

Release R1-10p (2018-03-29 16:24:10):

	Fixed issue with model output comparision checker.

Release R1-10m (2018-03-08 11:54:28):

	Cleaned up python code for code checking.

Release R1-10j (2018-02-26 12:34:43):

	Added/restructured regtests to verify correct behavior of TAGS, CHILD_TAGS,
	TAGS=skip, TAGS=quarantined.

Release R1-10d (2017-09-26 12:05:03):

	Updates to sphinx handling of math.

Release R1-10b (2017-06-08 13:31:17):

	Fixed dtest issue with setting child_tags environment variable with
	comma-separated list of tags.  Code cleanup for pylint.  Fixes/updates to
	sphinx docs.

Release R1-09y (2017-04-27 17:26:54):

	Various code cleanups.

Release R1-09u (2016-10-25 11:45:05):

	Updates from JSC: Split DUnit tests to support some tests not requiring SOA.
	Updated corresponding tests cases.

Release R1-09s (2016-05-02 18:40:38):

	Updates for Sphinx docs.

Release R1-09r (2016-04-18 15:28:43):

	Fix to make loop more robust (from Scott Nemeth at JSC).

Release R1-09q (2016-03-25 10:17:36):

	Minor sphinx docs fixes.

Release R1-09p (2016-03-24 14:45:55):

	Added post-processing to Sphinx docs to update links to download PDF files
	to open in the browser.

Release R1-09o (2016-03-24 14:19:23):

	Fixed issue with Dtest Sphinx docs being incorporated into the global
	docs.

Release R1-09n (2016-03-23 16:04:53):

	Added Dtest Tags presentation to the sphinx docs.  Search for 'Dtest
	Tags'.

Release R1-09m (2016-03-11 17:27:40):

	Fixed issue with CHILD_TAGS for dtest.

Release R1-09l (2016-03-11 15:37:37):

	Added support for CHILD_TAGS to allow child subdirectories to inherit a
	tag.

Release R1-09i (2015-12-15 09:30:26):

	Fix for 'srun dtest -c' on ssh-ed runs.

Release R1-09h (2015-12-10 14:48:27):

	Fixed truth suffix environment variables to be space or comma separated.

Release R1-09g (2015-12-10 10:52:13):

	Fixed issue with multiple tests for COMAPARE tests.

Release R1-09f (2015-12-09 16:40:30):

	Fixes for handling suffixes.

Release R1-09d (2015-12-09 14:57:16):

	Fixed problem with dtest --truth-suffix command line.

Release R1-09c (2015-12-09 14:05:19):

	Fix: --truth-suffix now adds to suffixes in DTESTDEFS files.

Release R1-09b (2015-12-09 12:03:29):

	Got dtest -c working again.

Release R1-09a (2015-12-08 17:13:15):

	Add regtest for giving DTEST additional truth suffixes via environment
	variables.

Release R1-09 (2015-12-08 16:33:09):

	Support additional dtest truth suffixes from DTEST_TRUTH_SUFFIX environment
	variable.

Release R1-08z (2015-12-08 15:38:58):

	Small improvements in truth suffix handling.

Release R1-08y (2015-12-08 15:10:26):

	Fix issue with COMPARE tests.

Release R1-08x (2015-12-08 10:21:05):

	Updated to support multiple truth suffixes.

Release R1-08w (2015-10-12 16:36:07):

	Added support for executing/excluding tests based on TAGS defined in the
	DTESTDEFS file in the same directory as the regtest.  From the sphinx
	docs, if you run dtest with the --exclude-tags option:

	      $ Drun dtest --exclude-tags logging

	then any regtest that defines TAGS to include 'logging' in its DTESTDEFS
	file will be excluded (skipped).

	You may also run only regtest that contain a specific tag:

	      $ <Drun> dtest --run-only-tags validation

	then only regtests that define TAGS to include 'validation' in its
	DTESTDEFS file will be executed.  Note that all regtests that are marked
	for skipping or quarantining, will not be run.

	Also, there are two special tags: 'skip' and 'quarantined'.  These do the
	same things as listing the regtests in parent DTESTDEFS file 'SKIP' or
	'QURANTINED' lists (respectivley).

	One advantage in using tagging to skip or quarantine a regtest is that
	skip/exclusion flag goes into the same directory as the regtest itself
	(not its parent).

Release R1-08v (2015-10-06 17:12:44):

	Added command line options for blacklist and exclude to dtest-kwstyle.

Release R1-08u (2015-09-23 11:02:30):

	Integrated DUnit, a testing framework derived from python's unittest. DUnit
	adds support for operations that we need such as comparing quaternions, etc.

Release R1-08i (2014-09-29 10:58:07):

	Support detecting comparison failure in unittest and doctest

Release R1-07z (2014-08-20 13:57:40):

	Updated sphinx docs.

Release R1-07u (2014-04-03 13:40:12):

	This is just a test of pyam's long message wrapping. Let us see if this
	works properly when used interactively.

Release R1-07t (2014-04-03 13:38:05):

	This is just a test of pyam's long message wrapping. Let us see if this
	works properly when used interactively.

Release R1-07b (2013-08-26 14:23:11):

	Avoid hanging on standard in

Release R1-06s (2013-07-22 09:07:14):

	Add "--prerequisite-imports"

Release R1-06r (2013-07-16 10:22:09):

	Show timeout value in report

Release R1-06f (2013-06-13 16:23:01):

	-

Release R1-06e (2013-06-13 15:12:14):

	Split on commas too in dtest-numerical-diff.

Release R1-06b (2013-06-07 17:54:24):

	Improve usability of dtest-numerical-diff

Release R1-05x (2013-05-28 16:52:24):

	Move testing tool from pyam to here

Release R1-05u (2013-04-25 17:59:49):

	Updated DCompareUtils.py to deal with checkpoint files that have no models.

Release R1-05o (2013-03-27 16:21:46):

	Print nice error messages instead of spitting out that stack.

Release R1-05l (2013-03-19 14:58:25):

	Add dtest-numerical-diff

Release R1-05i (2013-03-10 11:03:17):

	Add Python 3 support

Release R1-04y (2012-12-21 19:19:43):

	Highlight in red run-time errors in "failed tests" list

	This helps the user distinguish between comparison failures and run-time
	failures after running the tests in multiprocess mode.

Release R1-04r (2012-12-05 11:21:08):

	A couple of improvements to genRegtestMail and genRegtestHtml scripts from
	Scott Nemeth (JSC).  Updated regtests appropriately.

Release R1-04q (2012-12-04 16:28:02):

	Update to sphinx docs (fixed label).

Release R1-04o (2012-11-30 14:32:32):

	Replace redundant column with something useful

Release R1-04l (2012-11-15 09:22:29):

	Fixed lists for failed tests in genRegtestHtml.

Release R1-04g (2012-10-25 10:41:53):

	Make filtering out non-failing test emails easier

	The test status is appended to the email title.

Release R1-04d (2012-10-15 15:55:16):

	Add time taken to email test report

Release R1-04 (2012-09-20 13:31:09):

	Minor documentation fixes.

Release R1-03s (2012-08-09 17:02:08):

	Avoid clobbering another dtest instance

	This is done via lock file.

Release R1-03n (2012-07-21 11:12:23):

	Reorganize so that second table in email is alphabetical

Release R1-03j (2012-07-06 08:26:22):

	Add --ignore option

Release R1-03i (2012-07-05 08:50:01):

	Allow configuration filename to be "DTESTDEFS.cfg" for syntax
	highlighting. And implement the environment variable setting stuff
	correctly.

Release R1-03d (2012-07-01 15:29:28):

	Add optional support for emailing on failure

Release R1-03c (2012-06-28 21:41:34):

	Start Xorg if it is down yet DISPLAY is set

Release R1-03 (2012-06-25 12:34:48):
	Show status of DISPLAY

Release R1-02z (2012-06-25 08:27:54):
	Print test run time

Release R1-02y (2012-06-04 22:00:37):
	Make control-c handling kill processes faster

Release R1-02w (2012-06-01 11:09:58):
	Add "--fail-fast" and improve multiprocess output

Release R1-02s (2012-05-03 09:12:36):
	Converted nummarray references to numpy.

Release R1-02r (2012-04-28 22:48:16):
	Correctly filter "/home/"-prefixed strings in DCompareUtils

	This fixes the flip-flopping RoverVehiclesTest tests

Release R1-02n (2012-01-04 14:16:35):
	Add support back for running tests listed on the command-line

Release R1-02m (2011-12-23 20:23:29):
	Fix bug that causes DELETE not to work.

Release R1-02i (2011-12-12 22:12:38):
	Change order of modules failed test listing.

Release R1-02f (2011-12-10 18:57:45):
	Add support for reporting timed out tests.

Release R1-02c (2011-12-09 22:20:46):
	Improve separation of concerns.

Release R1-01w (2011-12-07 20:18:09):
	Fix the bug where no options are passed when run in parallel mode. This
	is accomplished by switching out the Make-based parallel processing with
	normal Python function calls (multiprocessing). This allows for running
	Python coverage in parallel. Also fix the problem where --dir doesn't
	work properly when not run from inside the module.

Release R1-01r (Mon Oct 17 13:54:49 2011):
	Tracing yam with DBI_TRACE.

Release R1-01q (Sat Oct 15 16:31:38 2011):
	Testing saving a module checked out with a version of pyam that appends
	to the "branches" list.

Release R1-01p (Wed Oct 12 19:47:01 2011):
	Testing saving a module checked out using pyam in a sandbox created by
	pyam.

Release R1-01j (Tue Sep 13 10:56:04 2011):
	Fixed coveragepy commadn transformation that can occur.

Release R1-01g (Fri Sep  9 12:28:10 2011):
	Added Valgrind memcheck support to dtest.

	Got rid of hacky coveragepy code. Replaced it with a more general test-
	command modifier feature.
	Added a --test-command-modifier which currently takes either
	'coveragepy' or 'memcheck' as argument.

Release R1-01e (Tue Aug  2 15:09:17 2011):
	Fixed issue with regupdate that it was ignoring emtpy files.

Release R1-01d (Tue Aug  2 14:22:16 2011):
	Fixed problem that dtest was not finding the top test directory
	(ROOTDIR) correctly.  This was revealed by the cmp.prg fix in
	the previous release.

Release R1-01c (Tue Jul 26  9:27:05 2011):
	cmp.prg will now throw an error when passed a non-existent filter file.

Release R1-01 (Mon Jul 11 15:21:02 2011):
	Fixed output/log issue (was skipping correct quarantined item but
	printing wrong name for it).

Release R1-00z (Mon Jun 27  8:55:23 2011):
	Fixed computation of total number failed that was not computed
	correctly due to the changes in the last release.

Release R1-00x (Fri Jun 24 10:33:21 2011):
	Updated email message script to show number of tests and pass rate at
	the top.

Release R1-00w (Thu Jun 23 15:15:12 2011):
	Fixed bug that deleted all files in test directory when running
	regtests with 'dtest' when 'DELETE =' is used in DTESTDEFS files (no
	value after equals sign).

Release R1-00v (Tue Jun 21 19:03:11 2011):
	Allow correct processing of truth file with alternative suffix.
        Documented testing utilities code and revised the Sphinx docs.

Release R1-00u (Tue Jun 21 10:29:59 2011):
	Documentation fixes for Dtest, including spelling corrections in the
        standalone executables.

Release R1-00t (Mon Jun 20 18:13:10 2011):
	Added --wholeFile option to compareDictsInFiles.

Release R1-00s (Mon Jun 20 15:37:16 2011):
	Added some documentation.

Release R1-00q (Fri Jun 17 12:52:29 2011):
	Added a documentation page for testing utilities and revised
        text on the Dtest overview page.  Also fixed a spelling error
        in the regtest mail generation script.

Release R1-00o (Thu Jun 16 17:08:28 2011):
	Moved various regtest-related utilities to this module and
	added a few regtests.

Release R1-00n (Thu Jun 16 16:42:52 2011):
	Separated the killableprocess implementation from dtest into a
	standalone Python script, and merged it with the Dutils lib.

Release R1-00j (Wed Jun 15  9:37:08 2011):
	Fixed checkpoint comparison utility to recognize changes in syntax
	due to recent DVar refactor.  Moved regupdate from dev-utils to this
	module.

Release R1-00i (Mon Jun  6 15:12:03 2011):
	Added skeleton documentation to Dtest.

Release R1-00h (Mon Jun  6 13:53:39 2011):
	Fixed oversight in truth suffix code.

Release R1-00g (Mon Jun  6 12:15:18 2011):
	Added fallback to .orig even if other truth suffix is given.
	If both are available, it will use the specified truth suffix.
	If the specified suffix is not available, it will use .orig.
	Renamed test directories to use underscores.

Release R1-00e (Mon Jun  6 11:12:31 2011):
	Added missing file5.txt in doctest.

Release R1-00d (Mon Jun  6 10:53:02 2011):
	Added speicifying eps to comparison programs.

Release R1-00b (Thu Jun  2 18:00:15 2011):
	Added compareDictsInFiles utility to compare a python dictionary in
	two different files.   The dictionary should be marked by distinctive
	user-definable textual markers before and after the dictionary.  Try:

	    compareDictsInFiles --help

	to see the usage.

Release R1-00a (Thu Jun  2 15:30:25 2011):
	Moved in DCompareUtils.py, Ddoctest.py, and related regtests (from
	DshellEnv).  Moved in dtest, compareCheckpointFiles, and
	compareModelFiles (from dev-utils).

Release R1-00 (Thu Jun  2 14:31:55 2011):
	Created.