Skip to content

Home

Jump to bottom
Yuanpeng Zhang edited this page Mar 16, 2023 · 4 revisions

CI/CD

The deployment on analysis clusters is conducted through Ansible playbooks mantidtotalscattering.yml and mantidtotalscattering-dev.yml

  • Certain version combo of tox, tox-conda and tox-gh-actions does not work together -- tox and tox-conda packages are installed with conda while tox-gh-actions is installed using pip. This is something we need to pay attention to in the future, but for the moment, the following version combo does work properly,

    tox => 3.27.1
tox-conda => 0.10.1
tox-gh-actions => 2.12.0

    The version of tox and tox-conda can be specified here while the version of tox-gh-actions is specified here. However, it seems that the requirements-dev.txt file will also be used at another separate stage of tox which will then read the requirements-dev.txt file and install all packages using pip. In this case, the installation will fail since pip cannot read the version definition like this tox=3.27.1. For the moment, it seems that specifying tox-gh-actions version in the github action YAML file is enough for solving all the version conflict issue.

    Refer to the link here for a typical error msg when it is not working with this regard.

Dev Notes

  1. As part of the pipeline, a dedicated routine for pre-calculating the absorption coefficient and constructing the grouping mechanism was developed for NOMAD. This routine will first access the ITEMS database to fetch the sample information. Then, it will automatically open the libreoffice software to load in the downloaded sample information csv file where users can edit the wrong information or fill in those missing ones. Then the program will perform the absorption coefficients calculation for all the pixels after which the grouping mechanism will be generated according to the similarity among the absorption coefficient spectra of all the detectors. Both the calculated absorption workspace (grouped, in an average manner, according to the generated grouping mechanism) and the generated grouping file will be saved to hard disk for later use. Also, a plain text file will be saved to include a list of reference spectra IDs for each of the identified group of pixels. The main program is a bash wrapper script which will execute several python scripts for performing the tasks mentioned above. On the top of the bash script, there are two lines containing the directory of those python scripts and the location of the central configuration file (for NOMAD, it is /SNS/NOM/shared/autoreduce/configs/mts_abs_pre_calc.json). The generated output files (i.e., the absorption coefficients workspace, the grouping file, and the plain text file, as mentioned above) will be found in the directory specified in the central configuration file (see the CacheDir entry, and specifically for NOMAD, it is /SNS/NOM/shared/mts_abs_ms). Each IPTS will have its own dedicated directory under the cache directory.

    The wrapper bash script and all the called python scripts can be found here.

  2. The mask is being applied within the Mantid AlignAndFocusPowder algorithm.

Clone this wiki locally