From f6f181215776743c9115c3da7bea2eb71c43261e Mon Sep 17 00:00:00 2001 From: escobar Date: Fri, 30 Jan 2015 15:47:36 +0100 Subject: [PATCH 01/69] added example of dummy toolchain for dependencies --- docs/Writing_easyconfig_files.rst | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/Writing_easyconfig_files.rst b/docs/Writing_easyconfig_files.rst index 7bb859e8..f56ef464 100644 --- a/docs/Writing_easyconfig_files.rst +++ b/docs/Writing_easyconfig_files.rst @@ -160,7 +160,16 @@ Example: builddependencies = [('CMake', '2.8.12', '', ('GCC', '4.8.2')] .. note:: By default, EasyBuild will try to resolve dependencies using the same toolchain as specified for the software being installed. - Exceptions can be specified on a per-dependency level (cfr. the ``CMake`` build dependency in the example). + Exceptions can be specified on a per-dependency level (cfr. the ``CMake`` build dependency in the example above). + +You can also define a dependency which uses the dummy toolchain: + +.. code:: python + + name = 'GTI' + ... + toolchain = {'name': 'goolf', 'version': '1.5.14'} + dependencies = [('PnMPI', '1.2.0', '', ('dummy', ''))] .. _configure_build_install_command_options: From 7a7228b2e1a2b8722318f358104d69cf919a893a Mon Sep 17 00:00:00 2001 From: Ward Poelmans Date: Sat, 7 Mar 2015 14:39:34 +0100 Subject: [PATCH 02/69] Added the clean-gists.py script to the usefull scripts list --- docs/Useful-scripts.rst | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/Useful-scripts.rst b/docs/Useful-scripts.rst index 7b954e01..31dc07f7 100644 --- a/docs/Useful-scripts.rst +++ b/docs/Useful-scripts.rst @@ -81,3 +81,17 @@ This results in ``broken.eb`` being rewritten/fixed, after creating a backup cop A script to set up an EasyBuild development environment. For more information, see :ref:`install_latest_develop_using_script`. + +.. _clean_EasyBuild_gists: + +``clean_gists.py`` +-------------------------------- + +*download from:* https://github.com/hpcugent/easybuild-framework/blob/master/easybuild/scripts/clean_gists.py + +When you use the ``--from-pr`` with ``-upload-test-report`` (see https://github.com/hpcugent/easybuild/wiki/Review-process-for-contributions#automated-testing-of-easyconfigs-pull-requests) option, your Github gist account can become cluttered with test reports. The ``clean_gists.py`` is here to help with that: it has serveral options to clean old test reports: +* ``clean_gists.py -p``: delete all gists from closed pull-requests. This is the default action. +* ``clean_gists.py -a``: delete all gists generated by Easybuild. +* ``clean_gists.py -o``: delete all gists without a pull-request. This happens when you use ``-upload-test-report`` without ``--from-pr``. + +By default, the script will use the same GitHub account that Easybuild uses. If you want a different account, you can use the ``-g`` option to specify a different username. The script will expect that a valid token for the username is present. The procedure on how to do that can be found at https://github.com/hpcugent/easybuild/wiki/Review-process-for-contributions#setting-things-up. From d574834cfd0644e3a054b6a99407be5800ed97be Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 9 Mar 2015 18:44:32 +0100 Subject: [PATCH 03/69] tweak documentation by @wpoely86 on clean_gists.py script --- docs/Useful-scripts.rst | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/docs/Useful-scripts.rst b/docs/Useful-scripts.rst index 31dc07f7..9426e610 100644 --- a/docs/Useful-scripts.rst +++ b/docs/Useful-scripts.rst @@ -82,16 +82,23 @@ This results in ``broken.eb`` being rewritten/fixed, after creating a backup cop A script to set up an EasyBuild development environment. For more information, see :ref:`install_latest_develop_using_script`. -.. _clean_EasyBuild_gists: +.. _clean_gists_script: ``clean_gists.py`` -------------------------------- *download from:* https://github.com/hpcugent/easybuild-framework/blob/master/easybuild/scripts/clean_gists.py -When you use the ``--from-pr`` with ``-upload-test-report`` (see https://github.com/hpcugent/easybuild/wiki/Review-process-for-contributions#automated-testing-of-easyconfigs-pull-requests) option, your Github gist account can become cluttered with test reports. The ``clean_gists.py`` is here to help with that: it has serveral options to clean old test reports: -* ``clean_gists.py -p``: delete all gists from closed pull-requests. This is the default action. -* ``clean_gists.py -a``: delete all gists generated by Easybuild. -* ``clean_gists.py -o``: delete all gists without a pull-request. This happens when you use ``-upload-test-report`` without ``--from-pr``. +When using ``--from-pr`` in combination with ``-upload-test-report`` (see +https://github.com/hpcugent/easybuild/wiki/Review-process-for-contributions#automated-testing-of-easyconfigs-pull-requests), +you can end up with a bunch of gists in your GitHub account containing test reports, that may no longer be relevant. -By default, the script will use the same GitHub account that Easybuild uses. If you want a different account, you can use the ``-g`` option to specify a different username. The script will expect that a valid token for the username is present. The procedure on how to do that can be found at https://github.com/hpcugent/easybuild/wiki/Review-process-for-contributions#setting-things-up. +To help with that the ``clean_gists.py`` script is available, to clean up gists containing test reports: + +* ``clean_gists.py -p``: delete all gists from closed pull requests (default action if no other action is specified) +* ``clean_gists.py -a``: delete all gists generated by Easybuild +* ``clean_gists.py -o``: delete all gists without a matching pull request (created by using ``-upload-test-report`` without ``--from-pr``) + +By default, the script will use the same GitHub account that Easybuild uses (see ``--github-user``); to specify a different GitHub account, use ``-g``. + +The script expects that a valid GitHub token for the used GitHub account username is available, see ref:`github_token`. From 0a9b5cf7db2761d5d87f55994e22c1ccef7e5bbe Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Tue, 10 Mar 2015 09:41:36 +0100 Subject: [PATCH 04/69] mention support for downloading over a proxy, tweak sections w.r.t. dummy toolchain and dependencies --- docs/Writing_easyconfig_files.rst | 45 ++++++++++++++++++++++++------- 1 file changed, 36 insertions(+), 9 deletions(-) diff --git a/docs/Writing_easyconfig_files.rst b/docs/Writing_easyconfig_files.rst index f56ef464..1ad7fc47 100644 --- a/docs/Writing_easyconfig_files.rst +++ b/docs/Writing_easyconfig_files.rst @@ -25,8 +25,8 @@ It consists of a plain text file (in Python syntax) with mostly `key-value` assi Easyconfigs typically follow a (fixed) strict naming scheme, i.e. ``-[-][-].eb``. -The toolchain label (which includes the toolchain name and version) is omitted when a ``dummy`` toolchain is used -(see also :ref:`dummy_toolchain`); the version suffix is omitted when it's empty. +The toolchain label (which includes the toolchain name and version) is omitted when a :ref:`dummy_toolchain` is used; +the version suffix is omitted when it's empty. .. note:: the filename of an easyconfig is only important w.r.t. dependency resolution (``--robot``), see :ref:`use_robot`. @@ -104,6 +104,8 @@ Common easyconfig parameters This section includes an overview of some commonly used (optional) easyconfig parameters. +.. _common_easyconfig_param_sources: + Source files and patches ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -114,6 +116,8 @@ Source files and patches Remarks: * sources are downloaded (best effort), unless already available +* proxy settings are taken into account, since the `urllib2 Python package `_ + is used for downloading (since EasyBuild v2.0) * patches need to be EasyBuild-compatible * unified diff format (``diff -ru``) @@ -135,6 +139,8 @@ Example: .. note:: Rather than hardcoding the version (and name) in the list of sources, a string template `%(version)s` can be used, see also :ref:`easyconfig_param_templates`. +.. _dependency_specs: + Dependencies ~~~~~~~~~~~~ @@ -159,18 +165,39 @@ Example: dependencies = [('PnMPI', '1.2.0')] builddependencies = [('CMake', '2.8.12', '', ('GCC', '4.8.2')] +For each of the specified (build) dependencies, the corresponding module will be loaded in the build environment +defined by EasyBuild. For the *runtime* dependencies, ``module load`` statements will be included in the generated +module file. + .. note:: By default, EasyBuild will try to resolve dependencies using the same toolchain as specified for the software being installed. - Exceptions can be specified on a per-dependency level (cfr. the ``CMake`` build dependency in the example above). + A different toolchain can be specified on a per-dependency level (cfr. the ``CMake`` build dependency in the example above). -You can also define a dependency which uses the dummy toolchain: +Loading of modules for dependencies with a ``dummy`` toolchain +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When a :ref:`dummy_toolchain>` is used, EasyBuild will only load the modules for each of the (build) +dependencies when an *empty* string is used as a toolchain version, i.e. :: + + toolchain = {'name': 'dummy', 'version': ''} + +When specifying a non-empty string as version for the ``dummy`` toolchain (e.g., ``dummy``), +modules for the (build) dependencies will *not* be loaded in the build environment as defined by EasyBuild. +Load statements for the runtime dependencies will still be included in the generated module file, however. + +Specifying dependencies using ``dummy`` toolchain +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To make EasyBuild resolve a dependency using the ``dummy`` toolchain, either specify '``dummy``' as toolchain name +in the tuple representing the dependency specification, or simply use ``True`` as 4th value in the tuple. + +For example, to specify PnMPI version 1.2.0 built with the ``dummy`` toolchain as a (runtime) dependency:: -.. code:: python - - name = 'GTI' - ... - toolchain = {'name': 'goolf', 'version': '1.5.14'} dependencies = [('PnMPI', '1.2.0', '', ('dummy', ''))] +which is equivalent to:: + + dependencies = [('PnMPI', '1.2.0', '', True)] + .. _configure_build_install_command_options: Configure/build/install command options From 48ce8c38469bc7f7c94972350b7139a77aef17fa Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Tue, 10 Mar 2015 09:41:40 +0100 Subject: [PATCH 05/69] prettify section name for dummy toolchain section in Concepts page --- docs/Concepts_and_Terminology.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Concepts_and_Terminology.rst b/docs/Concepts_and_Terminology.rst index 2daf443e..cfdb59d9 100644 --- a/docs/Concepts_and_Terminology.rst +++ b/docs/Concepts_and_Terminology.rst @@ -113,8 +113,8 @@ Recent releases of EasyBuild include out-of-the-box toolchain support for: .. _dummy_toolchain: -Dummy toolchain -~~~~~~~~~~~~~~~ +``dummy`` toolchain +~~~~~~~~~~~~~~~~~~~ The ``dummy`` toolchain is a special case. It is an `empty` toolchain, i.e. a toolchain without any components, and corresponds to using the readily available compilers and libraries (e.g., the ones provided by the OS, or From 7696f4a81e6ef56d6e677a9133e444b5e7e03f64 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Tue, 10 Mar 2015 09:41:51 +0100 Subject: [PATCH 06/69] bump release, update changelog --- docs/Changelog.rst | 5 +++++ docs/conf.py | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/Changelog.rst b/docs/Changelog.rst index 89c656af..583142d2 100644 --- a/docs/Changelog.rst +++ b/docs/Changelog.rst @@ -3,6 +3,11 @@ Changelog for EasyBuild documentation ------------------------------------- +* **release 20150310.01** (`Mar 10th 2015`): + + * document peculiarties w.r.t. dependencies in combination with a ``dummy`` toolchain (see :ref:`dependency_specs`) + * document ``clean_gists.py`` script (see :ref:`clean_gists_script`) + * mention taking into account of proxy settings for downloading sources (see :ref:`common_easyconfig_param_sources`) * **release 20150306.03** (`Mar 6th 2015`): add release notes for EasyBuild v2.0.0 (see :ref:`release_notes`) * **release 20150306.02** (`Mar 6th 2015`): diff --git a/docs/conf.py b/docs/conf.py index f5bc3630..38e3e15d 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -43,7 +43,7 @@ # The short X.Y version. version = '2.0.0' # this is meant to reference the version of EasyBuild # The full version, including alpha/beta/rc tags. -release = '20150306.03' # this is meant to reference the version of the documentation itself +release = '20150310.01' # this is meant to reference the version of the documentation itself # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: From a1077b8684c85adcfc42f48661753cacf20bcb41 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Tue, 10 Mar 2015 09:53:42 +0100 Subject: [PATCH 07/69] fix typo --- docs/Writing_easyconfig_files.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Writing_easyconfig_files.rst b/docs/Writing_easyconfig_files.rst index 1ad7fc47..812aca28 100644 --- a/docs/Writing_easyconfig_files.rst +++ b/docs/Writing_easyconfig_files.rst @@ -175,7 +175,7 @@ module file. Loading of modules for dependencies with a ``dummy`` toolchain ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -When a :ref:`dummy_toolchain>` is used, EasyBuild will only load the modules for each of the (build) +When a :ref:`dummy_toolchain` is used, EasyBuild will only load the modules for each of the (build) dependencies when an *empty* string is used as a toolchain version, i.e. :: toolchain = {'name': 'dummy', 'version': ''} From dfd73fa98fd07be108498bf06e3f4fc725311cb9 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Tue, 10 Mar 2015 09:54:40 +0100 Subject: [PATCH 08/69] add link to dummy toolchain section --- docs/Writing_easyconfig_files.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Writing_easyconfig_files.rst b/docs/Writing_easyconfig_files.rst index 812aca28..1d2b3769 100644 --- a/docs/Writing_easyconfig_files.rst +++ b/docs/Writing_easyconfig_files.rst @@ -180,7 +180,7 @@ dependencies when an *empty* string is used as a toolchain version, i.e. :: toolchain = {'name': 'dummy', 'version': ''} -When specifying a non-empty string as version for the ``dummy`` toolchain (e.g., ``dummy``), +When specifying a non-empty string as version for the :ref:`dummy_toolchain` (e.g., ``dummy``), modules for the (build) dependencies will *not* be loaded in the build environment as defined by EasyBuild. Load statements for the runtime dependencies will still be included in the generated module file, however. From de9778cc35210333993d33a49e66240035332161 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Tue, 10 Mar 2015 18:04:57 +0100 Subject: [PATCH 09/69] fix ref --- docs/Useful-scripts.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Useful-scripts.rst b/docs/Useful-scripts.rst index 9426e610..0fc1a879 100644 --- a/docs/Useful-scripts.rst +++ b/docs/Useful-scripts.rst @@ -101,4 +101,4 @@ To help with that the ``clean_gists.py`` script is available, to clean up gists By default, the script will use the same GitHub account that Easybuild uses (see ``--github-user``); to specify a different GitHub account, use ``-g``. -The script expects that a valid GitHub token for the used GitHub account username is available, see ref:`github_token`. +The script expects that a valid GitHub token for the used GitHub account username is available, see :ref:`github_token`. From 67bf8a59a90fd4ad6e847df1a467c1f8bf900c12 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Tue, 10 Mar 2015 18:10:26 +0100 Subject: [PATCH 10/69] punctuation --- docs/Writing_easyconfig_files.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Writing_easyconfig_files.rst b/docs/Writing_easyconfig_files.rst index 1d2b3769..3a0cc36d 100644 --- a/docs/Writing_easyconfig_files.rst +++ b/docs/Writing_easyconfig_files.rst @@ -25,8 +25,8 @@ It consists of a plain text file (in Python syntax) with mostly `key-value` assi Easyconfigs typically follow a (fixed) strict naming scheme, i.e. ``-[-][-].eb``. -The toolchain label (which includes the toolchain name and version) is omitted when a :ref:`dummy_toolchain` is used; -the version suffix is omitted when it's empty. +The toolchain label (which includes the toolchain name and version) is omitted when a :ref:`dummy_toolchain` is used. +The version suffix is omitted when it is empty. .. note:: the filename of an easyconfig is only important w.r.t. dependency resolution (``--robot``), see :ref:`use_robot`. From b9c1cc64e8c5c0444255013f2f3b6a71fb903406 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Tue, 10 Mar 2015 21:52:44 +0100 Subject: [PATCH 11/69] prettify formatting --- docs/Writing_easyconfig_files.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Writing_easyconfig_files.rst b/docs/Writing_easyconfig_files.rst index 3a0cc36d..7dcd8bfe 100644 --- a/docs/Writing_easyconfig_files.rst +++ b/docs/Writing_easyconfig_files.rst @@ -25,8 +25,8 @@ It consists of a plain text file (in Python syntax) with mostly `key-value` assi Easyconfigs typically follow a (fixed) strict naming scheme, i.e. ``-[-][-].eb``. -The toolchain label (which includes the toolchain name and version) is omitted when a :ref:`dummy_toolchain` is used. -The version suffix is omitted when it is empty. +The ``-`` label (which includes the toolchain name and version) is omitted when a :ref:`dummy_toolchain` is used. +The ``-`` label is omitted when the version suffix is empty. .. note:: the filename of an easyconfig is only important w.r.t. dependency resolution (``--robot``), see :ref:`use_robot`. From d72ceab3657b6ceb049d422e06b935dcf6cc0c8a Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Wed, 11 Mar 2015 21:00:48 +0100 Subject: [PATCH 12/69] enhance documentation w.r.t. cfgfile templates --- docs/Configuration.rst | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/docs/Configuration.rst b/docs/Configuration.rst index b966d154..61e31e94 100644 --- a/docs/Configuration.rst +++ b/docs/Configuration.rst @@ -156,20 +156,29 @@ An example configuration file that should make everything clear is shown below. Templates and constants supported in configuration files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -For each configuration option, a matching template is available to refer to its corresponding value. +Two types of template values are supported in configuration files: -A couple of predefined constants are available to be used (only) in EasyBuild configuration files. -An overview of supported constants is available via ``eb --avail-cfgfile-constants``. +* for configuration options defined in the configuration file (and only those) + * *syntax:* ``%(opt)s``, i.e., using the (lowercase) name of the configuration option +* for the default value of selected configuration options (see ``eb --avail-cfgfile-constants``) + * *syntax:* ``%(DEFAULT_OPT)``, i.e., using the uppercase name of the configuration option and prefixed with ``DEFAULT_`` -For both templates and constants, the syntax format is ``%(template_or_constant_name)s``. +.. note:: + These template values are only supported in configuration files, *not* in environment variable values or + command line option values. + +.. note:: + Using a template value for a configuration option that was not defined in the configuration file + will result in an error like ``ConfigParser.InterpolationMissingOptionError: Bad value substitution``. Example +++++++ -To include both the easyconfigs archive repository and the default list of robot search paths in the -active robot search path, the following configuration file entry can be used, featuring the template for the -``repositorypath`` configuration option and the provided ``DEFAULT_ROBOT_PATHS`` constant:: +To include both the (custom) location for the easyconfigs archive repository and the default list of robot search +paths in the active robot search path, the following configuration file entry can be used, featuring the template +for the ``repositorypath`` configuration option and the provided ``DEFAULT_ROBOT_PATHS`` constant:: + repositorypath = /home/example/easybuild/easyconfigs_archive robot-paths = %(repositorypath)s:%(DEFAULT_ROBOT_PATHS)s See also :ref:`controlling_robot_search_path`. From 27b2684671cd669203cb5b0259fd70d6adcceabb Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Wed, 11 Mar 2015 21:01:29 +0100 Subject: [PATCH 13/69] enhance documentation w.r.t. --robot and --robot-paths --- docs/Using_the_EasyBuild_command_line.rst | 38 ++++++++++++++++++----- 1 file changed, 31 insertions(+), 7 deletions(-) diff --git a/docs/Using_the_EasyBuild_command_line.rst b/docs/Using_the_EasyBuild_command_line.rst index d9d1cc2b..fdab8e70 100644 --- a/docs/Using_the_EasyBuild_command_line.rst +++ b/docs/Using_the_EasyBuild_command_line.rst @@ -426,23 +426,47 @@ part of the robot search path (in the order they are encountered). Controlling the robot search path ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -To control the robot search path, you can specify a (colon-separated list of) path(s) to ``--robot`` and/or +To control the robot search path, you can specify a (colon-separated list of) path(s) to ``--robot``/``-r`` and/or ``--robot-paths`` (or, equivalently, ``$EASYBUILD_ROBOT`` and/or ``$EASYBUILD_ROBOT_PATHS``):: eb --robot=$PWD:$HOME ... -Paths specified to ``--robot`` get precedence over paths specified via ``--robot-paths``. -Only ``--robot`` enables the dependency resolution mechanism; ``--robot-paths`` does not. +These two configuration options each serve a particular purpose: -By combining ``--robot`` and ``--robot-paths`` on the different configuration levels (see also :ref:`configuration_types`), you have full control over the robot search path. +* ``--robot``, ``-r``: + * intended to be used (only) as a command line option to ``eb`` (although it can also be defined through another + configuration type see :ref:`configuration_types`) + * enables the dependency resolution mechanism (disabled by default) + * optionally, a list of paths can be specified, which is *prepended* to the robot search path + * by default, the corresponding list of paths is *empty* +* ``--robot-paths``: + * intended to be defined in an EasyBuild configuration file, or via ``$EASYBUILD_ROBOT_PATHS`` + * does *not* enable the dependency resolution mechanism + * the specified list of paths is *appended* to the robot search path + * by default, the corresponding list of paths only contains the path to the easyconfig files which + are part of the EasyBuild installation + +For both options, the list of paths should be specified as a colon-separated (``:``) list. + +By combining ``--robot`` and ``--robot-paths`` using the different configuration types (see also +:ref:`configuration_types`), you have full control over the robot search path. A constant named ``DEFAULT_ROBOT_PATHS`` is available that can be used (only) in EasyBuild configuration files to refer to the default robot search path. For more information on using constants in EasyBuild configuration files, see :ref:`configuration_file_templates_constants`. -.. note:: The paths specified on the configuration type with the highest order of preference `replace` any paths - specified otherwise, i.e. values are not retained across configuration types. For more information - see :ref:`configuration_types`. +.. tip:: + Only use ``--robot`` to enable the dependency resolution mechanism; define ``robot-paths`` in your EasyBuild + configuration file or via ``$EASYBUILD_ROBOT_PATHS`` to specify which sets of easyconfig files EasyBuild + should consider, and in which order. By means of exception, a path can be specified to ``--robot`` to give a specific + set of easyconfig files precedence over others, for example when testing modified easyconfig files. + +.. note:: The paths specified on the configuration type with the highest order of preference *replace* any paths + specified otherwise, i.e. values are not retained across configuration types. That is: ``--robot`` *overrides* + the value in ``$EASYBUILD_ROBOT``, ``$EASYBUILD_ROBOT_PATHS`` *overrides* the ``robot-paths`` specification in an + EasyBuild configuration file, etc. Of course, the value specified for ``--robot`` does not directly affect the value + specified for ``--robot-paths``, on any configuration level, and vice versa. For more information on the relation + between the different configuration types, see :ref:`configuration_types`. Example use case ++++++++++++++++ From 5cadf357641c0a97d9c26fef9d4f528cdbfd2459 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Wed, 11 Mar 2015 21:07:26 +0100 Subject: [PATCH 14/69] fix formatting issues --- docs/Using_the_EasyBuild_command_line.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/Using_the_EasyBuild_command_line.rst b/docs/Using_the_EasyBuild_command_line.rst index fdab8e70..2a8b085e 100644 --- a/docs/Using_the_EasyBuild_command_line.rst +++ b/docs/Using_the_EasyBuild_command_line.rst @@ -434,12 +434,15 @@ To control the robot search path, you can specify a (colon-separated list of) pa These two configuration options each serve a particular purpose: * ``--robot``, ``-r``: + * intended to be used (only) as a command line option to ``eb`` (although it can also be defined through another configuration type see :ref:`configuration_types`) * enables the dependency resolution mechanism (disabled by default) * optionally, a list of paths can be specified, which is *prepended* to the robot search path * by default, the corresponding list of paths is *empty* + * ``--robot-paths``: + * intended to be defined in an EasyBuild configuration file, or via ``$EASYBUILD_ROBOT_PATHS`` * does *not* enable the dependency resolution mechanism * the specified list of paths is *appended* to the robot search path From 7f66f4eb21724b9cad78aacbfac501e7d1d300c1 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Wed, 11 Mar 2015 21:07:53 +0100 Subject: [PATCH 15/69] fix formatting issues --- docs/Configuration.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/Configuration.rst b/docs/Configuration.rst index 61e31e94..f8ada979 100644 --- a/docs/Configuration.rst +++ b/docs/Configuration.rst @@ -159,8 +159,11 @@ Templates and constants supported in configuration files Two types of template values are supported in configuration files: * for configuration options defined in the configuration file (and only those) + * *syntax:* ``%(opt)s``, i.e., using the (lowercase) name of the configuration option + * for the default value of selected configuration options (see ``eb --avail-cfgfile-constants``) + * *syntax:* ``%(DEFAULT_OPT)``, i.e., using the uppercase name of the configuration option and prefixed with ``DEFAULT_`` .. note:: From d0844f73a1bf2ab5f8f6854abed876ef157a68da Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Wed, 11 Mar 2015 21:12:51 +0100 Subject: [PATCH 16/69] clarify template value syntax --- docs/Configuration.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Configuration.rst b/docs/Configuration.rst index f8ada979..ee266f51 100644 --- a/docs/Configuration.rst +++ b/docs/Configuration.rst @@ -156,7 +156,7 @@ An example configuration file that should make everything clear is shown below. Templates and constants supported in configuration files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Two types of template values are supported in configuration files: +Two types of template values ``%(...)s`` are supported in configuration files: * for configuration options defined in the configuration file (and only those) @@ -164,7 +164,7 @@ Two types of template values are supported in configuration files: * for the default value of selected configuration options (see ``eb --avail-cfgfile-constants``) - * *syntax:* ``%(DEFAULT_OPT)``, i.e., using the uppercase name of the configuration option and prefixed with ``DEFAULT_`` + * *syntax:* ``%(DEFAULT_OPT)s``, i.e., using the uppercase name of the configuration option and prefixed with ``DEFAULT_`` .. note:: These template values are only supported in configuration files, *not* in environment variable values or From 2b96d7737973957e6defee45f4ae24a4f9bb37e4 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Wed, 11 Mar 2015 21:15:46 +0100 Subject: [PATCH 17/69] clarify note w.r.t. unknown template values --- docs/Configuration.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/Configuration.rst b/docs/Configuration.rst index ee266f51..19f92108 100644 --- a/docs/Configuration.rst +++ b/docs/Configuration.rst @@ -171,8 +171,9 @@ Two types of template values ``%(...)s`` are supported in configuration files: command line option values. .. note:: - Using a template value for a configuration option that was not defined in the configuration file - will result in an error like ``ConfigParser.InterpolationMissingOptionError: Bad value substitution``. + Using an unknown template value, i.e. either one for a configuration option that was not defined in the + configuration file, or a non-existing one for a particular default value, will result in an error like: + ``ConfigParser.InterpolationMissingOptionError: Bad value substitution``. Example +++++++ From 9020b76c1b73132e0ef25f09a9d5d69c3639f56b Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Wed, 11 Mar 2015 22:22:00 +0100 Subject: [PATCH 18/69] bump release, update changelog --- docs/Changelog.rst | 4 ++++ docs/conf.py | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/Changelog.rst b/docs/Changelog.rst index 583142d2..8ae859bb 100644 --- a/docs/Changelog.rst +++ b/docs/Changelog.rst @@ -3,6 +3,10 @@ Changelog for EasyBuild documentation ------------------------------------- +* **release 20150312.01** (`Mar 12th 2015`): + + * enhance documentation w.r.t. template values in configuration files (see :ref:`configuration_file_templates_constants`) + * improve documentation on ``--robot`` and ``--robot-paths`` (see :ref:`controlling_robot_search_path`) * **release 20150310.01** (`Mar 10th 2015`): * document peculiarties w.r.t. dependencies in combination with a ``dummy`` toolchain (see :ref:`dependency_specs`) diff --git a/docs/conf.py b/docs/conf.py index 38e3e15d..8305f97a 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -43,7 +43,7 @@ # The short X.Y version. version = '2.0.0' # this is meant to reference the version of EasyBuild # The full version, including alpha/beta/rc tags. -release = '20150310.01' # this is meant to reference the version of the documentation itself +release = '20150312.01' # this is meant to reference the version of the documentation itself # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: From 0d333e72984523ae2b9810a8d5e9e9d465dce5e0 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Thu, 12 Mar 2015 09:28:19 +0100 Subject: [PATCH 19/69] clarify overriding of default robot search path --- docs/Using_the_EasyBuild_command_line.rst | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/Using_the_EasyBuild_command_line.rst b/docs/Using_the_EasyBuild_command_line.rst index 2a8b085e..f1c495e8 100644 --- a/docs/Using_the_EasyBuild_command_line.rst +++ b/docs/Using_the_EasyBuild_command_line.rst @@ -431,31 +431,34 @@ To control the robot search path, you can specify a (colon-separated list of) pa eb --robot=$PWD:$HOME ... -These two configuration options each serve a particular purpose: +These two configuration options each serve a particular purpose, and together define the robot search path: * ``--robot``, ``-r``: * intended to be used (only) as a command line option to ``eb`` (although it can also be defined through another configuration type see :ref:`configuration_types`) * enables the dependency resolution mechanism (disabled by default) - * optionally, a list of paths can be specified, which is *prepended* to the robot search path + * optionally a list of paths can be specified, which is included *first* in the robot search path * by default, the corresponding list of paths is *empty* * ``--robot-paths``: * intended to be defined in an EasyBuild configuration file, or via ``$EASYBUILD_ROBOT_PATHS`` * does *not* enable the dependency resolution mechanism - * the specified list of paths is *appended* to the robot search path - * by default, the corresponding list of paths only contains the path to the easyconfig files which - are part of the EasyBuild installation + * the specified list of paths is included *last* in the robot search path + * by default, only the path to the easyconfig files that are part of the EasyBuild installation is listed + * **note**: setting this configuration option implies redefining the default robot search path! For both options, the list of paths should be specified as a colon-separated (``:``) list. By combining ``--robot`` and ``--robot-paths`` using the different configuration types (see also -:ref:`configuration_types`), you have full control over the robot search path. +:ref:`configuration_types`), you have full control over the robot search path: which paths are included, +the order of those paths, whether or not the easyconfig files that are part of the EasyBuild installation should be +considered, etc. A constant named ``DEFAULT_ROBOT_PATHS`` is available that can be used (only) in EasyBuild configuration files to refer -to the default robot search path. For more information on using constants in EasyBuild configuration files, see +to the default robot search path, i.e. the path to the easyconfigs that are part of the EasyBuild installation. +For more information on using constants in EasyBuild configuration files, see :ref:`configuration_file_templates_constants`. .. tip:: From 52e96eb3909318f62c56bb63bbf13354eadd3d95 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Thu, 12 Mar 2015 09:34:01 +0100 Subject: [PATCH 20/69] just refer to configuration_types once in the same section --- docs/Using_the_EasyBuild_command_line.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Using_the_EasyBuild_command_line.rst b/docs/Using_the_EasyBuild_command_line.rst index f1c495e8..9d9854c8 100644 --- a/docs/Using_the_EasyBuild_command_line.rst +++ b/docs/Using_the_EasyBuild_command_line.rst @@ -436,7 +436,7 @@ These two configuration options each serve a particular purpose, and together de * ``--robot``, ``-r``: * intended to be used (only) as a command line option to ``eb`` (although it can also be defined through another - configuration type see :ref:`configuration_types`) + configuration type) * enables the dependency resolution mechanism (disabled by default) * optionally a list of paths can be specified, which is included *first* in the robot search path * by default, the corresponding list of paths is *empty* From 80c82b8bbd599eabe8023ac81d71861367eeaad2 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 16 Mar 2015 09:15:24 +0100 Subject: [PATCH 21/69] include list of EasyBuild repositories cloned by install-EasyBuild-develop.sh script --- docs/Changelog.rst | 4 ++++ docs/Installation_Alternative.rst | 12 +++++++++++- docs/conf.py | 2 +- 3 files changed, 16 insertions(+), 2 deletions(-) diff --git a/docs/Changelog.rst b/docs/Changelog.rst index 8ae859bb..5a3d5429 100644 --- a/docs/Changelog.rst +++ b/docs/Changelog.rst @@ -3,6 +3,10 @@ Changelog for EasyBuild documentation ------------------------------------- +* **release 20150316.01** (`Mar 16th 2015`): + + * include list of EasyBuild repositories cloned by ``install-EasyBuild-develop.sh`` script + (see :ref:`install_latest_develop_using_script`) * **release 20150312.01** (`Mar 12th 2015`): * enhance documentation w.r.t. template values in configuration files (see :ref:`configuration_file_templates_constants`) diff --git a/docs/Installation_Alternative.rst b/docs/Installation_Alternative.rst index 8c72259c..98091326 100644 --- a/docs/Installation_Alternative.rst +++ b/docs/Installation_Alternative.rst @@ -174,8 +174,18 @@ Installation of latest development version using provided script ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ After you have forked each of the EasyBuild repositories on GitHub (+ vsc-base), you can set up a development version -of EasyBuild, in the following manner:: +of EasyBuild using the ``install-EasyBuild-develop.sh`` script. +This script will clone the different EasyBuild repositories from GitHub: + + * ``easybuild``: EasyBuild metapackage & documentation sources for http://easybuild.readthedocs.org + * ``vsc-base``: dependency for EasyBuild framework (logging, command line interface, ...) + * ``easybuild-framework``: EasyBuild framework + * ``easybuild-easyblocks``: collection of easyblocks + * ``easybuild-easyconfigs``: collection of easyconfig files + * ``easybuild-wiki``: EasyBuild wiki pages + +It can be used as follows:: # pick an installation prefix (adjust as you like) INSTALL_PREFIX=$(mktemp -d $HOME/EasyBuild-XXXXXX) # download script diff --git a/docs/conf.py b/docs/conf.py index 8305f97a..d6adf8b9 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -43,7 +43,7 @@ # The short X.Y version. version = '2.0.0' # this is meant to reference the version of EasyBuild # The full version, including alpha/beta/rc tags. -release = '20150312.01' # this is meant to reference the version of the documentation itself +release = '20150316.01' # this is meant to reference the version of the documentation itself # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: From 1feba71ee497609a0c6f95891017360ed8dd4c81 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 16 Mar 2015 18:31:10 +0100 Subject: [PATCH 22/69] link to different EB repos --- docs/Installation_Alternative.rst | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/docs/Installation_Alternative.rst b/docs/Installation_Alternative.rst index 98091326..a7c63233 100644 --- a/docs/Installation_Alternative.rst +++ b/docs/Installation_Alternative.rst @@ -178,12 +178,18 @@ of EasyBuild using the ``install-EasyBuild-develop.sh`` script. This script will clone the different EasyBuild repositories from GitHub: - * ``easybuild``: EasyBuild metapackage & documentation sources for http://easybuild.readthedocs.org - * ``vsc-base``: dependency for EasyBuild framework (logging, command line interface, ...) - * ``easybuild-framework``: EasyBuild framework - * ``easybuild-easyblocks``: collection of easyblocks - * ``easybuild-easyconfigs``: collection of easyconfig files - * ``easybuild-wiki``: EasyBuild wiki pages + * `easybuild `_: + EasyBuild metapackage & documentation sources for http://easybuild.readthedocs.org + * `vsc-base `_: + dependency for EasyBuild framework (logging, command line interface, ...) + * `easybuild-framework `_: + EasyBuild framework + * `easybuild-easyblocks `_: + collection of easyblocks + * `easybuild-easyconfigs `_: + collection of easyconfig files + * `easybuild-wiki `_: + EasyBuild wiki pages It can be used as follows:: # pick an installation prefix (adjust as you like) From 4517155e5d9aa935d608dff5a236e7f41e571467 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Tue, 17 Mar 2015 10:35:56 +0100 Subject: [PATCH 23/69] fix formatting issue --- docs/Installation_Alternative.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/Installation_Alternative.rst b/docs/Installation_Alternative.rst index a7c63233..897ccbd0 100644 --- a/docs/Installation_Alternative.rst +++ b/docs/Installation_Alternative.rst @@ -192,6 +192,7 @@ This script will clone the different EasyBuild repositories from GitHub: EasyBuild wiki pages It can be used as follows:: + # pick an installation prefix (adjust as you like) INSTALL_PREFIX=$(mktemp -d $HOME/EasyBuild-XXXXXX) # download script From 52805ce26daf4c330febd5b857c82c6db095a7a9 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Thu, 26 Mar 2015 16:50:30 +0100 Subject: [PATCH 24/69] documented deprecated functionality w.r.t. error reporting --- docs/Changelog.rst | 3 +++ docs/Deprecated-functionality.rst | 36 +++++++++++++++++++++++++++++-- docs/conf.py | 2 +- 3 files changed, 38 insertions(+), 3 deletions(-) diff --git a/docs/Changelog.rst b/docs/Changelog.rst index 5a3d5429..7ffac9e2 100644 --- a/docs/Changelog.rst +++ b/docs/Changelog.rst @@ -3,6 +3,9 @@ Changelog for EasyBuild documentation ------------------------------------- +* **release 20150326.01** (`Mar 26th 2015`): + + * documented deprecated functionality w.r.t. error reporting (see :ref:`depr_error_reporting`) * **release 20150316.01** (`Mar 16th 2015`): * include list of EasyBuild repositories cloned by ``install-EasyBuild-develop.sh`` script diff --git a/docs/Deprecated-functionality.rst b/docs/Deprecated-functionality.rst index 071d40c6..c4da0432 100644 --- a/docs/Deprecated-functionality.rst +++ b/docs/Deprecated-functionality.rst @@ -34,11 +34,43 @@ For authors of easyconfig files: For developers of easyblocks: -*(nothing yet)* +* ref:`depr_error_reporting` For EasyBuild framework developers: -*(nothing yet)* +* ref:`depr_error_reporting` + +.. _depr_error_reporting: + +Report errors by raising ``EasyBuildError`` rather than using ``error()``/``exception()`` log methods +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Use of the** ``error`` **and** ``exception`` **log methods is deprecated.** + +* *deprecated since:* EasyBuild v2.1.0 (April'15) +* *removed in:* EasyBuild v3.0 +* *alternative(s)*: **use** ``raise EasyBuildError(...)`` **instead** + +The `error() `_ and +`exception() `_ log methods defined by +EasyBuild (in the ``easybuild.tools.build_log`` module) do not ahder to the semantics of the standard Python log +methods, in the sense that they also raise an exception next to logging messages. + +This causes problems when 3rd party libraries are being used together with EasyBuild, since they may be using these +log methods without expecting an exception being raised. + +The custom definitions for the ``error()`` and ``exception()`` log methods will be removed in EasyBuild v3.0, +so they should no longer be used. Note that this applies both to the EasyBuild framework and to (custom) easyblocks. + +To report errors, an ``EasyBuildError`` should be raised instead:: + + from easybuild.tools.build_log import EasyBuildError + ... + def configure_step(self): + """Example configure step implementation where things may go wrong.""" + ... + if errors: + raise EasyBuildError("One or more errors were detected: %s", errors) .. _deprecation_policy: diff --git a/docs/conf.py b/docs/conf.py index d6adf8b9..91ae3d4b 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -43,7 +43,7 @@ # The short X.Y version. version = '2.0.0' # this is meant to reference the version of EasyBuild # The full version, including alpha/beta/rc tags. -release = '20150316.01' # this is meant to reference the version of the documentation itself +release = '20150326.01' # this is meant to reference the version of the documentation itself # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: From 67b70d075bd8daae2a6b8178bf5350a7c8e531ce Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Fri, 27 Mar 2015 09:00:33 +0100 Subject: [PATCH 25/69] reword section on deprecating log.error --- docs/Deprecated-functionality.rst | 42 +++++++++++++++---------------- docs/conf.py | 2 +- 2 files changed, 22 insertions(+), 22 deletions(-) diff --git a/docs/Deprecated-functionality.rst b/docs/Deprecated-functionality.rst index c4da0432..5c0eb57b 100644 --- a/docs/Deprecated-functionality.rst +++ b/docs/Deprecated-functionality.rst @@ -34,43 +34,43 @@ For authors of easyconfig files: For developers of easyblocks: -* ref:`depr_error_reporting` +* :ref:`depr_error_reporting` For EasyBuild framework developers: -* ref:`depr_error_reporting` +* :ref:`depr_error_reporting` .. _depr_error_reporting: -Report errors by raising ``EasyBuildError`` rather than using ``error()``/``exception()`` log methods -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Report errors by raising ``EasyBuildError`` rather than using log methods +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -**Use of the** ``error`` **and** ``exception`` **log methods is deprecated.** +**Use of the** ``error()`` **and** ``exception()`` **log methods is deprecated.** * *deprecated since:* EasyBuild v2.1.0 (April'15) * *removed in:* EasyBuild v3.0 * *alternative(s)*: **use** ``raise EasyBuildError(...)`` **instead** -The `error() `_ and -`exception() `_ log methods defined by -EasyBuild (in the ``easybuild.tools.build_log`` module) do not ahder to the semantics of the standard Python log -methods, in the sense that they also raise an exception next to logging messages. +The ``error()`` and ``exception()`` log methods defined by EasyBuild (in the ``easybuild.tools.build_log`` module) +do not match the semantics of the `standard Python log methods +`_, in the sense that they also raise an +exception next to logging messages. -This causes problems when 3rd party libraries are being used together with EasyBuild, since they may be using these -log methods without expecting an exception being raised. +This may cause problems when 3rd party libraries (e.g., `gc3pie `_) are being +used by EasyBuild, since they may be using these log methods without expecting an exception being raised. -The custom definitions for the ``error()`` and ``exception()`` log methods will be removed in EasyBuild v3.0, -so they should no longer be used. Note that this applies both to the EasyBuild framework and to (custom) easyblocks. +The custom definitions for the ``error()`` and ``exception()`` log methods will be removed in EasyBuild v3.0. -To report errors, an ``EasyBuildError`` should be raised instead:: +Hence, these log methods should no longer be used to report errors since they will not raise an exception anymore once +their custom definitions are removed. Note that this applies both to the EasyBuild framework and to (custom) easyblocks. - from easybuild.tools.build_log import EasyBuildError - ... - def configure_step(self): - """Example configure step implementation where things may go wrong.""" - ... - if errors: - raise EasyBuildError("One or more errors were detected: %s", errors) +To report errors, an ``EasyBuildError`` should be raised instead. For example: + +.. code:: python + + # make sure config.sh script is there + if not os.path.exists(os.path.join(self.builddir, 'config.sh'): + raise EasyBuildError("config.sh script is missing in %s", self.builddir) .. _deprecation_policy: diff --git a/docs/conf.py b/docs/conf.py index 91ae3d4b..62bce78d 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -43,7 +43,7 @@ # The short X.Y version. version = '2.0.0' # this is meant to reference the version of EasyBuild # The full version, including alpha/beta/rc tags. -release = '20150326.01' # this is meant to reference the version of the documentation itself +release = '20150327.01' # this is meant to reference the version of the documentation itself # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: From 32c8212f354a02b4800a121e0e58406247e67374 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Fri, 27 Mar 2015 09:40:50 +0100 Subject: [PATCH 26/69] fix typo and code formatting --- docs/Deprecated-functionality.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Deprecated-functionality.rst b/docs/Deprecated-functionality.rst index 5c0eb57b..458cd32d 100644 --- a/docs/Deprecated-functionality.rst +++ b/docs/Deprecated-functionality.rst @@ -66,10 +66,10 @@ their custom definitions are removed. Note that this applies both to the EasyBui To report errors, an ``EasyBuildError`` should be raised instead. For example: -.. code:: python +.. code:: # make sure config.sh script is there - if not os.path.exists(os.path.join(self.builddir, 'config.sh'): + if not os.path.exists(os.path.join(self.builddir, 'config.sh')): raise EasyBuildError("config.sh script is missing in %s", self.builddir) .. _deprecation_policy: From 54dfc6ab1835e3a502dcfcab1e47ea5cca67c325 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Fri, 27 Mar 2015 10:15:35 +0100 Subject: [PATCH 27/69] fix date in Changelog --- docs/Changelog.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Changelog.rst b/docs/Changelog.rst index 7ffac9e2..aff4af40 100644 --- a/docs/Changelog.rst +++ b/docs/Changelog.rst @@ -3,7 +3,7 @@ Changelog for EasyBuild documentation ------------------------------------- -* **release 20150326.01** (`Mar 26th 2015`): +* **release 20150327.01** (`Mar 27th 2015`): * documented deprecated functionality w.r.t. error reporting (see :ref:`depr_error_reporting`) * **release 20150316.01** (`Mar 16th 2015`): From 0fbe53d0ad13c8452ad8c9908e87f9fca7d220ea Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Tue, 7 Apr 2015 16:29:03 +0200 Subject: [PATCH 28/69] improve documentation on software/modules install paths --- docs/Changelog.rst | 7 +++ docs/Configuration.rst | 108 +++++++++++++++++++++++++++++++++-------- docs/Installation.rst | 4 ++ docs/conf.py | 4 +- 4 files changed, 100 insertions(+), 23 deletions(-) diff --git a/docs/Changelog.rst b/docs/Changelog.rst index aff4af40..396d7013 100644 --- a/docs/Changelog.rst +++ b/docs/Changelog.rst @@ -3,6 +3,13 @@ Changelog for EasyBuild documentation ------------------------------------- +* **release 20150407.01** (`Apr 7th 2015`): + + * add link to :ref:`unit_tests` page in dedicated section at :ref:`installation` page + (see :ref:`install_running_unit_tests:`) + * clarify relation between ``--installpath``, ``--prefix``, ``-subdir-*`` and ``-installpath-*`` configuration options + (see :ref:`installpath`) + * **release 20150327.01** (`Mar 27th 2015`): * documented deprecated functionality w.r.t. error reporting (see :ref:`depr_error_reporting`) diff --git a/docs/Configuration.rst b/docs/Configuration.rst index 19f92108..b89e3ed3 100644 --- a/docs/Configuration.rst +++ b/docs/Configuration.rst @@ -291,6 +291,8 @@ If any of these configuration settings is not provided in one way or another, Ea In practice, all of these have reasonable defaults (see ``eb --help`` for the default settings). +.. note:: The mandatory path-related options can be tweaked collectively via ``--prefix``, see :ref:`prefix` for more + information. .. _sourcepath: @@ -337,42 +339,86 @@ when the installation is completed (by default). .. _installpath: -Install path (``--installpath``) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Software and modules install path (``--installpath``, ``--installpath-software``, ``--installpath-modules``) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +defaults: + * *software install path:* ``$HOME/.local/easybuild/software`` (determined via ``--prefix`` and + ``--subdir-software``) + * *modules install path:* ``$HOME/.local/easybuild/modules/all`` (determined via ``--prefix``, + ``--subdir-modules`` and ``--suffix-modules-path``) + +There are several ways in which the software and modules install path used by EasyBuild can be configured: + +* using the direct configuration options ``--installpath-software`` and ``--installpath-modules``, + see :ref:`installpath_direct_options` +* using the parent install path configuration option ``--installpath`` or overall prefix path configuration option + ``--prefix`` (see :ref:`prefix`), along with the ``--subdir-*`` and ``--suffix-modules-path`` companion + configuration options (see :ref:`parent_installpath`) + +.. _installpath_direct_options: + +Direct options: ``--installpath-software`` and ``--installpath-modules`` +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +(available since EasyBuild v2.1.0) + +*default*: *(no default specified)* + +The ``--installpath-software`` and ``--installpath-modules`` configuration options allow to directly specify +the software and modules install paths, respectively. + +These configuration options have precedence over all of the other configuration options that relate to specifying the +install path for software and/or modules (see below). + +.. _parent_installpath: -*default*: ``$HOME/.local/easybuild/`` +Parent install path: ``--installpath`` +++++++++++++++++++++++++++++++++++++++ -The ``installpath`` configuration setting specifies the parent path of -the directories in which EasyBuild installs software packages and the +*default*: *(no default specified)* + +The ``--installpath`` configuration option specifies the *parent* path of +the directories in which EasyBuild should install software packages and the corresponding module files. -By default, the packages themselves are installed under ``/software`` -in their own subdirectory following the active module naming scheme -(e.g., ``/-``, by default). -The corresponding module files are installed under -``/modules/all``, and symlinks are installed in ``/modules/``. +The install path for software and modules specifically is determined by combining ``--installpath`` with +``--subdir-software``, and ``--subdir-modules`` together with ``--suffix-modules-path``, respectively. +For more information on these companion configuration options, see :ref:`installpath_subdirs`. + +Full install path for software and module file +++++++++++++++++++++++++++++++++++++++++++++++ -Different configuration options are available for changing the default behaviour, i.e., ---subdir-software, --subdir-modules, --suffix-modules-path, --module-naming-scheme, etc. +The full software and module install paths for a particular software package are determined by the active +module naming scheme along with the general software and modules install paths specified by the EasyBuild configuration. -For more information, see `Optional configuration settings`_. +Both the software itself and the corresponding module file will be installed in a subdirectory of the corresponding +install path named according to the active module naming scheme (default format: +``/-``). +Additionally, symlinks to the actual module file are installed in a subdirectory of the modules install path +named according to the value of the ``moduleclass`` easyconfig parameter. +For more information on the module naming scheme used by EasyBuild, see :ref:`module_naming_scheme`. -Setting ``$MODULEPATH`` -+++++++++++++++++++++++ +Updating ``$MODULEPATH`` +++++++++++++++++++++++++ -After (re)configuring EasyBuild, you need to make sure that -``$MODULEPATH`` environment variable is extended with the -``modules/all`` subdirectory of the ``installpath`` -so you can load the modules created for the software built with EasyBuild, i.e.: +To make the modules generated by EasyBuild available, the ``$MODULEPATH`` environment variable must be updated +to include the modules install path. The recommended way to do this is to use the ``module use`` command. +For example: .. code:: shell-session - $ module use /modules/all + $ EXAMPLE_INSTALLPATH=$HOME/easybuild + $ eb --installpath=$EXAMPLE_INSTALLPATH + $ module use $EXAMPLE_INSTALLPATH/modules/all It is probably a good idea to add this to your (favourite) shell ``.rc`` file, e.g., ``~/.bashrc``, and/or the ``~/.profile`` login scripts, so you do not need to adjust ``$MODULEPATH`` every time you start a new session. +.. note:: Updating ``$MODULEPATH`` is not needed for EasyBuild itself, since it updates ``$MODULEPATH`` itself + according to the modules install path it is configured with. + .. _easyconfigs_repo: Easyconfigs repository (``--repository``, ``--repositorypath``) @@ -469,6 +515,24 @@ template value here):: Optional configuration settings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. _prefix: + +Overall prefix +^^^^^^^^^^^^^^ + +*default:* ``$HOME/.local/easybuild`` + +The overall prefix path used by EasyBuild can be specified using the ``--prefix`` configuration option. + +This affects the default value of several configuration options: + +* source path (see :ref:`sourcepath`) +* build path (see :ref:`buildpath`) +* software and modules install path (see :ref:`installpath`) +* easyconfigs repository path (see :ref:`easyconfigs_repo:`) + +.. _installpath_subdirs: + Software and modules install path suffixes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -476,7 +540,7 @@ This section describes configuration options ``--subdir-software``, ``--subdir-modules``, ``--suffix-modules-path``, which are supported since v1.14.0. -*defaults*: ``software`` as software install path suffix, ``modules`` as +*defaults*: ``software`` as software install path suffix, ``modules/all`` as modules install path suffix The software and modules install path suffixes can be adjusted using the @@ -515,6 +579,8 @@ For example, to indicate that EasyBuild should be using ``Lmod`` as modules tool $ eb --modules-tool=Lmod ... +.. _module_naming_scheme: + Active module naming scheme (``--module-naming-scheme``) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/Installation.rst b/docs/Installation.rst index 8fc32c1b..825566bc 100644 --- a/docs/Installation.rst +++ b/docs/Installation.rst @@ -127,6 +127,8 @@ For example:: may make a big difference, if you have installed both versions 1.9.0 and 1.15.2, with respect to what is the version being loaded by default. +.. _install_running_unit_tests: + Running unit tests ~~~~~~~~~~~~~~~~~~ @@ -135,6 +137,8 @@ After completion of the bootstrap procedure and loading the python -m test.framework.suite +More details about the EasyBuild unit tests are available at :ref:`unit_tests`. + If this does not complete successfully, `please open an issue`_ to report it. .. _please open an issue: https://github.com/hpcugent/easybuild-framework/issues/new diff --git a/docs/conf.py b/docs/conf.py index 62bce78d..be19c05e 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -41,9 +41,9 @@ # other places throughout the built documents. # # The short X.Y version. -version = '2.0.0' # this is meant to reference the version of EasyBuild +version = '2.1.0dev' # this is meant to reference the version of EasyBuild # The full version, including alpha/beta/rc tags. -release = '20150327.01' # this is meant to reference the version of the documentation itself +release = '20150407.01' # this is meant to reference the version of the documentation itself # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: From c8a87371c01cd9c96c63fb3f2070a301969df66f Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Tue, 7 Apr 2015 16:50:13 +0200 Subject: [PATCH 29/69] fix formatting issues --- docs/Changelog.rst | 2 +- docs/Configuration.rst | 76 ++++++++++++++++++++---------------------- 2 files changed, 37 insertions(+), 41 deletions(-) diff --git a/docs/Changelog.rst b/docs/Changelog.rst index 396d7013..0dba25e3 100644 --- a/docs/Changelog.rst +++ b/docs/Changelog.rst @@ -6,7 +6,7 @@ Changelog for EasyBuild documentation * **release 20150407.01** (`Apr 7th 2015`): * add link to :ref:`unit_tests` page in dedicated section at :ref:`installation` page - (see :ref:`install_running_unit_tests:`) + (see :ref:`install_running_unit_tests`) * clarify relation between ``--installpath``, ``--prefix``, ``-subdir-*`` and ``-installpath-*`` configuration options (see :ref:`installpath`) diff --git a/docs/Configuration.rst b/docs/Configuration.rst index b89e3ed3..12200568 100644 --- a/docs/Configuration.rst +++ b/docs/Configuration.rst @@ -343,30 +343,29 @@ Software and modules install path (``--installpath``, ``--installpath-software`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ defaults: - * *software install path:* ``$HOME/.local/easybuild/software`` (determined via ``--prefix`` and - ``--subdir-software``) - * *modules install path:* ``$HOME/.local/easybuild/modules/all`` (determined via ``--prefix``, - ``--subdir-modules`` and ``--suffix-modules-path``) + +* *software install path:* ``$HOME/.local/easybuild/software`` (determined via ``--prefix`` and + ``--subdir-software``) +* *modules install path:* ``$HOME/.local/easybuild/modules/all`` (determined via ``--prefix``, + ``--subdir-modules`` and ``--suffix-modules-path``) There are several ways in which the software and modules install path used by EasyBuild can be configured: -* using the direct configuration options ``--installpath-software`` and ``--installpath-modules``, - see :ref:`installpath_direct_options` -* using the parent install path configuration option ``--installpath`` or overall prefix path configuration option - ``--prefix`` (see :ref:`prefix`), along with the ``--subdir-*`` and ``--suffix-modules-path`` companion - configuration options (see :ref:`parent_installpath`) +* using the direct configuration options ``--installpath-software`` and ``--installpath-modules`` (see + :ref:`installpath_direct_options`) +* using the parent install path configuration option ``--installpath`` (see :ref:`parent_installpath`) or overall + prefix path configuration option ``--prefix`` (see :ref:`prefix`), along with the ``--subdir-*`` and + ``--suffix-modules-path`` companion configuration options (see :ref:`parent_installpath`) .. _installpath_direct_options: Direct options: ``--installpath-software`` and ``--installpath-modules`` ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -(available since EasyBuild v2.1.0) - *default*: *(no default specified)* -The ``--installpath-software`` and ``--installpath-modules`` configuration options allow to directly specify -the software and modules install paths, respectively. +The ``--installpath-software`` and ``--installpath-modules`` configuration options (available since EasyBuild v2.1.0) +allow to directly specify the software and modules install paths, respectively. These configuration options have precedence over all of the other configuration options that relate to specifying the install path for software and/or modules (see below). @@ -383,8 +382,8 @@ the directories in which EasyBuild should install software packages and the corresponding module files. The install path for software and modules specifically is determined by combining ``--installpath`` with -``--subdir-software``, and ``--subdir-modules`` together with ``--suffix-modules-path``, respectively. -For more information on these companion configuration options, see :ref:`installpath_subdirs`. +``--subdir-software``, and combining ``--installpath`` with ``--subdir-modules`` and ``--suffix-modules-path``, +respectively. For more information on these companion configuration options, see :ref:`installpath_subdirs`. Full install path for software and module file ++++++++++++++++++++++++++++++++++++++++++++++ @@ -408,16 +407,15 @@ For example: .. code:: shell-session - $ EXAMPLE_INSTALLPATH=$HOME/easybuild - $ eb --installpath=$EXAMPLE_INSTALLPATH - $ module use $EXAMPLE_INSTALLPATH/modules/all + $ eb --installpath=$HOME/easybuild + $ module use $HOME/easybuild/modules/all It is probably a good idea to add this to your (favourite) shell ``.rc`` file, e.g., ``~/.bashrc``, and/or the ``~/.profile`` login scripts, so you do not need to adjust ``$MODULEPATH`` every time you start a new session. -.. note:: Updating ``$MODULEPATH`` is not needed for EasyBuild itself, since it updates ``$MODULEPATH`` itself - according to the modules install path it is configured with. +.. note:: Updating ``$MODULEPATH`` is not needed for EasyBuild itself, since itiupdates ``$MODULEPATH`` itself at + runtime according to the modules install path it is configured with. .. _easyconfigs_repo: @@ -434,18 +432,16 @@ Currently, EasyBuild supports the following repository types (see also ``eb --avail-repositories``): * ``FileRepository('path', 'subdir')``: a plain flat file repository; - ``path`` is the path where files will be stored, ``subdir`` is an - *optional* subdirectory of - that path where the files should be stored + ``path`` is the path where files will be stored, ``subdir`` is an + *optional* subdirectory of that path where the files should be stored * ``GitRepository('path', 'subdir/in/repo'``: a *non-empty* **bare** - git repository (created with ``git init --bare`` or ``git clone --bare``); - ``path`` is the path to the git repository (can also be a URL); - ``subdir/in/repo`` is optional, and specifies a subdirectory of the - repository where files should be stored in + git repository (created with ``git init --bare`` or ``git clone --bare``); + ``path`` is the path to the git repository (can also be a URL); + ``subdir/in/repo`` is optional, and specifies a subdirectory of the + repository where files should be stored in * ``SvnRepository('path', 'subdir/in/repo')``: an SVN repository; - ``path`` contains the subversion repository location (directory or - URL), the optional second value - specifies a subdirectory in the repository + ``path`` contains the subversion repository location (directory or + URL), the optional second value specifies a subdirectory in the repository You need to set the ``repository`` setting inside a configuration file like this: @@ -533,19 +529,19 @@ This affects the default value of several configuration options: .. _installpath_subdirs: -Software and modules install path suffixes -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Software and modules install path subdirectories (``--subdir-software``, ``--subdir-modules``, ``--suffix-modules-path``) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -This section describes configuration options -``--subdir-software``, ``--subdir-modules``, ``--suffix-modules-path``, -which are supported since v1.14.0. +*defaults*: -*defaults*: ``software`` as software install path suffix, ``modules/all`` as -modules install path suffix +* *software install path subdirectory* (``--subdir-software``): ``software`` +* *modules install path subdirectory* (``--subdir-modules``): ``modules`` +* *modules install path suffix* (``--suffix-modules-path``): ``all`` -The software and modules install path suffixes can be adjusted using the -``subdir-software`` and/or ``subdir-modules`` / ``suffix-modules-path`` configuration settings, -for example: +The subdirectories for the software and modules install paths (relative to ``--installpath``) can be specified using +the corresponding dedicated configuration options (available since EasyBuild v1.14.0). + +For example: .. code:: shell-session From 16beec573a19b428b54ceb1cea1b23282935e9c9 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Tue, 7 Apr 2015 17:04:25 +0200 Subject: [PATCH 30/69] fix formatting issues --- docs/Configuration.rst | 32 ++++++++++++++------------------ 1 file changed, 14 insertions(+), 18 deletions(-) diff --git a/docs/Configuration.rst b/docs/Configuration.rst index 12200568..f26d5ec8 100644 --- a/docs/Configuration.rst +++ b/docs/Configuration.rst @@ -27,7 +27,8 @@ is even fairly common). The order of preference for the different configuration types is as listed above, that is: * environment variables override the corresponding entries in the configuration file -* command line arguments in turn override the corresponding environment variables *and* matching entries in the configuration file +* command line arguments in turn override the corresponding environment variables *and* matching entries in the + configuration file Consistentency across supported configuration types @@ -159,12 +160,9 @@ Templates and constants supported in configuration files Two types of template values ``%(...)s`` are supported in configuration files: * for configuration options defined in the configuration file (and only those) - - * *syntax:* ``%(opt)s``, i.e., using the (lowercase) name of the configuration option - + * *syntax:* ``%(opt)s``, i.e., using the (lowercase) name of the configuration option * for the default value of selected configuration options (see ``eb --avail-cfgfile-constants``) - - * *syntax:* ``%(DEFAULT_OPT)s``, i.e., using the uppercase name of the configuration option and prefixed with ``DEFAULT_`` + * *syntax:* ``%(DEFAULT_OPT)s``, i.e., using the uppercase name of the configuration option and prefixed with ``DEFAULT_`` .. note:: These template values are only supported in configuration files, *not* in environment variable values or @@ -308,7 +306,8 @@ Looking for the files specified via the ``sources`` parameter in the .eb easyconfig file is done in the following order of preference: * ``/``: a subdirectory determined by the name of the software package -* ``//``: in the style of the ``easyblocks``/``easyconfigs`` directories: in a subdirectory determined by the first letter (in lower case) of the software package and by its full ``name`` +* ``//``: in the style of the ``easyblocks``/``easyconfigs`` directories: in a + subdirectory determined by the first letter (in lower case) of the software package and by its full ``name`` * ````: directly in the source path Note that these locations are also used when EasyBuild looks for patch @@ -343,11 +342,10 @@ Software and modules install path (``--installpath``, ``--installpath-software`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ defaults: - -* *software install path:* ``$HOME/.local/easybuild/software`` (determined via ``--prefix`` and - ``--subdir-software``) -* *modules install path:* ``$HOME/.local/easybuild/modules/all`` (determined via ``--prefix``, - ``--subdir-modules`` and ``--suffix-modules-path``) + * *software install path:* ``$HOME/.local/easybuild/software`` (determined via ``--prefix`` and + ``--subdir-software``) + * *modules install path:* ``$HOME/.local/easybuild/modules/all`` (determined via ``--prefix``, + ``--subdir-modules`` and ``--suffix-modules-path``) There are several ways in which the software and modules install path used by EasyBuild can be configured: @@ -355,7 +353,7 @@ There are several ways in which the software and modules install path used by Ea :ref:`installpath_direct_options`) * using the parent install path configuration option ``--installpath`` (see :ref:`parent_installpath`) or overall prefix path configuration option ``--prefix`` (see :ref:`prefix`), along with the ``--subdir-*`` and - ``--suffix-modules-path`` companion configuration options (see :ref:`parent_installpath`) + ``--suffix-modules-path`` companion configuration options (see :ref:`installpath_subdirs`) .. _installpath_direct_options: @@ -430,7 +428,6 @@ corresponding ``.eb`` file is uploaded to a repository defined by the ``reposito Currently, EasyBuild supports the following repository types (see also ``eb --avail-repositories``): - * ``FileRepository('path', 'subdir')``: a plain flat file repository; ``path`` is the path where files will be stored, ``subdir`` is an *optional* subdirectory of that path where the files should be stored @@ -533,10 +530,9 @@ Software and modules install path subdirectories (``--subdir-software``, ``--sub ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ *defaults*: - -* *software install path subdirectory* (``--subdir-software``): ``software`` -* *modules install path subdirectory* (``--subdir-modules``): ``modules`` -* *modules install path suffix* (``--suffix-modules-path``): ``all`` + * *software install path subdirectory* (``--subdir-software``): ``software`` + * *modules install path subdirectory* (``--subdir-modules``): ``modules`` + * *modules install path suffix* (``--suffix-modules-path``): ``all`` The subdirectories for the software and modules install paths (relative to ``--installpath``) can be specified using the corresponding dedicated configuration options (available since EasyBuild v1.14.0). From 217fc2e207c57fc4ecc3ffa50b7b87f70035ea8a Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Tue, 7 Apr 2015 17:10:03 +0200 Subject: [PATCH 31/69] fix moar formatting issues --- docs/Configuration.rst | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/docs/Configuration.rst b/docs/Configuration.rst index f26d5ec8..d636ff5a 100644 --- a/docs/Configuration.rst +++ b/docs/Configuration.rst @@ -342,10 +342,10 @@ Software and modules install path (``--installpath``, ``--installpath-software`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ defaults: - * *software install path:* ``$HOME/.local/easybuild/software`` (determined via ``--prefix`` and - ``--subdir-software``) - * *modules install path:* ``$HOME/.local/easybuild/modules/all`` (determined via ``--prefix``, - ``--subdir-modules`` and ``--suffix-modules-path``) + +* *software install path:* ``$HOME/.local/easybuild/software`` (determined via ``--prefix`` and ``--subdir-software``) +* *modules install path:* ``$HOME/.local/easybuild/modules/all`` (determined via ``--prefix``, + ``--subdir-modules`` and ``--suffix-modules-path``) There are several ways in which the software and modules install path used by EasyBuild can be configured: @@ -400,7 +400,9 @@ Updating ``$MODULEPATH`` ++++++++++++++++++++++++ To make the modules generated by EasyBuild available, the ``$MODULEPATH`` environment variable must be updated -to include the modules install path. The recommended way to do this is to use the ``module use`` command. +to include the modules install path. + +The recommended way to do this is to use the ``module use`` command. For example: .. code:: shell-session @@ -412,7 +414,7 @@ It is probably a good idea to add this to your (favourite) shell ``.rc`` file, e.g., ``~/.bashrc``, and/or the ``~/.profile`` login scripts, so you do not need to adjust ``$MODULEPATH`` every time you start a new session. -.. note:: Updating ``$MODULEPATH`` is not needed for EasyBuild itself, since itiupdates ``$MODULEPATH`` itself at +.. note:: Updating ``$MODULEPATH`` is not required for EasyBuild itself, since ``eb`` updates ``$MODULEPATH`` itself at runtime according to the modules install path it is configured with. .. _easyconfigs_repo: @@ -530,9 +532,10 @@ Software and modules install path subdirectories (``--subdir-software``, ``--sub ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ *defaults*: - * *software install path subdirectory* (``--subdir-software``): ``software`` - * *modules install path subdirectory* (``--subdir-modules``): ``modules`` - * *modules install path suffix* (``--suffix-modules-path``): ``all`` + +* *software install path subdirectory* (``--subdir-software``): ``software`` +* *modules install path subdirectory* (``--subdir-modules``): ``modules`` +* *modules install path suffix* (``--suffix-modules-path``): ``all`` The subdirectories for the software and modules install paths (relative to ``--installpath``) can be specified using the corresponding dedicated configuration options (available since EasyBuild v1.14.0). From 8446bdceaad1cd8b7a46585a9482f0c514dfcce0 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Tue, 7 Apr 2015 17:14:14 +0200 Subject: [PATCH 32/69] fix broken ref --- docs/Configuration.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Configuration.rst b/docs/Configuration.rst index d636ff5a..e2645f3a 100644 --- a/docs/Configuration.rst +++ b/docs/Configuration.rst @@ -524,7 +524,7 @@ This affects the default value of several configuration options: * source path (see :ref:`sourcepath`) * build path (see :ref:`buildpath`) * software and modules install path (see :ref:`installpath`) -* easyconfigs repository path (see :ref:`easyconfigs_repo:`) +* easyconfigs repository path (see :ref:`easyconfigs_repo`) .. _installpath_subdirs: From 80252ab3d3b7a82c36b583e333b922d9905615c7 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Tue, 7 Apr 2015 17:15:32 +0200 Subject: [PATCH 33/69] fix even moar formatting issues --- docs/Configuration.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Configuration.rst b/docs/Configuration.rst index e2645f3a..ee3a5d51 100644 --- a/docs/Configuration.rst +++ b/docs/Configuration.rst @@ -160,9 +160,9 @@ Templates and constants supported in configuration files Two types of template values ``%(...)s`` are supported in configuration files: * for configuration options defined in the configuration file (and only those) - * *syntax:* ``%(opt)s``, i.e., using the (lowercase) name of the configuration option + * *syntax:* ``%(opt)s``, i.e., using the (lowercase) name of the configuration option * for the default value of selected configuration options (see ``eb --avail-cfgfile-constants``) - * *syntax:* ``%(DEFAULT_OPT)s``, i.e., using the uppercase name of the configuration option and prefixed with ``DEFAULT_`` + * *syntax:* ``%(DEFAULT_OPT)s``, i.e., using the uppercase name of the configuration option and prefixed with ``DEFAULT_`` .. note:: These template values are only supported in configuration files, *not* in environment variable values or From 05d2308848ab3d4599d9b3fdd540b8998bcecbaa Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Tue, 7 Apr 2015 17:18:07 +0200 Subject: [PATCH 34/69] moar prefix refs --- docs/Configuration.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/Configuration.rst b/docs/Configuration.rst index ee3a5d51..fc63f916 100644 --- a/docs/Configuration.rst +++ b/docs/Configuration.rst @@ -297,7 +297,7 @@ In practice, all of these have reasonable defaults (see ``eb --help`` for the de Source path (``--sourcepath``) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -*default*: ``$HOME/.local/easybuild/sources/`` +*default*: ``$HOME/.local/easybuild/sources/`` (determined via :ref:`prefix`) The ``sourcepath`` configuration setting specifies the parent path of the directory in which EasyBuild looks for software source and install files. @@ -318,7 +318,7 @@ files in addition to the various ``easybuild/easyconfigs`` directories that are Build path (``--buildpath``) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -*default*: ``$HOME/.local/easybuild/build/`` +*default*: ``$HOME/.local/easybuild/build/`` (determined via :ref:`prefix`) The ``buildpath`` configuration setting specifies the parent path of the (temporary) directories in which EasyBuild builds its software packages. @@ -343,8 +343,8 @@ Software and modules install path (``--installpath``, ``--installpath-software`` defaults: -* *software install path:* ``$HOME/.local/easybuild/software`` (determined via ``--prefix`` and ``--subdir-software``) -* *modules install path:* ``$HOME/.local/easybuild/modules/all`` (determined via ``--prefix``, +* *software install path:* ``$HOME/.local/easybuild/software`` (determined via :ref:`prefix` and ``--subdir-software``) +* *modules install path:* ``$HOME/.local/easybuild/modules/all`` (determined via :ref:`prefix`, ``--subdir-modules`` and ``--suffix-modules-path``) There are several ways in which the software and modules install path used by EasyBuild can be configured: @@ -422,7 +422,7 @@ so you do not need to adjust ``$MODULEPATH`` every time you start a new session. Easyconfigs repository (``--repository``, ``--repositorypath``) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -*default*: ``FileRepository`` at ``$HOME/.local/easybuild/ebfiles_repo`` +*default*: ``FileRepository`` at ``$HOME/.local/easybuild/ebfiles_repo`` (determined via :ref:`prefix`) EasyBuild has support for archiving (tested) ``.eb`` easyconfig files. After successfully installing a software package using EasyBuild, the @@ -512,8 +512,8 @@ Optional configuration settings .. _prefix: -Overall prefix -^^^^^^^^^^^^^^ +Overall prefix path +^^^^^^^^^^^^^^^^^^^ *default:* ``$HOME/.local/easybuild`` From 89792e82b9a9e8f5cee94e952d065010eea5cf11 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Tue, 7 Apr 2015 17:20:14 +0200 Subject: [PATCH 35/69] include --prefix in section title --- docs/Configuration.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Configuration.rst b/docs/Configuration.rst index fc63f916..36659d87 100644 --- a/docs/Configuration.rst +++ b/docs/Configuration.rst @@ -512,8 +512,8 @@ Optional configuration settings .. _prefix: -Overall prefix path -^^^^^^^^^^^^^^^^^^^ +Overall prefix path (``--prefix``) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ *default:* ``$HOME/.local/easybuild`` From 7aa26fad7721805a8557ac7228dd018c4c62cb71 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Tue, 7 Apr 2015 17:27:31 +0200 Subject: [PATCH 36/69] final tweaks --- docs/Configuration.rst | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/docs/Configuration.rst b/docs/Configuration.rst index 36659d87..b17b541f 100644 --- a/docs/Configuration.rst +++ b/docs/Configuration.rst @@ -349,11 +349,9 @@ defaults: There are several ways in which the software and modules install path used by EasyBuild can be configured: -* using the direct configuration options ``--installpath-software`` and ``--installpath-modules`` (see - :ref:`installpath_direct_options`) -* using the parent install path configuration option ``--installpath`` (see :ref:`parent_installpath`) or overall - prefix path configuration option ``--prefix`` (see :ref:`prefix`), along with the ``--subdir-*`` and - ``--suffix-modules-path`` companion configuration options (see :ref:`installpath_subdirs`) +* using the direct configuration options ``--installpath-software`` and ``--installpath-modules`` (see below) +* via the parent install path configuration option ``--installpath`` (see below) +* via the overall prefix path configuration option ``--prefix`` (see :ref:`prefix`) .. _installpath_direct_options: @@ -381,7 +379,9 @@ corresponding module files. The install path for software and modules specifically is determined by combining ``--installpath`` with ``--subdir-software``, and combining ``--installpath`` with ``--subdir-modules`` and ``--suffix-modules-path``, -respectively. For more information on these companion configuration options, see :ref:`installpath_subdirs`. +respectively. + +For more information on these companion configuration options, see :ref:`installpath_subdirs`. Full install path for software and module file ++++++++++++++++++++++++++++++++++++++++++++++ @@ -394,6 +394,7 @@ install path named according to the active module naming scheme (default format: ``/-``). Additionally, symlinks to the actual module file are installed in a subdirectory of the modules install path named according to the value of the ``moduleclass`` easyconfig parameter. + For more information on the module naming scheme used by EasyBuild, see :ref:`module_naming_scheme`. Updating ``$MODULEPATH`` @@ -537,15 +538,15 @@ Software and modules install path subdirectories (``--subdir-software``, ``--sub * *modules install path subdirectory* (``--subdir-modules``): ``modules`` * *modules install path suffix* (``--suffix-modules-path``): ``all`` -The subdirectories for the software and modules install paths (relative to ``--installpath``) can be specified using -the corresponding dedicated configuration options (available since EasyBuild v1.14.0). +The subdirectories for the software and modules install paths (relative to ``--installpath``, see :ref:`installpath`) +can be specified using the corresponding dedicated configuration options (available since EasyBuild v1.14.0). For example: .. code:: shell-session $ export EASYBUILD_SUBDIR_SOFTWARE=installs - $ eb --subdir-modules=module_files ... + $ eb --installpath=$HOME/easybuild --subdir-modules=module_files ... Modules tool (``--modules-tool``) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ From a1a74696f629d5d1f93894e50b2de539304e736c Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Wed, 8 Apr 2015 00:30:38 +0200 Subject: [PATCH 37/69] fix remark --- docs/Changelog.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Changelog.rst b/docs/Changelog.rst index 0dba25e3..e5902d79 100644 --- a/docs/Changelog.rst +++ b/docs/Changelog.rst @@ -7,8 +7,8 @@ Changelog for EasyBuild documentation * add link to :ref:`unit_tests` page in dedicated section at :ref:`installation` page (see :ref:`install_running_unit_tests`) - * clarify relation between ``--installpath``, ``--prefix``, ``-subdir-*`` and ``-installpath-*`` configuration options - (see :ref:`installpath`) + * clarify relation between ``--installpath``, ``--prefix``, ``-subdir-*`` and ``--installpath-*`` + configuration options (see :ref:`installpath`) * **release 20150327.01** (`Mar 27th 2015`): From d2d8fd66f79a040750cbf22cbee91fdf9278a9a9 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Wed, 8 Apr 2015 00:36:09 +0200 Subject: [PATCH 38/69] include docs on --show-default-configfiles --- docs/Changelog.rst | 2 ++ docs/Configuration.rst | 15 +++++++++++++++ 2 files changed, 17 insertions(+) diff --git a/docs/Changelog.rst b/docs/Changelog.rst index e5902d79..b409adb3 100644 --- a/docs/Changelog.rst +++ b/docs/Changelog.rst @@ -9,6 +9,8 @@ Changelog for EasyBuild documentation (see :ref:`install_running_unit_tests`) * clarify relation between ``--installpath``, ``--prefix``, ``-subdir-*`` and ``--installpath-*`` configuration options (see :ref:`installpath`) + * mention ``--show-default-configfiles`` command line option in relevant section + (see :ref:`default_configuration_files`) * **release 20150327.01** (`Mar 27th 2015`): diff --git a/docs/Configuration.rst b/docs/Configuration.rst index b17b541f..e993615d 100644 --- a/docs/Configuration.rst +++ b/docs/Configuration.rst @@ -99,6 +99,21 @@ configuration files at ``/etc/easybuild.d/*.cfg`` and ``$HOME/.config/easybuild/ The configuration file located in ``$XDG_CONFIG_HOME`` will be listed *after* the ones obtained via ``$XDG_CONFIG_DIRS``, such that user-defined configuration settings have preference over system defaults. +A detailed overview of the list of default configuration files is available via ``eb --show-default-configfiles`` +(available since EasyBuild v2.1.0). For example:: + + $ XDG_CONFIG_DIRS=/tmp/etc:/tmp/moreetc eb --show-default-configfiles + Default list of configuration files: + + [with $XDG_CONFIG_HOME: (not set), $XDG_CONFIG_DIRS: /tmp/etc:/tmp/moreetc] + + * user-level: ${XDG_CONFIG_HOME:-$HOME/.config}/easybuild/config.cfg + -> /home/example/.config/easybuild/config.cfg => found + * system-level: ${XDG_CONFIG_DIRS:-/etc}/easybuild.d/*.cfg + -> {/tmp/etc, /tmp/moreetc}/easybuild.d/*.cfg => /tmp/etc/easybuild.d/config.cfg, /tmp/moreetc/easybuild.d/bar.cfg, /tmp/moreetc/easybuild.d/foo.cfg + + Default list of existing configuration files (4): /tmp/etc/easybuild.d/config.cfg, /tmp/moreetc/easybuild.d/bar.cfg, /tmp/moreetc/easybuild.d/foo.cfg, /home/example/.config/easybuild/config.cfg + Multiple configuration files ++++++++++++++++++++++++++++ From 4b5e4ef8aab56d39c4b2103660027aa39f860491 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Sat, 25 Apr 2015 15:47:11 +0200 Subject: [PATCH 39/69] add documentation on using external modules --- docs/Changelog.rst | 4 ++ docs/Using_external_modules.rst | 105 ++++++++++++++++++++++++++++++ docs/Writing_easyconfig_files.rst | 6 ++ docs/conf.py | 2 +- docs/index.rst | 1 + 5 files changed, 117 insertions(+), 1 deletion(-) create mode 100644 docs/Using_external_modules.rst diff --git a/docs/Changelog.rst b/docs/Changelog.rst index b409adb3..453620b9 100644 --- a/docs/Changelog.rst +++ b/docs/Changelog.rst @@ -3,6 +3,10 @@ Changelog for EasyBuild documentation ------------------------------------- +* **release 20150425.01** (`Apr 25th 2015`): + + * add documentation on :ref:`using_external_modules` + * **release 20150407.01** (`Apr 7th 2015`): * add link to :ref:`unit_tests` page in dedicated section at :ref:`installation` page diff --git a/docs/Using_external_modules.rst b/docs/Using_external_modules.rst new file mode 100644 index 00000000..60ac454c --- /dev/null +++ b/docs/Using_external_modules.rst @@ -0,0 +1,105 @@ +.. _using_external_modules: + +Using external modules +====================== + +Since EasyBuild v2.1, support for using modules that were not installed via EasyBuild is available. +We refer to such modules as *external modules*. + +This can be useful to reuse vendor-supplied modules, for example on Cray systems. + + +Using external modules as dependencies +--------------------------------------- + +External modules can be used as dependencies, by including the module name in the ``dependencies`` list (see +:ref:`dependency_specs`), along with the ``EXTERNAL_MODULE`` constant marker. + +For example, to specify the readily available module ``fftw/3.3.4.2`` as a dependency:: + + dependencies = [('fftw/3.3.4.2', EXTERNAL_MODULE)] + +For such dependencies, EasyBuild will: + +* load the module before initiating the software build and install procedure +* include a '``module load``' statement in the generated module file (for non-build dependencies) + +If the module is not available, EasyBuild will only report the module as being missing. It will *not* search for a +matching easyconfig file, or try to install the module in order to resolve the dependency. + + +Metadata for external modules +----------------------------- + +Since very little information is available for external modules based on the dependency specification alone (i.e., only +the module name), metadata can be supplied to EasyBuild for external modules. + +Using the ``--external-modules-metadata`` configuration option, the location of one or more files can be specified that +provide such metadata. The files are expected to be in INI format, with a section per module name and key-value +assignments denoting the metadata specific to that module. + +Format:: + + [modulename] + key1 = value1 + key2 = value2, value3 + +For example, the following file provides metadata for a handful of modules that may be provided on Cray systems:: + + [cray-hdf5/1.8.13] + name = HDF5 + version = 1.8.13 + prefix = HDF5_DIR + + [cray-hdf5-parallel/1.8.13] + name = HDF5 + version = 1.8.13 + prefix = /opt/cray/hdf5-parallel/1.8.13/GNU/49 + + [cray-netcdf/4.3.2] + name = netCDF, netCDF-Fortran + version = 4.3.2, 4.3.2 + prefix = NETCDF_DIR + + +The following keys are supported: + +* ``name``: specifies the software name(s) that is (are) provided by the module +* ``version``: specifies the software version(s) that is (are) provided by the module +* ``prefix``: specifies the installation prefix of the software that is provided by the module, or the name of an + environment variable defined by the module that contains the installation prefix + +Any other keys are simply ignored. + +.. note:: + When both ``name`` and ``version`` are specified, they must provide an *equal number of values* + (see for example the ``cray-netcdf`` example above). + +Using the provided metadata, EasyBuild will define environment variables that are also defined by modules that are +generated by EasyBuild itself. In particular, for each software name that is specified: + +* the corresponding environment variable ``$EBROOT`` is defined to the specified ``prefix`` value (if any) +* the corresponding environment variable ``$EBVERSION`` is defined to the corresponding ``version`` value (if any) + +For example, for the external modules for which metadata is provided in the example above, the following +environment variables are set in the build environment when the module is used as a dependency: + +* for ``cray-hdf5/1.8.1.13``: + + * ``$EBROOTHDF5`` = ``$HDF5_DIR`` + * ``$EBVERSIONHDF5`` = ``1.8.13`` + +* for ``cray-hdf5-parallel/1.8.13``: + + * ``$EBROOTHDF5`` = ``/opt/cray/hdf5-parallel/1.8.13/GNU/49`` + * ``$EBVERSIONHDF5`` = ``1.8.13`` + +* for ``cray-netcdf/4.3.2``: + + * ``$EBROOTNETCDF`` = ``$NETCDF_DIR`` + * ``$EBROOTNETCDFMINFORTRAN`` = ``$NETCDF_DIR`` + * ``$EBVERSIONNETCDF`` = ``4.3.2`` + * ``$EBVERSIONNETCDFMINFORTRAN`` = ``4.3.2`` + +The ``get_software_root`` and ``get_software_version`` functions that are commonly used occasionally in easyblocks +pick up the ``$EBROOT*`` and ``$EBVERSION*`` environment variables, respectively. diff --git a/docs/Writing_easyconfig_files.rst b/docs/Writing_easyconfig_files.rst index 7dcd8bfe..6a1f63ee 100644 --- a/docs/Writing_easyconfig_files.rst +++ b/docs/Writing_easyconfig_files.rst @@ -198,6 +198,12 @@ which is equivalent to:: dependencies = [('PnMPI', '1.2.0', '', True)] +Using external modules as dependencies +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Since EasyBuild v2.1, specifying modules that are not provided via EasyBuild as dependencies is also supported. +See :ref:`using_external_modules` for more information. + .. _configure_build_install_command_options: Configure/build/install command options diff --git a/docs/conf.py b/docs/conf.py index be19c05e..cd0fd0df 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -43,7 +43,7 @@ # The short X.Y version. version = '2.1.0dev' # this is meant to reference the version of EasyBuild # The full version, including alpha/beta/rc tags. -release = '20150407.01' # this is meant to reference the version of the documentation itself +release = '20150425.01' # this is meant to reference the version of the documentation itself # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: diff --git a/docs/index.rst b/docs/index.rst index 3e60597e..3363b94a 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -55,6 +55,7 @@ Advanced usage topics :maxdepth: 3 Integration_with_GitHub + Using_external_modules Other topics ============ From f909e27a3734344fd92f005899e58f79e7935298 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Sat, 25 Apr 2015 19:48:28 +0200 Subject: [PATCH 40/69] fix remark --- docs/Using_external_modules.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/Using_external_modules.rst b/docs/Using_external_modules.rst index 60ac454c..27e0642a 100644 --- a/docs/Using_external_modules.rst +++ b/docs/Using_external_modules.rst @@ -24,8 +24,9 @@ For such dependencies, EasyBuild will: * load the module before initiating the software build and install procedure * include a '``module load``' statement in the generated module file (for non-build dependencies) -If the module is not available, EasyBuild will only report the module as being missing. It will *not* search for a -matching easyconfig file, or try to install the module in order to resolve the dependency. +If the specified module is not available, EasyBuild will exit with an error message stating that the dependency can +not be resolved because the module could not be found. It will *not* search for a matching easyconfig file in order to +try and install the module to resolve the dependency. Metadata for external modules From 9985b0783b099105dd999063c77a605211f9bc1b Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Sun, 26 Apr 2015 21:46:05 +0200 Subject: [PATCH 41/69] add draft documentation on manipulating dependencies and partial installations --- docs/Changelog.rst | 5 +++++ docs/Manipulating_dependencies.rst | 6 ++++++ docs/Partial_installations.rst | 6 ++++++ docs/conf.py | 4 ++-- docs/index.rst | 2 ++ 5 files changed, 21 insertions(+), 2 deletions(-) create mode 100644 docs/Manipulating_dependencies.rst create mode 100644 docs/Partial_installations.rst diff --git a/docs/Changelog.rst b/docs/Changelog.rst index 453620b9..93bf0853 100644 --- a/docs/Changelog.rst +++ b/docs/Changelog.rst @@ -3,6 +3,11 @@ Changelog for EasyBuild documentation ------------------------------------- +* **release 20150427.01** (`Apr 27th 2015`): updates for EasyBuild v2.1.0 + + * add documentation on :ref:`partial_installations`, covering ``--stop``, ``--skip`` and ``--module-only`` + * add documentation on :ref:`manipulating_dependencies`, covering ``--filter-deps`` and ``--hide-deps`` + * **release 20150425.01** (`Apr 25th 2015`): * add documentation on :ref:`using_external_modules` diff --git a/docs/Manipulating_dependencies.rst b/docs/Manipulating_dependencies.rst new file mode 100644 index 00000000..bdff8a0c --- /dev/null +++ b/docs/Manipulating_dependencies.rst @@ -0,0 +1,6 @@ +.. _manipulating_dependencies: + +Manipulating dependencies +========================= + + diff --git a/docs/Partial_installations.rst b/docs/Partial_installations.rst new file mode 100644 index 00000000..4d85cc73 --- /dev/null +++ b/docs/Partial_installations.rst @@ -0,0 +1,6 @@ +.. _partial_installations: + +Partial installations +===================== + + diff --git a/docs/conf.py b/docs/conf.py index cd0fd0df..395d3a30 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -41,9 +41,9 @@ # other places throughout the built documents. # # The short X.Y version. -version = '2.1.0dev' # this is meant to reference the version of EasyBuild +version = '2.1.0' # this is meant to reference the version of EasyBuild # The full version, including alpha/beta/rc tags. -release = '20150425.01' # this is meant to reference the version of the documentation itself +release = '20150427.01' # this is meant to reference the version of the documentation itself # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: diff --git a/docs/index.rst b/docs/index.rst index 3363b94a..2cb13b74 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -55,6 +55,8 @@ Advanced usage topics :maxdepth: 3 Integration_with_GitHub + Manipulating_dependencies + Partial_installations Using_external_modules Other topics From 28e7f595fff41dfde97d7ddbaa8a3852af197077 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Sun, 26 Apr 2015 22:00:58 +0200 Subject: [PATCH 42/69] add subsections in pages on manipulating dependencies and partial installations --- docs/Manipulating_dependencies.rst | 4 ++++ docs/Partial_installations.rst | 18 ++++++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/docs/Manipulating_dependencies.rst b/docs/Manipulating_dependencies.rst index bdff8a0c..d1ad7256 100644 --- a/docs/Manipulating_dependencies.rst +++ b/docs/Manipulating_dependencies.rst @@ -3,4 +3,8 @@ Manipulating dependencies ========================= +Filtering out dependencies using ``--filter-deps`` +-------------------------------------------------- +Installing dependencies as hidden modules using ``--hidden-deps`` +----------------------------------------------------------------- diff --git a/docs/Partial_installations.rst b/docs/Partial_installations.rst index 4d85cc73..1b544063 100644 --- a/docs/Partial_installations.rst +++ b/docs/Partial_installations.rst @@ -3,4 +3,22 @@ Partial installations ===================== +Stopping the build and install procedure after a particular step using ``--stop`` +--------------------------------------------------------------------------------- +Skipping existing installations (partially) using ``-k``/``-skip`` +------------------------------------------------------------------ + +.. note:: + Unless the existing module is removed manually, using ``--skip`` requires ``--force`` to actually install + additional extensions and regenerating the module file. + +Only (re)generating module files using ``--module-only`` +-------------------------------------------------------- + +TODO: + +* ``--module-only`` (for example combined with ``--module-syntax=Lua``) +* ``--module-only`` with ``--force``: skip all steps, including sanity check: install module without actually installing +the software or making sure the software was installed before +* ``--module-only`` with 'migration' naming scheme From 30d1ec0645a8998f1dff11fc4dd8ea4fa8dac223 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Sun, 26 Apr 2015 22:03:10 +0200 Subject: [PATCH 43/69] fix typo --- docs/Manipulating_dependencies.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Manipulating_dependencies.rst b/docs/Manipulating_dependencies.rst index d1ad7256..e3398286 100644 --- a/docs/Manipulating_dependencies.rst +++ b/docs/Manipulating_dependencies.rst @@ -6,5 +6,5 @@ Manipulating dependencies Filtering out dependencies using ``--filter-deps`` -------------------------------------------------- -Installing dependencies as hidden modules using ``--hidden-deps`` ------------------------------------------------------------------ +Installing dependencies as hidden modules using ``--hide-deps`` +--------------------------------------------------------------- From 5f7befab75f3dc8ead7cf98264cf25054b6aa13d Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 27 Apr 2015 19:12:37 +0200 Subject: [PATCH 44/69] document --module-syntax, add note on errors being reported on unknown $EASYBUILD* environment variables --- docs/Configuration.rst | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/docs/Configuration.rst b/docs/Configuration.rst index e993615d..1a4b66d5 100644 --- a/docs/Configuration.rst +++ b/docs/Configuration.rst @@ -226,6 +226,8 @@ file. This can be used as an empty template configuration file: [basic] # Print build overview incl. dependencies (full paths) (def False) +.. _configuration_env_vars: + Environment variables ~~~~~~~~~~~~~~~~~~~~~ @@ -247,6 +249,8 @@ shown in the sections below. .. tip:: Any configuration option of EasyBuild which can be tuned by command line or via the configuration file, can also be tuned via a corresponding environment variable. +.. note:: If any ``$EASYBUILD``-prefixed environment variables are defined that do not correspond to a known + configuration option, EasyBuild will report an error message and exit. Command line arguments ~~~~~~~~~~~~~~~~~~~~~~ @@ -563,6 +567,8 @@ For example: $ export EASYBUILD_SUBDIR_SOFTWARE=installs $ eb --installpath=$HOME/easybuild --subdir-modules=module_files ... +.. _modules_tool: + Modules tool (``--modules-tool``) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -608,3 +614,32 @@ For more details, see the dedicated page: https://github.com/hpcugent/easybuild/ .. _`http://docs.python.org/2/library/configparser.html`: http://docs.python.org/2/library/configparser.html +.. _module_syntax: + +Module files syntax (``--module-syntax``) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +*default*: ``Tcl`` +*supported since*: EasyBuild v2.1 + +The syntax to use for generated module files can be specified using the ``--module-syntax`` configuration setting. + +Possible values are: + +* ``Lua``: generate module files in Lua syntax + * this requires the use of Lmod as a modules tool to consume the module files (see :ref:`modules_tool`) + * module file names will have the ``.lua`` extension +* ``Tcl``: generate module files in Tcl syntax + * Tcl module files can be consumed by all supported modules tools + * module files will contain a header string indicating that they are composed in Tcl syntax + +.. note:: + Lmod is able to deal with having module files in place in both Tcl and Lua syntax. When a module file in Lua + syntax (i.e., with a ``.lua`` file name extension) is available, a Tcl module file with the same name will be ignored. + +.. note:: + Using module files in Lua syntax has the advantage that Lmod does not need to translate from Lua to Tcl internally + when processing the module files, which benefits responsiveness of Lmod when used interactively by users. In terms + of Lmod-specific aspects of module files, the syntax of the module file does *not* matter; Lmod-specific statements + can be included in Tcl module files as well, by guarding them by a condition that only evaluates positively when + Lmod is consuming the module file. From 5392c2fce69f1bb07665d3a28028ab2a3a8bc136 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 27 Apr 2015 21:46:59 +0200 Subject: [PATCH 45/69] fix syntax issues, mention that Lua module files are ignored by Tcl-based env mods --- docs/Configuration.rst | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/Configuration.rst b/docs/Configuration.rst index 1a4b66d5..7e38c970 100644 --- a/docs/Configuration.rst +++ b/docs/Configuration.rst @@ -620,6 +620,7 @@ Module files syntax (``--module-syntax``) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ *default*: ``Tcl`` + *supported since*: EasyBuild v2.1 The syntax to use for generated module files can be specified using the ``--module-syntax`` configuration setting. @@ -627,15 +628,20 @@ The syntax to use for generated module files can be specified using the ``--modu Possible values are: * ``Lua``: generate module files in Lua syntax + * this requires the use of Lmod as a modules tool to consume the module files (see :ref:`modules_tool`) * module file names will have the ``.lua`` extension + * ``Tcl``: generate module files in Tcl syntax + * Tcl module files can be consumed by all supported modules tools - * module files will contain a header string indicating that they are composed in Tcl syntax + * module files will contain a header string ``#%Module`` indicating that they are composed in Tcl syntax .. note:: Lmod is able to deal with having module files in place in both Tcl and Lua syntax. When a module file in Lua - syntax (i.e., with a ``.lua`` file name extension) is available, a Tcl module file with the same name will be ignored. + syntax (i.e., with a ``.lua`` file name extension) is available, a Tcl module file with the same name will be + ignored. The Tcl-based environment modules tool will simply ignore module files in Lua syntax, since they do not + contain the header string that is included in Tcl module files. .. note:: Using module files in Lua syntax has the advantage that Lmod does not need to translate from Lua to Tcl internally From 704624fed41f520e9330c4b0dba8de91dfeb17dd Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 27 Apr 2015 21:50:39 +0200 Subject: [PATCH 46/69] mention Lmod-specific condition + conditional loads --- docs/Configuration.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/Configuration.rst b/docs/Configuration.rst index 7e38c970..1b9503ac 100644 --- a/docs/Configuration.rst +++ b/docs/Configuration.rst @@ -647,5 +647,7 @@ Possible values are: Using module files in Lua syntax has the advantage that Lmod does not need to translate from Lua to Tcl internally when processing the module files, which benefits responsiveness of Lmod when used interactively by users. In terms of Lmod-specific aspects of module files, the syntax of the module file does *not* matter; Lmod-specific statements - can be included in Tcl module files as well, by guarding them by a condition that only evaluates positively when - Lmod is consuming the module file. + supported by EasyBuild can be included in Tcl module files as well, by guarding them by a condition that only + evaluates positively when Lmod is consuming the module file, i.e. + ``if { [ string match "*tcl2lua.tcl" $env(_) ] } { ... }``. Only conditional load statements like + ``load(atleast("gcc","4.8"))`` can only be used in Lua module files. From d267af0b6ce72baaa4a1b59c25096238d64b44fd Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 27 Apr 2015 21:59:27 +0200 Subject: [PATCH 47/69] add quotes --- docs/Configuration.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Configuration.rst b/docs/Configuration.rst index 1b9503ac..214ed45a 100644 --- a/docs/Configuration.rst +++ b/docs/Configuration.rst @@ -649,5 +649,5 @@ Possible values are: of Lmod-specific aspects of module files, the syntax of the module file does *not* matter; Lmod-specific statements supported by EasyBuild can be included in Tcl module files as well, by guarding them by a condition that only evaluates positively when Lmod is consuming the module file, i.e. - ``if { [ string match "*tcl2lua.tcl" $env(_) ] } { ... }``. Only conditional load statements like - ``load(atleast("gcc","4.8"))`` can only be used in Lua module files. + '``if { [ string match "*tcl2lua.tcl" $env(_) ] } { ... }``'. Only conditional load statements like + '``load(atleast("gcc","4.8"))``' can only be used in Lua module files. From 51a348923c391373957fe410e15ce986eef40cad Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 27 Apr 2015 22:10:24 +0200 Subject: [PATCH 48/69] document prepending/appending to the robot search path --- docs/Using_the_EasyBuild_command_line.rst | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/Using_the_EasyBuild_command_line.rst b/docs/Using_the_EasyBuild_command_line.rst index 9d9854c8..3d044977 100644 --- a/docs/Using_the_EasyBuild_command_line.rst +++ b/docs/Using_the_EasyBuild_command_line.rst @@ -448,6 +448,7 @@ These two configuration options each serve a particular purpose, and together de * the specified list of paths is included *last* in the robot search path * by default, only the path to the easyconfig files that are part of the EasyBuild installation is listed * **note**: setting this configuration option implies redefining the default robot search path! + * append/prepend support For both options, the list of paths should be specified as a colon-separated (``:``) list. @@ -474,6 +475,23 @@ For more information on using constants in EasyBuild configuration files, see specified for ``--robot-paths``, on any configuration level, and vice versa. For more information on the relation between the different configuration types, see :ref:`configuration_types`. +.. _robot_search_path_prepend_append: + +Prepending and/or appending to the default robot search path +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +Prepending or appending to the default robot search path is supported via the ``--robot-paths`` configuration option. + +To *prepend* one or more paths, a list of paths followed by a '``:``' should be specified; analogously, to *append* +one or more paths, a list of paths preceded by a '``:``' should be specified. + +For example: + +* ``--robot-paths=/tmp/$USER:``: prepend ``/tmp/$USER`` to the robot search path +* ``export EASYBUILD_ROBOT_PATHS=:$HOME/myeasyconfigs:$HOME/test``: append ``$HOME/myeasyconfigs`` and ``$HOME/test`` to the + robot search path (in that order) +* ``--robot-paths=/tmp/$USER::$HOME/test``: prepend ``/tmp/$USER`` *and* append ``$HOME/test`` to the robot search path + Example use case ++++++++++++++++ From d257b8bfbbd8db8e2aa1c40fbbf10a8e440cdc3e Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 27 Apr 2015 22:11:33 +0200 Subject: [PATCH 49/69] fix syntax issue --- docs/Using_the_EasyBuild_command_line.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/Using_the_EasyBuild_command_line.rst b/docs/Using_the_EasyBuild_command_line.rst index 3d044977..b597f20e 100644 --- a/docs/Using_the_EasyBuild_command_line.rst +++ b/docs/Using_the_EasyBuild_command_line.rst @@ -25,10 +25,10 @@ This can be done in a number of ways: Whenever EasyBuild searches for easyconfig files, it considers a couple of locations, i.e. (in order of preference): -(i) the local working directory -(ii) the robot search path (see :ref:`robot_search_path`) +(i) the local working directory +(ii) the robot search path (see :ref:`robot_search_path`) (iii) the path to easyconfig files that are part of the active EasyBuild installation - (which is included in the default robot search path) + (which is included in the default robot search path) These locations are only considered for easyconfig files that are specified only by filename or using a relative path, *not* for easyconfig files that are specified via an absolute path. From 8925e32fbc170d7fcbe458b5a67ac2136d47de79 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 27 Apr 2015 22:13:21 +0200 Subject: [PATCH 50/69] fix long line --- docs/Using_the_EasyBuild_command_line.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/Using_the_EasyBuild_command_line.rst b/docs/Using_the_EasyBuild_command_line.rst index b597f20e..9e816efd 100644 --- a/docs/Using_the_EasyBuild_command_line.rst +++ b/docs/Using_the_EasyBuild_command_line.rst @@ -482,14 +482,14 @@ Prepending and/or appending to the default robot search path Prepending or appending to the default robot search path is supported via the ``--robot-paths`` configuration option. -To *prepend* one or more paths, a list of paths followed by a '``:``' should be specified; analogously, to *append* -one or more paths, a list of paths preceded by a '``:``' should be specified. +To *prepend* one or more paths, a list of paths followed by a '``:``' should be specified. + +Analogously, to *append* one or more paths, a list of paths preceded by a '``:``' should be specified. For example: -* ``--robot-paths=/tmp/$USER:``: prepend ``/tmp/$USER`` to the robot search path -* ``export EASYBUILD_ROBOT_PATHS=:$HOME/myeasyconfigs:$HOME/test``: append ``$HOME/myeasyconfigs`` and ``$HOME/test`` to the - robot search path (in that order) +* ``export EASYBUILD_ROBOT_PATHS=/tmp/$USER:``: prepend ``/tmp/$USER`` to the robot search path +* ``--robot-paths :$HOME/eb:$HOME/test``: append ``$HOME/eb`` and ``$HOME/test`` to the robot search path (in that order) * ``--robot-paths=/tmp/$USER::$HOME/test``: prepend ``/tmp/$USER`` *and* append ``$HOME/test`` to the robot search path Example use case From 81f5be866ae8ddc67ced902054348a2737df308a Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 27 Apr 2015 22:19:29 +0200 Subject: [PATCH 51/69] fix note --- docs/Using_the_EasyBuild_command_line.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Using_the_EasyBuild_command_line.rst b/docs/Using_the_EasyBuild_command_line.rst index 9e816efd..337d87fb 100644 --- a/docs/Using_the_EasyBuild_command_line.rst +++ b/docs/Using_the_EasyBuild_command_line.rst @@ -447,8 +447,8 @@ These two configuration options each serve a particular purpose, and together de * does *not* enable the dependency resolution mechanism * the specified list of paths is included *last* in the robot search path * by default, only the path to the easyconfig files that are part of the EasyBuild installation is listed - * **note**: setting this configuration option implies redefining the default robot search path! - * append/prepend support + * **note**: setting this configuration option implies redefining the default robot search path, unless a + prepending/appending list of paths is specified, see :ref:`robot_search_path_prepend_append` For both options, the list of paths should be specified as a colon-separated (``:``) list. From fdba5e96edd0d43c08ef4eaeacde75d9c2c757cd Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 27 Apr 2015 22:24:48 +0200 Subject: [PATCH 52/69] 'specifies to' --- docs/Using_the_EasyBuild_command_line.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/Using_the_EasyBuild_command_line.rst b/docs/Using_the_EasyBuild_command_line.rst index 337d87fb..70cbfbf0 100644 --- a/docs/Using_the_EasyBuild_command_line.rst +++ b/docs/Using_the_EasyBuild_command_line.rst @@ -488,9 +488,9 @@ Analogously, to *append* one or more paths, a list of paths preceded by a '``:`` For example: -* ``export EASYBUILD_ROBOT_PATHS=/tmp/$USER:``: prepend ``/tmp/$USER`` to the robot search path -* ``--robot-paths :$HOME/eb:$HOME/test``: append ``$HOME/eb`` and ``$HOME/test`` to the robot search path (in that order) -* ``--robot-paths=/tmp/$USER::$HOME/test``: prepend ``/tmp/$USER`` *and* append ``$HOME/test`` to the robot search path +* ``export EASYBUILD_ROBOT_PATHS=/tmp/$USER:`` specifies to prepend ``/tmp/$USER`` to the robot search path +* ``--robot-paths :$HOME/eb:$HOME/test`` specifies to append ``$HOME/eb`` and ``$HOME/test`` to the robot search path (in that order) +* ``--robot-paths=/tmp/$USER::$HOME/test`` specifies to prepend ``/tmp/$USER`` *and* append ``$HOME/test`` to the robot search path Example use case ++++++++++++++++ From 5512fdaf166785486284515dbc64c9ab8e469194 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 27 Apr 2015 22:39:32 +0200 Subject: [PATCH 53/69] complete documentation on --filter-deps and --hide-deps --- docs/Manipulating_dependencies.rst | 44 ++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/docs/Manipulating_dependencies.rst b/docs/Manipulating_dependencies.rst index e3398286..33e08347 100644 --- a/docs/Manipulating_dependencies.rst +++ b/docs/Manipulating_dependencies.rst @@ -3,8 +3,52 @@ Manipulating dependencies ========================= +A couple of different ways are available to manipulate the list of dependencies that are specified for the software +packages being installed. + +.. _filter_deps: + Filtering out dependencies using ``--filter-deps`` -------------------------------------------------- +To avoid that certain dependencies are being installed, a list of software names can be specified to ``--filter-deps``. +Any time a dependency with a name from this list is specified, it will be simply filtered out by EasyBuild, and +thus disregarded when resolving dependencies, loading modules for the dependencies in the build environment, and +including '``module load``' statements in the generated module files. + +For example: + +* overview of dependencies of HDF5:: + + $ eb HDF5-1.8.13-intel-2015a.eb -D + ... + * [ ] $CFGS/i/intel/intel-2015a.eb (module: intel/2015a) + * [ ] $CFGS/z/zlib/zlib-1.2.8-intel-2015a.eb (module: zlib/1.2.8-intel-2015a) + * [ ] $CFGS/s/Szip/Szip-2.1-intel-2015a.eb (module: Szip/2.1-intel-2015a) + * [ ] $CFGS/h/HDF5/HDF5-1.8.13-intel-2015a.eb (module: HDF5/1.8.13-intel-2015a) + +* overview of dependencies of HDF5, with zlib and Szip excluded:: + + $ eb HDF5-1.8.13-intel-2015a.eb --filter-deps=zlib,Szip -D + ... + * [ ] $CFGS/i/intel/intel-2015a.eb (module: intel/2015a) + * [ ] $CFGS/h/HDF5/HDF5-1.8.13-intel-2015a.eb (module: HDF5/1.8.13-intel-2015a) + + +.. _hide_deps: + Installing dependencies as hidden modules using ``--hide-deps`` --------------------------------------------------------------- + +Selected software packages can be marked to be installed as hidden modules (i.e., modules that are not visible via +'``module avail``', but can be loaded) whenever they are included as a dependency, via the ``--hide-deps`` +configuration option. + +For example (note the preceding '``.``' in the last part of the module names for zlib and Szip):: + + $ eb HDF5-1.8.13-intel-2015a.eb --hide-deps=zlib,Szip -D + ... + * [ ] $CFGS/i/intel/intel-2015a.eb (module: intel/2015a) + * [ ] $CFGS/z/zlib/zlib-1.2.8-intel-2015a.eb (module: zlib/.1.2.8-intel-2015a) + * [ ] $CFGS/s/Szip/Szip-2.1-intel-2015a.eb (module: Szip/.2.1-intel-2015a) + * [ ] $CFGS/h/HDF5/HDF5-1.8.13-intel-2015a.eb (module: HDF5/1.8.13-intel-2015a) From 02a432e65fb8f991b3c2b9c1de925c3979b2d9b6 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 27 Apr 2015 22:51:43 +0200 Subject: [PATCH 54/69] mention OS pkgs with --filter-deps, add note on Lmod's --show-hidden with --hide-deps --- docs/Manipulating_dependencies.rst | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/Manipulating_dependencies.rst b/docs/Manipulating_dependencies.rst index 33e08347..996b1b72 100644 --- a/docs/Manipulating_dependencies.rst +++ b/docs/Manipulating_dependencies.rst @@ -16,6 +16,9 @@ Any time a dependency with a name from this list is specified, it will be simply thus disregarded when resolving dependencies, loading modules for the dependencies in the build environment, and including '``module load``' statements in the generated module files. +This can be useful when particular tools and libraries are already provided by OS packages (or in some other way), +and should not be reinstalled as modules by EasyBuild. + For example: * overview of dependencies of HDF5:: @@ -52,3 +55,18 @@ For example (note the preceding '``.``' in the last part of the module names for * [ ] $CFGS/z/zlib/zlib-1.2.8-intel-2015a.eb (module: zlib/.1.2.8-intel-2015a) * [ ] $CFGS/s/Szip/Szip-2.1-intel-2015a.eb (module: Szip/.2.1-intel-2015a) * [ ] $CFGS/h/HDF5/HDF5-1.8.13-intel-2015a.eb (module: HDF5/1.8.13-intel-2015a) + +.. note:: + Using Lmod (version >= 5.75), hidden modules can be made visible in the output of '``module avail`` using the + ``--show-hidden`` option, for example:: + $ module avail bzip2 + + Use "module spider" to find all possible modules. + Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". + + $ module --show-hidden avail bzip2 + ----- /home/example/.local/easybuild/modules/all ----- + bzip2/.1.0.6 + + Use "module spider" to find all possible modules. + Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". From 655205cb01228ef4a01467dd217366aa2bb7d99e Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 27 Apr 2015 22:54:07 +0200 Subject: [PATCH 55/69] fix syntax issue --- docs/Manipulating_dependencies.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/Manipulating_dependencies.rst b/docs/Manipulating_dependencies.rst index 996b1b72..16a5043e 100644 --- a/docs/Manipulating_dependencies.rst +++ b/docs/Manipulating_dependencies.rst @@ -58,7 +58,8 @@ For example (note the preceding '``.``' in the last part of the module names for .. note:: Using Lmod (version >= 5.75), hidden modules can be made visible in the output of '``module avail`` using the - ``--show-hidden`` option, for example:: + ``--show-hidden`` option, for example:: + $ module avail bzip2 Use "module spider" to find all possible modules. From 11a2a0923da7972a0f60277205a322067d736239 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 27 Apr 2015 22:57:46 +0200 Subject: [PATCH 56/69] fix typo --- docs/Manipulating_dependencies.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/Manipulating_dependencies.rst b/docs/Manipulating_dependencies.rst index 16a5043e..76feb2ac 100644 --- a/docs/Manipulating_dependencies.rst +++ b/docs/Manipulating_dependencies.rst @@ -57,8 +57,10 @@ For example (note the preceding '``.``' in the last part of the module names for * [ ] $CFGS/h/HDF5/HDF5-1.8.13-intel-2015a.eb (module: HDF5/1.8.13-intel-2015a) .. note:: - Using Lmod (version >= 5.75), hidden modules can be made visible in the output of '``module avail`` using the - ``--show-hidden`` option, for example:: + Using Lmod (version >= 5.7.5), hidden modules can be made visible in the output of '``module avail``' using the + ``--show-hidden`` option. + + For example:: $ module avail bzip2 From b163847998534d765e4cfe94d70135ee7bc2ea92 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 27 Apr 2015 23:05:14 +0200 Subject: [PATCH 57/69] include reference to --hide-deps section where hiddendependencies is listd --- docs/Writing_easyconfig_files.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Writing_easyconfig_files.rst b/docs/Writing_easyconfig_files.rst index 6a1f63ee..f583e4b2 100644 --- a/docs/Writing_easyconfig_files.rst +++ b/docs/Writing_easyconfig_files.rst @@ -146,7 +146,7 @@ Dependencies * **dependencies**: build/runtime dependencies * **builddependencies**: build-only dependencies (not in module) -* **hiddendependencies**: dependencies via hidden modules +* **hiddendependencies**: dependencies via hidden modules (see also :ref:`hide_deps`) * **osdependencies**: system dependencies (package names) Remarks: From fbe63692185cbf15ee3c622080e8fd30ac15f410 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Wed, 29 Apr 2015 09:57:34 +0200 Subject: [PATCH 58/69] update release notes for EasyBuild v2.1.0 --- docs/Release_notes.rst | 150 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 147 insertions(+), 3 deletions(-) diff --git a/docs/Release_notes.rst b/docs/Release_notes.rst index b0a5f1cd..f0dea569 100644 --- a/docs/Release_notes.rst +++ b/docs/Release_notes.rst @@ -3,9 +3,153 @@ EasyBuild release notes ======================= -The latest version of EasyBuild provides support for building and installing **581** different software packages, -using 32 different (compiler) toolchains. It contains 143 software-specific easyblocks and 21 generic easyblocks, -alongside 3,344 easyconfig files. +The latest version of EasyBuild provides support for building and installing **608** different software packages, +using 33 different (compiler) toolchains. It contains 145 software-specific easyblocks and 22 generic easyblocks, +alongside 3,617 easyconfig files. + +v2.1.0 (April 28th 2015) +------------------------ + +feature + bugfix release + +**framework** + +* requires vsc-base v2.2.0 or more recent + + * added support for LoggedException (`vsc-base#160 `_, `vsc-base#163 `_, `vsc-base#165 `_, `vsc-base#167 `_) + * added support for add_flex action in GeneralOption (`vsc-base#162 `_) + * added support to GeneralOption to act on unknown configuration environment variables (`vsc-base#162 `_) + +* add support for only (re)generating module files: ``--module-only`` (`#1018 `_) + + * module naming scheme API is enhanced to include det_install_subdir method + * see :ref:`module_only` + +* add support for generating module files in Lua syntax (note: required Lmod as modules tool) (`#1060 `_, `#1255 `_, `#1256 `_) + + * see ``--module-syntax`` configuration option and :ref:`module_syntax` + +* deprecate ``log.error`` method in favor of raising ``EasyBuildError`` exception (`#1218 `_) + + * see :ref:`depr_error_reporting` + +* add support for using external modules as dependencies, and to provide metadata for external modules (`#1230 `_, `#1265 `_, `#1267 `_) + + * see :ref:`using_external_modules` + +* add experimental support for Cray toolchains on top of PrgEnv modules: ``CrayGNU``, ``CrayIntel``, ``CrayCCE`` (`#1234 `_, `#1268 `_) + + * see https://github.com/hpcugent/easybuild/wiki/EasyBuild-on-Cray + +* various other enhancements, including: + + * sort the results of searching for files (e.g., ``--search`` output) (`#1214 `_) + * enhance test w.r.t. use of templates in cfgfile (`#1217 `_) + * define '``%(DEFAULT_REPOSITORYPATH)s``' template for cfgfiles (see ``eb --avail-cfgfile-constants``) (`#1220 `_) + * also reset ``$LD_PRELOAD`` when running module commands, in case module defined ``$LD_PRELOAD`` (`#1222 `_) + * move location of '``module use``' statements in generated module file (*after* '``module load``' statements) (`#1232 `_) + * add support for ``--show-default-configfiles`` (`#1240 `_) + + * see :ref:`default_configuration_files` + + * report error on missing configuration files, rather than ignoring them (`#1240 `_) + * clean up commit message used in easyconfig git repository (`#1248 `_) + * add ``--hide-deps`` configuration option to specify names of software that must be installed as hidden modules (`#1250 `_) + + * see :ref:`hide_deps` + + * add support for appending/prepending to ``--robot-paths`` to avoid overwriting default robot search path (`#1252 `_) + + * see :ref:`robot_search_path_prepend_append` + + * enable detection of use of unknown ``$EASYBUILD``-prefixed environment variables (`#1253 `_) + + * see :ref:`configuration_env_vars` + + * add ``--installpath-modules`` and ``--installpath-software`` configuration options (`#1258 `_) + + * see :ref:`installpath` + + * use dedicated subdirectory in temporary directory for each test to ensure better cleanup (`#1260 `_) + * get rid of ``$PROFILEREAD`` hack when running commands, not needed anymore (`#1264 `_) +* various bug fixes, including: + + * make bootstrap script robust against having ``vsc-base`` already available in Python search path (`#1212 `_, `#1215 `_) + * set default value for unpack_options easyconfig parameter to ``''``, so ``self.cfg.update`` works on it (`#1229 `_) + * also copy rotated log files (`#1238 `_) + * fix parsing of ``--download-timeout`` value (`#1242 `_) + * make ``test_XDG_CONFIG_env_vars`` unit test robust against existing user config file in default location (`#1259 `_) + * fix minor robustness issues w.r.t. ``$XDG_CONFIG*`` and ``$PYTHONPATH`` in unit tests (`#1262 `_) + * fix issue with handling empty toolchain variables (`#1263 `_) + +**easyblocks** + +* replace '``log.error``' with '``raise EasyBuildError``' in all easyblocks (`#588 `_) +* one new generic easyblock: ``ConfigureMakePythonPackage`` (`#540 `_) +* new easyblocks for 2 software packages that require customized support: + + * TINKER (`#578 `_), Xmipp (`#581 `_) + +* various other enhancements, including: + + * enhance WIEN2k easyblock for recent versions + cleanup (`#486 `_) + * define ``$PYTHONNOUSERSITE`` in PythonPackage easyblock so user-installed packages are not picked up (`#577 `_) + * add support in CP2K easyblock for building on top of MPICH/MPICH2 (`#579 `_) + * enhance Hadoop easyblock to support parallel builds (`#580 `_) + * drop ``-noroot`` for recent FLUENT versions, honor ``installopts``, enable ``-debug`` (`#582 `_) + * include ``prebuildopts`` in build command for Python packages (`#585 `_) + * also install ``rosetta_tools`` subdirectory for Rosetta (`#586 `_) + * update SLEPc easyblock for v3.5 + style cleanup (`#593 `_) + * minor fix in HPL easyblock w.r.t. checking defined environment variables (`#595 `_) + * tweak CP2K easyblock w.r.t. LAPACK/FFTW support (`#596 `_) + * minor update to GCC easyblock to support GCC v5.x (`#598 `_) + * update sanity check in R easyblock for version 3.2.0 (`#602 `_) + +* various bug fixes, including: + + * fix definition of ``$MKLROOT``, should be set to '``mkl``' subdir of install dir (`#576 `_) + * add ``-libmpichf90`` to list of MPI libraries in NWChem easyblock (`#584 `_) + * stop using '``$root``' to make easyblocks compatible with module files in Lua syntax (`#590 `_) + * also set ``$PYTHONPATH`` before installing Python package in temporary directory in ``test_step`` (`#591 `_) + * unset ``buildopts``/``installopts`` before installing Python extensions in Python easyblock (`#597 `_) + * allow not including vsc-base sources when installing EasyBuild (gets installed with easybuild-framework) (`#600 `_) + * use ``self.initial_environ`` rather than ``self.orig_environ`` in EasyBuildMeta easyblock (`#600 `_) + +**easyconfigs** + +* added example easyconfig files for 27 new software packages: + + * AFNI (`#1322 `_, `#1521 `_), BCFtools (`#1492 `_), getdp (`#1518 `_), gmsh (`#1518 `_), gtest (`#1244 `_), hanythingondemand (`#1530 `_), mawk (`#1369 `_), Minimac (`#815 `_), Minimac3 (`#1502 `_), monty (`#1548 `_), Octave (`#1563 `_), pbs_python (`#1530 `_), pigz (`#1036 `_), Pygments (`#1536 `_), pyhull (`#1539 `_), pymatgen (`#1549 `_), PyQt (`#1322 `_, `#1521 `_), Ray (`#1494 `_), requests (`#1536 `_), seqtk (`#1524 `_), SIP (`#1322 `_, `#1521 `_), S-Lang (`#1369 `_), Spark (`#1554 `_), spglib (`#1549 `_), TINKER (`#1465 `_), tmux (`#1369 `_), Xmipp (`#1489 `_) + +* added easyconfigs for new (Cray-specific) toolchains (`#1538 `_): ``CrayGNU``, ``CrayIntel``, ``CrayCCE`` + + * initially supported software (using CrayGNU toolchains): CP2K, GROMACS, HPL, Python + numpy/scipy, WRF (`#1558 `_) + * see also https://github.com/hpcugent/easybuild/wiki/EasyBuild-on-Cray + +* added new easyconfigs for existing toolchains: ``goolf/1.5.16``, ``intel/2014.06`` +* added additional easyconfigs for various supported software packages: version updates, different toolchains, ... + + * including GCC v5.1.0, OpenFOAM v2.3.1, R v3.1.3 and v3.2.0, PETSc/SLEPc v3.5.3, WIEN2k v14.1 + +* various other enhancements, including: + + * include pbr dependency for lockfile Python extension in Python v2.7.9 easyconfigs + mock/pytz/pandas (`#1462 `_, `#1540 `_) + * include SQLite as dependency in Python v2.7.9 easyconfigs (`#1468 `_) + * set ``$LD_PRELOAD`` for Hoard and jemalloc (`#1470 `_) + * fix homepage in SCOTCH easyconfigs (`#1485 `_) + * adding missing six/ecdsa dependencies for dateutil/paramiko Python packages in Python easyconfigs (`#1504 `_, `#1505 `_, `#1506 `_, `#1507 `_, `#1508 `_, `#1509 `_, `#1510 `_) + * enable ``pic`` toolchain option in expat easyconfigs (`#1562 `_) + * extend list of source URLs for Bioconductor packages in R easyconfigs to include 'release' source URLs (`#1568 `_) + +* various bug fixes, including: + + * adding missing zlib dependency in all Tcl easyconfig files (`#1344 `_) + * fix homepage in FLUENT easyconfigs (`#1472 `_) + * use ``--with-verbs`` rather than deprecated ``--with-openib`` in OpenMPI configure options (`#1511 `_) + * stop relying on ``OS_NAME`` constant to specify OS dependencies in OpenMPI easyconfigs (`#1512 `_) + * replace use of '``$root``' with '``%(installdir)s``' to ensure compatibility with module files in Lua syntax (`#1532 `_) + * stop relying on '``$MKLROOT``' in ROOT easyconfigs (`#1537 `_) + * use proper Bundle easyblock for biodeps/PRACE (`#1566 `_) v2.0.0 (March 6th 2015) ----------------------- From c34ff9a7c7c85eb315b6a32cfb656435d8bfbd5b Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Wed, 29 Apr 2015 23:15:07 +0200 Subject: [PATCH 59/69] include last-minute merged PRs in release notes --- docs/Release_notes.rst | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/Release_notes.rst b/docs/Release_notes.rst index f0dea569..1ca5d99c 100644 --- a/docs/Release_notes.rst +++ b/docs/Release_notes.rst @@ -25,7 +25,7 @@ feature + bugfix release * module naming scheme API is enhanced to include det_install_subdir method * see :ref:`module_only` -* add support for generating module files in Lua syntax (note: required Lmod as modules tool) (`#1060 `_, `#1255 `_, `#1256 `_) +* add support for generating module files in Lua syntax (note: required Lmod as modules tool) (`#1060 `_, `#1255 `_, `#1256 `_, `#1270 `_) * see ``--module-syntax`` configuration option and :ref:`module_syntax` @@ -43,6 +43,7 @@ feature + bugfix release * various other enhancements, including: + * clear list of checksums when using ``--try-software-version`` (`#1169 `_) * sort the results of searching for files (e.g., ``--search`` output) (`#1214 `_) * enhance test w.r.t. use of templates in cfgfile (`#1217 `_) * define '``%(DEFAULT_REPOSITORYPATH)s``' template for cfgfiles (see ``eb --avail-cfgfile-constants``) (`#1220 `_) @@ -72,6 +73,7 @@ feature + bugfix release * use dedicated subdirectory in temporary directory for each test to ensure better cleanup (`#1260 `_) * get rid of ``$PROFILEREAD`` hack when running commands, not needed anymore (`#1264 `_) + * various bug fixes, including: * make bootstrap script robust against having ``vsc-base`` already available in Python search path (`#1212 `_, `#1215 `_) @@ -107,6 +109,7 @@ feature + bugfix release * various bug fixes, including: + * fix Score-P easyblock for compiler-only toolchains, include Qt as optional dependecy (`#552 `_) * fix definition of ``$MKLROOT``, should be set to '``mkl``' subdir of install dir (`#576 `_) * add ``-libmpichf90`` to list of MPI libraries in NWChem easyblock (`#584 `_) * stop using '``$root``' to make easyblocks compatible with module files in Lua syntax (`#590 `_) @@ -114,6 +117,8 @@ feature + bugfix release * unset ``buildopts``/``installopts`` before installing Python extensions in Python easyblock (`#597 `_) * allow not including vsc-base sources when installing EasyBuild (gets installed with easybuild-framework) (`#600 `_) * use ``self.initial_environ`` rather than ``self.orig_environ`` in EasyBuildMeta easyblock (`#600 `_) + * make GCC easyblock compatible with ``--module-only`` by setting default value for ``self.platform_lib`` in constructor (`#603 `_) + **easyconfigs** @@ -150,6 +155,9 @@ feature + bugfix release * replace use of '``$root``' with '``%(installdir)s``' to ensure compatibility with module files in Lua syntax (`#1532 `_) * stop relying on '``$MKLROOT``' in ROOT easyconfigs (`#1537 `_) * use proper Bundle easyblock for biodeps/PRACE (`#1566 `_) + * make source_urls in Cube and Scalasca easyconfigs compatible with --try-software-version (`#1574 `_) + * add patch for Cube to fix configure script w.r.t. Qt dependency, add --without-java-reader configure option (`#1574 `_) + v2.0.0 (March 6th 2015) ----------------------- From 66aaf0e38d73082f3c3f33784114857e79ab236a Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Wed, 29 Apr 2015 23:15:30 +0200 Subject: [PATCH 60/69] flesh out Changelog --- docs/Changelog.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/Changelog.rst b/docs/Changelog.rst index 93bf0853..fa8e2797 100644 --- a/docs/Changelog.rst +++ b/docs/Changelog.rst @@ -7,6 +7,10 @@ Changelog for EasyBuild documentation * add documentation on :ref:`partial_installations`, covering ``--stop``, ``--skip`` and ``--module-only`` * add documentation on :ref:`manipulating_dependencies`, covering ``--filter-deps`` and ``--hide-deps`` + * document ``-module-syntax`` configuration option (see :ref:`module_syntax`) + * add note on detection of unknown ``$EASYBUILD``-prefixed environment variables (see :ref:`:ref:`configuration_env_vars`) + * mention support for prepending/appending to ``--robot-paths`` (see :ref:`robot_search_path_prepend_append`) + * update release notes for EasyBuild v2.1.0 (see :ref:`release_notes`) * **release 20150425.01** (`Apr 25th 2015`): From 7671cfd9ea744ef5ef186e4f2853b2734feafe7e Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Thu, 30 Apr 2015 08:54:45 +0200 Subject: [PATCH 61/69] fix easyconfigs count + typo --- docs/Release_notes.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Release_notes.rst b/docs/Release_notes.rst index 1ca5d99c..459b8580 100644 --- a/docs/Release_notes.rst +++ b/docs/Release_notes.rst @@ -5,7 +5,7 @@ EasyBuild release notes The latest version of EasyBuild provides support for building and installing **608** different software packages, using 33 different (compiler) toolchains. It contains 145 software-specific easyblocks and 22 generic easyblocks, -alongside 3,617 easyconfig files. +alongside 3,618 easyconfig files. v2.1.0 (April 28th 2015) ------------------------ @@ -25,7 +25,7 @@ feature + bugfix release * module naming scheme API is enhanced to include det_install_subdir method * see :ref:`module_only` -* add support for generating module files in Lua syntax (note: required Lmod as modules tool) (`#1060 `_, `#1255 `_, `#1256 `_, `#1270 `_) +* add support for generating module files in Lua syntax (note: requires Lmod as modules tool) (`#1060 `_, `#1255 `_, `#1256 `_, `#1270 `_) * see ``--module-syntax`` configuration option and :ref:`module_syntax` From 4401458217b418143e36b728ba54ff051ad84978 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Thu, 30 Apr 2015 12:26:48 +0200 Subject: [PATCH 62/69] also cover extensions in concepts/terminolgoy page --- docs/Changelog.rst | 1 + docs/Concepts_and_Terminology.rst | 12 ++++++++++++ 2 files changed, 13 insertions(+) diff --git a/docs/Changelog.rst b/docs/Changelog.rst index fa8e2797..0df50e12 100644 --- a/docs/Changelog.rst +++ b/docs/Changelog.rst @@ -5,6 +5,7 @@ Changelog for EasyBuild documentation * **release 20150427.01** (`Apr 27th 2015`): updates for EasyBuild v2.1.0 + * also cover 'extensions' in page on concepts and terminology (see :ref:`extensions`) * add documentation on :ref:`partial_installations`, covering ``--stop``, ``--skip`` and ``--module-only`` * add documentation on :ref:`manipulating_dependencies`, covering ``--filter-deps`` and ``--hide-deps`` * document ``-module-syntax`` configuration option (see :ref:`module_syntax`) diff --git a/docs/Concepts_and_Terminology.rst b/docs/Concepts_and_Terminology.rst index cfdb59d9..f8eef70a 100644 --- a/docs/Concepts_and_Terminology.rst +++ b/docs/Concepts_and_Terminology.rst @@ -153,3 +153,15 @@ Therefore, reproducing installations becomes trivial. EasyBuild version |version| includes support for over 511 different software packages, spread over 2800 easyconfig files describing distinct builds. + +.. _extensions: + +Extensions +---------- + +Some software packages support installing additional add-ons alongside the 'main' software, either in the same +installation prefix, or in a separate location. Well-known examples include Perl modules (http://www.cpan.org/modules/), +Python packages (https://pypi.python.org/pypi), R libraries (http://cran.r-project.org/web/packages/), +Ruby gems (http://guides.rubygems.org/what-is-a-gem/), etc. + +In EasyBuild, we use the neutral term '**extensions**' to refer these add-ons. From 23783dc984762858a50e7e25460601a279b4595b Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Thu, 30 Apr 2015 12:27:15 +0200 Subject: [PATCH 63/69] flesh out page on partial installation support --- docs/Partial_installations.rst | 265 +++++++++++++++++++++++++++++++-- 1 file changed, 254 insertions(+), 11 deletions(-) diff --git a/docs/Partial_installations.rst b/docs/Partial_installations.rst index 1b544063..db9daf3f 100644 --- a/docs/Partial_installations.rst +++ b/docs/Partial_installations.rst @@ -3,22 +3,265 @@ Partial installations ===================== -Stopping the build and install procedure after a particular step using ``--stop`` ---------------------------------------------------------------------------------- +Several ways of performing partial installations are supported. These may be useful when debugging a particular issue +with the installation procedure being performed by EasyBuild, updating existing software installations or module files, +or after changing the EasyBuild configuration (e.g., switching to module files in Lua syntax or a different module +naming scheme). -Skipping existing installations (partially) using ``-k``/``-skip`` ------------------------------------------------------------------- +* :ref:`partial_installation_stop` +* :ref:`partial_installation_skip` +* :ref:`module_only` + + * :ref:`module_only_only_regenerate` + * :ref:`module_only_existing_different` + * :ref:`module_only_force` + +.. _partial_installation_stop: + +Stopping the installation procedure *after* a step using ``-s``/``--stop`` +-------------------------------------------------------------------------- + +To stop the installation procedure *after* a specific step in the installation procedure, the ``-s``/``--stop`` +command line option can be used; the name of the step must be supplied as an argument. + +The following step names are recognized (listed in execution order): ``fetch``, ``ready``, ``source``, ``patch``, +``prepare``, ``configure``, ``build``, ``test``, ``install``, ``extensions``, ``package``, ``postproc``, +``sanitycheck``, ``cleanup``, ``module``, ``testcases``. + +Example usage:: + + $ eb GCC-4.9.2.eb --stop configure + == temporary log file in case of crash /tmp/eb-X2Z0b7/easybuild-mGxmNb.log + == processing EasyBuild easyconfig /home/example/GCC-4.9.2.eb + == building and installing GCC/4.9.2... + == fetching files... + == creating build dir, resetting environment... + == unpacking... + == patching... + == preparing... + == configuring... + == COMPLETED: Installation STOPPED successfully + == Results of the build can be found in the log file /dev/shm/example/GCC/4.9.2/dummy-dummy/easybuild/easybuild-GCC-4.9.2-20150430.091644.log + == Build succeeded for 1 out of 1 + == temporary log file(s) /tmp/eb-X2Z0b7/easybuild-mGxmNb.log* have been removed. + == temporary directory /tmp/eb-X2Z0b7 has been removed. + + +.. _partial_installation_skip: + +Installing additional extensions using ``-k``/``-skip`` +------------------------------------------------------- + +For software applications that may include :ref:`extensions`, it is often required to install one or more additional +extensions without having to reinstall the software package (and all extensions) from scratch. + +For this purpose, the ``-k``/``--skip`` command line option is available. + +To actually skip an existing software installation and all installed extensions, a corresponding module must be +available already; if not, the installation procedure will be performed from scratch. + +To trigger the installation of missing extensions, ``--force`` must be used as well; without it, the installation +procedure will be skipped as a whole (since the module is already available). + +When ``--skip`` is combined with ``--force``, EasyBuild will: + +i) ensure all (extension) sources are available (and try to fetch them if needed) +ii) prepare the build environment; +iii) check which extensions have not been installed yet; +iv) install the missing extensions; +v) run the sanity check (which includes checking that all extensions are available) +vi) regenerate the module file (since it contains a list of installed extensions) + +Example usage:: + + $ eb Python-2.7.9-intel-2015a.eb --skip + ... + == Python/2.7.9-intel-2015a is already installed (module found), skipping + == No easyconfigs left to be built. + == Build succeeded for 0 out of 0 + +.. code:: + + $ eb Python-2.7.9-intel-2015a.eb --skip --force + ... + == building and installing Python/2.7.9-intel-2015a... + ... + == configuring [skipped] + == building [skipped] + == testing [skipped] + == installing [skipped] + == taking care of extensions... + == packaging [skipped] + == postprocessing [skipped] + == sanity checking... + == cleaning up... + == creating module... + == COMPLETED: Installation ended successfully .. note:: - Unless the existing module is removed manually, using ``--skip`` requires ``--force`` to actually install - additional extensions and regenerating the module file. + The '``skipsteps``' easyconfig parameter has a different purpose, i.e. to specify which installation steps should + *always* be skipped when the installation of a particular software package is performed, no matter when the software + or corresponding module is already available or not. + +.. _module_only: Only (re)generating module files using ``--module-only`` -------------------------------------------------------- -TODO: +Since EasyBuild v2.1, it is possible to only (re)generate the module file that matches the specifications in the +easyconfig file, using ``--module-only``. Depending on the use case, additional options should be supplied. + +Usually, ``--force`` is required either to ignore the existing module file (if the module is available under the same +name as the one being (re)generated), or to skip the sanity check that verifies the software installation (if no +software installation is available). + +Combining ``--module-only`` with ``--installpath-modules`` is also a common use case, to generate the module file in +a (test) location other than the software installation prefix (see :ref:`installpath`). + +Use cases: + +* :ref:`module_only_only_regenerate` +* :ref:`module_only_existing_different` +* :ref:`module_only_force` + +.. _module_only_only_regenerate: + +Only (re)generate (existing) module file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To only generate a module file (i.e., skip actually building and installing the software), or to regenerate an +existing module file, ``--module-only`` can be used. + +In the former case, enabling ``--force`` is required because the sanity check step that verifies whether the +installation produced the expected files and/or directories is not skipped unless forced. +In the latter case, ``--force`` must be used to make EasyBuild ignore that the module is already available +according to the modules tool. + +Example usage: + +* only generate module file:: + + $ module avail GCC + ---------------------------- /home/example/.local/modules/all ---------------------------- + GCC/4.8.2 + + $ eb GCC-5.1.0.eb --module-only --force + ... + == building and installing GCC/5.1.0... + == fetching files [skipped] + ... + == configuring [skipped] + == building [skipped] + == testing [skipped] + == installing [skipped] + ... + == sanity checking [skipped] + == cleaning up [skipped] + == creating module... + == COMPLETED: Installation ended successfully + ... + + $ module avail GCC + + ---------------------------- /home/example/.local/modules/all ---------------------------- + GCC/4.8.2 GCC/5.1.0 + +* regenerate existing module file:: + + $ module avail GCC/4.8.2 + + ---------------------------- /home/example/.local/modules/all ---------------------------- + GCC/4.8.2 + + $ ls -l /home/example/.local/modules/all/GCC/4.8.2 + -rw-rw-r-- 1 example example 1002 Jan 11 17:19 /home/example/.local/modules/all/GCC/4.8.2 + + $ eb GCC-4.8.2.eb --module-only --force + ... + == building and installing GCC/4.8.2... + ... + == sanity checking [skipped] + == creating module... + == COMPLETED: Installation ended successfully + ... + + $ ls -l /home/example/.local/modules/all/GCC/4.8.2* + -rw-rw-r-- 1 example example 1064 Apr 30 10:54 /home/example/.local/modules/all/GCC/4.8.2 + +.. _module_only_existing_different: + +Generate an additional module file for an existing installation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Generating an additional module file for existing installations is also supported. This can achieved by combining +``--module-only`` with additional configuration options that apply the module generator backend. + +Examples: + +* to generate a module file in Lua syntax and put it next to the already existing module file in Tcl syntax, + ``--module-only --module-syntax=Lua`` can be used:: + + $ module avail GCC/4.8.2 + + ---------------------------- /home/example/.local/modules/all ---------------------------- + GCC/4.8.2 + + $ ls -l /home/example/.local/modules/all/GCC/4.8.2* + -rw-rw-r-- 1 example example 1064 Apr 30 10:54 /home/example/.local/modules/all/GCC/4.8.2 + + $ eb GCC-4.8.2.eb --modules-tool=Lmod --module-only --module-syntax=Lua --force + ... + == building and installing GCC/4.8.2... + ... + == sanity checking [skipped] + == creating module... + == COMPLETED: Installation ended successfully + ... + + $ ls -l /home/example/.local/modules/all/GCC/4.8.2* + -rw-rw-r-- 1 example example 1064 Apr 30 10:54 /home/example/.local/modules/all/GCC/4.8.2 + -rw-rw-r-- 1 example example 1249 Apr 30 11:56 /home/example/.local/modules/all/GCC/4.8.2.lua + + .. note:: + Since only Lmod can consume module files in Lua syntax, Lmod must be used as a modules tool; + see also :ref:`module_syntax`. + + Only changing the syntax of the module file does not affect the module name, so Lmod will + report the module as being available. Hence, ``--force`` must be used here as well. + +* to generate a module file using a different naming scheme, ``--module-only`` can be combined with + ``--module-naming-scheme``:: + + $ eb --installpath=modules=$HOME/test/modules --module-only --module-naming-scheme=HierarchicalMNS --force + ... + == building and installing Core/GCC/4.8.2... + ... + == sanity checking [skipped] + == creating module... + == COMPLETED: Installation ended successfully + + $ module unuse $HOME/.local/modules/all + $ module use $HOME/test/modules/all + $ module avail + + ---------------------------- /home/example/test/modules/all ---------------------------- + Core/GCC/4.8.2 + + .. note:: Modules that are generated used different module naming schemes should *never* be mixed, hence the use + of ``--installpath-modules``, see also :ref:`installpath_direct_options`. + + .. note:: The modules files generated using the specified module naming scheme will ***not*** be tied to the existing + software installation in this case, since the name of the subdirectory of the software installation prefix + is also governed by the active module naming scheme. This is also why ``--force`` must be used in the + example above (to skip the sanity check that verifies the software installation). + + Thus, this is only useful to assess how the module tree would look like under a particular module naming + scheme; the modules themselves are useless since they point to empty installation directories. + + To tie a module file generated using to an existing software installation that was performed under a + different module naming scheme, a simple module naming scheme can be implemented that mixes two modules + naming schemes, by providing the name of the software installation subdirectory using one scheme, and the + module names (and other metadata) with the other. -* ``--module-only`` (for example combined with ``--module-syntax=Lua``) -* ``--module-only`` with ``--force``: skip all steps, including sanity check: install module without actually installing -the software or making sure the software was installed before -* ``--module-only`` with 'migration' naming scheme + See for example the `MigrateFromEBToHMNS module naming scheme that is used by the EasyBuild framework unit + tests `_. From dca5006211b804c72c016b07910340396feeeed1 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Thu, 30 Apr 2015 12:28:15 +0200 Subject: [PATCH 64/69] fix documentation release --- docs/Changelog.rst | 2 +- docs/conf.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Changelog.rst b/docs/Changelog.rst index 0df50e12..dbcc857b 100644 --- a/docs/Changelog.rst +++ b/docs/Changelog.rst @@ -3,7 +3,7 @@ Changelog for EasyBuild documentation ------------------------------------- -* **release 20150427.01** (`Apr 27th 2015`): updates for EasyBuild v2.1.0 +* **release 20150430.01** (`Apr 30th 2015`): updated documentation for EasyBuild v2.1.0 * also cover 'extensions' in page on concepts and terminology (see :ref:`extensions`) * add documentation on :ref:`partial_installations`, covering ``--stop``, ``--skip`` and ``--module-only`` diff --git a/docs/conf.py b/docs/conf.py index 395d3a30..b4aa7808 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -43,7 +43,7 @@ # The short X.Y version. version = '2.1.0' # this is meant to reference the version of EasyBuild # The full version, including alpha/beta/rc tags. -release = '20150427.01' # this is meant to reference the version of the documentation itself +release = '20150430.01' # this is meant to reference the version of the documentation itself # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: From 8ae886df98e6b2fb6a40ce4a9b0fb6eb7797c68d Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Thu, 30 Apr 2015 12:29:10 +0200 Subject: [PATCH 65/69] fix URL --- docs/Partial_installations.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Partial_installations.rst b/docs/Partial_installations.rst index db9daf3f..13d6a011 100644 --- a/docs/Partial_installations.rst +++ b/docs/Partial_installations.rst @@ -264,4 +264,4 @@ Examples: module names (and other metadata) with the other. See for example the `MigrateFromEBToHMNS module naming scheme that is used by the EasyBuild framework unit - tests `_. + tests `_. From 0e6a11e9d15e2149d1bbd98317e0cd4636ead379 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Thu, 30 Apr 2015 12:43:54 +0200 Subject: [PATCH 66/69] minor tweaks --- docs/Concepts_and_Terminology.rst | 11 +++++-- docs/Partial_installations.rst | 54 +++++++++++++++---------------- 2 files changed, 34 insertions(+), 31 deletions(-) diff --git a/docs/Concepts_and_Terminology.rst b/docs/Concepts_and_Terminology.rst index f8eef70a..c80e9687 100644 --- a/docs/Concepts_and_Terminology.rst +++ b/docs/Concepts_and_Terminology.rst @@ -160,8 +160,13 @@ Extensions ---------- Some software packages support installing additional add-ons alongside the 'main' software, either in the same -installation prefix, or in a separate location. Well-known examples include Perl modules (http://www.cpan.org/modules/), -Python packages (https://pypi.python.org/pypi), R libraries (http://cran.r-project.org/web/packages/), -Ruby gems (http://guides.rubygems.org/what-is-a-gem/), etc. +installation prefix, or in a separate location. In EasyBuild, we use the neutral term '**extensions**' to refer these add-ons. + +Well-known examples include: + +* Perl modules (http://www.cpan.org/modules/) +* Python packages (https://pypi.python.org/pypi) +* R libraries (http://cran.r-project.org/web/packages/) +* Ruby gems (http://guides.rubygems.org/what-is-a-gem/) diff --git a/docs/Partial_installations.rst b/docs/Partial_installations.rst index 13d6a011..83f0570c 100644 --- a/docs/Partial_installations.rst +++ b/docs/Partial_installations.rst @@ -13,8 +13,8 @@ naming scheme). * :ref:`module_only` * :ref:`module_only_only_regenerate` - * :ref:`module_only_existing_different` - * :ref:`module_only_force` + * :ref:`module_only_additional` + * :ref:`module_only` .. _partial_installation_stop: @@ -54,18 +54,16 @@ Installing additional extensions using ``-k``/``-skip`` For software applications that may include :ref:`extensions`, it is often required to install one or more additional extensions without having to reinstall the software package (and all extensions) from scratch. - For this purpose, the ``-k``/``--skip`` command line option is available. To actually skip an existing software installation and all installed extensions, a corresponding module must be available already; if not, the installation procedure will be performed from scratch. - To trigger the installation of missing extensions, ``--force`` must be used as well; without it, the installation procedure will be skipped as a whole (since the module is already available). When ``--skip`` is combined with ``--force``, EasyBuild will: -i) ensure all (extension) sources are available (and try to fetch them if needed) +i) ensure that all (extension) sources are available (and try to fetch them if needed); ii) prepare the build environment; iii) check which extensions have not been installed yet; iv) install the missing extensions; @@ -91,8 +89,7 @@ Example usage:: == testing [skipped] == installing [skipped] == taking care of extensions... - == packaging [skipped] - == postprocessing [skipped] + ... == sanity checking... == cleaning up... == creating module... @@ -100,19 +97,19 @@ Example usage:: .. note:: The '``skipsteps``' easyconfig parameter has a different purpose, i.e. to specify which installation steps should - *always* be skipped when the installation of a particular software package is performed, no matter when the software - or corresponding module is already available or not. + *always* be skipped when the installation of a particular software package is performed, no matter whether the + software or corresponding module is already available or not. .. _module_only: -Only (re)generating module files using ``--module-only`` --------------------------------------------------------- +Only (e)generating (additional) module files using ``--module-only`` +--------------------------------------------------------------------- Since EasyBuild v2.1, it is possible to only (re)generate the module file that matches the specifications in the easyconfig file, using ``--module-only``. Depending on the use case, additional options should be supplied. -Usually, ``--force`` is required either to ignore the existing module file (if the module is available under the same -name as the one being (re)generated), or to skip the sanity check that verifies the software installation (if no +Usually ``--force`` is also required, either to ignore the existing module file (if the module is available under the +same name as the one being (re)generated), or to skip the sanity check that verifies the software installation (if no software installation is available). Combining ``--module-only`` with ``--installpath-modules`` is also a common use case, to generate the module file in @@ -126,8 +123,8 @@ Use cases: .. _module_only_only_regenerate: -Only (re)generate (existing) module file -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Only (re)generating (existing) module file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To only generate a module file (i.e., skip actually building and installing the software), or to regenerate an existing module file, ``--module-only`` can be used. @@ -188,17 +185,17 @@ Example usage: $ ls -l /home/example/.local/modules/all/GCC/4.8.2* -rw-rw-r-- 1 example example 1064 Apr 30 10:54 /home/example/.local/modules/all/GCC/4.8.2 -.. _module_only_existing_different: +.. _module_only_additional: -Generate an additional module file for an existing installation -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Generating additional module files +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Generating an additional module file for existing installations is also supported. This can achieved by combining -``--module-only`` with additional configuration options that apply the module generator backend. +Generating an additional module file, next to the one(s) already available, is also supported. This can achieved by +combining ``--module-only`` with additional configuration options that apply to the module generation. Examples: -* to generate a module file in Lua syntax and put it next to the already existing module file in Tcl syntax, +* to generate a module file in Lua syntax, next to an already existing module file in Tcl syntax, ``--module-only --module-syntax=Lua`` can be used:: $ module avail GCC/4.8.2 @@ -223,7 +220,7 @@ Examples: -rw-rw-r-- 1 example example 1249 Apr 30 11:56 /home/example/.local/modules/all/GCC/4.8.2.lua .. note:: - Since only Lmod can consume module files in Lua syntax, Lmod must be used as a modules tool; + Since only Lmod can consume module files in Lua syntax, it must be used as a modules tool; see also :ref:`module_syntax`. Only changing the syntax of the module file does not affect the module name, so Lmod will @@ -232,7 +229,7 @@ Examples: * to generate a module file using a different naming scheme, ``--module-only`` can be combined with ``--module-naming-scheme``:: - $ eb --installpath=modules=$HOME/test/modules --module-only --module-naming-scheme=HierarchicalMNS --force + $ eb --installpath-modules=$HOME/test/modules --module-only --module-naming-scheme=HierarchicalMNS --force ... == building and installing Core/GCC/4.8.2... ... @@ -250,10 +247,11 @@ Examples: .. note:: Modules that are generated used different module naming schemes should *never* be mixed, hence the use of ``--installpath-modules``, see also :ref:`installpath_direct_options`. - .. note:: The modules files generated using the specified module naming scheme will ***not*** be tied to the existing - software installation in this case, since the name of the subdirectory of the software installation prefix - is also governed by the active module naming scheme. This is also why ``--force`` must be used in the - example above (to skip the sanity check that verifies the software installation). + .. note:: The modules files generated using the specified module naming scheme will most likely **not** be tied to + an existing software installation in this case (unless the software installation was already there somehow), + since the name of the subdirectory of the software installation prefix is also governed by the active + module naming scheme. This is also why ``--force`` must be used in the example above (to skip the sanity + check that verifies the software installation). Thus, this is only useful to assess how the module tree would look like under a particular module naming scheme; the modules themselves are useless since they point to empty installation directories. @@ -261,7 +259,7 @@ Examples: To tie a module file generated using to an existing software installation that was performed under a different module naming scheme, a simple module naming scheme can be implemented that mixes two modules naming schemes, by providing the name of the software installation subdirectory using one scheme, and the - module names (and other metadata) with the other. + module names (and other metadata for module files) with the other. See for example the `MigrateFromEBToHMNS module naming scheme that is used by the EasyBuild framework unit tests `_. From b22aa507c36bfd235bd896bbd24263a85a75e05c Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Thu, 30 Apr 2015 12:45:26 +0200 Subject: [PATCH 67/69] fix date in release notes --- docs/Release_notes.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Release_notes.rst b/docs/Release_notes.rst index 459b8580..bbb77183 100644 --- a/docs/Release_notes.rst +++ b/docs/Release_notes.rst @@ -7,7 +7,7 @@ The latest version of EasyBuild provides support for building and installing **6 using 33 different (compiler) toolchains. It contains 145 software-specific easyblocks and 22 generic easyblocks, alongside 3,618 easyconfig files. -v2.1.0 (April 28th 2015) +v2.1.0 (April 30th 2015) ------------------------ feature + bugfix release From 507353e77794a021fbbfc93400fb5da3feec42f2 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Thu, 30 Apr 2015 12:46:31 +0200 Subject: [PATCH 68/69] fix broken ref --- docs/Changelog.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Changelog.rst b/docs/Changelog.rst index dbcc857b..d2243e4a 100644 --- a/docs/Changelog.rst +++ b/docs/Changelog.rst @@ -5,11 +5,11 @@ Changelog for EasyBuild documentation * **release 20150430.01** (`Apr 30th 2015`): updated documentation for EasyBuild v2.1.0 - * also cover 'extensions' in page on concepts and terminology (see :ref:`extensions`) + * also cover extensions in page on concepts and terminology (see :ref:`extensions`) * add documentation on :ref:`partial_installations`, covering ``--stop``, ``--skip`` and ``--module-only`` * add documentation on :ref:`manipulating_dependencies`, covering ``--filter-deps`` and ``--hide-deps`` * document ``-module-syntax`` configuration option (see :ref:`module_syntax`) - * add note on detection of unknown ``$EASYBUILD``-prefixed environment variables (see :ref:`:ref:`configuration_env_vars`) + * add note on detection of unknown ``$EASYBUILD``-prefixed environment variables (see :ref:`configuration_env_vars`) * mention support for prepending/appending to ``--robot-paths`` (see :ref:`robot_search_path_prepend_append`) * update release notes for EasyBuild v2.1.0 (see :ref:`release_notes`) From 57e1a9ba192fcc18bad24507fc452844abcb9c53 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Thu, 30 Apr 2015 13:29:19 +0200 Subject: [PATCH 69/69] fix remark w.r.t. --skip and extension versions --- docs/Partial_installations.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/Partial_installations.rst b/docs/Partial_installations.rst index 83f0570c..89c5b805 100644 --- a/docs/Partial_installations.rst +++ b/docs/Partial_installations.rst @@ -95,6 +95,10 @@ Example usage:: == creating module... == COMPLETED: Installation ended successfully +.. note:: + Upgrading of extensions to a newer version does not work (yet) using ``--skip``, because the way in which extensions + are checked for availability, i.e. the extensions filter, is (usually) version-agnogstic. + .. note:: The '``skipsteps``' easyconfig parameter has a different purpose, i.e. to specify which installation steps should *always* be skipped when the installation of a particular software package is performed, no matter whether the @@ -182,7 +186,7 @@ Example usage: == COMPLETED: Installation ended successfully ... - $ ls -l /home/example/.local/modules/all/GCC/4.8.2* + $ ls -l /home/example/.local/modules/all/GCC/4.8.2 -rw-rw-r-- 1 example example 1064 Apr 30 10:54 /home/example/.local/modules/all/GCC/4.8.2 .. _module_only_additional: