Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use EESSI module in EESSI-install-software.sh #790

Open
wants to merge 14 commits into
base: 2023.06-software.eessi.io
Choose a base branch
from
+118 −81
Open

Use EESSI module in EESSI-install-software.sh #790

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
d27fcf4
Modify EESSI-install-software.sh so that it loads the EESSI module to…
Oct 16, 2024
4d22c13
Also replace source of easybuild configuration by loading the EESSI-e…
Oct 16, 2024
660697a
Add a quick build so we can test the EESSI-install-software.sh script
Oct 16, 2024
50b9b8d
Merge branch '2023.06-software.eessi.io' into use_module_in_eessi_ins…
casparvl Oct 16, 2024
41f8e02
Move test if we are in prefix
Oct 16, 2024
6034cc2
Merge branch 'use_module_in_eessi_install' of github.com:casparvl/sof…
Oct 16, 2024
ab27742
Unsetting all the other variables that EESSI-extend would consider, w…
Oct 16, 2024
c374bf4
Make sure to use the site installation path for install_cuda_librarie…
Oct 21, 2024
9c48e83
Make sure we reload EESSI-extend with the correct env vars set, since…
Oct 21, 2024
ad42fad
Remove patchelf, add util-linux. Patchelf is already in the stack, so…
Oct 22, 2024
e963240
Merge branch '2023.06-software.eessi.io' into use_module_in_eessi_ins…
casparvl Oct 22, 2024
ce5f2c8
Remove util-linux again, this was just to prove the PR worked
Oct 22, 2024
f38b24f
Remove util-linux again, this was just to prove the PR worked
Oct 22, 2024
abf1aff
Update EESSI-install-software.sh
casparvl Oct 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
192 changes: 111 additions & 81 deletions EESSI-install-software.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,26 +112,8 @@ fi

TMPDIR=$(mktemp -d)

echo ">> Setting up environment..."

source $TOPDIR/init/minimal_eessi_env

if [ -d $EESSI_CVMFS_REPO ]; then
echo_green "$EESSI_CVMFS_REPO available, OK!"
else
fatal_error "$EESSI_CVMFS_REPO is not available!"
fi

# make sure we're in Prefix environment by checking $SHELL
if [[ ${SHELL} = ${EPREFIX}/bin/bash ]]; then
echo_green ">> It looks like we're in a Gentoo Prefix environment, good!"
else
fatal_error "Not running in Gentoo Prefix environment, run '${EPREFIX}/startprefix' first!"
fi

# avoid that pyc files for EasyBuild are stored in EasyBuild installation directory
export PYTHONPYCACHEPREFIX=$TMPDIR/pycache

# Get override subdir
DETECTION_PARAMETERS=''
GENERIC=0
EB='eb'
Expand All @@ -148,45 +130,33 @@ if [ -z $EESSI_SOFTWARE_SUBDIR_OVERRIDE ]; then
echo ">> Determined \$EESSI_SOFTWARE_SUBDIR_OVERRIDE via 'eessi_software_subdir.py $DETECTION_PARAMETERS' script"
else
echo ">> Picking up pre-defined \$EESSI_SOFTWARE_SUBDIR_OVERRIDE: ${EESSI_SOFTWARE_SUBDIR_OVERRIDE}"
# make sure directory exists (since it's expected by init/eessi_environment_variables when using archdetect)
mkdir -p ${EESSI_PREFIX}/software/${EESSI_OS_TYPE}/${EESSI_SOFTWARE_SUBDIR_OVERRIDE}
# Run in a subshell, so that minimal_eessi_env doesn't change the shell environment for the rest of this script
(
# Make sure EESSI_PREFIX and EESSI_OS_TYPE are set
source $TOPDIR/init/minimal_eessi_env

# make sure directory exists (since it's expected by init/eessi_environment_variables when using archdetect)
mkdir -p ${EESSI_PREFIX}/software/${EESSI_OS_TYPE}/${EESSI_SOFTWARE_SUBDIR_OVERRIDE}
)
fi

# if we run the script for the first time, e.g., to start building for a new
# stack, we need to ensure certain files are present in
# ${EESSI_PREFIX}/software/${EESSI_OS_TYPE}/${EESSI_SOFTWARE_SUBDIR_OVERRIDE}
# - .lmod/lmodrc.lua
# - .lmod/SitePackage.lua
_eessi_software_path=${EESSI_PREFIX}/software/${EESSI_OS_TYPE}/${EESSI_SOFTWARE_SUBDIR_OVERRIDE}
_lmod_cfg_dir=${_eessi_software_path}/.lmod
_lmod_rc_file=${_lmod_cfg_dir}/lmodrc.lua
if [ ! -f ${_lmod_rc_file} ]; then
echo "Lmod file '${_lmod_rc_file}' does not exist yet; creating it..."
command -V python3
python3 ${TOPDIR}/create_lmodrc.py ${_eessi_software_path}
fi
_lmod_sitepackage_file=${_lmod_cfg_dir}/SitePackage.lua
if [ ! -f ${_lmod_sitepackage_file} ]; then
echo "Lmod file '${_lmod_sitepackage_file}' does not exist yet; creating it..."
command -V python3
python3 ${TOPDIR}/create_lmodsitepackage.py ${_eessi_software_path}
fi
echo ">> Setting up environment..."

# Set all the EESSI environment variables (respecting $EESSI_SOFTWARE_SUBDIR_OVERRIDE)
# $EESSI_SILENT - don't print any messages if set (use 'unset EESSI_SILENT' to let script show messages)
# $EESSI_BASIC_ENV - give a basic set of environment variables if set (use 'EESSI_BASIC_ENV=' to let script initialise a full environment)
EESSI_SILENT=1 EESSI_BASIC_ENV= source $TOPDIR/init/eessi_environment_variables
# If EESSI_VERSION is not set, source the defaults script to set it
if [ -z ${EESSI_VERSION} ]; then
source $TOPDIR/init/eessi_defaults
fi

if [[ -z ${EESSI_SOFTWARE_SUBDIR} ]]; then
fatal_error "Failed to determine software subdirectory?!"
elif [[ "${EESSI_SOFTWARE_SUBDIR}" != "${EESSI_SOFTWARE_SUBDIR_OVERRIDE}" ]]; then
fatal_error "Values for EESSI_SOFTWARE_SUBDIR_OVERRIDE (${EESSI_SOFTWARE_SUBDIR_OVERRIDE}) and EESSI_SOFTWARE_SUBDIR (${EESSI_SOFTWARE_SUBDIR}) differ!"
# If module command does not exist, use the one from the compat layer
command -v module
module_cmd_exists=$?
if [[ "$module_cmd_exists" -ne 0 ]]; then
echo_green "No module command found, initializing lmod from the compatibility layer"
# Minimal initalization of the lmod from the compat layer
source $TOPDIR/init/lmod/bash
else
echo_green ">> Using ${EESSI_SOFTWARE_SUBDIR} as software subdirectory!"
echo_green "Module command found"
fi

echo ">> Initializing Lmod..."
source $EPREFIX/usr/share/Lmod/init/bash
ml_version_out=$TMPDIR/ml.out
ml --version &> $ml_version_out
if [[ $? -eq 0 ]]; then
Expand All @@ -195,46 +165,60 @@ else
fatal_error "Failed to initialize Lmod?! (see output in ${ml_version_out}"
fi

echo ">> Configuring EasyBuild..."
source $TOPDIR/configure_easybuild

if [ ! -z "${shared_fs_path}" ]; then
shared_eb_sourcepath=${shared_fs_path}/easybuild/sources
echo ">> Using ${shared_eb_sourcepath} as shared EasyBuild source path"
export EASYBUILD_SOURCEPATH=${shared_eb_sourcepath}:${EASYBUILD_SOURCEPATH}
fi

# Make sure we start with no modules and clean $MODULEPATH
echo ">> Setting up \$MODULEPATH..."
# make sure no modules are loaded
module --force purge
# ignore current $MODULEPATH entirely
module unuse $MODULEPATH

# if an accelerator target is specified, we need to make sure that the CPU-only modules are also still available
if [ ! -z ${EESSI_ACCELERATOR_TARGET} ]; then
CPU_ONLY_MODULES_PATH=$(echo $EASYBUILD_INSTALLPATH | sed "s@/accel/${EESSI_ACCELERATOR_TARGET}@@g")/modules/all
if [ -d ${CPU_ONLY_MODULES_PATH} ]; then
module use ${CPU_ONLY_MODULES_PATH}
else
fatal_error "Derived path to CPU-only modules does not exist: ${CPU_ONLY_MODULES_PATH}"
fi
fi
# Initialize the EESSI environment
module use $TOPDIR/init/modules
module load EESSI/$EESSI_VERSION

# If in dev.eessi.io, allow building on top of softw
if [[ "${EESSI_CVMFS_REPO}" == /cvmfs/dev.eessi.io ]]; then
module use /cvmfs/software.eessi.io/versions/$EESSI_VERSION/software/${EESSI_OS_TYPE}/${EESSI_SOFTWARE_SUBDIR_OVERRIDE}/modules/all
# make sure we're in Prefix environment by checking $SHELL
# We can only do this after loading the EESSI module, as we need ${EPREFIX}
if [[ ${SHELL} = ${EPREFIX}/bin/bash ]]; then
echo_green ">> It looks like we're in a Gentoo Prefix environment, good!"
else
fatal_error "Not running in Gentoo Prefix environment, run '${EPREFIX}/startprefix' first!"
fi

module use $EASYBUILD_INSTALLPATH/modules/all
if [ -d $EESSI_CVMFS_REPO ]; then
echo_green "$EESSI_CVMFS_REPO available, OK!"
else
fatal_error "$EESSI_CVMFS_REPO is not available!"
fi

if [[ -z ${MODULEPATH} ]]; then
fatal_error "Failed to set up \$MODULEPATH?!"
# Check that EESSI_SOFTWARE_SUBDIR now matches EESSI_SOFTWARE_SUBDIR_OVERRIDE
if [[ -z ${EESSI_SOFTWARE_SUBDIR} ]]; then
fatal_error "Failed to determine software subdirectory?!"
elif [[ "${EESSI_SOFTWARE_SUBDIR}" != "${EESSI_SOFTWARE_SUBDIR_OVERRIDE}" ]]; then
fatal_error "Values for EESSI_SOFTWARE_SUBDIR_OVERRIDE (${EESSI_SOFTWARE_SUBDIR_OVERRIDE}) and EESSI_SOFTWARE_SUBDIR (${EESSI_SOFTWARE_SUBDIR}) differ!"
else
echo_green ">> MODULEPATH set up: ${MODULEPATH}"
echo_green ">> Using ${EESSI_SOFTWARE_SUBDIR} as software subdirectory!"
fi

# assume there's only one diff file that corresponds to the PR patch file
pr_diff=$(ls [0-9]*.diff | head -1)
# avoid that pyc files for EasyBuild are stored in EasyBuild installation directory
export PYTHONPYCACHEPREFIX=$TMPDIR/pycache

# if we run the script for the first time, e.g., to start building for a new
# stack, we need to ensure certain files are present in
# ${EESSI_PREFIX}/software/${EESSI_OS_TYPE}/${EESSI_SOFTWARE_SUBDIR_OVERRIDE}
# - .lmod/lmodrc.lua
# - .lmod/SitePackage.lua
_eessi_software_path=${EESSI_PREFIX}/software/${EESSI_OS_TYPE}/${EESSI_SOFTWARE_SUBDIR_OVERRIDE}
_lmod_cfg_dir=${_eessi_software_path}/.lmod
_lmod_rc_file=${_lmod_cfg_dir}/lmodrc.lua
if [ ! -f ${_lmod_rc_file} ]; then
echo "Lmod file '${_lmod_rc_file}' does not exist yet; creating it..."
command -V python3
python3 ${TOPDIR}/create_lmodrc.py ${_eessi_software_path}
fi
_lmod_sitepackage_file=${_lmod_cfg_dir}/SitePackage.lua
if [ ! -f ${_lmod_sitepackage_file} ]; then
echo "Lmod file '${_lmod_sitepackage_file}' does not exist yet; creating it..."
command -V python3
python3 ${TOPDIR}/create_lmodsitepackage.py ${_eessi_software_path}
fi

# install any additional required scripts
# order is important: these are needed to install a full CUDA SDK in host_injections
Expand Down Expand Up @@ -278,6 +262,52 @@ if command_exists "nvidia-smi"; then
${EESSI_PREFIX}/scripts/gpu_support/nvidia/link_nvidia_host_libraries.sh
fi


echo ">> Configuring EasyBuild..."

# Make sure that we use the EESSI_CVMFS_INSTALL
# Since the path is set when loading EESSI-extend, we reload it to make sure it works - even if it is already loaded
# Note we need to do this after running install_cuda_and_libraries, since that does installations in the EESSI_SITE_INSTALL
unset EESSI_USER_INSTALL
unset EESSI_PROJECT_INSTALL
unset EESSI_SITE_INSTALL
export EESSI_CVMFS_INSTALL=1
module unload EESSI-extend
module load EESSI-extend/${EESSI_VERSION}-easybuild

if [ ! -z "${shared_fs_path}" ]; then
shared_eb_sourcepath=${shared_fs_path}/easybuild/sources
echo ">> Using ${shared_eb_sourcepath} as shared EasyBuild source path"
export EASYBUILD_SOURCEPATH=${shared_eb_sourcepath}:${EASYBUILD_SOURCEPATH}
fi

# if an accelerator target is specified, we need to make sure that the CPU-only modules are also still available
if [ ! -z ${EESSI_ACCELERATOR_TARGET} ]; then
CPU_ONLY_MODULES_PATH=$(echo $EASYBUILD_INSTALLPATH | sed "s@/accel/${EESSI_ACCELERATOR_TARGET}@@g")/modules/all
if [ -d ${CPU_ONLY_MODULES_PATH} ]; then
module use ${CPU_ONLY_MODULES_PATH}
else
fatal_error "Derived path to CPU-only modules does not exist: ${CPU_ONLY_MODULES_PATH}"
fi
fi

# If in dev.eessi.io, allow building on top of software.eessi.io
if [[ "${EESSI_CVMFS_REPO}" == /cvmfs/dev.eessi.io ]]; then
module use /cvmfs/software.eessi.io/versions/$EESSI_VERSION/software/${EESSI_OS_TYPE}/${EESSI_SOFTWARE_SUBDIR_OVERRIDE}/modules/all
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be good to make a comment that explains the expected precedence in MODULEPATH:

/path/to/dev.eessi.io/modules:/path/to/accel/modules:/path/to/software.eessi.io/modules

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure you have caught the case though that would define this:

  • I want to install in dev.eessi.io
  • I want to install for an accelerator (and require accelerator dependencies)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder is this what we want to support:

/path/to/dev.eessi.io/accel/modules:/path/to/dev.eessi.io/modules:/path/to/software.eessi.io/accel/modules:/path/to/software.eessi.io/modules

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While my pr moved this section of code, I'm pretty sure it was already there, wasn't it? I'm on my phone currently, so can't easily check

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is on me, these changes come from #771, see also @boegel's comment about handling accel cases #771 (comment)

In a meeting we talked about fixing this in a separate PR, but if it's better to address this on this PR I can give it a go when I'm back at my desk this week

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be in favour of doing that in a separate PR, if @ocaisa agrees. This PR currently is mostly a reshuffle: some parts that can only be done after EESSI is initialized got moved down until after the EESSI module is loaded. While the pr replaced sourcing the various init scripts with loading the module, it should not modify behaviour in any way. I think it's more clear of we do a big fix like that in a separate PR

fi

module use $EASYBUILD_INSTALLPATH/modules/all

if [[ -z ${MODULEPATH} ]]; then
fatal_error "Failed to set up \$MODULEPATH?!"
else
echo_green ">> MODULEPATH set up: ${MODULEPATH}"
fi

# assume there's only one diff file that corresponds to the PR patch file
pr_diff=$(ls [0-9]*.diff | head -1)


# use PR patch file to determine in which easystack files stuff was added
changed_easystacks=$(cat ${pr_diff} | grep '^+++' | cut -f2 -d' ' | sed 's@^[a-z]/@@g' | grep 'easystacks/.*yml$' | egrep -v 'known-issues|missing')
if [ -z "${changed_easystacks}" ]; then
Expand Down
7 changes: 7 additions & 0 deletions scripts/gpu_support/nvidia/install_cuda_and_libraries.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,13 @@ for EASYSTACK_FILE in ${TOPDIR}/easystacks/eessi-*CUDA*.yml; do
# Load EasyBuild version for this easystack file _before_ loading EESSI-extend
module avail EasyBuild
module load EasyBuild/${eb_version}
# Make sure EESSI-extend does a site install here
# We need to reload it with the current environment variables set
unset EESSI_CVMFS_INSTALL
unset EESSI_PROJECT_INSTALL
unset EESSI_USER_INSTALL
export EESSI_SITE_INSTALL=1
module unload EESSI-extend
module load EESSI-extend/${EESSI_VERSION}-easybuild

# Install modules in hidden .modules dir to keep track of what was installed before
Expand Down