configure fails python consistency check in default TOSS 4 tce environment

[garlick]

Problem: on fluke, with the default tce environment loaded, configure fails

[garlick@fluke108:flux-core]$ module list

Currently Loaded Modules:
  1) intel-classic-tce/2021.6.0   2) mvapich2-tce/2.3.6   3) StdEnv (S)

  Where:
   S:  Module is Sticky, requires --force to unload or purge
 
[garlick@fluke108:flux-core]$ ./configure
[snip]
checking for Python site-packages path... /collab/usr/gapps/python/build/spack-toss4.1/var/spack/environments/python/._view/75prb56irmif5ejtirjthpx6kq3gqo52/lib/python3.9/site-packages
checking for Python platform specific site-packages path... 
checking python extra libraries... -lcrypt -lintl -lpthread -ldl  -lutil -lm -lm
checking python extra linking flags... -Xlinker -export-dynamic
checking consistency of all components of python development environment... no
configure: error: in `/g/g0/garlick/proj/flux-core':
configure: error: 
  Could not link test program to Python. Maybe the main Python library has been
  installed in some non-standard library path. If so, pass it to configure,
  via the LIBS environment variable.
  Example: ./configure LIBS="-L/usr/non-standard-path/python/lib"
  ============================================================================
   ERROR!
   You probably have to install the development version of the Python package
   for your distribution.  The exact name of this package varies among them.
  ============================================================================

See `config.log' for more details

config.log.txt

[chu11]

So the issue wasn't a pathing issue with tce's python 3.9 specifically, but a lib it depends on:

configure:16108: checking python extra libraries
configure:16115: result: -lcrypt -lintl -lpthread -ldl  -lutil -lm -lm
configure:16122: checking python extra linking flags
configure:16129: result: -Xlinker -export-dynamic
configure:16136: checking consistency of all components of python development environment
configure:16164: gcc -o conftest -g -O2  -I/collab/usr/gapps/python/build/spack-toss4.1/var/spack/environments/python/._view/75prb56irmif5ejtirjthpx6kq3gqo52/include/python3.9  -Xlinker -export-dynamic conftest.c -lm  -L/collab/usr/gapps/python/build/spack-toss4.1/opt/spack/linux-rhel8-ivybridge/gcc-10.3.1/python-3.9.12-qeu37vouuau7gmcuom6z7lxzrfmktk3h/lib -lpython3.9 -lcrypt -lintl -lpthread -ldl  -lutil -lm -lm -lcrypt -lintl -lpthread -ldl  -lutil -lm -lm >&5
/usr/bin/ld: cannot find -lintl
collect2: error: ld returned 1 exit status

for some reason -lintl couldn't be found.

Went "spelunking" trying to find out where libintl was in /usr/tce and couldn't find it. Eventually had to talk to some LC staff and learned that this is a known issue.

The fix is to add the following path LIBS="-L/collab/usr/gapps/python/build/spack-toss4.1/opt/spack/linux-rhel8-ivybridge/gcc-10.3.1/gettext-0.21-kzvsqybuil5uh6kgx74o3ogxehnywsi4/lib/"

Unfortunately, this isn't a workaround, it's probably the long term solution. Various config changes/hacks have been discussed, but all come with pros vs cons, and its not clear that any of them outweigh just leaving it as is.

:-(