vsc/__init__.py file missing when doing "pip install" with "--prefix=xx"

[LiangFr]

I use python 2.7.13 and pip 9.0.1 to install Easybuild 3.1.0 with the following command:
$ pip install --install-option "--prefix=${PWD}/EasyBuild" easybuild
Executable eb is generated. PATH and PYTHONAPTH are correctly set. While doing command:
eb --version
I got the following error:

  ......
  File "/mnt/home_local/eocoe/liang/ECAM/EB/EB-Liang/EasyBuild/lib/python2.7/site-packages/easybuild/tools/build_log.py", line 40, in <module>
    from vsc.utils import fancylogger
ImportError: No module named vsc.utils

The error occurs because of the absence of vsc/__init__.py. @boegel checked that when using pip, we have:
WARN: Skipping installation of /mnt/home_local/eocoe/liang/ECAM/EB/EB-Liang/EasyBuild/lib/python2.7/site-packages/vsc/__init__.py (namespace package)
This is a pip issue that has already reported to pip in github, see (pypa/pip#1924)

One temporary solution for those who want to use pip install with their own PREFIX is, after the pip install command, do:
touch ${PREFIX}/lib/python2.7/site-packages/vsc/__init__.py
The problem should the be solved.

Further actions may relate to the setup.py of vsc_base. We may let the program detect the case when doing pip install with PREFIX and force the program to create __init__.py instead of skipping it.

[verdurin]

In case it's useful, I saw this today when installing without admin rights on a CentOS 6 system, which I was using a Python 2.7.11 module, to get around a known problem with SSL libraries when using the native Python.

[JensTimmerman]

Since pip is claiming there isn't anything they can do about this we'll probably have to look at working around that limitation in pip. But I'm not sure how we can detect we're being used by (a broken) pip and not a working setuptools.

[boegel]

I think the proper fix here is to switch to using pkgutil.extend_path rather than pkg_resources.declare_namespace, which works fine with pip, see also https://packaging.python.org/guides/packaging-namespace-packages/#creating-a-namespace-package .

We made that change in EasyBuild recently (see easybuilders/easybuild-framework#2836) didn't encounter any real problems since we did.

The (big) downside is that once we start using pkgutil.extend_path somewhere, we have to start using it in all other vsc-* packages on short notice, since it doesn't mix well with pkg_resources.declare_namespace

[JensTimmerman]

We used to have declare namespace but this didn't work for zipped eggs (iirc), see discussion in
#21

and pr where we switched from declare namespace to extend_path: #6

it also failed for some other reasons
and my answer to my own question here:
https://stackoverflow.com/questions/12040689/add-a-subdirectory-to-a-python-namespace/12041007#12041007

but if you got it working in EB, you might have found the correct workarounds?