Update cloudformation config to use pdm export instead of venv

[spwoodcock]

What type of PR is this? (check all applicable)

• 🍕 Feature
• 🐛 Bug Fix
• 📝 Documentation
• 🧑‍💻 Refactor
• ✅ Test
• 🤖 Build or CI
• ❓ Other (please specify)

Describe this PR

• Prod has an issue

+ pdm run -vv flask db upgrade
[PdmUsageError]: Command 'flask' is not found in your PATH.

• The virtualenv for PDM is not being configured correctly.
• Instead of using a virtualenv in prod, it's probably better to use the included python environment on a fresh machine (each deploy is to a fresh machine).
• Using pdm export we can get the requirements in a txt file, then pip install the requirements.
• This should solve the issue as the packages will definitely be available in the PYTHONPATH of the default python binary in PATH.

Checklist before requesting a review

• 📖 Read the Tasking Manager Contributing Guide: https://github.com/hotosm/tasking-manager/blob/develop/docs/developers/contributing.md
• 📖 Read the HOT Code of Conduct: https://docs.hotosm.org/code-of-conduct
• 👷‍♀️ Create small PRs. In most cases, this will be possible.
• ✅ Provide tests for your changes.
• 📝 Use descriptive commit messages.
• 📗 Update any related documentation and include any relevant screenshots.
• 🔠 Does this PR introduce or change any environment variables? If so, make sure to specify this change in the description.

[optional] What gif best describes this PR or how it makes you feel?

[spwoodcock]

@dakotabenjamin I was going to also update Python 3.9 --> 3.10 in this script, as we use 3.10 in other places without issues (in the dockerfile for instance).

However I think this config runs on Ubuntu 20.04, where the python3.9 package is available.
We would need to upgrade to Ubuntu 22.04 for python3.10 (https://packages.ubuntu.com/jammy/python3.10).

[mahesh-naxa]

I replicated the exact PROD EC2 instance with the same AMI & Conf and landed at:

root@ip-10-0-8-64:/opt/tasking-manager# pdm install
+ pdm install
Lock file version is not compatible with PDM, installation may fail
Updating the lock file...
python.use_venv is on, creating a virtualenv for this project...
Virtualenv is created successfully at /opt/tasking-manager/.venv
See /tmp/pdm-lock-wv9xtlmw.log for detailed debug log.
[KeyError]: 'Requires-Python'
Add '-v' to see the detailed traceback

This is where the user-data code fails. With some digging i found that there's been some changes to importlib 5 days ago hence resulting in this.
References
https://importlib-metadata.readthedocs.io/en/latest/history.html#v8-0-0

pdm-project/pdm#2974

I downgraded with pip3 install importlib_metadata==v7.2.1 And everything ran as expected.

[mahesh-naxa]

I did try the fix by @spwoodcock, i believe this should have also fixed the issue alternatively. But i ran to

    Found existing installation: typing_extensions 4.12.2
    Uninstalling typing_extensions-4.12.2:
      Successfully uninstalled typing_extensions-4.12.2
  Attempting uninstall: setuptools
    Found existing installation: setuptools 52.0.0
    Uninstalling setuptools-52.0.0:
      Successfully uninstalled setuptools-52.0.0
  Attempting uninstall: PyYAML
    Found existing installation: PyYAML 5.3.1
error: uninstall-distutils-installed-package

× Cannot uninstall PyYAML 5.3.1
╰─> It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

I think this could be due to existing python dependencies existing in the host. I have yet to figure this one out.

[mahesh-naxa]

Since we have python version differences & different use of pdm, i believe there are stuff that still need to be figured out, I am well opting right now to just downgrading:

          'pip install --upgrade pip pdm==2.7.4',
          echo "Downgrade for importlib KeyError on missing keys ref: https://importlib-metadata.readthedocs.io/en/latest/history.html#v8-0-0"
          pip3 install importlib_metadata==v7.2.1 
          'wget -6 https://s3.dualstack.us-east-1.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-latest.tar.gz -O /tmp/aws-cfn-bootstrap-py3-latest.tar.gz',

Then we can borrow time to decide on upgrading AMI to ubuntu 22.04 for python 3.10 & decide on use of pdm as well.
cc: @spwoodcock @dakotabenjamin @ramyaragupathy

[spwoodcock]

@mahesh-naxa to fix the conflict with PyYaml use the --user flag with pip.
I will update the PR 👍

[spwoodcock]

Nice debugging to find the importlib error 😄

I would favour the --user fix, as you will have to remember to remove the importlib pin in the future otherwise

[mahesh-naxa]

Nice debugging to find the importlib error 😄

I would favour the --user fix, as you will have to remember to remove the importlib pin in the future otherwise

Yeah i agree.
Also @dakotabenjamin can you perform a test run of the commands on one of the unhealthy instance, I don't have access to the shell there. This should give us exactly what we need. **what works.
Also i see there are 8 unhealthy ones running since forever. Can't imagine the aws billing. 😰

[dakotabenjamin]

The install works but I am getting a "Command not found" for the flask command. On the server the bin is /root/.local/bin/ but it's not on the path. Setting that dir on the path fixes it.

[spwoodcock]

Would adding this be ok?

          'export PATH="/root/.local/bin:$PATH"',

[spwoodcock]

Superseded by #6481