Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fms_yaml_tools conda environment+packaging attempt #66

Open
wants to merge 21 commits into
base: main
Choose a base branch
from

Conversation

ilaflott
Copy link
Member

This is less an actual PR, and more a discussion with some diffs nearby to easily click through.

There was a clear and early intention to have fms_yaml_tools interface with fre-cli. These were not formally a dependency, however, and so those pieces of code were not tracking updates to this repository. I removed them when I realized exactly how far behind they were...

So, after that, I set about exploring how I could make fms_yaml_tools a formal dependency of the fre-cli conda package. This PR reflects that effort! The first step to making it a formal dependency is to make fms_yaml_tools a conda package itself. Conda packages are only permitted to rely on other conda packages within the conda ecosystem.

This PR is doing two main things:

  1. adds environment.yml to create a conda environment for fms_yaml_tools. If one does conda env create -f environment.yml and activates that created environment, one should be able to clone this repo (recursively) and pip install . successfully into that environment. I've tested this locally by hand and it works great- the unittests pass! Additionally, there is also now create_test_conda_env.yml, a workflow that clearly demonstrates what environment.yml is for and how to use it. A successful run can be seen here. Aside from style/neatness, this is largely ready as a prototype approach.

  2. Independent of the above capability, though slightly-overlapping, this PR also adds meta.yaml for initial conda packaging efforts. This is very similar to environment.yml but with a crucial difference- it's the target for conda build, which outputs a tarball that's capable of being uploaded to a conda channel repository. fre-cli's workflows check the conda package build/publishing using github workflows, which i've attempted to adapt to this repo, as seen in .github/workflows/build_conda.yml and .github/workflows/publish_conda.yml. These workflows are not working yet. I.e., the conda packaging of this repository is not ready as a prototype yet. I will detail the friction in my next comment.

ilaflott and others added 14 commits October 3, 2024 14:33
…v, test in conda env (create_test_conda_env).

also to run on push: workflow to build conda package but not publish the package (build_conda).
to run on merge to main branch: workflow to publish conda package (publish_conda).

environment.yaml builds usable environment with `conda env create -f environment.yaml`, which can then be activated.
the python within that environment may be used to pip install the `fms_yaml_tools` package into the conda env.

meta.yaml provides info for `conda build`, this is a first-pass

ultimate goal is a conda package fms_yaml_tools to link fre-cli's fre.yamltools into, ala fre.catalog into catalogbuilder
…r gfdl_msd_schemas/FMS folder to execute final test
whitespace change please workflow run
see if i can't convince conda build to use the gfdl_msd_schemas repo code anyways
probably not gonna work. gotta try it juuuuust one more thing though...
@ilaflott ilaflott added enhancement New feature or request help wanted Extra attention is needed question Further information is requested labels Oct 17, 2024
@ilaflott ilaflott self-assigned this Oct 17, 2024
@ilaflott
Copy link
Member Author

ilaflott commented Oct 17, 2024

To understand the friction with conda build, we'll look at the workflow logs for build_conda.yml. Note that publish_conda.yml differs from build_conda.yml only slightly. So if we understand the failure for build_conda.yml, we will understand what's holding publish_conda.yml back as well.

Poking through logs here, you come up against the relevant error message, thrown by conda build . pretty quickly:

fatal: detected dubious ownership in repository at '/__w/fms_yaml_tools/fms_yaml_tools'
To add an exception for this directory, call:

	git config --global --add safe.directory /__w/fms_yaml_tools/fms_yaml_tools
Error: Process completed with exit code 128.

This message is essentially useless and is points one to the wrong thing, i think, because of github's CI/CD oddities.

To get to the root of the real problem, one should clone this repo locally, pull in it's submodules (grabbing fms_yaml_tools/schema/gfdl_msd_schemas). cd into this repo's root directory, conda build ., and you will get largely the same error message but instead, an accurate complaint about fms_yaml_tools/schema/gfdl_msd_schemas ownership. Long story short, conda build checks for submodule repositories that do not belong to the one is trying to package, and blocks the build.

Locally, a git config --global --add safe.directory fms_yaml_tools/schema/gfdl_msd_schemas will enable a successful conda build ., IIRC. But this NOT succeeding in the pipeline blocks the potential for automatic conda-package publishing on gfdl's conda-forge channel. Unfortunate!

@ilaflott ilaflott marked this pull request as ready for review November 5, 2024 19:31
@ilaflott
Copy link
Member Author

ilaflott commented Nov 5, 2024

CREDIT TO @singhd789 and @rem1776 for pointing this out to me.

the solution is- no "by hand" git calls! instead of git submodule --init --update --recursive anywhere in the workflow file, lean on actions more:

- uses: actions/checkout@v4
  with:
    submodules: 'recursive'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed question Further information is requested
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant