Versioning for repos that are not tagged with named release branches?

[kdmccormick]

Some background: openedx/axim-engineering#924 (comment)

Many Open-edX-related Python repos don't get tagged in named releases. That's because they are pip-installed into repos which are tagged. If we also tagged the pip-installed repos, then their tags wouldn't necessarily line up with the version(s) that tagged repos pip-install. (The same logic applies to JS libraries).

This process (tagging only top-level repos) has been the case since at least Juniper, and probably well before that too. @nedbat might know when we stopped tagging packages, and whether or not it was for the reason described above.

Does that clash with the fact that we use named releases in RTD version picker in https://docs.openedx.org/en/latest/ and elsewhere? What should we do for these pip-installed repos?

This is an open point of confusion in the release process: https://openedx.atlassian.net/wiki/spaces/COMM/pages/19662426/Process+to+Create+an+Open+edX+Release#5b.-Ask-Axim-to-activate-hidden-release-versions-of-each-book

[feanil]

My gut feeling is that for libraries, we should tag docs for semantic version but we should also tag the relevant version with the openedx named release tag. The reason for this is that from a consumer perspective, if you're writing code against or trying to user features from a named release, you shouldn't have to know what numeric version maps to the named release you care about.

For example: If there is a feature in the ORA repo that is available in Quince, and you want to look at the relevant docs, you should be able to pull up the quince docs for ORA whether that's version 6.5 or 7.1 in the platform isn't really useful.

This is definitely not how we've been doing things but I believe it's the direction we should be moving in. In terms of automation, I think it's possible to update the release tagging script to take into account more catalog-info and for libraries, it could tag the latest released version with the relevant tree name as well. There's still some edge cases to work out there but I think the solution is tractable.

[kdmccormick]

I agree in spirit, but I'm still not sure how to practically achieve this.

Imagine we have package "bork-lib" with these versions:

• main (latest code)
• 6.3 (latest release)
• 6.2 <-- edx-platform pins this one
• 6.1 <-- course-discovery pins this one

When it comes to cut the release, which version of bork-lib should we choose? I think the answer depends on what the consumer is doing. If they are hacking on discovery, they'll want to see 6.1; for edx-platform, 6.2; if they are developing on a new repo, they'll want 6.3. As of right now, the standard cutting/tagging process would target main.

[kdmccormick]

For example: If there is a feature in the ORA repo that is available in Quince, and you want to look at the relevant docs, you should be able to pull up the quince docs for ORA whether that's version 6.5 or 7.1 in the platform isn't really useful.

I read this wrong the first time. Based on what you're saying, we'd just want to choose the latest release that exists at cut time, bork-lib==6.3 in the example above.

It'd be a tooling change, but not a very difficult one. I could get on board with that.

[nedbat]

We stopped branching and tagging dependent repos in 2016 (openedx/repo-tools@ffead27), so Eucalyptus was the last release that had them. I think the reason we stopped was to simplify the process, since we had more reliable information (find the version from the installing repo).

[mariajgrimaldi]

I found myself needing release-specific docs for events and filters, so developers know which events/filters are available for each release. Can those repos be included in the release process? I'm currently generating the docs for the open-release/*.master manually, but I'd like to know the state of this ticket to understand how maintainable this is in the long run.

[kdmccormick]

Coming to this ticket after all this time, I think the sensible solution would be something like this:

• For any given build of docs.openedx.org, we look at edx-platform's requirement pins, and use those to determine which package versions to pull in.
• We would do this both for docs.openedx.org master, as well as for previous named releases.
• Example:
  • To build master of docs.openedx.org, we would look at edx-platform's package metadata on master to determine that we should link to the docs build of openedx-filters v1.12.0.
  • To build redwood of docs.openedx.org, we would look at edx-platform's package metadata on open-release/redwood.master to determine that we should link to the docs build of openedx-filters v1.8.1.
• Open question: How would we ensure that the correct version of openedx-filters (and other packages) are build and available to be linked to?

@feanil , do you think that this ^ is both reasonable approach, and something that a core contributor would feasibly be able to orchestrate in docs.openedx.org?

[mariajgrimaldi]

I manually built versions for open-release/sumac.master for both libraries, using the latest versions available at the time. Would it be possible to consider using these release branches instead of programmatically determining the versions for the release? Or do we want to avoid tagging these pip installed libs? Libraries like openedx-aspects are already being tagged.

[kdmccormick]

@mariajgrimaldi My concern with adding release branches and tags to pip-installed libs is that they can fall out-of-sync with what edx-platform actually pins. Imagine:

• We cut branch edx-platform:zebrawood, which pins openedx-filters:3.0.0.
• But, the latest release of openedx-filters is actually 4.0.0.
• So, the problem: Should we create the branch openedx-filters:zebrawood based on the commit at 3.0.0, or 4.0.0?
  • Another problem: Imagine we choose 3.0.0. But later, someone backports a PR to edx-platform:zebrawood which updates the openedx-filters pin to 4.0.0. Now, openedx-filters:zebrawood is out of date!

These are problems we could solve, with process changes or tools or both. I have some underdeveloped ideas, but before I dive into those--does that make sense, and are you still interested in adding named release branches (and/or tags) to pip-installed repos?

[mariajgrimaldi]

Thank you for the detailed explanation, @kdmccormick. Yes, please go ahead. It'd be useful to establish a process for these kinds of requests.