transport-ops

Scripts and config files for provision and deploy transport and required services

New release process (GitHub actions + GitHub Container Registry)

To create a new release as a Docker image:

1. Update the code (Dockerfile and Github workflow tests), create a pull request, get your changes merged into master
2. Create a GitHub release (with tag name == release name), with a changelog on the page https://github.com/etalab/transport-ops/releases
   1. Use locally rake get_image_version to generate the correct tag / release name
   2. Then create a release using this generated name as both the tag and the release title. You can auto-generate the release notes.
3. On release creation (see https://github.com/etalab/transport-ops/blob/master/.github/workflows/docker.yml), a build will start, and should normally result into the publication of a GitHub-hosted Docker image named just like the release. You can find the resulting image here: https://github.com/etalab/transport-ops/pkgs/container/transport-ops This image URL can then be used by transport-site.

One major caveat: the workflow must exist at the moment the tag is created (https://github.community/t/workflow-set-for-on-release-not-triggering-not-showing-up/16286/7):

The trigger only executes when a release is created using a tag that contains the workflow.

How to rebuild the package for an existing release (security patches etc)

See #45:

• You can now re-run a workflow thanks to the workflow_dispatch event trigger: go to "actions" / "test and publish the docker image", "run workflow" on the right, pick the right tag.
• It is also possible to move an existing release to "pre-release" (then cancel the job automatically created) then back again to "release", this will re-trigger the release.
• On last resort, you can also delete the remote tag and the release, and recreate both.

LEGACY DOCUMENTATION below - kept until the moment we clean it

Image naming convention

In the past images were named "betagouv/transport:x.y.z".

Now we name them after the Elixir, Erlang and Alpine versions actually used (like the base hexpm image it uses).

• Base name: hexpm/elixir:1.10.4-erlang-23.2.7.2-alpine-3.13.3
• Replace hexpm/elixir: by betagouv/transport:elixir-
• Target name: betagouv/transport:elixir-1.10.4-erlang-23.2.7.2-alpine-3.13.3

You can use rake get_image_version to build the expected target name out of the Dockerfile in automated fashion.

Production configuration and deployment

The production requires the following services:

• transport-site: the web frontend

  • Hosted at https://transport.data.gouv.fr
  • Elixir application hosted on CleverCloud
  • The master branch of https://github.com/etalab/transport-site is deployed by CircleCI

• udata: references and/or stores the datasets and comments

  • Hosted at https://data.gouv.fr

Docker for transport-site CI

The Transport Site project is tested with CircleCI in a Docker container that is defined in transport-site/Dockerfile.

Auto-build of images

⚠️ Currently, the auto-build mechanism is broken (see #17). See "Manual build" as a work-around in the mean time.

When updating that Dockerfile, push a git tag in the format a.b.c and the image will be built by Dockerhub : https://hub.docker.com/r/betagouv/transport/

Then update https://github.com/etalab/transport-site/blob/master/.circleci/config.yml to match the new version.

Manual build and push of images

As a work-around for #17:

• Create a release with an upgraded version number
• Build the image locally with the correct tag, and without cache to make sure no left-over impact the build:

IMAGE_VERSION=$(rake get_image_version)
IMAGE_NAME=betagouv/transport:$IMAGE_VERSION
docker build transport-site --no-cache -t $IMAGE_NAME --progress=plain

• Carefully verify the versions (this will be translated into a testing script later):

docker run -it --rm $IMAGE_NAME /bin/bash -c 'node --version'
docker run -it --rm $IMAGE_NAME /bin/bash -c 'elixir --version'
# only major
docker run -it --rm $IMAGE_NAME /bin/bash -c "erl -noshell -eval 'erlang:display(erlang:system_info(system_version))' -eval 'init:stop()'"
# full version (https://stackoverflow.com/a/34326368)
docker run -it --rm $IMAGE_NME /bin/bash -c "erl -eval '{ok, Version} = file:read_file(filename:join([code:root_dir(), \"releases\", erlang:system_info(otp_release), \"OTP_VERSION\"])), io:fwrite(Version), halt().' -noshell"

• Read the docker push documentation
• List the local images with docker image ls
• Filter with docker image ls | grep "betagouv/transport" | grep $IMAGE_VERSION
• Push with docker image push $IMAGE_NAME
• Verify tag presence at https://hub.docker.com/repository/docker/betagouv/transport
• TODO: handle latest (but it is currently unused)

Useful tricks when upgrading

Before creating a tag, the following commands can be used to verify the versions:

cd transport-site
docker build . -t test:latest --progress=plain
docker run -it --rm test:latest /bin/bash -c 'node --version'
docker run -it --rm test:latest /bin/bash -c 'elixir --version'
docker run -it --rm test:latest /bin/bash -c "erl -noshell -eval 'erlang:display(erlang:system_info(system_version))' -eval 'init:stop()'"