Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update tests; consolidate pypi build and publish #46

Merged
merged 2 commits into from
Nov 3, 2023
Merged

Update tests; consolidate pypi build and publish #46

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9466a15
Update tests; consolidate pypi build and publish
rhiannonlynne Nov 3, 2023
558ce3c
Change workflow name
rhiannonlynne Nov 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 6 additions & 8 deletions .github/workflows/cache.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,12 @@ jobs:
id: rs-install
shell: bash -l {0}
run: |
export RUBIN_SIM_DATA_DIR=~/rubin_sim_data
mamba install --quiet rubin_sim
python -c "import rubin_sim; print(rubin_sim.__version__)" > ${{ github.workspace }}/rs_version
echo `cat ${{ github.workspace }}/rs_version`
mamba list rubin-sim | grep -v "#" | awk '{print $2}' > ${{ github.workspace }}/rs_version
echo "rs-version" `cat ${{ github.workspace }}/rs_version`
echo "rs-version=`cat ${{ github.workspace }}/rs_version`" >> $GITHUB_OUTPUT

- name: Cache rubin-sim-data
- name: Access rubin-sim-data cache
id: cache-rs
uses: actions/cache@v3
env:
Expand All @@ -47,18 +46,17 @@ jobs:
${{ env.cache-name }}-v

- if: ${{ steps.cache-rs.outputs.cache-hit == 'true' }}
name: Hit cache? List contents
continue-on-error: true
name: List cache contents.
shell: bash -l {0}
run: |
export RUBIN_SIM_DATA_DIR=~/rubin_sim_data
echo $RUBIN_SIM_DATA_DIR (contents)
echo $RUBIN_SIM_DATA_DIR contents
ls $RUBIN_SIM_DATA_DIR
echo "__contents of versions.txt__"
cat $RUBIN_SIM_DATA_DIR/versions.txt

- if: ${{ steps.cache-rs.outputs.cache-hit != 'true' }}
name: Download data
name: Download data.
shell: bash -l {0}
run: |
export RUBIN_SIM_DATA_DIR=~/rubin_sim_data
Expand Down
83 changes: 0 additions & 83 deletions .github/workflows/python_tests.yaml
View file
Open in desktop

This file was deleted.

75 changes: 47 additions & 28 deletions .github/workflows/build_pypi.yaml → .github/workflows/test_and_build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,28 @@
name: Build and Publish PyPI
name: Run CI

on:
# Run job at the end of each day
schedule:
- cron: "0 0 * * *"
# Run job on push to main
push:
tags:
- "*"
branches:
- main
# Run job on pull request to main
pull_request:
branches:
- main
# Run job manually on request
workflow_dispatch:

jobs:
tests:
name: Check Tests (${{ matrix.python-version }}, ${{ matrix.os }})
name: Run Tests (${{ matrix.python-version }}, ${{ matrix.os }})
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: ["ubuntu-latest"]
os: ["ubuntu-latest", "macos-latest"]
python-version: ["3.11"]
steps:
- uses: actions/checkout@v3
Expand All @@ -27,39 +37,48 @@ jobs:
show-channel-urls: true

- name: configure conda and install requirements
id: rs-install
shell: bash -l {0}
run: |
mamba config --set always_yes yes
mamba install --quiet --file=requirements.txt
mamba install --quiet --file=test-requirements.txt
mamba list rubin-sim | grep -v "#" | awk '{print $2}' > ${{ github.workspace }}/rs_version
echo "rs-version" `cat ${{ github.workspace }}/rs_version`
echo "rs-version=`cat ${{ github.workspace }}/rs_version`" >> $GITHUB_OUTPUT

- name: download rubin_sim_data components needed for unit tests
- name: Access rubin-sim-data cache
id: cache-rs
uses: actions/cache@v3
env:
cache-name: cached-rubin-sim-data
with:
path: ~/rubin_sim_data
key: ${{ env.cache-name }}-v${{ steps.rs-install.outputs.rs-version }}
restore-keys: |
${{ env.cache-name }}-v

- if: ${{ steps.cache-rs.outputs.cache-hit == 'true' }}
name: List cache contents.
shell: bash -l {0}
run: |
export RUBIN_SIM_DATA_DIR=~/rubin_sim_data
echo $RUBIN_SIM_DATA_DIR
ls $RUBIN_SIM_DATA_DIR
echo "__contents of versions.txt__"
cat $RUBIN_SIM_DATA_DIR/versions.txt

- if: ${{ steps.cache-rs.outputs.cache-hit != 'true' }}
name: Download data.
shell: bash -l {0}
run: |
export RUBIN_SIM_DATA_DIR=${{ github.workspace }}/data_dir
mkdir $RUBIN_SIM_DATA_DIR
cd $RUBIN_SIM_DATA_DIR
wget -q https://s3df.slac.stanford.edu/data/rubin/sim-data/rubin_sim_data/tests_2022_10_18.tgz
tar -xf tests_2022_10_18.tgz
echo "tests_2022_10_18.tgz complete"
wget -q https://s3df.slac.stanford.edu/data/rubin/sim-data/rubin_sim_data/throughputs_2023_09_07.tgz
tar -xf throughputs_2023_09_07.tgz
echo "throughputs_2023_09_07.tgz complete"
wget -q https://s3df.slac.stanford.edu/data/rubin/sim-data/rubin_sim_data/site_models_2023_03_28.tgz
tar -xf site_models_2023_03_28.tgz
echo "site_models_2023_03_28.tgz complete"
wget -q https://s3df.slac.stanford.edu/data/rubin/sim-data/rubin_sim_data/scheduler_2023_09_22.tgz
tar -xf scheduler_2023_09_22.tgz
echo "scheduler_2023_09_22.tgz complete"
wget -q https://s3df.slac.stanford.edu/data/rubin/sim-data/rubin_sim_data/skybrightness_pre_2023_09_19.tgz
tar -xf skybrightness_pre_2023_09_19.tgz
echo "skybrightness_pre_2023_09_19.tgz complete"
export RUBIN_SIM_DATA_DIR=~/rubin_sim_data
rs_download_data --force --dirs scheduler,site_models,skybrightness_pre,throughputs,tests --tdqm_disable

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this actually work here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well ... I haven't fallen into the problem of needing to use it.
It did work from the cache download script.
This should only trigger if the version of rubin_sim changed since the cache was created. The cache will be re-created every night, or can be manually triggered.
So it's just a backup, really. We could copy in the full "download all these files individually", but it seems like not a great option. It could also be deleted entirely, which was my first thought .. and then I figured, no it was worth leaving in as a "good example".
It's also worth noting that the cache build script is currently running on an ubuntu runner, but it would be perfectly fine to move that to a macOS runner if needed.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you wanted me to pull the download individual files option into a different file, to preserve for easy pasting later, I could do that?

- name: install schedview
shell: bash -l {0}
run: |
echo `pwd`
python -m pip install .
python -m pip install . --no-deps

- name: conda list
shell: bash -l {0}
Expand All @@ -68,7 +87,7 @@ jobs:
- name: run unit tests
shell: bash -l {0}
run: |
export RUBIN_SIM_DATA_DIR=${{ github.workspace }}/data_dir
export RUBIN_SIM_DATA_DIR=~/rubin_sim_data
pytest -r a -v --cov=schedview --cov=tests --cov-report=xml --cov-report=term --cov-branch

- name: Upload coverage to codecov
Expand Down Expand Up @@ -107,4 +126,4 @@ jobs:
uses: pypa/gh-action-pypi-publish@release/v1
with:
user: __token__
password: ${{ secrets.SP_PYPI_UPLOADS }}
password: ${{ secrets.SP_PYPI_UPLOADS }}
Loading