Skip to content

Commit

Permalink
Switch build-info and build images to run on public runners
Browse files Browse the repository at this point in the history 
We used to have low number of Public runners on GitHub actions and
Apache Software Foundation projects were competing for them, that's
why we switched `build-images` workflow to run on self-hosted
runners, and we had to add workflow-embedded list of committers to
make sure that build-info also uses self-hosted runners.

Recently however the ASF has 900 runners and if anything, it's the
public runners that are in abundance, while self-hosted runners are
limited, which sometimes leads to "Wait for CI images" jobs timeout
while waiting for self-hosted runners to build the image.

Those "wait-for-ci-images" jobs do almost nothing, just waiting, so
using self-hosted runners for those is not needed, we can easily
switch those to public runners as well.

This PR does the following changes:

* build-info jobs always run on public runners
* wait-for-ci-images job are always run on public runners
* all the other jobs run on either self-hosted or public runners
  depending who the actor is (committers - run on self-hosted,
  non-committers, run on public runners)
  • Loading branch information
@potiuk
potiuk committed May 21, 2023
1 parent 9a0f6b8 commit ca2f301
Show file tree
Hide file tree
Showing 10 changed files with 203 additions and 135 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/build-images.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,10 @@ jobs:
build-info:
timeout-minutes: 10
name: "Build Info"
runs-on: ${{ github.repository == 'apache/airflow' && 'self-hosted' || 'ubuntu-20.04' }}
runs-on: 'ubuntu-20.04'
env:
TARGET_BRANCH: ${{ github.event.pull_request.base.ref }}
outputs:
runs-on: ${{ github.repository == 'apache/airflow' && 'self-hosted' || 'ubuntu-20.04' }}
python-versions: "${{ steps.selective-checks.python-versions }}"
upgrade-to-newer-dependencies: ${{ steps.selective-checks.outputs.upgrade-to-newer-dependencies }}
all-python-versions-list-as-string: >-
Expand All @@ -66,6 +65,7 @@ jobs:
cache-directive: ${{ steps.selective-checks.outputs.cache-directive }}
default-branch: ${{ steps.selective-checks.outputs.default-branch }}
default-constraints-branch: ${{ steps.selective-checks.outputs.default-constraints-branch }}
runs-on: ${{ steps.selective-checks.outputs.runs-on }}
target-commit-sha: "${{steps.discover-pr-merge-commit.outputs.target-commit-sha ||
github.event.pull_request.head.sha ||
github.sha
Expand Down
63 changes: 3 additions & 60 deletions .github/workflows/ci.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -65,64 +65,7 @@ jobs:
# to find the list of authors and replace them, so any changes to the
# formatting of the contains(fromJSON()) structure below will need to be
# reflected in that workflow too.
runs-on: >-
${{ (
(
github.event_name == 'push' ||
github.event_name == 'schedule' ||
contains(fromJSON('[
"BasPH",
"Fokko",
"KevinYang21",
"Taragolis",
"XD-DENG",
"aijamalnk",
"alexvanboxel",
"aoen",
"artwr",
"ashb",
"bbovenzi",
"bolkedebruin",
"criccomini",
"dimberman",
"dstandish",
"eladkal",
"ephraimbuddy",
"feluelle",
"feng-tao",
"houqp",
"hussein-awala",
"jedcunningham",
"jgao54",
"jghoman",
"jhtimmins",
"jmcarp",
"josh-fell",
"kaxil",
"leahecole",
"malthe",
"mik-laj",
"milton0825",
"mistercrunch",
"msumit",
"o-nikolas",
"pierrejeambrun",
"pingzh",
"potiuk",
"r39132",
"ryanahamilton",
"ryw",
"saguziel",
"sekikn",
"turbaszek",
"uranusjr",
"vikramkoka",
"xinbinhuang",
"yuqian90",
"zhongjiajie"
]'), github.event.pull_request.user.login)
) && github.repository == 'apache/airflow'
) && 'self-hosted' || 'ubuntu-20.04' }}
runs-on: "ubuntu-20.04"
env:
GITHUB_CONTEXT: ${{ toJson(github) }}
outputs:
Expand Down Expand Up @@ -171,11 +114,11 @@ jobs:
skip-pre-commits: ${{ steps.selective-checks.outputs.skip-pre-commits }}
helm-test-packages: ${{ steps.selective-checks.outputs.helm-test-packages }}
debug-resources: ${{ steps.selective-checks.outputs.debug-resources }}
runs-on: ${{ steps.selective-checks.outputs.runs-on }}
source-head-repo: ${{ steps.source-run-info.outputs.source-head-repo }}
pull-request-labels: ${{ steps.source-run-info.outputs.pr-labels }}
in-workflow-build: ${{ steps.source-run-info.outputs.in-workflow-build }}
build-job-description: ${{ steps.source-run-info.outputs.build-job-description }}
runs-on: ${{ steps.source-run-info.outputs.runs-on }}
canary-run: ${{ steps.source-run-info.outputs.canary-run }}
run-coverage: ${{ steps.source-run-info.outputs.run-coverage }}
steps:
Expand Down Expand Up @@ -525,7 +468,7 @@ jobs:
wait-for-ci-images:
timeout-minutes: 120
name: "Wait for CI images"
runs-on: "${{needs.build-info.outputs.runs-on}}"
runs-on: "ubuntu-20.04"
needs: [build-info, build-ci-images]
if: needs.build-info.outputs.image-build == 'true'
env:
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/codeql-analysis.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ jobs:
env:
COMMIT_REF: "${{ github.sha }}"
VERBOSE: "false"
ACTOR: "${{ github.event.pusher.name }}"
run: breeze ci selective-check 2>> ${GITHUB_OUTPUT}

analyze:
Expand Down
16 changes: 15 additions & 1 deletion dev/breeze/src/airflow_breeze/commands/ci_commands.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
import click

from airflow_breeze.global_constants import (
COMMITTERS,
DEFAULT_PYTHON_MAJOR_MINOR_VERSION,
RUNS_ON_PUBLIC_RUNNER,
RUNS_ON_SELF_HOSTED_RUNNER,
Expand All @@ -42,11 +43,12 @@
from airflow_breeze.utils.common_options import (
option_answer,
option_dry_run,
option_github_repository,
option_verbose,
)
from airflow_breeze.utils.confirm import Answer, user_confirm
from airflow_breeze.utils.console import get_console
from airflow_breeze.utils.custom_param_types import BetterChoice
from airflow_breeze.utils.custom_param_types import BetterChoice, NotVerifiedBetterChoice
from airflow_breeze.utils.docker_command_utils import (
check_docker_resources,
fix_ownership_using_docker,
Expand Down Expand Up @@ -220,6 +222,14 @@ def get_changed_files(commit_ref: str | None) -> tuple[str, ...]:
envvar="GITHUB_EVENT_NAME",
show_default=True,
)
@option_github_repository
@click.option(
"--github-actor",
help="Actor that triggered the event",
envvar="GITHUB_ACTOR",
type=NotVerifiedBetterChoice(COMMITTERS),
default="",
)
@option_verbose
@option_dry_run
def selective_check(
Expand All @@ -228,6 +238,8 @@ def selective_check(
default_branch: str,
default_constraints_branch: str,
github_event_name: str,
github_repository: str,
github_actor: str,
):
from airflow_breeze.utils.selective_checks import SelectiveChecks

Expand All @@ -243,6 +255,8 @@ def selective_check(
default_constraints_branch=default_constraints_branch,
pr_labels=tuple(ast.literal_eval(pr_labels)) if pr_labels else (),
github_event=github_event,
github_repository=github_repository,
github_actor=github_actor,
)
print(str(sc), file=sys.stderr)

Expand Down
2 changes: 2 additions & 0 deletions dev/breeze/src/airflow_breeze/commands/ci_commands_config.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@
"--default-branch",
"--default-constraints-branch",
"--github-event-name",
"--github-repository",
"--github-actor",
],
}
],
Expand Down
54 changes: 53 additions & 1 deletion dev/breeze/src/airflow_breeze/global_constants.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,59 @@ def get_default_platform_machine() -> str:
BREEZE_INIT_COMMAND = ""
DRY_RUN_DOCKER = False
INSTALL_AIRFLOW_VERSION = ""
SQLITE_URL = "sqlite:////root/airflow/airflow.db"


COMMITTERS = [
"BasPH",
"Fokko",
"KevinYang21",
"Taragolis",
"XD-DENG",
"aijamalnk",
"alexvanboxel",
"aoen",
"artwr",
"ashb",
"bbovenzi",
"bolkedebruin",
"criccomini",
"dimberman",
"dstandish",
"eladkal",
"ephraimbuddy",
"feluelle",
"feng-tao",
"houqp",
"hussein-awala",
"jedcunningham",
"jgao54",
"jghoman",
"jhtimmins",
"jmcarp",
"josh-fell",
"kaxil",
"leahecole",
"malthe",
"mik-laj",
"milton0825",
"mistercrunch",
"msumit",
"o-nikolas",
"pierrejeambrun",
"pingzh",
"potiuk",
"r39132",
"ryanahamilton",
"ryw",
"saguziel",
"sekikn",
"turbaszek",
"uranusjr",
"vikramkoka",
"xinbinhuang",
"yuqian90",
"zhongjiajie",
]


def get_airflow_version():
Expand Down
24 changes: 22 additions & 2 deletions dev/breeze/src/airflow_breeze/utils/selective_checks.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,15 @@
from re import match
from typing import Any, Dict, List, TypeVar

from typing_extensions import Literal
if sys.version_info >= (3, 9):
from typing import Literal
else:
from typing_extensions import Literal

from airflow_breeze.global_constants import (
ALL_PYTHON_MAJOR_MINOR_VERSIONS,
APACHE_AIRFLOW_GITHUB_REPOSITORY,
COMMITTERS,
CURRENT_KUBERNETES_VERSIONS,
CURRENT_MSSQL_VERSIONS,
CURRENT_MYSQL_VERSIONS,
Expand All @@ -59,6 +64,8 @@
DEFAULT_PYTHON_MAJOR_MINOR_VERSION,
HELM_VERSION,
KIND_VERSION,
RUNS_ON_PUBLIC_RUNNER,
RUNS_ON_SELF_HOSTED_RUNNER,
GithubEvents,
SelectiveUnitTestTypes,
all_helm_test_packages,
Expand Down Expand Up @@ -295,13 +302,17 @@ def __init__(
commit_ref: str | None = None,
pr_labels: tuple[str, ...] = (),
github_event: GithubEvents = GithubEvents.PULL_REQUEST,
github_repository: str = APACHE_AIRFLOW_GITHUB_REPOSITORY,
github_actor: str = "",
):
self._files = files
self._default_branch = default_branch
self._default_constraints_branch = default_constraints_branch
self._commit_ref = commit_ref
self._pr_labels = pr_labels
self._github_event = github_event
self._github_repository = github_repository
self._github_actor = github_actor

def __important_attributes(self) -> tuple[Any, ...]:
return tuple(getattr(self, f) for f in self.__HASHABLE_FIELDS)
Expand Down Expand Up @@ -618,7 +629,7 @@ def _get_test_types_to_run(self) -> list[str]:
@staticmethod
def _extract_long_provider_tests(current_test_types: set[str]):
"""
In case there are Provider tests in the list of test to run (either in the form of
In case there are Provider tests in the list of test to run - either in the form of
Providers or Providers[...] we subtract them from the test type,
and add them to the list of tests to run individually.
Expand Down Expand Up @@ -765,3 +776,12 @@ def affected_providers_list_as_string(self) -> str | None:
if affected_providers == "ALL_PROVIDERS":
return _ALL_PROVIDERS_LIST
return " ".join(sorted(affected_providers))

@cached_property
def runs_on(self) -> str:
if self._github_repository == APACHE_AIRFLOW_GITHUB_REPOSITORY:
if self._github_event in [GithubEvents.SCHEDULE, GithubEvents.PUSH]:
return RUNS_ON_SELF_HOSTED_RUNNER
if self._github_actor in COMMITTERS:
return RUNS_ON_SELF_HOSTED_RUNNER
return RUNS_ON_PUBLIC_RUNNER
4 changes: 2 additions & 2 deletions images/breeze/output-commands-hash.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ ci:fix-ownership:fee2c9ec9ef19686792002ae054fecdd
ci:free-space:47234aa0a60b0efd84972e6e797379f8
ci:get-workflow-info:01ee34c33ad62fa5dc33e0ac8773223f
ci:resource-check:1d4fe47dff9fc64ac1648ec4beb2d85c
ci:selective-check:3a085894f24cb909812fbc8253a21e13
ci:e51cbc38a202b92b7dc6288f6344c412
ci:selective-check:6065c28e01fede4c5138989eb6615752
ci:32880df943c523dad03d2616e937e247
ci-image:build:3ffe4dd24ae7090415543e27d8504955
ci-image:pull:c16c6e57c748bfe9b365b4ffafb18472
ci-image:verify:aee88f55e8837028d19316356e29b009
Expand Down
Loading

0 comments on commit ca2f301

Please sign in to comment.