-
Notifications
You must be signed in to change notification settings - Fork 496
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement Automated Weekly Smoke Tests #4112
Comments
Fixes skypilot-org#4112 Implement an automated weekly smoke test run using GitHub Actions leveraging the existing `tests/test_smoke.py` script. * Add a new GitHub Actions workflow file `.github/workflows/weekly-smoke-tests.yml`. * Schedule the workflow to run weekly using cron syntax. * Set up the environment and install dependencies. * Run `pytest tests/test_smoke.py --terminate-on-failure`. * Handle different test groups (AWS, GCP, Azure, Lambda, Kubernetes) as defined in the script. * Add necessary cloud credentials as GitHub secrets. * Upload test results as artifacts.
i have some buildkite stuff i've already setup for GKE that would probably adapt well to running skypilot smoke tests. Happy to work with the team on sharing it since I'm depending pretty heavily on skypilot+k8s right now |
Another cost optimization - we can run a k8s cluster as a part of our CI and move many of the cloud agnostic tests (e.g., those which test core functionality of SkyPilot) to run on a Kubernetes cluster provisioned in github actions for the duration of the test. See: https://github.com/marketplace/actions/kind-kubernetes-in-docker-action Need to evaluate cost-benefit (test migration effort vs reduced cloud cost) before we implement the above. |
Implement Automated Weekly Smoke Tests
Problem
Currently, smoke tests for SkyPilot (implemented in
test_smoke.py
) are being run manually. This process could be improved by automating these tests on a weekly basis.Proposed Solution
Implement an automated weekly smoke test run using a suitable CI/CD platform or automation tool, leveraging the existing
test_smoke.py
script.Implementation Details
Automation Setup:
test_smoke.py
script in the automationEnvironment Setup:
Test Execution:
pytest tests/test_smoke.py --terminate-on-failure
Challenges to Address
Credit Control:
Test Stability and Retries:
Multi-Cloud Testing:
test_smoke.py
Next Steps
Feedback on implementation details and challenge mitigation strategies is welcome, particularly from those familiar with
test_smoke.py
and our current testing processes.The text was updated successfully, but these errors were encountered: