Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Add Support for GCP #193

Merged
merged 117 commits into from
Jan 3, 2025
Merged

Feature: Add Support for GCP #193

merged 117 commits into from
Jan 3, 2025

Conversation

haroldrubio
Copy link
Member

@haroldrubio haroldrubio commented Nov 25, 2024

This PR changes the expertise service to support GCP Vertex AI by:

Supporting the Expertise Application as a Container

  • Adds a Dockerfile to build a Docker image that runs the expertise service. Note: the service is NOT functional, it is mainly just to have a container that has the environment setup.
  • The image copies files from the GCS bucket specified in AIP_STORAGE_URL
  • Adds a workflow dispatch action that builds and pushes an image to the Artifact Repository
  • Adds some endpoints to be recognized as a container by Vertex AI

Adds Support for Execution in Vertex AI Pipelines

  • Adds a script to build and upload a Vertex AI Pipeline that uses the built Docker image to the Artifact Registry
  • Adds a script that gets run by the pipeline from the command line that runs the expertise model, and uploads the results and related files to GCS

Routes Requests to Vertex AI Pipelines

  • Reads config variables from the Flask config file to route requests from using local resources to Vertex AI
USE_GCP
GCP_PROJECT_ID
GCP_PROJECT_NUMBER
GCP_REGION
GCP_PIPELINE_ROOT
GCP_PIPELINE_NAME
GCP_PIPELINE_REPO
GCP_PIPELINE_TAG
GCP_BUCKET_NAME
GCP_JOBS_FOLDER
GCP_SERVICE_LABEL
  • Adds the interface to GCP in utils.py

@haroldrubio
Copy link
Member Author

There are some references to the expertise-utils folder that I use to store data that may have to be downloaded during container startup, are these alright to leave in or should I try to parameterize this

os.environ["AIP_STORAGE_URI"] = "gs://openreview-expertise/expertise-utils/"

AIP_STORAGE_URI="gs://openreview-expertise/expertise-utils/" \

@carlosmondra carlosmondra merged commit 2974db2 into master Jan 3, 2025
1 check passed
@carlosmondra carlosmondra deleted the feature/vertex-pipeline branch January 3, 2025 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants