Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs] Add docs to deploy project on Astro Cloud #90

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

pankajastro
Copy link
Collaborator

@pankajastro pankajastro commented Nov 1, 2024

Add documentation for deploying the project in the /dev directory on Astro Cloud. I created a Makefile target called "deploy" for this purpose.

I set up an Astro deployment on AWS Cloud and a RayCluster on GKE for this exercise.

To deploy on Astro Cloud, run the command make deploy.

Generated Docs

Screenshot 2024-11-01 at 12 55 21 AM

closes: #88

@pankajastro pankajastro marked this pull request as ready for review November 1, 2024 03:11
@pankajastro pankajastro requested a review from a team as a code owner November 1, 2024 03:11
@codecov-commenter
Copy link

codecov-commenter commented Nov 1, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.69%. Comparing base (5d2e992) to head (eca8821).

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #90   +/-   ##
=======================================
  Coverage   97.69%   97.69%           
=======================================
  Files           5        5           
  Lines         564      564           
=======================================
  Hits          551      551           
  Misses         13       13           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Comment on lines 32 to 34
.. code-block:: bash

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/your/service-account-key.json
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, we don't need to set this since we're passing the --key-file in the command below & that seems to suffice as I tried.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed it

--num-nodes 1 \
--machine-type e2-standard-4 \
--enable-autoscaling --min-nodes=1 --max-nodes=3 \
--no-enable-ip-alias
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
--no-enable-ip-alias
--no-enable-ip-alias --project <YOUR_PROJECT_ID>

Since we're not setting the project I think it's good to mention this.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think otherwise it would be nicer to specify the below command prior to these steps instead of providing --project for each of the commands

gcloud config set project <YOUR_PROJECT_ID>

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added this command

docs/getting_started/astro_setup.rst Outdated Show resolved Hide resolved
docs/getting_started/astro_setup.rst Outdated Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where do we specify to use this file in the make target?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed offline moved content from this file to Dockerfile

2. **Deploy the project on Astro Cloud**


Please ensure you use `Dockerfile.ray_google_cloud <https://github.com/astronomer/astro-provider-ray/blob/main/dev/Dockerfile.ray_google_cloud>`_ as the default. The command below uses `Dockerfile <https://github.com/astronomer/astro-provider-ray/blob/main/dev/Dockerfile>`_ by default.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we suggest how to use that Dockerfile.ray_google_cloud or modify the make deploy target to use this Dockerfile by default?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed offline moved content of Dockerfile.ray_google_cloud file to the Dockerfile

docs/getting_started/astro_setup.rst Show resolved Hide resolved

gcloud components install gke-gcloud-auth-plugin

4. **Create a GKE cluster**
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's also add a step suggesting the command on how to terminate this cluster once the usage is done. WDYT?

Copy link
Collaborator Author

@pankajastro pankajastro Nov 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a command to delete cluster

dev/Dockerfile.ray_google_cloud Outdated Show resolved Hide resolved
@@ -0,0 +1,22 @@
FROM quay.io/astronomer/astro-runtime:12.2.0

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#Change path to your service account key here

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to mount this key file? Can we specify how to do that?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed offline moved the content of Dockerfile.ray_google_cloud file to the Dockerfile and deleted this file

Makefile Outdated Show resolved Hide resolved
Add documentation for deploying the project in the /dev directory on Astro Cloud. I created a Makefile target called "deploy" for this purpose.

For this exercise, I set up an Astro deployment on AWS Cloud and a RayCluster on GKE.

To deploy on Astro Cloud, run the command make deploy.

closes: #88
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a document to deploy and setup on Astro Cloud
3 participants