Skip to content

Commit

Permalink
updates
Browse files Browse the repository at this point in the history
  • Loading branch information
venkatajagannath committed Aug 8, 2024
1 parent 645bd19 commit 60e55ae
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 62 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: Deploy Sphinx documentation to Pages

on:
push:
branches: [main]
branches: ["main","documentation"]

permissions:
contents: read
Expand Down
62 changes: 1 addition & 61 deletions docs/getting_started/code_samples.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,66 +17,6 @@ If you already have a Ray cluster set up, you can use the ``SubmitRayJob`` opera

In the below example(``ray_taskflow_example.py``), the ``@task.ray`` decorator is used to define a task that will be executed on the Ray cluster.

.. code-block:: python
from airflow.decorators import dag, task as airflow_task
from datetime import datetime, timedelta
from ray_provider.decorators.ray import task
RAY_TASK_CONFIG = {
"conn_id": "ray_conn",
"runtime_env": {
"working_dir": "/usr/local/airflow/dags/ray_scripts",
"pip": ["numpy"],
},
"num_cpus": 1,
"num_gpus": 0,
"memory": 0,
"poll_interval": 5,
}
@dag(
dag_id="ray_taskflow_example",
start_date=datetime(2023, 1, 1),
schedule_interval=timedelta(days=1),
catchup=False,
default_args={
"owner": "airflow",
"retries": 1,
"retry_delay": timedelta(minutes=5),
},
tags=["ray", "example"],
)
def ray_taskflow_dag():
@airflow_task
def generate_data():
import numpy as np
return np.random.rand(100).tolist()
@task.ray(config=RAY_TASK_CONFIG)
def process_data_with_ray(data):
import ray
import numpy as np
@ray.remote
def square(x):
return x**2
ray.init()
data = np.array(data)
futures = [square.remote(x) for x in data]
results = ray.get(futures)
mean = np.mean(results)
print(f"Mean of squared values: {mean}")
return mean
data = generate_data()
process_data_with_ray(data)
ray_example_dag = ray_taskflow_dag()
.. literalinclude:: ../../example_dags/ray_taskflow_example.py

Remember to adjust file paths, connection IDs, and other specifics according to your setup.

0 comments on commit 60e55ae

Please sign in to comment.