Skip to content

Commit

Permalink
Update api endpoint documentation for contributors (apache#42216)
Browse files Browse the repository at this point in the history
  • Loading branch information
@pierrejeambrun
pierrejeambrun authored Sep 16, 2024
1 parent d1e500c commit e321ce6
Show file tree
Hide file tree
Showing 2 changed files with 60 additions and 88 deletions.
147 changes: 59 additions & 88 deletions contributing-docs/17_adding_api_endpoints.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,122 +18,93 @@
Adding a New API Endpoint in Apache Airflow
===========================================

This documentation outlines the steps required to add a new API endpoint in Apache Airflow. It includes defining the endpoint in the OpenAPI specification, implementing the logic, running pre-commit checks, and documenting the changes.
This documentation outlines the steps required to add a new API endpoint in Apache Airflow. It includes implementing the logic, running pre-commit checks, and documenting the changes.

**The outline for this document in GitHub is available at top-right corner button (with 3-dots and 3 lines).**

Step 1: Define the Endpoint in ``v1.yaml``
----------------------------------------
1. Navigate to the ``v1.yaml`` file, which contains the OpenAPI specifications.
2. Add a new path for your endpoint, specifying the URL path, HTTP method (GET, POST, etc.), and a brief summary.
3. Define the parameters required for the endpoint, including types, required/optional status, and default values.
4. Describe the responses, including status codes, content types, and schema details.

Example:
Introduction
------------

The source code for the RestAPI is located under ``api_fastapi``. Endpoints are located under ``api_fastapi/views`` and contains different types of endpoints. The main two are ``public`` and ``ui``.
Public endpoints are part of the public API, standardized, well documented and most importantly backward compatible. UI endpoints are custom endpoints made for the frontend that do not respect backward compatibility i.e they can be adapted at any time for UI needs.
When adding an endpoint you should try as much as possible to make it reusable by the community, have a stable design in mind, standardized and therefore part of the public API. If this is not possible because the data types are too specific or subject to frequent change
then adding it to the UI endpoints is more suitable.


.. code-block:: yaml
paths:
/example/endpoint:
get:
summary: Example API endpoint
description: This endpoint provides an example response.
parameters:
- name: example_param
in: query
required: true
schema:
type: string
responses:
"200":
description: Successful response
content:
application/json:
schema:
type: object
properties:
message:
type: string
"400":
$ref: "#/components/responses/BadRequest"
"404":
$ref: "#/components/responses/NotFound"
Step 2: Implement the Endpoint Logic
Step 1: Implement the Endpoint Logic
------------------------------------
1. In the appropriate Python file, implement the endpoint's logic.
2. Ensure proper parameter handling and validation.
3. Implement the core logic, such as data retrieval or processing.
4. Add error handling for potential issues like missing parameters or invalid data.
1. Considering the details above decide whether your endpoints should be part of the ``public`` or ``ui`` interface.
1. Navigate to the appropriate views directory in ``api_fastapi/views``.
2. Register a new route for your endpoint with the appropriate HTTP method, query params, permissions, body type, etc.
3. Specify the appropriate Pydantic type in the return type annotation.

Example:

.. code-block:: python
@security.requires_access_dag("GET", DagAccessEntity.TASK_LOGS)
@provide_session
@unify_bucket_name_and_key
@provide_bucket_name
def get_example(
*,
example_param: str,
session: Session = NEW_SESSION,
) -> APIResponse:
# Implementation details here
pass
@dags_router.get("/dags") # permissions go in the dependencies parameter here
async def get_dags(
*,
limit: int = 100,
offset: int = 0,
tags: Annotated[list[str] | None, Query()] = None,
dag_id_pattern: str | None = None,
only_active: bool = True,
paused: bool | None = None,
order_by: str = "dag_id",
session: Annotated[Session, Depends(get_session)],
) -> DAGCollectionResponse:
pass
Step 2: Add tests for your Endpoints
------------------------------------
1. Verify manually with a local API that the endpoint behaves as expected.
2. Go to the test folder and initialize new tests.
3. Implements extensives tests to validate query param, permissions, error handling etc.


Step 3: Documentation
---------------------
Documentation is built automatically by FastAPI and our pre-commit hooks. Verify by going to ``/docs`` that the documentation is clear and appears as expected (body and return types, query params, validation)

Step 3: Run Pre-commit Hooks

Step 4: Run Pre-commit Hooks
-----------------------------
1. Ensure all code meets the project's quality standards by running pre-commit hooks.
2. Pre-commit hooks include static code checks, formatting, and other validations.
3. One specific pre-commit hook to note is the ``update-common-sql-api-stubs`` hook. This hook automatically updates the common SQL API stubs whenever it recognizes changes in the API. This ensures that any modifications to the API are accurately reflected in the stubs, maintaining consistency between the implementation and documentation.
3. Persisted openapi spec is located in ``v1-generated.yaml`` and a hook will take care of updating it based on your new endpoint, you just need to add and commit the change.
4. Run the following command to execute all pre-commit hooks:

.. code-block:: bash
pre-commit run --all-files
Optional: Adding Schemas
-----------------------------
In some cases, you may need to define additional schemas for new data structures. For example, if you are adding an endpoint that involves new data objects or collections, you may define a schema in a Python file. Here's an example:
Optional: Adding Pydantic Model
-------------------------------
In some cases, you may need to define additional models for new data structures. For example, if you are adding an endpoint that involves new data objects or collections, you may define a model in a Python file. The model will be used to validate and serialize/deserialize objects. Here's an example:

.. code-block:: python
class TaskLogPageSchema(Schema):
"""Schema for task log pagination details."""
total_pages = fields.Int(description="Total number of pages for task logs.")
current_page = fields.Int(description="Current page number.")
page_size = fields.Int(description="Number of logs per page.")
These schemas are defined to structure and validate the data handled by the API. Once defined, you can include these schemas in the OpenAPI specification file (e.g., v1.yaml) and reference them in the API endpoint definitions.

For example, in v1.yaml, you might add:
class DAGModelResponse(BaseModel):
"""DAG serializer for responses."""
.. code-block:: yaml
dag_id: str
dag_display_name: str
is_paused: bool
is_active: bool
last_parsed_time: datetime | None
components:
schemas:
TaskLogPage:
type: object
properties:
total_pages:
type: integer
description: Total number of pages for task logs.
current_page:
type: integer
description: Current page number.
page_size:
type: integer
description: Number of logs per page.
These models are defined to structure and validate the data handled by the API. Once defined, these models will automatically be added to the OpenAPI spec file as long as they are actually used by one endpoint.

Including schemas helps in automatically generating API documentation and ensures consistent data structures across the API.
After adding or modifying Pydantic models, make sure to run the pre-commit hooks again to update any generated files.

After adding or modifying schemas, make sure to run the pre-commit hooks again to update any generated files.
Situational: Legacy Endpoint Migration
--------------------------------------
When migrating legacy endpoint to the new FastAPI API:

Note on the New FastAPI API
---------------------------
As part of the AIP-84 you may be adding new endpoints to the FastAPI API by migrating some legacy endpoints. In case you are doing so don't forget to mark the legacy endpoint (Flask one) with the ``@mark_fastapi_migration_done`` decorator. This will help maintainers keep track of the endpoints remaining for the migration and those already migrated.
1. Implement a feature complete endpoint in comparison to the legacy one or explain why this is not possible in your context.
2. Make sure to have a good test coverage by copying over the legacy test cases to the new endpoint. This will guarantee an isofunctional new endpoint.
3. Mark the legacy endpoint with the ``@mark_fastapi_migration_done`` decorator. This will help maintainers keep track of the endpoints remaining for the migration and those already migrated.
1 change: 1 addition & 0 deletions docs/spelling_wordlist.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1797,6 +1797,7 @@ verticaql
Vevo
videointelligence
VideoIntelligenceServiceClient
views
virtualenv
virtualenvs
vm
Expand Down

0 comments on commit e321ce6

Please sign in to comment.