Yalp Tcefrep Project

Overview

This project provides a FastAPI service for accessing player metrics stored in Google BigQuery. It includes an in-memory caching mechanism to optimize performance by reducing the number of direct queries to BigQuery.

Prerequisites

• Git
• Terraform
• Python
• Google Cloud SDK
• Docker

Installation

1. Clone the Repository:

   git clone https://github.com/euhoro/yalp-tcefrep.git
   cd yalp-tcefrep

2. Add an Admin Service Account:

   Ensure you have the following service account with the necessary roles in your GCP project:

   • Email: [email protected]
   • Roles:
     • Artifact Registry Administrator
     • Artifact Registry Writer
     • BigQuery User
     • Container Analysis Occurrences Viewer
     • Editor
     • Owner
     • Storage Admin

3. Download the key file and place it in the root folder of the project as key.json.

4. Place the raw_data folders containing the parquet files in the root folder

5. Initialize and Apply Terraform:

   terraform init
   terraform apply

   This will create the yalp_tcefrep bucket and the necessary structure for the BigQuery table. This will create the api available in the gcp cloud for easy access

6. Setup Python Environment:

   Create a virtual environment and activate it:

   python3 -m venv venv
   source venv/bin/activate

7. Install Required Python Packages:

   pip install -r requirements.txt

8. Populate BigQuery Table:

   Run the following command to populate the BigQuery table:

   export GOOGLE_APPLICATION_CREDENTIALS="key.json"
   python notebooks/pandas_file_to_big_query.py

9. Build and Run Docker Container locally:

   docker build -t fastapi-metrics:v1.1.0 .
   docker run -p 8080:8080 fastapi-metrics:v1.1.0

Usage

The FastAPI service provides an endpoint to get player metrics. The service looks in the cache first and then queries BigQuery if the data is not cached. The cache is refreshed periodically.

Access the application:

• API Documentation

• Home

• Endpoint: GET /get_metric/

  • Request Body:

    {
      "player_id": "6671adc2dd588a8bda0367bb",
      "metric_name": "country"
    }

  • Response:

    {
      "player_id": "6671adc2dd588a8bda0367bb",
      "country": "USA",
      "query_time": 0.34,
      "total_time": 0.35
    }

    • query_time will be -1 if request reached the cache

Design

• Design Documentation