Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

A high-throughput and memory-efficient inference and serving engine for LLMs

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

Run any open-source LLMs, such as Llama 3.1, Gemma, as OpenAI compatible API endpoint in the cloud.

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and much more!

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.

SGLang is a fast serving framework for large language models and vision language models.

Superduper: Integrate AI models and machine learning workflows with your database to implement custom AI applications, without moving your data. Including streaming inference, scalable model hosting, training and vector search.

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

AICI: Prompts as (Wasm) Programs

RayLLM - LLMs on Ray

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

A throughput-oriented high-performance serving framework for LLMs

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

LLM (Large Language Model) FineTuning

Efficient AI Inference & Serving

This is suite of the hands-on training materials that shows how to scale CV, NLP, time-series forecasting workloads with Ray.

Multi-node production GenAI stack. Run the best of open source AI easily on your own servers. Easily add knowledge from documents and scrape websites. Create your own AI by fine-tuning open source models. Integrate LLMs with APIs. Run gptscript securely on the server

Finetune LLMs on K8s by using Runbooks

Since the emergence of chatGPT in 2022, the acceleration of Large Language Model has become increasingly important. Here is a list of papers on accelerating LLMs, currently focusing mainly on inference acceleration, and related works will be gradually added in the future. Welcome contributions!