Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support FP8 KV Cache #652

Open
wants to merge 26 commits into
base: main
Choose a base branch
from
Open

Support FP8 KV Cache #652

wants to merge 26 commits into from
+241 −39

Commits on Oct 8, 2024

  1. (feat) : support fp8 kv cache

    @ajtejankar
    ajtejankar committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    6f887aa View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2024

  1. fix a few things

    @ajtejankar
    ajtejankar committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    93fc0d1 View commit details
    Browse the repository at this point in the history

  2. add support for fp8 kv cache using flash infer

    @ajtejankar
    ajtejankar committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    939b479 View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2024

  1. add logging

    @ajtejankar
    ajtejankar committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    56589ed View commit details
    Browse the repository at this point in the history

  2. merge back previous code to provide fp8_kv as a quantization option

    @ajtejankar
    ajtejankar committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    9590a71 View commit details
    Browse the repository at this point in the history

  3. remove unnecessary comments

    @ajtejankar
    ajtejankar committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    1517b16 View commit details
    Browse the repository at this point in the history

  4. remove is_fp8_kv_supported function

    @ajtejankar
    ajtejankar committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    a5f1c25 View commit details
    Browse the repository at this point in the history

  5. fix attention api and kv_dtype

    @ajtejankar
    ajtejankar committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    52021c0 View commit details
    Browse the repository at this point in the history

  6. keep ruff happy

    @ajtejankar
    ajtejankar committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    7dd10f3 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2024

  1. use fp16 prefill with fp8 kv

    @ajtejankar
    ajtejankar committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    787b58e View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2024

  1. use fp16 prefill for fp8 kv cache (without prefix caching)

    @ajtejankar
    ajtejankar committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    8dae2d8 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into fp8-kv-flash-infer

    @ajtejankar
    ajtejankar committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    8540461 View commit details
    Browse the repository at this point in the history

  3. add window_left option

    @ajtejankar
    ajtejankar committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    d880464 View commit details
    Browse the repository at this point in the history

  4. fix merge conflicts

    @ajtejankar
    ajtejankar committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    bb75730 View commit details
    Browse the repository at this point in the history

  5. move paged_attention import location

    @ajtejankar
    ajtejankar committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    3031ba4 View commit details
    Browse the repository at this point in the history

  6. add llama and qwen models

    @ajtejankar
    ajtejankar committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    1742cde View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2024

  1. refactor according to Travis' comments

    @ajtejankar
    ajtejankar committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    c732810 View commit details
    Browse the repository at this point in the history

  2. refactor llama and qwen models

    @ajtejankar
    ajtejankar committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    90e85f5 View commit details
    Browse the repository at this point in the history

  3. fix flash_attn call bug in qwen

    @ajtejankar
    ajtejankar committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    be6e9ff View commit details
    Browse the repository at this point in the history

  4. fix flash_attn call bug in llama

    @ajtejankar
    ajtejankar committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    f2f9a08 View commit details
    Browse the repository at this point in the history

  5. refactor flash_causal_lm for better git diff

    @ajtejankar
    ajtejankar committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    7974ff9 View commit details
    Browse the repository at this point in the history

  6. refactor to use fp8_kv in reshape_and_cache

    @ajtejankar
    ajtejankar committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    cd29d93 View commit details
    Browse the repository at this point in the history

  7. add support for gemma models and fix ruff error in qwen

    @ajtejankar
    ajtejankar committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    488d428 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2024

  1. fix is_fp8_supported function

    @ajtejankar
    ajtejankar committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    813b64b View commit details
    Browse the repository at this point in the history

  2. fix fp8 error handling

    @ajtejankar
    ajtejankar committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    5dfb5e5 View commit details
    Browse the repository at this point in the history

  3. keep ruff happy

    @ajtejankar
    ajtejankar committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    bca4daf View commit details
    Browse the repository at this point in the history