Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve Key-Value Caching #383

Closed
wants to merge 3 commits into from
Closed

Improve Key-Value Caching #383

wants to merge 3 commits into from

Commits on Sep 18, 2023

  1. - Support passing in multiple tokens when using past_kv_cache. 

    - Add tests for past_kv_cache.
- Add documentation for past_kv_cache.
- Fix type hints for some components that assume left_attention_mask has same number of tokens as input. This was previously unnoticed because there were no tests that covered past_kv_cache.
    @UFO-101
    UFO-101 committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    9b0e81d View commit details
    Browse the repository at this point in the history

  2. Support freezing key-value caches.

    @UFO-101
    UFO-101 committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    033393e View commit details
    Browse the repository at this point in the history

  3. Integrate past_left_attention_mask into HookedTransformerKeyValueCach… 

    …e so that it doesn't need to managed manually. Remove from HookedTransformer forward().
    @UFO-101
    UFO-101 committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    5334792 View commit details
    Browse the repository at this point in the history