Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Here are the key points:

The Problem: When LLMs generate text, they typically use either greedy decoding (always picking the most likely token) or temperature sampling. Current sampling methods often struggle to balance diversity with accuracy, especially for reasoning tasks.

The Innovation: The authors discovered that when LLMs generate tokens, the logits (pre-softmax scores) naturally separate into two regions:

• A "noisy" region following a Gaussian distribution (background noise)

• An "informative" region containing the actually relevant tokens

The Solution: Top-nσ works by:

• Identifying the maximum logit value

• Selecting tokens that are within n standard deviations (σ) of this maximum

• Only sampling from these selected tokens

• Using temperature to control sampling within this filtered set

Key Benefits:

• Maintains consistent performance even at high temperatures, unlike other methods

• Computationally efficient as it operates directly on logits

• Outperforms both existing sampling methods and greedy decoding on reasoning tasks

• Works particularly well for tasks requiring careful reasoning

Results: The method was tested on four reasoning-focused datasets and showed superior performance, especially at higher temperatures where other methods typically fail.

The paper essentially shows that by being more selective about which tokens to sample from based on their statistical properties, you can get better and more reliable results from language models, particularly for tasks that require careful reasoning.

Motivation

Looks to be the best sampler yet, and will be a clear differentiator for llama.cpp

Possible Implementation

See white paper: "Top-nσ Not All Logits Are You Need"