What is top-p sampling? Selecting candidates by cumulative probability
Understand nucleus sampling, top-k, and temperature through probability examples, including why thresholds are not accuracy scores and why support must be checked.
Understand the concept
Top-p or nucleus sampling retains the highest-probability candidates needed to reach a cumulative threshold and samples within that set. Its size changes with the distribution rather than staying fixed. The threshold is not the probability of a correct answer. Check model support before attempting to tune it.
A cumulative threshold determines a dynamic candidate set
With candidate probabilities 0.6, 0.3, and 0.1, a threshold of 0.8 retains the first two, totaling 0.9. This simplified example explains filtering; real vocabularies and decoding stacks are larger. A threshold of 0.8 does not guarantee 80% accuracy.
Concentrated distributions need few candidates; flatter ones need more. This differs from top-k’s fixed count. Both operate on token selection, not on judging source truth, and do not replace verification.
Temperature and top-p differ but can interact
Temperature reshapes the distribution; top-p filters candidates from it. Changing both obscures attribution. Hold model and other settings fixed and inspect repeated outputs rather than one successful example.
The nucleus-sampling paper addresses text-generation degeneration, not a universally optimal method. Exact extraction may not benefit from varied wording. Define task metrics first; different output does not automatically mean higher quality.
Check current support and warnings rather than old tables
Hugging Face defines top_p in generation configuration, but cloud services may restrict it. Current Google guidance says relevant Gemini 4 and 3.6-and-later models ignore custom sampling values. A field appearing in a request does not establish that it takes effect.
This page reports no sampling experiment or universal setting. Check task errors and format stability and record configuration and responses. Persistent factual errors call for examining evidence and capability rather than repeatedly shrinking the threshold.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.