How AI picks the next word
A model writes one token at a time by sampling from a list of candidates. Turn the temperature up and down and watch it get predictable or wild, right in your browser.
What you're looking at
A language model doesn’t plan a sentence and type it out. It predicts one token at a time: for the blank above, it produces a probability for every word it might say next, then picks one, adds it to the text, and does the whole thing again for the word after that. The bars are those probabilities. Everything you change with the sliders changes how that pick is made.
The odds here are a fixed illustration, not a live model, so the demo can run entirely in your browser with nothing uploaded. A real model computes them across tens of thousands of tokens, but the mechanism, probability then sample, is exactly this.
The three dials
- Temperature flattens or sharpens the odds. Low temperature makes the top word dominate (safe, repetitive); high temperature evens everyone out (surprising, sometimes nonsense). At 0 it’s greedy: it always takes the single most likely word, so the same prompt gives the same answer every time.
- Top-k keeps only the
kmost likely words and ignores the rest, so a long tail of weird options can never be chosen. - Top-p (nucleus) keeps the smallest group of top words whose odds add up to
p, then samples from just those. It adapts: a confident step keeps few words, an uncertain one keeps more.
This is why the same prompt can give different answers, and why “make it more creative” usually means “raise the temperature.” It’s also why a model can state something wrong so fluently: a plausible-sounding but false word can simply win the draw. See AI hallucinations for that failure mode.
Under the hood The softmax that turns scores into odds optional
The model doesn’t emit probabilities directly. It emits a raw score (a logit)
for each candidate word, and softmax squashes those scores into positive numbers that
add up to 1. Temperature T is a divisor applied before the exponent:
p(word i) = exp(logit_i / T) / Σ exp(logit_j / T)
Small T makes the gaps between logits bigger after dividing, so the top word’s share
balloons toward 1 (that’s greedy in the limit). Large T shrinks the gaps, pushing
every word toward the same share. Top-k and top-p then zero out the words you excluded and the rest are
re-scaled to add back up to 1 before the draw.
FAQ
Is this a real language model?
No, and it doesn’t pretend to be. The candidate words and their scores are hand-picked so you can play with the mechanism. A real model produces the same kind of distribution over its full vocabulary; the temperature, top-k, and top-p math shown here is the real thing.
Why do I get a different word each time?
Because above temperature 0 the pick is a weighted random draw, not a lookup. The most likely word wins most often, but any word with odds above zero can come up. Set temperature to 0 to make it deterministic.
What temperature should I use in practice?
Lower (near 0 to 0.3) for factual, structured, or code output where you want consistency; higher (0.7 to 1.0) for brainstorming and creative writing. Above ~1.2 most models start to wander. Exact ranges vary by provider.
Related
Count the tokens a model reads, see what a context window holds, or watch a vague prompt force the model to guess.