AIF-C01 ยท Designing foundation model applications
What do temperature, top-p, top-k and maximum output length control?
Effect of temperature, top-p, top-k and maximum output length on randomness, determinism and response size.
Key points
- Temperature reshapes the next-token probabilities: low values make the model pick likely tokens (more deterministic), high values flatten the distribution (more varied).
- Top-k limits sampling to the k most likely tokens; top-p limits it to the smallest set whose cumulative probability reaches p. Maximum output length caps how many tokens the response may contain, which bounds size and output cost.
Source: https://docs.aws.amazon.com/bedrock/latest/userguide/inference-parameters.html
Practice questions
Which temperature setting is most likely to give nearly identical answers to the same prompt on repeated calls?
- Temperature close to 0
- Temperature close to the maximum
- Any value, because temperature never affects variation
- Temperature set equal to top-k
Show answer
Answer: A. Temperature close to 0
A temperature near 0 makes the model favour the highest-probability token, so output varies little. High values flatten the distribution and increase variation.
Which statements about sampling parameters are correct?
- Top-k sets the maximum number of tokens in the answer
- Top-k restricts sampling to the k most likely next tokens
- Top-p sets the language of the answer
- Top-p restricts sampling to the smallest set of tokens covering probability p
- Raising top-p narrows the set of candidate tokens
Show answer
Answer: B. Top-k restricts sampling to the k most likely next tokens, D. Top-p restricts sampling to the smallest set of tokens covering probability p
Top-k fixes how many candidate tokens are considered, while top-p fixes the share of cumulative probability, so the candidate count varies per step. Lowering either narrows the pool. Neither one sets the response length; that is the maximum output length.
Setting the maximum output length to a small number makes the model write a shorter, complete answer.
Show answer
Answer: False
False. The limit only cuts generation off at that many tokens, so a long answer is truncated mid-thought rather than rewritten more briefly. To get a short answer, also instruct the model to be brief.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
