Bitzu

AIF-C01 ยท Designing foundation model applications

What do temperature, top-p, top-k and maximum output length control?

Effect of temperature, top-p, top-k and maximum output length on randomness, determinism and response size.

Key points

  • Temperature reshapes the next-token probabilities: low values make the model pick likely tokens (more deterministic), high values flatten the distribution (more varied).
  • Top-k limits sampling to the k most likely tokens; top-p limits it to the smallest set whose cumulative probability reaches p. Maximum output length caps how many tokens the response may contain, which bounds size and output cost.

Source: https://docs.aws.amazon.com/bedrock/latest/userguide/inference-parameters.html

Practice questions

Single choice

Which temperature setting is most likely to give nearly identical answers to the same prompt on repeated calls?

  1. Temperature close to 0
  2. Temperature close to the maximum
  3. Any value, because temperature never affects variation
  4. Temperature set equal to top-k
Show answer

Answer: A. Temperature close to 0

A temperature near 0 makes the model favour the highest-probability token, so output varies little. High values flatten the distribution and increase variation.

Select all that apply

Which statements about sampling parameters are correct?

  1. Top-k sets the maximum number of tokens in the answer
  2. Top-k restricts sampling to the k most likely next tokens
  3. Top-p sets the language of the answer
  4. Top-p restricts sampling to the smallest set of tokens covering probability p
  5. Raising top-p narrows the set of candidate tokens
Show answer

Answer: B. Top-k restricts sampling to the k most likely next tokens, D. Top-p restricts sampling to the smallest set of tokens covering probability p

Top-k fixes how many candidate tokens are considered, while top-p fixes the share of cumulative probability, so the candidate count varies per step. Lowering either narrows the pool. Neither one sets the response length; that is the maximum output length.

True or false

Setting the maximum output length to a small number makes the model write a shorter, complete answer.

Show answer

Answer: False

False. The limit only cuts generation off at that many tokens, so a long answer is truncated mid-thought rather than rewritten more briefly. To get a short answer, also instruct the model to be brief.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More AIF-C01 topics in this module