Bitzu

AIF-C01 ยท Generative AI concepts

Tokens, chunking, embeddings and vectors

What a token is, how text is chunked, what embeddings and vectors represent and why they enable semantic search.

Key points

  • A token is the unit a model reads and writes: a word, part of a word or a punctuation mark.
  • Long documents are split into chunks so they fit the model and can be searched piece by piece.
  • An embedding model turns each chunk into a vector, a list of numbers.
  • Chunks with similar meaning get vectors that sit close together, which is what makes semantic search possible.

Source: https://docs.aws.amazon.com/bedrock/latest/userguide/key-definitions.html

Practice questions

Single choice

What does an embedding represent?

  1. A numeric vector that captures the meaning of the input
  2. A compressed copy of the original document
  3. The number of tokens in a prompt
  4. An encrypted version of the input text
Show answer

Answer: A. A numeric vector that captures the meaning of the input

An embedding is a numeric vector that captures meaning so items can be compared. It is not a compressed file, a token count or an encryption of the text.

Select all that apply

Which statements about tokens are correct?

  1. A token can be a part of a word or a punctuation mark
  2. Every word is always exactly one token
  3. A model's context window is measured in tokens
  4. Tokens are the vectors stored in a vector database
  5. Token-based pricing counts the tokens processed
Show answer

Answer: A. A token can be a part of a word or a punctuation mark, C. A model's context window is measured in tokens, E. Token-based pricing counts the tokens processed

Tokens can be whole words, word parts or punctuation, and billing and context limits are counted in them. Tokens are not always one per word, and they are not the same as embeddings.

True or false

True or false: semantic search through embeddings only finds text that shares the exact same words as the query.

Show answer

Answer: False

False. Embeddings place similar meanings near each other, so paraphrases and synonyms match even without shared words.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More AIF-C01 topics in this module