AIF-C01 ยท Generative AI concepts
Tokens, chunking, embeddings and vectors
What a token is, how text is chunked, what embeddings and vectors represent and why they enable semantic search.
Key points
- A token is the unit a model reads and writes: a word, part of a word or a punctuation mark.
- Long documents are split into chunks so they fit the model and can be searched piece by piece.
- An embedding model turns each chunk into a vector, a list of numbers.
- Chunks with similar meaning get vectors that sit close together, which is what makes semantic search possible.
Source: https://docs.aws.amazon.com/bedrock/latest/userguide/key-definitions.html
Practice questions
What does an embedding represent?
- A numeric vector that captures the meaning of the input
- A compressed copy of the original document
- The number of tokens in a prompt
- An encrypted version of the input text
Show answer
Answer: A. A numeric vector that captures the meaning of the input
An embedding is a numeric vector that captures meaning so items can be compared. It is not a compressed file, a token count or an encryption of the text.
Which statements about tokens are correct?
- A token can be a part of a word or a punctuation mark
- Every word is always exactly one token
- A model's context window is measured in tokens
- Tokens are the vectors stored in a vector database
- Token-based pricing counts the tokens processed
Show answer
Answer: A. A token can be a part of a word or a punctuation mark, C. A model's context window is measured in tokens, E. Token-based pricing counts the tokens processed
Tokens can be whole words, word parts or punctuation, and billing and context limits are counted in them. Tokens are not always one per word, and they are not the same as embeddings.
True or false: semantic search through embeddings only finds text that shares the exact same words as the query.
Show answer
Answer: False
False. Embeddings place similar meanings near each other, so paraphrases and synonyms match even without shared words.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
