AIF-C01 ยท Training and fine-tuning foundation models
Pre-training, continued pre-training, fine-tuning, distillation
Pre-training, continuous pre-training, fine-tuning and distillation and how they differ in goal, data and cost.
Key points
- Pre-training builds a model from scratch on a huge unlabeled corpus (most expensive).
- Continued pre-training keeps training an existing model on more unlabeled, domain text.
- Fine-tuning adjusts weights with a smaller labeled dataset for a task.
- Distillation transfers a large teacher model's behavior into a smaller, cheaper student model.
Source: https://docs.aws.amazon.com/bedrock/latest/userguide/custom-models.html
Practice questions
Which approach trains a foundation model from scratch on a very large unlabeled corpus?
- Fine-tuning
- Pre-training
- Distillation
- Prompt engineering
Show answer
Answer: B. Pre-training
Pre-training starts from randomly initialized weights. Fine-tuning and distillation start from an already trained model.
Which statements about distillation in Amazon Bedrock are correct?
- A larger teacher model produces responses used to train the student
- The result is a smaller, faster, cheaper model for a specific use case
- It requires a multi-terabyte unlabeled text corpus
- It increases the number of parameters of the student
- It builds a model from randomly initialized weights
Show answer
Answer: A. A larger teacher model produces responses used to train the student, B. The result is a smaller, faster, cheaper model for a specific use case
A teacher model generates responses to your prompts and those fine-tune a smaller student. It does not train a model from scratch, and it needs prompts, not a petabyte corpus.
True or false: fine-tuning a foundation model is about as expensive as pre-training it from scratch.
Show answer
Answer: False
False. Fine-tuning starts from trained weights and uses a comparatively small dataset, so it is far cheaper than pre-training.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
