Bitzu

AIF-C01 ยท Training and fine-tuning foundation models

Pre-training, continued pre-training, fine-tuning, distillation

Pre-training, continuous pre-training, fine-tuning and distillation and how they differ in goal, data and cost.

Key points

  • Pre-training builds a model from scratch on a huge unlabeled corpus (most expensive).
  • Continued pre-training keeps training an existing model on more unlabeled, domain text.
  • Fine-tuning adjusts weights with a smaller labeled dataset for a task.
  • Distillation transfers a large teacher model's behavior into a smaller, cheaper student model.

Source: https://docs.aws.amazon.com/bedrock/latest/userguide/custom-models.html

Practice questions

Single choice

Which approach trains a foundation model from scratch on a very large unlabeled corpus?

  1. Fine-tuning
  2. Pre-training
  3. Distillation
  4. Prompt engineering
Show answer

Answer: B. Pre-training

Pre-training starts from randomly initialized weights. Fine-tuning and distillation start from an already trained model.

Select all that apply

Which statements about distillation in Amazon Bedrock are correct?

  1. A larger teacher model produces responses used to train the student
  2. The result is a smaller, faster, cheaper model for a specific use case
  3. It requires a multi-terabyte unlabeled text corpus
  4. It increases the number of parameters of the student
  5. It builds a model from randomly initialized weights
Show answer

Answer: A. A larger teacher model produces responses used to train the student, B. The result is a smaller, faster, cheaper model for a specific use case

A teacher model generates responses to your prompts and those fine-tune a smaller student. It does not train a model from scratch, and it needs prompts, not a petabyte corpus.

True or false

True or false: fine-tuning a foundation model is about as expensive as pre-training it from scratch.

Show answer

Answer: False

False. Fine-tuning starts from trained weights and uses a comparatively small dataset, so it is far cheaper than pre-training.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More AIF-C01 topics in this module