Machine Learning Associate ยท Model evaluation and generalization
Underfitting vs overfitting and the bias-variance tradeoff
Underfitting versus overfitting, training versus validation error patterns, and responses such as regularization, simpler or more complex models, and more data.
Key points
- Underfitting (high bias): the model is too simple, so training and validation error are both high.
- Overfitting (high variance): training error is low but validation error is much higher.
- Remedies for overfitting: regularization, a simpler model, more training data.
- Remedies for underfitting: a more complex model, more or better features, less regularization.
- More data helps variance, not bias.
Source: https://scikit-learn.org/stable/modules/learning_curve.html
Practice questions
A model shows very low training error but much higher validation error. What is this pattern called?
- Underfitting
- Data leakage into the test set only
- Overfitting
- A well-fitted model
Show answer
Answer: C. Overfitting
A large gap with low training error is overfitting (high variance). Underfitting would show high error on both sets.
Which actions can reduce overfitting?
- Increase the regularization strength
- Add more training data
- Reduce the model's complexity, for example a shallower tree
- Add more layers to the network
- Train for many more epochs without monitoring validation error
Show answer
Answer: A. Increase the regularization strength, B. Add more training data, C. Reduce the model's complexity, for example a shallower tree
Stronger regularization, more data and a simpler model all lower variance. Adding layers and training longer increases capacity and tends to worsen overfitting.
True or false: collecting more training data is a reliable fix for an underfitting model.
Show answer
Answer: False
False. A model that is too simple stays too simple with more rows; both errors remain high. More data mainly helps when variance is the problem.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
