Underfitting and Overfitting in ML

- Overview

Overfitting and underfitting are two common modeling errors in machine learning (ML) where a model either memorizes training noise or fails to capture basic data patterns. Machine learning (ML) models should learn useful patterns from training data. Underfitting or overfitting occurs when a model learns too little or too much.

Underfitting means the model is too simple to capture all the genuine patterns in the data. Overfitting means the model has learned not only the underlying patterns but also the noise or random outliers present in the training data.

A good model strikes the right balance: it is complex enough to capture genuine patterns but not so complex that it "memorizes" the noise.

1. What is Underfitting? 

Underfitting happens when a machine learning (ML) model is too simple to discover the underlying structure of the data.

• Characteristics: High bias, low variance, and poor performance on both training and test datasets.
• Common Causes: Using models that are too basic (like a straight line for curved data), missing important features, or under-training.

How to Fix:

• Increase model complexity.
• Add more relevant input features.
• Reduce regularization. 

2. What is Overfitting? 

Overfitting occurs when a model learns the training data too well, capturing random noise and minor fluctuations instead of the true trend.

• Characteristics: Low bias, high variance, excellent performance on training data, but poor performance on new, unseen test data. 

• Common Causes: Overly complex models (like high-degree polynomials), too many parameters, or training on a small dataset with lots of noise.

How to Fix:

• Adding regularization (L1/L2).
• Gather more training data.

• Apply early stopping or cross-validation.

- How to Check if the Model is Overfitting or Underfitting? 

You can check if a machine learning (ML) model is overfitting or underfitting by comparing its performance metrics - such as error or accuracy - between the training data and validation/test data. 

1. Signs of Overfitting: 

Overfitting happens when a model memorizes the training data instead of learning general patterns. It performs great on training data but fails on new data. 

• High Training vs. Test Performance Gap: Training accuracy is very high (or training loss is near zero), but validation/test error is significantly higher.
• Diverging Loss Curves: During training, training loss continues to decrease, but validation loss starts to increase or plateau.
• Brittle Predictions: The model's predictions swing wildly or fail when small, minor changes are made to the input data.

 
2. Signs of Underfitting: 

Underfitting happens when a model is too simple to capture the underlying structure of the data. It performs poorly on both training and validation sets. 

• High Error Across the Board: Both training loss and validation/test loss remain high, or accuracy remains low across both sets.
• Stagnant Improvement: Training and validation metrics stop improving early in the training process and flatline at poor performance levels.

3. How to Test for Fit:

• Holdout Validation: Split your data into a training set and an independent test set to see how the model handles unseen examples.
• Cross-Validation: Use techniques like k-fold cross-validation to ensure your evaluation isn't biased by a single train-test split.
• Learning Curves: Plot training and validation errors over time or across different dataset sizes to visually spot divergence or high baseline error.

[More to come ...]