Underfitting Explained: Definition, Causes, Diagnosis Methods, and Common Misconceptions
Underfitting is a state in which a model fails to sufficiently learn the relationship between inputs and outputs, and the training error remains high. This article explains the causes of overly simple models and shows how to judge model status using training error and validation error, making it suitable for machine learning beginners and AI model diagnosis.
What Is Underfitting?
Underfitting refers to a situation in which a machine learning model has not sufficiently learned the relationships in the data. Common signs include training error that is difficult to reduce, or model performance on the training data that remains poor. It indicates that the model has not accurately captured the relationship between input variables and the output target, so not only does it perform poorly on the training set, but it may also produce a high error rate on unseen data.
From the perspective of model diagnosis, underfitting is not that “the model failed to memorize a few specific samples,” but rather that the model’s overall expressive capacity is insufficient, or that it has not learned the main relationships behind the data. For AI model training, the significance of identifying underfitting is to first determine whether the model has even learned the basic patterns in the training data before further discussing generalization ability.
Key point: If a model already performs poorly on the training data, underfitting should usually be considered first, rather than immediately discussing whether it has adapted too strongly to the training samples.
Understanding Underfitting by Fitting a Curve with a Straight Line
An intuitive analogy is using a straight line to fit a set of data points that clearly follow a curved distribution. The data itself has a curved trend, but the model can only express a linear relationship, so no matter how it is adjusted, this straight line will have difficulty fitting the main changes in the data.
The point of this analogy is not that “a straight line is always bad,” but that when the model form is too simple, it may be unable to express the trends and patterns that truly exist in the training data. In other words, underfitting reflects that the model does not have enough capacity to describe the relationship between inputs and outputs.
| Analogy element | Meaning in machine learning |
|---|---|
| Data points that clearly follow a curve | More complex trends or patterns exist in the data |
| A model that can only draw a straight line | The model form or expressive capacity is too simple |
| Poor fit even on the training data | Training error is high, and the model has not learned the key relationships |
| Poor performance on new data as well | The error rate on unseen data may also be high |
Why Does Underfitting Happen?
At the level of the basic definition, underfitting is often related to a model being too simple. If a model lacks the ability to capture the underlying trends and patterns in the training data, it will be difficult for it to learn the relationship between input examples and target values, and it will therefore perform poorly on the training data.
The Model Form Is Too Simple
When a model is too simple, the relationships it can express are limited. If the relationship between inputs and outputs in the real data is more complex, the model cannot effectively fit that relationship, and the training error will also be difficult to reduce further.
Failure to Capture Underlying Trends and Patterns
The goal of a machine learning model is usually not to memorize samples one by one, but to learn patterns from the training data that can be used for prediction. Underfitting means that the model has not captured these patterns, or has captured them insufficiently, so it may have a high error rate on both training samples and unseen samples.
The Relationship Between Inputs and Outputs Has Not Been Accurately Learned
If a model cannot describe the relationship between input variable X and target value Y, its predictions will deviate from the true target. In this case, even if you continue to observe the training error, you may find that the error remains at a relatively high level.
What Is the Difference Between Underfitting and Overfitting?
Both underfitting and overfitting mean that the model’s performance is not ideal, but the direction of the problem is different. Underfitting usually indicates that the model has not learned the key relationships in the training data; overfitting may appear as training error being much lower than validation error, meaning the model performs well on the training data but insufficiently on validation data or unseen data.
| Dimension | Underfitting | Overfitting |
|---|---|---|
| Performance on training data | Usually poor, with training error still high | Usually good, with relatively low training error |
| Performance on validation data or unseen data | May also be poor, with a relatively high error rate | May become noticeably worse, with validation error higher than training error |
| Typical error relationship | Training error is difficult to reduce, and the training set itself is not well fitted | Training error is much lower than validation error |
| Core issue | The model has not sufficiently learned the input-output relationship | The model adapts too strongly to the training data and lacks generalization |
| Diagnostic focus | First check whether performance on the training set is already poor | Compare the gap between training error and validation error |
It should be noted that low training error does not automatically mean low generalization error. Simply minimizing training error does not necessarily improve the model’s performance on unseen data. Therefore, when judging the model’s status, you should pay attention to its performance on both training data and validation data.
How Can Training Error and Validation Error Be Used to Judge Model Status?
Training error and validation error are important entry points for understanding underfitting and overfitting. Training error reflects how well the model fits the training data, while validation error or performance on unseen data helps determine whether the model has generalization ability.
First Look at Performance on the Training Data
If the model’s error on the training data remains high, or if its overall performance is poor, this is usually a signal of underfitting. Because the model has not even fitted the training samples it has already seen, it may not have learned the main relationship between inputs and outputs.
Then Combine This with Performance on Validation Data or Unseen Data
If the error rate is high on both the training set and unseen data, this is more consistent with underfitting. In this case, the issue is not just that the model is unstable on new data, but that the model has also failed to master the basic patterns in the training data.
Do Not Treat Training Error as the Only Objective
Training error can help detect underfitting, but it cannot be used alone to prove that a model is reliable. Because training error cannot be directly equated with generalization error, simply pursuing the minimization of training error may not bring better validation performance.
| Observation | Possible meaning | Next diagnostic focus |
|---|---|---|
| Training error remains high | Underfitting may exist | Check whether the model is too simple and whether it has learned the input-output relationship |
| Training error is low but validation error is high | Overfitting may exist | Focus on the gap between training performance and validation performance |
| Only a single error metric is available | It is difficult to judge whether the model is reliable | Observe performance on both training data and validation data from a generalization perspective |
After Detecting Underfitting, What Directions Should Be Analyzed?
After detecting underfitting, the first step is to confirm the nature of the problem: whether the model is too simple, whether it has failed to capture the relationship between inputs and outputs, and whether it is unable to learn the underlying trends and patterns in the training data. In practical analysis, it is not appropriate to directly equate an underfitting diagnosis with a fixed parameter-tuning approach; instead, you should first determine whether the observed phenomenon is consistent with underfitting.
You can conduct an initial check from the following perspectives:
| Check perspective | Questions to observe | Significance for diagnosing underfitting |
|---|---|---|
| Training set performance | Does the model still have a relatively high error rate on the training data? | If the training set performance is also poor, underfitting is more likely |
| Input-output relationship | Does the model accurately capture the relationship between input variables and the output target? | If the relationship has not been learned, it indicates insufficient model expression or learning |
| Data trends and patterns | Can the model describe the main trends in the training data? | If it cannot describe the trends, the model may be too simple |
| Distinguishing from overfitting | Is training error much lower than validation error? | If this phenomenon appears, it is closer to overfitting than underfitting |
You can use the checklist below for a quick review:
- Is performance on the training set already poor?
- Is the error rate on unseen data also high?
- Has the model failed to capture the relationship between inputs and outputs?
- Is the model too simple to describe the trends and patterns in the data?
- Is the current phenomenon different from the overfitting pattern of “training error being much lower than validation error”?
If most of these questions point to “the training data has not been fitted well either,” then underfitting can be treated as the main diagnostic direction. After that, you can carefully decide whether the model design or training process needs to be adjusted based on the specific task, model type, and data situation.