Computational Engineering / AI and machine learning for structural engineers
Overfitting and generalisation
Why training error falls forever, why test error turns round, and why the gap is invisible from the training data alone.
Lesson review: expert verification remains open
This lesson has received a full-context editorial review and a check of its technical explanations. Selected arithmetic has been checked separately. This is not independent engineering sign-off.
Checks needed for project use
- A computational engineering reviewer must check algorithms, convergence claims, statistical assumptions and numerical examples beyond the cases explicitly recorded.
- Reproduce seeded learning experiments and label model-selection data as validation, retaining an independent test procedure.
- Obtain qualified review for legal responsibility statements; this is not a legal opinion.
- Review domain guards and calibration before any real engineering use.
These are educational models and exercises. Check their assumptions before using the results. The full technical content and every diagram have not yet been independently verified.
Concrete and steel calculations retain their stated first-generation Eurocode teaching basis. Use the standards and amendments specified for your project; do not mix generations without an appropriate design basis.
Read the lesson register and outstanding review notesWhy this matters
Overfitting is usually explained with a picture of a wiggly line through some dots. That gets the shape right and misses the consequence, which is: you cannot detect it from the data you trained on. The training error is falling the whole time. Everything looks like progress.
The only way to see it is to hold data back, and that is not a technicality — it is the entire argument for a test set.
By the end of this lesson you should be able to
- Explain why training error falls with every added parameter
- Read the point where test error turns round
- Explain the role of a held-back test set
- Recognise the engineering analogue in curve-fitting practice
Training and validation error
In a nested, unregularised polynomial family, the minimum training loss cannot increase when another coefficient is added, because the simpler fit remains available. It may stay equal; a numerical fitting algorithm need not attain that minimum. Distinct input locations allow a polynomial of degree at most n−1 to interpolate n scalar observations, but this does not guarantee useful prediction.
Held-out validation error can improve and then worsen as flexibility increases. It need not follow a smooth U-shaped curve, and an error gap can also reflect distribution differences or data problems. Use representative evaluation and uncertainty estimates.
In the displayed study, degree 3 has error 1.431 on the set used to compare configurations, while degree 10 has 1.657. Because that set informs model choice, call it a validation set. Reserve separate test data, or use nested cross-validation, for an independent performance estimate after selection.
A finite noisy sample can favour a model more complex than the generating function. This does not establish a general preferred degree or a universal observations-per-parameter rule. Examine sample dependence, regularisation and the intended prediction domain.
Try it
Overfitting explorer
Fit a polynomial of increasing degree to noisy data drawn from a straight line. Two error curves, and only one of them turns round.
More data moves the best degree back towards the true relationship.
- Degree selected
- 3
- Training error
- 0.6990
- Test error
- 1.4310
- Test / training ratio
- 2.0
- Best degree by test error
- 3
- Points needed to fit exactly
- 12
4 parameters
a healthy model is near 1.5; nineteen is overfitting
degree 11 has one parameter per point
This is the best degree by test error. Note that the data came from a STRAIGHT LINE — the best model on a finite noisy sample is not necessarily the true one.
What to try
- Push the degree to 10 and watch the training error collapse while the test error rises. A model chosen on training error alone would pick this.
- Set the noise to zero. The two curves converge, because there is no noise for the extra parameters to learn.
- Increase the training points. The best degree moves back towards 1 — more data shifts the balance towards the true relationship.
- The polynomial fit itself is a least-squares solve, and at high degree the normal-equations matrix becomes ill-conditioned — the same arithmetic problem as Module 12's stiffness range.
What this shows: Training error falls with every added parameter; test error turns round — and nothing in the training data reveals the gap.
Worked example
Choosing a model on the wrong curve
Given
- The table above: training and test error against polynomial degree
- An engineer with access to the training data only
Find
What degree they would choose, and what it costs
Practice
A model achieves a training error of 0.089 and a test error of 1.657. What is the ratio of test error to training error?
Practice
Twelve training points are fitted with a polynomial. At what degree does the polynomial have exactly enough parameters to pass through every point?
Worked example
How many parameters is too many
Given
- A model is fitted to 40 measured floor deflections
- The candidate is a polynomial in span, depth and load, with all cross terms up to second order
Find
How many parameters that is, and whether 40 data points can support it
Practice
A full second-order polynomial in three variables has how many parameters, counting the constant?
Practice
40 data points are used to fit a model with 10 parameters. How many observations per parameter is that?
Practice
A model’s training error is 0.20 and its test error is 1.60. What is the ratio of test error to training error?
Check yourself
Why can the minimum unregularised training loss not increase in a nested polynomial model family fitted to the same data?
Check yourself
An engineer tries fifteen model configurations and reports the test error of the best one. What is wrong with that figure?
Check yourself
What does cross-validation buy that a single train/test split does not?
Summary
- Training error falls with every added parameter, necessarily
- Test error falls then rises; the gap is the model learning the noise
- Overfitting is undetectable from the training data alone
- No universal test/train error ratio establishes a healthy model
- Choosing a configuration on the test set consumes it — use a third split