Every chemometric model report carries a small table of error figures. RMSEC. RMSECV. RMSEP. Sometimes SEP, SEC, or bias alongside. The numbers look interchangeable - they are all in the same units as the property being predicted, they are all small, they all sound reassuring. They are not interchangeable, and treating them as if they are is how a model that behaves beautifully on the developer’s laptop ends up disappointing at the plant.

The three metrics answer three different questions. RMSEC says how well the model fits the data it was built on. RMSECV estimates how well it would generalise if you were to swap out some of the calibration samples. RMSEP says how well it did on samples that were held out entirely, or that came in fresh from the process. The gap between the three is more informative than any one number, and a plant reviewing a vendor’s model brief should read them together.

RMSEC: the fit residual

RMSEC is the root-mean-square error of calibration. It is computed on the samples used to fit the model, comparing the model’s predictions of those samples against their reference values. Formally, it is the square root of the mean of the squared residuals from the calibration set.

RMSEC is bounded below by two things: the reference method’s own uncertainty (you cannot calibrate a model to be more accurate than the truth it was trained against) and the number of latent variables in the model. Add more PLS components and RMSEC will fall - eventually to zero, if you keep adding until the model has as many components as samples. This is the trap. A vanishingly small RMSEC is a signal of overfitting, not of a good model.

RMSEC is useful for one thing: quickly ruling out a model that cannot even fit its own training data. If RMSEC is comparable to the reference method’s precision, the model has captured what is capturable. If RMSEC is much larger than the reference precision, the model is under-specified - too few components, wrong preprocessing, wrong data range - and no amount of cross-validation will fix that.

RMSEC alone is never sufficient evidence for anything else. A model report that reports RMSEC without RMSECV or RMSEP is telling you nothing about how the model behaves on samples it has not seen.

RMSECV: the leave-something-out estimate

RMSECV is the root-mean-square error of cross-validation. The calibration set is split into segments; the model is refit on all-but-one segment and used to predict the held-out segment; the residuals across all held-out predictions are pooled into a single root-mean-square number.

The two common variants are leave-one-out (LOO), where each individual sample is held out in turn, and k-fold, where the samples are partitioned into k groups (typically 5, 7, or 10). LOO is asymptotically optimistic for small sets and computationally cheap. K-fold is generally the more honest estimator for process work, especially when combined with grouped or structured folds that keep replicates of the same batch, day, or reactor together in the same fold. Grouped cross-validation prevents the model from being scored on “new” samples that are actually near-duplicates of training samples - a very common source of an artificially low RMSECV.

RMSECV is the number that should drive the choice of latent variables. Plot RMSECV against number of components; pick the model where RMSECV stops falling meaningfully. This is the standard PLS model-selection routine described in ASTM E1655 and in every chemometrics textbook since Martens and Naes.

RMSECV is still an internal estimate. It uses only the calibration set. If the calibration set does not span the process’s real variability - because it was collected over a narrow production window, in one season, on one raw-material lot, with one instrument - then RMSECV will be optimistic in ways that only production samples can reveal.

RMSEP: the test-set residual

RMSEP is the root-mean-square error of prediction. It is computed on an independent set: samples that were held out from calibration entirely, or new samples acquired later. It is the closest thing to the number an operator will experience at the plant.

The important word is independent. A test set that was drawn from the same day, same batch, same instrument, same operator as the calibration set is not independent - it is a second calibration set with a different label. The correct test set is drawn from a distribution the model should generalise to: later batches, different lots, an instrument that has been recalibrated, a season with different ambient conditions.

For process work, the most useful RMSEP is one computed over a rolling window on production samples once the model is deployed - a live monitoring figure, updated as fresh reference values come back from the lab. That number is what belongs on a control chart, not the RMSEP from the initial validation exercise.

RMSEP is often decomposed into two parts: SEP, the standard error of prediction (the scatter around the fitted line), and bias, the mean signed residual. A model can have a small SEP and a large bias, which is a systematic offset that a plant can usually correct with a small slope-and-intercept adjustment. Or a large SEP and no bias, which is genuine imprecision and cannot be adjusted away.

The gap between the three

The single most diagnostic quantity in a model report is not any of the three metrics on their own; it is the gap between them.

  • RMSEC and RMSECV close together: the model is not overfitting on the calibration set. This is a necessary but not sufficient sign of a healthy model.
  • RMSECV much larger than RMSEC: the model is overfitting. Reduce components, or reconsider preprocessing.
  • RMSEP close to RMSECV: the test set is representative, and the model likely generalises. This is what a plant wants to see.
  • RMSEP much larger than RMSECV: the test set contains variance the calibration set did not span. Either the model needs more calibration data or the test set is outside the model’s applicability. Either way, do not deploy yet.
  • RMSEP much smaller than RMSECV: the test set is not truly independent. Rerun with a properly held-out set.

Reading the three in isolation is how a vendor brief that reports only RMSEC (“RMSEC = 0.12 wt% for a 0-20 wt% range - excellent”) slips a badly overfit model past a busy reviewer. Reading them together makes the story hard to hide.

What the metrics do not tell you

None of RMSEC, RMSECV, or RMSEP address whether the samples in the test set were drawn from the same distribution as the samples the model will encounter in production. A model can pass all three with flying colours on a well-designed offline validation and still fail in production because the process shifted, a raw-material supplier changed, or the probe fouled. That is the job of ongoing outlier detection (Q residuals, Hotelling’s T-squared) and of periodic reference-lab comparison - the monitoring loop that keeps a deployed chemometric model honest.

Nor do these metrics say anything directly about the model’s behaviour on a second instrument. Moving a model from the instrument it was built on to another, even an identical unit, introduces new sources of variance that only an explicit calibration-transfer step can address. A model with a small RMSEP on the source instrument can have a much larger RMSEP on the target until it is transferred.

Finally, none of them replaces the reference method. Every chemometric error metric is a comparison against a reference value that itself has uncertainty. A model whose RMSEP is at or below the reference method’s precision is doing as well as it can. Chasing lower numbers past that point usually means the model is memorising reference-lab noise, which will look like accuracy on paper and like drift in production.

The short version for a review meeting

If you have five minutes to read a chemometric model report, look for four things.

  • RMSEC on its own is unhelpful; ignore any brief that reports only RMSEC.
  • RMSECV should be reported with the cross-validation scheme (LOO or k-fold, grouped or not). Ungrouped LOO on production data is a red flag.
  • RMSEP should be reported on a genuinely independent test set, with the sampling window disclosed. “Held out at random from the calibration set” is not independent.
  • The three numbers should be reported together. Their relative sizes are what tell you whether the model has been honestly evaluated.

None of this is new. ASTM E1655 has said most of it since the 1990s, and USP chapter 1039 codified it for the pharmaceutical context. What changes is how easy it is to skip in a fast vendor cycle. The metrics reward the diligent reader; they reveal the corner-cutting model.