Continuously Measure Model Quality and Performance

March, 2023 • Alex Serban, Koen van der Blom, Joost Visser

20 / 46 • Training •

This practice was ranked as basic.
Click to read more.

Intent

Detect errors early and improve experimentation agility.

Applicability

Quality monitoring should be used in any training experiment.

Description

For long running experiments – such as training deep neural networks for object recognition or language tasks – it is important to detect errors as early as possible in the training process. Moreover, it is important to continuously check model quality and performance on different benchmarks which match the production environment as close as possible.

By continuously monitoring the model’s quality and performance, one allows errors to be detected and contained early. Moreover, experiments can be stopped early, avoiding futile use of resources.

Ultimately, continuous monitoring enhances experimentation agility. It also helps to keep an experiment “log-book” and keep track of past experiments and hyper-parameter configurations.

Monitoring should also include regular snapshots of the model in order to return to different versions of the model and facilitate retraining.

Adoption