For long running experiments – such as training deep neural networks for object recognition or language tasks – it is important to detect errors as early as possible in the training process. Moreover, it is important to continuously check model quality and performance on different benchmarks which match the production environment as close as possible.
By continuously monitoring the model’s quality and performance, one allows errors to be detected and contained early. Moreover, experiments can be stopped early, avoiding futile use of resources.
Ultimately, continuous monitoring enhances experimentation agility. It also helps to keep an experiment “log-book” and keep track of past experiments and hyper-parameter configurations.
Monitoring should also include regular snapshots of the model in order to return to different versions of the model and facilitate retraining.
- Share Status and Outcomes of Experiments Within the Team
- Continuously Monitor the Behaviour of Deployed Models
- Rules of Machine Learning: Best Practices for ML Engineering
- Testing and Debugging in Machine Learning