Perform Checks to Detect Skew between Models
Intent
Avoid introducing errors in production pipelines.
Motivation
Test if a model that performs well during training and initial testing will also perform well in production i.e. test if the training data distribution reflects the production one.
Applicability
Model skew should be monitored in any production-level machine learning application.
Description
In a quickly changing environment or when the training data does not reflect the production distribution, it is not uncommon to have models that perform well during training and initial testing, but not in production. In order to avoid deployment of under-performing or sub-optimal models, it is recommended to continuously check possible skew between the production and training environments.
Make sure to:
- check performance skew between training and hold-out data,
- check skew between data generated in previous days,
- check skew between live data and training.
Adoption
Related
Read more
- Continuous Delivery for Machine Learning
- Rules of Machine Learning: Best Practices for ML Engineering
- Testing and Debugging in Machine Learning
- TFX: A tensorflow-based Production-Scale ML Platform