In a quickly changing environment or when the training data does not reflect the production distribution, it is not uncommon to have models that perform well during training and initial testing, but not in production. In order to avoid deployment of under-performing or sub-optimal models, it is recommended to continuously check possible skew between the production and training environments.
Make sure to:
- check performance skew between training and hold-out data,
- check skew between data generated in previous days,
- check skew between live data and training.
- Continuous Delivery for Machine Learning
- Rules of Machine Learning: Best Practices for ML Engineering
- Testing and Debugging in Machine Learning
- TFX: A tensorflow-based Production-Scale ML Platform