Monitoring plays an important role in production level machine learning. Because the performance between training and production data can vary drastically, it is important to continuously monitor the behaviour of deployed models and raise alerts when unintended behaviour is observed.
The monitoring pipeline should include:
- performance, quality and skew metrics,
- fairness metrics,
- model interpretability outputs (e.g. LIME),
- metrics for the perceived effect of the model, e.g. user interactions, conversion rates, etc.
- Perform Checks to Detect Skew between Models
- Enable Automatic Roll Backs for Production Models
- Continuously Measure Model Quality and Performance
- Continuous Delivery for Machine Learning
- Machine Learning Logistics
- Machine learning: Moving from experiments to production
- Testing and Debugging in Machine Learning
- TFX: A tensorflow-based Production-Scale ML Platform