Continuously Monitor the Behaviour of Deployed Models
30 / 46 •
Deployment •
This practice was ranked as medium.
Click to read more. •
This practice helps to increase
the team's agility.
Click to read more. •
This practice helps to increase
the traceability of ML components.
Click to read more.
This practice was ranked as medium. Click to read more. •
This practice helps to increasethe team's agility.
Click to read more. •
This practice helps to increasethe traceability of ML components.
Click to read more.
Intent
Avoid unintended behaviour in production models.
Motivation
Once a model is promoted to production, the team has to understand how it performs.
Applicability
Monitoring should be implemented in any production-level ML application.
Description
Monitoring plays an important role in production level machine learning. Because the performance between training and production data can vary drastically, it is important to continuously monitor the behaviour of deployed models and raise alerts when unintended behaviour is observed.
The monitoring pipeline should include:
- performance, quality and skew metrics,
- fairness metrics,
- model interpretability outputs (e.g. LIME),
- metrics for the perceived effect of the model, e.g. user interactions, conversion rates, etc.
Adoption
Related
- Perform Checks to Detect Skew between Models
- Enable Automatic Roll Backs for Production Models
- Continuously Measure Model Quality and Performance
Read more
- Continuous Delivery for Machine Learning
- Machine Learning Logistics
- Machine learning: Moving from experiments to production
- Testing and Debugging in Machine Learning
- TFX: A tensorflow-based Production-Scale ML Platform
30 / 46 •
Deployment •
This practice was ranked as medium.
Click to read more. •
This practice helps to increase
the team's agility.
Click to read more. •
This practice helps to increase
the traceability of ML components.
Click to read more.
This practice was ranked as medium. Click to read more. •
This practice helps to increasethe team's agility.
Click to read more. •
This practice helps to increasethe traceability of ML components.
Click to read more.