Continuously Monitor the Behaviour of Deployed Models

30 / 46 Deployment This practice was ranked as medium.
Click to read more.
This practice helps to increase
the team's agility.
Click to read more.
This practice helps to increase
the traceability of ML components.
Click to read more.


Intent

Avoid unintended behaviour in production models.

Motivation

Once a model is promoted to production, the team has to understand how it performs.

Applicability

Monitoring should be implemented in any production-level ML application.

Description

Monitoring plays an important role in production level machine learning. Because the performance between training and production data can vary drastically, it is important to continuously monitor the behaviour of deployed models and raise alerts when unintended behaviour is observed.

The monitoring pipeline should include:

  • performance, quality and skew metrics,
  • fairness metrics,
  • model interpretability outputs (e.g. LIME),
  • metrics for the perceived effect of the model, e.g. user interactions, conversion rates, etc.

Adoption

Related

Read more



30 / 46 Deployment This practice was ranked as medium.
Click to read more.
This practice helps to increase
the team's agility.
Click to read more.
This practice helps to increase
the traceability of ML components.
Click to read more.