Perform Checks to Detect Skew between Models

31 / 45 Deployment This practice was ranked as advanced. Click to read more.


Intent

Avoid introducing errors in production pipelines.

Motivation

Test if a model that performs well during training and initial testing will also perform well in production i.e. test if the training data distribution reflects the production one.

Applicability

Model skew should be monitored in any production-level machine learning application.

Description

In a quickly changing environment or when the training data does not reflect the production distribution, it is not uncommon to have models that perform well during training and initial testing, but not in production. In order to avoid deployment of under-performing or sub-optimal models, it is recommended to continuously check possible skew between the production and training environments.

Make sure to:

  • check performance skew between training and hold-out data,
  • check skew between data generated in previous days,
  • check skew between live data and training.

Adoption

Related

Read more



31 / 45 Deployment This practice was ranked as advanced. Click to read more.