Perform Checks to Detect Skew between Models

31 / 46 Deployment This practice was ranked as medium.
Click to read more.


Intent

Avoid introducing errors in production pipelines.

Motivation

Test if a model that performs well during training and initial testing will also perform well in production i.e. test if the training data distribution reflects the production one.

Applicability

Model skew should be monitored in any production-level machine learning application.

Description

In a quickly changing environment or when the training data does not reflect the production distribution, it is not uncommon to have models that perform well during training and initial testing, but not in production. In order to avoid deployment of under-performing or sub-optimal models, it is recommended to continuously check possible skew between the production and training environments.

Make sure to:

  • check performance skew between training and hold-out data,
  • check skew between data generated in previous days,
  • check skew between live data and training.

Adoption

3-12 months 1-2 years 2-5 years More than 5 years 020406080100
Adoption by team experiencePercentage of answers
1 member 2-3 members 4-5 members 6-9 members 10-15 members 16 or more members020406080100
Adoption by team sizePercentage of answers
Tabular dataTextImages/VideosAudioTime seriesGraphs020406080100
Adoption by data typePercentage of answers
TechcompanyNon-techcompanyGovernmentalOrganisationResearch020406080100
Adoption by org. typePercentage of answers

Related

Read more



31 / 46 Deployment This practice was ranked as medium.
Click to read more.