Capture the Training Objective in a Metric that is Easy to Measure and Understand
Intent
Ensure the machine learning objective is easy to measure and it is a good proxy for the true objective.
Motivation
Many times the true objective is hard to capture in a metric, and may lead to entangled measurements. Choosing a simple, observable metric as a proxy simplifies things, leads to better interpretability, and enhances communication within the team.
Applicability
All training objectives should be captured in an easy to comprehend metric.
Description
Choosing an objective to optimize is not trivial because (1) the objective may be hard to capture in a metric, and (2) the objective evolves over time.
In both cases, over-engineering a metric may lead to entangled measurements, which are hard to comprehend or assess.
Simple metrics, that are easy to measure and comprehend are considered better proxies for the true objective of a machine learning application. Working together with business or data analysts to ensure the metrics reflect business values helps to align the measurements with the true objective.
A great example can be found in the 13th rule for machine learning by Martin Zinchevich.
Adoption
Related
Read more
- Team Data Science Process Documentation
- How do teams work together on automated ML projects
- Operational Machine Learning
- Rules of Machine Learning: Best Practices for ML Engineering