Choosing an objective to optimize is not trivial because (1) the objective may be hard to capture in a metric, and (2) the objective evolves over time.
In both cases, over-engineering a metric may lead to entangled measurements, which are hard to comprehend or assess.
Simple metrics, that are easy to measure and comprehend are considered better proxies for the true objective of a machine learning application. Working together with business or data analysts to ensure the metrics reflect business values helps to align the measurements with the true objective.
A great example can be found in the 13th rule for machine learning by Martin Zinchevich.
- Team Data Science Process Documentation
- How do teams work together on automated ML projects
- Operational Machine Learning
- Rules of Machine Learning: Best Practices for ML Engineering