Enable Parallel Training Experiments

16 / 46 Training This practice was ranked as basic.
Click to read more.
This practice helps to increase
the team's agility.
Click to read more.


Intent

Avoid deadlocks during experimentation.

Motivation

Machine learning relies heavily on empirical processes. In order to allow fast experimentation and avoid deadlocks, it is recommended to think upfront of experiment parallelisation.

Applicability

Parallelisation should be considered in any machine learning application.

Description

The development of machine learning components is different than traditional software development because we expect that much of the code used during experimentation to be thrown away when the experiments failed.

This involves writing code that will be never used again, needless to say it will never reach production.

However, in order to enable fast experimentation and parallel deployment, the code must still be developed and managed wisely. Not using proper encapsulation or dependency management can significantly impact experiment parallelisation, and slow down development.

In order to ease parallelisation, make sure to:

  • think of and prepare the infrastructure for parallel processing and deployment,
  • encapsulate code,
  • avoid hidden dependencies

and use virtualization, e.g. Docker.

Adoption

Related

Read more



16 / 46 Training This practice was ranked as basic.
Click to read more.
This practice helps to increase
the team's agility.
Click to read more.