The development of machine learning components is different than traditional software development because we expect that much of the code used during experimentation to be thrown away when the experiments failed.
This involves writing code that will be never used again, needless to say it will never reach production.
However, in order to enable fast experimentation and parallel deployment, the code must still be developed and managed wisely. Not using proper encapsulation or dependency management can significantly impact experiment parallelisation, and slow down development.
In order to ease parallelisation, make sure to:
- think of and prepare the infrastructure for parallel processing and deployment,
- encapsulate code,
- avoid hidden dependencies
and use virtualization, e.g. Docker.