Assure Application Security

March, 2023 • Alex Serban, Koen van der Blom, Joost Visser

27 / 46 • Coding •

This practice was ranked as advanced.
Click to read more. •

This practice addresses requirements
from the EU guidelines for trustworthy ML.
Click to read more.

Intent

Prevent attackers from stealing or corrupting data, or from disrupting the availability of an application.

Motivation

Security incidents can lead to public data leaks, financial losses, or disrupt the availability of an application.

Applicability

Security is important for any application with an external interface or which processes personal or sensitive data.

Description

For any application that exposes an external interface or uses personal or sensitive data, it is imperative to reflect and take actions to improve its security. Security is known as an arms race, with attackers constantly improving their techniques, and defenders updating their systems in order to predict and prevent new threats. Therefore, ensuring security is a continuous task.

Besides classical cyber threats that apply to software systems, machine learning adds new security risks both during training and deployment. Training time attacks are known as data poisoning, and consist of attackers trying to alter the training data in order to induce malicious behaviour – such as misclassifying certain examples.

Test time (or inference) attacks are more diverse, and consist of adding small perturbations to test data in order to induce malicious behaviour (adversarial attacks), reverse engineering the model or checking if some data was used for training (membership attacks). Like other branches of machine learning, security is also a growing field of study.

As mentioned earlier, security requires a proactive approach, with some mechanisms including security code reviews, using security analysis tools, penetration testing, and actively performing red teaming exercises.

Adoption

Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims