What Is Adversarial Machine Learning?

Adversarial machine learning concerns attacks at inference time: inputs deliberately crafted so a trained model produces the wrong output. The model is not compromised, the infrastructure is not breached, and the system behaves exactly as built — on an input chosen to exploit how it learned.

This is distinct from model poisoning, which attacks training, and from prompt injection, which exploits the instruction channel of a language model. Different phase, different defences.

The attack families

Evasion gets the attention. The bottom three matter more for privacy: a model trained on health or financial records can leak information about individuals in it without ever exposing the underlying data store.

Where this actually bites in security

Security products are themselves models, and they are the most directly targeted case. Malware classifiers, spam filters and anomaly detection are all subject to evasion by attackers who can test against them freely — often using the same commercial product.

Defences, and their honest limits

• Adversarial training — include crafted examples in training. Helps against known attack styles; generalises imperfectly.
• Input validation and preprocessing — raises the effort required without closing the gap.
• Ensembles — harder to fool several dissimilar models at once, at the cost of complexity.
• Rate limiting and query monitoring — directly frustrates extraction and membership inference, and is frequently the cheapest meaningful control.
• Defence in depth — never let one model be the only thing between an input and a consequence.

Testing your own models

Adversarial robustness is testable, and almost nobody tests it because it falls between two teams: the security testers do not own the model and the data scientists are measuring accuracy.

A practical starting programme:

• Generate adversarial examples against your own model with published techniques and measure how far the accuracy falls
• Attempt extraction using only the query interface an external user has, and see how close the reconstruction gets
• Test membership inference if the training data contained personal information, since that is a privacy exposure as well as a security one
• Measure the query budget an attack requires, because that number tells you whether rate limiting is a sufficient defence
• Re-test after retraining, since robustness is not preserved across model versions

The query-budget number is the most actionable output. If evasion requires thousands of probes, monitoring and rate limiting are genuinely protective; if it requires a handful, they are not.

Where to start

Identify any model whose output directly triggers a consequential action with no second check — blocking, approving, flagging or authorising. Those are where evasion has a real-world effect, and they are usually a small list.

See prompt injection in business applications and AI governance. GuardsArm runs security risk assessments.