What Is Model Poisoning?

Poisoning attacks the model before it is ever deployed. Where adversarial machine learning manipulates inputs at inference, poisoning corrupts what the model learns — through the training data, the training process, or the artefact itself.

That timing is what makes it dangerous. A poisoned model behaves correctly on ordinary inputs and passes ordinary evaluation. The flaw only surfaces when the attacker supplies the trigger they chose.

The variants worth distinguishing

Backdoors are the hardest to catch. Accuracy on the test set stays high, behaviour looks normal, and the model misbehaves only on inputs carrying the trigger — which no standard evaluation contains.

Where the exposure comes from in practice

Few organisations train large models from scratch, which can feel like an exemption. It is not — it relocates the risk into the supply chain.

Systems that retrain automatically on live data are the clearest case. If production input reaches training without curation, anyone who can use the product can contribute to what it becomes.

Defences

• Provenance for every model artefact — where it came from, verified checksums, and a recorded decision to trust it
• Curate training data, with outlier and duplicate detection before it enters the set
• Hold out a trusted evaluation set the training pipeline never sees, so a skew is visible
• Test for triggers, not just for accuracy — standard metrics are specifically what a backdoor is designed to survive
• Control the retraining pipeline with the same rigour as production deployment, because that is what it is
• Version and be able to roll back, since the fastest response to a suspect model is returning to a known one

Detecting a model you already suspect

Backdoors are designed to survive ordinary evaluation, so the usual metrics will not surface one. What sometimes does:

• Behavioural differencing against a known-clean model on the same inputs, looking for disagreement clusters
• Trigger search, systematically probing for input patterns that produce anomalous confidence
• Activation analysis, since backdoor triggers frequently produce distinctive internal activation patterns
• Data provenance review, working backwards through what entered the training set and when
• Retraining from curated data as the decisive test, because if the behaviour disappears the data was the source

The last option is the most reliable and the most expensive, which is the honest position: detection is hard, and prevention through pipeline control and artefact provenance is considerably cheaper than forensics on a suspect model.

Where to start

Ask where your models came from and whether anyone verified the artefact. For most organisations the honest answer is that someone pulled a model from a public hub and it worked — which is an unverified dependency with more authority over outcomes than most libraries you would review carefully.

See AI vendor risk and AI governance. GuardsArm runs security risk assessments covering AI pipelines.