Executive Summary
Machine learning has moved from a security buzzword to a working component of the modern defensive stack. But its usefulness is uneven: in some areas ML is mature and reliable, in others it is promising but unproven, and in a few it is oversold. Knowing the difference is what separates effective adoption from wasted budget.
This whitepaper is a practical, application-by-application survey of where AI and ML genuinely work in cybersecurity today. Rather than treating AI as one monolithic thing, it examines specific use cases — what technique is used, how well it works, and where the limits are. It is grounded in real deployed capabilities and referenced against MITRE ATT&CK, the NIST AI RMF, and MITRE ATLAS.
AI in security is not one capability but dozens of distinct applications at very different levels of maturity. The practical question is never "should we use AI?" but "which specific application solves a problem we actually have?"
Key findings of this paper:
- Anomaly and behavioural detection is the most mature and valuable ML application in security today.
- Malware classification and phishing detection are proven, widely deployed ML use cases.
- Generative AI and LLMs add real value in investigation, summarization, and triage — with important caveats.
- Every ML application requires quality data, tuning, and human validation; none is a fire-and-forget solution.
How Machine Learning Fits Security Problems
Before surveying applications, it helps to understand why certain security problems suit machine learning and others do not.
What ML is good at
Machine learning excels at problems involving pattern recognition in large datasets, especially where the patterns are too numerous or subtle for humans to specify as rules. Security is full of these: distinguishing normal from anomalous behaviour, classifying files as malicious or benign, and spotting the linguistic signatures of phishing.
Supervised versus unsupervised
- Supervised learning trains on labelled examples — known-malicious and known-benign — to classify new samples. It works well where good labelled data exists, as in malware and spam classification.
- Unsupervised learning finds structure and anomalies without labels, making it suited to detecting novel behaviour that no one has seen before.
Where ML struggles
ML struggles where data is scarce or unrepresentative, where being wrong is very costly, and where adversaries actively adapt to evade it. Security has all three challenges, which is why ML augments rather than replaces human analysts and rule-based controls.
The reliable pattern across every successful security ML deployment is the same: ML narrows a vast problem to a manageable set of things worth a human's attention. It rarely makes the final call alone.
Anomaly and Behavioural Detection
This is the most mature and consistently valuable application of ML in security, and for good reason: it plays directly to ML's strengths.
User and entity behaviour analytics
UEBA systems learn the normal behaviour of users, devices, and service accounts, then flag meaningful deviations — a user accessing systems they never touch, a service account behaving like a human, logins from impossible locations. This catches threats that rule-based detection misses: compromised accounts, insider misuse, and novel attack patterns.
Network anomaly detection
ML models baseline normal network traffic and flag deviations that may indicate command-and-control, lateral movement, or data exfiltration. This is especially valuable for detecting activity from devices that cannot run an endpoint agent, including IoT and operational technology.
Why behavioural detection works
Unlike signature-based detection, which can only catch known threats, behavioural detection can flag never-before-seen activity because it defines the anomaly relative to normal rather than to a blacklist. This maps directly onto detecting the later stages of an attack in the MITRE ATT&CK framework — lateral movement and exfiltration — where behaviour betrays the adversary.
The tuning challenge
The cost of behavioural detection is false positives. A model with a poorly defined baseline floods analysts with benign anomalies. Effective deployment requires environment-specific tuning and continuous refinement — which is exactly what GuardsArm's threat detection service provides.
Malware and Phishing Classification
Two of the oldest and most successful applications of ML in security are now deeply embedded in commercial products, often invisibly.
Malware detection
Traditional antivirus relied on signatures — hashes of known-bad files — which cannot catch new or modified malware. ML-based detection classifies files by their features and behaviour rather than exact signatures, allowing it to flag previously unseen malware and variants. This is why modern endpoint protection combines signatures with ML-driven analysis.
Phishing and spam detection
Email security has used ML for years to classify spam and phishing, analyzing content, sender reputation, links, and structural features. As attackers use generative AI to remove the obvious tells, ML-based detection increasingly focuses on behavioural and contextual signals — who normally emails whom, and whether a message fits established patterns.
The adversarial dynamic
Both applications operate against adaptive adversaries who deliberately craft samples to evade the models — an evasion attack in MITRE ATLAS terms. This makes them a continuous arms race rather than a solved problem:
- Models must be retrained as adversaries adapt.
- Detection works best in layers, combining ML with reputation, sandboxing, and other signals.
- No single model is sufficient; defense in depth still applies.
Malware and phishing classification prove that ML security works in production at scale — while also demonstrating that ML is never finished. The model that worked last year needs retraining against this year's tactics.
Generative AI and LLMs in Security Operations
The newest wave of AI in security centers on large language models, which bring genuinely new capabilities alongside genuinely new risks.
Investigation and summarization
LLMs can read and summarize complex incident data, explain obscure log entries in plain language, and draft investigation reports. For an analyst facing a wall of technical data, an LLM that summarizes what happened and suggests next steps meaningfully accelerates the work.
Alert triage assistance
LLMs can help triage alerts by gathering context, correlating related information, and prioritizing — directly attacking the alert-fatigue problem that overwhelms SOC teams.
Natural-language interfaces
LLMs let analysts query security data conversationally rather than in specialized query languages, lowering the barrier to investigation and making powerful tools accessible to more of the team.
The caveats that matter
- Hallucination: LLMs can produce confident, plausible, incorrect output. In security, an analyst must verify conclusions, not accept them.
- Prompt injection: an LLM processing attacker-controlled data (logs, alerts, emails) can be manipulated through embedded instructions — a live threat catalogued in MITRE ATLAS.
- Data exposure: sending sensitive telemetry to external LLM services raises privacy and confidentiality questions.
LLMs are a powerful assistant to the security analyst, not an autonomous decision-maker. The value is real; so is the need to verify, constrain inputs, and keep a human accountable. GuardsArm designs LLM-assisted workflows with exactly these guardrails.
Emerging and Specialized Applications
Beyond the well-established use cases, ML is being applied across a widening range of security problems at varying maturity.
Vulnerability management
ML helps prioritize the flood of vulnerabilities by predicting which are most likely to be exploited, letting teams focus remediation where it matters rather than chasing every high CVSS score. Coupled with threat intelligence, this makes patching a risk-driven decision.
Fraud and abuse detection
In financial and online services, ML models detect fraudulent transactions and account abuse in real time, adapting as fraud patterns shift. This is one of the most commercially proven applications of security ML.
Automated response
ML and automation increasingly drive SOAR platforms that execute routine response actions — enriching alerts, containing low-risk threats — freeing analysts for complex work. Autonomy here is deliberately bounded to low-risk, well-understood actions.
Deception and threat hunting
ML helps analyze the data from honeypots and deception environments and supports threat hunters by surfacing subtle patterns worth investigating.
Choosing wisely
- Match the application to a real problem, not to the technology.
- Demand evidence of effectiveness before trusting a use case.
- Ensure data quality and human oversight for every deployment.
- Govern AI systems under the NIST AI RMF and secure them against the threats in MITRE ATLAS.
The practical path is application-by-application: adopt the mature, proven use cases now, pilot the emerging ones with clear success metrics, and govern all of them. GuardsArm helps clients map which ML security applications fit their environment and deploy them with the tuning and oversight they require.
Key Takeaways
- 1.AI in security is dozens of distinct applications at different maturity levels — adopt by specific use case, not by buzzword.
- 2.Anomaly and behavioural detection (UEBA, network analytics) is the most mature, valuable ML application and catches novel threats.
- 3.Malware and phishing classification are proven at scale but require constant retraining against adaptive, evasive adversaries.
- 4.LLMs genuinely accelerate investigation, triage, and summarization — but demand verification, input constraints, and human accountability.
- 5.Every ML application needs quality data, environment-specific tuning, human oversight, and governance under the NIST AI RMF and MITRE ATLAS.
Sources & Further Reading
- MITRE ATT&CK Framework
- MITRE ATLAS (Adversarial Threat Landscape for AI Systems)
- NIST AI Risk Management Framework (AI RMF 1.0)
- NIST SP 800-94, Guide to Intrusion Detection and Prevention Systems
- ENISA Artificial Intelligence and Cybersecurity guidance