AI Security
LLM Hallucination
as a Security Vector
Published: 2026-05-17 18:19 PDT (Oregon)
Hallucinated package names get installed. Hallucinated URLs get visited. Hallucinated code gets deployed to production. When AI confidently fabricates, the downstream damage is real — and increasingly weaponizable by attackers who've learned to predict what models hallucinate.
From Annoying to Dangerous
Hallucination has long been framed as a reliability problem: the model is wrong sometimes, users should verify. This framing misses the security dimension. A wrong answer that a user ignores is harmless. A wrong answer that a user acts on — running a suggested command, installing a recommended package, clicking a fabricated URL — converts a reliability failure into a security incident.
The shift from nuisance to threat happened when researchers demonstrated that hallucinations can be predicted and exploited. If an LLM consistently recommends a non-existent package called util-auth-helper when asked about authentication in a specific framework, an attacker can simply publish that package on npm with malicious contents and wait.
Slopsquatting: The Package Hallucination Attack
Researchers at Socket and others have documented "slopsquatting" — publishing packages that match LLM-hallucinated names. The methodology: probe LLMs with common developer queries, collect non-existent package recommendations, register those packages with malicious payloads, then wait for developers who follow the same AI recommendations to install them.
Developer asks LLM: "How do I validate JWT tokens in Express?" — LLM recommends express-jwt-validator (doesn't exist, hallucinated) — attacker has published express-jwt-validator on npm containing a credential-stealing payload — developer runs npm install express-jwt-validator — payload executes at install time via postinstall script.
A 2024 study found that LLMs hallucinate package names in approximately 5.2% of coding queries — and that the same hallucinated names recur consistently across models and sessions, making them predictable targets for pre-registration.
Other Hallucination Attack Surfaces
Hallucinated URLs
LLMs fabricate documentation links, API endpoints, and reference URLs. Attackers register these domains with phishing pages or malware distribution. Users following "official" links from AI responses land on attacker-controlled sites.
Hallucinated CVE Details
LLMs fabricate CVE numbers, affected version ranges, and patch instructions. Security teams acting on hallucinated vulnerability data may deploy unnecessary patches, miss real vulnerabilities, or apply incorrect mitigations.
Hallucinated Code Patterns
Subtly insecure code patterns recommended with confidence — incorrect crypto usage, flawed authentication logic, SQL queries that look parameterized but aren't. The code runs; the vulnerability ships to production.
Hallucinated CLI Flags
LLMs recommend non-existent or dangerous flags for system tools. A hallucinated --force-overwrite flag for a file operation may match a real flag with destructive behavior, or cause the tool to silently fail.
Why Confidence Makes It Worse
The security damage from hallucination is amplified by presentation. LLMs present hallucinated content with the same syntax-highlighted, well-formatted, authoritative tone as accurate content. There is no visual distinction between a correct package recommendation and a hallucinated one. The user interface creates the appearance of reliability where none exists.
This is fundamentally different from a web search result, where the user can visit the source, check publication dates, and assess credibility. An LLM recommendation launders the attacker's package through the model's perceived authority. The developer trusts the AI, not the package registry.
Mitigations
- Always verify packages exist before installing. Check the registry directly — npm, PyPI, crates.io. Package count and download history are meaningful signals. A package with 3 downloads and a creation date of yesterday deserves scrutiny.
- Use LLM output as a starting point, not an instruction. For security-sensitive operations (installs, system commands, config changes), treat AI suggestions as leads to verify, not commands to execute.
- Enable lockfiles and integrity checking.
package-lock.json,poetry.lock, and similar mechanisms ensure reinstalls match the originally resolved packages — they don't prevent initial installation of a malicious package, but they prevent silent upgrades. - AI coding tool providers: surface uncertainty. When recommending packages or URLs, models should retrieve and verify existence in real-time via tool calls rather than generating from memory. Grounded recommendations eliminate the slopsquatting attack surface.
- Security teams: monitor for hallucinated CVEs. Build a process for validating AI-sourced vulnerability information against authoritative databases (NVD, vendor advisories) before acting on it.