What Changed
For decades, phishing detection relied on a simple heuristic: real communications from real organizations are professionally written. Phishing attempts, largely produced by non-native English speakers with limited resources, were riddled with grammatical errors, awkward phrasing, and generic greetings like "Dear Valued Customer." Security training emphasized these signals. Users learned to be suspicious of bad writing.
LLMs made this heuristic obsolete overnight. Any LLM available since 2023 writes grammatically perfect English — and perfect French, Spanish, Japanese, and every other language — indistinguishable from native speaker prose. The "bad grammar = phishing" signal is gone, and it will not return. This is not a temporary capability gap that will be closed by better detection — it is a permanent shift in what attackers can produce at zero marginal cost.
The second shift is personalization at scale. Before LLMs, there were two categories of phishing: mass phishing (high volume, zero personalization — "Dear Customer, your account has been suspended") and spear phishing (low volume, high personalization — targeting specific individuals with tailored content). The cost structure meant you could have one or the other, not both. LLMs break this tradeoff. An attacker can now generate thousands of highly personalized messages per hour, each tailored to a specific target using scraped LinkedIn profiles, corporate websites, and social media data.
The Spear Phishing Pipeline
Modern AI-powered spear phishing is not a single tool — it's a pipeline that combines several AI capabilities with traditional data scraping. Understanding the pipeline is necessary for building defenses against it.
Stage 1 — Target profiling: scrape LinkedIn for role, tenure, connections, recent activity; scrape company website for org chart, projects, press releases; scrape personal social media for interests, relationships, communication style. Stage 2 — Context synthesis: feed profile to an LLM with prompt "write a spear phishing email targeting this person that references their specific situation and appears to come from [spoofed colleague/vendor/executive]." Stage 3 — Infrastructure: register a lookalike domain (acme-corp.com → acme-c0rp.com), configure DKIM-spoofing, set up credential harvest page. Stage 4 — Send at scale: the entire pipeline can be automated to generate and send hundreds of highly personalized attacks per day.
From: sarah.chen@acme-c0rp.com (spoofed)
Subject: Re: Q2 budget review — approval needed before EOD
"Hi David,
Following up on the Q2 planning session last Tuesday — I know you had questions about the infrastructure line items. I've updated the shared doc with the breakdown you asked for. Could you review and provide signoff before our 3pm board prep call? Just click the link below to access the finance portal and authenticate.
[credential harvest link]
Thanks,
Sarah"
Note: the name, role reference, timing detail, and context are all synthesized from scraped data. No human wrote this — an LLM generated it in under a second.
Voice Cloning + LLM = Vishing at Scale
The text-based phishing problem is compounded by voice-based attacks. Voice cloning technology can now produce convincing replicas of a target's voice from as little as three seconds of audio — available from public recordings, voicemails, or social media videos. Combine cloned voice with an LLM-driven real-time conversation engine, and you have a fully automated vishing (voice phishing) system that sounds like a colleague, family member, or executive.
The attack flow: the system calls the target using a spoofed caller ID matching a trusted contact. The call is handled by a real-time LLM agent speaking in the cloned voice of that contact, trained on scraped context about the target's relationship with that contact. The agent handles unexpected questions, maintains conversational context, and guides the target toward the attacker's objective — authorizing a wire transfer, revealing credentials, clicking a link.
Several high-profile financial fraud cases have already been publicly attributed to AI voice cloning attacks. The technology is available, the attack pipeline exists, and the cost-per-attack is declining rapidly. Teams that haven't updated their social engineering defenses for the voice cloning threat are operating with out-of-date playbooks.
Detecting AI-Generated Phishing
Detection of AI-generated phishing is significantly harder than detection of traditional phishing. The grammar-based heuristics that most email security tools rely on are useless against LLM-generated content. Several alternative detection approaches show partial promise:
Perplexity-based detection: AI detectors analyze the statistical distribution of token sequences. AI-generated text tends to have lower perplexity than human text — it is "too smooth," too consistently fluent. This signal is real but unreliable at the individual email level. False positive rates are high enough that perplexity-based detection cannot be used as a standalone control for phishing detection without significant human review overhead.
Behavioral signals: AI-generated phishing often has subtle behavioral tells that reflect the attack pipeline rather than genuine human communication. Emails that reference very recent events (scraped from the target's public activity) but contain generic emotional framing; messages that use the correct names and roles but miss subtle cultural/organizational context that only insiders would know; communications that arrive at unusual times for the claimed sender's timezone. These signals require domain expertise to evaluate and are not automatable at scale.
Context inconsistencies: the most reliable AI phishing signal is when the fabricated context doesn't quite match reality. The "Q2 budget meeting" that didn't happen, the "project" that uses an incorrect code name, the "colleague" who was actually out of office that week. Catching these inconsistencies requires the recipient to have ground truth about the claimed context — which is exactly why attackers invest in scraping accurate contextual data before launching attacks.
Defense: Technical Controls
DMARC (Domain-based Message Authentication, Reporting, and Conformance) with a reject policy is the single most effective technical control against email-based phishing. It prevents spoofing of your domain as a sender — attackers cannot send emails that appear to come from yourcompany.com without legitimate credentials. Critically, DMARC only protects your outbound identity; it does not prevent attackers from using lookalike domains or other organizations' email infrastructure. Enforce DMARC on all your domains, and monitor DMARC reports for spoofing attempts against your domain.
Modern email security platforms use LLMs specifically trained for phishing detection — the same technology that makes phishing harder to detect also makes it possible to build better detectors. Look for platforms that do behavioral analysis (does this email match the sender's normal writing style?), relationship analysis (has this sender ever emailed this recipient before?), and URL sandboxing (follow every link in a sandboxed environment and analyze the destination page before delivering the email). No detector catches everything, but layered technical controls significantly raise the cost of a successful attack.
Defense: Organizational Controls
Technical controls cannot catch every phishing attempt. The human layer matters enormously. Establish clear verification protocols for high-risk actions: any wire transfer over a threshold, any credential reset request, any software installation request. The protocol should require out-of-band verification — calling the requester on a known phone number, not replying to the email or calling numbers provided in the email. This should be a formal policy, not a cultural expectation, with defined consequences for bypassing it.
The most underinvested defense is cultural: building an organization where it is normal and expected to verify unusual requests, regardless of who the claimed sender is. AI-powered phishing creates urgency and authority to suppress the natural human instinct to verify. Train employees explicitly for this manipulation pattern: any request that creates urgency, invokes authority, or pressures quick action is a red flag — even if it comes from a known name and arrives in professionally written prose. The goal is not to make employees paranoid but to shift the default from "assume it's real" to "verify before acting."
The uncomfortable truth about AI-powered phishing is that it shifts the security challenge from a technical problem to a human problem at the margins. Technical controls can catch a large fraction of AI-generated phishing. But the most sophisticated attacks — targeted, contextually accurate, leveraging voice cloning and real-time conversation — will get through technical filters. The final line of defense is people who have been trained, repeatedly and specifically, to recognize manipulation attempts even when they are flawlessly executed. That training needs to be updated for the AI threat landscape, not the threat landscape of five years ago.