What Is AI-Powered Phishing and How LLMs Increase Phishing Risks
- Key Takeaways:
-
What is AI-powered phishing?
It’s a cyberattack method where generative AI tools are used to craft hyper-realistic and personalized phishing emails. -
How does AI improve phishing effectiveness?
By generating grammatically correct, contextualized, and convincing messages at scale. -
Why are AI-generated attacks harder to detect?
They evade traditional pattern-matching techniques by varying structure, tone, and vocabulary. -
What are the potential risks to organizations?
Increased click-through rates, credential compromise, and lateral movement from a single breach. -
What detection strategies are effective against AI phishing?
Behavior-based analytics, cross-channel correlation, and user activity monitoring. -
How does Stellar Cyber help detect AI-powered phishing?
By correlating phishing indicators across email, endpoint, and network layers within its Open XDR platform.
How AI and Machine Learning Improve Enterprise Cybersecurity
Connecting all of the Dots in a Complex Threat Landscape
Experience AI-Powered Security in Action!
Discover Stellar Cyber's cutting-edge AI for instant threat detection and response. Schedule your demo today!
Setting the Stage for AI Phishing: Click Rates are Driven by Two Levers
Phishing attacks – like many within cybersecurity – have a circular lifespan. A certain style of phishing attack becomes particularly popular and successful, it comes to the attention of security staff, and and employees are trained on its particularities. And yet, there’s no satisfying conclusion – unlike a software patch, employees still get caught out, often despite years of role experience and phishing training.
When trying to dig deeper, the most popular option for assessing an organization’s level of phishing preparedness is an overall click-through rate. This provides a simple snapshot of who fell for the internally-crafted mock phishing email. However, this metric is stubbornly variable. And when CISOs are looking for proof that their time and resource-intensive phishing training works, assessment leaders may even be tempted to reduce the complexity of these mock phishing attacks, seeking a lower click-through rate – indirectly cannibalizing the organization’s overall security stance.
In 2020, researchers Michelle Steves, Kristen Greene, and Mary Theofanos were finally able to categorize these infinitely-variable tests into a single Phish Scale (PDF). In doing so, they identified that the ‘difficulty’ of a phishing email scales with just two key qualities:
- The cues contained in the message; otherwise known as ‘hooks’, or characteristics of a message’s formatting or style that could blow its cover as malicious.
- The user’s context.
Generally, fewer cues led to higher click-through rates, as did how closely the email aligns with the user’s own context. To shed some light on the scale, the following example hit a formulaic 30 points of personal alignment out of a possible 32:
As an organization, NIST places a heavy emphasis on a safety, and nowhere is this more true than within the lab managers and IT teams. To take advantage of this, a test email was crafted from a spoofed Gmail address that claimed to be from one of NIST’s directors. The subject line stated “PLEASE READ THIS”; the body greeted the recipient by first name, and stated “I highly encourage you to read this.” The next line was a URL, with the text “Safety Requirements.” It concluded with simple sign-off from the (supposed) director.
This email – and others that focused on hyper-aligned safety-requirements – had average click-through rates of 49.3%. Even in shockingly short, single-line attacks – it’s the message’s cues and personal alignment that dictate its efficacy.
How AI Phishing is Supercharging Both Levers
Cues make up the majority of employee phishing training, as they offer a way for the recipient to peek behind the curtain of an attack before it happens. Chief of this are spelling and grammatical errors: this focus is so prevalent that many think spelling errors are purposefully added to phishing emails, in order to single out the vulnerable.
While a nice idea, this approach makes the vast majority of people even more vulnerable to phishing attacks. All attackers need to do now is bulletproof the message’s grammar and formatting to achieve just enough plausibility in a quick skim-read. LLMs are the perfect tool for this, offering native-level fluency for free.
And by eliminating the most obvious qualities of a phishing email, attackers are free to start gaining the upper hand. Steves et al’s study acknowledges how – more important than the cues – is how well an attack aligns to the recipient’s own premise. It’s this field that LLMs uniquely excel in.
LLMs Incredibly Efficient at Privacy Violations
Personal alignment is achieved by knowing your target; it’s why invoice phishing attacks fail in almost every department except finance. However, attackers are unlikely to study their victims for months in the wild; their relentless profit motive dictates that attacks need to be efficient.
Luckily for them, LLMs are able to conduct widespread data collection and inference campaigns for next to no cost. A 2024 study by Robin Staab et al (PDF) was the first to study how well pre-trained LLMs can infer personal details from text. A selection of 520 pseudonymized Reddit profiles were scraped for their messages, and ran through a selection of models to see what age, location, income, education, and occupation each commenter was likely to have.
For a window into how this works, consider a comment about commutes: “I…get stuck waiting for a hook turn”
GPT-4 was able to pick up on the small cue that is a “hook turn” – it’s a traffic maneuver particularly used in Melbourne. Other comments in completely different threads and contexts included mention of the price of a “34D”, and a personal anecdote about how they used to watch Twin Peaks after getting back home from highschool. Collectively, GPT was correctly able to infer that the user was a female living in Melbourne, between the ages of 45-50.
Repeating the process across all 520 user profiles, the researchers found that GPT-4 can correctly infer a poster’s gender and place of birth at a rate of 97% and 92% respectively. In the shadow of the prior study’s analysis of phishing in the workplace, the ability for LLMs to infer in-depth personal qualities from social media posts become particularly alarming when you stop and think about the quantity of information on other, less anonymous sites – such as LinkedIn.
This inference process, in aggregate, occurs 240 times faster than the human dataset could make the same conclusions, and at a fraction of the cost. Speculation aside, it’s this last component that makes AI-powered phishing so immensely powerful: cost.
LLMs Supercharge Phishing’s Economics
The profits of human-powered phishing campaigns aren’t bottlenecked by the number of people that click on them; they’re bottlenecked by the labor-demanding task of writing new or customized ones. As phishing attackers are overwhelmingly driven by financial gain, the balancing act between customization and pressing send has kept the scale of some operations in check.
With LLMs now able to produce masses of phishing messages in mere minutes – alongside inferring avenues of customization for each victim – attackers’ toolkits have never been so well-stocked.
Keep Pace with Stellar Cyber
Employee training takes time – and the pace that phishing is evolving threatens to leave thousands of businesses at risk. To handle this elevated threat level, Stellar Cyber offers integrated network and endpoint defenses that keep attackers out, even if they worm their way past an employee.
Endpoint monitoring allows for real-time insight into potential malware deployment, while network protection allows you to see and stop an attacker from establishing a foothold there. User and Entity Behavior Analytics (UEBA) allows you to assess every action in the context of what’s normal, further helping you spot signs of potential account compromise. Protect your team and keep attackers out with Stellar Cyber’s open XDR.
