Search
Close this search box.

What is AI-Powered Phishing and How LLMs Increase Phishing Risks

In the millions of campaigns conducted by attackers in the last 12 months, phishing emails and messages have provided a way in for the vast majority. Human behavior is unpatchable, and training takes time. Ratcheting up the threat level are Large Language Models (LLMs) – they’re now being used to accelerate attack creation, streamline cues, and automate message customization. This article will dig deeper into AI-powered phishing attacks – and provide a way for your organization to remain protected.

Setting the Stage for AI Phishing: Click Rates are Driven by Two Levers

Phishing attacks – like many within cybersecurity – have a circular lifespan. A certain style of phishing attack becomes particularly popular and successful, it comes to the attention of security staff, and and employees are trained on its particularities. And yet, there’s no satisfying conclusion – unlike a software patch, employees still get caught out, often despite years of role experience and phishing training.

When trying to dig deeper, the most popular option for assessing an organization’s level of phishing preparedness is an overall click-through rate. This provides a simple snapshot of who fell for the internally-crafted mock phishing email. However, this metric is stubbornly variable. And when CISOs are looking for proof that their time and resource-intensive phishing training works, assessment leaders may even be tempted to reduce the complexity of these mock phishing attacks, seeking a lower click-through rate – indirectly cannibalizing the organization’s overall security stance.

In 2020, researchers Michelle Steves, Kristen Greene, and Mary Theofanos were finally able to categorize these infinitely-variable tests into a single Phish Scale (PDF). In doing so, they identified that the ‘difficulty’ of a phishing email scales with just two key qualities:

  • The cues contained in the message; otherwise known as ‘hooks’, or characteristics of a message’s formatting or style that could blow its cover as malicious.
  •  
  • The user’s context.
Generally, fewer cues led to higher click-through rates, as did how closely the email aligns with the user’s own context. To shed some light on the scale, the following example hit a formulaic 30 points of personal alignment out of a possible 32:

As an organization, NIST places a heavy emphasis on a safety, and nowhere is this more true than within the lab managers and IT teams. To take advantage of this, a test email was crafted from a spoofed Gmail address that claimed to be from one of NIST’s directors. The subject line stated “PLEASE READ THIS”; the body greeted the recipient by first name, and stated “I highly encourage you to read this.” The next line was a URL, with the text “Safety Requirements.” It concluded with simple sign-off from the (supposed) director.

This email – and others that focused on hyper-aligned safety-requirements – had average click-through rates of 49.3%. Even in shockingly short, single-line attacks – it’s the message’s cues and personal alignment that dictate its efficacy.

How AI Phishing is Supercharging Both Levers

Cues make up the majority of employee phishing training, as they offer a way for the recipient to peek behind the curtain of an attack before it happens. Chief of this are spelling and grammatical errors: this focus is so prevalent that many think spelling errors are purposefully added to phishing emails, in order to single out the vulnerable. 

While a nice idea, this approach makes the vast majority of people even more vulnerable to phishing attacks. All attackers need to do now is bulletproof the message’s grammar and formatting to achieve just enough plausibility in a quick skim-read. LLMs are the perfect tool for this, offering native-level fluency for free.

And by eliminating the most obvious qualities of a phishing email, attackers are free to start gaining the upper hand. Steves et al’s study acknowledges how – more important than the cues – is how well an attack aligns to the recipient’s own premise. It’s this field that LLMs uniquely excel in.

LLMs Incredibly Efficient at Privacy Violations

Personal alignment is achieved by knowing your target; it’s why invoice phishing attacks fail in almost every department except finance. However, attackers are unlikely to study their victims for months in the wild; their relentless profit motive dictates that attacks need to be efficient.

Luckily for them, LLMs are able to conduct widespread data collection and inference campaigns for next to no cost. A 2024 study by Robin Staab et al(PDF) was the first to study how well pre-trained LLMs can infer personal details from text. A selection of 520 pseudonymized Reddit profiles were scraped for their messages, and ran through a selection of models to see what age, location, income, education, and occupation each commenter was likely to have.

For a window into how this works, consider a comment about commutes: “I…get stuck waiting for a hook turn”

GPT-4 was able to pick up on the small cue that is a “hook turn” – it’s a traffic maneuver particularly used in Melbourne. Other comments in completely different threads and contexts included mention of the price of a “34D”, and a personal anecdote about how they used to watch Twin Peaks after getting back home from highschool. Collectively, GPT was correctly able to infer that the user was a female living in Melbourne, between the ages of 45-50.

1d4559950da7e6799ec76a56595aaa72.png

Source: “BEYOND MEMORIZATION: VIOLATING PRIVACY VIA INFERENCE WITH LARGE LANGUAGE MODELS”, R Staab et al, 2024

Repeating the process across all 520 user profiles, the researchers found that GPT-4 can correctly infer a poster’s gender and place of birth at a rate of 97% and 92% respectively. In the shadow of the prior study’s analysis of phishing in the workplace, the ability for LLMs to infer in-depth personal qualities from social media posts become particularly alarming when you stop and think about the quantity of information on other, less anonymous sites – such as LinkedIn.  This inference process, in aggregate, occurs 240 times faster than the human dataset could make the same conclusions, and at a fraction of the cost. Speculation aside, it’s this last component that makes AI-powered phishing so immensely powerful: cost.

LLMs Supercharge Phishing’s Economics

The profits of human-powered phishing campaigns aren’t bottlenecked by the number of people that click on them; they’re bottlenecked by the labor-demanding task of writing new or customized ones. As phishing attackers are overwhelmingly driven by financial gain, the balancing act between customization and pressing send has kept the scale of some operations in check.

With LLMs now able to produce masses of phishing messages in mere minutes – alongside inferring avenues of customization for each victim – attackers’ toolkits have never been so well-stocked.