When phishing awareness starts learning from real attacks

Learn how organizations can use LLMs trained on real phishing emails to improve awareness programs.

Most organizations receive a large number of spam and phishing emails every day. Email security gateways filter these messages, label them, and block many of them before they reach users. After that, the emails usually sit in logs or archives and are rarely used again.

At the same time, phishing awareness programs continue to rely on simulated emails that are often generic. These simulations are usually based on common templates and broad attack patterns. They are not always tied to what the organization is actually seeing in the real world.

Follow my journey of 100 Days of Red Team on WhatsApp, Telegram or Discord.

This creates a gap. Real attackers keep changing their techniques, while awareness training often stays the same.

Turning email filters into a training pipeline

Now consider a different approach.

Email filtering tools already see real phishing emails every day. They collect the full content of the message, along with metadata such as sender domains, subjects, themes, and timing. Over time, this becomes a large and realistic dataset of attacker behavior.

An organization could take this data, clean it, remove sensitive information, and use it to train an internal large language model (LLM). That model could then be connected to the phishing awareness platform.

Instead of using static templates, the awareness tool could generate emails that closely resemble the phishing emails the organization actually receives.

From a defender’s point of view, this approach makes sense.

It uses data the organization already has. It reduces the gap between real attacks and training exercises. It produces phishing simulations that match the organization’s industry, vendors, and communication style.

Users are no longer trained on generic examples. They are trained on emails that look similar to what attackers are really sending.

For security teams looking to improve awareness metrics and make training more relevant, this idea is likely to be appealing. As AI adoption grows, it is reasonable to expect that some organizations will try this.

At this point, the threat model changes.

Why this LLM will need to be red teamed?

If an organization uses an LLM trained on real phishing emails to drive awareness training, that model becomes part of the security system. It influences how users learn, what they recognize as suspicious, and how they respond to future emails. Any system that shapes user behavior in this way needs to be tested from an attacker’s perspective before it is trusted.

From a red team point of view, this is not about challenging the idea itself. It is about validating how the model behaves when exposed to adversarial input and understanding the assumptions it makes. At a minimum, red teams should focus on the following areas:

• Training data influence and drift - Test how repeated phishing patterns affect the model’s output, since attackers already influence part of the training data through volume.

• Coverage of rare and high-context attacks - Test whether the model can simulate low-frequency, targeted phishing, not just common and well-represented patterns.

• Bias from automated email classification - Test how upstream filtering and labeling decisions shape what the model learns, as the LLM only sees what other systems classify as phishing.

• Exposure of organizational signals in generated content - Test whether generated awareness emails reflect internal language, vendors, or workflows that could be inferred over time.

Red teaming this type of system is not about blocking innovation. It ensures that the model behaves as expected when it is exposed to real attacker behavior.

TL;DR
- Organizations can use LLMs trained on real phishing emails to generate more realistic awareness campaigns.
- Red teaming is needed to ensure the model behaves as expected under adversarial conditions.
- Key areas to test:
  - Training data influence and drift
  - Coverage of rare and targeted phishing attacks
  - Bias from automated email classification
  - Exposure of internal organizational signals

Follow my journey of 100 Days of Red Team on WhatsApp, Telegram or Discord.