Microsoft blocks phishing scam which used AI-generated code to trick users

• AI generated code used in phishing campaign, blocked by Microsoft Defender
• Attackers used SVG file disguised as PDF, with hidden business themed code inside
• Security Copilot flagged AI style traits, like verbose identifiers and generic comments

AI code is now used across industries for a range of tasks, and in cybersecurity, both security teams and attackers are increasingly turning to large language models to support their work.

Defenders apply AI to detect and respond to threats at scale, while attackers experiment with it to craft phishing lures, generate obfuscated code, and disguise malicious payloads.

Microsoft Threat Intelligence recently detected and blocked a phishing campaign it believed used AI-generated code to hide its payload inside an SVG file.

Polished but not practical

The campaign used a compromised small business email account to send self addressed messages with actual targets hidden in BCC fields, and the attachment was named to resemble a PDF while carrying scriptable SVG content.

The SVG file included hidden elements made to look like a business dashboard, while a script inside it turned business related words into code that revealed a hidden payload.

When opened, the file redirected users to a CAPTCHA gate, a common social engineering tactic that can lead to a fake sign in page intended to harvest credentials.

The obfuscation relied on concatenated business words and formulaic code patterns rather than cryptographic techniques.

Security Copilot analyzed the file and flagged markers consistent with LLM output, such as long descriptive identifiers, repetitive modular structures, generic comments, and an unusual combination of XML declaration and CDATA.

These traits made the code look polished on the surface but not practical, which led analysts to believe it was probably generated by AI.

The researchers used AI powered tools in Microsoft Defender for Office 365 to piece together clues that were harder for attackers to hide.

The system flagged the unusual self-addressed email pattern, the odd SVG file disguised as a PDF, the redirect to a known phishing site, the hidden code inside the file, and the tracking methods used on the phishing page.

The incident was limited, easily blocked, and primarily targeted US organizations, but Microsoft notes that it illustrates how attackers are increasingly experimenting with AI to craft convincing lures and complex payloads.

Via Infosecurity Magazine

You might also like

• These are the best AI writers available now
• Microsoft Copilot targeted in first “zero-click” attack on an AI agent
• AI is creating code faster - but this also means more potential security issues