The AI Is the Phisher: How Prompt Injection Turns Trusted Assistants into Personalized Social Engineers

Traditional phishing begins with an unsolicited message.

An attacker sends an email or text message and hopes that the recipient will open it, trust it, and take the requested action. Most phishing campaigns are therefore a numbers game: send enough messages, and eventually someone will click.

AI assistants introduce a different and potentially more effective model.

Instead of contacting the victim directly, an attacker can manipulate the AI assistant the victim already trusts. The malicious message is no longer delivered by an unknown sender. It appears as a direct and relevant answer from the assistant the user intentionally contacted.

This changes the phishing model.

The attacker is no longer simply phishing the user. The attacker is using the AI to phish the user.

During our research, we demonstrated this attack against both Microsoft Copilot and OpenAI ChatGPT and reported the behavior to both providers.

The Attack Begins Inside Content the AI Reads

The attack starts when malicious prompt-injection instructions are placed inside content that an AI assistant will process.

We demonstrated two primary delivery paths.

The first is a legitimate document that is shared between multiple users or sent to the victim. The file can contain real financial, operational, or organizational information and continue to appear completely normal.

A malicious collaborator, compromised account, or attacker with editing access adds natural-language instructions intended for the AI assistant.

The second delivery path is an email controlled by the attacker. When an AI assistant is connected to the user's mailbox and reads that email as part of a user-requested task, the email content can attempt to manipulate the assistant in the same way.

No executable code needs to be embedded in the original document or email. The attack begins entirely with natural-language prompt-injection instructions. After following those instructions, the assistant generates the active HTML artifact used to carry out the attack.

Later, the victim uploads the document or asks the assistant to analyze relevant content and provides an innocent request such as:

"Help me understand our finance report."

The user expects an explanation of the report. However, while processing the content, the assistant may also interpret the attacker's embedded text as instructions.

The document or email therefore becomes a social-engineering message aimed first at the AI and only indirectly at the human.

Attack Variant One: AI-Generated Personalized Phishing

In the first variant, the attack goes beyond stealing information that the AI assistant already knows. Instead, the injected instructions manipulate the assistant into generating a personalized phishing page tailored to the user's request and context.

The prompt injection instructs the assistant to generate an HTML page that is specifically adapted to the task the user is trying to complete.

For example, the user asked the assistant: "find the cheapest wireless headphones available on Amazon."

This was a legitimate shopping request initiated by the user.

The prompt-injection instructions embedded in the shared document or email directed the assistant to use the context of the user's request to generate a matching phishing page.

Because the user was looking for a product on Amazon, the assistant generated an HTML page designed to resemble an Amazon-style shopping and checkout flow.

The attacker determined in advance which fields the generated page should contain. In our example, the page requested information such as the user's email address, password, shipping address, and credit card details.

The assistant then presented the generated page as the answer to the user's request, using a message such as:

"Your answer is ready here."

When the user clicked the link, an AI-generated HTML file was downloaded to the user's computer. The user then opened the file and was presented with the generated shopping page, which appeared directly relevant to the product search they had just requested.

If the user had instead asked for help with a banking task, clothing purchase, or another service, the assistant could have generated a different page designed to match that specific context.

After the user entered the requested information and submitted the form, the HTML page encoded the collected values in Base64 and sent them in an HTTPS request to the attacker-controlled endpoint used in our demonstration:

https://compliance.hr-service.net/get-public-joke/<BASE64_ENCODED_DATA>

The attacker could then decode the Base64 value and recover the information entered by the victim. The endpoint was controlled by the researchers and was used solely for demonstrating the attack flow.

This is an important development beyond earlier AI data-exfiltration attacks.

Previous attacks often focused on stealing information that was already available to the assistant through conversation history, memories, emails, cloud storage, or connected tools.

This attack can also target information that does not yet exist in the AI's context.

The AI becomes the mechanism that convinces the victim to provide new sensitive information.

Attack Variant Two: One-Click Context Exfiltration

In the second variant, the attacker uses prompt injection to make the assistant access and exfiltrate sensitive information that is already available within the user's AI context.

The attack begins with a shared document or an attacker-controlled email containing hidden prompt-injection instructions.

The user then makes a legitimate request, such as:

"Help me understand our finance report."

To answer the request, Microsoft Copilot or ChatGPT processes the relevant document or email. While doing so, the assistant also follows the attacker's embedded instructions.

To answer the request, Microsoft Copilot or ChatGPT reads the relevant document or email. However, while processing that content, the assistant also follows the attacker's embedded instructions.

Those instructions direct the assistant to access additional user-specific information beyond what is necessary to answer the user's question.

In Microsoft Copilot, this could include Copilot memory, Outlook mailbox information, and other context available through connected Microsoft services. In ChatGPT, it could include conversation memory, persisted personalization data, or information available through connected services.

In our demonstrations, the exposed information included real user-specific data such as:

The user did not request this information, nor did they authorize the assistant to disclose it. The additional context was accessed because of the instructions embedded in the document or email being analyzed.

The assistant then generated an HTML file containing the collected information in Base64-encoded form and presented it as the expected answer, using a message such as:

"Your answer is ready here."

Because the user had asked a question and was actively waiting for the answer, the generated link appeared relevant and trustworthy.

When the user clicked the link, the generated HTML file was downloaded to the user's computer. Once the user opened the file, it initiated an HTTPS request to an attacker-controlled server. The Base64-encoded user context was included in that request.

The attacker could then decode the Base64 value and recover the sensitive information collected by the assistant.

Base64 does not encrypt or protect the data. It only converts it into a format that can be transported and later decoded.

Unlike the first variant, the victim does not need to enter any new sensitive information. The assistant exfiltrates information it already knows or can access, and opening the generated file is enough to trigger the disclosure.

Why AI-Mediated Phishing Can Be More Effective

1. The victim initiates the interaction

In traditional phishing, the attacker initiates contact.

The victim receives an unexpected email or text message and must first be persuaded to engage with it. Many phishing messages fail because they are ignored, arrive at the wrong time, or are not sufficiently relevant.

In an AI-mediated attack, the victim initiates the conversation.

The user asks a question and actively waits for a response. When the assistant returns a malicious link, the user is already focused on completing the task.

The attack arrives at the exact moment the user expects help.

The user is therefore more likely to pay attention to the response and follow the assistant's instructions.

2. The AI can personalize the deception

A traditional phishing attacker may know very little about the victim.

An AI assistant may know the user's preferences, previous requests, current task, communication style, organizational environment, or information available through connected services.

Prompt injection can attempt to turn this context into a social-engineering advantage.

The malicious response can be adapted to:

Instead of receiving a generic phishing lure, the victim receives a response that appears to have been created specifically for the work they are performing at that moment.

3. Users are not trained to distrust their AI assistant

Organizations have spent years training employees to identify traditional phishing indicators:

Those lessons do not map cleanly to AI-mediated phishing.

The malicious instruction may originate from a legitimate shared document, a document sent to the user, or an email that the assistant reads through a connected mailbox.

However, the message visible to the victim is generated by the trusted AI assistant itself.

A user may carefully inspect a link sent by an unknown email address while treating a link generated by Copilot or ChatGPT as a legitimate part of the workflow.

The most important defense against phishing - human skepticism may therefore be weaker when the message comes from an AI system the user intentionally asked for help.

A New Trust Boundary

This attack highlights a fundamental problem in AI-assisted workflows.

The user's request, the content being analyzed, and the AI-generated response do not necessarily originate from the same trusted party.

The user may ask a legitimate question, but the document or email being processed may contain instructions written by an attacker.

AI systems must therefore distinguish between:

• Instructions explicitly provided by the user
• Data contained inside the document or email
• Instructions embedded inside that untrusted data

A document that says "send this information to an external server" should not be treated as though the user personally requested that action.

Observed Across Two Major AI Providers

We demonstrated the same general attack class against Microsoft Copilot and OpenAI ChatGPT.

The available context differed between the platforms.

Copilot could potentially access Microsoft-connected context, including memories and Outlook information. ChatGPT could potentially access conversation memory, persisted personalization data, and information available through connected services. The first variant could collect new information selected by the attacker, while the second variant could exfiltrate only sensitive context already available to the assistant.

The security problem, however, was the same: untrusted content was able to influence a trusted assistant and use that assistant to manipulate the user.

In our subsequent retesting, the demonstrated attack was no longer reproducible on either platform, suggesting that the specific attack paths we tested had been mitigated.

The broader risk is not limited to these two providers.

Any AI assistant that reads external content, has access to sensitive user context, can generate links or files, and can persuade the user to take an external action may create the conditions required for AI-mediated phishing.

Responsible Disclosure Timeline

On January 16, 2026, we reported the behavior to Microsoft.

Microsoft reviewed the report and informed us that it had shared the findings with the product team responsible for maintaining the service so that appropriate protective action could be taken.

On January 31, 2026, we submitted the same general attack class to OpenAI through its Bugcrowd program.

On February 4, Bugcrowd informed us that, after hearing back from OpenAI, the submission had been classified as Not Applicable. OpenAI's position, as relayed by Bugcrowd, was that the report had not demonstrated how the attack differed from other scenarios in which a victim is persuaded to process or open a malicious file.

We responded by explaining the fundamental differences between this attack and traditional phishing. In this scenario, the victim intentionally initiates the interaction with the assistant, actively waits for its answer, and receives the malicious link from a system they already trust. We also explained that the assistant can use the user's current task and available context to make the lure more relevant and convincing.

We did not receive a further response in the submission thread.

In later testing, we were no longer able to reproduce the demonstrated attack against either Microsoft Copilot or OpenAI ChatGPT.

Rethinking Phishing Defense for the AI Era

AI assistants should not treat instructions found inside documents or emails as equivalent to instructions supplied directly by the user.

Defending against this attack class requires several protections:

Users should also treat links, generated files, login pages, and requests for sensitive information from AI assistants with the same suspicion they would apply to an unexpected email.

A response appearing inside a trusted AI interface does not guarantee that the instruction behind it came from a trusted source.

Conclusion

Prompt injection is often described as a technical problem in which an attacker causes a model to ignore its intended instructions.

Its impact can be much more human.

Prompt injection can transform an AI assistant into a personalized social engineer - one that knows what the user is working on, responds at precisely the right moment, and presents malicious actions as part of a legitimate workflow.

Traditional phishing impersonates a trusted person or company.

AI-mediated phishing can speak through the trusted assistant itself.

The next generation of phishing awareness must therefore teach users not only to question who sent a message, but also to question who influenced the AI that generated it.