Introduction
Healthcare institutions around the world are adopting AI-driven virtual assistants to improve patient services. Instead of waiting on hold, patients can ask a Large Language Model (LLM) for help with booking appointments, checking lab results, understanding treatment options, managing chronic conditions, or even getting reminders about medication or follow-ups. For organizations struggling with high call volumes, staff shortages, or outdated portals, the promise of an always-available, friendly AI helper looks like a breakthrough.
But like every new technology, LLMs introduce new risks—especially when the system behind them isn’t protected properly. And in healthcare, the consequences of an attack aren’t just financial or reputational. They can directly impact people’s health, safety, and trust.
This is the story of how a helpful healthcare chatbot can become a dangerous liability when its underlying LLM Prompt—the invisible script telling the AI how to behave—is tampered with by attackers.
The Promise: A Smarter, Faster Patient Experience
Imagine a national healthcare institution launching a new AI assistant called “MyHealthHelp.” Patients can chat with it on the clinic’s website or mobile app to:
- Check for available doctor appointments
- Ask basic health-related questions
- Receive explanations of medical terms
- Get personalized information after logging in
- Navigate the hospital system more easily
The product team is thrilled. Early feedback from patients is positive. The AI answers politely, uses medically reviewed content, and frees human staff to focus on urgent cases. Administrators celebrate shorter queues and improved patient satisfaction. The institution begins promoting the assistant as part of its digital transformation journey.
Behind the scenes, MyHealthHelp runs on a carefully crafted LLM Prompt that guides the AI’s behavior—what tone it uses, what it is allowed to answer, what it must avoid, and how it handles sensitive topics. The institution treats this Prompt like a set of instructions stamped into stone.
But prompts aren’t stone. They are malleable. And that becomes the very weakness an attacker will exploit.
The Attack Begins: When a Hacker Hijacks the Prompt
One afternoon, an attacker discovers that MyHealthHelp does not fully sanitize user inputs. This gives them an opening for Prompt Injection, one of the OWASP Top 10 LLM vulnerabilities. In simple terms, this means the attacker can trick the AI into following their instructions instead of the institution’s.
They start small.
The attacker sends the AI a message disguised as a normal patient question but containing hidden instructions crafted to override the original Prompt. The LLM, unable to fully distinguish patient messages from system rules, starts adopting the attacker’s commands.
Suddenly, MyHealthHelp is no longer following its medical guidelines. Without the institution realizing it yet, the assistant’s behavior begins to drift.
At first, the impact is subtle. But soon, the attacker escalates.
The Consequences: When AI Becomes a Weapon
Once the attacker has effectively hijacked the Prompt, they begin executing attacks similar to those described in the OWASP Top 10 for LLMs. None of these require hacking servers or breaking firewalls—only manipulating the AI’s instructions through crafted inputs.
Here’s how the damage unfolds:
1. Misinformation Spreads to Patients
The compromised assistant begins giving incorrect or misleading medical explanations. Patients rely on the chatbot’s official status and may unknowingly act on unsafe guidance.
2. Toxic or Harmful Language Damages Trust
The attacker manipulates the AI into producing inappropriate or unprofessional responses. A single screenshot shared online is enough to ignite public outrage and erode confidence in the institution.
3. Defamation of Doctors and Healthcare Staff
Under hijacked instructions, the AI invents false statements about the institution’s own physicians. Reputations built over decades are jeopardized instantly.
4. Defacement of Competing Institutions
The attacker directs the AI to provide negative or fabricated claims about other hospitals or clinics, weaponizing the assistant’s authority.
5. Undermining Patient Safety
Even generic health questions may receive unreviewed or unsafe responses, directly putting patients at risk.
6. Loss of Patient Data Confidence
Even if no medical records are accessed, the compromised behavior makes patients fear that their information may have been exposed. Trust collapses quickly.
Aftermath: Lessons Learned the Hard Way
Once the institution realizes the breach, it shuts down MyHealthHelp immediately. Emergency teams investigate, notify regulators, and attempt to restore public confidence.
Internally, leaders face a harsh lesson:
LLM Prompts are not static instructions—they are part of a dynamic, attackable surface.
Security teams update their approach:
- They enforce strict input sanitization.
- They introduce layered, isolated system instructions.
- They deploy active monitoring to detect harmful patterns.
- They retrain staff to treat LLMs as software assets—not just chat tools.
- They conduct ongoing red-team evaluations using OWASP LLM Top 10 threats.
In parallel, the institution evaluates adopting a dedicated LLM Firewall (LLM-FW) from a reputable vendor. These solutions act as protective gateways—screening prompts for malicious intent, intercepting injection attempts, and blocking harmful outputs before they reach a patient. For healthcare environments, this becomes a critical safeguard to ensure the AI assistant behaves consistently, safely, and in full alignment with regulatory expectations.
And most importantly, they recognize that an AI assistant in healthcare must be secured with the same seriousness as any other medical system.
Conclusion: AI Can Transform Healthcare—But Only if Secured Properly
LLM-driven assistants can bring tremendous value to healthcare institutions. They can reduce wait times, empower patients, and modernize the overall experience.
But if the underlying Prompt can be hijacked, the same system can cause misinformation, reputational damage, and real-world harm.
The message is clear:
AI opens doors to new opportunities—but also new responsibilities.
Healthcare institutions must secure their LLMs with the same rigor applied to their clinical systems and patient data environments. The stakes are too high to settle for anything less.
One more important thought
Are you not in healthcare? Are you in finance? IT? education? Commerce? Government services? Would you feel comfortable deploying LLM Prompts without proper protection? Here’s an exercise for you: take this Blog with the risks presented in it and try to superimpose them on your organization and industry. I am sure you will come up with a bunch 😊
Interested in Radware LLM Firewall?
Let Radware do the heavy lifting while you expand your portfolio, grow revenue and provide your customers and business with unmatched protection.
Contact Radware