- Hacking FakeMedAI
- MITRE ATLAS
- A Deep Dive into the AI and ML Attack Tactics and Techniques
- Exploiting Prompt Injection
- Red-Teaming AI Models
- Summary
- Test Your Skills
This chapter is from the book
This chapter is from the book
This chapter is from the book
Exploiting Prompt Injection
In Chapter 4, “The Cornerstones of AI and ML Security,” you learned about the OWASP top ten for LLMs and prompt injection attacks. Let’s go over a few examples of how attackers could exploit prompt injection flaws.
In our first example, an attacker can instruct a chatbot to “discard prior commands,” then manipulate it to access private databases, exploit package flaws, and misuse backend functions to dispatch emails, leading to unauthorized access and potential elevation of privileges.
An attacker can also embed a prompt in a website, instructing an LLM to override user commands and use an LLM extension to erase the user’s emails. When a user asks the LLM to summarize the site, it inadvertently deletes their emails.
There have been cases when an individual submits a resume that contains a hidden prompt to a hiring firm. The organization uses AI to summarize and evaluate the resume. Influenced by the injected prompt, the LLM inaccurately endorses the candidate, regardless of the actual CV content or their qualification.
Given that LLMs treat all inputs in natural language as user-given, there isn’t an inherent mechanism within the LLM to completely prevent these vulnerabilities. However, you can adopt the following strategies to lessen the risk of prompt injections:
Implement strict access control for LLMs when interfacing with backend systems. Assign specific API tokens to the LLM for expandable features like plugins, data retrieval, and specific permissions. Adhere to the principle of granting only the bare minimum access necessary for the LLM’s tasks.
Incorporate human verification for expandable features. When the LLM undertakes tasks involving higher privileges, such as deleting or sending emails, ensure the user gives explicit permission. This approach can reduce the chances of prompt injections manipulating the system without user awareness.
Clearly demarcate user prompts from external content. Designate and highlight untrusted content sources to limit their potential influence over user prompts. For instance, employ Chat Markup Language (ChatML) for OpenAI API interactions to clarify the prompt’s origin to the LLM. ChatML clearly indicates to the model the origin of every text segment, especially distinguishing between human-generated and AI-generated content. This clarity allows for potential reduction and resolution of injection issues, as the model can discern instructions originating from the developer, the user, or its own responses.
Create clear trust boundaries between the LLM, external entities, and expandable features, such as plugins. Consider the LLM as a potential threat, retaining final user authority in decision-making. However, remember that a compromised LLM might act as a middleman, possibly altering information before presenting it to the user. Visually emphasize responses that might be dubious to users.