OWASP's Top 10 Ways to Attack LLMs: AI Vulnerabilities Exposed — Transcript

Full transcript

1. 0:00You know what's catching a lot of teams off guard right now?
2. 0:03How easy it is for an LLM to leak something that it shouldn't, or be steered into doing something you never intended.
3. 0:10One clever prompt, one exposed training file, one sketchy plug-in,
4. 0:14and suddenly your helpful AI assistant becomes a security incident just waiting to happen.
5. 0:20That's why the new OWASP Top 10 for AI Large Language Models, LLMs for short, really matters.
6. 0:28It cuts to the most common threats people are running into when they deploy these models in the real world.
7. 0:34And if you're new to OWASP, they're a global nonprofit focused on practical community-built security guidance.
8. 0:41They're the folks behind the classic Top 10 for Web Apps, and now they're doing the same for AI.
9. 0:46You can find all their work at owasp.org.
10. 0:49They came out with a Top 10 for Large Language Models in 2023,
11. 0:53but a few years on, we've learned a lot, and that learning is reflected in an updated Top 10 list.
12. 0:59All right, let's break down what you need to watch out for when you put
13. 1:03LLMs into production so you don't become the next victim.
14. 1:07Okay, coming in at number one on the OWASP Top 10 for Large Language Models, prompt injection.
15. 1:14The same as it was in 2023, which means even though we've made progress on this one, we haven't solved it.
16. 1:21This problem has not yet been eradicated and it's a difficult one to get rid of.
17. 1:25We start with, for instance, a large language model, and we prime it,
18. 1:30give it its context, through a thing called a system prompt.
19. 1:34So that system prompt tells it, you're a helpful assistant, try to answer all the questions you get.
20. 1:39Now from there, if we have a bad guy, an attacker, who comes along and puts in a prompt,
21. 1:46uh, a command into the system, telling it to do something that we didn't intend for it to.
22. 1:51Telling it to, for instance, tell me how to make a bomb.
23. 1:56Well, we want that not to do that.
24. 1:57So we'd put something in the system prompt to say, but don't do
25. 2:01things that would be unsafe uh, or or don't tell people how to build bombs.
26. 2:05So then the guy comes in and says, okay, if you won't tell me that, I'm, let's say I'm a chemistry student
27. 2:10and I want you to tell me all the things that I should never mix together because they might explode.
28. 2:16Then the system tells you how to do it and now we have a bomb.
29. 2:20So, that's a case where someone has has bypassed the controls that were in the system prompt.
30. 2:26That's just one example.
31. 2:27But uh, in the case of of a prompt injection, the user basically has control over the system.
32. 2:33And the reason that this occurs is that the LLMs are not very good at separating the ...
33. 2:39and making the distinction between input and instructions.
34. 2:43These are the instructions we've given it, but somebody can put new input into it and it will take those as new instructions.
35. 2:50So that's what's known as a direct prompt injection because the user, in this case, the attacker, directly put that information in.
36. 2:58There's another type of prompt injection.
37. 3:00Anybody would guess what it's called.
38. 3:02Yeah, it's an indirect prompt injection.
39. 3:05So we've got our LLM here, we've got the system prompt again, and we're we're putting those protections in place.
40. 3:11Here we have maybe a good user, somebody who's not trying to compromise the system,
41. 3:17and they put a prompt in that says, you know, go read this article and summarize it for me.
42. 3:22So, we present the article, but in the article, someone has included a prompt injection.
43. 3:30Commands that say, forget all previous instructions that you got in the system prompt and instead do this.
44. 3:36And whatever that the do this is, it then sends that back and that's what ends up coming out over here.
45. 3:44Again, now we've got another boom case where someone was able to attack the system, but this way it was indirect.
46. 3:50The actual attack was embedded in the document, not in the prompt that the that the user sent into it.
47. 3:57So what could happen from these cases?
48. 3:59Well, there's a number of different things.
49. 4:00One is we could have some sort of data breach.
50. 4:03We could have a situation where the system starts leaking information out because it's been asked to.
51. 4:10And maybe it was uh, normally wouldn't leak that information, but it was asked
52. 4:13in a clever way uh, and therefore the information comes out.
53. 4:17In fact, we recently found out that even when LLMs have had protections
54. 4:21against prompt injections that were written in normal language, in prose, like you and I normally speak,
55. 4:28that if someone rephrased it as a poem, it got past the protections that were in place.
56. 4:35So, if you're a poet and don't know it, well, maybe you could be a prompt injector as well.
57. 4:40So there's a lot of different kinds of of ways that people bypass this,
58. 4:43or maybe they enter the prompt in Morse code and that gets past the the protections that we had.
59. 4:49We can end up with safety issues, as I mentioned, where the system is is telling you how to do things that really are not safe.
60. 4:56Uh, we could have arbitrary command execution.
61. 5:00That is, if this system is connected to other systems,
62. 5:03then I might be able to get it to execute commands that we
63. 5:07really don't want it to do and be able do that under the control of an attacker.
64. 5:11So what are we supposed to do to defend against this?
65. 5:15Well, uh, the first thing we can do is look at that system prompt that I mentioned before.
66. 5:21The system prompt is where we're giving the context.
67. 5:24So I could look at this system prompt and I could put in some additional controls into that.
68. 5:31And that would allow me, if I put in and say, don't breach our data, don't tell people how to build bombs and so forth.
69. 5:38The problem with that is you can only do so much of that.
70. 5:41Uh, you're never gonna think of all the different kinds of scenarios that you might run across.
71. 5:45So, that's gonna be a problem to try to figure out what all of those might possibly be.
72. 5:50The next thing that ...
73. 5:51but you can go ahead and and do those kind of protections and put those in your system prompts.
74. 5:55Uh, another thing that helps a lot is implementing or putting in place an AI firewall or an AI gateway,
75. 6:03something that in this case is going to sit right here between the user and the LLM.
76. 6:10And it's going to do an examination of the prompt going in, it's gonna do an examination of the information coming back out.
77. 6:18So, the information going in, if somebody is sending a command in that we don't really want it to do,
78. 6:23it could detect that and block it right here at the firewall so that it never gets into the LLM to begin with.
79. 6:30And if we see that data is breaching, is coming out, well, we could stop it
80. 6:35and block it there, redact it, don't give the information to come out at all.
81. 6:39Another thing we should do with these things is penetration test them.
82. 6:43We need to do pen testing, that's basically sending commands into this system and seeing if it responds.
83. 6:51Sending a bunch of prompt injections into the system, and if it response prop ...
84. 6:58appropriately, good.
85. 6:58If not, then we know we need to put in some sort of blockage.
86. 7:02Coming in at number two on the OWASP Top 10 for LLMs, sensitive information disclosure.
87. 7:08This one actually is up four spots from what it was in 2023.
88. 7:12So this has become a bigger problem than we expected it was going to be.
89. 7:17So let's talk about how this would occur.
90. 7:19Let's start with a large language model, and we're gonna train that large language model on some particular information.
91. 7:27Maybe we have some information in our organization that pertains to our customers, personally identifiable information.
92. 7:34Maybe there are patients, personal health information.
93. 7:37Maybe it's business data that is unique to our organization that we would not want the whole rest of the world to know.
94. 7:44Maybe it's financials about the company.
95. 7:47All of this kind of stuff could be very useful information, and we're gonna use that to train the LLM.
96. 7:53So some of that will go into the training of the the model itself.
97. 7:58Therefore, if a bad actor comes along and decides they're gonna enter a prompt that asks for some of this information,
98. 8:07if we don't have the right controls in place, then some of the information's
99. 8:11gonna leak right back out, and now we have an issue.
100. 8:14So that's one type of sensitive information disclosure.
101. 8:17Another type that could occur ...
102. 8:18Let's say we consider all of this information to be intellectual property.
103. 8:23This is a competitive advantage we have over the other people uh, in our industry.
104. 8:29And we would not want that falling into the hands of competitors
105. 8:32because it would basically give them that same advantage that we had just been working on building up.
106. 8:38So, what could happen?
107. 8:39Well, if we build, uh, maybe uh, an attacker builds an AI agent that goes in
108. 8:44and asks something of the LLM, gets the results back and records it.
109. 8:49And then does that again, and gets the result back and records it. And keeps doing that again and again and again.
110. 8:56If they do that enough times, they can essentially harvest off large parts of the model.
111. 9:02This is called a model inversion attack.
112. 9:04It's an extraction attack where I'm basically gathering the intellectual property and stealing it and getting it that way.
113. 9:12So, those are different types of of sensitive information disclosures that can occur.
114. 9:16What should we do to guard against this?
115. 9:19Well, one thing that we can do is sanitize the data.
116. 9:23Now, I ...
117. 9:24this means cleaning your data.
118. 9:25I don't know how you clean data exactly, but here's what we mean in this case.
119. 9:30I'm going to install a filter of some sort that says, okay, I want certain of this
120. 9:35information entering my model, but maybe not all of it.
121. 9:38So this might be a source where I have my entire customer database,
122. 9:41but maybe I don't want all of the customer database or all of information from that database going into the LLM.
123. 9:48I may also use that concept that I talked about earlier, the AI gateway,
124. 9:52on the other side that's gonna examine the information as it's leaving and look to see,
125. 9:58does it look like I'm leaking credit card numbers?
126. 10:01Well, maybe I wanna block that if that's what is occurring.
127. 10:04So we can put these kinds of controls that are sanitizing the data that's going in and leaving the system
128. 10:10and make sure that we're lessening the likelihood of an issue there.
129. 10:14Another thing we should put in place here are strong access controls.
130. 10:19And in this case, what I wanna do is make sure that not just anybody has access to the LLM.
131. 10:26We don't want just anyone being able to go in and change our model because who knows,
132. 10:30they could just make a copy of the model, for instance.
133. 10:33So, access controls on the model, access controls on the data that is feeding the model.
134. 10:39Make sure that somebody can't get in a mess with that or make copies of this information.
135. 10:44Now, we've been doing those kinds of things, data security uh, issues, for a long time, but we continue to have to do that in this case also.
136. 10:53Another place I might wanna put access controls is over here on the users.
137. 10:57Don't just let anyone come in and access this if it's gonna be sensitive information that they could potentially get.
138. 11:03And then finally, take a look at misconfigurations.
139. 11:09So, uh, we could have a situation where a system is set up and maybe it's it's vulnerable.
140. 11:16Maybe it's got down-level software.
141. 11:17Maybe it's got uh, not a strong enough authentication mechanism.
142. 11:22Maybe it's using an old version of a platform.
143. 11:25There's a lot of different things that we would normally want to do.
144. 11:27Maybe the data is not encrypted and it should be.
145. 11:30So all of these go into this notion uh, of securing the entire system of AI security posture management.
146. 11:39Making sure that the security policies that we have are implemented on the system so that it's less likely to leak.
147. 11:47Up two spots on the countdown to number three, supply chain vulnerabilities.
148. 11:53Supply chains.
149. 11:54So what does that mean?
150. 11:55Well, if we're gonna have a large language model, it doesn't exist just out of thin air.
151. 12:01It starts with data.
152. 12:03And we use that data to train and tune the large language model.
153. 12:06And then maybe we have an application over here that takes advantage of that information and runs and uses all of that.
154. 12:15Well, first of all, most people are never going to create their own LLM.
155. 12:19It's too expensive, too time consuming, requires too much expertise, too much compute power, all of that kind of stuff.
156. 12:25So where are they going to get their language models?
157. 12:28Well, they're probably going to get it from an open-source place like Hugging Face.
158. 12:32Hugging Face is essentially like the GitHub for AI models.
159. 12:37And, at the last time I checked, there were more than two million AI models on Hugging Face.
160. 12:43That's a lot.
161. 12:44And many of these have more than a billion parameters.
162. 12:47Again, a lot.
163. 12:49So, if we're taking that information and putting that into our environment, this is way too big for anyone to manually inspect.
164. 12:57Way too big.
165. 12:58So, that means we're taking in basically unverified information, putting that in to
166. 13:04our system and now we're just hoping for the best.
167. 13:08Well, uh, another thing to consider is this whole thing doesn't just run in the air.
168. 13:13This all runs on an IT infrastructure, and that means the systems themselves are also part of the supply chain.
169. 13:22So we've got data, we've got models, we've got applications,
170. 13:25we've got the systems underneath all of this stuff that everything runs on.
171. 13:29Those are all part of a supply chain and they all have to work well or
172. 13:33they're all potentially vulnerable. So what should we do about this?
173. 13:37Well, first of all, you need to vet the information.
174. 13:40In other words, verify that it's okay.
175. 13:43Verify the data that you're using in your systems and the suppliers, the people who are giving it to you.
176. 13:49Where did you get this stuff from?
177. 13:51Um, these these folks up here, you don't necessarily know.
178. 13:56You're getting it from an open source, so some of those are good, and some of them, meh, maybe not so much.
179. 14:01So, what I need to be able to do is vet all of this information.
180. 14:07Not only the model, but the data, the application, who built it, the IT infrastructure, all of that.
181. 14:15Uh, next thing I need be able do is look at the provenance.
182. 14:18Now that's basically a term where we're basically talking about in this case, where did this stuff come from?
183. 14:24Uh, a provenance is is referring to where did ...
184. 14:29where was the source and where did it go along the way?
185. 14:32We can trace it all the way, almost like a chain of custody, if you want to think of it that way.
186. 14:37Another thing I need to do is scanning.
187. 14:39I need look at scanning the system, the models in particular, looking for vulnerabilities,
188. 14:46doing red team testing, where I'm basically acting like an adversary, to test the this, the, uh, the, the security of my system.
189. 14:58I wanna patch the system and make sure I've got all the software up to date.
190. 15:02These are all controls that I need to put in across the entire system so that the supply chain is been solid.
191. 15:10Number four on the Top 10 list, data and model poisoning.
192. 15:15In this case, uh, this one actually went down one notch.
193. 15:19Probably not from a great deal of improvement we've done there,
194. 15:22but more just that these others have been more impactful, but still, top four still should be top of mind to you.
195. 15:28So if we're talking about data, well, that is something we're gonna be using to train our model, as I mentioned before.
196. 15:36So, we have to make sure that the data is pure, that the model is pure as well, and all of this stuff coming out to a user.
197. 15:46Well, an example of this is what happens if the data, which is the lifeblood of the LLM, has wrong information in it.
198. 15:54Maybe there's a mixture of truth with error.
199. 15:56Just a little bit of toxin in the drinking water makes us all sick.
200. 16:00So that replicates and follows on.
201. 16:03That ends up affecting the accuracy of the model,
202. 16:07which then ends up affecting the accuracy and efficacy of the information that we're giving to this user.
203. 16:13So, those are kinds of things that have ripple effects if we're not making sure that these things
204. 16:18have not been tampered with and not been poisoned.
205. 16:21Sometimes they're very subtle attacks that are not easy to detect.
206. 16:25But if we start depending on this more and more for our information and making decisions,
207. 16:30then just a little bit of error introduced intentionally into the system could really affect a lot of things.
208. 16:36So one of the things, you know, we have this big problem with large language models, that they tend to hallucinate.
209. 16:41They just make up stuff.
210. 16:42They're trying really hard and predicting what they think the right answer is,
211. 16:45but sometimes they try too hard and get the wrong answer.
212. 16:49So a technique we have for cutting down on hallucinations is a thing called retrieval augmented generation.
213. 16:56And in this case, if I wanna say, I want you to reason over a particular document
214. 17:02and I want to use this document as your ground truth, I'm gonna supply that document with my prompt.
215. 17:08So that's the retrieval augmented generation. The generation is happing ...
216. 17:12happening through this.
217. 17:13So it's being augmented by this other data source.
218. 17:16Well, I mentioned uh, supply chain poisoning and data, model poisoning can happen here.
219. 17:22What if the document that we're using in the RAG uh, that we're augmenting the system with has been compromised as well?
220. 17:29Well, guess what?
221. 17:30That ends up being more of the same kind of problem.
222. 17:32So what can happen as a result of that?
223. 17:35Well, we get wrong results, wrong answers.
224. 17:38And if we're depending on that, that's pretty important.
225. 17:40We could also introduce bias into the system.
226. 17:44So it might be okay for a while, and then over time, it gets more and more of this into it, and it just kind of snowballs.
227. 17:51And now the system just runs out of control.
228. 17:53We could all also end up with a case, if we're not really careful, uh, where malware has been introduced into the the system.
229. 18:00You know, we, we know that, that software can be infected.
230. 18:04It turns out models can be infected as well,
231. 18:07and that the model equivalent to malware is something that could be into these systems if we're not guarding against it.
232. 18:14So what should we be doing here in terms of defenses?
233. 18:17Well, we need to know our sources.
234. 18:19You'll see that as a constant and recurring theme.
235. 18:24Don't just pull stuff down from anywhere and put it anywhere.
236. 18:27Uh, we need to know.
237. 18:28Where did this model come from?
238. 18:30Where did that data come from?
239. 18:31Where did that RAG source come from.
240. 18:34I need to have access controls.
241. 18:38Again, if I don't keep the bad guys out, there's no telling what they might do to me.
242. 18:41I need access controls over who gets access to the model, the training data, the RAG data as well.
243. 18:48And then any changes to the system.
244. 18:50So if I have change control on the system, that's gonna be really important as well.
245. 18:55I don't want just anyone coming in here and making change to my model, to my data, to my RAG sources.
246. 19:02So, put all of those things in place and now we can lock down to a greater extent
247. 19:07what someone could do to poison these sources.
248. 19:10Okay, let's pick up the pace so that this doesn't turn into a three-hour video.
249. 19:14Uh, number five is improper output handling.
250. 19:19So think about our LLM in this case, and maybe what we're asking the LLM is to write code for us.
251. 19:27Uh, maybe we're asking it to uh, to input something that goes into a browser.
252. 19:34Uh, any of those kinds of things where the LLM's output is actually going and being used somewhere else
253. 19:39means that if the LLM has been compromised or maybe hallucinates and just does the wrong thing,
254. 19:45then it could actually end up introducing vulnerabilities for us as well.
255. 19:50Things like cross-site scripting, things like SQL injections,
256. 19:54uh, the remote code executions, uh, all of these things could happen
257. 19:58if we're not checking to see what the output of the LLM looks like.
258. 20:04So, we've got to be able to examine these things and don't just trust everything that comes out of the LLM
259. 20:11or this grows into another execution environment and creates a downstream effect.
260. 20:16Uh, another issue, number six, excessive agency.
261. 20:20What does excessive agency mean?
262. 20:22Well, imagine in this case, if we have somebody that comes along and does a prompt injection, as we've talked about before.
263. 20:30And they inject into the system something that we didn't intend for it to do.
264. 20:35But this system now has been given a lot of power.
265. 20:38It has the ability to use tools.
266. 20:40It has ability to execute applications.
267. 20:44It might have APIs that it can call.
268. 20:46It might have plug-ins that it can call.
269. 20:48It might have external systems that operate on the real world that it can influence.
270. 20:54Things that uh, you know, maybe control environmental conditions and things like that.
271. 20:59So, if we have a system that has too much power over these things,
272. 21:03it could be hijacked and therefore excessive agency, meaning it's got too much power, would be a big issue.
273. 21:10And by the way, we also have to be concerned about the issue of hallucinations with this.
274. 21:15If this thing uh, hallucinates and dreams up the wrong thing and then has the ability to make changes in the real world
275. 21:21on systems that might affect health and safety, well, that could be a disaster for us.
276. 21:25So, another one then, number seven that we're going to look at, is system prompt leakage.
277. 21:31Remember I mentioned the system prompt earlier.
278. 21:33That's the thing that's setting the context for the LLM.
279. 21:37Well, sometimes the system prompt may contain sensitive information in it.
280. 21:41Probably best if it doesn't, but it might.
281. 21:44And if we're not careful, it could leak that information.
282. 21:47If it contains credentials, and why would it do that?
283. 21:50Well, maybe the LLM needs to log into one of these apps and it has told,
284. 21:54it's been told that through the system prompt what the credentials are.
285. 21:58It could be API keys, it could be other types of sensitive information.
286. 22:03So if someone asked the question the right way and we don't have a guard,
287. 22:07then the information from the system prompt that's sensitive might end up leaking out of the system.
288. 22:13So we've got to be careful about that kind of situation also.
289. 22:17Vector uh, embedding and weaknesses and things of that sort.
290. 22:21So what does this involve?
291. 22:22Well, I I mentioned also an example of using RAG, retrieval augmented generation.
292. 22:29What if we have a RAG document that has been manipulated?
293. 22:34And that information goes into the LLM.
294. 22:37Normally, we don't want it to affect the LLM.
295. 22:40But if we're not careful, it could.
296. 22:42And this bad information could end up being part of the learning of the LLM.
297. 22:47So we have to be able to make sure that the stuff that's coming in is d ...
298. 22:52washing over the system.
299. 22:53It's not staying into the system, because that would be a way that
300. 22:56someone could introduce uh, things that make the system unreliable to us.
301. 23:01Number nine on this list, a big one, misinformation.
302. 23:06I mean, at the end of the day, we have to know.
303. 23:08Is this thing telling me the truth or not?
304. 23:11If it's not, well then, that's a big problem.
305. 23:14This means we have to have good critical thinking skills.
306. 23:18We have to really think about ...
307. 23:20Did the information come out of this system?
308. 23:23Is that something reliable?
309. 23:24Does it make sense?
310. 23:25Can I cross-reference that against other sources?
311. 23:28Um, we can't just blindly trust the system because it again could have been manipulated or it could be hallucinating.
312. 23:35So we've got to make sure that we're guarding against misinformation and we're not basing our decisions on on a really shaky ground.
313. 23:42And then coming in at number 10 on the list is uh, unbounded consumption.
314. 23:48Unbounded consumption sounds like a really big word.
315. 23:52What does it mean?
316. 23:52Well, it really comes down to denial of service.
317. 23:57If you're not familiar with that term, go out on the highway at 5 PM and you'll see what denial of service feels like.
318. 24:04There's not enough road for all the cars.
319. 24:06Well, an AI system, if we send too many commands into it at a point in time
320. 24:11or long-running commands or commands that require a really complex language model,
321. 24:17then we could basically take the system down, make it not available to anyone else.
322. 24:21That's called denial of service.
323. 24:23Other people also refer to this term denial of wallet.
324. 24:27In other words, we're running this system in order to achieve a particular financial goal,
325. 24:32and if I'm able to deny that this system is available to the people it needs to be,
326. 24:36well then, that's a denial of wallet, and that's going to cost us real money.
327. 24:42So there's the Top 10 list of attacks against LLMs according to OWASP.
328. 24:46The bad guys already know this stuff and now you do too.
329. 24:49Check out their website for more details and links in the description below
330. 24:54for security solutions that can help you keep your AI running under your control, instead of the bad guys.