What is AI Technical Debt? Key Risks for Machine Learning Projects — Transcript

Full transcript

1. 0:00Let's address the elephant in the room.
2. 0:03Everyone's racing to ship AI.
3. 0:05Chatbots, agents, automations, and the demos look great.
4. 0:08But behind the scenes, hard-coded prompts, no evals, no versioning, security as an afterthought.
5. 0:15The elephant is AI technical debt, and it's growing faster than your model accuracy.
6. 0:21Oh yeah, the other elephant in the room is a broken shoulder,
7. 0:24which is what can happen if you go for a run and fail to take into account the patch of ice on the trail ahead.
8. 0:30Yet another form of technical debt resulting from poor planning, I'll admit, but I digress.
9. 0:36What is AI technical debt?
10. 0:38Well, it's basically trading off speed now for costs later.
11. 0:44Or another way to look at it is technical debt is future cost from present shortcuts.
12. 0:49It's the interest you have to pay because you didn't make a large enough down payment upfront.
13. 0:54Those future interest payments come in the form of bugs, refactoring costs and maintenance.
14. 1:00This practice has been around for a while, but it's really taking off with AI where there is such a rush to deploy something, anything, to remain competitive.
15. 1:09Let's take a look at this situation, try to understand it better and see what we can do to prevent it.
16. 1:16Ready, fire, aim.
17. 1:18Who does that?
18. 1:19Well, that's what causes technical debt is that kind of approach because basically what we're doing is we do implementation,
19. 1:26then we do deployment and then we go back and figure out how to plan this thing.
20. 1:31It's opposite, it's backwards.
21. 1:33The idea is we're going to deploy it now with imperfect results and then just fix it later in the field.
22. 1:38So, go fast and break things.
23. 1:41Well, that's what got me this.
24. 1:43So not always what you wanna do is go fast and break things.
25. 1:46It's sort of like trying to repair a plane when it's in the air.
26. 1:50Yeah, you can do that, but it's a lot more difficult and it's lot more expensive and there's a more risk involved.
27. 1:55So another way to look at this is if you start off with ad hoc design, well, then you're gonna end up with a more fragile system.
28. 2:03If you don't have any architecture that you've done up front, well, then you've got no future with this thing.
29. 2:09I mean, who would build a building and not do an architecture first?
30. 2:13You don't just start laying down bricks and then figuring out where they're gonna lie later.
31. 2:19And then ultimately what you end up with is a monolithic system instead of one that's modular.
32. 2:26And what we want is a modular system so I can make changes later and I have the flexibility that we need.
33. 2:31That'll actually give me more speed in the long run.
34. 2:34So, we're making strategic decisions.
35. 2:37And so to be fair, not all technical debt is bad.
36. 2:40Sometimes we take it on with full awareness that this is in fact what we're doing.
37. 2:44So we've got technical debt that's strategic and then we've got technical debt that's reckless.
38. 2:50Strategic technical debt.
39. 2:51That's something that we take consciously.
40. 2:54We know what we are doing, we know that there are some risks and we figured that the risks are worth it in this particular case, but we've thought it through.
41. 3:02We've documented what we're doing, it's time-bound.
42. 3:05We know this is not the permanent way the system's gonna work and we have plans in place already to remediate how this system is going to work
43. 3:13when we finally get the thing out into production or after it's through its first version.
44. 3:17So that's a way of getting out there fast.
45. 3:20Doesn't mean we always just sit there and overthink things, but in some cases we go fast but we have a plan to deal with it.
46. 3:28Now reckless technical debt is what's caused by poor discipline.
47. 3:32We basically just didn't take the time.
48. 3:35We didn't do the documentation and we have no remediation plan.
49. 3:39We have no future.
50. 3:41All we have is a future mess headed for us.
51. 3:45Okay, so why is this problem with technical debt particularly worse with AI?
52. 3:50Well, let's first of all, take a look at traditional software.
53. 3:53What are the issues that we deal with here?
54. 3:55Well, traditional software is deterministic.
55. 3:58So, you give it a certain set of inputs, you'll always get the same outputs, at least that's what we hope.
56. 4:02And it's predictable as a result.
57. 4:05It's testable.
58. 4:06We can have expected results and when we test it, we get those expected results every single time.
59. 4:13So, we've got a system that we can kind of understand, get our heads around.
60. 4:18Technical debt in the traditional software sense comes out though in the form of spaghetti code.
61. 4:24That kind of stuff where we've got go-tos and things are moving here and there and everything's interleaving and it's just a big hard-coded mess
62. 4:31with a lot of hard-coded assumptions.
63. 4:34We may even have passwords and API keys and stuff like that hard-coded into, baked into the formula, in other words.
64. 4:41So, it's very inflexible and therefore the cost of changes is very high.
65. 4:46So we're incurring that debt in regular traditional software.
66. 4:50And we've got missing tests.
67. 4:52We haven't really tested some of these things out and that's going to result in yet more debt because we're going to find out about those problems later
68. 4:59and they're going to be more expensive to fix later in the cycle.
69. 5:03Now, over here on the AI side, what's AI like?
70. 5:07Well, kind of the opposite.
71. 5:09It's probabilistic rather than deterministic.
72. 5:12It is non-deterministic is another way to say.
73. 5:14In other words, there's a probability you can give it certain inputs and you won't always get,
74. 5:20even with the same inputs, you will not necessarily always get the very same outputs.
75. 5:23It's sensitive to the inputs that you give, but you could give the same ones and still end up with different results.
76. 5:30It's all very context-dependent as well.
77. 5:34So, what you've said before will affect what the result might be this time versus what you'd done in another case with the same inputs might give a different result.
78. 5:43It's all of these things which, you know what, is kind of just like people.
79. 5:48You know, we're context-dependent and not always completely predictable and all that kind of stuff.
80. 5:54One way this has been described is change anything, changes everything.
81. 5:59So, that's kind of how we look at AI.
82. 6:02And ultimately, what this means is, because AI moves really fast, then as it moves fast, then debt compounds even faster.
83. 6:13Okay, so let's take a look and see what are the different forms of AI technical debt and how might they be manifested in our particular system.
84. 6:21Well, we're gonna look at four different aspects of this.
85. 6:23The first we're going to start off with is data because there is no AI without data.
86. 6:29We use data to train and tune our systems.
87. 6:31And all of the decisions that it comes out with are based on data.
88. 6:35So, that means we're subject to garbage in equals garbage out.
89. 6:40With an AI system, the garbage out can actually be amplified.
90. 6:44So we've got to make sure that the data that we're feeding in is good.
91. 6:47We've vetted the sources and we know that they're trustworthy and that they are correct.
92. 6:51We need to make that we don't have bias in the system.
93. 6:54If we fed it too much of one kind of data and not enough of the full spectrum, then it's only gonna be accurate on that particular segment.
94. 7:02And we need it not to be biased.
95. 7:03We want it to give good results across all different data types or we end up missing something important.
96. 7:09We need to check for drift.
97. 7:11We need make sure that even though this thing was trained on good data to begin with,
98. 7:15that it hasn't changed over time because again, it may have seen too much of one type of thing.
99. 7:20And then we may need to make sure that we haven't had a case of poisoning.
100. 7:25We were in such a rush to get this thing out that the system just got deployed
101. 7:29and somebody actually put some stuff into the data intentionally that's messing all of this up.
102. 7:35We may also wanna protect data by anonymizing it so that we, and that's the kind of thing that we tend to overlook if we're in a hurry,
103. 7:42so, we need to take advantage of anonymization services so that our model doesn't leak important information
104. 7:49like personally identifiable information or confidential information from the organization.
105. 7:54The next type of AI debt is with the model itself.
106. 7:58So, the model, we need to make sure, if you're incurring technical debt, one way it happens is from no version control.
107. 8:06You just bring in a model and deploy it and go.
108. 8:09We're not clear necessarily when we're going to make an update to the model.
109. 8:14We're not clear when we do, what the plan is for doing that.
110. 8:18We're not clear if there's any way to do evaluation of the model itself.
111. 8:23We need to have metrics where we're seeing and judging against model drift and things like that.
112. 8:29And then we also need to have the ability to do rollback.
113. 8:32And in an AI technical debt situation, we might not have had any of those capabilities, no ability to roll back.
114. 8:39So now, if we find that there's an error once this thing is deployed, it's gonna cost us a lot to pull this thing back in.
115. 8:45I need to do penetration testing against this model.
116. 8:48Look at all the different attack types that could be and figure out if my model is vulnerable to those things.
117. 8:54If I don't, it's yet more technical debt that occurs.
118. 8:59Another class of technical debt is prompt technical debt.
119. 9:03The prompt is what the user's putting into the system if we're talking about a chat bot, for instance.
120. 9:08So, we may have an undocumented system prompt.
121. 9:12The system prompts the thing that you feed into the large language model, telling it what its context is.
122. 9:18And if that's not documented, we don't know what kind of a system we're gonna end up with in the long run.
123. 9:24So we need to make sure that we've thought through those kinds of things.
124. 9:28We might have no validation of the inputs coming in, the prompts that are coming in.
125. 9:33This can be not just input from the user, but it could be input into the system and change the behavior of the system.
126. 9:41It will override the system prompt and that's what we call a prompt injection.
127. 9:46So, we to not incur those kinds of situations.
128. 9:50Prompt injection is something that can absolutely result from going too fast and not checking for all the cases.
129. 9:57We could result in data leakage.
130. 10:00As I mentioned before, if we're not checking the information that went into the model,
131. 10:04well, then, we could have the model with sensitive information at minimum when we're having information exfiltrating,
132. 10:11coming out of this thing as a prompt response, I wanna make sure that I'm not leaking sensitive data.
133. 10:17So that means, and also, that if I've not put any guardrails in place, well then I'm gonna incur more technical debt.
134. 10:25That could come in the form of lawsuits and things like that.
135. 10:28So one of the things that helps here is putting an AI gateway between the user and the AI model and have it check the inputs that are coming in,
136. 10:36block the things that look like prompt injections or look like they might be getting around the guardrails
137. 10:40and also redact the information that's coming out that might be of a sensitive nature.
138. 10:47And then, our last category of technical debt is organizational technical debt.
139. 10:52In this case, we have basically what the organization sees from all of this perspective.
140. 10:58We're asking questions if we didn't decide who owns this thing.
141. 11:02We didn't decide what is the governance policy for this thing?
142. 11:06If I don't have a policy in place, then I don't know if I'm doing the right thing or not.
143. 11:11And I'm gonna figure it out later, and it's gonna cost more later.
144. 11:14So it's much better if we have also done red teaming, basically testing the system out and putting it through its paces.
145. 11:22If I don't plan properly, the system could have latency.
146. 11:26It could be slow to respond and it's not gonna, and we did it in the prototype, we moved fast, it all looked great.
147. 11:31Then we put it in the field and it didn't work great because all of a sudden everybody starts hitting it and it doesn't have the capacity.
148. 11:39It could end up costing us more if we didn't plan for this.
149. 11:42All of these things that I've talked about, there's a cost associated with remedying every one of them.
150. 11:48And another one is scalability.
151. 11:50We've got to make sure that the system will keep up with the demand that's out there.
152. 11:55If we don't do all of these things, they incur technical debt.
153. 11:59And in the end, we end up with an AI that we don't trust.
154. 12:03Ready, fire, aim.
155. 12:05Okay, we can all agree that's not the way to do it.
156. 12:08We need to be able to do ready, aim, fire.
157. 12:11So what does that look like in terms of an IT project, and in particular, an AI project?
158. 12:17Well, it's the basics we need to remember.
159. 12:19We don't get to skip just because it's AI.
160. 12:22We start with requirements.
161. 12:23We move then into architecture.
162. 12:26Then we go to implementation.
163. 12:28Don't start with that.
164. 12:29Then we move into testing it.
165. 12:33Then we deploy it.
166. 12:34Then we evaluate what we've done, and then we feed all of that back into our requirements.
167. 12:40Nothing about this changes.
168. 12:41That's still true, and it still applies to AI projects.
169. 12:45If you think about it this way, AI technical debt is what results from speed minus discipline equals this massive compounding interest on the debt.
170. 12:57And what we want to do is burn down that debt.
171. 13:00In other words, if you don't put enough money down on that home loan, you're going to be paying on interest forever, and the house is going to cost way more.
172. 13:08Another way to think of it?
173. 13:10If you don't know where you're going, any road will take you there.
174. 13:13Because if you don't, there's no telling where you' re going to end up, and worse still, there's not telling how much you're gonna owe for the experience.