What is Human In The Loop with AI? How HITL Shapes AI Systems — Transcript

Full transcript

1. 0:00Human-in-the-loop is built around a simple question.
2. 0:03If AI can perform a task, should it perform that task on its own or
3. 0:10does a human need to be involved somewhere in the process?
4. 0:13Because as AI systems get more powerful and AI agents can run for like hours at a time autonomously,
5. 0:19we have to decide how much we humans are willing to let the AI work unsupervised.
6. 0:27So broadly human-in-the-loop is the idea that humans should be embedded somewhere in an AI system's workflow.
7. 0:37And human-in-the-loop is actually a spectrum of human involvement.
8. 0:40So on one end, you've got human in the loop itself, kind of strict HITL.
9. 0:48This is where the system literally stops and waits for a person to approve before it proceeds.
10. 0:53So the AI makes a recommendation or maybe it performs some kind of subtask.
11. 0:58But then a person is going to need to make the final action.
12. 1:02Give the go command.
13. 1:03Say, hey, let's do it.
14. 1:06Now in the middle, you've got something called human-on-the-loop.
15. 1:12This is where the AI operates autonomously, but a human monitors the process and
16. 1:18has veto power to hit the kill switch or override the system if things go pear shaped.
17. 1:24And then on the far side of the spectrum, you have got human-out-of-the-loop,
18. 1:31which is full autonomy with no human involved at all.
19. 1:37So the AI senses, decides and acts, it's doing all of that without any human intervention.
20. 1:42And in some cases, the system operates at speeds where human intervention wouldn't be possible anyway.
21. 1:49So some quick examples of this.
22. 1:52So for human-in-the-loop, let's consider an example would be medical AI.
23. 2:00So this is medical AI that flags, let's say, potential tumors on an X-ray, but requires a radiologist to make the final diagnosis.
24. 2:08This is a high-stakes environment, where the cost of a false positive or false negative is extremely high.
25. 2:15For human-on-the-loop, well, given that I drove my car to the studio today on FSD,
26. 2:23I'm going to say that supervised self-driving is a good example of that.
27. 2:31So that is where the car is doing all of the work of steering and maintaining speed,
28. 2:37but the driver, me, is still required to keep my eyes on the road to make sure that I can take over if I need to.
29. 2:45I still have to pay attention rather than using my time to do something better
30. 2:49like watching Jeff Crume YouTube videos on the drive-in.
31. 2:53And then for human-out-of-the-loop, a good example of that is something that happens very fast.
32. 3:00So for example, high-frequency trading is a good sample of that.
33. 3:05Because these algorithms, they operate at such speed and scale that humans cannot
34. 3:12physically react before thousands of trades have already been executed and the markets moved on.
35. 3:18So we need to decide whether we need a human-in the loop and if we do, where in the process should we put them?
36. 3:31Because there are really three places you can inject human involvement into an AI system.
37. 3:39Those three places are at training time, at tuning time and also at inference time.
38. 3:50And each one of these, it solves a different problem.
39. 3:54So let's start with training.
40. 3:57Now before a model can learn anything, somebody has to label the data it learns from.
41. 4:03That's a pretty well-known technique in machine learning, and it's called supervised learning.
42. 4:12This is humans looking at thousands of examples and then tagging them.
43. 4:18So, for example, looking at an email and saying, ah, yes, I think this email is spam, or maybe it is looking at and image.
44. 4:30So perhaps it looks at this image here, and it says this image is a stop sign.
45. 4:39Now without those human provided labels, the model has no ground truth to learn from.
46. 4:46But labeling data like this, it's expensive, it's slow, it' s pretty boring for humans.
47. 4:53And for specialized domains like medical imaging or legal document review, you're going to need an expert to do it.
48. 5:00So what if instead of labeling everything you could label just the stuff that matters most?
49. 5:08And the process for doing that is called active learning.
50. 5:14Now in active learning, the model trains on a small labeled dataset like this,
51. 5:19and then, it looks at all of the unlabeled data that we haven't got to yet.
52. 5:26So here's my data set with some unlabel data in it.
53. 5:28And then it says well I'm pretty confident about, let's say, these examples here, but these other ones here,
54. 5:39I'm kind of 50-50 on I'm, I'm not sure what these things are. I need a human to tell me.
55. 5:46So the human effort is concentrated on the really hard cases that the model cannot reliably perform itself.
56. 5:54Now let's move on to tuning.
57. 5:57So, for example, let's talk about if we want to tune a large language model.
58. 6:04And we want to tune it to respond in ways that are helpful and harmless and honest.
59. 6:09Now just a raw pre-trained LLM can predict text, but it really has no concept of what a good response looks like.
60. 6:18And you can't really write a formula that says hey, LLM.
61. 6:21I want you to be like helpful, but if you could dial back the creepiness factor, that would be nice.
62. 6:28So what do you do when you can t really write the rule for something like that?
63. 6:32Well, you get the human to show it.
64. 6:35And that is a technique that is called RLHF, reinforcement learning from human feedback.
65. 6:43So in RLHF, a model generates two responses, let's call them A and B, for every prompt.
66. 6:51And a human looks at both, and it says hmm I think option A is better this time.
67. 6:57So the human isn't labeling anything, or they're not correcting anything.
68. 7:02They're specifying a preference.
69. 7:04And RLHF collects thousands of these preference pairs.
70. 7:09And it uses them to train something called a reward model.
71. 7:14And a reward model is a completely separate model that learns to predict which option humans would prefer.
72. 7:23And then the reward model is used to coach the original LLM through reinforcement learning.
73. 7:29So the human's judgment is effectively getting incorporated into the system without needing a human there at runtime.
74. 7:38And speaking of runtime, the third place you can put human in the loop is at inference time, the live system in production.
75. 7:47And this especially applies to agentic AI.
76. 7:50Because agents, they do more than generate stuff like text and images.
77. 7:54Agents take actions.
78. 7:55They execute code.
79. 7:56They modify databases.
80. 7:58So here are a few common patterns for this.
81. 8:01One of them is confidence thresholds.
82. 8:04The model scores its own certainty on every prediction.
83. 8:08And if that prediction drops below a set threshold, then it routes that to a human.
84. 8:14So the AI handles the easy 90% and the humans handle the uncertain 10%.
85. 8:20Then there are approval gates.
86. 8:23So the AI proposes an action and a human has to explicitly approve it before the system proceeds.
87. 8:29And well, that's how many coding agents work today before doing something like making a change to a file system.
88. 8:36And then escalation queues.
89. 8:39The AI handles routine cases end to end, but it flags edge cases for human review.
90. 8:45So training time, tuning time, inference time.
91. 8:50What has human in the loop done here?
92. 8:53Well, labeling has given the model knowledge.
93. 8:59Our preference tuning here, that has given the model judgment.
94. 9:05And runtime oversight, that has given the model guardrails.
95. 9:11Now human-in-the-loop isn't without its trade-offs.
96. 9:15So let's talk about a couple.
97. 9:18One is scalability.
98. 9:20Every human touch point is a human-shaped bottleneck.
99. 9:25So if a system processes thousands of decisions per second, the humans are
100. 9:30gonna need to be a bit strategic about where they insert themselves.
101. 9:33And there is also consistency.
102. 9:37Two people will label the same data differently because, eh, humans.
103. 9:42Fatigue, bias, subjectivity, these are very human traits.
104. 9:46But the goal of human-in-the-loop was never to keep humans in the loop forever.
105. 9:53It's to keep them there until the system earns enough trust to move along that spectrum we talked about
106. 9:59from human-in-the-loop to human-on-the-loop, to human-out-of-the-loop entirely in some cases.
107. 10:08That's the maturity curve of every AI deployment.
108. 10:13Even the most autonomous systems we have today, they started with human-in-the-loop somewhere.
109. 10:19The human taught it, the human tuned it, human monitors it.
110. 10:22Until it was ready to go alone, which means maybe one day I really will be
111. 10:27able to watch Jeff Crume light board videos on my drive to the studio.