Holo4: A Model That Clicks, Codes and Calls Tools — Transcript

Full transcript

1. 0:00Okay, so imagine you've built an agent
2. 0:02that can use a computer. It looks at a
3. 0:04screen. It can click. It can type. But
4. 0:07now you drop that into a server with no
5. 0:09screen. It's just going to get stuck.
6. 0:11It's got nothing to look at. It was
7. 0:13built for visual kinds of things. If you
8. 0:15flip it around, if you've got an agent
9. 0:17that's great at calling tools and APIs,
10. 0:20but you then ask it to pull a report out
11. 0:22of some old app or something like that
12. 0:24that has no API, it's also stuck, right?
13. 0:27It doesn't have the ability to look at
14. 0:28the UI. it doesn't have the ability to
15. 0:30actually act on that. And so far up
16. 0:32until now, people have generally tended
17. 0:34to make models to do one side of that
18. 0:37line or the other. So either being able
19. 0:39to do the visual kind of stuff and act
20. 0:42on things, click on things, that kind of
21. 0:44thing, or being able to actually do the
22. 0:46API related stuff, the non-visual kind
23. 0:49of stuff. Unfortunately though, the
24. 0:51problem is that real work doesn't stay
25. 0:53on one side or the other of that line.
26. 0:55More and more as people using agents to
27. 0:57do a whole variety of different tasks,
28. 1:00it often means that it's got to click
29. 1:01through some legacy UI. It's then got to
30. 1:04be cleaning a CSV file. It's then got to
31. 1:06be posting it to a Slack or to some
32. 1:08other API. And right now, the way that
33. 1:11people have sort of handled this is you
34. 1:13tend to have lots of different models.
35. 1:14You chain models together. So you've got
36. 1:17one visual model that you force
37. 1:18everything as screenshots. You've got
38. 1:20another model which is doing your
39. 1:21orchestration and dealing with MCPS,
40. 1:24etc. So in this video I want to talk
41. 1:25about a model from a company in Paris
42. 1:27called H Company and they're the lab
43. 1:29behind Hollow Models. They've just
44. 1:31released Hollow 4 and that's what I want
45. 1:33to go through here. This is basically an
46. 1:35openw weight model trained to click to
47. 1:38write to run code to call MCPs to do API
48. 1:42tools etc. And most importantly to be
49. 1:44able to orchestrate and choose between
50. 1:47those actions as it goes along. And I
51. 1:49kind of feel there are two really
52. 1:50interesting parts here. There's the
53. 1:52model itself, which we'll look at, but
54. 1:54the other part is how you train a model
55. 1:56to make that choice. And I should point
56. 1:59out they've actually got a family of
57. 2:00hollow models here. So, we'll have a
58. 2:02look at those. We'll have a look at how
59. 2:03they fit into an agent, how they're
60. 2:06built, and also perhaps where they're
61. 2:07falling short, etc. All right, so let's
62. 2:09recap what an actual agent loop is. You
63. 2:12basically give the agent a task. The
64. 2:14model looks at what's going on and it
65. 2:16decides on one action, right? Then the
66. 2:20harness, which is just the code around
67. 2:22it, executes that action. It does the
68. 2:24click. It runs code. It calls the tool.
69. 2:27And then obviously it captures what
70. 2:29happened, whether that's a new
71. 2:30screenshot, whether that's just some
72. 2:32return from an API. And it feeds that
73. 2:35back into the model along with the
74. 2:37memory of the previous steps around it.
75. 2:40And this loop just goes round and round.
76. 2:42Right? This is fundamentally the base of
77. 2:44what's going on here. I guess the key
78. 2:46thing here is that the model itself
79. 2:47never actually touches anything. It only
80. 2:49decides. The harness actually does all
81. 2:51the doing. And that matters more and
82. 2:54more nowadays. When we look at
83. 2:55benchmarks, we see that models can score
84. 2:58really differently depending on the
85. 3:00harness that are actually running with
86. 3:02it. Now for Hollow 4, they've actually
87. 3:04optimized this to work with three main
88. 3:06lanes of sort of actions here. The first
89. 3:09are GUIs, right? Graphical user
90. 3:11interfaces. The model looks at a
91. 3:12screenshot, then says, "Click here, type
92. 3:15this." And you can kind of think of it
93. 3:17as the agent using the computer, a web
94. 3:20device, a browser. The way that you do
95. 3:22it, it works on anything that a human
96. 3:24can use, but it's going to be slow and
97. 3:26it can often break when someone moves a
98. 3:28button or changes the UI, etc. The
99. 3:30second lane is code. So, the model
100. 3:33writes a script, runs it in a sandbox or
101. 3:35a shell, and it gets an output back.
102. 3:37That's fast, that's precise, but it only
103. 3:40works when you can actually write some
104. 3:42software to do the particular task,
105. 3:44right? It can't just easily write a
106. 3:46Python code to know which button to
107. 3:48press on a visual output. The third lane
108. 3:51relates a bit to the second lane, but
109. 3:52this is where you've got tools. So, MCP
110. 3:55servers, APIs, that kind of thing. The
111. 3:58model calls a function and it gets back
112. 4:01structured data. And in many ways, this
113. 4:03is the most reliable of the three here.
114. 4:04But obviously the challenge is it only
115. 4:06works if there is an MCP or an API or
116. 4:10some kind of tool that is actually going
117. 4:12to do what you want it to do. Now the
118. 4:14challenge has been for many people
119. 4:15running agents, especially local agents
120. 4:18that these three different lanes often
121. 4:21require either different models or
122. 4:23different setups with your agent
123. 4:25harness. If you got something that's
124. 4:27doing screenshots, that's going to eat a
125. 4:29huge amount of context compared to just
126. 4:31JSON files or a few lines from a
127. 4:34terminal output, etc. And for exactly
128. 4:36that reason, most people have tended to
129. 4:38ignore running something that could
130. 4:40actually do visual stuff just because
131. 4:42it's often not practical to do locally.
132. 4:44And going back to the example I talked
133. 4:46about in the intro, something like
134. 4:48exporting a report from some kind of
135. 4:49legacy app through a guey. You want it
136. 4:52to go through an e-commerce site and
137. 4:53find different things that are not
138. 4:55available through their API. Then you
139. 4:57want to take that, clean it up for a CS
140. 4:59file with a Python script and then post
141. 5:01it to an API like a Slack, etc. is
142. 5:04really the type of goal that Hollow 4 is
143. 5:06sort of working on here. They talk about
144. 5:08this as being working across interfaces.
145. 5:11It's not just that it supports MCP as a
146. 5:14feature. It's trained to pick the right
147. 5:17lane between those images, code, or
148. 5:19tools. And in many ways, that's what
149. 5:21sets it apart. The earlier hollow models
150. 5:24that they already trained were using
151. 5:26function calling. The big difference
152. 5:27here, though, is that they've trained
153. 5:29this model to do all three together.
154. 5:32Graphical user interfaces is a first
155. 5:34class option, code is a first class
156. 5:36option, and tools are a first class
157. 5:39option. They're not just add-ons. Now,
158. 5:41the big question for me was, would you
159. 5:43use this as your only model or alongside
160. 5:45of another one? And the answer is
161. 5:47perhaps kind of both. HR runs all their
162. 5:49benchmarks with hollow as the sole model
163. 5:51doing both the planning and the acting.
164. 5:54But you could also put this underneath a
165. 5:57bigger model for doing orchestration or
166. 5:59a different model if you wanted to do
167. 6:00that. H actually has their own harness
168. 6:03and SDK called high agents which runs
169. 6:06their hosted agents and can plug into
170. 6:08things like clawed code over MCP. So
171. 6:10your main model plans the job and hands
172. 6:12the screen heavy work to hollow and I do
173. 6:15think that may make sense for certain
174. 6:17use cases where you just want to use
175. 6:19this for doing the gooey stuff for doing
176. 6:21all the computer use agents kind of
177. 6:23stuff. All right, so let's look at what
178. 6:25age company actually shipped here.
179. 6:27They've got two main models here. the
180. 6:29Holo 427B which is a dense model built
181. 6:32on the Quen 3.827B
182. 6:35and then they've also got the Hollow 4
183. 6:3735B A3B. Right? So that's basically a
184. 6:39mixture of experts model with 3B active.
185. 6:42That one is actually still built on the
186. 6:44Quen 3.6 model. Along with those two
187. 6:46models, they've also released Holotron
188. 6:49Nano, which is the same recipe for
189. 6:51actually making the first two, but
190. 6:53applied to Nvidia's Neotron 3 Nano Omni
191. 6:56there. So, I made a video about that
192. 6:58Neotron 3 Nano Omni. A mouthful of a
193. 7:01name there, but you can check that out
194. 7:03if you want to see more some of the
195. 7:04reasons why people are using that
196. 7:06smaller model. All right, so the weights
197. 7:08for all of these are on HuggingFace.
198. 7:09They've got a bunch of different
199. 7:10quantizations, etc. They've also got
200. 7:12their own API if you want to run it
201. 7:14through that. The full Bflat 16bit
202. 7:17version here is 54 gigs. So, I'm going
203. 7:20to be running this on my Dell Pro Max
204. 7:22machine with an RTX Pro 6000 in here.
205. 7:25So, a big shout out and thanks to Dell
206. 7:26for sponsoring the compute for this. One
207. 7:29of the challenges that you want to think
208. 7:30about wherever you're going to choose to
209. 7:32run this is that screenshots fill up
210. 7:34your KV cache very fast. So, you need to
211. 7:37make sure that you're leaving plenty of
212. 7:39headroom specifically for that. Okay.
213. 7:42So, if we look at the benchmarks in
214. 7:43here, we can see that they're actually
215. 7:44comparing both the 27B dense version and
216. 7:48the mixture of experts version and the
217. 7:50original Quen 3.8 827B
218. 7:53along with a bunch of models that were
219. 7:55sort of frontier perhaps a week, two
220. 7:57weeks ago. And I got to say I feel for a
221. 7:59lot of these companies when you
222. 8:00basically benchmark what you're about to
223. 8:02release and then the Frontier Labs go
224. 8:04and release something completely new.
225. 8:06But honestly, I feel like the Frontier
226. 8:08ones are more just for a guide of what
227. 8:10sort of state-of-the-art for this thing.
228. 8:12Obviously, none of those we can run
229. 8:14locally. And obviously, if you're paying
230. 8:15for API usage with images on those, that
231. 8:18can get very expensive very quickly. So,
232. 8:21if we look at things like OS World,
233. 8:22they're really not that far behind the
234. 8:24Frontier models. In fact, you can see
235. 8:26the 27B here is actually a little bit
236. 8:28ahead of what now is the old GPD 5.5
237. 8:32model. Now, when it comes to OSWorld,
238. 8:34yes, they're quite a bit behind. But
239. 8:36again, don't just look at the accuracy
240. 8:38scores here. Really pay attention to the
241. 8:41costs, right? because while Opus 5.5 and
242. 8:44Astra are coming out on top, you can see
243. 8:46that the cost of running those compared
244. 8:48to the 27B is like seven times more
245. 8:52here. The other big takeaway I would say
246. 8:54from the benchmarks here is it's really
247. 8:56kind of fascinating just how far behind
248. 8:58the mixture of expert model is. So when
249. 9:00we look at something like OS World, we
250. 9:02can see that the 35BE
251. 9:05is scoring less than half what the 27B
252. 9:09dense model is. And while it's not as
253. 9:11dramatic as that for the other
254. 9:12benchmarks in here, there really is a
255. 9:14clear advantage to using the dense model
256. 9:16for these kinds of tasks. And I guess it
257. 9:18makes more sense that if we compare sort
258. 9:20of active parameters, we're comparing 27
259. 9:23billion to 3 billion. And it just does
260. 9:26seem that for images and getting
261. 9:28representations out of images, the 27
262. 9:31billion dense model just does that so
263. 9:34much better here. All right, so just
264. 9:35quickly, how do they actually train the
265. 9:37model to choose a latent like this?
266. 9:39because this is one of the parts that I
267. 9:40find really interesting here. So, H
268. 9:42company built what they're calling an
269. 9:44agentic task factory. It's a set of
270. 9:46pipelines that build like interactive
271. 9:48environments and tasks that can be
272. 9:51checked automatically. So, things like
273. 9:52screenshots of real websites,
274. 9:55open-source software using specific
275. 9:57webs, specific MCP servers, and even
276. 10:00things like desktop environments. And
277. 10:02the interesting thing there is that some
278. 10:04of those environments are kind of hybrid
279. 10:06meaning that you could take the state of
280. 10:08that for example like a website or
281. 10:10something and you could actually do it
282. 10:11through either a guey way by looking at
283. 10:14the website through screenshots or you
284. 10:16could do it perhaps by an MCP way or by
285. 10:18using some code to look for something on
286. 10:21the website. So over many of these
287. 10:23different tasks then the model can
288. 10:25actually solve the task by clicking or
289. 10:28by calling a tool and then reinforcement
290. 10:30learning sits over all of that and
291. 10:32rewards the model for whatever works the
292. 10:34best in that situation. So that's how
293. 10:37the model learns that clicking through
294. 10:38five menus is perhaps going to be the
295. 10:41slow way when there's an API sitting
296. 10:43there. And also vice versa that perhaps
297. 10:46using a screenshot to get a bunch of
298. 10:48detailed information when there is no
299. 10:50detailed API available is going to be
300. 10:52the way that the RL reinforces it and
301. 10:54makes the model better at things like
302. 10:56that. So the training itself is
303. 10:58supervised fine-tuning on about 127
304. 11:00billion tokens then creating two
305. 11:03separate RL experts which get merged
306. 11:05back into the one model. On top of that
307. 11:07the other thing that they did was
308. 11:08rebuild their actual harness to actually
309. 11:10fit in with this. They were then able to
310. 11:12basically tag sort of what failed and
311. 11:15have engineers review those that look at
312. 11:17like how could we improve the harness to
313. 11:19actually work with this kind of output
314. 11:21from the model. So it's interesting here
315. 11:23that some of the biggest wins were
316. 11:24reliable memory that can track sort of
317. 11:26hundreds of steps and it can relate
318. 11:29those steps together perhaps in a better
319. 11:31way because it's being trained on these
320. 11:33three different lanes. All right, so
321. 11:35that same recipe was used for the Quen
322. 11:373.8 dense model, the 3.6e and the
323. 11:40Neotron. And I think it's pretty clear
324. 11:42that what they're really focusing on
325. 11:43here is really strong post-training
326. 11:45pipeline rather than overcustomizations
327. 11:48for any one specific model. They're
328. 11:50really setting themselves up here for
329. 11:52when the Quen 4 models come out. And
330. 11:54Alibaba has actually recently confirmed
331. 11:56that there is going to be a Quen 4 27B.
332. 12:00So, I think everyone's got to be looking
333. 12:01forward to that. You could imagine a
334. 12:03hollow 4.5 or five that actually
335. 12:05improves on all of this even more. All
336. 12:07right, so let's take a quick look and
337. 12:09demo at how this actually works. Okay,
338. 12:11so you can see here this is going to use
339. 12:15the guey element of this. So, it's
340. 12:17basically going to the hugging face
341. 12:18page. We've given it the task of open
342. 12:21the Holo 4 collection and list out the
343. 12:24models precisions. And you can see that
344. 12:26sure enough, it's gone to different
345. 12:28elements there. It's actually finding
346. 12:30the collections, the Holo 4 collection.
347. 12:33And it will go on for quite a while to
348. 12:35do this. But you'll notice at the bottom
349. 12:37that everything going through here is
350. 12:40actually a guey task in there. So, let
351. 12:43me just stop this one and let's look at
352. 12:45some of the other ones. I've built a UI
353. 12:48that's using a little mini harness at
354. 12:50the back end and that harness we can
355. 12:52give it an MCP if we turn it on. We can
356. 12:55give it access to tools, access to code,
357. 12:58etc. So, I'm going to try going through
358. 13:00the first task. So, here it's basically
359. 13:03got a madeup admin panel so that this is
360. 13:06running locally. You can actually see
361. 13:08that this is a list of orders there and
362. 13:11it's using the guey in this one. We've
363. 13:13got the MCP turned off. So, it doesn't
364. 13:15have the ability to use the MCP anything
365. 13:18here. So, it's literally having to
366. 13:21navigate that web page and go through
367. 13:24and get stuff. Now, it's getting it and
368. 13:26then it's actually using code to put it
369. 13:28all together. All right. So, you can see
370. 13:31finally it comes back with the answer
371. 13:33out. In this case, we can see that it
372. 13:35basically downloaded the CSV file. It
373. 13:38then ran Python on the CSV file. It then
374. 13:40gave us our answer back. Now it started
375. 13:42out with the guey there. If we look at
376. 13:44the bottom, we can see it went 1 2 3
377. 13:46guey steps. Then it went to code and
378. 13:48then it gave us our final answer out.
379. 13:51Let's look at the exact same thing
380. 13:52though. Now if we give it an MCP on. So
381. 13:56now we've got the MCP is actually on. So
382. 14:00it doesn't need to go to all the UI
383. 14:02stuff now, right? It can actually just
384. 14:04go and say, "Hey, we've got a tool
385. 14:06export orders." And it will go through.
386. 14:09It's going to basically get the same
387. 14:12kind of thing and we've got the answers
388. 14:14out at the end as well here going
389. 14:17through this. And you can see that
390. 14:19obviously that was a lot less steps and
391. 14:21none of them were gooey in this case. It
392. 14:23was basically just the tool and then
393. 14:25code and then output there. If you
394. 14:28wanted to basically just do everything
395. 14:30where it's doing various clicks as it
396. 14:32goes through, you can see it did a click
397. 14:34on the overdue. It did a click on the
398. 14:36export CSV file. Again, we had multiple
399. 14:39guies there and then it's running code
400. 14:41before. It's going to go back to the
401. 14:42model and give us our final response
402. 14:45out. Here's an example that I put
403. 14:46together where it's basically now just
404. 14:49using code. So, it's able to go through
405. 14:52a file, pull out different information,
406. 14:55reconcile the two different order
407. 14:57exports here, and list any mismatches
408. 15:00that it's finding in there. And the main
409. 15:02thing to notice here is that it's just
410. 15:05using code this time. It's basically
411. 15:06just putting that all together and then
412. 15:08at some point here it's going to give us
413. 15:10our final result out. And you can see
414. 15:12actually we got some environment issues
415. 15:14going on there where it's actually ran
416. 15:15into some problems with that. You can
417. 15:18see that eventually it's starting to use
418. 15:19pandas. And then finally we get our
419. 15:22answer out. Now it took quite a long
420. 15:24time there. Right, that was a minute 37
421. 15:26seconds. I've obviously cut some bits
422. 15:28out for the recording, but you can see
423. 15:30that it's done each of these steps. And
424. 15:33that's the whole thing with this model
425. 15:34is that it can choose to just do the
426. 15:37guey and use just the graphical user
427. 15:40interface and then process those
428. 15:42screenshots and then work out where to
429. 15:44actually click on the UI. It can do the
430. 15:47code like we've seen in this one. And it
431. 15:50can use tools like the MCP if they're
432. 15:52available and it sees those in the tool
433. 15:55repository going into the model etc.
434. 15:57Okay, so just to finish up, another cool
435. 15:59thing that they've published are the
436. 16:00actual trajectories of what's going on
437. 16:03in here. So you can actually come in and
438. 16:05have a look at how these trajectories
439. 16:06actually went. Now, the big question to
440. 16:08sort of finish up on is should you use
441. 16:10this? Well, I certainly think it's worth
442. 16:12checking out here. Unfortunately, the
443. 16:14biggest issue with this, I would say, is
444. 16:16the license that while it's open weight
445. 16:18and you can check this out, this is
446. 16:20actually a non-commercial license,
447. 16:22right? Even though the Quen base model
448. 16:24underneath this is Apache 2.0 0 and even
449. 16:27though their hollow 3.1 models were
450. 16:30Apache 2.0, unfortunately, this doesn't
451. 16:32fall into that category. So, you can
452. 16:34experiment, you can prototype, you can
453. 16:36do research with this, but if you want
454. 16:38to ship a product, you're going to have
455. 16:39to go through H's API, or you're going
456. 16:42to need to contact them directly. Now,
457. 16:44one thing that I do think that they
458. 16:45deserve some credit for is that every
459. 16:47trajectory behind these benchmark scores
460. 16:50that they've put out are actually open.
461. 16:52So you could actually go through and
462. 16:54learn a lot from those if you were
463. 16:56trying to make your own model for doing
464. 16:57this. So for me the takeaway here is
465. 17:00that agents choosing between clicking,
466. 17:02coding, calling, using guies, those
467. 17:04sorts of things is where things are
468. 17:06heading. Hollow 4 shows us that that
469. 17:09ability comes from these training
470. 17:11environments, comes from reinforcement
471. 17:13learning, comes from basically
472. 17:15incorporating the model to work with a
473. 17:17specific harness as well. It also shows
474. 17:19us some interesting things about the
475. 17:21dense versuse gap in there. Anyway, as
476. 17:24always, I've put the links to the blog
477. 17:26and the models and even the trajectory
478. 17:28viewer in the description there. Let me
479. 17:30know in the comments what you think. I'm
480. 17:32really curious to hear from you guys.
481. 17:33How many of you are actually doing sort
482. 17:36of computer use or browser agents,
483. 17:39things like that, versus just sticking
484. 17:41to basically tools, code, MCPS, APIs,
485. 17:45that kind of thing. I've covered some
486. 17:46browser use agents in the past and I've
487. 17:48certainly been pitched a bunch of them,
488. 17:50but I haven't always been convinced how
489. 17:51actually useful they are over just using
490. 17:54a normal model and a custom harness.
491. 17:56Anyway, as always, if you found the
492. 17:58video useful, click like and subscribe,
493. 18:00and I will talk to you in the next
494. 18:01video. Bye for now.