Stop Telling Agents How — Transcript

Full transcript

1. 0:01[music]
2. 0:12>> My name is Dan. I am the creator of Open
3. 0:15Pros. This year is Raymond I'm joined by
4. 0:16Raymond and
5. 0:18Jose two fabulous teammates, co-workers,
6. 0:22just phenomenal creators and engineers
7. 0:24all around. I'll introduce them a little
8. 0:26bit in a bit. Uh
9. 0:28just a little bit of background about
10. 0:30what we're what we're about to show you
11. 0:32here.
12. 0:33Uh
13. 0:345 years ago
14. 0:35in April of 2021, I was named the
15. 0:40best prompter in the Eleuther AI
16. 0:43back rooms Discord, which was a skill at
17. 0:46the time that I thought would be
18. 0:47tremendously valuable for many years
19. 0:49afterward and actually it lasted I think
20. 0:52around like two or three years maybe was
21. 0:55the lifetime of of that skill.
22. 0:57And I say that because a lot obviously,
23. 1:00as you all know, has changed in the last
24. 1:025 years.
25. 1:03But one thing has not changed and
26. 1:06I've been to a lot of workshops today.
27. 1:08We've got a lot of
28. 1:10amazing talks coming up.
29. 1:12But the one thing that seems to be
30. 1:15ever present in in all of the work that
31. 1:18we do and and that actually doesn't
32. 1:20change
33. 1:21with the latest harness or the latest
34. 1:22model is we all still need to
35. 1:25communicate our intent
36. 1:27to the thing and no matter how smart the
37. 1:29thing gets,
38. 1:31there is still this challenge of how do
39. 1:33I get out of my brain the thing that I
40. 1:35want and and get it into the super
41. 1:38intelligence that is just going to carry
42. 1:40it away and and do everything for me. Uh
43. 1:42Open Pros is is a is a language, it's a
44. 1:45framework, it's a
45. 1:47started as a project based on my own
46. 1:50love of language and my own
47. 1:52strong belief that language is this
48. 1:55really powerful
49. 1:58force of creation and I mean that like I
50. 2:00know that's a little bit cliche, but
51. 2:02that's like in the very literal sense
52. 2:04now with with the capabilities that we
53. 2:06have. You can write a sentence and the
54. 2:09things just make that sentence true. And
55. 2:12so the question then becomes like what
56. 2:14is the sentence that you write now that
57. 2:17you have this tremendously powerful
58. 2:19machine
59. 2:21to make it true. And the skill of
60. 2:22prompting from 5 years ago looks very
61. 2:24different from the skill of what we like
62. 2:27to think of as contract creation or
63. 2:28contract composition
64. 2:31today. And when you were dealing with a
65. 2:32model that was just completing the the
66. 2:35next token, there was there was good
67. 2:38patterns and bad patterns for for
68. 2:40prompting.
69. 2:41A few years later, you were filling out
70. 2:43context by hand and you were taking rag
71. 2:45results and and tool calls and stuff and
72. 2:48search results and stuffing them into
73. 2:49into the context window and you call
74. 2:51that context engineering.
75. 2:53And then the harness started doing that
76. 2:54for you and you were building a harness
77. 2:56and and and you were you were you
78. 2:57thought of the harness as this this
79. 2:59machine that was stuffing context for
80. 3:01you.
81. 3:02And and in the last 8 10 months, we've
82. 3:05seen a number of papers come out that is
83. 3:08just pushing everything up another layer
84. 3:10of abstraction and and the harnesses
85. 3:12themselves are are changing out from
86. 3:15under you. We
87. 3:16saw a great presentation by Anthropic
88. 3:18this morning on on flows and that's this
89. 3:21idea of this like self-modifying
90. 3:23harness topology that is managing sub
91. 3:25agents in a way that just was not
92. 3:27thinkable even just a year ago. So, if
93. 3:30if
94. 3:31is solved and the context is now solved
95. 3:34and now the harnesses are now solved,
96. 3:35like what's kind of the next step here?
97. 3:37And that's what we'd like to talk a
98. 3:39little bit about today and show you some
99. 3:40of the stuff we've been we've been
100. 3:41building in in support of that.
101. 3:44I like to think of it as as
102. 3:46once you can name the thing that you
103. 3:48want,
104. 3:49any harness can self-shape, can fan out,
105. 3:52do any sort of loop to get you there.
106. 3:55And the question is what are the good
107. 3:57best practices for naming the thing that
108. 3:59you want.
109. 4:00And and we think of that as as
110. 4:02declarative. We're we're going to borrow
111. 4:04concepts from classical software
112. 4:05engineering. I'm a software engineer of
113. 4:0712 years. Jose and Raymond have been
114. 4:09around the block with this as well.
115. 4:11Jose was early Google Cloud and he's
116. 4:13informed a number of the ways we think
117. 4:15about this. Raymond is just a phenomenal
118. 4:18uh uh
119. 4:19He told me to He told me to introduce
120. 4:20himself as a as a vibe code
121. 4:22extraordinaire, but that's way
122. 4:23underselling his his talents. He's a PhD
123. 4:26and two-time founder and has been doing
124. 4:29harness engineering for basically as
125. 4:30long as anybody I know. Um and and so so
126. 4:34it's this idea of borrowing old
127. 4:36engineering patterns and applying them
128. 4:38into this new world. We want to write
129. 4:40our prompts compositionally. We want to
130. 4:42compose
131. 4:43composition over inheritance.
132. 4:45We want to write our prompts
133. 4:47declaratively rather than procedurally
134. 4:50or imperatively. We want to tell it what
135. 4:52what to do, not how to do it because the
136. 4:55harness in its infinite wisdom and its
137. 4:57infinite intelligence is going to figure
138. 4:59that out. And if we tell it how to do
139. 5:01the thing, now we're actually leaving
140. 5:03cost and and intelligence on the table
141. 5:05because we could run it for a cheaper
142. 5:07model. A more intelligent model can
143. 5:09figure out how to do the thing. So it's
144. 5:10this idea of write what you want,
145. 5:12declare what you want, the outcome, the
146. 5:14goal, the end state. We think of that as
147. 5:17a contract.
148. 5:18And and
149. 5:20write it in a way such that you can
150. 5:22compose these because when you build a
151. 5:23system, when when you build an open
152. 5:25framework like this, when you build a
153. 5:27system that has many parts, maybe across
154. 5:29your company or across your team, you
155. 5:31want to do things in a way that allows
156. 5:32things to elegantly compose and that
157. 5:34lets you manage
158. 5:37complexity in in a in a more local way.
159. 5:39Um
160. 5:40with that, I'm going to hand it over to
161. 5:42Raymond who's going to tell you a little
162. 5:43bit about how it works and then we're
163. 5:45going to show you some some demos after
164. 5:46that. We'll we'll we'll open it up to
165. 5:49questions throughout. So, and please do
166. 5:50raise uh your hand if you have
167. 5:52questions, interject, please.
168. 5:55>> Awesome. Thank you, Dan. Okay, so there
169. 5:59um
170. 6:00I wanted to just help you understand
171. 6:01Open Pros and put it in a quadrant. Um
172. 6:05so,
173. 6:06if there's only two things that you
174. 6:07remember, the two things that I want you
175. 6:09to remember are it's just markdown.
176. 6:12So, we like to sometimes call it it's a
177. 6:14programming language for agents. We
178. 6:16sometimes call it a framework, but all
179. 6:18of it is just text. It's just one giant
180. 6:21prompt. Um and then the second thing to
181. 6:24just help you figure out in your mental
182. 6:26bucket where this fits is it's event
183. 6:28driven. It's not chat. We're not
184. 6:30chatting with these things. Um
185. 6:31I'll chat with an agent that will make
186. 6:33Open Pros programs later, but if you
187. 6:35only remember two things about us,
188. 6:36that's it. It's it's all text and we're
189. 6:39doing these event-driven multi-agent
190. 6:41workflows, not chatting. Um you can it's
191. 6:44open source, you can run it locally with
192. 6:45NPX skills.
193. 6:47Uh I'll walk through the hosted version
194. 6:48that I've built uh run.pros.md, and uh
195. 6:52we also will do it for you. If you want
196. 6:54to reach out, uh we'll help you out.
197. 6:56So, uh I guess quick question, like how
198. 6:59many people in here are from the the
199. 7:01finance side of things?
200. 7:03Okay, that's good. How many are like AI
201. 7:06engineers or developers? Okay. Then how
202. 7:09many people um
203. 7:11are maybe more like PMs or like
204. 7:13professionals but and you're using AI
205. 7:15but you don't know how to code?
206. 7:17Okay, fewer but Okay, good. So, I've got
207. 7:19why should I care for those three
208. 7:21audiences? So,
209. 7:22for the small regulated industries, one
210. 7:25of the things that we've seen um some
211. 7:28banks and uh other financial services
212. 7:30companies be excited about is like what
213. 7:32I said before, it's all text. So, we
214. 7:34don't need the procurement department,
215. 7:36we don't need the IT department. You can
216. 7:38use whatever coding agent you want,
217. 7:39whatever inference you want, whatever
218. 7:41guardrails you already have, whatever
219. 7:42stack you have, we can actually like
220. 7:44help you um you know, create these
221. 7:47multi-agent workflows without deploying
222. 7:49anything to you other than some markdown
223. 7:51files. Um so, that's one of the the
224. 7:54angles or or benefits of it being all
225. 7:56text.
226. 7:57Um for the folks that are uh not writing
227. 7:59code themselves, uh basically, if you
228. 8:01can read logical English, you can read
229. 8:03Open Prose. So, we're going to very
230. 8:05specifically not get into the syntax of
231. 8:08the language. Technically, it is a
232. 8:09programming language and it has a
233. 8:11syntax, um
234. 8:12but it's really meant to be more um read
235. 8:15by people and edited by people rather
236. 8:18than like handwritten from scratch by
237. 8:20people. Like you we don't want users
238. 8:23concerned with like learning the schema.
239. 8:25And so, you can kind of describe the
240. 8:27work, have your agent write the Prose
241. 8:28program,
242. 8:30you know,
243. 8:31that might give you some more ideas for
244. 8:32how you might want to edit it, or you
245. 8:33might see something you want to change,
246. 8:35uh and then that Prose program becomes
247. 8:37the reusable workflow.
248. 8:39Um
249. 8:40and then for the AI engineers and the
250. 8:43developers,
251. 8:44uh I think the angle where you might
252. 8:46want to use this is
253. 8:48it's you know, it's
254. 8:50not it's a framework that's not a
255. 8:52framework, and we'll talk a little bit
256. 8:54about that in a second, but even the
257. 8:56compiler for the framework, even the
258. 8:58virtual machine for the framework, is
259. 8:59itself just a giant prompt, and you're
260. 9:01literally like incepting cod code into
261. 9:04compiling the program for you, incepting
262. 9:06Codex into doing that. Uh so, there's uh
263. 9:09you can express very, very complex
264. 9:11multi-agent workflows that uh things
265. 9:14like the Anthropic um
266. 9:17uh dynamic What do they call them?
267. 9:18Dynamic agent workflows
268. 9:20um would probably be like an analogy.
269. 9:22You can do that in any coding agent in
270. 9:24just our markdown contract.
271. 9:27Uh okay. So, first on this idea that
272. 9:29it's just markdown, um
273. 9:31this can be a complete program, just as
274. 9:33an example. And again, I don't want to
275. 9:35get uh
276. 9:36so hung up in the syntax, um but you can
277. 9:40basically say, "Hey, here's the goal and
278. 9:42like here is the the contract that that
279. 9:45the that this maintains. Like here's the
280. 9:47definition of what this contract
281. 9:48maintains." Um and again, that's just
282. 9:51like a very silly simple one. Um
283. 9:55the uh
284. 9:58this idea here of like, well, what is it
285. 9:59that you can declare? So, again, I don't
286. 10:02want to get into the schema and the
287. 10:03types and the syntax, but what you or
288. 10:06your agent can declare on behalf of you
289. 10:09with uh in the CLI is called the pros
290. 10:11write function. In our app, it's
291. 10:12basically just the agent that drives the
292. 10:15whole app for you. You can make sub
293. 10:17agents.
294. 10:18Uh you can assign specific skills uh to
295. 10:21those specific sub agents. You can have
296. 10:23specific tools that are available. So,
297. 10:25for example, for search, I love Exa. I
298. 10:27think they're amazing and I don't want
299. 10:29it doing regular web search. I always
300. 10:30want it using Exa cuz it's faster and
301. 10:32cheaper for me. You can say, "Here's the
302. 10:34skill for that. Here's the CLI for
303. 10:35that." If you can't find those things,
304. 10:37just don't run the program. Like we're
305. 10:40not doing it the other way. So, you
306. 10:41might have skills that are specific to
307. 10:43your company or to your org. And then
308. 10:45essentially, you can think of it as like
309. 10:47wiring that program together and like if
310. 10:50it doesn't have all the dependencies,
311. 10:51like do not compile, right? Um and then
312. 10:55obviously that what are the inputs of
313. 10:56this program, what are the outputs that
314. 10:58this uh program has to deliver, this
315. 11:00contract has to deliver.
316. 11:02Um and again, like again, this could be
317. 11:05something if people want to nerd out and
318. 11:06ask Dan about how he built it. Like the
319. 11:08runtime itself is just a giant prompt,
320. 11:10too. So, um you know, some of my
321. 11:13favorite sentences in the uh in the
322. 11:15skill or the prompt is like "You are the
323. 11:18compiler."
324. 11:19This is not an analogy. You are the
325. 11:21compiler. And uh it seems to work. Um
326. 11:26So, uh yeah, and then this idea of a
327. 11:29event-driven. So, we want to get out of
328. 11:32this idea that everything is a
329. 11:33conversation, that everything is a chat,
330. 11:35um and build event-driven systems.
331. 11:38So, um
332. 11:40again, like
333. 11:42uh
334. 11:42just to compare these different workflow
335. 11:44trees, it's like something something
336. 11:46happens that might be just like a cron
337. 11:47job on a schedule, that might be a new
338. 11:49issue is opened in GitHub, or a new PR
339. 11:52uh is filed, or you have some webhook
340. 11:54system, right? Um that you're I don't
341. 11:56know, Stripe
342. 11:58isn't working, whatever, then you can
343. 12:01now um you know, trigger that workflow
344. 12:04that you've expressed
345. 12:06uh to go handle that entire thing end to
346. 12:08end and deliver on the contract.
347. 12:12Um and so, this idea of like starting to
348. 12:14build towards jobs, so you can set this
349. 12:16up yourself with your own agents in your
350. 12:18own system, and what I built um in
351. 12:20run.prose.md
352. 12:22is um this idea of like a job is a
353. 12:25contract plus a trigger. So, again, the
354. 12:28the trigger can be something super
355. 12:30trivial like a schedule, and so here I'm
356. 12:32basically saying like, "Okay, every day
357. 12:35uh run this."
358. 12:36Um but uh right now, the two the two
359. 12:39main triggers that we have implemented
360. 12:41in the app as is are um our webhooks,
361. 12:44which really
362. 12:45uh inevitably cover a lot of different
363. 12:46things, like a whole Slack app is just
364. 12:48webhooks, GitHub just webhooks. Um and
365. 12:52cron jobs. We're happy to add more if
366. 12:54you want to give us feedback on ones
367. 12:55that we can make easier for you. Um and
368. 12:59again, from from that trigger, then you
369. 13:01triage uh these workflows. So,
370. 13:04uh you can start to build out these, you
371. 13:05know, code factories from um
372. 13:08creating these standing jobs
373. 13:10um
374. 13:10of the Prose programs.
375. 13:13Uh what I want to show and I'll get
376. 13:15towards in a second is
377. 13:17um
378. 13:18one of the things that I've been
379. 13:19building to make sure that's ready in
380. 13:20time for this event is
381. 13:22um is webhooks and using uh GitHub uh
382. 13:26events specifically, and I've been
383. 13:29forcing myself to use the program to
384. 13:33develop the program.
385. 13:34And so, one of the examples that we now
386. 13:36have in the issue in the system is,
387. 13:39well, there's two different main ones.
388. 13:40One is like any GitHub issue that comes
389. 13:42in, you write a prose program for how
390. 13:44it's going to try to address the issue.
391. 13:47So, you can make that as complex or as
392. 13:49simple as you want. You can run
393. 13:51it could be a multi-agent workflow. It
394. 13:53could have specific, um, skills that it
395. 13:55has to follow or use models.
396. 13:58Um, and so, basically the input is
397. 14:00whatever that issue was, the output is a
398. 14:02commit. And it has to go all the way to
399. 14:04getting that thing done for you. Uh, and
400. 14:06then similarly, uh, and I'll show this
401. 14:07one more specifically is like doing it
402. 14:09with, uh, PR reviews. So, you can do
403. 14:12this event-driven kind of adversarial
404. 14:14review. And I don't know if you guys
405. 14:16heard, um, in one of the things I
406. 14:18thought was interesting this morning in
407. 14:19the Anthropic workshop where they're
408. 14:20talking about um, the challenges with
409. 14:23self-review. So, like, you're you're in
410. 14:25Claude Code and you're like,
411. 14:27"All right. Here,
412. 14:29I'm going to let you push to production,
413. 14:30but just could you just check like one
414. 14:33more time for me?" And like, it's it's
415. 14:35biased to, you know, think that it did a
416. 14:38good job. And so, so are humans, right?
417. 14:40And so, um, I'll show a workflow where
418. 14:43essentially the agents that are building
419. 14:45this get this backpressure from the app
420. 14:47itself
421. 14:48um, through through pull requests.
422. 14:51Um, okay. So, while we transition to
423. 14:55this, are there any questions now while
424. 14:57we switch over to the
425. 14:58to the web app?
426. 15:02>> Please, no question is too silly.
427. 15:05Is that a hand or a phone? I can't see
428. 15:06the lights here.
429. 15:09>> Maybe none.
430. 15:10>> Yeah, go ahead.
431. 15:10>> One over there.
432. 15:11>> [clears throat]
433. 15:12>> It's a great question. Uh, I'll field
434. 15:13this.
435. 15:14>> Yeah, you should take that.
436. 15:15>> Yeah.
437. 15:16Uh, so, we actually have this idea of
438. 15:20earlier I alluded to the what and the
439. 15:22how being two two different two
440. 15:23different ideas. Declare the outcome and
441. 15:26then the how is a little bit you you
442. 15:27leave it open, let the agent figure it
443. 15:29out.
444. 15:30On that spectrum, you actually do, to
445. 15:33your point, at at certain places want to
446. 15:37weigh in on the how. And in fact, we
447. 15:38have a
448. 15:40a feature a
449. 15:41syntax that lets you pin the how. And so
450. 15:45you can initially declare it very loose.
451. 15:47You can let a very intelligent agent run
452. 15:50and figure out how to get to the thing
453. 15:52that you've declared. And then once it
454. 15:55does that, you can say here's the agent
455. 15:56trace.
456. 15:58Let's take this trace and we will sort
457. 16:00of codify it or or I don't know, shrink
458. 16:02wrap it
459. 16:03into a how. And what that allows us to
460. 16:05do is actually drive down the the the
461. 16:08cost of completing the workflow. So you
462. 16:10you start with a very expensive model,
463. 16:12figures it out, and then you sort of, as
464. 16:15you say, make it more specific, more
465. 16:17deterministic. So you're kind of like
466. 16:19shifting the the the the the spectrum
467. 16:21from
468. 16:22totally at just a what to to how how to
469. 16:26do it. And and the cost comes down as
470. 16:28you do that.
471. 16:29So when we say it's all markdown, that's
472. 16:31where you can start it, but then you can
473. 16:35move your way to to uh more specific
474. 16:39procedural imperative style instructions
475. 16:42as well.
476. 16:43>> Yeah, I guess a couple comments on that.
477. 16:45So one, we do have a system where you
478. 16:46can take like a golden session and like
479. 16:50the like say make this into a reusable
480. 16:52workflow. Like that was the best cloud
481. 16:54code session I've ever had. Like I
482. 16:56wanted to do cuz that's this is we get
483. 16:58this all the time, right? And I think
484. 17:00there's there's a part of it where like
485. 17:02these things are non-deterministic.
486. 17:04I think a lot of the
487. 17:06quote-unquote pinned and stable
488. 17:08endpoints people have shown actually are
489. 17:10changing as all these companies are
490. 17:12compute constrained, but I also just
491. 17:14think there's this like changing
492. 17:15expectations thing, right? Which is like
493. 17:18the first time you tried like whatever
494. 17:20it was, like Fable or whatever, like
495. 17:21blew your blew your mind and you're
496. 17:23like, I want whatever. This workflow is
497. 17:25so amazing. I want this every time. And
498. 17:27then you kind of like boil the frog and
499. 17:29like have just different expectations
500. 17:31and all of a sudden doesn't happen again
501. 17:33and like some of that is just
502. 17:34reliability. And so so one thing you can
503. 17:36do is like take a really good session
504. 17:39and like codify it. But then the other
505. 17:41thing, just to give Dan a little bit
506. 17:42more credit of like building the
507. 17:44framework is like it's not just
508. 17:46literally like the markdowns you see are
509. 17:48like the tip of the iceberg of what it's
510. 17:51seeing. So like the last time I checked
511. 17:53the full like system prompt or skill or
512. 17:57however you
513. 17:58there's I mean it is a skill it is a
514. 17:59system prompt is like close to 70,000
515. 18:03tokens, 80,000 tokens of like
516. 18:05>> longer. We're trying to clean it up.
517. 18:07We're trying to
518. 18:07>> Yeah, well now that the agents are
519. 18:08smarter we can make it a little bit
520. 18:10shorter. But we basically have this full
521. 18:12type system that's like we're overriding
522. 18:14markdown with a very very clear type
523. 18:16system and a very specific prompt for
524. 18:18how the agent is going to deliver this
525. 18:20incredibly complex multi-agent um
526. 18:23product on the out um but maybe to be
527. 18:26explicit about where we don't mess with
528. 18:27it is like
529. 18:29um
530. 18:30So like Claude code when it runs it it
531. 18:32has its own system for sub-agents. We
532. 18:34don't want to mess with that. We don't
533. 18:36want where we don't want our prompt to
534. 18:38be um
535. 18:39in any way like specific to Claude code
536. 18:41cuz Codex has a totally different like
537. 18:43set of sub-agents. So the the we're like
538. 18:47I don't know, a meta harness, like
539. 18:49>> [laughter]
540. 18:49>> whatever. We're one layer we want you to
541. 18:50bring your own harness. We don't want to
542. 18:52be the harness. We want to be this uh
543. 18:54thing that is compiled by the harness.
544. 18:56So I guess that's a couple pieces of
545. 18:58>> Does that answer your question?
546. 18:59>> um
547. 19:01All right. So if you want to try it,
548. 19:03it's um it's here the QR code
549. 19:04run.prose.md uh the QR code goes to
550. 19:07that. If you want to start playing
551. 19:08around with it. Um what I want to do
552. 19:11in
553. 19:12um showcasing this is to
554. 19:17uh show off a couple of things. So,
555. 19:20the um the first thing is that um
556. 19:25the So, this is
557. 19:28the This is a Prose program in the edit
558. 19:30mode, right? It is technically Markdown
559. 19:33and Dan's overridden like the H3 headers
560. 19:36with all the types of this thing.
561. 19:39But, getting someone to like learn that
562. 19:42syntax and write it
563. 19:45from scratch, it's kind of a
564. 19:48not really the point. Um and so, what we
565. 19:53um what we want to do is So, even though
566. 19:56everything that executes this program is
567. 19:58here, and in fact, I've for the real
568. 20:02nerds in the audience, you can see that
569. 20:04I've over I've taken the front matter of
570. 20:06the Markdown file, and that is actually
571. 20:08all of the stuff that's happening in the
572. 20:10cloud. So, I could take this program
573. 20:12and export it as a pure Markdown file
574. 20:15and go send it to my friend, and they'll
575. 20:17it'll upload in their cloud this the
576. 20:19exact same way
577. 20:21um to keep it like composable and keep
578. 20:23it true to that idea of that we're pure
579. 20:24Markdown.
580. 20:26Um
581. 20:26anyways, the whole idea is like even
582. 20:27though that is the thing and we'd want
583. 20:29you to be reading this thing, it's not
584. 20:31like the entry point. So, what I've
585. 20:33built instead is um an agent which
586. 20:38hopefully will
587. 20:40talk to me here.
588. 20:42>> Hi, it's Prose. What are you hoping to
589. 20:44get done today?
590. 20:45>> Okay, so uh every time I have a new pull
591. 20:49request, I want you to um write me a
592. 20:53Prose program in my run-prose repo that
593. 20:57will review it as if you are a a grumpy
594. 21:01senior developer who doesn't like
595. 21:05AI-generated code.
596. 21:08>> Sure, I can do that. Hang on.
597. 21:11>> Okay.
598. 21:12So, I'm going to mute myself.
599. 21:15So, it's it's talking. It's going to try
600. 21:17to figure this out. And the the whole
601. 21:20idea is like anything
602. 21:24So, anything that can charge money or
603. 21:26spend money, you have to confirm. I
604. 21:29think that's like a best practice. Um
605. 21:33Oh, yeah.
606. 21:35Great. So, I had to confirm cuz that
607. 21:37that action costs money for it to So,
608. 21:40that's the pros rate function. So, it
609. 21:41basically takes my query and is going to
610. 21:44generate a new program um from that
611. 21:47including all the wiring. This step is
612. 21:48going to take a little bit longer. So,
613. 21:50while that's running, I'll show like the
614. 21:51manual way of doing that. Which is in
615. 21:54the jobs tab. I don't know why this is
616. 21:56cutting out. Um in the jobs tab, we can
617. 21:59go here, make a new job. The the uh on a
618. 22:03schedule is just a cron job. We have a a
619. 22:06web hook that's open-ended. So, like you
620. 22:08can have yourself or your agent um
621. 22:11either use our specific web hook or like
622. 22:13wire our thing to listen to your web
623. 22:15hook. But for Git GitHub, um
624. 22:17so, I'll do the same one. So, for GitHub
625. 22:19pull requests, we make that easy.
626. 22:22Um we're going to do it in the run.pros.
627. 22:24We're going to
628. 22:25uh I've just pre-wired this contract.
629. 22:27It's like, "Okay, you're going to review
630. 22:29my thing." And then I can pick, you
631. 22:31know, what
632. 22:32uh model that they have. We'll we'll add
633. 22:34more uh later. Right now, we have the
634. 22:36the like top three from
635. 22:39uh OpenAI.
636. 22:40Uh and then I would just uh
637. 22:42create that and deploy that. So,
638. 22:46then the other thing while this um
639. 22:49this one's working that I want to show
640. 22:50is how I've actually been using that as
641. 22:52back pressure. So, the main thing
642. 22:55Don't get too hung up on the um
643. 22:58whatever, the content up here. But the
644. 22:59thing that I want to show you here is
645. 23:02like the
646. 23:04the the volume. So, like in
647. 23:07whatever that is, like this entire thing
648. 23:09is goes back to 3 days ago, but then if
649. 23:11I go here, like these are all still from
650. 23:133 days ago or 4 days ago.
651. 23:15Or so, like in the span of a few days,
652. 23:18this is doing like dozens and dozens and
653. 23:20dozens of PRs, and every time I'm asking
654. 23:23for a new feature about this GitHub was
655. 23:26webhooks feature that I wanted to have
656. 23:28ready for this event, uh every single
657. 23:31one of those is firing off like a series
658. 23:33of review agents where it's like so some
659. 23:35of them are simple like okay, this one
660. 23:37clearly seems to have worked right away.
661. 23:40So, there's only one round. Uh some of
662. 23:42these had,
663. 23:44you know, 14
664. 23:46rounds of review before
665. 23:48uh it felt like the reviewer agents were
666. 23:50like aligned and it let this thing pass.
667. 23:53Um and even just today, like I noticed
668. 23:55something that I didn't love about um
669. 23:58how it was doing it, and I was able to
670. 24:02um you know, basically say okay,
671. 24:04um on our staging server, you know,
672. 24:06don't mess with production. I'm about to
673. 24:07give a talk. On the staging server, I
674. 24:09want you to like change the things the
675. 24:12exact ways in which this will get
676. 24:14triggered, but you have to use the
677. 24:15system yourself. So, um like both our
678. 24:18dev version of this and our prod version
679. 24:20of this are giving different reviews. Um
680. 24:22and now the agent that's running
681. 24:25uh here,
682. 24:27uh which we can see all the different
683. 24:29jobs in the the runs tab here,
684. 24:31essentially like cannot consider its job
685. 24:34to be complete until it's like um
686. 24:38a a like um like my developer agents
687. 24:41have like have to take that feedback
688. 24:42into account.
689. 24:43>> And just to emphasize like what's
690. 24:44different about something like this from
691. 24:46any other chat-based or or agent-based
692. 24:49flow that you've seen is all of these
693. 24:51are getting compared against a contract.
694. 24:53So, you could think of it as a fresh
695. 24:55session every time is getting
696. 24:56benchmarked against this contract that
697. 24:58you've written declaratively, which as
698. 25:00as Raymond alluded to earlier, gets the
699. 25:03models out of this like self-confidence
700. 25:06loop where you don't want the same model
701. 25:07checking the thing that was done. So,
702. 25:09you're spawning new sessions and and
703. 25:11bouncing them off of a contract, which
704. 25:13is I just a better uh pattern for for
705. 25:16how to do it. Yeah, go ahead.
706. 25:16>> Yeah.
707. 25:18>> But, doesn't [clears throat] it seem
708. 25:18like a great opportunity for one-shot
709. 25:19classifiers like Jeff?
710. 25:22>> Yeah. So, they um
711. 25:25>> We have a lot of good stuff in the pipe.
712. 25:27So, so the idea of checking against a
713. 25:29contract and and verifying it. Sorry,
714. 25:31Raymond.
715. 25:32>> Yeah.
716. 25:32>> Uh it it is super exciting on these
717. 25:34decision models where uh you you have a
718. 25:37contract, which is some ideal state, and
719. 25:40then you have your messy real-world
720. 25:41state, and you can apply a general class
721. 25:44decision model like like Jeff to say, is
722. 25:46this is this passing or not?
723. 25:48>> So, I think
724. 25:49>> in this, but it's coming.
725. 25:50>> Yeah, and I think I think without um
726. 25:53like
727. 25:54Dan, again, we're going a lot off
728. 25:56script, but it's a great question, and
729. 25:58uh the great Jeff-ification is upon us.
730. 26:01So, we need to speak to this. Um
731. 26:04is uh I think Dan and I both
732. 26:07been thinking about this a lot and
733. 26:08realized that like
734. 26:10um
735. 26:11OpenPros itself, the language that's all
736. 26:14markdown, I think does not have any
737. 26:16place in
738. 26:18the Jeff land because they're not
739. 26:20generative. Like, so the whole idea of
740. 26:24the lang- like it generating a language
741. 26:26and understanding a language, like it
742. 26:29uh it's not that. But, I think the the
743. 26:32two places where we think it would fit
744. 26:33really well, and in particular with the
745. 26:35hosted run.pros, is one on the back end.
746. 26:39So, the the downside or the weakness of
747. 26:42OpenPros is it is you are letting the
748. 26:44agent say, "I have met the contract.
749. 26:46Here it is." Right? And so, I'll often
750. 26:48say to people
751. 26:49uh like the intent behind OpenPros is
752. 26:53similar to something like DSPy where you
753. 26:55want to have like this abstraction layer
754. 26:58over which you can express very complex
755. 27:00workflows.
756. 27:01The implementation is the exact opposite
757. 27:03of DSPy. DSPy is like every LLM call is
758. 27:06a black box and you
759. 27:08you isolate the crap out of that thing.
760. 27:10It's all deterministic code. It's all
761. 27:12typed interfaces to like only let the
762. 27:14black box do that thing. Dan's approach
763. 27:17with open prose is the exact opposite
764. 27:18which is like AI is getting smart enough
765. 27:20to be the compiler. Like let everything
766. 27:23be in fuzzy land.
767. 27:25So the the the weakness of it though is
768. 27:27like if you run it on Haku or you run it
769. 27:29on Fable you're going to get a very
770. 27:30different result and and you're also
771. 27:32letting it say I'm done. So I think a
772. 27:34very natural place for like uh Jev or
773. 27:37these new like decision models or
774. 27:39whatever we want to call them system one
775. 27:40models is like you were saying like as a
776. 27:43verifier on the end of the job like very
777. 27:45very fast and cheap. Um and then the
778. 27:48other thing that I want to build that's
779. 27:50relevant to this this is like a router.
780. 27:52So like why would I have to go click the
781. 27:55button that like oh issues go to the
782. 27:57issue job but PRs go to the PR review
783. 27:59job. Why don't I just say like
784. 28:01okay all the GitHub events that come in
785. 28:04like you know solve the issues review
786. 28:07the PRs right?
787. 28:09Another question.
788. 28:10>> Don't give them any more ideas about the
789. 28:11game.
790. 28:11>> Okay.
791. 28:12>> I know I was going to say I was going to
792. 28:13>> [laughter]
793. 28:14>> Thank you.
794. 28:15It's just topical on the Twitter sphere
795. 28:16at the exact moment that like why
796. 28:18wouldn't you just RL this? Like if I got
797. 28:19all these beautiful runs here against
798. 28:21the contract why am I not just RLing
799. 28:22this?
800. 28:22>> So
801. 28:23So one of the ideas back to the language
802. 28:25question is one of the ideas of like why
803. 28:28why are we doing this at all in
804. 28:30language? Like the original human side
805. 28:32of this is like
806. 28:33you can commit it to get and you can
807. 28:35version control it or you can
808. 28:37you know use it in notion or or Jira and
809. 28:40and have anybody on your team using it.
810. 28:42But from from a engineering side it's
811. 28:45the fact that it's all text is like
812. 28:47perfect and beautiful for for RL. So,
813. 28:50we're not doing that right now, but like
814. 28:52that's absolutely something we're
815. 28:53talking with various prospects about is
816. 28:56like once you have all of your stuff
817. 28:58written as text, this is like
818. 29:00>> Yeah.
819. 29:00>> a whole lot better for for RL.
820. 29:02>> Yeah, we haven't hit volumes where this
821. 29:04is important, but like if you are
822. 29:07hosting the model and you have this
823. 29:09100,000-ish token prefix, then you can
824. 29:11like just it or you like, you know,
825. 29:14build these um
826. 29:15more token efficient representations.
827. 29:17So,
828. 29:18um
829. 29:19Yeah, if someone wants to do a trillion
830. 29:21tokens in Open Pros, we'll happily just
831. 29:23it for you. Um So, anyways, to go back,
832. 29:26thank you. Those are Those are really
833. 29:27good ones. We now have our grumpy senior
834. 29:30pull request review.
835. 29:32Um and again, like the key idea is so
836. 29:34the agent did all that for us. The key
837. 29:36idea here is um
838. 29:38uh like the by deploying this job, like
839. 29:41this will now start spending money when
840. 29:44whenever we get a new PR, so I have to
841. 29:46click that. So, then I'm going to
842. 29:47confirm that. And then once this is
843. 29:49deployed, it'll go when I go back over
844. 29:51to my jobs, it will um
845. 29:54it will look exactly like uh let's see.
846. 29:57So, if we just Oops, I did
847. 30:00switch that.
848. 30:02Once this loads, we should see
849. 30:06and the grumpy one is at the top or the
850. 30:07bottom.
851. 30:10Yeah, here we go. Grumpy AI skeptic PR
852. 30:12review. So,
853. 30:13um you can see I was practicing earlier.
854. 30:16Um
855. 30:18Okay. So,
856. 30:20let's see what in terms of this. I think
857. 30:22those are the main things. Um
858. 30:24Yeah, I'm in charge of run.pros.md, so
859. 30:26if you run into any issues, don't tell
860. 30:28my boss, you just send it straight to
861. 30:30me.
862. 30:31Uh and my agents will uh take care of it
863. 30:33for you.
864. 30:34Um
865. 30:35but yeah, I'm really excited about this
866. 30:37as just like a a simple, easy-to-use um
867. 30:40um way for having these like event-based
868. 30:42systems. If you want to see other
869. 30:43triggers or other features then that's
870. 30:45really fun for me. But again, you don't
871. 30:47need to use the hosted thing at all and
872. 30:49like you can uh
873. 30:51you can use the um the skill instead NPX
874. 30:53skills add open prose prose. Um but now
875. 30:58uh with that Dan is going to um
876. 31:01go through a specific example that is
877. 31:05for [clears throat] our finance friends
878. 31:08um and we're going to show that one
879. 31:11uh in
880. 31:13Codex and then he'll use that as an
881. 31:14opportunity to talk a little bit more
882. 31:16about how all this works under the hood.
883. 31:19>> Yep. Thank you.
884. 31:20>> Sure.
885. 31:21>> Um so
886. 31:24getting that working in the cloud is is
887. 31:26interesting. One of the more interesting
888. 31:28use cases that we've been talking about
889. 31:30with various customers and prospects is
890. 31:34um
891. 31:35you can this is
892. 31:37this is a framework or a language or
893. 31:39pattern that you can build in house and
894. 31:41you can use you can run it on prem. The
895. 31:43whole idea here is that you can run it
896. 31:45on prem. You don't need to use our cloud
897. 31:47version of it. We'd love you to try it
898. 31:49out in the cloud and and and and play
899. 31:51with it get a get a feel for that. Um
900. 31:53but if this is something that you want
901. 31:55to implement uh inside your organization
902. 31:58the most interesting use case that we're
903. 31:59seeing is on top of open models on top
904. 32:02of your own harness bring your own
905. 32:03harness and then use this as kind of the
906. 32:06application layer on on top of that and
907. 32:08that lets you build this full stack in
908. 32:10house. You don't have to ship any of
909. 32:12your data out to the labs. More
910. 32:13important than your data, you don't have
911. 32:14to ship out your institutional
912. 32:17We like to think of open prose actually
913. 32:19as like institutional operational
914. 32:21knowledge as documentation, right? Think
915. 32:24of it as like standard operating
916. 32:25procedures. So even more important than
917. 32:27your data is you don't want to be
918. 32:28shipping your standard operating
919. 32:30procedure out to the big labs.
920. 32:32Um and so so this fits really nicely in
921. 32:34this stack of of open open source
922. 32:37application layer
923. 32:39sits on top. We we publish this open
924. 32:41open core. Uh it has a permissive
925. 32:44license. You can bring it in house and
926. 32:45and use it. We have a number of big
927. 32:47companies who are are already doing
928. 32:48that. Um and and you can bring your own
929. 32:52harness. You can use it with the
930. 32:53approved models that your company
931. 32:54already has. Uh so you don't have to
932. 32:56feel like you're throwing data or your
933. 32:58organization's knowledge over the wall.
934. 33:00Um this is just I mean a a
935. 33:03we can get into this a little bit and I
936. 33:05want to field more questions. Um this is
937. 33:07a a workflow that we're building out for
938. 33:09a customer um that is interested in
939. 33:13using OpenProse to document their quant
940. 33:15models. So uh for those of you who are
941. 33:17familiar,
942. 33:19um quant models have all sorts of
943. 33:21regulatory um
944. 33:23um commitments whereby every every 3
945. 33:26years or every 1 year or on certain
946. 33:28event changes, you need to update the
947. 33:31the model documentation for these quant
948. 33:33models. So this is normally a process
949. 33:35that we found in our experience that is
950. 33:37uh human white collar, very smart person
951. 33:41intensive, right? These people uh maybe
952. 33:43some of you probably spend a a share of
953. 33:46your your year going through and and
954. 33:48writing documentation. That documents
955. 33:51what your model's being used for, under
956. 33:53what conditions was it tested, under um
957. 33:56uh who who what other models reference
958. 33:58it upstream, and a whole slew of other
959. 34:00required These are obviously required by
960. 34:02the regulators and your internal
961. 34:04auditors and your internal compliance.
962. 34:06Um so this is this is a a flow that that
963. 34:09we've found success running.
964. 34:12Uh it's it's perfect because in some
965. 34:14ways the regulatory docs are like an
966. 34:17OpenProse program, right? And so this
967. 34:20these are requirements written as prose.
968. 34:23Um and and so I'm going to kick off
969. 34:26You can see here uh
970. 34:28we're in Codex. You You run this in
971. 34:31Cloud Code. You could run this on your
972. 34:32own harness. The whole point is that
973. 34:34this is harness agnostic.
974. 34:36Uh so we we just say boot the kernel.
975. 34:40Uh and then pros run and then a specific
976. 34:43program. And so I'm going to kick this
977. 34:44off and while it runs I can show you
978. 34:46like what the kernel looks like
979. 34:48uh and then what the program itself
980. 34:49looks like.
981. 34:50Um just go back to Chrome.
982. 34:54And then
983. 34:54>> Right there.
984. 34:55>> It's uh
985. 34:57that one.
986. 34:58So this is the kernel. Um
987. 35:00it's markdown. It's not intended to be
988. 35:03read by you which is why it's formatted
989. 35:04like this. We originally had it in a
990. 35:06nice looking format and people
991. 35:07complained. They're like, "What is
992. 35:08this?"
993. 35:09Uh it's meant to be read by the agents.
994. 35:12Uh and it's just markdown. It's
995. 35:13operating instructions for what to
996. 35:15expect. You could think of it a lot like
997. 35:17a a bootloader or a resolution path, a
998. 35:20an index
999. 35:22uh system, a way of finding other
1000. 35:24documents that you that that the that
1001. 35:26the agent needs. Um
1002. 35:29and then the program itself
1003. 35:32looks and again this is what I
1004. 35:34referenced in Codex looks very similar.
1005. 35:36So this is again markdown. It contains a
1006. 35:39whole slew of the requirements that the
1007. 35:42regulators and the compliance and the
1008. 35:43auditors are looking for.
1009. 35:45Uh and this specifies, you know,
1010. 35:48uh the ideal state of of the end result
1011. 35:51is going to comply with this contract.
1012. 35:54Uh and so those are the only two things
1013. 35:56you pass in. You just pass in the kernel
1014. 35:57which is published publicly. You can all
1015. 35:59download that right now uh and and and
1016. 36:01try it out and then write any program
1017. 36:03that you want, any contract that
1018. 36:05requires some complex um
1019. 36:08set of conditions to satisfy and then
1020. 36:10the agents are just going to chug and
1021. 36:12and the you know, the better the
1022. 36:13harness, the better the model, the the
1023. 36:15faster and the cheaper that they will
1024. 36:16produce the output. Uh but the idea is
1025. 36:19that that you're you're declaring the
1026. 36:21the result rather than declaring how to
1027. 36:23how to go about about doing it.
1028. 36:26Um
1029. 36:27Let's see we can poke back and see. We
1030. 36:29ran this with a a relatively cheap model
1031. 36:31here so that it would run faster.
1032. 36:34Uh but it's still working.
1033. 36:36Um
1034. 36:39Oh, it looks like it's finishing up. So,
1035. 36:40this is I mean this is the document that
1036. 36:41it produced. So, in this case we're
1037. 36:44we're we're um creating documentation
1038. 36:47for the sofa curve. It's a public uh
1039. 36:49model. So, we're not showing any any uh
1040. 36:52internal uh bank data here. Um and this
1041. 36:57I don't know. Raise a hand. Anybody Are
1042. 36:59there any like quants in the room?
1043. 37:01To review it?
1044. 37:02How does this How does this look?
1045. 37:05Yeah, [laughter] all right. Well,
1046. 37:06we'll come we'll come we'll come find
1047. 37:08you afterwards. Um but but so that you
1048. 37:10know this is this is this is
1049. 37:11documentation that has to be produced um
1050. 37:14regularly for every model. A big bank
1051. 37:17has
1052. 37:184,000 or so models. They have to be re
1053. 37:20re re-documented yearly and uh this is a
1054. 37:24this is a a a problem that is worth
1055. 37:27anywhere between 10 to 50 million
1056. 37:28dollars a year uh for some of these
1057. 37:30bigger banks and and we're having great
1058. 37:32success in in um
1059. 37:34in doing it for them for for much
1060. 37:36cheaper.
1061. 37:37Uh granted there are some some um
1062. 37:41Getting this
1063. 37:42through compliance and the auditors and
1064. 37:44the regulators is is another part of the
1065. 37:46battle obviously as I'm sure you're all
1066. 37:48familiar with but um
1067. 37:50we we can benchmark this against
1068. 37:52historical results. We can benchmark
1069. 37:54this against how long it takes the teams
1070. 37:56to do it uh and we have quant you know
1071. 37:58quants who are on the floor who are able
1072. 38:00to to rate these and and give us a sense
1073. 38:03of how how well the models are doing on
1074. 38:05this and we're finding that it's I mean
1075. 38:07quite shocking shockingly good. Um for
1076. 38:10for much cheaper than than traditional
1077. 38:13uh
1078. 38:13manual way of doing this.
1079. 38:15Any questions? Any like
1080. 38:16>> the quants do the stuff that the quants
1081. 38:18are good at.
1082. 38:19>> Making the models.
1083. 38:20>> Not the regulatory paperwork which
1084. 38:22probably isn't very fun. I'm not a
1085. 38:24quant, but
1086. 38:26>> [snorts]
1087. 38:27>> Any questions?
1088. 38:28Great question. Uh
1089. 38:30yes. And so, actually, um Raymond
1090. 38:32showing off the chat agent a little bit
1091. 38:35as part of the kernel, as
1092. 38:37buried in the markdown
1093. 38:39at a at a path, we have uh best
1094. 38:42practices. The These are the types of
1095. 38:43things you want to be doing when you're
1096. 38:45declaring out out outcomes. Uh these are
1097. 38:48um and anti-patterns, things that you
1098. 38:50shouldn't be doing. And so, yes, there
1099. 38:52is there is a a world in which you're
1100. 38:55not typing the whole contract itself,
1101. 38:59but you're iterating, and and that's
1102. 39:00what Raymond was showing uh with another
1103. 39:03agent following our best practice guide
1104. 39:05on how to on on how to create the
1105. 39:07contract. But, yeah, I mean,
1106. 39:09getting getting intent out of people's
1107. 39:11heads is like the hard part, and and I
1108. 39:13think this is a piece of it. It's
1109. 39:14certainly not like the whole
1110. 39:15>> Yeah. I have a slightly like longer uh I
1111. 39:19don't want to make this too long. We
1112. 39:20only have 5 minutes. But, like, a
1113. 39:21slightly longer rant on this, which is
1114. 39:23that I do think that this is a big part
1115. 39:25of why
1116. 39:26chat like blew up so fast is like you do
1117. 39:30not need to know anything about what
1118. 39:32you're looking for, just start engaging
1119. 39:33with ChatGPT, and just like
1120. 39:35the magic of
1121. 39:37generative AI and like having it
1122. 39:39like come back to you, right? And so,
1123. 39:41like, if that's
1124. 39:43I don't I don't know that OpenProse is
1125. 39:46the best place if you're at that stage
1126. 39:48of something where you re you really
1127. 39:50don't even know the shape of what you're
1128. 39:51talking about, or you're not even sure
1129. 39:54like what it is that you're looking for.
1130. 39:56Um and like this go
1131. 39:59I'm like a very old-school DSPy user,
1132. 40:01and like my my joke about DSPy was like,
1133. 40:04"Oh, no, I have to know what I want now,
1134. 40:06right?" Because like the whole idea of
1135. 40:07DSPy is like you have to you create the
1136. 40:10signature of the thing that you want.
1137. 40:11And if you and you you you like see
1138. 40:14something like that, you're like, "This
1139. 40:15is so awesome." And then you sit down,
1140. 40:16and you're like, oh I I actually
1141. 40:19don't know what I want to do I, right?
1142. 40:21So, I think I think with the
1143. 40:23you know, Open Prose is a lot less
1144. 40:25strict than something like DS fire where
1145. 40:27you like you literally really needed to
1146. 40:28say the signature of the return object
1147. 40:31or nothing happens. Um
1148. 40:33but yeah, I would say like the built-in
1149. 40:36agent hopefully can like help with some
1150. 40:38of like you can just turn on the voice
1151. 40:39and be like, what can you do? Like and
1152. 40:42it can do almost anything you can do in
1153. 40:43the app like we
1154. 40:45you know, we have those hard-coded like
1155. 40:46confirm things for um
1156. 40:50for like anything that would cost money.
1157. 40:51Obviously, we want people like click a
1158. 40:53button for that. Um but yeah, I think I
1159. 40:56think it's um
1160. 40:58I don't know, I think it's also related
1161. 40:59to like the great jobification is like
1162. 41:02there are
1163. 41:04like chat isn't a great
1164. 41:06um
1165. 41:07interface and agents that are like RLF
1166. 41:10to be convincing chat partners are not
1167. 41:13necessarily like the best type of AI for
1168. 41:17workflows that don't have human
1169. 41:20conversation as part of them at all.
1170. 41:22Um
1171. 41:22>> And let me let me interject, which is
1172. 41:25just use cases like we tell people you
1173. 41:27don't want to use this if if you're
1174. 41:30doing a one-off, if you are like chat is
1175. 41:33perfect for that. You're if you're in
1176. 41:34exploratory mode.
1177. 41:36What this is good for is something you
1178. 41:38need to do a lot. It it needs to be
1179. 41:40reliable, it has to be repeatable, it
1180. 41:42needs to be auditable.
1181. 41:44Those types of workflows are what what
1182. 41:46we're what Open Prose is built for and
1183. 41:49meant for and so there is an implicit
1184. 41:51like you do know what you need
1185. 41:53in in here.
1186. 41:54>> Oh, we got a question from a real quant.
1187. 41:57>> And earlier I alluded to and I I
1188. 41:58wouldn't just call it inheritance, I
1189. 41:59would call it composition. So, we we in
1190. 42:02fact a core feature of of the framework
1191. 42:06and and the the pattern itself is it's
1192. 42:08built to be composed and so yes,
1193. 42:11injecting multiple contracts into the
1194. 42:14same
1195. 42:16uh run in in which the agent now has to
1196. 42:18satisfy many of them is is is a
1197. 42:22a core piece here. The other half
1198. 42:25the event-driven half
1199. 42:27uh which maybe looks a little bit you
1200. 42:28might call it inheritance. I I think of
1201. 42:30it more like um
1202. 42:32uh I don't know if you're familiar with
1203. 42:34like React React-style memoization where
1204. 42:36you have certain components. It's still
1205. 42:38composition. You have certain components
1206. 42:40that have upstream dependencies and when
1207. 42:44the data on on one of them changes, it's
1208. 42:47of course going to trickle down to
1209. 42:48everything below it.
1210. 42:50Um
1211. 42:51that is another place where Jev is super
1212. 42:53exciting and and the decision model
1213. 42:55super exciting because we had built that
1214. 42:56system and we had agents trying to
1215. 42:59determine has the state changed and if
1216. 43:02the state on upstream dependency
1217. 43:04changes, do we kick off downstream
1218. 43:06dependency?
1219. 43:07Uh but that's very expensive to run and
1220. 43:09Jev becomes suddenly this very cheap
1221. 43:11like semantic classifier or semantic
1222. 43:14memoizer, which you could think of it as
1223. 43:16like a cache and Jev is deciding when
1224. 43:19something is out of date and now it
1225. 43:21kicks off everything downstream from
1226. 43:22that. So, that's another kind of core
1227. 43:24component of of the framework.
1228. 43:27Minus the Jev part, we're still working
1229. 43:28we're still building that part in. Yes,
1230. 43:31and there is no magic here. It's all
1231. 43:32just English. It's all just language.
1232. 43:34Uh yeah, it's but yes, and that's where
1233. 43:37the composition composability comes in.
1234. 43:40>> The other piece to your point is like
1235. 43:41it's not just a template. So, like you
1236. 43:43could obviously even pre-LLMs, you could
1237. 43:45have
1238. 43:47um or very let's say not very early LLMs
1239. 43:50or even like encoders could like fill a
1240. 43:52template for you. Like you like prefix
1241. 43:54the template and like fill the thing.
1242. 43:56But here it's like, "Well, what can you
1243. 43:58template?" And so, you could template
1244. 44:01the entire multi-agent workflow. So,
1245. 44:03like this the simple example that I give
1246. 44:05is you could say like the output that I
1247. 44:07need is an article.
1248. 44:09You know, and maybe there's it's got
1249. 44:11like this type of article, it's going to
1250. 44:12be this length, whatever, right? Some
1251. 44:14constraints around that.
1252. 44:16And the input is like the topic. Like
1253. 44:19until you've got a good article for me,
1254. 44:22and the parent agent decides good, it
1255. 44:24decides when to exit, so that's where
1256. 44:26you're going to get a better result with
1257. 44:27a stronger model than a weaker one.
1258. 44:30I've got a researcher
1259. 44:33sub-agent. That agent has my access CLI
1260. 44:36and my access skill on how to use a CLI
1261. 44:38cuz I don't want it, you know, using any
1262. 44:40random web search. I've got a writer
1263. 44:43sub-agent that has my special like
1264. 44:46here's the last 10 articles I wrote that
1265. 44:48are really good, and you I want you to
1266. 44:50write like this. I've got a copy editor
1267. 44:52uh sub-agent, and that whole thing is
1268. 44:55just maybe
1269. 44:56>> But the really important 100 lines of
1270. 44:58markdown, and it might take them 10
1271. 45:00hours to do it.
1272. 45:01>> important parts to emphasize there is is
1273. 45:03the contract composition where so you
1274. 45:05you you go to like, I mean, these things
1275. 45:07have been around forever. Any Any
1276. 45:09Any workflow builder, drag-and-drop
1277. 45:10workflow builder.
1278. 45:12That's a different way of thinking about
1279. 45:13this because when you think about this
1280. 45:15in terms of that like DAG
1281. 45:18directed acyclic graph of contracts, you
1282. 45:21can have intermediary steps that you
1283. 45:23almost think of them as like checkpoints
1284. 45:25or like
1285. 45:27this intermediary step meets the
1286. 45:29contract, and the thing is going, and it
1287. 45:32hits this contract and fulfills this
1288. 45:33one, and then that causes the next one
1289. 45:35to keep going until it's fulfilled that
1290. 45:37contract. So, you can you can still in
1291. 45:39some ways declare the how to get there
1292. 45:42just by composing these intermediary
1293. 45:44contracts along the way and still
1294. 45:46letting the agents be a little bit
1295. 45:47flexible on on how to do achieve that
1296. 45:50intermediary step.
1297. 45:51>> Yeah, and and the shape of it. So, like
1298. 45:53one one example like when Claude um
1299. 45:56the dynamic workflows came out, they had
1300. 45:58this blog post that's like, here's the
1301. 45:59six types, right? And it's like one of
1302. 46:01them's like fan out, fan in, and like a
1303. 46:03map reduce with sub agents. One of
1304. 46:05them's like a
1305. 46:06adversarial review, one's like a loop
1306. 46:08until X condition is met, right? And all
1307. 46:12of those things you can express in a
1308. 46:14prose program and then run them on on
1309. 46:17any agent. So, you can
1310. 46:18like you can template the shape of the
1311. 46:21the multi-agent workflow um
1312. 46:25with no code. Um and again,
1313. 46:28the whole it's only possible because the
1314. 46:30the agents are like good at managing
1315. 46:33themselves and because most harnesses
1316. 46:34have sub agents. So, like if you're
1317. 46:36running plain vanilla pie, which has no
1318. 46:38sub agents at least last time I checked,
1319. 46:40you're going to
1320. 46:41want sub agents for it to do that more
1321. 46:44complex kind of workflow. But, like
1322. 46:46essentially any harness that has one of
1323. 46:49the requirements,
1324. 46:50>> Sub agents.
1325. 46:50>> sub agents is pretty much the only one
1326. 46:52for most
1327. 46:53>> context.
1328. 46:54>> Yeah, like um
1329. 46:55like bash and sub agents are actually
1330. 46:58you don't even need bash. We can run
1331. 46:59these in
1332. 47:00VM isolates in code mode, too. So,
1333. 47:02>> Any other questions? We can give 10
1334. 47:04minutes back to everyone. Uh
1335. 47:06it is based on
1336. 47:08my many years of engineering experience
1337. 47:11and dealing with people who sometimes
1338. 47:14are not very good at expressing
1339. 47:15requirements and other people who are
1340. 47:16very good at expressing requirements.
1341. 47:18And it's also, I mean, we we benchmark
1342. 47:20these internally and and run them
1343. 47:22against we
1344. 47:23we have sort of an evolutionary
1345. 47:25algorithm where we can shrink the kernel
1346. 47:27by seeing if if they can conform to
1347. 47:30that. So, it's based on practice and
1348. 47:33empirical results. Uh
1349. 47:35>> Yeah, you can just have that as a spec
1350. 47:37and you could have like let's say you
1351. 47:38like um
1352. 47:40uh Matt Pocock's like uh
1353. 47:42spec, you know, you like Matt I built a
1354. 47:45funny one just to like troll him that
1355. 47:46was like auto Pocock and it basically
1356. 47:49it was my way of proving that my way of
1357. 47:50wiring skills in works.
1358. 47:53So, you basically just he has a workflow
1359. 47:54that's like grill me, you know, and then
1360. 47:57you ask all these questions and then
1361. 47:58it's like TDD and then all these other
1362. 48:01ones and you can put the whole thing in
1363. 48:03one prose program and just be like do
1364. 48:05Matt Pocock's workflow and every
1365. 48:07sub-agent has to load at every stage of
1366. 48:10the program you have to load this the
1367. 48:12next skill in the sequence. Um so yeah,
1368. 48:15you can absolutely like
1369. 48:16>> do have people I I wasn't calling it
1370. 48:18spectrum driven development, but like
1371. 48:20almost by its very nature it is spectrum
1372. 48:22driven development. I'd love you to try
1373. 48:24this out. Try it or come find me after
1374. 48:26and like whatever spectrum driven
1375. 48:28development specification you have, we
1376. 48:30could compose with the
1377. 48:32boot kernel prose run and then we'll see
1378. 48:35how it does. I I think it would do quite
1379. 48:37well actually.
1380. 48:38That's a good good inspiration.
1381. 48:40I haven't tried it with the spectrum
1382. 48:42driven development contracts. Like I'd
1383. 48:43love to sit down and and and try it and
1384. 48:45see if it overrides it or if it starts
1385. 48:48colliding.
1386. 48:49>> That'd be interesting like um
1387. 48:51>> Here's Here's one
1388. 48:52>> See how much different skills like throw
1389. 48:54it off.
1390. 48:54>> Here's one thing we do have and software
1391. 48:57engineers using Open Prose to develop
1392. 49:00software.
1393. 49:01Uh
1394. 49:02more of our users are using Open Prose
1395. 49:05to do ops and product management and
1396. 49:08um
1397. 49:09quant model documentation and non I
1398. 49:11don't know software engineering stuff.
1399. 49:14Uh I think it's just the blend of like
1400. 49:15markdown documentation actually is more
1401. 49:18interesting to non-engineers um as a
1402. 49:21result. Yeah.
1403. 49:23Any other questions?
1404. 49:25Thank you all.
1405. 49:27Appreciate your time.
1406. 49:28>> [applause]