GPT-6 Astra Finally Built the Ultimate JARVIS — Transcript

Full transcript

1. 0:00Every time I've tried to build up a
2. 0:02Jarvis assistant, I've ended up with
3. 0:04something that was technically cool but
4. 0:06incomplete until GPG6 Astro came along.
5. 0:10I gave it a much harder challenge. Build
6. 0:13me a complete Jarvis. Not just a fancy
7. 0:17chatbot with a microphone, but a proper
8. 0:19local assistant that can talk to me
9. 0:21naturally, access my actual data, decide
10. 0:25on which tools it needs, and then
11. 0:27respond through a polished interface.
12. 0:30GPT6 did all of this in one single
13. 0:33prompt and the results blew my mind. So
14. 0:36in this video, I'm going to show you how
15. 0:39this Jarvis works, how you can build one
16. 0:42yourself, and most importantly, how you
17. 0:44can expand it to fit your own workflow.
18. 0:48Let me show you how it works and let's
19. 0:50dive in. Hey Jarvis, can you please prep
20. 0:52me for my next meeting? So as I gave it
21. 0:54this command, it's going to search
22. 0:56Gmail. It's going to search my content
23. 0:57vault, search Google Drive, find all of
24. 1:00the relevant information, and put it all
25. 1:03together into a full brief for today's
26. 1:06meeting.
27. 1:06>> You're meeting Brady Spear, Spearhead
28. 1:08Media, at 9:30, 11:05 p.m.
29. 1:11>> Also, Jarvis, can you please open up
30. 1:14Safari?
31. 1:16Now, the reason that this works so well
32. 1:19is that it didn't manually program any
33. 1:21of those commands. There's no specific
34. 1:24prepare meeting voice command hidden
35. 1:26somewhere. I just gave Jarvis the tools,
36. 1:29the processes, and most importantly,
37. 1:32access to an AI agent. Astra built
38. 1:34everything around this modular concept.
39. 1:37And really, there's three things that
40. 1:40GPT6 Astra nailed that made this version
41. 1:43work so much better than any other
42. 1:45Jarvis I've ever built. Now, the first
43. 1:47thing that I wanted was really obvious
44. 1:49because if I'm building Jarvis, I don't
45. 1:50want to type, right? I want to be able
46. 1:52to talk to it. So, Astra wired 11 labs
47. 1:55into the interface for both speech
48. 1:57recognition and for voice output. And
49. 2:00that really gives me a continuous voice
50. 2:02loop. I speak, Jarvis transcribes what
51. 2:06I'm saying. The assistant processes it
52. 2:08and 11 Labs streams the response back.
53. 2:12Now, the interface also changes
54. 2:14depending on what's happening. Hey
55. 2:17Jarvis, what's the capital of Somalia?
56. 2:20The capital of Somalia is Moadishu.
57. 2:22>> So most voice assistants that I've built
58. 2:24before have felt like speak, wait, wait,
59. 2:28get an audio file back, play it. But
60. 2:30this feels much more like an actual
61. 2:33conversation. Obviously, there is a
62. 2:35little bit of lag between the time that
63. 2:37I ask a question and the agent response
64. 2:40because there's tool calls. There's 11
65. 2:42labs transcription in between, but the
66. 2:45answer that I get back is pretty much
67. 2:47complete because the AI agent has
68. 2:49understood exactly what it is that I'm
69. 2:51asking. So, 11 Labs is only handling the
70. 2:54voice layer, right? It's not the
71. 2:56intelligence behind Jarvis. And that
72. 2:58part is slightly more interesting. If
73. 3:01you want the prompt to build this
74. 3:02yourself or if you want me to work with
75. 3:05you to build bespoke solutions for your
76. 3:07business, check out the links in the
77. 3:09description. Let's get back into the
78. 3:11video. Now, normally if I wanted to
79. 3:14build something like this, I'd have to
80. 3:15connect my app Jarvis directly to an LLM
81. 3:19API. But because this is running locally
82. 3:22on my Mac, I realized I already had
83. 3:25extremely capable AI agents sitting
84. 3:27there. codec cli cloud code. So instead
85. 3:32of building another API integration,
86. 3:34Astra connected the app directly to the
87. 3:37CLI. So the architecture basically looks
88. 3:40like this. Voice comes in, the local
89. 3:43Jarvis application sends the request to
90. 3:46the coding agent, whether that's cloud
91. 3:48code or codeex. The agent reasons about
92. 3:52what I asked. it can use the tools that
93. 3:54I've given it and then it returns a
94. 3:57structured response to the interface.
95. 3:59Now, this has two benefits. First,
96. 4:02depending on the CLI and the account
97. 4:04that I'm using, you can potentially use
98. 4:06the AI access you already pay for
99. 4:09instead of metering every single Jarvis
100. 4:12conversation through an API. But
101. 4:14honestly, the bigger benefit is that
102. 4:16these coding agents are already built to
103. 4:19operate inside a computer environment.
104. 4:21They already understand the concept of I
105. 4:23have tools, I have instructions, I need
106. 4:25to inspect something, I need to run
107. 4:26something, I need to combine the result.
108. 4:28And that makes them a surprisingly good
109. 4:31brain for something like Jarvis. But
110. 4:33that only works if you give the agent
111. 4:35the right tools. And this is where the
112. 4:38project became much more interesting.
113. 4:41Now, a normal LLM can tell you things,
114. 4:44but Jarvis needs to be able to find and
115. 4:46do things as well. So I gave the agent
116. 4:49control toolbox. Right now it's
117. 4:52connected to the Google Workspace CLI
118. 4:54which I've installed on my computer.
119. 4:56Through it, it can search my Gmail, read
120. 4:59emails, look at my calendar, and search
121. 5:01my Google Drive. It can also search my
122. 5:04Obsidian Vault. That's my second brain.
123. 5:07And these are all very simple tools that
124. 5:10Astra coded into this Jarvis for me.
125. 5:13things like search Gmail, read Gmail,
126. 5:15get calendar events, search vault, etc.
127. 5:18And the important part is that I never
128. 5:20call these tools manually. Jarvis itself
129. 5:24decides when it needs them. So if I say,
130. 5:27"Hey Jarvis, can you pull up my most
131. 5:30recent unread emails?"
132. 5:35>> You have several unread emails. Most are
133. 5:38cold outreach pitches with two hero
134. 5:40digests and a couple of stalled reply
135. 5:42threads worth a decision.
136. 5:44>> So if I open up this event inspector,
137. 5:46you'll see that it this is the request
138. 5:49that I sent it. And these are all the
139. 5:51events that happened. It kicked in the
140. 5:53search Gmail tool. It got a response
141. 5:55saying the tool was completed. It got
142. 5:57all this information back and it
143. 6:00processed all of that using the AI brain
144. 6:03itself plot code in this instance and
145. 6:06gave me back a structured response like
146. 6:08this. This is incredible and is really
147. 6:11the Jarvis that I've always wanted to
148. 6:13build. Now, the meeting prep example
149. 6:15that you saw at the beginning of the
150. 6:17video is the part that finally made it
151. 6:20feel intelligent cuz Jarvis had to check
152. 6:22my calendar so it knew who I'm meeting.
153. 6:26Then it had to search Gmail for any
154. 6:28recent correspondence with that person.
155. 6:30Then it searched my Obsidian second
156. 6:32brain for previous meeting notes. It
157. 6:34checked Drive for any relevant
158. 6:36documents. And it actually went ahead
159. 6:38and combined all of that into a single
160. 6:41briefing. And this is the bit that every
161. 6:44previous Jarvis attempt I've made was
162. 6:47missing. The assistant isn't just using
163. 6:50one integration at a time. It's
164. 6:52reasoning across them. So instead of
165. 6:54saying you have a meeting with Brady at
166. 6:579:30, it can say something more like
167. 6:59you're meeting Brady at 9:30. Here's
168. 7:02what you discussed in your recent email
169. 7:04threads and this is what you spoke about
170. 7:06on your last discovery call. Most of
171. 7:09these things are still unresolved. Now
172. 7:12this is genuinely useful and it actually
173. 7:14makes the slight lag from my question
174. 7:17and the response forgivable. Now, the
175. 7:19other thing that I really like about
176. 7:21this architecture is that it's
177. 7:23expandable. So far, you've seen me demo
178. 7:26the GWS CLI and the Obsidian Second
179. 7:29Brain. But what if I wanted to give it
180. 7:31access to more tools? If I want Jarvis
181. 7:33to control or do something new, I don't
182. 7:36have to rebuild the entire app because
183. 7:38the tool schema is already built in. So,
184. 7:41I just have to open up the Jarvis folder
185. 7:43in Codeex and ask it to add a new tool.
186. 7:46So, let's say I want to add a tool that
187. 7:48checks my max health. And when I call
188. 7:50the tool, it should show up as a popup
189. 7:53with a brief LLM response. I just have
190. 7:55to describe the tool that I want, hit
191. 7:57enter, and let Codeex do its thing. Now,
192. 8:00while we're waiting for this to build, a
193. 8:02tool basically needs just four things. A
194. 8:04name, a description, the inputs it
195. 8:06expects, and the code that actually
196. 8:07performs the action. So, hypothetically,
197. 8:10you could add anything, right? You could
198. 8:11add your home assistant. You could add
199. 8:13Spotify, Slack, YouTube Analytics,
200. 8:15access to a new database, a custom
201. 8:17script, pretty much anything because the
202. 8:20underlying brain that's building this
203. 8:22Jarvis is codeex. And once the tool
204. 8:25exists, the AI can decide when it makes
205. 8:28sense to use it. And because Astra built
206. 8:31the tool layer separately from the
207. 8:33interface, all of those new abilities
208. 8:36automatically plug into the same Jarvis
209. 8:39experience. Check my computer's health
210. 8:42for me, please. There it is. A snapshot
211. 8:45of the Mac running Jarvis. CPU load,
212. 8:48memory, storage, battery, and thermals.
213. 8:53Finally, and the real reason that I
214. 8:55wanted to make this video is that none
215. 8:57of the individual technologies here are
216. 9:00new. 11 Labs existed, coding agents
217. 9:04existed, tool calling existed. I've
218. 9:07already had an Obsidian vault for a very
219. 9:09long time. The difficult part has always
220. 9:12been getting all of these pieces to work
221. 9:14together without the project turning
222. 9:17into a giant mess. And that is where
223. 9:19Astra surprised me. I gave it a fairly
224. 9:23ambitious system to build a real-time
225. 9:25voice interface, a local AI agent, a
226. 9:28tool registry, permissions, UI,
227. 9:30processes, etc. And instead of giving me
228. 9:33another half working prototype, it
229. 9:36managed to wire the whole system
230. 9:38together into something I could actually
231. 9:40use. Previous models helped me build
232. 9:43parts of Jarvis, and this is the first
233. 9:46time one helped me build the actual
234. 9:49system. dig into the content vault and
235. 9:52get notes of my last meeting with Brady
236. 9:55Spear and go ahead and draft a reply to
237. 10:00the email thread with a summary of what
238. 10:02we had discussed la last time. Do not
239. 10:05send the email, just save it as a draft.
240. 10:11>> Your draft is saved in Gmail. It has not
241. 10:13been sent.
242. 10:14>> And here is the draft complete with all
243. 10:16the notes from the last meeting. saved
244. 10:20in Gmail. Let's go ahead and open up
245. 10:22Gmail and make sure that the draft is
246. 10:24actually there and that Jarvis is not
247. 10:26making this up. And there is the draft.
248. 10:29Awesome.