Building a $1,500 Lead Artifact: Opus 5 vs GPT-5.6 Sol — Transcript

Full transcript

1. 0:00Today, we're putting Opus's 5
2. 0:02head-to-head with Chat GPT's 5.6 Soul.
3. 0:05Now, this isn't some random
4. 0:06demonstration of some code base I have
5. 0:08zero intention on using. I'm actually in
6. 0:10the market for this. So, I figured what
7. 0:12better way to test these models than to
8. 0:15test it for something I need to build.
9. 0:17So, without further ado, let's build.
10. 0:20All right. So, as you can see here on
11. 0:21the screen, we have two fresh terminals
12. 0:23open. I just said hi on both to get the
13. 0:26meter logged and start working. What's
14. 0:29interesting is Opus 5 on a high cost 30
15. 0:32cents to say hi. Uh whereas GPT 5.6 Soul
16. 0:36only cost us just about 5 cents to say
17. 0:39hello.
18. 0:40>> [laughter]
19. 0:41>> Uh that's funny. Now, here's the prompt
20. 0:43for today.
21. 0:44You are building a front-end only
22. 0:46prototype of a client-facing AI ops
23. 0:48audit report. This is not a production
24. 0:50application, no real authentication, no
25. 0:53real back-end, no real database, no
26. 0:55payment processing, no scheduling
27. 0:56integration, no PDF generation service,
28. 0:59no multi- tenant logic. All data is
29. 1:02synthetic and hard-coded or mocked in
30. 1:04the front end. All export and send
31. 1:06actions are non-functional UI states and
32. 1:08must be visibly non-functional, not
33. 1:11misleadingly styled as real. Now, before
34. 1:14I continue reading the rest of this,
35. 1:15let's copy this prompt and let's get
36. 1:18started with this demonstration here.
37. 1:24Bam. Bam.
38. 1:26And I'm going to switch this to auto
39. 1:27mode.
40. 1:30We already have permission set to
41. 1:32approve for me.
42. 1:33Um so, we're just going to let those get
43. 1:35to work and we'll continue reading the
44. 1:37prompt. So, stack React plus Vite
45. 1:39Tailwind CSS
46. 1:41Shadcn UI components, Recharts for any
47. 1:44data visualization, ship a single
48. 1:47deployable front end, no server
49. 1:48required. Build these sections as one
50. 1:50flowing client portal.
51. 1:53So, we have a lot of specifics here. I'm
52. 1:55not going to read these line for line,
53. 1:57but if you do want a copy this, just
54. 1:58screenshot the page here.
55. 2:01Synthetic client
56. 2:03placeholder
57. 2:06what we heard source material,
58. 2:07opportunity candidates to score, right?
59. 2:09So, we're going to have a certain
60. 2:11scoring chart so to speak so that the
61. 2:13customer or client can visualize the
62. 2:16opportunities based off of these
63. 2:19specific metrics and then a few things
64. 2:22to note at the end intentionally not
65. 2:23specified visual design direction
66. 2:26information architecture copy tone chart
67. 2:29types section order beyond the list
68. 2:30above
69. 2:32and how confidently to score each
70. 2:33opportunity.
71. 2:35That judgment is what's being tested,
72. 2:37right? So,
73. 2:39while I gave it a rather detailed
74. 2:41prompt,
75. 2:42the reason I'm giving it this detailed
76. 2:44prompt is I actually want to use one of
77. 2:46these code bases moving forward. I have
78. 2:47a need for this. This is not me building
79. 2:49something I don't need. Uh hence the
80. 2:51purpose of this demonstration here. So,
81. 2:54I really wanted to be able to judge.
82. 2:56We're not going to factor in speed that
83. 2:58much. They should finish around the same
84. 3:00time I'd imagine.
85. 3:02But, I want to test its judgment in
86. 3:04visual design, how it's going to
87. 3:06architect this portal, and the overall
88. 3:09experience end to end, right? Uh that
89. 3:12way I can pick a code base I'm happy
90. 3:14with and then actually build the thing
91. 3:16on audit.clearmud.ai.
92. 3:20So, let's get back to the demo here.
93. 3:22And hey, they're both getting started.
94. 3:24So, what we're going to do
95. 3:26is whoop, let's also scroll down. I have
96. 3:28a lot of meters open. So, these are the
97. 3:30two meters associated with the project.
98. 3:32We can also confirm here by looking as
99. 3:34you can see audit app.opus all sessions.
100. 3:38That's what this one's called here.
101. 3:40The same goes for GPT.
102. 3:44audit.app.GPT
103. 3:45all sessions
104. 3:48And what we're going to do is we're just
105. 3:50going to fast forward through the boring
106. 3:51part.
107. 3:55All right. So, I wanted to end that time
108. 3:58lapse here. As you can see, GPT finished
109. 4:01in lightning speed.
110. 4:03Uh it finished first.
111. 4:06And it's already come in a lot cheaper.
112. 4:09I'm actually very shocked by this price.
113. 4:12Uh it even did some QA and it it I
114. 4:14thought it was going to be local, but it
115. 4:16deployed it to Vercel.
116. 4:18Very interesting. While Opus is still
117. 4:20working. So, before we check that out,
118. 4:23we are going to kind of prepare, copy
119. 4:26this,
120. 4:28add in a browser tab to the right,
121. 4:31paste that in there.
122. 4:32I'm going to hit enter, but I'm going to
123. 4:33immediately tab over because I do not
124. 4:35want to
125. 4:37um
126. 4:38cheat by looking at it first.
127. 4:42Wow.
128. 4:45It's a big price discrepancy there. So,
129. 4:47looks like this was 1.95
130. 4:50million input.
131. 4:52Okay.
132. 4:53I'll
133. 4:54I'll have to cross-reference. I think
134. 4:56that's accurate, right? So, very
135. 4:58interesting. Very interesting. So, we're
136. 5:00just going to fast forward through the
137. 5:01boring part until Opus finishes.
138. 5:06All right. So,
139. 5:09>> [laughter]
140. 5:10>> I was not expecting this.
141. 5:13Opus 5 appears to be insanely token
142. 5:17hungry.
143. 5:19It's so absurd that I'm going to have to
144. 5:22go back at the end of this demo to make
145. 5:25sure that my my API meter is accurate.
146. 5:28Um what?
147. 5:33>> [laughter]
148. 5:34>> Uh
149. 5:35so,
150. 5:36that took
151. 5:3727 minutes
152. 5:39on Opus, 11 minutes on ChatGPT. The
153. 5:43price kind of shows you can see the
154. 5:46tokens. I mean,
155. 5:48wow.
156. 5:49Now, one undesirable thing here I'm
157. 5:51already seeing, it's
158. 5:54it's sending us to an artifact through
159. 5:57our account.
160. 5:58I honestly thought they were going to
161. 6:00both spin up a local preview for me.
162. 6:03Uh chat GPT went above and beyond uh and
163. 6:06deployed the Versel. I know for a fact
164. 6:09my Claude CLI has access to that. So,
165. 6:11I'm I'm a bit curious like, okay, Claude
166. 6:14defaults to sending you to their
167. 6:16platform, whereas it looks like Codex
168. 6:18defaults to maybe
169. 6:21the last method of deployment uh because
170. 6:24I have been deploying a lot with
171. 6:26Codex. Um wow, okay.
172. 6:29Well,
173. 6:32let's go ahead and open this up. I might
174. 6:34have to log in to
175. 6:36Oh, no, I don't. Okay, great.
176. 6:39Do not want to share. All right. So,
177. 6:42since GPT finished first,
178. 6:45we're going to go with GPT's demo first.
179. 6:48Now, let's go full screen here.
180. 6:53And we'll bring that down.
181. 6:56All right, nice split screen. I love how
182. 6:58this looks, super professional.
183. 6:59Obviously, it's not our brand color, but
184. 7:01I didn't give it our palette. I didn't
185. 7:03give it uh a clear method front-end
186. 7:05design skill to follow. I wanted it to
187. 7:07use its best judgment.
188. 7:09Continue with Google. Okay.
189. 7:11Final report. So, this is just a
190. 7:13one-page scroll from what I can see.
191. 7:14We're We're scrolling down
192. 7:16and it allows you to kind of navigate
193. 7:18between Cool. This is a great-looking
194. 7:20brief.
195. 7:22Very professional.
196. 7:32Key clear pain points we heard in the
197. 7:34meeting.
198. 7:36Opportunity map.
199. 7:38Chart is not necessarily
200. 7:41Is it working as as as intended? Kind
201. 7:44of. I would have loved to see a little
202. 7:46bit more, but I get what it's doing with
203. 7:48the dots. Very impressive. Look at these
204. 7:51Look at this functionality. It's pretty
205. 7:53cool. I would make those a little bit
206. 7:54larger, right?
207. 7:57So, five for efficiency, revenue upside,
208. 8:00revenue protection, effort. Okay, great.
209. 8:05Feel like we would want to add an
210. 8:06overall score baseline as well.
211. 8:09Build the operating layer in the right
212. 8:10order.
213. 8:13Okay.
214. 8:15It's pretty solid.
215. 8:18From audit to operating rhythm, kind of
216. 8:20gives a timeline.
217. 8:23One thing it's missing though that I
218. 8:25realized I forgot to is like how would
219. 8:27we build it? Yes, this is an audit. This
220. 8:29is where what we would build,
221. 8:32um but I would want a tools section
222. 8:34here. And I realize this is my fault for
223. 8:36not including that, but I was hoping
224. 8:38that one of the models would decide uh
225. 8:40as far as giving certain recommendations
226. 8:43per uh pain point. But this is okay.
227. 8:47Listen, needs some work. Needs some
228. 8:49work, but solid solid output. Okay. Now,
229. 8:53let's go over to
230. 8:57Claude, shall we?
231. 8:58Very similar split screen approach.
232. 9:02Again, I would want to give it some some
233. 9:04coloring here.
234. 9:06Wow, very
235. 9:09very basic on the coloring side.
236. 9:20Okay, what we heard.
237. 9:26Okay, I would want to kind of change the
238. 9:28visual. I guess depending on the target
239. 9:30audience, this might work.
240. 9:32But I do love the idea of having that
241. 9:34vertical menu bar rather than this,
242. 9:37right?
243. 9:38Um so interesting difference that Opus
244. 9:40went with.
245. 9:42Cool little accordion effect, pain
246. 9:44points,
247. 9:45business consequence, okay. Now, again,
248. 9:49I was hoping one of these models would
249. 9:50have chosen. I guess it was something I
250. 9:52had to get specific with as far as what
251. 9:54tools or what solutions we would
252. 9:56recommend based on those pain points.
253. 10:00Very similar opportunity map.
254. 10:04Decided to go with this kind of
255. 10:06I forget what you call this chart, but
256. 10:08this approach.
257. 10:10Okay.
258. 10:16Feels more detailed, for sure.
259. 10:27But again,
260. 10:28okay.
261. 10:45Okay.
262. 10:49Interesting, eh? You know, I'm going to
263. 10:51be honest with you. While
264. 10:55I do like Opus's output, the fact that
265. 10:58it almost took three times as long and
266. 11:00three times the cost for that matter, um
267. 11:06I would have to go with GPT. It just it
268. 11:09was so much more efficient.
269. 11:11Uh obviously, I'm going to have to
270. 11:13improve on this a bit. This is not for
271. 11:16the public. This is something I'm going
272. 11:18to use internally once we perform the
273. 11:20audit. I'm also going to be building
274. 11:22um an audit
275. 11:24kind of dashboard for me and the
276. 11:26prospect who signs up and and and
277. 11:29schedules that booking to walk through
278. 11:31together.
279. 11:33Um but more importantly, right? We're
280. 11:35going to have a very fluid natural
281. 11:37conversation.
282. 11:38And I'm just going to be listening first
283. 11:40and foremost, and then I'm going to have
284. 11:42an input to where I can take that
285. 11:43transcript, that audio file from the
286. 11:45meeting, and input it into my platform
287. 11:48so that I can have Claude or Codex or
288. 11:51both
289. 11:52analyze that transcript, define the pain
290. 11:55points, define what we hear, right?
291. 11:58Define the pain points, create the
292. 11:59opportunity map, and then work together,
293. 12:02right? This is not me saying I'm going
294. 12:04to use one versus the other for that
295. 12:06process, that part of the pipeline and
296. 12:08workflow. I'm going to use them both. If
297. 12:10I'm being honest with you, I'll be using
298. 12:11both of them. Um
299. 12:13but I I just got to say it.
300. 12:16I like how much more professional this
301. 12:18one looks. This one looks too much like
302. 12:21I'm just This isn't my style,
303. 12:24right? So, I don't like the direction it
304. 12:26took it in. Had it packaged it
305. 12:29Had Had the packaging been a bit
306. 12:30different,
307. 12:32I might be leaning towards Opus here,
308. 12:34but
309. 12:35yeah.
310. 12:37Yeah.
311. 12:39In my book, Codex is the winner here. Um
312. 12:42now,
313. 12:43I'm very curious. Let's go over to my
314. 12:46model meter.
315. 12:52And I'm just going to take a screenshot.
316. 12:55Please reference this screenshot. They
317. 12:57are the
318. 12:59audit app Opus session for Opus and then
319. 13:03the audit app GPT session for the GPT
320. 13:05meter.
321. 13:07Are these numbers accurate? I'm just so
322. 13:10blown away by the differences. Can you
323. 13:12please double-check the numbers,
324. 13:15uh cross-reference them to these two
325. 13:16links. These are the official uh API
326. 13:19pricing docs.
327. 13:20And let me know your findings.
328. 13:25All right.
329. 13:27So, this is just a little bonus part of
330. 13:28the episode to make sure we're being
331. 13:30thorough and accurate.
332. 13:40I'm just blown away at the cost
333. 13:42difference. That is kind of insane. And
334. 13:44for me to still pick GPT, I thought for
335. 13:46sure I was going to go with Opus, but
336. 13:47Opus and Anthropic, it defaulted to it's
337. 13:50kind of like it feels like a
338. 13:55What's the word I want to use to
339. 13:56describe this? Uh
340. 13:59It's just not my style. It's not my
341. 14:00style, right? Uh it doesn't feel that
342. 14:03professional, if I'm being brutally
343. 14:05honest.
344. 14:09Aha! Okay, I was wondering. So, we're
345. 14:13going to have the updated pricing here
346. 14:14in a moment.
347. 14:16It'll be able to fix it in line.
348. 14:19Let's just patiently wait.
349. 14:21All right. So, as you can see here,
350. 14:23applying the Claude fix now, GPT figure
351. 14:26will remain unchanged because its
352. 14:28published sole rates are already
353. 14:29correct. Yeah, okay. So,
354. 14:33three times more than what it should be.
355. 14:35So, I'm guessing it's going to come down
356. 14:36to about 1185
357. 14:40or what?
358. 14:42Just under $12.
359. 14:45$13. Okay, that is more accurate of a
360. 14:48demonstration because the numbers
361. 14:51weren't adding up for me here. Um
362. 14:54>> [laughter]
363. 14:55>> Okay. Now, I'll make sure to commit this
364. 14:58uh
365. 14:58once done.
366. 15:00Please commit to get.
367. 15:05Merge to main. Thanks.
368. 15:08Now, for anybody curious, this is an
369. 15:10open-source model meter that I do have
370. 15:11live. If you just check out the ClearML
371. 15:13GitHub repository or the link in this
372. 15:15description, uh you'll be able to access
373. 15:17it and use it for yourself. Now,
374. 15:21that makes so much more sense.
375. 15:26So much more sense. I I I just I'm glad
376. 15:29I checked.
377. 15:32I'm glad I checked. Wow.
378. 15:36Yeah, listen. Like even with the the
379. 15:40savings here, like I just like GPT's
380. 15:43output more.
381. 15:47Listen, Opus
382. 15:48and and Anthropic models are always my
383. 15:50go-to when I'm in ideation, creation, uh
384. 15:54prototyping phases, but once I'm ready
385. 15:56to proceed and get ready to production,
386. 15:59I trust GPT to be far more efficient um
387. 16:03and produce a production-ready codebase.
388. 16:06Uh that's where I run all my security
389. 16:07audits with. Although some of the tests
390. 16:09I've been seeing run with Opus 5 and
391. 16:12some of the benchmarks,
392. 16:14apparently it does a very good job at
393. 16:15security audits. So, I think my new
394. 16:17workflow here is going to be running
395. 16:18security audits
396. 16:20for both, right? With both models
397. 16:22against that same repository.
398. 16:25So, this is not a I'm going to use one
399. 16:27versus the other demonstration. I'm
400. 16:29literally going to be using both of
401. 16:30these. So,
402. 16:31I hope you found some value in today's
403. 16:33video. Now, I'm not an AI expert. I'm
404. 16:35building in public and sharing what
405. 16:36actually works. If you'd like to see me
406. 16:38make a particular video or run a
407. 16:39specific topic, drop a comment below.
408. 16:41Let me know who you are, what you do,
409. 16:43who your target audience is, and what
410. 16:45your question is, and I will add a video
411. 16:46to my queue custom-tailored just for
412. 16:48you. Thank you so much for tuning into
413. 16:50today's video. My name is Marcelo. This
414. 16:52is Clear Mod and Clarity Matters.
415. 16:58>> [music]