YouTube2Text

I Tested Sonnet 5.5 vs Opus 5.5. What You Need to Know. — Transcript

by Nate Herk | AI Automation · 5,745 words · 787 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Cloud Sonnet 5.5 is here and it is a
  2. 0:02clear upgrade over Claude Sonnet 5.
  3. 0:04Running 30% faster as well as costing
  4. 0:0630% less for most work. So, I've been
  5. 0:08playing around with it all day on my
  6. 0:10actual real workflows. Things like
  7. 0:12building websites, taking a bunch of
  8. 0:13messy data and turning that into
  9. 0:15spreadsheets. Also, Excel sheets with
  10. 0:17different views as well as formulas
  11. 0:18inside of every single cell, motion
  12. 0:20design and editing videos, interactive
  13. 0:22dashboards that have real-time data
  14. 0:24syncing, project management and
  15. 0:25planning, and so many other things as
  16. 0:27well. But specifically, I've been
  17. 0:28testing Son of 5.5 against Opus 5.5, so
  18. 0:31you know exactly when you should be
  19. 0:32using which model. Now, in all these
  20. 0:34comparisons, I'll go over how fast they
  21. 0:35are, the input and output tokens, and
  22. 0:37how much they actually cost you in
  23. 0:38dollars. So, let's not waste any time
  24. 0:40and just get straight into this one.
  25. 0:41Okay, so before we jump into all of the
  26. 0:43different demos, I just wanted to lay a
  27. 0:44foundation here, which is how much does
  28. 0:46Sonnet 5.5 actually cost versus Opus
  29. 0:485.5? The answer is roughly half. So here
  30. 0:51you can see Sonnet 5.5 is $2 per million
  31. 0:53input tokens whereas Opus is four and
  32. 0:56Sonnet is $10 for the million output
  33. 0:57tokens whereas Opus 5.5 is 20. So Opus
  34. 1:01is roughly double the cost of Sonnet
  35. 1:025.5. So we're here to see if Opus is
  36. 1:04double the quality of Sonet 5.5 and
  37. 1:07where to use each model because Claude
  38. 1:09actually came out themselves and dropped
  39. 1:10this little guide that said, "Hey,
  40. 1:11here's when you should use each model.
  41. 1:13If you have something well scoped for
  42. 1:14everyday coding, fixing bugs, sonnet it
  43. 1:17high volume everyday development,
  44. 1:18sonnet. polished documents, slides,
  45. 1:20spreadsheets, one pagers, sonnet. You
  46. 1:22should basically be using Opus if it's
  47. 1:23complex work requiring careful judgment,
  48. 1:25including long horizon, agentic coding,
  49. 1:27and knowledge work. And for your hardest
  50. 1:29problems where you need the most
  51. 1:31intelligence, opus 5.5. I really like
  52. 1:33this section down here. Sonnet 5.5 fits
  53. 1:36best when the task has a clear spec and
  54. 1:38a way to check the result. And so, the
  55. 1:40way that I interpret this is basically
  56. 1:41if you have a task with an objective
  57. 1:43definition of done, then try that out
  58. 1:45with Sonnet 5.5 first. If you need some
  59. 1:47more creativity and you're looking for a
  60. 1:49thought partner to help you decide what
  61. 1:50the definition of done is and it's a bit
  62. 1:52more subjective, then that's where I
  63. 1:54would use opus as my model of choice for
  64. 1:56that task. And then what I'd do is I'd
  65. 1:58run different skills with different
  66. 1:59models and I'd see, okay, for this
  67. 2:01skill, Sonnet does better. For this
  68. 2:02skill, Opus does better. So in the
  69. 2:03future when I want to run this skill
  70. 2:05again, I know which model to use.
  71. 2:07Anyways, you guys know the drill here. I
  72. 2:09gave all of these different sessions the
  73. 2:11exact same prompt. And now we will be
  74. 2:13reviewing the results. I did Sonnet 5.5
  75. 2:15and Opus 5.5 both on high. When it comes
  76. 2:18to the actual effort level, they both
  77. 2:20used high. So the first thing was
  78. 2:22building a engaging and immersive but
  79. 2:24high converting landing page for perk
  80. 2:26form. And I said that they could use
  81. 2:27key.ai to generate anything they need.
  82. 2:29And you can see I had to correct it
  83. 2:30key.ai, not G.AI. So I should actually
  84. 2:33probably add that as a misspelling
  85. 2:34inside of my glido to correct this to
  86. 2:38key. Okay, cool. All right, so let's
  87. 2:40take a look at these two outputs. I'm
  88. 2:41not going to tell you which one was
  89. 2:42which until the end. I really like how
  90. 2:44when this one loads up, we have that
  91. 2:45text come in. Really cool. We can see
  92. 2:47that this can is interactive. It syncs
  93. 2:49to what my mouse is doing. Right off the
  94. 2:51bat, I noticed that this can is kind of
  95. 2:53a 3D rendering. So, it's not actually
  96. 2:55the exact can like pixel for pixel the
  97. 2:58way it should be. So, it had to kind of
  98. 2:59invent that. I don't love that. But, I
  99. 3:01do like the effect that it gave for us.
  100. 3:04I also like how in the back you can see
  101. 3:05Oh, wow. Actually, if I click and drag
  102. 3:07it like scrolls the can, but you can see
  103. 3:09in the background we can see like perk
  104. 3:11form. We can see text in the back, which
  105. 3:13I thought was pretty cool. Anyways,
  106. 3:14let's go ahead and scroll down. We can
  107. 3:16see the different flavors here. We can
  108. 3:17actually switch right from the hero
  109. 3:19section, which is pretty neat. As we
  110. 3:20scroll down, we get the can flooding
  111. 3:22away. And then we have two drinks,
  112. 3:23coffee, and protein coming into one,
  113. 3:26which gives us our one can, which is our
  114. 3:28perk form. We can see we scroll down
  115. 3:30into a little section where we have a
  116. 3:32gym bag, phone, keys, and as we scroll,
  117. 3:35we have the time going up, and now we
  118. 3:37grab our perk form. We have a little
  119. 3:39flavor section. So, we have the three
  120. 3:40different flavors. I'm able to see that
  121. 3:42I can click between them here. So, we
  122. 3:44have our salted caramel, we have our
  123. 3:45vanilla latte, we have our bold mocha.
  124. 3:47It's sort of starting to tell a little
  125. 3:48bit of a story here. And then that is
  126. 3:50basically the end. And we have these
  127. 3:51three dynamic cans coming back into
  128. 3:53view. Okay. So, that was one output.
  129. 3:55Here is the other one. So, right off the
  130. 3:56bat, this one isn't as like boom right
  131. 3:59in your face. It's not as much wow
  132. 4:00factor, I'd say. We do have sort of an
  133. 4:02element where it's syncing to the mouse.
  134. 4:04You can see as I'm moving, you can see
  135. 4:05these are on different layers. Also, the
  136. 4:07key is on a different layer over here.
  137. 4:08And I think that honestly this looks a
  138. 4:10little bit cheaper when we look at just
  139. 4:11kind of like the hero section. But as we
  140. 4:13scroll, we can see that those other two
  141. 4:15drinks kind of come down here. It's
  142. 4:17doing a similar style where we're like
  143. 4:18scrolling through a time. And I don't
  144. 4:21love this one as much. It's kind of
  145. 4:22making all of your eyes and all of your
  146. 4:24attention go to one spot. So it's
  147. 4:26definitely telling a different style of
  148. 4:27story here. Even though all the branding
  149. 4:29and everything else is consistent, this
  150. 4:30is using the actual real generated image
  151. 4:33inside of the brand guidelines for the
  152. 4:35product. So, that is better that it gave
  153. 4:37us the actual um I guess image in a real
  154. 4:41way rather than back here if you guys
  155. 4:43remember it was like a generated
  156. 4:44version. Here's a cool little animation
  157. 4:46here. We zoom into that image. All of
  158. 4:48these were different independently
  159. 4:50generated images. I think that this
  160. 4:51looks really nice. Ultimately, this is
  161. 4:53the better output in my mind. The one
  162. 4:55thing I don't like as much is this
  163. 4:57section. I think that this version tells
  164. 4:59a little bit of a better story, but
  165. 5:01ultimately this is the better output.
  166. 5:03Even though I don't like the hero
  167. 5:04section as much, this is the better
  168. 5:06output. So, this one was Opus 5.5 and
  169. 5:09this one was Sonet. I think Sonic killed
  170. 5:11the hero section. If this would have
  171. 5:12been a real CAN and then Opus 5.5 missed
  172. 5:15that, but everything else of Opus'
  173. 5:16version I think was better. So, if we
  174. 5:18look at the stats here, we can see that
  175. 5:21Sonnet ran for 28 minutes whereas Opus
  176. 5:23ran for 52 minutes. Sonnet cost 20
  177. 5:25million input tokens, 134K output
  178. 5:28tokens. Opus was 37 million and 148k
  179. 5:31output. and Opus cost about double here.
  180. 5:34So Opus ran for about twice as long and
  181. 5:36cost about twice as much, which is
  182. 5:39pretty consistent with what we saw when
  183. 5:41we actually look at the pricing. The
  184. 5:42question to me now would be, is Opus'
  185. 5:44output two times better? I would say no.
  186. 5:47So ultimately, what I would probably
  187. 5:48want to do here is I would reprompt
  188. 5:50Sonnet and say, "Hey, this is really
  189. 5:51good, but don't use like this fake
  190. 5:53image. Use our actual real images." And
  191. 5:55I bet that it could swap everything out.
  192. 5:57And then I bet that it could give us the
  193. 6:00level of output we were looking for with
  194. 6:01a real can and it wouldn't have costed
  195. 6:03us $13. So ultimately that's why I think
  196. 6:06that we need to give this run, we give
  197. 6:08the win here to Sonnet. But either way,
  198. 6:10this stuff is subjective. So take it as
  199. 6:12you will. But that is what we're going
  200. 6:13to do right now is I'm going to give
  201. 6:15this first output to Sonnet. And in my
  202. 6:18latest videos, you guys have been saying
  203. 6:19like, "Oh, please can you use the
  204. 6:21subscription um usage instead of the API
  205. 6:23billing?" No, I think this is a better
  206. 6:25way to show the actual cost. These costs
  207. 6:28are very much correlated to your
  208. 6:30subscription. So, just think about it
  209. 6:31like that. But the reason I'm not going
  210. 6:32to do subscription is because I don't
  211. 6:34know if you're on the 20 bucks a month
  212. 6:35plan or the 200 bucks a month plan. I
  213. 6:36also like to run all of these in
  214. 6:38parallel and I don't want to like sit
  215. 6:39and wait 52 minutes and then jot down
  216. 6:41what percent it was and then do this one
  217. 6:42for 28 minutes and then jot like, you
  218. 6:44know, I'm just gonna run this in
  219. 6:45parallel. Anyways, let's move on to
  220. 6:46number two. Real quick, guys, I've got
  221. 6:47this completely free website design
  222. 6:49skill that I'm giving away. I use this
  223. 6:50for every single website that I build,
  224. 6:52including our AI Automation Society
  225. 6:53site, which I think is pretty slick, and
  226. 6:56I absolutely love this website. It's
  227. 6:58called ScrollCraft, and it understands
  228. 6:59things like scroll driven animations,
  229. 7:01and layering, but also it has things in
  230. 7:03there like taste, typography, spacing,
  231. 7:06depth. It's just a really good website
  232. 7:08design skill in general, and it's
  233. 7:09completely free. So, the link for this
  234. 7:10will be down in the description, but
  235. 7:12let's get back to the video. All right,
  236. 7:13for number two, what I did is I said,
  237. 7:14"Hey, I need you to make me a motion
  238. 7:16showre skill about glido." and you do
  239. 7:19the research, make sure that all of this
  240. 7:21is consistent in branding, typography,
  241. 7:23you know, all of this kind of stuff. You
  242. 7:24are the motion designer here. So, just
  243. 7:26impress me. So, let's first look at
  244. 7:28Sonnet's output.
  245. 7:31Refactor this function to use uh async
  246. 7:33await instead of promises.
  247. 7:37I will fold those into the next
  248. 7:38revision. Reply saying it looks good and
  249. 7:40we can start next week.
  250. 7:42We try logic to the API client.
  251. 8:00Okay, really not bad at all. I thought
  252. 8:02there was a lot of creativity here where
  253. 8:03it's kind of showing off what the
  254. 8:04product actually looks like when you use
  255. 8:06it, which I thought was really nice. The
  256. 8:08voice over that it did, it generated
  257. 8:09that with maybe 11 Labs or something. I
  258. 8:11also really liked how these pixels here
  259. 8:13formed a G and that actually turned into
  260. 8:16an AI generated video that it made. So,
  261. 8:18I thought that was pretty nice. Overall,
  262. 8:20I was honestly not expecting a Sonnet
  263. 8:22model to do this good with this motion
  264. 8:24show reel. Now, let's go take a look at
  265. 8:26Opus' output.
  266. 8:29Refactor this function to use async
  267. 8:31await instead of promises.
  268. 8:58Okay. I mean, I don't think you can beat
  269. 9:00that. Opus 5.5 is just so good as of now
  270. 9:02at motion design. It just felt more
  271. 9:04glido. the background, the different
  272. 9:07animations it made, it just felt like
  273. 9:09even the copy was better. It also went
  274. 9:11to our website and pulled this quote
  275. 9:13from me that was on the site, which was
  276. 9:15very fast. But right here, this was
  277. 9:16actually the quote from the website. So
  278. 9:18ultimately, this output much better from
  279. 9:21Opus. Now, when we look at the cost
  280. 9:23here, Opus was faster and it was also
  281. 9:27less output tokens, but ultimately it
  282. 9:29was more expensive. So, Sonnet was
  283. 9:31$14.37 whereas Opus was $2055.
  284. 9:34I think though we're gonna give this one
  285. 9:37to Opus. If I'm still looking through
  286. 9:38the lens of if we had Sonnet try again
  287. 9:41to get up to $20.5,
  288. 9:43I still think Opus' would be a better
  289. 9:45output. It just had a better taste with
  290. 9:47like motion and the sound design. And
  291. 9:49sound design is something that I think
  292. 9:50is really hard to get right even as a
  293. 9:52human sometimes. And the fact that Opus
  294. 9:53is doing this so consistently, very
  295. 9:55good. So, I'm giving this one to Opus.
  296. 9:57Okay, so for number three, I said very,
  297. 10:00very vague. create me a researchbacked
  298. 10:0290-day action plan for getting how they
  299. 10:04AI to a thousand subscribers. So, let's
  300. 10:06go first take a look at Sonnet's output.
  301. 10:08Okay, so one thing to call out here is
  302. 10:10both of these models originally when I
  303. 10:11shot off that prompt, they just gave me
  304. 10:13a markdown file. And then for both of
  305. 10:14them, I gave them the exact same second
  306. 10:16prompt, which was turn that into some
  307. 10:18sort of document that I can view that's
  308. 10:20easier for me to read. And this is
  309. 10:22Sonnet's version. It created me an HTML
  310. 10:23file and it gave it to me locally. And
  311. 10:25then Opus did the same thing except for
  312. 10:26it put it as a clawed artifact. So,
  313. 10:28that's one little tiny difference, but
  314. 10:30besides that, this is Sonnet's output.
  315. 10:32Let's see. 1,000 subscribers in 90 days.
  316. 10:34It also threw our logo up here, AIS
  317. 10:36Media, September 28th. We have day one
  318. 10:38launch as well as the day 90 goal. It's
  319. 10:41interesting that it chose to launch this
  320. 10:42on November 2nd rather than like next
  321. 10:45month or something, Q4. Anyways, we have
  322. 10:47a timeline here. So, we have episodes on
  323. 10:50the bottom. We have subscriber gates and
  324. 10:53getting to a,000 by day 90. We can see
  325. 10:55that we have some other things here like
  326. 10:57promotions. Test send full list full
  327. 10:59list. We have Nate promotion sends out
  328. 11:02six of them. Community posts five of
  329. 11:04them. 500 subs. Holidays locked.
  330. 11:06Interesting. We have different phases.
  331. 11:08So what to do from October to November
  332. 11:10and then two weeks on launch and then a
  333. 11:12month in the next stage and then
  334. 11:14compounding. So it's showing us the
  335. 11:16different sort of milestones to hit in
  336. 11:17these different phases. It's showing us
  337. 11:19the different sources that we can get
  338. 11:20subscribers from. None of this is
  339. 11:21interactive. doesn't look like. It's
  340. 11:22just showing us what this could look
  341. 11:24like. We can also see that we got some
  342. 11:26scroll animations coming in there, which
  343. 11:27I thought was a nice touch. As you can
  344. 11:28see, the text comes in. It's also
  345. 11:30looking at shorts and LinkedIn and guest
  346. 11:32kit and scorecards and other decisions
  347. 11:35like that. Okay, so this is good. I
  348. 11:36wouldn't say that this is like super
  349. 11:38super helpful for me at the moment, but
  350. 11:40not too bad. And here we have the
  351. 11:41version with also scroll animations from
  352. 11:43Opus as a cloud artifact. This one chose
  353. 11:45to start October 1st to December 29th.
  354. 11:47So, it's actually like a Q4 90-day
  355. 11:50sprint, which I think is a bit more
  356. 11:51aligned based on like what's going on in
  357. 11:53our business right now. So, now we have
  358. 11:55the 90 days. This one gave us a little
  359. 11:57bit of a different sort of view, which I
  360. 12:00actually like better. I'd say we can see
  361. 12:02when we're going to launch. And from now
  362. 12:04until launch, we'd be kind of setting up
  363. 12:06auditions and we'd be doing guest
  364. 12:07applications and we'd be, you know,
  365. 12:09announcing all of this kind of stuff,
  366. 12:10recording these episodes. we'd be
  367. 12:12looking at getting out shorts and
  368. 12:14hitting the email list and community
  369. 12:16posts and all this kind of stuff. I
  370. 12:17think that this view is a little bit
  371. 12:19easier to digest than what's going on
  372. 12:21here. They're just a bit too much going
  373. 12:23on here and it's not as clear. Whereas
  374. 12:24this seems a little bit more clear to
  375. 12:26me. We have different phases here, but
  376. 12:27this one's basically each month, which I
  377. 12:28think is better. Build, launch,
  378. 12:31compound. Same thing. Where do these
  379. 12:32actual subscribers come from? The list,
  380. 12:34YouTube organic, Nate's channel, things
  381. 12:36like that. Here are things that we
  382. 12:37actually need to decide on. And then we
  383. 12:39can open the full plan which actually
  384. 12:41takes us to oh wow this takes us to like
  385. 12:44an actual very specific write up with
  386. 12:47like tasks. So this is basically
  387. 12:49something that I could throw into my
  388. 12:50OTAA now. And we have these actual tasks
  389. 12:52that we can check off. Okay, I think
  390. 12:54this one's pretty clear. Opus wins this
  391. 12:56test. Now when we take a look at the
  392. 12:58stats, Opus ran for 20 minutes. Sonnet
  393. 13:00ran for 14. Sonnet cost a little under
  394. 13:02$5, whereas Opus costs a little over $9.
  395. 13:05So, if we had Sonnet take another stab
  396. 13:07at it, where would Sonnet be? Now, this
  397. 13:10one gets really interesting to me
  398. 13:11because if you think about the actual
  399. 13:12prompt, I didn't really give it
  400. 13:15guidelines at all. I said, "Create me a
  401. 13:17researchbacked 90-day action plan." That
  402. 13:19was it. And if we think about what I
  403. 13:21said earlier, if you have an objective
  404. 13:23definition of done, use Sonnet. If you
  405. 13:25need more creativity and you're looking
  406. 13:26for a thought partner, use Opus. And
  407. 13:28this is definitely an example of me
  408. 13:29looking for a thought partner and
  409. 13:31looking for a model to just take a
  410. 13:32ambitious goal or sort of a vague goal
  411. 13:34and just run with it. I think that if I
  412. 13:36would have given both of these models a
  413. 13:38very very specific prompt on here's what
  414. 13:40I want, here's how it should look,
  415. 13:41here's blah blah blah, I think that opus
  416. 13:44and sonnet would have given me a similar
  417. 13:45result but sonnets would have been
  418. 13:46cheaper. So I do really think that
  419. 13:48that's interesting to think about the
  420. 13:50way that you talk to these different
  421. 13:51large language models. But in this case
  422. 13:52I can't ignore how much better opus did
  423. 13:55with this vague prompt even with the
  424. 13:56cost. So, this one is going to go to
  425. 13:58Opus as well. So, right now we're at
  426. 14:00Opus is winning two to one. And really,
  427. 14:01guys, I'm not trying to come in here and
  428. 14:03say like which model is better than
  429. 14:04which because ultimately Opus is a
  430. 14:06better model, but I'm just trying to
  431. 14:07hopefully show you guys like now you
  432. 14:09might have a better idea of when to use
  433. 14:11each model and or how to use each model.
  434. 14:13So, let's hop into number four. All
  435. 14:15right. So, in this one, I gave them a
  436. 14:17bunch of company data, mock company
  437. 14:19data, and I wanted to get an Excel sheet
  438. 14:22as well as a investor pitch deck. So,
  439. 14:26we're going to do Opus first. Here is
  440. 14:28the deck that Opus built. It generated
  441. 14:29this image in the back. We've got 18
  442. 14:31slides. We've got generated images here
  443. 14:33as well. We have the problem. We have
  444. 14:35the product. We have who we serve. We
  445. 14:37have some traction, so some statistics
  446. 14:39in here based on the mock data. We've
  447. 14:41got the business model, customers. Now,
  448. 14:44this doesn't look too bad. We've got
  449. 14:46visuals. We've got, you know, not too
  450. 14:47much going on on each slide, but I will
  451. 14:49say this is the infamous clawed design
  452. 14:52font with the Fs like that and the
  453. 14:54numbers. It doesn't look super premium
  454. 14:56in my mind. Like this doesn't feel as
  455. 14:57premium as I'd like it to, but either
  456. 14:59way, the slides aren't designed super
  457. 15:01super poorly or anything. So, that is
  458. 15:03the output. I mean, this really isn't
  459. 15:04bad at all from Opus 5.5. And then on
  460. 15:06the Excel sheet, we have a dashboard,
  461. 15:08inputs, monthly metrics, we have
  462. 15:10different views. The dashboard is really
  463. 15:11nice with all these stats and all of
  464. 15:12these different visualizations in here.
  465. 15:14We have the different inputs. We have
  466. 15:16metrics, quarterly, P&L. What I really
  467. 15:18like is in a lot of these cells, we see
  468. 15:21formulas. We see formulas basically
  469. 15:23everywhere. Which means if we had to
  470. 15:24come in here and change up the data, the
  471. 15:26whole sheet would update. I don't like
  472. 15:28when AI models will give you a Excel
  473. 15:31sheet with no formulas and now you have
  474. 15:33to basically it's all static. It's all
  475. 15:35hard-coded data and if you needed to
  476. 15:36make changes, you would have to go ask
  477. 15:38the agent to make those changes for you.
  478. 15:40I hate when it does that. So, it's
  479. 15:42really good that Opus is putting I mean
  480. 15:44there's tons and tons of slides here,
  481. 15:45like a ton of slides or sheets, I guess,
  482. 15:48tabs. What I like is that they have
  483. 15:49formulas inside. So, it's really not bad
  484. 15:51at all from Opus. We've even got some
  485. 15:53color coding and stuff. Very good Excel
  486. 15:55sheet. This is interesting because if
  487. 15:57you guys remember earlier, Opus made an
  488. 15:59artifact and Sonnet didn't. But in this
  489. 16:01time, Sonnet made an artifact. So, let's
  490. 16:03see how this one's looking. This is
  491. 16:05Bright Path Analytics. We see our intro
  492. 16:08right there. We have I believe this is
  493. 16:10the exact same font that Opus was using.
  494. 16:13So anyways, we have um company snapshot,
  495. 16:15the problem, the product, who we serve,
  496. 16:19business model, traction. I mean, this
  497. 16:21is very very similar. Like there's not
  498. 16:22really any difference in my mind between
  499. 16:25the two different decks. They feel and
  500. 16:28look very very similar and sonnet
  501. 16:31actually gave us more slides. Now, let's
  502. 16:33open up the Excel sheet we got from
  503. 16:34Sonnet. Immediately, it doesn't look as
  504. 16:36premium, but we go into the dashboard.
  505. 16:38The dashboard looks just fine. We have a
  506. 16:40lot of visualizations here. monthly
  507. 16:42data. We can see that we have lots of
  508. 16:44formulas in these calculation cells.
  509. 16:47Same thing with the P&L. We have some
  510. 16:49SAS metrics. We have assumptions. We
  511. 16:52have organic forecast. All of these are
  512. 16:54different lookups and formulas as well.
  513. 16:56Wow. Lots of different tabs here.
  514. 16:58Pipeline, customers, marketing,
  515. 17:00headcount. So, this one honestly might
  516. 17:01have given us like too much. I don't
  517. 17:02know. If this was real data and this was
  518. 17:04my real company, I'd be able to tell you
  519. 17:06guys better like how useful all this is.
  520. 17:08But ultimately, I wouldn't really be
  521. 17:10able to come in here and say that one of
  522. 17:12these outputs doing this sort of like
  523. 17:14document creation was significantly
  524. 17:16better than one of the other AI models.
  525. 17:19So, I really think in this case,
  526. 17:20whichever model was cheaper and faster
  527. 17:22is who I'm going to say won this round.
  528. 17:24And wow, in this case, Opus was actually
  529. 17:27cheaper and faster. So, we don't even
  530. 17:29have to discuss this one. Opus wins this
  531. 17:30one. No question. Cheaper, faster, and
  532. 17:33the outputs were pretty much identical
  533. 17:35in my mind. Now, what if we had some
  534. 17:37skills built around it? That would be
  535. 17:38another interesting thing to test. But
  536. 17:40in this case, I didn't utilize a skill.
  537. 17:41I just said, "Hey, make these things for
  538. 17:43me." All right. So, here's an example
  539. 17:44where it actually did use a skill. I
  540. 17:46have a skill called HTML explainer, and
  541. 17:48I asked it to use that skill or I didn't
  542. 17:50ask it to, but it did. Where I said,
  543. 17:51"Hey, what models could I run on my
  544. 17:53current device? I don't know anything
  545. 17:54about this, so make it easy to
  546. 17:55understand. Lots of pictures, blah,
  547. 17:56blah, blah." So, let's take a look at
  548. 17:57these outputs. All right, so this one is
  549. 17:59Opus. What AI can my PC run? You can see
  550. 18:01this is branded. It's just a super
  551. 18:03simple, you know, HTML explainer style
  552. 18:06skill that I use sometimes. So, we can
  553. 18:08see we have graphics card, memory,
  554. 18:10processor, storage. We see the graphics
  555. 18:12card memory versus other things, which
  556. 18:14is like a gaming PC at a data center
  557. 18:16chip. The 32 GB rule. The model has to
  558. 18:18fit in your graphics card. So, we
  559. 18:20probably couldn't run this 43 GB model,
  560. 18:22but we could run a 20 GB model. So, that
  561. 18:24is a nice little simple rule. We can see
  562. 18:26that we also have shunk copies. We have
  563. 18:28speed, so tokens per second on our
  564. 18:31graphics card. What to install? Everyday
  565. 18:33chat. It gives us a suggestion. Coding
  566. 18:35hard problems. Gemma 4 for fast replies.
  567. 18:38Super super cool. Too big for one PC.
  568. 18:40Start tonight. It tells us exactly what
  569. 18:42we would need to do. Okay, cool. Very
  570. 18:44simple. Not too bad at all. Okay. And
  571. 18:46now we have the exact same skill, exact
  572. 18:48same prompt, but this is the sonnet
  573. 18:50version. So, local AI models your PC can
  574. 18:52run. It actually made us a little
  575. 18:53visual, which I thought was cool. 32 GB,
  576. 18:55the number that decides what local
  577. 18:57means. Okay, so it starts off with what
  578. 18:59local means, which I don't think the
  579. 19:00other one did. The other one basically
  580. 19:02just went straight into RPC. So that's
  581. 19:04good that it did that. It broke it down
  582. 19:05simply with another little
  583. 19:07visualization. Three places to keep
  584. 19:08model desk, shelf, warehouse. Okay.
  585. 19:11Fast, slow, storage only. What fits?
  586. 19:13Green fits, amber crawls, red won't run.
  587. 19:16So it doesn't exactly tell me why
  588. 19:17because of the whole gigabytes thing,
  589. 19:19but it does show a nice little
  590. 19:20visualization and some other AI models.
  591. 19:22So that's not too bad. It also shows us
  592. 19:25speed, pick by the job, similar to the
  593. 19:27last one, and how to actually try one
  594. 19:29out right now. So ultimately, if we
  595. 19:32really think about how would we want to
  596. 19:34understand how this stuff works, I do
  597. 19:36think that Opus' output was better. Like
  598. 19:40this one is more visual, I'd say, but it
  599. 19:43doesn't actually explain it to me as
  600. 19:45well as the Opus one did, unfortunately.
  601. 19:47So I would say that Opus wins this
  602. 19:48output. But if we think about this cost
  603. 19:51here, similar timing, Sonnet cost half
  604. 19:54of what Opus cost. So, if I prompted
  605. 19:57Sonnet one more time and said, "Hey, I
  606. 19:59am still confused about this, this, and
  607. 20:00this. Can you just add some more info
  608. 20:03here?" I bet that Sonnet's output would
  609. 20:05be better. And I would also argue that I
  610. 20:07think that Sonnet's output visually is
  611. 20:09better. And that's one of the the pieces
  612. 20:11of this skill, which is a very, very
  613. 20:13simple skill. It's not refined at all,
  614. 20:15but ultimately, I think that if we
  615. 20:17prompted Sonnet again, we would get an
  616. 20:19output better than what Opus gave us for
  617. 20:21$3. So, I'm going to give this one to
  618. 20:23Sonnet. Okay. So, in this next one,
  619. 20:25number six, I said, "Scrape X for all
  620. 20:27the news that's been going on today.
  621. 20:29Give me an interactive dashboard so I
  622. 20:30can view what's going on and see what's
  623. 20:32coming out and just get prepped for all
  624. 20:34these different things going on in the
  625. 20:35AI space." So, let's take a look at
  626. 20:36Sonnet's output. This is a local host
  627. 20:39that it created from me. We can see that
  628. 20:40we can see when this was last scraped.
  629. 20:43We can hit refresh. We can see how many
  630. 20:45posts were tracked. So, 386 posts
  631. 20:47tracked from 676 scraped. We can see
  632. 20:50some headline news right up top. We can
  633. 20:52see what is trending. We can mark fits
  634. 20:54my channel or hide covered or pinned
  635. 20:57only. Okay, interesting. We can also see
  636. 20:59the heat score. So, Claude Sonet 5.5 100
  637. 21:02heat metastarts meta enterprise platform
  638. 21:0494 heat. We can maybe click into this
  639. 21:07stuff. Nope, I can just hover over. I
  640. 21:08can open details. Okay, cool. So, I can
  641. 21:10actually open the actual post that it's
  642. 21:12sourcing, which is pretty nice. We can
  643. 21:14mark it covered. We can add to plan. So,
  644. 21:17now we get this list with like a content
  645. 21:19plan over there. Not too bad. We can
  646. 21:21also sort by newest. We can sort by
  647. 21:23channel fit. We can look at the last 48
  648. 21:25hours. We can see what's coming this
  649. 21:27week, dev day by OpenAI. We can see the
  650. 21:29calendar. Okay, this is not a bad view
  651. 21:31at all. And now here we have Opus'
  652. 21:32version, which off the bat just kind of
  653. 21:34looks a little bit better to me. You can
  654. 21:36see that this one's still scraping X. As
  655. 21:38you can see, it's showing us a little
  656. 21:39progress bar. So, it is working, but
  657. 21:42this view looks a little bit more
  658. 21:44visual. It's showing us posts per hour
  659. 21:45on AI. It's showing us how many posts
  660. 21:47are rising and cooling off. And this
  661. 21:50view, in my opinion, is just a little
  662. 21:52bit better. I can also switch up here
  663. 21:53this week. Okay, nice little view here.
  664. 21:56We can see content plan, so I can add
  665. 21:58things to my list. We can see the raw
  666. 21:59feed as well. So, that's pretty cool. I
  667. 22:01can see sort of a heat score as well.
  668. 22:03It's tagging things by rumor or shipped
  669. 22:06or discussion. And it's pulling up real
  670. 22:08posts on the side here. And let's see,
  671. 22:10how do I add something to right there? I
  672. 22:13can click add to content plan. I can
  673. 22:14open the top post and it goes to X.
  674. 22:16Okay, very cool. So, I'm really starting
  675. 22:18to understand a pattern here, which is
  676. 22:20what I said at the beginning. If you
  677. 22:22know exactly what you want, Sonnet is
  678. 22:23probably going to be able to do a good
  679. 22:24job for you. But if you need the
  680. 22:25creativity and you send an open-ended,
  681. 22:28very vague goal, Opus is just going to
  682. 22:30handle it better. And that's become
  683. 22:32very, very clear to me. In this case,
  684. 22:34Opus ran for about 7 minutes longer and
  685. 22:36cost $6 more. And if we iterated four
  686. 22:40more times with Sonnet, or as much as it
  687. 22:41took to get to $8 of output, but we had
  688. 22:44a clearer goal, then Sonnet would
  689. 22:46definitely give us that level of output.
  690. 22:48There's nothing in here that Sonnet
  691. 22:50would have not been able to code or
  692. 22:51create for us. It's just that Sonnet
  693. 22:53doesn't think about that as much and it
  694. 22:55won't go kind of like outside the box as
  695. 22:57much as Opus will to give you a better
  696. 22:59experience. It's basically just going to
  697. 23:01give you what it thinks you want and
  698. 23:03it's going to be the the baseline. So,
  699. 23:05if we're sticking with my rule here,
  700. 23:06because Sonnet was much cheaper, I'm
  701. 23:08going to give this one to Sonnet. But
  702. 23:09ultimately, like right now on one
  703. 23:11prompt, Opus' output was better there
  704. 23:12and there's just no denying that. So,
  705. 23:14let's go on to the last one, number
  706. 23:16seven. All right. So, in this one, I
  707. 23:17gave Opus and Sonnet a link to one of my
  708. 23:20YouTube videos and I said, "Create me a
  709. 23:21YouTube resource guide." So, this is
  710. 23:23using a skill once again. So, the skill
  711. 23:25is defining what good looks like
  712. 23:27essentially. So, let's take a look at
  713. 23:29Sonnet's output. First, we have the AIS
  714. 23:32header up top. We have this font. We
  715. 23:34have Nate Herk linking to my YouTube
  716. 23:35channel. It starts off with the
  717. 23:37challenge and the rules. It goes over
  718. 23:39the S&P benchmark. It goes over the
  719. 23:42actual steps of how I built this. We
  720. 23:44have the problem, the solution, day one
  721. 23:46and day two results, day three, day
  722. 23:48four, day five, day six, day seven,
  723. 23:49final results, lessons learned, and we
  724. 23:51have key takeaways down here. They're
  725. 23:53pretty much the exact same length. This
  726. 23:55one is like seven pages, but it's really
  727. 23:58like I don't know, there's a big gap
  728. 24:00here, whereas this one, very similar.
  729. 24:02Same link, same title, same header, the
  730. 24:04overview, the rules, the benchmark, how
  731. 24:06the system was built, um day one, day
  732. 24:09two, final results, all this kind of
  733. 24:11stuff. It's very very similar, which is
  734. 24:13why this one would be essentially
  735. 24:16whichever one was cheaper is the one
  736. 24:18that I would go with here. In this case,
  737. 24:19you can see that Sonnet was cheaper and
  738. 24:21faster about half the speed. So twice as
  739. 24:25fast and a dollar40
  740. 24:27cheaper. So this one is going to go to
  741. 24:29Sonnet because the outputs were not too
  742. 24:31different enough to actually beble to
  743. 24:32say, "Oh, one's really better than the
  744. 24:34other." And it's because in this case, I
  745. 24:36know we've seen some other examples
  746. 24:37where it didn't always match, but in
  747. 24:39this case, I had a skill that I do use a
  748. 24:41lot. And a model like Sonnet that's less
  749. 24:43intelligent than Opus was able to just
  750. 24:44follow my instructions because it knew
  751. 24:46what I wanted more clearly. So, this one
  752. 24:47goes to Sonnet. So, ultimately in this
  753. 24:50test, we had three, four go to Sonnet
  754. 24:53and three go to Opus. So, we did
  755. 24:55technically have Sonnet winning this
  756. 24:56challenge, but I think you guys
  757. 24:57understand this wasn't really a
  758. 24:58head-to-head. It was more like how and
  759. 25:00when to use each model. And here are the
  760. 25:01final stats for you guys. You can see
  761. 25:03across these seven sessions, Opus ran
  762. 25:05for about 46 minutes longer. It spent
  763. 25:08about 38 million input tokens more as
  764. 25:11well as actually less output tokens,
  765. 25:13which is interesting. But it did cost
  766. 25:15about $16 more than Sonnet 5.5 in all
  767. 25:17these runs, which when you think about
  768. 25:19the fact that it's actually double the
  769. 25:20cost of Sonnet, it didn't cost double
  770. 25:22the cost in this experiment, which is
  771. 25:24also interesting. I've talked a lot
  772. 25:26about how I see each of these models
  773. 25:27fitting into my workflow now. And
  774. 25:29hopefully you guys have a better idea of
  775. 25:30how this all should work for you. But
  776. 25:32ultimately, you got to get in there and
  777. 25:33you got to test it on your own skills
  778. 25:35and your own prompts and your own
  779. 25:36processes. So, that's going to do it for
  780. 25:38this one. That was my experiment today.
  781. 25:39I hope you guys enjoyed. I hope you
  782. 25:40learned something new. And if you did,
  783. 25:41please give it a like. It helps me out a
  784. 25:42ton. And as always, I appreciate you
  785. 25:44guys making it to the end of the video.
  786. 25:45I'll see you on the next one. Thanks,
  787. 25:47everyone.

About this transcript

This page contains the full transcript of I Tested Sonnet 5.5 vs Opus 5.5. What You Need to Know. by Nate Herk | AI Automation, generated from the public captions YouTube serves with the video. The transcript has 5,745 words across 787 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.