I Tested Sonnet 5.5 vs Opus 5.5. What You Need to Know. — Transcript
Full transcript
- 0:00Cloud Sonnet 5.5 is here and it is a
- 0:02clear upgrade over Claude Sonnet 5.
- 0:04Running 30% faster as well as costing
- 0:0630% less for most work. So, I've been
- 0:08playing around with it all day on my
- 0:10actual real workflows. Things like
- 0:12building websites, taking a bunch of
- 0:13messy data and turning that into
- 0:15spreadsheets. Also, Excel sheets with
- 0:17different views as well as formulas
- 0:18inside of every single cell, motion
- 0:20design and editing videos, interactive
- 0:22dashboards that have real-time data
- 0:24syncing, project management and
- 0:25planning, and so many other things as
- 0:27well. But specifically, I've been
- 0:28testing Son of 5.5 against Opus 5.5, so
- 0:31you know exactly when you should be
- 0:32using which model. Now, in all these
- 0:34comparisons, I'll go over how fast they
- 0:35are, the input and output tokens, and
- 0:37how much they actually cost you in
- 0:38dollars. So, let's not waste any time
- 0:40and just get straight into this one.
- 0:41Okay, so before we jump into all of the
- 0:43different demos, I just wanted to lay a
- 0:44foundation here, which is how much does
- 0:46Sonnet 5.5 actually cost versus Opus
- 0:485.5? The answer is roughly half. So here
- 0:51you can see Sonnet 5.5 is $2 per million
- 0:53input tokens whereas Opus is four and
- 0:56Sonnet is $10 for the million output
- 0:57tokens whereas Opus 5.5 is 20. So Opus
- 1:01is roughly double the cost of Sonnet
- 1:025.5. So we're here to see if Opus is
- 1:04double the quality of Sonet 5.5 and
- 1:07where to use each model because Claude
- 1:09actually came out themselves and dropped
- 1:10this little guide that said, "Hey,
- 1:11here's when you should use each model.
- 1:13If you have something well scoped for
- 1:14everyday coding, fixing bugs, sonnet it
- 1:17high volume everyday development,
- 1:18sonnet. polished documents, slides,
- 1:20spreadsheets, one pagers, sonnet. You
- 1:22should basically be using Opus if it's
- 1:23complex work requiring careful judgment,
- 1:25including long horizon, agentic coding,
- 1:27and knowledge work. And for your hardest
- 1:29problems where you need the most
- 1:31intelligence, opus 5.5. I really like
- 1:33this section down here. Sonnet 5.5 fits
- 1:36best when the task has a clear spec and
- 1:38a way to check the result. And so, the
- 1:40way that I interpret this is basically
- 1:41if you have a task with an objective
- 1:43definition of done, then try that out
- 1:45with Sonnet 5.5 first. If you need some
- 1:47more creativity and you're looking for a
- 1:49thought partner to help you decide what
- 1:50the definition of done is and it's a bit
- 1:52more subjective, then that's where I
- 1:54would use opus as my model of choice for
- 1:56that task. And then what I'd do is I'd
- 1:58run different skills with different
- 1:59models and I'd see, okay, for this
- 2:01skill, Sonnet does better. For this
- 2:02skill, Opus does better. So in the
- 2:03future when I want to run this skill
- 2:05again, I know which model to use.
- 2:07Anyways, you guys know the drill here. I
- 2:09gave all of these different sessions the
- 2:11exact same prompt. And now we will be
- 2:13reviewing the results. I did Sonnet 5.5
- 2:15and Opus 5.5 both on high. When it comes
- 2:18to the actual effort level, they both
- 2:20used high. So the first thing was
- 2:22building a engaging and immersive but
- 2:24high converting landing page for perk
- 2:26form. And I said that they could use
- 2:27key.ai to generate anything they need.
- 2:29And you can see I had to correct it
- 2:30key.ai, not G.AI. So I should actually
- 2:33probably add that as a misspelling
- 2:34inside of my glido to correct this to
- 2:38key. Okay, cool. All right, so let's
- 2:40take a look at these two outputs. I'm
- 2:41not going to tell you which one was
- 2:42which until the end. I really like how
- 2:44when this one loads up, we have that
- 2:45text come in. Really cool. We can see
- 2:47that this can is interactive. It syncs
- 2:49to what my mouse is doing. Right off the
- 2:51bat, I noticed that this can is kind of
- 2:53a 3D rendering. So, it's not actually
- 2:55the exact can like pixel for pixel the
- 2:58way it should be. So, it had to kind of
- 2:59invent that. I don't love that. But, I
- 3:01do like the effect that it gave for us.
- 3:04I also like how in the back you can see
- 3:05Oh, wow. Actually, if I click and drag
- 3:07it like scrolls the can, but you can see
- 3:09in the background we can see like perk
- 3:11form. We can see text in the back, which
- 3:13I thought was pretty cool. Anyways,
- 3:14let's go ahead and scroll down. We can
- 3:16see the different flavors here. We can
- 3:17actually switch right from the hero
- 3:19section, which is pretty neat. As we
- 3:20scroll down, we get the can flooding
- 3:22away. And then we have two drinks,
- 3:23coffee, and protein coming into one,
- 3:26which gives us our one can, which is our
- 3:28perk form. We can see we scroll down
- 3:30into a little section where we have a
- 3:32gym bag, phone, keys, and as we scroll,
- 3:35we have the time going up, and now we
- 3:37grab our perk form. We have a little
- 3:39flavor section. So, we have the three
- 3:40different flavors. I'm able to see that
- 3:42I can click between them here. So, we
- 3:44have our salted caramel, we have our
- 3:45vanilla latte, we have our bold mocha.
- 3:47It's sort of starting to tell a little
- 3:48bit of a story here. And then that is
- 3:50basically the end. And we have these
- 3:51three dynamic cans coming back into
- 3:53view. Okay. So, that was one output.
- 3:55Here is the other one. So, right off the
- 3:56bat, this one isn't as like boom right
- 3:59in your face. It's not as much wow
- 4:00factor, I'd say. We do have sort of an
- 4:02element where it's syncing to the mouse.
- 4:04You can see as I'm moving, you can see
- 4:05these are on different layers. Also, the
- 4:07key is on a different layer over here.
- 4:08And I think that honestly this looks a
- 4:10little bit cheaper when we look at just
- 4:11kind of like the hero section. But as we
- 4:13scroll, we can see that those other two
- 4:15drinks kind of come down here. It's
- 4:17doing a similar style where we're like
- 4:18scrolling through a time. And I don't
- 4:21love this one as much. It's kind of
- 4:22making all of your eyes and all of your
- 4:24attention go to one spot. So it's
- 4:26definitely telling a different style of
- 4:27story here. Even though all the branding
- 4:29and everything else is consistent, this
- 4:30is using the actual real generated image
- 4:33inside of the brand guidelines for the
- 4:35product. So, that is better that it gave
- 4:37us the actual um I guess image in a real
- 4:41way rather than back here if you guys
- 4:43remember it was like a generated
- 4:44version. Here's a cool little animation
- 4:46here. We zoom into that image. All of
- 4:48these were different independently
- 4:50generated images. I think that this
- 4:51looks really nice. Ultimately, this is
- 4:53the better output in my mind. The one
- 4:55thing I don't like as much is this
- 4:57section. I think that this version tells
- 4:59a little bit of a better story, but
- 5:01ultimately this is the better output.
- 5:03Even though I don't like the hero
- 5:04section as much, this is the better
- 5:06output. So, this one was Opus 5.5 and
- 5:09this one was Sonet. I think Sonic killed
- 5:11the hero section. If this would have
- 5:12been a real CAN and then Opus 5.5 missed
- 5:15that, but everything else of Opus'
- 5:16version I think was better. So, if we
- 5:18look at the stats here, we can see that
- 5:21Sonnet ran for 28 minutes whereas Opus
- 5:23ran for 52 minutes. Sonnet cost 20
- 5:25million input tokens, 134K output
- 5:28tokens. Opus was 37 million and 148k
- 5:31output. and Opus cost about double here.
- 5:34So Opus ran for about twice as long and
- 5:36cost about twice as much, which is
- 5:39pretty consistent with what we saw when
- 5:41we actually look at the pricing. The
- 5:42question to me now would be, is Opus'
- 5:44output two times better? I would say no.
- 5:47So ultimately, what I would probably
- 5:48want to do here is I would reprompt
- 5:50Sonnet and say, "Hey, this is really
- 5:51good, but don't use like this fake
- 5:53image. Use our actual real images." And
- 5:55I bet that it could swap everything out.
- 5:57And then I bet that it could give us the
- 6:00level of output we were looking for with
- 6:01a real can and it wouldn't have costed
- 6:03us $13. So ultimately that's why I think
- 6:06that we need to give this run, we give
- 6:08the win here to Sonnet. But either way,
- 6:10this stuff is subjective. So take it as
- 6:12you will. But that is what we're going
- 6:13to do right now is I'm going to give
- 6:15this first output to Sonnet. And in my
- 6:18latest videos, you guys have been saying
- 6:19like, "Oh, please can you use the
- 6:21subscription um usage instead of the API
- 6:23billing?" No, I think this is a better
- 6:25way to show the actual cost. These costs
- 6:28are very much correlated to your
- 6:30subscription. So, just think about it
- 6:31like that. But the reason I'm not going
- 6:32to do subscription is because I don't
- 6:34know if you're on the 20 bucks a month
- 6:35plan or the 200 bucks a month plan. I
- 6:36also like to run all of these in
- 6:38parallel and I don't want to like sit
- 6:39and wait 52 minutes and then jot down
- 6:41what percent it was and then do this one
- 6:42for 28 minutes and then jot like, you
- 6:44know, I'm just gonna run this in
- 6:45parallel. Anyways, let's move on to
- 6:46number two. Real quick, guys, I've got
- 6:47this completely free website design
- 6:49skill that I'm giving away. I use this
- 6:50for every single website that I build,
- 6:52including our AI Automation Society
- 6:53site, which I think is pretty slick, and
- 6:56I absolutely love this website. It's
- 6:58called ScrollCraft, and it understands
- 6:59things like scroll driven animations,
- 7:01and layering, but also it has things in
- 7:03there like taste, typography, spacing,
- 7:06depth. It's just a really good website
- 7:08design skill in general, and it's
- 7:09completely free. So, the link for this
- 7:10will be down in the description, but
- 7:12let's get back to the video. All right,
- 7:13for number two, what I did is I said,
- 7:14"Hey, I need you to make me a motion
- 7:16showre skill about glido." and you do
- 7:19the research, make sure that all of this
- 7:21is consistent in branding, typography,
- 7:23you know, all of this kind of stuff. You
- 7:24are the motion designer here. So, just
- 7:26impress me. So, let's first look at
- 7:28Sonnet's output.
- 7:31Refactor this function to use uh async
- 7:33await instead of promises.
- 7:37I will fold those into the next
- 7:38revision. Reply saying it looks good and
- 7:40we can start next week.
- 7:42We try logic to the API client.
- 8:00Okay, really not bad at all. I thought
- 8:02there was a lot of creativity here where
- 8:03it's kind of showing off what the
- 8:04product actually looks like when you use
- 8:06it, which I thought was really nice. The
- 8:08voice over that it did, it generated
- 8:09that with maybe 11 Labs or something. I
- 8:11also really liked how these pixels here
- 8:13formed a G and that actually turned into
- 8:16an AI generated video that it made. So,
- 8:18I thought that was pretty nice. Overall,
- 8:20I was honestly not expecting a Sonnet
- 8:22model to do this good with this motion
- 8:24show reel. Now, let's go take a look at
- 8:26Opus' output.
- 8:29Refactor this function to use async
- 8:31await instead of promises.
- 8:58Okay. I mean, I don't think you can beat
- 9:00that. Opus 5.5 is just so good as of now
- 9:02at motion design. It just felt more
- 9:04glido. the background, the different
- 9:07animations it made, it just felt like
- 9:09even the copy was better. It also went
- 9:11to our website and pulled this quote
- 9:13from me that was on the site, which was
- 9:15very fast. But right here, this was
- 9:16actually the quote from the website. So
- 9:18ultimately, this output much better from
- 9:21Opus. Now, when we look at the cost
- 9:23here, Opus was faster and it was also
- 9:27less output tokens, but ultimately it
- 9:29was more expensive. So, Sonnet was
- 9:31$14.37 whereas Opus was $2055.
- 9:34I think though we're gonna give this one
- 9:37to Opus. If I'm still looking through
- 9:38the lens of if we had Sonnet try again
- 9:41to get up to $20.5,
- 9:43I still think Opus' would be a better
- 9:45output. It just had a better taste with
- 9:47like motion and the sound design. And
- 9:49sound design is something that I think
- 9:50is really hard to get right even as a
- 9:52human sometimes. And the fact that Opus
- 9:53is doing this so consistently, very
- 9:55good. So, I'm giving this one to Opus.
- 9:57Okay, so for number three, I said very,
- 10:00very vague. create me a researchbacked
- 10:0290-day action plan for getting how they
- 10:04AI to a thousand subscribers. So, let's
- 10:06go first take a look at Sonnet's output.
- 10:08Okay, so one thing to call out here is
- 10:10both of these models originally when I
- 10:11shot off that prompt, they just gave me
- 10:13a markdown file. And then for both of
- 10:14them, I gave them the exact same second
- 10:16prompt, which was turn that into some
- 10:18sort of document that I can view that's
- 10:20easier for me to read. And this is
- 10:22Sonnet's version. It created me an HTML
- 10:23file and it gave it to me locally. And
- 10:25then Opus did the same thing except for
- 10:26it put it as a clawed artifact. So,
- 10:28that's one little tiny difference, but
- 10:30besides that, this is Sonnet's output.
- 10:32Let's see. 1,000 subscribers in 90 days.
- 10:34It also threw our logo up here, AIS
- 10:36Media, September 28th. We have day one
- 10:38launch as well as the day 90 goal. It's
- 10:41interesting that it chose to launch this
- 10:42on November 2nd rather than like next
- 10:45month or something, Q4. Anyways, we have
- 10:47a timeline here. So, we have episodes on
- 10:50the bottom. We have subscriber gates and
- 10:53getting to a,000 by day 90. We can see
- 10:55that we have some other things here like
- 10:57promotions. Test send full list full
- 10:59list. We have Nate promotion sends out
- 11:02six of them. Community posts five of
- 11:04them. 500 subs. Holidays locked.
- 11:06Interesting. We have different phases.
- 11:08So what to do from October to November
- 11:10and then two weeks on launch and then a
- 11:12month in the next stage and then
- 11:14compounding. So it's showing us the
- 11:16different sort of milestones to hit in
- 11:17these different phases. It's showing us
- 11:19the different sources that we can get
- 11:20subscribers from. None of this is
- 11:21interactive. doesn't look like. It's
- 11:22just showing us what this could look
- 11:24like. We can also see that we got some
- 11:26scroll animations coming in there, which
- 11:27I thought was a nice touch. As you can
- 11:28see, the text comes in. It's also
- 11:30looking at shorts and LinkedIn and guest
- 11:32kit and scorecards and other decisions
- 11:35like that. Okay, so this is good. I
- 11:36wouldn't say that this is like super
- 11:38super helpful for me at the moment, but
- 11:40not too bad. And here we have the
- 11:41version with also scroll animations from
- 11:43Opus as a cloud artifact. This one chose
- 11:45to start October 1st to December 29th.
- 11:47So, it's actually like a Q4 90-day
- 11:50sprint, which I think is a bit more
- 11:51aligned based on like what's going on in
- 11:53our business right now. So, now we have
- 11:55the 90 days. This one gave us a little
- 11:57bit of a different sort of view, which I
- 12:00actually like better. I'd say we can see
- 12:02when we're going to launch. And from now
- 12:04until launch, we'd be kind of setting up
- 12:06auditions and we'd be doing guest
- 12:07applications and we'd be, you know,
- 12:09announcing all of this kind of stuff,
- 12:10recording these episodes. we'd be
- 12:12looking at getting out shorts and
- 12:14hitting the email list and community
- 12:16posts and all this kind of stuff. I
- 12:17think that this view is a little bit
- 12:19easier to digest than what's going on
- 12:21here. They're just a bit too much going
- 12:23on here and it's not as clear. Whereas
- 12:24this seems a little bit more clear to
- 12:26me. We have different phases here, but
- 12:27this one's basically each month, which I
- 12:28think is better. Build, launch,
- 12:31compound. Same thing. Where do these
- 12:32actual subscribers come from? The list,
- 12:34YouTube organic, Nate's channel, things
- 12:36like that. Here are things that we
- 12:37actually need to decide on. And then we
- 12:39can open the full plan which actually
- 12:41takes us to oh wow this takes us to like
- 12:44an actual very specific write up with
- 12:47like tasks. So this is basically
- 12:49something that I could throw into my
- 12:50OTAA now. And we have these actual tasks
- 12:52that we can check off. Okay, I think
- 12:54this one's pretty clear. Opus wins this
- 12:56test. Now when we take a look at the
- 12:58stats, Opus ran for 20 minutes. Sonnet
- 13:00ran for 14. Sonnet cost a little under
- 13:02$5, whereas Opus costs a little over $9.
- 13:05So, if we had Sonnet take another stab
- 13:07at it, where would Sonnet be? Now, this
- 13:10one gets really interesting to me
- 13:11because if you think about the actual
- 13:12prompt, I didn't really give it
- 13:15guidelines at all. I said, "Create me a
- 13:17researchbacked 90-day action plan." That
- 13:19was it. And if we think about what I
- 13:21said earlier, if you have an objective
- 13:23definition of done, use Sonnet. If you
- 13:25need more creativity and you're looking
- 13:26for a thought partner, use Opus. And
- 13:28this is definitely an example of me
- 13:29looking for a thought partner and
- 13:31looking for a model to just take a
- 13:32ambitious goal or sort of a vague goal
- 13:34and just run with it. I think that if I
- 13:36would have given both of these models a
- 13:38very very specific prompt on here's what
- 13:40I want, here's how it should look,
- 13:41here's blah blah blah, I think that opus
- 13:44and sonnet would have given me a similar
- 13:45result but sonnets would have been
- 13:46cheaper. So I do really think that
- 13:48that's interesting to think about the
- 13:50way that you talk to these different
- 13:51large language models. But in this case
- 13:52I can't ignore how much better opus did
- 13:55with this vague prompt even with the
- 13:56cost. So, this one is going to go to
- 13:58Opus as well. So, right now we're at
- 14:00Opus is winning two to one. And really,
- 14:01guys, I'm not trying to come in here and
- 14:03say like which model is better than
- 14:04which because ultimately Opus is a
- 14:06better model, but I'm just trying to
- 14:07hopefully show you guys like now you
- 14:09might have a better idea of when to use
- 14:11each model and or how to use each model.
- 14:13So, let's hop into number four. All
- 14:15right. So, in this one, I gave them a
- 14:17bunch of company data, mock company
- 14:19data, and I wanted to get an Excel sheet
- 14:22as well as a investor pitch deck. So,
- 14:26we're going to do Opus first. Here is
- 14:28the deck that Opus built. It generated
- 14:29this image in the back. We've got 18
- 14:31slides. We've got generated images here
- 14:33as well. We have the problem. We have
- 14:35the product. We have who we serve. We
- 14:37have some traction, so some statistics
- 14:39in here based on the mock data. We've
- 14:41got the business model, customers. Now,
- 14:44this doesn't look too bad. We've got
- 14:46visuals. We've got, you know, not too
- 14:47much going on on each slide, but I will
- 14:49say this is the infamous clawed design
- 14:52font with the Fs like that and the
- 14:54numbers. It doesn't look super premium
- 14:56in my mind. Like this doesn't feel as
- 14:57premium as I'd like it to, but either
- 14:59way, the slides aren't designed super
- 15:01super poorly or anything. So, that is
- 15:03the output. I mean, this really isn't
- 15:04bad at all from Opus 5.5. And then on
- 15:06the Excel sheet, we have a dashboard,
- 15:08inputs, monthly metrics, we have
- 15:10different views. The dashboard is really
- 15:11nice with all these stats and all of
- 15:12these different visualizations in here.
- 15:14We have the different inputs. We have
- 15:16metrics, quarterly, P&L. What I really
- 15:18like is in a lot of these cells, we see
- 15:21formulas. We see formulas basically
- 15:23everywhere. Which means if we had to
- 15:24come in here and change up the data, the
- 15:26whole sheet would update. I don't like
- 15:28when AI models will give you a Excel
- 15:31sheet with no formulas and now you have
- 15:33to basically it's all static. It's all
- 15:35hard-coded data and if you needed to
- 15:36make changes, you would have to go ask
- 15:38the agent to make those changes for you.
- 15:40I hate when it does that. So, it's
- 15:42really good that Opus is putting I mean
- 15:44there's tons and tons of slides here,
- 15:45like a ton of slides or sheets, I guess,
- 15:48tabs. What I like is that they have
- 15:49formulas inside. So, it's really not bad
- 15:51at all from Opus. We've even got some
- 15:53color coding and stuff. Very good Excel
- 15:55sheet. This is interesting because if
- 15:57you guys remember earlier, Opus made an
- 15:59artifact and Sonnet didn't. But in this
- 16:01time, Sonnet made an artifact. So, let's
- 16:03see how this one's looking. This is
- 16:05Bright Path Analytics. We see our intro
- 16:08right there. We have I believe this is
- 16:10the exact same font that Opus was using.
- 16:13So anyways, we have um company snapshot,
- 16:15the problem, the product, who we serve,
- 16:19business model, traction. I mean, this
- 16:21is very very similar. Like there's not
- 16:22really any difference in my mind between
- 16:25the two different decks. They feel and
- 16:28look very very similar and sonnet
- 16:31actually gave us more slides. Now, let's
- 16:33open up the Excel sheet we got from
- 16:34Sonnet. Immediately, it doesn't look as
- 16:36premium, but we go into the dashboard.
- 16:38The dashboard looks just fine. We have a
- 16:40lot of visualizations here. monthly
- 16:42data. We can see that we have lots of
- 16:44formulas in these calculation cells.
- 16:47Same thing with the P&L. We have some
- 16:49SAS metrics. We have assumptions. We
- 16:52have organic forecast. All of these are
- 16:54different lookups and formulas as well.
- 16:56Wow. Lots of different tabs here.
- 16:58Pipeline, customers, marketing,
- 17:00headcount. So, this one honestly might
- 17:01have given us like too much. I don't
- 17:02know. If this was real data and this was
- 17:04my real company, I'd be able to tell you
- 17:06guys better like how useful all this is.
- 17:08But ultimately, I wouldn't really be
- 17:10able to come in here and say that one of
- 17:12these outputs doing this sort of like
- 17:14document creation was significantly
- 17:16better than one of the other AI models.
- 17:19So, I really think in this case,
- 17:20whichever model was cheaper and faster
- 17:22is who I'm going to say won this round.
- 17:24And wow, in this case, Opus was actually
- 17:27cheaper and faster. So, we don't even
- 17:29have to discuss this one. Opus wins this
- 17:30one. No question. Cheaper, faster, and
- 17:33the outputs were pretty much identical
- 17:35in my mind. Now, what if we had some
- 17:37skills built around it? That would be
- 17:38another interesting thing to test. But
- 17:40in this case, I didn't utilize a skill.
- 17:41I just said, "Hey, make these things for
- 17:43me." All right. So, here's an example
- 17:44where it actually did use a skill. I
- 17:46have a skill called HTML explainer, and
- 17:48I asked it to use that skill or I didn't
- 17:50ask it to, but it did. Where I said,
- 17:51"Hey, what models could I run on my
- 17:53current device? I don't know anything
- 17:54about this, so make it easy to
- 17:55understand. Lots of pictures, blah,
- 17:56blah, blah." So, let's take a look at
- 17:57these outputs. All right, so this one is
- 17:59Opus. What AI can my PC run? You can see
- 18:01this is branded. It's just a super
- 18:03simple, you know, HTML explainer style
- 18:06skill that I use sometimes. So, we can
- 18:08see we have graphics card, memory,
- 18:10processor, storage. We see the graphics
- 18:12card memory versus other things, which
- 18:14is like a gaming PC at a data center
- 18:16chip. The 32 GB rule. The model has to
- 18:18fit in your graphics card. So, we
- 18:20probably couldn't run this 43 GB model,
- 18:22but we could run a 20 GB model. So, that
- 18:24is a nice little simple rule. We can see
- 18:26that we also have shunk copies. We have
- 18:28speed, so tokens per second on our
- 18:31graphics card. What to install? Everyday
- 18:33chat. It gives us a suggestion. Coding
- 18:35hard problems. Gemma 4 for fast replies.
- 18:38Super super cool. Too big for one PC.
- 18:40Start tonight. It tells us exactly what
- 18:42we would need to do. Okay, cool. Very
- 18:44simple. Not too bad at all. Okay. And
- 18:46now we have the exact same skill, exact
- 18:48same prompt, but this is the sonnet
- 18:50version. So, local AI models your PC can
- 18:52run. It actually made us a little
- 18:53visual, which I thought was cool. 32 GB,
- 18:55the number that decides what local
- 18:57means. Okay, so it starts off with what
- 18:59local means, which I don't think the
- 19:00other one did. The other one basically
- 19:02just went straight into RPC. So that's
- 19:04good that it did that. It broke it down
- 19:05simply with another little
- 19:07visualization. Three places to keep
- 19:08model desk, shelf, warehouse. Okay.
- 19:11Fast, slow, storage only. What fits?
- 19:13Green fits, amber crawls, red won't run.
- 19:16So it doesn't exactly tell me why
- 19:17because of the whole gigabytes thing,
- 19:19but it does show a nice little
- 19:20visualization and some other AI models.
- 19:22So that's not too bad. It also shows us
- 19:25speed, pick by the job, similar to the
- 19:27last one, and how to actually try one
- 19:29out right now. So ultimately, if we
- 19:32really think about how would we want to
- 19:34understand how this stuff works, I do
- 19:36think that Opus' output was better. Like
- 19:40this one is more visual, I'd say, but it
- 19:43doesn't actually explain it to me as
- 19:45well as the Opus one did, unfortunately.
- 19:47So I would say that Opus wins this
- 19:48output. But if we think about this cost
- 19:51here, similar timing, Sonnet cost half
- 19:54of what Opus cost. So, if I prompted
- 19:57Sonnet one more time and said, "Hey, I
- 19:59am still confused about this, this, and
- 20:00this. Can you just add some more info
- 20:03here?" I bet that Sonnet's output would
- 20:05be better. And I would also argue that I
- 20:07think that Sonnet's output visually is
- 20:09better. And that's one of the the pieces
- 20:11of this skill, which is a very, very
- 20:13simple skill. It's not refined at all,
- 20:15but ultimately, I think that if we
- 20:17prompted Sonnet again, we would get an
- 20:19output better than what Opus gave us for
- 20:21$3. So, I'm going to give this one to
- 20:23Sonnet. Okay. So, in this next one,
- 20:25number six, I said, "Scrape X for all
- 20:27the news that's been going on today.
- 20:29Give me an interactive dashboard so I
- 20:30can view what's going on and see what's
- 20:32coming out and just get prepped for all
- 20:34these different things going on in the
- 20:35AI space." So, let's take a look at
- 20:36Sonnet's output. This is a local host
- 20:39that it created from me. We can see that
- 20:40we can see when this was last scraped.
- 20:43We can hit refresh. We can see how many
- 20:45posts were tracked. So, 386 posts
- 20:47tracked from 676 scraped. We can see
- 20:50some headline news right up top. We can
- 20:52see what is trending. We can mark fits
- 20:54my channel or hide covered or pinned
- 20:57only. Okay, interesting. We can also see
- 20:59the heat score. So, Claude Sonet 5.5 100
- 21:02heat metastarts meta enterprise platform
- 21:0494 heat. We can maybe click into this
- 21:07stuff. Nope, I can just hover over. I
- 21:08can open details. Okay, cool. So, I can
- 21:10actually open the actual post that it's
- 21:12sourcing, which is pretty nice. We can
- 21:14mark it covered. We can add to plan. So,
- 21:17now we get this list with like a content
- 21:19plan over there. Not too bad. We can
- 21:21also sort by newest. We can sort by
- 21:23channel fit. We can look at the last 48
- 21:25hours. We can see what's coming this
- 21:27week, dev day by OpenAI. We can see the
- 21:29calendar. Okay, this is not a bad view
- 21:31at all. And now here we have Opus'
- 21:32version, which off the bat just kind of
- 21:34looks a little bit better to me. You can
- 21:36see that this one's still scraping X. As
- 21:38you can see, it's showing us a little
- 21:39progress bar. So, it is working, but
- 21:42this view looks a little bit more
- 21:44visual. It's showing us posts per hour
- 21:45on AI. It's showing us how many posts
- 21:47are rising and cooling off. And this
- 21:50view, in my opinion, is just a little
- 21:52bit better. I can also switch up here
- 21:53this week. Okay, nice little view here.
- 21:56We can see content plan, so I can add
- 21:58things to my list. We can see the raw
- 21:59feed as well. So, that's pretty cool. I
- 22:01can see sort of a heat score as well.
- 22:03It's tagging things by rumor or shipped
- 22:06or discussion. And it's pulling up real
- 22:08posts on the side here. And let's see,
- 22:10how do I add something to right there? I
- 22:13can click add to content plan. I can
- 22:14open the top post and it goes to X.
- 22:16Okay, very cool. So, I'm really starting
- 22:18to understand a pattern here, which is
- 22:20what I said at the beginning. If you
- 22:22know exactly what you want, Sonnet is
- 22:23probably going to be able to do a good
- 22:24job for you. But if you need the
- 22:25creativity and you send an open-ended,
- 22:28very vague goal, Opus is just going to
- 22:30handle it better. And that's become
- 22:32very, very clear to me. In this case,
- 22:34Opus ran for about 7 minutes longer and
- 22:36cost $6 more. And if we iterated four
- 22:40more times with Sonnet, or as much as it
- 22:41took to get to $8 of output, but we had
- 22:44a clearer goal, then Sonnet would
- 22:46definitely give us that level of output.
- 22:48There's nothing in here that Sonnet
- 22:50would have not been able to code or
- 22:51create for us. It's just that Sonnet
- 22:53doesn't think about that as much and it
- 22:55won't go kind of like outside the box as
- 22:57much as Opus will to give you a better
- 22:59experience. It's basically just going to
- 23:01give you what it thinks you want and
- 23:03it's going to be the the baseline. So,
- 23:05if we're sticking with my rule here,
- 23:06because Sonnet was much cheaper, I'm
- 23:08going to give this one to Sonnet. But
- 23:09ultimately, like right now on one
- 23:11prompt, Opus' output was better there
- 23:12and there's just no denying that. So,
- 23:14let's go on to the last one, number
- 23:16seven. All right. So, in this one, I
- 23:17gave Opus and Sonnet a link to one of my
- 23:20YouTube videos and I said, "Create me a
- 23:21YouTube resource guide." So, this is
- 23:23using a skill once again. So, the skill
- 23:25is defining what good looks like
- 23:27essentially. So, let's take a look at
- 23:29Sonnet's output. First, we have the AIS
- 23:32header up top. We have this font. We
- 23:34have Nate Herk linking to my YouTube
- 23:35channel. It starts off with the
- 23:37challenge and the rules. It goes over
- 23:39the S&P benchmark. It goes over the
- 23:42actual steps of how I built this. We
- 23:44have the problem, the solution, day one
- 23:46and day two results, day three, day
- 23:48four, day five, day six, day seven,
- 23:49final results, lessons learned, and we
- 23:51have key takeaways down here. They're
- 23:53pretty much the exact same length. This
- 23:55one is like seven pages, but it's really
- 23:58like I don't know, there's a big gap
- 24:00here, whereas this one, very similar.
- 24:02Same link, same title, same header, the
- 24:04overview, the rules, the benchmark, how
- 24:06the system was built, um day one, day
- 24:09two, final results, all this kind of
- 24:11stuff. It's very very similar, which is
- 24:13why this one would be essentially
- 24:16whichever one was cheaper is the one
- 24:18that I would go with here. In this case,
- 24:19you can see that Sonnet was cheaper and
- 24:21faster about half the speed. So twice as
- 24:25fast and a dollar40
- 24:27cheaper. So this one is going to go to
- 24:29Sonnet because the outputs were not too
- 24:31different enough to actually beble to
- 24:32say, "Oh, one's really better than the
- 24:34other." And it's because in this case, I
- 24:36know we've seen some other examples
- 24:37where it didn't always match, but in
- 24:39this case, I had a skill that I do use a
- 24:41lot. And a model like Sonnet that's less
- 24:43intelligent than Opus was able to just
- 24:44follow my instructions because it knew
- 24:46what I wanted more clearly. So, this one
- 24:47goes to Sonnet. So, ultimately in this
- 24:50test, we had three, four go to Sonnet
- 24:53and three go to Opus. So, we did
- 24:55technically have Sonnet winning this
- 24:56challenge, but I think you guys
- 24:57understand this wasn't really a
- 24:58head-to-head. It was more like how and
- 25:00when to use each model. And here are the
- 25:01final stats for you guys. You can see
- 25:03across these seven sessions, Opus ran
- 25:05for about 46 minutes longer. It spent
- 25:08about 38 million input tokens more as
- 25:11well as actually less output tokens,
- 25:13which is interesting. But it did cost
- 25:15about $16 more than Sonnet 5.5 in all
- 25:17these runs, which when you think about
- 25:19the fact that it's actually double the
- 25:20cost of Sonnet, it didn't cost double
- 25:22the cost in this experiment, which is
- 25:24also interesting. I've talked a lot
- 25:26about how I see each of these models
- 25:27fitting into my workflow now. And
- 25:29hopefully you guys have a better idea of
- 25:30how this all should work for you. But
- 25:32ultimately, you got to get in there and
- 25:33you got to test it on your own skills
- 25:35and your own prompts and your own
- 25:36processes. So, that's going to do it for
- 25:38this one. That was my experiment today.
- 25:39I hope you guys enjoyed. I hope you
- 25:40learned something new. And if you did,
- 25:41please give it a like. It helps me out a
- 25:42ton. And as always, I appreciate you
- 25:44guys making it to the end of the video.
- 25:45I'll see you on the next one. Thanks,
- 25:47everyone.
About this transcript
This page contains the full transcript of I Tested Sonnet 5.5 vs Opus 5.5. What You Need to Know. by Nate Herk | AI Automation, generated from the public captions YouTube serves with the video. The transcript has 5,745 words across 787 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.