YT YouTube2Text

YouTube2Text → Transcripts → finally a good small model

finally a good small model — Transcript

by Theo - t3․gg · 8,307 words · 1,184 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Anthropic is pretty good at making big
0:01models for coding. We've seen that with
0:03models like Fable and Opus and all the
0:05way back in the day with models like
0:06Sonnet 3.5 and the idea of tool calls
0:09kind of making us software engineers use
0:11AI the way that we do today. But there's
0:13always been a bit of a weakness with
0:14their lineup. It's the smaller models.
0:16And as cool as Haiku 4.5 was when it
0:19came out, it came out in October of last
0:21year and it was expensive at the time.
0:23Now it's just laughable. And I have put
0:26a lot of effort into adjusting all of my
0:28tooling, all of my agent and Claude MDs
0:29and all those things to make sure every
0:31tool in my system makes it very, very
0:33clear to all my agents that Haiku 4.5
0:36should be avoided at all costs because
0:39it's useless. It's just not very good.
0:42That's why I made this post a few weeks
0:44ago where I said that Anthropic has no
0:46small models that are worth using right
0:48now and also dunked on OpenAI saying
0:49they have no large models worth using
0:50cuz at the time they had Astra and 6.0
0:53Soul. They didn't have 6.1 Soul yet. And
0:55of course Google gets the fun jab at the
0:57end saying none of the Google models are
0:58worth using. But the thing I want to
1:00focus on here is this idea that
1:01Anthropic has no small models worth
1:04using. Historically, Anthropic puts
1:06almost all their effort into the biggest
1:08model and then poorly distills it into
1:10the cheaper ones. That's why when they
1:12had Mythos and Fable, the quality of
1:14Opus plummeted and we ended up with 4.6,
1:174.7, 4.8 and then eventually Opus 5
1:19which was terrible. That was all because
1:21of their focus on Anthropic's large
1:23models. That curse has been broken
1:25though with both Opus 5.5 and Sonnet
1:285.5. Which is why I'm so hyped for
1:31today's introduction of Haiku 5.5.
1:34They've been teasing this one for a bit
1:36and I I had high hopes but low
1:38expectations because this is one of the
1:40biggest missing pieces. The model the
1:42other models use to search for files, to
1:45triple check changes, to do all the
1:47things that a model shouldn't waste
1:49tokens on if the model is expensive. I
1:51got to the point where I had taught my
1:53Claude to call Soul in order to save
1:55money because I didn't trust it calling
1:57Haiku 4.5. So, how did Haiku 5.5 turn
2:01out? I don't want to spoil too much, but
2:04I think it's fair to say
2:06they made something pretty cool. The
2:08price is right, the performance seems
2:10solid, has a handful of tricks that make
2:12it a little extra special, especially
2:14considering how small this model is. But
2:16before we can get started, in honor of
2:18the nature of the small model, we should
2:20take a really small break for today's
2:21sponsor. One of the painful side effects
2:23of having such good models for coding is
2:25that all of our code bases are getting
2:26way bigger and more complex. And while
2:28AI review tools are good at helping us
2:29avoid bugs, are they actually helping us
2:31prevent major security issues? And even
2:34if it does find one, is it actually
2:35going to let you fix it? Because let's
2:36be real, most of these models that are
2:38this good just refuse outright once you
2:40start doing security stuff. That's why
2:42I've been so reliant on today's sponsor.
2:44You've already heard of them, it's Code
2:45Rabbit, but I'm showing something
2:46different today. But if you remember a
2:48few seconds ago, I said that the models
2:50can't really help with the security
2:51things because they're not allowed to.
2:53Well, Code Rabbit has made all of the
2:54deals they need to and found all the
2:56models they have to to get real security
2:58feedback for you as you're working. Not
2:59only will they give you real security
3:01callouts in your pull requests, they'll
3:02give you the tools needed to fix them.
3:04There's just like a little fix button
3:05that will run on their side with the
3:07models that are approved to do this.
3:08They also have something that's even
3:09more important, which is the ability to
3:11run deep scans across an entire repo.
3:14You click this little button in the
3:15corner, you choose the repo you want to
3:16run on, and then half an hour to an hour
3:18later, you have a ton of real feedback.
3:21I ended up addressing over half the
3:22things that were found in both of these
3:24reviews. And if I hop to this findings
3:26tab, I'm going to need my editor to
3:27start censoring things because the
3:28things it finds are actually quite
3:30useful to know about. And I'll be real,
3:32some of these are bad and I need to go
3:33address them. As our software's getting
3:35more complex, keeping it secure is only
3:36getting more important. Secure your
3:38stuff at swyd.link/coderabbit. Before we
3:40dive into everything about Haiku, I want
3:43to show one thing quick on Slapolitics.
3:46We currently show all of Anthropic's
3:47current models. Sadly, we don't have
3:49available 5.5 yet, but we do have Opus
3:505.5, Sonnet 5.5, and now Haiku. Let's
3:52just show the last Haiku model real
3:55quick.
3:56Yeah. Hopefully not much more needs to
3:58be said. Let's go through the
4:00announcement. Introducing Haiku 5.5, the
4:03cheapest, fastest, and most capable
4:05small model we've ever released. Haiku
4:075.5 is designed for high-volume,
4:09cost-sensitive tasks. It reliably
4:11handles quick and repetitive workloads,
4:13things like summaries, compactions,
4:15database queries, and classification
4:16requests. It pairs well with Opus and
4:18Sonnet as a sub-agent on coding work.
4:20And since it's also our fastest model to
4:22date, it works especially well for
4:24speed-sensitive tasks like live customer
4:26support and browser use. Haiku 5.5 is
4:28available at a much lower price than
4:304.5. On average, it's now about 75% less
4:34money to run. Along with this launch,
4:36we're making improvements to the value
4:38of our model range. We are having the
4:40price of Sonnet 5.5's cache reads. Woo!
4:42I actually missed that detail. Huge.
4:44That was the thing I roasted them on in
4:46the Sonnet 5.5 video. That is a huge
4:48change, actually, that makes the whole
4:49lineup make way more sense. That might
4:51actually be my favorite thing about this
4:52release now. Oh, [ __ ] I actually didn't
4:54know about that. So, while I sit here in
4:56rage that nobody mentioned this to me,
4:58cuz I I'm a huge fan of changes to cache
5:01read prices. I have stories I wish I
5:02could tell that I can't. If you know,
5:04you know.
5:05Let's just say I've been deep in the
5:06world of cache pricing as of recent. And
5:09the one complaint I had about Sonnet
5:11that also made Sonnet 5.5 much less
5:13appealing than 6.1 Soul from OpenAI is
5:15that the cache read price for Sonnet 5.5
5:19wasn't just too expensive for Sonnet, it
5:22was the same price as Opus. So, the
5:24thing that made up about 30 to 50% of
5:26your bill was the same for Sonnet and
5:29for Opus, which meant the price
5:31difference between these models was not
5:33particularly large if you use them for
5:35agentic work. Cache reads are essential
5:38to making these things reasonably cheap.
5:40So, with that in mind, let's look at
5:41price quick, cuz this is what's exciting
5:43to me. Sonnet 5.5 went from 20 cents per
5:46million cache reads to 10 cents, which
5:49is the same cash read price as a model
5:51like GPT-61 Soul, finally making those a
5:54little bit closer in price. I hope
5:56Artificial Analysis re-prices their
5:58charts with this new price so that I
5:59could show the different numbers cuz
6:00that's a really big improvement,
6:02especially for agentic work. Haiku 4.5
6:05was that same cash read price and then
6:07half the price across the board. Haiku
6:085.5 is a 10th the cash read price. It's
6:121 cent per mil tokens read. 12.5 cents
6:15per million cash token writes, 10 cents
6:17for normal 1 mil read, and 50 cents per
6:20mil out, where previously it was 5 bucks
6:23per mil out and Sonnet 5.5 was 10 bucks
6:26per mil out. This makes it 20x cheaper
6:28in output tokens and input tokens than
6:30Sonnet, comically cheaper in cash writes
6:32as well, about again, 20x cheaper, and
6:35then cash reads are a 10th the price,
6:38too. There is a catch, though. That is
6:40up to a 100k token context. This is a
6:43bit of a weird thing because in Tropic
6:45used to charge when you broke 270 or
6:47250k token context and doubled the
6:49price. They stopped doing that somewhat
6:51recently. That makes a lot of sense for
6:53tools like Claude Code because you don't
6:54have to like micromanage the context to
6:56keep your bill low, and cash reads are
6:58so cheap, it's kind of just fine. But a
7:01100k token limit seems very small,
7:03especially for like a long-running
7:04agentic work. I see this as a couple
7:07things. And also, I'll just share in
7:09advance here, if you go over 100k
7:10tokens, the price doesn't double, it
7:125x's. So, you go from 50 cents per mil
7:15out to 250 per mil out. You go from 10
7:16cents per mil in to 50 cents per mil in,
7:18etc. The reason for that, infra-wise, is
7:21because when you have a certain
7:23threshold of tokens in the context, that
7:25is now much more RAM you need to hold it
7:27all, which does make it more expensive.
7:29What I think happened here is they made
7:31the price for under 100k way cheaper,
7:34possibly like eating some of their own
7:36margin there, and they're making up a
7:38little bit of it with the over 100k. I
7:40see this a bit differently, though.
7:42First off, I see this as a method of
7:44making the main use case, which is quick
7:48reads and categorization checks like
7:50that, as absurdly cheap as possible.
7:52Something like Jev is the real
7:53competitor to a small model used in
7:55these ways where it's being handed text
7:57and categorizing, organizing, all these
7:58types of things. By making under 100k
8:00this cheap, they are making it harder
8:02and harder to justify spinning up
8:03another API when you could just throw it
8:05at Haiku. And then the other side is to
8:07make sure they don't bankrupt themselves
8:08when people use this inside of Claude
8:10code. This is a way of steering people
8:12towards the intended use case by making
8:14it really, really cheap if they use it
8:16on short, quick response things. But
8:18also making it reasonably expensive,
8:21still half the price Haiku was before,
8:22but not for free like it is in the under
8:26100k range if you're using it in tools
8:28like Claude code. It is worth noting
8:29that Anthropic employees like Lydia have
8:31come out publicly and said, "If you want
8:33to keep your bill cheap and not worry
8:35about this, you can manually set Haiku
8:375.5's auto compact window to 100k so
8:40that you're always working within that
8:41cheaper token pricing. Apparently it's a
8:43save per model, so this will only apply
8:45to Haiku including sub-agents. So if you
8:47go do this in your Claude code, if
8:48you're working on API billing or you're
8:50on like Bedrock or something, this
8:52allows you to guarantee the sub-agents
8:54will never cost a shitload of money. So
8:56now we need to talk about everything
8:57else, all the performance, all the use
8:59cases, and of course how it performs in
9:01my silly suite of benches. As we can see
9:04here, it absolutely decimated Haiku 4.5
9:07getting more than double the score in
9:09the vast majority of things. And one
9:11that my friends at Anthropic mentioned
9:12to me is that they were very happy to
9:15see that Haiku was no longer scoring a
9:16zero on Terminal Bench. They made their
9:19way up from zero to almost 40%. Also
9:22worth noting that as incredible as Luna
9:24is, and I still think GPT-6 Luna is an
9:25unbelievable model, GPT-6 Luna only got
9:28a 16.4%.
9:30So it more than doubled Luna's score for
9:32code. So if you're one of those people
9:34that thought Luna was decent at code, I
9:36question your judgment. I question a lot
9:37about you, honestly. But at the very
9:39least, you now have a model that is
9:40allegedly more than 2x better at coding
9:42for not too much more money. One of the
9:44most interesting numbers for me here is
9:46computer use. I've mentioned before that
9:48OpenAI has been pretty far ahead in
9:50computer use. And also, I don't think
9:52these benches are great because it
9:53doesn't show the gap as prominent as it
9:55is in my experience. Haiku slaughtered
9:58here. It did way better than 6 Luna did
10:02by like almost 40% or so. This makes
10:05this a very solid small model for doing
10:08computer use work, potentially. I
10:10haven't tried it yet. I've heard good
10:12things, though. And when you see the
10:13speeds, you'll see why I'm excited.
10:15Because Haiku 5.5 is seeing numbers from
10:18100 to 200 TPS, depending on the
10:20provider. Bias got notified by my chat
10:22that Prime tweeted he was using Haiku in
10:25some work he was doing with reasoning
10:27off for automations. He swapped to
10:28Haiku. It was a drop-in replacement. But
10:30he's actually seeing a worse pass rate
10:34and a much, much higher fail rate, as
10:36well as it being noticeably slower than
10:38what he was experiencing with Luna. This
10:40does not surprise me too much because
10:42the big thing
10:44that he did here is he turned off
10:46reasoning. He has both 6 Luna and Haiku
10:485.5 on no reasoning.
10:50OpenAI has trained their models to be
10:51good at that. Anthropic has trained
10:53their models to be bad at that. With
10:55Opus and Sonnet 5.5, if I recall
10:57correctly, I know they did with Opus. I
10:58think they did with Sonnet. They turned
10:59off no reasoning. So, you always have to
11:01let it reason at least a little bit.
11:02Haiku left no reasoning as an option
11:04because he will use it for work that
11:05they just want a response in quick. But
11:07it's not very good at that.
11:09I would honestly say that for like
11:10categorization work, if you're turning
11:12off reasoning anyways, you should just
11:13use Jev. I have been playing with the
11:15OpenAI decisions API, and I have plenty
11:17of thoughts to share about that in the
11:18near future. But for now,
11:19just use Jev unless you need like
11:21vision. And then Haiku or Luna makes
11:24sense. Probably or probably Luna cuz
11:26it's cheaper, especially with reasoning
11:27off. But uh
11:29yeah. Take that as you will. For what it
11:31is worth, when I tried out Haiku 55 on
11:34my machines with Claude code and my
11:35normal Claude code sub, I was seeing
11:37around 180 tokens per second for my just
11:41traditional prompting working on things
11:42like fish slop, which of course will be
11:44showing fish slop soon. So, while
11:45Prime's numbers with reasoning off
11:46weren't great, Haiku's numbers with
11:48reasoning on do seem very good in this
11:50tier, which means it's time to hop into
11:53slop politics. Let's look at tokens per
11:55task cuz I am genuinely very curious and
11:58for good reason. It seems like this is
12:00not the usual wasteful [ __ ] show that we
12:02see from these smaller models from
12:04OpenAI. For reference, Sonnet 5.5 on max
12:08did almost 200,000 tokens per task
12:10average, putting it at I think the
12:12highest ever on Artificial Analysis,
12:14where Fable did 78k tokens, under half
12:17as many. Yeah, it was 2.5 times the
12:19tokens per task with Sonnet 5.5, which
12:22resulted in a cost for Sonnet 55 that
12:25was slightly higher than Fable 51. The
12:27reason for this is an unreasonable
12:30setting. It's right here. It's max. So,
12:32on slop politics, you can click the word
12:34max, which makes it go away, and now we
12:36have a chart that makes a lot more sense
12:38because those max reasoning levels are
12:40bad. And now we see tokens per task
12:42numbers that make way more sense and
12:43also cost per task numbers that make way
12:45more sense. Where Haiku 55 is now all
12:47the way on the left here, cheaper than
12:48anything else, because I only have X
12:50high and max numbers for it. The max
12:52number is still pretty cheap, but more
12:54than I would want it to be, especially
12:56once you add back in models like 61
12:59Soul. Because 61 Soul on low is as cheap
13:02as Haiku on low, largely due to the
13:04token efficiency difference. But, let's
13:06go back to the cost versus intelligence
13:08cuz I think this is fun. You also see 45
13:10Haiku in here, which I am going to turn
13:13off because we don't care about that
13:14model anymore. It can finally be left
13:16where it belongs, the grave. And we'll
13:18notice some weird things here. All this
13:19data is from Artificial Analysis and it
13:21is far from representative of everything
13:23these models could or would or should be
13:25used for, but you'll see that aggregate
13:27across all their benches, Sonnet 5.5
13:30performs worse than Opus at the same if
13:33not a slightly higher cost, which meant
13:35that Opus and Sonnet aren't as
13:36complimentary as one might hope. But, if
13:38we look all the way to left here with
13:39Haiku 5.5, things are looking a lot more
13:42promising. Because even though these
13:43scores are close,
13:45Opus was almost five times as expensive
13:48as the run was with Haiku.
13:50Pretty solid. I have a fun feature here
13:53as well, the Pareto line, which is the
13:56best performance at a given price point.
13:58And this line moves around based on like
14:00what is the best you can do at that
14:01level of intelligence. And you can see
14:03some very interesting things here. Of
14:05course, as you expect, Haiku 5.5 does
14:07very well here with X-High and Max being
14:09the best value in that range within
14:11Anthropic's line of models. Sonnet 5.5
14:13sneaks in randomly with high because it
14:16is smarter than Opus low and cheaper
14:18than Opus medium. But, after Sonnet
14:20high, all the best value, according to
14:23this bench, is just Opus 5.5 medium,
14:25high, X-High, Max.
14:28But, here is where I must admit I was
14:29slightly misleading you because I only
14:31am showing Anthropic models right now.
14:33In fact, I'm only showing Anthropic
14:35models that are the latest for each of
14:37these lines. And also, Fable 5.1 doesn't
14:39even light up anymore cuz there's no
14:41reason to use that when you look at the
14:42Pareto line here. Ready for where things
14:44start to get more interesting though?
14:46First off, we can switch to linear
14:48pricing where you see the gap isn't
14:50quite as big as it seems because the log
14:52pricing makes things seem further away
14:54on the cheap side and closer on the
14:56expensive side than they really are. And
14:58you can see here that Sonnet X-High is a
15:01fine value at $2.75 per task. But, then
15:04Sonnet 5.5 Max makes literally no sense
15:06at $7.60
15:08per task. Do not if you're using Max
15:10reasoning levels, I have questions. But,
15:12I also will say very confidently, do not
15:14use Sonnet 5.5 on Max, it makes no sense
15:16at all. But, again, still just looking
15:19at Anthropic models. What happens if we
15:21put on like Gemini 4 Argon? Oh, not
15:25much. How about 38 flash? Still not
15:27much. In fact, it almost looks like we
15:28took the Haiku line and moved it to the
15:30right and down. So, we don't really need
15:32those anymore. We'll just turn those
15:33back off. How about
15:35uh I don't know, Grok 47?
15:38Still not very interesting. Still below
15:39all of this.
15:41How about 6 Astra?
15:42Oh, they got a point on. Astra on the
15:45low is around the same intelligence as
15:47Sonnet 55 on high. So, that adds to the
15:51Pareto line. But, here's where things
15:52get more interesting. Let's put on GPT-6
15:56Luna.
15:57Oh. Huh. That just extended the line a
16:00whole bunch to the left because Luna is
16:02cheaper than Haiku, always.
16:05It's also dumber, always, at least with
16:06the numbers they have. But, these are
16:08complementary. This is a pretty stable
16:10line. If you turn on the Pareto line, it
16:12has all of Luna, all of Haiku, then it
16:15briefly dips to Astra, then Sonnet, then
16:18Opus for the top. And if you were to
16:20turn off Astra
16:22and Sonnet, you have a pretty clear,
16:24simple line here for the best value per
16:26dollar.
16:27But, here is where things get messy.
16:29I'm going to turn off the Pareto line to
16:30make this clearer. I'm also going to
16:31turn off Fable and Sonnet cuz they don't
16:34really make sense here.
16:36Now, what happens when we turn on 61
16:38Sol? Oh.
16:41That's unfortunate.
16:44Turn on the Pareto line.
16:46And I guess technically Haiku is
16:48slightly smarter for meaningfully more
16:51money there, but then
16:53just barely more. Going from
16:5621.3 cents to 21.4 cents, you get a huge
17:00jump in intelligence. And if we were to
17:02just turn Haiku off here, oh, that's a
17:04much smoother line.
17:05So, if you're actually trying to
17:07optimize based on the benchmark scores
17:08for the best intelligence per dollar,
17:10your stock would look like 6 Luna, 61
17:13Sol, and Opus 55. There is one more
17:16hidden gem, diamond in the rough, that
17:18I'm surprised people aren't talking
17:19about as much.
17:20Mimo V26 Pro is an underrated open
17:23weight model. Pretty damn good for the
17:25price. But other than that,
17:28nothing really comes above this line.
17:30So, if you're trying to get the absolute
17:32most value and you're paying API prices,
17:356 Luna 61 Soul Opus gets you pretty much
17:38all of that. That said, Haiku 55 is neck
17:42and neck now. Are Soul and Luna still
17:45better per dollar? Slightly, usually.
17:47However, if you are trying to do
17:50everything through one provider, like
17:51you want to just use your Anthropic
17:52inference cuz you have a crazy spend
17:54target with them and you just use their
17:55APIs, or you're subscribed to Claude
17:57Code and don't want to also subscribe to
17:59Codex, or you want models that
18:01understand each other better cuz you're
18:02using Opus for everything and when Opus
18:04prompts Soul it's worse than when Opus
18:06prompts Haiku. I don't know how true
18:08that is, but it's real. There's a lot of
18:10reasons that you would want a model in
18:13that range in this family. And for once,
18:16it's no longer irresponsible to use
18:18Haiku like it was before. Haiku 4.5 was
18:21so bad and so unnecessarily expensive
18:25that there was no justifiable reason to
18:27use it. When I had my tweet that
18:28Anthropic has no small models worth
18:30using, OpenAI has no large models worth
18:31using, and Google has none that worth
18:32using, somebody replied, "All three
18:34statements are false." To which I
18:36replied, "LMAO, this guy uses Haiku
18:384.5." And then he got torn to teeny,
18:42tiny pieces in my replies because there
18:44is literally no reason to use Haiku 45
18:47at this point in September of this year,
18:49but also right now especially. Haiku 45
18:52was barely usable at the time and
18:54quickly became garbage and just wasn't
18:55touched because Anthropic was too
18:57focused on the big models. Now they're
18:59not. And while I do wish this model
19:01pushed the Pareto line a bit more, I am
19:03thankful Anthropic is embracing models
19:05in this range because together, these
19:08all actually look quite reasonable. Also
19:11remember, Sonnet 55 got way cheaper with
19:13the cash read price change, so that
19:15might actually adjust things even more.
19:17Good news, Artificial Analysis updated
19:19their data,
19:20which means I've updated mine. And now
19:21Sonnet is shifted quite a bit to the
19:24left, which makes it much more
19:25compelling on the Pareto line. Actually,
19:27not that much more. It does make this
19:29bottom Sonnet 5.5 high feel less awkward
19:32as a dip. But yeah, that is improved.
19:35Sonnet feels less bad now by quite a bit
19:37with that change. Okay, enough of the
19:40bench maxing. Let's talk about the model
19:42and how it actually works. Okay, one
19:44last fun thing from Anthropic's release
19:46notes here that is actually quite cool.
19:47They want to encourage people using
19:49Claude code to build more things with
19:51the Claude APIs. A lot of people, myself
19:53included, were scared of this
19:54announcement thinking this might be the
19:56end of using your sub and tools like T3
19:57Code cuz yes, in T3 Code you can use
20:00your sub with Anthropic, OpenAI, or
20:02whatever else. If it's Claude Code or
20:03Codex installed on your computer, it
20:05will just work with T3 Code. And a lot
20:07of us are scared cuz they had said they
20:08would kill that, and they haven't yet.
20:10But they did a cool thing here, actually
20:12really cool thing, where if you're
20:13subscribed as a 20x user, so you're on
20:15the $200 plan, you get $200 in API
20:18credit every month. 5x users get $100 in
20:20credit. And this is for the API. So
20:23you're effectively able to get an API
20:25key like you would if you were paying,
20:26but they pre-load it with 100 or $200
20:29every month to throw at whatever. So if
20:31you want to vibe code an app in Claude
20:33code that has Haiku or Sonnet built into
20:36it and then send it to your friends, you
20:37now have $200 of credit for that, which
20:39I actually think is really nice of them.
20:41I
20:42Do I say these things? Yeah, [ __ ] it. I
20:44chatted with them a lot about this back
20:46in the day cuz they wanted to make sure
20:47they didn't screw up things with how
20:49like Claude Code SDK and Agent SDK
20:51integrations were going to end up. They
20:54were going to do something a lot
20:55stupider with this.
20:57And I'm thankful they talked to people
20:58like me and that they listened because
21:00they turned what could have been an L
21:02into a really genuinely cool thing. That
21:04this is If you guys have noticed that
21:05I'm being nicer to Anthropic, it's cuz
21:07they're actually listening now and this
21:08is a great example of one of those
21:09things. Ooh, actually one more fun small
21:11detail here. They're updating the Claude
21:13Python and TypeScript SDKs to add
21:15support for computer use and browser
21:17use. That is potentially very useful for
21:19us. T3 code might level up from this. I
21:22just saw Elon reply to them with
21:24congrats on another great model. I would
21:26be surprised if Haiku 5.5 wasn't nicer
21:28to use than Grok 4.7 if I'm being real.
21:30Oh, I missed the artificial analysis
21:32jump chart.
21:35This is hilarious.
21:37Beautiful. From the furthest right end
21:39here to actually competitive. They are a
21:42bit more brutal showing it is not on the
21:45Pareto line, but uh yeah, the data is
21:47updated. It's more interesting than they
21:49give it credit. Regardless, I'm tired of
21:51talking about benchmarks. I want to
21:52showcase the awesome things this model
21:54can do. And not just by itself, but when
21:57it's used alongside models like Opus 5.5
22:00and Sonnet 5.5. The best use cases for
22:02Haiku 5.5 aren't clicking it in the UI
22:04or selecting it in Claude code and using
22:06it. It's letting your agents use it and
22:08letting your applications use it.
22:10Building it in so when data comes in,
22:12Haiku will audit it and categorize it.
22:14Or having Sonnet or Opus call Haiku for
22:17testing different things out, exploring
22:19the code base to find stuff. Anthropic
22:21made a really cool demo here where they
22:23had Opus 5.5 by itself try to build a
22:27safe egg drop in this emulation of like
22:30egg drop mechanics that they built. And
22:31on the other side, they had Opus 5.5
22:34plus 10 Haiku 5.5 sub agents do it. And
22:37I think the results here are actually
22:38quite interesting.
22:40The thing that you might notice pretty
22:41quickly here is that Opus is only trying
22:44one at a time cuz it's running single
22:45threaded. And if you were to have it run
22:47multiple threads, it would end up
22:49wasting way more money. Where on the
22:51right, we have Opus plus 10 Haiku sub
22:53agents and it's able to try more things
22:55at the same time and save a ton of money
22:57and get to a result faster as well.
23:00There you go.
23:01It took 3 and 1/2 minutes for Opus 5.5
23:04by itself to succeed at this task. It
23:06only designed 25 attempts and it cost
23:09$0.47 to do. But with Opus 5.5
23:11commanding 10 Haiku instances, it was
23:14able to do it in under a minute with 86
23:18attempts. Way more attempts because it's
23:20able to get more renditions out when
23:22it's that much cheaper and faster and
23:23only cost $0.14. So, for the types of
23:26tasks where the majority of answers are
23:28wrong, like if you're looking through a
23:30code base to find one file out of a
23:32bunch and you can't just do like a rip
23:34grep call to find it. Haiku will be way
23:36cheaper and faster cuz it can do a lot
23:38of things at once without causing crazy
23:39bills. So, for the subset of tasks where
23:42more attempts is more valuable than
23:44smart attempts, where you'd rather try
23:4610 times and verify each result than
23:49spend a little more time trying once and
23:51hoping it's right. If the gap between a
23:53random try from a dumb model and a
23:56thoughtful try from a smart model isn't
23:58that big, Haiku 5.5 is going to
24:00slaughter. This is a small subset of
24:02work and I am hoping Opus is smart
24:04enough to know what that subset is. I
24:06have not pushed it hard enough to know
24:07yet, but I am going to go edit my Claude
24:09MD to no longer force everything to be
24:12Opus or sole sub agents because this is
24:15cheap enough that it would make real
24:16sense. But you guys aren't here just to
24:18listen to me rant about sub agent
24:20architecture. You're here to see what
24:22the model's capable of. So, let's see
24:24some of the fun demos people have been
24:25making with Haiku 5.5. Multiple people
24:29from my community have been throwing
24:30together video demos cuz everyone knows
24:32Opus 5.5 the model got way better at
24:34programmatically creating video. And
24:36this example came from Arshan in the
24:38community.
24:42>> [music]
24:44[bell]
24:52>> This is
24:53insanely impressive for the money.
24:56God damn.
25:00Yeah, I am very impressed. And this cost
25:0260 cents an API cost to make.
25:05Was built in under 20 minutes. That's
25:07insane. A lot of why Haiku could make a
25:09video that good is that the Anthropic
25:11models have pretty good like taste built
25:14in is the best I can put it especially
25:15around design stuff. And I think they
25:17just have a lot of like really good
25:18design data and have obsessively trimmed
25:20that data set down to give the model
25:22good reference points. And you can see
25:24that even more so with the front-end
25:26design capabilities using which AI by
25:28our friend Dara.
25:30This is so comically better than Grok.
25:32Oh, that's cool actually. The little
25:34animation for how that text came and I
25:35haven't seen anything like that in one
25:36of these demos before actually.
25:40It's different, but it's it's cool. It's
25:43like a new thing. This is the blueprinty
25:45one. I don't love this diagram. It's
25:49cool it like has hover behaviors and
25:50changes things underneath.
25:52But uh
25:54not my favorite.
26:00This is kind of cute. Not my favorite. I
26:02like the fonts it chose. It's not
26:03picking cringe fonts as much as models
26:05used to.
26:07And this one's fine.
26:11If we turn off the design scale, how
26:13much worse is it?
26:15Yeah, now now we're back into classic LM
26:17design. It seems like the new Claude
26:19design skill helps their models a
26:21shitload right now. Yeah, the gap to
26:24that to this is hilarious.
26:26I had uninstalled the design skill. I
26:28definitely need to bring it back cuz it
26:29is it is much better now than it was
26:32before. These are great. And just like
26:33for comparison, we can look at Grok 4/7.
26:38>> [laughter]
26:39>> With the design skill off, it's even
26:41worse. My god. Like
26:47This one still kills me. I I'm still
26:49very very amused by this particular one.
26:53Oh, man.
26:54I'm sorry. I just I have to laugh a
26:56little, okay? So, when in chat said that
26:58they legitimately think they might have
26:59made better sites in raw HTML in
27:01elementary school than what they were
27:03seeing from Grok there, I don't
27:05disagree. Regardless, like for the the
27:08people who are picking Haiku as their
27:10model for cost-sensitive reasons or like
27:12you're using it in the $8 plan with open
27:14code and such, this is usable. It's not
27:16great. Like I'm not going to sit here
27:18and pretend that this is all you'll ever
27:20need doing front-end design and building
27:22applications, but like
27:24a model this cheap being decent is
27:26impressive. And it means like the $20
27:28Claude sub might be usable for code. I'm
27:30not going to spin one up to test it. I'm
27:32not getting more Claude accounts even
27:33for experiments. I think I have too many
27:35already if I'm being real with y'all,
27:36but uh regardless of all that, good
27:39designs. Out of curiosity, I'm throwing
27:41this model at one of my favorite tasks,
27:43which is auditing my open pull requests.
27:46T3 code has over 1,500 PR's right now.
27:48Actually, it's just under 1,500 PR's.
27:50It's probably 1,500 by the time I finish
27:51the sentence cuz we have so many coming
27:52in at all times. So, I have Haiku 55
27:56auditing all of them. Specifically in
27:58this case, it is addressing all the ones
28:00that have performance issues like PR's
28:02that are fixing performance-related
28:03stuff. And I have another thread here
28:04where it's just going to will go through
28:06and categorize all of them. Sentiently
28:07told to use six holes for sub-agents and
28:09corrected it later. You get the idea
28:10though. This is going to go through all
28:12my PR's and make me a nice HTML page
28:14showing them when it's done. While we
28:16wait for that though, I have my favorite
28:18demo.
28:19We got a new build of Fish Slap.
28:23Oof.
28:24This one has some problems.
28:27First off,
28:28no mouse move.
28:31Only moving with space, shift, and WASD.
28:35You got to turn.
28:38No sound either. I don't know if that's
28:40cuz I have it muted. Nope, it's just no
28:41sound.
28:46Yeah, it's a little rough.
28:49This is roughly the same tier as the
28:52Grok version.
28:54Which was offensively bad. I will say
28:56for this one, other than the lack of
28:57mouse move,
28:58there is no like a just like a
29:01straight-up incorrect offensively bad
29:03thing. So, for comparison with um GPD6
29:07Astra, the demo it made was stunning,
29:10but it got the like core movement wrong.
29:13Like the mouse movement was way too slow
29:15and it felt awful. And the gameplay's
29:17like core loop was a little screwy and
29:19there were like pretty apparent bugs in
29:20it.
29:21This version doesn't seem to have like
29:23like other than the mouse move thing,
29:25where it just doesn't work at all.
29:27Nothing else is like immediately like
29:29this is just [ __ ] wrong.
29:31Which is good. It's not It's not making
29:33bad decisions. It's just not making
29:36impressive ones. Like, "Oh, yeah, the
29:37corners are fucked." That's a good
29:38point, actually. Yeah, the way that this
29:39corner works is just bad. Okay, yeah,
29:42I'm taking back what I was saying there.
29:44This does have some [ __ ] stuff. It's
29:46subtle, but it has it.
29:48Also, the timing of the second alien
29:49attack made no sense. It shouldn't have
29:50been there that early. It is not
29:52honoring the original source the way I
29:54was hoping it would.
29:56It's not as like immediately offensively
29:59pathetic as a lot of other models do
30:02things. Like it's It's lows aren't as
30:04low as Astra's, but it's highs are a
30:06tenth as high as Astra's, if that's I
30:09think that's the simplest way I can put
30:10it. That bill of fish slop was about a
30:14dollar of usage total. It was a shitload
30:17of cash read tokens, 9.73
30:20million of them. A bunch of 1-hour cash
30:22writes, because it always writes 1 hour,
30:24which is annoying. New inputs,
30:26apparently only 102 tokens cuz it was
30:28caching constantly. And the output
30:30tokens were about 30 cents. 47 of 51
30:33requests exceeded 100k input tokens, so
30:34they used Haiku's higher price tier. If
30:36I had capped the context, the game
30:38probably would have come out much worse,
30:40but it would have been way cheaper. Look
30:42at that. The first PR review is in. They
30:45found 40 performance-related open PRs.
30:47Most are server or mobile fixes. Only a
30:49few are clean and high impact. Top of
30:51the list below is merge ready cuz GitHub
30:53reports clean. So it thinks that they're
30:54ready just cuz GitHub said so. It didn't
30:56seem to audit them thoroughly itself.
30:58Replaying a command it no longer freezes
31:00the server. Retry went from 18 seconds
31:01plus to 6 milliseconds. I'll do what I
31:04usually do here, which is I open up
31:06Opus. I paste the PR and I say something
31:08along the lines of this seems like it is
31:11a change worth landing. Can you figure
31:13out if that's the case, fix anything
31:15that you think needs to be fixed, and
31:16get this merged if you're confident it's
31:18ready to go. Full send. And now either
31:21that PR or something like it will be
31:22merged momentarily.
31:24The full send full stop combo has
31:26changed how I built. I'm so happy with
31:28it. While I wait for this PR summary
31:31view to generate, I want to answer one
31:33of the questions I see happening a lot
31:34in my chat. Will I use this model for
31:36code? No. I I want to be clear about
31:38this cuz I think you guys are thinking
31:40about these things wrong in general.
31:42Coding is not what makes LLMs expensive.
31:45It's not the process of going from a
31:47prompt to a JavaScript file you can
31:49execute that is the majority of your
31:51costs. Your costs come from everything
31:54else largely. Like once the context is
31:56in the model and it's cached, and
31:57generating the right code file and
31:59putting it in the right place is
32:00relatively cheap. Getting the context
32:02needed to do that is expensive.
32:04Verifying that it worked is expensive.
32:07Writing a shitload of code to test all
32:09the edges around it, to hook into GitHub
32:12so that you can monitor the PR when it's
32:13up, and all these other things, those
32:15are expensive. Regenerating it eight
32:17times because you got it wrong the first
32:19seven, that's expensive. Those things
32:21are expensive. But the act of going from
32:23all of the context and details needed to
32:25write code to the code file in your code
32:27base, even on the most expensive Fable,
32:31that is a few pennies. It is not that
32:33expensive to generate 500 output tokens
32:35once the input tokens are cached. What
32:37I'm trying to say here is I don't get
32:40why people are looking for a dumb model
32:43to do the code part. The code is cheap.
32:46The expensive part is all of the prep
32:48before you write the code file and all
32:50the verification after. And Haiku is a
32:53tool your models can use to do those
32:55parts more cheaply is valuable. And if
32:58that middle part, the coding, has edges
33:01around it, like you want to try five
33:03versions of the thing and you have a
33:05verification system where you know if it
33:06worked or not trivially, where you can
33:08run code to know, Haiku generating five
33:11or eight versions to test could be worth
33:13it. But for most problems, the smarter
33:16model writing the code is not that
33:18expensive and is preferred in general.
33:20When the model is unsure about a theory
33:22and it wants to vet four things to
33:25confirm it or wants to go look at eight
33:27apps in the world and like random
33:28open-source projects to compare against
33:30or read through a bunch of documentation
33:31to figure out some facts about some idea
33:33you have. That is when Haiku is
33:35incredible. But the execution of the
33:38plan, sure, if you write a really
33:40thorough detailed plan with a really
33:41smart model, a dumb model might be able
33:43to implement parts of it. Cool. Awesome.
33:45You already spent the money on the plan,
33:47though.
33:48Oof. This is an L for me.
33:51I hit a rate limit on GitHub, probably
33:53my fault cuz I was doing a lot of these
33:54types of things across a lot of data,
33:56but the script saved the error body as
33:58though it was API data and then failed
34:01because it would hit errors when it ran
34:02jq on that. The parallel per PR fetches
34:06that this run
34:08did used most of my 5,000 requests per
34:11hour, so now I'm locked out of T3 code
34:13GitHub integrations for a bit. The
34:15PR-star.json
34:17files parsed fine in aggregate, so their
34:18data is valid. Now it's waiting for the
34:20quota to reset, then it will refetch the
34:22181 PRs and fix the script so it rejects
34:24responses that aren't JSON arrays. Then
34:27it will recount CI and review. Cool.
34:30Except for the fact that it trigger
34:32anything to spin itself back up. It said
34:36that it's going to wait until the rate
34:37limits up, but it didn't trigger a wait.
34:39We would show that in the UI if it did.
34:41So, this thread is just done until I
34:43tell it to keep going.
34:45So,
34:46yeah, this this is why I don't like dumb
34:49models.
34:50Small models will corner themselves like
34:53this, make something harder for not just
34:55themselves, but for me. Like, this just
34:56ruined my T3 code experience for at
34:58least the next 50 minutes until my quota
35:00gets reset so I can go back to like
35:02doing PRs in side of T3 code. But, since
35:05my rate limits are [ __ ] anyways, let's
35:07show off how I would actually do this. I
35:10took the same prompt from before, but I
35:12changed two things.
35:14First, I told it to use lots of
35:16sub-agents and workflows to review these
35:17PRs and to use Haiku 55 for all
35:20sub-agents throughout the work. To say
35:22it in one more sentence, minimize work
35:24that you do yourself outside of actually
35:26getting the data and orchestrating the
35:27sub-agents. And now, the other change.
35:30I'm going to run this on Opus, because
35:31unlike Haiku, Opus is a smart model, and
35:34it should not only be able to not hit my
35:36rate limits, it should find ways to work
35:38around them, too. Here is where it's
35:40going to hit those rate limits, and it's
35:41probably going to find a way to work
35:42around it. Here it's Here's Opus being
35:45smart.
35:461,495 open PRs is a huge number to
35:48tackle with sub-agents. Maybe batches of
35:50around 15 PRs per Haiku agent will mean
35:52roughly 100 agents running. Beyond the
35:54typical small-scale workflow guidelines,
35:56but the user explicitly wants, quote,
35:57lots of sub-agents. Look at that. It's
36:00actually being thoughtful. This is going
36:02to take longer than I plan on sitting
36:03here to show you guys, but uh
36:06you get the idea.
36:07I think that Haiku is incredibly useful
36:09as a tool that you insert into your
36:11code, and that your agents can use as
36:14well, but I would not select this model
36:16as a tool I use in my coding agents,
36:20because I'm on the $200 tier, and I
36:22would rather just use Opus. And as big
36:24as the price gap is, you have to
36:26remember that when a model can't do
36:28something, and it ends up doing lots of
36:30things in the process, it might end up
36:33using more money. Cuz if Opus can solve
36:35a problem in 100k tokens, and Haiku
36:38burns a million before it figures out it
36:40can't, Haiku ends up more expensive. So,
36:43for a lot of work, not only is Haiku not
36:45able to do it, Haiku will cost more
36:47money as it fails to do it. And as
36:49things have continued to improve across
36:51the industry, in particular cost
36:53performance with both Anthropic and open
36:55AI, we've ended up in a fun place where
36:57the price gap between Opus 5 low and
37:00Haiku 5 5 max is not as big as you would
37:04think. It's 20 cents to 55 cents. So,
37:07it's around 2x more money for Opus if
37:10you're willing to deal with low. Also,
37:11we got the rest of the data for Haiku 5
37:135 because they finally finished running
37:14it, which makes Sol and Luna still quite
37:18compelling with the Pareto line. But, uh
37:20it does fill out for the Anthropic
37:23lineup quite well here, where
37:26have all of Haiku for the cheap, and
37:28most of Opus for the less cheap.
37:31This is a compelling lineup. If the only
37:32models you used were Opus and Haiku,
37:34you're probably not missing out on a
37:36whole lot. Now, Sonnet, I showed you
37:38guys Sonnet just doesn't offer much with
37:40the Pareto line. Actually, it offers
37:42nothing now. Maybe just barely on high
37:44there, interesting. But, yeah, I I think
37:46you can avoid Sonnet entirely and just
37:48use Opus and Haiku and have a very, very
37:49good time with your Anthropic models.
37:51But, that leaves us with a question.
37:53When should you use Sol?
37:54Well, first off, as you see here, Sol's
37:56actually quite a good bridge from Haiku
38:00over to Opus. It fits in the middle
38:02there. But, that's not how I would think
38:04of Sol. I wouldn't think of Sol as for
38:06things that Haiku is too dumb for, and
38:08Opus is too expensive for. I'd think of
38:10Sol as a different family with similar
38:14intelligence. Imagine that you have a
38:16team of people that work on the same
38:17things with you every day, and you have
38:19a really hard problem you have to solve.
38:20And you ask everyone around you, and you
38:22all, for the most part, agree on what
38:24the path is. That makes sense because
38:26you guys work together every day, so
38:27you've started to think about things in
38:28a similar way. Soul thinks about things
38:30in a different way. It's like you're
38:32asking your friend who works at a
38:33different company for their thoughts,
38:35and it's a unique perspective. And I
38:37found open AI models bring a unique
38:39perspective in particular with their
38:42thoroughness auditing code. I love 61
38:45Soul as an auditor. I have set up skills
38:48for all my agents where they use Opus as
38:50the thing that does the coding, and then
38:52they consult with Soul for opinions
38:54before they put up the PR, and I'm
38:56landing code so much faster simply
38:59because Soul catches all the dumb things
39:02Opus might miss. And by the time the PR
39:04is up, it's already been scrubbed by
39:05Opus and by Soul. So, I personally am
39:07sticking to my Opus and Soul duo, but
39:10when Opus needs to go collect a bunch of
39:12data or read through tons of [ __ ] or
39:14just categorize things, Haiku is going
39:17to be an incredible addition to my
39:21portfolio of models. I won't be the one
39:23to pick it, but I trust Opus to do a
39:26good enough job picking it at the times
39:27where it makes sense. And I think the
39:29result of all of this is a very
39:31compelling set of models. This isn't
39:33going to take too long. I am curious
39:35what the results are, but uh
39:38I think I've said all I have to on this
39:39model release. It's a useful tool. It's
39:42a nice change to see Anthropic making
39:44small models that are actually usable
39:46and useful for real world stuff, and
39:48it's still honestly a bit trippy seeing
39:50an Anthropic model so far to the left on
39:53the artificial analysis cost versus
39:55intelligence. Right there, neck and neck
39:58with Luna. Actually, I mentioned earlier
40:00the like Pareto line stuff. We only had
40:03X high and max for Haiku at the time.
40:05Now that they have the rest, it's so
40:07close to Luna that this no longer feels
40:09like a big missing piece of the
40:12Anthropic portfolio. And with that, I
40:14will go back to my opening statement.
40:16Anthropic has no small models that are
40:18worth using right now. Anthropic now has
40:21one small model that is sometimes worth
40:23using, and that is a very nice change.
40:26And with that, all I have left to say is
40:29that I can't wait for Fable 5.5. I
40:31understand that they're probably not
40:32going to put it out for a while cuz it
40:33kind of is outside of the spirit of
40:35pacing the frontier, but I'm so happy
40:37with this 5.5 lineup right now that I'm
40:39really hopeful Fable meets the bar
40:41they've set with model releases like
40:42Opus, Sonnet, and Haiku as part of this
40:44line. Anthropic is dominating across the
40:47field now, and if I was OpenAI, I would
40:49be very, very scared, and I would be
40:51praying that the next training run for
40:53Astra comes out unbelievable. Because
40:55right now I'm honestly struggling a bit
40:57to use my Codex subs because my
40:59Anthropic and Claude subs are just so
41:01much more useful. Let me know if you
41:03guys agree, if I'm over exaggerating
41:05this gap, or if it really does feel that
41:07way to y'all. Until next time.
41:09Peace, nerds.

About this transcript

This page contains the full transcript of finally a good small model by Theo - t3․gg, generated from the public captions YouTube serves with the video. The transcript has 8,307 words across 1,184 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.