OpenAI fights back — Transcript
Full transcript
- 0:04They're back.
- 0:05Okay, let me catch you guys up quick.
- 0:07Last week Anthropic dropped a new model,
- 0:09Opus 5.5, and it was unbelievably good.
- 0:12It was so unbelievably good that OpenAI
- 0:14rushed out two model drops, GPT-6 Soul
- 0:17and GPT-6 Luna. You might have noticed I
- 0:19didn't do a video on those models.
- 0:21There's a reason. They weren't that
- 0:23good. I was not particularly impressed
- 0:25with either of them and didn't really
- 0:27have much to say. But there was one
- 0:29other model I happened to get early
- 0:31access to that is now available for all.
- 0:34That model is called GPT-6.1
- 0:37Soul. And this model made it very, very
- 0:39hard to film that Sonnet 5.5 video
- 0:41because I knew something was coming.
- 0:43Something surprisingly cheap,
- 0:44surprisingly capable, and most
- 0:46surprisingly better than Astra. So,
- 0:49yeah. I got a lot to say about this one.
- 0:52I'm filming this at 2:00 in the morning
- 0:54right after filming my Sonnet 5.5
- 0:56videos. So, pardon me for stumbling over
- 0:58a few words here and there. I'm doing my
- 1:00best to get this out as reasonably
- 1:01quickly as possible because I want to
- 1:03have some coverage. And I'll be real, it
- 1:06is also quite fun to cover these things
- 1:08before they are out. So, you are getting
- 1:10my true honest take and not the
- 1:12distilled version of what everyone else
- 1:13is saying. I'm sure this model's going
- 1:15to cause some pretty crazy waves. So,
- 1:18it'll be nice to have my take out
- 1:20initially separately first. As always, I
- 1:22feel obligated to remind you I do have
- 1:23early access, but I'm not being paid in
- 1:25any way, shape, or form. OpenAI has no
- 1:27influence over what and how I say
- 1:29things, just when. They politely asked
- 1:31me to wait until the model is out to
- 1:33talk about it, which makes a lot of
- 1:34sense. But I have to wait for one other
- 1:36thing first. Today's sponsor. In order
- 1:38to build good software with agents, you
- 1:39need to get feedback. And let's be real,
- 1:41they're getting a lot of that feedback
- 1:42from our CI. That's why we've all been
- 1:44seeing our CI bills skyrocket. And also
- 1:46why we've been getting more and more
- 1:47frustrated with GitHub Actions. Today's
- 1:49sponsor is Depot, and they're here to
- 1:50solve all of this and more. Not only can
- 1:52they make your CI up to 10 times faster,
- 1:54as well as your Docker builds up to 40
- 1:56times faster, especially when they're
- 1:58downloading cache. They're also cheaper
- 2:00and they give better feedback for your
- 2:02agents. All this is possible due to
- 2:03Depot metal. They're running their own
- 2:05bare metal with AMD epic processors that
- 2:07are way faster than what you get from
- 2:09traditional CI providers like, of
- 2:11course, GitHub Actions. If you wanted to
- 2:12be a drop-in replacement, it absolutely
- 2:14can be, but Depot's APIs are so much
- 2:16better that you should probably use
- 2:17those instead. They enable
- 2:18parallelization and most importantly,
- 2:20resilience when GitHub inevitably goes
- 2:22down randomly for no good reason. We've
- 2:24had our releases get blocked because we
- 2:25weren't using Depot and I'm so thankful
- 2:28that I've been moving more and more
- 2:28stuff over. For example, when Ben moved
- 2:30Pick Thing over to Bun, we immediately
- 2:32had some CI failures. Normally, this
- 2:34would be obscure piles of text that our
- 2:36agents parse through for us, but when we
- 2:38use Depot, it becomes way easier to see.
- 2:40They'll even analyze the failures and
- 2:42give suggestions to make it much simpler
- 2:44to get this feedback back to our agents.
- 2:46This is especially useful when you tell
- 2:47your agents that you can use Depot
- 2:49because they'll no longer have to push
- 2:50changes and wait for that to trigger a
- 2:52build. They can just run the CLI to
- 2:54trigger the exact same CI that you'd be
- 2:57triggering through GitHub instead. No
- 2:59longer do you have to file PRs of broken
- 3:01code just to get feedback to your
- 3:02agents. They could just run a tool
- 3:03instead. Your agents will also get way
- 3:05better breakdowns of what is taking so
- 3:07long in your actual CI runs so that you
- 3:10can figure out how to improve them and
- 3:12make them faster and more reliable. You
- 3:14and your agents deserve faster Docker,
- 3:15faster build times, faster CI, better
- 3:17results, and ideally a cheaper price.
- 3:19Get all of that and more at
- 3:20slite.link/depot.
- 3:22Let's talk about this model a bit
- 3:23because it is not quite what I expected
- 3:26and it's probably not what you guys
- 3:27expected either. Especially when you
- 3:29consider that GPT-6 sold just came out
- 3:32like a week ago. It'll be a round a one
- 3:34week gap from 6.0 sold to 6.1 sold. I
- 3:37also want to disclose the numbers I'm
- 3:39currently showing on my screen are
- 3:41unlikely to be exactly accurate because
- 3:43I am running terminal bench for myself
- 3:46in the first two times I ran it, I
- 3:47screwed things up. The third one seems
- 3:49to be doing much better. I didn't run it
- 3:50on medium initially so there's a missing
- 3:52there, but low, high, x-high, and max,
- 3:55although the max run is incomplete, so
- 3:57I'm currently backfilling scores from
- 3:59x-high for the ones that max either got
- 4:01wrong or in previous run or didn't do
- 4:03yet cuz it takes like 8 plus hours in
- 4:05some of these tasks. I This bench is
- 4:07nuts. It's I'm more skeptical of
- 4:09benchmarks than ever now that I've been
- 4:11running a lot more of them myself in
- 4:12order to get the coverage I want to give
- 4:13here. For what it is worth, Terminal
- 4:15Bench 4 is state of the art score here,
- 4:18as is Deep SWE, although this one's
- 4:20weirder because it goes down on x-high
- 4:23and max and stays even on low and high,
- 4:26but those even low and high scores are
- 4:28scoring around what Astra did on high.
- 4:31The difference being it's doing it for
- 4:33comically cheaper. If you switch over to
- 4:35the log scale, you'll see what I mean.
- 4:37This model on low costs 21 cents versus
- 4:41Astra on low costing $1.46
- 4:43and Opus 5.5 on max getting the same
- 4:46score for $14.65.
- 4:49While I will gladly admit that Deep SWE
- 4:51is far from a perfect measure of how
- 4:54good a model is at day-to-day code work,
- 4:56the fact that 6.1 Soul is scoring the
- 4:58same as Opus and is also 73x cheaper is
- 5:02at least worth noticing. Here's where
- 5:05I'm going to say some of the things that
- 5:06I probably shouldn't. Considering that
- 5:08GPT-6 Soul came out last week on Tuesday
- 5:11and this model's coming out this week on
- 5:13Tuesday, I think it's reasonable to
- 5:16infer that 6.1 Soul was not meant to be
- 5:196.1 Soul. There are things I'm not
- 5:22supposed to say and I'm definitely
- 5:23walking a thin line here by sharing it,
- 5:25so I hope this proves I'm not paid off
- 5:27by OpenAI because I'm about to give you
- 5:29guys info that I Yeah, it just let let
- 5:31me get through this. First and foremost,
- 5:336.1 is significantly smarter than GPT-6
- 5:36Soul, thereby indicating this isn't just
- 5:38a dot one bump. There is something
- 5:40fundamentally different here. Next point
- 5:42is that it has meaningfully slower
- 5:43tokens per second. That tends to
- 5:45indicate the model is bigger. Hard to
- 5:47know for sure. Seems like this model
- 5:49might be different. Most importantly, we
- 5:51have Tibo's tweet. What am I referring
- 5:54to there? Well, right before I started
- 5:57filming, Tibo dropped quite a wall of
- 5:59text. The thing I want to emphasize here
- 6:01is first off that the Pro $200
- 6:04subscription is back, but more
- 6:05importantly,
- 6:07more importantly is this sentence. They
- 6:09are changing how they calculate the
- 6:10usage in the sub. In effect, if you do
- 6:12the math, it will net out at half the
- 6:15dollar in API spend compared to the old
- 6:18Pro $200 plan. Why in the world would
- 6:21they do this? Especially right now where
- 6:23there is allegedly an internal code red
- 6:25because Opus 5.5 is so unbelievably good
- 6:28and has made the $200 Claude code sub
- 6:30such an unbelievable value. The only
- 6:32reason in the world Tibo would post this
- 6:34right now is a I don't know. Maybe a new
- 6:36model is coming where their margins
- 6:38aren't as good, so the ability to
- 6:39subsidize has gone down because
- 6:41remember, you can get 8 to 9,000 dollars
- 6:44of usage in a month on the $200 Claude
- 6:47code plan, and you can get over 12 grand
- 6:50on the $200 Codex plan. I did actually
- 6:52run a lot of numbers before this, and
- 6:53the amount you could get on Astra did go
- 6:55down slightly closer to like 8 grand or
- 6:58so. Hard to know for sure because they
- 7:00differ for everyone everywhere, and it's
- 7:02hard to log all of this stuff, but from
- 7:03my math, roughly 9 grand a month of
- 7:05usage, and that's where the price for
- 7:08this model comes in. This model is $2
- 7:11per million input tokens and $10 per
- 7:12million out. This makes it way cheaper
- 7:14than 5.6 sole was at launch, half the
- 7:16price of 5.6 sole after discounts, and
- 7:18the same price as GPT-6 sole, 1/5 the
- 7:21price of Astra. However, this is not the
- 7:24whole story cuz cash reads matter, and
- 7:27the cash read price for this model is
- 7:29going to be 10 cents per mill in. That's
- 7:32a big deal. OpenAI has not changed cash
- 7:35read price as far as I know ever before.
- 7:38It's always been exactly 10% of the
- 7:40normal read price. That makes it a 90%
- 7:42discount, and now it's a 95% discount.
- 7:45That means they cut the cash read cost
- 7:47in half, massively reducing the cost for
- 7:50real-world agentic use, which to be
- 7:52clear is what we're using these for most
- 7:54of the time.
- 7:55So, this makes the model absurdly cheap
- 7:58for doing real-world code work. That
- 8:00also means that they are almost
- 8:01certainly cutting into their margins.
- 8:03Historically, these margins are rumored
- 8:05to be as high as 95%,
- 8:07like for every $10 you spend, they only
- 8:10have to spend 50 cents. And as crazy as
- 8:12that sounds, it makes a lot of sense,
- 8:14especially when you consider how
- 8:14expensive it is to make and train these
- 8:16models. But, that also gives them wiggle
- 8:18room to change things around a bit,
- 8:20which appears to be what's happening
- 8:21here. That also means that if they were
- 8:23to keep subsidizing the same level that
- 8:25they were on the subscriptions, that
- 8:27your electricity cost for your sub would
- 8:29be more than you're paying. So, I get
- 8:31why they have to change this. They've
- 8:33kind of just left the details out there
- 8:35for us to reverse-engineer, so
- 8:37take this as you will. 6.1 coming so
- 8:40fast seems to indicate it is not just a
- 8:43new snapshot of GPT-6. So, let's talk
- 8:45more about this model.
- 8:47As I was showing earlier,
- 8:49seems really good at terminal bench.
- 8:50Every time we refresh, the numbers
- 8:51change because new runs come in, and it
- 8:53looks like Max failed some things that
- 8:55XHi passed, which is why it just dropped
- 8:57a bit. But again, pretty much all of
- 8:59these, even high and XHi, are scoring
- 9:02higher than anything else ever has. And
- 9:05this is for me running this benchmark on
- 9:06random VMs on my network, so
- 9:09not the best suite to test against. I
- 9:11also had to drop three particular tasks
- 9:13from it because they expected an H100 to
- 9:15work against, which I make decent money.
- 9:17I don't make H100 money, okay? But none
- 9:20of this is real-world code work. So,
- 9:22let's talk a bit about that. Obviously,
- 9:25we'll have all the fun things like fish
- 9:26slop near the end, so stay tuned for
- 9:27that. But I just want to fixate a bit on
- 9:30the costs here because the most
- 9:32expensive run with 6.1 sole for me was
- 9:34about a dollar and 38 cents per task,
- 9:36and to cheapest run with Opus 5.5 was
- 9:40$5.12.
- 9:42That's a 4 to 5x gap from the cheapest
- 9:45Opus to the most expensive Soul. So, at
- 9:47this point, I would imagine you are
- 9:49hoping and praying this model is good
- 9:51and that it can actually replace Opus
- 9:535.5 for day-to-day work. And I promise
- 9:56we'll get some good answers to that in a
- 9:57bit. But first, we need to talk a bit
- 9:58about model behaviors here. Because this
- 10:01model is a part of the GPT-6 family,
- 10:04which means it has a behaviors that are
- 10:07worth talking about.
- 10:09I know I cite this diagram a lot, but
- 10:11there's a reason for it. The thing that
- 10:12made me so frustrated with GPT-6 Astra
- 10:15wasn't that it was less intelligent than
- 10:17the best models from Anthropic, because
- 10:19it was more intelligent than the best
- 10:20models from Anthropic, and I would argue
- 10:21in many ways still is. But there was a
- 10:23problem. It is also dumb. It is smart
- 10:26and dumb at the same time. GPT-6 Astra
- 10:28would just randomly spike into the
- 10:30dumbest I've seen a model do
- 10:32this year, even worse than like some of
- 10:34the small open weight models I play
- 10:35with. It's still so deeply frustrating
- 10:38that Astra does this, because on the
- 10:40other end, when it does well, it's
- 10:41unbelievable. But these spikes got to
- 10:44the point where I effectively churned. I
- 10:46was only using my Codex subs for
- 10:48computer use, and I ended up just
- 10:50leaning on the Fable 5.1 and obviously
- 10:51now Opus 5.5 for almost all of my
- 10:54day-to-day work. So, have they addressed
- 10:56the spikiness? Has GPT-6.1 Soul fixed
- 10:59the problems that I was so frustrated
- 11:02about with Astra? I would say mostly.
- 11:05Not entirely, but for the most part,
- 11:08yeah, this is a much better model. Its
- 11:10peaks are not as high. This is not the
- 11:12incredible revolutionary 3D capabilities
- 11:15that we saw with Astra. In fact, I would
- 11:17put it slightly below 5.5 Opus in most
- 11:20of those types of things. It is not as
- 11:22good at computer use as Astra, although
- 11:24it is close enough to the point where I
- 11:26have been happy using it for all of my
- 11:27day-to-day work. I actually had 6.1 Soul
- 11:30go through all of my emails and find
- 11:32invoices that I had forgotten to pay or
- 11:34was behind on, mostly like investing
- 11:36stuff, and set up new tabs in Chrome for
- 11:39every investment I needed to wire, fill
- 11:41out all the details for me, and just
- 11:42leave me to hit send. It didn't get a
- 11:44single thing wrong and called out
- 11:45additional stuff that I absolutely would
- 11:47have missed if I was doing this work
- 11:48myself. So, I'm literally trusting this
- 11:50model to wire money for me. It's
- 11:52trustworthy enough for that, and
- 11:54honestly, I don't know if I would have
- 11:55trusted Astra with that due to the
- 11:56spikiness. 6.1 Soul, much, much less
- 11:59spiky. From what I've heard from the
- 12:01other testers, they seem to agree with
- 12:02this analysis. I know for a fact that
- 12:04Julius and Ben, who have also been
- 12:05testing, have had a much better
- 12:07experience with this than Astra in terms
- 12:08of the spikiness. Julius called the
- 12:10model incredible. Ben called it
- 12:11incredibly boring. And I think that's
- 12:13the best place you can be for a model
- 12:15drop like this. But, as I had mentioned
- 12:16before, its peaks are not as impressive.
- 12:19While it does quality work the majority
- 12:21of the time, there are some tasks that
- 12:23are just at the edge of its capability
- 12:25that it will start to do weirder stuff
- 12:27on. For the most part, it's fine, but I
- 12:30I'm still reaching for Opus a decent
- 12:32bit. We'll talk more about the
- 12:32comparison later. I do default to this
- 12:35model for a bunch of stuff, though.
- 12:37First off, as I mentioned before,
- 12:39computer use. I can't wait for ultrafast
- 12:41to be like an actual thing you can use
- 12:43with OpenAI models, because when it is,
- 12:45this model is going to be crazy on it,
- 12:47because it can already figure out how to
- 12:49navigate computer use totally fine. If
- 12:51it can suddenly do it six times faster,
- 12:53it's going to be unbelievably fun. Still
- 12:55not quite as good as Astra, but more
- 12:57than good enough that for the price
- 12:58difference, I wouldn't even think twice
- 12:59about it. But, as I mentioned before,
- 13:01there are certain things I would still
- 13:03occasionally use Astra for that I am
- 13:05more than happy to use Soul for. One of
- 13:07those things is deep code reviews. I
- 13:10have still found OpenAI models and the
- 13:12like Rottweiler nature where they'll dig
- 13:14into a problem and shake it into tiny
- 13:16pieces until they find every single
- 13:17thing wrong with it. I find 6.1 Soul to
- 13:19be incredibly capable in this particular
- 13:21way. So, as you can probably guess, I
- 13:24had 6.1 Soul do some deep audits on
- 13:27Orchestrator V2 and other parts of my
- 13:29real-world codebases. In my Orchestrator
- 13:31V2 audit, it performed nearly
- 13:33identically to Astra. I do believe it
- 13:35was slightly higher a score. Okay, not
- 13:37in this analysis, but in my other
- 13:39analysis, it did actually score very,
- 13:40very slightly higher. But, it did it at
- 13:42about half the price, 297 versus 584.
- 13:46Sonnet was still cheaper, and Opus was
- 13:48slightly cheaper as well. The difference
- 13:50being neither of these models were
- 13:52anywhere near as thorough with their
- 13:54analysis. 6.1 Soul dug deep to find
- 13:59things, which is why it was able to get
- 14:00a score comparable to Astra, although it
- 14:02did admittedly burn way more tokens.
- 14:04Another task I've had a lot of fun
- 14:06testing with is asking the model to find
- 14:07opportunities to improve a codebase. In
- 14:10this case, to improve D3 code. This is
- 14:12the one where Grok 4.7 scored strangely
- 14:15well. Of course, Astra scored way better
- 14:17at an 83.8 versus the 80.7 from Grok
- 14:204.7. But, GBD 6.1 Soul hit it out of the
- 14:23park with an 87.4.
- 14:25I didn't save all the prices for these
- 14:27runs. It's been a bit, okay? But, for
- 14:29Opus 5.5, it cost five bucks. And for
- 14:32Sonnet 5.5, it cost almost $9. With GBD
- 14:366.1 Soul, it was $2.15.
- 14:39That's the difference. This model's
- 14:40token price is cheaper than Sonnet, but
- 14:43its token utilization is still
- 14:45maintaining OpenAI's usual efficiency,
- 14:48which results in just crazy price to
- 14:50performance. This whole thread was
- 14:52particularly fun because I had Opus 5.5
- 14:54review this model with a different name,
- 14:57obviously, so it didn't know what it
- 14:58was. I went and edited the history
- 14:59after. And it concluded very quickly
- 15:01this was a frontier tier model. Its
- 15:03reviews and bug repros matched the fixes
- 15:05that later merged. Its first draft code
- 15:07had real bugs, which review bots caught.
- 15:09Four reviewers are still running. Local
- 15:11team for code reviews backed frontier
- 15:13tier again. In a blind 10-model bench on
- 15:15the same prompt, 6.1 Soul placed first
- 15:18with the 87.4. It found the fish slop
- 15:20runs and compared those two. It did say
- 15:226.1 souls quality output was slightly
- 15:25below Astra's as well as the two open AI
- 15:27models with Opus 5.5 and sauna 5.5,
- 15:30which we will absolutely show you in a
- 15:31bit. But I do want to call out the price
- 15:33here cuz it only costs $7 to run versus
- 15:3615 for sauna 5.5 and 50 for Opus. Opus's
- 15:41honest tier call was that this model is
- 15:43incredible for scoped work, top of the
- 15:45frontier. For find what's wrong and tell
- 15:47me the truth, I would choose it over
- 15:48Astra and about level with Opus. For
- 15:51long unattended building, this was below
- 15:53frontier. Follows its process rules even
- 15:55when they stop all progress and it does
- 15:57not ask for help. This I absolutely
- 15:59noticed. I had mentioned before, well, a
- 16:01few times now, that my TS rust port that
- 16:03I'm working with Opus 5.5 is going way
- 16:05better than when I was working on that
- 16:06same port using Astra and soul in the
- 16:09past. I had that port running for a
- 16:11while with this model and it made no
- 16:13progress. It burned a shitload of
- 16:14tokens, but it didn't actually improve
- 16:16the compiler at all. Opus was able to
- 16:18from scratch restart it and get it
- 16:21working in a day after I had spent
- 16:23months and hundreds of thousands of
- 16:25dollars in tokens with this model as
- 16:27well as with Astra and 5.6 soul. Opus
- 16:29did it in like a grand in like a night
- 16:31with just two subscriptions with the
- 16:33Claude plan. So, for unattended long
- 16:36like heavy rewrite type stuff, Anthropic
- 16:38is just comically far ahead right now.
- 16:41And it also didn't have great judgment
- 16:42when I was using it for managing my
- 16:44fleet. And for those wondering, my fleet
- 16:45is all the computers I use for running
- 16:47all my agents in code cuz one computer
- 16:48is far from enough. I don't run any of
- 16:50them on this MacBook now. So, when I use
- 16:52this model to manage the fleet, it made
- 16:53a couple dumb mistakes here and there.
- 16:55To be fair, so is Opus. Astra's the only
- 16:57one that hasn't really made too many of
- 16:58those dumb mistakes, but like I'm going
- 17:00to be so real, I am entirely done using
- 17:02Astra after this model. After I had Opus
- 17:04do all of this review, I asked it how
- 17:06much does it think this model should
- 17:07cost. It guessed $5 per mil in, 50 cents
- 17:11cashed, and 30 per mil out, putting it
- 17:14at Opus's prices roughly. It said that
- 17:16cuz it's performing like Opus, its speed
- 17:19should add a premium cuz it is quite
- 17:20fast. And it's not a pro model, which is
- 17:23where expects those higher like $100 out
- 17:26tiered pricing things to come. Pro model
- 17:28isn't really a thing anymore. We just
- 17:29use Fable and Aster, but you get the
- 17:31idea. This is the funniest part of the
- 17:32whole thread though. If OpenAI wants
- 17:34people to adopt it, I would expect $3
- 17:37per mil in, 30 cents for cashed, and $20
- 17:40instead. That would still be a fair
- 17:42price for what it does. To which I
- 17:44responded, if I told you it was $2 and
- 17:46$10 out and 10 cents per mil cash read,
- 17:49what would you think? I'd call that very
- 17:51aggressive pricing. For how you use it,
- 17:52it costs about a quarter of what I
- 17:54guessed. The cash price does most of the
- 17:56work. Agent workloads are 96% cash
- 17:59reads, so 10 cents for cash reads
- 18:01matters more than the $2 and $10
- 18:02headline prices.
- 18:04When it looked at all of my sessions,
- 18:05its price guess would have been $5,700,
- 18:08but after looking at these new prices,
- 18:10it redid the math and it would have been
- 18:11$1,550.
- 18:13That is a massive decrease. And for all
- 18:16my PR review type tasks, it was
- 18:18expecting those to be up to $10 and it's
- 18:20actually only up to $3. And that's for
- 18:22like heavy PRs with tens of thousands of
- 18:24lines of code. According to Opus, so
- 18:27don't blame me, blame Opus for saying
- 18:28this. First off, Opus says it becomes
- 18:31the default model for scoped work.
- 18:32Second off, it says that bloated system
- 18:35prompts barely matter anymore because of
- 18:37the cash pricing. It's just noise.
- 18:39Third, it says long loops are still a
- 18:40bad idea, but not because of money. It's
- 18:42cuz according to it, the TSR port wasted
- 18:44four days and made no progress at all.
- 18:47And Opus even said they'd be suspicious
- 18:49of it lasting. They expect this price to
- 18:51go up in the future. I cannot fathom
- 18:53OpenAI ever increasing the price for a
- 18:55model, but Opus thinking they will is
- 18:57hilarious and shows just how good a
- 18:59value the model is. I love this call out
- 19:01here. GPT-4 1 sold did three rounds of
- 19:03work for about half the cost of Sonnet's
- 19:06single round. A lot of this comes down
- 19:08to how context was managed. Well,
- 19:10because 6.1 sold is much more efficient,
- 19:11so it's not doing as many calls that
- 19:14bloat the context. It's not outputting
- 19:16as many tokens that are like building up
- 19:17over time. So, the average number of
- 19:20tokens being read per request was only
- 19:23around 110,000 tokens versus 360,000 for
- 19:26Sonnet 5. The result was that Soul used
- 19:28under half as many input tokens as
- 19:30Sonnet, making this model significantly
- 19:32more efficient. Speaking of efficiency,
- 19:34I want to talk about these deep SW
- 19:35scores a tiny bit more because this is a
- 19:38weird bench for me to a fork and include
- 19:40in these things. I actually did it for a
- 19:42different reason, not to compare against
- 19:446.1 Soul, but to compare against a new
- 19:47release from OpenRouter, Jev Router.
- 19:50OpenRouter added Jev Router to try and
- 19:52optimize costs with your requests, and I
- 19:54thought it was an incredibly stupid
- 19:56idea. Once I started running it against
- 19:57benchmarks, I confirmed it's an
- 19:59incredibly stupid idea. It turns out a
- 20:01model that cannot reason, that is given
- 20:03a prompt into no context, cannot make a
- 20:05good decision around how hard the
- 20:07problem is. And Jev Router ended up
- 20:10being DeepSeek V4.1 Flash Router for the
- 20:13vast majority of its runs. Around 60% of
- 20:16all the requests went straight to
- 20:17DeepSeek 4.1 Flash. So, didn't like it
- 20:19that much. It also routes to other
- 20:21smarter models, which should give it
- 20:23more of an advantage, but it ended up
- 20:25being more expensive than GPT-6 Astro
- 20:27was on low, while also taking four to
- 20:30five times longer cuz 6 Astro low took
- 20:324.6 minutes, and Jev Router took 20. Jev
- 20:36Router's average task took 104 steps,
- 20:39whereas GPT-6 Astro took 19. You get the
- 20:42idea, it wasn't very good. But, the
- 20:44whole point of Jev Router is that it
- 20:45would be as cheap as possible to get a
- 20:48certain score. That was the promise on
- 20:50the tin. Whether or not you believe them
- 20:52is up to you, not me. I think it's
- 20:54Regardless, Jev Router was
- 20:57routing to DeepSeek 4.1 Flash for the
- 21:00majority of its requests. Despite Jev
- 21:02Router routing to the cheapest possible
- 21:05small open-weight models from whatever
- 21:07provider will give it away for free. 6.1
- 21:09Sonnet low got the same score for an
- 21:12eighth the price. OpenAI is here to
- 21:14destroy any wins anyone else has in
- 21:17terms of efficiency. Completing this
- 21:19bench in 4.8 minutes for 21 cents with
- 21:22the second highest score I've ever seen
- 21:23on it is a massive achievement, tying
- 21:26Opus 5.5, which took 50 minutes per task
- 21:30on max. A tenth the time and a
- 21:33seventieth the price for the same score.
- 21:35If your work fits within the things 6.1
- 21:38Sonnet does well, you should probably
- 21:40use it for everything. But, if your work
- 21:42doesn't fit in it particularly well, you
- 21:44should probably keep using Opus and
- 21:46maybe give Opus the ability to call 6.1
- 21:49Sonnet when it should for various tasks.
- 21:51I'm almost certainly going to be setting
- 21:52things up so that Opus 5.5 can call 6.1
- 21:55Sonnet to do investigation work to try
- 21:57and like root cause bugs, to do analysis
- 22:00of code bases to figure out what things
- 22:02need to be touched and why, to help me
- 22:04triage real-world work, to help me
- 22:06review the work that Opus does, and
- 22:08more. I'm kind of spoiling the ending
- 22:10here, aren't I? I'm going to keep using
- 22:12Opus 5.5 for now. Before I explain why,
- 22:15let me do the thing that I'm most
- 22:17excited for.
- 22:18Fish slop.
- 22:20The first thing you might have noticed
- 22:21is the inclusion of slop in fish slop.
- 22:25This model did the horrible thing I hate
- 22:28where it surrounded the game in a bunch
- 22:31of absolutely garbage UI. And coming to
- 22:34this right after the 5.5 Sonnet demo
- 22:37hurts me deeply
- 22:39because Sonnet 5.5 did not make graphics
- 22:41anywhere near this good looking,
- 22:43but at least it made a UI that was
- 22:45nowhere near this awful. And man, do I
- 22:48wish the bad UI is where the issues
- 22:50stopped. I will turn on the sound.
- 22:53Oh god, it's blaring.
- 23:00It is stunning looking.
- 23:02The fish are some of the best. The model
- 23:04for the sub is way better. The
- 23:06propellers work way better.
- 23:08I'm going to mute the sound cuz that is
- 23:10looking pretty bad. I haven't even heard
- 23:11it, honestly.
- 23:14But damn, like
- 23:16looks beautiful,
- 23:17but if you actually are playing it, one
- 23:19of the first things you'll notice is
- 23:20that the movement feels significantly
- 23:23worse than it does in either the Opus or
- 23:26the Sonnet versions that I've demoed in
- 23:28the past.
- 23:32Yeah, it it moves jank. It also has a
- 23:35significantly worse frame rate than the
- 23:37versions from the other models. It does
- 23:40have higher graphic fidelity, so that
- 23:41makes sense. Like the models here with
- 23:44the plants are significantly better than
- 23:47they were with the Sonnet version. The
- 23:49details of the fidelity of the extras in
- 23:50the tank is absolutely hilarious.
- 23:54Like
- 23:55yeah.
- 23:57But god damn, I'm so tired of the
- 23:58unnecessary text everywhere. This model
- 24:01does it worse than almost any I've ever
- 24:02seen before. We got take a breather,
- 24:05paused,
- 24:07your little world can wait, back to the
- 24:09reef, start a new tank, slot 01, feeder
- 24:13submarine,
- 24:15a little underwater chaos,
- 24:18the big little goal, your little
- 24:20ecosystem, little fish become big
- 24:22earners, four meals and they're all
- 24:24grown up,
- 24:26make the family a little bigger, a
- 24:27little golden overachiever. It is so
- 24:30many of these. There's like 20 plus of
- 24:32them.
- 24:33And I promise you guys, as soon as I saw
- 24:35this, I took a screenshot, I sent it to
- 24:37Open AI, and I crashed out in the slack
- 24:39because I cannot fathom how they haven't
- 24:41fixed this problem. This model
- 24:43is unacceptably garbage at UI. It has
- 24:46regressed again. And if you're looking
- 24:48for a model that can make front ends
- 24:49that don't suck, go spend your money
- 24:51somewhere else because it should not be
- 24:53spent here. This model sucks at front
- 24:55end. It sucks at design. It has no
- 24:56taste. And you're going to have to bring
- 24:58your taste yourself still. But it is
- 24:59admittedly really good at Blender. If
- 25:01you give it like a screenshot of a thing
- 25:03you want it to model in 3D and say,
- 25:04"Hey, you have Blender over the CLI. Go
- 25:06make this." It will, and it'll do a
- 25:08pretty damn good job.
- 25:09But I would never have it make the
- 25:10actual mechanics for my games because it
- 25:13feels awful to play. It also has like
- 25:16nowhere near as much gameplay loop. In
- 25:19fact, the first time I tried demoing
- 25:20this before filming,
- 25:22it just randomly game overed as I was
- 25:24like getting started in the first 30
- 25:26seconds. It never like said why.
- 25:28Actually, I think I technically beat it.
- 25:30I also want to like take this version
- 25:32and hand it to Opus or Sonnet and say,
- 25:33"Hey, can you make this play better?
- 25:35Because the graphics are good, but the
- 25:37game sucks." But when you combine how
- 25:39cheap it was to make this cuz like this
- 25:41was $5, I think, to generate. That's
- 25:44pretty insane. And if you combine that
- 25:47with like Ultra Fast, if that ever
- 25:49happens, suddenly you're going to be
- 25:51able to make a game in a few minutes on
- 25:54demand. We're actually now getting to
- 25:56that threshold where game development is
- 25:58about to flip upside down because of how
- 26:00models are finally understanding
- 26:02three-dimensional space and the tooling
- 26:03necessary to do these types of things.
- 26:05It's happening. As per usual, I was not
- 26:07allowed to put the code I wrote with
- 26:09this model inside of T3 Code or other
- 26:11open-source projects during the testing
- 26:13window. So, I had to use it exclusively
- 26:15on my internal projects like Lakebed, as
- 26:17well as for auditing other work, which
- 26:19means I mostly use this for auditing
- 26:21other work, and I was very impressed.
- 26:22This is a real PR I was working on to
- 26:24fix a bug where my new little work tree
- 26:27like setup window that would appear in a
- 26:29new thread in T3 Code would disappear if
- 26:31you left and came back. I had Claude
- 26:32Code work on this, but this problem went
- 26:34pretty deep. So, I wanted to make sure
- 26:37that whatever solution I came up with
- 26:39was very, very, very well vetted. While
- 26:41I personally still do not trust this
- 26:43model to write the code I'm trying to
- 26:45land, I absolutely trust it to review
- 26:47things. Ignore the GPT-6 soul there,
- 26:49just a placeholder. So, when I had 6.1
- 26:52Soul look through this, it found real
- 26:55problems that were entirely missed by
- 26:57Fable and by Opus. First, it called out
- 26:59that follow-up messages can stay blocked
- 27:01after the agent starts, which is very
- 27:03annoying if you want to queue a message,
- 27:05and also that recovered setup progress
- 27:07was disappearing too early. It figured
- 27:09all of these things out with a
- 27:10combination of reading the code and
- 27:11analyzing it as well as computer use. It
- 27:13was able to prevent me from merging a
- 27:14real regression in T3 code. So, I
- 27:17literally just copy-pasted those things
- 27:18to Claude and then told it to take
- 27:20another look. Said, "Better, but I'll
- 27:22still fix two small gaps." Remember, I
- 27:24can't use this model of the code for
- 27:26this project at the time, so I
- 27:27copy-pasted that again over to Claude,
- 27:29and it eventually got it good enough,
- 27:31and then I finally merged. But, that is
- 27:33what I like this model for, and I cannot
- 27:35wait to push its limits for actually
- 27:36coding. Although, I will say from the
- 27:38code that I did have the misfortune of
- 27:40reading, it is harder to justify merging
- 27:43this code than it is for code from Opus.
- 27:46Normally, I would make you guys wait for
- 27:48the Opus versus Soul video or the Sonic
- 27:50versus Soul video, but I'll just spoil
- 27:51the details now. I like using them in
- 27:53tandem because I find Soul to be way
- 27:55better at reviewing and digging into the
- 27:57details, but I find Opus a more pleasant
- 27:59collaborator and significantly better at
- 28:01actually implementing code without
- 28:03getting blocked constantly throughout
- 28:04its work. And even now with the Rust
- 28:06rewrite of TypeScript, I find myself in
- 28:08a similar pattern, where I have Opus 5.5
- 28:11just going and going and going, making
- 28:13the code base work and what work well,
- 28:15and then I had Soul come in and do an
- 28:17audit. And this is the funniest part.
- 28:19Remember before I said that I had Soul
- 28:21and Astra working on that TS Rust port
- 28:23for effectively months? I told Soul to
- 28:26come in, and it got it from 83.7% to
- 28:28100% in under a day. I was blown away by
- 28:32that, that it had somehow unblocked the
- 28:33work that Astra and Soul were doing, as
- 28:35well as 6.1 Soul. And I was absolutely
- 28:37blown away by that, that it had taken
- 28:39the work that 5.6 Soul, 6.1 Soul, and
- 28:41Astra had done over months, and got it
- 28:44unblocked where it had been stuck for
- 28:45weeks, and finished it. I was much more
- 28:47blown away when I had 61 Soul take a
- 28:50look at that work and critique it. And
- 28:53what it brought up was that of the 1.8
- 28:56million lines of code, 1.3 million were
- 28:59not being used. The reason was because
- 29:02Opus concluded whole of the code from
- 29:04all the other agents was useless slop
- 29:06that had no chance of being recovered,
- 29:07and it chose to rewrite it from scratch
- 29:09itself in another crate. So, on one
- 29:11hand, the only reason the code worked
- 29:13was Opus, but on the other hand, the
- 29:15only reason the slop was still around
- 29:16was also Opus. So, I had to have this
- 29:18model come in and clean up the mess that
- 29:21other OpenAI models had made because
- 29:23Opus didn't even notice the mess was
- 29:25still there. What I'm trying to say is
- 29:26this model absolutely has a place in
- 29:28your workflows. It could probably even
- 29:30be your default coding model, and you
- 29:32wouldn't have too many issues with it,
- 29:34but I still find Opus to be a better
- 29:35collaborator overall. That said, I have
- 29:38almost no reason to use Sonnet anymore
- 29:40because this will effectively take its
- 29:42place. And you bet your butt the moment
- 29:44this model comes out, I'll be going and
- 29:46making adjustments inside of my Claude
- 29:47config because I already have it set up
- 29:49so that I can use Soul inside of Claude
- 29:51code because I want to make sure Opus
- 29:53knows this is the model to have review
- 29:55its work and investigate the things
- 29:57going on in the code base. This is a
- 29:58damn good model, and I'm really happy to
- 30:00have it. I wish we had something bigger,
- 30:02smarter, and more capable overall. I was
- 30:04really hoping for something to truly
- 30:06dethrone Opus 5.5 as my daily driver.
- 30:08This isn't it, and I'm not planning on
- 30:10canceling any of my Claude subs as a
- 30:12result of this release, but I am
- 30:13planning on taking a lot more of my
- 30:15Codex subs in my day-to-day work,
- 30:17admittedly in Claude code. This is an
- 30:19awesome release and an unbelievable
- 30:21price for what you're getting, but this
- 30:23does potentially mark the start of the
- 30:25end of the subsidization era. So, make
- 30:27sure you're subscribed so that you can
- 30:28be here when I cover all of that and
- 30:30more. God, I hope this isn't get me
- 30:31canceled online. I have no idea how
- 30:33others feel about this beyond like a
- 30:34handful of early access testers I've
- 30:35talked to. I legitimately don't know if
- 30:37you all are going to love it or hate it
- 30:38or land somewhere between. I will know
- 30:40in a few hours, I guess, cuz it's a
- 30:43Yeah, it's 3:00 in the morning. I am
- 30:45going to go to bed now.
- 30:47Uh,
- 30:48this is a tiring one. Hopefully I did a
- 30:50good job. Let me know in the comments.
- 30:51And until next time,
- 30:53peace nerds.
- 30:55God, I'm so dead.
- 30:57Ah.
About this transcript
This page contains the full transcript of OpenAI fights back by Theo - t3․gg, generated from the public captions YouTube serves with the video. The transcript has 6,324 words across 892 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.