YouTube2Text

OpenAI fights back — Transcript

by Theo - t3․gg · 6,324 words · 892 segments · language en · Watch on YouTube

Full transcript

  1. 0:04They're back.
  2. 0:05Okay, let me catch you guys up quick.
  3. 0:07Last week Anthropic dropped a new model,
  4. 0:09Opus 5.5, and it was unbelievably good.
  5. 0:12It was so unbelievably good that OpenAI
  6. 0:14rushed out two model drops, GPT-6 Soul
  7. 0:17and GPT-6 Luna. You might have noticed I
  8. 0:19didn't do a video on those models.
  9. 0:21There's a reason. They weren't that
  10. 0:23good. I was not particularly impressed
  11. 0:25with either of them and didn't really
  12. 0:27have much to say. But there was one
  13. 0:29other model I happened to get early
  14. 0:31access to that is now available for all.
  15. 0:34That model is called GPT-6.1
  16. 0:37Soul. And this model made it very, very
  17. 0:39hard to film that Sonnet 5.5 video
  18. 0:41because I knew something was coming.
  19. 0:43Something surprisingly cheap,
  20. 0:44surprisingly capable, and most
  21. 0:46surprisingly better than Astra. So,
  22. 0:49yeah. I got a lot to say about this one.
  23. 0:52I'm filming this at 2:00 in the morning
  24. 0:54right after filming my Sonnet 5.5
  25. 0:56videos. So, pardon me for stumbling over
  26. 0:58a few words here and there. I'm doing my
  27. 1:00best to get this out as reasonably
  28. 1:01quickly as possible because I want to
  29. 1:03have some coverage. And I'll be real, it
  30. 1:06is also quite fun to cover these things
  31. 1:08before they are out. So, you are getting
  32. 1:10my true honest take and not the
  33. 1:12distilled version of what everyone else
  34. 1:13is saying. I'm sure this model's going
  35. 1:15to cause some pretty crazy waves. So,
  36. 1:18it'll be nice to have my take out
  37. 1:20initially separately first. As always, I
  38. 1:22feel obligated to remind you I do have
  39. 1:23early access, but I'm not being paid in
  40. 1:25any way, shape, or form. OpenAI has no
  41. 1:27influence over what and how I say
  42. 1:29things, just when. They politely asked
  43. 1:31me to wait until the model is out to
  44. 1:33talk about it, which makes a lot of
  45. 1:34sense. But I have to wait for one other
  46. 1:36thing first. Today's sponsor. In order
  47. 1:38to build good software with agents, you
  48. 1:39need to get feedback. And let's be real,
  49. 1:41they're getting a lot of that feedback
  50. 1:42from our CI. That's why we've all been
  51. 1:44seeing our CI bills skyrocket. And also
  52. 1:46why we've been getting more and more
  53. 1:47frustrated with GitHub Actions. Today's
  54. 1:49sponsor is Depot, and they're here to
  55. 1:50solve all of this and more. Not only can
  56. 1:52they make your CI up to 10 times faster,
  57. 1:54as well as your Docker builds up to 40
  58. 1:56times faster, especially when they're
  59. 1:58downloading cache. They're also cheaper
  60. 2:00and they give better feedback for your
  61. 2:02agents. All this is possible due to
  62. 2:03Depot metal. They're running their own
  63. 2:05bare metal with AMD epic processors that
  64. 2:07are way faster than what you get from
  65. 2:09traditional CI providers like, of
  66. 2:11course, GitHub Actions. If you wanted to
  67. 2:12be a drop-in replacement, it absolutely
  68. 2:14can be, but Depot's APIs are so much
  69. 2:16better that you should probably use
  70. 2:17those instead. They enable
  71. 2:18parallelization and most importantly,
  72. 2:20resilience when GitHub inevitably goes
  73. 2:22down randomly for no good reason. We've
  74. 2:24had our releases get blocked because we
  75. 2:25weren't using Depot and I'm so thankful
  76. 2:28that I've been moving more and more
  77. 2:28stuff over. For example, when Ben moved
  78. 2:30Pick Thing over to Bun, we immediately
  79. 2:32had some CI failures. Normally, this
  80. 2:34would be obscure piles of text that our
  81. 2:36agents parse through for us, but when we
  82. 2:38use Depot, it becomes way easier to see.
  83. 2:40They'll even analyze the failures and
  84. 2:42give suggestions to make it much simpler
  85. 2:44to get this feedback back to our agents.
  86. 2:46This is especially useful when you tell
  87. 2:47your agents that you can use Depot
  88. 2:49because they'll no longer have to push
  89. 2:50changes and wait for that to trigger a
  90. 2:52build. They can just run the CLI to
  91. 2:54trigger the exact same CI that you'd be
  92. 2:57triggering through GitHub instead. No
  93. 2:59longer do you have to file PRs of broken
  94. 3:01code just to get feedback to your
  95. 3:02agents. They could just run a tool
  96. 3:03instead. Your agents will also get way
  97. 3:05better breakdowns of what is taking so
  98. 3:07long in your actual CI runs so that you
  99. 3:10can figure out how to improve them and
  100. 3:12make them faster and more reliable. You
  101. 3:14and your agents deserve faster Docker,
  102. 3:15faster build times, faster CI, better
  103. 3:17results, and ideally a cheaper price.
  104. 3:19Get all of that and more at
  105. 3:20slite.link/depot.
  106. 3:22Let's talk about this model a bit
  107. 3:23because it is not quite what I expected
  108. 3:26and it's probably not what you guys
  109. 3:27expected either. Especially when you
  110. 3:29consider that GPT-6 sold just came out
  111. 3:32like a week ago. It'll be a round a one
  112. 3:34week gap from 6.0 sold to 6.1 sold. I
  113. 3:37also want to disclose the numbers I'm
  114. 3:39currently showing on my screen are
  115. 3:41unlikely to be exactly accurate because
  116. 3:43I am running terminal bench for myself
  117. 3:46in the first two times I ran it, I
  118. 3:47screwed things up. The third one seems
  119. 3:49to be doing much better. I didn't run it
  120. 3:50on medium initially so there's a missing
  121. 3:52there, but low, high, x-high, and max,
  122. 3:55although the max run is incomplete, so
  123. 3:57I'm currently backfilling scores from
  124. 3:59x-high for the ones that max either got
  125. 4:01wrong or in previous run or didn't do
  126. 4:03yet cuz it takes like 8 plus hours in
  127. 4:05some of these tasks. I This bench is
  128. 4:07nuts. It's I'm more skeptical of
  129. 4:09benchmarks than ever now that I've been
  130. 4:11running a lot more of them myself in
  131. 4:12order to get the coverage I want to give
  132. 4:13here. For what it is worth, Terminal
  133. 4:15Bench 4 is state of the art score here,
  134. 4:18as is Deep SWE, although this one's
  135. 4:20weirder because it goes down on x-high
  136. 4:23and max and stays even on low and high,
  137. 4:26but those even low and high scores are
  138. 4:28scoring around what Astra did on high.
  139. 4:31The difference being it's doing it for
  140. 4:33comically cheaper. If you switch over to
  141. 4:35the log scale, you'll see what I mean.
  142. 4:37This model on low costs 21 cents versus
  143. 4:41Astra on low costing $1.46
  144. 4:43and Opus 5.5 on max getting the same
  145. 4:46score for $14.65.
  146. 4:49While I will gladly admit that Deep SWE
  147. 4:51is far from a perfect measure of how
  148. 4:54good a model is at day-to-day code work,
  149. 4:56the fact that 6.1 Soul is scoring the
  150. 4:58same as Opus and is also 73x cheaper is
  151. 5:02at least worth noticing. Here's where
  152. 5:05I'm going to say some of the things that
  153. 5:06I probably shouldn't. Considering that
  154. 5:08GPT-6 Soul came out last week on Tuesday
  155. 5:11and this model's coming out this week on
  156. 5:13Tuesday, I think it's reasonable to
  157. 5:16infer that 6.1 Soul was not meant to be
  158. 5:196.1 Soul. There are things I'm not
  159. 5:22supposed to say and I'm definitely
  160. 5:23walking a thin line here by sharing it,
  161. 5:25so I hope this proves I'm not paid off
  162. 5:27by OpenAI because I'm about to give you
  163. 5:29guys info that I Yeah, it just let let
  164. 5:31me get through this. First and foremost,
  165. 5:336.1 is significantly smarter than GPT-6
  166. 5:36Soul, thereby indicating this isn't just
  167. 5:38a dot one bump. There is something
  168. 5:40fundamentally different here. Next point
  169. 5:42is that it has meaningfully slower
  170. 5:43tokens per second. That tends to
  171. 5:45indicate the model is bigger. Hard to
  172. 5:47know for sure. Seems like this model
  173. 5:49might be different. Most importantly, we
  174. 5:51have Tibo's tweet. What am I referring
  175. 5:54to there? Well, right before I started
  176. 5:57filming, Tibo dropped quite a wall of
  177. 5:59text. The thing I want to emphasize here
  178. 6:01is first off that the Pro $200
  179. 6:04subscription is back, but more
  180. 6:05importantly,
  181. 6:07more importantly is this sentence. They
  182. 6:09are changing how they calculate the
  183. 6:10usage in the sub. In effect, if you do
  184. 6:12the math, it will net out at half the
  185. 6:15dollar in API spend compared to the old
  186. 6:18Pro $200 plan. Why in the world would
  187. 6:21they do this? Especially right now where
  188. 6:23there is allegedly an internal code red
  189. 6:25because Opus 5.5 is so unbelievably good
  190. 6:28and has made the $200 Claude code sub
  191. 6:30such an unbelievable value. The only
  192. 6:32reason in the world Tibo would post this
  193. 6:34right now is a I don't know. Maybe a new
  194. 6:36model is coming where their margins
  195. 6:38aren't as good, so the ability to
  196. 6:39subsidize has gone down because
  197. 6:41remember, you can get 8 to 9,000 dollars
  198. 6:44of usage in a month on the $200 Claude
  199. 6:47code plan, and you can get over 12 grand
  200. 6:50on the $200 Codex plan. I did actually
  201. 6:52run a lot of numbers before this, and
  202. 6:53the amount you could get on Astra did go
  203. 6:55down slightly closer to like 8 grand or
  204. 6:58so. Hard to know for sure because they
  205. 7:00differ for everyone everywhere, and it's
  206. 7:02hard to log all of this stuff, but from
  207. 7:03my math, roughly 9 grand a month of
  208. 7:05usage, and that's where the price for
  209. 7:08this model comes in. This model is $2
  210. 7:11per million input tokens and $10 per
  211. 7:12million out. This makes it way cheaper
  212. 7:14than 5.6 sole was at launch, half the
  213. 7:16price of 5.6 sole after discounts, and
  214. 7:18the same price as GPT-6 sole, 1/5 the
  215. 7:21price of Astra. However, this is not the
  216. 7:24whole story cuz cash reads matter, and
  217. 7:27the cash read price for this model is
  218. 7:29going to be 10 cents per mill in. That's
  219. 7:32a big deal. OpenAI has not changed cash
  220. 7:35read price as far as I know ever before.
  221. 7:38It's always been exactly 10% of the
  222. 7:40normal read price. That makes it a 90%
  223. 7:42discount, and now it's a 95% discount.
  224. 7:45That means they cut the cash read cost
  225. 7:47in half, massively reducing the cost for
  226. 7:50real-world agentic use, which to be
  227. 7:52clear is what we're using these for most
  228. 7:54of the time.
  229. 7:55So, this makes the model absurdly cheap
  230. 7:58for doing real-world code work. That
  231. 8:00also means that they are almost
  232. 8:01certainly cutting into their margins.
  233. 8:03Historically, these margins are rumored
  234. 8:05to be as high as 95%,
  235. 8:07like for every $10 you spend, they only
  236. 8:10have to spend 50 cents. And as crazy as
  237. 8:12that sounds, it makes a lot of sense,
  238. 8:14especially when you consider how
  239. 8:14expensive it is to make and train these
  240. 8:16models. But, that also gives them wiggle
  241. 8:18room to change things around a bit,
  242. 8:20which appears to be what's happening
  243. 8:21here. That also means that if they were
  244. 8:23to keep subsidizing the same level that
  245. 8:25they were on the subscriptions, that
  246. 8:27your electricity cost for your sub would
  247. 8:29be more than you're paying. So, I get
  248. 8:31why they have to change this. They've
  249. 8:33kind of just left the details out there
  250. 8:35for us to reverse-engineer, so
  251. 8:37take this as you will. 6.1 coming so
  252. 8:40fast seems to indicate it is not just a
  253. 8:43new snapshot of GPT-6. So, let's talk
  254. 8:45more about this model.
  255. 8:47As I was showing earlier,
  256. 8:49seems really good at terminal bench.
  257. 8:50Every time we refresh, the numbers
  258. 8:51change because new runs come in, and it
  259. 8:53looks like Max failed some things that
  260. 8:55XHi passed, which is why it just dropped
  261. 8:57a bit. But again, pretty much all of
  262. 8:59these, even high and XHi, are scoring
  263. 9:02higher than anything else ever has. And
  264. 9:05this is for me running this benchmark on
  265. 9:06random VMs on my network, so
  266. 9:09not the best suite to test against. I
  267. 9:11also had to drop three particular tasks
  268. 9:13from it because they expected an H100 to
  269. 9:15work against, which I make decent money.
  270. 9:17I don't make H100 money, okay? But none
  271. 9:20of this is real-world code work. So,
  272. 9:22let's talk a bit about that. Obviously,
  273. 9:25we'll have all the fun things like fish
  274. 9:26slop near the end, so stay tuned for
  275. 9:27that. But I just want to fixate a bit on
  276. 9:30the costs here because the most
  277. 9:32expensive run with 6.1 sole for me was
  278. 9:34about a dollar and 38 cents per task,
  279. 9:36and to cheapest run with Opus 5.5 was
  280. 9:40$5.12.
  281. 9:42That's a 4 to 5x gap from the cheapest
  282. 9:45Opus to the most expensive Soul. So, at
  283. 9:47this point, I would imagine you are
  284. 9:49hoping and praying this model is good
  285. 9:51and that it can actually replace Opus
  286. 9:535.5 for day-to-day work. And I promise
  287. 9:56we'll get some good answers to that in a
  288. 9:57bit. But first, we need to talk a bit
  289. 9:58about model behaviors here. Because this
  290. 10:01model is a part of the GPT-6 family,
  291. 10:04which means it has a behaviors that are
  292. 10:07worth talking about.
  293. 10:09I know I cite this diagram a lot, but
  294. 10:11there's a reason for it. The thing that
  295. 10:12made me so frustrated with GPT-6 Astra
  296. 10:15wasn't that it was less intelligent than
  297. 10:17the best models from Anthropic, because
  298. 10:19it was more intelligent than the best
  299. 10:20models from Anthropic, and I would argue
  300. 10:21in many ways still is. But there was a
  301. 10:23problem. It is also dumb. It is smart
  302. 10:26and dumb at the same time. GPT-6 Astra
  303. 10:28would just randomly spike into the
  304. 10:30dumbest I've seen a model do
  305. 10:32this year, even worse than like some of
  306. 10:34the small open weight models I play
  307. 10:35with. It's still so deeply frustrating
  308. 10:38that Astra does this, because on the
  309. 10:40other end, when it does well, it's
  310. 10:41unbelievable. But these spikes got to
  311. 10:44the point where I effectively churned. I
  312. 10:46was only using my Codex subs for
  313. 10:48computer use, and I ended up just
  314. 10:50leaning on the Fable 5.1 and obviously
  315. 10:51now Opus 5.5 for almost all of my
  316. 10:54day-to-day work. So, have they addressed
  317. 10:56the spikiness? Has GPT-6.1 Soul fixed
  318. 10:59the problems that I was so frustrated
  319. 11:02about with Astra? I would say mostly.
  320. 11:05Not entirely, but for the most part,
  321. 11:08yeah, this is a much better model. Its
  322. 11:10peaks are not as high. This is not the
  323. 11:12incredible revolutionary 3D capabilities
  324. 11:15that we saw with Astra. In fact, I would
  325. 11:17put it slightly below 5.5 Opus in most
  326. 11:20of those types of things. It is not as
  327. 11:22good at computer use as Astra, although
  328. 11:24it is close enough to the point where I
  329. 11:26have been happy using it for all of my
  330. 11:27day-to-day work. I actually had 6.1 Soul
  331. 11:30go through all of my emails and find
  332. 11:32invoices that I had forgotten to pay or
  333. 11:34was behind on, mostly like investing
  334. 11:36stuff, and set up new tabs in Chrome for
  335. 11:39every investment I needed to wire, fill
  336. 11:41out all the details for me, and just
  337. 11:42leave me to hit send. It didn't get a
  338. 11:44single thing wrong and called out
  339. 11:45additional stuff that I absolutely would
  340. 11:47have missed if I was doing this work
  341. 11:48myself. So, I'm literally trusting this
  342. 11:50model to wire money for me. It's
  343. 11:52trustworthy enough for that, and
  344. 11:54honestly, I don't know if I would have
  345. 11:55trusted Astra with that due to the
  346. 11:56spikiness. 6.1 Soul, much, much less
  347. 11:59spiky. From what I've heard from the
  348. 12:01other testers, they seem to agree with
  349. 12:02this analysis. I know for a fact that
  350. 12:04Julius and Ben, who have also been
  351. 12:05testing, have had a much better
  352. 12:07experience with this than Astra in terms
  353. 12:08of the spikiness. Julius called the
  354. 12:10model incredible. Ben called it
  355. 12:11incredibly boring. And I think that's
  356. 12:13the best place you can be for a model
  357. 12:15drop like this. But, as I had mentioned
  358. 12:16before, its peaks are not as impressive.
  359. 12:19While it does quality work the majority
  360. 12:21of the time, there are some tasks that
  361. 12:23are just at the edge of its capability
  362. 12:25that it will start to do weirder stuff
  363. 12:27on. For the most part, it's fine, but I
  364. 12:30I'm still reaching for Opus a decent
  365. 12:32bit. We'll talk more about the
  366. 12:32comparison later. I do default to this
  367. 12:35model for a bunch of stuff, though.
  368. 12:37First off, as I mentioned before,
  369. 12:39computer use. I can't wait for ultrafast
  370. 12:41to be like an actual thing you can use
  371. 12:43with OpenAI models, because when it is,
  372. 12:45this model is going to be crazy on it,
  373. 12:47because it can already figure out how to
  374. 12:49navigate computer use totally fine. If
  375. 12:51it can suddenly do it six times faster,
  376. 12:53it's going to be unbelievably fun. Still
  377. 12:55not quite as good as Astra, but more
  378. 12:57than good enough that for the price
  379. 12:58difference, I wouldn't even think twice
  380. 12:59about it. But, as I mentioned before,
  381. 13:01there are certain things I would still
  382. 13:03occasionally use Astra for that I am
  383. 13:05more than happy to use Soul for. One of
  384. 13:07those things is deep code reviews. I
  385. 13:10have still found OpenAI models and the
  386. 13:12like Rottweiler nature where they'll dig
  387. 13:14into a problem and shake it into tiny
  388. 13:16pieces until they find every single
  389. 13:17thing wrong with it. I find 6.1 Soul to
  390. 13:19be incredibly capable in this particular
  391. 13:21way. So, as you can probably guess, I
  392. 13:24had 6.1 Soul do some deep audits on
  393. 13:27Orchestrator V2 and other parts of my
  394. 13:29real-world codebases. In my Orchestrator
  395. 13:31V2 audit, it performed nearly
  396. 13:33identically to Astra. I do believe it
  397. 13:35was slightly higher a score. Okay, not
  398. 13:37in this analysis, but in my other
  399. 13:39analysis, it did actually score very,
  400. 13:40very slightly higher. But, it did it at
  401. 13:42about half the price, 297 versus 584.
  402. 13:46Sonnet was still cheaper, and Opus was
  403. 13:48slightly cheaper as well. The difference
  404. 13:50being neither of these models were
  405. 13:52anywhere near as thorough with their
  406. 13:54analysis. 6.1 Soul dug deep to find
  407. 13:59things, which is why it was able to get
  408. 14:00a score comparable to Astra, although it
  409. 14:02did admittedly burn way more tokens.
  410. 14:04Another task I've had a lot of fun
  411. 14:06testing with is asking the model to find
  412. 14:07opportunities to improve a codebase. In
  413. 14:10this case, to improve D3 code. This is
  414. 14:12the one where Grok 4.7 scored strangely
  415. 14:15well. Of course, Astra scored way better
  416. 14:17at an 83.8 versus the 80.7 from Grok
  417. 14:204.7. But, GBD 6.1 Soul hit it out of the
  418. 14:23park with an 87.4.
  419. 14:25I didn't save all the prices for these
  420. 14:27runs. It's been a bit, okay? But, for
  421. 14:29Opus 5.5, it cost five bucks. And for
  422. 14:32Sonnet 5.5, it cost almost $9. With GBD
  423. 14:366.1 Soul, it was $2.15.
  424. 14:39That's the difference. This model's
  425. 14:40token price is cheaper than Sonnet, but
  426. 14:43its token utilization is still
  427. 14:45maintaining OpenAI's usual efficiency,
  428. 14:48which results in just crazy price to
  429. 14:50performance. This whole thread was
  430. 14:52particularly fun because I had Opus 5.5
  431. 14:54review this model with a different name,
  432. 14:57obviously, so it didn't know what it
  433. 14:58was. I went and edited the history
  434. 14:59after. And it concluded very quickly
  435. 15:01this was a frontier tier model. Its
  436. 15:03reviews and bug repros matched the fixes
  437. 15:05that later merged. Its first draft code
  438. 15:07had real bugs, which review bots caught.
  439. 15:09Four reviewers are still running. Local
  440. 15:11team for code reviews backed frontier
  441. 15:13tier again. In a blind 10-model bench on
  442. 15:15the same prompt, 6.1 Soul placed first
  443. 15:18with the 87.4. It found the fish slop
  444. 15:20runs and compared those two. It did say
  445. 15:226.1 souls quality output was slightly
  446. 15:25below Astra's as well as the two open AI
  447. 15:27models with Opus 5.5 and sauna 5.5,
  448. 15:30which we will absolutely show you in a
  449. 15:31bit. But I do want to call out the price
  450. 15:33here cuz it only costs $7 to run versus
  451. 15:3615 for sauna 5.5 and 50 for Opus. Opus's
  452. 15:41honest tier call was that this model is
  453. 15:43incredible for scoped work, top of the
  454. 15:45frontier. For find what's wrong and tell
  455. 15:47me the truth, I would choose it over
  456. 15:48Astra and about level with Opus. For
  457. 15:51long unattended building, this was below
  458. 15:53frontier. Follows its process rules even
  459. 15:55when they stop all progress and it does
  460. 15:57not ask for help. This I absolutely
  461. 15:59noticed. I had mentioned before, well, a
  462. 16:01few times now, that my TS rust port that
  463. 16:03I'm working with Opus 5.5 is going way
  464. 16:05better than when I was working on that
  465. 16:06same port using Astra and soul in the
  466. 16:09past. I had that port running for a
  467. 16:11while with this model and it made no
  468. 16:13progress. It burned a shitload of
  469. 16:14tokens, but it didn't actually improve
  470. 16:16the compiler at all. Opus was able to
  471. 16:18from scratch restart it and get it
  472. 16:21working in a day after I had spent
  473. 16:23months and hundreds of thousands of
  474. 16:25dollars in tokens with this model as
  475. 16:27well as with Astra and 5.6 soul. Opus
  476. 16:29did it in like a grand in like a night
  477. 16:31with just two subscriptions with the
  478. 16:33Claude plan. So, for unattended long
  479. 16:36like heavy rewrite type stuff, Anthropic
  480. 16:38is just comically far ahead right now.
  481. 16:41And it also didn't have great judgment
  482. 16:42when I was using it for managing my
  483. 16:44fleet. And for those wondering, my fleet
  484. 16:45is all the computers I use for running
  485. 16:47all my agents in code cuz one computer
  486. 16:48is far from enough. I don't run any of
  487. 16:50them on this MacBook now. So, when I use
  488. 16:52this model to manage the fleet, it made
  489. 16:53a couple dumb mistakes here and there.
  490. 16:55To be fair, so is Opus. Astra's the only
  491. 16:57one that hasn't really made too many of
  492. 16:58those dumb mistakes, but like I'm going
  493. 17:00to be so real, I am entirely done using
  494. 17:02Astra after this model. After I had Opus
  495. 17:04do all of this review, I asked it how
  496. 17:06much does it think this model should
  497. 17:07cost. It guessed $5 per mil in, 50 cents
  498. 17:11cashed, and 30 per mil out, putting it
  499. 17:14at Opus's prices roughly. It said that
  500. 17:16cuz it's performing like Opus, its speed
  501. 17:19should add a premium cuz it is quite
  502. 17:20fast. And it's not a pro model, which is
  503. 17:23where expects those higher like $100 out
  504. 17:26tiered pricing things to come. Pro model
  505. 17:28isn't really a thing anymore. We just
  506. 17:29use Fable and Aster, but you get the
  507. 17:31idea. This is the funniest part of the
  508. 17:32whole thread though. If OpenAI wants
  509. 17:34people to adopt it, I would expect $3
  510. 17:37per mil in, 30 cents for cashed, and $20
  511. 17:40instead. That would still be a fair
  512. 17:42price for what it does. To which I
  513. 17:44responded, if I told you it was $2 and
  514. 17:46$10 out and 10 cents per mil cash read,
  515. 17:49what would you think? I'd call that very
  516. 17:51aggressive pricing. For how you use it,
  517. 17:52it costs about a quarter of what I
  518. 17:54guessed. The cash price does most of the
  519. 17:56work. Agent workloads are 96% cash
  520. 17:59reads, so 10 cents for cash reads
  521. 18:01matters more than the $2 and $10
  522. 18:02headline prices.
  523. 18:04When it looked at all of my sessions,
  524. 18:05its price guess would have been $5,700,
  525. 18:08but after looking at these new prices,
  526. 18:10it redid the math and it would have been
  527. 18:11$1,550.
  528. 18:13That is a massive decrease. And for all
  529. 18:16my PR review type tasks, it was
  530. 18:18expecting those to be up to $10 and it's
  531. 18:20actually only up to $3. And that's for
  532. 18:22like heavy PRs with tens of thousands of
  533. 18:24lines of code. According to Opus, so
  534. 18:27don't blame me, blame Opus for saying
  535. 18:28this. First off, Opus says it becomes
  536. 18:31the default model for scoped work.
  537. 18:32Second off, it says that bloated system
  538. 18:35prompts barely matter anymore because of
  539. 18:37the cash pricing. It's just noise.
  540. 18:39Third, it says long loops are still a
  541. 18:40bad idea, but not because of money. It's
  542. 18:42cuz according to it, the TSR port wasted
  543. 18:44four days and made no progress at all.
  544. 18:47And Opus even said they'd be suspicious
  545. 18:49of it lasting. They expect this price to
  546. 18:51go up in the future. I cannot fathom
  547. 18:53OpenAI ever increasing the price for a
  548. 18:55model, but Opus thinking they will is
  549. 18:57hilarious and shows just how good a
  550. 18:59value the model is. I love this call out
  551. 19:01here. GPT-4 1 sold did three rounds of
  552. 19:03work for about half the cost of Sonnet's
  553. 19:06single round. A lot of this comes down
  554. 19:08to how context was managed. Well,
  555. 19:10because 6.1 sold is much more efficient,
  556. 19:11so it's not doing as many calls that
  557. 19:14bloat the context. It's not outputting
  558. 19:16as many tokens that are like building up
  559. 19:17over time. So, the average number of
  560. 19:20tokens being read per request was only
  561. 19:23around 110,000 tokens versus 360,000 for
  562. 19:26Sonnet 5. The result was that Soul used
  563. 19:28under half as many input tokens as
  564. 19:30Sonnet, making this model significantly
  565. 19:32more efficient. Speaking of efficiency,
  566. 19:34I want to talk about these deep SW
  567. 19:35scores a tiny bit more because this is a
  568. 19:38weird bench for me to a fork and include
  569. 19:40in these things. I actually did it for a
  570. 19:42different reason, not to compare against
  571. 19:446.1 Soul, but to compare against a new
  572. 19:47release from OpenRouter, Jev Router.
  573. 19:50OpenRouter added Jev Router to try and
  574. 19:52optimize costs with your requests, and I
  575. 19:54thought it was an incredibly stupid
  576. 19:56idea. Once I started running it against
  577. 19:57benchmarks, I confirmed it's an
  578. 19:59incredibly stupid idea. It turns out a
  579. 20:01model that cannot reason, that is given
  580. 20:03a prompt into no context, cannot make a
  581. 20:05good decision around how hard the
  582. 20:07problem is. And Jev Router ended up
  583. 20:10being DeepSeek V4.1 Flash Router for the
  584. 20:13vast majority of its runs. Around 60% of
  585. 20:16all the requests went straight to
  586. 20:17DeepSeek 4.1 Flash. So, didn't like it
  587. 20:19that much. It also routes to other
  588. 20:21smarter models, which should give it
  589. 20:23more of an advantage, but it ended up
  590. 20:25being more expensive than GPT-6 Astro
  591. 20:27was on low, while also taking four to
  592. 20:30five times longer cuz 6 Astro low took
  593. 20:324.6 minutes, and Jev Router took 20. Jev
  594. 20:36Router's average task took 104 steps,
  595. 20:39whereas GPT-6 Astro took 19. You get the
  596. 20:42idea, it wasn't very good. But, the
  597. 20:44whole point of Jev Router is that it
  598. 20:45would be as cheap as possible to get a
  599. 20:48certain score. That was the promise on
  600. 20:50the tin. Whether or not you believe them
  601. 20:52is up to you, not me. I think it's
  602. 20:54Regardless, Jev Router was
  603. 20:57routing to DeepSeek 4.1 Flash for the
  604. 21:00majority of its requests. Despite Jev
  605. 21:02Router routing to the cheapest possible
  606. 21:05small open-weight models from whatever
  607. 21:07provider will give it away for free. 6.1
  608. 21:09Sonnet low got the same score for an
  609. 21:12eighth the price. OpenAI is here to
  610. 21:14destroy any wins anyone else has in
  611. 21:17terms of efficiency. Completing this
  612. 21:19bench in 4.8 minutes for 21 cents with
  613. 21:22the second highest score I've ever seen
  614. 21:23on it is a massive achievement, tying
  615. 21:26Opus 5.5, which took 50 minutes per task
  616. 21:30on max. A tenth the time and a
  617. 21:33seventieth the price for the same score.
  618. 21:35If your work fits within the things 6.1
  619. 21:38Sonnet does well, you should probably
  620. 21:40use it for everything. But, if your work
  621. 21:42doesn't fit in it particularly well, you
  622. 21:44should probably keep using Opus and
  623. 21:46maybe give Opus the ability to call 6.1
  624. 21:49Sonnet when it should for various tasks.
  625. 21:51I'm almost certainly going to be setting
  626. 21:52things up so that Opus 5.5 can call 6.1
  627. 21:55Sonnet to do investigation work to try
  628. 21:57and like root cause bugs, to do analysis
  629. 22:00of code bases to figure out what things
  630. 22:02need to be touched and why, to help me
  631. 22:04triage real-world work, to help me
  632. 22:06review the work that Opus does, and
  633. 22:08more. I'm kind of spoiling the ending
  634. 22:10here, aren't I? I'm going to keep using
  635. 22:12Opus 5.5 for now. Before I explain why,
  636. 22:15let me do the thing that I'm most
  637. 22:17excited for.
  638. 22:18Fish slop.
  639. 22:20The first thing you might have noticed
  640. 22:21is the inclusion of slop in fish slop.
  641. 22:25This model did the horrible thing I hate
  642. 22:28where it surrounded the game in a bunch
  643. 22:31of absolutely garbage UI. And coming to
  644. 22:34this right after the 5.5 Sonnet demo
  645. 22:37hurts me deeply
  646. 22:39because Sonnet 5.5 did not make graphics
  647. 22:41anywhere near this good looking,
  648. 22:43but at least it made a UI that was
  649. 22:45nowhere near this awful. And man, do I
  650. 22:48wish the bad UI is where the issues
  651. 22:50stopped. I will turn on the sound.
  652. 22:53Oh god, it's blaring.
  653. 23:00It is stunning looking.
  654. 23:02The fish are some of the best. The model
  655. 23:04for the sub is way better. The
  656. 23:06propellers work way better.
  657. 23:08I'm going to mute the sound cuz that is
  658. 23:10looking pretty bad. I haven't even heard
  659. 23:11it, honestly.
  660. 23:14But damn, like
  661. 23:16looks beautiful,
  662. 23:17but if you actually are playing it, one
  663. 23:19of the first things you'll notice is
  664. 23:20that the movement feels significantly
  665. 23:23worse than it does in either the Opus or
  666. 23:26the Sonnet versions that I've demoed in
  667. 23:28the past.
  668. 23:32Yeah, it it moves jank. It also has a
  669. 23:35significantly worse frame rate than the
  670. 23:37versions from the other models. It does
  671. 23:40have higher graphic fidelity, so that
  672. 23:41makes sense. Like the models here with
  673. 23:44the plants are significantly better than
  674. 23:47they were with the Sonnet version. The
  675. 23:49details of the fidelity of the extras in
  676. 23:50the tank is absolutely hilarious.
  677. 23:54Like
  678. 23:55yeah.
  679. 23:57But god damn, I'm so tired of the
  680. 23:58unnecessary text everywhere. This model
  681. 24:01does it worse than almost any I've ever
  682. 24:02seen before. We got take a breather,
  683. 24:05paused,
  684. 24:07your little world can wait, back to the
  685. 24:09reef, start a new tank, slot 01, feeder
  686. 24:13submarine,
  687. 24:15a little underwater chaos,
  688. 24:18the big little goal, your little
  689. 24:20ecosystem, little fish become big
  690. 24:22earners, four meals and they're all
  691. 24:24grown up,
  692. 24:26make the family a little bigger, a
  693. 24:27little golden overachiever. It is so
  694. 24:30many of these. There's like 20 plus of
  695. 24:32them.
  696. 24:33And I promise you guys, as soon as I saw
  697. 24:35this, I took a screenshot, I sent it to
  698. 24:37Open AI, and I crashed out in the slack
  699. 24:39because I cannot fathom how they haven't
  700. 24:41fixed this problem. This model
  701. 24:43is unacceptably garbage at UI. It has
  702. 24:46regressed again. And if you're looking
  703. 24:48for a model that can make front ends
  704. 24:49that don't suck, go spend your money
  705. 24:51somewhere else because it should not be
  706. 24:53spent here. This model sucks at front
  707. 24:55end. It sucks at design. It has no
  708. 24:56taste. And you're going to have to bring
  709. 24:58your taste yourself still. But it is
  710. 24:59admittedly really good at Blender. If
  711. 25:01you give it like a screenshot of a thing
  712. 25:03you want it to model in 3D and say,
  713. 25:04"Hey, you have Blender over the CLI. Go
  714. 25:06make this." It will, and it'll do a
  715. 25:08pretty damn good job.
  716. 25:09But I would never have it make the
  717. 25:10actual mechanics for my games because it
  718. 25:13feels awful to play. It also has like
  719. 25:16nowhere near as much gameplay loop. In
  720. 25:19fact, the first time I tried demoing
  721. 25:20this before filming,
  722. 25:22it just randomly game overed as I was
  723. 25:24like getting started in the first 30
  724. 25:26seconds. It never like said why.
  725. 25:28Actually, I think I technically beat it.
  726. 25:30I also want to like take this version
  727. 25:32and hand it to Opus or Sonnet and say,
  728. 25:33"Hey, can you make this play better?
  729. 25:35Because the graphics are good, but the
  730. 25:37game sucks." But when you combine how
  731. 25:39cheap it was to make this cuz like this
  732. 25:41was $5, I think, to generate. That's
  733. 25:44pretty insane. And if you combine that
  734. 25:47with like Ultra Fast, if that ever
  735. 25:49happens, suddenly you're going to be
  736. 25:51able to make a game in a few minutes on
  737. 25:54demand. We're actually now getting to
  738. 25:56that threshold where game development is
  739. 25:58about to flip upside down because of how
  740. 26:00models are finally understanding
  741. 26:02three-dimensional space and the tooling
  742. 26:03necessary to do these types of things.
  743. 26:05It's happening. As per usual, I was not
  744. 26:07allowed to put the code I wrote with
  745. 26:09this model inside of T3 Code or other
  746. 26:11open-source projects during the testing
  747. 26:13window. So, I had to use it exclusively
  748. 26:15on my internal projects like Lakebed, as
  749. 26:17well as for auditing other work, which
  750. 26:19means I mostly use this for auditing
  751. 26:21other work, and I was very impressed.
  752. 26:22This is a real PR I was working on to
  753. 26:24fix a bug where my new little work tree
  754. 26:27like setup window that would appear in a
  755. 26:29new thread in T3 Code would disappear if
  756. 26:31you left and came back. I had Claude
  757. 26:32Code work on this, but this problem went
  758. 26:34pretty deep. So, I wanted to make sure
  759. 26:37that whatever solution I came up with
  760. 26:39was very, very, very well vetted. While
  761. 26:41I personally still do not trust this
  762. 26:43model to write the code I'm trying to
  763. 26:45land, I absolutely trust it to review
  764. 26:47things. Ignore the GPT-6 soul there,
  765. 26:49just a placeholder. So, when I had 6.1
  766. 26:52Soul look through this, it found real
  767. 26:55problems that were entirely missed by
  768. 26:57Fable and by Opus. First, it called out
  769. 26:59that follow-up messages can stay blocked
  770. 27:01after the agent starts, which is very
  771. 27:03annoying if you want to queue a message,
  772. 27:05and also that recovered setup progress
  773. 27:07was disappearing too early. It figured
  774. 27:09all of these things out with a
  775. 27:10combination of reading the code and
  776. 27:11analyzing it as well as computer use. It
  777. 27:13was able to prevent me from merging a
  778. 27:14real regression in T3 code. So, I
  779. 27:17literally just copy-pasted those things
  780. 27:18to Claude and then told it to take
  781. 27:20another look. Said, "Better, but I'll
  782. 27:22still fix two small gaps." Remember, I
  783. 27:24can't use this model of the code for
  784. 27:26this project at the time, so I
  785. 27:27copy-pasted that again over to Claude,
  786. 27:29and it eventually got it good enough,
  787. 27:31and then I finally merged. But, that is
  788. 27:33what I like this model for, and I cannot
  789. 27:35wait to push its limits for actually
  790. 27:36coding. Although, I will say from the
  791. 27:38code that I did have the misfortune of
  792. 27:40reading, it is harder to justify merging
  793. 27:43this code than it is for code from Opus.
  794. 27:46Normally, I would make you guys wait for
  795. 27:48the Opus versus Soul video or the Sonic
  796. 27:50versus Soul video, but I'll just spoil
  797. 27:51the details now. I like using them in
  798. 27:53tandem because I find Soul to be way
  799. 27:55better at reviewing and digging into the
  800. 27:57details, but I find Opus a more pleasant
  801. 27:59collaborator and significantly better at
  802. 28:01actually implementing code without
  803. 28:03getting blocked constantly throughout
  804. 28:04its work. And even now with the Rust
  805. 28:06rewrite of TypeScript, I find myself in
  806. 28:08a similar pattern, where I have Opus 5.5
  807. 28:11just going and going and going, making
  808. 28:13the code base work and what work well,
  809. 28:15and then I had Soul come in and do an
  810. 28:17audit. And this is the funniest part.
  811. 28:19Remember before I said that I had Soul
  812. 28:21and Astra working on that TS Rust port
  813. 28:23for effectively months? I told Soul to
  814. 28:26come in, and it got it from 83.7% to
  815. 28:28100% in under a day. I was blown away by
  816. 28:32that, that it had somehow unblocked the
  817. 28:33work that Astra and Soul were doing, as
  818. 28:35well as 6.1 Soul. And I was absolutely
  819. 28:37blown away by that, that it had taken
  820. 28:39the work that 5.6 Soul, 6.1 Soul, and
  821. 28:41Astra had done over months, and got it
  822. 28:44unblocked where it had been stuck for
  823. 28:45weeks, and finished it. I was much more
  824. 28:47blown away when I had 61 Soul take a
  825. 28:50look at that work and critique it. And
  826. 28:53what it brought up was that of the 1.8
  827. 28:56million lines of code, 1.3 million were
  828. 28:59not being used. The reason was because
  829. 29:02Opus concluded whole of the code from
  830. 29:04all the other agents was useless slop
  831. 29:06that had no chance of being recovered,
  832. 29:07and it chose to rewrite it from scratch
  833. 29:09itself in another crate. So, on one
  834. 29:11hand, the only reason the code worked
  835. 29:13was Opus, but on the other hand, the
  836. 29:15only reason the slop was still around
  837. 29:16was also Opus. So, I had to have this
  838. 29:18model come in and clean up the mess that
  839. 29:21other OpenAI models had made because
  840. 29:23Opus didn't even notice the mess was
  841. 29:25still there. What I'm trying to say is
  842. 29:26this model absolutely has a place in
  843. 29:28your workflows. It could probably even
  844. 29:30be your default coding model, and you
  845. 29:32wouldn't have too many issues with it,
  846. 29:34but I still find Opus to be a better
  847. 29:35collaborator overall. That said, I have
  848. 29:38almost no reason to use Sonnet anymore
  849. 29:40because this will effectively take its
  850. 29:42place. And you bet your butt the moment
  851. 29:44this model comes out, I'll be going and
  852. 29:46making adjustments inside of my Claude
  853. 29:47config because I already have it set up
  854. 29:49so that I can use Soul inside of Claude
  855. 29:51code because I want to make sure Opus
  856. 29:53knows this is the model to have review
  857. 29:55its work and investigate the things
  858. 29:57going on in the code base. This is a
  859. 29:58damn good model, and I'm really happy to
  860. 30:00have it. I wish we had something bigger,
  861. 30:02smarter, and more capable overall. I was
  862. 30:04really hoping for something to truly
  863. 30:06dethrone Opus 5.5 as my daily driver.
  864. 30:08This isn't it, and I'm not planning on
  865. 30:10canceling any of my Claude subs as a
  866. 30:12result of this release, but I am
  867. 30:13planning on taking a lot more of my
  868. 30:15Codex subs in my day-to-day work,
  869. 30:17admittedly in Claude code. This is an
  870. 30:19awesome release and an unbelievable
  871. 30:21price for what you're getting, but this
  872. 30:23does potentially mark the start of the
  873. 30:25end of the subsidization era. So, make
  874. 30:27sure you're subscribed so that you can
  875. 30:28be here when I cover all of that and
  876. 30:30more. God, I hope this isn't get me
  877. 30:31canceled online. I have no idea how
  878. 30:33others feel about this beyond like a
  879. 30:34handful of early access testers I've
  880. 30:35talked to. I legitimately don't know if
  881. 30:37you all are going to love it or hate it
  882. 30:38or land somewhere between. I will know
  883. 30:40in a few hours, I guess, cuz it's a
  884. 30:43Yeah, it's 3:00 in the morning. I am
  885. 30:45going to go to bed now.
  886. 30:47Uh,
  887. 30:48this is a tiring one. Hopefully I did a
  888. 30:50good job. Let me know in the comments.
  889. 30:51And until next time,
  890. 30:53peace nerds.
  891. 30:55God, I'm so dead.
  892. 30:57Ah.

About this transcript

This page contains the full transcript of OpenAI fights back by Theo - t3․gg, generated from the public captions YouTube serves with the video. The transcript has 6,324 words across 892 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.