OpenAI is getting desperate — Transcript

Full transcript

1. 0:00Recently, it feels like Anthropic can't
2. 0:01stop winning, but on the other hand, it
3. 0:03feels like OpenAI just can't stop
4. 0:04losing. From the horrible changes where
5. 0:06they reduced our limits terribly on the
6. 0:08$200 plan to the semi-flop of GPT-6 1
7. 0:11Soul and the lack of any real follow-up
8. 0:13for GPT-6 Astra to compete with
9. 0:15something as powerful as Opus 5.5 from
10. 0:17Anthropic, it's just kind of been a
11. 0:19losing streak for us Codex fans
12. 0:21recently. And I'm counting myself as one
13. 0:22of them because I've historically been
14. 0:23one of the defenders of Codex and
15. 0:25OpenAI. And right now, it's pretty hard
16. 0:27to justify using. All that said, it does
17. 0:29feel like we're finally getting back on
18. 0:31the right track. Thibault just made a
19. 0:32huge announcement that I'm really
20. 0:34excited about. As of October 29th, they
21. 0:36are doubling all usage limits for all
22. 0:38pro accounts, whether it's the $100
23. 0:40tier, the $200 tier, $500 tier, as well
24. 0:42as the business premium, effective
25. 0:44today. There are layers to this one.
26. 0:46They are making changes that might give
27. 0:48you less usage in certain case for
28. 0:50certain people. They are now bundling
29. 0:52things together they weren't before,
30. 0:53which can get a little nasty. And the
31. 0:55$200 tier in particular is uh
32. 0:59messy, to say the least. There's a lot
33. 1:01to dive into from how these limits will
34. 1:03affect our day-to-day use to how much
35. 1:05work we can actually get done in each of
36. 1:07the tiers to how much a dollar value are
37. 1:09we getting when we put in 200 bucks,
38. 1:11what do we take out? And of course, the
39. 1:12most important question, how does this
40. 1:14compare to the value we get from our
41. 1:16Claude code subscriptions? Because
42. 1:17normally when we compare value, we're
43. 1:19comparing API prices. The sub prices are
44. 1:21different and the amount of usage you
45. 1:23get is different, so how does this all
46. 1:24line up and which plan should you be
47. 1:26getting today? There's a lot of layers
48. 1:28to this one that make it complex, but
49. 1:29there is one thing I know you should get
50. 1:31and it's much easier to justify. Today's
51. 1:33sponsor, Agents have made a lot of
52. 1:34things easier. Sadly, auth isn't one of
53. 1:36them, and that's why I can't wait to
54. 1:38show you guys today's sponsor, WorkOS.
55. 1:40What? Passport? You've already heard me
56. 1:42talk about WorkOS. They just introduced
57. 1:44a very different new thing, and I want
58. 1:45to talk about this a bit cuz I think
59. 1:47it's actually pretty cool. Stop me if
60. 1:48you've been here before. You're opening
61. 1:50the laptop up after a long weekend off
62. 1:52and you're greeted by a ton of sign-in
63. 1:54screens for all the different things you
64. 1:55use. Wouldn't it be nice if there was
65. 1:57one a that everything was connected to
66. 1:59so that your computer was the off layer,
67. 2:01that would already be pretty cool. But,
68. 2:02where it gets way cooler is with agents,
69. 2:05because you can use this off layer to
70. 2:06allow your agents to do useful things
71. 2:08with the MCP servers for all the apps
72. 2:10that you're using. More importantly
73. 2:11though, for those enterprises in the
74. 2:13world that have strict policies here,
75. 2:15you can have company-wide policies
76. 2:17applied to the MCP servers that you are
77. 2:19governing through the passport platform.
78. 2:21This allows for your employees to set up
79. 2:22passport on their computer directly,
80. 2:24sign in to all of their apps, and now
81. 2:26everything is managed through this
82. 2:27layer, allowing them to use agents to do
83. 2:30real work. And once you have it
84. 2:31configured, your agents can now access
85. 2:33everything you've signed into, whether
86. 2:34using Grokbot, ChatGPT dots, Claude
87. 2:37code, or I don't know, that weird indie
88. 2:38T3 code thing, they have you covered.
89. 2:40Sign up for early access today at
90. 2:41soydev.link/passport.
91. 2:43Before we can really talk about this
92. 2:45announcement correctly, we need to go
93. 2:46back in time a little. Back in time to
94. 2:49when Tibo was SA dumping on X. All the
95. 2:53way back on September 28th this year,
96. 2:55feels like years ago at this point, Tibo
97. 2:57made this long-ass announcement about
98. 3:00how the $200 tier was going to have
99. 3:02changes. Specifically, if you were to do
100. 3:05the math, it would net out to half the
101. 3:06dollar amount in API spend compared to
102. 3:09the old Pro 200 plan. He then tries to
103. 3:12justify how we'll actually get more work
104. 3:13done than we would have on the previous
105. 3:16subscription a month ago, including
106. 3:18things like committing to not having
107. 3:195-hour limits anymore, to guaranteeing
108. 3:22you'll always get more work done with
109. 3:23increasing levels of quality. This is
110. 3:25actually a good guarantee we'll talk
111. 3:26about in a bit, this idea that as models
112. 3:28get smarter and more capable, you should
113. 3:30always be able to do more work on a
114. 3:32given subscription tier than you were
115. 3:33before. They will get there. The biggest
116. 3:36part here is C though. They don't want
117. 3:37to have an incentive on themselves to
118. 3:39artificially inflate API list prices to
119. 3:41make it look like they're getting a lot.
120. 3:43Instead, they want to continue to both
121. 3:44rapidly reduce prices and increase
122. 3:46capabilities of models on the API.
123. 3:49It is worth noting that this is around
124. 3:50the time when 601 soul dropped, and 601
125. 3:52soul was, I believe, like a day or two
126. 3:54after with OpenAI Dev Day. That model is
127. 3:57way cheaper than they had originally
128. 3:59probably planned to serve it for. That's
129. 4:02why they had to make this change because
130. 4:04if they made the model cheaper on API
131. 4:06and they give you the same dollar amount
132. 4:08outside of API, then they're eating more
133. 4:10compute than they would otherwise have
134. 4:11to. If you're curious how the economics
135. 4:13of all that work, take a look at the
136. 4:14video I just put out about Claude Ultra
137. 4:16plan where I discuss this all in much
138. 4:18more detail. There is something worth
139. 4:20noting in this announcement though. The
140. 4:22whole thing's about the pro $200 plan.
141. 4:24None of these changes affected the other
142. 4:27plans, which means that today's change
143. 4:30restores the $200 plan to where it was,
144. 4:32but it also doubled what you get for the
145. 4:34$100 plan and for the $500 plan. And the
146. 4:37$500 plan doubling is pretty nuts
147. 4:40because it means that that plan is now
148. 4:4150x usage compared to the pro tier. This
149. 4:44is a pretty insane amount of usage that
150. 4:46you can probably only blow through if
151. 4:48you're using fast and ultra fast
152. 4:49heavily. I cannot fathom how on normal
153. 4:52speeds using their best model right now,
154. 4:54which is 61 sole, not Astra, sole, how
155. 4:57you could ever get through the $500
156. 4:59tier, but yeah, worth noting. So, we
157. 5:02should start by looking at the numbers
158. 5:03of how much dollar value you're getting
159. 5:05even though as I just said, it's not the
160. 5:06right way to measure. It's a worthwhile
161. 5:08thing to note. And here we see that on
162. 5:10the $500 plan, you now get around 10.5
163. 5:13thousand dollars of usage. And on the
164. 5:15$200 plan, you get around 4.3k of usage.
165. 5:18So, those are still both heavily
166. 5:20subsidized, but remember, the $200
167. 5:23Claude plan gets you 12 grand in usage.
168. 5:25It does have weird restrictions like
169. 5:27half limits on Fable, which thankfully
170. 5:30at least as of the time of recording,
171. 5:32there is no Fable model worth using over
172. 5:34Opus, so the 50% limit doesn't affect
173. 5:36you quite as badly. But even then, the
174. 5:38$200 Claude plan with the 50% cut gets
175. 5:40you six grand of usage, where the $200
176. 5:43plan on Open AI, no matter what, only
177. 5:46gets you 4.3k. So, you're still getting
178. 5:48much less dollar value when you compare
179. 5:52the Open AI models and the Anthropic
180. 5:54models on the subscriptions. But how
181. 5:56does it compare for your real-world
182. 5:57usage? As I've been trying to explain in
183. 6:00my videos for a while now, Open AI
184. 6:02models are much cheaper. Not just per
185. 6:04token cuz per token pricing is garbage.
186. 6:07There's three ways models are
187. 6:08effectively being priced that we care
188. 6:10about. There is how much do they charge
189. 6:11per token in and per token out. There's
190. 6:13also like caching involved there, which
191. 6:15changes the economics entirely. There is
192. 6:18how much tokens it take to solve a
193. 6:20problem, which is the thing I think most
194. 6:22people miss is they look at the dollar
195. 6:23price per token and they forget that
196. 6:25certain models use more and fewer
197. 6:26tokens. I actually visualize this here,
198. 6:28by the way. Notice how all the OpenAI
199. 6:30models are pretty far on the left there.
200. 6:32And even the worst case with something
201. 6:35like 61 Soul doing 68K tokens on max,
202. 6:38it's still around the same number of
203. 6:40tokens as medium or high would be on
204. 6:43Fable. And if you drop down to X high
205. 6:46for 61 Soul, it uses fewer tokens than
206. 6:49anything Anthropic has released recently
207. 6:51other than Opus 5.5 on low. So, X high
208. 6:5561 Soul does fewer tokens than Opus 5.5
209. 6:59medium and obviously scores
210. 7:01significantly better because the low and
211. 7:03medium reasoning is just not going to
212. 7:05compare to something like 61 Soul at its
213. 7:08best. Yeah, 61 Soul X high scoring a 52
214. 7:11on the Artificial Analysis Intelligence
215. 7:13Index and Opus 5.5 medium is scoring a
216. 7:1648. $2.45 a task versus
217. 7:20$0.76 a task. All right, this is log
218. 7:22scale. If we switch to linear, you'll
219. 7:24see the gap even more so because you're
220. 7:26paying more than 2x the price for medium
221. 7:29Opus and getting worse results than X
222. 7:32high Soul. So, again, pricing type
223. 7:35number two, per task pricing, we see
224. 7:38OpenAI just slaughtering Anthropic here.
225. 7:41But, I mentioned there's a third thing,
226. 7:42which is the thing we're actually
227. 7:43talking about today, which is the
228. 7:44subscriptions. So, what happens when you
229. 7:46combine this with the amount of usage
230. 7:48you get in the $200 plans? Adam Holters
231. 7:51been working on this a lot recently and
232. 7:53his numbers have been fascinating to
233. 7:55look at. He adjusted the prices based on
234. 7:58the discounts you get on the subs and
235. 8:00the numbers look very different now as a
236. 8:03result. There is still a gap here to an
237. 8:05extent, like this is cost per task. But
238. 8:07remember, lower is better. And here, 61
239. 8:11Soul on Max is costing by the subs under
240. 8:16half as much as you would get from Opus
241. 8:185.5 Max. But remember, I almost never
242. 8:21recommend Opus 5.5 Max. I basically
243. 8:23never recommend Max on any of the model
244. 8:25lines, especially in profit models. I
245. 8:28mostly rock high with the occasional
246. 8:30shift to XI. And you'll notice high on
247. 8:33Opus 5.5 is around 5 cents per task
248. 8:37versus 61 Soul on high, which is 4 cents
249. 8:39a task. I would put those two neck and
250. 8:41neck as like the two I reach to. I do a
251. 8:43lot of Soul XI and I do a lot of Opus
252. 8:455.5 high. And they're priced almost
253. 8:47identically because once you cancel out
254. 8:50everything else, the $200 plan being
255. 8:53priced in, the capability price gap gets
256. 8:56a lot smaller. And the value you get
257. 8:59ends up balancing out quite a bit. This
258. 9:01gets more interesting when you look at
259. 9:02how it affects the Pareto line. The
260. 9:04Pareto line is what is the best value at
261. 9:06a given level for different levels of
262. 9:08capability. And if we look at it for EPI
263. 9:10prices, you see some weirdness, like for
264. 9:13example, Haiku technically being
265. 9:15slightly smarter even though it's 2x
266. 9:16more expensive than Terra on low is. But
267. 9:20Terra immediately takes back over as
268. 9:22well as 61 Soul on low. There's not much
269. 9:25reason API price wise to be defaulting
270. 9:28to Haiku, at least according to these
271. 9:29numbers.
272. 9:30But you'll see for the most part, it's a
273. 9:31pretty smooth curve with that one weird
274. 9:33Haiku blip. The rest is just open AI
275. 9:36dominance. Even if you add in open
276. 9:38weight models, open AI tends to dominate
277. 9:40a lot on the Pareto line right now. With
278. 9:42Opus only coming out as the better value
279. 9:44at the absolute top because it has the
280. 9:47best performance right now. And if you
281. 9:49doubt that, I'm sorry. Opus 5.5 is the
282. 9:51best model available, at least as of the
283. 9:53time of this recording. But when we
284. 9:54switch over to the subs, you see it
285. 9:56flipping back and forth a hell of a lot
286. 9:58more. In fact, now that I'm looking at
287. 10:01it more closely, similar to how there
288. 10:03was that one weird dip here where Haiku
289. 10:05just barely technically qualifies,
290. 10:08there's a similar one of those here
291. 10:09where Soul High just barely technically
292. 10:11qualifies. The entire rest of the curve,
293. 10:15other than like this one blip for Soul
294. 10:16Low, is just Anthropic. Turns out the
295. 10:20first chart I was looking at wasn't
296. 10:21exactly right. This one looks a lot
297. 10:24better for OpenAI. I thought the reason
298. 10:26was because they included the 2x limits
299. 10:29here and they might not have in the
300. 10:30other chart I was in. That is not the
301. 10:31case. The reason is much sillier. This
302. 10:34version has resets because if you didn't
303. 10:36know this, OpenAI resets a lot, 11 times
304. 10:39in the last 30 days. So, if we turn that
305. 10:41off, then the limits get a lot worse and
306. 10:45things look closer. Again, shout out to
307. 10:47Adam for all the work he put into making
308. 10:49these charts and getting this data.
309. 10:51Nothing else is quite as advanced here
310. 10:53and the sub-limits are very much
311. 10:55important for us to understand. And it's
312. 10:57crazy to see how much the numbers shift
313. 10:59when you turn the resets on and off. The
314. 11:01reason's pretty obvious when you look at
315. 11:03the numbers here. From April 16th to
316. 11:05October 31st, we got 52 resets in Codex
317. 11:08and we got 14 in Claude and we haven't
318. 11:10gotten a single Claude reset for 30 days
319. 11:11now, which is sad cuz I like Claude
320. 11:13resets. This is still based on the price
321. 11:15for artificial analysis work though. And
322. 11:18sadly, when we look at real numbers for
323. 11:20real-world tasks, the the gap closes a
324. 11:23bit more. I mentioned in some other
325. 11:25videos that I have a new skill I've been
326. 11:26using called Council, where I let my
327. 11:28agents review the work I'm doing across
328. 11:30different models at once. So, I have
329. 11:32some changes I want to make, I have a
330. 11:33proposal, a plan, or something. And then
331. 11:35the Council comes in across three to
332. 11:37five different model families to give
333. 11:38feedback. This ends up being a really
334. 11:40good generator for like-for-like tests
335. 11:43because now I have a bunch of different
336. 11:44models that are given the exact same
337. 11:46task, review this change. And this means
338. 11:48I've been able to go through and analyze
339. 11:50all of my usage.
340. 11:52I had Opus and Soul judge my usage of
341. 11:55Opus and Soul together and apart, fully
342. 11:58blind obviously, to see which reviews
343. 12:00they prefer. And as I would hope nobody
344. 12:02is surprised by, Opus 5.5 was preferred
345. 12:05by both quite a bit. Opus preferred Opus
346. 12:0766 out of the 83 times. Soul preferred
347. 12:10Opus 54 out of the 83 times. So, Soul
348. 12:13preferred Opus 65% of the time, Opus
349. 12:15preferred Opus 80% of the time. There
350. 12:16are occasions where both prefer Soul.
351. 12:19It's just much less often. The median
352. 12:22cost for an Opus review was $1.65 and
353. 12:24the median cost for a Soul review was 36
354. 12:27cents. When you factor this against the
355. 12:29expected API value that you get for a
356. 12:32sub, which is around $3,300 for Opus
357. 12:35with Claude and about $1,000 with Soul
358. 12:37on the ChatGPT Pro $200 plan, it comes
359. 12:40out to a much closer gap here with Opus
360. 12:43giving you around 205 reviews on that
361. 12:47plan and the ChatGPT Pro sub giving you
362. 12:49290 reviews. So, again, while the price
363. 12:53difference might look extreme, the
364. 12:54actual amount of usage that you're
365. 12:56getting between the two is a lot closer.
366. 12:58And when you consider how much more
367. 12:59capable Opus 5.5 is, that obviously ends
368. 13:03up being the better value. Soul is
369. 13:05slightly faster, which is nice, and its
370. 13:06reviews it gets its work done faster.
371. 13:09But generally speaking, this is the
372. 13:10number that matters. You can get around
373. 13:11200 views with Opus on Claude and 290
374. 13:14with Soul on ChatGPT Pro. So, if a 30 to
375. 13:1750% bump in things you can do in a given
376. 13:21month is super, super important to you
377. 13:23and you're willing to take the hit with
378. 13:2561 Soul's capability, then do it.
379. 13:28But if you want the best possible model
380. 13:30and you're scared of the Claude sub not
381. 13:32being quite as good of a value as Codex,
382. 13:34that's not the case right now. In fact,
383. 13:36it's actually kind of crazy how close
384. 13:39these are in terms of usage right now.
385. 13:42It's not as big of a gap as I had
386. 13:43expected, especially now that this has
387. 13:45been doubled. Oh, yeah, almost forgot
388. 13:47cuz I know people are getting really
389. 13:49huffy about this. ChatGPT and work are
390. 13:52now included within your limits.
391. 13:54Previously, using work and normal chat
392. 13:58did not count against the same limits
393. 13:59that you had for Codex. This affects
394. 14:02maybe like 30 people cuz I don't know a
395. 14:04lot of people who use chat and work
396. 14:06enough that those would even have a like
397. 14:081% dent in their Codex limits. But if
398. 14:10you are one of those people, I would say
399. 14:12I'm sorry, but I'm honestly just
400. 14:13confused. Like, I would like to know
401. 14:15more about your use case cuz that's
402. 14:16crazy to me. The doubling should still
403. 14:17be more than enough to cover both sides.
404. 14:20I don't think people being outraged
405. 14:21about that are making much sense. Usage
406. 14:23in work and chat is obviously going to
407. 14:25use exponentially fewer tokens than
408. 14:28day-to-day Codex usage. Just look at the
409. 14:30numbers, guys. Come on. Don't don't be
410. 14:31annoying about that. The other question
411. 14:33is dots. Apparently, dots will still not
412. 14:36be affected by these changes and it will
413. 14:38not use your limits when working within
414. 14:40dots. At least I think. I'll be real
415. 14:43though, I have no idea. I don't like
416. 14:45dots. I have not found it to be pleasant
417. 14:47compared to the other assistant tools
418. 14:48that exist. I have not set up Duet yet.
419. 14:50I did get invited. I've mostly just been
420. 14:52using Grok about though. It's been
421. 14:53surprisingly solid. It's kind of crazy
422. 14:55that I have my OpenAI subs for computer
423. 14:57use and verification. I have my Claude
424. 14:59subs for all my day-to-day code work.
425. 15:01And then I have my one Grok sub just for
426. 15:03Grok bot. But, yeah. That's where we
427. 15:05are.
428. 15:06I think that's all I have to say on this
429. 15:08one. This is definitely a good change
430. 15:09and it's almost annoying how much people
431. 15:12are pushing back on the small parts that
432. 15:13don't matter. It is worth calling out
433. 15:16that the $200 plan is back where it was
434. 15:18a month ago. It's not a huge bump for
435. 15:19$200 subs. It's just a restoration of
436. 15:22where they were. The $500 plan getting
437. 15:242x is crazy and the $100 plan getting 2x
438. 15:26is pretty interesting as well. Overall,
439. 15:29I think this is a really good change,
440. 15:30but it is missing something important. A
441. 15:32model that is worth using. OpenAI, I'm
442. 15:35begging you. We need something better.
443. 15:37We need something that actually can go
444. 15:38head-to-head with Opus 5.5 because that
445. 15:41model is just unbelievable and nothing
446. 15:43that you guys have released is quite
447. 15:45there yet. I can't wait to see what you
448. 15:46cook. Until next time.
449. 15:49Peace nerds.