YT YouTube2Text

YouTube2Text → Transcripts → the hard truth about token spend

the hard truth about token spend — Transcript

by Theo - t3․gg · 7,404 words · 1,040 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00By now you guys have probably heard of
0:01Lauren, aka Potato, aka the creator of
0:03the P stack. She made a lot of other
0:05really cool things like the React
0:06compiler. I think she's awesome. But she
0:08recently went from Meta to working at
0:10Cursor and has quickly taken this entire
0:12agentic dev world by storm with her
0:14unique way of doing things. When I
0:16checked Twitter at the start of today, I
0:17saw this post going in on Lauren and
0:20people like her for wasting allegedly
0:23132 million dollars of tokens a year.
0:27132 million dollars a year. I don't know
0:32about you guys, but as someone who has
0:34used Cursor a good bit myself, I don't
0:37know if I would put the last few months
0:39of Cursor's improvement in that 132
0:42million dollar range. That I mean, you
0:45can get like three GPUs for that. That's
0:47a lot of money. All of this data comes
0:49from her profile on the Cursor site. You
0:52could see that in the last 30 days,
0:54she's done 1.5 trillion with a T tokens.
0:58I go pretty hard with my token use and
1:01I'm still not able to hit 100 bill. So,
1:03how the hell did she manage to do 15
1:06times more token usage than someone like
1:09me who is actively trying to waste
1:11tokens? And what the hell is she doing
1:13with 132 million dollars worth every 30
1:16days? Is this number just fake? Is there
1:19things being shipped that we don't know
1:21about? Is she just doing it for the
1:23memes? What's going on here?
1:26As you can guess, I have a lot of
1:27thoughts and as you can probably have
1:30predicted by now, most of these numbers
1:32don't make any sense at all. But I'm not
1:34trying to say your spend isn't crazy, it
1:36absolutely is. But what she is doing
1:38here is much more pioneering than she is
1:41getting credit for on Twitter. And
1:43sadly, since it's my job to clear up
1:45this type of misinfo, I'm going to have
1:47to drag you guys through yet another
1:48explanation of how companies like
1:50OpenAI, Anthropic, Cursor, and SpaceX
1:52AI, Meta, and all these token maxing,
1:55token wasting companies are actually
1:57spending their tokens and what we are
2:00working towards as they do it. In
2:02particular, I have to explain why what
2:04Zeeg said here about people working at
2:06the labs not understanding how no one
2:09can ever spend this much on tokens.
2:12I need to explain why this is a really
2:13[ __ ] stupid thing to post. Before we
2:15go into stupid and bad things, I want to
2:17talk about something awesome. Today's
2:18sponsor. I don't know if this goes for
2:20everyone, but personally, I've been
2:21running a lot more agents recently. And
2:23with this comes a lot of additional
2:25friction. Things like more builds, more
2:27time waiting on CI, more time just
2:29constrained by compute and RAM and
2:31network speeds and more. At least I was
2:33until I moved all that work over to
2:35Depot. These guys are here to speed up
2:36your agents and their dev workflows. All
2:38of the stuff they've been building was
2:39already useful for devs who are trying
2:41to ship quickly, but it's so much more
2:43useful when your agents are hitting
2:44those same bottlenecks. If you were
2:46waiting a little bit for CI yourself,
2:48that might have slowed you down a bit.
2:49When your agents are filing dozens of
2:50PRs a day, every second counts. And
2:53they're not going to save you seconds,
2:54they're going to save you hours if not
2:55more. They can be used as a drop-in
2:57replacement for GitHub Actions, but it
2:59gets way cooler once you start adopting
3:01their CI pipelines, which guess what?
3:03Your agents can do very easily. This
3:05allows you to parallelize the work
3:07that's being done and share much more of
3:08it, so your agents are able to test
3:10their code much, much faster. Let's go
3:12through some of the tools that will help
3:14your agents move faster. First, we have
3:15Depot CI, a fully programmable engine
3:17your agents can use to make faster
3:19parallelizable CI. It can be used fully
3:21without GitHub, so all those outages
3:22won't affect you. If you're using mono
3:24repos for building, Depot Cache will
3:26help a ton as well. They already work
3:27with pretty much all mono repo tools,
3:29whether you're in TypeScript with Turbo
3:30Repo or Nx, or you're in other
3:31ecosystems with Bazel and Gradle and
3:33other tools like that. If you're using
3:35Docker, I hope you already are using
3:36Depot because they will make all of your
3:38Docker commands way faster. Their cache
3:40is shared between you, your team, and
3:42your CI, so the likelihood that you have
3:44to build a full image from scratch is
3:45near zero, massively speeding up those
3:48builds up to 10 times faster. And this
3:49all can be stored in the Depot registry,
3:51which handles all of the images, all of
3:53the artifacts, and more that you and
3:55your agents will need to be building. If
3:57you want to go all in, they just put out
3:58Depot code, which is a Git server that
4:00uses no desks, stores everything in blob
4:02storage, that's way faster and closer to
4:05your CI, so all of the steps that
4:06normally take minutes will start taking
4:08seconds instead. Real users have seen
4:10improvements between 10 and 20x faster
4:12for their builds, CI, and more. Get them
4:14yourself at soydev.link/depot.
4:16If you were to ask your agent to
4:17summarize the video you're about to
4:19watch, it would say something along the
4:20lines of Theo points out the flaws in
4:23the calculation that Peter did and
4:25showcases how models will get cheaper
4:27over time or something along those
4:29lines. And that is the like top-level
4:31what we're about to do. The more subtle
4:32lesson I want you to take from this is
4:34learning how to identify the types of
4:36people who say things that sound really
4:39smart and agreeable in this AI dev
4:41space, but are just entirely wrong if
4:44you look at the numbers. I'm
4:45particularly frustrated around the types
4:47of people who just gladly, happily post
4:51these giant, inflated, incorrect things
4:52cuz they want to complain, and I want to
4:54help you guys learn to avoid them.
4:56Because you got a lot of people falling
4:58for things like what Peter said here and
4:59what Zig followed up with. And this is
5:01all just egregious. And I don't mean any
5:04ill will towards Peter or towards Zig,
5:07but I legitimately think you should
5:08never listen to anything either says
5:10about AI because they clearly don't know
5:11what they're talking about. I'm going to
5:12change my opinion here in the future if
5:14they stop lying so aggressively and just
5:16being wrong in adamant ways on the
5:17internet. But if the majority of the
5:19things they say about AI are factually
5:21wrong, I don't know why you would hunt
5:23so deeply to find the others. So let's
5:25go through all the factual inaccuracies
5:27of what Peter said here. We start with
5:281.5 trillion tokens in last 30 days.
5:31That's true. These first two sentences
5:32are true. Assume an average rate of $8
5:36per million tokens. Now we're at 384k a
5:39day or 11 million in the last 30 days.
5:42Per year, that's 132 mil. First, I want
5:45you to remember that our friend Lauren
5:47works at SpaceX AI. So, there's a good
5:50chance she is using models cheaper than
5:52things like Opus, Fable, and Astra.
5:54She's probably using Grok a good bit.
5:56So, let's dig into this $8 per million
5:59number. The token count that is being
6:01shared on Cursor profiles is an
6:05aggregate number. 1.5 trillion tokens.
6:08This is not split between the four types
6:10of tokens. Cash read, cash write, normal
6:13read, and normal out. There's four types
6:15of tokens, and they're all priced very,
6:17very differently. A model like Opus 5.5
6:20is $5 per million in and $25 per million
6:24out. If you average those two numbers,
6:26you end up with a number like $12 to
6:28$15. Knowing that I am almost
6:30exclusively using Opus 5.5, what do you
6:33think my average cost per million
6:37tokens, if you combine all the tokens,
6:39is? Get a number in your head. Remember,
6:41the number we cited for Lauren here, who
6:43is using Grok a lot, which is a cheaper
6:45model, is $8 per mil. What do you think
6:48my number per mil is? Spoiler, it's 41
6:51cents. He's off by 20x. He is literally
6:55opening with a number that is an order
6:57of magnitude and a half off. These
6:59people don't know what they're talking
7:01about. They're just lying. They're
7:02either lying or they're stupid, and it's
7:03annoying cuz Peter's a smart person. I
7:05don't know why the [ __ ] he posted this.
7:07And we're just getting started, don't
7:08worry. The reason that this number is so
7:10egregious is because cash read on Opus
7:14isn't the usual $5. It's 20 cents. And
7:18despite that, half my cost is cash read,
7:21a little less than half, but around half
7:23is cash read, half is cash write, and
7:26the input and output tokens, the ones he
7:28is basing the price on, make up less
7:30than 8% of my total spent. He averaged
7:33the two numbers at the far end that are
7:36less than 10% of my spend, and said
7:38that's the price for the whole thing.
7:40And we're just getting started, guys.
7:42Don't worry, though, he he prices this
7:43in because we should assume Cursor has
7:45some insane deal with Anthropic and
7:47they're getting 90% off. Then it's only
7:5038k a day or 1.1 mil in September and
7:53October. And he compares this to how
7:55expensive engineers are and says that 13
7:57mil, which is allegedly her budget for
8:00inference over a year, gets her 26 staff
8:03level engineers at 500k a year each.
8:06Well, I'm going to be real with you,
8:07Peter. I don't know how many staff level
8:10engineers I'd want to hire without
8:11giving them a crazy token budget because
8:13I honestly believe 26 staff level
8:16engineers with $0 in tokens would be
8:19meaningfully less productive than just
8:21Lauren by herself with unlimited tokens.
8:24I would argue comically so. He does
8:25follow up though. Let's say his math is
8:28off by a factor of 10 or they're running
8:29their own GPUs at only 80 cents, it
8:32still comes out to $100,000 a month.
8:34Remember, he is pricing in huge
8:36discounts there and his number is still
8:38double my number as an end consumer.
8:40Entirely wrong. The only thing here that
8:42is right is that they are probably
8:44getting some insane deal from Anthropic
8:46because remember, Cursor is now owned by
8:48SpaceX AI and SpaceX AI is giving a
8:51meaningful amount of compute to
8:53Anthropic, renting it because Anthropic
8:54needs GPUs more than XAI does. And on
8:56top of that, the vast majority of
8:58Anthropic's revenue came from Cursor for
9:01many years. So they almost certainly
9:03have a crazy deal here. If I was to
9:06guess what the effective cost, so to
9:09speak, that they are spending on the
9:12stuff that Lauren is doing is, again,
9:15pretending that the price is across all
9:18the tokens and that agreeing that way
9:19makes any sense at all, which it doesn't
9:21for a million reasons, but assuming it
9:23does, I would guess at the high end, her
9:27price per million tokens with all of
9:29this combined would be closer to like 20
9:30cents. So again, look at his numbers, he
9:33said 80 cents is 100k a month. So my
9:36guess at 20 cents, which again, like
9:38this is not the right way to measure it,
9:39but given the [ __ ] show we're in here,
9:41that would knock that 100K a month down
9:44to 25K a month. And here is where I have
9:46an honest question for you guys. I don't
9:48know how much you all know about
9:48marketing spend. In fact, I actually do.
9:50It's not that much. I know cuz I talked
9:52to a lot of engineers about marketing.
9:53They don't get it. How much money do you
9:55think most companies would be willing to
9:56spend to get the type of attention that
9:59Lauren is getting on Twitter and in the
10:01dev AI world right now? I'll give you a
10:03hint. It's a lot more than $25,000 a
10:06month. I know companies that throw
10:08events that get less attention than one
10:11random tweet from Lauren. And some of
10:13those events cost over a million
10:14dollars. And they get less from it than
10:16a couple tweets from Lauren. And that's
10:18not even counting the fact that all of
10:20the tokens she is burning are actually
10:22improving the products. Both Cursor,
10:24which now finally is actually
10:26performant. When Lauren joined Cursor, I
10:28was very hopeful because Cursor's app
10:30was falling apart. It was super slow. It
10:32was buggy as hell. And Lauren joining
10:34cleaned up a bunch of the bugs and made
10:35it way, way faster. And more importantly
10:38is Grokbot, which is quickly becoming
10:40the coolest product out of SpaceX AI.
10:42She's filing thousands of PR's a month,
10:45landing endless changes across all of
10:47these products, fixing things
10:48constantly. And it's only probably
10:50costing 25 to like worst case 50 grand a
10:52month based on reasonable estimates of
10:55what her token spend would be.
10:57But fun facts that we'll add on top of
10:59here, not only are they getting a
11:01discount for the tokens they're getting
11:02from somebody like Anthropic, she's also
11:04doing very, very deep long agentic runs.
11:08So, she's probably hitting cash even
11:10more than I am. All of this combined, I
11:12wouldn't be surprised if her token spend
11:15was closer to 10 cents per mill based on
11:18this awful aggregate number that doesn't
11:20tell us anything. Which would again,
11:22pricing based on this, put her at closer
11:24to like 15K a month at worst. And now I
11:28want to talk a bit about my response
11:29here because I
11:31I have things to say.
11:33The $132 mil a year that he started with
11:35is just it's not real at all. Even
11:37assuming full prices today, worst case
11:40is closer to 3 mil. I don't even think
11:42that is accurate. I think it's a lot
11:44closer to like 1 mil at most, probably
11:47like 800k or so. But with the trend of
11:50how inference is getting cheaper, I
11:53think this could get as low as $1,000 a
11:55year within the year. Hear me out.
11:58One last artificial analysis flashbang
12:00warning because we need to go to the
12:02trends, specifically the efficiency
12:05trends.
12:06This chart shows at a given level of
12:08intelligence, how does cost go down over
12:10time? As you can imagine, smart models
12:13coming out tend to come out at a pretty
12:14high price. They're measuring the
12:16intelligence index task cost. So, this
12:18is how much money did it take to
12:20complete the average task in artificial
12:22analysis. When a model like Sonnet 4.5
12:26came out, it was the frontier. It was
12:28the first model to break a 20 on the
12:30intelligence index, and it cost 56 cents
12:32per task. That was in September of 2025.
12:35Now, in September of 2026, there are
12:37models that get that same score, which I
12:40can't see cuz of this line being broken.
12:42God, this is why I stopped using
12:43artificial analysis for things. Yeah. 6
12:46Luna low gets the same score as Sonnet
12:504.5 did about a year later for 0.45
12:54cents per task. It went from 56 cents
12:57per task to 0.45. That is a 100x
13:01decrease over a year. Or we could look
13:03at the models in the 30 range, where
13:06Sonnet 4.6 Max broke it for the first
13:08time at $2.50.
13:10And now, under a year later, 6 Luna high
13:13can do it for 3 cents. $2.50
13:17to 3 cents. That is nearly a 100x
13:19decrease in approximately 8 months. Do
13:23you see where we're going here? Not only
13:25is intelligence getting cheaper,
13:27it's getting cheaper faster.
13:29It used to take a year for a 10x
13:31decrease. Now it takes 6 to 8 months for
13:34a 100x decrease. And this is where I
13:36need to talk about what Zeig said.
13:39This is exact quote him responding to
13:41Peter. This is what the folks at the
13:42labs often don't understand. If you pay
13:45for your own tokens or you have a
13:46business that has to, then you'll
13:48finally understand why the things that
13:50you say are so silly. No.
13:53Just now. For those who haven't been
13:55paying close attention to recent videos,
13:57this is the same guy who posted saying
13:59you should probably just keep using
14:00models like GPT 5.6 Soul or Sonnet or
14:05Opus 5.0 because the cost savings were
14:08worth it and the big fancy models are
14:10just never worth it. That I had to do a
14:11whole video about why you should use
14:12smart models on. This is his new take
14:14that pisses me off. This is the one that
14:16made me crash out and go do all the
14:18numbers. My response to him was that the
14:20first computers cost millions of
14:22dollars. And I personally find it quite
14:24cool that folks like Lauren are pushing
14:25the limits while ignoring cost. On
14:28average, the quote cost for a given
14:29level of intelligence drops by around
14:3110x year over year. It's actually
14:34accelerated recently going as much as
14:364.5x decreases per month. As I mentioned
14:39before for the AI numbers, scores above
14:42a 10 used to cost 26 cents in January
14:442025. Now they're 1.5 cents. Or even
14:47things that are in the 50s costing $6
14:51when they dropped originally, just 3
14:53months later you're getting the
14:54equivalent for 20x cheaper
14:57in 2 and 1/2 months. Do you understand
15:01how quickly things are accelerating?
15:02They're not just accelerating with how
15:03much faster they get smarter, they're
15:05accelerating at how quickly things get
15:07cheaper at a given level of
15:08intelligence. That's only true due to
15:10benchmarks though and you're not
15:11actually getting net cheaper
15:13intelligence everywhere. Yeah, because
15:15we like to use the smarter thing. Yeah,
15:18like you can't complain iPhones aren't
15:20getting cheaper when you refuse to buy
15:22last year's iPhone. Things are getting
15:24cheaper. We just use it more. I was just
15:26so far as I like this you aren't
15:28actually getting net cheaper
15:29intelligence. You you are. You
15:32absolutely are. Chat's pointing out that
15:34it feels like you didn't even read what
15:35I said. Fair and more fair the other way
15:37because when he slopped back to me with
15:40this essay, I I was done. I just I
15:42couldn't do it anymore. The example I
15:44gave in the reply is that the smartest
15:45model this time last year was GPT-5 High
15:48and this was like agreed upon
15:49universally. Nobody thought there was a
15:50model smarter than 5 High. They might
15:52not have thought it was as smart as I
15:53did, but everyone agreed 5 High was the
15:55[snorts] best model at the time. GLM-53
15:57Flash and DeepSeek-41 Flash match or
16:01beat it in every benchmark, but more
16:03importantly here, I prefer it.
16:06I would never choose GPT-5 or O1 or any
16:09model released last year
16:11over GLM-53 Flash because 53 Flash is
16:14genuinely really pleasant to work with.
16:17It's good at long-running agentic tasks.
16:19You ask it to do a thing and it does it.
16:21You ask it to do another thing and it
16:22does it. Last year's models weren't just
16:24dumber. They were worse at working with
16:26you. So you wanted to use them less and
16:28when you did, you would use them less.
16:30So not only do I disagree with this
16:32premise that like things aren't getting
16:34cheaper. It's In particular, he talks in
16:36this post about how
16:39Where is it? Old models get retired so
16:41there's no true price drop regularly.
16:44What What? What? Are you expecting them
16:47to keep hosting the super inefficient
16:49version of the model when people have
16:51made more efficient better models for
16:52way cheaper? But like the cost for
16:54hosting doesn't inherently go down when
16:56they leave the model out. Like it
16:57wouldn't make sense for them to keep
16:58hosting. Like what what? I'm just this
17:01particular sentence.
17:04I'm sorry I'm crashing out so hard. It
17:06just really pisses me off when
17:07billionaire tech bros that I'm friends
17:09with like David
17:11are this egregiously wrong about [ __ ]
17:13Like in the here's the worst part. Most
17:15of what he says isn't factually wrong.
17:17None of what he says is relevant at all.
17:19I'm thankful he didn't watch my last
17:21video about his [ __ ] because he
17:22would not have interacted with me as
17:24politely as he did here.
17:26The thing I'm most upset about though is
17:28that he got close to my likes. So the
17:30reason I'm crashing out here isn't so
17:32much that David is saying really dumb
17:34things. It's that people want to listen
17:37and believe him. People are hearing the
17:39dumb things he is saying and being like,
17:40"Yeah, that sounds right to me."
17:42It's not. It's just wrong. And if you're
17:44one of those 17 people, you're either
17:46just misinformed, which is fine. No
17:48offense, that's why you're here. And if
17:50you are liking tweets like this and
17:51watching my videos, you clearly are
17:54trying to figure this out and I respect
17:56you for it. But pretty much everybody
17:57else who hit that button did it because
17:59they don't want to recognize where we're
18:02going. They want to say AI is just going
18:04to stay super expensive forever. It
18:05won't make sense. We need to use cheaper
18:07models. We need to write code ourselves.
18:08We need to do all these things. And
18:10they're liking things that are wrong cuz
18:11they make them feel better. But things
18:13are about to get worse as we go. As I
18:15mentioned, not only are 41 flash and 53
18:18flash from DeepSeek and GLM respectively
18:21way more pleasant to use, they're also
18:25comically cheaper. I ran these numbers
18:28because Prime was complaining that
18:30models weren't getting cheaper. So I ran
18:32the numbers cuz I wanted to figure out
18:34how wrong that was. So I compared the
18:36raw token price for GPT-5 versus V4
18:40flash, not even 41 flash yet. Before
18:43flash and eventually GLM-53. Input
18:45tokens were 21x cheaper. Output tokens
18:47were 55x cheaper. Even if you presume
18:50there's a three times increase in the
18:51number of tokens, the difference nets
18:52out to 20x cheaper. I then point out
18:55that GLM-53 flash is more pleasant to
18:57use and is 29 times cheaper than GPT-5
18:59in bench runs. You're getting better
19:01intelligence for a similar cost, not
19:03cheaper intelligence overall generally
19:05speaking.
19:07The old rates go away cuz there's no
19:08infra to run those old models or they
19:10don't want to use it because it doesn't
19:12cover any costs. And those 10x better
19:14intelligence numbers are just
19:15benchmarks, which we all know aren't
19:17reliable. I will agree that benchmarks
19:19aren't the most reliable thing with
19:20modern models, but it's not for the
19:22reason he's implying here. The reason
19:24benchmarks aren't reliable anymore is
19:26cuz they're all either saturated or
19:27stupid. Do you know why we like
19:29benchmarks so much though? We like them
19:32because last year models weren't smart
19:33enough to saturate them, so they were
19:35very useful. The fact that cheap models
19:38today can saturate the benches that
19:40models struggled on last year does not
19:42mean the bench is bad. It means as my
19:44chat is pointing out that what we need
19:46is better and newer benchmarks. Then he
19:48says a 1% gain might be a quote 10x
19:50improvement.
19:53Thank you, S John. I was trying to be
19:55charitable to him before, but arguing
19:56more intelligence for the same cost
19:58doesn't imply the same intelligence is
19:59cheaper is genuinely crazy. Yep. He's
20:02contradicting himself. Yes, he is. But
20:03this is the thing and what you have to
20:04realize with the anti-AI dev doomer,
20:07contradictions don't matter to them.
20:09Facts don't matter to them. Numbers
20:11don't matter to them. Usage doesn't
20:12matter to them. Progress doesn't matter
20:14to them. Words that align with their
20:16beliefs is the only thing that matters
20:18to them. So if somebody writes an essay,
20:19even if they used AI to write a slop
20:21essay,
20:22if the headline is something that makes
20:24them feel better,
20:25they will gladly share it with the
20:27world.
20:28I'm not saying it's not getting cheaper,
20:30but folks who quote those numbers do so
20:33without considering the practical rate
20:34of improvement and cost of applied AI.
20:37If I had a problem that a model could
20:38have solved a year ago for 10 cents,
20:40it's probably still costing me close
20:42enough to that. That's what I mean by
20:44quote, it's not actually getting cheaper
20:45in a lot of scenarios. It certainly
20:47hasn't gotten like 30x cheaper for that
20:49kind of space. And then there's problems
20:51where models still struggle to solve,
20:52and those haven't gotten 30 times more
20:53reliable. They've gotten better at it,
20:55but the numbers don't actually scale to
20:56outcomes. This might be the worst part.
20:59I tried something fun recently. I
21:01plugged GPT-4o into Open Code to see if
21:05it could update a page. It couldn't
21:07figure out how to call tools. It spit
21:09out broken HTML in the chat log and then
21:12stopped responding cuz it didn't know
21:14what to do with the history. I think
21:16every model released this year is
21:18minimum 30 times more likely to solve a
21:21task in a harness than a model like 408.
21:25I honestly think people like this need
21:27to be forced to use a model from a year
21:28ago to remember how bad things were.
21:31GPT-6 Luna is 100 times cheaper than
21:33GPT-5 and it's legitimately smarter and
21:34better in every way I can think of. 61
21:37Soul is 10 times smarter than GPT-5 and
21:39it's still cheaper than five was. If I
21:41had a problem that could be solved a
21:43year ago for 10 cents it's still the
21:44same, I just outright said this is
21:45false. The only way it would be the same
21:48cost is if you let the modern equivalent
21:50model do more work than it did before,
21:53probably work you did by hand before,
21:54like running the dev server and checking
21:56it in browser. I'll self-own a bit here.
21:58I remember making fun of WinSurf when
22:00they added the ability for the model to
22:02run commands. The way I referred to it
22:04is like, "Oh, this is baby beginner mode
22:06for people who don't know how to run a
22:06dev command. Why would I ever want this?
22:09I can tell the agent which file to edit
22:11and I would do it with command K in the
22:12file." I just didn't get why you would
22:14want to ask a model to run a command.
22:16You can just run the command. I was
22:18super wrong on that. I did not see the
22:19future where models would use tools and
22:22bash to do their work. It seems like
22:24he's forgotten about that time even
22:26though that time was not long ago. He
22:28challenged me to go find a task from
22:30three years ago that a model would nail
22:32in accuracy. I'm going to hop back to
22:34chat. Can y'all name a task that models
22:37from three years ago could nail in
22:38accuracy? Ugly to-do apps? Three years
22:40ago? No. An AI model's not making a
22:43to-do app from scratch.
22:44We are so used to what AI could do
22:46today, we don't know how to think back
22:48to what AI did three years ago. Three
22:50years ago it couldn't make a to-do app.
22:52You could set up an environment where it
22:54could build, give it the main file and
22:57tell it to make changes, but you
22:58couldn't tell it to go build the ugly
23:00to-do app. You had to set everything up
23:02for it and then let it edit a file. What
23:05else do we have in here? There was no
23:06GPT-2 to-do app now. I like that someone
23:09figured out a good example here. Skate
23:11Bench. Skate Bench, funny enough,
23:13isn't like the best example in the world
23:15here. I'm not going to pretend
23:16otherwise. Like, Skate Bench is letting
23:19models name skate tricks and its ups and
23:21downs, but I ended up running numbers
23:24for this because of who I am as a
23:25person. I ran Skate Bench on O1. If you
23:29don't remember,
23:31O1 was the first ever reasoning model.
23:33It was when OpenAI figured out that
23:36letting the model spin in circles before
23:39making its response could increase the
23:40quality of its answers. This model came
23:42out in December of 2024.
23:45O1 cost $15 per million tokens in and 60
23:49per million out. That makes it more
23:51expensive than Fable. Even better is the
23:54cash read price was only half off. It
23:56was 750 per million reads. Now, the norm
23:59is 90 95% off, not 50% off.
24:02So, this would have been $1.50 modern
24:05times worst case. And don't get me
24:07started on O1 Pro, which is $150 per mil
24:09and 600 per mil out. But, I I could have
24:12destroyed him by doing that, but I chose
24:13not to. I used normal O1. And here is
24:16what it cost to do Skate Bench.
24:19It got a score of 67.7%
24:22and it cost 12 cents per task. 1 year
24:25later, almost to the day. Actually, it
24:27is today. It was December 17th, 2024
24:29where O1 came out and December 17th,
24:302025 when Gemini 3 Flash came out.
24:33Higher score, not a tenth the price,
24:35almost a hundredth the price. 58x lower
24:39costs exactly 1 year later. This is
24:42where I realized I was wrong. I thought
24:44the number was a pretty consistent 10x.
24:46It's not. It's closer to a hundredx
24:49year-over-year decrease at a given level
24:51of intelligence. And I told my agents to
24:53keep going. What is the cheapest that
24:55gets close? GLM 5 3 Flash, crazy to have
24:59that come back, got a slightly lower
25:00score, 70.8% versus Actually, no, it's
25:03higher. 01 was 67.7. So, 53-Flash got a
25:08higher score at 70.8 at $0.016.
25:13That is 0.00016
25:17per question. 680 times cheaper in under
25:202 years. How the [ __ ] is intelligence
25:22not getting cheaper? It's not just
25:24getting cheaper, it's getting cheaper
25:25faster. Anyone who thinks otherwise is
25:27stupid and wrong. Like, I'm sorry. The
25:29numbers speak for themselves, and I was
25:31wrong about this, too. I thought we
25:32would hit a ceiling for how smart the
25:34models got and a floor for how cheap
25:36they could be. I couldn't have been more
25:37wrong. We're nearing the point where the
25:39computer that you run your agents on,
25:42not the one where the inference happens
25:43with the models, but the one where it's
25:45calling bash, the VM that runs on is
25:47going to be more expensive than the
25:48inference at this rate. Back to what Zig
25:50said here. So, yeah, intelligence is
25:53getting cheaper, but in practice, that
25:54doesn't mean tasks are getting cheaper
25:56at the same scale because the
25:57measurement in those benches is not
25:59related to actual outcomes. Yeah,
26:01because the difference is if the task
26:03was fix this bug, previously, you would
26:07try out the app, find the bug, go to the
26:10file in the code base, open it, select
26:13where the bug was, press command K, and
26:15tell Sonnet 3.5 to edit the code, save
26:19it yourself, command S yourself, rerun
26:21the dev server,
26:23open it up in the browser, and see if it
26:25worked. The same task now is me handing
26:29a screenshot of the bug to the model and
26:30saying fix. And yeah, in that case, the
26:32task didn't get cheaper, but I'm not
26:34doing the [ __ ] work anymore. Sure,
26:37closing that ticket on Jira costs the
26:39same amount, but I didn't do the work
26:41this time. If I was to keep using Cursor
26:44the old-fashioned way, where you select
26:46text and then tell it what to do to it,
26:48I bet it would have been way cheaper.
26:50But since we live in the real world,
26:52where applications are trying to free us
26:54up of our engineering effort as humans,
26:56no one builds that way anymore. Here is
26:59where I started to crash out enough that
27:00I realized I had to make a video. He
27:02gave the number of 3 years ago, which I
27:04thought was insane. Like nobody should
27:05compare to things 3 years ago. So I did
27:062 years instead. 3 years ago, there were
27:08no reasoning models. Tool calls didn't
27:10exist yet. It made no sense to do this
27:12comparison at that time scale, because
27:14the tasks models could do were crappy
27:16auto complete. O1 came out in December
27:18of 2024, gave the prices. I genuinely
27:21struggled to find real-world tasks that
27:22O1 could complete. I chose to make it
27:24name skateboard tricks. The cost was
27:26around 11 cents per question. 3 flash
27:28came out in December of last year. It
27:30got a higher score and it cost 0.2 cents
27:33per question. IDK why you're fighting
27:34this, man. It's so obviously true. Try
27:36using O1 for any real code work, then
27:38switch to GLM 53 flash, and tell me
27:40honestly which you prefer to use. It's
27:42going to be the one that is more than
27:4410x cheaper. And this is the part that's
27:46hardest is David was very careful to
27:48frame things in ways where he is not
27:50angry or upset in his verbage, while
27:53also triggering me really badly to be
27:56very angry in my verbage.
27:58It sucks because this is dumbass versus
28:00[ __ ] and they look equal, because
28:02one person's being a dick and the other
28:04person is being nice, but is entirely
28:06wrong. And I just I I need you guys to
28:08understand this isn't one of those like
28:10right to your opinion things. This isn't
28:12two different takes on the same thing.
28:14This is somebody who spends way too much
28:16time and energy. Like make fun of me all
28:18you want for being too into AI. I know
28:20my [ __ ] with this now. I do a lot of
28:22work with these models. I have built
28:23tons of benchmarks. I am evaluating this
28:25stuff obsessively. I have a terabyte of
28:27my own history with models in a
28:29clickhouse database that I set up on my
28:31network, so I can audit all of this. And
28:34I am thankful you guys are
28:36reflecting on this thing. I'm being too
28:38nice if anything.
28:39I appreciate you guys, cuz I feel I felt
28:42so insane when I was going through this
28:43this afternoon, which is It's I closed
28:45on IDK why you're fighting me on this.
28:47It's obviously true. If any real code
28:49work was done with these, you would
28:50understand." To which she responds,
28:53"Okay, then. We can all agree that
28:54suggesting intelligence is getting 10x
28:55cheaper a year is a silly statement to
28:57make. You can run the same exercise
28:59aligned to the benchmarks, by the way,
29:01and you won't find anywhere near the
29:02same pricing delta in applied use. Sorry
29:04to push back so much on this. I know
29:06you're smart and I respect the hell out
29:07of you, which is why it's so genuinely
29:08confusing that you have this take. A
29:10year ago, the flow was highlight text in
29:12editor, command K, read the code by
29:14hand. Now, the flow is "Hey Claude, I
29:17want to do X. Figure it out." I asked
29:18him to give me a set of real tasks that
29:20he used AI for 12 months ago, and I'll
29:21turn it into a benchmark to show it is
29:22actually 10 times cheaper. He responded
29:24with more essays. Yeah, I'm I'm annoyed.
29:28I My issue isn't that like some people
29:30are dumb about this. My issue is that
29:32these dumb takes have been going viral
29:34so often because people want to believe
29:36this. They want to ignore AI and say
29:39this is just the labs wasting tokens
29:41because they can, and they're trying to
29:42trick us all into wasting a bunch of
29:43money. No. These things are getting
29:45cheaper to the point where the labs are
29:46losing their margins fast. If you hate
29:48OpenAI and Anthropic, you should be
29:51listening to me here. You should be
29:52joining me here. You should be shouting
29:54louder than me about how much cheaper
29:55inference is getting and how Anthropic
29:57and OpenAI will have no margin as
29:59everything gets driven to the floor.
30:01Here's where I get to my final points on
30:02this.
30:03With my version of the numbers, I priced
30:05her usage at around 100K a month. With
30:08just a few months of further
30:10improvements, if she wasn't to reach for
30:12smarter models and instead used
30:13equivalent models that level of
30:15intelligence but cheaper, she could get
30:17her costs down to as low as $3,000 in
30:19just a few months. And if the trends
30:21continue, as low as $100 a month by this
30:23time next year. But we're not going to
30:25do that because we're chasing the
30:26dragon. We want to use the smartest. We
30:28want to improve and change our
30:29workflows. We want to solve tasks
30:31differently in ways that involve us less
30:33or perhaps solve harder tasks that
30:35models couldn't have solved before. But
30:37here is where my final point is. No one
30:40said this is normal. I've seen a lot of
30:42people claim that folks like Lauren are
30:44trying to normalize the idea of spending
30:47millions of dollars a month in tokens.
30:49No, they're not. Lauren is not trying to
30:52seem like a normal person. She is very
30:55strange. She is a weird one-of-one
30:58individual. It's why I've always found
31:00her so exciting and fascinating. She's
31:02just She's a phenomenal engineer. And
31:04I'm thankful nobody's trying to go after
31:06her engineering capability right now cuz
31:08I would crash out so hard I'd be
31:09canceled for life. She's one of the best
31:10engineers I know. Way before this AI
31:12stuff, too.
31:13Turns out she's also really good at the
31:14agentic engineering stuff. I said she's
31:16in the .0001 top percent of agentic
31:19engineers. Measuring her work as dollars
31:21in software out is reductive and wrong.
31:23Her spend should be looked at as a
31:25combination of product and eng,
31:26research, FTE work with P stack,
31:29marketing, and like product research and
31:32improvement. Is this worth $350,000 a
31:34day? Probably not.
31:36Is it worth a mill or two a month?
31:37Without question. It's not costing that
31:39much, but even if she legitimately
31:41wanted to spend two to three mill a
31:42month, if I was the CEO of Cursor, I
31:45wouldn't even think twice. I would ask
31:46if she wanted more. She's exploring what
31:48happens when you ignore the cost of
31:49intelligence. She's not doing it cuz
31:51Cursor expects every company to spend 10
31:53mill a year on tokens. She's doing it
31:55because that 10 mill on tokens is going
31:56to cost $10,000 in a year or two. And of
31:58course, Lauren popped in. She strives to
32:01be a power user of Grokbot and Cursor so
32:03she can give good feedback and make
32:05changes and make the experience better
32:06for users. And she does this. I have
32:08watched this happen live in our shared
32:10slacks and DMs. I think Lauren is a rare
32:14gem in a space full of people who
32:17admittedly don't know what they're
32:18doing. When there are so many like
32:20agentic engineers that are just throwing
32:22tokens at things they don't understand,
32:24I understand the desire to go after
32:26these people.
32:27Lauren's not one of them. If you want to
32:29attack somebody for how they're wasting
32:30tokens, go after me. I'm a much easier
32:33target. I'm throwing away 500k in tokens
32:36on a rewrite of TypeScript just for fun.
32:39Lauren is actually pushing the frontier.
32:41Steer your frustrations towards people
32:42who deserve it more, like me, and let
32:45Lauren invent the future, because that's
32:47what she's here to do. And please, for
32:49the love of God,
32:50when you see people sharing numbers that
32:52are obviously wrong, or maybe just seem
32:55like they might not be right, maybe just
32:58screenshot the tweet and throw it at
33:00Claude or Codex and ask them to verify
33:03the numbers, because even the dumbest AI
33:06models nowadays could probably explain
33:08to you why none of the numbers being
33:10shared here make any sense at all. The
33:12lesson I really want y'all to take from
33:14this is how to notice people sharing
33:16these types of like fear-mongering
33:18things, where they care more about
33:20drumming up emotions than sharing
33:23information. And also, what one last
33:26thing to notice with this, watch for
33:28when people go after someone's
33:29motivations instead of trying to
33:31understand what they're saying. This is
33:33the thing I do my best to not do. I care
33:36about the information, and when I am
33:38mean to somebody, it's not because I
33:40think that they are lying to someone
33:41paid them to or something like that. I
33:44say that what they said is dumb.
33:45The folks I'm warning you about here are
33:48going to do the opposite. They will
33:50never engage with the points. They're
33:51going to steer around them and say
33:53something else, like, "Well, yeah,
33:54you're paid by the labs." Or, "Well,
33:56yeah, they're doing this for marketing."
33:58Or, "They're trying to normalize this
34:00thing." Or, "They are just using the
34:02benchmarks to make the numbers look
34:03better." They will say anything other
34:05than, "You're right." Those people are
34:07not good sources of information, and I
34:10hate to draw these lines, but it's now
34:12at the point where I have to call out a
34:14good friend with David Cramer, and say
34:16you should not listen to a thing he says
34:17about AI, even though I did a panel with
34:19him 2 weeks ago about AI, because I
34:21legitimately think the way he talks
34:22about these things are actively harmful
34:24to the space that we are in. If we
34:27cannot talk about the facts, then we
34:29cannot have a conversation. And sadly,
34:32this level of egregiousness around the
34:34facts has degraded the conversation to
34:37[ __ ] and people like Zeig/Kramer the
34:39founder of Century who has been a
34:41sponsor of mine for a while and Zeig's a
34:43good friend of mine and even Peter who
34:46I've had my back and forth with and like
34:47we did not get along initially but we
34:49have fully come around and are decently
34:50friendly now. I hate calling out my
34:52friends like this
34:53but I'm calling out my friends because
34:55it would feel a lot worse calling out
34:57the dumber people doing this and I know
34:59that both Zeig and Peter are smart
35:02enough to listen and stop doing this
35:04[ __ ] going forward. I hate drama videos
35:06like this I really do and know that I'm
35:08carefully picking my targets as people
35:10who I know and trust and am friends with
35:12because I think they might actually fix
35:14their [ __ ] but until they do until
35:16they publicly acknowledge how wrong they
35:18were similar to how I have been doing
35:20forever now. How many times have I made
35:21the meme that I was so wrong about the
35:23intelligence ceiling or that I had no
35:26idea how powerful Wind Surf's idea of
35:28letting the agent run commands would be
35:30or that I thought intelligence would
35:31stop getting cheaper and now it's
35:32getting more cheaper more faster or that
35:34I didn't think Anthropic would ever be
35:35able to do RL and now they have the best
35:37RL models in the world. I was wrong a
35:39lot and I will gladly publicly shout
35:41about this because what I care about is
35:43getting the right information to y'all
35:45and I am sure the comment section is
35:46going to reflect exactly that.
35:49Oh man I am scared to see this one when
35:50it comes out but uh
35:52I've said what I have to hopefully I've
35:54helped you guys better understand these
35:55types of things and that you now have
35:57some things to recognize when people are
35:59being really really wrong on the
36:00internet. Uh
36:02I'm going to go do literally anything
36:03else with my time and until next time
36:05peace nerds.

About this transcript

This page contains the full transcript of the hard truth about token spend by Theo - t3․gg, generated from the public captions YouTube serves with the video. The transcript has 7,404 words across 1,040 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.