AI Just Got Cheaper. Here's Why That's a Problem for NVIDIA. — Transcript
Full transcript
- 0:00If you've moved work to a cheaper AI
- 0:01model this year, you're part of a
- 0:03problem Nvidia was already worried about
- 0:05back in August. The Wall Street Journal
- 0:07reports Jensen Huang told his finance
- 0:09team that chip wafers and memory Nvidia
- 0:12had committed to were coming at him like
- 0:14a quote avalanche. All of that needs
- 0:17paying customers, and the customers are
- 0:19getting smarter. You found out the
- 0:22cheaper model does most of the job, so
- 0:24you stop paying for the pricey one. I
- 0:26hear that over and over again. Demand
- 0:28for AI is very very real. It's
- 0:30absolutely enormous. What I want to know
- 0:33is whether it is growing at the like 10
- 0:35x or more pace that pays the bill for
- 0:38the capacity build out. Ramp tracks AI
- 0:40spending by businesses. It reported
- 0:42virtually no growth in token spending
- 0:44across August and September. Two nearly
- 0:47flat months followed by a rise of less
- 0:49than 2% is still extremely slow growth.
- 0:53You're nowhere near the 10-fold increase
- 0:55in spending that you want to see to
- 0:58sustain the kind of build out that the
- 1:01labs are talking about, at least not at
- 1:02that pace. Now, I will call out as
- 1:04someone who's watched the ramp spending
- 1:06numbers that they do bounce around a
- 1:09lot. There are some months when you see
- 1:1030 40% growth in a month and other
- 1:13months where it's completely flat. What
- 1:15makes this even more interesting is that
- 1:16if you dig into the cross tabs of ramp
- 1:18ramp's most recent report, the most
- 1:20expensive models actually gained
- 1:22spending share in October. So taking a
- 1:25larger share is good for the suppliers
- 1:28of those models, but it doesn't mean the
- 1:30overall market is actually growing
- 1:32quickly enough to sustain the capacity
- 1:34build out. And that's the problem I am
- 1:36interested in. That slow spending growth
- 1:38fits a lot of the conversations I'm
- 1:39having these days. Companies are
- 1:41figuring out that they can move work to
- 1:43a cheaper model over and over again.
- 1:45They test whether it does the job and as
- 1:47models get better, it often does and
- 1:49they use it. They don't need the most
- 1:51intelligent model for every step of the
- 1:52process and they know they don't need to
- 1:54pay for it. And that matches what
- 1:56businesses have told ramp. They're
- 1:57setting defaults that steer employees
- 1:59toward less models because those models
- 2:02do the work well enough. Here's a really
- 2:03simple example. Imagine, you know, a
- 2:06customer moving routine work to a
- 2:08cheaper model from the same lab. The lab
- 2:11is going to keep the customer as a
- 2:12paying customer. Token usage is going to
- 2:14go way, way up, but the bill either
- 2:17doesn't rise or doesn't rise nearly as
- 2:20fast. We've covered Anthropic's
- 2:22extraordinary growth here. We've covered
- 2:24the enormous computing commitments that
- 2:26go with that. The question now is where
- 2:28the next tenfold increase in spending is
- 2:31going to come from if more customers
- 2:33learn how to trade down to cheaper
- 2:35models. Google has just given customers
- 2:37another way to do that in the last week.
- 2:39Carbon and hydrogen are two of Google's
- 2:42new Gemini models. Carbon handles
- 2:44difficult reasoning and hydrogen is
- 2:46built for faster execution. Inside
- 2:48Google's anti-gravity software, a
- 2:49feature called Orchestrator will
- 2:51auto-switch between those models during
- 2:53tasks. Now, I know that anti-gravity is
- 2:56not a super popular software with
- 2:58developers right now, but the point is
- 3:01that more and more of these kinds of
- 3:02models are available. Think of coding
- 3:05jobs. You might want a very capable
- 3:06model to work out why something is
- 3:08broken and maybe plan the repair, but
- 3:11once the decision is made, you're going
- 3:12to pass a bunch of the rest of the
- 3:14coding work to a cheaper, faster model.
- 3:16And this is what I see in my own usage,
- 3:18and this is what I see in the usage of
- 3:20everyone around me who is using AI
- 3:22tools, not just at startups, but at
- 3:25traditional companies as well.
- 3:27In Google's own testing, the
- 3:29Orchestrator tool got very close to
- 3:32carbon success rate while consuming 24%
- 3:35less quota and finishing 19% faster.
- 3:38Frankly, some of the numbers I've seen
- 3:40from folks who are using open source
- 3:42models are even better. They get to 80
- 3:4590% cheaper or more, and they can still
- 3:48finish faster at an equivalent result.
- 3:50So for people who are consuming AI
- 3:52tokens right now, this is a phenomenal
- 3:54story. It's a phenomenally useful
- 3:56result. More work is now fitting inside
- 3:58the same allowance, whether you're
- 3:59coding or whether you're doing other
- 4:01work. Less time is going into waiting,
- 4:03the expensive model gets used less, the
- 4:05bill either goes down or it's flat. This
- 4:09is a direction I see companies moving to
- 4:11overwhelmingly. They want AI that works
- 4:14and they're getting much more deliberate
- 4:16about how much they pay for it.
- 4:18And so they don't pay attention to the
- 4:20new intelligence scores. Those only
- 4:22matter if it lets them do something they
- 4:24couldn't do before.
- 4:25So for work they already know how to
- 4:27complete, cheaper execution is often a
- 4:30much much more attractive announcement.
- 4:32There's also a clue to all of this in
- 4:35RAM's most recent November report about
- 4:37where new spending is coming from. Among
- 4:40the AI spending RAM can identify by
- 4:42department, customer success and support
- 4:44appear to have grown from less than 1%
- 4:47in June to more than 13% of AI spend in
- 4:50October. AI is starting to reach outside
- 4:53of engineering into other kinds of work
- 4:56and a lot of that work just needs a fast
- 4:59economical answer. It does not need a
- 5:00frontier answer. If a customer wants to
- 5:03know where the shipment is, the company
- 5:06just needs the right order information
- 5:08and they need to retrieve a correct
- 5:09status. That's it. It's not a fancy
- 5:11query and paying for deeper reasoning
- 5:13just does not make sense on most of
- 5:15those requests. And you can grow that
- 5:18kind of a business very substantially
- 5:20while choosing a cheaper model for most
- 5:22of it without changing your overall bill
- 5:25footprint and that's exactly the kind of
- 5:26pattern that the data reflects. Now look
- 5:29at that from the other side of the
- 5:30transaction for a minute. The labs have
- 5:32ordered computing capacity years and
- 5:35years and years and years in advance.
- 5:37Their customers are learning how to buy
- 5:38intelligence more efficiently at the
- 5:40same time. The labs do need enough
- 5:43additional business to make up for lower
- 5:45prices and then they need to grow far
- 5:47beyond that to justify their valuations.
- 5:50That's why the new reporting about
- 5:52Jensen matters so much. The Wall Street
- 5:55Journal describes an August meeting
- 5:57about Nvidia's financial support for
- 5:59prospective customers. Jensen Huang was
- 6:01reportedly worried about commitments
- 6:04piling up and major cloud providers
- 6:07looking at alternative chips as a
- 6:09result. The avalanche remark that I
- 6:11quoted earlier in this video came in
- 6:13that conversations and concerned supply
- 6:16that Nvidia had already committed to
- 6:18purchase as part of its own build out
- 6:20for chips and capacity to sell to data
- 6:23centers. We already know Nvidia has been
- 6:25deeply involved in financing the overall
- 6:27AI build out. I don't think that's a
- 6:29surprise to anyone at this point. What
- 6:31this report adds is concern inside
- 6:34Nvidia about how those commitments were
- 6:36developing and how they might lead to
- 6:38exposure for the company long term. None
- 6:41of this is a clear evidence that the
- 6:44industry has overbuilt so far. Unlike
- 6:46.com, we still have skyrocketing token
- 6:49demand, but it does give us reason to at
- 6:51least examine whether the capacity being
- 6:54ordered matches the willingness of
- 6:57business to pay more over time. In this
- 7:00world, especially as token prices drop,
- 7:02demand can still grow astonishingly
- 7:05quickly and yet disappoint everyone who
- 7:08financed the equipment. Suppose a
- 7:10business uses twice or four times as
- 7:12much AI but pays a half or a quarter as
- 7:14much per unit. That's a world we live
- 7:16in.
- 7:17Spending hasn't changed overall. The
- 7:19provider may serve far more requests,
- 7:21but they're not collecting another
- 7:22dollar from the customer. Now, that's
- 7:24not the only way cost is impacting this
- 7:27story. Better hardware and more
- 7:29efficient models are lowering provider
- 7:30costs as well, and that's part of what
- 7:32makes the cheaper service possible at
- 7:34all. But whether a particular data
- 7:35center earns money back depends on how
- 7:37much it can charge, how much business it
- 7:39wins, and what it costs to build out in
- 7:41the first place. Those are all numbers
- 7:43that are one, proprietary, and two,
- 7:45different by data center. A cheaper
- 7:46model can still be a good business if
- 7:48it's inexpensive enough to run, and a
- 7:51busy data center can still be a very bad
- 7:53investment if the price customers pay
- 7:55isn't covering costs. Like that just
- 7:57makes sense. At this point, we really
- 7:58need to understand on a per data center
- 8:01basis where the cost of supplying the
- 8:04work and the revenue from selling it add
- 8:06up or don't add up. And unfortunately,
- 8:08because most of this is, of course, very
- 8:10proprietary, we're not in a position to
- 8:12know that.
- 8:13We don' know for sure what the margins
- 8:16are. We don't know for sure where the
- 8:18triggers and guarantees are in the
- 8:20system, and we don't know for sure
- 8:22whether the data centers that are being
- 8:24built out are covering their costs as
- 8:27they go. And so when we think about the
- 8:30future, it's rational to ask ourselves,
- 8:32are we in a position where the market is
- 8:35growing healthily?
- 8:37And when I look at it, I see two
- 8:39stories. I see a story of skyrocketing
- 8:43token demand, more and more businesses
- 8:45leaning into AI. That's all true. But I
- 8:48also see a story of dramatically falling
- 8:51token costs.
- 8:52And the thing that I am looking at
- 8:54really carefully is whether frontier
- 8:58labs are going to be able to sustain the
- 9:02kinds of revenue multiples they're
- 9:04looking to grow at over the next year if
- 9:07overall token spend remains flat.
- 9:11There's still a substantial market for
- 9:12the most capable models. I don't
- 9:15question that. In fact, we see that in
- 9:16RAM's data where the most expensive new
- 9:19frontier models make up roughly 70% of
- 9:21the spend. The unresolved question is
- 9:23how quickly that market grows relative
- 9:26to the commitments being made around it.
- 9:27I don't have the answer, and anyone who
- 9:29says they the answer doesn't. I think
- 9:31the combination of customers trading
- 9:33down and the reported concern about
- 9:34incoming supply makes it a question we
- 9:37should all pay attention to. And like it
- 9:39or not, RAM's data has become at least
- 9:41as important as the Fed in understanding
- 9:44one of the key drivers of our economy
- 9:46right now. We have to understand AI
- 9:49spend to understand where the AI economy
- 9:52is going and growing. And just to be
- 9:55clear on the macro thesis, I still think
- 9:57we are dramatically underallocated to AI
- 10:00overall.
- 10:01It's just we need to see clear ROI for
- 10:04businesses to allocate more. And until
- 10:06they see that cash register ring,
- 10:09sticking at what, 0.1, 0.2, 0.3, 0.4% of
- 10:12overall revenue is something that's
- 10:15going to be where most businesses top
- 10:16out. And so, as we head into the next
- 10:19NVIDIA earnings call, which is right
- 10:20around the corner, I would want to hear
- 10:22more than just the quarterly results,
- 10:24right? And I think investors are going
- 10:26to be asking, what about the trajectory
- 10:29of NVIDIA as a business? And implicitly,
- 10:32they're going to be asking, what about
- 10:33the trajectory of the AI build out
- 10:35overall? Is there a bigger order book on
- 10:37the horizon? Are we seeing secure growth
- 10:41in demand and secure growth in financing
- 10:43such that we feel comfortable with the
- 10:44build out? That's going to be a big
- 10:47point to watch in the next week. And
- 10:49it's going to be even more important to
- 10:51review kind of how that earnings call
- 10:53goes because there is now a bipartisan
- 10:55initiative supported by Anthropic and by
- 10:57OpenAI to further restrict chip exports
- 11:01to China. This is something that I think
- 11:03was likely inevitable after the Jensen
- 11:05scandal broke, but it's going to affect
- 11:08the way NVIDIA's chip sales actually
- 11:11work, particularly if they close down
- 11:13sales, say, through Singapore or other
- 11:15places where it's possible for Chinese
- 11:17firms to rent compute. There is an extra
- 11:20piece of news here that I don't think
- 11:22has gotten enough attention that is
- 11:24going to add fuel to the fire as far as
- 11:26export controls. Will DePue examine six
- 11:29released open source models from four
- 11:31different Chinese labs, including Deep
- 11:33Seat and including Moonshot. He found in
- 11:36his research numerical fingerprints
- 11:39consistent with NVIDIA's Blackwell GB200
- 11:42and GB300 chips, both of which are
- 11:45subject to export controls and should
- 11:47never have appeared in open source
- 11:50models in China. Now, you might wonder,
- 11:52how did he figure that out? Well, that's
- 11:54the interesting part. A model can
- 11:56contain billions and billions of
- 11:57numerical values called weights. We sort
- 11:59of know that that's how that works at
- 12:00this point. Before training, those
- 12:03values have to be initially set or
- 12:05initialized. The hardware helps
- 12:07determine how those starting numbers are
- 12:10arranged and some traces of that initial
- 12:13state survived training. So, the
- 12:16researchers worked backward from
- 12:18released models to identify hardware
- 12:21that could reproduce the patterns that
- 12:23they saw. So, in other words, the model
- 12:26itself can contain evidence of the
- 12:28equipment used to create it. You don't
- 12:31necessarily need a leaked invoice or
- 12:33someone inside the company to tell you
- 12:35to see which chips were used. The
- 12:38fingerprints support Blackwell access,
- 12:40but what they can't tell you is where
- 12:42were those chips at the time, how was
- 12:44access to those chips obtained, did the
- 12:46full training run take place on them?
- 12:48There are just a bunch of investigative
- 12:49questions that have yet to be answered.
- 12:51But what this changes is that
- 12:54investigators now have something
- 12:55technical that they can examine
- 12:57independently of what a company may say
- 12:59about its own hardware and the story of
- 13:01how a model came to be. And I suspect
- 13:04from a narrative perspective, this is
- 13:06going to add fuel to the fire for export
- 13:09control. And that's what we see with
- 13:10OpenAI's recent policy shift here.
- 13:12Bloomberg reports that OpenAI has told
- 13:14lawmakers it supports a package of
- 13:17tighter chip controls. Now, Anthropic
- 13:19had already backed those measures. Those
- 13:21measures have bipartisan support. And so
- 13:24when both labs support it and both
- 13:25parties support it, support is now
- 13:28growing for really closing down the
- 13:30routes through which restricted
- 13:31computing power can still reach Chinese
- 13:33developers. One proposal, the Chip
- 13:35Security Act, would require location
- 13:37verification for advanced chips covered
- 13:39by export controls. The idea is pretty
- 13:42straightforward. A shipment going to any
- 13:44approved destination should not be able
- 13:47to disappear from view and in any
- 13:49situation end up somewhere else. This
- 13:51follows the China security story I
- 13:53covered. Huang admitted an affair with a
- 13:55Mega Speed executive whom US officials
- 13:57believe worked for Chinese intelligence.
- 14:00And part of the allegation there is the
- 14:02allegation that those chips were moved
- 14:05other places. Now, China denies those
- 14:07allegations and of course, we have to
- 14:09see how all of that plays out. But it's
- 14:12in the political wind right now and it's
- 14:15part of the story that is driving
- 14:17bipartisan support for this bill. Now,
- 14:20there's one other story I want to cover
- 14:22because it affects anyone being asked to
- 14:24trust these companies with more of their
- 14:26work, which is effectively what the labs
- 14:28are asking companies to do. I'm talking
- 14:30about, of course, the Microsoft 365
- 14:33outage.
- 14:34[snorts] The Microsoft 365 outage on
- 14:36November 12th disrupted Outlook, Teams,
- 14:38and other services that businesses
- 14:40depend on. Microsoft now says an earlier
- 14:42network capacity upgrade introduced a
- 14:44problem that surfaced after a later
- 14:47network change. It says the incident was
- 14:49not related to cyber activity or any AI
- 14:52agents on the system, and I believe
- 14:54Microsoft. That's the technical
- 14:56explanation. But my larger concern is
- 14:59what Microsoft's engineering culture
- 15:01does to its ability to prevent a
- 15:03mistake, contain it, and recover from
- 15:05it. In Microsoft's developer division,
- 15:07some managers have been told that AI use
- 15:09should factor into employee evaluations.
- 15:11And to be clear, Microsoft is not alone
- 15:13on that. I've heard similar stories from
- 15:15Meta, similar stories from Amazon and
- 15:17other large companies. The company also
- 15:19offered retirement packages to very
- 15:22long-serving employees in the spring,
- 15:24and I think that's relevant as well.
- 15:26When you reward people
- 15:28for using AI and you encourage
- 15:30experienced people to leave, you need to
- 15:32be really sure that you're keeping the
- 15:34knowledge required to run extremely
- 15:36complex systems that you are now
- 15:38changing at a higher rate. Former Azure
- 15:40engineer Axel Rietschin has described
- 15:43years and years of quick fixes and lost
- 15:46senior expertise inside Microsoft's
- 15:48cloud business. His account is focused
- 15:50on Azure, which is a separate service.
- 15:53But I think the point here is that there
- 15:56is a larger question that many companies
- 15:58are grappling with about companies'
- 16:00treatment of engineering knowledge,
- 16:02particularly in the age of AI.
- 16:05And Gergely Orosz's recent reporting on
- 16:07companies that include Uber and Meta has
- 16:09echoed similar themes of AI use and code
- 16:11output without equivalent attention to
- 16:13whether the software involved actually
- 16:15gets better. Ultimately, regardless of
- 16:17the technical trigger, that is the risk
- 16:19I see in asking companies to
- 16:22dramatically increase AI usage without
- 16:25care. You can measure a lot of activity
- 16:28while missing what makes the service
- 16:29dependable, and you can walk your human
- 16:32talent out the door precisely at the
- 16:33moment that you need it the most. An
- 16:35experienced engineer is there for a
- 16:37reason. They know all of the skeletons
- 16:39in the closet. They remember the
- 16:41failures, and when something breaks,
- 16:43they know which thing they have to look
- 16:45at to fix. When I was at Amazon, there
- 16:48was literally a guy on a boat that you
- 16:50called whenever AWS went down, and he
- 16:53was able to come and fix it. So, this is
- 16:56something that we desperately need more
- 16:58of as we lean into AI. And none of this
- 17:00is saying that AI doesn't add value.
- 17:03I'm just trying to call out that if we
- 17:05are going to ask companies, which the
- 17:07labs are doing, to increase spend, we
- 17:10need to do so in a way that allows
- 17:12companies to improve the reliability and
- 17:14security of their software along the
- 17:16way. And that requires keeping
- 17:18engineering cultures intact. Amazon's
- 17:20own explanation of a March incident
- 17:22around AI illustrates why this is so
- 17:25fraught. An engineer at the time
- 17:27followed incorrect AI advice drawn from
- 17:29an outdated wiki. Now, Amazon called the
- 17:31cause user error. That's true. They
- 17:34claimed it was unrelated to AI, which I
- 17:36kind of have some questions about, and
- 17:38they said its controls let the mistake
- 17:40have too much impact. I would agree with
- 17:41that. But even in that account, the
- 17:44quality of the information and the
- 17:46checks around the engineer mattered
- 17:48enormously. It's really, really
- 17:51important to have strong engineering
- 17:53cultures that allow that kind of system
- 17:56to grow in reliability over time rather
- 17:59than to degrade. We are not going to get
- 18:02an official cultural explanation for a
- 18:04root cause for the outage. We'll get a
- 18:06technical one, but we're not going to
- 18:08know whether staffing or AI incentives
- 18:10contributed in any way to the Microsoft
- 18:13outage. But I think in a lot of ways,
- 18:15that is the most relevant question we
- 18:17can ask coming out of a global outage
- 18:19like this.
- 18:20An ordinary network error doesn't answer
- 18:23management questions. It doesn' answer
- 18:25questions around how the error came to
- 18:27be in the first place, around how the
- 18:29impacts was so large, around why
- 18:30recovery was so difficult and took such
- 18:33a long time.
- 18:35Every company at this scale is committed
- 18:38to security and quality. I do not doubt
- 18:40that Microsoft is committed to security
- 18:42and quality of their software.
- 18:44But my larger concern is that we have
- 18:47seen a lot of public statements, not
- 18:49just from Microsoft, but from other big
- 18:50companies around dramatic changes in
- 18:54engineering culture in order to enable
- 18:56AI. And I think that one of the things
- 18:58I'm seeing at successful engineering
- 19:00cultures in the age of AI is that they
- 19:02are not taking away deep developer
- 19:04experience. They're actually building on
- 19:06it so that they can use AI effectively
- 19:09to grow the reliability and quality of
- 19:12software in their systems. That's
- 19:14something that we need to think about
- 19:15more. And I think that is one of the
- 19:17things that stands in the way of the 10x
- 19:20growth in spend that the labs are
- 19:21looking for. We need to build up
- 19:23engineering cultures. Ultimately, I
- 19:25don't see some of these trends changing
- 19:27anytime soon. Customers are going to
- 19:28keep learning how to buy AI more
- 19:30cheaply. Suppliers have a tremendous
- 19:32amount of capacity coming online in the
- 19:34next year or so. Governments are
- 19:36deciding who can use some of it and who
- 19:38can't. That's the China export controls.
- 19:41And the companies building and operating
- 19:42these systems need to get enough revenue
- 19:44in the door to keep that build out
- 19:45going. Just underline this three times.
- 19:47I expect people to use a lot more AI in
- 19:492027. I think the question that I have
- 19:52is whether the spend growth is going to
- 19:55be as sky high as the labs need. I
- 19:58expect spend growth in 2027 too. I just
- 20:02wonder if that spend growth is going to
- 20:03go to model makers or if it's going to
- 20:05go farther up the stack to places where
- 20:09AI is going to actually help businesses
- 20:10ring the cash register. AI token prices
- 20:13are going to continue to fall. It will
- 20:14be a buyer's paradise. It is going to be
- 20:17up to us to build strong engineering
- 20:19cultures so that we can actually
- 20:20implement AI systems that offer real and
- 20:23sustained benefits. That is going to be
- 20:25the work of 2027. I think that's the
- 20:27lesson of the do's in the last week or
- 20:29so. I'll see you next time.
About this transcript
This page contains the full transcript of AI Just Got Cheaper. Here's Why That's a Problem for NVIDIA. by AI News & Strategy Daily | Nate B Jones, generated from the public captions YouTube serves with the video. The transcript has 3,693 words across 555 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.