YT YouTube2Text

YouTube2Text → Transcripts → Inside AI Tokenomics: How to Profitably Turn Tokens Into Business Value | NVIDIA AI Podcast Ep. 299

Inside AI Tokenomics: How to Profitably Turn Tokens Into Business Value | NVIDIA AI Podcast Ep. 299 — Transcript

by NVIDIA · 5,576 words · 881 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Not all tokens are created equal. And
0:02there is a way to look at token value.
0:05There are two key factors that impact
0:08token value. One is the intelligence
0:11[music] embedded in the token, or how
0:13much intelligence does the token carry,
0:15and the other is how fast does it
0:17arrive.
0:19>> [music]
0:22>> Welcome to the NVIDIA AI podcast. I'm
0:24Noah Kravitz. I'm here with Shruti
0:26Kulkarni. Shruti is a member of the
0:29accelerated computing team here at
0:31NVIDIA, and she focuses on inference.
0:33And we're here to talk about tokenomics.
0:36As data centers become AI factories and
0:39produce intelligence for the new
0:41industrial revolution, this word
0:44tokenomics has been fluttered about.
0:46It's a useful term, but maybe we can
0:48break it down with your help, Shruti, so
0:50that it's really something that business
0:51leaders can understand and take into
0:53practice.
0:54>> Yes, absolutely. Well, first of all,
0:56thanks a lot for having me, Noah. I am
0:58very excited to dig into the economics
1:00of AI, or tokenomics. And as you said,
1:04it is a term that gets used quite a bit,
1:07and I welcome the opportunity to help
1:08define it, so to speak. So, the way to
1:11think about tokenomics is it's about how
1:13tokens are valued, supplied, consumed,
1:17and monetized. And what that essentially
1:20maps to is token utility, which is all
1:23about token value, token supply, and
1:26this is where your AI infrastructure
1:28decisions are, right? Thinking about
1:30what infrastructure to invest in that
1:32will maximize your token output while
1:34minimizing cost. Then there's token
1:37demand. This is where customers and
1:39organizations think through what is
1:42their number of users, how many use
1:43cases, what types of use cases. So,
1:46really sort of mapping out the volume
1:48and velocity of the tokens that they
1:49need. And then finally, there's token
1:51monetization, which is taking the tokens
1:54and turning it into business value. So,
1:56those are sort of the four pillars for
1:59tokenomics, and it's super important to
2:02understand all four of those and how
2:04they relate to each other to be able to
2:07deploy AI successfully.
2:09>> So, let's start at the top then with uh
2:11utility or value.
2:13How do you define the value of a token?
2:15Are all the tokens worth the same? Do
2:17they have different values? Is there a
2:19better way to look at it? How do you
2:20approach that?
2:21>> That's a really great question, and
2:23you're right, actually, that not all
2:26tokens are created equal. And there is a
2:29way to look at token value. There are
2:31two key factors that impact token value.
2:34One is the intelligence embedded in the
2:37token, or how much intelligence does the
2:40token carry? And the other is how fast
2:42does it arrive, which is essentially the
2:44interactivity. So, to unpack that a
2:47little bit, the intelligence of the
2:49token is dependent on the model that
2:53produced the token. So, more complex,
2:56more intelligent models will produce
2:58tokens that, in general, have much more
3:01intelligence. Yeah. And then, it also
3:03depends on the context that the models
3:05looked at. And, generally speaking, the
3:08longer the context that you sort of
3:11allow the model to look at, the better
3:13the accuracy, the better the
3:14intelligence of the tokens. Now, I say
3:16generally because there is There are
3:18cases where um if the context increase
3:22too much, then the model
3:25quality the output quality can degrade,
3:28but I don't want to rabbit hole into
3:29that. Generally, more context does kind
3:32of equate to better intelligence.
3:34>> Right.
3:35>> Um so, that's one aspect. And then, as I
3:37mentioned, how fast the token arrives is
3:40the token interactivity, which is
3:43essentially tokens per second per user.
3:45So, it's the rate of token generation.
3:48And so, if you look at token value as a
3:51spectrum, on the one hand, you have
3:53these basic models with shorter contexts
3:56generating tokens at not that fast a
3:59speed. And then on the other extreme is
4:01these more complex, more intelligent
4:03models with much larger context and
4:06generating tokens that that are really
4:08fast. And across that entire spectrum is
4:12your different use cases and how you map
4:15those use cases to the to the token
4:18value.
4:19>> Right.
4:19>> So, of course, there is a absolute way
4:21in which to think about the token value,
4:24but there is also a relative way with
4:27respect to the use cases um in how to
4:30think about it. And and we can unpack
4:32that a little bit if you want.
4:34>> So, is it fair to say that the value of
4:36the token um is tied at all to the task
4:39that it's hand as well?
4:41>> Yes, and that's that's exactly sort of
4:43what I was trying to get at when I said
4:45that like you have to think about
4:47mapping the right use case to the token
4:49value.
4:50>> Right.
4:50>> So, as an example, uh we said like I
4:53said earlier, right, that that tokens
4:56generated by more complex intelligent
4:58models are more valuable.
5:00But that's in an absolute sense.
5:02Relatively speaking, your use case may
5:05not require that more complex, more
5:08intelligent model. And then that
5:10additional value is completely useless
5:12to you. Um one example of this is
5:14domain-specific applications where in a
5:17very narrow context, a post- trained,
5:20which is fine-tuned, small language
5:22model, so a much smaller model can give
5:25you just the value you need. Um in fact,
5:27in some cases even better accuracy for
5:29that given task. So, you don't need
5:32always the the big large models. And so,
5:34relatively speaking, you need to map
5:37where on the spectrum of token value
5:40does your use case sit. Um same thing is
5:42true for the interactivity piece, which
5:44is agentic applications absolutely need
5:47the highly interactive tokens. But you
5:49may have applications like chat
5:51interfaces or enterprise search, which
5:54don't need that level of interactivity.
5:56And so that is very critical when you
5:59are thinking through your sort of AI
6:00deployment decisions of where to map
6:04your use case to what token value.
6:06>> So when a business leader's thinking
6:08about demand and thinking about mapping
6:10out tokens to use cases and the
6:13different use cases have, you know,
6:15different values associated with them.
6:17How How What's a good approach for
6:20someone who's, you know, looking at what
6:23their what their org is doing, what
6:24their different team members need? How
6:26do they start to get a handle on, well,
6:28how many tokens are we going to need to
6:30produce and how many of each kind?
6:32>> Yeah. So use cases are extremely
6:35important when thinking through token
6:37demand. And
6:39there are three layers in which you can
6:41think about this
6:43with improving levels of forecasting
6:47accuracy, if you will. So the basic sort
6:50of, you know, back of napkin math is
6:53look at how many users you have,
6:55how many requests or sessions
6:58the user is going to initiate in a given
7:01day or month, and then how many tokens
7:04you need per request or per session. And
7:08those three numbers put together will
7:10give you your base demand for a single
7:12day or a month or, you know, whatever is
7:13your time period of analysis.
7:15Now, that is the base and very, very
7:18sort of simplistic look at it.
7:20There are multipliers that you do need
7:22to account for that will dramatically
7:25change your understanding of your token
7:28requirement. And a couple of those are,
7:31number one, are you using reasoning
7:33models? As we know, like reasoning
7:35models use thinking tokens, which never
7:38get seen by the end user. And often
7:40times when AI is deployed, you can
7:42actually set thresholds on how many
7:44thinking tokens are allowed per
7:46interaction. And so
7:48when you are estimating demand, you do
7:50need to think through are we using
7:51reasoning models? What are our
7:52thresholds? What do we expect the peak
7:54and average to be on those, you know, on
7:57that on that use? So, that's one. Second
7:59is agentic. Agentic is a huge multiplier
8:02because any use case, if you are
8:04deploying it in this sort of agentic
8:06workflow context, then there are
8:08multiple sort of turns and loops that
8:11might happen that can increase your
8:13token demand significantly. And then
8:15finally, the last factor is something
8:17called cache hit rate or the KV cache
8:20hit rate. And for those listeners for
8:23whom this term might be new, KV cache is
8:25sort of like the short-term memory of a
8:28model. And so, anytime an input request
8:31comes in to a model, it needs to process
8:33it. But, if it's already seen that input
8:36request before, then many times it
8:40actually gets stored in the cache. And
8:42then when it comes in again, it doesn't
8:44need to recompute it. It can just use
8:47those cached values. So, those are some
8:49key factors to kind of look at to get to
8:51a higher degree of accuracy when
8:53[clears throat] thinking about token
8:55demand. And then the final one is demand
8:57variability, which is how is your demand
9:00changing in a day? Like sometimes you
9:03may have products that get used quite a
9:05lot in in the morning hours, but not so
9:08much in the evening or vice versa. Same
9:11thing with seasonal variability. For
9:13example, you know, retail
9:15providers or e-commerce will see a surge
9:19during the holidays when they are trying
9:20to push a lot of products out, right?
9:22So, you do need to think through those.
9:25And then of course, there is the user
9:27growth. So, you started with a base
9:29number of users, but you as a business
9:32are trying to constantly drive up user
9:34growth. So, you need to factor in how
9:37much you expect that to grow as you
9:39think through your token demand.
9:41>> So, demand of course leads us to supply.
9:43How do you start thinking about supply
9:46and you've you've mapped out, you know,
9:47sort of your baseline
9:49the conditions that you just outlined,
9:51Shruti?
9:52Um
9:53how do you go about then translating
9:55that into creating the supply necessary
9:58to get all these tasks done?
10:00>> So, when it comes to token supply,
10:02that's where a lot of the AI
10:04infrastructure decisions lie. And when
10:06you're making that decision, what you
10:08want is maximum tokens availability,
10:11token output, while minimizing your
10:13token cost. Now, when you think about
10:16cost or total cost of ownership, often
10:19times organizations and decision makers
10:21can gravitate towards the easily
10:24available metrics and what I like to
10:26call input metrics,
10:27>> Okay.
10:28>> such as the cost per GPU hour
10:31>> Sure.
10:31>> or the flops per dollar, which is
10:33essentially how many floating point
10:35operations are you getting for a dollar.
10:37And these are input metrics because they
10:40don't tell you anything about the actual
10:43delivered token output, which is a
10:45function of much more than just flops or
10:48just you know, the memory you have. It
10:51is a function of extreme co-design. And
10:54so, the metric that represents both your
10:59input but also the output is cost per
11:02token.
11:03>> Okay.
11:03>> It's a very simple metric that tells you
11:06uh
11:07what is the cost that you will you're
11:10paying of cost of generating one token.
11:13And it's essentially, you know, the cost
11:17of GPU divided by how many tokens does
11:20the GPU produce. So, in a way it
11:22incorporates both the input and the
11:24output and gives you a sense of your
11:27true ROI from the the AI infrastructure.
11:31>> It's interesting to hear you explain it
11:32because it sounds so simple and I can
11:35understand coming from the other point
11:37of view, right? Of well, we're outlaying
11:39for the GPUs and the server racks and
11:41all the interconnectivity. And so we you
11:44know, we count those costs. But looking
11:46at it from the other end, as you said,
11:48the output cost just makes so much sense
11:51because that's what you're trying to get
11:53at the end is the intelligence, the
11:54token. And so putting the price on that
11:56seems like a really kind of clear way to
11:59think about it. Does the cost per token
12:01metric vary at all or do you have to
12:03think about it differently depending on
12:05the use cases as you were talking about
12:06before?
12:08>> Uh cost per token is sort of the base
12:11metric. Now, of course, it will vary
12:13depending on all the other things, like
12:17the model, the context, the
12:19intelligence, basically, and then the
12:21interactivity. So, any tokens that are
12:25generated by a more complex model or are
12:27more interactive are going to be
12:29costlier to produce.
12:30They are just That's just physics,
12:31right?
12:32So, yes, it definitely does depend on
12:35the models, the context, as well as the
12:37interactivity.
12:38But, you know, you said it really well
12:40earlier that
12:42ultimately,
12:43if the business runs on the output,
12:46which is the tokens, right? It is It is
12:49kind of a fundamental mismatch if you
12:51are evaluating infrastructure based on
12:53the inputs, but your business runs on
12:55the output.
12:56>> Right.
12:56>> And and that's why cost per token starts
12:58to get at sort of the real ROI. Why?
13:00Because it measures both in in many
13:02ways.
13:02>> Yeah.
13:03So, Shruti, as we think through input
13:05metrics and cost per token, is there an
13:08example that comes to mind that can
13:10really kind of bring this idea to life?
13:12>> Yeah, absolutely. In fact, if you look
13:15at Nvidia Blackwell compared to Nvidia
13:18Hopper,
13:19um and if you look at just merely the
13:22the input metrics, which is the hourly
13:25GPU cost, that's 2x. And so so that's
13:29Blackwell being maybe 2x more expensive
13:31than Hopper. If you just look at flops
13:34per dollar, that's also 2x. So Blackwell
13:37does deliver 2x more flops per dollar.
13:40And that sounds like a huge advantage,
13:43which it is, but it also doesn't even
13:46scratch the surface of the true sort of
13:50benefit and value of Blackwell.
13:52>> Okay.
13:52>> And that's because Blackwell, when it
13:55comes to delivered output, delivers 50x
13:59more tokens per watt compared to Hopper.
14:03>> 50x.
14:04>> 50x.
14:05>> Fantastic.
14:05>> the same infrastructure footprint, the
14:07Blackwell NVL 72 system delivers 50x
14:11more tokens than than Hopper. And that
14:15translates to a 35x lower token cost.
14:20>> Amazing.
14:20>> Yeah. And so that really, I think, you
14:23know, puts brings the point home on why
14:27not just look at the input metrics, but
14:30look at a metric like cost per token,
14:32which represents both what you're
14:34paying, but also what you're truly
14:36getting. Does that make sense?
14:38>> So I'm glad you mentioned I was going to
14:39ask you to go back. You'd mentioned
14:41extreme co-design. We've talked about it
14:43before. Obviously, anyone familiar with
14:45the space has has heard the term.
14:47Um but maybe you can dig in a little bit
14:49to what it means, particularly in this
14:51context.
14:52>> Yes. Uh
14:54I actually welcome the opportunity to
14:55talk about extreme co-design because we
14:57get asked this question quite a lot. And
15:00so, you know, often we get asked why
15:03extreme co-design. What does co-design
15:05even mean? Like, is it just integration?
15:08And, you know, people may think that
15:10this is just splitting hair or just
15:12semantics, but I do think that the
15:14distinction is important. Because when
15:15you think about integration, you think
15:17about different parts, different sort
15:21of, you know, independent units that are
15:23then integrated
15:25post factor.
15:27Whereas co-design is about designing
15:30from the ground up simultaneously
15:33multiple parts of the same system
15:36knowing that they are all optimized
15:38towards the same outcome, that of lowest
15:41token cost.
15:41>> Right.
15:42>> And so that's why the word co-design is
15:44extremely important. And the reason it
15:47is called, or rather we call it, extreme
15:49co-design is what NVIDIA does, is
15:52because of the depth and breadth into
15:55which it extends. So it's co-design
15:58across
15:59just compute? No, it's compute, memory,
16:03storage, networking, right? Everything.
16:06I mean, the Vera Rubin platform has
16:07seven chips, but it goes even beyond
16:10that. There's all the software that sits
16:12on top. So everything from the CUDA
16:15kernels to the runtimes to the serving
16:17software,
16:18um as well as all the way out to the
16:21ecosystem. Everyone from our, you know,
16:24silicon partners, our our OEMs, our
16:27cloud providers that we work with, the
16:29various OSS frameworks that we work
16:31with,
16:32um the co-design extends beyond just
16:35sort of, you know, what's in a system,
16:37what's in an AI factory, all the way out
16:39to ecosystem. And that's one of the
16:42reasons why it's it's extreme.
16:44>> Yes.
16:44>> And so, um anyway, but but you you asked
16:47a question of a more specific question
16:49about what are some of the co-designs
16:52that help with the with the cost per
16:54token. And um I think one important one,
16:57which I think you've actually discussed
16:59in the previous podcast, is the mixture
17:00of experts models, how the Blackwell NVL
17:0372 is such a great fit for them because
17:06it kind of helps with the inter-GPU
17:08communication. And then all the software
17:10in terms of Dynamos, disaggregated
17:12serving, coupled with any of the
17:15runtimes that we support, whether it is
17:16TensorRT, vLLM, SGLang, um doing a
17:20technique called wide expert parallel
17:22that greatly optimizes uh the inference
17:25performance, and then thereby reduces
17:28the cost per token for those mixture of
17:30experts models.
17:31Um so, that's one great example. The
17:34other really good example is actually
17:36the Vera Rubin platform itself,
17:38[clears throat]
17:39um which is built for the age of agentic
17:42AI. And to really understand why that
17:45extreme co-design is required, maybe we
17:48can look at what an agentic workload is
17:51is like. Um
17:53so, thinking about an agentic workload,
17:56let's draw the parallel to the
17:57conversational workload. When
18:00in a conversational setting, um
18:03the
18:04user prompts something, say you prompt
18:05something, and then the LLM says
18:07something, and then you say something
18:08else, and then it says it back. So, you
18:11are taking, as a human, turns with the
18:13LLM, with the AI.
18:15In agentic, um
18:18it's actually AI taking turns with AI as
18:21well as with software, because uh
18:24main agent can sort of, based on the
18:28user input, decide to do some reasoning,
18:31then decide, "Oh, I need to do a tool
18:33call." So, call some software. Then it
18:35might decide, "Oh, I need a sub-agent or
18:37a specialized agent to go do some work."
18:40So, it's going to take a turn with the
18:42specialization till the specialization,
18:44you know, does its computation, comes
18:45back with the result, and this just
18:47keeps going. And And we love agentic for
18:49that. That's right. Um and it's it's
18:52it's multi-turn in a way that uh
18:55that is sort of has no user involvement
18:59other than the prompt that the user gave
19:01in terms of like, maybe say, "Book a
19:03ticket to Miami." And then it goes
19:05through all of this several turns to
19:08then finally produce an outcome.
19:11Um
19:11and the number of turns involved in a
19:13Gen Teck is is significantly higher than
19:16conversational.
19:18Uh, so the number of LLM calls, like the
19:20number of times the large language model
19:22is called is also higher. And in
19:24general, the token demand for that
19:26reason is also higher.
19:27>> yeah.
19:27>> And that's why extreme co-design is so
19:30critical because you you're using up so
19:32many tokens, so you have to lower the
19:34cost per token.
19:35Um, latency is really important
19:37[clears throat] because on every turn,
19:39even a couple of milli- milliseconds
19:41more add up to, you know, several
19:44potential seconds of delay for the end
19:47end result. And then, um,
19:50finally, coming back to the Vera Rubin
19:52platform, um,
19:53now that we've sort of described the
19:55agentic workload, uh, we can clearly see
19:59why the co-design is required to
20:01accelerate sort of the LLM itself or the
20:04reasoning and, um, the the AI itself,
20:07you need the Rubin GPU, you need the
20:10Groq 3 LPX solution and deliver that
20:13ultra-low latency, you need Vera CPU
20:16because it's going to do all the stool
20:18calling or you know, sandboxing for for
20:20code generation and code testing.
20:22Um, you need, uh,
20:24the CMX platform that we've talked
20:26about, which is, uh, the BlueField, uh,
20:29DPUs together with Spectrum-X, which
20:32allow for the KB cache or the short-term
20:34memory, as we discussed, to be
20:36offloaded, um, when needed so that it
20:39can be retrieved when required for a
20:41match with with an incoming request. And
20:44so, that's sort of, uh, another example
20:47of co-design where
20:49being able to develop all of these from
20:52ground up helps a lot.
20:54>> Right, right.
20:55So, we talked about extreme co-design
20:57and you mentioned all the different
20:59pieces that go into it, building,
21:00designing, and building from the ground
21:01up. Software is a part of that, but
21:04maybe you could double-click Sruti into
21:07how software plays a role and how
21:08important software really is.
21:10>> Yes, absolutely. So, software actually
21:14is the difference between what you get
21:17in the real world, the delivered token
21:20output, and the actual token cost versus
21:23what you see on a spec sheet. Software
21:26makes all the difference, right? Right?
21:28All the things on the spec sheet, the
21:29system design, are
21:32cannot be fully realized unless you have
21:36software that makes use of it and
21:39delivers really good output.
21:40>> Right.
21:41>> And the the other important thing about
21:43software is that it's cannot be
21:46piecemeal optimization. You need to have
21:49a robust software stack that can turn
21:52on, enable every single optimization.
21:56So, that can do, say, NVFP4
21:58quantization. It can also do MTP or
22:01speculative decoding. It can also do
22:03disaggregator serving. It can do the
22:05wide export panel, the KV cache
22:07offloading, the KV aware routing, and on
22:10and on and on, right?
22:12>> To be able to stack all of those
22:14optimizations together is really
22:16important, because that is what gets you
22:18the 50x.
22:19>> Right.
22:20>> The 50x more throughput that we see with
22:22Blackwell and the 35x lower token cost.
22:25And so, that's software is a huge, huge
22:28piece of that story, for sure. Um the
22:31the other thing about software is that
22:32it never stops. Open source software
22:34especially, it never stops. And it's not
22:37just the NVIDIA team that is building
22:39this software, it's the entire
22:40ecosystem.
22:41>> Right.
22:41>> Right? It's it's all the OSS frameworks,
22:44all of our partners, customers, the
22:46developer community, and every small
22:49optimization that they do, that's a drop
22:53that just keeps adding and adding to
22:55this massive ocean of advantage uh that
22:58is the NVIDIA ecosystem. And so, just as
23:01an example, on both VLM and SG language
23:05are these inference run times, we've
23:07seen eight times more performance in
23:10just about 6 months.
23:11>> Wow.
23:12>> And that's huge. Because from the same
23:14infrastructure footprint, you're getting
23:16so much more token output, and that's
23:18driving down your token cost as well.
23:21So, absolutely, software is a huge, huge
23:23piece of the puzzle.
23:26>> So, we're through three of the four
23:27pillars. The fourth one, perhaps the big
23:30one, monetization. How do you talk about
23:33monetization? How does a business leader
23:35think about Okay, so I understand the
23:39importance of extreme co-design. I
23:41understand the different value of tokens
23:43in different situations, in different
23:45tasks, and Inception is wonderful,
23:48Recurrent Neural Networks, all the
23:48things you've elucidated here.
23:51How does a business leader think about
23:53monetization of the tokens?
23:56>> Right. So,
23:58when it comes to monetizing tokens,
24:00there are various different ways in
24:02which, you know, you can go to market,
24:04but one of the best proxies is to just
24:07think through it as you're generating
24:09tokens, and then you're selling the
24:11tokens. And so, when you think about
24:13selling the tokens, how do how much do
24:14you sell them for? Um and it's sort of a
24:17classic exercise in figuring out your
24:19pricing, which is A, you need to think
24:22about what is the cost to produce the
24:24token, which is this lowest token cost
24:26that NVIDIA is helping with, right? But
24:29you do need to understand what is what
24:31is your token utility, and given that
24:34token utility and token value, what is
24:36your cost to produce the token? And you
24:38obviously want to charge more than that,
24:40uh right? So, there's that. Okay. So,
24:42that's cost-based pricing, right? And
24:44then you also obviously have to think
24:45through value-based pricing, which is
24:48essentially how much is the willingness
24:51to pay? What is [clears throat] this
24:53sort of uh token utility? Uh how how
24:57valuable is it to the people who are
24:59going to pay for it? So, you do need to
25:00take that into account. And then
25:02finally, before you think through the
25:04pricing, you also need to think through
25:06what is the demand distribution. Because
25:09ultimately, there are revenue goals and
25:12you know, kind of profit margin goals
25:14that you are working towards. And so, to
25:17to land at a place that you like, you do
25:19need to think through where will your
25:22sort of bulk demand be? And how will the
25:25demand taper off when it is, say, for
25:29tokens that are not as uh as much
25:32utility, there may not be many takers.
25:35But in the same way, tokens that are
25:36highly valuable, there will be fewer
25:39people who are willing to pay the
25:41premium for that. You do need to account
25:43for that sort of [snorts] demand um
25:45distribution. And then with those three
25:47things, you can figure out what the
25:50pricing for each token can be. And
25:54and and then deploy it successfully. So,
25:56the the the key thing here though is
25:58that, you know, pricing the tokens
26:01again, is obviously just one proxy. Um
26:04there will be uh customers who are
26:06building
26:08value-added services on top of those
26:10tokens. So, like customers building AI
26:12native products or something like that.
26:14And then in that case,
26:16you the process is similar, but
26:20you do need to think through then what
26:21is the additional value you're adding on
26:23top of on top of just generating those
26:26tokens as well.
26:27>> So, going back Sheri to something I was
26:29thinking about when you were explaining
26:30extreme co-design. Um
26:33when you get to a point, kind of a sweet
26:35spot, where you've you've you know, your
26:37infrastructure is humming and the cost
26:39per token is low,
26:41does that mean that ultimately, you
26:43won't need as many GPUs to produce the
26:46number of tokens that you really need?
26:48Or or
26:50what happens in that kind of a scenario?
26:52>> Right. This This is a great question.
26:54And what we see here is the classic
26:57Jevons paradox.
26:58>> Right. Okay.
26:59>> Which is essentially, you know, you
27:00would think that, "Okay, the GPUs are
27:03way more productive. They're, you know,
27:05generating so many more tokens. Do you
27:07need less of them?" And the answer is
27:10absolutely no. And the reason is is as
27:14you see the efficiency, new use cases
27:17get unlocked.
27:18>> Sure.
27:18>> And people just figured out we have all
27:21you know thriving research community,
27:23data scientists, ML engineers. They just
27:26figured out how to just use up that
27:29efficiency, how to absorb that
27:31efficiency, and do more with it.
27:33>> Right. People aren't going to run away
27:34from intelligence. They want to use it,
27:36you know.
27:36>> That's right. And if you look at the
27:38sort of macro pattern that we've seen so
27:40far, it's it's very telling. So, when
27:44generative AI became a thing and people
27:46were, you know, sort of generating
27:47summaries and and images, um that was
27:50great. Then we lowered the cost per
27:52token. And instead of, you know, needing
27:55less GPUs, they needed more GPUs and
27:58more tokens. Why? Because test time
28:01scaling and reasoning. And so, our
28:03researchers figured out that by us
28:05scaling at test time, we can generate
28:08better accurate, more intelligent
28:10responses. And um that was valuable for
28:13the use cases. And so, that happened.
28:15And it just it didn't just happen once.
28:17We are seeing that again now with
28:19Agentic.
28:19>> Right.
28:19>> Now that we've, you know, sort of
28:21figured out how to deploy these mixture
28:23of expert models, uh reasoning models
28:25efficiently, and lowered the cost per
28:27token for those significantly, now here
28:29comes another inflection point where
28:32it's like, "Okay, we've got more tokens.
28:34Let's do more with them."
28:36>> And and so, that's where the Agentic
28:38revolution is happening. And so,
28:40definitely it's it's it's Jevons paradox
28:43in action at the macro level. And I've
28:45also seen this play out at at sort of,
28:47you know, individual customer level. So,
28:50that's a great question.
28:52>> So, Shruti, can we kind of ground this
28:54in some examples of how businesses,
28:56organizations that you've been working
28:58with,
28:59are putting all of this into action and
29:01really, you know, extracting value from
29:03the tokens and using them to build
29:07>> Yeah, absolutely. So, when you think
29:09about taking tokens and turning it into
29:11business value, there's four primary, I
29:14would say, business models, so to speak.
29:17Number one is what we just discussed,
29:19which is selling tokens directly. And a
29:21lot of NVIDIA customers and partners are
29:24doing this and the examples are
29:25fireworks, Base Stand, Together AI, Deep
29:28Infra. There's just so many of them and
29:31all of them are helping their end
29:33customers
29:34build, you know, valuable services on
29:37top of the tokens that they are selling.
29:39So, that's number one. Number two is AI
29:42native companies [clears throat]
29:44who are building products from the
29:47ground up with AI in it, you know, sort
29:50of permeating through it from day one.
29:54And those are customers like Perplexity
29:56or Cursor, who have a coding engine,
30:00and and, you know, many, many, many,
30:02many others. So, that's sort of the
30:04second model. Third is you might use AI
30:07to enhance your existing products and
30:10infuse AI through your existing
30:11products. And again, lots of different
30:13examples. We have Shopify, we have
30:15Airbnb,
30:17there is Adobe. In fact, a lot of them
30:20are doing both. They are building AI
30:22native capabilities, but they're also
30:24using AI to improve their existing
30:28products. For example, Adobe is, you
30:29know, they've built their Firefly family
30:31of models and then they're using those
30:33models to infuse new capabilities into
30:37Photoshop, for example. And then the
30:40final bucket is is pretty much every
30:42organization today, which is trying to
30:46improve their internal operations, their
30:49internal processes, improve employee
30:51productivity by deploying AI. So,
30:54they're not necessarily sort of
30:55deploying external customer-facing
30:57products or services, but these are
30:59internal to their own operations. And
31:01again, Nvidia is working pretty much
31:04with with everyone across the board on
31:06something like that as well. So,
31:08there's there's So, those are the four
31:11key ways. I'm sure that that there are
31:13others that are more nuanced that I
31:15missed, but that's a useful framework to
31:18think about how to take the tokens and
31:20and turn them into business value.
31:21>> Right.
31:22So, for the business leader who's
31:24listening and comes away from this with
31:28a better understanding of the pillars
31:29and how they relate, and really what in
31:32tokenomics, right? What the cost of a
31:34token is and how you ascribe value and
31:36everything.
31:37How do they get started putting this
31:39into practice? What advice would you
31:41leave them with for thinking about how
31:42to put this into action in their own
31:44organizations?
31:45>> Um, I think the the the best place to
31:48start is to first just think through
31:50what is the final outcome. And usually
31:53that starts with your customers. Whether
31:55they are external customers or your own
31:57internal employees and you know,
31:59internal processes,
32:00that that that's immaterial. You have to
32:02start back from the customer need, from
32:05the use case. Because as we discussed,
32:07the use case actually dictates a whole
32:10lot. The user and the use case dictates
32:13what type of model will you use. It
32:14dictates what type of context lens you
32:17might need to support. It dictates what
32:19type of interactivity will you need. So,
32:21the intelligence and interactivity, and
32:23then those factors are what dictate what
32:26type of infrastructure you need.
32:28>> Right, right.
32:28>> Um, and then of course, we walk through
32:30the key metrics such as cost per token
32:32when making those infrastructure
32:34decisions. And and and then, you know,
32:37that that's the the supply, you know, so
32:39you you essentially walk back from token
32:42utility and and token demand, think
32:45through token supply, and then once you
32:47have a handle on all three of those, you
32:49think through your sort of monetization
32:51strategy, and and then go to market, and
32:54then you fly.
32:55>> Customer first, work back from that,
32:57easy enough. Um Shruti, thank you so
32:59much for taking the time to join the
33:01podcast and really break down tokenomics
33:03in a way that I think
33:05listeners, viewers can extract so much
33:07value from. To talk about extracting
33:09value. Um it's a really
33:11comprehensive, but yet really easy to
33:13follow and understand, you know,
33:15uh start to finish how this all comes
33:18together. So,
33:19>> Thank you.
33:20>> Yeah, thank you for having me.

About this transcript

This page contains the full transcript of Inside AI Tokenomics: How to Profitably Turn Tokens Into Business Value | NVIDIA AI Podcast Ep. 299 by NVIDIA, generated from the public captions YouTube serves with the video. The transcript has 5,576 words across 881 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.