YT YouTube2Text

YouTube2Text → Transcripts → The Window Before AWS Wakes Up: Nebius CTO on Racing the Hyperscalers

The Window Before AWS Wakes Up: Nebius CTO on Racing the Hyperscalers — Transcript

by Unsupervised Learning: With Jacob Effron · 10,282 words · 1,709 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00If Nvidia could have bought, like, could
0:01have produced like 10x the chips, this
0:04whole market would have been shaped
0:05different. Because there is not enough
0:07capacity for their scale, this is a
0:09relatively small business for them. For
0:11the hyperscalers, it's not just matter
0:14of, like, sales. It's also a matter of
0:16changing their infrastructure that was
0:19built at scale for other kinds of
0:22workloads. This is a massive investment
0:24on their part [music] that needs to be
0:25justified by the size of a potential
0:27business. And it's not that they're yet.
0:34>> The emergence of Nebius and the other
0:35hyperscaler competitors these past years
0:37has been a fascinating part of the AI
0:39story. It's something I've really been
0:40interested in, and so it was a lot of
0:42fun to have Nebius's CTO, Dima, on the
0:43podcast because we're able to dig into a
0:45bunch of interesting things about the
0:46business and how the space has unfolded.
0:48Talked about Nebius's really interesting
0:50origin story and the original
0:51relationship with Yandex and how the
0:53company came to be. We talked about what
0:54it's like to be inside one of these
0:55companies, how Nebius works, you know,
0:57with Nvidia, what it looks like to get
0:59their next generation of chips live. We
1:01talked about why the opportunity
1:03actually existed and why AWS, GCP, Azure
1:06didn't just capture this entire market.
1:09We also talked about Dima's view on a
1:11bunch of new technologies like Disagg
1:12and Prince, among other things. All in
1:15all, just awesome to get to talk with
1:16someone who's at the epicenter of the AI
1:18infrastructure market about his thoughts
1:20on everything about where the space is
1:21headed. I think folks will really enjoy
1:23it. Without further ado, here he is. I
1:25guess I'd love to talk just maybe about
1:26the broader like AI cloud market,
1:29uh, you know, today. And I think that
1:30there's a few maybe larger and prominent
1:32public,
1:33uh, you know, I guess termed neo clouds.
1:35Uh, and then, you know, it feels like
1:36there's a new one getting funded every
1:37day. I know you don't like that term.
1:39Uh, but maybe just talk a little bit
1:40about how you think about what
1:41differentiates, like, some of our
1:43listeners are probably thinking like,
1:45you know, all that really matters is
1:46just access to GPUs and like, you know,
1:48you wherever you can get them, you go.
1:50Um, maybe maybe use that as a straw man
1:52and and tell us why that's not the case.
1:54>> Well, I'm obviously biased because,
1:56like, first of all, I'm a software
1:58person.
1:59And
2:00uh, our product and business was
2:03launched on the idea that software is
2:06the mode, right?
2:07So, your ability to serve
2:10the customers that consume, and you
2:12know,
2:13tens of thousands GPUs of GPUs and then
2:16one interconnected cluster with the same
2:18platform as you serve like self-service
2:20customer that runs one GPU for half an
2:23hour was our like value prop from the
2:26very beginning. So, we believe that
2:30this is extremely important. Being able
2:32to build large scale,
2:34but have the possibility almost like
2:36sell it in retail in a sense. This is
2:39what we think like
2:42definitely differentiates us from the
2:44market from everyone on the market.
2:46Like small remark, neo clouds hate that
2:50term like with a passion because like
2:53it started as a distinction as in these
2:56are kind of cloud companies that are
2:58doing this weird strange new thing, but
3:00then everyone wanted in that market. So,
3:03like when we started there were like you
3:05know, five, seven companies that were in
3:08this area that were called neo clouds.
3:11Now, I think it's closer to like
3:143 to 500.
3:16>> Yeah.
3:16>> Depending on what you even
3:19are ready to call a neo cloud to the
3:21scale. And
3:23we really don't like this
3:25stretched umbrella term anymore. We are
3:28the AI cloud. We don't care. Like let's
3:31others other guys be neo clouds. We are
3:33the AI cloud.
3:35And this is our mental model. We are not
3:37competing with the neo cloud markets.
3:41We kind of believe we are past that. Our
3:43next competition point is taking on AWS
3:47and Google Cloud.
3:49Yes, we are way smaller. Yes, our like
3:52revenue is like what? I think like
3:551% of AWS nowadays.
3:58But we are running fast. We serve a very
4:02particular market. We
4:05We have a bet that this is the market
4:07that is going to grow, that is going to
4:09explode. So, like people are concerned
4:11that like these investments are huge.
4:14Like people put a lot into the
4:15infrastructure, etc., etc., etc. My view
4:19if I compare what we see in the market
4:21with like what was there 2 years ago, we
4:25are barely scratching the surface.
4:27I think the overall demand in the next 3
4:29years will like
4:31increase tenfold, hundredfold maybe. So,
4:35like we we are like at the at the verge
4:39of of another like explosive growth.
4:42Right now, we see these huge facilities,
4:45right? But a lot of those are privately
4:47owned for internal needs of a certain
4:49companies, right? So, they like
4:52Elon Musk is talking about his Colossus
4:54cluster and it's like it's immense
4:56scale. Like really, it's like real solid
4:58large-scale thing.
5:00But
5:01it's not for the broader market, right?
5:03It's for their internal needs. Maybe for
5:06some partners that they decide to share
5:07with.
5:08>> of uh
5:09Yeah.
5:09>> Yeah. Anthropic and OpenAI are massing
5:12the like the capacity, but they do it
5:14for their own product. And this is like
5:16a wholly separate discussion for these
5:18companies will essentially want to be
5:20everything. They will want to be the
5:22internet and contain everything and
5:24every product and every capacity angle
5:26within themselves.
5:28Uh
5:29there is a question of Google, Meta, and
5:33Microsoft who have a lot of capacity,
5:36but are constantly juggling and
5:37struggling between like pro- giving it
5:40out to the market uh for like uh to rent
5:44or giving it out to internal teams.
5:47Because all of all of these companies
5:49are extremely
5:51machine learning AI-enabled.
5:53They need a ton of capacity for internal
5:55usage, for runtime, and for research.
5:58So, they have to constantly juggle this.
6:00So, our bet is that there is a niche for
6:02a independent as in not affiliated with
6:05a large company with a huge internal
6:07consumption
6:09player that will serve the markets of
6:12the media players who want to be the
6:14large companies.
6:16>> I thought it would be interesting for
6:16our listeners to talk a little bit about
6:18the history of Nebius. I feel like it's
6:20a really interesting path that you you
6:22and you know had to end up here.
6:25And so maybe just tell a little bit
6:26about how the business came to be.
6:27>> I will start with a like with a moment
6:31I joined, right? So
6:33I am CTO of the company but I've been
6:36with the company from the very beginning
6:37like since before it was called Nebius.
6:39So the new company is established
6:43I think during 2024 also the deal was
6:46finalized. We were finally announced our
6:48separation from Yandex. No connections
6:51to the Russian assets anymore, clean
6:53slate, etc. etc. etc.
6:56And that was it.
6:57>> You know people have come to these AI
6:59cloud businesses in different ways and
7:01obviously you know I felt like the
7:02Yandex team was like legendarily really
7:05really strong
7:06you know on the engineering side. You
7:08know a lot of folks building out AI
7:09clouds came from like crypto or energy
7:12or like not not necessarily the same
7:14you know maybe technical heritage. To
7:17what extent I guess was the Yandex
7:18experience really relevant and to what
7:20extent was it like you know it turns out
7:22no one knows how to build a GPU cloud
7:23because it's all kind of new and
7:25you know there's there's everyone's
7:26learning on the fly.
7:27>> First we
7:30had and still have the core team that
7:33supports the hardware side of business
7:36that came from Yandex that had
7:37experience operating hardware like
7:41infrastructure at scale like at real
7:42scale.
7:44Hundreds of thousands, millions of
7:46servers span over multiple facilities,
7:48etc. etc. etc. These people like were
7:51the core of our build out like initial
7:54build outs and expansion and they are
7:56still with the with the company. So this
7:57definitely helps.
7:59I think like we would be way less
8:01successful if not for that core team.
8:04>> Yeah.
8:05>> On the The side it's interesting because
8:07it's a twofold. Like because like with
8:09hardware, it's very straightforward. We
8:11needed people with experience, we had
8:12people with experience, they started
8:14building out from the get-go.
8:16Uh on the software side, it's
8:19it was a weird one. First of all, we had
8:21to rewrite everything from scratch. So,
8:24our previous experience was kind of like
8:26garbled.
8:27Second, we had to grow so fast that by
8:31for example, by the end of 2024 when the
8:33new platform launched, like a
8:36significant portion of our engineering
8:38team, software engineering team, was not
8:39from Yandex anymore because we had to
8:41hire during like uh
8:44uh this period extensively. As a team,
8:46we were working on this cloud platform
8:49uh from like from the initial idea,
8:51right? So, we kind of just went with it
8:54and uh doubled down on software. Like we
8:58are not doing a managed bare metal, we
9:00are not doing a patchwork of third-party
9:02services. We want to build a holistic
9:04cloud experience, the holistic cloud
9:06platform.
9:07Uh
9:08and we kind of doubled down on that even
9:11with the whole rewriting of the
9:13platform, like doing it from scratch,
9:15etc. etc. etc. And of course, like
9:17having a ton of talented engineers on
9:19the team uh obviously helped. Like
9:22just for the scale of it, right? You
9:24have mentioned like people who like came
9:27from crypto or came from other avenues
9:29of life. They are mostly
9:32concentrated on hardware. Like they
9:34their idea is like you need to buy GPUs,
9:36you need them to run, and then you'll
9:37find the buyer.
9:40In our case, for example, in 2024, when
9:43our hardware team that actually like
9:45procured GPUs, uh did the build-outs, uh
9:51were working on our own servers and
9:52chassis, writing firmware and stuff,
9:55building the network.
9:57That whole team was roughly like 40 to
9:5950 people.
10:00>> Yeah.
10:00>> The software organization was already
10:02200.
10:05Right? So, we kind of can certainly
10:07think of ourselves as a software-first
10:09company. We are building the cloud on
10:12the solid foundation of hardware.
10:13>> Yeah.
10:14>> We're not a hardware-first company.
10:15>> Makes total sense. I guess I'd love to
10:17take our listeners like inside
10:19Nebulas and I feel there's so many
10:20interesting parts of the business, but
10:22maybe one place to start would just be I
10:24imagine as you're like getting, you
10:26know, new Nvidia chips that come in,
10:28like there must be a bunch of stuff you
10:29have to do within the business to to get
10:31this all all up and running. So, like
10:33what does that look like from, you know,
10:34I don't know, you're talking to Nvidia
10:35folks and they're, you know, giving you
10:37early insight into uh into the next
10:38generation of chips to like, you know,
10:40actually getting these live for a set of
10:42customers.
10:43>> That is a mess. No, I mean, like in a
10:45good way. You know, like
10:47this whole this whole market, this whole
10:50field is running at such a pace
10:53that's like everyone is like trying to
10:55catch up, right? And Nvidia, despite
10:58being like
10:59in the lead of this
11:01essentially leading and showing the
11:02direction for this market, is not the
11:04exception. So, first, Nvidia folks, like
11:08way before, show you the prototypes of
11:11what they want to build in the next like
11:13year, right? They kind of show you the
11:16schematics, like how which will be
11:19like
11:20optimized internally, where's your like
11:23PCI connections, how it that look like,
11:25what is the
11:27like future platform would look like.
11:30Then sometime later, they come to you
11:32and say, "Oh, like we've optimized
11:34things. It's now going to be different."
11:36You're like, "All right."
11:37Then they come to you and like, "Oh, we
11:40have like the first lab samples. Do you
11:42want to have a look?" Yes, of course we
11:43do want to have a look. "Oh, by the way,
11:45this doesn't look like the stuff you've
11:46showed us 2 months ago." "Oh, we've
11:48changed stuff because like we found like
11:50because they are so compressed in time,
11:53same as they everyone else, they are
11:55constantly iterating while these chips
11:58are being produced, right? So, they've
12:00It's not your like, you know, there is
12:02like an ocean in the consumer market
12:04that Apple iPhone that you see today was
12:07actually planned 3 years ago and final
12:10design more or less was settled like a
12:12year ago, and they take the final year
12:15to polish it, to build this product.
12:17Like, AI market is running at such a
12:20pace, like, that's
12:22uh
12:23whatever like they announce is it's
12:25indicative. And then teams are
12:26constantly iterating and uh
12:28>> the workflows are changing, right? So,
12:29I'm sure they're iterating uh as as that
12:31goes.
12:31>> Yes, like they're iterating, like, going
12:34further and further and further. And
12:35Nvidia is a behemoth now, right?
12:37Whatever they do influences the whole
12:39market. supply chains, there's
12:41everything. So, they have to uh like
12:43uh look at that. So, at some point,
12:46you'll get a sample, and it's more or
12:47less like what you'll get in production.
12:50Then you start building, like, you are
12:54uh
12:55testing everything. You like incorporate
12:57support for the new hardware, for the
12:59new restrictions or capabilities into
13:01the software platform. You figure out a
13:04way to isolate workloads from from the
13:06start. We are we are cloud, we are not
13:08providing bare metal hosts, right? So,
13:10we need to split, we need to make sure
13:12that our virtualization works, that our
13:13storage works, that our networking
13:15works.
13:16Sometimes [snorts] we are on the
13:17bleeding edge of this, so we like
13:21for some of the generations, we were
13:23probably
13:24the first team like maybe one of the
13:27first teams that actually built a like a
13:29cluster of a certain size.
13:32Uh so, we've uh hit some issues with
13:35like with the software that uh no one
13:37has seen before. So, we had to report
13:39that back to Nvidia, like, propose a
13:42patch, get there, like, fix from it,
13:45make sure that it runs, etc., etc., etc.
13:48So, like, we kind of feel like we are at
13:50the forefront of it. But,
13:52all in all, it's a pretty normalized
13:55process. It may sound messy, right?
13:57There are like multiple iterations, you
13:58go back and forth, you're in constant uh
14:01contact with like Nvidia hardware folks,
14:03with Nvidia software folks. Uh
14:06you're trying to solve the issues on
14:08your end at the same time, incorporating
14:10into a more complex platform, thinking
14:13how can you expose that to customers
14:15because, for example, working with uh
14:18like your usual black will chip like,
14:21you know, the
14:22B300s and then
14:24transferring that experience to GB300s,
14:27which are a whole different architecture
14:29in terms of isolation. You're talking
14:30racks rather than servers. So, you need
14:33to
14:33find a way how to do maintenance on the
14:36rack instead of the server where you
14:38just switch off and you move the
14:40workloads to another one.
14:42Uh
14:43despite all of that, this kind of works
14:45all the time and every day you like push
14:48it a little bit further and then in the
14:50end, yes, there is a product and the
14:52customers use it and that's it.
14:55>> One thing I think
14:56is would be something interesting your
14:57thoughts on is just the the differences
14:59in, you know, um kind of what end
15:01customers end up wanting. Like,
15:02obviously it can be easy to think, okay,
15:04everyone just wants to, you know, come
15:06to you and and get similar things, but I
15:08imagine you segment your customers or
15:09think about differences in what, you
15:11know, is important to different
15:12customers. How do you think about like
15:14the the key differences in in in these
15:16customer desires today?
15:18>> Hi, I'm Anne Marie and I'm the producer
15:20of Unsupervised Learning. Thank you so
15:22much for listening to the show and if
15:24you've enjoyed the conversation, please
15:26subscribe. Subscribing helps us grow,
15:28which helps us continue to lock down
15:30some amazing guests. Thank you so much
15:32for your listenership and see you next
15:34episode.
15:35>> How do you think about like the the key
15:37differences in in in these customer
15:39desires today?
15:40>> Uh
15:41>> [sighs]
15:42>> it's not that different, to be honest,
15:44because like every customer is uh
15:47like its own unique flower, right? But
15:50at the same time, workloads in principle
15:54are not that different.
15:56It's either training and you kind of
15:58know how training looks like or
16:00inference and you kind of know how
16:02inference looks like.
16:03Circling back a couple of years ago, for
16:05example, we kind of underestimated the
16:08storage component. We thought GPUs are
16:11the most important thing and they are.
16:13But
16:14uh alongside the whole training angle,
16:16you see a massive demand for storage.
16:19And it's not for the like amount of like
16:22HDs on VMs that you can provide to the
16:24customer. It's about throughput. It's
16:26about latency.
16:27In my prior life, pre-AI
16:30workloads and infrastructure build outs,
16:32I am I was talking about gigabytes per
16:34second if we were talking about network
16:36attached storage, right? It was like
16:38it's equivalent. Right now, we have
16:40customers routinely kind of like hitting
16:43our like a common share like one volume,
16:46right? Connected to a number of GPU
16:48servers
16:49with like, you know, 5, 6, 700 gigabytes
16:53per second of throughput.
16:56This is extreme.
16:58Like 2 years ago, I didn't even think
17:01about that.
17:03And now this is
17:05more or less like something we expect.
17:08At the same time, if you are targeting
17:10these numbers for the highest demand
17:12like the most demanding customers,
17:16your baseline is so high, you kind of
17:19can't serve anyone.
17:20>> Yeah.
17:20>> Right? So, if you hit this like use case
17:23where the customer needs 700 gigabytes
17:26per second from a like, you know, 3
17:28petabytes of shared file storage
17:29connected to a thousand of servers or
17:32like 8,000 GPUs,
17:34you kind of know how to serve a massive
17:37cohort.
17:38So, I'd say in terms of like performance
17:40or features per se,
17:42the customers are not that different.
17:44Then you need another subset of like
17:48what the customers really want. Because
17:51there are definitely different customers
17:53in terms of not their scale, not their
17:56size, not their workloads, but rather
18:00how mature they are. What are their
18:03obligations
18:05like
18:06for their clients, right? Because we can
18:09talk about like any labs. Like new labs
18:11routinely have a seed rounds of in the
18:13billions nowadays, right?
18:15So, most of that budget goes into
18:17commute. So, you can have like a
18:19contract for a billion dollars with a
18:22newly founded Neo lab. Who
18:25like they don't care that much about
18:28anything but GPU, storage, performance,
18:32availability. This is it. They like
18:34often times they don't even care about
18:36the location of those GPUs. They don't
18:38care if that's US, Europe, like France
18:42or Iceland, like
18:45Why why bother? Like
18:47Internet connection is good, storage is
18:49there, GPUs are working as a and we are
18:51talking about a billion-dollar contract.
18:53Right? At the same time, we now see and
18:57like start on boarding enterprise
18:59customers.
19:01These are companies whose like the
19:03contract with us will be smaller.
19:05Sometimes significantly smaller.
19:08But they are more established. They have
19:10customers and they have obligations for
19:12that customers.
19:13They have procedures.
19:15Uh
19:16they needs to make sure they're
19:17compliant to a number of regulations in
19:20like multiple
19:23uh countries.
19:24>> Yeah.
19:24>> So, now we are getting into that whole
19:26angle of being the real cloud, you know?
19:30Like certifications, compliance,
19:33security posture checks, let us look at
19:36your source code.
19:37Uh how do you solve problem X? How do
19:40you like protect your customers from
19:42attack vector Y?
19:45Etc. etc. etc. And I'd say if you're
19:48talking about differentiated customers,
19:50this is the most differentiating thing
19:53as opposed to what you would expect from
19:55like the size or the problems.
19:58>> You know, I've seen a lot of people are
19:58thinking through this this trade-off of
20:00of like throughput latency or a
20:02trade-off of like latency and and and
20:03kind of cost. Um and I imagine you have
20:06some customers that have workloads that
20:07are super latency-sensitive and they'll,
20:09you know, do small batches or whatever
20:11is required uh to to get to that
20:13latency. And then you have other
20:13customers that are much less
20:15latency-sensitive, maybe background
20:16agents or something. Is that all just,
20:19you know, uh determined on the software
20:21side? I guess if you have those
20:22different customers of like basically,
20:23how, you know, uh the batch size that
20:25you're running these workloads in, or do
20:27you actually construct
20:29the clusters in any different ways based
20:30on those workloads?
20:31>> No. No. No. No.
20:33This was the premise from the very
20:35beginning. We are building the universal
20:37cloud platform, and we need to make sure
20:38that our baseline is high enough to
20:43fit
20:44most of the workloads.
20:46>> Yeah.
20:47>> Then, there is the SLA play, right? So,
20:49customers come with, for example, for
20:51the managed inference part of the
20:52business, customers come with a
20:54different demands, and then you are in
20:57this kind of a benchmarks game, right?
20:59They want this particular footprint
21:02based on this particular traffic pattern
21:04fits this SLAs in terms of latency, or
21:08like, latency as time to first token, or
21:11the whole like request latency.
21:15And you start optimizing for that.
21:17But, this is never done on the hardware
21:19level, because hardware is good enough.
21:21This is rarely done on the lower
21:24infrastructure level, because it's
21:25universal, and again, good enough.
21:28No compromise for performance, etc.,
21:30etc. And you are going further away to
21:33the orchestration level, to the model
21:35level.
21:36Uh
21:38way further up the stack, all the way to
21:41how exactly does the request from the
21:43customer, like the endpoint, gets to the
21:46inference servers, and what are your
21:48performance
21:49hits along the way.
21:50>> Yeah. That makes sense.
21:51>> Yeah. So, this is like 100% software.
21:53>> I guess a lot of people I've talked to,
21:54you know, feels like this this disagg,
21:56you know, splitting prefill and decode
21:58is very in the air these days, and a lot
21:59of people talking about that. Like, is
22:01that something you guys think about now,
22:03or or at least, how do you how do you
22:04think about that kind of in the future?
22:05>> I think we are in the stage where
22:07everyone's talking about that, and no
22:08one's doing that properly.
22:11Uh so, because like,
22:13it is a complex problem to solve. And
22:16particularly in inference, you have a
22:18lot of other problems, like
22:21way lower hanging fruits that you can
22:24fix, address, and hit those SLAs.
22:27Uh so, we are quite cautious about that.
22:30We don't have it, for example, as a
22:32go-to solution for every customer for
22:35every workload because like first it
22:36doesn't make sense. Second, it's really
22:39hard to operate and uh, work with like
22:43provides
22:44proper SLAs around that is like
22:47uh, proving quite difficult as of today.
22:50We have some promising results in our
22:52like prototype uh, services but not as
22:55of today. At the same time, we have some
22:57like disaggregated setups for the most
22:59demanding customers in production right
23:01now. It's just a more
23:03you know, wide glove
23:05uh, manual approach.
23:07>> Yeah.
23:07>> But
23:08uh, especially for these uh, newer
23:10architectures of like like you know,
23:12those Grace Blackwalls, whole
23:15uh, node that has really fast
23:17interconnect, a lot of challenges that
23:19you've had before with disaggregated
23:20setups are kind of solved if you're able
23:23to fit them in the footprint of a rack
23:25because like the throughput is so much
23:27faster, like interconnect is so much
23:28faster, you
23:30kind of stop caring and that this is a
23:33way of just balancing the profile and
23:35the code nodes
23:36to match the particular pattern that uh,
23:39you see from the customer.
23:41>> What what's your mental model for like
23:42the workloads that do fit in a rack?
23:44>> I don't know. It's a very case-by-case.
23:46Well, these are definitely significant
23:47workloads like we are talking like
23:50thousands of requests per second, right?
23:52We are talking like billions of tokens
23:55that flow through. But at the same time,
23:58because we are the cloud, we can, for
24:00example, like share the rack between two
24:02workloads and not be that bothered about
24:05that, right? So, for example, on the
24:07cloud side,
24:08we for for this particular product, we
24:11only allow to consume it like a track
24:14level. You cannot order two Grace
24:16Blackwalls uh,
24:19nodes. It doesn't make sense.
24:21>> Yeah.
24:21>> Right? Like you'll have to allocate the
24:24whole rack and then like uh, structure
24:26workloads within it. But for example, we
24:29only were a client of such a solution as
24:31we do with managed inference, for
24:33example. You can collocate workloads,
24:35and you can find a way to do that. So,
24:38uh in a sense, you don't need to fit fit
24:41anything in the rack. It's just is there
24:44a workload that will benefit from this
24:47disbalance,
24:48like in intentional disbalance of
24:50prefilling the code nodes.
24:52>> Yeah.
24:52>> And can you fit more than one into a
24:54single rack, and how do you solve the uh
24:57packing problem?
24:58>> Yeah, that makes sense. You kind of
25:00mentioned there's like lower hanging
25:01fruit on the uh on the inference side,
25:03you know, that than than doing that
25:04maybe. What what are like some of the
25:05lower hanging fruits today that you
25:07still think we're we're not
25:08uh as as a as an industry optimized on?
25:10>> Well, first of all, I think as an
25:12industry we didn't optimize So,
25:14like, you know, [laughter]
25:16pardon my French, but uh in a sense, I
25:19strongly believe that
25:22running inference, particularly in the
25:24like variety of open weight for open
25:26source-ish models that are available
25:29today, is not a solved problem.
25:31Like, if you take something just to run
25:34with it as it's published like in the
25:36initial support in some of the open
25:38source inference servers, run with the
25:41novel models,
25:42your results will probably be
25:43underwhelming. It's not that fast, it's
25:46not that cheap, especially with like
25:48current prices on the infrastructure.
25:50It's easier to go and use like uh I
25:52don't know, Fable 5.1, Astra 6, etc.,
25:55etc., etc.
25:56But, because of the open architecture,
25:59because like uh
26:02of everything that we know as of today,
26:05you can optimize the way you process
26:08those requests. You can optimize the way
26:11the inference server is configured. You
26:14put a lot of effort in kernel
26:15optimizations, sometimes kernel over
26:18optimizations based like on the
26:21particular pattern that you see in the
26:23traffic.
26:24Right? You There is a lot of
26:27underexplored [snorts]
26:28potential in model weights optimization.
26:31And I'm not just talking about like
26:33stuff like speculative decoding, for
26:35example. I'm talking about like model
26:37pruning, selective pruning, compression.
26:40Uh because I believe that uh
26:43And I And I And we see that uh within
26:45our customer base,
26:4795% of our customers do not run vanilla
26:50open-source models.
26:52These are modified, fine-tuned, etc.,
26:54etc., etc.
26:56It means
26:57uh that these are the models that uh
27:01uh start to address a particular use
27:03case that the customer has.
27:05Particular use case usually means
27:07limited domain, scoped domain.
27:09Scoped domain usually means that you
27:12need to retain like frontier
27:14intelligence and reasoning for a
27:16particular subset of tasks, and you
27:18don't care about the rest.
27:21And now we are getting somewhere in
27:23terms of like maybe you don't need such
27:26a large model anymore. Maybe you need
27:29just like uh
27:30model that is selectively compressed in
27:34a way where it works for a particular
27:36task, and it doesn't work as good for
27:39the rest of it. Now, of course, that
27:41doesn't work with like open-ended, you
27:43know, deep research or uh broad coding
27:47tasks anymore. But I strongly believe
27:50that for
27:52that for real AI adoption, when we are
27:55talking about serious enterprise
27:57business, when we are talking about like
27:59practical AI that actually solves
28:01problems, we are getting more and more
28:04into these specialized agents, limited
28:06domains, where you
28:08even stop caring about the costs. You
28:11care more about predictability of the
28:13outcome, and you care about time as in
28:16like real seconds and minutes till we
28:19get that outcome.
28:20>> Yeah.
28:21>> And I think there is a lot of potential
28:23there, and we still
28:25will we will still see
28:28uh
28:29a couple of nice breakthroughs there.
28:31>> There's a ton of work around inference
28:33optimization, and a lot of that gets
28:34done in open source, right? Or like, you
28:36know, contributions that that share
28:37throughout the community.
28:39Uh there's obviously a bunch of work on
28:41uh on on like smaller purpose-built
28:43models and and compaction and a bunch of
28:45these things. Across all these problems,
28:46which do impact, you know, uh how
28:48enterprises consume inference, which
28:51ones like Nvidea should have a solution
28:52or offer a product around, and which
28:54ones you're like, "Hey, it's great that
28:56that exists and we'll benefit from that,
28:57but we don't need to be at the
28:58forefront."
28:59>> to be at the forefront of everything.
29:01Uh
29:03in a sense, these are additive and even
29:07maybe multiplicative, right? Like
29:10if you have a right subset of things
29:13optimizations that you introduce for a
29:15particular use case,
29:17it means that you can reach the targets
29:20easier, better, faster,
29:22in a more predictable fashion.
29:24So, we invest in all avenues. Like
29:28one of our acquisitions back in like uh
29:30the beginning of summer was targeting
29:32specifically that. Like, we basically
29:34bought the company for the talent in
29:36model optimization, model compression,
29:38speculative decoding, etc., etc., etc.
29:40Combined with existing talent, so we
29:42like we have like
29:43uh
29:44>> [snorts]
29:44>> team that is 2x the size. And we like we
29:47have serious manpower, like as in tons
29:49and tons of people who are working on
29:51these problems. Like, we have
29:52specialized groups for like kernel
29:55optimization, specialized groups for
29:57like model compression problems.
29:59Because this is like I feel
30:02if you want to talk about like running
30:05inference as a service at a serious
30:07level,
30:08this is the only way you can do it
30:10nowadays.
30:11>> And I guess like, you know, maybe along
30:12those lines you guys have done, you
30:13know, a a few acquisitions, and and one
30:14of them that that caught a few people's
30:16eyes was, you know, uh uh Tavily in the
30:18kind of like web search space. Um
30:21maybe talk a little through a little bit
30:22like, you know, the rationale for that,
30:24how you think about that fitting into
30:25the broader platform.
30:27>> So, first we were very impressed by the
30:28product and their go-to-market motion.
30:32It was almost a year ago now, right? But
30:35uh
30:35we felt like we knew how to help them
30:38run even faster with our expertise and
30:41technology and low-level stuff.
30:44Because actually a running a web search
30:46engine is
30:50is not a very simple task. Not neither
30:52from like the software development
30:55perspective nor from the
30:56>> Cuz a lot a lot of you used to run one.
30:58>> And some of us dabbled a little bit in
31:01our previous lives. Yes.
31:03Right? So, this was the premise. Like,
31:06we have a
31:08great team that is running real fast in
31:11terms of like market penetration and in
31:13general go-to-market motion. And we know
31:16how to make it complementary with our
31:18expertise on the low-level stuff and the
31:20running large-scale infrastructure and
31:22for example large-scale web index. It's
31:24not exactly like embedded into the rest
31:28of the offerings that we have now today.
31:30Like, we are exploring and we have
31:33something cooking for like for the
31:35managed inference combined with like
31:37agentic tools because for example, we
31:39also develop sandboxes nowadays, right?
31:41So, this is another like building blocks
31:44for building one of one of the building
31:46blocks that people use to build their
31:48agents. We have
31:50several other products that are in the
31:52earlier stages of development in that
31:55regard, too.
31:55>> Yeah, I mean, it seems like there's
31:56obviously a set of tools that folks are
31:57going to use to build agents and build
32:00on open source and it seems like you
32:02want to have a bunch of those tools that
32:03you will offer to to your end customers.
32:05And web search kind of being a
32:07key primitive for any any long-running
32:09agent. We'll consume a decent amount of
32:11tokens on on just pulling information
32:12in. So, yeah. And then I'd love to
32:14switch gears to just like the broader
32:16you know industry buildout and and kind
32:17of the you know I think I think there's
32:19lots of questions folks are asking about
32:20that. I figured one place to start that
32:22actually might just be interesting for
32:23our listeners is
32:25to what extent are like the data centers
32:27that you work with similar and you know
32:29like there's a reference design and they
32:31all kind of like you know I don't know
32:32to you it doesn't really matter which
32:33one's which. And like what are some of
32:35the differences in them that actually
32:36impact the business of you know running
32:39an AI cloud on on top of them.
32:41>> Well, to be frank with you, I have no
32:43idea what the walls of the facility are
32:45made of.
32:47So, they're probably different, but that
32:49doesn't really [laughter] help.
32:52In all seriousness, what's within the
32:55walls is something that we fully
32:57control.
32:58We do not rent data centers when someone
33:02like have built the like the network on
33:05their own or
33:07brought some GPU GPUs or servers, etc.
33:12If we don't own the facility and we rent
33:15it, for example, we rent
33:17well, the building,
33:19uh electricity, but that's it. All the
33:22internal build-out is controlled by us.
33:25It's uh like
33:27uh it's done
33:28uh according to our standards. So, from
33:31the software side of things, there is
33:33literally no difference which data
33:36center we're running. Is it our own
33:38facility?
33:39Is it something that is uh built within
33:41a rented facility? Or, for example,
33:44we've recently announced the so-called
33:46asset-light model when
33:47a third party brings capital in order to
33:51set up the facility and GPUs, and we run
33:54our platform on top of that.
33:56It's going to be the same. These data
33:59centers are built according to our like
34:01fitted out according to our spec.
34:03Uh hardware is something that is
34:05familiar to us because like unknown
34:07hardware is quite hard to adopt and make
34:11sure that you provide uh proper SLAs.
34:14Uh and that's it. So, we don't
34:16compromise there. We never rent
34:18something from people and present it as
34:21our own cloud. If you launch a virtual
34:23machine in Nebius cloud,
34:25you can be sure that it runs on our
34:27stack top to bottom.
34:29>> Yeah, makes uh makes sense. But, I guess
34:31there's a lot of dialogue around the the
34:33bottlenecks to the AI build-out today,
34:34and I think people talk about a bunch of
34:35different ones, you know, whether it's
34:37power or the labor to actually make, you
34:39know, data centers, cooling, networking,
34:41you know, uh HBM, uh
34:43GPUs as
34:45overall. Like, how do you think about
34:47what the what the key bottlenecks are
34:49there if you were to to kind of rank the
34:51across all of them the the the few that
34:53feel most most poignant right now?
34:55>> I'm sorry to disappoint. There is one
34:58single bottleneck and that's your
34:59ability to execute.
35:02Because yes, all of those are issues.
35:04Like
35:05there are GPU shortages in the markets,
35:07component shortages in the market.
35:09Nowadays CPUs shortages in the market.
35:11Like NVMe shortages in the market. There
35:15are like not enough facilities, not
35:17enough electricity, not enough not
35:19enough qualified like
35:22workforce to help you build it out.
35:25Not enough like small components like
35:28network transceivers for example. What
35:31else is
35:32not in abundance? Oh, oh, oh, you know,
35:34like
35:35those
35:38large scale
35:40I'm not sure I like I know the proper
35:42English terminology but like you know
35:44those machines that actually convert
35:46electricity and place like
35:48transformators.
35:49>> The transformers.
35:50>> Transformers. Yeah, like do you know the
35:52lead times on those these days? Ooh,
35:54years.
35:55Right? So like everything is a problem.
35:58And I think the single bottleneck and
36:02at the same time the single like mode
36:04that you can impose on that is like your
36:06ability to execute around all these
36:09problems [snorts]
36:10and concerns. Your ability to source,
36:12your ability to cut corners where like
36:15appropriate. Your ability to execute in
36:18general. Your ability and flexibility to
36:20move stuff between facilities. Okay,
36:22we're going to move this one from here
36:24to there and then we'll going to move it
36:26back candle and like yes, we'll postpone
36:28this one but is this means we'll be in
36:30time with this one. This is important
36:32etc. etc. etc. So I think that like
36:36in this particular field like an
36:38experienced project manager is probably
36:42in highest demand I've ever seen
36:44throughout my career.
36:46>> Yeah, that's uh I I I guess the uh the
36:48bottleneck is really good product
36:49managers uh which
36:50>> Yeah, exactly. Like your your overall
36:53ability to like have this large context
36:56of like problems everywhere and mind and
36:59navigate that
37:01becomes the most.
37:03>> Yeah. It's interesting cuz you obviously
37:04have uh you know, the hardware itself
37:07changes changing. Um you have a bunch of
37:09workloads changing. Obviously, we have
37:11much more of long-run agents. We have uh
37:13more post-training going on. You know,
37:14people running up source models. How
37:16does that actually impact the way that,
37:19you know, the overall like I got I guess
37:21the overall build-out of the clusters
37:23themselves? To what extent is it like,
37:24"Hey, you know, a workload's a workload
37:25and you know, we can we can run it."
37:27>> No, the workload's a workload. Uh like
37:30exactly like you said. Uh
37:32it is slightly different, but uh in a
37:35sense, you need a high enough baseline
37:38to not be like constrained by that.
37:41And then the rest is just your like
37:44uh optimization problem. The way like
37:46you pack things, the way you split
37:48things, right? So, like
37:50you don't need whole nodes for some of
37:53the workloads. So, you need to collocate
37:55workloads. You need to have the right
37:57primitives. You need to expose those
37:59through the platform, etc. etc. etc. But
38:01in a sense, like differentiated
38:04workloads almost never
38:07like impose any changes on on on on how
38:10you build on the lowest level. Yes, of
38:12course, like nowadays with like this uh
38:16new architectures like what like like uh
38:21What is it called? The Nvidia LPX like
38:23the
38:25the outcome of the Groq deal.
38:27Right? Uh this like adds to it.
38:31But in a sense,
38:33uh if you have experience running other
38:36generations, this is just another like
38:390.5 generation, right? Yes, this will be
38:43a little bit novel and it's connected in
38:48in a weird like compared to the previous
38:51ones way and it kind of like there are
38:54some restrictions on you can run and not
38:56run on that.
38:57But
38:59efficiency gains. This is like just and
39:01like a niche
39:03local edition to the platform to what to
39:05what you offer to the broader market.
39:08But in general, no. There is like way
39:11less custom like
39:13custom solutions that you would expect
39:15from that. The workload is a workload.
39:17>> Why didn't the hyperscalers like
39:20move into this market faster or win this
39:22market, you know, more convincingly
39:23early on in in in this first chapter at
39:25least?
39:26>> First of all, like sometimes there is an
39:29ocean that they didn't even try.
39:32There there there is a solid like solid
39:35amount of business that hyperscalers get
39:37from out out of this market. Like huge
39:40deals.
39:41They are very competitive. Sometimes
39:43people say, "Oh, you have to see AWS
39:44prices." Well, these are public prices.
39:47I assure you like for proper deals they
39:49are doing proper prices and they are
39:50very competitive and like
39:52like they are very aware of the market.
39:56I think in a sense like there are two
39:59two angles to that. First, like what
40:02I've mentioned before, a lot of these
40:04companies need to juggle like their
40:06internal consumption and what they can
40:09put for all like for for the broad
40:11market.
40:12And
40:14there is like we are in chip shortage
40:16world, right? So, like if Nvidia could
40:18have bought like could have produced
40:20like
40:2110x the chips,
40:23probably this
40:25this this whole market would have be
40:27shaped differently.
40:28Because they would like Microsofts and
40:30Googles of the world would both buy
40:32capacity for themselves and for the
40:34broader market to operate.
40:36This is probably like simply not enough
40:38capacity for their scale, right? Because
40:41there is not enough capacity for their
40:43scale, this is a relatively small
40:45business for them.
40:47AWS revenue is what like 100-billion or
40:49something?
40:50Uh
40:52I mean like
40:54uh a business for like two, three, four
40:56billion,
40:57it's not insignificant by any margin,
41:00but like compared to what they they're
41:02conservative large-scale established
41:05businesses,
41:06it's not that easy, either.
41:08>> Yeah. But, your point is basically like
41:10they're, you know, they get a chip
41:12allocation that is like, you know, not
41:13as large as they'd want. They have to
41:15use some of that, you know, a large
41:16amount of that for training themselves.
41:18There's only so big that that inference
41:19business or, you know, uh that that the
41:21cloud business can be given all that.
41:23And yeah, it's interesting to figure out
41:25how to prioritize. Do you go after like
41:27the, you know, the few folks with large
41:28contracts or do you try and get this
41:30like mid-market enterprise, uh you know,
41:33up-and-coming segment? Uh but, I guess
41:34you're you're kind of doomed either way
41:36if if you only have a certain number of
41:38chips to
41:38>> Yes, exactly. And then and and and and
41:41then at the same time, this becomes a
41:43relatively small like part of your
41:45revenue portfolio and you're like, do I
41:48even need to allocate resources to that?
41:50And
41:52that is especially a pressing question
41:54because like
41:56for the hyperscalers, it's not just uh
42:00a matter of like sales. It's also a
42:02matter of changing their infrastructure
42:05that was built at scale for other kinds
42:09of workloads.
42:10The networking, the like throughput
42:12performance, the storage performance, by
42:14the way,
42:15right, that I've mentioned earlier. So,
42:18this like this requires, if you want to
42:20have it universally like uh implemented
42:23on through the whole stack, this is a
42:25massive investment on their part
42:27that needs to be justified by the size
42:29of a potential business.
42:31And it's not that they're yet.
42:33I think if we start getting out of this
42:36like chip shortage in like, I don't
42:39know, two, three,
42:40five years, maybe,
42:42uh
42:44we will definitely see this shift. Uh
42:46and they will pay more attention because
42:49this will become like an increasingly
42:51large chunk of business. And they will
42:53start investing, they will like
42:55reorganize and redesign their like
42:59internals to be more accommodating to
43:01these demands, etc. etc. etc. And then
43:03with their whole sales power, they'll go
43:06and try to win this market. And the
43:08the question is whether the novel
43:11players like like us, for example, will
43:13be fast enough to grow
43:16and represent like significant
43:18competition when they get there.
43:21>> Yeah.
43:21>> And like I'm 100% that sure that they
43:24will
43:25pay more attention in the future.
43:28>> Yeah.
43:28>> Our bet is that we'll be fast enough to
43:31be in the league with them and fight as
43:34equals rather than a small player trying
43:38to run in their trails.
43:39>> Makes sense. I mean, what do you think
43:41happens to like you mentioned there's
43:42like these like, you know, 300 to 500
43:44like, you know, neo clouds everyone
43:45trying to get into this market. Like as
43:47you kind of, you know, you mentioned you
43:48kind of think like the chip shortage
43:49comes to an end at some point. Like as
43:51you zoom 5 years, 10 years in the
43:53future, like how many of these
43:54businesses are still around or like what
43:56is the end state of of of this market?
43:58>> Listen, I I I think the whole like uh
44:04AI consumption is going to grow.
44:07And it will be like universally
44:11used everywhere like AI is everywhere.
44:12Like right now like web services are
44:14everywhere, AI services will be
44:16everywhere.
44:19So in a sense, like you have AWS, right?
44:24But you also have Hetzner.
44:26You also have DigitalOcean. You also
44:28have Linode. You also have like tons and
44:31tons and tons of local like hosting
44:33providers that are like giving you
44:36virtual machines, bare metals, bare
44:37metals, etc. etc. etc.
44:39So this market is going to be vast. Will
44:42all of the current like three to 500
44:44companies survive the next 5 years?
44:47I'm not sure.
44:49But I am pretty sure that there will be
44:51a place for the very long tail of
44:53providers,
44:54down the line.
44:56Like
44:57some of this the current players will
44:58consolidate, some of them will die off,
45:01new new players will be launched, etc.
45:04etc. etc.
45:05It's a like it's not a bad market right
45:08now. It's not a bad business to be in.
45:10So, it will constantly attract capital
45:12and attention and investments, etc. etc.
45:15etc.
45:16And I trust that so like in 5 years time
45:18there's still going to be like hundreds
45:20of players in this market who found the
45:22investment into the right ones and
45:24several hundred million dollars and they
45:26can offer something
45:29to someone in the market. Same as it is
45:31with like with
45:35the small scale clouds like Digital
45:37Ocean these days, right? Like Digital
45:40Ocean did not die because AWS is
45:42growing.
45:42>> They're obviously doing a lot of things
45:43in-house like sandboxes and you know,
45:45web search and you know,
45:48helping people with their their models
45:49over time and
45:51you know, I think one question a lot of
45:52people are asking is you know, what
45:54parts of the AI infra stack end up just
45:57being subsumed into these future AI
45:59hyperscalers and like live, you know, in
46:00just one combined offering and then what
46:02are like the standalone pieces of AI
46:04infrastructure that will persist as
46:06companies?
46:06>> In a sense, I think it will like
46:08it will
46:09kind of mimic what's happened with like
46:12the
46:14legacy generation of clouds, previous
46:16generation of clouds,
46:18right?
46:19When your growth
46:21kind of dictates that you need to start
46:24addressing new and like new sectors, new
46:26markets, etc. because otherwise your
46:29growth is like bounded and you can't
46:31like
46:33can't show your projections, can't
46:35justify evaluations, etc. etc. etc.
46:38So, there will be a natural push for the
46:40larger companies to start like
46:43increasing
46:45their portfolio, like trying to get into
46:47new products, into new services, etc.
46:49etc. etc.
46:52It doesn't mean that there will be a
46:54one-size-fits-all solution.
46:56Because in a sense, if you're like if
46:58you're concerned by that,
47:01you will uh
47:03like
47:04we just need to stop doing whatever we
47:05are doing and just wait for for like
47:08OpenAI to rule the world, Anthropic to
47:10rule the world. Because these companies
47:12will eventually like you do realize that
47:15OpenAI has models,
47:17runs inference, they have their own
47:19search that they like they don't exactly
47:22expose it, but if you're building the
47:24agent on their platform, it has access
47:27to search.
47:28Uh they have all kinds of like uh
47:31sandboxing,
47:32local agents, cloud agents, etc. etc.
47:35etc. They both OpenAI and Anthropic are
47:38now investing in forward deployed
47:40organizations and are going to go and do
47:43the work that was traditionally like I
47:46don't know, Deloitte's or like
47:49Accenture's piece of pie.
47:51Right? So, they will be everywhere. And
47:53I think this is a natural motion for the
47:55large company to start getting into into
47:57a broader
47:59markets.
48:02But it doesn't mean this is they are
48:04going to be the only ones. So, even
48:06nowadays, there is a place for
48:09independent object storage players.
48:13All of a sudden. Right? Or independent
48:15compute players. Or independent
48:18small-scale inference players for that
48:20matter.
48:21Independent database players. Like like
48:24when when so like it seemed like Google
48:26had like big big query and whatever, all
48:30kinds of data stuff, but then Snowflake
48:33kind of like appears from out of
48:35nowhere.
48:36AWS has like all kinds of like managed
48:39databases, large-scale databases, like
48:42Aurora, Redshift, etc. etc. etc. But
48:44then MongoDB appears and is pretty
48:46successful. I can't say it's like super
48:49large-scale and probably by nowadays
48:52standards will never be because nowadays
48:54they started for the large companies
48:56like trillion plus.
48:58Right? But like 2 years ago, it was a
49:00very solid and very successful company.
49:03Or someone like ClickHouse appears with
49:05their cloud offering, right?
49:07Also, a massive managed database, an
49:09independent company that finds its niche
49:12and works it real well. So, I think like
49:16larger company, larger clouds will grow.
49:18They will have like products in every
49:21sector and will be pretty successful at
49:24selling them those because of bundling,
49:26because of their sales power, etc.,
49:28etc., etc. But there will always be a
49:30place for niche players. And because the
49:32market is huge, niche doesn't mean
49:35small. Niche doesn't mean small scale or
49:39low revenue or low valuation. These can
49:42be solid companies.
49:43>> Yeah. But it's interesting you mentioned
49:45like, you know, OpenAI and Anthropic
49:46having this kind of broad suite of
49:47offerings, too. I mean, in many ways
49:48they're obviously building their own AI
49:50clouds. And I thought one thing that was
49:51interesting recently was, you know,
49:53obviously limitation of those AI clouds
49:55versus others is is the the, you know,
49:57having just one model family, right? To
49:59be able to call from. Um but I thought
50:00it was really interesting that OpenAI
50:01announced that partnership with Base 10.
50:03I'm curious what your take on was was on
50:05that. I mean, is that them just trying
50:06to get into this general, you know,
50:07hyperscaler game across models?
50:10>> In a sense, we see a lot of push and
50:12concerns about people like claiming that
50:15OpenAI models or Anthropic models are
50:17too expensive.
50:19Everyone is puzzled or surprised by the
50:21bills they get get get at the end of the
50:23months.
50:24So, this might be like an answer to
50:27that. Well, if you're like, well, go try
50:29open open models.
50:32At the same time, open models are still
50:35behind the closed source closed models,
50:38right?
50:39However good those models are, they're
50:42still lagging behind.
50:43And given people an option to try like
50:47lower scale with the ability to switch
50:50back to the top frontier model, it's
50:53also be a good incentive. Like, anyway,
50:55I think like OpenAI is doing a very
50:59interesting move with that partnership.
51:01And I like I like it really much because
51:03like
51:05this kind of shows
51:07flexibility, right? They're not set in
51:10stone. Like they're exploring different
51:12options, different things that they can
51:14or cannot do. Like looking for different
51:18options to like uh so but I I I think in
51:21a sense
51:22this all plays into one single thing.
51:25Like OpenAI wants same as Anthropic.
51:28They want to be one single like touch
51:32point. Like they they want to be one
51:34single thing for everything.
51:36Even that is like okay, but sometimes I
51:39want to use open source models within my
51:41Codex controlled environment, by the
51:43way.
51:45Right? So like well, good for them, but
51:48if they uh company, the person, the user
51:51is still within your ecosystem, you
51:54still have a lot of options to steer
51:56them, to upsell them, to introduce them
51:58to new services,
52:00etc. etc. etc. So if that like
52:04alleviates one of the concerns, so be
52:06it. Why not?
52:07>> Well, I always like to end these
52:08interviews with some like overly broad
52:09questions that I stuff into the to the
52:11very end. And so if you'll if you'll
52:12humor me, I I guess first question
52:15is just what's one thing that you've
52:17changed your mind on in AI in the last
52:19year?
52:19>> In the last year? Oh, a lot has a lot
52:22has happened in the last year.
52:24No, like the obvious answer would be uh
52:26software engineering.
52:28Like a lot of people forget, but a year
52:31ago we lived in a totally different
52:33world.
52:33>> Yeah, it's amazing. Like dog years in AI
52:35world.
52:36>> Yeah, way less capable. Yes, way less
52:38capable in terms of the model
52:40capabilities and in terms of how good
52:43the modern harnesses and tools are. It's
52:46like uh the AI coding. Because like a
52:49year ago
52:51it was total Right now, it's
52:55it's a solid part of like your everyday
52:58tooling.
52:59But for example, uh if you're not
53:01talking about the industry trends or
53:03like talking about more like my personal
53:05thing.
53:07I've
53:09started relying more on the like
53:11AI-enabled like local agents and
53:14harnesses to offload a part of my
53:17cognitive load, to be honest.
53:19>> Well, obviously like you run a really
53:20large technical organization, and I
53:22think everyone's trying to figure out,
53:23you know, I mean, obviously the models
53:25only recently got really good on the
53:26coding side. And so, everyone's trying
53:27to figure out what it means to like
53:29build teams in the with you know, given
53:31the new model capabilities and run an
53:32organization. I'm curious like what you
53:34where you've settled and like, you know,
53:37you know, yeah, how you've thought about
53:39that.
53:39>> Cautiously.
53:41Conservatively.
53:42I'm very concerned that
53:45modern models
53:48and the attractive user experience that
53:51they provide, like modern agents, modern
53:54models, etc., etc., etc.
53:56Is something that will
53:59lead us to uh uh
54:02to issues.
54:04You like
54:05you see slop like
54:10being revealed in like
54:12in chats, in merge requests,
54:15all over. And I like I try to paste
54:18that, and I try to like
54:21be very observant and vigilant about
54:23that. So, for example, like we don't
54:25restrict token usage, right? People can
54:28use whatever they want. Our budget is
54:30quite significant. We don't impose like
54:32per person limitations or per department
54:35limitations.
54:36But they
54:38at the same point, we try to like look
54:41at the usage,
54:42figure out what it is used for,
54:45look at the most ineffective ways people
54:48use it.
54:49We
54:51like constructed a set of policies that
54:54doubles down on the idea that whatever
54:57you've done with the agent, it's still
54:59your result. It's your responsibility.
55:03It's your pull request. It's your code.
55:06It's you who is on call. It's you who's
55:08going to wake up during in the middle of
55:10the night if something goes awry, etc.
55:13etc. etc.
55:15And if something was created by a
55:16complete autonomous agent, it is marked
55:19like that. Uh
55:21complete autonomous agents will not
55:23touch like the core services for a
55:26while.
55:27Because like this is like this is like
55:29out of bounds. Like
55:30Sometimes people are surprised that like
55:33I work in a company that's kind of
55:35participates in this whole AI
55:38revolution, and I'm very conservative,
55:40but
55:41at the same time like I've seen how the
55:44sausage is made, and I like
55:47I'll be cautious.
55:48>> What's the biggest mistake you feel like
55:50you've made in in building Nebia so far?
55:52Obviously, a lot's gone right, but I'm
55:53curious you know something that in
55:54retrospect you you wish you guys had
55:55done differently.
55:56>> It might sound funny these days, but we
55:59underestimated the scale.
56:01Like and it's not about the growth of
56:03the company. We were pretty optimistic
56:05from the very beginning, but
56:08I personally and that was like my strong
56:10bias for the first like
56:13significant part.
56:15Uh I underestimated the scale that a
56:17single client operates at nowadays.
56:19Because like
56:215 years ago, if you were talking about a
56:2330 megawatt data center, it was a
56:25serious facility.
56:26It hosted like hundreds of thousands of
56:28servers like run
56:33glorious gloriously large for clothes.
56:36By now, 30 megawatt sometimes it is not
56:38enough for a single client.
56:41Which I'm still getting used to. Right?
56:45Yeah, the storage I've mentioned, right?
56:48Like
56:494 years ago, when you were talking about
56:51network attached storage, you were
56:52talking about like
56:54hundreds of megabytes, maybe gigabytes
56:56per second of throughput.
56:58Right now, some of our clients routinely
57:00fetch I don't know 700s of gigabytes per
57:04second from a single like shared file
57:07system.
57:09And just like I'm I'm I'm still like I
57:11underestimated it in the beginning and
57:13I'm still getting used to this scale.
57:15Like it's weird like I'm an
57:17infrastructure person. I've worked and
57:20I've seen these numbers throughout my
57:21career multiple times. And all of a
57:24sudden you add a couple of zeros to the
57:26end and call it a new norm.
57:29>> Yeah.
57:30>> Still puzzling.
57:31>> It's crazy. [laughter]
57:32Um
57:33and and I guess lastly like there's
57:34obviously a lot of you know unsettled
57:36questions in the in the AI field today
57:38that you know lots of people trying to
57:40figure out where where the world goes.
57:42I'm curious like you know what you're
57:44paying attention to and I guess you know
57:45with a lens toward you know if it goes
57:47one way or another it would really
57:49change kind of the strategy for for
57:51Nebius as a whole. What are you kind of
57:52most paying attention to on like these
57:54these unsettled questions of where the
57:56AI world's going?
57:57>> Well
57:59I stopped caring about like the new
58:01models being larger or smarter. I just
58:05accepted that it's a given.
58:07Uh
58:09I'm
58:10kind of trying to look out for like a
58:13new novel architectures and approaches
58:15but I don't see any as of today.
58:18I hope somewhere deep down in like a
58:22stealth lab something new is cooking but
58:24for now I frankly don't see anything
58:27that
58:28like a
58:28is a
58:30thing that I would be even like would
58:32would place even a a small bets on.
58:36But I'm more more interested in the
58:39whole outcome optimization part on the
58:42agent side.
58:44Right? Because like we we are used to
58:46talking about models and size of models
58:48and families of models etc. etc. etc.
58:51But now that we've shifted to this
58:54multi-turn long-running agents that we
58:57are talking not about requests but
58:59rather outcomes companies are talking
59:01about bidding for per outcome. Right?
59:05I'm very interested in how that part
59:07will shape up and we have like a small
59:09research group internally within Nebius
59:11that's working on this problem,
59:12actually.
59:13How is like
59:15how do you increase the chances of
59:17getting to that outcome? Not by
59:18increasing the model or like changing
59:21the model. That is, of course, part of
59:22the discussion. But, I believe there is
59:24a lot of optimization on the part where
59:26you're
59:27introduce more sophisticated or maybe
59:29more simplified tools.
59:31You introduce the harnesses that start
59:34to evolve according to a particular
59:36task. And this kind of gets us back to
59:39like uh
59:40significant CPU consumption because like
59:44a lot of that a lot of like modern AI
59:46workloads actually go back to becoming
59:49code. Like this codified, it's like very
59:52formal. You see like quality gates, you
59:54see sophisticated tools, etc., etc.,
59:57etc. And sometimes your agent works for
59:59minutes and minutes and minutes without
1:00:00even issuing a back end request to an
1:00:02LLM.
1:00:04Because the workloads uh start to like
1:00:06to sit on this part. And this is the
1:00:08part that really interests me because I
1:00:10think
1:00:11like a lot of future AI adoption lies in
1:00:15how reliable your agentic scenarios are
1:00:19and how reliably you reach those
1:00:21outcomes.
1:00:22And I think like the whole optimization
1:00:25loop outside the model is important part
1:00:27of that.
1:00:28>> 100%. Um well, this has been an awesome
1:00:30conversation. Uh really appreciate you
1:00:32coming on the show and uh yeah, thanks
1:00:34so much for for coming on.
1:00:35>> Thank you for having me.
1:00:36>> I'm Jacob Effron and this has been
1:00:38Unsupervised Learning, a podcast where I
1:00:40get to talk to the smartest people in AI
1:00:42and ask them tons of questions about
1:00:44what's happening with models and what it
1:00:46means for businesses in the world. As I
1:00:47hope is clear, I have a ton of fun doing
1:00:49this. It's a nights and weekends project
1:00:51in addition to my day job as an investor
1:00:53at Redpoint, but our ability to get
1:00:54these incredible guests on really comes
1:00:57from folks like you subscribing to the
1:00:58podcast, sharing it with friends. It's
1:01:00really what ultimately makes this whole
1:01:02thing [music] work. And so, please
1:01:03consider doing that and thank you so
1:01:04much for your support and listening.
1:01:06We'll see you next episode.

About this transcript

This page contains the full transcript of The Window Before AWS Wakes Up: Nebius CTO on Racing the Hyperscalers by Unsupervised Learning: With Jacob Effron, generated from the public captions YouTube serves with the video. The transcript has 10,282 words across 1,709 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.