Your Self-Hosted LLM Needs to Say No - Harshul Jain & Parv Khatri, Audible & SAP — Transcript
Full transcript
- 0:05Good morning everyone. My name is
- 0:07Hershel Jan and I have here with me
- 0:10Pearl. Today we are going to talk about
- 0:13saying no. It's called as admission
- 0:16control. How our system decides which
- 0:20request to get in when your coding
- 0:22agents overwhelm the LLMs you host
- 0:25yourself.
- 0:27Uh before we move on a quick intro about
- 0:30us.
- 0:32So I am a senior software engineer at
- 0:34Audible where I lead some of the MLAI
- 0:37infrastructure.
- 0:38Uh I am also writing an open-source
- 0:40handbook called as LLM inference at
- 0:43scale. It has received over 300 plus
- 0:45stars by now.
- 0:47I am also maintaining an open-source
- 0:49framework called agentship which helps
- 0:51you ship reliable and secure agentic
- 0:55harnesses in an ultra fast speed. Uh
- 0:59perv would you like to introduce
- 1:00yourself?
- 1:01>> Sure. Hi everyone. I'm Perv. I'm a
- 1:04senior software developer at SAP. I work
- 1:06on like cloud application and LLM
- 1:09infrastructure. I wrote an LLM fine-tune
- 1:12guide. I contribute to agentship and I'm
- 1:15like building a task tracker which is a
- 1:18local post task board for you and your
- 1:20agent.
- 1:22Uh apart from this if you would like to
- 1:24connect with us our LinkedIn QR codes
- 1:27are right there on this slide. Now back
- 1:29to you Hershel.
- 1:32Great. So
- 1:35here is the plan. We are going to start
- 1:37with understanding the motivation why
- 1:41any serving stack can admit so much of a
- 1:45traffic.
- 1:46Second, we try to dive into admission
- 1:48control where the decision to say no
- 1:51lives and what it needs to see. Third,
- 1:55we will go through some of the proven
- 1:57strategies, the different ways a system
- 2:00can actually make the admission
- 2:01decision. And fourth, we will try to
- 2:05dive into the benchmarking where you
- 2:07will learn how to prove that your policy
- 2:11for admission is working or it's not
- 2:13working.
- 2:16So let's get started with the
- 2:18motivation. Um
- 2:21and for this
- 2:26everything today uh let's try to
- 2:28understand what is inference. Everything
- 2:31today happens during this inference
- 2:34itself. So inference is simply a moment
- 2:37when someone sends a prompt to the model
- 2:40and gets back an answer.
- 2:43So the step one is like a user prompt
- 2:45comes in. It's a sequence of words. Uh
- 2:48for example, it could be like failing a
- 2:51fix a failing test. The model which is
- 2:55running on your GPUs writes the answer
- 2:57one token at a time. A token is roughly
- 3:00like a word as I mentioned. Uh it writes
- 3:03the first word then uses that word to
- 3:06pick the second uh then it uses both to
- 3:10pick the third and it keeps on
- 3:12continuing like that and it cannot write
- 3:16all of them at once because every word
- 3:18depends on the previous one. So those
- 3:22once all the words are generated they
- 3:24stream back to the user uh who ask for
- 3:28the result and that volt prompt and the
- 3:32answer out is like that one request. Now
- 3:34I want you to hold on to one small
- 3:37detail like model has no idea how long
- 3:41this answer is going to be. It only
- 3:44finds out whether it writes when it only
- 3:49finds out when it writes the first last
- 3:51token.
- 3:53Keep that in mind. It will matter in the
- 3:55few minutes.
- 4:01So a model never sits on the internet
- 4:05itself. Every request travels through a
- 4:08short pipeline. It starts in your app or
- 4:12your agent. Then it reaches your API
- 4:15gateway
- 4:16which checks your API key counts how
- 4:18many requests you are sending. Then a
- 4:21router picks a replica which which is
- 4:23just one running copy of the model and
- 4:26finally the requests land on the
- 4:28inference engine. The common some of the
- 4:31common inference engines are the VLMs,
- 4:34SGLAN, Tensor RT etc. engine is what
- 4:39actually hosts your model.
- 4:42It actually serves your model uh and it
- 4:45loads the model weights into your GPUs.
- 4:48It groups many requests into one batch
- 4:50so the GPU stays busy and for every
- 4:53request it keeps something called as a
- 4:56KV cache. In plain words, the KV cache
- 4:58is the memory the request builds up when
- 5:02it try when its answer is being written.
- 5:05Longer the answer, bigger this memory
- 5:07gets.
- 5:08Keep this picture in your head. Every
- 5:10box here makes some decision but one
- 5:13decision is missing and we will find it
- 5:16out in the few minutes.
- 5:19First, how much an engine can actually
- 5:22hold.
- 5:26Every deployment has a ceiling and it
- 5:28comes from the three targets that pull
- 5:30against each other. Quality is how big
- 5:33your and precise your model is. Latency
- 5:37is how fast the first token arrives and
- 5:40every token after it. Throughput is how
- 5:43many requests your GPUs can serve at a
- 5:45same time. You know the old saying good,
- 5:49fast, cheap, but you have to pick two.
- 5:51It's the same here. You cannot pick all
- 5:53the three. You cannot prioritize all the
- 5:55three. So you can't So every product has
- 5:59to pick two. For an example like an
- 6:01interactive chat, you would want to go
- 6:04with the quality and the latency because
- 6:06users are more sensitive to that. It's
- 6:09quite obvious to the users if the
- 6:11quality is down or the latency is high.
- 6:14For something like an async agents, you
- 6:16would want to go with a higher
- 6:17throughput as well as quality. If
- 6:20quality is less, you will have to redo
- 6:23the task and redoing is very very
- 6:25expensive because these are the
- 6:27background and the asynchronous tasks.
- 6:30And the high volume real-time systems
- 6:32such as the autocompletes they would try
- 6:35to favor latency as well as and the
- 6:37throughput in favor of the quality
- 6:39because you don't need an expensive
- 6:41model for doing the autocompletes for
- 6:43you. But whichever corner you give up
- 6:46there is still a maximum load you can
- 6:49carry while hitting your SLO
- 6:52your service level objective. Let's say
- 6:54that on this deployment it's eight
- 6:57requests at once. In a normal minute,
- 7:00you never notice that limit. Now,
- 7:03picture the 40 requests arriving in the
- 7:05same two seconds.
- 7:08It's such a high load.
- 7:15And this is not like hypothetical. It
- 7:17has happened to like biggest providers
- 7:19in the world. Here are the two public
- 7:21postmortms two years apart.
- 7:24On November 8, 2023, OpenAI after a day
- 7:27of unusual heavy traffic, maybe not 40
- 7:31requests, it would be some million
- 7:33requests, their routing servers ran out
- 7:36of the memory. They started failing
- 7:37their health checks. They added more
- 7:40replicas, then had to roll them back.
- 7:42The way out was to cut incoming traffic
- 7:45and ramp it up slowly. In January 21
- 7:502026 enthropic the sonnet 4.5 tire gear
- 7:55got slow first then it started returning
- 7:59errors over about 90 minutes. The cause
- 8:02they gave was simple not enough compute
- 8:04for the demand.
- 8:07And now let's look at what each team
- 8:10decided as their next set of the action
- 8:12item. Neither of them says we need a
- 8:15faster engine.
- 8:17Open AI's followup is graceful load
- 8:20shedding. Enthropics is detecting
- 8:24capacity limits earlier.
- 8:26It's the same idea in the different
- 8:29words which is decide on your request
- 8:32sooner
- 8:34to make sure which request gets in. And
- 8:38there are more incidents which we have
- 8:41linked in the appendix. If you are
- 8:43interested definitely go and read
- 8:44through them.
- 8:49So
- 8:52if you walk backward from those
- 8:54follow-ups,
- 8:55they both point at the same place, the
- 8:57front door. Here is the pipeline again
- 9:01with the missing step filled in.
- 9:05We have a client, we have an admit
- 9:08layer, the Q router, and then the
- 9:10engine.
- 9:12It's never the client talking directly
- 9:14to your engine or client talking to your
- 9:17queue directly. There is an admission
- 9:20phase in between.
- 9:22Admission is a decision made before a
- 9:25request takes any GPU memory and it only
- 9:28has three possible answers.
- 9:33Admission decides
- 9:36whether to start now or you want to
- 9:42hold the request. You want to accept it
- 9:44but make it wait. Third is the reject.
- 9:47That means not right now. It can throw
- 9:50different amount of errors here such as
- 9:52429s and we will dive into this bit more
- 9:56deeper in the second part.
- 10:00this that is the decision that was
- 10:01missing from our pipeline.
- 10:06So why does it have to happen before the
- 10:09engine
- 10:11and what would happen if you will
- 10:13actually accept the request what would
- 10:16be the cost of that yes
- 10:19so let's try to take a look at this like
- 10:22two phases of the inference like prefill
- 10:24and the decode. So first comes the
- 10:26prefill. That's the engine reading the W
- 10:29prompt, building the request KV cache.
- 10:31And then comes the decode. That's the
- 10:33engine writing the answer one token at a
- 10:35time. With every token, the cash grows a
- 10:38little more. All of that memory stays
- 10:40reserved until the request finishes.
- 10:45So a yes is not a moment. It's a
- 10:48commitment that lasts the request whole
- 10:50life. And here you can look at the
- 10:53example.
- 10:55There is a request A, there is a request
- 10:57B. Once you admit a request A which is a
- 11:00short answer request and then you admit
- 11:02a request B which is a long answer
- 11:04request. Now when you admit a request C
- 11:06which is again a short answer request,
- 11:09we don't know how longer it would take
- 11:11because there is a longer request which
- 11:13was already in the memory. Should we
- 11:16even take that request C or should we
- 11:18not even take the request C. So there is
- 11:20a cost associated of like admitting
- 11:23every request because admission could
- 11:26still result in missing out your SLOs's.
- 11:34Another impact apart from the SLOs's is
- 11:37your memory.
- 11:40Here is what it does to your GPU. So
- 11:43every running request it keeps growing
- 11:45your cache as it writes. So your total
- 11:47memory keeps climbing. When the memory
- 11:49gets to the full engine has three ways
- 11:51to cut. Either it will aait
- 11:54or like throw away your request. Either
- 11:57it will put it to your waiting state
- 11:59that means it will swap or it will just
- 12:03say okay I won't be able to serve and I
- 12:05I will probably return you an error.
- 12:09Every one of those throws I mean every
- 12:12one of those causes a certain delay in
- 12:16processing of the request. and missing
- 12:19out your SLOs's again while also wasting
- 12:21your memory.
- 12:30So why is it like our problem like we
- 12:33have con servers from open AI anthropic
- 12:37bold teams which are serving their own
- 12:39infrastructure
- 12:41for the inference.
- 12:44So it's a problem for us because
- 12:48there are teams who want privacy. They
- 12:51want to have their own models hosted on
- 12:54their own GPUs
- 12:57and they want it to be hosted on their
- 12:59own GPUs due to the cost perspectives
- 13:02but they have to also protect
- 13:06the burndown on those GPUs.
- 13:08So they need to effectively decide which
- 13:11request to serve across their inference
- 13:14ecosystem.
- 13:22On top of
- 13:24and on top of that like the traffic
- 13:26arriving at the door has also changed.
- 13:30Earlier there used to be a person that
- 13:32was chatting. He would send one request
- 13:34at a time. There would be a seconds of
- 13:36reading in between and then there would
- 13:38be a slow answer. But now coding agents
- 13:40they have taken over. There are tons of
- 13:42calls that these coding agents they send
- 13:45along with the retries, pauses,
- 13:49um
- 13:51like the undo calls as well and it there
- 13:54is such a overwhelming traffic that your
- 13:57cluster has to handle and what that
- 14:00means is cluster has a responsibility to
- 14:03take a wise decision of which requests
- 14:05to accept and which uh requests to
- 14:08reject.
- 14:15And this is exactly that visualization.
- 14:17You ask your agent that fix an o test.
- 14:21It runs all your it tries to run
- 14:24multiple tool calls that is reading of
- 14:27the files like around there could be
- 14:29like 14 files. There could be like
- 14:32around 10 tests that it has to run and
- 14:35it would combine together to make 40 50
- 14:38calls for just one request and all these
- 14:4240 requests they require like reaching
- 14:44out to the LLM getting back the
- 14:46response. Uh and in between some of
- 14:50these requests would be long running
- 14:52some of the requests would be short
- 14:55prompt request. So we need to decide
- 14:57effectively where which one to pick and
- 15:00which one not to pick.
- 15:09Yeah. And that was the like reason why
- 15:12we need to worry about
- 15:15admission control. Why we need to worry
- 15:18about
- 15:21handling the traffic right way across
- 15:23our infancy ecosystem.
- 15:32Okay. So this brings us to the admission
- 15:36control.
- 15:37Uh now moving on to the next slide. Uh
- 15:42that is slide number 18. So in this
- 15:45slide we will be covering three
- 15:47questions. Where does this decision
- 15:50from? which request to admit
- 15:52and which to reject sits back at your
- 15:55stack
- 15:57and the another question is what does it
- 16:00need to see to make a good call or
- 16:03judgment on the request and what does it
- 16:07returns back to the client. Moving on to
- 16:10the next slide. So here uh we see the
- 16:15pipeline again which each layer's jobs
- 16:17is written out. The agent sees its own
- 16:21call and decides when to retry.
- 16:25The gateway sees API key and request
- 16:28count and enforces a rate limit rate
- 16:31limiter for each and the router which
- 16:35sees replicas or healthy and which one
- 16:37to pick. Now the engine sees its KV
- 16:41cache and the current batch and decides
- 16:44the order of request to run and what to
- 16:47evict and print.
- 16:50So every layer decides something. Now
- 16:54ask each of them one question. Is the
- 16:57room for this request?
- 17:00The agent says not my job. The gateway
- 17:03says I cannot see it. And the router say
- 17:07not my job and the engine says I only
- 17:09find out when I have already said yes.
- 17:13The admission box is missing in in
- 17:17between.
- 17:20So now moving on to the next slide. Um
- 17:23so where to put this decision change
- 17:28what it can see and what it cost you to
- 17:31run. So there are three possible
- 17:34moments. The first is an invoice right
- 17:38in front of the engine. Here the policy
- 17:41sees the request itself plus whatever
- 17:43metrics the engine exposes. All you need
- 17:47is proxy or gateway. The engine stays
- 17:50exactly as it is. The second one is the
- 17:53Q boundary. Now the decision happens
- 17:56inside the engineuler
- 17:58as it picks the next batch to run.
- 18:03The third is during decoding when the
- 18:06scheduleuler can pause a request or take
- 18:08back its cache halfway through an
- 18:11answer. Those later movement can see
- 18:14more but they mean changing the
- 18:16engineuler.
- 18:18That means keeping your own modified
- 18:20copy of BLLM or SG lang and updating it
- 18:24every time original changes forever. So
- 18:28an ingress is the only movement that
- 18:30works with the engine exactly as it
- 18:33ships.
- 18:34So that's where we will stay for the
- 18:36rest of the talk. So now moving on to
- 18:39the next slide.
- 18:42Uh so what actually sits in ingress,
- 18:46right? Uh so you you you can think of it
- 18:49as a function. It can take in what it
- 18:52knows about the request and current
- 18:54state of the system and it returns one
- 18:58of the three answers
- 19:01like if we have to admit the request. So
- 19:04the admit is the request enters the
- 19:07engine now and memory is committed for
- 19:09this moment. If it's deferred with a
- 19:13time we can call toao hold on like uh
- 19:16hold on and come back after some time
- 19:19and there is another decision which is
- 19:22with uh reject with a code that no here
- 19:26is the reason why and until the policy
- 19:29says admit the request cost you nothing.
- 19:32It holds no GPU memory at all.
- 19:37Now moving on to the next slide.
- 19:43uh to make the call well the policy
- 19:46needs two views at the same time from
- 19:49the request and from the engine. So from
- 19:53the request how long the input is, which
- 19:56client or tenant sends it, which agent
- 19:59session it belongs to, and a deadline.
- 20:03And from the engine point of view, how
- 20:06long uh KV cache is, how long the oldest
- 20:10request has been waiting in the queue,
- 20:12how often the prompt is already cached,
- 20:15and how how loaded each pool of replica
- 20:19is. Now look back at the ST. The gateway
- 20:22has left side and the engine has the
- 20:26right side. Nobody has the both.
- 20:30And that's really why admission has to
- 20:33be its own layer. Now moving on to the
- 20:36next slide.
- 20:39Here are those eight signals turned into
- 20:41simple rule. Each one is compared to
- 20:44threshold and when it crosses it pushes
- 20:47towards the outcome in the colored pale
- 20:51from the request side.
- 20:53If the input is too big to fit in the
- 20:56memory headroom just refer it. If a
- 20:59client or tenant is over
- 21:02it share reject it with 429.
- 21:06If one agent session has too many calls
- 21:09in flight defer with extras. If waiting
- 21:13would make a request miss its deadline
- 21:15anyway reject it from the engine side.
- 21:19Uh you can see at the bottom half of
- 21:23bottom half side. While KB cache is
- 21:26under high water mark admit it. When the
- 21:31oldest request has waited longer than
- 21:33your target defer it. If the prompt is
- 21:36already cached admit it first because
- 21:39it's cheap to serve. And when every pool
- 21:41is full, reject it with 529. One rule of
- 21:45thumb, pick the signal that changes in
- 21:48second, like a QH, not a one minute
- 21:52average. A coding agent sends 40 calls
- 21:55in 2 seconds.
- 21:57A minute average won't even notice. So
- 22:00that's what the policy reads. Now what
- 22:03does it say back? Now moving on to the
- 22:06next slide.
- 22:08We will take the three answers one at a
- 22:11time and look at each from both sides.
- 22:15What the server sends and what your
- 22:17agent should do with it. First, yes, the
- 22:21server sends back a 200 and the response
- 22:24starts streaming. At that moment, three
- 22:26things happens inside the engine. It
- 22:29commits KV cache from the whole prompt
- 22:32and the request joins the running batch
- 22:34and decode one token per step alongside
- 22:37everyone else
- 22:40and it holds that memory until it
- 22:42finishes or until the client
- 22:44disconnects.
- 22:46That's the commitment from part one just
- 22:49in the form of an HTTP response which is
- 22:52why the admission decision has to happen
- 22:54before 200 never after it.
- 22:59Now the client side if we are writing
- 23:02the agent here is how to handle 200.
- 23:06Watch the stream not the status code.
- 23:10First time is properly start your
- 23:13timeout clock when the first token
- 23:15arrives not when you send the request
- 23:18under the heavy load prefill alone can
- 23:21take few seconds. Second, if you decide
- 23:24to give up, actually close the
- 23:26connection. The server only frees the KV
- 23:29cache when the request finishes or the
- 23:31client disconnects. An agent that times
- 23:34out but leaves the connection open is
- 23:36still holding the GPU memory. And the
- 23:39third one, uh, a stream can fail halfway
- 23:43through, right? And that that's no
- 23:46status for that, right? and you're
- 23:49you're left with partial answer and you
- 23:51have to decide whether to retry or carry
- 23:53on. In short, 200 promise to start. It's
- 23:58not a promise to finish. Right?
- 24:03The second answer is dipper which
- 24:05basically means not yet. The server
- 24:07sends back 503 with retry header.
- 24:11The header tells the client how many
- 24:13seconds to wait. The policy picks that
- 24:15number from the engine. For example, how
- 24:18long until the running batch finishes
- 24:20and freeze the enough cash
- 24:22and you differ when capacity is coming
- 24:25back soon? It's a burst, not so not a
- 24:27trend, right? And for example, like if
- 24:30you think about it like Q age is rising
- 24:34but the system isn't saturated.
- 24:39If the capacity isn't coming back
- 24:41anytime soon, a dipper just kicks kicks
- 24:44the can down the road, right? And in
- 24:48that case, we can just reject it with
- 24:50the honest answer.
- 24:53Moving on to the next slide. On the
- 24:55agent side, handling a defer is simple.
- 24:59Sleep for retry after time and a little
- 25:02random jitter, then retry again. The
- 25:05jitter just means a small random extra
- 25:07delay. So client don't like don't call
- 25:11or like don't at all come back at the
- 25:14exact same moment. They will have to
- 25:16wait.
- 25:19Look what the server sees from 10
- 25:21retries without retry after every agent
- 25:25picks its own delay. And that's usually
- 25:29that's usually zero. So they all come
- 25:32back at once. That's how 40 calls turns
- 25:35into 80.
- 25:38With retry after and jitter the retry
- 25:41spreads out across the window serves
- 25:43asked for.
- 25:50Okay. So if if we see the uh reject
- 25:55reject returns 429 or 529 the third
- 25:59answer is no.
- 26:01Uh and there there are two different
- 26:03codes for it because there are two
- 26:05different reasons. A 429s means too many
- 26:08requests.
- 26:10This particular client or agent session
- 26:13is using more than its share. The policy
- 26:16saw something like per client load, the
- 26:18number of concurrent calls or a kota.
- 26:21The problem is the caller and then the
- 26:24529 means the provider is overloaded.
- 26:28Enthropic uses this code. Every replica
- 26:31is sold. So nobody is getting in right
- 26:34now.
- 26:36The problem is pool not the caller.
- 26:40Now uh handling a reject. So how should
- 26:44agent react on 429? Reduce your own
- 26:48load. Cap how many calls that agent has
- 26:51in flight at once.
- 26:54And on 529 retrying harder only makes
- 26:57this worse. Waiting longer switching to
- 27:00different model provider or park the
- 27:02task for later. and one either and on
- 27:07either one keep a retry budget. Use
- 27:09expansion exponential back off with
- 27:12jitter which means waiting longer after
- 27:15each failed try and after few tries tell
- 27:19the user what's going on because a
- 27:21reject that gets retrieded at full speed
- 27:24is how you end up in one of the postmodm
- 27:26from the talk at we described at the
- 27:29starting.
- 27:33Now over to you Hershel
- 27:40three.
- 27:41Thank you P. Uh so this is the module
- 27:44three where we will try to learn about
- 27:46the different strategies uh that have
- 27:48been rolled out for the admission
- 27:50control. So far we have seen that no
- 27:52layer in the today's stack asks where uh
- 27:55whether there is a room and that layer
- 27:58is ours to add and now we give it a
- 28:01brain. So we will look at why counting
- 28:03requests isn't enough. Breaking a policy
- 28:06down into four different choices. See
- 28:08what four different research paper chose
- 28:12and finish with a decision tree that you
- 28:15can use onto your own ecosystem.
- 28:20So let's start with what most team
- 28:22already have which is counting. Yes.
- 28:26Counting the number of the requests.
- 28:30A simple idea is to rate limit at the
- 28:32gateway which gives API key a fixed
- 28:35allowance set in advance. It's great for
- 28:38keeping the clients fair, keeping your
- 28:40tenants fair with each other,
- 28:43but it does nothing to protect your GPUs
- 28:47because it doesn't change with the load
- 28:50that is happening onto your GPUs or your
- 28:52cluster. A fixed cap at the proxy gets
- 28:57closer. It counts how many requests are
- 29:00in flight on each replica. The problem
- 29:03is that the quick crep and a long
- 29:05planning to call
- 29:08count exactly the same while the real
- 29:10capacity of the GPU changes with every
- 29:12batch. So in practice it mostly cues and
- 29:17clients timeout ends up doing the
- 29:20refusing.
- 29:22Neither one knows whether our 40 calls
- 29:25are cheap or expensive.
- 29:27The cost of air policy does though and
- 29:31it estimates each request will actually
- 29:35cost what each request will cost in the
- 29:39KV memory and the tokens using the live
- 29:42state from your engine. And it says no
- 29:45before the request actually takes up the
- 29:47memory. So the short version is count is
- 29:50not a cost.
- 29:55How do you build your own policy? So
- 29:57there are four choices here. The first
- 30:00is the unit
- 30:02which tells you what you are actually
- 30:04admitting. Are you admitting a single
- 30:06request? Are you admitting an agent
- 30:09session or are you admitting a tenant?
- 30:13The two second is the signal
- 30:17which means what is that the policy is
- 30:20watching for taking an admission
- 30:22decision. This could be as simple as
- 30:24your memory pressure which is your KB
- 30:27pressure. It could be your Q depth. It
- 30:29could be your deadline slack. That means
- 30:32whether your request is able to meet
- 30:34your SLOs's or not.
- 30:37And whatever it is, it has to actually
- 30:39move when the system gets busy. So three
- 30:43is the rule which means how does the
- 30:46signal gets turned into a decision.
- 30:50So rule is used for that and that can be
- 30:54anything from a simple threshold that
- 30:56takes microscond to a full optimizer
- 30:58that takes seconds.
- 31:01Fourth is the action. Once you have the
- 31:04rule that gives you a decision what do
- 31:06you do with it?
- 31:09What does your how does your client gets
- 31:11impacted through those actions?
- 31:14That's the admit defer and the reject
- 31:17that per just talked about in the module
- 31:20two. Each choice limits the next one.
- 31:24For example, a deadline signal only
- 31:26works if your request actually carry the
- 31:28deadlines. So let's go through the first
- 31:31three.
- 31:37Choice one. Do you admit a single
- 31:39request or do you admit a poll sessions?
- 31:44Say you have two agent task 20 calls
- 31:46each room for almost half of them.
- 31:50You can uh point you can see the top
- 31:54row. So if you admit per request about
- 31:57half of each task gets in but neither
- 32:00task can finish because each one is
- 32:03stuck waiting on the calls that didn't
- 32:05make it. You have used all the capacity
- 32:08and completed nothing. If you admit per
- 32:12session, session A gets all of its call
- 32:15and finishes. Session B gets one clear
- 32:17answer. Come back later. And this
- 32:20picture is illustrative, but the logic
- 32:22holds.
- 32:24Our 40 calls are one session. Coding
- 32:27agents send sessions and yet most
- 32:30published policies admit only the single
- 32:33requests.
- 32:38So which metric do you watch for what
- 32:40particular problem?
- 32:44This is the
- 32:46question that we are trying to answer.
- 32:49It depends on the problem that you are
- 32:51trying to prevent. If you are if you
- 32:54want to keep the cash from overflowing,
- 32:57you definitely have to watch your KV
- 32:59pressure.
- 33:02If you want to avoid your wasted
- 33:04recomputee, you have to watch out
- 33:07the KV used together with the cash hit
- 33:10rate. A full cache is fine if it's being
- 33:12reused.
- 33:15If you want to keep your weights to be
- 33:18shorter, you have to watch for your
- 33:20queue depth and you have to take actions
- 33:22based on that. If you want to meet your
- 33:25latency targets, you need to watch your
- 33:27deadline slack. Take actions based on
- 33:30that. If you need to plan for your long
- 33:32answers, you need to predict for the
- 33:34output length.
- 33:37And look at the last column
- 33:40fails when it tells you when each signal
- 33:44stops working. Look at the first row. KV
- 33:48pressure fails when the cache never
- 33:50fills up.
- 33:52Remember that row? It comes back to bite
- 33:54someone in the part four.
- 34:01So what are the four ways to take that
- 34:03decision?
- 34:07That is how do you convert your signal
- 34:09into a decision from the cheapest to the
- 34:12most expensive. Here are the ways a
- 34:16threshold just compares one number to
- 34:18the one limit. Example is a policy named
- 34:23clued weight that was uh created
- 34:27somewhere in 2026.
- 34:29It takes tens of microsconds per
- 34:31decision.
- 34:34Second is a dual gate which needs two
- 34:37signals to agree before it acts. Example
- 34:40is a paper called concour
- 34:43uh which checks like KVUs and the cash
- 34:45hits under 100 microsconds.
- 34:49Third is an optimizer which solves like
- 34:51one big planning problem across all the
- 34:54cues at once. Example is a QLM.
- 34:57It's thorough but it takes about 5
- 34:59seconds per decision.
- 35:02And the fourth is the formal bound uh
- 35:05which does the math on the worst case.
- 35:08Can this request still meets its
- 35:10deadline?
- 35:12The example is the Kronos on which uses
- 35:16the like time to first token and it uh
- 35:19provides you like a microcond level
- 35:21latency.
- 35:23And now hold that cost column up against
- 35:26our agent. It sends 40 calls in 2
- 35:28seconds. A 5-second decision shows up
- 35:31after every party is over.
- 35:38So what the four papers did
- 35:42as the complete policy fluid weight
- 35:44admits single request uses the threshold
- 35:46on your KV use and reports 10 to 25%
- 35:50improvement. Conqueror is the one that
- 35:53admits your VO agents. It uses your dual
- 35:56gate and reports four to 4x
- 36:00uh the throughput of the SG lang but it
- 36:02was tested offline.
- 36:05Uh QLM admits the request plans across
- 36:09the W feed and reports 40 to 90% gains
- 36:13at about 5 seconds per decision. Kronos
- 36:16admits requests against their deadline
- 36:18slack with a formal bound and takes
- 36:21deadline misses from 99% down to zero.
- 36:25Definitely very amazing accomplishments.
- 36:29So each purper used a different model
- 36:32engine and a workload. So please don't
- 36:34read this as a leaderboard.
- 36:37The question that matters for this room
- 36:39is does any of them fit a coding agent
- 36:45and the answer is no. We placed every
- 36:49paper on the two axises. One axis is
- 36:52what it optimizes deadlines or the
- 36:55throughputs. Other is what it admits
- 36:57single request or the VO agent sessions.
- 37:01A coding agent needs a policy that
- 37:03admits bold sessions, works online with
- 37:06live traffic and runs engine you haven't
- 37:08fought
- 37:11and nothing sits there cleanly. Conc
- 37:13doesn't need a for but it was only
- 37:16tested offline.
- 37:18So in practice you will pick the closest
- 37:20match and build just build your own. And
- 37:23here is how you can choose which polic
- 37:27like which kind of a policy you want to
- 37:29try it out.
- 37:31So here is that decision tree.
- 37:34First
- 37:36the question that you have to answer is
- 37:38does your request carry any deadlines?
- 37:40If yes, set up a some sort of a formal
- 37:44bound on that deadline. This will take
- 37:46some microsconds level of latency.
- 37:50If there is no deadline requirement, the
- 37:53second question that you can ask is does
- 37:56the KV cache fill at the peak load? If
- 38:00your KV cache fills at the peak load,
- 38:05then
- 38:07you would want something like a way to
- 38:11detect the cache prefixes used. So you
- 38:14can use some sort of a dual gate
- 38:15mechanism or a KV threshold mechanism.
- 38:19Um and then if your KV cache gets filled
- 38:24at the peak load, it does not gets
- 38:27filled at the peak load, then you what
- 38:28you could do is you can set up a Q level
- 38:30of threshold on it.
- 38:35And that gets us to the benchmarking
- 38:38like once you have a policy, how do you
- 38:41benchmark?
- 38:43So I will be handing it over to pop.
- 38:50Thank you. So now moving on to part four
- 38:53which is benchmarking. As we just saw no
- 38:57publish policy fit a coding agent
- 39:00cleanly. So you are going to adopt one
- 39:02or you you will have to build your own
- 39:05and then you have to prove if it's
- 39:07actually working or not. So to now
- 39:11moving on to the next slide
- 39:14you would have to test your policies
- 39:16right so for that you need four things
- 39:19first is your cluster your policy and
- 39:23trace and score. So let's dig deeper
- 39:26into each. So your cluster of the first
- 39:29measure it it is capacity with no
- 39:33admission control at all. The load your
- 39:35replica can sustain on their own. Every
- 39:38load level at after this is described as
- 39:42multiple of your number like multiple of
- 39:45that number and uh about the policy u
- 39:49which signals it reads what limit it
- 39:52uses and what it does when it says no.
- 39:56The third one is trace which is a record
- 39:58of real traffic that you replay using
- 40:02code coding agent traffic ramping from
- 40:04half your capacity up to twice your
- 40:08capacity.
- 40:10Four is a score. Then they run the same
- 40:13trace twice. Once with the policy and
- 40:16once without and then you can compare
- 40:19your result. The score is where the most
- 40:22people go wrong. So let's look look at
- 40:25that. Now moving on to the next slide
- 40:28which is uh how how you score. The score
- 40:32we recommend is called goodput under
- 40:35admission. So you can see the formula on
- 40:38the screen. The top of the fraction is
- 40:41the request that met their SLO their
- 40:44latency target. That means the answer
- 40:47started on time and it never stalled for
- 40:50too long between tokens.
- 40:53The bottom of the fraction is every
- 40:56request that arrived including the one
- 40:59which got rejected. Why does the bottom
- 41:02matter so much?
- 41:05So try this. Uh refuse 95% of your
- 41:09traffic and serve the remaining 5%
- 41:11perfectly. If you only count what you
- 41:15admitted, your SLO attainment is 100%.
- 41:19Looks amazing, right? Just ship it.
- 41:22But no like good put under admission
- 41:25says 5%. Shedding looks three when you
- 41:29only account the request you kept. It
- 41:33isn't.
- 41:35So moving on to slide 42.
- 41:39The second thing to get right is the
- 41:42workload. A benchmark only tell you
- 41:44something if it looks like your real
- 41:47traffic.
- 41:49So replay agent traces with their fan
- 41:52out retries session not uniform random
- 41:56arrival.
- 41:57Describe load as multiple of the
- 42:00capacity you measured and sweep from
- 42:03half of it to double it.
- 42:08Set your guideline as multiple of your
- 42:11latency with no load. And remember a
- 42:14benchmark that never goes past capacity
- 42:17never actually tests the policy because
- 42:19the policy never has to say no. So you
- 42:22have to make sure that you reach at that
- 42:25point.
- 42:27Now
- 42:29move moving on to the seven uh seven
- 42:32things every admission result should
- 42:34disclose.
- 42:36Signal liess is one of the seven things
- 42:39every admission result should disclose.
- 42:42Here is the full full list that you can
- 42:45see on your screen.
- 42:47The load as a multiple of measured
- 42:50capacity
- 42:51deadline sets relative to latency with
- 42:54no load.
- 42:56Signal liveless which is no paper
- 42:59report.
- 43:01How many times you repeated the run and
- 43:04how much the result varied not one lucky
- 43:07run. Where each parameters comes from,
- 43:10turned on this setup, borrowed from
- 43:12somewhere else or left at the default.
- 43:16Whether the rejected request count in a
- 43:19good put denominator and the full
- 43:22configuration
- 43:25and the full configuration is model,
- 43:28engine, version, hardware, workload,
- 43:31SLOs's and baseline. None of this ask
- 43:35anyone to change how they evaluate. It
- 43:38just ask them to say what they did. Use
- 43:41it as a checklist when you read on on a
- 43:44paper and when your benchmark and when
- 43:47you benchmark your own policy. Doing all
- 43:51seven by hand is painful. So we built a
- 43:54tool that records all of this uh as as a
- 43:58default.
- 44:00So uh the tool is admin perf and it's it
- 44:05it's two things in one repository. So in
- 44:09production if you see it's a library you
- 44:13put in front of VLLM or SG lang it
- 44:16decides admit defer or reject for every
- 44:20request that's ingress layer from part
- 44:23two.
- 44:26So if you also see we on the right side
- 44:31uh it's uh in in a benchmark it's a
- 44:34harness that runs any policy against a
- 44:36real engine on a real GPU and reports
- 44:40what the policy cost
- 44:43both sides calls the same function
- 44:45policy decide. So the policy you
- 44:49benchmark is exactly the policy you
- 44:51deploy.
- 44:53We build it because across those 16
- 44:56paper no two share the same baseline
- 45:00engine workload or SLO admitf makes a
- 45:04head-to-head comparison possible.
- 45:07Now moving on to the next slide.
- 45:11So using its pive command. So you can
- 45:15see on the screen we have five commands.
- 45:18So infra up starts a real VLM on a real
- 45:22GPU. Infra smoke checks that it's
- 45:25actually serving. Then we have bench
- 45:27runs every policy several times each.
- 45:31Then we have bench computer tell you
- 45:34which policy won and how by how much.
- 45:37And then we have infraown which which
- 45:40basically means it should shuts all all
- 45:43off so you stop paying. And that's my
- 45:46favorite one.
- 45:48Uh and so now provisioning and
- 45:50benchmarking are separate commands on
- 45:52purpose. Loading a model takes minutes
- 45:55and every policy has to face exactly
- 45:58same deployment.
- 46:00If you are reprovisioned between policy,
- 46:02you would change the very thing you are
- 46:05trying to hold a constant.
- 46:08The deployment is constant. The policy
- 46:10is variable.
- 46:15So now uh here a policy must declare the
- 46:19signal it reads. This is how admit perf
- 46:22deals with the trap I just showed you.
- 46:25Every policy has to declare the signal
- 46:27it needs. This KV cache threshold policy
- 46:31says I require KV used correction which
- 46:35is the share of the KV cache in use.
- 46:40If the engine cannot report that signal,
- 46:42the run stops at the startup with an
- 46:45error. You can actually read without the
- 46:48check a missing signal reads as a zero
- 46:51and KV cache threshold that always sees
- 46:54zero admits everything. It's quietly
- 46:57turned into the no admit baseline while
- 47:01your results looks completely normal. So
- 47:04policy that cannot see its signal should
- 47:07fail loudly and not just fail silently.
- 47:14Um so here
- 47:17what we did is every decision is
- 47:20recorded with the state it saw. Every
- 47:22run leaves behind a folder you can open
- 47:25inspect. There would be four files as
- 47:27you can see on the screen. Those are
- 47:30manifest JSON summary JSON decision JSON
- 47:33and uh out outcome JSON. So
- 47:39what each does
- 47:41is
- 47:42uh you can take a look deeply once you
- 47:46run the results. Now we we will be
- 47:48looking at the shading cut. So shade
- 47:51shading 37% cut tail latency. So here
- 47:55are the four results how they will look
- 47:57like on a real GPU. The setup small
- 48:00model we used quen 2.5 uh and it's a
- 48:04half billion
- 48:06model on a A10 GPU. The engine was kept
- 48:10out at four sequ sequences at a time.
- 48:14Eight request at 14 per second two
- 48:17repeats.
- 48:20If you look at the first row with no
- 48:23admission control, the P95 time to first
- 48:27token was about two two and two and a
- 48:29half around 2 and a half seconds and P
- 48:3245 sorry P95 means 95% of the requests
- 48:37were faster than that and it swung by
- 48:40about 700 milliseconds either way.
- 48:44Now the last row if you see with a queue
- 48:47up policy that allows at most two
- 48:50request waiting 60 63% of requests were
- 48:54admitted and the P95 dropped to about
- 48:58around 1 second that's 2.5 times better
- 49:03with the exact the good same put
- 49:07now if we look at the signal matter more
- 49:10than the threshold
- 49:12but this This is the result I love most.
- 49:15Same run. KV cache usage peaked at half
- 49:18a percent. The queue got 24 requests
- 49:21deep on a small model. KV cache is huge
- 49:25compared to the four short sequences. So
- 49:28it never comes close to filling. All of
- 49:31the pressure showed up in the queue.
- 49:33Instead, a KV cache pressure policy
- 49:35would have botched a flat line and
- 49:38admitted everything. that that's the row
- 49:42of choice from two. So
- 49:48now here and you you can try it today.
- 49:50You don't need a GPU. There is a small
- 49:53fake engine that speaks the three
- 49:55endpoints admit puff uses and it gets
- 49:58busy under load just like a real one. So
- 50:02you can test the whole setup on your
- 50:04laptop. If your right policy is your own
- 50:06package,
- 50:08admit perf discovers it automatically.
- 50:11No change to the repo and contribute
- 50:13contributions are very welcome. New
- 50:16policies, workload, engine adapter or
- 50:19matrices.
- 50:25Okay.
- 50:29So if you want to remember like one
- 50:32slide just make it this one that
- 50:37when you are accepting the request in
- 50:40your inference clusters
- 50:42you need to have a mechanism to say no
- 50:45early something that works at the
- 50:47ingress something before saying yes to
- 50:51the request before that request goes
- 50:54into the GPU before that request takes
- 50:57up your memory.
- 51:00and per added clearly. There are a
- 51:03couple of error codes that your
- 51:06admission policies return. You need to
- 51:08handle those requests and those errors.
- 51:15And with that we would like to come to
- 51:18the closure. We would like to thank you
- 51:20all for your patience, for your time.
- 51:24And here are like two of our projects.
- 51:28It all started as part of the LLM
- 51:30inference at scale handbook. We and when
- 51:33we were doing the survey, we found the
- 51:35gaps and we prepared admit puff. So here
- 51:39is the source code for it. Feel free to
- 51:42go to it and like experiment with it.
- 51:46Raise the issues, contribute to it. idea
- 51:49is to
- 51:52build your admission policies, benchmark
- 51:54them and be confident that it uh helps
- 51:58you keep your cluster stable. Yeah.
- 52:01>> Thank you everyone.
- 52:02>> Thank you everyone.
About this transcript
This page contains the full transcript of Your Self-Hosted LLM Needs to Say No - Harshul Jain & Parv Khatri, Audible & SAP by AI Engineer, generated from the public captions YouTube serves with the video. The transcript has 6,829 words across 1,045 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.