YouTube2Text

Your Self-Hosted LLM Needs to Say No - Harshul Jain & Parv Khatri, Audible & SAP — Transcript

by AI Engineer · 6,829 words · 1,045 segments · language en · Watch on YouTube

Full transcript

  1. 0:05Good morning everyone. My name is
  2. 0:07Hershel Jan and I have here with me
  3. 0:10Pearl. Today we are going to talk about
  4. 0:13saying no. It's called as admission
  5. 0:16control. How our system decides which
  6. 0:20request to get in when your coding
  7. 0:22agents overwhelm the LLMs you host
  8. 0:25yourself.
  9. 0:27Uh before we move on a quick intro about
  10. 0:30us.
  11. 0:32So I am a senior software engineer at
  12. 0:34Audible where I lead some of the MLAI
  13. 0:37infrastructure.
  14. 0:38Uh I am also writing an open-source
  15. 0:40handbook called as LLM inference at
  16. 0:43scale. It has received over 300 plus
  17. 0:45stars by now.
  18. 0:47I am also maintaining an open-source
  19. 0:49framework called agentship which helps
  20. 0:51you ship reliable and secure agentic
  21. 0:55harnesses in an ultra fast speed. Uh
  22. 0:59perv would you like to introduce
  23. 1:00yourself?
  24. 1:01>> Sure. Hi everyone. I'm Perv. I'm a
  25. 1:04senior software developer at SAP. I work
  26. 1:06on like cloud application and LLM
  27. 1:09infrastructure. I wrote an LLM fine-tune
  28. 1:12guide. I contribute to agentship and I'm
  29. 1:15like building a task tracker which is a
  30. 1:18local post task board for you and your
  31. 1:20agent.
  32. 1:22Uh apart from this if you would like to
  33. 1:24connect with us our LinkedIn QR codes
  34. 1:27are right there on this slide. Now back
  35. 1:29to you Hershel.
  36. 1:32Great. So
  37. 1:35here is the plan. We are going to start
  38. 1:37with understanding the motivation why
  39. 1:41any serving stack can admit so much of a
  40. 1:45traffic.
  41. 1:46Second, we try to dive into admission
  42. 1:48control where the decision to say no
  43. 1:51lives and what it needs to see. Third,
  44. 1:55we will go through some of the proven
  45. 1:57strategies, the different ways a system
  46. 2:00can actually make the admission
  47. 2:01decision. And fourth, we will try to
  48. 2:05dive into the benchmarking where you
  49. 2:07will learn how to prove that your policy
  50. 2:11for admission is working or it's not
  51. 2:13working.
  52. 2:16So let's get started with the
  53. 2:18motivation. Um
  54. 2:21and for this
  55. 2:26everything today uh let's try to
  56. 2:28understand what is inference. Everything
  57. 2:31today happens during this inference
  58. 2:34itself. So inference is simply a moment
  59. 2:37when someone sends a prompt to the model
  60. 2:40and gets back an answer.
  61. 2:43So the step one is like a user prompt
  62. 2:45comes in. It's a sequence of words. Uh
  63. 2:48for example, it could be like failing a
  64. 2:51fix a failing test. The model which is
  65. 2:55running on your GPUs writes the answer
  66. 2:57one token at a time. A token is roughly
  67. 3:00like a word as I mentioned. Uh it writes
  68. 3:03the first word then uses that word to
  69. 3:06pick the second uh then it uses both to
  70. 3:10pick the third and it keeps on
  71. 3:12continuing like that and it cannot write
  72. 3:16all of them at once because every word
  73. 3:18depends on the previous one. So those
  74. 3:22once all the words are generated they
  75. 3:24stream back to the user uh who ask for
  76. 3:28the result and that volt prompt and the
  77. 3:32answer out is like that one request. Now
  78. 3:34I want you to hold on to one small
  79. 3:37detail like model has no idea how long
  80. 3:41this answer is going to be. It only
  81. 3:44finds out whether it writes when it only
  82. 3:49finds out when it writes the first last
  83. 3:51token.
  84. 3:53Keep that in mind. It will matter in the
  85. 3:55few minutes.
  86. 4:01So a model never sits on the internet
  87. 4:05itself. Every request travels through a
  88. 4:08short pipeline. It starts in your app or
  89. 4:12your agent. Then it reaches your API
  90. 4:15gateway
  91. 4:16which checks your API key counts how
  92. 4:18many requests you are sending. Then a
  93. 4:21router picks a replica which which is
  94. 4:23just one running copy of the model and
  95. 4:26finally the requests land on the
  96. 4:28inference engine. The common some of the
  97. 4:31common inference engines are the VLMs,
  98. 4:34SGLAN, Tensor RT etc. engine is what
  99. 4:39actually hosts your model.
  100. 4:42It actually serves your model uh and it
  101. 4:45loads the model weights into your GPUs.
  102. 4:48It groups many requests into one batch
  103. 4:50so the GPU stays busy and for every
  104. 4:53request it keeps something called as a
  105. 4:56KV cache. In plain words, the KV cache
  106. 4:58is the memory the request builds up when
  107. 5:02it try when its answer is being written.
  108. 5:05Longer the answer, bigger this memory
  109. 5:07gets.
  110. 5:08Keep this picture in your head. Every
  111. 5:10box here makes some decision but one
  112. 5:13decision is missing and we will find it
  113. 5:16out in the few minutes.
  114. 5:19First, how much an engine can actually
  115. 5:22hold.
  116. 5:26Every deployment has a ceiling and it
  117. 5:28comes from the three targets that pull
  118. 5:30against each other. Quality is how big
  119. 5:33your and precise your model is. Latency
  120. 5:37is how fast the first token arrives and
  121. 5:40every token after it. Throughput is how
  122. 5:43many requests your GPUs can serve at a
  123. 5:45same time. You know the old saying good,
  124. 5:49fast, cheap, but you have to pick two.
  125. 5:51It's the same here. You cannot pick all
  126. 5:53the three. You cannot prioritize all the
  127. 5:55three. So you can't So every product has
  128. 5:59to pick two. For an example like an
  129. 6:01interactive chat, you would want to go
  130. 6:04with the quality and the latency because
  131. 6:06users are more sensitive to that. It's
  132. 6:09quite obvious to the users if the
  133. 6:11quality is down or the latency is high.
  134. 6:14For something like an async agents, you
  135. 6:16would want to go with a higher
  136. 6:17throughput as well as quality. If
  137. 6:20quality is less, you will have to redo
  138. 6:23the task and redoing is very very
  139. 6:25expensive because these are the
  140. 6:27background and the asynchronous tasks.
  141. 6:30And the high volume real-time systems
  142. 6:32such as the autocompletes they would try
  143. 6:35to favor latency as well as and the
  144. 6:37throughput in favor of the quality
  145. 6:39because you don't need an expensive
  146. 6:41model for doing the autocompletes for
  147. 6:43you. But whichever corner you give up
  148. 6:46there is still a maximum load you can
  149. 6:49carry while hitting your SLO
  150. 6:52your service level objective. Let's say
  151. 6:54that on this deployment it's eight
  152. 6:57requests at once. In a normal minute,
  153. 7:00you never notice that limit. Now,
  154. 7:03picture the 40 requests arriving in the
  155. 7:05same two seconds.
  156. 7:08It's such a high load.
  157. 7:15And this is not like hypothetical. It
  158. 7:17has happened to like biggest providers
  159. 7:19in the world. Here are the two public
  160. 7:21postmortms two years apart.
  161. 7:24On November 8, 2023, OpenAI after a day
  162. 7:27of unusual heavy traffic, maybe not 40
  163. 7:31requests, it would be some million
  164. 7:33requests, their routing servers ran out
  165. 7:36of the memory. They started failing
  166. 7:37their health checks. They added more
  167. 7:40replicas, then had to roll them back.
  168. 7:42The way out was to cut incoming traffic
  169. 7:45and ramp it up slowly. In January 21
  170. 7:502026 enthropic the sonnet 4.5 tire gear
  171. 7:55got slow first then it started returning
  172. 7:59errors over about 90 minutes. The cause
  173. 8:02they gave was simple not enough compute
  174. 8:04for the demand.
  175. 8:07And now let's look at what each team
  176. 8:10decided as their next set of the action
  177. 8:12item. Neither of them says we need a
  178. 8:15faster engine.
  179. 8:17Open AI's followup is graceful load
  180. 8:20shedding. Enthropics is detecting
  181. 8:24capacity limits earlier.
  182. 8:26It's the same idea in the different
  183. 8:29words which is decide on your request
  184. 8:32sooner
  185. 8:34to make sure which request gets in. And
  186. 8:38there are more incidents which we have
  187. 8:41linked in the appendix. If you are
  188. 8:43interested definitely go and read
  189. 8:44through them.
  190. 8:49So
  191. 8:52if you walk backward from those
  192. 8:54follow-ups,
  193. 8:55they both point at the same place, the
  194. 8:57front door. Here is the pipeline again
  195. 9:01with the missing step filled in.
  196. 9:05We have a client, we have an admit
  197. 9:08layer, the Q router, and then the
  198. 9:10engine.
  199. 9:12It's never the client talking directly
  200. 9:14to your engine or client talking to your
  201. 9:17queue directly. There is an admission
  202. 9:20phase in between.
  203. 9:22Admission is a decision made before a
  204. 9:25request takes any GPU memory and it only
  205. 9:28has three possible answers.
  206. 9:33Admission decides
  207. 9:36whether to start now or you want to
  208. 9:42hold the request. You want to accept it
  209. 9:44but make it wait. Third is the reject.
  210. 9:47That means not right now. It can throw
  211. 9:50different amount of errors here such as
  212. 9:52429s and we will dive into this bit more
  213. 9:56deeper in the second part.
  214. 10:00this that is the decision that was
  215. 10:01missing from our pipeline.
  216. 10:06So why does it have to happen before the
  217. 10:09engine
  218. 10:11and what would happen if you will
  219. 10:13actually accept the request what would
  220. 10:16be the cost of that yes
  221. 10:19so let's try to take a look at this like
  222. 10:22two phases of the inference like prefill
  223. 10:24and the decode. So first comes the
  224. 10:26prefill. That's the engine reading the W
  225. 10:29prompt, building the request KV cache.
  226. 10:31And then comes the decode. That's the
  227. 10:33engine writing the answer one token at a
  228. 10:35time. With every token, the cash grows a
  229. 10:38little more. All of that memory stays
  230. 10:40reserved until the request finishes.
  231. 10:45So a yes is not a moment. It's a
  232. 10:48commitment that lasts the request whole
  233. 10:50life. And here you can look at the
  234. 10:53example.
  235. 10:55There is a request A, there is a request
  236. 10:57B. Once you admit a request A which is a
  237. 11:00short answer request and then you admit
  238. 11:02a request B which is a long answer
  239. 11:04request. Now when you admit a request C
  240. 11:06which is again a short answer request,
  241. 11:09we don't know how longer it would take
  242. 11:11because there is a longer request which
  243. 11:13was already in the memory. Should we
  244. 11:16even take that request C or should we
  245. 11:18not even take the request C. So there is
  246. 11:20a cost associated of like admitting
  247. 11:23every request because admission could
  248. 11:26still result in missing out your SLOs's.
  249. 11:34Another impact apart from the SLOs's is
  250. 11:37your memory.
  251. 11:40Here is what it does to your GPU. So
  252. 11:43every running request it keeps growing
  253. 11:45your cache as it writes. So your total
  254. 11:47memory keeps climbing. When the memory
  255. 11:49gets to the full engine has three ways
  256. 11:51to cut. Either it will aait
  257. 11:54or like throw away your request. Either
  258. 11:57it will put it to your waiting state
  259. 11:59that means it will swap or it will just
  260. 12:03say okay I won't be able to serve and I
  261. 12:05I will probably return you an error.
  262. 12:09Every one of those throws I mean every
  263. 12:12one of those causes a certain delay in
  264. 12:16processing of the request. and missing
  265. 12:19out your SLOs's again while also wasting
  266. 12:21your memory.
  267. 12:30So why is it like our problem like we
  268. 12:33have con servers from open AI anthropic
  269. 12:37bold teams which are serving their own
  270. 12:39infrastructure
  271. 12:41for the inference.
  272. 12:44So it's a problem for us because
  273. 12:48there are teams who want privacy. They
  274. 12:51want to have their own models hosted on
  275. 12:54their own GPUs
  276. 12:57and they want it to be hosted on their
  277. 12:59own GPUs due to the cost perspectives
  278. 13:02but they have to also protect
  279. 13:06the burndown on those GPUs.
  280. 13:08So they need to effectively decide which
  281. 13:11request to serve across their inference
  282. 13:14ecosystem.
  283. 13:22On top of
  284. 13:24and on top of that like the traffic
  285. 13:26arriving at the door has also changed.
  286. 13:30Earlier there used to be a person that
  287. 13:32was chatting. He would send one request
  288. 13:34at a time. There would be a seconds of
  289. 13:36reading in between and then there would
  290. 13:38be a slow answer. But now coding agents
  291. 13:40they have taken over. There are tons of
  292. 13:42calls that these coding agents they send
  293. 13:45along with the retries, pauses,
  294. 13:49um
  295. 13:51like the undo calls as well and it there
  296. 13:54is such a overwhelming traffic that your
  297. 13:57cluster has to handle and what that
  298. 14:00means is cluster has a responsibility to
  299. 14:03take a wise decision of which requests
  300. 14:05to accept and which uh requests to
  301. 14:08reject.
  302. 14:15And this is exactly that visualization.
  303. 14:17You ask your agent that fix an o test.
  304. 14:21It runs all your it tries to run
  305. 14:24multiple tool calls that is reading of
  306. 14:27the files like around there could be
  307. 14:29like 14 files. There could be like
  308. 14:32around 10 tests that it has to run and
  309. 14:35it would combine together to make 40 50
  310. 14:38calls for just one request and all these
  311. 14:4240 requests they require like reaching
  312. 14:44out to the LLM getting back the
  313. 14:46response. Uh and in between some of
  314. 14:50these requests would be long running
  315. 14:52some of the requests would be short
  316. 14:55prompt request. So we need to decide
  317. 14:57effectively where which one to pick and
  318. 15:00which one not to pick.
  319. 15:09Yeah. And that was the like reason why
  320. 15:12we need to worry about
  321. 15:15admission control. Why we need to worry
  322. 15:18about
  323. 15:21handling the traffic right way across
  324. 15:23our infancy ecosystem.
  325. 15:32Okay. So this brings us to the admission
  326. 15:36control.
  327. 15:37Uh now moving on to the next slide. Uh
  328. 15:42that is slide number 18. So in this
  329. 15:45slide we will be covering three
  330. 15:47questions. Where does this decision
  331. 15:50from? which request to admit
  332. 15:52and which to reject sits back at your
  333. 15:55stack
  334. 15:57and the another question is what does it
  335. 16:00need to see to make a good call or
  336. 16:03judgment on the request and what does it
  337. 16:07returns back to the client. Moving on to
  338. 16:10the next slide. So here uh we see the
  339. 16:15pipeline again which each layer's jobs
  340. 16:17is written out. The agent sees its own
  341. 16:21call and decides when to retry.
  342. 16:25The gateway sees API key and request
  343. 16:28count and enforces a rate limit rate
  344. 16:31limiter for each and the router which
  345. 16:35sees replicas or healthy and which one
  346. 16:37to pick. Now the engine sees its KV
  347. 16:41cache and the current batch and decides
  348. 16:44the order of request to run and what to
  349. 16:47evict and print.
  350. 16:50So every layer decides something. Now
  351. 16:54ask each of them one question. Is the
  352. 16:57room for this request?
  353. 17:00The agent says not my job. The gateway
  354. 17:03says I cannot see it. And the router say
  355. 17:07not my job and the engine says I only
  356. 17:09find out when I have already said yes.
  357. 17:13The admission box is missing in in
  358. 17:17between.
  359. 17:20So now moving on to the next slide. Um
  360. 17:23so where to put this decision change
  361. 17:28what it can see and what it cost you to
  362. 17:31run. So there are three possible
  363. 17:34moments. The first is an invoice right
  364. 17:38in front of the engine. Here the policy
  365. 17:41sees the request itself plus whatever
  366. 17:43metrics the engine exposes. All you need
  367. 17:47is proxy or gateway. The engine stays
  368. 17:50exactly as it is. The second one is the
  369. 17:53Q boundary. Now the decision happens
  370. 17:56inside the engineuler
  371. 17:58as it picks the next batch to run.
  372. 18:03The third is during decoding when the
  373. 18:06scheduleuler can pause a request or take
  374. 18:08back its cache halfway through an
  375. 18:11answer. Those later movement can see
  376. 18:14more but they mean changing the
  377. 18:16engineuler.
  378. 18:18That means keeping your own modified
  379. 18:20copy of BLLM or SG lang and updating it
  380. 18:24every time original changes forever. So
  381. 18:28an ingress is the only movement that
  382. 18:30works with the engine exactly as it
  383. 18:33ships.
  384. 18:34So that's where we will stay for the
  385. 18:36rest of the talk. So now moving on to
  386. 18:39the next slide.
  387. 18:42Uh so what actually sits in ingress,
  388. 18:46right? Uh so you you you can think of it
  389. 18:49as a function. It can take in what it
  390. 18:52knows about the request and current
  391. 18:54state of the system and it returns one
  392. 18:58of the three answers
  393. 19:01like if we have to admit the request. So
  394. 19:04the admit is the request enters the
  395. 19:07engine now and memory is committed for
  396. 19:09this moment. If it's deferred with a
  397. 19:13time we can call toao hold on like uh
  398. 19:16hold on and come back after some time
  399. 19:19and there is another decision which is
  400. 19:22with uh reject with a code that no here
  401. 19:26is the reason why and until the policy
  402. 19:29says admit the request cost you nothing.
  403. 19:32It holds no GPU memory at all.
  404. 19:37Now moving on to the next slide.
  405. 19:43uh to make the call well the policy
  406. 19:46needs two views at the same time from
  407. 19:49the request and from the engine. So from
  408. 19:53the request how long the input is, which
  409. 19:56client or tenant sends it, which agent
  410. 19:59session it belongs to, and a deadline.
  411. 20:03And from the engine point of view, how
  412. 20:06long uh KV cache is, how long the oldest
  413. 20:10request has been waiting in the queue,
  414. 20:12how often the prompt is already cached,
  415. 20:15and how how loaded each pool of replica
  416. 20:19is. Now look back at the ST. The gateway
  417. 20:22has left side and the engine has the
  418. 20:26right side. Nobody has the both.
  419. 20:30And that's really why admission has to
  420. 20:33be its own layer. Now moving on to the
  421. 20:36next slide.
  422. 20:39Here are those eight signals turned into
  423. 20:41simple rule. Each one is compared to
  424. 20:44threshold and when it crosses it pushes
  425. 20:47towards the outcome in the colored pale
  426. 20:51from the request side.
  427. 20:53If the input is too big to fit in the
  428. 20:56memory headroom just refer it. If a
  429. 20:59client or tenant is over
  430. 21:02it share reject it with 429.
  431. 21:06If one agent session has too many calls
  432. 21:09in flight defer with extras. If waiting
  433. 21:13would make a request miss its deadline
  434. 21:15anyway reject it from the engine side.
  435. 21:19Uh you can see at the bottom half of
  436. 21:23bottom half side. While KB cache is
  437. 21:26under high water mark admit it. When the
  438. 21:31oldest request has waited longer than
  439. 21:33your target defer it. If the prompt is
  440. 21:36already cached admit it first because
  441. 21:39it's cheap to serve. And when every pool
  442. 21:41is full, reject it with 529. One rule of
  443. 21:45thumb, pick the signal that changes in
  444. 21:48second, like a QH, not a one minute
  445. 21:52average. A coding agent sends 40 calls
  446. 21:55in 2 seconds.
  447. 21:57A minute average won't even notice. So
  448. 22:00that's what the policy reads. Now what
  449. 22:03does it say back? Now moving on to the
  450. 22:06next slide.
  451. 22:08We will take the three answers one at a
  452. 22:11time and look at each from both sides.
  453. 22:15What the server sends and what your
  454. 22:17agent should do with it. First, yes, the
  455. 22:21server sends back a 200 and the response
  456. 22:24starts streaming. At that moment, three
  457. 22:26things happens inside the engine. It
  458. 22:29commits KV cache from the whole prompt
  459. 22:32and the request joins the running batch
  460. 22:34and decode one token per step alongside
  461. 22:37everyone else
  462. 22:40and it holds that memory until it
  463. 22:42finishes or until the client
  464. 22:44disconnects.
  465. 22:46That's the commitment from part one just
  466. 22:49in the form of an HTTP response which is
  467. 22:52why the admission decision has to happen
  468. 22:54before 200 never after it.
  469. 22:59Now the client side if we are writing
  470. 23:02the agent here is how to handle 200.
  471. 23:06Watch the stream not the status code.
  472. 23:10First time is properly start your
  473. 23:13timeout clock when the first token
  474. 23:15arrives not when you send the request
  475. 23:18under the heavy load prefill alone can
  476. 23:21take few seconds. Second, if you decide
  477. 23:24to give up, actually close the
  478. 23:26connection. The server only frees the KV
  479. 23:29cache when the request finishes or the
  480. 23:31client disconnects. An agent that times
  481. 23:34out but leaves the connection open is
  482. 23:36still holding the GPU memory. And the
  483. 23:39third one, uh, a stream can fail halfway
  484. 23:43through, right? And that that's no
  485. 23:46status for that, right? and you're
  486. 23:49you're left with partial answer and you
  487. 23:51have to decide whether to retry or carry
  488. 23:53on. In short, 200 promise to start. It's
  489. 23:58not a promise to finish. Right?
  490. 24:03The second answer is dipper which
  491. 24:05basically means not yet. The server
  492. 24:07sends back 503 with retry header.
  493. 24:11The header tells the client how many
  494. 24:13seconds to wait. The policy picks that
  495. 24:15number from the engine. For example, how
  496. 24:18long until the running batch finishes
  497. 24:20and freeze the enough cash
  498. 24:22and you differ when capacity is coming
  499. 24:25back soon? It's a burst, not so not a
  500. 24:27trend, right? And for example, like if
  501. 24:30you think about it like Q age is rising
  502. 24:34but the system isn't saturated.
  503. 24:39If the capacity isn't coming back
  504. 24:41anytime soon, a dipper just kicks kicks
  505. 24:44the can down the road, right? And in
  506. 24:48that case, we can just reject it with
  507. 24:50the honest answer.
  508. 24:53Moving on to the next slide. On the
  509. 24:55agent side, handling a defer is simple.
  510. 24:59Sleep for retry after time and a little
  511. 25:02random jitter, then retry again. The
  512. 25:05jitter just means a small random extra
  513. 25:07delay. So client don't like don't call
  514. 25:11or like don't at all come back at the
  515. 25:14exact same moment. They will have to
  516. 25:16wait.
  517. 25:19Look what the server sees from 10
  518. 25:21retries without retry after every agent
  519. 25:25picks its own delay. And that's usually
  520. 25:29that's usually zero. So they all come
  521. 25:32back at once. That's how 40 calls turns
  522. 25:35into 80.
  523. 25:38With retry after and jitter the retry
  524. 25:41spreads out across the window serves
  525. 25:43asked for.
  526. 25:50Okay. So if if we see the uh reject
  527. 25:55reject returns 429 or 529 the third
  528. 25:59answer is no.
  529. 26:01Uh and there there are two different
  530. 26:03codes for it because there are two
  531. 26:05different reasons. A 429s means too many
  532. 26:08requests.
  533. 26:10This particular client or agent session
  534. 26:13is using more than its share. The policy
  535. 26:16saw something like per client load, the
  536. 26:18number of concurrent calls or a kota.
  537. 26:21The problem is the caller and then the
  538. 26:24529 means the provider is overloaded.
  539. 26:28Enthropic uses this code. Every replica
  540. 26:31is sold. So nobody is getting in right
  541. 26:34now.
  542. 26:36The problem is pool not the caller.
  543. 26:40Now uh handling a reject. So how should
  544. 26:44agent react on 429? Reduce your own
  545. 26:48load. Cap how many calls that agent has
  546. 26:51in flight at once.
  547. 26:54And on 529 retrying harder only makes
  548. 26:57this worse. Waiting longer switching to
  549. 27:00different model provider or park the
  550. 27:02task for later. and one either and on
  551. 27:07either one keep a retry budget. Use
  552. 27:09expansion exponential back off with
  553. 27:12jitter which means waiting longer after
  554. 27:15each failed try and after few tries tell
  555. 27:19the user what's going on because a
  556. 27:21reject that gets retrieded at full speed
  557. 27:24is how you end up in one of the postmodm
  558. 27:26from the talk at we described at the
  559. 27:29starting.
  560. 27:33Now over to you Hershel
  561. 27:40three.
  562. 27:41Thank you P. Uh so this is the module
  563. 27:44three where we will try to learn about
  564. 27:46the different strategies uh that have
  565. 27:48been rolled out for the admission
  566. 27:50control. So far we have seen that no
  567. 27:52layer in the today's stack asks where uh
  568. 27:55whether there is a room and that layer
  569. 27:58is ours to add and now we give it a
  570. 28:01brain. So we will look at why counting
  571. 28:03requests isn't enough. Breaking a policy
  572. 28:06down into four different choices. See
  573. 28:08what four different research paper chose
  574. 28:12and finish with a decision tree that you
  575. 28:15can use onto your own ecosystem.
  576. 28:20So let's start with what most team
  577. 28:22already have which is counting. Yes.
  578. 28:26Counting the number of the requests.
  579. 28:30A simple idea is to rate limit at the
  580. 28:32gateway which gives API key a fixed
  581. 28:35allowance set in advance. It's great for
  582. 28:38keeping the clients fair, keeping your
  583. 28:40tenants fair with each other,
  584. 28:43but it does nothing to protect your GPUs
  585. 28:47because it doesn't change with the load
  586. 28:50that is happening onto your GPUs or your
  587. 28:52cluster. A fixed cap at the proxy gets
  588. 28:57closer. It counts how many requests are
  589. 29:00in flight on each replica. The problem
  590. 29:03is that the quick crep and a long
  591. 29:05planning to call
  592. 29:08count exactly the same while the real
  593. 29:10capacity of the GPU changes with every
  594. 29:12batch. So in practice it mostly cues and
  595. 29:17clients timeout ends up doing the
  596. 29:20refusing.
  597. 29:22Neither one knows whether our 40 calls
  598. 29:25are cheap or expensive.
  599. 29:27The cost of air policy does though and
  600. 29:31it estimates each request will actually
  601. 29:35cost what each request will cost in the
  602. 29:39KV memory and the tokens using the live
  603. 29:42state from your engine. And it says no
  604. 29:45before the request actually takes up the
  605. 29:47memory. So the short version is count is
  606. 29:50not a cost.
  607. 29:55How do you build your own policy? So
  608. 29:57there are four choices here. The first
  609. 30:00is the unit
  610. 30:02which tells you what you are actually
  611. 30:04admitting. Are you admitting a single
  612. 30:06request? Are you admitting an agent
  613. 30:09session or are you admitting a tenant?
  614. 30:13The two second is the signal
  615. 30:17which means what is that the policy is
  616. 30:20watching for taking an admission
  617. 30:22decision. This could be as simple as
  618. 30:24your memory pressure which is your KB
  619. 30:27pressure. It could be your Q depth. It
  620. 30:29could be your deadline slack. That means
  621. 30:32whether your request is able to meet
  622. 30:34your SLOs's or not.
  623. 30:37And whatever it is, it has to actually
  624. 30:39move when the system gets busy. So three
  625. 30:43is the rule which means how does the
  626. 30:46signal gets turned into a decision.
  627. 30:50So rule is used for that and that can be
  628. 30:54anything from a simple threshold that
  629. 30:56takes microscond to a full optimizer
  630. 30:58that takes seconds.
  631. 31:01Fourth is the action. Once you have the
  632. 31:04rule that gives you a decision what do
  633. 31:06you do with it?
  634. 31:09What does your how does your client gets
  635. 31:11impacted through those actions?
  636. 31:14That's the admit defer and the reject
  637. 31:17that per just talked about in the module
  638. 31:20two. Each choice limits the next one.
  639. 31:24For example, a deadline signal only
  640. 31:26works if your request actually carry the
  641. 31:28deadlines. So let's go through the first
  642. 31:31three.
  643. 31:37Choice one. Do you admit a single
  644. 31:39request or do you admit a poll sessions?
  645. 31:44Say you have two agent task 20 calls
  646. 31:46each room for almost half of them.
  647. 31:50You can uh point you can see the top
  648. 31:54row. So if you admit per request about
  649. 31:57half of each task gets in but neither
  650. 32:00task can finish because each one is
  651. 32:03stuck waiting on the calls that didn't
  652. 32:05make it. You have used all the capacity
  653. 32:08and completed nothing. If you admit per
  654. 32:12session, session A gets all of its call
  655. 32:15and finishes. Session B gets one clear
  656. 32:17answer. Come back later. And this
  657. 32:20picture is illustrative, but the logic
  658. 32:22holds.
  659. 32:24Our 40 calls are one session. Coding
  660. 32:27agents send sessions and yet most
  661. 32:30published policies admit only the single
  662. 32:33requests.
  663. 32:38So which metric do you watch for what
  664. 32:40particular problem?
  665. 32:44This is the
  666. 32:46question that we are trying to answer.
  667. 32:49It depends on the problem that you are
  668. 32:51trying to prevent. If you are if you
  669. 32:54want to keep the cash from overflowing,
  670. 32:57you definitely have to watch your KV
  671. 32:59pressure.
  672. 33:02If you want to avoid your wasted
  673. 33:04recomputee, you have to watch out
  674. 33:07the KV used together with the cash hit
  675. 33:10rate. A full cache is fine if it's being
  676. 33:12reused.
  677. 33:15If you want to keep your weights to be
  678. 33:18shorter, you have to watch for your
  679. 33:20queue depth and you have to take actions
  680. 33:22based on that. If you want to meet your
  681. 33:25latency targets, you need to watch your
  682. 33:27deadline slack. Take actions based on
  683. 33:30that. If you need to plan for your long
  684. 33:32answers, you need to predict for the
  685. 33:34output length.
  686. 33:37And look at the last column
  687. 33:40fails when it tells you when each signal
  688. 33:44stops working. Look at the first row. KV
  689. 33:48pressure fails when the cache never
  690. 33:50fills up.
  691. 33:52Remember that row? It comes back to bite
  692. 33:54someone in the part four.
  693. 34:01So what are the four ways to take that
  694. 34:03decision?
  695. 34:07That is how do you convert your signal
  696. 34:09into a decision from the cheapest to the
  697. 34:12most expensive. Here are the ways a
  698. 34:16threshold just compares one number to
  699. 34:18the one limit. Example is a policy named
  700. 34:23clued weight that was uh created
  701. 34:27somewhere in 2026.
  702. 34:29It takes tens of microsconds per
  703. 34:31decision.
  704. 34:34Second is a dual gate which needs two
  705. 34:37signals to agree before it acts. Example
  706. 34:40is a paper called concour
  707. 34:43uh which checks like KVUs and the cash
  708. 34:45hits under 100 microsconds.
  709. 34:49Third is an optimizer which solves like
  710. 34:51one big planning problem across all the
  711. 34:54cues at once. Example is a QLM.
  712. 34:57It's thorough but it takes about 5
  713. 34:59seconds per decision.
  714. 35:02And the fourth is the formal bound uh
  715. 35:05which does the math on the worst case.
  716. 35:08Can this request still meets its
  717. 35:10deadline?
  718. 35:12The example is the Kronos on which uses
  719. 35:16the like time to first token and it uh
  720. 35:19provides you like a microcond level
  721. 35:21latency.
  722. 35:23And now hold that cost column up against
  723. 35:26our agent. It sends 40 calls in 2
  724. 35:28seconds. A 5-second decision shows up
  725. 35:31after every party is over.
  726. 35:38So what the four papers did
  727. 35:42as the complete policy fluid weight
  728. 35:44admits single request uses the threshold
  729. 35:46on your KV use and reports 10 to 25%
  730. 35:50improvement. Conqueror is the one that
  731. 35:53admits your VO agents. It uses your dual
  732. 35:56gate and reports four to 4x
  733. 36:00uh the throughput of the SG lang but it
  734. 36:02was tested offline.
  735. 36:05Uh QLM admits the request plans across
  736. 36:09the W feed and reports 40 to 90% gains
  737. 36:13at about 5 seconds per decision. Kronos
  738. 36:16admits requests against their deadline
  739. 36:18slack with a formal bound and takes
  740. 36:21deadline misses from 99% down to zero.
  741. 36:25Definitely very amazing accomplishments.
  742. 36:29So each purper used a different model
  743. 36:32engine and a workload. So please don't
  744. 36:34read this as a leaderboard.
  745. 36:37The question that matters for this room
  746. 36:39is does any of them fit a coding agent
  747. 36:45and the answer is no. We placed every
  748. 36:49paper on the two axises. One axis is
  749. 36:52what it optimizes deadlines or the
  750. 36:55throughputs. Other is what it admits
  751. 36:57single request or the VO agent sessions.
  752. 37:01A coding agent needs a policy that
  753. 37:03admits bold sessions, works online with
  754. 37:06live traffic and runs engine you haven't
  755. 37:08fought
  756. 37:11and nothing sits there cleanly. Conc
  757. 37:13doesn't need a for but it was only
  758. 37:16tested offline.
  759. 37:18So in practice you will pick the closest
  760. 37:20match and build just build your own. And
  761. 37:23here is how you can choose which polic
  762. 37:27like which kind of a policy you want to
  763. 37:29try it out.
  764. 37:31So here is that decision tree.
  765. 37:34First
  766. 37:36the question that you have to answer is
  767. 37:38does your request carry any deadlines?
  768. 37:40If yes, set up a some sort of a formal
  769. 37:44bound on that deadline. This will take
  770. 37:46some microsconds level of latency.
  771. 37:50If there is no deadline requirement, the
  772. 37:53second question that you can ask is does
  773. 37:56the KV cache fill at the peak load? If
  774. 38:00your KV cache fills at the peak load,
  775. 38:05then
  776. 38:07you would want something like a way to
  777. 38:11detect the cache prefixes used. So you
  778. 38:14can use some sort of a dual gate
  779. 38:15mechanism or a KV threshold mechanism.
  780. 38:19Um and then if your KV cache gets filled
  781. 38:24at the peak load, it does not gets
  782. 38:27filled at the peak load, then you what
  783. 38:28you could do is you can set up a Q level
  784. 38:30of threshold on it.
  785. 38:35And that gets us to the benchmarking
  786. 38:38like once you have a policy, how do you
  787. 38:41benchmark?
  788. 38:43So I will be handing it over to pop.
  789. 38:50Thank you. So now moving on to part four
  790. 38:53which is benchmarking. As we just saw no
  791. 38:57publish policy fit a coding agent
  792. 39:00cleanly. So you are going to adopt one
  793. 39:02or you you will have to build your own
  794. 39:05and then you have to prove if it's
  795. 39:07actually working or not. So to now
  796. 39:11moving on to the next slide
  797. 39:14you would have to test your policies
  798. 39:16right so for that you need four things
  799. 39:19first is your cluster your policy and
  800. 39:23trace and score. So let's dig deeper
  801. 39:26into each. So your cluster of the first
  802. 39:29measure it it is capacity with no
  803. 39:33admission control at all. The load your
  804. 39:35replica can sustain on their own. Every
  805. 39:38load level at after this is described as
  806. 39:42multiple of your number like multiple of
  807. 39:45that number and uh about the policy u
  808. 39:49which signals it reads what limit it
  809. 39:52uses and what it does when it says no.
  810. 39:56The third one is trace which is a record
  811. 39:58of real traffic that you replay using
  812. 40:02code coding agent traffic ramping from
  813. 40:04half your capacity up to twice your
  814. 40:08capacity.
  815. 40:10Four is a score. Then they run the same
  816. 40:13trace twice. Once with the policy and
  817. 40:16once without and then you can compare
  818. 40:19your result. The score is where the most
  819. 40:22people go wrong. So let's look look at
  820. 40:25that. Now moving on to the next slide
  821. 40:28which is uh how how you score. The score
  822. 40:32we recommend is called goodput under
  823. 40:35admission. So you can see the formula on
  824. 40:38the screen. The top of the fraction is
  825. 40:41the request that met their SLO their
  826. 40:44latency target. That means the answer
  827. 40:47started on time and it never stalled for
  828. 40:50too long between tokens.
  829. 40:53The bottom of the fraction is every
  830. 40:56request that arrived including the one
  831. 40:59which got rejected. Why does the bottom
  832. 41:02matter so much?
  833. 41:05So try this. Uh refuse 95% of your
  834. 41:09traffic and serve the remaining 5%
  835. 41:11perfectly. If you only count what you
  836. 41:15admitted, your SLO attainment is 100%.
  837. 41:19Looks amazing, right? Just ship it.
  838. 41:22But no like good put under admission
  839. 41:25says 5%. Shedding looks three when you
  840. 41:29only account the request you kept. It
  841. 41:33isn't.
  842. 41:35So moving on to slide 42.
  843. 41:39The second thing to get right is the
  844. 41:42workload. A benchmark only tell you
  845. 41:44something if it looks like your real
  846. 41:47traffic.
  847. 41:49So replay agent traces with their fan
  848. 41:52out retries session not uniform random
  849. 41:56arrival.
  850. 41:57Describe load as multiple of the
  851. 42:00capacity you measured and sweep from
  852. 42:03half of it to double it.
  853. 42:08Set your guideline as multiple of your
  854. 42:11latency with no load. And remember a
  855. 42:14benchmark that never goes past capacity
  856. 42:17never actually tests the policy because
  857. 42:19the policy never has to say no. So you
  858. 42:22have to make sure that you reach at that
  859. 42:25point.
  860. 42:27Now
  861. 42:29move moving on to the seven uh seven
  862. 42:32things every admission result should
  863. 42:34disclose.
  864. 42:36Signal liess is one of the seven things
  865. 42:39every admission result should disclose.
  866. 42:42Here is the full full list that you can
  867. 42:45see on your screen.
  868. 42:47The load as a multiple of measured
  869. 42:50capacity
  870. 42:51deadline sets relative to latency with
  871. 42:54no load.
  872. 42:56Signal liveless which is no paper
  873. 42:59report.
  874. 43:01How many times you repeated the run and
  875. 43:04how much the result varied not one lucky
  876. 43:07run. Where each parameters comes from,
  877. 43:10turned on this setup, borrowed from
  878. 43:12somewhere else or left at the default.
  879. 43:16Whether the rejected request count in a
  880. 43:19good put denominator and the full
  881. 43:22configuration
  882. 43:25and the full configuration is model,
  883. 43:28engine, version, hardware, workload,
  884. 43:31SLOs's and baseline. None of this ask
  885. 43:35anyone to change how they evaluate. It
  886. 43:38just ask them to say what they did. Use
  887. 43:41it as a checklist when you read on on a
  888. 43:44paper and when your benchmark and when
  889. 43:47you benchmark your own policy. Doing all
  890. 43:51seven by hand is painful. So we built a
  891. 43:54tool that records all of this uh as as a
  892. 43:58default.
  893. 44:00So uh the tool is admin perf and it's it
  894. 44:05it's two things in one repository. So in
  895. 44:09production if you see it's a library you
  896. 44:13put in front of VLLM or SG lang it
  897. 44:16decides admit defer or reject for every
  898. 44:20request that's ingress layer from part
  899. 44:23two.
  900. 44:26So if you also see we on the right side
  901. 44:31uh it's uh in in a benchmark it's a
  902. 44:34harness that runs any policy against a
  903. 44:36real engine on a real GPU and reports
  904. 44:40what the policy cost
  905. 44:43both sides calls the same function
  906. 44:45policy decide. So the policy you
  907. 44:49benchmark is exactly the policy you
  908. 44:51deploy.
  909. 44:53We build it because across those 16
  910. 44:56paper no two share the same baseline
  911. 45:00engine workload or SLO admitf makes a
  912. 45:04head-to-head comparison possible.
  913. 45:07Now moving on to the next slide.
  914. 45:11So using its pive command. So you can
  915. 45:15see on the screen we have five commands.
  916. 45:18So infra up starts a real VLM on a real
  917. 45:22GPU. Infra smoke checks that it's
  918. 45:25actually serving. Then we have bench
  919. 45:27runs every policy several times each.
  920. 45:31Then we have bench computer tell you
  921. 45:34which policy won and how by how much.
  922. 45:37And then we have infraown which which
  923. 45:40basically means it should shuts all all
  924. 45:43off so you stop paying. And that's my
  925. 45:46favorite one.
  926. 45:48Uh and so now provisioning and
  927. 45:50benchmarking are separate commands on
  928. 45:52purpose. Loading a model takes minutes
  929. 45:55and every policy has to face exactly
  930. 45:58same deployment.
  931. 46:00If you are reprovisioned between policy,
  932. 46:02you would change the very thing you are
  933. 46:05trying to hold a constant.
  934. 46:08The deployment is constant. The policy
  935. 46:10is variable.
  936. 46:15So now uh here a policy must declare the
  937. 46:19signal it reads. This is how admit perf
  938. 46:22deals with the trap I just showed you.
  939. 46:25Every policy has to declare the signal
  940. 46:27it needs. This KV cache threshold policy
  941. 46:31says I require KV used correction which
  942. 46:35is the share of the KV cache in use.
  943. 46:40If the engine cannot report that signal,
  944. 46:42the run stops at the startup with an
  945. 46:45error. You can actually read without the
  946. 46:48check a missing signal reads as a zero
  947. 46:51and KV cache threshold that always sees
  948. 46:54zero admits everything. It's quietly
  949. 46:57turned into the no admit baseline while
  950. 47:01your results looks completely normal. So
  951. 47:04policy that cannot see its signal should
  952. 47:07fail loudly and not just fail silently.
  953. 47:14Um so here
  954. 47:17what we did is every decision is
  955. 47:20recorded with the state it saw. Every
  956. 47:22run leaves behind a folder you can open
  957. 47:25inspect. There would be four files as
  958. 47:27you can see on the screen. Those are
  959. 47:30manifest JSON summary JSON decision JSON
  960. 47:33and uh out outcome JSON. So
  961. 47:39what each does
  962. 47:41is
  963. 47:42uh you can take a look deeply once you
  964. 47:46run the results. Now we we will be
  965. 47:48looking at the shading cut. So shade
  966. 47:51shading 37% cut tail latency. So here
  967. 47:55are the four results how they will look
  968. 47:57like on a real GPU. The setup small
  969. 48:00model we used quen 2.5 uh and it's a
  970. 48:04half billion
  971. 48:06model on a A10 GPU. The engine was kept
  972. 48:10out at four sequ sequences at a time.
  973. 48:14Eight request at 14 per second two
  974. 48:17repeats.
  975. 48:20If you look at the first row with no
  976. 48:23admission control, the P95 time to first
  977. 48:27token was about two two and two and a
  978. 48:29half around 2 and a half seconds and P
  979. 48:3245 sorry P95 means 95% of the requests
  980. 48:37were faster than that and it swung by
  981. 48:40about 700 milliseconds either way.
  982. 48:44Now the last row if you see with a queue
  983. 48:47up policy that allows at most two
  984. 48:50request waiting 60 63% of requests were
  985. 48:54admitted and the P95 dropped to about
  986. 48:58around 1 second that's 2.5 times better
  987. 49:03with the exact the good same put
  988. 49:07now if we look at the signal matter more
  989. 49:10than the threshold
  990. 49:12but this This is the result I love most.
  991. 49:15Same run. KV cache usage peaked at half
  992. 49:18a percent. The queue got 24 requests
  993. 49:21deep on a small model. KV cache is huge
  994. 49:25compared to the four short sequences. So
  995. 49:28it never comes close to filling. All of
  996. 49:31the pressure showed up in the queue.
  997. 49:33Instead, a KV cache pressure policy
  998. 49:35would have botched a flat line and
  999. 49:38admitted everything. that that's the row
  1000. 49:42of choice from two. So
  1001. 49:48now here and you you can try it today.
  1002. 49:50You don't need a GPU. There is a small
  1003. 49:53fake engine that speaks the three
  1004. 49:55endpoints admit puff uses and it gets
  1005. 49:58busy under load just like a real one. So
  1006. 50:02you can test the whole setup on your
  1007. 50:04laptop. If your right policy is your own
  1008. 50:06package,
  1009. 50:08admit perf discovers it automatically.
  1010. 50:11No change to the repo and contribute
  1011. 50:13contributions are very welcome. New
  1012. 50:16policies, workload, engine adapter or
  1013. 50:19matrices.
  1014. 50:25Okay.
  1015. 50:29So if you want to remember like one
  1016. 50:32slide just make it this one that
  1017. 50:37when you are accepting the request in
  1018. 50:40your inference clusters
  1019. 50:42you need to have a mechanism to say no
  1020. 50:45early something that works at the
  1021. 50:47ingress something before saying yes to
  1022. 50:51the request before that request goes
  1023. 50:54into the GPU before that request takes
  1024. 50:57up your memory.
  1025. 51:00and per added clearly. There are a
  1026. 51:03couple of error codes that your
  1027. 51:06admission policies return. You need to
  1028. 51:08handle those requests and those errors.
  1029. 51:15And with that we would like to come to
  1030. 51:18the closure. We would like to thank you
  1031. 51:20all for your patience, for your time.
  1032. 51:24And here are like two of our projects.
  1033. 51:28It all started as part of the LLM
  1034. 51:30inference at scale handbook. We and when
  1035. 51:33we were doing the survey, we found the
  1036. 51:35gaps and we prepared admit puff. So here
  1037. 51:39is the source code for it. Feel free to
  1038. 51:42go to it and like experiment with it.
  1039. 51:46Raise the issues, contribute to it. idea
  1040. 51:49is to
  1041. 51:52build your admission policies, benchmark
  1042. 51:54them and be confident that it uh helps
  1043. 51:58you keep your cluster stable. Yeah.
  1044. 52:01>> Thank you everyone.
  1045. 52:02>> Thank you everyone.

About this transcript

This page contains the full transcript of Your Self-Hosted LLM Needs to Say No - Harshul Jain & Parv Khatri, Audible & SAP by AI Engineer, generated from the public captions YouTube serves with the video. The transcript has 6,829 words across 1,045 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.