YT YouTube2Text

YouTube2Text → Transcripts → How to Protect Your Systems with Throttling & Rate Limiting - 5 Real-World Use Cases

How to Protect Your Systems with Throttling & Rate Limiting - 5 Real-World Use Cases — Transcript

by Arpit Bhayani · 2,947 words · 413 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00So what is throttling? Throttling is a
0:02technique that ensures that the flow of
0:04data request or anything that is sent to
0:06a target system is sent at a rate which
0:10is acceptable to the target system.
0:12Right? This would ensure that the target
0:14system does not go down. If there are
0:17many users or many subsystems trying to
0:21send data to one such target machine, we
0:24need to ensure that this target machine
0:26is not overwhelmed by the number of
0:27requests that it gets. Right? Which is
0:30where a component called rate limiter
0:32kicks in who ensures that the data is
0:35forwarded to the target system at an
0:37acceptable rate. Right? Throttling is
0:39generally perceived as a defensive
0:41measure where you are being defensive of
0:45your system not going down. Right? And
0:49there are three typical strategies to
0:52like when you are when you know that
0:53there are far too like far too many
0:55requests coming at your rate limiter.
0:58what you like basically what your
1:00throttling mechanism would do, what your
1:02rate limiter would do. There are
1:03typically three things. First, slowing
1:05down. In case of slowing down, what
1:07you'll do is when the when there are a
1:10lot of requests coming in onto your rate
1:11limiter, it would slow down and slowly
1:14drip the request to the target machines,
1:17which means that your rate limiter is
1:19acting as a buffer. Right? A classic
1:21example of this is a normal queueing
1:24like a normal message broker like SQS,
1:26Revit MQ or something where a lot of
1:28requests that is coming in are buffered
1:30there and the consumers are reading it
1:32at their own pace where the pace at
1:34which they're comfortable with. So the
1:35systems are not overwhelmed and there is
1:38a buffer to keep all of those stuff
1:40handy. Second is when a throttler
1:44rejects a request. So it might be
1:46possible where you are getting a lot of
1:49request and if you get
1:52requests which are larger than or or or
1:54or which is more than a particular
1:56number of requests you would want to
1:57reject it where you know that your
2:00system is not able to handle will not be
2:02able to handle the load so you'll be
2:04rejecting those requests. A classic
2:06example of this is when you get a lot of
2:08requests coming onto your API servers.
2:10This is a synchronous communication.
2:11There is no buffering happening there.
2:13So then if you if you know that your API
2:15servers are designed to handle 100
2:17requests per second but if you're
2:19getting 1 million requests per second
2:21what will you do with those others? You
2:23will be rejecting them. So which is
2:25where when you get all of these requests
2:27your rate limiter can choose to reject
2:30the surplus request will only allow 100
2:33requests per second to drip in but would
2:35reject the surplus request. So there is
2:36no buffering here. And the third one is
2:40ignore. So this ignore is similar to
2:42reject but when rejecting you typically
2:46tell the client that your request is
2:48rejected while in case of ignoring you
2:50say I'll not even tell the client but
2:53I'll just ignore right. So this is a
2:56very popular technique to fool an
2:59attacker where your attacker sees 200
3:02okay but behind the scenes your load
3:05balancer has actually rejected the
3:06request. So instead of sending 429 as an
3:08error code, you just send 200 okay to
3:11the attacker and he and that person
3:14thinks key array your uh that you have
3:17accepted a request and and he or she is
3:19actually abusing your system. But in
3:21fact that doesn't but in fact that is
3:24not happening. Instead you are just
3:26ignoring the request from the load
3:27balancer while returning a successful
3:29response to the attacker. So multiple
3:32ways and obviously it's always the
3:34combination of these three that you will
3:36apply on the kind of endpoints that you
3:38have. It's not always like one over
3:40other right. So why do we even need
3:43throttling in the first place? So
3:44throttling we know that it is all about
3:47streamlining whatever comes to a target
3:49machine. Right? But why why do we need
3:52to do that? First reason to prevent
3:54system abuse. Right? There might be like
3:56you you have built your system and you
3:58love it but then you see like but there
4:01are a lot of users who are using your
4:03system and then one user goes rogue and
4:05says okay I'll just th I'll just make as
4:07much of API calls as I can to bring your
4:10system down. So in order to prevent your
4:13business in order to prevent your
4:15product in order to prevent your service
4:16from going down what would you do? You
4:19would want to add a rate limiter to
4:22ensure that your system is not going
4:24down. your system is not overwhelmed.
4:26Second, to only allow traffic that could
4:29be handled. Now here, this is different
4:32from attack. Like these are all genuine
4:35traffic. No user has gone wrong. Uh no
4:37user has gone rogue. Here this
4:40particular use case talks about when
4:43you you have a website and some very
4:47famous person on Twitter tweeted about
4:50it and your website went viral. All the
4:53users coming onto your website, they're
4:54all legitimate users. None of them is an
4:57attacker. But if you know that if you
4:59accept requests from all of those users,
5:00those HTTP requests coming in or those
5:02packets coming in, you know that your
5:04database will go down, right? You would
5:07know that you will not be able to handle
5:09those many requests. So now what would
5:11you do? You would want to allow the
5:13traffic that could be handled so that
5:15your website can still continue to
5:17function for the existing set of people.
5:20Right? This is where you need to vary.
5:22You need to be wary of the fact on which
5:24request to serve and which request to
5:25not. Third is control consumption cost.
5:30So typically when you consume a request
5:33you would be using some compute power
5:35your EC2 machines virtual servers uh
5:37some cache some clusters whatnot to
5:40serve those request. What you would want
5:44to ensure is if you accept a large
5:45number of requests from the user and you
5:47are spinning up a lot of infrastructure
5:49to process it, that would add to a lot
5:51of cost, right? A classic example of
5:54this is a very expensive machine
5:56learning model which you might be
5:57running. So you might have exposed ML as
5:59a service and there are a lot of users
6:01hitting your system and what you would
6:04happen is every request that comes in
6:06requires you to spin up a huge GPU
6:08machine to do some sort of processing,
6:10right? But if you accept a large chunk
6:13of request, your infrastructure cost
6:14will blow up. Your AWS cost or or your
6:17AWS bill will shoot skyhigh and that
6:20might be a problem. So which is where in
6:23order to ensure that you are operating
6:24well within the limits, you would want
6:27to ensure that you would you would
6:30typically want to ensure that you are
6:32accepting the requests that could be
6:34handled within your acceptable limits.
6:36Now this another use case for this uh
6:40control consumption cost is where if you
6:43are accepting a lot of request then and
6:45you are and you have to process it and
6:47that becomes and it might not be worthy
6:51enough for you to execute those process
6:53or to run those process on your infra.
6:55So you might want to just say no and
6:58it's okay for me to drop those request I
7:00don't I just don't want to blow up my
7:02infra right there you would be where
7:05there this particular technique comes in
7:06handy and fourth and the most important
7:09one most important one is to prevent
7:11cascading failures to prevent cascading
7:13failures like when it's not just that
7:15when a lot of requests comes in only one
7:18system is going down when that
7:20particular system goes down it will have
7:22ripple effect it will have cascading
7:24effect over every other system which is
7:26connected to it. So if one system goes
7:28down, it might take down other systems
7:30with it and slowly and steadily your
7:32entire website will go down. So which is
7:34where having a rate limiter at the at
7:37the source point itself would ensure
7:39that the first system is not going down.
7:41So it will prevent your gas heading
7:42failures from happening. Right? So now
7:45let's look at use cases of throttling.
7:47And now in this one we will look at five
7:49use cases of throttling. Two external
7:51rate limiters, three internal rate li
7:53rate limiters. Use case. First use case
7:55very common prevent catastrophic DOS
7:58attack. DOS attack are are there like
8:01when your website grows when your
8:04business is popular, website is popular,
8:06there will be some attacker trying to
8:07abuse your system and you have to be
8:11very defensive about it that no matter
8:14how big the attack is, your website is
8:16not going down. So which is where what
8:18you do is any request that comes in
8:20always hits a always hits a rate limiter
8:23first. The rate limiter will ensure that
8:26it is only forwarding the request which
8:29can be handled by your infra which is a
8:32legitimate request and only those
8:34requests go to your system. Request from
8:37an attacker will not go. Now you say how
8:39how will I identify someone is an
8:40attacker? Obviously you'll see a surge
8:42of request from a particular IP or from
8:45a particular user or from a particular
8:46access token. You will be preventing
8:48you'll be configuring rules in your rate
8:50limiter to say gee do not accept any
8:53more request from this user. Do not
8:54accept any more request from this
8:56authentication token. Right? And what
8:58happens to the surplus request in case
9:00of a DOS attack? They are all dropped.
9:03They are all dropped. The rate it's the
9:05job of the rate limiter to not buffer
9:07it. just drop those request preventing
9:10your system from going down or from
9:12getting abused. This is an example of an
9:14external rate limiter which this rate
9:17limiter interfaces right with your user
9:20or right with your external uh public
9:22facing endpoints. Second is to
9:25gracefully handle the surge of users.
9:27This is similar to an abuse but
9:31dissimilar at the same time. similar
9:33with respect to the traffic pattern
9:35dissimilar with respect to these all
9:37being legitimate users. So again as I
9:40said in the use cases as I said in the
9:44previous part let's say your website
9:45went viral there are a lot of legitimate
9:48users coming in none of them is an
9:49attacker but you have not scaled up your
9:52infra if you get a lot of request your
9:55DB would go down and it will take
9:57everything with it which is where you
10:00need to be wary of the fact that if
10:03there is a legitimate search in the
10:05users I need to gracefully handle
10:08my system handling those request. It is
10:10okay for me to send to the user key hey
10:14my website is slowing down. I am
10:16expecting too much of traffic so please
10:18come back after some time rather than
10:19saying the website is down like like
10:21basically uh rather than giving them a
10:24generic error called 503 service
10:26temporarily unavailable. So with this
10:28rate limiter what you would still do is
10:30there will be request of some users that
10:32would go through request of other users
10:34might just drop right. This way you are
10:38continuing to function for a partial set
10:41of users while not continuing to
10:43function for other set of users. So at
10:45least your business is not completely
10:47affected but partially affected. Right?
10:50Third use case. Now this use case starts
10:52for internal rate limiters. Now I'll
10:55give you an a very solid example. Let's
10:57say you are a CI/CD company like let's
10:59say circle CI who offers multi-tiered
11:02pricing. Let's say there is three tier
11:04tier one, tier two, tier three. Because
11:05you are a CI/CD company, people will use
11:07your tool to build uh their binaries,
11:11build their images, run their test,
11:12right? Build their entire CI/CD. So now
11:15you have three tiers where you have tier
11:17one which is a free tier in which you
11:19give 200 minutes of build time for free.
11:21Then you have tier two in which you
11:22give,000 minutes of build time for let's
11:24say $5 a month. And the tier three is
11:27where you charge customer $50 a month
11:29and you give infinite build time. Now
11:31here what you need to do is this is not
11:33something which is external, right? This
11:36is something which is internal to your
11:38ecosystem. So what anytime any request
11:40comes in for the user to build
11:42something, it comes to our API server.
11:44API server before triggering the build
11:48what it would do is it would check with
11:49the rate limiter key. Hey, has this user
11:52crossed the limit is or basically in
11:55simple term is this request allowed? Has
11:57this user used his or her kota? If not,
12:00I will go and trigger the build workers
12:03for that particular job. Otherwise, I
12:05would not. Right? And here the build
12:07workers will keep on pushing the build
12:09stat uh the build statistics in the
12:11database. Your rate limiter will use the
12:12database to see how much each customer
12:15has consumed. And depending on that it
12:18would say true or false or yes or no to
12:20our API server and AP server can then
12:22either reject the request or accept the
12:23request and continue with the build. So
12:25this is a case of an internal rate
12:27limiter where you are using this to
12:30ration the to ration the compute
12:33capacity to customer depending on the
12:36amount of money that that customer has
12:38paid you. So depending on a multi-tered
12:40pricing that you had, you wanted to
12:42ration your compute to that
12:44corresponding consumer.
12:47Fourth use case is you are not overusing
12:49a third party system. For example, let's
12:52say you are consuming an extremely
12:54expensive third party API and they have
12:57on demand pricing. For example, you are
13:00uh using
13:02a a service from a third party provider
13:04which is giving you very deep very
13:07amazing deep learning capabilities but
13:09every single API call is costing you $5.
13:13If that is the case, what you need to
13:14ensure for your cost to not bloat up for
13:18your cost, your infrastructure cost or
13:20your bill for that external vendor is
13:22not bloating up, you would want to have
13:24an internal rate limiter. you like these
13:27are your worker instances. They before
13:30making call to this expensive vendor
13:32they call they invoke or they check with
13:34the rate limiter key hey should I make a
13:37call to this vendor or not right because
13:39every single call going to this vendor
13:41is $5 every API call. So which is where
13:45because it is very expensive you need to
13:48ensure that hey you are operating within
13:50the limits otherwise your infra cost or
13:53your bill with the vendor will go up
13:55right which is why this is another
13:56example where a rate limiter is not just
13:58external but also internal and fifth and
14:01the final use case that I want to talk
14:03about is to protect an unprotected
14:06system. So a classic example of this is
14:11hard deleting. We know that hard
14:13deleting is hard on the DB because it
14:15requires a lot of rebalancing and you
14:17know whatn not a lot of discios
14:19rebalancing and whatnot. So now because
14:21your DB does not give you rate limiting
14:23by default and if let's say you want to
14:26delete a million rows a million rows
14:28hard delete onto your database if you do
14:30it in one query your DB will definitely
14:33choke up there will be massive table
14:35locks being taken your DB will become
14:36unresponsive your website will go down
14:39which is where to protect your
14:41unprotected system in this case DB what
14:44you would do is you would want to
14:46streamline the delete requests that are
14:49coming in right so let's you want to
14:50delete a million rows in one shot.
14:53Instead of deleting it in one shot, you
14:55would do like every 1 or 1,000 rows,
14:58every two or 1,000 rows, something like
15:00that. You would ration or you would want
15:02to distribute the load uniformly
15:05across a time span. Let's say you want
15:07to delete a million rows every single
15:09day. Split it into 24 hours time. Do it
15:13some number of deletes every minute or
15:15some number of deletes every hour. This
15:17way you are not making or you are not
15:20bombarding your unprotected system with
15:22an expensive call and ensuring that your
15:25unprotected system is not going down.
15:26Again an example of an internal load uh
15:30internal rate limiter
15:33right. So rate limiters again busting
15:35the myth rate limiters are not only
15:37external they help you more on the
15:39internal cases as well where you would
15:42want to streamline where you would want
15:44to protect your unprotected system keep
15:46your infrastructure cost in check and
15:48whatnot. All right nice hope you learned
15:53something new today uh that basically
15:56that's it for this video. If you guys
15:58like this video give this video a thumbs
16:00up. If you guys like the channel give
16:01this channel a sub. I post three
16:03in-depth engineering videos every week
16:05and I'll see you in the next one.

About this transcript

This page contains the full transcript of How to Protect Your Systems with Throttling & Rate Limiting - 5 Real-World Use Cases by Arpit Bhayani, generated from the public captions YouTube serves with the video. The transcript has 2,947 words across 413 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.