Replication and Sharding - System Design — Transcript
Full transcript
- 0:00hey what's going on everybody in this
- 0:01lesson we're going to be talking about
- 0:02replication and sharding for databases
- 0:05so how we go from a single database
- 0:07instance to multiple instances but
- 0:10everything working together as a single
- 0:12unit or you might hear the term cluster
- 0:14so a cluster is when you have multiple
- 0:17database
- 0:19instances so this is going to use
- 0:22technology such as replication and
- 0:24sharding this is going to probably be
- 0:26the last video before we start talking
- 0:29about no SQL databases and the different
- 0:31types and how they work I'm really
- 0:33excited for that video this is going to
- 0:35lay the foundation just to make sure we
- 0:37got everything covered before we go into
- 0:39that content I also want to show off my
- 0:41wife's beautiful new
- 0:45sweater
- 0:47just relax okay so when I explain the
- 0:51majority of these Concepts let's first
- 0:53apply this to relational databases so
- 0:56traditional databases think Oracle MySQL
- 0:59postra
- 1:00many of the nosql databases are more
- 1:03friendlier when it comes to splitting
- 1:04data up across multiple nodes so let's
- 1:07first think from a relational database
- 1:09point of view now we've talked about
- 1:11some of these principles in the earlier
- 1:13videos in this series but you don't
- 1:15necessarily have to go watch those if
- 1:17you're just jumping in here basically we
- 1:18have a computer it's running the
- 1:20database software this is our database
- 1:22instance and we first want to vertically
- 1:25scale our database instance so that
- 1:28means increasing the compute power of
- 1:30just this single server generally if we
- 1:32get to the point where we're working
- 1:34with multiple
- 1:36instances we don't want to be doing this
- 1:39with commodity hardware for traditional
- 1:41database systems what I mean by that is
- 1:43you don't want to be working with a
- 1:44bunch of average servers you want to be
- 1:47working with a really powerful server
- 1:49before you worry about splitting it up
- 1:51across multiple and this is adequate for
- 1:53many use cases if you want to do
- 1:55additional research you can look up
- 1:57sites or apps that have been very
- 2:00successful using just a single database
- 2:02instance and often before we have to
- 2:04worry about using multiple instances
- 2:06there are other things we can do to make
- 2:08things more efficient so that we're not
- 2:10putting as much pressure on that
- 2:12database server so an example of this
- 2:14would be caching we could cach data in
- 2:17something like reddis which can reduce
- 2:19the amount of requests we have to make
- 2:20to the database so here are two things
- 2:23that we can do we can reduce the
- 2:25database load with things like I just
- 2:27mentioned redus caching we can increase
- 2:31the single instance compute power so
- 2:34vertically scale the database instance
- 2:36so let's take a moment to talk about
- 2:38nosql and how we can actually help with
- 2:40this problem nosql is a category of
- 2:43databases so this doesn't describe a
- 2:44single database instance or a bunch of
- 2:47databases that all work exactly the same
- 2:49way and what that means is there's often
- 2:51a bunch of specialized databases that
- 2:53might help with specific things and this
- 2:55specialization means we might use a
- 2:57nosql database for a certain part part
- 3:00of our application or backend but not
- 3:02necessarily for just everything like we
- 3:05might use a relational database for
- 3:07sometimes that's a case there might be a
- 3:09nosql database that you decide hey this
- 3:11is going to be my main database for my
- 3:13data storage but often it's the case
- 3:15you'll have a main relational database
- 3:18but you might use nosql databases for
- 3:20some specific use cases and it's not
- 3:23going to store all of the data for
- 3:24everything so we've already talked about
- 3:26one of these which is redus
- 3:30or some kind of inmemory cache this
- 3:33could be classified as a nosql database
- 3:35so redus could be used for any type of
- 3:37data that doesn't need to be preserved
- 3:39forever another example is elastic
- 3:42search this is a database that is
- 3:45specialized in searching shocker right
- 3:48but there might be a scenario where you
- 3:50might be doing a lot of this searching
- 3:53and you don't want to have to throw that
- 3:54load on your main database so you can
- 3:56put the data that you need to work with
- 3:58into elastic search and then just use
- 4:00that database for that purpose there are
- 4:03also client side databases so this would
- 4:06be index DB so this is specifically
- 4:10referring to the web browser it has
- 4:13index DB available for data storage that
- 4:17doesn't need to be stored on the server
- 4:19so this might be something that you know
- 4:21is used actively but again doesn't need
- 4:24to be stored forever so maybe it's some
- 4:26kind of local game or some settings or
- 4:28something that don't don't require
- 4:30permanent storage but you want to store
- 4:32that information somewhere and one
- 4:34option for doing that would be to use a
- 4:37local database such as index DB so those
- 4:39are just three examples where putting
- 4:41some of the data in other locations
- 4:44might actually help to reduce the total
- 4:47required computation of the main
- 4:50relational database system you can also
- 4:52use a data warehouse so if you're doing
- 4:53a bunch of analytics and this is going
- 4:55to be computationally expensive it might
- 4:58make sense to actually bring the data
- 5:00from a relational standard database for
- 5:03transactional processing to a data
- 5:06warehouse and this data warehouse is
- 5:08more efficient for analytics so that
- 5:11will mean the main database is reserved
- 5:14for really just transactional processing
- 5:17which you'll also hear as
- 5:21oltp which is short for online
- 5:23transactional processing or for data
- 5:26warehousing you might hear oap for
- 5:28analytical process processing I guess
- 5:30I'll just write it over here however my
- 5:32big head might get in the way so we have
- 5:35transactional analytical so that's a
- 5:38variety of different ways you can reduce
- 5:40the demand of a single database instance
- 5:43now if this kind of stuff interests you
- 5:45and you want to know more that's what
- 5:47we're going to be talking about probably
- 5:48in the next lesson so stay tuned for
- 5:50that so we've gone through efforts to
- 5:52reduce the load on this database and
- 5:54it's finally good to go maybe there
- 5:57might actually be some other things that
- 5:58we want to do to to make sure that our
- 6:00database layer is good to go completely
- 6:02so now let's talk about how we can
- 6:04continue to improve our database layer
- 6:06we've already done what we could to
- 6:08reduce the demand on the database and
- 6:11we've made an effort to vertically scale
- 6:12the single database instance before
- 6:14having to worry about multiple database
- 6:17servers so what other concerns might we
- 6:19have well the first thing I want to
- 6:21mention does not actually reduce the
- 6:23total load on the database but it does
- 6:25help with data durability and high
- 6:27availability which is to have a fail
- 6:29over so we want to remove the single
- 6:32point of failure which is having a
- 6:34single instance that's probably my first
- 6:37biggest concern here is if we have a
- 6:39single instance and you know I don't
- 6:41know it explodes in a big fiery ball of
- 6:46fire even if we were able to recover
- 6:49this data the database instance might be
- 6:51offline longer than we would like so
- 6:54we'll first worry about high
- 6:55availability and then we're going to
- 6:57talk about some of the options to
- 6:58actually reduce comp computational load
- 7:00by introducing more servers if we go
- 7:04back in time before our database
- 7:06exploded we can now have a standin node
- 7:11and this is a database instance that's
- 7:13basically a copy of the current instance
- 7:17and the data is going to be replicated
- 7:20from one node to the other so this is
- 7:23where we introduced the idea of
- 7:24replication so with replication you can
- 7:26just think of data being copied over to
- 7:28another server this database is ready to
- 7:30go so if our main database did explode
- 7:34into a fiery ball of fire we would then
- 7:36be able to direct traffic to this other
- 7:38database without having any interruption
- 7:41in the user experience so we removed the
- 7:43single point of failure we're good to go
- 7:45so let's look at replication a little
- 7:47bit closer and we'll ask ourselves how
- 7:49does the second database get the data
- 7:52now obviously the answer is replication
- 7:55but let's talk a little bit in more
- 7:56detail and technically if you want to do
- 7:58some research you can look up the right
- 8:00ahead log which will get you started in
- 8:02this area so w so that's basically a log
- 8:06that all things that happen to the
- 8:07database goes to that log and that can
- 8:09then be used to replicate the same
- 8:11information to another instance but
- 8:14practically we write each incoming right
- 8:17to both
- 8:21databases and this is important to think
- 8:24about and understand there are two
- 8:26there's a microphone right here which is
- 8:28why I'm always like looking here so if
- 8:30I'm always just like staring off in the
- 8:32distance I just for some reason think
- 8:34that's like a camera or something I
- 8:36don't know I lost my train of thought
- 8:38what was I saying now this writing to
- 8:40both instances is done automatically for
- 8:42us so it's not like we actually have to
- 8:44write the query twice or something like
- 8:47that but the main thing to understand is
- 8:49that there are two rights in this
- 8:52process so we're not actually reducing
- 8:56the demand we're doubling the rights and
- 8:59we're doubling the number of instances
- 9:01that we write to so yes we doubled the
- 9:03compute having two instances but we're
- 9:05now doing twice as many wres and that's
- 9:07the reason why it doesn't even help us
- 9:10if we were to make both of these
- 9:12writable if you could write to either of
- 9:14these you might think hey that could
- 9:17double our computation power right
- 9:20basically horizontally scaling our
- 9:22database but if the data is synced
- 9:25across both of these then we're doing
- 9:27just double the amount of Rights so it
- 9:29doesn't matter that we now have two
- 9:31servers we're just making it more
- 9:33complicated so imagine each one of these
- 9:36I'm just making these numbers up but
- 9:37let's say could do a th000 transactions
- 9:40per second and then let's say we wanted
- 9:41to somehow make these both writable we
- 9:43might now be able to process 2,000
- 9:46transactions per second but a single
- 9:50right now becomes two
- 9:55rights so then this number is really
- 9:57just cut back down in half to a00
- 10:01long story short replication does not
- 10:04help with
- 10:05Io if it's a one: one replication and
- 10:09we'll talk about this more but the
- 10:11concept of replication will be used even
- 10:13if it's not a direct copy so when we get
- 10:16into sharding there still might be some
- 10:18replication it just might not be a
- 10:20complete duplicate on every single
- 10:23computer let's say this was your
- 10:24database server and you want to add more
- 10:27nodes but not just for the sake of
- 10:29having standin nodes but you actually
- 10:30want to be able to write to multiple
- 10:32nodes a multimaster database cluster or
- 10:35you might hear it as active active so
- 10:38now let's say instead of just a single
- 10:39node we have three nodes and we write to
- 10:43one of these nodes there might be
- 10:45replication from one node to the next
- 10:48node so that introduces redundancy it
- 10:51improves the
- 10:52durabil oh I just B had a heart
- 10:56attack Claire seriously so this
- 11:00increases the durability of our data
- 11:02because we now have redundancy if one of
- 11:04these catches on fire we still have a
- 11:07backup but we don't necessarily have to
- 11:09replicate it to all of the nodes so
- 11:11let's say maybe we write to this node
- 11:14and it's replicated to this node and
- 11:17then we write to this node and it's
- 11:19replicated to this node so for every
- 11:21write it's written to two nodes but we
- 11:26have three nodes of compute in this
- 11:28situation we process in total more
- 11:31transactions because we don't replicate
- 11:34the transaction to all of the nodes we
- 11:37only replicate it to some of the nodes
- 11:39so this is The Logical next step to
- 11:41database clustering however it does
- 11:43introduce a variety of challenges so if
- 11:45you're just working with traditional
- 11:48relational databases it is possible to
- 11:51do something like this and there's often
- 11:53different companies or services that
- 11:55will do this kind of thing for you so
- 11:56you can have multimaster replication and
- 11:59have a database cluster where you can
- 12:00write to multiple notes it helps with a
- 12:02variety of things first it helps with
- 12:04redundancy of data so you don't lose
- 12:06anything it helps with total compute
- 12:08power so you can increase the total
- 12:11write capacity and it could potentially
- 12:13help with latency because you can space
- 12:15out data appropriately where data is
- 12:17being requested so I'm definitely not
- 12:19saying it's not possible I'm just saying
- 12:21it's a new layer of complexity and
- 12:23really you usually don't want to go to
- 12:25this unless you're like an Enterprise
- 12:27level software where this is needed now
- 12:29when it comes to nosql databases there
- 12:31are some that were designed with this
- 12:33problem in mind so if you're going to go
- 12:35the route of having multiple nodes it
- 12:36might be worth considering a nosql
- 12:39database but there are challenges with
- 12:41that as well you have to think about how
- 12:43you're actually going to structure the
- 12:44data as it's usually a little bit
- 12:46different than a relational database so
- 12:48if you're brand new it might take a
- 12:49little bit of learning on how to
- 12:51properly structure that database so that
- 12:52data is located in the right location
- 12:56and data that needs to be accessed
- 12:58together can be accessed together and
- 13:00then there's all kinds of different
- 13:01configurations like the replication
- 13:02Factor how many times you want to
- 13:04replicate data there's also other Quorum
- 13:06settings that you'll need to consider so
- 13:08that you're writing and reading to the
- 13:10proper number of noes for your use case
- 13:13we'll get into that don't you guys worry
- 13:15I got your back new videos coming soon
- 13:18so be sure to subscribe and if you do
- 13:20enjoy this type of material and you want
- 13:21to get better break into software
- 13:23engineering or Advance your software
- 13:25engineering career I do work with people
- 13:27directly in my mentorship program so if
- 13:29that's something of interest to you I'll
- 13:30have a link down in the description
- 13:32check it out and hopefully we'll get to
- 13:34work together now let's go back to just
- 13:35relational databases before you go the
- 13:38clustering route there's some simple
- 13:40things you can do with replication that
- 13:41will make things better so say this is
- 13:43our database we can first like I
- 13:46mentioned add in a standin node oh my
- 13:50gosh that was awful if anything goes
- 13:52wrong with this guy you got a second one
- 13:54ready to go so that's basically a backup
- 13:57for computation a run database you can
- 14:00also back up your data which doesn't
- 14:03actually keep a database instance
- 14:05running ready to go it's really just the
- 14:08actual content the files so if you're a
- 14:10really high-tech company and you want to
- 14:12do some Cutting Edge stuff then yeah
- 14:14back up your data Additionally you can
- 14:17make replicas that are for
- 14:19reading this will offer a few benefits
- 14:22it will allow us to individually scale
- 14:24the reads and writs if we're directing
- 14:27reading to a separate instance
- 14:30basically the data will just be
- 14:32replicated from the rights to the read
- 14:35database and then we can adjust the
- 14:37compute power either up or
- 14:39down for these individual instances so
- 14:43we just really have to worry about being
- 14:44able to write fast enough on this node
- 14:47and then we can adjust this node as
- 14:48needed to meet the reading requirements
- 14:51same idea here so now we can actually
- 14:53horizontally scale the reading capacity
- 14:57thanks to replication from the main
- 14:59right node so you can vertically scale
- 15:01an individual node or you can add more
- 15:04read replicas a benefit of adding more
- 15:07of them is that you can spread them out
- 15:08in different areas so hey maybe this
- 15:10one's in the US maybe this one's in
- 15:12Europe so if you have a user and they're
- 15:15reading data from the website well
- 15:17that's going to be a lot faster than
- 15:19reading the data all the way from over
- 15:21here you could also have standin nodes
- 15:23for these guys as well if you needed so
- 15:25if for some reason you want to make sure
- 15:28that this node does not go down even
- 15:31though it's just for reading you can
- 15:33still have a stand in node to fill in if
- 15:35something were to go wrong this is less
- 15:37important than this guy here because if
- 15:40you didn't have the standing node and
- 15:42this exploded well you could just direct
- 15:45to a different read node and your
- 15:47application's still going to work now
- 15:49when you have this setup where you're
- 15:50writing to one main node and it's
- 15:53replicating down to read nodes this is
- 15:55usually eventually consistent and what
- 15:58this means is that the data on this node
- 16:01is going to match this data exactly it's
- 16:05just going to be a little bit delayed so
- 16:07say you write a blog post and it goes to
- 16:11this node it's then replicated down to
- 16:14this node but you're very fast right you
- 16:17write that blog post and you immediately
- 16:19check to see if that blog post is there
- 16:21maybe your read is routed to this node
- 16:24and it says oh no there's no blog post
- 16:26there you would say that that's
- 16:28inconsistent the the data doesn't match
- 16:30but it's eventually consistent so if you
- 16:32just wait long enough eventually that
- 16:34blog post is going to appear when you
- 16:35read and everything's fine that's okay
- 16:39in the majority of cases so social
- 16:41medias blogging websites most apps the
- 16:45eventual consistency is fine however
- 16:47there may be scenarios where that's not
- 16:48okay and you need to make sure that your
- 16:50replication is consistent which in that
- 16:52situation it would actually
- 16:55write to the read node
- 16:59and confirm that before the right is
- 17:02confirmed on the main
- 17:04node so that's going to slow things
- 17:08down but we'll make sure that all of the
- 17:11nodes have the same data this might be
- 17:14an option you can configure depending on
- 17:16the database you use or the hosting
- 17:18provider you use it might be the case
- 17:20that if you just want to do very simple
- 17:22replication automatic based on your
- 17:24cloud provider that they don't give you
- 17:26a ton of configuration options but you
- 17:29can do some research on your needs the
- 17:31different databases and different
- 17:32providers it's going to change if you're
- 17:34using Cloud databases or if you're
- 17:37hosting these yourself but I'm just
- 17:39giving you an overview of the different
- 17:40approaches and Concepts there's eventual
- 17:42consistency and then there's just
- 17:44consistent and if you want to know more
- 17:46about these different concerns check out
- 17:48my previous video on the cap theorem
- 17:50that's going to go in a lot more depth
- 17:51now another question you might have is
- 17:53if you have multiple read nodes how is
- 17:55the data distributed there are database
- 17:57load balancers but it's often the case
- 17:59that this load balancing is not done
- 18:01just out of the box for you so you can
- 18:04do different things like weighted DNS
- 18:06routing or using multiple connection
- 18:08strings so two reading connection
- 18:11strings and round robbing them in your
- 18:13code or you can look into some of the
- 18:16load balancers so ha proxy if you want
- 18:19to learn more about that or use a
- 18:21hosting service that does this kind of
- 18:23thing for you so for example Amazon
- 18:25Aurora postgress compatible is going to
- 18:28be able to to do that kind of load
- 18:29balancing for the reading all built in
- 18:32so there are options out there to do
- 18:34that kind of thing you'll just need to
- 18:35do some additional research now in the
- 18:37case of Aurora while if that's something
- 18:40you want to know a lot of more
- 18:41information about go look up dedicated
- 18:43content for that but that's something
- 18:45that can be provisioned saying hey
- 18:47here's the server size that I want or it
- 18:50could be serverless where it's basically
- 18:52just paid on a peruse basis so those are
- 18:56two different options and you can do
- 18:57research on that as well well guys
- 18:59that's pretty much everything I wanted
- 19:00to cover in this video well actually I
- 19:02had a bunch of other notes but I'm like
- 19:04I'm kind of getting bored so we'll end
- 19:05it there now when it comes to charting
- 19:07which we didn't talk about in tons of
- 19:08detail this is really just splitting up
- 19:10data across multiple nodes where it's
- 19:12not exactly one to one replication this
- 19:15is going to come up a lot more with no
- 19:16SQL databases which we'll talk about in
- 19:18more detail in the upcoming lessons
- 19:20basically with sharding we need to
- 19:21figure out how we split up data writing
- 19:23to some nodes not writing to some nodes
- 19:25and all of those different configuration
- 19:27options so we'll get into some of that
- 19:29in more detail as we go if you enjoy my
- 19:31content please be sure to subscribe I'll
- 19:33also have a link to the backend mind map
- 19:35which will be great if you want to know
- 19:37more about backend Technologies and I'll
- 19:39have a link to my mentorship program if
- 19:40you want to really level up your
- 19:42software engineering game and work with
- 19:44me closely so check that out as well and
- 19:46I'll see you in the next one
About this transcript
This page contains the full transcript of Replication and Sharding - System Design by Caleb Curry, generated from the public captions YouTube serves with the video. The transcript has 3,615 words across 520 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.