YouTube2Text

Replication and Sharding - System Design — Transcript

by Caleb Curry · 3,615 words · 520 segments · language en · Watch on YouTube

Full transcript

  1. 0:00hey what's going on everybody in this
  2. 0:01lesson we're going to be talking about
  3. 0:02replication and sharding for databases
  4. 0:05so how we go from a single database
  5. 0:07instance to multiple instances but
  6. 0:10everything working together as a single
  7. 0:12unit or you might hear the term cluster
  8. 0:14so a cluster is when you have multiple
  9. 0:17database
  10. 0:19instances so this is going to use
  11. 0:22technology such as replication and
  12. 0:24sharding this is going to probably be
  13. 0:26the last video before we start talking
  14. 0:29about no SQL databases and the different
  15. 0:31types and how they work I'm really
  16. 0:33excited for that video this is going to
  17. 0:35lay the foundation just to make sure we
  18. 0:37got everything covered before we go into
  19. 0:39that content I also want to show off my
  20. 0:41wife's beautiful new
  21. 0:45sweater
  22. 0:47just relax okay so when I explain the
  23. 0:51majority of these Concepts let's first
  24. 0:53apply this to relational databases so
  25. 0:56traditional databases think Oracle MySQL
  26. 0:59postra
  27. 1:00many of the nosql databases are more
  28. 1:03friendlier when it comes to splitting
  29. 1:04data up across multiple nodes so let's
  30. 1:07first think from a relational database
  31. 1:09point of view now we've talked about
  32. 1:11some of these principles in the earlier
  33. 1:13videos in this series but you don't
  34. 1:15necessarily have to go watch those if
  35. 1:17you're just jumping in here basically we
  36. 1:18have a computer it's running the
  37. 1:20database software this is our database
  38. 1:22instance and we first want to vertically
  39. 1:25scale our database instance so that
  40. 1:28means increasing the compute power of
  41. 1:30just this single server generally if we
  42. 1:32get to the point where we're working
  43. 1:34with multiple
  44. 1:36instances we don't want to be doing this
  45. 1:39with commodity hardware for traditional
  46. 1:41database systems what I mean by that is
  47. 1:43you don't want to be working with a
  48. 1:44bunch of average servers you want to be
  49. 1:47working with a really powerful server
  50. 1:49before you worry about splitting it up
  51. 1:51across multiple and this is adequate for
  52. 1:53many use cases if you want to do
  53. 1:55additional research you can look up
  54. 1:57sites or apps that have been very
  55. 2:00successful using just a single database
  56. 2:02instance and often before we have to
  57. 2:04worry about using multiple instances
  58. 2:06there are other things we can do to make
  59. 2:08things more efficient so that we're not
  60. 2:10putting as much pressure on that
  61. 2:12database server so an example of this
  62. 2:14would be caching we could cach data in
  63. 2:17something like reddis which can reduce
  64. 2:19the amount of requests we have to make
  65. 2:20to the database so here are two things
  66. 2:23that we can do we can reduce the
  67. 2:25database load with things like I just
  68. 2:27mentioned redus caching we can increase
  69. 2:31the single instance compute power so
  70. 2:34vertically scale the database instance
  71. 2:36so let's take a moment to talk about
  72. 2:38nosql and how we can actually help with
  73. 2:40this problem nosql is a category of
  74. 2:43databases so this doesn't describe a
  75. 2:44single database instance or a bunch of
  76. 2:47databases that all work exactly the same
  77. 2:49way and what that means is there's often
  78. 2:51a bunch of specialized databases that
  79. 2:53might help with specific things and this
  80. 2:55specialization means we might use a
  81. 2:57nosql database for a certain part part
  82. 3:00of our application or backend but not
  83. 3:02necessarily for just everything like we
  84. 3:05might use a relational database for
  85. 3:07sometimes that's a case there might be a
  86. 3:09nosql database that you decide hey this
  87. 3:11is going to be my main database for my
  88. 3:13data storage but often it's the case
  89. 3:15you'll have a main relational database
  90. 3:18but you might use nosql databases for
  91. 3:20some specific use cases and it's not
  92. 3:23going to store all of the data for
  93. 3:24everything so we've already talked about
  94. 3:26one of these which is redus
  95. 3:30or some kind of inmemory cache this
  96. 3:33could be classified as a nosql database
  97. 3:35so redus could be used for any type of
  98. 3:37data that doesn't need to be preserved
  99. 3:39forever another example is elastic
  100. 3:42search this is a database that is
  101. 3:45specialized in searching shocker right
  102. 3:48but there might be a scenario where you
  103. 3:50might be doing a lot of this searching
  104. 3:53and you don't want to have to throw that
  105. 3:54load on your main database so you can
  106. 3:56put the data that you need to work with
  107. 3:58into elastic search and then just use
  108. 4:00that database for that purpose there are
  109. 4:03also client side databases so this would
  110. 4:06be index DB so this is specifically
  111. 4:10referring to the web browser it has
  112. 4:13index DB available for data storage that
  113. 4:17doesn't need to be stored on the server
  114. 4:19so this might be something that you know
  115. 4:21is used actively but again doesn't need
  116. 4:24to be stored forever so maybe it's some
  117. 4:26kind of local game or some settings or
  118. 4:28something that don't don't require
  119. 4:30permanent storage but you want to store
  120. 4:32that information somewhere and one
  121. 4:34option for doing that would be to use a
  122. 4:37local database such as index DB so those
  123. 4:39are just three examples where putting
  124. 4:41some of the data in other locations
  125. 4:44might actually help to reduce the total
  126. 4:47required computation of the main
  127. 4:50relational database system you can also
  128. 4:52use a data warehouse so if you're doing
  129. 4:53a bunch of analytics and this is going
  130. 4:55to be computationally expensive it might
  131. 4:58make sense to actually bring the data
  132. 5:00from a relational standard database for
  133. 5:03transactional processing to a data
  134. 5:06warehouse and this data warehouse is
  135. 5:08more efficient for analytics so that
  136. 5:11will mean the main database is reserved
  137. 5:14for really just transactional processing
  138. 5:17which you'll also hear as
  139. 5:21oltp which is short for online
  140. 5:23transactional processing or for data
  141. 5:26warehousing you might hear oap for
  142. 5:28analytical process processing I guess
  143. 5:30I'll just write it over here however my
  144. 5:32big head might get in the way so we have
  145. 5:35transactional analytical so that's a
  146. 5:38variety of different ways you can reduce
  147. 5:40the demand of a single database instance
  148. 5:43now if this kind of stuff interests you
  149. 5:45and you want to know more that's what
  150. 5:47we're going to be talking about probably
  151. 5:48in the next lesson so stay tuned for
  152. 5:50that so we've gone through efforts to
  153. 5:52reduce the load on this database and
  154. 5:54it's finally good to go maybe there
  155. 5:57might actually be some other things that
  156. 5:58we want to do to to make sure that our
  157. 6:00database layer is good to go completely
  158. 6:02so now let's talk about how we can
  159. 6:04continue to improve our database layer
  160. 6:06we've already done what we could to
  161. 6:08reduce the demand on the database and
  162. 6:11we've made an effort to vertically scale
  163. 6:12the single database instance before
  164. 6:14having to worry about multiple database
  165. 6:17servers so what other concerns might we
  166. 6:19have well the first thing I want to
  167. 6:21mention does not actually reduce the
  168. 6:23total load on the database but it does
  169. 6:25help with data durability and high
  170. 6:27availability which is to have a fail
  171. 6:29over so we want to remove the single
  172. 6:32point of failure which is having a
  173. 6:34single instance that's probably my first
  174. 6:37biggest concern here is if we have a
  175. 6:39single instance and you know I don't
  176. 6:41know it explodes in a big fiery ball of
  177. 6:46fire even if we were able to recover
  178. 6:49this data the database instance might be
  179. 6:51offline longer than we would like so
  180. 6:54we'll first worry about high
  181. 6:55availability and then we're going to
  182. 6:57talk about some of the options to
  183. 6:58actually reduce comp computational load
  184. 7:00by introducing more servers if we go
  185. 7:04back in time before our database
  186. 7:06exploded we can now have a standin node
  187. 7:11and this is a database instance that's
  188. 7:13basically a copy of the current instance
  189. 7:17and the data is going to be replicated
  190. 7:20from one node to the other so this is
  191. 7:23where we introduced the idea of
  192. 7:24replication so with replication you can
  193. 7:26just think of data being copied over to
  194. 7:28another server this database is ready to
  195. 7:30go so if our main database did explode
  196. 7:34into a fiery ball of fire we would then
  197. 7:36be able to direct traffic to this other
  198. 7:38database without having any interruption
  199. 7:41in the user experience so we removed the
  200. 7:43single point of failure we're good to go
  201. 7:45so let's look at replication a little
  202. 7:47bit closer and we'll ask ourselves how
  203. 7:49does the second database get the data
  204. 7:52now obviously the answer is replication
  205. 7:55but let's talk a little bit in more
  206. 7:56detail and technically if you want to do
  207. 7:58some research you can look up the right
  208. 8:00ahead log which will get you started in
  209. 8:02this area so w so that's basically a log
  210. 8:06that all things that happen to the
  211. 8:07database goes to that log and that can
  212. 8:09then be used to replicate the same
  213. 8:11information to another instance but
  214. 8:14practically we write each incoming right
  215. 8:17to both
  216. 8:21databases and this is important to think
  217. 8:24about and understand there are two
  218. 8:26there's a microphone right here which is
  219. 8:28why I'm always like looking here so if
  220. 8:30I'm always just like staring off in the
  221. 8:32distance I just for some reason think
  222. 8:34that's like a camera or something I
  223. 8:36don't know I lost my train of thought
  224. 8:38what was I saying now this writing to
  225. 8:40both instances is done automatically for
  226. 8:42us so it's not like we actually have to
  227. 8:44write the query twice or something like
  228. 8:47that but the main thing to understand is
  229. 8:49that there are two rights in this
  230. 8:52process so we're not actually reducing
  231. 8:56the demand we're doubling the rights and
  232. 8:59we're doubling the number of instances
  233. 9:01that we write to so yes we doubled the
  234. 9:03compute having two instances but we're
  235. 9:05now doing twice as many wres and that's
  236. 9:07the reason why it doesn't even help us
  237. 9:10if we were to make both of these
  238. 9:12writable if you could write to either of
  239. 9:14these you might think hey that could
  240. 9:17double our computation power right
  241. 9:20basically horizontally scaling our
  242. 9:22database but if the data is synced
  243. 9:25across both of these then we're doing
  244. 9:27just double the amount of Rights so it
  245. 9:29doesn't matter that we now have two
  246. 9:31servers we're just making it more
  247. 9:33complicated so imagine each one of these
  248. 9:36I'm just making these numbers up but
  249. 9:37let's say could do a th000 transactions
  250. 9:40per second and then let's say we wanted
  251. 9:41to somehow make these both writable we
  252. 9:43might now be able to process 2,000
  253. 9:46transactions per second but a single
  254. 9:50right now becomes two
  255. 9:55rights so then this number is really
  256. 9:57just cut back down in half to a00
  257. 10:01long story short replication does not
  258. 10:04help with
  259. 10:05Io if it's a one: one replication and
  260. 10:09we'll talk about this more but the
  261. 10:11concept of replication will be used even
  262. 10:13if it's not a direct copy so when we get
  263. 10:16into sharding there still might be some
  264. 10:18replication it just might not be a
  265. 10:20complete duplicate on every single
  266. 10:23computer let's say this was your
  267. 10:24database server and you want to add more
  268. 10:27nodes but not just for the sake of
  269. 10:29having standin nodes but you actually
  270. 10:30want to be able to write to multiple
  271. 10:32nodes a multimaster database cluster or
  272. 10:35you might hear it as active active so
  273. 10:38now let's say instead of just a single
  274. 10:39node we have three nodes and we write to
  275. 10:43one of these nodes there might be
  276. 10:45replication from one node to the next
  277. 10:48node so that introduces redundancy it
  278. 10:51improves the
  279. 10:52durabil oh I just B had a heart
  280. 10:56attack Claire seriously so this
  281. 11:00increases the durability of our data
  282. 11:02because we now have redundancy if one of
  283. 11:04these catches on fire we still have a
  284. 11:07backup but we don't necessarily have to
  285. 11:09replicate it to all of the nodes so
  286. 11:11let's say maybe we write to this node
  287. 11:14and it's replicated to this node and
  288. 11:17then we write to this node and it's
  289. 11:19replicated to this node so for every
  290. 11:21write it's written to two nodes but we
  291. 11:26have three nodes of compute in this
  292. 11:28situation we process in total more
  293. 11:31transactions because we don't replicate
  294. 11:34the transaction to all of the nodes we
  295. 11:37only replicate it to some of the nodes
  296. 11:39so this is The Logical next step to
  297. 11:41database clustering however it does
  298. 11:43introduce a variety of challenges so if
  299. 11:45you're just working with traditional
  300. 11:48relational databases it is possible to
  301. 11:51do something like this and there's often
  302. 11:53different companies or services that
  303. 11:55will do this kind of thing for you so
  304. 11:56you can have multimaster replication and
  305. 11:59have a database cluster where you can
  306. 12:00write to multiple notes it helps with a
  307. 12:02variety of things first it helps with
  308. 12:04redundancy of data so you don't lose
  309. 12:06anything it helps with total compute
  310. 12:08power so you can increase the total
  311. 12:11write capacity and it could potentially
  312. 12:13help with latency because you can space
  313. 12:15out data appropriately where data is
  314. 12:17being requested so I'm definitely not
  315. 12:19saying it's not possible I'm just saying
  316. 12:21it's a new layer of complexity and
  317. 12:23really you usually don't want to go to
  318. 12:25this unless you're like an Enterprise
  319. 12:27level software where this is needed now
  320. 12:29when it comes to nosql databases there
  321. 12:31are some that were designed with this
  322. 12:33problem in mind so if you're going to go
  323. 12:35the route of having multiple nodes it
  324. 12:36might be worth considering a nosql
  325. 12:39database but there are challenges with
  326. 12:41that as well you have to think about how
  327. 12:43you're actually going to structure the
  328. 12:44data as it's usually a little bit
  329. 12:46different than a relational database so
  330. 12:48if you're brand new it might take a
  331. 12:49little bit of learning on how to
  332. 12:51properly structure that database so that
  333. 12:52data is located in the right location
  334. 12:56and data that needs to be accessed
  335. 12:58together can be accessed together and
  336. 13:00then there's all kinds of different
  337. 13:01configurations like the replication
  338. 13:02Factor how many times you want to
  339. 13:04replicate data there's also other Quorum
  340. 13:06settings that you'll need to consider so
  341. 13:08that you're writing and reading to the
  342. 13:10proper number of noes for your use case
  343. 13:13we'll get into that don't you guys worry
  344. 13:15I got your back new videos coming soon
  345. 13:18so be sure to subscribe and if you do
  346. 13:20enjoy this type of material and you want
  347. 13:21to get better break into software
  348. 13:23engineering or Advance your software
  349. 13:25engineering career I do work with people
  350. 13:27directly in my mentorship program so if
  351. 13:29that's something of interest to you I'll
  352. 13:30have a link down in the description
  353. 13:32check it out and hopefully we'll get to
  354. 13:34work together now let's go back to just
  355. 13:35relational databases before you go the
  356. 13:38clustering route there's some simple
  357. 13:40things you can do with replication that
  358. 13:41will make things better so say this is
  359. 13:43our database we can first like I
  360. 13:46mentioned add in a standin node oh my
  361. 13:50gosh that was awful if anything goes
  362. 13:52wrong with this guy you got a second one
  363. 13:54ready to go so that's basically a backup
  364. 13:57for computation a run database you can
  365. 14:00also back up your data which doesn't
  366. 14:03actually keep a database instance
  367. 14:05running ready to go it's really just the
  368. 14:08actual content the files so if you're a
  369. 14:10really high-tech company and you want to
  370. 14:12do some Cutting Edge stuff then yeah
  371. 14:14back up your data Additionally you can
  372. 14:17make replicas that are for
  373. 14:19reading this will offer a few benefits
  374. 14:22it will allow us to individually scale
  375. 14:24the reads and writs if we're directing
  376. 14:27reading to a separate instance
  377. 14:30basically the data will just be
  378. 14:32replicated from the rights to the read
  379. 14:35database and then we can adjust the
  380. 14:37compute power either up or
  381. 14:39down for these individual instances so
  382. 14:43we just really have to worry about being
  383. 14:44able to write fast enough on this node
  384. 14:47and then we can adjust this node as
  385. 14:48needed to meet the reading requirements
  386. 14:51same idea here so now we can actually
  387. 14:53horizontally scale the reading capacity
  388. 14:57thanks to replication from the main
  389. 14:59right node so you can vertically scale
  390. 15:01an individual node or you can add more
  391. 15:04read replicas a benefit of adding more
  392. 15:07of them is that you can spread them out
  393. 15:08in different areas so hey maybe this
  394. 15:10one's in the US maybe this one's in
  395. 15:12Europe so if you have a user and they're
  396. 15:15reading data from the website well
  397. 15:17that's going to be a lot faster than
  398. 15:19reading the data all the way from over
  399. 15:21here you could also have standin nodes
  400. 15:23for these guys as well if you needed so
  401. 15:25if for some reason you want to make sure
  402. 15:28that this node does not go down even
  403. 15:31though it's just for reading you can
  404. 15:33still have a stand in node to fill in if
  405. 15:35something were to go wrong this is less
  406. 15:37important than this guy here because if
  407. 15:40you didn't have the standing node and
  408. 15:42this exploded well you could just direct
  409. 15:45to a different read node and your
  410. 15:47application's still going to work now
  411. 15:49when you have this setup where you're
  412. 15:50writing to one main node and it's
  413. 15:53replicating down to read nodes this is
  414. 15:55usually eventually consistent and what
  415. 15:58this means is that the data on this node
  416. 16:01is going to match this data exactly it's
  417. 16:05just going to be a little bit delayed so
  418. 16:07say you write a blog post and it goes to
  419. 16:11this node it's then replicated down to
  420. 16:14this node but you're very fast right you
  421. 16:17write that blog post and you immediately
  422. 16:19check to see if that blog post is there
  423. 16:21maybe your read is routed to this node
  424. 16:24and it says oh no there's no blog post
  425. 16:26there you would say that that's
  426. 16:28inconsistent the the data doesn't match
  427. 16:30but it's eventually consistent so if you
  428. 16:32just wait long enough eventually that
  429. 16:34blog post is going to appear when you
  430. 16:35read and everything's fine that's okay
  431. 16:39in the majority of cases so social
  432. 16:41medias blogging websites most apps the
  433. 16:45eventual consistency is fine however
  434. 16:47there may be scenarios where that's not
  435. 16:48okay and you need to make sure that your
  436. 16:50replication is consistent which in that
  437. 16:52situation it would actually
  438. 16:55write to the read node
  439. 16:59and confirm that before the right is
  440. 17:02confirmed on the main
  441. 17:04node so that's going to slow things
  442. 17:08down but we'll make sure that all of the
  443. 17:11nodes have the same data this might be
  444. 17:14an option you can configure depending on
  445. 17:16the database you use or the hosting
  446. 17:18provider you use it might be the case
  447. 17:20that if you just want to do very simple
  448. 17:22replication automatic based on your
  449. 17:24cloud provider that they don't give you
  450. 17:26a ton of configuration options but you
  451. 17:29can do some research on your needs the
  452. 17:31different databases and different
  453. 17:32providers it's going to change if you're
  454. 17:34using Cloud databases or if you're
  455. 17:37hosting these yourself but I'm just
  456. 17:39giving you an overview of the different
  457. 17:40approaches and Concepts there's eventual
  458. 17:42consistency and then there's just
  459. 17:44consistent and if you want to know more
  460. 17:46about these different concerns check out
  461. 17:48my previous video on the cap theorem
  462. 17:50that's going to go in a lot more depth
  463. 17:51now another question you might have is
  464. 17:53if you have multiple read nodes how is
  465. 17:55the data distributed there are database
  466. 17:57load balancers but it's often the case
  467. 17:59that this load balancing is not done
  468. 18:01just out of the box for you so you can
  469. 18:04do different things like weighted DNS
  470. 18:06routing or using multiple connection
  471. 18:08strings so two reading connection
  472. 18:11strings and round robbing them in your
  473. 18:13code or you can look into some of the
  474. 18:16load balancers so ha proxy if you want
  475. 18:19to learn more about that or use a
  476. 18:21hosting service that does this kind of
  477. 18:23thing for you so for example Amazon
  478. 18:25Aurora postgress compatible is going to
  479. 18:28be able to to do that kind of load
  480. 18:29balancing for the reading all built in
  481. 18:32so there are options out there to do
  482. 18:34that kind of thing you'll just need to
  483. 18:35do some additional research now in the
  484. 18:37case of Aurora while if that's something
  485. 18:40you want to know a lot of more
  486. 18:41information about go look up dedicated
  487. 18:43content for that but that's something
  488. 18:45that can be provisioned saying hey
  489. 18:47here's the server size that I want or it
  490. 18:50could be serverless where it's basically
  491. 18:52just paid on a peruse basis so those are
  492. 18:56two different options and you can do
  493. 18:57research on that as well well guys
  494. 18:59that's pretty much everything I wanted
  495. 19:00to cover in this video well actually I
  496. 19:02had a bunch of other notes but I'm like
  497. 19:04I'm kind of getting bored so we'll end
  498. 19:05it there now when it comes to charting
  499. 19:07which we didn't talk about in tons of
  500. 19:08detail this is really just splitting up
  501. 19:10data across multiple nodes where it's
  502. 19:12not exactly one to one replication this
  503. 19:15is going to come up a lot more with no
  504. 19:16SQL databases which we'll talk about in
  505. 19:18more detail in the upcoming lessons
  506. 19:20basically with sharding we need to
  507. 19:21figure out how we split up data writing
  508. 19:23to some nodes not writing to some nodes
  509. 19:25and all of those different configuration
  510. 19:27options so we'll get into some of that
  511. 19:29in more detail as we go if you enjoy my
  512. 19:31content please be sure to subscribe I'll
  513. 19:33also have a link to the backend mind map
  514. 19:35which will be great if you want to know
  515. 19:37more about backend Technologies and I'll
  516. 19:39have a link to my mentorship program if
  517. 19:40you want to really level up your
  518. 19:42software engineering game and work with
  519. 19:44me closely so check that out as well and
  520. 19:46I'll see you in the next one

About this transcript

This page contains the full transcript of Replication and Sharding - System Design by Caleb Curry, generated from the public captions YouTube serves with the video. The transcript has 3,615 words across 520 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.