YouTube2Text

Intro to NoSQL Databases - System Design — Transcript

by Caleb Curry · 10,599 words · 1,535 segments · language en · Watch on YouTube

Full transcript

  1. 0:00welcome to your system design topic all
  2. 0:02about nosql databases this is going to
  3. 0:04be a fairly comprehensive introduction
  4. 0:07so we're going to talk about what a
  5. 0:09nosql database is how the data is
  6. 0:11structured how things work behind the
  7. 0:13scenes how they're different than
  8. 0:14relational databases all kinds of good
  9. 0:17stuff in this video now this is not a
  10. 0:19Hands-On video we are going to talk
  11. 0:21about some design for nosql databases
  12. 0:24but if you're looking for some Hands-On
  13. 0:26practice you have to check out the
  14. 0:27upcoming lessons but I will promise you
  15. 0:29that with this one video you will
  16. 0:31finally have a detailed to grasp on
  17. 0:33nosql databases we will also be talking
  18. 0:36about some of the most common nosql
  19. 0:38databases by name so you know where
  20. 0:40those databases fit in the nosql
  21. 0:43database world so you won't just leave
  22. 0:44this video knowing some basic concepts
  23. 0:46you'll know some of the most common
  24. 0:47databases and when you might want to use
  25. 0:49them so I'm really excited about this
  26. 0:51video I feel like the entire system
  27. 0:52design playlist has been building up to
  28. 0:55this moment right here and I'm going to
  29. 0:57interrupt it with a message from our
  30. 0:59sponsor which is my mentorship program
  31. 1:01so if you're ready to seriously level up
  32. 1:03your skill and your career in software
  33. 1:05engineering get a better grasp of
  34. 1:07backend systems and designing
  35. 1:09applications this mentorship program is
  36. 1:11for you you don't have to come in with
  37. 1:13any prerequisite knowledge we meet you
  38. 1:15where you are in terms of skill build up
  39. 1:18a plan a customized road map to get you
  40. 1:20to your goals so it's a really great
  41. 1:22opportunity if you're ready to level up
  42. 1:24put in the work and get to that next
  43. 1:25level of skill so I'll leave a link for
  44. 1:27that down below would highly encourage
  45. 1:29and appreciate you checking that out
  46. 1:31other than that if you're not interested
  47. 1:32in the mentorship program the least you
  48. 1:34can do for me is to subscribe because
  49. 1:36we're trying to get to a million Subs
  50. 1:37we're so close we just have like 330,000
  51. 1:40more to go not too many though so we're
  52. 1:42on our way so if you want to support
  53. 1:44this content be sure to subscribe and
  54. 1:45share it with another person so before
  55. 1:47we talk about no SQL databases
  56. 1:51nsql which by the way stands for not
  57. 1:53only SQL before we get into that we
  58. 1:56should first talk about SQL databases
  59. 2:00you might hear it as SQL I generally say
  60. 2:02SQL but really not a huge deal which one
  61. 2:04you say so this stands for structured
  62. 2:07query language so structured query
  63. 2:10language is a language
  64. 2:13standard for communicating to relational
  65. 2:18databases so this is an important word
  66. 2:20so
  67. 2:21relational the word relation here refers
  68. 2:25to
  69. 2:26tables commonly people will think that
  70. 2:29this refers to relationships between
  71. 2:31tables but really the term relation just
  72. 2:33means a table so this is a database that
  73. 2:37everything is stored in different tables
  74. 2:39now there are a bunch of different
  75. 2:40relational databases but they all
  76. 2:43typically are built off of the SQL
  77. 2:46standard language so working with them
  78. 2:48is pretty common there might be a few
  79. 2:50variations between one database and
  80. 2:52another but they're all pretty much
  81. 2:54built on this SQL standard so that
  82. 2:57summarizes SQL but let's talk about what
  83. 3:00it looks like to store data in tables
  84. 3:02before we talk about nosql we're not
  85. 3:04going to get into a ton of detail but we
  86. 3:05do want to make sure we have that
  87. 3:06Foundation built if you're already
  88. 3:08really comfortable with relational
  89. 3:09databases then you can just you know
  90. 3:12watch this to refresh your memory and
  91. 3:14we'll get into nosql in just a second a
  92. 3:17relational database will have tables and
  93. 3:19a table can be visualized as columns and
  94. 3:22rows so typically we'll have some column
  95. 3:26headers such as ID
  96. 3:30and then name and then let's say this is
  97. 3:33an account for a website we can just say
  98. 3:35if their accounts verified or not so
  99. 3:37we'll have an ID of one name
  100. 3:40Caleb verified is true we'll have
  101. 3:43another user two this will be
  102. 3:45John verified is false and another user
  103. 3:50three s will say that she is verified so
  104. 3:53these are
  105. 3:55rows so we have three rows
  106. 3:57currently each row having three three
  107. 4:00columns so you can see that it is a
  108. 4:03rectangle and every row has the same
  109. 4:06data the data is very structured so this
  110. 4:09follows a structure or what you will
  111. 4:11often hear as a
  112. 4:13schema in this situation every single
  113. 4:16record or row has an ID a name and then
  114. 4:19a Boolean true or false if we want to
  115. 4:22store certain information in a
  116. 4:24relational database sometimes we'll
  117. 4:25split it across multiple tables so let's
  118. 4:28say these are accounts for a website and
  119. 4:30we also want to store devices for that
  120. 4:32user so we could send push notifications
  121. 4:35for example we might have another table
  122. 4:37so let's say this is the user
  123. 4:39table we might have a devices
  124. 4:43table and this might have a device ID
  125. 4:46and then we'll have some way to refer
  126. 4:48back to the user table so we could have
  127. 4:50a user
  128. 4:52ID and then we'll just say device for
  129. 4:55Simplicity here so this would be the
  130. 4:57other table
  131. 5:01and it's going to look something like
  132. 5:02this where we might have a device ID 37
  133. 5:06user IDs 2 and then device would be I
  134. 5:09don't know Samsung or whatever doesn't
  135. 5:11really matter what the data here is the
  136. 5:13point is we have another table that
  137. 5:16points back to the original table this
  138. 5:20here describes a relationship and
  139. 5:23relationships build the entire structure
  140. 5:26of relational databases you're going to
  141. 5:27have data split up across multiple
  142. 5:29tables I've done a couple database
  143. 5:31design videos here on YouTube you may
  144. 5:33have seen them this is usually fine and
  145. 5:35in fact it's often recommended if you
  146. 5:37have data that nicely fits split up
  147. 5:40across multiple tables you have a bunch
  148. 5:41of data and it's connected in some way
  149. 5:45however there are scenarios where this
  150. 5:47is not ideal and this is where the nosql
  151. 5:51world comes
  152. 5:53in and in this situation you'll often
  153. 5:56not have multiple tables with
  154. 5:57relationships you will have nested data
  155. 6:00in a single collection or table and the
  156. 6:04way you design your data is a little bit
  157. 6:05different in this example you could
  158. 6:07imagine just taking the devices for a
  159. 6:09certain user and storing them with the
  160. 6:11user so the data might look like this
  161. 6:13the ID in this case will go with John is
  162. 6:15two the name is
  163. 6:19John verified is false and then we can
  164. 6:24have another devices
  165. 6:27attribute and this could just be nested
  166. 6:29data so we could have an array with the
  167. 6:32device information
  168. 6:34here and this could be a simple string
  169. 6:37or it could be an object
  170. 6:39itself so here it could be a primitive
  171. 6:42such as a string or a number or it could
  172. 6:45be an object where we basically have
  173. 6:47this data in a similar structure just
  174. 6:49now inside of this nested array that's
  175. 6:54associated with this devices key so this
  176. 6:56is pretty similar to what you might see
  177. 6:59with Json or objects in programming
  178. 7:02languages so Json this is one example of
  179. 7:06how you could represent data without
  180. 7:08having to have multiple tables but even
  181. 7:12in the scenario where the data you're
  182. 7:14working with is like this and this is
  183. 7:16what it looks like when you're building
  184. 7:18the application you still might split
  185. 7:20this out into individual tables at the
  186. 7:22database level so there's a disconnect
  187. 7:25between the way we work with the data in
  188. 7:27the application and the way it stores in
  189. 7:29the database and this layer is where you
  190. 7:31write SQL or you use some kind of object
  191. 7:34relational mapper kind of getting into
  192. 7:36the details there but the point is the
  193. 7:39way we work with the data is not always
  194. 7:41the same as how it's represented in the
  195. 7:43database but with a nosql database we
  196. 7:45could actually store it in the database
  197. 7:48like this instead of splitting it up
  198. 7:50across tables and now all of the
  199. 7:52information about John is together as a
  200. 7:55single unit let's go through another
  201. 7:57quick example of this where this is
  202. 7:58going to be really handy say you have
  203. 8:00some course website and you're going
  204. 8:02through the lessons we have lesson
  205. 8:06one and then we might have like a video
  206. 8:09player for it and you can just imagine
  207. 8:11this being displayed on a
  208. 8:13website maybe we have some notes for the
  209. 8:17video or maybe some
  210. 8:21downloads that's a paperclip like
  211. 8:23attachment it looks nothing like a
  212. 8:24download but you get the point lesson
  213. 8:26three we may have some kind of quiz or
  214. 8:29some information that's perhaps a little
  215. 8:31bit different than these other lessons
  216. 8:33and also as we're viewing this we might
  217. 8:35get some additional information on the
  218. 8:36side such as the author bio maybe a
  219. 8:40picture of
  220. 8:42them so some information about the
  221. 8:45course and you can imagine all of this
  222. 8:48information while it's being viewed
  223. 8:52together is actually stored in a bunch
  224. 8:54of different tables in the database so
  225. 8:57we might have a course table an
  226. 8:59Associated lesson table an attachments
  227. 9:02table for downloads we might have
  228. 9:04quizzes and basically we have a bunch of
  229. 9:06information spread out across a bunch of
  230. 9:08tables but when we are actually
  231. 9:10accessing the information on the web
  232. 9:12page it's all brought together in a
  233. 9:14single location so you can think of this
  234. 9:16almost as a single unit where this is
  235. 9:19kind of like a snapshot of the
  236. 9:22course so the question is if we always
  237. 9:25access this information together why do
  238. 9:27we then store it spread out in different
  239. 9:29tables wouldn't it just make sense to
  240. 9:31store it all in a single object in the
  241. 9:34database and sometimes it does make
  242. 9:36sense to do this putting it all together
  243. 9:38in the database other times it does not
  244. 9:40make sense to do it putting it all
  245. 9:42together in a database if you have a
  246. 9:44bunch of different structures and you
  247. 9:45need to access those structures in
  248. 9:47arbitrary ways you know maybe you need
  249. 9:49to grab all of the quizzes or all of the
  250. 9:52comments for the lessons or do some
  251. 9:55arbitrary queries where you don't know
  252. 9:57those queries ahead of time then it
  253. 10:00might not make sense to put everything
  254. 10:01together but there are scenarios where
  255. 10:03it does make sense to put everything
  256. 10:05together and in this scenario we can
  257. 10:07basically store all of this information
  258. 10:09in a single object that then goes into
  259. 10:12the database so this could be stored as
  260. 10:15a single unit into the database so what
  261. 10:20is accessed together is stored together
  262. 10:24access together stored together now I
  263. 10:27mentioned that sometimes it makes sense
  264. 10:28to do this sometimes it doesn't that's a
  265. 10:31design decision you will have a stronger
  266. 10:33intuition of when you should do
  267. 10:35something like this where you put it all
  268. 10:36together and when you should not by the
  269. 10:38end of this video but it's still
  270. 10:40something that's going to depend a lot
  271. 10:42on the application we'll get into some
  272. 10:44of the details of choosing a relational
  273. 10:46database over a nosql database and it
  274. 10:48often comes down to how the data is
  275. 10:50accessed and the scale of the
  276. 10:53application I want you now to build a
  277. 10:55stronger intuition and understanding of
  278. 10:57why people say no ql might be more
  279. 11:01scalable this is usually only important
  280. 11:04if you get to a scale where you have to
  281. 11:07do data sharding we've talked about
  282. 11:09sharding earlier on in this playlist and
  283. 11:12I'll draw a quick scenario of that so
  284. 11:15assume we are working with a relational
  285. 11:17database we might have a few nodes split
  286. 11:20up so each one of these is a database
  287. 11:23instance but they work together as a
  288. 11:25unit if we're working with a relational
  289. 11:27database and we split this data up
  290. 11:28across different nodes where every node
  291. 11:31doesn't have the full data set just a
  292. 11:34portion of the data set that's the idea
  293. 11:36behind
  294. 11:40charting the problem is that we might
  295. 11:44access one course but we end up having
  296. 11:47to retrieve information from three
  297. 11:50different nodes or multiple nodes across
  298. 11:52the world so the benefits of sharding
  299. 11:55are lost we basically split up the data
  300. 11:58so that we can have a higher throughput
  301. 12:02but now to read that data which is split
  302. 12:04up across multiple nodes we then have to
  303. 12:06access multiple nodes so we've lost the
  304. 12:09benefit of splitting it up in the first
  305. 12:11place I have no idea what's going on
  306. 12:12here but the idea is we have to access
  307. 12:14multiple nodes to build this individual
  308. 12:17page switching over now to
  309. 12:20nosql everything that is accessed
  310. 12:22together is stored together so that
  311. 12:25means even if we have a cluster of say
  312. 12:28three nodes
  313. 12:32all of this information is going to be
  314. 12:35accessed from just a single note because
  315. 12:37if you access just a piece of the
  316. 12:39information it all comes together it's
  317. 12:41stored as a single unit so we don't have
  318. 12:43to then go and access these other nodes
  319. 12:46which is obviously going to be better so
  320. 12:49that's one of the first benefits of
  321. 12:50nosql is the scalability and I'm not
  322. 12:53just trying to throw around database
  323. 12:55buzzwords saying you need to get a nosql
  324. 12:57database I'm just trying to help you
  325. 12:59understand why people say nosql
  326. 13:01databases are potentially more scalable
  327. 13:03it's because the data that is used
  328. 13:06together is stored together so nosql
  329. 13:09databases are typically cluster
  330. 13:13friendly and specifically talking about
  331. 13:16multimaster write and read notes not
  332. 13:20just read replication which is a bit of
  333. 13:22a different solution and doesn't
  334. 13:25actually increase the right throughput
  335. 13:27of a system so this is for both write
  336. 13:29and reads the next major benefit which
  337. 13:32has some drawbacks we're going to get
  338. 13:34into is that no SQL databases are often
  339. 13:37schema so let me write up here some
  340. 13:39example data and then I'll talk about
  341. 13:41what it means to be schema so here we
  342. 13:44have one
  343. 13:45object where we have a user ID a name
  344. 13:48and an email we could also have in here
  345. 13:52nested data so we could have the devices
  346. 13:54which is an array of device objects but
  347. 13:57I don't feel like writing all that out
  348. 13:58so this is what get the idea of this
  349. 14:00being schema is that this is one example
  350. 14:04of a user but we could have another user
  351. 14:07that follows a different schema so in
  352. 14:10this situation we have another user here
  353. 14:12that instead of having an email they
  354. 14:14have a phone number but here's the
  355. 14:16important thing it's not so much that
  356. 14:18this will have an email and it's just
  357. 14:19empty and this will have a phone number
  358. 14:21and it's just empty that would be the
  359. 14:23case for a table because we have to fill
  360. 14:25in every value in this situation because
  361. 14:29the attributes are actually associated
  362. 14:31with the values in the object we can
  363. 14:33just remove an attribute Al together so
  364. 14:36we don't have the email in here at all
  365. 14:38it's not empty it's not none it's not
  366. 14:40null it's just not there same thing up
  367. 14:43here we don't have a phone number at all
  368. 14:45so it's hard to actually write this in
  369. 14:47table form because every individual
  370. 14:50record could have different values
  371. 14:53however it's often the case that you
  372. 14:55could attempt to do this and get like a
  373. 14:57close representation of what it might
  374. 14:58look like even though it might not
  375. 15:00accurately represent how it's stored on
  376. 15:02the computer but if we wanted to do that
  377. 15:04it might look something like this where
  378. 15:05we have an ID and I'm just going to call
  379. 15:07this ID just to keep it shorter a
  380. 15:13name an
  381. 15:15email and a phone the first row will
  382. 15:18have values for the first three the next
  383. 15:20row will have values for the ID name and
  384. 15:23phone but these would just be empty this
  385. 15:26is how you could represent it in a table
  386. 15:29even if that's not how it's actually
  387. 15:30stored so sometimes even with the nosql
  388. 15:33databases they will give you some kind
  389. 15:34of table representation but we're not
  390. 15:36just storing a bunch of empty values
  391. 15:38we're just not storing values for those
  392. 15:39at all so the database being schema list
  393. 15:42can be a good thing it can be a bad
  394. 15:44thing let's talk about some of the pros
  395. 15:45and cons if you're prototyping some
  396. 15:48software you're trying to build quickly
  397. 15:50a schema database makes it a thousand
  398. 15:52times easier basically you just write
  399. 15:55how things should look in the software
  400. 15:57and you store it in the database it does
  401. 15:58not plan meaning if you wanted to add a
  402. 16:00new attribute you just add in the new
  403. 16:02attribute you don't have to worry about
  404. 16:04updating all of the previous records by
  405. 16:07adding a column as you would have to do
  406. 16:08in a relational database this allows
  407. 16:10people to very easily work through their
  408. 16:12application prototyping adding data
  409. 16:15changing data not having to think of the
  410. 16:17exact data structure up front the big
  411. 16:20downside to this is that you're kind of
  412. 16:23pushing certain problems down the road
  413. 16:25and they might come back to bite you so
  414. 16:28if you're drawing a web page and you're
  415. 16:30trying to present data that comes from
  416. 16:32the database now if you're unsure if
  417. 16:36that data is going to be there because
  418. 16:37it's a schema list design you have to
  419. 16:40conditionally render the possible
  420. 16:42attributes because there's a chance the
  421. 16:44attribute might not be there or there's
  422. 16:46a chance different attributes might be
  423. 16:48there so we're basically removing the
  424. 16:51static typing of the database and now we
  425. 16:53have to dynamically consider all of the
  426. 16:55possibilities at the application Level
  427. 16:58because even if a database is schema
  428. 17:01there's still an implicit schema data
  429. 17:04will generally follow some structure and
  430. 17:06that's how you decide what to display on
  431. 17:08a web page so if your data is all over
  432. 17:10the place and you're not following some
  433. 17:12schema the amount of work you're going
  434. 17:14to have to do at the application Level
  435. 17:16goes up so no schema over here means an
  436. 17:20increase of work on the application to
  437. 17:24be careful to not mess anything up you
  438. 17:27don't want the web page to say undefined
  439. 17:29or something weird being displayed or
  440. 17:31you don't want to throw any exceptions
  441. 17:32trying to access data that's not there
  442. 17:35so those are the major pros and cons I
  443. 17:37think people assume oh it's schema list
  444. 17:39it's going to be easier but really
  445. 17:41they're just putting the work at the
  446. 17:42application Level and later on they're
  447. 17:44not going to have as much trust in the
  448. 17:47database because there's not an enforced
  449. 17:49schema compare this to a scenario where
  450. 17:51we do have a
  451. 17:54schema and we label columns that are
  452. 17:58never supposed to be empty as not null
  453. 18:01we're working with a backend language
  454. 18:03that's statically typed meaning we know
  455. 18:06the types of the data we're expecting at
  456. 18:08compile time then overall the amount of
  457. 18:10work is going to go down at the
  458. 18:13application Level there might be a
  459. 18:14little bit more thinking at the database
  460. 18:17level so you might need to plan ahead a
  461. 18:19little
  462. 18:21bit so the more planning and the more
  463. 18:23prep you can do here the less issues you
  464. 18:26can have down the road that being said
  465. 18:28there are scenarios where things being
  466. 18:35scheessel in here that the user can put
  467. 18:38on the
  468. 18:39screen maybe it's a dashboard with
  469. 18:42different data or maybe they can plug in
  470. 18:45their own modules with some custom
  471. 18:48plugins in this situation it might be
  472. 18:50very handy to go for something schema
  473. 18:54because we don't necessarily know all
  474. 18:55the data structure that's going to be
  475. 18:57used for all of these different modules
  476. 18:59you might have to think a lot ahead of
  477. 19:01time to think of every possible scenario
  478. 19:03or you might have to continually update
  479. 19:05possible scenarios back in the database
  480. 19:07sounds like a pain in the butt hey maybe
  481. 19:09we could just go schema and let that be
  482. 19:11a little bit more fluid there however
  483. 19:14this doesn't mean you have to use a
  484. 19:16nosql database because a lot of the
  485. 19:18relational databases have introduced a
  486. 19:21schema component which is basically to
  487. 19:24store Json in a column so you might have
  488. 19:27a table for widgets where you have a
  489. 19:28widget ID and then the widget
  490. 19:32data and the data type for this is
  491. 19:36Json which does give you that
  492. 19:38flexibility but still within a database
  493. 19:41table so just if you want some schema
  494. 19:43list but you don't really want a full
  495. 19:45nosql database that's totally possible
  496. 19:47within a relational database and it's
  497. 19:50probably what I would first recommend
  498. 19:51versus jump and ship and going straight
  499. 19:53to a nosql database the other major
  500. 19:55thing that I would be aware of is that
  501. 19:57the design is often query driven so
  502. 20:01instead of asking how do I store this
  503. 20:04data you might ask how will I access
  504. 20:07this data so the relational databases
  505. 20:10are often pretty good at just doing
  506. 20:11arbitrary queries joining from whatever
  507. 20:13tables you need very very flexible
  508. 20:16whereas it's often the case with nosql
  509. 20:18databases you have to think ahead on
  510. 20:21what queries you're going to be using to
  511. 20:23access that data because the storage is
  512. 20:26a little bit more strict on what's
  513. 20:27possible for the queries now that's not
  514. 20:30always the case especially with nosql
  515. 20:32because there are tons of different
  516. 20:34databases within the nosql category but
  517. 20:37a quick example of this would say Hey
  518. 20:39you have to access by the key or by some
  519. 20:41index and you can't do any arbitrary
  520. 20:43queries that might not be 100% the case
  521. 20:46for the database you choose but that's a
  522. 20:48way to think about it where you have to
  523. 20:49think about what the queries are going
  524. 20:50to be ahead of time and if you need to
  525. 20:52change that then you might need to add
  526. 20:53in an index or change things a little
  527. 20:55bit so there's less rules like
  528. 20:57normalization and things like that with
  529. 20:59nosql but there are more design
  530. 21:02decisions about how you'll access the
  531. 21:04data and you'll become more familiar
  532. 21:06with that and become better at it as
  533. 21:08time goes on all right so at this point
  534. 21:10we have a pretty good foundational
  535. 21:12knowledge of nosql how it's different
  536. 21:14than SQL relational versus
  537. 21:16non-relational databases you might also
  538. 21:18hear structured versus unstructured now
  539. 21:20I want to talk about some of the
  540. 21:22different types of nosql databases
  541. 21:24because it's a large category there are
  542. 21:26a bunch of different types so what talk
  543. 21:28about some of the main ones there's
  544. 21:30probably others out there but this will
  545. 21:32cover most of nosql so I'm going to put
  546. 21:35four up here the
  547. 21:37first is key
  548. 21:41value and I'm just going to write all of
  549. 21:43them up here so you can know what we're
  550. 21:45going to be talking about the second is
  551. 21:47document databases the third is wide
  552. 21:51column and then the fourth here is graph
  553. 21:56databases and there's probably a couple
  554. 21:59others like maybe you could say time
  555. 22:00series databases databases specifically
  556. 22:03used for working with events in time or
  557. 22:07maybe uh search specific nosql databases
  558. 22:10ideal for searching through large data
  559. 22:12sets there's probably a bunch of other
  560. 22:14variations but this is going to get you
  561. 22:16the majority of scenarios so let's do a
  562. 22:18quick overview and then I will erase and
  563. 22:20we'll go into the details of each of
  564. 22:22these so key value document wide column
  565. 22:26these are all relatively similar where
  566. 22:28you'll have some key with some
  567. 22:30Associated data so for key value you can
  568. 22:33just imagine literally having a simple
  569. 22:35key for accessing the data and then an
  570. 22:39Associated value which can be quite
  571. 22:42complex such as an object document
  572. 22:44database is going to be pretty similar
  573. 22:46but it might have more insight into this
  574. 22:48value instead of just being like a
  575. 22:49snapshot of some data you might be able
  576. 22:51to query certain things within that
  577. 22:55value wide column you can imagine a
  578. 22:57relational database you have rows and
  579. 22:59columns the difference here though is
  580. 23:01that instead of every row having the
  581. 23:03same columns exactly a wide column you
  582. 23:06can have certain columns for certain
  583. 23:09rows and each row can have its own set
  584. 23:12of columns and then lastly graph
  585. 23:15databases you can visualize a graph as
  586. 23:17nodes and the connection between nodes
  587. 23:21so these are great for analyzing
  588. 23:23connections between data so you can
  589. 23:25imagine social networks or things like
  590. 23:28that so that's the highle overview
  591. 23:31probably didn't give you enough detail
  592. 23:32but I just wanted to show you what we're
  593. 23:35going to talk about now we're going to
  594. 23:36rewind talk about it in a little bit
  595. 23:38more detail so let's talk about key
  596. 23:40value databases to understand this
  597. 23:43database best you should first
  598. 23:45understand hashmaps a hashmap is a data
  599. 23:47structure that has a
  600. 23:50key with an Associated value and the key
  601. 23:54is actually used to determine the
  602. 23:56storage location here
  603. 24:00so typically this will be processed
  604. 24:02through a hash
  605. 24:04function and I don't know if we've
  606. 24:06talked about hash functions yet in this
  607. 24:08series but basically it's a function
  608. 24:10that takes an
  609. 24:12input and gives out a consistent length
  610. 24:18output we can go into the details of the
  611. 24:20different types of hash functions how
  612. 24:22they work but the idea is it can take
  613. 24:24any input and give out a consistent
  614. 24:26length output the same input will give
  615. 24:28you the same output so we have this
  616. 24:30hashing function and basically we'll
  617. 24:32take the key put it through that
  618. 24:33function and use this output to
  619. 24:36determine where the data is actually
  620. 24:39stored so it seems kind of complex if
  621. 24:42you're not familiar with this the reason
  622. 24:44why we do this is because it allows us
  623. 24:46to very quickly access data by a key so
  624. 24:51let's say we did this with the key five
  625. 24:53we would go through this process we
  626. 24:55would hash five take that output and
  627. 24:57then use that somehow to decide where
  628. 24:59that data is stored next time when we
  629. 25:01want to access five so not actually
  630. 25:03insert the data but actually retrieve
  631. 25:05that data we would just go through that
  632. 25:07same process we would process it through
  633. 25:09the hash function figure out what that
  634. 25:11output is and then use that to retrieve
  635. 25:14the exact location for the associated
  636. 25:17value so to repeat the key is the
  637. 25:20basically primary key used to identify
  638. 25:23the data but also the storage location
  639. 25:27is derived from that key through this
  640. 25:29hashing process so we know exactly where
  641. 25:31that data is stored on disk or in memory
  642. 25:34in the case of a data structure like a
  643. 25:36hash map when we take this idea and
  644. 25:38bring it to a database now the storage
  645. 25:40is on disk so we can go from key to
  646. 25:43exact value on
  647. 25:47disk in constant
  648. 25:50time so it's a very very quick operation
  649. 25:53because we don't have to search through
  650. 25:55any data we just process it through the
  651. 25:57hash it gives us the location that's all
  652. 25:59done for you probably automatically this
  653. 26:01is just how hashmaps work and key value
  654. 26:04databases are built on the ideas of
  655. 26:06hashmaps so if you've used dictionaries
  656. 26:09or associative arrays you're using
  657. 26:11hashmaps now you can just imagine
  658. 26:12instead of storing that data in memory
  659. 26:15you're just going to store it on disk
  660. 26:17and boom you have a key value database
  661. 26:19or in certain scenarios that data won't
  662. 26:21even be stored on disk so a popular
  663. 26:25example of this is redus redus is an
  664. 26:27inmemory cache and this will allow you
  665. 26:30to basically put a layer between the
  666. 26:34database and the application for
  667. 26:36commonly accessed data so reddis is my
  668. 26:39go-to when I'm thinking of key value
  669. 26:41data stores but there are plenty of
  670. 26:42other ones out there so another one that
  671. 26:44you could look into would be Dynamo DB
  672. 26:47now the thing to remember with key value
  673. 26:50databases as a general rule is that the
  674. 26:54value is seen as a single unit you can
  675. 26:58just think of it as a box that contains
  676. 27:00all the data you need the database is
  677. 27:03not intelligently aware of what's going
  678. 27:05on in this box you just give it a key it
  679. 27:08gives you back the thing at that Key's
  680. 27:10destination let's say this object that
  681. 27:13we were retrieving was a video on a
  682. 27:16website and then we had a
  683. 27:18comments and this comments might be a
  684. 27:21nested collection of a bunch of data
  685. 27:23this is something where you're
  686. 27:24interacting a couple of layers in you're
  687. 27:27not probably going to have a bunch of
  688. 27:28capabilities with the key value database
  689. 27:30to manipulate deeply into the uh nested
  690. 27:34data typically you're just going to work
  691. 27:35with it as a snapshot then work with
  692. 27:38that data in your application and then
  693. 27:40store the data back in the database if
  694. 27:43you wanted to do an update and most
  695. 27:45likely even if you did work with some
  696. 27:47attribute in the bigger object the
  697. 27:50bigger object is going to be accessed
  698. 27:52anyways so you're probably not going to
  699. 27:54save any write or read throughput by
  700. 27:57working at at a layer in of the nested
  701. 28:01object hopefully I'm not getting too
  702. 28:03abstract here it's hard to go through
  703. 28:04this without Hands-On examples hope to
  704. 28:06release some of those here soon so be
  705. 28:08sure to subscribe but for now the idea
  706. 28:10here is that you should think of the
  707. 28:12value just as a snapshot of some data
  708. 28:14you then work with it at the app
  709. 28:19level instead of doing a bunch of
  710. 28:21complex operations for certain
  711. 28:23attributes in that value so now let's
  712. 28:26talk about document databases this this
  713. 28:28is very similar or an idea but now you
  714. 28:30have a little bit more understanding of
  715. 28:32what the associated value is so these
  716. 28:35databases are much better for working
  717. 28:37with that nested data and that value
  718. 28:39being maybe a couple of layers deep so
  719. 28:41big examples of this would be mongodb I
  720. 28:44think couchdb would be another good
  721. 28:46example and I'm sure the cloud providers
  722. 28:48out there have some of their own
  723. 28:49document databases so there's probably
  724. 28:51more to this list but when I think of
  725. 28:53document databases mongodb is what comes
  726. 28:54to my mind this is a database where you
  727. 28:56can just access data by its ID but it
  728. 29:00also gives you a very strong query
  729. 29:01language where you can access nested
  730. 29:03data and manipulate a couple of layers
  731. 29:05in so the knowledge of the actual value
  732. 29:09associated with the key is much stronger
  733. 29:11so you could say that the value is
  734. 29:16transparent and the document database
  735. 29:18has knowledge of what's in that value
  736. 29:20and could potentially even index nested
  737. 29:23data so if you're looking for that key
  738. 29:25value like experience but you want to
  739. 29:27have complex control over the actual
  740. 29:31object the document then a document
  741. 29:34database makes sense for you next up we
  742. 29:36have wide column
  743. 29:38databases and I wanted to make sure I
  744. 29:41clarify that this is different than a
  745. 29:44column oriented database which is pretty
  746. 29:47common in any kind of data warehousing
  747. 29:50applications so if we basically took a
  748. 29:54relational database and instead of
  749. 29:56organizing data on disk by by row we
  750. 29:59organized it by column so instead of
  751. 30:01getting all of the data for one
  752. 30:03particular order together you might get
  753. 30:06all of the total order
  754. 30:09amounts and because this column is
  755. 30:11stored together on disk we can very
  756. 30:13easily do
  757. 30:18analytics with that data it's
  758. 30:22preorganized for convenient analytical
  759. 30:24processing so if you've heard of like o
  760. 30:28L A online analytical processing
  761. 30:31compared to O TP online transactional
  762. 30:36processing so this is a column oriented
  763. 30:38database data warehouse this is not what
  764. 30:40we're talking about here we're talking
  765. 30:41about nosql wide column data store and
  766. 30:45in this situation you're still going to
  767. 30:46have rows and for any row you'll have
  768. 30:49dedicated columns with
  769. 30:52values so it might look something like
  770. 30:54this but the difference here is between
  771. 30:56one row and the other you you can have
  772. 30:58different columns so you might store one
  773. 31:01thing for a certain row and store
  774. 31:02something else for another row the
  775. 31:04benefit here is that a column value is
  776. 31:06only written if there's a value for it
  777. 31:08compared to a relational database where
  778. 31:11every single row is going to have to
  779. 31:12have something for that column even if
  780. 31:15it's just going to be null so when we're
  781. 31:17talking wide column databases you're
  782. 31:19probably going to hear Google big
  783. 31:21table as well as Apache
  784. 31:24Cassandra these are really the defining
  785. 31:27dat databases you can check out the
  786. 31:29white papers for these okay I don't
  787. 31:32really like white papers they kind of
  788. 31:33put me to sleep not going to lie but
  789. 31:35there's actually some good stuff in some
  790. 31:37of these white papers about databases I
  791. 31:39use dark mode so it's not even a white
  792. 31:41paper anymore much better for your eyes
  793. 31:43by the way Apache Cassandra is really
  794. 31:45cool because as you scale instead of
  795. 31:47having a massive really powerful
  796. 31:51vertically scaled database instance
  797. 31:54Apache Cassandra is designed to go
  798. 31:56horizontal with commodity Hardware so
  799. 31:59it's perfectly able to run the database
  800. 32:02across many smaller weaker
  801. 32:05nodes and these are all connected in the
  802. 32:08database cluster so that's where Apache
  803. 32:11Cassandra really shines and the number
  804. 32:14of nodes you can have in one of these
  805. 32:16clusters is insane I don't know what the
  806. 32:18highest Apache Cassandra cluster has
  807. 32:21been but I've heard of clusters being in
  808. 32:23the tens of thousands of nodes all
  809. 32:26working together to store data to
  810. 32:28support some system so that's pretty
  811. 32:30wild I also wanted to mention
  812. 32:34hbas this is kind of like Google big
  813. 32:37table but in the Hadoop ecosystem so if
  814. 32:41you know anything about that you might
  815. 32:43be familiar with h base and then I also
  816. 32:45wanted to mention silid DB which is
  817. 32:48basically Apache Cassandra Java
  818. 32:50Rewritten in C++ now if you're doing
  819. 32:52additional research on nosql on the
  820. 32:54internet you'll probably find some
  821. 32:56written stuff and some videos by by
  822. 32:57Martin Fowler great stuff about nosql
  823. 33:00databases he categorizes these three as
  824. 33:03aggregate databases so what is an
  825. 33:06aggregate you can think of it as a
  826. 33:08logical single unit of information it's
  827. 33:11what we are trying to store and work
  828. 33:12with so in these three databases so far
  829. 33:16you will basically store going back to
  830. 33:18what I was saying everything that's
  831. 33:19accessed together together on disk so on
  832. 33:23his website he gives the example of an
  833. 33:26order with the line items even though
  834. 33:31these are logically two different things
  835. 33:35and would be split up into two tables
  836. 33:37probably in a relational database when
  837. 33:39we're talking about Aggregates the thing
  838. 33:42we're trying to discuss here is the
  839. 33:44order so this is going to be considered
  840. 33:47a single unit an aggregate in the
  841. 33:51scenario so you read and write
  842. 33:54Aggregates and in general you want to
  843. 33:56avoid operations that go outside of the
  844. 33:58aggregate boundary because then you have
  845. 34:00data Integrity risks unless you have
  846. 34:02some kind of transaction mechanism but
  847. 34:04updates to a single aggregate are Atomic
  848. 34:07so you can basically do anything with
  849. 34:09that Object Store it back on disk and
  850. 34:11you can rest assured that the full thing
  851. 34:14is going to be saved at once now graph
  852. 34:17databases were excluded because that has
  853. 34:19data split up across these relationships
  854. 34:21it's kind of like a relational database
  855. 34:24but instead it's split up as a graph
  856. 34:26we're not going to talk a ton on on
  857. 34:27graph databases we might do some
  858. 34:29dedicated content to it later down the
  859. 34:31road but for now you just need to
  860. 34:33understand that a graph database stores
  861. 34:35things as a graph where data is stored
  862. 34:38in nodes and they are connected neo4j is
  863. 34:41the biggest example of a graph database
  864. 34:43if you want to get some practical
  865. 34:44hands-on experience now I wanted to
  866. 34:46briefly mention something that you
  867. 34:48should be aware of when we're talking
  868. 34:50about nosql databases and that is the
  869. 34:52idea of a replication Factor as well as
  870. 34:55the concept of consistent hashing we're
  871. 34:57not going to get into the details of
  872. 35:00that in this lesson but this is a very
  873. 35:02common system design problem so it's
  874. 35:04probably worth doing additional research
  875. 35:06on it so we'll likely talk about this in
  876. 35:08a dedicated lesson coming soon but for
  877. 35:11now let's just get the highle overview
  878. 35:13basically since we are in a database
  879. 35:15system that splits the data up across
  880. 35:18multiple nodes we need to be careful
  881. 35:20that one node doesn't go out and we lose
  882. 35:22a portion of our data so consider a
  883. 35:25system where we have a couple of
  884. 35:26different database nodes and these are
  885. 35:29all connected in a database cluster and
  886. 35:32any of the nodes can write or read data
  887. 35:35so there is not just read replication
  888. 35:38you can write to any one of these
  889. 35:39however it's likely the case you won't
  890. 35:41just write to one you'll actually write
  891. 35:43to a few so you might start by writing
  892. 35:45to this one and then that data is
  893. 35:48replicated by the database to the next
  894. 35:51nodes so you don't just have the data in
  895. 35:54one location that's a single point of
  896. 35:56failure for that data so we want to
  897. 35:58replicate it we have some data
  898. 36:00redundancy in that case and our data is
  899. 36:02much safer same idea when we read data
  900. 36:05you might actually read from multiple
  901. 36:07nodes so you could read from both of
  902. 36:09these nodes confirming that the data
  903. 36:11exists on both of these nodes helping
  904. 36:13ensure that we get the most up-to-date
  905. 36:14data in this scenario there are three
  906. 36:17values that you should consider
  907. 36:18important the first being the
  908. 36:21replication Factor the next is the
  909. 36:24number of right acknowledgements and
  910. 36:26then the next is the number number of
  911. 36:28read
  912. 36:28acknowledgements so I'll go through an
  913. 36:31example again you don't have to
  914. 36:32understand all the details of this
  915. 36:33because we're going to do a dedicated
  916. 36:35lesson on it but consider an RF of three
  917. 36:39what this means is that data is
  918. 36:40replicated to three nodes in the cluster
  919. 36:43not every single node consider the case
  920. 36:45where you might have a cluster of
  921. 36:47thousands of nodes you don't put that
  922. 36:49data on every single node so a
  923. 36:51replication factor of three would mean
  924. 36:53that when we write to a database
  925. 36:55instance that data ultimately will be
  926. 36:57replicated to three nodes total the W
  927. 37:01here is how many we have to actually
  928. 37:03confirm have a successful right before
  929. 37:06we consider the right to be a success so
  930. 37:08consider the case where W is 2 we might
  931. 37:11write to this node and this node before
  932. 37:15we say hey the right was a success the
  933. 37:17read similar idea we might read from two
  934. 37:20nodes to confirm that the read was a
  935. 37:22success so we read from this node for
  936. 37:25example and this node for example with
  937. 37:27data replication that might be
  938. 37:29eventually consistent so something
  939. 37:31that's replicated from one node to the
  940. 37:33other will take some amount of time so
  941. 37:35because that right number is lower than
  942. 37:38the actual replication count it might
  943. 37:41not replicate to all of the RF nodes
  944. 37:44immediately it only needs to write to
  945. 37:45two of them for that operation to be
  946. 37:48considered successful the rest would be
  947. 37:49replicated afterwards so then the
  948. 37:52question is when we're reading how are
  949. 37:55we sure that we're getting the most up
  950. 37:57toate data and the formula for this is
  951. 37:59if the right acknowledgements plus the
  952. 38:01number of read acknowledgements is
  953. 38:03greater than the replication Factor then
  954. 38:07we should get consistent
  955. 38:12data because in that situation there's a
  956. 38:14guaranteed node overlap in that set of
  957. 38:19replication Factor nodes so in this
  958. 38:22situation we write to
  959. 38:24two and then we read from two and you
  960. 38:28can see there's an overlap right here so
  961. 38:32if we write to two and then immediately
  962. 38:33read from two we're sure that we're
  963. 38:35going to get the most up to-date data
  964. 38:37because at least one of the nodes has
  965. 38:39the newest data however if you don't
  966. 38:40need these requirements you could of
  967. 38:42course lower these numbers so we could
  968. 38:43say R is one in that situation we might
  969. 38:46read from a node that does not have the
  970. 38:49most up to-date data but if we don't
  971. 38:51needed to have the most up-to-date data
  972. 38:53this is going to be a faster operation
  973. 38:55so this is going to ultimately reduce
  974. 38:57latency ly and it's a trade-off between
  975. 38:59consistency and latency now there's also
  976. 39:00the concept of a coordinator so we make
  977. 39:03a request maybe we hit this node this
  978. 39:05node acts as a coordinator directing the
  979. 39:08request to the correct location where
  980. 39:11that data is stored but this brings up
  981. 39:13another question how do we know which
  982. 39:16node the data is stored in if we're
  983. 39:18given a bunch of nodes and we say hey
  984. 39:20find the associated value for this key
  985. 39:23how do we know where to access it so
  986. 39:25that's another question we're going to
  987. 39:26talk about in this has to do with some
  988. 39:28kind of algorithm to find the correct
  989. 39:31node typically it's going to be
  990. 39:33consistent hashing so we'll talk about
  991. 39:35what that means now this whole thing is
  992. 39:37called Quorum consensus and this is
  993. 39:39basically a system for multiple nodes to
  994. 39:42work together to give the correct data
  995. 39:44so they all have to be using the same
  996. 39:46consensus mechanism so that's what these
  997. 39:48parameters help Define now I want to
  998. 39:51briefly talk about the idea of
  999. 39:52consistent hashing let me bring back our
  1000. 39:55database cluster and we'll draw these in
  1001. 39:57a ring so you can just imagine a
  1002. 39:59circle and that's a pretty bad Circle
  1003. 40:03but if you remember back to when we had
  1004. 40:05the hashmap data
  1005. 40:08structure we had a key and a value and
  1006. 40:11when we were just working with memory
  1007. 40:12the key was used to determine the memory
  1008. 40:14location it's going to work very similar
  1009. 40:16here but now instead of some location in
  1010. 40:18memory it's going to be some location in
  1011. 40:20this database cluster so the way we get
  1012. 40:23that position is the key goes through
  1013. 40:26some kind of hash function function so
  1014. 40:28let's go through an example data just so
  1015. 40:30this can be a little bit more concrete
  1016. 40:32let's say the key is 1 2 3 and this has
  1017. 40:34my user data so username Caleb and so
  1018. 40:40forth we take this
  1019. 40:43key process it through a hash
  1020. 40:48function this gives us a hash value and
  1021. 40:52that hash value can be represented as a
  1022. 40:53number somewhere between zero and
  1023. 40:56whatever the max value is depending on
  1024. 40:59the hash function that's kind of
  1025. 41:01irrelevant the point is it gives us some
  1026. 41:03number back that can be used to
  1027. 41:05determine the location so let's just say
  1028. 41:09this circle represents the full range of
  1029. 41:12values we'll say it starts right here
  1030. 41:14this is zero and for Simplicity this is
  1031. 41:17going to be
  1032. 41:19100 this will be
  1033. 41:22200 right here will be
  1034. 41:25300 and here will be 4 400 and then
  1035. 41:28let's say we process our key through
  1036. 41:30this hash function and we get
  1037. 41:34255 and I know we you might be thinking
  1038. 41:36well why don't we just use this key here
  1039. 41:38well this doesn't have to be just a
  1040. 41:39simple number basically we need to take
  1041. 41:41some key process it through to get some
  1042. 41:45number that we can then use to determine
  1043. 41:47its location so where would 255 go on
  1044. 41:49this ring somewhere about right
  1045. 41:52here
  1046. 41:54255 and then by convention we basically
  1047. 41:57just put it on the server right here so
  1048. 42:00it's within this range so it goes on the
  1049. 42:03server clockwise on this ring so
  1050. 42:06basically in one quick operation we just
  1051. 42:08take the key hash it get the hash value
  1052. 42:12that tells us exactly which server that
  1053. 42:14that data will be stored on then going
  1054. 42:17to the replication Factor thing it's
  1055. 42:19going to be replicated to a couple more
  1056. 42:21notes so given this information if you
  1057. 42:24can even interpret my handwriting given
  1058. 42:27this key can you tell me how you would
  1059. 42:29find that information to read that
  1060. 42:31information not to make a write it's
  1061. 42:34going to be the same exact process so we
  1062. 42:36do a query for ID
  1063. 42:39123 behind the scenes that gets
  1064. 42:41processed through the hash function
  1065. 42:43it'll give us the same value that's a
  1066. 42:45critical part of the hash function the
  1067. 42:47same input needs to give the same output
  1068. 42:49which is true for hash functions so we
  1069. 42:51then can immediately jump to the correct
  1070. 42:54location in this database cluster there
  1071. 42:57still might be like a coordination node
  1072. 42:59so for example we might make the initial
  1073. 43:01request to say this node saying we're
  1074. 43:05trying to access key 1 2 3 and then this
  1075. 43:08node can do the magic calculation stuff
  1076. 43:10and say oh yeah here's exactly where you
  1077. 43:12need to go to get that data otherwise if
  1078. 43:15we didn't have this coordination node
  1079. 43:17then we would need some external system
  1080. 43:19to figure out which node to get the data
  1081. 43:22from so there might still be some
  1082. 43:24routing going on but it's a very quick
  1083. 43:27quick operation well that's the overview
  1084. 43:28of consistent hashing I kind of ruined
  1085. 43:30all the fun because now the next video
  1086. 43:33on these topics you'll be like I already
  1087. 43:35know all this stuff but we might be able
  1088. 43:37to go into some more depth I feel like
  1089. 43:39having a highle understanding of these
  1090. 43:40topics is good to know for nosql but I
  1091. 43:43don't really feel like you have to know
  1092. 43:45the inner details of everything in order
  1093. 43:47for you to understand nosql in general
  1094. 43:50and use it as a developer so I'm trying
  1095. 43:52to just give you a little bit here so
  1096. 43:53you have an idea but if we really want
  1097. 43:55to go into the details that would makes
  1098. 43:57sense to do in a dedicated lesson now
  1099. 43:59let's talk about another challenge with
  1100. 44:00nosql databases so we have the data
  1101. 44:04distributed across multiple nodes as
  1102. 44:06we've seen but we have to give some kind
  1103. 44:08of ID to identify how to find that
  1104. 44:11information in the database cluster so
  1105. 44:14in the previous example I used the key 1
  1106. 44:162 3 this ID has to come from somewhere
  1107. 44:21now if you're coming from a relational
  1108. 44:22database system you're probably familiar
  1109. 44:25with some kind of autoincrement
  1110. 44:27or
  1111. 44:28equivalent so this is not always a
  1112. 44:31native feature in nosql databases
  1113. 44:34because of the distributed nature of
  1114. 44:36nosql just consider the smallest example
  1115. 44:39of this if we have two
  1116. 44:41nodes and let's say the last used ID was
  1117. 44:441 2 3 but the data is split up across
  1118. 44:47these nodes so the highest on this one
  1119. 44:49might be 1 2 3 and maybe it has had the
  1120. 44:52last few rights so it might have 1 2 2 1
  1121. 44:562 1 one and this one the most latest
  1122. 44:59might be
  1123. 45:01120 so then let's say we write something
  1124. 45:04to this node what id do we use because
  1125. 45:07this node currently does not know that
  1126. 45:10123 exists over here on this node right
  1127. 45:13so it's not just going to naturally know
  1128. 45:15to use 124 unless these are sharing some
  1129. 45:19kind of coordinator node to distribute
  1130. 45:22IDs so that's one possible solution you
  1131. 45:24basically have an ID distributor
  1132. 45:27and this will basically take a request
  1133. 45:30and give back an ID for this
  1134. 45:34autoincrement feature so we basically
  1135. 45:36just have a centralized ID Creator so
  1136. 45:39when we have an external ID generator we
  1137. 45:42now have to basically move that ID
  1138. 45:44generation outside of the database so
  1139. 45:47often with nosql databases you will be
  1140. 45:49responsible at the application Level to
  1141. 45:51give an ID so we have a database node
  1142. 45:54and we're inserting data the insert is
  1143. 45:57going to have the ID
  1144. 46:01already so this comes from the
  1145. 46:03application not from the database so
  1146. 46:06that means our application can
  1147. 46:08communicate
  1148. 46:09to that centralized ID
  1149. 46:13server now there's problems to this
  1150. 46:16because now we have a single point of
  1151. 46:17failure we built a distributed database
  1152. 46:19and we're relying on a centralized
  1153. 46:21service so you might do something where
  1154. 46:24we actually load balance this and we
  1155. 46:25have two ID
  1156. 46:28servers and then maybe this one
  1157. 46:30distributes odds this one distributes
  1158. 46:32evens there are various Solutions I'm
  1159. 46:34sure but the main thing you need to
  1160. 46:36understand here is that it's no longer
  1161. 46:38the database's job to generate the ID
  1162. 46:41it's the application's job so the ID is
  1163. 46:44decided
  1164. 46:45here not here and that means anything
  1165. 46:48you can do at the application Level to
  1166. 46:50generate an ID could potentially be a
  1167. 46:52valid solution to this problem so we've
  1168. 46:54talked about one which is a central
  1169. 46:57ized Auto increment kind of complex
  1170. 47:01probably wouldn't want to do this if I
  1171. 47:02was just building some system and I
  1172. 47:04wanted to use a nosql database so there
  1173. 47:07are a couple of other really good
  1174. 47:08options the next option would be to use
  1175. 47:11some kind of natural
  1176. 47:15key so a natural key is great here
  1177. 47:18because it's associated with the data
  1178. 47:20that's being inserted so you don't have
  1179. 47:22to use any external resources so that's
  1180. 47:24a great option and that won't require an
  1181. 47:26external system you can do that all
  1182. 47:28within the application that has the data
  1183. 47:31you're working with right there the
  1184. 47:32third option would be to generate
  1185. 47:34something
  1186. 47:35random at the application Level that's
  1187. 47:38not Auto incremented so a uu
  1188. 47:42ID or you might hear guid universally
  1189. 47:46unique identifier and globally unique
  1190. 47:49identifier same idea basically this is
  1191. 47:51just a 128bit value that can uniquely
  1192. 47:54identify any data because with 128 bits
  1193. 47:58the value is large enough that the
  1194. 48:00chances of collision is extremely low so
  1195. 48:02this is great but the downside here is
  1196. 48:04it's not searchable very easily if
  1197. 48:06you're trying to access a value and you
  1198. 48:08have the ID already this is going to
  1199. 48:10work great because you can just access
  1200. 48:12that exact ID no problem but if you need
  1201. 48:15to search through the data there's no
  1202. 48:17logical sorting because the IDS are
  1203. 48:19random so that's the big downside here
  1204. 48:21as well as every single key taking up
  1205. 48:23128 bits even if you're working with
  1206. 48:25really small amounts of data every
  1207. 48:27single one is going to use that full bit
  1208. 48:29size the fourth solution which is a
  1209. 48:32little bit more complicated in terms of
  1210. 48:34understanding but this is pretty common
  1211. 48:37for different Services out there is to
  1212. 48:39use a
  1213. 48:42snowflake and this is a 64bit
  1214. 48:47value that's not random but actually has
  1215. 48:49some encoded information in that value
  1216. 48:53so something like one bit is a sign bit
  1217. 48:55that's reserved I think there's 41 bits
  1218. 48:58to a timestamp that actually indicates
  1219. 49:00when the ID was created 10 bits to the
  1220. 49:03system ID so whatever server is creating
  1221. 49:06the snowflake ID to avoid conflicts
  1222. 49:09across servers and then a 12-bit
  1223. 49:11sequence ID which is an auto
  1224. 49:13incrementing value that resets every
  1225. 49:14millisecond so you can create a lot of
  1226. 49:16values every millisecond so all those
  1227. 49:18put together will be 64bit but the value
  1228. 49:21means so much more than just a
  1229. 49:23128bit randomly generated number so this
  1230. 49:26is something you you'll see for large
  1231. 49:27applications that want to have some
  1232. 49:29identifying information inside of the ID
  1233. 49:33and they're going through additional
  1234. 49:34effort to save space and that
  1235. 49:37information is sortable because there's
  1236. 49:39a timestamp so there are some really
  1237. 49:40good benefits to the snowflake idea but
  1238. 49:43it does take a little bit of additional
  1239. 49:45understanding we might do some dedicated
  1240. 49:46content on that now to wrap things up I
  1241. 49:49wanted to go through two examples so
  1242. 49:51talking about how you might design
  1243. 49:53something in nosql so let's first start
  1244. 49:55talking about a video system so we're
  1245. 49:58creating some video platform and we have
  1246. 50:02a way for users to create comments so I
  1247. 50:05wanted to talk about this example
  1248. 50:07because this is going to introduce some
  1249. 50:09interesting things about Keys as well as
  1250. 50:12secondary indexes which are very
  1251. 50:13important in nosql so again thinking
  1252. 50:16ahead of what types of queries we might
  1253. 50:18be making so let's talk about what a
  1254. 50:20comment might look like we might have
  1255. 50:23the video
  1256. 50:25ID a
  1257. 50:27timestamp we're not going to be using
  1258. 50:30the snowflakes or any IDs with
  1259. 50:32timestamps embedded in them so we're
  1260. 50:34just going to have a video ID which
  1261. 50:35we'll just assume is a uu ID that was
  1262. 50:38generated nothing too fancy so we'll
  1263. 50:40have a separate Tim stamp value and then
  1264. 50:43we'll have a user
  1265. 50:46ID this is who posted the comment and
  1266. 50:49then we'll have the actual content of
  1267. 50:51the
  1268. 50:52comment so initially there's going to be
  1269. 50:56some challenges here because we don't
  1270. 50:57have relationships so this user ID this
  1271. 51:01video ID it does not connect to another
  1272. 51:04table with some kind of foreign key
  1273. 51:07constraint so if we want to have that
  1274. 51:09connection we have to do it at the
  1275. 51:11application Level basically we have an
  1276. 51:13app
  1277. 51:17here this accesses two tables from the
  1278. 51:20database or maybe
  1279. 51:22three
  1280. 51:23[Music]
  1281. 51:24users comments and videos and then we do
  1282. 51:28whatever we need with these three
  1283. 51:30individual sets of data at the
  1284. 51:31application Level so you just got to be
  1285. 51:33more careful with data Integrity if the
  1286. 51:35nosql system has some form of
  1287. 51:38transactions across tables you might
  1288. 51:39want to use those for certain scenarios
  1289. 51:42but the idea here is that this user ID
  1290. 51:44and this video ID are just values
  1291. 51:47they're not foreign keys to some other
  1292. 51:49tables and that's a big difference
  1293. 51:51inside of nosql compared to a relational
  1294. 51:53database this first attribute here is
  1295. 51:56often going to be called the partition
  1296. 51:59key and notice how we don't have a
  1297. 52:02comment ID here this might seem strange
  1298. 52:05but we have to think about how we are
  1299. 52:07accessing the data in this aggregate
  1300. 52:11oriented style of database design we
  1301. 52:14would access the video and then we need
  1302. 52:16to get all of the associated comments so
  1303. 52:19by setting the video ID as the partition
  1304. 52:22key in the comments we can access all of
  1305. 52:25the comments for particular video ID
  1306. 52:28very easily so if we had a comment ID it
  1307. 52:31could be here but it wouldn't be the
  1308. 52:34primary key of the table it would just
  1309. 52:36be another attribute so we basically are
  1310. 52:40going to group things by video ID and
  1311. 52:42that's the idea of a partition key so
  1312. 52:44things that are accessed together are
  1313. 52:46going to be stored together on the same
  1314. 52:48partition so I'm going to remove that
  1315. 52:49comment ID but we have another problem
  1316. 52:52here we still need a primary key which
  1317. 52:53is unique if we made it just the video
  1318. 52:56ID
  1319. 52:57that's not going to work because we have
  1320. 52:59multiple comments per video so we
  1321. 53:01actually need to use a composite primary
  1322. 53:03key which will consist of two values so
  1323. 53:06that's going to be the partition key
  1324. 53:08plus this here which is often called the
  1325. 53:12sort key or you might hear it as the
  1326. 53:15clustering
  1327. 53:17key and when you hear clustering here if
  1328. 53:19you're familiar from relational
  1329. 53:21databases the idea of a clustered index
  1330. 53:24it determines the sort of the data this
  1331. 53:27determines the order in which that data
  1332. 53:29is stored so these two combined can be
  1333. 53:32our primary key so we will not have any
  1334. 53:35duplicate values where we have the same
  1335. 53:37video ID with the same timestamp so this
  1336. 53:39helps us understand the basics of how we
  1337. 53:41would structure a table in a nosql
  1338. 53:44database now let's talk about secondary
  1339. 53:46indexes this is very important if you
  1340. 53:49want to access the data from a different
  1341. 53:52perspective such as all the comments
  1342. 53:54from a certain user so right now it's
  1343. 53:56highly inefficient because the data is
  1344. 53:59stored by video ID and timestamp we
  1345. 54:01would have to do a table scan to get all
  1346. 54:03of the comments with a certain ID so we
  1347. 54:06would add an index here and you can
  1348. 54:08imagine this index basically taking the
  1349. 54:11original structure of the data and
  1350. 54:14reorganizing it by a different key so
  1351. 54:18now we have the ability to retrieve by
  1352. 54:20video ID and we also have the ability to
  1353. 54:22retrieve by user ID however this would
  1354. 54:25obviously lead you to believe that
  1355. 54:26there's additional work happening here
  1356. 54:29so anytime you add an index that index
  1357. 54:31needs maintained you're basically
  1358. 54:33putting that data in two different
  1359. 54:35arrangements so think clearly about what
  1360. 54:38queries you really need so you can
  1361. 54:39create those secondary indexes but if
  1362. 54:41you're not using one of them then it's
  1363. 54:43not really a good idea to keep them
  1364. 54:44around just in case you could always add
  1365. 54:46an index later with some upfront work
  1366. 54:49but the idea here is we think ahead
  1367. 54:51about what queries we might be making
  1368. 54:53against our data now one more quick
  1369. 54:54comment about the information that I
  1370. 54:56just erased is if we have some ID that
  1371. 55:00has a Tim stamp or some kind of sort
  1372. 55:02embedded in it then we might not need a
  1373. 55:04dedicated Tim stamp for the clustering
  1374. 55:08key so for example if we were using
  1375. 55:10snowflakes in that situation we could
  1376. 55:12just use the comment ID as the
  1377. 55:14clustering key or the sort key instead
  1378. 55:16so it might look like this video ID and
  1379. 55:19then comment ID this would be our
  1380. 55:22primary key we could organize by by
  1381. 55:26video that's how the information would
  1382. 55:28be accessed we want the comments for a
  1383. 55:30certain video and then this being an ID
  1384. 55:35that has some sort information embedded
  1385. 55:38in it can be used to sort the data from
  1386. 55:42newest to oldest or whatever you
  1387. 55:47need this is basically the replacement
  1388. 55:50for the timestamp if the IDS have that
  1389. 55:53timestamp information embedded in them
  1390. 55:56so there's definitely benefits of using
  1391. 55:58something like a snowflake so I actually
  1392. 56:00wanted to do a second example of Discord
  1393. 56:03and a popular article online on how
  1394. 56:06Discord stores billions of messages
  1395. 56:08recently updated to how they store
  1396. 56:10trillions of messages they use
  1397. 56:12snowflakes in their system and their
  1398. 56:15primary key looks like
  1399. 56:19this channel ID message ID so very
  1400. 56:24similar situation here where we have
  1401. 56:27some Channel let's just say it's General
  1402. 56:30and we want to get all of the messages
  1403. 56:32for that channel so that's why this is
  1404. 56:34the partition key but we don't just want
  1405. 56:37the messages thrown in there we want
  1406. 56:39them to be sorted in some way so this
  1407. 56:42secondary part of the key is a
  1408. 56:44clustering key that determines the sort
  1409. 56:47and this
  1410. 56:49is a snowflake I believe this would be a
  1411. 56:52snowflake as well it's just not used for
  1412. 56:54the Sorting in this scenario so we take
  1413. 56:56the message IDs we can retrieve the
  1414. 56:59timestamp information from within the
  1415. 57:01snowflake to then give a sort from
  1416. 57:05newest at the
  1417. 57:07bottom scrolling up to the
  1418. 57:10oldest so that's a really quick
  1419. 57:14overview of discord's ID architecture
  1420. 57:20they used Cassandra they switched to syb
  1421. 57:22and now if you go read those articles
  1422. 57:24you'll have a pretty good foundational
  1423. 57:26to understand what they're talking about
  1424. 57:28and why it works for them I've given a
  1425. 57:30lot of information in this video and
  1426. 57:32this of course is not going to apply
  1427. 57:34universally to all nosql database types
  1428. 57:37so definitely do your research for the
  1429. 57:39individual database you might be working
  1430. 57:41with this is really just designed to
  1431. 57:43give you that really good foundation you
  1432. 57:46should now be able to go into nosql
  1433. 57:48databases with some level of confidence
  1434. 57:50as well as having more confidence going
  1435. 57:52into any of the white papers
  1436. 57:53understanding what they're talking about
  1437. 57:54with consistent hashing and things like
  1438. 57:56that so the final summary here is how do
  1439. 57:59you choose between a nosql database and
  1440. 58:02a relational database hopefully you have
  1441. 58:04a good understanding of that answer by
  1442. 58:06now ultimately it depends so hopefully
  1443. 58:10that helps one thing I would consider
  1444. 58:12doing though is wrapping your database
  1445. 58:15in a very easy to consume API or SDK so
  1446. 58:20you can imagine having your database
  1447. 58:22here and then wrapping this so that it's
  1448. 58:25super easy to interrup act with based on
  1449. 58:27the most common things you need to do so
  1450. 58:29for
  1451. 58:30example get users this could be a
  1452. 58:33function call if you're working with the
  1453. 58:35database directly in your application or
  1454. 58:38you could actually just expose different
  1455. 58:40API
  1456. 58:43endpoints such as get
  1457. 58:45users post a user basically writing out
  1458. 58:50all of the different interaction points
  1459. 58:52the interface to the database so that in
  1460. 58:55theory while it's not not as easy in
  1461. 58:56practice you could take out the database
  1462. 58:59substitute it in for something else so
  1463. 59:03you could even switch between SQL and
  1464. 59:06nosql but the applications that interact
  1465. 59:09with this
  1466. 59:11interface have the same interface and do
  1467. 59:14not have to change any of their code
  1468. 59:17because they're still going to get users
  1469. 59:19structured just how they would expect
  1470. 59:20and you can do any transformation logic
  1471. 59:22or anything you need to do here inside
  1472. 59:25of the data layer
  1473. 59:27so whether that's through an API or an
  1474. 59:28SDK or however you want to do it
  1475. 59:31basically adding some interface to
  1476. 59:34separate the consumption of the database
  1477. 59:37from the database itself can be very
  1478. 59:39handy for creating scalable systems so
  1479. 59:43that you don't break all of the
  1480. 59:44interacting applications if you switch
  1481. 59:46from one database to another so if that
  1482. 59:48doesn't make sense you can imagine you
  1483. 59:49have a bunch of different microservices
  1484. 59:51and all of these ultimately talk to the
  1485. 59:54database
  1486. 59:56starting out you might use the exact
  1487. 59:59database drivers that you're working
  1488. 1:00:00with so imagine you're working with some
  1489. 1:00:02nosql database and you're writing direct
  1490. 1:00:05for that database in all of these
  1491. 1:00:07individual applications but then you
  1492. 1:00:08want to switch it out you now have to go
  1493. 1:00:10into all of these different applications
  1494. 1:00:12updating that code which could be a
  1495. 1:00:15disaster and make it pretty much
  1496. 1:00:16impossible to switch out a database or
  1497. 1:00:18to do any major changes like that so
  1498. 1:00:21while I don't love abstraction for
  1499. 1:00:23abstraction sake just making things
  1500. 1:00:24complicated I do like the idea of
  1501. 1:00:27abstracting away the database itself and
  1502. 1:00:30just giving a nice interface so in that
  1503. 1:00:33situation all of these apps would only
  1504. 1:00:36talk to this outer
  1505. 1:00:39layer in this case I'd probably make it
  1506. 1:00:42an API so that we could do very easy
  1507. 1:00:45cross app
  1508. 1:00:46communication they just hit the API get
  1509. 1:00:49the data they need they don't care what
  1510. 1:00:51the database is or where the data is
  1511. 1:00:54stored they just know that if they hit
  1512. 1:00:55the API they get the data and they can
  1513. 1:00:58use it in their app so that's one
  1514. 1:01:00specific example you could probably
  1515. 1:01:02structure it differently but that's just
  1516. 1:01:04an overview of an approach that you
  1517. 1:01:05could use to basically reduce the
  1518. 1:01:09possibilities of things going south if
  1519. 1:01:11you need to make a change at the data
  1520. 1:01:13layer if you've made it this far you are
  1521. 1:01:15a legend I have to say thank you and I
  1522. 1:01:17would really appreciate a subscribe to
  1523. 1:01:19the channel if you enjoy this type of
  1524. 1:01:21content again check out the mentorship
  1525. 1:01:23program I'll have links for that down
  1526. 1:01:25below go through that if you really want
  1527. 1:01:27to level up your software engineering
  1528. 1:01:29game so that's the only two asks I have
  1529. 1:01:31for you subscribe check out the
  1530. 1:01:32mentorship program and I'll see you next
  1531. 1:01:34time if you have any suggestions on
  1532. 1:01:36additional content or things you would
  1533. 1:01:38like to hear about in more detail let me
  1534. 1:01:40know drop a comment would really
  1535. 1:01:42appreciate it

About this transcript

This page contains the full transcript of Intro to NoSQL Databases - System Design by Caleb Curry, generated from the public captions YouTube serves with the video. The transcript has 10,599 words across 1,535 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.