Intro to NoSQL Databases - System Design — Transcript
Full transcript
- 0:00welcome to your system design topic all
- 0:02about nosql databases this is going to
- 0:04be a fairly comprehensive introduction
- 0:07so we're going to talk about what a
- 0:09nosql database is how the data is
- 0:11structured how things work behind the
- 0:13scenes how they're different than
- 0:14relational databases all kinds of good
- 0:17stuff in this video now this is not a
- 0:19Hands-On video we are going to talk
- 0:21about some design for nosql databases
- 0:24but if you're looking for some Hands-On
- 0:26practice you have to check out the
- 0:27upcoming lessons but I will promise you
- 0:29that with this one video you will
- 0:31finally have a detailed to grasp on
- 0:33nosql databases we will also be talking
- 0:36about some of the most common nosql
- 0:38databases by name so you know where
- 0:40those databases fit in the nosql
- 0:43database world so you won't just leave
- 0:44this video knowing some basic concepts
- 0:46you'll know some of the most common
- 0:47databases and when you might want to use
- 0:49them so I'm really excited about this
- 0:51video I feel like the entire system
- 0:52design playlist has been building up to
- 0:55this moment right here and I'm going to
- 0:57interrupt it with a message from our
- 0:59sponsor which is my mentorship program
- 1:01so if you're ready to seriously level up
- 1:03your skill and your career in software
- 1:05engineering get a better grasp of
- 1:07backend systems and designing
- 1:09applications this mentorship program is
- 1:11for you you don't have to come in with
- 1:13any prerequisite knowledge we meet you
- 1:15where you are in terms of skill build up
- 1:18a plan a customized road map to get you
- 1:20to your goals so it's a really great
- 1:22opportunity if you're ready to level up
- 1:24put in the work and get to that next
- 1:25level of skill so I'll leave a link for
- 1:27that down below would highly encourage
- 1:29and appreciate you checking that out
- 1:31other than that if you're not interested
- 1:32in the mentorship program the least you
- 1:34can do for me is to subscribe because
- 1:36we're trying to get to a million Subs
- 1:37we're so close we just have like 330,000
- 1:40more to go not too many though so we're
- 1:42on our way so if you want to support
- 1:44this content be sure to subscribe and
- 1:45share it with another person so before
- 1:47we talk about no SQL databases
- 1:51nsql which by the way stands for not
- 1:53only SQL before we get into that we
- 1:56should first talk about SQL databases
- 2:00you might hear it as SQL I generally say
- 2:02SQL but really not a huge deal which one
- 2:04you say so this stands for structured
- 2:07query language so structured query
- 2:10language is a language
- 2:13standard for communicating to relational
- 2:18databases so this is an important word
- 2:20so
- 2:21relational the word relation here refers
- 2:25to
- 2:26tables commonly people will think that
- 2:29this refers to relationships between
- 2:31tables but really the term relation just
- 2:33means a table so this is a database that
- 2:37everything is stored in different tables
- 2:39now there are a bunch of different
- 2:40relational databases but they all
- 2:43typically are built off of the SQL
- 2:46standard language so working with them
- 2:48is pretty common there might be a few
- 2:50variations between one database and
- 2:52another but they're all pretty much
- 2:54built on this SQL standard so that
- 2:57summarizes SQL but let's talk about what
- 3:00it looks like to store data in tables
- 3:02before we talk about nosql we're not
- 3:04going to get into a ton of detail but we
- 3:05do want to make sure we have that
- 3:06Foundation built if you're already
- 3:08really comfortable with relational
- 3:09databases then you can just you know
- 3:12watch this to refresh your memory and
- 3:14we'll get into nosql in just a second a
- 3:17relational database will have tables and
- 3:19a table can be visualized as columns and
- 3:22rows so typically we'll have some column
- 3:26headers such as ID
- 3:30and then name and then let's say this is
- 3:33an account for a website we can just say
- 3:35if their accounts verified or not so
- 3:37we'll have an ID of one name
- 3:40Caleb verified is true we'll have
- 3:43another user two this will be
- 3:45John verified is false and another user
- 3:50three s will say that she is verified so
- 3:53these are
- 3:55rows so we have three rows
- 3:57currently each row having three three
- 4:00columns so you can see that it is a
- 4:03rectangle and every row has the same
- 4:06data the data is very structured so this
- 4:09follows a structure or what you will
- 4:11often hear as a
- 4:13schema in this situation every single
- 4:16record or row has an ID a name and then
- 4:19a Boolean true or false if we want to
- 4:22store certain information in a
- 4:24relational database sometimes we'll
- 4:25split it across multiple tables so let's
- 4:28say these are accounts for a website and
- 4:30we also want to store devices for that
- 4:32user so we could send push notifications
- 4:35for example we might have another table
- 4:37so let's say this is the user
- 4:39table we might have a devices
- 4:43table and this might have a device ID
- 4:46and then we'll have some way to refer
- 4:48back to the user table so we could have
- 4:50a user
- 4:52ID and then we'll just say device for
- 4:55Simplicity here so this would be the
- 4:57other table
- 5:01and it's going to look something like
- 5:02this where we might have a device ID 37
- 5:06user IDs 2 and then device would be I
- 5:09don't know Samsung or whatever doesn't
- 5:11really matter what the data here is the
- 5:13point is we have another table that
- 5:16points back to the original table this
- 5:20here describes a relationship and
- 5:23relationships build the entire structure
- 5:26of relational databases you're going to
- 5:27have data split up across multiple
- 5:29tables I've done a couple database
- 5:31design videos here on YouTube you may
- 5:33have seen them this is usually fine and
- 5:35in fact it's often recommended if you
- 5:37have data that nicely fits split up
- 5:40across multiple tables you have a bunch
- 5:41of data and it's connected in some way
- 5:45however there are scenarios where this
- 5:47is not ideal and this is where the nosql
- 5:51world comes
- 5:53in and in this situation you'll often
- 5:56not have multiple tables with
- 5:57relationships you will have nested data
- 6:00in a single collection or table and the
- 6:04way you design your data is a little bit
- 6:05different in this example you could
- 6:07imagine just taking the devices for a
- 6:09certain user and storing them with the
- 6:11user so the data might look like this
- 6:13the ID in this case will go with John is
- 6:15two the name is
- 6:19John verified is false and then we can
- 6:24have another devices
- 6:27attribute and this could just be nested
- 6:29data so we could have an array with the
- 6:32device information
- 6:34here and this could be a simple string
- 6:37or it could be an object
- 6:39itself so here it could be a primitive
- 6:42such as a string or a number or it could
- 6:45be an object where we basically have
- 6:47this data in a similar structure just
- 6:49now inside of this nested array that's
- 6:54associated with this devices key so this
- 6:56is pretty similar to what you might see
- 6:59with Json or objects in programming
- 7:02languages so Json this is one example of
- 7:06how you could represent data without
- 7:08having to have multiple tables but even
- 7:12in the scenario where the data you're
- 7:14working with is like this and this is
- 7:16what it looks like when you're building
- 7:18the application you still might split
- 7:20this out into individual tables at the
- 7:22database level so there's a disconnect
- 7:25between the way we work with the data in
- 7:27the application and the way it stores in
- 7:29the database and this layer is where you
- 7:31write SQL or you use some kind of object
- 7:34relational mapper kind of getting into
- 7:36the details there but the point is the
- 7:39way we work with the data is not always
- 7:41the same as how it's represented in the
- 7:43database but with a nosql database we
- 7:45could actually store it in the database
- 7:48like this instead of splitting it up
- 7:50across tables and now all of the
- 7:52information about John is together as a
- 7:55single unit let's go through another
- 7:57quick example of this where this is
- 7:58going to be really handy say you have
- 8:00some course website and you're going
- 8:02through the lessons we have lesson
- 8:06one and then we might have like a video
- 8:09player for it and you can just imagine
- 8:11this being displayed on a
- 8:13website maybe we have some notes for the
- 8:17video or maybe some
- 8:21downloads that's a paperclip like
- 8:23attachment it looks nothing like a
- 8:24download but you get the point lesson
- 8:26three we may have some kind of quiz or
- 8:29some information that's perhaps a little
- 8:31bit different than these other lessons
- 8:33and also as we're viewing this we might
- 8:35get some additional information on the
- 8:36side such as the author bio maybe a
- 8:40picture of
- 8:42them so some information about the
- 8:45course and you can imagine all of this
- 8:48information while it's being viewed
- 8:52together is actually stored in a bunch
- 8:54of different tables in the database so
- 8:57we might have a course table an
- 8:59Associated lesson table an attachments
- 9:02table for downloads we might have
- 9:04quizzes and basically we have a bunch of
- 9:06information spread out across a bunch of
- 9:08tables but when we are actually
- 9:10accessing the information on the web
- 9:12page it's all brought together in a
- 9:14single location so you can think of this
- 9:16almost as a single unit where this is
- 9:19kind of like a snapshot of the
- 9:22course so the question is if we always
- 9:25access this information together why do
- 9:27we then store it spread out in different
- 9:29tables wouldn't it just make sense to
- 9:31store it all in a single object in the
- 9:34database and sometimes it does make
- 9:36sense to do this putting it all together
- 9:38in the database other times it does not
- 9:40make sense to do it putting it all
- 9:42together in a database if you have a
- 9:44bunch of different structures and you
- 9:45need to access those structures in
- 9:47arbitrary ways you know maybe you need
- 9:49to grab all of the quizzes or all of the
- 9:52comments for the lessons or do some
- 9:55arbitrary queries where you don't know
- 9:57those queries ahead of time then it
- 10:00might not make sense to put everything
- 10:01together but there are scenarios where
- 10:03it does make sense to put everything
- 10:05together and in this scenario we can
- 10:07basically store all of this information
- 10:09in a single object that then goes into
- 10:12the database so this could be stored as
- 10:15a single unit into the database so what
- 10:20is accessed together is stored together
- 10:24access together stored together now I
- 10:27mentioned that sometimes it makes sense
- 10:28to do this sometimes it doesn't that's a
- 10:31design decision you will have a stronger
- 10:33intuition of when you should do
- 10:35something like this where you put it all
- 10:36together and when you should not by the
- 10:38end of this video but it's still
- 10:40something that's going to depend a lot
- 10:42on the application we'll get into some
- 10:44of the details of choosing a relational
- 10:46database over a nosql database and it
- 10:48often comes down to how the data is
- 10:50accessed and the scale of the
- 10:53application I want you now to build a
- 10:55stronger intuition and understanding of
- 10:57why people say no ql might be more
- 11:01scalable this is usually only important
- 11:04if you get to a scale where you have to
- 11:07do data sharding we've talked about
- 11:09sharding earlier on in this playlist and
- 11:12I'll draw a quick scenario of that so
- 11:15assume we are working with a relational
- 11:17database we might have a few nodes split
- 11:20up so each one of these is a database
- 11:23instance but they work together as a
- 11:25unit if we're working with a relational
- 11:27database and we split this data up
- 11:28across different nodes where every node
- 11:31doesn't have the full data set just a
- 11:34portion of the data set that's the idea
- 11:36behind
- 11:40charting the problem is that we might
- 11:44access one course but we end up having
- 11:47to retrieve information from three
- 11:50different nodes or multiple nodes across
- 11:52the world so the benefits of sharding
- 11:55are lost we basically split up the data
- 11:58so that we can have a higher throughput
- 12:02but now to read that data which is split
- 12:04up across multiple nodes we then have to
- 12:06access multiple nodes so we've lost the
- 12:09benefit of splitting it up in the first
- 12:11place I have no idea what's going on
- 12:12here but the idea is we have to access
- 12:14multiple nodes to build this individual
- 12:17page switching over now to
- 12:20nosql everything that is accessed
- 12:22together is stored together so that
- 12:25means even if we have a cluster of say
- 12:28three nodes
- 12:32all of this information is going to be
- 12:35accessed from just a single note because
- 12:37if you access just a piece of the
- 12:39information it all comes together it's
- 12:41stored as a single unit so we don't have
- 12:43to then go and access these other nodes
- 12:46which is obviously going to be better so
- 12:49that's one of the first benefits of
- 12:50nosql is the scalability and I'm not
- 12:53just trying to throw around database
- 12:55buzzwords saying you need to get a nosql
- 12:57database I'm just trying to help you
- 12:59understand why people say nosql
- 13:01databases are potentially more scalable
- 13:03it's because the data that is used
- 13:06together is stored together so nosql
- 13:09databases are typically cluster
- 13:13friendly and specifically talking about
- 13:16multimaster write and read notes not
- 13:20just read replication which is a bit of
- 13:22a different solution and doesn't
- 13:25actually increase the right throughput
- 13:27of a system so this is for both write
- 13:29and reads the next major benefit which
- 13:32has some drawbacks we're going to get
- 13:34into is that no SQL databases are often
- 13:37schema so let me write up here some
- 13:39example data and then I'll talk about
- 13:41what it means to be schema so here we
- 13:44have one
- 13:45object where we have a user ID a name
- 13:48and an email we could also have in here
- 13:52nested data so we could have the devices
- 13:54which is an array of device objects but
- 13:57I don't feel like writing all that out
- 13:58so this is what get the idea of this
- 14:00being schema is that this is one example
- 14:04of a user but we could have another user
- 14:07that follows a different schema so in
- 14:10this situation we have another user here
- 14:12that instead of having an email they
- 14:14have a phone number but here's the
- 14:16important thing it's not so much that
- 14:18this will have an email and it's just
- 14:19empty and this will have a phone number
- 14:21and it's just empty that would be the
- 14:23case for a table because we have to fill
- 14:25in every value in this situation because
- 14:29the attributes are actually associated
- 14:31with the values in the object we can
- 14:33just remove an attribute Al together so
- 14:36we don't have the email in here at all
- 14:38it's not empty it's not none it's not
- 14:40null it's just not there same thing up
- 14:43here we don't have a phone number at all
- 14:45so it's hard to actually write this in
- 14:47table form because every individual
- 14:50record could have different values
- 14:53however it's often the case that you
- 14:55could attempt to do this and get like a
- 14:57close representation of what it might
- 14:58look like even though it might not
- 15:00accurately represent how it's stored on
- 15:02the computer but if we wanted to do that
- 15:04it might look something like this where
- 15:05we have an ID and I'm just going to call
- 15:07this ID just to keep it shorter a
- 15:13name an
- 15:15email and a phone the first row will
- 15:18have values for the first three the next
- 15:20row will have values for the ID name and
- 15:23phone but these would just be empty this
- 15:26is how you could represent it in a table
- 15:29even if that's not how it's actually
- 15:30stored so sometimes even with the nosql
- 15:33databases they will give you some kind
- 15:34of table representation but we're not
- 15:36just storing a bunch of empty values
- 15:38we're just not storing values for those
- 15:39at all so the database being schema list
- 15:42can be a good thing it can be a bad
- 15:44thing let's talk about some of the pros
- 15:45and cons if you're prototyping some
- 15:48software you're trying to build quickly
- 15:50a schema database makes it a thousand
- 15:52times easier basically you just write
- 15:55how things should look in the software
- 15:57and you store it in the database it does
- 15:58not plan meaning if you wanted to add a
- 16:00new attribute you just add in the new
- 16:02attribute you don't have to worry about
- 16:04updating all of the previous records by
- 16:07adding a column as you would have to do
- 16:08in a relational database this allows
- 16:10people to very easily work through their
- 16:12application prototyping adding data
- 16:15changing data not having to think of the
- 16:17exact data structure up front the big
- 16:20downside to this is that you're kind of
- 16:23pushing certain problems down the road
- 16:25and they might come back to bite you so
- 16:28if you're drawing a web page and you're
- 16:30trying to present data that comes from
- 16:32the database now if you're unsure if
- 16:36that data is going to be there because
- 16:37it's a schema list design you have to
- 16:40conditionally render the possible
- 16:42attributes because there's a chance the
- 16:44attribute might not be there or there's
- 16:46a chance different attributes might be
- 16:48there so we're basically removing the
- 16:51static typing of the database and now we
- 16:53have to dynamically consider all of the
- 16:55possibilities at the application Level
- 16:58because even if a database is schema
- 17:01there's still an implicit schema data
- 17:04will generally follow some structure and
- 17:06that's how you decide what to display on
- 17:08a web page so if your data is all over
- 17:10the place and you're not following some
- 17:12schema the amount of work you're going
- 17:14to have to do at the application Level
- 17:16goes up so no schema over here means an
- 17:20increase of work on the application to
- 17:24be careful to not mess anything up you
- 17:27don't want the web page to say undefined
- 17:29or something weird being displayed or
- 17:31you don't want to throw any exceptions
- 17:32trying to access data that's not there
- 17:35so those are the major pros and cons I
- 17:37think people assume oh it's schema list
- 17:39it's going to be easier but really
- 17:41they're just putting the work at the
- 17:42application Level and later on they're
- 17:44not going to have as much trust in the
- 17:47database because there's not an enforced
- 17:49schema compare this to a scenario where
- 17:51we do have a
- 17:54schema and we label columns that are
- 17:58never supposed to be empty as not null
- 18:01we're working with a backend language
- 18:03that's statically typed meaning we know
- 18:06the types of the data we're expecting at
- 18:08compile time then overall the amount of
- 18:10work is going to go down at the
- 18:13application Level there might be a
- 18:14little bit more thinking at the database
- 18:17level so you might need to plan ahead a
- 18:19little
- 18:21bit so the more planning and the more
- 18:23prep you can do here the less issues you
- 18:26can have down the road that being said
- 18:28there are scenarios where things being
- 18:35scheessel in here that the user can put
- 18:38on the
- 18:39screen maybe it's a dashboard with
- 18:42different data or maybe they can plug in
- 18:45their own modules with some custom
- 18:48plugins in this situation it might be
- 18:50very handy to go for something schema
- 18:54because we don't necessarily know all
- 18:55the data structure that's going to be
- 18:57used for all of these different modules
- 18:59you might have to think a lot ahead of
- 19:01time to think of every possible scenario
- 19:03or you might have to continually update
- 19:05possible scenarios back in the database
- 19:07sounds like a pain in the butt hey maybe
- 19:09we could just go schema and let that be
- 19:11a little bit more fluid there however
- 19:14this doesn't mean you have to use a
- 19:16nosql database because a lot of the
- 19:18relational databases have introduced a
- 19:21schema component which is basically to
- 19:24store Json in a column so you might have
- 19:27a table for widgets where you have a
- 19:28widget ID and then the widget
- 19:32data and the data type for this is
- 19:36Json which does give you that
- 19:38flexibility but still within a database
- 19:41table so just if you want some schema
- 19:43list but you don't really want a full
- 19:45nosql database that's totally possible
- 19:47within a relational database and it's
- 19:50probably what I would first recommend
- 19:51versus jump and ship and going straight
- 19:53to a nosql database the other major
- 19:55thing that I would be aware of is that
- 19:57the design is often query driven so
- 20:01instead of asking how do I store this
- 20:04data you might ask how will I access
- 20:07this data so the relational databases
- 20:10are often pretty good at just doing
- 20:11arbitrary queries joining from whatever
- 20:13tables you need very very flexible
- 20:16whereas it's often the case with nosql
- 20:18databases you have to think ahead on
- 20:21what queries you're going to be using to
- 20:23access that data because the storage is
- 20:26a little bit more strict on what's
- 20:27possible for the queries now that's not
- 20:30always the case especially with nosql
- 20:32because there are tons of different
- 20:34databases within the nosql category but
- 20:37a quick example of this would say Hey
- 20:39you have to access by the key or by some
- 20:41index and you can't do any arbitrary
- 20:43queries that might not be 100% the case
- 20:46for the database you choose but that's a
- 20:48way to think about it where you have to
- 20:49think about what the queries are going
- 20:50to be ahead of time and if you need to
- 20:52change that then you might need to add
- 20:53in an index or change things a little
- 20:55bit so there's less rules like
- 20:57normalization and things like that with
- 20:59nosql but there are more design
- 21:02decisions about how you'll access the
- 21:04data and you'll become more familiar
- 21:06with that and become better at it as
- 21:08time goes on all right so at this point
- 21:10we have a pretty good foundational
- 21:12knowledge of nosql how it's different
- 21:14than SQL relational versus
- 21:16non-relational databases you might also
- 21:18hear structured versus unstructured now
- 21:20I want to talk about some of the
- 21:22different types of nosql databases
- 21:24because it's a large category there are
- 21:26a bunch of different types so what talk
- 21:28about some of the main ones there's
- 21:30probably others out there but this will
- 21:32cover most of nosql so I'm going to put
- 21:35four up here the
- 21:37first is key
- 21:41value and I'm just going to write all of
- 21:43them up here so you can know what we're
- 21:45going to be talking about the second is
- 21:47document databases the third is wide
- 21:51column and then the fourth here is graph
- 21:56databases and there's probably a couple
- 21:59others like maybe you could say time
- 22:00series databases databases specifically
- 22:03used for working with events in time or
- 22:07maybe uh search specific nosql databases
- 22:10ideal for searching through large data
- 22:12sets there's probably a bunch of other
- 22:14variations but this is going to get you
- 22:16the majority of scenarios so let's do a
- 22:18quick overview and then I will erase and
- 22:20we'll go into the details of each of
- 22:22these so key value document wide column
- 22:26these are all relatively similar where
- 22:28you'll have some key with some
- 22:30Associated data so for key value you can
- 22:33just imagine literally having a simple
- 22:35key for accessing the data and then an
- 22:39Associated value which can be quite
- 22:42complex such as an object document
- 22:44database is going to be pretty similar
- 22:46but it might have more insight into this
- 22:48value instead of just being like a
- 22:49snapshot of some data you might be able
- 22:51to query certain things within that
- 22:55value wide column you can imagine a
- 22:57relational database you have rows and
- 22:59columns the difference here though is
- 23:01that instead of every row having the
- 23:03same columns exactly a wide column you
- 23:06can have certain columns for certain
- 23:09rows and each row can have its own set
- 23:12of columns and then lastly graph
- 23:15databases you can visualize a graph as
- 23:17nodes and the connection between nodes
- 23:21so these are great for analyzing
- 23:23connections between data so you can
- 23:25imagine social networks or things like
- 23:28that so that's the highle overview
- 23:31probably didn't give you enough detail
- 23:32but I just wanted to show you what we're
- 23:35going to talk about now we're going to
- 23:36rewind talk about it in a little bit
- 23:38more detail so let's talk about key
- 23:40value databases to understand this
- 23:43database best you should first
- 23:45understand hashmaps a hashmap is a data
- 23:47structure that has a
- 23:50key with an Associated value and the key
- 23:54is actually used to determine the
- 23:56storage location here
- 24:00so typically this will be processed
- 24:02through a hash
- 24:04function and I don't know if we've
- 24:06talked about hash functions yet in this
- 24:08series but basically it's a function
- 24:10that takes an
- 24:12input and gives out a consistent length
- 24:18output we can go into the details of the
- 24:20different types of hash functions how
- 24:22they work but the idea is it can take
- 24:24any input and give out a consistent
- 24:26length output the same input will give
- 24:28you the same output so we have this
- 24:30hashing function and basically we'll
- 24:32take the key put it through that
- 24:33function and use this output to
- 24:36determine where the data is actually
- 24:39stored so it seems kind of complex if
- 24:42you're not familiar with this the reason
- 24:44why we do this is because it allows us
- 24:46to very quickly access data by a key so
- 24:51let's say we did this with the key five
- 24:53we would go through this process we
- 24:55would hash five take that output and
- 24:57then use that somehow to decide where
- 24:59that data is stored next time when we
- 25:01want to access five so not actually
- 25:03insert the data but actually retrieve
- 25:05that data we would just go through that
- 25:07same process we would process it through
- 25:09the hash function figure out what that
- 25:11output is and then use that to retrieve
- 25:14the exact location for the associated
- 25:17value so to repeat the key is the
- 25:20basically primary key used to identify
- 25:23the data but also the storage location
- 25:27is derived from that key through this
- 25:29hashing process so we know exactly where
- 25:31that data is stored on disk or in memory
- 25:34in the case of a data structure like a
- 25:36hash map when we take this idea and
- 25:38bring it to a database now the storage
- 25:40is on disk so we can go from key to
- 25:43exact value on
- 25:47disk in constant
- 25:50time so it's a very very quick operation
- 25:53because we don't have to search through
- 25:55any data we just process it through the
- 25:57hash it gives us the location that's all
- 25:59done for you probably automatically this
- 26:01is just how hashmaps work and key value
- 26:04databases are built on the ideas of
- 26:06hashmaps so if you've used dictionaries
- 26:09or associative arrays you're using
- 26:11hashmaps now you can just imagine
- 26:12instead of storing that data in memory
- 26:15you're just going to store it on disk
- 26:17and boom you have a key value database
- 26:19or in certain scenarios that data won't
- 26:21even be stored on disk so a popular
- 26:25example of this is redus redus is an
- 26:27inmemory cache and this will allow you
- 26:30to basically put a layer between the
- 26:34database and the application for
- 26:36commonly accessed data so reddis is my
- 26:39go-to when I'm thinking of key value
- 26:41data stores but there are plenty of
- 26:42other ones out there so another one that
- 26:44you could look into would be Dynamo DB
- 26:47now the thing to remember with key value
- 26:50databases as a general rule is that the
- 26:54value is seen as a single unit you can
- 26:58just think of it as a box that contains
- 27:00all the data you need the database is
- 27:03not intelligently aware of what's going
- 27:05on in this box you just give it a key it
- 27:08gives you back the thing at that Key's
- 27:10destination let's say this object that
- 27:13we were retrieving was a video on a
- 27:16website and then we had a
- 27:18comments and this comments might be a
- 27:21nested collection of a bunch of data
- 27:23this is something where you're
- 27:24interacting a couple of layers in you're
- 27:27not probably going to have a bunch of
- 27:28capabilities with the key value database
- 27:30to manipulate deeply into the uh nested
- 27:34data typically you're just going to work
- 27:35with it as a snapshot then work with
- 27:38that data in your application and then
- 27:40store the data back in the database if
- 27:43you wanted to do an update and most
- 27:45likely even if you did work with some
- 27:47attribute in the bigger object the
- 27:50bigger object is going to be accessed
- 27:52anyways so you're probably not going to
- 27:54save any write or read throughput by
- 27:57working at at a layer in of the nested
- 28:01object hopefully I'm not getting too
- 28:03abstract here it's hard to go through
- 28:04this without Hands-On examples hope to
- 28:06release some of those here soon so be
- 28:08sure to subscribe but for now the idea
- 28:10here is that you should think of the
- 28:12value just as a snapshot of some data
- 28:14you then work with it at the app
- 28:19level instead of doing a bunch of
- 28:21complex operations for certain
- 28:23attributes in that value so now let's
- 28:26talk about document databases this this
- 28:28is very similar or an idea but now you
- 28:30have a little bit more understanding of
- 28:32what the associated value is so these
- 28:35databases are much better for working
- 28:37with that nested data and that value
- 28:39being maybe a couple of layers deep so
- 28:41big examples of this would be mongodb I
- 28:44think couchdb would be another good
- 28:46example and I'm sure the cloud providers
- 28:48out there have some of their own
- 28:49document databases so there's probably
- 28:51more to this list but when I think of
- 28:53document databases mongodb is what comes
- 28:54to my mind this is a database where you
- 28:56can just access data by its ID but it
- 29:00also gives you a very strong query
- 29:01language where you can access nested
- 29:03data and manipulate a couple of layers
- 29:05in so the knowledge of the actual value
- 29:09associated with the key is much stronger
- 29:11so you could say that the value is
- 29:16transparent and the document database
- 29:18has knowledge of what's in that value
- 29:20and could potentially even index nested
- 29:23data so if you're looking for that key
- 29:25value like experience but you want to
- 29:27have complex control over the actual
- 29:31object the document then a document
- 29:34database makes sense for you next up we
- 29:36have wide column
- 29:38databases and I wanted to make sure I
- 29:41clarify that this is different than a
- 29:44column oriented database which is pretty
- 29:47common in any kind of data warehousing
- 29:50applications so if we basically took a
- 29:54relational database and instead of
- 29:56organizing data on disk by by row we
- 29:59organized it by column so instead of
- 30:01getting all of the data for one
- 30:03particular order together you might get
- 30:06all of the total order
- 30:09amounts and because this column is
- 30:11stored together on disk we can very
- 30:13easily do
- 30:18analytics with that data it's
- 30:22preorganized for convenient analytical
- 30:24processing so if you've heard of like o
- 30:28L A online analytical processing
- 30:31compared to O TP online transactional
- 30:36processing so this is a column oriented
- 30:38database data warehouse this is not what
- 30:40we're talking about here we're talking
- 30:41about nosql wide column data store and
- 30:45in this situation you're still going to
- 30:46have rows and for any row you'll have
- 30:49dedicated columns with
- 30:52values so it might look something like
- 30:54this but the difference here is between
- 30:56one row and the other you you can have
- 30:58different columns so you might store one
- 31:01thing for a certain row and store
- 31:02something else for another row the
- 31:04benefit here is that a column value is
- 31:06only written if there's a value for it
- 31:08compared to a relational database where
- 31:11every single row is going to have to
- 31:12have something for that column even if
- 31:15it's just going to be null so when we're
- 31:17talking wide column databases you're
- 31:19probably going to hear Google big
- 31:21table as well as Apache
- 31:24Cassandra these are really the defining
- 31:27dat databases you can check out the
- 31:29white papers for these okay I don't
- 31:32really like white papers they kind of
- 31:33put me to sleep not going to lie but
- 31:35there's actually some good stuff in some
- 31:37of these white papers about databases I
- 31:39use dark mode so it's not even a white
- 31:41paper anymore much better for your eyes
- 31:43by the way Apache Cassandra is really
- 31:45cool because as you scale instead of
- 31:47having a massive really powerful
- 31:51vertically scaled database instance
- 31:54Apache Cassandra is designed to go
- 31:56horizontal with commodity Hardware so
- 31:59it's perfectly able to run the database
- 32:02across many smaller weaker
- 32:05nodes and these are all connected in the
- 32:08database cluster so that's where Apache
- 32:11Cassandra really shines and the number
- 32:14of nodes you can have in one of these
- 32:16clusters is insane I don't know what the
- 32:18highest Apache Cassandra cluster has
- 32:21been but I've heard of clusters being in
- 32:23the tens of thousands of nodes all
- 32:26working together to store data to
- 32:28support some system so that's pretty
- 32:30wild I also wanted to mention
- 32:34hbas this is kind of like Google big
- 32:37table but in the Hadoop ecosystem so if
- 32:41you know anything about that you might
- 32:43be familiar with h base and then I also
- 32:45wanted to mention silid DB which is
- 32:48basically Apache Cassandra Java
- 32:50Rewritten in C++ now if you're doing
- 32:52additional research on nosql on the
- 32:54internet you'll probably find some
- 32:56written stuff and some videos by by
- 32:57Martin Fowler great stuff about nosql
- 33:00databases he categorizes these three as
- 33:03aggregate databases so what is an
- 33:06aggregate you can think of it as a
- 33:08logical single unit of information it's
- 33:11what we are trying to store and work
- 33:12with so in these three databases so far
- 33:16you will basically store going back to
- 33:18what I was saying everything that's
- 33:19accessed together together on disk so on
- 33:23his website he gives the example of an
- 33:26order with the line items even though
- 33:31these are logically two different things
- 33:35and would be split up into two tables
- 33:37probably in a relational database when
- 33:39we're talking about Aggregates the thing
- 33:42we're trying to discuss here is the
- 33:44order so this is going to be considered
- 33:47a single unit an aggregate in the
- 33:51scenario so you read and write
- 33:54Aggregates and in general you want to
- 33:56avoid operations that go outside of the
- 33:58aggregate boundary because then you have
- 34:00data Integrity risks unless you have
- 34:02some kind of transaction mechanism but
- 34:04updates to a single aggregate are Atomic
- 34:07so you can basically do anything with
- 34:09that Object Store it back on disk and
- 34:11you can rest assured that the full thing
- 34:14is going to be saved at once now graph
- 34:17databases were excluded because that has
- 34:19data split up across these relationships
- 34:21it's kind of like a relational database
- 34:24but instead it's split up as a graph
- 34:26we're not going to talk a ton on on
- 34:27graph databases we might do some
- 34:29dedicated content to it later down the
- 34:31road but for now you just need to
- 34:33understand that a graph database stores
- 34:35things as a graph where data is stored
- 34:38in nodes and they are connected neo4j is
- 34:41the biggest example of a graph database
- 34:43if you want to get some practical
- 34:44hands-on experience now I wanted to
- 34:46briefly mention something that you
- 34:48should be aware of when we're talking
- 34:50about nosql databases and that is the
- 34:52idea of a replication Factor as well as
- 34:55the concept of consistent hashing we're
- 34:57not going to get into the details of
- 35:00that in this lesson but this is a very
- 35:02common system design problem so it's
- 35:04probably worth doing additional research
- 35:06on it so we'll likely talk about this in
- 35:08a dedicated lesson coming soon but for
- 35:11now let's just get the highle overview
- 35:13basically since we are in a database
- 35:15system that splits the data up across
- 35:18multiple nodes we need to be careful
- 35:20that one node doesn't go out and we lose
- 35:22a portion of our data so consider a
- 35:25system where we have a couple of
- 35:26different database nodes and these are
- 35:29all connected in a database cluster and
- 35:32any of the nodes can write or read data
- 35:35so there is not just read replication
- 35:38you can write to any one of these
- 35:39however it's likely the case you won't
- 35:41just write to one you'll actually write
- 35:43to a few so you might start by writing
- 35:45to this one and then that data is
- 35:48replicated by the database to the next
- 35:51nodes so you don't just have the data in
- 35:54one location that's a single point of
- 35:56failure for that data so we want to
- 35:58replicate it we have some data
- 36:00redundancy in that case and our data is
- 36:02much safer same idea when we read data
- 36:05you might actually read from multiple
- 36:07nodes so you could read from both of
- 36:09these nodes confirming that the data
- 36:11exists on both of these nodes helping
- 36:13ensure that we get the most up-to-date
- 36:14data in this scenario there are three
- 36:17values that you should consider
- 36:18important the first being the
- 36:21replication Factor the next is the
- 36:24number of right acknowledgements and
- 36:26then the next is the number number of
- 36:28read
- 36:28acknowledgements so I'll go through an
- 36:31example again you don't have to
- 36:32understand all the details of this
- 36:33because we're going to do a dedicated
- 36:35lesson on it but consider an RF of three
- 36:39what this means is that data is
- 36:40replicated to three nodes in the cluster
- 36:43not every single node consider the case
- 36:45where you might have a cluster of
- 36:47thousands of nodes you don't put that
- 36:49data on every single node so a
- 36:51replication factor of three would mean
- 36:53that when we write to a database
- 36:55instance that data ultimately will be
- 36:57replicated to three nodes total the W
- 37:01here is how many we have to actually
- 37:03confirm have a successful right before
- 37:06we consider the right to be a success so
- 37:08consider the case where W is 2 we might
- 37:11write to this node and this node before
- 37:15we say hey the right was a success the
- 37:17read similar idea we might read from two
- 37:20nodes to confirm that the read was a
- 37:22success so we read from this node for
- 37:25example and this node for example with
- 37:27data replication that might be
- 37:29eventually consistent so something
- 37:31that's replicated from one node to the
- 37:33other will take some amount of time so
- 37:35because that right number is lower than
- 37:38the actual replication count it might
- 37:41not replicate to all of the RF nodes
- 37:44immediately it only needs to write to
- 37:45two of them for that operation to be
- 37:48considered successful the rest would be
- 37:49replicated afterwards so then the
- 37:52question is when we're reading how are
- 37:55we sure that we're getting the most up
- 37:57toate data and the formula for this is
- 37:59if the right acknowledgements plus the
- 38:01number of read acknowledgements is
- 38:03greater than the replication Factor then
- 38:07we should get consistent
- 38:12data because in that situation there's a
- 38:14guaranteed node overlap in that set of
- 38:19replication Factor nodes so in this
- 38:22situation we write to
- 38:24two and then we read from two and you
- 38:28can see there's an overlap right here so
- 38:32if we write to two and then immediately
- 38:33read from two we're sure that we're
- 38:35going to get the most up to-date data
- 38:37because at least one of the nodes has
- 38:39the newest data however if you don't
- 38:40need these requirements you could of
- 38:42course lower these numbers so we could
- 38:43say R is one in that situation we might
- 38:46read from a node that does not have the
- 38:49most up to-date data but if we don't
- 38:51needed to have the most up-to-date data
- 38:53this is going to be a faster operation
- 38:55so this is going to ultimately reduce
- 38:57latency ly and it's a trade-off between
- 38:59consistency and latency now there's also
- 39:00the concept of a coordinator so we make
- 39:03a request maybe we hit this node this
- 39:05node acts as a coordinator directing the
- 39:08request to the correct location where
- 39:11that data is stored but this brings up
- 39:13another question how do we know which
- 39:16node the data is stored in if we're
- 39:18given a bunch of nodes and we say hey
- 39:20find the associated value for this key
- 39:23how do we know where to access it so
- 39:25that's another question we're going to
- 39:26talk about in this has to do with some
- 39:28kind of algorithm to find the correct
- 39:31node typically it's going to be
- 39:33consistent hashing so we'll talk about
- 39:35what that means now this whole thing is
- 39:37called Quorum consensus and this is
- 39:39basically a system for multiple nodes to
- 39:42work together to give the correct data
- 39:44so they all have to be using the same
- 39:46consensus mechanism so that's what these
- 39:48parameters help Define now I want to
- 39:51briefly talk about the idea of
- 39:52consistent hashing let me bring back our
- 39:55database cluster and we'll draw these in
- 39:57a ring so you can just imagine a
- 39:59circle and that's a pretty bad Circle
- 40:03but if you remember back to when we had
- 40:05the hashmap data
- 40:08structure we had a key and a value and
- 40:11when we were just working with memory
- 40:12the key was used to determine the memory
- 40:14location it's going to work very similar
- 40:16here but now instead of some location in
- 40:18memory it's going to be some location in
- 40:20this database cluster so the way we get
- 40:23that position is the key goes through
- 40:26some kind of hash function function so
- 40:28let's go through an example data just so
- 40:30this can be a little bit more concrete
- 40:32let's say the key is 1 2 3 and this has
- 40:34my user data so username Caleb and so
- 40:40forth we take this
- 40:43key process it through a hash
- 40:48function this gives us a hash value and
- 40:52that hash value can be represented as a
- 40:53number somewhere between zero and
- 40:56whatever the max value is depending on
- 40:59the hash function that's kind of
- 41:01irrelevant the point is it gives us some
- 41:03number back that can be used to
- 41:05determine the location so let's just say
- 41:09this circle represents the full range of
- 41:12values we'll say it starts right here
- 41:14this is zero and for Simplicity this is
- 41:17going to be
- 41:19100 this will be
- 41:22200 right here will be
- 41:25300 and here will be 4 400 and then
- 41:28let's say we process our key through
- 41:30this hash function and we get
- 41:34255 and I know we you might be thinking
- 41:36well why don't we just use this key here
- 41:38well this doesn't have to be just a
- 41:39simple number basically we need to take
- 41:41some key process it through to get some
- 41:45number that we can then use to determine
- 41:47its location so where would 255 go on
- 41:49this ring somewhere about right
- 41:52here
- 41:54255 and then by convention we basically
- 41:57just put it on the server right here so
- 42:00it's within this range so it goes on the
- 42:03server clockwise on this ring so
- 42:06basically in one quick operation we just
- 42:08take the key hash it get the hash value
- 42:12that tells us exactly which server that
- 42:14that data will be stored on then going
- 42:17to the replication Factor thing it's
- 42:19going to be replicated to a couple more
- 42:21notes so given this information if you
- 42:24can even interpret my handwriting given
- 42:27this key can you tell me how you would
- 42:29find that information to read that
- 42:31information not to make a write it's
- 42:34going to be the same exact process so we
- 42:36do a query for ID
- 42:39123 behind the scenes that gets
- 42:41processed through the hash function
- 42:43it'll give us the same value that's a
- 42:45critical part of the hash function the
- 42:47same input needs to give the same output
- 42:49which is true for hash functions so we
- 42:51then can immediately jump to the correct
- 42:54location in this database cluster there
- 42:57still might be like a coordination node
- 42:59so for example we might make the initial
- 43:01request to say this node saying we're
- 43:05trying to access key 1 2 3 and then this
- 43:08node can do the magic calculation stuff
- 43:10and say oh yeah here's exactly where you
- 43:12need to go to get that data otherwise if
- 43:15we didn't have this coordination node
- 43:17then we would need some external system
- 43:19to figure out which node to get the data
- 43:22from so there might still be some
- 43:24routing going on but it's a very quick
- 43:27quick operation well that's the overview
- 43:28of consistent hashing I kind of ruined
- 43:30all the fun because now the next video
- 43:33on these topics you'll be like I already
- 43:35know all this stuff but we might be able
- 43:37to go into some more depth I feel like
- 43:39having a highle understanding of these
- 43:40topics is good to know for nosql but I
- 43:43don't really feel like you have to know
- 43:45the inner details of everything in order
- 43:47for you to understand nosql in general
- 43:50and use it as a developer so I'm trying
- 43:52to just give you a little bit here so
- 43:53you have an idea but if we really want
- 43:55to go into the details that would makes
- 43:57sense to do in a dedicated lesson now
- 43:59let's talk about another challenge with
- 44:00nosql databases so we have the data
- 44:04distributed across multiple nodes as
- 44:06we've seen but we have to give some kind
- 44:08of ID to identify how to find that
- 44:11information in the database cluster so
- 44:14in the previous example I used the key 1
- 44:162 3 this ID has to come from somewhere
- 44:21now if you're coming from a relational
- 44:22database system you're probably familiar
- 44:25with some kind of autoincrement
- 44:27or
- 44:28equivalent so this is not always a
- 44:31native feature in nosql databases
- 44:34because of the distributed nature of
- 44:36nosql just consider the smallest example
- 44:39of this if we have two
- 44:41nodes and let's say the last used ID was
- 44:441 2 3 but the data is split up across
- 44:47these nodes so the highest on this one
- 44:49might be 1 2 3 and maybe it has had the
- 44:52last few rights so it might have 1 2 2 1
- 44:562 1 one and this one the most latest
- 44:59might be
- 45:01120 so then let's say we write something
- 45:04to this node what id do we use because
- 45:07this node currently does not know that
- 45:10123 exists over here on this node right
- 45:13so it's not just going to naturally know
- 45:15to use 124 unless these are sharing some
- 45:19kind of coordinator node to distribute
- 45:22IDs so that's one possible solution you
- 45:24basically have an ID distributor
- 45:27and this will basically take a request
- 45:30and give back an ID for this
- 45:34autoincrement feature so we basically
- 45:36just have a centralized ID Creator so
- 45:39when we have an external ID generator we
- 45:42now have to basically move that ID
- 45:44generation outside of the database so
- 45:47often with nosql databases you will be
- 45:49responsible at the application Level to
- 45:51give an ID so we have a database node
- 45:54and we're inserting data the insert is
- 45:57going to have the ID
- 46:01already so this comes from the
- 46:03application not from the database so
- 46:06that means our application can
- 46:08communicate
- 46:09to that centralized ID
- 46:13server now there's problems to this
- 46:16because now we have a single point of
- 46:17failure we built a distributed database
- 46:19and we're relying on a centralized
- 46:21service so you might do something where
- 46:24we actually load balance this and we
- 46:25have two ID
- 46:28servers and then maybe this one
- 46:30distributes odds this one distributes
- 46:32evens there are various Solutions I'm
- 46:34sure but the main thing you need to
- 46:36understand here is that it's no longer
- 46:38the database's job to generate the ID
- 46:41it's the application's job so the ID is
- 46:44decided
- 46:45here not here and that means anything
- 46:48you can do at the application Level to
- 46:50generate an ID could potentially be a
- 46:52valid solution to this problem so we've
- 46:54talked about one which is a central
- 46:57ized Auto increment kind of complex
- 47:01probably wouldn't want to do this if I
- 47:02was just building some system and I
- 47:04wanted to use a nosql database so there
- 47:07are a couple of other really good
- 47:08options the next option would be to use
- 47:11some kind of natural
- 47:15key so a natural key is great here
- 47:18because it's associated with the data
- 47:20that's being inserted so you don't have
- 47:22to use any external resources so that's
- 47:24a great option and that won't require an
- 47:26external system you can do that all
- 47:28within the application that has the data
- 47:31you're working with right there the
- 47:32third option would be to generate
- 47:34something
- 47:35random at the application Level that's
- 47:38not Auto incremented so a uu
- 47:42ID or you might hear guid universally
- 47:46unique identifier and globally unique
- 47:49identifier same idea basically this is
- 47:51just a 128bit value that can uniquely
- 47:54identify any data because with 128 bits
- 47:58the value is large enough that the
- 48:00chances of collision is extremely low so
- 48:02this is great but the downside here is
- 48:04it's not searchable very easily if
- 48:06you're trying to access a value and you
- 48:08have the ID already this is going to
- 48:10work great because you can just access
- 48:12that exact ID no problem but if you need
- 48:15to search through the data there's no
- 48:17logical sorting because the IDS are
- 48:19random so that's the big downside here
- 48:21as well as every single key taking up
- 48:23128 bits even if you're working with
- 48:25really small amounts of data every
- 48:27single one is going to use that full bit
- 48:29size the fourth solution which is a
- 48:32little bit more complicated in terms of
- 48:34understanding but this is pretty common
- 48:37for different Services out there is to
- 48:39use a
- 48:42snowflake and this is a 64bit
- 48:47value that's not random but actually has
- 48:49some encoded information in that value
- 48:53so something like one bit is a sign bit
- 48:55that's reserved I think there's 41 bits
- 48:58to a timestamp that actually indicates
- 49:00when the ID was created 10 bits to the
- 49:03system ID so whatever server is creating
- 49:06the snowflake ID to avoid conflicts
- 49:09across servers and then a 12-bit
- 49:11sequence ID which is an auto
- 49:13incrementing value that resets every
- 49:14millisecond so you can create a lot of
- 49:16values every millisecond so all those
- 49:18put together will be 64bit but the value
- 49:21means so much more than just a
- 49:23128bit randomly generated number so this
- 49:26is something you you'll see for large
- 49:27applications that want to have some
- 49:29identifying information inside of the ID
- 49:33and they're going through additional
- 49:34effort to save space and that
- 49:37information is sortable because there's
- 49:39a timestamp so there are some really
- 49:40good benefits to the snowflake idea but
- 49:43it does take a little bit of additional
- 49:45understanding we might do some dedicated
- 49:46content on that now to wrap things up I
- 49:49wanted to go through two examples so
- 49:51talking about how you might design
- 49:53something in nosql so let's first start
- 49:55talking about a video system so we're
- 49:58creating some video platform and we have
- 50:02a way for users to create comments so I
- 50:05wanted to talk about this example
- 50:07because this is going to introduce some
- 50:09interesting things about Keys as well as
- 50:12secondary indexes which are very
- 50:13important in nosql so again thinking
- 50:16ahead of what types of queries we might
- 50:18be making so let's talk about what a
- 50:20comment might look like we might have
- 50:23the video
- 50:25ID a
- 50:27timestamp we're not going to be using
- 50:30the snowflakes or any IDs with
- 50:32timestamps embedded in them so we're
- 50:34just going to have a video ID which
- 50:35we'll just assume is a uu ID that was
- 50:38generated nothing too fancy so we'll
- 50:40have a separate Tim stamp value and then
- 50:43we'll have a user
- 50:46ID this is who posted the comment and
- 50:49then we'll have the actual content of
- 50:51the
- 50:52comment so initially there's going to be
- 50:56some challenges here because we don't
- 50:57have relationships so this user ID this
- 51:01video ID it does not connect to another
- 51:04table with some kind of foreign key
- 51:07constraint so if we want to have that
- 51:09connection we have to do it at the
- 51:11application Level basically we have an
- 51:13app
- 51:17here this accesses two tables from the
- 51:20database or maybe
- 51:22three
- 51:23[Music]
- 51:24users comments and videos and then we do
- 51:28whatever we need with these three
- 51:30individual sets of data at the
- 51:31application Level so you just got to be
- 51:33more careful with data Integrity if the
- 51:35nosql system has some form of
- 51:38transactions across tables you might
- 51:39want to use those for certain scenarios
- 51:42but the idea here is that this user ID
- 51:44and this video ID are just values
- 51:47they're not foreign keys to some other
- 51:49tables and that's a big difference
- 51:51inside of nosql compared to a relational
- 51:53database this first attribute here is
- 51:56often going to be called the partition
- 51:59key and notice how we don't have a
- 52:02comment ID here this might seem strange
- 52:05but we have to think about how we are
- 52:07accessing the data in this aggregate
- 52:11oriented style of database design we
- 52:14would access the video and then we need
- 52:16to get all of the associated comments so
- 52:19by setting the video ID as the partition
- 52:22key in the comments we can access all of
- 52:25the comments for particular video ID
- 52:28very easily so if we had a comment ID it
- 52:31could be here but it wouldn't be the
- 52:34primary key of the table it would just
- 52:36be another attribute so we basically are
- 52:40going to group things by video ID and
- 52:42that's the idea of a partition key so
- 52:44things that are accessed together are
- 52:46going to be stored together on the same
- 52:48partition so I'm going to remove that
- 52:49comment ID but we have another problem
- 52:52here we still need a primary key which
- 52:53is unique if we made it just the video
- 52:56ID
- 52:57that's not going to work because we have
- 52:59multiple comments per video so we
- 53:01actually need to use a composite primary
- 53:03key which will consist of two values so
- 53:06that's going to be the partition key
- 53:08plus this here which is often called the
- 53:12sort key or you might hear it as the
- 53:15clustering
- 53:17key and when you hear clustering here if
- 53:19you're familiar from relational
- 53:21databases the idea of a clustered index
- 53:24it determines the sort of the data this
- 53:27determines the order in which that data
- 53:29is stored so these two combined can be
- 53:32our primary key so we will not have any
- 53:35duplicate values where we have the same
- 53:37video ID with the same timestamp so this
- 53:39helps us understand the basics of how we
- 53:41would structure a table in a nosql
- 53:44database now let's talk about secondary
- 53:46indexes this is very important if you
- 53:49want to access the data from a different
- 53:52perspective such as all the comments
- 53:54from a certain user so right now it's
- 53:56highly inefficient because the data is
- 53:59stored by video ID and timestamp we
- 54:01would have to do a table scan to get all
- 54:03of the comments with a certain ID so we
- 54:06would add an index here and you can
- 54:08imagine this index basically taking the
- 54:11original structure of the data and
- 54:14reorganizing it by a different key so
- 54:18now we have the ability to retrieve by
- 54:20video ID and we also have the ability to
- 54:22retrieve by user ID however this would
- 54:25obviously lead you to believe that
- 54:26there's additional work happening here
- 54:29so anytime you add an index that index
- 54:31needs maintained you're basically
- 54:33putting that data in two different
- 54:35arrangements so think clearly about what
- 54:38queries you really need so you can
- 54:39create those secondary indexes but if
- 54:41you're not using one of them then it's
- 54:43not really a good idea to keep them
- 54:44around just in case you could always add
- 54:46an index later with some upfront work
- 54:49but the idea here is we think ahead
- 54:51about what queries we might be making
- 54:53against our data now one more quick
- 54:54comment about the information that I
- 54:56just erased is if we have some ID that
- 55:00has a Tim stamp or some kind of sort
- 55:02embedded in it then we might not need a
- 55:04dedicated Tim stamp for the clustering
- 55:08key so for example if we were using
- 55:10snowflakes in that situation we could
- 55:12just use the comment ID as the
- 55:14clustering key or the sort key instead
- 55:16so it might look like this video ID and
- 55:19then comment ID this would be our
- 55:22primary key we could organize by by
- 55:26video that's how the information would
- 55:28be accessed we want the comments for a
- 55:30certain video and then this being an ID
- 55:35that has some sort information embedded
- 55:38in it can be used to sort the data from
- 55:42newest to oldest or whatever you
- 55:47need this is basically the replacement
- 55:50for the timestamp if the IDS have that
- 55:53timestamp information embedded in them
- 55:56so there's definitely benefits of using
- 55:58something like a snowflake so I actually
- 56:00wanted to do a second example of Discord
- 56:03and a popular article online on how
- 56:06Discord stores billions of messages
- 56:08recently updated to how they store
- 56:10trillions of messages they use
- 56:12snowflakes in their system and their
- 56:15primary key looks like
- 56:19this channel ID message ID so very
- 56:24similar situation here where we have
- 56:27some Channel let's just say it's General
- 56:30and we want to get all of the messages
- 56:32for that channel so that's why this is
- 56:34the partition key but we don't just want
- 56:37the messages thrown in there we want
- 56:39them to be sorted in some way so this
- 56:42secondary part of the key is a
- 56:44clustering key that determines the sort
- 56:47and this
- 56:49is a snowflake I believe this would be a
- 56:52snowflake as well it's just not used for
- 56:54the Sorting in this scenario so we take
- 56:56the message IDs we can retrieve the
- 56:59timestamp information from within the
- 57:01snowflake to then give a sort from
- 57:05newest at the
- 57:07bottom scrolling up to the
- 57:10oldest so that's a really quick
- 57:14overview of discord's ID architecture
- 57:20they used Cassandra they switched to syb
- 57:22and now if you go read those articles
- 57:24you'll have a pretty good foundational
- 57:26to understand what they're talking about
- 57:28and why it works for them I've given a
- 57:30lot of information in this video and
- 57:32this of course is not going to apply
- 57:34universally to all nosql database types
- 57:37so definitely do your research for the
- 57:39individual database you might be working
- 57:41with this is really just designed to
- 57:43give you that really good foundation you
- 57:46should now be able to go into nosql
- 57:48databases with some level of confidence
- 57:50as well as having more confidence going
- 57:52into any of the white papers
- 57:53understanding what they're talking about
- 57:54with consistent hashing and things like
- 57:56that so the final summary here is how do
- 57:59you choose between a nosql database and
- 58:02a relational database hopefully you have
- 58:04a good understanding of that answer by
- 58:06now ultimately it depends so hopefully
- 58:10that helps one thing I would consider
- 58:12doing though is wrapping your database
- 58:15in a very easy to consume API or SDK so
- 58:20you can imagine having your database
- 58:22here and then wrapping this so that it's
- 58:25super easy to interrup act with based on
- 58:27the most common things you need to do so
- 58:29for
- 58:30example get users this could be a
- 58:33function call if you're working with the
- 58:35database directly in your application or
- 58:38you could actually just expose different
- 58:40API
- 58:43endpoints such as get
- 58:45users post a user basically writing out
- 58:50all of the different interaction points
- 58:52the interface to the database so that in
- 58:55theory while it's not not as easy in
- 58:56practice you could take out the database
- 58:59substitute it in for something else so
- 59:03you could even switch between SQL and
- 59:06nosql but the applications that interact
- 59:09with this
- 59:11interface have the same interface and do
- 59:14not have to change any of their code
- 59:17because they're still going to get users
- 59:19structured just how they would expect
- 59:20and you can do any transformation logic
- 59:22or anything you need to do here inside
- 59:25of the data layer
- 59:27so whether that's through an API or an
- 59:28SDK or however you want to do it
- 59:31basically adding some interface to
- 59:34separate the consumption of the database
- 59:37from the database itself can be very
- 59:39handy for creating scalable systems so
- 59:43that you don't break all of the
- 59:44interacting applications if you switch
- 59:46from one database to another so if that
- 59:48doesn't make sense you can imagine you
- 59:49have a bunch of different microservices
- 59:51and all of these ultimately talk to the
- 59:54database
- 59:56starting out you might use the exact
- 59:59database drivers that you're working
- 1:00:00with so imagine you're working with some
- 1:00:02nosql database and you're writing direct
- 1:00:05for that database in all of these
- 1:00:07individual applications but then you
- 1:00:08want to switch it out you now have to go
- 1:00:10into all of these different applications
- 1:00:12updating that code which could be a
- 1:00:15disaster and make it pretty much
- 1:00:16impossible to switch out a database or
- 1:00:18to do any major changes like that so
- 1:00:21while I don't love abstraction for
- 1:00:23abstraction sake just making things
- 1:00:24complicated I do like the idea of
- 1:00:27abstracting away the database itself and
- 1:00:30just giving a nice interface so in that
- 1:00:33situation all of these apps would only
- 1:00:36talk to this outer
- 1:00:39layer in this case I'd probably make it
- 1:00:42an API so that we could do very easy
- 1:00:45cross app
- 1:00:46communication they just hit the API get
- 1:00:49the data they need they don't care what
- 1:00:51the database is or where the data is
- 1:00:54stored they just know that if they hit
- 1:00:55the API they get the data and they can
- 1:00:58use it in their app so that's one
- 1:01:00specific example you could probably
- 1:01:02structure it differently but that's just
- 1:01:04an overview of an approach that you
- 1:01:05could use to basically reduce the
- 1:01:09possibilities of things going south if
- 1:01:11you need to make a change at the data
- 1:01:13layer if you've made it this far you are
- 1:01:15a legend I have to say thank you and I
- 1:01:17would really appreciate a subscribe to
- 1:01:19the channel if you enjoy this type of
- 1:01:21content again check out the mentorship
- 1:01:23program I'll have links for that down
- 1:01:25below go through that if you really want
- 1:01:27to level up your software engineering
- 1:01:29game so that's the only two asks I have
- 1:01:31for you subscribe check out the
- 1:01:32mentorship program and I'll see you next
- 1:01:34time if you have any suggestions on
- 1:01:36additional content or things you would
- 1:01:38like to hear about in more detail let me
- 1:01:40know drop a comment would really
- 1:01:42appreciate it
About this transcript
This page contains the full transcript of Intro to NoSQL Databases - System Design by Caleb Curry, generated from the public captions YouTube serves with the video. The transcript has 10,599 words across 1,535 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.