YT YouTube2Text

YouTube2Text → Transcripts → Moran's I : Data Science Concepts

Moran's I : Data Science Concepts — Transcript

by ritvikmath · 3,159 words · 513 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00[Music]
0:01hey everyone welcome back so in this
0:03video we'll be talking about moran's eye
0:05which is a measure of spatial
0:06correlation now let's just show you an
0:08example right away just to get an idea
0:10for what kind of questions can be
0:12answered using moran's eye
0:14so what i've shown here is the election
0:15map for the 2016 presidential election
0:17in the united states
0:19so you see that the red states voted for
0:20donald trump and the blue states voted
0:22for hillary clinton
0:24so as you can see there's a little bit
0:25of clustering here and there you can see
0:26some areas where
0:28red states cluster together in some
0:29areas where blue states cluster together
0:31so a natural quantity that you might
0:32want to measure is the magnitude of this
0:34clustering through the united states so
0:36that would help answer the question of
0:37do red states cluster together are red
0:39states more likely to be near other red
0:41states and blue states near other
0:43blue states or is it more random
0:44geographically
0:46so that's the type of question that can
0:47be answered with moran's eye
0:49now instead of going into that example
0:51which is going to be very complicated
0:52let's look at a smaller example and
0:54you'll be able to see how we can answer
0:56the initial question
0:57pretty easily so rather than just give
0:59you the formula for moran's eye you know
1:01on this channel i don't like to just
1:02throw formulas at you because i don't
1:04see that as an effective method of
1:05teaching
1:06instead we'll be looking at two examples
1:08and then we'll be building up the
1:09formula so that by the end you're going
1:11to have the formula
1:12but also the understanding about what
1:13each component means
1:15okay so let's pretend like we're looking
1:17at the same example except with a much
1:19smaller country so let's say that this
1:21is our country
1:22it has eight states so each state is
1:24each of these cells and a one let's say
1:26means that that state voted republican
1:28in the last election and let's say zero
1:30means that it voted democrat in the last
1:32election so we're going to look at two
1:33possible scenarios here and look at what
1:35the moran's eye
1:36is for each one so our first scenario is
1:39up here so the states are numbered with
1:41the green values so 1
1:422 3 4 5 6 7 8 and the same numbering
1:46down here for the second scenario now we
1:48see in the first scenario there is a
1:50very obvious clustering all of the
1:52states that voted republican or have
1:54ones in them are to the left and all the
1:56states that voted democrat or have a
1:58zero in them are to the right
2:00so we would expect moran's eye to be
2:02rather high just a quick note morant's
2:04eye
2:04is bounded between negative one and one
2:06so that the interpretation doesn't
2:07change
2:08based on whether your values are
2:09measured in feet or meters or
2:11whatever other units you might have so
2:13we have this case where we'd expect
2:15moran's eye to be rather high towards
2:17the one side because there is a strong
2:19positive spatial correlation and what
2:21that means in layman's terms is that
2:23there's a
2:23strong indication that states with
2:26similar voting patterns cluster together
2:28now let's look at the bottom example
2:30the bottom example is actually the exact
2:32opposite scenario we see that if we have
2:34a republican state or one that has a one
2:36in it
2:36then it's going to be bordered by
2:38democrat states that have zero in them
2:40and vice versa if we have a democrat
2:42state that has zero in it it's going to
2:44be bordered by republican states
2:46so this is a prime example of negative
2:48spatial correlation and we don't see
2:50this much in real life and it doesn't
2:51get talked about too much but
2:53i do want to give it to you so you can
2:55have the full picture
2:57negative spatial correlation means that
2:58the value at any given geography tends
3:01to be very
3:01far from the value at neighboring
3:03geographies so it's even difficult for
3:05me to come up with a natural example of
3:07this because we typically think of
3:09either having
3:10a positive spatial correlation where
3:12similar values cluster together
3:14or having no spatial correlation where
3:16there's no real pattern
3:18in the geography you're looking at but
3:19thinking about having a negative spatial
3:21correlation is a little bit more
3:23difficult but that doesn't mean that it
3:24doesn't exist it just means that we
3:25think about a little bit less but this
3:27is a graphical example of a negative
3:29spatial correlation
3:30where the value at any given geography
3:32is the opposite or very
3:34far from the value at neighboring
3:36geographies now just to reiterate again
3:38we have n states here so our n is equal
3:40to eight and that's just the number of
3:42units that you have if you're looking at
3:43the map of the united states then your n
3:45would be equal to 50 because we have 50
3:47states so the last component you'll need
3:49before we start the calculations of
3:50moran's eye is to define a weight matrix
3:53and this ends up being actually a very
3:54important part of what your moran's i
3:56values are going to be
3:58this weight matrix where each entry is
4:00given by w i
4:01j where i goes from one to n and j goes
4:04from one to n
4:05basically tells the story of what is the
4:07weight or what is the connectedness of
4:09geography i to geography j
4:11for example if i'm looking at geography
4:12one and two that would be these first
4:14two cells here
4:15then i'm going to give that a weight of
4:17one because
4:19those two states or those two
4:20geographies border each other
4:22however if i'm looking at geography one
4:25and three
4:26i'm going to give that a weight of zero
4:27because those geographies don't border
4:29each other
4:29another example would be looking at
4:31geography one and five those two do
4:33border each other spatially so i'm going
4:35to give that a weight of one
4:37so this weight matrix completely tells a
4:39story about what is the connection or
4:41strength of connection between one
4:42geography and any other geography that's
4:45in your situation so the weight matrix
4:47i've chosen for this example is that
4:48very simple one where if you see a state
4:50having a border
4:52not a diagonal border but either a
4:53horizontal or vertical border with
4:55another state we're going to give that
4:56weight
4:571 and if they don't have such a border
4:59i'm going to give that a weight of 0. so
5:01that's the story that's being told
5:02mathematically
5:03over here now that's not the only way
5:05you can define a weight matrix for
5:07example you can also do something like a
5:09k
5:09nearest neighbor weight matrix where
5:11you're going to give the
5:12five nearest neighbors or three nearest
5:14neighbors of a geography a weight of one
5:17or you might have a decaying weight
5:18matrix where the nearest neighbors get a
5:21weight of something near one
5:22but as you get further away the weight
5:24is not zero but it is going to zero so
5:26for example the weight between one and
5:28four
5:28would be low in that case so there's a
5:30lot of ways you can define this weight
5:32matrix and that's going to impact the
5:34final calculation of moran's eye so the
5:36way you define this weight matrix is an
5:37important factor
5:38in the final values okay but anyway now
5:41that we have our weight matrix defined
5:43and by the way this big w is just going
5:44to be the sum
5:45of all of the values in my weight matrix
5:47and that's going to be equal to 20.
5:49quick note why is it going to be equal
5:50to 20
5:51because if we look at the situation and
5:53you count the number of pairs of
5:54bordering states you find that there are
5:5610 pairs
5:57and each pair is symmetric so for
5:59example the weight matrix contains a 1
6:02for state 1 connected to 2 and if you
6:04look at the transpose element
6:06or the matching element on the other
6:07side of the weight matrix that's also a
6:091. so
6:09there's 10 pairs times 2 and we get
6:1320 as the total sum of all the elements
6:15in the weight matrix and we're going to
6:16call that variable
6:17big w so now that we have the whole
6:20situation set up we're ready to go ahead
6:21and just do the calculation and explain
6:23why this calculation
6:24intuitively does capture the spatial
6:27correlation
6:28so i've broken it down for you in three
6:29easy steps the first step
6:31is to compute x bar x bar is simply just
6:34the mean
6:35across the entire geography of the
6:37variable you care about
6:39here the variable we care about is easy
6:41it's binary either one if the state
6:42voted republican and zero of the state
6:44voted democrat
6:45so we see that the mean would be
6:47one-half because if i add up all these
6:49binary variables in either case i'm
6:51going to get four
6:52and there's eight states total so four
6:54over eight is one-half so we have
6:55x-bar is equal to one-half so the next
6:57step step two is to compute the total
6:59variation
7:00in our eight-state country based on how
7:03far
7:03each value is away from that mean we
7:05just computed so that is given
7:07mathematically down here
7:08so what we're doing here is going
7:10through each state from one to n
7:12and we're simply doing the x i of that
7:14state again that's either one or zero
7:16minus the mean we just calculated and
7:19squaring it
7:20so this is a sum of square differences
7:22between each state's
7:24democrat or republican value and the
7:26mean value
7:27now notice that this doesn't take
7:28anything into account just yet about the
7:30weights between these states or the fact
7:32that one state borders another
7:33this is simply just the total variation
7:36a measure of how much variation there is
7:38total in the entire geography so we see
7:40that if we do this calculation we get
7:41two so the last step and the step that's
7:43most
7:44interesting and makes this truly a
7:45spatial correlation rather than just a
7:47regular correlation
7:49is that we take that weight matrix into
7:51account so what we're doing here is
7:52summing over every i
7:53j pair that's what this two sums are
7:55doing and for each ij pair we're
7:57computing the following value we're
7:59first going to compute this value we're
8:00going to compute
8:01x i minus x bar times x j minus x bar
8:04so you might have seen something similar
8:06when you're doing a correlation or a
8:08covariance calculation
8:09and this is basically measuring whether
8:11state i and state j's difference from
8:12the mean
8:13are in similar or opposite directions
8:15for example let's say that x
8:17i was democrat and x j was republican in
8:19that case we would have
8:20a one here and a zero here this quantity
8:23would be positive one half and this
8:24quantity would be negative one-half so
8:26that we would have a negative one-fourth
8:28total
8:28and the fact that that's negative tells
8:30us that these two states are on opposite
8:32sides of the mean so they're not similar
8:34on the other hand if we had two
8:35republican states like one and one
8:37then we would have one minus of half and
8:39one minus a half and so we would have
8:41one fourth
8:42and the fact that one fourth is positive
8:43would tell us that these two states
8:45do lie on the same side of the mean so
8:47this part is basically just computing
8:49whether or not these two states we care
8:51about
8:52and again we're looping over every pair
8:53of states here and each time we're
8:55looking at two states we're seeing if
8:56these two states do match up
8:58in terms of where they are relative to
9:00the mean now here's the most important
9:02part
9:03we take that measure whether it's
9:05positive or negative or near zero
9:07and we weight it by the corresponding
9:09entry w i j
9:10in the weight matrix now why is this
9:13important because it adds that very
9:14necessary
9:15spatial layer to the story it basically
9:17says that okay
9:18you might have that two states aren't
9:20similar for example one is republican
9:22and one's democrat
9:23but i really only care about that as
9:25much as their weight
9:26for example let's say those two states
9:28don't border let's say we're looking at
9:29state one and state four
9:31so that we would have a negative value
9:33when we do this part of the computation
9:35the question is should i care about that
9:37and their weight is zero because they
9:39don't border so according to my weight
9:41matrix i created here their weight is
9:42zero
9:43so i actually don't even care about that
9:45term at all now compare that with
9:47whether i'm looking at state one and
9:48state two
9:49so that i get positive 1 4 in this part
9:52here
9:52the question is should i care about that
9:54their weight is 1
9:56because they do border so that does get
9:58included in this term so you see why
10:00this term
10:00even though it looks a little bit
10:01complex does exactly the job that we
10:03need because the second two terms
10:05compute the strength of similarity for
10:07any pair of states that we see in our
10:09geography
10:09and the first term weights that strength
10:12by how
10:13close together these states are
10:15literally like geographically
10:17so if these two states are really close
10:18geographically their weight is going to
10:20be really high
10:21and we put a big emphasis on this term
10:23here
10:24however if those two states are very far
10:25apart we would expect their weights to
10:27be very low
10:28and so we don't put a big emphasis here
10:31so this here is computing the variation
10:33between states but weighted by how close
10:36those states are
10:38geographically to each other and if we
10:39do that literal calculation
10:41for the top example here we see that it
10:43is eight times one-fourth where does
10:45that come from again there are 10
10:47pairs of neighboring states here for
10:48example here's 1 2
10:503 4 5 6 7 8
10:539 10. so there are 10 pairs of
10:56neighboring states here
10:57and each one is symmetric so really
10:59there's 20 pairs of neighboring states
11:01if you look at it from the side of
11:02one state versus a different state now
11:04eight of these pairs are connecting ones
11:06to ones
11:07that means that we have eight times
11:09one-fourth
11:10because the one-fourth is what's going
11:11to come out if you do this x i minus x
11:13bar x j minus x bar
11:15and x i and x j are both one similarly
11:19eight of these pairs are connecting
11:20zeros to zeros and we get the exact same
11:22calculation here
11:24four of these pairs are connecting ones
11:26to zeros where are those four
11:28we have one two and they're both
11:31symmetric so we have three and four
11:33so we have four where the measure of
11:35covariance is going to be negative one
11:37fourth
11:38and so if we do this whole calculation
11:40we get three for this step three for the
11:42top one
11:43and we can do a very similar calculation
11:45for the bottom one it's actually much
11:46easier
11:47because every pair you can look at is
11:49connecting a one to a zero
11:50so we simply get twenty negative one
11:53fourths so we get negative five on the
11:54bottom few so we're actually all done
11:56doing our calculations hopefully that
11:58you understood the calculations but more
12:00importantly understood the intuition
12:02behind why we are computing each of the
12:04values in these steps
12:06the last thing we have to do is just put
12:07them all together and this is the final
12:09formula for moran's i
12:11so moran's i is given by big n over big
12:14w
12:14big n is again the number of states we
12:16have eight and big w
12:18is the sum of all the entries in the
12:20weight matrix 20.
12:22these are simply just normalization
12:23factors just so that the final moran's i
12:26is bound in between negative one and one
12:27and then
12:28we do step three divided by step two now
12:30let's quickly think about why that makes
12:32sense
12:32remember step two was the total
12:34variation not taking into account weight
12:36or geographical similarities between
12:38states at all that's just how much
12:40variation is there in my country overall
12:42so that becomes our denominator step
12:44three
12:45was how much variation is there but now
12:47i do care about whether that is
12:48variation between states that are close
12:50together or very far apart
12:52so we see that it's very natural to do
12:54step three's value divided by step two's
12:56value because now we kind of get a
12:57measure of
12:58the fraction of variation that is
13:00attributed to states that are close
13:02together so we do step three over step
13:04two
13:04and we do these calculations and we get
13:06that it's three-fifths
13:08for the top example and we get negative
13:11one for the bottom example so this does
13:13totally match up to our understanding
13:15this negative one for the bottom example
13:16because this is a prime example
13:19of negative spatial correlation and
13:21three-fifths for the top example is very
13:23close to one
13:24in fact the only reason it's not exactly
13:26one is because i have such a small
13:28example if you took this example of
13:30eight states and made it huge
13:31then you would get this approaching
13:33pretty much one so the last couple
13:34things i'll say is the real
13:36applications of moran's eye again this
13:38negative spatial correlation case is not
13:40talked about as much because it's harder
13:42to find real world examples
13:43so a lot of times you'll be checking
13:45whether there's positive spatial
13:46correlation versus whether there's
13:48zero spatial correlation zero spatial
13:50correlation although i didn't do an
13:52example
13:52would be where if these ones and zeros
13:54are pretty much randomly interspersed
13:56there is no connection to geography
13:58and if that were the case then you would
13:59get spatial correlation around
14:01zero so a lot of times researchers are
14:03looking at a map of something whether it
14:05is the election or whether it's heights
14:07or whether it's population densities
14:09in given city areas and they're really
14:11asking the question about
14:12is there a statistically significant
14:14positive spatial correlation here
14:16or is it really just zero and if you
14:18find that there is a positive spatial
14:20correlation there then you can start
14:22saying that okay
14:23we do see clustering of population
14:25densities here
14:26by geography or election by geography
14:29and so on
14:30so hopefully this helped you understand
14:31moran's eye this very widely used
14:33measure of spatial correlation please
14:34like and subscribe for more videos just
14:36like this
14:37i'll see you next time

About this transcript

This page contains the full transcript of Moran's I : Data Science Concepts by ritvikmath, generated from the public captions YouTube serves with the video. The transcript has 3,159 words across 513 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.