Least squares - 13 - Multiple linear regression - Matrix form and an example — Transcript

Full transcript

1. 0:09to help us work with these multiple
2. 0:11linear
3. 0:12models efficiently we need to use some
4. 0:15matrix notation we will look at that in
5. 0:17this video the first step here is that
6. 0:20we're going to remove the intercept from
7. 0:22consideration this is not as bad as it
8. 0:25sounds the intercept is seldom of
9. 0:27interest and we can always recover it if
10. 0:29we want to know it we remove it by first
11. 0:32stating the regular regression model
12. 0:34then in the second line we restate the
13. 0:36regression model at the point which we
14. 0:38know the model will pass through the
15. 0:40averages thirdly we subtract the second
16. 0:43line from the first line this creates a
17. 0:47new regression model where our variables
18. 0:48are in their centered form in other
19. 0:50words instead of X I we use X I minus
20. 0:54the mean of all X values instead of Y I
21. 0:56we use Y I minus the mean of the Y's
22. 0:59recall and centering is a process that
23. 1:01simply shifts the variable but maintains
24. 1:04all of the relationships that it has
25. 1:06when we build our model with the
26. 1:08centered x and y variables or as we
27. 1:11sometimes say deviation variables we now
28. 1:13have a model where the intercept is 0
29. 1:15and does not need to be estimated notice
30. 1:18from the second line over here we can
31. 1:20always recover that intercept value back
32. 1:22if we really are interested in it now
33. 1:25sometimes you will see me build
34. 1:27regression models with the intercept in
35. 1:28it especially in the designed
36. 1:30experiments part of the course so you
37. 1:32really don't have to do this step but it
38. 1:35does help for interpretation later on so
39. 1:38now we have a model without an intercept
40. 1:40but we have K parameters beta 1 beta 2
41. 1:43up to beta K and each of them are
42. 1:46multiplied by their corresponding X
43. 1:47variable right at the end we have an
44. 1:50error epsilon I and all of that is added
45. 1:53up in a linear combination to give you a
46. 1:55value of y I we can express this
47. 1:58observation as a linear combination in
48. 2:00vector form here we have a vector X and
49. 2:03a vector beta and this is for the I
50. 2:05thought servation so we collect our
51. 2:08vector X the input data and the vector
52. 2:11beta and we multiply them using regular
53. 2:14matrix multiplication ad the epsilon
54. 2:16value to the end and then we get our Y I
55. 2:18value there are K values of X multiplied
56. 2:22by their corresponding K coefficients of
57. 2:24beta and we add
58. 2:26to get our single y value and we will
59. 2:29call that vector X I to emphasize that
60. 2:32that is the row vector for the I thought
61. 2:34servation of X now if we have multiple
62. 2:38observations of X then we can stack
63. 2:40those vectors in a matrix and form
64. 2:43capital matrix X Row 1 is the first
65. 2:46observation x 1 Row 2 for the next
66. 2:49observation x subscript 2 and so on up
67. 2:52to the last row x subscript n I've also
68. 2:56replaced the beta's now with bees to
69. 2:59emphasize that we're going to estimate
70. 3:01the bee values rather than us knowing
71. 3:03what the population model is so we have
72. 3:05a matrix of X observations n rows and K
73. 3:09columns there are n rows for every
74. 3:11observation and K columns for the K
75. 3:14input variables multiply that here by
76. 3:17our vector b1 b2 up to be K so K entries
77. 3:21of B and then we add a vector of errors
78. 3:24at the end I've emphasized the
79. 3:27dimensions of each of the vectors and
80. 3:28matrices over here now the objective
81. 3:31function for multiple linear regression
82. 3:33is exactly the same as regular least
83. 3:35squares minimize the sum of squares of
84. 3:38the errors if we take that vector of
85. 3:41errors over here lowercase e the sum of
86. 3:44squares of the errors can easily be
87. 3:46shown to be e transpose e well that
88. 3:48vector e can be written as Y minus
89. 3:51matrix X times vector B if we sub that
90. 3:55in we get the second row over here and
91. 3:57if we expand that matrix vector
92. 3:59transpose we can use some basic rules of
93. 4:02matrix vector algebra to get this
94. 4:04revised objective function we've seen
95. 4:07before that we can minimize the
96. 4:08subjective function by setting the
97. 4:10partial derivative of it with respect to
98. 4:13the search variables to 0 this means the
99. 4:16partial derivatives with respect to each
100. 4:18of the entries in vector B are set equal
101. 4:20to 0 this will give us K equations in K
102. 4:24unknowns we have K partial derivative
103. 4:27equations and there are K values of B b1
104. 4:30b2 up to BK now these K linear equations
105. 4:34in K unknowns can be solved very
106. 4:36compactly using what is probably the
107. 4:38best-known
108. 4:39formula in least-squares that the
109. 4:41entries in vector B are equal to X
110. 4:43transpose X inverse multiplied by X
111. 4:46transpose Y don't forget that formula
112. 4:49you're going to see it reoccurring
113. 4:51several times in this module as well as
114. 4:53in the next module on designed
115. 4:55experiments and it's time for an example
116. 4:58now where we've got two vectors X 1 and
117. 5:00X 2 shown here on the screen let's use
118. 5:03these to predict the vector Y the first
119. 5:06step we're going to have to take is to
120. 5:07Center the vectors X and the vector Y
121. 5:09will do that and the results are shown
122. 5:12here but before we go ahead and even
123. 5:14solve this equation what do you expect
124. 5:17the slope to be for X 1 as it relates to
125. 5:20Y and what do you expect the slope to be
126. 5:22for X 2 as it relates to Y examine the
127. 5:25data and write down your answer keep it
128. 5:27aside and we'll verify it later now
129. 5:30let's go solve this mathematically and
130. 5:32try to verify our intuition the first
131. 5:35step we need to do is assemble those two
132. 5:37individual vectors into a matrix capital
133. 5:40X the first column will represent the
134. 5:43entries for variable X 1 the second
135. 5:45column in matrix X represents the second
136. 5:48variable X subscript 2 notice that we
137. 5:51have n observations 6 in this case so
138. 5:54there must be six rows in the X matrix
139. 5:56and two columns one for each of the
140. 5:58input variables X is always an N by K
141. 6:01matrix and the Y vector is always an N
142. 6:04by 1 vector of the output values these
143. 6:07are assembled now as shown here on the
144. 6:09screen so we are ready to go calculate
145. 6:12vector B the two coefficients of our
146. 6:15regression model X transpose X inverse
147. 6:17multiplied by X transpose Y let's go
148. 6:20calculate the first part X transpose X
149. 6:23that's shown over here then we can go
150. 6:26calculate X transpose Y separately shown
151. 6:29over here you're welcome to use any
152. 6:31software tool to do this but here on the
153. 6:34screen are the our instructions in case
154. 6:36you wanted to try it over there but we
155. 6:38actually need the inverse of X transpose
156. 6:40X and you can also do that in our with
157. 6:43this line of code so now we have X
158. 6:45transpose X inverse and we can go
159. 6:47multiply it by X transpose Y
160. 6:50I want to point out that X transpose X
161. 6:53will always be a K by K matrix K being
162. 6:56the number of variables in our model in
163. 6:58this case K is 2 it is going to be an
164. 7:01important point later on that the
165. 7:03entries in X transpose X represent the
166. 7:06covariance of the X variables with each
167. 7:08other for example the entry in the one
168. 7:11two position in this matrix is the same
169. 7:14as the entry in the to one position
170. 7:15that's because the covariance of
171. 7:18variable X 1 with variable X 2 is the
172. 7:21same as the covariance of variable X 2
173. 7:24with variable X 1 you'll recall that
174. 7:26from an earlier video and that that
175. 7:28should be the same number the entries on
176. 7:30the diagonal are also covariances but
177. 7:33they're covariances of the variables
178. 7:35with themself so we simply call that
179. 7:36variance in this case the 55.5 over here
180. 7:40is directly proportional to the variance
181. 7:43of the x one variable the 62 is
182. 7:45proportional to the variance of the X 2
183. 7:48variable because the diagonal entries in
184. 7:51X transpose X are variances they should
185. 7:53always be positive variances are by
186. 7:56definition positive values and the off
187. 7:58diagonal elements in X transpose X will
188. 8:01always be symmetrical once you've
189. 8:02calculated the upper diagonal part you
190. 8:05can just copy the values over to the
191. 8:07lower diagonal part now here's a
192. 8:09question I want to leave for advanced
193. 8:11students what will the x transpose x
194. 8:13matrix look like when the variables x1
195. 8:15and x2 and so on are entirely
196. 8:18uncorrelated with each other what will
197. 8:20you observe in those off diagonal
198. 8:22entries you'll see the answer to that
199. 8:25coming up in the next module but you
200. 8:27should be able to guess what it is now
201. 8:29now if x transpose x represents
202. 8:32variances and covariances what do you
203. 8:34think the entries in X transpose Y
204. 8:36represents
205. 8:38if you guess that it's the covariance of
206. 8:40the entries of X with Y you write the
207. 8:43first entry in the X transpose Y result
208. 8:45is the covariance of variable x1 with Y
209. 8:49and the second entry is the covariance
210. 8:52of X 2 with Y now take a look at the
211. 8:55signs do they match your expectations
212. 8:57what you wrote down earlier look back at
213. 9:00the raw data and make sure at least that
214. 9:03the sign information matches your
215. 9:05expectation remember right at the start
216. 9:08of the section on linear models we said
217. 9:10that for covariances at the very least
218. 9:12you can interpret the sign of that
219. 9:14result the magnitude of the value isn't
220. 9:16so useful and those results that we see
221. 9:19there the 36.5 and the -36 seem to make
222. 9:23sense the positive sign for the first
223. 9:25entry indicates that the first X
224. 9:27variable is positively correlated with
225. 9:30the Y variable and we see that in the
226. 9:32raw data is that first column increases
227. 9:34the column of Y also increases the
228. 9:37second entry in X transpose Y is
229. 9:39negative indicating that that column in
230. 9:41X has a negative correlation with Y and
231. 9:44that's also observed in the raw data
232. 9:46there
233. 9:48now one final thing to mention before we
234. 9:50end this video is that you should never
235. 9:52calculate your least squares model using
236. 9:54this X transpose X inverse formula
237. 9:56multiplied by X transpose Y for real
238. 9:59general data it works in some specific
239. 10:02cases very well and that's why we will
240. 10:04use it later on but in general there's
241. 10:06very poor numerical conditioning with
242. 10:08calculating the inverse in this way
243. 10:10sophisticated least squares packages
244. 10:13such as R will calculate the linear
245. 10:15model using the QR decomposition instead
246. 10:18it's a similar type of linear algebra
247. 10:20step and interested students should go
248. 10:22investigate that in their own time the
249. 10:25our software package uses the QR
250. 10:27decomposition behind-the-scenes and not
251. 10:29this inverse and since we're talking
252. 10:31about R let's go see now how R solves
253. 10:34this example problem use the LM command
254. 10:38and say why is regressed on two
255. 10:40variables x1 plus x2 try that out and
256. 10:44use the summary command on that model
257. 10:46result to see if it matches the output
258. 10:49that you calculated with the X transpose
259. 10:51X inverse X transpose Y equation there
260. 10:54is one last entry we need to go
261. 10:56calculate and that is the standard error
262. 10:57standard error you will recall is equal
263. 11:00to the objective function of the least
264. 11:01squares model divided by n minus K and
265. 11:04then we take the square root so in other
266. 11:07words standard error is calculated as e
267. 11:09transpose e divided by n minus K and
268. 11:12then we take the square root once we
269. 11:14have that error vector it is easy to go
270. 11:16calculate the standard error in the next
271. 11:19video we're going to look at the
272. 11:20important topic of interpreting the
273. 11:22multiple linear regression coefficients