Machine Learning Fundamentals: Sensitivity and Specificity — Transcript

Full transcript

1. 0:00when the well runs dry you might be
2. 0:04thirsty but this still StatQuest you
3. 0:07can watch it StatQuest hello I'm Josh
4. 0:13Starmar and welcome to StatQuest today
5. 0:15we're gonna continue our series on
6. 0:17machine learning fundamentals and we're
7. 0:19going to talk about sensitivity and
8. 0:21specificity they're gonna be clearly
9. 0:23explained this StatQuest follows up on
10. 0:27the one that describes the confusion
11. 0:29matrix so if you're not already down
12. 0:31with that
13. 0:32check out the quest the first half of
14. 0:35this video will explain how to calculate
15. 0:37and interpret sensitivity and
16. 0:40specificity when you have a confusion
17. 0:42matrix with two rows and two columns and
18. 0:45the second half will show you how to
19. 0:47calculate and interpret sensitivity and
20. 0:50specificity when you have three or more
21. 0:52rows and columns
22. 0:55even if you're already down with the
23. 0:57confusion matrix let's remember that
24. 0:59rows correspond to what was predicted
25. 1:01and columns correspond to the known
26. 1:05truth when there are only two categories
27. 1:09to choose from in this case the two
28. 1:11choices were has heart disease or does
29. 1:14not have heart disease then the top
30. 1:17left-hand corner contains the true
31. 1:19positives true positives are patients
32. 1:23that had heart disease that were also
33. 1:25predicted to have heart disease true
34. 1:29negatives are in the bottom right hand
35. 1:30corner true negatives are patients that
36. 1:34did not have heart disease and were
37. 1:36predicted not to have heart disease the
38. 1:40bottom left-hand corner contains the
39. 1:42false negatives false negatives are when
40. 1:45a patient has heart disease but the
41. 1:48prediction said they didn't lastly the
42. 1:52top right hand corner contains the false
43. 1:54positives false positives are patients
44. 1:57that do not have heart disease but the
45. 1:59prediction says that they do once we
46. 2:03filled out the confusion matrix we can
47. 2:05calculate two useful metrics sensitivity
48. 2:09and specificity in this case sensitivity
49. 2:13tells us what percentage of patients
50. 2:15with heart disease were correctly
51. 2:18identified
52. 2:20sensitivity is the true positives
53. 2:23divided by the sum of the true positives
54. 2:26and the false negatives
55. 2:29specificity tells us what percentage of
56. 2:32patients without heart disease were
57. 2:34correctly identified specificity are the
58. 2:39true negatives divided by the sum of the
59. 2:42true negatives and the false positives
60. 2:46in the StatQuest on the confusion
61. 2:48matrix we applied logistic regression to
62. 2:51a testing data set and ended up with
63. 2:55this confusion matrix let's start by
64. 2:59calculating sensitivity for this
65. 3:01logistic regression here's the formula
66. 3:05for sensitivity and for true positives
67. 3:08we plug in 139 and for false negatives
68. 3:13we plug in 32 when we do the math we get
69. 3:18zero point eight one sensitivity tells
70. 3:22us that 81% of the people with heart
71. 3:25disease were correctly identified by the
72. 3:27logistic regression model
73. 3:30now let's calculate the specificity
74. 3:34here's the formula for specificity and
75. 3:37for true negatives we will plug in 112 and
76. 3:42for false positives we will plug in 20 when
77. 3:47we do the math we get 0.85 specificity
78. 3:52tells us that 85% of the people without
79. 3:55heart disease were correctly identified
80. 3:58by the logistic regression model now
81. 4:02let's calculate sensitivity and
82. 4:03specificity for the random forest model
83. 4:06that we used in the confusion matrix
84. 4:08StatQuest
85. 4:10here's the confusion matrix here's the
86. 4:14formula for sensitivity and when we plug
87. 4:17in the numbers we get zero point eight
88. 4:19three
89. 4:21here's the formula for specificity and
90. 4:24when we plug in the numbers we get zero
91. 4:27point eight three again
92. 4:30now we can compare the sensitivity and
93. 4:33specificity values that we calculated
94. 4:35for the logistic regression to the
95. 4:38values we calculated for the random
96. 4:40forest
97. 4:42sensitivity tells us that the random
98. 4:45forest is slightly better at correctly
99. 4:47identifying positives which in this case
100. 4:50are patients with heart disease
101. 4:54specificity tells us that logistic
102. 4:56regression is slightly better correctly
103. 4:59identifying negatives which in this case
104. 5:02are patients without heart disease we
105. 5:06would choose the logistic regression
106. 5:08model
107. 5:08if correctly identifying patients
108. 5:10without heart disease was more important
109. 5:13than correctly identifying patients with
110. 5:15heart disease
111. 5:18alternatively we would choose the random
112. 5:21forest model if correctly identifying
113. 5:23patients with heart disease was more
114. 5:26important than correctly identifying
115. 5:28patients without heart disease BAM
116. 5:33in the confusion matrix stat quest we
117. 5:36calculated this confusion matrix when we
118. 5:39tried to predict someone's favorite movie
119. 5:42now let's talk about how to calculate
120. 5:44sensitivity and specificity when we have
121. 5:47a confusion matrix with three rows and
122. 5:49three columns the big difference when
123. 5:53calculating sensitivity and specificity
124. 5:55for larger confusion matrices is that
125. 5:59there are no single values that work for
126. 6:01the entire matrix instead we calculate a
127. 6:06different sensitivity and specificity
128. 6:07for each category
129. 6:11so for this confusion matrix we'll need
130. 6:14to calculate sensitivity and specificity
131. 6:16for the movie troll 2 for the movie Gore
132. 6:20police and for the movie cool as ice
133. 6:25let's start by calculating sensitivity
134. 6:28for troll 2 for troll 2 there were 12
135. 6:32true positives people that were
136. 6:35correctly predicted to love troll 2 more
137. 6:38than Gore police and cool as ice
138. 6:41so for true positives we'll plug in 12
139. 6:46and there were 112 plus 83 which equals
140. 6:50195 false negatives people that love to
141. 6:55troll 2 but were predicted to love
142. 6:57Gore police or cool as ice
143. 7:00so for false negatives will plug in 195
144. 7:04and when we do the math we get 0.06
145. 7:11sensitivity for troll 2 tells us that
146. 7:13only 6% of the people that loved the
147. 7:16movie troll 2 more than Gore police or
148. 7:19cool as ice were correctly identified
149. 7:22now let's calculate the specificity for
150. 7:25troll 2 there were 23 plus 77 plus 92
151. 7:31plus 17 equals 209 true negatives people
152. 7:36that were correctly predicted to like
153. 7:38Gore police or cool as ice more than
154. 7:41troll 2 so for true negatives will plug
155. 7:45in 209
156. 7:47and there were 102 plus 93 equals 195
157. 7:52false positives people that loved gore
158. 7:55police or cool as ice the most but were
159. 7:59predicted to love troll 2 so for false
160. 8:02positives will plug in 195 and when we
161. 8:07do the math we get 0.52 specificity
162. 8:12for troll 2 tells us that 52% of the
163. 8:16people who loved Gore police or cool as
164. 8:18ice more than troll 2 were correctly
165. 8:21identified
166. 8:23calculating sensitivity and specificity
167. 8:26for Gore police is very similar let's
168. 8:30start by calculating sensitivity
169. 8:34there are 23 true positives people that
170. 8:37were correctly predicted to love Gore
171. 8:39police the most
172. 8:42and 102 plus 92 equals 194 false
173. 8:47negatives people who loved Gore police
174. 8:50the most but were predicted to love
175. 8:52troll 2 or cool as ice more when we do the
176. 8:56math we get 0.11 sensitivity
177. 9:01for Gore police tells us that only 11%
178. 9:04of the people that loved Gore police
179. 9:06were correctly identified now let's
180. 9:10calculate specificity there were 12 plus
181. 9:1493 plus 83 plus 17 equals 205 true
182. 9:19negatives people correctly identified as
183. 9:22loving troll 2 or cool as ice more than
184. 9:25gore police
185. 9:27and 112 plus 77 equals 189 false
186. 9:32positives people predicted to love gore
187. 9:35police even though they didn't and when
188. 9:39we do the math we get 0.52 specificity
189. 9:44for gore police tells us that 52% of the
190. 9:47people that loved troll 2 or cool as
191. 9:50ice more than gore police were correctly
192. 9:52identified
193. 9:54lastly calculating sensitivity and
194. 9:57specificity for cool as ice follows the
195. 10:00same steps we identify the true
196. 10:03positives the false positives the true
197. 10:07negatives and the false negatives and
198. 10:11then plug in the numbers first for
199. 10:15sensitivity then for specificity double bam
200. 10:20if we had a confusion matrix with
201. 10:24four rows and four columns then we would
202. 10:28have to calculate sensitivity and
203. 10:29specificity for four different
204. 10:31categories little bam
205. 10:38in summary sensitivity equals the true positives
206. 10:41divided by the sum of the true positives
207. 10:44and the false negatives and specificity
208. 10:48equals the true negatives divided by the
209. 10:51sum of the true negatives and the false
210. 10:53positives we can use sensitivity and
211. 10:57specificity to help us decide which
212. 11:00machine learning method would be best
213. 11:02for our data if correctly identifying
214. 11:06positives is the most important thing to
215. 11:08do with the data we should choose a
216. 11:10method with higher sensitivity if
217. 11:13correctly identifying negatives is more
218. 11:16important then we should put more
219. 11:17emphasis on specificity
220. 11:21hooray we've made it to the end of
221. 11:23another exciting StatQuest if you like
222. 11:26this StatQuest and want to see more
223. 11:28please subscribe and if you want to
224. 11:30support stack quest well consider buying
225. 11:33one or two of my original songs alright
226. 11:35until next time quest on