Machine Learning Fundamentals: The Confusion Matrix — Transcript

Full transcript

1. 0:01If you feel confused,
2. 0:05don't sweat it.
3. 0:07StatQuest is here.
4. 0:12StatQuest.
5. 0:14Hello, I'm Josh Starmer and welcome to
6. 0:17StatQuest. Today we're going to cover
7. 0:19another machine learning fundamental,
8. 0:21the confusion matrix, and it's going to
9. 0:23be clearly explained.
10. 0:26Imagine that we have this medical data.
11. 0:29We've got some clinical measurements
12. 0:31like chest pain, good blood circulation,
13. 0:33blocked arteries, and weight.
14. 0:36And we want to apply a machine learning
15. 0:38method to them to predict whether or not
16. 0:40someone will develop heart disease.
17. 0:43To do this, we could use logistic
18. 0:45regression
19. 0:47or K nearest neighbors
20. 0:49or random forest
21. 0:51or some other method. There are tons to
22. 0:54choose from.
23. 0:55How do we decide which one works best
24. 0:58with our data?
25. 1:00We start by dividing the data into
26. 1:02training and testing sets.
27. 1:05Note, this would be an excellent
28. 1:08opportunity to use cross-validation. And
29. 1:10if you're not familiar with that, well,
30. 1:12check out the StatQuest.
31. 1:15Then we train all of the methods we're
32. 1:17interested in with the training data.
33. 1:21And then test each method on the testing
34. 1:23set.
35. 1:26Now we need to summarize how each method
36. 1:28performed on the testing data.
37. 1:32One way to do this is by creating a
38. 1:34confusion matrix for each method.
39. 1:38The rows in a confusion matrix
40. 1:40correspond to what the machine learning
41. 1:42algorithm predicted.
42. 1:44And the columns correspond to the known
43. 1:47truth.
44. 1:49Since there are only two categories to
45. 1:51choose from,
46. 1:52heart disease or does not have heart
47. 1:56disease,
48. 1:58then the top left corner contains true
49. 2:01positives.
50. 2:03These are patients that had heart
51. 2:05disease that were correctly identified
52. 2:08by the algorithm.
53. 2:10The true negatives are in the bottom
54. 2:12right-hand corner.
55. 2:14These are patients that did not have
56. 2:16heart disease that were correctly
57. 2:18identified by the algorithm.
58. 2:21The bottom left-hand corner contains
59. 2:23false negatives.
60. 2:25False negatives are when a patient has
61. 2:28heart disease, but the algorithm said
62. 2:30they didn't.
63. 2:32Lastly, the top right-hand corner
64. 2:34contains false positives.
65. 2:37False positives are patients that do not
66. 2:39have heart disease, but the algorithm
67. 2:42says they do.
68. 2:44For example, when we applied the random
69. 2:46forest to the testing data,
70. 2:49there were 142 true positives, patients
71. 2:53with heart disease that were correctly
72. 2:55classified,
73. 2:57and 110 true negatives, patients without
74. 3:00heart disease that were correctly
75. 3:02classified.
76. 3:04However, the algorithm misclassified 29
77. 3:08patients that did have heart disease by
78. 3:10saying they did not. These are false
79. 3:13negatives.
80. 3:15And the algorithm misclassified 22
81. 3:17patients that did not have heart disease
82. 3:20by saying that they did. These are false
83. 3:22positives.
84. 3:25The numbers along the diagonal, the
85. 3:27green boxes, tell us how many times the
86. 3:30samples were correctly classified.
87. 3:33The numbers not on the diagonal, the red
88. 3:36boxes, are samples that the algorithm
89. 3:39messed up.
90. 3:41Now we can compare the random forest
91. 3:43confusion matrix
92. 3:46to the confusion matrix we get when we
93. 3:48use K nearest neighbors.
94. 3:51K nearest neighbors was worse the random
95. 3:54forest at predicting patients with the
96. 3:56heart disease,
97. 3:58107 versus 142,
98. 4:02and worse at predicting patients without
99. 4:05heart disease, 79 versus 110.
100. 4:10So, if we had to choose between using
101. 4:11the random forest and K-nearest
102. 4:13neighbors, we would choose the random
103. 4:16forest.
104. 4:18Bam!
105. 4:20Lastly, we can apply logistic regression
106. 4:22to the testing data set and create a
107. 4:24confusion matrix.
108. 4:27These two confusion matrices are very
109. 4:29similar and make it hard to choose which
110. 4:32machine learning method is a better fit
111. 4:34for this data.
112. 4:36We'll talk about more sophisticated
113. 4:38metrics like sensitivity, specificity,
114. 4:41ROC, and AOC that can help us make a
115. 4:44decision in the next Stack Quests.
116. 4:48Now that we have the basic confusion
117. 4:50matrix figured out, let's look at a more
118. 4:52complicated one.
119. 4:55Here's a new data set.
120. 4:57Now the question is, based on what
121. 5:00people think of these movies, Jurassic
122. 5:02Park III, Run for Your Wife, Out Cold,
123. 5:06spelled with a K, and Howard the Duck,
124. 5:09can we use a machine learning method to
125. 5:11predict their favorite movie?
126. 5:15If the only options for favorite movie
127. 5:17were Troll 2, Gore Police, or Cool as
128. 5:21Ice,
129. 5:23then the confusion matrix would have
130. 5:24three rows and three columns.
131. 5:28But just like before, the diagonal, the
132. 5:31green boxes, are where the machine
133. 5:33learning algorithm did the right thing,
134. 5:36and everything else is where the
135. 5:38algorithm messed up.
136. 5:40In this case, the machine learning
137. 5:42algorithm didn't do very well, but can
138. 5:45you blame it? These are all terrible
139. 5:47movies.
140. 5:49Bam!
141. 5:51Ultimately, the size of the confusion
142. 5:53matrix is determined by the number of
143. 5:55things we want to predict.
144. 5:58In the first example, we were only
145. 6:00trying to predict two things, if someone
146. 6:02had heart disease or if they didn't.
147. 6:05And that gave us a confusion matrix with
148. 6:08two rows and two columns.
149. 6:11In the second example, we had three
150. 6:13things to choose from.
151. 6:15And a confusion matrix with three rows
152. 6:18and three columns.
153. 6:20If we had four things to choose from, we
154. 6:23get a confusion matrix with four rows
155. 6:25and four columns.
156. 6:28And if we had 40 things to choose from,
157. 6:30we get a confusion matrix with 40 rows
158. 6:33and 40 columns.
159. 6:35Double bam!
160. 6:38In summary, a confusion matrix tells you
161. 6:41what your machine learning algorithm did
162. 6:43right
163. 6:44and what it did wrong.
164. 6:47Hooray! We've made it to the end of
165. 6:49another exciting StatQuest. If you like
166. 6:52this StatQuest and want to see more,
167. 6:54please subscribe. And if you want to
168. 6:55support StatQuest, well, consider buying
169. 6:58one or two of my original songs. All
170. 7:01right, until next time, quest on.