Machine Learning Fundamentals: Cross Validation — Transcript

Full transcript

1. 0:00StatQuest
2. 0:01Check it out
3. 0:03talking about
4. 0:05Machine-learning. Yeah StatQuest
5. 0:08Check it out
6. 0:09Talking about cross-validation
7. 0:12StatQuest
8. 0:15Hello, I'm Josh stormer and welcome to StatQuest today we're going to talk about cross validation and it's gonna be clearly explained
9. 0:25Okay, let's start with some data
10. 0:28We want to use the variables chest pain good blood circulation
11. 0:33Etc
12. 0:34To predict if someone has heart disease
13. 0:37Then when a new patient shows up
14. 0:40we can measure these variables and
15. 0:43Predict if they have heart disease or not
16. 0:47However, first we have to decide which machine learning method would be best
17. 0:53we could use logistic regression or
18. 0:56K nearest neighbors
19. 0:59Or support vector machines and
20. 1:03Many more machine learning methods. How do we decide which one to use?
21. 1:09Cross-validation allows us to compare different machine learning methods and get a sense of how well they will work in practice
22. 1:19Imagine that this blue column represented all of the data that we have collected about people with and without heart disease
23. 1:27We need to do two things with this data
24. 1:30One we need to estimate the parameters for the machine learning methods in
25. 1:36In other words to use logistic regression we have to use some of the data to estimate the shape of this curve
26. 1:44in machine learning lingo
27. 1:47Estimating parameters is called training the algorithm
28. 1:51The second thing we need to do with this data is evaluate how well the machine learning methods work in?
29. 1:58Other words we need to find out if this curve will do a good job categorizing new data in
30. 2:06In machine learning lingo
31. 2:09Evaluating a method is called testing the algorithm
32. 2:13Thus using machine learning lingo we need the data to
33. 2:18one train the machine learning methods and
34. 2:22to test the machine learning methods a
35. 2:27A terrible approach would be to use all the data to estimate the parameters ie to train the algorithm
36. 2:35Because then we wouldn't have any data left to test the method
37. 2:40Reusing the same data for both training and
38. 2:43Testing is a bad idea because we need to know how the method will work on data. It wasn't trained on a
39. 2:52Slightly better idea would be to use the first seventy-five percent of the data for training and the last 25% of the data for testing
40. 3:02We could then compare methods by seeing how well each one categorized the test data
41. 3:09But how do we know that using the first?
42. 3:11Seventy-five percent of the data for training in the last 25% of the data for testing is the best way to divide up the data
43. 3:21What if we use the first 25% of the data for testing
44. 3:26Or what about one of these middle blocks?
45. 3:29Rather than worry too much about which block would be best for testing cross-validation uses them all one at a time and summarizes the results at the end
46. 3:41For example cross-validation would start by using the first three blocks to train the method and
47. 3:49then use the last block to test the method and
48. 3:53Then it keeps track of how well the method did with the test data
49. 3:58then it uses this combination of blocks to train the method and
50. 4:03this block is used for testing and
51. 4:07Then it keeps track of how well the method did with the test data, etc
52. 4:12Etc, etc
53. 4:16in the end every block of data is used for testing and we can compare methods by seeing how well they performed in
54. 4:25This case since the support vector machine did the best job classifying the test data sets. We'll use it
55. 4:33BAM!!!
56. 4:36Note: in this example, we divided the data into 4 blocks. This is called four-fold cross validation
57. 4:45However, the number of blocks is arbitrary
58. 4:49In an extreme case we could call each individual patient (or sample) a block
59. 4:56This is called "Leave One Out Cross Validation"
60. 4:59Each sample is tested individually
61. 5:03That said in practice it is very common to divide the data into ten blocks. This is called 10-fold cross-validation
62. 5:14Double BAM!!!
63. 5:16One last note before we're done
64. 5:20Say like we wanted to use a method that involved a tuning parameter a parameter that isn't estimated but is just sort of guessed
65. 5:28For example Ridge regression has a tuning parameter
66. 5:33Then we could use 10-fold cross validation
67. 5:36to help find the best value for that tuning parameter
68. 5:40Tiny Bam!
69. 5:42Hooray we've made it to the end of another exciting StatQuest if you like this StatQuest and want to see more please subscribe
70. 5:50And if you want to support StatQuest well
71. 5:54Please click the like button down below and consider buying one of my original songs
72. 5:59Alright until next time quest on