But what is a neural network? | Deep learning chapter 1 — Transcript

Full transcript

1. 0:04This is a 3.
2. 0:06It's sloppily written and rendered at an extremely low resolution of 28x28 pixels,
3. 0:10but your brain has no trouble recognizing it as a 3.
4. 0:14And I want you to take a moment to appreciate how
5. 0:16crazy it is that brains can do this so effortlessly.
6. 0:19I mean, this, this and this are also recognizable as 3s,
7. 0:22even though the specific values of each pixel is very different from one
8. 0:27image to the next.
9. 0:28The particular light-sensitive cells in your eye that are firing when you
10. 0:32see this 3 are very different from the ones firing when you see this 3.
11. 0:37But something in that crazy-smart visual cortex of yours resolves these as representing
12. 0:42the same idea, while at the same time recognizing other images as their own distinct
13. 0:47ideas.
14. 0:49But if I told you, hey, sit down and write for me a program that takes in a grid of
15. 0:5428x28 pixels like this and outputs a single number between 0 and 10,
16. 0:59telling you what it thinks the digit is, well the task goes from comically trivial to
17. 1:04dauntingly difficult.
18. 1:07Unless you've been living under a rock, I think I hardly need to motivate the relevance
19. 1:10and importance of machine learning and neural networks to the present and to the future.
20. 1:15But what I want to do here is show you what a neural network actually is,
21. 1:18assuming no background, and to help visualize what it's doing,
22. 1:22not as a buzzword but as a piece of math.
23. 1:25My hope is that you come away feeling like the structure itself is motivated,
24. 1:28and to feel like you know what it means when you read,
25. 1:31or you hear about a neural network quote-unquote learning.
26. 1:35This video is just going to be devoted to the structure component of that,
27. 1:38and the following one is going to tackle learning.
28. 1:40What we're going to do is put together a neural
29. 1:43network that can learn to recognize handwritten digits.
30. 1:49This is a somewhat classic example for introducing the topic,
31. 1:52and I'm happy to stick with the status quo here,
32. 1:54because at the end of the two videos I want to point you to a couple good
33. 1:57resources where you can learn more, and where you can download the code that
34. 2:00does this and play with it on your own computer.
35. 2:05There are many many variants of neural networks,
36. 2:07and in recent years there's been sort of a boom in research towards these variants,
37. 2:12but in these two introductory videos you and I are just going to look at the simplest
38. 2:16plain vanilla form with no added frills.
39. 2:19This is kind of a necessary prerequisite for understanding any of the more powerful
40. 2:23modern variants, and trust me it still has plenty of complexity for us to wrap our minds
41. 2:28around.
42. 2:29But even in this simplest form it can learn to recognize handwritten digits,
43. 2:33which is a pretty cool thing for a computer to be able to do.
44. 2:37And at the same time you'll see how it does fall
45. 2:39short of a couple hopes that we might have for it.
46. 2:43As the name suggests neural networks are inspired by the brain, but let's break that down.
47. 2:48What are the neurons, and in what sense are they linked together?
48. 2:52Right now when I say neuron all I want you to think about is a thing that holds a number,
49. 2:58specifically a number between 0 and 1.
50. 3:00It's really not more than that.
51. 3:03For example the network starts with a bunch of neurons corresponding to
52. 3:08each of the 28x28 pixels of the input image, which is 784 neurons in total.
53. 3:14Each one of these holds a number that represents the grayscale value of the
54. 3:19corresponding pixel, ranging from 0 for black pixels up to 1 for white pixels.
55. 3:25This number inside the neuron is called its activation,
56. 3:28and the image you might have in mind here is that each neuron is lit up when its
57. 3:32activation is a high number.
58. 3:36So all of these 784 neurons make up the first layer of our network.
59. 3:46Now jumping over to the last layer, this has 10 neurons,
60. 3:49each representing one of the digits.
61. 3:52The activation in these neurons, again some number that's between 0 and 1,
62. 3:56represents how much the system thinks that a given image corresponds with a given digit.
63. 4:03There's also a couple layers in between called the hidden layers,
64. 4:06which for the time being should just be a giant question mark for
65. 4:09how on earth this process of recognizing digits is going to be handled.
66. 4:14In this network I chose two hidden layers, each one with 16 neurons,
67. 4:17and admittedly that's kind of an arbitrary choice.
68. 4:21To be honest I chose two layers based on how I want to motivate the structure in
69. 4:24just a moment, and 16, well that was just a nice number to fit on the screen.
70. 4:28In practice there is a lot of room for experiment with a specific structure here.
71. 4:33The way the network operates, activations in one
72. 4:35layer determine the activations of the next layer.
73. 4:39And of course the heart of the network as an information processing mechanism comes down
74. 4:43to exactly how those activations from one layer bring about activations in the next
75. 4:48layer.
76. 4:49It's meant to be loosely analogous to how in biological networks of neurons,
77. 4:53some groups of neurons firing cause certain others to fire.
78. 4:58Now the network I'm showing here has already been trained to recognize digits,
79. 5:01and let me show you what I mean by that.
80. 5:03It means if you feed in an image, lighting up all 784 neurons of the input layer
81. 5:08according to the brightness of each pixel in the image,
82. 5:11that pattern of activations causes some very specific pattern in the next layer
83. 5:16which causes some pattern in the one after it,
84. 5:18which finally gives some pattern in the output layer.
85. 5:22And the brightest neuron of that output layer is the network's choice,
86. 5:26so to speak, for what digit this image represents.
87. 5:32And before jumping into the math for how one layer influences the next,
88. 5:36or how training works, let's just talk about why it's even reasonable
89. 5:40to expect a layered structure like this to behave intelligently.
90. 5:44What are we expecting here?
91. 5:45What is the best hope for what those middle layers might be doing?
92. 5:48Well, when you or I recognize digits, we piece together various components.
93. 5:54A 9 has a loop up top and a line on the right.
94. 5:57An 8 also has a loop up top, but it's paired with another loop down low.
95. 6:01A 4 basically breaks down into three specific lines, and things like that.
96. 6:07Now in a perfect world, we might hope that each neuron in the second
97. 6:11to last layer corresponds with one of these subcomponents,
98. 6:14that anytime you feed in an image with, say, a loop up top,
99. 6:18like a 9 or an 8, there's some specific neuron whose activation is
100. 6:22going to be close to 1.
101. 6:24And I don't mean this specific loop of pixels,
102. 6:26the hope would be that any generally loopy pattern towards the top sets off this neuron.
103. 6:32That way, going from the third layer to the last one just requires
104. 6:36learning which combination of subcomponents corresponds to which digits.
105. 6:41Of course, that just kicks the problem down the road,
106. 6:43because how would you recognize these subcomponents,
107. 6:45or even learn what the right subcomponents should be?
108. 6:48And I still haven't even talked about how one layer influences the next,
109. 6:51but run with me on this one for a moment.
110. 6:53Recognizing a loop can also break down into subproblems.
111. 6:57One reasonable way to do this would be to first
112. 6:59recognize the various little edges that make it up.
113. 7:03Similarly, a long line, like the kind you might see in the digits 1 or 4 or 7,
114. 7:08is really just a long edge, or maybe you think of it as a certain pattern of several
115. 7:13smaller edges.
116. 7:15So maybe our hope is that each neuron in the second layer of
117. 7:18the network corresponds with the various relevant little edges.
118. 7:23Maybe when an image like this one comes in, it lights up all of the
119. 7:27neurons associated with around 8 to 10 specific little edges,
120. 7:31which in turn lights up the neurons associated with the upper loop
121. 7:35and a long vertical line, and those light up the neuron associated with a 9.
122. 7:40Whether or not this is what our final network actually does is another question,
123. 7:44one that I'll come back to once we see how to train the network,
124. 7:47but this is a hope that we might have, a sort of goal with the layered structure
125. 7:51like this.
126. 7:53Moreover, you can imagine how being able to detect edges and patterns
127. 7:56like this would be really useful for other image recognition tasks.
128. 8:00And even beyond image recognition, there are all sorts of intelligent
129. 8:04things you might want to do that break down into layers of abstraction.
130. 8:08Parsing speech, for example, involves taking raw audio and picking out distinct sounds,
131. 8:12which combine to make certain syllables, which combine to form words,
132. 8:16which combine to make up phrases and more abstract thoughts, etc.
133. 8:21But getting back to how any of this actually works,
134. 8:23picture yourself right now designing how exactly the activations in one layer might
135. 8:27determine the activations in the next.
136. 8:30The goal is to have some mechanism that could conceivably combine pixels into edges,
137. 8:35or edges into patterns, or patterns into digits.
138. 8:39And to zoom in on one very specific example, let's say the hope
139. 8:43is for one particular neuron in the second layer to pick up
140. 8:46on whether or not the image has an edge in this region here.
141. 8:51The question at hand is what parameters should the network have?
142. 8:55What dials and knobs should you be able to tweak so that it's expressive
143. 8:59enough to potentially capture this pattern, or any other pixel pattern,
144. 9:03or the pattern that several edges can make a loop, and other such things?
145. 9:08Well, what we'll do is assign a weight to each one of the
146. 9:11connections between our neuron and the neurons from the first layer.
147. 9:16These weights are just numbers.
148. 9:18Then take all of those activations from the first layer
149. 9:21and compute their weighted sum according to these weights.
150. 9:27I find it helpful to think of these weights as being organized into a
151. 9:31little grid of their own, and I'm going to use green pixels to indicate
152. 9:34positive weights, and red pixels to indicate negative weights,
153. 9:37where the brightness of that pixel is some loose depiction of the weight's value.
154. 9:42Now if we made the weights associated with almost all of the pixels zero
155. 9:46except for some positive weights in this region that we care about,
156. 9:50then taking the weighted sum of all the pixel values really just amounts
157. 9:53to adding up the values of the pixel just in the region that we care about.
158. 9:59And if you really wanted to pick up on whether there's an edge here,
159. 10:02what you might do is have some negative weights associated with the surrounding pixels.
160. 10:07Then the sum is largest when those middle pixels
161. 10:10are bright but the surrounding pixels are darker.
162. 10:14When you compute a weighted sum like this, you might come out with any number,
163. 10:18but for this network what we want is for activations to be some value between 0 and 1.
164. 10:24So a common thing to do is to pump this weighted sum into some function
165. 10:28that squishes the real number line into the range between 0 and 1.
166. 10:32And a common function that does this is called the sigmoid function,
167. 10:35also known as a logistic curve.
168. 10:38Basically very negative inputs end up close to 0, positive inputs end up close to 1,
169. 10:43and it just steadily increases around the input 0.
170. 10:49So the activation of the neuron here is basically a
171. 10:52measure of how positive the relevant weighted sum is.
172. 10:57But maybe it's not that you want the neuron to
173. 10:59light up when the weighted sum is bigger than 0.
174. 11:02Maybe you only want it to be active when the sum is bigger than say 10.
175. 11:06That is, you want some bias for it to be inactive.
176. 11:11What we'll do then is just add in some other number like negative 10 to this
177. 11:15weighted sum before plugging it through the sigmoid squishification function.
178. 11:20That additional number is called the bias.
179. 11:23So the weights tell you what pixel pattern this neuron in the second
180. 11:27layer is picking up on, and the bias tells you how high the weighted
181. 11:31sum needs to be before the neuron starts getting meaningfully active.
182. 11:36And that is just one neuron.
183. 11:38Every other neuron in this layer is going to be connected to
184. 11:42all 784 pixel neurons from the first layer, and each one of
185. 11:46those 784 connections has its own weight associated with it.
186. 11:51Also, each one has some bias, some other number that you add
187. 11:54on to the weighted sum before squishing it with the sigmoid.
188. 11:58And that's a lot to think about!
189. 11:59With this hidden layer of 16 neurons, that's a total of 784 times 16 weights,
190. 12:06along with 16 biases.
191. 12:08And all of that is just the connections from the first layer to the second.
192. 12:12The connections between the other layers also have
193. 12:14a bunch of weights and biases associated with them.
194. 12:18All said and done, this network has almost exactly 13,000 total weights and biases.
195. 12:2313,000 knobs and dials that can be tweaked and turned
196. 12:27to make this network behave in different ways.
197. 12:31So when we talk about learning, what that's referring to is
198. 12:34getting the computer to find a valid setting for all of these
199. 12:37many many numbers so that it'll actually solve the problem at hand.
200. 12:42One thought experiment that is at once fun and kind of horrifying is to imagine sitting
201. 12:47down and setting all of these weights and biases by hand,
202. 12:50purposefully tweaking the numbers so that the second layer picks up on edges,
203. 12:54the third layer picks up on patterns, etc.
204. 12:56I personally find this satisfying rather than just treating the network as a total black
205. 13:01box, because when the network doesn't perform the way you anticipate,
206. 13:04if you've built up a little bit of a relationship with what those weights and biases
207. 13:09actually mean, you have a starting place for experimenting with how to change the
208. 13:13structure to improve.
209. 13:14Or when the network does work but not for the reasons you might expect,
210. 13:18digging into what the weights and biases are doing is a good way to challenge
211. 13:22your assumptions and really expose the full space of possible solutions.
212. 13:26By the way, the actual function here is a little cumbersome to write down,
213. 13:29don't you think?
214. 13:32So let me show you a more notationally compact way that these connections are represented.
215. 13:37This is how you'd see it if you choose to read up more about neural networks.
216. 13:40Organize all of the activations from one layer into a column as a vector.
217. 13:48Then organize all of the weights as a matrix, where each row of that matrix corresponds
218. 13:50to the connections between one layer and a particular neuron in the next layer.
219. 13:58What that means is that taking the weighted sum of the activations in
220. 14:02the first layer according to these weights corresponds to one of the
221. 14:05terms in the matrix vector product of everything we have on the left here.
222. 14:14By the way, so much of machine learning just comes down to having a good
223. 14:17grasp of linear algebra, so for any of you who want a nice visual
224. 14:21understanding for matrices and what matrix vector multiplication means,
225. 14:24take a look at the series I did on linear algebra, especially chapter 3.
226. 14:29Back to our expression, instead of talking about adding the bias to each one of
227. 14:33these values independently, we represent it by organizing all those biases into
228. 14:38a vector, and adding the entire vector to the previous matrix vector product.
229. 14:43Then as a final step, I'll wrap a sigmoid around the outside here,
230. 14:46and what that's supposed to represent is that you're going to apply the
231. 14:50sigmoid function to each specific component of the resulting vector inside.
232. 14:55So once you write down this weight matrix and these vectors as their own symbols,
233. 15:00you can communicate the full transition of activations from one layer to the next in an
234. 15:05extremely tight and neat little expression, and this makes the relevant code both a lot
235. 15:10simpler and a lot faster, since many libraries optimize the heck out of matrix
236. 15:14multiplication.
237. 15:17Remember how earlier I said these neurons are simply things that hold numbers?
238. 15:22Well of course the specific numbers that they hold depends on the image you feed in,
239. 15:27so it's actually more accurate to think of each neuron as a function,
240. 15:31one that takes in the outputs of all the neurons in the previous layer and spits out a
241. 15:36number between 0 and 1.
242. 15:39Really the entire network is just a function, one that takes in
243. 15:43784 numbers as an input and spits out 10 numbers as an output.
244. 15:47It's an absurdly complicated function, one that involves 13,000 parameters
245. 15:51in the forms of these weights and biases that pick up on certain patterns,
246. 15:55and which involves iterating many matrix vector products and the sigmoid
247. 15:59squishification function, but it's just a function nonetheless.
248. 16:03And in a way it's kind of reassuring that it looks complicated.
249. 16:07I mean if it were any simpler, what hope would we have
250. 16:09that it could take on the challenge of recognizing digits?
251. 16:13And how does it take on that challenge?
252. 16:15How does this network learn the appropriate weights and biases just by looking at data?
253. 16:20Well that's what I'll show in the next video, and I'll also dig a little
254. 16:23more into what this particular network we're seeing is really doing.
255. 16:27Now is the point I suppose I should say subscribe to stay notified
256. 16:30about when that video or any new videos come out,
257. 16:33but realistically most of you don't actually receive notifications from YouTube, do you?
258. 16:38Maybe more honestly I should say subscribe so that the neural networks
259. 16:41that underlie YouTube's recommendation algorithm are primed to believe
260. 16:44that you want to see content from this channel get recommended to you.
261. 16:48Anyway, stay posted for more.
262. 16:50Thank you very much to everyone supporting these videos on Patreon.
263. 16:54I've been a little slow to progress in the probability series this summer,
264. 16:57but I'm jumping back into it after this project,
265. 16:59so patrons you can look out for updates there.
266. 17:03To close things off here I have with me Lisha Li who did her PhD work on the
267. 17:07theoretical side of deep learning and who currently works at a venture capital
268. 17:10firm called Amplify Partners who kindly provided some of the funding for this video.
269. 17:15So Lisha one thing I think we should quickly bring up is this sigmoid function.
270. 17:19As I understand it early networks use this to squish the relevant weighted
271. 17:23sum into that interval between zero and one, you know kind of motivated
272. 17:26by this biological analogy of neurons either being inactive or active.
273. 17:30Exactly. But relatively few modern networks actually use sigmoid anymore.
274. 17:34Yeah. It's kind of old school right?
275. 17:35Yeah or rather ReLU seems to be much easier to train.
276. 17:39And ReLU, ReLU stands for rectified linear unit?
277. 17:42Yes it's this kind of function where you're just taking a max of zero
278. 17:47and a where a is given by what you were explaining in the video and
279. 17:52what this was sort of motivated from I think was a partially by a
280. 17:56biological analogy with how neurons would either be activated or not.
281. 18:01And so if it passes a certain threshold it would be the identity function but if it did
282. 18:06not then it would just not be activated so it'd be zero so it's kind of a simplification.
283. 18:11Using sigmoids didn't help training or it was very difficult to
284. 18:15train at some point and people just tried ReLU and it happened
285. 18:20to work very well for these incredibly deep neural networks.
286. 18:25All right thank you Lisha.