YT YouTube2Text

YouTube2Text → Transcripts → EfficientML.ai Lecture 3 - Pruning and Sparsity Part I (MIT 6.5940, Fall 2024)

EfficientML.ai Lecture 3 - Pruning and Sparsity Part I (MIT 6.5940, Fall 2024) — Transcript

by MIT HAN Lab · 7,658 words · 1,151 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:01all right good afternoon everyone let's
0:03get started welcome to the efficient ml.
0:07today we are going to introduce lecture
0:08three about pruning and sparsity to
0:11accelerate the inference of neural
0:14networks so we'll have two lectures
0:17about pruni and sparcity so today we
0:19will have the first lecture about
0:21that so this is the part overview of the
0:24part one of this lecture about efficient
0:27difference we are going to cover four
0:30sections okay we are going to start
0:33with talking about the um here talking
0:37about pring and
0:38sparcity followed by quantization like
0:41integer quantization rp4 rp8 and also
0:45neural architecture search as the third
0:47part how to design efficient neural
0:49architectures even before you compress
0:52them and also we are going to conu that
0:54by knowledge distillation how to use a
0:57larger Network to distill to teach a
1:00smaller Network okay um so this is the
1:03agenda for uh the first part of this
1:06course about pruning we are going to
1:08conclude it by a real world example
1:12designing mcet and tiny tiny engine um
1:15to fit the neural network into a
1:17microcontroller by using all the
1:19techniques we learned uh in these four
1:22sections so today we are going to jump
1:25into the first part which is about
1:28pruning so before jumping into pring
1:30let's start with some uh motivation and
1:32background so when you first introduce
1:35this ml perf Benchmark which is the
1:38Olympic game for efficient air Computing
1:41for air Hardware okay so ml perf
1:43basically uh tests the performance in
1:47latency in throughput given the
1:49particular accuracy on The Suite of
1:51benchmarks there are two divisions the
1:54open Division and the Clos division so
1:56in the Clos division you cannot change
1:58neuron Network you can only apply
2:01quantization on the open division you're
2:03freid to change the neural network
2:05architecture and apply techniques such
2:07as pruning so this is the latest
2:09Benchmark from ndia Blackwell platform
2:13which was just a few weeks ago in late
2:16August so in the Clos division um the
2:19offline samples per second can reach
2:2444488 tokens per second this is Lama 2
2:2770b primer large language model Runing
2:30on a single Nvidia h200 GPU in the open
2:36division on the second column the L The
2:39throughput increased to more than 11,000
2:42tokens per second this is a big
2:45Improvement and how is that achieved and
2:48that is actually using the techniques we
2:49are going to learn in this lecture about
2:52pruning so uh this technique applied two
2:55pruning techniques one is St pruning
2:58reducing the number of layers from 8 to
3:0032 followed by the width pruning
3:03reducing the uh Channel Dimension from
3:0628,000 to 14 14,000 and as a result
3:10there's about two and a half X speed up
3:13running the at a good
3:17accy so hopefully that motivates uh the
3:20importance of pruning and from the
3:22hardware perspective why do we need prum
3:25so memory is very expensive computation
3:28is much cheaper uh a memory movement is
3:31more than two orders of magnitude than
3:33the arithmetic operations like we have
3:35seen in the previous lecture the 32 bit
3:38ad takes less than one p jeol while a
3:4132bit dam access Dam memory access is
3:45causing more than 600 PJs so uh data
3:49movement is much more expensive so to
3:52make deep burning more efficient we want
3:54to reduce the amount of memory reduce
3:56the model size reduce the activation
3:59size
4:01okay so with that as the motivation
4:03let's start with the agenda we are going
4:05to start by introducing what is pruning
4:08taking a dense neuron Network into a
4:10sparse neuron Network in general we can
4:13formulate pruning uh in this way so we
4:16want to minimize the loss of the prun
4:18model uh given the input X here the L
4:25indicates go back so here the L
4:28indicates the objective function for the
4:31neuron Network training we want to
4:33minimize this loss function when we have
4:36the pr weight WP okay subject to the
4:40number of non zeros okay the number of
4:43non zeros should be smaller than a
4:45threshold okay so we only want to have a
4:47limited amount of non zero elements in
4:50the neuron Network to make it sparse
4:53okay to make it a sparse and still
4:55minimize the
4:57loss and then we are going to introduce
5:00the pruning granularity in P grein
5:02versus fine grain and then the pring
5:05Criterion which neuron which synapsis to
5:07remove which neuron which synapsis to
5:09keep Follow by determining the prunin
5:12ratio what is the redundancy can we
5:15reduce a lot can we reduce just a little
5:17bit how do we select the protein ratio
5:20while maximizing the um amount of new
5:24connections that is PR and minimizing
5:26the loss of accuracy and finally we're
5:29going to talk about about how to fine
5:30tune how to Rin the network so that we
5:33can recover the accuracy as the same as
5:36before Runing okay so with that let's
5:40start with the first part about
5:41introduction to pruning so the pruning
5:44mechanism actually actually happens in
5:48human brain so according to this study
5:50from nature a newborn child has about uh
5:552,500 synapses per neuron okay and when
5:59uh K grows to to four years old this
6:03number surges to 15,000 synapsis per
6:06neuron surges a lot but during
6:09adolesence this number didn't keep
6:11increasing but started to
6:13decrease from 15,000 to only 7,000
6:16synapsis per neural actually the
6:19adolescence time is when we go to school
6:21when we go to college that's the time we
6:23when we learn the most amount of
6:25knowledge through our lifetime and
6:27proing natur happens in that time
6:31and pruning also happens in artificial
6:33neuron networks okay so we can make
6:36neuron networks smaller by removing
6:40those synapsis and neurons that is
6:42redundant so on the left hand side is
6:45showing a dense neuron Network before
6:48pruning here consist of three fully
6:51connected layers every each layer is
6:54densely connected on the right hand side
6:57we can do pruning on this both the
6:59synapsis
7:00and also on the neurons so that not all
7:03the neurons is connected to each
7:07other and how does this impact the
7:11accuracy so I carried an experiment back
7:13in 2015 we first train the neural
7:16network and this is Alex net where can
7:20achieve uh this is accuracy we can
7:22achieve this is the Baseline accuracy
7:24and this is the distribution of the
7:26weight for particular layer roughly from
7:29formulates a normal
7:31distribution and then we can gradually
7:34prove the connections so previously was
7:36Den now it becom sparse okay as we are
7:40getting sparer we remove those small
7:43connections those small weights as a
7:46result you can see the distribution all
7:49the weights centered around zero they
7:52disappeared we remove remove them
7:54meaning that we've zero them M okay so
7:57the distribution changed the small value
8:00disappear and you can see the more we
8:02prove the less the accuracy the more we
8:05prove the less accuracy the Y AIS is the
8:08accuracy loss okay the xais is the pring
8:12raion the primet pro away um on the
8:16right hand side you Pro away all the
8:18parameters you can expect the accuracy
8:20will drop to zero and on the left hand
8:23side here we are starting from 40% ring
8:27ratio on the left
8:29so this is quite unfortunate right
8:31people TR these neural networks to get
8:34high accuracy suddenly due to pruning
8:37you immedately reduce the accuracy by
8:39here is like 1% here is like 2% that is
8:43full B right and we do
8:45better we can actually put in the
8:48remaining weights to recover that RK
8:50okay so comparing this one versus after
8:54we train the remaining weights that
8:56survived the pruning okay train this
8:59weights that survived the pruning and
9:03actually the curve moved to the upper
9:06right corner what do me move into the
9:08upper right corner we can Pro more
9:11parameters we can Pro more parameters or
9:14we can achieve the higher accuracy given
9:16the same tring ratio okay and as a
9:19result after the ring the weight
9:22distribution also shifted from this way
9:26to this way becomes smoother okay
9:30that's pretty good actually we can Pro
9:32away here is 90% of the parameters
9:36without hting the accuracy here on Alex
9:38that's pretty good can we do even better
9:42how can we put more without losing the
9:46accuracy actually we can do this process
9:50iteratively right not just do one
9:52iteration but compare here and here we
9:56can do another round of tring okay
9:59so from the green curve to the red curve
10:03we can p even more so about 90% P away
10:0790% of the parameters without hurting
10:09the
10:10accuracy so that debutes the whole
10:13process you're between the original
10:16network uh you prune some of the weights
10:18according to the magnitude and then you
10:20retrain the remaining weights that
10:22survive pruning and don't go too uh too
10:26aggressive at each step you want to go
10:29any smaller steps um each step should be
10:33less aggressive so that you can push the
10:35boundary of this pruning
10:38process as a result these are the
10:41networks where uh including Alex Net v
10:44Google net res squeet at a time we can
10:47see alexnet can be proved from 61
10:50million parameters to only six million
10:52parameters about nine times cor ratio
10:56and the max reduction is actually
10:58smaller it's 3x
10:59in the last lecture we learned the
11:01difference between number of parameters
11:04versus the number of Max how to
11:05calculate them for convolution layer for
11:07FC layer and also for tension layer so
11:11they are not
11:13equal for example even networks that is
11:16super small like squeez net which are
11:20Cur and the uh for is design in 2016
11:24it's already pretty small to begin with
11:26it has the same accuracy as alex9 but
11:28being 6 times smaller xet 61 million
11:32parameters sque net only one million
11:35parameter so we can still Pro such a
11:37very small and compact model OKAY from
11:39one million parameter to only 0.38
11:42million
11:43parameters that's 3.2x reduction for the
11:46model
11:48size not only for such um uh image tasks
11:54but also for visual language tasks like
11:56caption we can still also PR it for
11:59example the first image uh given the
12:02caption original caption is a basketball
12:04player in the white uniform is playing
12:07with the ball okay it's indeed the case
12:10and if you PR away 90% of the parameters
12:12it says a basketball player in wning
12:15form is playing with a basketball it's
12:17pretty accurate second
12:19image uh the Baseline model says a brown
12:22dog is running through the grassy field
12:24versus through 90% of brown dog is
12:26running through a grassy area including
12:29the third one riding a surfboard on a
12:31wave a man in the white sweet riding
12:33wave on the beach
12:3690% what if we PR more like pruning away
12:4095% we are more
12:43aggressive so on the third image the
12:45original model Baseline model says a
12:48soccer player in red is running the
12:50field versus if you thr away 95% it says
12:54a man in the red shirt and black and
12:57white black shirt is running through
12:59field it's getting drunk right so there
13:01is a limit how much you can cannot Pro
13:04too much and we're going to later see a
13:06demo showing the process when we are
13:08doing the pruning from 70% 80% 90% all
13:12the way to 99% and we break and here we
13:15break model and I highly recommend you
13:17try this experiment offline at home
13:20we're going to give you the code for uh
13:23this ion
13:25not so in recent years PR spity is
13:29getting quite popular uh this whole
13:31domain started uh back in 1980s 1980s
13:35with the paper optimal brain
13:38damage and then throughout the years it
13:40uh gradually uh increased until 2015
13:462016 when I published this paper called
13:48Deep compression which show that these
13:50modern neuron networks in a large scale
13:53data set large scale GP 20 we can
13:56aggressively prune it there's a lot of
13:58opportunity to optimize the hard rare
14:00efficiency by looking through at the
14:02algorithm level right and then we can
14:05also design specialize accelerators like
14:07the E the efficient inference engine to
14:11accelerate directly on a sparse and
14:14compressed model in recent years the
14:17number of Publications per year just
14:19searching very fast in recent
14:22years and this has been adopted in
14:25Industry like ring in
14:27Industry left hand side other the three
14:29papers I published first two during my
14:32PhD and the last two during when I was
14:35at MIT about how to accelerate building
14:38specialized accelerators to accelerate
14:42matrix multiplication and and um
14:45different workload and N media adopted
14:48sparcity in the gpus after the a100 GPU
14:51Andia introduced a 24 sparity we're
14:54going to introduce that soon so turning
14:56a dense Matrix into a sparse Matrix and
14:59have about 2x theoretical speed up and
15:03about 1.5x measure the speed
15:06up and sigings now part of AMD also use
15:10the sparcity to um optimize their models
15:13using this AI Optimizer um um acquired
15:18previously my startup and build it into
15:20a software tool chain to Pro the model
15:22and f t the model and as a result you
15:24can get a faster
15:27inference Okay so so let's go to the
15:30next chapter about how to determine the
15:32pruning granularity okay in what pattern
15:36should we Pro the neuron
15:40Network so pruning can be performed at
15:43different granularities can be very fine
15:46grein or very CL green you can imagine
15:49there is a very big design space so
15:52turning from this very dense 2D weight
15:55Matrix we can make it
15:57a uh the right one means it's preserved
16:01and the white rectangles means it's
16:03pruned fine green pruning is most
16:07flexible and we can Pro any weights in
16:09any location but the drawback what is
16:13drawback here it's hard to accelerate
16:16right for gpus different threads prefers
16:20to be uh doing the work in L step manner
16:23everything want to be paralyzed to the
16:25same thing no branching right but
16:27there's a lot of uh irregularities which
16:30is not Hardware
16:31friendly uh what is the advantage
16:34here you have the most flexibility you
16:38can prune any weight you want and so the
16:41pruning ratio is the highest using such
16:44fine
16:46graining in contrast we can also do
16:50forse grain okay so in this case we are
16:53pruning the entire row we are pruning
16:55the third Row the fourth row um and the
16:58another row we put through three rows in
17:02the middle okay what is good about that
17:05we can condense it into a dense Matrix
17:09still apply dense Matrix Matrix
17:12multiplication to do the arithmetic okay
17:16so the good thing is about it's easy to
17:18accelerate it's very structured it's
17:20very regular you can just apply then
17:23symmetric symmetric
17:24par but you can imagine this is less
17:27flexible okay if you prune the whole row
17:30has to be prune or the whole column has
17:31to be prune and the pruning ratio is
17:34much less compared with the previous
17:37example the fine grain
17:43example okay so let's talk about the
17:45pruning granity not just for the FC
17:48layer but also for the convolution layer
17:50okay in the convolution layer there are
17:53four dimensions for the convolution
17:56kernal CI is the number of input
17:59channels Co is number of output channels
18:02and we have KH and KW for the kernels
18:05height and kernel width and these four
18:08dimensions give us more choices more
18:11degree of freedom to select the pruning
18:16granularity so we start with the um fine
18:20grain
18:22pruning in this notation we have K and
18:25KW equal to three it's a 3X3 Kern
18:29and here we have three output
18:32channels and here we have two input
18:34channels so we read it right the four
18:36dimensional convolution kernel and
18:39visualize that in this way so this is
18:42the fine green pruning atom based
18:45pruning Vector pruning kernel level
18:47pruning all the way to channel pruning
18:49this is the whole landscape and now
18:52let's dive deeper into each of them and
18:55talk about what is good about it what is
18:57bad about it what is the treal
18:58everything is about fit
19:01off so fine grain pry okay it's
19:04irregular but it's most flexible you you
19:07can have the highest pruning ratio if
19:09you're just targeting compressing the
19:11weight without worrying about
19:13acceleration or paradism this is the way
19:16to go okay um it's very has very
19:19flexible pruning indices and you already
19:23have larger compression ratio since we
19:25can very flexibly find those redundant
19:30weights for example here we can compress
19:33it by up to like an order of magnitude
19:35for these different neur
19:39networks it can also deliver speed up on
19:41Specialized Hardware if you have a um
19:44the capability to design specialized
19:46Hardware like eie the efficient
19:48inference engine which I published in 20
19:50iscar
19:522016 um you can do that but it's not
19:55easily accelerated on off the shelf
19:58Hardware
20:00a second category the PN based okay so
20:05the pruning prun Kel has some ATS okay
20:09like this is one pattern rotated by 90
20:12degrees same pattern these patterns are
20:15the same okay so this is actually give
20:18you uh more um regularity compared with
20:23the fine
20:26grainy one notable pattern is the N2 M
20:30sparity for example 2 to four sparity so
20:35start with a dense Matrix on the left
20:37hand side we can prun it to a toal four
20:40sparse Matrix so give
20:43was one minute to take a look at the
20:46pattern anyone can tell me what is the
20:48pattern here on the right hand side the
20:52Matrix
20:59so what pattern do we see on the second
21:022 four sparse
21:10Matrix right every row has four missing
21:13values right right and actually if you
21:15look deeper um like the first row four
21:19missing values and actually uh this is
21:21due to every four group of four elements
21:24you must have at least two uh zeros okay
21:28so two to four means out of four entries
21:32at least two entries has to be zero and
21:35that is the case for every group group
21:38of four elements and that's why we call
21:41it two to
21:43four so it's 50% Spar cting and how many
21:48bits do you need to indicate the
21:51location of the N zero or the zero
21:54element you have four Pointes therefore
21:57you need two bits to
21:59indicate where the non Zer where the
22:01zero
22:02are okay so that is the metadata the
22:05overhead you have to store every you
22:08have to store two bits for the
22:13index so it it really well maintain the
22:15accuracy so this is the test of accuracy
22:18across different benchmarks like resent
22:2050 exception bird Etc comparing the uh
22:25dense uh accuracy versus the sparse
22:28accuracy you can see it's pretty much
22:30the same 76.1
22:3276.2 the accuracy is very well pretty
22:36well maintained using this 2 to four
22:38roughly 50% sparity
22:42ratio okay so we can also do this
22:46channel level sh okay we omitted this
22:50middle two figures for um it's following
22:55the S similar principle we are just
22:57getting more and more regular but less
23:00and less degree of Freedom uh the
23:03extreme case is the channel pring where
23:06we are pruning away the entire Channel
23:09okay um so the pro is that we can
23:13directly speed it up due to the reduced
23:16number of channels leading to a neuron
23:18network with a smaller number of Channel
23:21and it's still Dan you don't need any
23:23specialized Hardware just using CPUs
23:26using whatever Hardware you originally
23:27have you can directly accelerate it but
23:31the car is you can have a smaller
23:34compression ratio for example um for
23:39convolution neuron Nets like mov net you
23:42can Pro away only about 30% of the
23:45parameters of mobile
23:47net so here we are showing the neuron
23:50net gr with five layers and we can prune
23:53the channels with different sparity
23:55ratio across different layer
24:00and there are two ways one is to do
24:02uniform shrinking so for all the layers
24:04you apply exactly the same sparcity in
24:07this case 30%
24:10sparcity but that is really not as good
24:13as having a smarter way to figure out
24:16the redundancy and sparcity for each
24:19layer individually like on the right
24:21hand side um we'll later discuss how to
24:24find the optimal sparity ratio to give
24:27ioc to different layers we're going to
24:30talk about sensitivity analysis in the
24:33next
24:34lecture this also applies to recent
24:36large language models where a convention
24:39in the to is that all the layers are
24:42repeating the same Transformer building
24:43block exactly the same number of
24:46channels across different layers um it's
24:49it's very homogeneous it's very easy to
24:52partition especially for model level um
24:55model paradism distribute the weight
24:57across mod for gpus uh but if you want
25:00to extract the uh the inference
25:02efficiency to the extreme different
25:04layer indeed may have different
25:07redundancy in spity ratio and we don't
25:10necessarily have to keep the same am
25:13amount of channel number across
25:15different layers Ur sensitivity analysis
25:18help to analyze that so we are going to
25:20talk about in lecture two of
25:24pring and this phenomenum is further
25:27demonstrated on this figure
25:29comparing the uniform scating for all
25:31the layers just uniformly shrink it by
25:34the same percentage PR away the same
25:36percentage for all the layers versus
25:40using an optimal um um a better
25:44optimized U Spar C ratio search
25:47algorithm here is AMC automatic model
25:50compression to search that and here is
25:53the latency versus the accuracy trade
25:55off the search approach have a lower
25:59latency and higher accuracy in this
26:03case so the student did this work was
26:05the TA for our class last year his name
26:08is G he's join open ey after graduation
26:11this is his
26:12work okay all right so let's jump into
26:16the next part how to determine the
26:19Bruning prer there are so many ways in
26:23neural network so which one do we keep
26:25which one do we PR away what is the
26:27criteria
26:28for the weights for the synapsis and for
26:30the neurons so we are going to talk
26:33about this pruning criteria okay of
26:37course we want to reduce though we want
26:39to P away those less important
26:42parameters so that we can maintain the
26:45accuracy or minimize the loss uh for
26:48example in this case we have only three
26:51waves um 10 x0 - 8 X1 plus 0.1 X2 I
26:58really want to show very simple example
27:01for um for intuition right give you some
27:04intuition which ones to select to Pro if
27:08out of these three weights out of these
27:10three weights 10 minus 8
27:120.1 if one weight has to be removed you
27:15have the capacity to hold only two
27:17parameters which one should you remove
27:21intuitively 0.1 right because it's
27:24smallest likely to have the smallest
27:26impact
27:28so that is actually the most simple well
27:32the most very effective way to
27:35determining um the pring criteria just
27:38select the small on super super easy so
27:41the importance we just use the um
27:45magnitude of the weight to indicate the
27:47importance of the weight if the
27:49importance is small the magnitude is
27:51small then we just remove it way also
27:54called magnitude based pruning so we
27:57want to maintain the weights with large
28:00absolute value and PR the weights with
28:03very small absolute value okay that is
28:06very simple theistic but turn turned out
28:09to be working super well uh in both
28:11Academia and Industry for so many
28:14years in the example on on the bottom we
28:17have four weights we find out our one
28:19Norm for each of them and this is our
28:22one norm and we keep the red the largest
28:25and remove the smallest so this becomes
28:27the pr weights so just use the uh
28:30absolute value to determine the pruning
28:34uh
28:35criteria okay what about for uh four
28:40screen pring for example if you want to
28:42Pro away the whole role of this
28:46Matrix uh we can apply L1 Norm or L2
28:50Norm for example here we start with L1
28:52Norm magnitude based ofon so we find the
28:56L1 Norm for the first row which is 3 + 2
29:00which is five we also find the our one
29:03Norm of the second row which is six okay
29:06and here we compare five is smaller than
29:08six so we are going to thr away five so
29:11our one Norm I only leave the second row
29:15unpr similarly we can also apply our two
29:18Norm so this is the way to calculate our
29:21two Norm um it's very simple I won't
29:24repeat it here um that is the Lar
29:28um characteristic just use the magnitude
29:31no matter if it is tensor or if it is a
29:34um um just a single value okay and in
29:38general we can use the lp Norm U to
29:40determine the uh Runing
29:45criteria we can also apply this scaling
29:48based pruning technique for example here
29:51we have N filters from Filter zero
29:53filter one all the way to filter n minus
29:56one and we apply scating Factor as to
29:59associate each future with a skating
30:01Factor okay so here is the skating
30:05Factor associated with each Channel like
30:08the first channel will be multiplied
30:10with 1.17 second channel will be
30:12multiplied with
30:140.1 and that scaling factor is learnable
30:18okay so that's learnable you have one
30:20learnable parameter for the whole
30:22channel so it's actually very parameter
30:24efficient you only have un numbers to
30:27learn right here
30:30and you want to minimize uh the scaling
30:32factor to try to push them uh to to zero
30:36so that we can easily PL them away like
30:38here if the scaling factor is 0.1
30:40compared with 1.17 this is pretty small
30:43so likely we are going to remove uh the
30:46second filter filter one okay so we can
30:49later sort um the scating factor and pro
30:54away the channels with a very small SC
31:00Factor so originally this was the number
31:04filters and removed uh the channels with
31:08small scaling factor and the the neuron
31:12Network becomes something on the right
31:14hand side the filters and output
31:16Channels with small scaling Factor will
31:19be approved that's
31:30of course there are many other
31:32characteristics we're also going to
31:33cover very soon so this is the first
31:36theistic very simple but there are more
31:39complicated ones let's talk about
31:44that and to continue talk about the
31:47scaling Factor right here uh the scaling
31:50Factor can be reused from the batch
31:52normalization layer which we learned
31:54from the previous lecture uh for the
31:56batch normalization you have one scaling
31:58Factor per Channel okay you have one
32:01scaling Factor per Channel that's
32:02exactly the scaling factor that is here
32:05so you can reuse the same scaling factor
32:08from the batch normalization L to
32:10simplify the
32:12calculation what other theistic we have
32:15magnitude may not be the best one right
32:19um it's hard to tell us is hard to tell
32:21which is the best heris depending on the
32:24data set depending on the neural network
32:27is experiment Al stuff but here let me
32:30introduce the design space so when you
32:32are doing such pring tasks in the future
32:35you at least know what is the the way to
32:37think about it what are the choices
32:40there's never a conclusion which one is
32:42the best and I still try
32:44it second order based approving okay so
32:48let's apply the tailor expansion to
32:50approv the network this is the pr
32:53Network versus the original Network um
32:56PR the network you basically give a
32:58preservation to the Del the W to the
33:00weights or the Delta W which is equal to
33:04um this is the first order second order
33:09and third order approximation okay so we
33:12can remove the third order and
33:15Beyond and only keep the uh first and
33:19second order the paper optimal brain
33:22damage suggest that um the last term is
33:26we can neglect them and then the sing
33:30has converged so the gradient should be
33:32very close to zero you already converged
33:34at a local Minima so first order you can
33:37neglect them so only the second order
33:40term is
33:43there a second order term has two parts
33:47the error caus by deleting each
33:49parameter assumed to be independent so
33:52the cross term can also be neglected and
33:55as a result we only have one term in
33:58middle um which is the second order
34:01term and we use that to determine
34:05whether this weight is important or
34:09not okay so the
34:11importance summarize this way where uh
34:15the H is the H
34:19Matrix but the down side is that the hro
34:22Matrix is difficult to compute so we
34:24have to apply some approximation to
34:27compute to estimate the H
34:32Matrix okay so beyond the synapsis we
34:35can also prove the
34:37neurons when we remove removing the
34:40neurons and removing the synapsis they
34:44are very um highly related so removing
34:48the neuron is equal to actually a very
34:51CL weight to for example in a linear
34:56layer right here removing one neuron
34:59right here means we are removing all the
35:02weights associated with this output
35:04neuron meaning that we are reducing one
35:06row in the weight
35:08Matrix similarly in the convolution
35:11layer when we are removing some of the
35:14output channels it also means we are
35:17removing the entire
35:18kernel corresponding to that
35:25channel okay so uh let's talk about one
35:28way to determine uh which activation to
35:32remove using the percentage of zero
35:35percentage of zero based through me okay
35:38since Ru will generate a lot of zeros
35:42okay so um here we have a batch of two
35:45this is batch one this is batch two and
35:48we have three channels and each channel
35:50is 4x4 so there are two uh two feature
35:53map
35:54batches and we just um calculate the uh
35:59average percentage of zeros in these two
36:02batches so rather than using complicated
36:05math let's just see that from a a simple
36:09example okay so there are two badges um
36:13what is the average number percentage of
36:16zero of Channel Channel Zero channel one
36:19channel three for channel one here we
36:21have five n zeros five zeros okay and in
36:26the second image we have 1 2 3 4 5 six
36:29we have six uh zeros and how many
36:32elements do we have in total four by
36:34four we have patch of two so 2 * 4 is 4
36:3732 so this is the average percentage of
36:41zero for Channel Zero okay and similarly
36:44we can calculate that for channel two
36:47well five zeros here seven zeros here so
36:50average percentage of Z is 12 ID 32 and
36:54similarly we can calculate for channel
36:56two
36:58and we compare them and we are going to
37:01remove the Channel with the most the
37:04largest amount of average percentage of
37:07zeros since they are supposed to be
37:10redundant and this is done by rather
37:13than using a static way for measuring
37:15the weight and when we are ping the
37:17weight you don't have to run any input
37:19you just statically look at the weight
37:22but for the for the activations right
37:26you have to really look at
37:28you have to really run a few
37:30samples so that you
37:32can so that you can um calculate the
37:36average percentage of zeros okay so in
37:39this case we run a batch size of two and
37:42then we calculate this the6 of these two
37:45samples so that we can calculate average
37:47percentage of
37:50zeros all right let's take a short break
37:53before jump into the next
37:56method all right welcome back let's
37:59resume the lecture so just now we talk
38:02about activation plan right so a common
38:06question here is the tensor here is no
38:09longer the weight okay this is the
38:11activation tensor those six matrixes
38:14those are the activations not the weight
38:16that's um um if I didn't explain that
38:19clearly before now is the time to
38:21clarify these are the activations and we
38:23are trying to prove the activation
38:26Channel and we run through the network
38:29with two input examples therefore we can
38:32get two categories okay so this is the
38:35first uh this is the first
38:38batch this is the second batch we run
38:41two samples both are activations and we
38:44collect the statistics about these
38:46activations to perform activation
38:49pruning all right so
38:53next regression based pruning so what is
38:57regression based cling usually um if you
39:00run the network end to end to calculate
39:03the loss that's could be pretty
39:06expensive for example if you want to Pro
39:08Lama and you want to use the last loss
39:12as the end loss as your supervision when
39:15we are doing the rining that could be
39:17super expensive um as opposed to that we
39:20can um minimize the Reconstruction error
39:24of the corresponding layer okay to layer
39:27wise
39:27layerwise reconstruction so you only
39:30need to um work on only one matrix
39:34multiplication to minimize the change of
39:36that Matrix
39:38modifcation for example right here we
39:40have a activation times the weight um
39:44the activation has batch size of two and
39:46four input channels the weight has four
39:50input channels and eight output channels
39:52and we get an output answer of batch
39:55size times Co okay
39:58and then we we try to prove some of the
40:00channels on the weight for example here
40:03we try to Pro the second channel on the
40:06weight okay so what is modified with the
40:08second Channel actually that's the
40:10second channel of the activation the
40:13modifcation uh they will correspond to
40:15each other so what is the dimension of
40:18the output after doing such after doing
40:21such um channel
40:23throughing it's the same it's still F
40:26size by Co okay because we are putting
40:30in the CI Dimension so the B size and Co
40:33those two dimensions are not impacted so
40:38um although it's just a matrix
40:39multiplication but if you want to see
40:41the pry of large language model
40:42basically everything boils to matrix
40:44multiplication so this can be quite
40:46cral and now we try to minimize um the
40:50error between the unpruned output tenser
40:54versus the pruned output tenser okay um
40:58that makes the optimization local and a
41:01lot easier compared to you back
41:03propagate across entire all the layers
41:08and supervise it in that
41:10way so how do we do proving here um we
41:15can view this matrix multiplication into
41:18four parts four is the CI Dimension okay
41:21so c i Dimension CI has four is four so
41:25four input channels
41:27and we color them in four different
41:29colors okay so X x0 multiply with w0 X1
41:35minus X1 will multiply with W1 X2 will
41:40multiply with W2 okay so that's the
41:42corresponding activation and
41:44corresponding uh weight channel so we
41:48can view them as the outer product okay
41:51the outter product of XC times WC and
41:57and sum sum them together okay in this
41:59case we have four terms okay one two
42:03three four four terms to sum together
42:06alter
42:08product and then we apply a scaling
42:10factor to each outer product beta C
42:13which is the speeding factor
42:16for each auor product we sum them up we
42:21try to minimize the difference between
42:24the original Z and the the pruned Z Z
42:27such that subject to we want to minimize
42:31the zero Norm of beta to have to to make
42:35beta as close to zero as possible if
42:38beta is zero what does it
42:40mean it means this Al product doesn't
42:44exist this uto product doesn't exist
42:47like this channel okay so it means um
42:50the channel is approved so how many
42:53betas do we have in this example
42:58we have four right four colors we have
43:01four betas if we PR in this way means
43:05beta the first beta so beta beta 1 beta
43:090 beta 1 beta 3 two beta
43:123 means beta beta one is zero okay
43:15corresponding to the white region that
43:19is
43:20pro and how do we solve that problem so
43:23we can first fix W and then solve beta
43:26to select the pr the channel and then we
43:29can fix beta to solve for w to minimize
43:33the rec reconstruction error so we can
43:36do this way iteratively okay we first a
43:38fix W to find among these four beta this
43:43four other products which other product
43:45if we remove them will have the minimum
43:48impact on the
43:51output and after selecting that that
43:53data for example after selecting the
43:55second channel to be pred
43:57we can solve W to minimize the
44:00Reconstruction error and we can do such
44:02process
44:04iteratively now this could be quite
44:06helpful for pruning uh L language models
44:10where it's super expensive to back
44:12propagate all the way uh to the very end
44:15and back properly to the very
44:18beginning all right so far we talk about
44:21what is pruning primarities of pruning
44:24criteria to select the weights to prune
44:28and we are going to show a demo to
44:31strengthen our um understanding of try
44:34so let's now switch gear to the ring
44:41demo all right um this is The Notebook I
44:49prepared let's maximize The Notebook
44:54make the font size a bit larger so that
44:57can see see
45:03it okay so in this s we prepared a amist
45:08data set uh to Pro the network to
45:13classify the H written digits from zero
45:16to n okay so if you random guess the
45:19accuracy should be
45:2010% um so let's run the my python
45:24notebook start from the beginning let's
45:26first to the
45:30setup this is using Google collab the
45:34same infrastructure we are going to use
45:37for our
45:38homeworks so for the free version I
45:41would say you can have uh this P4 GPU
45:46which is a not super Advanced GPU but it
45:50should be enough for learning purpose to
45:54be enough for this
45:55lecture we run the
45:58setup and we prepare the ne Network
46:02model and then let's
46:06visualize the images now we have 10
46:09digits to
46:11classify and then we are going to
46:13retrain the neural network on the
46:16administ data
46:18set this training has started remotely
46:22Google Cloud this is now running on the
46:25t4g viu
46:30okay play about one I has completed the
46:36accuracy already is already
46:3998% right
46:42here so we are going to repeat the
46:45training process training for five iPods
46:49this is iPod
46:50one actually I 2 already finished
46:5498.6% accuracy
46:59you can of course upgrade to a better
47:02GPU by using the PA version um but I
47:05think the threee version is also enough
47:09for our Labs actually from Lab One to
47:12lab four we are all going to use Google
47:14cab probably lab four will be a bit
47:16slower since we are going to run large
47:18language model and lab five will be uh
47:22written in C++ so make sure you learn
47:25have those uh PR requisites to um
47:29manipulate the pointers deal with the
47:31body
47:34threading okay the last iPod you know
47:37already 90 9
47:4498.9% the last ioc has finished we 98.9
47:49n% which is actually pretty
47:54satisfying okay that's first EV valuate
47:57the accuracy and mod size of this St
47:59model
48:01before
48:0398.9% almost 99% accuracy and this is
48:07the digit the second row is the
48:08prediction we can see in this batch of
48:12examples we are all
48:15correct so then let's do the pruning
48:18apply a pruning
48:20ratio let's start with some moderate p
48:23ratio let's maybe start with uh s
48:28% I SC away um 70% of the parameters and
48:33see what
48:36happens the accuracy dropped as expected
48:40right the accuracy dropped from 99% to
48:4594.6% okay and if you see the
48:48visualization some of the mark uh
48:52classified not correctly from8 to 9 is
48:56is a
48:57mistake so what can we do about it let's
49:00find tune the pr model to get a higher
49:05accuracy and we find tune for two IO in
49:10this case here's the entire code to do
49:14that learning rate the momentum the
49:16weight
49:17Decay let's just find human the model to
49:19see if we can recover the accuracy from
49:2394% back to 99%
49:27one IO already finished the increase to
49:3198.78% which is a good starting
49:34point let's fine tune for another
49:39IPO the accuracy recovered to uh
49:4498.85%
49:46already quite close to
49:4998.9% and we already P away 70% of the
49:53parameters only with 30% of the
49:56parameters
49:58left okay so what shall we do um let's
50:02visualize
50:05them let's load the
50:08model and for these 10 examples they
50:12actually all
50:16correct I showing that by pruning and
50:20retraining the remaining models we can
50:22pretty much recover the accuracy loss
50:24from 94% back to 98% tax so let's do
50:28something more aggressive rather than
50:30pulling away uh 70% let's do
50:4090% unfortunately the model accuracy
50:43dropped to only 20%
50:4620% this as expected right because we
50:49are already proving with 90% of the plan
50:52very natural accuracy will drop a lot
50:55for example this four that get
50:56classified as three let's see how this
51:00fine tuning can help recovery the
51:02accuracy so let's fine tune the pro the
51:04model to get a higher
51:11accuracy let's find Cate using the same
51:15uh schedule for two IO and of course
51:18feel free to adjust the learning rate um
51:21and also the momentum and also the way
51:24Decay or or increasing the number of IPO
51:27to see if we can have a um better result
51:30in
51:34practice okay to IO finish
51:3998.3% isn't that amazing we P away 90%
51:42of parameters after retraining we can
51:44still get 98% of
51:46accuracy
51:48so let's load the model again classify
51:52the 10
51:53digits actually all these cases are
51:56correct
51:57and the test accuracy is
52:0298.35% how about let's do something more
52:05aggressive some someone can tell me a
52:08number you want to
52:09try 99% okay okay 99% only 1% of the
52:14primers is left it's it's a very
52:20aggressive all right 10% it's random gu
52:24right you have 10 digits you have 10%
52:27accuracy so that's basically random
52:32guess everything is full yeah some
52:34already recognized everything is
52:36predicted as two after Runing away 99%
52:39of the CR DPS so let's find tune that
52:43for IPO two IPO and see what happens and
52:47we increase it a little bit or the first
52:50IPO give you 9% of accuracy back
53:06okay second one finished
53:1033% so let's run the demo
53:19again okay like this one is correctly
53:22predicted and unfortunately the
53:24remaining one is not correctly predicted
53:26right so that's the where uh was pretty
53:29much the the limit 99% will not work you
53:33can see we exactly played the entire
53:36curve from 70% 90% 99% even with r cing
53:42we can recover the accuracy up to uh 90%
53:46but if you prune too aggressively like
53:4899% the accuracy is going to drop very
53:52aggressive and even cannot be recovered
53:54from your tring
53:56but every the technology is improving
53:59very rapidly maybe some of you come come
54:01up with the new algorithms to push the
54:04frontier even more aggressive fing ratio
54:06and even higher accuracy so we have a
54:08lab for lab one we are going to release
54:13in on next Tuesday so you can play with
54:16the um such pruning and retraining
54:19process to figure out a good training
54:22schedule question
54:34oh the question is what is the
54:36difference between the training initial
54:39training versus the fine tuning okay so
54:41the fine tuning we train for two
54:43IO and learning rate 0.1 momentum 0.9
54:47and weight Decay e minus 4 and we go to
54:51see the original uh training process the
54:54learning rate is larger okay the
54:56learning rate is one but in the printing
54:59process we reduce it by 10x reduce it
55:02since it's pretty much converged so we
55:04reduce learning rate and initial
55:07training has five IO and the fine tuning
55:09took only two IPO yeah
55:17right question
55:26yeah so training a smaller model from
55:28scratch even TR for a longer
55:31time is you practically practically
55:34worse than pruning a larger model so
55:37during the optimization you redundancy
55:39is helpful for you to get away from the
55:42local minimum for example you have a
55:45sadle point and then you add another
55:47dimension um if you get a stock in local
55:50minimum if you add another dimension if
55:52it is SLE structure you can go even even
55:56lower um so over parameter over
55:59parameterization helps with optimization
56:02and after the optimization you can Pro
56:04away them
56:10away
56:20question when removing a row at the
56:23column from The Matrix
56:27can
56:37you uh so those are athal techniques SD
56:41versus pruning um you can also Pro on
56:45top of a
56:47asvd uh
56:49Matrix so pruning um asbd low rank
56:54approximation quantization ination um
56:58there are different AAL techniques to
57:01apply
57:06here all right so in the next lecture
57:09we're going to cover a few um techniques
57:12in next Tuesday from how to find the
57:14pruning ratio for each layer we find
57:16that super crucial compared with uniform
57:19pruning ratio and how to try and find
57:21human PR layer and automated ways to
57:24find the pr ratios right than manually
57:26find them and also how to the system and
57:29Hardware support for different
57:31granularities together with lab one
57:33we'll be out by next Tuesday actually we
57:36already have all the labs available
57:37online so if you're eager to try what we
57:40have tested today feel free to grab lab
57:43one from our course website which is
57:45efficient
57:47ml. and here are the reference for today
57:51which conclude today's lecture thank you

About this transcript

This page contains the full transcript of EfficientML.ai Lecture 3 - Pruning and Sparsity Part I (MIT 6.5940, Fall 2024) by MIT HAN Lab, generated from the public captions YouTube serves with the video. The transcript has 7,658 words across 1,151 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.