YT YouTube2Text

YouTube2Text → Transcripts → Inteligência Artificial DESTRUINDO no Dino Chrome 2.0

Inteligência Artificial DESTRUINDO no Dino Chrome 2.0 — Transcript

by Universo Programado · 7,156 words · 1,196 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Attention, this video is absolutely
0:03stunning, so to fully enjoy it, watch it
0:06on a television.
0:10In 2019 I published this video. In it,
0:13we saw a neural network learning to play
0:15the famous Google Chrome dinosaur game.
0:17Throughout the video, it became clear
0:19that the original version of the game
0:20was too easy. The network learned to
0:22play without any difficulty.
0:24So, to make it more difficult, I added a
0:26new obstacle that could only be jumped
0:28over with a specific action.
0:30But that didn't help much either. The
0:33network managed to master the game
0:35anyway.
0:36Then, recently rewatching this video, I
0:38realized how little we've taken
0:40advantage of this game.
0:42The fact that it's simple leaves a lot
0:44of room for us to add interesting
0:46mechanics. So in today's video, we're
0:49going to do exactly that.
0:51We're going to revisit this game and
0:53enhance it even more.
0:55We're going to improve its aesthetics
0:57and add several other obstacles so that
0:59the game is now really difficult.
1:02Our goal here will be to find out how
1:04long the neural network can keep
1:06learning to play.
1:08And here's a spoiler: it was much more
1:10than I expected.
1:18So, to begin, let's start with the game
1:20version from the previous video and make
1:22some modifications. First, let's
1:24reorganize the screen layout.
1:27We'll move all the information to the
1:29bottom to free up space at the top
1:31because we need that upper area to be
1:33clear.
1:35On the left, there's some training
1:37information and a graph showing the
1:38learning progress.
1:40On the right, there's some information
1:42about the best individual and a small
1:44screen showing them playing in real time
1:46.
1:47In this graphical interface, we also
1:49have a button that zooms in on the
1:50neural network and shows the weights of
1:52the connections individually.
1:55Second, let's remove the infinite speed
1:57from the scenario and remove the little
1:59plane with spikes because they will now
2:01interfere with the new ideas we're going
2:03to implement.
2:05Now the speed starts at five and
2:07increases until it reaches 10. When it
2:09reaches 10, it will start to decrease
2:11until it returns to five.
2:13It will oscillate in this way infinitely
2:16. Before, the dinosaurs 'score was per
2:19pixel traveled.
2:21Now it will be per obstacle jumped, just
2:23so we have a more intuitive metric of
2:24how good the individual really is.
2:27Okay, let's put it to training just to
2:29see if everything is working. After
2:31about eight minutes of training, there
2:33it is, mastered the game as always.
2:36The best individual jumped over 15,000
2:37obstacles before the match was
2:39interrupted because it lasted too long.
2:41Perfect. The next step is to improve the
2:44game's visuals.
2:46Since they're in the desert, let's make
2:48the ground sand-colored and the cacti
2:50green.
2:52The mountains will be in shades of brown
2:54to match the desert style, and a blue
2:55sky with white clouds just to complement
2:57it.
2:59Now, so the scenery doesn't look too
3:01static, let's add a sun that gradually
3:03sets. When it reaches the horizon, the
3:06sky will turn orange and then it will be
3:08night. And of course, if we look behind
3:11the scenes of the game, it's not so
3:13pretty.
3:15The sun is making a kind of crazy circle
3:16, there are mountains, but no ground,
3:18and the clouds are teleported to the
3:19other side.
3:21Games always trick the player. Okay,
3:23since I haven't changed any of the game
3:25mechanics, everything should still be
3:27working. Let's practice here quickly,
3:29and there it is, they're still mastering
3:31the game. Perfect. Let's also improve
3:33the enlarged graph and the enlarged
3:35neural network.
3:37Now their background will be transparent
3:39, but with reduced lighting to give that
3:41sense of integration, but without losing
3:44legibility.
3:45Perfect. Now, let's make the game harder
3:47.
3:48First, let's slightly reduce the
3:50distance between the obstacles and set
3:52their size to random within a range.
3:55When the obstacle is generated, we'll
3:58multiply its size by a random number
4:00between 0.75 and 1.25.
4:02This allows its size to be reduced or
4:05enlarged by up to 25%.
4:08This randomness, in addition to making
4:10the game harder, also generalizes
4:12learning, because now the network will
4:13be exposed to a larger set of sizes.
4:17And this set will make it harder for it
4:19to simply memorize what to do for each
4:21specific obstacle.
4:23After about 18 minutes, they don't care
4:25anymore.
4:26They've learned to dodge everything in
4:28the same way. Awesome. Now let's add two
4:31super cacti. One will be taller
4:33vertically and the other taller
4:34horizontally. And to get past them,
4:37we're going to add a new action to the
4:39dinosaur: the double jump.
4:42So the idea now is for them to learn to
4:44identify the super cacti and use the
4:46double jump to get past them.
4:48And as the name suggests, the double
4:50jump can only be used after the normal
4:52jump.
4:53It only works if the dinosaur is already
4:55in its first jump.
4:56Let's put it to the test and see what
4:58happens.
5:04After about 15 minutes, there it is. We
5:07have one good thing and one bad thing.
5:09The good thing is that they managed to
5:10learn. Easy. The best individual has
5:12already jumped over 4,500 obstacles.
5:15And the bad thing is that they learned
5:16to use the double jump all the time,
5:17even when they don't need to.
5:19Since there's no limit or drawback,
5:21they'll take advantage of it. This is
5:24the famous alignment problem. The
5:27algorithm achieves the objective you
5:28asked for, but not in the way you wanted
5:30.
5:31This happens because there's a tendency
5:33for the algorithm to follow the simplest
5:35path.
5:36And this tendency exists because the
5:38simplest paths are the most abundant.
5:41Among all the possible behaviors for
5:43this neural network, the simplest ones
5:46are more numerous.
5:48So, thinking in terms of probability,
5:50the chance of the algorithm converging
5:52on one of them is higher.
5:53Okay.
5:55Since it exploited a flaw in our design
5:56to win more easily, let's do the
5:58following.
5:59Let's add energy dynamics.
6:03Every time it uses the double jump, it
6:05will expend some energy. The energy will
6:07be limited and will regenerate slowly.
6:10This will prevent it from using the
6:12double jump unnecessarily all the time.
6:15Otherwise, when it reaches the
6:16supercactus, it will be out of energy to
6:18jump and will spin. They will start with
6:21100 energy and the double jump will cost
6:2420. But here a small problem arises.
6:27Since the obstacles in the scenario are
6:29generated randomly, several supercacti
6:31may appear in sequence.
6:33Then they will run out of energy to jump
6:35.
6:37Not because they used it incorrectly,
6:38but because the scenario didn't give
6:39them enough time to regenerate their
6:41energy.
6:43So, to get around that, every time the
6:45individual manages to jump over a
6:46supercactus, their energy will be
6:48regenerated by 20, which is precisely
6:50the cost of the double jump.
6:53This way, supercacti can come infinitely
6:55, one after the other, and the
6:56individual will never run out of energy
6:59because of them.
7:00Let's see what happens.
7:07Now the game has really gotten harder,
7:09because it took an hour and a half for
7:10them to reach a higher performance level
7:12. And this time something rather
7:14counterintuitive happened.
7:17When the bird comes very high, they use
7:19the double jump instead of going
7:20underneath, which makes no sense,
7:22because they are wasting energy
7:23unnecessarily.
7:26When I saw this, I thought, "Wait, if
7:28they are wasting energy like that, how
7:30do they manage to survive more than 800
7:32obstacles?"
7:34The energy should have been reset at the
7:36beginning of the game. Then I watched
7:38him play and noticed something very
7:40interesting. They only waste energy when
7:43they have some left over. When it gets
7:46close to 40, they stop wasting it. And
7:48that makes perfect sense.
7:51To survive, they only need to ensure
7:52they have at least 20 energy points to
7:54jump over the supercactus when it comes
7:56.
7:57If their energy is at 35, for example,
7:59and the individual uses a double jump,
8:01it drops to 15, leaving them in a
8:03dangerous situation. If the supercactus
8:05comes now, they don't have the energy to
8:07jump. But if it's at 40, it drops to 20
8:10, guaranteeing another double jump.
8:13In other words, they learned to expend
8:16energy freely, as long as it's greater
8:18than 40. The evolutionary process ended
8:20up selecting not the individuals that
8:22conserve energy to the maximum and keep
8:24it always fixed at 100, but rather the
8:26individuals that always have 20 energy
8:28saved.
8:29They don't need to have energy to spare
8:32, they need to not have energy shortages
8:34. These are the situations that make me
8:37enjoy implementing this type of project
8:39. Situations where the algorithm
8:41surprises you and does something smarter
8:43than you were expecting.
8:46And speaking of waiting, it is with
8:48great satisfaction that I tell you, it's
8:50finally over, because the Programmed
8:52Universe course has arrived.
8:58Now you're going to learn real computing
9:01. In these classes, I'll teach you
9:04everything you need to create projects
9:05like the ones on this channel. And when
9:08I say everything, I mean everything,
9:09from the fundamentals to the
9:11implementation.
9:12First, I'll teach you all the computing
9:14aspects, starting with binary and going
9:16all the way to programming logic.
9:19The classes have interactive animations
9:20that, in addition to facilitating your
9:22learning, also make everything much more
9:24interesting.
9:25This one, for example, is the class on
9:27memory.
9:28You can run this simulator on your
9:30computer to interact with the memory
9:32yourself and explore how it works
9:34however you want.
9:37After you've learned to program, I'll
9:38teach you how to use a graphics library
9:40and we'll create a game just like I do
9:42here on the channel.
9:44Then, when our game is ready, we'll move
9:46on to the machine learning stage. We'll
9:49start by understanding how algorithms
9:51work and then we'll implement them. In
9:54this module, we'll implement a neural
9:56network from scratch. And finally, we'll
9:59put it all together and have our neural
10:01network learn to play the game we
10:03created.
10:05In other words, by the end of this
10:06course, you will have built a project
10:08from start to finish, just like the one
10:09you're seeing in this video.
10:12And all these topics are taught
10:14gradually, sequentially, and from
10:16absolute zero.
10:17A true ladder of knowledge. So, it
10:20doesn't matter if you know nothing about
10:22programming or mathematics, you will be
10:24able to learn.
10:26And just to be clear, this is a computer
10:28science course with machine learning,
10:30and its objective is to give you the
10:32knowledge base you need to understand
10:34how algorithms work and thus have the
10:36autonomy to build your own projects.
10:39On the course page, there are some free
10:41lessons for you to try and see if you
10:43like them. There's also a button to open
10:46the project we're going to create
10:48directly in your browser.
10:50Then you can play around with the
10:52training parameters and see what happens
10:53.
10:54The link to the page is in the video
10:56description.
10:57Access is lifetime and the current price
10:59is a pre-launch promotional value. It
11:02will be adjusted when the course is
11:04finished.
11:06So go ahead, visit the page, try the
11:08free lessons and the project to find out
11:10if this course makes sense for you.
11:13Now, back to the video. Perfect. Since
11:16they've learned, let's make it more
11:18difficult.
11:20First, let's change the color of this
11:22gray text here, which looks really bad.
11:25Okay, improved.
11:27Now let's add two new actions to the
11:29dinosaur, walking forward and walking
11:31backward, so that it can move
11:32horizontally across the scenario.
11:35And let's see the impact of this on the
11:36game.
11:41After 1 hour and 40 minutes of training
11:43, they are already able to jump over
11:4511,000 obstacles.
11:47The first thing you can notice here is
11:49that when they are about to jump over
11:51the obstacle, they take a step forward
11:53and when they finish jumping, they go
11:54back, as if they were trying to get past
11:56the obstacle as quickly as possible and
11:58then recover the distance traveled.
12:01Also notice the speed and precision with
12:03which they return to the ground after
12:06passing the obstacle, making the most of
12:08every millimeter.
12:13The second thing is that the double jump
12:15over birds has curiously reappeared.
12:19Perhaps this behavior actually gives
12:21them an advantage, and evolution is
12:22always going in that same direction.
12:26The third thing is that now that there's
12:28horizontal movement, the individual can
12:29jump over the horizontal super cactus
12:31without using the double jump.
12:34And this new possibility has caused the
12:35population to split into two types of
12:37individuals.
12:39And the detail here is that, with the
12:41way I implemented the energy mechanic,
12:43if it passes the super cactus without
12:45using the double jump, it literally
12:46gains 20 energy, since it didn't spend
12:48anything to jump.
12:51In other words, once again the
12:52evolutionary process found a loophole in
12:55the game's design and generated
12:56individuals that take advantage of the
12:58flaw, individuals that can regenerate
13:00energy faster than normal.
13:03And last but not least, they learned to
13:05stay stuck to the left corner of the
13:06screen without moving much around the
13:08scenery.
13:10And of course, that wasn't the idea.
13:13So to fix this, we have to give them a
13:14reason to move around the map, something
13:16that makes it more advantageous to move
13:18than to stand still.
13:21And nothing is better to encourage
13:23someone to do something than money.
13:25>> We want money!
13:27We want money!
13:29>> Let's add a coin that will randomly
13:32change position during the game.
13:35When the individual touches it, they
13:37will earn some extra points. To start,
13:39let's set the coin to be worth one point
13:42, the same value as a cactus, just to
13:44see what happens.
13:51And after an hour and 20 minutes of
13:52training, they learned to play. But to
13:55our dismay, they are ignoring the coin.
13:58>> What a shame.
13:59>> This is probably happening because it
14:01gives few points.
14:03So at the beginning of the training, the
14:05individuals who focused on jumping over
14:07the cacti ended up earning many more
14:08points than those who learned to chase
14:10the coin.
14:11Then, over generations, this behavior
14:14was reinforced.
14:15So, let's increase the number of points
14:18the coin is worth to five to see what
14:20happens.
14:21After about 30 minutes of training, I
14:23noticed the graph was very stagnant and
14:25decided to take a look at what was
14:27happening.
14:29As you can see, catching the coin is now
14:31so rewarding that their behavior has
14:33evolved entirely in that direction.
14:36They learned to chase the coin perfectly
14:38, but their jumping ability left
14:39something to be desired.
14:42So I interrupted the training, reduced
14:44the number of coin points to three, and
14:45put them back to training.
14:48And this was the result. They remained
14:50fixated on chasing the coin and didn't
14:52learn to jump. The record was only 24
14:55obstacles. So I interrupted the training
14:58again and reduced it to two points. I
15:00put them back to training and when I
15:02looked at the graph, I thought it had
15:03worked, but it hadn't.
15:05They simply, ironically, went back to
15:07ignoring the coin.
15:10In other words, the problem isn't the
15:12number of coin points; it's somewhere
15:14else, and most likely it's here.
15:17Perhaps the neural network is too small
15:19.
15:21Perhaps this size isn't able to store
15:23and represent both behaviors at the same
15:26time.
15:27This means we need to increase its size
15:30. But the question now is, should we
15:32increase the width or the depth of the
15:34network?
15:35Add more neurons to the existing layer
15:38or add a new layer?
15:40If we pay attention to the network in
15:43operation, it seems that some neurons
15:45are being activated very little.
15:47This may indicate that what is lacking
15:49is not the quantity of neurons, but the
15:52quantity of layers.
15:54Perhaps the network doesn't have enough
15:56complexity to learn to jump and chase
15:58the coin at the same time.
16:01So, let's add another layer and see if
16:03that solves it.
16:10And to our delight, it worked!
16:13They are jumping and chasing the coin at
16:15the same time. At this moment, the best
16:17individual has already scored more than
16:1915,000 points. Notice how well they are
16:21spread across the map.
16:24Some individuals are giving more
16:25importance to the coin than others, and
16:27that's a great sign.
16:29This demonstrates that the diversity of
16:32the population is still high, and if we
16:34wanted, we could leave them here
16:36training for a few more hours and they
16:38would get even better.
16:40Also notice that they are not rushing
16:43towards the coin like crazy.
16:45They converge in her direction.
16:48They get closer, but also move away when
16:50necessary.
16:52It's as if they're positioning
16:54themselves in the scene in such a way
16:56that the coin is in their path, as if
16:58the coin isn't the objective of the jump
17:00but a consequence, a bonus, which makes
17:02perfect sense.
17:05And since the coin was just a reason for
17:07the individuals to move around the scene
17:08, we can consider that the objective was
17:10successfully achieved.
17:12So, of course, let's make it more
17:14difficult.
17:18First, let's add a moon, because this
17:20night is too simple.
17:22Now, let's add another type of obstacle
17:24, an obstacle that moves around the
17:26scene. Let's add a giant ball that will
17:29come towards them.
17:32But to be realistic, the ball obviously
17:33has to roll.
17:36And for it to roll correctly and not
17:38look like it's slipping on the ground,
17:40we have to figure out how much we need
17:42to increase its angle in each frame of
17:44the game.
17:45In other words, we have to find out the
17:48angular velocity. And finding that
17:50velocity is actually quite easy. Check
17:52it out.
17:54If a point on the ball's surface touches
17:56every point on the ground, then the ball
17:58is 100%sliding.
18:01If a point on the ground touches every
18:03point on the ball's surface, then the
18:06ball is 100%sliding.
18:08Now, if each point on the ball's surface
18:10touches only a single point on the
18:12ground, then it's rolling.
18:15Rolling is the middle ground between
18:17sliding and slipping.
18:19And if each point on the ball's surface
18:22only interacts with a single point on
18:24the ground, this implies that these two
18:26lengths will always be equal.
18:29And now the fun begins.
18:31>> I'm afraid we'll have to use math.
18:37>> The blue length is precisely the
18:38horizontal displacement, which is the
18:40horizontal velocity times the number of
18:42frames the ball traveled.
18:45For the red length, we can use the arc
18:47length formula.
18:50According to this formula, a segment of
18:52the circumference measures exactly pi
18:54times the radius times the angle divided
18:57by 180. If we isolate this angle from
18:59the equation, we will have that the
19:01angle is equal to the horizontal
19:02velocity times the number of frames
19:04times 180, all divided by pi times the
19:06radius.
19:09But there's a catch: this angle isn't
19:11what we're looking for. This angle is
19:14all of this.
19:15It's the total angle of the entire
19:17displacement.
19:18We need the angular velocity.
19:21We need to find out the measure of the
19:23angle of these little segments that,
19:25along the frames, were added together
19:27and resulted in the entire path.
19:31And to find this little angle is simple
19:33, we just divide the total angle by the
19:34number of frames that formed it.
19:37If the object rotated, for example, 200
19:40degrees in 10 frames, then its angular
19:42velocity was 20 degrees per frame.
19:45Now, on the left side of the equation,
19:47we have the angular velocity.
19:49But any modification we make to one side
19:51of the equation must be made to the
19:53other to maintain equality. So we divide
19:55the right side by the squares as well.
19:59Now the squares in the numerator will
20:01neutralize the squares in the
20:02denominator, the famous cancellation.
20:06And on the left side, we can replace the
20:08fraction with the name of the velocity
20:09itself.
20:10And voilà, we found the angular velocity
20:13.
20:14It will be 180 times the horizontal
20:16velocity of the ball divided by pi times
20:18the radius.
20:20And since we are the ones who define the
20:22horizontal velocity and the radius of
20:24the ball, then our work is done.
20:26We already have everything necessary to
20:28calculate the angular velocity.
20:31If we apply this relationship to the
20:33game, it's there, spinning perfectly.
20:36And to give it a special touch, let's
20:38replace this texture with the Chrome
20:39icon.
20:41There, now we're good.
20:44And to be able to jump this new obstacle
20:45, we're going to add another action to
20:47the dinosaur, which is the super jump.
20:50This jump will work like the normal jump
20:52, except for the maximum height it
20:53reaches, which will be much higher.
20:56It will cost 50 energy, because, well,
20:58it's pretty overpowered.
21:00And when the character is going up,
21:02we're going to change its sprite to this
21:04one.
21:05As soon as he's coming down, we'll put
21:07this one in.
21:09I tried some other sprites too, like
21:11these, but they didn't turn out as good
21:12.
21:14So the super jump will be like this.
21:17Now, of course, let's train.
21:25After 1 hour and 20 minutes of training
21:27, there it is.
21:31When Chrome appears, they start walking
21:34backwards, as if they want to gain time
21:36or space.
21:41Notice that they've learned again to use
21:43the double jump to get over the bird.
21:45But this time the strategy has changed a
21:47bit. Now that they need the super jump
21:50to get past Chrome, their energy can't
21:53drop below 50. In other words, they've
21:55learned the same strategy, but adapted
21:58to the new circumstances.
22:01The better they get, the harder the
22:03training becomes, because besides taking
22:06longer to lose the game, which slows
22:08down the evolutionary process, the
22:10random mutations end up spending more
22:12time worsening the individuals than
22:14improving them.
22:16That's why the graph always has more or
22:18less this format, because at the
22:20beginning of training, when the
22:21individual is still bad, any negligible
22:23little mutation can improve them.
22:26Then the aptitude increases very quickly
22:28, but as they improve, these mutations
22:30become scarce.
22:32Only a few, very specific ones, will be
22:34able to improve them even further. All
22:37the others will ruin them. The moral of
22:40the story is, the better you get, the
22:42harder it is to improve.
22:44It's easier to be good at many things
22:46than excellent at just one.
22:49As the game is now, there are many
22:51obstacles to jump over, and none that
22:53necessarily require going under.
22:56So, let's add one.
23:00Let's add a mountain with a tunnel that
23:02can only be passed through crouching,
23:04like a cave.
23:07I tried to find a cool mountain sprite
23:08on Google, but none were the way I
23:10wanted.
23:11So I went to Gemini and spent a long
23:13time generating various types of
23:15mountains. And in the end I chose this
23:17one for two reasons.
23:19First, we need the mountain to be tall
23:22enough to touch the top of the screen;
23:24otherwise, these guys will find a way to
23:26get over it.
23:28Second, the fact that it's somewhat
23:30square will make calculating the
23:32collision easier.
23:34And consequently, it will save us
23:36processing power, because in this type
23:38of project, every drop of processing
23:40power is very precious.
23:42We already have 500 characters with 500
23:45neural networks being processed in real
23:48time.
23:49The last thing we want is an obstacle
23:51with a complicated collision that will
23:53further burden the training.
23:56And to give the feeling that the
23:57dinosaurs are passing through the
23:59mountain, we're going to divide its
24:00sprite into four layers.
24:03First, we'll draw this one, which will
24:04be the background.
24:06Then, on top of the dinosaurs, we'll
24:08draw this one, which is similar to the
24:10background but without the section where
24:11they pass.
24:13This serves to give a 3D feel when they
24:15collide with the mountain, as if they
24:17were colliding in the middle of it.
24:20Then, we draw this shadow to reduce the
24:22brightness of the tunnel and create the
24:25feeling that it's dark.
24:27And lastly, we half-closed the doors to
24:29give the feeling that the dinosaurs are
24:31passing through them to enter the tunnel
24:33.
24:35Perfect. Now let's put them to the test
24:37.
24:42And after an incredible 15 minutes, they
24:45had already mastered the game. They are
24:47going under the mountain perfectly and
24:50jumping over the other obstacles
24:51normally.
24:53The curious thing is that when they
24:54enter the mountain, they start walking
24:56forward as if they want to get out as
24:57quickly as possible.
24:59And after they get out, they start going
25:01back. To tell the truth, I didn't
25:03understand the point of this behavior. I
25:05can't see an advantage in getting out of
25:08the mountain quickly.
25:10Maybe it's a randomly acquired behavior
25:12.
25:14If you see an advantage in this, tell me
25:15in the comments.
25:17Okay, this mountain was easy. It didn't
25:20add much difficulty to the game. In fact
25:22, it even seems to have made it easier,
25:24because now they learned much faster.
25:27So let's add a really difficult obstacle
25:30.
25:32Since the game already has obstacles to
25:34go over and now to go under, let's add
25:36one to go through the middle.
25:39Now things are going to get crazy.
25:42This obstacle is a bit strange to
25:44imagine, because the top part would be
25:46kind of floating, disconnected from the
25:48bottom part.
25:50The first thing I thought of was the
25:52pipes from Flappy Bird, but that would
25:54be too disconnected from the game's
25:55theme.
25:57Then I thought of a wall of birds, but
25:59that would increase the computational
26:01cost because of the collisions.
26:04So let's do this: let's put a giant tree
26:06with a passage in the middle.
26:08It's not the best solution, but it will
26:10work well.
26:12Again, I went to Google to look for a
26:14tree sprite, but this time I managed to
26:16find this one.
26:19It wasn't exactly the way I needed it,
26:21so I changed its color in Photoshop and
26:23asked Gemini to add the passage.
26:26Just like we did with the mountain, the
26:28tree will also be divided into layers.
26:32This one will be drawn in the background
26:34and this one on top of the dinosaur to
26:35give the 3D feeling that they are
26:37passing through it.
26:39Perfect. Now, to pass through the tree,
26:41we're going to add another action to the
26:43dinosaur, which will be the dash.
26:46When the individual uses the dash, it's
26:47like a quick forward sprint in a
26:49straight line.
26:51The amount of movement is fixed, and if
26:53the individual is in the air, vertical
26:55movement is interrupted and paused while
26:58the dash is happening.
27:00It will cost 30 energy because it's kind
27:02of overpowered.
27:05So the idea now is for them to learn to
27:07position themselves in front of the tree
27:09passage and use the dash to quickly
27:10cross it.
27:13But to make it even more difficult, the
27:15tree passage will be at a height that
27:17can only be reached with a double jump.
27:20In other words, they will have to learn
27:23to combine two abilities.
27:25First, they will have to use the double
27:27jump to reach the height of the passage
27:29and then use the dash to cross it.
27:32But before we put them to practice,
27:34let's change the color of this mountain
27:35because when I look at it, I can only
27:37see a mountain of poop.
27:40Okay, a little better. Let's see if they
27:43can learn this.
27:49After about 30 minutes of training, they
27:52had already mastered the game. But two
27:55problems appeared.
27:57First, they managed to get through the
27:59tree using only the double jump with
28:01ease.
28:02Look at the precision of these guys.
28:04There's no way around it, they always
28:06take the easiest path. If it's possible
28:09, it will happen.
28:10Second, they've gone back to ignoring
28:12the coins and are stuck in the corner of
28:15the screen again.
28:16To solve the tree problem, we'll
28:18increase its width and decrease the
28:20height of the passage.
28:23And if that's still not enough, we'll do
28:24it again.
28:26At some point, it will have to become
28:27impossible to pass without the dash. And
28:30to solve the coin problem, we'll use the
28:32final blow. We'll change the reward
28:35format. As it is now, the reward is only
28:38given when the individual touches the
28:40coin.
28:41If they don't touch it, they get nothing
28:43, zero.
28:44This is called a sparse reward.
28:47In this type of reward, the delivery is
28:50binary.
28:51Either we reach the correct state and
28:53get the entire reward, or we get nothing
28:55.
28:56And this isn't very good, because at the
28:58beginning of training the agent's
29:00behavior is random.
29:02And achieving rewards by performing
29:03random actions is quite improbable.
29:07This means that, in practice, the agent
29:09will spend most of its time receiving no
29:10reward at all.
29:12And if it receives no reward, its
29:14behavior doesn't improve.
29:16And if its behavior doesn't improve, it
29:18continues to miss out on rewards.
29:21If the rewards are sparse, the agent
29:23will be forced to explore the scenario
29:25blindly.
29:27It will be forced to guess actions
29:29thousands of times until it eventually
29:32gets it right and earns some reward.
29:35This makes training much slower. And
29:38depending on the complexity of the
29:40scenario, it may even be impossible.
29:43Because the probability of achieving the
29:46reward with random actions may be very,
29:49very close to zero.
29:52So we're going to fix this by
29:54transforming these sparse rewards into
29:56dense rewards.
29:58With dense rewards, it's the opposite.
30:02We'll provide the agent with small
30:04rewards all the time, so that every time
30:06it performs an action, it knows whether
30:08it helped or not.
30:11Even if it only helped a little, that's
30:13enough.
30:14By transforming the rewards into dense
30:16rewards, we move from a discrete and
30:18binary format to a continuous format.
30:22And this continuity ends up creating a
30:24path of rewards that will lead the agent
30:27to the objective.
30:29In the case of the coin, the reward will
30:31no longer be whether the individual
30:32touched it or not, but rather the
30:34distance they are from it.
30:36The closer they are, the more they earn
30:38.
30:39In this way, the evolutionary process
30:41will have the ability to distinguish
30:43which individuals are better, even if no
30:45one has picked up any coin.
30:48Just the fact that some are closer to
30:50the coin than others is enough for
30:52evolution to prioritize those
30:53individuals.
30:55And of course, in the Programmed
30:57Universe course we will also see all
30:59these techniques in detail. Okay, let's
31:02test it.
31:08After 2 hours and 40 minutes of training
31:10, there it is, nice and neat. They went
31:13back to chasing the coin and are now
31:15using the dash to get past the tree.
31:17Check out this sequence of dinosaurs
31:19using the dash, how satisfying.
31:24And since they managed to learn, we will
31:25continue.
31:27But before adding the next obstacle,
31:28let's improve some sprites.
31:30First, the Chrome image had a very
31:33innocuous aspect.
31:34So I asked Gemini to add some spikes to
31:37it to make it look more hostile. And
31:40after a few tries, it looked like this.
31:42Then, I still thought this image of the
31:44mountain was too ugly.
31:46So I took this sprite here, which was
31:48one of the first ones, changed its color
31:50to gray, and asked Gemini to put it in
31:52the shape of Sugarloaf Mountain.
31:54And there it is, that was the result.
31:57He removed the mountain doors, but
31:58that's okay, we'll put them back.
32:01Perfect.
32:02Now we're going to put that spike from
32:04the last video back, but this time there
32:05won't be a plane to get past it.
32:08This time the spike will be so long that
32:10the individual will have to learn to
32:12combine three abilities to get past it.
32:15First, they'll have to use the super
32:17jump, then the double jump, and finally
32:18the dash.
32:20Notice that now, to jump over the spike
32:22, the individuals will need to use all
32:24their energy.
32:26This will force them to keep their
32:28energy at 100 all the time.
32:30They can no longer waste energy
32:32unnecessarily. Let's see what happens.
32:41After 40 minutes, how were they?
32:44Showing off.
32:47And as expected, they're no longer using
32:49the double jump to get over the bird,
32:51which is a great sign.
32:54But they're still far from perfect. Here
32:57, for example, he wasted energy where he
32:59shouldn't have, and when it came to the
33:00spike, he lacked the energy to complete
33:02the combo.
33:03So let's keep training a little longer
33:05to see how good they can become.
33:09Oops, wait a minute, there was a small
33:10unforeseen event here. This was already
33:13in the middle of the night, and I simply
33:15fell asleep while waiting for the
33:17training to happen.
33:19In other words, they trained all night,
33:21and when I woke up, they were like this
33:24.
33:33When I woke up, the agents were
33:34incredibly sharp.
33:36After more than 8 hours of training, the
33:38best individual jumped over more than
33:405,000 obstacles without losing.
33:43Look at that exponential growth
33:54; they're trying to buy time for the
33:56spike to detach from the corner of the
33:59screen because if they jump with the
34:01spike stuck, they'll lose.
34:04So I let them train for another two
34:06hours, and the best individual reached a
34:09score of 12,000 obstacles.
34:12Look how he dodges everything, flowing
34:14through the scenery, so skillful,
34:15confident, and unpretentious.
34:18He's got talent, man, he's got talent.
34:22Then at that moment I thought, what if
34:24we remove the energy limit and let the
34:26individuals use their skills freely, let
34:28them literally spam their skills?
34:31What strategy will they learn?
34:34So I went and retrained them without the
34:36energy limit.
34:38And they learned to jump over everything
34:40with the super jump, which actually
34:42makes sense.
34:44Look how they get past the tree now on
34:45the descent with the super jump.
34:48But there's a catch.
34:50Removing the energy cost makes several
34:52behaviors that weren't possible before
34:55now possible.
34:57Therefore, now there are several winning
34:59strategies, several behaviors that can
35:02survive the obstacles.
35:04But during training, the population
35:06converges on one of them randomly.
35:09This means that if we put them to train
35:11again, the next strategy learned may be
35:14very different from the previous
35:16strategy.
35:17And that's exactly what happened.
35:20I restarted the training and now they've
35:22learned to do this weird, kind of blocky
35:24movement, using the super jump to go up
35:27and the dash to go forward all the time
35:29.
35:30When they're in the tree, they use the
35:32super jump to reach the height of the
35:33opening and then the dash to go through
35:35it.
35:37Look at this craziness, them spamming
35:38the super jump and pressing down at the
35:40same time.
35:45Awesome. Now let's go back to the energy
35:48limit and continue making the game
35:50harder.
35:51But first, one more aesthetic
35:53improvement.
35:54Let's add some smoke to the super jump
35:56and the dash to give a sense of momentum
35:59.
36:00Let's improve the tree sprite so that it
36:02connects to the ground in a more organic
36:04and natural way.
36:06I asked Gemini to improve its roots, and
36:08out of all his exaggerations, the one I
36:10liked the most was this one.
36:13Then I tried to improve the dinosaur
36:15sprites themselves. I asked Gemini to
36:17replace it with a dinosaur that looks
36:20angry. And there it is, sensational.
36:23First, a silly dinosaur, then a cute one
36:25, and finally a more realistic one.
36:28The sprites turned out incredibly well.
36:32However, in the game, the dinosaur is
36:34drawn very small, so all the details
36:36from these high-quality sprites end up
36:38blending together and looking a bit
36:40blurry.
36:41So I decided not to mess with our
36:44pixelated little dinosaur.
36:46Now let's make it harder. We're going to
36:48add a bird that comes from the sky
36:50towards the dinosaur. If the bird
36:52touches it, it's game over.
36:54And to avoid that, the dino will be
36:56equipped with a weapon. So the challenge
36:59now will be learning to aim and shoot at
37:02the bird.
37:03And the difficulty lies in the fact that
37:05the player will have to do this while
37:07jumping over obstacles. Dodge and shoot
37:10at the same time.
37:12Even though all the birds come at the
37:14same time, each dinosaur will have its
37:16own bird.
37:17One dinosaur's shot won't hit another's
37:20bird. And one dinosaur's bird won't
37:22affect another's dinosaur either. When
37:24the bird is red, it means it's in
37:26aggressive mode. It's going to attack
37:28the dinosaur and smash it. I'm
37:31>> going to break you, you crazy thing. I'm
37:32serious, I'm going to break you.
37:33>> When the bird is blue, it means its
37:36dinosaur has already gone to meet God
37:38and the bird will retreat.
37:41When the bird is gray, kind of
37:42transparent, it means the dino managed
37:44to hit it with a shot and then,
37:45unfortunately, it fell asleep without
37:46sleep.
37:49The dino can rotate the weapon clockwise
37:51or counterclockwise and the weapon will
37:52shoot in the direction it's pointed.
37:55Rotating and shooting will have to cost
37:57energy to prevent the agent from
37:59spinning and shooting wildly. Therefore
38:02, we'll have to increase the dino's
38:04energy. Let's put it at about 120 just
38:06to start.
38:08For the agent to be able to see the bird
38:10, we'll pass it the distance and angle
38:12between it and the bird.
38:14This is called polar coordinates. But
38:17there's a problem. The angle is a
38:19circular metric.
38:210º is equal to 360º.
38:241º is very similar to 359º. This will
38:28confuse the neural network.
38:30It will have to learn to treat the
38:32minimum value the same way it treats the
38:34maximum value.
38:36And this goes a little against how a
38:37neural network works.
38:39Is it possible? Yes, but it will
38:41unnecessarily hinder learning.
38:44So, instead of passing us the raw angle
38:46, we'll pass us its sine and cosine.
38:50This way, we can break down a circular
38:52quantity into two linear quantities.
38:55Now, let's go, let's train.
39:03And now things got really much harder,
39:06because they had to train for 4 hours to
39:08reach a reasonable level.
39:11In the beginning, there are some crazy
39:13ones who just shoot randomly and
39:14immediately run out of energy and lose
39:16the game.
39:17In the first attack by the birds, half
39:19the population is already down.
39:22This happens because, as we've already
39:23said, at the end of the training, the
39:25random mutations end up doing more harm
39:27than good to the individuals.
39:29We can see the evolutionary process as a
39:32search process.
39:33We are looking for a set of values,
39:36literally a list of numbers that, when
39:39inserted into the weights of the neural
39:42network connections, make them behave
39:45the way we want, in a winning way.
39:48This is the best individual so far.
39:50Check out his headshots.
39:56But even though they had learned, they
39:59went back to ignoring the coins and
40:01sticking to the wall. But now this makes
40:04a lot of sense.
40:06If they keep chasing the coin, they'll
40:08be forced to learn to shoot in a 180-
40:10degree range, because the bird can come
40:12from any direction.
40:16But if they stick to the wall, they
40:17reduce that range to 90 degrees, which
40:20is much easier.
40:22And since hitting the bird is worth more
40:24than getting the coin, they end up
40:26converging on this strategy.
40:28Obviously, this wasn't the result I
40:30expected.
40:31This behavior of sticking to the wall
40:33greatly reduced the difficulty of
40:35shooting the birds.
40:37So I started thinking about ways to
40:39solve this and came up with an
40:40interesting solution.
40:43Notice that the act of jumping over
40:45obstacles is quite independent of the
40:47act of shooting.
40:49It's as if they were two distinct tasks
40:52in the same agent. Then I thought, what
40:54if we use two neural networks? One just
40:58to avoid obstacles and the other just to
41:00shoot. Like GTA, while one drives, the
41:03other shoots.
41:05And best of all, if we separate the
41:07agent's neural network into two, we can
41:10reuse a network that we've already
41:12trained.
41:14We can reuse that neural network that
41:16already knows how to jump over obstacles
41:19very well. Then we'll only need to train
41:21the network that shoots. So that's
41:24exactly what we're going to do. Here we
41:27have two separate networks. The top one
41:29only shoots, and the bottom one jumps
41:32and chases the coin.
41:34The top network starts completely random
41:36because it's the one that will be
41:38refined during training.
41:41For the bottom network, we'll load that
41:44agent that trained for 10 hours and
41:46jumped over 12,000 obstacles.
41:48We'll put its neural network into all
41:51the individuals in this new training.
41:54Everyone will have the same neural
41:56network that jumps.
41:57Only the network that shoots will be
41:59different for each one.
42:01During training, the bottom network will
42:03remain frozen. It won't be modified in
42:06any way. And since it already knows how
42:08to chase the coin, it will move around
42:10the scenario normally.
42:13In other words, the jumping network's
42:15behavior will force the other network to
42:17learn to shoot in a 180-degree circle.
42:20And this strategy worked incredibly well
42:24.
42:25In just 5 minutes of training, the
42:27shooting neural network had already
42:29managed to take down 700 birds.
42:32Look how they're doing it.
42:34The bottom network is moving around the
42:35scene, while the top one protects from
42:37the birds.
42:38Awesome.
42:39Now there's just one little thing
42:41missing to make it perfect. The bird is
42:44taking too long to return.
42:46Let's make a new bird appear as soon as
42:49the previous one is shot down, so it
42:51keeps shooting all the time.
42:54Now that's crazy!
42:57Check them out shooting during the jump
42:58. Amazing.
43:03We started like this, now we're like
43:06this.
43:07The secret is always to start with the
43:09simple things that work and only then
43:11increase and improve them.
43:13If you start straight with the complex,
43:15errors will appear and you won't be able
43:17to find where they are.
43:20Here I realized that the video was
43:22getting gigantic and that the AI hadn't
43:23given any sign that it would be defeated
43:25.
43:26So there's no way around it.
43:28If you want this video to have a sequel
43:30so we can improve it even more until the
43:32game becomes literally impossible, let
43:34me know in the comments.
43:37While you're at it, leave me a
43:38suggestion for what mechanics would be
43:40interesting to add to the game.
43:43If you want to play this modified
43:44version of the game, it's available here
43:46in the channel's members area.
43:48Just go there and download it.
43:50And of course, the link to the course
43:52where I'll teach you everything you saw
43:53in this video is in the description.
43:56See you soon!

About this transcript

This page contains the full transcript of Inteligência Artificial DESTRUINDO no Dino Chrome 2.0 by Universo Programado, generated from the public captions YouTube serves with the video. The transcript has 7,156 words across 1,196 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.