YT YouTube2Text

YouTube2Text → Transcripts → How to make advanced image recognition bots using python

How to make advanced image recognition bots using python — Transcript

by Kian Brose · 3,112 words · 419 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Hey, so today I'll teach you how to make
0:02some bots. We'll be covering how to make
0:03a simple piano tail bot, how to make an
0:05aim booster bot that can track an
0:06object. Okay, so let's get started. Step
0:09one, download Python and remember to
0:11check the add to path checkbox when
0:13installing. Step two, open the console
0:15with administrator privileges and run
0:17the following commands. You can find
0:18them in the video description. These are
0:20all Python libraries that we are going
0:22to use throughout the tutorial. I'll
0:23explain what they do a bit more into the
0:25video. Step three. Now that we have
0:27completed this setup, create a new
0:28folder and create a text file named b.
0:30py. You can name it to whatever you want
0:32as long as it ends with the py
0:34extension. Now begins the interesting
0:36part of the tutorial. Right click your
0:37newly created Python file, edit with
0:39idle, edit with idle, and paste this
0:41into the code editor. You can find this
0:43in the description. These are the
0:45libraries we downloaded a second ago.
0:46Let's find tiles online. This one should
0:49do. For this game, we have four rows
0:51where black squares will come down from
0:53each row. The fastest and easiest way to
0:55bought this game is to check a single
0:57pixel from each row. We will check if
0:59that pixel is black, and if it is, we're
1:01going to click it. Okay, so first we
1:03need to know the RGB value of black,
1:05which is the tile's color. For this,
1:07open a new Python shell by pressing
1:08Windows search Python and open. Write
1:11import pi autogi and then pi autog gui
1:15display mouse position. The shell will
1:17display the pixel you're pointing at and
1:19its RGB value. Hover over any black part
1:22of the screen. You can see the color for
1:24black is 0 red, zero green, and zero
1:26blue. Go back to the shell and press
1:28Ctrl C to stop it from running. We now
1:31want to figure out the X aka horizontal
1:34position for each row. Open the Python
1:36shell again and type import pygi and
1:38then pygi display moss position. Hover
1:42over the first, second, third, and
1:44fourth row. Go back to the shell and
1:46copy the data from each respective row.
1:48We will only be using one Y vertical
1:50position. So I'll put mine to around
1:52400, which is around the middle of the
1:54game. For me, the first house pixel is x
1:57597. Second x676, and so on. It will be
2:01different for you, but don't worry. Open
2:03the bot.py file. There are two ways to
2:05perform a click. The first is by using
2:07pygi click function, which looks like
2:09this. The the downside for this is that
2:12it's pretty slow. So instead, we will
2:14use the wind32 API, which runs much
2:16faster. To do this, we start the script
2:19by creating a click function like so.
2:21This uses the Win32 API library to move
2:23the mouse to XY position with set cursor
2:26pause, pressing down, holding for 0.01
2:29seconds, and releasing. We want the 0.01
2:32second delay since clicks sometimes fail
2:34to execute if you click too fast. We
2:36then write while keyboard is pressed Q
2:38equal equal false. This line checks if
2:41the key Q is pressed down. It returns
2:43true if it's pressed down and false if
2:44it's not. We're going to use this to
2:46stop the bot when we want to stop it.
2:49Okay. So now what we'll do is check the
2:51RGB value of the first row's pixel. To
2:53do this, we use the if pi autogy.pixel
2:56and then your pixel's location 0 equal
2:59equals 0. This will return the RGB value
3:01of the pixel located in X 597 and Y 400.
3:06The zero checks for the R value. Whereas
3:08if you were to switch the zero for a
3:10one, it would check for the G value. And
3:11if you were to switch it for a two, it
3:13would check the B value. There's no need
3:15to check for more than one in our case.
3:16So the R value is enough. Now that the
3:19script can tell when the pixel is black,
3:21we want to click it immediately when it
3:23does. For this, we call the click
3:24function we created and make it click
3:26the first tile's position, which is the
3:28same position we scanned. We do this by
3:30writing click and then your pixel's
3:32position. All right, so repeat this
3:33process for the remaining three tiles.
3:35And don't forget to add the correct
3:36pixel location. And there you go. Press
3:38run, run module, switch back to piano
3:40tiles, and press start.
3:45So, as you can see, yay, it works.
3:49[Applause]
3:50[Music]
3:52[Applause]
3:59Now that you know how to make a piano
4:00tile spot, let's move on to the aim
4:02booster bot. Start by going to
4:03aimbooster.com. Enable flash player if
4:05it's not enabled and press reload. In
4:07this game, you have to press the circles
4:09that appear randomly in this area of the
4:11screen.
4:12There are two approaches to solving this
4:14game. The first is to take a picture of
4:16the circle and locating it on the
4:18screen. This approach would be too slow.
4:20So instead, we will take a screenshot of
4:22the game area and check every one out of
4:24five pixels to see if that pixel is the
4:27color of the center of the target.
4:28Before doing this, however, I'll
4:30demonstrate how to do image recognition.
4:32Okay, so through my amazing artistic
4:33talent, I have made what appears to be a
4:35stickman in paint. What I'll teach you
4:37now is how to detect if the stickman is
4:39on the screen or not. Go back to the
4:41folder you created with the bot.python
4:43file. Create a new file and name it
4:45stickman.py.
4:46Edit the file with idle like we did
4:48before and paste in the same libraries
4:50as in the piano tiles bot. Go back to
4:52the stick man you drew. In my case, this
4:54one. On Windows, press shift Windows S.
4:57This will open up a screenshot tool that
4:58allows you to select a part of the
5:00screen to screenshot. Drag over the
5:02stick man you drew to save the
5:03screenshot. Open a new paint file and
5:05paste the stickman. You now want to trim
5:07it as small as possible by selecting the
5:09area that contains the stick man and
5:11cutting excess like so. Once you have
5:13cut the excess, press file, save as, and
5:15save it as stickman.png in the same
5:18folder as the script. Saving it in the
5:20same location as the script is
5:21important. Open the stickman script you
5:23just created. We will now create a while
5:25loop with while one. Inside this loop,
5:29we will write the script that is
5:30actually going to find the image. If pi
5:33go.locate on screen stickman.png png is
5:36not equal none. This line uses pi autogi
5:38to locate the stickman on the screen. If
5:41it doesn't find anything, the output of
5:43this line will be none. So if it's not
5:45none, it means it found it. So if you
5:48write the following script, if it finds
5:50the image on the screen, it will print
5:51out I can see it and then wait for 0.5
5:54seconds. The delay so that the script
5:56doesn't spam the console. And if it
5:58doesn't find the image, it writes I am
5:59unable to see it. Run the script by
6:02pressing run module. And as you can see,
6:04the script can see the image. But if you
6:06move the window over the stickman, it
6:08stops working.
6:10Now, one of the questions you might have
6:11is what if the background of the image
6:13I'm trying to detect changes? For
6:15example, if I were to put this
6:17disturbance here, the script wouldn't
6:19detect the stickman since it's not an
6:21identical match. To fix this, we simply
6:23add the confidence parameter in the
6:25locate on screen like this. Let me
6:27explain. When the bot looks at the
6:29screen, it checks every pixel and gives
6:30a percentage chance probability of
6:32finding a match to the image you're
6:34asking to search for. By default, pgy
6:37only gives an output when it's 100% sure
6:39of having a match. But by using
6:41confidence 0.8, we tell it that if it's
6:4480% sure or more to have found the
6:46image, it finds it. You can, of course,
6:48adjust the value to whatever you need.
6:50But wait, there's more. Now, you might
6:51be asking yourself, how can I make the
6:53script run faster? The first and most
6:55obvious way is by checking a smaller
6:57region of the screen. So instead of
6:59searching for the image on the entire
7:01screen, we have to tell it where to
7:02look. Finding the correct region is a
7:05bit of a process, but I'll explain as
7:06best as I can. Create a new Python
7:08script in the folder. Copy the path to
7:10the folder by clicking here and copying
7:12this. Open the script and write the
7:14following code. It's in the description
7:16as well. But remember to change this
7:18path to your folder's path you copied a
7:20second ago. This script saves the image
7:22on the region 0 0 100 100 as saved
7:25image.png. The first number is the x
7:28position that the screenshot starts on.
7:30The second number is the y position that
7:32the screenshot starts on. The third
7:34number is the width and the fourth is
7:36the height. So let's say I do region 100
7:38200 500 500. The script will take a
7:41screenshot starting 100 pixels from the
7:43right, 200 pixels down, and have a width
7:45and height of 500 pixels. Now we just
7:48need to tweak the values manually. Run
7:50the script. Look at the output until we
7:52find the region of the stickman. So as
7:55you can see, running the script saves an
7:57image with the region I gave it. I found
7:59that the region for the stickman in my
8:00case is around 150, 175, 350, and 600.
8:04Keep in mind, it doesn't have to be too
8:06close to the object. It's fine to check
8:08the area around the stickman as well.
8:10Open the stickman script again and write
8:12region equal your region and then
8:14grayscale equals true. The grayscale
8:16parameter gives a speed boost of around
8:1830%, so it's always a nice thing to
8:20have. Do not use grayscale if you're
8:22getting fake results, but in our case,
8:24it's fine. Run the newly modified
8:26script, and as you can see, it can
8:28detect the stickman even though parts of
8:29it are hidden and or modified.
8:34All right, so back to the aim booster
8:35bot. Create a new Python file in the
8:37same folder as all the other files we
8:39made. Name it aimooster. py. Paste in
8:42the libraries we used in the two last
8:43examples and then rewrite the same click
8:45function we used in piano tiles.
8:47Remember to add a timesleep 2 to give
8:50you two seconds to switch over to aim
8:51booster when starting the script. So as
8:54I explained a bit earlier, we will be
8:55taking a screenshot of this region of
8:57the screen where the circles appear and
8:58then check every one out of five pixels
9:00to see if the color of that pixel
9:02matches the color of the center of the
9:03target. So first we want to figure out
9:06what the region for the game area is. To
9:08do this, open the script you created
9:10during the stickman demo that takes a
9:11screenshot of a region and saves it and
9:14tweak the region values until you get
9:15this region, which is the region where
9:17the circles are going to be appearing.
9:20Now that you know the game region, let's
9:22start the script with the while keyboard
9:24is pressed equal equal false so that we
9:26can stop the bot from moving the mouse
9:27when we want it to stop. Write the
9:29following line. This line uses pi autogy
9:32screenshot function to take a screenshot
9:33of the screen and store it in the pick
9:35variable. In our case, it takes a
9:37screenshot of the region we give it,
9:38which for me is 660, 350, 600, 400. Your
9:43region will be different, so don't mind
9:44it if it's not exactly the same. We then
9:47use the size function to get the width
9:48and height of the pick variable, like
9:50so. This will store the width in the
9:52variable width and height in the
9:54variable height. Pretty
9:55self-explanatory.
9:57Now, we want to check what the RGB value
9:59of the center of the target is. For
10:01this, I took a screenshot of the game
10:03while playing, pasted the picture in
10:05Microsoft Paint, opened the Python
10:07shell, wrote import pi autogy and pi
10:09autogy the display mouse position. I
10:12then hover over the center of the target
10:14and the console will print out the RGB
10:16value of the pixel I am pointing to. The
10:18center of the target has an RGB value of
10:20255, 219, and 195. Copy the RGB values.
10:25Go back to the script and paste it
10:26anywhere as a comment. Note to make a
10:28comment in Python, use the hashtag
10:30symbol. Okay, so since checking every
10:32single pixel in our screenshot for a
10:34match is a bit overkill, we will instead
10:36check for the first pixel, then the
10:37fifth, then the 10th, etc., etc. But
10:39first, let me explain how a for loop
10:42works. Let's take this example for x in
10:44range 0 10 1, print x. This will output
10:48every single number starting from 0 up
10:50to 9. This means that the script will
10:51set x to zero, print out x, so zero, and
10:55then increase x by 1, which is the third
10:57value here. It will do this while x is
10:59smaller than 10. The maximum value this
11:01can output is 9. If we were to run 4x in
11:04range 0 151, print x, the script will
11:08print out every number from 0 to 14. If
11:10we were to increase the third value here
11:12to two for x in range 0 152 print x, we
11:16will instead get increments of two
11:17instead of 1. So the output would be 0 2
11:204 6 8 10 12 14. So as you might have
11:25already guessed we will be using
11:26increments of five for our image. So for
11:28example let's say that our image has a
11:30width of 500 pixels and a height of 600
11:32pixels. We write for x in range 0 5005
11:36for y in range 0 65. RGB equal pick.get
11:41pixel xy.
11:44The script will save the RGB values of
11:46the pixel in position XY of our picture
11:48pick thanks to the get pixel function.
11:50So for example, the script will first
11:52check the pixel x0 y 0 then x0 y5 then
11:56x0 y 10 then x0 y15 etc etc until it
12:00hits the maximum value of y 595 since we
12:04have increments of five and y has to be
12:06smaller than 600 and store the red value
12:08in r, green value in g and a blue value
12:10in b. All right. So, your script should
12:13now look like this. Since my game has a
12:15region width of 600 and a height of 400,
12:18I will use the width variable here and
12:20the height variable here instead of
12:21hard-coded numbers. You can use
12:23hard-coded numbers if you want, but for
12:25simplicity, I will use the variables.
12:27So, now that we can successfully check
12:28the color of every fifth pixel, we want
12:30to perform a click if the color matches
12:32the center of the target's color. You
12:34can code this in two different ways. In
12:36my case, the center of the color of the
12:38target is always going to be the same.
12:40So we can add a if b which is the blue
12:42value equal equal 195. This checks if
12:45the blue value of the pixel located in
12:47xy is equal equal to the value 195 which
12:51is the blue value of the center of our
12:53target. To check for the red and green
12:54values change it to either r for red and
12:57g for green. But in this case blue is
12:59more than enough. But if you wanted to
13:02create a bot similar to this one where
13:04the center is not always going to be the
13:06same color then you can use range again.
13:08If you were to use range, the loop would
13:10look like this. If B in range 180 210,
13:14this will check if the blue value stored
13:16in B is between 180 and 210.
13:20In my case, checking for a range is
13:22unnecessary. So I will just stick for if
13:25B equal equal 195.
13:28Okay. So now that the script can tell
13:30when the color of the pixel is the color
13:32we want it to be, we want to click. For
13:34this, simply call the click x + 660 y +
13:38350.
13:39If you wonder why I have these values,
13:41remember that the click function will
13:43click a pixel based on your screen
13:45resolution. So a click 0 0 will click at
13:48the top left of the screen. But in our
13:50case, we want to perform a click
13:51relative to the screenshots region.
13:53Since our region's first pixel, aka
13:55pixel 000 of our screenshot starts at
13:58660 pixels to the right and 350 pixels
14:01down, we want the click function to also
14:03be 660 pixels right and 350 pixels down.
14:08Since the script runs a bit too fast, it
14:10might click the circles a bit too fast.
14:12So, we add a time sleep after the click
14:14to prevent misclicks. I'll put mine to
14:160.05. We then end the script with a
14:18break to stop the script from clicking
14:20the same place again.
14:22Save the script. Run the script. Tab
14:24over to aim booster and press challenge.
14:27And voila, there you have it. A working
14:29aim booster bot. Remember that you have
14:30to hold Q to stop the bot. If you
14:33experience accuracy issues, just
14:34slightly tweak the delay after the click
14:36so the bot doesn't have the time to
14:38click twice. So before ending the video,
14:41I just wanted to ask if the fast pace of
14:43this video was okay or if it was way too
14:45hard to follow and if you want a more
14:46detailed in-depth video about botting.
14:49All right, so I hope you enjoyed this
14:50fast-paced tutorial. If it helped you
14:52out, please consider subscribing and
14:53leaving a like. If you have any tips on
14:56how I can improve the video, do leave
14:57them in the video description. Have a
14:59great day.

About this transcript

This page contains the full transcript of How to make advanced image recognition bots using python by Kian Brose, generated from the public captions YouTube serves with the video. The transcript has 3,112 words across 419 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.