YT YouTube2Text

YouTube2Text → Transcripts → 2026 FREE Data Analyst Bootcamp [24 Hours+] for FREE | SQL, Excel, Python, Power BI, GitHub, AWS

2026 FREE Data Analyst Bootcamp [24 Hours+] for FREE | SQL, Excel, Python, Power BI, GitHub, AWS — Transcript

by Alex The Analyst · 334,400 words · 45,258 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Hello everybody and welcome to the 2026
0:02data analyst boot camp. Back in 2024, I
0:04released the first ever data analyst
0:06boot camp on YouTube. It was about 24 or
0:0825 hours worth of content that I've been
0:10creating for the past four, five, or
0:12even 6 years. Believe it or not, since
0:14then, I've been creating even more
0:16content. And so, I'm adding a lot of
0:17those new things into this boot camp to
0:19make it even longer and cover even more
0:21things. In the original boot camp, we
0:23covered a ton of stuff already. And I'm
0:24going to read it because it's too many
0:26things. Uh we covered my SQL, Excel,
0:28Tableau, PowerBI, Python, Pandas, Python
0:31projects, building a portfolio website,
0:33creating a resume, practicing for
0:35technical interviews, Azure, AWS, and
0:38then how to use LinkedIn to land a job.
0:40That is a ton of stuff. And those are
0:42just videos I've been making for the
0:43past several years, and I put it into
0:45one really long video. In the boot camp
0:47that you're watching right now, we are
0:48adding a ton of new content. The first
0:50thing being a data fundamentals
0:52playlist. These videos cover things like
0:54what is data, what are data types, what
0:56is data cleaning. A lot of things that
0:57data analysts need to know. We're also
0:59going to be adding in our entire Git and
1:01GitHub series because Git and GitHub are
1:03just really useful to know and they're
1:04being used more and more especially with
1:06AI becoming so popular. I'm also adding
1:09in the R for data analyst series which
1:11is basically just R programming for data
1:13analysts. And so if you like R, you are
1:15in luck. I am adding it to this boot
1:17camp. Lastly, we're going to be adding
1:19in data bicks. And data bicks is a
1:21phenomenal data platform to know how to
1:23use. It's super popular and a lot of
1:25people say you either choose data bricks
1:26or snowflake and I'm going to create a
1:28snowflake series in the future, but
1:29we're adding in the data bricks series
1:31and so that's going to cover how to use
1:32data bricks and how to create ETL
1:34pipelines within data bricks as well. So
1:36there's a ton of new things that we're
1:37adding to this boot camp and it all is
1:40here just to help you learn the skills
1:41that you need in order to become a data
1:43analyst. If you did not know, I also
1:45have a paid platform called Analyst
1:47Builder. Analyst Builder is where I have
1:48all of my full courses. It is my paid
1:50platform and so all the things that
1:52we're covering within this boot camp, I
1:54have full courses that dive even more in
1:56depth and have more advanced projects
1:58within Analyst Builder. We also have a
2:00part of the platform where you can
2:01practice for interviews. So you can
2:03practice SQL and Python and R all within
2:06the platform. And there's also general
2:08technical interview questions. If you
2:10don't know how to answer some of these
2:11questions, we have an entire part of the
2:13platform where you can practice and get
2:15better at answering those questions. And
2:17if you are looking for projects to
2:19build, we have a whole section on the
2:20platform where you can find projects and
2:22you can build them out and share them
2:24with others. Analyst Builder is my paid
2:26platform. YouTube is where I have all of
2:27my free content. So, they're two
2:29separate things. If you just want to
2:31stick to the free content, this boot
2:32camp is the best boot camp you are going
2:34to find on YouTube. Hands down. I
2:37promise. But if you're wanting to go
2:38even more in depth and you're willing to
2:40spend a little bit of money, Analyst
2:41Builder is where I would go. I truly
2:43hope that you learn a ton from this boot
2:45camp. I have spent years creating the
2:47content to put into this boot camp and I
2:49hope that you really enjoy it. I have no
2:51idea how long this boot camp is going to
2:53be. The last one was like 25 hours. So,
2:55I hope you enjoy the somewhere between
2:5725 and 30ome hours worth of learning.
3:00We're [music] going to start at the very
3:01beginning assuming you haven't started
3:03this process at all of becoming a data
3:05analyst. If you already have, you can
3:07kind of find and identify where you are
3:09in this process and then go from there.
3:11Now, before we dive into everything, I
3:13want to warn you I will be mentioning my
3:15own channel a lot in this video. I have
3:17videos and playlists on just about every
3:19single topic that we're going to be
3:20talking about today. I'll have all the
3:22links to those videos in the description
3:23so you can dive into those topics more
3:25in depth. So, I hope that's okay. And
3:27it's all completely free. I've been
3:28building this out for the past 3 years
3:30and honestly, you can probably get 90%
3:33of the way to learning everything you
3:34need for data analytics just on my
3:36channel. So, now that I've warned you,
3:37let's jump to number one and that is
3:39learn the data analyst skills. Now,
3:41there are literally a hundred different
3:42things that you can learn for data
3:43analytics. You can learn things like
3:44AlterX or a cloud platform or different
3:46programming languages, but there are
3:48some core skills that I recommend you
3:50start out with before kind of branching
3:51into some of those other skills. The
3:53number one skill that I always recommend
3:55people start with is SQL. SQL is just
3:57one of those fundamental skills I think
3:59everybody should learn. Even if you
4:00don't use SQL, you'll use some variation
4:03of SQL if your company has a large
4:05enough data set. SQL is used to actually
4:07query and retrieve data from a database.
4:09So, if your company collects data, which
4:11every company does, they're going to put
4:13it somewhere to store. It's usually
4:14stored in a database, and SQL is how you
4:17get that data from the database. I think
4:19SQL is also fairly easy to learn, which
4:21makes it really good when you're just
4:22starting out. I have several playlists
4:24dedicated to SQL, starting from beginner
4:26all the way to advanced, and you can
4:27learn all of that for free. One other
4:29reason why I think you should learn SQL
4:30first, is that a lot of companies
4:32interview or have a technical interview
4:34during the interview process on SQL.
4:36That's something that really caught me
4:37off guard when I was first starting out
4:38because I thought it was going to be
4:39more behavioral. I didn't even know what
4:41a technical interview was. So, knowing
4:43SQL actually became a really important
4:45part of interviewing and getting a job
4:47as a data analyst. The second skill that
4:49I will learn is a business intelligence
4:50tool like Tableau or PowerBI. Now, there
4:53are a ton of different BI tools. I can
4:55literally name 10 off the top of my head
4:56that I've used throughout my career. But
4:58what I will say is that learning
4:59something like Tableau or PowerBI is
5:01pretty transferable to almost all those
5:03other BI tools. They're all fairly
5:05similar in how they do things and how
5:08they show and display the data. You most
5:10likely won't have a technical interview
5:11asking you about Tableau or PowerBI like
5:13to build something for them. That
5:15usually does not happen. But the
5:17combination of SQL where you can query
5:19your data and then taking that data to
5:20build something that is a really really
5:23great combination to learn right away. I
5:25have entire series on both Tableau and
5:26PowerBI with projects on my channel. The
5:29third skill that I would learn is Excel.
5:31Now most people have used Excel. They
5:33know what Excel is and how it's used,
5:34but it can be used a little bit
5:36differently for a data analyst. For
5:38example, in Excel, a lot of people
5:39haven't cleaned data in Excel or built
5:42charts and graphs using Excel, and those
5:44are things that data analysts would
5:45probably do. Excel is also just a
5:47fundamental skill that every company is
5:49going to expect you to know. So, I have
5:51an entire playlist dedicated to Excel to
5:53actually walk you through how to use it
5:55for data analysis. The fourth skill that
5:57I recommend you learn is Python. Now, a
5:59lot of people will have Python higher up
6:00on their list. They only use Python.
6:02They don't use SQL or a BI tool. They
6:05just do everything in Python. Now,
6:06Python is a fantastic tool. You can use
6:08it to manipulate your data to create
6:10data visualizations and a ton more like
6:13web scraping and regular expression and
6:15100 different other things. But it can
6:17be kind of hard to learn. It took me a
6:19long time to really learn the basics
6:21very well. That's really the only reason
6:23why it is farther back. I feel like SQL
6:25and a BI tool are really easy to learn
6:27and really pack a big punch. Whereas
6:30Python can be quite tough to learn in my
6:32experience and you may not use it as
6:34often as you would something like SQL or
6:36a BI tool. If you're interested in
6:38learning Python, I have an entire series
6:40dedicated to Python as well as projects
6:42that you can build. Again, I warned you
6:43there's going to be a lot of
6:44self-promotion in this video. I have
6:46videos on just about every single one of
6:47these topics. The fifth and the last
6:50skill that I recommend you learning, and
6:51this is the only one that I don't have a
6:53series on yet, and I will make those, is
6:56learning a cloud platform like AWS,
6:58Google Cloud Platform, or Azure. There's
7:00no denying that these platforms have
7:02played a huge impact on how we use data
7:04as a whole in the data analyst industry.
7:06They can be kind of tough to learn,
7:08though, if you aren't using it hands-on
7:10in an actual job. I think that learning
7:12a cloud platform is already something
7:14that most people should start working
7:15towards because in the future it's only
7:17going to become more prevalent. After
7:19you learn all of these skills, the next
7:20thing that I recommend you do is
7:22actually build projects with those
7:23skills. Now, what does building a
7:25project actually mean? It means taking a
7:27skill and then building something out of
7:29it that you can then show a potential
7:31employer. For example, if you went
7:33through and learned Tableau, you could
7:34go and take a data set and you could
7:36build a visualization and a dashboard in
7:38Tableau and that would be a project.
7:41With these projects, you can build
7:42something called a portfolio. And I
7:44usually call it a portfolio website. A
7:46portfolio website is a website that you
7:48create where you store all of your
7:49projects and then you can share that
7:51with recruiters and hiring managers so
7:53that they can see all of your work. Now,
7:55do you absolutely need a portfolio to
7:57show employers? No, you don't. But it
8:00does help in two different ways. The
8:02first thing that it may do is actually
8:03help you land the interview. If you have
8:05a link on your resume and they click on
8:06it, they may see your skills and see
8:08your projects and be like, "Man, this
8:10person really knows what they're doing.
8:11This is exactly what we need." The
8:13second reason that I recommend building
8:15projects is because most likely during
8:17your interview, you're going to get
8:18asked questions like, "How have you used
8:20SQL? How have you used Tableau?" And if
8:23you don't have any experience in that,
8:25you're just going to say, "Well, you
8:26know, I've taken courses to learn it."
8:28But with a project, you can be a lot
8:30more specific. You'll be able to say,
8:32"Well, I actually just built out this
8:33project in Tableau. I took the data and
8:36cleaned it in Excel and then I put it in
8:38Tableau and built out this dashboard and
8:39here are the insights that I found from
8:41this data set." It's just a much better
8:43answer. And as a hiring manager myself,
8:45I can tell you that it is definitely
8:47beneficial to build out these projects.
8:49The next step that I recommend you take
8:50in becoming a data analyst is building a
8:53data analyst resume. The resume, to say
8:55the least, is extremely important. It's
8:58what's going to actually allow you to
8:59land an interview to potentially get a
9:01job. Now, if you were like me when I was
9:03first starting out, I had a resume. It
9:06just had nothing to do with data
9:07analytics. So, how do you make a data
9:10analyst resume if you don't have any
9:12experience as a data analyst? Well, you
9:14are asking the perfect questions because
9:16the very first things that we talked
9:18about are what are going to go on your
9:19resume, those skills, and those
9:21projects. If you have no experience or
9:24degree, like myself who has a
9:26recreational therapy degree, if you have
9:28no background in this, it can be really
9:29daunting to kind of display that you
9:31know what you're doing and that a
9:33company should hire you. So, what I
9:35usually recommend is right beneath your
9:36contact at the top, you put your skills
9:38and your projects that you built out on
9:41your resume. Things like work experience
9:43and education should go on your resume
9:44as well, but just a little bit lower.
9:47You want them to see those things before
9:49they see that your last work experience
9:50was at Domino's and you have a degree in
9:52marine biology. It's just not relevant
9:55to data analysis. And if you put those
9:57things at the top, they're probably
9:58going to rule you out right away. The
10:00fourth step to become a data analyst is
10:02actually applying. You have the skills,
10:04you have the projects, you have the
10:05resume, now you're ready to start
10:07applying for those data analyst jobs.
10:09Now, there's a lot of different opinions
10:10on how you need to go about applying for
10:12data analyst jobs, but I'll give you my
10:14take on it, and this has been the most
10:16successful for me and my career. The
10:17first thing that I want to mention is
10:18actually what I would not do, which is
10:20just blindly apply on Glass Door,
10:22Monster, Zip Recruiter, and all these
10:24other platforms to just any data analyst
10:26job that you can find. Now, I'm not
10:28against this. I think you should do
10:30that, but I don't think that's the only
10:31thing that you should do because the
10:33chances of you getting a call back or
10:35actually hearing something back are
10:36extremely low. To really increase your
10:39chances of becoming a data analyst, I
10:41highly, highly, highly recommend working
10:42with a recruiter. A recruiter is
10:44literally someone who is there to help
10:46you find a job. Now, when I first
10:47started out, I didn't understand what a
10:49technical recruiter was at all. I was
10:51kind of nervous or scared to work with
10:53them. But it's actually pretty simple. A
10:56company has a position that they want to
10:57fill and they don't want to spend hours
10:59and hours and hours to find someone to
11:01fill that position. So, they hire a
11:03recruiter. a recruiter is going to go
11:04out and try to find someone to fill that
11:06position, aka you. And so if you go and
11:09talk to that recruiter and they have a
11:10position that opens up, they will help
11:12you get that interview. And then if you
11:14get a job, let's say for $50,000, the
11:17company is going to pay that recruiter,
11:18let's say, 10% of your salary. So
11:20they'll give them $5,000. So you don't
11:23actually lose or have anything to lose
11:25using a recruiter. You can reach out to
11:27recruiters in several ways, and I've
11:29done every variation, but I'll tell you
11:31my most successful way, which was using
11:33LinkedIn. There are tens of thousands of
11:35recruiters on LinkedIn. I made an entire
11:37video of how you can reach out to
11:38recruiters and what to say to recruiters
11:40on LinkedIn to help you land a job. So,
11:42be sure to check out that video when you
11:44actually get to that point. But, you can
11:46also just cold email and cold call these
11:48recruiting companies. But to me, it's
11:50just not as effective as reaching out
11:52directly on LinkedIn. And this is just a
11:54bonus one. The last thing that you need
11:56to do is accept a job offer. So, in step
11:58number four, after you apply to those
12:00jobs, you do actually have to go in,
12:01interview, and then get a job offer,
12:03which you will accept. I just thought
12:05I'd mention that just in case that was
12:06not super clear. Now, that was a lot of
12:09stuff. Let's talk about time frames to
12:11actually complete all of these things.
12:13Now, doing all of these things from
12:15scratch is going to take a while, but
12:16let's break it down by each step and see
12:18how long I generally think it's going to
12:20take. Let's start with step number one,
12:22which is actually learning the skills.
12:23Now, just to be upfront, this one
12:25probably is going to take the longest
12:26for most people. For most people to
12:29learn all of these skills, it's going to
12:30take around 3 to four months. Now, if
12:32you don't learn a cloud platform and
12:34Python, which are the last ones that I
12:36recommend, and you just focus on SQL,
12:38ABI tool, and Excel, I think you can do
12:40that in under 3 months. That is very
12:43dependent though on how much time you
12:44have to study. That time frame is more
12:47for someone who has several hours per
12:49day, maybe 3 hours in the end of the
12:51night after you go to work. That is
12:52someone who has quite a bit of time to
12:54dedicate to learning during their week.
12:56Of course, that time frame is going to
12:57take longer if you don't have as much
12:58time to dedicate to learning. Now, let's
13:00look at number two, which was creating
13:02projects and a portfolio of projects.
13:04From my experience, when you're first
13:05starting out, it takes a lot longer to
13:07actually create these projects. It can
13:08take one or two weeks per project. I
13:11usually recommend people doing three to
13:12five projects in their portfolio before
13:14they start applying. And since they can
13:16take anywhere from 1 to two weeks,
13:18you're looking at anywhere from 3 to 6
13:20weeks. The next step was to create a
13:22data analyst resume. Now, in my opinion,
13:24this one should take the shortest out of
13:25every single step here because you're
13:27really just kind of reformatting a
13:29resume or creating a resume. You're just
13:31adding skills, you're adding your
13:32projects, and then kind of reformatting
13:34it to make it look nice. This should
13:36hopefully take under a week, but if you
13:37use something like a professional
13:39service where they help you build a
13:40resume, it can take one to two weeks.
13:42The two last steps which kind of go hand
13:44inand are step four and five, which is
13:46actually applying for jobs and then
13:48landing a job. Now, this process can
13:50take as little as a month, or it can
13:52take as long as 6 months or a year. It
13:54really depends on how you're applying,
13:56where you're applying, and just the kind
13:59of luck that you're having with actually
14:00landing interviews. I've seen people who
14:02have never had any experience land a job
14:04within a month of starting to apply. And
14:06it's incredible. It's amazing, but it
14:08doesn't happen too often. You're usually
14:10looking at around 2 to four months on
14:13average to land your first data analyst
14:15job. If you put all of those together
14:17and kind of average everything out,
14:19you're looking at around six months
14:20total for the entire process. Now, I
14:22don't want that to discourage you, okay?
14:242023 is a long year. You have a lot of
14:27time and it doesn't have to take 6
14:29months. You could do it faster. You
14:30could do it in 3 months and just prove
14:32me wrong. But if you are really focused
14:34and you are really driven to become a
14:35data analyst this year, I know that you
14:37can do it. Now, to maybe boost your
14:39spirits and make you feel a little bit
14:40better, I didn't know any of these
14:42things when I first started out. I
14:43didn't have anyone telling me kind of a
14:44plan on what to do. I had to go out and
14:47figure all these things out by myself
14:48and it took me almost a year to land my
14:50first real data analyst job. So, with
14:52all that being said, I hope that this
14:53video is helpful. I hope you now have a
14:55path on how to become a data analyst
14:57this year and that my channel can be a
14:59big part of that. What's going on
15:00everybody? Welcome back to another
15:01video. Today, we're going to be starting
15:03our data fundamentals series. [music]
15:10>> [music]
15:10>> Now, in this series, I'm going to be
15:12walking through some really core
15:13fundamental concepts about data. If
15:15you've been on my YouTube channel for
15:16any amount of time, you know that we
15:17talk a lot about data, specifically
15:20about how to use tools to work with your
15:23data. But up until now, I haven't really
15:24dived in and broken down the core
15:26concepts of what data is and how it's
15:28used in the real world. So, in this
15:30series, that's what I aim to do. And in
15:32this video, we're going to be starting
15:33off with what is data. Let's jump over
15:35to my screen and take a look. All right.
15:36Right. So, in this lesson, we're going
15:37to be talking about what data is. Again,
15:38we're starting from the very basics.
15:41We're working our way up through this
15:43series. So, by definition, data is just
15:45raw facts and figures. It doesn't even
15:48necessarily have to be on a computer. It
15:50could be on a notepad. If you're writing
15:52down 1 2 3 4 5, that is data that you're
15:54writing down. That's going to be harder
15:56to use that data necessarily because,
15:59you know, on a computer, it's easier to
16:00process and use data. But that is data.
16:04Some examples of data are things like a
16:07number, the number 45, that is a piece
16:09of data. A word could be data. Just the
16:12word completed or a sentence that is
16:14also data. And then a date could be data
16:16as well. So we have 1210 of 2024. This
16:20is a piece of data. Now the thing about
16:23data is is that data is everywhere. It's
16:26in everything we do all the time. But
16:29without collecting it, without context
16:31of what this data actually means, it's
16:33basically useless. Look at some examples
16:36of how data is used in the real world.
16:38Something that a lot of you will use
16:40almost every single day. So, let's take
16:42a look at this first one. This is a
16:43weather app. I use my weather app almost
16:46all the time. I got to send my kids to
16:47school and I want to check if it's
16:49rainy, if it's hot, if it's cold, if
16:50they need a jacket, if they don't. And
16:52that is something that requires a lot of
16:54data. Meteorologists use tools to
16:57collect data on all of these things.
16:59Then they're presented in these apps. So
17:01they collect data about weather,
17:03temperature, humidity, location, time,
17:05and they take all of this and they
17:07aggregate it. That just means they bring
17:09all of it together into one place and
17:11they aggregate these numbers. And
17:13finally, they'll present it to you based
17:15off of your data, your location, where
17:17you actually are. And so that is
17:19something that is extremely data
17:20dependent. They typically will process
17:22this as well as forecast, which means
17:24they're going to kind of predict out
17:26what they think is going to happen in
17:27the next hour, two hours, maybe it's a
17:29day or several days ahead. They can also
17:31use this data to predict what might
17:34happen in your location. Another example
17:36would be a bank app. So, I go on my
17:38phone all the time to check my banking
17:40statements to make sure that, you know,
17:41I didn't spend too much money. And
17:43everything in your banking app is data.
17:46And so when you go on this app, like you
17:48can see in this image down here, you can
17:50see where you spent your money, you can
17:51see the amount you spent your money, the
17:53date and the time that you spent your
17:54money. These are all different points of
17:57data that the bank has collected. Banks
17:59collect hundreds, if not thousands of
18:01data points, but really common ones are
18:02things like bank deposits and
18:04transactions and payments that you've
18:05made. All of these things so they can
18:07put it in your dashboard so that you can
18:09track your money. Now, let's take a look
18:10at different types of data because there
18:13isn't just one type of data. Data isn't
18:15just one thing. Data is really complex
18:18and there's a lot of different things to
18:20it. So, really briefly, I'm going to
18:21touch on structured, semistructured, and
18:23unstructured data. Structured data is
18:26really neat and really easy to visualize
18:29and see, as you can see in this uh image
18:31right here. That typically refers to
18:32something that has columns and rows,
18:34something like an Excel file.
18:36Everybody's used an Excel file before,
18:38and so that is something that is very
18:40structured, very easy to kind of
18:42visualize and use and understand. The
18:44exact opposite side of this, we have
18:46unstructured data. Unstructured data is
18:49kind of all over the place and it might
18:51be a lot harder to use. For example,
18:53that could be something like a
18:55photograph that you took or maybe a
18:57video or an audio file. These are all
18:59examples of unstructured data that you
19:01can't really put into something like an
19:03Excel file for it to be in columns and
19:05rows. Then we have something called
19:06semistructured data. Semi-structured
19:08data is closer to structured data than
19:11it is unstructured data, but it is more
19:14complex than something like an Excel
19:16file. For example, semiructured data
19:18might be something like a JSON file.
19:20Now, we haven't gotten to file formats
19:22and file types yet in this series, but
19:24when we do, I'll dive into what JSON
19:26files are because JSON files store
19:29things differently than something like
19:30structured data where they nest data in
19:32kind of these hierarchies. Let's take a
19:34look at the two main ones. Structured
19:36versus unstructured data. Structured
19:38data is the data that I primarily work
19:40with as a data analyst. This is
19:42typically going to be something like a
19:44row and column in an Excel file or a CSV
19:46file or in something like a relational
19:49database. A database is just a place
19:51where you will store a lot of data and
19:53all of that data typically connects in
19:55some way. So you can work with a lot of
19:57it. Structured data is also quantitative
19:59versus qualitative like unstructured
20:02data. Structured data is numbers based
20:04and it's measurable and so you can
20:05easily kind of track it and use it.
20:07Whereas qualitative could be something
20:09like a survey where it might be free
20:11text where it's someone saying you know
20:13I had a really great experience at this
20:15and I really liked this and I liked
20:16this. That's a little bit harder to
20:18actually put into numbers how much that
20:21person liked. But quantitative may be
20:23hey how much did you like this on a
20:25scale of 1 to 10? And if the person puts
20:27a seven that's a very specific answer.
20:29Now the amount of data that sits as
20:31structured data is significantly less
20:34than unstructured data. Structured data
20:35is only about 20% of enterprise data
20:38compared to 80% of unstructured data.
20:41And so there's a lot more unstructured
20:43data in the world. And so what a lot of
20:45people in the data world do is they try
20:47to take that unstructured data and they
20:49try to make it structured. And so that
20:52is a big part of what a lot of data
20:54professionals do. Lastly, and we just
20:55touched on this, but structured data
20:57tends to be things like numbers, dates,
20:59strings, which are things like words and
21:01text, versus unstructured data, which is
21:03things like images, audio, video, and
21:05others. That being said, data is
21:07everywhere. It is all around us all the
21:09time, and especially in this
21:11technological world we live in, it is
21:13literally in everything we do. But the
21:16challenge especially for people in data
21:18who are working with data, the challenge
21:20is to collect it, to organize and
21:23analyze it so that you can then use that
21:25data. What's going on everybody? Welcome
21:27back to another video. Today we're going
21:29to be taking a look at KPIs and metrics.
21:31[music]
21:37Now, if you've worked in the data world,
21:38you've probably heard these terms before
21:40because they are very popular to throw
21:42around and use. but you may not know
21:44exactly what they mean. So, in this
21:46lesson, we're going to be diving into
21:47KPIs and metrics. So, with that being
21:49said, let's jump over to my screen. So,
21:51let's dive into KPIs and metrics. And
21:54let's start off with metrics. So, what
21:57exactly is a metric? A metric is
22:00something that, by the way, people throw
22:01out all the time and they typically get
22:03metrics and KPIs confused. And so, we
22:06will be really good going through this.
22:07I'm going to talk about metrics first
22:08and then how metrics relate to KPIs. So
22:12a metric is any measurement that
22:14provides information using data. So if
22:17you're taking data and you're tracking
22:19that data and you're measuring that
22:20data, that is a metric. It can be
22:22anything. For example, let's say we have
22:25a website and we sell things on this
22:27website. It's like an e-commerce
22:28platform. A metric that we might have is
22:31website visits. How many visits per
22:34month do we actually have? And let's say
22:36this month we had 15,000 monthly
22:39visitors. We also could look at sales
22:41numbers. So, we sold 500 units of
22:43whatever product we're selling. So,
22:45these are core metrics. Now, if you look
22:48over on this right hand side, we have
22:50this dashboard. This is something that a
22:51lot of companies will have. It'll be
22:53something to track their metrics or at
22:55least visualize their metrics. It could
22:57be on revenue or margins or all these
22:59different things. This is something that
23:01when you're working with data, people
23:03want to know. They want to be able to
23:04measure and track specific data. Now
23:07let's see how this compares to a KPI.
23:10KPI stands for key performance indicator
23:13and this is a specific metric that
23:15directly measures progress towards a
23:18goal. So every single KPI is going to be
23:21a metric but not every metric is going
23:24to be a KPI. For example, let's look
23:26over on this image on the right hand
23:28side. We start with all of our data and
23:30then we take that data and we say what
23:32metrics do we want to look at within
23:34this data? Then we determine what
23:36metrics do we actually want to track.
23:38Which metrics align with our goals. So
23:41for example, in our website that we
23:43have, we have 15,000 monthly visitors.
23:45But let's say our target KPI is going to
23:47be those monthly visitors, but our goal
23:49is 20,000. And so we have a metric that
23:53we're tracking. And now we're saying we
23:55want this to be a goal that we achieve.
23:57And it's also measurable. For our sales
23:59numbers, we have 500 units sold. One of
24:01our sales KPI could be, okay, this month
24:04we're selling 500, but next month we
24:06want to increase our sales by 2% each
24:09month going forward. So that could also
24:11be a KPI. You may be asking, how do you
24:13choose a good KPI? And this is a very
24:16good question because I've worked with a
24:18lot of different data teams and they
24:20just sometimes randomly choose KPIs.
24:22They're like, this seems like a good one
24:24to achieve or get better at. That may be
24:26true, but it may not actually progress
24:28you towards the goal that you're
24:30wanting. And so when you're trying to
24:31choose a good KPI, let's say you're
24:33tracking these metrics, you're saying I
24:35want to progress towards a goal. How do
24:37we choose what KPIs we want? The first
24:40thing you have to ask is what is your
24:41goal? Maybe for our website it was we
24:44want to increase sales. That's our
24:45number one thing. Then we have to
24:47identify which metric actually best
24:49tracks our progress towards that goal.
24:51On a website there are a ton of
24:52different data points that you can do.
24:54For example, you can see how long a user
24:56is actually on your website. You can see
24:58how many times one user gets onto your
25:00platform every day. Maybe they get on
25:02three or four or five times a day. Those
25:04are metrics, but that may not be what
25:06your goal actually is to sell your
25:08products. And then lastly, you have to
25:09say, is this metric actionable? Maybe
25:11you can't make people sign onto your
25:13website 3, four, five times a day if
25:15that's what you're wanting to do. That
25:16may not be an actionable metric to
25:18actually track, but something like the
25:20daily users is because maybe you could
25:22send out a daily email or you can do
25:24some type of marketing to get more
25:25people on your platform. These are
25:27things that you have control over and
25:29that are actionable. So, really quickly,
25:30let's just break down key performance
25:32indicators versus business metrics. A
25:34KPI is going to provide a direct line of
25:36action to a business goal. Whereas
25:38business metrics don't necessarily
25:40contribute directly towards a goal. KPIs
25:42are measured actively as a benchmark to
25:44a goal. Whereas business metrics can be
25:47measured, but they aren't typically
25:48checked and monitored. For example, at a
25:50previous company I was working with, we
25:51had lots of metrics that we kept kind of
25:54tabs on, but we only put the key
25:56performance indicators or the most
25:58important ones into dashboards or maybe
26:00into reports that we would then provide
26:02to our users. Next, KPIs are a set of
26:05standards that are vital to the
26:06business. Whereas metrics may or may not
26:08be vital to the business. Again, could
26:10be anything versus KPIs are like the
26:12core things that you want to track and
26:13get better at. And then lastly, this is
26:15something I mentioned before. All KPIs
26:18are business metrics, but not all
26:19business metrics are KPIs. You specify
26:22exactly what metrics you want to be KPIs
26:25to help move your business or grow your
26:27business, however you want to do that.
26:29So, in a nutshell, a metric helps us
26:31understand what's happening in our data.
26:32And KPIs show us whether we're reaching
26:34our goals using our data. I hope that
26:37this was helpful. This was not something
26:38that I knew when I first started uh
26:40working in data. It's something I kind
26:42of figured out along the way. But if you
26:44are just getting into data, or maybe you
26:46already are and you don't know what this
26:47is, now you know. And now you can go
26:49into business meetings and you can go
26:50into conversations being a little bit
26:52more knowledgeable. What's going on
26:54everybody? Welcome back to another
26:55video. Today we're going to be taking a
26:57look at data types.
27:04If you've worked with data at all, you
27:05know that data types are very important.
27:07And if you haven't, you're in the right
27:08place cuz we're going to talk a lot
27:10about what data types are and how they
27:12are used. So with that being said, let's
27:13jump onto my screen and take a look. So
27:15let's dive into it. Let's take a look at
27:17what data types are. Now data types,
27:20really quickly before we jump into the
27:22definition, data types are something
27:23that if you work with any data in any
27:26way, data types do pertain to you. These
27:29are things that you need to know. These
27:31are things that eventually will either
27:33help you or hinder you on whatever
27:35you're trying to get done. So a data
27:38type is an attribute associated with a
27:40piece of data that tells a computer
27:42system how to interpret its value. Every
27:44single programming language, every
27:46database has similar but slightly
27:48different data types. I've worked with
27:50all types of programming languages and
27:52lots of different databases and they all
27:54act somewhat similarly, but they can be
27:56different at times. So knowing the core
27:58concepts that we're going to cover in
28:00this lesson can be really helpful to
28:02kind of traverse the different types of
28:04data types in different systems. To get
28:05started, there are a ton of different
28:08data types. I mean, there's probably 50,
28:0960, 70 different data types. I'm just
28:11throwing out random numbers, but they
28:13all fall under certain categories.
28:16They're usually either strings, which
28:18are things like free text, numbers, like
28:201 2 3 or date and time. So, January 1st
28:23of 2024 at 6 p.m. These are kind of the
28:26core fundamental data types. But under
28:29each of these are a lot of other data
28:31types as well. Now within string you
28:33could have something like a name. John
28:35Smith, Emma Johnson. This is very simple
28:38data that you can categorize and you can
28:40use. Next it could be an address. 123
28:42Main Street, New York, New York. This is
28:44something that can determine a location.
28:46Then we have something like product
28:47categories, electronics, furniture, etc.
28:49These product categories are all
28:51strings, but then can be used later on
28:53in the data process for things like
28:55aggregation. Maybe you want to see how
28:56many customers are buying electronics at
28:58your store versus furniture at your
29:00store. You have to collect that data.
29:02You do that with string data. Next, we
29:04have numerical or numbers. This is
29:07probably the most simple data type. It's
29:09integers. 25 100 - 15. Then we also have
29:13decimals. These are both two separate
29:15data types. So decimals is like 12.5
29:1899.99.
29:20Now within certain systems, especially
29:21databases, they'll have something like a
29:24big integer for really large numbers.
29:26They'll have something like a small
29:27integer. They'll also have just a
29:29regular integer. Within decimals, they
29:30may have something like a double point
29:32system or a decimal data type. So within
29:35each of these, they may have even
29:37subcategories of data types. Next, we
29:39have date time. The dates could be
29:41something simple like 1210 of 2024. We
29:44have a day, a month, and a year or it
29:46could be month day year. There are
29:48different formats within dates. Then we
29:51have times or timestamps. So times just
29:54indicate the time of the day. Then
29:56timestamps are both date and time
29:58combined. So the timestamps tend to be
30:01very specific. Then we have dates which
30:03give a general day and then times that
30:06just give a specific time. Timestamps
30:08can be very useful, but often times when
30:11you're working with it in the real
30:13world, you'll break out these two data
30:15into separate columns. You have date in
30:17one, time in another, and that may allow
30:19you to do more advanced things with this
30:21data. So understanding what data type
30:24your data is is really important because
30:26that allows you to work with it and
30:28analyze that data in different ways. For
30:30example, if we're working with numerical
30:32data, we can use that to calculate
30:34averages. You can say the average person
30:36is spending this much on our website per
30:38day. That would be an example of how you
30:40can use numerical data. Or with string
30:42data types or categorical data, you can
30:45group it and count on those occurrences.
30:47Or you can do any type of aggregation.
30:49sum, max, min, average, median, all
30:52these different things. You can see how
30:54many people are buying things in our
30:56electrical department versus our
30:57furniture department. And that is
30:59worthwhile data to use. That really is
31:01the basics of data types. Now, if you go
31:04into specific things like if you go into
31:06SQL or if you go into Python or Excel, I
31:09have a lot of lessons specifically for
31:11data types in those systems and then it
31:13gets a little bit more complex. And so
31:15if you want to really dive into data
31:17types in a specific system, go into any
31:19of my playlists, whether it's SQL,
31:21Excel, uh PowerBI, Tableau, they all
31:23have different data types and you can
31:25learn a lot more about data types within
31:27that specific tool. What's going on
31:29everybody? Welcome back to another
31:30video. Today we're going to be taking a
31:31look at file types.
31:39Now, file types or file formats are
31:41everywhere. If you've saved any type of
31:43file ever to your computer, you had to
31:44save that as a specific file type. So,
31:47in this lesson, we're going to be diving
31:48into exactly what file types and file
31:50formats are and how they are used. So,
31:52with that being said, let's jump over to
31:53my screen and take a look. All right, so
31:54let's take a look at what file types
31:57are. File types refer to the format in
31:59which data is stored in a file. And
32:01you're going to see even in the next
32:02slide, there are so many different file
32:05types out there. There are a lot. And
32:07we're going to get into specific file
32:08types in this video that you can see a
32:10lot of the different types of files that
32:12are very commonly used. Each file type
32:14is designed for a specific purpose and
32:16it depends on what kind of data it's
32:18holding and how it's going to be used or
32:20shared. You really can think of these
32:21like a container, whether it's a bowl or
32:24a plate or it's a vase. Each one of
32:26these containers is designed to hold
32:29different things. And that's kind of
32:30like what a file type is used for. Now,
32:32if you go right now, you look at your
32:34file explorer if you're using a Windows
32:36machine. If you go to your view and you
32:37look at the details, the view is right
32:39next to the sort button up there, you
32:41can see the different type of file type.
32:44And with all this is my actual, you
32:46know, file explorer right now. If you go
32:47look in my downloads, we have a lot of
32:50different stuff. I have MP4 files. I
32:52have different file folders that holds
32:53different files. I have PGs, I have
32:55CSVs, I have PDFs, I have all sorts of
32:58stuff. To the right of this you can see
33:00the size as well. All these different
33:03file types hold the data differently and
33:06some take a lot more data than others.
33:08So that size is how much data is
33:11actually stored within that file. Let's
33:12take a look at probably the most simple
33:14type of data file or data format that
33:16you'll see which is a text file. A text
33:18file is super simple and just stores
33:20data as plain text often used for
33:23unformatted or tabular data. This is
33:25data that I would work with all the time
33:27as a data analyst. We would work with
33:29text files and CSV which stands for
33:32commaepparated values. This image on the
33:34right hand side is being stored right
33:36now as a text file. But if I then went
33:38and saved it as a CSV, it would separate
33:41these values based off of the commas. So
33:44country, salesperson, order amount,
33:45quarter, this data would be separated
33:48and all the commas going down on each
33:50row would be separated into basically
33:51columns and rows. DSV and text files are
33:54super common and they can store a lot of
33:56data very simply because they're just
33:59storing plain text and so it doesn't get
34:01super complex and so it's a very popular
34:03type of file format. Next we have
34:05structured file types. Now these are
34:07files that store data in a predefined
34:10structure typically rows and columns or
34:12in a hierarchal format. One of the most
34:15common types is one that I'm sure almost
34:16everyone has used. That's an XLSX.
34:19That's going to be an Excel file. your
34:21typical Excel workbook. And so right
34:23here on the right hand side, this is
34:24your standard workbook. You're going to
34:26have columns and rows and different
34:27worksheets and you'll be able to do
34:29different things with that data in these
34:31Excel files. We also have a DB file and
34:34this stands for a database file. So
34:37oftent times in the data world, if
34:39you're working with a customer or a
34:40client, they might give you an entire
34:42backup of their database and you can go
34:44and use that. And within that database,
34:47they're going to have columns and rows,
34:48especially if it's a relational
34:50database. They're going to have columns
34:51and rows just like an Excel file, but on
34:54a much larger scale. It typically can
34:56hold a lot more data. And then, of
34:58course, you get more complex things
34:59within a database file, like database
35:02schemas to connect and bring all that
35:04data together. Next, we have
35:05semistructured file types. Now, these
35:08get a little bit more complex. These
35:10file types have a loose structure, often
35:11used to store complex data
35:13relationships. Some common formats for
35:15this are JSON and XML files. Now, these
35:17can get a lot more complex than just
35:19columns and rows because in columns and
35:21rows, it's pretty simple. It's pretty
35:22straightforward. But something like a
35:24JSON file, which we have on the right
35:26hand side, you can have data nested
35:27within other data, which just means data
35:29within data within data. And so, there
35:31can be lots of layers to this data. This
35:33is another one that's just really
35:34popular and really common within data
35:36professionals. And they are really great
35:38for storing data that's a little bit too
35:40complex for something that's really
35:41simple like columns and rows. Next, we
35:43have unstructured file types. Now, you
35:45saw this within my file explorer. These
35:47are files that don't follow a specific
35:49format. They don't have columns and
35:51rows. They're often just raw data or
35:53multimedia data. For example, this video
35:55that I'm recording right now, I'm
35:56recording onto an MP4 file. This image
35:59on the right of this beautiful little
36:01hummingbird is apng or there's actually
36:04lots of different formats for images,
36:06but PNG is probably the most common one.
36:08Lastly, we have big data and specialized
36:11file types. So, these file types are
36:13designed specifically to handle really
36:15largecale data efficiently. One of the
36:18most common and one that if you've
36:19worked in the data world or you've
36:20worked in Azure or maybe even AWS that
36:23you might be familiar with, it's
36:24something called a parquet file. Paret
36:27files are very common, especially if
36:29you're working in something like Spark.
36:30That's something where I've used it
36:32quite a bit or even data bricks where
36:33you're bringing in massive amounts of
36:35data and you want to do that in a really
36:37efficient manner. So, it has all these
36:39different properties that they bring in
36:40to these paret files that a normal CSV
36:43file would never be able to do or
36:45handle. Now, what file type you choose
36:46can be extremely extremely important.
36:49And I've made a lot of mistakes over my
36:51years as a data analyst storing data in,
36:53you know, one file format versus
36:54another. And that has caused issues. And
36:57so, and so knowing how data actually is
36:59collected and used and stored and
37:01analyzed and shared, you have to take
37:03all these things into account when
37:05you're determining what kind of file
37:06type you want. Now, in the next lesson,
37:09we're going to be looking at data
37:10collection. And we're going to address
37:12this exact thing because if you store
37:14data in a specific way and then you want
37:16to put it in a database, it may not be
37:17possible if you store it incorrectly.
37:19And so, there are a lot of things to
37:20consider within just data collection.
37:23But this has been our lesson on file
37:25types. What's going on everybody?
37:26Welcome back to another video. Today
37:28we're going to be talking all about data
37:29collection. [music]
37:36Now, data collection is an extremely
37:38part of getting data. In fact, I was on
37:40a data collection team for over 3 years.
37:42I absolutely love data collection and so
37:44I'm really excited to talk about this
37:46topic. Let's not waste any time. Let's
37:48jump on my screen and take a look. Let's
37:49take a look at data collection. Now,
37:52what exactly is data collection? Just as
37:54a definition, data collection is the
37:56process of gathering data from different
37:58data sources to use in analysis,
38:00decision-m and problem solving. A data
38:03source means where the data is actually
38:05being created. And data is created all
38:07around us. It could be in a hospital ehr
38:09system. It could be in your bank
38:11account. It could be through APIs. It
38:13could be through a website or it could
38:14be in a CSV file. So why is data
38:16collection important? And what I really
38:18should say is why is it really
38:20important? One, it ensures that you have
38:22raw material or the raw data needed to
38:25make informed decisions. You can then
38:26use that data to identify trends,
38:29patterns, and opportunities within that
38:31data. And it also lays the foundation
38:32for data quality. This is something that
38:35we're going to cover actually in our
38:36next lesson when we look at data
38:37cleaning. So, if you collect data poorly
38:40or if you do not process that data
38:41correctly, that can lead to bad data,
38:44which can give you incorrect results and
38:45then of course you're going to make bad
38:47decisions with bad data. Now, data
38:49collection does not just happen. It
38:50doesn't just magically appear. This is a
38:52very calculated and specific process
38:54that needs to happen in order for you to
38:56get that data. Let's take just a really
38:58quick example. You are actually running
39:00an online shop. You have a website.
39:02You're selling stuff. On the right,
39:03you're selling t-shirts and jackets and
39:05other clothes. I'm assuming now you want
39:08to track how many customers actually put
39:10something into their cart. You might be
39:12able to compare how many customers put
39:14something into their cart versus how
39:16many customers actually bought
39:17something. And that could be useful
39:18information to you. that data just
39:20doesn't appear. A data collection system
39:22is set up to collect that data and place
39:24it into something like a database so it
39:26can be analyzed and used. Now these data
39:28collection systems can be something that
39:29you manually create and we'll talk about
39:31that in a little bit or it could be
39:33something that you just pay for. So it
39:34could be a system that you say, "Hey, I
39:36want you to collect this data." They go
39:38and do it for you and you don't have to
39:39actually do the work, but there is a
39:41data collection system in place. Now
39:43whether it's you or some system you paid
39:45for, all this data is going to be
39:47processed via a data pipeline. Now this
39:50is a system that's going to automate the
39:51movement of data from one place to
39:53another while often transforming it
39:55along the way. Let's take a look at this
39:57ETL pipeline. ETL stands for extract,
40:01transform and load. And you can see that
40:03uh through this process. So on the left
40:05hand side we have all these different
40:06data sources. It could be different
40:08parts of your website or it could be
40:10different websites or it could be
40:11different locations of the data
40:12entirely. We're going to extract that
40:14raw data and we're going to put it into
40:17a staging area. This staging area is
40:19kind of like a temporary hold for your
40:21data where then you can work with data
40:22professionals like data engineers,
40:24database developers, data analysts, data
40:26scientists who can then determine how we
40:28want to transform that data. Now
40:30transforming data is something we'll
40:32cover again in the next lesson when we
40:34talk about data cleaning. But
40:35transforming the data makes that raw
40:37data more usable for whatever you're
40:39trying to use it for. After we transform
40:41that data, we're going to load that data
40:42into something like a data warehouse, a
40:44database, maybe it could even be an
40:46Excel file. It could be as simple as
40:48that. But then from there, we can use
40:50that to analyze our data. And that's
40:52really what data collection is in a
40:54nutshell. Now, I worked on a data
40:56collection team for many years, and I
40:58was a data analyst, and I would often
41:00work at every single step of this
41:02process. I would have to go and talk to
41:03the client and see exactly what data
41:05they had available and sometimes they
41:07didn't even have data available and so
41:09they said hey we want to collect this
41:10new type of data as part of my job is I
41:13would help them understand how to start
41:14collecting that data so then we could
41:16create these data pipelines to then
41:18transfer it into a database to use data
41:20collection is not a one-time thing
41:22either data collection is always
41:24happening these ETL pipelines they break
41:27or they need new ones or the source data
41:29changed on the left hand side of this
41:31diagram the source data changed from one
41:33part of the website to another. And so
41:34you need to go back and you need to fix
41:36that data pipeline to correctly collect
41:38all the data that you're wanting. And so
41:40this is a very active process. This
41:42typically doesn't just happen one time
41:44and then it's done. It typically happens
41:46one time, then you adapt it and you keep
41:48changing or maybe fixing it over time.
41:50What's going on everybody? Welcome back
41:51to another video. Today we're going to
41:53be talking all about data cleaning.
42:00Now data cleaning is an essential part
42:02of working with any [music] data and so
42:03this is a very core fundamental concept
42:06that anybody who works with data will
42:07need to know. As a data analyst I've
42:09cleaned data in almost any way that you
42:11can imagine. So I am very excited to
42:13talk about this topic. Let's not waste
42:14any time. Let's jump on my screen and
42:16take a look. All right. So let's take a
42:17look at what data cleaning actually is.
42:20Data cleaning is the process of
42:22identifying and fixing issues aka dirty
42:25data in your data to ensure it's
42:27accurate, consistent, and complete.
42:29Let's take a look at these patient names
42:31on the right hand side really quickly.
42:33These are all my names, but just in
42:35different versions or variations. This
42:37is very dirty. So, we have Alex
42:39Freeberg, Alexander Freeberg, Alex F,
42:41Alexander F, Alex Freeberg with a U,
42:43Alejandro Freeberg. These are all the
42:46same person. That's all me. But if
42:49you're looking at this in a database,
42:50it's going to consider all these people
42:52different. And so what we would have to
42:54do to clean up this data is we would
42:56have to make all of these names
42:57consistent and the same. Maybe we do
42:59that by saying, okay, all these people
43:01have the same date of birth. They're all
43:03male. They all have the same social
43:05security number. But with all of those
43:07things, we could say, yes, these are all
43:09the same person. We can make all of
43:10these patient names Alex Freeberg with
43:13an E. That would really clean up this
43:14data a lot. Why do we need to clean data
43:17at all? What is the reason that we do
43:19this? One is to ensure accuracy because
43:22you're going to be giving this data to
43:24stakeholders or your boss or a customer
43:27and they're going to expect that this
43:28data is accurate. But if you're giving
43:31them bad data, then they may make bad
43:32decisions with that data and that's
43:34going to come back on you. Next, it
43:36improves efficiency. When you are
43:38working with clean data, it is 10 times
43:40easier to analyze and use that data in
43:43any way you need rather than it being
43:45really messy and really difficult to
43:47work with. Lastly, it builds trust and
43:49stakeholders and customers and clients,
43:51they want to be able to trust that what
43:54you're giving them, the data that you're
43:55using is actually accurate and complete.
43:58And so those stakeholders are really
43:59going to trust you if that data is clean
44:00and accurate and you can find insights
44:02with that data. Now, let's take a look
44:04at some other examples of dirty data.
44:06And these are not all of the examples,
44:09but these are many of them. Here in this
44:10data, we have columns and rows like an
44:12Excel file. We have a name, a phone
44:14number, an email address, and an
44:16address. In each of these, you can see a
44:18lot of mistakes. Take for example in the
44:20name column. We have things like
44:22punctuation marks and abbreviations.
44:25These are the same person, just like we
44:26looked at in the example with my name.
44:28These are all the same person, but each
44:30of these records requires a different
44:31row of data because it isn't
44:32standardized or cleaned. Let's look at
44:34phone number. This one shows us missing
44:37data. And this is a very common one
44:39where you're collecting phone numbers
44:41and all of a sudden you just don't have
44:42a phone number. So you're missing that
44:44data. Now, what are you going to do with
44:45that when you're cleaning it? Maybe you
44:47need to populate that data. Maybe that
44:49is something that you can get from
44:50another data source or you know it and
44:52you can actually put it in there. That
44:54is a way you can clean that data up.
44:56Next, we have mixed numbers and letters.
44:58It may look like 831262149,
45:01but that one is actually an L. And so
45:04this is just a classic case of bad data.
45:06They should all be in the same format
45:08with the phone number, but they're not.
45:10Some have dashes, some have commas, some
45:12don't have anything, and some even have
45:13letters. That's very, very messy. Next,
45:16we have email address. Now, within this
45:18one, they're all somewhat different,
45:20right? We all have different email
45:21addresses, and again, that's just a
45:23standardization, making them all the
45:24same issue. But we also have something
45:26like a non-printable character. Now,
45:29there are specific characters within
45:30computer systems that you just shouldn't
45:32be putting or really can't put in
45:34specific data types. So, if this is a
45:36string, that may be a character that you
45:38shouldn't have or can't have. That's
45:40going to mess up your data.ly in the
45:42address section, we have two issues
45:43here. We have messy structure and
45:45format. So, some are capitalized.
45:49So, so some of the data is capitalized
45:51versus not capitalized. We also have
45:53incomplete data. For example, we have
45:55Emily Renzelli Boulevard, but we don't
45:57have a number. We don't have a city. We
45:59don't have a state. They're just
46:00incomplete. And that makes it really
46:02difficult to work with that data. Now,
46:04here we have something called a data
46:05cleaning cycle. And I really like this
46:07visualization because it shows that data
46:10cleaning is not just a one-time thing.
46:12It is something that you have to
46:13continuously do. And I have and I have
46:16experienced this a lot over my years as
46:18a data analyst. Once you clean data
46:20isn't necessarily perfectly cleaned. You
46:22kind of have to go back and clean it as
46:24you go. And it's never really perfect.
46:26it's just really usable. This cycle
46:28typically starts at the very top. So we
46:30have importing data. This is just
46:32bringing in data from other systems,
46:34usually through some data collection
46:35system like a data pipeline that we
46:37talked about in the data collection
46:39lesson last time. Next we have merging
46:41data sets. This is combining multiple
46:43data sources into one data set. Next we
46:45have rebuilding missing data. This is
46:47where you're handling incomplete data or
46:49you're actually filling in that missing
46:51data. Next we have standardization and
46:52normalization. Standardization is where
46:54you're making sure that the data follows
46:56a consistent format. So all the date
46:58formats are the same. For example, if we
47:01go back to this one, we can make sure
47:02that the address is all standardized.
47:05They all have an address, a city, a
47:07state, and maybe a zip code if you need.
47:09They all have the same data. Next, we
47:11have normalization. This is a little bit
47:13different because you're adjusting data
47:15to a common scale without distorting the
47:17data itself. Next, we have dduplication.
47:20And this is where you may have
47:21duplicates of data in your columns. For
47:24example, maybe we have a 100 columns of
47:26Alex Freeberg and we don't need 100
47:28columns. It's all the exact same data.
47:30We would want to remove 99 of those to
47:32just keep the one that we actually need.
47:34Next, we have what they call
47:35verification and enrichment. This is
47:37really just data quality. You're going
47:38through and you're validating your data,
47:40making sure that your data is accurate.
47:42Then lastly, you're saving this clean
47:44data in the format that you need for
47:46your next processes, whether it's
47:47analyzing or you're using in some type
47:49of product. Now, one thing I want to
47:50mention on here is that this cycle is
47:53not perfect. In fact, I usually do
47:55dduplication earlier on within this
47:58cycle, but it really depends on the data
48:00itself. Sometimes you're importing,
48:02merging, rebuilding, and then going back
48:04and getting more data and then
48:05importing, merging, and rebuilding. So,
48:07this isn't a perfect cycle. This isn't
48:09exactly what you should do every time,
48:11but these are often the steps that I am
48:13taking when I'm cleaning data. Now, data
48:15cleaning is very specific to different
48:17tools, and I have a lot of different
48:19videos on how to exactly clean data with
48:21full projects on data cleaning in SQL
48:23and Tableau and PowerBI and Python and
48:26Excel. And so, if you want to learn
48:27hands-on how to actually clean data, go
48:30check out those videos cuz they are
48:31fantastic to learn how to actually clean
48:33data. Today, we are going to be starting
48:35our MySQL tutorial series.
48:43Now, the entire MySQL series will be
48:45broken up in three smaller series. We'll
48:47have our beginner, our intermediate, and
48:48our advanced. This lesson is the very
48:50first lesson in the beginner series
48:52where we're going to walk through all of
48:53the beginner or the basics of MySQL.
48:56Today, we're going to be setting
48:57everything up. So, we'll be installing
48:58MySQL and then creating our database
49:00that we'll use to actually learn MySQL.
49:02Now, before we get started, I wanted to
49:04let you know that I created three full
49:05MySQL courses over on
49:07analystbuilder.com. I built a crash
49:09course for MySQL for people who are
49:10going to be interviewing or taking
49:11technical interviews. I also created a
49:13full MySQL course that's going to cover
49:15everything from the basics all the way
49:17to the intermediate level. And then
49:18lastly, I created an advanced course
49:20that's going to teach you a lot of the
49:21more advanced things that an analyst
49:22would typically use. Those courses are
49:24going to go really in-depth and they're
49:25going to have a lot of practice
49:26questions along the way and we'll also
49:29have full guided projects in there as
49:30well. I'll have links in the description
49:32to all of those courses if you want to
49:33check those out. Now, without further
49:35ado, let's jump on my screen and install
49:36my SQL and create our database. All
49:38right, so let's get started by
49:40downloading MySQL. We're going to come
49:42right over here to
49:43dev.mmysql.com/downs/installer.
49:48And I will have that link in the
49:49description so you don't have to write
49:50all that out, but you should be seeing
49:52this page right here. Now, we have to
49:54select an operating system. I'm using a
49:56Windows machine. And if you aren't, if
49:59you're using Linux or Mac or something
50:00else, it should populate it for you. But
50:03if it doesn't, just select this dropown
50:04and select your operating system. Next,
50:06we have two different downloads. We can
50:08install the MySQL installer community or
50:11MySQL installer web community. This one
50:13is very small, but then you actually do
50:16have to download the installer. It just
50:17gets it from the web. This one I'm going
50:19to download the actual installer. It's
50:21larger, but this is the one I'm going to
50:23do. So, I'm going to go ahead and select
50:25download. It's going to ask me if I want
50:27to log in or create an account, and I
50:30don't. I'm going to say no thanks. Just
50:31start my download.
50:34I'm going to save this in this desktop
50:36folder. Doesn't really matter where you
50:37save it. We're going to save that. And
50:39it's going to download. It should be
50:41done in just a few seconds. I'm going to
50:43go ahead and click on it. And it's going
50:44to open it up when it's finished. And we
50:46should get the interface or the UI for
50:48the actual installation for MySQL. So,
50:51here is the MySQL installer. And first
50:54thing we need to do is choose a setup
50:56type. Now, we're going to keep the
50:57developer default unless you really know
51:00what you're doing. And you can select
51:02the server only, the client only, full,
51:04which is literally everything MySQL has
51:06to offer, or custom. So, we're going to
51:09keep this developer default, just
51:11installing the things that we kind of
51:12need. So, let's go ahead and select
51:14next. And for whatever reason on my
51:16computer, it's saying this path already
51:18exists. You probably won't get that, but
51:21I'm just going to go ahead and select
51:22next. And then I'll select yes. It keeps
51:24doing that. I can't explain why, but it
51:27keeps doing that for me even though I've
51:28deleted it from my computer completely.
51:29U but it just remembers it somewhere in
51:31its memory. Now the next thing we need
51:34is to check requirements. Now I just
51:37have this one. It says I need to
51:38download this uh Visual Studio. I'm not
51:41going to do that, but on your screen you
51:42may have multiple multiple requirements.
51:45Typically you're looking at something
51:47like this Microsoft Visual C++
51:50redistributable package. What you need
51:52to do is download this. All you have to
51:54do is click download. Once you download
51:56and install that on your computer and
51:58then we go back, all of those should be
52:00gone. That's the one that I see the most
52:02when I'm actually working with these
52:03requirements. I had to install it myself
52:05when I got this new laptop. So, go ahead
52:08and install that if you need to. But if
52:09yours looks like mine, we don't need
52:11this Visual Studio for what we're going
52:12to do. We're going to go ahead and
52:14select next. It is giving us a prompt
52:17that we haven't satisfied all the
52:18requirements, but that's okay. We're
52:19going to go ahead and select yes as
52:21well. Now we're ready to install all of
52:23these things. These are all things that
52:25MySQL wants you to install. The most
52:27important are the server and the
52:28workbench, but it does not hurt to have
52:30all these other things as well. Some of
52:32these connectors are also important. So,
52:34we're going to go ahead and execute.
52:36This will take just a few minutes. I'll
52:38skip ahead uh when they're all done, but
52:39this should take just a few minutes and
52:41then we'll continue on installing my
52:43SQL. So, everything just completed and
52:46now we're going to select next. And now
52:48we need to actually configure our
52:49product. Now really the only one that we
52:51actually need to configure is the
52:53server. The router says we need to
52:55configure it and the samples and
52:56examples say we need to configure it as
52:58well, but really it's just the server.
53:00Let's go ahead and select next. Now
53:02we're not going to change anything for
53:04this type and networking unless you know
53:06what you're doing with the port, the X
53:07protocol port. Uh we're not going to
53:09change any of this. We'll go ahead and
53:11select next. Next thing we need to do is
53:13select an authentication method. I'm
53:15going to be using a password. I'm not
53:17going to be using the legacy
53:18authentication method. So, I'm just
53:20going to go ahead and create a password.
53:22Now, [snorts] for you, and I keep
53:24getting this error, and I can't explain
53:25why right here for you, you should be
53:27creating a password at the bottom. It's
53:29remembering my password somehow, and I I
53:31really can't explain it, but I'm going
53:33to create my password or check my
53:34password. Uh, this is one that I already
53:37created uh before I deleted it off my
53:39computer, but it's still there. Um, so
53:41it's saying my password is still good,
53:43but if you need to, you should be
53:45entering a password and then confirming
53:46your password and saving it. And then
53:48you should also be checking it as well.
53:50And then we're going to configure this
53:51as a MySQL server as a Windows service.
53:53I'm going to keep that checked. Uh, and
53:55we're going to start the MySQL server at
53:57system startup. I like that
53:58automatically being there. I don't want
54:00to mess with that. So, I'm going to keep
54:01it as it has it. We're going to go ahead
54:03and select next. And the last thing we
54:05do is just need to execute this. And
54:07then everything we put in there is going
54:09to actually go. So, let's run this and
54:11execute it. And that just finished. So,
54:13let's go ahead and select finish. Now,
54:15it says configuration complete for the
54:17server, but we also need to configure
54:18these other two. Let's take a look at
54:20these really quickly. We're not going to
54:22do anything on this. It even says we
54:24really don't need to do this. We just
54:25need to click finish and configuration
54:28not needed. Next, we'll do samples and
54:30examples. And we can input our password.
54:34And all this is really going to do is
54:36put in some sample databases for us in
54:38our database, which if you want, you
54:40definitely can do that. Uh, I just
54:42connected. That worked. I'm going to hit
54:44next and execute. And it's basically
54:47just going to put in a database or two,
54:49some sample ones for you to look at. And
54:51the configuration is complete. You don't
54:53have to do that one, but we'll see that
54:55in just a second. We're going to select
54:57next. And now the installation is
54:59completely done. and we can start my SQL
55:02workbench after setup and start my SQL
55:04shell after setup. Now, I'm not going to
55:06do the shell, so I'm going to actually
55:07uncheck that and we're going to select
55:09finish.
55:11Now, my SQL just popped up for us, and
55:13this is exactly what you should be
55:15seeing. Now, there's a lot of things in
55:16my SQL to learn and know how to do.
55:19We're not going to be taking a look at
55:20all of that stuff today, but in future
55:22lessons, we'll walk through a lot of
55:24these different things that kind of
55:25correlate with different lessons or
55:26things that we're working on in my SQL.
55:28The first thing that we're going to
55:29click on is right over here. This is our
55:31local instance. This is local to just
55:34our machine. It's not a connection to,
55:36you know, some other database on the
55:37cloud or anything like that. It's just
55:39our local instance. We're going to go
55:41ahead and click on this. So, this is
55:43what you should be seeing right here.
55:44This is where we're going to actually
55:45write all of our SQL code. And I'll show
55:47you all this in just a second. But this
55:49is where we can actually create our
55:50database. And our database is going to
55:52go right over here on this lefth hand
55:53side. This silica one is actually a
55:56sample database. It has a bunch of
55:58tables and views store procedures
56:00functions has all these things in here
56:02if you want to go ahead and mess around
56:03with that. What we're about to do is
56:05create our own database that we're going
56:07to be using throughout this entire
56:09series both beginner, intermediate, and
56:11advanced. We'll use a lot of this and
56:13sometimes we'll import some other ones
56:15for different use cases, but this will
56:17serve for most of what we're trying to
56:19do throughout this entire series. Now,
56:21what I'm going to do is I'm going to go
56:22ahead and I'm going to say open a SQL
56:24script file in a new query tab. And
56:26right here, it opened up to a folder
56:28that I already created, this mysql
56:30beginner series folder. Within it, we
56:32have this right here, the parks and
56:35recreate_b.
56:37Now, in order to get this, you just have
56:38to go to the GitHub and download this
56:40file. That's all you have to do. We're
56:42then going to open this file. So, let's
56:44click on it. We're going to say open.
56:46And what you're now seeing is basically
56:48the query editor. This is where you can
56:50write your code. Now, we're not
56:52importing a database. We're actually
56:53creating it by running code. Now,
56:56because this is the first lesson in the
56:57beginner series, I'm going to assume
56:59that you don't know a ton about MySQL.
57:01Really, all this is doing is creating
57:03the database name and then we're
57:05inserting a few tables into that
57:08database and then we're inserting data
57:10into those tables. So, this is all of
57:12our data that will go into these tables
57:14that we create. We only have one, two,
57:18three different tables that we're going
57:19to be using. So, all you have to do to
57:21run this is click this lightning button
57:23right up here. We're going to go ahead
57:25and execute this. If we come down to the
57:28bottom and we pull this up, this is our
57:30output. This says six rows affected. And
57:34we have a bunch of other things like
57:35create table, create table, insert,
57:37insert, create table, insert into. These
57:40things are all working perfectly. So now
57:42if we go ahead and click refresh in our
57:45schemas with this refresh button right
57:47here, this parks and recreation table is
57:50populated. If we go under the tables, we
57:52see all of these things. So, now that
57:54we've actually created our database and
57:56our tables, that's really all we were
57:58trying to do in this lesson. But I just
58:00want to open up a table really quickly,
58:02show you what it looks like, show you
58:04how we can run code, and then in the
58:06next lesson, we'll start actually
58:07learning how to query this data. So,
58:10let's go up to employee demographics.
58:12We're going to rightclick and select
58:14rows limit 1000. This is going to open
58:16up a new window right up here and it's
58:19going to say select everything from this
58:22database dot this table employee
58:25demographics and it ends with a
58:27semicolon. Now right down here we have
58:29this output window. This is the actual
58:32data that sits in our table. We have
58:35columns right here. So employee ID,
58:38first name, last name, age, gender, and
58:41birth date. And then here are all of our
58:43employees on each row. So these are all
58:45separate rows. We have Leslie Nope, Tom
58:47Havford, and it goes on and on. So this
58:49is all of our data. The most important
58:51things to know when we're actually
58:52working with this, and I'm going to zoom
58:54in, is if we hover over this query right
58:57here, and we run it, we can select this
58:59execute, which is this lightning bolt
59:01with this I, we're going to execute
59:04this, and it's going to run this because
59:05we're highlighted over it. Now, if we
59:08have two queries, let's say this one
59:10right here, but let's change it to
59:12employee salary. We'll do underscore
59:16salary. Let's say we want to query this
59:18table. So now if we highlight over this
59:21and we go up and select the lightning
59:22bolt with the I, now we're looking at a
59:25different table. But if we select, even
59:28if we're hovering over this, if we
59:30select this button, we're going to
59:32execute everything in this editor
59:34window. So let's run this. And now you
59:37can see at the bottom we have two
59:38outputs, the employee demographics and
59:41the employee salary. So this button is
59:43going to run everything in this editor
59:45window. Whereas if we select this
59:47lightning bolt with the I, we're doing
59:49everything that's just under where we
59:51have the cursor, where we have it
59:53highlighted. The very last thing that I
59:55want to mention is that right over here,
59:56you may have this up and you probably
59:59don't want that. We're not going to do
1:00:00any SQL editions in this series. You can
1:00:03get rid of that by clicking this button
1:00:04right here. So, starting in the next
1:00:06lesson, we'll have everything ready to
1:00:08go and we will start learning the basics
1:00:09of my SQL. I hope you're able to follow
1:00:12along, get everything set up how we have
1:00:14it on this screen. I am super excited
1:00:16about this series. I just I love SQL.
1:00:18So, I'm really, really, really excited
1:00:20to get started on this with you guys. I
1:00:22will see you guys in the next lesson.
1:00:24[music]
1:00:36Hello everybody. In this lesson, we're
1:00:38going to be learning about the select
1:00:39statement in MySQL. The select statement
1:00:42is used to work with columns and specify
1:00:44what columns you want to see in your
1:00:45output. The first thing that we need to
1:00:47do is open up a tab or an editor window.
1:00:50So, let's come right up here to the
1:00:51left-hand side and we're going to create
1:00:53a new tab. And I'm going to zoom in just
1:00:56a little. Now, what we need to do is we
1:00:59need to select the actual table that
1:01:01we're going to be querying off of. If
1:01:04you remember from the very first lesson
1:01:05when we set everything up, we came over
1:01:07here and we rightclicked and did select
1:01:10rows limit 1000. We're not going to do
1:01:12that. We're going to actually write it
1:01:13out. So, what we need to do to select
1:01:16that table, the employee demographics
1:01:18table, is we need to select
1:01:21everything. That's what this star means.
1:01:23The star means everything. All tables,
1:01:26all rows. Now, we do have a limit on
1:01:28here. We have a limit to 1,000 rows. So,
1:01:31if we had a table that had 50,000 rows,
1:01:33this limiter would be an issue. It would
1:01:36still limit it to 1,000 rows. We would
1:01:38have to change that to 2,000, 5,000,
1:01:41probably all the way up to 50,000 if we
1:01:43wanted to view everything. If we had,
1:01:45say, a million rows, we would need to
1:01:47come up here and say, don't limit, and
1:01:49it would give us a million rows. The
1:01:51reason they do this is mostly to keep
1:01:53the processing time low. If you have a
1:01:56million rows, it's going to take a long
1:01:57time for the output to actually appear.
1:02:00So, let's come right back here. The next
1:02:02thing that we need to do is we need to
1:02:03say select everything. And now we need
1:02:05to say where we're selecting it from.
1:02:08So, we're going to come right down here
1:02:09and we're going to say from. And now we
1:02:12need to specify what table. And we're
1:02:14going to say employee
1:02:17demographics. And at the end, we need a
1:02:20semicolon. Now, why do we need a
1:02:22semicolon? This is going to tell my SQL
1:02:25that this is the end of this query. So,
1:02:28if we write another one down here, which
1:02:30we will in just a second, it'll be able
1:02:32to distinguish between the two queries.
1:02:34We're going to go ahead and we're going
1:02:35to run this and we'll just use this
1:02:37execute right here instead of this one.
1:02:40And there we have our entire table. So,
1:02:42we were able to get our table. Now,
1:02:44there is one thing that is potentially
1:02:46wrong depending on what you're using it
1:02:48for. But what we didn't do is we did not
1:02:51specify the actual database before it.
1:02:54We only specified the table. And this
1:02:56works perfectly fine because if you look
1:02:59over here on this left-h hand side, we
1:03:00have parks and recreation. It's in
1:03:03black. It's bold. That means that we're
1:03:05hitting off of this database. What's
1:03:08going to happen though if we come down
1:03:09here to the CIS database and we double
1:03:10click on it? Now this database is
1:03:13highlighted. So now when we're selecting
1:03:16this table, we're trying to select this
1:03:18table from the CIS database. Let's go
1:03:20ahead and try this.
1:03:23If you notice, we have no output. Let's
1:03:25come right down here and pull this up.
1:03:28It's going to say employee cis.mp
1:03:30employee demographics doesn't exist. So
1:03:34it's assuming that we're highlighting
1:03:36this CIS database. That means we're
1:03:38trying to pull from that database. Now
1:03:40we can still have this highlighted and
1:03:43still select the correct database by
1:03:45saying parks_and_recreation
1:03:51and let me spell that right dot. So now
1:03:54we're selecting everything from parks
1:03:56and recreation employee demographics. If
1:03:59we run this, we do get the correct
1:04:02output. That's just something to
1:04:04consider, especially when you're working
1:04:06with a lot of databases and a lot of
1:04:08tables. It's usually best practice to
1:04:11actually put the database in front of
1:04:14the table name. Although throughout this
1:04:16lesson, we probably won't be doing that
1:04:17every time since we're only going to be
1:04:19using this parks and recreation
1:04:20database. Let's go ahead and double
1:04:22click this so we have this highlighted
1:04:24again. And let's click all of this.
1:04:27Let's copy all of this. We're going to
1:04:29come down just a little bit right here.
1:04:31Now, so far we've only selected
1:04:33everything, but we don't have to do
1:04:35that. We can actually just select one
1:04:37column if we would like to. For example,
1:04:39if we got rid of that star, we say first
1:04:42name. We're selecting the first name
1:04:45column from this table. If we highlight
1:04:48this query and we hit the execute button
1:04:51with the I.
1:04:54Now, we are only going to return in our
1:04:56output all of the first names. And we
1:04:58can add a lot more. Let's actually look
1:05:00at all these. We can separate multiple
1:05:03columns with a comma. So we can do first
1:05:05name, last name, and then we could do
1:05:09birth date. So now we have three
1:05:12separate columns. Let's go ahead and run
1:05:14this. And now we have first name, last
1:05:17name, and birth date in our output. Now
1:05:19the way we just wrote it is all on one
1:05:21line. And that's perfectly acceptable
1:05:23because my SQL is going to read it the
1:05:25exact same as if we did it in a
1:05:27different format as long as it's still
1:05:29in this order. But sometimes you'll see
1:05:32it like this where it's select first
1:05:34name, last name, comma, birth date all
1:05:37on different rows. Now, there's a lot of
1:05:39different use cases for this or reasons
1:05:41for this, but it typically can be easier
1:05:44to read. Also, if you're doing any type
1:05:46of functions or calculations in the
1:05:48select statement, it's easier to
1:05:50separate those out on its individual
1:05:53row. Now, again, we won't always be
1:05:54doing this, but it does help sometimes
1:05:57if you're doing that. It just makes it
1:05:58easier to visualize. For example, if we
1:06:01added the age. So, let's add age in
1:06:03here. Let's run this. Let's say we were
1:06:06doing a calculation where we wanted to
1:06:07add, you know, 10 years to their age.
1:06:10So, we'll say age and we'll actually
1:06:12create a new row for this or new column.
1:06:15We'll do age + 10. So, now we can easily
1:06:19see that we're doing plus 10 here. And
1:06:20this is another thing that you can do in
1:06:22the select statement, things like
1:06:24calculations. So, if we go up here and
1:06:26we run this, we'll now have an age
1:06:29column, but we'll also have an age + 10
1:06:32column where it just adds 10 to the age.
1:06:34And we can at least visualize and really
1:06:36easily see this when we're doing these
1:06:38calculations. Now, something really
1:06:40important to know about any type of
1:06:42calculations, any math within my SQL is
1:06:45that it follows the rules of PEMDOS.
1:06:48Now, PEMDOSS is written like this. It's
1:06:50PMDES.
1:06:52Now, what I just did right here with
1:06:54this pound or this hashtag is actually
1:06:56create a comment. So, this code isn't
1:06:58going to actually run, but it's just for
1:07:00note-taking or seeing things in your
1:07:02actual editor window. I'll come back to
1:07:05comments in just a second, but just
1:07:06wanted to explain what that was. Now,
1:07:08what PEMDOS is is the order of
1:07:11operations for arithmetic or math within
1:07:15my SQL. This stands for parentheses,
1:07:18exponent, multiplication, division,
1:07:20addition, and subtraction. So this is
1:07:22the order that these calculations are
1:07:24going to run in the execution engine
1:07:27that my SQL has. So if I do age + 10,
1:07:31and we'll put that all in parenthesis,
1:07:33and then we come over here and we add*
1:07:3510. So we're doing plus 10 here and then
1:07:37a time 10 here. What's going to actually
1:07:40happen is it's going to say age + 10. So
1:07:4344 + 10= 54. Then we're multiplying time
1:07:4710. The parenthesis executes first
1:07:50because it comes first in this order.
1:07:52Parenthesis. Multiplication comes next
1:07:54because it's this one. And then anything
1:07:56else after that if we did, you know,
1:07:58plus 10 could run this. And you'll
1:08:02notice that it still follows the logic.
1:08:0410 was just added at the very end to all
1:08:06of these outputs. Now let's go right
1:08:08back up here. Let's select everything
1:08:09again from this table. Let's pull up
1:08:12this table. so we can see it a little
1:08:13better. And let's go down because the
1:08:17last thing that I want to show you is
1:08:19something called distinct. Now, this is
1:08:21really, really useful and you use this a
1:08:23lot in my SQL. What distinct is going to
1:08:26do is it's going to select only the
1:08:28unique values within a column. Let's go
1:08:31ahead and copy this employee
1:08:33demographics. Bring it right down here.
1:08:36Let's say select and let's do first
1:08:40name. So now we're just selecting the
1:08:42first name. Let's come right down here.
1:08:48There we go. So now we're selecting just
1:08:50the first name from this column. Now
1:08:52these are all unique values. So if we
1:08:55come right here and we say distinct,
1:08:58nothing should happen to this table cuz
1:08:59these are all unique values. Let's go
1:09:01ahead and run this.
1:09:03As you can see, the output looks exactly
1:09:05the same. But what if we were do
1:09:08something like gender? So let's come
1:09:11here. Let's do gender. Let's run this.
1:09:14Keeps going down. I don't know why it's
1:09:15doing that. Um but now we have male and
1:09:18female. Now these are not all unique. We
1:09:20have female, female, female, and female.
1:09:23And the rest are males. So there's only
1:09:25two unique values here. So if we come
1:09:27right here and we say distinct gender,
1:09:30now there should only be two in the
1:09:32output, male and female. Let's go ahead
1:09:34and run this.
1:09:36So now we get male and female in our
1:09:38output. Now this works perfectly in one
1:09:41column but what happens if we have two
1:09:42columns. So let's do first name, gender.
1:09:47Let's go and run this.
1:09:49Now the combination of first name and
1:09:52gender are no longer unique. Now Leslie
1:09:56and female are being grouped together
1:09:58and it's taking the distinct between
1:10:00both of these columns. So when we're
1:10:02only working with gender, it's only
1:10:04looking at this one column for both male
1:10:06and female. it reduces it down to the
1:10:08only two unique values. But because we
1:10:11add the first name, all of these values
1:10:13are unique. So therefore, the name plus
1:10:16the gender combination is always going
1:10:18to be unique. The very last thing that I
1:10:20want to show you in this lesson doesn't
1:10:22actually pertain to the select
1:10:23statement, but I want to save this code.
1:10:25Let's say we wanted to update this or
1:10:27upload this into our GitHub or save this
1:10:30and send it to somebody. We can do that.
1:10:32We can save it by clicking this save
1:10:34button right here. I'm going to go ahead
1:10:36and click this. And now we're in our
1:10:37MySQL beginner series folder. I'm just
1:10:40going to save this. And I can save this
1:10:41as anything I want. So I'm going to say
1:10:42two dot select statement
1:10:46tutorial. So now when I save this,
1:10:49you'll notice that the name gets changed
1:10:51up here to two select statement
1:10:53tutorial. Let's exit out of this. I'm
1:10:55going to open up and now I'm going to
1:10:57come here to the select statement
1:10:58tutorial. I'm going to open it. And now
1:11:01I have our code again exactly as we had
1:11:04it written before. I just wanted to show
1:11:05that to you in case you wanted to save
1:11:07your code as you go throughout this
1:11:08series because that's usually what I do
1:11:10when I'm working with this stuff or
1:11:12learning these things. Like to save my
1:11:13code as I go along. So with that being
1:11:16said, that is the end of the select
1:11:18statement. In the next lesson, we're
1:11:20going to be learning about the where
1:11:21statement where we can actually filter
1:11:22our data.
1:11:26[music]
1:11:36Hello everybody. In this lesson, we're
1:11:38going to be taking a look at the wear
1:11:39clause. The wear clause is used to help
1:11:41filter our records or our rows of data,
1:11:44whereas the select statement is used to
1:11:46help filter or select our actual
1:11:49columns. So, when we're using the wear
1:11:51clause, we're only going to return the
1:11:52rows that fulfill a specific condition.
1:11:55Let's take a look at exactly how this
1:11:57works. Let's say we come right up here.
1:11:59We're going to say where. And let's go
1:12:02down with that one. Let's say where. And
1:12:04now we need to specify what column we're
1:12:06about to create this condition for. So
1:12:08we're going to say first name. So we're
1:12:11saying where the first name we'll say is
1:12:14equal to and let's do quotes. And let's
1:12:16say Leslie. So we're saying the first
1:12:19name has to be equal to this value right
1:12:22here, which is Leslie for Leslie. Nope.
1:12:25If we run this, there's only going to be
1:12:28one row that's returned because Leslie
1:12:30is the only Leslie in this entire table.
1:12:33Now, we just used an equal sign, and
1:12:36that's actually called a comparison
1:12:37operator. And there's a few other
1:12:39comparison operators that you can use.
1:12:42Let's take a look at some of these other
1:12:43ones. Let's pull this down right down
1:12:46here. And let's actually highlight the
1:12:49select from, and we're going to run it
1:12:51with this one right here. It's going to
1:12:53only select everything from the whole
1:12:55table. So, we didn't select that wear
1:12:57clause. Let's go right down here and
1:12:59let's look at this salary field. So, I'm
1:13:03going to say where the salary and I'm
1:13:05going to do a different comparison
1:13:07operator called greater than. So, when
1:13:09the salary is greater than 50,000. Now,
1:13:13one thing I want to note before we
1:13:14actually run this is that right down
1:13:16here we have Tom Havford who makes
1:13:18exactly 50,000. And I think there's one
1:13:21more, Jerry Gurgich, which also makes
1:13:24exactly 50,000. If we run this, you'll
1:13:28notice that both Tom and Jerry are not
1:13:30in this output. But in the salary field,
1:13:32everything is greater than 50,000. The
1:13:35reason for that is that Tom and Jerry
1:13:38made exactly 50,000. What we're saying
1:13:41right here is where the salary is only
1:13:43greater than. If we want to include Tom
1:13:46and Jerry, we have to say greater than
1:13:48or equal to. And now we'll select 50,000
1:13:51or above. Whereas right here, before
1:13:54when we were doing just this, it was
1:13:56greater than 50. It didn't include the
1:13:5850,000. Let's go ahead and include it
1:14:00and run this. And now you'll notice that
1:14:03Tom and Jerry were both included because
1:14:06they had exactly 50,000 and we said
1:14:08greater than or equal to. Now we can do
1:14:11the exact same thing but with less than.
1:14:14So we have less than 50,000.
1:14:17And now we only have two people who make
1:14:19less than 50,000. That's April and Andy.
1:14:22And if we say less than or equal to, and
1:14:25we run that, now we include both Tom and
1:14:27Jerry who make exactly 50,000. So it's
1:14:30less than or equal to $50,000. Now what
1:14:33we're going to do is head on over to a
1:14:36different table. We're going to do the
1:14:38demographics table.
1:14:41Make sure I spell that right. And let's
1:14:43add our semicolon. Let's run this. And
1:14:46what we want to look at is the gender
1:14:49really quick. So we're going to say
1:14:51where the gender is equal to, we'll do
1:14:55in quotes female. And if we run this, we
1:15:00get all the genders that are equal to
1:15:02female. But we do have something called
1:15:05the not equal to. And it looks like
1:15:07this. It's an exclamation point and an
1:15:10equal sign. This is going to say where
1:15:12the gender is not equal to female. So if
1:15:14we run this, you'll notice that the
1:15:17gender is all male. Now, now so far
1:15:19we've worked with things like integers,
1:15:21which are numbers. We've worked with
1:15:23characters or strings like names. But
1:15:27there's a different type of data type as
1:15:28well in here. We have a date column for
1:15:31these birth dates. Now, in the wear
1:15:33clause, we can also filter on birth
1:15:35dates. Let's come over here and we'll
1:15:37say birth
1:15:39date. Let's say it's greater than and
1:15:41within quotes we'll say 1985-01-01.
1:15:47This is kind of the standard default
1:15:49date format within my SQL which is year,
1:15:52month and day. If we go ahead and run
1:15:55this, we can also take all the people
1:15:57who are greater than or born greater
1:15:59than 1985. So all of these dates are
1:16:01greater than 1985. Now the next thing
1:16:04that I want to take a look at is logical
1:16:06operators in the wear clause. So logical
1:16:09operators are things like and or and
1:16:13not. Now these are called and let's add
1:16:16this logical operators. So logical
1:16:19operators allow us to have different
1:16:22logic. And let's take a look at how this
1:16:24works exactly. Let's copy this down
1:16:25because we already have this one written
1:16:27out. We're saying where the birth date
1:16:28is greater than 1985.
1:16:31We can also say where the gender is
1:16:33equal to male. So we can say and the
1:16:37gender is equal and then we'll say male.
1:16:40So we're adding a different complexity
1:16:42or an additional conditional statement
1:16:45within our wear clause. Let's go ahead
1:16:47and run this. So now we're only
1:16:49selecting birth dates that are greater
1:16:51than 1985 and where the gender is equal
1:16:54to male. Only the rows that fulfill both
1:16:57of those are returned. Now the and says
1:17:00both this and this have to be true. But
1:17:05we could change this and we could say
1:17:07or. What this means is is either this
1:17:10one has to be true or this one has to be
1:17:13true in order for it to be returned. So
1:17:15let's go ahead and run this. You'll
1:17:17notice that Jerry Girkitch was born much
1:17:20before 1985. But since he has a male
1:17:23gender, he is in our output. And we
1:17:26could also use the not operator by
1:17:28saying or not gender equal to male. So
1:17:32now what this is saying is the birthday
1:17:34could be greater than 1985 or it could
1:17:38not be equal to male which is female. So
1:17:40if we look at Leslie nope she was born
1:17:43before 1985 but because she is female
1:17:46she is in the output. Now like we talked
1:17:48about in the last lesson there is
1:17:50something called PEMDOS and that
1:17:52actually applies to these logical
1:17:53operators as well. So, if we run this
1:17:56entire table, let's go ahead and run
1:17:58this.
1:18:00If we're looking at this entire table,
1:18:02let's say we want to get someone very,
1:18:03very specific. Let's say we're going to
1:18:06do uh where the first underscore name is
1:18:10equal to Leslie
1:18:13and their age has to be equal to 44.
1:18:18That's extremely specific. And we can
1:18:19actually just do it like this. We don't
1:18:21need quotes um for integers. We could
1:18:24just do the number if we'd like to. This
1:18:26is very specific. This is only one
1:18:27person. But if we put this in
1:18:29parentheses,
1:18:32we can add an or over here. We can say
1:18:35or the age is greater than. Let's just
1:18:38do 55. Let's go ahead and run this and
1:18:40then we'll take a look at it. So within
1:18:42these parentheses, we have an and
1:18:44operator. What that means is both this
1:18:47condition has to be met and this
1:18:49condition has to be met. And that's only
1:18:50one person. That's Leslie note. But then
1:18:52outside of these parenthesis, we have
1:18:55another conditional statement or the age
1:18:57is greater than 55. So what we're saying
1:18:59within these parenthesis is that this is
1:19:01an isolated conditional statement.
1:19:03Within these parenthesis, if this is
1:19:06true, then in our output, it'll be
1:19:08returned. But then we have an or
1:19:10condition which says or someone with the
1:19:12age of greater than 55 can also be in
1:19:14the output. So these parentheses can be
1:19:16really helpful when you're actually
1:19:18using it in the wear clause with these
1:19:20and, ors, and nots. Now, I want to take
1:19:22a look at just one more thing. And let's
1:19:25bring this down here.
1:19:28And let's get rid of this entire thing.
1:19:33Now, the last thing that we're going to
1:19:34take a look at is a like statement. Now,
1:19:38the like statement is super unique
1:19:40because we can look for specific
1:19:42patterns. We're not necessarily looking
1:19:44for an exact match. Like here if we said
1:19:48where first name is equal to Jerry. If
1:19:55we're looking for Jerry it has to be
1:19:57exactly Jerry. But if we take this out
1:20:00say J and then we run it we get no
1:20:03output. It has to be an exact match. But
1:20:06here's where the like statement comes in
1:20:08because we can actually say like jer and
1:20:12we can add two special sequences or
1:20:15special characters within our like
1:20:18statement. So those special characters
1:20:21are the percent sign and the underscore.
1:20:26The percent sign means anything and the
1:20:28underscore means a specific value. Let's
1:20:31see how that actually works. So what
1:20:33we're going to do is we're going to say
1:20:34like jer percent sign. That's the first
1:20:38one in this like statement. What this
1:20:40says is the first name is like starting
1:20:43with jer but then has anything after it.
1:20:47Doesn't matter what it is. As long as it
1:20:49has jer at the very beginning it will be
1:20:52returned. Let's go ahead and run this.
1:20:54Now the only person who starts with jer
1:20:56is Jerry. But what if I took the j out
1:21:00of here? Now it's saying it starts with
1:21:02eer and that's not anybody. What we can
1:21:06do is we can add another percent at the
1:21:08beginning. This is going to say anything
1:21:10comes before anything comes after. All
1:21:13we're looking for is e rer somewhere in
1:21:16their name. Let's go ahead and run this.
1:21:18There still is only one person and
1:21:20that's Jerry. Now let's come up here and
1:21:23let's get rid of this and let's say
1:21:24we're looking for everyone's name who
1:21:26starts with a. We can do that really
1:21:28easily by saying a percent sign. All
1:21:31that says is it starts with a. We don't
1:21:33have a percent sign before it, which
1:21:35would say this string just has to have
1:21:37an a somewhere in it. If we have it like
1:21:40this, this means an a has to come at the
1:21:42beginning. Let's go and run this. In our
1:21:45output, we have April and an Andy. Now,
1:21:48let's take a look at the underscore. If
1:21:51we get rid of this percent sign and we
1:21:54do two underscores one, two, this is
1:21:58going to say it starts with an A and
1:22:00then it has two characters after it. No
1:22:03more, no less. So if we run this, an is
1:22:07going to be the only person who's
1:22:08returned cuz she has an A and then two
1:22:10characters after it. Now if we want
1:22:12Andy, we can specify that by doing
1:22:15another underscore. That's 1 2 3.
1:22:18And now Andy is the only one in our
1:22:21output. Now there was also April in
1:22:23there, but she had more than three
1:22:24characters. But we can actually get her
1:22:27in our output by doing a percent sign.
1:22:29So we can combine both the underscore
1:22:32and the percent sign. And this is going
1:22:34to say it starts with an a has 1 2 3
1:22:38characters and then it can have anything
1:22:41after that. So it just has to have at
1:22:43least an A and have one two three
1:22:45characters after it. So let's run it.
1:22:48Now you can see April comes into here
1:22:50because she does have a the P, R, and I
1:22:54are the three next characters, but then
1:22:56we have a percent sign that allows that
1:22:58L to be in the output as well. Now, we
1:23:01don't just have to do this with strings
1:23:03or text like April and Andy. We could
1:23:05also do this with birth dates. For
1:23:06example, Andy's birth date is 1989. We
1:23:09could say where the birth date is like.
1:23:14Let's say we want to look at everyone
1:23:16who is 1989
1:23:18or born in 1989. Let's go and run this.
1:23:21And Andy is the only person born in
1:23:231989. But again, we looked at the year
1:23:26at the very beginning. So that is how
1:23:28the like statement works. It looks for a
1:23:31specific sequence within that column
1:23:33that you can search for. So it doesn't
1:23:35have to be an exact match as long as it
1:23:37has that specified sequence that you've
1:23:39put in there anywhere within that cell
1:23:41or that column. So that is everything
1:23:43that we're going to look at for the wear
1:23:45clause. In the next lesson, we're going
1:23:46to take a look at the group by and the
1:23:48order by within my SQL.
1:24:02Hello everybody. In this lesson, we're
1:24:04going to be taking a look at group by
1:24:06and order by in my SQL. Now, when you
1:24:08use the group by clause in MySQL, it's
1:24:11going to group together rows that have
1:24:12the same values in the specified column
1:24:15or columns that you're actually grouping
1:24:17on. Once you group those rows together,
1:24:19you can run something called an
1:24:20aggregate function on those rows. Let's
1:24:23see how this actually works. Let's go
1:24:25ahead and copy this right here. We'll
1:24:27bring that down. And let me go back up
1:24:30one. Let's go ahead and write gender
1:24:33right here. Now, we want to group on
1:24:37this gender column. And we're going to
1:24:39say group by gender. Let's go ahead and
1:24:44run this. We'll see what we get. And so,
1:24:46we have male and female. Now, we could
1:24:50get the exact same output by saying
1:24:52select distinct gender from this table.
1:24:55What is group by doing that the gender
1:24:58actually isn't doing? Well, it's
1:24:59actually rolling up all of these values
1:25:02into these rows. So later when we run
1:25:05aggregate functions like average, min,
1:25:07max, we'll do it based off of these rows
1:25:10and all those rows are rolled up into
1:25:12these two rows. And we'll see that in a
1:25:14little bit. Now, what if I was to come
1:25:16up here and in this demographics, we
1:25:18have a first name. What would happen if
1:25:21I'm selecting the first name, but I'm
1:25:23grouping by the gender? Let's go ahead
1:25:25and run this.
1:25:27If we come right down here, we pull this
1:25:29up. You can see that the select list is
1:25:32not in group by clause and contains
1:25:34non-aggregated columns. What this means
1:25:37is that when you are selecting a column,
1:25:39if it's not an aggregated column like
1:25:41say average of something, if we're not
1:25:44using the aggregate functions in the
1:25:45select statement, it has to be in the
1:25:47group by. These have to match. So this
1:25:50gender has to match this group by if
1:25:52we're not performing an aggregate
1:25:54function on it. Let's go ahead and run
1:25:56this. And now it works properly. Now
1:26:00let's go back up. Let's run this query
1:26:02because I want to select everything
1:26:03again. But let's say we wanted to take a
1:26:06look at the average ages for gender. So
1:26:09what we're going to do is we're
1:26:10selecting gender. We're also grouping by
1:26:12gender. But what we're going to do is
1:26:14add a comma and we'll say the average.
1:26:17That's avg. That stands for average. And
1:26:19then we're going to put in here age. So
1:26:22now this right here is an aggregate
1:26:24function. This does not need to go in
1:26:26the group by. We're just grouping on the
1:26:28gender and then we're performing this
1:26:31aggregate function or kind of a
1:26:32calculation based off of those grouped
1:26:35rows for gender. So let's go ahead and
1:26:37run this and take a look at the output.
1:26:39So what this is telling me is that for
1:26:41the males, all of the male rows that
1:26:43were grouped, the average age is 41,
1:26:47let's say three, and for female, the
1:26:50average age is 38.5.
1:26:53So, super quickly, you can tell that the
1:26:55average age of females is lower than the
1:26:57average age of males. Now, we'll take a
1:26:59look at aggregate functions more in just
1:27:01a little bit. Let's actually go to a
1:27:04different table. Let's come right down
1:27:06here. We're going to go to the salary
1:27:08table and
1:27:11just select everything for now.
1:27:14Let's go ahead and run this. Now, what
1:27:16we're going to actually be grouping on
1:27:18is this occupation right here. Now,
1:27:20there's a lot of unique values. It's um
1:27:23not as distinct as the gender which only
1:27:25had two values. You'll notice we do have
1:27:27a few that are the same. We have ones
1:27:29like office manager. So when we come up
1:27:31here say occupation.
1:27:34And of course we need to group by the
1:27:36occupation as well. Now let's run this.
1:27:39And you'll notice that office manager
1:27:41only has one row. Let's say we also want
1:27:44to group on the salary. Let's say
1:27:46salary. Now we can group on multiple. So
1:27:50we're going to say salary like this. So
1:27:53we're grouping on the occupation as well
1:27:55as the salary. Now let's run this.
1:27:59You'll notice that we have two rows for
1:28:01office manager. Now this is because this
1:28:03salary and this salary for those two
1:28:06employees are different. We have 50,000
1:28:08and 60,000. For this I just wanted to
1:28:11demonstrate that if these had both been
1:28:1250,000 there would only be one row.
1:28:15Office manager 50,000. But because this
1:28:17is a unique value different than 50,000,
1:28:20they have their own individual rows
1:28:22which we would then perform our
1:28:24aggregate calculations on. Let's go and
1:28:26get rid of that because we will not be
1:28:28using that anymore. I just wanted to
1:28:29demonstrate it really quickly. So before
1:28:31we were looking at gender and average
1:28:33age and we're also grouping on the
1:28:35gender. We can perform other aggregate
1:28:37functions as well. Let's take a look at
1:28:39some of those. We could look at the max
1:28:43age as well. The max is going to show us
1:28:46the highest value within each of those
1:28:48groupings. So we have a male and female.
1:28:51The max age for those for the male is 61
1:28:54and the highest age for the female is
1:28:5646. We can do the exact same thing
1:28:58except we can say min or the exact
1:29:01opposite thing. We can say the minimum
1:29:03age. So this is going to be the lowest
1:29:05for both the male and the female. Go and
1:29:07run this. Now we have female and male
1:29:10and the minimum age is 29 and 34. And
1:29:13there is one last one that I want to
1:29:15show you which is count. We're going to
1:29:17do count. Now count is going to count
1:29:20the actual rows within this age column.
1:29:24So if we run this, you'll see that we
1:29:27have four females for count and we have
1:29:29seven males. It's just telling us a
1:29:31count of how many values is in this
1:29:34column when we're actually grouping on
1:29:36the gender. So that's how we can use
1:29:38group by to actually roll up and group
1:29:42all of these similar values within a
1:29:43column or columns and perform our
1:29:46aggregate functions on them. Now let's
1:29:48come down here and what we're going to
1:29:50take a look at is order by. So we're
1:29:52going to say order by. Now let's
1:29:55actually pull in this demographics table
1:30:00right here. We're just going to say
1:30:02select everything
1:30:05and run this really quickly after we add
1:30:07a semicolon. So order by order by is
1:30:12going to actually sort the result set in
1:30:14either ascending or descending order.
1:30:17Let's take a look at how this works. At
1:30:19the very end, we could say order by and
1:30:23we could order by the first underscore
1:30:26name. So, we're going to take this
1:30:28column and we're going to order all of
1:30:30our rows based off of this one column.
1:30:32Let's go ahead and run this. So, it's
1:30:34going to do it based off ascending
1:30:36order, which means smallest to largest.
1:30:38Now, this is a text column or a
1:30:40character column. So, we do it A to Z.
1:30:43So, Andy and April all the way down to
1:30:46Tom. Now, by default, this is in ASC
1:30:50order, ascending order. And if we run
1:30:52this, it's going to be the exact same
1:30:53output. But we can change this to do it
1:30:56the opposite, highest to lowest or Z to
1:30:58A by doing descending. So now if we run
1:31:02this, you'll see that goes Tom all the
1:31:04way down to Andy. Now let's take a look
1:31:07at ordering on something like gender and
1:31:10age because we can do both at the same
1:31:12time. So let's order by the gender
1:31:14first. Let's go ahead and run this.
1:31:18And you'll see that all the females are
1:31:20grouped together and then all the males
1:31:22are grouped together because that's just
1:31:24the order in which it is. But we can do
1:31:26an additional column. We can also do it
1:31:28based off of the age. Let's go ahead and
1:31:30run this.
1:31:32So now within the female since that came
1:31:35first in our order by, we're ordering by
1:31:38the gender and then we're also ordering
1:31:40by the age after we've ordered by the
1:31:43gender. So now it's 29 all the way up to
1:31:4546. Then 34 for males all the way up to
1:31:4861. Now we can change this just for the
1:31:52age. Let's say we want to do age
1:31:54descending. So gender will stay the same
1:31:56in ascending order, but now age will be
1:31:59in descending order. Let's go ahead and
1:32:01run this.
1:32:02Now female and male stayed the same, but
1:32:04now it starts at the highest down to the
1:32:06lowest. Now this is something that I
1:32:08would absolutely do in real life except
1:32:12sometimes you can make mistakes and
1:32:13sometimes you do the wrong column first.
1:32:15Let's do age and then we'll do gender.
1:32:19Now, if we run this, the gender is not
1:32:22going to be used at all. And this is
1:32:25because there are no unique values that
1:32:27are going to be on the same row. So,
1:32:28notice all these values are completely
1:32:31unique. So, the gender never is actually
1:32:34used to order anything on because if
1:32:36there were things like 34, 34, 34, 34,
1:32:39these would be ordered based off of the
1:32:42gender. But since there's no unique
1:32:44fields, this is really pretty useless.
1:32:46That's why the order of the order by or
1:32:49the columns that you place in the order
1:32:51by are actually quite important. Now,
1:32:52the last thing that I want to show you,
1:32:54and I'll just go back to gender and age,
1:32:56is that you don't actually have to use
1:32:59the column names. We can use the column
1:33:02positions. Now, I will preface this by
1:33:04saying I don't recommend doing this, but
1:33:07I sometimes do it in shorthand for just
1:33:09a quick query. um if I know the column
1:33:12position and I don't want to write out
1:33:14the whole name. So sometimes I do it
1:33:16although it's not best practice but
1:33:17let's take a look at it. So gender is
1:33:20the 1 2 3 4 5th column. So I'm going to
1:33:24replace this with five and age is the 1
1:33:272 3 4 column. So these are the positions
1:33:31of the fields but not the names of them.
1:33:33If we run it, we're going to get the
1:33:35exact same output because these
1:33:37represent these columns appropriately.
1:33:40But again, I just don't recommend it.
1:33:44It's kind of a slippery slope that I've
1:33:46fallen down myself uh many times. And
1:33:48when you get to more advanced SQL and
1:33:51you're creating things like store
1:33:52procedures and triggers and all these
1:33:54things, this can actually cause a lot of
1:33:56issues. If you were to add any columns
1:33:58or remove any columns, then you'd be
1:34:00ordering by the wrong column because
1:34:03let's say this last name got removed. We
1:34:05didn't want it for some reason. Then the
1:34:08gender is 1 2 3 4. Now we're ordering on
1:34:12the wrong column and that would be a big
1:34:14mistake. So just by best practice, it is
1:34:18better to do gender,
1:34:21age, but I just wanted to show you that
1:34:23in case you want to be like me and kind
1:34:25of go down the wrong path. Uh so that is
1:34:28everything we're going to take a look at
1:34:30with group by and order by. In the next
1:34:32lesson, we're going to be taking a look
1:34:34at having [music]
1:34:34versus where.
1:34:48Hello everybody. In this lesson, we're
1:34:50going to take a look at the difference
1:34:51between having and where. Now, in the
1:34:54last lesson, we looked at group by and
1:34:56order by. The most obvious thing to do
1:34:58would be to come right here and say
1:35:00where, and we're going to say this
1:35:02column, which is actually named this.
1:35:04We'll say where the average age, let's
1:35:06say, is greater than 40, which would
1:35:09only be the males. So, let's go ahead
1:35:11and run this. And as you can see, we're
1:35:14not getting any output. Let's bring this
1:35:16up and take a look at the error. It says
1:35:19invalid use of the group by function.
1:35:22What's actually happening is something
1:35:24to do with this group by gender right
1:35:26here. When we're selecting gender and
1:35:28then we're performing an aggregate
1:35:30function, this occurs only after the
1:35:34group by actually groups those rows
1:35:36together. So when we're trying to filter
1:35:39based off of this column right here of
1:35:41average age, it really hasn't been
1:35:42created yet because this group by hasn't
1:35:45happened. That's where the having clause
1:35:47comes into play. So let's go ahead and
1:35:50what we're going to do is we're going to
1:35:51get rid of this. We're going to come
1:35:53right down here and instead of where
1:35:56we're going to say having. Now having
1:35:59was specifically created for this exact
1:36:02example. It comes right after group by.
1:36:05And after group by we can filter based
1:36:07off of these aggregate functions. So now
1:36:09if we run this, we're going to get an
1:36:12output that only has where the average
1:36:14age is greater than 40. Now let's take a
1:36:18look at just one more example. And I'm
1:36:20going to show you how you can use both
1:36:21in one query. So instead of
1:36:24demographics, let's look at the salary
1:36:26table.
1:36:28And let's run it.
1:36:30Now in this salary table, we have this
1:36:33occupation. And remember, we have this
1:36:34office manager that happens twice. And
1:36:37this is going to be our main example. So
1:36:39we're going to say occupation.
1:36:41And then we'll say the average salary.
1:36:45Now we'll need to come down here and
1:36:47we'll say group by. Now we're going to
1:36:50say occupation.
1:36:52So this should look pretty similar
1:36:55because right here we have our office
1:36:57manager and one of the office managers
1:36:59made 50, one of the office managers made
1:37:0260. So the average is 55,000.
1:37:04Now I can use the where by saying where.
1:37:09Then I'll say occupation
1:37:11like and let's see people who are
1:37:14managers. So I'll say percent manager
1:37:18percent and close that quote. So they're
1:37:20like a manager. And then I want to see
1:37:23where a manager makes more than let's
1:37:25say 75,000. So I won't actually say
1:37:28where. I'm going to say having an
1:37:30average salary.
1:37:33And I need to add a space there. Having
1:37:35an average salary greater than let's say
1:37:3875,000.
1:37:40And let's run this.
1:37:42So now I filtered at the row level right
1:37:46here in the wear clause. But then down
1:37:49here I filtered at the aggregate
1:37:51function level. This having is only
1:37:54going to work for aggregated functions
1:37:57after the group by actually runs. So
1:38:00that is the difference between the
1:38:01having clause and the wear clause. The
1:38:03wear clause you're most likely going to
1:38:05use a lot more. But if you do want to
1:38:07filter on those aggregated function
1:38:09columns, you have to use the having
1:38:11clause. I hope that that was really
1:38:13helpful. And in the next lesson, we're
1:38:15looking at our very last lesson in our
1:38:17beginner series. We're going to look at
1:38:19limit and aliasing.
1:38:28>> [music]
1:38:33>> Hello everybody. In this lesson, we're
1:38:35going to be taking a look at limit and
1:38:37aliasing. Limit is just going to specify
1:38:40how many rows you want in your output.
1:38:42If we take this table for example, if we
1:38:44come right here and we say limit, let's
1:38:47do three. If we run this, it's only
1:38:49going to take the top three that we
1:38:52have. Let's go and run this. As you can
1:38:54see, we have employee 1, three, and
1:38:56four, Leslie, Tom, and April. Now, this
1:38:59seems super straightforward, really,
1:39:00really easy, but it can be combined with
1:39:03order by to actually be really powerful.
1:39:06For example, let's say we wanted to take
1:39:08the three oldest employees. All we'd
1:39:11have to do is come right under here. We
1:39:14say order by, and we'll order by the age
1:39:17in descending order. So we're going to
1:39:20order on age descending and then it's
1:39:22going to take the top three. So if we
1:39:24run this and very quickly we have the
1:39:26top three oldest people in this table.
1:39:28Now there is one additional parameter
1:39:30that we can use in limit and all we have
1:39:33to do to access it is have a comma here.
1:39:36Now what this is going to do and I'll
1:39:37put a one here. What this is going to do
1:39:39is it's now going to say we're going to
1:39:41start at position three and then we're
1:39:44going to go one row after it. Now, I
1:39:47actually want to take one of these
1:39:49people. So, let's start at position two
1:39:51and select the next one after it, which
1:39:53should be Leslie. Nope. So, we're going
1:39:55to start at position two, and we're
1:39:57going to select the one right after it.
1:39:59So, we're going to start at position
1:40:00two, and then one means we're taking the
1:40:03next one row. Let's go ahead and run
1:40:05this. And as you can see, we got Leslie
1:40:08nope in our output. Now, let's come
1:40:10right down here. We are going to now
1:40:12look at
1:40:14aliasing. Now, aliasing is just a way to
1:40:18change the name of the column for the
1:40:21most part. And it can also be used in
1:40:23joins, but we're going to take a look at
1:40:25joins or aliasing joins in the
1:40:27intermediate series. In a previous
1:40:29lesson, we looked at a group by that
1:40:31looked like this. We selected gender,
1:40:33then we said from I believe it was
1:40:36employee
1:40:38demographics. Then we said group by
1:40:42gender. And we also had the average and
1:40:46I think it was age. There we go. And
1:40:49we'll add our semicolon. Let's go ahead
1:40:51and run this. In our output, we have
1:40:53gender as our gender column, the same as
1:40:55the column name. But then average age is
1:40:58average age. And so if we want to
1:41:01actually do something like a having
1:41:03where we say having the average age,
1:41:07let's say greater than 40 like we had
1:41:10it. we have to actually use this
1:41:12aggregate function in our having clause
1:41:14and we don't want to always have to do
1:41:16that. We can actually change the name of
1:41:18this column and subsequently use it
1:41:20throughout our query with that alias
1:41:22name. So I'm going to say as and that's
1:41:25the keyword to actually change it. We'll
1:41:27say as and we'll do average
1:41:30age. So now we've changed this name to
1:41:34average age and we can come down here to
1:41:36having and say having the average age
1:41:39greater than 40. And when we run this it
1:41:42works perfectly. And you'll notice that
1:41:43the name of the column was actually
1:41:45changed. Now this as isn't actually 100%
1:41:49needed. It's kind of implied. Even if we
1:41:51get rid of it, it's implied there's like
1:41:53this as in there somewhere. Um but we
1:41:56don't have to have it. If we took it out
1:41:58and ran it like this, it would still
1:42:00work exactly the same. So that is how we
1:42:03can use limit and aliasing in SQL. And
1:42:06congratulations, this is the end of the
1:42:08beginner series in my SQL. In the
1:42:11intermediate series, we're going to take
1:42:12a look at things like joins, unions,
1:42:15case statements, subqueries, and window
1:42:17functions.
1:42:30Hello everybody. In this lesson, we're
1:42:32going to be taking a look at joins.
1:42:34Joins allow you to combine two tables or
1:42:36more together if they have a common
1:42:39column. That doesn't mean the column
1:42:41name has to be the exact same, but at
1:42:43least the data within it are similar
1:42:45that you can use. There are several
1:42:47joins that we're going to look at today
1:42:49like inner joins, outer joins, and self
1:42:51joins. These are the two tables that
1:42:53we'll be working with the most
1:42:54throughout this lesson. We have the
1:42:56employee demographics table as well as
1:42:58the employee salary table. Now, within
1:43:00the employee demographics table, we do
1:43:02have this employee ID column. And if we
1:43:05look at the employee salary, we also
1:43:06have the employee ID column. So, in this
1:43:09instance, the column name is actually
1:43:10the exact same. And of course, the data
1:43:12inside of it is also very similar. So
1:43:15let's start by writing out an inner
1:43:17join. This is probably one of the most
1:43:18common joins, one of the most simple
1:43:20joins as well. An inner join is going to
1:43:23return rows that are the same in both
1:43:26columns from both tables. So let's see
1:43:28how we can actually write out this join.
1:43:30Let's come right down here and let's
1:43:32copy this. This will be the first table
1:43:34that we start with. And then we'll join
1:43:36the salary table onto this demographics
1:43:38table. So what we need to do is we need
1:43:41to come right here and we need to say
1:43:43join. Now by default join represents an
1:43:48inner join although we can write inner
1:43:50join here to make it more explicit
1:43:52explicitly writing out inner join. Then
1:43:55we're going to come up here and we're
1:43:57going to say employee salary. So we're
1:44:00selecting everything from the employee
1:44:02demographics and we're doing an injoin
1:44:04on the employee salary. Now we have to
1:44:07tell my SQL exactly what columns we're
1:44:10supposed to be joining on. I'm going to
1:44:12hit enter and I'm going to hit tab. Now,
1:44:14you don't have to hit tab. It just looks
1:44:16more codelike and it's easier to read.
1:44:19And that's how I've done it for other
1:44:20programming languages as well. So,
1:44:22that's how I'm going to show you how to
1:44:24do it. What we need to do is say on.
1:44:26Now, this keyword is going to allow us
1:44:28to say we're joining the demographics
1:44:30table to the salary table based on these
1:44:33two columns. So, from the demographics
1:44:35table, we're doing the employee ID is
1:44:38equal to and then in the salary table,
1:44:41it's also the employee ID. Let's do
1:44:43employee now and you spell it right.
1:44:46Employee ID.
1:44:49Now, if we try to run this and let's do
1:44:51this, we're going to get an error. And
1:44:54let's bring this up. It's going to say
1:44:55column employee ID on the in clause is
1:44:58ambiguous. Now, what does it mean
1:45:00ambiguous? That means that it doesn't
1:45:03know what table this employee ID is
1:45:05from. Is it from the employee
1:45:07demographics table? Is it from the
1:45:08employee salary table? We don't know
1:45:10because it's ambiguous. Now what we can
1:45:13do is we can specify it by saying
1:45:15employee demographics
1:45:17dot employee ID and then employee salary
1:45:21do employee ID. Now if we run this
1:45:26we're going to get the output that we're
1:45:27looking for. And let's take a look at
1:45:29this real quick. Let me bring this up.
1:45:32So we're pulling everything from the
1:45:34employee demographics that's right here
1:45:36all the way through the birth date. Then
1:45:38we're pulling the employee salary table.
1:45:40That's the employee ID all the way.
1:45:43Let's scroll over through the department
1:45:45ID. So, we're basically pulling in all
1:45:47of the rows or all the columns from both
1:45:49tables, but we're not pulling in all of
1:45:51the rows. Remember, an interjoin is only
1:45:54going to bring over the rows that have
1:45:56the same values in both columns that
1:45:58we're tying on. So, in this employee ID,
1:46:00we're missing number two. Are we missing
1:46:04any other ones? No, we're only missing
1:46:06number two. Let's go back up and I'm
1:46:09going to run both of these tables and
1:46:10we're going to take a look. So, let's
1:46:13run this.
1:46:14So, you'll notice in the employee salary
1:46:16table, we have a number two right here
1:46:19and that's Ron Swanson. But in the
1:46:21employee demographics table, we don't
1:46:24have that. I believe that Ron Swanson
1:46:27did this that Leslie Nope would not know
1:46:29when his birth date was because he
1:46:31didn't want to bring that information. I
1:46:32think that makes the most sense.
1:46:34Although Ron was not willing to give a
1:46:37comment on that. Now, if we run this
1:46:39again, you'll notice that two is not in
1:46:42there. Since two is not in the employee
1:46:45demographics table, the employee ID 2 is
1:46:47not going to be populated or brought
1:46:49over into this output from the employee
1:46:52salary table. Now, really quickly, this
1:46:54is honestly uh giving me some anxiety
1:46:56because this is so incredibly long.
1:46:58Something that I mentioned in the
1:47:00beginner's series is that you can use
1:47:01something called aliasing when using
1:47:03joins and it's really helpful. This is
1:47:05what I mean. So right here we have
1:47:07employee demographics. We're going to
1:47:08call this DEM. You can also do as DEM.
1:47:13You can do as SAL. These are just short
1:47:16names for demographics and short name
1:47:18for salary. And we can replace these and
1:47:21say DEM.mp employee ID and SA.mployee
1:47:26ID. Oh, that looks so much better. Now,
1:47:28we're going to run this and it'll be the
1:47:30exact same output, but now we're using
1:47:32these aliases, which just makes it so
1:47:34much easier to read. Now, one last thing
1:47:37that I want to show you while we're just
1:47:38looking at the inner join is selecting
1:47:41the actual columns. Let's say we wanted
1:47:43to select the employee ID
1:47:46and we wanted to select age and then we
1:47:49wanted to select their occupation.
1:47:52If we try to run this, we're going to
1:47:54get an error and it's going to be almost
1:47:56the exact same error that we got before,
1:47:58which is column employee ID in field
1:48:00list is ambiguous. So in our field list,
1:48:04which is right up here in the select
1:48:05statement, we have this employee ID. It
1:48:08does not know which employee ID to pull
1:48:10from, whether it's the demographics or
1:48:12the salary. So we have to tell it which
1:48:14one to pull from. So let's pull it from
1:48:16the demographics by saying DM.PMP
1:48:18employee ID. Now when we run this, we're
1:48:21able to get information from both tables
1:48:24in our output without having all of the
1:48:26information. And if there are columns
1:48:29that are similar in both tables, we have
1:48:32to denote that by using this alias or
1:48:35the table name. All right, so that is
1:48:37inner joins. Now, let's move down here.
1:48:40I'm going to copy this and we're going
1:48:41to come right down here and we're going
1:48:43to look at outer joins next. And let's
1:48:46put that right here. Now for outer joins
1:48:48we have a left join and we have a right
1:48:51join or a left outer and a right outer
1:48:53join. A left join is going to take
1:48:55everything from the left table even if
1:48:57there's no match in the join and then it
1:48:59will only return the matches from the
1:49:01right table. The exact opposite is true
1:49:04for a right join. Let's see how this
1:49:06actually works. Let's start by changing
1:49:08this to a left join or a left outer
1:49:12join. They're both the same and you can
1:49:13use them uh similarly. I'm just going to
1:49:16say left join and we're joining it on
1:49:18the exact same things and I'm going to
1:49:20take everything because I think that'll
1:49:21be easier to visualize and I'm going to
1:49:23run this. Now you may notice that this
1:49:25looks exactly the same and that's for a
1:49:28very good reason. It's because in the
1:49:31left table which is the employee
1:49:33demographics table the from statement
1:49:35that's our left table and then the join
1:49:38where we're actually joining on that's
1:49:40our right table. So this is our right
1:49:41table. So since we're doing a left, it's
1:49:43taking everything from the employee
1:49:45demographics. Now remember, the employee
1:49:48demographics didn't have Ron Swanson. It
1:49:51had no information. So everything in the
1:49:53right table had a match. Let's change
1:49:56this to a right join. And what this is
1:50:00going to do, and I want to make it all
1:50:01cap, make it all the same. What this is
1:50:03going to do is it's going to take
1:50:05everything from the employee salary
1:50:07table, but if there is not a match in
1:50:10the employee demographics, it just will
1:50:12have nulls. Let's go ahead and run this.
1:50:15So now it looks a little bit different.
1:50:17Now we're taking everything from the
1:50:19employee salary. So we're taking Ron
1:50:21Swanson, but if there is not a match, it
1:50:24will still populate that row, but it'll
1:50:27have all nulls in it. Then any of the
1:50:29information that is overlapping or the
1:50:31same, it will bring over. So employee ID
1:50:33is matched to employee ID 1. Then we'll
1:50:36bring all that information over from the
1:50:38left table. And that's essentially what
1:50:39a left and a right join is. With a left
1:50:42table, you're taking everything from the
1:50:43left table and then matches from the
1:50:45right table. If you do a right join,
1:50:47you're taking everything from the right
1:50:49table, but only matches on the left
1:50:50table. And again, it populates it with
1:50:53nulls. Now, let's go down and look at
1:50:55our last type of join that we're going
1:50:56to look at. And this is a self, let me
1:51:00spell that right, a self join. Now what
1:51:02is a self join? It is a join where you
1:51:04tie the table to itself. Now why would
1:51:07you want to do this? Let's take a look
1:51:09at a very serious use case. Let's do
1:51:12select everything. Let's do this from uh
1:51:16employee
1:51:18salary and let's run this. Now, let's
1:51:21say it's December 1st and the employee
1:51:23and Rex department decided to do a
1:51:25secret Santa and they wanted to assign
1:51:29based off of their employee ID the
1:51:31person who they're going to have as a
1:51:33secret Santa. We can help orchestrate
1:51:35this very easily using my SQL. Well,
1:51:38very easily is subjective, I guess, but
1:51:40let's take a look at how we can do this.
1:51:42So, just like any other join, the first
1:51:44thing we're going to do is select
1:51:45everything from employee salary and then
1:51:47say join. And then we're going to say
1:51:50employee salary again. So we're tying it
1:51:52to itself. Now when we come down here
1:51:55and let me do this. When we come down
1:51:57here and we say on, we have to specify
1:52:01which table we're pulling from. Are we
1:52:03pulling from the left table which is
1:52:05like the first table we're pulling from?
1:52:06Are we pulling from when we're joining
1:52:08on the right table? We need to be able
1:52:10to distinguish these two tables because
1:52:11they are the same. So, I'm going to say
1:52:14EMP1 and I'm going to say EMP2 just to
1:52:18say this is employee table one and
1:52:20employee table 2. So, we're going to tie
1:52:22them based off the employee ID because
1:52:24we know those will be the exact same
1:52:25because we're pulling from the same
1:52:27table. So, we'll do emp1
1:52:30employee.
1:52:32And just so you know, if it populates
1:52:34like this, you can hit tab and it'll
1:52:36auto uh finish that for you. and
1:52:38emporid.
1:52:43Now, if we run this, let's do this. The
1:52:47output that we're going to get is
1:52:48literally just a one forone match. It's
1:52:51all the columns and all the rows because
1:52:52they all match exactly. But now what
1:52:55we're going to do is we're going to
1:52:56assign an employee ID to the next
1:52:58employee ID and that will be their
1:52:59secret Santa. So, just keep it really
1:53:01simple. The next highest person with an
1:53:04employee ID, that is their secret Santa.
1:53:06So let's do an employee ID + one is
1:53:11equal to employee 2 employee ID. So
1:53:14we're adding one over here and we're
1:53:16saying that's equal to this employee ID
1:53:18over here. Let's run this.
1:53:21So now you can see Leslie Nope is now
1:53:25going to be assigned to Ron Swanson who
1:53:27has an ID of two. Ron Swanson is going
1:53:30to be assigned to Tom Havford which I'm
1:53:32sure he's really happy about. and so on
1:53:34and so forth. Now, let's bring this down
1:53:37here. And what we're going to do is try
1:53:40to simplify this and simplify this
1:53:42output a little bit because this is a
1:53:43little bit chaotic down here. So, we're
1:53:45going to specify what columns we want in
1:53:47our output. What we're going to want is
1:53:49the employee ID, first name, last name,
1:53:52and then employee ID, first name, last
1:53:54name of the person who they got for
1:53:56Secret Santa. So, we're going to start
1:53:58with emp1.mp
1:54:01employee
1:54:03employee
1:54:04id and we can call this we'll just say
1:54:07as emp
1:54:10Santa then we'll do a comma and we'll
1:54:12come down now I need to spell employee
1:54:16right so we have our employee ID and now
1:54:18we need our first name and last name now
1:54:20remember Leslie nope is going to be the
1:54:22secret Santa for Ron Swanson I don't
1:54:24know if I made that clear but that's I
1:54:26guess how it works so now we need to do
1:54:29uh emp1 one dot and we'll do first_ame
1:54:34and we'll do as we do as first_ame
1:54:39Santa. We can do a comma. We'll do the
1:54:42exact same thing except for the last
1:54:45name. So last_ame
1:54:50last name Santa. Now all we have to do
1:54:54is do a copy all this bring it down here
1:54:57and change this to two. They're pulling
1:54:59from the second table.
1:55:01And it'll look just like this. And get
1:55:03rid of this comma. And this is done.
1:55:06Let's run it.
1:55:08And let's bring this up. So we have
1:55:10employee Santa, first name Santa,
1:55:13Leslie, last name Santa. Nope. Then
1:55:16employee Santa. And we actually need to
1:55:18change these names. That is one thing we
1:55:20need to do. We'll just change it to
1:55:22employee name, first name,
1:55:26employee, and last name employee.
1:55:32And now when we run this, we have our
1:55:33Santa. And then we just have the
1:55:35employee who this person is going to be
1:55:37the Santa for. Now, this is kind of a
1:55:39silly way to look at it, but in essence,
1:55:41this is exactly how a self join works.
1:55:43Now the very very very last thing I
1:55:46promise you the last thing that I want
1:55:47to show you is how we can join multiple
1:55:50tables together. So we're going to say
1:55:51joining multiple you spell that right
1:55:55multiple
1:55:57tables together. Now not just one table
1:56:00together to another table. I'm talking
1:56:02about one table to another table to
1:56:04another table. So let's go all the way
1:56:06back up. We're going to take this right
1:56:09here and bring it all the way down. And
1:56:12what we're now going to do is we're
1:56:14going to tie in this table right here,
1:56:16the parks department. Let's actually
1:56:18look at this table and let's select
1:56:21everything real quick and let's run
1:56:23this. Now, let's go down here and we're
1:56:26going to say select everything. We'll do
1:56:28this from
1:56:30park
1:56:32departments and let's run this. Now,
1:56:34this is something called a reference
1:56:36table. This is not a table that most
1:56:39likely you'll ever add a bunch of
1:56:42information to. It's there to reference
1:56:44that we have these department names.
1:56:46Tables like the salary table or employee
1:56:48demographics table are going to change
1:56:50pretty often as people get raises or as
1:56:52they get older with their age. Those are
1:56:54going to be updated fairly often.
1:56:56Whereas this parks department table is
1:56:58just there for reference. Now, if we
1:57:00look down here in the columns, we have a
1:57:02department ID. Then we have a department
1:57:04name. So, we have the ID and the name of
1:57:07that ID. If we run our join and we
1:57:10scroll all the way to the right, you'll
1:57:13notice we have a DPT ID. This stands for
1:57:16department ID that's in the salary
1:57:18table. So, what we want to do is join
1:57:21this department ID to the department ID
1:57:24from the parks and wreck. So, what we
1:57:25can do is we're going to say inner join.
1:57:29And now we're going to join. Let's
1:57:31scroll down just a hair. There we go.
1:57:34Now, we're going to take this and do it
1:57:36off this. So, we're going to call this
1:57:38PD for short and we're going to say
1:57:41we're joining it on. Now, we cannot join
1:57:44this parks department to the employee
1:57:46demographics table. Why is that? Well,
1:57:49the employee demographics table only has
1:57:51employee ID all the way through birth
1:57:53date. There's no common column that we
1:57:56can tie to this parks department. The
1:57:58only table that has a common column is
1:58:01this department ID in this salary table.
1:58:04So what we need to do is actually take
1:58:05S. So we'll say SA dot and then we're
1:58:09going to say DP and we'll say department
1:58:12ID is equal to the PD dot and we need to
1:58:16take the department ID. Now notice these
1:58:19are not the exact same name. They are a
1:58:21little bit different but they have the
1:58:23same values. One thing I forgot to
1:58:25mention is that in this parks department
1:58:27there's no repeating. That's why it's a
1:58:29reference. Whereas in the salary, the
1:58:30department ID repeats several times
1:58:32because multiple people are in the same
1:58:34department. So this reference table also
1:58:36usually does not have duplicates. Uh
1:58:38just one other thing to note, but we
1:58:40have now tied it successfully. Let's try
1:58:42to run this.
1:58:45And if we come down here, go all the way
1:58:47to the right. We now have the department
1:58:49ID 11111 and the department ID and
1:58:52department name. Parks and recreation,
1:58:54healthcare, public works, finance,
1:58:56public works, and parks and recreation.
1:58:58So this worked perfectly. So this is how
1:59:00you can tie multiple tables together. If
1:59:03you have common columns between them,
1:59:05even though employee demographics has no
1:59:08column that's related to the parks
1:59:09department table, we can still tie them
1:59:11together based through this employee
1:59:14salary because employee demographics can
1:59:16tie to employee salary. Employee salary
1:59:18can tie to the parks department. And
1:59:20that really is the majority of what you
1:59:22need to know in order to use joins.
1:59:24Well, now in the next lesson, we're
1:59:25going to be taking a look at something
1:59:26called a union.
1:59:40Hello everybody. In this lesson, we're
1:59:42going to be taking a look at unions in
1:59:43MySQL. A union allows you to combine
1:59:46rows together, not like columns like we
1:59:48were doing before with joins where one
1:59:50column is next to the other. A union
1:59:53allows you to combine the rows of data
1:59:55from separate tables or from the same
1:59:57table. It's up to you. But you do that
1:59:59by taking one select statement and using
2:00:02a union to combine it with another
2:00:04select statement. Let's see how this
2:00:06actually looks. So what we're going to
2:00:08do is right after this select statement,
2:00:10we're going to come here and say union.
2:00:12Then we're going to go right below the
2:00:14union and we're going to do another
2:00:16select statement. So we're going to copy
2:00:18this, place it right here. But instead
2:00:21of the demographics table, just for
2:00:22example, we'll do the salary table. Now,
2:00:25if we look at the demographics table,
2:00:27let's say we want to take age and
2:00:31gender. And let's go and take a look
2:00:33really quickly at the salary table. And
2:00:37let's say we want to take first name and
2:00:40last name. So, we'll do first_ame
2:00:43and last_ame.
2:00:45Now, let's go ahead and run this and see
2:00:47what it looks like. And let's pull this
2:00:49up. So, as you can see, we have age and
2:00:52gender. That's from the very first
2:00:53select statement. And that's also the
2:00:55column names. But then we have all of
2:00:57the data for the age and gender. And
2:00:59then below, if we move this over a
2:01:02little bit, we have the last name and
2:01:05first name from the employee salary
2:01:07table. It's just down here. Now, what I
2:01:10just demonstrated is that this doesn't
2:01:12always work for everything. You can't
2:01:14just combine random data together
2:01:16because this is bad data. We shouldn't
2:01:18have age and gender mixed with first
2:01:20name and last name. Really, when you're
2:01:22using this, you need to keep the data
2:01:24the same. So, for us, we should take the
2:01:27first I'll actually just copy this. The
2:01:29first and last name from the employee
2:01:31demographics as well. And let's run
2:01:33this. And now we have all the names from
2:01:36all of the tables. Now, you may be
2:01:39thinking, where did all the other data
2:01:40go? Before we had a lot of rows, but now
2:01:43we only have a unique row for each one.
2:01:46Well, by default, this is actually a
2:01:49union distinct. And if you remember,
2:01:52distinct is only going to take unique
2:01:53values. So when we're doing this, union
2:01:56is going to remove all the duplicates.
2:01:58And the first name and last name from
2:02:00salary overlaps a lot with the employee
2:02:03demographics table. So when we ran this,
2:02:06the only one that's actually somewhat
2:02:08unique to one table is that in the
2:02:09employee salary table, we have Ron
2:02:11Swanson, whereas we don't have that in
2:02:13the employee demographics. Now, if we
2:02:16wanted to show all of them without the
2:02:18distinct, there is something called a
2:02:20union all. If we run this
2:02:24now, we're going to get all of the
2:02:25results without removing any of the
2:02:27duplicates. So, if we scroll down, we're
2:02:29going to have duplicates in here, but
2:02:32we're just showing all of the results
2:02:33from this table and from this table. Now
2:02:36that we know how to actually use a
2:02:37union, let's look at a use case. So,
2:02:40let's go right down here and let's copy
2:02:43this. Why not?
2:02:45and let's put it right down here. Now,
2:02:48let's say in the employee demographics,
2:02:50we wanted to take the first name and
2:02:52last name where the age is greater than
2:02:5750. And let's run this. So, there's only
2:03:00one person, but let's label them. Let's
2:03:04add a label. We're going to say, comma,
2:03:06old. So, this person is old. And if we
2:03:10run this, it says first name, last name,
2:03:12and old. And we can even call this as
2:03:16label. And if we run this, the label is
2:03:19old. So Jerry Giritch, he's the only old
2:03:21person in this demographics table. Now,
2:03:24why are we doing this? Well, the parks
2:03:26department is trying to cut their budget
2:03:28a little bit. They want to identify
2:03:30older employees that they can push out.
2:03:32And they also want to identify high paid
2:03:35employees who they can reduce their pay
2:03:36or push them out to save money. So, we
2:03:39just identified someone who's older who
2:03:40are going to want to try to push out.
2:03:42But let's in the same output find people
2:03:45who are also highly paid. So now we can
2:03:48come down here. We can say union and
2:03:51let's do this like this. I need to spell
2:03:53this right. All right. Union and let's
2:03:56take this. We're not going to be using
2:03:58[clears throat] this exact same query,
2:04:00but we actually need to pull from the
2:04:01salary table. So the employee salary. So
2:04:04we also want the first name and last
2:04:06name. But let's say where their salary
2:04:10is greater than let's say 70,000 cuz
2:04:13that's a lot of money. If you're making
2:04:14more than 70 uh for sure the parks
2:04:16department is going to try to get rid of
2:04:17you. But for the label we're going to
2:04:20change it to a highly paid employee.
2:04:25Now let's go ahead and run this. So now
2:04:28we have Leslie Nope and Chris Trager.
2:04:31They're both labeled as highly paid
2:04:33employees. Now 50 I think is just a
2:04:36little too low. Um if I'm being
2:04:38completely honest, I think we need to
2:04:39change this and we should do a union and
2:04:43then add another select statement. Let's
2:04:46bring this down. I think the 50 is too
2:04:49low. Let's change it to 40.
2:04:52And let's add one more thing. Let's say
2:04:56and
2:04:58the gender is equal to male.
2:05:02And then we'll go down here and say
2:05:04where the gender is equal to female
2:05:07because we want to separate this out. So
2:05:09we want to know who's the old man. Oh,
2:05:12that's actually old lady. This is the
2:05:14female one. And for up here where it's
2:05:16male, we'll say old man. So we have
2:05:20three different select statements using
2:05:22two separate unions. We're selecting the
2:05:24first name and the last name in all of
2:05:26them, keeping the data consistent. And
2:05:28then in our third column, we're labeling
2:05:30it either old man, old lady, or highly
2:05:33paid employee. Let's go ahead and run
2:05:35this. And let's look at our output. Now,
2:05:39you may notice something really quickly
2:05:42that Chris Trager and Leslie Nope are an
2:05:44old man and an old lady. And Leslie Nope
2:05:47and Chris Trager are both highly paid
2:05:49employees. So, these people meet
2:05:51multiple criteria. Yeah. So, let's
2:05:53actually order by and then we'll do
2:05:55first_ame,
2:05:57last name because we want to order by
2:06:00these to see. So, let's run.
2:06:03And now we can easily see that Chris
2:06:05Trager is both of these. Donna is just
2:06:08an old lady. Jerry's just an old man.
2:06:10And Leslie is both an old lady and a
2:06:12highly paid employee. So, now we can
2:06:15send this to whoever we need to send it
2:06:16to to make sure that these people get
2:06:18looked at first so that our job is still
2:06:21secure. The job market is tough these
2:06:22days. You got to do what you got to do.
2:06:24So that is how we use union. And let's
2:06:28just take one more look at it. There we
2:06:30go. So this is how we can use unions.
2:06:33It's kind of a real use case. I've done
2:06:34something very similar to this in my
2:06:36real job, but you know, this is just an
2:06:39example of how you can have multiple
2:06:41select statements all combined or
2:06:44combining the rows using a union. In the
2:06:46next lesson, we're going to be taking a
2:06:48look at case statements.
2:06:53>> [music]
2:07:01>> Hello everybody. In this lesson, we're
2:07:03going to be taking a look at string
2:07:05functions. Now, string functions are
2:07:07built-in functions within MySQL that
2:07:09will help us use strings and work with
2:07:11strings differently. Now, we're going to
2:07:12look at a ton of different ones. They
2:07:14all have different use cases, but I'll
2:07:16try to walk through some of those as we
2:07:18go along. But we'll look at a lot of
2:07:20different string functions in this
2:07:21lesson. We'll start off with one that's
2:07:23really simple. This one is called
2:07:24length. So, if we select and then we say
2:07:28length and let's say we put in and I
2:07:32don't know why it's popping up like
2:07:33that. Let's say we put in something like
2:07:35sky or skyfall or really anything. If we
2:07:39run this, it's going to give us the
2:07:41length of how long this string is. So,
2:07:44if I come down here and we say select
2:07:47everything from employee
2:07:51demographics
2:07:53and let's add a semicolon here. Let's
2:07:55run this one right down. Now, what we
2:07:58can do is we can look at how long each
2:08:01person's name is. What we can do is just
2:08:04take the first name, but then we'll also
2:08:07do the length of the first underscore
2:08:11name. So if we run this now, we get
2:08:14Leslie, Tom, Jerry, Donna, and it gives
2:08:16us the length of their name. If we
2:08:19wanted to, we could even order by this.
2:08:20So we could do order by, and we could
2:08:23just do two for now. And we can order by
2:08:26the length from the shortest name all
2:08:28the way to the longest name. Now, one
2:08:30use case that I've used length for in my
2:08:32actual job was when I was working with
2:08:34phone numbers. I wanted to make sure
2:08:36that they were exactly 10 characters
2:08:38long. Otherwise, something went wrong
2:08:40somewhere in the data cleaning process.
2:08:42So, I would go and look at the length
2:08:44and I would make sure they're all 10.
2:08:45And if any were above 10, I would go and
2:08:47specifically look at those and try to
2:08:48clean those and fix those up. Now, let's
2:08:50go on to the next one. And these next
2:08:52ones are pretty simple ones. At least I
2:08:54think they're fairly simple. We're going
2:08:56to look at upper first. And it's doing
2:08:58the same thing as the other one. We'll
2:09:00do upper. And let's say we're going to
2:09:02do sky. If we select upper sky, it's
2:09:06going to give us an all uppercase. Or we
2:09:09can copy this and we can do lower. So
2:09:13now let me add semicolons otherwise it's
2:09:15going to drive me crazy. Uh let's try
2:09:18this lower. Now it's going to do all
2:09:21lower even if I make it all capital. So
2:09:24if I say all capital sky, it's going to
2:09:26make it all lower. So if we come back
2:09:28up, let's copy this
2:09:31and instead of doing the length now
2:09:33we'll do upper.
2:09:36Let's go ahead and select this. So we
2:09:38have Leslie and then we have the upper
2:09:40first name. So upper allcase Leslie. Now
2:09:43this is actually really good. This is
2:09:45really helpful especially with
2:09:46standardization is what I found a great
2:09:48use case for it because sometimes it'll
2:09:50be all capital tom and sometimes I'll
2:09:53put it in as T lowercase OM. uh and just
2:09:56making them all uppercase or all
2:09:57lowercase can help correct those really
2:10:00simple standardization issues within a
2:10:02single column. The next one that we're
2:10:03going to look at is trim. Now there's
2:10:06multiple trims. We have trim, left trim,
2:10:08and right trim. Trim is basically going
2:10:11to take the white space on the front or
2:10:13the end and get rid of it, which is
2:10:15really really helpful. So what we're
2:10:16going to do is we're going to come right
2:10:18here and say select and we'll start off
2:10:20with trim. And let me add a semicolon
2:10:23every time. So then we'll do trim. And
2:10:26for our actual string, we'll do
2:10:30something a little bit odd. We'll do
2:10:31some spaces. And then we'll do sky. And
2:10:34then we'll add some spaces. Let's run
2:10:36this and add our semicolon. That's going
2:10:38to be the end of me in this lesson. I
2:10:40was just adding semicolons. Now it fixes
2:10:43it completely. Now, what if we don't add
2:10:45sky at all? We'll [snorts] just keep it
2:10:47like this. Well, you can see that
2:10:49there's spaces before and there's spaces
2:10:51after. But that's what trim does. trims
2:10:54gets rid of the leading and the trailing
2:10:57white spaces. Now, if we come up here
2:10:59and we just do the left trim, it's only
2:11:02going to remove from the left hand side.
2:11:05So, we're only getting rid of the
2:11:06left-hand side white spaces. This right
2:11:09hand side, as you can see, is really
2:11:10long. It's still there. And if we do RT
2:11:14trim, we go ahead and run this one. It
2:11:16gets rid of the white space on this
2:11:18side, but it doesn't get rid of the left
2:11:20space on this side. Now, let's keep
2:11:22going. We have a lot to cover still.
2:11:24We're going to move on to what I think
2:11:25is probably my most favorite string
2:11:27function if I'm allowed to have a
2:11:28favorite string function and that's
2:11:30substring. But I'm going to kind of work
2:11:33us into substring a little bit by
2:11:35looking at two smaller functions which
2:11:37is left and right. So let's select
2:11:40everything and we'll do that from the
2:11:43employee demographics again. Let's add
2:11:44our semicolon. Now I'm going to run
2:11:47this. Now I want to get the first name
2:11:50and I'm going to do left of the first
2:11:54underscore name just like this. Now when
2:11:57you're using this is actually going to
2:11:59be an error. Let's see if I highlight
2:12:00over this if it'll tell me what the
2:12:02error is. It says the parenthesis is not
2:12:04a valid position. They're expecting
2:12:06something else. And basically what
2:12:08they're telling us is that this is not
2:12:10how it should be written. We're looking
2:12:12for a different value. That different
2:12:14value is actually a number. We're going
2:12:16to do comma and let's do four. That's
2:12:18what it was looking for. It didn't want
2:12:20this at the end. It needed this comma
2:12:22four. And what we're actually specifying
2:12:24is how many characters from the left
2:12:26hand side do we want to select. So we're
2:12:29selecting the first name and we're going
2:12:31from the left four characters. Let's go
2:12:33ahead and run this. And so we have
2:12:35Leslie, Tom, April all the way down. You
2:12:38can see that there's only four
2:12:39characters in each one. So, someone like
2:12:41Chris, the S is no longer going to be
2:12:44there because we're only looking at the
2:12:45first four characters. Now, we can do
2:12:47the exact same thing. And let's actually
2:12:49copy this uh down here. So, we'll come
2:12:52we'll go like this. Try to make this a
2:12:54little more professional and we'll do
2:12:57right. So, now we'll do right. If we do
2:13:00the right four, it's going to go from
2:13:02the right hand side of the string and go
2:13:05left four. So, we're looking at the far
2:13:08four for most right characters. Now,
2:13:10this can be useful in certain instances,
2:13:13but if I'm being honest, I don't use
2:13:15these that much. For the most part, I'm
2:13:18pretty addicted to using substrings. I
2:13:20love substrings. I think they're
2:13:21fantastic. And let's look at substrings
2:13:25like this. So, a substring is going to
2:13:27allow us to do a few different things.
2:13:30Let's do first_ame.
2:13:32The second thing that we put within this
2:13:34function is the position that we want to
2:13:36start at. So let's say we want to start
2:13:38at the third position. And then we
2:13:41specify how many characters we want to
2:13:43go. So with this we specified four, but
2:13:46let's just do two. So now we're going to
2:13:49the third position and we're going over
2:13:51to the right two characters. Let's go
2:13:53ahead and run this. So with Leslie we
2:13:56get SL. So we go 1 2 and three. We start
2:14:00at the third position and then we take
2:14:02two characters, the S and the L. I have
2:14:05found this one to be extremely extremely
2:14:08useful. Let's take this for example.
2:14:10Let's do comma. Let's do birth
2:14:13date. And let's run this. I'm keeping
2:14:16everything in here. Although it might be
2:14:18a bit much, but let's say we have this
2:14:19birthday. And this middle column is the
2:14:22month. And we're running some, you know,
2:14:24query. We want to find the month that
2:14:26everyone is born. So we can do that very
2:14:29easily using substring and we wouldn't
2:14:32have been able to do this very easily
2:14:34using left or right. So now we're going
2:14:36to take this birth date and we're going
2:14:38to use the substring and we want to
2:14:39select these middle characters. So what
2:14:41we need to do since it's all
2:14:42standardized we do 1 2 3 4 5 6. We start
2:14:46at position six and we want to select
2:14:49one and two. Let's go ahead and run
2:14:52this.
2:14:53And now we've pulled out all of the
2:14:55months. So we can say as birth_mon
2:15:00month and now we could save that put it
2:15:03into a temp table add it as a new column
2:15:05in our table whatever we want to do. Uh
2:15:07but now we have this information that we
2:15:09desperately desperately wanted to know.
2:15:11So that is left right and substring.
2:15:13Again substring is it's fantastic. Now
2:15:15let's keep going. The next thing that
2:15:17we're going to take a look at is
2:15:18replace. Now replace will replace
2:15:21specific characters with a different
2:15:23character that you want. So, let's
2:15:26actually copy all this right here
2:15:28because I don't want to keep writing
2:15:29this out. And we'll say select
2:15:32everything.
2:15:33Now, what we're going to do is we're
2:15:35going to take the first underscore name
2:15:37and then we're going to say replace. And
2:15:40then we'll also do the first underscore
2:15:42name, but we can specify what we want to
2:15:44replace and then what we want to replace
2:15:46it with. So, we have two more parameters
2:15:48that we need to put in this function. So
2:15:51let's say A and let's replace it with a
2:15:54Z. Let's just see what that does. Let's
2:15:57go ahead and run this. And so now when
2:16:00we see the letter A and we are
2:16:01specifying a lowercase A like mark that
2:16:05is replaced with a Z. So that's really
2:16:07all replace does. It specifies what you
2:16:09want to replace and then what you're
2:16:10going to replace it with. Now let's take
2:16:12a look at the next one and we're going
2:16:13to take a look at a function called
2:16:15locate. So, if I say select and let's do
2:16:19locate. I'm going to give it a string.
2:16:22I'll say Alexander. That's my name. And
2:16:24I'm going to specify what I'm looking
2:16:26for. So, let's close this parenthesis.
2:16:28The string that we're actually looking
2:16:29for comes first. So, what we're going to
2:16:31do is I'm looking for the letter X in my
2:16:34name. So, we'll do X and Alexander.
2:16:37Let's go ahead and run this. And it
2:16:39tells us that it is in position four.
2:16:41So, we have one, two, three, and four.
2:16:44That's where our position is. That's
2:16:46where it locates that sequence that
2:16:47we're looking for. Now, if we pull this
2:16:50down here,
2:16:52place this right here, and we'll change
2:16:54this locate. Now, let's say we're still
2:16:57looking at the first name, but we want
2:16:59to locate people that have an a
2:17:02like this in their name. Let's go ahead
2:17:05and run this. And we get zeros for
2:17:08everybody except for an andy. So, this
2:17:12might be something where we put it into
2:17:13a CTE or a temp table. Then we can
2:17:15filter down based off of these results
2:17:17to where it only equals one. Now the
2:17:20last one that we're going to take a look
2:17:21at and let's go right here. We're going
2:17:23to do first name last underscore name.
2:17:26Now this one is super super useful
2:17:29because what we can do is have a
2:17:31concatenation
2:17:33of multiple columns. So let's go down
2:17:36right here. So we have first name and
2:17:38last name. But if we come down and we
2:17:40say concat, we can then combine these
2:17:43columns into one single column. So we'll
2:17:46do concat. It'll do first underscore
2:17:49name and then comma last underscore
2:17:52name. And if we run this, it's going to
2:17:55be Leslie and nope combined into Leslie
2:17:58nope. Now this doesn't look perfect,
2:18:00right? We don't want it to look like
2:18:01that. All we have to do is come in here
2:18:03and we could do a little space. So we'll
2:18:06add a space in there. And if we run that
2:18:08now we have Leslie and nope. And we
2:18:10could call this as full_name.
2:18:15And this is something that I've done a
2:18:17million times in my real job where
2:18:19there's multiple columns. We want to
2:18:21create one column out of it or take two
2:18:23columns and create one column. Happens
2:18:25all the time. So this concat is really
2:18:27really helpful to combine those columns
2:18:29really quickly. So that is all we're
2:18:31going to take a look at in this string
2:18:32functions lesson. In my full course you
2:18:34can find in the description below. I
2:18:35also have lessons on numeric functions,
2:18:37date and time functions, converting
2:18:39different data types, all in [music] the
2:18:41functions module.
2:18:52[music]
2:18:54Hello everybody. In this lesson, we're
2:18:56going to be taking a look at case
2:18:58statements in MySQL. A case statement
2:19:00allows you to add logic in your select
2:19:02statement. Sort of like an if else
2:19:04statement in almost all other
2:19:05programming languages or even things
2:19:07like Excel. Let's see how this actually
2:19:09works. So let's bring this down and
2:19:12let's take this employee demographics
2:19:14table and let's take the first name and
2:19:17let's take the last name and let's add a
2:19:21case statement.
2:19:23How we need to do this is we have to say
2:19:25case. So that's going to signify that
2:19:27we're starting a case statement. And
2:19:29then I'm going to go over here and say
2:19:31tab. So this is where our logic comes
2:19:33into play. So I'm going to say when the
2:19:36age let's say is less than or equal to
2:19:4030 then so I'm saying if the age is less
2:19:44than or equal to 30 then what's going to
2:19:46happen? We'll just keep it really simple
2:19:48for now. We'll just say that this person
2:19:50is young. And then if we want to end the
2:19:53case statement, we'll come down here and
2:19:55say end. So this is a complete case
2:19:58statement. Let's go ahead and run it.
2:20:01And let's take a look at the output. So
2:20:03we have the first name, we have the last
2:20:05name, and then we have this case
2:20:07statement right here. And if their age
2:20:09is less than or equal to 30, they're
2:20:10young. Let's actually add the age right
2:20:13here just so we can visually see that as
2:20:15well. So we have the age. So this person
2:20:18is the only person who's under or equal
2:20:20to the age of 30. That's April. So, she
2:20:23has a label of young. The great thing
2:20:26about case statements is you can add
2:20:28multiple when statements. So, we can
2:20:29come down here and say when. And then we
2:20:32can do something like when age and maybe
2:20:34we'll say between. So, I don't know if
2:20:37in previous lessons we've looked at
2:20:38between, but between just says between
2:20:40this number and this number. So, we'll
2:20:42say between 31 and 50. If they're
2:20:46between 31 and 50, well, good night. uh
2:20:50that person is old. So, we're going to
2:20:52have it just like this. We're going to
2:20:54run it.
2:20:56And now we have a lot of people who are
2:20:59old. These are all people between the
2:21:01ages of 31 and 50. But we still have
2:21:04more people outside of the age of 50 or
2:21:06older than 50. So, we could do when the
2:21:10age and now we can say greater than or
2:21:13equal to 50. and we're gonna say then
2:21:17and then we're gonna say on death's
2:21:20door. Uh because good night if you're
2:21:22over 50, my parents are gonna love me
2:21:24for this one. So let's go ahead and run
2:21:26this. And then if we look at this, we
2:21:29have on death store right there. Now
2:21:31this is huge. This is massive. So let's
2:21:33actually name this and we'll just say as
2:21:36at the end of end, so right after end,
2:21:38we'll say as age uh bracket and let's
2:21:42run this.
2:21:44And this looks a lot better. So now we
2:21:47have this age bracket just signifying
2:21:49kind of where people are at. And most
2:21:51people are quite old. Poor Jerry. Uh you
2:21:54know, can't catch a break that guy. Now
2:21:56let's go down and let's take a look at a
2:21:58different table. So let's select
2:22:00everything. We'll do from employee
2:22:04salary.
2:22:05Now that we have our employee salary
2:22:07table, here is the scenario that we are
2:22:09given. The Pawne Council sent out a memo
2:22:12of their bonus and pay increase for end
2:22:14of year and we need to follow it and
2:22:16determine people's end ofear salary or
2:22:18the salary going into the new year and
2:22:20if they got a bonus how much was it. So
2:22:23the first thing we need to do is we need
2:22:25to get the pay increase and bonus and
2:22:29their pay increases look like this. So,
2:22:31if they made less than 50,000, then that
2:22:35equals a 5% raise. Very generous. And if
2:22:38they made greater than 50,000, that
2:22:42equals a 7% raise. Very, very generous.
2:22:45Lastly, if they work in the finance
2:22:48department, that equals a 10% bonus.
2:22:52Just cash that goes into their bank
2:22:54account. Very, very generous, but only
2:22:56the finance department gets it. So,
2:22:58these are the guidelines that the Pawne
2:22:59Council sent out and it is our job to
2:23:02determine and figure out those pay
2:23:04increases as well as the bonuses. So,
2:23:06let's come right down here. We're going
2:23:08to have our salary employee. I actually
2:23:10want to be able to see these. Let me
2:23:12pull this up just a touch. There we go.
2:23:15So, we want to be able to write this
2:23:17out. So, first thing we should do is
2:23:18just select the columns that we need.
2:23:20First name, last name, probably salary
2:23:23as well. And now what we can do is
2:23:26determine this first one which is if
2:23:28they make less than 50,000 they get a 5%
2:23:31raise. So let's say case and I'll also
2:23:35add uh end in here and we're going to
2:23:38say when their salary is less than
2:23:4250,000
2:23:43what's going to happen then we say then
2:23:47salary. So, we're taking their initial
2:23:48salary and we're saying plus then we're
2:23:51going to do salary times 0.05.
2:23:56And if we run this
2:23:59should work. Let's pull this up really
2:24:01quickly. So, April Lgate, she made under
2:24:0450,000. So, she got a raise and her new
2:24:07salary is 26,250.
2:24:10We could actually call that. We'll say
2:24:12uh as a actually let's do new salary
2:24:17because that's their new salary. And
2:24:19let's run this. So the new salary is
2:24:2226,250.
2:24:24Andy Dwire is now making 21,000. Now
2:24:27this calculation you can do it different
2:24:29ways. We could do it exactly like this
2:24:30or we could just do times 1.05.
2:24:34Should be the exact same thing. Uh just
2:24:36however you would like to write it out.
2:24:38it's just you know adding it
2:24:40[clears throat] or multiplying it by
2:24:41this. So let's take this and now we're
2:24:45going to say when it is greater than
2:24:4850,000 so let's say greater than 50,000
2:24:51they get a 1.07.
2:24:54So this is the 7% increase. This is a 5%
2:24:56increase. This is a 7% increase. And
2:24:59let's run this
2:25:02and let's put this up here.
2:25:05So now if they made greater so 50,000
2:25:08that's 75,000 they got a 7% increase.
2:25:11Now unfortunately we did not make the
2:25:13rules the Pawnie Council did and the
2:25:17people who made exactly 50,000
2:25:19unfortunately were not part of those
2:25:21brackets. Uh and that just wasn't up to
2:25:23us. We couldn't control that. So
2:25:25unfortunately Tom Havford and Jerry
2:25:28Gurggic just didn't get races this year.
2:25:30And that's not our fault. Okay. That's
2:25:32not our fault. Now, the next thing that
2:25:35we need to do is determine the bonuses.
2:25:38Now, let's come right back up here
2:25:40really quickly and let's just copy this
2:25:43because what we need to determine is
2:25:47how we know that somebody is in the
2:25:50finance department because if they're in
2:25:52the finance department, that means they
2:25:53get a 10% bonus. That's really
2:25:55important.
2:25:57Now, it's not in the employee
2:25:59demographics. We don't have anything
2:26:00about the department. But if we look in
2:26:03the salary and we run this, we do have
2:26:06the department ID. Now, let's open up
2:26:11and let's pull this up right here. We'll
2:26:13look at the parks department. And in the
2:26:18parks department, here we go. The
2:26:20finance is department ID of six. So if
2:26:24we're looking at the salary,
2:26:26there's only one person who's in uh
2:26:29department ID equal to six. So what we
2:26:32can do is another case statement. We can
2:26:35say comma we'll do case and end
2:26:40and we'll do another one. We're going to
2:26:42say when
2:26:44dep
2:26:46so when the department ID is equal to
2:26:49six, then we're going to give them a
2:26:52bonus. So, we're going to say salary
2:26:54times.10
2:26:57and we'll call this as bonus. Let's go
2:27:01ahead and run this
2:27:03and let's pull it up. So, he gets a
2:27:06$7,000 bonus this year. That's Ben Wyatt
2:27:09uh because he was part of the finance
2:27:11department that just did uh an
2:27:12exceptional job this year apparently
2:27:14according to the Ponyie Council. So,
2:27:17that is how case statements work.
2:27:19They're really powerful, really useful.
2:27:21I honestly use them quite often and
2:27:23they're just a way to really add some
2:27:25logic and some, you know, labeling or
2:27:28even do calculations like we did right
2:27:30here with the salary. In the next
2:27:32lesson, we're going to be taking a look
2:27:33at sub queries in MySQL.
2:27:47Hello everybody. In this lesson, we're
2:27:49going to be taking a look at sub queries
2:27:51in MySQL. Now, subquery is basically
2:27:54just a query within another query. We
2:27:57can do this in a few different ways, and
2:27:59I'm going to try to show you a lot of
2:28:00the different variations within this
2:28:02lesson. The first way that we're going
2:28:03to use a subquery is in the wear clause.
2:28:06Then, we'll take a look at the select
2:28:08and the from clause. Also, let's take
2:28:09this demographics table that we have
2:28:11down here. What if we only wanted to
2:28:13select the employees who worked in the
2:28:16actual parks and recck department? Well,
2:28:18we could do that if we had a few joins.
2:28:21We have this salary table and one
2:28:24actually represents that they work for
2:28:26the parks and wreck. If we come over
2:28:28here and we open this up, we can see
2:28:30that parks and wreck is the department
2:28:32ID of one. So, we do have that option.
2:28:35We could just join these two tables
2:28:36together. But sometimes we don't want to
2:28:39do that and we'll use a subquery. Let's
2:28:41see how it works in the wear clause. So,
2:28:44let's go ahead and get rid of this. So
2:28:45what we're going to do is we're going to
2:28:47say select everything from employee
2:28:49demographics where and now we want to
2:28:52pull because this is the salary table.
2:28:54We want to pull employee IDs where the
2:28:57department ID is equal to one. But
2:28:59remember we're querying off of this
2:29:02table. So let's actually pull this up.
2:29:05This is what we're working with. So we
2:29:07want to say where the employee
2:29:10ID that's referencing this column in the
2:29:13demographics table is in what we're
2:29:16going to do is we're going to do a
2:29:18parentheses here and we can even come
2:29:20down and put a parenthesis down here. So
2:29:22what we're going to do now is write our
2:29:24query which is our sub query and this is
2:29:26our outer query. So now we're going to
2:29:28write an entirely other query within
2:29:31this. So, we'll say select. And now
2:29:33we're going to say employee
2:29:35id. And let's just bring this over. I
2:29:38usually have it something like this. And
2:29:41I'm going to try to bring this down a
2:29:43little bit. So, select everything. And
2:29:45then we'll do from and then instead of
2:29:48employee demographics,
2:29:50we'll do employee salary.
2:29:54And let's just format this a little
2:29:55better. So, select the employee ID from
2:29:58employee salary. And remember we wanted
2:30:00to do where the department ID is equal
2:30:03to one. Now let's bring this back up.
2:30:07And this is what the query is going to
2:30:09look like. Now just by itself, let's run
2:30:12this subquery or this inner query. When
2:30:15we run this, it's going to create this
2:30:17list of just employee IDs where the
2:30:20department ID is equal to one. So when
2:30:22we say where the employee ID from the
2:30:25employee demographics table is in, it's
2:30:28going to try to match those employee IDs
2:30:31to this list of employee IDs. So just
2:30:34remember 1 2 3 4 5 6 and 12. Let's go
2:30:37ahead and run this entire query.
2:30:40Now we have 1 3 4 5 6 12. If you
2:30:44remember from previous lessons, the two
2:30:47is Ron Swanson and he's only in the
2:30:49salary table. So since we're doing just
2:30:52the employee demographics table, he's
2:30:54not in here. So what we're doing is
2:30:55we're selecting everything from the
2:30:57employee demographics where the employee
2:30:59ID in this table matches or is in the
2:31:04select employee ID from the salary table
2:31:07where the department ID is equal to one.
2:31:09In essence, this is what a subquery is.
2:31:12It's a query within a query. Now, what
2:31:14would happen if we have the employee ID,
2:31:16but we also wanted to say the department
2:31:18ID because we just wanted to view this.
2:31:20Let's go ahead and try to run this.
2:31:23We are going to get no output and we're
2:31:25going to get an error that says operand
2:31:27should contain one column. The operand
2:31:30referring to this entire thing right
2:31:32here cuz this is an operator. So, this
2:31:35is our operand and we're returning two
2:31:37columns in here which is saying we
2:31:39cannot do. We have to only have one. So
2:31:43now if we run this, it works perfectly
2:31:46well. And let's bring that down. Now we
2:31:48can also use the subquery in a select
2:31:51statement. So let's take a look at that.
2:31:53Next, let's go down here and let's say
2:31:56we want to do select everything from
2:31:59employee salary. And let me spell that
2:32:03right.
2:32:04Let's say we want to look at all the
2:32:06salaries just like how we have it now.
2:32:08But in a column next to it, we also want
2:32:10to compare it to the average salary for
2:32:13everyone. So we'll be able to see, you
2:32:15know, whether somebody's salary is above
2:32:16average or below average. So what we
2:32:19would try to do potentially is do
2:32:22something like uh first name salary and
2:32:26average salary. And we try to run this.
2:32:30And of course, we're going to get an
2:32:31error. It's going to basically tell us
2:32:33that we need to group by if we're doing
2:32:35this. So, let's go back down and let's
2:32:38actually add that group by.
2:32:41And we'll say group by first name and
2:32:43salary.
2:32:45And we'll look at this output.
2:32:48And this is not looking good at all.
2:32:50It's just looking at the average salary
2:32:52for each unique row, which is Leslie
2:32:5575,000. So, the average is 75,000. This
2:32:58is not what we're looking for. This is
2:32:59not what we want. Here's what we really
2:33:02do want. we want to just take the
2:33:04average salary of this entire column
2:33:07regardless of group by or anything else.
2:33:09So let's get rid of this and let's see
2:33:10how we can do that. So let's come right
2:33:13down here. We're going to say select
2:33:17select the average salary and then we're
2:33:19going to say from add our parenthesis
2:33:22because this is our subquery from the
2:33:25employee salary table just like that.
2:33:28Now if we run this we should get the
2:33:30exact output we're looking for. So the
2:33:33average salary is 57,250
2:33:38and we have our salary right here. So we
2:33:40can compare really quickly just like
2:33:42that. Now we can also use a subquery in
2:33:45the from statement. So let's go down
2:33:48here and let's say select everything
2:33:51from employee_demographics.
2:33:55Let's have it autocomplete for me. So we
2:33:57have the employee demographics table.
2:33:59Now let's create a group by based off
2:34:01the gender column and add some
2:34:02aggregated functions and I'm going to
2:34:04show you how you can use this as a
2:34:06subquery. So let's go up here say gender
2:34:10and then we'll let's go ahead and add
2:34:12our group by. So we'll say group by
2:34:14gender as well. Now let's add a few
2:34:16things. We'll do average we'll do
2:34:18average age and then we can do let's
2:34:21just do all of them based off the age.
2:34:23We'll just do age,
2:34:26min of age, and count
2:34:31of age.
2:34:33When I try to write fast, it doesn't
2:34:35always go right. So, we have this. Let's
2:34:37run this. And this is what our output is
2:34:40going to look like. Now, what if we
2:34:41wanted to get the average of the oldest
2:34:44age or the average of the smallest ages
2:34:47or, you know, see what the average count
2:34:49is for males and females? Well, we can't
2:34:52do that given this table. But let's do
2:34:56something right here. Select everything.
2:34:58And then we're going to say from and in
2:35:01our from statement, we're going to have
2:35:03a parenthesis. We're going to paste our
2:35:05select statement and then close the
2:35:07parenthesis. So, we're going to select
2:35:09everything from this output that is
2:35:13right down here. So if we run just this,
2:35:17we're going to get an error and forgot
2:35:19this was going to happen. But every
2:35:20drive table must have its own alias. So
2:35:23you have to name a table. I forgot it
2:35:25does that. All we have to do to fix this
2:35:27is just name it. So we'll say as and
2:35:29we'll say aggregated table. We'll just
2:35:32call it aggregated table. So let's run
2:35:34this. And we get the exact same output.
2:35:37But here's the neat thing is we can now
2:35:40select we can do gender. And these are
2:35:43actually the column names now. So I can
2:35:45do the average of this column right
2:35:49here. But I can't do it just like this
2:35:53because it's going to give us an error.
2:35:55And I'll show you why in just a second.
2:35:57Says unknown column age in field list.
2:36:00So what it's saying is is we're trying
2:36:02to perform an aggregated function on the
2:36:05aggregation of an age column, but we
2:36:08don't have an age column in our table
2:36:11right here. Let me run this again. We
2:36:14have a column named this exact thing. So
2:36:17what we actually need to do is do this
2:36:19back tick and back tick. This is the
2:36:21actual name of the column. It's not an
2:36:23aggregation anymore. The back tick on my
2:36:26laptop is right above the tab on the far
2:36:28left hand side. Um right under the
2:36:30escape. That's where mine is. Uh so
2:36:32these back ticks, it's not a quote like
2:36:35this. It's a back tick. So you just need
2:36:37to find that on your keyboard. But now
2:36:39if we run this, it looks like we
2:36:41encountered another error. Says in
2:36:43aggregated query without group by.
2:36:45That's right. Now we need a group by. So
2:36:48now we need a group by gender.
2:36:51Sometimes you got to figure this out on
2:36:52the fly and it should work. There we go.
2:36:57So now we can perform aggregations on
2:37:00this table. Now, this doesn't actually
2:37:02work perfect because we're still
2:37:04grouping by the female male. But let's
2:37:06get rid of this for a second. And we'll
2:37:08get rid of this group by entirely. And
2:37:10if we run this, we're now looking at the
2:37:13averages of this column right here, max
2:37:16age. Now, when you're doing something
2:37:18like this, it's actually really smart to
2:37:21rename these. We'll say as average age.
2:37:25We'll say as max age. And it makes it so
2:37:28much easier. you don't have to do these
2:37:30back ticks anymore. Um, and as min age
2:37:34and so on and so forth and I would
2:37:36probably format this better and stuff
2:37:38like that. Well, we don't have to go
2:37:39through everything, right? I'm just kind
2:37:41of giving you an example.
2:37:43But then when we're using this table,
2:37:46these columns are actually named this.
2:37:49So I don't have to do these back ticks
2:37:50anymore. I can just take this whole
2:37:53thing. Oops. Get rid of that back tick.
2:37:56Now I can just take this column because
2:37:58this is the column name. So let's go
2:38:00ahead and run this and it's still going
2:38:02to work perfectly. So this one's pretty
2:38:04cool because you're basically creating
2:38:06this kind of like a temp table. Um
2:38:08you're just creating your own little
2:38:10output. Then you can query off of it and
2:38:12you can do you know more advanced
2:38:14calculations this way. It's actually
2:38:15really useful. But there are better ways
2:38:18to do something like this uh like a CTE
2:38:21or a temp table that we'll look at in
2:38:23the advanced series. But this is at
2:38:25least how you can do it and you can
2:38:27actually try it out using subqueries. So
2:38:29that is all we're going to look at today
2:38:31for sub queries. In the next lesson,
2:38:33we're going to take a look [music] at
2:38:34window functions.
2:38:47Hello everybody. In this lesson, we're
2:38:49going to be taking a look at window
2:38:51functions. Now, window functions are
2:38:53really powerful and are somewhat like a
2:38:55group by, except they don't roll
2:38:57everything up into one row when
2:38:59grouping. Window functions allow us to
2:39:01look at a partition or a group, but they
2:39:04each keep their own unique rows in the
2:39:06output. We're also going to look at
2:39:07things like row numbers, rank, and dense
2:39:09rank at the end of this lesson. So,
2:39:11before we jump into writing a window
2:39:13function and seeing how the syntax
2:39:15works, let's actually write out a group
2:39:17by, and then we'll compare the two when
2:39:19we actually do write the window
2:39:20function. Let's say we want to take this
2:39:21demographics table and we want to take
2:39:23this gender and compare it to the actual
2:39:25salaries. So what we actually need to do
2:39:28is we need to say join and we're going
2:39:31to join on the employee salary. Let's go
2:39:34like this.
2:39:36Get rid of all of this and we'll do
2:39:38salary and we're going to say on and
2:39:42let's do dm and sal for the aliases.
2:39:46We'll say dm.mp employee id is equal to
2:39:51sal employee id. Now we're going to come
2:39:54up here and we're going to say gender
2:39:57comma and we want to look at the average
2:39:59salary. And we need to get rid of this
2:40:01right here and we need to come down to
2:40:03the bottom and say group by gender. Now
2:40:07let's go ahead and run this query. See
2:40:10if it works. And it did. So we have our
2:40:12gender and we have our average salary
2:40:15from our salary table. And we can rename
2:40:18this as average and we'll do average
2:40:22salary. Just like that. So this is how
2:40:25group by works. It rolls everything up
2:40:28into one row. Now let's try doing
2:40:31something pretty similar except we're
2:40:34going to use a window function. Let's
2:40:36come right down here and let's paste
2:40:38this and let's start writing out our
2:40:40window function. Now we don't have to
2:40:41use the group by. We're going to go
2:40:43ahead and get rid of that. And right
2:40:45here for gender, we can keep that the
2:40:47exact same. All we're really going to
2:40:50change is this part right here. We're
2:40:52going to say average salary. And that is
2:40:55part of creating a window function.
2:40:58Typically with a straightforward window
2:40:59function, all we have to put is over
2:41:02with a closed parenthesis. This is going
2:41:04to say we're looking at the average
2:41:06salary over and normally in here you'll
2:41:09specify something and we'll get to that
2:41:10in a little bit, but we're just going to
2:41:11look at an average salary over
2:41:13everything. So let's go ahead and run
2:41:16this output. So this is going to look a
2:41:18little bit different, right? So the male
2:41:20and female all have their own individual
2:41:24rows, which is not the same as group by.
2:41:26And this average salary is looking at
2:41:28the average salary of everybody. We're
2:41:31not breaking it out by the gender like
2:41:34we did up here. Here we rolled it up.
2:41:37Now we're looking at the average salary
2:41:39for the entire column. Now what we can
2:41:42do is actually partition by. Now
2:41:44partition by is going to separate it out
2:41:46kind of like grouping it. So let's say
2:41:48partition
2:41:50partition by and we'll say gender. So
2:41:53just like when we did the group by, the
2:41:55group by rolled everything up into one
2:41:57row. This is not going to roll
2:41:59everything up, but it is going to
2:42:00perform this calculation based off of
2:42:03the different genders, the unique values
2:42:05in this column. Let's go ahead and run
2:42:07this.
2:42:09And if you'll notice, the female is
2:42:1153,750,
2:42:13the male 57,428.
2:42:16Now, let's go and compare these. I'm
2:42:17going to run this and this query. Let's
2:42:20run this.
2:42:21So if we look at our group by, it's the
2:42:24exact same numbers except we have it on
2:42:28their own individual rows. Now, why
2:42:30would we want this? Well, let's say we
2:42:33want additional information. So let's
2:42:35just look at this one for now. So in
2:42:38this one, let's say we wanted to add
2:42:39additional things like the first name.
2:42:42So we'll do demirst_ame.
2:42:46We can do last or dem.last_ame.
2:42:50So, we can add other information and it
2:42:53doesn't affect this column at all
2:42:56because we're using a window function.
2:42:58If we try to add these exact things, and
2:43:00I'm going to go up here and do it. If we
2:43:02try to add these exact things to this,
2:43:05let's see if um yeah, that works. And
2:43:08then we also have to group by this.
2:43:12If we run this query now, it's going to
2:43:14be completely different because we're
2:43:17using a group by. We're grouping by the
2:43:19first name, the last name, and the
2:43:21gender. We're breaking everything out
2:43:23based off of the unique values in these
2:43:25columns. Whereas down here,
2:43:29it's completely independent
2:43:32of what's going on in these other
2:43:33columns. All we're doing is we're doing
2:43:35a window function just based off of that
2:43:38column. So, I think that's pretty
2:43:40amazing. And there's a lot of additional
2:43:43functionality that we can do with these
2:43:45window functions. And we're going to
2:43:47take a look at a lot of those things in
2:43:48just a little bit. Let's try another
2:43:50example really quickly. Let's literally
2:43:52just copy this, paste it down here. And
2:43:55all we're going to do is we're going to
2:43:56change this to sum.
2:43:59So now instead of the average salary,
2:44:02we're looking at the sum of salaries and
2:44:05we're still partitioning by the gender.
2:44:07Let's go ahead and run this and let's
2:44:09pull this up.
2:44:11So, all the men together make $42,000.
2:44:16All the females make $215,000.
2:44:20Now, what we're about to do is something
2:44:22called a rolling total. If you've never
2:44:25heard of a rolling total, a rolling
2:44:26total is super cool and can be done
2:44:29within my SQL. A rolling total is going
2:44:32to start at a specific value and add on
2:44:34values from subsequent rows based off of
2:44:37your partition. So, all we have to do is
2:44:39add an order by. And we're going to
2:44:42order by let's say the employee
2:44:44ID. Let's go ahead and take a look at
2:44:46this. And it looks like the employee ID
2:44:49is ambiguous. I had a feeling. So, I
2:44:52just need to say uh dem employee ID.
2:44:56Let's try this one. So, now we have
2:44:59something called a rolling total. I'm
2:45:00going to actually name it as
2:45:02rolling_total
2:45:05because this is super cool. that window
2:45:08functions can do this and this is
2:45:10something that a lot of people in like
2:45:12finance do. I did it myself when I
2:45:14worked in healthcare and it partitions
2:45:17based off the female and you can't see
2:45:19the employee ID but there's an employee
2:45:20ID that we're kind of ordering on in the
2:45:22background. Now what it's doing is it's
2:45:24starting with Leslie nope and she made
2:45:2675,000 then the next person April she
2:45:29made 25,000 which equals 100. And just
2:45:32to actually see this better, I'm going
2:45:34to add salary.
2:45:36And so Leslie Nome had 75,000. Then
2:45:40we're adding this 25,000 to the 75 and
2:45:42we get 100. Then we're adding the 60,000
2:45:46to 160,000. Then we're adding 55,000 to
2:45:49215,000. So we're adding every single
2:45:53time we're adding this salary to the
2:45:56already existing total all the way up to
2:45:59our grand total, which was 215,000. The
2:46:02exact same thing happens with the males.
2:46:04So, we start with 50,000, then we add
2:46:0650, then we add 90, then we add 70, all
2:46:09the way up to 42,000. Now, you can do
2:46:11this in a lot of different
2:46:12configurations on a lot of different
2:46:14columns, but in essence, this is exactly
2:46:17what a rolling total is. That's how it
2:46:19works. And we were able to partition
2:46:21based off of this column. We don't have
2:46:23to use partition by. We could do this
2:46:25completely regardless of the partition,
2:46:27but I thought it was interesting to at
2:46:29least break it out by female versus
2:46:32male. So, now that we know how to use a
2:46:34window function, let's look at some
2:46:36special things that you can really only
2:46:38do with window functions or window like
2:46:41functions. So, we're going to bring this
2:46:42down and what we're going to do is get
2:46:44rid of this entire thing and we're going
2:46:47to look at something called row number
2:46:49and we're going to look at rank and then
2:46:50we'll look at dense rank. So let's look
2:46:52at row number. And this is just like an
2:46:57aggregate function like we're doing the
2:46:59average age or average salary or
2:47:01something like that. This is what we're
2:47:02doing. We're doing a row number. Now
2:47:04we're going to do this over and we'll
2:47:06just do everything for right now. So
2:47:08let's go ahead and run this and just see
2:47:10what it looks like. Let's bring it up.
2:47:13And what we're doing is we're saying,
2:47:15okay, we have first name, last name,
2:47:17gender, salary. That's all great. But
2:47:18then we get to row number. and we're
2:47:20doing a row number based off of
2:47:22everything. It doesn't matter what it
2:47:24is. So, we're starting at one, which is
2:47:26the very first row, and we go all the
2:47:28way down to the bottom, just like an
2:47:30employee ID. So, let's actually add
2:47:32that. Let's do dm
2:47:34employee id just like this.
2:47:38So, we have this 1 2 3 4 5 6 7 8 9 10
2:47:4111. Now, if you remember on this table,
2:47:44we are missing Ron Swanson. So, it kind
2:47:47of skips that, but it's basically like
2:47:48an employee ID. We're kind of giving it
2:47:50its own unique value. And these row
2:47:53numbers are not going to repeat itself
2:47:55if you do it like this. Now, they can
2:47:58repeat themselves if we do a partition.
2:48:00And let's do a partition on the gender
2:48:02again because we know how to do that
2:48:03one. We'll do partition
2:48:06by, let me spell that right, partition
2:48:08by the gender. Now, we're going to add a
2:48:10row number based off the gender, but
2:48:12again, it's broken out or partitioned by
2:48:15gender. Let's look at this.
2:48:18Now it goes for the females 1 2 3 4.
2:48:22Then for the males it restarts 1 2 3 4 5
2:48:266 7. Now this is just in a random order
2:48:29based off how the you know data was
2:48:30stored in the table itself. Now what if
2:48:33we wanted to kind of rank these based
2:48:35off of the highest salary first down to
2:48:37the lowest salary? You nailed it. We
2:48:40just add an order by. So we'll order by
2:48:43salary. And if we want to do it from
2:48:44highest to lowest, giving the highest
2:48:47salary, the number one, and the lowest
2:48:49salary, you know, later down, we'll do
2:48:51descending.
2:48:52And let's run this. And you'll see that
2:48:55for female, we're still partitioning by
2:48:57gender. For female, the highest salary
2:48:59is one. Next is two, three, and then
2:49:01four. Then for males, the highest salary
2:49:04is one, all the way down to seven. So
2:49:06that's what row number does. just gives
2:49:08a row number based off of whatever
2:49:10you're partitioning by or ordering by in
2:49:13your window function. Now, let's go over
2:49:15here and add a comma. And we're going to
2:49:17add and let's go down just a hair. Let's
2:49:22add rank. So, I want to do rank and
2:49:25we'll do our parenthesis. Now, rank is
2:49:27going to give it more of an official
2:49:29rank. And let's see how this works. So,
2:49:32we'll do rank and we'll do over
2:49:34partition by salary descended. The exact
2:49:36same thing.
2:49:39And while we're here, I'm going to
2:49:41rename these. I'm going to say as
2:49:43as row_num
2:49:46and we'll call this one uh rank num. So
2:49:51let's go ahead and run this. And it
2:49:54looks very very very similar except for
2:49:57one small thing. This right here. So
2:49:59when we're using the row number,
2:50:01whatever we are partitioning by, it's
2:50:03not going to have duplicate rows within
2:50:04that partition. It just won't. So even
2:50:06if there's 50,000 right here, it's just
2:50:09going to automatically assign it based
2:50:10off of something that it's running in
2:50:12the background, whether it's the order
2:50:13of how the data is stored in the table
2:50:15or some other order by that you are
2:50:17using on the table. Now rank is a little
2:50:20bit different because rank is going to
2:50:22take it just like it did the ronum
2:50:24except when it encounters a duplicate
2:50:26based off of the order by which is the
2:50:29salary, it's going to assign it the same
2:50:32number. So this is five and five. What's
2:50:35unique about rank is that the next
2:50:37number is not going to be the next
2:50:38number numerically. It's going to be the
2:50:40next number positionally. So this is 1 2
2:50:443 4 five. This is kind of like a six.
2:50:47And then it goes to seven. So it skips
2:50:50number six. Now there's another one.
2:50:53Let's copy this rank. There's another
2:50:55type of rank called dense rank. And
2:50:58we'll do dense rank. So, we'll do dense
2:51:04rank. And let's run this.
2:51:07And let's pull this up.
2:51:09There we go. Now, dense rank is ever so
2:51:13slightly different than rank in the fact
2:51:14that when it gets down to duplicates,
2:51:17it's still going to duplicate them. So,
2:51:18it's going to have a five and a five,
2:51:20but it's going to give the next number
2:51:23numerically, not positionally. That is
2:51:25the only real difference between rank
2:51:27and dense rank. And again, row number is
2:51:30just not going to have duplicates. It's
2:51:32going to give it its own unique within
2:51:33that partition. So, I know I just threw
2:51:35a lot at you, but that's row number,
2:51:37rank, and dense rank in a nutshell. And
2:51:40you can review this, mess around with
2:51:41it, all of these things, because, you
2:51:43know, these are actually really, really
2:51:44useful. So, that's all we're going to
2:51:46take a look at in this window functions
2:51:47lesson. I hope all of that made sense. I
2:51:49hope you kind of got an understanding of
2:51:51how it can work and how powerful these
2:51:52window functions can be. And this is
2:51:54actually the last lesson in the
2:51:55intermediate MySQL series. Thank you
2:51:57guys so much for watching. I really
2:51:59appreciate it. If you like this video,
2:52:01be sure to like and subscribe and
2:52:02[music] I'll see you in the next video.
2:52:10[music]
2:52:16Hello everybody and welcome to the first
2:52:17lesson in the advanced MySQL tutorial
2:52:20series. Today we are going to be looking
2:52:21at CTE. Now CTE stand for common table
2:52:25expression. They're going to allow you
2:52:27to define a subquery block that you can
2:52:29then reference within the main query.
2:52:32Now, that may not make perfect sense,
2:52:34but we've looked at subqueries in the
2:52:35past or in previous lessons in the
2:52:37intermediate series. So, you kind of
2:52:38understand that it's kind of like a
2:52:40query within a query, except we're going
2:52:42to name this subquery block, and it'll
2:52:44be a little bit more standardized, a
2:52:46little bit better formatted than
2:52:48actually using a subquery. Let's take a
2:52:50look at the basics of writing a CTE.
2:52:53Let's pull this down really quickly. And
2:52:55all we're going to do is we want to
2:52:58create this as a CTE. So we'll say with
2:53:01and that is our keyword to define our
2:53:04CTE. So we're going to say with name our
2:53:06CTE and we'll just call it CTE and we'll
2:53:09do underscore example. And then we're
2:53:12going to say as. So this is how we
2:53:14define it. And now we need to actually
2:53:16put it in parenthesis. Now you can do
2:53:18this in several different ways. I'm
2:53:20going to do it kind of like this just to
2:53:23really emphasize that this is within the
2:53:26CTE. Now, CTE are unique because you can
2:53:29only use the CTE immediately after you
2:53:32create it. So, if we come right down
2:53:34here and we come right below it, we'll
2:53:37say select everything and we're going to
2:53:40say from CTE example. So, we'll say from
2:53:44CTE example. And let's bring this back
2:53:47up. Now, if we run this, we're going to
2:53:49get the exact same output. Now, this
2:53:52should seem pretty familiar, almost like
2:53:54we're using a subquery. And within our
2:53:58subquery, we have this right here. We're
2:54:00kind of building our own little table.
2:54:02And then we can query off of it down
2:54:04below. So, we can come down here and
2:54:06let's actually change the names in here.
2:54:08We're going to say average
2:54:11cell. And we'll change all of these real
2:54:14quick just because don't uh I don't like
2:54:18having to actually put the, you know, uh
2:54:20the tick marks. I don't like doing that.
2:54:22So here we're going to say max, then
2:54:25we'll say min,
2:54:28and then we'll say count. And let's go
2:54:31ahead and run this again. And so now we
2:54:33have these different names. And when we
2:54:35come right here, we can say select, and
2:54:38then we'll just do something really
2:54:39simple. Let's do the average of average
2:54:43cell. So the average salary and let's
2:54:47run this. And so this is the average
2:54:49between both the males and the females.
2:54:52Kind of the purpose of these CTE is to
2:54:54be able to perform more advanced
2:54:56calculations. Something that you can't
2:54:58easily do or can't do at all within just
2:55:01one query. Another reason to use a CTE
2:55:04is just the readability. You can
2:55:06absolutely write this using a subquery.
2:55:08And let's do that really quickly. And
2:55:10it's just going to be a little bit
2:55:11tougher to read and look at. So let's
2:55:14come right up here. We're going to say
2:55:18from and we'll do right here. We'll say
2:55:21select everything. We'll do select
2:55:24average cell
2:55:26from here. And we're going to need to
2:55:27name this. So we'll say um do example
2:55:32subquery.
2:55:34We'll get rid of this. And then we just
2:55:36need to get rid of this. And we can run
2:55:39this query. And we get the exact same
2:55:41output. Now, if I formatted this exactly
2:55:44the same, just like this, and the names
2:55:47down there. If we look at this, the
2:55:49syntax is just a little bit more
2:55:50difficult to read. We're selecting the
2:55:52average of average sal from, and then we
2:55:54have our subquery right here, and then
2:55:56we're naming it at the bottom. If we
2:55:59scroll up and compare this, this one
2:56:01just looks a lot better. Now, when
2:56:04you're writing in my SQL, sometimes it
2:56:05doesn't matter if it looks pretty or
2:56:07not, as long as it gets the job done.
2:56:09That is true, especially if you're just
2:56:10going to be using it yourself. But in a
2:56:12more professional environment, when
2:56:14you're using this in your actual job,
2:56:16they're going to be people who've been
2:56:17using this for 10, 20 years, and they're
2:56:19going to expect you to write it well.
2:56:21They don't want it to be really messy.
2:56:22They aren't most likely going to want it
2:56:24to be written like this. I've been using
2:56:26it for quite a long time and I much
2:56:28prefer CTE over subqueries just visually
2:56:31and it makes it a lot quicker to
2:56:33actually read through. So that is just
2:56:35one of the reasons although you get the
2:56:37exact same output. Now there is some
2:56:39additional functionality within CTE as
2:56:41well. Now one thing that I mentioned
2:56:43just a second ago is that when you build
2:56:45a CTE you can only use it immediately
2:56:48after. You can't use it right below it.
2:56:50So, let's go ahead and let's copy this
2:56:52query and we're going to bring it right
2:56:54here. If we try to run this and let's do
2:56:57this. We're going to get an error and
2:57:00let's pull this up. It says table parks
2:57:03andrec.ct
2:57:04example doesn't exist. So, we're looking
2:57:07for a table called CTE example in our
2:57:11database, but it's not there. Now, the
2:57:13reason this happens is because you're
2:57:15creating a CTE. You're not creating a
2:57:17permanent object like a temp table,
2:57:19which we'll look at in the next lesson.
2:57:21And you're not creating a real table and
2:57:23you're not creating a view. You're
2:57:24really not creating anything. It's just
2:57:26a common table expression to create this
2:57:29table right here. This basically almost
2:57:31like a temporary table almost, but then
2:57:35you're just using it to query off of it.
2:57:37You're not saving it. You're not storing
2:57:39it in memory. You're not really doing
2:57:41anything with it. It's just like writing
2:57:42a regular query. So this is why you can
2:57:45only write it immediately after creating
2:57:47the CTE. You can't write it down below
2:57:49and reuse it because it's just like
2:57:51calling a query that you wrote before.
2:57:53It just isn't going to work. Now the
2:57:55next thing that I want to take a look at
2:57:57and let's copy this down here. Next
2:58:00thing I want to take a look at is that
2:58:01you can actually create multiple CTE
2:58:05within just one. And so if we wanted to
2:58:07do a more complex query or joining more
2:58:10complex queries together, we can do that
2:58:12all within one CTE. So let's come right
2:58:15here. Let's get rid of all of this.
2:58:19And we're going to say uh from the
2:58:22demographics table, we're going to say
2:58:24where birth date and let's just do as
2:58:28larger than 1985-1.
2:58:32So we have one query and we'll take just
2:58:36a few columns from this table. So we'll
2:58:39take let's say the employer or employee
2:58:42ID. We'll take the gender and the birth
2:58:46underscore date. So this is one query
2:58:49and we're filtering just based off of
2:58:51this birth date. Now when we create
2:58:53this, this is the CTE example, but we
2:58:57can have a comma here. We can come down
2:58:59below and then we can say ct
2:59:03example 2 and I need to combine that. So
2:59:06two and then we can say as and then we
2:59:09have another query. So then right here
2:59:12we could say select everything. We'll
2:59:15change that in a second from employee
2:59:18say salary
2:59:19and in the salary we'll just do a simple
2:59:22one. We'll do where salary is greater
2:59:25than 50,000.
2:59:28And we'll actually just take the
2:59:31employee id and the salary. Now if I
2:59:36come right down here, I can say select
2:59:38everything from CTE example which is and
2:59:40let me scroll up so we can see
2:59:41everything.
2:59:43That's our original. Our CT example is
2:59:45this first query right here. Then we're
2:59:48creating our second one right here. And
2:59:51we can join basically on these two
2:59:54common table expressions. So now we can
2:59:56say join and then we'll do CTE example
3:00:022. I need to change that X
3:00:05and then we'll say on then
3:00:08[clears throat] we're just going to do
3:00:09TT example
3:00:12employee ID is equal to TTample2
3:00:17employee
3:00:19ID and not an equal sign but a dot.
3:00:22There we go. Now, if we run this, it
3:00:25should work and we can pull this down
3:00:27and look at our output. Now, this is
3:00:29just an example. This isn't a real use
3:00:31case because, of course, we could just
3:00:33join these two tables together normally,
3:00:36but you can imagine you have a much more
3:00:37complex query or you're doing a lot of
3:00:40functionality within this table and you
3:00:42just want a certain subsection of this
3:00:44table and you're wanting to combine
3:00:46those. This is how you can do that with
3:00:48a CTE. So now we have all of our
3:00:51information right here. And that can be
3:00:53extremely extremely helpful. Now, one
3:00:55last thing that I want to show you.
3:00:56We're going to go all the way back up
3:00:58really quickly right here. Let's run
3:01:01this one one more time and let's
3:01:04actually take everything
3:01:07and let's run this. So here we have our
3:01:11gender, average salary, max salary,
3:01:13men's salary, and count salary. The last
3:01:15thing that I want to show you, and this
3:01:17is more of something that's just
3:01:18somewhat helpful, you don't have to
3:01:20actually do it in your main query, is
3:01:22before we went in here and we changed
3:01:24all the column names by doing an alias
3:01:27by saying as and then saying the average
3:01:29salary. And the as is just implied here.
3:01:32But we're changing this via an alias. We
3:01:34don't have to do this. In fact, we could
3:01:36come right here and we could do a
3:01:38parenthesis. We could call it gender.
3:01:41We'd call it average salary, max salary,
3:01:47miners
3:01:49salary, and let's do countd
3:01:54salary. So now, if we were to run this,
3:01:56let's change this up. We'll do capital
3:01:58on this one. If we wanted to run it like
3:02:00this, when we run this, it'll change all
3:02:02of those names to what we have it right
3:02:04here. So this will be the default. This
3:02:07will overwrite the column names that you
3:02:09have in your actual CTE expression or
3:02:12the query that you have within your CTE.
3:02:14So, that is all we're going to take a
3:02:15look at in this lesson on CTE. These are
3:02:18very, very helpful, definitely help with
3:02:20more complex queries, and they're just
3:02:22really easy to read and understand,
3:02:24which is why I personally use them a
3:02:26lot. In the next lesson, we're going to
3:02:28be taking a look at temp tables, and
3:02:30we'll also compare temp tables to CTE,
3:02:32and we'll [music] take a look at the
3:02:33difference.
3:02:46Hello everybody. In this lesson, we're
3:02:48going to be taking a look at temporary
3:02:50tables. Now, temporary tables are tables
3:02:53that are only visible to the session
3:02:55that they're created in. So, if I create
3:02:57a temp table right now and I exit out of
3:03:00my SQL and I come back in, it's not
3:03:02going to be there anymore. And we'll
3:03:03look at that in just a little bit. Now,
3:03:05temporary tables can be used for a lot
3:03:06of things, but how I've mostly used
3:03:08them, especially as a data analyst, is
3:03:10for storing intermediate results for
3:03:12complex queries, somewhat like a CTE,
3:03:15but also for using it to manipulate data
3:03:18before I insert it into a more permanent
3:03:20table. So, let's take a look at how we
3:03:22can create a temp table. There's two
3:03:24ways that you can do it. I'll show you
3:03:25the first way, which I don't think is as
3:03:27popular, and then I'll show you the
3:03:28second way, which is how I typically use
3:03:30it the most. Now, the first way to
3:03:32create a temp table is to create a
3:03:36temporary
3:03:38table. I need to sound it out like that.
3:03:39It's the only way I can spell. So, we're
3:03:41going to do temp table. So, this is our
3:03:43name. Now, if we just took this out and
3:03:46we created a table, this would create a
3:03:48table in our parks and recreation
3:03:50database. But we don't want that. We
3:03:53want to create a temporary table that
3:03:55just lives inside of our memory or the
3:03:57memory within our computer. Now, we're
3:03:59going to create this temporary table
3:04:01much like we would a regular table. And
3:04:03we're going to need to name the columns
3:04:05as well as the data types. So, let's do
3:04:07first name. And our data type can be
3:04:10varchar. Let's say 50. And we'll do a
3:04:13comma. Then we'll do last name. We're
3:04:16going to keep this really simple. We'll
3:04:17do varchchar 50 again. And then for our
3:04:21last one, we'll do favorite
3:04:24movie. And for this one, it needs to be
3:04:26longer. So we'll do varchchar let's say
3:04:28100. Now let's get rid of this and let's
3:04:31actually run this after we do our
3:04:32semicolon. Let's actually run this and
3:04:36nothing's going to happen. Let's click
3:04:37refresh. Nothing's going to happen. At
3:04:38least you can't see it happening. Let's
3:04:41pull this up. And you can see that
3:04:43create temporary table says zero row is
3:04:46affected, but it was created. Now in
3:04:48order to actually see it, we can do
3:04:50select everything. And we'll do this
3:04:53from our temp table. And we'll add a
3:04:56semicolon. And then we run this. And we
3:04:59have this empty table right here. Now,
3:05:02what's really great about these temp
3:05:03tables is then you can insert data into
3:05:06it. And it basically is like a real
3:05:08table except it just lives in memory and
3:05:11they go away after a while. But you can
3:05:13reuse this temp table over and over and
3:05:16over again. Now, let's insert some data
3:05:18into here and then we'll take a look at
3:05:20this again. So, let's come right down
3:05:21here. Let's insert data. We'll do insert
3:05:24into and we want to insert that into the
3:05:26temp table. And we're just going to say
3:05:28values. Now, we just say values. I'll
3:05:30use myself for this one. We'll do uh
3:05:32Alex Freeberg. And what's my favorite
3:05:36movie? Give me a comma. That'll be Lord
3:05:39of uh I think it's like that. Lord of
3:05:41the Rings, the Two Towers.
3:05:46Uh it's probably my favorite movie of
3:05:47all time. Now, let's go ahead and insert
3:05:49this data.
3:05:51And let's pull this down here. And let's
3:05:53run it all the way down here after we
3:05:56add our semicolon. And when we run this,
3:05:59you'll notice that now we have data in
3:06:01here. So now we can use this table much
3:06:03like any real table. So that's the first
3:06:06way to create a temp table. Not my
3:06:08personal favorite way, although there
3:06:10have been some use cases where I've done
3:06:12it like that. I'm going to show you the
3:06:13way that I typically do it. And for
3:06:16this, let's select everything from the
3:06:19employee
3:06:20salary table. Let's run this. Now, let's
3:06:24say I just wanted a subsection of this
3:06:26data to sit in this temp table where the
3:06:29salary is greater than let's say 50,000.
3:06:32I could easily easily do this. I'm going
3:06:33to say create temporary table and let's
3:06:38do this one as salary over
3:06:4150k. Now, one thing about naming either
3:06:45temp tables or CTEs or sub queries or
3:06:47any of these things where you need to
3:06:48name something, I try to typically name
3:06:50it something that actually makes sense.
3:06:53So, the salary over 50k is something I
3:06:55would actually name it in my real work.
3:06:58I wouldn't normally name it something
3:06:59like tempt table. The reason for that is
3:07:02because when you're in a work
3:07:03environment and you have lots of temp
3:07:05tables, you're creating really advanced
3:07:06store procedures, really advanced
3:07:08queries, you have hundreds or even
3:07:10thousands of tables and different
3:07:11databases, it gets really complex. So
3:07:14naming conventions are actually pretty
3:07:16important or they become more important
3:07:18uh the more you get entrenched in this
3:07:19stuff. So just something to think about.
3:07:22Now we're creating this temp table. Now,
3:07:23we don't have to really insert data into
3:07:26it more than we're just going to select
3:07:28data from an already existing table. So,
3:07:31I'm going to say select everything from
3:07:35I'm going to say employee salary
3:07:38and we're just going to say where the
3:07:40salary is greater than 50,000. Now, I
3:07:44want uh Tom and Jerry, I want them to be
3:07:47included as well. So, I'll actually say
3:07:49greater than or equal to. So now we're
3:07:52creating a temporary table based off of
3:07:54an already existing table and we're just
3:07:56selecting data into this temporary
3:07:59table. So when we run this now we can
3:08:03select the salary over 50k and let's run
3:08:08this and it works perfectly. Now the
3:08:12great thing about tempt tables is they
3:08:13last as long as you are within that
3:08:15session. Meaning if I copy this query,
3:08:18let's go to a new window and let's paste
3:08:21this in here and let's zoom in a little
3:08:24bit and let's run this. It still works
3:08:27even in a new window. But if I'm to exit
3:08:30out and come back in, then it is no
3:08:33longer going to be working. Now, let's
3:08:35exit out of this. Let's come back in and
3:08:36we'll see if these temp tables still
3:08:38work. Let's go ahead and exit out. Oh
3:08:40jeez, I'm embarrassed. All right, let's
3:08:42go to my SQL.
3:08:45Let's come over here to the local
3:08:46instance. So now it pulls right back up.
3:08:49Zoom in once again on both these and
3:08:51let's try to pull up our salary over 50k
3:08:54temporary table. Let's run this. And
3:08:57we're not getting an output. Let's go
3:09:00back. It's going to say error code. The
3:09:02table salary over 50k does not exist. So
3:09:05it only lasted as long as we were within
3:09:08this session. So that is how we create
3:09:11our temp tables and that's how we use
3:09:13our temp tables. Now in the last lesson
3:09:15we had looked at CTE. CTE and tempt
3:09:18tables both have their own use cases
3:09:20within my SQL. For temp tables this is
3:09:22usually for the more advanced things. So
3:09:25I'm usually using these in store
3:09:26procedures when I'm really manipulating
3:09:29data and I'm doing a lot more complex
3:09:31queries overall and oftent times I'll
3:09:33use multiple temp tables and I'm joining
3:09:35them together and I'm just doing a lot
3:09:36of more advanced stuff. With CTE, it's
3:09:39typically more simple things because you
3:09:41can't make as advanced CTE or as complex
3:09:44CTE. So with those, I'm usually keeping
3:09:47it to just one level of transformation.
3:09:49I have my base CTE or my base subquery
3:09:52or query, however you want to call that,
3:09:54and I'm changing it or doing one level
3:09:57of advanced thing on top of that query.
3:10:00That's what a CT is really great for.
3:10:02Temp tables, you can just get a lot more
3:10:04advanced with it. They also last within
3:10:05the session. And if I'm using it
3:10:07multiple times throughout something like
3:10:09a store procedure, then it makes so much
3:10:12sense to use a temporary table. So this
3:10:14has been our lesson on temporary tables.
3:10:16In the next lesson, we're going to be
3:10:18taking a look at string functions.
3:10:20[music]
3:10:32Hello everybody. In this lesson, we're
3:10:34going to be taking a look at stored
3:10:35procedures. Store procedures are a way
3:10:37to save your SQL code that you can reuse
3:10:39over and over again. When you save it,
3:10:42you can call that stored procedure, and
3:10:43it's going to execute all the code that
3:10:45you wrote within your store procedure.
3:10:47It's really helpful for storing complex
3:10:50queries, simplifying repetitive code,
3:10:52and just enhancing performance overall.
3:10:54So, let's take a look at how we can
3:10:55create a stored procedure. Now, we're
3:10:57going to start by just creating a really
3:10:58simple query. We'll make it a little bit
3:11:00more advanced as we go along and take a
3:11:02look at the different things within
3:11:04store procedures that you can do. Now,
3:11:06let's change this query. Let's say where
3:11:09the salary is greater than let's do
3:11:1250,000. Let's actually do greater than
3:11:14or equal to 50,000. We want to include
3:11:17Tom uh and Jerry as well. So, let's go
3:11:19ahead and run this. Now, what we want to
3:11:21do is save this really complex code
3:11:24within a store procedure. Let's come
3:11:26right down here and we can create a
3:11:29super super super simple store procedure
3:11:32by just saying create procedure
3:11:36and pasting that. Now we just have to
3:11:38name it. So we have create procedure and
3:11:40we'll call this large
3:11:43salaries and then we do a closed
3:11:46parenthesis. Now this is as simple as it
3:11:49can possibly be. It does not get any
3:11:51simpler than this. So let's go ahead and
3:11:52run this. And if we go down, we pull
3:11:56this up, you can see that it says create
3:11:58procedure, zero rows affected. So it
3:12:00looks like it worked. And if we come
3:12:02over here to this refresh button, you
3:12:04should see now that under store
3:12:06procedures, it drops down and we have
3:12:08our large salaries. That's exactly what
3:12:10should have happened. We wanted to save
3:12:12that into our parks and recreation. Now,
3:12:14if you wanted to be careful, you could
3:12:16say use parks
3:12:20recreation. This is not a bad idea, but
3:12:23you don't have to. But you can specify
3:12:25what database within your actual editor
3:12:27window. Sometimes that is helpful. But
3:12:30now we've created it. Now let's see how
3:12:32we can call it. All we have to do is say
3:12:34call. We're going to copy this entire
3:12:37thing including the parenthesis.
3:12:39And let's end it with that's right, a
3:12:43semicolon. Let's go ahead and run this.
3:12:45And as you can see, it worked because we
3:12:47got the exact output. So we actually
3:12:49called this store procedure and this
3:12:52code ran. So it's just a select
3:12:54statement. So it worked perfectly. Now
3:12:56you can also come over here to large
3:12:58salaries and there's this little tiny
3:13:00little button here that looks like a
3:13:01lightning bolt. And if you click it,
3:13:03it's going to open up a different window
3:13:05and we'll say call parks and
3:13:07recreation.large salaries. So you can do
3:13:10it that way as well, but uh we're not
3:13:12going to be doing it that way. Now what
3:13:13we've written right here is not best
3:13:16practice by any means. And I'm going to
3:13:18copy this down here because there's a
3:13:21lot of different things that you need to
3:13:22take into account when you're creating a
3:13:24store procedure. For example, this right
3:13:27here is most likely not what you're
3:13:29going to be putting into a store
3:13:30procedure. This is super super simple.
3:13:32Typically, you'll be having multiple
3:13:34queries. And let's see what happens if I
3:13:37try to put another query in here. And
3:13:39let's get rid of this. So, we're going
3:13:41to select everything where the salary is
3:13:43greater than 50,000. Then we'll select
3:13:45everything where it's greater than
3:13:4610,000 which is everybody. Let's call
3:13:48this large salaries 2. So we have two
3:13:51different statements in here and we want
3:13:53them all to be under this large salaries
3:13:55too. Let's select everything and let's
3:13:58run this and we're getting an output
3:14:01which is already not a good sign but we
3:14:03created the store procedure and then we
3:14:06selected everything. So what's actually
3:14:09happening here? Pull this back down.
3:14:12What's happening is is this is creating
3:14:14the store procedure and this is just
3:14:16some other you know random query. But
3:14:19that's not what we want. What we want is
3:14:21everything or both of these queries
3:14:23within one store procedure. The best
3:14:25practice is to use something called a
3:14:27delimiter. Now this right here is a
3:14:29delimiter. The semicolon. So the
3:14:32semicolon separates our queries from one
3:14:34another. It tells my SQL hey you know
3:14:36this is a different query. Don't be
3:14:38mixing these and cause errors. You know,
3:14:41that's essentially what a delimiter
3:14:42does. Now, we can change the delimiter
3:14:45by coming up here and saying delimiter,
3:14:48and we can change it to almost anything
3:14:50we want. Now, in my actual job, I've
3:14:52seen it done many different ways. I've
3:14:54seen these forward slashes. I've also
3:14:56seen dollar signs. This is probably the
3:14:58one that I've seen the most when I
3:14:59worked with data engineers, data
3:15:01scientists, database developers. You
3:15:03This one I see a lot. And then you'll
3:15:05come into the code and you'll say begin.
3:15:09And let's go over here and let's tab all
3:15:12of this. And then we'll say end. Now
3:15:15when we end, we're going to end it with
3:15:18this dollar sign. So here's what's
3:15:20happening. We're changing the delimiter
3:15:21right here to dollar sign. We're
3:15:23creating our store procedure and within
3:15:25it, we are keeping all of this. So all
3:15:29of this code is going to go into this
3:15:31one stored procedure. Then at the end,
3:15:34we are saying this is the end right here
3:15:37of this stored procedure. These
3:15:39semicolons no longer are the delimiter
3:15:42that's telling us when it is the end of
3:15:44the store procedure. That's what the
3:15:45delimiter does. Now, it is best practice
3:15:47at the end to change it back, right? Uh
3:15:50let me spell it right because if you
3:15:52don't, then you're going to have to
3:15:54start using uh these dollar signs for
3:15:56everything. And how do you spell
3:15:58delimiter? Oh man, there we go. Now,
3:16:01we've changed it back to a semicolon
3:16:04afterwards. So, then we can go and write
3:16:05other queries and it'll um act
3:16:07appropriately. Uh, let's go down. So,
3:16:10this is getting closer to best practice.
3:16:14Let's go ahead and run this entire
3:16:16thing.
3:16:17And if we pull this up, we're not
3:16:19getting an output. That's a good sign.
3:16:20If we pull this up, it's saying we
3:16:22already created number two. Change that
3:16:25to three. My apologies. Let's go down
3:16:27here.
3:16:29Now, we've created the store procedure
3:16:31three. Now, let's go over here. We're
3:16:33going to rightclick on this. We're going
3:16:36to say alter stored procedure. And now
3:16:39you can see that we have both of these
3:16:41queries within this stored procedure.
3:16:44Let's get rid of this. And we're going
3:16:45to go and call this. So let's copy this
3:16:49large salaries three.
3:16:51Bring this all the way down.
3:16:54And let's say call
3:16:58that store procedure. If we run it,
3:17:00you'll notice we get two outputs. We
3:17:03have six and seven. This result six is
3:17:06where it's greater than 50,000 or 50,000
3:17:09uh or greater. This one is where it's
3:17:11greater than 10,000 which is essentially
3:17:13the entire table. Now, so far we've done
3:17:15everything just by writing it all out.
3:17:17And that's fantastic. But you can also
3:17:20come over here to store procedures. You
3:17:22can rightclick and say create stored
3:17:24procedure. Now let's actually copy this.
3:17:28We're just going to create the exact
3:17:29same thing. We'll create store
3:17:30procedure. And we can just paste this in
3:17:33here. And let's go ahead and do that.
3:17:35There we go. And sure, we'll call it new
3:17:38procedure. Why not? And if we say apply,
3:17:42you'll notice that it generates this
3:17:44script right here. And we can apply it
3:17:46and we can create it. We will in just a
3:17:48second, but let's take a look at it. So,
3:17:50we're going to use parks and recreation.
3:17:52That's what I was mentioning before.
3:17:54We're then going to say drop procedure
3:17:56if exists. Now, this is something that I
3:17:59was going to show you later, but I'll
3:18:00just show it to you now. Sometimes it is
3:18:02really beneficial to write something
3:18:04like this before you create it in case
3:18:06you've already created a store procedure
3:18:08with that name that you're wanting to
3:18:10replace. So it's checking if it's there
3:18:12and if that new procedure is already
3:18:14there, it's just going to drop it. Then
3:18:16it comes down and let me see if I can
3:18:17zoom in on this. And then it's going to
3:18:20create our delimter which it uses dollar
3:18:22signs. So my SQL is even, you know,
3:18:23validating what I was saying earlier.
3:18:25We're going to use parks and recreation
3:18:27again. And now again we have to use
3:18:29instead of a semicolon we're using
3:18:30dollar signs. Then we're creating the
3:18:33procedure which is new procedure we're
3:18:35saying begin and then it's even changing
3:18:38the delimter back. So basically
3:18:40everything that I said this is kind of
3:18:42doing it for you automatically. Now when
3:18:44I click apply it went ahead and executed
3:18:47that SQL statement and our new one is
3:18:50ready. So, we can go ahead and alter
3:18:52that store procedure, and it looks
3:18:54exactly the same as this one out here,
3:18:58which was uh large salaries number
3:19:01three. So, it looks exactly the same.
3:19:03Now, let's go ahead and get rid of this.
3:19:05Get rid of this, and let's go down
3:19:08below. The next thing I want to take a
3:19:10look at is something called a parameter.
3:19:13Now, before I actually get into this,
3:19:14I'm going to copy all this down here
3:19:16because I don't want to rewrite all of
3:19:17it, uh, if I'm being honest. So let's
3:19:20paste this in here. Now parameters are
3:19:22variables that are passed as an input
3:19:25into a store procedure in the allow the
3:19:27store procedure to accept an input value
3:19:30and place it into your code. Let's take
3:19:32a look at what that actually means. Now
3:19:34before I do anything, I'm just going to
3:19:35change this to uh number four so I don't
3:19:38forget. So let's get rid of all of this.
3:19:42We're going to keep it somewhat simple
3:19:43because we're looking at something new.
3:19:45Now, when I say we're passing through a
3:19:46parameter, I'm talking about when we're
3:19:48calling it. So, let's say we've already
3:19:51created this one. I'm not going to, you
3:19:52know, run this yet, but let's say we've
3:19:54created it. Let's say I want to pass in
3:19:56an employee ID. I want to pass in a
3:19:58specific person and I want to retrieve
3:20:00their salary. I know their employee IDs.
3:20:03I just want it to pull up their salary
3:20:05for us. So, what we're going to do is
3:20:07we'll get rid of this. And when we're
3:20:09calling it, put this down. When we're
3:20:11calling it, I'm going to pass through a
3:20:12value like one. That's Leslie. Nope. And
3:20:15then I want the salary to be the output.
3:20:18So I'm going to select the salary. So
3:20:20we're selecting salary from the employee
3:20:23salary. But how do we know that this one
3:20:26is the person we're looking for? Well,
3:20:28when we're actually creating this
3:20:29parameter, we create it right in here.
3:20:32That's what tells the store procedure to
3:20:34accept an input value when we're calling
3:20:37it down below. We're going to call this
3:20:39employee
3:20:41ID. Now, after we call it, after we name
3:20:44our parameter, we need to then give it a
3:20:47data type. So, I'll call this an
3:20:48integer. So, we're telling the store
3:20:50procedure, when somebody calls this
3:20:51store procedure, they have to pass
3:20:53through an integer. It can't be a string
3:20:56or it can't be a date. It has to be an
3:20:58integer. Now what we're going to go do
3:21:00is right down here we'll say where the
3:21:04employee ID that's from this column in
3:21:07the actual table we'll say is equal to
3:21:10the employee
3:21:12ID which is our parameter right here.
3:21:15Now you may be thinking that's really
3:21:17confusing. They're named the exact same
3:21:18thing. Can I change it? The answer is
3:21:21yes. I actually encourage it. So there
3:21:22are some naming conventions that are out
3:21:24there that I think are helpful ones that
3:21:25I personally use. Um, but remember this
3:21:28is just kind of a variable parameter
3:21:30name. You can kind of call it whatever
3:21:31you want. So if I wanted to say Huggy
3:21:34Muffin, I could. Uh, and this could be
3:21:37Huggy Muffin. So let's try it with Huggy
3:21:39Muffin. I just came up with that off the
3:21:41top of my head, so don't judge me. Um,
3:21:43but we're going to create the store
3:21:44procedure. And then when we call it
3:21:46later, we want it to return the salary
3:21:49where the employee ID right here is
3:21:52equal to whatever was passed through
3:21:55that parameter, that input parameter.
3:21:57We're going to keep it as one. So it
3:21:58should return 75,000. Let's go ahead.
3:22:01We're going to create this. And now
3:22:03let's go right down here and we're going
3:22:05to run it. And we can see that that is
3:22:07the salary and it worked perfectly. Now
3:22:10like I was saying, that is not what I
3:22:11would actually name it. Uh there are
3:22:13some naming conventions like underscore
3:22:16param at the end. So you kind of want to
3:22:18keep it at least I recommend you try to
3:22:21keep it similar to what you're actually
3:22:23looking for. And you can either end it
3:22:25in underscore param or there's another
3:22:27way that you can do it which is come
3:22:28right over here and do p underscore. And
3:22:31these are just ways that you can tell
3:22:33the code or you can just be able to
3:22:35visually see the difference in the code.
3:22:36So this is just what I recommend. Then
3:22:39you put it right down here. you say
3:22:41where the employee ID is equal to P
3:22:43employee ID saying this is the parameter
3:22:46that's being passed through and put into
3:22:48our actual query. So that is all we're
3:22:50going to take a look at in this lesson.
3:22:52In the next lesson, we're going to be
3:22:54taking a look at triggers [music] and
3:22:55events.
3:23:08Hello everybody. In this lesson, we're
3:23:10going to be taking a look at triggers
3:23:12and events. A trigger is a block of code
3:23:14that executes automatically when an
3:23:16event takes place on a specific table.
3:23:19For example, let's take a look at these
3:23:20two tables. Now, when a new employee is
3:23:22hired, they're put into this table with
3:23:25their salary information and everything,
3:23:26but sometimes people forget or don't add
3:23:30their information like uh you know who
3:23:33right here. Uh they're not put into this
3:23:35demographics table. And we want to
3:23:37change that because we want to have
3:23:38everybody in here. So when somebody is
3:23:41put into this salary table, we want it
3:23:43to automatically update with the
3:23:45employee ID, first name, and last name
3:23:48into this table right here for the
3:23:50employee ID, first name, and last name.
3:23:52So we're going to write a trigger when
3:23:54data is updated into the salary, it's
3:23:57going to also update the employee
3:23:58demographics for us. Now, let's go right
3:24:00down here and we're going to take a look
3:24:03at how we can do that. Now, if you
3:24:04watched the last lesson on store
3:24:06procedures, we'll do a lot of the same
3:24:08writing style or same formatting for
3:24:11triggers and events. So, we're going to
3:24:12start with is the delimter. We're just
3:24:14going to do that right off the bat
3:24:15before we get into anything. And we're
3:24:16going to change that to the double
3:24:18dollar sign. Now, the delimiter again in
3:24:21case we have multiple lines of code,
3:24:22which we're going to have. If we have
3:24:24multiple lines of code when we're
3:24:25creating this trigger, this delimter is
3:24:28going to help us have multiple queries
3:24:30within our create trigger statement. So
3:24:33this is really important. We'll just
3:24:34start out by doing that. Now let's
3:24:36create our trigger. And we do need to
3:24:40name this. So we'll say employee_insert.
3:24:45And we'll just call it like that. Did I
3:24:47spell that right? Yeah. Employee insert.
3:24:49So we have our create trigger. We've
3:24:52named it. Now we need to specify what
3:24:54event needs to take place in order for
3:24:57this to be triggered. So we're going to
3:24:59say after an insert and I need to spell
3:25:03insert right after an insert on and
3:25:05we'll do the employee
3:25:08salary table. So after we insert onto
3:25:11the employee salary table down below
3:25:13we're going to write what's actually
3:25:14going to happen. Now we're writing after
3:25:16because we're doing it where when new
3:25:18information is put on the salary table
3:25:20it's automatically updated into the
3:25:22demographics table. But you could also
3:25:25do before which means if data is deleted
3:25:28from the employee salary table something
3:25:30could happen but we're not doing any
3:25:32deleting or any updating we're doing
3:25:35insertion. So we're going to say after
3:25:38an insert on now the next part that we
3:25:40need to write is for each row. Now this
3:25:43for each row means that the trigger is
3:25:46going to get activated for each row that
3:25:48is inserted. So, if we had an insert
3:25:51statement that inserted four different
3:25:52people who were just hired, that means
3:25:54this trigger is going to be activated
3:25:56four times. Now, some SQL databases like
3:25:59Microsoft SQL Server have things like
3:26:01batch triggers or table level triggers
3:26:03that'll only trigger once for all four
3:26:06of them. And in my opinion, those are
3:26:08really, really nice. Uh, I've used
3:26:10those. I like them. The way that my SQL
3:26:12has it right here is not the most
3:26:13optimal way to do it, unfortunately. but
3:26:15we don't have access to the batch level
3:26:17or the table level triggers at this
3:26:19time. So this is really just the setup
3:26:21for what we're about to write. So after
3:26:23it's inserted on the employee salary
3:26:25table for each row, what is going to
3:26:28happen? We're going to go down here and
3:26:29we're going to say begin and we'll have
3:26:32end. Now the code that we're going to
3:26:34write here is what's going to happen
3:26:36after this event takes place. So what
3:26:40we're going to do is we want to take
3:26:42from this table. Let's bring this back
3:26:44up real quick. When we insert a new
3:26:47person, we want to take the employee ID,
3:26:50the first name, and the last name and
3:26:52automatically put it into the
3:26:54demographics table. So, we want to say
3:26:58insert and let me do tab insert into
3:27:02we're going to insert into the employee
3:27:06demographics
3:27:08table. And we're not taking everything.
3:27:10So, let's actually specify what columns
3:27:12we're doing. We're doing employee id
3:27:14first name and then the last underscore
3:27:20name. Now we need to specify what the
3:27:22values are. Now from the employee salary
3:27:25table we're taking employee ID, first
3:27:27name and last name. But we don't want to
3:27:29take all of them, right? We don't want
3:27:31to take every single employee ID, every
3:27:33single first name, every single last
3:27:34name. We only want to take the new
3:27:36values that were just inserted. Well,
3:27:39lucky for us, there is something that we
3:27:41have for this. So, let's do values, new
3:27:44parenthesy. We have something called
3:27:45new. Now, new is going to say we're only
3:27:48taking the new rows that were inserted.
3:27:50There's also an old like this where it
3:27:54takes rows that were deleted or updated,
3:27:56but of course, for us, we're going to be
3:27:58using new. So, we'll say new employee
3:28:01ID, new_ame,
3:28:05and then new.ast_ame. last underscore
3:28:08name and we'll close that. Then we'll
3:28:10come down here and we'll do our
3:28:14delimiter and we'll change this as well.
3:28:16We'll say delimiter back to a semicolon.
3:28:20Now, we're getting this error because we
3:28:22need this right here. So, let's recap
3:28:25what we've created, then we'll actually
3:28:26create it and try it out. So, we're
3:28:29creating our trigger called employee
3:28:31insert. After a row is inserted into the
3:28:35employee salary table, for each row,
3:28:38here's what's going to happen. We are
3:28:39going to insert into the employee
3:28:41demographics table the employee ID, the
3:28:43first name, and the last name. Those are
3:28:45the columns that we're going to insert
3:28:46into. Then we're taking the values new.
3:28:50ID, new name, and new.ast name. Now,
3:28:54MySQL understands that when we say new,
3:28:56we're talking about the event that takes
3:28:58place. So, this is the data that's being
3:29:00inserted. It just knows that. So, let's
3:29:02go ahead and create it. We're going to
3:29:04run this. And it should work. Let's pull
3:29:06this up. Says create trigger. So, that
3:29:08worked. Now, the thing about triggers,
3:29:11uh, that's unfortunate, it does doesn't
3:29:13have its own little section under here,
3:29:15right? But it does have under the
3:29:18employee salary. Let's go right here.
3:29:21And then under the triggers. So, we can
3:29:23find it, which is great. So, we have
3:29:26this employee insert. If we rightclick,
3:29:29we can't really do anything with it.
3:29:30That's the unfortunate thing. We can't
3:29:32alter it. We can't change it. We can't
3:29:34drop it. We can't do anything. Um,
3:29:35that's the unfortunate part. But let's
3:29:37actually test it. So now we're going to
3:29:39say insert into and we're going to
3:29:41insert into this employee salary. That's
3:29:44how we're going to trigger it. So insert
3:29:45into the employee salary. And then we'll
3:29:48do employee ID. These are all the
3:29:51columns. The first underscore name, last
3:29:54underscore name, occupation.
3:29:57Uh, bear with me for a second. Then we
3:29:59have salary and then department ID. So
3:30:03this is what we're inserting into. Now
3:30:05we have to do our values. Now this
3:30:06should be shorter hopefully. We'll do
3:30:0913. We'll call him uh Jean Ralph.
3:30:14There we go. Last name is Sapperstein.
3:30:19Just like that. And not actually just
3:30:21like that. That's not spelled right. So
3:30:23we have Sapperstein. His occupation is
3:30:26entertainment
3:30:28720 CEO. How much is he making? Uh let's
3:30:32say a million. Is that a million? A
3:30:35million. He's making a million dollars.
3:30:37And he's really not part of any uh
3:30:39department. So we're just going to have
3:30:40null. So what we're about to do is we're
3:30:43only inserting on the employee salary
3:30:45table, but we're putting all the values
3:30:47that we need into the appropriate
3:30:49places. Let's add a semicolon. Let's go
3:30:52ahead and run this. Make sure it worked.
3:30:56And it says insert into and then one row
3:30:59affected. Now let's come back up. Let's
3:31:02look at our salary table first and get
3:31:05rid of this.
3:31:07If we pull this up, you can see Jean
3:31:09Ralph Sapperstein. He was added. Let's
3:31:13go over to the demographics is the
3:31:14moment of truth. Let's see if it worked.
3:31:18And as you can see, it worked perfectly.
3:31:20We have Jean Ralph Sapperstein. Now,
3:31:22they do need to come back and fill in
3:31:23this information, but it's already in
3:31:26here kind of queuing them up saying,
3:31:27"Hey, we need this person's age, gender,
3:31:29birth date, all that other information."
3:31:31So, that is how we can create a trigger
3:31:34based off of a specific table and then
3:31:37when it happens, it just automatically
3:31:40does it for us. We don't have to really
3:31:41think about it. We just know that we've
3:31:43created a trigger and we can actually go
3:31:44and insert data on that table and that
3:31:46trigger is going to work. it's going to
3:31:48do what it's supposed to do. And that's
3:31:50really, really helpful in the real world
3:31:52when you're working with a ton of
3:31:53tables. A ton of things need to be
3:31:55automatically done and you don't want to
3:31:57have to manually do this. So, having
3:31:59these triggers can save you a ton of
3:32:01time. Now, let's scroll down and we're
3:32:03going to take a look at
3:32:05events. Now, event is kind of similar to
3:32:07a trigger. A trigger happens when an
3:32:09event takes place, whereas an event
3:32:12takes place when it's scheduled. So,
3:32:14this is more of a scheduled automator
3:32:17rather than a trigger that happens when
3:32:18an event takes place. These can be
3:32:20fantastic for a lot of things like when
3:32:22you're importing data. You can pull data
3:32:24from a specific file path on a schedule.
3:32:27You can build reports that are exported
3:32:28to a file on a schedule. You can do it
3:32:31daily, weekly, monthly, yearly, really
3:32:33whatever you'd like. It's just super
3:32:35helpful for automation in general. Now,
3:32:37let's say the Pawne Council comes up
3:32:39with some new legislation. They need to
3:32:42save some money, especially in the parks
3:32:43and recck department. We're just
3:32:44spending too much or they're spending
3:32:46too much. And what they want to do is
3:32:48retire people who are over the age of 60
3:32:50immediately and give them lifetime pay.
3:32:54So what we want to do is create an event
3:32:57that checks it, let's say every month or
3:32:59every day. And then if they're over a
3:33:02specific age, we are then going to
3:33:04delete them from the table and they will
3:33:06be retired. This is a fake example, so
3:33:09you know, go with it. So, what we're
3:33:10going to do is come right down here.
3:33:12We'll select everything from employee
3:33:15demographics
3:33:17and let's run this and let's pull this
3:33:20up. So, let's say if they are over the
3:33:22age of 60, which unfortunately is Jerry
3:33:25Gurgage, like I don't make the rules,
3:33:27but if they're over the age of 60, they
3:33:29are going to be automatically retired.
3:33:32So, let's come right over here. We're
3:33:35going to say create event and we'll call
3:33:38this the delete
3:33:40and delete
3:33:43retirees. Now before when we were
3:33:46creating the trigger, we were saying
3:33:47based off of a specific event, but here
3:33:49we're going to schedule it. We're going
3:33:51to say on schedule and then we're going
3:33:54to say every and we could do one month.
3:33:57Maybe we'll look every single month. But
3:33:59here we'll do let's do every 30 seconds.
3:34:03every 30 second. Now, we're going to go
3:34:05down. We'll say do. And this is going to
3:34:07say here's what needs to happen every 30
3:34:10seconds. So, we'll say begin and end.
3:34:14Now, what's going to happen every 30
3:34:16seconds is we're just going to start
3:34:17with a select statement. And I'll just
3:34:19copy this. Actually, we'll start with a
3:34:21select statement, but then we'll update
3:34:22it to a delete statement. But we'll come
3:34:25right here. We'll say where the age is
3:34:29greater than or equal to 60. So, if we
3:34:32just run this query right here,
3:34:36that's only one person. That's Jerry
3:34:37Gurgg. Now, if we want to write this
3:34:39correctly, we'll do the delimiter.
3:34:42We'll have the dollar signs. We'll have
3:34:45the dollar sign right down here as well.
3:34:47And we'll say delimiter back to a
3:34:50semicolon. Now, every 30 seconds, we
3:34:53don't want to select people who are that
3:34:54age. We want to delete. So, let's go
3:34:57right here. We're going to change this
3:34:58because now we know it should be
3:34:59deleting the right person. and we're
3:35:01going to go ahead and create this event.
3:35:04Let's go ahead and run this.
3:35:06And let's make sure it was created
3:35:08properly.
3:35:10It looks like create event zero is
3:35:12affected. This should be working. Let's
3:35:14go back up to the demographics table and
3:35:16let's run this.
3:35:19And let's pull this down and pull this
3:35:22up. And as you can see, unfortunately,
3:35:25Jerry Gurgich is no more. Um, you know,
3:35:28he's just too old. and the Pony Council,
3:35:30they recognized that. And so it wasn't
3:35:32my rule. That was unfortunately Ponyie
3:35:34Council's rule. Now, really quickly, if
3:35:36that did not work, let's say you
3:35:38couldn't create your event at all. Let's
3:35:41go down here. I'm going to show
3:35:43variables
3:35:44and we'll run it just like this.
3:35:48Um, I'm going to show you how you may
3:35:50need to fix this. So, we can say where
3:35:53variables is like, and then we'll say,
3:35:56uh, event. do it just like this. So, I
3:36:00have a ventuler where the value is on.
3:36:03If yours is off, which sometimes that
3:36:05can happen, you're just going to update
3:36:07this to on. Now, another issue could
3:36:10have happened, and I just want to
3:36:11explore this for just one second. You
3:36:12may not have permissions to delete
3:36:14things. If you do not come right up
3:36:16here, let's try to figure this out
3:36:18together. It's actually edit
3:36:19preferences,
3:36:21and I want to say it's right here into
3:36:23the SQL editor at the very bottom. Yep.
3:36:26So, save updates rejects updates and
3:36:29deletes with no restrictions. This needs
3:36:31to be unchecked. So, go to preferences,
3:36:33go to the SQL editor down at the bottom,
3:36:35uncclick this if that didn't work. Now,
3:36:38if everything worked perfectly, you
3:36:39don't need to change a thing. But if it
3:36:41didn't, I just wanted to work through,
3:36:42you know, some uh troubleshooting that
3:36:44you may just have to Google or chat GBT
3:36:47or something to try to figure out. So,
3:36:49that is how we can create an event in my
3:36:51SQL to run on a schedule. Now, typically
3:36:53you wouldn't do it on every 30 seconds.
3:36:56you would do something like every 1
3:36:57month or every 1 year or you know a
3:37:00longer time frame but you get the
3:37:02picture of what we're trying to do. So
3:37:04that is how we create triggers and
3:37:05events and this is also the end of the
3:37:08advanced my SQL series. If you made it
3:37:11this far absolutely fantastic work in
3:37:13the next two lessons those are going to
3:37:14be our projects that we're going to work
3:37:16on for this series. We'll have a data
3:37:18cleaning project and we'll have an
3:37:19exploratory data analysis project. Both
3:37:22of those are going to include a ton of
3:37:24things that we've looked at in this
3:37:25series and even some new things that we
3:37:27didn't look at in the actual lessons
3:37:28themselves. So, thank you guys for
3:37:30watching. I hope you enjoyed this entire
3:37:32series. If you did, be sure to leave a
3:37:34like and a subscribe below. I will see
3:37:36you in those projects.
3:37:41[music]
3:37:50Hello everybody and welcome to the very
3:37:52first project in the MySQL series. Today
3:37:55we're going to be focusing on data
3:37:56cleaning. Now if you don't know what
3:37:58data cleaning is, it's basically where
3:38:00you get it in a more usable format. So
3:38:02you fix a lot of the issues in the raw
3:38:04data that when you start creating
3:38:06visualizations or start using it in your
3:38:08products that the data is actually
3:38:10useful and there aren't a lot of issues
3:38:11with it. So that's really what data
3:38:13cleaning is. Now, what we're about to do
3:38:14is create a database. We're going to
3:38:16import a data set. This is a real data
3:38:18set. And what we're going to do is we're
3:38:20going to clean the data. So, I'm going
3:38:21to show you and walk you through all the
3:38:23steps in order to clean the data. The
3:38:25data set that we're going to be working
3:38:26with will be in the GitHub. So, you can
3:38:27just go and download that. I'll have a
3:38:29link somewhere in the description. But,
3:38:31let's get started. First thing we're
3:38:33going to do is create a new database.
3:38:34So, we'll go right over here to create a
3:38:36new schema. And we're just going to call
3:38:38this one. We'll do this is world
3:38:42layoffs. So, if you can't tell already,
3:38:45uh, we're going to do world layoffs. Uh,
3:38:46that's the data set that we're going to
3:38:47be doing. We'll just click apply. And
3:38:50that creates our world layoffs right
3:38:52here. Now, we're going to go into here.
3:38:54There are no tables. We're going to
3:38:56right click on tables and go to table
3:38:58data import wizard. Now, we haven't done
3:39:00this yet uh in this series. We haven't
3:39:02imported any data, but that's what we're
3:39:04doing here. We're going to show you how
3:39:06to import data. So, we'll go ahead and
3:39:08click browse. And as you can see right
3:39:10here, we have this layoffs data set.
3:39:12Let's open this up and we're going to
3:39:15click next and we're going to create a
3:39:18new table. There's no existing table in
3:39:20this database. You can drop it if it
3:39:22exists uh if you'd like to. It doesn't
3:39:23matter. This is new. We're going to go
3:39:25ahead and select next. Now, right here
3:39:27is where you configure import settings.
3:39:29Now, MySQL is going to automatically
3:39:31assign a data type based off of the data
3:39:33in these columns. So, we'll take a look
3:39:35at the data later. Now, there is one
3:39:38thing that you can take a look at real
3:39:39quick. We have this date column. Now, in
3:39:42here, it assigned it as a text. That's
3:39:44because the format. We are going to
3:39:46import this as the raw data. We're not
3:39:48going to try to change anything in the
3:39:50import settings. We're just going to
3:39:51assume this is how the data was in the
3:39:53table. So, we're not going to change
3:39:54anything. Although, this may be
3:39:57something that you would want to change
3:39:58to something like a date time and go and
3:40:01fix that. But, we're going to import
3:40:03this as the raw data. Let's go ahead and
3:40:05select next. We're going to import it.
3:40:08We just select next. Now, this could
3:40:09take a little bit. Uh, so while this is
3:40:11importing, I'm just going to skip ahead.
3:40:13This should take just a few minutes to
3:40:14import. All right, this just finished.
3:40:16Let's select next. And we imported 2,361
3:40:20records. Let's go ahead and select
3:40:22finish. We can get rid of this. And
3:40:26let's refresh this. Perfect. We have our
3:40:29layoffs table. So, we'll select
3:40:31everything. And I'm going to go and
3:40:33double click on the world layoffs
3:40:35because I don't want to write out the
3:40:36whole thing every time. So we're going
3:40:37to say from layoffs and let's see what
3:40:41we get. So let's take a look at the data
3:40:44that we're going to be working with in
3:40:46this data cleaning project. So this data
3:40:48set is layoffs from around the world
3:40:50starting I think 2021. And we'll take a
3:40:52look at that in this date column later.
3:40:54But it has the company. So it has the
3:40:56company that did the layoffs. It has the
3:40:58location of where they are, what
3:41:00industry they are part of, how many they
3:41:02laid off, the percentage that they laid
3:41:04off. So the percentage of their company,
3:41:06the date, the stage, which refers to the
3:41:09stage that the company is in, whether
3:41:10it's a series B, post IPO, uh they don't
3:41:13know. Then there's the country, and then
3:41:15we have funds raised millions. So we
3:41:18have a lot of information here. And in
3:41:21the next project, we're going to be
3:41:22doing exploratory data analysis. So
3:41:24we're cleaning all of this data and then
3:41:27in the next lesson, we're going to
3:41:28actually dive into it and try to find
3:41:30trends and patterns and all these other
3:41:32things. So what we are going to do is
3:41:34we're going to go through multiple
3:41:36steps. Step number one is we are going
3:41:39to try to remove duplicates if there are
3:41:42any. That is the first thing I typically
3:41:44do especially if I know this data
3:41:46shouldn't have any duplicates or it'd be
3:41:48you know repetitive or unnecessary to
3:41:50have duplicates. The second thing is
3:41:52going to be to standardize
3:41:55the data.
3:41:56That just means that if there are issues
3:41:58with the data with spellings or things
3:42:00like that, we just want to standardize
3:42:02it to where it's all the same as it
3:42:03should be. Number three is we'll look at
3:42:06the null values or blank values. And
3:42:10there's a lot of null values in here.
3:42:12There's even a blank value right here.
3:42:14And we're going to see if we can
3:42:16populate that if we can. And there are
3:42:18times where you should, there are times
3:42:20where you shouldn't. I'll kind of walk
3:42:21through that as well. And lastly, we
3:42:24want to remove any columns and rows that
3:42:27aren't necessary. And there's a few
3:42:28different ways to do that. Uh, this one
3:42:30is a little bit, you know, um, let me
3:42:33write this actually real quick. Remove
3:42:34any columns. So, I'm just going to say
3:42:36there are instances where you can do
3:42:38this. There are instances where you
3:42:39shouldn't do this. When you're working
3:42:41with massive data sets and you have a
3:42:43column that's, you know, completely
3:42:44irrelevant, completely blank, you don't
3:42:46have any ETL process that is required
3:42:48for it. Um, you can get rid of it. it
3:42:51can save you time when you're querying
3:42:52your data. Now, with that being said, uh
3:42:55and we'll talk about this later, in the
3:42:57real workplace, oftent times you have
3:42:59processes that automatically import data
3:43:01from different data sources. If you
3:43:02remove a column from the raw data set,
3:43:05that's a big big problem. So, what we're
3:43:08going to do is something I would
3:43:10actually do in my real work, which is I
3:43:12would create some type of staging or raw
3:43:14data set. Let's say this one's our raw
3:43:16one. And we could have even called this
3:43:18layoffs raw. We're going to create
3:43:21another one. We're going to create a
3:43:22table. So, we'll say create table. And
3:43:25let's call this one layoffs_staging.
3:43:29And we literally just want to copy all
3:43:31of the data from the raw table into the
3:43:35staging table. So, we can do that really
3:43:37quickly by just saying like layoffs.
3:43:42And if we run this and we refresh,
3:43:45you'll see we have the staging database.
3:43:48And let's copy this.
3:43:51Here we go. We'll do layoffs_staging.
3:43:56And so now we have all of the columns.
3:43:58And all we have to do is insert the
3:44:00data. So we're just going to say insert.
3:44:03Then we're going to say layoffs staging
3:44:06right here. And we'll select everything
3:44:10from
3:44:12layoffs.
3:44:14And let's run this. And if we select the
3:44:17table, we now have all the data over. So
3:44:20super super easy. And now we have these
3:44:23two different tables. Now again, why do
3:44:24we do this is because we're about to
3:44:26change the staging database a lot. If we
3:44:29make some type of mistake, we want to
3:44:31have the raw data available. This does
3:44:34happen. This is something that you do in
3:44:36the real workplace because you're not
3:44:37going to work on the raw data. It just
3:44:39you shouldn't do it. It's not best
3:44:40practice. So I'm going to show you what
3:44:42I would actually do in my, you know,
3:44:43like a real job. So, that's what we're
3:44:46going to do now. We're only going to be
3:44:47working off the staging database, and we
3:44:49can copy this and make different
3:44:51databases for different things. Um, as
3:44:53long as we have our raw data, we can
3:44:55really do anything we want going
3:44:56forward. Uh, and that's what we're going
3:44:58to do. So, the number one thing we're
3:45:01going to look at is to make sure that we
3:45:03are removing duplicates. We want to make
3:45:05sure we don't have any duplicate data in
3:45:06here, and if so, we're going to get rid
3:45:07of it. Now, really quickly, if you did
3:45:09my Microsoft SQL Server project, we did
3:45:12something very similar, but we had an
3:45:14extra column over here that gave the
3:45:16unique row ID, which made it really easy
3:45:19to remove the duplicates. Here, there is
3:45:23no identifying factor that's going to be
3:45:25easy for that. So, I'm just going to
3:45:26tell you up front, removing these
3:45:27duplicates is not going to be easy, but
3:45:29we'll walk through it every step of the
3:45:31way. So, what we can do is try and do
3:45:33something like a row number and we'll
3:45:35match it against all of these columns
3:45:37and then we'll see if there are any
3:45:39duplicates. Now, I'm just we're starting
3:45:41off strong. Okay, we're jumping into
3:45:43kind of some of the more advanced
3:45:44things. It does get actually easier as
3:45:46we go, but this is the actual order that
3:45:48I follow. So, uh I'm going to keep it.
3:45:50So, let's try to identify duplicates.
3:45:53So, let's copy this.
3:45:56Let's pull this down. Do underscore
3:45:58staging.
3:46:00There we go. Now, what we can do is we
3:46:02can do row number and we'll do that
3:46:04partition by basically we could do every
3:46:07single one of these columns. That's kind
3:46:09of what we're doing. So, what we can do
3:46:12is we can say everything. Then we can do
3:46:14a comma and we'll say row number and be
3:46:19just like this. And we're going to do
3:46:21this over and we want to partition by
3:46:24all of these columns essentially. We
3:46:27could just do a few for now to see if we
3:46:29get any hits and then we can look at
3:46:30that. But they're going to be multiple
3:46:32companies that have layoffs in the same
3:46:34location and industry. Although their
3:46:36total laid off would probably be
3:46:37different. The date would probably be
3:46:38different. So if we do something like uh
3:46:41company, let's do industry. We will do
3:46:46total_laid
3:46:49off,
3:46:50percentage
3:46:52laid off. And then let's do date. Now
3:46:56I'm doing date with the back ticks
3:46:58because date is a keyword in my SQL. So
3:47:02if we do it like this, it just really
3:47:04makes it easy. So we're going to
3:47:06partition by all of these things. So
3:47:08let's do partition by and let's bring
3:47:12this down real quick. So I'm just going
3:47:14to say over partition by and we're going
3:47:16to call this as row_num.
3:47:20Now let's try running this. Let's see if
3:47:22it works really quickly. It's important.
3:47:25And over here you can see that we have
3:47:28our row number. Now these mostly are
3:47:30unique and these all look unique. I'm
3:47:32not going to scroll through all of them,
3:47:33but we want to be able to filter on
3:47:35this. So we can filter where the row
3:47:36number is greater than two. If it has
3:47:38two or above, that means there's
3:47:39duplicates. That means there's an issue.
3:47:42So let's go ahead and we're going to
3:47:44take this. We'll put it into either a uh
3:47:47subquery or a CTE. I'll create a CTE for
3:47:49this uh because it's really easy. So
3:47:51we'll say four or not four, we'll say
3:47:54width and then we'll do uh duplicate_ct
3:47:59as then we'll just do our parenthesis.
3:48:02We'll paste this in here and get rid of
3:48:04that right there. And now we're going to
3:48:06say
3:48:08select everything from this duplicate
3:48:12CTE. Then we'll say where row num is
3:48:16greater than one. Let's run this and add
3:48:19a semicolon. Let's run this. And you can
3:48:22see that these ones have duplicates. So
3:48:25these are our duplicates actually. And
3:48:28we want to get rid of these exact rows.
3:48:30Now just to confirm that these are uh
3:48:32the duplicates, let's look at this one.
3:48:35I've never heard of this company. Um but
3:48:37we'll take it really quick
3:48:40and let's select we'll say
3:48:44where company is equal to. We'll call
3:48:48this ODA. So, let's run this.
3:48:53And it looks like these
3:48:55No, no, no, no. These aren't duplicates.
3:48:58That's a good thing we checked. Okay,
3:49:00because it looks like um these aren't
3:49:03the exact same. Although they're very,
3:49:05very close, these technically are not
3:49:07duplicates. So, I'm glad we checked
3:49:09this. We need to do this partition by
3:49:11over every single column. That's what
3:49:13I'm realizing. So, we'll do company,
3:49:16location. I'm glad I'm genuinely glad
3:49:18we're, you know, it's good to make
3:49:20mistakes um and figure things out as you
3:49:22go. It really is important. So, company,
3:49:24location, industry, total laid off,
3:49:26percentage laid off, date, then we'll do
3:49:29stage,
3:49:30and then we'll do country, and then
3:49:34funds
3:49:36raised
3:49:38millions. So, we're changing the CTE to
3:49:41partition over everything. So, now let's
3:49:43run this. Okay, ODA is not in there.
3:49:46That's the only one we checked. Um, but
3:49:48let's look at Casper. I know this. These
3:49:50are the um Aren't these the mattress
3:49:52people? Didn't know they had layoffs.
3:49:54Poor guys. Um, all right. Let's take a
3:49:55look. It looks like this row and this
3:50:00row are duplicates. These are our
3:50:02duplicates. So, we are going to want to
3:50:04remove only one of those. We don't want
3:50:07to remove all of those. So, um, just
3:50:11looking at this one example, it looks
3:50:12like this, uh, query is working well. So
3:50:15here's our duplicates. Now we need to
3:50:17identify these exact rows. We don't want
3:50:20to delete both of them. When we looked
3:50:22at Casper, there's the real one that we
3:50:24want to keep. Then there's a duplicate
3:50:25that we want to remove. We don't want to
3:50:27remove both. That would be bad. Now in
3:50:29my SQL, it's a little bit trickier to
3:50:31remove things than it is in something
3:50:33like Microsoft SQL Server, Postgrace
3:50:35SQL. Um they have different ways that
3:50:37they can delete rows. For example, in
3:50:39Microsoft SQL Server, we could literally
3:50:41identify these row numbers in the CTE
3:50:43and delete them from it and it would
3:50:45delete it from the actual table. We
3:50:46can't do that in my SQL. And I'll show
3:50:49you uh let's actually copy this.
3:50:54We'll go like this and we'll say uh
3:50:57let's say we want to delete these. We'll
3:50:59say delete from we're deleting this from
3:51:02where the row number is uh greater than
3:51:04one. What am I writing right here?
3:51:06Delete. There we go. So delete from this
3:51:08duplicate CTE where the row number is
3:51:10greater than one. That's all these
3:51:11duplicates. We want to remove them.
3:51:13Let's try to do this. Let's run it.
3:51:16Let's go down. If we look at the bottom,
3:51:18it says the target table duplicate CTE
3:51:21of the delete is not updatable. So you
3:51:23cannot update a CTE. A delete statement
3:51:27is like an update statement. Um
3:51:29essentially. So what we are going to do
3:51:31is we're going to do something a little
3:51:33bit different because this is how I
3:51:34would love to do it. That makes it super
3:51:36super easy to remove duplicates. But
3:51:38that is not always the way that things
3:51:40happen in the real world. I think what
3:51:42we should do is take this right here and
3:51:44let's run this. We should take this
3:51:46right here and put this into let's say a
3:51:48staging two database and then we can
3:51:51delete it because we can filter on
3:51:53[clears throat] these row nums and we
3:51:54can delete those which are equal to two.
3:51:56So it's essentially like you know
3:51:59creating some type of table and then uh
3:52:01just deleting the actual column. So
3:52:03we're that's exactly what we're going to
3:52:05do. So it's essentially just creating
3:52:06another table that has this extra row
3:52:08then deleting it where that row is equal
3:52:11to two. So you know somewhat fairly
3:52:13straightforward but um let's try it and
3:52:17let's see what happens. So we're going
3:52:19to come down here and do is create our
3:52:22table. Uh let's try doing that with
3:52:24here. Let's uh let's copy to clipboard a
3:52:29create statement. Let's see if this
3:52:30works. Perfect. That's exactly what I
3:52:33wanted. Now, all we're going to do is
3:52:36say we're creating the table layoff
3:52:38staging 2. Now, this is a create table
3:52:41statement and we're naming the columns
3:52:43and then we're also assigning the data
3:52:45type. So, we have all these things, but
3:52:48we want one more. Let's do a comma and
3:52:51we want to add row_num.
3:52:54And I need to underscore num. And that
3:52:56should be an integer data type. So, we
3:52:59just keep it just like this. Let's go
3:53:02ahead and copy this and let's run it.
3:53:07See if it worked. Bring this up. Looks
3:53:10like it worked properly.
3:53:12Uh, and let's say
3:53:15let's go back up.
3:53:18I want to rewrite things that I don't
3:53:20have to.
3:53:22Let's run this. So, now we have this
3:53:24empty table. So, we want to insert
3:53:27this information right here. So, we're
3:53:30going to insert into. So, we'll insert
3:53:33into
3:53:34and then we'll do this right here. So,
3:53:36insert into staging two. Now, let's try
3:53:39to run this. See if it works. And let's
3:53:42run it. And let's select that table. And
3:53:46now we have it. So, let's pull this back
3:53:47up and I'll walk through what we just
3:53:49did because I know I'm going quick, but
3:53:50we have so much to cover um in this
3:53:52lesson. So we just inserted basically a
3:53:56copy of all these columns but in this
3:53:59new table we added one more the row num.
3:54:01So now we can filter and we can say
3:54:04where I need to spell that right where
3:54:07row num is equal to two or we should
3:54:10should say greater than one because some
3:54:12might have multiple duplicates. And
3:54:14there you go. Here are our duplicates.
3:54:16Now we're going to delete these. So all
3:54:18we have to do is come right back down.
3:54:21Where'd I go? Copy this. Come right back
3:54:24down here and we're just going to say
3:54:26delete from. We just did a select
3:54:29statement. I always recommend doing that
3:54:30to identify what you're deleting. Then
3:54:32you change it to delete. And now if we
3:54:35run this
3:54:37go. And I'm actually going to keep this.
3:54:39Um let me see. There we go. And let's
3:54:42run it again. And now they're gone. And
3:54:45if we say um just the whole table.
3:54:50This looks wonderful. Now, this row num
3:54:52is going to be a column at the end that
3:54:53we probably don't need anymore, right?
3:54:55It's a redundant column. It adds up
3:54:57extra space in memory and storage and
3:54:59all these other things and processing
3:55:00times. We're just going to get rid of
3:55:01it. Uh that'll be at the very end, I'm
3:55:03sure. So, it looks like we are good to
3:55:07go. That's how we remove duplicates.
3:55:08Now, um there are different
3:55:10[clears throat] ways to do it when you
3:55:12have different columns. Like if you have
3:55:13a unique column over here, makes it so
3:55:16much easier. So, so so so much easier.
3:55:18But we didn't have that. So we had to
3:55:19kind of do a workaround. Uh welcome to
3:55:21the real world. Now let's look at
3:55:23standardizing
3:55:26data. So standardizing data is finding
3:55:29issues in your data and then fixing it.
3:55:32So I'm already noticing right here. It
3:55:35looks like we have a space at the
3:55:36beginning. Uh we could easily just do a
3:55:38trim on this column. Um and let's I'm I
3:55:42don't even think I was um I did this
3:55:44when I wrote out all the the scripts for
3:55:46this. Let's just do from this table. Why
3:55:49am I writing it all out again? We
3:55:50actually want to select the company and
3:55:53then the or actually we'll just do
3:55:55distinct company. Distinct
3:55:58company. Let's run this.
3:56:02And
3:56:04if we do a trim around this, let's run
3:56:09this again.
3:56:11And that looks better. So, if we do uh
3:56:13company
3:56:15company, comma, and then we'll just do
3:56:17the trim. I don't want to
3:56:20we don't need to do distinct right now.
3:56:21We'll do the company. This just looks
3:56:23better. So, we're going to update that.
3:56:25Uh it's super easy. Now, if you ran into
3:56:28an issue just a second ago, uh I may
3:56:31need to help you change that. So, if you
3:56:33couldn't update or delete those things
3:56:35earlier, I should have told you this
3:56:36earlier. I apologize. All you need to go
3:56:39is to edit. You just need to go to edit,
3:56:41go to preferences at the very bottom, go
3:56:43to SQL editor, go all the way down to
3:56:45the bottom, and right here we have safe
3:56:48updates on. If you have this selected,
3:56:50that means you can't update anything.
3:56:51That's a problem. So, what you need to
3:56:53do is select this uh or unselect it like
3:56:56I have it and save it. You may have to
3:56:59even restart your MySQL potentially uh
3:57:01in order for the changes to take effect,
3:57:03but then you should be able to update
3:57:05that. Now, all we're going to do is
3:57:07update this table.
3:57:10and we're going to set. And now we need
3:57:12to come back here and we'll say we're
3:57:14going to set the company equal to trim.
3:57:18Now, if you don't know what trim is or
3:57:20you haven't taken that lesson, trim just
3:57:22takes off the white space off the end.
3:57:24So, it took the white space out of here
3:57:26or off the right hand side as well. So,
3:57:28we're going to update this and let's do
3:57:30a semicolon. A semicolon. Let's run
3:57:33this. Let's select this again. And it
3:57:36was updated properly. So, we're already
3:57:38off to a great start. Now, the next
3:57:41thing that I want to take a look at is
3:57:43the actual industry. So, let's go back.
3:57:47Let's copy this
3:57:50and let's take a look at the industry.
3:57:52So, we'll do industry and we'll run it.
3:57:55Now, if you look in here, there's a ton
3:57:58of different industries. Um, and there's
3:58:01marketing and marketing. Oh, because I
3:58:03haven't done distinct.
3:58:05Uh, please ignore me. Let's do distinct.
3:58:08And there's a ton of different
3:58:09industries in here. Transportation,
3:58:11healthcare, consumer, uh there's a blank
3:58:14one, which we'll take a look at.
3:58:16Aerospace. There's a lot of really
3:58:18unique ones. Let's actually order this.
3:58:19We'll do order by uh and let's just do
3:58:23one, which is the first column. We're
3:58:24just ordering our own stuff. So, we have
3:58:26null. We have blank. That's a problem.
3:58:28We'll take a look at that later.
3:58:31Uh but this is an issue. Crypto,
3:58:32cryptocurrency, and cryptocurrency.
3:58:34These are all the same thing. These
3:58:35should all be uh on or labeled the exact
3:58:39same thing. The reason we need to change
3:58:41this is because when we start doing uh
3:58:43the exploratory data analysis
3:58:45visualizing it, these would all be their
3:58:48own rows, their own unique thing, which
3:58:50we don't want. We want them all to be
3:58:52grouped together so we can accurately
3:58:54look at the data. Let's take a look at
3:58:56any other ones. Fintech and finance,
3:58:59that could be the same thing. I'm not
3:59:01100% sure. I'm not a fintech person. Um,
3:59:05I think for now the only one that I'm
3:59:07confident in changing is this one right
3:59:10here, which is cryptocurrency. So, let's
3:59:13go ahead and update that. So, all we
3:59:15have to do and we need to actually let's
3:59:17select really quickly where it's like
3:59:20crypto. So we'll say uh where industry
3:59:23and we want to select everything
3:59:27where the industry
3:59:29is like and we'll just do crypto. They
3:59:34all start with crypto, right? Yeah. So
3:59:35we'll do crypto just like this and let's
3:59:38run this and let's just take a look. Lot
3:59:43of layoffs in the crypto industry. Good
3:59:45night. All right. Let's find where it's
3:59:46cryptocurrency. Okay. So, even this one,
3:59:49it's crypto. And I know Gemini crypto.
3:59:51Crypto. And then it says cryptocurrency.
3:59:53So, these should be all crypto. You see
3:59:56how 95% of them are crypto. So, we're
3:59:58going to update these other ones. Oh,
4:00:01this one is C R Y PT. Is that how you
4:00:03spell crypto? Jeez, I don't know
4:00:05anything. All right. So, we want to
4:00:06update all of them to be crypto. So,
4:00:09what we're going to do is we're going to
4:00:11say update
4:00:14layoffs industry 2. We want to set the
4:00:18industry equal to crypto
4:00:22just like this where and we can do it a
4:00:26few different ways. We can say industry.
4:00:28We I think we can do like let's try this
4:00:30real quick. I I some of this stuff I
4:00:32don't have planned out. I'm just kind of
4:00:33going with it as we go. Um which I like
4:00:35better. You know, we kind of we work
4:00:37together on this. We figure these things
4:00:39out together. That's what I like. Um
4:00:41then we'll do like crypto just like
4:00:43this. Exactly like we had it up here.
4:00:45So, if it's like crypto, it should be
4:00:47crypto. Let's try this. Let's see if it
4:00:50ran because it may not have. I can't
4:00:52remember. Yeah, it worked. Okay, so it
4:00:54updated uh three rows and that looks
4:00:57correct. Now, let's go back up and let's
4:01:02run this. And as we scroll down, they
4:01:05are all the exact same. Beautiful,
4:01:07beautiful, beautiful. So if we do uh
4:01:10distinct industry again, let's get rid
4:01:14of this.
4:01:16If we run this query and we scroll down,
4:01:19crypto is its own thing. Beautiful.
4:01:23And it looks great. We can look at those
4:01:26later on how we can update those. Um but
4:01:29let's keep going. Let's look at our
4:01:32whole table again. And these blanks and
4:01:34these nulls are actually an issue. We do
4:01:36need to deal with them. But I I my
4:01:38instinct is telling me go fix it. Um but
4:01:41my you know tutorial side is saying okay
4:01:43stick with uh the tutorial the order
4:01:46that we agreed on. Um so let's go take a
4:01:49look. So we've looked at company, we've
4:01:51looked at industry. Um let's just real
4:01:53quick look at uh distinct
4:01:56uh location. Now it's good to look at
4:02:00most of these things, right? There could
4:02:01be small tiny issues that you just never
4:02:04saw. Um, and we're just going to order
4:02:07by
4:02:08order by one. Just do a real quick just
4:02:12a scan to see if we find any issues.
4:02:17Um, that could be an issue, but that
4:02:19could just be another language if I'm
4:02:21being honest. I don't know. I'm as I'm
4:02:23just scrolling through here because I
4:02:25want to make sure because this is not
4:02:26something I had in my uh pre-written
4:02:28script. This looks pretty good to me.
4:02:31Um, let's do everything. We'll run this
4:02:35and now let's look at country. So we'll
4:02:38do distinct country and let's run this
4:02:43and let's scroll down
4:02:47again. This is sometimes just what I
4:02:50actually do. All right, we got an issue
4:02:52right here. Super common. Somebody put a
4:02:54period at the end. Some dingus. Uh and
4:02:57we're not going to judge that person. I
4:02:58don't know who it was or who ruined this
4:03:00data set, but um yeah, that's a problem.
4:03:03So, we're going to need to just update
4:03:04that. It looks pretty simple. Um, but
4:03:07I'll just say where country is equal to
4:03:10or let's say like and then I'll say like
4:03:14United States.
4:03:17There we go. And oops. I'm going to say
4:03:20select everything. So, I just want to
4:03:21see
4:03:23um where it's at. Oh jeez, there's too
4:03:26many.
4:03:28Let me see if I can spot it.
4:03:31I can't spot it. It looks like they're
4:03:32supposed to be United States, not United
4:03:34States dot. That's the issue. Um, we can
4:03:37easily easily fix this. And we can
4:03:40probably Let's do um really quickly,
4:03:45let's do select, oops, select distinct,
4:03:49and then we'll do country, comma, and
4:03:52then we'll do a trim because we want to
4:03:55get rid of that um that one. We'll do
4:03:59country. Now, just doing the trim won't
4:04:02fix it. Let's go to the bottom. So, that
4:04:04doing the trim doesn't fix it. But
4:04:06here's what you can do. It's a little
4:04:08trick of the trade here. We're going to
4:04:09do something called trailing, which
4:04:11means coming at the end. So, what's
4:04:13trailing? The period from country. Let's
4:04:17try running this. Scroll to the bottom.
4:04:20And it fixed it. So, this is a little um
4:04:23a little advanced little tidbit for the
4:04:26trim here. We can do trailing from the
4:04:28country and we're looking for something
4:04:30that's not a whites space. We're
4:04:32specifying we're looking for a period.
4:04:33So now what we can do is we can say
4:04:35update. We can set the country do update
4:04:39um this table and we'll oops and we'll
4:04:44set what am I doing? What's going on
4:04:46here? We'll set the country equal to and
4:04:50we'll do it just like this. But we're
4:04:51only going to do it for a country,
4:04:53right? Uh so we'll say is equal to trim
4:04:57and we'll say where country is equal to
4:05:01or actually let's say like
4:05:04and let me see if I have this. I don't.
4:05:08Let's
4:05:10let's just say like United States like
4:05:11we had before.
4:05:14Just like this. So let's go ahead and
4:05:16update this after I put my semicolon in.
4:05:18Let's run this. And let's run this
4:05:21again.
4:05:22It shouldn't need to fix it anymore.
4:05:24It's just one row. That's perfect.
4:05:26That's exactly what we wanted. Now, one
4:05:29thing that's really important, uh, and
4:05:31this is, you know, this is a
4:05:32longitudinal,
4:05:34it's not the right word at all. Give me
4:05:36a second. I can't I can't speak and
4:05:38write at the same time. So, sometimes I
4:05:39just say, uh, dumb things. Um, uh, if we
4:05:43want to do not longitudinal, but, um,
4:05:46time series, that's the word I'm looking
4:05:48for. If we're trying to do time series
4:05:50um exploratory data analysis, time
4:05:52series visualizations later on, this
4:05:55needs to be changed. Right now it's text
4:05:57and we can look at that by going right.
4:05:59Actually, let's refresh this. We're not
4:06:02looking at staging. We're looking at
4:06:03staging two. If we look at the columns
4:06:05and we come down here to date, it is a
4:06:08text column. That's not good. If we're
4:06:10trying to do uh time series stuff, we
4:06:13want to change this to a date column.
4:06:15Now, how can we do that? Let's take a
4:06:17look. So, let's do date and let's not
4:06:20actually do it like that. Let's do date
4:06:21backslash. So, we're just going to look
4:06:23at the date. Now, let's change this
4:06:27because we want to format it how we want
4:06:29to format it with it, which is month,
4:06:31day, year. So, how can we do this? Well,
4:06:34there's something that's very, very
4:06:36helpful, works perfectly in this
4:06:38situation, and is exactly what we're
4:06:39going to do. It's called string to date.
4:06:41So we're going to do string underscore
4:06:43there it is right there undersc_2
4:06:46date. It literally helps us go from a
4:06:49string which is a text that's the data
4:06:51type to a date. So it's perfect. Now all
4:06:54we need to do is pass through two
4:06:55parameters. We have to pass through the
4:06:57column which is the date column and then
4:06:59what format we want it in. Now if you
4:07:02haven't done date formats before I'm
4:07:03going to kind of walk you through it
4:07:04while we're looking at it. Um in order
4:07:07to format this properly you use a
4:07:09percent sign. This is going to be a
4:07:10formatting for a month. A lowercase M. A
4:07:14capital M is something completely
4:07:15different. I believe it's spelled out. I
4:07:17need to I we can look at that in a
4:07:18second if we want to actually. And then
4:07:20we can do this right here. And then
4:07:22we'll do another one. So we're
4:07:23formatting it in the way that we want
4:07:25it, but also converting it to an actual
4:07:29uh date column. So now we want month and
4:07:31then we want day lowercase day. We'll do
4:07:34a forward slash and then another percent
4:07:36sign and then a capital Y which stands
4:07:38for I believe the four um four number
4:07:42long year. Uh I have a let's just um
4:07:45let's look at this real quick. So it
4:07:47worked perfect. So we're it's taking in
4:07:49this format that it's in right over here
4:07:52and converting it into the date format.
4:07:55So this is the standard date format that
4:07:57you're going to find in my SQL. Now
4:07:59let's see what happens really quickly
4:08:00just for fun. Uh, let's see if we do
4:08:02capital M. Uh, it looks like that's not
4:08:05going to work at all. Uh, let's do
4:08:07lowercase Y and um
4:08:11just formatted it to 2020. I think it
4:08:14took the first two numbers it looks
4:08:16like. I don't know why it's doing that
4:08:18if I'm being honest. Um, but if we keep
4:08:21it with the capital Y as we should, this
4:08:24looks perfect. This looks exactly like
4:08:27what we're trying to do. So, you can
4:08:28mess around with it. It depends on the
4:08:30how the data is formatted in your
4:08:32original column when it converts it to
4:08:34the string to date. And there's a lot of
4:08:36different stuff. You should just look up
4:08:37um date formatting in my SQL. Really
4:08:39interesting stuff. So, we're going to
4:08:40update this date column to this, which
4:08:43is our new date column. Let's go ahead
4:08:45and do that. We're going to say update.
4:08:48You guys should be getting used to this
4:08:49by now. That's the whole point is
4:08:50getting used to doing these things. So,
4:08:52we're going to set date equal to and
4:08:57then we're going to put in this right
4:08:58here, the string to date. Go ahead and
4:09:00do this. And let's run it.
4:09:04Make sure it worked. 2355
4:09:07rows. It looked like it did every single
4:09:09one. Uh, but let's go ahead and get rid
4:09:12of this
4:09:14and let's run it.
4:09:16And it looks like it worked perfectly.
4:09:18Now, there were some nles. It looks like
4:09:22and that'll be something we have to look
4:09:23at later when we talk about nulls. But
4:09:26um overall I believe this looks proper.
4:09:30Now if we refresh this, let's refresh.
4:09:33Let's come down to the date. You'll
4:09:34notice it is still a text. It's date.
4:09:38It's called text, but now it's in the
4:09:39date format. Now that's really
4:09:41important. And maybe I should have done
4:09:43that earlier if I'm being honest. Um
4:09:45tried to convert it to a date column. It
4:09:46wouldn't work. It would give us an
4:09:48error. Um you just have to trust me on
4:09:49that one. But now we can do it where we
4:09:52can change it to a date column. So let's
4:09:55do alter table. Now only do this, never
4:10:00ever do this on your raw table. Only do
4:10:02this on things like a staging table
4:10:03because we're about to completely change
4:10:05the data type of the actual table. So we
4:10:08want to change the layoff staging too.
4:10:10And then we're going to come down here
4:10:11and we're going to say modify column.
4:10:14And what column are we modifying? It's
4:10:17this date column.
4:10:19There we go. And we want to change it to
4:10:21what data type? A date. And am I
4:10:24spelling this right? Yeah. I just need a
4:10:26semicolon here. Whenever I see an error,
4:10:28I always you got to just look for the
4:10:30semicolons. So, let's go and run this.
4:10:33And let's refresh. See if it worked. And
4:10:35the date was changed to a date, which is
4:10:38perfect. That's all we wanted to do. Uh
4:10:40just to make sure we were doing what uh
4:10:43or we'll set ourselves up later in the
4:10:45future really well. Let's look at our
4:10:48table.
4:10:50All right, this is very good. So, we
4:10:54fixed a few uh just issues with the
4:10:57company. I believe something with the
4:10:59industry or the cryptocurrency. We
4:11:01changed the country. Um I'm just going
4:11:03to go ahead and tell you right now, this
4:11:05one uh we're not going to look at until
4:11:07we look at the um nullles and whatnot in
4:11:10just a second. So, we're not looking at
4:11:11that one yet. And then uh we have this
4:11:13extra column that we've done. So we've
4:11:15done a lot so far, but the next thing in
4:11:18the process, step one was remove
4:11:19duplicates. Step two was
4:11:20standardization. Step three is working
4:11:22with null and blank values. Now this is
4:11:25going to happen. You're going to have
4:11:27nles and you're going to have uh blank
4:11:30values in here. I it's somewhere um it's
4:11:33just going to happen. And so we need to
4:11:35think about what we're going to do with
4:11:36that information. Whether we want to
4:11:38make them all nulls, make them all
4:11:39blanks, try to populate that data. Let's
4:11:42see what we're going to do. So let's
4:11:45start off with the total laid off. We'll
4:11:48just do uh where total_laid
4:11:53off is null. So in order to look at the
4:11:56null, we say is null. Let's try equal to
4:12:00null. It's not going to give it to us.
4:12:02We have to say where it is null. So we
4:12:05have these values. These are completely
4:12:08null. Uh there's quite a few of them.
4:12:10But remember this is also useful
4:12:13information. But if they have two nulls,
4:12:16uh that probably is pretty useless to
4:12:19us. Um that's something I think we'll
4:12:21take a look at in a little bit.
4:12:22Actually, we'll say we and we may save
4:12:25this query. Percentage
4:12:28uh laid off is null. So if they're both
4:12:32null like these, these are all I believe
4:12:36fairly useless to us. These might be
4:12:38ones that we remove. So let's actually
4:12:40look at this. Um in step four we look at
4:12:43removing rows and columns. But one thing
4:12:46we should take a look at I remember this
4:12:48industry.
4:12:50Let's do uh industry
4:12:53do distinct.
4:12:55This industry had some missing values
4:12:59and let's take a look at that. Okay. So
4:13:02we have a missing value and we have a
4:13:04null here. So let's look at this query
4:13:09and let's say where
4:13:12industry is null or
4:13:17do industry
4:13:18is equal to a blank like this. We'll
4:13:22select everything. Let's run this. All
4:13:26right. So it looks like there are a few
4:13:30that are blank. Now, what we can try to
4:13:32do is see if any of these have one
4:13:34that's populated. Let's take Airbnb for
4:13:36example. Let's search for this really
4:13:38quickly. And this is 100% um you know,
4:13:42it's just helpful. It's really really
4:13:43helpful to be able to populate data that
4:13:46is populatable. Is that a word? Um let's
4:13:49try it. So, we'll say uh select
4:13:52everything. I just wanted to do where
4:13:56spell that right where company is equal
4:13:58to and let's do Airbnb.
4:14:03There we go. Let's run this.
4:14:07And it looks like we have this one right
4:14:08here. So, for example, um these whether
4:14:12they have them or not, we're going to
4:14:14try to populate these. If this Bs or
4:14:17Carvana or Jewel had multiple layoffs,
4:14:20these ones should if these ones aren't
4:14:22blank. If they have one that's not
4:14:23blank, we should be able to populate it.
4:14:25For example, um not the one I was trying
4:14:27to do. If we look at Airbnb, this one
4:14:30has travel. So, we know this is the
4:14:32travel industry. So, we can populate
4:14:34this with travel. Again, we want this
4:14:36data to be uh the same. So, if we're
4:14:39trying to look at, you know, what
4:14:41industries were impacted the most, this
4:14:43row isn't going to be affected or this
4:14:45row won't be in our output because it's
4:14:46blank. We want that to be traveled to
4:14:49represent the data properly. So, we want
4:14:50to update it. So, if this one has
4:14:52travel, we should be able to update this
4:14:54row with this travel right here. So,
4:14:58let's see how we can write this. And let
4:15:00me give myself some rows right here. All
4:15:03right. Now, what we're going to need to
4:15:04do is try to do a join here. So, let's
4:15:09try running out in a select statement
4:15:10and then we'll just change it to an
4:15:11update if it works. So, we're going to
4:15:14select everything and we're going to do
4:15:16this from staging two. from staging two
4:15:20and we'll call this ST2
4:15:23and then we'll join on itself because
4:15:26what we're going to do is we're going to
4:15:27check in this table does it have one
4:15:29that is blank and not blank. If so
4:15:32update it with the non-blank one that's
4:15:35essentially uh in layman terms what
4:15:37we're trying to write but writing it out
4:15:38could be a little bit more difficult.
4:15:40Um, so we're going to join on itself and
4:15:44we'll call this let's actually call this
4:15:45table one. T1 and T2 because they're the
4:15:48exact same table. Uh, and we'll do this
4:15:51on and we're going to say T1
4:15:55company is equal to T2 company. So the
4:16:00company has to be the same. That's
4:16:02important. And we probably should do the
4:16:04location is the same as well. Now we'll
4:16:07do and T1.loation location is equal to
4:16:12T2.loation. I'm imagining, you know,
4:16:15there's another Airbnb in like South
4:16:18America somewhere that's called Airbnb,
4:16:20but you know, I'm just imagining a
4:16:22scenario, right, where we have to think
4:16:23about different use cases rather than
4:16:24just large companies. So, those other
4:16:27ones, they may have ones that are in
4:16:29different locations. We don't want those
4:16:31um we don't want to change them if
4:16:32they're not the same. So, these are the
4:16:34same. Now what we want to find is we're
4:16:37going to say oops we want to say where
4:16:40then we'll do t1.industry
4:16:44is null and then we want to check that
4:16:47t2.industry is not null. We'll say and
4:16:51t2.industry
4:16:55is not null. And let's just run this.
4:16:58Let's see if we get anything. So let's
4:17:00think this through because we got
4:17:01nothing in our output. We're selecting
4:17:04everything. We're joining on the company
4:17:07and the company um and the location
4:17:09where T1 industry is null and T2
4:17:12industry is not null. Let's just get rid
4:17:14of this for a second. I just want to see
4:17:15if this changes anything. It doesn't.
4:17:17And it's possible actually that instead
4:17:20of doing is null, we could do or
4:17:24and this I'm glad we're walking through
4:17:26this. We can do or
4:17:28is equal to blank. And let's try running
4:17:31this. There we go.
4:17:34Okay, so it looks like there's Jewel,
4:17:38Carvana, and Airbnb. These ones all have
4:17:41industries um where it's null or blank
4:17:44and an industry is not null. So that's
4:17:47really good. Now, if we scroll over, see
4:17:49the industry here. This is our T1. This
4:17:51is our first table. If we scroll over, I
4:17:53bet we'll see the T2 industry where it's
4:17:55not null. Let's scroll over. And here's
4:17:59our industry. We have travel,
4:18:00transportation, and consumer. So, this
4:18:04worked exactly as we had hoped. I can
4:18:06even um pull this up here just to show
4:18:09kind of show you a little bit easier
4:18:11what that's doing. And we'll do
4:18:12T2.industry.
4:18:14This is kind of like what we're trying
4:18:15to do. So, if it's blank, this one is
4:18:19going to be populated into here if there
4:18:21is one that is not blank. So, that's
4:18:23essentially what we're going to do.
4:18:24Let's write the update statement and
4:18:26we're going to see if it works. This we
4:18:28have to translate this to an update
4:18:30statement. So we'll do update and we're
4:18:32going to update uh this right here. So
4:18:35we'll say update T1 and then we'll do
4:18:38the join
4:18:40right there. And now we have to do a set
4:18:42statement. So we'll set uh the
4:18:44T1.industry
4:18:47equal to and I'll just copy this
4:18:50T2.industry. I just don't like I don't
4:18:52like writing things out. Um then we say
4:18:54where. So we do this
4:18:58just like that.
4:19:01and let's add a semicolon.
4:19:03Okay, let's confirm. So, we're updating
4:19:05this table T1. We're joining on T2 where
4:19:08the company is the exact same. We're
4:19:11setting T1 industry equal to T2
4:19:13industry. So, the T1 should be the blank
4:19:15one. So, where the T1 industry is null
4:19:17or blank and T2 industry is not null.
4:19:22Let's go ahead and run this semicolon.
4:19:26See if there were about three updated.
4:19:28Yep. Rematch zero rows affected though.
4:19:32Let's go take a look. We have to let's
4:19:36run this query. Looks like those are
4:19:38still null. Let's run this. Uh, that one
4:19:41is still blank. Now, let me think here.
4:19:43I'm I'm trying to think of why this
4:19:44didn't work. And I want to walk you
4:19:47through my thought process.
4:19:49It is possible that because these are
4:19:52blanks and not nulls that it's not
4:19:55working. I and I will say that is
4:19:56something I typically do where I set
4:19:59these blanks to nulls first. So let's
4:20:03actually try that and see if that
4:20:06changes anything. I'm just going to
4:20:08update uh this. I'm going to say set the
4:20:14industry
4:20:16equal to null. We'll say where industry
4:20:21is equal to blanks. So, we're just
4:20:24changing it to null where it's blank.
4:20:26Let's try this.
4:20:29And let's go back down here to our
4:20:30select statement.
4:20:32So, these are all nulls. Okay. I think I
4:20:36think this is now going to work because
4:20:38now you can see on this side it's going
4:20:40to it there's only one option for it to
4:20:42populate it. Before there were those um
4:20:45blanks which I think was causing the
4:20:46issue. Um, let's get rid of this part
4:20:51because now we have no nulls.
4:20:53And now let's try running this. We're
4:20:56workshopping this on the fly, guys. Uh,
4:20:58let's see. Three rows affected. Heyo.
4:21:01All right. Let's go see if it worked.
4:21:03Um, let's run this query. And we have
4:21:06none. That's perfect. Let's look at
4:21:07Airbnb.
4:21:11All right. All right. Ran into some
4:21:12issues, but we worked through it. We
4:21:14figured out the issue and now it's
4:21:16working properly. And we can even come
4:21:18back up here to select everything. And
4:21:21it looks like Bailey's is the only one
4:21:23that still has a null. Let's look up
4:21:25Bailey's real quick and we'll say our
4:21:28company is like uh Bailey.
4:21:33Let's run this. Yeah. And there's only
4:21:36one. So there wasn't another row. All
4:21:38these other ones like Carvana and um I
4:21:41can't remember where the other Jewel and
4:21:43Airbnb, those ones had an extra row.
4:21:45They did multiple layoffs. This one only
4:21:47did one layoff. So, we don't have
4:21:49another populated row where it's not
4:21:51null to actually populate the null row.
4:21:54That's really all that happened. Uh
4:21:56that's why that worked that way. So, I'm
4:21:58really happy that worked. Awesome job,
4:22:00guys. Uh I was starting to question
4:22:02myself. Do I even know how to use my
4:22:05SQL? I mean, I was really starting to
4:22:06question my abilities here.
4:22:08um take a look. Uh I think that is all
4:22:12we're going to do for populating null
4:22:15values. Now, here's why. Things like
4:22:18total laid off, percentage laid off, um
4:22:21funds raised, how are we going to
4:22:23populate that with the data that we have
4:22:25here? I don't believe we can. Now, we
4:22:28might be able to populate Oops. We might
4:22:31be able to populate some of this if we
4:22:33had the um company total like if we had
4:22:36the original total before laid off
4:22:38because then we could do calculations
4:22:40like um oh these companies went
4:22:42completely out of business. That's not
4:22:44good. At 1% that means 100% was laid
4:22:46off. Um [clears throat] but if we had
4:22:49you know the total they had 50 employees
4:22:52and 100% were laid off. We could
4:22:53populate the total laid off. Whoops. Did
4:22:56it again. We could populate the total
4:22:58laid off by saying if this is 50 100%
4:23:01was laid off that's 50 people were laid
4:23:03off. We don't have that data so we can't
4:23:05go and populate it I don't believe funds
4:23:08raised we might be able to scrape some
4:23:10data from the web and populate this but
4:23:12that's a totally different thing um not
4:23:15part of this project. So I think the
4:23:17data cleaning for the null values and
4:23:19blank values I think that's going to be
4:23:21done. Um, it's possible that the stage
4:23:24could be the same and if you want to go
4:23:25check, you can, but we're going to keep
4:23:27chugging along because we want to remove
4:23:29columns and rows that we need to. Now,
4:23:32if you remember, we were looking at this
4:23:33before. Did I save that uh query? Let's
4:23:36go look. Here we go.
4:23:39Bring this down to the bottom.
4:23:42All right, these rows. Let's let's
4:23:46really take a look at these um and think
4:23:47about if this is going to be help to us.
4:23:49Um what we are trying to do with this
4:23:53data in the near future is we're not
4:23:55just trying to identify a company or a
4:23:57location that had layoffs and maybe we
4:23:59are maybe that maybe we are trying to do
4:24:01that but these have no layoffs and no
4:24:04percentage laid off. So in my opinion I
4:24:07don't know if these laid off any at all.
4:24:10Um I believe that we can get rid of
4:24:13these. Now deleting data is a very
4:24:17interesting thing to do. You have to be
4:24:19confident. Am I 100% confident? No, not
4:24:21really. But I'm confident enough to know
4:24:23that what we're about to look at in the
4:24:25next one, we're going to be using these
4:24:26total laid off a lot, percentage laid
4:24:28off a lot when we're looking at um you
4:24:31know, actually querying the data and
4:24:33doing some exploratory data analysis.
4:24:35So, we're going to use these a lot. I
4:24:36don't think uh these I'm not even sure
4:24:40if these are accurate. I'm not even sure
4:24:42if they actually did have a layoff. It's
4:24:44saying they did, but it doesn't show if
4:24:46they laid off any. So, um, can we delete
4:24:49this? Yes. Should we delete this? It's
4:24:52iffy. Uh, I'm not 100% if I'm being
4:24:54completely honest. And there's a lot of
4:24:56rows like that. This is This could be
4:24:58like 100 or so. Really not. I mean, I
4:25:01could run a query and run it, but I
4:25:02don't want to. I don't It's not a big
4:25:03deal. The point being, I don't think we
4:25:06need this information. So, we're going
4:25:07to get rid of it if nothing else just to
4:25:10show that you can do it. So, now we'll
4:25:12say uh delete. And then we'll do from
4:25:15here. There we go. So now we're going to
4:25:18delete these rows. Let's try to select
4:25:20them again. And they are gone. So we
4:25:22deleted the ones where the total laid
4:25:23off was blank and the percentage laid
4:25:25off was blank. We just I can't trust
4:25:26that data. I really can't. Um and let's
4:25:30go back down.
4:25:32Come right here.
4:25:35Semicolon. So I sometimes I have to walk
4:25:37myself through these things. Um all
4:25:40right. this row num. I mean, come on. We
4:25:44don't need that anymore. Let's get rid
4:25:46of it. Um, so what we can do now, it's a
4:25:49little bit different syntax. We want to
4:25:51drop a column from this table. So, we
4:25:54have to do the alter table again. So,
4:25:56we're going to alter table layoff
4:25:59staging two. And then we're going to say
4:26:02drop column and row_num.
4:26:07If we run this
4:26:09then we run the table again should be
4:26:12gone and it is. So this is it. This is
4:26:16our finalized clean data. Now in the
4:26:19next project we're going to be doing
4:26:21exploratory data analysis on this
4:26:23cleaned data. We're going to finding
4:26:25trends and patterns and running complex
4:26:27queries. It's going to be phenomenal.
4:26:29I'm super excited about it and I love
4:26:30this data cleaning one. Um I made some
4:26:33mistakes. I'll be the first one to
4:26:34admit. But cleaning data is not always a
4:26:37straightforward thing. Um, you know, you
4:26:39have to you kind of mess around with it,
4:26:41figure it out. Uh, and and you know,
4:26:44that's what we did. Uh, whoa, took a
4:26:47while. So, just to recap, we removed
4:26:50duplicates, we standardized the data, we
4:26:53looked at the null values or blank
4:26:54values, and we removed any columns
4:26:58or rows. So, we did a lot. Um, and if
4:27:00you go back and you actually scroll
4:27:02through here and look at some of this
4:27:03code that we wrote, uh, it's not super
4:27:06beginner stuff. So, if you're following
4:27:08along with these things and you are
4:27:10getting this project, this is a
4:27:11fantastic project to put on your
4:27:13portfolio. I myself would put this
4:27:15project on my portfolio because it's a
4:27:17very, very relevant thing. So, I hope
4:27:19this was helpful. I'm just going to keep
4:27:21scrolling while I talk, but I hope this
4:27:22was helpful. I hope you learned
4:27:23something. We did a we did a lot of
4:27:25different things that we didn't even do
4:27:26in the lessons, which I like doing
4:27:28because you can't cover every single
4:27:30aspect of my SQL in lessons, right?
4:27:32Sometimes you just got to get in there,
4:27:34get into the nitty-gritty, clean some
4:27:36data, and you'll find uh or discover new
4:27:38things, try new things. Um, and now
4:27:42we're getting to the bottom. And awesome
4:27:44work, awesome, awesome, awesome work.
4:27:46Uh, this is an A1 project. I think this
4:27:50should be in everyone's uh portfolio. If
4:27:52I don't see it in your portfolio and you
4:27:54know you send it to me, I'm going to say
4:27:56it's the garbage portfolio. So, this is
4:27:58a good one. So, with that being said,
4:28:01thank you guys so much for watching. I
4:28:02If you made it all the way to the end,
4:28:04you're still listening to me. Awesome
4:28:05work. Really awesome work. For real. I
4:28:09you know, you're just following along
4:28:10with the tutorial. That's what it feels
4:28:11like. But by the end of this, I I just
4:28:14know you're learning a ton and you're
4:28:16you're trying new things and you're
4:28:18really pushing yourself beyond just
4:28:20simple tutorials. So trust me when I say
4:28:23this is not easy. Not everyone was able
4:28:25to make it to the end. So great work
4:28:26getting here. So I will uh see you guys
4:28:29in the next project when we actually
4:28:31explore this data. We'll walk through a
4:28:33lot of different ways to do that. So
4:28:35thank you again for watching. If you
4:28:37like this, be sure to like and subscribe
4:28:39below. I put out tons of content about
4:28:41all this stuff and I absolutely love it.
4:28:43It is definitely one of my passions in
4:28:44life. So go ahead and do that and I will
4:28:47see you in the next video.
4:29:00Hello everybody. In this project, we're
4:29:02going to be focusing on exploratory data
4:29:04analysis. Now in the first project we
4:29:06worked with this exact data set and we
4:29:08cleaned up the entire thing and that was
4:29:10a really good project and it set us up
4:29:12to explore the data and with all that
4:29:15clean data we'll be able to look at our
4:29:17data much better and find better
4:29:19insights while we are using it. Now
4:29:20normally when you start the EDA process
4:29:22or the exploratory data analysis process
4:29:24you have some idea of what you're
4:29:26looking for sometimes not always and
4:29:29sometimes when you're exploring the data
4:29:31you also find issues with the data that
4:29:32you then have to clean. So even though I
4:29:35did a data cleaning video and then an
4:29:37exploratory data analysis video and
4:29:38they're kind of separate projects,
4:29:40sometimes those coincide together where
4:29:43you're exploring it and cleaning it at
4:29:44the same time. Now what we're going to
4:29:46be doing here with this data set, we're
4:29:48just going to be kind of exploring it. I
4:29:50don't have any agenda. I don't have any,
4:29:53you know, one thing that I want to look
4:29:54at. I just kind of want to look at
4:29:56everything and we'll kind of discover
4:29:58and go uh about things as we are
4:30:01learning and looking at this data set.
4:30:02We will however start off really simple
4:30:05with kind of the basics, work a little
4:30:06bit more towards the tougher stuff and
4:30:08then at the end we'll have some more
4:30:09advanced things that I think will be
4:30:11really fun. So with that being said,
4:30:13let's start off with kind of more easier
4:30:15things. We'll kind of just ease our way
4:30:17into exploring this data set. Let's pull
4:30:20this down and let's copy this right down
4:30:22here. Now we're going to be working with
4:30:25this total laid off and percentage laid
4:30:27off or most likely this total laid off
4:30:29quite a bit. The percentage laid off
4:30:31isn't super helpful because we don't
4:30:33know how large the company is. We don't
4:30:35have another column here that says
4:30:37here's how many total employees they
4:30:38had. And then okay, they had a
4:30:40percentage laid off. You know, we won't
4:30:43work as much with this one, but we'll
4:30:44work quite a bit with this total laid
4:30:46off. Let's look real quick. We could
4:30:48look at something like the max uh total.
4:30:52And I need to use a parenthesis max
4:30:54total laid off. And let's look at this.
4:30:59So on one day there was somebody out
4:31:02there who had the max total laid off of
4:31:0512,000 people. That's a lot of people to
4:31:08lay off in one, you know, one go. That's
4:31:11a lot. Let's also take a look at the max
4:31:15and I think it was percentage laid off.
4:31:20Let's run this. And it looks like one.
4:31:22Now one represents 100. That means 100%
4:31:26of the company was laid off. Um, and
4:31:29that's, you know, that's not great. Uh,
4:31:31that just means an entire company went
4:31:32under essentially. We can actually take
4:31:34a look at that because I'm interested to
4:31:36see, you know, if there's any companies
4:31:37I recognize or can see, um, where, let
4:31:42me come right down here where the
4:31:45percentage laid off is equal to one.
4:31:49Let's go ahead and look at this and
4:31:52let's take a look. So, we have this
4:31:55ahead. I'm just going to go through here
4:31:56and see if I recognize any of these. Uh,
4:32:00some in the crypto space. BlockFi. I
4:32:03feel like I recognize that one. I don't
4:32:05know. Uh, let's keep going. Deliveroo.
4:32:09It's not good. They left like let go of
4:32:11120 people. Uh, I'm just curious. I
4:32:13mean, I'm I'm just kind of scrolling
4:32:14through here trying to see if I
4:32:15recognize any. These are companies that
4:32:17like completely went under or or lost
4:32:20all their employees. Volt Bank.
4:32:24Interesting. just interesting to me.
4:32:25We're going to be taking a look at a lot
4:32:27of stuff. Um, but these are companies
4:32:28that completely went under and that's,
4:32:31you know, unfortunate. We can also order
4:32:33by uh total_laid
4:32:36off in that's not how you spell it. In
4:32:39descending, we'll see which company went
4:32:41under had the largest. So, this one had
4:32:442000. Construction company had 2400
4:32:46people they went um under. Doesn't say
4:32:49what stage they were at, but that's in
4:32:50the United States. We can also take a
4:32:52look at and there's another column over
4:32:53here called funds raised in millions.
4:32:56Let's look at that one. So I want to see
4:32:58um these are companies that had a lot of
4:33:01funding or potentially a ton of funding.
4:33:05Uh let's go over. So this is like $2.4
4:33:08billion I believe. Like I think this is
4:33:10like a ton of money. Um Quibby I believe
4:33:14I know this company uh in BlockFi. I I
4:33:16thought I had heard of them. I'm pretty
4:33:17sure I know who that is. So, Whibby is
4:33:19one that I'm definitely familiar with.
4:33:21It was like a short form uh media
4:33:24company. Yeah. Yeah. And then there's
4:33:26British Volt, which looks like an
4:33:28electric company that went under. So,
4:33:29you know, some big companies that went
4:33:31under um in 2023, 2020, 2022. So, that's
4:33:36interesting. So, we have a lot of
4:33:37companies here and we're just looking at
4:33:39um that had total laid off. But, let's
4:33:42take a look. Let's let's use group by
4:33:45real quick. I want to look at the
4:33:46company and I also want to look at the
4:33:49sum of the total laid off. And for that
4:33:54we need to use a group by the company
4:33:57and let's just start with this and I'm
4:33:59sure we'll use an order by in a second.
4:34:01Yeah, let's order by
4:34:04order by let's just do two for now in
4:34:07descending
4:34:09and two stands for one two this is the
4:34:12total it off. So, uh, for the total for
4:34:15this table, and we don't know how far go
4:34:17back it goes. We haven't checked that
4:34:18yet. We'll check that in a second, but
4:34:20for this table, you should recognize a
4:34:23lot of these companies. So, I think it
4:34:25starts in like 2020 until like sometime
4:34:28in 2023, but this is Amazon let go of
4:34:311,800 people, Google 12,000. I'm
4:34:33guessing that's at one time because that
4:34:35was the max that we looked at earlier.
4:34:37Uh this is Facebook or Meta, Salesforce,
4:34:40Microsoft, Phillips, Uber, Dell, Cisco,
4:34:43Pelaton. I mean these are a ton of big
4:34:45companies. Arvana, they let go of
4:34:48thousands and thousands and thousands of
4:34:49people. Twitter, that's not surprising,
4:34:52uh given what's the change of things.
4:34:54Groupon, um ton of ton of people or a
4:34:57ton of companies and that's a lot of
4:34:58people that have been let go. Now, let's
4:35:00really quickly uh before we keep going,
4:35:02I want to look at our date ranges real
4:35:04quick. So, let's select everything. Um,
4:35:07whoops. We'll do from there. And how do
4:35:10we want to do this? Let's do minimum of
4:35:12date. And let me do it like this. Date.
4:35:17And then we'll do uh the max as well
4:35:19because I want to look at the date range
4:35:21that we have here.
4:35:23Let's run this.
4:35:25It looks like it starts in 2020 of 311.
4:35:28So right when like I believe the
4:35:29pandemic started or the uh COVID 19
4:35:32started. I want to say that's like right
4:35:34when it hit at least us in the United
4:35:35States. Then this is almost exactly
4:35:37three years later. So early 2023. So
4:35:41just in those three years, you know,
4:35:44here's some of what we're looking at.
4:35:46These companies have let go of quite a
4:35:47few people or had layoffs. We could also
4:35:49take this exact thing. Oops. What did I
4:35:53do here? Copy this again. We can also
4:35:56take this exact thing and look at quite
4:35:57a few other things. There was um the
4:36:00industry. So we can look at industry
4:36:02like what industry got hit the most
4:36:04during this time or had the most
4:36:06layoffs. Um all we're looking at right
4:36:08now is total laid off. We can also look
4:36:10at um percentage in a little bit but
4:36:13looks like consumer got hit really hard,
4:36:15retail really hard. That makes a lot of
4:36:17sense with shops closing down because
4:36:19people couldn't come in for the corona
4:36:21virus. Now we're just making
4:36:22assumptions, right? Um but you know
4:36:25during that time it was mostly COVID
4:36:27that impacted a lot of stuff. Then we
4:36:29have transportation, finance,
4:36:30healthcare, food, real estate. Um, yeah,
4:36:34there's a lot a lot of people. Let's
4:36:35look at the lowest ones. Manufacturing,
4:36:38fintech, aerospace, energy, legal. So,
4:36:42low numbers on those, high numbers on
4:36:45these. So, really, really interesting.
4:36:47Uh, let's go back up. Just want to look
4:36:50at our whole table really quickly. See
4:36:51what we got while we're looking at this
4:36:54stuff. And let's run this. Now we looked
4:36:57at the company, looked at the industry.
4:36:59I would really be interested to look at
4:37:00the country as well. Which countries at
4:37:03least from this data set and we can copy
4:37:05or we can go right here country because
4:37:09I believe that United States had the
4:37:12most. Holy mackerel, they had by far the
4:37:17most. Uh then India, this is 256,000
4:37:21people um lost their jobs. We'll look I
4:37:24think we'll look at the dates in a
4:37:25little while like at kind of like time
4:37:27series like how many per year per month
4:37:29per day or whatever we want to look at
4:37:32but goodness gracious uh that's a lot of
4:37:34people within just three years in the
4:37:35United States India Netherlands Sweden
4:37:38Brazil Germany uh United Kingdom then it
4:37:40goes down and down and down but these
4:37:42are just reported um from this data set
4:37:45that I I had gotten. So really really
4:37:48interesting. Good night. the United
4:37:50States had much more than than most for
4:37:53sure. Um, let's actually look at that
4:37:55date real quick or we can look at it by
4:37:57year. Um, so we have this date and if we
4:38:01do it like this and we can
4:38:06do by date real quick. So this is going
4:38:08to do it by individual date. And let's
4:38:11order by let's do one. So this is the
4:38:14most recent date. So it's literally by
4:38:16date that's reported. Um, we don't want
4:38:19that. Let's do it by the year. So, 2020,
4:38:222021, 2022, 2023. We can do that fairly
4:38:26easily. We'll use this year function.
4:38:30And we'll group by
4:38:33the year as well.
4:38:36Let's try running this. There we go. It
4:38:39looks like in 2020, 80,000 people. 2021
4:38:43uh 16,000.
4:38:45160,000. in 2022. This looks like the
4:38:48worst year. And then it's only we only
4:38:50have three months of data in 2023.
4:38:53There's 125,000. Holy smokes. So in
4:38:562023, it looks like we're ramping up
4:38:58because I'm recording this in 2023,
4:39:00about a month after this data set that
4:39:02we got this data set. There's 125,000
4:39:05people um around the world, you know,
4:39:07but just in those first three months. So
4:39:10this is going to be a lot higher than
4:39:12even 2022. That's pretty wild. Um very
4:39:16very interesting
4:39:19one other one one while we're looking at
4:39:20group by um there's there was a column
4:39:24and you can go back and look at it if
4:39:25you'd like but it's called stage and
4:39:27this shows the stage of the company and
4:39:30if we run this and we're all just
4:39:31looking at total but if you look at the
4:39:34um stage of the company this is like the
4:39:37different series that they're in A B C D
4:39:40A I believe is like a series A funding
4:39:42that's like a super super starting oh
4:39:44This is like a seed phase. Then there's
4:39:46series A and then it goes up up up up
4:39:48until usually they go um like they do
4:39:51IPO or they get acquired or something.
4:39:53Now if we go up here and we do two
4:39:55descending I want to see which one had
4:39:56the most. So this is post IPO. This is
4:39:58the Amazon, the Googles of the world,
4:40:00the large large companies that are post
4:40:03IPO or initial public offering. Then
4:40:05there's unknown. We don't know which
4:40:07that is. Um a lot of you know layoffs
4:40:09from acquisitions CD B all the way down.
4:40:14So, it looks like um most of it's coming
4:40:16from, you know, these ones right here.
4:40:19Really, really interesting. Let's go
4:40:21look at percentages. I'm just going to
4:40:22literally crying to say literally. I'm
4:40:25going to literally copy these. Um and
4:40:29with percentages, I don't think uh let
4:40:33me look at percentage. I don't think the
4:40:34sum is going to be a good indicator. I
4:40:36don't know if this is a good one to even
4:40:37look at because and then we're looking
4:40:40at company right now because percentages
4:40:42refer to a percent of the company,
4:40:45right? So, we don't have hard numbers
4:40:48because we don't know how large these
4:40:49companies are. So, now that we're
4:40:51actually looking at this, this
4:40:52percentage laid off isn't super
4:40:54relevant. Um, really the one that's kind
4:40:56of more, you know, has better this is a
4:40:59better use for what we're looking at is
4:41:01this total laid off because again, we
4:41:02don't know these sums. We could we could
4:41:04look at like the average, right? Um but
4:41:07again, that just doesn't help us that
4:41:09much. I don't think um I think we're
4:41:14going to really dive into that too much
4:41:17is my uh is my feeling. Now, one thing
4:41:21that I would be really interested in is
4:41:24to kind of look at the progression of
4:41:26layoff, right? Uh you could call this a
4:41:29rolling sum. So, start at the very
4:41:31earliest of layoffs and do a rolling sum
4:41:33until the very end of these layoffs. Um,
4:41:36and let's go to the bottom. This is
4:41:37where it's going to start getting a
4:41:39little tougher. Um, and there's, you
4:41:42know, we're just doing a little bit of
4:41:43exploratory data analysis. You know, do
4:41:45digging into this a little bit. You can
4:41:47go and dig into this as much as you'd
4:41:50like. You don't have to just do what I'm
4:41:51doing, but I'm just trying to show you
4:41:52some stuff. Now, let's try to do rolling
4:41:54total of layoffs. Um, we could do that
4:41:57on the day, although I feel like that's
4:42:00going to be way too many rows. Let's do
4:42:02it based off the month. So, right here
4:42:04in this month. Now, let's see if we do
4:42:07just the month. Let's do something. I'll
4:42:09show you the month. And that's going to
4:42:10be an issue. And I'll in my head I
4:42:12already know. But let's look at it. We
4:42:15could do something like select um from
4:42:18and let's get this.
4:42:22There we go.
4:42:24So, if we do um we'll do substring. Let
4:42:27me add a semicolon. Let's do substring.
4:42:31And we want to pull out this month right
4:42:34here. So, we'll go one, two, three,
4:42:36four, five, six. So, start at position
4:42:39six. Um, and this is of course in the
4:42:42date column. We'll start at position six
4:42:44and then we'll take two. Let's just run
4:42:46this really quickly. And there's our
4:42:49month. So, this we can do this as month,
4:42:52right?
4:42:54um or like this.
4:42:57Is that correct? Yeah. So, as month. So,
4:43:00this is our month that we're doing it.
4:43:01Now, if we group on this and we do like
4:43:04something like a sum of total uh laid
4:43:08off, I think that's the column. And then
4:43:10we do a group by on this month. So, it'
4:43:13be like this right here.
4:43:17We'll do
4:43:19group by
4:43:22this. Let's try running this. We should
4:43:25be able to do month as well. Let's try
4:43:27this real quick as well because I don't
4:43:29want to have this if I don't have to run
4:43:32it. Perfect. So the months right here
4:43:35don't show us the year. So if we're
4:43:37trying to get a rolling to total of just
4:43:39the month, it's actually would work fine
4:43:42when we actually implement the the
4:43:44rolling total use um you know a window
4:43:46function. But the issue with this is
4:43:48it's just going to show us month. So
4:43:49this is 2020. This is January of 2020,
4:43:532021, 2022, 2023, any other years we
4:43:56have it. This is not a great rolling
4:43:58total. What if we did one all the way to
4:44:02I want to say it's seven, six, seven.
4:44:04Let's try this. Now, this is going to
4:44:07give us a much better Let's order this.
4:44:09Order by one.
4:44:12This is just our first column. So, now
4:44:14uh well, we should do it where it's not.
4:44:16Give me a second. I'm I'm I'm figuring
4:44:18this out as we go. We'll do where uh the
4:44:21month write that where the month is not
4:44:25null. I'm just going to get rid of that
4:44:27one.
4:44:34And of course, uh that doesn't work
4:44:37because we're looking at the substring.
4:44:38So, let's try doing this.
4:44:42There we go. Um it just wasn't reading
4:44:44in that month that I was trying to use.
4:44:46Let's go down. Now, here's what we're
4:44:48going to do is we want to take it from
4:44:50the very first month and we're grouping
4:44:52everything. So, these are all the
4:44:54layoffs from 2020 of 03. So, that's
4:44:57March of 2020. Then we have April, May,
4:44:59and these are the layoffs. So, this is
4:45:01really good. This is exactly what I was
4:45:03imagining in my head. So, we want this.
4:45:06This is just, you know, 12 months in a
4:45:08year, and we go all the way to the
4:45:09bottom. And I want to do a rolling sum
4:45:12of this. So, let's see how we can do
4:45:14that. And we'll use this logic in a
4:45:16little bit. Let's copy this
4:45:19and let's do select everything. We'll do
4:45:22right here. Now, what we actually want
4:45:24to do now that I'm thinking about it is
4:45:25we want to take this data and we want to
4:45:28do the rolling sum based off this exact
4:45:30thing. So, we actually need to take uh
4:45:33this. Let's get rid of this. And we'll
4:45:36do it with a CTE. So we'll say width and
4:45:39we'll do rolling_total
4:45:43that we'll say as and then we'll put
4:45:46this in here just like that. So with
4:45:50rolling total as now we're going to say
4:45:53select and we'll just do from here. Now
4:45:56what we need to do is we need to select
4:45:58the month. So let's go ahead and select
4:46:00that month and we'll take it just like
4:46:02this. So we'll select the month and we
4:46:04need to do a rolling total. All we have
4:46:05to do for that is the sum of which
4:46:08column we're doing. Let's actually
4:46:10change this real quick. Um we're going
4:46:12to call this as
4:46:15um total
4:46:18off. I'm just going to keep it simple.
4:46:21So the sum of total off. So now we're
4:46:24doing that, but we want to do it over.
4:46:26And all we need to add into here is an
4:46:28order by. We're not going to partition
4:46:29by anything because in here we already
4:46:32did a group by. So it's, you know, kind
4:46:34of like partitioning it. We just need to
4:46:36say order by and we just need to order
4:46:38by the month, I believe. So let's try
4:46:41that.
4:46:44And let's run it. Let's do that. And we
4:46:47actually need to since we're doing um
4:46:50this, we need this at the end. And we
4:46:52can rename this if we'd like. So we can
4:46:54do this as rolling total all lowercase.
4:46:58Let's try running this and let's see
4:47:01what we get.
4:47:05Okay. And this looks correct. So,
4:47:08starting in 2020 of 03, we had 9,000
4:47:11layoffs. Then the next total we added
4:47:14onto here. Now, this visually isn't the
4:47:17best. I would like the month right here
4:47:20as well. So, let me actually add um let
4:47:24me create its own row. Put a comma here.
4:47:28Then right here, I want to keep this
4:47:31total off so we can visually see better.
4:47:35Much better. Okay, so we have the month
4:47:38and as it goes down, we're having more
4:47:41laid off. Now, this is our rolling
4:47:43total. Here's essentially how this
4:47:45works. It starts with 9,628.
4:47:48Then it adds on the next month, which is
4:47:5026,000, which equals 36,000. Then it
4:47:53adds on the next month, and we get 62.
4:47:56adds on the next month 69, right? It
4:47:58keeps going all the way down. This just
4:48:00shows each month how many were laid off
4:48:03and this shows a monthby-month
4:48:05progression all the way down to the
4:48:06bottom. So, let's keep let's just, you
4:48:08know, take a look. In 2020 of 03, we had
4:48:109,000. By the end of 2020, we had about
4:48:1481,000 or so. Then, at the beginning,
4:48:17right here all the way down to 2021. By
4:48:21the end of 2021, we only had 96,000. So
4:48:242021 was a good year. It looks like um
4:48:27comparatively we had 90 80 well let me
4:48:30see 91,000 people let go and here we
4:48:34only have 96,000 let go. So that's what
4:48:37uh 81 that's only like 15,000 people.
4:48:40That's like nothing um comparatively.
4:48:42Then in 2022
4:48:44uh things start ramping up dramatically.
4:48:47It looks like we have um 12,000 people,
4:48:5017,000, 16,000, and they're adding up.
4:48:53It's going from 97 all the way up to
4:48:56good night right before the holidays in
4:48:592022 of this past year. I mean, we had
4:49:03uh 247,000 people. So, that's like uh
4:49:0613ome,000. My math my math's really bad.
4:49:09It's like 150,000.
4:49:11And then we only have Oh, we have even
4:49:13more here actually. And then we only
4:49:15have the first three months of 2023. So
4:49:18these months right here were really
4:49:22devastating. It's just around the world.
4:49:24Now, we can also break this out
4:49:26potentially
4:49:27by country. So we can see how many per
4:49:30country, but this is just around the
4:49:31world. That's a lot of people losing
4:49:33their jobs all the way up to 383,000.
4:49:35So, in this range, 383,000
4:49:38from March of 2023 all the way back to
4:49:41March of 2020 lost their jobs. And this
4:49:44is just reported. I'm sure there was uh
4:49:46you know, much more than that, but this
4:49:47is that like tech companies, larger
4:49:50companies that have like series A
4:49:52funding, IPOs, etc. Um, but a lot of
4:49:54small businesses went out of business.
4:49:56Um, so we don't we don't have that
4:49:58information in this data set. So I think
4:49:59that's what we're going to do next is
4:50:01kind of look at the company maybe
4:50:03because I'm always interested in the
4:50:04company and actually earlier let's not
4:50:07do that one earlier we're looking at the
4:50:09company the sum of totally loft let's um
4:50:13let's bring this down let's run that
4:50:15because that that's what rolling total
4:50:17is by the way rolling totals are great
4:50:19really good for visualizations as well
4:50:22um let's see yeah so I want to take a
4:50:24look at these companies but I want to
4:50:26see how much they were laying off per
4:50:28year. So, instead of just looking at it
4:50:30as a total, we'll break it out by the
4:50:32year. Now, I'm just going to warn you,
4:50:34this probably going to this most likely
4:50:36will be our last one in the in the
4:50:37lesson. This is going to be probably our
4:50:38hardest one yet. Um potentially. We'll
4:50:41see. Maybe the other one was earlier. Uh
4:50:44was harder earlier.
4:50:46Now, let's use this kind of as um a
4:50:51starting point. But what we're going to
4:50:53need to do is we want to take the
4:50:54company, but I also want the date. So I
4:50:56need to do a comma then date. So we need
4:51:00our date here and I'm going to do that.
4:51:03I need to group by the date as well. So
4:51:05we'll do date and let's run this. All
4:51:08right. Now this is just doing the, you
4:51:11know, company and the exact date. We
4:51:13don't want to do that. Let's actually do
4:51:14the year. Let's just look at the year. I
4:51:16think that'll be plenty. You could also
4:51:18do the exact same thing as we did above
4:51:19with the substring. Um although I think
4:51:21that's going to get a little messier. um
4:51:24stuff, you know, just a thought. Let's
4:51:27run this. Okay, so now we're looking at
4:51:30just the year. We're grouping by year
4:51:32and let's order by uh let's say the
4:51:35company. And we'll do that in sending.
4:51:39There we go. And let's run this. So now
4:51:41we have it open. Let's see who, you
4:51:43know, you can see people who made
4:51:44multiple layoffs. This is in 2020, they
4:51:46let go of 200. And then in 2023, they
4:51:49let go of 155. This is a company I've
4:51:52never heard of. So this is already
4:51:53looking really good. Now let's say we
4:51:56wanted to use this and what we want to
4:51:58do is we want to rank which years they
4:52:02laid off the most employees. Now this is
4:52:04just a small uh sample. We'll look at
4:52:06more in just a little bit. We can
4:52:07actually look at um let's just do three
4:52:11uh three descending just like this
4:52:15should be large companies. So, you know,
4:52:17some of these companies like Microsoft,
4:52:20even Amazon right here and Amazon right
4:52:22there, they let go of multiple or
4:52:24thousands of people in different years.
4:52:26So, I want to rank those. I want to say,
4:52:29you know, the highest one uh based off
4:52:31of the laid off should be ranked number
4:52:34one. That's the year that they laid off
4:52:35the most people. So, let's go ahead and
4:52:38try to do that. Thing we need to do is
4:52:40uh do a CTE. We'll start with that. Let
4:52:43me um me add some more things down here
4:52:47so we're good to go. So let's do we'll
4:52:51do with let's do uh company. So this is
4:52:54going to be the company year_year.
4:52:57We'll do it as and that's what this is
4:53:00going to be. This is our company year.
4:53:02And we can do select everything from
4:53:05company year. It's going to be the exact
4:53:07query that we're looking at. Let's go
4:53:10ahead and run this. Okay. So this is
4:53:12good. Now I do want to change these
4:53:14columns and I can do that right here.
4:53:16We'll do company
4:53:18um let's call this years and then we'll
4:53:21do I can do total laid off again. So
4:53:24total_laid
4:53:26off just the sum right total laid off
4:53:29per year. So let's go ahead and run this
4:53:31now. There we go. We have company years
4:53:34and total laid off. So this looks much
4:53:37better. And what we're going to do is
4:53:38select everything, but we want to
4:53:41partition it uh probably based off this
4:53:43years right here. And then we want to
4:53:46rank it based off how many they laid off
4:53:48in that year. So we'll get to see who
4:53:50laid off the most people per year. Cuz
4:53:53some companies like Amazon, they let
4:53:55they let off multiple people per year,
4:53:57but was at the highest per year. That's
4:53:58kind of what we're going to look at. Um
4:54:00so we'll do dense_rank
4:54:03and we're going to do that over. Now,
4:54:05we're going to partition by, oops,
4:54:08that's not how you spell partition.
4:54:09Partition by. We want to partition by
4:54:11the years. So, all of the 2021 layoffs
4:54:14will be in the same partition. All the
4:54:162022 will be in the same partition. And
4:54:18we'll do years. And we want to also
4:54:21order by the total laid off. Now, we
4:54:24want to do that in descending. So, we'll
4:54:26do total laid off descending. And then
4:54:29we want to um add this dense rank to it.
4:54:32So, let's try it. Let's run this. Good
4:54:36night. That's a a big one. So, let's
4:54:40take a look. So, in 2021,
4:54:44it looks like um or 2020, it looks like
4:54:47Uber had the highest. Now, we want to
4:54:49take out these nulls. So, let's do um
4:54:52where years,
4:54:56let's say, is not null. And let's run
4:54:59that. There we go. So in 2020 and that's
4:55:02what we're partitioning on first. It
4:55:03looks like this is one two three. These
4:55:05are the top ones. Um and let's order by
4:55:10and let's do the let's order by the
4:55:12rank. Um first let's call this as
4:55:17bring it down. What do we want to call
4:55:20this?
4:55:22We'll call this as ranking. There we go.
4:55:26So order by ranking
4:55:30ascending.
4:55:33There we go. Now we have our ranking. So
4:55:36in 2020, this is the biggest one of
4:55:39layoffs. 2021, this was the biggest
4:55:42layoff. I guess we'll have to take a
4:55:44look in Meta. In 2022, they had the
4:55:46biggest layoff. And Google had the
4:55:47biggest layoff total for 2023. So, this
4:55:51looks correct,
4:55:53but I kind of want to filter on this
4:55:55ranking to be able to only filter maybe
4:55:58the top like five um companies per year.
4:56:02And I think we can do that. Let's
4:56:06actually get rid of this. I think what
4:56:08we should do is we should add this as
4:56:10another CTE and query off of that. So
4:56:13now we'll call this company_year_rank.
4:56:18So now we have the year rank as
4:56:21we'll have our query oops have our
4:56:24query.
4:56:26So now this is our company year rank. So
4:56:28now if we do select everything from
4:56:31company year rank
4:56:34this we run it. Okay. So now we have our
4:56:38rankings. Let's come down. Now we have
4:56:41our rankings but I just want to filter
4:56:43it based off of that ranking. We'll say
4:56:45uh where ranking is greater than or
4:56:48equal to let's say five. We'll look at
4:56:50the top five rankings. Let's run this.
4:56:54And I said greater than I wanted uh less
4:56:56than. Run that.
4:56:59That's looking good. Okay. So, really
4:57:02quickly, we have in 2020
4:57:05we had these are the top five people who
4:57:07laid people off. Uber, Booking.com,
4:57:10Groupon, Swiggy, Airbnb. In 2021, the
4:57:13largest layoff was Bite Dance, which I
4:57:16think is Tik Tok, right? Uh Catera,
4:57:18Zillow,
4:57:20uh yeah, these are the top five. So 2021
4:57:22and or 2022 and 2023 were definitely the
4:57:25largest as well. We have Meta 11,000
4:57:28people, Amazon, Cisco, Pelaton,
4:57:31and Carvana, as well as Phillips. They
4:57:33tied. That's why we have the dense
4:57:35ranking because some of these will be
4:57:36ties. Then we have Google
4:57:39uh in 2023 all the way down to Dell.
4:57:42These are all ones I know. Microsoft,
4:57:44Ericson, Amazon, Salesforce, and Dell.
4:57:47So, this is really, really interesting
4:57:49just looking at a year-by-year snapshot,
4:57:52right? These are the total laid off for
4:57:54each company. And we could even go back
4:57:56and change this for like industry or,
4:57:59you know, really whatever we want to
4:58:01change this to. This is just an
4:58:02interesting query in general to look at,
4:58:05you know, per year. here. And we could
4:58:06go back and change for month or lots of
4:58:09stuff we can change in here, but this is
4:58:10really interesting to me. Um, it just
4:58:13looks like a lot of the large tech
4:58:14companies had some took some big L's,
4:58:17took some big hits. Um, let's recap this
4:58:20query really quickly in case, you know,
4:58:21it's tough to follow. But we created
4:58:24this query up here and we're looking at
4:58:26the company by the year and how many
4:58:29people they let off. Then right over
4:58:32here we said with the company year we
4:58:35changed these columns. This is our CTE.
4:58:37So we created our first CTE.
4:58:40Then we went and we gave it a rank and
4:58:43we wanted to you know filter on that
4:58:45rank. So we did this rank as another CT.
4:58:48We just did a comma had a second CTE and
4:58:51we hit off the first CT the company year
4:58:54which is right here. So we hit off our
4:58:57first CTE to make this second CTE. And
4:58:59then finally we um queried off of the
4:59:02final CTE. Definitely not an easy query
4:59:05to kind of think through and walk
4:59:06through, but I hope you know you're able
4:59:07to follow um because you know that's a a
4:59:10really good query. This is something
4:59:11I've definitely done in my real job when
4:59:14I was working with a lot of healthcare
4:59:16data. This is a lot of stuff that I
4:59:18would do. And so this is a you know
4:59:20pretty good um pretty good query to know
4:59:23how to do. But with that being said uh
4:59:25we are done with this lesson. I hope
4:59:27this wasn't too short. I don't know how
4:59:28long I ran, but um you know, we looked
4:59:30at a lot of different stuff. Let's go
4:59:32back to the top again. We were just
4:59:34exploring the data. We looked at lay it
4:59:36off a lot. Um looked a lot at the
4:59:39company, uh when these dates actually
4:59:42started for these layoffs in this data
4:59:44set. We looked at the country, the
4:59:46actual year of layoff. Uh then we went
4:59:48to a little bit more difficult things.
4:59:51We looked at it per month. So per month,
4:59:54how many layoffs they had, and then we
4:59:56did a rolling total. This one was a
4:59:58pretty good one using that substring.
5:00:00Um, I love substrings, man. They're
5:00:02awesome or lady. They're awesome. Uh,
5:00:05and then we came down here and we did
5:00:07the one we just did with multiple CTEs
5:00:09in the company. I think it was a a
5:00:11really really good solid project. Um,
5:00:14combine that with that data cleaning
5:00:15project and man, you got a just a really
5:00:17good start with some MySQL projects and
5:00:19this one can be expanded upon. Don't
5:00:21stop where I stopped. Right? Let me go
5:00:23back to the top. Don't stop where I
5:00:25stopped. Right? This data set has so
5:00:29much data in it. You can do a lot of
5:00:31different things. And even if you want
5:00:34to, you could go and find these
5:00:36companies right over here and you could
5:00:39try to uh get their total uh total
5:00:42company that they had and you could use
5:00:45this column a lot more. That'd be really
5:00:47interesting with some calculations
5:00:48there. So with that being said, that is
5:00:50the end of our exploratory data analysis
5:00:52project. I hope you enjoyed it. I hope
5:00:54you learned something both in the data
5:00:56cleaning project and in this exploratory
5:00:58data analysis project. That's what this
5:00:59is all about and getting the confidence
5:01:02and gaining the experience to create
5:01:04these projects and add those to your
5:01:06portfolios. Speaking of which, if you
5:01:08haven't already, check out my video on
5:01:10how to create a free portfolio website
5:01:13uh using GitHub. Awesome. I highly
5:01:15recommend it. You can add these to your
5:01:17portfolio. So, with that being said,
5:01:19thank you so much for watching. I really
5:01:21appreciate it. If you like this video,
5:01:22if you learned anything at all, be sure
5:01:24to like and subscribe below. Check out
5:01:25my channel for tons of other videos just
5:01:27like this one and more. I will see you
5:01:30in the next video.
5:01:44What's going on everybody? Today we are
5:01:45starting our Excel tutorial series.
5:01:50>> [music]
5:01:53>> Now, there are so many things that you
5:01:55can do in Excel. So, I don't know how
5:01:56long the series is going to be. It could
5:01:58be 15 or even 20 videos. But what I do
5:02:00know is that I'm going to be covering
5:02:02just about every single thing that I've
5:02:03used since I became a data analyst. And
5:02:05I want to show you how to do it. Uh so,
5:02:07it won't just be the more concrete
5:02:09things. Um you know, like pivot tables,
5:02:11charts, VLOOKUPs, things like that.
5:02:13It'll also be some of the more nuanced
5:02:14things like how to deal with missing
5:02:16data or how to deal with dirty data and
5:02:18how to clean that up within Excel. And
5:02:20so those are things that you may not be
5:02:22able to do, you know, if somebody wasn't
5:02:24showing you how to do it. And so that's
5:02:25what I'm going to try to help you
5:02:27because I know that that is something
5:02:28that you will need to do or learn how to
5:02:30do in Excel. Without further ado, let's
5:02:32jump on my screen and get started with
5:02:33our very first Excel tutorial. All
5:02:34right, so I'm going to go ahead and get
5:02:35rid of myself. We are going to be
5:02:36looking at something absolutely pivotal
5:02:39in your data analytics career, and that
5:02:41is pivot tables. Uh, and I think that's
5:02:43really appropriate. It is probably one
5:02:45of the most commonly used things I think
5:02:48that data analysts use to convey
5:02:50information in Excel. It's super easy to
5:02:52group things together to display
5:02:53information in a very easily
5:02:56understandable way, especially for
5:02:58people who are not data analysts, right?
5:03:00I use this a lot for other managers or
5:03:02for higherups um, who don't want to get
5:03:05into SQL or or, you know, aren't super
5:03:07techsavvy in like Python or Tableau.
5:03:09They just want it in in Excel. And so I
5:03:12use it all the time for that reason. And
5:03:14so we're going to be using this data set
5:03:15right here, bike store sales in Europe.
5:03:17I will include this link in the
5:03:18description. Um we're not going to look
5:03:20at the columns just yet. We're going to
5:03:21download it. Um I've already downloaded
5:03:23it a few times, [clears throat] but we
5:03:25are going to go to
5:03:28um our downloads. We're going to open it
5:03:30up and we're going to open up this sales
5:03:33right here. And give it a second.
5:03:38All right. Perfect. And so here's what
5:03:39it looks like, at least on my screen.
5:03:41I'm going to spread it out just a little
5:03:44bit. Um, and really quickly, let's take
5:03:47a very quick glance at this. So we have
5:03:50a date, a day, a month, a year. So some
5:03:53um some date information.
5:03:56Um, then we have some customer age
5:03:59information. So how old was the
5:04:00customer? Again, this is bike sales. So
5:04:03what did um, you know, what did they
5:04:05buy? And then we have some demographic
5:04:07information. So this is their age group.
5:04:09We have uh the gender, the country,
5:04:12state, uh the product category, the
5:04:16subcategory, the actual product that was
5:04:17purchased, and then we have things like
5:04:20um you know how much these things cost,
5:04:22the quantity that was that was ordered.
5:04:25So we have order quant quantity, unit
5:04:27cost, unit price. Then we have the
5:04:29profit cost and revenue. all things that
5:04:32we almost everything in here we can in
5:04:36some way put into a pivot table. Now,
5:04:37I'm not going to go through every single
5:04:39variation of that, but we are going to
5:04:41be um looking at a lot of this um
5:04:44revenue over here because I think it's
5:04:46it's pretty easy to show the value of a
5:04:48pivot table with especially with um you
5:04:50know currency or money. So, what we're
5:04:54going to do to get started is we're
5:04:56going to go up to insert and we're going
5:04:58to click on insert and then we're going
5:05:00to click on pivot table. Now, really
5:05:02quick, there is a recommended pivot
5:05:03tables and if you click on that, what
5:05:05will come up is some recommendations
5:05:07that Excel gives based on the data that
5:05:09you have. Um, and it can kind of give
5:05:12you some ideas of of what you can do
5:05:15with pivot tables. It's going to
5:05:16generate it for you. We're not going to
5:05:18do that. We're going to build our own.
5:05:21Uh but let's click on pivot table and
5:05:24it's going to auto select basically
5:05:26everything and that's fantastic. Um but
5:05:29what if it doesn't come like that I I
5:05:31just erased that. If it doesn't come
5:05:32like that you can click right here. You
5:05:34can click excuse [clears throat] me you
5:05:36can click control shift and then the
5:05:38right arrow and then the down arrow and
5:05:40that is going to select all of our data.
5:05:43Um and you have right here a new
5:05:44worksheet or an existing worksheet.
5:05:46We're going to create a new worksheet.
5:05:48just tends to get too clogged up if we
5:05:50put it on the same worksheet that
5:05:51already has a lot of data in it. So,
5:05:54right over here are pivot table fields,
5:05:57and these are all of our columns that we
5:05:59just looked at. And we're going to be
5:06:01able to select those and kind of drag
5:06:02and drop. Now, if you just took the
5:06:04Tableau um tutorial series that I just
5:06:07finished doing last week, then this is
5:06:10going to be pretty familiar. um you're
5:06:12going to start seeing a little bit of um
5:06:15hopefully some patterns about how the
5:06:18data is kind of displayed. And so we
5:06:19have our filters down here. We have
5:06:21columns, rows, values.
5:06:24All of these things uh we will be using
5:06:27I'll show you how to use today as well
5:06:28as some additional things. Um one thing
5:06:32that we want to start with uh for this
5:06:34demonstration is we're going to be
5:06:35looking at kind of the um these bottom
5:06:38ones right here, profit, cost, and
5:06:40revenue. And we're going to be doing
5:06:42that per country uh per country and
5:06:44state and we'll kind of do some drill
5:06:46downs. Um and I'll show you how those
5:06:48work. So for just to start out, we're
5:06:50going to take the country right here and
5:06:53you'll see it populate right over here.
5:06:54In fact, um let me zoom in maybe once.
5:06:58Uh yeah, that should be fine. I don't
5:07:00know if I want I might zoom in again in
5:07:02just a little bit. Um so we have our
5:07:04country and and it's just like this.
5:07:06Very very simple. Oops. Um now I'm going
5:07:09to include the state. Now, I'm going to
5:07:10drag this um all the way and I'm going
5:07:13to put it under. You can put it above or
5:07:14you can put it below. I'm going to put
5:07:16it below. Uh it definitely makes the
5:07:18most sense there. Now, when you do that,
5:07:21it it um kind of populates it in an
5:07:25expanded way, but you can collapse this
5:07:28very easily. We're going to go right
5:07:29here. We're going to rightclick. We're
5:07:31going to go go down to expand and
5:07:33collapse, and we're going to collapse
5:07:34the entire field. And so now here are
5:07:37all of our um all of our countries as
5:07:40they were before, but now each of them
5:07:41has this plus sign to the left. And if
5:07:43you click on it now, we can go and we
5:07:45see this state that we that we added to
5:07:47these rows. And what this is going to do
5:07:50is it kind of is like a rollup or it's
5:07:51like a grouping. Um and so if you you
5:07:54know have taken the SQL um tutorial
5:07:57series and you've done uh things with
5:07:59group by, this is very similar to that.
5:08:01Um, and if you've done the uh Tableau
5:08:05tutorial series, it's kind of like a
5:08:06drill down. It's [clears throat] very,
5:08:08very similar. So, you can drill into the
5:08:10information. So, we um can put some
5:08:13values in here. Uh, and what we're what
5:08:16that's going to do is that's going to
5:08:18kind of create some some context to what
5:08:21this what we're grouping by. So, just
5:08:24for um visual purposes, let's add this
5:08:27revenue. So this is the revenue that is
5:08:30bike uh bike sales revenue, right?
5:08:32That's what we're looking at. So this is
5:08:34the sum of the revenue for these bike
5:08:39sales per country. Now if we drop down
5:08:41right here, we can see that in Australia
5:08:45uh New South Wales had uh 92, what is
5:08:48that? 9,23,495.
5:08:52Queensland had 5 million, you know,
5:08:55etc., etc. So now we can break it down.
5:08:57we can't it's we don't just have to look
5:08:59at Australia. We can now drill down even
5:09:01further to the actual state is what
5:09:04they're calling it. Um the actual state
5:09:07within Australia and so it's super super
5:09:09useful and you can do that for every
5:09:10single one. And so we can look at
5:09:13Canada, we can look at France and we can
5:09:15really drill down into uh the revenue
5:09:17for each of these countries as well as
5:09:20the states within them. Now over here,
5:09:23this is not the most uh pretty. Um it
5:09:26just says sum of revenue and then it has
5:09:28some numbers. Not not the most pretty
5:09:30thing I've ever seen. Um really quick,
5:09:33we can go like we can um kind of
5:09:35highlight over these and we can go back
5:09:37to home. You can do it in a couple
5:09:38different ways. We can go to home and
5:09:40we'll type currency. Now it has these
5:09:42two. Z00 zeros at the end. You can get
5:09:45rid of those really easily by going like
5:09:47that. Um already this looks quite a bit
5:09:50better just visually. um especially if
5:09:51you're looking at it in uh you know
5:09:54dollars, you can change the currency um
5:09:56to different currencies if you want to
5:09:59do that. Now, we don't just have to do
5:10:03uh the sum of revenue. We can do a lot
5:10:05of different things. So, let's go to the
5:10:07value field settings. So, we can
5:10:10customize this name. So, we can do um
5:10:14revenue. Oops, be good if I could spell
5:10:17revenue per country.
5:10:20Um, that's fine. That, you know, it's
5:10:22just a placeholder to try to show you.
5:10:24But we don't have to just do that. Um,
5:10:26you know, we could do the count, the
5:10:28average, the max, the min. We can do
5:10:30just about anything we want. Um, but
5:10:33let's keep it the sum right now. Um, and
5:10:38if we want to, we can show this value as
5:10:41different things. So, we percentage the
5:10:44percentage of column total, percentage
5:10:45of row total. Let's do really quick just
5:10:48for demonstration purposes the
5:10:49percentage of grand total. So when we do
5:10:53that we can see that the United States
5:10:55the per revenue per country United
5:10:58States has 32%. Just between these um
5:11:02you know these countries and Australia
5:11:04has the next one. So you know it might
5:11:07be kind of hard to glance at this really
5:11:09quickly to know who has the highest. Um,
5:11:12but what we can do is we can go right
5:11:14here and we can go to sort and we can do
5:11:16largest to smallest. And there we have
5:11:19the United States on top. Now, when you
5:11:21do it right here, it's not sorted
5:11:24largest uh to smallest. You'd have to go
5:11:26and again click sort and do largest to
5:11:29smallest. And so now we can see that
5:11:30California has the has the um, you know,
5:11:33biggest percentage. They're pulling in
5:11:3520% of that 32% of revenue.
5:11:39So, I'm just going to click control-z a
5:11:41few times and get us back to where we
5:11:44just were. Um, and what I want to do is
5:11:47I want to show you a few different
5:11:49things uh pretty quickly. So, we want to
5:11:52pull in this profit and this cost. Uh,
5:11:54and so I'm going to pull in this cost
5:11:56next. And then I'm going to pull in this
5:11:59profit. Again, uh, I'm going to change
5:12:03the currency on this.
5:12:06And I'm not going to change the names um
5:12:07right now, but you absolutely can do
5:12:11that. Now, the revenue is the how much
5:12:14is actually being sold. So, you know,
5:12:16for the United States, it was 27
5:12:19million. Now, the cost is how much did
5:12:22it cost to manufacture or or store um or
5:12:26distribute all of these products. So,
5:12:28that was 16 million. And the profit is
5:12:30actually how much money is being made at
5:12:33the end of the day after um you know all
5:12:36their costs after all their employee
5:12:37costs after everything they're still
5:12:39making the United States is still making
5:12:41$11 million.
5:12:43Now you might look at this and you might
5:12:45say well you know I can kind of glance
5:12:47at it and say know that this profit is
5:12:49correct based off these two numbers. Um
5:12:52but we can do a calculated field. Um,
5:12:55and if you remember what calculated
5:12:57fields are, that's something from
5:12:58Tableau. Very uh basically the exact
5:13:01same thing. And so we can create an
5:13:03additional column right here that is a
5:13:05calculated field that can add and
5:13:06subtract these things to make sure that
5:13:08our numbers are adding up correctly. So
5:13:11let's do that really quickly. Uh let's
5:13:13go to pivot table analyze. We're going
5:13:16to go over to fields, items, and sets
5:13:19and go to calculated field. Now we can
5:13:21name this anything. Um, and I'm just
5:13:24going to for demo purposes, I'm going to
5:13:26say, um, oops, calculated
5:13:30field demo. Uh, I'm sure yours will be
5:13:34different. Now, um, if you want to, you
5:13:37can go in here and this is the formula.
5:13:38It's almost like, um, you know, we
5:13:40haven't looked at formulas. This is our
5:13:42first tutorial. But, you know, when we
5:13:43look at formulas, it's basically the
5:13:45same thing as writing an if inside of a
5:13:47cell, but here it gives us kind of this
5:13:50um open text to do how we uh do what we
5:13:53want with it. Now, what we're going to
5:13:55do is we're going to do revenue. I'm
5:13:58going to insert that. I'm going to get
5:14:00rid of this. I'm going to do revenue.
5:14:04And so, that's the the the very large
5:14:06number. And then we're going to
5:14:08subtract.
5:14:09And we're going to subtract our cost.
5:14:12We're going to insert that. And let's do
5:14:15this. And click okay. So this is our
5:14:18calculated field demo column that we
5:14:21just created. And as you can see, it
5:14:23matches our uh sum of profit column
5:14:25exactly. And that's exactly what we want
5:14:27to see. We want to kind of check to make
5:14:29sure that this revenue and cost uh
5:14:32fields are generating the correct
5:14:34profit. And sometimes those are off. And
5:14:36so it's really good to kind of check
5:14:37those and have that additional column.
5:14:39Um you probably wouldn't have this if
5:14:41you were um you know going to submit
5:14:43this to somebody. Uh just so you know
5:14:46now that this is an actual column. You
5:14:48can't go here and do something like cut
5:14:50or and paste it over here. You know
5:14:53that's not it won't let you do that.
5:14:55What it is is now an actual um column.
5:14:58And so we can go and remove that and we
5:15:00can add it back at any moment. So, if we
5:15:02want to go back and add that um oops,
5:15:05add that down here, we can do that
5:15:07because we've created that column. It's
5:15:09now permanently there unless we go and
5:15:11delete all of that data. Uh and so we
5:15:14can just click this check mark and it
5:15:16will get rid of it for us. All right.
5:15:17Now, the last thing that we have not
5:15:19used down here is the filters. Now, the
5:15:22filters is exactly what it sounds like.
5:15:24It's going to allow you to filter on
5:15:26certain things. Um, but probably not
5:15:29things that you already have included in
5:15:31your pivot table. So, if you add
5:15:33something like the country down here,
5:15:36um, it's going to kind of expand
5:15:38everything and then if you then go and
5:15:40filter on it, it kind of breaks it down.
5:15:44That's really not what the filter is
5:15:46kind of used for or meant for. Um, for
5:15:49example, right up here we have uh
5:15:52customer gender. Okay, so let's take the
5:15:54customer gender and we'll put it in this
5:15:55filters. Now we can see all of the
5:15:58revenue, all of the cost, all the
5:16:01profit, and we can do that based off of
5:16:03the gender. So we can filter by a
5:16:05gender, not really having to change
5:16:07anything about our pivot table. And so
5:16:09at a super quick glance, we can see that
5:16:12uh the males are the the profit from the
5:16:15males is 16.487 487 million and the
5:16:19profit from the females is 15.733
5:16:23million. So at a super uh basic level at
5:16:25a really quick glance we can see that
5:16:27the men or the males are you know
5:16:30spending a little bit more than the
5:16:32females by about about $700,000.
5:16:35Now let's go ahead and create one more
5:16:37pivot table. Uh we are going to create a
5:16:39pivot table right over here. Let's go
5:16:41back to the sales
5:16:43right here again. control shift right
5:16:46down. It's going to select all of our
5:16:48data and we're click okay. So, one thing
5:16:53that we're going to look at is we're
5:16:54going to use some of this date
5:16:56information right here. So, let's select
5:16:58our country just like we did before. Um,
5:17:01and what we want to do is see, you know,
5:17:03what year were we performing our best?
5:17:06when were we doing our absolute best uh
5:17:08with oops let me go back
5:17:13uh with our sales. So I'm going to
5:17:15select the year and put that in our
5:17:17columns. And so now we have 2011 through
5:17:212016 and we want to look at our revenue.
5:17:25So let's put our revenue right down
5:17:26here. And now we have all of our
5:17:29revenue. Now let's again make this into
5:17:32a currency.
5:17:35Just like that. And super quickly now,
5:17:38we can get a really quick glance at how
5:17:40Australia was doing each year. And we
5:17:43can see that there was a huge uptick in
5:17:45uh 2013 and a huge uptick in 2015. That
5:17:49didn't happen for every single country.
5:17:51Uh did go up uh for most countries, very
5:17:54slightly for some, but we can see on a
5:17:57large scale from um year to year what
5:18:01that's like. And so within just a few
5:18:03minutes, we're able to create some
5:18:04really useful pivot tables that anybody
5:18:06could look at and understand. And that's
5:18:08really the biggest use of these pivot
5:18:10tables is that you can kind of group
5:18:12these things together, show some uh
5:18:14information, data at at kind of a broad
5:18:16larger scale and make it to where
5:18:19anybody who's looking at it can
5:18:20understand it. That is why pivot tables
5:18:22are so useful. And so I hope that this
5:18:24video was helpful. I hope that I was
5:18:26able to walk through it and help you
5:18:27better understand how pivot tables work
5:18:29and how you can use them when you are
5:18:31working within Excel. Thank you guys so
5:18:33much for watching. I really appreciate
5:18:35it. If you like this video, be sure to
5:18:36like and subscribe below and I'll see
5:18:38you in the next video.
5:18:51What's going on everybody? Today we're
5:18:53going to be looking at formulas in
5:18:54Excel.
5:18:58[music]
5:19:01Now, I know what you're thinking.
5:19:02There's absolutely no way that you're
5:19:03going to be able to show us every single
5:19:05formula in Excel. And you're absolutely
5:19:07right. But I am going to show you some
5:19:09of my favorites and the ones that I
5:19:10found the most useful. And then you can
5:19:12go ahead and practice those and try
5:19:14those out. And if there are ones that
5:19:16you really want me to do and you think
5:19:17that I missed, put it in the comments
5:19:20below and I will see those and I'll try
5:19:21to make a list of those and make another
5:19:24video on formulas and include all of
5:19:26those as well. And now before we jump
5:19:28into the actual tutorial, I want to give
5:19:29a huge shout out to the sponsor of this
5:19:31series and that is Udemy. You guys
5:19:33already know if you've watched any of my
5:19:35videos that I absolutely love Udemy. I
5:19:37mean honestly, they were the ones who
5:19:39got me started and were able to give me
5:19:40affordable courses for me to get started
5:19:42as a data analyst. I learned SQL and
5:19:45Excel and Python all through Udemy
5:19:47courses. And so if you are looking for a
5:19:49platform to take a course, I absolutely
5:19:51recommend you look at Udemy. They have
5:19:53fantastic sales going on right now,
5:19:54especially during the holiday season in
5:19:56this new year. And so if you're looking
5:19:58to take a full-fledged Excel course, I
5:20:00have some of my favorites in the
5:20:01description below. And now without
5:20:03further ado, let's jump onto my screen
5:20:04and get started with the tutorial. All
5:20:06right. Now before we start, I want to
5:20:07say that this is not like every other
5:20:09tutorial that I have created. This one
5:20:11is very streamlined. Okay. So, I already
5:20:14know exactly what I'm going to do.
5:20:15There's not going to be much messing
5:20:16around. I've left little notes here and
5:20:19there. Um, and I'm going to try to get
5:20:21through it because there's a lot of them
5:20:22to get through. Um, so all these ones at
5:20:24the bottom. Now, these are ones that I
5:20:26use a lot that I think are useful.
5:20:28Again, if you know other ones that you
5:20:30use a lot that think that I should be
5:20:32using, which I know there are ones that
5:20:33I left out of here, you know, put it in
5:20:35the comments. Um, I'll see the ones that
5:20:37people are liking and I will I will
5:20:39create more videos on these because I
5:20:40know there are so many. I also will save
5:20:43this um Excel in uh on the GitHub so you
5:20:47can go and download it. It'll be exactly
5:20:48what you're looking at right now. I
5:20:50highly recommend trying these formulas
5:20:52out for yourself so you can get a feel
5:20:54for how they work and how they're
5:20:55actually used and you can mess around
5:20:56with it yourself. So, um as you can see
5:20:59at the bottom, we're going to start with
5:21:01max min and then we're going to go on to
5:21:03some more I think a little bit more uh
5:21:06difficult things. Um and all these
5:21:08things are super useful. I'll try to
5:21:09talk about how you can actually use it
5:21:11as we go through it. Some are super
5:21:13self-explanatory, but some may not be.
5:21:16So, this one I think is super
5:21:18self-explanatory, but again, one that
5:21:20you're going to use all the time. Um,
5:21:22and so, uh, what we can do is we can say
5:21:24equal, and that's how you kind of start
5:21:26off saying this is going to be a formula
5:21:28in this cell. Equal means, uh, I am now
5:21:31creating a formula. And we're going to
5:21:32say m a x. And I'll hit tab. And so,
5:21:36it'll kind of populate it. And right
5:21:38here, if you've never seen a formula
5:21:39before, it'll kind of give you what the
5:21:41inputs need to be. So, it's going to say
5:21:43max of number one, number two, etc.,
5:21:45etc. What we're going to do is we're
5:21:47going to give a range. So, we're going
5:21:48to go from here down to here. You don't
5:21:51have to close the parentheses, but you
5:21:52can. I'm going to. And then you hit
5:21:54enter. And so, for this date, it's going
5:21:57to give us the max date. Now, these are
5:21:59um the start dates for these people
5:22:02right here. And so if we just kind of
5:22:04glance through here, we can see that
5:22:062013 was the last year and this one is
5:22:09actually the latest in that year. And so
5:22:11it gave us the correct one. The min is
5:22:13going to do the exact opposite. It's
5:22:16going to give us the smallest. And so
5:22:19we'll give it the same range. We'll
5:22:20close the parenthesis. And it's going to
5:22:22say December 7th of 1995. And we can see
5:22:26that that is correct. So Michael Scott
5:22:29started in 1995, the earliest of all the
5:22:31employees. Um, and you can do the exact
5:22:33same thing for really any of these
5:22:36columns. Um, we can see who the who's
5:22:38making the most money or at least what
5:22:40the highest salary is. U, so we'll do
5:22:43max. And then we'll do the salary range.
5:22:46And so this is this one again. Uh,
5:22:49Whoops. What did I do? Oh, I did the
5:22:51wrong range, didn't I? No, I didn't do
5:22:54the wrong range. It's just There it
5:22:57goes. uh this column was a date range or
5:23:01a date column for whatever reason. So
5:23:02let me get rid of that. Uh and then we
5:23:05can do equals min and we'll do again
5:23:08we'll do the salary. And at a quick
5:23:11glance we can see that Beasley is making
5:23:13the least and 65,000 is Michael Scott
5:23:17who's making uh that. So super simple.
5:23:21It shows the max. It shows the min. You
5:23:23can select a range. There you go. Let's
5:23:25move on to if and ifs. Now, if is um I
5:23:30think pretty straightforward. So, all
5:23:32you're going to do is you're going to
5:23:33say if this then that. Um ifs is a
5:23:38little bit different. So, ifs is you can
5:23:40you can put multiple conditions. And as
5:23:42we're writing it, I'll show you kind of
5:23:43what it the conditions that need to be
5:23:45met. All right. So, we're going to click
5:23:47right here. We're going to say equal.
5:23:49We're going to do if hit tab. And we
5:23:51need a logical test. Uh, and so we're
5:23:53going to give it a range or or or
5:23:55something. We're going to say if it's
5:23:56equal, greater to um something like
5:23:58that. Then we're going to say if the
5:24:00value is true, what's the what is going
5:24:01to be the output? Or if the value is
5:24:03false, what's going to be the output? So
5:24:05let's do this right here.
5:24:09We'll do this age range. And so if they
5:24:12are greater than let's say let's do 30.
5:24:17If they're greater than 30, we're going
5:24:19to do a comma. And so if the value is
5:24:21true, what what should be the output? If
5:24:24they're greater than 30, we're going to
5:24:25call them old. And then if it is false,
5:24:29so if they're younger than 30, what
5:24:31should it say? And we're going to say
5:24:34young.
5:24:36And we'll close the parenthesis. And
5:24:38there you go. So if they're over 30,
5:24:42then they are going to have young. Or if
5:24:44they're younger than 30, they're going
5:24:45to have young. Now, this is something
5:24:48where you need to specify if you want 30
5:24:50and over or over 30. We chose over 30.
5:24:54So, 30 is not included in that. Um, so
5:24:57they're going to be young.
5:24:59Now, uh, let's get We don't actually
5:25:01need two of these. That's pretty
5:25:03self-explanatory. The ifs is a little
5:25:05bit different, right? You can have
5:25:06multiple conditions. So, let's open that
5:25:08up real quick. So, ifs. And now we have
5:25:12a logical test value. If uh that's true
5:25:16then you can do logical test two value
5:25:18if that's true. Um so you can have
5:25:21multiple multiple multiple things. Now
5:25:23this one is a little bit different. In
5:25:25this one oops let me get out of this. In
5:25:29this one you had a value of true a value
5:25:31of false. Ifs does not have that. Ifs is
5:25:35going to give you um different ranges
5:25:38and different specific conditions. And
5:25:41you can't say if this one's false.
5:25:43you're just going to have multiple
5:25:44conditions. So, let's do equals and ifs
5:25:48tab and we'll do our first logical test.
5:25:50So, let's do um
5:25:54if the salesman
5:25:56or if that equals to salesman,
5:26:01we're going to say we're going to
5:26:04respond with sales.
5:26:07So, that's if the value is true. That's
5:26:09what we want the output to be. Now we're
5:26:12going to go on to our logical test two.
5:26:14So you're going to see this pattern,
5:26:16right? If this is our conditional or
5:26:18logical test. So if this is true, this
5:26:21is what's going to be returned. So
5:26:23you'll notice that's just a a pretty
5:26:25simple pattern. We can just do random
5:26:26things. So if it's equal to sales, um,
5:26:30and we'll just do the same one. If that
5:26:33is equal to
5:26:36say HR, we can say fire immediately.
5:26:42And now we're going to say
5:26:45if it's equal to
5:26:51regional
5:26:54manager
5:26:56and we say give Christmas bonus and
5:27:02we'll close the parenthesis and let's
5:27:03see what we get. So, as [clears throat]
5:27:06you can see, there's no default value
5:27:08for true or false. Like like this one,
5:27:11there was a logical test and if it was
5:27:13true, there was a value and if it was
5:27:15false, there was a value. So, for every
5:27:16single one, you'll get a value. For this
5:27:18one, that's not exactly going to happen.
5:27:20As you can see, there are these NAS.
5:27:23Now, when that happens, it just means
5:27:25nothing met that condition. So, we never
5:27:27said anything about supplier relations.
5:27:29We never said anything about
5:27:30accountants. But if it was part of that
5:27:33ifs statement then it got something. Um
5:27:35and so that is how the ifs works. Now
5:27:39let's move on to length. Uh this is
5:27:42exactly what we're going to do. But you
5:27:44know some of the uses for this u for the
5:27:46length I've used it for a lot of
5:27:48different things. Um one thing that I've
5:27:50used it for in the past and you know max
5:27:52and ifs you know you can use it for
5:27:54almost anything. Length is there's a lot
5:27:57of different use cases. one, I used to
5:27:59work with a lot of um customer data or
5:28:02or patient data. They had like social
5:28:03security numbers and if you know there
5:28:05was bad social security numbers, we
5:28:07didn't want to include that. And so we
5:28:09do like the length of that. And if a
5:28:11social security number was let's say 10
5:28:13numbers or 11 numbers where it should
5:28:15only be nine or or you know, however
5:28:18many they are, I think it's nine. Then
5:28:20we know that that social security number
5:28:21is incorrect. And then we can get rid of
5:28:23that or discard it from our results.
5:28:25That's just an example, right? Um, so
5:28:27for this, oops, why' I do that? I did
5:28:30control Z to undo that if you didn't
5:28:32know how to do that. Uh, so we're going
5:28:33to do equals leen, which is length. Um,
5:28:37and again, if you didn't see that, it
5:28:39returns the number of characters in a
5:28:41text string. So, let's go right here and
5:28:45let's go to uh let's go to their last
5:28:48name and we'll give it a range. So, it's
5:28:51going to tell us how many characters are
5:28:54in that string. So for Halpert it's
5:28:56seven characters. For Fenderson it's 10
5:29:00characters. And we're able to see a
5:29:02length. And so again there are a lot of
5:29:03different use cases for this. Uh the
5:29:05social security number was one. Another
5:29:07one is phone numbers. Right? If you look
5:29:09at the length of the phone numbers and
5:29:10there's uh ones that are like 12 numbers
5:29:13long. You know those might not be ones
5:29:15that are accurate and you need to go
5:29:16look at them and see if you want to
5:29:18include them in your results or your
5:29:19output. So that is how length is done.
5:29:22Let's move right over to the left and
5:29:24right. Um I I might be going a little
5:29:28fast, but uh you know, I'm keeping it
5:29:30I'm keeping it live. I'm keeping us on
5:29:31our feet. Uh so let's keep going. Left
5:29:34and right um are kind of like
5:29:37substrings. If you've taken the the SQL
5:29:40um tutorial series that I've done, uh
5:29:43substrings are where you can choose a
5:29:44certain part of the text string and you
5:29:47can extract data from that. Um and it
5:29:50usually have to reference a certain
5:29:51number. So, a certain amount of
5:29:53characters. That's the exact same thing,
5:29:55except uh uh unfortunately there's no
5:29:57substring. There's substitute, but
5:29:59there's no substring. Left and right is
5:30:01really the closest thing that we have.
5:30:03So, let's kind of take a look real quick
5:30:05and see what we can do. So, we're going
5:30:08to do left, and it's going to say
5:30:10returns the specified number of
5:30:11characters from the start of a text
5:30:13string. So, we're starting from the very
5:30:14far left, and we need to choose our
5:30:17text, and then choose the number of
5:30:19characters that we're going to be
5:30:20looking over.
5:30:22So let's go over here and let's just
5:30:24choose, you know, start simple. Uh we'll
5:30:27get a little bit more advanced. So we
5:30:29have uh this is our text range. So these
5:30:31are the the the ones that we want to
5:30:33look at. And then how many characters do
5:30:34we want to look forward um and we'll
5:30:36just choose three as an example. And so
5:30:39you can see that it takes the first
5:30:41three characters from every single um
5:30:44thing. Now you can also do this with
5:30:45numbers. It doesn't just have to be um
5:30:48you know name with with actual words or
5:30:51letters. You can do the exact same
5:30:52thing. So you can say write
5:30:56um and we're going to choose our our
5:30:57string. Uh and let's do this one. So you
5:31:00know all of them start with 100 um and
5:31:02we'll just say we want to take the last
5:31:04one. So this one is going to start from
5:31:07the very far right and go over one
5:31:09character. So right here you can see
5:31:11this is our range and I just chose one.
5:31:13So starting from the very far right, we
5:31:15go over one character and that's what we
5:31:16take. And so that can definitely be
5:31:19useful. Another one that you can do and
5:31:21this one is one that I have used so many
5:31:23times. I mean honestly countless times
5:31:25in in actually using this in my job. Uh
5:31:28so we're going to go from the right and
5:31:29we're going to look at a date. So, you
5:31:32know, sometimes you have these date
5:31:34structures, month, month, day, day,
5:31:36year, year, year or year. Um, you know,
5:31:38day, month, year, all these different
5:31:40and sometimes you just want to extract
5:31:43either the month or the year or or
5:31:45something like that, the day. And so, we
5:31:47want to come in here and we're just
5:31:48going to extract the Oops, I wanted to
5:31:50make that a range. We want to extract
5:31:52the year of the start dates. So, we're
5:31:55going to do that. And then we're going
5:31:56to go over four because we want to take
5:31:58the first four characters from the right
5:32:00to give us the entire year. So let's do
5:32:03that. And now we can see exactly the
5:32:05year. And this can be just super super
5:32:07useful. This is again one that I've used
5:32:10a lot. And so that is one that you might
5:32:11want to remember in case you're ever
5:32:12doing analysis on, you know, start and
5:32:14end dates or or anything with um date
5:32:16data. Uh again, one that I highly
5:32:19recommend remembering. Let's go over to
5:32:22date to text. I actually probably should
5:32:24have included that. um before because I
5:32:27actually used it in this one. Um if you
5:32:30notice right here, this is a text. So in
5:32:32in this one we just did that was a text.
5:32:35You can't do this right on um start and
5:32:38end dates when it's a date uh format.
5:32:41And let me show you. So this is a date.
5:32:44Now if I do equals and you know we just
5:32:48did this uh let's do on the end date and
5:32:52I mean I'll do the whole range. Give me
5:32:54a second and we'll do four.
5:32:57It's giving us completely random
5:32:58numbers. Why is that? Because underneath
5:33:00the date range there are um numbers,
5:33:04right? So if I go right here and I make
5:33:07this
5:33:08general, it's going to have a numbers.
5:33:10And look, these are the first four
5:33:12characters from the right. And so it's
5:33:14doing what it's supposed to do, but uh
5:33:16it's not doing what we actually want.
5:33:17And that's the issue. So how can we
5:33:20convert this? Now there are a ton of
5:33:22different ways. Um, but the quickest,
5:33:25probably the easiest besides actually
5:33:28writing writing it out like this, like
5:33:3011-2-201,
5:33:33which then converts it to a date format.
5:33:36Um, but what you can do, you know, just
5:33:38so you know, you can create it as a
5:33:40text. You can do 11-2-201.
5:33:45And now it will stay a text string. And
5:33:48as you can tell, these are a little bit
5:33:50different because this one is uh
5:33:51formatted or situated on the right and
5:33:53this one's on the left. That's how you
5:33:54can tell the difference. Now, if you
5:33:57don't want to do it by hand uh
5:33:59completely manually and waste hours of
5:34:01your time, you can do it in a very
5:34:04simple way. So, we're going to do uh
5:34:07text. So, this is the exact um formula
5:34:10that we're going to use. So, let's get
5:34:11rid of that one. Oops. There we go. So,
5:34:14we're going to do equals. We're going to
5:34:17do uh oops text. It says converts a
5:34:20value to text in a specific number
5:34:22format. So for a date format, we can
5:34:26choose a date format and then it'll
5:34:28convert it to a text for us, which saves
5:34:31so much time, I promise you. Uh let's do
5:34:34all of these just like we did. And then
5:34:36we need to tell it what the format is.
5:34:39If we don't, if we tell it something
5:34:41incorrect, it's going to give us a
5:34:43completely terrible output or just give
5:34:44us an error altogether. So this is a day
5:34:47day month year year format and that is
5:34:51what we're going to do. So we're going
5:34:52to dd/mm/y
5:34:55y and close that up. And there you go.
5:35:00And now we will because it's in a
5:35:02formula what we need to do is
5:35:06copy this
5:35:09and paste it right over here. And now
5:35:11you can see that is a general. This is
5:35:14something that we can use as a string.
5:35:16And let's just check it just to make
5:35:18sure. So we're going to do write, we're
5:35:20gonna do this one. Let's do all of them.
5:35:23And we'll do four. And there you go. So
5:35:27now it works. That is what we are
5:35:29looking for. Um, and you can do that.
5:35:31Imagine doing that with millions of rows
5:35:33or, you know, let's say 10,000 rows.
5:35:36It's going to be a breeze, right? It's
5:35:38going to take you 2 minutes or a minute
5:35:40to do everything that you want to do
5:35:42instead of having to just do a bunch of
5:35:44mess to convert it to a string, which I
5:35:46promise you I've done and it just takes
5:35:48forever. It's it's terrible. So, that is
5:35:51uh date to text. Super helpful formula.
5:35:54Let's go over to trim. Now, I I
5:35:57purposefully messed up this column. Now,
5:36:00why do I did I mess it up like this?
5:36:03Because when you're working with real
5:36:04data, you're going to get data like
5:36:05this. it it it's messy, it's dirty, it
5:36:08just has random spaces at the end for no
5:36:12reason. Um because sometimes you're
5:36:15going to be working with um data that is
5:36:18inputed by a user. It's not like a drop-
5:36:21down option. So imagine somebody's
5:36:22typing this in, they accidentally put a
5:36:24space or they accidentally put an enter
5:36:26or something and then they submit it and
5:36:28this is how it's going to look in the
5:36:29database. Um, and if you're a data
5:36:32engineer or you know you're working with
5:36:33the raw data, if they don't clean that
5:36:35up, then you're going to be working with
5:36:37that that dirty data. And I I guarantee
5:36:39you if you're working as a data analyst,
5:36:41you're going to see stuff like this. Not
5:36:43with maybe a last name, but all sorts of
5:36:45data. So, we're going to go right here.
5:36:47We're going to say equals trim. Do open
5:36:50parenthesis. Actually, this says removes
5:36:52all spaces from a text string except for
5:36:53a single space between words. So like
5:36:57you know if it said help space uh or gym
5:37:01space helper it won't take the space in
5:37:03between there because it it kind of
5:37:04understands that the in normal language
5:37:07space is supposed to be there. So it
5:37:08won't do that. Um but we'll take that.
5:37:11We'll give it this range.
5:37:14Close that up. And there you go. Now it
5:37:16is nice and clean. Much more usable. Now
5:37:19let's look at concatenate. one that I
5:37:22have used just way way way too many
5:37:26times. Um, and something that I've used
5:37:29concatenate for, and you'll see this one
5:37:31in a lot of demonstrations for a good
5:37:33reason, is because a lot of people use
5:37:35it for this. Um, so what you can do is
5:37:39you can say equals um, and well, let me
5:37:42tell you what concatenate does real
5:37:44quick.
5:37:45So, what concatenate does, oops, I'm
5:37:48totally messing up here. Um, but it
5:37:51joins two or more text strings into one
5:37:53string. It basically joins things
5:37:55together and adds them together. So,
5:37:58let's do concatenate. And we're going to
5:38:00add this first and last name. Again, one
5:38:02that gets used all the time, but that's
5:38:04because um it really is useful. So, you
5:38:07can do this. And you can say now I want
5:38:10to include this. So, concatenating this
5:38:12and this. And let's take a look. So, it
5:38:15says Jim Halbert, but it's all
5:38:16connected. And that's typically not how
5:38:19people write their names. So, what we
5:38:21can do is we can go back in here and we
5:38:23can do what my demonstration up here
5:38:25already tells us to do, which is we're
5:38:27just going to add another thing in here.
5:38:29And if we add two parentheses, we can
5:38:31include anything in here. We can include
5:38:33a dash, we can include an exclamation
5:38:35point, or we can just include a space.
5:38:38So, let's just include a space really
5:38:40quick. And just like that, it works
5:38:44perfectly. And so, now we have the full
5:38:45name. Now, something that you could use
5:38:48it for is something like generating uh
5:38:50an email. This is something that you
5:38:53absolutely could do. Um, and it's, you
5:38:56know, pretty simple. So, I'm going to do
5:38:58it like this. I'm going to say, oops,
5:39:01what' I do? I'm gonna say um dot and
5:39:07then at the end I'm going to say at oops
5:39:11comma quotation atgmail.com.
5:39:17And now I've created emails for all of
5:39:19these people. So just something that you
5:39:22can do with this um and something that
5:39:24it it absolutely is used for and you'll
5:39:26see that demonstration almost everywhere
5:39:28because honestly it gets used a lot um
5:39:30by data analysts. And so uh you know
5:39:33just a good one to know understanding
5:39:35how that that concatenation works. Um
5:39:38let's go over to the next one. So
5:39:40[clears throat] we are going to do
5:39:42substitute. Now substitutees really
5:39:44interesting. Um there are different ways
5:39:46you can do it. I'm going to show it to
5:39:48you on these dates real quick. Uh that's
5:39:51what we're going to look at. So changing
5:39:53a date format, changing how uh what it's
5:39:56supposed to look like is absolutely
5:39:58something that happens all the time. And
5:40:00um you know sometimes you'll even get it
5:40:02like this
5:40:04where it'll look like it'll be messy.
5:40:06It'll be different a different um I
5:40:09guess format. So this one has all the
5:40:12other ones have um slashes where these
5:40:15ones have dashes. And you know what you
5:40:19can do is if you want to well let me
5:40:23actually go with the no instances real
5:40:24quick because this one is uh actually
5:40:26makes the most sense. Um, so we'll do
5:40:29equals and we're going to say
5:40:31substitute.
5:40:33And oops, and let me say substitute
5:40:34replaces existing text with new text in
5:40:38a text string. So if we do an open
5:40:41parenthesis, it says we take the text,
5:40:43we have the old text, we have the new
5:40:45text, and then we have how what instance
5:40:48or how many times uh or or or what
5:40:51instance are we looking at it? And I'll
5:40:52explain that in a little bit.
5:40:55So the text that we're going to be
5:40:56looking at is this one right here. So
5:40:58let's take this range. And the old is
5:41:02we're going to take this dash. And so
5:41:06let's take the dash.
5:41:08And then what do we want to replace it
5:41:10with? We want to replace it with this
5:41:12slash right here. I think it's a forward
5:41:14slash. Isn't that what it's called? Is
5:41:15that called a forward slash? Am I crazy?
5:41:17Um and we're not going to put an
5:41:19instance. Notice that that's in a
5:41:20bracket. That means it's optional. We're
5:41:22going to do none of that. Um, and what
5:41:24it's going to do is it's going to fix
5:41:26this. So, this one is now in the correct
5:41:28format that we want. Uh, and that's
5:41:31fantastic. That's, you know, that's what
5:41:33we tried to accomplish given what we
5:41:35had. Now, let's fix that. If we want to
5:41:37do the exact same thing, uh, we can say,
5:41:40uh, what are we doing? Substitute. We
5:41:42can do substitute. We can do open
5:41:44parentheses. We'll give the range. And
5:41:47now, let's say we want to change all of
5:41:49them to a different format. So instead
5:41:51of the um forward slash, I'm going to
5:41:54keep calling it that if that's correct,
5:41:57we want to give it a dash. And so then
5:41:59we close that. And now all of them are
5:42:01in this new format. So it it's able to
5:42:03substitute a specific value for a new
5:42:06value. And if you don't include an
5:42:08instance, then it'll do it to every
5:42:11single one in there. So, let's go over
5:42:15here and we're going to actually use the
5:42:17the um the uh the instance num and I'll
5:42:21show you what that does. Uh and so,
5:42:24really quick, we'll do the exact same
5:42:25thing that we just did. We'll do the
5:42:29forward slash. And we want to replace it
5:42:32with this one again, this dash, but we
5:42:36only want to do it on the first instance
5:42:38of that forward slash. And so as you can
5:42:42see all the ones that um all the ones
5:42:45that were replaced are the very first
5:42:46instance whereas the second instance
5:42:49which is the second time it appears in
5:42:51this string does not get touched.
5:42:54So if we take this
5:42:56and we put it right over here and we
5:43:00move it to two,
5:43:02it's kind of the opposite. So the first
5:43:04one wasn't touched, the second one was.
5:43:06So, we're choosing which instance or
5:43:08which time it shows up in that string
5:43:10and then it replaces it. If you do not
5:43:12choose an instance, it chooses all of
5:43:15them. So, this can be super useful if
5:43:17you want to do like a bulk replace um
5:43:20but you only want to do it on a specific
5:43:22column um and you just want to use a
5:43:23formula really quick, right? Um and so
5:43:25you can use this in a lot of different
5:43:27ways. So, that's how you're able to
5:43:28actually do it with the first instance,
5:43:30the second instance, and if you don't
5:43:32include an instance at all. Let's go
5:43:34over to the sum. Uh, this is one I think
5:43:37everyone knows how to use, but I want to
5:43:40show you two other ones um as well. So,
5:43:44let's go to the sum and we're just going
5:43:45to do equals the sum. And I hope you
5:43:47know what this is. Well, not hope. I if
5:43:49you don't know what this is, it just
5:43:50adds up all the numbers in a range. So,
5:43:53we're going to add. Sum means add. So,
5:43:55we're going to take this and it's going
5:43:56to give us the uh what all these
5:43:58salaries are together. So, super super
5:44:01simple. Sum is one of probably the most
5:44:03basic formulas that you can do. Um, sum
5:44:06if is a little bit different. You can
5:44:10add an if statement, which we learned
5:44:13right back here. You can add an if
5:44:15statement and then add it if it meets a
5:44:18certain criteria. All right, so we're
5:44:21going to do equals sum if and then
5:44:24you're going to need to give a range and
5:44:26criteria and you can include a sum range
5:44:28if you would like.
5:44:30So, we're going to do the salary again.
5:44:33We're going to do a comma. And now,
5:44:34here's our criteria. Let's do if they
5:44:37have greater than 50,000 for their
5:44:41salary and close that parenthesis. So,
5:44:44now it's only going to add up if their
5:44:47salary is greater than 50,000. Now, his
5:44:50is 50,000 exactly, so that won't count,
5:44:52but we have 63 and 65,000, which does
5:44:55equal 128,000. So it it just gives a
5:44:59specific criteria or an if statement
5:45:02then it does the addition. Uh so super
5:45:04useful in that one. So that is how you
5:45:06do a sum if and sum ifs is kind of the
5:45:08same thing as we did back here. There's
5:45:11the if and the ifs. So the ifs is going
5:45:13to be if it has it meets multiple
5:45:16conditions. So let's take a look at that
5:45:18one. So let's do um equals some ifs. Now
5:45:23uh oops. Now [clears throat] the syntax
5:45:26for this one is going to be a little bit
5:45:28different. You'll see that in just a
5:45:29second. But this adds the cells
5:45:32specified by a given set of conditions
5:45:34or criteria. So let's do no paren open
5:45:37parenthesy. We'll give the sum range. So
5:45:39let's do um the same one as before. Then
5:45:43we have our criteria range. So what are
5:45:46we looking at? What's um this is the
5:45:48area that's going to be added after all
5:45:50these if statements are done. Right? So,
5:45:53[clears throat] we have to initially set
5:45:54that. Now, we're going to say, okay,
5:45:56what criteria are we basing this off of?
5:45:59So, let's put a comma. And we're going
5:46:00to base it off of let's do this one.
5:46:03We'll say um if the uh gender, so we'll
5:46:08do comma if that's female.
5:46:12Oops. If that's female. And then we'll
5:46:15give another one. We can say if they're
5:46:17female and
5:46:19let's say they are greater than oops
5:46:22greater than 30 and we'll close that up
5:46:26and it's going to give us 88,000. So
5:46:28female female uh there's one two right
5:46:32here. So it's going to be this one and
5:46:34this one that equals 88,000. So that's
5:46:38how that works. you're able to
5:46:39incorporate several different conditions
5:46:43into uh the sum formula. So again, I
5:46:46know this one's super simple, but you
5:46:48you can use it in a much more complex
5:46:50way if you use the sum if and the sum
5:46:52ifs. Um almost the exact same thing for
5:46:56this count. I'm not going to go super in
5:46:58depth into this one. Um, I'll just kind
5:47:01of show you because count is um count
5:47:06and sum are kind of on the same level of
5:47:09difficulty. They're both pretty
5:47:10beginner. This is just going to give you
5:47:12a count of how many cells um are there.
5:47:16So, let's give this range. Um, and so
5:47:18it's not going to add it. It's just
5:47:20going to give us a count. So, if we do
5:47:21right here and scroll over them, like
5:47:23highlight them, this countdown here,
5:47:25oops, this countdown here is nine. And
5:47:27so it's going to give us that count. But
5:47:31we can do a count with conditions
5:47:33exactly how we did it in the sum.
5:47:36So if we do count if, oops, I did not
5:47:38spell that right. If we do count if,
5:47:41we're going to give a range and a
5:47:42criteria, exact same as we did before.
5:47:45Uh, so let's do this. I mean, you can do
5:47:48this on basically any of these. It
5:47:49doesn't really for this demonstration,
5:47:51it doesn't really matter. um but we'll
5:47:53say if their salary is greater than
5:47:5645,000. So how many people this is going
5:47:59to give us how many people have a salary
5:48:01over 45,000 and that's five. So before
5:48:04in the sum if we did that um we did
5:48:0750,000 it adds everything together. The
5:48:10count is just going to count the amount
5:48:12of cells that meet that criteria. And
5:48:15again count ifs
5:48:18uh we're going to have a criteria range
5:48:20and then we will specify what if
5:48:23statements we want to be uh to occur in
5:48:26order to count those cells. So let's do
5:48:30we want you know we want to count let's
5:48:32it can be any range or it can be any of
5:48:34these. We'll do the ID this time. And
5:48:37now we can say [clears throat] you know
5:48:39we want it to be as our criteria one we
5:48:43can say we want it to be greater than we
5:48:45want their ID to be greater than 1005
5:48:50and let's say we want them to be
5:48:56male.
5:48:58So they have an ID over a certain um a
5:49:01certain range and then they are a male.
5:49:04So there's only three people that meet
5:49:06that criteria. And so it'll be um
5:49:09Michael, Stanley, and Kevin. Those are
5:49:11our three people. And so it gives us a
5:49:12count. Very useful to give quick numbers
5:49:15like this. Something I I genuinely use a
5:49:18lot. Um and I know I've said that a lot
5:49:21during this tutorial, but that's because
5:49:23everything I'm showing you are things
5:49:25that I've used a lot. So I don't feel
5:49:26like um you know, I'm speaking out of
5:49:28turn here. Let's look at this one. This
5:49:30one is very
5:49:32um has some specific use cases. Um
5:49:36notice that this is a text right now. Um
5:49:39if you do it when it is uh in a date
5:49:42format, it actually will not work. I
5:49:44mean I can you can test it out yourself.
5:49:46You just got to trust me. It's not going
5:49:47to work. So what this does is it's going
5:49:51to give you the range from this day to
5:49:53this day. That's what it's going to do.
5:49:55So let's do uh oops days. is gonna we
5:49:59want to choose our end date. So this is
5:50:01our end date. That's kind of backward
5:50:03from what you think. End date to start
5:50:04date. You think start date to end date.
5:50:06So you have to start with this one and
5:50:08then we're going to choose the start
5:50:09date. And now it's going to tell us how
5:50:12many um how many uh days was it from
5:50:18here to here. And this one it's 5,56.
5:50:22So network days is extremely similar
5:50:25except it takes out holidays and it
5:50:27takes out weekends. And you can see how
5:50:29many working days has this person uh how
5:50:33many working days or network days has
5:50:35this person worked not including you
5:50:37know weekends and holidays? Have they
5:50:39actually worked since their start date
5:50:41and their end date. So let's do network
5:50:44days. And we mean our start date, our
5:50:46end date. And you can specify extra
5:50:49holidays if you'd like, but there are a
5:50:51already standard set holidays in there
5:50:54that it takes out. Um, so you know, if
5:50:57you want to do that, you can. So, we're
5:50:59going to do the start date. Again, this
5:51:01one's different. This one says start
5:51:02date, end date. And then we're going to
5:51:04give the end date.
5:51:06And if you notice,
5:51:08they are going to be different numbers.
5:51:10Dramatically lower because it's taking
5:51:12out weekends and holidays. So this is
5:51:14how many days uh calendar days they've
5:51:17worked and this is how many days they've
5:51:18actually been in the office and worked.
5:51:21And that is it. Um again there are so
5:51:25many formulas I mean literally hundreds
5:51:27of formulas that you can utilize and use
5:51:31and are out there for you to try out
5:51:34yourself. If there are specific ones
5:51:36that I did not cover in this video,
5:51:39please put it in the comments below so
5:51:41that I can, you know, show you how to do
5:51:44these things. I I I will say I probably
5:51:46used a majority of the ones that you're
5:51:47going to put in the comments already.
5:51:49And if I haven't used it, I'll take a
5:51:51look at it and see if it's really useful
5:51:52and I'll show you that. So, thank you
5:51:55guys so much for watching. I hope that
5:51:57this has been helpful. I I feel like a
5:51:59lot of these things are not things that
5:52:01I learned before I started. Almost all
5:52:03these are ones that I learned while I
5:52:06was on the job. And so I'm hoping that
5:52:07you can get ahead of the curve and you
5:52:09can learn these things before you
5:52:10actually start so that when you get in
5:52:12there, you're just like killing it with
5:52:14the formulas and people are like, "Whoa,
5:52:16this guy is like this guy knows what
5:52:17he's doing in Excel. Give him all the
5:52:19Excel work." And then you become like,
5:52:20you know, just the Excel guy. Um, and
5:52:23everyone, you know, loves you for it. So
5:52:25with that being said, thank you so much
5:52:26for watching. I really do hope this
5:52:28helped. If you like this video, be sure
5:52:30to like and subscribe below. and I'll
5:52:32see you in the next video.
5:52:35[music]
5:52:43[music]
5:52:45What's going on everybody? Welcome back
5:52:46to another video. In this Excel
5:52:48tutorial, we'll be looking at XOOKUP.
5:52:53[music]
5:52:56Now, if you don't already know what
5:52:57XLOOKUP is, it is a new feature in Excel
5:52:59to kind of replace VLOOKUP or to be a
5:53:02much better option, at least in my mind,
5:53:04is a much better option than VLOOKUP.
5:53:06And so, if you're someone who's either
5:53:08used VLOOKUP a lot and you're trying to,
5:53:10you know, learn this new option or if
5:53:12you've never used it before, this video
5:53:13will be super helpful cuz I'll walk you
5:53:15through kind of the options and what
5:53:17XLOOKUP can do as well as the difference
5:53:18between XLOOKUP and VLOOKUP. But before
5:53:21we get into the tutorial, I want to give
5:53:22a huge shout out to today's sponsor, and
5:53:23that is Udemy. Udemy is the go-to place
5:53:26if you want a full-fledged course in
5:53:27Excel. I have three options of courses
5:53:29that I have taken on Udemy. So, I'd
5:53:31highly recommend checking those out.
5:53:33They are having a huge sale on all their
5:53:35courses during this time. And so, if you
5:53:37are in the market for a course, I highly
5:53:39recommend checking out Udemy and getting
5:53:40one there. Now, without further ado,
5:53:42let's jump on my screen and start the
5:53:43tutorial. All right, so let's get me off
5:53:45the screen because we all know why we're
5:53:47here. So, I didn't include this in the
5:53:49formulas video last week because I knew
5:53:52this was going to be a large one and a
5:53:54lot of people are going to want to know
5:53:55how to do this, what the difference
5:53:56between VLOOKUP and XLOOKUP is. So, it
5:53:58has its own dedicated video to it. So,
5:54:01let's get started. It is a formula. So,
5:54:03we're going to come in here in this
5:54:04cell. We're going to hit equal and then
5:54:06we're going to start typing XLOOKUP.
5:54:08Now, I'm going to hit tab in just a
5:54:10second, but let's read what this says.
5:54:12It says searches a range or an array for
5:54:14a match and returns the corresponding
5:54:16item from a second range or array. By
5:54:18default, an exact match is used. So,
5:54:21really useful to know. Um, we'll talk a
5:54:23little bit more about that in just a
5:54:24second. Let's hit tab and it's going to
5:54:27complete it and it's going to start
5:54:29giving us or it's going to tell us what
5:54:30our input values need to be. We're going
5:54:33to have our lookup value. We're going to
5:54:35have our lookup array, our return array,
5:54:38and then some optional things like if
5:54:40not found. So, if your option isn't
5:54:42found, you know, what will be um you
5:54:45know, the the uh output that it gives us
5:54:48a match mode and a search mode. And I'm
5:54:50going to show you um kind of how to use
5:54:52every single one of these things. As you
5:54:54can see at the very bottom, I've kind of
5:54:55already set up all of the instructional
5:54:58um the instructional content for this
5:55:01video. And so, we'll kind of get through
5:55:03all these different scenarios. So let's
5:55:05just start really quickly with um how to
5:55:08use it very simply with the lookup
5:55:11lookup array and return array. So we're
5:55:13going to come in here and we're going to
5:55:15give it our lookup value. Now Toby
5:55:17Fenderson right over here in A3 is going
5:55:20to be our lookup value. So that's who
5:55:22we're going to be searching for. Now
5:55:24we're going to hit comma and now we're
5:55:26going to be needing to look up uh or to
5:55:28input our lookup array. Now an array is
5:55:30just uh you know a range basically. So,
5:55:33we're going to do this is where it's
5:55:35going to be searching for um that value.
5:55:38This is where it searches for A3. So,
5:55:40here's Toby Fenderson. Here's Toby
5:55:42Flenderson. So, it will find it in this
5:55:44array right here. Then, we're going to
5:55:46hit comma. And now, we need to give it
5:55:49the return array, what it's going to
5:55:50return on that row when it finds it. So,
5:55:53we're going to return his email. Keep it
5:55:55really simple. So, what it should do,
5:55:57and let's close this parenthesy. What it
5:55:59should do is it should take Toby
5:56:01Flenderson. It's going to search in this
5:56:03column or in this array and then it's
5:56:06going to return the email when it finds
5:56:10Toby Fenderson. So, it's on Toby
5:56:11Fenderson is on row six. So, it's going
5:56:14to find Toby Fenderson. It's going to
5:56:16come over here and it's going to return
5:56:18Toby Flenderson at dundermland
5:56:20corporate.com. That's what it should do.
5:56:22Let's see what it actually does. Let's
5:56:24hit enter and it returns it. Now, if we
5:56:28drag it down like this, it'll apply it
5:56:30to all of these names right here. And it
5:56:33works exactly how it's supposed to. Um,
5:56:35again, if you have never used VLOOKUP,
5:56:38you don't know how good you have it.
5:56:39Okay, VLOOKUP um was extremely useful,
5:56:42but just uh a bit complicated and I'll
5:56:44talk about that near the end of the
5:56:46video when we compare VLOOKUP to XOOKUP.
5:56:49But just know that if you're using
5:56:50XOOKUP for the first time and you're
5:56:52just getting into using Excel, you guys
5:56:54have it good. Okay? So just know that.
5:56:57Um now let's go over here to XLOOKUP
5:57:00multiple rows because you can return
5:57:03more than one output with um with
5:57:08XOOKUP. So let's go right in here and
5:57:11we're going to basically write the exact
5:57:12same thing um as we did before. So let's
5:57:16write XOOKUP. We're going to do Toby
5:57:18Fenderson as our value. We're going to
5:57:20search here and we're going to do
5:57:23something a little bit different this
5:57:24time. we want to include our end date
5:57:27and the email. So, what we're going to
5:57:29do is we're going to start here. We're
5:57:30going to go down all the way to the
5:57:32bottom of end date. And then we're also
5:57:34going to include the email. And when we
5:57:36do that, it will uh in in the output
5:57:40give us a row or a column for end date
5:57:42and a column for email. So, an output
5:57:44for both. So, let's hit enter. And now
5:57:48we can see that we have the end date
5:57:49here and the email here. Now, one of the
5:57:52downsides or or something that I'm not a
5:57:56huge huge fan of is well, first off, I
5:57:58love that you can do this. That's
5:58:00fantastic. Um, but they have to be right
5:58:03next to each other. So, you you're only
5:58:05going to get that output exactly how it
5:58:07is in the columns. So, if I went and did
5:58:10this range, um, I would include all of
5:58:13that. Um, so, you know, let's just, for
5:58:16example, let's pull that down here. So
5:58:19let's take this
5:58:21and put it right here. If I did instead
5:58:25of zero or or 0 O2 to P10, if I
5:58:29[clears throat] included age to email,
5:58:31this whole range and I hit enter, it's
5:58:33all going to be included. So, you know,
5:58:36that's one of the small downsides of of
5:58:40that functionality of when you can use
5:58:41multiple rows is that it's going to use
5:58:44the rows exactly as they are. you can't
5:58:46really customize it within the formula.
5:58:49You can move around um these columns to
5:58:52how you want it. Um so that is something
5:58:55to note. And again, you can pull this
5:58:58down and it'll be applied to all of
5:59:00those names. Let's go over to XLOOKUP
5:59:03exact match. So let's open this up.
5:59:06We're going to do equals XLOOKUP as
5:59:08we've been doing. And we're actually
5:59:09going to be looking at the if not found
5:59:11and the match mode u both you know on
5:59:14this tab right here.
5:59:15So, let's do what we've been doing
5:59:17before. We take our value that we're
5:59:19looking up. We take the um array that
5:59:23we're looking and we're going to do the
5:59:26email. And you know, as you can see,
5:59:29this says Toby Flender, not Toby
5:59:32Flender. So, what we are going to do is
5:59:34we're going to hit comma. And if it's
5:59:36not found, you can return um a value or
5:59:39a string that you want to return. Now,
5:59:42for simple purposes or for simple
5:59:45instructional purposes, we're going to
5:59:46do not found.
5:59:50And then we're going to close that off.
5:59:52So, let's do this. And Toby Flenderson
5:59:55was not found. And so, it was returned
5:59:57not found. If Toby Flender was actually
6:00:00in this full name, then it would have
6:00:03returned the email. And then if along
6:00:05the way, you know, one of these was not
6:00:07part of it, then, you know, we would
6:00:09have uh we would have had the not found.
6:00:12All right. So, let's go right up here.
6:00:14We're actually just going to copy this
6:00:16uh because I want to reuse it. Um and
6:00:19then we're going to go right here. I'm
6:00:20going to hit a comma. Now, this is our
6:00:22match mode option. And so, we have four
6:00:26different options that we can choose
6:00:27from. A zero is an exact match. And that
6:00:29is uh by default, that is what we have
6:00:32or what we use. Then there's a minus
6:00:34one. That's an exact match or next
6:00:36smaller item. Then there's a one which
6:00:38is an exact match or next larger item.
6:00:41And then there's a two which is a
6:00:42wildcard character match. Now we're
6:00:44going to do that and we are going to um
6:00:47you know try this out and it's not going
6:00:50to work. And not just because I forgot
6:00:51to put A4. Um it's doing it because it's
6:00:55searching for Beasley but if there's not
6:00:58a wildcard option already put in here um
6:01:01it doesn't recognize it. So, we need to
6:01:03indicate where that wild card needs to
6:01:05be. So, we're going to do a double
6:01:07apostrophe or quotation marks. We're
6:01:08going to put an asterisk right here. And
6:01:10then do another one. And we're going to
6:01:12hit an amperand. So, we're going to have
6:01:15an amperand right here. And what that's
6:01:17going to say is anything that comes
6:01:19before A4. Anything that comes before
6:01:22beasley is okay. Doesn't matter what it
6:01:24is. As long as it has Beasley at the
6:01:26end, that is going to be okay. So we're
6:01:28going to have Pam that comes before
6:01:30Beasley and that's going to tell it and
6:01:32it's going to say okay I know that
6:01:34anything that comes before beasley is
6:01:35all right and so when we hit enter is
6:01:37now going to return the output that we
6:01:40are looking for and we can include that
6:01:42on these as well. Now this one is
6:01:45Meredith um and so Meredith is at the
6:01:49beginning so we have Meredith Palmer.
6:01:51So, we can actually take this and we're
6:01:54going to put this at the end. Put the
6:01:57amberand right here. And now it'll work.
6:02:00And the exact same thing for Kevin Malo
6:02:04right here. Kevin Malone. So, I just
6:02:06didn't include uh the ne at the end. And
6:02:10so, it's still going to work if we
6:02:12include that asterisk at the end. Now, I
6:02:14know I said we were looking at search
6:02:16order, but I'm actually going to kind of
6:02:17give you an exact match first and then
6:02:19search order. but it's just kind of
6:02:21easier to show it over here. So, I'm
6:02:23going to do X lookup. I'm going to look
6:02:26up this value. Do a comma. Here's the
6:02:29range. This is our start date that it's
6:02:30going to be looking for. And I want to
6:02:33return the full name. Now, no value in
6:02:37here has 11200.
6:02:40But what we can do is we can do comma
6:02:43and then a comma for the match mode and
6:02:45do an exact match or next larger.
6:02:48And I know this is in the exact match
6:02:50part, but it, you know, kind of refers
6:02:53to search order a little bit um where it
6:02:55searches for the next largest value.
6:02:58That's what that's what that number one
6:02:59represents, the next larger value. So we
6:03:01have 11200. And if we look right here,
6:03:03the next value above 11200 is 15200. And
6:03:08so it should return Angela Martin. Let's
6:03:10see if that works. And there it is. Now
6:03:14let's look up the actual search order.
6:03:16Um, so let's do equals xookup.
6:03:20This is the value that we want to be
6:03:21searching for and we're going to be
6:03:23looking in this start date and comma and
6:03:28we want to return the name. Now let's
6:03:31get over to search mode. Now the search
6:03:33mode performs a search starting at the
6:03:36first item. So at the very top going
6:03:38down. So by default it searches from
6:03:40first to last, but you can reverse that
6:03:43and do search from last to first. We're
6:03:45going to do a binary search, which is
6:03:47where it sorts in ascending order or
6:03:49sorts in descending order. Um, and
6:03:51that's with the actual value. And so, we
6:03:55won't be able to show this binary search
6:03:56or um ascending or descending because
6:04:00our values are the same. But if we had
6:04:03different values and we were looking up
6:04:05um using this um next largest, we would
6:04:09be able to show that. But I'm going to
6:04:10show you the search from first to last
6:04:11and last to first. So let's put in by
6:04:14default. And this is what it would be.
6:04:16Search from first to last. What the
6:04:17default would be. So it starts at the
6:04:19very top. It goes down and finds the
6:04:22first 56 2001 and returns Toby
6:04:25Flenderson. Now if we go in here and we
6:04:28hit minus one, that is going to search
6:04:30from last to first. So it's going to
6:04:32start at the bottom and go to the top.
6:04:33And the first one that it finds is
6:04:35Michael Scott. So that's that first one
6:04:37starting from the bottom. And then the
6:04:40Michael Scott right there. So these two,
6:04:42the exact match and the search order can
6:04:44kind of be combined into um this one
6:04:46right here where you're using this one
6:04:49um which is uh you know exact match or
6:04:51next larger and you can include that in
6:04:54this binary search in this one as well.
6:04:56All right, now let's head over to the
6:04:58xookup horizontal. I think we're we only
6:05:01have a few left. Yep, xookup horizontal.
6:05:02Then we'll do xookup with sum and then
6:05:04I'm going to show you the vlookup at the
6:05:06end. So let's go right here. Let's say
6:05:08equals xookup. The value that we want to
6:05:11be searching for is February. That's
6:05:12what we're looking for. Hit comma. And
6:05:14where do we want to search to find
6:05:16February? We want to search in uh these
6:05:18calendar months. And then we hit another
6:05:21comma. And now we're going to be
6:05:22searching for paper. So let's do paper.
6:05:26And we'll hit enter. And it found
6:05:29February. And it returned paper right
6:05:32here. And we can do that for paper,
6:05:34printer, and manila folders. And so it's
6:05:36going to give us the 310, the 40, and
6:05:39the 118 from February. Now, let's go
6:05:41right over here to XLOOKUP with some um
6:05:43I actually it's basically a carbon copy
6:05:46of this. Uh let's take this over here
6:05:49real quick
6:05:52and place it right there because it's
6:05:54the exact same thing except at the end
6:05:57we're going to use I'm going to show you
6:05:58how to use sum with the XOOKUP at the
6:06:02same time. Now, um, we're going to be
6:06:05using the formula sum and so we're going
6:06:09to do sum and then within the sum, our
6:06:12first number is going to be an xookup
6:06:14and then our next value is also going to
6:06:17be an xookup. So, let's do xookup.
6:06:22And now we're going to search for our
6:06:23very first value. Oops, our very first
6:06:26lookup value. So, we're going to go to
6:06:29I1
6:06:31and then we're going to search this
6:06:33again.
6:06:35And we want whatever value oops goes
6:06:39into that. So, let's close that
6:06:41parenthesis. And now we're going to do a
6:06:43colon and another x lookup.
6:06:47And now let's do March. So, now we're
6:06:51going to search for March. We're going
6:06:53to do our search range where we're
6:06:55searching for that March. And we want
6:06:57the paper as well.
6:07:00And let's close that. And then we also
6:07:02need to close that parenthesy. So now we
6:07:06are basically adding this February and
6:07:08this March. So it's going to be 310 +
6:07:11150. It's adding those um two values and
6:07:14it should be uh what 460. So let's see
6:07:17if that is our output and it is. So, you
6:07:21can do this with a lot of things, not
6:07:23just some, but you're able to use
6:07:24XLOOKUP within different formulas. If
6:07:27you're searching for a specific value
6:07:28and a specific value um in in another um
6:07:31cell, you can add those together using
6:07:34XLOOKUP, which is uh honestly, it's
6:07:36pretty great. So, let's go over to
6:07:38VLOOKUP. So, I wanted to show you this
6:07:40because I wanted to show you where it
6:07:42came from and what we used to do um
6:07:45unless you are continuing to use VLOOKUP
6:07:47and what we can do now. So, AxOookup, I
6:07:49just showed you kind of everything. Um,
6:07:51but super quickly, I'm going to show you
6:07:52how VLOOKUP used to work um in a super
6:07:55short way so that you can understand how
6:07:58it used to be used and how it is used uh
6:08:00how XLOOKUP is used now. So, let's go in
6:08:03here and we're going to say equals and
6:08:05we're going to do a VLOOKUP. And so, we
6:08:08have a lookup value. And so, we're going
6:08:10to click this. We're going to hit comma
6:08:13just like we did before. And now we're
6:08:14going to do a table array. And the table
6:08:17array is a little different in that
6:08:19you're searching an entire area. So
6:08:22let's do uh H2
6:08:26all the way through O oops 010. So
6:08:31that's what that's what our table array
6:08:34is going to be. Then we're going to do a
6:08:36comma. And now we have to do a column
6:08:38index number. Which number um are we
6:08:42going to be um searching for? which um
6:08:45value are we going to be searching for
6:08:47in here? And so we want to search for
6:08:49eight because this is 1 2 3 4 5 6 7 8.
6:08:53We want to return that email and we're
6:08:55searching for the name right here in
6:08:58this very first column. So we have that
6:09:00comma and we're going to do eight. And
6:09:02then in the range lookup, you can do
6:09:04true, which is an approximate match, or
6:09:06false, which is an exact match. And
6:09:08we'll do false. I don't know why it's
6:09:11not autodoing it, but there we go. And
6:09:14now we will do it and it's going to
6:09:16return it just as we had it. Um,
6:09:20a lot of people uh I guess not everybody
6:09:23but some people didn't like and the
6:09:25reason why they created XLOOKUP you had
6:09:27to do those ranges and if you ever went
6:09:30in here and then we let's say we um
6:09:34added another column which happens to
6:09:37data now it gives us completely
6:09:39different um different data. So let's
6:09:42say for whatever reason we added uh
6:09:44address. So now we have these people
6:09:46address. Well, now it's going to give us
6:09:48a different um value. It's going to have
6:09:50this end date because if we go in here
6:09:52now it doesn't um now the eth is this
6:09:56end date and the ninth is this email. So
6:09:58if you have a VLOOKUP that you use for
6:10:02um you know a calculation or a table
6:10:04that you've created or different things
6:10:06in Excel, you then have to go through
6:10:07here and manually change this. And so a
6:10:10lot of people didn't like that because
6:10:11if you, you know, needed to change data
6:10:13or you needed to change something or add
6:10:15an additional column, you'd have to go
6:10:16back and fix all of your VLOOKUPs, they
6:10:19wouldn't just automatically u move with
6:10:22it, which is what happens with XLOOKUP.
6:10:24And just to prove this, uh, let's go
6:10:26back to the very first one, which is the
6:10:28XOOKUP. And right now the email is
6:10:31looking at O2 and through O10. Um, we're
6:10:35just going to insert right here. And
6:10:37that will be our new column. We'll do
6:10:39address. Oops.
6:10:42Address. And notice that it hasn't
6:10:43changed. And why is that? Because it
6:10:45auto changed for us from P2 to P10.
6:10:49Understanding that it wanted to stick
6:10:51with when something was inserted here,
6:10:52it wanted to stick with the original
6:10:54data or the original array that was
6:10:56selected. And so XLOOKUP does that work
6:10:59for you. and it makes it a little bit
6:11:01easier to automate things and create
6:11:04these processes in Excel without having
6:11:06to go fix it later, which you had to do
6:11:08with VLOOKUP. So, that is it for today.
6:11:10I hope that you know how to use Xookup a
6:11:12little bit better now that you have
6:11:13watched this. Uh, if you enjoyed this
6:11:15video, be sure to like and subscribe
6:11:17below and I will see you in the next
6:11:18video.
6:11:31What's going on everybody? Welcome back
6:11:33to another Excel tutorial. Today we'll
6:11:35be looking at conditional formatting.
6:11:39[music]
6:11:42Now, if you've never heard of
6:11:43conditional formatting before, that's
6:11:45okay. I had never heard of it before I
6:11:47became a data analyst. And so, now that
6:11:49I've been using Excel a lot, of course,
6:11:50I use it quite a bit. And so I want to
6:11:52show you how to use it. Conditional
6:11:54formatting is basically just a way to
6:11:56see patterns and trends in data. And
6:11:57that's a super simple way of putting it.
6:12:00Um, but it's very easy to use and so
6:12:03hopefully I can show you how to use it
6:12:05uh really easily and a lot of the things
6:12:06that I use the most and some of the
6:12:08things that I use it for so that you can
6:12:10also know how to use conditional
6:12:11formatting. Now before we jump into the
6:12:13tutorial, I want to give a huge shout
6:12:14out to the sponsor of this Excel series
6:12:16and that is Udemy. You guys know by now
6:12:18that I absolutely love Udemy. I've been
6:12:20using them for years and I've taken
6:12:21literally hundreds of courses on Udemy
6:12:24and I've learned so so much especially
6:12:25when I was first starting out as a data
6:12:27analyst. Uh I learned a lot through
6:12:29their Excel courses on Udemy and so I
6:12:32have actually put the ones that I really
6:12:33like and I have taken and enjoyed and
6:12:35think you would as well in the
6:12:37description. So if you want to take
6:12:38those, be sure to check those out.
6:12:40Again, huge shout out to Udemy for
6:12:41sponsoring the series. Now without
6:12:43further ado, let's jump onto my screen
6:12:44and get started with the tutorial. All
6:12:46right, so let's jump right into it. On
6:12:47this home tab right here, if we go all
6:12:49the way over to the right, there is
6:12:51conditional formatting. And the
6:12:53description that it gives us is easily
6:12:54spot trends and patterns in your data
6:12:56using bars, colors, and icons to
6:12:58visually highlight important values. And
6:13:00that is exactly how I would have defined
6:13:03it. U really good job, Microsoft.
6:13:05Exactly how I would have done it. So
6:13:06what you'll see right away is nothing
6:13:08too complex. So we have some highlight
6:13:10cell rules. Um, we have some top bottom
6:13:13rules, data bars, color scales, icon
6:13:16sets, and then at the bottom we can
6:13:18create a rule, we can clear the rule,
6:13:19and we can manage our rules. So, if you
6:13:21create a rule, then you can manage it.
6:13:24So, we're going to start with these icon
6:13:26sets, and I'm going to show you how to
6:13:27use those, and we'll work our way to the
6:13:29top, and then I'll show you how to
6:13:30create some rules yourself, and how that
6:13:33all works. So, let's start off with the
6:13:36icon sets. I'm going to go over here to
6:13:37sales. Um and for this data we kind of
6:13:41have this um you know trend or or
6:13:44pattern that you can kind of see over
6:13:46time. So over the months um so if we go
6:13:49right here and let's use that
6:13:52conditional forming let's use that icon
6:13:54sets and right here we can use these
6:13:57directional. So you know we have this
6:13:59kind of time series each month that
6:14:01shows us how much paper they're selling.
6:14:03And if we do this right here, it's going
6:14:05to show us if it's kind of average or if
6:14:08it's below average or if it's above
6:14:11average or if it's going up. So, at a
6:14:13really quick glance, you can kind of see
6:14:15the pattern of this data set. It's kind
6:14:17of going mostly yellow and red. There's
6:14:20only two months where it's going up
6:14:22significantly. Now, we don't have to
6:14:24only do that for one row or one column.
6:14:27You can apply it to all of them. But, as
6:14:30you can see, all of these are red. Now,
6:14:32why are they all red? It's because
6:14:34they're using numbers for everything.
6:14:36So, they're comparing these 24s and
6:14:38these 50s and 65s against these 450s and
6:14:42750s. And so, they're all going to be
6:14:44red. But, if we do it individually, if
6:14:46we do it each row, if we take it just
6:14:49like this, and then we go to icon sets
6:14:51and do it, it's going to be much more
6:14:53representative of the actual printers,
6:14:56not of all the numbers as a whole. And
6:14:58you can do other things. The arrows are
6:15:00ones that you'll probably see the most
6:15:02often. That's the one I've used if I
6:15:04ever do use them. Um, but you can, you
6:15:07know, do ones like this where they have,
6:15:10you know, kind of a trend upward or a
6:15:12trend downward. Um, and so there's just
6:15:14several more arrows. This one only gives
6:15:16you three. As you can see, this one
6:15:18gives you five. Um, and you can do, you
6:15:21know, colors or shapes or or different
6:15:23indicators and all these different
6:15:25things. Um, and honestly, it's kind of
6:15:27whatever you want to use, whatever makes
6:15:28sense for your data, but you know, I've
6:15:30really only ever seen like these colors
6:15:32being used. I've never really seen these
6:15:34flags or anything like that. But again,
6:15:36it just depends on what industry you
6:15:38work in. You might you might see that.
6:15:39Let's go right over here to the
6:15:41demographics. Um, and let's look at our
6:15:45color scales. Now, color scales are
6:15:47going to be the probably the most
6:15:48obvious thing that in data bars are
6:15:50going to be the most obvious things in
6:15:52here. Um, if you go right here and and
6:15:55you look at this color scale, if it's
6:15:57high, if it's among the top ones, it's
6:16:00green, the lowest, it's red. And you can
6:16:03change that um to really any colors you
6:16:05want, any colors that they offer you.
6:16:07Um, and it it does exactly what it does.
6:16:11It's a color scale, a gradient of the
6:16:13colors from high to low or low to high.
6:16:16And so any color that you do, you'll be
6:16:18able to kind of see um, you know, what's
6:16:20good and what's not good. That really is
6:16:24um color scales in a nutshell. Data bars
6:16:28are again super super straightforward.
6:16:31It's going to be either a gradient fill
6:16:33or a solid fill. So let's look at the
6:16:34gradient fill. If we do a blue gradient
6:16:37fill, actually let's get rid of our um
6:16:40let's go over here. Let's go to clear
6:16:42rules from selected cells. We haven't
6:16:44looked at that yet, but that's how you
6:16:46clear it. Let's go to data bars and
6:16:49we'll use this blue gradient. So with
6:16:52this blue gradient, you know, this one
6:16:53is or sorry, this one is the highest
6:16:55one. So it's going to be completely
6:16:57filled. And this one is 36,000 almost
6:16:59half of this. Um pretty close. And so
6:17:02it's almost half. Um this one again, you
6:17:05know, it's not used very often. I you
6:17:09don't see these a lot to be honest. You
6:17:11just don't. Um but if you do see it,
6:17:14that's how you use it. That's how it can
6:17:16be done. Again, pretty easy. Uh, as I
6:17:19just showed a second ago, if you want to
6:17:21clear the rules, you can clear it from
6:17:22the selected cells. That's what we're
6:17:23doing. So, I have column G selected, and
6:17:25I'm going to I'm going to clear that. If
6:17:27you want to clear the rules, the entire
6:17:28sheet, you can do that as well. So, it
6:17:30would affect every single column and
6:17:32row. We'll just do this for now. So, now
6:17:36let's go look at the top bottom rules.
6:17:38So, this is the top 10 items, top 10%,
6:17:41bottom 10 items, bottom 10%, above
6:17:44average, and below average. and they're
6:17:45going to do exactly what you think they
6:17:47are going to do. If you select above
6:17:49average, it is going to select or
6:17:51highlight the cells that are above the
6:17:54average in column G. So, let's look at
6:17:56the salaries that are above average. All
6:17:58right. And so, uh the ones that are at
6:18:01the very top are Michael Scots, Toby
6:18:03Flenderson's, and Dwight Shroo. Uh no
6:18:07shock there. Um I believe the average is
6:18:09somewhere around like 48,500
6:18:12or something. So, I think this one just
6:18:14is just below it. And so, all these
6:18:16other ones are below average. And that's
6:18:18just because, you know, Michael Scott
6:18:20and Dwight Tru are and Toby are kind of
6:18:23bringing up that average quite a bit.
6:18:24So, everyone else is going to fall
6:18:26beneath that. And so, at a super quick
6:18:28glance, you're able to just highlight
6:18:30the cells and you're able to see who is
6:18:33above average. And, you know, you can do
6:18:36this in a lot of different ways in
6:18:37Excel, but this is just a really simple,
6:18:39fast way to do that. Um, let's get rid
6:18:42of that real quick and let's go back up
6:18:44here. And now we can Oops. Let's go to
6:18:47top bottom rules. And now we can see the
6:18:48below average. And it's going to
6:18:50highlight all the other ones. And so it
6:18:52works exactly how you think it is going
6:18:54to work. And this is the default way
6:18:56that it highlights these cells. So it
6:18:58highlights them this kind of um
6:19:00see-through red and then it highlights
6:19:02the actual text or or the um characters
6:19:05in there red as well. Now, I'm not going
6:19:07to go through and show you every single
6:19:09one of these top bottom rules. I think
6:19:11they're pretty self-explanatory. I just
6:19:13kind of wanted to show you what happens
6:19:14when you do use one of them. It's going
6:19:16to highlight that cell. So, let's go up
6:19:19here to the highlight cells rules. And
6:19:21honestly, these are the ones that I use
6:19:24by far the most. Uh all these other ones
6:19:26combined I do not use more than this
6:19:29highlight cells rules. Um and the one in
6:19:31here that I use more than any other
6:19:33conditional formatting rule is this
6:19:34duplicate values. So, I'll start with
6:19:36that really quick and I'll kind of show
6:19:38you a few of these other ones. But this
6:19:40duplicate values to me is one of the
6:19:43most useful ones. Um, and so let's kind
6:19:46of show you how that works. If we go to
6:19:49the start date, you can see that we have
6:19:51a duplicate value right here. And if we
6:19:54go over here to conditional formatting,
6:19:56highlight cells, rules, and duplicate
6:19:58values. It is going to highlight um the
6:20:02duplicate. And that says duplicate right
6:20:03here. Now, we can go through here and
6:20:06click on unique. Um, and then it would
6:20:08highlight all the ones that are not
6:20:10duplicates. Um, so you can use it, you
6:20:13know, kind of in a similar inverse way.
6:20:15Uh, it's just different different, but I
6:20:17use the duplicate almost always. Um,
6:20:20another thing that you can do is go over
6:20:22here and you can change the color. Um,
6:20:25or you can even do a custom um, which I
6:20:28just never do that. It's not um,
6:20:30something I spend a lot of time doing. I
6:20:32typically just stick with this one. So,
6:20:34you can do that and it's going to
6:20:35highlight um you know something that has
6:20:38a duplicate value in there. Now, why do
6:20:41I use this so much? Well, I work with a
6:20:44lot of different types of data sets, but
6:20:46one thing that you'll find in almost all
6:20:48of them is they have some type of ID and
6:20:52they're going to have some type of um
6:20:54personal information, whether that's a
6:20:57social security number or an address or
6:21:01um you know
6:21:03or a cell phone number or something like
6:21:05that. There is going to be data that is
6:21:08going to identify that person. Now, I
6:21:10work a lot with pharmaceutical data, a
6:21:12lot with pharmacy data, um, as well as
6:21:16healthcare data. So, like names, social
6:21:18security numbers, addresses, phone
6:21:19numbers, all of those things, all that
6:21:21customer or or client information. And
6:21:23oftent times when I get a new data set
6:21:25and I have it in Excel or I convert it
6:21:27to Excel, I will start using these
6:21:29duplicates to try to find issues with
6:21:32the data and I find them all the time.
6:21:34either there's an employee ID or some
6:21:36type of customer ID or client ID that
6:21:38has a duplicate in there that should not
6:21:40be in there or there's multiple social
6:21:42security numbers or there's an issue in
6:21:44some other way and I'm able to find
6:21:45those things and spot those patterns
6:21:48using this duplicates and I promise you
6:21:50I use this one almost every single time
6:21:52I open a new data set or I work with a
6:21:54new client working with their data. Um
6:21:56and so I wanted to show you this one. I
6:21:58wanted to really press upon you that
6:22:00this one is a really, really, really
6:22:02good one to know and learn how to use.
6:22:04It's not complicated. It's not hard. It
6:22:06just shows you, you know, you know, if
6:22:09there's a duplicate value, but I wanted
6:22:10you to know how I use it and how often I
6:22:13use it so that you can, you know, pick
6:22:15that up and put that in your toolkit in
6:22:16your back pocket so that you can use
6:22:18that later on if you have uh if you have
6:22:20a similar need or if you're trying to do
6:22:22something similar to what I was just
6:22:24talking about. So, that is how
6:22:26duplicates work. Again, super great.
6:22:29It's obviously not super useful when
6:22:31you're only using um 10 rows, but when
6:22:32you have, you know, 50,000, 100,000, and
6:22:35there should be zero duplicates in
6:22:37there, and you highlight it, and then uh
6:22:40you come right here, use the filter,
6:22:43and we're going to filter, and we're
6:22:45going to sort by the color, and it
6:22:48allows you to sort by the color, and you
6:22:50have duplicates in there, then that's a
6:22:51problem. And you identified a problem
6:22:53super quickly. Uh, and you know, some of
6:22:56those things they slip by because nobody
6:22:58checks it. And so that's something that
6:23:00I I often check. And if you go here and
6:23:02you sort by color and there isn't an
6:23:03option to do um this this pink red
6:23:06color, then that means there aren't any
6:23:07duplicates. And that's a really good
6:23:09thing. Most of the time that's a really
6:23:10good thing. So let's go ahead and we're
6:23:13going to clear that as well as
6:23:16get rid of our conditional formatting
6:23:19rules.
6:23:20Now, another one that I use a lot is
6:23:23this one right here, which is the text
6:23:27that contains. Honestly, this one comes
6:23:30a lot in handy, especially when you're
6:23:32looking for like a specific keyword. In
6:23:35my uh case, a lot of times I was using
6:23:39this when I was going through drug
6:23:41names. I am not a doctor. I do not
6:23:43pretend to be a doctor. And so when I
6:23:44was looking for Lorazzipam or something
6:23:46like that, um I would just search for
6:23:48like Lorazz or something and and not
6:23:51Lorax but Lauraz, you know, I I would
6:23:53just search for it and then all the ones
6:23:56that contain that would pop up. I can
6:23:57bring them to the top and I can see
6:23:59them. And to me that's super super
6:24:02useful and I would do that all the time.
6:24:04And so in this case we're looking at
6:24:05emails. And let's say we all only wanted
6:24:08to pull all the ones that are Gmail. And
6:24:10so now we can go through and we can you
6:24:12know click okay and that's going to pop
6:24:14up or we want all the ones that have
6:24:17dunder oops dunder mifflin. And if we
6:24:21click on that all the ones that are
6:24:22dunder mifflin come up or have dunder
6:24:24mifflin in it. And again we can um sort
6:24:27by or we can um and so we can sort by
6:24:31right here and we can bring all those to
6:24:33the top. And so super super useful. Um,
6:24:36and another use for it that you may not
6:24:38think of is something like if it's, you
6:24:41know, there's some incorrect data in
6:24:43there. This happens often with phone
6:24:45numbers, addresses, um, start dates or
6:24:49or or dates in general, date formats
6:24:52where you can go in here and you can say
6:24:55text that contains and if you know you
6:24:57put in a oops a dash and it has it in
6:25:01there, then you know that that is that
6:25:03is wrong. Now that is really all I
6:25:05wanted to show you in the highlight
6:25:06cells [clears throat] rules. Uh the
6:25:07duplicate values and the text contains
6:25:09are by far the ones I use the most. All
6:25:12the other ones I have used. Um these
6:25:14ones not so much. But in these highlight
6:25:16cells rules I use you know these two all
6:25:18the time. Um sometimes I use this
6:25:21between I don't really use these other
6:25:23ones as much although I have used them.
6:25:25And so if you got nothing else from this
6:25:27video I just wanted you to know that
6:25:28these two are super useful. and if you
6:25:31haven't used them before to maybe try
6:25:32them out and see how you can apply them
6:25:34to your own data sets. Now, we've looked
6:25:36at all of these preset ones and
6:25:38conditional formatting, but you can also
6:25:41do a new rule. And so, if we click on
6:25:43new rule right here, and we go down to
6:25:45use a formula to determine which cells
6:25:47to format, we can add our own formula in
6:25:50here that will then highlight exactly
6:25:53what we want. And so if there isn't a
6:25:55preset rule that you like and it doesn't
6:25:58have the option that you want, you can
6:26:00do almost any formula that you want in
6:26:02our formulas video that we did a few
6:26:04weeks ago and you can put it in here.
6:26:05And then you can format uh what you want
6:26:08the cell to look like if it meets that
6:26:10criteria. So let's take this right over
6:26:12here. Um and before we start this
6:26:14formula, I just want you to note that,
6:26:17you know, I have H11 highlighted. That's
6:26:19going to come into play in just a little
6:26:21bit, but I wanted you to be aware that
6:26:23H11 is the cell that we're highlighted.
6:26:25So, what we're going to do is we are
6:26:27going to create our formula. Now, if
6:26:30you've never created a formula, I highly
6:26:32recommend uh watching my formulas
6:26:34tutorial because that is going to show
6:26:35you how to do this. Um, but we're all
6:26:37we're going to do is we're going to do
6:26:39equals. That's how you start the uh how
6:26:42you actually create a formula. And we're
6:26:44going to give it this range right here.
6:26:46And so it's going to take everything
6:26:48from G2 to G10. Now, these dollar signs
6:26:51are super important. If you don't know
6:26:53how to use them or you don't know what
6:26:54they do, um you're going to mess up this
6:26:57formula a lot. Uh and so what this
6:27:00dollar sign basically does is it's
6:27:02basically hard coding it in there. It is
6:27:04only going to look at G2 and is only
6:27:06going to look at G10 or through G10
6:27:09because that colon. And this can come
6:27:11into play because if you have something
6:27:14selected like the H11, it's going to
6:27:16mess it up because now if you have H11
6:27:19selected like we do, you'll see this in
6:27:20a second. It's not going to be applied
6:27:23to this. Um, and again, I'll show you
6:27:26that in just a minute. But we don't want
6:27:27this hardcoded in there. Okay. But we do
6:27:31have to select the proper range in a
6:27:33second. Um, so we're going to get rid of
6:27:35this. We're going to get rid of the
6:27:36dollar signs because we want it to be
6:27:38pretty fluid and be able to applied to
6:27:40be applied basically anywhere we want.
6:27:42Let's go into this formula. Um, if it
6:27:46meets our criteria, let's give it um
6:27:49let's give it a border and we'll give it
6:27:52um we'll give it some color. We're going
6:27:55to say if this is greater than 50,000.
6:27:58So, let's hit okay. And nothing
6:28:01happened. So, let's go back and see why.
6:28:04So, if we go to our manage rules, you
6:28:06can see that it still has the G2 to G10
6:28:08is greater than 50,000, but it only is
6:28:10being applied to this H11 cell, which
6:28:13really makes no sense. Um, so if we had
6:28:16wanted to get it done the first time, we
6:28:18needed to have basically selected that
6:28:19G2 to G10 right away. Um, but we can do
6:28:22that now. So, let's get rid of this. And
6:28:25we're going to say G2 to G10.
6:28:30And that is hardcoded in there. That
6:28:32should be fine still. Um but let's see
6:28:35what it does.
6:28:37And so now every single thing is
6:28:39highlighted. And why is that? Uh that's
6:28:42because when we changed it, it also
6:28:45changed the format of it because we
6:28:47changed the cell that we were looking
6:28:48at. So we need to come back here. And
6:28:51that's why again you want to do this the
6:28:52right way the first time. We're going to
6:28:53come back here. We're going to give it
6:28:54this range.
6:28:56And we're going to get rid of these
6:28:58dollar signs.
6:29:02And now we're going to hit okay. And so
6:29:05now it's being applied G2 to G10 and G2
6:29:09to G10. And we'll keep it like that. And
6:29:11we'll apply it. And now it works
6:29:13properly. So now everything that's above
6:29:1550,000 is being highlighted. Again, if
6:29:17that was confusing, um it it is
6:29:19confusing. It genuinely is. And so if
6:29:22you wanted to do this right the first
6:29:23time without having to make a bunch of
6:29:25changes, you'd want to highlight these
6:29:27before you start. And then you want to
6:29:29go in and create the rule. We'll do this
6:29:32really quick just to kind of show you
6:29:33what I'm talking about. We'll say
6:29:34equals. We'll give it this range.
6:29:39Get rid of these real quick because
6:29:42again, I don't want this
6:29:45hardcoded in there. It will ruin our
6:29:46formula. And then we'll say greater than
6:29:4930. Um, and we'll give it this nice
6:29:52green. Uh, and so now if they're over
6:29:55the age of 30, it will be highlighted.
6:29:57And we didn't have to go back and change
6:29:58anything. We didn't have to go back and
6:30:00fix anything like we did in the first
6:30:01one. Um, that was all for demonstration
6:30:04purposes. But again, you need to really
6:30:06be aware of that. That is something that
6:30:08I think almost everybody's going to mess
6:30:10up at some point. If you don't already
6:30:12know about it, then you definitely are
6:30:14going to make that mistake. Now, if we
6:30:16come over here in this area, uh, we go
6:30:18to our manage rules and not just the
6:30:20current selection, but this whole
6:30:21worksheet, then you can see that we have
6:30:23these two formulas. Now you can go in
6:30:25and edit any of these by double clicking
6:30:26or clicking on it and then hitting edit
6:30:28rule. You can also delete these rules or
6:30:31duplicate these rules. Um I just wanted
6:30:33to show you what you are able to do with
6:30:34them. But if we uh go ahead and we get
6:30:37rid of this. Um so let's say we delete
6:30:40that rule and we hit apply uh you know
6:30:42the rule is going to go away. That's
6:30:44that I mean it's as simple as that. So
6:30:46that is how you can create your own
6:30:48rule. I want to be again very specific
6:30:52in the fact that that is a confusing
6:30:54piece. And if you mess that up, you're
6:30:56going to be, you know, fixing a bunch of
6:30:58different stuff and not understanding
6:31:00why your rule is not working properly.
6:31:02It's just because it's confusing. Those
6:31:04dollar signs are are really important to
6:31:06watch out for. And that is all there is
6:31:08to it with conditional formatting.
6:31:10Again, conditional formatting is um you
6:31:12know, it's not anything super confusing.
6:31:15We've looked at more complicated things,
6:31:16but it's a really really useful tool to
6:31:19use to look at these patterns and trends
6:31:21super quickly and to find um these
6:31:24outliers or these specific values that
6:31:25you're looking for very quickly. And if
6:31:27you're looking at just thousands and
6:31:30tens of thousands or hundreds of
6:31:31thousands of rows, this is one of the
6:31:33fastest ways to find these things
6:31:35without having to kind of wait and
6:31:37filter and use these um these these
6:31:40filters right here because again, this
6:31:41can just take forever. Um, and so if you
6:31:44haven't or if you've never worked with a
6:31:46ton of data and tried to use this
6:31:47before, it can take honestly like 10
6:31:50minutes for something simple that you
6:31:52could do with conditional formatting in
6:31:53like 10 seconds. So definitely something
6:31:55to mess with and use when you are
6:31:57working with your own data sets. Uh, I
6:31:59hope this was helpful. I mean, honestly,
6:32:01I use this all the time. So, you know, I
6:32:03hope that somebody out there can can use
6:32:05this uh for their own work that they're
6:32:07currently using. Thank you guys so much
6:32:09for watching. I really appreciate it.
6:32:10Again, huge shout out to Udemy for
6:32:12sponsoring this Excel series. If you
6:32:14like this video, be sure to like and
6:32:15subscribe below. I'll see you in the
6:32:17next video.
6:32:20[music]
6:32:30What's going on everybody? Welcome back
6:32:31to another Excel tutorial. Today we will
6:32:33be looking at charts.
6:32:40Now, if you have data in Excel and you
6:32:42want to visually show that with bars or
6:32:44graphs or anything like that, you can do
6:32:46that really simply. And I'm going to
6:32:48show you how to do that today. And a lot
6:32:50of people are a little bit intimidated
6:32:51because they think it's a little bit
6:32:53complicated. But I promise you, by the
6:32:55end of this video, you will know how to
6:32:57do it like a pro. It's not that
6:32:59difficult. It's just you need to know
6:33:01where to look, where to click, and how
6:33:02to actually filter through things to
6:33:04make sure that you're visually showing
6:33:05the things that you want to show. But
6:33:07before we actually jump into the
6:33:08tutorial, I want to give a huge shout
6:33:09out to the sponsor of this Excel series,
6:33:11and that is Udemy. You may not know
6:33:13this, but I probably get at least 15 to
6:33:1550 companies every single month reaching
6:33:17out to me wanting to sponsor the channel
6:33:19and promote their product. And I turn
6:33:21down almost every single one because I
6:33:23either don't know their product or I
6:33:24don't believe in their product. And so,
6:33:26I'm not going to, you know, go and
6:33:27promote that on my channel. But Udemy is
6:33:29one that I have consistently promoted
6:33:31over the past year. And that's because I
6:33:32truly believe in their product. I've
6:33:34been taking courses off their platform
6:33:35for years and I've honestly learned so
6:33:38much and I cannot recommend them enough.
6:33:40So, if you want to take a full-fledged
6:33:41Excel course, I have my recommendations
6:33:44in the description if you want to check
6:33:45those out. Thank you again to Udemy for
6:33:48sponsoring this Excel series. So,
6:33:49without further ado, let's jump on my
6:33:51screen and get started with this
6:33:52tutorial. All right, so let's jump right
6:33:54into it. Right here we have the Dunder
6:33:55Mifflin sales report and over here we
6:33:58have all the products that they were
6:33:59selling along with the months that they
6:34:01were sold in. And so in January they
6:34:03sold 450 reams of paper. Down here we
6:34:07have the total items per month. And so
6:34:09in January they sold 898 units of uh
6:34:13products or or things that they sold.
6:34:15And at the very end we have the year-end
6:34:17total. So this is the total amount of
6:34:18paper that they sold throughout the
6:34:20year. Now we're going to use this data
6:34:22right here for all of our charts. Now
6:34:25you may not have data exactly like this.
6:34:27It can come in lots of different
6:34:28flavors, but you're going to get the
6:34:30basic gist of how to use charts, how to
6:34:33edit it, how to customize it to fit what
6:34:36you need, and then we're going to kind
6:34:37of put it right over here and kind of
6:34:39create its own sheet where we can kind
6:34:42of visualize all the things that we want
6:34:44to show.
6:34:46So, let's jump right back over here into
6:34:48sales. And first thing we need to do is
6:34:51kind of highlight the data that we're
6:34:52going to be working with. Now, I'm going
6:34:54to start with everything, but um you
6:34:56know, I'll show you along the way. We
6:34:57don't actually want everything, but we
6:34:59can filter that stuff out as we go. So,
6:35:02let's go right here, and we're going to
6:35:04insert, and we're going to go over to
6:35:07charts. Now, this is the chart section.
6:35:08There's lots of different types of
6:35:10charts. Um but the first thing that
6:35:12we're going to be looking at is right
6:35:14here. This is a 2D column or kind of
6:35:16like a bar chart. And we're just going
6:35:18to click right here. And we're going to
6:35:20pull this down.
6:35:22So, now that we have this down here,
6:35:24there are a few things that I want to
6:35:26show you before we actually really get
6:35:28into it, and I kind of want to show you
6:35:29the options that you have. So, if you go
6:35:31up here, we have different uh chart
6:35:34styles. And so, if I hover over them,
6:35:37you can see that each one kind of looks
6:35:40a little bit different. And it really
6:35:42doesn't matter. Uh it doesn't really
6:35:45change the data in any way, just how you
6:35:47visualize it. And so if that is
6:35:49important, if that is something that you
6:35:51um you want to stick with a certain
6:35:53theme or a certain look, then go for
6:35:55that. Uh the other thing that's really
6:35:58nice to have over here is this switch
6:36:00row and column. So right down here, you
6:36:02can see this purple and you can see this
6:36:04red. Those are our rows and columns. And
6:36:07we can switch that right here. So if we
6:36:09go like this now, instead of the months
6:36:11being right here, the months are the
6:36:13colors and the actual product is right
6:36:16here. Let's click it again and it'll go
6:36:18back. And so now we have this kind of
6:36:20time series. So now we have January
6:36:22through the end of year total. Now this
6:36:24one is one that I think is super
6:36:27helpful. You know it you can do it down
6:36:29here as well. If you go to this filter
6:36:31um but both of these are super helpful
6:36:34because you sometimes just want to
6:36:36select all the data and then kind of get
6:36:37in there and mess with it. Something
6:36:39that we want to get rid of is this total
6:36:41items per month. So we want to remove
6:36:43that. And then we also want to remove
6:36:45this year-end total because both of
6:36:47those are are kind of the end result.
6:36:51They're not the actual data per month or
6:36:53or per product. So, we're going to get
6:36:55rid of those and we're going to apply
6:36:56that. And as you can see, just right off
6:36:59the bat, our data has changed
6:37:00dramatically. Uh, and that's because we
6:37:02aren't including these these large large
6:37:05numbers that were kind of throwing off
6:37:07uh the visualization for us. So, this
6:37:10one right here as is already pretty
6:37:13good. Um, what we can do right here is
6:37:15we can change this and we're just going
6:37:17to say
6:37:19products sold
6:37:22per month.
6:37:25Now, what we can do if we want to move
6:37:27it to another um to another sheet is we
6:37:30can actually move the chart and we can
6:37:32select where we want to move it. We can
6:37:34move it to chart sheet and we can do
6:37:35that. Or something that I do um almost
6:37:3899% of the time is I just copy and I
6:37:41come over here and I'm going to paste
6:37:43it. And so now we have this um this
6:37:47chart right over here as well as back
6:37:50here. And so I typically tend to do that
6:37:53because now we can still go over here
6:37:55and change this one as much as we want.
6:37:57So if we want to go in here, we can
6:37:58alter this one and it won't affect the
6:38:00other one. So we just have basically two
6:38:02copies. So, we're going to keep this one
6:38:04right here. This is going to be our
6:38:05first visualization.
6:38:07Um, and as I said, it's it's fairly
6:38:09straightforward. If you've ever done any
6:38:11types of charts or graphs before, um,
6:38:13right here, it's January, February,
6:38:15March, April, May. And if you hover over
6:38:17these, you can see that that's the the
6:38:19paper. And if we just glance, you know,
6:38:22the paper is their biggest product by
6:38:23far. And so, that blue um, which is
6:38:26their paper, is going to be the biggest
6:38:27every single month. So, that makes
6:38:30perfect sense. Now, what if we want to
6:38:32change up uh the the kind So, what if we
6:38:35want to change up the kind of
6:38:37visualization that it offers us? Well,
6:38:40we have a lot of different options.
6:38:42Let's go right over here to change chart
6:38:44type. Now, this is going to offer you
6:38:47just about everything you could possibly
6:38:50imagine or want and even things that you
6:38:52absolutely would never ever want ever.
6:38:55Um, and so I'm going to show you some of
6:38:57the good ones and I'm going to show you
6:38:58some just absolutely insane ones that uh
6:39:01Excel came up with which cannot I I
6:39:04could not imagine a scenario that these
6:39:06are ever used. Um, but within these
6:39:08columns you can do they're called
6:39:10cluster columns. Uh, these stacked
6:39:13columns. So it would look just like
6:39:14this. Those are often used as well.
6:39:18Um, and then we have ones that they're
6:39:21just not used often. Let's look let's
6:39:22take a look at this one right here.
6:39:25I mean, it's tough. It's tough to look
6:39:27at. Um, but let's let's put it right
6:39:29here. This is basically the same thing
6:39:32that we just had except visualized in a
6:39:35different um we'll call it more unique
6:39:37way. Uh, and let's for the sake of it,
6:39:40let's put it over here. Um, these two
6:39:42things show the same information. They
6:39:45show the same data. Just one is shown
6:39:48well and one is not shown well. Um, I'm
6:39:51not a fan of these 3D type of
6:39:54visualizations.
6:39:55I I just don't like them. But maybe you
6:39:58do and and you want to use that. That's
6:40:00fantastic. Let's go back. Um, something
6:40:04else that you'll probably use a lot are
6:40:06things like these um these line graphs.
6:40:09Okay, so these are line graphs and
6:40:11they're different types. So there are
6:40:12these stacked um 100% stacked line lines
6:40:16with markers, different flavors for this
6:40:20this type of line graph. And so you can
6:40:22go in here and take a look. Again, um
6:40:26not my favorite, but they have it as an
6:40:29option if you ch so choose to do this.
6:40:32Um but I kind of I'm kind of a simple
6:40:33guy. Um but I'm going to go in here and
6:40:36it's pretty clustered. Um, I want to
6:40:39kind of take the ones that have the
6:40:41highest sales
6:40:43or the highest total amount sold. So,
6:40:45that would be paper, manila folders, and
6:40:49three ring binders. So, let's go in
6:40:51here. We want to keep paper. We want to
6:40:55keep uh manila folders.
6:40:58And we want to keep three ring binders.
6:41:01And let's apply that. And so, now it's a
6:41:03lot cleaner. And we're just going to
6:41:06copy this. and we're going to put it
6:41:08over here. And I'm just putting these
6:41:10all over here for you u because we'll
6:41:11look at this at the end and just kind of
6:41:13see different options and and ways to do
6:41:15things as we have gone through this
6:41:17tutorial. So let's go back here. Now
6:41:20something else that we haven't looked at
6:41:22is the actual colors and color schemes
6:41:25that you can do. So let's go right here
6:41:27to these chart styles and we can go to
6:41:29color. Now, color is um something that
6:41:33probably is quite overlooked um in
6:41:36actual charts and graphs. Some terrible
6:41:38colors like this or or this um where
6:41:41they're really close together,
6:41:42especially when you have a lot of them.
6:41:44Um for example, let's just pretend we
6:41:47put all of them back really quickly. It
6:41:52is near impossible to distinguish these
6:41:54colors. Um we wouldn't
6:41:57we wouldn't want that. Let's go back to
6:41:59this color. You know, when you have it
6:42:01like uh in some of these colors at
6:42:03least, it at least distinguishes them so
6:42:06you can kind of see what you're working
6:42:07with, but when you have it in these
6:42:09monochromatic options, sometimes they're
6:42:12just impossible to distinguish. So, be
6:42:14sure to choose the right colors that
6:42:16you're using so that if somebody who's
6:42:19never seen this data before looks at it,
6:42:21they can easily distinguish uh the
6:42:23product and the month that you are
6:42:26looking at. But let's go just back up
6:42:28here. We'll choose this default option.
6:42:30Um, oh, let's choose this one right
6:42:32here. This one's nice, although there's
6:42:33lots of yellows and oranges. Let's see
6:42:35this one. This one's not bad. Greens,
6:42:38blues, uh, and like [snorts] yellows.
6:42:41So, that's nice. Um, other things that
6:42:44we want to look at, and there are these
6:42:46chart elements right here. Other things
6:42:48that we can add are things like data
6:42:51labels. Um, and right here, it's super
6:42:54messy. Um, but if we went back and we
6:42:57got rid of some of these things like the
6:43:00printer, staples, highlighters, pens,
6:43:03and total, if we apply that, it's a
6:43:05little bit easier to distinguish. Um,
6:43:08and that's, you know, something that you
6:43:10may be interested in doing. You can also
6:43:12add this data table at the bottom, which
6:43:15is the actual columns and rows that you
6:43:18have for this visualization right here.
6:43:19Now, let's expand this quite a bit. I'm
6:43:21going to make this extremely large. If
6:43:24you have something like this, it
6:43:25actually can be pretty nice. Um, you
6:43:27know, maybe we get rid of these data
6:43:29labels, but it can be easy because
6:43:32you're putting it all in one place. You
6:43:33can also make this two separate
6:43:35visualizations. So, you can have one
6:43:36visualization just like this, and right
6:43:38underneath it, you can have the actual
6:43:40rows and columns, but this option allows
6:43:42you to put it all in one. So, let's put
6:43:44this back down because that is way too
6:43:47big. And uh, wait, let's expand it a
6:43:51little bit. Now, if you notice right
6:43:52here, we have our legend up top. Um, it
6:43:55is possible to actually change that. You
6:43:57can go right here and you can move this
6:44:00um kind of wherever you want. Um, but
6:44:03it's not exactly easy to put based off
6:44:06how we have it right here. If we go in
6:44:08to this chart elements, we go down to
6:44:10legend and we hit this little arrow
6:44:12right here. We can select it on the
6:44:15right, the top, the left, and the
6:44:17bottom. Or we can just go to more
6:44:19options, uh, which allows us to push it
6:44:21anywhere. But, um, let's say I want to
6:44:24do it just like this. I'm going to put
6:44:25on the right. And I actually want to
6:44:27bring it down right here. And, you know,
6:44:31that's just an option if you want to
6:44:33kind of customize it a little further.
6:44:34Makes it a little cleaner. Uh, you can
6:44:36do that with almost any of these things.
6:44:37So, if you click on this, oops. If you
6:44:39click on this, you can move this
6:44:41anywhere as well. So, if you want to
6:44:43move this over here on top of it, you
6:44:44can and make it look terrible. or you
6:44:46can move it uh right back over here. You
6:44:48know, this is something that you can
6:44:50move around. Uh you just kind of want to
6:44:52make sure you're doing it the right way.
6:44:54So, let's get this back where it was.
6:44:56There we go. Now, before we go any
6:44:57further, let's copy that and put it
6:45:00right over here with our other uh charts
6:45:03and graphs. And if you see over here on
6:45:06this side, we have this format chart
6:45:08area. Notice I haven't showed you this
6:45:10at all yet. That is because I genuinely
6:45:12just don't use this almost at all. Um,
6:45:16there are some good stuff in here. Um,
6:45:18and I'm sure that, you know, if you are
6:45:20someone who really wants to go in there
6:45:21and super customize it, you can do that.
6:45:24Um, but I honestly I just never get in
6:45:26here and I never, you know, change the
6:45:28glow or the shadows. Um, just not
6:45:32something I use. And and some of these
6:45:33are only for these three 3D formatting,
6:45:35which I never use. And so, I'm not going
6:45:38to show you and walk through these
6:45:39things. Again, I I really don't use it.
6:45:41And so if you want to go in there and
6:45:43mess with it, uh, you know, by all
6:45:45means, go for it. It's just not
6:45:46something that I want to take the time
6:45:48to show you. And with that being said,
6:45:50let's go back over to this chart sheet
6:45:52that we have. And it was super super
6:45:55easy to get these um charts and graphs
6:45:59and and whatnot. There are lots of
6:46:01different options. Again, if we go back
6:46:03here and we go up here to chart design
6:46:05and go to the change chart type and
6:46:08again there are a ton of different
6:46:10options like a pie chart um like this.
6:46:13It's it's you know you can try to figure
6:46:16this out and use these. Um but you know
6:46:20I wanted to show you the ones that
6:46:21you'll probably use the most which are
6:46:22these columns and line charts. And they
6:46:25all kind of are similar in their own
6:46:28way. This bar chart is basically, you
6:46:30know, this column chart just on its
6:46:32side. And so they all have their
6:46:34different flavor. They all have their
6:46:35different way of visualizing the data,
6:46:37but in essence, they're using the data
6:46:39in a similar way to to visualize it and
6:46:41represent the data itself, especially
6:46:43things like these box and whisker plots
6:46:45or these waterfall charts. Uh, you know,
6:46:47these are things that usually require
6:46:50specific data to kind of use. Uh, and
6:46:52and so I'm just using data that you'll
6:46:54probably see the most of. um like this
6:46:57this sales data. So, I hope that this
6:46:59has given you a pretty good um you know
6:47:02quick understanding of how to use these,
6:47:04how to customize them, how to copy and
6:47:06paste them over to a different sheet to
6:47:09create some type of little uh chart and
6:47:12visualization sheet that you can use to
6:47:14show your employers and visualize the
6:47:16data that you are working with. Thank
6:47:18you guys so much for watching. I really
6:47:20appreciate it. Again, huge shout out to
6:47:21Udemy for sponsoring this Excel series.
6:47:23If you like this video, be sure to like
6:47:25and subscribe [music] below, and I'll
6:47:27see you in the next video.
6:47:36[music]
6:47:40What's going on everybody? Welcome back
6:47:41to the Excel tutorial series. Today we
6:47:43will be looking at how to clean data in
6:47:45Excel.
6:47:51Now, knowing how to clean data in Excel
6:47:53is actually extremely useful, and there
6:47:55are a ton of techniques to do this. I'm
6:47:57going to be showing you the ones that I
6:47:58probably use the most, and I feel like
6:48:00are the most helpful to kind of do the
6:48:02bulk or the majority of the data clean
6:48:04that you're going to do in Excel. Like I
6:48:06said, there's so many different ways and
6:48:08very specific things that you can do,
6:48:10but I'm going to highlight some of the
6:48:12bigger ones that I find the most useful.
6:48:13And some of you may be thinking, well,
6:48:15I'll just do my data cleaning in SQL or
6:48:17Python or when I get it ready to put it
6:48:18in Tableau. Um, but honestly, a lot of
6:48:21the data cleaning, at least a lot of the
6:48:23big stuff, I tend to do in Excel if the
6:48:25data set is small enough to fit in
6:48:27Excel. And so, I think it's actually
6:48:28really, really useful to know how to do
6:48:30this because you'll most likely be doing
6:48:32it more than you think. Now, before we
6:48:34jump into the tutorial, I want to give a
6:48:36shout out to the sponsor of this video
6:48:37and is a brand new sponsor. It is
6:48:39Unlocked by Z by HP. Unlocked is a movie
6:48:42that's actually broken up into four
6:48:43parts and each of them have a unique
6:48:45data science challenge associated with
6:48:47it. Now, I'm going to read this next
6:48:48part because it's extremely interesting.
6:48:50Each challenge represents a different
6:48:52topic. So, there's data visualization,
6:48:54text analysis, audio signal processing,
6:48:56and computer vision. And you can submit
6:48:58your answers and your work on their
6:48:59website for a chance to win one of 10
6:49:01ZBook Studio laptops or a free trip to
6:49:03the Kaggle World Championships. So, I'll
6:49:06leave a link in the description where
6:49:07you can go watch the movie and then do
6:49:08the challenges and then submit your
6:49:09answers for a chance to win. You should
6:49:11also go check out their hackathon where
6:49:12you can do these projects with other
6:49:14people just like you who are trying to
6:49:15figure out these answers and submit them
6:49:17to win as well. So, go check that out.
6:49:19Thank you again to the sponsor of this
6:49:21video, Unlocked by Z by HP. Now, without
6:49:24further ado, let's jump onto my screen
6:49:25and get started with the tutorial. All
6:49:27right, so let's jump right into it. I
6:49:28have this US president's data set. I got
6:49:30the base data set from Kaggle. Uh, but I
6:49:33added some of my own data and then I
6:49:35messed some stuff up as well just to
6:49:37kind of demonstrate some of these things
6:49:39that we're going to be looking at today.
6:49:40This is not a full project. So, you
6:49:43know, we're not actually going to be
6:49:44using this to create any visualizations
6:49:46or anything like that. So, you know, all
6:49:47this is just for demonstration purposes,
6:49:50but we will be doing a full project in
6:49:53about two or three videos uh in this
6:49:56Excel series where we're going to be
6:49:57doing from start to finish with a real
6:49:59data set. So, you know, if that's
6:50:00something that you're you're wanting,
6:50:02then we will absolutely be doing that.
6:50:04Now, something that you may be wondering
6:50:05is how do you actually identify what you
6:50:07need to clean in the data. What do you
6:50:09know to look for? Well, some of the
6:50:11obvious things are things like
6:50:12formatting and standardization. So,
6:50:15things like, you know, this James Monroe
6:50:16is in all caps. That happens all the
6:50:18time with real data. Um, and and so, you
6:50:21know, you want to standardize that or
6:50:23this all lowercase. You want to
6:50:24standardize that. You want that all to
6:50:25be the same. There's also things like um
6:50:29right here where we have this wig and
6:50:31this wig with a bunch of random stuff
6:50:33after it. This happens all the time
6:50:36where it's not completely standardized.
6:50:38Um and you may even notice um you know
6:50:41there are some spelling errors in here
6:50:43and I'll we'll kind of look through that
6:50:44in a little bit. And then you know there
6:50:47are things like additional spaces where
6:50:50there shouldn't be spaces. There are
6:50:51things like currencies that you need to
6:50:53be aware of if you were importing this
6:50:55into or going to be importing this into
6:50:56a SQL database. Um, things like
6:50:58currencies can be just a problem or be
6:51:03really um unnecessary. It may actually
6:51:06cause more issues in the long run. So,
6:51:07you may just want to, you know, take
6:51:09that to the base uh value. And then
6:51:12dates are always an issue. Always,
6:51:14always, always. Um, so always look at
6:51:16your dates. Make sure they're they're
6:51:17formatted correctly. Make sure they're
6:51:19all the same. These are the types of
6:51:21things that right when I glance at this
6:51:22data set, these are things that I'm
6:51:24looking for. Um, one other thing that is
6:51:27actually the first thing that we're
6:51:28going to start out with is you want to
6:51:30make sure that your data is
6:51:31[clears throat] not duplicated because
6:51:34if your data has duplicate data in it
6:51:36and you don't want that, it's not
6:51:38supposed to be there. There are some
6:51:40specific use cases where duplicated data
6:51:42is okay. Um, you know, you want to get
6:51:45rid of that and it's very easy to do in
6:51:47Excel. Uh the first thing we're going to
6:51:49do, we're going to go up uh to this data
6:51:51tab. We're going to go right over here
6:51:52and we're going to get see if there's
6:51:54any uh duplicates in our data. So, we're
6:51:56just going to go up to remove
6:51:57duplicates. It's going to automatically
6:51:59choose all of your columns to to check
6:52:02against. So, it's going to for from A
6:52:04all the way through I, it's going to see
6:52:06is the exact same data in all these
6:52:08rows. And if it is, it's going to get
6:52:09rid of it. Um and so, we're going to
6:52:11click okay.
6:52:13And it did find one duplicate. And I'll
6:52:15show you that one real quick. um because
6:52:17you know it was right here. So Barack
6:52:20Obama was here twice and then I'm going
6:52:22to hit control I hit control Z to go
6:52:24back. I'm going to hit control Y to go
6:52:26forward and it removed that uh that row
6:52:29completely. Now in this example you may
6:52:32be able to spot that with your eye but
6:52:33in a real data set where you have 10,000
6:52:36100,000 rows there's absolutely no way
6:52:38you're going to see that or very very
6:52:40unlikely that you are going to see that
6:52:42there's duplicated data in there. So
6:52:44just running a a a quick um ddup or or
6:52:47removing of duplicates that is really
6:52:49important to make sure that you um have
6:52:52gotten rid of those things. So that's
6:52:54one of the first things that I do. Um
6:52:56we're going to go into a lot of these
6:52:58different uh columns and I'm going to
6:53:00kind of show you different techniques or
6:53:01things that I do when I look at actual
6:53:04data. So I'm going to come right over
6:53:06here. I'm going to insert. And this is
6:53:08what I actually do. I I usually create a
6:53:10separate column especially when I'm
6:53:11working with this because I don't want
6:53:12to change this one. Um I don't want to
6:53:16go in here and you know say um equals
6:53:19upper equals proper etc. There's a lot
6:53:22of different ways that you can change um
6:53:23names or not a lot but the main ones
6:53:26that you can change names and all of
6:53:27them are completely okay. So for example
6:53:30I'm going to hit equal upper oops upper
6:53:33and I'm going to go like this and close
6:53:35my parenthesy. So, I selected this cell.
6:53:37I closed my parenthesy. I hit enter. It
6:53:40is and I'm going to hit um in the bottom
6:53:42right. I'm going to hit double click
6:53:43this. It's going to apply it to all of
6:53:45them. It is completely okay to have your
6:53:47data like this if you want it to be like
6:53:49that. Um if you want it to be all lower,
6:53:51you can do that. If you want it to be in
6:53:52proper case, you can do that. Um there
6:53:55are oops, there are different um uses
6:53:59for all of them. And honestly, as long
6:54:01as it's all the same, typically it's
6:54:03okay. But if um you know, for example,
6:54:05if you're selling this to like a
6:54:06thirdparty company or something like
6:54:08that, they may have um what they want
6:54:11for their ingestion process when they
6:54:13take your file in. If you send, you
6:54:15know, a weekly file or a monthly file,
6:54:17they may want it exactly how they want
6:54:19it, and you can change that to to what
6:54:21they want. Um but as long as it's
6:54:23standardized for you, it's all the same
6:54:25for you, that is a good thing. So now we
6:54:28have all of these um in the proper case.
6:54:31That's typically what I I do or I use
6:54:34upper. Those are the ones I use the
6:54:36most. I don't usually use um lower. And
6:54:39if you go in here and you type in lower,
6:54:42you know, it changes it to all lower. I
6:54:44don't typically do that. Um and I'm
6:54:46going to add I'm going to Oops. I'm
6:54:48going to say president dash fixed. And
6:54:52so now all of these names um all of
6:54:55these uh different uppercase and
6:54:57lowercase these are all fixed and and it
6:54:59just makes it so much easier to read and
6:55:02you don't have different um uppercase
6:55:04and lowerase issues. It's all the same.
6:55:06So I'm going to keep that right there.
6:55:08Uh if we move a little bit to the right,
6:55:13if you look at this prior, now this
6:55:15prior is a mess. It's it has stuff all
6:55:19over. And to be honest, this is not
6:55:21really something that I would probably
6:55:23be using um like in a real data set. I
6:55:27would look at this column and I would
6:55:28say this is pretty useless. Um if I had
6:55:30a very specific use case for this this
6:55:33data in this column, I might try to, you
6:55:35know, parse it out and do something. But
6:55:37I don't uh this this is a completely
6:55:39useless column to me. So I'm actually
6:55:40going to skip this one. I'm going to go
6:55:42to this party one. And this party one to
6:55:45me is looks pretty important because
6:55:47this is something that I know I can
6:55:48group by um and I can create
6:55:50visualizations with and and kind of
6:55:52break that out. And if you look right
6:55:55here, we're going to add um we're going
6:55:57to add a filter. So now let's open up
6:55:59party and take a look. So, uh, if we
6:56:02look right here, we have Democratic,
6:56:04Democratic-Republican,
6:56:05Federalist, nonpartisan, Republican,
6:56:07Republicans, wig, and wig with a a date
6:56:10and some information in the back of it,
6:56:12and then some blanks. Um, and it's
6:56:15really important when we're when we're
6:56:17looking at these um ones that we think
6:56:18we might group by that we have these um
6:56:22properly grouped. So, Republican and
6:56:24Republicans to me right off the bat
6:56:26looks like a spelling error. And so, I'm
6:56:28just going to deselect all. I'm going to
6:56:30go to Republican. Republicans.
6:56:33And it's literally Republican all the
6:56:36way down except for this last one. And
6:56:38to me, that's just something that I
6:56:40would update. So I would just go right
6:56:41here. I do that. If I didn't do that and
6:56:44then I try to create, let's say, a pivot
6:56:46table on here, I'll have its own group
6:56:48of Republicans, and it wouldn't be added
6:56:50to Republican. And maybe that's on
6:56:52purpose, but let's just presume that we
6:56:55know this data extremely well. That's
6:56:56not supposed to be like that, right?
6:56:58Again, that that just comes back to
6:56:59knowing your data really well,
6:57:02understanding what it um you know what
6:57:04it should look like, and we know that it
6:57:05should not be like that. So, we're going
6:57:07to fix that. Uh the next thing that
6:57:09we're going to fix um and as you can
6:57:10see, it it got rid of it. Next thing
6:57:12we're going to fix is this wig. Um
6:57:16that's just like an error. That's that's
6:57:18some issue on the the data side, and
6:57:23we're just going to [clears throat] fix
6:57:23that by updating it. And that's it. I
6:57:27would always be keeping um a a copy of
6:57:30this with the raw data uh somewhere else
6:57:33because this is presumably like a
6:57:35working document. This is not a um you
6:57:39know you you aren't saving over your
6:57:41original file. Let's just say that. And
6:57:43then let's take a look at these blanks
6:57:44real quick. Um okay. So there are these
6:57:49rows right here that have nothing. I I
6:57:51think we're okay. But if we see anything
6:57:53different 47 48. Okay. So, yeah, it's
6:57:56just these ones right here that have no
6:57:58data in it anyways. It's just seeing it
6:58:00in the filter. So, not an issue at all.
6:58:03So, okay, we're looking good. We've gone
6:58:06all the way over. We we fixed this
6:58:07president. We skipped this one. Um we we
6:58:10cleaned up this party. And I kept this
6:58:12one in here because I'm not exactly sure
6:58:14if that's a Democratic or Republican.
6:58:16So, I'm going to keep it its own thing.
6:58:18Um I'm not a huge uh history buff on
6:58:22that aspect. The next one right here is
6:58:25um the next one right here is really
6:58:28easy. Uh this is something that happens
6:58:30all the time especially on actually uh
6:58:33most often it's happens on numerical
6:58:35data. So like uh you know there'll be a
6:58:39number of 101 then there'll be a space
6:58:41after it for absolutely no reason. Uh
6:58:43and it happens all the time. It does
6:58:46happen like this as well um where you'll
6:58:48see this and all you got to do is do
6:58:50trim and select the uh the cell. We're
6:58:53going to close that parenthesy and we're
6:58:55going to apply that all the way down.
6:58:57What is so fantastic about the trim is
6:58:59that it's really intuitive and it knows
6:59:02basically everything it needs to do. For
6:59:05example, um it gets rid of the um spaces
6:59:09before. It gets rid of extra spaces in
6:59:11the middle and um it'll get rid of extra
6:59:15spaces at the end um which you wouldn't
6:59:17be able to see but they are there and
6:59:19they they absolutely can cause issues.
6:59:21If you have spaces at the end that you
6:59:23cannot see um let's take this one for
6:59:25example like if I had spaces at the end
6:59:27that can cause issues when you insert or
6:59:30or or put that into a database. Um that
6:59:32happens a lot with numbers. um you know
6:59:35when you're putting that into SQL that
6:59:37can cause issues and so you really it is
6:59:39important to actually do that trim um
6:59:41and you can do that on all of your
6:59:43columns or just ones that you know
6:59:45you're having issues with but once you
6:59:47import that data into SQL you will know
6:59:48if there's an issue or not um when you
6:59:50actually try to start using it. So we're
6:59:52going to say vice and we're going to say
6:59:55fixed. Oops. There we go.
6:59:58Uh this next one is one that you'll run
7:00:02into a lot when you're working with
7:00:03numerical data. You will encounter so
7:00:07many different issues. Um one that I run
7:00:10into a lot is I I've worked with a lot
7:00:12of cost data or pricing data. And when
7:00:15it's in an Excel, it sometimes comes in
7:00:18with um these currencies like a dollar
7:00:20sign, a pound sign, things like that.
7:00:23And when you put that into SQL, it just
7:00:27is a nuisance, right? You're not going
7:00:29to be able to run um it's going to go in
7:00:33as a text or it's going to be like a
7:00:35string, right? Because it has that
7:00:37special character and you don't want
7:00:38that. You don't want to have to then go
7:00:40in and then change things around. You
7:00:42just want to be able to start um you
7:00:44know doing calculations on those
7:00:46numbers. So, what you can do is
7:00:48sometimes it'll come in as a text.
7:00:50Sometimes it'll come in as um a
7:00:52currency, which I think this one's a
7:00:54currency. We are just going to change
7:00:55that to be a number. And then we're
7:00:58going to get rid of these. Oops.
7:01:02And get rid of those. That it doesn't
7:01:05look as pretty, but that is much more
7:01:07useful than actually having the currency
7:01:10on there um with the decimals. This
7:01:12actually is so much easier when you when
7:01:14you want to use it for almost anything
7:01:16because you're able to add and uh do
7:01:19things properly in other systems. In
7:01:21Excel, I think it does understand it. Um
7:01:23but you know that can cause issues. So
7:01:26there is how you do that. The next thing
7:01:29that we're going to look at is these
7:01:30dates. And just notoriously whenever I
7:01:33see a date field, I know there's going
7:01:34to be an issue with it. It's very rare
7:01:37that I get a date field that is perfect.
7:01:40uh it just it it is genuinely is um is a
7:01:44novelty when that happens and most of
7:01:47the time it has to do with um let's say
7:01:49a date comes into Excel and it's in a
7:01:52text format or a date comes into Excel
7:01:53and they're not the same. In this
7:01:55example they are not the same um and we
7:01:58just want them to all be similar. They
7:02:00say date on if you look right here it
7:02:02says date. It says date. It looks like
7:02:05it should be the same. Um, but if we go
7:02:09like this, it all looks the same, right?
7:02:12There's no issues at all. If we were to
7:02:16um try to use that, it may or may not be
7:02:19an issue, but we don't want to leave
7:02:21that to chance later on if you're using
7:02:22this with Python or something like that,
7:02:24it can cause issues. Uh, maybe not in
7:02:26SQL because it may um see the underlying
7:02:29um what's in the underlying cell, not
7:02:31just what we see, but some systems
7:02:34won't. And so, you want to make sure
7:02:35that they're all the same. And so you
7:02:37know what we were doing back here with
7:02:39um oops with a party and we were looking
7:02:42at this uh this filter and identifying
7:02:44the issues. I usually do that on date
7:02:47fields as well. And and oftent times um
7:02:49you know just for just for demonstration
7:02:51purposes oftent times I will get
7:02:54something like that and then I'll come
7:02:56up here and I'll notice that there's
7:02:58this one random number that happens all
7:03:01the time. All the time. Um, and so, you
7:03:04know, you want to make sure that you um
7:03:07that you look at these things and just
7:03:09just do at least a quick glance, if not
7:03:12kind of doing a kind of a deep dive into
7:03:14it. But all we're going to do is we're
7:03:16going to do both of these and we're
7:03:18going to do a short date and let's take
7:03:20a look and see if that fixed it. And so
7:03:22now that they are all the same format
7:03:24and that is fantastic. That is exactly
7:03:26what we want. Uh we're going to go back
7:03:28through here. We're going to get rid of
7:03:31these. Um, again, this is a working um
7:03:36this is a working document. Oops. Uh, we
7:03:39need to we're I'm going to do um control
7:03:42shift down. Oops. Let me go back up. Do
7:03:46control shift down and copy. And what
7:03:49I'm going to do right now is I'm
7:03:50actually going to copy. And let me do it
7:03:53right here. I'll show you. Sometimes I
7:03:54do this. Doesn't just depends. I'm going
7:03:56to go right here. I'm going to hit
7:03:58rightclick. And I'm going to paste as a
7:04:01value, which means it's not going to
7:04:03take the um calculation or the formula
7:04:05that I just did. Uh it's going to
7:04:07actually paste it as that value. So, we
7:04:09just replaced it. Um right here, you can
7:04:12see up here it says equals trim of G2.
7:04:15This now, now that I copied and pasted
7:04:17it over as a value, um it got rid of
7:04:22that um calculation and now it is
7:04:24actually a string. So, we don't need
7:04:26this anymore. And I'll do the same thing
7:04:29over here as well.
7:04:31I'm going to controll shift down copy
7:04:37and I just hit the right key uh or the
7:04:39left key sorry. Now I'm going to
7:04:41rightclick and I'm going to do paste as
7:04:44a value. And again has this proper and
7:04:47now it doesn't have the proper. It's
7:04:49actually the value that was here. So
7:04:51that's really important to note. Uh and
7:04:53we're going to get rid of that one. And
7:04:55so now what we have is is already
7:04:58looking much better. Now one of the last
7:05:00things I want to look at is deleting
7:05:01columns that we are not going to use.
7:05:03And this is why it's so important to
7:05:05keep a backup or or the raw data not in
7:05:08this file because if you start saving
7:05:10over this file and this is your raw file
7:05:12uh that can mess up a lot of things and
7:05:14that happened to me before and it's
7:05:16terrible and then you have to request
7:05:18another file or you have to go back and
7:05:20find it or something like that. It's
7:05:21terrible. Um, so, so this is our working
7:05:24document. So, we can mess with this and
7:05:26do whatever we want for our purposes.
7:05:28Now, for us, um, I can already tell you
7:05:31that this prior is a bunch of nonsense.
7:05:33And we do not need it. We're not going
7:05:35to use it for anything. And it and if we
7:05:37have, um, this is a small very small
7:05:39data set. This only has like um, let's
7:05:41say, you know, one, two, three, four,
7:05:44five, six, seven, eight. We have like
7:05:46eight columns that we're, you know, kind
7:05:47of using that has data. Eight or nine.
7:05:50Now, that's a small data set. I've had
7:05:52ones with literally like hundreds um and
7:05:55and it has so many columns uh so much
7:05:58data and sometimes it's good to just
7:06:00trim it back to the things you know
7:06:02you're going to use. This to me is
7:06:03absolutely useless. Um we're going to
7:06:05delete that. And then right over here,
7:06:07it's pretty redundant. Um it's just one
7:06:10number off. But if we scroll down just a
7:06:12little bit, um it goes it's basically
7:06:15just counts. It's a I you could even
7:06:17call it a unique um identifier if you
7:06:20want. Sure, why not? But we don't need
7:06:22both. Um, so we're going to get rid of
7:06:23this first one. And now we have more of
7:06:25the useful and relevant data rather than
7:06:27the stuff that we absolutely know that
7:06:29we are not going to use. Um, these date
7:06:31updateds and date created, we may never
7:06:33use them, but we might. Um, so it's it
7:06:36doesn't hurt to keep it on hand. Those
7:06:38other ones are ones that we are almost
7:06:39certain we will never use again. Keep a
7:06:42backup just in case you need it. You can
7:06:44always go back and get it. So, you know,
7:06:47if you go back to what we started with
7:06:48and you look at what we have now, it is
7:06:50much cleaner. It's much more usable. And
7:06:53these are small, subtle changes. Um,
7:06:55especially with this very small data set
7:06:57of only like 50 rows or or 46 rows. But
7:07:00you're going to be working with data
7:07:01sets that are thousands, tens of
7:07:03thousands, hundreds of thousands of
7:07:05rows. And you need to know how to kind
7:07:07of look at this data, standardize it,
7:07:09um, format it properly for what you're
7:07:11going to be using it for. If you're
7:07:13keeping it in Excel, there are different
7:07:15things that you may do than if you're
7:07:16putting it into a database or going to
7:07:18be using it in, you know, um using
7:07:22Python to to access it. So, you need to
7:07:25kind of know your use case. But these
7:07:27are some things that I do all the time
7:07:30to kind of clean up the data before I
7:07:32use it for something. Whether I'm
7:07:33creating pivot tables or I'm inserting
7:07:35it into or I'm putting it into SQL,
7:07:38these are things I do all the time. And
7:07:39so hopefully that helps give you kind of
7:07:41an idea of some of the things that you
7:07:43should be looking for when you're
7:07:45actually cleaning data. And it's really
7:07:46important to understand why you're
7:07:48actually making these changes and the
7:07:50reason you're making these changes
7:07:51because some of the things that I did
7:07:52today may not be things you want to do
7:07:54on a different data set that has
7:07:56different uses and different um purposes
7:07:58for. So, you know, take everything that
7:08:00I've said and and apply it um with a
7:08:03little grain of salt to your data set
7:08:05because your specific needs may be
7:08:06different than what I wanted when I was
7:08:09cleaning my data set. So, I hope this
7:08:11was helpful. I hope you this gave you a
7:08:13small glimpse of some of the things that
7:08:14I'm looking for when I clean a data set
7:08:16or I get a new data set in and I'm kind
7:08:18of, you know, analyzing it, figuring out
7:08:20what I need to fix in it. I hope this
7:08:22has been helpful. Uh with that being
7:08:24said, thank you so much for watching. I
7:08:26really appreciate it. If you like this
7:08:28video, be sure to like and subscribe
7:08:29below. And I'll see you in the next
7:08:31video.
7:08:43What's going on everybody? Welcome back
7:08:45to the Excel tutorial series. Today
7:08:47we're going to create an entire project
7:08:48in Excel.
7:08:51[music]
7:08:54Now, if you've never done a complete
7:08:56project in Excel where you take the
7:08:58data, you clean it, and then you create
7:09:00an actual dashboard where people can
7:09:01click on things and filter things, this
7:09:04is going to be a really great learning
7:09:05opportunity, as well as potentially, you
7:09:07know, a simple project that you can use
7:09:09for your portfolio. Or you can spice
7:09:10things up and go a little farther than
7:09:12what we're going to be doing in today's
7:09:13video. I will walk you through every
7:09:15single step of the way, and hopefully we
7:09:16learn something together. And without
7:09:18further ado, let's jump right into it.
7:09:20Let's jump onto my screen and get
7:09:21started with the project. All right, so
7:09:23this is the data set that we're going to
7:09:25be working with. I will leave a link in
7:09:26the description to my GitHub where you
7:09:28can go and download it so you can be
7:09:29working with the exact same data set
7:09:30that I am using. Now, before we actually
7:09:33get into this data and start looking at
7:09:34it, I'm going to show you what the final
7:09:36dashboard is going to look like. Um,
7:09:38we're going to create a few different
7:09:39types of visualizations. Nothing too
7:09:41crazy. Um, and then we'll create some
7:09:43filters as well, so we can kind of, you
7:09:45know, create some interactive filters
7:09:47with our data. So, let's go right on
7:09:49over to our data set. Now, I'm going to
7:09:53hide this because we are not going to
7:09:55use that. But what I am going to do
7:09:56before we do anything is I'm going to
7:09:58create a dashboard
7:10:01and I'm going to create a pivot table.
7:10:04Oops.
7:10:06And I'm going to create a
7:10:09working sheet. So, um, all these things
7:10:13have different uses and I'll explain
7:10:16that as we go along. So, this is our
7:10:18data set. Um, I'm going to copy this
7:10:21over to our working sheet. When I go
7:10:23into, you know, an Excel and I'm working
7:10:26on something, I don't like to, you know,
7:10:28use just the one that I was using in
7:10:30case I mess something up and it saves
7:10:31over it or some issue. I like to create
7:10:33a working sheet and keep the raw data
7:10:35right over here. It just makes my life
7:10:37easier. I don't have to save it and
7:10:38then, you know, open up a different
7:10:40Excel to compare them. So, we have our
7:10:42bike buyers. This is our working sheets.
7:10:44This is our raw data. This is the one
7:10:45we're actually be working on today. So,
7:10:48let's um let's start looking at it
7:10:50really quick and just kind of glance and
7:10:51see what data we're working with and
7:10:54then we'll start cleaning it up, making
7:10:56it more useful for what we are going to
7:10:57be using it for and then we'll start
7:11:00building out the dashboard. So, right
7:11:03here we have an ID that should be a
7:11:06unique ID to each person. Uh this is
7:11:08their marital status, so married or
7:11:10single. This is their gender, male,
7:11:13female. We have their income, children,
7:11:16their education, their occupation, do
7:11:18they own a home, how many cars they own,
7:11:21how long their commute is, the region
7:11:24where they live, their age, and if they
7:11:26purchased a bike. And this column right
7:11:28here is extremely important. This is
7:11:30going to tell us whether they did or did
7:11:31not buy a bike. So, we got their
7:11:33information. They're looking for a bike,
7:11:35but they either decided not to buy a
7:11:36bike or they did buy a bike. And we're
7:11:38going to be using that one a lot in in
7:11:40this video. And so um you know this is
7:11:44basically the data set that we're
7:11:46working with um some of the demographics
7:11:48and and information behind the person.
7:11:51So what we want to do when we are
7:11:53cleaning the data before we do anything
7:11:56uh I like to see if there are any
7:11:57duplicates in here um what we're going
7:11:59to do is come right up here. We can go
7:12:02to uh
7:12:05where is it? Right here we got remove
7:12:07duplicates. So we're going to click on
7:12:08that. It selects every single one. We
7:12:11just want to see if there's any useless
7:12:13duplicated data that we do not need. Uh,
7:12:16and the data is a header. So, we're
7:12:17going to click okay.
7:12:19All right. So, we had a ton of
7:12:20duplicates in there. Uh, for whatever
7:12:22reason. So, we do have duplicates in
7:12:24there. So, I'm glad we did that.
7:12:25Otherwise, we would have uh, you know,
7:12:28not good data. We don't want that. Let's
7:12:32start right over here. Um, the ID, of
7:12:34course, we're not going to change. The
7:12:35marital status and gender are M's, S's,
7:12:39Fs, and M's. Um, this isn't inherently a
7:12:43bad thing to have it like this, but, you
7:12:45know, we have to think about it from the
7:12:46perspective of someone who's going to be
7:12:48using this dashboard. Do they know what
7:12:50M and S is? Do they know what M uh and F
7:12:53is? And if they don't, it's better to
7:12:55just spell it out for the most part. Um,
7:12:58so let's just do that. So, we're going
7:13:00to click on the column B. We're going to
7:13:02hit CtrlH. That's going to bring up our
7:13:04find and replace. Now, there's an M in
7:13:07both of these columns and there's
7:13:09different things. One is married and one
7:13:11means male. So, what we're going to do
7:13:13is we're going to search by columns. Um,
7:13:17and we'll have match case. I don't think
7:13:18that's going to change anything, but
7:13:19that just means an exact match. Uh, and
7:13:21we're going to do M equals and we're
7:13:24going to replace it with married. And
7:13:26we'll replace all. Awesome. And then
7:13:29we'll do S is single. This one is super
7:13:33easy. We're going to do the exact same
7:13:35thing right here. So, column C and hit
7:13:38control H. We'll do still has by column.
7:13:41So, we'll do M is male.
7:13:46We'll replace all of those and F is
7:13:50female. And replace all those. That's
7:13:53great.
7:13:55Uh, you know, the next column right here
7:13:57is income. And in a se in a previous
7:14:00video, I talked about how I don't
7:14:01typically like it in this format. And
7:14:03that's true. Um, if you're doing
7:14:05calculations on it or or any other
7:14:07thing, it can mess it up sometimes
7:14:08having the dollar sign or it being a
7:14:10currency. We're not really going to mess
7:14:13with it too much right now. Um, what we
7:14:15can do is just kind of we'll make sure
7:14:19all of it's currency. Um, we'll just go
7:14:21like that to make it a little simpler,
7:14:23but we're not going to change it to like
7:14:25a numeric. Um, we will use this in the
7:14:29visualization. We'll see how it looks
7:14:30and if we need to, we'll come back and
7:14:32change it. If not, we'll keep it how it
7:14:34is. Um, so that's all we're going to do
7:14:36to that one. Uh, the children, those
7:14:39look good. We have education,
7:14:42partial college, partial high school.
7:14:44This looks fine to me. Um, if there's
7:14:46any spelling errors or anything like
7:14:48that, of course, we need to clean that
7:14:49up. It doesn't look like there is.
7:14:51Occupation,
7:14:54skilled manual, manual. Okay, those
7:14:56should be separate. Are they a
7:14:58homeowner?
7:14:59Should just be yes or no. All right, we
7:15:03have cars. 1 2 3 4. Good night. Who owns
7:15:06four cars? Um, and then that we have the
7:15:07commute distance. Uh, and you know,
7:15:09there's nothing terrible about this.
7:15:11It's giving you ranges. Um, which can be
7:15:13a good thing. I say let's keep it for
7:15:17now, but I have a feeling when we get
7:15:19further and we start using it in the
7:15:20visualization, we may want to change
7:15:22this. So, let's just hold off for now.
7:15:24Um, but if needed, we will come back to
7:15:27this and we will change this. Um, and
7:15:30then we have our region and that looks
7:15:33totally fine. And we have our age. Now,
7:15:36when you're using ages, typically you
7:15:38have some type of like age bracket or or
7:15:41age range. And you do that because there
7:15:44are so many ages in here, right? It's 25
7:15:47all the way down to 89. And if you're
7:15:49using that on some type of
7:15:50visualization, it could just get really
7:15:52messy. And so you'll create kind of, you
7:15:54know, just brackets around these so that
7:15:57you can kind of condense it and make it
7:15:59a little bit easier to understand. So
7:16:02let's do that and just create a new
7:16:04column and then we can use that for our
7:16:07dashboard. So let's go right up here.
7:16:08We're just going to create a new column.
7:16:11Uh we'll call this age brackets.
7:16:15And what we can do is we can use an if
7:16:18statement to kind of say if it's older
7:16:22than or less than and and and kind of
7:16:24give them these ranges. Um that's one
7:16:27way to do it and that's the way we're
7:16:28going to do it right now. So let's go up
7:16:31here and what we want to do is we want
7:16:34to say is going to we're going to say
7:16:36equals and we're going to do if and
7:16:38we're going to close that parenthesis.
7:16:40Now, what we're going to say is if this,
7:16:44we'll go right back up here. If this is
7:16:47less than, so we're going to do this 31
7:16:51and we're going to say comma. So, if
7:16:53they are less than 31, what do we want
7:16:56to call them? What do we want their
7:16:58their, you know, name to be? We'll call
7:17:02them adolescent. Oops, that's not how
7:17:06you spell adolescent. Adolescent. Um,
7:17:09and then if they're not, what we're
7:17:10going to do is we're going to say it's
7:17:12invalid.
7:17:15Okay. And let's just see if this one
7:17:16works first.
7:17:18All right. It's not working at all. Um,
7:17:21okay. So, basically what we did was um
7:17:24incorrect. We did it backward. Uh, we
7:17:26want to do I said, uh, L2 is greater
7:17:29than 31. No, we want to do like this.
7:17:32So, let's do that now.
7:17:35All right. and it should pull up where
7:17:37if they're under the age of 31. So if
7:17:40they're 30 or below is basically what
7:17:42it's saying. So if they're 31 they'll be
7:17:45invalid, but if they're 30 or below it's
7:17:47adolescent. So it is working properly.
7:17:50Um and let's see what it see what it
7:17:52says. Perfect. So this one is working
7:17:54and and now what we want to do is we
7:17:56actually want to build on this and make
7:17:58it uh kind of like a nested if statement
7:18:01if you've ever heard of that or done
7:18:03that before. So this is our first if
7:18:05statement and this is going to be this
7:18:08is invalid. This is our value if false
7:18:10statement. This whole statement is going
7:18:13to become our value if false for a
7:18:16different if statement. Um so let let me
7:18:20write it out and hopefully that'll make
7:18:22sense. But we're going to say if do open
7:18:25parenthesis and we're going to do it
7:18:26like this. And let's just get rid of
7:18:27this for a second.
7:18:31All right. Uh what did I do? And let me
7:18:34do oops, give me a second.
7:18:39Okay, we have our if. Let me just write
7:18:41that out again. We have our if. There we
7:18:44go. So now what we're going to do is
7:18:46we're going to write basically the next
7:18:49part of it. So we're going to say if
7:18:51that L2 is and we're going to do this
7:18:54time we're going to do greater than or
7:18:55equal to 31. So now it's going to
7:18:58include that 31. So right here we did
7:19:00anything less than 31. So is 30 and
7:19:03below. This one is going to be 31 and
7:19:06above. So we're gonna say these people
7:19:08are middle age.
7:19:12And if not, then it's going to go to
7:19:15this if statement. And then we need to
7:19:17close it, I believe. So now let's try
7:19:18this.
7:19:20All right.
7:19:22Fantastic. Now if um everybody should be
7:19:25in one of these areas, right? everyone
7:19:27should either be an adolescent or
7:19:29middle-ag because basically all we're
7:19:30saying is is if they're older than 31 or
7:19:3330 or below. That's all these two
7:19:35statements do. So we have um you know
7:19:38our next group. Now we can add and go
7:19:41even further into this and now we can
7:19:44use this entire thing as the um what was
7:19:47it called? The value if false uh
7:19:50section. So that's what we're going to
7:19:52do. We're going to do one more. So we're
7:19:53going to have three different
7:19:54categories. So, we're going to say if
7:19:56and do an open parenthesis. And we're
7:19:59going to say if Oh, actually, let's do
7:20:01it. Um,
7:20:03let's not do it to this one. Let's do it
7:20:06to this top one just easier.
7:20:09Uh, so we're going to say if open
7:20:11parenthesis, we're going to say L2. And
7:20:15this time we're going to say anybody
7:20:17over the age of 50. Uh, or we can do 55.
7:20:21Let's do 55. So, we do 55. and we're
7:20:24going to call them old and we'll do
7:20:28comma and this is the value if statement
7:20:31and we need to close the parenthesis. So
7:20:33let's try this. Anybody over the age of
7:20:3655 should have old. Um you know maybe
7:20:39we'll do 54. So anybody who is 55 is
7:20:43considered old. I think that's fair. I
7:20:45think that's fair guys. Oops. I should
7:20:47have done I should have done that to
7:20:49this one. Let me get out of this and
7:20:51we'll do 54. Uh my dad is 55. That's why
7:20:56I'm doing it like this. This is for you,
7:20:57Dad. Uh because he should be in this old
7:21:00category to be fair. So now we have
7:21:02adolescent, adolescent, middle-ag, and
7:21:05old. These are three categories. So we
7:21:07can now have these buckets, these
7:21:09different groups of ages, and it's much
7:21:12more usable than these individual ages.
7:21:14Um and so we will be using this in our
7:21:17in our dashboard for sure. Now, our next
7:21:19one is the purchased bike. Uh, and we're
7:21:22not going to do anything with that. So,
7:21:24uh, you know, that is that is that one.
7:21:27And, you know, there wasn't a ton to
7:21:30clean up here. We removed some
7:21:31duplicates. Um, I don't know why it says
7:21:34that. What did I do? Married.
7:21:38Married. What does this mean even mean?
7:21:42I don't Did I write that? Did I mess
7:21:43this up, guys? Oh,
7:21:47when I did the M and the S uh
7:21:51replacement in there, it replaced it
7:21:53with married and single. It's supposed
7:21:55to say marital status. Oops.
7:21:59Thanks for catching that, guys. Thanks
7:22:00for catching that. I hope that's how you
7:22:02spell marital. Uh we'll see. So, uh we
7:22:06are going to keep it just like this. Now
7:22:08what we are going to now
7:22:12now what we are going to do is build
7:22:15pivot tables with this data. So we had
7:22:17our raw data, we have our working sheet
7:22:20and now we want to create pivot tables.
7:22:22And pivot tables is how you actually
7:22:24help build your dashboards or help build
7:22:26your visualizations. So we're going to
7:22:28go right here. We're going to hit
7:22:30Whoops. Get rid of that. We're going to
7:22:33go right here. We're going to insert and
7:22:35we're going to say pivot table. And it's
7:22:37going to ask us what range.
7:22:40So we're going to go back to the working
7:22:41sheet and we'll just click here and hit
7:22:43control A.
7:22:46This is going to select all of our data
7:22:48for us. So it's really easy. And we're
7:22:51going to hit okay. And so now we have
7:22:54all of our
7:22:56uh pivot I don't need I don't need to
7:22:57pull it out that far. That was way too
7:22:58far. And now we have all of our pivot
7:23:00table information over here. And so that
7:23:03should make it really easy to you know
7:23:05actually build out. So what we're going
7:23:07to do is start selecting what columns
7:23:09and what data we actually want to work
7:23:11with. So the first one that we are going
7:23:12to build out is a dashboard that is
7:23:15basically looking at the average income
7:23:17of somebody who either bought or did not
7:23:19buy a bike. So we need in this one we're
7:23:24going to need their income. That's
7:23:25definitely going to be a value right
7:23:26here. Um but we want to break it out by
7:23:30male and female. So let's look at their
7:23:32gender. We're going to pull that down
7:23:34into the rows. So, um, this is basically
7:23:36a sum and no, let's look at
7:23:40let's make this an average. So, I just
7:23:41went to the, um, I clicked right here. I
7:23:44went to the value field settings and
7:23:46we're just going to do an average.
7:23:49All right. And then we are going to make
7:23:51these, um, and as you can see, there's
7:23:55four decimal points. Um, we'll keep it
7:23:57as is right now, but we may need to go
7:23:58back and change something. Then we're
7:24:00going to look at if they purchased a
7:24:01bike or not, and we're going to put that
7:24:03right here.
7:24:05So we can see that uh right here for the
7:24:08people who did not buy a bike the
7:24:10females their their average salary was
7:24:1253,000 the average salary for the
7:24:15average salary for males was 56,000 for
7:24:17yes the ones who did buy a bike the
7:24:20average salary was 55 for female and 60
7:24:23for males. So the people who had a
7:24:25little bit more money are buying bikes
7:24:27and you can also see that uh the men are
7:24:29making more money in this data set just
7:24:31overall in general. Um, so
7:24:35let's make the visualization really
7:24:37quick, but you know, I don't know. I'm
7:24:39not a huge fan of these decimal points.
7:24:41And maybe we can just change that in the
7:24:42visualization. We'll see. Um,
7:24:46oops. That's not what I meant to do.
7:24:49Um, let's do that. So, what we are going
7:24:53to do is we're going to click into here.
7:24:54We're going to click insert and we're
7:24:56going to go to these recommended charts.
7:24:58And it's going to bring up basically
7:25:00every single type that we would want.
7:25:02Um, and we can just click in here and
7:25:04see which one looks good. Uh, oh yeah, I
7:25:07love those 3D ones. Those are my
7:25:09favorite. You guys know that. Uh, let's
7:25:11let's use this one right here. Pretty
7:25:13simple. Um, whoops. Let's pull this
7:25:15right over here. And as is, it looks
7:25:19pretty good. Um, you know, it shows
7:25:22male, female. We have the average or the
7:25:25incomes right here, whether they did or
7:25:27did not purchase it. Um, and so at a
7:25:30glance, it's pretty easy to see. Let's
7:25:32see if there's anything. Um, you know,
7:25:36if you want to change up style-wise, go
7:25:38for it. I'm just going to keep it as is.
7:25:40Um, but let's see if there's anything we
7:25:42need to add, right? Do we want to add
7:25:43these access titles? Uh, for the most
7:25:46part, I I tend to do that. Um, it makes
7:25:50it pretty easy to see. So, we can go in
7:25:52here and we can just click it like this
7:25:54and we'll say income
7:25:57and we'll say
7:26:00we'll do gender. So, that's what that
7:26:03is.
7:26:05And let's go back in here. Do we want to
7:26:07add a chart title? We definitely want to
7:26:10add a chart title. Uh, for most of
7:26:11these, we'll add a chart title for sure.
7:26:13So, we'll say average income
7:26:16per purchase.
7:26:18Um, I don't know if that's 100% right,
7:26:20but we'll we'll we'll use it. Uh, if we
7:26:22need to change it to be, you know, by
7:26:24gender or something, we can. But, um,
7:26:26for now, let's see. Do we want to add
7:26:28data labels? Uh, definitely not. Uh, a
7:26:31data table. Um, we can do this. It may
7:26:34make it a little easier to read. I will
7:26:36say that again, these numbers are just
7:26:38these decimal points are really throwing
7:26:39me off. Let's go see if um, we can
7:26:41change it in here. Let's go to
7:26:45see if we can just make these numbers.
7:26:47Okay. And um we can keep it like that or
7:26:51we can even do something like this. Add
7:26:54commas.
7:26:56Yeah, I'm going to keep it just like
7:26:57this. I I think this just looks the
7:26:59best. Um again, I'm I'm getting adding
7:27:01commas here. I'm changing the um decimal
7:27:04place right here. It just makes it look
7:27:07a little nicer, a little cleaner. Um so,
7:27:10let's keep this exactly how it is. um we
7:27:15can always change things if we want to
7:27:17uh if we want to come back to it. So
7:27:19that we created our pivot table and then
7:27:20we created our visualization. Basically
7:27:23exactly what we're going to do for all
7:27:24of these because again all of these need
7:27:27um you know all of these need pivot
7:27:29tables in order to create the
7:27:30visualization. So let's um get out of
7:27:32here. We are going to scroll down and
7:27:35we're going to create our next pivot
7:27:37table. And once we get done with all of
7:27:39the pivot tables that we need or all the
7:27:41visualizations that we need, then we
7:27:42will um we will start. So we're going to
7:27:46do control A
7:27:48do okay and basically do the exact same
7:27:50thing that we did. Um this time we're
7:27:52going to look at the distance. So for
7:27:54this one I wanted to see you know I try
7:27:57to you know I created this already. I've
7:27:58already done this entire project through
7:28:00but I haven't really talked about why or
7:28:02what we're going to look at for this
7:28:04one. you know, we're looking at is their
7:28:08income, does it change whether they
7:28:10bought or didn't buy one? Um, so if they
7:28:12said yes, you know, is there a reason?
7:28:15Are they making more money? Is, you
7:28:16know, our price points are the
7:28:18customers, do they make more money, so
7:28:20should we cater to them or not? Uh,
7:28:22that's a good question. Uh, another
7:28:24thing is, you know, we're we sell bikes
7:28:26or this person sells bikes. So,
7:28:28commuting distance definitely makes a
7:28:30difference. you know, does the person
7:28:32who is buying a bike live one mile away
7:28:35from where they work or 20 miles away?
7:28:37Uh, this will help us determine, this
7:28:39next visualization will help us
7:28:40determine, you know, who who is doing
7:28:42that or who's buying it. So, what we're
7:28:45going to do is we are going to look at
7:28:48the um that one that we were looking at
7:28:52earlier, the commute distance. So, we're
7:28:54going to bring that right over here. So,
7:28:55we have these, you know, one mile, 10
7:28:58mile, 1.2, etc.
7:29:01Now we are going to uh again we're going
7:29:03to look at if they purchased a bike.
7:29:05That's really important. And let's make
7:29:08that the column as well. So now what we
7:29:10have is a count of these nos and yeses
7:29:12whether they did or did not buy a bike.
7:29:14Um one of the issues I already see and
7:29:17we'll I'm going to visualize it and then
7:29:18I'll show you this 10 miles you know
7:29:21it's right next to the 0.1. So it's not
7:29:23an order. Um and that could be that
7:29:27could be an issue. Um, so we may have to
7:29:30revise that somehow to put it at the
7:29:32very bottom because we can either do
7:29:34ascending or descending. Uh, either one
7:29:38I don't think is going to work. So, we
7:29:40may have to work through that in just a
7:29:41second. Um, don't know if I did that in
7:29:43my plan for that. Um, yeah. So, it has
7:29:46this big dip. Um,
7:29:50yeah. So, let's let's create it. Um,
7:29:52that's okay. We're going to figure this
7:29:54one out together because I honestly um I
7:29:57didn't plan for this one. So, okay, we
7:29:59have 0.1 miles. That's exactly where it
7:30:01needs to be. The one, the two, the five,
7:30:04that's exactly where it needs to be.
7:30:05This 10 miles is not. And let's see if I
7:30:09change that 10 m 10 plus miles to 10
7:30:12miles plus. Let's see if that'll put it
7:30:15down here because I I don't know if it's
7:30:17looking at I don't know if it's reading
7:30:20it weird. Um, but let's go into this
7:30:22working sheet and let's go right here
7:30:26and we're going to do Ctrl H and we'll
7:30:28do Oops, not this one.
7:30:31Um, 10 miles plus. Let's get that in
7:30:35there. And we're going to do 10
7:30:38uh miles
7:30:40plus. I I don't know if that's actually
7:30:42going to work. Um, we will see. So,
7:30:45let's go back to the pivot table. Let's
7:30:48re go to the data. Let's refresh. Uh,
7:30:52no, it didn't. It didn't change it. Um,
7:30:54okay. So, let's think about this. Maybe
7:30:57if we change it to like a letter, it
7:31:00might change down here. So, start it
7:31:01with uh miles. That could work. Um,
7:31:04let's try it. Okay, it's already
7:31:07selected.
7:31:09Let's do is it 10 plus miles? Okay. So,
7:31:12let's do
7:31:15um
7:31:17m uh more than 10 miles
7:31:22and we'll replace all. Let's get rid of
7:31:25this.
7:31:27Let's go to the pivot and refresh. All
7:31:31right. Okay. So, it's not perfect, but
7:31:34it works. Um and for what we're doing, I
7:31:37think we'll keep it how it is. So, we
7:31:40have our second one. Uh, and
7:31:44you know, there are different ways you
7:31:45can kind of change this one. Um, you
7:31:48know, on the last one, we did a ton of
7:31:49different stuff. We can do
7:31:53just do
7:31:56commute distance.
7:31:58And we can say,
7:32:01what do we want to say on this one? What
7:32:02is this? Oh, this is the count. Um, do
7:32:06we have to do we have to keep this one?
7:32:10Um, no. There we go. I'm just going to
7:32:13do um just one and say
7:32:19commute distance.
7:32:22And let's add a title
7:32:25chart title. We can make this one um
7:32:28let's say distance
7:32:33per customer. Uh that's not 100% true
7:32:36because it's no or yes. Um that's that's
7:32:38the important part of this. It's
7:32:41distance um average distance. Uh let's
7:32:45see. We'll just say customer commute.
7:32:52All right. And we'll keep it just like
7:32:53that. All right. Perfect. I don't think
7:32:57um let me see. I don't think there's
7:32:59anything else we need to add on that
7:33:00one. All right. Now, let's go right down
7:33:02here. We're going to create our very
7:33:04last one. Uh, we only had three. So, you
7:33:07know, sometimes you'll have a ton.
7:33:08Sometimes you'll have like one on each
7:33:11sheet and you'll create multiple sheets.
7:33:12But, um, do control A. Um, now we have
7:33:17our thing. Now, this one we're going to
7:33:19be looking at these age brackets that we
7:33:22were looking at that we created. Um,
7:33:24something that I do honestly a lot is is
7:33:28kind of bracket things in into groups
7:33:30like this. And you know, for this I'm
7:33:32just kind of made them up, but um you
7:33:35know, it's good to know how to do this
7:33:39because I I promise you this one happens
7:33:41a lot or I use this one a ton. And then
7:33:44we just want to look at who purchased a
7:33:46bike. Uh so the same thing as we did
7:33:48before. So like purchased a bike, count
7:33:50of the purchase. Um you know, pretty
7:33:52easy. So we just have the count of
7:33:53either no or yes for these age ranges.
7:33:56Um and let's go to the insert. We'll go
7:34:00to recommendation. Um, I personally like
7:34:03a good line for this one. Um, so let's
7:34:08This is already interesting. You could
7:34:10do something like this.
7:34:13That's nice. See this one versus this.
7:34:16It just adds a dot. Oh, it looks nice.
7:34:18We'll keep that one. Um,
7:34:21so just really quick at a glance, really
7:34:24interesting. People under the age of 30
7:34:25are not buying that many bikes. um age
7:34:2830 to 54.
7:34:31Uh 31 to 54 buying a ton of bikes. Uh
7:34:34they are they buy more bikes or look at
7:34:37bikes more than anybody. Really
7:34:38interesting. Um but we'll make the
7:34:40dashboard a little bit. Um let's make
7:34:43these chart titles. We'll do
7:34:46ver oops the horizontal.
7:34:49We'll just call this
7:34:51age bracket.
7:34:55Um, and then we'll add a chart title.
7:34:58Um, again, you can add some extra stuff
7:35:01if you want to. Um, but you don't need
7:35:04to. Uh, none of this other stuff we
7:35:06really need. I'm just kind of looking at
7:35:07the stuff we do need or do want. Uh, so
7:35:10what do we want to call this one? Let's
7:35:12call it customer age
7:35:15brackets. Um, and it's not perfect, but
7:35:19we'll keep it as is for comparison. Um,
7:35:22let me see if I can copy um
7:35:26or or use this um real quick. Instead of
7:35:29the age brackets, I'm going to get rid
7:35:32of this and use the age.
7:35:36And then let's use
7:35:39um let's insert recommendation.
7:35:42We use a line and we'll use this. So
7:35:48this compared to this, just think of it
7:35:51like if a customer or consumer or or not
7:35:55a customer, if somebody you're working
7:35:57with is trying to use this dashboard,
7:35:58understand this dashboard, this is going
7:36:00to be just it's going to I don't know.
7:36:03It might melt their brain. It just makes
7:36:05no sense. It makes sense. It's just all
7:36:07over the place. It's really hard to make
7:36:08sense of this. It really is. I mean, you
7:36:10can kind of see a pattern going up
7:36:12around like the mid30s and then it
7:36:14trends downward, but it's hard to see.
7:36:17Um, it really is. So, doing these um
7:36:20these brackets really helps. And you can
7:36:22even add, you know, adolescent um you
7:36:26know, 0 to 30 underneath it. And in
7:36:29fact, we may want to do that. Um why
7:36:31not? Why not? Let's do that. Whoops.
7:36:34Um so, why don't why don't we do that?
7:36:37Why don't we go back? I'm just gonna I'm
7:36:39doing this on the fly. Why don't we go
7:36:40back?
7:36:42Uh, what am I doing? Whoops.
7:36:44And this is all calculated, but let's do
7:36:47adolescent
7:36:490 to
7:36:5130.
7:36:54Let's do middle-aged
7:36:5631 through 54
7:37:00and then old 55 plus. Let's see if this
7:37:03breaks anything. I hope it doesn't.
7:37:06Um, and we'll go back to our pivot
7:37:08table. Let's refresh the data.
7:37:14Uh, okay. It did mess with stuff.
7:37:17Okay, never mind, guys. That was a
7:37:19terrible idea. Don't do that. Um,
7:37:22perfect. [clears throat]
7:37:23Uh, let's get rid of that. That was a
7:37:25terrible idea. Don't do that. I'm glad
7:37:27we tested it out, though. I like I like
7:37:29to see if it was going to work. No, it
7:37:31messed with the um the order of things.
7:37:34Um, I I intentionally named them
7:37:37adolescent, middle-ag, and old because
7:37:39it's it it makes sense for the
7:37:41visualization. Um, but you know, if if I
7:37:46change something and it messes with it,
7:37:48I'm not going to mess with it. It was
7:37:49just an idea on the fly, guys. Come on.
7:37:51All right. So, let's start building out
7:37:53our dashboard. Now, um, when we're
7:37:56building our dashboard, what I
7:37:57personally like to do is to have this
7:37:59pivot table sheet, and then I will copy
7:38:02them over. And later, we'll hide these
7:38:04other sheets. Um, and I'll explain that
7:38:07in a little bit, but I like to have this
7:38:09this one for us. So, we're going to copy
7:38:11this. So, I just click on it, hit C.
7:38:13We're going to paste it right over here.
7:38:17Uh, let's just make them small for now.
7:38:19That's Oh gosh, no, let's not do that.
7:38:21Oh, these look terrible. Okay. Anyways,
7:38:24um
7:38:25let's copy this one over.
7:38:29Oops.
7:38:31Okay. What did I just do?
7:38:34Oh, I didn't copy this one. Whoops.
7:38:38It's not copying. Okay, we're going to
7:38:42go copy.
7:38:44Hit paste.
7:38:46Fantastic.
7:38:48Oops. Guys, look away. This is This is
7:38:51tough to watch. This is tough for me to
7:38:52watch. I'm the one doing it and it's
7:38:54tough for me to watch. All right, let's
7:38:55go to this last one.
7:38:58I'm going to try it again. All right, it
7:39:00worked this time. So, now we have um our
7:39:04our three visualizations. This is
7:39:06perfect. But now we actually want to
7:39:08create a dashboard. Now, how do you do
7:39:09that? How do you make it look nice? Um
7:39:11and then we're going to add some, you
7:39:12know, filters and stuff like that. How
7:39:13do we make it look nice? Um what
7:39:16happened here? What changed? What do we
7:39:18do?
7:39:21Oh my goodness gracious. All right,
7:39:23let's copy this.
7:39:26Let's paste this.
7:39:29Let's get rid of this. I don't even know
7:39:30how that happened. I've never seen that
7:39:31before. That was wild. Uh, Excel is
7:39:34trying to destroy my whole video. I
7:39:36mean, I'm doing this for you, Excel.
7:39:38Good night. Okay,
7:39:41no problem at all. What we're going to
7:39:42do and how you make this at least look
7:39:44nice. Um, first off, we can get rid of
7:39:47these grid lines pretty easily. And I
7:39:50recommend when you do that when you make
7:39:51a dashboard. It just makes it look
7:39:52cleaner. It makes it look like an actual
7:39:53dashboard. Um, let's go to view and grid
7:39:57lines. So, we can get rid of these grid
7:39:59lines. It just makes it look nicer. Um,
7:40:01we're going to make, you know, we can
7:40:03choose any color here. I'm just going to
7:40:05get choose a color.
7:40:08I like this. And let's We're basically
7:40:12creating like a header, right? if you're
7:40:13using like Tableau or something. Um
7:40:15we're going to merge and center. So it
7:40:16takes every single cell that we have
7:40:18highlighted, creates into one. Let's
7:40:20call this um bike sales. Uh I have I
7:40:24think I called it bike sales dashboard.
7:40:26Let's just call it that. Um you know,
7:40:29see what happens. Let's get that. Let's
7:40:33make it white and make it much larger
7:40:36than it is.
7:40:38Okay. Okay.
7:40:41Sure. Let's do that. Doesn't look bad.
7:40:46Um, what is it doing? There we go. Uh,
7:40:49let's make that center. Perfect. Um,
7:40:52it's not perfect, but we're going to use
7:40:53it. All right. So, now we kind of want
7:40:56to organize these. And, you know,
7:40:58everybody has their different way of
7:41:00doing it. Uh, I'm just going to start
7:41:03building it out myself and just see how
7:41:06it looks.
7:41:09Uh, and then we'll go from there. I like
7:41:10this one there. Um, we could put
7:41:14this one. I This one's a kind of a
7:41:16longer one, so I'll probably put it at
7:41:17the bottom. Let's see how it looks.
7:41:20Um, but we'll put this one right here.
7:41:24Try to line it up. Jeez. Let's Let's
7:41:27zoom in a little bit. Let's try to line
7:41:29this up. See what it looks like.
7:41:33Let's extend it to the end.
7:41:36That doesn't [clears throat] look too
7:41:37bad. Uh, needs to move up just a hair.
7:41:40Uh, and I'll show you how to kind of
7:41:41align these in a second. But, um, that
7:41:45looks [clears throat] not bad. And we'll
7:41:47kind of try to align these as well. And
7:41:49let me zoom out and extend this the
7:41:52length of this just to make it look
7:41:54nice. Um, you know, now what you can do,
7:41:59and you know, this is something that's
7:42:01pretty simple, is you can get both of
7:42:04these, and we're going to go to shape
7:42:06format, and we can just align these. is
7:42:08really nice to align especially if like
7:42:10the top um maybe like the left to right
7:42:13but like we're gonna align these to the
7:42:14top and they just kind of align
7:42:16themselves on the very top. Now these
7:42:18look much better. This one is a larger
7:42:21dashboard or a larger visualization. So
7:42:23I'm going to keep it how it is. Um and
7:42:27I'm going to keep this one how it is. So
7:42:28it is going to be a little bit smaller
7:42:30as you can tell. And then we'll have
7:42:31this one. Um and I'm going to do that.
7:42:36Um I this is going to bother me if I
7:42:39don't align these. So let me do this. Go
7:42:43shape format align to the right.
7:42:47And it's not exactly what I wanted to
7:42:50happen because
7:42:53oh jeez, what am I doing? That's not
7:42:55exactly what I wanted to happen. I
7:42:56actually wanted this one to align uh
7:42:58this one to align with this one. It did
7:43:00the opposite. Um so let me just scoot
7:43:02this back. All right. Visually it looks
7:43:05fine. But that's how you do it if you
7:43:06want to do it. Um I I I if you have
7:43:09multiple of them like this, it you can
7:43:11make it look bad. So we have our
7:43:13dashboards. This is already looking
7:43:14really good. I I like how this looks.
7:43:17Colors are coordinated. It we have a
7:43:19kind of a theme throughout. Um and it
7:43:22looks nice. I actually I actually kind
7:43:24of want to change this one um to
7:43:28um
7:43:30let's see.
7:43:34Maybe if I did like that it look nicer
7:43:36than all of them. Yeah, this does look
7:43:37nicer. Um it doesn't change much either.
7:43:41Guys, I'm Should I do it? All right,
7:43:43we're going for it. We're changing the
7:43:45design on the fly. Should I do it for
7:43:47all of them?
7:43:49Let's see.
7:43:52It doesn't fit. Doesn't fit. Um, all
7:43:54right, guys. Just ignore what I'm doing.
7:43:57Uh, don't do any of this. I'm just
7:43:58messing around at this point. So, this
7:44:01[clears throat] is really great to have.
7:44:02It really is. And what we want to do is
7:44:05there are other elements. There are
7:44:07other things that people would like to
7:44:08be able to filter by and be able to look
7:44:10at, but it's not in this visualization.
7:44:12Um, to be more specific, one field
7:44:16that's could be really interesting is
7:44:17married versus single. are single people
7:44:19buying more or um married people buying
7:44:22more. You know, it it'd be nice to
7:44:24filter on it. So, we're going to click
7:44:25on uh any of these actually and we're
7:44:27going to go up to pivot chart analyze
7:44:30and we'll click insert slicer. Now, we
7:44:33can choose which ones we want to be able
7:44:35to filter on all at the same time or one
7:44:37at a time. I'm just going to do the
7:44:38first one by itself and then I'll show
7:44:40you how to do other ones. Um but this
7:44:43one is the marital status. So, this is
7:44:44the married single, the one we were just
7:44:46looking at. And we can drag this right
7:44:49over here.
7:44:51Bring it a little bit.
7:44:54All right. And we don't need all that
7:44:56space. So, we're going to boop boop boop
7:44:58boop all the way up. Now,
7:45:01while we're doing this, um it only
7:45:04because we selected this uh this
7:45:06visualization, it only is working on
7:45:07that one right now. We of course wanted
7:45:10to apply to all of them. Is not hard to
7:45:12do. All we're going to do is we're going
7:45:14to click on we're going to make sure
7:45:16we're clicking on this. We're going to
7:45:17go up to slicer. We're going to hit
7:45:18report connections. Um and if you
7:45:21remember, we have this um this pivot
7:45:23table that we're working with. Um and
7:45:26this is where all of our pivots are
7:45:28coming from. So, we're going to actually
7:45:30apply it to all of them. This is our
7:45:32sheet. Um and this is the name of the
7:45:34pivot table. Now, again, we created that
7:45:36fourth one. We're not using it, but
7:45:37we're going to apply it to all of them.
7:45:39So now when we click on it, it's going
7:45:42to apply to all of them. So at a quick
7:45:44glance, let's see what single people are
7:45:46doing. Um,
7:45:50interesting. Interesting. Um, you know,
7:45:53when I'm looking at the just these
7:45:54numbers right here, married people,
7:45:56these individuals are making a lot more,
7:45:59like eight
7:46:02um sometimes 8 to like 10,000 more on
7:46:05average uh than their single
7:46:06counterpart. Um, you know, again, that's
7:46:09a rough estimate, but it's it's
7:46:10interesting. So, now what we can do is
7:46:12we're going to create more of these. So,
7:46:14we're going to go to uh pivot chart
7:46:16analyze. We're going to go to slicer.
7:46:18Now, we already did marital status, but
7:46:20what if we want to look at things like
7:46:22uh region and maybe something like their
7:46:26education. So, let's bring up both of
7:46:29those. And look, now two of them come
7:46:31up. So, let's add the region right here.
7:46:35Bring that in just a little bit. See if
7:46:37we can match it. Nailed it. All right.
7:46:40Now, we're going to put that up. We'll
7:46:43bring this one down.
7:46:45Just like this. Bring it over. See if I
7:46:48can match it again. Come on.
7:46:52Nail. Almost nailed it. I don't know if
7:46:54I nailed it, but it's close. All right.
7:46:56Kind of bring this up a little bit.
7:46:58Bring this up. And we have to do the
7:47:01exact same thing that we did with this
7:47:03one because right now again it only
7:47:04applies to that one um chart. So what we
7:47:07want to do is we want to go to slicer
7:47:09report connections. Add it to all of
7:47:11them. Okay. Do the same thing with
7:47:15education
7:47:16for connections. Bada bing bada boom. We
7:47:19are looking good. And now uh let's get
7:47:23rid of all of them. It's just going to
7:47:24be everybody. So now we can kind of
7:47:27slice and dice and choose what we want.
7:47:30We want to look at people who have a
7:47:31bachelor's degree who live in Europe and
7:47:34are single. And this is the information
7:47:36that we have on those people. So now we
7:47:38can narrow it down by certain
7:47:40demographics even further and look at
7:47:42this key information. So we may not, you
7:47:45know, look at counts and averages of
7:47:46these things, but we're able to filter
7:47:48on them. Uh, and that's really great to
7:47:51know. So, bachelor's degrees on average
7:47:53are making 60s 70,000. Um, let's look at
7:47:58um let's look at graduate degrees. Okay,
7:48:01a little more.
7:48:03Um, but you know, again, I'm just
7:48:06looking at random stuff. Um, but you can
7:48:08mess around with this. Take a look at
7:48:10some stuff. Um, this to me, I want to
7:48:13make this color darker. Feel like it'
7:48:15look nicer darker. There we go. Oh yeah,
7:48:18that's way better. This to me is it's a
7:48:21good dashboard, right? You have key
7:48:24information that you're looking at, nice
7:48:27visualizations, it's colorcoordinated,
7:48:29you have these slicers on the side. Um,
7:48:32to me, this is a fantastic just simple
7:48:36dashboard. And there are so many other
7:48:38things that you can do with this data
7:48:40and you can make it unique and you can
7:48:42add your own spin on it. And I highly
7:48:44recommend that you do that. Push
7:48:45yourself. go past what we just did today
7:48:48and add your own stuff and and use this
7:48:50and then you can add this to your
7:48:51portfolio website and show this off and
7:48:53show people that you know how to use
7:48:55Excel, which is a fantastic thing to
7:48:57know how to use and show off. So, with
7:49:00that being said, I hope that this
7:49:01project was helpful. I hope that you
7:49:03learned something along the way. I know
7:49:05I did. Um, I was learning things as we
7:49:07were going and I hope that you didn't
7:49:08mind that I took some detours along the
7:49:10way um for your amusement as well as my
7:49:13learning. Uh, so with that being said,
7:49:15thank you so much for joining me. I
7:49:17really appreciate it. I hope you have a
7:49:19good day and goodbye.
7:49:33What's going on everybody? Welcome back
7:49:35to another video. Today we are starting
7:49:36our Tableau tutorial series.
7:49:44Now, this series is for absolute
7:49:46beginners. So, if you have never used
7:49:47Tableau before, you are in the perfect
7:49:49place. I'm going to take you all the way
7:49:50from the very beginning of installing it
7:49:52and just understanding what Tableau is
7:49:54and how you can use it all the way to
7:49:56creating dashboards and sharing it. Now,
7:49:58personally, I hate those videos that are
7:50:00like 3 hours long and they just expect
7:50:02you to go through it. Uh, I like to
7:50:04break my videos up into chunks. So, if
7:50:06you have ever done my SQL tutorials,
7:50:08you'll know that I like to break things
7:50:09up so it gives you time to try them out
7:50:11and do them yourself and then you can
7:50:13move on to the next video. So, I'm going
7:50:14to be breaking this up into five
7:50:16separate videos. But in this video, I'm
7:50:18going to show you how to install Tableau
7:50:19for free. I'm going to show you the user
7:50:21interface. We're going to download a
7:50:23data set that you can find on Kaggle.
7:50:25And then we will build our first
7:50:26visualization together. With that being
7:50:28said, let's jump over to my screen and
7:50:29we'll get started. All right. So, the
7:50:31very first thing that we need to do is
7:50:32you need to actually download Tableau.
7:50:35So, we're not going to be using Tableau.
7:50:36We're going to be using a free version
7:50:38called Tableau Public. It has a lot of
7:50:40the same features except, of course,
7:50:42it's not uh every single feature that
7:50:44regular Tableau has, but it is
7:50:46absolutely perfect for learning it and
7:50:48for using it and and you can even build
7:50:51um you know, dashboards and share those
7:50:52for your portfolio. Um I'm going to put
7:50:56this link in the description so you can
7:50:57just go and click on that and and all
7:51:00you have to do is input your email right
7:51:01here. We're going to click download the
7:51:03app. Um, and then it should start to
7:51:05download. And then you can save that.
7:51:07And then you're going to open this up.
7:51:09Now, I'm going to open it up. I don't
7:51:11know what it's going to do. I already
7:51:12have it downloaded. Um, but it should
7:51:15open up and look hopefully like what
7:51:17you're seeing on uh my screen in just a
7:51:19second. Let's see what it does. Um, I
7:51:22hope you can see this, but it says
7:51:24Tableau Public. Um, it says I already
7:51:27have it set up, but you're going to
7:51:28click install and go through all that um
7:51:30all that setup stuff. Uh, so I'm going
7:51:31to exit out of here, but I'm going to go
7:51:34over here and type in Tableau Public.
7:51:37Uh, and it's 2021.3. That's the current
7:51:40version that they have out if you're
7:51:42doing this in the f.
7:51:46Um, so you should be able to pull this
7:51:48up right here. Now, um, I'm going to go
7:51:52and get our data set that we're going to
7:51:54be using, and I'm going to show you how
7:51:55to get that as well, and then we will
7:51:56actually jump into Tableau and start,
7:51:59uh, using it. So, let's go over here.
7:52:01I'm going to get a data set from Kaggle.
7:52:03I wanted something pretty generic uh to
7:52:05show you. In future videos, I'm going to
7:52:07show you some special or not special,
7:52:10but just different visualizations that
7:52:12you might use. Um, and we'll get
7:52:14different data sets for those because,
7:52:15of course, not one data set covers all
7:52:18these other types of visualizations. So,
7:52:20um, we're starting off pretty simple
7:52:21right here. We're going to be getting
7:52:22one called video game sales. Um, and we
7:52:26can take a really quick look at it. Um,
7:52:28here are some of the fields that you're
7:52:29going to be having. Uh, like rank, name,
7:52:31platform, the year, genre, and then some
7:52:34sales data. And this is what it actually
7:52:36looks like. It's called VG sales. So,
7:52:38video game sales. It's in a CSV. And um,
7:52:41you know, here are the fields. And we
7:52:44have our data. And all we are going to
7:52:46do is we're going to download that.
7:52:49And I will save it. Now, when you
7:52:51download it, it's going to be saved into
7:52:53a zip file. So, we need to go to our
7:52:55downloads. Uh, let's refresh this.
7:52:59Here's our archive. We need to go in
7:53:01here. You can just copy it and paste it
7:53:04right back into here. Um, and just so
7:53:07you know, that is a uh a CSV, so be
7:53:11aware of that. So, what we want to do is
7:53:13we want to come in here. Now, since it
7:53:14is a CSV, this is not we're not going to
7:53:16be using Microsoft Excel. We're going to
7:53:18be using the text file. So, we'll come
7:53:20in here. We'll take VG sales. Now, uh,
7:53:24one thing I want to do before I do that
7:53:25is I'm going to rename mine, uh, VGSK
7:53:29sales_1.
7:53:30Um, I've already prepared for this and
7:53:33so I already have that in there. Um, but
7:53:36so I want to make a distinct one for
7:53:37myself. You do not have to do that. So,
7:53:39we'll come back here. Um, and then we're
7:53:42going to do text file and VG sales.
7:53:45We're going to open that up.
7:53:48And
7:53:50when it pulls up right here, um, you can
7:53:53bring in other tables and then you can
7:53:55start to join them together and create
7:53:57those relationships. We are not going to
7:53:59be doing that in this video. We'll do
7:54:00that in a separate one. Um, as for, you
7:54:04know, just getting started, you know,
7:54:05we're not going to be using that. But
7:54:06you can see um, some of these
7:54:10things or some of these fields. And if
7:54:13you notice, they they um they're either
7:54:17ABC or they're a number. So it starts to
7:54:20categorize
7:54:21what this field type is. So is it a
7:54:24string? Is it numeric? It starts to
7:54:26automatically do that. And that's all
7:54:28done within Tableau.
7:54:30And so it just kind of reads it and
7:54:32that's what it does. Um what we're going
7:54:34to do is we're going to click right down
7:54:36here. It's called go to worksheet. Um,
7:54:38the worksheets are where you're going to
7:54:39actually start being able to build your
7:54:41visualizations, your charts, your
7:54:43graphs, all these things. Um, and so,
7:54:45you know, we have this in here now. And
7:54:47so, we're just going to click right here
7:54:49on go to worksheet. As you can see here
7:54:52is VG sales_1.
7:54:55You will not have the underscore one if
7:54:56you did not add that like I did. Uh, but
7:54:59right down here, you can see all the
7:55:00fields that we just imported from that
7:55:02data set. And they even created one
7:55:03right here for us. Uh, they just
7:55:06generated that field. um based on the
7:55:08file. So it's a count of all the rows
7:55:10really. So what I'm going to do is I'm
7:55:13just going to walk you through uh
7:55:16basically what we're looking at some of
7:55:17the things that we're going to be using
7:55:18today. There will be things that I don't
7:55:20talk about, but I'm going to highlight
7:55:23those in in future videos when we start
7:55:25using those or going over them. Um and
7:55:28so let's just start with the most
7:55:29obvious one. It's way over here. I'm
7:55:32sure you saw it when we uh this first
7:55:34came up on the screen because it has all
7:55:36these different charts and
7:55:37visualizations and graphs. And uh these
7:55:41will become available as you start
7:55:43dragging and dropping our data into this
7:55:46sheet. And so if I go right here, it
7:55:49says four scatter plots try zero or more
7:55:51dimensions, two to four measures. So
7:55:54what our dimensions are are right here
7:55:56and what our measures are are right down
7:55:58here. And so typically uh things like
7:56:01like you say genre or names or or
7:56:04strings like that are going to be these
7:56:07uh dimensions and then a lot of a lot of
7:56:09times the numerical is going to be are
7:56:11going to be measures. Next what I want
7:56:14to show you is right here. So you can
7:56:17take something like global sales and you
7:56:20can drag it right here into your rows
7:56:23and then it takes your rows and so it
7:56:26automatically created a sum of global
7:56:28sales. Now if we take that away, let's
7:56:31say we drag it right here, it's going to
7:56:33give us a column. Now you can also do it
7:56:36right up here. You don't have to um drag
7:56:39it on screen. You can also
7:56:42just add it to the column or the row.
7:56:44That's typically what I do. I it's just
7:56:46more intuitive to me. Um or you can drop
7:56:49it in this section right here and it
7:56:51does its best to assign it some type of
7:56:54um some type of visualization. And so
7:56:57that's [clears throat] what it always is
7:56:58trying to do. It is trying to say okay
7:57:01this is what you're trying to do. Let me
7:57:03try to get the best visualization for
7:57:05the data that you're giving me. Now,
7:57:08while we are here, um, it went down here
7:57:11into [clears throat] marks. And marks is
7:57:13a very important area. It's where you
7:57:16can add color, size, text, detail, and
7:57:19tool tip. And I'm not going to go into
7:57:21what all those are cuz I'm just going to
7:57:22show you. So, let's start pulling some
7:57:24fields in here and creating a
7:57:25visualization. And then I'm going to
7:57:27show you how all of that works,
7:57:29including filters as well. So, the first
7:57:31thing that we are going to look at is
7:57:33global sales. And let's put that in the
7:57:36rows. And then I'm going to take year
7:57:39and I'm [clears throat] going to make
7:57:40that the column. And this is basically
7:57:43exactly what uh I wanted to do. Now, as
7:57:46of right now, it has only the year and
7:57:49it's looking at global sales for
7:57:52everything. But we want to break that
7:57:53out a little bit better. I want to break
7:57:56it out by let's do genre. So different
7:57:59genre of games. Now, if I add that right
7:58:02here to this columns, it is going to
7:58:05break it up by year and genre. If I add
7:58:09it right here is going to break it out
7:58:12by the year, of course, but then in each
7:58:15individual row has the different genre.
7:58:18That's not what we want. We want to keep
7:58:20this type of line graph. Uh, and what
7:58:25we're going to do is we're going to add
7:58:26it to marks. And you can't really see it
7:58:29based off of these colors, but they're
7:58:31all different. So, we have action genre,
7:58:34we have the sports genre, racing, uh,
7:58:37role playing, all these different genres
7:58:39within it. Now, we can get rid of that
7:58:41because we don't need it anymore. Uh,
7:58:43and this is where these um these marks
7:58:46really come in handy because you can
7:58:48start basically doing what you want with
7:58:51them. So, for the genre, I want to be
7:58:54able to see all these different genres
7:58:55with different colors. To me, that just
7:58:57makes the most sense. So, I'm going to
7:58:58put color right here. And automatically,
7:59:00it assigns every single genre its own
7:59:03color and gives us this legend right
7:59:06over here. And so, it's really easy to
7:59:10see. Well, when you have smaller
7:59:11numbers, it's much easier. But I know
7:59:13that red is sports. And I can go right
7:59:15here and find red. And that is sports.
7:59:18So, it makes it a lot easier than when
7:59:19it is all the same color, blue. So what
7:59:23you can do after that is you can also
7:59:25add things like uh a label to it. So if
7:59:28we take label and we or we take genre,
7:59:31put label, you can click right here and
7:59:34you can get rid of the labels that you
7:59:36have and you can see them right down
7:59:37here. Or you can also change uh the
7:59:40font. So if you want to make it orange
7:59:42or or whatever color, you can do all
7:59:44those same things. And you can also do
7:59:47things like changing where you see these
7:59:49things. So for action, you're going to
7:59:52see it a ton because for each year
7:59:54action is is at the is on the higher end
7:59:57and so you're seeing those in those mins
7:59:59and maxes. You can also do it for a
8:00:01selected area. So if I come in here and
8:00:03I select it, it's then going to show me
8:00:06what those are. So label is really
8:00:08really uh useful, really helpful. Let me
8:00:11get rid of that really quick. Uh you can
8:00:13also do it where the lines end. So line
8:00:16ends is at the beginning and the end.
8:00:18And you can also take that away or put
8:00:21that back on. So labels are really
8:00:23important. Labels aren't very helpful
8:00:25when you're doing, at least I don't find
8:00:26that it's super helpful when you're
8:00:28doing things like genre. So when you're
8:00:30doing your dimensions. So I'm going to
8:00:31get rid of that. And I'm actually going
8:00:33to bring our global sales over here. And
8:00:36let's label that.
8:00:39And right now I think it's labeling the
8:00:42uh line ends. We want to do the min and
8:00:44max. Now, if we do min and max on the
8:00:47table, it's just going to give us the
8:00:49max and the min, which is zero, and then
8:00:52139.4,
8:00:54it's a little bit more useful if we do
8:00:55it per each line. Uh, this at least
8:00:58gives us some context. I probably
8:00:59wouldn't do this in an actual visual
8:01:01visualization. But to give you some um
8:01:03understanding of just how it works. So
8:01:05now I know that um right over here the
8:01:08min and the max or the min uh sorry the
8:01:11max for these for action and for sports
8:01:14is right around 138 139. So it's pretty
8:01:17easy to see. Um and you can again go in
8:01:21here and you can remove the max or
8:01:24remove the mins whichever one you feel
8:01:26is best. Uh you'll probably keep the
8:01:29maximums in there for each category. And
8:01:31so this is a really quickly becoming uh
8:01:33a pretty usable visualization. And it's
8:01:36not the only label that you can add. We
8:01:38still are using year over here. So we
8:01:41can always drop year in there as well.
8:01:43Create a label. And so now we have let's
8:01:46see for this one is a puzzle genre. So
8:01:49we also have the year that it had the
8:01:52maximum uh sales. And so you know just
8:01:56some [clears throat] things that you can
8:01:57do. You don't have to add that.
8:02:00Now, let's go up here and we're going to
8:02:01take a look at filters because filters
8:02:04are really important. You know, if you
8:02:05are making this for a client or you are
8:02:08making this for somebody, you want them
8:02:10to be able to filter down uh to very
8:02:13specific information that they want to
8:02:15see. So, let's take uh the platform.
8:02:18Lots of different platforms.
8:02:20Um as you can see, you know, PS4, Xbox,
8:02:23um if you're familiar with these, we'll
8:02:26click all of these. Um, and we'll click
8:02:29okay. So now this is an option as a
8:02:32filter and all we're going to do is
8:02:33we're going to click on this arrow right
8:02:35here and we're going to say show filter.
8:02:38Now right now all of them are selected.
8:02:41So every single one is being taken into
8:02:43account for this visualization. But
8:02:46let's say we come down here and we say
8:02:48okay I don't want to see sales for any
8:02:50of these PS the original PlayStation 2,
8:02:53three or four. So, I'm going to get rid
8:02:54of this one, this one, this one, and
8:02:58this one. And you could immediately see
8:03:00the the changes that were happening. So,
8:03:03now none of the numbers, none of those
8:03:05sales are being accounted for and and
8:03:07being added to the sum of global sales
8:03:10right here at all.
8:03:12So, uh that is just how a filter can
8:03:16work. And you can also do that. and you
8:03:21can get rid of all of them and you can
8:03:23go in and actually just pick very
8:03:25specific sales. So if you only want to
8:03:27see the PlayStation sales, you can go in
8:03:29there and do that as well. So really
8:03:32really handy filters are things that you
8:03:35you'll at least want to have as an
8:03:37option for most of your your
8:03:39visualizations. At least that's what I
8:03:40found, especially when you're doing
8:03:41client facing work. They like to uh get
8:03:44in there and mess around and look at
8:03:45different look at it in different ways.
8:03:47And so that's one that I I think is is
8:03:50really useful to to have.
8:03:53The very last thing that we want to do
8:03:57is we want to actually add this to a
8:04:00dashboard. Now let's say we come right
8:04:03down here and we add a new worksheet.
8:04:05And actually we might change one more
8:04:06thing on that last one, but we'll just
8:04:08make a really simple one. Um we'll just
8:04:11give it genre and we'll give it global
8:04:14sales as the rows. Um, and this nifty
8:04:18button right up here, which is a sorting
8:04:20button. So, I'm going to sort like that.
8:04:23I'm going to add the genre in just as we
8:04:26did. I'll give it different colors.
8:04:28Perfect now, we have two really quick
8:04:30different visualizations. Right. What I
8:04:33want to do is just show you how to
8:04:34combine those because what you are going
8:04:37to do is you're going to actually come
8:04:39in here and you're going to do new
8:04:42dashboard. That's what this button is
8:04:43right here. Now, when we come in here,
8:04:45the size is extremely small. It's very
8:04:48easy to fix that. All we're going to do
8:04:50is click right here. We're going to go
8:04:52to this range or this dropdown, and
8:04:54we're going to click automatic. So, now
8:04:56it is a much larger size for us to
8:04:58actually drop our visualizations into.
8:05:01Uh, and let's put sheet one. And we'll
8:05:05put uh let's put it up top. So, now it
8:05:08looks a little bit like this. Uh, not
8:05:10perfect, but again, if I wanted to make
8:05:13this look a lot better, I definitely
8:05:14would. And then you can go over here and
8:05:17you can rename these things. You can
8:05:18also do that back when we were in our
8:05:20actual worksheets, but you can also do
8:05:22it here as well and then start um, you
8:05:25know, customizing and building it out.
8:05:27That's not what this video is for. That
8:05:28is the last video. We're going to build
8:05:30an entire dashboard. It'll be kind of
8:05:32like a small project. You can put that
8:05:33in your portfolio. Um, if you have
8:05:36gotten this far and you want to jump
8:05:38straight into it and you don't want to
8:05:40wait for these other videos to come out
8:05:42or you don't you just want to jump
8:05:43straight into creating an entire
8:05:45portfolio project, I have an entire
8:05:47portfolio project series that covers
8:05:50SQL, Python, and Tableau. And so, go
8:05:53check out that series. I have one video
8:05:56dedicated to Tableau. It's like 45
8:05:58minutes or an hour long and it covers a
8:06:01lot of the things that we're going to
8:06:02hear in here as well as a few other
8:06:04things. But I appreciate you checking
8:06:07out this video. In future videos, we're
8:06:09going be going over things like creating
8:06:10bins, calculated fields, doing joins,
8:06:13and then creating a final project and
8:06:15putting it all together. So, thank you
8:06:17so much for joining me. I really
8:06:19appreciate it. If you like this video,
8:06:21be sure to like and subscribe below, and
8:06:23I will see you in the next video.
8:06:37What's going on everybody? Welcome back
8:06:38to the Tableau tutorial series. In this
8:06:40video, we're going to be going over bins
8:06:42and calculated [music] fields.
8:06:49All right, so let's jump right into it.
8:06:51The first thing that we're going to look
8:06:52at are bins. And bins are basically just
8:06:55groupings or ranges of numerical values.
8:06:58So we cannot create bins uh for genre,
8:07:01name, platform or anything like that. We
8:07:03have to do something with this sign
8:07:04right here, which means that it is a
8:07:06numeric. So year or all of this sales
8:07:09data or this ranking data and we're
8:07:11going to use what we worked on in our
8:07:13very first tutorial. And so what we're
8:07:16going to be using to kind of demonstrate
8:07:17how bins work is this year right down
8:07:19here. So, right now we have a range of
8:07:221993 all the way up to 2018. And we're
8:07:25going to create some bins to group and
8:07:27create ranges for these years. And it's
8:07:30pretty simple. All we're going to do is
8:07:32we're going to come right over here to
8:07:34year and this little drop down on the
8:07:36side and we're going to go down to
8:07:37create and go down to bins. Now, it's
8:07:41going to say the size of bin and it's
8:07:43going to give you a recommendation based
8:07:46off of the information that is already
8:07:48provided. The min and the max, the
8:07:50ranges of these values. You know, you
8:07:52don't have to do this, but usually um it
8:07:55it does give some good estimation on
8:07:57what you might be considering. If you
8:07:59were thinking, hey, maybe do a bit of
8:08:01like 20 and they're recommending two.
8:08:03Think about why they might be doing
8:08:04that. We're going to change ours to
8:08:06five. And you can always change what
8:08:08this field is going to be. I'm just
8:08:10going to give it an old exclamation
8:08:12point just to um really spice things up
8:08:14here. So, we're going to click okay. And
8:08:18as you can see, it adds it right up
8:08:19here. Is no longer um it is no longer a
8:08:23numeric. Now, it is a categorical. So,
8:08:26now it's this is no longer just uh 1 2 3
8:08:294 5. It's ranges. It's groups. And we're
8:08:32going to get rid of this year really
8:08:33quick. Actually, let's keep it up there
8:08:35for a second. Uh see what happens. But
8:08:37we're going to bring this up and we'll
8:08:39get rid of this year. And this is is
8:08:43what kind of it spits out for us. Now, I
8:08:46did look at the data when I was prepping
8:08:48for this. There are some nulls in the
8:08:49years. Um, and so all we're going to do
8:08:51for this is we're just going to go like
8:08:52this and we're going to exclude the
8:08:55nulls. Uh, probably not something you
8:08:58should be doing uh if you're doing this
8:08:59for work, but this is for demonstration
8:09:01purposes, so we can do whatever we want.
8:09:04But as you can see, we now have these
8:09:07ranges. So this range starts at 1990
8:09:11and it includes 1990 all the way up to
8:09:141994 and then it's 1995 to 1999.
8:09:18And so just really quickly, we can tell
8:09:21that the years 2000 to 2004 were a huge
8:09:25huge huge uh season or group of of years
8:09:28for game sales. So these are the global
8:09:31sales for for these video games. And so
8:09:34it is really helpful. It's very useful.
8:09:37Um you can do this on a lot of different
8:09:39information. We could do this on this
8:09:41sales data. You can do this on age. You
8:09:43can do it on years like we did. And it
8:09:45can be very very useful. And so uh
8:09:48really quickly that is how bins work. I
8:09:51would say it's pretty straightforward.
8:09:53Now this is a perfect time to segue into
8:09:56the next part of the video which is
8:09:57calculated fields. uh right over here on
8:10:00this left hand side. We see that the
8:10:02global sales which are in millions goes
8:10:04all the way up to 900 million and
8:10:06created these beautiful bins right down
8:10:08here. But let's look at within these
8:10:11from 1999 to 2015. Let's see which of
8:10:14these has the highest percentage. Of
8:10:16course, it's going to be this one. But
8:10:18we can do something called a quick table
8:10:21calculation. Um we'll create a our own
8:10:23calculation later. I'll show you how to
8:10:25do that. But we're going to do a quick
8:10:26table calculation and we're going to do
8:10:28the percent of total. And so now we have
8:10:31these bins and instead of just seeing
8:10:33the total amount of sales that they had,
8:10:35we see the actual percentages based off
8:10:37these year ranges, which is really
8:10:40useful, something that you could
8:10:41absolutely put uh in some real work that
8:10:43you do for a client. Now, really quick,
8:10:46just to show you something that you can
8:10:47do, if you click control and you drag
8:10:50this over here, you can actually save
8:10:52that calculation. So we can say
8:10:54percentage of global sales and that
8:10:59actually saves it as uh you know a
8:11:01measure for us. So that was a quick
8:11:03calculation but let's look how to
8:11:04actually create a calculated field. So
8:11:08if we do this right here what is going
8:11:10to come up is just the global sales and
8:11:13you can do a lot of what you would
8:11:14basically do in Excel. Multiplication
8:11:16division subtraction a few other things
8:11:18but we're going to keep it super super
8:11:19simple today. All I'm going to do is I'm
8:11:22going to take global sales and I'm going
8:11:24to subtract. I'm going to do an open
8:11:26bracket and I'm going to say EU sales
8:11:28and it auto completes for me. I'm going
8:11:31to click okay. And it created
8:11:33calculation two. I'm going to come in
8:11:35here and I'm just going to say global
8:11:38sales minus
8:11:41EU sales.
8:11:43And let's drag this over. These are
8:11:46different. Um, one's percentage, one is
8:11:51in terms of sum. And so I'm just going
8:11:54to bring this in right here. And so now
8:11:56we are comparing against the same thing.
8:11:58And if we look at the global sales, we
8:12:00have probably right around 950
8:12:04millionish in this 2000 uh to 2004 bin.
8:12:08And for global sales minus the EU sales,
8:12:10we're looking at, you know, 650 million.
8:12:13So there is a noticeable difference. And
8:12:15this is just one of the ways that you
8:12:17can use uh calculated fields to actually
8:12:20just show the difference between two
8:12:22numbers or you can do more advanced
8:12:24calculations depending on the data that
8:12:25you actually have. So that's it for this
8:12:27video. I hope you learned a little bit
8:12:28more about bins and calculated fields.
8:12:31In the next video, we're going to be
8:12:32looking at a ton of different
8:12:34visualizations and graphs and charts and
8:12:36just exploring what options are really
8:12:39are out there for visualizing our data.
8:12:41Thank you guys so much for joining me. I
8:12:43really appreciate it. If you like this
8:12:45video, be sure to like and subscribe
8:12:47below and I will see you in the next
8:12:48video.
8:12:50[music]
8:12:58[music]
8:13:01What's going on everybody? Welcome back
8:13:02to the Tableau tutorial series. In this
8:13:05video, we're going to be looking at lots
8:13:06of different visualizations, including
8:13:08the scatter plot and density maps.
8:13:09[music]
8:13:16Now, before we jump into the tutorial, I
8:13:18have some very exciting news. In just
8:13:20two days, on October 7th, I am going to
8:13:22be partnering with AlterX to host a
8:13:23webinar. This webinar is completely for
8:13:26data analysts who are wanting to change
8:13:28careers to become a data analyst. Now,
8:13:30you did hear that right. I will be the
8:13:31host of the event, but we will be
8:13:33bringing on guests as well who are
8:13:34industry experts who actually change
8:13:36careers to become data analysts, much
8:13:38like myself. They'll be sharing their
8:13:39stories of how they actually transition
8:13:41careers along with the tools that they
8:13:43found extremely useful and helpful to
8:13:45make that switch and they'll be giving
8:13:46lots of advice along the way. So, if you
8:13:49are somebody who is wanting to change
8:13:50careers to become a data analyst or just
8:13:52wanting to learn about data analytics,
8:13:54this is an absolute fantastic place to
8:13:57learn a lot more about that. I will
8:13:58leave a link in the description. So, be
8:14:00sure to go and sign up for that. Again,
8:14:01I'm going to be there, so it should be
8:14:03really fun. Without further ado, let's
8:14:05jump on my screen and start the
8:14:06tutorial. Now, we are about to look at a
8:14:08ton of different visualizations. Uh over
8:14:11here, you can see just an array of them,
8:14:14but not all of them are ones that I
8:14:17actually think are useful or ones that I
8:14:19would actually recommend using. And so,
8:14:21I'm going to take you through some of
8:14:22the ones that I absolutely think are
8:14:25worth learning and using and trying out.
8:14:27Uh, and I'm just going to kind of just
8:14:30show you how I might use them, how they
8:14:32might look, how you can navigate them a
8:14:34little bit. Now, before we do that, we
8:14:36do need to go download one data set.
8:14:39It's this Starbucks location worldwide.
8:14:41Yes, we're going to do a little bit of
8:14:43longitude latitude here. And all we have
8:14:46to do is click this downloads button and
8:14:49it will download. We're going to do that
8:14:51into downloads. We'll save that. Uh,
8:14:54yeah, I've already done that, but you
8:14:56know, I'm doing this with you guys. I'm
8:14:58doing it for you. So, let's go to our
8:15:00downloads.
8:15:02Now we have here we want to come in
8:15:04here. We're going to copy it or um you
8:15:07can cut it. Uh and then we're going to
8:15:10paste it here. Yeah. Replace it.
8:15:12Perfect. And now we have it ready to go.
8:15:16We'll come in here. Let's do a new
8:15:18sheet. And I already have it in there,
8:15:20but uh I'm just going to show you what I
8:15:22would do. Do new data source. Uh we'll
8:15:24do text file. We'll do directory. And we
8:15:28will open it.
8:15:31And let's see what data we have in here
8:15:33before we actually begin. Uh just super
8:15:35quickly. We have the brand. So, um
8:15:39whatever company has it. And then a
8:15:41bunch of um location information. Street
8:15:45address, city, the state. This is all in
8:15:48the United States. So, that's basically
8:15:51it. And what we are going to do is we're
8:15:53going to go over to this sheet three.
8:15:56And we have this directory 2. That's the
8:15:58one I just pulled in. uh exact same
8:16:00thing as directory but so the first
8:16:02visualization that we are going to look
8:16:04at is a bar and line graph. So what
8:16:06we're going to take is the year right
8:16:08here and take these global sales and
8:16:11these NA sales and we're going to be
8:16:14doing this one right here. So this has a
8:16:17combination of two separate uh types of
8:16:20visualizations. So sometimes you just
8:16:22have line, sometimes you just have these
8:16:24uh these bar graphs or these bar charts.
8:16:27Uh, and we're combining the two. And
8:16:29it's very nice. I like how this looks.
8:16:32Now, if you notice, if I put this NA
8:16:35sales behind it, now it kind of cuts
8:16:37off. So, now this global sales is in
8:16:39front. We're going to, you know, put
8:16:41that back. I just wanted to show you
8:16:43that uh right here there's all sum of
8:16:46global sales, sum of NA sales. So, if we
8:16:48go into this all, we click this
8:16:50dropdown, we can change it to a line.
8:16:52Uh, we can change it basically whatever
8:16:54we want. I just hit control-z to reverse
8:16:56that. But what we can do is we can go in
8:16:59here and we can change this color. And
8:17:03let's see if we can just make it red. Is
8:17:05that possible?
8:17:08See what I did? I made it orange. That
8:17:10works for me. Um, just something to
8:17:12stick out a little bit more. Choose
8:17:14whatever color you want. And this is a
8:17:16really nice visualization. This is one
8:17:17that I have used in the past. We're
8:17:19looking at global sales versus the NA
8:17:21sales. And so it's very easy to see the
8:17:24distinction between the two and how one
8:17:26was doing a specific year versus how the
8:17:28other one was doing in that same year.
8:17:31And so I really like this. If you want
8:17:32to do something uh like keeping it
8:17:34consistent, you can do two bars. I don't
8:17:37really like this one as much. Um and you
8:17:40can again you can really change it up.
8:17:42Um there's lots of different ones that
8:17:44you can do. Again, I prefer the line,
8:17:46but you know, do whatever you think is
8:17:49best. I'm going to change it back
8:17:50because this is not how I want to keep
8:17:52it. But there you go. So that is the
8:17:54[clears throat] first one that we are
8:17:55going to look at. Let's move on to the
8:17:58second one. And we actually will be
8:18:00using our our Starbucks data here. Now
8:18:04when you bring in data that has um any
8:18:07type of map or or um address or postal
8:18:11code or things like that or or country,
8:18:13it's typically going to create this
8:18:15latitude and longitude and it's going to
8:18:17generate that. Now, what we want to do
8:18:19is bring this longitude right up here
8:18:22and this latitude right there.
8:18:26And if you do the show me right now,
8:18:28it's giving us this. But what we want to
8:18:30do is add what we're looking for. So,
8:18:33what will we actually be trying to
8:18:36search for on this map? You can do
8:18:38anything from like a postal code. Um,
8:18:41and it will drag us right here. Let's
8:18:44come over to this. This allows us to
8:18:46kind of scroll around a little bit. Um,
8:18:50we're going to mess around with this one
8:18:51for just a little bit. And see if I can
8:18:56That's nice. That might be too big. Let
8:18:59me back up one. So, at least in the
8:19:02continental US, a little bit down here.
8:19:05This these are the postal codes. So,
8:19:06right now, we're looking at postcodes.
8:19:08Uh, and
8:19:11there are a lot that you can do with
8:19:13this. Um really color will make almost
8:19:15no difference. It just becomes this
8:19:17mess. So you don't typically want to do
8:19:19something like that. At least not for
8:19:21this. Let's go to size. And if we make
8:19:25it really small, you can kind of see
8:19:29these groupings, these pairings um
8:19:31typically of like larger cities or major
8:19:34major metropolitan areas. And so you can
8:19:37do this and it's and it's really really
8:19:39easy. I don't recommend uh labeling
8:19:42this. I don't even know if it'll do it.
8:19:43Um, it would be an absolute mess to try
8:19:46to label all these postcodes.
8:19:48But let's bring this out and let's bring
8:19:50these state and provinces in. Now, right
8:19:53now, we have these little tiny tiny uh
8:19:56dots on here. And I think what we want
8:19:59to do is not increase the size, but over
8:20:04here we want to actually do this and
8:20:06make it a map. And so now it's going to
8:20:08fill in all the states. And we can, you
8:20:11know, why not? We'll add some color
8:20:12here. Um, but we can
8:20:16Oh, it has a numbered. I didn't think
8:20:17they were numbered. Um,
8:20:20oh, that's interesting. I haven't seen I
8:20:22didn't look at that before. I was just
8:20:24uh found that interesting. But now we
8:20:26can see what uh what states Starbucks is
8:20:29in. And as you can see, they're in all
8:20:3150 states. But it's something
8:20:33interesting to um look at to think
8:20:35about. Now, if we go right up here, we
8:20:38can again choose a different type. and
8:20:40we're going to go to the density. Now,
8:20:43right now, it's just doing a density on
8:20:44the uh the state. We're get rid of that.
8:20:47We're going to bring back postal code.
8:20:49I'm just switching it up on you a little
8:20:50bit. And you can do it as small or as
8:20:53big as you'd like. Um you know, I like
8:20:56to do somewhere in the middle. Um
8:20:58probably right
8:21:00right about there is fine. Um I don't
8:21:03think it's going to make sense to really
8:21:04add any color here. Again, all these
8:21:05poster codes are different, so it's just
8:21:07going to be complete mish mash. But uh
8:21:09this is kind of how you can use a
8:21:11density map. And you can do this AC with
8:21:14uh countries, you can do this with
8:21:16postal codes, you can do this with any
8:21:18type of kind of like address or
8:21:19locationbased data. So that is how you
8:21:23can use a map. Again, there's lots of
8:21:26different ways to use a map. And so I'm
8:21:28not going to show you every single way,
8:21:30but in a really brief way, this is how
8:21:31you can use a map to actually visualize
8:21:33your data that does have location uh
8:21:36based information in it. So, let's go
8:21:38over to sheet three. Uh, and this data
8:21:40that we have over here, it just allows
8:21:43for a lot of different types of
8:21:44visualizations. So, we're going to use
8:21:46this one. Um, and there are lots of
8:21:48other ones that you might see out there,
8:21:51like this one right here. Uh, we
8:21:53obviously wouldn't be using this. We
8:21:55might do something like this. Change the
8:21:59label.
8:22:00Um, and maybe add Why are both of these
8:22:03in here? Um, let's get rid of this.
8:22:06Oops, that's not what I meant. Let's
8:22:07actually add that. Let's do the sum of
8:22:10global sales and we'll just make that
8:22:12into a label as well. So,
8:22:16what you can do with these and and how
8:22:18you're able to use them and visualize
8:22:20them. Again, these are not you'll see
8:22:23these often, but these are not often
8:22:25ones that I would recommend you use.
8:22:27That's very similar to these packed
8:22:29bubbles. Um, you can add these global
8:22:33sales in here. again add the label. It
8:22:37just uh it sometimes is not as
8:22:39straightforward the information that
8:22:42it's trying to tell you, right? You kind
8:22:44of have to search for it a little bit.
8:22:45You kind of have to look around. Um but
8:22:49you can find some good visualizations in
8:22:51here for very specific types of data.
8:22:54And so these are just ones to consider.
8:22:56Uh one that you'll see all the time is
8:23:00uh this guy right here. And uh let me
8:23:03see if I can expand this a little bit
8:23:06cuz this is
8:23:08very small. Um let's see. I have this I
8:23:13just want global sales
8:23:15and let's label that
8:23:19the size.
8:23:22How do I expand this? Haven't done this
8:23:25in a while. Let me just expand this. I
8:23:28don't use pie charts. What is happening?
8:23:32This is a incredibly large pie chart. Oh
8:23:35my gosh. I am making this um this is
8:23:38becoming a problem. There we go. Uh and
8:23:41what I actually wanted to do was label
8:23:42the uh genre as well as I've been doing
8:23:45in all the other ones. Uh and we'll
8:23:48label this. Now look, whether you are a
8:23:52fan of pie charts or not, you have to
8:23:55understand that people use them. Uh some
8:23:57people just like how they look. And for
8:24:00certain data, it can do well. For things
8:24:04that have a lot of different um
8:24:06groupings or categories, it usually
8:24:08isn't super great. Uh but it does give
8:24:11you some type of order of things, give
8:24:14you a quick glance, and people use them,
8:24:16right? So, let's not pretend like
8:24:20it's like the the the hideous stepchild.
8:24:22All right? People use it. People have it
8:24:25in their dashboards and their
8:24:26visualizations all over. So, it's best
8:24:28to just know what they look like, know
8:24:30how to do them, know um how to use them
8:24:32best. Again, I'm not a super huge huge
8:24:35fan of it myself. I've used it once or
8:24:38twice, but one to look out for. And
8:24:41again, you can come over to here and use
8:24:43is called a box and whisker plot. Um
8:24:46it's good for these large um
8:24:49distributions. You know, this is like
8:24:53the median, upper, upper, lower, lower.
8:24:55I don't use these a lot, but I know a
8:24:57lot of people who love them. Something
8:25:00to just look at and consider, mess
8:25:03around with it a little bit. It's
8:25:04pretty, I think, straightforward, and it
8:25:07does give you some good insight into
8:25:09your data if you know how to use it.
8:25:11Now, there is one last one that I want
8:25:13to show you. I'm just going to create it
8:25:14on a new sheet. Make it easy. Uh, we'll
8:25:18do year here. We'll do sum of Let's do
8:25:22NA sales. Why not?
8:25:25And we are going to make this like this.
8:25:28Now, it's very similar to a line chart.
8:25:31But when we break it out by the genre
8:25:34and we add some color, you know, it's
8:25:37just a different way to visualize this
8:25:40information. You can uh you know,
8:25:43potentially add some stuff in here like
8:25:45some labels if you uh want to depending
8:25:48on how it looks for you. But this is
8:25:51just another way to visualize the data.
8:25:53So, wanting to give you guys some
8:25:55options, wanting to give you some things
8:25:58that you might want to look at if you
8:26:01haven't already used these before. These
8:26:03are ones all every single one that I've
8:26:04showed you are ones that I've at least
8:26:06used once. Um, this one I maybe have
8:26:09literally only used once, but the first
8:26:12ones that I showed you, the ones I
8:26:13pointed out as the ones that I really
8:26:15wanted you to know are great
8:26:18visualizations to learn how to use and
8:26:21learn how to make useful for the data
8:26:23that you have. With that being said,
8:26:24that is all that we are looking at in
8:26:26this video. Again, I tried to keep it
8:26:28super easy. Just wanted to show you some
8:26:30different visualizations, the data that
8:26:31you can use to get those visualizations,
8:26:33and just some other options in case you
8:26:35wanted to get a little bit uh
8:26:37spontaneous, a little bit out there, a
8:26:39little bit funky uh to show your boss or
8:26:41something like that. Thank you guys so
8:26:43much for watching. I really appreciate
8:26:45it. If you like this video, be sure to
8:26:47like and subscribe below and I will see
8:26:49you in the next video.
8:26:52[music]
8:26:58>> [music]
8:27:05>> What's going on everybody? Welcome back
8:27:06to another video. Today we're looking at
8:27:08joins in Tableau.
8:27:13[music]
8:27:15Now before we get into the tutorial, I
8:27:17want to give a huge shout out to today's
8:27:18sponsor and that is Udemy. They are
8:27:20having a massive Black Friday sale.
8:27:22Everything is about 85% off. So, if
8:27:25you've been looking at a course, now is
8:27:26the time to buy it. If you are looking
8:27:29at learning and taking a actual full
8:27:31Tableau course, there are fantastic ones
8:27:33on Udemy that I have taken myself. So,
8:27:35be sure to go and check out Udei while
8:27:37they're having this huge sale. I will
8:27:38include a link in the description if you
8:27:40want to check them out. Now, let's get
8:27:42into the tutorial. All right, let's get
8:27:43started. And first, we're going to start
8:27:44off in Excel. I'm going to kind of walk
8:27:46you through the data that we're working
8:27:47with and then we're going to put it into
8:27:49Tableau and I'm going to show you how to
8:27:51do all those joins in Tableau. So the
8:27:53first table that we have is this
8:27:55demographics table. We have employee ID,
8:27:57name of employee, employee age and
8:27:59employee gender. Now look right here
8:28:01because this will be important uh going
8:28:03forward. In the demographics table, we
8:28:06have 10 uh individuals and they each
8:28:09have an employee ID. Now, when we go to
8:28:11the job title, we have our employee ID,
8:28:14employee name, and the job title. But
8:28:17this one is missing Ryan Howard is
8:28:19missing his employee ID. And then the
8:28:22very last one, there are only seven
8:28:25employee IDs and no names. Um, and so
8:28:28we're going to use all of that and I'm
8:28:29going to show you how to actually do the
8:28:31joins in Tableau. Tableau does a really
8:28:34fantastic job of visualizing it for you,
8:28:36so it takes a lot of the guesswork out.
8:28:38Um, I am going to include a link to my
8:28:40joins video in SQL because these two are
8:28:42very closely connected and and if you
8:28:45understand how the joins work in in SQL,
8:28:48you'll understand how the joins work in
8:28:49Tableau, it's almost the exact same
8:28:52thing. So, with that being said, let's
8:28:55jump over to Tableau. So, I'm going to
8:28:57pull this up and go right over here. And
8:29:00now we have uh where we can connect to
8:29:03our data. And so, we're going to click
8:29:05Microsoft Excel. I'm going to scroll
8:29:07down here to Tableau joins file. I'm
8:29:10going to open this up. And I have it
8:29:11open so I can't use it. So, let me get
8:29:13rid of that and let's open it again.
8:29:16Perfect. So, now what we're going to do,
8:29:19and I'm going to show you how to
8:29:20actually open up the joins um in a
8:29:22second. But what you need to understand
8:29:23is when you first come here, Tableau
8:29:26doesn't automatically allow you to to
8:29:29use the joins. They use something called
8:29:31relationships. And there are joins on
8:29:33the back end, but they call it
8:29:35relationships because they are inferring
8:29:36all of these things. They're trying to
8:29:38go in and make that inference for you.
8:29:40So, it takes a lot of the work off of
8:29:42you. And most of the time that works.
8:29:44And and you know, you just plug these
8:29:46two things in here like a demographics
8:29:48and the job title. And it is going to,
8:29:52you know, help you build those what they
8:29:54call relationships. And you can click on
8:29:57this and learn how the relationships
8:29:58differ from joins. Again, there's not a
8:30:00huge difference, but it's not as
8:30:02customizable, and you can't as easily do
8:30:05left joins or full joins or all these
8:30:07things that we're about to look at. So,
8:30:09uh, I'm going to take this one off. And
8:30:11what we're going to do to actually be
8:30:13able to look at the joins and and choose
8:30:16what joins we want to use is we're going
8:30:17to do this dropdown. We're going to
8:30:19click open. And so, now we are in a
8:30:22place where we can actually create the
8:30:25joins. Uh, and again, it's just much
8:30:28more customizable. And so, um, back when
8:30:31I was using Tableau regularly, I would
8:30:35use the relationships when it was pretty
8:30:36simple and straightforward because
8:30:38almost they almost always got it right.
8:30:40But, uh, you know, the joins, it it just
8:30:44makes more sense in the way it
8:30:45visualizes it for me. So, most of the
8:30:47time I'd be using the joins. So, let's
8:30:50pull over this job title right here.
8:30:53And it's going to make this connection.
8:30:55Now, before if you remember just about,
8:30:57you know, 30 seconds ago when it
8:30:59connected them, it was just a line. And
8:31:01and so it gave us this option down here
8:31:03to kind of edit the relationship. But
8:31:05now it's giving us this visualization.
8:31:07And so let's click on it really quick.
8:31:09And what is going to come up is the
8:31:11different types of joins that you can
8:31:13do. You can do an inner join, a left
8:31:14join, a right join, and a full outer
8:31:17join. And then you can actually choose
8:31:20the different uh data sources and how
8:31:22you're connecting them. So again, um I'm
8:31:25going to walk through a little bit of
8:31:26this, but I think the SQL video that I
8:31:29did on this shows it so well. Um I would
8:31:32just highly recommend using that. Um and
8:31:34I recommend learning SQL, too.
8:31:36[clears throat] So, you know, two birds,
8:31:37one stone. So, I'm going to get into
8:31:40each of the joins, how they work, what
8:31:43data is going to be displayed. Um and
8:31:45these visualizations are really going to
8:31:47be helpful, and I think that it's it's
8:31:50just nice that they have it because it's
8:31:51a little reminder. Okay. um you know
8:31:54this is what this joint is or this is
8:31:55what that joint is. So super super
8:31:57simple. So right now we have the
8:31:59demographics table and we have the job
8:32:02title table. And so what it's doing
8:32:04right now and let's get rid of this.
8:32:06What it's doing right now is it's doing
8:32:07an injoin. And so it's pulling
8:32:10everything that overlaps if it matches
8:32:12on the employee ID and the employee ID.
8:32:16And so right now you only see one
8:32:18through nine. But if you remember in the
8:32:20demographics table, we had uh 1,000 all
8:32:23the way through 10. So where is that
8:32:2510th one? Well, the 10th one is not
8:32:27there. And that is because in this job
8:32:30title employee ID, it only went up to
8:32:331009. And then Ryan Howard just didn't
8:32:37have an employee ID in there for
8:32:38whatever reason. So that data is going
8:32:39to be missing. Now when you are using
8:32:42actual data sets very large data sets
8:32:45which we will use in the next video when
8:32:47we walk through an entire project
8:32:50um when you use large data sets this can
8:32:53be the difference between clean data and
8:32:56very wrong data and and visualizing it
8:32:59correctly and showing completely wrong
8:33:01numbers. And so you really need to be
8:33:03sure you understand how your data works
8:33:05together when you're doing these joins.
8:33:07So how can we fix this? How can we um
8:33:11make it to where we can see all of the
8:33:13data? Well, right now we're only making
8:33:15it to where if the employee ID is equal
8:33:17to the employee ID. So, we only are
8:33:19going to see through 1009 and through
8:33:211009. We're never going to see Ryan. So,
8:33:24there are two different types of joins
8:33:25that we could do to make it see it. And
8:33:28then there's something else that we can
8:33:29join on to where we can see that data.
8:33:31The first one that we can look at is the
8:33:33right uh join. And what this does is
8:33:36it's going to take everything that is
8:33:38the same, but also everything from this
8:33:41job title table regardless of if it has
8:33:43a match in the demographics table. So
8:33:45it's pretty, you know, this
8:33:46visualization does it all. It's going to
8:33:48show everything in the right table
8:33:49regardless. And it's only going to show
8:33:52things from this table if there's a
8:33:54match. So let's try this one. And we
8:33:56should see Ryan Howard in the job title
8:33:58table. So let's click on it. And if we
8:34:01scroll down, there's going to be null,
8:34:02null, null, null, null until we get to
8:34:06over here where we now have the data
8:34:09that we had in that actual table. But
8:34:12again, this wasn't a match. And so we
8:34:14weren't able to see that data. So this
8:34:16gives us a way to where we can see all
8:34:19of it. Um, all everything from that
8:34:22right table, this job title table. And
8:34:24now we're going to click on the full
8:34:25outer. Now, the full outer is going to
8:34:28take everything from both regardless of
8:34:30if there is a match at all. And so,
8:34:33right here, you're going to see Ryan
8:34:34Howard and Ryan Howard. Now, why are
8:34:35there two different rows for it? Well,
8:34:37because in the demographics table, there
8:34:40was an employee ID. So, we're seeing the
8:34:42employee ID, Ryan Howard, his age, and
8:34:44his gender. And over here, there was no
8:34:48match, right? But in the job title
8:34:51table, again, this one didn't have an
8:34:53employee ID. And so we we are going to
8:34:55be able to see this data, but over here
8:34:59it has no match. And so that's why it's
8:35:02showing us two different rows is because
8:35:04there was no connection. There was no
8:35:06match there. That's what a full outer
8:35:08join is going to do. Now, just for uh
8:35:12the purposes of seeing what this one
8:35:13does as well, we have the lefth hand
8:35:15table. Um and now we are able to see the
8:35:19110 or or that we didn't see before. um
8:35:23and it's putting in nulls over here
8:35:25because there's no match. So that's that
8:35:27is um what we have so far. Now like I
8:35:31said just a second ago, there is a way
8:35:33that we can do this without using the
8:35:36employee IDs. We're allowed to use a
8:35:38different join clause. Now there is the
8:35:41name of the employee in both of them.
8:35:42This one is called name of employee and
8:35:44in the job title it's called employee
8:35:46name. They don't have to have the same
8:35:48column name in order to join it. You can
8:35:50do whatever you want. So, I'm going to
8:35:54get rid of this one.
8:35:56And now we are only tying it on the
8:35:59employee name. And let's do an inner
8:36:02join. And it should be basically
8:36:06everything um except the only piece of
8:36:08data that wasn't filled in, which is
8:36:10that 1,0 over on the job title table.
8:36:14And so this way was a slightly different
8:36:17maybe uh less thought of way because
8:36:19normally you do it if there's an ID you
8:36:21go on the IDs but because we had a lack
8:36:26of data for in in one of the tables in
8:36:28the job title table we decided to use a
8:36:31different column to to join on and now
8:36:34we're able to look at all the data
8:36:36together. So, super quickly, that is an
8:36:40inner join, a left joint, a right joint,
8:36:42and a full outer join. And it's pretty
8:36:44easily visualized here. And you're able
8:36:47to uh change what you're joining on
8:36:50right here. But you're also you can do
8:36:52multiple. So, if we want to do the
8:36:54employee ID and the employee ID, you can
8:36:56do that as well. And you can keep going
8:36:58as as many as you'd like. Um, and right
8:37:03here, you can change some of these
8:37:05things. Uh I don't there aren't a lot of
8:37:08use cases for this. Um but you know you
8:37:11can absolutely do this um and mess
8:37:13around with this as seen. I'm not going
8:37:14to go through it in the tutorial because
8:37:16again 95 plus% of the joins you're
8:37:20doing, you're going to want to do it to
8:37:21where this equals this. Um and if you
8:37:23want to get into where it doesn't equal
8:37:25or or all these other things, which is
8:37:27more complicated, I think it's much
8:37:30better to learn that in SQL. Uh that's
8:37:32my personal preference. And so, um,
8:37:34again, all in the SQL tutorial if you
8:37:36want to check that one out. So, you're
8:37:37able to join on multiple things. Now,
8:37:39let's get rid of that one because we can
8:37:42actually bring in this salary one as
8:37:44well. And what you'll see right down
8:37:46here
8:37:48is that we have our employee ID and this
8:37:51is all coming from the demographics. So,
8:37:52employee ID, name of employer, employee
8:37:55age, employee gender. Then right over
8:37:58here, we have the job title table. So
8:38:01employee ID, job title, employee name,
8:38:04job title, and then right over here was
8:38:08or is our salary table. And so we have
8:38:10employee ID, salary, and employee
8:38:12salary. So again, this is a way that you
8:38:15can put all of this data into one place.
8:38:17And in just a second, we'll go into the
8:38:19worksheet right down here. I'm going to
8:38:21show you kind of how it looks because it
8:38:23looks a little bit different um than
8:38:24previous tutorials. And so I want to
8:38:27show you how that actually all works
8:38:29together. Um, but again, you can create
8:38:32these joins um as well and do the exact
8:38:36same thing that we just looked at and
8:38:37customize the joins, customize what
8:38:39you're what you're um uh joining on. And
8:38:43then you have your finished product. And
8:38:45so right now we have our demographics
8:38:47plus Tableau joins file. And we can
8:38:50rename that if we want. I'm going to
8:38:52call this um demographics plus joins
8:38:56demo and click enter. And so now that is
8:39:00saved. So now let's go down to the go to
8:39:03worksheet. We're going to click on that.
8:39:05And so up here on our left side, this
8:39:07may look a little bit different than it
8:39:08normally does. Um because it's broken
8:39:11out um on the measure names and the
8:39:13measure values. It's broken out by the
8:39:15tables that they were joined on. So, we
8:39:18can pull in the employee gender now, and
8:39:20we can pull in the employee name now.
8:39:23Um, and [snorts] we can pull in the
8:39:24employee ID again if we want to from the
8:39:27job title table, and we can pull in the
8:39:30employee ID from the salary table. We
8:39:31could do that if we wanted to. It makes
8:39:33no sense uh uh for actually creating any
8:39:35visualizations, but you know, you can do
8:39:37that. And so, you probably you wouldn't
8:39:39be able to do that if you hadn't joined
8:39:40these together. And so down here in the
8:39:43measure values, the values that we have
8:39:45are from the demographics table and the
8:39:47salary table. All of the um all of the
8:39:51stuff from the employee title, none of
8:39:54those things were um values. And so we
8:39:57can't use there are going to be no
8:39:58values down here. And so really quick,
8:40:01let's take the name of the employee.
8:40:03Let's take their salary. Sure, why not?
8:40:06Um let's order that.
8:40:10Let's take the employee salary.
8:40:13We'll do color.
8:40:15And uh let's expand this out a little
8:40:19bit.
8:40:21Maybe one more time. Oops. Just like
8:40:24that. And there you go. So that is how
8:40:26you do joins in Tableau. And I think
8:40:28Tableau does a really fantastic job of
8:40:30making it pretty simple. They have the
8:40:32different types of joins when you click
8:40:34on that that join button. And it shows
8:40:36you the inner and the left and the right
8:40:37and the full outer. and they make it
8:40:39pretty simple. Um, and and it's just
8:40:42really useful to be able to see that
8:40:45while you're creating it and see the
8:40:46output below like we just did a second
8:40:48ago. It just makes it so simple to
8:40:51create those joins and then just keep
8:40:52going because you already know what your
8:40:54output is going to be and you can kind
8:40:55of mess around with it and make sure
8:40:57you're getting the data that you need.
8:40:58In the very next video, we're going to
8:41:00be doing an entire project in Tableau.
8:41:02We're going to be using a lot more data
8:41:04and it's going to be a a complete
8:41:06project that you can add to your
8:41:07portfolio and it's going to be a really
8:41:09good time. So, I hope that you join me
8:41:11for that one. I appreciate your time. I
8:41:13hope that this was helpful. Thank you
8:41:15guys so much for watching. I really
8:41:16appreciate it. If you like this video,
8:41:18be sure to like and subscribe below and
8:41:20I'll see you in the next video.
8:41:23[music]
8:41:33What's going on everybody? Welcome back
8:41:35to the Tableau tutorial series. This is
8:41:36our very last video in the series and
8:41:39today we'll be doing an entire project.
8:41:43[music]
8:41:46Now, if you watching this video, I hope
8:41:48that you watch the other four videos in
8:41:49this series just so you can get the
8:41:51basics down. You kind of know what
8:41:52you're doing. Uh this won't be a crazy
8:41:55hard project. This is a beginner
8:41:57tutorial series, so I'm trying to make
8:41:58this super easy so you can follow along.
8:42:01Nothing super complicated, I promise.
8:42:03And if you were wanting to go above and
8:42:04beyond and just make a lot of different
8:42:06dashboards or try a lot of different
8:42:07things, there's a ton of data in here.
8:42:10And so I'll show you some of the things
8:42:11that I would do, you know, as we go
8:42:13through it of the things that I would be
8:42:14looking at and some of the different
8:42:15visualizations that I might do as well.
8:42:18But again, in this video, we're going to
8:42:19be singing to a lot of the basics. But
8:42:21I'll switch over to my screen in just a
8:42:22second. and I will show you the final
8:42:23product and then we will actually walk
8:42:25through step by step of how to do the
8:42:27entire dashboard and at the end you
8:42:29should have a completed project that you
8:42:30can add to your portfolio or you know
8:42:32just share on LinkedIn if you want to do
8:42:34that as well. With that being said,
8:42:35let's jump over to my screen and let's
8:42:37get started. All right, so let's get me
8:42:38off screen and show you what we're going
8:42:40to be working on today. This is the
8:42:41final dashboard that we're actually
8:42:43going to be building. And so it it's
8:42:45nothing crazy, right? I'm sure you have
8:42:47seen all of these things before. Um, and
8:42:49I'm just going to help you kind of build
8:42:50it out, show you what to do, the buttons
8:42:53to click. Um, and it's really going to
8:42:55be a simple walkthrough. By the end of
8:42:57this, you should be able to do all these
8:42:58things very easily. And I highly
8:43:01encourage looking at the data and
8:43:03looking at these visualizations and
8:43:04seeing what else you can do with it.
8:43:06There's a lot of different colors, a lot
8:43:08of different visualizations um, that you
8:43:10can do with this data. I'm just showing
8:43:12you this today. And so the more you go
8:43:15out there and the more you do this on
8:43:16your own and you mess around with stuff
8:43:19and and choose different things and see
8:43:20how it all works, the better you're
8:43:22going to get. And so I highly highly
8:43:23encourage doing that. Uh so what we are
8:43:26going to be working with today is an
8:43:28Airbnb data set. I'm going to show you
8:43:30that in just a second and I'm going to
8:43:32show you the data and we're going to
8:43:34just jump right into it. All right. So
8:43:36this is the data set that we are going
8:43:37to be using. This is the Seattle Airbnb
8:43:40open data set. And let's scroll down
8:43:43really quick. Um there's three different
8:43:45CSVs in here. And so this is some of the
8:43:48data that we're going to be working
8:43:49with. Um some date and listings, and
8:43:52some pricing. And then there's the
8:43:54actual listing that shows um the actual
8:43:57street address, the location, the price,
8:44:00the bedrooms, all of these good stuff.
8:44:02And then there's a reviews.
8:44:05Um, and it has, you know, some comments
8:44:07and, you know, talks about some of the
8:44:09reviews. So, this is what we're going to
8:44:12be working with, but you don't have to
8:44:14go in here and download it. I have
8:44:16already combined all of these CSVs into
8:44:19one. I've put it on the GitHub, so I'll
8:44:22have a link below, so you can just click
8:44:23on that and you don't have to do all the
8:44:25stuff that I did to get this set up. Um,
8:44:27just so you know, this is from 2016, so
8:44:29this data set is a little bit old. If
8:44:32you want to, you can come right here and
8:44:34I will leave this link as well and you
8:44:36can get the data set from you know what
8:44:38is this a couple weeks ago. Uh this is
8:44:41they they are continuing to update this.
8:44:43This is always updated and so you can go
8:44:44ahead and download these but some of
8:44:46these are the CSV.gz. Um so you may need
8:44:49to like convert it. I don't want to go
8:44:51through that process um on you know in
8:44:54the video and so I am just going to go
8:44:57with what is literally in Kaggle um and
8:45:00use that. But if you want to have an
8:45:02updated one for your project, I just
8:45:04advise you to go in here and grab it
8:45:06yourself and that should be perfectly
8:45:08good. So go ahead and download the data
8:45:11set from the GitHub and we should be
8:45:13good to go. So this is the Excel that I
8:45:15was just talking about. This has all of
8:45:17our CSVs in one place. This is, you
8:45:19know, an Excel workbook. So in this
8:45:22reviews, actually, let's start with the
8:45:24listings cuz that's kind of where it all
8:45:25stems from. Uh we have our listing and
8:45:28the date or the data in here is um you
8:45:31know really extensive. There's a lot of
8:45:33data in here. So let's get over really
8:45:35quick. Um the listing refers to the
8:45:38actual home that they're renting out the
8:45:41Airbnb. So it shows their location.
8:45:45Um and there's a lot more location
8:45:47information over here. I'm getting into
8:45:48it in in just a second. So, there's the
8:45:50neighborhood, the city, state, um, zip
8:45:53code, all stuff that, you know, may be
8:45:55useful. There's a latitude and
8:45:57longitude.
8:45:59It shows what type of property it is, so
8:46:01that's really good. Um, right over here,
8:46:04it has, you know, how many bathrooms,
8:46:06bedrooms, and beds. Um, you know,
8:46:08sometimes if it's a five bedroomedroom
8:46:09house, it's has seven beds. So, that's
8:46:12why there's those two different um
8:46:14fields. I don't know if you're familiar
8:46:16with Airbnb and and you know what they
8:46:18have on there, but just something to
8:46:20note. Uh they have the price. This is
8:46:22the price per day. There's a weekly
8:46:24price, a monthly price, and if there's a
8:46:26deposit needed, uh and then a cleaning
8:46:29fee as well. So, a bunch of financial
8:46:32data that's, you know, super useful. We
8:46:34go into it a little bit, but there's so
8:46:36much you can do with that. Um, you know,
8:46:38if you want to dig into that, and that's
8:46:40kind of it. The rest of it's pretty uh
8:46:42pretty useless. Um, and there's a lot
8:46:44of, so there's so much data in here,
8:46:45almost, you know, more than half by far
8:46:48is nothing you would put in any type of
8:46:50visualization. Um, and this is pretty
8:46:52common. Uh, you're not going to get
8:46:55data every column where you're going to
8:46:57be able to use it. A lot of times it's
8:46:59just a lot of useless junk. And so you
8:47:01have to know what you're looking for and
8:47:02know uh, you know, what's actually
8:47:04useful. So that's the listing. Then we
8:47:06have reviews. Now
8:47:09what's really a little bit confusing in
8:47:11here and something that you just need to
8:47:12kind of understand about the data u and
8:47:14something that if you're if you get a
8:47:16data analyst job you need to understand
8:47:18your data because it's very easy to come
8:47:20in here and say okay there's an ID ID
8:47:22field and here's an ID field. So that
8:47:25means that those are the same. Well not
8:47:27in this case um this ID field is
8:47:29actually the reviews ID not the reviewer
8:47:32ID that refers to like the person. This
8:47:35is the reviews ID. this listing ID is
8:47:38the actual ID right there. So, really
8:47:43important to note. Um, and then the line
8:47:47and so then they just have their comment
8:47:48there, what they left as a review. And
8:47:50then on the calendar, um, I don't know
8:47:52why I'm scrolled down. Uh, we have this
8:47:55listing ID again. So, again, that
8:47:57listing ID is equal to the ID in this
8:47:59listing table. And we have a date and a
8:48:02price. So, this refers to a specific
8:48:03location. and on this day they got $85
8:48:07for it. Somebody rented it out. Um, and
8:48:10so then there's these like T's and Fs.
8:48:12Um, let's try to find a blank one really
8:48:14quick. Here's a blank one. So there's
8:48:16these T's and Fs. Uh, the T means that
8:48:20it was taken. Um, the F means that it's
8:48:22vacant. Uh, I don't know exactly what it
8:48:25means. Uh, what the TF means, but that
8:48:27we can deduce that much from this. And
8:48:29so you can see when and how much this
8:48:32person was making or this home made uh
8:48:34in that time. So really really good data
8:48:38in here. There's a lot to work with. Um
8:48:41and and so we're just going to be kind
8:48:42of I'll give you a little bit of a use
8:48:44case for it in a second and then we're
8:48:46going to start trying to answer some of
8:48:48those the building out some of the
8:48:50visualizations for that use case. Uh
8:48:53again, you could have 20 different use
8:48:55cases for this data or more um honestly
8:48:58for this data where you could build out
8:48:59different dashboards and different
8:49:00reports literally with just this data,
8:49:03but you know, we're doing a pretty
8:49:05general broad project and so it's hard
8:49:08to answer all of them. So, let's jump
8:49:11over to Tableau. We're going to get
8:49:13started on this and we are going to
8:49:15build out everything. All right, so
8:49:18let's come right here. Uh this is a
8:49:20Microsoft Excel. We'll open that up. Do
8:49:24this one. We will open it
8:49:28and give it just a second. Says it's
8:49:30executing the query. It's pulling the
8:49:32data in. All right. So, we have our
8:49:36calendar, our listing, and our reviews.
8:49:38Those are the different tabs at the
8:49:40bottom. We're going to start with the
8:49:41listing. This is the the kind of the
8:49:44main one has um you know the there's I
8:49:47didn't show you, but there's about 3,600
8:49:50locations that they had in there. Uh
8:49:54let's just have it update automatically.
8:49:57I don't know why we need to click on
8:49:59that, but um so we have this listings,
8:50:02we have [clears throat] our uh calendar
8:50:04and our reviews.
8:50:06What we're going to do is we're going to
8:50:07come in here and we're going to open it
8:50:09as we did in our very last video uh for
8:50:12the joins. So, now that we've opened it,
8:50:14we can kind of go in here and we can do
8:50:16the joins as um as needed. And so, let's
8:50:21go over here and we're going to uh let's
8:50:24start with calendar.
8:50:26Put it right there. That was super slow.
8:50:28I apologize.
8:50:31[clears throat and cough] All right,
8:50:32let's wait for it to
8:50:36get the data. Start setting everything
8:50:38up.
8:50:41Did not think it would take this long. I
8:50:43apologize.
8:50:47No, take your time. So, let's click on
8:50:50here. And right now, it has the uh the
8:50:53join based on the price, which obviously
8:50:56is not going to work. Um, and if you
8:50:58remember, there is no ID in this
8:51:01calendar. It's just the listing ID. Um,
8:51:03we can actually look right here. There's
8:51:05just the listing ID. So, we're actually
8:51:06going to put listing ID is equal to ID.
8:51:12And right down here, we can see that we
8:51:14have a lot of of well, you can't see it
8:51:17u, but we show that there is a lot of
8:51:19data. Um, and so we know that that is
8:51:23correct. We know that that is now
8:51:24pulling in data correctly because it's
8:51:26showing up down here. So, that's a good
8:51:28thing. Now, in this listings, there are
8:51:32about 3,600
8:51:34um about 3,600 listings. And so,
8:51:39that all the data that's in listings is
8:51:41going to be in there. But on the
8:51:43calendar, because we converted from a
8:51:46CSV to an Excel workbook, it isn't able
8:51:48to store as much information. So, some
8:51:49of the ones in calendar may have gotten
8:51:51cut off. So, we can just keep it this
8:51:53inner join because we know that if it's
8:51:55in listings, it's going to be in
8:51:56calendar. We know that it if it um there
8:52:00may be some in calendar that aren't in
8:52:02listings. So, if we really um you know,
8:52:06if we really really wanted to, we could
8:52:08do a full outer or something like that.
8:52:10I I haven't really thought through this
8:52:12as I'm talking through it in my head,
8:52:13but we know that uh everything that's in
8:52:16listing is going to be in calendar. Uh,
8:52:18and so, you know, we don't really need
8:52:20to do anything other than an inner join.
8:52:24And we can also pull in these reviews.
8:52:29And it's going to do the same thing as
8:52:30before where it's just kind of pulling
8:52:31in the data. And it defaults to ID
8:52:34equals ID. Now, we know that that is not
8:52:37correct um because the ID in here is
8:52:40referring to the review ID. We need to
8:52:42go to the listings ID. So, we need the
8:52:44ID be able to, you know, be part of that
8:52:47listings ID. If we do the ID,
8:52:51it goes down to 2555
8:52:54rows. If we do how it's supposed and
8:52:56there because that's just, you know,
8:52:57it's random luck. There happen to be
8:52:58some numbers that are in both fields um
8:53:01that tie together. If we do the correct
8:53:04one where we hit the listing ID, it
8:53:06bumps it up to I think 2,373,000.
8:53:09Oh, maybe more than that. Uh 23 million
8:53:13rows, right? A lot lot lot more. And so
8:53:16it's super important to get these joins
8:53:18right to tie them together on the right
8:53:19fields. If you just do it based off what
8:53:21Tableau tells you because it has that
8:53:23automated um you know it goes into these
8:53:27fields and says okay these are the same
8:53:28exact column name. So they're most
8:53:31likely going to be what you're looking
8:53:33for. Well, it was incorrect in this
8:53:35point. So it's really important to check
8:53:37those things and make sure you're
8:53:38pulling in the right data. Again we're
8:53:39going to keep it that inner join. Um,
8:53:42you know, if you wanted to, you know,
8:53:44try to see if there's any other data
8:53:45that correlate. We're keeping it simple
8:53:46today, but sometimes you need to join on
8:53:48multiple things. Uh, so just, uh, uh,
8:53:52you know, a tip. So, let's get out of
8:53:54here. Um, and we are good to go. So,
8:53:56this is our listings plus Tableau full
8:53:59project. That's what we'll that's what
8:54:01we'll be working with. Um, and we we
8:54:03were able to tie all three of these um,
8:54:06you know, I guess you'd call them tables
8:54:08or sheets or whatever you want to call
8:54:09them. we were able to tie them together.
8:54:12So, let's go over here to our first
8:54:14worksheet. Uh, and let's see.
8:54:18All right. So, this says Tableau public
8:54:19only works with less than 15 million
8:54:21rows of data. We have 23 million rows of
8:54:23data. That is, uh, that's a problem. Um,
8:54:26and when I did this before, it didn't do
8:54:29that. So, I, you know, we're going to
8:54:31work through this together. So, this is
8:54:33date reviews. I believe this is date for
8:54:36um
8:54:39this is date for the calendar which is
8:54:43going to be a lot of rows of data and so
8:54:45I'm sure that's part of it. Let's see.
8:54:50Let's do years.
8:54:52We only want 2016. Oops. We only want
8:54:562016.
8:54:59Let's do Okay,
8:55:03let's see what that does. Let's see if
8:55:04that gets us under what we need. Um, we
8:55:06only want 2016 data anyways.
8:55:10So, if it's in 2017, we were going to
8:55:12take it out. Um, anyways, so we'll see
8:55:15if that gets us underneath. I have
8:55:17absolutely no if this take ends up
8:55:19taking like 20 minutes, I will just cut
8:55:22it and you know, you won't have to wait
8:55:24as long as I'm waiting. So, let's see
8:55:26how long it takes.
8:55:31All right. So, it took about 20 minutes
8:55:33and it did absolutely nothing. Um,
8:55:37one thing I do know is that we don't
8:55:40actually use this review tables at all.
8:55:42Uh, this is just for demonstration
8:55:44purposes. So, we're going to remove
8:55:46that. And let's see if that helps us in
8:55:50any way.
8:55:53Because if it does, we're just going to
8:55:55keep it as is. Um, you know, the reviews
8:55:57table is really just for demonstrating
8:56:00how to do the joins. Uh, but we weren't
8:56:03actually using any of the data from any
8:56:04of the visualizations,
8:56:05although you could.
8:56:08Again, I'm going to see how long this
8:56:10takes. Uh, and I'll cut ahead.
8:56:15All right. So, that worked uh,
8:56:17perfectly. It apparently took out all
8:56:19the data that we needed or all the rows
8:56:20that we needed to get under that level.
8:56:22Again, I was just doing that to show you
8:56:24the the that joins how you needed to
8:56:26change the columns to make sure that it
8:56:29joined properly. We don't actually use
8:56:31it for any of the visualizations. So,
8:56:32their end product is going to be totally
8:56:34fine. I don't know why uh this didn't
8:56:37happen to me when I when I created this
8:56:38whole thing already. Um so, I'm just
8:56:41going to move forward because uh I make
8:56:43mistakes. So, uh let's keep moving. The
8:56:47first one that we are going to make is
8:56:49that uh is that colorful one. I'll
8:56:51probably pop it up on screen so you can
8:56:53see it. Uh well, if I remember, I'm
8:56:55going to pop it up on screen. Um it's
8:56:56the colorful one. It's the price by zip
8:56:59code. So, we're going to be looking at
8:56:59these zip codes and kind of see um you
8:57:02know, how expensive
8:57:04is each zip code. Um and before we
8:57:08actually start, I just remembered I want
8:57:11to talk to you about the use case for
8:57:12this data.
8:57:14I want to imagine you to imagine that
8:57:16you're working for somebody and they're
8:57:17like, "Hey, where, you know, I want to
8:57:20start an Airbnb business. I want to know
8:57:22where I should go. Where should I buy up
8:57:25buy a home, put it up on Airbnb, and
8:57:28start renting it out? Where's the best
8:57:29place? You know, what are some of the
8:57:31factors that I should be looking at?"
8:57:33Uh, and so that's kind of what our use
8:57:35case is. So, we're gonna the some of the
8:57:37things that he cares about are things
8:57:38like bedrooms, um, location, which is
8:57:42really important, and how much price
8:57:44he's actually going to get, how much
8:57:46money can he charge. And so, he's trying
8:57:48to optimize that to make sure that
8:57:50whatever rental he gets, he can make a
8:57:52the most profit from instead of choosing
8:57:54something that, you know, he thinks
8:57:56would work, but, you know, in the end,
8:57:57he's actually not making that much
8:57:58money. So, those things are important.
8:58:01So, that's our use case. We're trying to
8:58:02help this guy out, help him find a
8:58:05really good Airbnb. Um, so let's take a
8:58:08look at these zip codes real quick. We
8:58:09have uh quite a few of them. And there's
8:58:13one that's null. Uh, we'll exclude that.
8:58:15Or if if it doesn't have a zip code,
8:58:16we'll just exclude those because they're
8:58:18not going to show up on the these
8:58:19visualizations anyways. Um, and so we
8:58:22want to look at the price. So we just
8:58:24want to find uh the price, which should
8:58:27actually be down here,
8:58:29and not the sum. Uh,
8:58:33no. We want to look at the average
8:58:36price. And let's order that. This is
8:58:40great. Um, so this is the most expensive
8:58:42one. Uh, zip code 98134 at $26
8:58:47uh per
8:58:49for the average price. Uh, but let's
8:58:51give that some color really quick. Let's
8:58:54uh Where's the zip code? It's up here.
8:58:56So, let's take that zip code. We're
8:58:58going to put it right over here. We're
8:58:59going to do color. and it's going to
8:59:01give it some uh assorted colors. Now,
8:59:04these colors are gonna um when we do the
8:59:06map just a little bit, these colors will
8:59:09um match what we're doing in there. And
8:59:11so, you know, I I like to try to color
8:59:14coordinate things. Um we're not doing
8:59:16going too crazy with the colors today.
8:59:18So, this is our very first
8:59:19visualization. Congratulations. It is uh
8:59:21it is complete. So, uh, we can label
8:59:25this one. And we can just do
8:59:28price by zip code. And I'll make that
8:59:34bold. I don't know. I usually like it
8:59:36bold. We'll apply. We'll do like that.
8:59:38And boom. First one is done. Uh, and
8:59:41this is our starting place to say, uh,
8:59:44hey, person who's looking to buy this
8:59:46Airbnb, here are the zip codes where
8:59:49they are able to charge the most, um,
8:59:52for for their Airbnb. So, let's go over
8:59:55to the second sheet. And we are going to
8:59:57be doing the map. And so, um, map is
9:00:00pretty easy, but it it's pretty easy
9:00:03once you actually get the data that you
9:00:05need. Although there's a lot of
9:00:07different data that you can use for the
9:00:10actual um map right here, you need
9:00:13something that shows um the location and
9:00:16there's a lot of things that show
9:00:18location in here. In fact, they already
9:00:20um provide a latitude and longitude. And
9:00:22then at the bottom, they generated a
9:00:25latitude and longitude from from some
9:00:27different um fields. And then there's
9:00:29just a bunch of different um state.
9:00:31There's um states, there's zip codes,
9:00:35there are uh I think another one I uh
9:00:38yeah, like country. There's a lot of
9:00:40location data in here. So, which one do
9:00:43we want to use? We want to stay
9:00:45consistent. We don't want to deviate
9:00:47from that and start using different um
9:00:49longit long longitude and latitudinal uh
9:00:52coordinates because that could throw off
9:00:54our our results completely. We want to
9:00:56stay consistent with what we're using.
9:00:58So, we actually want to use this zip
9:01:00code. But when we pull it up here, it's
9:01:02going to give us uh basically the same
9:01:04um you know, it's going to show these
9:01:05zip codes, but we're going to right over
9:01:07here, we're going to click on this one.
9:01:09And now it's going to separate them out.
9:01:11So now we have all of these um you know,
9:01:14kind of separated out. What you might
9:01:15get when you first do this, um is it
9:01:18might look like this. You may have to
9:01:20zoom in. Um I know that that happened to
9:01:22me the other time.
9:01:25Let me go to here. That's what happened
9:01:26to me uh just when I first did it. So,
9:01:30uh, know that that may happen. And
9:01:33we want to change the colors the exact
9:01:36same way that we did them before. So,
9:01:37we're just going over here. We're doing
9:01:39color. And these colors do um they do
9:01:45or should match up with the um with the
9:01:48other ones. Let me um exclude this. Let
9:01:52me see if it does. 98134. That's the
9:01:55blue.
9:01:57And right over here, 98134, it's a blue.
9:02:00I I I believe they are going to be the
9:02:02same. Yep. And so, just scrolling back,
9:02:05if you look at the zip code on the far
9:02:07right, uh they are the same. So, if you
9:02:10look at like this section right over
9:02:11here, I I just wanting to make sure I'm
9:02:13not going crazy uh before I get into
9:02:15this and realize I'm not correct at all.
9:02:18So, uh now what we want is, you know,
9:02:21this doesn't really give us any
9:02:22information. If I was just to glance at
9:02:24this map, I would have no idea what
9:02:27you're trying to show me um any
9:02:29information off this. So, we want to
9:02:30show some actual information. So, first
9:02:33thing that we're going to do is we're
9:02:35going to actually add the label to this
9:02:37so that you can see it. You know, when
9:02:39you're going over here and you see,
9:02:41okay, here's this um zip code. Um in the
9:02:45dashboard when we create it, you can
9:02:46click on this. But if you just want to
9:02:49do it visually without having to click
9:02:50anywhere, you'll be able to see, okay,
9:02:5298134, that's right here. So, this
9:02:54location right here is, you know, able
9:02:56to charge a lot of money. It's probably
9:02:58a really nice neighborhood. So, um, and
9:03:01we can back that up by putting the
9:03:05average price. So, these these two
9:03:07visualizations are really they really go
9:03:09hand in hand. We're going to add oops,
9:03:12not the sum.
9:03:14This one needs to be the average. So you
9:03:16go to this measure, the sum, go to
9:03:18average, and there you go. And these
9:03:22should match. So this should be 206.6.
9:03:24Um, I'm looking at the average price
9:03:26right here. And then we go over here.
9:03:2998134 206.6. So this all matches. Um,
9:03:32and we can uh we can actually change
9:03:35that size a little bit if you wanted to
9:03:36actually get it in um get it within each
9:03:40of these things. You know, adjust it as
9:03:43you see fit. I think that's fine right
9:03:45there. Um, no need to
9:03:48mess with it anymore.
9:03:51All right. So, let me see. I think that
9:03:53is everything for this one. I don't know
9:03:54if I want to add anything else. Uh, no.
9:03:58I'm going to keep it how it is. So, that
9:04:00is our second visualization. Again,
9:04:02these ones are directly uh correlated
9:04:06and and you know this there's just
9:04:08different ways to visualize it. This one
9:04:10you can see actually on the map where it
9:04:11is and the average price. This one you
9:04:13can see from highest to low. So again,
9:04:15you know, sometimes when you're doing
9:04:16these visualizations, you're going to
9:04:18have these accompanying um uh these
9:04:22accompanying visualizations in your
9:04:24dashboard. That's very normal. So, let's
9:04:28move over to the third one. And for this
9:04:31third one, um you know, something that
9:04:34our guy was looking at is he's like,
9:04:36"Okay, well, you know, I'm thinking
9:04:38about listing it on Airbnb, but I also
9:04:41want to live in it. So, I want to know
9:04:42the best times to actually um you know,
9:04:46put it on the market for people to be
9:04:48able to use. And so, I was like, "Okay,
9:04:51man. No problem. Uh let's let's take a
9:04:53look at when when are people spending
9:04:55the most money in Airbnbs." And we
9:04:58actually had that calendar. Um if you
9:05:00remember, let's look let's see this
9:05:02calendar. So, we had this available, the
9:05:05date, the listing, all of that stuff.
9:05:08Um, and [clears throat]
9:05:10let's look at the date in here. Uh, and
9:05:14we obviously don't want it like this. We
9:05:16want it to be more uh more of a time
9:05:19series. And we're going to do be doing
9:05:21that based off of uh the price for the
9:05:25calendar. So, let's go see if we can
9:05:27find that really quick.
9:05:29Football. Okay, here's the price.
9:05:33Where is that calendar one?
9:05:37Let me see. Okay, there's the calendar.
9:05:40Oh, here.
9:05:43All right. I totally forgot where that
9:05:45was supposed to be. O, that looks
9:05:46terrible.
9:05:48Okay. Um, let's see. Let's let's start
9:05:51working on this cuz this needs some
9:05:53work. Obviously, uh, this is the worst
9:05:56visualization I have ever seen. Um, so
9:05:59we need to work on this a little bit.
9:06:01What we need to do is we need to change
9:06:04Whoops. we need to change some some the
9:06:06way that these dates are are seen. So
9:06:09right here is act these are two separate
9:06:12things. So if I go right here and I do
9:06:13it by quarter, it's just going to change
9:06:15the quarters here, right? That's that
9:06:17isn't really helpful. We actually want
9:06:19to keep the year here. What we want to
9:06:21do it is by year. We want to separate it
9:06:24by year. Um but we want to separate it.
9:06:27Let's just do I don't know. Let's try
9:06:28week and see what it looks like. Okay,
9:06:30this is great. This is this is what
9:06:31we're looking at again. Um, if we went
9:06:34back and changed this like quarter, it
9:06:37uh changed it quarter and then change it
9:06:39to week, it would show the quarters, but
9:06:43it wouldn't show
9:06:46everything, [snorts] right? This isn't
9:06:47all the data that we need. And so, you
9:06:49know, you really need to make sure that
9:06:51you're doing this correct. I by default,
9:06:54it's almost always year. But if you're
9:06:56looking at it via quarter, so like let's
9:06:58say somebody comes in, you say, "Hey,
9:07:00what quarters? I want to break these out
9:07:02by quarters um and not year-over-year.
9:07:06That's how you would do this. But in the
9:07:07year, we want to break it out by uh the
9:07:10week. And you see this huge drop off um
9:07:16at the end. Well, that is actually
9:07:17because the data doesn't go past that.
9:07:20Um there's just like one day of data or
9:07:22one one um week of data in here with
9:07:26actual um with January of 2017 data. So
9:07:29it just drops off cuz this is an this is
9:07:31a sum. So it only adds up to like um
9:07:35591,000 compared to like the 2 million.
9:07:38So we want to get rid of that. Um and
9:07:41how do we do that? Uh let's see. I think
9:07:43it's filterup.
9:07:46Is it format? No, it's not format. What
9:07:48am I thinking? Bear with me. Uh let's
9:07:52filter. Well, I was looking for it. I
9:07:54just couldn't find it.
9:07:56Uh let's bring it back to the 31st.
9:07:59Let's see if that fixes what we need.
9:08:02Perfect. Uh that that's all you had to
9:08:04do. Um and the reason that this is
9:08:07helpful and often times you'd have
9:08:10several years worth of data in here. Um
9:08:13and then you could have you could do
9:08:14even do something like this. Um like
9:08:16this one where it has multiple lines.
9:08:19The reason that this is helpful is
9:08:21because if I'm telling my friend, let's
9:08:24I mean just I'm going to say it's a
9:08:25friend or business partner, whatever you
9:08:27whatever you want to use this use case
9:08:29for. I'm going to tell him, hey, the
9:08:31beginning of January all the way until
9:08:34like, you know, even February, it's like
9:08:38really low. It's half. So, there's not a
9:08:40lot of people traveling because everyone
9:08:42travels when? At the end of the year.
9:08:44So, in November, December for the
9:08:46holidays to visit family. Um, and then
9:08:48in the summer for vacations, I would
9:08:51tell him just based off this one thing,
9:08:53I would say, "Hey, over the summer and
9:08:56then at the end of the year and during
9:08:57the holidays, that's when I would be
9:09:00renting out your Airbnb." Okay, so just
9:09:03this one very simple visualization can
9:09:05help him understand the best times um to
9:09:08do that. That may have been intuitive.
9:09:09You may have already known that, but you
9:09:11can prove it with the data, which is
9:09:13always really helpful. Um, and let's
9:09:16see. Is there anything else that we need
9:09:17to do with this?
9:09:19Uh, I'm just going to label it. And I'm
9:09:22going to say,
9:09:23um,
9:09:26revenue
9:09:28for year.
9:09:31Let's do bold. Do apply. There we go.
9:09:35Did I label this last one? I didn't.
9:09:38Let's label that last one.
9:09:42And we'll do
9:09:44price per zip code.
9:09:48Price per zip code. We'll just keep it
9:09:50at that. Let's keep it simple.
9:09:53Um and let's do that. All right. I
9:09:56believe we have two more.
9:09:58So, we have done um we've done three of
9:10:02them. Um we got the zip codes, we've got
9:10:06the um you know, the time of the year.
9:10:09Now, something else that he was wanting
9:10:10to know is um you know, just how things
9:10:13affect it. And something that's going to
9:10:14affect the price of the actual Airbnb is
9:10:19going to be the amount of bedrooms. So,
9:10:21the the larger the house, the more
9:10:22bedrooms, the more it's going to cost
9:10:24typically. So, we can take a look at
9:10:28that. Let's pull in these bedrooms.
9:10:32Um and that will be our columns.
9:10:36Uh no, it won't. what we need to do. Um,
9:10:39and so I I knew this was going to
9:10:41happen. I just forgot it until right uh
9:10:42until right now. Well, we this right now
9:10:45is actually a um it's a a value, right?
9:10:49So it's a number. And that's totally um
9:10:52reasonable because if we go right here,
9:10:55we do count distinct. That's because
9:10:57there's only seven values, right? It
9:10:58goes there's zero bedrooms, 1 2 3 4 5 6
9:11:017 all the way up to seven bedrooms.
9:11:03Right? Now it has it as a numerical
9:11:04value. we want to um change that to
9:11:08create it as um these measure names, not
9:11:12a value. So, we're going to um we're
9:11:16going to remove this. We're going to go
9:11:18right down here. We're going to click
9:11:19this dropdown and we're going to say
9:11:21convert to dimension.
9:11:24And so now we're going to add it as a
9:11:26dimension. So there, that looks um much
9:11:29more normal. I really quick. I'm going
9:11:31to I'm going to keep these in here for a
9:11:33second, but we're going to get rid of
9:11:33these nulls and zeros because if a home
9:11:35has zero bedrooms, that's a problem. Um,
9:11:39and so we want to look at the price
9:11:42again. Let's go down here in the
9:11:45listings. It should be the price. Now,
9:11:47this is the price for the location per
9:11:49day. Um, if you want to look at monthly
9:11:52or or you know, stuff like that, they
9:11:54have that data. Um, but we're just going
9:11:56to do the price, the average price, not
9:11:58the sum.
9:12:00Um, although this is helpful. So, just
9:12:02really quick before we change it, this
9:12:04is going to show you which ones make the
9:12:06which ones are bringing in the most
9:12:08money. It also may show you which ones
9:12:09are the most common. Um, those are all
9:12:11different visualizations that we can do,
9:12:13but the one that brings in the most
9:12:15money uh that brought in 63 or that has
9:12:18$63 million worth of um worth of
9:12:23listings. So, they all add up. Those
9:12:26onebedrooms are doing phenomenal. half
9:12:29of that are two bedrooms at 30 million,
9:12:32three bedrooms at 18 million, and so on
9:12:33and so forth. So, there's a ton of
9:12:36one-bedroom ones. We may even keep we
9:12:39could even keep that in there. Um, you
9:12:41know, if we wanted to.
9:12:44Um, and then we do something similar
9:12:46later, but you can keep something like
9:12:47this in there. What we will do really
9:12:50quick though is we're going to do the
9:12:51same thing that we been doing is keeping
9:12:53average.
9:12:55Um, and we are going to get rid of this
9:12:58because if it doesn't have the bedrooms,
9:13:00you know, that's not helpful to us. And
9:13:02if it has zero bedrooms, that's that's
9:13:04genuinely a problem. I will not be
9:13:05renting an Airbnb with my family uh that
9:13:08has zero bedrooms in it. So, now we have
9:13:10this.
9:13:12And it would be really helpful to be
9:13:13able to see that in the visualization. I
9:13:15mean, it's just kind of
9:13:17hard to see it as is. I mean, it just
9:13:21does not hurt to add that right here.
9:13:24do a label. Um, why is it angled like
9:13:27that? Maybe I just need to
9:13:31move it out more.
9:13:34That looks much better. Um, that's the
9:13:37average price. That cannot be right.
9:13:40That's the sum. That's why. So, let's go
9:13:42over here. Let's make that average as
9:13:44well. Much better because uh if the
9:13:47price was $3 million
9:13:50for a three-bedroom, I would not be
9:13:52going there. So, this is really really
9:13:56useful information for our friend,
9:13:58right? If um he wants to, you know, get
9:14:01into those one that onebedroom area, you
9:14:03know, you're not going to be making a
9:14:04lot of money. It may be low cost
9:14:06upfront, but he's not going to be making
9:14:07a lot of money. It significantly goes up
9:14:11when you reach these five and sixbedroom
9:14:13homes, which makes sense. I mean, if it
9:14:15has five or six bedrooms in it, it's
9:14:16probably a really large, really nice
9:14:18home, and you can charge a lot more
9:14:20money. And our friend is uh extremely
9:14:22wealthy. he can buy whatever he wants.
9:14:23And so he may be looking at these um
9:14:25larger ones, seeing that there's a much
9:14:27higher return um on his investment the
9:14:30higher and the more bedrooms he goes. So
9:14:33we're going to keep it just as it is.
9:14:37Um and let me see is there's anything
9:14:39else that we want to do with this. No,
9:14:41we're going to keep it just like this.
9:14:42Uh and the last one is by far the
9:14:44easiest and we actually just discussed
9:14:45it a little bit. We want to know, you
9:14:48know, what's his competition look like?
9:14:50So, um, for those for the bedrooms
9:14:52specifically, so let's go back up to the
9:14:56bedrooms,
9:14:58we want that one to be right here in our
9:15:01rows. So, we show um these and then we
9:15:04just want a count of um how many
9:15:08listings there are. So, we can do that
9:15:11via the listings ID. So, here's our
9:15:13listings. Each ID represents one
9:15:16location or one home. So, we're going to
9:15:18do that right here. Uh, that looks
9:15:21absolutely terrible.
9:15:24That looks terrible. What am I doing
9:15:26wrong here? Oh, let me see.
9:15:31U, one thing we need to do is we want to
9:15:33get rid of these nulls and zeros.
9:15:35Do that really quick.
9:15:38Um, and then [clears throat] we don't
9:15:40want to do just the ID because I I'm
9:15:42realizing now uh what I'm doing. I need
9:15:46to convert this to a numeric so we can
9:15:49do a count on it. So let's um Oops. Let
9:15:53me see what what is happening. This is
9:15:55terrible. All right, let's put this
9:15:57back. Let's make Let me see if I can
9:15:59just um do an attribute.
9:16:04Let's do
9:16:07the count
9:16:10and let's do
9:16:13text.
9:16:15Um, no. It needs to be a distinct count
9:16:19because that's that's basically like um
9:16:23a count of the numbers themselves, not
9:16:27each individual ID. Okay, it took some
9:16:31figuring out. I'm going to keep that in
9:16:32there because you guys need to see uh a
9:16:35lot of you guys like seeing when I make
9:16:36mistakes, so you know, it makes it feel
9:16:38like when you make mistakes, it's okay.
9:16:39Um, and I'm all about that. So, I'm
9:16:41leaving that in there. You guys can see
9:16:42me fail a little bit. Um, I just forgot
9:16:44how to do that for a second. And this is
9:16:47exactly what we're looking for, right?
9:16:49We want, we now, it showed us in that
9:16:51visualization that we were looking at
9:16:52earlier before we um switched it to the
9:16:55average price. This is showing us that
9:16:58there are for onebedrooms, there's 1,800
9:17:01onebedroom, two that have 483, three
9:17:03that have 206, four that have 55, only
9:17:06five that have 20, and six that have
9:17:07five. So, the more you go up, the less
9:17:10and less it is, or the less and less
9:17:12competition there's going to be. Now, is
9:17:13there a lot of demand for fourbedroom,
9:17:155bedroom, sixbedroom? Uh, that's for our
9:17:17friend to figure out. Um, well, maybe
9:17:19we'll help them out with that later um
9:17:22in the with the data. You know, we could
9:17:24look at the reviews that we had. Um,
9:17:26there's so much data in here and we
9:17:28could absolutely figure that out. But
9:17:29for what it's worth, we're giving him
9:17:31this initial stuff and he'll have
9:17:32follow-up questions for us later. That's
9:17:34how it always works, I promise. Um, so
9:17:37now we're good with this one. Let's
9:17:39label this one. Did I label the last
9:17:41one? I will go back and look. Um,
9:17:45distinct I I'm going to butcher this
9:17:47one. I'm going to do distinct count of
9:17:51of bedroom listings. I don't that may
9:17:55not make sense at all, but we're keeping
9:17:57it. So, we're going to do bedroom.
9:17:59Apply. Okay. Let me see if I added the
9:18:02label on this one. I didn't. Let me do
9:18:05that real quick.
9:18:08We'll do average
9:18:11price per bedroom.
9:18:14Again, I'm Oops,
9:18:18you didn't see that. I'm just going with
9:18:20whatever is coming to my head. This
9:18:22probably wouldn't be what I would keep
9:18:23if I this were like an actual project,
9:18:25but it works for now. So, we have our
9:18:29five visualizations. 1 2 3 four and
9:18:31five. And let's create our dashboard.
9:18:34That's going to be this button right
9:18:35here. So, we're going to click that. We
9:18:38are going to uh go right here and we're
9:18:41going to say automatic because we want
9:18:43to use this entire area. And so, now
9:18:46we're just going to start um you know
9:18:49pulling them over. And I'm just going to
9:18:50start from the very first one and go to
9:18:53the very last one. Keep it really
9:18:55simple. So, this very first one, we'll
9:18:58pull it over it. You know, it's going to
9:19:00take up the entire space until you start
9:19:02adding all the other ones. We'll include
9:19:04this one right here. Um, and well, let's
9:19:08leave it as it is, you know. We'll
9:19:09adjust it once it gets to its final
9:19:11place. Now, we have number three. We'll
9:19:14add this one on this side. It looks
9:19:17terrible right now, but give it a
9:19:19second. Uh, then we have number four.
9:19:21We're going to add that across the top.
9:19:24Okay. It's already starting to look a
9:19:25little better.
9:19:27And, um, maybe I I You don't have to
9:19:31keep this in here. Um,
9:19:34but you definitely can. Uh, let's start
9:19:37to adjust things a little bit.
9:19:41Oops.
9:19:43Okay. Let's
9:19:45see see if I can zoom in one more. No,
9:19:49I'm going to do it just like that.
9:19:50Actually, let me see
9:19:56if I can make it even just a little bit
9:19:58closer. Perfect. Uh, that's the best
9:20:00you're going to get. Um, if you didn't
9:20:02see, I used this um magnifying and then
9:20:04I could click on the area that I wanted
9:20:06to see. So, we're going to keep that
9:20:08just like that.
9:20:10We're going to move this over because
9:20:11that is um definitely not as important.
9:20:15Um, and then we're going to move this
9:20:18way over as well to keep it just like
9:20:20that. Again, this is something where if
9:20:22you want to, you can click on this. Um,
9:20:25it didn't I don't know why uh I can't
9:20:27remember how to get those connected, but
9:20:28it's you definitely can. Um but okay, I
9:20:32was just clicking on the wrong one.
9:20:33That's why
9:20:35that is why. But you can click over here
9:20:38and you you know it'll filter um based
9:20:40on So if I go to this one. Oops. Dang.
9:20:45Oh jeez, what am I doing? Oh, this is a
9:20:47travesty. Okay, let's try to get this
9:20:50back.
9:20:52All right, I'm not touching it, guys.
9:20:53You get the gist. You can mess around
9:20:55with it yourself. I'm not messing this
9:20:56up. Okay. So, the next thing we need to
9:20:58add is the very last one. That's going
9:21:00to go right up here. And then we're just
9:21:02going to kind of move it off to the
9:21:05side.
9:21:08And
9:21:10let's see.
9:21:14Add
9:21:17Yeah, this caption. Um, if you've never
9:21:20seen something like this before,
9:21:23um, and I actually want to make this
9:21:24bigger as well.
9:21:27Jeez, give me a second. It's It's kind
9:21:30of lagging a little bit.
9:21:36And make this a little bit. Maybe I
9:21:40don't want it as wide, but I definitely
9:21:42want a little taller.
9:21:47Give it a second. Yeah. Let me scooch
9:21:50this back.
9:21:53Just like that.
9:21:55That's fine. Uh, we can keep it like
9:21:58that. In my original one, I didn't have
9:22:00this. Um, you can get rid of this if you
9:22:02want. You know, you can, um, you know,
9:22:06just exit out right here if you want to
9:22:07do that. But there you have it. Uh, this
9:22:10is the entire thing. So, we started from
9:22:13the very start. Um, we started with this
9:22:15one, then this one. Uh, did some um, and
9:22:19this is, you know, all the zip all of
9:22:22our zip code work. Then we took a look
9:22:24at the calendar where we looked at the
9:22:26price and did some time series
9:22:27visualization. And then we're looking at
9:22:30the bedrooms and and the count of
9:22:32bedrooms. And so this should be really
9:22:33helpful for a friend. It should be an
9:22:35initial dashboard to get him going. And
9:22:37once he sees this, he's going to have a
9:22:39million other questions and he's going
9:22:40to want another dashboard for different
9:22:42data that's in there. He's going to ask
9:22:44about, okay, well, what if I want to do
9:22:45it weekly or, you know, I want to rent
9:22:47it out for the month or, you know, how
9:22:49many um reviews are people, fivestar
9:22:52reviews are people giving on, you know,
9:22:54onebedroom, two-bedroom, threebedroom.
9:22:56These are all things that, you know, he
9:22:59may ask and then we'd have to build out.
9:23:00In the real world, this is what happens
9:23:02all the time. You know, they make a
9:23:04request and then they're like, "Oh, this
9:23:05is great, but I also want this." So, um
9:23:08you know, your friend is is going to be
9:23:11right in line with just about everyone
9:23:12else um that has ever gotten a dashboard
9:23:15uh for work or for personal use. With
9:23:19that being said, this is it. Um we have
9:23:21done the entire thing. Now, if you want
9:23:23to share this, it is super super easy to
9:23:26share. Um and I'm going to try to
9:23:28remember how to share it. Uh, so we're
9:23:29going to do save to Tableau public as
9:23:33and we're going to do this and we're
9:23:34going to make it um let's do air B&B. Is
9:23:39it like is it a capital B? Is it like
9:23:40that? No, that doesn't look right.
9:23:42Airbnb.
9:23:44Uh, we'll do full project and we'll
9:23:47save.
9:23:49And that is being created right now. Um,
9:23:52and I will save this. So, if you guys
9:23:54want to go look at this, you can. Um,
9:23:56and I'll provide a link in the
9:23:58description as well for that and see if
9:24:00yours looks um similar to mine or better
9:24:02than mine.
9:24:05Give it a second because it's thinking.
9:24:10All right. So, here it is. So, here's
9:24:13our final our final project. Um, and if
9:24:15you followed step by step, then you
9:24:17should get this exact or very very
9:24:19similar to this one. Again, I encourage
9:24:22you to if you want to have the upto-date
9:24:25data to go to that um link in the
9:24:27description that has um the the most
9:24:31recent data and they update that I
9:24:32believe monthly. So, you can go there,
9:24:34get the most recent data and then you
9:24:36can do stuff and you can create a
9:24:37beautiful project just like this um but
9:24:39with the you know the most recent data.
9:24:40Again, I use the Kaggle data just so you
9:24:42guys can remember. And I encourage you
9:24:44to look at the different data points
9:24:46that are in the Excel. There is so much
9:24:48in there and you can use honestly like
9:24:51there's probably 30 or 40 other fields
9:24:53that you could be using in there that we
9:24:55never even touched. Um, but for this
9:24:58project, we're keeping it pretty simple.
9:25:00And so go do that. Make completely
9:25:02unique dashboards and and visualizations
9:25:05and create projects and add it to your
9:25:07portfolios so that you can create uh a
9:25:09fantastic portfolio website and get a
9:25:12job. And that's what this is all about.
9:25:14Um, it's about upskilling and and
9:25:16getting these skills that you can, you
9:25:18know, get a job or or do better in your
9:25:20job. So, I hope this has been helpful. I
9:25:22really appreciate you guys joining me
9:25:24and and doing this entire project with
9:25:26me. I have no idea how long this is.
9:25:28This probably this could be like an hour
9:25:29for all I know. Um, so thank you so much
9:25:32for sticking with me this entire time.
9:25:33If you like this video, be sure to like
9:25:35and subscribe below and I will see you
9:25:37in the next video.
9:25:41>> [music]
9:25:51>> What's going on everybody? Welcome back
9:25:52to another video. Today we're going to
9:25:54be starting our PowerBI tutorial series.
9:26:02Now, I am super excited to start this
9:26:04series with you guys. We're going to be
9:26:05breaking this up in about six or seven
9:26:07videos. I don't really like those super
9:26:09long videos where it's like four hours
9:26:11long. I like breaking mine up into
9:26:13chunks. So, that's what we're going to
9:26:14do. This is the beginner series. And so,
9:26:16we're going to start with the very
9:26:17basics and we're just going to work our
9:26:18way up. And I'm going to walk you
9:26:19through every single step of the way.
9:26:20It'll be very easy to follow. Everything
9:26:23will be provided for you so that all you
9:26:25have to do is really follow along and by
9:26:27the end of it, you should know PowerBI a
9:26:28lot better and you should have a lot
9:26:30more confidence using it. All right. So,
9:26:31the first thing I'm going to do is
9:26:32download PowerBI desktop. I will leave
9:26:34this link in the description. So, you
9:26:36can just click on it, go to it and
9:26:37download it. We're going to click this
9:26:39download free button. And once we click
9:26:42it, you can go to the Microsoft Store.
9:26:45And I already have it downloaded. So,
9:26:46when you see it, uh it'll already say
9:26:48downloaded. But, um for you, you can go
9:26:51in here, you can click download, and it
9:26:53will download it for you. I'm on
9:26:54Microsoft, uh but it may look a little
9:26:56bit different for you if you're on a
9:26:58different system. But once that is done,
9:27:00we are going to open up PowerBI. So,
9:27:02let's go right down here to our search.
9:27:04Let's go to PowerBI.
9:27:09And it is going to open up for us. All
9:27:11right. So, right away, this is what it's
9:27:13going to look like when you open it. And
9:27:14we're going to go right over here to get
9:27:16data. And let's click on that. It's
9:27:20going to open up this window and it's
9:27:21going to give us a lot of different
9:27:23options for where we can get data from.
9:27:26Now some of these are free and some you
9:27:28need to upgrade from but you just taking
9:27:30a quick glance through here you have a
9:27:32ton of options. There's databases
9:27:34there's um you know blob stoages there's
9:27:38postgrade SQL or different SQL databases
9:27:40um there's Google Analytics there's a
9:27:42lot of places and you can go through the
9:27:44process to connect to that data and you
9:27:46can pull that data in from those data
9:27:48sources. Now for what we are doing we're
9:27:50just going to be using an Excel. I'm
9:27:52going to leave the Excel that I'm going
9:27:54to be using in the description. You can
9:27:56go and download it and walk through this
9:27:57with me. So, what we're going to do is
9:27:59click on Excel workbook and we're going
9:28:01to click connect. So, we're going to go
9:28:03right here in our PowerBI tutorials
9:28:05folder and we're going to click on
9:28:06Apocalypse Food Prep. So, let's click on
9:28:09that and it is going to connect and pull
9:28:11that data in. Now, right here we have
9:28:14our navigator and so if you had a lot of
9:28:16different sheets, you can click on that
9:28:18and choose which ones to pull in. I just
9:28:20clicked on it right over here and we're
9:28:22able to preview the data, but I can't
9:28:25load or transform it yet. I need to
9:28:27select which sheets I'm bringing in. So,
9:28:30we only have one. So, that's the only
9:28:31one we're going to bring in. So, you can
9:28:32go ahead and load the data or you can
9:28:34click on transform data. It's going to
9:28:36take us to PowerBI Power Query, which is
9:28:39going to allow us to transform our data.
9:28:41So, I'm going to have an entire video on
9:28:43how to transform the data, but I'm going
9:28:45to give you a really quick glance at it
9:28:47to kind of show you what it is. So right
9:28:49up here it says our power query editor
9:28:52and this is uh the window to basically
9:28:54transform your data and get it ready for
9:28:56your visualizations. Now you can do this
9:28:58in Excel if you want to and do that
9:29:00beforehand or you can do it here. And
9:29:02there are lots of things that we can do
9:29:03in here as you can see at the top. Again
9:29:06I'll have an entire video dedicated to
9:29:08just Power Query. But let's take a quick
9:29:10look at the data and see if there's
9:29:11anything we want to transform quickly
9:29:13before we actually go and start building
9:29:15our visualizations.
9:29:18So over here we have the store where we
9:29:20purchased it. We have the product that
9:29:22we purchased, the price that we paid,
9:29:24and the date that we bought it. Now, the
9:29:26first thing that jumps out to me is that
9:29:27this just says date on it. Um, we might
9:29:30want to say date
9:29:34purchased and we're going to hit enter.
9:29:36And if you noticed right over here on
9:29:38these applied steps, it says renamed
9:29:40columns. everything that you do, every
9:29:43single step that you apply to transform
9:29:45this data is going to be right over
9:29:47here. And if I want to, if I go back and
9:29:49I say, you know, I really didn't want to
9:29:50rename that column, I can just click X,
9:29:53and it is going to get rid of that and
9:29:55take it back to its original state. So
9:29:58again, I'm just going to say purchased,
9:30:01and we're going to enter that. Now, this
9:30:04is our apocalypse food prep. So this is
9:30:06food that we are buying for the
9:30:08apocalypse um for this example. And if
9:30:10we look at our products, we have bottled
9:30:12water, canned vegetables, dried beans,
9:30:15milk, and rice. And all of that stuff
9:30:16makes sense except for the milk. U milk
9:30:19will not stay or last long in the
9:30:22apocalypse. So I think what we're going
9:30:23to do is we're going to filter that out
9:30:24really quickly. And we're going to click
9:30:26okay. And right over here again, it says
9:30:29filtered rows. And so now, if we scroll
9:30:32down, there's no milk. So what we are
9:30:34going to do is we are going to go over
9:30:36here to close and apply.
9:30:39and it is going to actually load the
9:30:41data into PowerBI desktop.
9:30:45So on this lefth hand side, it
9:30:46immediately takes us to the report tab.
9:30:49And what we want to do is go right here
9:30:51to the data tab
9:30:53and take a look at our data. So again,
9:30:55there's our date purchased. And as you
9:30:58can see, the milk is not in there.
9:31:01Another tab that we're going to take a
9:31:03look at um and again in this report tab,
9:31:05this is where we actually build our
9:31:06visualizations. The data is where we can
9:31:09see the data and and change it up a
9:31:11little bit and change some small things
9:31:12about it like sorting the columns or
9:31:14even creating a new column. And over
9:31:16here we have this other tab and is
9:31:18called model. And this is especially
9:31:20useful when you have multiple tables or
9:31:22multiple Excels and you need to join
9:31:24them to kind of connect them together.
9:31:26We don't have that, but in a future
9:31:28video I'm going to walk through how to
9:31:29use this entire tab. So now let's go
9:31:31back to the data tab. And I want to just
9:31:33look at the data really quickly before
9:31:35we go over to the report tab and we
9:31:37start building our first visualization.
9:31:39As you can see, I've been buying these
9:31:41different products in different months.
9:31:42So, this rice I've been purchasing in
9:31:44January, February, March, and April. And
9:31:47I've been buying it from three different
9:31:48locations because I wanted to see if I
9:31:50was spending less money at one location
9:31:52on all of the products. So then I would
9:31:54just shop there in the future and save a
9:31:56lot of money. Or if there were specific
9:31:57products that were really cheap at one
9:31:59location, but others they were cheaper
9:32:01at a different location and so I should
9:32:03just buy like the dried beans at Costco,
9:32:06but everything else I should be buying
9:32:07at Walmart. And so that's what we're
9:32:08going to look at in just a little bit.
9:32:10So let's go over to the report tab.
9:32:12Right up here at the top there's this
9:32:14data section. So you can kind of choose
9:32:15if you want to add any more data now
9:32:17that we are here. We can also write
9:32:20queries or transform the data like we
9:32:22were looking at in the power query
9:32:23editor window. Over here in the insert,
9:32:25we can add a new visualization or a text
9:32:27box. And then in the calculation
9:32:29section, we can create a new measure or
9:32:31a quick measure. And then over here we
9:32:33have share where you can actually
9:32:35publish your report or your dashboard
9:32:37online. Now over on the visualization
9:32:38section on this far right, this is a
9:32:40very important area. This is where a lot
9:32:43of the actual creating of the dashboards
9:32:45happen. So, let's take a look really
9:32:47quick and we'll get into a lot of these
9:32:49things as we're actually building our
9:32:51dashboard. So, we're not just sitting
9:32:52here looking and talking. We're going to
9:32:53be actually building and doing. All
9:32:55right. So, we're going to click right
9:32:56here on this drop down on sheet one and
9:32:59it's going to show us all of our
9:33:00columns. Now, two of the things that we
9:33:02wanted to look at were where are we
9:33:04spending the least amount of money
9:33:06buying the exact same product. That'll
9:33:08help us determine where we want to shop.
9:33:09And the second thing was, should I be
9:33:11buying all my products at the same place
9:33:13or are there certain products that
9:33:15they're going to be cheaper at a
9:33:16specific store and I should buy it
9:33:17there? So, let's start out with the
9:33:19first one, which we're just going to
9:33:21see, uh, with the store and the price,
9:33:25uh, where we're spending the least
9:33:27amount of money. And just at a quick
9:33:29glance, we can see we're spending the
9:33:30least amount of money at Costco at $210
9:33:32versus Target $ 219 and Walmart at 225.
9:33:36And that really answers our question,
9:33:38but we want to visualize it better, be
9:33:40able to see it in a an easier way. So,
9:33:42we're going to go right over here, and
9:33:44we can click on a lot of these, but the
9:33:46one that probably makes the most sense
9:33:47is the stocked column chart,
9:33:50and it's going to show Walmart, Target,
9:33:52and Costco. Now, they're all the same
9:33:54color. Let's add a legend. So, we're
9:33:56just going to drag store over here down
9:33:58to this legend. And let's make this
9:34:01larger while we're working on it. So now
9:34:04we can see we're spending the most
9:34:05amount of money at Walmart. Uh right in
9:34:07between at Target and then at Costco is
9:34:09the lowest. And so right there we know
9:34:11that Costco is the place to go for our
9:34:13Apocalypse food prep. But is it going to
9:34:16be that way for every product? Uh I
9:34:19don't know. Let's take a look. Let's put
9:34:22this up in this corner and let's start a
9:34:24new one. We're going to need to select
9:34:26the product for sure and the price and
9:34:30probably additionally the store as well.
9:34:33And let's click on
9:34:36let's not do this one. We need a
9:34:37clustered column chart. That's what we
9:34:39need. Let's bring this over here. Let's
9:34:42expand this quite a bit. And so really
9:34:45at a glance, this is giving us
9:34:47everything that we need. We can see each
9:34:49product right here and we can see how
9:34:51much we're paying per store. And so for
9:34:54rice, we're paying it looks like a lot
9:34:57more for uh our rice at Walmart, while
9:35:00at Target is actually where we are
9:35:02paying the least. Now, if we look at all
9:35:04of these, it looks like for Costco, the
9:35:07only one that we're really paying a lot
9:35:08more on is on our rice. But for our
9:35:11dried beans, our bottled water, we're
9:35:14paying quite a bit less. And really,
9:35:16it's pretty negligible for these canned
9:35:18vegetables. We're paying maybe what, 60
9:35:20cents, 50 60 cents more per can. That's
9:35:23pretty negligible. But for the big
9:35:25ticket items, um, we're really spending
9:35:27a lot less at Costco. If we wanted to sp
9:35:30to save just a little bit more money, we
9:35:32could go to Target for our rice. Now, if
9:35:34I want to make this more like a
9:35:36dashboard and we're only keeping these
9:35:38two things, I'm going to kind of size
9:35:40them kind of like this. Whoops. Going to
9:35:43show you that in a little bit. I'm going
9:35:44to size them a little bit like this. So,
9:35:48now that we have that looking good, we
9:35:50want to change the title of both of
9:35:51these. So, what we're going to do is go
9:35:53over here in our visualizations and
9:35:55format your visual. Uh, and we are going
9:35:58to go to this general, go to title, and
9:36:01now we can name it anything we really
9:36:03want. For this, we're going to say best
9:36:06store
9:36:08for product.
9:36:11And while we're in here, one other thing
9:36:13that I wanted to do is I want to go to
9:36:14this visual. Go right down here to these
9:36:17data labels. Now, we haven't added any
9:36:19data labels. So, I'm going to click on,
9:36:21and you'll see exactly what it does. Uh,
9:36:24it just puts the labels and the numbers
9:36:25above it, so you don't have to actually
9:36:27like hover over it and see what it is.
9:36:29Now, it is actually rounding these
9:36:30numbers. So, what we're going to do is
9:36:32go down here. We're going to go down to
9:36:35values and we'll go down to display
9:36:38units and it's on auto, so it's auto
9:36:40rounding those numbers. And we're just
9:36:42going to say none so we can see the
9:36:44actual value of these numbers.
9:36:47And we can do the exact same thing over
9:36:49here. It probably is a good thing to do.
9:36:53Um, and it just is going to visualize it
9:36:55a little bit differently in here, but
9:36:56you can always change that if you want
9:36:58to. Go over here to title and we're
9:37:02going to say total by store.
9:37:06And now we're going to take a look. And
9:37:10so in a matter of minutes, we were able
9:37:11to take our data from an Excel, put it
9:37:14into PowerBI, transform it a little bit.
9:37:17Then we were able to create these
9:37:18visualizations that gave us concrete
9:37:20answers to some very important topics.
9:37:23We now know that Costco is the place to
9:37:25go for basically every single product
9:37:27except if we're buying rice. And if we
9:37:30want to save just a few dollars, we're
9:37:32going to head over to Target. And that's
9:37:33genuinely going to change my shopping
9:37:35habits for the next several years until
9:37:36the apocalypse happens. So, in future
9:37:38videos, we're going to dive into a lot
9:37:40of the things that we looked at today,
9:37:41but just in more detail. And then at the
9:37:43very end of this series, we're going to
9:37:44have an entire project where we really
9:37:46use every single part of PowerBI and
9:37:48create a beautiful dashboard. And so,
9:37:50that's all we have for our very first
9:37:52video in our PowerBI series. I hope it
9:37:54was helpful. If you like this video, be
9:37:55sure to like and subscribe below and
9:37:57I'll see you in the next video.
9:38:10What's going on everybody? Today we're
9:38:12continuing our PowerBI tutorial series
9:38:13and in this video we're going to be
9:38:15looking at Power Query.
9:38:22Now [music] Power Query is really great
9:38:23because it allows you to actually
9:38:24transform the data before you actually
9:38:26get it into PowerBI. So, if you want to
9:38:28make any changes like adding or deleting
9:38:30a column or changing the data type or a
9:38:32ton of other things, you can do all of
9:38:34that in Power Query. Now, without
9:38:36further ado, let's jump on my screen and
9:38:38get started with the tutorial. All
9:38:39right, so before we jump over to PowerBI
9:38:41and start using Power Query, I wanted to
9:38:43take a look at the data. And this is the
9:38:45Excel from our last video called
9:38:47Apocalypse Food Prep. And in that video,
9:38:49we went through and we bought some rice,
9:38:51some beans, water, vegetables, and milk
9:38:53all for the apocalypse getting prepared
9:38:56for that. Now, we decided to buy some
9:38:58additional things like rope, some
9:39:00flashlights, duct tape, and a water
9:39:02filter, several water filters. And after
9:39:06we purchased those, uh, our boss or
9:39:08whoever we're working with there,
9:39:10somebody decided to go and make a pivot
9:39:12table. Now, in this pivot table, they
9:39:14kind of broke it out by Costco, Target,
9:39:15and Walmart, and had all the items had
9:39:18some subtotals as well as some grand
9:39:20totals right here. And then [snorts]
9:39:22they decided to kind of copy and paste
9:39:25that into this. And you'll see this a
9:39:27lot when you're working with uh people
9:39:29who use Excel. They like to kind of make
9:39:31things like this, maybe make it into
9:39:33like a table or or format it a little
9:39:35bit differently, but you'll see stuff
9:39:36like this a lot. So, this is what we're
9:39:38going to actually pull into Power Query
9:39:41and work with. Now, we're going to
9:39:42imagine that this is all we have. This
9:39:45is the only thing we were working with.
9:39:46And I'll kind of reference this pivot
9:39:48table a little bit, but we're going to
9:39:50pretend this is all we have. and we want
9:39:52to transform it to make it a lot more
9:39:53usable to where we can make
9:39:55visualizations with it. So, let's hop
9:39:56over to PowerBI and pull this Excel in.
9:39:59So, what we're going to do is click
9:40:00import data from Excel. We're going to
9:40:02click Apocalypse Food Prep and click
9:40:03open. And then it's going to bring up
9:40:05this window right here. Now, this is
9:40:07where we can choose what data to bring
9:40:09in. So, we can take a preview and just
9:40:11click on it real quick. And this is the
9:40:13pivot table that we were looking at. So,
9:40:15it does have that pivot table. So, we
9:40:17are able to pull in just a pivot table.
9:40:19And then we have the purchase overview
9:40:21where it's kind of that formatted um
9:40:24thing that we were just looking at with
9:40:25all the colors. We're going to pull both
9:40:26of those in. So we're going to pull in
9:40:28the pivot table and the purchase
9:40:30overview. Now we could just load it or
9:40:32we could transform it and we're going to
9:40:34click transform and that's going to
9:40:35bring us to power query. So let's click
9:40:37on transform data. So now really quick
9:40:39before we actually jump into working
9:40:41through this and transforming it, I want
9:40:43to show you what the Power Query editor
9:40:45looks like. So, if we go right over
9:40:47here, we have our queries, and these are
9:40:48the tables that we actually pulled in,
9:40:50and we can click on those and kind of go
9:40:52back and forth between them. Now, up
9:40:54top, we have our ribbon, and the ribbon
9:40:56offers a lot of functionality. We have
9:40:58things like remove columns, keep rows,
9:41:00remove rows, split columns. These are
9:41:03all things that we're likely to use when
9:41:05using this Power Query editor. There's
9:41:07also another tab called transform where
9:41:09there's a lot of functionality here as
9:41:11well. things like unpivoting a column or
9:41:14transposing columns and rows and using a
9:41:17first row as a header. Some of the
9:41:18things that we'll be looking at today.
9:41:20There's also another tab called add a
9:41:22column. And this one's pretty
9:41:24self-explanatory where you can add
9:41:25additional columns like deleting a
9:41:27column, creating an index column, or a
9:41:30conditional column. Those are the three
9:41:31main ones. There's also view, tools, and
9:41:34help, but we're not going to really be
9:41:35looking at those today. And then on the
9:41:37far right side, we have our query
9:41:39settings. You can do things like change
9:41:41the name. So we can call it pivot table
9:41:442022 and it'll update right over here on
9:41:47our query side. And we have our applied
9:41:50steps. Now our applied steps are
9:41:52extremely important and very very
9:41:54useful. Anytime we make any change to
9:41:56transform this data, it's going to be
9:41:58documented right here. And then we can
9:42:00go back and look at it or we could even
9:42:02delete that change in the future if we
9:42:04want to and go back to a previous
9:42:06version of what we just did. So when we
9:42:08loaded the data into PowerBI, it did a
9:42:10few things for us. It chose the source,
9:42:12the navigation, and it promoted the
9:42:13headers. And then it also changed the
9:42:16data type. So if we want to check, we
9:42:18can actually see those things or change
9:42:19those things like this source right
9:42:21here. We can click on this little icon
9:42:23and it's going to bring up the actual
9:42:25path where we got this file. So if we
9:42:27wanted to change that or or it changes
9:42:29in the future, we can come here and we
9:42:31can change this file path. But we're not
9:42:33going to do that right now. So, let's
9:42:34click on cancel and let's go back down
9:42:36to change type. So, it promoted these
9:42:39headers and obviously these headers are
9:42:41not correct. We're looking at this pivot
9:42:42table and not the purchase overview, but
9:42:44it changed these column headers. And so,
9:42:47in the future, if we wanted to, we could
9:42:48easily change those, but it did that for
9:42:50us. And it changed the type as well. So,
9:42:53if you look right here, it says ABC123.
9:42:56All the way over here to where it just
9:42:58says ABC. ABC means it's only going to
9:43:01be text where ABC123 means it could be
9:43:03basically anything uh text or it could
9:43:06be numeric. So now let's go over to
9:43:08purchase overview and this is the one
9:43:10that we're actually going to be working
9:43:11on the most but we might be looking at
9:43:13pivot table just a little bit to kind of
9:43:15reference it and see some of the
9:43:16differences. So before we do anything
9:43:18let's just take a look at how PowerBI
9:43:20decided to take this data in. So, it
9:43:22chose this apocalypse food prep overview
9:43:24as kind of the first column. And that
9:43:26was kind of our header or the title of
9:43:28what we were looking at before. And then
9:43:30all these other columns are basically
9:43:31column 1 2 3 4 5s. So, that's something
9:43:34that we're going to want to change in
9:43:35just a little bit. There's also all
9:43:37these blank uh columns right here at the
9:43:39top and kind of these null values as we
9:43:42go along. And we'll take a look at those
9:43:44and we kind of are going to want to get
9:43:46rid of some of this and just clean this
9:43:47up to make it more usable for our
9:43:50PowerBI visualizations. This may be
9:43:52perfectly fine and acceptable in an
9:43:54Excel, but when you're pulling it into
9:43:55PowerBI, the real reason you're pulling
9:43:57it in is to create visualizations, not
9:43:59just it to look good in an Excel. So,
9:44:02we're going to need to clean this up
9:44:03quite a bit. So, let's go right up top.
9:44:06The first thing that I want to do is I
9:44:08want to get rid of these top rows. So,
9:44:09we're going to go to this top ribbon and
9:44:11we're going to click remove rows. And
9:44:13we're going to select remove top rows.
9:44:15and we're going to select two cuz we
9:44:17have one two rows of all nulls and those
9:44:20are completely useless. We just want to
9:44:22get rid of them right away. So let's
9:44:24click okay and it removed those. The
9:44:27next thing that we want to do is these
9:44:29this location product and the all these
9:44:32dates these are actually the column
9:44:34headers that we wanted. So what we need
9:44:37to do now is we want to go over to
9:44:39transform and we want to say use first
9:44:42row as headers
9:44:44and just like that we have location
9:44:47products and these dates as our headers
9:44:49exactly how we wanted them. Now let's
9:44:51say for whatever reason you know we made
9:44:53a mistake and we needed to go back we
9:44:55would just select remove top rows and
9:44:58that would be perfectly fine. Now you
9:45:00can see over here it promoted the
9:45:01headers but it's also changed the data
9:45:03type. So before if we went to before we
9:45:07removed the headers, these were all
9:45:08ABC123
9:45:10ABC123 cuz it had a lot of different
9:45:12data types in there. So it just kind of
9:45:14made a generic data type. But when we
9:45:16promoted these headers, the first thing
9:45:18that it decided to do was also change
9:45:20this data type for us, giving us its
9:45:23best guess as to what this data type is.
9:45:26And it decided to do this decimal. So
9:45:28this one two is a decimal, but we're
9:45:30actually going to change that. And all
9:45:32you have to do is click on this 1.2 two
9:45:34or or the data type that it has right
9:45:36here for you. And we're going to click
9:45:38on fixed decimal number. And let's do
9:45:41replace current. And now it's just a
9:45:44little bit better. So now it's 2.70 2.5.
9:45:47And that's normally how we would read uh
9:45:49values like this because this is money.
9:45:52So we would normally read it to the
9:45:53second decimal just like that. And if we
9:45:55have it on the second decimal for some,
9:45:57we should probably have it on the second
9:45:58decimal for all of them. So, really
9:46:00quickly, I'm going to go through and I'm
9:46:02just going to change that. And it should
9:46:04be pretty quick. So, hang with me for
9:46:06just a second.
9:46:08All right, that is perfect. Now, for the
9:46:11purposes of what we're about to do, we
9:46:13don't actually need these subtotals or
9:46:15this Costco total, Target total, and
9:46:18Walmart total as well as the grand
9:46:19total. Really, we want to get rid of
9:46:21those. And so, what we're going to do is
9:46:23we're going to go right over here. We're
9:46:24going to click on this dropdown and
9:46:25we're going to try to filter this data
9:46:27before we actually load it into PowerBI.
9:46:30So, we're going to filter and we're
9:46:32going to say remove empty. And let's
9:46:35remove those. And it's going to take out
9:46:37all of those nulls. If we wanted to try
9:46:39to filter this out by saying something
9:46:40like Costco total or Target total, we
9:46:44could do that by going right here,
9:46:45clicking this drop down on products,
9:46:47going to text filters, and saying does
9:46:49not contain. And let's do insert. And
9:46:54we're going to say does not contain. And
9:46:55we want to say total. And let's click
9:46:59okay. And again, it filtered out all of
9:47:02those things. So there's a few different
9:47:03options that you can do if you want to
9:47:04filter out rows that contain either null
9:47:07values or specific values. Now, the next
9:47:09thing that we're going to do is actually
9:47:11get rid of a column, this grand total
9:47:13column. And so what we're going to do is
9:47:14we're going to click on the very top
9:47:16part where it says grand total. We're
9:47:18going to go back over here to home and
9:47:20we're going to click on remove columns
9:47:22and it says insert. That's because we're
9:47:24on this filtered rows one right here.
9:47:26Um, but what we're going to do is just
9:47:27insert that and it'll insert it right
9:47:29there. That's totally fine. We can just
9:47:31move it to the bottom. Now, we got rid
9:47:32of this column entirely. Now, this looks
9:47:36really good visually. I like how this
9:47:38looks. I like how everything is set up.
9:47:40The biggest thing about this is that
9:47:43when you're actually wanting to use this
9:47:44for visualizations, these columns as
9:47:46dates doesn't really work too well. And
9:47:50so what we're going to want to do is
9:47:52we're going to want to transpose this or
9:47:54pivot this to where these dates are
9:47:56actually rows. So what we're going to do
9:47:58is select the first date, which is
9:48:00January 1st, all the way through April
9:48:021st. And we're going to hit shift and
9:48:04click on that April 1st right there to
9:48:05select all of them at the same time. And
9:48:08then we're going to go over here to the
9:48:09transform tab and we're going to click
9:48:12unpivot columns and let's see what this
9:48:14does. And so now what we've done is
9:48:16we've basically recreated our original
9:48:19Excel that we had. So let's go back and
9:48:20take a look really quickly at that. So
9:48:22this looks almost identical to what we
9:48:24have in PowerBI right now. And this is
9:48:26extremely usable and very good for
9:48:28visualizations and is much much better
9:48:31than this. But again, we were pretending
9:48:33that this is what we were given at the
9:48:35beginning. So you have to imagine, you
9:48:37know, somebody just handing you this and
9:48:38you need to make it much more usable for
9:48:40visualizations in the future, which
9:48:42happens a lot. And we actually wanted to
9:48:45create this. We just weren't given this.
9:48:47Now, a few last things that we might
9:48:48want to do is we want to clean this up
9:48:50just a little bit. We're going to select
9:48:51the data type and change this to date.
9:48:54And then we're going to select the
9:48:55value. And I double clicked on the
9:48:58value. And I actually want to call this
9:49:00cost uh or product cost. product_cost
9:49:07and then for the location I actually
9:49:08want this to be called store. So now
9:49:12this looks really good but I want to
9:49:14show you one thing really quickly on
9:49:15this pivot table 2022. So let's go back
9:49:18here. This looks very similar to how we
9:49:21had it when it first started. One thing
9:49:23I wanted to show you uh really quickly
9:49:26and I want to click on this first one.
9:49:28We're going to make this our column
9:49:30header and then we're going to try to
9:49:31pivot or unpivot this January, February,
9:49:34March, April. So really quickly, let's
9:49:36do that. So we're going to transform use
9:49:39first row as headers.
9:49:42So now we have this January, February,
9:49:43March, April. Now if you notice, these
9:49:46are not dates. These are actually text.
9:49:49It says January, February, March, and
9:49:51April. So if we go to do this and we
9:49:55click unpivot
9:49:57and here's the columns that are created
9:49:59when we unpivot it. It is January,
9:50:02February, March, and April. These are
9:50:04not dates. So we cannot go and change
9:50:06this to a date because that would error
9:50:09out because it's actually text. So it's
9:50:11something that you want to look out for.
9:50:12It's something that you need to be aware
9:50:13of. And you can change that in the pivot
9:50:16table. So you want to be aware of how it
9:50:18actually sits and looks in the Excel or
9:50:20whatever data source you're pulling from
9:50:22before you actually pull it into Power
9:50:23Query to transform. And now the very
9:50:26last thing that we need to do to
9:50:27finalize all of this is go over here to
9:50:29close and apply. And once we click that,
9:50:32everything that we've worked on is going
9:50:33to be applied to the actual data and
9:50:35it's going to load into PowerBI to
9:50:37create our visualizations. So let's go
9:50:38ahead and click on that. And so now the
9:50:40data has been pulled into PowerBI. Let's
9:50:42go right down here to data and we can
9:50:44see the data right here. If we need to
9:50:46transform this data again, we can bring
9:50:48it back into the Power Query Editor
9:50:50window by just clicking the transform
9:50:52data button, and it's going to bring us
9:50:54right back. So, I hope that this was
9:50:55helpful. Thank you so much for watching.
9:50:57If you like this video, be sure to like
9:50:59and subscribe below and check out all my
9:51:01other videos and everything data analyst
9:51:03related. I'll see you in the next video.
9:51:06[music]
9:51:17What's going on everybody? Welcome back
9:51:18to the PowerBI tutorial series. Today
9:51:20we're going to be taking a look at
9:51:21building relationships.
9:51:27[music]
9:51:29Now, when you import multiple tables
9:51:30from either the same data source or
9:51:32multiple data sources, you want to tie
9:51:34them together so that when you're
9:51:35creating your visualizations, everything
9:51:37is connected. So, in this tutorial,
9:51:39we'll be walking through how to create
9:51:40those relationships to make sure that
9:51:42all of your tables are connected
9:51:43properly. And without further ado, let's
9:51:45jump on my screen and get started with
9:51:46the tutorial. All right. So, before we
9:51:48jump over to PowerBI and start creating
9:51:49our relationships and our model, I want
9:51:51to take a look at the data in Excel. We
9:51:53realized we were buying so many products
9:51:55for the apocalypse that we decided to
9:51:57start our own store. And we have several
9:51:59customers and some client information
9:52:01down here. And so, I wanted to take a
9:52:03look at some of the columns and these
9:52:04tables that we're going to be looking
9:52:05at. First thing we have is the
9:52:08apocalypse store. These are the things
9:52:10that we are selling. I know it's a very
9:52:12limited inventory, but these are the
9:52:14really high sellers. These are the ones
9:52:16that I wanted to sell. So, we have this
9:52:18product ID, our product name, price, and
9:52:21production cost. Then we have this
9:52:23apocalypse sales. This is how many sales
9:52:26we've actually made to our customers.
9:52:28So, we have this customer ID, our
9:52:31customer name, product ID, order ID,
9:52:34units sold, and the date it was
9:52:35purchased. And then we have our customer
9:52:37information right here. Here are all of
9:52:40our clients. So, we have this customer
9:52:41ID, customer, address, city, state, and
9:52:45zip code. So, now that we've taken a
9:52:46look at our data, let's go and load it
9:52:48into PowerBI. So, we're going to say
9:52:50import data from Excel. We're going to
9:52:52choose this model right here. And we're
9:52:54going to click open. And we are going to
9:52:55want all three of these. So, I'm going
9:52:57to click on all of them, and we're just
9:52:58going to load it. We're not going to
9:52:59transform the data at all.
9:53:04So, now the data has been loaded. Let's
9:53:06go right over here on the left hand side
9:53:08to our model tab. And let's scoot this
9:53:10over just a little bit and move back.
9:53:14And we're going to move these tables up
9:53:16to where it's a little bit easier to
9:53:18see.
9:53:19So, right off the bat, you can already
9:53:22see that there are these lines between
9:53:23these tables. So, there are already
9:53:25relationships that PowerBI has
9:53:27automatically detected and created. From
9:53:30my experience, PowerBI actually does a
9:53:31really good job at creating these
9:53:33relationships automatically, but we're
9:53:35going to go in and take a look at these
9:53:37and kind of see what everything means,
9:53:39and then we're going to go back and
9:53:40create these relationships from scratch
9:53:42just to make sure that we know how to do
9:53:43every single part. So, to get us
9:53:44started, let's double click on this line
9:53:46connecting the customer information
9:53:48table to the apocalypse sales table.
9:53:51and it's going to bring up this edit
9:53:53relationship page right here. So, this
9:53:55line right here connecting these two
9:53:56tables actually gives us quite a bit of
9:53:58information without actually having to
9:54:00click into this edit relationship page.
9:54:02What this is showing is that we have a
9:54:04one to many relationship and there's
9:54:07only one or a single cross filter
9:54:09direction. And you can find both of
9:54:11those things right down here. And I'm
9:54:13going to walk through what those mean in
9:54:14just a little bit. On this page, you can
9:54:16also see the columns that PowerBI
9:54:18decided to choose in order to tie these
9:54:20two tables together. Now, for our
9:54:22example, they decided to use the
9:54:24customer and customer right here from
9:54:26the customer information table as well
9:54:28as the apocalypse sales. But I don't
9:54:30really want to use those specifically
9:54:32because on this apocalypse sales table,
9:54:35I might remove this customer information
9:54:37and just keep the customer ID. it may
9:54:39have chosen these customer columns
9:54:41because they have the exact same name
9:54:42and really the same information, but I
9:54:45want to use this customer ID anyways.
9:54:47So, what I'm going to do is I'm going to
9:54:48click on that column and click on this
9:54:50column. And then I'm going to click
9:54:51okay. And if we go back into it by
9:54:54double clicking again, we're going to
9:54:56see that it now save that. And if we did
9:54:58what we just did before, which is kind
9:55:00of hover over it, it's going to show us
9:55:01what those two tables are joined on. So,
9:55:03opening this back up, let's go down here
9:55:05to this cardality and cross filter
9:55:07direction. Cardonality has several
9:55:09different options that you can choose
9:55:10from. You have one to many, one to one,
9:55:13one to many, and many to many. Now, for
9:55:15this example, we're looking at
9:55:17apocalypse sales, and we're going
9:55:18apocalypse sales down to customer
9:55:20information. Now, there are a lot of
9:55:23rows in the apocalypse sales, but
9:55:24there's very few in this customer
9:55:26information, and there's only one
9:55:28customer per row, whereas in the
9:55:30apocalypse sales up here, the customer
9:55:33can have several rows for several
9:55:34different orders. So that's why the
9:55:36cardality is many to one. Now if we flip
9:55:40this and we say we want the customer
9:55:41information here and we want the
9:55:43apocalypse sales down here and we tie
9:55:46that together. Now it's going to flip
9:55:47and it's going to say one to many. Now
9:55:49let's look at the cross filter
9:55:51direction. And there's only two options
9:55:52here. It's either single or both. And if
9:55:54we choose both and we click okay, this
9:55:57now goes from a single arrow pointing in
9:55:59one direction to two arrows pointing in
9:56:01both directions. But what does this
9:56:03really mean? So, in order to demonstrate
9:56:05this, I'm going to put this back to a
9:56:07single direction. And what we're going
9:56:08to try to do is connect the data over
9:56:10here or the columns over here to the
9:56:12columns in this apocalypse store. So,
9:56:14let's go over here to build a
9:56:16visualization. And what we're going to
9:56:18do is we're going to take this customer
9:56:20information and let's just say we want
9:56:21to look at state. So, I'm going to click
9:56:24on state right here. And I'm just going
9:56:25to make this into a table. And the
9:56:28customer information table is only tied
9:56:30right now to this sales table. So, we're
9:56:33actually going to go over to the
9:56:34Apocalypse store, and we want to see how
9:56:37many product IDs are being bought in
9:56:39these different states. So, really
9:56:41quickly, we're going to come up here and
9:56:42create a new measure. And all we're
9:56:45going to say is this measure is the
9:56:46count of Apocalypse store product ID.
9:56:51And we're going to create that. And now
9:56:53we're going to select it. So, it's added
9:56:55to that table. So now what this is
9:56:56showing is that there are 10 product IDs
9:56:58which there are 10 products for each of
9:57:01these states. But that's not actually
9:57:03technically correct because not every
9:57:06state purchased these 10 different
9:57:08items. If we go back to our model and we
9:57:11change both of these to
9:57:14a both direction,
9:57:17then we're going to go back and see what
9:57:18changed in our numbers. So now let's go
9:57:21back to our visualization. And now we
9:57:24can see that Minnesota actually only
9:57:25ordered seven different product IDs.
9:57:28Missouri 8, New York 9, and Texas 10.
9:57:31This is actually much more accurate than
9:57:33before. When you use the both option, it
9:57:36takes these tables and treats them as if
9:57:38they are a single table. But the single
9:57:40option is not going to do that. And so
9:57:41for our example, if we're trying to
9:57:43connect this table to this table and one
9:57:45of the last things that I want to show
9:57:46you is this option right down here,
9:57:48which says make this relationship
9:57:50active. Now if we don't click this and
9:57:52there are other options in here that
9:57:54connect these things like the customer
9:57:55to the customer then that may be the
9:57:58active relationship. But if I select
9:58:00this is the active relationship that
9:58:01means this is going to become the
9:58:03default relationship between these two
9:58:04tables. So now let's come out of here.
9:58:06We're going to click cancel.
9:58:08We're going to zoom in just a little bit
9:58:10and bring these tables a little bit
9:58:12closer so we can zoom in just a little
9:58:14bit more. Now we are going to go ahead
9:58:17and delete these. So we're going to say
9:58:19delete. Yes. And delete. Yes. So, just
9:58:25for demonstration purposes, we're going
9:58:26to build these relationships from
9:58:27scratch. So, we're going to come over to
9:58:29the customer information table and we're
9:58:31going to drag it all the way over here
9:58:33and put it on top of this custo ID or
9:58:35the customer ID in Apocalypse sales. And
9:58:38it's going to automatically create that
9:58:40relationship. And we can open this up.
9:58:43And as you can see, it created the
9:58:44relationship between this customer ID in
9:58:46the apocalypse sales and the customer ID
9:58:48in the customer information. It also
9:58:50defaulted the cardality from many to one
9:58:52and the cross filter direction to
9:58:54single. So we're going to go ahead and
9:58:56change that to both and click okay. And
9:58:58then we're going to come over here to
9:59:00the product ID and Apocalypse store and
9:59:02drag this over the product ID in the
9:59:03apocalyp sales.
9:59:06And again, if we open it up, it created
9:59:08that relationship for us. It created the
9:59:10cardality automatically. And we're going
9:59:12to change this cross filter direction to
9:59:13both and click okay. And so on a really
9:59:16small scale, that is how it works. Of
9:59:19course, it becomes a little bit more
9:59:20complex the more tables that you add and
9:59:23the more relationships that are created,
9:59:25but this is how you're going to actually
9:59:26create the relationships in the model
9:59:28tab within PowerBI. I hope that this
9:59:30tutorial has helped you understand this
9:59:32concept a little bit better. Thank you
9:59:34guys so much for watching. I really
9:59:35appreciate it. If you like this video,
9:59:37be sure to like and subscribe below and
9:59:39I'll see you in the next video.
9:59:41[music]
9:59:52What's going on everybody? Welcome back
9:59:53to the PowerBI tutorial series. Today
9:59:56we're going to be taking a look at DAX.
10:00:01[music]
10:00:03Now DAX stands for data analysis
10:00:06expressions and it's basically a library
10:00:08of functions and operators that help you
10:00:10build formulas. You can use DAX to
10:00:12create measures and calculated columns
10:00:14within PowerBI which can really give you
10:00:16a lot of insight into your data.
10:00:18Honestly, it is not super complicated
10:00:20and hopefully by the end of this video,
10:00:21you'll have a lot more confidence
10:00:23actually using DAX and PowerBI. So,
10:00:25without further ado, let's jump on my
10:00:26screen and get started with the
10:00:27tutorial. All right, so let's take a
10:00:29look at our tables and data before we
10:00:31get started. So, we have two tables, the
10:00:32Apocalypse Sales, the Apocalypse store.
10:00:35For this apocalypse sales table, we have
10:00:37the customer, product ID, order ID,
10:00:39units sold, and the date it was
10:00:41purchased. And then for the Apocalypse
10:00:44store, we have product ID, product name,
10:00:46price, and production cost. Now, these
10:00:49are joined together or they do have a
10:00:52relationship together via the product
10:00:54ID. So, what we're going to be using are
10:00:56these new measures and new columns to
10:00:58create our DAX functions. So, really
10:01:01quickly, let's go over to this report
10:01:03tab and let's drop down our fields over
10:01:06here so we can see everything. And so,
10:01:08to get us started, we're going to go
10:01:09right up here to Apocalypse Sales. We're
10:01:11going to rightclick and click new
10:01:13measure. And it's going to open up this
10:01:15right here, which is basically our bar
10:01:17where we can create our functions. And
10:01:19so, right here, it's automatically given
10:01:21us the name measure, but we can change
10:01:23that. And we're going to say count of
10:01:26sales. So, now we can start writing our
10:01:29DAX function. And that's just going to
10:01:30be the name of it and what's going to
10:01:31show up right over here once we click
10:01:33enter. So let's go over here and we're
10:01:36going to say count. And as we're typing,
10:01:39it's automatically giving us options. It
10:01:41has something called IntelliSense. If
10:01:43you've ever used other Microsoft
10:01:44products, IntelliSense is their kind of
10:01:46autocomp completion that helps you look
10:01:49at other options very quickly. And so
10:01:51we're just going to click on this count.
10:01:53And it's prompting us to put in a column
10:01:55name. And so we can come down here and
10:01:57we can select one or we can type it out
10:01:59and it'll try to predict and help us
10:02:01choose which column to select. So for
10:02:04us, we're going to use this order ID,
10:02:05but let's just start typing it out.
10:02:07We'll say order ID. And then we can
10:02:10click on it and we're going to close
10:02:12this parenthesis and click enter. Or you
10:02:14can go over here and click this check
10:02:16mark, but we're just going to click
10:02:17enter. And so over on this right side,
10:02:20it finalized that and saved that. And we
10:02:22can actually look at that by clicking on
10:02:24this box next to it.
10:02:26and we want to look at this in a table.
10:02:29So now we can see that there are 74
10:02:31sales. Now for this we want to see who's
10:02:34buying our products. We want to see what
10:02:36our what our client name is. So we're
10:02:39going to go over here and we're going to
10:02:40choose customer and we're going to put
10:02:42customer on top of sales and we're just
10:02:45going to take a look at it like this. So
10:02:48now we can see that our number one
10:02:50customer is Uncle Joe's Prep Shop. He
10:02:52has 22 orders. Now, they have the most
10:02:54orders with us, but it doesn't
10:02:55necessarily mean that they're spending
10:02:56the most money with us, but we can take
10:02:58a look at that later. The next thing
10:03:00that I want to take a look at is how
10:03:02many products we're actually selling.
10:03:04What are our big products that we're
10:03:05selling? We have 10 different items, but
10:03:08I don't know exactly which one is
10:03:10selling the best. If if one is doing
10:03:12really poorly and getting no orders,
10:03:14this is something that I want to look
10:03:15into. So, all we're going to do is go
10:03:16right back up here to Apocalypse Sales
10:03:18again, rightclick, and select new
10:03:21measure. And for this one, we're going
10:03:23to call it the sum of products sold.
10:03:28And all [snorts] we're going to start
10:03:29out with is by doing sum. And if this
10:03:33seems familiar to something like Excel,
10:03:36you're 100% correct. It is very similar.
10:03:38And remember, these are both Microsoft
10:03:40products. So there's going to be similar
10:03:42functionality in both of them. And so
10:03:45this DAX is going to have a lot of
10:03:47similarities to exactly how it has it in
10:03:49Excel. So, we're going to do an open
10:03:51bracket. And now, what we're going to
10:03:53choose is this units sold. We want to
10:03:56sum up all of these units sold and see
10:03:58how many we're actually selling. So,
10:04:00we're going to say units sold. I'm going
10:04:03to hit tab. It's going to autocomplete
10:04:05that. I'm going to close my parenthesis
10:04:07and I'm going to come over here and
10:04:08click this check box. So, now it's
10:04:11created that measure and we're already
10:04:12selected in this table. So, all we have
10:04:14to do is click the check mark and it's
10:04:17going to show us that we have 3,000
10:04:19total products sold and we can go
10:04:22through here and see what the big
10:04:23sellers are. And probably the biggest
10:04:25one that I see right off the bat is this
10:04:27multi-tool survival knife. So, these DAX
10:04:29functions that you can write can be very
10:04:31simple and lead to really good insights
10:04:33that you can use for the visualizations
10:04:35later on. Now I want to take a look at
10:04:37the difference between something like
10:04:38sum which is an aggregator function and
10:04:40something like sumx which is an iterator
10:04:43function. Because if you add x to some
10:04:45of these aggregator functions you can
10:04:47create them or or make them into an
10:04:50iterator function. So you can have sum
10:04:52and sumx or average and average x.
10:04:55Adding x onto the end of them can make
10:04:57them into an iterator function. So let's
10:04:59take a look and see how that actually
10:05:00works. I'm going to show you the
10:05:02difference and then I'm going to talk
10:05:03through the difference at the end. So,
10:05:05really quickly, let's go back to our
10:05:06data and let's go to the Apocalypse
10:05:09store. Now, what we have right here is
10:05:11we have the price and we have the
10:05:13production cost. And we want to see how
10:05:14much profit we're getting from each of
10:05:16these as well as we can take a look at
10:05:18the units sold and see how much money we
10:05:20are actually making. So, what we're
10:05:23going to do is we're going to come back
10:05:24over here. We're going to go to
10:05:26Apocalypse store. We're going to
10:05:28rightclick and create a measure. And in
10:05:30just a little bit, we're going to be
10:05:31creating a new column. and that'll kind
10:05:32of show the difference really well. So,
10:05:35we're going to create this new measure
10:05:36and we're going to name it profit
10:05:39and we're going to come over here and
10:05:41what we're going to do is we're going to
10:05:42take the sum oops we're going to start
10:05:45with our sums. We're going to take the
10:05:46sum of the price and then we're going to
10:05:50close that parenthesis and we're going
10:05:51to subtract the sum of the production
10:05:55cost.
10:05:57So, all that does is it says if
10:05:58something cost $20, if we sold it for
10:06:00$20 and it only cost us $10, that's $10
10:06:03in profit for that item. And then what
10:06:05we're going to want to do is we're going
10:06:07to actually want to encapsulate that
10:06:09really quickly because we're about to
10:06:10use multiply. And then we're going to
10:06:14sum. And now we're going to take the
10:06:16units sold. So, how many units were
10:06:19actually sold at that profit that we
10:06:21just made. So, let's see if that works.
10:06:23And let's click the check right here.
10:06:26And so we have the profit. So let's
10:06:27click on the profit. Oops, that's not
10:06:30what I wanted to do. Let's use a new
10:06:31one. Let's create a new uh table. We're
10:06:34going to click profit.
10:06:36And let's make it a table. And I'm going
10:06:38to pull this right over here.
10:06:40Now, we have our profit, but what I
10:06:42really want to know is which customer is
10:06:44spending the most money at my store. So,
10:06:47we're going to come right over here.
10:06:48We're going to click on customer and
10:06:51customer at the top. And just at a
10:06:52glance, we can see that Uncle Joe's prep
10:06:54shop is spending the most money at the
10:06:56store. Now, what I want to show you is
10:06:58the difference between sum and sumx. So,
10:07:01what I'm going to do is I'm going to go
10:07:03back to this profit and going to copy
10:07:06this
10:07:07this entire thing and we're going to go
10:07:09back here to this table. Now, we just
10:07:12created a measure and we were able to
10:07:14break it down by each customer. So,
10:07:17let's go back over here. Now, let's go
10:07:20up here to home and we're going to
10:07:22create a new column. And we're going to
10:07:25call this profit
10:07:30column. And we're going to literally
10:07:32paste the exact same thing into here.
10:07:35And we're going to hit enter.
10:07:39And each row is the exact same thing.
10:07:43So, what it's doing is it is going
10:07:44through the price. It's adding all of it
10:07:47up and calculating it at the bottom.
10:07:49It's adding the production cost. It's
10:07:50going all the way down and calculating
10:07:52it at the bottom. And then it's going
10:07:54over and looking at how many units it
10:07:56sold. And then it's performing this
10:07:58calculation up here. And then it gives
10:08:00us the total. And it's doing it for
10:08:02every single row. But that's not really
10:08:05what we want it to show. What we want it
10:08:07to show is the profit for each row. What
10:08:10we want it to say is here's the price
10:08:12for the rope, the production cost for
10:08:13the rope, and then how many units we
10:08:16actually sold. and then it'll calculate
10:08:18that and give us the actual profit for
10:08:20just that row. But we cannot do it by
10:08:23just using this sum. What we need to do
10:08:26is use something called sumx. So let's
10:08:29add another column. Let's go back to
10:08:31home. I'm going to say new column.
10:08:34And now we're going to say
10:08:37profit
10:08:39oops underscorec column
10:08:44sum x. And now we're going to use sum x
10:08:50and hit tab. And we need to choose the
10:08:52table that we want to put this in. So
10:08:54we're going to say apocalypse sales
10:08:56because that's table that we're looking
10:08:57at right here. We're going to say comma.
10:09:00And now we need to input an expression
10:09:01which it says it returns the sum of an
10:09:03expression evaluated for each row in a
10:09:06table. Before when you're just using
10:09:07sum, it's looking at all of these
10:09:09combined. Now it's taking it row by row.
10:09:12So what we're going to do is basically
10:09:13input the same thing as we did before.
10:09:15I'm going to copy. I'm going to paste
10:09:16that. It's not going to be correct. I
10:09:18need to get rid of these sums,
10:09:20but it's basically the exact same
10:09:22equation.
10:09:23Give me just a second. And let's get rid
10:09:26of this sum.
10:09:28and let's see if this works. So, let's
10:09:31click the check button.
10:09:34And now this looks a lot better. So,
10:09:37what this is now showing us is at a row
10:09:39level, this nylon rope made us 51,000,
10:09:42almost $52,000.
10:09:44The waterproof matches made us $15,000.
10:09:48And we can go down and look at each item
10:09:50and see how much that actually made us
10:09:53versus this profit column. And so that
10:09:56is the biggest difference between sum
10:09:58and sumx. Hopefully that made sense. I
10:10:00know that sum and sumx and and the
10:10:02difference between an aggregator
10:10:04function and an iterator function can be
10:10:05a little bit confusing, especially if
10:10:07you've never done it before, but
10:10:08hopefully that was a good example for
10:10:10you to understand that concept. Now,
10:10:11let's go back over here to apocalypse
10:10:14sales. Right here, we have a date
10:10:16purchase. Now, in the DAX function, we
10:10:18have some ways that we can interact with
10:10:20dates. And so, I want to take a look at
10:10:22those really quickly. So, we're going to
10:10:24go right up here and click on new
10:10:25column.
10:10:27And we're just going to leave that as
10:10:29column, but what we're going to say is
10:10:31day. So, there's a few different ones.
10:10:33We have day, dates, YTD, next day,
10:10:37previous day, and weekday. And they all
10:10:40are pretty self-explanatory. If you
10:10:42click on it, let's click on weekday. It
10:10:45says it's going to return a number from
10:10:461 to 7 identifying the day of the week
10:10:49of a date. So, let's use this really
10:10:52quickly. And so we're going to say date,
10:10:55purchased,
10:10:56and click tab, hit comma,
10:11:00and it's going to give us a three
10:11:02different options. Basically, it's a
10:11:03one, a two, and a three. Um, right here,
10:11:06if you hit this button, read more, you
10:11:08can read more on it. This is going to
10:11:10say Sunday's equal to 1, Saturday's
10:11:11equal to 7. I like this one personally,
10:11:13which is Monday equals 1. In my brain,
10:11:16it just makes more sense. So, I'm going
10:11:17to click on two. I'm going to close that
10:11:20parenthesis and we're going to I guess
10:11:22I'll say uh let's say day of week for
10:11:26the column. Let's click that checkbox.
10:11:30And now Saturdays are equal to sixes,
10:11:33Mondays are equal to one. This allows us
10:11:36to see which day of the week people are
10:11:38buying the most products on or or which
10:11:40day of the week is somebody submitting
10:11:43their orders on. And so let's go over to
10:11:45our report. Let's get rid of this. Just
10:11:49going to move this. Oh jeez, I hate
10:11:52moving stuff sometimes. All right,
10:11:54really quickly, I want to show you the
10:11:56difference between what we just did and
10:11:57what we already have. So, we have this
10:12:00um date purchased. And let's make that
10:12:04into a bar graph.
10:12:07And what we're going to be taking a look
10:12:08at is actually the units sold. So, right
10:12:12here we have this. And obviously for we
10:12:15don't want 2022. We're going to get rid
10:12:16of the year. We only have one quarter
10:12:19right here. We can see January,
10:12:21February, March. So we can tell that
10:12:23January has the most sales or the most
10:12:26units sold in that month. If we get rid
10:12:28of that, we go down to day. We do have
10:12:30some information, but we don't know what
10:12:32day of the week it is. It could change
10:12:34from month to month, and it's really
10:12:37hard to tell exactly what if there's any
10:12:39pattern there at all. That's where what
10:12:41we just created comes in handy. So,
10:12:43let's recreate this exact same thing,
10:12:45but instead we're going to use day of
10:12:47week. So, we're going to select day of
10:12:48week in units sold. Let's drag that
10:12:52down.
10:12:54Move this over right here. And this day
10:12:56of the week should be on the x axis.
10:12:59And it's really easy now to see if
10:13:02there's a pattern here. There's really
10:13:03not, at least not for this fake data
10:13:05that we have. Um, but just I I want
10:13:08these uh data labels on really quickly.
10:13:12Um it's not easy to see if there's any
10:13:14pattern. Again, Monday has the most. So
10:13:17maybe that that I mean it goes down a
10:13:19little bit and then it picks back up. So
10:13:20maybe middle of the week is our least uh
10:13:22sales day. Our Wednesdays and Thursdays
10:13:24are a little bit lower than the rest.
10:13:26And the beginning and the end of the
10:13:28week tend to be the highest. Again, not
10:13:30a huge pattern, but you know, it's much
10:13:32easier to see if there is a pattern from
10:13:34week to week or what day of the week now
10:13:36that we use this weekday function. And
10:13:38so this can be really, really useful.
10:13:41Let's go back here to our data. And now
10:13:43we're going to look at our last DAX
10:13:44function for this video. Let's go up
10:13:46here and create a new column. And we're
10:13:49going to be looking at something called
10:13:51the if statement. Now, if you've ever
10:13:52used Excel, I'm sure you have heard of
10:13:54this. And you can do the exact same
10:13:56thing here in PowerBI. And so we're
10:13:58going to name this one order size. Order
10:14:02size. And so all we're going to say is
10:14:05if we're going to click on this one
10:14:07right here. We need to perform our
10:14:09logical test. And then we want to say if
10:14:11it's true, what's our value? And if it's
10:14:13false, what is our value? So what we're
10:14:16going to be looking at is units sold. So
10:14:18we're looking at order size. So we're
10:14:20going to say if units sold is greater
10:14:24than 25.
10:14:26What's going to happen? If it is true,
10:14:28if the order is larger than 25, you want
10:14:30to say it's a big order.
10:14:33And if it's not, we want to say it's a
10:14:36small order.
10:14:38Super simple. We'll close that
10:14:40parenthesis. We'll click okay. And now,
10:14:43really quickly, we're able to see if
10:14:44this is a big order or a small order.
10:14:47And so, that is all I have for you
10:14:49today. There are a lot of other DAX
10:14:51functions, but the ones that we looked
10:14:52at today are ones that are very common,
10:14:55ones that you'll see the most. And there
10:14:57can be a lot of really complex and
10:14:59intricate DAX functions that you can
10:15:00create. And in our project at the end of
10:15:03this series, I will be sure to include
10:15:05some more complex DAX functions. But
10:15:08hopefully this gave you a good
10:15:09introduction into DAX so you know how to
10:15:11use it a little bit better. Thank you
10:15:13guys so much for watching. I really
10:15:15appreciate it. If you like this video,
10:15:16be sure to like and subscribe and check
10:15:18out all of my other videos on everything
10:15:20data analyst related. I will see you in
10:15:22the next video.
10:15:35What's going on everybody? Welcome back
10:15:37to the PowerBI tutorial series. Today
10:15:39we're going to be looking at how to
10:15:40drill down in visualizations. [music]
10:15:48So when I say drill down, I mean you're
10:15:50basically adding another layer beneath
10:15:52the top layer of the visualization. And
10:15:54when somebody clicks or drills down into
10:15:56that data, they can see more insights
10:15:58and more information on the top level of
10:16:01data. When you drill down, you can also
10:16:03drill up. And I will show you how to do
10:16:04that in this tutorial. So without
10:16:06further ado, let's jump on my screen and
10:16:07get started with the tutorial. All
10:16:08right, so before we get started, I
10:16:10wanted to remind you that you can find
10:16:11the data that we're going to be working
10:16:12with in this tutorial in the
10:16:14description. You can go and download it
10:16:15from my GitHub. Now, the two tables that
10:16:18we're going to be looking at are
10:16:18Apocalypse Sales and Purchase Tracker.
10:16:21And if you've ever created any
10:16:23visualizations, you've probably seen
10:16:24something like this where you'll have
10:16:26the store and the price. And this is the
10:16:28the things that we actually bought. So
10:16:30this is the total amount of apocalypse
10:16:32prepping uh equipment that we bought.
10:16:35And we'll put the store in this legend
10:16:37right here. And you've probably seen
10:16:39something like this. And if you're
10:16:40anything like me, you're going to be in
10:16:41a meeting and you're going to be
10:16:42presenting this and some higher up is
10:16:44going to be like, "Hey, Alex, that looks
10:16:45great. But I want to, you know, see what
10:16:48things we actually bought and targeted,
10:16:49how much this cost. can you create a
10:16:50visualization for that? And you're going
10:16:52to be like, well, I could or I could use
10:16:55drill down. And so, you could have done
10:16:57this in the first place, uh, which you
10:16:58should have. So, what we're going to do
10:17:00is all we're going to do is we're going
10:17:01to say we're going to say the product
10:17:04right here. And these are going to be
10:17:05the actual things. And we're going to
10:17:06put it right under store. Now, you can't
10:17:08see these things, right? But there is a
10:17:11a hierarchy here. So, once we added
10:17:14this, these options became available.
10:17:16Let's take it out. And all those just
10:17:18disappeared.
10:17:19And then if we add it back right here,
10:17:23they came back. And so you can do right
10:17:26here, which is click to turn on drill
10:17:28down. You can go to the next level in
10:17:30the hierarchy, or you can even expand
10:17:32all down one level in the hierarchy. So
10:17:34let's look at each of those really
10:17:35quickly. So let's click on this one.
10:17:37It's just going to turn on drill down
10:17:38mode. So now if I go and I click on
10:17:41target, it's going to drill down into
10:17:43these. And if we want to, I can then put
10:17:46product under this legend.
10:17:48And we can see all of those things. But
10:17:50of course, if we go back up, it's going
10:17:53to be all broken up into this clustered
10:17:54column chart, which is more like um
10:17:57this, which isn't exactly what we were
10:18:00going for, but it works. Now, uh let me
10:18:02get rid of this. I actually want store
10:18:04in the legend. Now, if we turn that off
10:18:06and we click, it doesn't do that
10:18:08anymore. So, what it does now is it just
10:18:11highlights Walmart. It highlights
10:18:12Costco. It highlights Target. So, we're
10:18:15going to keep that on. Uh but we can
10:18:17also do something called going down the
10:18:19next level of hierarchy. So let's click
10:18:21on that. And so now this is going to go
10:18:24down to the next level down to this
10:18:26product level because that is the next
10:18:27level. And now it's going to show us
10:18:29each of those things but it's going to
10:18:30have it broken out by the store. And so
10:18:33it's a completely different
10:18:34visualization but all within the same
10:18:37realm of the data that we're looking at
10:18:38and what we actually care about. So
10:18:40let's go back up in the hierarchy and
10:18:43then let's use this one right here which
10:18:44is expand all down one level in the
10:18:46hierarchy. And so this one is again
10:18:47extremely similar except it just
10:18:50visualizes it differently. And now what
10:18:51it's doing is Walmart rice, Target dried
10:18:54beans, Costco rice. So instead of having
10:18:56it all uh like this one where it's
10:18:59stacked on top of each other, it's
10:19:01breaking it down individually. So this
10:19:04one column would become three separate
10:19:06columns. Now I'm going to minimize this
10:19:08right here. Uh, I'm actually going to go
10:19:09back up in the hierarchy just for visual
10:19:12purposes. Now, I'm going to show you one
10:19:14more example. We're going to use this
10:19:16Apocalypse sales up here. And this is
10:19:18one that I actually use all the time.
10:19:20So, the one you've seen, you know,
10:19:22you'll get stuff like that, especially
10:19:23if you're working with like sales and
10:19:25stuff, but I work in operations, right?
10:19:27So, I have a lot of order IDs, product
10:19:31IDs, stuff like that. Now, this one,
10:19:33this one genuinely I use quite often.
10:19:36I'll have a customer. And let's make it
10:19:38we'll just go like this. We have a
10:19:40customer and we have units sold. And
10:19:43let's use the customer as the legend. So
10:19:47let's make this one quite a bit larger.
10:19:51And I'll have something like this. And
10:19:53they'll say, okay, well, we want to see
10:19:55the order IDs that go with it because we
10:19:58want to know what orders are actually
10:20:00happening for each of these people.
10:20:01Obviously, I'm not using this exact
10:20:03data, but very, very, very similar. And
10:20:06all you have to do is take these order
10:20:08IDs and slide it right under here under
10:20:10customer. And this visualization right
10:20:13here is something I've done a thousand
10:20:15times because what happens is is someone
10:20:18some stakeholder in our company is
10:20:20saying, "Hey, Alex, we want this and we
10:20:21want to know we want to drill down on
10:20:23this IP address. We want to drill down
10:20:26on this certain database. We want to
10:20:28drill down on something and we want to
10:20:29see the order IDs within them." So then
10:20:32all you do is you turn on drill mode or
10:20:34drill down mode. you'll click on it and
10:20:36you can see every single order ID that's
10:20:38in there and then they can go and look
10:20:40those up in their system and resolve
10:20:41them or whatever they're trying to do
10:20:43with it and it helps a ton and it's very
10:20:46very useful. This one is extremely
10:20:47applicable and that's really all drill
10:20:49down is again you have these different
10:20:51hierarchies as well um but for different
10:20:53things it's not as useful as you can see
10:20:56we also have this hierarchy which again
10:20:58is not as useful. So, it just depends on
10:21:01the data that you're using and how you
10:21:03want to use this drill down effect. But
10:21:05I promise you that drill down is used
10:21:07all the time, especially when you're
10:21:09giving presentations where people want
10:21:11to know more information than just the
10:21:13the visualization that you're
10:21:14presenting. So, I hope that this has
10:21:16been helpful. I hope that you understand
10:21:17drill down a little bit better. If you
10:21:19like this video, be sure to like and
10:21:20subscribe and check out all my other
10:21:22videos on PowerBI. Thank you and I'll
10:21:24see you in the next video.
10:21:26[music]
10:21:37What's going on everybody? Welcome back
10:21:39to the PowerBI tutorial series. Today
10:21:41we're going to be taking a look at
10:21:42conditional formatting.
10:21:48Now, conditional formatting may sound
10:21:50familiar because we looked at it in the
10:21:52Excel series, and it's very similar how
10:21:54you use it in Excel versus how you use
10:21:56it in PowerBI. Conditional formatting
10:21:58allows you to take a table or a matrix
10:22:00within PowerBI and use those cells to
10:22:03color code them and create gradients and
10:22:04different visualizations within the
10:22:06actual table or matrix. I'm excited to
10:22:08start this one. So, let's jump over my
10:22:10screen and get started with the
10:22:11tutorial. All right, so before we get
10:22:12started, if you want to use the data
10:22:13that we're using in this video, you can
10:22:15find it in the description on my GitHub.
10:22:17Now, conditional formatting is super
10:22:18simple, and you've most likely used it
10:22:20in Excel before, but you can also use it
10:22:22in PowerBI. And let me show you how to
10:22:24do that. So, the first thing we're going
10:22:25to do is come over to our Apocalypse
10:22:27store, and we're going to pull up our
10:22:30product name as well as the price. And
10:22:34what we can do is come over here, and
10:22:36we're going to go to price. And it has
10:22:38to be under the columns. So, you can't
10:22:40come over here and do this. We're going
10:22:42to come right over here to price and
10:22:43we're going to rightclick and let's go
10:22:45to conditional formatting and we have
10:22:47background color, font color, icons, and
10:22:49web URL. Let's take a look at background
10:22:52color first. This is most likely the one
10:22:53that we'll look at the most. So, we're
10:22:55going to get this popup and I'm going to
10:22:57slide this over. Now, there's a lot of
10:23:00different things we can customize in
10:23:01here. And the first thing I want to take
10:23:03a look at is format style. We have the
10:23:04gradient and what it's going to say is
10:23:06the lowest value will be this color,
10:23:08highest value will be this color. It'll
10:23:10give us this gradient color scale. And
10:23:12so we'll use that in just a little bit.
10:23:14But we can also create rules kind of
10:23:16like an if statement. And if it is
10:23:19between this range and this range, we'll
10:23:20give it a color. And if it's between a
10:23:22different range and a different range,
10:23:23we'll give it a different color. So
10:23:25we'll also try that one. And then we
10:23:27have this field value. Uh and this one
10:23:29is one that uh honestly I don't use that
10:23:31much. I've used it maybe once. And what
10:23:34you can do is select a text field like
10:23:36customer and you can do some
10:23:38summarizations on the first and last.
10:23:40And that is it. So what we're going to
10:23:42do is we're going to look at gradient
10:23:44specifically for not the customer but
10:23:47we're going to go back to the apocalypse
10:23:49store and we're going to do it on the
10:23:51price.
10:23:52Now what I'm going to do is keep it as
10:23:54the count because this is what the
10:23:55default is and we're going to go back
10:23:57and fix it later. But what we want our
10:23:59lowest value to be is this bright green
10:24:02showing that this it's it's a cheap
10:24:03product that's easy to purchase. The
10:24:06highv value ones are going to be just
10:24:08this shade of red, more expensive. And
10:24:10we'll do it on the count. Now remember
10:24:12the count is on each of these and we're
10:24:14not doing a count of how many are sold.
10:24:16We're doing a count of each product. So
10:24:17it's just one per row. So it all should
10:24:20be the same color. Let's take a look. So
10:24:22it is all the same color. But what we
10:24:24really want to show is the actual price,
10:24:27not just the count of the price. So
10:24:29let's go back to conditional formatting.
10:24:31We're going to click the background
10:24:32color again. And this time we're going
10:24:34to change the summarization.
10:24:36Now you can do sum, you can do average,
10:24:39minimum, maximum. It really doesn't
10:24:41matter for this example. The number is
10:24:43the same regardless of really which one
10:24:45we choose. So we can just choose the
10:24:47minimum. And it's going to choose the
10:24:48minimum of each row, which is the price.
10:24:51So, we're just going to select minimum
10:24:52for this example. We'll select okay, and
10:24:55it should correct it accordingly, which
10:24:57means the bright green is the lowest,
10:24:58and it goes all the way up to the
10:25:00highest, which is the red. Now, let's go
10:25:02over here to apocalypse sales. We'll add
10:25:05in the units sold,
10:25:07and let's move that out a little bit.
10:25:11And I'm doing that on purpose because
10:25:12we're about to look at something within
10:25:14the conditional formatting. So, let's go
10:25:16to units sold, and we'll look at the
10:25:17conditional formatting for this one.
10:25:19Now, if you noticed, we now have a new
10:25:22one on here called data bars. Now, we're
10:25:25able to see data bars on units sold and
10:25:27not price because units sold is
10:25:30something like a sum, an average,
10:25:32something that's aggregated. But let's
10:25:33take a look at data bars cuz I want to
10:25:35show you how to use this and then we'll
10:25:36go back to the background color. So, for
10:25:39data bars, we are going to taking a look
10:25:41at the lowest or the highest value.
10:25:44Again, we're going to go from bright
10:25:46green all the way to
10:25:50this exact red. It's going to be from
10:25:52left to right. And what it's going to
10:25:54show you is if it is a positive number,
10:25:55which all of these are, is going to be a
10:25:57green bar basically representing the
10:25:59number that you see in here along this
10:26:01line. So, let's click okay.
10:26:05And we're going to be able to see the
10:26:07highest numbers. And let's scooch this
10:26:09over quite a bit so you can kind of get
10:26:10a better understanding. and we're going
10:26:12to do it from highest to lowest. So, we
10:26:15sold the most multi-tool survival knives
10:26:19at 477. And so, this entire bar, this
10:26:22row is entirely filled up or almost all
10:26:24the way filled up. While as it gets
10:26:26lower and as we sell only 182 solar
10:26:30battery flashlights, the bar is going to
10:26:32represent that and show that. Now, I'm
10:26:34about to completely mess up this
10:26:35visualization on purpose because it's
10:26:37about to get very messy to show you that
10:26:39you can do a little bit too much. Uh, it
10:26:41is possible. What we're going to do is
10:26:43we're going to go right over here to
10:26:45this background color units sold and
10:26:46instead of gradient, let's look at
10:26:48rules. Now, with the price, we just did
10:26:51a gradient scale, but we can do
10:26:54basically groups of these and say if a
10:26:56number is greater to or equal than this
10:26:58number, then it's going to be a certain
10:27:00color. And then if it's in a different
10:27:01range, we can give it a different color.
10:27:03So, we're going to say if it's greater
10:27:04than or equal to zero, and we're going
10:27:07to say number, not percent. And if it's
10:27:10less than 266, because we have got 265
10:27:14right here, let's make it a nice uh like
10:27:17gold, a beautiful, lovely mustard gold.
10:27:20Just just great. Now, we're going to say
10:27:22if it's greater than or equal to, we'll
10:27:25do 266 because this says less than 266.
10:27:28So, it should be greater than or equal
10:27:30to 266 number. And if it is less than
10:27:34we'll say 500.
10:27:36Now, we want to do this one and we'll
10:27:39give it uh let's do like a peach. And
10:27:41we'll click okay. And now we have
10:27:43another conditional formatting on top of
10:27:45that that can give us more information.
10:27:48Now, again, you should not do this. It's
10:27:51just too many. Now, let's go one step
10:27:53further and make it even more ridiculous
10:27:54and show you one more thing before I
10:27:56show you how you may actually want to
10:27:58use this. Uh let's go back to unit sold.
10:28:01We're going to rightclick, go to
10:28:02conditional formatting, and you can do
10:28:04something called icons. Um, font color
10:28:07is the exact same thing as background
10:28:08color except it changes the the font.
10:28:10And so I'm not really going to look into
10:28:11that one. Icons are very simple,
10:28:14extremely similar to Excel and how
10:28:16you've seen them. And the rules that you
10:28:18can apply to them are basically the same
10:28:21as if you're doing like a gradient. And
10:28:23it's these if statements that we saw
10:28:24before. Now, it auto gives us this right
10:28:28here, which basically says 0 to 33%, 33
10:28:31to 67, 67 to 100. If it's in the bottom
10:28:34third percent, it gives us this red, the
10:28:36middle is yellow, and the top is green.
10:28:38So, we can go through and change all of
10:28:40this. But honestly, this looks pretty
10:28:42good. So, let's click on it. And so, the
10:28:45ones that are our least sellers are
10:28:46these red ones right here. And the top
10:28:49sellers are up here. Now, this is just
10:28:51based on units sold. And this looks
10:28:53absolutely terrible. So, let's kind of
10:28:55take this exact information, but make it
10:28:58a little bit better. So, we're going to
10:29:00create a new visualization or at least a
10:29:02new table. So, let's click on product
10:29:04name and we'll take the price, units
10:29:08sold, and revenue. And what I think
10:29:11makes the most sense for looking at
10:29:12revenue is these data bars right here.
10:29:14But there's only one problem. I can't do
10:29:17that because it's not summarized like
10:29:20unit sold was. But what I can do is to
10:29:23get that those data bars is I can come
10:29:25right down here instead of saying don't
10:29:26summarize, I can summarize it. I can
10:29:29just click the sum. So it now is
10:29:32summarized. It's the exact same number.
10:29:34But if I right click on here as sum of
10:29:37revenue, I go to conditional formatting.
10:29:39I can now use those data bars. And so
10:29:41we're going to use those data bars. And
10:29:43we're going to say for the lowest value
10:29:44and the highest value. And let's just
10:29:47make it a nice
10:29:49a darker green. I don't want it to Well,
10:29:51that's that's hideous. Let's make it
10:29:53this color right here. A nice dark
10:29:55green. And there's no negatives, so it
10:29:56doesn't really matter. We're going to go
10:29:58left to right. And you can show the bar
10:30:00only, but we're going to keep it because
10:30:02I want to see it. And we're going to go
10:30:04just like this. We're going to order.
10:30:07And this is pretty telling. Um,
10:30:10honestly, I did not think the
10:30:12weatherproof jackets were performing so
10:30:14well, but I mean, they are by far our
10:30:16number one seller. So, you know, our
10:30:18weatherproof jackets, multi-tool,
10:30:20survival knives, and the nylon rope are
10:30:22perform outperforming all of our other
10:30:25products. So, those might be the ones
10:30:26that I focus on the most. While duct
10:30:29tape, the N95 masks and waterproof
10:30:31matches, I mean, those are those are
10:30:32garbage. So, I might be looking to
10:30:34replace those in the near future with
10:30:35some other items that might sell a
10:30:37little bit better. So, that's how you
10:30:38use conditional formatting, and it's
10:30:40actually pretty useful. There are a lot
10:30:41of times where I've done something like
10:30:42this in an actual visualization for
10:30:44work, and it looks something like this.
10:30:47It just depends on what you're
10:30:48visualizing, but this is very much a
10:30:51simple thing that you can do to just add
10:30:53a little bit more information and and
10:30:55actual visuals to this little chart or
10:30:58table that you're going to create.
10:30:59Sometimes it's just better to have these
10:31:01simple visualizations on this table
10:31:03rather than just having the numbers
10:31:04themselves. Makes it a little bit more
10:31:06easy to read and understand. So again, I
10:31:09hope that this was helpful. Thank you
10:31:10guys so much for watching. I really
10:31:12appreciate it. If you like this video,
10:31:13be sure to like and subscribe and check
10:31:15out all my other videos on PowerBI. And
10:31:17I'll see you in the next video.
10:31:23[music]
10:31:30What's going on everybody? Welcome back
10:31:32to the PowerBI tutorial series. Today
10:31:34we're going to be taking a look at bins
10:31:35and lists.
10:31:42Now, bins and lists are really useful
10:31:44because they allow you to group things
10:31:45together to analyze and visualize them
10:31:47easier. So, in this tutorial, I'll show
10:31:49you how to create your bins and lists
10:31:51and then we'll create some
10:31:52visualizations to show you how it can be
10:31:53helpful. So, without further ado, let's
10:31:55jump on my screen and get started with
10:31:56the tutorial. All right, so before we
10:31:58get started, I wanted to let you know
10:31:59you can go and download the data that
10:32:01we're going to be using in this tutorial
10:32:03in the description below. It is on my
10:32:05GitHub. So, we are going to be looking
10:32:07at bins and lists today. Um, and for
10:32:10this, we're going to be going over here
10:32:12to this apocalypse sales. Uh, and let's
10:32:15open up our data right over here. And we
10:32:18want to look at apocalypse sales really
10:32:20quickly. I feel like more people would
10:32:22know what a bin is. So, we'll kind of
10:32:23start with a list. Just go a little bit
10:32:24backwards than we normally would. Uh,
10:32:26I'm going to use this customer or we're
10:32:28going to use this customer column right
10:32:30here for a list really quickly. And you
10:32:32can do that in two ways. You can come up
10:32:33here and you can rightclick on the
10:32:35customer and go to new group. Or you can
10:32:38come over here under this uh the field
10:32:40section on the far right and go to
10:32:43customer, rightclick and click new
10:32:45group. So let's click on that now.
10:32:48And right now is only giving us the list
10:32:52type. It's not giving us bins because
10:32:54bins have to be numeric. So we really
10:32:56can't do that at the moment. Um, so
10:32:58we're going to call this just customer
10:32:59groups just or or we'll actually call it
10:33:02list just so it's easier to recognize
10:33:04when we create it. And so all we're
10:33:05going to do is we're going to basically
10:33:07group these, but it's going to be called
10:33:10a list. And so what we're going to do is
10:33:12we're going to select and we're going to
10:33:14select and we're going to say group and
10:33:17click on this group button. And then it
10:33:18creates [clears throat] this Alex the
10:33:19analyst apocalypse preppers and uh this
10:33:22prep for anything prepping store. So
10:33:24that it kind of named it for us. But if
10:33:27we double click on it, then we can
10:33:30rename this and we can call this the
10:33:33best prepping stores.
10:33:37And then we have these last two and we
10:33:40can we can click on one and then click
10:33:43control and click on the other one. So
10:33:45we get both of them. And then we can
10:33:47click group and we can call this and
10:33:50we'll double click and we'll call this
10:33:52the worst
10:33:54prepping stores.
10:33:56Um, and then that's it. And that's all
10:34:00we have to do. And what we're then going
10:34:02to do, and if you want to undo this and
10:34:04you want to switch it up and do
10:34:05whatever, you can click ungroup, but
10:34:06we're not going to do that. We're going
10:34:07to click okay.
10:34:09And here is the column that it created.
10:34:12And it basically tells us what list we
10:34:14put it in. If it's Uncle Joe's prep
10:34:15shop, that's in the worst prepping
10:34:17stores list. And if it's the Alex the
10:34:19Analyst Apocalypse preppers, that is in
10:34:21the best prepping stores. So, it's kind
10:34:23of like an if statement. You could even
10:34:25create a calculated column, do it on
10:34:28this customer, create an if statement.
10:34:30This is just a lot faster and a lot
10:34:32easier than doing that, but it basically
10:34:34would do the exact same thing. Now, you
10:34:36can use lists as well on things like
10:34:39numeric. So let's say we have order ID
10:34:43and we'll go to new group and it's going
10:34:45to auto go to bin because typically
10:34:47that's what you'll use but you can do
10:34:49list as well. And let's say you know we
10:34:52want to say we want to call these like
10:34:55we'll group these and call these the
10:34:57first
10:34:59um we'll call this the first customers
10:35:01or the first orders because we're
10:35:03looking at order ids. Look at the first
10:35:04orders. And then we will go back here.
10:35:08We're going on the left side. We're
10:35:10going to click. Oops. We're going to go
10:35:11back to the top. We're going to hit
10:35:13shift group all of these. And we'll say
10:35:17the latest orders.
10:35:20And you absolutely can do this. Um,
10:35:22again, this is kind of like an if
10:35:23statement, right? So, you're saying if
10:35:25it falls between this range and this
10:35:27range, then it's called the first
10:35:28orders. And if it's between this range
10:35:30and this other range, it's the latest
10:35:32orders. Um, again, it's just a much
10:35:35simpler version of an if statement. And
10:35:37so, you don't have to write it all out.
10:35:39you can just have this user interface
10:35:40kind of do it for you. Uh, and and it's
10:35:43really really useful. So, now let's talk
10:35:44about bins. And by far the easiest way
10:35:46to demonstrate this, and I'll show you
10:35:48one other way. U, but by far the easiest
10:35:50way to show this is by using age. And
10:35:53so, uh, for absolutely no reason
10:35:55whatsoever, these customer IDs, uh, who
10:35:58are right here in this customer
10:35:59information, they decided to give us
10:36:01some of their buyer information who are
10:36:03actually buying their products on their
10:36:05website or in their store. they just
10:36:06decided to give it to us as well as some
10:36:08uh simple demographic information. I I
10:36:11don't know why. But what we're going to
10:36:12use bins for is grouping these age
10:36:16brackets. So, you know, you might be
10:36:18interested and say, well, I want to know
10:36:21if my core population who are buying my
10:36:23products are within a certain range. And
10:36:25you don't want to look at every single
10:36:26age because then it just, you know, in
10:36:29your visualizations, it's not going to
10:36:30look right. You want to kind of group
10:36:32them, make it easier to visualize. So,
10:36:34what we're going to do is we're going to
10:36:35go through here and we're going to
10:36:36basically go by 10. So, 10, 20, 30, 40,
10:36:4050, 60, and see what age bracket these
10:36:42people fall in. So, we're going to go to
10:36:44age. We're going to rightclick and we're
10:36:45going to say new group. And we're going
10:36:47to go to bin and we'll leave it as a
10:36:49default age bins. Um, and you can do two
10:36:52things. You can do the size of the bins,
10:36:54which splits it uh uh which splits it by
10:36:57this number right here. Or you can go
10:36:59based on the number of bins. So, if you
10:37:02only want to do five different bins,
10:37:04it'll calculate that for you and it'll
10:37:06say, "Okay, if you only want five bins,
10:37:09you're going to have to do it at 12.2.
10:37:11If you want 10 bins, it could be 6.1."
10:37:15But it is completely up to you on how
10:37:17you want to do that. Um, you can do the
10:37:19size and we'll just say every 10, which
10:37:22is what we're going to do. Or you can go
10:37:23through and then you can create, you
10:37:25know, the how many bins you actually
10:37:26want. So, let's go ahead and click okay.
10:37:30and it's going to create those bins for
10:37:32us. So, if somebody is 78, they're going
10:37:34to be in the 70s bin. If somebody is 41,
10:37:37they'll be in the 40 bin. If somebody is
10:37:4029, they'll be in the 20 bin. And so on
10:37:42and so forth. So, when we go to
10:37:44visualize this, we don't have, you know,
10:37:4671, 72, 73, 74. Have a lot more things
10:37:50on our visualization. It'll just be the
10:37:5270 or it'll just be the 20. Now, we can
10:37:54also use bins on dates as well. So,
10:37:57let's go back to apocalypse sales. we
10:37:59have this date purchased. So we can
10:38:01create a bin for this as well. So let's
10:38:03go to date purchased. Let's go new
10:38:05group. Now you can also create a list
10:38:08and that's totally fine if you would
10:38:10like to do that. Um and it would look
10:38:12kind of like this where you can go
10:38:14through and you can select it and you
10:38:16can say okay this group all these dates
10:38:20you can group those and say this is
10:38:21going to be January. Uh and you can do
10:38:25that and that's totally okay. Um, but
10:38:27for this one, we're going to do bins. I
10:38:28think it's a little bit easier to do
10:38:30bins because what we can do is go right
10:38:32here and we can specify if we want
10:38:34seconds, minutes, hours, days, months,
10:38:36or years. And so, um, for the data that
10:38:38we have, it goes January, February, and
10:38:41March. So, we're going to do months, and
10:38:43we're going to say the bin size is going
10:38:45to be one month. So, each month should
10:38:47have its own bin. So, it'll be three
10:38:48bins total. So, we're going to select
10:38:50okay.
10:38:52And as you can see on this right side,
10:38:54we have January of 2022 and that
10:38:56correlates to the January over here.
10:38:58Then it goes down to February and then
10:39:01it goes down to March. And then when we
10:39:04visualize this uh we don't have to do
10:39:06this the hierarchy stuff that we do in
10:39:08here where we filter it down down to
10:39:10months. We can just use this right here
10:39:12and that will be our months column. So
10:39:14now let's go over to our visualizations
10:39:15and we'll see how this looks really
10:39:17quickly. We're not going to look at all
10:39:18of them but we will take a look at a few
10:39:20of them. So the first one that we can
10:39:22look at is age. So let's look at the
10:39:24buyer ID and then we'll do age as well.
10:39:28And so let's spread this out.
10:39:32And we can see our distribution of our
10:39:34buyers. So it looks like we have very
10:39:36few uh who are in the 10 range, thank
10:39:39goodness. And we can even put the age
10:39:41right under here under the age bins. And
10:39:44we have this now we kind of have this
10:39:46drill down. And so if we go right here
10:39:48and we drill down right there, this will
10:39:51actually give us the breakdown. So this
10:39:52is what it would have kind of looked
10:39:54like, our visualization would have
10:39:56looked like if we had just kept it the
10:39:57age cuz now we're drilling down into the
10:39:59age. And so it looks like we have one
10:40:0118-year-old and maybe a 20-year-old as
10:40:04well. Um let's go back up. Yeah. So it
10:40:07looks like we only have one buyer ID.
10:40:08Yeah. So there's only one 18-year-old.
10:40:10so of legal age to start buying, you
10:40:12know, all these prepping equipment and
10:40:14probably uh buying online and stuff like
10:40:16that, which makes sense, right? So, uh
10:40:19this gives you kind of a quick breakdown
10:40:20in the bins rather than um doing it the
10:40:23alternative way. So, now let's take a
10:40:25look at the customer list as well as the
10:40:28units sold. And it looks like the best
10:40:31prepping store uh is actually performing
10:40:33much worse, surprisingly uh than the
10:40:36worst prepping store. And so I hope this
10:40:38gave you a really good idea of how to
10:40:40use bins and lists within PowerBI. Thank
10:40:42you so much for watching. If you like
10:40:43this video, be sure to like and
10:40:45subscribe and check out all my other
10:40:46videos on PowerBI. I'll see you in the
10:40:48next video.
10:40:50[music]
10:40:57[music]
10:41:01What's going on everybody? Welcome back
10:41:02to the PowerBI tutorial series. Today
10:41:05we're going to be taking a look at all
10:41:06types of visualizations.
10:41:13Now, when you're working in PowerBI,
10:41:15there are a lot of different options to
10:41:17create visualizations, and you may not
10:41:19always be sure which one to use. And so,
10:41:21that's what this video is for. I'm going
10:41:22to walk you through a lot of the
10:41:24visualizations that I like and I use a
10:41:26lot as well as kind of point out some of
10:41:28the ones that I don't like as much so
10:41:30that you get kind of a feel for the ones
10:41:32that I think are really popular and that
10:41:34are used the most. So without further
10:41:35ado, let's jump into PowerBI and start
10:41:37taking a look. All right, before we jump
10:41:38into it, there is a link in the
10:41:40description where you can get the data
10:41:41that we're going to be using for these
10:41:42visualizations if you want to practice
10:41:44them yourself. Before we actually get
10:41:47into it, we do need to combine this. And
10:41:50if you download that Excel and you see
10:41:51this, you'll have to do the same thing.
10:41:54All we have to say is that this product
10:41:56ID is the same as this product ID
10:41:58purchased. And now we are good to go. Do
10:42:02one to many. And it's okay if it's one
10:42:03way. So right over here under this
10:42:05visualizations tab, there are lots of
10:42:07different options and it can be a little
10:42:09bit overwhelming. You don't really know
10:42:11which one to choose. There are some in
10:42:13here that I have almost never used for
10:42:15my job ever. So, I'll point those out as
10:42:17we go through, but the main focus is
10:42:19going to be focusing on the ones that I
10:42:21do use or that I have used and showing
10:42:23you how to actually create that
10:42:24visualization, maybe spice it up just a
10:42:26little bit. But, we have a lot of them
10:42:28to go through. So, let's jump right into
10:42:31it. And the very first one that we're
10:42:32going to start with, probably the
10:42:33easiest one and the one that you'll
10:42:34recognize the most is a stacked bar
10:42:37chart. And what we're going to do is go
10:42:38ahead right over here to the product
10:42:41name. And we want this unit sold as
10:42:44well. So, we're going to click product
10:42:46name, and it's going to go straight into
10:42:47the y-axis for us. And then we're going
10:42:49to click units sold, and that will go
10:42:51into the x-axis automatically. It just
10:42:54kind of intuitively knows, but sometimes
10:42:56it will make a mistake. And then you can
10:42:58just fix it or flip it. And we do want
10:43:01this uh let me make this much larger. We
10:43:04do want this to be a little bit more
10:43:05color-coded. That is what this legend is
10:43:07down here. So, what we're going to do is
10:43:09drag this product name down to the
10:43:11legend. And now we have each product as
10:43:14its own color. And in previous videos,
10:43:17we have gone through and looked at some
10:43:19of these visual and general options that
10:43:21you have when you're actually creating
10:43:23these visualizations, but we're going to
10:43:24do some of them while we're in here as
10:43:26well. So, we're just going to go down
10:43:28here. We're going to choose data labels
10:43:31and we're going to shrink that. And if
10:43:34you go higher, the higher you go, the
10:43:35less you see. So, if you want all of
10:43:37them all the way down to the green,
10:43:39we're going to go right about there. and
10:43:40we're going to make it smaller. So now
10:43:42we can go ahead and click anywhere
10:43:43outside of that visualization and now we
10:43:45can create a new one. If we had just
10:43:47kept it like this where we were still
10:43:49interacting with this visualization and
10:43:51we clicked on a different one, it would
10:43:53have then changed our visualization
10:43:55completely which we don't want. So let's
10:43:57hit control Z, click out of it and now
10:44:00we can create a new one. Let's go right
10:44:02over here to this 100% stacked column
10:44:04chart. I'm going to click on it, drag it
10:44:07over here and make it much larger. And
10:44:10we're going to come right over here to
10:44:12this customer information. And we're
10:44:14going to click on customer. And then
10:44:16we're going to go up to units sold and
10:44:18click on units sold. And we want to
10:44:21break these out. And so basically what
10:44:22this is doing is it's breaking it out by
10:44:24each of these shops. And we can see the
10:44:27total of what they're buying, the units
10:44:29sold. But we want to see exactly what
10:44:32products make up this percentage or this
10:44:34100%. So we're going to go right over
10:44:36here to product name. We're going to
10:44:38drag that down to the legend. And as you
10:44:40can see, now we have each of these
10:44:43products and each of the products is up
10:44:44here. So this backpack, we can see the
10:44:47backpack right here. Backpack right here
10:44:49and right here. And we can see which
10:44:50customer is buying what percentage of
10:44:52their purchases. So for this Prep for
10:44:55Anything prepping store, they have a
10:44:56very large percentage, 40% is duct tape.
10:44:59So they're buying a lot of duct tape. So
10:45:02really quickly, we're able to see what
10:45:03clients are purchasing or which clients
10:45:05are purchasing what products the most.
10:45:06So, just like this Alex Analyst
10:45:08Apocalypse preppers, they're buying a
10:45:10lot of water purifiers. We like drinking
10:45:12clean water. Um, you know, that's just
10:45:14what my audience likes. And so, you
10:45:16know, we can easily get a quick glance
10:45:18of that. Again, we're going to go in
10:45:20here. I tend to like putting these data
10:45:22labels on here. That's just what I
10:45:25prefer. So, you know, something like
10:45:27this. It looks nice. It looks clean. Um,
10:45:29we can always go back and change these
10:45:32names, which we'll do for this one. So,
10:45:33we're going to go over here, go to
10:45:35title. We'll go down to the text and
10:45:38we'll do customer.
10:45:41Oops.
10:45:43Customer purchase. Oh jeez. Breakdown.
10:45:49Pretend I'm really good at spelling. And
10:45:52we're going to do it just like that.
10:45:54We'll get out of there. So now we have
10:45:55customer purchase breakdown. And that
10:45:57looks really nice. It's a good uh a good
10:46:00visualization. And we're going to bring
10:46:01that right over here. We're going to
10:46:04have a lot on the screen. And so I may
10:46:06have to uh make them smaller or larger
10:46:09to fit everything. All right, so let's
10:46:12go on to our next one. Another really
10:46:14common visualization is this one right
10:46:17here, which is the line chart. And the
10:46:19line chart is great, especially when
10:46:21you're using things like dates. I have
10:46:24found this one to be the best and a lot
10:46:26of people use this as well. So we're
10:46:27going to go right over here and click on
10:46:29date purchased and then units sold. And
10:46:32on the x-axxis, you can see it's broken
10:46:34up by year, quarter, month, and day. So,
10:46:36we don't want to do it that high level.
10:46:37We only have three months of data in
10:46:39here. So, we're going to get rid of the
10:46:40year. We're gonna get rid of the
10:46:42quarter. And then we at least have this.
10:46:45And let's break it out because right now
10:46:47we're looking at all of the units sold.
10:46:49So, we're going to drag the product name
10:46:51right down here to the legend. And now
10:46:53it breaks it out by the actual product.
10:46:55And for each month in January, February,
10:46:57or March, you can follow these products
10:46:59and see how they did in each of those
10:47:01months. And if we wanted to, we can come
10:47:02right over here to the filter on the
10:47:04product name and we could filter it by
10:47:06maybe the top three. So let's do
10:47:08multi-tool survival knife, the nylon
10:47:12rope, and the duct tape. And we can have
10:47:15it just like this. And you know, you can
10:47:18do those for any product that you want,
10:47:20but again, we just want to do it for
10:47:21those three just for an example. And
10:47:23that really doesn't give us a ton of
10:47:25information. We could even go down to
10:47:26the day and, you know, it might give us
10:47:29a little bit more information. And so
10:47:31we'll keep it like that. And we can go
10:47:33over here,
10:47:35change the name as well. We're not going
10:47:36to do this for all of them. Again, we're
10:47:38just looking at the different types of
10:47:39visualizations I think are really good
10:47:41to know. But we'll change this one as
10:47:43well to products purchased
10:47:48by date.
10:47:50We'll keep it just like that. Again,
10:47:52nothing fancy. We're just trying to look
10:47:53at a bunch of different stuff. So, let's
10:47:55put this over here
10:47:57down here. Now, let's click out of
10:47:59there. And there are other ones in here
10:48:02um that are definitely useful and you
10:48:04absolutely can use. Um like this one is
10:48:06a stacked bar chart. This one is a
10:48:08stacked column chart. It's basically the
10:48:09same thing just a different orientation.
10:48:12We went to here just a different
10:48:14orientation. It's the same thing. Um
10:48:17just like this clustered bar chart,
10:48:19clust column chart. It's just its
10:48:21orientation either horizontal or
10:48:22vertical. Then we have things like an
10:48:25area chart, a stacked area chart. Not
10:48:28really things that I've used too much in
10:48:30previous positions. One that I have used
10:48:33though is a line and clustered column
10:48:35chart. So it kind of combines a few of
10:48:38these with, you know, you have these bar
10:48:41charts as well as line charts into one
10:48:44visualization. So let's look at this one
10:48:45because this is one that I have used
10:48:47several times in my actual job. So for
10:48:49our xaxis, we'll use the product name.
10:48:53Then we'll look at something like the
10:48:55price. And so let's make this a lot
10:48:57larger
10:48:59so we can actually see it. So now we
10:49:02have the price and now we can look at
10:49:04something like the production cost and
10:49:06that can be
10:49:08our line yaxis. So now we're looking at
10:49:11the price of it, how much someone is
10:49:12actually paying for it. And then we're
10:49:14looking at how much it's costing us to
10:49:16actually produce that product. And so
10:49:18really quickly at a glance you can kind
10:49:19of see that it's around the halfway to
10:49:212/3 point on most of these. You can see
10:49:24that the production cost is always lower
10:49:27than the actual price because of course
10:49:29we're out here to make a profit on these
10:49:30products. So, let's minimize this one.
10:49:33We're going to put this one right down
10:49:34here. Let's make it even smaller. Let's
10:49:37click out of that. And the next one that
10:49:39we're going to take a look at is a
10:49:41scatter chart. So, let's click on that
10:49:43and make it much larger. Oops.
10:49:47There we go. So, let's use the price and
10:49:50the production cost again. And so our
10:49:53x-axis is the price. Our yaxis is the
10:49:56production cost. But now we need to fill
10:49:57in this values right here. So let's go
10:49:59over here and click on the product name
10:50:01and drag that into values. And so now we
10:50:03have our values. We just don't know what
10:50:05they are, but we can see it. So let's
10:50:08drag this down to legend as well. And it
10:50:10breaks it out. And we kind of have this
10:50:12scatter plot. And you know, for this
10:50:14fake data that we're using, it doesn't
10:50:16really show a lot. Uh, but if you're
10:50:19using real data, you can definitely find
10:50:20outliers and trends and patterns using
10:50:22this type of visualization. Let's go
10:50:24ahead and make that one small as well. I
10:50:27get right down into the corner.
10:50:30Now, let's go right over here and we
10:50:32have the the dreaded pie charts. Um, and
10:50:34donut charts. Now, look, I think it's
10:50:36kind of a joke in the data analyst
10:50:38community about pie charts and doughut
10:50:40charts, but at the same time, people use
10:50:42them and they request them. And so,
10:50:43sometimes you're going to use it whether
10:50:45you like it or not. So, let's click on
10:50:47the doughut chart and let's make this
10:50:50one a lot larger.
10:50:52And let's go over here and let's click
10:50:54on state. And we're also going to click
10:50:57on total purchased. And that's really
10:51:00all you have to do. These ones are
10:51:03pretty straightforward. You can change a
10:51:05few different things like where these
10:51:07labels are. If you want them inside, you
10:51:09can also do that. That would look
10:51:11totally fine. Um, again, I'm just not a
10:51:14super huge fan, but you will get this
10:51:15one requested. People like this and want
10:51:17to see it. And the reason a lot of
10:51:19analysts don't like using this is
10:51:21because when you start glancing at
10:51:23these, it's really hard to tell the
10:51:25difference between these sizes. If you
10:51:27look at something like this, you can
10:51:29easily see that this is larger. Like, if
10:51:31you're looking at this one, the
10:51:32multi-tool survival knife is obviously
10:51:34the longest, and it gets shorter,
10:51:35shorter, shorter, shorter. But when you
10:51:37start getting in here, it's really hard
10:51:38to approximate the size. I would not be
10:51:40able to tell the difference between this
10:51:425.63, 5.78, two, 7.72. I would not be
10:51:47able to tell really the difference
10:51:48between these or or kind of the the
10:51:50difference between them very easily.
10:51:53That's why a lot of people don't want to
10:51:54use them in general. So again, I want to
10:51:57show you this one because I think it's
10:51:59worth noting and worth knowing how to
10:52:01use, but I don't really push people
10:52:04towards this because I don't think it's
10:52:06the best visualization available most of
10:52:08the time. All right, the next two are
10:52:10super easy, but are used all the time.
10:52:13Uh maybe more than some of these even,
10:52:15but they're just so easy to use. So, I
10:52:18kind of saved them for last. This one is
10:52:20the card. And all the card is is it
10:52:23displays one number or multiple numbers
10:52:25if you want to use a multiro card, but
10:52:27we'll just look at the card for now. All
10:52:29we're going to look at is the total
10:52:30purchased. And it's just going to
10:52:32display it just like this. And you can
10:52:34make it as large or as small as you'd
10:52:36like. And normally it goes on like the
10:52:38top. and you'll put card here, a card
10:52:40here. Just for example, I'll kind of
10:52:43show you how this might look. So, it
10:52:44look something like this, right? And at
10:52:47the top, it'll have different usually
10:52:48high overarching information. And this
10:52:51is super common to see, and I'm sure if
10:52:53you've looked at other people's
10:52:54visualizations, you'll see something
10:52:55like this. This is usually totals or
10:52:58averages or something like that in here
10:53:00where it's super easy to look at. So,
10:53:02like right here, this is total
10:53:04purchased, and we can go in and look at
10:53:06the minimum. And then we can go over
10:53:08here and this one can be account. And so
10:53:11it gives us a lot of information just at
10:53:13a really quick glance. And then we have
10:53:15all of our more in-depth colorful
10:53:17visualizations that kind of have more
10:53:19information than just a single piece
10:53:21like the card does. And then the very
10:53:22last one that I'm going to show you is
10:53:24this one right here, which is the table.
10:53:26And this one is obviously extremely
10:53:28popular. It's like an little Excel
10:53:30table. And we can go in here and we can
10:53:32get the customer wherever that is. And
10:53:36then we'll also get the units sold. And
10:53:38this is what it looks like. And it's
10:53:40super easy. And oftentimes you'll have
10:53:42it like on the side as well. Uh and all
10:53:44the other visualizations over here. And
10:53:46so, you know, if we're going to take all
10:53:48these visualizations and pretend they
10:53:49were like a real thing. You know,
10:53:52there's a lot in here, but we'll just
10:53:54kind of really quickly do this. Um, you
10:53:57know, we [snorts] might have something
10:53:58like this. And we'll make this larger
10:54:01and make this wider.
10:54:04And you know, we have a lot of
10:54:06information just in here. And this is
10:54:07not a project, so don't go put this on
10:54:09your portfolio. I'm just threw a ton of
10:54:12random visualizations on, you know, this
10:54:14dashboard. But you can already see a lot
10:54:17of these you most likely have seen in
10:54:19other people's work and other people's
10:54:20visualizations on LinkedIn or on
10:54:22YouTube. These are very common, very,
10:54:25very popular. And again, we did not go
10:54:27through all of the ones over here. There
10:54:29are maps that you can use, but I haven't
10:54:31used maps ever in my job. There are
10:54:34things like gauges and decomposition
10:54:37trees and waterfall charts and uh tree
10:54:40maps and all these different things, but
10:54:43I really have never used those in my
10:54:45actual job. And I don't see them a lot
10:54:48in others people's work either.
10:54:49Otherwise, I would be telling you to
10:54:51learn these and use these. But again,
10:54:53try them out. See which ones you like.
10:54:55If you like this video, be sure to like
10:54:56and subscribe below and go check out all
10:54:58the other PowerBI tutorial videos that I
10:55:00have on my channel. and I will see you
10:55:02in the next video.
10:55:07[music]
10:55:15What's going on everybody? Welcome back
10:55:16to the PowerBI tutorial series. Today we
10:55:19are going to be working on our final
10:55:20project.
10:55:27Now this is our final project of the
10:55:28PowerBI tutorial series. So, if you have
10:55:30not watched all of those videos leading
10:55:32up to this, I recommend going and
10:55:34watching those videos so you can make
10:55:35sure that you know all the things we're
10:55:37going to be looking at in today's
10:55:38project. I am really excited to work on
10:55:40this project with you because I think it
10:55:41is a really good one and it uses real
10:55:43data that we collected about a month ago
10:55:45where I took a survey of data
10:55:47professionals and this is the raw data
10:55:49that we're going to be looking at and so
10:55:51I think it's just really interesting
10:55:52that we collected our own data now we're
10:55:54using it for a project. We're going to
10:55:55transform the data using Power Query and
10:55:57then we'll actually create the
10:55:58visualizations and finalize the
10:56:00dashboards as well as create a theme and
10:56:02a different color scheme to kind of make
10:56:04it a little bit more unique. Without
10:56:05further ado, let's jump on my screen and
10:56:07get started with the project. All right,
10:56:08so before we jump into it, I wanted to
10:56:10let you know that you can get the data
10:56:11below. It is on my GitHub. You can go
10:56:13and download this exact file that we're
10:56:15going to be looking at. Now, in the past
10:56:17several projects, we have been using
10:56:20this fake apocalypse data set. You know,
10:56:22it was fun. it was, you know, whatever.
10:56:25This data set is real. This is a real
10:56:27data set. It was a survey that I took
10:56:28from data professionals. I posted on
10:56:30LinkedIn and Twitter and all these other
10:56:32places. And we had about 600 700 people
10:56:34who responded to the questions. So
10:56:36before we actually get into it and start
10:56:38cleaning the data and doing all this
10:56:41stuff in PowerBI, I just wanted to show
10:56:43you the data. All right. So this is the
10:56:45CSV that I downloaded from the survey
10:56:47website that I used. And this is
10:56:49completely raw data. I haven't done
10:56:51anything to it at all. But let's go
10:56:53through the data really quickly and
10:56:54we'll kind of see what we have. And we
10:56:56are not going to make any changes at all
10:56:58in Excel. We're going to do all of our
10:57:00transformations or at least a few
10:57:02transformations in PowerBI because again
10:57:05this is a PowerBI tutorial and project.
10:57:07So I want you to kind of learn how to
10:57:09use that and not use Excel because you
10:57:11can go through my Excel tutorial if you
10:57:13want to do that. So let's just look at
10:57:15it in Excel and then we'll move it over
10:57:16to PowerBI and actually start
10:57:18transforming the data. So we have this
10:57:20unique ID. These are all the people that
10:57:22actually took it. Oops. Don't want to do
10:57:24that. We have an email, which this is
10:57:26completely anonymous. I didn't collect
10:57:27any data or user data on this. Then we
10:57:31have the date taken. Um, and let's get
10:57:32into the actual good information. Then
10:57:35we have all of these questions. So, we
10:57:37have question one, which title fits you
10:57:39best? And they can choose things. Now,
10:57:42uh, let's add a filter really quickly
10:57:44that we can look at this. Now you had
10:57:47the pre-selected ones which were like
10:57:49data analyst, architect, engineer, but
10:57:51then there was an option where you could
10:57:52say other and you could specify what
10:57:54that was. So if you look in here, we're
10:57:57going to have all these different other
10:58:00please specify with different titles,
10:58:02right? And there were a lot of them. Now
10:58:07typically what you want to do is really
10:58:09clean this up. And we're not going to be
10:58:11doing a ton ton ton of data cleaning,
10:58:14but we are going to do some in PowerBI,
10:58:16but none in here. But typically with
10:58:18this amount of data in the way that it's
10:58:20formatted, we would do so much data
10:58:22cleaning um with this one. I mean,
10:58:24there's a lot of work to be done. Um
10:58:27like this current year salary, this is
10:58:29one that I would absolutely be cleaning
10:58:32up because it's a ranges and it has a
10:58:34dash and a k and all these numbers. This
10:58:37is something that I would be cleaning up
10:58:38and using, but we're not going to be
10:58:40cleaning this up right now. So, anyways,
10:58:43let's just get into it. Let's see what
10:58:44questions we asked. Uh, we have the
10:58:46yearly salary. What industry do you work
10:58:48in? Favorite programming language?
10:58:51Then there were a lot of different
10:58:53options. So, this was like one question
10:58:55where they picked multiple options. So,
10:58:57is how happy are you in your current
10:58:59position with the following? You have
10:59:00your salary, work life balance.
10:59:04Um, then we have co-workers, management,
10:59:07upward mobility, learning new things.
10:59:10Um, and they could rank it from 0 to 10.
10:59:12So, some people ranked upward mobility a
10:59:1410, some ranked it a zero or a one. Um,
10:59:18and again, they can answer however they
10:59:20want. How difficult was it to break into
10:59:23data? Very difficult, very easy. Um, if
10:59:27you're looking for a new job, we have,
10:59:29you know, what would you be looking for?
10:59:30Remote work, better salary, etc. We have
10:59:33fe male, female, which country are you
10:59:35from? And then this is more like
10:59:36demographics. So, if you're a male, how
10:59:38old you are, and this was in a range.
10:59:41So, this is like a a a sliding bar. So,
10:59:43you can slide it to the exact age you
10:59:45had. There's some people who are
10:59:48apparently 92. Um, which if that's true,
10:59:50I mean, good for you, man. Or
10:59:52[clears throat] woman. Actually, really
10:59:54quickly, I'm going to see just just
10:59:56while we're here, I'm going to see if
10:59:57this is a male male or a female. Oh,
10:59:59it's a female from India. Very cool. Um,
11:00:02so we have all this information and it
11:00:05is a lot of information when you have
11:00:07something like this. I mean, there is so
11:00:10much data cleaning that can be done. I
11:00:12mean, I already see like 20 plus
11:00:17different things that I would need to do
11:00:19to make this a lot better. Um, and we
11:00:21also have date taken and the time taken
11:00:24as well as how long it they took on it,
11:00:26like the time spent. Really just really
11:00:29interesting data. But again, this is a
11:00:32beginner tutorial series. This is the
11:00:34beginner project. So, we're not going to
11:00:36get do anything too crazy. I will be
11:00:39using this exact data set in a future
11:00:41video doing a lot more data cleaning and
11:00:45creating a much more advanced
11:00:46visualization with what we have and what
11:00:48we're looking at right here. But for
11:00:50this video, we're just going to be doing
11:00:51a pretty simple visualization and
11:00:54dashboard that you can use uh to
11:00:56practice with or put on your portfolio
11:00:58if you know that's where you're at right
11:00:59now. So, let's get out of here and let's
11:01:02put this into PowerBI. So, let's exit
11:01:04out and let's come right over here to
11:01:06import data from Excel. We'll click on
11:01:08PowerBI final project and open.
11:01:12Give that a second. Doing this all in
11:01:14real time. and we only have the one. So,
11:01:16we'll do we won't be practicing any
11:01:18joins or anything, but we're not going
11:01:20to load it. We're going to transform
11:01:21this data. So, let's put it into Power
11:01:24Query editor.
11:01:27And now we have all of our data in here.
11:01:29And it should look extremely familiar.
11:01:33Now, when I'm looking at this, when I
11:01:35start looking at this information, I
11:01:38kind of need to know beforehand what I
11:01:41want to get out of this. Do I need to
11:01:43clean every single column? Do I just
11:01:45need to clean a few of them? Do I need
11:01:46to get rid of columns? That's kind of
11:01:48where my head's at. And so, right off
11:01:50the bat, I can already tell you that
11:01:52there are columns that we can just
11:01:53delete to get out of our way. So, we're
11:01:55going to do that at the beginning so
11:01:57that we don't have to do that later on
11:01:59or they're just in our way. So, I'm
11:02:00going to click on browser and then I'm
11:02:02going to hit shift and I'm going to go
11:02:04over here to refer.
11:02:06I'm just going to go up here to remove
11:02:07columns. And everything that we do is
11:02:10going to go over here to this applied
11:02:11steps. If you've been following this
11:02:13series, um you know, we can remove
11:02:15things, add things, but anything we do
11:02:18will show up right over here. So, we can
11:02:20track it and go back if we need to. Now,
11:02:23one column that I know for sure that I'm
11:02:25going to be using quite a bit is this
11:02:27which title fits you best in your
11:02:28current role because I I specifically
11:02:30wanted to do a breakdown of diff
11:02:32people's roles and how much they make
11:02:34and different stuff like that. So I know
11:02:36that I want to use this but as we saw
11:02:38before
11:02:40there's kind of the issue is is it's not
11:02:41very clean right it has data analyst
11:02:44data architect engineer scientist
11:02:46database developer and then like a
11:02:49hundred different options and then a
11:02:52student or or none of these right um
11:02:57and so for the purpose of this video
11:03:00right here we are not going to take
11:03:02every single one of these options
11:03:04because this involves a lot more data
11:03:06cleaning. Let me give you an example.
11:03:07This says software engineer. This also
11:03:10says software engineer. The and with AI.
11:03:14These two would typically be combined or
11:03:17standardized to software engineer. But
11:03:20it's not very easy to do that in
11:03:22PowerBI. We could do that in Excel, but
11:03:24not really in PowerBI or even SQL if we
11:03:27pull this from a SQL database. Um, and
11:03:29you can find lots of different, you
11:03:31know, options of that. We have data
11:03:33manager and data manager. If we
11:03:34separated these out, these would be
11:03:37different options when we created our
11:03:39visualizations, and we don't want that.
11:03:40So, what we are going to do, uh, and
11:03:43this is going to be kind of a an easy
11:03:45way out to just make sure that this is
11:03:48pretty clean and doesn't we don't have a
11:03:49thousand different options. We're going
11:03:51to create this to other. So, we're going
11:03:53to simplify this a lot. And then we're
11:03:57going to use this. So we'll have maybe
11:03:59six or seven options instead of the, you
11:04:01know, let's say 50 that we would have if
11:04:03we actually did the harder work, which
11:04:06is break it out, standardize it, and
11:04:08clean it up that way. So what we're
11:04:10going to do is we're going to click on
11:04:11this right here. We're going to go up
11:04:13here to split column in this ribbon up
11:04:15top. We'll go to split column, and we
11:04:18want to do it by a delimiter. And if you
11:04:21notice, let me see if I can move this
11:04:23over. If you notice, we have other and
11:04:25then we have this parenthesis. and in no
11:04:27other option or way is there
11:04:29parenthesis. So what we're going to do
11:04:31is we're going to use a custom and we're
11:04:34going to use this open parenthesis. What
11:04:37that's going to do is it's going to
11:04:38separate it by this parenthesis. It's
11:04:39going to leave the other. It's going to
11:04:41create separate columns just one
11:04:44separate column for each of these. And
11:04:46we can do that at each occurrence or we
11:04:48can do the leftmost. And we really we
11:04:50only need it for the leftmost cuz
11:04:51there's only one of these uh left-handed
11:04:54or left-sided uh brackets or or what is
11:04:58it whatever this is called. And then
11:05:00let's go and click okay. And it should
11:05:02create another column. So it's going to
11:05:04have 0.1
11:05:062. And now we have if we click on this
11:05:10now we only have these options. We have
11:05:12analyst architect engineer data
11:05:14scientist database developer other and
11:05:16student looking or none. That is what we
11:05:19want. It makes it so much simpler and
11:05:21it's not perfect, but again, I'm trying
11:05:24to show you what we are able to do in
11:05:26PowerBI. So now we're just going to
11:05:28remove that column and we're going to go
11:05:30and do the exact same thing to this one
11:05:32as well because I know that we want to
11:05:34use this. And I really wanted to use
11:05:37this one as well. But if we look at this
11:05:39one also, um there's a lot. So I said,
11:05:42"What is your favorite programming
11:05:44language?" and people there were
11:05:45pre-selected answers like JavaScript,
11:05:47Java, C++, Python, R things like that
11:05:51and then there was an other option and
11:05:53in this other option I mean it was free
11:05:55text so they can fill it in as they
11:05:57want. I mean there's four, five, six
11:05:59[snorts] different ways that people put
11:06:01SQL that is something I would
11:06:02standardize and you know that would be
11:06:06the way I cleaned it but that's not how
11:06:08we did it in here. So we're going to do
11:06:09the same thing. We're going to keep that
11:06:10other. So we're going to split this
11:06:12column again. We're use a delimiter. And
11:06:15for this delimiter though, we're going
11:06:17to use a colon. So we're going to say
11:06:20we're going to do a colon right there.
11:06:21We'll just do the leftmost. We'll click
11:06:24okay. And then we have our options. And
11:06:28it's much simpler. Now, I really would
11:06:30have rather kept all these and because
11:06:32SQL's in there quite a bit, but you
11:06:34know, a lot of people don't think SQL is
11:06:36even a programming language. So, uh,
11:06:38we're going to delete that column. Now,
11:06:40one that I just skipped and I kind of
11:06:41wanted to go back to is this current
11:06:44yearly salary. I really want to use
11:06:47this. Let's see if we can use it. I
11:06:50Here's what I want to do with it. And
11:06:51this is not perfect. Um, but for this
11:06:53video, I want to try it. What I want to
11:06:55do is break up these numbers 106 125 and
11:06:59then take the average of those numbers.
11:07:01Then we'll use some docs in there. So,
11:07:03we'll take 106 125 create that into two
11:07:06separate columns. Then we'll create a
11:07:08third column that will give us the
11:07:10average of those two numbers. So we'll
11:07:11do 106 + 125 / 2 and then we'll have the
11:07:16average of that. Now that is not perfect
11:07:19but it's going to give us at least you
11:07:21know an average a kind of roundabout
11:07:23number because they gave us this range.
11:07:25They said my salary is between 106
11:07:27125,000. So if we say that their salary
11:07:29was 112,000
11:07:31at least gives us it makes it usable.
11:07:33It's a numeric value instead of being
11:07:35this which is text which we really we
11:07:38could use and and I'll show you how to
11:07:40do that because we're going to keep this
11:07:41column. I'll create a copy of this and
11:07:43I'll show you the difference between
11:07:44this and using the average but for but
11:07:49for this data cleaning portion let's
11:07:51just try it. Let's see what we can do
11:07:53and see if we can make it work. So first
11:07:56let's create a duplicate. So we're going
11:07:59to uh duplicate the column. So now we
11:08:03have this copy at the very very end and
11:08:06we can use this one instead of having to
11:08:08use the original way way way back here.
11:08:11So we're going to leave that one how it
11:08:12is and we're going to use this one. So
11:08:16let's go ahead and split this one up.
11:08:18We're going to click on the column
11:08:19header. Then we're going to click on
11:08:21split column. And we'll do it by digit
11:08:23to non-digit.
11:08:26And if you look at it right here, it's
11:08:29broken it out kind of um in the fact
11:08:31that now in this one we just have
11:08:34numeric values. And in this one we have
11:08:37K dash numeric or just dash numeric. And
11:08:42now this can be easily cleaned. Whereas
11:08:44this one we can just completely get rid
11:08:46of because it's only K. So we'll just
11:08:48remove that column. And then in this one
11:08:51we're going to rightclick. We're going
11:08:53to click on replace values. And so if it
11:08:56just has we just do a K. We'll replace
11:08:59with nothing. Do okay. And then for the
11:09:02last one, we'll go to replace values and
11:09:06we'll do the dash or the minus sign and
11:09:08we'll place that with nothing. And so
11:09:10now we have our values as well. Oh, we
11:09:13also have a plus. Let me get rid of that
11:09:14because that's when some people had 250
11:09:17or 225,000 plus. So for that one, the
11:09:20average is just going to be 225. We'll
11:09:22have to specify that in our DAX. I
11:09:24forgot. But actually, if somebody has
11:09:26225, let me find this plus really quick.
11:09:30Uh, let me filter by it because it's a
11:09:33lot faster. What we actually want to do
11:09:36for the purpose of this one is we want
11:09:37to put 225 here so that when we do 225 +
11:09:41225 / 2, it comes out to 225. That's
11:09:44just what we're going to put it as. And
11:09:46there's only two people. So, uh, I'm
11:09:48actually going to replace this. I'm
11:09:50going to do replace values. So I'm going
11:09:51to say plus
11:09:53with 225
11:09:55and we'll click okay. Awesome. We can
11:09:58unfilter these. Select all. So we're
11:10:02going to go right up here to add column
11:10:04and we're going to say custom column.
11:10:07And we're going to go right over here.
11:10:09Actually, let's make it uh average
11:10:13salary.
11:10:14Let's make it average salary. So we're
11:10:17going to insert this.
11:10:19We're going to say
11:10:22parentheses and we're going to say plus
11:10:26this insert
11:10:28and close the parenthesis divided by
11:10:31two. And it says no syntax errors have
11:10:34been detected. Let's click on okay. And
11:10:38it's giving us an error. So it's saying
11:10:40we cannot apply operator plus to types
11:10:42text and text which makes uh perfect
11:10:45sense. These aren't uh numbers. So,
11:10:46let's make it a whole number. And let's
11:10:48make it a whole number. And then let's
11:10:52see if this will actually work now.
11:10:56Or maybe just need to try a whole
11:10:58another one. So, let's try transform or
11:11:01add column. Custom column.
11:11:04Let's try this all again. See if uh I
11:11:06can make it work.
11:11:08Insert
11:11:10this one
11:11:12plus
11:11:14this one.
11:11:16and we'll do divided by two. And let's
11:11:19try this one. And there we go. So now
11:11:22let's get rid of this column.
11:11:24Columns. And we can actually remove
11:11:27these ones as well
11:11:29because now we have this um
11:11:33average salary column
11:11:37which [clears throat] when we look at
11:11:39this or when we use this uh we can let
11:11:41me see if I can just move this way way
11:11:43way over. All right. I might cut because
11:11:45this is taking forever. So if you take
11:11:47the average of these two numbers, you'll
11:11:49get 53. If you take the average of 0 and
11:11:5140, you'll get 20. So now we have this
11:11:53average salary. And again, when we get
11:11:56to the actual visualization part, I'll
11:11:58show you why this isn't as useful as
11:12:00having this average salary. And just a
11:12:02reminder, this is not perfect. Uh I
11:12:05wouldn't typically do this, especially
11:12:06if I had it in Excel or if I was, you
11:12:09know, creating this survey in a
11:12:11different way. I would probably have a
11:12:13very specific value where they can do it
11:12:14on a slider, but this is how it is. So,
11:12:17we've at least made it usable or more
11:12:19usable in my mind. And we have a few
11:12:22other things that we can change like
11:12:23what industry do you work in where we
11:12:25can break this one out. So, I'm going to
11:12:27go ahead and break this one out as well
11:12:29as
11:12:31this one right here. Which country do
11:12:32you live in? I'm going to break both of
11:12:34those out to where it's the country or
11:12:36other. I'm not going to have these other
11:12:38values, although there are a lot of them
11:12:40because there's a lot of people who live
11:12:41in these different countries, but we
11:12:44can't really do that super well in here
11:12:46because again, the same issue kept
11:12:48happening. Argentina, Argentina,
11:12:50Argentine, a Australia. So, we can't
11:12:53normalize those values unless we spend
11:12:55just copious amount of time doing that.
11:12:58So, I'm going to go ahead and do these.
11:13:00I'm going to fast I'm going to fast
11:13:02speed this so it goes a lot faster. So,
11:13:04I'm just going to go silent and let this
11:13:06happen really quick. And then we'll get
11:13:08to the end and we'll actually start
11:13:09building our visualizations.
11:13:19All right. So, we've split them up and
11:13:22as you can see we have all these options
11:13:23as well as other and I think you know
11:13:27there is let me tell you there is so
11:13:29much more that we could do with this. I
11:13:31mean, just so many other things, but
11:13:35this is like what the bare minimum of
11:13:37what we need for this project. So, let's
11:13:40go ahead and close and apply this. And
11:13:43if we need to come back at any point and
11:13:45actually fix anything or change
11:13:47anything, we can. So, it's not like
11:13:49that's permanent. Um, so as you can see,
11:13:50we have everything over here. We have
11:13:52all of our data as it is transformed in
11:13:55here as well. And now we can start
11:13:59building out our visualization. So,
11:14:01let's go back to our report
11:14:03and let's start building something out.
11:14:05All right. So, let's add a title to our
11:14:07dashboard.
11:14:10Make this right at the top.
11:14:13Call this the data
11:14:16professional
11:14:18survey
11:14:19breakdown.
11:14:22And let's make that quite a bit larger.
11:14:26Make it bold. Why not? And we'll put
11:14:29that in the center. And now let's um
11:14:33let's add some effects. Let's change
11:14:35that background to something like that's
11:14:38too dark. Something like this. And I do
11:14:41not like that bold. Let's take that off.
11:14:44There we go. So something like this just
11:14:46as a quick title to what we're about to
11:14:49do, what we are about to build. So we're
11:14:51going to start off with the most simple
11:14:53visualizations that we're going to do
11:14:54and we'll kind of work our way towards
11:14:56kind of the harder ones. So, the first
11:14:58one that we're going to start off with
11:14:59is a card. And the cards are obviously
11:15:02like just super super easy. They usually
11:15:05just display one piece of information.
11:15:08So, we're going to go right over here to
11:15:09the very bottom at the unique ID and
11:15:12we're going to select it and we're going
11:15:15to say a count of distinct or a count,
11:15:18it doesn't matter. Um, and it says 630
11:15:21count of unique ID. Now, we're not going
11:15:23to keep that as is. We're actually going
11:15:24to go right over here. We're going to
11:15:26say rename for this visual. And it says
11:15:28count of unique ID, but we're going to
11:15:30say count of
11:15:32survey takers. And you can say whatever
11:15:36you want here, but in in general, that
11:15:38is what it is. We're we're counting how
11:15:40many people um you know took this
11:15:43survey. And that's just a kind of a
11:15:45total maybe I say total amount or of
11:15:48survey takers, but you can say count of
11:15:50survey takers. How many people took the
11:15:52survey? So, let's click out of there.
11:15:54Let's click on card. Let's make it about
11:15:57the same size. We're going to drag it up
11:15:59here and try to make them about the
11:16:02same. We will in a little bit. We'll
11:16:04make them the same size. Um, but for
11:16:06this one, we're going to look at age.
11:16:08So, we're going to look at current age.
11:16:10So, we click on that and we'll say want
11:16:13the average age. So, our average age
11:16:16taker is almost 30 years old. So, let's
11:16:18go right over here. We're going to say
11:16:20rename for this visual. We'll say
11:16:22average age of survey.
11:16:27This might be too long.
11:16:30Average age of survey taker. Again, name
11:16:32it whatever you'd like. So again, these
11:16:34are meant to be high-level numbers. So
11:16:36when somebody is looking at your
11:16:38dashboard, they can just really quickly
11:16:40glance at this and know exactly what it
11:16:42is instead of like some of these other
11:16:44visualizations that we're about to
11:16:45create. They don't really have to dig
11:16:47into it, look at the x-axis, the y-axis,
11:16:49the the different uh legend colors and
11:16:52whatnot. They can just see these high
11:16:54numbers and get a really quick glance of
11:16:56the data. Now, let's create our first
11:16:58visualization. And what we're going to
11:17:00do for that one is a clustered bar
11:17:02chart. So, let's go ahead and click on
11:17:04the clustered bar chart. We can create
11:17:06as small or as large as we'd like. And
11:17:09for this one, we're going to be looking
11:17:10at the job titles. Now remember we kind
11:17:13of changed the job titles or you know u
11:17:17transform those if you want to say that.
11:17:19So we're going to look at job titles and
11:17:21then we're going to look at their
11:17:22average salary and if you remember we
11:17:25transformed that one as well. We have
11:17:27all average salary. Now this one is it
11:17:30looks like a text right now so it may
11:17:32not work properly. And what we're
11:17:33actually going to do is go over here.
11:17:36I want to see the average salary.
11:17:41So, let's click on average salary and
11:17:42see if we can change this data type from
11:17:44a text to a decimal number. Let's click
11:17:48yes. I forgot to do that when we were
11:17:50transforming it. And there we go. This
11:17:52is perfect. Um, so now we can go back
11:17:55and we can select our average salary.
11:17:59And as you can see, it has this um this
11:18:01function symbol. And so now we can click
11:18:03on it and it'll look a lot better. And
11:18:06although this says average salary as the
11:18:07title, it's actually doing a count or
11:18:09the sum. So we can click average right
11:18:12here. And what we want to do is actually
11:18:15break this down by the job title. And so
11:18:19now we can see data scientists are
11:18:21making the most by far. They're making
11:18:23average of 93,000 at least from the
11:18:26survey takers that took it. Then we have
11:18:28our data engineers making 65,000.
11:18:31Data architects are making 63. And then
11:18:34where are the data analysts? Data
11:18:36analysts are right here making 55. So
11:18:38again, we had 630 people take this
11:18:41survey. And so the vast majority of them
11:18:44were data analyst. So this one's
11:18:46probably the most accurate out of all of
11:18:47them. And I actually don't like how this
11:18:50looks as the clustered bar chart. Let's
11:18:52try the stacked bar chart and put this
11:18:55as the legend. That's more what I was
11:18:57going for. I don't know. I didn't want
11:19:00as skinny because when you're doing this
11:19:01one, it typically they have multiple
11:19:03options per um
11:19:06x-axis. And so I think that's why it was
11:19:08that little skinny line. But this one is
11:19:10more what I was looking for. But let's
11:19:12make that smaller. And let's definitely
11:19:14change that title cuz good night. Um
11:19:17this is like incredibly long. So let's
11:19:19go over here to this format visual.
11:19:23We'll go to the general the title and
11:19:27we're just going to say average
11:19:30salary by job title. Just like that. And
11:19:36this looks a lot better. Now, we're not
11:19:38going to kind of format our whole
11:19:41dashboard yet. We're going to create our
11:19:42visualizations and then we're going to
11:19:44kind of organize everything and kind of
11:19:46play Tetris with it to make it look the
11:19:48best. So, we're just going to minimize
11:19:51this and put it right up here for now.
11:19:55Um, but we will go back and kind of make
11:19:57everything look better at the end. And
11:19:59actually, while we're here, I also want
11:20:01to change this as well. So, rename for
11:20:05this, we're going to say job title.
11:20:08Oops. Why did I do that? Job
11:20:13title. And for this one, we're just
11:20:16going to say
11:20:19average salary.
11:20:22There we go. Looks much better, much
11:20:24cleaner. Uh, took away a lot of the
11:20:27anxiety that I was feeling about 2
11:20:29minutes ago when we first put that up
11:20:30there. So, let's go on to our second
11:20:32visualization. The next one that I'm
11:20:34interested in is actually what
11:20:36programming language people were using
11:20:38the most. So, we have salary. There's a
11:20:40thousand different things we can look at
11:20:41in here, but I want to know, you know,
11:20:43what is people's favorite programming
11:20:45language? So, let's take a look at that.
11:20:47So, we have favorite programming
11:20:50language. Let's find that. So, we have
11:20:51our favorite programming language and we
11:20:53also have how many people actually took
11:20:56it or the unique people. So, right now,
11:20:58this is columns. We don't want that.
11:21:01Let's um let's do a clustered column
11:21:03chart. Click on this right here. And it
11:21:07looks like
11:21:09here we go. That is kind of what we're
11:21:12looking for. And instead of count of
11:21:13unique ID, we'll say count of
11:21:17let's do count of voters.
11:21:21And for favorite programming language,
11:21:22we'll say
11:21:25favorite oops favorite programming
11:21:28language and get rid of that as well.
11:21:31And then we're going to go into here
11:21:32also and change the title and say
11:21:37favorite programming
11:21:40languages
11:21:41or favorite pro programming language
11:21:44just like this. Now let's make this a
11:21:46lot bigger so you can see it. But really
11:21:48[clears throat] quickly at a glance you
11:21:50can see Python is by far the most
11:21:52popular are other C++ JavaScript Java.
11:21:54Now all we're seeing is the count. So
11:21:56it's all the same. It's just blue. We
11:21:58can see how many people voted for each
11:22:00one. But if we wanted to break it out
11:22:01similar to how we did with the job
11:22:03titles, we could still do that. So all
11:22:06we'd have to do is break it out uh or
11:22:07bring this job title down to the legend.
11:22:10And now it breaks it out like this. And
11:22:12that's not exactly what I was going for.
11:22:14I was going more for something like this
11:22:17where we can see the still the whole
11:22:18count. But now we can see who is
11:22:21actually voting for these things. So,
11:22:23I'm just not a huge fan of the colors
11:22:24that are pre-selected here and kind of
11:22:27the whole theme of this dashboard. At
11:22:29the very end, we're going to completely
11:22:32revamp this, change a bunch of colors,
11:22:34the background, and make this look a lot
11:22:36nicer rather than just the white
11:22:37background like we have it. Um, and so
11:22:40for now, let's just
11:22:43make this a lot smaller and put it into
11:22:46this corner. These will not be staying
11:22:48there, but we need to we need room to
11:22:50create our next visualizations. and just
11:22:52a cleaner space to do things. Now, the
11:22:54next thing that I really want to include
11:22:56is a way to break down where they're
11:22:58from, their country, because especially
11:23:00something like salary is very dependent
11:23:02on your country. Whereas the average
11:23:04salary in the United States for a data
11:23:05analyst may be like 60,000. In another
11:23:09country, it could be 20,000. That could
11:23:11bring down the average quite a bit. So,
11:23:13we need a way to be able to break that
11:23:15down. Now, we can do something like a
11:23:18filled map, and there's no problem with
11:23:20that at all. Um, but, you know, for what
11:23:24we're building, what we're creating,
11:23:26it's not probably going to work out the
11:23:28best. I mean, this looks okay. We could
11:23:31stick it in the corner or something. Um,
11:23:33and you can do that and that's perfectly
11:23:34fine. I think what I'm going to do is
11:23:36something like a tree map, which I don't
11:23:40use a lot, but I want something where
11:23:42they can just click on it. They can look
11:23:44at the values
11:23:46distinct.
11:23:48They can look at the values and just
11:23:49click on it and it'll be right there for
11:23:51them. So they don't have to filter it
11:23:53out on their own or know geography and
11:23:55look at this map. They can just read
11:23:56Canada, other United Kingdom, India,
11:23:58United States and click on that. And so
11:24:00for example, let's click over here on
11:24:02United States. The numbers change quite
11:24:04a bit. Now the average salary for a data
11:24:06scientist is 139,000. For a data
11:24:09analyst, it's 80. And if we look at
11:24:11India, you know, the average salary for
11:24:14a data scientist is 68. The average
11:24:16salary is 26 for a data analyst. That
11:24:18doesn't mean that they make less money
11:24:20in India. That just means that the cost
11:24:22of living is probably lower in India.
11:24:24Therefore, they don't need the higher US
11:24:26dollars salary because again, this was
11:24:28all done in US dollars. So, just
11:24:30something to think about. Uh, let's
11:24:31click out of that. So, we'll keep that
11:24:33one as well. So, now let's create our
11:24:35next visualization. This is one that I
11:24:37do not get to use enough in my actual
11:24:39job. So, we're going to use it in this
11:24:40project. Um, and it's going to be this
11:24:42gauge right here. So, let's add that
11:24:44one. Put it right over here. We're going
11:24:46to add two of those. Let's just go ahead
11:24:49and add another one while we're at it
11:24:52because we're going to have them kind of
11:24:53like right here, right next to each
11:24:54other. The first one, and these ones are
11:24:56really good for kind of looking at these
11:24:58kind of surveys, and I don't get to work
11:25:00with surveys enough, but we can see, you
11:25:02know, how happy are they in terms of
11:25:04work life balance. So, we can add that.
11:25:07We're going to add work life balance.
11:25:08Um, and right now it's doing a count and
11:25:11we don't have minimum or maximum values
11:25:13in there yet. So, it's going to look
11:25:14kind of weird, but we're going to look
11:25:15at the average rate or the the average
11:25:18score of these. Then, we're going to
11:25:20pull this over to the minimum value. We
11:25:22want to put that at the minimum and pull
11:25:25this over and add the maximum value. So,
11:25:28now it actually has 0 to 10. And it
11:25:31shows that the average person is happy
11:25:34with uh which one was this? The average
11:25:36person is happy with their work life
11:25:37balance. Uh they rate about a 5.74
11:25:40overall. Now let's really quickly change
11:25:45the title of this because this is
11:25:46ridiculous. I want to say happy with
11:25:50work life balance. So this is their
11:25:53rating. Uh you know change it to
11:25:54whatever title you want. That's what I'm
11:25:56going to do. And we'll also do happy
11:25:58with their salary. So let's click on
11:26:01salary. We'll add that to minimum.
11:26:05And we'll add the maximum value as well
11:26:07to make sure that we know how to use
11:26:09that.
11:26:11And then we'll take [clears throat] the
11:26:12average. So not many people are happy
11:26:14with their salary. I'm just finding out.
11:26:15I mean this is a real survey. This is
11:26:17real data. So I mean it's uh pretty
11:26:19interesting. Let's go to the title.
11:26:22Let's go to happy with or maybe it's
11:26:25happiness. Happiness with salary. Maybe
11:26:29that's what we should make it. And I'm
11:26:31going to change that over here as well.
11:26:32I think it sounds better.
11:26:34Some of this I've already planned out,
11:26:36some I haven't. This is not something
11:26:37I've planned out. So, uh, so we're going
11:26:39to say happiness with work life balance,
11:26:41happiness with salary. Really
11:26:43interesting. Um, we may go back and
11:26:45tweak these just a little bit in the
11:26:46future, but the very last visualization
11:26:48that we're going to do is male versus
11:26:50female. Kind of got to have that in
11:26:52there. Um, I don't typically like pie
11:26:55charts and doughut charts, but uh, you
11:26:57know, I'm feeling I'm just feeling it.
11:26:59So, let's try it. Um, and we will do,
11:27:04let's see, let's make this larger. So,
11:27:06we have male, female,
11:27:08and what do we want to look at? Like,
11:27:10what do we want to measure? So, we have
11:27:11male versus female. We can measure
11:27:14anything. Um, but maybe what we'll do is
11:27:17the average salary. Again, I mean, we've
11:27:19kind of only looked at salary once in
11:27:22this one right here. um and a little bit
11:27:24of like how happy they are. But we'll
11:27:26look at the average salary between males
11:27:29and females. And then we'll look at not
11:27:33the current age. Oops, I meant average
11:27:36salary. And then we'll look at the
11:27:39average.
11:27:41And it looks like the average salary is
11:27:43actually really close versus males
11:27:45versus females. 55 for female versus 53
11:27:49for males. So actually the females are a
11:27:52little bit higher. Uh, congratulations.
11:27:53So, they're just a little bit higher in
11:27:56terms of pay. So, now we need to start
11:27:58organizing all of this, cleaning it up,
11:28:00making it look a lot better than it does
11:28:02right now. It looks great. Uh, you know,
11:28:06but we can do a lot more with this. So,
11:28:07I'm going to we're we're going to keep
11:28:09these or all these kind of over on this
11:28:11left-hand side. I'm going to put this I
11:28:14want this up here. We also need to
11:28:15change that title. I want this up here.
11:28:18Um, and again, we're going to kind of
11:28:19change the theme as we go.
11:28:23I just want to format it, right?
11:28:26I'll have it just like this. Let's
11:28:28change the title of this.
11:28:32Let's go to title and we're going to say
11:28:34country of survey takers.
11:28:38Uh I'm not the the survey takers. I'm
11:28:40not really stuck on that. If you find
11:28:42something better, you think of something
11:28:43better, I would go with that. But um you
11:28:47know, it definitely doesn't look bad.
11:28:48And where did this where did my other
11:28:49visualization go? There it goes. Um, I
11:28:52think this one I want to make kind of
11:28:54more tall. Um, so I might move it this
11:28:57way. Jeez, this is such a I hate I hate
11:29:00having a lot of visualizations on here.
11:29:01It just really uh is annoying to me. So,
11:29:04what we're going to do, we're going to
11:29:07step this to the side. Put this to the
11:29:09side as well.
11:29:12Make it to where it's just
11:29:16Okay. I didn't want it to cut off.
11:29:19We'll do that. Might make these
11:29:25these a little bigger actually. So, I
11:29:27want it to kind of match the size
11:29:31like right there. I'll match this.
11:29:34Perfect. This one I kind of want to
11:29:36bring over here and bring it down a
11:29:40little bit. Maybe something like this.
11:29:44Maybe. I'm not sure. I'm not I'm not
11:29:46sold on that. Um, I added a few
11:29:48different visualizations that I didn't
11:29:49have in my original. So, now I'm kind of
11:29:51having to do this on the fly. So, um, I
11:29:53might fast forward some of the parts
11:29:55where I'm like really thinking about it
11:29:56or taking too much time on it. But, I'm
11:29:58going to bring this down a little bit
11:30:00actually because I don't like how close
11:30:01that is to, um, the the text above it.
11:30:06But one thing we do need to do,
11:30:12I'm going to put this up kind of like
11:30:14this. I think that looks fine. I think
11:30:17I'm going to put this at the very
11:30:18bottom. So, let's make some room for it.
11:30:22Right. Just like that. Stretch it to the
11:30:24side. And we'll lower it.
11:30:28And I think we'll keep that as is.
11:30:32Kind of like this. Um, okay. There's a
11:30:35lot going on in here. And there are some
11:30:37things I'm just noticing as we're
11:30:39walking through this that I kind of
11:30:40missed. Um, like I need to change some
11:30:43titles and stuff like that. So, let me
11:30:44go ahead and change some of those
11:30:46things. So, we're going to do title.
11:30:49We're going to do average salary by
11:30:54gender or by sex.
11:30:58Do like that. Average salary by sex. I
11:31:00also don't like that it's in the middle.
11:31:03Um, I don't like that it's on the
11:31:06outside. I want them on the inside for
11:31:07this. So, let's go to the details. Let's
11:31:11go to inside and see if that looks any
11:31:13better. Oh, that looks terrible. Um, let
11:31:16me see if I can change that. Maybe I
11:31:18don't. No, I definitely want it. Um, I
11:31:23guess we'll do outside. I You can't even
11:31:25see the information. Oh, the decimal is
11:31:28crazy long. Um, let me go and see if I
11:31:30can change that decimal to just like a
11:31:32whole number or like 1.1.
11:31:35Uh, because that's a problem. So, maybe
11:31:37I need to go over here to the value.
11:31:42All right. All right. So, I think I want
11:31:43to change this one. It's just not
11:31:44working out exactly how I wanted. And
11:31:47you guys know if I make mistakes, I'm
11:31:48going to keep it in here so you guys can
11:31:50see it. I I hope that this was going to
11:31:52turn out better, but it didn't. Um, one
11:31:54that I do want to add because this is
11:31:56kind of a a breakdown and a nice
11:31:59visualization. I want to add this
11:32:00difficulty piece. So, I want to add
11:32:02this. How difficult was it for you to
11:32:04break into data science? So, let's get
11:32:06rid of these. And I want to click on
11:32:08this really quickly. See what it gives
11:32:09us. Um, bum values. Okay. So now this
11:32:15shows us percentages um of how easy it
11:32:18was. Again, it's neither easy nor
11:32:20difficult. Difficult, easy, very
11:32:22difficult, very easy. These numbers make
11:32:25absolutely no sense. We need to kind of
11:32:27order them a little better. So I'm going
11:32:29to come over here to slices. We have our
11:32:31colors over here. We want very difficult
11:32:34to be like the most difficult. Um so
11:32:38we're going to make that red.
11:32:41And then we want difficult to be maybe
11:32:43like an orange.
11:32:45Let's see if we can find an orange.
11:32:46There we have an orange. This does not
11:32:48look red enough. There we go. Oh, no,
11:32:52no, no. Very difficult is red. Difficult
11:32:54is orange. We have neither easy nor
11:32:57difficult. And that's kind of a neutral.
11:32:59Um, let's see if we have something
11:33:00neutral in here.
11:33:04Kind of like this yellow. I don't know.
11:33:06Let's try it out. Then we have easy and
11:33:09very easy. And these will be like our
11:33:11blues. So, I'm going to keep that um I'm
11:33:15going to keep that kind of like a dark
11:33:18blueish.
11:33:20And then our blue for super easy is just
11:33:23going to be like really blue. Um and
11:33:28that doesn't look bad. The I mean, look,
11:33:29I'm I'm not a color person. I I'm not
11:33:32great with colors, and we're going to
11:33:34kind of organize this in just a little
11:33:35bit, but this looks better to me. Um,
11:33:38but we need to change up some stuff as
11:33:40well, like the title. Need to do
11:33:44difficulty to break into data.
11:33:50There we go. And we're also going to
11:33:53change
11:33:54this title right here. We'll just say
11:33:57difficulty.
11:34:00Difficulty.
11:34:02This looks better to me. Um, again, not
11:34:05perfect and there's a thousand different
11:34:07things you could have done, but that's
11:34:08just what we're going to do. I need to
11:34:09go through here and see what I need to
11:34:11change. So, right off the bat, I can see
11:34:12I need to change this um
11:34:16to let's see right here. I'm going to
11:34:19rename this job title just like we did
11:34:23in this one right here. Uh, count of
11:34:26voters. That's fine. Programming
11:34:29language breaking into difficulty,
11:34:31happiness, happiness, average count.
11:34:33Okay. Okay. So, what we have here is
11:34:38very close to a finished product. Now,
11:34:41it's not 100% complete. I mean, I I do
11:34:44want to make it look a little nicer
11:34:46rather than just the typical white. So,
11:34:48what we're going to do, we're going to
11:34:50go up here. We'll go to uh what is it?
11:34:53View. And we have all these different
11:34:55filters. And we're just going to play
11:34:56around with it. See if we can find
11:34:58something that we like. Um,
11:35:01this doesn't look too bad. It's uh not
11:35:04really my style. Uh, we can do this one.
11:35:07Frontier. This is pretty neat. I kind of
11:35:10am digging this. We might come back to
11:35:12it. I like the natural tones. I don't
11:35:14know why I said tones like that, but I
11:35:16did. Um, this one's [clears throat] not
11:35:19bad, but I don't I don't It's not That's
11:35:22not my I don't like how dark that is.
11:35:24Um, and so maybe it's like, you know,
11:35:28uh, we change like the background color
11:35:30of all of these as well as match it with
11:35:33um, match it with something else.
11:35:36Whatever you want, genuinely, you
11:35:38customize this however you want. I kind
11:35:40of like this one. It's kind of groovy,
11:35:42man. And, um, it's [clears throat] not
11:35:44perfect by any means, but what we can
11:35:48do, and we can customize this current
11:35:49theme. We can come in here, customize
11:35:51this theme however we'd like. I
11:35:55personally [clears throat] don't want
11:35:56color five, which is the data analyst
11:35:58color. I don't like it to I don't want
11:36:01to go go and change it because I don't
11:36:03like it, but I don't really like that
11:36:04color per se. You know, I might want to
11:36:07choose a different color. Um, but it has
11:36:09to be like this muted like that. It has
11:36:11a style to it. So, you can come in here
11:36:14and you can customize this and make it
11:36:16however you'd like and and really mess
11:36:19around with it. Play around with it. For
11:36:21me, uh I'm just going to keep it how it
11:36:23is because I don't really want to mess
11:36:25with it and break it or anything like
11:36:26that. So, let me just up just a tiny
11:36:30bit. So, this is it. This is the
11:36:33project. I hope that it was helpful. Um
11:36:36I am not joking when I say that I'm
11:36:39because I'm going to do a different
11:36:40project. I'm gonna go really in depth in
11:36:42another project. It's probably gonna be
11:36:44like a two-hour project. It's gonna be
11:36:45crazy long. Um well, for a YouTube
11:36:47video, but I can see doing a thousand
11:36:51different things with this data,
11:36:52creating a really great dashboard,
11:36:55really cleaning the data, which is a
11:36:57large part of of actually doing this.
11:36:59And we didn't do much data cleaning at
11:37:01all. There's just so much you can do
11:37:02with this. And so, really dig into this.
11:37:04See what you like, see what you don't
11:37:06like, see what you want to clean, what
11:37:08you don't want to clean. You could put
11:37:09it in SQL. You could put it in um Excel
11:37:12and just and just standardize the data
11:37:15to make it a lot more usable. Do
11:37:17whatever you want with it. I mean, I I
11:37:19took this survey for you guys that we
11:37:20could use it. So, go out and use it and
11:37:24make the best dashboard that you can
11:37:26possibly do. So, I hope that this was
11:37:28helpful. I hope that you enjoyed this.
11:37:29Thank you so much for watching this
11:37:32video. If you like this Thank you so
11:37:35much for watching. If you like this
11:37:36video, be sure to like and subscribe
11:37:38below, and I'll see you in the next
11:37:39video.
11:37:41[music]
11:37:52What's going on everybody? Welcome back
11:37:53to another video. Today, we're going to
11:37:55be starting our Python tutorial series.
11:38:02Now, I am extremely excited for this
11:38:04series. We're going to be walking
11:38:06through all the things that you need to
11:38:07know to get started in Python. We'll be
11:38:09looking at variables, data types, for
11:38:11loops, y loops, operators, and a ton
11:38:14more. After this beginner series, we're
11:38:15going to be going into another set of
11:38:17series where we look at pandas, mapplot,
11:38:19lib, seabor, web scraping, and more.
11:38:21Now, in this video, we're just going to
11:38:23be setting up our environment to where
11:38:24we can learn Python in future videos. In
11:38:26this series, we're going to be using
11:38:27Jupyter Notebooks for all of our
11:38:28tutorials because I feel like it's a
11:38:30really great place to learn the basics.
11:38:32But then in future videos, I'll show you
11:38:33different IDEs that you can use for your
11:38:35Python code. I genuinely cannot wait to
11:38:37get started on this series. I absolutely
11:38:38love Python. So without further ado,
11:38:40let's jump on my screen. I'm going to
11:38:42show you how to install Jupyter
11:38:43Notebooks. All right, so let's get
11:38:44started by downloading Anaconda.
11:38:46Anaconda is an open- source distribution
11:38:48of Python and our products. So within
11:38:51Anaconda is our Jupyter notebooks as
11:38:53well as a lot of other things, but we're
11:38:54going to be using it for our Jupyter
11:38:56notebooks. So let's go right down here.
11:38:58And if I hit download, it's going to
11:38:59download for me because I'm on Windows.
11:39:02But if you want additional installers,
11:39:03if you're running on Mac or Linux, then
11:39:06you can get those all right here. Now,
11:39:08if you are running on Windows, just make
11:39:09sure to check your system to see if it's
11:39:11a 32-bit or a 64. You can go into your
11:39:14about in your system settings to find
11:39:16that information. I'm going to click on
11:39:18this 64-bit.
11:39:20It's going to pop up on my screen right
11:39:22here, and I'm going to click save.
11:39:25Now, it's going to start downloading it.
11:39:26It says it could take a little while,
11:39:28but honestly, it's going to take
11:39:29probably about 2 to 3 minutes, and then
11:39:31it will get going. Now that it's done,
11:39:33I'm just going to click on it, and it's
11:39:35going to pull up this window right here.
11:39:37We are just going to click next because
11:39:38we want to install it. This is our
11:39:40license agreement. You can read through
11:39:42this if you would like. I will not. I'm
11:39:44just going to click I agree. Now we can
11:39:47select our installation type. And you
11:39:49can either select it for just me or if
11:39:50you have multiple admin or users on one
11:39:53laptop, you can do that as well. For me,
11:39:56it's just me, so I'm going to use this
11:39:58one as it recommends. Now, it's going to
11:40:00show you where it's installing it on
11:40:01your computer. This is the actual file
11:40:04path. It's going to take about 3.5 gigs
11:40:07of space. I have plenty of space, but
11:40:09make sure you have enough space. And
11:40:10then once you do, you can come right
11:40:12over here to next. And now we can do
11:40:15some advanced options. We can add
11:40:17Anaconda 3 to my path environment
11:40:20variable. And when you're using Python,
11:40:22you typically have a default path with
11:40:25whatever Python IDE or notebook that
11:40:28you're using. I use a lot of Visual
11:40:30Studio Code. So if I do this, I'm
11:40:32worried it might mess something up. So I
11:40:34am not going to do this. It also says it
11:40:36doesn't recommend it. Again, messing
11:40:37with these paths is kind of something
11:40:39that you might want to do once you know
11:40:40more about Python. So, I don't really
11:40:42recommend you having this checked. We
11:40:44can also register Anaconda 3 as my
11:40:46default Python 3.9. You can do this one.
11:40:50And I'm going to keep it this way just
11:40:51so I have the exact same settings as you
11:40:53do. So, let's go ahead and click
11:40:54install. And now, it is going to
11:40:57actually install this on your computer.
11:40:59Now, once that's complete, we can hit
11:41:01next. And now, we're going to hit next
11:41:03again. And finally, we're going to hit
11:41:06finish. But if you want to, you can have
11:41:08this tutorial and this getting started
11:41:10with Anaconda. I don't want either of
11:41:13them cuz I don't need them. But if you
11:41:15would like to have those, keep those
11:41:16checked and you can get those. Let's
11:41:18click finish. Now, let's go down and
11:41:20we're going to search for Anaconda and
11:41:23it'll say Anaconda Navigator and we're
11:41:26going to click on that and it should
11:41:28open up for us. So, this is what you
11:41:30should be seeing on your screen. This is
11:41:31the Anaconda Navigator and this is where
11:41:34that distribution of Python and R is
11:41:37going to be. So we have a lot of
11:41:38different options in here and some of
11:41:40them may look familiar. We have things
11:41:42like Visual Studio Code, Spider, R
11:41:45Studio, and then right up here we have
11:41:47our Jupyter notebooks and this is what
11:41:50we're going to be using throughout our
11:41:51tutorials. So let's go ahead and click
11:41:53on launch. And this is what should kind
11:41:55of pop up on your screen. Now I've been
11:41:57using this a lot. Um, so I have a ton of
11:41:59notebooks and files in here. But if you
11:42:03are just now seeing this, it might be
11:42:04completely blank or just have some, you
11:42:07know, default folders in here. But this
11:42:09is where we're going to open up a new
11:42:11Jupyter notebook where we can write code
11:42:13and all the things that we're going to
11:42:14be learning in future tutorials. And you
11:42:16can use this area to save things and
11:42:19create folders and organize everything.
11:42:21If you already have some notebooks from
11:42:23previous projects or something, you can
11:42:25upload them here. But what we're going
11:42:26to do is go right to this new. We're
11:42:29going to click on the dropdown and we're
11:42:30going to open up a Python 3 kernel. And
11:42:33so we're going to open this up right
11:42:34here. Now, right here is where we're
11:42:36going to be spending 99% of our time in
11:42:39future videos. This is where we're going
11:42:41to write all of our code. So right here
11:42:43is a cell and this is where we can type
11:42:45things. So I can say print. I can do the
11:42:48famous hello world and then I'll run
11:42:51that by clicking shift enter. And this
11:42:53is where all of our code is going to go.
11:42:55These are called cells. So each one of
11:42:58these are a cell. And we have a ton of
11:42:59stuff up here. And I'm going to get to
11:43:01that in just a second. But one thing I
11:43:03wanted to show you is that you don't
11:43:04only have to write code here. You can
11:43:06also do something called markdown. And
11:43:08so markdown is its own kind of you could
11:43:10say language, but um it's just a
11:43:12different way of writing, especially
11:43:13within a notebook. So all we're going to
11:43:15do is do this little hashtag. And
11:43:18actually I think it's a pound sign, but
11:43:19I'm going to call it hashtag. We're
11:43:20going to do that. We're going to say
11:43:22first notebook. And then if I run that,
11:43:25we have our first notebook. And we can
11:43:26make little comments and little notes
11:43:27like that that don't actually run any
11:43:29code. They just kind of organize things
11:43:31for us. And I'm going to do that in a
11:43:33lot of our future videos. So, just
11:43:34wanted to show you how to do that. Now,
11:43:35let's look right up here. A lot of these
11:43:37things are pretty important. Uh, one of
11:43:40the first things that's really important
11:43:41is actually saving this. So, let's say
11:43:43we wanted to change the title to I'm
11:43:45going to do a aaa because I want it to
11:43:47be at the beginning. Um, so I can show
11:43:49you this. I'm going do a aaa new
11:43:51notebook and I'm going to rename it and
11:43:54then I'm going to save that. So if I go
11:43:56right back over here, you can see a aaa
11:43:59new notebook. That green means that it's
11:44:02currently running. And when I say
11:44:04running, I mean right up here. And if we
11:44:07wanted to, we go ahead and shut that
11:44:08down, which means it wouldn't run the
11:44:10code anymore. And then we'd have to run
11:44:12up a new cluster. Uh so let's go ahead
11:44:14and do that. I didn't plan on doing
11:44:15that, but let's do it. So we have no
11:44:17notebooks running. And right here it
11:44:19says we have a dead kernel. So this was
11:44:21our Python 3 kernel. And now since I
11:44:24stopped it, it's no longer processing
11:44:25anything. So let's go ahead and say try
11:44:27restarting now.
11:44:30And it says kernel is ready. So it's
11:44:32back up and running and we're good to
11:44:34go. The next thing is this button right
11:44:36here. Now this is an insert cell below.
11:44:38So if I have a lot of code I know I'm
11:44:40going to be writing, I can click a lot
11:44:42of that. And I often do that because I
11:44:44just don't like having to do that all
11:44:46the time. So I make a bunch of cells
11:44:48just so I can use them. You can also
11:44:50delete cells. So say we have some code
11:44:52here. We'll say here and we have code
11:44:56here. And then we have this empty cell
11:44:58right here. We can just get rid of that
11:44:59by doing this cut selected cells. We can
11:45:02also copy selected cells. So if I hit
11:45:04copy selected cells and I can go right
11:45:07here and say paste selected cells. And
11:45:10as you can see it pasted that exact same
11:45:12cell. You can also move this up and
11:45:14down. So, I can actually take this one
11:45:16and say I wanted it in this location. I
11:45:19can take this cell and move it up or I
11:45:21can move it down. And that's just an
11:45:23easy way to kind of organize it. Instead
11:45:25of having to like copy this and moving
11:45:27it right down here and pasting it, you
11:45:28can just take this cell and move it up,
11:45:30which is really nice. Now, earlier when
11:45:32I ran this code right here, I hit shift
11:45:35enter. You can also run and it'll run
11:45:37the cell below. So, you can hit run and
11:45:39it works properly. If you're running a
11:45:41script and it's taking forever and it's
11:45:43not working properly, at least it's you
11:45:45don't think it's working properly, you
11:45:47can stop that by doing this interrupt
11:45:49the kernel right here and anything
11:45:51you're trying to do within this kernel
11:45:52if it's just not working properly, it'll
11:45:54stop it. You can restart it. Then you
11:45:56can try fixing your code. You can also
11:45:58hit this button if you want to restart
11:45:59your kernel and this button if you want
11:46:01to restart the kernel and then rerun the
11:46:03entire notebook. As we talked about just
11:46:06a second ago, we have our code and our
11:46:08markdown code. We're not going to talk
11:46:10about either of these because we're not
11:46:11going to use that throughout the entire
11:46:13series. The next thing I want to show
11:46:14you is right up here. If you open this
11:46:17file, we can create a new notebook. We
11:46:19can open an existing notebook. We can
11:46:21copy it, save it, rename it, all that
11:46:23good stuff. We can also edit it. So, a
11:46:26lot of these things that we were talking
11:46:27about, you can cut the cells and copy
11:46:28the cells using these shortcuts if you
11:46:30would like to. We also go to view and
11:46:32you can toggle a lot of these things if
11:46:34you would like to, which just means
11:46:35it'll show it or not show it depending
11:46:37on what you want. So, if we toggle this
11:46:38toolbar, it'll take away the toolbar for
11:46:41us. Or if we go back and we toggle the
11:46:43toolbar, we can bring it back. We can
11:46:45also insert a few different things like
11:46:47inserting a cell above or a cell below.
11:46:49So, instead of saying this plus button,
11:46:51you can just say A or B, routing above
11:46:54or below. We also have the cell in which
11:46:56we can run our cells or run all of them
11:46:58or all above or all below. And then we
11:47:01have our kernels right here, which we
11:47:03were talking about earlier where we can
11:47:04interrupt it and restart those. There
11:47:06are widgets. We're not going to be
11:47:08looking at any widgets in this series,
11:47:10but if it's something you're interested
11:47:11in, you can definitely do that. Then we
11:47:13have help. So, if you are looking for
11:47:15some help on any of these things,
11:47:16especially some of these references,
11:47:17which are really nice, you can use
11:47:19those. And you can also edit your own
11:47:21keyboard shortcuts. And now that we
11:47:23walked through all of that, you now have
11:47:24Anaconda and Jupyter Notebooks installed
11:47:26on your computer in future videos. This
11:47:28is where we're going to be writing all
11:47:29of our Python code. So, be sure to check
11:47:31those out so we can learn Python
11:47:32together. Thank you guys so much for
11:47:33watching. I hope you were able to get
11:47:34everything installed correctly. I am
11:47:36super excited for this series ahead of
11:47:38us. If you like this video, be sure to
11:47:40like and subscribe below and I will see
11:47:41you in the next video.
11:47:44[music]
11:47:53[snorts]
11:47:54Hello everybody. Today we're going to be
11:47:56learning about variables in Python. A
11:47:58variable is basically just a container
11:48:00for storing data values. So you'll take
11:48:03a value like a number or a string and
11:48:05you can assign it to a variable and then
11:48:07the variable will carry and contain
11:48:10whatever you put into it. So for
11:48:12example, let's go right over here. We're
11:48:14going to say x and this is going to be
11:48:16our variable. We're going to say is
11:48:17equal to. Now we can assign the value to
11:48:20it. So let's say I want to put 22. X is
11:48:25now equal to 22. So we won't have to
11:48:28write out the number 22 in later scripts
11:48:30that we write. we can just say x because
11:48:32x is equal to 22. It now contains that
11:48:36number. So now we can hit enter and say
11:48:38print. We'll do an open parenthesis and
11:48:41we'll say x. Now I'm going to hit shift
11:48:43enter. And now it prints out that 22
11:48:46because we are printing x and x is equal
11:48:49to 22. This is our value and this is our
11:48:52variable. One really great thing about
11:48:54variables is that it assigns its own
11:48:56data type. It's going to automatically
11:48:58do this. So, we didn't have to go and
11:49:00tell X that it's an integer. It just
11:49:02automatically knew that 22 is a number.
11:49:04So, we can check that by saying type and
11:49:07then open parenthesis and writing X. And
11:49:10we'll do shift enter again. And this
11:49:13says that X is an integer type. Now, we
11:49:15only assigned an integer to X. Let's try
11:49:19assigning a string value or some text to
11:49:21a variable. So, we'll say Y is equal to
11:49:25uh let's say mint chocolate chip. I'm
11:49:28feeling some ice cream today. So, we'll
11:49:30say mint chocolate chip. Now, if we
11:49:33print that again, we'll do print open
11:49:36parenthesis y and do shift enter. It'll
11:49:39print mint chocolate chip. And if we
11:49:42look at the type, we can see that the
11:49:44type is a string this time and not an
11:49:47integer. Now, again, we did not tell it
11:49:49that x was an integer and y was a
11:49:51string. It just automatically knew this.
11:49:54Let's go over here really quickly. We're
11:49:56going to add several rows in here
11:49:57because we're about to write a lot of
11:50:00different variables and really learn
11:50:02in-depth how to use variables. The next
11:50:04thing to know about variables is that
11:50:05you can overwrite previous variables.
11:50:08Right now we have mint chocolate chip
11:50:10and that is assigned to the variable y.
11:50:12So if I go down here, I say print y, I
11:50:16hit shift enter, it's going to print out
11:50:18mint chocolate chip. But if I go right
11:50:20above it, I say y is equal to and let's
11:50:24say chocolate. If I print that out, it's
11:50:28now going to say chocolate. Whereas up
11:50:29here, I'm reassigning it to y, it's
11:50:32still going to say mint chocolate chip.
11:50:35So if I come right down here and I copy
11:50:39this, and I'm going to paste this right
11:50:41here. Initially, it is going to assign Y
11:50:43to chocolate. But then right here it
11:50:46will automatically overwrite Y as mint
11:50:48chocolate chip. And when we hit shift
11:50:50enter it's going to show mint chocolate
11:50:52chip. Variables are also case sensitive.
11:50:55So if I come up here and I say a capital
11:50:58Y, this is a lowercase Y and this is a
11:51:00capital Y. It is going to print out the
11:51:03correct one instead of mint chocolate
11:51:05chip. And then if I go down here to the
11:51:07print and I type the capital Y, it will
11:51:11give us the mint chocolate chip. Up till
11:51:13now, we've only assigned one value to
11:51:15one variable. But we can actually assign
11:51:18multiple values to multiple variables.
11:51:21So let's do x comma y comma z is equal
11:51:26to and now we can assign multiple values
11:51:29to all of those. So we can say chocolate
11:51:34and then we'll do a comma. Oops, a
11:51:37comma. Then we can say vanilla and then
11:51:41we'll do another comma and we'll say
11:51:44rocky road. Now this is going to assign
11:51:48chocolate to x, vanilla to y and rocky
11:51:51road to z. So what we can do is we'll
11:51:54say print and we'll go print print.
11:51:59We'll say x y and z. So it prints out
11:52:04chocolate, vanilla, and rocky road. And
11:52:06these are our three different values. We
11:52:09can also assign multiple variables to
11:52:11one value. And we can do this by saying
11:52:14x is equal to y is equal to z is equal
11:52:17to and we can put whatever we would
11:52:19like. Let's do root beer bloat. Then
11:52:23we'll come back up here. We'll copy this
11:52:27and let's print off our x, our y, and z.
11:52:30And they are all the exact same. Now, so
11:52:33far we've really only looked at integers
11:52:34and strings, but you can assign things
11:52:37like lists, dictionaries, tupils, and
11:52:39sets all to variables as well. So, let's
11:52:42go right down here. So, let's create our
11:52:44very first list. I'm going to say ice
11:52:47cream is equal to, and that is our
11:52:49variable right there. The ice cream is
11:52:51our variable. So, now we're going to do
11:52:53an open bracket like this. And we're
11:52:56going to come up here and copy all of
11:52:58these values and we're going to stick it
11:53:00within our list. So now within ice cream
11:53:04we have three string values chocolate
11:53:06vanilla and rocky road all within this
11:53:09list. So what we can do is we can say x
11:53:13comma y comma z is equal to ice cream.
11:53:19So now these three values chocolate
11:53:21vanilla and rocky road will be assigned
11:53:23to these three variables x y and z. And
11:53:26we can copy this print up here and we'll
11:53:30hit shift enter. And now the X, Y, and Z
11:53:34all were assigned these values of
11:53:36chocolate, vanilla, and rocky road. Now
11:53:38something that we just did which is
11:53:40really important or something that you
11:53:41really need to consider is how you name
11:53:43your variables. So right here we have
11:53:46ice cream. Now this to me is exactly how
11:53:49I usually write my variables. But there
11:53:52are many different ways that you can
11:53:53write your variables. So, let's take a
11:53:54look at that really quickly and let's
11:53:57add just a few more because I have a
11:53:59feeling we're going to go a little bit
11:54:00longer than what we have. So, there are
11:54:02a few best practices for naming
11:54:04variables. First, I'm going to show you
11:54:05kind of what a lot of people will do.
11:54:08I'll show you some good practices and
11:54:09I'm going to show you some bad practices
11:54:11as well that you should avoid doing. The
11:54:14first thing that we're going to look at
11:54:14is something called camel case. And
11:54:17let's say we want to name it test
11:54:20variable case. Oops. Case. Now if we
11:54:24have a test variable case, the camel
11:54:26case is going to look like this. We'll
11:54:28have lowercase test and then we'll have
11:54:30uppercase variable and uppercase case is
11:54:34equal to. This is what this variable is
11:54:37going to look like. And we can assign it
11:54:39vanilla swirl.
11:54:43And this is what your camel case will
11:54:45look like. It's going to be lowercase.
11:54:47And then all the rest of those uh
11:54:49compound words or however you want to
11:54:50say that, these letters are going to be
11:54:52capitalized to kind of separate where
11:54:54the words end and begin. Let's go right
11:54:56down here. We're going to copy this. The
11:54:59next one is called Pascal case. So
11:55:02Pascal case is going to look just a
11:55:04little bit different. Instead of the
11:55:06lowercase at test, it's going to be a
11:55:08capital T in test. So test variable
11:55:11case. Again, this is a very similar way
11:55:14of writing it. Very similar to camel
11:55:15case. um but just a capital at the
11:55:18beginning. Now, let's look at the last
11:55:20one. And this one is my personal
11:55:22favorite. This one is going to be the
11:55:24snake case. Now, this one is quite a bit
11:55:27different in the fact that you don't use
11:55:29any capital letters and you separate
11:55:32everything using underscore. So, we're
11:55:34going to write test
11:55:36variable_case.
11:55:39Now, typically, let me have them all in
11:55:41there. Typically, these are the best
11:55:43practices. These are what you typically
11:55:46want to do, but probably the best one to
11:55:49use is this snake case right here. What
11:55:52a lot of people say is that it improves
11:55:54readability. If you take a look at
11:55:56either the camel case or the Pascal
11:55:58case, which you will see people do, it's
11:56:01not as easy to distinguish exactly what
11:56:03it says. And the name of a variable is
11:56:06important because you can gain
11:56:07information from it if people name them
11:56:09appropriately. So when I'm naming
11:56:11variables, I usually write it in snake
11:56:13case because I just find it a lot easier
11:56:15to read because each word is broken up
11:56:18by this underscore. So now let's look at
11:56:20some good variable names. These are all
11:56:22ones that you can use or could use. So
11:56:24let's do something like test var. So
11:56:27test var is completely appropriate. We
11:56:30can also do something like test_var
11:56:33oops underscore. We could do underscore
11:56:37test_var.
11:56:39You'll see that often as well where
11:56:41people will start it with an underscore.
11:56:44You can do test var
11:56:49capital T oops capital T capital V in
11:56:54test var or you could even do something
11:56:56like test
11:56:58var 2. Now adding a number to your
11:57:01variable is not inherently a bad thing.
11:57:03Usually it's semifowned upon but there
11:57:06are definitely some use cases where you
11:57:07can use it. But one thing that you
11:57:10cannot do is do something like
11:57:14putting the two at the front. If you put
11:57:16the two at the front, it no longer
11:57:17works. It won't run properly at all. So,
11:57:20we're going to take that out. So, we
11:57:22can't do that. So, I'm going to use this
11:57:23as an example of what you should not do.
11:57:25You also can't use a dash. So, something
11:57:28like test dash var 2. That doesn't work
11:57:32either. And you also can't use something
11:57:35like a space
11:57:38or a comma or really any kind of symbol
11:57:41like a period or a backslash or equal
11:57:44sign. None of those things will work
11:57:46within your variable. Now, another thing
11:57:48that you can do within your variable is
11:57:50use the plus sign. So, let's assign
11:57:52this. We'll say x is equal to and we'll
11:57:56do a string. We'll say ice cream
11:57:59is my favorite.
11:58:02and then we'll do a plus sign and we'll
11:58:05say period. Now what this will do is it
11:58:08will literally add these two strings
11:58:11together. So let's do print and we'll do
11:58:14x. So now it says ice cream is my
11:58:18favorite. One thing that we cannot do in
11:58:21a variable is we cannot add a string and
11:58:24a number or an integer. So we can't do
11:58:26ice cream is my favorite too. If we try
11:58:29to do that it will give us this error
11:58:30right here. So in this error it's saying
11:58:32you can only concatenate a string not an
11:58:35integer to a string. So only a string
11:58:37plus a string for this example. You can
11:58:40also do and we'll say x is equal to or
11:58:43we'll say y
11:58:45we'll say y is equal to
11:58:493 + 2 and it should output five because
11:58:52you can also do an integer and an
11:58:54integer. Now, so far we've only been
11:58:56outputting one variable in the print
11:58:58statement, but you can actually add
11:59:00multiple variables within a print
11:59:02statement. So, let's go right down here.
11:59:05We're going to say, let's get some more
11:59:07right there. So, we'll say x is equal to
11:59:11ice cream and we'll say y is equal to
11:59:17is. And then the last one, z is equal to
11:59:22my favorite. and we'll do a period at
11:59:25the end. Now we can go to the bottom and
11:59:27we can say print x + y + c. And when we
11:59:33enter that
11:59:35and when we run and when we run that we
11:59:37get ice cream is my favorite. Now we can
11:59:39actually add a space before is a space
11:59:42before my and when we hit shift enter it
11:59:45says ice cream is my favorite. You can
11:59:47also do this exact same thing with
11:59:49numbers as well. So we'll say x is equal
11:59:53to 1 2 and y z is equal to 3. So this
11:59:57should equal six. Now one thing that we
12:00:00tried to do was assign to one variable a
12:00:02string plus an integer and that did not
12:00:04work. But what you can do is you can
12:00:07take something like this and you can say
12:00:09ice cream
12:00:11and we'll get rid of this one and we'll
12:00:14get rid of the z. Now saying plus is
12:00:16actually not going to work. Let's try
12:00:18running this. So again, we can't
12:00:20concatenate these, but what we can do in
12:00:22the print statement is we can separate
12:00:24it by a comma. So when we add this
12:00:26comma, it should work properly. Let's
12:00:28hit enter. And it says ice cream 2.
12:00:31Again, this makes no sense, but you are
12:00:33able to combine a string and an integer
12:00:36separating by a comma. Now, this is the
12:00:37meat and potatoes of variables. There
12:00:40are some other things as well, but some
12:00:41of those things are a little bit more
12:00:42advanced and not something I wanted to
12:00:44cover in this tutorial. Although, we may
12:00:46be looking at some of those things in
12:00:47future tutorials. But this is definitely
12:00:50the basics, what you really, really need
12:00:52to know about variables. I hope that
12:00:54this video was helpful. If it was, be
12:00:56sure to like and subscribe below. And I
12:00:58will see you in the next video.
12:01:01[music]
12:01:08>> [music]
12:01:11>> Hello everybody. Today we're going to be
12:01:13talking about data types in Python. Data
12:01:15types are the classification of the data
12:01:17that you are storing. These
12:01:19classifications tell you what operations
12:01:20can be performed on your data. We're
12:01:22going to be looking at the main data
12:01:24types within Python, including numeric,
12:01:26sequence type, set, boolean, and
12:01:29dictionary. So, let's get started
12:01:30actually writing some of this out. And
12:01:32first let's look at numeric. There are
12:01:34three different types of numeric data
12:01:36types. We have integers, float, and
12:01:38complex numbers. Let's take a look at
12:01:40integers. An integer is basically just a
12:01:43whole number whether it's positive or
12:01:44negative. So an integer could be a 12.
12:01:47And we can check that by saying type.
12:01:50We'll do an open parenthesis and a
12:01:52closed parenthesis. And if we say the
12:01:54type of 12, it's going to give us an
12:01:56integer. Or if we say a -12, that is
12:01:59also an integer. We can also perform
12:02:01basic calculations like -12 + 100 and
12:02:04that'll tell us it is also an integer.
12:02:06So whether it's just a static value or
12:02:09you're performing an operation on it,
12:02:10it's still going to be that data type if
12:02:12those numbers are whole numbers whether
12:02:14negative or positive. Now let's take
12:02:16this exact one and let's say 12 and
12:02:20we'll do plus 10.25.
12:02:23When we run this, it's no longer going
12:02:24to be a whole number. It'll now be a
12:02:26float. So let's check this. And now this
12:02:29is a float type because it is no longer
12:02:31a whole number. It's now a decimal
12:02:32number. And the last data type within
12:02:34the numeric data type is called complex.
12:02:37Let's copy this right down here. Now
12:02:39personally this is not one that I've
12:02:40used almost ever, but it is one just
12:02:43worth noting. So you can do 12 plus and
12:02:46let's say 3 J. And if we do this, it's
12:02:50going to give us a complex. The complex
12:02:52data type is used for imaginary numbers.
12:02:55For me, it's not often used, but if you
12:02:57do use it, J is used as that imaginary
12:03:00number. If you use something like C or
12:03:04any other number, it's going to give you
12:03:06an error. J is the only one that will
12:03:08work with it. Now, let's take a look at
12:03:10boolean values. So, we'll say boolean.
12:03:13The boolean data type only has two
12:03:16built-in values, either true or false.
12:03:18So, let's go right down here and say
12:03:20type true.
12:03:23And when we run this, it'll say bool,
12:03:25which stands for boolean. We can do the
12:03:27exact same thing with false, and that is
12:03:30also boolean. And this can be used with
12:03:32something like a comparison operator. So
12:03:34let's say one is greater than five. And
12:03:38let's check this. This is giving us a
12:03:40boolean because it's telling us whether
12:03:42one is greater than five. Let's bring
12:03:44that right down here. This will give us
12:03:46a false. So it's telling us that one is
12:03:49not greater than five. And just as we
12:03:51got a false, we can say one is equal to
12:03:53one. And this should give us a true. So
12:03:56now let's take a look at our sequence
12:03:57type data types. And that includes
12:03:59strings, lists, and tupils. We'll start
12:04:02off by looking at strings. In Python,
12:04:05strings are arrays of bytes representing
12:04:07Unicode characters. When you're using
12:04:09strings, you put them either in a single
12:04:11quote, a double quote, or a triple
12:04:12quote. I call them apostrophes. It's
12:04:14just what I was raised to call them, but
12:04:16most people who use Python call them
12:04:18quotes. So right here we have a single
12:04:20quote and that works well. We can do a
12:04:24double quote and that works also. And as
12:04:28you can see they are the exact same
12:04:29output. And then we have a triple quote
12:04:32just like this. And this is called a
12:04:34multi-line. So we can write on multiple
12:04:36lines here. So let's write a nice little
12:04:39poem. So, we'll say the ice cream
12:04:42vanquished my longing for sweets upon
12:04:47this diet. I look away.
12:04:50It no longer exists
12:04:54on this day. And then if we run that,
12:04:56it's going to look a little bit weird.
12:04:59It's basically giving us the raw text,
12:05:01which is completely fine. But let's call
12:05:03this a multi-line.
12:05:06And we're going to call this a variable
12:05:08multi-line. And we're going to come down
12:05:10here and say print.
12:05:13And before I run this, I have to make
12:05:15sure that this is ran. So now let's
12:05:18print out our multi-line. And now we
12:05:20have our nice little poem right down
12:05:22here. Now something to know about these
12:05:24single and double quotes is how they're
12:05:25actually used. So if we use a single
12:05:28quote and we say, "I've always wanted to
12:05:33eat a gallon of ice cream." and then we
12:05:36do an apostrophe at the end. Obviously,
12:05:38something went wrong here. What went
12:05:40wrong is when you use a single quote and
12:05:43then within your text, within your
12:05:45sentence, you have another apostrophe,
12:05:47it's going to give you an error. So,
12:05:49what we want to do is whenever we have a
12:05:52quote within it, we need to use a double
12:05:55quote. These double quotes will negate
12:05:57any single quotes that you have within
12:05:59your statement. They won't, however,
12:06:01negate another double quote. So you need
12:06:03to make sure you aren't using double
12:06:05quotes within your sentence. If you want
12:06:07to do something like that, you need to
12:06:08use the triple quotes like we did above.
12:06:11So we can do double double and then
12:06:15let's paste this within it.
12:06:19And anything you do within these triple
12:06:21quotes will be completely fine as long
12:06:23as you don't do triple quotes within
12:06:25your triple quotes. We'll say this is
12:06:27wrong. So even though it's between these
12:06:29two triple quotes, it doesn't work
12:06:31exactly. Again, you just have to
12:06:33understand how that works. You have to
12:06:34use the proper apostrophes or quotes
12:06:36within your string. And just to check
12:06:38this, we can always say, here's our
12:06:40multi-line. We can always say type of
12:06:45multi-line.
12:06:47And that is still a string. One really
12:06:50important thing to know about strings is
12:06:52that they can be indexed. Indexing means
12:06:54that you can search within it. And that
12:06:56index starts at zero. So, let's go ahead
12:06:58and create a variable. And we'll just
12:07:00say a is equal to and let's do the all
12:07:04popular hello world. Let's run this. And
12:07:08now when we print this string, we can
12:07:10say a and we're going to do a bracket.
12:07:13And now we can search throughout our
12:07:14string using the index. So all you have
12:07:17to do is do a colon. We can say five.
12:07:21What this is going to do is it's going
12:07:22to say zero position zero all the way up
12:07:24to five, which should give us the whole
12:07:26hello, I believe. Let's run this. and
12:07:29it's giving us the first five positions
12:07:30of this string. We can also get rid of
12:07:33the colon and just say something like
12:07:35five. And then when we run this, it's
12:07:38actually going to give us position five.
12:07:41So this is 0 1 2 3 4 and then five is
12:07:45the space. Let's do six so we can see
12:07:47the actual letter. And that is our W. We
12:07:50can also use a negative when we're
12:07:52indexing through our string. So we could
12:07:54say -3 and it'll give us the L because
12:07:57it's -1 2 and 3. We can also specify a
12:08:01range if we don't want to use the
12:08:02default of zero. So before we did 0 to 5
12:08:05and it started at zero because that was
12:08:07our default but we could also do 2 to 5.
12:08:10Let's run this. And now we go position 0
12:08:131 and then we start at two l. Now we can
12:08:17also multiply strings and we have this a
12:08:20hello world. So we can do a * 3 and if
12:08:24we run this it'll give us hello world
12:08:26three times and we can also do a + a and
12:08:31that is hello world hello world. Now
12:08:34let's go down here and take a look at
12:08:35lists. Lists are really fantastic
12:08:37because they store multiple values. The
12:08:40string was stored as one value multiple
12:08:42characters but a list can store multiple
12:08:45separate values. So let's create our
12:08:47very first list. We'll say list really
12:08:50quickly and then we'll put a bracket and
12:08:53a bracket means this is going to be a
12:08:55list. There are other ones like a
12:08:57squiggly bracket and a parenthesis.
12:09:00These denote that they are different
12:09:01types of data types. The bracket is what
12:09:03makes a list a list. So to keep it super
12:09:06simple, we'll say 1 2 3 and we'll run
12:09:09this. And now we have a list that has
12:09:10three separate values in it. The comma
12:09:13in our list denotes that they are
12:09:14separate values. And a list is indexed
12:09:17just like a string is indexed. So
12:09:19position zero is this one. Position one
12:09:21is the two and position two is the
12:09:24three. Now when we made this list, we
12:09:26didn't have to use any quotes because
12:09:27these are numbers. But if we wanted to
12:09:30create a list and we wanted to add
12:09:32string values, we have to do it with our
12:09:34quotes. So we'll say quote cookie dough.
12:09:38Then we'll do a comma to separate the
12:09:40value. And then we'll say strawberry.
12:09:44And then we'll do one more and this will
12:09:46just be chocolate. And when we run this,
12:09:48we have all three of these values stored
12:09:50in our list. Now, one of the best things
12:09:52about list is you can have any data type
12:09:54within them. They don't just have to be
12:09:56numbers or strings. You can basically
12:09:59put anything you want in there. So,
12:10:01let's create a new list. And let's say
12:10:04vanilla.
12:10:05And then we'll do three. And then we'll
12:10:08add a list within a list. And we'll say
12:10:11scoops.
12:10:13comma spoon. And then we'll get out of
12:10:17that list. And then we'll add another
12:10:19value of true for boolean. And now we
12:10:22can hit shift enter. And we just created
12:10:25a list with several different data types
12:10:28within one list. Now let's take this one
12:10:31list right here with all of our
12:10:32different ice cream flavors. We'll say
12:10:34ice cream is equal to this list. Now one
12:10:38thing that's really great about lists is
12:10:40that they are changeable. That means we
12:10:42can change the data in here. We can also
12:10:44add and remove items from the list after
12:10:47we've already created it. So let's go
12:10:49and take ice cream and we'll say ice
12:10:51cream.append.
12:10:53And this is going to append it to the
12:10:54very end of the list. We'll do an open
12:10:57parenthesis. Let's say salted caramel.
12:11:01Now when we run this and we call it just
12:11:04like this, it's going to take this list
12:11:07add salted caramel to the end and we'll
12:11:10print it off. And as you can see, it was
12:11:12added to the list. And just like I said
12:11:14before, let me go down here. We can also
12:11:17change things from this list. So let's
12:11:19say ice cream. And then we need to look
12:11:21at the indexed position. So we're going
12:11:23to say zero. And that's going to be this
12:11:25cookie dough right here. We can say that
12:11:27is equal to. So we can now change that
12:11:30value. So let's call that butter pecan.
12:11:34And now when we call it,
12:11:37we can now see that the cookie dough was
12:11:39changed to butter pecan. Another thing
12:11:41that you saw just a little bit ago is
12:11:43something called a list within a list.
12:11:45Basically a nested list. So we had
12:11:48scoops spoon true. Let's give this and
12:11:51we'll say nested
12:11:54list is equal to. Now when we run this,
12:11:57we now have this nested list. So if we
12:12:00look at the index and we say zero, we'll
12:12:03get vanilla. If we say two, we'll get
12:12:06scoops and spoons. Now since we have a
12:12:08list within a list, we can also look at
12:12:10the index of that nested list. So let's
12:12:13now say one. And that should give us
12:12:16just spoon. And you can go on and on and
12:12:19on with this. You can do lists within
12:12:20lists within lists. And all of them will
12:12:23have indexing that you can call. Now
12:12:25let's go down here and start taking a
12:12:26look at tupils. So a list and a tupil
12:12:29are actually quite similar, but the
12:12:31biggest difference between a list and a
12:12:33tupil is that a tupil is something
12:12:35called immutable. It means it cannot be
12:12:37modified or changed after it's created.
12:12:39So let's go right up here. We're going
12:12:41to say tupil and let's write our very
12:12:45first tupil. So we'll say tupil
12:12:48scoops
12:12:50is equal to and then we'll do an open
12:12:52parenthesis. Now these open parenthesis
12:12:54you've seen if you do like a print
12:12:55statement but that's different because
12:12:57that's executing a function. This is
12:13:00actually creating a tupil which is going
12:13:01to store data for us. So we'll say 1 2 3
12:13:052 and 1. Let's go ahead and create that
12:13:09tupil. And we can just check the data
12:13:11type really quickly. And it's a tupole.
12:13:14And just like we saw before a tupil is
12:13:17also indexed. So if we go at the very
12:13:19first position which is a one, we will
12:13:22get the output of a one. But we can't do
12:13:25something like append and then add a
12:13:28value like three. If we do that, it's
12:13:30going to say tupil object has no
12:13:32attribute append. It's just because you
12:13:34cannot change or add anything to a
12:13:37tupil. Just like we were talking about
12:13:38before, typically people will use tupils
12:13:41for when data is never going to change.
12:13:43An example for this might be something
12:13:45like a city name, a country, a location,
12:13:48something that won't change. They
12:13:49definitely have their use cases, but I
12:13:51don't think they're as popular as just
12:13:52using a list. So, now let's scroll down
12:13:54and start taking a look at sets. But
12:13:57really quickly, let me add a few more
12:14:00cells for us. And let's say sets.
12:14:05Now a set is somewhat similar to a list
12:14:08and a tupil, but they are a little bit
12:14:11different in the fact that they don't
12:14:12have any duplicate elements. Another big
12:14:15difference is that the values within a
12:14:17set cannot be accessed using an index
12:14:19because it doesn't have an index because
12:14:21it's actually unordered. We can still
12:14:23loop through the items in a set with
12:14:25something like a for loop, but we can't
12:14:26access it using the bracket and then
12:14:28accessing its index point. So let's go
12:14:31ahead and create our very first set. So,
12:14:33we're going to say daily_pints.
12:14:36Then, we're going to say equal to. And
12:14:38to create a set, we're going to use
12:14:40these squiggly brackets. I don't know if
12:14:42there's an actual name for those, if I'm
12:14:43being honest. I call them squiggly
12:14:45brackets, and that's what we're going to
12:14:46go with. We're going to put in a one, a
12:14:48two, and a three. So, let's go ahead and
12:14:50run this.
12:14:52And let's look at the type. And as you
12:14:55can see, it is a set. Now, when we print
12:14:57this out, it's going to show us one, a
12:15:00two, and a three. And those are all the
12:15:02values within our set. But if we copy
12:15:04this and we'll say daily pints log, this
12:15:07is going to be every single day. Maybe I
12:15:11had different values.
12:15:13Now when we run this and we do the exact
12:15:15same thing. Now when we print this,
12:15:20it's going to have just the unique
12:15:21values within that set. Now a use case
12:15:23for set and this is something that I've
12:15:25done in the past is comparing two
12:15:27separate sets. Maybe you have a list or
12:15:29a tupil and you convert that into a set
12:15:31and that will narrow it down to its
12:15:33unique values. Then you can compare the
12:15:35unique values of one set to the unique
12:15:37values in another set. And then we can
12:15:39see what's the same and what's
12:15:40different. So let's go down here and
12:15:42let's say wife's
12:15:45daily and we'll just copy this right
12:15:48here. We'll say is equal to let's do our
12:15:51squiggly lines. Let's do one two. Let's
12:15:54do just random numbers.
12:15:57So now this is my daily log and this is
12:16:00my wife's daily log. And now we can
12:16:02compare these values. So let's go right
12:16:04down here. Let's say print. We'll do my
12:16:09daily logs and then we'll do this bar
12:16:12right here. And this is going to show us
12:16:13the combined unique values. It's
12:16:15basically like putting them all in one
12:16:17set and then trimming it down to just
12:16:19the unique values. So we'll take wife's
12:16:21daily pints log. And when we run this,
12:16:24we actually need to run this first. When
12:16:26we run this, we should see all the
12:16:27unique values between these two sets.
12:16:30And so, as you can see, 0 1 2 3 4 5 6 7
12:16:3324 31. So, these are all the unique
12:16:36values between these two sets.
12:16:39We can also do another one. And instead
12:16:42of this bar, we're going to do this
12:16:44symbol right here, which I believe is
12:16:46called an amperand. Don't quote me on
12:16:48that. But when we run this, it's going
12:16:50to show what matches. That means which
12:16:53ones show up in both sets. So the only
12:16:56ones that show up in both sets are 1 2 3
12:16:59and five. We can also do the opposite of
12:17:01that by doing a minus sign. And this is
12:17:04going to show us what doesn't match. And
12:17:06so we have 4 6 and 31. Now where is our
12:17:1024 that was in our wife's daily pints
12:17:12log? It's in this one, but we're
12:17:14subtracting the values on this one. So
12:17:16let's reverse this and we'll say daily
12:17:19pints log
12:17:21and let's run it. Now those are our
12:17:23other values. So, we're taking the
12:17:24values of this and then we're
12:17:26subtracting all the ones that are the
12:17:28same and getting the remaining values.
12:17:31And then for our last one, we can get
12:17:33rid of this and we'll do this symbol
12:17:36right here. And this is going to show if
12:17:38a value is either in one or the other,
12:17:41but not in both. So, let's run this. So,
12:17:44these values are completely unique only
12:17:47to each of those sets. Now, the very
12:17:50last one that we are going to look at in
12:17:51this video is dictionaries. So, let's go
12:17:54right down here. Let's add a few cells
12:17:57and let's say dictionaries.
12:18:00Now, I saved dictionary for last because
12:18:02this one is probably the most different
12:18:04out of all the previous data types that
12:18:06we've looked at. Within a data type, we
12:18:08have something called a key
12:18:11value pair. That means when we use a
12:18:14dictionary, it's not like a list where
12:18:16you just have a value, value, comma,
12:18:18value. we have a key that indicates what
12:18:21that value is attributed to. So let's
12:18:24write out a dictionary to see how this
12:18:26looks. So we're going to say dictionary
12:18:29cream. And just like a set, we use a
12:18:32squiggly line. But the thing that
12:18:34differentiates it is that in a
12:18:36dictionary, we'll have that key value
12:18:37pair. Whereas in a set, each value is
12:18:40just separated by a comma. So let's
12:18:42write name. And this is our key. And
12:18:45then we do a colon. And this is then
12:18:47where we input our value. So we're going
12:18:49to say Alex freeberg. And then we
12:18:53separate that key value pair by a comma.
12:18:56And now we can do another key value
12:18:57pair. So we'll say weekly intake and a
12:19:03colon. And we'll say five pints of ice
12:19:06cream. Do a comma. And then we'll do
12:19:09favorite ice creams. And now what we're
12:19:12going to do is we're going to put in
12:19:13here a list. So within this dictionary,
12:19:16we can also add a list. We'll do MCC for
12:19:19mint chocolate chip. And then we'll add
12:19:21chocolate, another one of my favorites.
12:19:23So now we have our very first
12:19:25dictionary. Let's copy this and run it.
12:19:29And let's just look at the type. And as
12:19:32you can see, it says that this is a
12:19:34dictionary. Let's also print it out.
12:19:37Now, if we want to, we can take our
12:19:39dictionary cream and say values with an
12:19:43open parenthesis. And when we execute
12:19:45this, we'll see all of the values within
12:19:47this dictionary. So, here's our values
12:19:49of Alex Freeberg, five, mint chocolate
12:19:51chip, and chocolate. We can also say
12:19:54keys, and when we run this, all of the
12:19:56keys, the name, weekly intake, and
12:19:58favorite ice creams. And we can also say
12:20:03items. So, this key value pair is one
12:20:06item. And this key value pair is another
12:20:08item. Now one difference between
12:20:11something like a list and a dictionary
12:20:13is how you call the index. But you can't
12:20:15call it by doing something like this
12:20:17where you just do a bracket oops and say
12:20:20zero. So this would in theory take this
12:20:24very first one, right? Our very first
12:20:26key value pair. That's going to give us
12:20:28an error. How you call a dictionary is
12:20:29actually by the key. So it doesn't
12:20:31technically have an index, but you can
12:20:33specify what you want to call and take
12:20:35it out. So we're going to say name and
12:20:38this is going to call that key right
12:20:40here. And when we run this we'll get the
12:20:43value which is Alex Freeberg. One other
12:20:46thing that you can do is you can also
12:20:48update information in a dictionary which
12:20:50we can't with some other data types. So
12:20:52for this for the name it was Alex
12:20:54Freeberg. Now let's say Freeberg and
12:20:59when we update that I'm also going to
12:21:01print the dictionary. get rid of this.
12:21:06So, it's going to update Christine
12:21:08Freeberg in that value of the name. So,
12:21:11let's go ahead and run this. And now, it
12:21:14changed the name from Alex Freeberg to
12:21:15Christine Freeberg. We can also update
12:21:18all of these values at one time. So,
12:21:21let's copy this
12:21:24and I'm going to put it right down here.
12:21:26I'm going to say
12:21:27dictionary.cream.update.
12:21:29Then we're going to put a bracket or not
12:21:32a bracket but a parenthesis around
12:21:33these. So now what we're going to do is
12:21:36update this entire thing. Let me take
12:21:38this say print this dictionary. Now we
12:21:43can update this to anything we want. So
12:21:46instead of here I can say
12:21:49I'll say weight
12:21:51and because of all that ice cream I now
12:21:53weigh 300 lb. So let's run this. And as
12:21:58you can see, it did not delete our key
12:22:00value pair right here. Instead, it just
12:22:02added to it. When you're using the
12:22:04update, we can't actually delete. That's
12:22:06the delete statement, and I'll show you
12:22:08that in just a second. But all we did
12:22:10was added this new value. It also is
12:22:12going to check and see if you changed
12:22:14anything with your key value pair. So,
12:22:16we can go in here and change this value.
12:22:18And we'll say 10. So, now when we run
12:22:20this, the value of this key value pair
12:22:23was changed. But let's say we do want to
12:22:25delete it. We'll say deel. that stands
12:22:27for delete part of this dictionary
12:22:30cream. And now let's specify the key
12:22:32which will also delete the value with
12:22:34it. Well, let's specify the key that we
12:22:36want to get rid of. And let's say wait.
12:22:39And then let's print that again.
12:22:43And as you can see, the weight was
12:22:46deleted from that dictionary. So that is
12:22:48all we're going to cover in this data
12:22:49types video. Thank you guys so much for
12:22:51watching. I really appreciate it. If you
12:22:53like this video, be sure to like and
12:22:54subscribe below, and I'll see you in the
12:22:56next video.
12:23:09Hello everybody. Today we're going to be
12:23:11taking a look at comparison, logical,
12:23:12and membership operators in Python.
12:23:14Operators are used to perform operations
12:23:16on variables and values. For example,
12:23:19you're often going to want to compare
12:23:20two separate values to see if they are
12:23:22the same or if they're different within
12:23:24Python. And that's where the comparison
12:23:26operator comes in. Right here, you can
12:23:27see our operators. You can also see what
12:23:29they do. So, this equal sign, equal sign
12:23:32stands for equal. We have the does not
12:23:34equal, the greater than, less than,
12:23:36greater than or equal to, and less than
12:23:38or equal to. And honestly, I use these
12:23:40almost every single time I use Python.
12:23:42So, these are very important to know and
12:23:44know how to use. So, let's get rid of
12:23:45that really quickly and actually start
12:23:47writing it out and see how these
12:23:48comparison operators work in Python. The
12:23:50very first one that we're going to look
12:23:51at is equal to. Now, you can't just say
12:23:5310 is equal to 10. Let's try running
12:23:56that really quickly by clicking shift
12:23:58enter. It's going to say cannot assign
12:24:00to literal. That's because this is like
12:24:02assigning a variable. We're trying to
12:24:03say 10 is equal to 10 and then we can
12:24:06call that 10 later. But that's not how
12:24:08this actually works. What we're trying
12:24:09to do is to determine whether 10 is
12:24:11equal to 10. So, we're going to say
12:24:13equal sign equal sign. And then if we
12:24:15run that by clicking shift enter again,
12:24:17it's going to say true. Now, if we put
12:24:19something else like 50 in there and we
12:24:21try to run this, it's going to say
12:24:23false. So, really what you're going to
12:24:25get when you use these comparison
12:24:26operators is either a true or a false.
12:24:29If we take this right down here, we can
12:24:31also say does not equal. And we're going
12:24:33to use an exclamation point equal sign.
12:24:35And that says 10 is not equal to 50. And
12:24:37that should be true. You can also
12:24:39compare strings and variables. So, let's
12:24:41go right down here and we're going to
12:24:43say vanilla is not equal
12:24:48to chocolate. And when we run this,
12:24:50it'll say false. Now, if it was the
12:24:53same, just like when we did our numbers,
12:24:54it should say true. And we can also
12:24:56compare variables. So, we'll say x is
12:24:59equal to vanilla and y is equal to
12:25:03chocolate. And then when we come down
12:25:05here, we can say x is equal to y. And
12:25:08it'll give us a false. and we say x is
12:25:12not equal to y and it'll give us a true.
12:25:15The next one that we're going to take a
12:25:16look at is the less than. So let's copy
12:25:18this one right up here. Let's scroll
12:25:20down and let's say 10 is less than 50.
12:25:26Now this will come out as true. Now
12:25:28let's say we put a 10 in here. Before 10
12:25:31was of course less than 50. But is 10
12:25:34less than 10? No. That's false because
12:25:37they are the same. So if we want an
12:25:38output that is true, all we would have
12:25:40to add is an equal sign right here. And
12:25:42this would say 10 is less than or it is
12:25:45equal to 10. And now it's true. Of
12:25:49course, we can say the exact same thing
12:25:50by saying greater than. So 10 is equal
12:25:53or greater than 10. That'll be true
12:25:55because 10 is equal to 10. But we can
12:25:58also say 50 is greater or equal to 10
12:26:01because 50 is obviously greater than 10.
12:26:03Now let's look at logical operators that
12:26:05are often combined with comparison
12:26:07operators. So our operators are and or
12:26:10and not. So if you have an and that
12:26:12returns true if both statements are
12:26:14true. If it's or only one of the
12:26:17statements has to be true. And the not
12:26:19basically reverses the result. So if it
12:26:21was going to return true, it would
12:26:23return false. I don't use this not one a
12:26:26lot, but I will show you how it works.
12:26:28So let's actually test that out. So
12:26:30before we were saying 10 is greater than
12:26:3250 and of course this returned false. So
12:26:35now let's add a parentheses around this.
12:26:3810 is greater than 50 and we're going to
12:26:39say and we'll do an open parenthesis. 50
12:26:43is greater than 10. Now this statement
12:26:45right here is true. 50 is greater than
12:26:4710. So we have a true statement and a
12:26:50false statement. But this and is going
12:26:52to look at both of them. It's going to
12:26:53say they both need to be true in order
12:26:56to return a true. So let's try running
12:26:58this. and we still have a false. If we
12:27:01want it to return true, we're going to
12:27:02have to change this to make it a true
12:27:04statement. So 70 is greater than 50 and
12:27:0650 is greater than 10. When we run this,
12:27:09it should return true. Now let's look at
12:27:11the or. So let's copy this and we'll say
12:27:1510 is greater than 50 or 50 is greater
12:27:19than 10. Now this is a false statement
12:27:21and this is a true statement. So if even
12:27:23one of them is a true statement, the
12:27:25output should be true. And again we can
12:27:27do this even with strings. So we can do
12:27:30vanilla
12:27:32and chocolate.
12:27:35There we go. And vanilla is actually
12:27:38greater than chocolate because v is a
12:27:40higher number in the alphabetical order.
12:27:42So v is like 20some whereas chocolate is
12:27:45three. Right? So it actually looks at
12:27:46the spelling for this. So if we say or
12:27:49here it will come out true. And if we
12:27:52say and here, it should also be true
12:27:54because V is greater than C and 50 is
12:27:57greater than 10. So this should also be
12:27:59true. Now let's copy this right here.
12:28:02And we're going to say not. So what we
12:28:05had before is 50 is greater than 10.
12:28:08That returned true. But now all we're
12:28:10doing is putting not in front of it. So
12:28:12instead of returning true, it's going to
12:28:13return false. So now let's take a look
12:28:15at membership operators. And we use this
12:28:17to check if something whether it's a
12:28:19value or a string or something like that
12:28:21is within another value or string or
12:28:24sequence. Our operators are in and not
12:28:26in. So it's pretty simple. If it's in,
12:28:28it's going to return true if the
12:28:30sequence with a specified value is
12:28:31present in the object just like we were
12:28:33talking about. And for not in, it's
12:28:35basically the exact same thing if it's
12:28:37not in that object. So let's start out
12:28:38by taking a look at a string. We're
12:28:40going to say ice cream is equal to I
12:28:44love chocolate ice cream.
12:28:48And then we're going to say love in ice
12:28:52cream. And that will return true. So all
12:28:55we're doing is searching if the word
12:28:56love or that string is in this larger
12:28:59string. We could also just do that by
12:29:01literally copying this and putting this
12:29:03where this is. So we can check is this
12:29:05string part of this string and it'll say
12:29:08true. We can also make a list. So we'll
12:29:10say scoops is equal to and then we'll do
12:29:13a bracket and we'll say 1 2 3 4 5. Then
12:29:17we'll say two in scoops. So all we're
12:29:21doing is searching to see if two is
12:29:22within this list. And that should return
12:29:25true. Now if we put a six here and we
12:29:28said not in, it will also return true
12:29:32because six is not in scoops and that is
12:29:34true. And just like we did, we could
12:29:36also say wanted scoops and we'll say
12:29:40eight. So I wanted eight scoops. So we
12:29:43can say wanted scoops in scoops. And
12:29:46this should return true because there's
12:29:48not an eight within the scoops that we
12:29:50wanted. And if we said in and we said we
12:29:54wanted eight, is that within our list
12:29:56that we created? And that's going to
12:29:58return a false. So that is a quick
12:30:00breakdown of comparison, logical, and
12:30:02membership operators. I hope that this
12:30:04was helpful. Thank you guys so much for
12:30:06watching. If you like this video, be
12:30:08sure to like and subscribe and I will
12:30:10see you in the next video.
12:30:16[music]
12:30:23Hello everybody. Today we're going to be
12:30:25taking a look at the if statement within
12:30:27Python. Now, it's actually the if l if
12:30:29else statement, but that's a mouthful,
12:30:30so I'm just going to call it the if else
12:30:32statement. Now, we have this flowchart,
12:30:34and I apologize for it being blurry, but
12:30:36this is the absolute best one that I
12:30:37could find. Right up top, we have our if
12:30:39condition. Now, if this if condition is
12:30:42true, we're going to run a body of code.
12:30:44But if that condition is false, we're
12:30:46going to go over here and go to the LF
12:30:48condition. The LF condition or statement
12:30:50is basically saying if the first if
12:30:52statement doesn't work, let's try this
12:30:54if statement. If this LF statement is
12:30:56true, it goes to this body of code. If
12:30:58it's false, it'll come over here to the
12:31:00else. And the else is basically if all
12:31:02of these things don't work then run this
12:31:05body of code. Now you can have as many
12:31:07ill if statements as you want but you
12:31:08can only have one if statement and one
12:31:10else statement. So let's write out some
12:31:12code and see how this actually looks.
12:31:14Let's first start off by writing if.
12:31:15That is our if statement. And now we
12:31:17have to write our condition which is
12:31:19about to be either met or not met. So
12:31:21we'll say if 25 is greater than 10 which
12:31:24is true. We'll say colon and then we're
12:31:27going to hit enter and it's going to
12:31:29automatically indent that line of code
12:31:30for us. And this is our body of code. So
12:31:33if 25 is greater than 10, our body of
12:31:35code will execute. So for us, we're just
12:31:38going to write print and we'll say it
12:31:40worked. Now if we run this, it's going
12:31:42to check is 25 greater than 10. If that
12:31:45is true, print this. So let's hit shift
12:31:49enter. And it worked. Now let's take
12:31:52this exact code. We'll paste it right
12:31:54down here. And we'll say is less than.
12:31:57And right now, this if statement is not
12:31:59true. So, it's not actually going to
12:32:01work. As you can see, there's no output.
12:32:04There's nothing that happened really.
12:32:05But it did check to see if 25 was less
12:32:07than 10, but it just wasn't true. Now,
12:32:10we can use our else statement. So, we're
12:32:12going to come right down here, and we're
12:32:13going to say else, and we'll do a colon,
12:32:16and we'll hit enter. Again,
12:32:17automatically indenting. And we're going
12:32:18to say print. and we're going to say it
12:32:22did not work dot dot dot. So what it's
12:32:25going to do is it's going to come up
12:32:26here and check is 25 less than 10. No,
12:32:30it's not. So this body of code is not
12:32:32going to be executed. It's going to go
12:32:33right down to this else statement. Now
12:32:35this else statement is going to be
12:32:36printed. There's no condition on this.
12:32:38So the if statement has a condition. 25
12:32:40is less than 10. This has no condition.
12:32:42So if this doesn't work, if this is
12:32:44false, it's going to come down here and
12:32:46it will run this body of code. Let's run
12:32:48this by clicking shift enter. And as you
12:32:51can see, our output is it did not work.
12:32:54Now, let's go back up here and put
12:32:56greater than because this is now true.
12:32:58It's going to say if 25 is greater than
12:33:0010, print it worked. And then it's going
12:33:02to stop. It's not going to go to this
12:33:04else statement at all. So, let's run
12:33:06this. And our output is it worked. So,
12:33:09what if we have a lot of different
12:33:10conditions that we want to try? Let's
12:33:12come right down here. This is where the
12:33:14lf comes in. So really quickly, let's
12:33:16change this to a not true, a false
12:33:19statement. We're going to go down and
12:33:20say l if and we're going to say if it
12:33:24is, and let's say 30, we'll say l if
12:33:30worked.
12:33:32So now it's going to check is 25 less
12:33:35than 10. No, it's not. Let's look at the
12:33:37next condition. Is 25 less than 30? And
12:33:40if it is, we'll print l if worked. So
12:33:43let's try running this. and lf worked.
12:33:46Now, we can do as many of these LF
12:33:48statements as we want. We can do let's
12:33:51just try a few of them right here. So,
12:33:53we'll say if 25 is less than 20 is less
12:33:58than 21 and let's do 40 and let's do 50.
12:34:03So, we'll say LF, LF2, LF3, and LF4.
12:34:08Now, if you look at this, the first one
12:34:10that is actually going to work is this
12:34:1325 to 40 right here. Once this one is
12:34:16checked and it comes out as true, none
12:34:18of the other LF or else statements will
12:34:20work. So, let's try this one. It should
12:34:21be LF3.
12:34:23And this one ran properly. Now, within
12:34:26our condition so far, we've only used a
12:34:27comparison operator. We can also use a
12:34:30logical operator like and or or. So, we
12:34:33can say if 25 is less than 10, which
12:34:36it's not. and let's say or actually and
12:34:39we'll say or one is less than three
12:34:43which is true. If we run this now it
12:34:46will actually work. So we can use
12:34:47several different types of operators
12:34:49within our if statement to see if a
12:34:51condition is true or not or several
12:34:53conditions are true. There's also a way
12:34:55to write an if else statement in one
12:34:57line if you want to do that. So we can
12:34:59write print. We'll say it worked
12:35:03and then we'll come over here and say if
12:35:0510 is greater than 30 and then we'll
12:35:08write else print and we'll say it did
12:35:13not work just like we had before except
12:35:16now it's all occurring on one line. So
12:35:18let's just try this and see if it works.
12:35:21So it's saying print it worked if 10 is
12:35:24greater than 30 which it wasn't. So, it
12:35:25went to the else statement and then it
12:35:27printed out our body right here.
12:35:29Although we didn't have any indentation
12:35:30or multiple lines, it was all done in
12:35:32one line. Now, there's one other thing
12:35:34that we haven't looked at yet. Uh, and
12:35:36I'm going to show it to you really
12:35:37quickly. And that's a nested if
12:35:39statement. So, when we run this, it's
12:35:41going to say it worked. It works because
12:35:43it says 25 is less than 10 or 1 is less
12:35:46than three. Since this is true, it's
12:35:49going to print out it worked. But we can
12:35:51also do a nested if statement. So we can
12:35:53do multiple if statements as well. So
12:35:56we're going to hit enter and we'll say
12:35:57if and we'll do a true statement here.
12:35:59So we'll say if 10 is greater than five.
12:36:03Let's do a colon hit enter and then
12:36:06we'll say print and then we'll type a
12:36:07string saying this nested if statement
12:36:12oops worked.
12:36:14Now let's try this out and see what we
12:36:16get. So it went through the first if
12:36:18statement. It said it was true and it
12:36:20prints out it worked. This is still the
12:36:22body of code. So it goes down to this
12:36:24next if statement and it says if 10 is
12:36:26greater than five, we're going to print
12:36:28this out. And you could do this on and
12:36:30on and on. It can basically go on
12:36:32forever and you can create a really
12:36:34in-depth logic. And that actually
12:36:36happens a lot when you start writing
12:36:37more advanced code. So I hope that this
12:36:39was helpful. I hope that you understand
12:36:40the if else statement better. I hope
12:36:42that you understand how nested if
12:36:43statements work as well. Thank you guys
12:36:46so much for watching. If you like this
12:36:47video, be sure to like and subscribe
12:36:49below, and I'll see you in the next
12:36:50video.
12:37:03Hello everybody. Today we're going to be
12:37:05learning about for loops in Python. The
12:37:07for loop is used to iterate over a
12:37:09sequence, which could be a list, a
12:37:11tupil, an array, a string, or even a
12:37:13dictionary. Here's the list that we'll
12:37:15be working with throughout this video.
12:37:16And I have this little diagram right
12:37:18here which kind of explains how a for
12:37:20loop works. The for loop is going to
12:37:22start by looking at the very first item
12:37:24in our sequence or our list. And that's
12:37:26going to be our one right here. It's
12:37:28going to ask is this the last element in
12:37:31our list? And it is not. So it's going
12:37:34to go down to this body of the for loop.
12:37:36Now we can have a thousand different
12:37:38things that can happen in the body of
12:37:39the for loop as we're about to look at
12:37:41in just a second. Then it's going to go
12:37:43up to the next element and ask is this
12:37:45the last element reached. So it'll be no
12:37:48again because it'll be going to the two
12:37:50and then the three and then the four and
12:37:51the five. Once it reaches the five,
12:37:54it'll go to the body of the for loop and
12:37:56then when it asks if that's the last
12:37:58element, the answer would be yes because
12:38:00it's iterated through all the items
12:38:02within the list and then we would exit
12:38:04the loop and the for loop would be over.
12:38:06Now, that may not have made perfect
12:38:07sense, but let's actually start writing
12:38:09out the syntax of a for loop so we can
12:38:11understand this better. To start our for
12:38:13loop, we're going to say four. And then
12:38:15we're going to give it a temporary
12:38:17variable for this for loop. So, it's a
12:38:19variable. As it iterates through these
12:38:21numbers, it's going to assign the
12:38:23variable to that number. So, for this
12:38:25one, we're just going to say number
12:38:26because it's pretty appropriate because
12:38:28these are all numbers. And then we're
12:38:30going to say in integers. Now, right
12:38:34here, you can put just about anything.
12:38:35This could be the list. This could be a
12:38:37tupole. This could be a string even. But
12:38:40that is what we're going to iterate
12:38:41through. So we're saying for the
12:38:43variables, each of these numbers within
12:38:45this list of integers. And then we're
12:38:48going to write a colon. This is the body
12:38:50of code that's going to actually be
12:38:52executed when we run through and iterate
12:38:54through our list. So for our first
12:38:56example, we're going to start off super
12:38:58simple. And all we're going to do is say
12:39:00print open parenthesis and say number.
12:39:03as it iterates through the 1 2 3 4 and
12:39:06five number becomes our variable that is
12:39:09going to be printed. So during that
12:39:11first loop our one will be printed
12:39:13because that will be assigned right
12:39:15here. Then through the next iteration
12:39:17the two will be assigned and it'll be
12:39:19put right here in each loop until the
12:39:22very end. So let's hit shift enter. And
12:39:26as you can see it did exactly that. Now
12:39:28in this body and I'll copy and paste
12:39:30this down here. In this body, we really
12:39:32can do just about anything we want. We
12:39:34don't even have to use this variable
12:39:36number right here. We can just print yep
12:39:40if we wanted to. And what it's going to
12:39:42do is for each iteration, all five of
12:39:44those, every time it loops through, it's
12:39:46going to print off yep. So, let's hit
12:39:49shift enter. And it printed it off for
12:39:52us. So, really, we weren't even using
12:39:54the numbers within the list. We were
12:39:56really just using it as almost a
12:39:58counter. Now let's copy this integers
12:40:00once again. Let's go right up here and
12:40:02let's go copy this for loop that we
12:40:05wrote. Now we do not have to call this
12:40:09number. This can be anything you want.
12:40:11Any variable name that you'd like to
12:40:13name it. We could call it jelly and we
12:40:17can do
12:40:18jelly plus jelly.
12:40:22I think you're getting the picture,
12:40:23right? When it loops through that one,
12:40:25it's doing 1 plus one. when it loops
12:40:27through the two, it's doing 2 + 2. That
12:40:30is basically how a for loop works. Now,
12:40:32for a dictionary, it's going to handle
12:40:33it a little bit differently. So, let's
12:40:35create a dictionary really quickly. So,
12:40:38we'll say ice cream
12:40:41dictionary is equal to we're going to do
12:40:43a squiggly brackets. So, we're going to
12:40:45say name and we're going to say colon.
12:40:48We need to assign our value for that
12:40:50item. So, we're going to say Alex
12:40:53freeberg. We'll do our next one
12:40:54separated by a comma and we'll say
12:40:57weekly intake and I'll say five scoops
12:41:01per week. The next one we will do is
12:41:04favorite ice creams. And for this one,
12:41:08we're going to do something a little bit
12:41:09different. For this, we're going to have
12:41:10a list within this dictionary. So, we'll
12:41:13say within our list of my favorite ice
12:41:16creams, we'll say mint chocolate chip.
12:41:18And I'll just do MCC for that. and we'll
12:41:21separate that out by a comma and we'll
12:41:24say chocolate. So now we have this
12:41:26dictionary ice cream dict. And within it
12:41:28we have my name, my weekly intake, and
12:41:30my favorite ice creams with a list in
12:41:34there as well. Let's hit shift enter.
12:41:36And now we're going to start writing our
12:41:37for loop. Now the for loop is going to
12:41:39look very similar, but to call a
12:41:41dictionary, it's just a little bit
12:41:43different. So, we're going to say for
12:41:45the cream in ice cream
12:41:50dictionary dot values and then we're
12:41:53going to do parentheses and then a
12:41:54colon. Now, we're going to print the
12:41:58cream. So, in order to indicate what we
12:42:01actually want to pull, we have to
12:42:02specify within the dictionary what we
12:42:05want. Are we pulling the item? Are we
12:42:07pulling the value? We need to specify
12:42:09this. So, that's why we have this dot
12:42:11values right here. So let's run this and
12:42:13see what we get. So as you can see, we
12:42:15are pulling in the values right here.
12:42:17That's why we're pulling in Alex
12:42:18Freeberg five and mint chocolate chip/
12:42:21chocolate. Now we are able to call both
12:42:24of those, both the key and the value. So
12:42:27let's go right down here and we can do
12:42:29both the key and the value. So we can
12:42:32pull two things at one time. And we're
12:42:35going to do this by saying dot items. So
12:42:38we could also do key if we just wanted
12:42:40to do a key, but we want to do items. So
12:42:43we want to do both of them. So we're
12:42:45going to go right down here and say for
12:42:47key and value in ice cream dictionary
12:42:50items print and let's write key and then
12:42:53we'll do a comma and then let's give it
12:42:56a little arrow or something like that.
12:42:58Uh something like this and then we'll do
12:43:00a comma and we'll say value. And let's
12:43:03print this off and see what we get.
12:43:06So it's looping through and for each key
12:43:08and value it's saying here is the key.
12:43:11So that's the name. Then we have weekly
12:43:13intake. Then we have favorite ice
12:43:14creams. It's giving us a little arrow
12:43:16and then we're also printing off the
12:43:18value. So we have name Alex Freeberg.
12:43:20Weekly intake five. Favorite ice creams
12:43:23mint chocolate chip and chocolate. So
12:43:25now let's talk about nested for loops.
12:43:27We've looked at for loops. We understand
12:43:28how they work and why they do what they
12:43:30do. But what about a nested for loop? a
12:43:33for loop within a for loop. For this
12:43:35example, let's create two separate list.
12:43:38Let's create flavors.
12:43:40And let's make that a list by making it
12:43:43a bracket. We'll do vanilla, the
12:43:47classic, chocolate,
12:43:51and then cookie dough, all great
12:43:54flavors. So, that's our first list. And
12:43:57then we're going to say toppings. And
12:43:59we'll do a bracket for that as well. And
12:44:01we'll say hot fudge.
12:44:05And then we'll do Oreos.
12:44:09And then we'll do marshmallows.
12:44:12Is that how you spell marshmallows?
12:44:15I think it's an E. That looks wrong. I
12:44:18might be spelling it wrong, but that's
12:44:19okay. So, let's save this by clicking
12:44:22shift enter. And now we have our flavors
12:44:24and our toppings. So, now let's write
12:44:27our first for loop. So we're going to
12:44:28say 41 as in our number one for loop.
12:44:32We're going to say in flavors and we'll
12:44:34do a colon. We'll click enter. Now we
12:44:37can write our second for loop. So we're
12:44:39going to say for two in toppings and
12:44:43we'll do a colon and enter. And then
12:44:45we're going to say print and we'll do an
12:44:47open parenthesis. And then we're going
12:44:49to say one. So we're printing the one in
12:44:52flavors. And then we're going to say
12:44:54one, comma, we're going to say top
12:44:57topped with comm, two. So what this is
12:45:02essentially going to do is we're going
12:45:04to say for one, we're going to take the
12:45:06very first one in flavors and then we're
12:45:09going to loop through all of two as
12:45:11well. So we're going to loop through hot
12:45:13fudge, Oreos, and marshmallows. And once
12:45:17we print that off, then we will loop all
12:45:19the way back to flavors and look at the
12:45:22next iteration or the next sequence
12:45:24within the first for loop. So let's run
12:45:26this really quickly and see what we get.
12:45:29So as you can see, it goes vanilla,
12:45:32vanilla, vanilla, and vanilla is topped
12:45:34with the hot fudge, the Oreos, and the
12:45:36marshmallows. And then we start
12:45:38iterating through our second one in our
12:45:40first for loop. So there's that
12:45:41hierarchy. So we're iterating completely
12:45:43through this one before we actually go
12:45:45to the very first for loop and start
12:45:47iterating through that one again. Now
12:45:48that is essentially how a nested for
12:45:50loop works. These nested for loops can
12:45:52get very complicated. In fact, for loops
12:45:55in general can get very complicated the
12:45:57more you add to it and the more you're
12:45:59wanting to do with it. But that is
12:46:00basically how a for loop and a nested
12:46:02for loop works. Thank you guys so much
12:46:04for watching. Be sure to like and
12:46:06subscribe below and I'll see you in the
12:46:07next video.
12:46:12>> [music]
12:46:20>> Hello everybody. Today we're going to be
12:46:21taking a look at while loops in Python.
12:46:23The while loop in Python is used to
12:46:25iterate over a block of code as long as
12:46:27the test condition is true. Now the
12:46:29difference between a for loop and a
12:46:31while loop is that a for loop is going
12:46:32to iterate over the entire sequence
12:46:34regardless of a condition. But the while
12:46:36loop is only going to iterate over that
12:46:38sequence as long as a specific condition
12:46:40is met. Once that condition is not met,
12:46:42the code is going to stop and it's not
12:46:44going to iterate through the rest of the
12:46:45sequence. So if we take a look at this
12:46:46flowchart right here, we're going to
12:46:48enter this while loop and we have a test
12:46:50condition right here. The first time
12:46:52that this test condition comes back
12:46:53false, it's going to exit the while
12:46:55loop. So let's start actually writing
12:46:56out the code and see how this while loop
12:46:58works. So let's create a variable. We're
12:47:00just going to say number is equal to
12:47:02one. And then we'll say while. And now
12:47:04we need to write our condition that
12:47:05needs to be met in order for our block
12:47:07of code beneath this to run. So we're
12:47:09going to say while number is less than
12:47:12five. And then we'll do colon enter. And
12:47:15now this is our block of code. We're
12:47:17going to say print and then we'll say
12:47:18number. Now what we need to do is
12:47:20basically create a counter. We're going
12:47:22to say number equals number plus one. If
12:47:26you've never done something like this,
12:47:27it's kind of like a counter. Most people
12:47:28will start it at zero. In fact, let's
12:47:30start it at zero. And then each time it
12:47:32runs through this while loop, it's going
12:47:34to add one to this number up here. And
12:47:36then it's going to become a 1, a 2, a
12:47:38three each time it iterates through this
12:47:40while loop. Now once this number is no
12:47:42longer less than five, it'll break out
12:47:45of the while loop and it will no longer
12:47:47run. So let's run this really quick by
12:47:49hitting shift enter. So it starts at
12:47:51zero and it's going to say while the
12:47:53number is less than five, print number.
12:47:55So the first time that it runs through
12:47:57it is zero and so it prints zero and
12:48:00then it adds one to number and then it
12:48:03continues that y loop right here and it
12:48:05keeps looping through this portion. It
12:48:06never goes back up here to this line of
12:48:08code. This is just our variable that we
12:48:10start with. And then once this condition
12:48:12is no longer met once it is false then
12:48:15it's going to break out of that code.
12:48:17Now that we basically know how a while
12:48:18loop works let's look at something
12:48:20called a break statement. So let's copy
12:48:22this right down here. And what we're
12:48:24going to say is if number is equal to
12:48:28three, we're going to break. Now with
12:48:30the break statement, we can basically
12:48:32stop the loop even if the while
12:48:34condition is true. So while this number
12:48:36is less than five, it's going to
12:48:37continue to loop through. But now we
12:48:40have this break statement. So it's going
12:48:41to say if the number equals three, we're
12:48:43going to break out of this while loop.
12:48:45But if this is false, we're going to
12:48:47continue adding to that number just like
12:48:49normal. So let's execute this. So, as
12:48:51you can see, it only went to three
12:48:52instead of four like before because each
12:48:55time it was running through this while
12:48:57loop, it was checking if the number was
12:48:58equal to three. And once it got to
12:49:00three, this became true. And then we
12:49:02broke out of this while loop. The next
12:49:04thing that I want to look at, and we'll
12:49:05copy this right down here, is an else
12:49:08statement, much like an if statement.
12:49:10But we can use the else statement with a
12:49:11while loop, which runs the block of
12:49:13code, and when that condition is no
12:49:15longer true, then it activates the else
12:49:17statement. So, we'll go right down here
12:49:19and we'll say else and we'll do a colon
12:49:22and enter. And then we'll say print and
12:49:25we'll say no longer
12:49:29less than five. Now, because this if
12:49:31statement is still in there, it will
12:49:32break. So, let's say six. And then we'll
12:49:36run this. And so, it's going to iterate
12:49:37through this block of code. And once
12:49:39this statement is no longer true, once
12:49:41we break out of it, we're going to go to
12:49:43our else statement. Now, as long as this
12:49:45statement is true, it's going to
12:49:46continue to iterate through. But once
12:49:48this condition is not met, then it will
12:49:50go to our else statement and we'll run
12:49:52that line of code. Now, the else
12:49:53statement is only going to trigger if
12:49:55the while loop no longer is true. If we
12:49:58have something like this if statement
12:49:59that causes it to break out of the while
12:50:01loop, the else statement will no longer
12:50:03work. So, let's say if the number is
12:50:05three and we run this, the else
12:50:07statement is no longer going to trigger.
12:50:09So, this body of code will not be run.
12:50:11Now, the next thing that I want to look
12:50:12at is the continue statement. If the
12:50:13continue statement is triggered, it
12:50:15basically rejects all remaining
12:50:17statements in the current iteration of
12:50:18the loop and then we'll go to the next
12:50:20iteration. Now to demonstrate this, I'm
12:50:22going to change this break into a
12:50:24continue. So before when we had the
12:50:26break, if the number was equal to three,
12:50:28it would stop all the code completely.
12:50:31But when we change this to continue,
12:50:33which we'll do right now, what it's
12:50:35going to do is it's no longer going to
12:50:36run through any of the subsequent code
12:50:38in this block of code. It's just going
12:50:40to go straight up to the beginning and
12:50:42restart our while loop. So, what's going
12:50:44to happen when we run this is it's going
12:50:46to come to three. It's going to become
12:50:48three and it's going to continue back
12:50:49into the while loop, but it's never
12:50:51going to have that number change to be
12:50:53added to one to continue with the while
12:50:55loop. This will basically create an
12:50:57infinite loop. Let's try this really
12:50:59quickly. And as you can see, it's going
12:51:01to stay three forever. Eventually, this
12:51:03would time out, but I'm just going to
12:51:05stop the code really quick. So if we
12:51:06just change up the order of which we're
12:51:09doing things, we're going to say there
12:51:12and we're going to put this down here.
12:51:14So what it's going to do now, instead of
12:51:16printing the number immediately and then
12:51:18adding the number later, we're going to
12:51:20add the number right away and then we're
12:51:22going to say if it is three, we're going
12:51:24to continue and it's going to print the
12:51:25number. So let's try executing this and
12:51:27see what happens. So as you can see, we
12:51:29no longer have the three in our output.
12:51:31What it did was when we got to the
12:51:33number three, it continued and didn't
12:51:35execute this right here, which prints
12:51:37off that number. So, that really is the
12:51:39basics of the while loop. I hope that
12:51:41this was helpful. I hope that you
12:51:42learned something in this video. If you
12:51:44did, be sure to like and subscribe
12:51:45below, and I'll see you in the next
12:51:47video.
12:51:59Hello everybody. Today we're going to be
12:52:01taking a look at functions in Python. A
12:52:03function is a block of code which is
12:52:05only run when you call it. So right here
12:52:07we're defining our function and then
12:52:09this is our body of code that when we
12:52:11actually call it is going to be ran. So
12:52:14right here we have our function call and
12:52:15all we're doing is putting the function
12:52:17with the parenthesis. That is basically
12:52:19us calling that function and then we
12:52:21have our output. Throughout this video,
12:52:23I'm going to show you how to write a
12:52:24function as well as pass arguments to
12:52:26that function and then a few other
12:52:27things like arbitrary arguments, keyword
12:52:30arguments, and arbitrary keyword
12:52:32arguments. All of these things are
12:52:33really important to know when you are
12:52:34using functions. So, let's get started
12:52:36by writing our very first function
12:52:38together. We're going to start off by
12:52:39saying def. That is the keyword for
12:52:41defining a function. Then, we can
12:52:44actually name our function. And for this
12:52:45one, we're just going to do first
12:52:48function. And then, we do an open
12:52:50parenthesis. and then we'll put a colon.
12:52:52We'll hit enter and it'll automatically
12:52:54indent for us. And this is where our
12:52:56body of code is going to go. Now, within
12:52:57our body of code, we can write just
12:52:59about anything. And in this video, I'm
12:53:00not going to get super advanced. We're
12:53:02just going to walk through the basics to
12:53:03make sure that you understand how to use
12:53:05functions. So, for right now, all we're
12:53:07going to say is print. We'll do an open
12:53:09parenthesis. We'll do an apostrophe, and
12:53:11we'll say we did it. And now, we're
12:53:14going to hit shift enter. And this is
12:53:16not going to do anything. At least you
12:53:18won't see any output from this. If we
12:53:20want to see the output or we actually
12:53:21want to run that function and some
12:53:23functions don't have outputs, but if we
12:53:25want to run that function, what we have
12:53:27to do is just copy this and put it right
12:53:29down here. And now we're going to
12:53:31actually call our function. So let's go
12:53:33ahead and click shift enter. And now
12:53:35we've successfully called our first
12:53:37function. This function is about as
12:53:38simple as it could possibly be. But now
12:53:40let's take it up a notch and start
12:53:42looking at arguments. So, let's go right
12:53:44down here and we're going to say define
12:53:47number
12:53:49squared. We'll do a parenthesis and our
12:53:52colon as well. Now, really quickly, when
12:53:54you're naming your function, it's kind
12:53:55of like naming a variable. You can use
12:53:57something like x or y, but I tend to
12:53:59like to be a little bit more
12:54:00descriptive. But now, let's take a look
12:54:02at passing an argument into a function.
12:54:04The argument is going to be passed right
12:54:06here in the parenthesis. So, for us, I'm
12:54:09just going to call it a number. And
12:54:11then, we're going to hit enter. And now
12:54:13we'll write our body of code. And all
12:54:14we're going to do for this is type print
12:54:16and open parenthesis. And we'll say
12:54:18number and we'll do two stars. At least
12:54:21that's what I call it, a star. And a
12:54:23two. And what this is going to do is
12:54:24it's going to take the number that we
12:54:26pass into our function. It's going to
12:54:28put it right here in our body of code.
12:54:30And then for what we're doing, it's
12:54:32going to put it to the power of two. And
12:54:33so when the user or you run this and
12:54:36call this function, this number is
12:54:38something that you can specify. It's an
12:54:40argument that you can input that will
12:54:42then be run in this body of code. So
12:54:44let's copy this right here and then
12:54:47we'll put it right down here into this
12:54:49next cell and we'll say five. And so
12:54:52this five is going to be passed through
12:54:53into this function and be called right
12:54:56here for this print statement. Let's run
12:54:58it and it should come out as I believe
12:55:0025. That is my fault. I forgot to
12:55:02actually run this block of code. So I'm
12:55:04going to hit shift enter. So now we've
12:55:06defined our function up here. And now we
12:55:08can actually call it. So now we'll hit
12:55:10shift enter and we got our output of 25.
12:55:13Now in this function we only called one
12:55:15argument but you can basically call as
12:55:17many arguments as you want. You just
12:55:19have to separate them by commas. So
12:55:21let's copy this and we'll put it right
12:55:24down here. Now we'll say number squared
12:55:28custom and then we'll do number and then
12:55:31we'll do power. So now we can specify
12:55:35our number as well as the power that we
12:55:37want to raise it to. So instead of
12:55:38having two, which is what you call
12:55:40hard-coded, we can now customize that
12:55:42and we'll have power. And now when we
12:55:45call this function, we can specify the
12:55:47number and the power and both of those
12:55:49will go into this body of code and be
12:55:51run. And we can customize those numbers.
12:55:53So let's copy this
12:55:56and we'll say
12:55:585 to the power of three. And let's make
12:56:02sure I ran this. So let's do shift
12:56:04enter. And now we will call our
12:56:06function. And let's hit shift enter. And
12:56:08we got 5 to the power of three, which is
12:56:11125. And just one last thing to mention
12:56:13is if you have two arguments within your
12:56:16function and you are calling it right
12:56:18here, you have to pass in two arguments.
12:56:20You can't just have one. So if we have a
12:56:22five right here, it's going to error
12:56:23out. We have to specify both arguments
12:56:27for it to work. Now let's take a look at
12:56:30arbitrary arguments. Now, arbitrary
12:56:33arguments are really interesting because
12:56:35if you don't know how many arguments you
12:56:37want to pass through, if you don't know
12:56:38if it's a one, a two, or a three, you
12:56:40can specify that later when you're
12:56:42calling the argument. So, you don't have
12:56:44to do it up front and know that
12:56:45information ahead of time. So, let's
12:56:47define our function. So, we're going to
12:56:48say define and then we're going to say
12:56:50number_s
12:56:53and we'll do an open parenthesis and a
12:56:55colon. Now within our argument right
12:56:58here, typically we would just specify
12:57:00here's what our argument will be. It
12:57:02will be number or it will be a word,
12:57:04right? But what we're going to do is
12:57:05something called an arbitrary argument.
12:57:07So it's unknown. So we're going to put
12:57:09star and we'll say args. Now you will
12:57:12see something exactly like this.
12:57:14Typically if you're looking at tutorials
12:57:15that'll have star args in there or if
12:57:17you're looking at just a generic piece
12:57:19of code, this is what it will look like.
12:57:21But for us, we're going to actually put
12:57:23number. So again, we have the star and
12:57:25then we have our arbitrary argument
12:57:28right here. And then we'll hit enter and
12:57:30we're going to say print open
12:57:32parenthesis. And this is where it's
12:57:34going to get a little bit different. So
12:57:35we're going to say number and then we're
12:57:37going to do an open bracket and let's
12:57:38say zero and then we'll do that times
12:57:42and then we'll say number again with a
12:57:45bracket of one. So in a little bit once
12:57:47we run this and then we call this number
12:57:49args function right here we're going to
12:57:51need to specify the number zero and the
12:57:54number one that's going to be called. So
12:57:56let's go ahead and run this and then we
12:57:58are going to call it and let's say 5a 6
12:58:04comma 1 2 8. So right up here we did not
12:58:08know how many arguments we were going to
12:58:10pass through. It could be five it could
12:58:12be a thousand. We could also call in a
12:58:15tupole and that's what this is right
12:58:16here. We're calling in a tupole. So what
12:58:19it's going to do now is when it calls
12:58:20this number, it's going to call the very
12:58:22first within that tupole which will be
12:58:23that five. And then it'll also call in
12:58:25this number which will be the first
12:58:27position which is the six. So let's hit
12:58:30shift enter and it's going to multiply
12:58:32these numbers together. So 5 * 6 is
12:58:34equal to 30. Now like I just said this
12:58:37is a tupole. So we don't actually have
12:58:38to write out these numbers like we just
12:58:40did. we can pass through a tupil when we
12:58:43are actually calling this function.
12:58:45Let's do that right up here. Let's just
12:58:47create um let's call it args tupil and
12:58:51we'll do open parentheses and we'll do
12:58:54the same numbers. Let's just copy it to
12:58:57make it easier.
12:58:59And now we've created this tupil right
12:59:01here which we can then pass in. And this
12:59:03is a lot more handy, a lot more
12:59:05specific. And this is most likely how
12:59:07someone would do something like this.
12:59:09But let's now create this.
12:59:12And now we can copy args tupole and pass
12:59:15it through. Now, really quickly, this is
12:59:18going to fail. And I'm doing that on
12:59:19purpose, but I want to show you what you
12:59:20need to do in order to pass through this
12:59:22tupole. So, right now, it's going to say
12:59:25tupil index is out of range. All you
12:59:28have to do in order to use this is you
12:59:30have to specify a star before it just
12:59:32like you did when you were creating your
12:59:34argument up here. we have to put a star
12:59:36in front of our tupil that we just
12:59:38passed through. And now let's try
12:59:39running this. And now it works properly.
12:59:42Now the last two things that we're going
12:59:43to look at are keyword arguments and
12:59:45arbitrary keyword arguments. There are
12:59:47more things that you can learn and do
12:59:49within functions, but again I'm just
12:59:51trying to teach you the basics to make
12:59:52sure that you understand how they work.
12:59:54So let's go right up here. And a keyword
12:59:56argument is kind of similar to this
12:59:58right here. And let's actually copy this
13:00:01and put it right down here. Now, a
13:00:04keyword argument is very similar in that
13:00:06you're going to specify your arguments
13:00:08right here. But what we did up here, let
13:00:11me bring this down. When we actually
13:00:14called the function, what we did was we
13:00:17just put in a five and a three. And when
13:00:19we did that, it automatically assigned
13:00:21number to five and power to three. And
13:00:24that's totally fine and you can do that.
13:00:26But if you want a little bit more
13:00:28control, you can use a keyword argument.
13:00:30So right here we could say power is
13:00:34equal to five and number is equal to
13:00:39three. So I just switched it around,
13:00:40right? Number was assigned to five and
13:00:42power was assigned to three, but I just
13:00:44switched it to show you how this might
13:00:46work. So let's run both of these. And
13:00:49now it's 3 to the power of five, which
13:00:51is 243.
13:00:53So that essentially is a keyword
13:00:54argument. Again, it just gives you a
13:00:56little bit more control. you don't have
13:00:58to put them in specific positions like
13:01:00if you're just calling multiple
13:01:01arguments. Now let's come right down
13:01:03here. We're going to create basically
13:01:04another custom function. Uh so for this
13:01:07one we're going to write define number_g
13:01:12and then we'll do an open parenthesis a
13:01:14colon and enter. And what this one is is
13:01:17this one is a keyword argument or an
13:01:20arbitrary keyword argument. Now, to
13:01:22specify an arbitrary argument, all we
13:01:24did was a star and then we input number.
13:01:28But if we're doing a keyword argument,
13:01:30we actually have to have two stars right
13:01:32here. So, let's start taking a look. And
13:01:34again, if you're doing arbitrary, it
13:01:36means we don't really know how many
13:01:38keyword arguments we want to pass into
13:01:40our function. So, we're just going to
13:01:42put starst star number. And then later
13:01:44within our body of code, and when we're
13:01:45calling it, we'll be able to specify it.
13:01:48And just like the arbitrary argument
13:01:50before, the arbitrary keyword argument
13:01:52means we really just don't know how many
13:01:54keyword arguments we're going to need to
13:01:55pass into our function. So to
13:01:57demonstrate this, let's write print do
13:02:00an open parenthesis and we'll say my
13:02:02oops need to do an apostrophe.
13:02:05My number is we'll do just like that
13:02:10little space and we'll say plus. And
13:02:11this is kind of where it gets a little
13:02:13interesting or a little bit more tricky.
13:02:15So we're going to say is number. So this
13:02:17is us calling our number and then we're
13:02:19going to do a bracket and then I'm
13:02:22actually going to go to calling the
13:02:24function. It's a little bit backward or
13:02:26a little bit different than what you
13:02:28might think. But when we're calling it,
13:02:29what I'm going to do is I'm going to say
13:02:31integer
13:02:33is equal to let's just do some random
13:02:35number. Now when we're calling that
13:02:37keyword within our body of code, what
13:02:39we're going to do is we're going to
13:02:40actually type out integer just like
13:02:43this. And this looks a little bit
13:02:46different, but what this allows us to do
13:02:48is we can put as many keyword arguments
13:02:50in here as we want later, and I'll show
13:02:52you in just a second. But for us, we're
13:02:53just creating this key and this value
13:02:56when we are calling it within the
13:02:58function. So now when we create this and
13:03:00we run this,
13:03:02oh, whoops, I forgot this has to be a
13:03:04string. Um, so let's run this again.
13:03:08Now we'll say my number is 2309.
13:03:12Then we're going to add, we'll say plus,
13:03:15and this isn't going to look great, but
13:03:16we'll say my other number because this
13:03:19will all be in the same line. That's
13:03:20okay. My other number. And then we'll
13:03:23say number. And we can specify again
13:03:26what we want in there. So now we can go
13:03:29down here to where we're calling it.
13:03:31We'll just put a comma. And we'll say
13:03:34integer oops, integer
13:03:382 is equal to, and we'll do a random
13:03:40number. And then we'll put integer two
13:03:43right here. And then we'll add plus
13:03:46right here so we don't error out. We'll
13:03:48create this. We'll run this. And as you
13:03:51can see, both numbers were passed
13:03:53through. Again, the syntax is terrible.
13:03:55But now you can see that you have this
13:03:56arbitrary keyword argument right here.
13:03:59And all we have to do is put number
13:04:02number. And we can pass through as many
13:04:03of these arbitrary keyword arguments as
13:04:05we want as long as we just specify it
13:04:08within our function when we're calling
13:04:09it. So, that's all we're going to look
13:04:11at in today's video on functions. There
13:04:13are, of course, other things that you
13:04:14can do within functions, and it can get
13:04:15a little bit more advanced, but I wanted
13:04:17to show you the basics, the meat and
13:04:19potatoes of things that I definitely
13:04:20think you should know in order to get
13:04:22started using functions. I hope that you
13:04:24were able to understand functions better
13:04:25because of this video. If you did, be
13:04:27sure to like and subscribe below,
13:04:28[music] and I will see you in the next
13:04:30video.
13:04:42Hello everybody. Today we're going to be
13:04:44talking about converting data types in
13:04:46Python. In this video I'm going to show
13:04:47you how to convert several different
13:04:49data types including strings, numbers,
13:04:51sets, tupils, and even dictionaries. So
13:04:54let's start off by creating a variable.
13:04:55We'll say num_int is equal to 7. And we
13:04:59can check that data type by saying type
13:05:02and then inserting our variable num int.
13:05:06And that will tell us that our data type
13:05:08for this variable is an integer. Let's
13:05:10go ahead and create another one. We're
13:05:12going to say num string is equal to. And
13:05:15for this one, we'll also do a seven. But
13:05:18let's check the type. And we'll do an
13:05:20open parenthesis. We'll say the type of
13:05:22num string. And that one is a string.
13:05:25Now let's say we wanted to add those.
13:05:27We'll say num
13:05:29sum. So the sum of num intint plus num
13:05:35string. Now when we're adding these two
13:05:37values, it is not going to work. It's
13:05:39going to give us an error and it's going
13:05:41to say unsupported operand for int and
13:05:44string. So it cannot add both an integer
13:05:46and a string. What we need to do in
13:05:48order to add these two numbers is to
13:05:50convert that string into an integer. So
13:05:53let's go right up here. Let's add
13:05:55another cell and let's say
13:05:58num_string_converted [clears throat]
13:06:02is equal to and we want to convert it
13:06:04into an integer. So all we have to do to
13:06:07convert it into an integer is type int
13:06:10and then we're going to say num
13:06:13string. And that is as easy as it's
13:06:16going to get. All we have to do is say
13:06:18integer with our num string inside of
13:06:21it. And then it's going to convert it.
13:06:23And we can even check it right after by
13:06:25saying type numstring converted. And
13:06:28let's run this. And now we can see that
13:06:30it was converted into an integer. So now
13:06:32let's add that numstring converted right
13:06:35here.
13:06:37Let's copy and replace that string with
13:06:39the string converted.
13:06:41And let's actually print out that num
13:06:45sum. And it worked properly. Now, we did
13:06:49not specify what type of value this
13:06:51numsum was going to be. But because
13:06:55it was two integers in here, it's going
13:06:57to automatically apply that data type of
13:06:59integer to that num sum. Let's go right
13:07:01down here. And now let's look at how we
13:07:04can convert lists, sets, and tupils. So
13:07:07now let's say we have a list type, and
13:07:10that's equal to 1 2 3. And we can check
13:07:14it again by saying type
13:07:17and that is a list. Let's say we want to
13:07:20convert it to a tupil. It's fairly easy.
13:07:23All we're going to do is write tupil say
13:07:26list type. That list type is now going
13:07:29to be a tupil. And we can check that by
13:07:32saying type and wrapping it around this
13:07:35tupil. And it shows us that it is
13:07:38converting that list into a tupole. Now
13:07:41we can also convert a list into a set.
13:07:43But it may change the actual values
13:07:47within it. Let's check that out really
13:07:49quickly. So let's say we have this list
13:07:51and let's add a few more values to this.
13:07:55Just like that. Now let's say we want to
13:07:57convert it to a set. So we're going to
13:07:59run this and we'll say set
13:08:03of list type. And let's try running this
13:08:06and see what the output is. So this is
13:08:08something that you really need to be
13:08:10aware of when you are converting data
13:08:12types because set does not act the same
13:08:14as a list. A set is basically going to
13:08:16take the unique values in the list and
13:08:18convert it to a set and it fundamentally
13:08:20changes the data that was in that
13:08:22original list. And just to check the
13:08:24data type, we can say type.
13:08:27I'm just doing this for all of them. And
13:08:29as you can see, that is now a set. Now
13:08:31let's go down here and take a look at
13:08:32dictionaries. Now, let's say we have a
13:08:36dictionary called dictionary type and
13:08:39we'll do a squiggly bracket and we'll
13:08:42say name and we'll do a colon and we'll
13:08:45say Alex. Then we'll do age and a colon
13:08:50and we'll say 28
13:08:53and then we'll do hair
13:08:57colon.
13:08:59And so really quickly, let's take that
13:09:01dictionary type and just confirm that it
13:09:04is a dictionary. And it is. And now what
13:09:07we're going to do is take a look at all
13:09:09of the items within that dictionary. So
13:09:12we're going to do dictionary type do
13:09:14items open parenthesis. And this is
13:09:17going to show us all the items within
13:09:19it. Now we can also take this and look
13:09:21at something like the values.
13:09:25And when we run that, these are our
13:09:27values. So within our dictionary we have
13:09:29items and that's what this is right
13:09:31here. This is one item. And then within
13:09:34that we have our values which are right
13:09:36here. So Alex, 28 and NA. And then we
13:09:39have something called a key. And this is
13:09:42the key. The name, age, and hair are all
13:09:45keys. And we can look at that by saying
13:09:49keys. So let's say we want to take all
13:09:51of the keys and put that into a list.
13:09:54What we're going to do is we're going to
13:09:55take this right here. say list.
13:09:58We'll do an open parenthesis. We'll type
13:10:00that in right there. So, it says a list
13:10:02and we're converting these keys into a
13:10:04list. And let's run that. And now this
13:10:07is a list. And let's just check the type
13:10:10as well just to confirm.
13:10:13And as you can see, it was converted
13:10:14properly into a list. And we can do the
13:10:17exact same thing with values.
13:10:22And the values can also be converted
13:10:24into a list. Now, we can also convert
13:10:26longer strings that aren't just numbers
13:10:28like we did above in our very first
13:10:29example. So, let's do long string and
13:10:33we'll say I like to party. Now, we're
13:10:37going to take this string and we're
13:10:39going to say list long string. So, we're
13:10:43going to convert this string into a
13:10:45list. And let's see what happens. So, it
13:10:47took every single character in that
13:10:49string and put it into a list. And we
13:10:51could also do a set as well. That one's
13:10:54a lot shorter because it's only looking
13:10:55at unique values. So, that is how you
13:10:58convert data types in Python. Thank you
13:11:00guys so much for watching. I really
13:11:01appreciate it. If you like this video,
13:11:03be sure to like and subscribe below and
13:11:04I'll see you in the next video.
13:11:07[music]
13:11:18Hello everybody. Today we're going to be
13:11:20working on building a BMI calculator in
13:11:22Python. Now, before we get started, I
13:11:24want to show you this BMI calculator
13:11:25that I found online. And it shows you
13:11:27the basic calculation that they use. And
13:11:29that's the one we're going to use in
13:11:30this video. And they also have this
13:11:32calculator right down here. And some
13:11:34ranges that we can use for our
13:11:36calculator as well. So, for reference, I
13:11:38weigh about 170.
13:11:41I'm about 5'9. Let's calculate this. So,
13:11:44I'm about a 25.1 BMI, which falls into
13:11:48the overweight category. That's
13:11:50unfortunate, but we can see exactly how
13:11:53this works and how ours should work when
13:11:55we actually build it. So, we're going to
13:11:57kind of reference this throughout the
13:11:58video. So, let's go right over here to
13:12:01our BMI calculator. We need to calculate
13:12:03weight and height and then run this
13:12:06calculation right here. So, let's go
13:12:07ahead and copy this
13:12:10and we're going to put it right down
13:12:11here.
13:12:14And so, now we have our calculation. So
13:12:17what we need is we need input from a
13:12:20user and there is an input function
13:12:22within Python that we're going to be
13:12:24using. So let's actually give me a few
13:12:26more cells. So the first thing that we
13:12:28need to calculate is their weight. So
13:12:30let's type out weight right here. We'll
13:12:32say weight is equal to and this is where
13:12:33we'll use our input function. So we'll
13:12:35say input and when we actually run this
13:12:38it's just going to give us this blank
13:12:39square or a user can input something.
13:12:42We'll say Alex. So this is our output is
13:12:45what the actual user input and it does
13:12:47save it to this variable. So if we say
13:12:50print weight, it will still print out
13:12:53Alex. Now this is where we want the user
13:12:55to just like we did before where they'll
13:12:58input their weight. So we want to kind
13:13:00of give them a prompt for this. We'll
13:13:02put a string in here. So I'll do a
13:13:04double quote and then I'll say enter
13:13:08your weight in and we're using pounds.
13:13:12Let's say pounds
13:13:14colon space. So now when we do this,
13:13:17it'll say enter your weight in pounds.
13:13:19I'll say 170. And then when we run this,
13:13:22it does store that. Now let's do print.
13:13:24I should have saved it. Wait again.
13:13:27Oops. Now it's only storing the value of
13:13:30170. It's not actually storing this
13:13:32string right here. So that's really
13:13:33important for when we do our
13:13:34calculations later. Um I'm going to I'm
13:13:38going to save this right down here
13:13:39because I'm sure I'm going to use that
13:13:40later. Um, so we have that as working.
13:13:44Now, we need to also do our height. So,
13:13:46let's copy this. And we'll put it right
13:13:49here. And we'll do height
13:13:53and enter your height in inches. So, now
13:13:56for this one, if we hit enter,
13:14:00it's actually running. Let's stop it
13:14:01really quick and interrupt it. Let's try
13:14:04running this. So, it's going to say,
13:14:06enter your weight in pounds. That's the
13:14:07first input. Say 170.
13:14:11And then when I hit enter, it's going to
13:14:13prompt me for that second input. And so
13:14:15in inches, 59 is 69 in. And then I can
13:14:20hit enter again. And now we have both of
13:14:24our inputs. Now we need this calculation
13:14:26right down here. And just like that. So
13:14:31now we have weight in pounds* 703
13:14:34divided by height in inches by height in
13:14:37inches. So we actually have weight and
13:14:39it's already written in there but I'm
13:14:41just going to do it like this. We'll do
13:14:42weight time 703. So that's pounds there.
13:14:46Our weight in pounds time 703 divided by
13:14:49now we have our height in inches
13:14:52times the height in inches. So this is
13:14:55our calculation right here. So let's do
13:14:58this exact same thing. Let's run this.
13:15:01And this times of course is not going to
13:15:03work. Whoops. We need to do our star for
13:15:06both of these. All right. Now, this is
13:15:08our calculation. So, let's run this. So,
13:15:11we have 170 and that's pounds and inches
13:15:15was 69. Hit enter.
13:15:19And it says cannot multiply the sequence
13:15:21of non- integer type of string. Ah,
13:15:23that's because these are being stored in
13:15:25strings. So, if right down here I do and
13:15:28we'll do type of height and we run that.
13:15:33This is actually a string. So, we want
13:15:36to change that because we don't need
13:15:37that anymore. Get rid of that.
13:15:40So, we don't want it to be a string. We
13:15:42need those to be integers or floats or
13:15:45really anything besides a string. It
13:15:47just needs to be numerical. Uh, so
13:15:49integer float really. So, let's do
13:15:50integer. And then we'll wrap that input
13:15:52in it. And we'll do the same thing for
13:15:55this one.
13:15:57Now, we have an integer for our weight,
13:15:59an integer for our height. So now when
13:16:01we're running this calculation, it
13:16:03should work properly. Let's run this
13:16:05again. Our pounds are 70.
13:16:08Our height is 69 in.
13:16:13And it's not giving us our output
13:16:15because we're not printing anything.
13:16:16Okay. So I just need to do
13:16:19print
13:16:21BMI. So let's try this again. 170 69.
13:16:26And there is our BMI 25.1. So it worked
13:16:29the exact same as this one. So they
13:16:32input well we input our height, we
13:16:35inputed our or we inputed our weight, we
13:16:36inputed our height, and then it
13:16:38calculated our BMI. The next thing that
13:16:40we need to do is we need to kind of give
13:16:43the user some context. Is that good? Is
13:16:46there BMI in within a good range? A bad
13:16:48range? We don't know. Uh so let's go
13:16:50ahead and I'm going to see if I can copy
13:16:53this. Know if this will work or not.
13:16:56Let's go ahead and copy this right down
13:16:57here. Perfect. So what we now need to do
13:17:00is we need to say okay if the user has
13:17:04given us this input we want to give them
13:17:06or tell them if they are a normal
13:17:09weight, overweight, obese, severely
13:17:12obese, anything like that. And we have
13:17:13these ranges. So that should help us out
13:17:16quite a bit. So let's just write our if
13:17:18statement and then we'll include it up
13:17:20here. But let's go down here and we'll
13:17:23say if and then we'll do BMI and let's
13:17:26just say BMI is greater than zero. So if
13:17:31it's greater than zero if they had any
13:17:33input where the BMI was not zero which
13:17:36should be every time if they do it
13:17:37properly and they don't you know put a
13:17:39string in there or something or type out
13:17:4140 which maybe we should make a prompt
13:17:43for that if that happens. Then we can
13:17:45say if we'll do BMI
13:17:49and now we need to give that first
13:17:50range. So this range right here. So if
13:17:52it's under 18.5 so we need to do a less
13:17:56than. So if it's less than 18.5
13:18:00and it just says under it doesn't say
13:18:02under or equal to. So I'll keep it at
13:18:0418.5. So if it's under 18.5
13:18:08then let's give kind of the output.
13:18:10We'll say print
13:18:12and the output or the basically the
13:18:15prompt is underweight. So we'll just say
13:18:18you are under
13:18:22under case underweight and just like
13:18:25that. Um
13:18:27then we're going to pass several LF
13:18:30statements through here. But let's just
13:18:32say else. So I guess this would be like
13:18:36if they are if they don't input
13:18:39something properly or something messes
13:18:41up maybe we could write something like
13:18:44um print oops
13:18:47I'm thinking all this through. We can
13:18:49write print enter valid inputs
13:18:54or something like this or we can always
13:18:57change that. But let's really quickly
13:19:00let's run this.
13:19:02Okay. So, I'm not in that range. Uh,
13:19:04let's make the next one. So, then I can
13:19:07be within a certain range. Oops.
13:19:10And we need we should need one more
13:19:12minimum. So, we'll say LF
13:19:15and LF.
13:19:18These next two are this 24.9. So, it's
13:19:22going to check this one first. So, if
13:19:23it's 18.5 or below 18.5, it's
13:19:27automatically going to print this one.
13:19:29So this next one, we don't have to do
13:19:30like a range or anything. We can just
13:19:33say if it's below if it's between 25 and
13:19:3729.9. So this one actually should be
13:19:40less than or equal to. Um, this one is
13:19:44normal. Oh, whoops. 24.9.
13:19:47So this one is 24.9.
13:19:50This one is going to say you are normal
13:19:53weight. So let's run this now.
13:19:58Let's see. BMI was 25.1.
13:20:02Oh, guys, I'm just messing up here. I
13:20:04apologize. All right, this is the one
13:20:06that I was part of. So, now it's going
13:20:08to be I'm part of the overweight crowd.
13:20:11Now, let's run this. And now our prompt
13:20:13is you are overweight because remember
13:20:14the BMI was saved right here as 25.1
13:20:19down here. If we run through this, it's
13:20:22saying no, you're not in Oops.
13:20:26Get rid of that. No, you're not in under
13:20:2818.5. You're not under 24.9. If you're
13:20:32under 29.9,
13:20:34you are overweight. So, that did work
13:20:36properly. So, that's really good. And I
13:20:38don't think I want this to be our output
13:20:41for the person because we're going to
13:20:42add this up here. It's just going to
13:20:43give us the BMI. And then the output is
13:20:46going to say you are overweight. Uh
13:20:48let's make it a little bit more
13:20:49customized. Um I'm going to say name is
13:20:53equal to input. And then we'll say enter
13:20:57your name.
13:21:00Um, so it'll be enter your name. We'll
13:21:02do Alex 70
13:21:0569. There's our BMI. Now it's going to
13:21:09run through this logic or it will run
13:21:10through this logic in just a second.
13:21:12When
13:21:14we actually finish this, so then we have
13:21:1734.9.
13:21:21And let's do one more.
13:21:25Oops. And then this one's going to be
13:21:27for 39.9.
13:21:31So this one was overweight. This one is
13:21:34obese.
13:21:36Severely obese. So we'll say severely is
13:21:40that how you spell it? Severely obese.
13:21:41And then anything that's over that 40
13:21:44and over. So if it's not this one,
13:21:46anything else should be se morbidly
13:21:50obese. So actually this else statement
13:21:52right here should say
13:21:55uh you are
13:21:59you are severely obese. This is going to
13:22:01say morbidly morbidly obese. Now I added
13:22:06that name up here because I wanted to
13:22:08add that down below actually. So, we're
13:22:11going to say uh name plus and then we'll
13:22:16do like comma
13:22:20you are underweight. So, it'll be a
13:22:22little bit more personalized. Uh I think
13:22:24it'll I think it'll be a nice touch. I
13:22:27really do. We'll do it like this. And
13:22:29we'll say you and let's go back and do
13:22:31that to all of them.
13:22:33And let me see how quickly I can do
13:22:35this.
13:22:38Oh, whoops. What' I do? Get rid of that.
13:22:42Name plus you. Like that. Jeez, you guys
13:22:48are seeing me mess up a ton. Name plus
13:22:51you. And then
13:22:55name plus you. So now let's run this.
13:22:58And now it's a little more personalized.
13:23:00It says Alex, you are overweight. So
13:23:03this is all really good. Now this is an
13:23:06if statement. Um, what we had done
13:23:08before I think is actually what we
13:23:09should put right down here. So, we'll
13:23:10say else and then if that doesn't work,
13:23:13we'll say, what do we say? Enter valid
13:23:16input. We'll just put that. Um, and let
13:23:19let me see if I can test this out. Don't
13:23:23I don't know if this will error out or
13:23:25if this will even work.
13:23:27Let me just see if I can mess with it
13:23:29and see if I can get it to work.
13:23:30Actually, let's copy this. We're going
13:23:34to copy this whole thing. We're going to
13:23:36include it right here.
13:23:38And now we have basically our entire
13:23:41calculator. So, um, let's run this.
13:23:45Enter your name. We'll say Alex.
13:23:48Enter your pounds, 170. Enter your
13:23:51inches, 69. And then it's going to say
13:23:5425.1.
13:23:56Alex, you are overweight. And that's
13:23:58perfect. We could even go as far as
13:24:00adding like some feedback. We say you
13:24:03are overweight. And then it would be a
13:24:05period and we could say um you need to
13:24:09exercise more stop sitting and writing
13:24:14so many Python tutorials. So now if we
13:24:18run this we'll do Alex
13:24:2117069.
13:24:23It says Alex you are overweight. You
13:24:25need to exercise more and stop sitting
13:24:26and writing so many Python tutorials.
13:24:30Period. And that's it. This is the
13:24:34entire project. Um, you can go a ton
13:24:38farther. You can include much more
13:24:40complex logic. You could even build out
13:24:42a UI to create your own, you know, app
13:24:45just like this where it has this input
13:24:47and this UI. You can build that out
13:24:49within Jupyter Notebooks with Python.
13:24:52Um, but that's not really what this
13:24:54tutorial is for. This is just to kind of
13:24:56help you um, think through some of the
13:24:58logic of creating something like this.
13:25:00So, you know, I hope that this was
13:25:02helpful. I hope that this was fun. I
13:25:03like creating stuff like this. We have
13:25:05two other projects that we're going to
13:25:06do and maybe I'll include more, but we
13:25:08have two right now that I have planned.
13:25:10Um, and I hope those are helpful. This
13:25:12is probably our easiest one and they'll
13:25:14get a little bit more difficult in the
13:25:16next projects. So, I hope that this was
13:25:18fun. I hope that this was helpful and
13:25:20that you can now kind of utilize those
13:25:22Python skills that you've been working
13:25:23on. If you like this video, be sure to
13:25:25like and subscribe below and I'll see
13:25:27you in the next video.
13:25:29[music]
13:25:40Hello everybody. Today we're going to be
13:25:42creating an automatic file sorter for
13:25:43your files in File Explorer. Now, out of
13:25:46all the projects that we've done in this
13:25:47series so far, I think this one might be
13:25:48the most difficult, but I also think
13:25:50this one is the most cool because it has
13:25:52some real life applications. So, without
13:25:54further ado, let's take a look at some
13:25:56files that we have right down here in my
13:25:58file explorer. So, I have this beautiful
13:26:00picture of Rosie uh right here. This is
13:26:03a PNG file. I have a CSV file and a text
13:26:06file. And I want to sort all of them
13:26:09into their own folders depending on what
13:26:11kind of file it is. So, if I go right in
13:26:14here and I click on this one, I go to
13:26:16properties, I can see that this is a PNG
13:26:19file. Um, if I go into this one, I don't
13:26:21need to, but if I go into this one, it's
13:26:22a CSV file. And of course, this one is a
13:26:25text file. So, I want three separate
13:26:28folders in here, and I want them to
13:26:31automatically go into those folders
13:26:33without me having to drag and drop and
13:26:35going and clicking. Now, we only have
13:26:37four files here, but imagine if we have
13:26:40thousands of files, how much time that
13:26:42could save us. So, let's get out of here
13:26:45and let's start writing our code. So
13:26:48we're going to say import OS,
13:26:52and then we're going to say shutil.
13:26:55Now OS obviously stands for operating
13:26:57system. Shutil, uh, I don't know what it
13:27:00actually supposed to stand for, but what
13:27:02it will allow us to do is do some
13:27:03high-level operations on our files in
13:27:06file explorer. So we're going to go
13:27:07ahead and import those. And now that we
13:27:10have those imported, uh, something
13:27:11that's going to be very important for us
13:27:13to have throughout this whole thing, and
13:27:15this is anytime I'm working with like
13:27:16directories or something like this, we
13:27:18want to get this path down. So, I'm
13:27:20going to go ahead and copy this path.
13:27:23And we're just going to say path is
13:27:25equal to, and we'll do this right here.
13:27:28So, let's run this. And I need to put an
13:27:31R right here to make this a raw text.
13:27:34Um, so when you don't have the R, uh,
13:27:36it's going to read in these, you know,
13:27:38these backslashes and these colons and
13:27:40different stuff. If we do R, it's just
13:27:41going to read it in as the raw string
13:27:43and that's what we want. So, here's what
13:27:45we need to do there. There's a few
13:27:47different things that have to happen
13:27:48when we are writing this out. One thing
13:27:50is is we need to go in here and we need
13:27:52to see this path and we need to see are
13:27:54there folders in here already? Um, if
13:27:56not, we need to create a folder. So,
13:27:59that's one of the first things that we
13:28:01need to do. The next thing that we need
13:28:03is it needs to check each of these files
13:28:05individually, identify what kind of file
13:28:08it is, and then put it into the correct
13:28:11folder. So, we have to create the
13:28:12folder, then check these, and then place
13:28:15it into the correct folder. So, let's go
13:28:18right out of here. So, what we're going
13:28:20to start doing is we're going to start
13:28:22working with these paths and these
13:28:24directories. And some of these things
13:28:25you may never have seen before, but
13:28:27that's okay. I'll try to explain it as I
13:28:28go through. So the first thing that
13:28:30we're going to write is os.list
13:28:33directories. Uh and what this is
13:28:35actually going to do is show us all the
13:28:36files in there. So we're going to say
13:28:38path. So it should show us all the files
13:28:41within path. And so here are our
13:28:43results. So we have the data
13:28:45professional results, fake text file,
13:28:48our image, and our other image. So this
13:28:50is actually showing us what files are in
13:28:52that path. And that's super important
13:28:54because we're probably going to have to
13:28:56loop through this in some way later. Um,
13:28:58I wrote this all out before, so I kind
13:29:01of remember, but I'm doing this all off
13:29:02the top of my head. So, I guarantee you
13:29:04throughout this I'll make some mistakes.
13:29:06But what we now need to do is we need to
13:29:09create folders or check if there's a
13:29:11folder and create it if it isn't there.
13:29:12That's um the next step that we need to
13:29:14take. So, let's go right down here and
13:29:17we want to check if this path exists
13:29:19already. So, if that folder already
13:29:21exists. So, we're going to say
13:29:22os.path.exists.
13:29:26So this is going to check does this path
13:29:28just like this path up here does it
13:29:30already exist and then we're going to do
13:29:32an open parenthesis. We'll say path. So
13:29:34that's our path. Now we need to add a
13:29:37folder name to this. Um we could
13:29:40hardcode it. So we could do plus we
13:29:43could say CSV files and that could work.
13:29:46So it would say does this path already
13:29:48exist? And we can try running this. And
13:29:50it's going to say false. So this doesn't
13:29:52already exist. But the thing is is we
13:29:55need to create three separate paths. So
13:29:56we could do this by just hard coding it
13:30:00in by saying CSV files, image files, um,
13:30:03and text files. Or we can just put this
13:30:06all in a list and loop through it. I
13:30:08think it's just going to be easier to do
13:30:10that or I don't know, visually it's
13:30:12going to be easier. So we'll do uh
13:30:14folder_names
13:30:17and we'll say is equal to and we'll
13:30:19create a list. So I think I want to call
13:30:21it CSV files. comma um image files or
13:30:26PNG files, whatever you want to write.
13:30:28And then we'll do text files.
13:30:33Do text files. And then we can go right
13:30:36down here. Um a little for loop. Uh I
13:30:39think what we'll do, well actually let's
13:30:41write folder
13:30:44names. Um then we can put something like
13:30:48uh let's write loop. Why not? Um, so a
13:30:52little trick for the for loop is going
13:30:54to say for and we'll say loop in and
13:30:57we'll just do a range because we want it
13:30:59to basically go through here. We don't
13:31:01want it to actually give us these file
13:31:02names. We just want it to count 0 1 and
13:31:05two. So if we do range from 0 to two 0
13:31:10uh 0 1 2 that should work. If we do um
13:31:13this then when it loops through it's
13:31:15going to call folder name and say zero
13:31:17which would be CSV files, image files
13:31:19and text files. Um, so let's
13:31:24uh yeah, I need a colon. Let's run
13:31:26through this really quickly. Uh,
13:31:28shouldn't do anything.
13:31:30But what we can do now is we can say,
13:31:33okay, if this does not exist, what we
13:31:37can do is actually create it. So we'll
13:31:40say if not. So if this does not exist
13:31:44then what we're going to do is take this
13:31:49and we'll say osmake
13:31:54directory and then we'll do just like
13:31:57that. Um I think it's make directory s I
13:32:02think that's correct. Um so let's test
13:32:04this out really quickly. Let's see if
13:32:06this works.
13:32:08and invalid syntax. I need a colon.
13:32:11Okay, so I just ran this. Let's see if
13:32:14it did actually make those folders.
13:32:17Let's refresh it. And it didn't. So,
13:32:21let's just print this off. Um, so if
13:32:24not, let's just print. Let's see. Does
13:32:27this actually work?
13:32:29Let's do if.
13:32:33Okay.
13:32:34Ah. Okay. So, I think I know what might
13:32:38be happening. I think it's giving us It
13:32:40may actually be Let Let's check this
13:32:41really quick. Go to Python tutorials.
13:32:44Oh, no.
13:32:46I think it's creating
13:32:49Yeah, it's creating these Python
13:32:50tutorial images right here. Whoops.
13:32:52Okay, so I just figured it out. Um,
13:32:55let's go back into Python tutorials.
13:32:57Don't take a look at any of those
13:32:58notebooks. Those are secret. Um, we were
13:33:01creating them in the wrong place. Um,
13:33:04and that's because of this right here.
13:33:05We need a backslash. So, we need to
13:33:07actually include a backslash right here
13:33:10in this path. We didn't have that. Um,
13:33:15uly scanning string literal.
13:33:18Okay. So, this backslash could cause an
13:33:21issue. Let's see if I can do forward
13:33:22slashes on all these. Just stick with
13:33:25me, guys. I might cut this out. I might
13:33:26not. We'll see if this is important.
13:33:29Just going to keep talking while we're
13:33:30doing it. Um, let's run this.
13:33:35Okay. So, now that we're doing these
13:33:36forward slashes, we're still checking.
13:33:39Let's make sure we can still check those
13:33:40files. Good. Now, when we loop through
13:33:43this, I'm not going to Well, yeah, I can
13:33:45print it off. Doesn't matter. I'm going
13:33:47to print it and we'll see if that name
13:33:48works. And then we're also going to um
13:33:53uh well, I said if so, if it exists,
13:33:56then make it. No, no, no. So, if not, I
13:33:59think the not did make sense. We just
13:34:00weren't sure. We had to do some um
13:34:02checking. So, if it exists, then we're
13:34:05going to create it. And we'll keep the
13:34:06print in there because it doesn't really
13:34:07matter. So, it's going to create the CSV
13:34:10and image, but it didn't create the
13:34:12text. Let's see. Okay, let's uh I don't
13:34:17know why this would work, but let's run
13:34:19it. Okay, so I think I just had the
13:34:21wrong range. So, now we have our images
13:34:24all right, we have our folders, all
13:34:26three folders. Now, we need to write a
13:34:28script that will read in these and check
13:34:32and see what kind of file it is and
13:34:34place it into the correct folder.
13:34:36So, let's come right down here and let's
13:34:39see what we need to do. So, now I think
13:34:42we need to use this right here. Um, I
13:34:45think we need to loop through this to be
13:34:47able to check each one. So, we need to
13:34:49name this. So, we'll just do um file_ame
13:34:53is equal to run that. So now we have
13:34:55this file name um and what we can do is
13:34:59loop through this. So let's say
13:35:03let's say for file in file name. So
13:35:07we're going to loop through this. Now
13:35:09when it goes through it needs to check
13:35:12the it's going to check the file path
13:35:14and in the file path it'll say txt.csv.
13:35:18So let's say um if I think it should be
13:35:22CSV. Let's test it on this one. But if
13:35:26CSV is in
13:35:28file name or actually it's file. So if
13:35:33if it's in file
13:35:35and not in and oh not not in but if it's
13:35:40also not in this I believe because we're
13:35:44going to check we're going to check each
13:35:45of those folders. So, we're going to
13:35:48loop through and it's going to check and
13:35:50see if the CSV. So, if that string is in
13:35:54the file,
13:35:56then what we want to do is check that
13:36:00it's also not in here. That's actually
13:36:03just the folder. We also need um also
13:36:07we're not doing that for loop anymore.
13:36:09Um,
13:36:11okay. I'm sorry. I'm talking this
13:36:13through. I'm figuring it out as I go
13:36:15because I may have forgotten some of
13:36:17this. So, we're going to say this.
13:36:19That's the CSV files. So, we need to
13:36:22check this one. Um, let's do it like
13:36:26this. Oops. Okay. So, it's going to
13:36:30check to see if CSV files and I think it
13:36:33needs that in between it. So, it's going
13:36:34to say the path. So, there's our path
13:36:37plus
13:36:39slash CSV files. Um, actually, no. It
13:36:43needs to be like this cuz we're going to
13:36:44check that. Then I got it. All right, I
13:36:46figured it out now. Then we're going to
13:36:48check if this file is in there. Yeah.
13:36:51So, that's right. So, it says if the CSV
13:36:55is in the file, um, which is right where
13:37:00am I looking?
13:37:02Oh, file name. So, if it's in that list
13:37:04of the actual files, which is all of
13:37:06these. if we find CSV in any of these
13:37:09files and it's not already in here. So,
13:37:13it's going to say path plus CSV files.
13:37:16Did I say files? Yeah, CSV files plus
13:37:20file. Okay, that all looks correct. So,
13:37:23if it's not in there, we're going to use
13:37:25shuttle.move. Now, this is how we
13:37:27actually move the file. It gives us the
13:37:29ability to move what we want. Then,
13:37:31we'll say move. We need to take it from
13:37:33our initial path to our new path. So,
13:37:36we're going to specify we'll separate by
13:37:38a comma. We need to specify its original
13:37:41path, which it should just be this
13:37:46without this.
13:37:48I think it should be file path because
13:37:51this is where it is now. It's in the fi
13:37:53this path with that file name. Then, we
13:37:56need to say we want to move it to here.
13:37:59That is what we want to do. Um,
13:38:03yeah. So, let's check it with just this
13:38:05one. and see if it works. Okay, it ran
13:38:08through it. Let's go check. Aha, now
13:38:10that CSV file is gone. Perfect. That is
13:38:13exactly what we wanted to happen. Now we
13:38:15can just recreate this for
13:38:19um
13:38:21for both our PNG files or our image
13:38:23files and our text files. So we'll say
13:38:25LF and LF
13:38:29and let's do PNG.
13:38:33Then we'll do image files
13:38:36and image files because again we're just
13:38:39doing the exact same thing. I can do
13:38:40text files. The next one's going to be
13:38:43text files. Text files. So this one's
13:38:46going to check for txt.
13:38:48Now do we need anything else? Um, we'll
13:38:52just say else and we'll print off print
13:38:57this file type is not included or or if
13:39:02there's multiple files, we'll say there
13:39:04are files in this path
13:39:10that were not moved.
13:39:12Okay. So, if we run through this, it's
13:39:17going to catch our CSV, catch our PNG,
13:39:19catch our text, and if not, it'll say
13:39:21there are files in this path that were
13:39:23not moved. Exclamation point. All right.
13:39:25Now, let's run through this.
13:39:28Uh,
13:39:30uh, that's because if LF l
13:39:35and then it's going to this else
13:39:37statement. Uh, I don't know. Let's let's
13:39:40circle back around to that in a second.
13:39:42All of them were moved properly. That's
13:39:46really good
13:39:49really quickly. I I'll I'll check and
13:39:51see. I just don't I'm going to take that
13:39:52out for now. So, I'm just going to run
13:39:54it. Um I we may or may not go back to
13:39:56that, but let's check and see if
13:39:58everything worked properly. So, let's go
13:40:00into the CSV file. And we have our CSV
13:40:03file. Let's go into our image files. And
13:40:05we have our images. And let's go into
13:40:08our text file. And there are our text
13:40:12files. Now, is there anything else that
13:40:15we need to do? I don't believe so. But
13:40:18what I can do is I can take all this.
13:40:23I can include it in here.
13:40:26And I'm going to
13:40:30basically restart it
13:40:34just to see if it works properly from
13:40:36scratch. Right. I just want to make sure
13:40:38that I didn't miss anything. Um, and
13:40:40we'll delete these.
13:40:42So, we have our I'm just going to rerun
13:40:45everything. We We imported,
13:40:48we created our path. These are our file
13:40:50names. And then when we run this, it
13:40:52should take our folder names, check
13:40:54through them. If they aren't already
13:40:56created, it's going to create it. Don't
13:40:59need it to print. So, let's get rid of
13:41:00that. Then for the file within our file
13:41:04names, and it check it checks each one.
13:41:06We check if there's a CSV and if it's
13:41:09already in that file, if it's already in
13:41:12that folder, I mean, if it's in that
13:41:14folder, then it doesn't do anything. But
13:41:15if it isn't, so and not it's not in
13:41:18there, it is going to move it to that
13:41:20location. So, it's going to check CSV,
13:41:22PNG, and text. I think everything should
13:41:25work properly. Let's run this.
13:41:29And it looks like it's working. Good,
13:41:31good, good. And perfect. It worked
13:41:35exactly how I had hoped. Um,
13:41:38that's great. So, this is the automatic
13:41:41file sorter in file explorer project.
13:41:45Uh, you can go even a step further. So,
13:41:47I had to come in here and manually run
13:41:49this. You can go a step further and put
13:41:51a timer on this where it automatically
13:41:53does this maybe every hour, every day,
13:41:57every 30 minutes. You can run this in
13:41:59your background, especially if you
13:42:00create um like an execution for this.
13:42:04You can run this in your background. Um
13:42:06if you are curious on how to do that, I
13:42:08think I did something similar to that in
13:42:10my web scraping project. Um my Amazon
13:42:14web scraping project if you want to go
13:42:15check that one out. But we're not going
13:42:16to do it in this project. This is all I
13:42:18wanted to show you how to do. So, I hope
13:42:20that this was helpful. I hope that this
13:42:21project was, you know, interesting and
13:42:24that you liked it. And I hope that you
13:42:25learned something. And so if you did, be
13:42:27sure to like and subscribe below and I
13:42:29will see you in the next video. What's
13:42:31going on everybody? Welcome back to
13:42:32another video. Today we're going to be
13:42:34starting our Python web scraping
13:42:35tutorial series. Now, this is more of a
13:42:37continuation of the Python tutorial
13:42:39series, but because we're going to be
13:42:40focusing on web scraping for three or
13:42:42four videos, I wanted to just make it
13:42:44its own little minieries. In this
13:42:46series, I'm going to show you the basics
13:42:47of web scraping. how to actually look at
13:42:49HTML, how to inspect a web page, how to
13:42:51pull that data in, and then even put it
13:42:53into a CSV file so you can save it and
13:42:55use it. Now, in this series, we're just
13:42:56covering the basics, which is a
13:42:58fantastic place to start, but in future
13:42:59series, I'll be going into some of the
13:43:01more advanced web scraping topics as
13:43:03well. So, without further ado, let's
13:43:04jump on my screen and get started with
13:43:05web scraping. Now, the first thing that
13:43:07we need to learn is HTML. HTML stands
13:43:10for hypertext markup language, and it's
13:43:12used to describe all of the elements on
13:43:15a web page. Now, when we actually go to
13:43:17a website and start pulling data and
13:43:19information, we need to know HTML so we
13:43:21can specify exactly what we want to take
13:43:24off of that website. So, that's where
13:43:26HTML comes in. And we're going to look
13:43:27at the basics, understanding just the
13:43:29basic structure of HTML. Then, we'll go
13:43:31look at a real website. And you'll kind
13:43:33of see that it's a little bit more
13:43:34difficult than what we just have right
13:43:36here. But, this is the basic building
13:43:38blocks to get to what the HTML actually
13:43:40looks like on a website. Now, this is
13:43:43basically what HTML looks like. We have
13:43:45these angle brackets with things like
13:43:47HTML, head, title, body, and then you'll
13:43:51notice that at the end we'll have a body
13:43:54and then we'll have a body at the
13:43:56bottom. This forward/body denotes that
13:43:59this is the end of the body section in
13:44:02HTML. So everything inside of this is
13:44:05within this body. So there is this
13:44:07hierarchy within HTML. We have HTML and
13:44:11HTML at the bottom which encapsulates
13:44:13all the HTML on the website. Then we
13:44:15have things like head and head, body and
13:44:18body. Now within these sections, we
13:44:20usually have things like classes, tags,
13:44:22attributes, text, and all these other
13:44:24things. Things that we'll get to in
13:44:25different lessons, but one of the
13:44:27easiest ones to notice and look at are
13:44:29tags. Things like a P tag or a title
13:44:32tag. Now, within these tags, because
13:44:34this is a super simple example, we have
13:44:37these strings here. my first web page.
13:44:39And this is what's called a variable
13:44:41string. And this is actual text that we
13:44:43could take out of this web page. Now
13:44:45that you understand the super basics of
13:44:47HTML, let's actually go to our website.
13:44:49And I'm going to have a link down below,
13:44:51but it's going to be this one right
13:44:52here. This is basically just a website
13:44:54that you can, you know, practice web
13:44:56scraping on. It's called
13:44:57scrapethesite.com.
13:44:59And what we're going to do is look at
13:45:01the HTML behind this web page. And you
13:45:03can do this on any website that you go
13:45:05on. So we're going to rightclick. We're
13:45:07going to go down to inspect.
13:45:10Now, right off the bat, this looks a lot
13:45:13more complicated and a lot more complex
13:45:15than the very simple illustration that
13:45:17we were looking at. But let's kind of
13:45:20roll this up just a little bit. You'll
13:45:22notice we have HTML and HTML at the
13:45:24bottom. We have a head and there is the
13:45:26end of the head and then a body and the
13:45:28end of the body. So in a super simple
13:45:31sense it is similar but just the
13:45:34information that's within it is a lot
13:45:36more difficult. Now if we look at this
13:45:38title right here, this is our title tag.
13:45:40If we click this little arrow, this is
13:45:43our drop down. You'll notice that here
13:45:45we have this string hockey teams forms
13:45:47searching and pageionation. Now let's
13:45:50say we didn't know we didn't want to
13:45:52click on that and go find it. There is
13:45:54something that's super helpful within
13:45:55this inspection page that you can click
13:45:57on right here. It says select an element
13:45:59in the page to inspect it. So, we're
13:46:01going to click on that. And as we go
13:46:03through our page and let's click on this
13:46:05title. It's going to take us to exactly
13:46:07where this is in our HTML. This is
13:46:10extremely helpful, extremely useful. For
13:46:13example, let's say the data I want is
13:46:15down here. I want to take in the Boston
13:46:17Bruins. I can click on it and it's going
13:46:19to take me to where that is exactly in
13:46:21the HTML. This is where we can start
13:46:23writing our web scraping script to
13:46:25specify, okay, I'm looking for a TR tag.
13:46:27I'm looking for a TD tag. I'm looking
13:46:29for the class called team. This is all
13:46:32information and things that we can use
13:46:33to specify exactly what we want to pull
13:46:36out of our web page. Now, there are
13:46:38other things that we didn't really look
13:46:40at as well in just our simple
13:46:42illustration. Let's come right over
13:46:44here. There's things like hrefs. Now,
13:46:46these are hyperlinks. So, if we went and
13:46:49then clicked on this, this is just
13:46:51regular text, but inside of it is this
13:46:53hyperlink where if we clicked on it, it
13:46:55would take us to another website. And
13:46:57typically that's denoted by this href
13:46:59right here. Then you'll typically see
13:47:01things like a P tag which usually stands
13:47:03for a paragraph. Now the last thing that
13:47:05I want to show you while we're here and
13:47:07we're going to learn a lot more in the
13:47:08next several lessons. But if we come
13:47:10right down here there is this actual
13:47:12entire table here. And let's try to find
13:47:15this table. And I'm having trouble
13:47:17selecting the entire thing. But let's
13:47:18select this team name. And if we look at
13:47:20this team name you can see that this is
13:47:22encapsulating the table. So this table
13:47:24tag. Now, these are super helpful
13:47:26because it takes in the entire table.
13:47:28Now, if we wrap this up and we look just
13:47:30at this, it says class table and then we
13:47:33have the end of this table tag. Now,
13:47:35when we open it, it's going to have all
13:47:38of this information. So, as you can see,
13:47:39as I'm highlighting over it, we have
13:47:41these TH tags. Then, we have these TD
13:47:44tags and even these TR tags, which is
13:47:48the individual data. And this is
13:47:49something that we'll look at when we're
13:47:51actually scraping all of the data from
13:47:52this table in a future lesson. So this
13:47:55is how we can use HTML, how we can
13:47:57inspect the web page and see exactly
13:47:59what's going on kind of under the hood.
13:48:00And then in future lessons, we'll see
13:48:02how we can use this HTML to specify
13:48:04exactly what data we want to pull out.
13:48:06Thank you guys so much for watching. If
13:48:08you like this video, be sure to like and
13:48:10subscribe below. I will see you in the
13:48:11next lesson.
13:48:24Hello everybody. In this lesson, we're
13:48:26going to be taking a look at beautiful
13:48:28soup and requests. Now, these packages
13:48:30in Python are really useful. These are
13:48:32the two main ones that I used when I was
13:48:34first starting out with web scraping. It
13:48:36can get a lot of what you want done in
13:48:38order to get that information out. Now,
13:48:39of course, there are other packages that
13:48:41you can use that may be a little bit
13:48:42more advanced, but again, this is just
13:48:44the beginner series. In a future series,
13:48:46we'll look at other packages as well
13:48:48that have some more advanced
13:48:49functionality. So, what we're going to
13:48:50be doing is we're going to import these
13:48:52packages. And then we're going to get
13:48:53all of the HTML from our website and
13:48:56make sure that it's in a usable state.
13:48:58And then in the next lesson, we're going
13:49:00to kind of query around in the HTML,
13:49:02kind of pick and choose exactly what we
13:49:04want. We'll look at things like tags,
13:49:06variable strings, classes, attributes,
13:49:08and more. So let's get started by
13:49:10importing our packages. What we're going
13:49:12to say is from BS4, this is the module
13:49:16that we're taking it from. We're going
13:49:17to say import. Then we'll do beautiful
13:49:22soup. Then we're going to come down and
13:49:24we're going to say import requests. Now
13:49:27let's go ahead and run this. I'm going
13:49:28to hit shift enter. And it works well
13:49:30for me. Now if this does not work for
13:49:32you, you may potentially need to
13:49:34actually install BS4. So, you may have
13:49:36to go to your terminal window and say
13:49:38pip install bs4. I'll just let you
13:49:40Google how to do that if you need to do
13:49:41that because it's pretty easy. But if
13:49:43you're using Jupyter Notebooks through
13:49:44Anaconda, like how we set it up at the
13:49:46beginning of this Python series, then
13:49:48you should be totally fine. It should be
13:49:50there for you. The next thing that we
13:49:51need to do is specify where we're taking
13:49:53this HTML from. So, what we need to
13:49:56actually do is come right over here to
13:49:57our web page and we need to get the URL.
13:50:00So, we're going to go here. We're going
13:50:01to copy this URL. And I'm just going to
13:50:03put it right here for a second. And what
13:50:06we're going to do is we're going to be
13:50:07using this URL quite a bit. So we just
13:50:09want to assign it to a variable. So
13:50:11we'll just say URL is equal to and then
13:50:13we'll put it right in here. Now we can
13:50:16get rid of that. So now this is our URL
13:50:18going forward. This is where we're going
13:50:19be pulling data from. Let's go ahead and
13:50:22run this. Now we're going to use
13:50:23requests and what we're going to do is
13:50:25we're going to say requests.get
13:50:28and then we're going to put in URL. Now
13:50:31this get function is going to use the
13:50:33request library. It's going to send a
13:50:34get request to that URL and it's going
13:50:37to return a response object. Let's go
13:50:39ahead and run this.
13:50:41As you can see here, I got a response of
13:50:43200. If you got something like a 204 or
13:50:46a 400 or 401 or 404, all of these things
13:50:50are potentially bad. Something like a
13:50:52204 would mean there was no content in
13:50:54the actual web page. 400 means a bad
13:50:57request. So, it was invalid. The server
13:50:59couldn't process it and you don't get
13:51:00any response. If you got a 404, that
13:51:03might be one that you're familiar with.
13:51:04That's an error that means the server
13:51:05cannot be found. The next thing that
13:51:07we're going to do is take the HTML. Now,
13:51:09if you remember, we come right back here
13:51:12and we inspect this. We have all of this
13:51:14HTML right here. Now, on this web page
13:51:16specifically, right now, it's completely
13:51:19static. It's not a bunch of moving stuff
13:51:21or anything like that. usually when
13:51:23you're looking at HTML if you're looking
13:51:24at something like Amazon and those web
13:51:26pages can update. But when you actually
13:51:28pull that into Python, you're basically
13:51:29getting a snapshot of the HTML at that
13:51:32time. So what we're going to do is bring
13:51:34in all of this HTML which is our
13:51:36snapshot of our website and then we can
13:51:39take a look at it. So we're going to
13:51:41come right down here and now we're going
13:51:42to say beautiful soup. So now we'll use
13:51:45the beautiful soup package library. So
13:51:47we need to say beautiful soup and we're
13:51:49going to do an open parenthesis. We're
13:51:50going to do two things. There's two
13:51:52parameters that we need to put in here.
13:51:53First, we need to put in this get
13:51:55request. So, we actually need to name
13:51:57this and we'll call this page. We'll say
13:52:00page is equal to. And let's run this.
13:52:02And now, we're going to put that page in
13:52:04here. And what we're going to say is
13:52:06text. So, the page is what's sending
13:52:08that request. And then the text is
13:52:10what's retrieving the actual raw HTML
13:52:12that we're going to be using. Then,
13:52:14we're going to put a comma here. And
13:52:16what we need to specify is how we're
13:52:18going to parse this information. Now,
13:52:19this is an HTML. So what we're going to
13:52:21do is HTML just like this. This is a
13:52:25standard. This is already built in to
13:52:26this library. So we don't need to go any
13:52:28further. But it's basically going to
13:52:29parse the information in an HTML format.
13:52:32Let's go ahead and run this. Let's see
13:52:34what we get. And as you can see, we have
13:52:37a lot of information. And as we scroll
13:52:40down, I'll try to point out some things
13:52:41that we've already looked at in previous
13:52:43lessons. Um
13:52:47something like this th tag that should
13:52:49be very similar. That's the title. Then
13:52:51we have these TD tags. And then of
13:52:53course, if we scroll down even further,
13:52:55we'll have things like a TR tag. So
13:52:57these are all things that we looked at
13:52:58in that first lesson when learning about
13:53:00HTML. Now again, we want to assign this
13:53:02to a variable. So we're going to say
13:53:05soup. That's going to say equal to this
13:53:08information right here. Now I'm not
13:53:10going to go into all the history behind
13:53:11beautiful soup. But what I will say is
13:53:13the guy who created this beautiful soup
13:53:15library, uh, what he said was is that it
13:53:17takes this really messy HTML or XML,
13:53:20which you can also use it for. Uh, and
13:53:22it makes it into this kind of beautiful
13:53:24soup. So, I just thought that was kind
13:53:25of funny. Uh, but that's why we're
13:53:26calling it soup right here. And we're
13:53:28going to go ahead and run this. And
13:53:30we'll come right down here and we'll say
13:53:32print soup. And let's run it. And now we
13:53:36have everything in here. So, we have our
13:53:38HTML, our head, we have some href and
13:53:42some links in here. Let's scroll down a
13:53:44little bit more. And then we have our
13:53:46body right there. And of course, we have
13:53:48a bunch of information in here. Now, in
13:53:50the next lesson, what we're going to be
13:53:52doing is learning how to kind of query
13:53:54all of this to take specific information
13:53:56out and basically understand a lot of
13:53:58what's going on in this HTML to make
13:54:00sure we can actually get what we need.
13:54:01Now, if this looks really kind of messy
13:54:04to you and it just doesn't make a lot of
13:54:06sense, there is one more thing that I'm
13:54:08going to show you and we'll come right
13:54:09down here. So, we'll say soup.pritify.
13:54:13And if you've ever used a different type
13:54:15of programming languages, uh, Pritify is
13:54:17very common in a lot of them where it'll
13:54:19just make it a little bit more easy to
13:54:21visualize and see. Uh, you'll notice
13:54:22that it kind of has this hierarchy built
13:54:24in. Whereas, if we scroll up, there's no
13:54:27hierarchy built in. It's all just down
13:54:28this lefth hand side. So if you kind of
13:54:31want to view it and just kind of
13:54:32visually see the differences, this does
13:54:34help a lot. But it doesn't actually help
13:54:37a lot when you're, you know, querying it
13:54:39or using, you know, find and find all,
13:54:41which is what we're going to look at in
13:54:43the next lesson. So that is our lesson
13:54:44on beautiful soup and requests. In the
13:54:47next two lessons, we're going to be
13:54:48looking at find and find all as well as
13:54:50really diving into things like variable
13:54:52strings and tags and classes and all
13:54:53those things. And then in the last
13:54:55lesson, we're going to do kind of this
13:54:56mini project where we try to get all the
13:54:57data from this web page that we've been
13:54:59using from that table and put it into a
13:55:02pandas dataf frame. So, thank you guys
13:55:04so much for watching. I really
13:55:05appreciate it. If you like this video,
13:55:07be sure to like and subscribe below and
13:55:09I will see [music] you in the next
13:55:10lesson.
13:55:22Hello everybody. In this lesson, we're
13:55:24going to be taking a look at find and
13:55:26find all. Really, we're going to be
13:55:28looking at a ton of different things in
13:55:30this lesson. This is where we really
13:55:31start digging in, seeing how we can
13:55:33extract specific information from our
13:55:36web page. But in order to do that, let's
13:55:38set everything up where we actually
13:55:39bring in the HTML like we did in the
13:55:41last lesson. And we're just going to
13:55:43write all this out one more time just
13:55:44for practice if nothing else. And then
13:55:47we'll get into actually getting that
13:55:49information from the HTML. So, we're
13:55:51going to start by saying from BS4 import
13:55:55beautiful
13:55:57soup. There we go. And import requests.
13:56:01We'll go ahead and run this. Then we're
13:56:03going to come up here, grab our HTML or
13:56:07sorry, our URL. So, we'll say URL is
13:56:09equal to and we'll have that right here.
13:56:13Now, we need to say page is equal to and
13:56:16then we'll do requests.get get and then
13:56:19we'll put in our URL right here. And
13:56:21we're going to come over here and run
13:56:22this. And lastly, we need to say soup.
13:56:25So we'll say soup is equal to beautiful
13:56:29soup. There we go. And then within our
13:56:31parenthesis, we need to specify the
13:56:33page.ext because we need that. And our
13:56:35parser, which is HTML.
13:56:39And there we go. And let's go ahead and
13:56:41run this. Let's print it out. Make sure
13:56:43it's working.
13:56:45And there we go. So, we have our soup
13:56:48right here. All this should look really
13:56:50similar to uh our last lesson. And so,
13:56:54now we've brought in our HTML from our
13:56:56page. We have a lot a lot a lot of
13:56:58information in here. Now, really
13:57:00quickly, let's come over and let's
13:57:02inspect our web page.
13:57:05Now, in here, we have a ton of
13:57:08information, right? We have bunch of
13:57:10different tags and classes and all these
13:57:11other things. But how do we actually use
13:57:14these? Well, that's where the find and
13:57:16find all is going to come into play. And
13:57:18they're pretty similar, and you'll see
13:57:20that in just a little bit. But let's say
13:57:22we want to take uh one of these tags.
13:57:24And let's come down. Let's say we just
13:57:27want to take this div tag. Now, there's
13:57:30going to be a lot of different div tags
13:57:33in our HTML, but let's just come right
13:57:36here. Let's go down and let's say we're
13:57:39going to call soup. We're going to say
13:57:40soup. That's all of our information. and
13:57:41we're going to sayfind.
13:57:43Now, within our parentheses, we can
13:57:45specify a lot of different things, but
13:57:47we're going to keep it really simple
13:57:48right now. We're just going to say div.
13:57:51Let's go ahead and run this. What this
13:57:52is going to bring up is the very first
13:57:55div tag in our HTML. And that's going to
13:57:57be this information right here. Now,
13:58:00let's copy this. And we're going to do
13:58:02the exact same thing except we're going
13:58:05to say find_all.
13:58:08Now, let's run this. Now we're going to
13:58:10have a ton more information. Really all
13:58:13find and find all do is that they find
13:58:16the information. Now find is only going
13:58:19to find the first response in our HTML
13:58:22list. That's the div class container.
13:58:24Let's go back up to the top. That's our
13:58:27div class container. But find all is
13:58:29going to find all of them. So it'll put
13:58:31it in this list for you. So it's going
13:58:33to have this first one and it goes down
13:58:34to uh this for slashd, which should be
13:58:38right here. And then we have a comma
13:58:40which separates our next div tag. So
13:58:43that is how we can use it. Now what if
13:58:45we want to specify one of these div
13:58:47tags? We pulled in a ton of them, but we
13:58:49want to just look for one of them. Well,
13:58:51this is something where the class comes
13:58:53in handy because right now we have class
13:58:55is equal to container. Class is equal to
13:58:57co MD-12.
13:59:00I don't know what these are at the off
13:59:01the top of my head, but um usually
13:59:04they'll be somewhat unique and we can
13:59:06use these to help us specify what we're
13:59:08looking for. For example, just kind of
13:59:10glancing at this, we can also use this a
13:59:12tag if we wanted to look at this. So, we
13:59:14could say, oh, we're looking for uh
13:59:16these hrefs. So, we have an href here
13:59:19and this right down here, we have this
13:59:20href as well, which again u if you
13:59:23remember from a previous lesson, that
13:59:25stands for a hyperlink. Now something
13:59:27like the class or the href um or these
13:59:31ids these are all attributes. So we can
13:59:34specify or kind of filter down based off
13:59:36of these. Now let's try it. So what we
13:59:38can do is we can do class first and this
13:59:39is kind of the default uh within
13:59:42something like find all is you can even
13:59:44do class underscore. We can come right
13:59:46back up. We have this div and then
13:59:48here's our class. So again we have to
13:59:50have the div and the class. So if we
13:59:52took this a tag, this is an a tag which
13:59:55would go right here with the class of
13:59:57something like navl link or something
13:59:59like nav link again down here. We need
14:00:01to specify that more but we have our
14:00:03div. So we'll say cl co md12 right here.
14:00:08And let's go ahead and run this. And now
14:00:10it's going to pull in just that
14:00:11information. Now we're still getting a
14:00:13list because we have multiple of these.
14:00:15So this div class uh col md-12 doesn't
14:00:19just happen once. If we scroll down,
14:00:22we'll see it multiple times. Something
14:00:24like right here. Uh or actually, let me
14:00:27see, right here. So, here's this comma.
14:00:29Then here's our next one. So, we have
14:00:31two of these uh div tags with a class of
14:00:34coal- md-12. And in each of these, we
14:00:38have different information. This looks
14:00:40like a paragraph with this p tag right
14:00:42here. And let's scroll back up. Uh, so I
14:00:46also think we should try out doing
14:00:48something like this P tag. Typically
14:00:50these P tags stand for paragraphs or
14:00:52they have text information in them.
14:00:54Let's try a P tag really quickly. Let's
14:00:56just see what we get. And let's run
14:00:59this. And it looks like we get multiple
14:01:01P tags. Now, if we come back here, you
14:01:04can see that there's this information
14:01:06and it's this information that we're
14:01:07pulling in. And I'm just, you know,
14:01:09noticing that from right here. And then
14:01:12we have this information right here. And
14:01:14it looks like there's one more which is
14:01:16this href which looks like this open
14:01:18source. So data via and then that uh
14:01:21hyperlink or that link right there. So
14:01:24we have three different P tags. Now just
14:01:26to verify and make sure that that's
14:01:28correct, what we could do is come over
14:01:30here. We're going to click on this
14:01:32paragraph. It's going to take us to that
14:01:34P tag where the class is equal to lead.
14:01:37Let's come over here and look at this
14:01:39paragraph. Now we have another P tag
14:01:42right over here where the class is equal
14:01:44to glyphicon glyphicon/education.
14:01:48I have no idea what that means. Um and
14:01:50then we'll go to our last one which is
14:01:52right here where the P tag is equal to
14:01:56uh we have a tag href class uh and a
14:01:59bunch of other information. So let's say
14:02:01we just wanted to pull in this paragraph
14:02:04right here. Let's go here and see how we
14:02:06can specify this information. So it
14:02:08looks like P where the class is equal to
14:02:10lead. That looks like it's going to be
14:02:13unique to just that one. So if we come
14:02:15down here, we're going to say comma and
14:02:18it was class. So you can do uh class
14:02:21underscore is equal to and then we're
14:02:24going to say lead. Let's try running
14:02:26this. And we're just pulling in that
14:02:29information. Now let's say we actually
14:02:31want to pull in this paragraph. We
14:02:33actually want this text right here. And
14:02:36this is a very real use case. You know,
14:02:38let's say I'm trying to pull in some
14:02:39information or or a paragraph of text.
14:02:42Well, let's copy this. And what we're
14:02:44going to then do is say text. And let's
14:02:48run this. Now, we're going to get an
14:02:49error right here. And this is a very
14:02:51common error because we're trying to use
14:02:54find all. Unfortunately, find all does
14:02:57not have a text attribute. We actually
14:03:00need to change this to find. Typically,
14:03:03when I'm working with these find and
14:03:05find alls, I'm using find all most of
14:03:07the time until I want to start
14:03:09extracting text. Then when I specify it,
14:03:12I'll change this back to find, just like
14:03:14this. Now, let's try this. And now we're
14:03:17getting in parentheses this information.
14:03:20Now, this is all wonky. It needs to
14:03:22definitely be cleaned up a little bit.
14:03:24But if we code back up, it's no longer
14:03:26in a list. And we no longer have things
14:03:29like these P tags in here or this class
14:03:33attribute. So we're really just trying
14:03:34to pull out this information. Now again,
14:03:37this does not look perfect. We could
14:03:39even try to do something like strip.
14:03:42Look like there's some white space. That
14:03:44cleans it up a little bit. This
14:03:46definitely looks a little better. Um and
14:03:48we could definitely go in here and clean
14:03:50this up more. But just for you know an
14:03:52example, this is how we can then extract
14:03:54that information. Now, let's look at one
14:03:56more example. This is some information,
14:03:58and this is what we're going to do kind
14:04:00of our little mini project in the next
14:04:01lesson on. Let's say we wanted to take
14:04:03all this information. Well, what if we
14:04:05wanted to pull in something like the
14:04:06team name? That's going to be in right
14:04:09here in this TR tag. And each of these
14:04:12TR tags have TH tags underneath them.
14:04:15So, if we scroll down, you'll notice
14:04:17that each row is this TR tag. So, let's
14:04:22go ahead and search for let's do th.
14:04:25Let's just search for that first. So,
14:04:27let's come right back up here. Let's use
14:04:29this find all.
14:04:32And we'll get rid of this text for right
14:04:35now. And let's just say we want to look
14:04:38for
14:04:39the TR. Is that what we said we were
14:04:41looking for? No, TH. So, let's say we're
14:04:43looking for TH. Let's go ahead and run
14:04:46this. So, we're going to have underneath
14:04:47this th we have team name, year, wins,
14:04:50losses, and notice these are all the
14:04:53titles. So, these titles are the only
14:04:56ones with these TH tags. If we go down,
14:04:59you'll notice that the date is actually
14:05:01TD tags. So, now let's go back and look
14:05:05for TD. We'll say D. And this is going
14:05:09to be a lot longer. We have a lot of
14:05:11information, but these are all the rows
14:05:13of data. Let's see if we can just get
14:05:15one piece of this data. We're going to
14:05:17get back. We want just this team name.
14:05:19That's all we're trying to pull in for
14:05:21now. Um, and then we'll try to get this
14:05:24row. And then in the next lesson, we're
14:05:26going to try to get all of this
14:05:27information, make it look really nice,
14:05:29and then we'll put it into a Pandanda's
14:05:31data frame. So, let's just get this team
14:05:33name right now. Let's go ahead. We're
14:05:36going to say th. Let's run this. And we
14:05:39have this th. And now that we know we're
14:05:42getting this information in, we can do
14:05:47find. Let's run this. So there's our
14:05:50team name. We're just going to say text.
14:05:54And again, we can do dot strip just like
14:05:57that. And bam, we have our team name. So
14:06:00you can kind of start getting the idea
14:06:02of how we're pulling this information
14:06:04out. We're really just specifying
14:06:06exactly what we're seeing in this HTML.
14:06:09And what's really, really helpful. And
14:06:10you know something that I do all the
14:06:12time is I'm inspecting it. I'm just kind
14:06:15of searching like how what do I want?
14:06:17What piece of information do I want?
14:06:18Then I go ahead and click on it and then
14:06:20I'm looking you know where is this
14:06:22sitting in the hierarchy. It's within
14:06:23the body. It's within this table with
14:06:26the class of table. Then it's down here
14:06:28where this TR tag and then this TD tag.
14:06:31So I'm looking kind of at the hierarchy
14:06:33and I'm specifying exactly what I'm
14:06:35looking for. So that is what we're going
14:06:36to look at in today's lesson. And that's
14:06:38how we can use find and find all. We
14:06:40were able to look at classes and tags
14:06:43and attributes and variable strings,
14:06:45which is this right here, getting that
14:06:47text uh and variable strings. And we
14:06:50were look at find and find all and how
14:06:52it's pulling that information in and how
14:06:54we can specify exactly what we're
14:06:55looking for. Now, in the next lesson,
14:06:57which is definitely going to be the most
14:06:58exciting one, we're going to try to pull
14:07:00in all of this information. So every
14:07:03single thing because we'll be able to
14:07:05put all this information into a dataf
14:07:07frame which then we can use pandas to
14:07:09really search and manipulate that data
14:07:12within that data frame. So with that
14:07:13being said that is the end of this
14:07:15lesson. If you like this video be sure
14:07:17to like and subscribe. I will see you in
14:07:19the next lesson.
14:07:22[music]
14:07:27>> [music]
14:07:32>> Hello everybody. In this lesson, we are
14:07:34going to be scraping data from a real
14:07:35website and putting it into a Pandanda's
14:07:37dataf frame and maybe even exporting it
14:07:39to CSV if we're feeling a bit spicy.
14:07:41Now, in the last several lessons, we've
14:07:44been looking at this page right here.
14:07:46And I even promised that we were going
14:07:48to be pulling this data, but as I was
14:07:50building out the project, I just I
14:07:52honestly thought it was a little bit too
14:07:53easy since in the last lesson, we kind
14:07:55of already pulled out some information
14:07:57from this table and I want to kind of
14:07:59throw you guys off. So, we're going to
14:08:00be pulling from a different table. We're
14:08:02going to be going on to Wikipedia and
14:08:04looking at the list of the largest
14:08:05companies in the United States by
14:08:06revenue and we're going to be pulling
14:08:08all of this information. So, if you
14:08:10thought this was going to be easy in a
14:08:11little mini project, uh it's now a full
14:08:13project because why not? So, let's get
14:08:17started. Uh, what we're going to do is
14:08:19we're going to import beautiful soup and
14:08:20requests. We're going to get this
14:08:22information and we're going to see how
14:08:24we can do this and it's going to get a
14:08:26little bit more complicated, a little
14:08:28bit more tricky. We're going to have to,
14:08:29you know, format things properly to get
14:08:31it into our pandas data frame to make it
14:08:33looking good and making it more usable.
14:08:36So, let's go ahead and get rid of this
14:08:37easy table. We don't want that one. Uh,
14:08:39and we're going to come in here and
14:08:41we're just going to start off. This
14:08:42should look uh really familiar by now.
14:08:44We're going to say from BS4 import
14:08:49beautiful
14:08:50soup. I don't know if you've noticed,
14:08:52but I've messed up spelling beautiful
14:08:53soup in every single uh video I've
14:08:56noticed. Uh let's run this. And now we
14:08:59need to go ahead and get our URL. So
14:09:01let's come up here. Let's get our URL.
14:09:05Say URL is equal to. And we'll just keep
14:09:08it all in the same thing really quickly
14:09:10because we know this by heart by now,
14:09:12right? Uh we'll say request.get
14:09:15and then URL to make sure that we're
14:09:17getting that information. It give us a
14:09:19response object. Um hopefully it'll be
14:09:21200. That'll mean a good response. And
14:09:24then we'll say soup is equal to and then
14:09:26we'll say beautiful soup. And we'll do
14:09:29our page.ext.
14:09:30Now we're pulling in the information
14:09:32from this URL. And then we use our
14:09:34parser which will be oops html.
14:09:38And let's go ahead and run this. Looks
14:09:41like everything went well. Let's print
14:09:42our soup. Now, this is completely new to
14:09:45you. It's completely new to me. I don't
14:09:47know what I'm doing. Uh but it looks
14:09:49like we're pulling in the information.
14:09:50Am I right? So, we got a lot of things
14:09:53going for us. Uh the uh stuff was
14:09:56imported properly. We got our URL. We
14:09:58got our soup, which is uh not beautiful
14:10:01in my opinion. But let's keep on
14:10:04rolling. Let's come right down here.
14:10:05Now, what we need to do is we need to
14:10:07specify what data we're looking for. So,
14:10:10let's come and let's inspect this web
14:10:12page. Now, the only information that
14:10:14we're going to want is right in here.
14:10:16We're going to want these uh titles or
14:10:18these headers. Whoops. So, we're going
14:10:21to want rank name, industry, etc. And
14:10:23then we are for sure going to want all
14:10:25of this information. Let's just scroll
14:10:27down, see if there's anything tricky in
14:10:28here.
14:10:31All right, that looks pretty good. Uh,
14:10:33and there is another table. So, there's
14:10:35not just one table in here. There are
14:10:37two tables in this page. So that might
14:10:41change things for us. But let's come
14:10:43right back and let's inspect our page by
14:10:46using this little button right here. And
14:10:49let's specify in let's see if I can
14:10:51highlight just this page. Oh, it's not
14:10:54going. Oh, let's do that right there. So
14:10:57now we have this uh wiki table sorter.
14:11:00Now I'm going to actually come right
14:11:02here. I'm going to copy and I'm just
14:11:04going to say copy the outer HTML. I'm
14:11:07just going to paste in here real quick.
14:11:09And that's a ton of information. I
14:11:11didn't think it was going to copy all of
14:11:12it. And we're just going to delete that.
14:11:13I just wanted to keep that class uh
14:11:15because I wanted to then come right down
14:11:19here at the bottom and just see what
14:11:21this table uh looks like. I don't know
14:11:24if it's part of it or if it's a if it's
14:11:26its own table.
14:11:28Um I can't tell. Let's look at this rank
14:11:31and let's come up. So it says uh it's
14:11:34under this table
14:11:36and it looks like it's its own table but
14:11:38it says wiki table sort sortable jQuery
14:11:41table sortter wikip sortable jQuery
14:11:44table sortter. So, it looks like there
14:11:47are two tables with the same class,
14:11:50which shouldn't be a problem if we're
14:11:53using find to get our text because we
14:11:55should be taking the first one, which
14:11:56will be this table. And this is the
14:11:58table we want. Um, and if we wanted this
14:12:02one, we could just use find all and
14:12:04since it's a list, we could use indexing
14:12:07to pull this table, right? Um, but I
14:12:10think we're going to be okay with just
14:12:12pulling in this one.
14:12:14So, let's go ahead and let's do our
14:12:16find. So, we'll do soup.find.
14:12:20And we could find all or we could just
14:12:22do find uh table. Let's just try this
14:12:25and see what we get. And if it pulls in
14:12:28the right one that we're looking for,
14:12:29that would be great. Now, this does not
14:12:32look correct at all. Um I don't know
14:12:35what table it's pulling in. Oh, maybe
14:12:37it's this right here. This might be a
14:12:40table. Yeah, it is. So we have this uh
14:12:43box more citations. So actually we are
14:12:45going to have to do exactly like what I
14:12:47was talking about. Uh let's pull this
14:12:51and we well we could do comma class uh
14:12:54right here. And let's do both. You know
14:12:56what? This is a learning opportunity.
14:12:58Let's do both. So let me go back up to
14:13:01the top because I need these. Um and
14:13:04what we're going to do is come right
14:13:07down here. I want to add in uh another
14:13:10thing. Actually, I'll just push this one
14:13:12up. There we go. So, we're going to say
14:13:15find_all.
14:13:17Let's run this. So, now we have
14:13:19multiple. And again, we got that weird
14:13:21one first, but if we scroll down, here's
14:13:23our comma. And then here's our wik wiki
14:13:27table sortable. And then we have rank,
14:13:30name, industry, all the ones that we
14:13:32were hoping to see. And I guarantee you
14:13:34if you scroll all the way to the bottom,
14:13:37um, we're going to see
14:13:40potentially Wells Fargo, Goldman Sachs.
14:13:43I'm pretty sure those are, um,
14:13:46let's see. Yeah, here we go. Like Ford
14:13:48Motor, Wells Fargo, Goldman Sachs.
14:13:50That's this table right here. So now
14:13:52we're looking at the third table, but
14:13:54again, this is a list, so we can use
14:13:56indexing on this. And we'll just choose
14:13:58not position zero because that's this
14:14:00one right here, which we did not like.
14:14:03Well, now we'll take position one. Let's
14:14:05run this.
14:14:07Let's go back up to the top. And this is
14:14:09our table right here. Rank, name,
14:14:12industry. This is the information that
14:14:14we were actually wanting just to
14:14:16confirm. Rank name, industry, etc. So,
14:14:20this is the information we're wanting
14:14:22and we're able to specify that with our
14:14:23find all. And this is the information we
14:14:26want. So, we now want to make this the
14:14:28only information that we're looking at.
14:14:30So, I'm just going to copy this. We
14:14:32didn't need to use our class for this
14:14:33one. You could, probably could have. Um,
14:14:35but we could. So, let's actually um put
14:14:37this right down here. This will be our
14:14:38table. We'll say equal to, but then I'll
14:14:41come right here and I'm going to say
14:14:44soup.find.
14:14:46And this is just for demonstration
14:14:48purposes. We'll do table,
14:14:50blast is equal to, and then we'll look
14:14:54at this right here. Whoops. Me do this.
14:14:58And let's see if we get the correct
14:15:00output.
14:15:01And let's run this. And looks like we're
14:15:03getting a none type object. Uh, if I
14:15:06remember, it looks like the actual class
14:15:08is this right here. So, let's run this
14:15:12instead. And I got to get rid of the
14:15:14index. There we go. Okay. So, we were
14:15:17able to pull it in just using the find.
14:15:19So, the find table class. And it says
14:15:21wiki table sortable. At least that's the
14:15:24HTML that we're pulling in right here.
14:15:27Let me go back because I don't I don't
14:15:31know if that's what I was seeing
14:15:32earlier.
14:15:34Let's just get this rank. Let's go back
14:15:36up. Oh, where's the rank?
14:15:39Go. Rank. There we go. So, here's our
14:15:41rank. And let's go up to the table and
14:15:45there's our class.
14:15:47Yeah. And and that's just uh to me
14:15:49that's a little bit odd. So, it says
14:15:50wiki table sortable jQuery-t
14:15:58um in our actual Python script that
14:16:00we're running, it was only pulling in
14:16:03the wiki table sortable. So, it wasn't
14:16:06pulling in the jQuery-t
14:16:09uh I'm not 100% sure, but all things
14:16:12that we're working through and we were
14:16:14able to uh we were able to figure out.
14:16:17So, we're going to make this our table.
14:16:20We're going to say tables equal to uh
14:16:22soup.findall.
14:16:24And let's run this. And if we print out
14:16:26our table, we have this table. Now, this
14:16:29is our only data that we are looking at.
14:16:31Now, the first thing that I want to get
14:16:33is I want to get these titles or these
14:16:36headers right here. That's what we're
14:16:37going to get first. So, let's go in
14:16:40here. We can just look in this
14:16:41information. You can see that these are
14:16:42with these TH tags. And we can pull out
14:16:46those TH tags really easily. Let's come
14:16:49right down here. We're just going to say
14:16:52TH. And we can get rid of this. Let's
14:16:55run this. Now, these are our only TH
14:16:58tags because everything else is a TR tag
14:17:00for these rows of data. So, these TH
14:17:03tags are pretty unique, which makes it
14:17:05really easy, which is really great
14:17:07because then we can just do world_titles
14:17:10is equal to. So, now we have these
14:17:12titles, but uh they're not perfect, but
14:17:15what we're going to do is we're going to
14:17:17loop through it. So, I'm going to say
14:17:18world titles and I'll kind of walk
14:17:20through what I'm talking about. This is
14:17:22in a list and each one is within these
14:17:25th tags. So, th and then there's our um
14:17:28string that we're trying to get. So, we
14:17:30can easily take this list and use list
14:17:34comprehension and we can do that right
14:17:36down here. So, I'm going to keep this to
14:17:38where we can see it. Um we'll do world
14:17:41table titles. That's equal to. Now we'll
14:17:46do our list comprehension. Should be
14:17:47super easy. Uh we'll just say for title
14:17:51in world_titles. And then what do we
14:17:54want? We want title.ext.
14:17:57That's it. Um because we're just taking
14:17:59the text from each of these. We're just
14:18:01looping through and we're getting rank.
14:18:03Then we're looping through getting name.
14:18:04Looping through getting industry. That's
14:18:06it. So let's go and print our world
14:18:10table titles and see if it worked.
14:18:14And it did. Uh, this looks like it needs
14:18:16to be cleaned up just a little bit. So,
14:18:19let's go ahead and do that while we're
14:18:21here before we actually put it into the
14:18:23uh, Pandas dataf frame. Oops. I just
14:18:26wanted uh, I just wanted this actually.
14:18:30So, what we're going to do is try to get
14:18:31rid of those backslash ends. If we do
14:18:34strip, that may actually not work. Yeah.
14:18:36Uh, because this is a list. What we need
14:18:38to do is we can actually do it
14:18:40dot.ext.strip,
14:18:42strip right here. Let's try to do it in
14:18:44there. There we go. So, now we have uh
14:18:46this and now this world tables is good
14:18:50to go. Now, I'm actually noticing one
14:18:52thing that may be odd. Yeah. So, we have
14:18:56rank name, industry, it goes to
14:18:58headquarters, but then in here we're
14:19:00getting rank name, industry, and then
14:19:02the profits,
14:19:04which is from
14:19:06this table right here, which we don't
14:19:10want. Uh let's scroll back up. Let's
14:19:13kind of backtrack this and see where
14:19:15this happened. We did find all table.
14:19:18We're looking at the first one, right?
14:19:21And then we're doing headquarters.
14:19:26Uh so we're doing print table. Ah, okay.
14:19:28I think I found the issue here. And
14:19:30let's backtrack again. This is we're
14:19:32working through this together. We're
14:19:33going to make mistakes. Uh the table is
14:19:35what we actually wanted to do. We just
14:19:37did soup.allTth, find all th which is
14:19:39going to pull in that secondary table. U
14:19:42gez we were not thinking here. Um so now
14:19:45we need to do find all on the table not
14:19:49the soup because now we were looking at
14:19:50all of them. Oh what a rookie mistake.
14:19:52Okay. Uh let's go back. Now let's look
14:19:55at this. Now it's just down to
14:19:57headquarters. Okay. Okay. Let's go ahead
14:20:00and run this. Let's run this. Now we
14:20:03just have headquarters. Now let's run
14:20:05this.
14:20:06Now we are sitting pretty. Okay, excuse
14:20:09my mistakes. Hey, listen. You know, if
14:20:11it happens to me, it happens to you. I
14:20:13promise you. This is, you know, this is
14:20:14a project. This is a little project
14:20:16we're creating here. So, we're going to
14:20:17run into issues, and that's okay. We're
14:20:19figuring it out as we go. Now, what I
14:20:21want to do before we start pulling in
14:20:22all the data is I want to put this into
14:20:25our pandas data frame. We'll have the
14:20:27uh, you know, headers there for us to
14:20:29go, so we won't have to get that later,
14:20:31and it just makes it easier uh, in
14:20:32general, trust me. So, we're going to
14:20:34import pandas as pd. Let's go ahead and
14:20:37run this. And now we're going to create
14:20:38our data frame. So, we'll say pd dot.
14:20:42Now, we have these world uh table
14:20:44titles. So, what we're going to do is
14:20:46pd.data
14:20:47frame. And then in here for our columns,
14:20:50we'll say that's equal to the world
14:20:52table titles. And let's just go ahead
14:20:55and say that's our data frame and call
14:20:57our data frame right here. Let's run it.
14:20:59There we go. So, we were able to pull
14:21:02out and extract those headers and those
14:21:04titles of these columns. We're able to
14:21:06put it into our data frame. So, we're
14:21:08set up and we're ready to go. We're
14:21:09rocking and rolling. The next thing we
14:21:11need, let's go back up. Next thing we
14:21:14need is to start pulling in this data
14:21:16right here. So, we have to see how we
14:21:18can pull this data in. Now, if you
14:21:20remember
14:21:22that we had those TH tags, those were
14:21:24our titles. As you can see, I'm
14:21:26highlighting over it. But down here now
14:21:28we have these TD tags and those are all
14:21:31encapsulated within a TR tag. So these
14:21:34TR represent the rows, right? Then the D
14:21:39represents the data within those rows.
14:21:41So R for rows, D for data. So let's see
14:21:44how we can use that in order to get the
14:21:46information that we want. So let's go
14:21:48back up here. Just going to take this
14:21:50because again we're only pulling from
14:21:52table, not soup. Not soup. What were we
14:21:56thinking? Um, and let's go ahead and
14:21:58let's look at TR. Let's run this. Now,
14:22:01when we're doing this TR, these do come
14:22:04in with the headers. So, we're going to
14:22:07have to later on, we're going to have to
14:22:08get rid of these. We don't want to pull
14:22:09those in um and have that as part of our
14:22:12data. But, if we scroll down, there's
14:22:14our Walmart.
14:22:16Um, we have the location. These are all
14:22:19with these TD tags. And then, of course,
14:22:23it's separated by a comma. And then we
14:22:25have our TD2. So above we had our TD1.
14:22:29So row one, row two, row three, all the
14:22:31way down. Now we will easily be able to
14:22:34use this, right? Because this is our
14:22:36column data. And we can even call it
14:22:38that column
14:22:40data is equal to we'll run that. Um, and
14:22:44what we're going to do is we're going to
14:22:45loop through that because it was all in
14:22:46a list. So we're going to loop through
14:22:48that information, but instead of looking
14:22:49at the TR tag, we're going to look at
14:22:51the TD tag. So let's come right down
14:22:54here. We'll say for the row in column
14:22:57row
14:22:59and we'll do a colon. Now we need to
14:23:01loop through this. We'll do something
14:23:03like row.find_all.
14:23:06And then what are we looking for? We're
14:23:08not looking for the tr looking for the
14:23:10TD. And just for now, let's print this
14:23:14off.
14:23:15See what this looks like. Apparently, I
14:23:18didn't run this uh column data, that's
14:23:21why.
14:23:24And let's run this. And what we actually
14:23:27need to do is something almost exactly
14:23:30like this.
14:23:32And I'm going to put it right below it.
14:23:35Um, instead of printing this off because
14:23:38again, this is all in a list. We're
14:23:40using find all. So we're we're printing
14:23:42off another list which isn't actually
14:23:44super helpful. um for each of all these
14:23:48data that we're pulling in. What we can
14:23:50do is we can call this uh the row data
14:23:54and then we'll put the row data in here.
14:23:56So we'll say for and we'll say in row
14:24:00data. So we'll just say for the data in
14:24:02row data and we'll take the data we'll
14:24:05exchange that and now instead of uh
14:24:08world table titles we can change this
14:24:11into uh individual
14:24:14row data right and now let's print off
14:24:18the individual row data. So it's the
14:24:20exact same process that we were doing up
14:24:23here and that's how we cleaned it up and
14:24:25got this. And we may not need to strip
14:24:27but let's just run this and see what we
14:24:29get. There we go. Um, and strip I'm sure
14:24:31was helpful. Let's actually get rid of
14:24:33this.
14:24:34Yeah, strip was helpful. It's the exact
14:24:37same thing that happened on the last
14:24:38one. So, let's keep that actually. Let's
14:24:41run this. And now, let's just kind of
14:24:43glance at this information. Let's look
14:24:45through it. This looks exactly like the
14:24:48information that's in the table. Let's
14:24:50just confirm with this first one. Uh, 25
14:24:53uh two, what am I saying? 572754
14:24:562.4 4 2300
14:24:59572752.4
14:25:002300. So this looks exactly correct. Now
14:25:04we have to figure out a way to get this
14:25:07into our table because again these are
14:25:09all individual lists. It's not like
14:25:12we're just, you know, putting all this
14:25:14in at one time. We can't just take the
14:25:16entire table and plop it into um into
14:25:19the data frame. We need a way to kind of
14:25:21put this in one at a time. Now, if
14:25:23you're just here for web scraping and
14:25:24you haven't taken like my Panda series,
14:25:26that's totally fine. That's not what
14:25:28we're here for anyways. Um, but what we
14:25:30can do, we'll have our individual row
14:25:32data and we're going to put it in kind
14:25:35of one at a time. Now, the reason we
14:25:37have to do that is because when we had
14:25:39it like this, and let's go back. When we
14:25:41had it like this, it's printing out all
14:25:43of it. But what it's really doing, and
14:25:45let's get rid of it. Um, what it's
14:25:47really doing is it's kind of doing it
14:25:48like this. It's printing it off one at a
14:25:51time and it's only going to save that
14:25:53current row of data. This last one, it's
14:25:57only going to save that as it's looping
14:25:59through. So, what we actually want to do
14:26:01is every time it loops through, we
14:26:03append this information onto the data
14:26:06frame. So, as it goes through, and
14:26:08eventually it's going to end up with
14:26:09this one, but as it goes through, let's
14:26:11run this. As it goes through, it puts
14:26:14this one in. And then the next time it
14:26:15loops through, it puts this one in. And
14:26:17the next time it loops through, etc.,
14:26:19all the way down. Um, so let's see how
14:26:22we can do this. So we have our data
14:26:23frame right here. Let's get rid of this.
14:26:27Let's bring our data frame in. Now
14:26:29again, like I just mentioned, if you
14:26:30don't know pandas and you haven't
14:26:32learned that, uh, you know, go take my,
14:26:34uh, series on that. It's really good.
14:26:36And we do something very similar to this
14:26:37in that series. So I'm not going to kind
14:26:39of walk through the entire logic. Um,
14:26:41but there is something called LOC, which
14:26:44stands for location when you're looking
14:26:45at the index on a dataf frame. And we're
14:26:48going to use that to our advantage. So,
14:26:50we're going to say the length of the
14:26:52data frame. So, we're looking at how
14:26:54many rows are in this data frame. And
14:26:56then, we're going to say that's our
14:26:57length. Then, we're going to take that
14:27:00length and use it when we're actually
14:27:03putting in this new information. Pretty
14:27:05um pretty cool. So, we're going to say
14:27:07df.loc loc and then a bracket and we're
14:27:10putting in that length. So we're
14:27:12checking the length of our data frame
14:27:14each time it's looping through and then
14:27:16we're going to put the information in
14:27:18the next position. That's exactly what
14:27:20we're doing. So let's go ahead and put
14:27:22in the individual row data. Um so let's
14:27:26just recap. We're looping through this
14:27:29TR. This is our column data. So these TR
14:27:32that's our row of data. Then we're as
14:27:35we're as we're looping through it, we're
14:27:37doing find all and looking for TD tags.
14:27:39That's our individual data. So that's
14:27:42our row data. Then we're taking that
14:27:43data, each piece of data, and we're
14:27:46getting out the text and we're stripping
14:27:48it to kind of clean it. And now it's in
14:27:50a list for each individual row. Then
14:27:53we're looking at our current data frame,
14:27:55which has nothing in it right now. We're
14:27:57looking at the length of it. And we're
14:27:59appending each row of this information
14:28:02into the next position. So, let's go
14:28:04ahead and run this. It's working. It's
14:28:07thinking. And it looks like we got an
14:28:09issue. And not set a row with mismatched
14:28:12columns. Now, we're encountering an
14:28:14issue. Not one that I got earlier, but
14:28:16we're going to cancel this out. We're
14:28:19going to figure this out together. So,
14:28:20let's print off our individual row data.
14:28:24Let's look at this. This one is empty.
14:28:26Uh this is I'm almost certain is
14:28:29probably the issue. Um I didn't
14:28:31encounter this issue when I wrote these
14:28:33uh when I wrote this lesson. Um but I'm
14:28:35almost certain that this is the issue
14:28:37right here. So let's do the column data,
14:28:39but let's start at position. Um let's
14:28:42try one and not parentheses. I need
14:28:46brackets because this is a list, right?
14:28:48So it should work. And there we go. So
14:28:51now that first one's gone. So now we
14:28:53just have the information. I didn't even
14:28:55think about that um just a second ago,
14:28:57but I'm glad we're running into it in
14:28:59case you ran into that uh issue. Let's
14:29:02go ahead and try this again.
14:29:04And it looked like it worked. So, let's
14:29:06pull our data frame down. I could have
14:29:08just wrote DF. Let's pull our data frame
14:29:10down. And now this is looking fantastic.
14:29:14Now, um these three dots just mean
14:29:16there's information in there, just
14:29:17doesn't want to display it. But it looks
14:29:19like we have our rank, we have our name,
14:29:22we have the industry, revenue, revenue
14:29:24growth, employees and headquarters for
14:29:26every single one. So this is perfect.
14:29:29Now this is exactly what I was hoping to
14:29:31get. Now you can go in and use pandas
14:29:33and manipulate this and change it and
14:29:35you know dive into all the information
14:29:36in there. But we can also export this
14:29:40into a CSV if that's what you're
14:29:42wanting. So we could easily do that by
14:29:44saying we'll do df.2 2 CSV and then
14:29:49within here we're just going to do R and
14:29:51specify our file path. So let's come
14:29:53down here to our file path and we'll go
14:29:55to our folder for our output. So we're
14:29:58just going to take this path and let me
14:30:01do it like that. So I have this path in
14:30:02my one drive documents Python web
14:30:05scraping folder for output. So you know
14:30:07I already made this um and I'm just
14:30:08going to put this right down here. Now I
14:30:11do have to specify what we're going to
14:30:12call this. Um we'll just call this
14:30:15companies. And then we have to say CSV.
14:30:18That is very important. Now if we run
14:30:20this, I already know just because uh we
14:30:23have this rank and this index here.
14:30:25We're going to keep this index in the
14:30:26output. Not great. Uh but let's run it.
14:30:30Let's look at our output.
14:30:33There's our companies. And when we pull
14:30:34this up, as you can see, this is not
14:30:37what we want because we have this extra
14:30:38thing right here. Now, if we were
14:30:40automating this, this would get super
14:30:41annoying. So, what we're going to do is
14:30:43go back and just say index equals false.
14:30:45Let's go out of here. And now we're just
14:30:47going to come right down here. We're
14:30:48going to say, comma, index equals false.
14:30:52And so, it's going to take this index,
14:30:53and it's not going to import or actually
14:30:55export it into the CSV. Now, let's go
14:30:58ahead and run this.
14:31:01Let's pull up our folder one more time.
14:31:05And let's refresh just to make sure.
14:31:07Should be good. And now this looks a lot
14:31:10better. So, we're able to take all of
14:31:12that information and put it into a CSV
14:31:15and it's all there. So, this is the
14:31:17whole project. So, if we scroll all the
14:31:19way back up, let's just kind of glance
14:31:21at what we did here. Scroll down. We
14:31:24brought in our libraries and packages.
14:31:26We specified our URL. We brought in our
14:31:29soup. Um, and then we tried to find our
14:31:32table. Now, that took a little bit of uh
14:31:35testing out, but we knew that the table
14:31:37was the second one. So, in position one.
14:31:40So we took that table. We were also able
14:31:42to specify it using find but then we
14:31:45used the class and of course we just
14:31:47wanted to work with that table. That's
14:31:48all the data we wanted. So we specified
14:31:51this is our table and we worked with
14:31:53just our table going forward. Of course
14:31:55uh we encountered some small issues user
14:31:58errors on my end but we were able to get
14:32:00our world titles and we put those into
14:32:03our data frame right here using pandas.
14:32:06Then next we went back and we got all
14:32:08the row data and the individual data
14:32:10from those rows and we put it into our
14:32:13pandas dataf frame. Then we came below
14:32:16and we exported this into an actual CSV
14:32:19file. So that is how we can use web
14:32:21scraping to get data from something like
14:32:23a table and put it into a pandas dataf
14:32:26frame. I hope that this lesson was
14:32:27helpful. I know we encountered some
14:32:28issues. That's on my end and I
14:32:30apologize. But if you run into the same
14:32:32issues, hopefully that helped. Uh, but I
14:32:34hope this was helpful and if you like
14:32:36this, be sure to like and subscribe
14:32:37below. I appreciate you. I love you and
14:32:40I will see you in the next lesson.
14:32:43[music]
14:32:54So, the first thing that we need to do
14:32:55is import our pandas library. So, we're
14:32:58going to say import and we're going to
14:32:59say pandas. Now, this will import the
14:33:01pandas library, but it's pretty common
14:33:04place to give it an alias and as a
14:33:07standard when using pandas. People will
14:33:09say as pd. So, this is just a quick
14:33:12alias that you can use. Uh, that's what
14:33:13I always use and I've always used it cuz
14:33:15that's how I learned it and I want to
14:33:17teach it to you the right way. So,
14:33:18that's how we're going to do it in this
14:33:19video. So, let's hit shift enter. Now
14:33:22that that is imported, we can start
14:33:24reading in our files. Now, right down
14:33:26here, I'm going to open up my file
14:33:27explorer. And we have several different
14:33:30types of files in here. We have CSV
14:33:33files, text files, JSON files, and an
14:33:36Excel worksheet, which is a little bit
14:33:38different than a CSV. So, we're going to
14:33:41import all of those. I'm going to show
14:33:42you how to import it, as well as some of
14:33:45the different things that you need to be
14:33:46aware of when you're importing. So,
14:33:48we're going to import some of those
14:33:49different file types, and I'll show you
14:33:51how to do that within pandas. So, the
14:33:53first thing that we need to say is PD
14:33:55dot and let's read in a CSV because
14:33:58that's a pretty common one. We'll say
14:34:00read
14:34:02CSV. And this is literally all you have
14:34:05to write in order to call that in. Now,
14:34:08it's not going to call it in as a string
14:34:10like it would in one of our previous
14:34:11videos if you're just using the regular
14:34:14operating system of Python. When you're
14:34:16using pandas, it calls it in as a data
14:34:18frame. And I'll talk about some of the
14:34:19nuances of that. So, let's go down to
14:34:21our file explorer. We have this
14:34:23countries of the world CSV. You just
14:34:25need to click on it and rightclick
14:34:28and copy as path. And that's literally
14:34:31going to copy that file path for us. You
14:34:33don't have to type it out manually. You
14:34:34can if you'd like. And we're just going
14:34:36to paste it in between these
14:34:38parentheses. Now, if we run it right
14:34:40now, it will not work. I'll do that for
14:34:42you. It's saying we have this Unicode
14:34:44error. Uh, basically what's happening is
14:34:46is it's reading in these backslashes and
14:34:49this colon and all those backslashes in
14:34:51there and this period at the end. What
14:34:53we need to do is read this in as a raw
14:34:55text. So, we're just going to say R. And
14:34:57now it's going to read this as a literal
14:35:00string or a literal value and not as you
14:35:03know with all these backslashes which
14:35:05does make a big difference. When we run
14:35:07this, it's going to populate our very
14:35:09first data frame. So, let's go ahead and
14:35:10run it. And now we have this CSV in here
14:35:14with our country and our region. Now if
14:35:17we go and pull up this file and let's do
14:35:18that really quickly. Let's bring up this
14:35:20countries of the world. It automatically
14:35:22populated those headers for us in the
14:35:24data frame. But we don't have any column
14:35:27for those 0 1 2 3. So if we go back, as
14:35:30you can see right here, there's this
14:35:32index. And that's really important in a
14:35:34dataf frame. It's really what makes a
14:35:35dataf frame a dataf frame. And we use
14:35:37index a lot in pandas. We're able to
14:35:39filter on the index, search on the
14:35:41index, and a lot of other things which
14:35:42I'll show you in future videos. But this
14:35:45is basically how you read in a file.
14:35:48Now, if we go right up here in between
14:35:49these parentheses and we hit shift tab,
14:35:52this is going to come up for us. Let's
14:35:54hit this plus button. And what this is
14:35:57is these are all the arguments or all
14:35:59the things that we can specify when
14:36:02we're reading in a file. And there are a
14:36:04lot of different options. So, let's go
14:36:05ahead and take a look really quickly.
14:36:07Really quickly, I wanted to give a huge
14:36:08shout out to the sponsor of this entire
14:36:10Panda series, and that is Udemy. Udemy
14:36:12has some of the best courses at the best
14:36:14prices, and it is no exception when it
14:36:16comes to pandas courses. If you want to
14:36:18master pandas, this is the course that I
14:36:19would recommend. It's going to teach you
14:36:20just about everything you need to know
14:36:22about pandas. So, huge shout out to
14:36:24Udemy for sponsoring this Pandanda
14:36:25series. And let's get back to the video.
14:36:27The first thing is obviously the file
14:36:28path. We can specify a separator, which
14:36:32there is no default. So when we're
14:36:34pulling in this CSV, when we're reading
14:36:36in the CSV, it's automatically going to
14:36:38assume it's a comma because it's a
14:36:39commaepparated uh file. You can choose
14:36:42delimters, headers, names, index
14:36:45columns, and a lot of other things as
14:36:47you can see right here. Now, I will say
14:36:49that I don't use almost any of these. Uh
14:36:53the few that I'm going to show you
14:36:54really quickly in just a second are up
14:36:56the very top, but you can do a ton of
14:36:59different things, and I'm just going to
14:37:00slowly go through them. So that's what
14:37:02those are. You can also go down here.
14:37:04This is our doc string and you can see
14:37:07exactly how these parameters work. It'll
14:37:10show you and give you a text and walk
14:37:12you through how to do this. Again, most
14:37:14of these you'll probably never use, but
14:37:17things like a separator could actually
14:37:18be useful and things like a header could
14:37:20be useful because it is possible that
14:37:23you want to either rename your headers
14:37:25or you don't have a header in your CSV
14:37:28and you don't want it to autopop
14:37:30populate that header. So that is
14:37:31something that you can specify. So for
14:37:33example, this header one and I'll show
14:37:35you how to do this. Uh the default
14:37:36behavior is to infer that there are
14:37:38column names. If no names are passed,
14:37:41this behavior is identical to header
14:37:42equals zero. So it's saying that first
14:37:45row or that first index, which is like
14:37:47right here, that zero is going to be
14:37:50read in as a header. But we can come
14:37:53right over here and we'll do comma
14:37:55header is equal to and we could say
14:37:58none. And as you can see, there are no
14:38:01headers now. Instead, it's another
14:38:03index. So, we have indexes on both the
14:38:05x-axis and the y-axis. And so, right
14:38:08now, we have this zero and one index
14:38:10indicating the first column and the
14:38:12second column. If we want to specify
14:38:14those names, we can say the header
14:38:16equals none. Then we can say names is
14:38:19equal to and we'll give it a list. And
14:38:22so, the first one was country and what's
14:38:25that second one? Oh, region. So, they're
14:38:28right here. That's the first um the
14:38:30first row, but we'll rename it and we'll
14:38:32just say country and region. And when we
14:38:36run that, we've now populated the
14:38:37country and the region. Uh we're just
14:38:39pretending that our CSV does not have
14:38:41these values in it and we have to name
14:38:42it ourselves. That's how you do it. But
14:38:45let's get rid of all that because we
14:38:47actually do want those in there. So,
14:38:48we're just going to get rid of those and
14:38:50read it in as normal. And there we go.
14:38:54Now, typically when you're reading in a
14:38:55file, what you need to do is you want to
14:38:58assign that to a variable. Almost always
14:39:00when you see any tutorial or anybody
14:39:03online or even when you're actually
14:39:05working, people will say DF is equal to.
14:39:08DF stands for dataf frame. Again, this
14:39:10is a dataf frame. In the next video in
14:39:13this series, I'm going to walk through
14:39:14what a series is as well as what a dataf
14:39:17frame is because that's pretty important
14:39:18to know when you're working with these
14:39:20data frames. But we'll assign it to this
14:39:22value and then we'll say we'll call it
14:39:24by saying df and we'll run it. And
14:39:27that's typically how you'll do things
14:39:28because [clears throat] you want to save
14:39:29this data frame. So later on you can do
14:39:31things like dataf frame dot and you can
14:39:34uh you know pass in different modules
14:39:36but you can't really do that. It's not
14:39:38as easy to do it if you're calling this
14:39:39entire CSV and importing it every time.
14:39:42So let's copy this because now we're
14:39:45going to import a different type of
14:39:47file. So now we've been doing read CSV,
14:39:50but we can also import text files. Now
14:39:53you can do that with the read CSV. We
14:39:55can import text files. Let's look at
14:39:57this one. We have the same one. It's
14:39:59countries of the world except now it's a
14:40:00text file cuz I just converted it for
14:40:02this video. I'll copy that as a path.
14:40:05And so now when we do this, oops, let me
14:40:07get those
14:40:09quotes in there. It'll say world.txt.
14:40:12It will still work. As you can see, this
14:40:15did not import properly. Um, we have
14:40:17this country back slasht region and then
14:40:20all of our values are the exact same
14:40:21with this backslash t. That's because we
14:40:23need to use a separator. And I'll show
14:40:26you in just a little bit how we can do
14:40:27this in a different way. But with that
14:40:29read cvv, this is how we can do it.
14:40:31We'll just say is equal to we need to do
14:40:35back slasht. Now let's try running this.
14:40:38And as you can see, it now has it broken
14:40:40out into country and region. We could
14:40:43also do it the more proper way. Okay.
14:40:45And this is the way you should do it.
14:40:46And I'll get rid of these really
14:40:48quickly. But just want to keep them
14:40:50there in case you want to see that. But
14:40:52you can also do read table. And let's
14:40:57get rid of this separator. And now we
14:40:59have no separator. It's just reading it
14:41:01in as a table. Let's run this. And it
14:41:03reads it in properly the first time.
14:41:05This read table can be used for tons of
14:41:08different data types, but typically I've
14:41:09been using it for like text files. Um,
14:41:11we can also read in that CSV. So, let's
14:41:14change this right here to CSV. We can
14:41:16read it in as a CSV, but just like we
14:41:19did in the last one when we read in the
14:41:20text file using read CSV, this read
14:41:23table, you're going to need to specify
14:41:24the separator. So, I'll just copy this
14:41:28and we'll say comma. And now it reads it
14:41:32in properly. Again, you can use that for
14:41:34a ton of different file types, but you
14:41:35just need to specify a few more things
14:41:37if you don't want to use the more
14:41:38specific readers function when you're
14:41:41using pandas. Now, let's copy this
14:41:43again. We're going to go right down
14:41:45here. And now, let's do JSON files. JSON
14:41:49files usually hold semistructured data,
14:41:51um, which is definitely different than
14:41:53very structured data like a CSV where it
14:41:55has columns and rows. So, let's go to
14:41:58our file explorer. We have this JSON
14:42:01sample. We will copy this in as path.
14:42:07Let's paste it right here. And we'll do
14:42:09read_json.
14:42:11Again, these different functions were
14:42:12built out specifically for these file
14:42:15types. That's why, you know, each one
14:42:17has a different name. So, now we're
14:42:18reading this in as the JSON.
14:42:21Let's read it in. And it read it in
14:42:24properly.
14:42:27Now, let's go ahead and copy this and
14:42:29take a look at Excel files because Excel
14:42:31files are a little bit different than
14:42:32other ones that we've looked at. Um, so
14:42:35let's just do readers
14:42:37cell
14:42:39and let's go down to our file explorer
14:42:41and let's actually open up this
14:42:43workbook. As you can see, we have sheet
14:42:46one right here, but we also have this
14:42:48world population which has a lot more
14:42:50data. Let's say we just wanted to read
14:42:52in sheet one. We can do that or by
14:42:55default it's going to read in this world
14:42:57population because it's the first sheet
14:42:58in the Excel file. Well, let's go ahead
14:43:01and take a look at that. Let's get out
14:43:03of here. And let's say, oops, I forgot
14:43:06to copy the file path. Let's go ahead
14:43:09and copy as path.
14:43:12And we'll put it right here.
14:43:15And let's just read it in with no
14:43:17arguments or anything in there or no
14:43:19parameters. When we read it in, it's
14:43:21reading in that very first sheet. So,
14:43:24this is the one that has all of the
14:43:25data. Now, let's say we wanted to read
14:43:27in that extra sheet name or the second
14:43:29sheet name. We'll just go comma
14:43:31sheet_name
14:43:34says equal to and then we can specify
14:43:36sheet was it sheet one like this. Yes it
14:43:39was. So we just had to specify the sheet
14:43:41name right here and then it brought in
14:43:44that sheet instead of the default which
14:43:46is the very first sheet in that Excel.
14:43:48Now that definitely covers a lot of how
14:43:50you read in those files. Again you can
14:43:52come in here and hit shift tab and this
14:43:54plus sign and take a look at all the
14:43:56documentation and you can specify a lot
14:43:58of different things. things that I
14:44:00didn't think were very important for you
14:44:02guys to know, especially if you're just
14:44:03starting out. The ones that we looked at
14:44:05today are what I would say are like the
14:44:07ones that I use almost all the time. So,
14:44:09I wanted to show you those, but if
14:44:11you're interested in any of these other
14:44:12ones or you have very unique data and
14:44:14you need to do that. Um, you know, it's
14:44:16worth really getting in here and
14:44:18figuring things out. A few other things
14:44:20that I wanted to show you just in this
14:44:21kind of first video or this intro video
14:44:23on how to read in files. Um, one thing
14:44:26that you may have noticed, especially in
14:44:27this file right here, is we're only
14:44:30looking at the first five and then the
14:44:33last five. So, if we wanted to see all
14:44:35the data, all the data is in these like
14:44:37little three dots right here, right? We
14:44:39want to be able to see that data. But
14:44:43right now, we can't. And that's because
14:44:44of some settings that are already within
14:44:46pandas. And all we need to do is change
14:44:49that. So, this one has 234 rows and four
14:44:52columns. So, obviously, we can see all
14:44:53the columns. Well, let's just change the
14:44:55rows. All we'll say is PD set
14:45:00option. Now, what we need to do is we're
14:45:02going to change the rows. We're not
14:45:04going to change the columns, at least
14:45:06not on this one. So, we'll say quote
14:45:09display
14:45:13rows. Now, if we just run this for
14:45:16whatever data we bring in, it's going to
14:45:18be able to show the max rows. And then
14:45:19we'll say 235.
14:45:22Although there's 234 rows. I'm just
14:45:24going to be safe. Let's run this.
14:45:27And now it has changed it. So let's read
14:45:29in this file again. And you'll see how
14:45:31it's changed. Now we have all of the
14:45:34numbers. And we have this little bar on
14:45:37the right that allows us to go down all
14:45:39the way to the bottom and all the way to
14:45:41the top. So now we can actually look and
14:45:43kind of skim and see our values. I like
14:45:45that better than just having that, you
14:45:47know, shorter version. Um, we can do the
14:45:50exact same thing on columns as well. So,
14:45:52if we look at this one, this is our JSON
14:45:54file. It has the same thing right here.
14:45:56We have what was it 38 columns, but we
14:45:59can only see I think it's maybe it's 20
14:46:02or something like that. I can't
14:46:03remember. Um, but we have 38. We can
14:46:05only see like let's say 15 of them or 20
14:46:07of them. We'll do the exact same thing
14:46:10and we'll just say PD set options
14:46:15doc columns and we'll set that to 40 for
14:46:20that one. When we run this, oops, let's
14:46:23get over here. When we run this one
14:46:26again, we can now scroll over and see
14:46:29every single one of our columns. Now,
14:46:31that one is a, in my opinion, a lot more
14:46:33useful. I like being able to see every
14:46:35single column. So definitely something
14:46:37that you should be using, especially
14:46:39when you have these really large files.
14:46:41You want to be able to see a lot of the
14:46:42data and a lot of the columns. So when
14:46:44you're slicing and dicing and doing all
14:46:46the things we're about to learn in this
14:46:47panda series, you know, you know what
14:46:49you're looking at. I also want to show
14:46:51you just how to kind of look at your
14:46:53data in these data frames as well.
14:46:55That's also pretty important. So let's
14:46:56go right down here. And the very last
14:46:58one that we imported was this one right
14:47:01here, this read Excel. So this data
14:47:02frame is the only one that's going to
14:47:04read in. Let's run it. Um, this is the
14:47:07last one to be run. So, this variable
14:47:09right here, DF, uh, it won't be applied
14:47:11to all these other ones. Um, which we
14:47:13can always go back and change those.
14:47:15Typically, you'll do something like
14:47:16dataf frame 2. You want to do something
14:47:18like that. Um, so let's keep dataf frame
14:47:202. Oops. So, what we're going to do is
14:47:23we're going to bring dataf frame 2 right
14:47:25down here. And we want to take a look at
14:47:27some of this data. We want to know a
14:47:28little bit more about it. Something that
14:47:30you can do is dataf frame 2.info.
14:47:33And we'll do an open parenthesis. And
14:47:35when we run this, it's going to give us
14:47:37a really quick breakdown of a little bit
14:47:39of our data. So we have our columns
14:47:41right here. Rank, CCA3, country, and
14:47:44capital. It's saying we have 234 values
14:47:48in those columns. Because there's 234
14:47:52scroll up here. Because there's 234
14:47:55uh rows, that tells me that there's no
14:47:58missing data in here, at least not, you
14:48:00know, completely missing like null
14:48:02values. there is something in each of
14:48:04those rows. The count tells me it's
14:48:06non-null, so there's no null values. And
14:48:08it tells me the data type. So, it's
14:48:09reading in as an integer, an object, an
14:48:11object, and an object. And it also tells
14:48:14us how much memory it's using, which is
14:48:16also pretty neat because when you get
14:48:17really really large data types, memory
14:48:20usage and and knowing how to work around
14:48:21that stuff does become more important
14:48:23than when you're working at these really
14:48:25small, you know, sample sizes that we're
14:48:27looking at. We can also do oops, let me
14:48:30get rid of that. can also do dataf frame
14:48:322 and we'll do shape. And for this one,
14:48:36we do not need the parenthesis.
14:48:39And all this is going to tell us is we
14:48:40have 234 rows and four columns. We're
14:48:44also able to look at uh the first few
14:48:47values or rows in each of these data
14:48:50frames. So we can just say dataf frame 2
14:48:52head. And if we do that, it's going to
14:48:54give us the first five values. But we
14:48:56can specify how many we want. We can say
14:48:59head 10. It'll give us the first 10 rows
14:49:01right here. We can do the exact same
14:49:04thing. And let's go right down here and
14:49:06we'll say tail. So they'll give us the
14:49:08last 10 rows within our data frame. Now
14:49:12let's copy this. And let's say we don't
14:49:14want to actually look at all of these
14:49:16values or all these columns. We can
14:49:18specify that by saying df2 and oops,
14:49:21let's get rid of all of this.
14:49:24And we'll say with a quote we'll say
14:49:27rank. And now we can take just a look
14:49:30the rank data. Now we can't do that by
14:49:33doing the index or at least not like
14:49:35this. If we want to use this index that
14:49:38is right here, we can. But there's a
14:49:40very special function called LO and I
14:49:42look for that. And I'm going to have an
14:49:44entire video on this because it does get
14:49:45a little bit more complex. But there's
14:49:48DF2. Loc. And there's lo and stands for
14:49:52location and eyelocation. That's only
14:49:54for the indexes. Whether it's the xaxis
14:49:57or the y-axis, those are the indexes.
14:49:59And for location, it's looking for the
14:50:02actual text, the actual string of the
14:50:04index. So if we come up here, that dataf
14:50:07frame 2, we can specify 224 and it'll
14:50:10give us this information right here in a
14:50:12little different format. So let's go
14:50:15bracket and we'll say 224. And when we
14:50:18run this, it gives us our rank CCA
14:50:21country capital with our values over
14:50:23here. Kind of like a dictionary almost.
14:50:26Now let's copy this and we'll say
14:50:28df2.lo.
14:50:31And right now these look the exact same,
14:50:34but we haven't really talked a lot about
14:50:36changing the index. And you can change
14:50:38the index to a string or a different
14:50:40column or something like that. And we'll
14:50:42look at that in future videos. The
14:50:43eyelock looks at the integer location.
14:50:45So even if these um let's go right up
14:50:48here even if this index had changed to
14:50:51let's say this rank or the CCA3 or
14:50:53country or whatever you make this index
14:50:55the look will still look at the integer
14:50:58location. So that 224 would still be 224
14:51:01even if it was Usuzbekistan.
14:51:03So then when we look at this, it's going
14:51:05to be the exact same. But if we had
14:51:08changed that index, this loc is the one
14:51:10that we could search on and we could
14:51:12search.
14:51:16Is that how you spellistan?
14:51:19Hey, I nailed it. So that is how you use
14:51:22LO and I look. Again, I just wanted to
14:51:24show you a little bit about how you can
14:51:25look at your data frame or search within
14:51:27your dataf frame. Now, in future videos,
14:51:28I'm going to dive a lot deeper into a
14:51:30lot of the concepts that we just looked
14:51:32at because I just kind of touched on
14:51:33them. I wanted you to have a brief
14:51:35introduction to them so that in future
14:51:37videos, I'm not just dropping everything
14:51:39on you all at once. So, hopefully this
14:51:40was a good quick introduction to those
14:51:42topics. Uh, you should be able to read
14:51:44in a file now, see your data frame, and
14:51:46kind of look at it in a few different
14:51:48ways that we just looked at. And I hope
14:51:49that that was helpful. And if it was, be
14:51:51sure to check out all my other videos on
14:51:53Python and pandas. And if you like this
14:51:55video, be sure to like and subscribe
14:51:56below. and I will see you in the next
14:51:58video.
14:52:00[music]
14:52:10Hello everybody. Today we're going to be
14:52:12looking at filtering and ordering dataf
14:52:13frames in pandas. There are a lot of
14:52:15different ways you can filter and order
14:52:17your data in pandas and I'm going to try
14:52:19to show you all of the main ways that
14:52:21you can do that. So let's kick it off by
14:52:23importing our data set. So we're going
14:52:24to say dataf frame is equal to and we'll
14:52:26say pandas and I need to import my
14:52:30pandas. So we'll say import p andas as
14:52:33pd. That's pretty important I think. Um
14:52:35so pdread
14:52:37csv and we'll do r and then we'll say
14:52:42the world population cv. So let's run
14:52:44this all our data frame right here. And
14:52:48this is the data frame that we're going
14:52:50to be filtering through and ordering in
14:52:52pandas. So, let's kick it off. The first
14:52:55thing that we can do is filter based off
14:52:57of the columns. So, the data within our
14:53:00columns. So, Asia, Europe, Africa, or
14:53:02whatever data we may have in that
14:53:04column. Let's go right down here. We're
14:53:06going to say DF. And then within it,
14:53:09we're going to specify what column we're
14:53:11going to be filtering on. So, we're
14:53:12going to say DF with another bracket,
14:53:14and we'll say rank. So, we're going to
14:53:16be looking at this rank column right
14:53:18here. And then we'll say in that rank
14:53:21column we want to do greater than 10.
14:53:24And that's actually going to be a lot of
14:53:25them. Let's do less than. So when we run
14:53:27this, it's only going to return these
14:53:30values that are less than 10. We can
14:53:32also do less than or equal to, you know,
14:53:34all of these um comparison operators. So
14:53:37less than or equal to. So now we have
14:53:39all of the ranks 1 through 10. Now if we
14:53:42look at these countries, we can specify
14:53:44by specific values almost exactly like
14:53:46we did here. But instead of doing a
14:53:48comparison operator like we did right
14:53:50here and including those names, let's
14:53:52say Bangladesh and Brazil, we can use
14:53:55the isin function almost like an in
14:53:57function in SQL if you know SQL. So
14:53:59let's go right down here and we're going
14:54:01to say specific
14:54:04countries. So right now we're just going
14:54:06to make a list of the countries that we
14:54:08want and then we'll say Bangladesh
14:54:14and Brazil.
14:54:17So let's go right down here and we'll
14:54:20say okay for these specific countries
14:54:22from the data frame let's do our bracket
14:54:25we'll say in this country column so
14:54:28we'll do data frame and then another
14:54:30bracket for country. So in this country
14:54:34column we can do is in and then an open
14:54:38parenthesis and then look for our
14:54:40specific countries. So, we're looking at
14:54:43just this column and we're saying is in.
14:54:45So, we're looking at are these values
14:54:47within this column and we're getting
14:54:50this error and this looks very very odd.
14:54:53Let me um this doesn't look right. There
14:54:56we go. I just had some syntax errors. I
14:54:59apologize. Made it way more complicated
14:55:01than it needed to be. But here's how you
14:55:03use this is in function. So, we're
14:55:06looking at Bangladesh and Brazil. And we
14:55:08return those rows with Bangladesh and
14:55:10Brazil. Really quickly, I wanted to give
14:55:12a huge shout out to the sponsor of this
14:55:14entire Pandanda series, and that is
14:55:15Udemy. Udemy has some of the best
14:55:17courses at the best prices, and it is no
14:55:19exception when it comes to pandas
14:55:21courses. If you want to master pandas,
14:55:22this is the course that I would
14:55:24recommend. It's going to teach you just
14:55:25about everything you need to know about
14:55:26pandas. So, huge shout out to Udemy for
14:55:28sponsoring this Panda series. And let's
14:55:30get back to the video. We can also do a
14:55:32contains function kind of similar to is
14:55:35in, except it's more like the like in
14:55:38SQL as well. I'm comparing a lot of this
14:55:40to SQL because when you're filtering
14:55:41things, I always my brain always goes to
14:55:43SQL. But in pandas, it's called the
14:55:46contains. So let's do let's actually
14:55:49copy this because I don't want to make
14:55:50the same mistake again. Let's do that.
14:55:53And we'll do the bracket, but instead of
14:55:57in we're going to docontains
14:56:00and then an open parenthesis. So we're
14:56:03going to be looking for a string. If it
14:56:05contain if it contains let's do United
14:56:09almost like United States or or any
14:56:11other United. So let's run this and as
14:56:14you can see we have United Arab
14:56:16Emirates, United Kingdom, United States,
14:56:18United States Virgin Islands. So we can
14:56:20kind of search for a specific string or
14:56:23a number or a value within our data or
14:56:26within that column of country. Now so
14:56:28far we've only been looking at how you
14:56:30can filter on these columns. We can also
14:56:32filter based off of the index as well.
14:56:35And there's two different ways you can
14:56:37do it or two of the main ways. There's
14:56:39filter and then there's lo and lo stands
14:56:42for location and i look stands for
14:56:44integer location. And if you've seen
14:56:46other previous videos, I've kind of
14:56:48mentioned those so we can take a quick
14:56:49look at all of those. So really quickly,
14:56:52we need to set an index because the
14:56:54index right now is uh not the best.
14:56:56We'll set our index to country.
14:57:00So let's say df2
14:57:03is equal to df set_index
14:57:08and we'll say country. I'm just doing
14:57:10df2 because later on I want to use that
14:57:13data frame again. So I'm just going to
14:57:14assign it to another data frame so we
14:57:17can just easily switch back and forth.
14:57:19So now we have this index as the country
14:57:22and what we can do is use the filter
14:57:24function. So let's go down here. We'll
14:57:26say df2.filter
14:57:31and we'll do an open parenthesis. And
14:57:32now we can specify our items. So these
14:57:35are actually going to be specifying
14:57:36which columns we want to keep. So we're
14:57:38going to say items is equal to then
14:57:41we'll make a list. We'll say continent.
14:57:44Hope that's how we spell continent. I'm
14:57:46always messing up with my uh
14:57:48my stuff here, my spelling. Then we'll
14:57:50do CCA3 because why not? You can specify
14:57:53whichever ones you want. When we run
14:57:56this, it's going to only bring in those
14:57:58two columns. Now, by default, it's
14:58:01choosing the axis for us. But we can
14:58:03also specify which axis we want to
14:58:05search on. So, if we say axis is equal
14:58:08to zero, it's actually going to search
14:58:10this axis. This is the zero axis. This
14:58:12is the one axis. So, where our columns
14:58:15are is one. So, if we go back and do
14:58:17one, we're searching on that one axis or
14:58:20those header axises again. And this is
14:58:22the default, but you can specify that.
14:58:24So, if you just want to search on, you
14:58:27know, filtering right here, you can do
14:58:29that. And let's actually copy this and
14:58:32do that right down here, just so you can
14:58:33see what it looks like. But let's search
14:58:35for Zimbabwe. And we'll do Zimbabwe. And
14:58:39we'll be looking at the zero axis, which
14:58:42is the up and down on the left hand
14:58:44side. And when we filter on that, we can
14:58:46filter by Zimbabwe by looking just at
14:58:49the country index. We can also use the
14:58:52like just like we did before. And I'll
14:58:54show you the exact same demonstration
14:58:56that we did, which you can say like is
14:58:59equal to and instead of having to put in
14:59:01a concrete um text, you can just say
14:59:04United just like we did before and we're
14:59:06searching where the access is equal to
14:59:07zero, which again is this left-handed
14:59:10access. So now we're looking for United
14:59:12and it's going to give us all of the
14:59:14countries or all the indexed values that
14:59:16have United in it. Like we were talking
14:59:18about before, we also have LO and I
14:59:20look. So we can say dataf frame 2.lo.
14:59:26Now this is a specific value. So we'll
14:59:29do United States. So location is just
14:59:32looking at the actual name or the value
14:59:35of it, not its position. So if we search
14:59:37for United States, it's going to give us
14:59:39this right here where it gives us all of
14:59:41the columns for United States and then
14:59:43all of the uh values for United States.
14:59:47Or we can do
14:59:49the eyel which is the integer location
14:59:52which is not the exact same because
14:59:55we're looking at the string for the lo.
14:59:58We're looking at this string but
15:00:00underneath it there still is a position
15:00:02that's that integer location. Let's do a
15:00:04completely random one. Let's just say
15:00:07three. If we look at the third position,
15:00:09it's going to give us ASM, which I'm not
15:00:12exactly sure what it is, but it still
15:00:14gives us basically the same kind of
15:00:15output, which is the columns and the
15:00:18values. So, that's another way that you
15:00:19can search within your index when you're
15:00:21actually trying to filter down that
15:00:23data. Now, let's go look at the order
15:00:26by, and let's start with the very first
15:00:28one that we looked at. Let's do data
15:00:29frame. That's why I kept it because I
15:00:31wanted to use it later. Now we can sort
15:00:33and order these values instead of it
15:00:35just being kind of a a jumbled mess in
15:00:37here. We can sort these columns however
15:00:40we would like. Ascending, descending,
15:00:42multiple columns, single columns. And
15:00:44let's look at how to do that. So we'll
15:00:45say data frame and then we'll do dataf
15:00:47frame. Look at rank again just like we
15:00:50were doing above. And let's do dataf
15:00:53frame where it's less than 10. I should
15:00:55have just gone and copied this. I
15:00:57apologize. So now we have this data
15:00:59frame that is greater than 10. Now we
15:01:02can do dots sort
15:01:06values and this is the function that's
15:01:08going to allow us to sort everything
15:01:10that we want to sort. So we can do by is
15:01:13equal to and we'll just order it by the
15:01:16exact same thing that we were doing uh
15:01:17or calling it on. So we'll do rank. So
15:01:20now what this is going to do, it's going
15:01:22to order our rank column. And as you can
15:01:25see, it did that 1 2 3 4 5. We can also
15:01:28do it with ascending or descending. So
15:01:31if you want to, you can look in here and
15:01:33see what you can do. So we'll do
15:01:34ascending. We'll say that's equal to
15:01:37true.
15:01:39And so that's the automatic default. So
15:01:41that didn't change anything. But if we
15:01:43say false, it's going to be descending
15:01:45from highest to lowest. So now we have
15:01:47it in the opposite direction. Now we
15:01:49don't have to just order or sort this on
15:01:52one single column. We can do multiple
15:01:54columns and we can do that by making a
15:01:56list right here. Whoops. Make a list
15:02:01just like that. And we'll input
15:02:03different ones as well. So now let's
15:02:05input our country.
15:02:08And when we run this, it will give us
15:02:10rank of 9876
15:02:12as well as the country of Russia,
15:02:15Bangladesh, Brazil. Now, if you noticed,
15:02:18the country really didn't change because
15:02:19the rank stayed the exact same. That's
15:02:22because there's an order of importance
15:02:23here and it starts with the very first
15:02:25one. If we change this around and we
15:02:29look at this one and put a comma right
15:02:32here. Now the country is going to be
15:02:34descended and the rank would come
15:02:36second. So it's not going the rank isn't
15:02:38going to really have any effect here. So
15:02:41now we have the country United States,
15:02:42Russia, Pakistan and the rank really
15:02:45didn't get ordered at all. Now, if we
15:02:47want to see how that can actually work,
15:02:49let's do continent right here. And let's
15:02:52actually put it right here and do
15:02:54country here. So, if we run this, it's
15:02:57first going to come and it's going to
15:02:58organize or sort the continent. Then,
15:03:02it's going to come back and go to the
15:03:03country and then it's going to sort the
15:03:05country. So, keep So, keep your eye
15:03:08right here in this Asia area because
15:03:10we're going to sort this differently
15:03:12than ascending. So we have ascending
15:03:14false and that applies to both of these.
15:03:16It's false and false. But we can specify
15:03:19which one we want to do. We can do a
15:03:21false here and a true here. So we'll do
15:03:23false, true. And what this is going to
15:03:26do is it's going to say false for the
15:03:28continent. So the continent right here
15:03:30is going to stay the exact same. And so
15:03:32that is a lot of how you can filter and
15:03:35order your data within pandas. I hope
15:03:37that this was helpful. I hope that you
15:03:38enjoyed this video. If you liked it, be
15:03:40sure to like and subscribe below. Check
15:03:42out all my other videos on Python and
15:03:44pandas and I will see you in the next
15:03:45video.
15:03:48[music]
15:03:58Hello everybody. Today we're going to be
15:04:00looking at indexing in pandas. If you
15:04:02remember from previous videos, the index
15:04:04is an object that stores the access
15:04:06labels for all pandas objects. The index
15:04:08in a dataf frame is extremely useful
15:04:10because it's customizable and you can
15:04:12also search and filter based off of that
15:04:14index. In this video, we're going to
15:04:16talk all about indexing, how you can
15:04:17change the index and customize that, as
15:04:19well as how you can search and filter on
15:04:21that index. And then we're also going to
15:04:23be looking at something a little bit
15:04:24more advanced called multi-indexing. And
15:04:27you won't always use it, but it's really
15:04:29good to know in case you come across a
15:04:31data frame that has that in it. So,
15:04:33let's get started by importing pandas.
15:04:36import pandas as pd. Now we'll get our
15:04:39first data frame. We'll say df is equal
15:04:41to pdread_csv.
15:04:44And I've already copied this, but we're
15:04:47going to do r and we're going to put
15:04:49this file path. So I have this world
15:04:51population cv. I will have that in the
15:04:54description just like I do in all of my
15:04:56other videos. Let's run df and let's
15:04:59take a look at this data frame. So we
15:05:01have a lot of information here. We have
15:05:03rank, country, continent, population, as
15:05:07well as the default index from zero all
15:05:09the way up to 233. Now, if you haven't
15:05:11watched any of my previous videos on
15:05:13pandas, the index is pretty important,
15:05:15and it's basically just a number or a
15:05:17label for each row. It doesn't even
15:05:19necessarily have to be a unique number.
15:05:22Um, you can create or add an index
15:05:24yourself if you want to, and it doesn't
15:05:26have to be unique, but it it really
15:05:28should be unique, especially if I want
15:05:30to use it appropriately for what we're
15:05:32doing. the country is actually going to
15:05:33be a pretty great index because the
15:05:36country, you know, is going to be all
15:05:38unique because we're looking at every
15:05:39single row as a different um country as
15:05:42well as the population. So, let's go
15:05:44ahead and create this country or add
15:05:45this country as our index. Now, we can
15:05:48do this in a lot of different ways, but
15:05:50the first way that you can do this if
15:05:52you already know what you are going to
15:05:54create that index on is we can just go
15:05:56right in here when we're reading in this
15:05:57file and we'll say comma index
15:06:01oops I spelled that completely wrong
15:06:03index column and we'll say that is equal
15:06:06to and then we're going to say quote
15:06:09country. So, we're taking this country
15:06:12and we're going to assign it as the
15:06:13index. Now, let's read this in. And as
15:06:16you can see, this is our index. Now, it
15:06:19looks a little bit different. We didn't
15:06:21have this country header right here,
15:06:23which is specifying that this is still
15:06:24the country. But you can tell that this
15:06:26is the index based off the um bold
15:06:29letters, as well as it being on the far
15:06:30left. And all the regular columns for
15:06:33the data is over here, while the country
15:06:35header is right here, and it's lower
15:06:37than all the others. Just a quick way
15:06:39that you can see that that is the index.
15:06:41Now before we move on, I want to show
15:06:42you some other ways that you can do this
15:06:44as well. But I'm going to show you how
15:06:46to reverse this index before we move on.
15:06:49And we'll say dataf frame. So we had our
15:06:52dataf frame right here. So we have dataf
15:06:54frame dot we'll say reset_index.
15:06:58And then we'll say in place is equal to
15:07:00true, which means we don't have to
15:07:02assign this to another variable and all
15:07:04that stuff. It'll just be true. So now
15:07:06when we run that data frame again, the
15:07:08index was reset to the default numbers.
15:07:11So now let's go down here and I'll show
15:07:13you how to do this in a different way.
15:07:14You can do df do we'll say set index and
15:07:18then we'll just say country. So very
15:07:21similar to when we were reading in that
15:07:22file and we said set the index or that
15:07:24index column, we set index column equals
15:07:27country. If we do this and we run it in,
15:07:30it works. But if we say dataf frame
15:07:33right down here, it's not going to save
15:07:35that. If we want to save it just like we
15:07:37did above, we're going to say in place
15:07:40is equal to true. That is going to save
15:07:43it to where we don't have to assign it
15:07:45another variable. So now when we run
15:07:47this, the data frame right here, which
15:07:49is going to populate this, the data
15:07:50frame is going to say in place is equal
15:07:52to true. So that country will now be our
15:07:54index again. Let's run this. And there
15:07:57we go. Really quickly, I wanted to give
15:07:59a huge shout out to the sponsor of this
15:08:01entire Panda series, and that is Udemy.
15:08:03Udemy has some of the best courses at
15:08:05the best prices and it is no exception
15:08:07when it comes to pandas courses. If you
15:08:09want to master pandas, this is the
15:08:10course that I would recommend. It's
15:08:11going to teach you just about everything
15:08:13you need to know about pandas. So, huge
15:08:15shout out to Udemy for sponsoring this
15:08:16pandas series. And let's get back to the
15:08:18video. Now, what's really great about
15:08:19this index is we're able to search based
15:08:21off just this index. And so, we can
15:08:23filter on it and basically look through
15:08:25our data with it. And there are two
15:08:27different ways that you can do that. At
15:08:28least this is a very common way that
15:08:30people who use pandas will do to kind of
15:08:32search through that index. The first one
15:08:34is called lock and there's lock and
15:08:36eyelock. That stands for location or
15:08:38integer location. Let's look at lock
15:08:41first. Let's say df.lock
15:08:44and then we'll do a bracket. Now we're
15:08:46able to specify the actual string, the
15:08:48label. So let's go right up here and
15:08:50let's say Albania.
15:08:52So we'll say Albania. So again, this is
15:08:55just looking at the location. Let's run
15:08:57this.
15:08:58Now it's going to bring up all the
15:09:00Albania data just like here where it's
15:09:02kind of looks like a column in a column
15:09:05and we can get this exact same data but
15:09:08using eyelock right here. And when we
15:09:12ran lock we're searching based off
15:09:14Albania which is in the 01 position. So
15:09:17if we actually pull the one position for
15:09:19that integer
15:09:21the eyelock we can look at the one
15:09:25position and this should give us the
15:09:27exact same data. Now let's take a look
15:09:29at multi-indexing and we'll come back to
15:09:32a little bit of this in a second. So
15:09:35multi-indexing is creating multiple
15:09:37indexes. We're not just going to create
15:09:39the country as the index. Now we're
15:09:41going to add an additional index on top
15:09:43of that. So let's pull up our data
15:09:45frame. Right now we have the country but
15:09:47let's do dotreset
15:09:50index
15:09:51and we'll say in place equals true.
15:09:55Oops. Let's run it. So now we have our
15:09:58data frame. Now let's set our index. But
15:10:01this time when we set our index we're
15:10:03going to add the country as the index as
15:10:05well as the continent as an index. So
15:10:08we'll say dataf frame set_index.
15:10:12Then we'll do a parenthesis and instead
15:10:14of just doing country like we did
15:10:16before, we're going to create a list.
15:10:19Oops. And we'll do it like that. And
15:10:22then we'll say
15:10:24oops continent
15:10:26and separated by a comma. So we have
15:10:29continent and country. Let's just say in
15:10:33place is equal to true. Now when we run
15:10:36this, we're going to have two indexes.
15:10:38Let's see what this looks like.
15:10:42And let's run this. So now we have
15:10:45country as well as continent as our
15:10:48index. Now you may notice that these
15:10:50indexes are repeating themselves on this
15:10:53continent index. We have Europe right
15:10:55here and Europe right here as well as
15:10:58Asia and Asia. And it looks a little bit
15:11:01funky, but we are able to sort these
15:11:04values and make it look a lot better. So
15:11:06let's go ahead and try this. We'll do df
15:11:09do. sort_index.
15:11:12And when we run this, it should sort our
15:11:14index alphabetically. And we can also
15:11:16look in here and see what kind of things
15:11:19we can, you know, specify. We can
15:11:21specify the axis, but it's automatically
15:11:23going to be looking at the zero. This is
15:11:25zero and this is one. So we have two
15:11:27axes within our data frame. You can
15:11:29choose the level, whether it's ascending
15:11:31or not ascending, in place, kind,
15:11:34string, sort, remaining, all of these
15:11:36different things. The only one that I
15:11:38really, you know, think is worth looking
15:11:40at is the ascending. We already know
15:11:41some of these other ones. But if we look
15:11:43at ascending,
15:11:45let's run it. Now, it's sorted these.
15:11:47And so now it's kind of grouped
15:11:49together. So we have Africa and all the
15:11:51African ones as well as South America
15:11:53and all the South American ones. Let's
15:11:56really quickly say PD do
15:12:00set option
15:12:03and we'll say display.mmax
15:12:07doc columns and just like this let's run
15:12:10it and I need to specify whoops specify
15:12:14right here let's see how many rows we
15:12:16have
15:12:18235 so let's do 235
15:12:21let's run this and now when we run this
15:12:24you can see that Africa is all grouped
15:12:26together and all the countries are in
15:12:28alphabetical order under it. And then we
15:12:30go all the way down to Asia and again
15:12:33just all in alphabetical order. If we
15:12:35wanted to we could say ascending
15:12:38equals true
15:12:40and then when we run this oh meant say
15:12:43false and then when we run this it's the
15:12:46exact opposite. So it starts with South
15:12:47America the last one and then goes in
15:12:49reverse alphabetical order. We could
15:12:51also say false make it a list and do
15:12:54comma true
15:12:57and just like this and then it would
15:12:59sort this first column as false and this
15:13:02next column as true. So you can really
15:13:04customize it but you know for what we're
15:13:06doing we don't need any of that. We just
15:13:08need to be able to see this right here.
15:13:09So now when we try to search by our
15:13:11index like we did before we did dataf
15:13:14frame.lo.
15:13:16Now when we did that and we said you
15:13:18know let's say Angola when we specified
15:13:21Angola it's not going to work properly
15:13:24because it's searching in this first
15:13:26index for the first string that we have
15:13:29we can search Africa
15:13:32let's search for Africa
15:13:35and now we have all of the African
15:13:37countries and if we want to specify to
15:13:40Angola we can also go down another level
15:13:43oops by doing angola
15:13:46And now we have what we were looking at
15:13:48before where we're calling all of the
15:13:50data within those. But we couldn't do it
15:13:52just based off Africa because we had an
15:13:54additional index right here. So once we
15:13:56called both indexes, now we get this
15:13:58view. But let's look at that eyel.
15:14:13So, you think it may pull up Angola.
15:14:16Let's go ahead and run this. And it's
15:14:18still pulling up Albania. Let's go right
15:14:21up here. If you remember when we didn't
15:14:24have the multiple indexes, it was
15:14:26pulling up Albania. The difference when
15:14:28you're doing these multi-indexes is that
15:14:31the LO is able to specify this, whereas
15:14:35this one does not go based off that
15:14:37multi-indexing. it's going to go based
15:14:39off the initial index or the
15:14:41integerbased index. So that's a lot
15:14:44about indexing in pandas. We'll cover
15:14:46even a few more things in future videos
15:14:48as we get more and more into pandas. But
15:14:51this is a lot of what indexing looks
15:14:53like within pandas. And again, super
15:14:55important to learn how to do and know
15:14:56how to do because it's a pretty
15:14:57important building block as we go
15:14:59through this pandas series. So I hope
15:15:02you enjoyed this video on indexing. If
15:15:04you did, be sure to like and subscribe
15:15:06below and I will see you in the next
15:15:07video.
15:15:09>> [music]
15:15:20>> Hello everybody. Today we're going to be
15:15:22taking a look at the group by function
15:15:23and aggregating within pandas. Group by
15:15:26is going to group together the values in
15:15:28a column and display them all on the
15:15:30same row. And this allows you to perform
15:15:32aggregate functions on those groupings.
15:15:35So let's start reading in our data and
15:15:37take a look. So we're going to do import
15:15:39pandas as pd.
15:15:42And then we're going to say our data
15:15:43frame is equal to and we'll say pd read
15:15:48csv.
15:15:49We'll do an open parenthesis r and our
15:15:52file path. And we're going to be looking
15:15:54at the flavors CSV right here. So right
15:15:57here we have our flavor of ice cream. We
15:16:00have our base flavor, whether it was
15:16:01vanilla or chocolate, whether I liked it
15:16:04or not, the flavor rating, texture
15:16:06rating, and its overall or its total
15:16:08rating. Now, these are all my own
15:16:10personal scores. So, you know, I've
15:16:12spent years researching this, so these
15:16:13are all very accurate, but this should
15:16:15be a low stress environment to learn
15:16:17group by and the aggregate functions.
15:16:19So, the first thing that we can do is
15:16:21look at our group by. Now, you can't
15:16:24group by well, you can you can group by
15:16:26flavor, but as you can see, these are
15:16:28all unique values. What we need is
15:16:30something that has duplicate values or
15:16:32or similar values on different rows that
15:16:35I'll group together. So, this base
15:16:37flavor is actually a perfect one to
15:16:39group it on. And we'll do that by saying
15:16:42df.group
15:16:44by do an open parenthesis. And we'll
15:16:46just specify base flavor. And this will
15:16:50then group together those values. And I
15:16:52need to make sure I can spell properly.
15:16:55This will group those flavors together.
15:16:57So let's run this. And as you can see,
15:17:00it actually is its own object. So it has
15:17:02a group by dataf frame group by object.
15:17:05So now that we've grouped them, let's
15:17:07give it a variable. So we'll say group
15:17:10by
15:17:12frame. Let's say that's equal to. Let's
15:17:15copy this. We'll run it. And now what we
15:17:19need to do is run our aggregations in
15:17:21order to get an output. So we're going
15:17:23to say mean and that's all we're going
15:17:27to put just for now just to get an
15:17:29output that we can take a look off and
15:17:30then we'll build from there. So let's go
15:17:32ahead and run this. And right here we
15:17:36have our base flavor which is now saying
15:17:38is the index of chocolate or vanilla.
15:17:41And then it's taking the mean or the
15:17:42average of all the columns that have
15:17:44integers. Notice that it did not take
15:17:47the liked column and it did not take the
15:17:49flavor column because those are strings
15:17:51and they cannot aggregate those and
15:17:52we'll take a look at that later. But it
15:17:54took all the values that have integers
15:17:56and then it gave us the average of those
15:17:58ratings. Really quickly, I wanted to
15:18:00give a huge shout out to the sponsor of
15:18:02this entire Pandanda series and that is
15:18:04Udemy. Udemy has some of the best
15:18:05courses at the best prices and it is no
15:18:07exception when it comes to pandas
15:18:09courses. If you want to master pandas,
15:18:11this is the course that I would
15:18:12recommend. and it's going to teach you
15:18:13just about everything you need to know
15:18:14about pandas. So, huge shout out to
15:18:16Udemy for sponsoring this Panda series
15:18:18and let's get back to the video. So,
15:18:19right off the bat, as averages with
15:18:22chocolate, I have a much higher rating
15:18:23overall than the ones with vanilla
15:18:25bases. Now, we can actually combine all
15:18:28of this together into one line. And we
15:18:30can do something like this. So, we'll
15:18:32say
15:18:34df.group by we'll say mean just like
15:18:39this. And this will actually run it.
15:18:41Before we didn't have any aggregating
15:18:43function on there, so it didn't run. But
15:18:45now that we combine it all into one, it
15:18:47will run properly. Now there are a lot
15:18:49of different aggregate functions, but
15:18:51I'm going to show you some of the most
15:18:52popular ones or the most common ones
15:18:54that you will see. So let's copy this
15:18:56right here. So we can do
15:19:00count. And when we run this, we can look
15:19:02at the count. And this will show us the
15:19:04actual count of the rows that were
15:19:06aggregated. So for chocolate, we had
15:19:08three. So there's going to be three all
15:19:09the way across. And for vanilla, we had
15:19:11six. So, we're looking at a higher count
15:19:14of vanilla, which if you're comparing it
15:19:16to this mean up here, that could be a
15:19:19big skew towards the chocolate because
15:19:21if you have one or two good chocolates,
15:19:23it could really pull the numbers up.
15:19:24Whereas, if you had two good vanillaas,
15:19:26but all the other ones were bad, it
15:19:28pulls that average down. So, knowing the
15:19:30count of something is really good.
15:19:33Let's take a look at the next one. And
15:19:35we can do min and max. And I'll just run
15:19:37these really quickly. We can do min. And
15:19:40when we run this, the first thing that
15:19:42you should notice is that it now has a
15:19:44flavor and a liked column. And that's
15:19:46because min and max will actually look
15:19:48at the first letter in the string or the
15:19:50first set of letters if there are um you
15:19:52know chocolate something. It'll look at
15:19:54the first and then it'll actually
15:19:56populate it. So chocolate with the ch
15:19:59chocolate is the very first or the
15:20:02minimum value for that string. And for a
15:20:05cake batter, that is the minimum value
15:20:07in vanilla as well. Now, with the liked,
15:20:09it's interesting because apparently I
15:20:11liked all the chocolate ones. I'm going
15:20:12to go take a look. So, chocolate I
15:20:14liked. Chocolate I like. Chocolate I
15:20:16like. So, there is no no option in this
15:20:18liked column. So, yes, was the only
15:20:20option. And now, let's look at max.
15:20:22Whoops.
15:20:24And it should do the exact opposite,
15:20:26which is going to take the highest value
15:20:28even if it's a string. So, Rocky Road,
15:20:30the letter R comes later in the
15:20:31alphabet. So, that's what it's looking
15:20:33at. and so does vanilla. And then we
15:20:35have yes as well. And then of course
15:20:38right here it's taking the max value. So
15:20:41before when we were looking at min, I
15:20:42just focused on those, but it still does
15:20:44the exact same thing to these integer um
15:20:47columns as well. So for the max value
15:20:50for vanilla, it was mint chocolate chip.
15:20:52That was our base. So I had a rating of
15:20:5410 for this vanilla row or grouping. And
15:20:58then we can also look at the sum.
15:21:01And there are all the sums for these.
15:21:03And again, it only does integer because
15:21:05we can't add the strings. Here are the
15:21:07sum or the total values for all of them.
15:21:10And for the total values, since we had,
15:21:11you know, six rows that were grouping
15:21:13into this vanilla, we now have a lot or
15:21:16a much higher score for vanilla. Now,
15:21:19that's a really simple way to do your
15:21:20aggregations. But there is actually an
15:21:22aggregation function. And let's take a
15:21:25look at this because this is um a little
15:21:27bit more complex. Although when I write
15:21:29it out or show you hopefully it makes a
15:21:31lot of sense. We can do agg. So this is
15:21:35our aggregate function and what we need
15:21:36to pass into our aggregate function is
15:21:39actually a dictionary. So let's do an
15:21:41open parenthesis and we're going to do a
15:21:43squiggly bracket and then we need to
15:21:46specify what we're going to be
15:21:47aggregating on or what column. So let's
15:21:49do this flavor rating. Let's copy this.
15:21:53We'll do flavor rating and I need to put
15:21:55that as a string. And then we'll do a
15:21:58colon. And now we can specify what
15:22:00aggregate functions we want. So we've
15:22:02done sum, count, mean, min, and max, all
15:22:05of those. And we can actually put all of
15:22:07those into here and perform all of those
15:22:09aggregations on just one column. So
15:22:12let's make a list. And then let's say
15:22:15mean,
15:22:17max,
15:22:19count, and uh what's another one? Sum.
15:22:23So let's do all four of those only on
15:22:26this flavor rating column.
15:22:29And when we run this, we have our base
15:22:31flavor right here, chocolate and
15:22:33vanilla. But now we don't have multiple
15:22:35columns. We have one column with
15:22:38multiple columns of our aggregations.
15:22:40And it is possible to pass in multiple
15:22:43columns like that. So we'll do texture
15:22:46rating.
15:22:47And we'll just come right here and do a
15:22:49comma. Then we'll say uh uh texture
15:22:53rating
15:22:54and then a colon. I don't know why I
15:22:58spelled it out when I copied it, but I
15:23:00did. And then we'll do the exact same
15:23:02ones. And now when we run it, we're
15:23:04getting the exact same columns. Mean,
15:23:06max, count, and sum for flavor rating.
15:23:09Then mean, max, count, and sum for our
15:23:11texture rating. Now, so far, we've only
15:23:13grouped on one column, but we can
15:23:16actually group on multiple columns.
15:23:18Let's go back up here to our data. And I
15:23:20should have just copied this down here.
15:23:22Let's go back down and just look at
15:23:24this. So really, we only grouped it on
15:23:27this base flavor, but you can do
15:23:30multiple groupings or group by multiple
15:23:32columns. So let's do our base flavor,
15:23:34which we did already, as well as the
15:23:37liked column. So we're going to say
15:23:39df.group group by then we'll do an open
15:23:43parenthesis and then instead of just
15:23:45passing through one string we're going
15:23:48to do a list and we'll say base flavor
15:23:53oops comma and then we'll do liked. So
15:23:57now when it groups this, it should put
15:24:00two groupings. And let's run this and
15:24:02just see. Oops, I got to say, let's just
15:24:05do mean.
15:24:07So now we have our chocolate and a
15:24:10vanilla. And remember, chocolate only
15:24:12had yes. So that's the only one that
15:24:14it's going to group on. But vanilla had
15:24:17a no and a yes. So if we look at the
15:24:20vanilla, we have our base flavor
15:24:21vanilla. And then within liked, we have
15:24:24no and a yes, which can show us that
15:24:27within our vanilla, when we group on
15:24:28these, our nos were really low. But our
15:24:31yeses were really high. We actually had
15:24:33a pretty similar rating or very close to
15:24:35the same rating as the ones we really
15:24:37liked in chocolate. And just like we did
15:24:39above, we can take this
15:24:42and I'm going to copy this and it'll
15:24:44perform it on each of those rows. Let me
15:24:47close that. And what did I do wrong? Oh,
15:24:50I need the squiggly bracket.
15:24:53And it'll show us each of those. So, the
15:24:55mean, max, count, and sum for all of the
15:24:58chocolate, and vanilla, as well as the
15:25:00groupings of liked, yes, and no. Now,
15:25:03after we've looked at all that, and
15:25:04that's how I usually do it, there is one
15:25:07uh shortcut function that can give you
15:25:09some of these things just really
15:25:10quickly. And so, let's go back up here
15:25:13and take this. It's just called
15:25:16describe. Um, and if you've ever done
15:25:17it, it's just going to give you some
15:25:19highlevel overview of some of those
15:25:21different aggregations. So, let's run
15:25:23this. And it's going to give us our
15:25:25chocolate and vanilla. And within each
15:25:27column, it's going to give us our count,
15:25:29our mean, our standard deviation, I
15:25:31believe is what that is. Our minimum,
15:25:3325%, 50, 75, and 100, which is our max.
15:25:37Then our count, and our mean. So, a lot
15:25:39of those aggregate functions. But the
15:25:41describe is, you know, a very
15:25:43generalized um function. We can't get as
15:25:46specific as we were with the previous
15:25:48ones that we were looking at, but I just
15:25:50wanted to throw this out there in case
15:25:51this is something that you'd be
15:25:52interested in because it, you know,
15:25:54technically is showing a lot of those
15:25:56aggregate functions just, you know, all
15:25:58at one time. So, that is our group by
15:26:00and aggregate functions within pandas. I
15:26:02hope that that was helpful. I hope that
15:26:03you understood, you know, everything
15:26:04that we were working on. If you like
15:26:06this video, be sure to like and
15:26:08subscribe and check out all my other
15:26:09videos on Python as well as pandas. And
15:26:11I will see you in the next video.
15:26:15>> [music]
15:26:25>> Hello everybody. Today we're going to be
15:26:27talking about merging, joining, and
15:26:28concatenating data frames in pandas.
15:26:30This whole video is basically around
15:26:32being able to combine two separate data
15:26:34frames together into one dataf frame.
15:26:36These are really important to understand
15:26:38when we're actually using the merge and
15:26:40the join. Right here we have what's
15:26:42called an inner join. And the
15:26:44[clears throat] shaded part is what's
15:26:45going to be returned. It's only the
15:26:46things that are in both the left and the
15:26:49right dataf frames. Then we have an
15:26:51outer join or a full outer join. And
15:26:54this will take all the data from the
15:26:56left data frame and the right data frame
15:26:58and everything that is similar. So
15:26:59basically it just takes everything. We
15:27:01also have a left join which is going to
15:27:03take everything from the left and then
15:27:05if there's anything that's similar,
15:27:07it'll also include that. And then the
15:27:09exact opposite of that is the right join
15:27:11which is going to give us everything
15:27:12from the right dataf frame and it's
15:27:14going to give us everything that is
15:27:15similar but it's not going to give us
15:27:17anything that is just unique to the left
15:27:19dataf frame. So this is just for
15:27:21reference because in a little bit when
15:27:22we start merging these these become very
15:27:24important. So I just wanted to kind of
15:27:26show you how that works visually. So
15:27:28let's get started by pulling in our
15:27:29files. So first we're going to say
15:27:31import and as pd. We'll run this and
15:27:36then we'll say dataf frame one and we'll
15:27:38also have a dataf frame two and these
15:27:39are the different data frames the left
15:27:41and [clears throat] the right dataf
15:27:42frame that we'll be using to join merge
15:27:45and concatenate. So we'll say dataf
15:27:47frame one is equal to [clears throat]
15:27:48pd.csv
15:27:50read and we'll do r and here is our file
15:27:55path. So we have this lo csv that's our
15:27:58Lord of the Rings CSV and let's call
15:28:00that really quickly so we can see what's
15:28:02in there. And I'm having a dyslexic
15:28:05moment uh because it's supposed to be
15:28:06read_csv.
15:28:08Uh I apologize for that. But this is our
15:28:11dataf frame. This is our dataf frame
15:28:12one. We have three columns. It's their
15:28:14fellowship ID 101 2 3 and four. Their
15:28:18first name Froto Sam Wise Gandalf and
15:28:20Pippen and their skills hiding gardening
15:28:22spells and fireworks. So this is our
15:28:24very first dataf frame that we're going
15:28:26to be working with. Let's go down a
15:28:27little bit. Let's pull this down here.
15:28:31And we're just going to say dataf frame
15:28:32two. Dataf frame two. And this is the
15:28:35Lord of the Rings 2. So let's pull this
15:28:38one in. Now, as you can see, it's very
15:28:40similar. We have fellowship ID 1 2 6 78.
15:28:44So we have three different IDs here. We
15:28:47don't have 67 and 8 in this upper this
15:28:50first data frame. We also have the first
15:28:52name. So Froto and Sam or Sam Wise are
15:28:55in the very first and the second data
15:28:57frame. But now we have three new people.
15:28:59Baramir, Eland, and Legalis. And now we
15:29:02have this age column, which again is
15:29:04unique to just this second dataf frame.
15:29:06Really quickly, I want to give a huge
15:29:07shout out to the sponsor of this video,
15:29:08and that is Zenesk. I've been using
15:29:10Zenesk for my company's customer
15:29:12analytics, and has been absolutely
15:29:13phenomenal. They're going to be hosting
15:29:14a conference called Zenesk Relate on May
15:29:1610th, and they're going to talk all
15:29:18about customer analytics, chat bots, and
15:29:20AI in this space. You can attend in
15:29:22person in San Francisco, or you can
15:29:23attend virtually, but space is limited,
15:29:26so be sure to apply if you want to
15:29:27attend. So, if you are a business leader
15:29:29and you want to make the most out of
15:29:30your customer data or you want to learn
15:29:32customer data analytics, I will leave
15:29:34links in the description. Again, huge
15:29:36shout out to Zenesk for sponsoring this
15:29:37video. Now, the first one that I want to
15:29:39look at is merge. And I want to look at
15:29:41merge first because I think this one is
15:29:42the most important. I use this one more
15:29:44than any of the ones that we're going to
15:29:46talk about today. The merge is just like
15:29:49the joins that we were just looking at,
15:29:51the outer, the inner, the left, and the
15:29:53right. And there's also one called
15:29:54cross, and I'll show you that one.
15:29:56Although if I'm being honest, I don't
15:29:58really use that one that much, but it's
15:30:00worth showing just in case you come into
15:30:01a scenario where you do want to do that.
15:30:04So, let's go right down here. And I want
15:30:05to be able to see these while we do it.
15:30:08So, we're going to say dataf frame one.
15:30:10And when we specify dataf frame one as
15:30:13the very first dataf frame, we say dataf
15:30:16frame. This is automatically going to be
15:30:19our left dataf frame. Then if we do our
15:30:23parenthesis right here and we say dataf
15:30:24frame 2, this is our right dataf frame.
15:30:27And let's see what happens when we do
15:30:29this. So what it's going to do and this
15:30:32we didn't specify this. It's just a
15:30:34default. It's going to do an inner join.
15:30:36So it's only going to give us an output
15:30:38where specific values or the keys are
15:30:41the same. Now you can't see this, but
15:30:42what is happening is is it's taking this
15:30:44fellowship ID and saying I have 1001
15:30:47here, a 10, 102 here. This is the exact
15:30:51same as up here with this fellowship ID
15:30:53and fellowship ID of 101 and two. But
15:30:56when we look at 1000 3 and 4, those
15:30:59aren't in this right data frame. And 678
15:31:02is not in this left data frame. So the
15:31:04only ones that match are this 101 and
15:31:07two. And that's why they get pulled in
15:31:09down here. But because we didn't
15:31:11explicitly say here's what I want to
15:31:14join or merge between these two data
15:31:16frames, it actually is looking at the
15:31:18fellowship ID and the first name. So
15:31:21it's taking in these unique values of
15:31:22Froto and Samwise, which are the same in
15:31:25both, which is why it pulled it over.
15:31:27But really quickly, let's just check and
15:31:29make sure that we did it on the inner
15:31:32join because again, we didn't specify
15:31:35anything. That was just the default. So,
15:31:37we're going to say how is equal to and
15:31:39then we'll say inner. And if we run
15:31:42this, it's going to be the exact same
15:31:43because again the inner is the default.
15:31:46But now, just to show you how it's kind
15:31:48of joining these two uh data frames
15:31:50together, I'm going to say on is equal
15:31:53to and then I'm only going to put
15:31:56fellowship ID. So, let's run this. Now,
15:31:59the first thing that you may have
15:32:00noticed is this first name underscorex
15:32:02and this first name underscorey.
15:32:05What the merge does as kind of a default
15:32:07is when you are only joining on a
15:32:09fellowship ID, we have this right dataf
15:32:11frame with fellowship ID, the left dataf
15:32:13frame with the fellowship ID. If you're
15:32:15just joining on these and you're not
15:32:17joining on the first name and the first
15:32:19name, then it's going to separate those
15:32:21into an underscorex and an underscorey.
15:32:24And even though they have the exact same
15:32:26values, since we are not merging on that
15:32:28column, it automatically separates that
15:32:31into two separate columns. So we can see
15:32:33the values within each of those columns.
15:32:35If we went into this on and we make a
15:32:37list and let's do it like that
15:32:41and we say comma and then we write first
15:32:44name oops first name and then we run
15:32:48this. It's going to look exactly like it
15:32:51did before. Again, it automatically
15:32:53pulled in both of these columns when it
15:32:55was merging it the first time even
15:32:57though we didn't write anything. But if
15:32:59we actually write this, it's doing
15:33:00exactly what it was doing when we just
15:33:01had DF2. We're just now writing it out.
15:33:05Now, there are other arguments that we
15:33:06can pass into this merge function. Let's
15:33:08hit shift tab and let's scroll down
15:33:11here. So, within this merge function, we
15:33:13have a lot of different arguments that
15:33:14you can pass into it. First, we have
15:33:16this right, which is the right data
15:33:18frame, which is this data frame 2. Then,
15:33:20we have the how and the on, which we've
15:33:22already shown how to do. There's a left
15:33:25on, right on, left index, right index.
15:33:28not something you'll probably use that
15:33:30much, but you definitely can if you want
15:33:32to look into that. And there's all these
15:33:33doc strings which show you exactly how
15:33:35to use all of these. So, if you're
15:33:37interested in looking at the left and
15:33:38the right and the left index, it's all
15:33:40in here. But one that is really good is
15:33:42the sort and you can sort it saying
15:33:45either it's false or true. Then we have
15:33:47these suffixes. Now, if you remember
15:33:49when we took these out, what it
15:33:51automatically did was it put in these
15:33:54underscorex and underscorey. You can
15:33:56customize that and you can put in
15:33:59whatever you'd like. Instead of the
15:34:00underscorex_y,
15:34:02you can put in some custom um string for
15:34:05that. We also have an indicator and a
15:34:07validate. Again, all the things that you
15:34:09can go in here and look at. I'm just
15:34:11going to show you the stuff that I use
15:34:12the most. So, these things right here
15:34:14are things that I definitely use the
15:34:16most. So, now that we've looked at the
15:34:17inner join, let's copy this right down
15:34:20here. And let's look at the outer join.
15:34:23And these get a little bit more tricky.
15:34:25I think the inner joint is probably the
15:34:26easiest one to understand.
15:34:29Let's look at the outer is spelled o u t
15:34:32e r. I don't know why I always want to
15:34:34say o u t e r, but let's run this and
15:34:37see what we get. So now this looks quite
15:34:40different. The inner join only gave us
15:34:43the values that are the exact same. This
15:34:46one is going to give us all of the
15:34:48values regardless of if they are the
15:34:50same. So we have 1 2 3 4 6 7 and 8. So
15:34:55let's scroll back up here. So we have 1
15:34:582 3 4 1 2 and 6 7 and 8. So we don't
15:35:01have a 1005. And then if you notice in
15:35:04this data frame right here, if the value
15:35:07doesn't have So if we can't join on the
15:35:10fellowship ID or the first name like
15:35:12legal wasn't one that we joined on or
15:35:14that has a similar value in the left
15:35:16dataf frame, it just gives us an nan
15:35:19which is not a number. And it's going to
15:35:21do that for any value where it couldn't
15:35:23find that join or it couldn't match uh
15:35:25something within that either ID or first
15:35:27name. So in age, we also have that for
15:35:30the ones that weren't in the right data
15:35:32frame. We only had 101 and 102. So we'll
15:35:36have the age for both Froto and Sam, but
15:35:38for Gandalf and Pippen, we don't have
15:35:41their corresponding IDs. And so it's
15:35:43just going to be blank for Gandalf and
15:35:45Pippen. And you can see that right here.
15:35:48So again, outer joins are kind of the
15:35:50opposite of inner joins. They're going
15:35:52to return everything from both. If there
15:35:55is overlapping data, it won't be
15:35:56duplicated. Now, let's go on to the left
15:35:59join. And I'm going to pull this down
15:36:01right here. And now we're just going to
15:36:03say how is equal to left. And let's run
15:36:06this. So what this is going to do is
15:36:10it's going to take everything from the
15:36:12left table or the left data frame right
15:36:14here. So everything from dataf frame
15:36:16one. Then if there is any overlap, it'll
15:36:19also pull the over overlapped or the,
15:36:21you know, whatever we're able to merge
15:36:22on from dataf frame 2. So let's go back
15:36:25up to our dataf frame one and two. So
15:36:27it's going to pull everything from this
15:36:28left dataf frame because we're
15:36:30specifying we're doing a left join. So
15:36:33everything from the left dataf frame
15:36:34will be in there. We're also going to
15:36:36try to bring in everything from the
15:36:38right, but only if it matches or or is
15:36:41able to merge. So just this information
15:36:43right here will come over. We weren't
15:36:46able to join on 10006, 10007 or 1008. So
15:36:50really, none of that information is
15:36:52going to come over. So let's go down and
15:36:53check on this. So again, we have 1 2 3 4
15:36:57all of the data with this first name and
15:37:00skills. Everything is in here. But then
15:37:03we are trying to bring over the age, but
15:37:05we only have matches with 10001 and 102.
15:37:08So only these two values will come in.
15:37:10Let's look at the right join because
15:37:12it's basically the exact opposite.
15:37:15Let's look at the right.
15:37:17And this is basically the exact opposite
15:37:19of the left in the fact that now we're
15:37:21only looking at the right hand. And then
15:37:24if there's something that matches in
15:37:25dataf frame one, then we will pull that
15:37:28in. So this is basically just looking
15:37:30like dataf frame 2 except we're pulling
15:37:32in that skills column. And since only
15:37:35101 and 102 are the same, that's why the
15:37:39skills values are here. Now, those are
15:37:41the main types of merges that I will use
15:37:43when I'm using a dataf frame or when I'm
15:37:46trying to merge a dataf frame. But there
15:37:48also is one called a cross or a cross
15:37:50join. Uh, and let's look at this one.
15:37:52And this one is quite a bit different.
15:37:55Here we go. Let's run this. So, this one
15:37:58is different in that it takes each value
15:38:01from the left dataf frame and compares
15:38:03it to each value in the right data
15:38:05frame. So for Frodo in this left data
15:38:08frame, it looks at the Froto in the
15:38:10right dataf frame, Sam Wise in the right
15:38:12dataf frame, Legololis, Elron, and
15:38:14Baramir all in the right dataf frame.
15:38:16Then it goes to the next value, Sam
15:38:18Wise, and does the exact same thing.
15:38:20Roto, Samwise, Legololis, Elron,
15:38:22Baramir. And it does that for every
15:38:24single value. So let's go right back up
15:38:27here. So it's taking this this 1001.
15:38:31It's comparing it to one two three four
15:38:33five. Then it's taking Sam Wise and it's
15:38:36comparing it to one two three four five
15:38:38Gandalf one two three four five Pippen
15:38:40and then you kind of see that pattern
15:38:41and that's what a cross join is. Um
15:38:43there are very few in my opinion reasons
15:38:46for a cross join although you'll if you
15:38:48ever do like an interview where you're
15:38:50being interviewed on Python you will
15:38:52sometimes be asked on cross joins but
15:38:54there aren't a lot of instances in
15:38:57actual work where you really use or need
15:38:59a cross join. Now, let's take a look at
15:39:02joins. And joins are pretty similar to
15:39:05the merge function. And it can do a lot
15:39:08of the same thing, except in my opinion,
15:39:10the join function isn't as easily
15:39:12understood as the merge function. It's a
15:39:14little bit more complicated. Um, but
15:39:17let's take a look and see how we can
15:39:19join together these data frames using
15:39:21the join function. So, let's go right up
15:39:22here. We're going to say dataf frame
15:39:24one.
15:39:26And then we'll do dataf frame two. very
15:39:29similar to how we did it before. And
15:39:31let's try running this. And it's not
15:39:33going to work. Um, when we did the merge
15:39:35function, it had a lot of defaults for
15:39:37us. Let's go down and see what this
15:39:39error is. It says the columns overlap,
15:39:41but no suffix was specified. So, it's
15:39:44telling us that it's trying to use the
15:39:45fellowship ID and the first name just
15:39:48like the join did, except it's not able
15:39:50to distinguish which is which. And so,
15:39:53we need to go in there and kind of help
15:39:54it out a little bit. Again, a little bit
15:39:57more hands-on than the merge, but let's
15:40:00see what we can do to make this work.
15:40:02Let's do comma and we'll say on and
15:40:05let's really quickly let's open this up
15:40:06and kind of see what we have. So, this
15:40:09one has less options than the merge
15:40:11does. We have other and that's our other
15:40:13data frame. We can do on and we're going
15:40:15to specify, you know, what column do we
15:40:17want to join on and then we can look at
15:40:19how do we want it to be a left, an
15:40:21inner, an outer, the same kind of types
15:40:23of joins as the merge. Then we have that
15:40:25left suffix, right suffix. And that's
15:40:28right here is kind of part of the issue
15:40:30that we were just facing is that those
15:40:32columns are the same. But if we say left
15:40:34suffix, it'll give us an underscore
15:40:37whatever we want to specify. Any string
15:40:39for columns that are both in the left
15:40:41and the right, we can give it a unique
15:40:43name. So we'll no longer have that
15:40:45issue. And then we can also sort it like
15:40:47we did on the other one. But anyways,
15:40:48let's go back to our on. We'll say on is
15:40:50equal to and then we'll say fellowship
15:40:55ID. Let's try running this. And we're
15:40:58still getting an error. It's just not as
15:41:00simple as the merge. So let's keep
15:41:02going. So now let's specify the type. So
15:41:04we'll say how is equal to and we'll do
15:41:06an outer.
15:41:08And if we run this, it still doesn't
15:41:10work. We're still getting the exact same
15:41:11issue as the left suffix and the right
15:41:13suffix. So now let's finally resolve it.
15:41:16I just wanted to show you how a little
15:41:18bit more frustrating it was. But now
15:41:19let's say uh L suffix is equal to and
15:41:24now it automatically when we did the
15:41:26merge did an underscorex but we can do
15:41:28let's do underscore
15:41:30uh left and then we can do a comma we'll
15:41:34do right suffix
15:41:36and we'll say is equal to and we'll do
15:41:39underscore right. Now when we run this
15:41:42it should work properly. Let's run this.
15:41:45So this is our output and obviously it
15:41:47looks quite a bit different over here.
15:41:49We have this fellowship ID. Then we also
15:41:52have fellowship ID left, first name
15:41:54left, fellowship ID right, and first
15:41:57name right. So it just doesn't look
15:41:59right. Now something I didn't specify
15:42:01when I first started this cuz I kind of
15:42:02wanted to show you is that the join
15:42:05usually is better for when you're
15:42:06working with indexes. Before when we
15:42:09were using the merge, we were using the
15:42:12column names and that worked really well
15:42:13and is pretty easy to do. But as you can
15:42:16see right here, when we're trying to use
15:42:17these column names, it's not working
15:42:19exceptionally well. Let's go ahead and
15:42:21create our index and then I can show you
15:42:23how this actually works and how it works
15:42:25a little bit better when we're working
15:42:26with just the index. Although you can
15:42:28get it to work just the same as the
15:42:30merge. It's just a lot more work. So
15:42:32let's go right down here and let's go
15:42:35and say DF4. So we'll create a new data
15:42:37frame. We'll say df1
15:42:40set index and we'll do an open
15:42:44parenthesis and we'll say we want to do
15:42:46this index on the fellowship
15:42:50ID and then we're going to do the join.
15:42:52So now we're going to say join. So we're
15:42:54setting an index. So we're setting that
15:42:56index on the fellowship ID. Now we're
15:42:58going to join it on df2
15:43:02set_index.
15:43:04And then we're also going to do that on
15:43:06the fellowship ID. And I'll just copy
15:43:08this.
15:43:13Oh jeez, I hate it when I do that. Okay,
15:43:16now we also want to do and specify the
15:43:19left and the right index. So I'll just
15:43:20copy this because we do need to specify
15:43:23this. Now let's try running the data
15:43:26frame for. So really quickly, just to
15:43:29recap, we were setting the indexes. We
15:43:31were doing the same thing above, right?
15:43:33Right? We have this join. We were
15:43:34joining dataf frame one with dataf frame
15:43:362. Now we're joining dataf frame one
15:43:39with dataf frame 2 except in both
15:43:41instances we're setting the index as
15:43:43fellowship ID. So we're joining now on
15:43:46that index. So now let's run this. And
15:43:48this should look a lot more similar to
15:43:50the merge than the join that we did
15:43:52above except now the fellowship ID right
15:43:55here is actually an index. So it's just
15:43:57a little bit different. But we can still
15:44:00go in here and do how is equal to outer.
15:44:04Oops, let's say outer. So we can still
15:44:07specify our different types of joins or
15:44:09the different way that we can merge or
15:44:11join these data frames together. We can
15:44:13still specify that. Again, it's just a
15:44:15little bit different. And that's why for
15:44:17most instances, I'm using that merge
15:44:19function because it's just a little bit
15:44:20more seamless, a little bit more
15:44:22intuitive. The join function can still
15:44:24get the job done, but as you can see, it
15:44:26takes a little bit more work. Now let's
15:44:28look at concatenate. Concatenating dataf
15:44:30frames can be really useful. And the
15:44:32distinction between a merge and join
15:44:34versus the concatenate is that the
15:44:36concatenate is kind of like putting one
15:44:38data frame on top of the other rather
15:44:40than putting one dataf frame next to one
15:44:42another which is like the merge and the
15:44:44join. So concatenating them is just a
15:44:46little bit different in how it'll
15:44:47operate. But let's actually write this
15:44:49out and see how this looks. Let's go up
15:44:51here and we'll say pd.conat.
15:44:55We'll do an open parenthesis and then
15:44:57we're going to concatenate dataf frame
15:44:59one, dataf frame 2. That's all we have
15:45:02to write. And let's run this. And so
15:45:05just like I said, it literally took the
15:45:07first dataf frame 1 2 3 4 and put it on
15:45:10top of the right data frame 1 2 6 7 8.
15:45:14So that is our left data frame. This is
15:45:16our right data frame. And they're
15:45:17literally just sitting one on top of the
15:45:19other. But just like when we merge
15:45:21either with a left or a right, when you
15:45:23have these skills and there aren't any
15:45:25values that populate for them, it is
15:45:27going to say not a number. And since
15:45:29we're not actually joining, we're not
15:45:30joining on one and two. Even though this
15:45:33one and this one is the same rows, it's
15:45:35not populating that value because again,
15:45:37we're not joining these together. We're
15:45:39just concatenating and putting one on
15:45:40top of the other. Now if we go into this
15:45:43concat we say shift tab there are a lot
15:45:47of different things that we can do which
15:45:48if you remember the zero axis is the
15:45:51left-hand index and the axis of one is
15:45:54the top index which is the columns so
15:45:56you can specify that and we can also do
15:45:59joins and this is the one that I'm going
15:46:01to take a look at but there are other
15:46:02ones that you can um look into as well
15:46:05but let's look at join let's do comma
15:46:08and we'll say join is equal to and let's
15:46:10do an inner join. So let's see what
15:46:13happens with this. As you can see, it is
15:46:15only taking the columns that are the
15:46:17same. That's what this inner is doing.
15:46:19It's joining these columns together. And
15:46:21the ones that were different, they
15:46:23didn't take because again, we weren't
15:46:25able to combine them. They aren't
15:46:27similar between both data frames. Let's
15:46:29do an outer. And now it's going to take
15:46:32all of them. And like I said, that's
15:46:34doing this on these columns right here.
15:46:35But we can also do it on this axis as
15:46:38well. So let's go ahead and say axis is
15:46:41equal to 1. And when we run this now
15:46:44it's joining us on this index right here
15:46:46of 0 1 2 3 4. So now these ones are
15:46:49being joined together and it's putting
15:46:51it side by side much like a merge would.
15:46:54So that's how concatenate works. And I'm
15:46:56going to show you one more thing. And
15:46:58again it's not up here in this you know
15:47:00title because it's not one that I
15:47:01recommend but it's one called append.
15:47:04The append function is used to append
15:47:06rows from one dataf frame to the end of
15:47:08another dataf frame. And then we can
15:47:09return that new dataf frame. And so
15:47:11let's do dataf frame one.append.
15:47:14We'll do an open parenthesis. And we'll
15:47:16say dataf frame 2. Very similar to how
15:47:18we've been doing other things. And let's
15:47:20run this. And as you can see, this is
15:47:22almost exactly like how the concatenate
15:47:24did when we first did it. But if we read
15:47:26kind of this warning, it's saying the
15:47:28frame.append not append method is
15:47:30deprecated and will be removed from
15:47:32pandas in the future version. Use
15:47:34pandas.conat instead. So it's literally
15:47:36warning us, you know, append is on its
15:47:38way out. If you want to do exactly what
15:47:40you're doing right here, go and try
15:47:42concat or concatenate because that'll do
15:47:44the exact same thing. So I'm not really
15:47:46going to show you any other variations
15:47:48of append because there's no reason it's
15:47:50going to be on its way out in the next
15:47:52version. So that is our video on merge,
15:47:54join, and concatenate and append as well
15:47:57uh in pandas. And I hope that that was
15:47:59helpful. I hope that you learned
15:48:00something. I mean, this stuff is really
15:48:02important because oftentimes you're not
15:48:03just working with one CSV or one JSON or
15:48:06one text file. You're working with
15:48:07multiple of them and you need to combine
15:48:09them all into one data frame. And so
15:48:11this is a really, really important
15:48:13concept and thing to understand. With
15:48:15that being said, be sure to like and
15:48:17subscribe, check out all my other videos
15:48:18on Python and pandas, and I will see you
15:48:20in the next video.
15:48:24>> [music]
15:48:34>> Hello everybody. Today we're going to be
15:48:35building visualizations in pandas. In
15:48:38this video we'll look at how we can
15:48:39build visualizations like line plots,
15:48:41scatter plots, bar charts, histograms,
15:48:44and more. I'll also show you some of the
15:48:46ways that you can customize these
15:48:47visualizations to make them just a
15:48:48little bit better. With that being said,
15:48:50let's go right over here, start
15:48:51importing our libraries. And we'll start
15:48:53with importing pandas spd. And this one
15:48:57is really all you need to actually
15:48:58create the visualizations in pandas. But
15:49:00we may get a little bit crazy. Uh, and
15:49:03so we're going to do a few different
15:49:04ones as well, like import numpy
15:49:08as np. And then we're going to do import
15:49:11mattplot lib.pipplot
15:49:16as plt. Now I may or may not use this. I
15:49:19just, you know, when I get into
15:49:20visualizations, I may want to change
15:49:21some different things. So, we're going
15:49:23to at least have them here in case we do
15:49:25want to use them. Let's go ahead and run
15:49:27this.
15:49:29So, now let's get our data set that
15:49:30we're going to be using. So, let's say
15:49:32dataf frames equal to pdread
15:49:36csv.
15:49:38And let's get this in right here. Now,
15:49:40we're going to be doing these ice cream
15:49:41ratings. Let's take a look at this
15:49:43really quickly. Now, these values are
15:49:46completely randomly generated. They're
15:49:48not real in any way. Um, but that's what
15:49:51we're going to be using cuz I just
15:49:52wanted something kind of generic,
15:49:54something that wouldn't be too crazy
15:49:55confusing, just something that we could
15:49:57use and you guys can understand that
15:49:58they're just numerical values. But let's
15:50:00also set that index really quick. So,
15:50:03we'll say dataf frame set_index
15:50:06and then we'll say date and then we'll
15:50:08say that's equal to the dataf frame. And
15:50:10we have this date column right here as
15:50:13our index. So, we have uh January 1st,
15:50:152nd, 3rd, 4th, and then we have our
15:50:17ratings right here. And again, these are
15:50:20all just integers and they're pretty
15:50:21easy or or really easy to demonstrate
15:50:23how you can visualize these. So, that's
15:50:25why we're using it today. So, the way
15:50:27that we visualize something in pandas is
15:50:29we use something called plot. So, let's
15:50:31just take our data frame. We'll do dataf
15:50:33frame.plot
15:50:35and we'll do our parenthesis. Now, let's
15:50:37go in here really quickly. Let's hit
15:50:39shift tab. And this is going to come up.
15:50:41And this is pretty important because
15:50:44this kind of is going to tell us what we
15:50:46can do within this plot. And
15:50:48unfortunately, there isn't like a quick
15:50:50overview. We just have this doc string,
15:50:52but we have our parameters right here.
15:50:54These are what we can pass in to kind of
15:50:56customize our visualization. So the data
15:50:59is going to be our data frame. Then we
15:51:01have our X and Y labels. We can specify
15:51:04the kind, and this one's important
15:51:05because we can specify what kind of
15:51:08visualization do we want. We can do a
15:51:10line plot, horizontal, a vertical bar
15:51:13plot, histogram, box plot, and then a
15:51:16few others including area, pi, density,
15:51:18all these other things. We can also
15:51:20specify if we want it to be a subplot.
15:51:22And a lot of these things that I'm
15:51:24specifying, you know, I'm going to show
15:51:25you how to do. You can use uh different
15:51:28indexes, you can add titles, add grids,
15:51:30legends, styles, all these different
15:51:33things. I mean, you can go through here
15:51:34because there are a lot, but you can
15:51:36specify and and, you know, customize all
15:51:38of these things. We won't be going into
15:51:41all of them, but I will show you some of
15:51:42the ones that I probably use the most
15:51:44and that I think are the most useful to
15:51:46know right away. So, let's get out of
15:51:47here. And we're just going to do
15:51:48df.plot.
15:51:50And when we run this, we'll get this
15:51:52right here. And that was super super
15:51:54easy. Created a line plot by literally
15:51:56doing just about nothing. Um, but by
15:51:59default, it's going to give us a line
15:52:01plot. So if we come up here,
15:52:04we say kind and let me get that out of
15:52:06the way is equal to line and we run
15:52:10this. So by default without us actually
15:52:12having to input anything, it's giving us
15:52:14that line plot as a default. So uh we
15:52:17can specify it's a line plot. As you can
15:52:19see, we already have all of our data
15:52:21right here. We didn't have to specify
15:52:22anything. It kind of automatically took
15:52:24it in. It is visualizing all three of
15:52:27these columns. And it has this little um
15:52:30legend right here. And we can specify
15:52:32where we want that. Uh there is an
15:52:34argument to be able to do that. It also
15:52:36gave us these tick marks of 2 4 6 8 10.
15:52:40Again, it read in and said it's only
15:52:42going from 0.0 to 1.0. That is kind of
15:52:46the peak. And so it kind of
15:52:48automatically gave us these ticks for
15:52:50us. Again, that's another thing that you
15:52:51can specify. We make it go up to 2, 5,
15:52:5410, a,000, whatever you want it to be.
15:52:56And then we're doing this based off of
15:52:58this date value right here. Really
15:53:00quickly, I wanted to give a huge shout
15:53:01out to the sponsor of this entire Panda
15:53:03series, and that is Udemy. Udemy has
15:53:05some of the best courses at the best
15:53:07prices, and it is no exception when it
15:53:08comes to pandas courses. If you want to
15:53:10master pandas, this is the course that I
15:53:12would recommend. It's going to teach you
15:53:13just about everything you need to know
15:53:15about pandas. So, huge shout out to
15:53:16Udemy for sponsoring this Panda series.
15:53:18And let's get back to the video. If we
15:53:20wanted to break these out by the actual
15:53:23column, we could go in here and say
15:53:25subplot is equal to true. And it's
15:53:29actually subplots. Whoops. And now we
15:53:32can run that. And then we can see each
15:53:34of those columns being broken out by
15:53:36themselves. Instead of them all being in
15:53:38one visualization, it's now uh three
15:53:41separate visualizations. Now, let's go
15:53:43right over here. We're going to get rid
15:53:44of the subplots. I want to show you just
15:53:45some of the different arguments that you
15:53:47can use to make this look nice. uh
15:53:49because I don't want to do this on every
15:53:51single visualization. I just want to
15:53:52show you what you can do. So, we have
15:53:54this one right here. We can add a title.
15:53:57Notice there's no title or anything
15:53:58really telling us what that is. So, we
15:54:00can say comma title and we'll say ice
15:54:04cream ratings. If we run this, we now
15:54:08have this nice title right here. Now, we
15:54:10can also customize the labels or the
15:54:12titles for the X and Y axis. It
15:54:14automatically took this date which is
15:54:16right here. This is our date index. It
15:54:19automatically took that for us, but we
15:54:21can customize that if we'd like to. All
15:54:24we have to do is comma and then we'll
15:54:25say x label is equal to. And so our x is
15:54:29this date one right here. And we can say
15:54:32daily rating. And then we can do the y
15:54:36label. We'll say y label is equal to and
15:54:39for this one we can say scores.
15:54:42Hope you cannot hear my dog in the
15:54:43background because they are being
15:54:44insane. Uh but let's go ahead and run
15:54:46this. And now we have these daily
15:54:48ratings on the x- axis and on the y-
15:54:50axis we have scores. Now let's go right
15:54:53down here and start taking a look at our
15:54:55next kind of visualization which is
15:54:57going to be a bar plot. So we'll do
15:54:59df.plot.
15:55:01We'll do kind is equal to and for this
15:55:04one we're going to say bar. Now this is
15:55:06what your typical bar plot will look
15:55:08like and a lot of the arguments that we
15:55:09just did on the line plot you can also
15:55:12apply to this bar plot. Something that's
15:55:14unique to the bar plot is that you can
15:55:16also make it a stacked bar plot. All we
15:55:18have to do is go in here. We'll say
15:55:20comma and we'll say stacked is equal to
15:55:23true. So now it's going to make it a
15:55:25stacked bar chart instead of just you
15:55:27know your regular bar chart. Let's go
15:55:29ahead and run this. And as you can see
15:55:31this is now stacked on top of one
15:55:32another with each of these columns all
15:55:35representing the values that they have.
15:55:37Now we don't always [snorts] have to do
15:55:38every single column. We can also specify
15:55:40the column that we want. So let's take
15:55:42the flavor rating for example. We could
15:55:45do
15:55:47flavor oops flavor rating. Good night
15:55:51flavor rating. And then it's only going
15:55:54to take in that flavor rating column.
15:55:56And if you notice, we don't have a
15:55:57legend. That's only when you have
15:55:59multiple values, which we are only
15:56:01looking at this one column. So all the
15:56:03values are right here. Now in this bar
15:56:04chart, it automatically defaults to a
15:56:07vertical bar chart, but you can change
15:56:09it to a horizontal bar chart. Let's go
15:56:11ahead and take a look at how to do that.
15:56:13Bring back all of them. We'll do df.plot
15:56:17dot and then we'll say barh. And I don't
15:56:21know if I can keep in that kind equals
15:56:22bar. Let me run this. Yeah, I need to
15:56:24get rid of that because the bar.h is its
15:56:26own um this is its own function. So now
15:56:30I'm going to run this. It should just
15:56:31have a stacked bar chart except now it
15:56:34should be horizontal. So now you can see
15:56:37this worked properly. It's basically the
15:56:39exact same thing as a vertical bar
15:56:41chart, just now horizontal, which may
15:56:43look better, especially depending on if
15:56:45you have values like this or, you know,
15:56:48something else that just looks better
15:56:49being horizontal. Now, the next one that
15:56:51we're going to take a look at is the
15:56:53scatter plot. So, we're going to say
15:56:55df.plot.catter.
15:56:58And if we run this, we're going to get
15:57:00an error. What we need in order to run
15:57:03this properly is we need to specify the
15:57:05x and the y axis in order for this
15:57:07scatter plot to work. So let's go here
15:57:11and we'll say x is equal to and we can
15:57:14take any of our columns that we have up
15:57:16here. So we'll say x is equal to texture
15:57:21rating and then oops y is equal to we'll
15:57:26do overall rating.
15:57:28Now, when we run this, it should work
15:57:30properly. Let's go ahead and take a
15:57:31look. Now, if we go in here and we do
15:57:34shift tab, we can also see some other
15:57:37things that we can specify. So, let's go
15:57:39right down here. So, we have our X and
15:57:41we have our Y, and those are the ones
15:57:42that we just did. We can also pass
15:57:44through an S, which is going to tell us
15:57:46or or change the size of the actual dots
15:57:50right here in our scatter plot. Then, we
15:57:52can also do a C, which is the color of
15:57:55each point. Let's start with the S.
15:57:58Let's say S is equal to and let's just
15:58:00do 100. We'll see what that looks like.
15:58:02So we have a much larger number. Let's
15:58:04do 500 and see what that looks like. So
15:58:07we can make these much larger on our
15:58:09visualization depending on what you're
15:58:11looking for. We can also look at the
15:58:12color. Let's put comma C. So for color
15:58:16we can say color is equal to and let's
15:58:19do uh yellow. Let's see if this works.
15:58:23So now we've changed it to yellow. That
15:58:24looks absolutely terrible, but it does
15:58:27work. Now, let's move on to the
15:58:29histogram. Histogram is always a good
15:58:31one. It's very similar to something like
15:58:33a bar chart, but what's great about a
15:58:35histogram is you can specify the bins.
15:58:37Um, so let's go ahead and say
15:58:39df.plot.hist.
15:58:43Then we'll do an open parenthesis. And
15:58:46let's go ahead and hit shift tab in
15:58:48here. Take a look at this one as well.
15:58:51So some of our parameters are the actual
15:58:53columns or the data frames that we want
15:58:55to pull in. We can choose the bins and
15:58:58they have a default of 10 in here. And
15:59:00so let's take a look at how this works.
15:59:02So we'll just run this as it is. So this
15:59:06is by default what this histogram is
15:59:08going to look like. Let's go ahead and
15:59:10specify our bins. We'll just say it was
15:59:1310 by default. Let's just do 20. See
15:59:16what that looks like. There are smaller
15:59:17columns right off the bat. And remember,
15:59:20histograms are really good for showing
15:59:22distribution of variables. You know,
15:59:24that's really what a histogram is for.
15:59:26But of course, since these are
15:59:27[clears throat] completely random
15:59:29numbers, this histogram isn't going to
15:59:30make any sense at all. But you can at
15:59:32least kind of see visually how it works.
15:59:34And if I didn't mention it before, which
15:59:36I should have, the bins represent how
15:59:38many kind of tick marks are down here.
15:59:40So, if we just do one, it's only going
15:59:43to be one very large uh, you know,
15:59:46histogram. We could even go further down
15:59:50from 10 and do five. So now there's only
15:59:52one, two, three, four, five. So the
15:59:55distribution gets smaller and things get
15:59:57more compact. As you spread it out
16:00:00again, like we did 100, [clears throat]
16:00:03it's going to spread it out a lot. Um,
16:00:05and this is what it shows. You know,
16:00:07it's showing the distribution of those
16:00:09bins across however many you want. So
16:00:12the 10 by default, you know, it usually
16:00:14is pretty good for a lot of different
16:00:15things. Now, let's go down here and look
16:00:17at the box plot. And the box plot is a
16:00:20pretty interesting one. Let's go ahead
16:00:22and visualize it really quickly, and
16:00:23then I'll kind of explain how this one
16:00:25works. So, let's do df.boxplot.
16:00:28Let's run this. And really, what we're
16:00:30looking at is some different markers
16:00:32within our data. This line right here is
16:00:34the minimum value within that column. We
16:00:37also have the bottom of the box, which
16:00:38is the 25th percentile of all the values
16:00:42within just this column. This is 50%.
16:00:45Then we have 75% and then up here we
16:00:48have our maximum value. So I can take a
16:00:50glance at this and see that we have a
16:00:51low minimum a high maximum and it
16:00:54definitely skews towards the lower
16:00:56range. Whereas if I look over here we
16:00:59have a lower minimum and a higher
16:01:01maximum and you can see that this m
16:01:03medium point is at 6 versus 04 over
16:01:05here. So the skews a lot higher. Now
16:01:07let's go down here and take a look at an
16:01:09area plot. We'll do df.plot plot area.
16:01:14And let's just run this. This is what
16:01:16we're going to get by default. Now,
16:01:18something I wanted to show you earlier,
16:01:20I just haven't gotten around to. I want
16:01:21to show you something called figure size
16:01:23or fig size. Um, so for this, it's know
16:01:26it's just looks small, looks a little
16:01:27bit cramped. So, let's say we want to
16:01:29increase the size of this. And we'll say
16:01:31fig size, oops, fig size is equal to,
16:01:34and let's just do a parenthesis and say
16:01:3710, 5. That should be pretty large. This
16:01:40is going to make it a lot larger. Just
16:01:42something I wanted to throw in there.
16:01:44But I look at these area charts as
16:01:45pretty similar to like a line chart. If
16:01:47we went and compared those be pretty
16:01:49similar. Um, but they're different
16:01:51visually. And you know, you absolutely
16:01:53can use these for different types of
16:01:55visualizations. But I don't use this one
16:01:57a lot if I'm being honest. That's why
16:01:58it's kind of towards the end of the
16:02:00video, but you definitely can do it.
16:02:02Let's go on to our very last one of the
16:02:04video. That's going to be the beautiful
16:02:06pie chart. Let's say df.plot.py.
16:02:09pi. We're going to open parenthesis and
16:02:12let's run it. We're going to get this
16:02:14error. That's because we need to specify
16:02:17what column we're working with here. So,
16:02:19let's just say the y and that's what we
16:02:21need. Let me open this up for us.
16:02:25Right here, we have our y and this is
16:02:27our our label or our column that we're
16:02:29going to plot. That's really all we
16:02:30need. So, we can just say y is equal to
16:02:34labor rating. Oops. Labor rating. Let's
16:02:38run this. And now we get this
16:02:40visualization right here. Let's make
16:02:42this one a little bit bigger. Big size
16:02:46is equal to 10,
16:02:50six. So now it's a little bit bigger. It
16:02:52definitely depends. So this legend is
16:02:54going to autopop populate. You know, you
16:02:56can make this as big as you want. And
16:02:59obviously it's going to look a little
16:03:00bit better if you do it larger. And
16:03:02these colors autopop populate. Now you
16:03:03can customize these colors. Although I
16:03:05found these ones to be just when you
16:03:07have a lot of them, it's harder to
16:03:08customize them as easily. But, you know,
16:03:11definitely look into it. These are
16:03:12things that everything in here is almost
16:03:14something that you can customize in some
16:03:16way. Although, it does get a little bit
16:03:18tricky. You definitely have to do some
16:03:19research and some Googling around just
16:03:21to kind of figure out how to do those
16:03:23things. Now, one last thing that I
16:03:25wanted to show and something, you know,
16:03:27I could have probably done at the
16:03:28beginning, um, is you can actually
16:03:30change what visual this is. And we can
16:03:33do that pretty easily. Within Mattplot
16:03:36Lib, there are different styles. Um, and
16:03:38so let's go right here. Let's add a new
16:03:41row or a new cell. And we'll say print.
16:03:44We'll do plt. So that's that mapplot lib
16:03:47right here. We'll do
16:03:48plt.style.available.
16:03:53And what this is going to do, whoops.
16:03:54What this is going to do is show us all
16:03:56these different types of stylings that
16:04:00you can do to kind of change up this
16:04:01visualization. And then once we find the
16:04:04one that we like, we'll just do
16:04:05plt.style
16:04:08use. And then in the parenthesis, we'll
16:04:11just specify which one we want. Now,
16:04:13there's all these seabor ones. And
16:04:15seabour is a really great um really
16:04:18great library. Let's try seabour deep. I
16:04:21haven't tried this one at all. Let's go
16:04:23ahead and try this. It just changes some
16:04:25of the colors, some of the visuals. We
16:04:27can try something like 538.
16:04:31Let's try this. That looks quite a bit
16:04:34different. And let's try something like
16:04:38um classic. I don't know what this one
16:04:40looks like. Let's just try it.
16:04:42So, you can try out all these different
16:04:44styles. Find one that you like. Find one
16:04:46that you think looks really nice. And
16:04:48you can run with it through all your
16:04:49visualizations. So this has been our
16:04:51video on visualizing data in pandas. I
16:04:53think it's a really good introduction on
16:04:55how you can visualize data within
16:04:56Python. And in future videos we'll look
16:04:58at map lib and seaborn which are some
16:05:01really great libraries for visualizing
16:05:03data which I use a lot. So I hope that
16:05:05you enjoyed this video. If you did be
16:05:07sure to check out all my other videos on
16:05:08Python and pandas and I will see you in
16:05:10the next video.
16:05:23Hello everybody. Today we're going to be
16:05:25cleaning data using pandas. Now there
16:05:27are literally hundreds of ways that you
16:05:29can clean data within pandas, but I'm
16:05:31going to show you some of the ones that
16:05:32I use a lot and ones that I think are
16:05:34really good to know when you are
16:05:35cleaning your data sets. So we're going
16:05:37to start by saying import pandas as pd
16:05:41and we're going to run that. And now
16:05:43we're going to import our file. So we're
16:05:45going to say dataf frame is equal to pd.
16:05:47So that's pandas read underscore and we
16:05:50actually have this in an excel file. So
16:05:52we'll say read oops say read excel do an
16:05:56open parenthesis and we'll do r and then
16:05:59we'll paste the path right here. And now
16:06:01we're just going to call that variable.
16:06:02So we'll call dataf frame and we'll
16:06:03actually read it in and look at the
16:06:05data. So let's scroll down here and
16:06:07let's take a look at this data frame or
16:06:09this excel file that we're reading in.
16:06:10So, right off the bat, we have this
16:06:12customer ID that goes from 1001 all the
16:06:14way down to 1,020.
16:06:17We have this first name, and everything
16:06:20looks pretty good here, except in this
16:06:22last name column, uh, looks like we have
16:06:25some errors. We have some forward
16:06:27slashes, some dots, some null values.
16:06:31Um, so definitely going to have to clean
16:06:32that up because we don't want that in
16:06:34the data. We have phone number, and it
16:06:37looks like we have a lot of different
16:06:39formats. um as well as NAS, not a
16:06:42number. Um just lots of different stuff.
16:06:46So, we're going to need to standardize
16:06:47that. So, clean it up and then
16:06:48standardize it to where it all looks the
16:06:50same. Um we also have address. And it
16:06:54looks like on some of these we just have
16:06:55a street address, but on some of the
16:06:57other ones we have like a street address
16:07:00and another location as well as a zip
16:07:02code in some of them. So, we'll probably
16:07:05want to split those out. We have a
16:07:07paying customer uh which is yes and nos
16:07:09and some of those are not the same. So I
16:07:12have to standardize that. We have a do
16:07:14not contact kind of the same thing as
16:07:16the paying customer. And we have this
16:07:18not useful column which we'll probably
16:07:20just want to get rid of. Okay. So the
16:07:22scenario is is that we got handed this
16:07:24list of names and we need to clean it up
16:07:26and hand it off to the people who are
16:07:28actually going to make these calls to
16:07:30this customer list. So, they want all
16:07:32the data in here standardized and
16:07:34cleaned so that the people who are
16:07:35making those calls can just make those
16:07:36calls as quickly as possible, but they
16:07:39also don't want columns and rows that
16:07:41aren't useful to them. So, things like
16:07:43this not useful column, we're probably
16:07:45going to get rid of. And then ones that
16:07:47say do not contact, if it says yes, we
16:07:50should not contact them, we probably
16:07:52will want to get rid of those somehow.
16:07:54So, that's a lot of what we're going to
16:07:55be doing to clean this data set.
16:07:57Normally the very first thing that I do
16:07:59when I'm working with a data set most of
16:08:01the time except very rare cases when
16:08:03you're actually supposed to have
16:08:04duplicates is I actually go and drop the
16:08:07duplicates from the data set completely.
16:08:09All you have to do for that is say
16:08:11df.drop
16:08:14duplicates. So they make it super easy
16:08:17for you. Let's just run it. And up here
16:08:20is our original data set. We have this
16:08:2319 and 20. And those are obviously
16:08:25duplicates. They have the exact same
16:08:26data. It's just a duplicate row that we
16:08:28need to get rid of. If we look right
16:08:31down here, we no longer have that 20. We
16:08:33now just have one row of Anakin
16:08:36Skywalker. And of course, we want to
16:08:38save that. So, we're just going to say
16:08:40DF is equal to and DF. So, now it's
16:08:44going to save that to the dataf frame
16:08:46variable again. And now when we run
16:08:48this, our dataf frame now does not have
16:08:50any duplicates. That's definitely one of
16:08:52the easier steps that we're going to
16:08:54look at. Uh things are going to get
16:08:55quite a bit more complicated as we go,
16:08:57but I'm starting out, you know, kind of
16:08:59simple so that we can kind of get a feel
16:09:01for it and then we'll start getting into
16:09:03the really tough stuff. So the next
16:09:05thing that I want to do is remove any
16:09:07columns that we don't need. I don't want
16:09:08to clean data that we're not going to
16:09:10use. So if we're just looking through
16:09:12here, you know, they may need, you know,
16:09:14first name, last name, phone number for
16:09:16sure. Address might give them some
16:09:18information of where they're calling to
16:09:20or time zone. So we want that. This not
16:09:23useful column looks like a pretty good
16:09:25candidate to delete and it's very easy
16:09:28to do that. We're going to go right down
16:09:29here and we're going to say df do drop.
16:09:34We'll do an open parenthesis. Drop just
16:09:36means we are dropping that column. And
16:09:38we can specify that by saying columns is
16:09:41equal to and then we'll paste in that
16:09:44column that we want to delete. So let's
16:09:46run this and see what it looks like. and
16:09:48it literally just drops that column
16:09:50exactly like we were talking about. It
16:09:52no longer has that column. Again, we
16:09:54want to save that. We can always do in
16:09:56place equals true. Um, if you follow
16:09:58this tutorial series, you can always do
16:09:59in place equals true and that'll save it
16:10:01as well. But just for our workflow, most
16:10:04of the time I'm going to assign it back
16:10:05to that variable. Um, just for keeping
16:10:08it the same. Really quickly, I wanted to
16:10:10give a huge shout out to the sponsor of
16:10:12this entire Panda series, and that is
16:10:14Udemy. Udemy has some of the best
16:10:15courses at the best prices and it is no
16:10:17exception when it comes to pandas
16:10:19courses. If you want to master pandas,
16:10:21this is the course that I would
16:10:22recommend. It's going to teach you just
16:10:23about everything you need to know about
16:10:25pandas. So, huge shout out to Udemy for
16:10:27sponsoring this pandas series. And let's
16:10:28get back to the video. Now, let's kind
16:10:30of go column by column and see what we
16:10:32need to fix. And we'll start on this
16:10:34left hand side. This customer ID to me
16:10:36looks perfectly fine. I'm not going to
16:10:38mess with it at all. The first name at a
16:10:41glance also looks perfectly fine. and I
16:10:44don't see anything wrong with it
16:10:45visually, which is a good thing. Um,
16:10:47although sometimes that can be deceiving
16:10:49and that can cause errors down the line,
16:10:50but we're not going to uh assume that
16:10:53there are errors in here. Now, let's
16:10:54look at this last name. Now, the last
16:10:56name, obviously, I'm I'm seeing some
16:10:58obvious things, things that we talked
16:10:59about when we were first looking at this
16:11:00data set. We have this forward slash,
16:11:04which we definitely need to get rid of.
16:11:06We have null values, so not a number.
16:11:09Right here, we have some periods as well
16:11:11as an underscore right here. So all
16:11:13those things I think we should clean up
16:11:15and get rid of it so that when the
16:11:17person is making these calls, you know,
16:11:19it's all cleaned up for them. So how are
16:11:21we going to do that? We can actually do
16:11:23this in several different ways, but
16:11:25let's just copy this last name. The
16:11:27first one I'm going to show you is
16:11:29strip. And we'll write it kind of like
16:11:30this. We'll say data frame and then
16:11:32we'll specify the column that we're
16:11:34working with because we don't want to
16:11:36make these changes or strip all of these
16:11:38values from everywhere. We only want to
16:11:40do it on just this column. If we do this
16:11:43and we don't specify the column name, it
16:11:45will apply it to everywhere. So, if
16:11:46we're trying to do these, yeah, let's
16:11:48say these underscores, maybe that would
16:11:51mess with something else in another
16:11:53column, and we don't want that. So, we
16:11:55just want to specify just this last
16:11:57name. So, let's go last name.
16:12:02Strip. Now, what strip does, and let's
16:12:04see if we can open this up really
16:12:06quickly. We can't. Um, but what strip
16:12:08does, I was just I was hitting shift tab
16:12:11in here to see if it could bring up um,
16:12:12you know, some of the notes on it. But
16:12:14what strip does is it takes either the
16:12:16left side or the right side. Well, lrip
16:12:19takes from the left side. Rrip takes
16:12:21from the right side and strip takes from
16:12:23both. But you can strip values off the
16:12:26left and the right hand side. And we can
16:12:28specify those values. Now, for what
16:12:30we're doing in this column, we can just
16:12:32use strip because, as you can see, this
16:12:34forward slash, these dots, as well as
16:12:37this um underscore are all on the far
16:12:40sides. If there was a value like
16:12:43swan_son,
16:12:45the strip wouldn't work at all because
16:12:46it's not on the outside of of the value
16:12:49or the word. So, we can use strip. I'll
16:12:51also show you how to use replace. And
16:12:54replace is another really good option
16:12:56for things like this. But let's start
16:12:58with strip and just see what it looks
16:13:00like and see if we can get what we need
16:13:01done. So let's just run this for now.
16:13:04See what happens. So it looks like
16:13:07nothing has changed because again we're
16:13:09not specifying any specific value. Just
16:13:11by default it's only taking out white
16:13:13space. So like spaces that shouldn't be
16:13:15there. That's what it does by default.
16:13:17Now we can specify within this exactly
16:13:21what values we want to take out. So,
16:13:23let's go ahead and do that. Let's say
16:13:26left strip and let's try to take out
16:13:28these dots real quick. So, we're just
16:13:29going to do a parenthesis dot dot dot.
16:13:32Now, let's run this and see what it
16:13:34looks like for this one. Potter, it is
16:13:38now gone. So, those three dots were
16:13:40there before. Let's just show it. So,
16:13:42they were there and then when I ran it
16:13:45like this, now they're gone. That's what
16:13:47the L strip does. It takes it only off
16:13:50the left hand side. Now, we can also do
16:13:52a forward slash. So, we'll do something
16:13:54like this, and it'll get rid of the
16:13:56white. But, as you can see, now we
16:13:59aren't taking out these three dots, so
16:14:00they're still there. Now, is it possible
16:14:03to do something like this, where we put
16:14:06these values inside of a list. Um, let's
16:14:08try it. So, we'll say just like this 1 2
16:14:113. Let's run it. And no, it doesn't. Um,
16:14:15this lrip actually sits within the the
16:14:17realm of regular expression. So if
16:14:20you've ever worked with regular
16:14:21expression, you know it gets very
16:14:23complicated, very complex. So you want
16:14:25to keep it kind of simple, especially
16:14:26with these values where we're just
16:14:28taking a few out. So what we're going to
16:14:30do is we're going to do dot dot dot and
16:14:33we're going to take it out one by one.
16:14:35Now in order to save this, because we
16:14:37want to save this, we want to take out
16:14:38that value. We don't just want to say
16:14:40dataf frame equals cuz that would be uh
16:14:43very bad. What this would say is now
16:14:45this data frame is only equal to these
16:14:47values that we're seeing right here. We
16:14:49want to only apply it to this column. So
16:14:53we're going to go like this. So now when
16:14:55we do it and then we call the entire
16:14:58data frame, it's only applying this to
16:15:01this one column, the last name column.
16:15:04So let's run it. And now when we go down
16:15:07to Potter right here, it's cleaned up.
16:15:10So we're going to do the same thing but
16:15:12for those other values.
16:15:14And we'll do it just like this. So,
16:15:15we'll do a forward slash and it's a left
16:15:19strip and then we'll do I'll do the left
16:15:21strip on this underscore to just to show
16:15:23you that it won't work and then we will
16:15:27go on from there. So, it's not pulling
16:15:29it because we're looking at the left
16:15:30hand side only. We need to use R strip.
16:15:33So, now let's use R strip.
16:15:37And now that looks perfect. Has no
16:15:39underscore. So, that's how you can use
16:15:41strip for either the left side, the
16:15:43right side, or just strip by itself,
16:15:45which covers both sides. Now, I showed
16:15:47you all of that because I am going to
16:15:48show you a different way to do it. Um,
16:15:50and I apologize because I somewhat lied
16:15:52to you earlier. Um, let's run
16:15:56this right here. Actually, we're just
16:15:57going to pull it in like this.
16:16:00We're going to remove the duplicates
16:16:01again. Bear with me. We're going to drop
16:16:04that column. And then now we're sitting
16:16:07with that data frame again with those
16:16:08exact same mistakes. I just wanted to
16:16:10reset it for a second. There is a way uh
16:16:12that you can do this and I just wanted
16:16:14to, you know, kind of show you how you
16:16:16can do it. You can do this right here.
16:16:21And we'll say so we're now again we're
16:16:23just looking at this column, just this
16:16:25column and we're using strip and let's
16:16:27get rid of R because we want to do apply
16:16:29it to everywhere. You can input all of
16:16:32those values individually and it will
16:16:35clean it up. So let's say we want to get
16:16:36rid of numbers. We'll do 1 2 3. Then we
16:16:39can do the dot. So that's going to be
16:16:41for a period or for our dot dot dot
16:16:43Potter. We could also do the underscore
16:16:46and we can do the forward slash. So we
16:16:48put it all in one string right here. Now
16:16:52let's take a look at this. We'll get rid
16:16:54of this really quickly. Now let's take a
16:16:56look. And all of them were removed. I
16:16:59showed you how to do it before because
16:17:00that's at least how my mind would think
16:17:02about it. I'd think, oh, I can put it in
16:17:03a list and run it through this L strip
16:17:05or this right strip and it would work.
16:17:07Um, but that's not how strip works. You
16:17:09have to kind of combine it all into one
16:17:10value. So, uh, yes, I deceived you. I
16:17:13apologize. But now when we call dataf
16:17:16frame and we assign it to that column,
16:17:19so the last name column, we're assigning
16:17:20what we just did to this last name
16:17:22column. Everything should look perfect.
16:17:26And it does. So, our customer ID, first
16:17:28name, last name are all cleaned up. Now,
16:17:30we're going to come to a much more
16:17:32difficult one. This is probably, if I'm
16:17:34being honest, the hardest one. I said we
16:17:36were going to work up, but this is
16:17:37probably the hardest one of the whole
16:17:39video. Working with phone numbers. And
16:17:41look at all these different types of of
16:17:45formats. I mean, it is um it's not going
16:17:48to be fun. And imagine, you know,
16:17:49there's 20,000 of these. You can't just
16:17:51go and manually clean those up. You need
16:17:54something to kind of automate that. So,
16:17:57that is what we're going to do. So,
16:17:59let's go right down here. We'll copy the
16:18:01data frame and I'm going to pull it
16:18:04right here. So now we need to clean up
16:18:06this phone number. What we want is it
16:18:09all to look exactly the same unless it's
16:18:12blank and we'll keep it blank. We don't
16:18:13want to populate that data. But we want
16:18:16all of them to look exactly like this
16:18:18one. And what we're going to do is right
16:18:21off the bat we're going to take all of
16:18:23the non-numeric values and just
16:18:25completely get rid of them. Strip it
16:18:27down to just the numbers. So this 1 2
16:18:293-643 or forward slash will just be the
16:18:33numbers. Same with these bars and these
16:18:36slashes and everything. All of these
16:18:39will just be numeric. Then we'll go back
16:18:41and reformat it how we want to format
16:18:44it, which will look exactly like this
16:18:46one. Um, but we just want to do it for
16:18:48the entire column. So let's go right up
16:18:50here and we're going to try replace for
16:18:52the first time. So let's do phone
16:18:55number. it just oops that's not what I
16:18:58wanted. So we're going to do a bracket
16:19:02say phone number dot string.replace
16:19:06just like we did before. Now we're going
16:19:08to use some regular expression in here
16:19:10and I'll kind of do a really high
16:19:12overview although I'm not going to dive
16:19:14super deep into the regular expression.
16:19:16Then we're going to do a parenthesis and
16:19:18within there we're going to do a
16:19:20bracket. Um I can't remember what this
16:19:22is called. Is it called a carrot? I
16:19:24think it's called a carrot. uh program.
16:19:25I'm just going to call it that. It may
16:19:27not be correct, but I think it's an
16:19:28upper arrow. So, it's an upper arrow. A
16:19:31dash oops a-z.
16:19:38Now, at a super high level, what that
16:19:40character or that first thing is doing.
16:19:41It's saying we're going to return any
16:19:42character except and then we specify
16:19:45anything A to Z. A to Z, upper or lower
16:19:48case. And then actually, I think this
16:19:50should be like this. A to Z. Uh and then
16:19:530 to 9. So any value like ABC 1 2 3
16:19:56those are not going to be matched. It's
16:19:58going to match all of them except these
16:20:00values. And then we're going to replace
16:20:02them by saying comma and we're going to
16:20:04replace them with nothing. So this is
16:20:06just an empty string. So literally we're
16:20:09taking everything that is not an a b c a
16:20:111 2 3. So a letter or a number. We're
16:20:14replacing all of that and then we're
16:20:16replacing it with nothing. So let's run
16:20:18this and see what it looks like. And it
16:20:20looks like that worked properly. Now, we
16:20:23do have this NA because we had an N- A
16:20:26for I don't know, maybe that was Creed
16:20:29Bratton. Um, but it worked for basically
16:20:31everything else. We're going to go
16:20:33through the entire process and then at
16:20:35the end we'll remove any values. We want
16:20:37them to just be completely null. We we
16:20:39don't want them to even see Nan and
16:20:41wonder what that is. We just want it to
16:20:43be blank. And we'll do that at the very
16:20:44end. So, now that we know that that
16:20:47worked, let's assign it. We'll do df
16:20:51phone number is equal to and then we'll
16:20:53say dataf frame. And this looks a lot
16:20:57more standardized than it did before
16:20:59already. But now what we want to do is
16:21:01try to format this. Um, and I've done
16:21:04this many many times. I always use a
16:21:06lambda. You can definitely use a for
16:21:09loop. I just I don't do it that way
16:21:10myself. So I'm going to show you how to
16:21:12do it using a lambda. Let's get rid of
16:21:14this. And we're gonna say DF phone
16:21:17number. We've already done that. I'm
16:21:19just going to get rid of it. Now we're
16:21:20gonna say DF phone number. Then we're
16:21:22going to say apply. We'll do an open
16:21:25parenthesis. And then this is where
16:21:26we're going to build out our lambda. So
16:21:28we'll say lambda x colon. Now this is
16:21:32where we're going to kind of format it.
16:21:34So what I want to do is I want to take
16:21:35the first three strings 1 2 3. Then I
16:21:38want to add a slash. And then the next
16:21:40three strings add a slash or a dash. uh
16:21:43and then that be the value that's
16:21:45returned. So it's not super difficult.
16:21:47We're just going to do x then a bracket.
16:21:50Let me get rid of that. An x and then a
16:21:52bracket. And then we want the 0 to
16:21:54three. So it goes 0 1 2. So 0 1 2. It
16:22:00doesn't include the three. It goes up to
16:22:02three. So 0 1 2. That's our third first
16:22:04three values. Then we'll do plus and do
16:22:08a quote and do a dash. So this is our
16:22:11first kind of sequence. And I'm just
16:22:13going to copy this. We'll do plus and
16:22:17instead of three or we are going to
16:22:19start at three because that now it's
16:22:20inclusive. So we're going to go from
16:22:21three and we're going to go all the way
16:22:23up to six. So it should be three, four,
16:22:26five, our next three values. Then we
16:22:28have a dash and we'll copy this and
16:22:32we'll say plus. And now we go from six
16:22:37all the way to 10. Now let's try running
16:22:40this. And as you can see, we get an
16:22:44error. Now, I already know what the
16:22:45error is. Float object is not
16:22:48subscriptable, which means we're trying
16:22:50to um basically look at it like a
16:22:52string. Right now, it's not a string.
16:22:54It's actually a number. So, let me get
16:22:57rid of this for just a second. I want to
16:22:59show you what it's talking about. So,
16:23:01right now, we have values that are
16:23:04floats and values that are strings or
16:23:07not even a number. So, we have values
16:23:08that are strings or not a number. So if
16:23:11we want to actually look through it like
16:23:13kind of like indexing if we want to do
16:23:14that they all have to be strings. So we
16:23:17need to change this entire column into
16:23:20strings before we can apply this um
16:23:23formatting. Now when I was creating this
16:23:25if I'm being honest my first thought
16:23:27when I was doing this was to do it like
16:23:28this string df phone number. Um let's
16:23:32just run that.
16:23:34This is what the values look like. Um,
16:23:36and I don't remember why or why it was
16:23:40doing this. I can't I can't remember.
16:23:42But I looked into it quite a bit and I
16:23:43was like, "Oh, I need to apply this
16:23:47string, converting it to a string on
16:23:49each value, not the entire row or not
16:23:52the entire column." So, how we can do
16:23:54that is actually fairly easy because
16:23:56we've already done a lot of the heavy
16:23:57lifting. We're just going to copy this
16:24:00and we're going to say x.
16:24:04So string of x. And again, lambda is
16:24:07like a little anonymous function. So you
16:24:10could do this by saying for um x in this
16:24:14uh column. We could do a for loop and
16:24:17then say for every x it equals the
16:24:18string of x and then it changes it to a
16:24:20string. But a lambda just does it a lot
16:24:22quicker. Um so we're going to say so
16:24:25let's do that really quickly. And all of
16:24:28our values look exactly the same, and
16:24:29that's how we want it. So, we're just
16:24:31going to copy this, apply it.
16:24:36Good. And now we're going to take this
16:24:41and we're going to run this again. Just
16:24:43ignore all my commented out stuff.
16:24:45Pretend I don't have that. Um, so now
16:24:47when we run this, it should work. There
16:24:50we go. Now if we look at these numbers 1
16:24:522 3-545-5421
16:24:59and it does that for every single one
16:25:01where there's values even where there's
16:25:02nan or na it's still adding those values
16:25:06but we expected that so let's apply it
16:25:11says equal to and then we'll look at the
16:25:14data frame and this looks almost exactly
16:25:18what we're hoping for we just need to
16:25:19get rid of these
16:25:20So, this nan- dash and this na dash, we
16:25:24need to get rid of those. And that is
16:25:25super easy to do. Um, we're just going
16:25:28to say, so now that we've done it, and
16:25:30we'll comment that out. We'll say df
16:25:35and let's copy this. Ignore the
16:25:38messiness. I do apologize for that. It's
16:25:39very messy. Um, but if you're following
16:25:41along with me, you get what we're doing.
16:25:43So, df phone number. So only on the
16:25:46phone number sayreplace
16:25:50no open parenthesis. Now we can specify
16:25:53this value. So we want to take this
16:25:56exact value
16:25:58and replace it with nothing. And let's
16:26:01just see if that does work. It does. Now
16:26:04we have these nas.
16:26:07And so let's actually I'll paste that
16:26:10right down here. We're going to do this
16:26:13is equal to and then we're just going to
16:26:15take this entire string put it right
16:26:17here and put this value as our what
16:26:21we're looking for and then replacing and
16:26:23then when we call that data frame it
16:26:26should work properly and it is perfectly
16:26:29cleaned. So we have every single value
16:26:33all the exact same. they don't have
16:26:35different characters or different um you
16:26:37know formatting and we got rid of all
16:26:39the ones that we don't have or don't
16:26:41need. Um all the ones that were just
16:26:43random values. So this column is now
16:26:46completely cleaned up. Again, definitely
16:26:48one of the more difficult ones. Um ones
16:26:51that I've done a thousand times. I've
16:26:52had to work with a lot of phone numbers
16:26:54and stuff like that. This one does get
16:26:56very tricky, especially if you have like
16:26:57a plus one, which is like an area code.
16:27:00Um that can get tricky as well. But this
16:27:02is on a kind of a high level. This is
16:27:04how you can do that. And it's pretty
16:27:06neat how you can actually, you know,
16:27:07clean up and standardize those phone
16:27:09numbers. So, let's go right down here.
16:27:11Uh, let's run it. The next thing that
16:27:13we're going to look at is this address.
16:27:16Now, let's just pretend that the people
16:27:17who are on the call center want all
16:27:20these separated into three different
16:27:21columns. They can read it easier, see
16:27:23what the zip code is, where they live,
16:27:25uh, you know, whatever they want it for.
16:27:26Let's just say we want to do that. This
16:27:28is, you know, again, for this use case,
16:27:30it may not make sense, but you have to
16:27:32do this. I do this all the time. Um, you
16:27:34need to split those columns. Now,
16:27:36luckily, all of these things are
16:27:38separated by a comma. So, we can specify
16:27:41that. We're going to split on this
16:27:42column. And then we'll be able to create
16:27:44three separate columns based off of this
16:27:46one column, which is exactly what we
16:27:48want. And we can name it as well. And we
16:27:51can do that very easily by using this
16:27:53split. So, we're going to say df and we
16:27:56want to specify oops.
16:27:59Oh jeez, not again. So, we want to
16:28:02specify that we're looking at the
16:28:04address. Then we're going to say dot
16:28:07string dotsplit.
16:28:10We'll do an open parenthesis. Now, the
16:28:12very first value that we need to specify
16:28:14is what we're splitting on. So, we want
16:28:16to split on the comma. So, we want to
16:28:18specify that. And then we need to
16:28:20specify how many values from left to
16:28:23right it should look for. Now we'll just
16:28:25start with one and then we'll go from
16:28:28there. Let's just see what this looks
16:28:30like.
16:28:32So
16:28:34it doesn't really look like it did
16:28:37anything. Let's do two. Well, let's go
16:28:39back to one and then let's say expand
16:28:43equals true. When we expand it, it's
16:28:46actually going to uh separate it, I
16:28:47believe. Okay, so we're expanding. We're
16:28:49now we're only doing this with one
16:28:51comma. So we're only looking at the very
16:28:53first comma and splitting it. But in
16:28:55some of these, well just in one there is
16:28:58an additional comma. So we should do it
16:28:59up to two. Let's do this. Okay. So now
16:29:03we have three columns. If we just save
16:29:06it like this, it's going to give us
16:29:07these 012. These basically these indexed
16:29:09values for these columns. And we don't
16:29:12want that. We want to specify what these
16:29:14actually are. And we can do that by
16:29:16saying df. And let me just do is equal
16:29:18to we'll do bracket and then within
16:29:21there we're going to specify our list.
16:29:23So we have three of them that we have.
16:29:26So I'm going to do um the first one this
16:29:29is the street address. So we'll say
16:29:31street address. The next one is um it's
16:29:36sh is not a state uh but these all are
16:29:39state. So I'm just going to say state.
16:29:42And then the very last one that looks
16:29:45like a zip code. So we'll say zip and
16:29:48we'll do underscore [clears throat]
16:29:49code. In fact, I also want to do street
16:29:52address. Um, so what this is now going
16:29:55to do is these three columns are going
16:29:56to be applied to these three names and
16:29:58they'll basically be appended. It
16:30:00doesn't replace the address. We're not
16:30:03saying DF address equals the DF address.
16:30:05We're not replacing it. We're now
16:30:07creating different columns. So let's run
16:30:10it. And then let's also call it. So
16:30:12they're right over here on this right
16:30:14hand side. I couldn't see them at first.
16:30:16But it did exactly what we needed it to
16:30:18do. So now if we wanted to at the very
16:30:20end, if we want to, we're not going to,
16:30:22we could just delete this address and
16:30:24keep the street address, the state, and
16:30:27the zip code. Another really common
16:30:30thing that you can do, this happens
16:30:32often again with like first name, last
16:30:34name. Well, you have Alex Freeberg, but
16:30:36it's Alex, Freeberg or Alex Space
16:30:38Freeberg, and you can separate those out
16:30:40into different columns. Now, the next
16:30:42one that we want to look at is this
16:30:44paying customer. and the paying customer
16:30:46and do not contact are very similar. Um,
16:30:50in the fact that it's yes, no, NY, yes,
16:30:53no, NY.
16:30:55Um, and so let's go right on down here
16:30:58and we're going to say df dot and we
16:31:00want to just replace these values as all
16:31:04yeses or all nos, but just with the same
16:31:07formatting um, just to keep it
16:31:09consistent. So, let's make anything
16:31:10that's an N into a no. Anything that's a
16:31:13a Y into a yes. I like it spelled out.
16:31:16So, let's change anything that's uh a
16:31:18yes into a Y. Anything that's uh a no
16:31:23into an N. That's usually how I do it.
16:31:26Just saves on data because it's less
16:31:28strings, although it's be often very
16:31:30minimal. Um, but let's specify the
16:31:34customer.
16:31:36We'll s say df bracket paying customer.
16:31:40Then we'll do string.replace.
16:31:43So now we're just going to look for
16:31:45those specific values. So if it's a y,
16:31:49oops, a capital y, then we'll say yes.
16:31:54Now let's run it. And now we have no
16:31:56more y's. We now just have yeses.
16:31:59Although now these are yeses. Okay, we
16:32:02don't want to do that. Let's do if we're
16:32:06looking because it's taking it's
16:32:08literally looking up here and saying
16:32:09okay there's here's a y. Um let's change
16:32:12the let's change that y into a y. So now
16:32:14it's doing y es. Uh we don't want that.
16:32:17So let's look for the yes
16:32:20and change into a y. Now when we run
16:32:22this that looks a lot better. Um so
16:32:26we'll do
16:32:28df paying customers equal to and then
16:32:31we'll copy this. We'll do the exact same
16:32:34thing. No.
16:32:36And N.
16:32:38Then let's call it. And now that entire
16:32:42column looks really good except for that
16:32:44value right there. But I'm going to
16:32:46leave that because I'm just going to
16:32:47apply it to the entire thing all at once
16:32:50to get rid of those at the end instead
16:32:52of just going column by column. And then
16:32:54it's literally going to be the exact
16:32:55same thing. So I'm not even going to
16:32:57scroll down. Whoops. I'm just going to
16:33:00put it right up here because this is the
16:33:02exact same thing. I'm going save us all
16:33:04some time.
16:33:08And when we run this, this looks exactly
16:33:10like what we're looking for. Again, some
16:33:12not a number values, but we can get rid
16:33:14of that in just a second by doing our
16:33:16place over the entire data frame. And
16:33:19that is basically the end of cleaning up
16:33:21individual columns. Now, let's go right
16:33:24down here. We're going to say
16:33:25df.string.replace
16:33:27replace and then we'll first do these
16:33:30values.
16:33:32Oops. So, we'll do Oops. Let me do that.
16:33:36There we go. And replace that with
16:33:38nothing. And let's just see what it
16:33:40looks like. Oops. Data frame object has
16:33:43no value string. Well, that's cuz we
16:33:45were looking at columns before. Yeah, I
16:33:47think I just need to get rid of this
16:33:49string. We're not looking we're doing it
16:33:51across the entire data frame. Now, let's
16:33:53try that.
16:33:54Okay, that worked appropriately. And
16:33:57we'll just say data frame is equal to.
16:34:00And then we'll copy this. And we'll do
16:34:03the nan as well.
16:34:06And we'll do
16:34:11and now when we do this, it is not going
16:34:13to replace these because these aren't
16:34:15actually a value because we're looking
16:34:17for that string. We actually need to use
16:34:18and I I completely forgot this. I'm not
16:34:20going to lie to you. Um let's get rid of
16:34:23this. uh to get rid of those values
16:34:24because it's literally not a number
16:34:26there. It is technically empty. Um I
16:34:30forgot we can do um or we could not even
16:34:33specify it. We'll do df.fill na. So
16:34:37we're going to fill these values if
16:34:39there's nothing in them. We're going to
16:34:41fill it and we're going to say
16:34:44blank. And when we run that, every value
16:34:47that doesn't have something in it is
16:34:49going to show up blank. Even over here
16:34:51where we only had a few all of them
16:34:53throughout the data frame if it doesn't
16:34:54have a value it is now blank. So let's
16:34:57apply that
16:35:00and we'll run this.
16:35:02And now all of our cleaning we're
16:35:05actually cleaning up the individual
16:35:07columns is completely done. We've
16:35:10removed columns. We've split columns.
16:35:12We've formatted and cleaned up phone
16:35:14numbers. We've also taken values off of
16:35:17first name or or this last name column
16:35:20and then we formatted and just kind of
16:35:22standardized paying customer and do not
16:35:24contact. Now they also asked us to only
16:35:28give them a list of phone numbers that
16:35:30they can call. So if we take a look some
16:35:33of these do not contacts are why which
16:35:35means we cannot contact them. And then
16:35:39there are some that don't even have
16:35:40phone numbers. So, we don't want to give
16:35:42the people the call center numbers that
16:35:45or or people who don't have numbers. So,
16:35:48we want to remove those. Now, there's a
16:35:50few different ways that we can do this.
16:35:53But let's start with and we'll just go
16:35:55by do this. Do not contact. It seems
16:35:58like the most obvious one. Now, if it's
16:36:00blank, we want to give them a call. We
16:36:03only want to not call them if they've
16:36:05specifically said we cannot call them.
16:36:07So, if it's why, we're not going to call
16:36:09them. So what we need to do and it's not
16:36:12anything like this. We probably need to
16:36:15loop through this column and then look
16:36:18at each row that has a value of this and
16:36:21drop that entire row. Uh and we probably
16:36:24will need to do that based off this
16:36:26index instead of doing it based off just
16:36:29this column. Uh that may not make sense,
16:36:32but let's actually let's actually start
16:36:34writing it. So we'll do 4x in and we
16:36:38need to look at our index. So we're just
16:36:40going to do let's do in df.index
16:36:44and we'll do a colon enter. And then we
16:36:47want to look at these indexes. How do we
16:36:50look at these indexes? We use lock.
16:36:52That's going to be df.loc.
16:36:55And then we need to look at the value
16:36:57which is this x right here. So each time
16:37:00it looks at the index, it's looking at
16:37:02the value. But we want to look at the
16:37:04value of this column. Do not contact. I
16:37:08don't know if I copied this before. Let
16:37:09me copy it. We only want to look at the
16:37:11value in this one column. If we didn't,
16:37:14it would look at um a different value.
16:37:16So we don't want that. So we're looking
16:37:18at just that value if it's equal to y.
16:37:22So if this value is equal to y, then we
16:37:26want to drop it. So we actually need to
16:37:27say if.
16:37:29So if this value x in this column is
16:37:33equal to y then we want to do df.drop
16:37:37and then we'll say x and we I think we
16:37:41have to say in place equals true here
16:37:43otherwise it won't take effect. Um
16:37:46otherwise you have to say like df is
16:37:48equal to df.ai and I don't I don't want
16:37:50to start messing with that. Let's just
16:37:51do in place equals true.
16:37:54Um and let's see if that works. I I
16:37:58can't remember if this is going to work
16:37:59or not. Invalid syntax. Okay. Need a
16:38:02colon.
16:38:04And now let's try to run this.
16:38:07Okay. Okay. Yeah. If we look at our
16:38:09index, we can already tell that there
16:38:11are ones missing. The one the one is
16:38:13missing, the three is missing. Uh let's
16:38:16see. And the 18 is missing. So, we
16:38:18already got rid of those values. And you
16:38:20can you can see that there's no Y's in
16:38:21here anymore, which is really good. We
16:38:24can if we want to, and we probably
16:38:25should. We should probably populate that
16:38:28um really quickly.
16:38:30Um let me just go up here really quick.
16:38:35I'll copy this. We probably should
16:38:38populate that. And I didn't plan on
16:38:40doing this. So, um if it's blank, oops.
16:38:44If it's blank, give it an N. And we want
16:38:47to attribute it to do not contact.
16:38:52Do
16:38:53not contact. Whoops.
16:38:57Let's see if that works.
16:39:00And we probably need to do string.
16:39:03Let's just see if it works.
16:39:06So, if it's blank,
16:39:09dude. Okay. I don't know why it's giving
16:39:10us a triple N.
16:39:13Maybe there's Maybe I need to strip this
16:39:15or something.
16:39:17Uh, okay. Never mind. Let's not do that.
16:39:22But now we basically need to do the
16:39:24exact same thing for this phone number.
16:39:26Um because if it's blank, we don't want
16:39:29them calling it. Um so we can copy this
16:39:32entire thing. Go right down here. And
16:39:35but now we're looking at phone number.
16:39:38So now we're looking just at the values
16:39:41within phone number. And we only want to
16:39:43look at if it's blank. So if it
16:39:44literally has no value, we want to get
16:39:46rid of it. Let's run this and see if it
16:39:49works again. It should. Good. And now
16:39:52our list is getting much smaller. So you
16:39:54can see in our index a lot of um those
16:39:58rows were removed. And okay, good.
16:40:01Actually, this worked itself out because
16:40:02these all have ends. Um so right now
16:40:05we're sitting really good. Everything
16:40:07looks really um standardized, cleaned.
16:40:11Everything looks great. I might drop
16:40:13this address. If you want to, you can
16:40:15drop this address, but besides that,
16:40:17this is all looking really good. this
16:40:18paying customer doesn't uh the yes and
16:40:20nos aren't really anything. Um, now we
16:40:24could and we probably should before we
16:40:26hand this off to the client or the
16:40:29customer called list, we probably should
16:40:30reset this index because they might be
16:40:33confused as why there's numbers missing
16:40:34or you know they might use
16:40:36[clears throat] this index um to show
16:40:38how many people they've called or I
16:40:39don't know something like that. So let's
16:40:41go right down here. We're gonna say df
16:40:44dot and then we'll do reset
16:40:47index.
16:40:49And let's just see what this looks like.
16:40:51Um, it does work, but as you can tell,
16:40:53it didn't uh get rid of that index
16:40:55completely. It actually took the index
16:40:57and saved that original one. We do not
16:41:00need to save that. Whoops. Let's put it
16:41:02right in here. Now, we're just going to
16:41:03do drop equals true. And when we do
16:41:06that, it just completely resets. It
16:41:08drops the original index and gives us a
16:41:10new index. And that is what we want.
16:41:13Let's do df equals. And this is our
16:41:16final product. Now, one thing that I you
16:41:20definitely could have done here, um, and
16:41:22I made this a little probably more
16:41:23complicated than it needed to be. Um,
16:41:25that was just how my brain was working
16:41:26at the time when I'm, you know, typing
16:41:28this out. We could have done df.rop
16:41:33a um, which is literally going to look
16:41:35at these null values. Um, before
16:41:39we couldn't do that with this one
16:41:40because these aren't we're not looking
16:41:41at NA, we're looking at Y's. So, we
16:41:43couldn't do that. But because we're
16:41:45looking at null values, we could have
16:41:46also done drop NA. Um, and done subset
16:41:51is equal to and then done it just on
16:41:54this phone number and then done like
16:41:58this and done in place equals true. So,
16:42:01we could have also done this then said
16:42:04df equals. Um, I can't I mean I can run
16:42:07it. It's just not going to do anything.
16:42:09I can run it on the different column,
16:42:11but that'll mess everything up. But this
16:42:13is another way you can do it. And I'll
16:42:15just save it in case you want to. Um,
16:42:17I'll say another way to drop null
16:42:21values.
16:42:23There you go. And that'll just be a note
16:42:24for us in the future. Um, but this is
16:42:27our final product. It looks a lot
16:42:31different than when we first started. I
16:42:33mean, we had mistakes here. Completely
16:42:36different formatting in the phone
16:42:37number, different address, everything
16:42:38that we just talked about. Um, and this
16:42:40looks just a lot lot better. And you can
16:42:42tell why it's really important to do
16:42:44this process because again, we're
16:42:46working on a very small data set. I I
16:42:49purposely, you know, created this data
16:42:51set with these mistakes because, you
16:42:53know, when you're looking at data that
16:42:55has tens of thousands, a hundred
16:42:56thousands, a million rows, these are all
16:43:00things that are going to be applied to
16:43:01much larger scale and you won't be able
16:43:02to as easily see them. Um, you'll have
16:43:05to do some exploratory data analysis to
16:43:08find these mistakes and then you're
16:43:10going to need to clean the data or doing
16:43:12it at the same time when you're
16:43:13exploring the data. Uh, so you'll clean
16:43:15it up as you go. But these are a lot of
16:43:18the ways that I clean data, a lot of the
16:43:20things that you can do to make your data
16:43:22just a lot more standardized, a lot more
16:43:24um visually better. And then it really
16:43:27helps later on with visualizations and
16:43:29your you know actual data analysis. So I
16:43:32hope that that was helpful. I know that
16:43:33this was a long video. I'm sure it was.
16:43:35Uh but I hope that you got something out
16:43:38of this and you learned some of the
16:43:39techniques on how to actually clean data
16:43:40in pandas. If you like this video, be
16:43:42sure to like and subscribe. Check out
16:43:44all my other videos on pandas as well as
16:43:46Python. and I will see you in the next
16:43:47video.
16:44:00Hello everybody. Today we're going to be
16:44:02looking at exploratory data analysis
16:44:04using pandas. Exploratory data analysis
16:44:07or EDA for short is basically just the
16:44:10first look at your data. During this
16:44:12process, we'll look at identifying
16:44:13patterns within the data, understanding
16:44:15the relationships between the features
16:44:17and looking at outliers that may exist
16:44:18within your data set. During this
16:44:20process, you are looking for patterns
16:44:22and all these things, but you're also
16:44:23looking for um mistakes and missing
16:44:25values that you need to clean up during
16:44:27your cleaning process in the future.
16:44:29Now, there are hundreds of ways to
16:44:30perform EDA on your data set, but we
16:44:33can't possibly look at every single
16:44:35thing. So I'm just going to show you
16:44:36what I think are some of the most
16:44:38popular and the best things that you can
16:44:40do when you're first looking at a data
16:44:41set. The first thing that we're going to
16:44:43do are import our libraries. So we'll do
16:44:45import pandas as pd.
16:44:48We're also going to import seabor and
16:44:51mapplot lib. Now, during this
16:44:53exploratory data analysis process, I
16:44:56often like to visualize things as I go
16:44:59because sometimes you just can't fully
16:45:01comprehend it unless you just visualize
16:45:03it and it gives you a a larger broader
16:45:06glimpse of everything. So, we're going
16:45:08to import and let's do seabour oops
16:45:13sns and then we'll import
16:45:16mattplot.lib.pipplot
16:45:19piplot
16:45:20as plt.
16:45:23Let's run this.
16:45:26That should work. Okay, perfect. Now, we
16:45:29need to bring in our data set. So, we've
16:45:31worked with that world population data
16:45:32set. That is the exact one that we're
16:45:34going to use now. So, we'll say dataf
16:45:36frame equals pdread
16:45:40csv. Do r and we'll paste in our CSV.
16:45:45And this is what it should look like.
16:45:47Although your path may be different, be
16:45:48sure to make sure that you have the
16:45:50correct file path. Then we'll read it
16:45:52in. Now, this data set should look
16:45:54extremely familiar if you've done some
16:45:56of my previous pandas tutorials, but I
16:45:59did make some alterations to this one.
16:46:01Took out a little bit of data, put in a
16:46:03little bit of data here and there. um to
16:46:05change things up because if it was just
16:46:08exactly how I pulled it, which I got
16:46:09this data set from Kaggle, if it was
16:46:11exactly how we pulled it, like we've
16:46:13looked at in the previous videos, it's
16:46:15too simple. You know, we wouldn't
16:46:16actually be able to do some of the
16:46:17things that I would like to show you.
16:46:19So, be sure to actually download this
16:46:21exact data set for this video because it
16:46:24is a little bit different.
16:46:26But what we're going to do now is just
16:46:28try to get some highlevel information
16:46:30from this. Now, if yours looks just a
16:46:32little bit different, like your values
16:46:34are in scientific notation, uh I have
16:46:37applied this so many times I think it's
16:46:39um you know, still applied to this, you
16:46:41can do something and we'll write it
16:46:43right down here. We're going to do PD
16:46:45set option and we'll do an open
16:46:49parenthesis and we'll say
16:46:51display.flat_mat.
16:46:55And so we're going to change that float
16:46:56format by just saying lambda x colon and
16:47:00then we're going to change basically how
16:47:02many um decimal points we're looking at.
16:47:05So let's just do here. So we'll do a
16:47:09quote percent sign 2f. So we're
16:47:12formatting it. Whoops. 2F. So we're
16:47:14going to format it and we'll do percent
16:47:17x. This is going to format it
16:47:19appropriately. I'm I can run it. Um, and
16:47:21actually it will change it cuz this is
16:47:23at 0.1 I believe last time I did it. So
16:47:25let's run this. And then let's run this
16:47:28again. It'll change it to 0 2. So that's
16:47:30two. I like it at 0.1. We don't really
16:47:33need it any Well, let's keep it at 0 2.
16:47:36Why not? We're going to keep it at 0 2.
16:47:38That's how you change that. And I like
16:47:40looking at it like this a lot better
16:47:41than scientific notation. So just
16:47:44something to point out. Um, let's go
16:47:46down here and let's just pull up data
16:47:48frame. So we have this data. One of the
16:47:51first things that I like to do when I
16:47:53get a data set is to just look at the
16:47:55info. So we're going to do info. And
16:47:57this gives us just some really highlevel
16:48:00information. This is how many columns we
16:48:02have. Here are the column names. Here
16:48:04are how many uh values we have. And if
16:48:07you notice, this is where it kind of
16:48:09gets. So we have 234 in each of these.
16:48:13So in each of these columns, we have 234
16:48:15until we get to this 2022 population.
16:48:18Once we get there, we start losing some
16:48:21values. And then at the world population
16:48:24percentage, we have all of our values,
16:48:26all 234 of them. The count tells us that
16:48:29it's non-null, so it does have values in
16:48:31it. And then we also have the data
16:48:32types, and these come in handy later.
16:48:35Um, and these are really great to know,
16:48:37and we'll be able to kind of use those
16:48:39in a few different ways later on in this
16:48:41tutorial. Really quickly, I wanted to
16:48:43give a huge shout out to the sponsor of
16:48:44this entire Panda series, and that is
16:48:46Udemy. Udemy has some of the best
16:48:48courses at the best prices and it is no
16:48:50exception when it comes to pandas
16:48:51courses. If you want to master pandas,
16:48:53this is the course that I would
16:48:54recommend. It's going to teach you just
16:48:55about everything you need to know about
16:48:57pandas. So, huge shout out to Udemy for
16:48:59sponsoring this Panda series. And let's
16:49:00get back to the video. The next thing
16:49:02that I really like to do, and this one
16:49:04is df.describe.
16:49:07This allows you to get really a
16:49:09highlevel overview of all of your
16:49:11columns very quickly. You can get the
16:49:13count, the mean, the standard deviation,
16:49:16the minimum value and the maximum value
16:49:19as well as your 25, 50 and 75
16:49:23percentiles of your values. So just at a
16:49:26super quick glance, there is a row
16:49:28somewhere in here and there, this
16:49:30country, their population is 510 for
16:49:332022. And in fact, if you go back to
16:49:361970, it was higher. It was at 752. I
16:49:39that's just interesting. Then if we look
16:49:41at the um max population, one has 1.42
16:49:45billion. I believe that's China. And
16:49:47then over here in 1970, we have 822
16:49:50million. Again, I still believe that's
16:49:52China. But this gives you just a really
16:49:54nice high level of all of these values
16:49:57or all these different calculations that
16:49:59you can run on it. And we can run all
16:50:01these individually on even specific
16:50:03columns. But you know, this is just a
16:50:05nice highle overview. One thing that we
16:50:07just talked about was the null values
16:50:09that we're seeing in here. Um, I'd like
16:50:12to see how many values we're actually
16:50:13missing because that is a problem. Um,
16:50:15we don't want to have too many missing
16:50:18values or could really obscure or change
16:50:21the data set entirely and so we don't
16:50:23want that. So, we'll say df.isnull
16:50:26and then we'll do a parenthesis and
16:50:27we'll say dot sum. And when we do this,
16:50:31whoops,
16:50:33dotsum. There we go. When we do this,
16:50:36it's going to give us all the columns
16:50:38and how many values we're actually
16:50:40missing. Now, we have 234
16:50:42rows of data. So, we have 41 47755424.
16:50:48Um, so we have we definitely have data
16:50:50missing. What we choose to do with it in
16:50:54the data cleaning process, maybe we want
16:50:55to populate it with a median value.
16:50:57Maybe we just want to delete those
16:50:59countries entirely if the data is
16:51:01missing. um you know, I don't think
16:51:03you're going to do that, but these are
16:51:05things that you need to think about when
16:51:07you're actually finding these missing
16:51:09values. This is what the EDA process is
16:51:11all about. We want to find different um
16:51:14either outliers, missing values, things
16:51:17that are wrong with the data or we can
16:51:19find insights into it while we're doing
16:51:21this as well. So, this is definitely
16:51:23something that I would consider um when
16:51:25I'm actually going through that data
16:51:26cleaning process. Really, really
16:51:28important information to know. Now,
16:51:29let's go right down here. go to our next
16:51:32cell, say df.
16:51:35This is going to show us how many unique
16:51:37values, and it's actually n unique. Uh,
16:51:40this is going to show us how many unique
16:51:42values are actually in each of these uh
16:51:46columns. And this one makes the most
16:51:48sense um for continent because I think
16:51:51there's only seven continents, right? Um
16:51:54but we have six right here. And for all
16:51:56of these, each of these ranks,
16:51:58countries, capitals should all be
16:52:00unique. That makes perfect sense. As
16:52:02well as these, you know, these
16:52:03populations are such specific numbers
16:52:05and such large numbers. I would be
16:52:07shocked if any of these were similar.
16:52:09And then for these world population
16:52:11percentages, it's much lower. And again,
16:52:14that makes a lot of sense because when
16:52:15we're looking at, and we'll pull it up
16:52:17right here. When we're looking at these
16:52:19world population percentages,
16:52:22um, a lot of them are really low.
16:52:230.00.01,
16:52:26like this one, um, 0.2. there are a lot
16:52:29of really low values for those small
16:52:31countries and so those are all um you
16:52:33know one unique value. Now let's say we
16:52:36just have this data right here and we
16:52:38want to take a look at some of the
16:52:39largest countries and we can easily do
16:52:42that. We could even we could say max and
16:52:44take a look at the largest country but I
16:52:46want to be a little bit more strategic.
16:52:47I want to be able to look at some of the
16:52:49top range of countries and we can do
16:52:51that based off this 2022
16:52:54population. So we'll say df.sort sort
16:52:58values. This is how we sort and um not
16:53:00filter but um order our data. So we'll
16:53:03do sort values and then we'll do by is
16:53:06equal and then we'll specify that we
16:53:08want uh this 2022 population and then
16:53:11we're going to say comma and we'll say
16:53:14actually let's just run this as is but
16:53:16we'll do head because we just want to
16:53:18look at the top values. So now we're
16:53:20just looking at the very top values. So,
16:53:23what we're looking at is actually these
16:53:252022 population. Um, that's what we're
16:53:28filtering on or sorting on basically.
16:53:30And we're looking at the very bottom
16:53:32values because it's sorting ascending.
16:53:34So, from lowest to highest. So, this
16:53:36Vatican City in Europe is um, you know,
16:53:40510. That's the value that we were
16:53:42looking at earlier. Now, we can do comma
16:53:45ascending equal to false because it was
16:53:47by default true. We can do false.
16:53:49Whoops. We can do false and then it'll
16:53:52give us the very largest ones. So if we
16:53:54just take a look at the top five largest
16:53:56by population, we're looking at China,
16:53:59India, United States, Indonesia, and
16:54:01Pakistan. And we can even specify that
16:54:04we want the top 10 in this head. We can
16:54:07bring in the top 10. We also have
16:54:09Nigeria, Brazil, Bangladesh, Russia, and
16:54:12Mexico. And you can do this for
16:54:13literally any of these columns. Whether
16:54:16you want to look at continent, capital,
16:54:18country, um you can sort on these and
16:54:20look at them and you can even look at,
16:54:22you know, things like growth rate, world
16:54:23percentage. This one seems really
16:54:25interesting. Let's just look at this one
16:54:27really quickly before we move on to the
16:54:28next thing. Um if we look at this world
16:54:32percentage, just China alone, I believe,
16:54:35yep, just China alone is 17.88%
16:54:39of the world. So 17.88 88 and 17.77
16:54:44and that's China and India and those are
16:54:46very large countries with a high high
16:54:48high population. That makes a lot of
16:54:50sense why that is the highest world
16:54:52population percentage. Again, just
16:54:54getting in here looking around. That's
16:54:56all we're really doing. Now, I want to
16:54:58look at something and I have always like
16:55:00doing this which is looking at
16:55:01correlations. Um, so a correlation
16:55:03between usually only numeric values. We
16:55:07can do that by saying df.co C O R R and
16:55:10a parenthesis. And we'll run this. And
16:55:13what this is is it is comparing every
16:55:16column to every other column and looking
16:55:19at how closely correlated they are. So
16:55:22this 2022 population, if we look across
16:55:24the board, it's very highly I mean this
16:55:27is a one one. This is highly correlated
16:55:30to each other. And that almost for all
16:55:32of these populations, they're very very
16:55:34closely tied to each other, which makes
16:55:35perfect sense because for most
16:55:38countries, they're going to be steadily
16:55:40increasing. And so they're probably
16:55:42almost exactly correlated. But we can
16:55:45look at these populations and if you
16:55:47look at the area, it's only somewhat
16:55:50correlated. And that's because in some
16:55:52countries, you know, they have a very
16:55:54high population but a small area. Or
16:55:56vice versa, small area and a very high
16:55:58population. So there isn't a onetoone
16:56:00correlation there, but it's hard to
16:56:02really just glance at this um and
16:56:04understand everything that's there. We
16:56:06could just visualize it and it would be
16:56:08a lot easier. So let's go ahead and do
16:56:11that. Let's go down here. We're just
16:56:13going to visualize this using a heat map
16:56:16basically. So we're going to say
16:56:17sns.heatmap
16:56:20and an open parenthesis. And the data
16:56:23that we're going to be looking at is
16:56:24df.core
16:56:26correlation. And then we also want to
16:56:28say anote equals true. We'll kind of
16:56:32show you what that looks like in just a
16:56:33little bit. Um, but let's do plt.show.
16:56:38And this will be our first look. And I
16:56:40need to say show, not shot. Um,
16:56:45we can get a little glimpse of what it
16:56:46looks like, but this looks um,
16:56:47absolutely terrible. Let's change the
16:56:50figure size really quickly. So, I want
16:56:52to make this much larger than it already
16:56:54is. We'll do pltr
16:56:58params rc params. Oops. Right there. Do
16:57:03an open parenthesis. And then right here
16:57:05we're going to do in quotes. Do figure
16:57:09size. This actually needs to be in
16:57:11brackets I believe.
16:57:13Just like this, not parentheses. We'll
16:57:16say fig size is equal to. And now we can
16:57:19specify the value that we want. Let's do
16:57:2110 comma 7 and see if this looks any
16:57:24better.
16:57:25No. No, that doesn't look good. Do 20.
16:57:30Okay, that looks a lot better. And um
16:57:34you know, this is just a quick way
16:57:36because it gives you basically a
16:57:37color-coded system. Highly correlated is
16:57:40this tan all the way down to basically
16:57:42no correlation or negative correlation
16:57:44even, which is black. So when we're
16:57:46looking at these 2022 populations and
16:57:48these are populations right down here on
16:57:51this axis, we can see that all of these
16:57:53are extremely highly correlated very
16:57:57very quickly. Whereas the rank really
16:57:59has nothing to do it's it's negatively
16:58:02correlated doesn't really have anything
16:58:03to do with it. Then for the population
16:58:05and the world population percentage it
16:58:08again is quite correlated except for the
16:58:12area density and growth rate. So I find
16:58:15that really interesting that you know
16:58:17the density the growth rate in the area
16:58:19aren't really all that associated or
16:58:23correlated with the population numbers
16:58:26that is I kind of would have assumed
16:58:29that on some level they went handinand
16:58:31the area does um which you know again
16:58:34makes sense you know larger area larger
16:58:35population that kind of thing but even
16:58:37density um I guess I guess density and
16:58:40growth rate um growth rate I can see
16:58:42because that's a percentile thing that
16:58:44could Definitely not correlated. I
16:58:46thought the density would be more
16:58:48correlated than it is. All that to say
16:58:50is this is one way that you can kind of
16:58:52look at your data, see how correlated it
16:58:54is to one another. That can definitely
16:58:56um help you know what to analyze and
16:58:58look at later when you're actually doing
16:59:00your data analysis. Let's go right down
16:59:02here. Um something that I do almost all
16:59:05the time when I'm doing any type of
16:59:07exploratory data analysis like this, I'm
16:59:09going to group together columns, start
16:59:11looking at the data a little bit closer.
16:59:14Um, so let's go ahead and group on the
16:59:16continent. So let's look at it right
16:59:19here. Let's group on this continent
16:59:20because sometimes when you're doing this
16:59:22EDA, you already know kind of what the
16:59:24end goal of this data set is. You know
16:59:27kind of what you're looking for, what
16:59:28you're going to visualize at the end
16:59:29that you really comes in handy when
16:59:30doing this. But sometimes you don't.
16:59:33Sometimes you're just going in blind.
16:59:34And so far we've really just been going
16:59:36in blind. We're just throwing things at
16:59:38the wind, kind of seeing some overviews,
16:59:40um, looking at correlation. That's all
16:59:42we've done. Now I kind of want to get
16:59:44more specific. I want to have like a use
16:59:46case, something I'm kind of looking for.
16:59:49Not doing full data analysis or not
16:59:51diving into the depths, but something we
16:59:53can kind of aim for. So the use case or
16:59:55the question for us is are there certain
16:59:57continents that have grown faster than
16:59:59others and in which ways? So we want to
17:00:03focus on these continents. We know that
17:00:04that's the most important column for
17:00:06this use case, this very fake use case.
17:00:08Um, so we can group on this continent
17:00:11and we can look at these populations
17:00:13right here because we can't really see
17:00:15growth. You can see a growth rate, but
17:00:18the density per uh kilometer, we don't
17:00:21have multiple values for that. It's just
17:00:23a static one single value. Same for
17:00:25growth rate, same for world population
17:00:27percentage. But we have this over a long
17:00:30span, many many years um you know 50
17:00:33years of data here. So this we can see
17:00:36which countries have really done well or
17:00:38which continents have really done well.
17:00:40So without you know talking about it
17:00:42even more let's do dfgroup by and then
17:00:45we'll say continent. Oops. Let me just
17:00:50copy this. I'm I am not good at
17:00:52spelling. We're going to say dfgroup by
17:00:54and then we'll do mean. And we can just
17:00:57do it just like this. And now we have
17:01:00Africa, Asia, Europe, North America,
17:01:03Oceanana, and South America.
17:01:07Okay, so if I'm being completely honest,
17:01:10I knew most of these. All right, I'm no
17:01:12geography expert, but I I knew most of
17:01:14these. I don't know what this ocean is.
17:01:16Um this that [clears throat] I don't I
17:01:19genuinely don't know what that is. Um so
17:01:22let's just search for that value and
17:01:25see. We'll come back up here in just a
17:01:27second, but I want to I want to kind of
17:01:29understand um what this is. So, we're
17:01:31going to df um then we'll say content.
17:01:36Let me sound that out for you guys. Um
17:01:39then we'll do string.contains.
17:01:42Oops. Contains. Good night. And then I
17:01:46want to look for ocean. Uh and let's
17:01:50let's run this. Oh, I need to do like
17:01:53this.
17:01:57Now let's run this. So now we're looking
17:01:59at our data frame and we're seeing when
17:02:01the values have this continent as ocean.
17:02:05Um okay. So these look like islands I'm
17:02:08guessing. So we have Fiji, Guam,
17:02:12um New Zealand,
17:02:15Papa New Guinea. Yeah, these look like
17:02:17all I'm I'm guessing based off the
17:02:20continent Oceanana. Um, Oceanania Oceana
17:02:25Oceania, guys, this is tough for me.
17:02:28Okay, I'm doing my best. I, you know,
17:02:30this is part of the EDA process. I don't
17:02:32know what that means. I don't know what
17:02:33Oceanana Oceania.
17:02:37Jeez, I'm just going to call it
17:02:38Oceanana. That's so wrong, but I'm just
17:02:40going to so easy for me to say, you
17:02:42know, I I now am seeing this and it
17:02:45[snorts] looks like islands. Um, which
17:02:48would make sense because
17:02:51for their average, they have the highest
17:02:52average rank. Um, and I'm guessing
17:02:56that's because they're just mostly small
17:02:58continents. So, let's let's order this
17:03:00really quickly. We're going to do dot
17:03:02sort
17:03:04values. Do an open parenthesis. And I
17:03:07want to sort on the population. We're
17:03:09just doing the average population. Um,
17:03:12we'll do by um equal. So on the average
17:03:16population and we'll do ascending equals
17:03:20false. So when we're looking at this
17:03:22average or the mean population, Asia has
17:03:25the highest population on average. Then
17:03:28we have South America, Africa, Europe,
17:03:31North America, and then Oceanana at the
17:03:35very bottom, which makes perfect sense.
17:03:36Again, small islands. Um world
17:03:40population percentage. So each of the
17:03:42countries each of those countries in
17:03:44Asia makes up about 1% on average.
17:03:47Really interesting um to know and just
17:03:50kind of look at this and the density in
17:03:54Asia is far higher than double almost
17:03:57double every single other continent. Um
17:04:01really really interesting actually now
17:04:02that I'm looking at this. But you know
17:04:04that's something that I would actually
17:04:06look into and I I would be like what is
17:04:08this Oceanana or Oceania? what does that
17:04:11mean? And you know, let me look into
17:04:13that. Let me explore that more because I
17:04:15want to know this data set. I'm trying
17:04:16to really understand this data set well.
17:04:18But what I want to do now is I want to
17:04:20visualize this. Um because I just feel
17:04:23like looking at it, I don't it's hard to
17:04:25visualize. And again, the use case that
17:04:27we're saying is is which continent has
17:04:29grown the fastest. Like it could be
17:04:31percentage- wise, it could be um you
17:04:34know, as just a whole on average. Let's
17:04:36take a look. So we're going to take this
17:04:39and let's copy it like this. Let's bring
17:04:42this right down here. So let's look at
17:04:44this. So if I try to visualize this and
17:04:49let's do that. Let's do DF2 is equal to
17:04:52because I'm I already know it's not
17:04:54going to look good just based off how
17:04:56the data is sitting. Um we can do DF2.
17:05:00Oops, what am I doing? I don't need to
17:05:03do that, but I will. Okay, DF2. and
17:05:05we'll do df2.lot
17:05:08and we'll run it just like this. Um,
17:05:11[clears throat]
17:05:12as you can see, Asia, South America,
17:05:15Africa, Europe, North America, Oceanana,
17:05:18we can kind of understand what's
17:05:20happening, but these are the actual um
17:05:23values that are being visualized, not
17:05:26the continents, which is what I wanted
17:05:28um in order to switch it. And it's
17:05:30actually pretty easy and this is
17:05:31something that um you know is good to
17:05:34know. We can actually transpose it to
17:05:36where these these continents become the
17:05:38columns and the columns become the
17:05:40index. And all we have to do is say df2
17:05:45transpose
17:05:47and we'll do this parenthesy right here.
17:05:49And let's just look at it and then we'll
17:05:51save it. So now all these columns are
17:05:55right here.
17:05:57and all of the indexes are the columns.
17:06:00So let's say df3 is equal to and I'm
17:06:03just doing that so I don't you know
17:06:04write over the df for my earlier dataf
17:06:06frames. So now we have this dataf frame
17:06:08three. So now let's do dataf frame
17:06:11three.plot and it should look quite a
17:06:13bit different.
17:06:15Uh whoops I didn't run this. Let's run
17:06:18this and run this.
17:06:22And as you can see this does not look
17:06:24right at all. And the reason is is
17:06:26because we're not only looking at uh the
17:06:29correct columns. We have this density in
17:06:31here. We're population percentage rank.
17:06:33We don't need any of those. The only
17:06:35ones that we want to keep are these ones
17:06:37right here. This population. Now, we can
17:06:40do that. And we can just go right up
17:06:42here. This is where we created that data
17:06:44frame 2 that we transposed. We can go
17:06:46right up here and we can specify within
17:06:49this. We actually only want specific
17:06:51values. Now, we can go through and
17:06:54handwrite all of these. And by all
17:06:56means, go for it. But I am going to go
17:06:59down here. I'm going to say df.c
17:07:01columns. And I'm going to run this. It's
17:07:04going to give us this list of all of our
17:07:06columns. And I'm just going to You can
17:07:09just copy this.
17:07:11And you can put it right in here. Again,
17:07:13you get list with I think it needs to be
17:07:15like this if I'm Let me try running
17:07:17this. Okay. So, this worked properly.
17:07:19You can do it just like this or a little
17:07:21shortcut if you want to do it like that.
17:07:24If you want to do a shortcut like um I I
17:07:27would hope you would you would just do
17:07:29df.c [clears throat]
17:07:30columns just like how we looked at down
17:07:32here except since this is our an index
17:07:36we can search through it. So we can just
17:07:37say 0 1 2 3. Okay. So we can do five up
17:07:42to 13 because I think it's seven. And
17:07:44we'll just let's see if this works.
17:07:48Uh it may not. I may actually need to go
17:07:49like this. Let's see. There we go. So,
17:07:53you can just use, you know, the indexing
17:07:55to save you some visual space. Gives you
17:07:58the exact same output. So, now we have
17:08:00this. This is our DF2. Now, let's go
17:08:02down and transpose it. So, now we just
17:08:05have these populations and we have our
17:08:06continents right here. And then now
17:08:09we're going to plot it. And this looks
17:08:12good, although it's backward. Um, okay.
17:08:16It's backward.
17:08:18So, what I actually want to do is not
17:08:22this. Uh, that is a quick way to do it,
17:08:25although not the best way to do it. Um,
17:08:28so I'm actually going to copy all of
17:08:30these. And although I said it would save
17:08:32us time, it did not at all. So, I'm
17:08:35going to
17:08:36put a bracket right here.
17:08:39I'm going to paste this in here. And I'm
17:08:41literally going to change these up. I
17:08:44might speed this up or I might just have
17:08:47you sit through this because, you know,
17:08:49this is an interesting part of the
17:08:51process and I want, you know, you to get
17:08:52the full experience. You know what? Now
17:08:54that I'm talking about it, that is what
17:08:56we're going to do. You guys can hang out
17:08:57with me. This is a good time. We have
17:09:002010,
17:09:022015,
17:09:042020,
17:09:06and 2022. Now, let's run it. What did I
17:09:10do? Oh, too many brackets. There we go.
17:09:13So now it's ordered appropriately. We
17:09:15have 1970 all the way up to 2022. This
17:09:17is how we want it. Let's transpose it
17:09:20appropriately. Let's run it. And now we
17:09:23have basically have the inverted uh
17:09:25image of this. Now just at a glance and
17:09:28we haven't done anything to this except
17:09:30for literally what we are looking at. At
17:09:32a glance, we can see that from 1970,
17:09:36China, you know, Asia and China are
17:09:38already in the lead by quite a bit. And
17:09:41it continues to drastically go up,
17:09:44especially in the 2000s. Like right
17:09:46here, it explodes like just straight up,
17:09:50then kind of starts going up and just
17:09:52leveling off. Every other continent,
17:09:55especially ocean, ocean, is just really
17:09:58low. It It never has done a bunch. Let's
17:10:00see. Look at green. green has gone up um
17:10:02from you know point let's say 0.1
17:10:06up to about 02 so they've almost doubled
17:10:09um in the last 50 years and again you
17:10:12can just get an overview a highle
17:10:14overview of each of these you know
17:10:17continents over the span of this time so
17:10:20this is kind of one way that we can you
17:10:22know look at that use case we're not
17:10:25going to harp on that too long I just
17:10:26wanted to give you an example like you
17:10:28know when you're looking at this
17:10:30sometimes times you'll have something in
17:10:31mind of what you're looking for and you
17:10:33go exploring and just kind of find
17:10:35what's out there and find what you see.
17:10:37Um, the next thing I want to look at is
17:10:39a box plot. Now, I personally I love box
17:10:42plots. You know, they're really good for
17:10:44finding outliers and there's a lot of
17:10:48outliers. I already know this because
17:10:50the average, the 25th, 50th percentile
17:10:53are very low and then there's some
17:10:54really just big outliers. But for your
17:10:57data set, it may not be that way. And
17:10:59those outliers may be something that you
17:11:01really need to look into. And box plots
17:11:03have been something that I've used a lot
17:11:05where I found those outliers that way
17:11:06and started to dig into the data to find
17:11:08those outliers and you know came across
17:11:11some stuff that I'm like oh I have to
17:11:12clean this up. I have to go back to the
17:11:13source. Really um really really powerful
17:11:16and useful to be able to find these. So
17:11:18all you have to do is df.boxplot.
17:11:21Yeah. Let's take a look at it. And this
17:11:24already looks good as is. Maybe I'll
17:11:26make it a little bit wider. Um, let's do
17:11:28fig size. Oops.
17:11:32Sorry. Fig size is equal to let's try 20
17:11:37by 10.
17:11:39Um, okay. That didn't help at all. I
17:11:42apologize. I thought it would, but let's
17:11:44keep going. [clears throat] What this is
17:11:45showing us is that these little boxes
17:11:48down here, which are actually usually
17:11:50much larger because you have a more
17:11:52equal distribution of of um numbers or
17:11:54values in this small value. This is
17:11:57where our averages lie. This number
17:12:00right here is the upper range. And then
17:12:03all these values, all these open
17:12:05circles, those actually stand for
17:12:07outliers. So we're looking at the 2022
17:12:10population. There's a lot of outliers.
17:12:12[clears throat]
17:12:12Now for our data set, knowing our data
17:12:15set is really important. Outliers are to
17:12:17be expected, especially when most
17:12:20countries or continents are small. So
17:12:22we're looking at, you know, all of these
17:12:24little dots are outlier countries
17:12:27um or outlier values, which each value
17:12:30corresponds to a country. So if this was
17:12:32a different data set, I would be, you
17:12:34know, searching on these and trying to
17:12:36find these so that I can see what's
17:12:38wrong with them, if anything, or if they
17:12:40are real um numbers. Like if this was
17:12:42revenue, everyone's revenue is way down
17:12:43here, and then there's one company
17:12:44that's making like $10 trillion. that'd
17:12:47be an outlier up here and it would
17:12:49definitely be something that you want to
17:12:50look into for our data set knowing that
17:12:52you know we're looking at population.
17:12:54This is more than acceptable and you
17:12:56know oddly enough but that's what box
17:12:59plots are really good for showing you
17:13:01some of those cortiles the upper and the
17:13:02lower um as well as denoting these
17:13:05points that fall outside of those normal
17:13:06ranges for you to look into. So really
17:13:09really useful. So now let's go down
17:13:11here, pull up our data frame again, and
17:13:13we've kind of just zoomed into the whole
17:13:16EDA process. There was one last thing
17:13:18that I wanted to show you. Uh, this is
17:13:20the very last thing that we're going to
17:13:21look at. We're ending on really a low
17:13:22point if I'm being honest because the
17:13:24last kind of stuff was more much more
17:13:25exciting. But there is something
17:13:28dftypes.
17:13:31Oops. Let's do df.dtypes.
17:13:34And we'll run this. Now just like info
17:13:36it gave us these values but we're
17:13:39actually able to search on these values
17:13:41now. So these um object float and
17:13:44integer we can search on those which is
17:13:47really great because we can do include
17:13:49equal and we can do something like
17:13:51number and none of these are numbers
17:13:54right or none of them explicitly say
17:13:56number but when we run it I'm getting an
17:13:59error series object not oh that's
17:14:01because I'm doing um dtypes is for a
17:14:04series we need to do select underscore
17:14:08dtypes now let's run this now it's only
17:14:11returning um the columns in this data
17:14:14frame where the data types are included
17:14:17in this number. So you won't see any you
17:14:19know country or any of those text or the
17:14:22strings. If we wanted to do that, we go
17:14:25in here and say object
17:14:28and run that. And this is another really
17:14:30quick way where we can just filter those
17:14:33columns to look for specific whether
17:14:36it's numeric. Um we could even do float
17:14:38in here. And so now it's not including
17:14:41that rank which was an integer. So we
17:14:43can specify the type of data type and
17:14:45it'll filter all of the columns based
17:14:47off of that which you know when you're
17:14:49doing stuff like this you it is good to
17:14:51know what kind of data types you're
17:14:53working with and look at just those
17:14:54types of data types because there might
17:14:56be some type of analysis you want to
17:14:57perform on just that whether it's
17:14:59numeric or just the string or integer
17:15:02columns within your data set. So again,
17:15:04ending on a low note, I apologize. Um,
17:15:06you know, everything else that we looked
17:15:08at, all those other things that we
17:15:09looked at are all things that I
17:15:11typically do uh in some way or another
17:15:14when I'm looking at a data set.
17:15:16Exploratory data analysis is really just
17:15:19the first look. you're looking at it,
17:15:21you're going to be cleaning it up, doing
17:15:22the data cleaning process, and then
17:15:24you're going to be doing your actual
17:15:26data analysis, actually finding those
17:15:28trends and patterns, and then
17:15:29visualizing it um in some way to find
17:15:32some kind of meaning or insight or value
17:15:35from that data. And again, there's a
17:15:37thousand different ways you can go about
17:15:39this. It it does typically um you know,
17:15:42depend on the data set, but these are a
17:15:44lot of the ways that you'll clean a lot
17:15:46of different data sets. And so, you
17:15:48know, that's why I went into the things
17:15:49that we looked at in this video. So, I
17:15:51hope that you guys liked it. I hope that
17:15:52you enjoyed something in this tutorial.
17:15:54If you like this video, be sure to like
17:15:55and subscribe, as well as check out all
17:15:57my other videos on pandas and Python.
17:15:59And I will see you in the next video.
17:16:02[music]
17:16:13What's going on everybody? Welcome back
17:16:15to another video. Today we are back with
17:16:17another data analyst portfolio project
17:16:18where we will be scraping data from
17:16:20Amazon using Python.
17:16:24[music]
17:16:27Now you may be asking do I need to know
17:16:29web scraping to become a data analyst
17:16:31and the answer is no you absolutely
17:16:33don't need to know it but it is a very
17:16:35cool skill to learn and in fact I have
17:16:37used it in my job in the past and so it
17:16:39is useful but you really don't need to
17:16:42know it. something that it is used for
17:16:44is kind of creating your own data sets.
17:16:46Um, and we're going to be looking at one
17:16:48where you can create your own data set
17:16:49today, but there are a lot of other uses
17:16:51for web scraping and I'm sure I'll talk
17:16:53a little bit more about that while we're
17:16:54actually walking through the project.
17:16:56One last thing I want to say before we
17:16:57get started is that this is most likely
17:16:59an intermediate project. So, if you are
17:17:01just now learning the basics of Python,
17:17:02this might be a little bit challenging
17:17:04for you, but I still recommend going
17:17:06through it because I will do my best to
17:17:08walk through everything every single
17:17:09step of the way and and kind of explain
17:17:11all of the concepts and so you can still
17:17:13learn something even if you aren't super
17:17:15good at Python right now. With that
17:17:17being said, let's jump over to my screen
17:17:18and get started on the project. All
17:17:19right, so we are going to get started.
17:17:21And if you didn't watch the last
17:17:22project, I had people download Anaconda.
17:17:25Uh we use Jupiter notebooks. Um, and
17:17:28I'll show you how to get to that in just
17:17:29a second, but I'll I'll leave this link
17:17:30in the description if you haven't done
17:17:32that already and you are just doing this
17:17:34project. Um, but you'll go, you'll
17:17:36download Andaconda, you know, download
17:17:38super easy. Um, and you're going to open
17:17:39up Jupyter Notebooks. I'll launch it
17:17:41right now. I already have it open. Uh,
17:17:43but I'll open up another one just for,
17:17:45you know, the purposes of demonstration.
17:17:48What we are going to do today and what
17:17:50we um what people voted on. I mean,
17:17:53there's like there was like 8,000 people
17:17:55that voted um in the poll that I made of
17:17:58what data you wanted me to scrape. There
17:17:59was like Amazon cryptocurrency weather
17:18:03um something else, I don't remember.
17:18:05Overwhelmingly, I mean, like 70% of
17:18:07people, maybe even 80%, I you know,
17:18:09don't don't fact check me on that, voted
17:18:11for Amazon. Um and so I'm going to do
17:18:14it. Now, there are many things that you
17:18:17can scrape um off of Amazon. Just a ton
17:18:20of stuff. Um, and I'm going to show you
17:18:24how to do it. I'm going to show you how
17:18:26to make it useful, how to make a data
17:18:28set. Um, and it's going to be really
17:18:31interesting, but there are lots of other
17:18:33ways to do this. And so, I think, um,
17:18:35and I have already kind of created it.
17:18:37I'm going to show you how to do it off
17:18:39of this page. Um, when you're actually
17:18:41in an item, and you can scrape, you
17:18:43know, basically anything in here. Um,
17:18:45and I'll show you how to do that.
17:18:47Another thing that is a little bit more
17:18:49advanced and that's why this first video
17:18:51is starting off I think on the more easy
17:18:53side. It's not easy but it's easier. The
17:18:56next thing the next video that I'm going
17:18:58to make is how to actually do um
17:19:02basically do multiple items, right? So
17:19:05this item, this item, this item, this
17:19:07item, and then traverse through the
17:19:10different pages. So there's 20 pages. Um
17:19:13you want all of that data. How do you
17:19:15get all of that? That'll be the next
17:19:17project. Um, I don't know when I plan on
17:19:19doing that. I have it like 90% of the
17:19:21way done. Um, but I have this one
17:19:23completed and so I wanted to get that
17:19:25out to you guys now. But that'll
17:19:26probably be the next project. I think
17:19:27that is much more difficult. Um, and so
17:19:30if you can understand this one and you
17:19:32get it and and you understand it, then
17:19:34the next project you should be able to
17:19:35understand too is just a little bit more
17:19:37complicated. So with that being said,
17:19:40um, we are going to actually get into
17:19:41the project. I'm going to delete one of
17:19:43these. Um, all we're going to do is go
17:19:45to new, do Python 3. It'll open up a new
17:19:50one. We'll call this um Amazon
17:19:54Web
17:19:55Scraper
17:19:57um project. That's what we'll call it.
17:20:00Did I spell that right? Perfect. Um, the
17:20:03first thing that we need to do uh or
17:20:05that we should do is upload um or or or
17:20:10import our libraries. So, I'm going to
17:20:12say um import Oops. What am I doing? Was
17:20:16off to a terrible start. There we go.
17:20:19Import libraries. Now, I'm not going to
17:20:21write out all the libraries. Um I have
17:20:23some things that I'm going to be copying
17:20:25and pasting throughout this. I won't
17:20:27there's only a few things that I'm
17:20:28copying and pasting. You can take a
17:20:29quick glance. Um some of the things that
17:20:31I just don't want to waste time on. Um
17:20:32because this could be a long video. I
17:20:34don't know. I don't want to waste time
17:20:36on stuff like this. Um and so, you know,
17:20:39I'm just going to copy and paste it. You
17:20:41guys are going to I'm going there will
17:20:43be a link below if you haven't clicked
17:20:44it already that will go to the GitHub
17:20:46page where you can literally have all of
17:20:48this code already written. I do
17:20:51recommend writing it all yourself
17:20:52because you will learn it much better. I
17:20:54promise because then you'll make
17:20:55mistakes and you'll figure it out and
17:20:56all that all that good stuff. But you
17:20:58will have that code available. So just
17:20:59go copy and paste it. Um that's what I
17:21:01would do. But what we are we are going
17:21:03to be using today is uh something called
17:21:05beautiful soup requests. Um, then we're
17:21:09going to be using time and datetime. And
17:21:12a potential one if you want to get, and
17:21:14I'm going to show you this at the end.
17:21:16This is not really part of the project.
17:21:17It goes above and beyond. But this
17:21:19library right here is for sending emails
17:21:22to yourself. Um, and I'll show you how
17:21:24uh you can use it if you want to. I
17:21:27already have the whole code written out.
17:21:28Um, you can just steal it and try it out
17:21:30yourself and see if you can get it to
17:21:32work. But this one is not um as
17:21:34important. I'll put it down here. So,
17:21:38um, let's move on. Now, one thing I want
17:21:40to say before we get too into it is
17:21:42[clears throat] that, well, give me a
17:21:43second
17:21:45is that right here in front of me is a
17:21:48different laptop. Now, it took me a
17:21:51solid, I would say, you know, 10 hours
17:21:55or so to write all of this. It took over
17:21:58the course of like two weeks in my free
17:21:59time. I'd pick it up. It took me a
17:22:01solid, you know, two weeks on and off,
17:22:04an hour here, an hour there. to finish
17:22:06this project. Um, and I made a ton of
17:22:09mistakes and messed a bunch of things up
17:22:11and I finally got it to work. Um, you
17:22:13know, after a bunch of revisions, that's
17:22:14typically how things go when I do
17:22:16projects. And so, uh, I'm about to give
17:22:19you a streamlined version of this
17:22:21because I have all the code right down
17:22:24here. And so, I'm going to be glancing
17:22:25at this a lot. Um, just so I don't make
17:22:29this video 20 hours of trying to
17:22:30remember all the code off the top of my
17:22:32head. I have it written out already. I
17:22:34already did the project. It works. It's
17:22:35beautiful. It's a good project. So, um I
17:22:37don't want to waste your time and I just
17:22:39want you to know that, you know, you
17:22:42nobody should be able to do this off top
17:22:44of their head in an hour. Most people
17:22:46won't. Um it takes time. You make
17:22:49mistakes. Um but [clears throat]
17:22:52uh let's get started on the project. Now
17:22:55in this uh in this what we're going to
17:22:59have to do is we going to have to tell
17:23:03beautiful soup and requests where we are
17:23:05actually getting this data from. What
17:23:07website um what is our computer you know
17:23:10some information from our computer. I'm
17:23:12going to again there's going to be a
17:23:14little copying and pasting in here
17:23:15because you don't ever you will never
17:23:16ever ever need to know this. Um but
17:23:19right here we're going to basically
17:23:21connect [clears throat] to the website.
17:23:22So, I'm just going to say connect to
17:23:24website and we're going to say URL is
17:23:27equal to and let's go get our
17:23:30[clears throat] URL.
17:23:32So, we have this right here. So,
17:23:33literally just go up here, do you know
17:23:36uh control A, copy that. Oops, that's
17:23:40the actual project. Get rid of that.
17:23:43Uh, paste it in here. And that is our
17:23:45URL. We will use that in just a second.
17:23:48Uh, what am I doing?
17:23:52me just get some room here. And then we
17:23:55what we're going to need is something
17:23:57called headers. Now again, you will
17:24:00never ever ever need to know this. So
17:24:02I'm just going to say headers. Um what
17:24:04I'm going to do is I'm going to copy
17:24:05this. I'm going to show you how to get
17:24:06this really quick. Um but is something
17:24:09called headers. So
17:24:14uh let me show you how to use how to get
17:24:16this
17:24:18and why you don't need to know any of
17:24:20this. So, what this headers is is this
17:24:22something called a user agent. You need
17:24:24to do this for your computer. Um, and
17:24:27you can do that by going to this link
17:24:29right here. So, I'm going to put this
17:24:31link in the description so that you can
17:24:33go and get that. And there's something
17:24:34right here called the user agent. So,
17:24:37all you have to do is copy this just
17:24:40like this. Do copy. I'm going to go back
17:24:43here and I'll show you that it's I'm
17:24:45going to copy it in. Um, it'll be the
17:24:47exact same. So, there you go.
17:24:50It's the exact same um [clears throat]
17:24:53all of this extra stuff except encoding
17:24:57except um this HTML stuff connection
17:25:01close all the you don't need to know any
17:25:02of it. I promise you'll never come in
17:25:04handy ever in life.
17:25:06Actually there will be one person who
17:25:08that becomes in handy for and then
17:25:09they'll message me. Um but we are now
17:25:13connecting um using our computer using
17:25:16this URL and then what we want to write
17:25:19is we want to write page we're going to
17:25:22say equals and this is where we start
17:25:23using uh these libraries. So we're going
17:25:25to use requests.get
17:25:28and we are going to pull in that URL and
17:25:31we're just going to say headers is equal
17:25:34to our headers right here. So, uh, we
17:25:38have this, and this is where we're going
17:25:40to actually start
17:25:43getting the data, bringing in the data.
17:25:46Um, and it's not going to look like that
17:25:47at first, but I'll try to print some
17:25:49stuff out as we go along the way so that
17:25:52you can kind of see what it looks like
17:25:53and how we're going to kind of make it
17:25:55more useful because it comes in very
17:25:57dirty uh, when we first get it. And some
17:26:00of the things I'm going to show you will
17:26:01just help clean that up. Um, and before
17:26:04we actually go any any further, I don't
17:26:06want my head to be here for the entire
17:26:07time. I'm going to get rid of myself so
17:26:08you can just see the page. Uh, I just
17:26:12it's less distracting. Uh, I hate when I
17:26:15feel like people are always watching me.
17:26:16So, I want people to just focus on the
17:26:18code. Uh, so I will see you in a little
17:26:21bit. Let's get back into it. All right.
17:26:23So, what we are going to do is we are
17:26:24actually going to start using the
17:26:26beautiful soup library. All right. So,
17:26:28we are going to say soup one is equal
17:26:31to, and this is where we actually start
17:26:33bringing beautiful soup. And you guessed
17:26:34it, you're going to say beautiful soup.
17:26:36And then in parenthesis, we're going to
17:26:38do page.content.
17:26:40Um, and again, these aren't really
17:26:43things that you need to remember or need
17:26:45to memorize. We're just pulling in the
17:26:47content from the page. That's really all
17:26:49we're doing right now. And it comes in
17:26:51as HTML. So, we're going to do
17:26:52HTML.parser.
17:26:55Uh, and let's see if I can print out.
17:26:57Uh, actually, let me just do soup one. I
17:27:00don't like I don't like doing uppercaps
17:27:01on stuff.
17:27:03Let's see if anything prints out real
17:27:05quick. So, we are literally pulling in
17:27:09all of the HTML.
17:27:11Um, and let me go show you really quick
17:27:14because we're going to get to this in a
17:27:15second anyways. Um, if you come here,
17:27:19this is this is a static page basically
17:27:23written in HTML. Um, if you have never
17:27:25seen HTML before, um, you know,
17:27:29actually a lot of this is, you know,
17:27:32just stuff that most people will never
17:27:34use. Uh, it's just good to know. Some of
17:27:37this stuff is good to know. So, as you
17:27:38see, I'm scrolling on this right side.
17:27:39By the way, I did rightclick and inspect
17:27:42or control shift I, whichever one works
17:27:45better for you. But, as I'm scrolling
17:27:47over this, you should see it kind of
17:27:48highlighting different areas. Um, it's
17:27:51hard to kind of get what you want. Let's
17:27:52say we want this title. Um, what I can
17:27:55do is I can click select element, go
17:27:58right here. Um, and then we can select
17:28:00like a t the the the header or the title
17:28:02of the the page. Now, I just want to
17:28:05show you though of what we're pulling
17:28:07in. So, we're pulling in this doc type
17:28:09HTML. All of this is coming in. So,
17:28:12that's what this is right here. This doc
17:28:14type HTML and we're pulling every single
17:28:17thing in. That is what we're doing right
17:28:19now. Uh so let's get or let's go down a
17:28:23little bit. Let's do soup two. We're
17:28:25just going to do a very uh you know uh
17:28:27an upgrade to soup one basically. We'll
17:28:30do beautiful soup again.
17:28:34And then we're going to do uh soup one.
17:28:37So we're pulling in that content again.
17:28:40So that's soup one. And we're going to
17:28:42do
17:28:44pritify. Uh, if you don't know what that
17:28:46is, it is common in a lot of different
17:28:49languages and a lot of different stuff.
17:28:51Um, it just makes things look better. It
17:28:54that's really all it is.
17:28:57Uh, I don't know why I'm using double
17:28:59quotes.
17:29:01I don't know why I can. You can do
17:29:03single ones if you want. Um, and now
17:29:04let's do beautiful soup 2. And it should
17:29:07just be a it should be better formatted.
17:29:10Um, and let's see if that's true. And it
17:29:13is. So, before if you did, if you can
17:29:15tell, it was it didn't have basically
17:29:16any formatting. It has a little bit of
17:29:17formatting now. Um, it'll help in a
17:29:20second. Um, and you'll see that. But
17:29:24now, [clears throat] what we want to do
17:29:25is go back and we want to actually get
17:29:27the data that we want. Now, you can get
17:29:29any data you want. I'm going to show you
17:29:32simple things, really, really easy. Um,
17:29:35in my in my in in my opinion, it gets
17:29:38more difficult the more complicated
17:29:40stuff you start pulling. Um and and
17:29:42you'll understand that as we go into it.
17:29:45So what I'm going to do is I'm going to
17:29:46select this and I'm going to select this
17:29:49um the title. I want that. And so if you
17:29:52do span ID, it's equal to product uh
17:29:56title. So we need to remember that. Um
17:29:58class, we don't need to know class, I
17:30:01believe.
17:30:03Uh we're going to be [clears throat]
17:30:03using that ID, this um ID equals product
17:30:07title. So that's what we're going to be
17:30:08using. um class will come in in the next
17:30:11video when we start looking at these uh
17:30:13but not in this one. So let's remember
17:30:16ID equals product title. So let's go
17:30:18back over here. So we have this soup 2.
17:30:21It's basically all of that HTML in it
17:30:24right down here. That that is what we're
17:30:26pulling in. So we need to kind of
17:30:27specify what we actually want. So let's
17:30:30say title. That's what we're going to be
17:30:31getting. Um and we're going to do soup
17:30:342. So using taking all that content and
17:30:37we're do find and we're going to do open
17:30:40parenthesis and we're going to say we
17:30:41want to find that ID where it's equal to
17:30:45product title
17:30:48and then we're going to do dot get
17:30:52text and then we're going to do open
17:30:55parenthesis. So now let's um
17:30:58[clears throat] let's print the title
17:31:02and see what we get. All right. So that
17:31:04is exactly what we're looking for. It's
17:31:06funny got data MIS um t-shirt. That that
17:31:12is what we're trying to pull in. So
17:31:13that's perfect. That's exactly what we
17:31:15want. We don't uh let me let me me just
17:31:18do this. Save me some time later on. We
17:31:21don't only want the title. We are also
17:31:22going to be pulling in the price. So if
17:31:25[clears throat] you can guess uh we'll
17:31:27be doing some uh a data set on the
17:31:31actual pricing.
17:31:33Um, and so let's go back here. We're
17:31:36going to again use this right here and
17:31:38we're going to go to this price.
17:31:41And it says again, we're going to look
17:31:43at this ID. The ID equals price block
17:31:46our price. So fairly easy. You can copy
17:31:49this. I'm just going to write it out.
17:31:51Um, we're going to say price is equal to
17:31:55soup 2.find.
17:31:58And then it's going to be again ID is
17:32:01equal to and then it's going to be price
17:32:03block_rric.
17:32:06Did I spell that right? Oops.
17:32:10Excuse me. [clears throat] There we go.
17:32:12And the exact same thing.get
17:32:15text
17:32:17parenthesis.
17:32:18Uh, and there's a get text, there's a
17:32:20get all or get all text. Um, so you know
17:32:24that get text is a specific thing that
17:32:26we are using. you we might use a
17:32:29different one later on. Um but that that
17:32:32is what we have. So now let's
17:32:35let's print the title and print what why
17:32:38do I have all this
17:32:41too much uh too much space? So let's t
17:32:44print the title and print the price. Now
17:32:47let's see what we get. Okay, so we have
17:32:49[clears throat] our title and we have
17:32:51our price. I mean, you know, I don't
17:32:53know what all this white space is over
17:32:54here. Um, but it looks like there's a
17:32:57lot of white space over here. We'll have
17:32:59to get rid of that uh in a little bit as
17:33:01we clean it up a little bit. You can, if
17:33:05you want, do things like um you can get,
17:33:10and this is up to you. I'm not going to
17:33:11do this right now, but I'm just going to
17:33:12show you how to do it. you can get this
17:33:14where you're pulling in the ratings um
17:33:17which is you know if you want to look at
17:33:20like how the ratings over time or or
17:33:22what ratings are for specific products
17:33:24that could be really useful. Um you can
17:33:27pull basically anything you can go down
17:33:29the product details and look at
17:33:30dimensions uh anything you want on this
17:33:33page. It is static so you can go in here
17:33:37and pull anything. It's you just have to
17:33:39pull it from the HTML know where you're
17:33:40looking pull it in. Um, and now when we
17:33:43go back here, excuse me. I'm going to
17:33:45show you now kind of how to use this,
17:33:47right? Because we have this, but how are
17:33:50we going to use it? Um, that's kind of
17:33:52the important part, I think. First thing
17:33:54we need to do is clean this up a little
17:33:56bit because it it just is, you know, if
17:34:01we try to use this, it wouldn't be super
17:34:03useful because it'd be it's just a
17:34:05little bit dirty. It's not super clean.
17:34:08Um, so what we want to do is let's start
17:34:11with the price. Why not? Uh, we're going
17:34:14to say price.
17:34:16Um, and that's just going to take uh
17:34:19basically the the junk off of either
17:34:22side. And so let's run that real quick.
17:34:25So this is what we have. But what we can
17:34:27also do is I don't want that dollar
17:34:29sign. I just want the numeric value. Um
17:34:32later on we are going to be putting this
17:34:33and we're going to be um creating a
17:34:35process to put this into an Excel file.
17:34:38Again, we're trying to create a data
17:34:40set. I don't want you to have to copy
17:34:41and paste stuff. This is all going to be
17:34:43automated basically to input this data
17:34:46into an Excel file for you or a CSV file
17:34:48for you. So um you know, think about
17:34:51making it useful in a CSV or in an Excel
17:34:54later on. So what we can do is do a
17:34:57bracket and we're going to do one and
17:35:00then everything after that. So basically
17:35:01it's just going to take everything from
17:35:03the first position onward. Uh so let's
17:35:06run that and there we go. So let's just
17:35:09say price is equal to price.
17:35:13Um and pull uh just do everything after
17:35:16that first um that first not value. What
17:35:20am I saying? What's the word for that? I
17:35:22can't remember the word. The first
17:35:23space. That's not the right word, but
17:35:25all right, let's do the title. Um, this
17:35:27is basically going to be the exact same
17:35:29thing. Um, super easy. So, we're just
17:35:31going to do title strip and open
17:35:34parentheses. Um, and we can, you know,
17:35:38if you want to do this exact same thing.
17:35:42So, now we have it. It's a little bit
17:35:43cleaner. So, this is what it originally
17:35:44looked like. And now this is what it
17:35:46looks like. So, you know, nothing super
17:35:51crazy, but you know, something
17:35:52interesting to know. Now we are about to
17:35:56in the very next part what we are going
17:35:58to do let me just add a few of these
17:36:00because makes me feel better. Um what we
17:36:02are about to do is we're going to create
17:36:05our CSV to insert this data into the CSV
17:36:08and then later on what I'm going to do
17:36:10is show you kind of how to um automate
17:36:12this process to pull this data um
17:36:17to create a data set. Right? Just
17:36:18pulling this one time and putting it
17:36:19into a CSV really doesn't do anything.
17:36:21you can just copy and paste that and
17:36:23save yourself a lot of time. Um, what
17:36:25I'm going to show you is is um basically
17:36:28doing it over over time and just having
17:36:31it automated in the background. That is
17:36:33what I'm going to show you. Um, I guess
17:36:34a spoiler, but what we need to do is we
17:36:38need to
17:36:40create uh create the CSV, insert it into
17:36:44the CSV, and then create a process to
17:36:46append more data into that CSV. Um, I'm
17:36:50doing a lot of talking. Let's do some
17:36:51writing. So, what we need to do is we're
17:36:54going to use um I should have done this
17:36:56at the top. Maybe I'll go back and add
17:36:59that later on. We're going to do import
17:37:01CSV. Now, in a CSV, what you want is you
17:37:04want headers and then you want the data,
17:37:06right? So, for our headers, and we're
17:37:08going to call it header. We're going to
17:37:10do um we're going to do a bracket. And
17:37:12let's make the first one a title because
17:37:16that's going to be uh we can call it
17:37:18title. You can call it product, whatever
17:37:21you want. I'm just going to call it
17:37:22because I've been using title, I'm going
17:37:23to call it title. And then we'll also
17:37:25have
17:37:27price.
17:37:29Now, we need our data. So, I'm going to
17:37:31say data is equal to. Now, this is
17:37:33important. Um, right now,
17:37:35[clears throat] how our data is, and I
17:37:37can do this right here. We're going to
17:37:38do type um title or no, let's do type
17:37:42price.
17:37:44So, these are strings. And that's
17:37:46important to know. Um, again, I don't
17:37:49want to get too much into, you know,
17:37:51dictionaries and arrays and lists and
17:37:53and strings and all these things, but
17:37:55this is a string and you can't put
17:37:57[clears throat] that right now. It's not
17:37:59super usable. What we're going to do is
17:38:01make this a list. Um, and so I'm doing
17:38:05an open bracket and I'm going to say our
17:38:07data is title,
17:38:11price. Oops, price. Now, oops. If I do
17:38:18type oops of data, I'll just run that.
17:38:22It's a list now. Um, and this is
17:38:24important because you can run into a lot
17:38:27of issues with this stuff. It's really
17:38:29important to remember what what type um
17:38:34how do I say this? Uh, how your data is.
17:38:37Is it a list? Is it an array? Is it a
17:38:39dictionary? Um, you know, what is it?
17:38:42These things are important. they do play
17:38:44a big impact especially with this type
17:38:46of stuff. So just want to show you that
17:38:47really quick. But what we are now going
17:38:50to do is create a CSV. Um you're create
17:38:54an Excel. I I call an Excel CSV, you
17:38:57know, whatever you want to call it. So
17:38:59what we are going to do is we're going
17:39:01to say with and we're going to say open.
17:39:04And now we're going to name our file.
17:39:06You can name this whatever you want. I'm
17:39:08going to call it uh
17:39:12um Amazon
17:39:16Web Scraper
17:39:18data set. That's real long. Uh CSV. And
17:39:22we're going to do underscore W and that
17:39:24means write.
17:39:27Um oh, whoops. That's not right. Just
17:39:30like I was wondering why that was uh in
17:39:32black. Uh so we're going to do W, which
17:39:34means write. Um, and then we're going to
17:39:36do new line. And if you don't know what
17:39:39new line is, uh, all that does is when
17:39:42we insert the data, it doesn't have a a
17:39:45space in between each CSV. And then we
17:39:48are going to do encoding
17:39:51is equal to oops is equal to UTF8.
17:39:58And that is it. And we'll just say as
17:40:00uh, let's do f. So some of
17:40:03[clears throat] that stuff you don't
17:40:04need to know. Some of it's useful. This
17:40:06W definitely needs to know. This new
17:40:08line is is good to know. And um I'll
17:40:10take it I might take it out just to show
17:40:12you what it actually does because it's
17:40:13annoying if you don't have it. I
17:40:15promise. Um but you know that that new
17:40:18line is important. This encoding, you
17:40:20know, good to know. I think that's by
17:40:22default is is it's like that. Uh
17:40:24anyways, what we're going to do now is
17:40:26we're going to uh it's something within
17:40:28the CSV
17:40:30within the CSV um library. So, we're
17:40:33going to do something called CSV writer
17:40:37and oops CSV.riter
17:40:41and we're going to do open parenthesis
17:40:43and that is that and we'll just call
17:40:45that writer
17:40:48and then we'll do and this is where we
17:40:51need to [clears throat] actually create
17:40:52the header. So uh we're going to do
17:40:54writer is dot sorry writer.right
17:40:59row uh and this is just for the initial
17:41:05um the initial
17:41:07import or or or um not import the
17:41:11initial insertion of the data into the
17:41:13CSV. This is what's important. The next
17:41:15one that we're going to write is for
17:41:17when we're actually appending the data,
17:41:18which is going to be a little bit
17:41:19different. But anyways, we're going to
17:41:21do write
17:41:21>> [clears throat]
17:41:21>> row open parenthesis. And this is where
17:41:24that header is going to go. So, we're
17:41:26going to the these headers are going to
17:41:28be the title and the price.
17:41:31And then for our last one, we're going
17:41:32to actually write the data, which is
17:41:34this data right here. And we're going to
17:41:36say writer
17:41:38write row. And we're going to do data.
17:41:42So this one we are creating the CSV
17:41:46and then we are inserting the header and
17:41:49inserting the data. So super easy. Um
17:41:54yeah I think that's fairly
17:41:55straightforward right now. Let's do this
17:41:59and let's see what happens. So I just
17:42:02ran it. Um let's go over here in here
17:42:06somewhere. Amazon web scraper data set.
17:42:10Let's open that up.
17:42:13And there we go. Oh jeez.
17:42:16This isn't good. Can't verify my um
17:42:21my subscription. Uh why does it say
17:42:23$6.99? Uh I'm going to go back and look,
17:42:26but I think I know the issue. Um but
17:42:30this is exactly what we want. Now, of
17:42:32course, we want more data and maybe a
17:42:34little bit more useful data. Um and I'll
17:42:36show you how to get that in just a
17:42:37second, but we just created that out of
17:42:40thin air. Uh that was not I didn't have
17:42:42that saved before. So we have this data
17:42:44set and the issue was is that I ran this
17:42:48multiple times. So now it's 699. If I do
17:42:51it again it's 99. Uh and if I did it
17:42:54again it's you got it gets rid of
17:42:55everything. So I'm just going to run
17:42:56this again. Run this again.
17:43:01Uh now everything's back to normal.
17:43:04Okay. So now if we run this, it's going
17:43:07to overwrite this Amazon Web Scraper
17:43:10data set.csv
17:43:11and it will put the data in properly. So
17:43:15there we go. Oh jeez, guys. This is
17:43:18embarrassing.
17:43:20I'm embarrassed.
17:43:22No, I don't want this. Okay, perfect.
17:43:26Um, guys, I if you can't tell, I'm in
17:43:30need of some um I'm in need of I'm in
17:43:33need of some help here, but [laughter]
17:43:36I'm just kidding. I'm I'm doing fine. Uh
17:43:38I just I don't know why that uh I don't
17:43:41have my uh subscription activated. It's
17:43:43not going to matter for this video, I
17:43:45guess, but that's really random. Um so,
17:43:47we got what we need. That's perfect.
17:43:50Now what we want to do after this um I
17:43:54guess actually what is important is some
17:43:56more useful data. Something that I like
17:44:00to do a lot when I do this type of this
17:44:02type of stuff is I like to have some
17:44:04type of date stamp um or some type of
17:44:06timestamp to know when I collected this
17:44:09data. It usually comes in handy later
17:44:11on. Um I I have never regretted putting
17:44:14it in there. I'll show you really quick
17:44:16how you can do it. Uh, you're going to
17:44:17do import datetime.
17:44:20Jeez, I hate having to format stuff like
17:44:22that. And what you can do is you can do
17:44:24date. Let me get date time. And you do
17:44:29date today open parenthesis. And that is
17:44:33going to give us this right here. Uh,
17:44:36and so we're just going to do um today,
17:44:39that's what we'll call it, is equal to
17:44:41this.
17:44:42And we'll say print today. And there we
17:44:46go. So that is today's date is the 21st
17:44:49of August in 2021.
17:44:52So today is now um is now this.
17:44:55[clears throat] So actually I'm going to
17:44:57get rid of that. I'm going to put it
17:44:59back up here. I'm going to put it right
17:45:02there. I'm going to run it again. Let's
17:45:05add this right here. We'll do um
17:45:10we'll do we'll call it date
17:45:13and then we'll add today.
17:45:17And we'll just run this again.
17:45:19And what we can do just to check the
17:45:24data without having to open up the data
17:45:26every single time, which is super
17:45:27annoying, is we're going to use pandas.
17:45:29Again, I should have imported this at
17:45:31the top. I'm just kind of um I'm not
17:45:33doing this off the top of my head, but
17:45:35uh I didn't have it 100% planned. So,
17:45:37import pandas and we're just going to
17:45:38say pdread_csv
17:45:42and then we'll read it in. Um, what you
17:45:45can do or what I often do is I go to
17:45:48properties and I go right here
17:45:55and we'll say boom boom backslash
17:46:01this right here. This I am doing off the
17:46:03top of my head. I don't do this often. I
17:46:04think I have this memorized by now. Uh,
17:46:06I I I hope. And then we'll do print. Oh,
17:46:11no. We don't have to do print. We'll
17:46:12just do this. uh what I do our uh let's
17:46:15actually call this um data frame and
17:46:20we'll do print.
17:46:23Let's see what happens. Perfect.
17:46:25[clears throat] Okay. So, what we have
17:46:27now is the new our new header, our new
17:46:30data that we added in there. So, we have
17:46:33our title, we have our price, and we
17:46:36have our date. Now, again, you can
17:46:38customize this whatever you want to add.
17:46:39go back here. Um, you know, find what
17:46:42you want. You know, do you want it to
17:46:44make sure it has a men's option or
17:46:47different colors or you want to pull in
17:46:49this information? Whatever you want. It
17:46:51it really does not matter. Um, just
17:46:53matters that you know, you get what you
17:46:56need for whatever purpose, whatever
17:46:58you're making this for. This is more of
17:46:59an introductory video to how to scrape
17:47:02data from Amazon. Um the next video will
17:47:04probably be a little bit more difficult
17:47:06and in-depth, but this is kind of let's
17:47:08get you guys started. So um we now have
17:47:11this and this is beautiful.
17:47:14Now, something that
17:47:18you want to do when you're scraping data
17:47:21and you're getting um I [clears throat]
17:47:24guess data over time, and that's kind of
17:47:25what we're doing. It's going to be
17:47:27almost like um a price tracker
17:47:29[clears throat] over time is you want to
17:47:32then append data to this. So, we can't
17:47:36only create it. And that's what this
17:47:38does because if I run this a 100 times,
17:47:39it'll only give me this first row. we
17:47:41need to now append data to this. So, um
17:47:45let's
17:47:46let's pull this down here. Um again, I'm
17:47:50I'm not I haven't added a bunch of
17:47:52notes. I'm going to say now we are
17:47:55appending data to this CSV. I haven't
17:47:58added a ton of notes. I'll try to go
17:47:59back maybe afterwards and add some notes
17:48:01for people who like to read notes. Um
17:48:05[clears throat]
17:48:06so, what we are now going to do is we're
17:48:07going to change this W to an A+. Now,
17:48:10this is going to be how we append the
17:48:13data. Um, and we no longer need the
17:48:15header. So, we don't aren't going to do
17:48:17the header anymore. And there we go. So,
17:48:20now instead of
17:48:22excuse me, so now instead of creating
17:48:24that header again, creating that first
17:48:26row of data again, we are ignoring the
17:48:30data and we're now going to the next
17:48:31nearest free row and appending data,
17:48:35which means to add on data to that. Um,
17:48:39and so if I run this, which I'm not
17:48:40going to right now, oh, I mean, why not?
17:48:43I can I can run it. Um, and then we can
17:48:45read this in. So now there there's our
17:48:48data. I'll run it a few more times.
17:48:51I ran it like three or four more times.
17:48:53I I run that in. And there we go. Now
17:48:55it's all the exact same data. Super um
17:48:57boring, but very very uh, you know, good
17:49:02to have. Now, we don't want to have to
17:49:04come in here and run this every day.
17:49:06Let's say we're going to do this daily.
17:49:08Um, we don't want to have to come and
17:49:09write run this every single day, right?
17:49:11We want a way where it does it while we
17:49:14sleep. It does it in the background of
17:49:16our laptop. Um, and is easy to do,
17:49:18right? I don't want to come in here
17:49:21every single morning with set an alarm
17:49:22on my phone every single morning. Come
17:49:24in here. I want to automate this.
17:49:27[clears throat] So, uh, how are we going
17:49:29to do that? Give me one second. Uh, if
17:49:32you didn't know, I have three kids and
17:49:34one of them is waking up. I will be
17:49:35right back. All right. I think he is
17:49:38asleep. Um, at least let's hope he's
17:49:40asleep. So, now what we're
17:49:42[clears throat] going to do is we are
17:49:43going to
17:49:45put this all
17:49:48into
17:49:50uh this check
17:49:53price.
17:49:55[clears throat] Now, you may never have
17:49:57used Oh jeez, what are these things
17:49:59called? Oh my gosh. Super
17:50:03used all the time. you'll know what I
17:50:06what it is. Uh
17:50:09not a function. I don't even remember
17:50:11what it's called. Maybe there's a
17:50:13function. Um I can't think. I'm having
17:50:15like a writer's block or whatever that
17:50:17is. We're going to put it all in here
17:50:19and then we're going to be able to use
17:50:20this price check later. Um because we
17:50:22want to be able to automate this. So,
17:50:24let's go back all the way up here.
17:50:27We are going to use this. So, let's copy
17:50:30all of that in
17:50:34and oh jeez, I hate this.
17:50:41All right. Everything just like that.
17:50:43Um, so this pulls in our data.
17:50:46Pulls in uh or or yeah, pulls in all of
17:50:49our data down to the title and the
17:50:51price. We want to
17:50:54make it look right.
17:50:58So, we're going to put it right here.
17:51:01So, now we have it formatted properly.
17:51:04Um, we want to add our date time.
17:51:11Just like that. I don't know if there's
17:51:14a better I'm sure there's a better way
17:51:15to do this.
17:51:17Um, then we need
17:51:21this right here.
17:51:27And just like that. Like that. So now we
17:51:29have our header and our data. And then
17:51:31we want to pull this in right here.
17:51:36Boom. Boom. Boom. Okay.
17:51:40[clears throat] So everything that we
17:51:42just wrote out, we are now putting into
17:51:45this check price. Uh you can call it
17:51:48whatever you want. Doesn't matter. But
17:51:51let's run that. See if we get any
17:51:52errors. We don't. So this is now good to
17:51:56go. Basically
17:51:58um what we are going to use this for um
17:52:02and what this is going to do is we are
17:52:04going to put this on a timer. Um you
17:52:06know have you ever wanted to like check
17:52:09something once a day, once every 10
17:52:12seconds, once a minute, whatever you
17:52:14want and you don't want to have to
17:52:15actually pull up your phone and look at
17:52:17it. This is how we are going to do that.
17:52:19So, we had something called, let's see,
17:52:23time. This this library time right here.
17:52:25That's what we're going to use right
17:52:26now. So, we're going to say while oops,
17:52:31while true
17:52:34and go like [snorts] this, do a colon.
17:52:37We're going to say check price. That's
17:52:40what we just wrote out. And we're going
17:52:43to do time.
17:52:45Now, this is completely up to you how
17:52:49much time you want to put in here. For
17:52:51the purposes of demonstration, I'm going
17:52:53to put 5 seconds, which means every 5
17:52:57seconds, it is going to run through this
17:52:59entire process. And so, let's run this
17:53:02really quick. And I'm going to run it
17:53:04for let's say 30 seconds. And then I'm
17:53:07going to
17:53:09pull this in right here.
17:53:13So, we just looked at it earlier. We had
17:53:15four um well, five [clears throat] rows
17:53:19of data, right? What we are going to do
17:53:22is in just a second I'm going to stop
17:53:24this, you know, maybe after 30 seconds
17:53:25or so and we're going to see how much
17:53:27data is in there. Uh and let's stop it
17:53:30right now. It's been going far enough.
17:53:32Um and now let's run it. So, now we have
17:53:35five, six, seven, eight. So, I guess I
17:53:36ran for 20 seconds.
17:53:38We can,
17:53:40that was for demonstration purposes.
17:53:42I've never do any some anything ever
17:53:44every 5 seconds. Um, unless it was like
17:53:45Black Friday on Amazon. [clears throat]
17:53:48We can put this as
17:53:51long or as short as you want. You can
17:53:53run it every second if you want. Um,
17:53:55that doesn't make sense to me, but you
17:53:57can. What we can do is do a little bit
17:54:00of math. Uh, and I don't know this off
17:54:02the top of my head, so I'm going to uh
17:54:04do the math with you live. Pretty
17:54:07exciting stuff. Got the calculator out.
17:54:10So, there are 60 seconds in a minute.
17:54:14And this goes by seconds, by the way.
17:54:16And you could do, you know, you can do
17:54:19some um some string up here of
17:54:23calculating this, but I'm just going to
17:54:24put in the number because it's easier.
17:54:27Uh maybe not easier. I'm just going to
17:54:28do it. There's 60 seconds
17:54:30[clears throat] um in a minute. There
17:54:33are 60 seconds or 60 minutes in an hour.
17:54:36So that's one hour. Uh, and we can do 24
17:54:39hours in a day. So that that's 86,
17:54:44400, I believe. Did I read that right?
17:54:47Oops. Did I read that right?
17:54:50Yes. So this now, if I ran this, and I'm
17:54:54going to this is going to check the
17:54:57price every single day. And this is the
17:54:59entire point of this um of of this
17:55:04project. Not the entire point, but this
17:55:06is a big part of this project is we want
17:55:08to create our own data set. Now,
17:55:10something that I personally really love
17:55:13is a data set that has,
17:55:16you know, that I can do some type of
17:55:18time series with. Now, this is not
17:55:21exciting. It's probably not super
17:55:23exciting for this, right? But you get
17:55:27the idea that if this price were to
17:55:30change, we would then see that reflected
17:55:33in the data at some point.
17:55:35You can do this on any item you could
17:55:38ever imagine on Amazon. It's the exact
17:55:40same process and some items change
17:55:43often. This t-shirt will most likely
17:55:46never change. Um, and so, you know,
17:55:48again, this is for demonstration
17:55:50purposes. The code itself will be nice
17:55:52to put in a project, although the data
17:55:54set that you get from this probably
17:55:56won't be the best, I would imagine.
17:55:59But notice that this is running. Um, I
17:56:01can then minimize this and this can run
17:56:04on my computer basically as long as my
17:56:07computer uh is is working. Um, one thing
17:56:12I will say before I go on to some more
17:56:15stuff, one thing that I will say is that
17:56:18I personally when I did this for a when
17:56:21I um created this, I did something
17:56:25similar and I put this in Visual Studio
17:56:27Code um and I didn't put it in Jupyter
17:56:31Notebooks. That's a personal preference.
17:56:34I would look into that if that is
17:56:35something that you want. Um, I think
17:56:37Visual Studio Code is a little bit
17:56:39easier for automating these types of
17:56:41tasks. Um, but for illustrator purposes
17:56:44and for demonstration purposes, you
17:56:46cannot beat Jupyter Notebooks. That's
17:56:48why I did it. So, with all that being
17:56:50said, that is basically the end of the
17:56:52project. Now, um, I'm not going to stop
17:56:54this and read it again, but you get the
17:56:57point. Um, we now have um a data set
17:57:03that Oh, jeez. All this again. That now
17:57:06has um data. I'm getting out of here. Oh
17:57:09jeez, it's hounding me. Let me get out
17:57:10of here. Oh no.
17:57:13I This is embarrassing, guys. I'm
17:57:15embarrassed. We now have a
17:57:17[clears throat] CSV file with data in
17:57:19it. Now, you run this in the background
17:57:20of your computer. You can do that. I
17:57:22have done it. I've ran it for weeks. I
17:57:25have ran it for months. Um, if you
17:57:27restart your computer, just come back in
17:57:29here and restart running this process.
17:57:31Um, it's the same for any automated
17:57:34process unless you start using some
17:57:36online um, automation service which will
17:57:39run it regardless of your computer. They
17:57:41do it, you know, either in the cloud or
17:57:44on some um, server. So, you know that
17:57:47this is a really good option. Again, if
17:57:49if you restart your computer or
17:57:51something happens, you lose connection,
17:57:52just come in here, run this through the
17:57:54script again. um except for the one
17:57:57where it deletes all your data. Don't
17:57:59run that one again. Only run that one
17:58:01time. [clears throat]
17:58:02Um and then you will and in fact what I
17:58:06would do is then um I would just comment
17:58:09this out, right? I'd come in here and I
17:58:12would just comment this out
17:58:14so that anytime I come back in here, I
17:58:16would never accidentally delete all my
17:58:18data.
17:58:19But that is what this project does. Now,
17:58:22something really interesting, something
17:58:24that I have done in the past that I
17:58:25thought was really cool, really useful.
17:58:29I actually did it for um I actually did
17:58:32it for some watches that I was watching,
17:58:36especially on Black Friday. It's when I
17:58:38used it. I was interested in a price
17:58:42drop or a specific price change. And
17:58:46what I did was is I said, and I don't
17:58:50know
17:58:52So, what I basically did was is I said
17:58:56if the price is lower than let's say
17:59:00let's say we wanted to drop below $14,
17:59:04it would then send an email. Um, and I'm
17:59:07going to show you the script that I
17:59:09used. It still works. Um, and if this is
17:59:12something that you are interested in,
17:59:14this could be a completely different
17:59:15project. I just think it's interesting
17:59:17and I wanted to show it to you. Although
17:59:19I wouldn't say this this is part of the
17:59:21um final project. Let me just come in
17:59:24here
17:59:26and we're going to create this. Super
17:59:31simple. Um and that's super simple.
17:59:34We're sending a mail. We're connecting
17:59:35to a server. We we're using Gmail. We're
17:59:38logging into our account. That is my
17:59:40email. You will not get my password.
17:59:42We're creating the subject, the body. um
17:59:45we we configure or or just kind of
17:59:47create this message and then we send a
17:59:49mail. So then I have this define uh or
17:59:53this send mail. I I'm blanking on what
17:59:56this is called. I'm going to call it a
17:59:57function, but that's probably not right.
17:59:59So if that price drops below a certain
18:00:02point, it'll send me an email. Um I have
18:00:05used this and I used it and was able to
18:00:07buy a watch that was like, you know,
18:00:09let's say 140 bucks for like 90 bucks um
18:00:12on a Black Friday sale. I was really
18:00:13really happy about that. So, this can be
18:00:16used in that way as well. Um, not
18:00:18something you have to write into your
18:00:19project, just something I'm going to
18:00:20include down here if you want to try it.
18:00:24I think it's super interesting,
18:00:25something really fun. Um, really fun to
18:00:29mess around with. I enjoyed this. So,
18:00:32with that being said, uh, this is this
18:00:36is the project. Um I in the next one and
18:00:39I promise you this one is probably going
18:00:41to get a lot more
18:00:43difficult. If you thought this one was
18:00:45easy, which I hope maybe I hope you do,
18:00:46then that means you're, you know, pretty
18:00:48good at Python, you know, in the next
18:00:51the next um web scraping project. And I
18:00:54hope to do many of these. I might do um
18:00:56even all the ones that I put in that
18:00:58poll, but I started with the one that
18:00:59was the most popular.
18:01:01Um you know, if you were able to get
18:01:03through this, I think that that is
18:01:05fantastic. I think this is a solid
18:01:08project to create um a data set and so
18:01:12use this how you will you can copy my
18:01:14code exactly I don't have a problem with
18:01:16that again I don't think this is
18:01:18beginner there are some a little bit
18:01:20more um advanced things and I not even
18:01:22advanced just like intermediate level
18:01:24things um that you kind of learn as you
18:01:26get into it and so um I hope that this
18:01:29was instructional I hope I explained it
18:01:31you know well um and I hope that this is
18:01:34useful Again, you know, when you
18:01:36actually use this, you'll have 22, 23,
18:01:4124, 25. You know, you'll see a price
18:01:44change, a price change, a price change,
18:01:46a price change. Go use a product or go
18:01:50to something that you are interested in
18:01:51or you know, fluctuates often. Um, and
18:01:54there are plenty of those on Amazon. I
18:01:57promise you, there's some that literally
18:01:58change almost every other day, like down
18:02:00a dollar, up a dollar. Um, and then
18:02:03Black Friday just goes crazy um, with
18:02:06these price changes. So, use this as you
18:02:08will. I hope that this was
18:02:09instructional. I hope that it's useful.
18:02:12I think I said that before is, you know,
18:02:14I'm doing this because I think it's
18:02:15really interesting. It's really useful.
18:02:18Um, this to me again was a good
18:02:22introduction,
18:02:23a really good introduction to web
18:02:25scraping because in this next one it
18:02:27gets quite a bit more difficult. Um, I
18:02:30would say on a scale of like difficulty,
18:02:33this is like maybe a four and it'll
18:02:35probably jump up to like a seven on this
18:02:37next one. Um, just just much more
18:02:41um technical or or coding heavy. So, um,
18:02:45you know, look forward to that if that's
18:02:47something that you look forward to. With
18:02:49that being said, I'm going to go back
18:02:50over here for my sendoff. With that
18:02:53being said, I hope this was helpful. I
18:02:56hope that you learned something. Um,
18:02:59don't get mad at me if it was too easy.
18:03:01Don't get mad if you was me if it was
18:03:02too hard. Uh, I'm doing my best over
18:03:04here. So, I appreciate your patience.
18:03:06Thank you so much for watching. I really
18:03:08appreciate it. If you like this video,
18:03:11be sure to like and subscribe below, and
18:03:13I will see you in the next video.
18:03:26What's going on everybody? Welcome back
18:03:28to another video. Today we're going to
18:03:29be creating a script to automatically
18:03:31take data from a crypto API.
18:03:39Now, this project stems from an earlier
18:03:41video that I did where I walked through
18:03:42what an API was and how you can use it.
18:03:44And in that video, I showed you how to
18:03:46use Coin Market Cap's API so you could
18:03:48start pulling in their crypto data. And
18:03:49in this video, we're going to take it
18:03:50one step further and automate that
18:03:52process. Then, we're going to do a
18:03:53little bit of transformation with the
18:03:54data. I'm going to show you some cool
18:03:56stuff on how you can use it and maybe
18:03:58we'll do a little bit of visualization
18:03:59at the end, but that is not the main
18:04:01point of this video. It's mostly around
18:04:03the automation piece and a little bit of
18:04:05the data cleaning piece as well. Now,
18:04:07fair warning, this is not a beginner's
18:04:08level project. It's probably more like
18:04:10an intermediate project. And it's not
18:04:12even a complete project per se because
18:04:14we're not doing all the data cleaning.
18:04:16We're not doing all the visualizations.
18:04:18But if you follow along, we're going to
18:04:20cover a lot of different things and
18:04:22you're really going to set yourself up
18:04:23to be able to do just about anything you
18:04:25want with this data or different APIs
18:04:27that you pull from. So with that being
18:04:28said, let's jump on my screen and get
18:04:30started with the project. All right, so
18:04:31this is where we stopped in our last
18:04:33video. So if you haven't watched it, now
18:04:35is the time to go back and do that. I'll
18:04:37have a link in the description. Also,
18:04:39all the code that we're going to be
18:04:40looking at today and working through is
18:04:42going to be in a GitHub repo below. So,
18:04:45you can go and get all the code and have
18:04:47it completely finished and just follow
18:04:48along or you can code it from scratch
18:04:51along with me. I do recommend writing it
18:04:53from scratch if you can because I think
18:04:55you'll learn more and you'll make
18:04:56mistakes and you'll learn from that as
18:04:58we go through it. But, it is up to you.
18:05:00So, let's get started. And as you can
18:05:03see, uh we have this script right here
18:05:05and I'm starting basically from scratch.
18:05:07I have a completed one up here. Actually
18:05:09going to get rid of those. Um, and what
18:05:12we're going to do is we're going to
18:05:13start from exactly where we started in
18:05:15our last one. I'm going to run the
18:05:16script. Um, this is going to pull from
18:05:19our API
18:05:21and we're going to look at the
18:05:23dictionary setter option and do our JSON
18:05:26normalize. So, this is where we
18:05:27literally left off from the from the
18:05:30last video. So, we have all of this data
18:05:34and
18:05:36what we want to do with it is we want to
18:05:38kind of automate that process, right?
18:05:40Because we don't want to have to come in
18:05:41here, run this, and you know, put into a
18:05:45CSV manually or something like that. We
18:05:47want to automate this data collection
18:05:49process so that we can just have the
18:05:51data ready for us to use. Um, and it all
18:05:53be ready to go. So, we're going to be
18:05:56using this script. Um but you know we we
18:05:59might want to add a little bit more to
18:06:01it before we do that. Uh the first thing
18:06:03that I want to do before um before
18:06:07anything is something that I like to do
18:06:10when I'm creating these automation
18:06:11scripts is I I like to add a timestamp.
18:06:14Uh and the reason for that is because I
18:06:17want to know when I ran or when each of
18:06:20those um loops you can say runs through
18:06:23an and does those automated runs, right?
18:06:25So, if I do it every day, I want to know
18:06:27what time of day I ran it, making sure
18:06:29each run ran successfully. And so, all
18:06:33I'm going to do is I'm going to add a
18:06:34new column at the end, just call it
18:06:36timestamp. So, let's go right up here
18:06:40and we're going to say PD dot and
18:06:43there's something called to datetime.
18:06:45So, we're going to do two
18:06:48date
18:06:50time and then we're going to do now. And
18:06:54what this is literally going to do is
18:06:56take the the date uh the the time stamp
18:06:59of right now when it's running and it's
18:07:03going to show that. Now we need to of
18:07:05course add a new uh a new column for
18:07:08that. So all we're going to do is we're
18:07:09going to say data frame. Whoops. Say
18:07:12dataf frame. And let me see real quick.
18:07:16So we just have the data.
18:07:19We need to add we need to create this
18:07:20data frame right here. So dataf frame
18:07:22equals and then this JSON normalized and
18:07:25we're going to say dataf frame and then
18:07:26we're going to do a bracket and we're
18:07:28going to say timestamp and we'll do well
18:07:31all these lowercase. We're going to keep
18:07:34with the the lowercase. We're going to
18:07:35say timestamp
18:07:38and we do that bracket and we'll say
18:07:39equals. So what this going to do is
18:07:41going to first off it's going to create
18:07:42this data or or assign this df as our
18:07:45data frame and then we're going to add
18:07:47this timestamp and add this new column.
18:07:50And so let's run this really quickly
18:07:54and let's go all the way to the right.
18:07:56And this is our time stamp. And this is
18:07:59the time uh that it is right now. This
18:08:01is the day that I'm running it. This is
18:08:03the time that I'm running it. And so
18:08:04this is working properly. Now, if you
18:08:07look really quickly, there is a last
18:08:09updated in here. And this is very close
18:08:13to this time stamp, but it is not the
18:08:15same thing. Um, but if you looked
18:08:17through this data and you really dug
18:08:18into it a little bit, there's this last
18:08:21update is coming from Coin Market Cap's
18:08:24API and this is when the actual um
18:08:28cryptocurrency was updated in their
18:08:29system. And so it is going to be really
18:08:31close, but it's not going to be exact.
18:08:33And so I don't like to rely on built-in
18:08:36ones that, you know, are coming from an
18:08:38API or something. I want to make one
18:08:39myself that's running on the system
18:08:40where I'm creating the automated process
18:08:42just like just something I do. Um, so
18:08:46now we have this original data frame
18:08:49created, right? We h we now have what we
18:08:53need, but what we want to do is to keep
18:08:56adding data to this. Um, we don't want
18:08:58it to just go through um, you know,
18:09:01create these 5,000 rows. We want it to
18:09:04create 5,000 5,000 5,000 over time,
18:09:07whether it's a day, an hour, a week, um,
18:09:10whenever you want to run it. So, um,
18:09:12what I'm actually going to do is I'm
18:09:14going to limit this a lot. I just want
18:09:15to look at the top, let's say, 15. So,
18:09:18we're do that. We're going to run
18:09:19through all this again. So, now I just
18:09:21have top 15. It's going to be um easier
18:09:25to to see and it won't take as much time
18:09:28to run our scripts. Again, you can keep
18:09:30as many as you'd like. If you want a
18:09:32100, 200, all 5,000, you do whatever
18:09:34you'd like. But what we are now going to
18:09:37do is we're going to create a function
18:09:40using this original script. So we again
18:09:42we have this data frame and we are going
18:09:45to create an automated process that is
18:09:48going to or an automate a script to
18:09:49automate this that is going to append
18:09:51data to this data frame right here. So
18:09:53that's kind of you know the big thing
18:09:55that we're trying to accomplish in this
18:09:56project. Um, so let's go up here and
18:10:00we're going to we'll just take from here
18:10:04all the way to here. I'm just going to
18:10:07copy this and
18:10:10going to paste it down here. Now what we
18:10:12need to do is we need to create a
18:10:14function. So we're going to say def
18:10:17and we're going to call this the
18:10:18API_runner.
18:10:20This is going to run our API um whenever
18:10:24we need it to run. Now, when you are
18:10:27formatting um something for a function,
18:10:30it it needs to be formatted properly.
18:10:33And so, what we need to do is you need
18:10:34to go over here. I'm going to hit tap.
18:10:36We're going to do this all the way down.
18:10:37I'm just going to skip forward when it's
18:10:38all the way done. All right. So, now we
18:10:40have this URL. And what we want to add
18:10:43because this is again, this is going to
18:10:44run through kind of this this automated
18:10:47process. We're going to run this um this
18:10:49function there. What we want is to also
18:10:51add this right here. So, we need to take
18:10:53this and we're going to need to add
18:10:56this.
18:10:58We'll just put it down here.
18:11:01Okay.
18:11:04And let's do that. So, what we have so
18:11:07far is really close to what we want our
18:11:11function to be. Um, we have this
18:11:14function that we're going to be running
18:11:16through. It's going to call this
18:11:17function. It's going to call the the
18:11:20API. We're going to use our key. We are
18:11:22going to um you know test it, load it,
18:11:25format it, format it right here. Then
18:11:28we're going to add this timestamp and
18:11:29then we will have this. Now, right now
18:11:32it's just call it's just going to print
18:11:34this data frame basically. But that's
18:11:36not what we want right now. What we want
18:11:38is to actually append this data. So when
18:11:41it gets to here, when it gets to this
18:11:43data that's going to be right um right
18:11:45here, what we want to do now since we
18:11:48already have the original dataf frame
18:11:50set up up top is we now want to say that
18:11:52this is going to be dataf frame 2. And
18:11:55we're going to say it's going to append
18:11:57it to dataf frame 2. And so the original
18:11:59dataf frame, we're going to say dataf
18:12:01frame 2.append
18:12:04and we're going to say df2. All this
18:12:07does is this says this new data that's
18:12:10going to be coming in every time. Let's
18:12:12say it's a loop and it's just looping
18:12:13through pulling the data, pulling the
18:12:15data, pulling the data. We're going to
18:12:17create this data frame. We're going to
18:12:19add add this time stamp like like we
18:12:21want and then we're going to append that
18:12:23to this original data frame. So, as of
18:12:27right now, this looks good. I will we'll
18:12:29run it in a second. I'll create it. So,
18:12:33I just created it.
18:12:35>> [clears throat]
18:12:35>> So now we need to actually create our
18:12:37script to automatically run this. So
18:12:39we're going to do something called
18:12:41import OS. And let me tell you there's a
18:12:44thousand different ways to do this. And
18:12:46there are better ways to do this, but
18:12:48they're much more complex, much more
18:12:50complicated, and some cost money in
18:12:53order to do it. I'm going to show you
18:12:55different options on how to do this in
18:12:57future videos on how to automate your
18:12:59Python scripts. But this one to me is
18:13:02one I've used a lot um many many times
18:13:04for different projects and it works. So
18:13:07I'm not going to show you the most
18:13:09complicated thing in the world. I'm
18:13:10going to show you something that I've
18:13:11just used a lot. And so we're going to
18:13:13say from time import time from time
18:13:18import sleep. That one's important.
18:13:21And now we're going to create our loop.
18:13:24So, what these um what the time and the
18:13:26sleep and the OS uh or your operating
18:13:29system, what what these are going to do
18:13:31is they're going to give us the ability
18:13:34to track the time and we're going to be
18:13:36able to run through and call this
18:13:39function in certain intervals that we
18:13:42want. So, let's create our for loop.
18:13:45We're going to say for i in. Now you can
18:13:49create this specific part in different
18:13:53ways, but what I'm going to do is I'm
18:13:55going to say range of one. Uh let's say
18:13:57333.
18:13:58And I say 333. And if you remember from
18:14:01the first video on the API, you only
18:14:03have 333 runs per day. And so if I ran
18:14:09ran this 333 times today, that would be
18:14:13our max. And so that's why I'm using
18:14:15that 333 just for reference. So now
18:14:18we're going to do API_Runner.
18:14:22So in this loop we're going to call this
18:14:24function up here and then I'm going to
18:14:26say I want to prove or or show have an
18:14:29output to show that this is running
18:14:31through successfully. So I'm just going
18:14:33to and you can write anything here.
18:14:34We're just going to say API runner
18:14:38completed
18:14:41uh completed successfully.
18:14:44Successfully. How do you spell that?
18:14:47successfully.
18:14:48That doesn't look right.
18:14:51I'm just going to say completed. All
18:14:52right, forget that. I don't remember how
18:14:54to say uh spell successfully. If that's
18:14:57if it's spelled it right, you guys spell
18:14:58it that way, but I can't remember. Now,
18:15:00we're going to use this sleep right
18:15:02here. Now, this counts it in seconds.
18:15:05You can change it to minutes, hours,
18:15:07whatever. We're going to have it run
18:15:09every minute, which is every 60 seconds.
18:15:12And so this is going to I'm just going
18:15:14to say it's going to sleep for one
18:15:17minute.
18:15:18And then we're going to say exit.
18:15:23So all this is going to do and this is
18:15:26again fairly simple. It's just a simple
18:15:29for loop. And what it says is it's going
18:15:31to call this API. It's going to tell us
18:15:34that it ran successfully and then it's
18:15:36going to wait for 60 seconds and it's
18:15:37going to run again. That's it.
18:15:40So, let's run this and see what happens.
18:15:43See if what we did works. So, it ran the
18:15:45first time. Now, I'm not going to I'm
18:15:49not going to bore you because I'm doing
18:15:50this live. Exactly what we're about to
18:15:52get is what we're going to use. I didn't
18:15:53run it overnight or or for a week so
18:15:56that we have a bunch of data. I'm what
18:15:58you were going to work with, I'm going
18:15:59to work with as well. So, I'm going to
18:16:01wait a few minutes. I'm going to let
18:16:02this run. I want you to do the same
18:16:04thing. I'm going to let this run for
18:16:06maybe like five minutes or so and we'll
18:16:09work with what we have and we'll keep
18:16:11going with the project because again
18:16:13we're not the point of this project is
18:16:15not to create the final product where
18:16:17we're creating all the visualizations
18:16:18that will most likely be in another
18:16:21video where we're taking all this data
18:16:23and doing all these things with it. The
18:16:24point of this video is to automate it,
18:16:26clean it up to where we have it to where
18:16:28we can really use it and then I'm going
18:16:30to let you guys loose and you guys can
18:16:31do whatever you want with it. And I
18:16:33think it's really setting you up for a
18:16:36lot of successful projects in the future
18:16:38that you can do all by yourself without
18:16:39me having to walk you through it. So, as
18:16:41you can see, it's already ran through
18:16:43twice. I'm going to pause for a second.
18:16:45I'm going to let that run through uh
18:16:47just a few more times and then we will
18:16:48continue with the project. All right, we
18:16:51are back. And of course, it's only ran
18:16:53what, five times. Um it has not reached
18:16:56the limit of 333. So, we are perfectly
18:16:58fine. What I'm going to do is I'm just
18:16:59going to stop this by clicking this uh
18:17:01square up here and it's going to give us
18:17:03some error and then we're going to check
18:17:05it and we will see what we have. I don't
18:17:09know why it's taking so long if I'm
18:17:10being honest. All right, so I
18:17:11interrupted it and let's run this. Let's
18:17:14see what we got. I hope we have more
18:17:15than 15 because if not, I'm going be
18:17:17very upset.
18:17:20Okay,
18:17:22so okay. Well,
18:17:25uh I made a mistake. Um, I was supposed
18:17:28to put data frame right here and I had
18:17:32dataf frame 2. So, um, take change your
18:17:37script. Do not do what I just did. We're
18:17:39supposed to be append. It's supposed to
18:17:41be dataf frame append. And we're
18:17:43supposed to be appending the original d
18:17:45this data frame two to the original data
18:17:48frame. So, um, I messed up on that one.
18:17:51Let's rerun that. Let's rerun that. Um,
18:17:55let's see. Um,
18:17:58local variable DF reference before
18:18:00assignment. Okay, this is perfect
18:18:02because this happened to me before. Um,
18:18:05we're running into all sorts of good
18:18:06stuff. I like to keep this stuff in my
18:18:08videos. I laugh because I hate running
18:18:10into mistakes, but everybody says they
18:18:12they are happy that I do this. Um, so
18:18:15I'm going to keep doing it. I'm not
18:18:16going to cut this out. I promise. Um,
18:18:18but what we actually need to do is we
18:18:20need to go back up to this function
18:18:22because what happened was is we called
18:18:24this data frame
18:18:27and now it's it's because it's in a
18:18:29function, it's in what they would call a
18:18:31local variable. What we need to do is we
18:18:34now need to state that this is a global.
18:18:38Um, it's just called a global. That's
18:18:41all it is. Um, and so what we're going
18:18:42to do is we're going to do tab. We're
18:18:44going to say global say df.
18:18:48And what this should do is this should
18:18:50declare it as a global variable and it
18:18:53should let this run properly. Let's hope
18:18:56it does.
18:18:58All right, it's running. Um, again, I
18:19:01ran into mistakes. Let me tell you
18:19:03something while we're here for just a
18:19:05second. This project I ran into probably
18:19:08a hundred mistakes or a hundred errors
18:19:11or issues that I had to research for
18:19:13hours um and hours. I'm legitimately on
18:19:16Stack Overflow and just googling and
18:19:18figuring figuring these things out.
18:19:20There were a lot of new things that I
18:19:21had never run into before um just on
18:19:23this project. And so um everything that
18:19:26you're seeing is from after I went
18:19:28through all of those things or after I
18:19:30fixed all of those things and had to
18:19:32really work through them. It was it was
18:19:33very um it was frustrating at times. I
18:19:36just I couldn't figure it out. And so
18:19:38what you're looking at is kind of the
18:19:39polished version of that now that I have
18:19:41everything laid out because I I can't
18:19:43spend 10 hours on a project. nobody
18:19:45would watch it. So, just know that if
18:19:48you are running into some of these
18:19:49mistakes or you run into mistakes later
18:19:51on when you're expanding this project,
18:19:53that's completely normal. So, what we're
18:19:55going to do is we're going to let this
18:19:56run for a little bit and then after
18:19:59maybe three or four minutes, we'll come
18:20:01back and we'll keep going with the
18:20:03project. All right. So, let's run this
18:20:06and check and see if we have uh the data
18:20:09that we're looking for. Uh, and it looks
18:20:12like we do. Let's go actually back up
18:20:14here really quick up.
18:20:18We want to set this to display max rows
18:20:21because I want to be able to see all the
18:20:23rows and not just um a few of them. So,
18:20:27and that just instead of it gives us
18:20:29this scrolling instead of that dot dot
18:20:31dot that shows us just a few. So,
18:20:33there's our original 15. Then we have
18:20:36the next um the next loop and then we
18:20:40have the next loop. And let me scroll
18:20:42over to the timestamps and I'll show you
18:20:44what I mean. Um, so this was ran on
18:20:4652651.
18:20:48Let's go down. 526 at 150 2905.
18:20:55I say 1501 2905. And the next one you
18:20:59can see was ran at 3006.
18:21:0331. These are all the ones minute after
18:21:05each other. My original one was from
18:21:07earlier.
18:21:0932 33. Yeah. So, you can see 32, 31,
18:21:133030 or um 3029. And this one was about
18:21:1615 minutes ago when I first um ran the
18:21:19original data frame, right? All right,
18:21:22guys. This is Alex from the future. I've
18:21:24actually completed this entire project
18:21:26uh in the video, and you're about to see
18:21:28all that after this. But I wanted to
18:21:30show you one more thing that you can do
18:21:31in this function up here that I didn't
18:21:33show you uh originally that I'm coming
18:21:36back to show you, and that's how to
18:21:37actually put it into a CSV. Now, all
18:21:40we've done in this one is we we've kept
18:21:43it all enclosed in a dataf frame, and
18:21:45that's it. And that may be great, but a
18:21:48lot of you guys are going to want to
18:21:50automate this and put it into a CSV. And
18:21:53I want to show you how to do that. All
18:21:54right. So, what I'm going to show you
18:21:55really quickly is right here in this uh
18:21:58in this folder right here, I have all
18:22:00these different API 3es and fours. These
18:22:02were tests that I did before. But what
18:22:04you can do is instead of just putting it
18:22:06into a dataf frame, you can actually
18:22:08append the data to a CSV and have that
18:22:11CSV sitting out there for you instead of
18:22:13just keeping it all in a dataf frame.
18:22:16And there's a lot of different uses for
18:22:17that. You may want to have that file
18:22:21separately from here just in case
18:22:23something times out or something breaks,
18:22:25which is a legitimate concern, or your
18:22:27computer shuts off or or something like
18:22:28that. That is a legitimate concern. So
18:22:31what we're going to do is we're going to
18:22:32say um if not and this is basically an
18:22:36if statement. We're going to say os.path
18:22:41dot is file. So what this is going to do
18:22:44is check if there's already a file under
18:22:47this name. And we're going to do r dot
18:22:50or or r. Um, if you have never done um
18:22:55if you've never done CSV stuff before,
18:22:58uh, it's really important that you put
18:23:00that you you're going to get an error
18:23:01every time. So, we're going to take this
18:23:03right here and we're going to copy that
18:23:06and we're going to put that right here.
18:23:08And then we're also going to do a
18:23:11slash and then we're going to name it
18:23:12basically. Um, let's name this API
18:23:15because I don't think I have that one in
18:23:16there. I think I deleted it. Yeah. So, I
18:23:18don't have API. So, I'm just going to
18:23:19keep it API.csv. CSV
18:23:22and then I'm going to close that
18:23:23parentheses and then we're going to add
18:23:26a colon right here and we're going to
18:23:28say if that does not exist we are going
18:23:32to write this to it and create it. So,
18:23:35we're going to say dataf frames. That's
18:23:37this dataf frame right here.
18:23:40Dataf frame dot and we're going to say 2
18:23:44CSV. And we're going to do that R. And
18:23:48then we're going to copy this. So, let's
18:23:52just let's just replace it like that.
18:23:57And then we're going to say comma
18:24:00header
18:24:02oops header is equal to
18:24:06column
18:24:08names. So what this is going to do is if
18:24:12we run through this and what we would
18:24:14have to do is um I'll talk about this in
18:24:17a little bit. We'll have to change this
18:24:19up a little bit. But what this is going
18:24:21to do is going to check to see if this
18:24:24file right here exists. If it does not,
18:24:27it is going to create it and create the
18:24:30column headers based off the this data
18:24:32frame. That is what that does. Now, what
18:24:35we want to do is say else. And this next
18:24:39part that we're going to write is saying
18:24:40if there's already the API file there,
18:24:43we want to append the data. We don't
18:24:45want to overwrite it or anything like
18:24:47that. We want to append the data. So,
18:24:48we're going to say we're basically going
18:24:50to copy this.
18:24:52Maybe not the whole thing, but I already
18:24:54did it. Um, so we're going to copy that
18:24:57and we're going to say mode. Oops. Mode
18:25:01equals A.
18:25:04And A stands for append. And then we're
18:25:06going to say header. Oops, keep messing
18:25:09up header. And we're going to say false.
18:25:11Oops. We're going to say false, which
18:25:14means when it appends the data, it's not
18:25:16going to use those col the column
18:25:17headers every time, which you don't want
18:25:19because every time you append it, if you
18:25:21added the headers, every 15 rows, every
18:25:2515 rows, you're going to have another
18:25:27headers that you're going to have to
18:25:28like go out into that CSV and filter out
18:25:30and and get rid of them. So, we're going
18:25:32to say header equals false. Now, just a
18:25:34second ago, I said you would need to
18:25:36mess with this just a little bit, and
18:25:37you would because every time um you'd be
18:25:41putting in this dataf frame, which it's
18:25:43already appending it to this data frame.
18:25:45So, every time you'd be creating a lot
18:25:47of duplicates if you kept it exactly as
18:25:49is. What you were going to need to do is
18:25:51basically take it back to its to its um
18:25:53bones. Um so, you need to
18:25:57kind of keep it like this. So, what you
18:26:00need to do is just now run this and it
18:26:02would work perfectly. Uh, let's test it
18:26:05really quick. Um, to see if it works.
18:26:07Uh, because I'm I'm promising you
18:26:09something. I want to make sure it
18:26:10actually works. Let's run it this time.
18:26:13Okay. So, it just ran for the first
18:26:15time. So, it should have created this
18:26:17file. Let's go see if that works
18:26:19properly.
18:26:21So, now it just created that file. And
18:26:23now we're going to see if it actually
18:26:26appends the data. So, let's wait just
18:26:28one time. Um, and then I'm going to stop
18:26:30it. I'm going to see if it works. Again,
18:26:32I'm just verifying to make sure that
18:26:34what I'm telling you is actually
18:26:35working. Uh, because if it doesn't, I
18:26:38would feel terrible. Uh, we don't want
18:26:39that. And while that's running,
18:26:41actually, I'm going to add this because
18:26:45now I want to show you how to call it.
18:26:47Um, super easy. We're just going to do
18:26:49PD.
18:26:53CSV. Do that.
18:26:56We're going to call this
18:27:00just like that.
18:27:02And then we're going to say data frame.
18:27:04And we're just going to do 72.
18:27:08Something random because I've already
18:27:10done this whole project. I don't want to
18:27:11mess anything up. So we're going to say
18:27:13[cough] data frame [clears throat] 72.
18:27:15So now let's stop this.
18:27:18Um, and what we're going to do is once
18:27:21that stops, we're going to run this and
18:27:23see if it actually um worked and to see
18:27:26make sure that this actually pulled the
18:27:28data in. All right, so we interrupted
18:27:29it. The file is ready to be read in. So,
18:27:33let's read it in. There's our file. Um,
18:27:38let's see. What did I mess up or did I
18:27:40mess anything up?
18:27:42Ah, I didn't mess anything up. This is
18:27:44the index for this file. And we already
18:27:47had this in here. we'd probably be able
18:27:48to get rid of it. But if you see, we
18:27:50have 0 1 2 3 4 5 6 7 8 9 14. Then we
18:27:54have zero 1 2 3 4. And if we look at the
18:27:57time stamp, it should be 1 minute apart.
18:27:59So it's 11945.
18:28:02It said 120 45. So this worked exactly
18:28:05as planned. Um again, you have two
18:28:08different options. You can just keep it
18:28:09how it was before. And I'll leave both
18:28:11of those options, you know, in the in
18:28:13the script so that you can kind of
18:28:15choose which one you want. But um that's
18:28:18how you do that. So then right here
18:28:20you're appending it to a CSV file. And
18:28:22then if you just keep this and you get
18:28:24rid of all this, you're just appending
18:28:25it to a dataf frame. Now please continue
18:28:28with the rest of the video that I
18:28:30already have done. Um but again I'm
18:28:32future Alex. So uh please continue with
18:28:34the rest of the video. Okay. So we have
18:28:37all this data. We have we have so many
18:28:40columns we can do. Now, you know, if you
18:28:44want to completely just go and do your
18:28:45own thing, you absolutely can do that.
18:28:48I'm going to mess around with a few
18:28:49things. Um, kind of show you something
18:28:53that I did that I thought was really
18:28:55interesting, um, in order to visualize
18:28:58this data a little bit and transform it
18:28:59a little bit to make it more usable. Um,
18:29:02but we're not doing a full data clean.
18:29:04That's not what this project is. We're
18:29:05not doing a full data cleaning of this
18:29:07data. That would be a m a very large
18:29:09undertaking because honestly, this needs
18:29:11a lot of work.
18:29:12One thing that I do want to clean up
18:29:14really quick uh is is this right here.
18:29:18This the math will be fine. It's just
18:29:21the way that it's shown on here is in
18:29:22state uh the scientific notation and I
18:29:24don't like it. So, what I'm going to do
18:29:27really quickly
18:29:29is just um get rid of that. So, we're
18:29:31going to uh we're going to say PD
18:29:36set and do underscore option and this is
18:29:40going to be do parenthesis. We're going
18:29:44to say display this is just this how
18:29:47this is formatted. So we're going to
18:29:48display uhflat_mat
18:29:54and we're going to say comma and we're
18:29:57now we're going to use this lambda say x
18:30:01colon and we're going to say
18:30:05percent
18:30:075f
18:30:10that right there and we're going to say
18:30:12percent x. Now, if you don't know what
18:30:15lambdas is, lambdas are, um, I highly
18:30:18recommend looking those up. Um, again,
18:30:21this is not a beginner tutorial. Whoops.
18:30:25No such keys. Display floor format. That
18:30:28makes sense. Uh, this is float. Yeah,
18:30:32guys, this is not a beginner's level.
18:30:34All right. Uh, you can't use the floor
18:30:36format. This is the float format. All
18:30:38right. So, now let's take a look at this
18:30:39uh this df uh this data frame that we
18:30:41have. So, we're just going to hit df.
18:30:43Click enter. And now our numbers are a
18:30:45little bit more easily readable. I
18:30:47prefer it this way. You do not have to
18:30:49do this. I'm doing this just because
18:30:50this is what I prefer.
18:30:53So, let's jump right into it. Um,
18:30:55something that when I saw this data, I
18:30:58was like, something that I really
18:30:59thought was interesting is this percent
18:31:02change of 1 hour, percent change 24
18:31:05hours, 7 days, 30 days, 60 days, 90
18:31:07days. If you're not in crypto or you
18:31:09don't do investing or anything like
18:31:10that, what this is going to show us is
18:31:14how I mean it's pretty obvious how much
18:31:16the price of this coin has changed over
18:31:19the last hour, 24 hours, 7 days. So, as
18:31:22you can see, it's it's barely fluctuated
18:31:24over the past 24 hours. A little bit
18:31:27over the past um 7 days, a lot over the
18:31:30last 30 days, 60 days, and 90 days. 20 -
18:31:3426% - 33%. We're in May. Hey, we just
18:31:37had a kind of a crash in crypto a couple
18:31:38weeks ago. So, I mean, this tracks,
18:31:41right? But I want to visualize this, see
18:31:45this, and kind of see um, you know, how
18:31:49this is going to look and how if I can
18:31:52gain any insight from that information
18:31:54and just having it all displayed for me.
18:31:56But in its current state, um, you know,
18:32:00we really cannot do that. Um, now
18:32:04another issue, not an issue, but another
18:32:07thing that we have to take into
18:32:07consideration is we have
18:32:10Bitcoin right here. We have Bitcoin
18:32:13right here after different polls. Now,
18:32:15we just did it a minute after each
18:32:16other, but for your project, you may do
18:32:18it a a run each day, a run every hour or
18:32:23something like that, right? And
18:32:26if you did that, your data could be very
18:32:29different. And so you may just want to
18:32:32take this first one. But what I'm going
18:32:35to do for the sake of this project, I'm
18:32:36going to group them. So let's go down
18:32:39here and we're going to say df.group
18:32:45by. And so if you've ever done something
18:32:47like SQL, uh this is how you group by in
18:32:50pandas. Basically, we're going to group
18:32:52by uh the name. So so on bitcoin,
18:32:55ethereum to other. So, we're going to
18:32:57we're going to do that on name.
18:33:01And uh I'm not going to I'm going to say
18:33:04sort is equal to false. Oops. I'm not
18:33:08going to sort it. Uh you could say true
18:33:10there, but we're not going to. And I
18:33:13guess you'll see why later. We're going
18:33:15to do an open bracket.
18:33:17And now we need to choose what we're
18:33:19going to group by uh or what we're going
18:33:21to what columns we're going to have. So,
18:33:24I'm going to do another open bracket.
18:33:25And I'm just going to copy and paste
18:33:27these. So I'm going to start right here
18:33:29at quote percent one hour. So I'm gonna
18:33:32do boom and then
18:33:36go over one. And we're going to take 24
18:33:40hours.
18:33:42Paste that comma.
18:33:46We have the 7-day 30-day.
18:33:49And we're going to do like that.
18:33:54And I'm just going to do comma. I'm
18:33:57going to do the same one, but I'm just
18:33:58going to manually change it to 30-day
18:34:02rid of that at the end. I don't know
18:34:03what that is. Uh then we're going to do
18:34:0760 days
18:34:10and comma. And we're going to do our
18:34:12last one, which is 90 days. And let's
18:34:17see what that gives us.
18:34:19Uh doesn't give us anything.
18:34:22Okay, I know what's wrong here. Um, we
18:34:25forgot to add basically the what we're
18:34:28we have we're grouping by something. We
18:34:30need to have like an average, uh, a
18:34:32mean,
18:34:34a mode, or something like that, right?
18:34:37So, all we have to do is go to the end
18:34:39right here and let's just do mean. We're
18:34:42going to do an average.
18:34:44Um, and and so we're taking this number.
18:34:48So, let's say this is for Bitcoin. So,
18:34:50we're going to take this number in this
18:34:52one hour for every time it's Bitcoin,
18:34:53it's going to group them all together.
18:34:55Um, and then it's going to average them.
18:34:58So, in the past five minutes where it's
18:35:00been running, we're going to take the
18:35:02average or the mean of that. So, let's
18:35:05run this again. And so, now this is our
18:35:08output. Let's take a look.
18:35:11Oops, I meant down here. Let's run this
18:35:14now.
18:35:18Now what we have is all of these um
18:35:20cryptos. These are all 15 that we have.
18:35:22And this is the average um for this 1
18:35:24hour, 24, 7 days, 30 days, 60 days, and
18:35:2890 days. So now we have all of our
18:35:30cryptocurrencies over here. We have our
18:35:33percent changes up top and then our
18:35:35averages um here as well.
18:35:38And so now what we're going to do is,
18:35:41you know, if you try to visualize this
18:35:43as is, it doesn't really work because
18:35:46these percent changes are up here as
18:35:48columns and we don't really want them as
18:35:51columns because that it just doesn't
18:35:52work for visual for actually creating
18:35:54the visualizations. We really need these
18:35:56to be rows. And so my initial thought
18:35:59when I was doing this was I of course I
18:36:01need to pivot. Um, you know, if you've
18:36:03ever used pivot like in Excel or or
18:36:06PowerBI or something like that, that was
18:36:07my first thought and I tried everything
18:36:10and I could not could not get it to work
18:36:11and I almost gave up until I I ran
18:36:14across um something called stacking or a
18:36:17stack and and so this was not something
18:36:20that I I I think I have used it before,
18:36:22but I I couldn't remember to be if I'm
18:36:24being completely frank. I couldn't
18:36:25remember how to do this. So, I just did
18:36:28um once I saw what it was, I did stack.
18:36:31Let's make that day four. And you don't
18:36:33have to do this. Uh you can keep this
18:36:34all the original data frame. I'm just I
18:36:37like for visual purposes. You can see
18:36:38like the progression that we're making.
18:36:40Um but I like to, you know, create it
18:36:43new data frame. And I can always go back
18:36:45and look at this data frame three um as
18:36:48we go. But you don't you don't have to
18:36:49do that. That's just what I'm doing. So
18:36:52now let's take a look at this. Now uh up
18:36:54here we had Bitcoin and we had all these
18:36:56columns and we had uh these numbers as
18:37:00rows. But now we have all of these as
18:37:03rows as well. This how we have this is
18:37:06much much more usable. Um and if you've
18:37:09ever done something like pivot or this
18:37:11stacking before, you'll know that you
18:37:13you kind of have to do it if you really
18:37:15want to visualize this well.
18:37:18But um you because we just stacked it,
18:37:21it kind of changed it. So if we look at
18:37:24um let's look at the type of let's do
18:37:27type of data frame three. This is before
18:37:31um before we stacked it. This was in a
18:37:34dataf frame. But now let's go and look
18:37:36at dataf frame 4. So this is a series.
18:37:40This is no longer a data frame. So we
18:37:43have to remember that that's that's
18:37:44really important because we can no
18:37:46longer treat it as a data frame. It's
18:37:48now a series. So we want to get it back
18:37:50to a data frame. We don't want it to be
18:37:53like that because you can't really use
18:37:55it in the series. So what we're going to
18:37:57do and let me just create a few of these
18:37:59so it can be up here better. So now what
18:38:02we're going to do is we're going to say
18:38:04dataf frame 4 dot and something called
18:38:07two frame. So we're going to make this
18:38:10into a frame. And now we're going to
18:38:12specify the name. And it doesn't mean um
18:38:15the name like right here. We actually
18:38:18mean the name of these values right
18:38:20here. This is part of the stacking
18:38:22process in in these columns or these two
18:38:25columns. So let's go right here and
18:38:28we're going to call it let's just say
18:38:31values
18:38:32and let's make this data frame five
18:38:38and let's see the output whoops for data
18:38:41frame five. And now so there's that
18:38:44values and now this already looks a lot
18:38:48better. Right? So it's in this it's in
18:38:50this more um this is already a data. So
18:38:53this is a data frame. So let's look type
18:38:55dataf frame five. So now it's in a data
18:38:58frame. But
18:39:00the issue is is that this name is kind
18:39:03of acting like a an index which we don't
18:39:07want because we want to be able to use
18:39:09this. So it doesn't really have an index
18:39:11at the moment. So we need to give it an
18:39:14index. But typically when you give an
18:39:17index you'll do something like um we'll
18:39:19say dataf frame five we'll do set
18:39:22index and then you'll do something like
18:39:25um name. So let's just do dat 6 is equal
18:39:30to we'll see we'll see what happens
18:39:32here. It's going to give us an error.
18:39:34Oops. What I meant is we're going to do
18:39:36day frame five bracket
18:39:40name. That's a column right? We're going
18:39:42to do that. And it's basically going to
18:39:45say that that's not going to work. And
18:39:48what we need to do is what or at least
18:39:50what I want to do and what we're going
18:39:52to do in this video is I'm going to
18:39:54create numbers. I really would just want
18:39:57it to be numbered. 1 2 3 4 5. That's
18:39:59what I want. Um, but we don't have that
18:40:02right now. I can't just will it into
18:40:04existence. So now what we're going to do
18:40:06is kind of create uh an index basically
18:40:08out of thin air. So we're going to do
18:40:10PD.index index
18:40:13and we're going to say uh you know we
18:40:16basically want how many um rows are in
18:40:20here. So that's what we want our our um
18:40:23index to be. We want it to count how
18:40:25many are in here. Now you can make this
18:40:26dynamic and I it probably wouldn't be
18:40:28that hard, but I'm going to take the
18:40:30super lazy route. Um and I'm just going
18:40:32to say
18:40:34let's do df5
18:40:38or oops df5.count. count
18:40:42and there's 90 values in here. So I'm
18:40:45going to do is I'm going to do a range
18:40:49of 90. Uh and this is not uh I would
18:40:53definitely make this dynamic but I'm
18:40:55again I'm just being
18:40:58being a little bit lazy. We're call this
18:41:00index is equal to and I'm going to put
18:41:03this index right here. So now this is a
18:41:05number. So now it's going to
18:41:08literally index this for us. Now I've
18:41:11ran into this issue many times. Um, and
18:41:14so what I need to actually do is to
18:41:16reset this index and then do it properly
18:41:18the first time. Uh, so let's do re let's
18:41:22get rid of this. Let's reset this index.
18:41:24Um, and it actually fixed itself. Um, so
18:41:29what was happening was is we were
18:41:31indexing something that was already
18:41:32indexed and we're causing issues
18:41:35in in a nutshell. So we reset the index
18:41:37and now this is what it looks like and
18:41:39this is exactly what we want. This is
18:41:42really how we wanted it formatted in
18:41:44order to for our visualizations. We have
18:41:46multiple rows for the bitcoin. Um each
18:41:49of these columns are is now a row with
18:41:51the value attached to it. Exactly what
18:41:53we wanted. So, um, really quick, I for
18:41:58whatever reason it it makes that, uh,
18:42:01level one. I don't know why, but we're
18:42:03just going to rename that column really
18:42:05quickly. So, we're going to do dataf
18:42:07frame six dot rename.
18:42:10And then we're going to do an open
18:42:12parenthesy. Say columns equal to, and
18:42:17we're going to do one of these bad boys.
18:42:18Oops. One of these bad boys. This this
18:42:20type of bracket. And we're going to say
18:42:23level underscore one and we do a colon
18:42:27and then oops
18:42:30and then a colon [snorts] and then we
18:42:32want to change it to and I'm just going
18:42:33to call this the percent
18:42:36change. So let's call this dataf frame
18:42:397.
18:42:42You don't have to do that. I'm just
18:42:44doing it. So now this looks much much
18:42:47better. Now let's try to visualize this
18:42:50one. Um because we haven't done any
18:42:51visualizations yet. We've just been
18:42:52messing with the data a little bit. I I
18:42:54you know I kind of want to see how we
18:42:56can use this. This is something that I
18:42:58personally am interested in. So I kind
18:43:00of wanted to see visualize how these
18:43:02changed over these these time periods.
18:43:04Um but we need to um import some stuff
18:43:07in order to be able to visualize this.
18:43:10So we're going to import Seabor as SNS.
18:43:14And if we need to um we're going to
18:43:16import mapplot lib as well. I don't know
18:43:19if we'll use it right now or at all, but
18:43:22um we're going to we're going to add it
18:43:25in here either way. So now those are
18:43:28added. And so what we're going to do is
18:43:31come right here. We're going to do
18:43:32SNS.plot.
18:43:36And we're going to oops we're going to
18:43:39say the xaxis is equal to and we want to
18:43:43do this as the percent change percent
18:43:48change.
18:43:50And then we have the yaxis. Now we want
18:43:54the y-axis to be these values right
18:43:56here. Say comma y is equal to and we're
18:44:01going to say values.
18:44:03Oops. And then we're going to say comma
18:44:06and we'll say we want to basically
18:44:09create a legend. Um I guess you could
18:44:11call it. We're going to say hue is equal
18:44:14to name. Um I'll show you what it looks
18:44:16like without it. And then you know you
18:44:18can see that that we need that. We're
18:44:22going to say the data is equal to this
18:44:24data frame seven.
18:44:28Data frame seven.
18:44:31And then we are going to say the kind
18:44:34is equal to.
18:44:38Now, let's run this and see what we get.
18:44:42And super quickly with just, you know,
18:44:44limited um inputs, here's what we have.
18:44:48Now, this looks really good. We can
18:44:51narrow this down if we wanted to to a
18:44:53few less because there's a lot here and
18:44:55there's a lot of colors. But again,
18:44:57[cough] that's just because
18:44:57[clears throat] we have a lot of
18:44:59different stuff. But there's a few that
18:45:01are doing really well. I think this is
18:45:03Tron.
18:45:05Um, and then we have a few that are not
18:45:08doing so well, but it's really hard to
18:45:10see. If you look down here, it's really
18:45:12hard to see this. Um, and that's just
18:45:16because of the the column names. And so,
18:45:18I actually want to change these column
18:45:20names or these values so that when we
18:45:23visualize it right down here, it it
18:45:26doesn't look like that. I kind of want
18:45:27this to be, you know, at least one good
18:45:30visualization you can take out of here.
18:45:32Now, this is definitely not perfect or
18:45:33complete by any means, but you know, you
18:45:35can take take that away from here. Um,
18:45:38so let's um I did alt enter, which adds
18:45:42another row. I could have just pushed
18:45:43plus. I was kind of the lazy way. Um,
18:45:46what I'm going to do is I'm going to
18:45:49change these um these values in here.
18:45:53So, how I'm going to do that is I'm
18:45:54going to do dataf frame seven. And we
18:45:55only want to look at this one column.
18:45:58So, we'll do that right there.
18:46:02and we want to say dotreplace
18:46:06and we're going to do an open uh uh
18:46:09parentheses and then a bracket. Now what
18:46:12we need to do is I'm just to show you um
18:46:15one of them is I'm going to say this one
18:46:18hour
18:46:20do that. Oops. And then what I need to
18:46:22do is a comma another bracket. This is
18:46:25what it's going to change to. I'm just
18:46:26going to say 1 hour. Oops. 1 hour. Um,
18:46:29and we'll do this one really quick and
18:46:31then I'm going to I don't want you to
18:46:33have to watch me type all this out, but
18:46:34I'm going to go through and basically do
18:46:35all of this uh for those. But let's
18:46:37let's see this really quick. And so now,
18:46:40as you can see that um the originally it
18:46:42said quote usdp percent change 1 hour is
18:46:45now only 1 hour. Now this didn't
18:46:49actually [clears throat] do anything. We
18:46:50need to apply it to this right here. So
18:46:53I'm going to say dataf frame 7 is equal
18:46:56to and then we'll run dataf frame 7
18:47:00again. So now that has actually changed
18:47:03that value. Now I'm going to go through
18:47:05and I'm going to update that for every
18:47:07single one. All right. So I basically
18:47:09just put the other ones um in here that
18:47:11we wanted to change with commas
18:47:13underneath. So I have 24 hours, comma,
18:47:16with the seven days, 30 days, 60 days,
18:47:1990 days, and then this bracket over
18:47:20here, which tells uh it what to change
18:47:23it to. 24, 7 days, 30 days, 60 days, 90
18:47:26days. So let's run this. I haven't even
18:47:29tried it yet. Uh, and it looks like it
18:47:32obviously worked properly. So now, let's
18:47:35go back down here and let's run this
18:47:37again.
18:47:39And look at that. It looks so much
18:47:41cleaner, so much nicer.
18:47:43Um and as you I mean all of them with
18:47:46that one hour change has very little
18:47:48change and then you can look back so we
18:47:51can see back within 90 days it's gone a
18:47:54lot of these have gone down which again
18:47:56if you're following crypto you know
18:47:57there's a big crash recently um
18:48:00especially with with you know all these
18:48:01altcoins um that you're seeing right
18:48:03here went down a ton. So, I think this
18:48:06is um avalanche or die or whatever.
18:48:09These ones are, you know, went down
18:48:11dramatically, whereas there's one up
18:48:14here, this lone wolf um that's just
18:48:16that's just did doing really well for
18:48:18whatever reason. So, it's really
18:48:19interesting um to see. Now, this is a
18:48:22pretty specific um visualization that I
18:48:26personally wanted to see and I thought
18:48:28was interesting. You can do absolutely
18:48:30whatever you want to do with this data.
18:48:32I mean, there's so much here. You can do
18:48:35a lot, I mean a lot with this data,
18:48:37especially depending on how long you
18:48:39track it, right? I only did this over
18:48:41the course of like five minutes, but if
18:48:43you set this up, um, and you can track
18:48:46it over a longer time. Now, um, let's
18:48:50say you wanted to do something much
18:48:52simpler, uh, you just wanted to look at
18:48:54like Bitcoin over that time that you,
18:48:58you know, uh, uh, took the data in.
18:49:00That's going to be a lot simpler than
18:49:01what we just did, and I'll show you how
18:49:02to do that really quickly. So, we're
18:49:04going to look at the data frame and we
18:49:06are going to say uh or we're going to
18:49:09take specific columns. We just want um a
18:49:13few columns that we want to keep or or
18:49:15pull from. So, we're going to take oops
18:49:18we're going to take the name column.
18:49:21We're going to do uh
18:49:24might be easier if I copy them, but I'm
18:49:26just going to write them out. Quote,
18:49:28USUSD
18:49:29price. This is the price of the actual
18:49:32cryptocurrency.
18:49:34Then we're going to do
18:49:36time stamp.
18:49:39And let's make this data frame. And
18:49:42we're just going to do 10 for absolutely
18:49:44no reason.
18:49:47Maybe I should have made it nine. It
18:49:48would have been easier. So now we just
18:49:49have these um these columns. And you
18:49:53know we have all these separate columns.
18:49:56So what we can do and the re kind of the
18:49:58reason I want to show you this is you
18:49:59can just query this really quickly and
18:50:01just take the columns that you want. So
18:50:04let's say we just wanted to look at
18:50:05Bitcoin. So we're going to say dataf
18:50:08frame 10 dot query do open parenthesis
18:50:12and we're going to say name is equal and
18:50:16equal is not like that uh when you're
18:50:18doing it like this you need to say equal
18:50:20equal equal to
18:50:23oops ignore that uh is equal to bitcoin
18:50:28and we're going to do it just like that
18:50:30and we're going to say dataf frame 10 is
18:50:32equal to let's try running that think
18:50:36Something's wrong with it. Try like
18:50:39this.
18:50:41All right, let's try that. There we go.
18:50:44It was just the I needed a double
18:50:46quotation instead of a single quotation.
18:50:47That was the issue. So now we have
18:50:49Bitcoin. We have the price and we have
18:50:51these timestamps. So this is the actual
18:50:52time when we ran it. So this is the
18:50:55original data frame. And then in the,
18:50:56you know, this this project, it took me
18:50:5815 more minutes to get this one. And
18:50:59then we had it running properly for the
18:51:01next five minutes. So that's you know
18:51:03that's actually what we have. Now if we
18:51:06want to just visualize this really
18:51:09simply what we can do is we're going to
18:51:11say
18:51:13uh we're going to do SNS
18:51:15line plot and that's going to be like a
18:51:17little line chart or line graph whatever
18:51:20whatever you want to call it. Then we're
18:51:23going to say x is equal to and we'll say
18:51:27quote
18:51:29no actually we want the time stamp to be
18:51:31on the x axis. Um and then we'll do y is
18:51:35equal to quote usd
18:51:40price.
18:51:42And let's see if that works.
18:51:46Could not interpret timestamp for the
18:51:49parameter.
18:51:51Uh that's because it's not understanding
18:51:55that the data
18:51:57equals dataf frame 10. Now let's try
18:52:01this. All right. So this is uh looks
18:52:05terrible. Let me
18:52:08just say snss. Set theme
18:52:14open parentheses. We'll do style is
18:52:17equal to dark.
18:52:22This looks a little better. Now again,
18:52:25we are looking just at a very very short
18:52:29time series, but we can look at just
18:52:33Bitcoin or we can look at multiple and
18:52:36we're showing this, you know, this line
18:52:38that's showing us this trajectory over
18:52:40time. So, you can get really creative
18:52:42with this. You can run this for a long
18:52:43time. You can show Bitcoin over days,
18:52:46weeks, or months, however long you run
18:52:48this. And so that's really all I've got.
18:52:51Um, honestly, like I said, this is not a
18:52:53I wouldn't say this is a complete full
18:52:56project, but I'm showing you how to do
18:52:58something to enable you to kind of run
18:53:00with it and run with the ball and do
18:53:02basically whatever you want with this.
18:53:04You can pull it from, you know, data
18:53:06from a different API. You can use this
18:53:08exact API and data, but I wanted to show
18:53:12you just a few things that I initially
18:53:14saw that I might do with the data. And
18:53:17you have so much. Let me go back to this
18:53:19original data frame.
18:53:21Uh right, we'll use this one right here.
18:53:24This one right here. Look at all this
18:53:26data. I mean, you have so so so much
18:53:29data. Actually, let's go to this one.
18:53:30This one's better. You have so much
18:53:32data. So many numbers here. Um so many
18:53:35columns that we didn't even look at that
18:53:37you can use. Um and so, you know,
18:53:40there's a lot that you can use here. And
18:53:44I'm really trying to just set you up so
18:53:46that you can run with it and do whatever
18:53:48you want. I could have done a thousand
18:53:49different things here, but you know, I
18:53:51tried to just show you two things that
18:53:53you can do with the data that I thought
18:53:55were pretty interesting or or simple to
18:53:57do. And you know, I want you guys to go
18:54:00out and do something way way better than
18:54:02what I did. So, I hope that this was
18:54:04helpful. I hope that this showed you how
18:54:05to automate that process so you don't
18:54:08have to sit there and click it and
18:54:10append it and do all these different
18:54:11things. that I can show you how to kind
18:54:13of automate this process and hopefully
18:54:15that will be helpful in your future
18:54:16projects. So, with that being said,
18:54:19thank you so much for watching. If you
18:54:21made it all the way to the end, you guys
18:54:22are fantastic. If you like this video,
18:54:24be sure to like and subscribe below and
18:54:26I'll see you in the next video.
18:54:29[music]
18:54:40What's going on everybody? Welcome back
18:54:41to another video. Today I'm going to be
18:54:43walking you through how to create your
18:54:44very own portfolio website.
18:54:49[music]
18:54:52Now, we just completed our data analyst
18:54:54portfolio project series where we walked
18:54:55through four projects in SQL, Tableau,
18:54:58and Python. And so, if you have
18:55:00completed those projects, you now want
18:55:02to share them with potential employers.
18:55:03And I think the best way to do that is
18:55:05to create your own website. In just a
18:55:07little bit, I'm going to show you two
18:55:08options on how you can actually create
18:55:10your own website. The first one is a
18:55:11website builder like wix.com. And the
18:55:14second one is hosting your own website
18:55:16through something called GitHub Pages.
18:55:18Now, if you have never created your own
18:55:19website before, it can sound a little
18:55:20bit daunting, but don't worry. I'm going
18:55:22to walk you through every single step of
18:55:24the way from the very start to the very
18:55:25end. And once you reach the end, you
18:55:27will have a complete data analyst
18:55:29portfolio website. So, without further
18:55:30ado, let's jump on my screen and let's
18:55:32get started. All right. So, the website
18:55:33that you're looking at right now is the
18:55:35actual website that we are going to
18:55:36build in this video. Um, it is hosted on
18:55:39GitHub pages or github.io. So, this is
18:55:42actually being hosted right now by
18:55:43GitHub pages. So, if you type this in,
18:55:45I'll leave a link in the description. If
18:55:47you type this in, um, you will get this
18:55:50page and you can check it out for
18:55:51yourself if you don't want to just watch
18:55:53me look at it. Um, so, you know, it has
18:55:56this little header and you can write a
18:55:57little bit about yourself. And then
18:55:59these are actual projects. So this is
18:56:01our data cleaning in SQL project. Um and
18:56:04then there's the COVID uh data
18:56:06exploration, Tableau dashboards, movie
18:56:08correlation with Python. Um this is a
18:56:10future video. I plan on doing a few more
18:56:13of these projects because I just really
18:56:15enjoy them. So um you know, and then
18:56:18there's this contact information at the
18:56:20bottom. So it's a really simple
18:56:23website and it gets the point across.
18:56:26And uh I have something similar to this
18:56:28for my own personal one. I I use a
18:56:30different variation, but um this all
18:56:33comes from this website, HTML 5. There
18:56:37are lots of templates, lots of options
18:56:39that you can use. Um again, the one
18:56:41we're going to be working with is this
18:56:43one, but I use a different one for mine,
18:56:46and they are really good. I mean, super
18:56:48easy to build and customize yourself.
18:56:53And I will say again, I have no
18:56:54experience doing this. I just watched a
18:56:57YouTube video that showed me how to do
18:56:58this and now I am creating my own
18:57:01YouTube video to show you how to do
18:57:02this. So, it's coming um pretty much
18:57:04full circle. So, like I said, there's no
18:57:07no real narrative to it. It just clicks
18:57:09to your project. Um if you click on this
18:57:11and let's just open a new tab, it'll
18:57:13take you right to our the GitHub
18:57:15project. Um and then you this whoever's
18:57:18checking this out like a an employer or
18:57:20a recruiter can see your code. So, super
18:57:23simple. Another way that you can do this
18:57:25is kind of creating your own website
18:57:28through like a template or something
18:57:30like that. Um, almost like a blog style.
18:57:33So, I imagine it being very something
18:57:35very similar to this where there's this
18:57:37introduction and you can talk about, you
18:57:38know, where you got the data set, how
18:57:40you got the data. Um, and then you can
18:57:42kind of have a more narrative uh
18:57:45approach with screenshots and with some
18:57:47code as well. So, you know, this person
18:57:49included screenshots. Um, and then
18:57:51there's the code right here that I can
18:57:53actually copy um, and paste that and it
18:57:56just walks through the logic of how the
18:57:59project was done. Um, there's a story to
18:58:02it really. And so that might be
18:58:04something that you're interested in.
18:58:05Now, I have done something like this in
18:58:08the past and I used Wix and there's a
18:58:10you can do this completely for free. Um,
18:58:12the one we're doing today is completely
18:58:13free as well, but you know, if you want
18:58:16the customized
18:58:18um the customized URL, you do have to
18:58:20pay for it on Wix, but you can get a
18:58:23free Wix website with the Wix um in the
18:58:26URL. So, you know, try this out. These
18:58:29are super easy and you can find
18:58:31thousands of templates and a million
18:58:32tutorials on how to do them. Um, so
18:58:34that's not the one we're going to be
18:58:35working on today. So, with that being
18:58:38said, uh the very very first thing that
18:58:41we need to do before we do anything is
18:58:43actually download Visual Studio Code.
18:58:46This is where we're going to download
18:58:47that HTML and we're going to be working
18:58:49with it in there. Um again, I don't know
18:58:52if I said this before, but it seems a
18:58:55little bit intimidating at first, but
18:58:56once we actually start looking at it,
18:58:58it's a lot easier than it looks. I
18:58:59promise you. So, if you are me and you
18:59:02have a Windows computer, you'll just go
18:59:04right here. you'll install it. Um, super
18:59:07easy to install. I'm not going to walk
18:59:08you through how to do that. Um, of
18:59:10course, I already have it up and running
18:59:12uh down here. So, once you have that
18:59:15installed, what you're going to do is
18:59:17you're going to come to this website. A
18:59:18link should be in the description. We
18:59:20are going to download this. All you have
18:59:22to click is the free download. It's
18:59:26going to pop up. I'm going to put it in
18:59:27my downloads. I'm going to click save.
18:59:31Fantastic. Uh, so let's go to the
18:59:35downloads and it should be right here.
18:59:36Now, if we open this up, it has a few
18:59:39different things in it. Okay, so um I'm
18:59:41using the Brave browser, so that's going
18:59:43to be right here. So that's just the
18:59:45symbol. But for you, if you're using
18:59:46Google Chrome, that should be the symbol
18:59:48there as well. But this is everything
18:59:50that you should be seeing. And what we
18:59:53want to do is we want to take it out of
18:59:54this um zip folder because it's there
18:59:59are things that can read into it with
19:00:01Visual Studio Code, but I want to make
19:00:02this as user friendly as I possibly can.
19:00:05So, what we're going to do is we're
19:00:07going to make create a new folder. I'm
19:00:09just going to call it massively or you
19:00:11can call it um port website. Whatever
19:00:14you want to call it. I'm just going to
19:00:15do port website. Um and we are just
19:00:19going to I'm going to copy this in. I'm
19:00:21not going to cut it in just in case I
19:00:23make a mistake. So, gonna put all of
19:00:27those um all of those things in here.
19:00:30And now what we're going to do is we're
19:00:33going to go to Visual Studio Code right
19:00:35here. And you should be greeted with
19:00:38this um this right here. And we're just
19:00:40going to click open folder. And we're
19:00:42going to go to port website. And we're
19:00:44going to go select folder.
19:00:46And you're going to say yes, I trust
19:00:48this one. And right over here is all of
19:00:51the documents that we were just looking
19:00:53at. Now, the one that the only one
19:00:56really that we're going to be working
19:00:57in, um, we'll work a little bit in the
19:00:59images, um, because I'll show you how to
19:01:01add your own images. The really the only
19:01:04one we're going to be working in is this
19:01:06index. So, again, it looks complicated.
19:01:10Um, if you've never looked at HTML
19:01:12before, um, it does look a little bit
19:01:14complicated, but HTML to me is one of
19:01:18the more easily understood languages.
19:01:21Um, once you start kind of getting into
19:01:23it, which we're about to, we're going to
19:01:24walk through the entire process, it
19:01:26actually makes a lot of sense and it is
19:01:28pretty simple. Um, something that you're
19:01:31going to want is you're going to want
19:01:33something called a live view. So, like
19:01:35if I click right here and I click open
19:01:37with live server, you don't have that
19:01:38yet, I'm guessing, unless you've done
19:01:40this before. Um, it's going to open up
19:01:42this website. And this is what we're
19:01:44looking at right now. So, it says a
19:01:46bunch of um gibberish or some language
19:01:49that I do not know. And so, we can view
19:01:53this live. Um, in just a second, I'm
19:01:56going to take myself off screen, but
19:01:58before I do that, um, let's download or
19:02:02let's, um, search for that that live,
19:02:07um, I think it's called live share, live
19:02:09server.
19:02:11Let me see what this called.
19:02:13Yeah, live server. So, come right here.
19:02:16It's called this live server. There it
19:02:17is. Yeah, that's the one. So, this is
19:02:20our live server. You just need to click
19:02:21install. takes like 5 seconds and it
19:02:24should be completely installed. Um, what
19:02:26this does is it just hosts a local
19:02:29website. It's not something that anybody
19:02:31can access. Um, but it connects to your
19:02:33code and when we make updates, it'll
19:02:35make a li you can see it live. You can
19:02:36see those updates live. So, I'll show
19:02:38you all that in a second. Just be sure
19:02:40to um be sure to download that or
19:02:42install that. Uh, with that being said,
19:02:45let's get out of this. Let's go. Let's
19:02:47go back right here. Uh with that being
19:02:50said, I am going to take myself off
19:02:51screen so that you can see everything
19:02:53that I am seeing as well. Um it's been
19:02:56really great seeing you. Have lots of
19:02:59different videos coming up, lots of new
19:03:01projects. Um I just I really enjoy this
19:03:04project series. I think I'm just going
19:03:05to do more of them. So uh all right, I'm
19:03:08going to get myself off screen. So let's
19:03:11look at what we actually need to do. So
19:03:14I'm going to um
19:03:17So let me see. Okay, so we're already
19:03:19connected to the live. Um, actually, I
19:03:22got rid of it. Whoops.
19:03:25Let's pull this over. And let's pull
19:03:28that.
19:03:30And we're going to
19:03:32open in live server. So, if we look
19:03:36right over here, and I know this is
19:03:38going to be a little bit squish, and I'm
19:03:39sorry about that. Um, but if we look
19:03:42right over here, this says this is
19:03:44massively. So, you can change that.
19:03:47That's that's this right here. And you
19:03:49can say we're going to say Alex the
19:03:52analyst
19:03:53portfolio. And we'll get rid of this
19:03:55massively. I'm going to hit control
19:03:57save. You can also go up here and hit
19:04:00save. But I'm I'm hit control S. So I
19:04:03hit control S. And just like that, it
19:04:07updates on the website. Now again, this
19:04:09is just a local, so it's nothing that
19:04:11anybody can see, so don't worry.
19:04:13But what we're going to do is I'm going
19:04:15to walk you through the entire process
19:04:16of creating this and then at the end I
19:04:19will show you how to host it on GitHub.
19:04:21Um and it's honestly it's it's a fairly
19:04:24easy process. It's just takes a little
19:04:25bit of time to customize it all. So
19:04:28let's get into it. So we have this um
19:04:31you may not be able to see it. Let me
19:04:32actually pull this up. So it says
19:04:33massively by HTTP. We're going to
19:04:36customize that. Customize that as well.
19:04:38Whoops. I don't want to do that every
19:04:39single time. I'm I'm gonna try not to go
19:04:41full and go back and everything like
19:04:43that. So, we're just gonna say Alex the
19:04:46analyst portfolio.
19:04:50Um, control S. And right up here, it
19:04:52changed it. You may not be able to see.
19:04:54Yeah. Don't ask me that again. Thank
19:04:55you. Uh, right up here, you probably
19:04:56can't see at the moment. We'll see that
19:04:58later. Um, but it it customizes this um
19:05:01tab, which is really cool.
19:05:04So, let's go right down here. Now, this
19:05:07is where it says a free, fully
19:05:08responsive HTML uh 5 template. We can
19:05:14customize that and I highly encourage
19:05:16that you do. So, what you can do, and
19:05:20they actually included their Twitter
19:05:22handle right here, and you can do the
19:05:24same. If you look at this one right
19:05:27here, I included my Alex the Analyst
19:05:30handle that that goes to my YouTube
19:05:31channel. And you can do the exact same
19:05:33thing. include your LinkedIn or your
19:05:35GitHub profile or whatever you want to
19:05:36include in there. Um, and so, you know,
19:05:40be aware that you can do that. So, let's
19:05:43say um, oops, I need to click back in
19:05:46here. So, we're going to say
19:05:50um,
19:05:52data analyst skilled in and then again,
19:05:56don't write what I'm writing. Um, you
19:05:58can it's I'm just going to make it
19:06:00really simple, but you know, this part
19:06:02is meant to be a little bit about you um
19:06:04as who you are. So, I'm going to say
19:06:06data analyst skilled in SQL, Tableau,
19:06:10and Python.
19:06:13And then I'm just going to get rid of
19:06:15all of this.
19:06:18Yep.
19:06:19Everything from here over
19:06:22and control S.
19:06:24And so, super simple. Um, actually, let
19:06:26me Where was that? Four.
19:06:30Four. Here it is. We don't need that.
19:06:33Actually, we don't need any anything
19:06:36from here over.
19:06:39Probably here, honestly. See what that
19:06:42looks like. Um, and yeah, and I can,
19:06:44again, you can use any website right
19:06:46here that you want, and you can
19:06:48customize what it looks like. So, I'm
19:06:49going to say Alex the Analyst. Um, and
19:06:52then whatever URL you want to include in
19:06:54there, that's what you need to put. So
19:06:55now if I save, oops, if I hit control S.
19:06:58So now it says Alex the analyst. Um, so
19:07:02pretty easy.
19:07:04Now we're going to go down and
19:07:07[clears throat] you can use this however
19:07:09you want to use it. I would you can even
19:07:11make this um you can make this like one
19:07:15of your one of your readmes like about
19:07:17you and put the link for that. I decided
19:07:19to include um again on this one I
19:07:22decided to include the project that I
19:07:24thought that we've done that was like
19:07:26the the most impressive or the I don't
19:07:28know the coolest one. I don't know if
19:07:31you consider data cleaning and SQL cool
19:07:33but um I do I think it's cool. So I
19:07:36included that one as my very first one.
19:07:38So that's what we're going to do um
19:07:39right here.
19:07:41So, we're going to go down and it's
19:07:44going to say,
19:07:47let's say it says this is massively.
19:07:49That's not it. Uh, cool. So, let's see
19:07:53what Oh, okay. I know what that is.
19:07:54We'll come back to this up here um in
19:07:57just a little bit. I'm going to go full
19:07:58screen. I'll show you what this is and
19:08:00then we'll come back to it. But, if we
19:08:02go right down here, this is our what
19:08:04they're calling a featured post and then
19:08:06the ones below this are posts. So, in
19:08:09our featured post, um I'm going to get
19:08:11rid of the date. I don't want them to
19:08:13know that I just created it like um I
19:08:16don't know. Oops. I keep doing uh
19:08:19control A selecting everything. Whoops.
19:08:22So, we're going to say um data cleaning
19:08:26in SQL.
19:08:29And we'll get rid of this
19:08:32and control S. Again, I'm just updating
19:08:34it a lot so that you see what I'm doing
19:08:36and where it's going. And we're going to
19:08:38get rid of basically all of this and go
19:08:42back. And we're just going to say
19:08:45in this project we clean data in we
19:08:49clean let's do we clean housing data in
19:08:52SQL server
19:08:55and control S. So, super easy. Again, uh
19:08:57give a little bit more description. I
19:08:59did in my other one. Um and you have the
19:09:01you have you can see that website. So,
19:09:02go check it out. And then we'll have an
19:09:05image and I'm going to show you um at
19:09:07the end. We're going to go back and redo
19:09:09all the images, but I'm not going to do
19:09:11that at this very moment.
19:09:14Um so,
19:09:16what we're now you can have this full
19:09:18story. I chose to do view project.
19:09:24And if I hit Ctrl S, it says view
19:09:26project. I think that just looks better,
19:09:28especially if you're displaying a
19:09:29project. I think it is nice. Uh, now we
19:09:32go into all the indiv individual posts.
19:09:34Um, actually, no, wait. What I want you
19:09:37I want to show you really quick is how
19:09:39you actually link it to this. So, let's
19:09:41go right over here. This is our COVID uh
19:09:44that's our COVID one. Here's a data
19:09:47cleaning project. So, all you have to do
19:09:49is take um take this website. So that's
19:09:53the URL and you're going to put it right
19:09:56here. Now there's three different
19:09:57places. This href is places are places
19:10:00where you can put a link to a website.
19:10:02Um and on here it references this right
19:10:06here. So you can they can click on this
19:10:08data cleaning and SQL. They can click on
19:10:10the image um as because you know this
19:10:12href is right next to this image. They
19:10:15can also click on the view project
19:10:18button. So you can put it in all three.
19:10:20Um and you'll just go like this. You'll
19:10:22you'll stick the URL right where that um
19:10:26hashtag or pound sign is.
19:10:30And then we're going to save that. Oops.
19:10:33Oh, I I this is embarrassing. I am not a
19:10:36website. I am not a web developer as you
19:10:38can see. Um but then if I go in here and
19:10:42I right click and I say open link, it is
19:10:44going to take me to that project. So,
19:10:46super simple. And we're going to do
19:10:48basically that for all of these. Um, I'm
19:10:50only going to show you three and then
19:10:51you can do the rest, but I want to show
19:10:53you how to also do the um put the
19:10:55Tableau. It's the exact same thing, but
19:10:58you know, it's different. So, wanted to
19:10:59show it to you. So, the next one that
19:11:02we're going to do is go down to posts
19:11:05and
19:11:07again, I'm going to get rid of this
19:11:08date. You can keep that in there if you
19:11:09want. Excuse me. And that's totally
19:11:12fine. Just update
19:11:14the date. Um, this is that said Magna.
19:11:17Again, I think this might be like some
19:11:18language. I just don't know about. The
19:11:20next one is data exploration
19:11:23in SQL.
19:11:25And I'm going to get rid of this.
19:11:28And we'll save that. Perfect.
19:11:32And we'll do view project.
19:11:38Cool.
19:11:40And yeah, so now we need to um customize
19:11:44this summary. And so I'm just going to
19:11:47say something really simple. Um data
19:11:51exploration of COVID 19
19:11:57data set in SQL Server.
19:12:02There we go. Let's save that. We have
19:12:05view project. Now let's go get our
19:12:07project. So this is the data
19:12:09exploration. We're going to take this.
19:12:12We're going to copy it and we're going
19:12:14to put it right in here.
19:12:17and right in here as well. And if you
19:12:20want to, you can also include it right
19:12:22up here. So, we have it in all three
19:12:24places. Uh, again, once you click on
19:12:27these, they will come up. Let's go to
19:12:30the next one. We're going to get rid of
19:12:33this.
19:12:35This one is going to be our Tableau
19:12:37projects. So, actually, let me just copy
19:12:38that while we're here. This is going to
19:12:40be our Tableau projects. So if you have
19:12:43one specific project that you want to
19:12:45include, what you would need to do is
19:12:47actually go in here, click view, grab
19:12:50that URL. What I am doing is I am just
19:12:53sharing my Tableau public page. So if
19:12:56you have tons of projects in here and um
19:13:00you want to display all of them then or
19:13:03you want them to be able to see all of
19:13:04them and go and pick and see and choose
19:13:06what they want to look at, then just
19:13:08choose this URL that we're choosing
19:13:09right here. So, um, in here or on in
19:13:13the, um, HTML, we're going to put I'm
19:13:16going to put Tableau projects.
19:13:20And
19:13:21let's go like this.
19:13:24And then we will get rid of uh, that
19:13:28hashtag,
19:13:30pound sign, whatever you want to call
19:13:31it.
19:13:33And we'll hit Crl S. And oh, we got to
19:13:37do the um
19:13:39this as well.
19:13:43This is my This is going to be a
19:13:46terrible Don't use this. This is my
19:13:48Tableau.
19:13:50This holds I'm just This is bad. This
19:13:52holds all of my Tableau
19:13:56dashboards.
19:13:58Don't Please don't do this. Um I am
19:14:01doing this because I don't want to take
19:14:03forever in a video to make it perfect.
19:14:05Um, and then you know, you're going to
19:14:07do the exact same thing. So, in this one
19:14:10right here, I included four. So, I'm
19:14:12going to keep four. Um,
19:14:16let me do the uh, no, I'm just going to
19:14:18do these three. I'm not going to take up
19:14:21more of our time. Um, so we did those.
19:14:25I'm just going to keep these three in
19:14:26for visual purposes. But once you get
19:14:29down here, um, you know what we're going
19:14:32to do is delete some of this, right? So
19:14:34we this is our data exploration and
19:14:37where's our Tableau
19:14:40this is our Tableau right here. So
19:14:42Tableau projects they're separated by
19:14:44these articles. So what we're going to
19:14:45do is go around right here and we're
19:14:47going to go down down down down
19:14:49to right here. This is going to get rid
19:14:51of all these other articles or all these
19:14:53other what they're calling um posts. So
19:14:57we're going to get rid of those and
19:14:58we're going to hit save.
19:15:01And now, as you can see, we have our
19:15:03header, we have our first project, and
19:15:06we have our second and our third. I
19:15:08would include those other projects that
19:15:10we've done in here so that it looks
19:15:12good. This is this footer right here. We
19:15:15don't need that cuz we don't have any um
19:15:17anything else in there. So, we're going
19:15:19to get rid of that as well. And now we
19:15:21just have this information. Now,
19:15:24I don't have anything where they can do
19:15:27the name, email, message, or you can
19:15:28keep that in there if you'd like. Um,
19:15:30but I am going to get rid of this. So,
19:15:33we're going to go right here. That's
19:15:35this section. So, don't delete this
19:15:37section. We want that. I'm going to
19:15:38delete this footer section is what
19:15:40they're calling it. And now we have this
19:15:44address, phone, email, social. Um, and
19:15:46I'm going to get to the social in just a
19:15:48second. It's again super easy.
19:15:51But for the address, I just put
19:15:53location. I don't want to give somebody
19:15:54my address or put it on a website
19:15:55anywhere. Um, it's not something I want
19:15:58to do. So, what we're going to do is
19:16:00just put I'm going to put Dallas and
19:16:03Texas. And we can keep it like that. And
19:16:06we'll hit Oops. We'll hit save. And
19:16:09it'll have Dallas, Texas. Um, hate the
19:16:12look of the zeros. 67890. So, we're
19:16:16going to do that. Phone number one, two,
19:16:20three, five, six, 789.
19:16:24And then email. and we'll put
19:16:28Alex thean analyst 95gmail.com.
19:16:34If you have issues with this, um, you
19:16:36can email me, but
19:16:39I'll try I will try to respond to all
19:16:41your emails. I get a lot. Um, so I will
19:16:44do my best. That is my actual email if
19:16:46you are curious. Now, um, now that we
19:16:49have this, we also have these this
19:16:51social media. Now I want to display my
19:16:55LinkedIn and I also want to display my
19:16:59GitHub. So what I'm going to do right
19:17:01here is I'm going to go over here and do
19:17:02LinkedIn.
19:17:05Perfect. Let's go to this. So I'm going
19:17:09to take my LinkedIn URL
19:17:13and I am going to get rid of these first
19:17:16two because I'm only going to include
19:17:19two. And for this one, I'm going to do
19:17:23uh LinkedIn.
19:17:25Oops. LinkedIn in. And then for right
19:17:29here, I'm going to replace that with
19:17:31linked in.
19:17:34And what you're going to do is put this
19:17:37link right here. And then we're going to
19:17:39go get do get the GitHub.
19:17:42So, let's do GitHub. Oh, who's is this
19:17:45sign up? What is going on? Um,
19:17:49I don't there. Let's just go back here.
19:17:52That was something I was like viewing a
19:17:54while back or something. Um, so we're
19:17:56going to take the GitHub and we're going
19:17:58to put that right here.
19:18:01So, it already has it as um the GitHub.
19:18:05Is this supposed to be lowercase?
19:18:08I think it is. Let me see if this is
19:18:10lowercased as well. Yeah. Um, so do it
19:18:13like that. Do it lowercased. Um, I
19:18:15forgot that that was how they did it.
19:18:18Um,
19:18:19and oh, that's the label. That doesn't
19:18:21matter as much. But this right here is
19:18:22the class is actually the important part
19:18:24because then when we go back here, there
19:18:27is no LinkedIn image. But when we save
19:18:29it, oops, when we save it, it has the
19:18:33LinkedIn image because it's already a
19:18:34class that was created in this HTML um,
19:18:37template.
19:18:39So, we have that. Um, and let me bring
19:18:42this full screen really quick because
19:18:44there are a few things that we couldn't
19:18:45see in that that screen. These right
19:18:48here are things that we could not see
19:18:51before. Um, and these as well. So, what
19:18:56we can do is we're going to go down
19:18:57here. We're just going to copy these
19:18:59social. We're going to replace them
19:19:00right here so they can have those. And
19:19:02then we're going to get rid of these two
19:19:03right here. And this says this is
19:19:05massively um, and we're going to change
19:19:07that as well. Let's make this full
19:19:09screen for the first time. Feels good.
19:19:12Um, I hate doing split screen, but I do
19:19:14it for you guys. Um, so [clears throat]
19:19:18this is Massively. And we're just going
19:19:20to put we're just going to get rid of
19:19:21these two. This is um it's called the
19:19:24navigator, the the different tabs. We're
19:19:26going to get rid of those two tabs. And
19:19:27then for this, I'm just going to call it
19:19:30projects.
19:19:31And I'll once I once we go back and
19:19:33update all of this, then you will um
19:19:36you'll see those changes.
19:19:38So, let's see. So, we made those
19:19:40changes. Here's our social or the social
19:19:42medias uh social media stuff. We're
19:19:45going to go and copy these two.
19:19:50And we're going to replace all of these
19:19:53with this.
19:19:56Um
19:19:57[clears throat] and let's save that. and
19:20:00let's go back. So now, as you can see,
19:20:02those two are gone. This says projects.
19:20:04There's only two right here. And if you
19:20:06click on it, it's going to go to my
19:20:09LinkedIn or your LinkedIn when you do
19:20:11it. Um, and this will take you to the
19:20:14GitHub. So, it is all working as
19:20:17intended. This is great. Um, when you
19:20:19scroll down and it says massively, we
19:20:21can change that as well, and we should.
19:20:23Let's do that really quick. Um, we'll
19:20:26just say
19:20:28Alex the analyst
19:20:30and we'll update that.
19:20:33And there we go. So, in a nutshell, this
19:20:37is all the a lot of it. Um, we need
19:20:40images.
19:20:41And I don't think I set this up for this
19:20:44video. So, I'm going to I'm going to
19:20:47like cut myself off for like two
19:20:49seconds. Go pull those images in um
19:20:51because it could take like a few
19:20:53minutes. I don't want to waste your
19:20:54time. And then I'll come back. So, I'll
19:20:55see you in two seconds. All right, so I
19:20:57just pulled over the images that we are
19:20:59going to use. Let's go to the downloads.
19:21:02Um, they're right here. They're the
19:21:03housing, Tableau, and COVID. Um, if I
19:21:06open up this COVID one, this is what the
19:21:08image looks like. This is what we're
19:21:09going to use for that COVID project. So,
19:21:12I'm going to copy these. I'm going to go
19:21:13into the port website um that we just
19:21:16had. I'm going to go to images and I'm
19:21:18going to insert these in here.
19:21:20So, now that we have those images in
19:21:22here, let's go back
19:21:25and let's see what we got. So, we just
19:21:28put these images in this um you'll have
19:21:31this folder right here. And you can open
19:21:34it up and you can see all of these that
19:21:36we have. So, all we're going to do is go
19:21:38and replace the images, these these, you
19:21:41know, um temporary images that they had
19:21:43for us, and we should be golden. and
19:21:47then we're going to actually upload it
19:21:48to to GitHub and then create our website
19:21:51for free. So, let's go right down here.
19:21:54This is our very first uh one. This is
19:21:57our data cleaning in SQL. This is with
19:21:59the housing data. So, this image right
19:22:02over here, it says images/pick01.jpeg.
19:22:06So, uh JPEG, I don't know why I said it
19:22:08like that. So, this is the housing. So,
19:22:10what we're going to do right here is do
19:22:12housing and it'll autocomplete for us.
19:22:14Um, so that housing should be in there.
19:22:16Now, next one is the data exploration in
19:22:19SQL. That was with the COVID. So, we're
19:22:22going to get rid of this. Want to say
19:22:23COVID. Um, because that is the image
19:22:26that I have right over here. And then
19:22:28the last one is, excuse me, Tableau. So,
19:22:31let's go right over here. Let's do
19:22:33Tableau.
19:22:36Let's get rid. Oh, I got to save that.
19:22:38Uh, control S. Perfect. And now, let's
19:22:43look at it.
19:22:45There you go. There you go. Oh, this one
19:22:47still says full story. Go change that.
19:22:49Um, I'm gonna go change it. Just doesn't
19:22:51feel right.
19:22:53Uh, view project. That's not useful.
19:22:59Okay. [cough and clears throat] Crl S.
19:23:02Perfect. Okay. So, now this looks a lot
19:23:06better. Um, and when we host it um
19:23:09through GitHub pages or github.io, Oh,
19:23:11this is going to be what it looks like.
19:23:14I mean, it is. And you can add a lot
19:23:16more to it. You can take away from it.
19:23:18You can add as many projects as you
19:23:20want. You can keep adding. You can copy
19:23:21those articles or those posts and you
19:23:23can just keep adding them. Um, so this
19:23:27is kind of what it's going to look like.
19:23:30And
19:23:32it was not that hard. I don't think I
19:23:34hope this was not too difficult. I
19:23:35really don't think it is. um it's really
19:23:37just using a template and kind of
19:23:38understanding a little basics of HTML.
19:23:41So um we are going to take this and we
19:23:44we have this saved already. We have this
19:23:46all saved.
19:23:48What we are going to do now is upload
19:23:51this to GitHub. So let's go right over
19:23:54here. Let's go to here and let's go to
19:23:58repositories
19:24:00and how do where where's the new one?
19:24:03Oh, I need to sign in. Okay, I'm going
19:24:06to get rid of this part so you can't see
19:24:07it. So, we are going to say a new
19:24:10repository.
19:24:11We're going to call it Alex theanalyst
19:24:152.github.io.
19:24:19So, we're going to write it just like
19:24:20that. You know, if your name's um Alex
19:24:25Jimmy, I don't know why I said Jimmy.
19:24:28Alex Jimmy, Alex Jimmy.github.io,
19:24:31um you can always go back after the fact
19:24:33and change this. So, it's not a big deal
19:24:35whether you change it or not. And we're
19:24:38going to create this repository.
19:24:41We're going to say upload an existing
19:24:43file.
19:24:45And instead of choosing them, what we're
19:24:47going to do is just go right over here,
19:24:50go to this, and we're just going to copy
19:24:51this in. Or not copy it in, but drag it
19:24:53in. Okay. So, we're going to take this,
19:24:56drag it in right here. And it can take a
19:24:57it'll take a little bit. Says 75, but it
19:25:00shouldn't take [clears throat] that
19:25:01long.
19:25:05And let's just wait for it. I was taking
19:25:08a sip of water. I apologize.
19:25:10But it is literally uploading just
19:25:12everything that we had in there. So all
19:25:13the updates and all the changes and all
19:25:14the stuff that we um had. And it looks
19:25:17like it's done. So let's just write
19:25:20initial [snorts] commit.
19:25:23Commit changes.
19:25:25It is processing it.
19:25:28All right. And it should be done very
19:25:31very soon as long as I have a good
19:25:33internet connection.
19:25:36We shall see.
19:25:40[cough and clears throat]
19:25:41Stick with me. It's taking its time.
19:25:45Um, while it's loading, let's go over to
19:25:48Oh. Oh, there it is. So, perfect. So,
19:25:50here's everything that we have. Has this
19:25:52read me that it generated. Let's go over
19:25:54to settings.
19:25:56And we have this u github.io. io. And if
19:26:02we go right down here to GitHub pages,
19:26:04pages settings now has its own dedicated
19:26:07tab. Let's check it out here. So, it is
19:26:12um
19:26:14it's currently disabled, but we're going
19:26:15to say we want it to do pull from the
19:26:17main. Um I think it's the docs. We'll
19:26:20see. I'm going to save this. Your site
19:26:22is ready to be published. Let's open
19:26:25this up. Okay. Site not found. Maybe
19:26:28it's from the root. Save.
19:26:33Um, your site is having a build a
19:26:35problem. Let me see if I can actually
19:26:37change the name. I already have an Alex
19:26:40[clears throat] analyst, but I'm going
19:26:41to see it's already taken. Um, I'm just
19:26:43going to try this one one more time. Oh,
19:26:46and now it's working. Uh, I have no idea
19:26:49why it uh didn't work before, but this
19:26:52is fantastic. It was giving me all this.
19:26:55I was maybe I was just reading too much
19:26:56into that. I had I had never tried to
19:26:58create another umio
19:27:02or or GitHub pages on this. So anyways,
19:27:05thanks for sticking with me through all
19:27:06that um stuff. So now we have our actual
19:27:11website. Um it doesn't look the same up
19:27:14here because of that thing that we were
19:27:16just looking at. It should just be this
19:27:19part right here. But um this is an
19:27:21actual website now and it's being ho
19:27:23hosted through GitHub and it's
19:27:25completely free. If you want to pay you
19:27:29can hide this from your GitHub. Um your
19:27:32repository has to be public. Uh
19:27:34something I didn't mention when you're
19:27:36doing this your repository has to be
19:27:39public. Um if I change the visibility to
19:27:42private um you will not be able to see
19:27:46it anymore. You'll have to then pay if
19:27:48you want to make this repository
19:27:49private. you have to then pay. I think
19:27:50it's like $4 a month or something like
19:27:52that. So, worth looking into. Um, if you
19:27:56don't want to display that on your
19:27:57GitHub, worth looking into, but this is
19:28:01our final product. I mean, it looks
19:28:03pretty fantastic. And you can use any of
19:28:05these templates, right? There are lots
19:28:07of different templates that are
19:28:08fantastic. I mean, they look amazing.
19:28:12They look professional. Um, it's really
19:28:14up to your style. like this one looks
19:28:16kind of cool, a little bit um edgy for
19:28:18for my taste, but uh this one looks
19:28:21really good, too. Might may be able to
19:28:23add some more narrative to that one. So,
19:28:25again, go through it, make your make a
19:28:28good choice in it, and then update it
19:28:30how we updated it. Uh, I will include
19:28:34the um, let's see. I will include
19:28:38everything that's in here and I'll keep
19:28:40this on my on this GitHub so that you
19:28:43can go in there and if you want to
19:28:44download these images, you can download
19:28:45the images that I used. Um, or you can
19:28:48go find your own. Just um, you know,
19:28:49look for try to get like HD images on
19:28:53Google. Just type in Google images and
19:28:54search for whatever image you want to
19:28:56search. Try to get an HD image. With
19:28:58that being said, that is the entire
19:29:00project. I I I I hope this didn't go too
19:29:02long. Um this may have gone, you know,
19:29:05this may have gone like 30 45 minutes,
19:29:08but in the end of it, at the at the end,
19:29:10which is where we are now, we have an
19:29:12entire website. It was completely free.
19:29:15And I hope that you can now host the
19:29:16projects and you can create create more
19:29:18projects. I will be coming out with more
19:29:20projects myself that hopefully will be
19:29:22interesting to you in the future. So,
19:29:25with that being said, thank you guys for
19:29:27joining me. for you who stuck it out to
19:29:28the very end. You are fantastic. You
19:29:31know, send me a post to your website on
19:29:33LinkedIn and tag me in it because I love
19:29:35seeing um you guys do these projects and
19:29:38this stuff. So, I'm super excited to see
19:29:40all of these um that you guys tag me on
19:29:42on LinkedIn and whatnot. So, with that
19:29:44being said, this is it. I hope you
19:29:46learned something. I hope that it worked
19:29:48for you and I appreciate you watching.
19:29:51Be sure to like and subscribe below and
19:29:54I will see you in the next video.
19:29:55[music] Goodbye.
19:30:08What's going on everybody? Welcome back
19:30:09to another video. Today I'm going to
19:30:11help you create a data analyst resume.
19:30:19Now, when I say data analyst resume,
19:30:21it's not that much different than a
19:30:23regular resume, except that it's going
19:30:25to be catered for a data analyst job. In
19:30:27just a second, we're going to take a
19:30:28look on my screen at a sample resume.
19:30:30I'll have the template in the
19:30:32description so you can just go and
19:30:33download it and fill in your
19:30:34information, but it's a fantastic
19:30:36starting place to actually creating your
19:30:37resume. When we're looking at this
19:30:39resume, we'll take a look at each
19:30:40section and kind of dissect each part of
19:30:42it. And then at the very end, I'll give
19:30:43some extra tips on what you should
19:30:45include and how to actually write your
19:30:46resume as well. So, without further ado,
19:30:48let's jump on my screen, take a look at
19:30:50the resume, and see how you can create
19:30:51your own data analyst resume. So, here's
19:30:53our sample resume. I'm just going to
19:30:55walk through the entire thing super
19:30:56quick, and then we'll break down each
19:30:58section individually. I'll give my
19:30:59thoughts and some tips on each section.
19:31:02And remember, you can download this
19:31:03exact thing in the description below.
19:31:05I'll have a link. I'll probably put it
19:31:06on my GitHub or somewhere else, but
19:31:08it'll be free to download. Uh, so you
19:31:10can go ahead and do that. But let's zoom
19:31:12in just a little bit. So at the very top
19:31:14we have our header. We have some just
19:31:17basic uh contact information. Then we
19:31:20have skills. Then we have projects. And
19:31:22notice the projects are up here at the
19:31:24top. And we'll get to that later about
19:31:25the order of where you should be putting
19:31:27your things. Then we have work
19:31:29experience. And then we have education.
19:31:31So really quickly, I'm going to zoom out
19:31:34and I hope you can still see it. The
19:31:37order is actually quite important. Now,
19:31:39there is one piece that is not in here
19:31:41right now, and that is a summary
19:31:43section. I don't have a summary section
19:31:45on my real resume. I just I don't think
19:31:48it's useful or helpful. I don't have
19:31:50one. You can include one, and it would
19:31:52be right up here at the very top. Now,
19:31:54why do we have the skills and projects
19:31:56at the top? Well, it's because that most
19:32:00people who are trying to break into data
19:32:02analytics don't have any experience in
19:32:04data analytics. If I am reading this
19:32:07resume as a hiring manager and the first
19:32:09thing that I look up here and I see is
19:32:11experience and it's not analyst, it's a
19:32:14teacher or a nurse or something, I'm
19:32:15going to be like, uh, this person
19:32:17doesn't have any experience, I don't
19:32:18want to hire them. The first thing that
19:32:20you want to have on your resume is
19:32:21something that is good for the hiring
19:32:23manager to see. The first several things
19:32:25you should put all your best stuff at
19:32:26the top. That's my uh what I believe.
19:32:29So, I think that these skills are really
19:32:32strong, a lot of great skills. And then
19:32:34these projects are all really good
19:32:36projects. Now, this is just a sample.
19:32:37These aren't all real projects. Um, or
19:32:40they are real real projects. They're
19:32:41just not, you know, ones that I built
19:32:43myself. It's just a sample. So, uh, then
19:32:47right here we have our work experience.
19:32:49Now, if you're, like I said, a nurse or
19:32:50a teacher or a lawyer or something
19:32:52that's not relevant to data analytics,
19:32:53you want that at the bottom. Um, and
19:32:55then you're going to want to tie in, uh,
19:32:57some things in these descriptions. And
19:32:58then the education at the bottom. My
19:33:00education was terrible. Okay, I had a
19:33:02bachelor's in recreational therapy which
19:33:05had nothing to do with data analytics.
19:33:07So for a tech job has was not good. I
19:33:09always had mine at the bottom. So let's
19:33:12start at the very top and walk through
19:33:14each section. So at the very top you
19:33:18want to have maybe a title, but for sure
19:33:20your full name. You definitely want to
19:33:23include your phone number if you're okay
19:33:24with them calling you, but definitely an
19:33:26email. for sure include things like a
19:33:29LinkedIn profile or a GitHub profile.
19:33:31You can also put your portfolio. In
19:33:33fact, I highly recommend putting your
19:33:34portfolio because it just looks good or
19:33:36if they check it out, that's a really
19:33:38good thing. And then your location
19:33:40because sometimes your job is going to
19:33:41be location-based, whether you're in
19:33:43Dallas or another metropolitan city.
19:33:45It's just nice to have that on there.
19:33:47This should be the simplest one to fill
19:33:49out unless you haven't built out
19:33:51something like a portfolio, you just
19:33:52don't include it. Um, but this one
19:33:54should be the simplest one, right?
19:33:56you're just putting contact information,
19:33:58maybe a link to a website. Next, we have
19:34:00the skill section. And this one on my
19:34:03own personal resume I have at the very
19:34:04top. I typically recommend anyone who
19:34:07does not have experience, who is trying
19:34:08to break into data analytics to put this
19:34:10at the top as well and have these skills
19:34:13and know these skills. That's important.
19:34:15Um, but when the hiring manager first
19:34:17initially sees this, there's just going
19:34:18to be a mental check. Okay, they have
19:34:20the skills that we're looking for. Let's
19:34:22move on to the rest of the resume. Um,
19:34:24but you want as many mental checks for
19:34:26what they're looking for at the
19:34:28beginning. Just going to I'm going to
19:34:30keep repeating that. Um, this is how I
19:34:33personally write my skills. So, I write
19:34:36something like SQL and then I'll say SQL
19:34:38server, my SQL, Postgrade SQL. Now, I
19:34:40have used all these different types of
19:34:42SQL in my actual job. If you don't you
19:34:45haven't done that and you're just
19:34:46starting out maybe you put something
19:34:48like um you know sub queries store
19:34:52procedures joins whatever the actual
19:34:54things within SQL I don't really think I
19:34:57don't recommend that as much because
19:35:00typically people know what SQL is like
19:35:02if they use SQL they know what SQL is so
19:35:04they're just going to expect that you
19:35:05know those things now for something like
19:35:07Python it's different because there are
19:35:09packages something like R there are
19:35:10packages and libraries within them so
19:35:12you can specify I I have worked with
19:35:15pandas in my actual job and I look for
19:35:17people who know pandas as well because
19:35:18you know we use it. So actually
19:35:20specifying these packages or libraries
19:35:23is really helpful. So this is how I
19:35:25would put these things on a resume. Now
19:35:28this is another resume. This is our
19:35:30sample two. I'm going to maybe include
19:35:32this one down below. Although I don't
19:35:34like this format as much but if you like
19:35:36it you can. But here's another way that
19:35:38you can um show these skills. Just a
19:35:41different way to do it. I want to show
19:35:42you both ways. Um, where you have like
19:35:44Python and the libraries underneath it.
19:35:46I've even seen it to where people will
19:35:47write out almost like um, let me go down
19:35:50here. They'll write out like a
19:35:51narrative. Um, they'll do Python
19:35:54and then they'll have like a colon and
19:35:56then they'll say use to um, manipulate
19:36:02data and I'm not spelling that right in
19:36:04pandas dot dot dot and they write it
19:36:06out. You can do that as well. Again, I
19:36:08like bullet points because it's to the
19:36:10point. That's exactly what you need.
19:36:12Let's get rid of this one real quick.
19:36:14So, this is the one uh that I like. So,
19:36:18that's the skills section. Let's move
19:36:20down to the projects. Now, the project
19:36:23section is almost primarily for people
19:36:26who are just starting out. Once you get
19:36:28experience, typically you maybe have one
19:36:30project on there or no projects at all.
19:36:33But the project section is used as kind
19:36:35of um in lie of actual experience,
19:36:38right? I've always said that you need to
19:36:41build projects not just for your resume
19:36:43but also for the interviews. So then
19:36:45when you get into an interview you can
19:36:47point to these projects and say yes I've
19:36:49used SQL I did it in this project and
19:36:51they may have seen it and you can walk
19:36:53them through how you actually used it.
19:36:55It gives you more credibility than just
19:36:57saying you know how to use SQL. So
19:36:59within the project section we're going
19:37:02to have a project. This one says data
19:37:04science job market exploratory data
19:37:06analysis. So this is a personal project
19:37:09and then within it they did some really
19:37:11great stuff. Here's usually what I
19:37:14recommend and this is in here which is
19:37:16you specify what you did. You say I used
19:37:18Python and what did you do to analyze
19:37:21this and gain insights in the job
19:37:23market. Then you walk through some of
19:37:25the things that you actually did things
19:37:26like regex techniques. You used pandas
19:37:29mapplot lib. You built a wordcloud.
19:37:31These are keywords that somebody will
19:37:34look for and they even highlighted them
19:37:36which I personally like and do as
19:37:38myself. They highlighted these things so
19:37:41that the viewer or the um hiring manager
19:37:43is actually seeing them making sure that
19:37:45they're bold so that they are catching
19:37:46their eye. So I personally do this and I
19:37:49recommend this. That's all it needs to
19:37:51be. It just needs to be I built a
19:37:53Tableau dashboard doing this from this
19:37:56data set. I cleaned it in SQL and you
19:37:58show those skills. Something that's
19:38:00important in both the skills section and
19:38:03the project section is using and
19:38:05highlighting your skills as much as
19:38:08possible. Especially if you don't have
19:38:10any experience, if you've never had a
19:38:12job before. Once you have a job and you
19:38:14come down to like the work experience,
19:38:16then it kind of speaks for you. But if
19:38:18you don't, you want the projects and the
19:38:20skills to speak towards your skills and
19:38:22credibility. So, we have this right
19:38:24here. Now, one thing that's not in here
19:38:26that I actually do recommend is a
19:38:28hyperlink. maybe right here or actually
19:38:32this being a hyperlink to the project
19:38:35because they might read this and be like
19:38:36I we work with you know data science job
19:38:39market data I don't know and then
19:38:42they'll click on this link and they can
19:38:43see your work that is the one thing that
19:38:45I would change in this other than that
19:38:47this is exactly how I would have it very
19:38:49very very similar to my own um and a lot
19:38:51of this that I did I actually took from
19:38:54other résumés and formatted it how I
19:38:56prefer and like it um so again some of
19:38:58this personal preference and you can
19:38:59change it however you want. That's just
19:39:01how I like it. So, that is the project
19:39:04section. Now, we're going to go down to
19:39:05the work experience section. Now, this
19:39:07person does have a little bit of analyst
19:39:11uh experience. So, you know, if you
19:39:14don't, that's okay, but you put your
19:39:17previous experience. Now, here's what I
19:39:19recommend. If you've been a teacher for
19:39:2115 years, you've been a nurse for 10
19:39:23years, you've had 10 different jobs,
19:39:24don't put all your experience on here.
19:39:26Um, maybe put your last two jobs going
19:39:29back maybe three years. I don't
19:39:31recommend you filling it up because it's
19:39:33not going to be super relevant. Unless
19:39:34you're applying for a healthcare data
19:39:36analyst position and you have a nursing
19:39:38degree, then it's relevant and that
19:39:40experience is super helpful because it's
19:39:41domain experience, right? Then you may
19:39:43go back five years. Just, you know, use
19:39:46your discretion. But what do you need to
19:39:47include? Of course, your title, where
19:39:50you worked, your location, and the
19:39:52times. That's standard for almost any
19:39:54resume. But within here, uh, what you
19:39:57really want to do is highlight again the
19:39:58skills if you can. If you can't, that'll
19:40:01change. But in here, he says,
19:40:03"Implemented a new reporting using Excel
19:40:05Pivot and VBA, which reduced processing
19:40:08time by 50%." These types of um
19:40:11quantitative information. I reduced
19:40:13time. I I I saved the company money. I I
19:40:16did something quantitative. Putting that
19:40:19in here is always helpful. always highly
19:40:21recommended, although it can be tough to
19:40:23measure these things, right? Typically,
19:40:25what I recommend, especially if you're
19:40:26first starting out, is to highlight
19:40:28skills. If you're a teacher, you've
19:40:30probably used Excel and you've probably
19:40:32used Excel for closer to data analytics
19:40:34than you'd think, just in a teacher way
19:40:36and not a data analytics way. But you
19:40:39can reward these things and make them
19:40:41sound good. If you are a a nurse, like I
19:40:43was saying, you've used Excel, you've
19:40:46used a health information system, you've
19:40:49used uh some type of database, talk to
19:40:52that. Include that in here. Um, and it
19:40:54can be hard to write these out. And I'm
19:40:57going to show you a way in just a little
19:40:58bit about how you can write these out
19:41:00and think about these things or have a
19:41:01way to help you write them or give you
19:41:03ideas. We'll get to that in a second.
19:41:06Lastly, we have the education piece.
19:41:07This is again really simple. at the very
19:41:10bottom, education, what your degree was,
19:41:12where you went. Um, and if you have, you
19:41:14know, some helpful things to include,
19:41:16you can do that, and then when you
19:41:18actually went. Now, you can include
19:41:20other things in here as well, like boot
19:41:22camps, if you went to a boot camp, or
19:41:24you could also include things like a
19:41:25GPA. Although, I don't personally
19:41:27recommend it. GPA has never been
19:41:29anything that I've ever cared about or
19:41:31I've seen anyone care about, ever. Um,
19:41:34so you don't normally have to include
19:41:35it. One other thing that you can include
19:41:37at the very bottom is something like
19:41:40certifications. Uh I personally don't
19:41:42put a lot of stock in certifications
19:41:44unless it is one that I have recommended
19:41:46in previous video like the Tableau
19:41:48certification or Tableau desktop
19:41:50certification. If you're applying to a
19:41:51job that uses Tableau that actually
19:41:54could be really good. So definitely
19:41:56include that. But one's on Udemy, one's
19:41:58on Corsera, or like my Alex the analyst
19:42:02boot camp that I have on my channel. I
19:42:04wouldn't really include that in your
19:42:06resume. It's mostly for learning. If you
19:42:08get something like the Tableau one or
19:42:10the AWS uh cloud one or the um Azure
19:42:13cloud one, those are all actual
19:42:15certifications that can help you and
19:42:16give you credibility towards a certain
19:42:18skill. Now, really quickly, let's just
19:42:19take a glance at the other resume. This
19:42:21is ré 2. So, we have the education at
19:42:24the top. doesn't have to be at the top
19:42:26unless it's relevant which you could put
19:42:28at the top. We have a skills section.
19:42:30They again this is the project same
19:42:31projects and then work experience. So
19:42:33this is just a little bit different um
19:42:35order. So you can do it like this as
19:42:37well and different way you can write the
19:42:39skills and you can also include a
19:42:41summary section as well. So that's the
19:42:43meat and potatoes of how I would create
19:42:45a data analyst resume. Now writing it is
19:42:48actually a different beast, right? You
19:42:49have to actually write it out, get
19:42:51something on the resume and then apply
19:42:53using that resume. But it can be hard to
19:42:55come up with these ideas. So, uh, I just
19:42:58want to show you something that a lot of
19:42:59people have been using. I personally
19:43:01haven't written a resume in a little
19:43:03while. So, I don't use it for my own
19:43:05resume or haven't used it, but I will.
19:43:07Um, and that's using chat GBT or some
19:43:09variation, whether it's on Bing or, you
19:43:11know, you get some different version or
19:43:13some new product that's out there at the
19:43:14moment. I'm just going to show you how
19:43:16to do it in chat GBT. Some of the things
19:43:17that you can prompt it to do, and
19:43:19that'll be it. I'm just going to show
19:43:21you kind of some ideas that it can
19:43:22generate for you to help you write these
19:43:24things. All right. All right. So, here
19:43:25on my screen, we're on Chad GBT. If you
19:43:27haven't used it, I'll leave a link in
19:43:29the description. I also have a whole
19:43:30video on how to use Chad GBT for a data
19:43:32analysis. Um, so I like Chad GBT. Now,
19:43:35I've already written out these questions
19:43:37because I don't want to wait for the
19:43:38responses. But here's what I asked it to
19:43:40do, and you can do some variation of
19:43:42this, whether you're a nurse or a lawyer
19:43:44or a teacher, whatever. I said, I'm a
19:43:47math high school teacher trying to
19:43:48become a data analyst. How can I use my
19:43:50experience on my resume to help me get a
19:43:52job? This is just to help provoke some
19:43:55ideas and it says, you know, you most
19:43:57likely have some skills. Emphasize your
19:43:59quantitative skills. So, those are some
19:44:01of the things you can focus on. Showcase
19:44:02your ability to commute complex
19:44:04concepts, which is really important in
19:44:06data analytics, being able to present
19:44:07information, which teachers have.
19:44:10Highlight your experience with
19:44:11technology. Hopefully, you're using some
19:44:13type of uh, you know, database for
19:44:15students or, you know, Excel or
19:44:17something like that. And you can
19:44:18highlight that and showcase your ability
19:44:20to solve problems. Now, the next thing
19:44:22that I asked it was, I built a COVID
19:44:24Tableau dashboard using Tableau. How can
19:44:28I add this to my resume? And then it's
19:44:30going to tell you exactly how you can do
19:44:32that. It's going to say, include the
19:44:33link to your dashboard, which I also
19:44:35recommend. Provide a brief description,
19:44:37highlight your data visualization
19:44:38skills, include screenshots or images,
19:44:40which that's what I would be putting in
19:44:42the project itself, not on your resume.
19:44:44Then provide context for the data. All
19:44:47really good stuff. Really great. Now,
19:44:49the last thing is kind of what I'm
19:44:50trying to get at as a whole. It can help
19:44:53you write things. So, I'm going to say
19:44:55write a two sent I said write a two.
19:44:57Write two sentences highlighting my
19:44:59COVID tablet dashboard to add to my
19:45:01resume. And it's going to say developed
19:45:03a COVID tablet dashboard to visualize
19:45:05pandemic trends using real-time data
19:45:07sources demonstrating strong data
19:45:09visualization and analysis skills. So,
19:45:12this can help you generate those
19:45:14descriptions in your work experience.
19:45:16that can help you generate the
19:45:17descriptions in your projects. And this
19:45:20can be really helpful to just generate
19:45:21some ideas because I personally really
19:45:23struggle with like highlighting my
19:45:25skills and descriptions within those
19:45:27things. This can be a way to kind of
19:45:30help you do that. So don't, you know,
19:45:32just copy and paste, but let it prompt
19:45:34you. Let it give you ideas. Now, the
19:45:36last thing that I want to mention is
19:45:37just your overall resume as a whole. The
19:45:40template that I use, the template that I
19:45:42recommend is very, very friendly to
19:45:45these automated systems that check your
19:45:47resume. If you did not know, most
19:45:50companies, especially big companies, use
19:45:52these automated systems that scan your
19:45:54resume, see if it has what they're
19:45:56looking for, and then that resume, if it
19:45:58gets through that system, gets passed on
19:46:00to a recruiter or hiring manager.
19:46:02Typically, most companies don't go
19:46:04straight to the hiring manager. So, you
19:46:06need a resume that can pass through
19:46:08those initial systems and pass those
19:46:10tests. The résumés that I've shown you
19:46:12today will do that. They have bullet
19:46:14points. They have the keywords. They
19:46:15have everything you need. That's why I
19:46:17recommend or partially why I recommend
19:46:19this type of resume. Other ones that
19:46:21have images and different fonts and
19:46:23different stylings can cause issues with
19:46:26these automated systems where it just
19:46:28doesn't read it properly or, you know,
19:46:30it doesn't read the right words that you
19:46:32want it to read. So, just know that
19:46:35these types of résumés have different
19:46:37uses, right? You're not just handing it
19:46:39off to somebody to where they can read
19:46:40it and it's needs to be visually
19:46:42stimulating. Really, what you need is
19:46:44you need it to get through those initial
19:46:46systems, which these résumés, uh, if you
19:46:48write them well, you have good, you
19:46:50know, skills and the right things on
19:46:51your resume. They will pass through that
19:46:53first layer to get to those hiring
19:46:55managers. So, again, be sure to download
19:46:57those. Those are completely free. I just
19:46:59I highly recommend using them. I think
19:47:00they're really good. So, be sure to
19:47:02download those, use those, just put in
19:47:04your own information. Be sure to build
19:47:06out your own projects. Don't just keep
19:47:08the ones that are on there because
19:47:09you'll need to be able to speak to them.
19:47:11Sometimes recruiters or hiring managers
19:47:12are going to ask you about them, how you
19:47:14built it, what you did, and you can also
19:47:16point to those projects in your actual
19:47:18interview. So, I hope that this was
19:47:20helpful. I hope that your resume is
19:47:22ready to go. I hope that you're ready to
19:47:24start applying for those data analyst
19:47:25jobs. Thank you guys so much for
19:47:27watching. I really appreciate it. If you
19:47:29like this video, be sure to like and
19:47:30subscribe below, and I'll see you in the
19:47:32next video.
19:47:34[music]
19:47:45What's going on everybody? Welcome back
19:47:46to another video. Today we are going to
19:47:48be solving easy SQL technical interview
19:47:50questions.
19:47:56Now, when I was interviewing for data
19:47:58analyst positions, I almost always got
19:48:00some type of technical interview. And
19:48:02the vast majority of the technical
19:48:03interviews that I got were in SQL. Not
19:48:05only that, but I was also on a hiring
19:48:07team. And then later as a hiring
19:48:08manager, I almost always conducted some
19:48:10type of SQL technical interview. Now,
19:48:12why do hiring managers conduct these
19:48:14interviews? It's because they want to
19:48:15make sure that you actually know the
19:48:16skill. Because you can just put SQL on
19:48:18your resume and not actually know it at
19:48:20all. And then when they hire you, they
19:48:21have to spend 2, three, four months
19:48:23training you on the basics of SQL for
19:48:25you to actually understand it and use
19:48:27it. That is not what any hiring manager
19:48:28wants. And so that's why they conduct
19:48:30these SQL technical interviews. So in
19:48:32this series, we're going to start with
19:48:33easy questions. That'll be in today's
19:48:34video. Then we'll go on to medium, hard,
19:48:36and then very hard SQL technical
19:48:38interview questions. So without further
19:48:40ado, let's jump on my screen and take a
19:48:41look at the easy SQL interview
19:48:43questions. We're going to be practicing
19:48:44these questions on analystbuilder.com.
19:48:46And if we come right over here, we can
19:48:48filter to the free questions and we can
19:48:50filter to the easy questions. These are
19:48:53all the free and easy questions that you
19:48:54can go and take right now. I will leave
19:48:56a link in the description. I will also
19:48:58leave a link to the two questions that
19:48:59we're going to be looking at today, but
19:49:01go ahead and check out this if you want
19:49:03to try out all of these different
19:49:05questions. There's also different
19:49:06difficulties. So, we're going to be
19:49:08working through the moderate and the
19:49:10hard ones in future videos. And then
19:49:12we'll also be looking at the very
19:49:14difficult ones in the very last video in
19:49:16this series. But let's go ahead and get
19:49:18rid of this because we're going to go
19:49:19over to our very first question. Now,
19:49:21really quickly, I just want to show you
19:49:22the interface before we actually dive
19:49:23into the question. Now, we're going to
19:49:25be solving this in my SQL, but you can
19:49:28also practice in Postgra SQL and
19:49:30Microsoft SQL Server. Whichever one you
19:49:32have an interview coming up for, if that
19:49:34company uses Microsoft SQL Server, come
19:49:36in here and use Microsoft SQL Server.
19:49:38And we also have Python as well. But
19:49:41we're going to be doing this in MySQL.
19:49:43This is where we'll write our actual SQL
19:49:44code. Then we have the prompt. Now, in
19:49:47an interview, typically they're going to
19:49:49give you some type of prompt and then
19:49:50the data. They're going to ask you to do
19:49:52something with the data and our data is
19:49:55right down here. So, this is our data.
19:49:57Now, this is a practicing platform,
19:49:59right? To practice for technical
19:50:01interview questions or practice for
19:50:03technical interviews. So, we also have
19:50:05hints and expected output, as well as a
19:50:07video explanation walking through this
19:50:09question, showing you exactly how to do
19:50:11it. Um, but if you can't get it at all,
19:50:14you can always come in here and look at
19:50:16the solution for any of these. But let's
19:50:18go ahead and start this question. The
19:50:21question is called car failure. It says,
19:50:23"Cars need to be inspected every year in
19:50:25order to pass inspection and be street
19:50:28legal. If a car has any critical issues,
19:50:30it will fail inspection. Or if it has
19:50:33more than three minor issues, it will
19:50:35also fail. Write a query to identify the
19:50:37cars that passed inspection. Output
19:50:39should include the owner name and
19:50:42vehicle name ordered by the owner name
19:50:44alphabetically.
19:50:45Let's go take a look at this data. So we
19:50:48have the owner name, vehicle name, minor
19:50:50issues, critical issues, and that's all
19:50:53we have. So it's just a simple one. Um,
19:50:56again, this is an easy question at least
19:50:59on Analyst Builder. And so, let's try to
19:51:02solve this without any of the hints or
19:51:04looking at the expected output because I
19:51:07think we can solve this one. Now, the
19:51:09first thing that we need to take into
19:51:10consideration is we're going to need to
19:51:11be filtering. So, we'll need to filter
19:51:14down some data. So, if a car has any
19:51:16critical issues, it'll fail. So, if
19:51:18critical issues is 1 2 3, it's going to
19:51:22fail. So, it can't have uh no critical
19:51:26issues.
19:51:27And then I'll say and because even
19:51:30though this says or right here, it
19:51:32doesn't mean one or the other. It means
19:51:34if it has this or it has this, it fails.
19:51:36So, we actually need an and here. So,
19:51:38both conditions need to be met or more
19:51:41than three minor issues.
19:51:45So, you can't have either of those. Now,
19:51:48in our output for what we're going to
19:51:50get down here, we need to have just two
19:51:54things. We just need the owner name and
19:51:56vehicle name. So, let's put that owner
19:51:58name and vehicle name. And then lastly,
19:52:02we just have to order by the owner name.
19:52:07And that's going to be ascending ASC.
19:52:10Ascending is just A to Z. So, I think we
19:52:12can go ahead and start writing this out
19:52:15because I think that's all we need to
19:52:16do.
19:52:17Now, let's go ahead and run this. And
19:52:19so, we have our data right down here.
19:52:21Now, here's what we need to do. We are
19:52:23trying to identify cars that passed the
19:52:26inspection. So, we need to filter out
19:52:29the ones that had a critical issue or
19:52:31that had three or more minor issues. So,
19:52:34what we're going to do is we're going to
19:52:35come right over here, and the first one
19:52:37we'll do is critical issues. So, we're
19:52:39going to say where critical
19:52:43issues
19:52:45and we need to say is equal to zero. And
19:52:48let's run this. And it looks like I had
19:52:50a space here. Let's run this again.
19:52:52There we go. So now these are cars that
19:52:56have no critical issues. So they're p
19:52:58these ones are passing. But if we just
19:53:00look at our data right down here, we
19:53:02have some that have four, four, five.
19:53:06So, we also have to say and and I'm
19:53:09going to copy this so I don't have
19:53:11another uh problem with that. So, I'm
19:53:13going to say where it's less than or
19:53:16equal to and I'm going to say three. So,
19:53:18if has three or less, it should be in
19:53:21our output. So, let's run this. And I
19:53:24say greater than. Let me do less than.
19:53:27There we go.
19:53:29And we can see that now the minor issues
19:53:31are all three or less. And let's just
19:53:33make sure or if it has more than three.
19:53:36So if it had four or five, it should not
19:53:38be in our output. So this has 3 2 0 222
19:53:41and this has 0000. So these are cars
19:53:45that should pass. Now right now we've
19:53:48been selecting everything, but what we
19:53:50really need to do is just select the
19:53:52columns that we need. That's going to be
19:53:54owner name and vehicle.
19:53:59So let's run this. And this looks really
19:54:01good. And the last thing that we need to
19:54:03do is order by. So we need to order by
19:54:07the owner name. Let's get that owner
19:54:09name. And we need to do that in
19:54:11ascending. Now by default when you use
19:54:14order by uh to organize and sort a
19:54:17column, it automatically isn't
19:54:18ascending. But I like to at least put it
19:54:21there, you know, explicitly so I can see
19:54:23it. And this to me should be the correct
19:54:26answer. Now, what we need to do to check
19:54:28our answer and actually show if we got
19:54:30the answer right is just click this
19:54:32check answer button. We can also do
19:54:33control uh shift enter. But I'm going to
19:54:36click the uh check answer. And there we
19:54:39go. We got the solution correct. So,
19:54:42let's go ahead and come up here. We're
19:54:44going to go to our next question. This
19:54:46is another easy question on analyst
19:54:48builder. Again, I'm going to leave a
19:54:49link in the description if you want to
19:54:50try out this exact question. But let's
19:54:52take a look at this one. This is called
19:54:54apply discount. It says, "A computer
19:54:56store is offering a 25% discount for all
19:54:59new customers over the age of 65 or
19:55:02customers that spend more than $200 on
19:55:04their first purchase. The owner wants to
19:55:06know how many customers received that
19:55:08discount since they started the
19:55:10promotion. Write a query to see how many
19:55:12customers received that discount." Okay.
19:55:16And let's go down and look at the data.
19:55:18So, we have customer ID, we have their
19:55:21age, and then we have their total
19:55:23purchase. And let's just see if there's
19:55:25any duplicates in this customer ID.
19:55:28It doesn't look like it. It looks like
19:55:30they're all just one purchase. The
19:55:32reason I was looking at that is it says
19:55:34all new customers and it looks like
19:55:36these are all new customers. It doesn't
19:55:38look like there's any repeat customers
19:55:39that are old customers. So, we're going
19:55:41to assume these are all new customers.
19:55:43If there was a multiple 101s, I would
19:55:46look for something like a transaction ID
19:55:48or a transaction date. But those types
19:55:51of questions are a little usually a
19:55:52little bit more difficult um in like the
19:55:54medium hard types of questions. But
19:55:57let's go make some notes real quick. So
19:56:00we have to filter on two different
19:56:01things. They have to meet uh one of
19:56:04these criteria in order to be in the
19:56:06output. They either have to be over the
19:56:08age of 65, so over 65, and let's do over
19:56:1265,
19:56:14or they just have to meet one of these,
19:56:16or have spent more than 200 or spent
19:56:20more than 200. And that's shouldn't be
19:56:24capitalized. Uh, so that's what we need.
19:56:26And then the owner is just wanting to
19:56:27know how many customers received that
19:56:30discount. So, we're going to do a count
19:56:34on the number of customers
19:56:37uh that fit that filter. I'll write it
19:56:40like that. So, it's just going to be a
19:56:42number in our output. This one should be
19:56:44uh I would say even a little bit simpler
19:56:46than the last one because we're not
19:56:48having to filter or really um do
19:56:51anything like that. Now, let's run this
19:56:54and let's come down here. So we're going
19:56:55to do a count eventually on this
19:56:57customer ID, but what we need to do is
19:56:59filter on both the age and the total
19:57:01purchase. So let's write this out. So
19:57:05age needs to be and I need to write
19:57:07where where age is greater than 65. Now
19:57:12I'm not saying greater than or equal to
19:57:14because right here it says for new
19:57:16customers over the age of 65. If it said
19:57:20age 65 or over, I would say greater than
19:57:23or equal to. I want to be these
19:57:24questions can be very specific
19:57:26and then we'll say and or actually or
19:57:30because it's either of these need to be
19:57:31true or the total purchase and again it
19:57:35says more than 200. So we're going to
19:57:38say greater than 200. Now let's run this
19:57:41and let's just kind of look down here.
19:57:44There's a lot of people that fit this
19:57:45bill. It looks like
19:57:48um yeah the majority of people. That's
19:57:50great. So we're going to give them a
19:57:51discount. These are people that received
19:57:52a discount. So all we're going to do is
19:57:54a count on this customer ID. So let's do
19:57:57account on customer ID. Let's run this.
19:58:01And our answer is 14. All we have to do
19:58:03is check this answer to make sure uh it
19:58:06is correct. So let's hit the check
19:58:08answer. And there we go. Your solution
19:58:11is correct. Now one thing I will say
19:58:13about just technical interviews in
19:58:15general is oftentimes they want to make
19:58:17sure you know how to write it. But even
19:58:19more so, they're really checking to make
19:58:20sure you understand how SQL works. So,
19:58:23when I'm writing this out, if I'm in an
19:58:25actual interview, I would be talking
19:58:27this out loud to the interviewer who's
19:58:29interviewing me. I would say, "Okay, I
19:58:31looks like I need to filter on this and
19:58:33I need to do a count on these things."
19:58:35Now, you only would know how to do that
19:58:38if you know my SQL or if you know SQL,
19:58:40right? Again, they really want to hear
19:58:42your thought process. And so walking
19:58:44through it exactly how I did in the
19:58:46actual interview is exactly what the
19:58:49interviewer wants to hear. Now, writing
19:58:51it correctly is still very important,
19:58:53especially as you get to the more uh
19:58:56difficult questions, right? You want to
19:58:57know certain functions and certain ways
19:58:59to write things, but you should be
19:59:01talking all this out loud during your
19:59:03interview. Now, one other really cool
19:59:04thing, I'm just going to show you this
19:59:05at the end, is if I go over to my
19:59:07profile, I'm actually earning points for
19:59:10these questions. So right down here, I
19:59:12just earned a couple extra points
19:59:14towards my my SQL badge. And I can go
19:59:16all the way up to expert and then master
19:59:18as well. And so I am well on my way. And
19:59:21then in future videos when we do the
19:59:22medium and the hard and then the very
19:59:24hard questions, we earn even more points
19:59:26for those cuz they are more difficult
19:59:27and they go towards these badges. So
19:59:29that is how we solve those easy SQL
19:59:31technical interview questions on Analyst
19:59:33Builder. Go ahead and try those out.
19:59:34There's tons of other free questions on
19:59:36the platform that you can just try out
19:59:38and there's lots of easy ones, but then
19:59:39we're also going to be taking a look at
19:59:40medium, hard, and very hard in future
19:59:43lessons. With that being said, I hope
19:59:44you enjoyed this video. If you did, be
19:59:46sure to like and subscribe below and I
19:59:48will see you in the next video.
20:00:02What's going on everybody? Welcome back
20:00:03to another video. Today we're going to
20:00:05be solving Medium SQL technical
20:00:06interview questions.
20:00:11[music]
20:00:13Now, if you watched [snorts] the first
20:00:14video in this series, you saw that we
20:00:16answered some easy SQL technical
20:00:18interview questions. Now, we're going to
20:00:19be solving medium level questions. The
20:00:21medium questions are going to be a
20:00:22little bit more difficult, but I will
20:00:24say that if you are practicing for a SQL
20:00:26technical interview, I highly recommend
20:00:28practicing the easy and the medium
20:00:30questions. But let's not waste any time.
20:00:32Let's head over my screen and take a
20:00:33look at our medium level questions. But
20:00:35actually before we take a look at our
20:00:36two questions up here, you can come over
20:00:38here, go to the free questions, go to
20:00:40the difficulty, and go to moderate. And
20:00:43these are all the medium level questions
20:00:45that you can try for free on
20:00:47analystbuilder.com. If you've not tried
20:00:48out analyst builder, I highly, highly,
20:00:50highly recommend it. That is my data
20:00:52analytics learning platform that I'm
20:00:53extremely proud of. You can take my full
20:00:55courses and try out these technical
20:00:57interview questions all in one place.
20:00:59But let's head over to our first
20:01:02question which is called tech layoffs.
20:01:04It says tech companies have been laying
20:01:06off employees after a large surge of
20:01:07hires in the past few years. Write a
20:01:10query to determine the percentage of
20:01:11employees that were laid off from each
20:01:13company. Output should include the
20:01:15company and the percentage to two
20:01:17decimal places of laid-off employees.
20:01:19Order by company name alphabetically. Uh
20:01:22and I think this is extremely accurate
20:01:24because uh that just happened. Uh, I'm
20:01:27recording this in late late late late
20:01:302023. Um, but I'm sure I'll release this
20:01:32in 2024 and I think you guys know what
20:01:34I'm talking about. Uh, it was just a bad
20:01:36year for 2022 2023 with all the layoffs.
20:01:39Now, we have Apple, Microsoft, Google,
20:01:42Amazon, Facebook, Tesla, and these are
20:01:46the employees fired. And what we're
20:01:49trying to do is output a percentage. So,
20:01:52we need to look at the company and then
20:01:54percentage of laid-off employees. So, if
20:01:57they had zero laid-off employees, the
20:01:59percentage should be zero. But let's say
20:02:02they had 6,000 of 181,000. We need to
20:02:05see what percentage of the entire
20:02:07company size was laid off. Was it 1% 2%?
20:02:11Um, that's what we're trying to
20:02:13determine. Now, we always can use hints
20:02:16and expected output. if we need help or
20:02:19if we need the video walkthrough, we can
20:02:20use it, but I don't think we'll need it.
20:02:22Let's come over here and make some notes
20:02:24before we get started. Now, before we
20:02:26write anything, remember when you're in
20:02:27a technical interview, whether it's for
20:02:29data analysis, data engineering, data
20:02:31science, it doesn't matter. When you are
20:02:33applying for these jobs, writing it out
20:02:35correctly is important. But I would say
20:02:37even more importantly, it's about how
20:02:39you actually talk through the problem
20:02:41because that really shows your skill
20:02:43level. If you're walking through it and
20:02:44you're just typing random stuff and it
20:02:46kind of makes sense, but you're not
20:02:48talking out loud, they may not
20:02:49understand that you really know what
20:02:51you're talking about. And so, you want
20:02:53to practice these questions, know what
20:02:55you're talking about, and while you're
20:02:56solving them, do what I'm about to do,
20:02:59which is I'm going to kind of talk
20:03:00through the steps that I need to do,
20:03:02then I'm going to write it out. That's
20:03:03what I recommend during actual
20:03:05interviews. So, the first thing that we
20:03:08need is we need to find the percentage.
20:03:10Now, this is going to be a calculation.
20:03:12So I'll write percentage calculation.
20:03:14Now how do we determine what the
20:03:17percentage is? What we need to do is
20:03:19employees fired divided by the company
20:03:22size times 100. So it's employees
20:03:27fired divided by comp size times 100.
20:03:33That's the calculation that we need. So
20:03:36we are going to go and do that in just a
20:03:38little bit. But after that our output
20:03:40needs something. So, our output needs
20:03:43and I need to comment this out. Our
20:03:45output needs the company name and it
20:03:48needs the percentage.
20:03:51So, we're definitely going to need to
20:03:52include both of those. And then lastly,
20:03:55we need to order by the company name as
20:04:00C, which means ascending. So, A to Z.
20:04:02So, this is what we need to do. Now,
20:04:05let's do one thing first.
20:04:08Let's just pull this up. But we don't
20:04:11really need to start looking at the
20:04:13output or the order by just yet. Let's
20:04:15keep everything. We'll keep this uh
20:04:18comma here. Let's keep everything, but
20:04:21let's start working on our calculation.
20:04:24So, let's see if our calculation is
20:04:26correct. It should be employees fired
20:04:28divided by the company size.
20:04:31I'm going to put this all in parenthesis
20:04:34um just to make sure we're doing PEMDOS
20:04:37correctly. and let's multiply it times
20:04:39100. So, let's run this.
20:04:42And here we go. Now, this looks correct
20:04:45just glancing at this because here we
20:04:48have 0%. Because Apple didn't lay anyone
20:04:51off. Uh 3% 6,000 of,800. That also looks
20:04:55correct. This one, I think, is the most
20:04:57um straightforward one. 15,000 into
20:05:01140,000. That should be around 10%. But
20:05:04because it's 11uh 15,000 to 140,000,
20:05:08it's a little more than 10%. And so this
20:05:10one looks very right to me. Now, one
20:05:13thing I didn't say mention here is we
20:05:15need to round to two decimal places. So
20:05:18let's go ahead and round this before
20:05:20anything. So let's put round and let's
20:05:23wrap this entire thing. Now we need to
20:05:26round this to two decimal places. So we
20:05:29need to do is do a comma two here
20:05:31because that says round to two decimal
20:05:33places. So let's run this. And there we
20:05:36go. That all looks correct. Now we can
20:05:39say this as let's just rename this as
20:05:42percentage
20:05:43just so we don't have that really long
20:05:46name. Uh it basically makes this the
20:05:48column name that's way too long. So
20:05:50we're going to call that percentage. Now
20:05:51the only thing that we need in our
20:05:53output is company name and percentage.
20:05:55So let's come back here and let's put
20:05:58company and let's run this. And this
20:06:01looks good except we need to order by
20:06:03the company name ascending. So we'll say
20:06:07uh order by we'll do company and then we
20:06:10can say ascending although by default
20:06:13order by is in ascending. We just I like
20:06:16explicitly writing it. So let's run
20:06:18this. And there we have Amazon, Apple,
20:06:22Facebook, Google, uh, Microsoft Tesla.
20:06:25So, this looks great. I think this is
20:06:27our final answer. Let's go ahead and
20:06:30check our solution. And there we go. We
20:06:33got the solution correct. Now, if you
20:06:35remember in the last video, we checked
20:06:36our profile. We earn points towards our
20:06:38badges. A medium question, you earned 25
20:06:41points. Um, and so if you're following
20:06:43along, you're doing these questions,
20:06:44then you should go check uh your profile
20:06:46out because you should have uh in your
20:06:48profile, you should have more points.
20:06:49Now, let's go to the next question. This
20:06:52one is called separation. It says, "Data
20:06:55was input incorrectly into a database.
20:06:58The ID was combined with the first name.
20:07:01Write a query to separate the ID and
20:07:03first name into two separate columns.
20:07:06Each ID is five characters long." All
20:07:10right. I've seen this in real databases
20:07:13uh a million times. Usually when we're
20:07:15like getting data from like an Excel
20:07:17file or something, we always have issues
20:07:20with Excel file or CSV files or stuff
20:07:22like that. So let's talk about how we
20:07:24are going to actually solve this. Now
20:07:26we're doing this in my SQL but again you
20:07:28can do this in Python, Postgra SQL,
20:07:30Microsoft SQL Server, whichever one you
20:07:33are practicing or or you know you have
20:07:35an interview coming up. Whichever one
20:07:36you have an interview coming up go ahead
20:07:38and use that one. Now, I think one of
20:07:40the main things that I'm interested in
20:07:42right here is that each ID is five
20:07:44characters long. Um, because we need to
20:07:46separate this out. So, it doesn't matter
20:07:48how long this name is. What matters,
20:07:50it's kind of kind of the key to this is
20:07:52how long this ID is. Cuz if it was 3 4 5
20:07:556, we might have to use something like
20:07:57regular expression to separate that out
20:07:59to extract all the numbers. But luckily,
20:08:02it's all five characters. So, with this
20:08:04um we should be able to use something
20:08:06like substring. This will be to pull out
20:08:10numbers and then names. So that's what
20:08:14we need. Uh and then we'll have two
20:08:16separate columns. So the output will
20:08:18then be the ID and the first name. And I
20:08:23believe that's all we need to do is
20:08:25separate them out into two separate
20:08:27columns. And then uh we needed to figure
20:08:29out how to actually separate it. So I
20:08:32think that's all we need. Let's pull
20:08:34this up.
20:08:36There we go. Now I'm going to keep
20:08:38everything. So just keeping this column,
20:08:41but then we'll separate it out into two
20:08:43and we'll see what this looks like. Now
20:08:45this substring is going to take a few
20:08:48different parameters. First we need to
20:08:50pass through the string. Now when I say
20:08:52string, I mean the column that contains
20:08:54the string cuz it's going to go through
20:08:56each row of that data. We need to select
20:08:58the ID and then we need to specify the
20:09:01start position and the end position. Now
20:09:04that's where this comes into place. uh
20:09:06where that five characters long comes
20:09:08into place. So because it's five
20:09:10characters long, we should start at
20:09:11position one and then we'll do a comma
20:09:14and end at position five. So start at
20:09:16position one and take through position
20:09:18five. Let's just run this and see if it
20:09:20works.
20:09:22There we go. So this pulled out just the
20:09:25first five characters in this string.
20:09:29Now we also need to pull out the full
20:09:30first name. And we can actually label
20:09:32this. Um let me bring this down. We'll
20:09:34do as um let's name it something. I'll
20:09:38say new ID. That's what we'll name it.
20:09:41And then we'll do the next one. So this
20:09:43will be substring.
20:09:46Now we also are going to pass through
20:09:47the ID. But this time we're not starting
20:09:49at position one. Now we need to start at
20:09:51position six. But we don't know how many
20:09:55characters are in each name. It could be
20:09:56S. It could be Harry. It could be uh
20:10:00Escariat. I don't even know if that's a
20:10:01name, but it could be really long. Um
20:10:03Alexander. That's my name, a long name.
20:10:05So, we don't know how long it could be.
20:10:07So, we could put something like 20 here.
20:10:10And if we run this, it's going to
20:10:12extract it. But what if someone's name
20:10:14is longer than 20 characters? That's not
20:10:16good. Luckily though, that uh third
20:10:19parameter, the end position, we can just
20:10:22leave blank and it'll go from the sixth
20:10:23position to the very end of that string.
20:10:26So, let's run this. And that looks
20:10:28really good. So we're going to say as
20:10:30first
20:10:32name. And there we go. So let's run this
20:10:35again. So we have the new ID and we have
20:10:38the first name. We can get rid of this
20:10:41initial one that has everything. And I
20:10:44believe this should be our full output.
20:10:46Now we have this new ID and we have this
20:10:49first name. Let's go ahead and check our
20:10:51answer. And there we go. Your solution
20:10:54is correct. Now these are just two of
20:10:56the medium questions on the platform.
20:10:57There are a ton of others. So, I'm going
20:10:59to leave links in the description to
20:11:00these questions as well as just to the
20:11:02questions page. So, you can go on there
20:11:04and you can practice and you can really
20:11:06get comfortable writing these out and
20:11:08using them because when you feel
20:11:10comfortable going into that SQL
20:11:12technical interview, you're going to do
20:11:13a lot better and you'll know how to talk
20:11:15through these things. And if you're ever
20:11:17having trouble with them, you can always
20:11:19go to this video explanation where I
20:11:21will walk through it and tell you my
20:11:22exact thought process on how I solve
20:11:25these questions. And honestly, I think
20:11:27that's one of the best features about
20:11:28the whole platform because when I was
20:11:30first starting out, I didn't have this.
20:11:32And so, I'm really happy that this is
20:11:34here for you. So, you can really learn.
20:11:35It's really just a learning platform to
20:11:38get better at these skills. So, if you
20:11:39have a technical interview coming up
20:11:41either in Python or SQL, try out
20:11:43analystbuilder.com. It is phenomenal. I
20:11:45created all the content myself. We also
20:11:47have full courses on there, so you can
20:11:49go ahead and check that out as well. In
20:11:51the next video, we're going to be going
20:11:52on to the hard questions. And it's going
20:11:54to be quite a big leap from medium to
20:11:56hard. And then after that, we're also
20:11:58going to be going into very hard
20:12:00questions. I would say the very hard
20:12:01questions are more of like a challenge.
20:12:03They're very difficult. They're a lot of
20:12:05fun. But go ahead and check all that out
20:12:07on Analyst Builder. With that being
20:12:09said, if you like this video, be sure to
20:12:10like and subscribe, and I will see you
20:12:12in the next video.
20:12:25What's going on everybody? Welcome back
20:12:27to another video. Today we are going to
20:12:28be solving hard SQL technical interview
20:12:30questions.
20:12:34[music]
20:12:37These hard interview questions are
20:12:38something that you would get in kind of
20:12:39a medium or a senior level data analyst
20:12:42position. The easy and the medium are
20:12:44more towards the entry level or slashmid
20:12:46level somewhere in that range. The hard
20:12:48ones are not something that you're going
20:12:50to get in the kind of more entry level
20:12:52range. These are questions that you
20:12:53might see in kind of a more advanced SQL
20:12:56technical interview. I've been on the
20:12:57interview side where I've interviewed
20:12:59for a ton of data analyst positions. I
20:13:00was also a hiring manager and then even
20:13:02before that I was on a hiring team where
20:13:04we conducted a ton of SQL technical
20:13:06interviews. And so the questions that
20:13:07we're going to look at today are very
20:13:08very similar to ones that I have seen in
20:13:10the real world or even given myself to
20:13:12interviewees. With that being said,
20:13:14let's jump on my screen and take a look.
20:13:15All right, so we're here on
20:13:16analystbuilder.com. We're going to go
20:13:18over here to the questions tab. We're
20:13:20going to filter to the free ones and
20:13:22then we'll go to the hard ones. Now,
20:13:26there are a lot more hard ones here on
20:13:28Analyst Builder, but under the free tab,
20:13:31uh we only have three. Looks like I
20:13:33didn't uh get this one right, but we
20:13:35have that one today. So, we'll see if I
20:13:37get this one right today. You can go try
20:13:38out these questions completely for free,
20:13:40and we're going to be taking a look at
20:13:42temperature fluctuations and Kelly's
20:13:43Third Purchase. And then there's another
20:13:45one called Cake Vers Pie, which I think
20:13:46may be the hardest of these three. So,
20:13:49you might want to go ahead and try to
20:13:51take that one and see if you can get it.
20:13:53Now, this is Kelly's third purchase.
20:13:54We'll start with that one and then we'll
20:13:56do temperature fluctuations. Uh, we'll
20:13:58see how quickly I can do these two
20:13:59because these are hard, but I think we
20:14:02can do it. So, let's look at Kelly's
20:14:05third purchase. It says, "At Kelly's Ice
20:14:07Cream Shop, Kelly gives a 33% discount
20:14:09on each customer's third purchase. Write
20:14:11a query to select the third transaction
20:14:13for each customer that received that
20:14:15discount. Output the customer ID,
20:14:17transaction ID, amount, and the amount
20:14:20after the discount as discounted amount.
20:14:23Order the output on customer ID in
20:14:25ascending order. Note transaction IDs
20:14:27occur sequentially. The lowest
20:14:29transaction ID is the earliest ID. Now,
20:14:33that's really important. We'll have to
20:14:35remember that. Now before we jump in
20:14:36anything, let's um let's go look at the
20:14:39data, but then let's start making some
20:14:40notes. So we have a customer ID, we have
20:14:44the transaction ID, and the amount that
20:14:47they spent. Now, this is the amount that
20:14:49eventually we'll need to use to
20:14:51calculate
20:14:53the end uh uh amount that they paid with
20:14:56the discount. So they get 33% off
20:14:59whatever this number is on their third
20:15:01purchase if that if you're tracking
20:15:03that. So, what we need to do, and let's
20:15:06start making some notes. One, we're
20:15:07going to need to apply a discount. So,
20:15:10that's going to be 33%.
20:15:13We have to identify though the person's
20:15:16third purchase. So, when they come in
20:15:18three times on that third purchase, they
20:15:21then get to get that discount. So, how
20:15:24can we do that? Well, I I'm almost
20:15:26certain just looking at this data
20:15:28because we have 101 for a customer ID.
20:15:301001
20:15:321001 what we can do is we need to order
20:15:36this transaction ID and then give it
20:15:38some type of rank now because each
20:15:41transaction ID should be unique and
20:15:43we'll double check that but it should be
20:15:44unique we should just be able to use row
20:15:47number but we could also use rank or
20:15:49dense rank um but they should give the
20:15:52exact out same output for each of them
20:15:55it shouldn't matter so I think using
20:15:56just row number um and then filter ing
20:16:01when it equals
20:16:03three. So, we're going to apply a row
20:16:06number based off the customer ID and the
20:16:08transaction ID. And then for each
20:16:11customer, we'll give it a row number.
20:16:13And then when it's three, which is the
20:16:15third transaction, that's the one we
20:16:17give the discount to. Um, in our output,
20:16:21let's look at what our output is going
20:16:23to be. Our output is going to be, let's
20:16:27take a look. Select the third
20:16:28transaction. Output customer ID,
20:16:30transaction ID, amount. So all columns,
20:16:34all columns
20:16:36with uh and I'll just copy this
20:16:39discounted amount.
20:16:41So really everything with just that new
20:16:43column. And then we need to order by the
20:16:47customer
20:16:49ID.
20:16:50So we got a lot to do. Uh this this
20:16:54definitely doesn't look like of course
20:16:55is a difficult question. is a hard one
20:16:57but it doesn't look like a super
20:16:59straightforward one. So the first thing
20:17:01that we need to do is we have to
20:17:04identify this row number because we
20:17:07cannot apply the discount until we know
20:17:10which data to apply it to the output in
20:17:13the order by will come at the very end.
20:17:15So let's look at this. Let's do um let's
20:17:18do a comma here. We'll come down here.
20:17:21Let's do row number. Now this is a
20:17:23window function. It's it, you know, if
20:17:25you haven't used these before or you
20:17:27haven't taken like my full course and
20:17:28and you know, worked through these
20:17:30things, row number uh is a window
20:17:32function that's going to apply to a
20:17:34window or kind of like a group by is is
20:17:37what I compare it to. When you group by,
20:17:39all of those customer IDs with 101 are
20:17:42going to be grouped into one row. With
20:17:44uh a window function, they aren't going
20:17:46to be grouped into one row. they'll just
20:17:48be in a window where you'll see each row
20:17:50individually and you can apply something
20:17:52to each row instead of grouping it and
20:17:56aggregating the data. So, it's really
20:17:58unique um and really useful. So, we're
20:18:00going to do row number and what we need
20:18:02to do this is over the transaction ID
20:18:06and we need to order by order by
20:18:09transaction ID and that's going to be
20:18:11ascending. So, the earliest one it says
20:18:13lowest transaction ID is the earliest.
20:18:15So, we need to start with the earliest,
20:18:17then go to the highest and pick the
20:18:18third one. So, let's just run this.
20:18:22And I need to do this over. I said row
20:18:24number. I didn't write that right at
20:18:26all. So, we'll do over and then we write
20:18:28it. So, we're doing we're applying this
20:18:31row number. The over is the keyword that
20:18:33we use to specify that this is what we
20:18:36are doing it on. And so now we have
20:18:39this. And we can't just do this because
20:18:43it's applying the row number
20:18:44appropriately based off of only the
20:18:47transaction IDs from from lowest to
20:18:49highest. Here's the thing though. We
20:18:51have to do it per each customer. So it's
20:18:53each customer's third. So we have to use
20:18:55partition by before the order by. Now
20:18:58partition by is going to separate it out
20:19:01by the customer ID. It's kind of that's
20:19:03kind of like the grouping part. Um,
20:19:06and so we'll use partition by customer
20:19:10ID. And why did I copy that? By customer
20:19:13ID. And now let's run this. And now when
20:19:17we come down, it should say 1001 101.
20:19:22And notice that we have this um row
20:19:25number applying at the customer ID
20:19:27level. And then when it gets to the last
20:19:30customer ID and it goes to the next one,
20:19:32it restarts. So now this is that third
20:19:35person's transaction 1001 and then at
20:19:3710002 this is the third transaction. Now
20:19:41here's the tricky part about trying to
20:19:44then use this row number is I cannot
20:19:47come down here and say where and let's
20:19:50label this. We'll say as row_num.
20:19:54Let's run that. Uh whoops because I have
20:19:57this as blank. Let's comment that out
20:20:00real quick. So I have this row num but I
20:20:05cannot say where row num is equal to
20:20:09three. Let's try it. So it's going to
20:20:12say unknown column. It doesn't
20:20:14understand that that's a column. And you
20:20:16may be thinking well you know in
20:20:18aggregations with group by you can use
20:20:19the having statement. Well let's try the
20:20:21having. And let's run this. It says the
20:20:25window function is allowed only in the
20:20:26select list and order by clause. We
20:20:29cannot use in the having. So what we
20:20:31need to do is we need to actually make
20:20:33this as uh its own little output is what
20:20:37I'll say. Now we can do that in two
20:20:39different ways. We can use a CTE and we
20:20:41can use common table expression to kind
20:20:43of store this data down here how it is
20:20:46and then we can query off it later or we
20:20:48can put it in a subquery. Or if you know
20:20:51we wanted to get really advanced and
20:20:53we're using um actual MySQL database, we
20:20:56could use something like a temporary
20:20:58table or a view or something. We could
20:21:01do other things, but for here let's wrap
20:21:04all of this in a um let me come right
20:21:09here. Let's wrap all this in
20:21:12a subquery. And when you have uh a
20:21:16subquery in a from statement, you have
20:21:19to label it. So you have to give it a
20:21:20name. So we're just going to call as row
20:21:22numbers.
20:21:24And let's select everything. And what
20:21:27this is doing is we're selecting
20:21:29everything from this data right down
20:21:33here. This table that we've essentially
20:21:35created. So what we're going to do is
20:21:37we're going to select everything. Now
20:21:40what we need to do is we need to say
20:21:43where row num is equal to three. And
20:21:48let's run this. So now we have each
20:21:51person's
20:21:52row num three. This is the third
20:21:54person's transaction. Now this is really
20:21:57good. So what we need to do next is we
20:21:59need to then calculate this amount. So
20:22:02it gets a 33% discount now. So let's
20:22:05select the columns that we actually want
20:22:07in our output. We need customer ID. We
20:22:11need transaction
20:22:13ID. We need the amount. Now we need to
20:22:17calculate the discounted amount.
20:22:19Remember we have to label this last one.
20:22:22Um we'll say as discounted amount. So
20:22:25this next column is going to be this
20:22:26calculation. Now we have to give a 33%
20:22:30discount. So we can't say amount times
20:22:34let me bring this down like this. We
20:22:38cannot say amount times 0.33. Let's run
20:22:43this and let me see. I just spelled
20:22:46transaction ID wrong. Transaction
20:22:50ID. That's it. Always gets me. Um, this
20:22:54is actually a This is 33% of this
20:22:58number. That's what this is. Now, 33% of
20:23:02this number is not a 33% discount. We're
20:23:05giving them a 67% discount. What we want
20:23:08is 67%
20:23:11of the amount. Let's run this. This
20:23:15right here is 33% off the total amount.
20:23:18It's a discount. So instead of paying
20:23:20$94, this person only had to pay 62.98.
20:23:25Now the last thing we need to do, the
20:23:27very last thing is order by customer ID.
20:23:30And it looks like it already is, but I'm
20:23:33going to do it anyways. We'll do order
20:23:35by customer ID ascending. And let's run
20:23:40this. This should be our final output.
20:23:44And you know, it took a little bit of
20:23:46work to get there. We had to use this
20:23:47subquery, but I'm pretty sure this is
20:23:48right. Let's go ahead and check this
20:23:51answer.
20:23:52And there we go. Our solution is
20:23:54correct. Now, remember, there's other
20:23:56ways to write this. There isn't just one
20:23:59way. Um, this is kind of the difficult
20:24:01part about hard interviews or like
20:24:03senior level data analyst interviews for
20:24:05for SQL um technical interviews. The
20:24:08difficult thing is there's not only one
20:24:09way to answer it. And so it starts
20:24:12getting down to okay, what's the best
20:24:13way to solve it? Walk through your
20:24:16thought process. So everything that I
20:24:18just did where I walked through and I
20:24:20said, okay, I could use any of these,
20:24:22but row number makes the most sense for
20:24:24this data, understanding the difference
20:24:26between those and why I'm choosing one
20:24:28over the other um is really helpful for
20:24:31the interviewer to understand and gauge
20:24:33kind of your skill level. That's why I
20:24:35recommend you write it out well, but
20:24:37also talk about it. The thought process
20:24:39is kind of the most important part. Now,
20:24:41if you tried this question, you could
20:24:43not get it or you couldn't solve it, you
20:24:45can always get a hint, um, you can take
20:24:47a look at the expected output or you can
20:24:50go up to the video explanation where I
20:24:51walk through this entire question or
20:24:54just go look at the solution and let's
20:24:55see if I wrote it the same way. Well, I
20:24:57called it Rn for row number, but this is
20:24:59essentially the same although I wrote
20:25:01out the column names. It's essentially
20:25:03the same, but you could have done a CTE
20:25:05uh with this as well. But that is how
20:25:07you would solve this Kelly's third
20:25:09purchase. I will leave a link in the
20:25:11description if you want to try that one
20:25:12out. Now, let's go up here. Let's go to
20:25:15temperature fluctuations. So, this
20:25:17question says, write a query to find all
20:25:19dates with higher temperatures compared
20:25:21to the previous dates. Yesterday, order
20:25:25dates in ascending order. Okay, let's
20:25:28look at the data. So, we have our date
20:25:30over here and the temperature. So, it
20:25:32looks like for example this one, this is
20:25:34the 2nd of January.
20:25:36This temperature was 70. The previous
20:25:39days was 65. So, we want to identify
20:25:42this date. And I think it's just the
20:25:44date um to find all the dates with
20:25:46higher temperatures. It looks like our
20:25:48output is just going to be the dates.
20:25:51And I'll write that real quick. Um
20:25:52output
20:25:54just date column.
20:25:57Now, how are we going to do this? How
20:25:58are we going to compare this? Uh
20:26:01initially, there are two things that I
20:26:03think we could do. One, we could use a
20:26:05window function. We could use a lag uh a
20:26:08lag function on this which would look at
20:26:11the previous rows data. So if we ordered
20:26:13on the date which it already looks like
20:26:15it's ordered, we can use the lag
20:26:17function to look at the previous value.
20:26:19So here is 70. Then we use the lag
20:26:21function. It would pull over 65 over
20:26:23here. That would perfectly fine uh way
20:26:25to do it. The other way we could do it
20:26:27is we could do a self join. So we could
20:26:30tie the table to itself, but instead of
20:26:32doing it where the date is equal to the
20:26:34date, we say the date minus one. That's
20:26:37another way that we could solve this. So
20:26:39you can go ahead and try whichever way
20:26:40you would like to. I think I prefer the
20:26:43self join. Um I just think the last one
20:26:47we did uh a row number which is a window
20:26:50function. So I don't want to do another
20:26:51window function, right? Although you
20:26:53could solve this with a window function.
20:26:54Um I'm going to try a self join. So
20:26:57let's pull this over. Um, I think I'm
20:26:59going to do a self join but on the
20:27:04previous day where there's a one day
20:27:05difference. So where the day is one day
20:27:09off is what I'll say. Um, then we can
20:27:13use that to compare. So use the
20:27:17temperatures
20:27:20to say where one is higher than the
20:27:22other.
20:27:25And that should make more sense uh in
20:27:27just a second when we start writing it
20:27:28out. But now we also need to order by
20:27:31dates ascending. That's it. So we have
20:27:34our data down here. And in order to do a
20:27:38self join, we're just going to say um
20:27:40I'm going to say inner join, but we can
20:27:42do if there's a different type of join
20:27:43you want to do, you can also do that.
20:27:45But I'm going to say on temperatures and
20:27:47we need to label these differently. So
20:27:49we'll do T1 for temperatures one and
20:27:52then T2. So now it's like we have this
20:27:54table over here which is temperatures.
20:27:56We have this table over here that's
20:27:57temperatures. And we're going to join
20:27:59them together. Now what are we going to
20:28:01Oh, let's do T2. Now what are we joining
20:28:04these together on? Um we're going to be
20:28:07joining this on the dates. Now there's a
20:28:10few different ways that we can write
20:28:11this, but there is a function called
20:28:13date diff where we can take one date and
20:28:17compare it to a different date and make
20:28:18sure it's one day different.
20:28:21So, let's go ahead and take a look at
20:28:23that. Let's do um a join on and we'll do
20:28:27date diff and then we'll do t1.
20:28:32And it's autopop populating it for us.
20:28:35But date diff, t2 dot and then we'll say
20:28:39date right there. And it should be a one
20:28:42day difference. Let's try this. Let's
20:28:44run this.
20:28:47And the reason why it's not pulling up
20:28:48is because we have all the same uh
20:28:51column names. Now, when it has the same
20:28:54column name, it's just showing up as
20:28:57just one. One is overlaying the other.
20:28:59So, what we're going to do is T1.ATE,
20:29:03T1.
20:29:07And let's get rid of that. Then, we're
20:29:09going to label these other ones
20:29:10different. So, we'll do T2.
20:29:13as date 2 and then we'll copy this
20:29:20and we'll do t2 temperature as
20:29:24temperature 2. Now let's run this and
20:29:27let's see what happens. So we have uh
20:29:31this date compared to the previous date
20:29:34this date compared to the previous date.
20:29:36So 3 versus 2 and let's keep going. 4 3
20:29:415 4 6 to 5. And so every single date has
20:29:45the previous date. Now what we can do is
20:29:48we can compare this temperature to this
20:29:51temperature. Now we can do that in a
20:29:53wear statement or we could just do it in
20:29:55the join. We can make it a conditional
20:29:57uh part of the condition in the join.
20:30:00Maybe I'll write out both. But let's say
20:30:01and we'll do T1 dot uh T1.
20:30:09Is greater than T2.
20:30:13So let's run this. So now we have 70
20:30:17compared to 65. And it looks like the
20:30:20other one wasn't as high. So the third
20:30:22day is gone. 58 compared to 55.
20:30:2590 compared to 58. 82 compared to 70. 88
20:30:29compared to 82. These are all of our
20:30:31dates right here in this column that
20:30:34this temperature was higher than the
20:30:36previous day's temperature. And so what
20:30:39we should be able to do is just get rid
20:30:42of all these columns and just take the
20:30:45date. And let's run this. And there we
20:30:48go. And all we have to do now is order
20:30:50by. And I think it's already correct,
20:30:54but I'm just going to I always like to
20:30:57write it out if it's asking us to do it.
20:30:59take do this in ascending
20:31:01and so let's run this. Yeah, in
20:31:04ascending order. I didn't write that. Oh
20:31:06yeah, here I did. I wrote it right here.
20:31:07So now this looks correct to me. Let's
20:31:10go ahead and check our answer. And there
20:31:12we go. Our solution is correct. Now
20:31:14again, there's multiple different ways
20:31:16to solve this. Genuinely off the top of
20:31:18my head, I could think of two, probably
20:31:19another one, another third option. Um,
20:31:22just off the top of my head, because
20:31:23I've been using my SQL for a while,
20:31:25walking through that in your interview,
20:31:27saying, "I think I could do it in this
20:31:28way or this way, but here's why I'm
20:31:30choosing this way." That really tells an
20:31:33interviewer, "This guy knows what he's
20:31:35talking about." Or, "Girl, this person
20:31:37knows what they're talking about. They
20:31:39understand it. They get it. I can trust
20:31:41that this person will know how to do the
20:31:43work that we're going to give them if we
20:31:45hire them." You want to give them a lot
20:31:47of confidence. That's all I'm going to
20:31:48say. Now, if you don't know how to do
20:31:50this or you've never done something like
20:31:51this before, that's what this platform
20:31:53is for. Um, so that when you get into
20:31:55those interviews or you know you want to
20:31:57learn and go take a course, when you get
20:31:59into those interviews, you can
20:32:00confidently say, I know this skill. Uh,
20:32:02and so if you had trouble with that one,
20:32:04you can always go to hints. You can
20:32:06always go to the expected output video
20:32:08solution solution. Um, let's see how I
20:32:11solved it here. Oh, I solved it the
20:32:12exact same, but I could I could have
20:32:14solved it a different way. I think the
20:32:15lag function would have done just as
20:32:18well. Uh it may have been simpler
20:32:20actually. So this is the one that I
20:32:23used, but honestly the lag function in a
20:32:25window function may even be better. So
20:32:28we solved this Kelly's third purchase.
20:32:30We solved this temperature fluctuations.
20:32:33I will leave links in the description
20:32:34for both of those. Go ahead and try
20:32:36those out yourself. And again, even back
20:32:39in the question section, there's this
20:32:40cake versus pie, uh, which is probably
20:32:43the most difficult of the hard ones, uh,
20:32:45under the free tier. Now, if you go back
20:32:48under the, you know, there's a, uh,
20:32:50where you can pay for a subscription
20:32:51under those, there's like 20 hard
20:32:53questions, and they're all very unique,
20:32:55very different, focusing on data
20:32:56cleaning, window functions, uh,
20:32:58different types of joins, and they're
20:32:59all really unique uh, and fun to do. But
20:33:02this cake versus pie one is really
20:33:04interesting. I want um I want you guys
20:33:06to go try this one. I'll leave this one
20:33:08in the description as well. This really
20:33:10interesting, difficult question. So,
20:33:11those were our two hard SQL interview
20:33:13questions. Uh they were pretty
20:33:15challenging. You know, window function
20:33:16and then self join. Two things that are
20:33:18a little bit more complex than you'll
20:33:20see in easy and medium questions. Uh in
20:33:22the next lesson, we'll be solving a very
20:33:24hard question. So, if you have not check
20:33:26out analyst builder.com, it's one of the
20:33:28best platform for data analysts. I
20:33:29created all the content on there, all
20:33:31the courses, all the questions, and we
20:33:33have so much more coming to the
20:33:34platform. If you like this video, be
20:33:36sure to like and subscribe below, and I
20:33:37will see you in the next video.
20:33:46[music]
20:33:51What's going on everybody? Welcome back
20:33:52to another video. Today, we're going to
20:33:54be solving a very hard SQL technical
20:33:56interview question.
20:34:03Now, if you've been following along with
20:34:04the entire series, we had videos on
20:34:06easy, medium, and hard SQL technical
20:34:08interview questions, and we solved two
20:34:10questions in each of those videos. But
20:34:12we are on to the very hard questions.
20:34:15These very hard questions are in fact
20:34:16very difficult. So, I'm only going to be
20:34:18doing one, although there's multiple on
20:34:20the platform that you can try, but I'm
20:34:22just going to be doing one because it's
20:34:23going to take a long time to solve it.
20:34:25Now, practicing these questions is meant
20:34:26to help you learn as well as feel
20:34:27comfortable and get ready for these
20:34:29technical interviews that you're going
20:34:30to get as a data analyst. The easy and
20:34:33medium questions are more geared toward
20:34:34entrylevel beginners or maybe even
20:34:36mid-level for the medium questions. The
20:34:38hard questions are geared more toward
20:34:40mid-level or senior level data analysts.
20:34:42And then there's the very hard. I don't
20:34:44think that you're going to get a
20:34:46technical interview this difficult. I
20:34:47know I haven't, nor have I ever given
20:34:49one, even though I've given tons of
20:34:51technical interviews before. Um, I've
20:34:54never gotten a question this hard. These
20:34:55are more of a challenge. Really a
20:34:58challenge of can you figure this out?
20:35:00Uh, because it's pretty difficult. So, I
20:35:02hope that you find this really
20:35:03interesting. I want you to try it out.
20:35:05But with that being said, let's jump
20:35:06onto my screen. All right. So, we're
20:35:08here on Analyst Builder. Let's go over
20:35:09to the questions page. Let's filter down
20:35:12to very hard. Now, I will note these are
20:35:16not under the free tier. If you go to
20:35:17the free, the very very hard ones are
20:35:20not under the free tier. So, if you want
20:35:21to try these, these are the very hard
20:35:23ones. We have consecutive visits,
20:35:25Twitter addiction, employee hierarchy,
20:35:27biggest spenders, complex address, Uber
20:35:30cancellation rates. Now, today we're
20:35:32going to be trying this complex address.
20:35:34But if you want to try out any of these
20:35:36other ones, head over to
20:35:37analybuilder.com. They are super super
20:35:39fun. But let's try out this complex
20:35:42address question. It says, "You are
20:35:44given a database containing customer
20:35:46addresses. Write a query to break out
20:35:48the address column into separate columns
20:35:50for street, city, state, and postal
20:35:53code. Note, some addresses may have
20:35:55additional unit or suite information.
20:35:58For example, sweet 5A or unit B, which
20:36:01should not be included as part of the
20:36:03street. So, let's go down here. Let's
20:36:05look at the addresses.
20:36:07We have 123 Main Street, sweet 5A. So,
20:36:10that's for example, that sweet 5A should
20:36:12not be included, it says. Then we have
20:36:14New York. That's the city. Uh, I'm
20:36:17guessing that's New York City. Then it's
20:36:19New York. Uh, then 1 2 3 4 5. Then the
20:36:22same thing. Minneapolis is the city.
20:36:25Then we have the state. And we have the
20:36:26zip code. I think that's all we need to
20:36:28Yeah, the postal code. So, let's start
20:36:31making some notes here. So, we have to
20:36:34the output needs to be uh street and I
20:36:38should just copy this street all the way
20:36:42down to postal code.
20:36:45There we go. So that's what our output
20:36:47needs to be. There's nothing on
20:36:48ordering. Uh I think the most difficult
20:36:51part is going to be just breaking it
20:36:54out. So breaking everything out. Now how
20:36:57are we going to do this? There's um one
20:36:59main way that I would be doing this and
20:37:01this is using substring. Now I've done
20:37:04this a thousand times in my real job.
20:37:07This is an extremely realistic thing.
20:37:10Happens all the time. data comes in just
20:37:12like this or sometimes separated by
20:37:14commas or just spaces and you have to
20:37:16figure that out, right? So, this is
20:37:19interesting because this is separated by
20:37:20no it's just a space. These are
20:37:22separated by these dashes. Um, and then
20:37:26we can't include this 5A. So, I'm going
20:37:28to use substring and we'll see if this
20:37:31works. Um, but with this we can choose
20:37:34our delimiter and that's really
20:37:36important. So we can say whether it's a
20:37:37space, whether it's a a dash or
20:37:41something like that. But we also have to
20:37:42note I'll just write note right here.
20:37:46Can't include things like sweet or unit.
20:37:50So we have to remove that somehow. Um
20:37:54let's go ahead and pull up the data over
20:37:56here
20:37:58and let's take a look. And I wanted to
20:37:59do this in my SQL. Now you can do this
20:38:01in Python. You can do this in Microsoft
20:38:05SQL. I had started out writing this in
20:38:08postgra SQL which uh the syntax is
20:38:11actually a little bit potentially maybe
20:38:13a little bit different actually it's
20:38:14different in Microsoft SQL server I
20:38:16believe postgrace SQL the substring is
20:38:18the same don't quote me on that but
20:38:20we're going to be using my SQL because
20:38:21I've been using that throughout the
20:38:22entire series
20:38:24but if you want to you can use Python my
20:38:27SQL Postgra SQL Microsoft SQL Server
20:38:30whichever one you feel comfortable using
20:38:33now let's run this so I'm going to keep
20:38:35everything here, but I'm going to add to
20:38:38it so we can confirm that the it's
20:38:41accurate. Like our output is correct. So
20:38:43that's why I'm going to keep the
20:38:44everything there. Now, we're going to
20:38:46use this substring. And the first one
20:38:49that we have to figure out is um this
20:38:53one right here. Now, it should be fairly
20:38:56easy with something like one that does
20:38:59not include sweet 5A. And for example,
20:39:02we would just do substring and we have
20:39:05to pass through some parameters. The
20:39:07first one that we need to pass through
20:39:08is just the string. Now this entire
20:39:10string is kept in address. So when I say
20:39:13we're passing through the string, we're
20:39:15passing through a column where it has
20:39:17multiple rows with strings in it. That's
20:39:20all that's all I'm saying. The next
20:39:22parameter is our delimiter. A delimter
20:39:24is something that how are we separating
20:39:26this out from itself? So I'm going to
20:39:29put in quotes I'm going to put a dash.
20:39:33And then the next parameter is where are
20:39:35we starting? So are we looking at the
20:39:38first delimiter, the second delimiter,
20:39:40the third, fourth, fifth? Because this
20:39:41one has multiple delimiters. This one
20:39:43has one right here and it has one over
20:39:45here. So if I put a one here,
20:39:48um, we get null. And that's because I
20:39:51wrote substring. We actually need
20:39:53substring_index.
20:39:57Uh, I'm thinking of Python. In Python,
20:40:00I'd be using substring. In my SQL, I
20:40:02need substring index.
20:40:04There we go. So, now we're doing this on
20:40:06the first delimter right here. But if I
20:40:08change this to two, now we're looking at
20:40:11this delimiter. So, that's the first
20:40:14delimiter. Second delimiter. So, we only
20:40:17want the first delimiter. But here's the
20:40:20issue with this. We have And there's a
20:40:22unit right here. So we have one that has
20:40:24unit and we have one that's sweet 5A.
20:40:28Here's what we need to do. We have to
20:40:32get rid of it if it has a sweet 5A or if
20:40:35it has unit B. So we'll need a case
20:40:38statement for this. Um what we'll write
20:40:40is we'll write case
20:40:43and this will be our else. So if it has
20:40:46auite in it or it has a unit in it, we
20:40:50will use the substring index on it. But
20:40:52if it doesn't, we're just going to treat
20:40:54it as normal. This is like our normal
20:40:56one. So, I'm going to say, and we'll do
20:40:58it right here. I'll have to format this
20:41:01in a little bit, but we'll say when when
20:41:03the address is like, and now we're
20:41:07looking for a pattern. We're going to
20:41:08search does this pattern exist in this?
20:41:12Now, uh, it's possible in some instances
20:41:16if you're using like a million rows of
20:41:18data, suite could be in like one two
20:41:21three Sweet Street. I've never seen that
20:41:24before. Um, I don't think we need to
20:41:26take that into account today, but that
20:41:28might be something to consider in a in a
20:41:30real world example, but this is a real
20:41:32world example, but we're just going to
20:41:34include sweet. And I'm going to include
20:41:36and I'm going to use these um
20:41:39these wild cards. So this means anything
20:41:42can come before this, anything can come
20:41:44after this. It just has to have a space
20:41:47suite. Now you don't have to add the
20:41:50that you could do it just like that. It
20:41:53makes more sense to me because we
20:41:55actually need to remove that in the
20:41:57future. And I'll explain that in a
20:41:58little bit. But when that happens,
20:42:01when there is a suite in there, what do
20:42:03we want to do? Well, we need to use
20:42:05substring index on it. But we cannot use
20:42:09the dash. The dash has got us in trouble
20:42:12last time. It kept the sweet 5A in
20:42:14there. What we need to do is we need to
20:42:16use this suite as our delimiter. So then
20:42:20when it gets to that delimiter, it's not
20:42:21included, right? So then we come over
20:42:24here, we put the suite in there. Let's
20:42:26run it. And I got a syntax error. And
20:42:29that's because we need an end. And we
20:42:32need and we can label it as well. But
20:42:34this will be our street. Uh we have to
20:42:36have an end to signify that the case is
20:42:39done. I'm pretty sure that's the issue.
20:42:41There we go. And that fixed it. That's
20:42:44because like the delimter with the dash,
20:42:47when it got to the dash, it took
20:42:49everything before it. So now if we find
20:42:51a suite, we're using that space suite as
20:42:55the delimiter. Now, if we put this as
20:42:58the delimiter, you may not be able to
20:43:00see it, but there should be a space
20:43:03right here, and that might cause dirty
20:43:05data. I'm going to actually I'm going to
20:43:06keep it like that. We'll see if my
20:43:08hypothesis uh is correct. Now, we're
20:43:11going to do the exact same thing
20:43:14except now we're looking for which one
20:43:16was it? Unit. So, if it includes sweet 5
20:43:20A or unit B, those are the examples. We
20:43:22need to get rid of that and we'll use
20:43:25unit. And let's run it.
20:43:28And there we go. So, this looks perfect.
20:43:31This looks exactly like what we should
20:43:33be getting in our output. So now uh
20:43:37Whoops. Now we need to come here down to
20:43:38the street. So now we have the street,
20:43:41but we need to get next we need the
20:43:44city. So the city is right after the
20:43:48street. But here's the thing. It's in
20:43:51the middle. And if you've ever worked
20:43:52with data like this, it's a little bit
20:43:54tricky because uh a delimiter only goes
20:43:58to one point, right? So what we can do
20:44:01is use a double uh substring index. So
20:44:06we're collecting the substring index and
20:44:08then uh within that text of that
20:44:10substring index we do another substring
20:44:12index. So I think that's what we need to
20:44:14do here. So let's come down here and we
20:44:17can just copy this because it's already
20:44:20written up for us. Um and let's run
20:44:23this. Now what if we say two here? Let's
20:44:27run this. Um we can do it this way and
20:44:31then go backward we can do a minus. So
20:44:34in substring index instead of a positive
20:44:37one looking forward that's starting from
20:44:39um the left hand side of the string and
20:44:42looking this way for the first one we
20:44:44can do negative which starts from the
20:44:45right hand side and looks this way. So
20:44:48left to right when it's positive right
20:44:50to left when it's negative. So then we
20:44:52can wrap this as a substring index.
20:44:56And now this whole thing right here is
20:44:59our string, which is this. This is our
20:45:01string. So now we're going to look
20:45:03backward. We're going to do minus one.
20:45:06Um, actually our our delimiter first is
20:45:09a dash because we're looking to this
20:45:11dash. And then we need to go to negative
20:45:14one. And let's try this out. And there
20:45:18we go. New York, Minneapolis, Goldsboro,
20:45:20Maples, Flower Town. So this looks
20:45:24perfect. So, let's keep that exactly how
20:45:27we have it. Now, for the next one that
20:45:29we need, uh, and I'm going to label this
20:45:31as city.
20:45:34Now, we're going to copy this whole
20:45:36thing, bring it right down here. We'll
20:45:38do this as state. Now,
20:45:42obviously, this isn't going to be our
20:45:43answer, but we can run it. So, here's
20:45:47what we need to do. Before we took this
20:45:49whole string and we got here and then we
20:45:52went backward to this delimiter and
20:45:55selected New York. What we now need to
20:45:57do is we need to go we need to select
20:46:01this New York Minneapolis. Now there's a
20:46:03space here and so what we should do is
20:46:05we should go backward to this index
20:46:09right there. I think we need to do that
20:46:10first. So we'll use that as our our
20:46:14starting place.
20:46:17And so we'll do negative one. Um,
20:46:20yeah. So I'll just do it. So we'll go
20:46:22negative one, but then we need to go
20:46:23forward. So this is going to be our
20:46:25string right here. New York 1 2 3 0 5.
20:46:27So then we need to go forward. And our
20:46:29delimiter should be a space.
20:46:32Let's do a space right here. Let's run
20:46:34this. And there we go. We have New York,
20:46:37Minneapolis, North Carolina,
20:46:39Massachusetts, I think, and Florida. So
20:46:42that's our state. And we have one more,
20:46:44and that's going to be our very last
20:46:46one. Now, this one should actually be a
20:46:47little bit simpler because we're just
20:46:49starting at the very end. It's not in
20:46:51the middle, which is a little bit
20:46:54tricky, right? So, we're going to come
20:46:55in here. We're going to say as, and this
20:46:57is postal code. I really want to stick
20:47:00to exactly what they told us to call
20:47:02them. I don't want to go changing it.
20:47:06So, now we're looking for a space
20:47:07delimiter, but we're looking backward
20:47:10negative 1. So, we're starting from the
20:47:12right hand side and going to the first
20:47:14space. And that should give us that 1 2
20:47:163 4 5. Let's run this. And there we go.
20:47:21Now I'm going to get rid of everything.
20:47:25And let me actually come up here. I'm
20:47:26going to get rid of everything.
20:47:28And let's run this. So now we have the
20:47:31street, city, state, and postal code.
20:47:33This all looks correct. I don't know. I
20:47:37genuinely don't know if I check this
20:47:39answer if it's going to be correct or
20:47:40not. My hypothesis, what I think is
20:47:42going to happen is is is not going to be
20:47:45correct because I think somewhere in our
20:47:47output, we have some spaces that we
20:47:49can't see. For example, this suite, if
20:47:53we're using just the suite and not the
20:47:55space suite, I think we have an extra
20:47:57space at the end of this one right here.
20:48:00So, it goes sweet space, which I don't
20:48:03think is correct. And it shouldn't you
20:48:05shouldn't have that in actual output in
20:48:07a real database. You don't want uh uh
20:48:09leading or trailing spaces. That's dirty
20:48:12data. So, let's try checking it. And
20:48:14there we go. Our answer is wrong. Let's
20:48:17try just fixing this and seeing if
20:48:19that's it. If not, it could also be one
20:48:22of these. But let's try this one now. Uh
20:48:25there we go. So, that is that was the
20:48:27exact issue. And it's really hard to
20:48:28catch. If you're not looking for it, you
20:48:31may not catch it because it is a little
20:48:33bit tricky. Um, but that is I let me see
20:48:38if I added this in as a hint.
20:48:41Maybe I did, maybe I didn't. But that is
20:48:43something that you need to account for
20:48:44in real data. So, um, I know that there
20:48:48were people in the Discord, if you
20:48:50haven't joined Analyst Builder and tried
20:48:51out these questions, um, we have a we
20:48:53have a Discord with like 2,000 people in
20:48:55it. And this question when people have
20:48:57been trying to solve it, have been
20:48:58really having issues with this exact
20:49:00thing. Is I it looks correct, but it's
20:49:02not correct. um that is a real world
20:49:06solution, a real world issue. And so
20:49:08that is how you solve this question. Now
20:49:12you know I have a whole video
20:49:13explanation on how to do this. Um as
20:49:16well as you can just come down here.
20:49:17Let's look at the solution. So yeah,
20:49:19this is exactly how I wrote it. Um there
20:49:22are other ways to write this actually.
20:49:24But that is how you can solve it. Now
20:49:27remember, you can come in here even in
20:49:30the in the questions and go to the
20:49:32difficulty and you can take a look. You
20:49:34can check out all these other questions
20:49:37um and see what they look like and what
20:49:40the questions are. These are really
20:49:41tough. U for example, this Uber
20:49:44cancellation rates is a really
20:49:45interesting one. Find the cancellation
20:49:47rates of requests with unbanned users.
20:49:49Both client and driver must not be
20:49:51banned each day between 2023, 1223, and
20:49:561225. Round the cancellation rates to
20:49:58two decimal points. So, this is another
20:50:00really interesting question. Um, and
20:50:03there's actually two tables here. I'm
20:50:05not doing this one today, uh, but these
20:50:08are more questions. I'll do more of
20:50:10these videos in the future just because
20:50:12I really love them. They're super fun.
20:50:14Um, but if you want to try that question
20:50:16out, I will leave a link in the
20:50:18description. You can go ahead and check
20:50:19that out. If you have not tried out
20:50:20Analyst Builder already, I highly
20:50:22recommend it. I created all the content
20:50:24myself. All the questions, all the
20:50:26courses are all done by me. So, if you
20:50:27like my YouTube channel, you will love
20:50:29Analyst Builder. All premium, really
20:50:31high uh quality content. So, go ahead
20:50:33and try that out. Thank you guys so much
20:50:36for watching. I really appreciate it. If
20:50:38you like this video, be sure to like and
20:50:39subscribe below, and I'll see you in the
20:50:41next video.
20:50:54What's going on everybody? Welcome back
20:50:55to another video. Today we're going to
20:50:57be starting our Azure series.
20:51:04Now, if you don't know, Azure is a
20:51:06cloud-based platform owned by Microsoft.
20:51:08Azure is one of the biggest cloud
20:51:10platform and has millions of users all
20:51:11around the world. I myself use it for
20:51:14many years and I absolutely love Azure.
20:51:15I think Azure is a fantastic cloud
20:51:17platform and that's why we're going to
20:51:18be starting to learn it. I think knowing
20:51:20a cloud-based platform is an essential
20:51:22skill today for any data analyst, data
20:51:24scientist, data engineer out there. So,
20:51:25that's what this entire series is going
20:51:26to be for. We're going to get our
20:51:27account set up. We'll look at things
20:51:29like account storage, SQL databases, uh
20:51:31even some things like data pipelines.
20:51:33So, using Azure data factory and Azure
20:51:35Synapse Analytics, these resources and
20:51:37tools within Azure are ones you're
20:51:39absolutely going to see. And so these
20:51:40are the ones that we're going to be
20:51:41focusing on in this lesson. We're just
20:51:42going to be getting everything set up.
20:51:44So we're going to be creating a
20:51:45Microsoft account. We're going to be
20:51:46creating an Azure account. And then
20:51:47we'll be doing a walkthrough of the UI
20:51:49just so you can kind of see it and get
20:51:51familiar with it. So with all that being
20:51:52said, let's jump on my screen and take a
20:51:54look. We're going to start right here on
20:51:56the azure.microsoft.com/free
20:51:59over here. Now this is going to be a
20:52:00little bit different if you're in a
20:52:01different country, but this is for uh me
20:52:04here in the US. But I'll have this link
20:52:06in the description so that you can come
20:52:08to it. you can create your account and
20:52:10then we'll get into all the uh UI and
20:52:12how everything looks once we actually
20:52:14set up our account. Now, before we
20:52:15actually create it, just go down here.
20:52:18Here are some of the things that you're
20:52:18going to get when you create your
20:52:20account. You're going to get uh popular
20:52:21services free for 12 months, which is
20:52:24really, really great. Um 55 services
20:52:26that you're always going to be free. And
20:52:27then we're going to get this credit
20:52:28right here. Now, this credit is really
20:52:29important because some of the things
20:52:30that we're going to be looking at in
20:52:32this series are not things that are
20:52:34completely free all the time. So, uh,
20:52:36we're going to be using some of this
20:52:38credit throughout this series. So, you
20:52:39want to make sure you can get that. You
20:52:40just need to create a new account, and
20:52:42that's what we're going to be doing. And
20:52:43then we don't have to use our money. We
20:52:44can use Azure's money, and we will all
20:52:47thank them for that. Let's come over
20:52:48here. We're going to click on start
20:52:50free. So, we're going to start our free
20:52:52trial. Now, if you already have an
20:52:53account, you can use it. If not, we're
20:52:56going to click on use another account.
20:52:58Now, again, for this, you can sign in
20:52:59with things like GitHub. You can sign in
20:53:01with a previous account. If you do not
20:53:03have an account, I'm just going to show
20:53:04you how to create one really quickly,
20:53:06and that's what we'll use going forward.
20:53:08If you already have an account, if
20:53:09you're already signed in, go ahead and
20:53:10skip a little bit forward, maybe like a
20:53:12minute or so, and we should be done
20:53:13creating this account. But let's go
20:53:14ahead and see how we can do one. We're
20:53:16going to go to create one. We're going
20:53:18to call this uh Alex the analyst
20:53:22atoutlook.com.
20:53:25And there we go. So, we have Outlook or
20:53:26we can do Hotmail. Hotmail feels like
20:53:28it's from like the 90s, so I'm going to
20:53:29do Outlook. at least feels uh somewhat
20:53:31current. Then we need to create our
20:53:33password. I don't want any tips. So, I'm
20:53:35going to do a password right here. And
20:53:37let's go ahead and click next. You're
20:53:39going to fill in when you were born. And
20:53:41we'll go ahead and click next. And it
20:53:43should be creating our account. Looks
20:53:44like we need to do a little uh puzzle
20:53:46here. Looks like we need to rotate it
20:53:48where it's pointing.
20:53:51And there we go. So, now we're signed
20:53:53into our Microsoft account, but now we
20:53:55need to actually create an Azure
20:53:56account. And those are two separate
20:53:58things. You need two accounts. So, we're
20:54:00going to go through. We're going to fill
20:54:02in all this information. Your name, your
20:54:04phone number, uh your address, and all
20:54:06these different things. Then, we're
20:54:07going to have to fill out uh our card,
20:54:10so a debit card or credit card so that
20:54:12we can actually if we go over the $200
20:54:14or if we use a service that is not free,
20:54:17then it is going to charge us. We won't
20:54:19be doing that, thank goodness. But you
20:54:21do need to have a card on file in case
20:54:22you do that. Let's go ahead and fill out
20:54:24all this information. Then, we'll go to
20:54:25the next part. All right. So, we just
20:54:26created our profile. Now, we need to
20:54:28identify our verification by card. One
20:54:30thing I will note in the profile, I did
20:54:32have to verify my phone number. So, they
20:54:34sent me a text. I put in the number and
20:54:36there you go. Next thing we need to do
20:54:38is we need to put in our debit card. So,
20:54:40go ahead and do that. And once we fill
20:54:42that out, [clears throat] you're going
20:54:42to see this right here, which is a
20:54:44little welcome to Microsoft. You can go
20:54:46ahead and look through this if you'd
20:54:48like. I'm just going to get out of it
20:54:49because we don't really need it. Now,
20:54:53this is going to be the first thing that
20:54:55you see. This is just the homepage of
20:54:57Azure. This is where you can access
20:54:59different services and resources and
20:55:01look at your profile and notifications
20:55:02and all these different things. Now, the
20:55:04first thing that we're going to do is
20:55:05just take a look at some of these
20:55:07resources because there are a lot of
20:55:10resources within Azure. And so, if we
20:55:12come down right here, let's go ahead and
20:55:14click into one of these. This is Azure
20:55:16Synapse Analytics. If we want to uh
20:55:18actually use Azure Synapse Analytics, we
20:55:21need to create a Synapse workspace. then
20:55:24we can start using this resource and
20:55:26start using this application and all we
20:55:28would have to do is create this. So we'd
20:55:29come in here to create synapse
20:55:31workspace. We would start filling out
20:55:33all these things and then we would
20:55:34actually have access to that resource.
20:55:37And so this one specifically is one that
20:55:38we will be looking at in this series. Uh
20:55:41let's go down and just really quickly
20:55:43take a look at some of these. We're
20:55:44going to be looking at data factory. So
20:55:46creating workflows and data pipelines.
20:55:48We will be using Azure Synapse
20:55:50Analytics. And if we go down to
20:55:52databases, we'll take a look at SQL
20:55:54databases as well as maybe one or two
20:55:56others because knowing how to use
20:55:58databases uh with the resources within
20:56:01Azure is actually quite important. And
20:56:02then if we go down to storage way down
20:56:06here, let's keep going. We go into
20:56:09storage, we will be looking at storage
20:56:11accounts, and that's actually where you
20:56:13can access something like a data lake.
20:56:15And so this can be pretty intimidating
20:56:17just looking at this because there's so
20:56:18many different things. But in this
20:56:20series, we'll really focus in on the
20:56:22things that I think you need to know
20:56:23that you're definitely going to be using
20:56:25as a data professional or especially a
20:56:27data analyst. But you know, even things
20:56:29like uh let's come back up here.
20:56:33Even things like data factories are used
20:56:35by data engineers, database developers,
20:56:36data scientists, it's everybody. And so
20:56:39knowing how to use these tools uh within
20:56:41Azure extremely important. So that's
20:56:43what we're going to be focusing on in
20:56:44the next several lessons. We'll be
20:56:46getting into specific resources within
20:56:48Azure and how to use them. That is all
20:56:50we're going to take a look at in this
20:56:51lesson because we're just getting set
20:56:52up, creating the accounts, looking at
20:56:54some of kind of the user interface of
20:56:56how Azure actually looks. And in the
20:56:58next several lessons, we'll be diving
20:56:59in, getting hands-on experience with a
20:57:01lot of these tools. Thank you guys so
20:57:03much for watching. I really appreciate
20:57:04it. If you have not checked it out
20:57:06already, I have a full course on AWS and
20:57:08Azure on analystbuilder.com. Be sure to
20:57:11go check it out. I'll leave a link in
20:57:12the description. If you like this video,
20:57:14be sure to like and subscribe, and I
20:57:16will see you in the [music] next video.
20:57:28[music]
20:57:30What's going on everybody? Welcome back
20:57:31to another video. Today we're going to
20:57:33be taking a look at account storage in
20:57:35Azure.
20:57:38>> [music]
20:57:41>> Now, account storage is super important
20:57:43within Azure. This is where a lot of
20:57:45companies are going to store a lot of
20:57:46their data. Using account storage in
20:57:48Azure is a super flexible way to store
20:57:50your data. It can store just about
20:57:51anything and you can even upgrade it
20:57:53into a data lakeink. And so, we'll be
20:57:54taking a look at all these things in
20:57:56this lesson just to get you familiar
20:57:57with how to use account storage because
20:57:59you absolutely will be using that within
20:58:01Azure. Without further ado, let's jump
20:58:02on my screen and take a look. So, let's
20:58:04start by taking a look at our resources.
20:58:06Right here in storage, we have our
20:58:08storage accounts. Now, as you can see,
20:58:10there's lots of different options for
20:58:13storage, but by far the one that you're
20:58:15probably going to use the most is
20:58:17storage accounts. Now, we do have right
20:58:20over here, you'll notice data lakeink
20:58:21storage gen 1. If we click into this and
20:58:24let's say we wanted to create a data
20:58:26lakeink, it says right up here, Azure
20:58:29data lakeink storage gen 1 will be
20:58:31retired on February 29th of 2024. We
20:58:33recommend that you migrate your Azure
20:58:34data lakeink storage gen 1 to Azure data
20:58:38lakeink storage gen 2 and that is
20:58:39located in the storage account. So let's
20:58:42come back here. We're going to go right
20:58:44into storage accounts and let's really
20:58:47quickly set up a storage account. Now
20:58:50this is some helpful information if
20:58:53you've never used a storage account
20:58:54before. It says you can store about 500
20:58:56terabytes. It has general purpose
20:58:58storage use for object stores. No SQL
20:59:01data store. uh you can define and use
20:59:03queries for message processing and you
20:59:05can also set up file shares. At the very
20:59:07end we have blob storage accounts for
20:59:09hot and cool access tiers. These are all
20:59:11things that we'll look at uh in this
20:59:13lesson. And so that is just kind of a
20:59:15preview of a lot of the things that
20:59:16we're going to be looking at. So let's
20:59:18go ahead and create our storage account.
20:59:20Now this is a brand new account uh as we
20:59:22set it up in the last lesson. So I'm
20:59:24going to be walking through this with
20:59:25you as if I have a brand new account.
20:59:28Now, you have to have your subscription,
20:59:30which if you're using the free tier,
20:59:31you'll have a free Azure subscription,
20:59:33but we have to have a resource group.
20:59:36Now, we haven't created a resource
20:59:37group. Let's go ahead and create a new
20:59:39one. And we'll just call this one Alex
20:59:41the analyst. And we'll click okay. Next,
20:59:44we need to specify a storage account
20:59:46name. We'll call this one Alex the
20:59:48Analyst storage. I'm guessing that's
20:59:51unique. Uh the storage account name has
20:59:53to be unique across all of Azure. So if
20:59:56you type in something kind of generic,
20:59:58usually it will already have been taken.
21:00:00So you have to do something pretty
21:00:01specific. Now this part and the next
21:00:04part uh these are pretty important. This
21:00:06region says choose the Azure region
21:00:08that's right for you and your customers.
21:00:10Not all storage account configurations
21:00:12are available in all regions. Now if
21:00:14you're just setting this up for
21:00:15yourself, just to store some data on the
21:00:17cloud or you have a little app that
21:00:18you're creating or something like that,
21:00:20it's very easy. You're just going to
21:00:22select your local region. And for me,
21:00:25that's US East. But what if you have a
21:00:28company or a product or an app that's
21:00:30being used by people all around the
21:00:33world? For example, you have a client-f
21:00:35facing app that they're using and
21:00:36they're going in and they're uh
21:00:38retrieving data from some type of
21:00:40storage account. Or maybe they're
21:00:41running a query on your website of data
21:00:43that's stored in a storage account. And
21:00:46what's happening is is it's stored
21:00:47locally to you, but they're way over
21:00:50here in uh let's say they're in central
21:00:53India. And so in order to get from
21:00:55central India to the US, it's going to
21:00:57take a lot longer to retrieve that data.
21:00:59And so it could take 5 10 15 20 seconds
21:01:02for it to get to them in India versus if
21:01:05it's just locally in US East. And so you
21:01:08need to know where your customers or
21:01:10where your clients are located who are
21:01:12going to be using this. Now, just for
21:01:13this lesson, we're going to keep it in
21:01:15East US because that's where I am
21:01:16located. Uh but that is something you
21:01:19really need to consider, especially as
21:01:20you get more advanced uh with using
21:01:22Azure. But, you know, just for the
21:01:24basics, you know, you don't really need
21:01:26to be thinking about that. I just want
21:01:28to walk you through my thought process
21:01:29as we're going through this. Next, we
21:01:31have to specify our performance. This is
21:01:33just going to determine how quickly and
21:01:35how easily you can retrieve your data.
21:01:37We're just going to go with standard,
21:01:38but premium would just allow you to
21:01:39retrieve that data even quicker. Next,
21:01:41we have redundancy. Now, redundancy is
21:01:44important, and it's one of the benefits
21:01:45of the cloud, which is if a server goes
21:01:48out that's storing your data, they're
21:01:50going to have a backup of that data
21:01:52somewhere else. And you can specify
21:01:54where that is going to be located. You
21:01:55can either do it a locally redundant
21:01:58storage, a geo redundant storage, a zone
21:02:00redundant storage, or a geozone
21:02:02redundant storage. And so if your data
21:02:03is super important, it is critical to
21:02:06what your website does, it is critical
21:02:08to your clients, if it was deleted by
21:02:10accident in any way or destroyed in any
21:02:12way, your whole company would collapse.
21:02:14You're going to choose this one down
21:02:15here. But if it's just some local files
21:02:17that you're uploading for someone to
21:02:19pick up, uh, and it's not that
21:02:20important, you're going to do something
21:02:21like locally redundant storage. And so
21:02:23that that's what we'll choose. Uh, just
21:02:25something to consider though. Now, there
21:02:27are other things you can do in advanced
21:02:30networking, data protection, encryption,
21:02:32tags, and all these different things.
21:02:34But if I'm being honest, 99% of the
21:02:36time, you're never going to use any of
21:02:38these unless you really, really, really
21:02:40know what you're doing or that's your
21:02:42job. you're some type of database
21:02:43administrator creating these storage
21:02:44accounts for people on your team. Most
21:02:46likely, you're never doing that. So,
21:02:48let's go down here. We're going to click
21:02:50create and it's going to say we have our
21:02:52deployment in progress. So, it's going
21:02:54to start creating the resources needed
21:02:55for that storage account. And then we're
21:02:58going to actually get into the storage
21:02:59account and start using it. And just
21:03:00like that, took about 10 seconds. It
21:03:03says that your deployment is complete.
21:03:05We don't need to go to the resource up
21:03:07here. We're going to go to the resource
21:03:08right here.
21:03:10Now what we're looking at is the user
21:03:12interface for this specific storage
21:03:14account. So we have Alex the analyst
21:03:16storage. If we go back to all services,
21:03:19we come into storage accounts. You can
21:03:21now see that we have one. We don't have
21:03:23to just have that uh boilerplate text
21:03:26and then create. If we want to create
21:03:27another one, we'll come right up here.
21:03:29But we can come into the storage account
21:03:31and we can look at this overview. So
21:03:33this is just some of the information on
21:03:35the location, the subscription, uh the
21:03:38subscription ID, the type of
21:03:39performance, the replication or
21:03:41redundancy, and a few other things as
21:03:44well. Now, there's a ton of things on
21:03:46this sidebar on this left hand side. We
21:03:49have things like activity log, tags,
21:03:51diagnosis, solve problems, access, data
21:03:54migration, events, storage browser,
21:03:55storage, mover, and all these different
21:03:57options. It's kind of overwhelming, but
21:03:58I know just from experience that you're
21:04:00not going to use almost any of these. In
21:04:02fact, things like monitoring are
21:04:04typically used like something like logs
21:04:06are typically used by it if there's an
21:04:08issue. Most likely, you're not going to
21:04:10be coming in here and taking a look at
21:04:12all these things or creating uh
21:04:14different alerts. You may be working
21:04:16with metrics if you're in the IT
21:04:18department. But again, a lot of this
21:04:20stuff you're not going to be working
21:04:21with specifically. Most of the time,
21:04:23we're going to be here in this storage
21:04:25browser. And this is where the data is
21:04:27actually uploaded, stored, and accessed.
21:04:30So if you come in here, let's come up
21:04:32here to a blob container. Let's say we
21:04:35want to create a blob container just to
21:04:36store a bunch of data. Maybe it's for a
21:04:38client or an application or whatever. We
21:04:41just want to be able to store that data
21:04:42in the cloud. And this kind of the
21:04:43simplest version of being able to use
21:04:45the storage browser or just storage in
21:04:48general. So what we can do is we can add
21:04:51a container. Now we're going to name
21:04:53this container. We'll call this uh ATA
21:04:56container. There we go. And if we come
21:04:58down here, we do have an advanced tab.
21:05:00You're most likely not going to use it
21:05:02because this is uh deals with
21:05:04encryption. And just this is completely,
21:05:07you know, as a sidebar above and beyond
21:05:09what you probably need to know. But when
21:05:11you start trying to access data later on
21:05:13in different applications, whether it's
21:05:14a SQL database or you're using it in
21:05:16PowerBI or whatever you're using uh this
21:05:18data for, if you have it encrypted,
21:05:20you're going to have to have a way to
21:05:21unencrypt it within Azure. And so it is
21:05:24an additional level of security, but it
21:05:26makes it a little bit more difficult to
21:05:27access that data later on if it's not
21:05:30super sensitive data. So just something
21:05:32to think about. Let's go ahead and
21:05:34create this. And now we have this ATA
21:05:37container. So we're doing great over
21:05:39here. So we've already created a storage
21:05:41account. We've gone into the storage
21:05:42browser. We've created a container. And
21:05:45these blob containers are amazing. I've
21:05:47used them thousands of times for so many
21:05:50different things. And blob containers
21:05:52are really just used for anything.
21:05:54Anything you need, you can use and you
21:05:56can dump inside of a blob container,
21:05:58whether it's structured,
21:05:59semi-structured, or unstructured data.
21:06:01And so let's see how that actually
21:06:02works. Let's go into this. And we have
21:06:04nothing in here. And we want to upload
21:06:07some data. So let's come up here to
21:06:09upload. We're going to go and browse for
21:06:10some files. So in here we have a bunch
21:06:13of different files. We have a SQL text
21:06:16file, a PNG which is just an image. Uh a
21:06:19Jupyter notebook file and a CSV. All of
21:06:21these are completely different. None of
21:06:23these are similar almost any way. And so
21:06:26what we're going to do is we're going to
21:06:26select all these. We're going to open
21:06:28these up and we can upload these. But
21:06:31really quickly, let's come in here and
21:06:33take a look at some of these advanced
21:06:35options because this part actually is
21:06:38something that you might use. The really
21:06:40the most important one in here is this
21:06:42access tier. If we hover over it, it's
21:06:45this piece that's kind of important. It
21:06:46says optimize storage costs by placing
21:06:48your data in the appropriate access
21:06:51tier. And you can come over here and
21:06:53look at all the access tiers if you'd
21:06:54like. But let's take a look at what
21:06:56these access tiers look like and what
21:06:57they actually do. We have four options.
21:07:00We have hot, cool, cold, and archive.
21:07:02Now, this refers to how the data is
21:07:04actually going to be stored in the blob
21:07:06storage. If it is hot, that means that
21:07:09you can just retrieve it anytime you
21:07:11want right away within milliseconds.
21:07:13It's just going to be ready to go and
21:07:14it's going to be there for you. But if
21:07:16you go with an option like cool or cold,
21:07:19it's not going to be there hot and ready
21:07:21just to be able to pick up and use that
21:07:23data. It's going to be sitting in a data
21:07:24store where if you want to retrieve it,
21:07:26you may have to wait a little bit. And
21:07:28so these options are actually a lot more
21:07:30cost-effective because if you're not
21:07:32using that data actively, you can just
21:07:34plop it in there as a data store where
21:07:36you're not using it for any application
21:07:37or any project and you have the data
21:07:40stored securely but you don't have to
21:07:41pay a ton of money for it. Whereas if
21:07:43you store it in hot, it's going to cost
21:07:45more money to store that. Lastly is
21:07:47archive. And archive means you most
21:07:50likely won't ever use it. This is a
21:07:52contract that was signed uh six years
21:07:54ago. you need it on file, but most
21:07:56likely you're never going to use this.
21:07:58If you do, you're willing to wait 5 or
21:08:0010 minutes cuz it's not going to be an
21:08:02emergency to get that data or that file
21:08:04or whatever it is. And so these are the
21:08:07different tiers that you can use. Now,
21:08:09we'll just use hot because that is the
21:08:11default option. But if you have a use
21:08:13case where it's not important that that
21:08:15data is quickly accessible, you don't
21:08:17need it right away, then these other
21:08:19options are going to be a lot cheaper.
21:08:21So, let's come down here. We're going to
21:08:23go ahead and upload these files. And
21:08:26there we go. Then what we're going to do
21:08:27is we're going to actually upload one
21:08:29more. And this is going to be for a
21:08:30future lesson when we actually access
21:08:33some of this data. We're going to browse
21:08:34for files. And we're going to select CSV
21:08:36file 2. I just made a copy of this. And
21:08:38all we're going to do is we're going to
21:08:39open this up. And for this, we're going
21:08:42to choose uh you can do any of these
21:08:44honestly, but let's put it in archive
21:08:46just to be have the most dramatic
21:08:47effect. Let's go ahead and upload this.
21:08:50And as you can see here in this access
21:08:52tier, we have hot inferred inferred
21:08:53inferred and then we have archive. And
21:08:55so later on in a future lesson when we
21:08:58try to access some of this data, we're
21:09:00going to try to access both of these and
21:09:01you'll see what actually happens when
21:09:04you have data stored in archive or cool
21:09:06or cold. It's quite similar. We'll see
21:09:08how these are retrieved. Now the next
21:09:10thing I want to take a look at is right
21:09:12over here under settings. Now under
21:09:14settings, you'll see this data lakeink
21:09:16gen 2 upgrade. Let's go ahead and click
21:09:18on this. It says that you can upgrade to
21:09:21a storage account with Azure Data Link
21:09:23Gen 2 capabilities. So, if you need
21:09:25things like data analytics and big data
21:09:27storage, you should consider upgrading
21:09:29to Azure Data Link Gen 2. Now, we're not
21:09:32diving into data links within Azure. I
21:09:34may do that in a future lesson, but this
21:09:37is where you can access to create a data
21:09:39lakeink. You can upgrade your storage
21:09:41account into a data lakeink. They used
21:09:43to have a completely separate data
21:09:44lakeink gen 1 which is what we just
21:09:46looked at earlier but now this is all
21:09:48located within the storage account. So
21:09:49you just upgrade from this location then
21:09:52you'll have those data lakeink
21:09:53capabilities. So that's just something
21:09:54that I wanted to mention while we're
21:09:56here. Now the last thing that I actually
21:09:58want to look at within the storage
21:10:00account is actually the IM which is the
21:10:02access control. Now, within here, you're
21:10:05going to have complete access to this
21:10:07because if you come over here and view
21:10:08your access, you can see that you're
21:10:10going to have grants full access to
21:10:12manage everything. And if you come over
21:10:14here, you can read it more. You you have
21:10:15access to everything cuz you created it.
21:10:18But what if we want other people to have
21:10:20access to this? Cuz right now, this is a
21:10:23private account. Nobody else can access
21:10:25this. If you want other people to have
21:10:27access to this, you're just going to go
21:10:28to add. You can add a role assignment or
21:10:30a co-administrator.
21:10:32Let's just say we're going to add a role
21:10:33assignment. You can say this person has
21:10:36the ability to read, but you can't make
21:10:38any changes. So, I'm going to click on
21:10:40this one. Then I'm going to come up here
21:10:41to members. Now, right now, I am the
21:10:44only person in my Azure account. So, if
21:10:47I wanted to add somebody, let's say Bob.
21:10:50If I wanted to add Bob, he'd have to be
21:10:52have an Azure account. But I would click
21:10:53on Bob and I'd say, okay, I'm going to
21:10:54give Bob this access. I'm going to
21:10:56select him and we're going to give him
21:10:58just the read access. We don't want him
21:11:00to, you know, delete all of our files by
21:11:02accident. He's not the brightest uh bulb
21:11:04of the bunch. So, we're just going to
21:11:06give him that access. Review and assign.
21:11:10And we'll go ahead and add that role
21:11:12assignment. Now, you can see that my
21:11:14access I am an owner, but I'm also a
21:11:16reader. And so, that is how you grant
21:11:17access to storage accounts. You can also
21:11:20create roles, deny assignments, uh
21:11:22create classic administrators, but
21:11:24typically this is done by a database
21:11:26administrator, but it is something that
21:11:29is extremely extremely frustrating about
21:11:31Azure just in general because any single
21:11:34tiny thing you want to do within Azure,
21:11:36you're going to have to request access.
21:11:37And so when you're first getting started
21:11:38up at a company, they're going to give
21:11:40you a lot of the base access. They're
21:11:42going to, you know, create your
21:11:43accounts. They're going to give you some
21:11:44access to PowerBI or the data lake or a
21:11:47SQL database. But whenever you want to
21:11:49use anything outside of that, you have
21:11:51to request IM access. And so that's just
21:11:53something I want you to be aware of
21:11:55because if you want access to specific
21:11:58storage accounts, you're saying, "Oh,
21:11:59this team, you know, wants me to work on
21:12:01their data or use something with their
21:12:03data, but I don't have access to it."
21:12:05That's because you weren't given access
21:12:06to it. You just have to request it or go
21:12:08to your database administrator, whoever
21:12:10runs that, to ask for permission. And so
21:12:12that's kind of the nuts and bolts of
21:12:14what most people are going to use
21:12:16storage accounts for. There are things
21:12:18like fileshares, cues, tables. Honestly,
21:12:20I you don't use them that much and so
21:12:22I'm not going to dive into it. These
21:12:24blob containers within storage accounts,
21:12:26the data lake, which uh is an
21:12:28upgradeable option, these are kind of
21:12:29the more important things. And so
21:12:31knowing how to store, where to store,
21:12:33and all these uh different options is
21:12:35very important. And so we'll be coming
21:12:37back to some of this data or putting in
21:12:39new data for different lessons when we
21:12:42actually start accessing data within a
21:12:44storage account. So, I hope that that
21:12:45was helpful. If you have not already, I
21:12:47have a full course on Azure and AWS over
21:12:49on analysts.com. I will leave a link in
21:12:52the description if you want to check it
21:12:53out. If you like this video, be sure to
21:12:55like and subscribe. I will see you in
21:12:57the next video.
21:13:11What's going on everybody? Welcome back
21:13:12to another video. But today we're going
21:13:14to be taking a look at SQL databases in
21:13:16Azure. [music]
21:13:22By now I think you all know how much I
21:13:24love SQL. I think it's one of the best
21:13:25skills for any data professional to
21:13:27have. But using it in the cloud is a
21:13:29little bit different than using it on
21:13:31your local computer. So in this lesson
21:13:32we're going to see how you can use a SQL
21:13:34database in Azure. Without further ado,
21:13:36let's jump on my screen and take a look.
21:13:37All right. So the first thing that we're
21:13:39going to do is we're going to come right
21:13:40in here into databases under the
21:13:42resources. Now we have a lot of
21:13:44different options in here and there's a
21:13:45ton of different databases that you can
21:13:48choose from and it kind of depends on
21:13:49what your company does. I'm only going
21:13:52to be showing you the SQL databases but
21:13:53other popular ones are things like using
21:13:55my SQL or Postgrace SQL with flexible
21:13:58servers as well as things like Azure
21:14:00Cosmos DB. They all have different use
21:14:02cases and they all have different ways
21:14:04uh that they are implemented. But by far
21:14:07the most common or the one that I've
21:14:08used the most in my career is SQL
21:14:11databases. So let's come right in here
21:14:13and what we need to do is we need to
21:14:15create a SQL database and let's click on
21:14:18create SQL database and let's actually
21:14:20create it and then we'll see how we can
21:14:21use it. So what we're going to do is
21:14:23come right down here to subscription. We
21:14:25have to select our resource group which
21:14:27you should have already created. Let's
21:14:29create a database name. Let's call this
21:14:31Alex the analyst DB for database. Now we
21:14:36have to select a server but we haven't
21:14:37created a server. So we need to create a
21:14:39new server. And again I'm going to call
21:14:41this uh we'll do ATA for Alex the
21:14:43analyst. I'll call this server. It looks
21:14:46like this needs to be lowercase. Let's
21:14:48do ATA server. And then we'll do YT at
21:14:50the end. Have to make it unique. There
21:14:52we go. All right. We're overcoming some
21:14:53hurdles here. Next we have to choose an
21:14:56authentication method. We can use the
21:14:58Microsoft Entra only authentication SQL
21:15:01and the entra authentication or just SQL
21:15:03authentication. Now what that means is
21:15:05is if you come in here and set your uh
21:15:07Microsoft Entra admin you can set it as
21:15:09yourself and that does help if you're
21:15:12already signed into Azure going to be
21:15:14using it within Azure. This can be very
21:15:16helpful. It's actually kind of the
21:15:18default u method. Let's get out of here.
21:15:20This is the default method. If you come
21:15:22down here though you can also create an
21:15:24admin login and a password. Both of
21:15:26these have their, you know, place and
21:15:28sometimes you need to use both. Um, and
21:15:31so just choose the one, the
21:15:33authentication method that you want for
21:15:34that server. For us, I think we're just
21:15:36going to stick with the, uh, Entra
21:15:38admin. We can always change that if we
21:15:40want to. So, let's go ahead and we're
21:15:42going to select ourselves here. We're
21:15:44going to select that. And there we go.
21:15:46Let's go ahead and click okay. And that
21:15:49should create our server. And there it
21:15:52goes. And now we need to finish creating
21:15:53our SQL database. So, do you want to use
21:15:56a SQL elastic pool? If you look at this
21:15:59tool tip right here, basically helps you
21:16:00manage your resources, but we're not
21:16:02going to be looking at the elastic
21:16:04pools. For our workload environment,
21:16:05we're just going to choose development.
21:16:07Production is going to be a lot faster
21:16:09because if it's in a production
21:16:10environment, you're going to want better
21:16:11speed, better compute, all these
21:16:13different things. Development is going
21:16:14to be a little bit slower. We can also
21:16:17choose a cheaper database or a more
21:16:20budget friendly database. So, we don't
21:16:22have to uh choose what it gives us. We
21:16:24can come in here and we can define this.
21:16:26So maybe you want to have a provision
21:16:28tier instead of serverless, which I
21:16:30don't really recommend. Uh serverless is
21:16:33uh quite nice for scalability, but we're
21:16:35just going to keep it at is. But if you
21:16:36want to come in here and change some of
21:16:38this configuration, you're free to do
21:16:39that. Uh just don't, you know, if you
21:16:42don't know what it is, I wouldn't mess
21:16:43with it. Let's come back to the create
21:16:45SQL databases. And then we have our
21:16:47backup storage redundancy. You can
21:16:49either do locally, zone, or geo. We're
21:16:51just going to stick with our local.
21:16:53Let's go ahead to review and create.
21:16:56It's going to tell us our cost, which is
21:16:58very, very, very low. Um, if you don't
21:17:00even have the free $200 that they're
21:17:03giving you, which will be uh which can
21:17:05be used for this, it's going to cost you
21:17:06like a dollar uh for, you know, what
21:17:09we're going to be doing or maybe even
21:17:10like 10 cents uh if I'm being honest.
21:17:12Let's go ahead and create this. It's
21:17:14going to take a little bit of time to
21:17:16set up all these servers and all the
21:17:17databases and all those things. And then
21:17:19once it is done, we'll take a look. All
21:17:21right, so our deployment is complete.
21:17:23You can come in here and look at some of
21:17:25the details. We created SQL databases,
21:17:27the server, SQL server, SQL server. And
21:17:28so all these things are ready to go.
21:17:31Let's go ahead and click on go to
21:17:33resource.
21:17:35And let's exit out of this. All this
21:17:38information is just an overview of our
21:17:40SQL database. Now, we have down here
21:17:44some of the more important things that
21:17:45we're going to be taking a look at.
21:17:46We're not going to be diving into all of
21:17:48them because uh I'm just going to show
21:17:49you the most common way. Now we have
21:17:52configure access connect application and
21:17:54start developing. Now in the real world
21:17:58when people are actually using the SQL
21:18:00databases and when they are getting in
21:18:02here and setting everything up you can
21:18:03do this in a few different ways. One is
21:18:06you can connect to a MySQL database and
21:18:09this is a very common practice where
21:18:11they connect it to a database management
21:18:12system. It could be MySQL Workbench or a
21:18:15ton of others that are out there. And
21:18:16you can do that by configuring it and
21:18:18you're going to get some of that
21:18:19information. You're going to plug it in
21:18:20and connect it. What we're going to be
21:18:22looking at is not that option, although
21:18:24uh that is something that happens often.
21:18:27I'm going to show you Azure's tool for
21:18:29this and it's going to be open Azure
21:18:31Data Studio. So, we're going to open up
21:18:32Azure Data Studio. We're going to click
21:18:34on this right here. And you're going to
21:18:36need to download the Azure Data Studio.
21:18:39Now, I already have this. So, I'm going
21:18:40to come down here and go to Azure Data
21:18:43Studio. And this should resemble a few
21:18:46different things. It should resemble a
21:18:48little bit of Visual Studio Code and it
21:18:50should resemble something like Microsoft
21:18:51SQL Server. It's kind of a combination
21:18:53of both. You have a search, you have
21:18:55some notebooks that you can use,
21:18:57different projects, an explorer, source
21:18:59control, extensions. It has a ton of
21:19:02stuff. And so, this was really popular
21:19:03when I was using Azure. Everybody used
21:19:05this. um as well as sometimes we
21:19:07connected to MySQL databases or uh
21:19:09Microsoft SQL Server databases and just
21:19:11use those database management systems
21:19:13but oftent times we would have
21:19:15everything in Azure data studio. So
21:19:18let's come right up here. We are going
21:19:19to connect. Now we have to specify our
21:19:22server name. Let's go back really
21:19:24quickly. We're going to come right over
21:19:25here.
21:19:27We're going to go back to uh this right
21:19:29here. We need to select our server. So
21:19:31this is our server. It's ATA server YYT.
21:19:34Uh we could even come in here into the
21:19:36server and we can just uh copy this if
21:19:38we want to. But we're going to get that.
21:19:40We have our server. We have our Windows
21:19:42authentication type. And so it can
21:19:44either be a SQL login, a Windows
21:19:46authentication, but we chose the
21:19:48Microsoft Entra ID. Now, right here,
21:19:50it's recognizing the analyst builder at
21:19:52Outlook.com. That was for the course
21:19:54that I have on analyst builder for AWS
21:19:56and Azure. We need to add in our Alex
21:19:58the analyst atlook.com. So let's come in
21:20:00here and we need to sign into our
21:20:02account. So let's go ahead and sign in.
21:20:03And there you go. Your account was added
21:20:06successfully. Let's go back. And there
21:20:08we go. Now we're signed in. And we need
21:20:10to select our database. Now, it's not
21:20:12popping up the database right away,
21:20:14which should be called like Alex the
21:20:16Analyst DB or something like that. Let's
21:20:18go ahead and try to connect and see what
21:20:20happens. It looks like we're getting an
21:20:21error here. I think let's actually come
21:20:24back here.
21:20:26I think our server name is actually this
21:20:28one right here. I just chose the actual
21:20:31server, but this is the connection that
21:20:33we actually need to make. So, uh I'm
21:20:35actually quite certain about that. Let's
21:20:36go ahead and click on this. And now it's
21:20:39saying our connection was denied since
21:20:40deny public network access is set to
21:20:43yes. So, this is something I was waiting
21:20:46to see because we need to configure this
21:20:48just a little bit. So, let's come over
21:20:50here and we need to go to configure
21:20:52access. So, we're going to select
21:20:55configure. And it says public network
21:20:57access is disabled, but we can enable
21:21:00this. And then what we can do is we can
21:21:02add in our IP address. So, we're going
21:21:04to add in your client IP v4 address. And
21:21:08all we have to do is click save. So, now
21:21:10it's going to update. And it's going to
21:21:11say, okay, you can access this. Let's
21:21:13not, you know, get too crazy and too
21:21:15wild here. Now, we can go back and we're
21:21:19going to select this. And we already
21:21:21have that selected. So now you can see
21:21:23that we have our two databases. We have
21:21:25the master which is the one that you're
21:21:26going to get and then we have the one
21:21:27that we actually created. So all of that
21:21:30to show that you do have to configure a
21:21:32few things. Make sure you're doing it
21:21:34properly. And now we can come down here
21:21:36and we can connect to it. And so now
21:21:39right in here we can come into our
21:21:41tables. We don't have any tables but we
21:21:43can come into these tables and views and
21:21:45uh all of these different things. Now we
21:21:47can actually use this. So now we are
21:21:48connected to our resource. So, we're
21:21:50connected to our server and we can
21:21:52actually access the databases, create
21:21:54them, do all of our querying, all of our
21:21:56uh things that we need to do with our
21:21:58data and we have all these options on
21:22:00the left hand side. Now, this isn't an
21:22:02Azure Data Studio tutorial, uh, but
21:22:04there's tons of stuff that you can do in
21:22:06here. So, if you've used something Azure
21:22:08Data Studio or if you've used Microsoft
21:22:10SQL Server, this should seem really
21:22:12familiar. You should feel right at home.
21:22:14Now, I want you to be able to actually
21:22:16use this. I don't just want this to be
21:22:18something pretty. So I need to show you
21:22:19one other thing that you need to do.
21:22:21Let's go ahead and try to create a table
21:22:24here. Right down here we have this
21:22:25script create new table. We can keep it
21:22:28new table just with the one column. Uh
21:22:30it doesn't really matter. Let's just say
21:22:32this is ready to go. Let's go ahead and
21:22:34publish these changes.
21:22:37Then we're going to come down here and
21:22:38we're going to update our database. Now
21:22:41we can come over here and we have a new
21:22:43table. So we can actually uh open this
21:22:45up. We'll select the top 10,00. And of
21:22:48course, we don't have any data in it,
21:22:49but uh we have a working table. So now
21:22:52this table is being stored on a server
21:22:54in a cloud. And this is great. So if
21:22:57you've ever used something like
21:22:58Microsoft SQL Server, you have all these
21:22:59tables and databases and all these
21:23:00things you're working with. That's how
21:23:01it's actually used in the real world,
21:23:03except you'd probably see a ton more
21:23:05tables. You'd have access to a bunch of
21:23:07different servers for different clients
21:23:09and different uh data. So that worked
21:23:11perfect. And what you can now do is
21:23:13let's do CtrlN. just get a new query
21:23:16window available. I'm going to paste in
21:23:18here just like this. Um, we're already
21:23:20selecting our database. We don't have to
21:23:22say use this database go. I'm just, you
21:23:25know, that's what I'm used to, so I'm
21:23:26going to keep it in there uh for any,
21:23:28you know, if you have a put this in a
21:23:29store procedure or something like that.
21:23:30I don't want it to fail out. But this is
21:23:32just a super simple table. Um, and we're
21:23:34going to go ahead and run this. Looks
21:23:36like that should be done. Let's refresh
21:23:39or actually refresh this table. And we
21:23:41have this products. Let's open this up.
21:23:44And now we have data in here. These are
21:23:46little under uh underlined in red.
21:23:49Sometimes if you do control ctrlalttr,
21:23:52it'll refresh it. Or maybe it's control
21:23:54shift r. That's okay. It'll get rid of
21:23:56it eventually. It just doesn't recognize
21:23:57it uh yet, but it will. Um so anyways,
21:24:00we have our data in here and now we can
21:24:02write regular queries. And so this isn't
21:24:05a SQL lesson. I'm not going to show you
21:24:06how to write SQL, but I have hundreds of
21:24:08other lessons and courses on how to
21:24:10learn SQL. And so this is uh kind of the
21:24:13nuts and bolts of how you set everything
21:24:15up and this is how people actually use
21:24:16it. So this shouldn't be too
21:24:18intimidating if you know how to use
21:24:19MySQL Workbench or Microsoft SQL Server.
21:24:22And so uh that is really awesome. Now if
21:24:25we come back here, we're just going to
21:24:26take a look at a few more things. This
21:24:28is just within our server, but we don't
21:24:30want to look um at our server. We want
21:24:32to go back to our database within our
21:24:36SQL database. right here. We already
21:24:38looked at a little bit of configuration
21:24:39and Azure data studio. Again, remember
21:24:42if you need to uh connect it to my SQL
21:24:44or something like that. Now, just within
21:24:46the SQL database, there are a few other
21:24:48things that you can do. One, they have
21:24:50something called a query editor um where
21:24:52you can come in here and you can query
21:24:54off of let's say you have tables you can
21:24:57query off of in here. I can assure you
21:24:59that almost nobody ever uses this,
21:25:01almost ever. Uh this is not really
21:25:03something that people use. It's there to
21:25:05kind of test connections sometimes. So
21:25:08if you're just setting up a new server
21:25:09or a new So if you're just setting up a
21:25:12new server or new database or whatever
21:25:14it is, you can kind of check to make
21:25:15sure it's working. But you won't use
21:25:17this in your real work. Uh that it's
21:25:19just not doesn't make sense. Now let's
21:25:21come over here and take a look at this
21:25:23lefth hand side. There are some
21:25:24interesting things that I want to show
21:25:25you. One is this power platform. When
21:25:28you start working with a large amount of
21:25:30data, you have a server and a database
21:25:32set up and you're using it. You have
21:25:33tons of real data in there. you're like,
21:25:34"Okay, now it's time to connect this."
21:25:36Well, you can use things like PowerBI,
21:25:38Power Apps, and Power Automate. All
21:25:41these things just kind of automatically
21:25:42integrate into it. There's also
21:25:44integrations, but just knowing how, you
21:25:46know, these are actually used. You most
21:25:47likely won't use them, but just knowing
21:25:49kind of what these are and how they
21:25:50work, you most likely won't use these
21:25:52too much. For people like you and I,
21:25:54PowerBI is something that we'll probably
21:25:55use quite a bit. And we most likely
21:25:59won't use monitoring too much, but I
21:26:02will say I've had to come into the
21:26:03monitoring quite a bit over my years to
21:26:05debug a bunch of stuff. So if a store
21:26:07procedure is failing, if the database is
21:26:09failing and you know you need to figure
21:26:11it out, you can come into the logs. If
21:26:13you need to see how much compute, how
21:26:14many resources you're using, you can
21:26:16look at the metrics. So there are some
21:26:17reasons to come in here. Typically
21:26:19though, this is more IT related. this
21:26:22isn't as much of what a data analyst
21:26:24will typically do unless you work like I
21:26:25said in IT where they monitor a lot of
21:26:28those things to keep cost down and keep
21:26:29things running and you know going
21:26:31smoothly. So, I hope that that was
21:26:32helpful. I really appreciate you guys
21:26:34watching. If you have not already, be
21:26:35sure to check out my full AWS and Azure
21:26:37course on analystbuilder.com.
21:26:40And if you like this video, be sure to
21:26:41like and subscribe below. I will see you
21:26:43in the next video.
21:26:57What's going on everybody? Welcome back
21:26:58to another video. Today we're going to
21:26:59be taking a look at Azure Data Factory
21:27:01in Azure. [music]
21:27:08Azure Data Factory is a super cool tool
21:27:10within Azure because it allows you to
21:27:12create data pipelines and different
21:27:13workflows to extract data and clean
21:27:15data, transform it, and put it places.
21:27:17And so it does a lot of different
21:27:18things. It's one of those skills I
21:27:20started using when I started getting a
21:27:21little bit more advanced in Azure, but I
21:27:24don't think you have to wait till you're
21:27:25really advanced as an analyst or a data
21:27:27scientist or engineer. I think you
21:27:28should start learning it now because
21:27:30it's a really really great skill to
21:27:31know. So with all that being said, let's
21:27:33jump on my screen and take a look. All
21:27:34right, so let's get started by coming
21:27:36right down here to analytics and under
21:27:38here we have data factories. Let's go
21:27:40ahead and click on it and let's create a
21:27:43data factory. So let's come in here. We
21:27:45need to give it a name. choose our
21:27:47resource group and we'll be rocking and
21:27:49rolling. So, let's call this the Alex
21:27:51the analyst and we'll do ADF just like
21:27:55that. For the region, we're good. And
21:27:57for version, we only have one option.
21:27:58So, let's go and review and create this.
21:28:02Now, as we know, this is going to be uh
21:28:03deploying this. It's going to take just
21:28:05a minute and then it'll be done and then
21:28:07we'll get going. And there we go. That
21:28:08literally took maybe 15 seconds. And so,
21:28:11uh just set up the data factory version
21:28:14two. And let's go to our resource. And
21:28:17there we go. Now, this should look uh,
21:28:21you know, fairly straightforward. We
21:28:22just have some of this information up
21:28:24here. We have our launch studio. And
21:28:26then down here, we have some of the
21:28:28monitoring. So, when you're actually
21:28:29running these automated systems,
21:28:31pipelines, everything that we're going
21:28:33to be doing, um, you have some data on
21:28:35it. And you can see some of the, uh,
21:28:37data on that. Now, they do have some
21:28:39quick starts, tutorials, template,
21:28:41galleries, and training modules. go
21:28:42ahead and take those cuz I've looked at
21:28:44a lot of these and they're really great.
21:28:45Uh, but what we're going to be doing in
21:28:48uh this lesson, which we're going to
21:28:50cover a lot of stuff, is I'm going to
21:28:52help get you set up. I'm going to help
21:28:53show you how to do different things. And
21:28:55there's going to be a lot of stuff that
21:28:56we cover. So, I'm going to be moving
21:28:58pretty quick, but let's go ahead and
21:29:00launch our studio. And here we go. So,
21:29:03this is the Azure data factory. There's
21:29:06a bunch of different things that we can
21:29:07do in here. We can ingest data. We can
21:29:09orchestrate. So create codef free data
21:29:11pipelines. And we can transform data.
21:29:13Now we're going to be looking at a lot
21:29:14of this but not all of it. So stick with
21:29:17me. The first thing that we are going to
21:29:19do is we're going to work on an
21:29:20ingestion because being able to pull
21:29:23data in is actually a pretty important
21:29:25thing to know how to do. We're just
21:29:26going to select run once and we're going
21:29:28to select next. Now uh there's a lot of
21:29:32different places that you can ingest
21:29:33data from where your source data is
21:29:36stored but for us we are going to select
21:29:39the data that we put in the last lesson.
21:29:41So in our last lesson we have put some
21:29:43data um in a SQL database which this is
21:29:46our server right here ATA server YT and
21:29:49we have this table right here. So we're
21:29:51going to ingest this data and so what
21:29:54we're going to do is we're going to come
21:29:55over here. We're going to type in uh
21:29:57SQL. Should have Azure SQL database
21:30:00should be right there. And we don't have
21:30:01a connection yet. So let's just select
21:30:03our new connection. This is basically
21:30:05like connecting to the Azure data studio
21:30:08uh before. And so this should seem
21:30:10really really familiar. So we have our
21:30:12ATA server YT. We have our Alex the
21:30:15analyst DB. And so now we should uh for
21:30:18our Azure subscription, we have our
21:30:19subscription. And let me go back because
21:30:21now it's not remembering. Uh so what we
21:30:24need to do now is we need to select our
21:30:27uh system assigned manage identity. So
21:30:30our manage identity is this one right
21:30:31here, Alex the analyst adf. Now that's
21:30:34going to be really important in just a
21:30:35second because this is by far the most
21:30:38confusing and you know frustrating part
21:30:40of doing this if you've never done this
21:30:42before. So let's just say we want to go
21:30:44ahead and create this. It was
21:30:46successfully created but we're getting
21:30:48this connection failed. And if we look
21:30:49at this, it says it cannot connect to it
21:30:52because basically it cannot open the
21:30:54server requested by the login. Now what
21:30:57we need to do is we need to edit this
21:30:59and we need to do something quite
21:31:00important. We need to take this manage
21:31:02identity name and we're going to
21:31:04actually update our database and make
21:31:07sure that this identity name is in there
21:31:08so it recognizes it and can connect to
21:31:10it. So let's go ahead and copy this and
21:31:13let's come right over here. Let make
21:31:14sure I have that. There you go. Let's do
21:31:16controlN in our Azure Data Studio. Now,
21:31:19this right here is the most important
21:31:21thing, but we have to put that in a few
21:31:23different places. So, I'm going to write
21:31:25out all the code and then I'll explain
21:31:26it to you and I'll have it to where you
21:31:27can just copy and paste it yourself. You
21:31:29don't have to write it all out. All
21:31:30right. So, I went ahead and wrote
21:31:31everything out. What we're doing is
21:31:33we're creating a user and that's going
21:31:35to be our user for Azure Data Factory
21:31:37that we have right here. So, we're
21:31:39adding that and then we're altering the
21:31:40role to make them a member so we can get
21:31:42access. Now, if we run just this, let's
21:31:45go ahead and run this. You'll notice
21:31:46that we don't have anything in this
21:31:49database principles and this database
21:31:50role members. This is our uh CIS
21:31:53databases. So, we're just checking there
21:31:55isn't uh that account in there. So, what
21:31:57we now need to run is this top part. And
21:31:59again, I'll have this as a copy and
21:32:00paste down below. Let's go ahead and run
21:32:02this. And looks like it worked. And now,
21:32:04let's check these two queries again. And
21:32:07there we are. So now we are in both of
21:32:10these uh CIS databases or CIS tables
21:32:12that we need to be in. So we should be
21:32:14good to go. Let's go ahead and go back
21:32:17and we're going to come right in here.
21:32:20And now that we have all that connected.
21:32:23Let's make sure let's see. Let's make
21:32:25sure everything's good in here. Let's go
21:32:28ahead and test this connection. And
21:32:29there we go. You can see that the
21:32:31connection is successful. Now, what we
21:32:34can do is cancel out of here because now
21:32:36this should be working.
21:32:39Now, what we need to do next is select
21:32:41what data we actually want in the
21:32:42output. Let's go ahead and select our
21:32:44products. We'll select next. You can
21:32:46preview the data if you want, but we
21:32:48don't need to do any of that right now.
21:32:50Let's go ahead and select next. Now,
21:32:51where are we going to place this data?
21:32:53Because we have data sitting over here
21:32:55in Azure Data Studio right in here that
21:32:59we want. And let's say we want to put it
21:33:02in the Azure blob storage. Maybe this is
21:33:04a report or some type of query that we
21:33:07want to send to a client. Uh, and so
21:33:09that's what we're going to do to get it
21:33:11into the Azure blob storage. We need to
21:33:13select our subscription and our account
21:33:16storage name. And we'll go down here to
21:33:19create. Next, we have to choose our
21:33:21folder path. So, let's pull up our
21:33:23storage account. We can duplicate this
21:33:26over here. And let's come back and let's
21:33:29see if we can get it right here. Here's
21:33:31our storage account. Let's go to Alex
21:33:33the Analyst storage. Let's go to our
21:33:35storage browser. We're going to go to
21:33:38our blob containers. Now, we should have
21:33:40uh one blob container in here. Here we
21:33:43go. Ata container. So, we can come back
21:33:45here. We're just going to select browse.
21:33:46I did all that just to show you where
21:33:48the data was coming from. Uh but that's
21:33:49in our uh storage account over here. Or
21:33:53is it called account storage? Storage
21:33:55accounts. If we just uh come in here and
21:33:59we say, "Okay, we want a file." That
21:34:00doesn't really work cuz we're putting
21:34:02data into something. So, we can't put it
21:34:04into a different file. We need to select
21:34:06a folder that we're going to place it
21:34:08into. So, we're going to select the ATA
21:34:10container. We're going to select okay.
21:34:11For our file name, we'll call this one
21:34:13SQL database output. And that should be
21:34:17good. Let's go ahead and select next.
21:34:20Now, these are the file format settings.
21:34:22We need to specify how we actually want
21:34:24this data to sit once we move it into
21:34:26the blob storage. Now, this is a
21:34:28delimited text. You can choose JSON
21:34:30files or C files, paret files, whichever
21:34:33files you want. So, we're going to do a
21:34:34comma delimited file, which is should be
21:34:36a CSV. We shouldn't need to add any
21:34:38compression onto it. Uh, because this
21:34:40isn't a massive amount of data or
21:34:42multiple files at all. So, we should be
21:34:44able to go ahead and select next. Next,
21:34:46we need to specify our task name. Now
21:34:49we're going to call this one uh SQL to
21:34:53blob
21:34:55and we should be good to select next.
21:34:57And this is the whole process. So we
21:34:59have Azure SQL database going to Azure
21:35:02blob storage. Let's go all the way down.
21:35:05We'll select next. And this is uh
21:35:07creating this. It says our whole
21:35:09deployment is complete. It validated the
21:35:11copy runtime environment, created the
21:35:13data sets, created the pipelines, and
21:35:15ran the pipelines. Go ahead and select
21:35:17finish. If we come over here to our
21:35:19storage, you can see that right here we
21:35:21have our SQL database output and uh
21:35:25let's see if we can just click into it
21:35:27real quick. Let's go ahead and download
21:35:28this just to see what it looks like. I'm
21:35:30just going to put this in downloads.
21:35:32We'll go ahead and save that. If we open
21:35:34up this file, uh just ignore those
21:35:36pictures of me and my wife on the
21:35:37Segway. Um if we open up this file, we
21:35:41can open it up as a let's just open up
21:35:43in Notepad. And you can see here's our
21:35:46data. Now, this isn't in a CSV format.
21:35:48Um, and that's okay because it's really
21:35:50easy to change it. But it is CSV uh
21:35:52separated. So, it was a text file. It
21:35:54meant to be I kind of want it to be a
21:35:56CSV, but we can very easily uh change
21:35:59that if we do CSV here. And now we've
21:36:02changed it into a CSV. Let's open it up.
21:36:04And there we go. So, uh you can
21:36:07definitely change that and we may
21:36:09actually look at that at some point in
21:36:10this lesson. But very easy to change
21:36:12that to a CSV file because it is comma
21:36:14separated. Um, so all this looks great.
21:36:16This looks really, really good. I don't
21:36:18need to save this. Let's go ahead and go
21:36:20back. Now, let's go back to Azure Data
21:36:22Factory. One thing I want to point out
21:36:24really quickly, uh, before we get into
21:36:26some other things is, uh, for our recent
21:36:29resources, we have our SQL to blob. You
21:36:31can find that right over here in this
21:36:34author. So, when we go over to author,
21:36:37you can find our pipelines and we have
21:36:39some data sets as well for our
21:36:40destination and our source data set. But
21:36:43that's regardless of what we're looking
21:36:44at in our pipelines. We built our SQL to
21:36:47blob. So if we come over here, this is
21:36:49what it's doing. And we can click on
21:36:51this and we can see uh kind of what it's
21:36:53doing. Down here we have our source
21:36:56data. That's where the data is coming
21:36:57from. That's our table. You can also
21:37:00have it write a query. So you can change
21:37:01this to a write a query from that table
21:37:04if you want to do some advanced stuff.
21:37:06Uh um you know, this data is super
21:37:07simple. But if we wanted to just select
21:37:09units in stock where it's greater than
21:37:11100, right? you can just write the query
21:37:13in here, copy it and put it right in
21:37:16here and then you can have a query
21:37:17instead of pulling over the whole table.
21:37:19And so that's actually uh really really
21:37:21useful. And so that is all really
21:37:23interesting stuff. Now within what we're
21:37:25looking at right here, we'll take a look
21:37:27at that in just a second. Um but what
21:37:29we're going to do is we're going to come
21:37:31over here to transform data. Let's go
21:37:33ahead and select this. You can see right
21:37:35up here we have our SQL to blob. That's
21:37:38our pipeline. And we have a bunch of
21:37:39stuff. And then we have our data flow
21:37:41right down here. Now what our data flow
21:37:44is is we're able to take different data
21:37:46sources, different things, transform it,
21:37:48join it, uh do aggregations, do anything
21:37:50we want to it, and then spit it out
21:37:51wherever we want. If we want to pull all
21:37:53these different files and put into a SQL
21:37:55database, we can do that. If we want to
21:37:57take a bunch of different data and we
21:37:59want to put it into a file, put into
21:38:01blob storage, we can do that. Or if you
21:38:04want to take multiple tables from a SQL
21:38:06database and then put it into a file, we
21:38:09can do that. And so it's kind of
21:38:11limitless what you can do within here.
21:38:13You just have to kind of know how it
21:38:14works and and kind of piece everything
21:38:16together. So let's start building this
21:38:19transformation. Let's come in here and
21:38:21we're going to add a data source. Now
21:38:24within our data source, we can come down
21:38:25here and we need to select a data set.
21:38:27Now these are two data sets that we've
21:38:29already used. uh these were in the ones
21:38:32that we did when we created the SQL to
21:38:34blob. Let's say we want to create a new
21:38:36data source. If we select a new data
21:38:38source, you'll notice all the different
21:38:40options for this, right? We have a ton a
21:38:42ton a ton of different places and
21:38:44applications that we can pull in. Um if
21:38:46you just want to scroll down, there's a
21:38:48lot. And so there's a lot of different
21:38:50things. Now, we're on a free tier, so um
21:38:52we may not be able to use some of these,
21:38:54but you know, if your company's paying
21:38:56for it, you should be able to do it. So
21:38:58let's just say we're going to take it
21:38:59from Azure blob storage. We're going to
21:39:02come right here. Ours is uh a delimited
21:39:05text. Let's go to our link service and
21:39:08go to our storage. And we need to select
21:39:11our data. So let's go to ATA container.
21:39:13Let's take this CSV file. CSV. So we
21:39:18just came in here. We're just selecting
21:39:20our data source. And you'll see we
21:39:21select it from our blob storage. And
21:39:23there's the path to it. Let's go ahead
21:39:25and select okay. And there we go. Now,
21:39:28one thing we will want to do as we're
21:39:30going throughout this process is to turn
21:39:31on data preview. So, we have to turn on
21:39:34the debug mode. Debug mode is right up
21:39:36here. So, data flow debug. And you can
21:39:38let it live for a certain amount of
21:39:40time. This does cost money. It's very
21:39:42cheap, but uh it's well worth it because
21:39:44as you're doing all these different
21:39:46things, you're going to want to preview
21:39:48the data and see what data looks like as
21:39:50a final result before you push it
21:39:51somewhere, before you put it into a
21:39:53database or a file. So, we're just going
21:39:55to wait for just a second while that um
21:39:57while that debug session starts and then
21:39:59we'll have our data preview available.
21:40:01Now, this is taking forever and I don't
21:40:03want to wait around forever to keep
21:40:04going. Uh it should work at some point
21:40:07and maybe we'll see it throughout the
21:40:09process, but uh we're just going to keep
21:40:11going because if yours is taking as long
21:40:13as mine, we don't need to wait on it
21:40:14because it's not vital. You don't have
21:40:16to have it. Um so, let's come down here.
21:40:19What we can do is we can add another
21:40:22source if we want to. You don't have to,
21:40:25but if you have multiple sources, you
21:40:26can add that. Uh so if you wanted to add
21:40:28a source down here, you just add uh
21:40:30another source. We aren't going to be
21:40:32doing that. So we can just delete that.
21:40:34Let's click this little button right
21:40:36here cuz there we're get a little
21:40:38different options than we have been uh
21:40:40before. We have a bunch of things that
21:40:43we can do to this data. So let's just
21:40:46scroll down really quickly. You should
21:40:48notice and you should be able to see a
21:40:50ton of these and one of the most
21:40:52important is actually this destination
21:40:54sync destination at the end and we'll
21:40:55take a look at in a bit. But we have all
21:40:57of these options to actually change and
21:41:00transform our data. And so if we want to
21:41:02clean the data, if we want to filter the
21:41:04data, if we want to aggregate the data,
21:41:06uh if we want to pivot the data, there's
21:41:07so many different things that you can do
21:41:09to it. Now, really quick, uh while we're
21:41:11here, I'm just going to show you the
21:41:12actual data that we're working with. If
21:41:14we open up this file, we have this CSV
21:41:16file. Let's go ahead and open it.
21:41:19And this is the data that is actually in
21:41:22that file that we upload. And there's a
21:41:24lot. There's about 32,000
21:41:26rows of data. We have things like state
21:41:29name, county, city, place, type, etc.
21:41:32And so there's a ton of data in here.
21:41:34Let's say we just want to filter this
21:41:36data. So let's go back. Let's go ahead
21:41:38to don't save. I just wanted to show you
21:41:39what it looked like. Let's go ahead and
21:41:41let's say we want to filter our data. So
21:41:43we're going to come down here. We're
21:41:44going to go to a row modifier which is
21:41:46filter. And so what we need to do is
21:41:48come down here and I highly recommend
21:41:50opening up the expression builder
21:41:52because what we can do is we can say we
21:41:54want to filter on one of these columns.
21:41:56Now over here in the dataf flow
21:41:58expression builder we have our
21:41:59expression up here and we have all of
21:42:01our elements. So we have things like
21:42:03functions, input schemas, parameters,
21:42:06cache lookups, dataf flow libraries. 99%
21:42:09of the time you're going to be using
21:42:11functions and your input schema. So our
21:42:13input schema is our table and the
21:42:15functions are all the functions that we
21:42:17have in here. So we can do one there's
21:42:20one called equals.
21:42:22Let's do this one right here. So we can
21:42:24do equals. You have your expression. You
21:42:25can even say over here here's some
21:42:27examples that you can do. But what we're
21:42:29going to do is we're going to go over
21:42:31here the input schema and we'll get rid
21:42:34of this. We have something called a
21:42:36state name. So if we come up here,
21:42:38select in there, select state name. So
21:42:40if we do state name and then in here
21:42:42let's do Alabama. So what we're saying
21:42:45is is we're going to filter where the
21:42:47state name is equal to Alabama. That's
21:42:50what we're going to do. So we're going
21:42:51to save and finish this and that should
21:42:54be good to go. Let's see if we have a
21:42:55data preview. This hasn't been working
21:42:57for me at all. I hope it's working for
21:42:58you. It's just taking it abnormally long
21:43:01time. Maybe because they're not
21:43:02prioritizing uh me as a small account,
21:43:05which is understandable, but um you
21:43:08know, thanks Azure. So now we have this
21:43:10source. We've filtered our data. And now
21:43:13and you can do so many different things
21:43:15in here. We're just not this isn't a
21:43:17full um you know lesson on how to use
21:43:20this. Now we need something called a
21:43:22sync. Now what the sync is is how you
21:43:24can actually save and publish it because
21:43:26you can't publish it without a sync. The
21:43:28sync is how you can specify where you're
21:43:30placing this data. So what we're going
21:43:33to do is we can place this anywhere we
21:43:35want. We can place this as a new file.
21:43:37We can put this in our SQL uh database.
21:43:39Now, if we come in here, we're going to
21:43:40have several different options here, but
21:43:43let's go ahead and select a new one and
21:43:47and let's say we want to put it back in
21:43:49the blob storage. So, we're cleaning the
21:43:51data up and we're going to save it as a
21:43:53delimited text file. Let's go ahead and
21:43:55select continue. So, now we want to put
21:43:58it in our blob storage. Again, we need
21:44:00to select our container. Container is
21:44:02going to be ATA container. You could set
21:44:04up another container if you want. Now, I
21:44:07don't think we want to select a file
21:44:09name because uh we're choosing the file
21:44:11path where we're going to place it. So,
21:44:12I think we just need to select okay
21:44:14here. Now, there's one last thing that
21:44:15we want to do before we actually publish
21:44:18all of this. Let's go to settings. And
21:44:21we have some options in here, which is a
21:44:23file name option. Now, when you're doing
21:44:27files in here, you in fact, you should
21:44:29probably publish this and just try it
21:44:30and see what happens. But often times
21:44:32you're going to want this as a single
21:44:34file, but it doesn't always happen when
21:44:36you're using Azure. Sometimes it'll make
21:44:38it 2, three, four, five, six files
21:44:40depending on what you're doing. So I
21:44:41like to output this as a single file.
21:44:44And then we get to output as a single
21:44:46file. And we can select the name. So we
21:44:47can just put this as we'll do filtered
21:44:50data. Uh, and that's it. I'm actually
21:44:53not sure if I need to do CSV right here.
21:44:56Let's just try it as CSV. Let's go ahead
21:44:57and try to publish this. It looks like
21:44:59it says the file name option output to
21:45:00single file requires single partition be
21:45:03selected in the partition type. So if we
21:45:05come over here I think it's in optimize
21:45:07for the partition option. You can use
21:45:09current partitioning but we need to
21:45:11select single partitioning. It now says
21:45:13it is fixed. We can close that and let's
21:45:16go ahead and publish all. So what we are
21:45:19going to do is we're going to review
21:45:20this. We have our different data sets.
21:45:22Um and then we have our data flow. So
21:45:25our data flow we're going to go ahead
21:45:26and publish. These are all changes that
21:45:28we've made uh throughout this process.
21:45:31It's going to deploy those and then it's
21:45:32going to uh run this data flow one and
21:45:35then we should see a new data set uh
21:45:37which is filtered based off of our
21:45:39source data. We should see that in our
21:45:42storage account. We should be able to
21:45:44come right up here and it looks like uh
21:45:47all those things are completed. Now I
21:45:49forgot we're actually not going to see
21:45:51this uh because it didn't actually run.
21:45:53It just saved as a data flow. We need to
21:45:55come up here and we need to create a new
21:45:57pipeline. So let's come up here, create
21:45:59a new pipeline. And what we're going to
21:46:02do is we're going to go down to our data
21:46:04flow. We're just going to drag this
21:46:06over. So this is our data flow. And we
21:46:08can name this anything we want. We can
21:46:10say uh transform.
21:46:13I need to spell that right. Uh transform
21:46:17data. Let's just call it this. And then
21:46:19we need to add a trigger. So we're going
21:46:22to go ahead and say trigger now. And
21:46:23that should run that data flow that we
21:46:25created. So let's go ahead and trigger
21:46:27now. And actually we need to publish it
21:46:30first. So let's go ahead and publish
21:46:31all. We'll publish this. And that
21:46:35publishing is completed. And now we can
21:46:37trigger it. So now we're going to say
21:46:39okay. And it's going to start running.
21:46:41And so this is one of the things just to
21:46:43kind of understand about building data
21:46:44flows and pipelines is that the pipeline
21:46:47you can do a lot of different things in
21:46:49it, but you're mostly chaining together
21:46:51different data flows. When you actually
21:46:52get in the data flow, you're chaining
21:46:54together all the different combinations
21:46:56that you want to do with different data
21:46:58sets and transformations and data
21:46:59cleaning and all these different things
21:47:01and then you put it into a pipeline. And
21:47:04that pipeline when it runs, which I
21:47:06think it's still running, when it runs,
21:47:08it runs that data flow that you created.
21:47:11And so these data flows can get really
21:47:13complex and you can chain multiple data
21:47:14flows together. You can say once this
21:47:16data flow is done running, then do this.
21:47:18Once this data flow is done running, do
21:47:20this next data flow. And so there's a
21:47:22lot of different things that you can
21:47:23chain together. And it's pretty awesome.
21:47:26So we're just going to wait on this for
21:47:27just a second. And when it is complete,
21:47:30we'll take a look at the output. And
21:47:32there you can see that it succeeded.
21:47:35Let's come over here and take a look.
21:47:37Let's go ahead and refresh this. Let's
21:47:39go to our blob container.
21:47:41And there we go. We have our filtered
21:47:43data.csv.
21:47:45So you can go ahead and check that out.
21:47:46Uh so those two things are working
21:47:49great. Let's go back home really quick.
21:47:52So we've worked on ingestion, we worked
21:47:54on data transformation, and lastly we
21:47:56have this orchestrate. And let's just
21:47:58come in here really quickly and take a
21:48:00look at how this looks. So as I was
21:48:02telling you before, we have these uh
21:48:04different pipelines that we've created.
21:48:06We've created a few extras that don't do
21:48:07anything. Um but what we can do is
21:48:10within this we can orchestrate this by
21:48:12taking multiple different pipelines,
21:48:14multiple workflows, multiple data sets
21:48:17and orchestrating all of it into an
21:48:20actual full pipeline. Now with that, and
21:48:22we're not actually going to be uh doing
21:48:23that right now, but with that, you can
21:48:25see that we have a ton of options right
21:48:28here. And in fact, you can take data
21:48:30from a ton of different places. You can
21:48:32copy data, you can do a Spark job
21:48:34function or even an Azure function. And
21:48:36you can customize so many different
21:48:38things and take data from so many
21:48:39different places and you can use so many
21:48:41resources within Azure to do just about
21:48:44anything you want. And so uh and I don't
21:48:46say that you know as in like you can
21:48:48actually do anything but just look at
21:48:50all these options. There are so many
21:48:51things uh that you can do. And so I've
21:48:55just given you a brief kind of
21:48:56introduction to ingesting data uh and
21:48:59transforming data. And then to build
21:49:01these out you just kind of combine those
21:49:03things together. So you say okay I want
21:49:05this uh pipeline right here and then if
21:49:08that works then we want to you know
21:49:10transfer it over here and actually we
21:49:12need to do something like this on
21:49:14success and so you can say if this works
21:49:16then on the success of that we'll do
21:49:19this and then just for an example we can
21:49:21say um we'll add this down here and
21:49:23we'll say if it fails we'll do this one
21:49:26and so this is just a demonstration of
21:49:28how this looks. This isn't actually what
21:49:30you should do by any means. Um, but you
21:49:33can give it some different instructions.
21:49:35You can say if this successfully works,
21:49:37if this pipeline works, then go do this
21:49:39piece. And then if this works, go do
21:49:41this piece. Oftent times you're going to
21:49:43chain these together. And this is the
21:49:44full orchestration of Azure Data Factory
21:49:46that we're not looking at in this
21:49:48lesson, but you can imagine uh you're
21:49:50taking data from a client. So the client
21:49:52drops data into a specific location. You
21:49:54ingest that data. So you have a data
21:49:56ingestion data flow. Once that data gets
21:49:59in, then maybe you have one for
21:50:01transforming your data. So if the data
21:50:03adjusts properly, then you're going to
21:50:04come up here and you're going to
21:50:06transform the data. Then you'll have
21:50:08another one afterwards. You'll be over
21:50:09here and you'll say once it's
21:50:10transformed and then we're going to send
21:50:12an email to me who is the person who
21:50:14owns that and saying, "Hey, this
21:50:16ingestion process worked or this data
21:50:18pipeline worked." And so you can get
21:50:19really advanced. You can also keep it
21:50:21really simple. And in my time as a data
21:50:23analyst, I worked with a ton of data
21:50:24engineers and data scientists and
21:50:27database developers who use this all the
21:50:29time. So I get in there and I get to
21:50:31mess around with it quite a bit. And
21:50:32oftent times it was mostly keeping it
21:50:34kind of simple. You're ingesting data,
21:50:36you're transforming the data, and you're
21:50:37placing it in somewhere. There were some
21:50:39use cases where we did a lot more
21:50:40advanced stuff, but this is the meat and
21:50:42potatoes of it. So play around with
21:50:43this, mess around with it, try to get
21:50:45these different things to work, try to
21:50:47create a full endto-end project. I think
21:50:49that'd be really cool. And maybe I'll do
21:50:51that in a future video. So that is in
21:50:54general what ADF kind of is. That's the
21:50:57meat and potatoes of Azure Data Factory.
21:50:59And I hope you're able to follow along
21:51:00and really understand that so you can
21:51:02start building on top of that and trying
21:51:04out your own stuff. If you found that
21:51:06helpful, be sure to check out my full
21:51:07Azure and AWS course on
21:51:09analystbuilder.com.
21:51:11And if you haven't already, be sure to
21:51:12like and subscribe below. And I will see
21:51:14you in the next video. [music]
21:51:27What's going on everybody? Welcome back
21:51:28to another video. Today we're going to
21:51:30be taking a look at Azure Synapse
21:51:31Analytics in Azure.
21:51:36[music]
21:51:38Now Azure Synapse Analytics is meant to
21:51:40be this all-in-one place to go for all
21:51:42your data analyst needs. It's going to
21:51:44have ETL. It's going to have workflows.
21:51:45It's going to be able to query data. So,
21:51:47it has a lot going for it. So, we're
21:51:49going to be taking a look at that and
21:51:51I'll talk a little bit about the
21:51:52comparisons between Azure Synapse
21:51:54Analytics and a few of the other
21:51:55resources and tools within Azure. So,
21:51:57with all that being said, let's jump on
21:51:58my screen and take a look. All right, so
21:52:00let's go down to our analytics tab.
21:52:02Let's go over to Azure Synapse Analytics
21:52:06and let's create a Synapse workspace. As
21:52:08you know, we need to select our
21:52:10subscription and our resource group and
21:52:12we need to name this. So we'll call this
21:52:14Alex the analyst and this is called uh
21:52:17oh wait I need to make them all
21:52:19lowercase. So we'll do Alex the analyst
21:52:22ASA Azure Synapse Analytics. That should
21:52:25be good. Now the next thing that we need
21:52:27to do is select a data lakeink storage
21:52:29gen 2 when we are using Azure Synapse
21:52:32Analytics. We are using a data lakeink.
21:52:34So we need to create an account name for
21:52:36this. So we'll do Alex the analyst ASA.
21:52:40That should work. Then we need a file
21:52:43system. Now we don't have one. So we're
21:52:45going to create the Alex the analyst FS
21:52:48and let's select okay. Now it says right
21:52:51here assign myself the storage blob data
21:52:53contributor role for the data lakeink
21:52:55storage gen 2 account to interactively
21:52:57query it in the workspace. So we're
21:52:59making oursel the person who is the main
21:53:01contributor so we can actually use this
21:53:03data lakeink gen 2 that we're using.
21:53:05Let's go ahead to review and create. It
21:53:08is going to cost us cuz we're using a
21:53:09serverless SQL per terabyte. It's going
21:53:11to cost about 5 USD for that and that's
21:53:14not a big uh cost for us. So, we're
21:53:16going to go ahead and create that and
21:53:17that should be covered under your 200.
21:53:19If you had the free $200 uh credit, that
21:53:22should be covered under that as well.
21:53:23All right, so that was deployed uh
21:53:26successfully. If you look in here, we
21:53:27did a few different things or it did a
21:53:29few different things. One, it created a
21:53:30new storage account for us and that's
21:53:32going to be our data lake. Next, it
21:53:34created our Synapse workspace. So, we
21:53:36have the new storage account and we have
21:53:37a new workspace. So let's go to this
21:53:39resource group here and let's actually
21:53:41come into our Synapse workspace and you
21:53:44can see a lot of our information up here
21:53:46but then the most important part which
21:53:48is open Synapse Studio. So let's go
21:53:50ahead and open this up. Then we need to
21:53:52go ahead and sign in here. Now let's
21:53:54just take a look at some of the things
21:53:56that's in here because this should seem
21:53:59pretty reminiscent of Azure Data
21:54:02Factory. And I say that because in our
21:54:04last lesson we looked at Azure Data
21:54:06Factory and we had a lot of these
21:54:07different things. We had things where we
21:54:08can ingest data. We could transform data
21:54:11and then we can place it somewhere. And
21:54:12believe it or not, you're able to do
21:54:14that within Azure Synapse Analytics as
21:54:16well. Not only that, within Azure
21:54:18Synapse Analytics, we're going to be
21:54:20able to explore our data and analyze it
21:54:23as well. You can come in here and see
21:54:24how all that works, but this is going to
21:54:26act very much like an Azure Data Studio.
21:54:29If you remember from our SQL databases
21:54:31video, we looked at Azure Data Studio
21:54:33and how that works and how you can
21:54:35actually access the data. And we'll be
21:54:37coming back to uh some of these things
21:54:39in a little bit. So, we're going to keep
21:54:41this up. Let's come over here and let's
21:54:43take a look at our data. Right now, we
21:54:45have no data in here. So, let's come in
21:54:48here. Now, if we connect to a SQL
21:54:49database, we're actually creating a SQL
21:54:51database or a lake database, but we
21:54:53don't need that. We're going to connect
21:54:54to external data. In here, we can
21:54:56connect to our Azure blob storage. So,
21:54:58we're going to go ahead and say
21:55:00continue, and we're just going to bring
21:55:02in some of our data that we've been
21:55:04working with. So, we're going to go to
21:55:05the Alex analyst storage. This is where
21:55:08we have some uh data in here and we need
21:55:10to select our subscription right there.
21:55:13There we go. We'll go ahead and create
21:55:15that. So, now if we come over here in
21:55:17our workspace, if we go over here to
21:55:19linked, we have our uh blob storage.
21:55:22Then we have our data link. Now within
21:55:24our data lakeink uh we don't have any uh
21:55:27data but if we come up here to the blob
21:55:29storage and so as you can see here we
21:55:32have all of our data that we have in
21:55:34that blob storage. If we wanted to
21:55:36access one of those we can do something
21:55:38like select new script. We do a new
21:55:41notebook where we can load to a data
21:55:42frame or a spark table. We can create a
21:55:45data flow. We can take that as a source.
21:55:47We can transform the data and we can
21:55:48load the data to somewhere else. So
21:55:50there's a lot of things that we can do
21:55:52with this data. One other thing I want
21:55:54to show you just with the data section
21:55:56is we have these browse galleries. And
21:55:59within here, there's a ton of real and
21:56:02really unique data that we can uh use
21:56:05and we can look at. For example, let's
21:56:06come over here to New York City uh
21:56:08safety data. Let's go ahead and click
21:56:10continue. There's a bunch of data in
21:56:12here. We're going to add this data set.
21:56:14It's going to take just a second, but it
21:56:16looks like it was created in this sample
21:56:18data sets uh in Azure blob storage. If
21:56:21we come over here, we can click on city
21:56:23safety New York. Let's go ahead and
21:56:25right click on it, select top 100 rows,
21:56:28and we'll go ahead and run this. As you
21:56:30can see down below, we're just looking
21:56:31at, you know, data. So, this is how we
21:56:34can run it with SQL. And with this, we
21:56:37can do just about anything we want,
21:56:38right? If we wanted to do some type of
21:56:40group by or joins or anything we want to
21:56:43do, you can do it. Uh, this is going to
21:56:45be by default SQL Server. Um, so you can
21:56:48do a lot of things uh that you want to
21:56:50do in there, but let's just say we
21:56:51wanted to group by category
21:56:55um and subcategory.
21:56:58And then up here, we'll select the top.
21:57:01We'll just do all of it, but we'll do
21:57:03select category and subcategory.
21:57:07And then we'll just do let's do a count
21:57:10of the subcategory.
21:57:13We're just going to get a count of it.
21:57:14Let's go ahead and run this. And let's
21:57:17take a look. Okay, it looks like our
21:57:20output is correct. Uh I was getting
21:57:21worried at first when I saw the nulls at
21:57:23the beginning, but we could probably uh
21:57:25clean this up a little bit, but we don't
21:57:27need to. Okay, we're just taking a look
21:57:29at what Azure Synapse Analytics can do.
21:57:31So this is where we can query data. We
21:57:34can also save this. We can publish this.
21:57:36And now that's been saved. So if I came
21:57:38over here to the develop, you can see we
21:57:40have SQL script one. Now we just got to
21:57:42change the name of this if we want to
21:57:44change it to something else. We can come
21:57:45over here. We can say rename. We can sol
21:57:48say this is the uh was this New York
21:57:50City. We'll say group by uh category.
21:57:54We'll just select okay. And there we go.
21:57:56So now we've saved that script. And so
21:57:57that's really important. That's really
21:57:59really useful. One other thing that we
21:58:01can do down here is we can also look at
21:58:04this as a chart. Now, this data in and
21:58:06of itself isn't very helpful as a line
21:58:08chart, but if we did a bar chart for the
21:58:11category column was the category and
21:58:13maybe we just want to select the top 10.
21:58:18We'll order by let's do the count of
21:58:22subcategory.
21:58:24Let's try running this again and we can
21:58:25go look at the table really quickly. And
21:58:29I need to do this descending. Whoops.
21:58:31All right, let's run this again. All
21:58:33right. So, there's the day that we want.
21:58:35Let's go back to this chart. Uh, again,
21:58:38this isn't a good visualization. We
21:58:40would do something like a bar chart. Uh,
21:58:42we would have the category here, and
21:58:44that'll be fine. But you can customize
21:58:46this a little bit more if you'd like to.
21:58:48Um, but we're able to look at little
21:58:50visualizations in our output when we're
21:58:52working with the SQL scripts. If we want
21:58:54to, we can save that as an image. And
21:58:56that would be great. Again, we can
21:58:58publish this and we can save this and we
21:59:01are good to go. Now the next thing that
21:59:04we can do, let's go back to our data.
21:59:06Next thing we can do is we can also
21:59:09create a notebook. So let's load this
21:59:11into a data frame. And you're going to
21:59:13notice this looks a little bit
21:59:14different. Now before we can actually
21:59:16run this, we have to create or attach a
21:59:19spark pool. We don't have one. So we're
21:59:21going to go create one uh really
21:59:22quickly. And we'll just call this uh ATA
21:59:26pool. There we go. We're going to go
21:59:28back down to review and create. going to
21:59:32create this.
21:59:34That was successfully deployed. So, our
21:59:36uh pool is ready to go. So, let's go
21:59:38back here. We're going to go to our ATA
21:59:41pool and we're going to choose spice
21:59:43spark. That's the one I've almost always
21:59:45used. You can also use SQL in here. Now
21:59:48um SQL in uh Azure Synapse Analytics or
21:59:51in other tools maybe like data bricks
21:59:53and other things like that they don't
21:59:55always have a SQL option but now it's
21:59:56being added a lot more but I used to
21:59:59have to use Scola and Pispark and all
22:00:01sorts of things but now if you know SQL
22:00:02you can do that in here too. Um so
22:00:04you're good to go. We're going to this
22:00:05is Pispark though. So we're going to go
22:00:07ahead and run this. And right here we're
22:00:09running into an issue. Your Spark job
22:00:10requested 24 vores. However, the
22:00:11workspace has a 12 core limit and that's
22:00:14because we need to message Azure support
22:00:16and request more. Um, and unfortunately
22:00:19for this video, we're not going to do
22:00:21that. So, let's get a smaller data set.
22:00:23So, we're going to come back here to our
22:00:25storage and let's add a new file.
22:00:30It's going to be this products file
22:00:32right here. Let's go ahead and upload
22:00:34this. And there we go. Now, we can go
22:00:37back here. We're take a look within our
22:00:40container. And we have this
22:00:42products.csv. So let's right click on
22:00:44this. Let's go to new notebook. You can
22:00:46load to dataf frame or spark table.
22:00:48Whichever one we're going to load this
22:00:50to a spark table. Let's choose our pool.
22:00:54And we're using Python. So now let's go
22:00:56ahead and try to run this. Now this
22:00:57didn't work. It says either the cores of
22:00:59the memory of the driver executes
22:01:00exceeded the spark pool node size. So
22:01:03for this what we're going to do is we're
22:01:04going to come back here. We're going to
22:01:06edit this. We're going to make it just
22:01:07the smallest one possible because we're
22:01:10just exceeding uh what we are working
22:01:13with right here. So now that's updated.
22:01:15Let's go back.
22:01:17Let's come right here. All right. And we
22:01:20got our output. Took a while, but that's
22:01:22cuz we're using a really small pool over
22:01:25here. Now, if you use a larger pool,
22:01:28it'll go a lot faster. Uh so if you
22:01:31select a ton more nodes and uh you get a
22:01:34larger node size just in general, it's
22:01:36going to go a lot faster. And so right
22:01:38now, oops, let's get out of here. Right
22:01:41now, uh before when we tried it with
22:01:44this one, it wasn't working because our
22:01:46node sizes were just too large for what
22:01:49they're allowing us to have. We have to
22:01:50again ask them to increase that. But
22:01:52again, that's mostly for really large
22:01:55data sets. For this one, we have a
22:01:57really tiny data set. and we get our
22:01:59data right down here. And so everything
22:02:02is working properly. But what we can do,
22:02:04which is really neat because this is a
22:02:06notebook, is we can come down here, we
22:02:08can write more code. So if you use
22:02:09something like a Jupyter notebook or
22:02:10there's tons of other notebooks out
22:02:12there that are really great. Um, if you
22:02:13use a notebook, you know, you have some
22:02:15code up here. You can put code down
22:02:17here. You can also put markdown. And so
22:02:19you can add in, you know, any title that
22:02:21you want to uh add in here. You can say
22:02:24this is a title. I'll go ahead and run
22:02:27this. And another really cool thing
22:02:29about using a notebook, especially
22:02:31within here, is that you don't just have
22:02:33to use Python. If you uh for some reason
22:02:36you say, okay, I want to use SQL, you
22:02:38can do that using this magic command,
22:02:40and you can says SQL and then you can
22:02:43just write SQL like normal. So you can
22:02:45select everything. We'll come right down
22:02:47here. We'll say from and right now it's
22:02:49called dataf frame. But I don't know if
22:02:52that's actually going to work. Let's go
22:02:53ahead and run this. And that's not going
22:02:55to work because it's not uh in a table.
22:02:58So we can do dataf frame dot and we can
22:03:00say create and there's this option for
22:03:03create or replace temp view and then we
22:03:06can name it whatever we want. So we can
22:03:08say this dataf frame to table. So we'll
22:03:10call this uh that right there. Let's go
22:03:12ahead and run this and then we'll come
22:03:14down here and replace that right here.
22:03:17Now we can run this and we just created
22:03:20it into a temporary view. And now we can
22:03:22run this just like we're running SQL.
22:03:24And so there's lots of options that you
22:03:26can do with this. Um I have a hundred
22:03:29tutorials on using Jupyter notebooks and
22:03:31how to use uh Python. It's very similar
22:03:33within this. And so it's really neat to
22:03:36be able to combine the two uh languages.
22:03:38I guess if you want to call SQL a
22:03:40programming language, it's technically
22:03:42not. Um but we have Pispark, so we can
22:03:44use Python and we have this as well. And
22:03:47so lots of options, lots of really good
22:03:50stuff in here. And so that's a big part
22:03:51of working with data within Azure
22:03:54Synapse Analytics. Now if we come down
22:03:57here to integrate, we can create now
22:03:59pipelines, link connections and copy
22:04:02data tool. Not using them, but we can
22:04:04use these things. And this is very very
22:04:07similar to Azure data factory. There are
22:04:09also some really good uh options in here
22:04:12if you want to kind of copy these. Let's
22:04:15say right over here it says bulk copy
22:04:16from files to database. If you want to
22:04:18do that, you can come in here, click
22:04:22continue, and then you can use this. So
22:04:24you can open this pipeline. You can say
22:04:26this is what I'm looking for. And you
22:04:28select your link service and your other
22:04:30link service. You open the pipeline, it
22:04:32creates that pipeline for you. So just
22:04:35something to be aware of. I think that's
22:04:37really awesome. And in fact, I've used
22:04:39these uh many times. Let's go ahead and
22:04:41click back. So there's tons and tons of
22:04:44just sample things where if you are
22:04:46doing one of these, let's say you want
22:04:47to create a system that deletes files
22:04:49older than 30 days. There you go. And so
22:04:51there's tons of things in here that you
22:04:53can use. Um as well as there's different
22:04:55SQL scripts you can use, different
22:04:57notebooks, different data sets. These
22:04:59are all free things that you can use
22:05:01within Azure Synapse Analytics. But
22:05:04let's get out of here. Let's go ahead
22:05:06and close this out. So let's say we do
22:05:08actually want to use something in
22:05:09integrate. Let's uh use this copy data
22:05:12tool. And this should look extremely
22:05:15extremely familiar if you use the Azure
22:05:18data factory uh if you took that lesson.
22:05:21Now, we're just going to copy it from
22:05:22blob storage.
22:05:24We'll go to next. We'll go to browse and
22:05:28we'll just select a file.
22:05:31So, we're going to come in here. We have
22:05:32our products uh data. So, we're going to
22:05:35take that products data. We're going to
22:05:37select next. and it wants to know what
22:05:39our file format is. Right now, we have
22:05:40it kind of as a CSV and we don't need
22:05:43any compression. So, we're going to go
22:05:44ahead and keep it as is. And we need to
22:05:46select our destination. Now, we can put
22:05:48this in almost any place we want. In
22:05:51fact, Azure uh SQL database would be a
22:05:53perfectly good spot for it. We're just
22:05:55demonstrating how to move one thing to
22:05:58another. So, I'm going to select Azure
22:05:59blob storage. We're just going to place
22:06:01it in the exact same one, exact same
22:06:03container. We're going to make some uh
22:06:05weird copy of this. Let's go ahead and
22:06:07click okay. And we need to select the
22:06:10file name. Now the file name is going to
22:06:12be products_copied.
22:06:14It's all we're going to do. CSV. Uh so
22:06:18now we'll select next. We'll select
22:06:20next. And our task name is going to be
22:06:22copy data. We'll do underscore. So copy
22:06:25data. And we'll select next. And we're
22:06:27going to select next. Now I'm not going
22:06:30to deep dive into the transformation and
22:06:32the pipelines and all those things
22:06:33because that's what we did in Azure data
22:06:35factory and in Azure Synapse Analytics.
22:06:38It's very very similar. Now you can see
22:06:40here we have this pipeline built. We can
22:06:42come right over here and this pipeline
22:06:44is ready to go. All we have to do is we
22:06:47have to publish it and then add a
22:06:49trigger. So let's go ahead and publish
22:06:51all. We're going to go ahead and publish
22:06:55this. Now that we've published it, we
22:06:56can add this trigger. We're going to say
22:06:58trigger now. We're going to click okay.
22:07:01So, as we looked at in Azure Data
22:07:03Factory, once we publish it, it saves
22:07:05it. We can add this trigger. We can run
22:07:07it to trigger it now to actually
22:07:09activate that process. And now, it's
22:07:11going to copy that data over. Now, that
22:07:13should take a little bit of time, but we
22:07:15can view that pipeline running. And it
22:07:17looks like it just succeeded. We can go
22:07:20back to our storage account. We're going
22:07:22to refresh our storage uh really
22:07:26quickly. go back to our blob containers
22:07:29and there you can see we have
22:07:30products_copied.csv.
22:07:33So even within Azure Synapse Analytics
22:07:36we can create these workflows and these
22:07:38pipelines. So let's go back over here
22:07:41back to the integrate. So this integrate
22:07:44allows us to do just about almost
22:07:46everything that we were doing within
22:07:48Azure data factory. It is a little bit
22:07:50more limiting but we copied some data.
22:07:53That's what we worked with. And there's
22:07:54a bunch of other things in here as well.
22:07:56You can create full data pipelines
22:07:58within Azure Synapse Analytics just like
22:08:01you can Azure Data Factory. But within
22:08:04Azure Synapse Analytics, we can also use
22:08:06the data. So we can take a look at the
22:08:08data and we can uh run queries on it and
22:08:11we can create different notebooks as
22:08:13well. And this is really awesome stuff
22:08:15that we're able to do. So Azure Synapse
22:08:17Analytics is really meant for a more
22:08:19mature data team. They're using large
22:08:22amounts of data. Again, this sits on top
22:08:24of a data lake. So you're creating all
22:08:26these pipelines, you're working with all
22:08:27these different data sets and you're
22:08:29querying the data, you're using the
22:08:30data, you're transforming the data.
22:08:32That's what this is really really for.
22:08:34And so if we come back over here to the
22:08:36home, you can see that we ingested some
22:08:39data. We explored and analyzed some
22:08:41data. We also have visualized data. Now,
22:08:44if we click in here, we can connect to
22:08:46PowerBI. We don't have an actual
22:08:48workspace for PowerBI set up, so we
22:08:51can't do that. But this is meant to be
22:08:53something that you're supposed to do.
22:08:55You're supposed to be able to ingest it,
22:08:56explore it, and then visualize it all in
22:08:59one place. So, it makes the work for
22:09:01data analysts or even data scientists
22:09:03quite easy. If you have kind of a
22:09:05smaller team or depending on your team,
22:09:07what data and tools you use, you may be
22:09:09able to do most of your work within
22:09:11Azure Synapse Analytics. Again, just
22:09:13depending on uh, you know, how
22:09:15everything's configured. But if you have
22:09:16a lot more complex systems, a lot larger
22:09:19data sets coming from a lot of different
22:09:21data sources, you may need to go branch
22:09:24out and use some of these other tools
22:09:25like SQL databases. Azure data factory
22:09:27as standalone tools, but sometimes
22:09:30you're able to get it all within Azure
22:09:32Synapse Analytics, all your ingestion,
22:09:34your data analysis, and your data
22:09:36visualization all in one place. And so
22:09:38it's a really powerful tool, and this is
22:09:40one that I used for quite a while. And
22:09:42so, uh, just coming in here, this is
22:09:45really bringing me back to one of my
22:09:46previous data analyst jobs where we were
22:09:48in here all the time. Now, one thing to
22:09:50note, and I didn't really mention this,
22:09:52is that this is actually a workspace.
22:09:54And so, uh, we have this workspace that
22:09:57you can then share and you can
22:09:59collaborate with people on. And so, if
22:10:01you're over here in a notebook and
22:10:02you're trying to run it, it's not
22:10:04working or, you know, whatever it is.
22:10:06Um, we can run these run this one real
22:10:08quick. It'll take a second. Um, we can
22:10:10run these and you can collaborate with
22:10:13other people and they can see your code.
22:10:14So then you can, you know, publish this
22:10:16and you can save this and all these
22:10:18different things and other people can
22:10:20view this. And so it's kind of more of a
22:10:21collaborative environment as well, which
22:10:24is really nice. So get in here, mess
22:10:26around with it a little bit. This is
22:10:28kind of just a little crash course on
22:10:30how to get up and running and some of
22:10:31the features of Azure Synapse Analytics.
22:10:35And this is a really great tool to know
22:10:36how to use. And I hope that that was
22:10:38helpful. I hope you got kind of a good
22:10:40overview of how Azure Synapse actually
22:10:42works. If you have not already, be sure
22:10:44to check out my full course on Azure and
22:10:46AWS on analybuilder.com. Be sure to like
22:10:48and subscribe below and I will see you
22:10:50in the next video.
22:10:53[music]
22:11:04What's going on everybody? Welcome back
22:11:05to another video. Today we're going to
22:11:06be starting our AWS series.
22:11:14Now AWS to me always seemed really
22:11:16complicated compared to Azure because I
22:11:18started with Azure and I knew it really
22:11:20well. And so when I started using AWS,
22:11:23it just seemed like a whole another
22:11:24world. So if you've already gone through
22:11:25my Azure series, that's really good
22:11:27because I'm going to reference it quite
22:11:28a bit to make some comparisons, but we
22:11:30still will be focusing mainly on the AWS
22:11:33portion of it. In this series, we'll be
22:11:34getting everything set up. We're looking
22:11:36at S3 buckets, Amazon Athena, Glue, Glue
22:11:39Data Brew, and Quicksite. And so that's
22:11:41a lot of different tools within AWS. And
22:11:43these are ones that I think are really,
22:11:44really important to know how to use. In
22:11:46this video, we're going to be creating
22:11:47an account and doing a walkthrough of
22:11:48the user interface. So without further
22:11:50ado, let's jump on my screen and get
22:11:52started. All right, so what we're going
22:11:53to be doing is creating an AWS account,
22:11:55looking at the UI within AWS, just
22:11:57getting familiar with it before we jump
22:11:59into some of the tools within AWS. Now,
22:12:02here I'll have this link down in the
22:12:04description. we are going to get started
22:12:06for free. So let's come right here and
22:12:08what we want is the AWS free tier. With
22:12:11the free tier, we get a lot of things
22:12:13within AWS completely for free. Of
22:12:15course, there are some things that are
22:12:16not underneath the umbrella of the free
22:12:18tier and so we won't be using those, but
22:12:21we will be able to do everything in this
22:12:22series with just the free account. So
22:12:25let's go ahead and create our free
22:12:27account.
22:12:28We need to sign up for AWS. Let's go
22:12:31ahead and verify this email address. So,
22:12:33I'm going to use my Alex
22:12:34theanalystytgmail.com
22:12:38and my account name is going to be Alex
22:12:41thean analyst. We'll do AWS. I got the
22:12:45code. I'm going put it in here and
22:12:47verify this. Now, we need to select our
22:12:49password. So, I'm going to put my
22:12:50password in right here.
22:12:54There we go. Let's go ahead and click
22:12:56continue. Now, within this free tier, we
22:12:58get a few different things. one uh the
22:13:01free services that were within AWS,
22:13:03they're never going to expire. You'll
22:13:04get them free always, which is really
22:13:06great. Um we also get 12 months free for
22:13:09certain services, and we'll look at that
22:13:10in just a little bit. And these things
22:13:12activate from when our trial actually
22:13:14starts. So, let's go ahead and fill out
22:13:17all this information, and then we will
22:13:20continue. Now, if we do use any
22:13:22services, we have to have billing on
22:13:24hand in case we use something that is
22:13:26not free or we go above the free tier.
22:13:29And so you have to input some type of
22:13:30credit or debit card number just in case
22:13:32you do that. Within what we're doing, we
22:13:34shouldn't be doing that or if we do,
22:13:35it'll cost like 10 cents. And so it
22:13:37should be super super cheap. So go ahead
22:13:39and fill in your information here. Next,
22:13:41we need to confirm our identity. I'm
22:13:43going to have it send me a text message
22:13:45so that I can uh fill this out. So go
22:13:48ahead and do that as well. All right,
22:13:50we're going to go ahead and select
22:13:51continue. And now we need to sign up for
22:13:53a support plan. Now, we're using a free
22:13:55account, so we don't really need
22:13:57support. Now, if you are just you're
22:13:59feeling wild, uh, and you want to get
22:14:01something like developer support, you
22:14:02can. If you're encountering issues, um,
22:14:05or if you're using AWS for business,
22:14:07maybe you want business support. Uh, you
22:14:09know what? Go for it. I'm not going to
22:14:10stop you. But we're going to be using,
22:14:12uh, the free support because, uh, I
22:14:14don't want to pay for it. So, let's go
22:14:16ahead and complete our sign up. And just
22:14:18like that, we have created our AWS
22:14:21account. Let's go to the AWS management
22:14:23console. Now, we are actually going to
22:14:25sign in with our account. Now, we are a
22:14:27root user. So, we're going to come in
22:14:29here. We're going to put in our email
22:14:30address and sign in. All right, we are
22:14:32all signed in. Let's go ahead and click
22:14:35next. Done and done and done and get rid
22:14:38of all this stuff. Now, this is our
22:14:39console home. You'll notice that it's
22:14:42very, very blank. We don't have anything
22:14:43that we've recently visited. We don't
22:14:45have any applications running. We don't
22:14:47have any cost uh going either. And so
22:14:50once we actually start using some of
22:14:52these services when we get into glue and
22:14:54start automating things you know you
22:14:56might have a cost but it should be under
22:14:57the free tier but you will have a cost
22:15:00when we start using databases and
22:15:02instances and all these different
22:15:03things. These things are compute and
22:15:06resources that AWS offers. So we'll be
22:15:08able to see and monitor a lot of those
22:15:10things within this home console right
22:15:13here. You can also customize this
22:15:15console home if you want to. So, if you
22:15:17want to come in here and add widgets,
22:15:18you can add different metrics or
22:15:20different things that you want in here,
22:15:22but we're not going to be doing that in
22:15:23this lesson. So, let's take a look at
22:15:25this UI really quick. On this lefth hand
22:15:27side, we have services. If we come here,
22:15:30this is all the services that AWS
22:15:32offers. And I'm just going to kind of
22:15:34slowly scroll down. We have compute,
22:15:36containers, storage, databases, machine
22:15:39learning. Let's keep going down a little
22:15:41bit. We have analytics. For this series,
22:15:44we're going to be focusing on things in
22:15:45this analytics tab. So, right in here,
22:15:48we're also going to be looking at some
22:15:50stuff in the databases and then of
22:15:53course S3 for S3 buckets. And this is
22:15:56where you can access all the resources
22:15:58and all the services within AWS. Another
22:16:01thing to note is this right up here,
22:16:03which is our region. Now, I'm in US
22:16:06East, but make sure you have the
22:16:08appropriate one that you're using. the
22:16:10region actually is uh pretty important
22:16:13if you choose one that's really far
22:16:14away. You're going to have some latency
22:16:16and some delays on retrieving data or
22:16:18using different services. And so make
22:16:20sure this is the correct one for you.
22:16:23Lastly, if we come over here, you can
22:16:25notice in your account we have account
22:16:26organization service quotas, billing and
22:16:28cost management and security
22:16:30credentials. The two that I think are
22:16:32really important is account and billing
22:16:34and cost management. If we go over to
22:16:36billing, this is an actual service uh
22:16:39within AWS. You can look at all of your
22:16:42costs. And so, as you start using these
22:16:44different services, you're going to want
22:16:46to come in here and make sure you're
22:16:47not, you know, spending too much money.
22:16:49Just as an example, we use AWS for
22:16:51Analyst Builder. And so, in here, we
22:16:53track all of our costs, all of our
22:16:55bills, everything associated with AWS
22:16:58for our platform. And so in here we
22:17:00track a lot of stuff and we have
22:17:01different metrics and different flags
22:17:03that we uh have in case we go over a
22:17:05certain amount or if something isn't
22:17:07working. And so we monitor uh you can
22:17:09create monitoring stuff. You can monitor
22:17:11a lot of stuff in this billing and cost
22:17:13management home uh which also was really
22:17:15important when I was a manager and we
22:17:17were using AWS. And then lastly, of
22:17:19course, we have our account within here.
22:17:21There's a ton of stuff like bills,
22:17:23payments, you know, if you want to
22:17:24change your credit card and all these
22:17:26different things. This is a really good
22:17:27place to come and just be familiar with.
22:17:29So, let's just go back to the services
22:17:30real quick while we close out and we'll
22:17:33go to all services and view all
22:17:35services. So, this is what we're going
22:17:37to be focusing on in the next several
22:17:38lessons. We're going to be looking at
22:17:40things like S3 storage. Let's come down
22:17:43here. I think we're going to be looking
22:17:44at Athena, AWS Glue, Data Brew, AWS
22:17:47Glue, and a few others as well. So,
22:17:49we're going to be in here. We're going
22:17:50to be learning a ton of stuff about AWS.
22:17:53And hopefully by the end of the series,
22:17:54you'll be really familiar with AWS, feel
22:17:56really confident putting it on your
22:17:58resume and actually knowing how to use
22:18:00it. So, I hope that this was helpful
22:18:01getting everything set up. If you have
22:18:03not already, be sure to check out my
22:18:04full AWS and Azure course on
22:18:06analystbuilder.com.
22:18:08And if you like this video, be sure to
22:18:09like and subscribe. I will see you in
22:18:11the next video. [music]
22:18:17>> [music]
22:18:24>> What's going on everybody? Welcome back
22:18:26to another video. Today we're going to
22:18:27be taking a look at S3 buckets in AWS.
22:18:35Now S3 buckets are super flexible ways
22:18:38of storing your data within AWS and they
22:18:40kind of connect to all the other data
22:18:42aspects of AWS as well. So, if you have
22:18:44data sitting in an S3 bucket, you can
22:18:46connect it to a SQL database, you can
22:18:47connect it to a data visualization tool
22:18:49or an ETL tool or a ton of other things.
22:18:52So, we're going to be diving into how to
22:18:53set up your S3 bucket, how you can
22:18:54actually use it. We're going to upload
22:18:56some data and we'll talk about little
22:18:57tips and little nuances of using S3
22:18:59buckets that I've learned over the
22:19:00years. So, with that being said, let's
22:19:01jump onto my screen and take a look. All
22:19:03right, so let's get started by coming
22:19:04into our services. We're going to go
22:19:06down to storage and we're going to click
22:19:07on S3. Now, you'll notice we have S3
22:19:10Glacier over here. And I'll briefly
22:19:13mention this uh a little bit in this
22:19:16video because it is worth noting. But
22:19:18let's come over here to S3. So this is
22:19:21kind of the homepage for Amazon S3.
22:19:24We'll take a look at a few things in
22:19:26here really quickly. They do have this
22:19:28little video here which if you're just
22:19:30using S3, you should look at. Now you
22:19:32may be wondering what does S3 stand for?
22:19:34It stands for simple storage service.
22:19:37It's meant to be a really simple way to
22:19:38store just about anything in the cloud.
22:19:40You can store essentially any type of
22:19:42file whether it's structured,
22:19:43semi-structured, unstructured, it could
22:19:45be almost anything. And so let's take a
22:19:47look at some of the things in here. It
22:19:48says store and retrieve any amount of
22:19:50data from anywhere. Amazon S3 is an
22:19:52object storage service that offers
22:19:54industryleading scalability, data
22:19:56availability, security, and performance.
22:19:59Let's come down here and take a look at
22:20:00some of the benefits and features. You
22:20:02can read through all of these things,
22:20:04and they're all really good. I'll read
22:20:05through this one in a little bit, but it
22:20:06says data performance and durability,
22:20:08security compliance and auditing,
22:20:10granular data control, and flexible
22:20:12storage options. So, right down here, it
22:20:14says save cost without sacrificing
22:20:16performance. Store data across a wide
22:20:17range of cost-effective storage classes
22:20:19and support different data access levels
22:20:21that are all designed for specific use
22:20:23cases. Now, in this lesson, we're going
22:20:25to be taking a look at some of this,
22:20:26which is their storage classes, cuz
22:20:28these are quite important and they do
22:20:30affect the cost that it's going to take
22:20:32in order to store your data because uh
22:20:35it's not free. Uh storing your data in
22:20:37the cloud is not free. You can also look
22:20:39at some of their use cases as well as
22:20:42some of their case studies if you would
22:20:44like. Now, let's actually get into it.
22:20:47We're going to come in here and we are
22:20:48going to create our very first S3
22:20:50bucket. So, I'll click on create bucket.
22:20:51The first thing that we need to do is do
22:20:53some more of our general configuration.
22:20:55We're going to keep this a general
22:20:57purpose. And we need to give it a name.
22:20:59So, I'm just going to call this one Alex
22:21:01the analyst bucket. And if we had a
22:21:05pre-existing bucket, we've already
22:21:07configured it with uh different
22:21:08configurations. We can just select that
22:21:10bucket and it'll copy over all the uh
22:21:12things that we want. The next thing we
22:21:14need to do is need to come right down
22:21:16here for object ownership. You can have
22:21:18ACL's disabled or ACL's enabled. If we
22:21:21go with the recommended route, it means
22:21:23that all the objects in this bucket are
22:21:25owned by this account, your account that
22:21:26you created. But if you do it where it's
22:21:29enabled, it says objects in this bucket
22:21:31can be owned by other AWS accounts. So
22:21:33with this one, it's just a little bit
22:21:35more secure because you're not saying
22:21:36other people can own it, which means
22:21:38they can delete it or change it in any
22:21:40way. Next, we're going to come down here
22:21:41to block public access for this bucket.
22:21:44Now, this part is actually very
22:21:47interesting because I've had a lot of
22:21:49use cases where you just want to block
22:21:51all public access, but if you start
22:21:53really getting into AWS and you start
22:21:55using a bunch of different tools and
22:21:57things, sometimes you need to get rid of
22:21:58it. Um, and you need to come in and not
22:22:01only that, there's much more advanced
22:22:03things in order to grant different
22:22:05bucket policies or create different
22:22:06bucket policies. And, um, we most likely
22:22:09won't get into all that in this lesson,
22:22:11but it can get quite advanced. And so
22:22:12this piece is pretty deceivingly simple.
22:22:15Um, but we're just going to keep it as
22:22:17we block all public access, but as you
22:22:19uh start opening it up to different
22:22:21services within AWS, you may want to
22:22:23turn this off so that you can have
22:22:25different services hitting off of your
22:22:26S3 bucket or the data within your S3
22:22:28bucket. So, we're just going to keep
22:22:30this as uh all public access off. I'm
22:22:32just giving you lots of extra
22:22:34information while we're in here. Some of
22:22:35my thoughts. Next, we're going to do
22:22:37bucket versioning. We don't need to have
22:22:38any type of versioning or version
22:22:40control within our bucket. What this
22:22:42means is keeping multiple variants of an
22:22:44object in the same bucket. It's used to
22:22:45preserve, retrieve, and restore every
22:22:47version of every object stored in your
22:22:49S3 bucket. And that is of course going
22:22:51to cost a little bit extra. So we don't
22:22:53uh need that at all. Next, you can add
22:22:56tags. And tags are helpful if you have a
22:22:59lot of different buckets and you maybe
22:23:01it's per client. You have some type of
22:23:03tag for a specific client. Perfectly
22:23:05normal. Uh next we have default
22:23:08encryption. Now encryption in general is
22:23:11really interesting within the cloud.
22:23:12I've run into lots of use cases where
22:23:15it's been really difficult to work with.
22:23:17If you want to pull data into certain
22:23:18services, you have to decrypt it uh
22:23:20because you've encrypted it in one area.
22:23:22And so if it's in the S3 bucket and it's
22:23:24encrypted, uh especially with kind of
22:23:26more advanced options, it can be
22:23:28somewhat difficult. And so just
22:23:29something to take into consideration if
22:23:31you want to kind of up your encryption.
22:23:33But for most use cases, you're just
22:23:35going to use the server side encryption
22:23:36with Amazon S3 managed keys with the
22:23:39bucket key enabled. Next, let's come
22:23:41down here to advanced settings. There's
22:23:44only one thing in here that we need to
22:23:45look at, which is the object lock. And
22:23:47remember, everything that you put inside
22:23:49of an S3 bucket is an object. Now, if we
22:23:52just read this right here, it says
22:23:53storage objects use a write once, read
22:23:55many, which is a worm model. And this
22:23:57helps prevent objects from being deleted
22:23:59or overwritten for a fixed amount of
22:24:01time or indefinitely. So, if you want to
22:24:03drop a file in there and you know you're
22:24:04going to need it, you don't want it to
22:24:06be deleted for any amount of time, you
22:24:08can lock that and you can have an object
22:24:10lock on that object. It can never be
22:24:12deleted. We of course do not need that.
22:24:14So, we're going to keep that disabled.
22:24:16And now we're ready to create our very
22:24:18first bucket. I clicked on create
22:24:20bucket. Uh looks like we can't use
22:24:22uppercase and I actually knew that. Uh
22:24:24let's fix this. Now, this does have to
22:24:27be unique. This is global. So, it says
22:24:29right here you have to use a unique
22:24:30name. You can't just use like Alex. Uh,
22:24:32someone's probably already chosen that.
22:24:34Let's go ahead and create our bucket.
22:24:36And now you can see right in here we
22:24:37have our very first bucket. Very
22:24:40exciting. Let's click into our first
22:24:42bucket, our only bucket. Let's go ahead
22:24:44and click into here. And we have a few
22:24:46different things in here. We have our
22:24:47objects, and that's going to be any
22:24:49files that we upload into this bucket.
22:24:51And within this, we can create tons of
22:24:53folders and subfolders and sub
22:24:55subfolders and all these different
22:24:56things. We also have properties. So this
22:25:00gives you a little bit of information on
22:25:02how you actually created this. You have
22:25:04permissions. So if you want to grant uh
22:25:06access to it. I talked a little bit
22:25:08about this which is a bucket policy. So
22:25:11if we turn off this all public access
22:25:13and we turn on this or we allow this
22:25:15bucket policy, we can create our own
22:25:17bucket policy and that's written in JSON
22:25:19and we can create our own bucket policy
22:25:21for public access to this specific
22:25:24bucket. That gets a little bit more
22:25:26advanced, but it is really fun. I've uh
22:25:28done that quite a bit. You have metrics
22:25:30on this bucket. You can manage this
22:25:32bucket and you can create access points
22:25:34to this bucket. So, there's a lot of
22:25:36things just within a single S3 bucket
22:25:38that you can do. But the most popular
22:25:41one that you're going to be using is
22:25:42this one right here, which is just
22:25:44adding, creating, and deleting, and
22:25:46using objects in general. So, let's go
22:25:48ahead and upload our very first file.
22:25:51We're going to go over here to add
22:25:52files, but you could add a folder if you
22:25:54have a whole folder, but we're just
22:25:55going to add one file. And within my
22:25:56sample files here, we have a bunch of
22:25:58very real healthcare data. It's not
22:26:01real. Uh it's just completely fake data.
22:26:03Now, what we're going to do is we're
22:26:04going to upload one. Then we're going to
22:26:05come back. We're upload another one in a
22:26:07different way. And I'm going to
22:26:08demonstrate that and uh explain that in
22:26:10a little bit. Now, what we're going to
22:26:11do is we're going to select this file.
22:26:13We're going to open this up. And now we
22:26:15have this real healthcare data 1.csv.
22:26:18And that's what we're going to be
22:26:19uploading. Now, let's come down here. We
22:26:22have our destination. This is telling us
22:26:23where we're actually placing this. We
22:26:26have our permissions and of course we
22:26:28have uh bucket enforced and so if we
22:26:30wanted to grant access to other accounts
22:26:32we need to change some of our access
22:26:34policies. But lastly we have properties
22:26:37and this piece is really interesting.
22:26:39This is our storage class. This is how
22:26:41uh Amazon S3 is actually going to store
22:26:43your data. Now I highly recommend going
22:26:45in to learn more and looking at their
22:26:47Amazon S3 pricing because it's very
22:26:49fascinating how they do this. Um, and
22:26:51it's also really important that you
22:26:52understand the differences between these
22:26:54different options that we have. By
22:26:56default, we have standard, and it's
22:26:59designed for frequently accessed data
22:27:01within milliseconds for access. So, if
22:27:03you're going to be using this data,
22:27:04you're going to be hitting off of it for
22:27:05different applications, services, uh,
22:27:08visualizations, whatever you're using it
22:27:09for, you're going to want to be able to
22:27:11access that pretty quickly. And so, this
22:27:13is a great option. If you need it even
22:27:15faster, you have S3 Express one zone,
22:27:18which is singledigit millisecond
22:27:20response times for the most frequent
22:27:22access data. Now, I don't think it talks
22:27:24about cost in here, uh, but you can use
22:27:28a calculator, and this is going to be
22:27:29costly, right? It's going to cost more
22:27:31to get faster and lower latency
22:27:34responses to your data. Now, if we come
22:27:36down here, you'll notice we have this
22:27:38glacier tier. Now, if we went back to
22:27:40our resources, remember we talked about
22:27:42S3 Glacier. I was going to mention that.
22:27:44Well, you can use these different
22:27:47glaciers which allows you to store it
22:27:50for long periods of time at a much lower
22:27:52cost, but it stores it a little bit
22:27:54differently. This one specifically is
22:27:56instant retrieval. It's very similar to
22:27:59almost standard, but you're kind of
22:28:00storing it for a long long time. You may
22:28:03not need it for 10 years, but when you
22:28:04do need it, you need it right away,
22:28:06which is not a lot of use cases if
22:28:08you're using Glacier. But then you have
22:28:09something like Glacier Deep Archive.
22:28:11This is longived archive data accessed
22:28:14less than once a year with retrieval of
22:28:16hours. And so this is going to be data
22:28:18that you don't need it right away. Uh
22:28:20you just want to be able to store it and
22:28:22have the security of putting it in the
22:28:24cloud, but you may not use that for
22:28:26years. And when you do need it, you
22:28:27know, you're just you're okay with
22:28:28waiting a little bit. It's going to cost
22:28:30almost nothing to store. It's very very
22:28:32very little. And oftent times if you're
22:28:34doing something like this, you may even
22:28:36put backups of databases. You may put
22:28:38backups of entire code bases in here.
22:28:41things that you may never use again. Um,
22:28:43and there's lots of different use cases
22:28:45as well. So, I just wanted to walk
22:28:47through that with this file. We're going
22:28:48to do standard. And on the next file we
22:28:50do, we're going to do Glacier deep
22:28:52archive. And just look at the difference
22:28:54here. Now, let's come down. Uh, we don't
22:28:56need encryption. We don't need any type
22:28:59of check sums, tags, or metadata. That
22:29:02stuff is almost uh never used or very
22:29:04very infrequently. So, we've uploaded
22:29:06this file. Let's go ahead and go to our
22:29:08destination. You can see that now we
22:29:10have an object in our bucket which is
22:29:13fantastic. You can even come over here
22:29:15and you can see the storage class is
22:29:18standard. Now let's go in. We're going
22:29:20to upload one more but now we're going
22:29:22to upload it as a different storage
22:29:24class. So let's go to upload. Let's go
22:29:26to add files. Let's go to healthcare
22:29:28data 2. We're going to open that up. The
22:29:31only thing we're going to change is now
22:29:33we're going to go down to the deep
22:29:35archive and we're going to go ahead and
22:29:38upload this. Easy peasy. And there we
22:29:42have two files. Now we have one in
22:29:45standard, one in Glacier deep archive.
22:29:47Let's go into the first one. If we want
22:29:48to use this in any way, it's almost
22:29:51instantly retrievable. We can download
22:29:53this. Uh we have a bunch of object
22:29:55options. So we can download it. We can
22:29:58copy it. We can move it. We can change
22:30:00the storage class. We do a bunch of
22:30:01different things. But let's come back
22:30:04and we're going to go to that file too.
22:30:07Let's go ahead in here. Notice right
22:30:09here we're getting a totally different
22:30:11uh message. It says the object is
22:30:13starting Glacier deep archive storage
22:30:15class. In order to access it, you must
22:30:16first restore it. So you have to
22:30:19initiate a restore right over here. And
22:30:22it can take 30 minutes to several hours.
22:30:24Notice we cannot download this. We also
22:30:27can't really do anything with it until
22:30:30we actually initiate that restore and it
22:30:32is restored for us to be able to access
22:30:34it. And that's just something that I
22:30:36think is worth noting about storing
22:30:38things in S3. There are different
22:30:39storage classes depending on your use
22:30:41case and what you're using. That's ADA
22:30:434. And of course, you should look at the
22:30:45S3 pricing for both of these cuz this
22:30:47one is going to cost more than this one.
22:30:50Now, there are costs associated with
22:30:52restoring it, but if you're restoring it
22:30:54once per year or maybe once every 5
22:30:56years, it's going to be significantly
22:30:58less than your standard storage class.
22:31:00Now, this is kind of the meat and
22:31:01potatoes of using S3. Of course, you can
22:31:04create folders and subfolders. And in
22:31:06fact, when we get into some other
22:31:08lessons, especially things like glue, uh
22:31:10having folders, subfolders, and
22:31:12different things like that is actually
22:31:14really important because we have
22:31:15something called a glue crawler. You
22:31:17know, a crawl through your folders and
22:31:19subfolders and retrieve certain data.
22:31:21And so having a folder structure
22:31:23actually becomes more important in those
22:31:25lessons, but we'll of course get to that
22:31:27when we actually start looking at it.
22:31:29One other thing to mention, and this is
22:31:30just uh kind of a neat addition that
22:31:32they have in here. If we go into this
22:31:34file and we go down to query with S3
22:31:37select, you can access this data and
22:31:40take a look at it. And so, uh, let's say
22:31:42we have, uh, our CSV here. It's comma
22:31:45separated. We can query this data using
22:31:48SQL. And so, let's just run this with
22:31:50the limit five on there. We'll run our
22:31:52SQL query. We have our query results
22:31:55right down here. We can have it
22:31:56formatted and it looks like a little
22:31:58table. So, we can come in here. we can
22:32:00look at uh this data down here and just
22:32:03see what's in there. And so that might
22:32:04be useful to you. Now, in the next
22:32:07lesson that we're going to be having,
22:32:08we're going to use Amazon Athena, which
22:32:10is basically a tool to query off of S3
22:32:13buckets. That's its primary use. And so,
22:32:16most of the time, I honestly am not
22:32:18using this almost ever, but you can even
22:32:19see right here it says Amazon Athena.
22:32:22That's what we're going to be looking at
22:32:23in the next lesson. We'll be seeing how
22:32:24Amazon Athena works, how you can query
22:32:26data, kind of set up a pseudo database
22:32:29like structure. I'll talk about the pros
22:32:31and cons of Amazon Athena versus other
22:32:34tools because Amazon Athena is not for
22:32:36every use case. So, I hope that that was
22:32:39helpful. I hope you enjoyed it. If you
22:32:40have not, I have a full AWS and Azure
22:32:43course on analystbuilder.com. Be sure to
22:32:44check it out. If you like this video, be
22:32:46sure to like and subscribe below. I will
22:32:48see you in the next video.
22:32:51>> [music]
22:33:02>> What's going on everybody? Welcome back
22:33:03to another video. Today we're going to
22:33:05be taking a look at Amazon Athena in
22:33:07AWS.
22:33:11[music]
22:33:13Now Athena is a tool that allows you to
22:33:15query data in your S3 bucket without
22:33:18having to put that data into some type
22:33:20of database. you're just querying it
22:33:21directly. So let's say you just got a
22:33:23big file from a client, you put it in an
22:33:25S3 bucket, but you don't want to
22:33:26actually create a table and take the
22:33:28time to do all that whole process. You
22:33:30just want to take a look at the data.
22:33:31You just want to query it really
22:33:32quickly. Well, Athena allows you to do
22:33:34that. And so if that's the case, it can
22:33:36save you a lot of time and money. And so
22:33:37this is a really great tool that a lot
22:33:39of companies use. We'll talk a lot more
22:33:41about Athena in just a second. So let's
22:33:43jump on my screen and take a look. All
22:33:44right, so let's come into services and
22:33:46we're going to go all the way down to
22:33:47the analytics tab and we're going to go
22:33:49into Athena. Now, before we jump into
22:33:51actually querying data and setting
22:33:53everything up, let's take a look at some
22:33:54of the information they have on Amazon
22:33:56Athena. This right here, this start
22:33:59quering data instantly is probably one
22:34:01of the uh biggest pieces of why you
22:34:04would use Amazon Athena. And I'll talk a
22:34:06little bit about a little later on about
22:34:08who this is for, why you'd want to use
22:34:10this versus other tools uh within AWS.
22:34:13But Amazon Athena is an interactive
22:34:15query service that makes it easy to
22:34:17analyze data in Amazon S3 and other
22:34:20federated data sources using standard
22:34:22SQL. So primarily though for you and I
22:34:25most likely you're going to be using
22:34:27this just to hit off of an Amazon S3
22:34:29bucket. You can even see how it works
22:34:30right down here. It says point to your
22:34:32data source. You're going to use Amazon
22:34:34Athena. You're going to query it and you
22:34:36can analyze those results down here in
22:34:40the benefits that are right here. You
22:34:42have start querying now. Oh, it's
22:34:43powerful, cost effective, fast. All of
22:34:45these things are great. Honestly,
22:34:47powerful, cost effective, and fast could
22:34:49be a ton of different tools in AWS. But
22:34:51this piece right here is quite unique to
22:34:53it in the fact that you can query it as
22:34:55it just sits in an S3. You don't have to
22:34:57bring that data over into a database and
22:35:00actually store it in the database. You
22:35:01can keep it in your S3 bucket and still
22:35:04query off of it. And so that is uh kind
22:35:06of the real use case and the real reason
22:35:09why you would want Amazon Athena just to
22:35:10look at data uh within an S3 bucket. Now
22:35:13we do have two options here. Query your
22:35:14data with Trino SQL or analyze your data
22:35:17using PIS spark and spark SQL. We are
22:35:19just going to be looking at the Trino
22:35:21SQL in this lesson but of course you can
22:35:23use PIS spark and spark SQL as well.
22:35:26Let's go ahead and launch our query
22:35:27editor. And this is what you should see.
22:35:30Now it says uh we need to edit some of
22:35:33our settings. And I'll explain that in
22:35:35just a little bit. Um, but over here on
22:35:37the lefth hand side, we have our data.
22:35:40So we have our data source, we have our
22:35:42database, we have tables and views, and
22:35:45then we have where we query our data,
22:35:47and then down here where our results
22:35:49will be. So there's a lot of stuff just
22:35:52in here, but this is um not super crazy
22:35:55advanced. Uh if you've used anything
22:35:57like MySQL or if you've used Microsoft
22:35:59SQL Server or anything like that, this
22:36:01is like a really I don't want to say
22:36:02dumbed down version that's not super uh
22:36:05kind to say, but it's it's a really
22:36:07simplified version of it. It's not very
22:36:10uh difficult to understand. You can also
22:36:12save your queries in recent and saved
22:36:15queries. And of course, we have some
22:36:17settings over here. Now, we're going to
22:36:19see how we can set up our database,
22:36:21create a table or two, see how all that
22:36:23works with actually hitting off of data
22:36:25within our S3 bucket and we will have to
22:36:28fix this piece which is you have to set
22:36:30up a query result location in Amazon S3.
22:36:33That's for your results or your metadata
22:36:36and uh that piece becomes quite
22:36:38important later on. So, let's come down
22:36:40here. We have our data source. This is
22:36:41going to be our AWS data catalog. Now,
22:36:44we haven't covered this in a previous
22:36:45lesson, but you have something called a
22:36:49data catalog, and we're going to look a
22:36:50lot at that actually in either the next
22:36:53lesson or the lesson after that when we
22:36:54look at uh AWS Glue and Glue Data Brew.
22:36:58But that's how they kind of organize it.
22:36:59And so, we're not going to be messing
22:37:00with that uh here. But we have to choose
22:37:03a database. And notice we don't have any
22:37:05database. So, what we need to do is we
22:37:07need to start pulling in data. And when
22:37:09we're pulling in that data, we'll be
22:37:10able to create a database. Now, right
22:37:13here we have create. You can create a
22:37:14table from a data source or create with
22:37:16SQL. So, if you want to go the
22:37:18oldfashioned way where you're creating a
22:37:19table like this and you can and you can
22:37:22specify the column, the column types or
22:37:25data types, the location, all these
22:37:27things, you can do that. Uh, but we're
22:37:29not going to be doing that. We are going
22:37:31to be doing it with S3 bucket data. Now,
22:37:34we also have AWS Glue Crawler. Now, I'm
22:37:37just going to open this up really quick.
22:37:39We will be doing this when we get to
22:37:41glue. Uh when we start looking at glue
22:37:43because crawlers are great. Crawlers
22:37:45allow you to specify the data source and
22:37:47it pulls it in and infers based off of
22:37:50the column names and and the data types
22:37:52that are in there. It builds the table
22:37:54for you and it's very helpful and it
22:37:56kind of helps automate it as well. We're
22:37:58not going to be doing that. Uh you can
22:38:00look at crawlers here, but we're not
22:38:02going to be doing that in this lesson.
22:38:04So this is what crawlers are and that's
22:38:06within AWS Glue, but we're not we're not
22:38:08going to be looking at that in this
22:38:09lesson. So what we are going to be doing
22:38:11is just creating it from an S3 bucket
22:38:13data. Now we need to call this table
22:38:15something. So let's call this healthcare
22:38:18and I need to spell health right. Uh
22:38:20healthcare data. I I'm having trouble
22:38:22spelling. Let's go down to the database
22:38:25configuration. Now we don't have a
22:38:27database. So we need to create a
22:38:29database. And this is super easy. Um
22:38:31this is actually the table name. Let's
22:38:33call this um patient data because I want
22:38:36to call the uh the database healthcare
22:38:39data. So I'll call this one healthcare
22:38:41data. I spelled that much better that
22:38:42time. So we've specified here's what our
22:38:45table is going to be called. Here's what
22:38:47our database is going to be called. Now
22:38:49we need to specify our data set. And
22:38:51this is a kind of an odd part and you'll
22:38:53see that in just a little bit. Let's
22:38:55come in here and we have our Alex the
22:38:57analyst bucket. Let's go ahead and click
22:38:58into it. Notice though within this
22:39:01bucket we have two data sets. I I really
22:39:03only want this one if I'm being honest.
22:39:05I just want this one. But you can't do
22:39:08that. Uh not within Athena and within
22:39:10other parts of AWS as well. This is just
22:39:12a a kind of one of those nuances uh
22:39:15within it. We have to specify a file
22:39:17path, not a file itself. So we have to
22:39:20choose the entire folder. Let's choose
22:39:23this. Let's come down here to the data
22:39:26format. Now this data format that's in
22:39:29here is a CSV. So we're going to come in
22:39:32here with an Apache Hive and we're going
22:39:34to specify that is a CSV file. Of
22:39:36course, we want the delimiter which is
22:39:38right down here to be a comma. So we
22:39:40should be good to go. But because we are
22:39:43not using a crawler, we have to manually
22:39:45enter these column details. So I'm going
22:39:48to come down here. We are going to pull
22:39:50up this file. Let's zoom in a little
22:39:52bit. These are uh everything that we
22:39:55have. Now we can there is an option in
22:39:57here actually. We can copy all of this
22:40:00and we'll come in here to add bulk
22:40:02columns and you can do it like this and
22:40:04then you can put in the data type and
22:40:07that's perfectly fine if you want to do
22:40:08that. Uh but we're not going to be doing
22:40:10that. I'm going to actually pull this
22:40:12up. We will bring this over to the side
22:40:17right here. And there we go. So now we
22:40:20can see it. So we have our first column.
22:40:22It's going to be uh patient
22:40:25ID. It doesn't have to be the exact same
22:40:27as over here in the file. Now, this file
22:40:29type is just numeric. So, we can come in
22:40:31here and we'll choose integer right
22:40:34here. And that's all we need to specify.
22:40:37Now, we have several other columns. So,
22:40:39let's just kind of bulk place them in
22:40:41here. We have name, we have age,
22:40:45we have diagnosis,
22:40:48and we have treatment. Let's see if
22:40:50there's anything else in here. I think
22:40:52we have files. Yeah, one more. So let's
22:40:54add in files. And just to show you, we
22:40:58can do name underscore. Is this their
22:41:00full name? Yeah, full name. So we'll do
22:41:03full underscore name. And we're doing
22:41:06that just to demonstrate that doesn't
22:41:07have to copy this exactly. Um but for
22:41:10the full name, this should be string or
22:41:12text. So we could use char. We could use
22:41:15uh string. See if there's any other ones
22:41:17that we could use. Probably those two.
22:41:18We'll call this one string. For age, it
22:41:21should be integer as well. for diagnosis
22:41:25treatment those are both string so we'll
22:41:28do string and string and then for files
22:41:32this one is integer as well we can make
22:41:36this a little bit larger now there's a
22:41:38bunch of different things that we can
22:41:39specify different types of compression
22:41:42if you have compression so if you're
22:41:43using a zip file or anything like that
22:41:46you can specify uh that it's sitting in
22:41:48it there is partitioning as well and
22:41:50this is a little bit more advanced this
22:41:52is a way that you can group specific
22:41:54information together, but this is not
22:41:56something that we need to worry about,
22:41:58especially with our simple data set.
22:42:00There also is bucketing. This is a way
22:42:02to bucket multiple columns together. And
22:42:05then it's stored in a way that when you
22:42:06try to retrieve it, it's retrieved a lot
22:42:08faster and easier. Again, this is a bit
22:42:11more complicated and a little bit
22:42:12advanced. So, we're not going to be
22:42:13taking a look at that at the moment.
22:42:15Let's go ahead and create this table.
22:42:18Now, we're getting this error that says
22:42:20no output location provided. Now, output
22:42:22location is required either through the
22:42:24work group result configuration setting
22:42:26or as an API input. So, let's go fix
22:42:29this really quickly. We're going to
22:42:31actually I don't want to have to redo
22:42:32all this. So, let's actually duplicate
22:42:34this. Let's go into our query editor and
22:42:38we need to go into this work group. So,
22:42:40let's come over here. Let's go into our
22:42:42workg groupoups and this is our primary
22:42:44work group. Let's come in here and we
22:42:47need to come over to edit. Now within
22:42:50this we had chosen uh Athena SQL and if
22:42:53we come down we have this query result
22:42:55configuration. We have to specify this
22:42:58so that our output and our metadata is
22:43:02put somewhere. So let's go ahead and
22:43:04browse this. Now just like when we chose
22:43:06our file we cannot specify like a file
22:43:10to put it in. We have to specify a file
22:43:12path. Now we're going to specify that
22:43:14this is our path but this is not the
22:43:16best way to do it. I'm doing this for
22:43:18demonstration purposes only. Uh we'll go
22:43:20back and do it the right way in a little
22:43:22bit. So this is where we're going to put
22:43:23our output, our results, metadata, all
22:43:25that stuff. Let's go ahead and choose
22:43:27this.
22:43:29And uh you can add life cycle uh
22:43:32configuration as well as assign a bucket
22:43:34owner and some encryption, but we don't
22:43:36need to do that. Let's save these
22:43:38changes and let's go back
22:43:42and let's go and create this table.
22:43:46So, it created this for us. The query
22:43:48was successful. Let's refresh this. And
22:43:51it's giving us a default database.
22:43:53That's actually not the one that we put
22:43:55it in. We put it into healthcare. So,
22:43:57let's go over to healthcare. And there
22:43:59is our patient data. You can see we have
22:44:01patient ID, full name, age, diagnosis,
22:44:03treatment, files with the associated
22:44:06data types. And what we can do, whoops.
22:44:08What we can do is we can come right over
22:44:10here and let's say we want to preview
22:44:13this table. So that is completed. Let's
22:44:16go down here. And the data looks pretty
22:44:18good. We have this one file uh because
22:44:20we have two files. It's reading in that
22:44:23second file uh column names, but we have
22:44:27some data in there. And that's really
22:44:28really good. Now, we're limiting this to
22:44:3010. And let's get rid of this. But
22:44:33actually uh before we do that, I'm going
22:44:34to keep it the same because before we do
22:44:36that, we need to go look at the
22:44:39ramifications of what we actually did
22:44:41because uh unfortunately it's not a good
22:44:44thing what we did. Let's go to our S3
22:44:46bucket and let's see what data is in
22:44:49this Alexi Analyst bucket here. Now you
22:44:52can see we have a bunch of stuff. It's
22:44:54not just our two files anymore. Now we
22:44:56have a text file, a text file, metadata,
22:44:59CSV. This is our output and some extra
22:45:01stuff as well as the metadata. And
22:45:03what's going to happen if we come over
22:45:06here and we try to query all of our
22:45:08data. Let's go ahead and run this.
22:45:11Now, you'll notice that we're pulling in
22:45:13a lot of bad bad data. This has
22:45:16completely ruined our query and this is
22:45:18kind of defeated the purpose of setting
22:45:20up using Amazon Athena. What we need is
22:45:24we need a separate folder location
22:45:26within our S3 bucket to dump all this
22:45:28stuff. Otherwise, you'll notice it's
22:45:30just going to keep growing. We're just
22:45:31going to have more metadata, more CSV,
22:45:33more text files. That is not a good
22:45:35thing. So, here's what we're going to
22:45:37do. We're going to create a folder.
22:45:39We're going to call this one uh metadata
22:45:42healthcare. I think I spelled that
22:45:44right. And we're going to create this
22:45:45folder. Now, I'm going to take all of
22:45:48these things and I'm going to delete
22:45:51them. So, let's go ahead and delete all
22:45:53these files. And we have to specify
22:45:55this. I'm going to do a little cheat.
22:45:56I'm going to copy and paste this. There
22:45:59we go. Let's delete our objects.
22:46:01And those were deleted. And now you can
22:46:03see we just have our uh real healthcare
22:46:06data here. And we have our metadata
22:46:09healthcare up here. Now what we need to
22:46:12do is we have to go back. We're going to
22:46:14go over to our work group and we're
22:46:15going to specify the new metadata folder
22:46:19that we're going to be placing this in.
22:46:20So let's go back down to settings.
22:46:23Actually, it's in the query result
22:46:24configuration. Let's go to browse. We're
22:46:27going to go in here and we're now we're
22:46:28going to specify the metadata healthcare
22:46:31folder. So, let's go ahead and specify
22:46:33that. Let's save our changes and we'll
22:46:36go back to our query editor. Now, when
22:46:40we run this, let's go ahead and run this
22:46:41again. Now, we have our output. Our data
22:46:45is in there. Let's come up here. Let's
22:46:47refresh this.
22:46:50The data is not in here anymore. Now,
22:46:52it's in this metadata healthcare. And
22:46:54that's great, right? Let's come back
22:46:56here and let's run this once again.
22:47:00Let's come down and you're going to
22:47:02notice something uh peculiar is what
22:47:04I'll call it. What's peculiar is the
22:47:06fact that now we have the same metadata.
22:47:09Now, why is this why is this happening?
22:47:11And why am I showing you this in the
22:47:13first place? It's because when you are
22:47:16specifying a file path, you are not only
22:47:18specifying a file path, you're
22:47:20specifying the folders and the
22:47:22subfolders. So when we come in here and
22:47:25we look at this bucket, if we just
22:47:28specify that we're pulling all the data
22:47:30out of this bucket, then we're pulling
22:47:32not just this data, we're pulling a
22:47:34folder within a subfolder. And so now
22:47:37we're pulling this data as well. So we
22:47:39need to if we want to keep it all in one
22:47:41bucket, which we can, we need to create
22:47:44a new folder called our patient
22:47:47data. So we're going to click on our
22:47:50file. We're going to click move and we
22:47:54need to specify our destination. So,
22:47:55we're just going to put this in the
22:47:57patient data. So, let's go ahead and
22:47:59choose this and let's move it. And we're
22:48:02going to close this out. Now, uh you'll
22:48:05notice that this data is a Glacier deep
22:48:09archive. So, we cannot move this and we
22:48:11cannot query it. And that's also
22:48:12something that I wanted to mention while
22:48:14we were here. I somewhat forgot this.
22:48:16The data that we are pulling in uh this
22:48:19data down here is only from one file. We
22:48:22don't have two files of data in here.
22:48:24The only data that we have or that we
22:48:26were pulling in was from this real
22:48:28healthcare data one. We were able to hit
22:48:30off of that data cuz the storage class
22:48:32is standard. But because when we set
22:48:35this up, this was a Glacier deep
22:48:38archive, we cannot query off of that
22:48:40data. And so what we're going to do is
22:48:42I'm going to come back up here to
22:48:43patient data. I'm going to go to upload.
22:48:46So, we have more than one file in here.
22:48:48Let's go ahead and add a file. I'm going
22:48:49to specify this number two, but now
22:48:52we're just going to keep it as is. We're
22:48:54not going to Oh, if we go to down to
22:48:56properties, we're just going to keep it
22:48:57as standard. So, let's go ahead and
22:48:59upload this. There we go. Let's close
22:49:02this out. Now, the data that we're
22:49:05hitting off of is right here. So, we're
22:49:08not hitting off of this file path. Now,
22:49:11we're going to be hitting off of this
22:49:12patient data. So let's go back. What we
22:49:15need to do now is we need to come back
22:49:17up here and we need to create a new
22:49:20table. So this new one, we're going to
22:49:22do create new table. This is going to be
22:49:23uh patient data 2 and we're going to
22:49:27choose our healthcare data. Now our data
22:49:31set is going to be different. So we're
22:49:33going to go into the Alex analyst
22:49:35bucket, but now we're going to specify
22:49:36patient data. Now, this is important
22:49:39because there are no folders and
22:49:42subfolders underneath patient data. It's
22:49:45just this one folder with our two files
22:49:47in it. When we actually run this and we
22:49:49query off of this data, it's going to go
22:49:52all the metadata is going to go into
22:49:53here, but will not mess up our data
22:49:56source. And that is very important to
22:49:58understand. Now, often times, just as
22:50:01you know, a little bit of side note,
22:50:03often times when you're working with
22:50:04this, you can either set it up like this
22:50:05or sometimes people and departments will
22:50:08create entire separate buckets just for
22:50:10their metadata. So, they can store all
22:50:11their metadata in it and then they can
22:50:12often they just delete it or whatever.
22:50:14So, let's go back here. We're going to
22:50:16recreate this. We're going to go to
22:50:18Apache Hive and we'll do uh CSV. Note,
22:50:22if you do iceberg or delta lake or lake
22:50:25formation governed table, these ones
22:50:27don't have options for a CSV anyways. So
22:50:30that's why we're not looking at it. Um,
22:50:32but for Apache Hive, they do have our
22:50:35CSV option, which is common delimited.
22:50:38Now, we need to go through and we need
22:50:39to specify our columns. Again, I'm going
22:50:41to skip this because you can just do
22:50:43this yourself. But we're just going to
22:50:44do this and then we're going to create
22:50:45our table. All right, let's go down.
22:50:47We're going to go ahead and create this
22:50:49table. Query was successful. Let's go
22:50:52back to our healthcare data and let's
22:50:54refresh this. So now we have our patient
22:50:57data 2. Let's go ahead and preview this
22:51:00table.
22:51:01And this is looking good. Let's get rid
22:51:04of this limiter. So let's come over
22:51:06here. Let's run this. Now you notice we
22:51:09have 42 results which is because we have
22:51:13uh one right here. This is from file.
22:51:16It's actually file two and this is file
22:51:17one but I got that wrong. Uh, but we
22:51:19have both files in here that we're
22:51:21hitting off of. So, within our S3
22:51:23bucket, we have two files of exact data
22:51:25and it's creating a union between these.
22:51:27It's reading in both of these files and
22:51:29it's giving us our output. And of
22:51:31course, we have this right here, which
22:51:33we don't want. So, we can always filter
22:51:35that out. Uh, whoops. Let's come up
22:51:37here. Let's say uh where let's do
22:51:41patient
22:51:43ID. There we go. We'll do is not null.
22:51:48And we'll run this. And now this looks
22:51:52really good. And so now we're querying
22:51:54off of our data and we're not getting
22:51:55any of that metadata issues that we were
22:51:58experiencing. We come back here in our
22:52:00metadata. It's storing all of this. And
22:52:03most of this is going to be pretty
22:52:04useless to most people. And so often
22:52:06times you'll have uh some type of event
22:52:08or something in place to delete this
22:52:10because you don't want to store this
22:52:12long term or most people won't. There
22:52:13may be some use cases too, but most of
22:52:15the time you're going to get rid of
22:52:16this. And so this is Amazon Athena. Now
22:52:19you can also create views. If you want
22:52:21to tie a bunch of tables together and
22:52:23create a view, you can do that. And of
22:52:25course this is SQL. So you can write uh
22:52:28you know SQL statements like you
22:52:30normally would with any SQL environment
22:52:32or SQL editor. And so if we want to
22:52:34filter, if we want to join uh we can do
22:52:37a lot of those things. And so this is a
22:52:39fully functioning uh quer editor. Now if
22:52:41we come down there are some other things
22:52:43that we have. We can copy this data. We
22:52:46can download these results. We can clear
22:52:49this whole thing. We can also create a
22:52:51table from this query or a view from
22:52:53this query which is really useful. We
22:52:55can also come up here if we want to save
22:52:57one of our queries. We'll come over here
22:52:59to query five and we're going to save
22:53:01as. And we'll call this one uh filter on
22:53:06nullles. It's a terrible description.
22:53:07We're just going to save that. So we
22:53:09have this filter on nulls. And if we
22:53:10ever want to reference this again, we
22:53:12can uh come back here and we can pull up
22:53:15this query and we'll have it available.
22:53:18Now, if we come over here on this
22:53:19left-hand side, you'll notice we have a
22:53:22notebook editor and a notebook explorer.
22:53:24So, there are options to create
22:53:26notebooks and run notebooks as well.
22:53:29That's something I would create a whole
22:53:30another lesson on because it's not as
22:53:32straightforward as using the SQL option.
22:53:35But another thing we can do which is
22:53:37within jobs right here is we have
22:53:39something called workflows. Now
22:53:41workflows are how you can create
22:53:43different pipelines and different
22:53:45orchestrations with your data and with a
22:53:48bunch of different tools within AWS.
22:53:50This is definitely a little bit more
22:53:52advanced and not something that most
22:53:54people are going to use within Athena.
22:53:56For a lot of orchestration data flows,
22:53:57data pipelines, you're going to be using
22:53:59Glue. And so within Glue, we'll be
22:54:01taking a look at how we can, you know,
22:54:03set everything like this up. But you can
22:54:05come in here and you can take a look at
22:54:06a bunch of different ones. You can, you
22:54:08know, these are different options. You
22:54:10can execute multiple queries, query
22:54:11large data sets, keep data uh up to
22:54:14date, and you can come in here and
22:54:16there's lots of different uh things that
22:54:17you can do, but again, we're not going
22:54:20to be covering it, but go ahead and take
22:54:21a look and see if you want to try it
22:54:22out. It can get a little bit
22:54:24complicated, a little bit complex. These
22:54:26are things that people do all the time
22:54:27in AWS, especially people like data
22:54:30engineers, database developers,
22:54:31analytics engineers, people like that.
22:54:33And so when we get to Glue, we'll look
22:54:35at how to create different workflows,
22:54:37data pipelines, uh, and all that stuff.
22:54:39So with all that being said, this is the
22:54:42meat and potatoes of Athena. Now, who is
22:54:45this for exactly? When I was using
22:54:48Amazon Athena, it was mostly to query
22:54:50data really quickly. A client was giving
22:54:52me a file, put it into an S3 bucket so I
22:54:54could start querying off of it, and that
22:54:56was it. uh most of the time we didn't
22:54:58have a full scale production environment
22:55:01over on this lefth hand side here with
22:55:03tables and views and databases um we
22:55:05kept the data separate of course for
22:55:07different clients and different projects
22:55:09but this was not our primary place to go
22:55:12I have consulted and worked with other
22:55:14teams where I come in and they're like
22:55:16hey here's where we're trying to get
22:55:17we're trying to get to this place and
22:55:19I'm like okay show me your current you
22:55:20know setup how you're working with your
22:55:22data how your data flows work and
22:55:23everything like that and they'll come in
22:55:24and they're like we're only using Amazon
22:55:26Athena for quering data and creating
22:55:28workflows and automations. If you want
22:55:30to be able to scale to a much larger
22:55:32scale, this won't work. There are
22:55:34several limitations within Amazon
22:55:36Athena. As you try to scale, as you try
22:55:39to add more data sources, as you try to
22:55:41automate all these things, it gets very
22:55:43difficult to do within Amazon Athena.
22:55:45And so that's when you need to start
22:55:46creating different databases like using
22:55:48something like RDS or another type of
22:55:50SQL server where you can store your
22:55:52data, just not within Amazon Athena.
22:55:55hitting off of S3 buckets as your main
22:55:58uh main way to create kind of your
22:56:01database is not optimal by any means.
22:56:04And so they might work for kind of a
22:56:05smaller company who's just trying to hit
22:56:07off some data, but it doesn't work at
22:56:09scale very well. So Amazon Athena is
22:56:11really good for getting quick insights
22:56:13into your data. It's really good when
22:56:15you have a ton of data stored in S3 and
22:56:17you just want to get simple insights
22:56:19into it, want to do simple joins,
22:56:21aggregations, and you're not trying to
22:56:23do anything too complex with it. This is
22:56:25a great place to go because it's
22:56:27actually very cheap compared to other
22:56:29resources and tools within AWS. This
22:56:31really shouldn't be used as a full-scale
22:56:33database, especially as your company
22:56:35gets larger. You run into a lot of
22:56:37roadblocks, a lot of issues, and I've
22:56:39seen that firsthand. And so, it's just
22:56:40not something I'm going to recommend.
22:56:42But Amazon Athena definitely has its
22:56:44place and that's why we're looking at it
22:56:45because you most likely if you're using
22:56:47AWS, you're working as a data scientist,
22:56:50data analyst, business analyst, you'll
22:56:51use Amazon Athena. A lot of companies
22:56:54have it. A lot of companies will use it.
22:56:55And so I hope that this was helpful. I
22:56:57hope you're able to get up and running,
22:56:58understand some of the nuances,
22:57:00especially with the folder paths. I
22:57:02always found that a little bit confusing
22:57:03until I just used it more. but
22:57:05understanding how to uh put your tables
22:57:08in to the healthcare in here um and
22:57:10actually get that uh metadata folders
22:57:12working as well. If you liked it, be
22:57:14sure to check out my full AWS and Azure
22:57:16course on analyst builder.com. And if
22:57:18you like the video, be sure to like and
22:57:20subscribe. And I will see you in the
22:57:21next video.
22:57:35What's going on everybody? Welcome back
22:57:36to another video. Today we're going to
22:57:38be taking a look at Glue and Glue datab.
22:57:44[music]
22:57:47Now, within the Glue umbrella, which
22:57:48includes Glue data brew, it's mostly for
22:57:51building pipelines and creating ETL
22:57:52processes. This is how you get data from
22:57:54one place to another. You can extract
22:57:56data, transform data, load the data, and
22:57:58do a ton of other things with it. So, in
22:58:00this video, we're going to take a look
22:58:01at both Glue and Glue Data Brew because
22:58:03they're slightly different and they have
22:58:04different purposes. So, without further
22:58:06ado, let's jump on my screen and take a
22:58:07look. All right, let's go down here.
22:58:09We're go down to analytics. Now, we're
22:58:11going to be taking a look at both AWS
22:58:14Glue Data Brew and AWS Glue. Now, we're
22:58:17going to start with Data Brew because I
22:58:18think it's a little bit more
22:58:19userfriendly. There's some nice
22:58:21animations. The UI is really uh a little
22:58:23bit, I would say, easier to understand.
22:58:26And this is often a place where you'll
22:58:28come for a lot of your transformations.
22:58:29So, let's come in here and you can go
22:58:31ahead and take a look at a lot of stuff
22:58:33in here. On this right hand side, we
22:58:35have data sets, projects, recipes, uh
22:58:38jobs, and we'll take a look at a few of
22:58:39those while we're in here. And you can
22:58:42look at some of the benefits as well.
22:58:45And uh Glue Data Brew is awesome. It's
22:58:47very much a more visual way to prepare
22:58:50your data. It even says it's a visual
22:58:52data preparation tool. And so it's a way
22:58:54to visualize how your data is changing
22:58:56and you can see how it's changing. We
22:58:58are not going to just create a project.
22:59:00We're going to create a sample project.
22:59:01And this is going to give us some sample
22:59:03data. We're going to work with it. We're
22:59:04going to be cleaning it up, transforming
22:59:06it a little bit, and then we'll see how
22:59:07we can automate that as well. So let's
22:59:09go ahead and create our sample project.
22:59:11Let's go ahead and select popular baby
22:59:13names in 2020. And then we have to go
22:59:16down here and choose a RO name. Now we
22:59:19haven't talked a lot about IM roles uh
22:59:22within AWS, but for certain things like
22:59:25glue data brew glue and a few other
22:59:27things you need IM access. So you can
22:59:29say create new IM ro and that's what
22:59:31we're going to do. Let's go down here
22:59:33and we need to actually create um a new
22:59:36role for AWS Glue Brew Service. So, it's
22:59:40going to create and I'll explain that in
22:59:42just a little bit. We'll call this one
22:59:43Alex the analyst and I'll call it data
22:59:47brew. Let's go ahead and create this
22:59:50project. Now, this needs to set up our
22:59:53session. It's provisioning some compute
22:59:55and getting our session ready, getting
22:59:56our data ready, all of these things in
22:59:58order to use data brew. But while we're
23:00:01in here and while we're waiting for this
23:00:02to uh be ready, I want to talk a little
23:00:04bit about the UI. So over here we have
23:00:07something called a recipe. Now the
23:00:09recipe is for when we actually use all
23:00:12of these things and we apply different
23:00:13changes to the data set, we make any
23:00:16transformations, whether we join the
23:00:17data, we group the data, we pivot the
23:00:19data, it's going to say, okay, you did
23:00:21this and then this and then this and
23:00:22then this. And you can come in here and
23:00:24you can edit this. And so you can say,
23:00:26oh, I don't want to actually group it.
23:00:27Let me get rid of that. And that'll be
23:00:29really easy to do and I'll show you that
23:00:30in a little bit. But then we can also
23:00:33save this recipe. We can just say, "Oh,
23:00:35I want to save this for a future data
23:00:37set." Or if you want to use that recipe
23:00:39on, you know, if you're bringing in
23:00:40multiple uh data sets of the same exact
23:00:43data or you have multiple uh data sets
23:00:45coming in that are all the same, then
23:00:47you may use the same recipe on all of
23:00:49them to transform that data consistently
23:00:51every single time. Looks like our data
23:00:53is done or our session is ready. But
23:00:55let's look at this top area. So, this is
23:00:57where we can make all of our
23:00:58transformations. We can filter, we can
23:01:00sort, we can come in here and we can
23:01:02click on the clean. And you'll notice
23:01:04there are tons of different options in
23:01:06here. A lot of ones that you should be
23:01:08pretty familiar with if you've ever
23:01:09cleaned data before or if you've ever
23:01:11done any of my projects in my SQL or
23:01:13Excel or Tableau or uh you know, Python
23:01:16and all these different ones on data
23:01:17cleaning. And these are a lot of really
23:01:20popular things in order to clean the
23:01:22data. So there's clean, extract, uh you
23:01:25can look at duplicates, outliers, you
23:01:27can merge your data, you can perform
23:01:29functions, you can apply different
23:01:31functions to your data. We also have
23:01:34things like pivot, group, join, union,
23:01:36and others. And there's just a ton in
23:01:38here. Now, what we're going to do is
23:01:40first we're going to look at our data,
23:01:42and then we're going to go through and
23:01:44we're going to do a few changes to it so
23:01:46you can see how the recipes work and
23:01:48then we'll save our recipe. So let's
23:01:50come in here. This is a new data set to
23:01:52both of us. This is our uh baby names
23:01:54data. So we have count, we have gender,
23:01:58we have the ID, we have the name, and
23:02:00then the year. And so this is pretty
23:02:02interesting information. Let's see if
23:02:03it's all uh 1880. This looks like just a
23:02:06small sample of the data. And we can
23:02:09come in here and we can also uh take a
23:02:11look at what kind of data types they
23:02:13assigned to this data as well. Now, what
23:02:15we're going to do is something quite
23:02:17simple. We just want to look at the male
23:02:20names and we want to take a look at the
23:02:22most popular male names per year. So
23:02:25that's going to require a bit of
23:02:26grouping. So we're going to have to
23:02:27group on both the year and the name and
23:02:30then we're have to filter based off of
23:02:32the gender. Now one other thing to note
23:02:35is that right now we're just working
23:02:36with a sample of the data. when we
23:02:39actually get to the final process and we
23:02:41actually create this data cleaning
23:02:43process. We can apply this to the entire
23:02:46data set or we can apply it to the
23:02:47sample. So right now we're just going to
23:02:49be working with this sample, but we'll
23:02:52of course be using the full data set
23:02:54later on. And you can even come up here
23:02:56and you can look at this. You can say
23:02:57first n rows, last n rows, and random.
23:03:01Now, right now we're just looking at the
23:03:02first n rows, the first 500 rows of our
23:03:05data set. And that may not be a perfect
23:03:07sample size. It actually may be better
23:03:10to pull random rows because what if we
23:03:12have male or female or other things.
23:03:15Right now it just looks like we have
23:03:16female in our sample and that may not be
23:03:19representative of the entire data set.
23:03:21So let's come here to random rows. Let's
23:03:23choose 500. We're going to load this
23:03:25sample. And as you can see we do we do
23:03:27have males in here. And so I'm really
23:03:29glad we did that because now we kind of
23:03:31have a better representation of our
23:03:34data. And so I think what we need to do
23:03:36let's bring over our recipe. I think
23:03:37what we need to do or what we're going
23:03:39to do is one, I want to filter where the
23:03:42gender is equal to male. Then we're
23:03:45going to come over here and we're going
23:03:46to try to find the most common name per
23:03:48year. Or maybe we'll just group by the
23:03:50year and the name and do a count on the
23:03:52name. I think that would be really
23:03:53interesting. So we're really
23:03:54transforming it quite a bit. So what
23:03:57we're going to do is we're going to
23:03:58filter on this gender. So we're going to
23:04:00go over here to filter and we're going
23:04:02to go down to by condition. Now we want
23:04:04to say where gender is equal to male. So
23:04:07you can either do contains and we can do
23:04:09contains an M or we can say it is
23:04:11exactly. Either one of these will be
23:04:14perfectly fine. We're going to do that
23:04:16on gender right here. So we have 300
23:04:19females, 200 males. We don't want the
23:04:21females, so we can get rid of that one.
23:04:23We're going to only keep the males. We
23:04:25can also enter a value in here. Uh but
23:04:27we don't need to do that. And so now
23:04:29this is ready to go. We're going to go
23:04:30down and we can either preview changes,
23:04:32but we're just going to go ahead and
23:04:33apply this. And as you can see right
23:04:35here in our recipe, it says under our
23:04:38applied steps, we have filter values by
23:04:40gender. We can either edit this or we
23:04:42can delete this. So, at any time if we
23:04:44want to change any of this, uh, we can
23:04:46do that. But, as you can see in our
23:04:48preview in our sample, we now only have
23:04:50200 rows and it's all filtered by males.
23:04:54Now, what we can do is we're going to
23:04:56group this data. We want to do it based
23:04:58off of the year and the name. So, let's
23:05:00come up here. We're going to group and
23:05:03we need to select our columns. So, we're
23:05:05going to start with the year and we're
23:05:07going to group by the year. And then
23:05:08we're going to select the name as well.
23:05:11And we'll come down here and we'll say
23:05:13group by. But we also wanted and you'll
23:05:16notice we also wanted to aggregate these
23:05:19values. So, we want the count. So, what
23:05:21we're going to do is we're going to come
23:05:22in here and we're going to say based off
23:05:24of the name, we want a count of the
23:05:27names for that year and name. So, we're
23:05:28going to do a count here. And so, for
23:05:31this new column, you can call it the
23:05:32name count, and that'll be perfectly
23:05:34fine. This is uh just like an example
23:05:36down here. But it shouldn't be a string.
23:05:39This should be an integer. And so, if we
23:05:41come down here, let's see. There should
23:05:43be some that have multiple um higher
23:05:45than one,
23:05:47at least not in our preview. But still,
23:05:50we're working on a sample here. So,
23:05:51let's go ahead and finish this. And now
23:05:54we have our data right here. Now, again,
23:05:57this is only working off the 200 rows.
23:05:59We could have 100,000, 50,000 uh in our
23:06:02data. But we grouped off of the year and
23:06:06then we grouped off of the name and then
23:06:07we got a count of the name. Now, what we
23:06:10need to do is I also want to filter or
23:06:12sorry, sort on this year and we can do
23:06:15that uh just by sorting here manually.
23:06:18So we can come over here and it's just
23:06:19going to do this within our uh window.
23:06:22But I actually want to apply a sort to
23:06:24here. So I want to do this ascending. So
23:06:27the smallest year to the largest year.
23:06:29We'll do this based off of the year.
23:06:31Then we'll come down here. We'll click
23:06:33apply. And now we have our years over
23:06:35here in ascending from smallest all the
23:06:38way down to largest. So now that we've
23:06:40applied all of our steps and really
23:06:42transform our data on our sample data
23:06:44set, let's go ahead and save this
23:06:45recipe. We're going to go ahead and
23:06:47publish this. So you can add some notes
23:06:49if you would like. We're going to
23:06:50publish this. And so now we've saved
23:06:52that recipe. And in fact, if we come
23:06:54over here to recipes, you can see that
23:06:56we've saved this. So we have that
23:06:58available. If we go back to our
23:07:00projects, we can come up here and we can
23:07:02create this job. So we're going to call
23:07:05this one the uh name aggregator. This is
23:07:09the data set associated with it. And we
23:07:12can choose our output. So, where do we
23:07:14want this output to be? Let's put it in
23:07:16S3. Uh, we can keep it right here. Let's
23:07:19go to our S3 location. And let's just
23:07:22put it in the Alex the analyst bucket.
23:07:24Let's select that. And we have some
23:07:26additional options down here. So, we
23:07:28have some advanced settings. One, the
23:07:30maximum number of units. So, that's to
23:07:32do with how many nodes you want on this
23:07:34job when it actually runs. You can
23:07:37specify if it times out or how many
23:07:38times you want it to retry it. This one
23:07:41is probably the more important one is if
23:07:42you want to schedule this right here. So
23:07:45you can come in here and you can create
23:07:46a new schedule and you can specify I
23:07:48want this recurring every and this says
23:07:501 hour but maybe you want it recurring
23:07:52or doing on a specific time of the month
23:07:55and you want to automate this and run
23:07:58this maybe every week or every month and
23:08:00get an output of the data that you have.
23:08:03So this can be really really useful. You
23:08:06can add tags for permissions. We just
23:08:08need to specify our role and we can
23:08:10create and run this job. This is going
23:08:13to take a little bit to run because it's
23:08:15going to be working in a much larger
23:08:16data set. But in a little bit, it's
23:08:18going to run this entire job. It's going
23:08:20to output our CSV into our S3 bucket.
23:08:24And then we're going to go and take a
23:08:26look at it. So, let's go over here to
23:08:28jobs and see its status is still
23:08:31running. Let's wait for just a little
23:08:33bit. And it's going to be doing that on
23:08:35the entire uh project. that's not just
23:08:37going to be doing that on the sample.
23:08:39It's using the uh full data set. And so
23:08:41let's wait for this to be done and then
23:08:42let's go look at our output. As you can
23:08:44see, it succeeded. It took about 2
23:08:47minutes. It finished up very quickly or
23:08:49just a second ago. Let's come up here to
23:08:50our S3 bucket and it should be right in
23:08:54here. So let's go ahead and refresh
23:08:56this. And now you can see we have this
23:08:57new folder name aggregator. Let's click
23:08:59in it. And now we have all of these
23:09:02CSVs. Now, this is to be expected
23:09:04because this is the default option
23:09:06within Glue Data Brew. I personally do
23:09:09not like this. I want a single CSV
23:09:11output as I'm sure most of you do. Let's
23:09:13come back up here to our jobs and let's
23:09:15go into this name aggregator. Let's come
23:09:17over here and we're going to select edit
23:09:19job. And now we have right here. This is
23:09:22our uh job that we just created. Now,
23:09:25what we need to do is get rid of this.
23:09:27So we need to go to settings because
23:09:28right now for file partitioning we have
23:09:32file output options autogenerate files
23:09:35default file output setting that
23:09:36generates multiple files. This usually
23:09:39results in the fastest job runtime which
23:09:41can be important uh for some use cases
23:09:44especially as you scale your company or
23:09:46your business or your department. You
23:09:48know these things do matter a lot but
23:09:50for what we're doing we want a single
23:09:52file output. We also have an option out
23:09:55here for our file storage where we can
23:09:57create a new folder for each run or
23:09:59replace output files for each job run.
23:10:02We'll keep it the create, but depending
23:10:04on what you're doing, uh you may want to
23:10:06replace it as well. Let's go ahead and
23:10:08save this. Then we're going to come down
23:10:10here and we're going to save it as well.
23:10:13And now we need to actually run this. So
23:10:15now we're going to click run job and
23:10:17we're going to run the same one except
23:10:19as a single file output. Let's go ahead
23:10:21and run this. Now it's going to be
23:10:22running. It's going to take a little bit
23:10:24longer than the minute 37 seconds, but
23:10:26hopefully not by a lot. So, let's wait
23:10:28just a little bit and then we will see
23:10:30what it looks like in our S3 bucket. All
23:10:32right, this succeeded and somehow took
23:10:34less time. Uh, so AWS is just messing
23:10:37with us right now. They're lying to us
23:10:38completely. Let's go back to our bucket.
23:10:41We're going to come back just to the
23:10:43bucket. Let's go ahead and refresh this.
23:10:45There's our next one. I think this is
23:10:48the second one. Uh they have different
23:10:50names obviously because we didn't
23:10:52overwrite. Let's come into here. And now
23:10:54we have our CSV. You can go ahead and
23:10:56download that. And in fact, I'm not just
23:10:58going to tell you to download it. I'll
23:10:59download it, too. So, let's come in
23:11:00here. Let's go and download this. All
23:11:02right. Let's open this up. And let's
23:11:04take a look. So, let's come over here.
23:11:07Let's filter this real quick cuz there's
23:11:09a lot in here, I'm assuming. Let's come
23:11:12over. And it looks like there's only one
23:11:14only most common name per year. Uh,
23:11:17apparently I completely misunderstood
23:11:18the data. Uh, this is probably what they
23:11:21did was they took they told us the most
23:11:23common year or the most common name per
23:11:25year. Um, and I just wasn't thinking
23:11:27about it. So, I think we misunderstood
23:11:28the assignment. Actually, you know what?
23:11:30I'll take the blame. I'll take the
23:11:32blame. I misunderstood the assignment.
23:11:33But, we do have 25,779
23:11:36rows. Uh, and so, you know, just a
23:11:38little example of how to use it.
23:11:41Although I'm not an expert on that data
23:11:42set. So, uh I take full blame for that.
23:11:45But you can see the process that we took
23:11:47on how to clean the data, change the
23:11:48data, and get a correct CSV output.
23:11:51Let's go back over to AWS. Let's go back
23:11:53to our analyst bucket, and we'll stick
23:11:56right here for just a second. Now, one
23:11:58thing to note, and this is something
23:12:00that you may, some people may have
23:12:02encountered when they were trying to run
23:12:04this job. It's it may say, oh, you don't
23:12:06have access or the ability to do that.
23:12:08You may have gotten an error in some
23:12:10way.
23:12:11What you may need to do, and this is
23:12:13something that uh my friend Cassoon over
23:12:15at Analyst Builder, he helped me
23:12:17understand, is that sometimes AWS
23:12:20doesn't give you full functionality
23:12:21until you have certain services running
23:12:23like an EC2 instance. Now, we're not
23:12:25covering EC2 instances in this series,
23:12:28but you may need to come in here into
23:12:30the EC2 and you can just go to services,
23:12:32go make sure to go to EC2, you may need
23:12:35to just launch an instance and just
23:12:37launch it. And what happened after I
23:12:39launched it was it sent me an email and
23:12:41said, "Hey, you now have full
23:12:42functionality for different things. It's
23:12:44super easy. You just come in here, make
23:12:46sure you select your uh pair. You can
23:12:47just say you don't want one, and then
23:12:49you launch an instance." And that's it.
23:12:51And after that, you'll hopefully within
23:12:53a few minutes or, you know, within the
23:12:54day, you'll get an email saying you have
23:12:56full functionality. You may need to do
23:12:58this. That's just something I want to
23:12:59make mention of because I know I
23:13:02encountered that when I first did it.
23:13:04Now, I had never encountered this before
23:13:06because I've used it in a workplace
23:13:08where somebody already had all this
23:13:10stuff set up, right? We this was a fully
23:13:12functioning production and development
23:13:13environment and so I didn't have to
23:13:15worry about this. But this is a
23:13:16completely new account. And so this is a
23:13:18free instance. You know, you don't have
23:13:20to spend money on this because you get
23:13:22two EC2 free tier offers and you can use
23:13:25those and it should uh be good to go.
23:13:27And you need to do that for what we're
23:13:28about to do in just a second. Um, but
23:13:31just wanted to mention this. So that is
23:13:34how you can use glue datab. Uh and of
23:13:38course I want you to get in here. I want
23:13:39you to take a look at a bunch of this
23:13:41other stuff. You can create rules as
23:13:43well for invalidating your data and you
23:13:45can send yourself emails. Uh if you know
23:13:47it doesn't look good. You can come in
23:13:49here and look at all your data sets that
23:13:51you have. This is the one that we were
23:13:52using the data set national baby names.
23:13:54And of course you can always connect to
23:13:56new data sets. So if you want to come in
23:13:58here, you want to pull data out of an S3
23:14:00bucket or from Redshift or you know
23:14:03other options within AWS, you can do
23:14:05that. So all really really good stuff.
23:14:07Now let's go back. We're going to go to
23:14:09all services and let's click up here. We
23:14:12are going to go all the way down to
23:14:15Glue. Now here's what I'll say about
23:14:17Glue before we even click into it. Glue
23:14:18data brew is very visual. I think it's
23:14:20actually fairly easy to use uh compared
23:14:24to glue. Glue, I think, is a little bit
23:14:26more complicated. Um, not entirely, but
23:14:29it is. Um, and so let's look at glue
23:14:32really quickly. Now, over on this lefth
23:14:34hand side, we have a ton of different
23:14:36stuff. Um, and we're not going to be
23:14:37looking at everything in here. I'm going
23:14:39to look at kind of the more important
23:14:41things, but we have something like a
23:14:42data catalog. And this we looked at in
23:14:44our lesson with Amazon Athena. They have
23:14:47the data catalog to set up your
23:14:49databases and tables and your schemas.
23:14:51And we had crawlers. Now, this is the
23:14:53first thing that we're going to look at
23:14:54within Glue because this is how you can
23:14:56kind of automate pulling in data,
23:14:59getting the data types, and pulling in
23:15:00that data. And so, it's really useful.
23:15:02The next is creating ETL jobs. So, those
23:15:05are the two things that we'll be looking
23:15:06at in this lesson. There's a ton of
23:15:09other things that it does. So, be sure
23:15:12to get in here and just check everything
23:15:13out because uh Glue does a lot of
23:15:16different stuff, and I'm just not
23:15:17covering everything, of course, because,
23:15:19you know, this would be a 10-hour uh
23:15:20lesson. So we have a few different
23:15:22things. One, you can prepare your
23:15:24account for AWS Glue. Here's your
23:15:26catalog for your data sets. And then
23:15:28here's how you move and transform your
23:15:30data. And so what we're going to do is
23:15:32we will need to set up ROS and users.
23:15:34We're just not going to do it right now,
23:15:35but we will need to set that up. And I'm
23:15:36actually going to show you some of the
23:15:38IM uh you know on the back in the IM uh
23:15:42resource how that actually looks. But
23:15:44let's come over here to crawlers.
23:15:46And we don't have any crawlers created,
23:15:48but we need to. Let's create a crawler.
23:15:52And let's create this. And we'll call
23:15:54this the uh Alex the analyst crawler
23:15:58example. Let's go ahead and click next.
23:16:02Now, we need to specify our data source.
23:16:04Now, just let me back up one second. You
23:16:07know, we want to pull in data to be able
23:16:08to use it in different services, and we
23:16:10want to do it kind of automatically,
23:16:12like I mentioned. And so let's say we're
23:16:13pulling in data and we want to put it
23:16:14into a database or we want to pull in
23:16:16data and we want to put it into Athena.
23:16:18These are things that we can automate.
23:16:20And so uh we want to specify a data
23:16:23source here. So let's go ahead and add a
23:16:25data source. We have our S3 bucket.
23:16:28That's what we want to do. And this is
23:16:29data that we used in previous lessons.
23:16:31So if you didn't take those lessons, be
23:16:33sure to go and do that. So we're going
23:16:35to browse this data. Let's go into the
23:16:37Alex analyst bucket. We're going to go
23:16:38into patient data. We can specify single
23:16:43files but let me just tell you it won't
23:16:45work. Um and you know that's just
23:16:47because of how the folder system works
23:16:49within glue. So we need to specify the
23:16:52entire patient data folder. Let's go
23:16:55ahead and choose this. And now we have
23:16:57on subsequent crawler runs crawl all
23:16:59subfolders subfolders only. If you add
23:17:01new subfolders to your folder then based
23:17:04off of an event. So an event would be
23:17:06like something triggering and then you
23:17:08uh it runs based off of that. So, we're
23:17:10just going to do crawl all subfolders.
23:17:12Let's add this data source. Let's go
23:17:14ahead and click next. Now, this is the
23:17:17part where we need to select an IM RO.
23:17:18We don't have one. So, let's go ahead
23:17:20and create a new IM roll. I'm going to
23:17:22call this Alex glue crawler roll. You
23:17:27can call this anything. Let's go ahead
23:17:29and create this.
23:17:31And it says it successfully created it.
23:17:33Let's go ahead and view this.
23:17:38So now we're in a totally different part
23:17:40of AWS. This is the identity and access
23:17:43management. That's our IM. So right here
23:17:46we created an IM role. This is the Alex
23:17:48group.
23:17:50This is the Alex Glue crawler role. And
23:17:53then down here we have the permissions
23:17:55policies. So right here this is going to
23:17:58be giving us access to I believe the S3
23:18:00bucket. We can come in here and actually
23:18:02look at the policy. So the permissions
23:18:04are read and write on a specific uh data
23:18:08source and that's going to be our bucket
23:18:10with the patient data and I think we can
23:18:12come up here to policy versions. Look
23:18:16right here and so this is what it looks
23:18:18like inside of it. Now this is in JSON
23:18:20and you can actually customize these and
23:18:22I'm not going to go into all how to do
23:18:24that but sometimes you need to depending
23:18:26on the data the data source if it's
23:18:28encrypted if it's not um what
23:18:30permissions you want it to do. But what
23:18:32this is doing is it's allowing us to get
23:18:34an object and put an object in this
23:18:36specific resource. So that's our patient
23:18:38data and that star is just a wild card
23:18:40to say anything in that file path. So if
23:18:42we go back and let's come back to this
23:18:44role. We also have this AWS glue service
23:18:47role. Now this one right here we created
23:18:50when we created the crawler to specify
23:18:53the file path of the data set. This is
23:18:55AWS manage. This is one that AWS creates
23:18:58themselves and we can look at all the
23:19:00things. Let me scroll down. We can look
23:19:02at all the things that it does uh for
23:19:04that AWS Glue service role. Gives us
23:19:07access to a ton of things for EC2
23:19:08instances, S3, everything within Glue.
23:19:11And we can come down here and take a
23:19:14look at the buckets, the access to S3 as
23:19:17well as a bunch of other stuff. And so
23:19:19this is what the IM roles look like. And
23:19:21you can create custom policies uh if you
23:19:24want to. And you can give access to
23:19:26different things. And so again, we're
23:19:28not covering this entirely in this
23:19:30lesson because this is not a lesson on
23:19:32IM roles, but I think it is interesting
23:19:34and worth uh worth knowing. So we've
23:19:36created our role. Let's go down over
23:19:38here. Let's click next. And now we need
23:19:41to specify where we're going to be
23:19:43putting this. So we're going to put this
23:19:45data in our healthcare data. So this is
23:19:48our healthcare data. Again, we already
23:19:50had created this database back in our
23:19:52Amazon Athena. And we have the option to
23:19:55uh create a table name prefix or use the
23:19:58maximum table threshold for the prefix.
23:20:01Let's just say uh crawler. There we go.
23:20:04And now we have this crawler schedule
23:20:05down here. Now you can do it on demand
23:20:07or you can specify whether you want a
23:20:10specific day of the month, whether
23:20:11weekly, daily, monthly, whatever you
23:20:13need. You can create this uh
23:20:16scheduleuler in order to run this
23:20:18crawler. Let's go ahead and click next.
23:20:20And let's just review this real quick.
23:20:21This is everything that we chose. We're
23:20:23going to go ahead and create our
23:20:25crawler. So now it says one crawler was
23:20:27successfully created. It's this one
23:20:29right here, Alex Analyst crawler
23:20:30example. And uh it has not run yet. So
23:20:34we've have no crawler runs. But let's go
23:20:36ahead and run this crawler. So now we're
23:20:38going to start the crawler. It's going
23:20:39to start up this that we just created to
23:20:41pull in that patient data. And once it
23:20:44is completed, we're going to go take a
23:20:46look at it in Amazon Athena like we
23:20:48looked at in a previous lesson. So right
23:20:50down here it says it's running. So once
23:20:52that's done running, we're going to take
23:20:53a look at that data. All right, it says
23:20:54that was completed. Now we can come back
23:20:58here to the data catalog connections.
23:21:00Let's go ahead and refresh this.
23:21:03You can see we have our crawler that's
23:21:05our uh prefix that we used and we have
23:21:08patient data. This patient data and
23:21:10patient data 2 were data cataloges and
23:21:12part of our data catalog are tables that
23:21:15we created in Amazon Athena. Now let's
23:21:18come up here. Let's just duplicate this
23:21:20really quickly. We're going to go back
23:21:22to Amazon Athena. Let's see if it's
23:21:24right here. So, here we go. We have
23:21:26Athena.
23:21:27And now we can access that data right
23:21:30here. So, now if we want to, I can say,
23:21:32uh, let's just preview the table.
23:21:35It's running. And we have our data in
23:21:38here. And so, this is perfect. Um, if
23:21:40you remember, we had our patient data
23:21:42and patient data 2. We manually entered
23:21:45that data to pull that in. And that was
23:21:47not super fun. So when we created it, we
23:21:49did it from an S3 uh bucket data. But
23:21:52with the AWS glue crawler, we can do
23:21:54this automatically and then it'll
23:21:56refresh our data and we can create this
23:21:58uh without having to manually do this
23:22:00every time we get a new data source,
23:22:02which let me tell you, when you start
23:22:03getting into a production environment or
23:22:05a development environment, whichever one
23:22:06you're in, when you have to start
23:22:08bringing in lots of data and the data is
23:22:10coming in daily or weekly or monthly,
23:22:12you do not want to have to manually do
23:22:14this. It takes up so much time. It is
23:22:16much better to automate this with an AWS
23:22:18glue crawler. So that's during that
23:22:20scheduling part right in here and we
23:22:23come over to crawlers. We come in here.
23:22:26This is just on demand, but we can edit
23:22:29this and we come down here and we can
23:22:31schedule it down here if we want to. But
23:22:34you can also now we have it on demand.
23:22:36So anytime we want to run it, we can
23:22:37also just run it on demand. Um so it's,
23:22:40you know, really really great to have
23:22:42and use. So that's how we use crawlers.
23:22:44And crawlers are amazing. Oftentimes, if
23:22:47you're using this in your work, you're
23:22:48going to have tons of them, like 50 to
23:22:50100 of them um running, and you'll be
23:22:53scheduling them, and some of them will
23:22:54break because a data set changed or
23:22:56whatever. Um and so this is a really
23:22:59great thing to test out, try with
23:23:01different data sets, uh and really get
23:23:03familiar with it because crawlers are
23:23:04amazing. Let's go down here. Let's go to
23:23:07ETL jobs. So, this is the last thing
23:23:10that we're going to be taking a look at
23:23:11in this lesson. There is of course more
23:23:13to AWS glue, but this is kind of like
23:23:16the two most important things I would
23:23:18say. So, let's go over here to a visual
23:23:21ETL. And what we're going to do is we're
23:23:23going to build out I don't think I have
23:23:25an unsaved job. We're going to build out
23:23:27um our very first ETL job within AWS
23:23:31Glue. Now, what we have here, and I'll
23:23:33talk a little bit more about this in a
23:23:35little bit. We have parent node, and we
23:23:37have uh the child node. I think it's
23:23:40node if that's the right term, but we
23:23:41have parent and child uh different nodes
23:23:44that we have in here and we need to make
23:23:46sure we chain them properly. Now,
23:23:48luckily we have this ETL visual tool and
23:23:50that's what we're in right now is be
23:23:52able to see what we're doing when we're
23:23:54in here. Now, let's come over here and
23:23:56let's go back to our sources because we
23:23:58have to have a source to start out with.
23:24:00So, let's go to Amazon S3 and we're
23:24:03going to click into this Amazon S3. So,
23:24:06we're going to pull data from our Amazon
23:24:09S3 bucket. We need to specify our
23:24:11location and we can either make it
23:24:13recursive or not recursive and I'll
23:24:15explain that in just a second. Now,
23:24:17let's come in here. We'll go into our
23:24:18bucket. We'll go into our patient data.
23:24:21Now, let's specify just this first file
23:24:23here. We really don't need recursive
23:24:25because I don't have any sub
23:24:26directories. So, we can turn that off or
23:24:28you can keep it on. We also have a data
23:24:31format. The our data format is not uh no
23:24:34format. We have a CSV format which is
23:24:36commaepparated.
23:24:38Now what it's kind of prompting us to do
23:24:40down here and I highly recommend doing
23:24:42it is getting a data preview. So we can
23:24:44see the data as we're transforming it
23:24:45and changing it. So I am going to come
23:24:48in here and I am going to specify this
23:24:50role that we had created earlier. And
23:24:52we're going to start this session. This
23:24:54could take just a little bit of time but
23:24:55we're going to get a data preview down
23:24:57right here at the bottom. And now our
23:24:59data preview is ready. We're looking at
23:25:02uh this file right here. Next, what
23:25:03we're going to do is we're going to come
23:25:05over here to add a node and we're going
23:25:06to go to transform and we're going to
23:25:09join. Now, we have to specify
23:25:13uh a few things here. One, we can name
23:25:15this. So, it's just called a join. Next,
23:25:17our node parent, and this is what I was
23:25:19talking about before with the parents.
23:25:22So, this parent is our first data
23:25:24source, which is just called Amazon S3,
23:25:26which we could change. We can come in
23:25:28here. We can call this uh patient data
23:25:31one. We can call this patient data 1.
23:25:33Then we'll come down here and go to the
23:25:35join. And notice if you come in in here,
23:25:37we only have one data source. So we need
23:25:40to specify a second data source. So
23:25:43let's come back out here. Let's come up.
23:25:45Let's go back to our sources and we'll
23:25:47pull in our second one. I'm actually
23:25:50going to change uh this view really
23:25:52quickly. I like it going left to right,
23:25:54but some people like it going up to
23:25:56down. Um, you can just do that using
23:25:58this button right here, which is the
23:25:59direction, but I like this direction a
23:26:01little better. So, we're going to come
23:26:02into this bucket. Let's call this
23:26:04patient data 2. And let's specify
23:26:09our other patient data, which is our
23:26:10second, which is number two, which
23:26:12actually says file one in it, but you
23:26:14know, just ignore that. Okay. Um, we are
23:26:18going to do that right there. We already
23:26:19have our preview. And now we need to go
23:26:21back to our join and specify these are
23:26:24the two things that we are joining
23:26:26together. Now if we do uh an inner join,
23:26:29it's not going to work cuz these data
23:26:30sets are extremely uh familiar. In fact,
23:26:33if we come in here and look at the data,
23:26:35there aren't any patient datas to join
23:26:37on or patient IDs to join on. Um they're
23:26:40all unique. And so what we're going to
23:26:41do is we should actually be doing a
23:26:44union here. Let's actually get rid of
23:26:45this. You know what? That was my
23:26:47mistake. Let's go to transform.
23:26:50Let's come down. Let's find a union or
23:26:52let me search for it. So, I'm going to
23:26:54do a union. So, we're going to do a
23:26:55union right here. There we go. And let's
23:26:59go into our union. And we're going to
23:27:00specify our parent nodes. We have boom
23:27:02and boom. That's one and two. And it's
23:27:04going to be working on this data
23:27:06preview. There we go. Let's scroll down
23:27:08a little bit. And now they're all in
23:27:10there, which is perfect. That's exactly
23:27:12what we want. File one and file two,
23:27:14which is actually file one and file two.
23:27:16So, we are good to go. This is looking
23:27:18really, really good. Let's come back
23:27:20here and let's take a look at what comes
23:27:24next. So, we've uh created our sources.
23:27:27We had two data sources. Let me pull
23:27:28this over here. And I can't pull this
23:27:31over anymore. Um, but we have our data
23:27:33sources. And again, you can pull this
23:27:35data in from anywhere. We've done one
23:27:37transformation. Now, we can do other
23:27:40transformations as well. We can even
23:27:42write a SQL query. We can look at uh
23:27:44filling in missing values if we need to.
23:27:46We can aggregate our data, drop
23:27:49duplicates. These are a lot of the
23:27:50different things that we looked at in
23:27:51Glue Data Brew. Um, and there's even
23:27:53more options uh in here as well that are
23:27:56kind of unique to Glue. But lastly, what
23:27:59we need to do is then we need to specify
23:28:01our target location. Where do we
23:28:04actually want this data to go? And so we
23:28:06can put it into things like an Azure uh
23:28:09SQL database. It could go into
23:28:11Snowflake. and go into a SQL server,
23:28:13Postgrace, MySQL, Redshift, or we can
23:28:15just put it back in an S3 bucket, which
23:28:17is by far uh the most simplest thing you
23:28:19can do. We could also take this data and
23:28:22put into the glue data catalog. And so,
23:28:24if we want to use this in Athena, we
23:28:26have a much, you know, bigger process
23:28:27and then we want to put it into or query
23:28:30that data in Athena, we can do that. So,
23:28:32there's a lot of different places and
23:28:34things that we can do. Let's just come
23:28:36back and put it into an S3 bucket
23:28:38because that is going to be the simplest
23:28:40thing to do. So we have our process and
23:28:42there at the very end we're going to put
23:28:44it into a CSV file. We don't need any
23:28:47type of compression here but uh we do
23:28:50have an option data catalog update
23:28:52options. Let me zoom in while we're
23:28:54here. We have our data catalog update
23:28:56options. So we can create a table in the
23:28:58data catalog on subsequent runs update
23:29:00the schema and add new partitions or we
23:29:02can create a table in data catalog and
23:29:04on subsequent runs keep existing schema
23:29:06and add new partitions. So, I'm going to
23:29:08click on this one right here. And we'll
23:29:10specify our database. That's going to be
23:29:11our healthcare data. Our table name,
23:29:14we'll call this one uh the ETL. I'll do
23:29:17underscore patient data. And then we'll
23:29:20come up here for our S3 target location.
23:29:23So, we're going to come back. We'll just
23:29:24place it in the Alex the analyst bucket.
23:29:27Let's go ahead and choose this. And we
23:29:29have a fully functioning ETL process. We
23:29:32took two data sets, we union them
23:29:34together, and then we have our output.
23:29:36And of course, we can see every step of
23:29:38the way what we're doing here. Now, if
23:29:40we wanted to uh because we already have
23:29:42this full process, right? If we wanted
23:29:45to, what if we wanted to add in an extra
23:29:47part? Let's say we wanted to come in
23:29:49here and we wanted to transform this
23:29:51union. Let's do an aggregation. So, we
23:29:53wanted to perform an aggregation on
23:29:55this. So, let's go to aggregation.
23:29:58Now, this doesn't look right. Right. We
23:30:00can't union it and then aggregate it and
23:30:04also send it out. Well, we can, but it
23:30:07doesn't really make sense for what we're
23:30:08doing. So, what we're going to do is
23:30:10let's say we want to union on maybe a
23:30:12diagnosis and look at the average age or
23:30:15maybe a treatment. Uh, could be
23:30:16anything. But let's come down here.
23:30:19Let's click on this aggregate. What we
23:30:22want to do is we want to put it where
23:30:24this is its parent. And then for the S3,
23:30:26the new parent for this is going to be
23:30:28the aggregation. So, this is correct
23:30:31because you can see we have this line
23:30:32flowing here. This part is correct. we
23:30:34will need to aggregate it in just a
23:30:36second. But we need to come up here and
23:30:38we need to change this parent node to
23:30:40the aggregate and then get rid of the
23:30:43union. And so now we've changed our
23:30:47workflow here. We've changed the ETL
23:30:49process. So now we need to go to
23:30:50aggregate. Now we're going to come over
23:30:52here and we need to be able to aggregate
23:30:54our data. So we have to uh select the
23:30:56fields to group by and we have to
23:30:58perform our aggregation on a specific
23:31:00column. But when we come in here, notice
23:31:04all of these are string. We have string,
23:31:06string, string, string, and string. And
23:31:09that's not correct. Uh that's actually
23:31:11incorrect. What we need to do is we need
23:31:14to come in here and after this union, we
23:31:17need to add an additional step. That's
23:31:18going to be our change schema. And then
23:31:22we'll select this aggregate real quick.
23:31:24And we're going to put it on the change
23:31:26schema. There we go. So we're just
23:31:28adding multiple steps here, but we need
23:31:30to specify what our data actually is.
23:31:32And we can see when we make the change
23:31:34what'll happen. So we can keep this or
23:31:36we can make this an integer. The uh name
23:31:39is going to be a string. The age needs
23:31:41to be an integer as well. We have string
23:31:44for both of these. And then a file needs
23:31:46to be an integer. And so this is looking
23:31:49really really good. Everything should be
23:31:52working properly. We shouldn't have any
23:31:54mistakes. Um, often times it'll, you
23:31:56know, show all blanks if you're doing
23:31:58something wrong. So now we can come over
23:32:01here to our aggregation and say we
23:32:03wanted to do this based off of the
23:32:06diagnosis. Then we come down to here.
23:32:08This is the um aggregation. Which field
23:32:10do we want to aggregate on? It's going
23:32:12to be on age. And we'll just do let's do
23:32:15an average. Just keeping it simple. So
23:32:18now this is good. And we can even take a
23:32:20look at this right here. So this is our
23:32:24data that we're going to be outputting
23:32:26into a CSV file. And then for our data
23:32:30target, we've already specified
23:32:32everything that we need. And of course
23:32:34creating the data catalog as well. So
23:32:37this is ready to go. Let's go ahead and
23:32:40uh let's go over to job details just
23:32:41really quick. We want to change this.
23:32:43We're going to say uh first ETL. There
23:32:46we go. We can come down here and there's
23:32:49a few different things that you can
23:32:50change if you want. You can change the
23:32:53type of job that it's going to be. You
23:32:55can change the glue version, the
23:32:56language, the worker type, bunch of
23:32:59different stuff in here, but I don't
23:33:00recommend uh you doing any of that if
23:33:03I'm being honest. The other thing while
23:33:05we're in here, and I'm just going to
23:33:07mention this, although I'm not going to
23:33:08show this to you uh because this gets
23:33:10quite complicated. I do cover this in
23:33:12the AWS and Azure course on Analyst
23:33:14Builder, but it gets a little bit
23:33:16complicated in here. But this is the
23:33:18code that's being generated from your
23:33:20visual. So this visual is actually
23:33:22writing uh to AP py file. Now you can
23:33:25get in here and change a lot of things.
23:33:28Um and actually it's helpful if you want
23:33:31certain functionalities to get in here
23:33:32and change these things, but we're not
23:33:34going to be doing that in this lesson. I
23:33:36just wanted to show this to you. If you
23:33:37want to edit the script, you won't be
23:33:39able to use the visual ETL anymore cuz
23:33:41that's for kind of simpler visual things
23:33:44where you're doing it exactly how they
23:33:46have it kind of made for you. But if you
23:33:48want to go and start doing custom
23:33:49things, which you can do, and it's
23:33:50pretty awesome, you're just going to
23:33:52have to confirm this, and then you won't
23:33:53be able to use the visual anymore. So,
23:33:55just be warned. Uh that is uh that is
23:33:58something. So, job has not been saved.
23:34:00Let's go ahead and save this. This is
23:34:02our first ETL. We successfully created
23:34:04this. And now, we need to run this. So,
23:34:06let's go ahead and run this. Let's go to
23:34:08our run details. This process is
23:34:11running, and we had multiple steps of
23:34:12our first ETL uh job that we created.
23:34:15Now we're going to let it run and then
23:34:16when it's done running we'll take a look
23:34:18at the output. It looks like our run
23:34:20failed this happens. Let's see what
23:34:23actually occurs. It says an error
23:34:24occurred when calling the py write
23:34:27dynamic frame access denied. Um so let's
23:34:31go back. Uh let's go to visual. Let's go
23:34:35see which role we actually took. Or
23:34:38maybe uh that's in a different place cuz
23:34:40maybe we need to go to the im and give
23:34:42uh different uh different options here.
23:34:44And here we go. So now we have this AWS
23:34:46uh glue service ro. Let's come over
23:34:48here. Let's go in and actually look at
23:34:51this glue crawler ro. Maybe we'll give
23:34:53it admin privileges uh just for this
23:34:55example, though you probably wouldn't do
23:34:57that in real life. But let's go over to
23:35:00uh let's go let's go to the EC2 cuz we
23:35:02don't need that one. Let's go to the IM.
23:35:05So let's go back to all services. It
23:35:08should be right over here. Yep. In the
23:35:10security identity compliance.
23:35:12Let's click into here. Let's go to our
23:35:16rolls.
23:35:18And it should be this Alex glue crawler
23:35:20roll. Let's make sure that's the right
23:35:21one. Alex glue crawler roll. So, let's
23:35:23go into this Alex glue craw crawler
23:35:25roll. It's tough to say. Um, it is
23:35:27possible that we didn't have the correct
23:35:29access. So, let's go into I guess we we
23:35:32are getting um we're getting a lesson in
23:35:34a little bit in IM. We're going to
23:35:35attach a policy to this. We're just
23:35:37going to give our guy straight up admin
23:35:38access. So, we're going to say uh admin.
23:35:40Now, there's a bunch of different admins
23:35:42in here. There should be one just says
23:35:44admin full access. Maybe it's this one.
23:35:47Yeah, it's allow everything. So, this is
23:35:48the one we're looking for. So, we're
23:35:50just going to attach this policy to this
23:35:53person. I think it was an S3 uh thing. I
23:35:56think we just didn't have access to the
23:35:58correct bucket or something. Um I'm not
23:36:01100% sure, but this is saved. It was
23:36:03attached to the RO. Let's go and try to
23:36:05run this again. So, let's go. Um
23:36:09there's our visual, but we can just run
23:36:11this again. Then we'll go to runs and
23:36:14hopefully this will not fail again. If
23:36:16it does, we'll try to work through it,
23:36:18right? These things happen. This is very
23:36:20common. So, let's go ahead and give it a
23:36:22go and see if this one works this time.
23:36:24And just like that, we figured it out.
23:36:27We always do. It succeeded. Um so, it's
23:36:29just a permission issue. And uh believe
23:36:32it or not, that is extremely extremely
23:36:34common. Um so, that succeeded. Let's go
23:36:37over here to our query editor. Let's go
23:36:39ahead and refresh this. We have our ETL
23:36:42patient data. Let's go ahead and preview
23:36:45this table.
23:36:47Now, it's being separated by commas. It
23:36:49didn't uh doesn't look like it separated
23:36:51out exactly how we wanted or maybe we
23:36:53did it wrong. Uh I'm not sure, but it's
23:36:57in there. It's just not in the right
23:36:59format. So, that's something you can
23:37:00definitely fix. Let's go over to our
23:37:02bucket. Let's go right here and refresh
23:37:05this. If we come down here, we have a
23:37:08ton of these really tiny files, like 21
23:37:11bytes. Uh, really, really tiny. Let's go
23:37:14down to the bottom. This unsave. Let's
23:37:17see what this is. We have these CSVs in
23:37:19here. Let me let's go into this really
23:37:22quick and let's download this and let's
23:37:25open this up. And so, this is our
23:37:28original data its. So, this is part of
23:37:30the original uh data set. This is not
23:37:33part of our actual output.
23:37:35Let's go back.
23:37:40If we come down here, let's take a look
23:37:42at any of these. This is just part of
23:37:44the run object. And so that is not a
23:37:47file.
23:37:49And let's scroll down. So we're getting
23:37:50this really weird output. Let's actually
23:37:52go back. Let's go back to our visual.
23:37:57And so for the format, we chose CSV,
23:38:00which should be fine. uh we have
23:38:03compression type as none but maybe we
23:38:05need a compression type and then we
23:38:07putting this in the healthcare data and
23:38:09so maybe we don't want the CSV type or
23:38:11maybe we need some type of compression
23:38:13maybe we need to zip this up so let's
23:38:16try doing this in something like a
23:38:18parquet file really great file maybe we
23:38:21want to put in a snappy compression
23:38:23feeling wild um I'm going to keep it
23:38:25like this just for now let's try it but
23:38:28it looks like we need to aggregate this
23:38:30properly and so let's Go back in here.
23:38:32Let's choose diagnosis. I guess it
23:38:34forgot what we were doing over here.
23:38:36We'll do age aggregate function is
23:38:39average. And now that's working again.
23:38:41So let's go over here. Let's uh try a
23:38:44parquet file. Let's save this. And then
23:38:47we are going to run this. Let's come
23:38:49over here to run details and let's let
23:38:52that run and see what happens. It looks
23:38:54like this one failed. Let's come back
23:38:55over here.
23:38:57Looks like we may need some compression.
23:38:59Let's go back to our visual.
23:39:01Let's add snappy compression. Although
23:39:04I'm not a fan. Let's go ahead and save
23:39:06this
23:39:08and let's run this and let's try again.
23:39:13All right, this one succeeded. Let's go
23:39:15down here. Let's click refresh really
23:39:17quick.
23:39:19Go down to our patient data. Actually,
23:39:22we have it right here. Let's go ahead
23:39:23and run again.
23:39:25Let's go down. Still doing the same
23:39:27thing. Let's go to our S3 bucket. I
23:39:30should have cleaned this up first
23:39:31because that's just going to keep
23:39:32getting messy. But if we scroll down,
23:39:35now we have all these parquet files.
23:39:38Now, this is still a lot better than
23:39:39what we had because now we can pull in
23:39:41these parquet files and we can join them
23:39:44together. And that's just kind of a
23:39:45typical run with an ETL job. Um, and so
23:39:48this is actually a lot lot better. If we
23:39:51want it all in one file, unfortunately,
23:39:54there is not a very easy way to do this.
23:39:56We can come in here and go to Amazon S3.
23:40:00There's not really a great way or an
23:40:02easy way to be able to do this. And so
23:40:04what you need to do if if you want it
23:40:05all in one file or you want it in a
23:40:07specific format, you have to script it
23:40:08out yourself. Um, again, it's not crazy
23:40:11hard to do once you've gotten in here
23:40:13and you kind of understand a little bit.
23:40:14But if you want to be able to do some of
23:40:16these custom things within using ETL,
23:40:18this is how you do it. Um, within the
23:40:20visual, it kind of limits you. So this
23:40:22would still be good. this type of ETL
23:40:25process um would still be perfectly fine
23:40:27because it's just going to append all of
23:40:29this uh data if you're putting into a
23:40:31database or something like that. That'd
23:40:33be perfectly fine. And so not as clean
23:40:35of an output as data brew was when we
23:40:38were working with these ones up here,
23:40:41but we are able to do different things
23:40:42that data brew isn't able to do within
23:40:45Glue Studio. And so this is how you use
23:40:47it. This is how you create these ETL
23:40:48jobs. Sometimes you got to get in and
23:40:50you have to customize a little bit to
23:40:51get the exact output that you want or
23:40:54are looking for. Um, but I'm going to
23:40:55have to come in here. I'm going to clean
23:40:57this up. Uh, because this is a mess now,
23:41:00but this is how you use uh, glue and
23:41:03glue data brew. Once you get in here and
23:41:04start trying it out and messing with all
23:41:06this stuff, you'll understand, right? As
23:41:08you start using it more, this is not
23:41:10uncommon. This actually happens all the
23:41:12time and is not a bad thing if you know
23:41:13how to use the data correctly. Now, like
23:41:15I said, there is a ton more to AWS Glue
23:41:18than just these two things, although
23:41:21those are kind of uh some of the bigger
23:41:22things that I wanted you to know how to
23:41:24use, but there are things like workflows
23:41:26and triggers and things like schemas.
23:41:30And these are of course the most popular
23:41:31ones. So, make sure to know these ones
23:41:33and we covered some of them in this
23:41:34lesson. But Glue is very expansive and
23:41:37you're going to use it for a lot of
23:41:38different things. Now, if you're a data
23:41:39analyst and you're working on something
23:41:41like a data collection team, uh, which
23:41:43is something that I worked on for many
23:41:44years, then you might be getting in here
23:41:46and using this quite a bit. But if
23:41:48you're just a data analyst and you're
23:41:49waiting for the data engineer or
23:41:51database developer or whoever's, you
23:41:52know, bringing in this data, if you're
23:41:54waiting for them to bring in the data,
23:41:55most likely you're not going to get in
23:41:57here that much. You might inform them.
23:41:59You might say, "Hey, I was looking at
23:42:01the data. Uh, the data looks bad or
23:42:03there was something wrong with it." So,
23:42:04you're looking for quality issues
23:42:06immediately. Then you'd relay that to a
23:42:08data engineer and say, "Hey, could you
23:42:09go and check out that job that brought
23:42:11in that data and then they would go and
23:42:13check it." That's typically how you know
23:42:15that would actually work. So, just
23:42:16something to be aware of uh with Glue.
23:42:19But, uh, Glue, Glue datab
23:42:23crawlers, as well as ETL, all stuff that
23:42:25I recommend you testing out, trying out,
23:42:27figuring out how it all fits together
23:42:29and works. With all that being said,
23:42:30that is the end of the lesson. If you
23:42:32haven't already, be sure to check out my
23:42:33AWS and Azure course on Analyst Builder.
23:42:36I go a lot more in depth into Glue and
23:42:39everything in it. We even create some
23:42:40custom scripts in here to put it all in
23:42:43one file, which is really uh useful to
23:42:45know how to do. So, if you're curious
23:42:46about how to do that, be sure to check
23:42:48out that course. If you have not, be
23:42:50sure to like and subscribe below, and I
23:42:52will see you in the next video.
23:42:55[music]
23:43:00>> [music]
23:43:05>> What's going on everybody? Welcome back
23:43:07to another video. Today we're going to
23:43:08be taking a look at Quicksite in AWS.
23:43:11[music]
23:43:17Now Quicksite is AWS's data
23:43:19visualization [music]
23:43:19tool. And so a lot of companies when
23:43:21they get into AWS's ecosystem, they have
23:43:24really simple needs for their data
23:43:25visualization and they just use the
23:43:27internal tool which is QuickSite. It's
23:43:29not the most robust tool you've ever
23:43:30seen in your life, but we're going to
23:43:32take a look and see how it actually
23:43:33works and I'll make a lot of comparisons
23:43:35to things like PowerBI and Tableau which
23:43:37are really popular data visualization
23:43:39tools. So without further ado, let's
23:43:41jump on my screen and take a look. I'll
23:43:42scroll down. Let's go to analytics. And
23:43:45under the analytics, we have Quicksite
23:43:47right here. Now, in order to get started
23:43:49with Quicksite, we actually have to
23:43:51create an account. So, we have to sign
23:43:53up for Quicksite. So, let's go ahead and
23:43:55say sign up for Quicksite. For the
23:43:56authentication method, we just want to
23:43:58use the IM federated identities. You can
23:44:01uh use some of these other ones if you'd
23:44:03like. Be sure to specify your region.
23:44:05Make sure you fill out your account
23:44:06info. And then down here, you can uh use
23:44:10an existing role or you can just use the
23:44:11quicksite manage role. Highly recommend
23:44:13just using default, much easier. and
23:44:15allow access and autodiscocovery for
23:44:18these resources. So, if you want them to
23:44:21be able to use your S3 buckets, be sure
23:44:23to click on your S3 bucket and we can
23:44:24say uh here's the bucket we want. Uh,
23:44:27you know, maybe I'll just choose both of
23:44:28them, but here are the buckets that we
23:44:30want you to be able to access as well as
23:44:32all these other things as well. And then
23:44:34we'll come down here and we don't want
23:44:37pageionated reports. And then we'll
23:44:38click finish. So, now it's creating our
23:44:40account and then we'll get access in
23:44:42just a second. All right, that took
23:44:44about 10 seconds. This is the name that
23:44:45I gave uh our account for Quicksite. And
23:44:49let's go ahead and go to Quicksite. This
23:44:51is what's new in Quicksite. Let's go
23:44:52ahead and close out of that. Now, this
23:44:55is the UI for Quicksite. Now, just
23:44:58before we get into, you know, creating
23:45:00stuff and building stuff and all these
23:45:02different things. I just want to talk
23:45:03about Quicksite in general. This is
23:45:05AWS's tool for data visualization. And
23:45:08of course, there are other options.
23:45:10There's PowerBI, there's Looker, there's
23:45:12uh Tableau, there's lots of different
23:45:14options and all those are perfectly
23:45:16acceptable. And in fact, in some
23:45:18instances, they may be better for some
23:45:20things, but within AWS's ecosystem, this
23:45:23is their BI data visualization tool. And
23:45:26so, because it's already integrated into
23:45:27all their services, it does make it a
23:45:29little bit easier to use than some of
23:45:30the other ones. Um, and that's what
23:45:32they're hoping for. They're hoping you
23:45:34use their services and stay in their
23:45:35ecosystem. So you don't have to go
23:45:37anywhere else for any service ever
23:45:39because AWS wants your money and your
23:45:40business of course. Now within Quicksite
23:45:42you can share reports, you can create
23:45:44folders that you can uh share with
23:45:46customers and clients. You can create
23:45:48dashboards. You can of course create
23:45:49stories. We have analysis data sets and
23:45:53topics. And then over here we have some
23:45:55sample analysis. So these are different
23:45:58um things that they've already created
23:46:00visualizations for. But the first thing
23:46:02I want to show you is right over here in
23:46:05data sets. Now these are the data sets
23:46:07that we already have access to. Sales
23:46:09pipeline. Uh let's just click in on one.
23:46:12We get a very slight overview of kind of
23:46:14what this data looks like. We can
23:46:16refresh this data. And so if we're
23:46:18connected to a data source, we can
23:46:20refresh this or we can schedule this.
23:46:22And then we can look at permissions and
23:46:24usage for people who have access uh to
23:46:27this data and how they can use it as
23:46:29well. So just some interesting things
23:46:31within data sets. If we want a new data
23:46:33set, we just come in here and we specify
23:46:35where we want to pull in our data.
23:46:37Whether it's an S3 bucket or we want to
23:46:39upload a file or it's coming in from
23:46:41Athena. There's a lot of different
23:46:42places uh that we can pull it in. There
23:46:45is something up here I just want to
23:46:46note. We're not actually going to be
23:46:47pulling in any data for this. We're
23:46:49going to be using one of their samples
23:46:50just for demonstration purposes. But
23:46:52there's something called SPICE in here.
23:46:54Spice actually stands for superfast
23:46:56parallel in-memory calculation engine.
23:46:59and it's used to do things really
23:47:01quickly in Quicksite. But for example,
23:47:03let's just say we were uh uploading a
23:47:05file. We can choose uh SQL database
23:47:09output. I don't even know what's in that
23:47:10file. Let's see what the preview says.
23:47:12Okay, we have some products. This is our
23:47:14products file. But let's say we select
23:47:16next. We want to bring in uh this file.
23:47:19It's going to import it into SPICE. And
23:47:22so that's going to cost money. That's
23:47:23going to put it into their super fast
23:47:24parallel in-memory calculation engine.
23:47:27and then you were going to pay for that
23:47:29service. Now, that isn't always the
23:47:30case. We don't have to uh we're not
23:47:33going to do that. We're not we're going
23:47:34to cancel that import. But you don't
23:47:35always have to do that, especially if
23:47:37you have other types of data connections
23:47:38to like my SQL or Postgre SQL. You may
23:47:40not need that. But just be aware of what
23:47:42that is cuz that does cost money. Let's
23:47:45go up here to quicksite. We're going to
23:47:46come on back and let's take a look at
23:47:48some of these samples that they have.
23:47:50Let's go into this sales pipeline
23:47:52analysis. So right now we're working
23:47:54with sample data and we're not going to
23:47:56be doing a full project in here in
23:47:59Quicksite. Really what we're doing is
23:48:00just seeing how it works and how
23:48:02everything is built out and how you can
23:48:04customize and build out your own
23:48:06dashboards analysis as well. So if you
23:48:08come in here you can see that we have
23:48:09different types of visualizations very
23:48:12standard types of visualizations uh for
23:48:14any BI tool. And if we click into one of
23:48:17these, let's say we're clicking into
23:48:18this one, you'll see we have an x-axis,
23:48:21a value, and a color. On the lefth hand
23:48:24side, we also have all of our fields.
23:48:26And you can see whether it's a dimension
23:48:28or if it is a measure. Next, you can see
23:48:30that up here we have our data set. This
23:48:32is the data that we are using. And right
23:48:35over here is the type of visualization
23:48:37chart. Now, this should look really
23:48:39familiar if you're familiar with Tableau
23:48:41or PowerBI, both of which I have series
23:48:43on YouTube. These are different types of
23:48:45visualizations that you probably have
23:48:47seen. And so if we wanted to uh change
23:48:50this, we can click on a different type
23:48:52of chart and it's going to visualize
23:48:54that data based off of that visual type.
23:48:57It's not going to be extremely extremely
23:48:59useful. These are ones that I would use
23:49:01all the time is just clicking on one of
23:49:03these and then filling in the x-axis,
23:49:06the value, the color, and all of these
23:49:08different things that it's prompting you
23:49:09to provide. Another thing you'll notice
23:49:11is we have different sheets up here. So
23:49:13within this analysis, we can add new
23:49:15sheets. So we'll make this uh
23:49:17interactive. And now we have a blank
23:49:19sheet here that we can build our
23:49:21visualizations off of. Let's say we
23:49:22wanted some type of doughut chart in
23:49:24here. We have our doughut chart. And
23:49:26it's going to uh prompt us to fill in
23:49:28what it actually needs. So we need a
23:49:30value. Then we need a group slashcolor.
23:49:33So let's say we want the uh region to be
23:49:36our group and color. And for the value,
23:49:39we'll do the weighted uh revenue right
23:49:42here. And we're going to do the sum.
23:49:44That should work. And so now you can see
23:49:46that this is interactive. You can go
23:49:47ahead and click on this. You can click
23:49:49into these different options. You can
23:49:51change these colors. Let's say we want
23:49:52to make this one uh black. There we go.
23:49:55And there's other options up here like
23:49:57format this uh visualization or if we
23:50:00want to maximize it or minimize it. And
23:50:02if we want to add any more
23:50:03visualizations, we just click add and we
23:50:06can add it right over here. And so
23:50:08everything in here is quite
23:50:09customizable. And you can rename these
23:50:11sheets. Let's call this one uh sum by
23:50:14region.
23:50:16Call it whatever we want. This is just
23:50:17for this one uh visualization. But
23:50:19everything is in here is quite
23:50:20customizable. Now once we have our
23:50:22visualizations, we have this properties
23:50:24right over here. And we can change a lot
23:50:26of the titles, subtitles. If it's a
23:50:29doughnut chart, we can make it large,
23:50:31small, medium. We can show the total
23:50:33versus not showing the total. So
23:50:35everything in here is very, very
23:50:37customizable. And we can specify if we
23:50:39want the data labels and the legend on.
23:50:42So if we don't want the legend, we can
23:50:44turn it off and on. Or if we don't want
23:50:45the data labels, we can turn those off
23:50:47and on as well. We also have over here
23:50:50an interactions tab. So if you want to
23:50:52customize some of these interactions or
23:50:55if you want to customize some of these
23:50:56tool tips, you can also do that as well,
23:50:58which happens when you hover over it. So
23:51:01right here on this right hand side, it's
23:51:02giving us the uh field for region and
23:51:05the weighted. And you'll notice when we
23:51:07hover over it, you can see it's the US
23:51:09with the weighted revenue. So we can
23:51:10specify what we want in here. Maybe we
23:51:12want to um add something else. Maybe we
23:51:15want to add the date doesn't make sense
23:51:17for here, but maybe the uh segment.
23:51:20Let's go ahead and do that. And we hover
23:51:21over it. You can see there's a count of
23:51:24the segment which is 12,824
23:51:26for this data set. So you can customize
23:51:28these tool tips as well. Now when we're
23:51:30done with our visualizations, we can
23:51:32come over here to file and we can
23:51:35publish this and save as analysis. So
23:51:37just saving is going to save it for you
23:51:39to pick up later and do whatever you
23:51:41want with it. But let's say we want to
23:51:43publish this. Publishing this dashboard,
23:51:45we're just going to call this uh sample
23:51:47dashboard is going to make it available
23:51:49to everyone. So let's go ahead and
23:51:51publish this dashboard. So now we have
23:51:53uh this first sheet that was created by
23:51:55them, but then we have the second one
23:51:57that was created by us. Now once we've
23:52:00published it, we can then come over here
23:52:02and we can share this. So we can share
23:52:05this dashboard and we can come in here
23:52:07and we can choose anyone who has access
23:52:09to Quicksite. We can say I want to share
23:52:11this with my colleague or the customer
23:52:12or whoever it is and I want to use this.
23:52:15The other thing that we can do is we can
23:52:17copy this embed code. What you can do is
23:52:19you can say hey I want to embed this on
23:52:21my website or on this platform and I
23:52:23want to uh show this visualization. You
23:52:26can embed that using the code uh from
23:52:29this right here. We also can copy this
23:52:31link. And so we'll copy the link. Come
23:52:34right over here.
23:52:38And now we have access to this
23:52:40dashboard. So anybody who has access to
23:52:41it can just use it as a link and that's
23:52:44really helpful as well. But what I
23:52:45thought was really interesting uh within
23:52:47all this is the embedding code. Because
23:52:49not every BI tool is embeddible but
23:52:52quicksite because of course it's with
23:52:53AWS it's all internetbased. It's not you
23:52:56know local to your computer. It's very
23:52:57embeddible into almost you know anything
23:52:59you want to put it into. So let's go
23:53:01back to our sample dashboard. We'll go
23:53:03back to Quicksite. So that's our
23:53:05different analysis, but let's go over to
23:53:07our dashboards. When we actually
23:53:09published our analysis, it went right in
23:53:12here to our dashboard. So now this is a
23:53:14completed dashboard. We would again
23:53:16share with our internal teams or our
23:53:18customers. We also can create data
23:53:20stories. This is something you do need
23:53:22to upgrade in order to receive access to
23:53:24this. But this is very very similar to
23:53:27data stories in something like Tableau.
23:53:29If you've watched my Tableau series,
23:53:30it's very very similar. you can watch
23:53:32this uh video as well. But you create
23:53:34these little stories and you can add
23:53:36narrative and you know why the data is
23:53:38doing what it's doing and what impact
23:53:39that has and it's pretty good. I
23:53:41personally don't use data stories all
23:53:43that much. I mostly would create a lot
23:53:45of KPI metric dashboards and stuff like
23:53:48that but you know there are use cases
23:53:50for data stories of course. Lastly we
23:53:52have these folders. Let's go ahead and
23:53:55create a new folder. We call this one
23:53:57Alex the analyst folder. Once that
23:53:59folder is created, we can come back to
23:54:02our dashboard. We can right click on
23:54:04this and say add to folder. Then we need
23:54:06to specify our folder. So we'll go to my
23:54:08folders and we'll add this. And so now
23:54:12within my folder, if I click in here, I
23:54:14have this dashboard. Now, this is really
23:54:17just an organizational tool for the most
23:54:19part. But when you start using this in a
23:54:21production environment, you're going to
23:54:22have a ton of different dashboards. And
23:54:24so you can come in here and create
23:54:25different ones for different customers,
23:54:27different clients, internal teams so
23:54:29that you can organize all these
23:54:31different dashboards and data stories
23:54:32and whatnot. So you have it all in one
23:54:34place. And so again, that's just super
23:54:36usable, really userfriendly. Now, this
23:54:39is really all we're going to cover in
23:54:41this video. We're not doing a full
23:54:42project in Quicksite. I'm really just
23:54:44demonstrating how to use it, some of the
23:54:46things that they have in here that I
23:54:48think are really interesting, and how
23:54:50you can publish and share and do all
23:54:51those things within Quicksite. So, I
23:54:53hope that that was helpful and if you
23:54:54have not already, be sure to check out
23:54:55my full Azure and AWS course on
23:54:57analystbuilder.com. And if you like this
23:54:59video, be sure to like and subscribe
23:55:00below and I will see you in the next
23:55:02video. What's [music] going on
23:55:03everybody? Welcome back to another
23:55:05video. Today, we're going to be learning
23:55:06data bricks in under two hours.
23:55:14Now, for the past 5 weeks, we've been
23:55:15diving into data bricks. And in this
23:55:17video, we're just putting all that
23:55:18together so you can follow along really
23:55:19easily. We're going to start by taking a
23:55:21quick walkthrough of data bricks and
23:55:22seeing everything that it has to offer.
23:55:24Then we're going to start importing data
23:55:25not just from a flat file but also
23:55:27connecting to a data source. Then we're
23:55:29going to start using their SQL editor as
23:55:30well as their notebooks and visualizing
23:55:32data. After this we'll be using their AI
23:55:34tools like Genie and their AI assistant.
23:55:36And finally we'll be building a full
23:55:38project at the end. The best thing about
23:55:40all this is it is completely free. Data
23:55:41bicks has something called the data
23:55:42bicks free edition. I will leave link in
23:55:44the description. You can make an account
23:55:46by just using your email. That is it. So
23:55:48be sure to go and create your free
23:55:49account so you can follow along with
23:55:51this entire video because we have a lot
23:55:52of things that we are going to cover.
23:55:54With all that being said, I hope you
23:55:55learn a ton because data bicks is a
23:55:57fantastic platform. So let's jump right
23:55:59into it. Now data bicks is an amazing
23:56:01platform. I've been using it for many
23:56:02years and it's built on top of Apache
23:56:04Spark which means it is very good at
23:56:06handling large amounts of data. It's
23:56:08designed for entire teams of data
23:56:10engineers, data analysts, data
23:56:11scientists to work with data
23:56:13collaboratively. That means ingesting
23:56:14the data, analyzing the data and
23:56:16visualizing data all in one place. In
23:56:18this lesson, we're going to be diving
23:56:20into data bricks and look at some of the
23:56:21core functionality and features that
23:56:23data bricks free edition has to offer.
23:56:25And then in the next several lessons,
23:56:26we're going to get really hands-on and
23:56:28start digging into these features and
23:56:29start building things out. With that
23:56:30being said, let's jump on my screen and
23:56:32get started. Before we jump into the
23:56:34platform itself, I just want to show you
23:56:35that this is where you sign up. So, I'm
23:56:37going to have this link down below. So,
23:56:39you can just click on it. You can create
23:56:40an account and you can sign in. There is
23:56:42no credit card information that they
23:56:44take. You just literally sign in and you
23:56:45are good to go. But on this page, you
23:56:47can sign up for the free edition or
23:56:49login, which we'll log in in just a
23:56:50second. But you can also learn a lot
23:56:52more about data bicks free edition and
23:56:55everything that they have to offer. You
23:56:56can see that they have built-in AI
23:56:58agents and they have AI builtin and that
23:57:00is something that we're going to cover
23:57:01in this series. You can also visualize
23:57:04your data and create full dashboards.
23:57:06And of course, you can interact with
23:57:08your data with Python and SQL, either
23:57:10through notebooks or, you know, through
23:57:11a code editor. Let's go back up and
23:57:14let's go and sign up for the free
23:57:16edition. All we have to do is log in.
23:57:19You can do that with Google, with a
23:57:20Microsoft account, or just an email. I'm
23:57:22going to sign in with my Google account.
23:57:24And all I have to do is say where I'm
23:57:25from. So, this is Alex. We have the free
23:57:27edition, and I'm from the United States.
23:57:29Let's go ahead and click continue. And
23:57:31just like that, we are signed up for the
23:57:32Data Bricks free edition. It was about
23:57:34as seamless as it can possibly be. Now,
23:57:37there's a lot to cover. Data Bricks has
23:57:39so many different features. They have so
23:57:41many different things that you can do
23:57:42from building interactive dashboards to
23:57:44writing SQL queries and sharing that
23:57:45with your team and being able to work
23:57:47with your team. You can even build AI
23:57:49agents. There is a lot of things that
23:57:51you can do. And some of this might be a
23:57:52little bit intimidating if you've never
23:57:54used a platform like this before. If
23:57:56you're used to just working something
23:57:57like SQL or R or Tableau, it's a little
23:58:00bit more advanced than that. So there's
23:58:02a lot of things that they have that are
23:58:03all combined into one place. And so
23:58:05we're going to walk through a lot of
23:58:06these things that you see on this lefth
23:58:07hand side to see exactly what you can do
23:58:10in data bricks. Now I'm going to start
23:58:11at the very top because this is a
23:58:14workspace. And a workspace is basically
23:58:16the place that you work and you can
23:58:18collaborate in as well. So you can give
23:58:20your teammates access to your workspace.
23:58:22So if you have data in here, if you have
23:58:23code in here, if you have
23:58:24visualizations, whatever it is, you can
23:58:26share that with them and they can get
23:58:27access to all of your work. Next, let's
23:58:29go to catalog. You can think of catalog
23:58:32as like a schema. So if you have a
23:58:34schema in a database, you're going to
23:58:35have all of your tables and your views
23:58:36and your store procedures and everything
23:58:38underneath it. And this is very similar.
23:58:40So if I go into a workspace or if I go
23:58:43down here into the samples, we have
23:58:45different databases. And so we can click
23:58:47into these databases and we can look at
23:58:49all these different tables. And so this
23:58:50is where you're going to have access to
23:58:51view all of your data and files and
23:58:53tables within data bricks. Next, let's
23:58:55take a look at jobs and pipelines. This
23:58:58is where we start getting into some
23:59:00automation. So here we have an ingestion
23:59:02pipeline, an ETL pipeline, and a job.
23:59:05And each of these does a slightly
23:59:07different thing. If you create an
23:59:08ingestion pipeline, that's going to be
23:59:09like the extraction of the ETL process.
23:59:12You're extracting data to bring it in.
23:59:14Then we have our ETL pipeline, which is
23:59:16the entire process of transforming it,
23:59:18loading, and actually putting it into a
23:59:19table within data bricks. And then we
23:59:22have our jobs that's going to
23:59:23orchestrate it. We're going to be able
23:59:24to say, here's when we actually run
23:59:25these pipelines so that we can time it.
23:59:28We can do it, you know, daily, weekly,
23:59:29monthly, whenever we want to run these
23:59:31pipelines. So, we don't have any
23:59:33created, but when we do, they'll be down
23:59:35here at the bottom. Next, let's look at
23:59:38compute. Now, compute is very simple in
23:59:40the free edition because you don't have
23:59:42any options here. We just have our
23:59:44serverless starter warehouse. This is
23:59:46what you get for free with the free
23:59:48edition. Now, if you go to the full
23:59:50datab bricks product, then you're going
23:59:51to be able to kind of customize your
23:59:53compute to your needs. But for this free
23:59:55edition, we have uh the owner, we have
23:59:57the size, which is a 2x small, and then
24:00:00we have whether it's active or not. Now,
24:00:02right now, we're not running anything.
24:00:04We're not looking at any tables. We're
24:00:05not running any queries. So, it's not
24:00:07active at the moment, but the second
24:00:09that we open something up and start
24:00:10working with real data, it will activate
24:00:12the serverless compute and we'll be able
24:00:14to use it for free. One small nuance to
24:00:16this is there actually is one other type
24:00:17of compute, you just can't see it, and
24:00:19that's for when you run Python in a
24:00:21notebook. It's called a generic compute.
24:00:22It's just for that one specific use
24:00:24case. For the most part, you're going to
24:00:25be using this serverless compute, but I
24:00:27thought it was worth mentioning. Notice
24:00:29when I clicked on compute, it also took
24:00:31me right down here to the SQL warehouse.
24:00:34This is our SQL warehouse that we are
24:00:36using. Next, let's go down to the
24:00:38marketplace. Now, the marketplace is
24:00:40basically just, hey, here's a lot of
24:00:42companies that work with us, that we
24:00:44partner with, we have connectors to, and
24:00:46we have, you know, partnerships with,
24:00:47and allows you to work with them a lot
24:00:49easier. So, here we go. Partner connect
24:00:51integration. So if you want to connect
24:00:53to Fiverr, to PowerBI, to Tableau, DBT,
24:00:56Prophecy, you can easily connect to
24:00:59these. I know at my previous job I
24:01:01connected it with PowerBI a lot and so
24:01:03these are great connectors where you can
24:01:05just kind of search and you can see,
24:01:06hey, do they connect to this tool that
24:01:08we use and most of the time they do have
24:01:10that connector. Now the marketplace has
24:01:12other things as well. If we come over
24:01:13here to products, we have things like
24:01:15tables or files which is data. You can
24:01:18also search for models and notebooks and
24:01:20all sorts of other things. So you can
24:01:22come in here and we can search for
24:01:24files. And so these are all free
24:01:26resources and files that you can search
24:01:28for and use. If you can't find data here
24:01:30in the marketplace, you can always just
24:01:31go to something like Kaggle and get free
24:01:33data and bring that into data bricks as
24:01:35well. Now let's come over here and go to
24:01:37the SQL editor. You can see we have kind
24:01:39of these uh larger overarching sections
24:01:42up here, but then we have SQL and then
24:01:44we have data engineering. Then we have
24:01:46AI and machine learning. Right now we're
24:01:48in the SQL section. So, right in here,
24:01:50we're able to come in and we're able to
24:01:52write SQL just like any other platform.
24:01:54The neat thing about this though is that
24:01:56they have AI integrated into it. It
24:01:58helps a lot with writing some of the
24:01:59base queries that you're going to write.
24:02:01Now, we're going to have a whole lesson
24:02:02on Genie. And so, we're going to dive
24:02:04into that and see how you can generate
24:02:05code and do a lot of different things
24:02:07with Genie, which is their AI system
24:02:09within Data Bricks. And Genie just
24:02:10allows you to get insights from your
24:02:12data just using natural language. But
24:02:14you'll be able to write SQL right here
24:02:15in this editor as well as have multiple
24:02:17tabs and choose what workspace you're
24:02:20working with in and choose what
24:02:21workspace you are working in. Right now,
24:02:23we're just on the default, but if you
24:02:24had different workspaces, you'd be able
24:02:26to just click on those and select where
24:02:28you want your queries to be pointed at.
24:02:29Next, let's take a look at our queries.
24:02:31So, once we get in and we start
24:02:33developing all these queries, we want to
24:02:34save them. And organization is a big
24:02:36piece of that. I know that I used to
24:02:38have like 20 30 different kind of
24:02:40queries that I was saving on my file
24:02:42explorer back in my old job and then I
24:02:44started working in data bricks. I'm like
24:02:46hey I can just save those within this
24:02:48and then I don't have to email someone
24:02:49my query and then they pull it up and
24:02:51copy and paste like it's all here within
24:02:53data bricks which is really great. Next
24:02:55let's go look at dashboards. Now these
24:02:58are some sample dashboards and we're
24:02:59going to have a whole lesson on
24:03:00analyzing and visualizing data within
24:03:02data bicks. So we'll be able to build
24:03:04something like this. Let's click into
24:03:06one of these dashboards right here. It
24:03:08says my warehouse is starting. That's
24:03:10because it's connecting to the data to
24:03:11be able to visualize our data. So, let's
24:03:14get rid of this filter really quick. So,
24:03:16this is what our dashboard looks like.
24:03:18And you're able to create all these
24:03:19different visualizations. You're able to
24:03:20drill down into the data. And this is
24:03:23all connected to data that is already in
24:03:25data brick. So, it's all in one place.
24:03:27Next, let's go take a look at Genie. And
24:03:29they have a whole section just for this.
24:03:31And they also have an entire section
24:03:33that is just devoted to this warehouse.
24:03:36Now this is something that a lot of
24:03:37platforms are trying to build to but
24:03:39data bricks already has it which is they
24:03:41have all the data they have the ability
24:03:42to query and analyze and visualize data
24:03:44all in one place but being able to use
24:03:47AI with it to get a lot of those things
24:03:49done faster. And so you can ask it
24:03:50questions and it's going to prompt you
24:03:52because this is kind of a uh you know
24:03:54starter section and you can ask it
24:03:56questions and it's going to run these
24:03:58queries and it's going to get you that
24:03:59information really quickly. In one of
24:04:01our future lessons, we're going to be
24:04:02diving into just Genie, focused only on
24:04:04what it can do, how it helps you analyze
24:04:06your data faster, and how it can be
24:04:08really useful to you as a user. The next
24:04:10thing that we're going to take a look at
24:04:11is alerts. Now, an alert is basically
24:04:13like a trigger. When a condition is met,
24:04:15it's going to do something. And so, you
24:04:17can customize these alerts and set these
24:04:19conditions, and they'll email you or
24:04:20they'll message you when these
24:04:22conditions are true. Now, in this
24:04:24section up here, this is where we're
24:04:25going to be spending the majority of our
24:04:27time. There are other things like the
24:04:30job runs which is where you can create
24:04:32your jobs which is part of these uh jobs
24:04:34and pipelines and then of course the
24:04:36data ingestion as well which allows you
24:04:38to connect to your data. So if you have
24:04:40a data source that you want to point at
24:04:42maybe it's Google Analytics you can just
24:04:44click on this and connect to your data
24:04:46or if you just want to upload a file you
24:04:48can do that as well. Down here we have
24:04:50our AI and machine learning section. So
24:04:52you can click on the playground. This is
24:04:54another area where you can interact with
24:04:55the AI within data bricks and you can
24:04:57select some custom parameters as well as
24:05:00the prompts that you are using. We can
24:05:02also look at experiments. This is where
24:05:04you can build out AI agents and machine
24:05:06learning models and you can actually
24:05:07test them out. And so this is a
24:05:08fantastic place where you can actually
24:05:09learn how to use a lot of these
24:05:11foundational models and you can take the
24:05:12data and you can deploy it and you can
24:05:14work through these issues and really
24:05:15learn hands-on how to work with them.
24:05:17Naturally, they're going to have AI
24:05:19integrated into all this. So you can
24:05:20work with the data bricks assistant to
24:05:22get help with all of your coding. So as
24:05:24you're going along, if you get stuck,
24:05:25you can always get help with your code.
24:05:27As I said before though, we are going to
24:05:28be spending most of our time right up
24:05:30here. Of course, we'll need to ingest
24:05:32our data. But after that, we're going to
24:05:34be spending a lot of our time
24:05:35visualizing, see how we can work with
24:05:37our data and how we can use their
24:05:38integrated AI to be able to do our work
24:05:40faster. At the end of this entire
24:05:42series, we'll be building a full project
24:05:44using a lot of the things that we're
24:05:45going to be looking at in this series. I
24:05:47think building and getting hands-on is
24:05:48the best way to learn. So, I cannot wait
24:05:50to get started on that. There are a lot
24:05:52of ways to work with data in data bricks
24:05:54and we're going to cover a lot of them
24:05:55in this lesson. We're going to upload
24:05:57data like a CSV and a JSON file. Then,
24:05:59we're going to connect to an external
24:06:00data source and we're going to bring
24:06:01that data in. Then, we're going to see
24:06:03how we can actually use our data working
24:06:04in the SQL editor as well as using a
24:06:06notebook. The best way to follow along
24:06:08is to create a data bicks account. I
24:06:09will leave a link in the description.
24:06:11It's data bicks free edition, so it is
24:06:13completely free. All you have to do is
24:06:14create an account and sign in and you'll
24:06:16have access to everything. You don't
24:06:17have to put in a credit card at all. It
24:06:19is completely free which is amazing.
24:06:21With all that being said, let's jump on
24:06:22my screen and get started. We are
24:06:24starting out fresh. We haven't uploaded
24:06:26any data or ingested any data into data
24:06:28bicks yet. So, you are right where you
24:06:30need to be. Now, there's actually a lot
24:06:32of different ways you can bring data
24:06:34into databicks. Right now, we're in our
24:06:36workspace, but let's go over to our
24:06:39catalog and let's go into our workspace.
24:06:42Let's go into default. Right now, we
24:06:44have no data in our default workspace.
24:06:47Now, what we're going to do is let's
24:06:48click into this default and we're going
24:06:51to come right over here and we go to
24:06:52create. Now, we have a few different
24:06:54options. These two are the only ones
24:06:56that are really relevant to what we're
24:06:57doing right now. So, let's create a new
24:06:59volume here. And I'm just going to call
24:07:00this uh YouTube series. And then I'm
24:07:04going to go ahead and create this.
24:07:08Now, you'll see right over here under
24:07:10default, we now have this folder called
24:07:12YouTube series. If I click in this
24:07:14YouTube series, we can upload to this
24:07:18volume. So, we can do that right here.
24:07:20We don't have to actually go outside of
24:07:22it or go to any of these other options.
24:07:24We just click on upload to this volume.
24:07:26So, let's go ahead and click browse. And
24:07:29I have all these different files. Now, I
24:07:31created these. They're very simple
24:07:32files. We're not getting crazy in this
24:07:34lesson, but I have a customers CSV and a
24:07:37customer's JSON. So, one's a CSV file,
24:07:40one's a JSON file. I also have orders
24:07:42CSV, orders JSON, product CSV, and
24:07:45products JSON. We're not going to use
24:07:47all these, although I will have all
24:07:49these files in a GitHub. You can just
24:07:51find them in a link below. So, you can
24:07:52download these exact files if you want
24:07:54to work alongside me. But let's bring in
24:07:56the customers CSV. So, I'm going to
24:07:58click on this customer CSV. And you can
24:08:01see this is the destination path that we
24:08:03are using. Let's go ahead and upload
24:08:05this file. And very quickly, like in 1
24:08:09second it took, we now have our
24:08:11customers CSV CSV. Now, I just named it
24:08:14that. So, you know, if the CSV wasn't
24:08:16there, we'll still be able to see it.
24:08:18Let's click into this CSV really
24:08:20quickly. We have our customer ID, first
24:08:22name, last name, country, and signup
24:08:24date. And then you can see our data is
24:08:27all separated by a comma. Now, this data
24:08:30is just being stored as a file. This
24:08:32isn't actually being stored as a table.
24:08:34So, you can see it just sits in here as
24:08:36a CSV. So, if we go over to our files,
24:08:39it's going to sit there as an actual CSV
24:08:41file. Now, we can still query off of
24:08:44this data just as it is as a file. We
24:08:46don't actually have to have this in a
24:08:48table format. Let's come right over here
24:08:51and let's copy this path. And then we're
24:08:54going to go down to our SQL editor and
24:08:56let's come over here to a SQL query.
24:08:59Now, if we want to access this data, we
24:09:02can say select everything and then we'll
24:09:05say from and I'm going to put this path
24:09:08in here. I should be able to use back
24:09:11ticks just like this. And let's try
24:09:13running this. And actually, I need to
24:09:15put CSV dot right here. Now, let's try
24:09:20running this. And as you can see, we
24:09:22were able to read in our data without it
24:09:25actually being in a table. A CSV file is
24:09:28one of the easiest files to work with.
24:09:30It's just values that are separated by
24:09:31commas. And so when we do this, when we
24:09:34say CSV dot and then we provide the
24:09:36path, we are reading in this data as if
24:09:39it is in a table. Now, if you come from
24:09:41just a SQL background, this may seem
24:09:43very unintuitive to you. And that's
24:09:45okay. Let's go back to our catalog and
24:09:48we're going to come over here to
24:09:50workspace. We're going to go to default.
24:09:52And instead of going into our volume
24:09:56right here, we're going to come back to
24:09:58our default. And so now we're going to
24:10:01create a table. When we go to create a
24:10:03table, we're still using our serverless
24:10:05starter warehouse here, but now we have
24:10:07the ability to connect to a data source
24:10:10or to upload a specific file format.
24:10:12Let's go ahead and click browse. We just
24:10:15uploaded the customers CSV, but let's go
24:10:17down to the orders JSON because JSON is
24:10:20a totally different format. Let's go
24:10:22ahead and open this up. It's going to
24:10:24read in that JSON and make it tabular,
24:10:27which is fantastic because JSON by
24:10:29definition is not a structured format.
24:10:31And so being able to read in that data
24:10:33really easily and then put it into
24:10:35columns and rows is very helpful. So now
24:10:39we're going to do a create a new table.
24:10:41We can name this table anything we want.
24:10:43I'm just going to get rid of this JSON
24:10:45because once it reads it in, it's like a
24:10:47table anyways. So, we're going to keep
24:10:49it as orders here. Then, we're going to
24:10:51create this table. Now, you'll notice
24:10:53over here under our default, we have
24:10:55tables and we have volumes. So, they are
24:10:58separated out because they're two
24:11:00totally different things. One is going
24:11:02to be storing different files in kind of
24:11:04a folder format and then one is the
24:11:06tables underneath our default workspace.
24:11:10So let's come down here to orders and
24:11:12that's actually right over here. So now
24:11:14under our orders we can see customer ID,
24:11:16order date, order ID, product ID,
24:11:18quantity, and total amount. We can get a
24:11:21little bit of metadata on this file. And
24:11:23if we come up here, we can come here and
24:11:25just create a query. So let's click on
24:11:27create query. It's going to open up a
24:11:29new query in here. And now we're just
24:11:32selecting everything from our orders
24:11:34table. It's already hitting off of our
24:11:36default schema in our workspace. So, we
24:11:39don't have to start doing, you know,
24:11:40workspace
24:11:42dot uh default.
24:11:45We don't have to have all that. You can
24:11:47uh but I'm just going to hit control-z
24:11:50here. Let's go ahead and run this. And
24:11:52now we can see all of our data in this
24:11:54really pretty view. So, so far we've
24:11:56ingested a CSV file and we just did that
24:11:58as a CSV file. We were still able to
24:12:01read that data in which is really
24:12:03fantastic. But we're also able to just
24:12:05create tables and then read that data in
24:12:08like any other SQL database. But now
24:12:10let's come over here to data ingestion.
24:12:13There's a lot of different ways you can
24:12:14access data that is not just sitting in
24:12:17a file. Right off the bat we have our
24:12:18data bricks connectors. Things like
24:12:20Salesforce, Workday, Service Now, Google
24:12:22Analytics, Azure SQL Server. These are a
24:12:24lot of the things that I've used in my
24:12:26actual work. Almost all companies are
24:12:28going to have at least one of these. And
24:12:29so connecting to that data source,
24:12:31bringing it in is really common. We can
24:12:32also bring in data from a file like we
24:12:34did before or create a table from Amazon
24:12:36S3. Then we have our fiverr connectors
24:12:39right down at the bottom. I know
24:12:41personally I've worked at different
24:12:42companies. I've consulted with
24:12:43companies. They use Google Drive as like
24:12:45their store of information. That's where
24:12:46they keep everything. So let's go over
24:12:48here to Google Drive and I'm going to
24:12:51actually connect this. So I'm going to
24:12:52say I want to put it in my workspace.
24:12:55Let's go ahead and click next. Really
24:12:57quickly, I have a file over here,
24:12:59orderscv.csv.
24:13:02It's sitting in a Google Drive. So,
24:13:04that's what we're going to go and try to
24:13:05connect to. Let's go ahead and click
24:13:06next. This is my email that I'm using
24:13:09for this Google Drive. We're going to
24:13:11connect this to Fiverr. Since this is a
24:13:13new account for Fiverr, I need to create
24:13:15a password. So, I'm going to do that
24:13:17really quick. We're going to come right
24:13:18down here and we're going to go to
24:13:19share. And all we have to do is make
24:13:22sure that this is not restricted. So,
24:13:23we're going to say anyone with this
24:13:25link, which means fiverr as well. And
24:13:27then we're going to copy this link and
24:13:30put it in our folder URL. Let's go ahead
24:13:33and save and test this. And it looks
24:13:35like our connection test passed. Let's
24:13:36go ahead and click continue. And now
24:13:38what we're going to do is sync our data.
24:13:40We'll start the initial sync. And it
24:13:42should be very quick because I do not
24:13:45have much data in this folder. So, it
24:13:47looks like our connection was
24:13:48successful. Let's come over here to our
24:13:50schema. our one file that we have in our
24:13:53Google folder is synced up. So, we
24:13:55should be good to go. Now, let's come
24:13:57back to our data bricks. Let's come
24:13:59right over here and we're going to go
24:14:01into our catalog. We're going to go into
24:14:03our workspace. And now we have default
24:14:06and we also have Google Drive. Let's
24:14:08click on our Google Drive and we have
24:14:10this orders CSV.
24:14:13And you'll notice that we now have it in
24:14:15here as a table. Let's go ahead and
24:14:17create a query for this so we can look
24:14:19at our data. So now we have
24:14:21workspace.google
24:14:22drive. It's connecting to a different
24:14:24schema. So we have select everything
24:14:25from orders CSV. Let's run this.
24:14:29And now we have our data. It also adds
24:14:32this in which is five transync which is
24:14:35a really really useful column because if
24:14:37you're syncing this data consistently,
24:14:39you really want to know when that data
24:14:41gets put into this table. I promise you
24:14:43that's really helpful that they put that
24:14:44in there. And then we have all of our
24:14:46data that we have. And so that's how we
24:14:48can connect to outside data. In this
24:14:49case, we use Fiverr, but sometimes
24:14:51you'll just do a direct connection
24:14:52depending on your data source. Now, this
24:14:55SQL editor works like any other SQL
24:14:57editor. You're going to be able to make
24:14:58joins and aggregations and all sorts of
24:15:00different things, but there is a
24:15:02different way to interact with your
24:15:04data. Let's come right over here. Let's
24:15:06go to new. We can also go and use a
24:15:08notebook.
24:15:10We can add code. We can add text. You
24:15:13can also use an AI assistant to help you
24:15:15with these things. So, we have markdown
24:15:17file right here. And I can say uh this
24:15:19is my first text
24:15:24right here. And then I have my code down
24:15:27here. So I can start writing and typing
24:15:28my code. Now I can specify right here
24:15:31whether I want it to be SQL, Scala, R or
24:15:34Python. So for this notebook, you can
24:15:36use any of these. In this cell right
24:15:38here, I have Python. But let's add
24:15:41another one. And I can use SQL in the
24:15:43next one. And then in the next one, I
24:15:47could use R. And so you don't have to
24:15:49just use one. You can use multiple. Now,
24:15:51it depends on what you're doing, whether
24:15:53you want to use a SQL editor or you want
24:15:55to come in here and use a notebook. When
24:15:57I'm just kind of querying data, I'm just
24:15:59looking at it. I'm not doing a lot of
24:16:00transformations. I don't need a big
24:16:02programming language. I'm just quering
24:16:03the data. I'm going to be using a SQL,
24:16:06you know, editor right here most of the
24:16:08time. And I can always save these
24:16:09queries and I can pass them along. But
24:16:11if I'm really digging in and I need to
24:16:13be able to break things out and leave
24:16:15notes and I'm going to share this with
24:16:16my team, a notebook is kind of the way
24:16:18to go. So let's look at SQL really
24:16:19quick. We already have a query for this,
24:16:21albeit a very simple query. But let's
24:16:24come over here and let's run this SQL
24:16:28query right here. So let's go ahead and
24:16:29run this. And so now if we scroll down,
24:16:32we're going to be able to see our data
24:16:34just like we did in the SQL editor. But
24:16:37now let's go write the same thing but in
24:16:39Python. In order to do that, let's come
24:16:41up here. And this is already in Python.
24:16:43So, we'll just say spark.t.
24:16:46And we need to read in uh the
24:16:49appropriate table. And all that is is
24:16:51orders. And let's go ahead and run this.
24:16:54And it is reading it in as a data frame.
24:16:57Let's call this uh dataf frame. And
24:17:00let's come right down here and we'll say
24:17:01display dataf frame. And let's run this.
24:17:07And now we're going to get our data in a
24:17:09table. Just like before, of course, with
24:17:12this, it's going to save this data, but
24:17:13you can always come in here and you can
24:17:15rename and export it and you can put it
24:17:17into Git. You can share this with your
24:17:20friends because all your friends really
24:17:21care about your notebooks that you are
24:17:23writing. Or you can create a new
24:17:24notebook and start from scratch. But
24:17:27this is a totally different way to
24:17:29interact with your data in data bricks.
24:17:31This is where most of your work is going
24:17:32to be done. It's right here querying
24:17:34data, whether it's a SQL query or over
24:17:37here in a notebook writing a bunch of
24:17:39code. whether it's in Scola or Python or
24:17:41R or SQL itself. Now, I hope that was
24:17:44really helpful because in the next
24:17:45lesson, we're going to be analyzing data
24:17:47with SQL and then also building out
24:17:48visualizations in data bricks. Now, in
24:17:51the last lesson, we looked at the SQL
24:17:52editor as well as notebooks. And so,
24:17:54what we're going to be doing is we're
24:17:55going to be looking at some data. We're
24:17:56going to be analyzing that data and then
24:17:58we're going to be putting it into a
24:17:59dashboard and creating different
24:18:00visualizations. This is actually the
24:18:02dashboard that we're going to be
24:18:03creating in this lesson. And I know what
24:18:04you're thinking, Alex, you put the exact
24:18:06same visualization twice. That makes no
24:18:08sense. but it will make sense once we
24:18:10get to it in the lesson. I highly
24:18:11recommend following along. All you have
24:18:13to do is have a data bricks free edition
24:18:15account. I will leave a link in the
24:18:17description so you can make that account
24:18:18and follow along. With that being said,
24:18:20let's jump onto my screen and get
24:18:21started. All right, so we're getting
24:18:22started right here on the dashboards.
24:18:25All we're going to do is we're going to
24:18:26come up here and we're going to create
24:18:28our own dashboard. So, we get started
24:18:30with this blank slate and it has all
24:18:32these little arrows and I recommend you
24:18:34read through these really quickly, but
24:18:35this is how you add filters. This is how
24:18:37we get our data which is right up here
24:18:39and I'll show you that in just a second.
24:18:40And this is how we actually add our
24:18:42visualizations to what they're calling
24:18:44our canvas. Let's come right up here and
24:18:46let's go to our data. Now within our
24:18:50data set, we are able to write SQL
24:18:52queries. And I will say this is one of
24:18:54my personal favorite things about this
24:18:56is you can write the query and then you
24:18:58can use that query to then create a
24:19:00visualization. We're going to do that in
24:19:01this lesson because I like being able to
24:19:03visualize and kind of see my
24:19:04aggregations when I'm doing some type of
24:19:06group by in SQL. I like to see what the
24:19:08actual output is and you can really
24:19:10easily do that in here. You can also
24:19:12come down here and you can just come in
24:19:14and select some of your data. Now, we
24:19:16are actually going to be using a sample
24:19:18data set. Everyone should have this data
24:19:20set. It's right here called the bake
24:19:22house. And so, we can come in here and
24:19:24we can also just select one of our data
24:19:26sets. When we select this sales
24:19:28transaction, it's going to start our
24:19:30warehouse. So now it's connected to our
24:19:32data and our serverless warehouse is
24:19:34running. So now we have access to this
24:19:36data. If I come right here, I click on
24:19:38these three dots and I can click add to
24:19:41dashboard. So I'm going to click add to
24:19:42dashboard. It's going to read in this as
24:19:45a SQL query. So if we go back up here,
24:19:47we now have sales transactions as one of
24:19:49our data sets. So we can just come in
24:19:52here and write it ourselves or we can
24:19:54also come into the catalog and just
24:19:55select a data set and have it run it for
24:19:57us. Now before we start actually diving
24:19:59in and kind of analyzing and visualizing
24:20:01this data, there are other ways you can
24:20:04actually analyze data. And we looked at
24:20:06this in a previous lesson when working
24:20:07with data set. But as it pertains to
24:20:10connecting it to a dashboard, what we
24:20:12can do is let's come right up here and
24:20:14let's click on a notebook. Let's say we
24:20:16want to run that exact same query. And
24:20:18I'll just go back to the dashboard. I'm
24:20:21going to rename this really quick. I'm
24:20:23going to rename this as our transaction,
24:20:26if I can spell this right, dashboard.
24:20:28All right. So, if I go to my data, I
24:20:30could just copy this. And then I can
24:20:32come right over here to the SQL editor,
24:20:34and we'll actually have access to uh our
24:20:38notebook right over here. So, this is
24:20:40just our fresh notebook. We haven't used
24:20:42it. I'm just going to make this SQL just
24:20:44so it can you can see how easy this is.
24:20:46But, we're going to run this exact same
24:20:47thing. We're going to get our output
24:20:49right down here. Now, let's say this is
24:20:51the output that we want. It isn't, but
24:20:53this is the output we want. But if we
24:20:54come right over here to these three dots
24:20:56and we scroll down, I can click add to
24:20:58dashboard. So, I'm going to add this to
24:21:00my dashboard. I'm going to say add to
24:21:02existing dashboard. And when I click on
24:21:04this, I'm going to click on the
24:21:05transaction dashboard and I'm going to
24:21:07import this. This data is then going to
24:21:09be imported over as an untitled data
24:21:12set. We can rename this. Uh I'm just
24:21:14going to say that this is from notebook.
24:21:17So this is our data from the notebook.
24:21:19It is the exact same query as you can
24:21:21see exact same data. But you don't just
24:21:24have to create your data from right here
24:21:26in the create from SQL or in this
24:21:28dashboard tab. Now let's start actually
24:21:30working on our dashboard and kind of
24:21:32analyzing the data as we go. Let's just
24:21:34start by taking a look at our data. So
24:21:35we're working in the sales transactions.
24:21:37We have transaction ID, customer ID,
24:21:39franchise ID. If we scroll over to the
24:21:41right hand side, we have the date and
24:21:44time that this transaction went through,
24:21:46the product that was purchased, the
24:21:48quantity, the unit price, total price,
24:21:50the way that they paid, and their card
24:21:53number. Now, this is not real data, so
24:21:55don't try to steal any of this card
24:21:56number, but we are going to be using
24:21:58this data to create our visualizations.
24:22:01Now, typically when I'm analyzing and
24:22:02then visualizing data, I have an end
24:22:04goal in mind. I know what I'm going to
24:22:05be doing with the data, and so I know
24:22:07how I need to analyze it. if I need to
24:22:09be use a group by or a window function,
24:22:11if I need to clean the data or pivot the
24:22:13data. These are things that I'll know
24:22:14ahead of time. Now, in this lesson,
24:22:16we're going to keep it kind of simple,
24:22:17just learn how to build these things.
24:22:19But in the last lesson in this series,
24:22:21we're going to be building a full
24:22:22project. It's going to have really messy
24:22:24data that we need to dig into. But here,
24:22:26we're going to learn a lot of the
24:22:27fundamentals. Let's start with that
24:22:28first bar chart that we saw earlier on.
24:22:31So, we're going to come right here.
24:22:32We're going to go create from SQL. And
24:22:35let's call this one. Let's rename this
24:22:37before we actually write it. We're going
24:22:38to call this one product sales
24:22:41descending. Now, what this means is is
24:22:44we're going to take the product sales.
24:22:46So, right over here, we're going to take
24:22:48the product and then we're also going to
24:22:50look at this total price. So, we're
24:22:51going to calculate the sales. This is
24:22:53actually going to be quite easy. If you
24:22:55know SQL, this should be uh pretty
24:22:56straightforward because we're just going
24:22:57to be using a group by for this. So, I'm
24:22:59going to come right over here and I'm
24:23:01going to say I want to take the product.
24:23:04So, let's take the product. And I use
24:23:05tab for autocomplete here. So, I'm going
24:23:07to do uh comma, then I'm going to do the
24:23:09sum, and then I'm going to take the sum
24:23:11of total price. And again, I'm just
24:23:13going to hit tab. So, now that we have
24:23:15the product and the sum of total price,
24:23:18all we have to do is come right down
24:23:19here and say group by and we're going to
24:23:21group by the product. And we should also
24:23:24order by. So, we're going to order by
24:23:26and we'll do total price and we'll do
24:23:29that descending. And so, all we're
24:23:31doing, and let's run this. All we're
24:23:32doing is we're taking the product and
24:23:34we're grouping it. And then we're taking
24:23:35the sum of all of that total price. And
24:23:37I actually need to order by the sum of
24:23:40total price. I just took the column
24:23:43itself, but in this query, we're using
24:23:45the sum of total price. Let's try
24:23:47running that one more time. Listen, it
24:23:48happens to the best of us. So now we
24:23:51have the product right down here, and
24:23:53we're ordering it based off the sum of
24:23:55this total price. Now I'm going to leave
24:23:57this just like this. Although typically
24:24:00I would use an alias. I would say as and
24:24:03I would say total
24:24:05price or something like this, right? But
24:24:07I'm not going to do that because I want
24:24:09to show you in the dashboard how we can
24:24:10easily rename this. We don't have to use
24:24:13this column name. So now this data right
24:24:15here is ready for us to visualize. We
24:24:18can use this. So now let's come over to
24:24:21our untitled page and let's title this.
24:24:24We're going to say uh dashboard. So
24:24:27we're just going to name this dashboard.
24:24:28And we can get rid of these global
24:24:29filters for now, but we will need that
24:24:32in a little bit. Now, you have all these
24:24:34options down here at the bottom. We have
24:24:36the move, we have add visualization, add
24:24:38a text box, add a filter, undo, and
24:24:41redo. Now, let's focus on adding a
24:24:43visualization first, and we'll worry
24:24:45about the text box later because we'll
24:24:48give it kind of a header of transaction
24:24:50dashboard. You don't have to, but we
24:24:52will for this dashboard. Now, when you
24:24:54first create a visualization, we're
24:24:56going to have this widget on this right
24:24:58hand side. This is where you basically
24:25:00build out your visualizations. We're
24:25:02going to come in here and we're going to
24:25:04select the data that we want. We just
24:25:06built this product sales descending.
24:25:08Let's go ahead and click on this. We do
24:25:10want this bar chart, but let's come in
24:25:12here and let's look at all these
24:25:14different options. There is a lot of
24:25:16different types of visualizations that
24:25:18we can create, and a lot of these are
24:25:20the ones that you'll use 99% of the
24:25:22time. So, we're going to click on this
24:25:24bar chart. Now, we have to select the
24:25:26x-axis and the y-axis. In our chart, the
24:25:29axes are like this. We have we have two
24:25:31separate axes. And so, we need to select
24:25:33what data goes on each of those axes.
24:25:36So, let's select our x-axis. For this,
24:25:38we're going to do the sum of the total
24:25:40price. And then for the yaxis, we're
24:25:42going to select the product. Now, this
24:25:45looks perfectly fine, right? I'm going
24:25:47to uh expand this a little bit just for
24:25:49a second. This looks perfectly fine as
24:25:52is, but there's a lot of little things
24:25:54that we can do to make it a lot better.
24:25:57The first thing that I'm noticing is
24:25:59that I, you know, I'm having a little
24:26:01tough time reading this. I'm saying,
24:26:03okay, how much is this one exactly? It's
24:26:05a little over 11,000. And if I hover
24:26:07over it, it'll tell me the exact number.
24:26:09But that's not good for a customer or
24:26:11somebody to see. I'm going to add these
24:26:13labels. And now we can see it right
24:26:16away. So, we don't have to kind of
24:26:17guesstimate. we can see the exact
24:26:19number. The next thing I'm noticing is
24:26:21that it's kind of all over the place. It
24:26:23looks like it's in alphabetical order
24:26:25here, but if you remember in our data
24:26:27back here, we had it in descending order
24:26:30based off of the sum of total price. So,
24:26:32highest to lowest, that's descending.
24:26:34And I want that in our dashboard as
24:26:36well. All we have to do is we're going
24:26:38to click on this. We're going to go down
24:26:40to these three bars right above product.
24:26:42And then we're going to click buy Xaxis
24:26:45and then right here is the descending.
24:26:47So now we've ordered our data from
24:26:50highest to lowest. It's no longer
24:26:52alphabetical. Now there are some other
24:26:53things I want to highlight. We don't
24:26:54have to add these, but we absolutely
24:26:57can. First, let's add a title. So we're
24:26:59just going to call this total price
24:27:02by product. Keep it super simple. But we
24:27:05can also add a description. For example,
24:27:08let's say we wanted to add some context
24:27:09to this. Or maybe we wanted to highlight
24:27:11that this is our biggest seller. So, we
24:27:13can say Golden Gate Ginger is our
24:27:16highest
24:27:19selling product
24:27:21eight years in a row. Now, I'm just
24:27:24making this up. Uh, this isn't real.
24:27:25This is just as an example, but let's
24:27:27say we had some historical data and this
24:27:29is now showing, hey, this is still our
24:27:31best seller and I just wanted to add
24:27:32that as some context. You can definitely
24:27:34do that. You don't have to, but you can.
24:27:36One other thing is let's take a look at
24:27:38these colors because we don't have to
24:27:41just keep these colors. We can also use
24:27:43a custom color. So they can be you know
24:27:45whatever color you think is best for
24:27:47your dashboard. We can also click on
24:27:49this plus sign and we can create some
24:27:51other options. Now if we click on
24:27:52product and this is one that I don't
24:27:54necessarily recommend. So each product
24:27:56is going to get its own color on each
24:27:59row. It's a little bit redundant. Now
24:28:01let's go back over here. Let's get rid
24:28:03of our product and let's select our
24:28:06total price. Now we have this gradient
24:28:09scale from blue to white. This isn't my
24:28:11favorite. I actually prefer if we come
24:28:14right up here and let's get the green
24:28:17blue. And so this is a great way to
24:28:19analyze and visualize at the same time.
24:28:21Sometimes you already know what you're
24:28:22going to be building out and so you can
24:28:24work with the data while you're building
24:28:26your dashboard. And now I can very
24:28:27easily see I'm like Golden Gate Ginger
24:28:29Man that is our product. like we're
24:28:31killing it with this product. Uh but
24:28:32Richard Oasis, nobody likes that. It's
24:28:34doing okay, but it is our worst seller.
24:28:37Now, just remember, we use this product
24:28:40sales descending and let's go take a
24:28:42look at this. This is the only data
24:28:45available to us in that visualization.
24:28:48Let's come over here and let's look at
24:28:51our sales transactions. We have our
24:28:54sales transactions right here. This is
24:28:55all of our data. And so there are times
24:28:58where we don't even need to write a
24:29:01custom query for each visualization.
24:29:03Let's take a look at this. So let's use
24:29:05our sales transactions to create a
24:29:07visualization in our dashboard. Let's
24:29:09come down here. Let's create our new
24:29:11visualization. And the next one that
24:29:13we're going to take a look at is payment
24:29:15type. So let's change this data set to
24:29:17the sales transaction. So now we're
24:29:19using two different data sets. Just note
24:29:22that for future reference because that
24:29:24will come into play. But we're going to
24:29:26come down here and let's say we wanted
24:29:28to create a pie chart for the angle. And
24:29:32I'm just going to come down here. We're
24:29:34going to do payment method right here.
24:29:36It's going to give us a count. And we
24:29:37don't have to do count distinct. In
24:29:39fact, uh we probably shouldn't because
24:29:41it's just going to if we hover over it,
24:29:42you're going to see three because
24:29:43there's only three options. But if we do
24:29:45a count and hover over it now, it's
24:29:473.33,000.
24:29:49So now we're going to be able to see how
24:29:51many transactions are using each payment
24:29:53method. Now, for the color, this one is
24:29:56actually really important for a pie
24:29:58chart. Let's come in here and we want to
24:30:00do this based off of the payment method.
24:30:02We want to break it out. So, we have
24:30:04this payment method. We have Mastercard,
24:30:06AX, and Visa. Again, we can see the
24:30:09split based off of the color, but I
24:30:11really like labels. I think they're just
24:30:13super important. So, I'm going to add
24:30:15these labels in right here. Now, this is
24:30:16an example where it says count of
24:30:18payment method right here on the
24:30:20dashboard. I don't like that. I mean,
24:30:22that's just by default. It's going to
24:30:23take the uh name of it. But I'm going to
24:30:25come in here. I'm going to change this
24:30:27display name. So, all I'm going to do is
24:30:29I'm just going to call this payment
24:30:30method breakdown.
24:30:33And let's change that. And that looks a
24:30:36lot better. It just doesn't seem so I
24:30:38just tossed it in there. Seems like you
24:30:39intentionally named this dashboard. So,
24:30:42we were able to go in to this sales
24:30:44transactions that has a lot of different
24:30:46fields, a lot of different columns, and
24:30:48we're able to just kind of pick out
24:30:49which one we want to use. So far, our
24:30:51dashboard is looking great. We're just
24:30:53going to build one last visualization
24:30:55and then we're going to work on
24:30:56filtering. So, let's come right down
24:30:58here and let's create one last
24:31:00visualization. And this is going to be
24:31:02our dashboard. So, we're going to come
24:31:04in here. We're going to use sales
24:31:05transactions again, but this time we
24:31:08want to see our transactions over time,
24:31:10right? We have this date column and we
24:31:12want to use this. Let's use a line
24:31:14chart. And if we go back to our data, we
24:31:17have this date time. And this is really
24:31:19useful. We want to utilize this and see
24:31:21how many transactions are we having over
24:31:23time. Maybe there's a certain day of the
24:31:24week that people are just making a lot
24:31:26of transactions. This is really useful
24:31:27data for someone to know. So, let's go
24:31:30back to our dashboard and let's come
24:31:32down here to our x-axis. And for this,
24:31:34we want it to be our date column right
24:31:37down here. So, we'll choose our date
24:31:38time. Now, by default, it's going to
24:31:41select monthly, but you can come in here
24:31:43and you can transform this. It's going
24:31:45to take that datetime column. It's going
24:31:47to be able to automatically change it to
24:31:49basically anything you want. So, let's
24:31:51just choose daily for now, but we can
24:31:53change it later on. We just have to
24:31:55choose our y-axis. So, let's click on
24:31:57our plus sign in order to see our sales
24:31:59over time. Let's take a look at our
24:32:01quantity. It's going to do the sum of
24:32:03quantity here. And if we go back just to
24:32:06check on this data, the quantity is the
24:32:08actual amount that we sold. So, we sold
24:32:10eight, we sold 36, we sold 40. And so,
24:32:12it's aggregating that data for us, which
24:32:14is really nice. If we come right over
24:32:16here, we can hover over this and we can
24:32:18kind of see each day and the sum of
24:32:22quantity. Now, again, we need to
24:32:24transform this a little bit because it's
24:32:26just sum of quantity, date, time, um,
24:32:29doesn't display the best. We want to
24:32:31customize this. We're going to just say
24:32:32date to keep it simple. And then for the
24:32:34sum of quantity, we're going to say
24:32:36quantity sold and keep it just like
24:32:39this. for our title because I think this
24:32:42one may need one. We'll say quantity of
24:32:44sales over time. Right there we go. Now
24:32:49we have our dashboard built. It looks
24:32:52great. I'm super happy with this. One
24:32:54last thing we should add, and this is
24:32:56optional. You don't have to do this, but
24:32:58I'm going to add this title really
24:33:00quick. I'm going to call this our
24:33:02transaction. Let me spell that right.
24:33:05Transaction dashboard. And I'm going to
24:33:08format it just a little bit. I'm going
24:33:10to do it like this. I'm going to
24:33:13increase the size. Just like that. You
24:33:16can change it to be a different color,
24:33:18but it just kind of adds a little bit of
24:33:21finesse to it, right? We can put that up
24:33:24and make this smaller. Um, we can make
24:33:26this smaller if we want to go kind of
24:33:28that route. But it is up to you. The
24:33:31next thing that I want to show you
24:33:32though is adding a filter. Now, filters
24:33:36are quite important. customers, clients,
24:33:38managers, whoever is using this
24:33:40dashboard are going to want to filter in
24:33:42some way. They're going to want to say,
24:33:43"Oh, I want to filter on this product or
24:33:44I want to filter on this month or this
24:33:46year or whatever it is." And so adding a
24:33:49filter is really important. We can add a
24:33:51global filter right here. Now, let's
24:33:53come over here to our widget because we
24:33:55do have the ability to change the type
24:33:57of filter. We can select multiple
24:33:59values. It could just be a single value.
24:34:01Could be a date picker or range picker.
24:34:03So, a range of dates instead of just a
24:34:05single date. It could be a text entry
24:34:07where they're searching for something.
24:34:08Maybe it's a product. You can also do a
24:34:10range slider. Let's click on this range
24:34:13slider and let's come in here. We're
24:34:15going to go down to the sales
24:34:17transactions and let's go down to the
24:34:20quantity. Now, what this means is is
24:34:22we've created this global filter on
24:34:24quantity. Quantity is a numeric data
24:34:27type. So, we have this slider where we
24:34:29can basically say, hey, I only want to
24:34:30see where the quantity was over a
24:34:32certain amount. So we have a picker
24:34:34where we can select a range for the
24:34:36quantity. Maybe we only want to see the
24:34:38transactions that have a quantity
24:34:40greater than 30. For example, we would
24:34:42be able to do that in this dashboard. We
24:34:44can click in here and we can customize
24:34:46this. Maybe the minimum is zero and the
24:34:48maximum is let's say 100. I'm just going
24:34:51to set it for now. Let's come right over
24:34:53here and let's say we only want to look
24:34:55at it where it's greater than 30. So I'm
24:34:58going to click on that. You'll notice
24:35:00that this doesn't actually change at
24:35:02all, but this one changed and this one
24:35:05changed. And let's just highlight that a
24:35:08little more. Let's kind of slide this
24:35:09around.
24:35:12And you'll notice
24:35:15that only these ones are changing, but
24:35:17this one is not changing. The reason for
24:35:19that is the data set that we chose. So,
24:35:22let's come back here. Right up here, we
24:35:24have the product sales descending. And
24:35:27then if you go down here, we have our
24:35:29sales transactions and we have our sales
24:35:32transactions. But what are we actually
24:35:34filtering on? We are filtering on the
24:35:36quantity. And if you remember, let's go
24:35:38back to our product sales description.
24:35:41We don't have the quantity in here at
24:35:43all. And so this quantity is not
24:35:44connected to this data set. And so when
24:35:46we're applying this filter to this
24:35:48dashboard, it is not connected to this
24:35:51right here. Now, the way that we can fix
24:35:53that is we can replicate this exact
24:35:55dashboard. And this is what I was kind
24:35:57of talking about earlier, which is why
24:35:59we're going to create a second
24:36:00dashboard, but we're going to create the
24:36:03exact same thing, but we're going to use
24:36:06the other data set. So now we're in
24:36:08sales transactions. We're going to
24:36:09create our bar chart. For the x-axis,
24:36:11we're going to do the sum of total
24:36:13price. And for the yaxis, it's going to
24:36:16be the product. Let's find it right
24:36:18here.
24:36:19Let's scan this all the way over. Let's
24:36:22go down, add our labels, create our
24:36:25custom
24:36:27colors based off of the total price, and
24:36:30we'll change that coloring to be the
24:36:33green blue. And you can already see that
24:36:35this is filtered based off of our global
24:36:38filter right over here. So, let's get
24:36:39rid of this. Let's just make it the
24:36:41same. And we actually need to do one
24:36:43more thing. We need to
24:36:45go like this.
24:36:47So now we have the exact same
24:36:49visualization,
24:36:51but this one is going to be connected to
24:36:54our filter. So this is a really
24:36:55important just thing to understand when
24:36:57you're building out these dashboards is
24:36:59people really like their filters. They
24:37:01want to be able to do that and this is
24:37:03actually going to be a big thing that
24:37:04people request once you build your
24:37:06dashboard. You're going to build it out.
24:37:07It's going to be great and they're like,
24:37:08"Hey, I want to be able to filter on
24:37:10this. I want to be able to filter on
24:37:12this." And so you're going to have to
24:37:13build that out and you may have to go
24:37:15back and connect your data in certain
24:37:16ways to be able to accommodate certain
24:37:18filters. So that's just something to
24:37:20think about and something to know. But
24:37:22now this one is connected just like
24:37:24this. And so I would actually replace
24:37:27this one with this new visualization
24:37:30right down here because I want it to be
24:37:32connected to all of our other data for
24:37:34these types of global filters. Now there
24:37:36are of course some other things that we
24:37:38could do. We can come in here and change
24:37:39some of these uh axes. We could add some
24:37:41more context in here. This is our sample
24:37:43data set. This is as far as we're going
24:37:44to go in this lesson. But like I
24:37:46mentioned earlier in the last video in
24:37:48this series, we're going to be building
24:37:49out a full project using real raw data.
24:37:52So we're going to have to do some data
24:37:53cleaning. We're going to have to really
24:37:54dig in and analyze our data and
24:37:55visualize and create our dashboard. The
24:37:57first we're going to be looking at is
24:37:58Genie. And Genie is built for business
24:38:00users to be able to get insight from
24:38:02their data just using natural language.
24:38:04Next, we're going to be using their AI
24:38:05assistant to help us code. So we'll be
24:38:07using that in the SQL editor as well as
24:38:08the notebooks. It's going to help us
24:38:10generate code, but it's also going to
24:38:11help us diagnose and fix issues if we
24:38:13run into them. Next, we're going to use
24:38:14their AI assistant in the dashboards
24:38:16tab. So, it's going to help us create
24:38:17visualizations. Now, what's so amazing
24:38:19is you're going to be able to try all
24:38:20these things completely for free because
24:38:22in Data Bricks free edition, you can use
24:38:24all of their AI tools completely for
24:38:26free. Be sure to use the link in the
24:38:27description to create your account so
24:38:29you can follow along and practice and
24:38:31actually use all these AI tools. With
24:38:33that being said, let's jump on my screen
24:38:34and get started. When you first pull up
24:38:36data bricks, we're going to need to come
24:38:38right down here within the SQL section
24:38:40to Genie. Let's go ahead and click on
24:38:42this. And this about sums it up for
24:38:44Genie. You can ask questions about your
24:38:46data in natural language. And that's
24:38:48what it is. It's just a way to converse
24:38:50with your data. Ask questions about your
24:38:52data. Let's come over here. We're going
24:38:54to go to new. And we want to connect to
24:38:56a data source. We are going to use a
24:38:58sample data set. Let's actually go to
24:39:00all. It's within the samples right down
24:39:03here. this New York City uh data set.
24:39:06This is the data that we're going to be
24:39:07using for this video. Let's go ahead and
24:39:10create this.
24:39:11It's going to spin up our SQL warehouse
24:39:13because that is what we're going to be
24:39:15using to interact with this data. Now,
24:39:17while this is going and while it's uh
24:39:19just starting up and we can get rid of
24:39:21this, we now have this data that sits
24:39:23right over here. Now, I'm just going to
24:39:24give you kind of an introduction to kind
24:39:26of this interface, but we're right here
24:39:28in this configure tab. We can come over
24:39:31here and we can create custom
24:39:32instructions for whatever we want this
24:39:34genie to be. So if this genie is
24:39:36supposed to be for a specific team or a
24:39:38specific project or whatever it is, you
24:39:40can give it these guidelines because
24:39:41then you can share this with other
24:39:43people and they can interact with it. So
24:39:44if it's people on your team who are
24:39:46interacting with a specific data source,
24:39:48then everyone can interact with that AI
24:39:50in the same way. That's pretty cool. If
24:39:52we come right over here to settings, we
24:39:53can also name this. So, I'm just going
24:39:55to call this one uh datab bricks genie
24:40:00taxi. And I could add some description,
24:40:02but I'm not going to at the moment. Now,
24:40:04right over here, you'll notice in this
24:40:06space, we have a little bit of kind of
24:40:08prompting that is just, hey, here are
24:40:10some things that you can click on and we
24:40:11can give you that information. We can
24:40:13also add a sample question. So, if you
24:40:15want to add one for your team or whoever
24:40:17you're working with or just for
24:40:18yourself, you can add a question right
24:40:20here. Let's just go ahead and save this.
24:40:23and we're going to get out of here so
24:40:25that we have just this Genie interface
24:40:27right here. Now, as I mentioned before,
24:40:29Genie is really great for a business
24:40:31user, someone who's just going to be
24:40:32using natural language in order to ask
24:40:34questions about the data. If you are a
24:40:37more technical user, you're most likely
24:40:39going to be using the AI assistant that
24:40:40we're going to be looking at in the SQL
24:40:42editor and the notebooks. You can edit
24:40:43and dive into the code and get into the
24:40:45more programming side of data bricks.
24:40:47Let's just go ahead and ask it to
24:40:49explain this data set because we haven't
24:40:51taken a look at this data set at all. In
24:40:53fact, I don't even know what it looks
24:40:54like. It's just a New York City taxi
24:40:56data set. So, it says here's where it's
24:40:58located. It contains taxi trips,
24:41:00cleaning pickup, drop off, trip
24:41:01distance, fair amount, and
24:41:03pickoff/dropoff
24:41:05zip codes. This is the only table in
24:41:07here. Now, we can actually come over
24:41:08here to configure and we can click on
24:41:10this table and we can see a very quick
24:41:12kind of sample of this. But I'm going to
24:41:14ask it to show me a sample of the data,
24:41:18please. I like to be polite uh when
24:41:20working with AI. You just never know,
24:41:22right? I just want to make sure I'm I'm
24:41:24being uh respectful here. So, now it's
24:41:26going to give me a sample of our data.
24:41:28Now, you'll notice right over here we
24:41:30have this show code, and this is what
24:41:32it's going to do for basically anything.
24:41:34It's going to show us kind of what it's
24:41:36doing under the hood, which I really
24:41:38like. You can also edit in here. So, if
24:41:40we want to come in here and we want to
24:41:41say, okay, let's limit it to 20 because
24:41:43we only have a small sample down here.
24:41:45Now, we can run this again and it's
24:41:48going to keep all this information, but
24:41:49we're now rerunning and kind of changing
24:41:51the code as we go if you want to do
24:41:53that. So, now let's take a look just at
24:41:55our data really quickly. This is for
24:41:57taxi data. So, we have a pickup time, a
24:41:59drop off time, how long the distance
24:42:01was, how much it cost, the pickup zip,
24:42:04and the drop off zip. So, fairly simple
24:42:06data. Now, there's a lot of things I
24:42:08could ask about this data set. I'm going
24:42:09to keep it pretty simple. As we go
24:42:11further along in this video, we're going
24:42:13to get a little bit more technical, a
24:42:15little bit more challenging to the AI to
24:42:17really see what it can do. So, I'm just
24:42:19going to say, what zip code
24:42:23are people
24:42:25being picked up at
24:42:28the most? So, I just want to know where
24:42:31are most people being picked up? Maybe
24:42:32there's a specific zip code that 99% of
24:42:36these users are getting picked up at and
24:42:38that would be really useful information.
24:42:40So, let's see what Genie comes up with.
24:42:41So, it says it right here. The most
24:42:43common pickup zip is 10,01 with 1,227
24:42:47pickups recorded. Now, you can see that
24:42:49right down here, but again, you have to
24:42:51remember that this is mostly for
24:42:52business users. And so, I think it is
24:42:54good that they include this information
24:42:55up here just as a narrative that you can
24:42:57read. Let's go and look at the code. So,
24:42:59it looks like uh they're looking at the
24:43:01pickup zip, which is exactly what we
24:43:02would want to do. They're doing just a
24:43:04count of everything, and then they're
24:43:06grouping on that pickup zip as well, but
24:43:08then ordering it, descending on the
24:43:10pickup count, and limiting by one. This
24:43:12is how I would have written it as well.
24:43:13This isn't anything crazy complex. But I
24:43:16think it did a really good job answering
24:43:17this question. Now, as I've been testing
24:43:20and working within Genie, it is pretty
24:43:22good with answering a lot of the
24:43:23questions about the data. Let's get a
24:43:25little bit more difficult with this one.
24:43:27So, let's come down here and let's say,
24:43:30what time of day do most people get a
24:43:34ride? Now, this is kind of how I would
24:43:36imagine someone who's not very technical
24:43:38might ask this question, right? It's not
24:43:40very specific, and I'm curious as to how
24:43:42it's going to handle it. So, let's see
24:43:44what code it writes and the output that
24:43:46we get. So, it looks like it used the
24:43:48pickup hour. It says the highest number
24:43:49of rides occur at 1,800, which is 6 p.m.
24:43:53with,455
24:43:55rides during that hour. Let's look at
24:43:56the code really quick. So it looks like
24:43:58it's using this hour right here within
24:44:01our datetime. It's running a count. It's
24:44:04kind of filtering a little bit and
24:44:05grouping on this and doing kind of the
24:44:07same thing it did in our previous query.
24:44:09Now Genie does have the ability to
24:44:11create visualizations as well. Let's see
24:44:13if it can visualize this. I'm going to
24:44:15say uh can you visualize this?
24:44:20All right, it did exactly what I asked
24:44:23it to do. I asked it to visualize it and
24:44:25it literally is visualizing uh the
24:44:27pickup hour and the count amount. This
24:44:30is not exactly what I had in mind. I
24:44:31actually wanted to see all the pickup
24:44:33times over time. So, I'm going to ask it
24:44:35if it can show me all the pickup hours
24:44:37and the counts and visualize it. So, I
24:44:39just went ahead and wrote that. I said,
24:44:41"Write a SQL query to show me all the
24:44:42pickup times and the counts and then
24:44:45visualize it." Let's see if it's able to
24:44:47do this one because the previous
24:44:49visualization was not exactly what I was
24:44:51hoping for. Although it gave me exactly
24:44:53what I asked for. So it looks like it
24:44:55did this properly. And let's just look
24:44:57at this. They really are just doing the
24:45:00same query, but now they're not, you
24:45:01know, filtering on uh it descending and
24:45:04then limiting it by one. Now we have all
24:45:07of them. And whoops, let's go right down
24:45:09here. And now we have a visualization of
24:45:12this data. As you can see right here,
24:45:15this is our highest one. This is our
24:45:161,800 or our 6 p.m. And so now we have
24:45:19this visualization. Now, the thing about
24:45:21Genie is it's kind of its own standalone
24:45:24thing. So, if we want, we can come up
24:45:27here and we can copy this. So, we can
24:45:29copy all of this and bring this over to
24:45:31the SQL editor. But again, that's not
24:45:33really the use case here. The use case
24:45:34is that a business user can come in here
24:45:36and ask questions about the data and get
24:45:38their questions answered without having
24:45:39to go into the SQL editor or a notebook
24:45:41or build a dashboard. So, with that
24:45:43being said, let's come right over here
24:45:45to our SQL editor and let's go right
24:45:48over here and let's take a look within a
24:45:51new query window. Let's take a look at
24:45:54the AI assistant that they have. Now,
24:45:56they have one right here, but you can
24:45:58also see a button right up here. They
24:46:00are both the assistant, but they are
24:46:02used in a different way. I'm going to
24:46:04demonstrate that in just a little bit.
24:46:06We're going to generate some code, but
24:46:07then we can ask larger context questions
24:46:10about the code and the data and
24:46:12revisions and insight into our data with
24:46:15our assistant up here, but we can't
24:46:16really do that with this one cuz this is
24:46:18purely for generating code. Let's go
24:46:20ahead and click on generate code. Now,
24:46:22we don't have this connected to our
24:46:23data. So, I'm just going to say, show me
24:46:26our New York City taxi data. And I'm
24:46:29going to ask it to generate this. Now,
24:46:32this is very generic. I don't even know
24:46:34if it's connected to this data. Let's
24:46:35just see what it does. It does not know
24:46:38exactly what data I'm talking about. And
24:46:40that's because we're in this
24:46:40workspace.default.
24:46:42I just wanted to see if it would be able
24:46:43to pick up on it, but it's not doing
24:46:45that. Let's click over here. Let's go to
24:46:48our samples. And for our schema, let's
24:46:51come down here to New York City. I'm
24:46:53going to accept this really quick
24:46:54because I want to demonstrate something.
24:46:56Let's go ahead and try to run this. It's
24:46:58not going to work. And that's okay. But
24:47:00what we can do is we can come right down
24:47:03here and we can say let's diagnose this
24:47:05error. So let's click on the diagnose
24:47:08and the assistant in the top right is
24:47:10now going to be activated. It's going to
24:47:11do this forward/fix. And so now it's
24:47:14going to say okay we need to use the
24:47:16proper schema and even if we hadn't
24:47:18switched over honestly it would have
24:47:20picked this up. This assistant up here
24:47:22is slightly more contextaware I've
24:47:25noticed than the actual code generation
24:47:27right down here. So now we can try
24:47:29running this. And we can run this right
24:47:30over here. And we're going to be able to
24:47:32see our data at the bottom. And I'm
24:47:34going to click this button in. It's
24:47:36going to say replace active query
24:47:37content. And so now I'm going to paste
24:47:39all this code right into this window.
24:47:42Let's get rid of this for now. And what
24:47:44we're going to do is I'm going to edit
24:47:47this code. So as you can see, it
24:47:48highlights all of our code. And I'm
24:47:50going to do two things. One, I'm just
24:47:52going to do this forward slash. And
24:47:54we're going to get some options here.
24:47:56We're going to say forward
24:47:57sldoc/explain.
24:47:58We have all these options and it tells
24:48:00you what they do on this right hand
24:48:02side. So we can explain the code,
24:48:04improve the formatting, optimize the
24:48:06code, replace parameters, fix errors in
24:48:08our code. There's a lot of different
24:48:10options and they're just kind of
24:48:11defaults. And these are things that you
24:48:12can use really easily. We just fixed our
24:48:14code, so we're not going to do that
24:48:16right now. But what I'm going to ask it
24:48:17to do is do something totally different.
24:48:19Let's actually run our code really quick
24:48:20because I want you to be able to see the
24:48:22data in here. But we have two really
24:48:25useful columns, a pickup time and a drop
24:48:27off time. And I'm going to ask it to
24:48:29calculate the difference between this
24:48:31and give us the average of those times.
24:48:33What's the average length of ride? It
24:48:35says the trip distance, but it doesn't
24:48:37tell us how long the trip took. So, I'm
24:48:39going to go into the code. I'm going to
24:48:41say I want the average time it took
24:48:46between
24:48:49pickup and drop off. Now, this is a
24:48:52little bit more difficult. This is more
24:48:54difficult than anything we did in Genie.
24:48:56Let's see if it's able to write this
24:48:57code correctly. Now, we can see what
24:48:59it's getting rid of with the red lines
24:49:01and with the green lines. We can see
24:49:02what it's actually keeping. We can
24:49:04accept this either by clicking accept or
24:49:06clicking tab. I'm going to go ahead and
24:49:08click tab. And basically, what it's
24:49:09doing is it's taking the drop off time
24:49:11and it's taking the pickup time. It's
24:49:13subtracting it and then it's taking the
24:49:15average. And so, we're just going to go
24:49:17ahead and run this.
24:49:19And in seconds, that works perfect. But
24:49:22I don't want it in seconds. I want it in
24:49:23minutes. So I'm going to edit this. I'm
24:49:25gonna say I want the average to be in
24:49:29minutes,
24:49:30not seconds.
24:49:33It's going to rewrite this and it's
24:49:34going to divide it by 60. And that's
24:49:36really uh all it needs. Let's go ahead
24:49:38and tap this. Let's run our code. And
24:49:41now we have average trip duration in
24:49:43minutes. It's around 15 minutes. Now,
24:49:45this is really great, but somebody who
24:49:47doesn't know what this is might be very
24:49:49confused. So, I'm going to come up here
24:49:52and I'm going to use one of these
24:49:53forward slashes and I'm going to say add
24:49:55comments to code. So, I'm just going to
24:49:57hit enter. It's going to start
24:49:59commenting on this code and I can click
24:50:02accept. It's just going to say
24:50:03calculates the average trip duration in
24:50:04minutes by subtracting pickup from drop
24:50:07off timestamps. Very simple, very
24:50:09straightforward, but it is really
24:50:10helpful for somebody who's going to be
24:50:12coming behind you and reading this code.
24:50:13Now, as I mentioned before, this is for
24:50:15coding, but if you have other questions
24:50:17about the data, you can also come over
24:50:20here. So, I'm just going to say, tell me
24:50:22a bit about this data. Now, this is just
24:50:26kind of a really simple overview of the
24:50:28columns and kind of what's in it. It's
24:50:30pretty straightforward, but now let's
24:50:32ask it. We're going to say, give me a
24:50:35simple data dictionary
24:50:37for each column. Now, I could have said
24:50:40for the whole data set. I don't know why
24:50:42I said column, but let's see what it
24:50:44gives us.
24:50:49And just like that, it is going to give
24:50:51us a data dictionary, basically a
24:50:52description for each column. And this is
24:50:55really useful. It reads in the data and
24:50:57just gives you a little bit of
24:50:59information about what is in that data,
24:51:01as well as the data type and the column
24:51:02name. So, this is really useful. Now, as
24:51:04I mentioned before, this isn't only in
24:51:06the SQL editor. We can also do this
24:51:08within a notebook. So let's come over
24:51:10here and take a look at the assistant
24:51:13within a notebook. You can see we have
24:51:15this generate. It's going to look very
24:51:17similar as it did before in the SQL
24:51:18editor. Now it can do a lot of similar
24:51:21things. It's going to generate code but
24:51:23it can do it in different languages now.
24:51:25So we can do markdown, Python, SQL,
24:51:27Scola and R. By default though it is in
24:51:29Python. So let's come over here. Let's
24:51:32go ahead and ask it to
24:51:35give us the sample New York City taxi
24:51:40data and just see what it comes up with.
24:51:43It looks like this one is going to be
24:51:44perfect. Let's go ahead and run this.
24:51:46And it's going to give us exactly what
24:51:48we want. We have our data right here.
24:51:50Now, one of the things that I want to
24:51:52highlight is that what you can do cuz
24:51:54when you're working within these
24:51:55notebooks, you can use whatever language
24:51:56you'd like. You can convert it to other
24:51:58languages. So let's come up here to our
24:52:01code. I'm going to say, can you convert
24:52:06this to SQL? It's going to take this,
24:52:09which was a dataf frame, a spark.sql
24:52:12dataf frame, and it displayed it. It's
24:52:13now converting it to SQL right here. And
24:52:16then we're reading in just the SQL.
24:52:18Let's go ahead and tab this, and we'll
24:52:20run it. And we should get the exact same
24:52:22output. So that's one of the things that
24:52:25you can do in notebooks that of course
24:52:26you're not going to do in a SQL editor
24:52:27because you're just using SQL. So that's
24:52:29how the assistant works within a SQL
24:52:31editor as well as a notebook. But let's
24:52:33head over here to our dashboards. Let's
24:52:36go and create a new dashboard. And what
24:52:38we're going to do is we're going to pull
24:52:40in that data that we were using before
24:52:42in our samples. So let's go to our
24:52:44catalog. Let's go to our samples New
24:52:46York City. And we'll click on this. And
24:52:49we're going to say add to dashboard.
24:52:52So now we have this data right up here.
24:52:54This is our trips data. And what we're
24:52:56going to do is come right over here and
24:52:58we're going to add a visualization. The
24:53:01first thing that's going to come up
24:53:02right here is the assistant. It said ask
24:53:04the assistant to create a chart or to
24:53:06create anything that we want. And so
24:53:08let's ask it to create something really
24:53:10simple. I wanted to create a card with
24:53:14the average fair amount. And let's go
24:53:18ahead and run this. So, I just wanted to
24:53:20show us the average price for one of
24:53:22their fairs for the taxi. And there we
24:53:24go. So, super easy, super simple. Now,
24:53:27let's ask it a little bit more difficult
24:53:29question. Create a line chart
24:53:33with the count of pickup times.
24:53:38And let's go ahead and run this. Now, it
24:53:41did do this. And what's nice is we have
24:53:43it right over here. It did this based
24:53:44off of the month. I'm going to change
24:53:46this manually to the day so that we can
24:53:49see a little bit easier what days have a
24:53:53lot or have a little. And we're just
24:53:55doing a simple count on the pickup
24:53:57dates. That's it. Now, with both of
24:53:58these, I didn't click the accept button,
24:54:00but once you do, once you click accept,
24:54:02and you can of course uh change it, it
24:54:05is going to stay there like you just
24:54:06created it as a normal visualization.
24:54:09Now, let's say I want to make a change
24:54:10to this. Let's come up here and say
24:54:11change the daily
24:54:15and I need to spell this right. Daily to
24:54:17hourly. Let's go ahead and run this. And
24:54:20so there going to be a lot of people
24:54:21they don't know how to come in here or
24:54:23don't want to come in here. They can do
24:54:25this by just writing change the daily to
24:54:27hourly. I'm going to accept this. Now
24:54:29this uh looks wild but let's uh look at
24:54:33it. This is looking correct to me. Take
24:54:36away that global filter. And now we're
24:54:39looking at it hourly. And so we have a
24:54:41very very accurate and specific view of
24:54:44hourby hour the pickup times. And so
24:54:46this is how we can create different
24:54:47visualizations by just asking it. We can
24:54:50of course come in here because it's
24:54:52still using the exact things that we
24:54:53would use to create these. And so if we
24:54:56want to add labels, I'm going to say add
24:54:58labels. I wouldn't on this data set.
24:55:01That's going to look terrible. Um but
24:55:02it's going to turn on the labels for
24:55:04you. And this uh looks horrible. But it
24:55:08did exactly what I requested. Uh it's
24:55:10too granular to add labels. We would
24:55:12have to change this to uh probably
24:55:14weekly maybe um and add the labels for
24:55:17it to even make somewhat sense. There we
24:55:19go. So this looks a lot better. But we
24:55:20can do a lot of the things that we can
24:55:22manually do just using this. Now if you
24:55:25already know how to create these
24:55:27visualizations just by memory, you won't
24:55:29need to use this all the time. But let's
24:55:30say you just can't figure out how to do
24:55:32it yourself. You can always ask. It is
24:55:34there to assist if you need it. You
24:55:35don't have to use it. But it can be
24:55:38useful if you're like, I just can't
24:55:39figure out how to create this. Let me
24:55:41just ask the assistant and see how they
24:55:42would create it. As you can see, there
24:55:44are a ton of ways that you can use AI
24:55:46and data bricks. You can use the genie.
24:55:47You can use in the SQL editor with the
24:55:49assistant. We can also use this
24:55:50assistant up here, which gives a little
24:55:52larger context and we can use it in the
24:55:54notebooks. So, it's integrated into a
24:55:56lot of different places that you would
24:55:58use as an analyst or a scientist or a
24:56:00data engineer. Now, within this entire
24:56:02data brick series, we've learned a lot
24:56:03of things and we're going to apply all
24:56:05of that into our final project. If
24:56:07you've never done one of my projects on
24:56:09this channel before, they can be a
24:56:10little bit long because I leave
24:56:12everything in. I don't cut out a lot of
24:56:13stuff. So, if I make mistakes, you're
24:56:15going to see it so you can learn from it
24:56:17as well. I will leave a link in the
24:56:18description so you can go to the data
24:56:19bricks free edition and create an
24:56:21account so you can follow along with
24:56:22this project. I will also have the data
24:56:24set that we're going to be using in a
24:56:26GitHub. So all you have to do is go and
24:56:27download that data set and you can
24:56:29import it and use it just like I'm
24:56:30doing. We are going to be working with
24:56:32real raw data. So we are going to
24:56:33encounter some issues and some things
24:56:35that we'll have to clean up along the
24:56:36way. But we're going to learn a lot and
24:56:37this is what we're going to end up with
24:56:38at the final project. We're creating
24:56:40this United States emissions breakdown
24:56:42dashboard. It's going to be an awesome
24:56:44project. So I cannot wait to do this
24:56:45with you. Let's go ahead and jump onto
24:56:47my screen and get started. Now, before
24:56:48we jump into data bricks and actually
24:56:50start building out our project, I want
24:56:52to take a look at the data because
24:56:54there's a lot of data here. And when I
24:56:56say a lot, I mean, you know, there's
24:56:58only 3,000 rows, but there's a ton of
24:57:01different fields. So, if we start
24:57:02scrolling over, we're going to see
24:57:04there's so many different things that we
24:57:06can work on. And I don't want to focus
24:57:08on everything because it's going to be
24:57:10impossible to get everything in there. I
24:57:12have highlighted with yellow the ones
24:57:14that we're going to be working with.
24:57:15This is the emissions uh for megat tons
24:57:17of CO2. We have our population,
24:57:19latitude, longitude, the county name,
24:57:22county, state name, and state
24:57:23abbreviation. We're going to be focusing
24:57:25on emissions, although there's so many
24:57:27other things we could look at in this
24:57:29data set. But we just don't have that
24:57:31much time and we don't have, you know,
24:57:33days and weeks to comb through this and
24:57:36build everything out for, you know, all
24:57:37the different scenarios and dashboards
24:57:39that we could build out. So, this is the
24:57:41data set that we're going to be working
24:57:42with. Again, you can get that in the
24:57:44GitHub. Let's get out of this. And what
24:57:46we're going to do is I'm right here. We
24:57:48are in our databicks free edition
24:57:50account. I'm going to go ahead and I'm
24:57:52going to create a new catalog. So, I'm
24:57:54going to call this one I'm going to say
24:57:55this is our emissions uh catalog. I'm
24:57:58going to go ahead and create this. And
24:58:00now we're going to have uh let's get out
24:58:02of this. We're going to have our
24:58:03emissions. And I'm going to go under my
24:58:05default and I'm going to come right over
24:58:08here and I'm going to create a table.
24:58:10So, we're going to drop this entire
24:58:12thing into this. Let's go ahead and
24:58:13browse. And we're going to come up here
24:58:15to emissions data 2023. Now, this data
24:58:18is large. May take just a second uh to
24:58:21bring it all in. Um but, you know, it's
24:58:23not like 10 gigabytes or anything. So,
24:58:25we should be able to create this quite
24:58:27well. I'm going to rename this to just
24:58:30emissions data because I think it looks
24:58:32cleaner. You can keep the 2023 if you
24:58:35would like, but I'm just going to take
24:58:37uh emissions data.
24:58:41All right. So, our data is in here. This
24:58:43all looks good. We are not going to
24:58:45transform any of this, although we may
24:58:48need to transform some of it, right? But
24:58:50it's going to autodetect all of our data
24:58:53types. So, you know, if we need to
24:58:55change something, we're going to do that
24:58:56after the fact. We're going to take it
24:58:57as it is with the raw data. Now, all we
24:59:00have to do is come down here to create
24:59:02table, and it's going to create our
24:59:05emissions data table for us. Now, it's
24:59:08going to give us some of this AI
24:59:09suggested description. This looks good
24:59:11to me. I'm going to go ahead and just
24:59:12accept this just to have it in this data
24:59:15set, which was uh nice to have. Now,
24:59:17what I'm going to do is just give you
24:59:19the scenario, right? You were hired by
24:59:21the EPA to create a dashboard and they
24:59:23want to focus specifically on emissions,
24:59:25but they want a breakdown of things like
24:59:27where it's actually coming from in the
24:59:29United States, where the most emissions
24:59:31are coming from. They want to see what
24:59:32states and counties are emitting the
24:59:34most emissions, as well as just in
24:59:37general as a population, how much
24:59:39emissions do we have per person per
24:59:41area. So, these are things that we're
24:59:42going to have to dive into. We're going
24:59:43to write some SQL queries in order to do
24:59:45this. Let's come over here and let's go
24:59:48to our SQL editor.
24:59:51Now, this is from a previous lesson. I'm
24:59:53going to go ahead and get rid of all
24:59:54these previous queries. And we're going
24:59:56to get started just with a fresh new
24:59:58query. And we will do this by selecting
25:00:01emissions and going to default. And it
25:00:05should be our emissions
25:00:08data. Let's go ahead and run this. Now,
25:00:12as our dashboards, that's going to be
25:00:13kind of our final product that we're
25:00:14going to hand off. I want to start with
25:00:16our location data because I personally
25:00:18am super interested in this. We of
25:00:20course have the state name and the
25:00:22county name. That's not going to be as
25:00:24specific as something like latitude and
25:00:26longitude. And so I want to use this
25:00:29data along with our emissions data. And
25:00:31let's come right over here and find
25:00:33that. It's right here. So it's GHD
25:00:36emissions M tons CO2E. It's a long name,
25:00:41but that is the column that we want. So,
25:00:44let's come right here and instead of
25:00:46getting all of or taking all the data,
25:00:48I'm going to do uh latitude. I'm going
25:00:51to do tab to autocomplete, we're going
25:00:53to do longitude and let's see if it can
25:00:56find the CO2.
25:00:59There it is. So, I'm going to go ahead
25:01:00and hit tab. And I'm just going to call
25:01:03this as emissions. I feel like that's
25:01:06just going to be easier. Uh let's go
25:01:08ahead and run this. And there we go.
25:01:11Now, in terms of actually analyzing this
25:01:14data, latitude and longitude tends to be
25:01:16a little bit more difficult to work
25:01:18within something like a county or a
25:01:19state because it is so specific. So, you
25:01:21can't really aggregate on it. It's more
25:01:23of a visual thing. So, let's actually
25:01:25just take this over. I'm going to copy
25:01:28this. I'm just going to bring this over
25:01:29to my dashboard. And we're going to
25:01:31create our new dashboard. Uh, let's
25:01:34rename it really quickly before we get
25:01:35into data. I'm going to call this our
25:01:37emissions
25:01:39dashboard. As we build things out, we
25:01:41will then come and add to it. We don't
25:01:43have to do it that way. We could get all
25:01:45the data up front and then uh go from
25:01:48there. But I'm going to create from SQL.
25:01:51And I'm going to run this. And we
25:01:53actually need to specify where this is
25:01:55coming from. So let me get rid of this.
25:01:58I'm going to call this uh emissions
25:02:02default emissions data. So I was just
25:02:04tabbing there. It got our catalog, then
25:02:06our schema, and then our table. I just
25:02:08tabbed along the way. It makes it, you
25:02:10know, kind of easy to work with. Let's
25:02:12go ahead and run this. And now we have
25:02:15our data right down here. Now, like I
25:02:17said, this isn't the easiest data to
25:02:20work with if you don't really know
25:02:22latitude and longitude data, but it's
25:02:24really easy to visualize. So, let's come
25:02:26up here and we are going to add I'm
25:02:29going to get rid of our filters, but I'm
25:02:31going to add a visualization.
25:02:33And the data that we'll connect with is
25:02:36apparently called our untitled data set.
25:02:38Let me change this. So I'm going to
25:02:40rename this as our location data. We
25:02:44will use some other location data, but
25:02:46it's like state and county. And so we
25:02:48should be good. Let's use our location
25:02:50data. And what we need is actually a
25:02:53map. And we didn't do a map when we were
25:02:55creating and building dashboards in a
25:02:57previous lesson. So this is a new one
25:02:58for us. But let's come right down here.
25:03:00We're going to go to a point map. Now
25:03:02you can see we already have the
25:03:03coordinates available for us. Longitude
25:03:05and latitude. So this should be really
25:03:07easy. We're just going to do uh for the
25:03:09longitude we'll do longitude. And for
25:03:12the latitude, believe it or not, this is
25:03:13going to sound insane. We're going to do
25:03:15latitude. Now, this is a little tricky
25:03:18to work with sometimes. It's kind of a
25:03:20double click. So I'm going to double
25:03:22click in. I'm going to double click in.
25:03:23And you can kind of scroll back. There
25:03:26are some things that we need to
25:03:27customize because this is just super
25:03:29blurry. I'm going to go to the size and
25:03:33I'm going to decrease the size a little
25:03:35bit. I think that makes it a lot better.
25:03:38We can always, you know, doubleclick and
25:03:41zoom out just a little just to have
25:03:44enough. I will say if we zoom out more,
25:03:46you'll see some information right here.
25:03:48I think this is Hawaii. Don't cancel me
25:03:50if I'm wrong about that. Geography is
25:03:52not my uh best area. Then we have Alaska
25:03:55up here. So maybe we want to include all
25:03:58of that. Um, that would be fine if we
25:04:01do, but I'm going to come right in here.
25:04:03I think this is what I want to keep,
25:04:05which is mainland United States. So,
25:04:07we're going to keep this. This to me is
25:04:10really useful and it makes a lot of
25:04:12sense. This is some of the most densely
25:04:14populated areas over here in the uh kind
25:04:17of the Mid East and the East. And then
25:04:20right over here, based on emissions,
25:04:22right, emissions is much lower. Even in
25:04:24California, there's a lot of people in
25:04:25California and on the west coast, but
25:04:28not as many emissions. And so most of
25:04:30our emissions, just looking at our, you
25:04:32know, our visualization are over here on
25:04:34the right hand side. Kind of
25:04:36centralized. What is this like Ohio? Uh,
25:04:38Chicago's in Illinois. It's like
25:04:40Illinois, Ohio area. I'm not a geography
25:04:42expert. Don't hate. I just I don't know
25:04:44geography that well. So, this is really
25:04:47interesting. I would not have guessed
25:04:49this, but this is official EPA data. So,
25:04:51this is, you know, pretty useful. Now,
25:04:53real quick, I'm just going to come in
25:04:55here and I'm going to create our title.
25:04:57We're going to call this the United
25:04:59States
25:05:00uh emissions
25:05:02breakdown. And I need to spell this
25:05:05properly. Let's put this in the middle.
25:05:08Let's uh make it bold. And we'll make it
25:05:12larger. Not going to get crazy with it.
25:05:15Maybe we'll do one more. Let me see.
25:05:17That's too big. All right. Let's go back
25:05:18to 24.
25:05:20Underneath it though, I'm going to add
25:05:21like some notes. Uh, make it a lot
25:05:25smaller. We'll say this data was
25:05:28collected by the EPA. I'm going to say
25:05:32uh Environmental.
25:05:36It's always tough to watch people watch
25:05:37me spell. By the way, I'm not good at
25:05:39environmental uh protection agency. And
25:05:43then I'm going to say in 2023. I think
25:05:46this is fine for now. If we want to add
25:05:48something later to it, we can. Or make
25:05:50it larger. Uh, whatever we want to do.
25:05:52But I'm going to keep it just like this
25:05:54for now. Actually, I think I do want to
25:05:57make this bigger or this part at least
25:05:58bigger. It seems like it should be. I
25:06:01feel like it should be bigger. That just
25:06:03feels better. I don't know why. Uh,
25:06:05don't get onto me. Now, we can add more
25:06:08visualizations to this, and of course,
25:06:10we will. But the first one, this one is
25:06:13probably the easiest one because we're
25:06:15not really digging into the data itself.
25:06:17We're more just trying to visualize it
25:06:19because latitude longitude data is, you
25:06:20know, pretty specific. Now, let's come
25:06:22back over here to our SQL editor. I'm
25:06:24going to try using some AI here on this
25:06:25next one because uh the next one's going
25:06:28to be a little bit trickier, I think. Uh
25:06:30let's just create a new query because I
25:06:32think that's perfectly fine. Uh, but I
25:06:35want to make sure it's in the right
25:06:36schema. It is not. Let's go to
25:06:39emissions. Let's go to default. In fact,
25:06:42I could have just copied this over. Um,
25:06:44honestly, so I'm going to come over
25:06:46here. Let's paste this in here. Now, I'm
25:06:49going to edit this. I now want to focus
25:06:52on a new thing that we're looking at. I
25:06:54want to take a look at some data. Now,
25:06:56I'm going to do this because this is
25:06:57what I personally do. I'm going to
25:06:58duplicate this and I'm going to bring it
25:07:01back to uh the catalog
25:07:05and go to emissions, go to default, go
25:07:08to emissions data and look at the sample
25:07:11data. The reason I'm doing this is
25:07:13because I want to be able to just
25:07:15oneclick over, take a look at my raw
25:07:17data. You can also just create a query
25:07:20and tab over. Um, but then I don't know,
25:07:22that's not my that's not my workflow. I
25:07:24like having a different thing for it.
25:07:25You can do that if you'd like. uh just
25:07:27do you know select everything from
25:07:29emissions data. Now what we are going to
25:07:31do is we're going to use the AI and what
25:07:34we're going to be looking at is I want
25:07:35to take a look at this county name as
25:07:37well as the population. So for each
25:07:40county so this is in Alabama. This is
25:07:43the county state name in Alabama. I want
25:07:45to look for this county and this
25:07:47population. Let's take a look at our
25:07:50emissions. But more specifically I want
25:07:52to take a look at the emissions per
25:07:54person. So, we're going to have to do a
25:07:55calculation here. So, I'm going to ask
25:07:58the AI. I'm going to say I want to look
25:08:00at the emissions per person in each
25:08:06county. Let's go ahead and generate
25:08:08this. Let's see if it will get exactly
25:08:11what I want. It may or may not. And
25:08:14let's accept this. I'm just going to hit
25:08:16tab. So, we're casting this as a double.
25:08:19And then we're casting the population as
25:08:21a double. And then we're saying that's
25:08:23the emissions per person. Now, this
25:08:25theoretically should work, right? But
25:08:27let's go over to our GHG emissions right
25:08:32here. Now, here's the thing, and this is
25:08:34not, you know, a data brick specific
25:08:37thing, but this happens all the time in
25:08:39almost any platform is this is a number.
25:08:42This is a 100% a number. Has a comma
25:08:44there. Should be rented as a number. But
25:08:46you can see right here, I can just tell
25:08:48you right now, this query is going to
25:08:50fail um because it needs to be changed.
25:08:52Let's go ahead and try to run this.
25:08:56Now, the reason why this is going to
25:08:57fail is because this is a string, right?
25:09:00We have letters, we have characters that
25:09:02are not numeric in this column, and that
25:09:05is an issue. And so, what we need to do
25:09:07is we need to convert this or we can
25:09:11also in here ask it to diagnose this
25:09:13error. I'm gonna see if the assistant
25:09:16can do it for us because if it can then
25:09:18that would be fantastic. Let's see if
25:09:19it's able to do this. This is kind of a
25:09:21specific fix. It needs to look into the
25:09:23data.
25:09:26And it looks like it got it perfect. It
25:09:27says you need to remove the thousands
25:09:29separator before you cast it. This looks
25:09:31excellent. Let's actually uh replace
25:09:34this. Let's get rid of that. And now
25:09:37let's try running this. And wait a
25:09:39second. We have latitude and longitude
25:09:41here.
25:09:42Let's go back because I didn't want
25:09:45latitude and longitude in the first
25:09:46place. I was just looking at this and I
25:09:48got mixed up. Um, okay, that's fine.
25:09:50We'll keep this. But I am before we even
25:09:53run this, I am going to say I don't want
25:09:56latitude. Did I spell it right? In
25:09:59longitude, I want the
25:10:03county and the population for those
25:10:06columns. I'm just using AI here. I
25:10:09normally I could just fix this myself. I
25:10:12just want to test it because I think
25:10:14it's, you know, an interesting thing.
25:10:15All right, let's accept this and let's
25:10:17go ahead and try running it and see what
25:10:20happens. All right, we're running into
25:10:22some issues. And I'm just going to tell
25:10:23you that's going to happen and that's
25:10:25okay. Let's take a look. One, we I don't
25:10:27think we even have a county column.
25:10:29Let's go right over here. We have a
25:10:31county name column, which actually
25:10:33should be fine since we're not grouping
25:10:35on anything. But I actually want this
25:10:37column right here. So, I'm going to copy
25:10:39this as the column name and we'll come
25:10:43right back here. Population should be
25:10:45correct. And then we're working with a
25:10:48replace. I believe that we need back
25:10:51ticks for this and not brackets.
25:10:57Let's just go ahead and try that. And
25:10:59let's run it. All right. Now, this is
25:11:01working. Listen, AI is not perfect. Uh
25:11:04sometimes we got to step in and do what
25:11:06we do. So, this looks correct to me.
25:11:09Let's take a look just really quick
25:11:10because we are breaking it down by the
25:11:12county state name. Um, which is what I
25:11:15would do. I wouldn't want to use this
25:11:18county name because what if we have
25:11:19another county name that's Baldwin
25:11:21County, right? And when we start trying
25:11:23to visualize this data, then that could
25:11:25be issues because it's going to probably
25:11:27try to group that data and it just it'll
25:11:29cause issues. Now, what we're going to
25:11:31do with this is we're actually going to
25:11:32look at the top 10 emissions uh based
25:11:36off of probably the emissions per
25:11:38[snorts] person. I think that's all we
25:11:40should do. So, I'm just going to add it
25:11:42this time. I'm not going to ask the AI.
25:11:44So, I'm just going to say order by and
25:11:46then I'm going to say emissions per
25:11:47person. And we'll do descending. And
25:11:50this should be really Actually, I should
25:11:51have done I should limit it by like 10
25:11:53or something. I'm going to say limit 10.
25:11:57There we go. Now, let's run this. And
25:12:00I'm super interested. This is real data.
25:12:02So, you know, I'm really curious. So,
25:12:05emissions per person, the population is
25:12:07only 5,000. They have a lot of
25:12:08emissions. That's in New England. This
25:12:11is in North Dakota. New England. North
25:12:12Dakota. Very interesting. So, based off
25:12:16the emissions per person, these are the
25:12:19places. And this is not what I would
25:12:20have guessed. I would have guessed like
25:12:22New York or, you know, I don't know,
25:12:25North Carolina or something like that.
25:12:27But these are we got New England and
25:12:29North Dakota. And then I think Mo is
25:12:32Missouri.
25:12:34I don't quote me on that, but this looks
25:12:36good to me. Um, this query was a little
25:12:38bit more challenging than our last one.
25:12:40I think you would agree. Let's bring
25:12:42this over. We're going to go back to our
25:12:44dashboard. I could also make this
25:12:47another tab. Um, and it looks like that
25:12:49reset. We'll do that at the end. Um,
25:12:52we'll just place that how we want it at
25:12:54the end. Now, what we actually need to
25:12:56do is put this in our data. So, I'm
25:12:59going to come in here. I'm going to
25:13:01place this in there. Again, it's not um
25:13:06it's right here. The emissions, we can
25:13:09change this. So, we don't have to um we
25:13:12don't have to do that. Let's go ahead
25:13:13and run this.
25:13:15And our data is working. Even though
25:13:16it's underlined, it's still reading it
25:13:18in fine. We do have it uh correctly
25:13:20done. We can always like we did before
25:13:23uh do emissions
25:13:26and it shouldn't say emissions data
25:13:27emissions.default
25:13:30emissions data just to get rid of those
25:13:32red lines. We can do that. Now we have
25:13:35our data down here. This is really
25:13:38useful. We don't necessarily have to
25:13:41even use this column, but we do have the
25:13:43population for emissions. And uh let's
25:13:47get rid of our limit because I want to
25:13:49make a specific type of visualization
25:13:51with this. It's a scatter plot. Scatter
25:13:53plots are really great because we're
25:13:54going to be able to use the population
25:13:56and the emissions per person on a
25:13:58scatter plot and kind of take a look.
25:13:59Hey, as the population increases, does
25:14:02the emissions per person increase as
25:14:04well or does it decrease? A scatter plot
25:14:06would be great for this. Let's call this
25:14:09uh let's do emissions
25:14:11per person. That's what we're going to
25:14:13call this one.
25:14:15Let's come over here and we're going to
25:14:17come in and we're going to choose our
25:14:19emissions per person. Now on our xaxis
25:14:23we can choose the emissions per person.
25:14:25We can switch this around, see which one
25:14:27works better. Then we'll also choose the
25:14:29population. Let's scroll down a little
25:14:31bit. Now we need to change this from a
25:14:33bar chart cuz that does not look right.
25:14:35We're going to change this into a
25:14:37scatter plot. Now for the scatter plot,
25:14:40we want the raw data. We don't want any
25:14:42aggregation. So, let's say none here.
25:14:44And for the sum of population, none as
25:14:47well. That should be good. We can also
25:14:51really quick change the size because
25:14:53it's a little blurry. I want to see a
25:14:54little bit more granular of what we're
25:14:56looking at. We have this one up here.
25:14:59Uh, this emissions per person is 0.5.
25:15:01Okay, so that's low, but the population
25:15:02is 10 million. I'm curious where this
25:15:05actually is. Let's come down here to the
25:15:08tool tip. Let's it add in the county
25:15:12state name. So now when I hover over it
25:15:15and it's so small. Um now when I hover
25:15:18over it, this is Los Angeles County. Um
25:15:21this is actually really good. There is a
25:15:23very low emissions for that amount of
25:15:25people. And you can see that because
25:15:26it's over on this left hand side. If we
25:15:29had something um over here, this is not
25:15:31going to be good. Uh this is one of our
25:15:33New England ones. They have emissions
25:15:35per person at 5.95 and that's in uh I
25:15:38think megat tons. So that's a lot of
25:15:40emissions. Their population is 5.16,000.
25:15:44That's not a lot. That's a little over
25:15:465,000. They have a lot of emissions. So
25:15:49as you can see, the areas that actually
25:15:51have a higher population tend to have
25:15:53less emissions. Um, overall, I mean,
25:15:56these are Cook County, Harris County in
25:16:00Texas, uh, Maricopa County in Arizona,
25:16:03but a lot of these ones are actually
25:16:05fairly low per person. In the areas that
25:16:09don't have a lot of people, they
25:16:10actually tend to looks like they have
25:16:12higher emissions for the lower
25:16:14population.
25:16:16I mean, some, of course, are over here
25:16:18with very low emissions per person. It's
25:16:20just really interesting. I'm I'm kind
25:16:22of, you know, I'm kind of looking into
25:16:24this as we go. Let's add this title.
25:16:27We'll say emissions per person. We'll
25:16:30say emissions verse population.
25:16:35And uh we could add some type of
25:16:37description. We don't have to, but I'm
25:16:39just going to say higher populations
25:16:43tend to have lower emissions per person.
25:16:48Um, and you know, we could go more in
25:16:50depth and add more notes. That would be
25:16:52useful, but we're not going to. But I
25:16:54think that's really interesting. Uh, I'm
25:16:56kind of fascinated by that myself. So,
25:16:58we have this emissions verse population
25:17:00one. Really quick, I'm going to add a
25:17:02title to this one. I'm going to say
25:17:05emissions
25:17:07per location. I'm just going to keep it
25:17:10like that. I think that's uh perfectly
25:17:12fine.
25:17:13Uh, let's scroll down really quick.
25:17:15We're going to add in some more
25:17:17visualizations. I'm going to just put
25:17:19these here so that we can scroll down a
25:17:21little easier. Um, but now we have these
25:17:23ones in here with us. Let's come back to
25:17:26our SQL editor and let's add a new tab.
25:17:30So, I'm going to say a new query here.
25:17:32Now, what I want to look at is the total
25:17:34emissions by state because I want to see
25:17:37what states are just doing the worst.
25:17:38They have so many emissions. They're the
25:17:40top 10 worst states in America. I
25:17:42wouldn't say that. That's actually
25:17:43that's a bit extreme. We will need to
25:17:45reuse this really quick. Um, we're going
25:17:47to have to reuse this replace. I'll just
25:17:50copy this over. In fact, I feel like I
25:17:52can write this quite quickly. If we were
25:17:55in the emissions default here, um, what
25:17:58we need to do is we need to come over
25:18:01here and let's see. So, I don't want to
25:18:04have to take this and break it out by
25:18:06the state over here. We have the state
25:18:08abbreviation. So, let's use it. Let's
25:18:10copy this column name. Let's bring it
25:18:13back. And all we're going to do is just
25:18:17keep the state abbreviation. We don't
25:18:19need to do any of these calculations. Uh
25:18:21we should be able to get rid of all of
25:18:23that. And let me see what I'm doing
25:18:26wrong here. Okay, it looks like I have
25:18:27an extra parenthesis. Then we're just
25:18:30going to say total emissions.
25:18:34That should be fine. Then we need to
25:18:36group by. So we're going to say group by
25:18:38because if we look at our data, we have
25:18:40a lot in Alabama, right? We have a lot
25:18:42of counties in each state. So, we have
25:18:44to group on this state abbreviation and
25:18:47then I'm not going to go over to it, but
25:18:50then we're going to group and do our sum
25:18:54right here. Right? And we're have to
25:18:55actually do the sum in just a second.
25:18:57So, we're going to say group by the
25:18:58state abbreviation.
25:19:00And there we go. And we'll take this.
25:19:05And then right up here, we just need to
25:19:07do the sum. So, I actually did need that
25:19:09extra parenthesis. I got rid of it. I
25:19:11was I was so shortsighted. I got too
25:19:14excited. And we'll keep the top 10
25:19:16actually. Um because I know that when we
25:19:19start visualizing this, we're not going
25:19:21to need uh all of them. Uh although it
25:19:25would be Let me just comment this out
25:19:26for a second because I am curious
25:19:29to look at all the states and see which
25:19:31one has the um highest. So the total
25:19:33emissions is Texas, Florida, Ohio,
25:19:35Illinois, Georgia. you know, that's a
25:19:37lot of the south and southeast and and
25:19:40not a lot from uh the west coast. Let's
25:19:42come down here. We have Vermont,
25:19:45Richmond, AK, is that Arkansas, uh DC,
25:19:50me, Maine. So, actually some of the
25:19:52Northeast is actually quite good with
25:19:54New Hampshire. Uh but then the West
25:19:56Coast and like kind of the Midwest tends
25:19:58to be really low emissions probably
25:20:01because their populations are lower, but
25:20:04I guess maybe not because of what we
25:20:06looked at earlier. Anyways, let's limit
25:20:09this by 10. And then I'm going to copy
25:20:12this over. Yeah, we'll go back to our
25:20:14dashboards. Let me go back to the
25:20:15emissions dashboard.
25:20:20Let's scroll down.
25:20:22And actually, we got to go to the data
25:20:24first. Let's create this. And we'll run
25:20:27this. I need to have our emissions
25:20:31and our default.
25:20:33That should be good. Let's run this
25:20:35data. And then I'm going to rename this.
25:20:38I'm going to say uh total
25:20:41emissions per state.
25:20:44That should be good. So now we have
25:20:46these. These are our top 10. What I
25:20:49actually want to look at this for is
25:20:51percentage-wise. We're going to have to
25:20:53do some other analysis because I do want
25:20:55to add something that you can't
25:20:57necessarily visualize. Um, we're just
25:20:59going to have to dig into that in the
25:21:01data really quickly and perform a little
25:21:03calculation. It's not nothing crazy. Um,
25:21:06but we may use AI for it. We'll see if
25:21:08it can uh it can work for us. We'll do
25:21:10totally emissions by state and we're
25:21:14going to do a pie chart right here. Now
25:21:18for the angle, all we have to do is the
25:21:21total emissions, but we want to break it
25:21:23out our color. We want to break it out
25:21:25by the state abbreviations. Uh, we
25:21:28definitely need to add the labels here.
25:21:31Let's add our labels. There we go. Um,
25:21:35and it has it over there. So, we have
25:21:37this little legend right down here. We
25:21:39have Texas. That's 20% in the top 10.
25:21:42That's in our top 10. These are the
25:21:44total emissions. If we tried to add all
25:21:46of them, which you know what? Let's just
25:21:48go try it. We can get rid of these 10.
25:21:51Uh we can do that. Let's run this. We'll
25:21:55have all of our data.
25:21:58Let's go back to our dashboard.
25:22:01And now we have all of them. So, it
25:22:03looks like Texas is 10%. But that is a
25:22:06lot, right? We have a lot of different
25:22:08colors. And it's really hard to see uh
25:22:11almost anything if I'm being honest.
25:22:15If we want to, and maybe this is a good
25:22:17uh thing to do, is we can add a filter
25:22:20right here. And I haven't even I haven't
25:22:23done this at all in this video yet, but
25:22:25I'm just going to say um maybe a range
25:22:29slider might be good. Uh for the field,
25:22:33we can do let's see total emissions per
25:22:37state. We'll look at the total
25:22:38emissions. And let's see if I can just
25:22:41kind of scroll up
25:22:44and do something like this. That might
25:22:46work. But then again, it's then, you
25:22:49know, only doing the calculation or the
25:22:51percentage based off of that. So, we're
25:22:53just reinventing the wheel here. I'm
25:22:55just demonstrating it can be done. Um,
25:22:57but I don't think we need this. All that
25:22:59being said, we're actually going to
25:23:01delete that.
25:23:03But that's [laughter] okay. Let's go
25:23:04back. We're going to add the limit 10.
25:23:06Listen, I'm here for education purposes,
25:23:09right? I want you to know what you can
25:23:10do. If you want to do that, you can. Um,
25:23:14but we're not going to do it. Let's go
25:23:15back here. Now, we have this. This looks
25:23:18good, but I don't like uh this title,
25:23:21actually. Let's get
25:23:24right here. I'm just going to say
25:23:26emissions
25:23:28percentage or something like that is
25:23:30fine. Feel like that's good enough for
25:23:32like a title even. We can and I think we
25:23:35should come in here and add some type of
25:23:37description, some information here. So,
25:23:40I'm just going to say the top 10 states
25:23:42account for X amount of emissions for
25:23:47the whole for all of the US. Now, I'm
25:23:51saying X amount because we don't have
25:23:54that information yet. We actually need
25:23:56to write that query. Let's go back to
25:23:59our SQL editor and let's dig into this.
25:24:02I am gonna try to use this assistant
25:24:05because I feel like it might be able to
25:24:07understand um what I'm doing. I'm gonna
25:24:10paste this in and I'm just going to say
25:24:13I want to know what percentage
25:24:17I need to write this right. But what
25:24:19percentage
25:24:21of emissions
25:24:23are the top 10
25:24:26states for the whole country?
25:24:30and let's see if it understands that. I
25:24:33don't know if I even wrote that well.
25:24:35Um, but let's see if it did with it if
25:24:38it does. Uh, this is a CTE it's writing.
25:24:40I'm just going to see what it's writing.
25:24:41It's writing a CTE. It's calculating the
25:24:44top 10 and then it's selecting
25:24:48it's selecting the sum of the total
25:24:50emissions.
25:24:52Okay, this could work. Let me take this
25:24:55query real quick and I'm actually going
25:24:57to open up a new tab or a new query.
25:25:01And let's make sure. Oops. I need to get
25:25:03all this.
25:25:05Oh, it's doing it for a notebook. That's
25:25:06all right. Um, I'm going to get rid of
25:25:08that because I'm going to come down here
25:25:11to emissions to default. Let's paste
25:25:14this in here. And this may work. Let's
25:25:18run this.
25:25:20Okay, so this is the top 10 emissions.
25:25:22This is the top 10 percentage.
25:25:25I believe this is correct. Um you can
25:25:28validate this but what it's doing is we
25:25:30don't need this SQL here. What it's
25:25:33doing is it's creating a CTE calling it
25:25:35the top 10. Then we're selecting the
25:25:37state abbreviations. We have this query.
25:25:39It's taking the total or the sum for the
25:25:41top 10. And then we're using that later
25:25:44on. So now we're doing select and this
25:25:46is the sum of the total emissions from
25:25:49up here. And then we're taking the sum
25:25:51of the total emissions divided by this
25:25:54number right here which is all of them.
25:25:56Now, wait a second. I may be wrong on
25:25:58this because we're only pulling from the
25:26:00top 10. So, maybe it's just looking at
25:26:04the total emissions from up here
25:26:08as top 10 emissions. It's taking the sum
25:26:11of the total emissions and then
25:26:12selecting the sum from emissions data.
25:26:15No, you know what? No, it is taking it
25:26:18from it's doing a subquery in here. I
25:26:20missed that. So, this subquery is
25:26:22actually selecting all of the emissions
25:26:25from the emissions data. This is great.
25:26:28I'm going to say it accounts for 51%.
25:26:30This is uh great job, AI. I'm glad it uh
25:26:34did a good job. I'm I I think I am going
25:26:37to add uh make a duplicate of this. I
25:26:41just that's how I that's my personal
25:26:42workflow. Okay. So, let's come in here
25:26:45and we're going to say that this
25:26:47accounts for the top 10 account for 51%.
25:26:54Uh, let's make sure I spell this right.
25:26:56The top 10 states,
25:26:59I'm actually going to say these 10
25:27:00states. These 10 states account for 51%
25:27:05of all emissions.
25:27:08I'm going to say
25:27:11of
25:27:14What am I What am I doing
25:27:17of all
25:27:19emissions
25:27:22in the US? I'm just I'm having a tough
25:27:25time writing.
25:27:26That is a useful statistic.
25:27:29That is a really useful statistic. So, I
25:27:31think this one is done. I think this is
25:27:33a really good visualization. Look, pie
25:27:35charts have its place. Percentages like
25:27:37this, I think, are really useful. Um,
25:27:40let's come back. Let's go to our SQL
25:27:42editor. I'm going to have that over
25:27:43here. So, I'm going to have my SQL
25:27:44editor, emissions dashboard, and the
25:27:46data. Now, what I want to do is I want
25:27:49to name and shame a little bit. Okay. I
25:27:52want to take a look the county state
25:27:53name. What specific county? And we kind
25:27:56of looked at this earlier when we were
25:27:58getting our population data because we
25:28:00had to ground it with a group by
25:28:02something, but now we're actually using
25:28:03the county state name. We're going to
25:28:05take a look at the top 10 counties
25:28:08within the US. and we want to create a
25:28:11bar chart for this. So, let's uh use
25:28:14this and come right back here. Uh this
25:28:17was by far the hardest query uh that
25:28:19we've written today. AI did well. Um we
25:28:22you know we gave it a lot of the
25:28:23context, but I think it did a good job.
25:28:26Let's bring in some of our previous like
25:28:29this was a really simple
25:28:32um thing here. Actually, let's use this
25:28:36because it has the uh a lot of the
25:28:39information we're already wanting. Now,
25:28:41all we're doing is we're just looking at
25:28:44the total emissions, not per person. So,
25:28:47let's get rid of
25:28:50this. We're literally just looking at it
25:28:52like this. So, we have the county state
25:28:54name. Let's go to emissions default. Um,
25:28:58we have the county state name from
25:29:00emissions data. And we don't have to do
25:29:03this. We need to do it.
25:29:07Let's close this parenthesis and call
25:29:09this as um total emissions. And then we
25:29:14have to use this alias for this part to
25:29:17order by it down here. We don't have to
25:29:20technically. We could just copy this
25:29:22whole cast uh and replace, but we're not
25:29:24going to. So now we want to look at the
25:29:26top 10. So this should work. Now we have
25:29:30the county state name in Arizona, Texas,
25:29:33Illinois, Florida. Geez, just so many so
25:29:36many emissions. Uh, you know, think of
25:29:40think of the world. Think of the think
25:29:41of your country. Let's come in here. Uh,
25:29:45no, we need to go back to the data.
25:29:46Let's create from SQL and we're going to
25:29:49put this in here. Let's go back to our
25:29:51emissions. Let's run this. This should
25:29:54be good. Awesome. So now we have the
25:29:57highest total emissions right here.
25:30:01I mean, listen, even though Los Angeles
25:30:04County is in the top 10, it's because
25:30:07their population is four times as much
25:30:10as some of these. I mean, I just it's
25:30:12it's really interesting. That's why we
25:30:14did the emissions per person earlier
25:30:15because there are outliers and, you
25:30:17know, the total population does matter.
25:30:20Let's come in here and we're just going
25:30:21to rename this uh county shaming. All
25:30:25right, that's what we're going to call
25:30:26it because that's what we're doing.
25:30:28Let's be honest. All right, we're just
25:30:30going to make a bar chart. We're going
25:30:32to go to the county shaming. We're going
25:30:34to use a bar chart
25:30:36for our x-axis. Uh, let's use totally
25:30:40emissions. Then for our yaxis, we can
25:30:43use the county state name. Um, and I do
25:30:46want to look at that from highest to
25:30:47lowest. So, let's do it like that. We
25:30:51can change the coloring if you want. I
25:30:53definitely want tool tips or not tool
25:30:55tips, sorry, labels. I definitely want
25:30:57labels on this. This is just our total
25:30:59emissions in like megatons. So, I'm
25:31:01going to say that I want to add a title.
25:31:03I'm going to say uh total emissions.
25:31:07I got to spell right. Emissions by M
25:31:10ton. M ton I think of CO2.
25:31:16Uh, and I want to make sure that's
25:31:18correct because listen, this dashboard,
25:31:21I'm gonna send this to the EPA right
25:31:23away. Uh, so we have m tons of CO2.
25:31:26Maybe CO2E. I don't know if I should be
25:31:30putting that in here. Let's add that.
25:31:32All right. This looks great. Um, let's
25:31:37adjust this real quick. I'm actually
25:31:40going to come over this way. Double
25:31:44click.
25:31:45Double click. Come out a little bit. Uh,
25:31:50this this is something that I just have
25:31:53not gotten the hang of. I don't know if
25:31:54I'm doing something wrong, but let's
25:31:56click in on this
25:31:58and uh have that right there. We need to
25:32:01bring back
25:32:03our emissions versus population and
25:32:06bring this up.
25:32:09Now, I do Oh, what did I do here? Oh, I
25:32:12think I might have clicked on. Yeah, I
25:32:14zoomed in. Whoops. Uh, don't mind me.
25:32:17Listen, I make mistakes all the time. I
25:32:20want to clean this up just a little bit.
25:32:22Things like the total emissions. I just
25:32:25want to say total emissions, right? I'm
25:32:28cleaning up these X and Y axes. I'm
25:32:31going to call this uh the county name.
25:32:35Uh, maybe I should say county state
25:32:37name, I guess, to be more accurate. That
25:32:41looks good. Emissions. This one looks
25:32:43good. Here we need Oops, let me zoom
25:32:45out.
25:32:47We need to rename this one emissions per
25:32:50person. We're gonna call this
25:32:53uh emissions.
25:32:55Oops. Did I spell that right? Emissions
25:32:59per person. That I does not look right,
25:33:02but maybe it is. And I'm I'm going to
25:33:04change this population to be
25:33:05capitalized. I just I think it looks
25:33:07better. Population.
25:33:11And uh let's see. Emissions per
25:33:14location. Emissions versus population. I
25:33:17actually should say emissions.
25:33:20Where's the title here? Oh, I got to do
25:33:23it up here. I'm going to say emissions
25:33:26for continental
25:33:29US. Did I spell that right? No.
25:33:31Continental US. My spelling is so
25:33:34terrible. But we're only looking at the
25:33:36continental US here, so that might just
25:33:37be worth noting. Um, we are done.
25:33:42We are done here. We have our United
25:33:45States emissions breakdown and I think
25:33:48this looks great. Uh, we dug into some
25:33:50of the data a little bit deeper than we
25:33:52probably needed to, but I think this was
25:33:54great and we're able to use AI and build
25:33:56out our dashboard. I am super happy with
25:33:58how this turned out. Remember, this is a
25:34:00real data set. There's so much other
25:34:02data in here. So if you want to, you can
25:34:04use the assistant in here and say, "Hey,
25:34:07what other things can I look at for
25:34:08emissions and it'll give you
25:34:10suggestions, but this is what I wanted
25:34:11to build because I was super interested
25:34:13in it." And emissions is just, you know,
25:34:15something that everybody understands.
25:34:18People are curious about it and so
25:34:19visualizing it, being able to see it and
25:34:21kind of shame some of these counties
25:34:23that just got so many emissions, but
25:34:26it's worth it. It's worth uh worth
25:34:28building it out. So I hope that you
25:34:29enjoyed this. I hope that you found this
25:34:31helpful. Before we go, I want to give a
25:34:33huge shout out to the sponsor of this
25:34:34entire series and that was Data Bricks.
25:34:36Data Bricks has been so awesome to work
25:34:38with because I already love Data Bricks.
25:34:40It was an easy fit. And what's even
25:34:41better is they have this data bicks free
25:34:43edition. If you have not already, if you
25:34:44just watched through this without
25:34:45following along, create a Data Bricks
25:34:48free edition account, go ahead and do
25:34:49that. I will leave a link in the
25:34:50description so you can build something
25:34:52like this completely for free. It is an
25:34:54amazing, amazing deal. With that being
25:34:56said, thank you guys so much for
25:34:58following along through this entire
25:34:59project. I had a lot of fun. and I hope
25:35:01you did too. If you have not already, be
25:35:03sure to like and subscribe and I will
25:35:04see you in the next video.
25:35:07[music]
25:35:18What's going on everybody? Welcome back
25:35:19to another video. Today we're going to
25:35:21build ETL pipelines in data bricks in
25:35:23under one hour.
25:35:28>> [music]
25:35:30>> Now, in this video, we're going to cover
25:35:32several different things. First, we're
25:35:33going to work on data ingestion, just
25:35:35getting data in. Second, we're going to
25:35:37actually build out our ETL pipelines.
25:35:39And then third, we're going to work on
25:35:40data orchestration or creating jobs in
25:35:43data bicks. At the very end, we're going
25:35:44to have a full end-to-end project where
25:35:46we pull data in from a folder in an AWS
25:35:49S3 bucket. Then, we automate it with an
25:35:51ETL pipeline to clean that data. Now,
25:35:53this video is made from several shorter
25:35:56videos that we have done on data bricks
25:35:57in previous lessons, but we're putting
25:35:59them all into one long video, so you can
25:36:02watch it all at one time. Let's not
25:36:03waste any more time. Let's jump into the
25:36:05first part, which is data ingestion.
25:36:07Before we jump into this data
25:36:08engineering series and start doing all
25:36:10the things, I want to slow down for just
25:36:12a second to take a look at what ELT is
25:36:14in data bricks. This is the process that
25:36:17we're going to be walking through for
25:36:18this entire series. We're extracting
25:36:20data or getting our data into data
25:36:22bricks. That involves loading the data
25:36:24into different schemas and having that
25:36:25data available and then we transform
25:36:28that data. Now ELT might sound odd
25:36:30because most people are used to ETL
25:36:32where you extract data, you transform it
25:36:34and then you load it into the database.
25:36:37With a lot of modern data workflows, it
25:36:38doesn't actually make much sense to
25:36:40transform your data before because
25:36:42compute is quite cheap these days and so
25:36:44you can just load your data into data
25:36:46bricks and then transform it after. Now
25:36:48there's something called the medallion
25:36:49architecture. We're going to take a look
25:36:51more at that in the next lesson when we
25:36:52take a look at bronze, silver, and gold
25:36:54architectures. Now, this is a really
25:36:56great way to kind of stage your data and
25:36:57it's been like this for a long time even
25:37:00before data bricks, but we'll be going
25:37:02into why and how we actually do that
25:37:04within data bricks. Our data when we
25:37:06actually get it into data bicks is being
25:37:08stored in a delta table. It's kind of
25:37:10like a delta file type, which is
25:37:11basically just a parquet file that has
25:37:13this log system where you can kind of
25:37:14revert back and see previous changes to
25:37:17the actual document. And so we store our
25:37:18data in these delta tables and then we
25:37:21can do all of our transformations on it
25:37:22within a notebook or within SQL queries.
25:37:24Now that we've got that out of the way,
25:37:26let's actually jump into data bicks and
25:37:28see how we can do this. All right, here
25:37:30we are on data bicks and we're going to
25:37:31be doing two things. One, we're just
25:37:32going to upload a CSV file. It's
25:37:34probably the simplest way to get data
25:37:36into databicks, but then we're also
25:37:38going to connect to an S3 bucket. And so
25:37:40I'm going to show you how you can do
25:37:42that really easily. And we're going to
25:37:43get all of our data into data bricks.
25:37:45Now we are just working with sample data
25:37:47for this lesson but at the last one when
25:37:50we start doing our full ETL process and
25:37:52automating this entire thing then we'll
25:37:54be using real data and so it'll be a lot
25:37:56more a little bit more complex. There's
25:37:58a few ways to ingest data. One you can
25:38:00just click on this bring in data and
25:38:03it's going to take you right down here.
25:38:04But we can also just go to our data
25:38:06ingestion. And so when we click on this
25:38:08we're going to upload files to a volume
25:38:11or we're going to create and modify a
25:38:13table. Now, these are two separate
25:38:15things and these are important things to
25:38:16understand. Let's actually come over
25:38:18here to catalog for a second. And what
25:38:20we're going to do is we're going to come
25:38:20over here and we're going to create a
25:38:22new catalog. And we're just going to
25:38:24call this one our data engineering. Uh
25:38:27that's all we're going to call it. I was
25:38:29going to keep going, but we'll call it
25:38:30the data engineering one. And let's go
25:38:33ahead and view this catalog. Now, when
25:38:35we create a schema, we're going to say
25:38:38this is uh video one. Let's go ahead and
25:38:42create this.
25:38:43We have within our data engineering we
25:38:46have our default and then we have this
25:38:47information schema but we also have this
25:38:49video one. Now we don't have any data in
25:38:53the schema but what we can do is we can
25:38:56create different ways to store our data.
25:38:58We can store it in a volume or we can
25:39:00store it in a table. Now in a previous
25:39:02series I kind of dove into these and how
25:39:04you can store your data as well as how
25:39:06to access the data once you put it in.
25:39:08We're going to be putting all of our
25:39:09data into tables. So, I'm just going to
25:39:11set up one table. Now that we're here,
25:39:13though, we can do the same thing that we
25:39:15would do if we came over to our data
25:39:17ingestion, which is basically just drop
25:39:19a file in here like you would on any
25:39:21platform. Let's just go over to the data
25:39:23inestion just so we get the full
25:39:25experience. We're going to create or
25:39:27modify a table. We're going to select
25:39:29our CSV file. So, it's just our users_y,
25:39:32and we're going to go ahead and upload
25:39:34this. So, now we have this preview of
25:39:36our data, and we're going to specify
25:39:38what we want to do with it. We could
25:39:39create a table. We could overwrite an
25:39:41existing table. And we want to put this
25:39:43in our data engineering video one. So if
25:39:45it doesn't automatically populate, you
25:39:46can always just specify where you want
25:39:48to place it. And we're going to call
25:39:50this underscore CSV because we're going
25:39:52to be bringing in the same file from an
25:39:54S3 bucket. So I just want to specify
25:39:56where we got this data. Let's come down
25:39:59here and we're going to create our
25:40:00table.
25:40:02So now we have our data sitting in our
25:40:04video one schema. So this is our
25:40:06users_y_csv.
25:40:08This is the simplest way to get data
25:40:11into data bricks. But I also have this
25:40:13exact same data sitting right over here
25:40:16in an S3 bucket and I want to use it. I
25:40:19want to connect to this data. I want to
25:40:21pull it in automatically. And that's
25:40:23going to really help us later on down
25:40:24the line when we start automating this
25:40:26whole process because we're going to
25:40:27create a connection to this data source
25:40:29so we can automatically pull this data
25:40:31in. And that's a big part of just data
25:40:34engineering in general, which is
25:40:35creating systems that can automatically
25:40:38ingest, transform, and load your data.
25:40:40So, what we're going to do is we're
25:40:41going to come right over here. Now, I
25:40:43just want to show you this. I'm going to
25:40:44have a link down below so that you can
25:40:46see this as well. But this is basically
25:40:48just how you're going to create the
25:40:50connection. I'm going to show it to you
25:40:51in a second. It is very, very simple.
25:40:54So, let's come right over here. And what
25:40:56we're going to do is we're going to come
25:40:57down to data ingestion. Now, we want to
25:41:00go to the data bricks connectors and we
25:41:02want to go to our Amazon S3 bucket right
25:41:05here and we need to create an external
25:41:07location. So, we're basically connecting
25:41:09our databicks account to our Amazon S3
25:41:12bucket. And then we can bring that data
25:41:15in very easily. What we're going to be
25:41:17using is this AWS quickart. Let's go
25:41:19ahead and select next. We need to put in
25:41:21our bucket name. So, I'm going to come
25:41:23over here. Let's click in here. We can
25:41:26actually get it right here. there.
25:41:27There's other places to get it, but I'm
25:41:29just going to copy it from here. Uh,
25:41:32we're going to go back to our catalog
25:41:35explorer, and there's our bucket name.
25:41:38So, now what we're going to do is we're
25:41:39going to generate this new token, and
25:41:41we're going to copy this. Now, we're
25:41:43going to come over here to launch in
25:41:44quick start. And all we have to do is
25:41:47it's going to connect to our account.
25:41:48So, that makes it pretty easy if you're
25:41:49already logged in. Then, we're going to
25:41:51come down here and we're going to say I
25:41:54acknowledge and we're going to say
25:41:56create stack. It's going to come right
25:41:58here and it's going to say create in
25:41:59progress. It's just going to validate it
25:42:01for a second. I had already done this uh
25:42:03before when I was making this video
25:42:05earlier just to confirm everything was
25:42:07working smoothly. And what it's going to
25:42:08do is it's just going to say create
25:42:10complete and then you're going to be
25:42:12good to go. All right. So, that took
25:42:13about 2 minutes and it says it is
25:42:15complete. So, all we're going to do is
25:42:17come back here and we're going to
25:42:19refresh this page. So, I'm going to go
25:42:21ahead and refresh.
25:42:24And now that that connection is active,
25:42:26we now have access to our users
25:42:29dirty. Let's come under here. We're
25:42:31going to click on this and we're going
25:42:32to go to preview table. So now we get
25:42:34this preview of the exact same data set.
25:42:36There's nothing changed. I'm not trying
25:42:37to trick you. All we have to do is we're
25:42:39going to come up here to data
25:42:40engineering and we're going to go to
25:42:41video one. And then we need to name
25:42:43this. So I'm going to call this one
25:42:46dirty
25:42:48data
25:42:50S3. So I'm just naming it this purely so
25:42:54that we know which one came from which.
25:42:56Let's come down here to create table.
25:42:58And now we can see over here we have our
25:43:00dirty data S3 and our users_y_csv.
25:43:04I named it completely wrong but these
25:43:06are the exact same data sets and now we
25:43:09have them in from two separate
25:43:10locations. Now this is really important
25:43:13especially as we start automating a lot
25:43:15of this. If you have data that's sitting
25:43:17in an S3 bucket and you have other
25:43:18systems that then upload it into that
25:43:20bucket, we're going to be able to ingest
25:43:22that data automatically, whether it's
25:43:24updated or if it's a new file. And we'll
25:43:26set all sorts of triggers and schedules
25:43:28and all sorts of really cool things in
25:43:30later lessons. So that's how we ingest
25:43:32data within data bicks. In our next
25:43:34lesson, we're going to be transforming
25:43:36data within an actual data pipeline. So
25:43:38we're going to have the entire ingestion
25:43:39process as well as the transformation
25:43:41process all in one place. Now, in the
25:43:44last lesson, we worked on data ingestion
25:43:46into data bricks. So, we were able to
25:43:47connect to just a local file, just kind
25:43:49of reading that file in and then we were
25:43:51also able to connect to an AWS S3
25:43:54bucket. Now that we have that data
25:43:55pulled in and we have it actually
25:43:56sitting in our schema, we need to clean
25:43:58this data up a little bit. And so, we're
25:44:00going to need to transform this data,
25:44:01which is part of the extract, transform,
25:44:03and load within an ETL process. So, we
25:44:06extracted and we ingested that data. And
25:44:08now, we need to clean up our data
25:44:10because it is messy. That's the
25:44:11transformation piece of the ETL process.
25:44:14Once we have this done, we can put it
25:44:15into an ETL pipeline and then it sits
25:44:17there and it does a lot of the heavy
25:44:19lifting for us and we'll talk about that
25:44:20in this lesson. Now, really quickly
25:44:22before we jump into things, I want to
25:44:24talk about this bronze, silver, and gold
25:44:25medallion architecture that is very
25:44:27popular within data bricks. Now, we've
25:44:29actually already covered this bronze
25:44:31level, which is just our raw data. We
25:44:34ingested our data from our S3 bucket and
25:44:37it's just sitting there in this raw
25:44:38format. This is data that we are just
25:44:40never going to touch. What we're going
25:44:41to do is we're going to create
25:44:42transformations on that data and then
25:44:44we're going to put it into a different
25:44:46table or even a different schema or
25:44:48catalog. When it gets to that location
25:44:50and the data is actually changed, that's
25:44:52going to be in our silver. So this
25:44:54silver layer or architecture is
25:44:56basically just once you clean it up and
25:44:58you have it in a lot better state where
25:45:00there aren't a lot of duplicates, there
25:45:01aren't a lot of issues with the data,
25:45:03that's where it's going to sit where you
25:45:04can then transform it into your gold
25:45:06architecture or layer. Gold is just
25:45:08production ready. You are ready to start
25:45:10using this data. You're going to put it
25:45:11into dashboards. You're going to put it
25:45:13into reports. You're going to put it
25:45:14into your apps. Whatever you're using
25:45:16that data for. Back when I was just
25:45:18using Microsoft SQL Server or any other
25:45:20tool, we would call this raw staging and
25:45:23production. The raw is bronze, the
25:45:25staging is silver, and of course, the
25:45:27production is gold where we actually use
25:45:29that data. So, what we're going to do in
25:45:31this lesson is we're going to actually
25:45:32use this. We already have our bronze. we
25:45:34need to transform my data into silver
25:45:35and then find a business use case to
25:45:37create the gold table. So now that we
25:45:39have this background information, let's
25:45:41go onto our screen and start building
25:45:42this out. So in the last lesson, we
25:45:44brought in this dirty data S3. And this
25:45:48is what our data looks like. We have
25:45:49this user ID, first name, last name,
25:45:51their email, their sign up, the country,
25:45:54and referral source. Now, this is just
25:45:56our raw data. This is our bronze layer
25:46:00right here. Now, in this video, we're
25:46:02not going to do it exactly how I would
25:46:04do it in the real world. I'm just going
25:46:05to kind of keep it all in one place for
25:46:08us. So, within this video one or within,
25:46:11you know, whatever schema you created,
25:46:13we're going to keep our silver and our
25:46:15gold tables all within this one schema.
25:46:18That is not typically how it is done.
25:46:20Here's what you typically would do.
25:46:21You're going to have a data engineering
25:46:23bronze catalog. Then you have a data
25:46:26engineering silver catalog. Then you
25:46:28have a data engineering gold catalog.
25:46:31And all these cataloges would hold the
25:46:34different levels. And so you're not just
25:46:36usually working with one small project
25:46:38like we are in, you know, this lesson,
25:46:40but typically you're working with lots
25:46:42of different projects and you're working
25:46:43with lots of different customers and you
25:46:44want those to be separated out so you
25:46:46don't kind of get them confused and you
25:46:48don't know which data you're supposed to
25:46:49be hitting off of. That typically is how
25:46:51it's done in a real workplace
25:46:53environment. We're just going to do it
25:46:54all right now within this video one
25:46:57schema. So this is our bronze layer
25:47:00right here. This is the file that we are
25:47:02going to be using. Now in order to
25:47:03transform our data to get it to silver,
25:47:06here's what we need to do. Let's come up
25:47:07to our new. Let's go down to our
25:47:10notebook. So we have this notebook right
25:47:12here. Let's call this bronze to silver
25:47:17transformation. There we go. And I'm
25:47:19going to give you a little spoiler here.
25:47:21Uh we're going to create another
25:47:22notebook and we're going to call this
25:47:24one silver to gold. And so we want to uh
25:47:28separate these out. You don't have to
25:47:31separate these out, but for the sake of
25:47:32what we're going to be doing in this
25:47:34lesson, I do want to show you how kind
25:47:35of you set things up and you actually,
25:47:38you know, organize things within an ETL
25:47:39pipeline. And then in the next lesson
25:47:41when we look at jobs and orchestration
25:47:43and automation, this will also come into
25:47:46play and I'll talk all about that. So
25:47:48let's create these two different things.
25:47:50Now, I can write all this out because,
25:47:53you know, I know this data set. It's
25:47:55pretty simple and I already know what's
25:47:56wrong with it. So, I can go in and I can
25:47:59just fix it. I can write this out
25:48:00manually. But, uh, you know, let's get a
25:48:03little creative. Let's take a look at
25:48:05how we can use AI in order to see if it
25:48:08can do most of the heavy lifting for us.
25:48:10Now, in our sample data, I'm going to
25:48:11give you two things that need to be
25:48:13changed because there's only really two
25:48:14big issues. The first thing is in this
25:48:16date column, we actually have it as a
25:48:19string. And that's a problem, right? We
25:48:21need it to be a date column. And the
25:48:23issue is is this right here. We have one
25:48:25date field that is 2.29.24
25:48:28instead of the uh forward slashes.
25:48:31That's an issue. We also have a USR_109
25:48:35as a user ID. And if we go down, we have
25:48:38a 1009 right over here. So we have a
25:48:41duplicate user ID. And in a primary key,
25:48:45like a user ID typically would be,
25:48:47that's an issue. So, we have two issues
25:48:49we need to solve. I am going to try to
25:48:51get the AI, which is the agentic AI,
25:48:54which is uh this one up here that we're
25:48:55going to be using to try to write this
25:48:57out and get it right. So, let's come
25:49:00back to our workspace. Let's come to our
25:49:02bronze to silver transformation and
25:49:05let's bring up our AI assistant. Now,
25:49:09I'm going to describe what I want it to
25:49:11do and then we're going to see if it's
25:49:14able to write it out. I myself could
25:49:15write it out very accurately in probably
25:49:17maybe three to four minutes, but this is
25:49:20not a coding tutorial. I want to show
25:49:22you guys how ETL pipelines work in data
25:49:24bricks, not how to necessarily transform
25:49:26the data. So, let's try this out. So,
25:49:28I'm going to say take my data set and I
25:49:31can pull it up over here just so we can
25:49:33see it. I'm going to go to data
25:49:34engineering video one just so I can see
25:49:37the data. take my data set in the data
25:49:40engineering catalog in video one schema
25:49:46called dirty data S3. Now I like to be
25:49:50super explicit cuz I don't want there to
25:49:52be confusion especially as you have like
25:49:54hundreds of tables you don't want it to
25:49:55read in the wrong tables. I like to be
25:49:57super explicit. We're going to ask it.
25:49:59I'm going to say there is something
25:50:01wrong in the date column making it a
25:50:05string. I want you to identify and fix
25:50:09that issue. There's also duplicates in
25:50:12the data set. I want you to remove
25:50:15duplicates on the ID. Now, I'm being
25:50:18slightly vague, right? I'm not telling
25:50:20it exactly what it needs to do, but I'm
25:50:22going to let this run and we're going to
25:50:23see if it's able to identify the issues
25:50:25and write the code. I do want it in
25:50:28Python. I think that's just the easiest
25:50:30way to transform this data. And so I'm
25:50:31going to say use Python and pandas and
25:50:36let's give it a go. So let's let this
25:50:37think for just a little bit and we'll
25:50:39see what it comes up with. So that took
25:50:41about a minute or so. It did a lot of
25:50:43different things and now it wants to
25:50:44actually run this code. Now before we do
25:50:47that, you can have it ask every time or
25:50:49you can just allow it to run the code
25:50:51after it's done. I'm going to ask it to
25:50:53ask every time just because uh you know
25:50:55I want to make sure. Now, it does have a
25:50:57lot of printing just to show the work
25:51:00that it's doing. I myself don't want
25:51:02this in my output. So, I will ask it to
25:51:05change that in just a second, but it
25:51:06does identify that there's a period. It
25:51:08replace it with a forward slash. It
25:51:10converts it to two date time, which
25:51:12looks correct, and it also formats it
25:51:14for us. Uh, then it comes down here and
25:51:16it's doing just a ton of kind of pretty
25:51:18unnecessary things before it gets to
25:51:21this df.drop duplicates on the user ID.
25:51:24and we're keeping the first one, which
25:51:25is perfectly fine. And then lastly, it's
25:51:27doing a lot of verification. I basically
25:51:30don't want 80% of this code. I just want
25:51:33the simple stuff. So, all I'm going to
25:51:34say is I like the transformations you've
25:51:37done, but get rid of all the print
25:51:41statements. All right, looks like it's
25:51:43done. And as you can see, it cleaned up
25:51:45the code immensely. Uh, this is uh
25:51:48really looking good. I'm going to go
25:51:49ahead and I'm going to accept all. You
25:51:51can see the diffs down here by the way
25:51:52for all the code that it's writing or
25:51:54taking away. We're going to accept all
25:51:56and we are going to run this ourselves.
25:51:58We can I'll just click run all here. Uh
25:52:00but we're going to run this ourselves
25:52:01and then we'll verify and make sure that
25:52:04this actually looks good. So let's open
25:52:06this up. Let's come down here and let's
25:52:08just do a display. We'll do dataf
25:52:11frame_clean which is what it named it.
25:52:14So now let's look at this new dataf
25:52:16frame that it has created. That should
25:52:18be a lot cleaner than before. So now if
25:52:20we come down here, we have our 1009.
25:52:23Let's go see if our 10009 was removed.
25:52:25It was. And let's come over here to our
25:52:28signup date. And it looks like that now
25:52:30is converted to a timestamp, which is
25:52:32perfectly fine. Uh we could also do it
25:52:34as just a date column, but honestly, it
25:52:38really doesn't matter. Uh this is a
25:52:40great uh a great change and it cleans it
25:52:42up immensely. So now it's all
25:52:43standardized. It's actually in a date
25:52:45column or a time stamp column and that
25:52:47works great. Now, all we need to do as
25:52:49the last part of this process is we have
25:52:51to write this table to a new table and
25:52:54that's going to be our silver table. So,
25:52:56I'm going to come down here. I'm going
25:52:58to say and I could put it as the genie
25:53:00code or I can come over here. I tend to
25:53:03like using the side a lot more. I don't
25:53:04know why, but I'm going to say uh write
25:53:07this cleaned table to a new table in the
25:53:11same schema and call it S3_cleaned
25:53:17silver. And so let's go ahead and let
25:53:19that run. And it should take just a
25:53:21second and we'll have that code for us.
25:53:23Let's go down here really quick. We have
25:53:25this. This looks great. I'm going to
25:53:27allow this to run for us. So it's going
25:53:29to run this code. Now it is giving us
25:53:31this warning and this is a very fair
25:53:33warning. We're using overwrite right
25:53:34here. And basically what we're doing is
25:53:36every time we run this, we're
25:53:38overwriting the previous data that's in
25:53:40that table. For now, I'm just going to
25:53:42use that cuz it's not a huge deal. You
25:53:44know, as you start getting more
25:53:46sophisticated with your data pipelines,
25:53:48you are going to want to think about
25:53:49things like adding data to your existing
25:53:52data instead of overwriting. But, you
25:53:53know, that can get a little bit more
25:53:55advanced depending on your data and your
25:53:57data need. Now, it's going to run this
25:53:59and I'm going to accept all. And then
25:54:01let's come right over here to our data
25:54:03engineering video one. And now we have
25:54:06this S3 cleaned silver. So our bronze to
25:54:09silver transformation is complete. This
25:54:12is all we needed to do in order to
25:54:14transform our data. And now we have our
25:54:17raw data. And let's come actually back
25:54:19to our catalog. And we'll just take a
25:54:21look at this. We can get rid of our
25:54:22genie code real quick. So we're going to
25:54:24come over here. So our raw data is still
25:54:27going to be raw. Let's go ahead and run
25:54:28this. This is our bronze level, right?
25:54:31We still have the raw data. We still
25:54:33have the duplicates. But when we come
25:54:35over to our silver, this is now going to
25:54:37be our cleaned level. So, now that we
25:54:40have all of our transformations
25:54:41completed, we've taken it from bronze to
25:54:43silver. Now, we want to create our
25:54:45silver to gold transformations as well.
25:54:48Let's come back to our workspace and
25:54:50we'll come down here to the silver to
25:54:52gold, which is going to still be right
25:54:54up here for us. Now, let's give it a use
25:54:55case, right? We could use this table
25:54:58just as it raw and we could hit off of
25:55:00it and we could build dashboards and all
25:55:02sorts of things. Sometimes we want to
25:55:03track certain KPIs or certain things
25:55:05that you can't just get from the raw
25:55:06data. So I'm just going to give it a
25:55:08simple use case. Let it write it out and
25:55:10we'll create our silver to gold
25:55:11transformation. So let's come right on
25:55:13here. I'm going to say that I want to
25:55:14know the best day of the week that
25:55:16people are clicking on certain ads. And
25:55:18we're going to see what it creates for
25:55:20us. So, uh, I want to create a new table
25:55:25called insights
25:55:27gold. And I want it to show me the best
25:55:31days of the week and what ads people
25:55:35clicked on the most. And let's run this
25:55:37and just see what it does. All right, so
25:55:39it went and did a lot of work for us. It
25:55:41did not take long. This is maybe 15
25:55:43seconds. It's doing some group buys on
25:55:46uh some different columns and it's
25:55:48getting some counts for us on different
25:55:50signups and referral sources. Let's go
25:55:53ahead and allow it to run this and let's
25:55:55see what it does. Now, it's given us a
25:55:57few things as far as outputs. One, this
25:56:00first one is extracting the day of the
25:56:01week and analyzing signup patterns. So,
25:56:03Thursday, Tuesday, Monday, and it's
25:56:05giving us kind of the day of the week
25:56:07when we had the most signups. And then
25:56:09if we come down here, we also have
25:56:10another one where we're getting the
25:56:12referral source, basically social media,
25:56:14organic referral, Google ads or partner,
25:56:16the total clicks and the countries
25:56:18reached. And if we go down here, we have
25:56:21this last table, but it hasn't been run
25:56:22yet because this is actually creating
25:56:24our table. And so this one should be
25:56:26really interesting, but let's actually
25:56:28stop it really quick. And then I'm going
25:56:30to accept and then run this as well. I
25:56:33just want to see what this one is. And
25:56:35so then we have uh day name, the
25:56:38referral source, signups, and unique
25:56:40countries. I think this is the one that
25:56:43I, you know, was kind of hoping for when
25:56:45I asked it to run it for us, but it gave
25:56:47us different options, which I like. Now,
25:56:49all we have to do is we have to get rid
25:56:51of this. And we're going to let that
25:56:52run. And so, let's accept that. And
25:56:55let's run this as well. Now, this
25:56:57display is literally just displaying uh
25:56:59right up here. So, we aren't actually
25:57:01reading this in. But uh let's come back
25:57:04into our catalog and let's go see if we
25:57:06have that gold table now. So now we have
25:57:08our video, we have our insights gold and
25:57:12let's just look at our sample data. And
25:57:14there we go. And so this would be like
25:57:16our gold table that we can now use. We
25:57:18now have some insights into our data.
25:57:20Now all we've done so far, if we come
25:57:23back in here, all we've done so far is
25:57:26we've just written code. We haven't
25:57:28necessarily created any type of
25:57:30pipeline. And so now this is the part of
25:57:32the video where we're going to get into
25:57:34building an actual pipeline. And I did
25:57:35it this way very specifically. This is
25:57:37how I tend to write my code. I come into
25:57:39a notebook. I write out my code. And
25:57:41then I'm like, okay, this is looking
25:57:43good. Let me now go create my pipeline.
25:57:46So let's come right over here. We're
25:57:47going to come down to our runs. And
25:57:50there's this thing right here that says
25:57:51ETL pipeline. Now let's get rid of this.
25:57:54We also have this right here, which is
25:57:57kind of what we're going to cover a lot
25:57:58in the next lesson, but I want to talk
25:58:00you through really quickly while we're
25:58:01here. The difference. Now, we created
25:58:04two separate notebooks. One from bronze
25:58:06to silver and one from silver to gold.
25:58:07Now, sometimes with simpler pipelines
25:58:09like the one we just created, it could
25:58:11be totally fine to just come in here,
25:58:13create a job, and say, "Do this one and
25:58:15then do this one." Right? That's all
25:58:17we're doing. We put it on a schedule or
25:58:18we can create uh, you know, a different
25:58:20trigger for that. And we'll look at that
25:58:21in the next lesson. But if you have a
25:58:23more complex pipeline, you're typically
25:58:26going to want to use this right here,
25:58:27which is our ETL pipeline. Let's go
25:58:29ahead and click in on this ETL pipeline.
25:58:31And let's come down here to start with
25:58:33an empty file. Now, you can start with
25:58:36sample code in SQL, sample code in
25:58:38Python, or if you have ones that you've
25:58:40already done, you can do that. We don't
25:58:42have anything, and I don't really want
25:58:44to kind of explain all of the sample
25:58:46code that they're going to be creating.
25:58:47Let's just start with an empty file.
25:58:49Now, we need to specify the language
25:58:51that we're using. And this is very
25:58:53important because once you create it,
25:58:55that's kind of the one that you're going
25:58:56to stick with. We're going to use
25:58:58Python. And it's just asking for a
25:59:00folder path. And so, we'll keep that.
25:59:01And we'll say yes. And now what we have
25:59:04looks very similar, right? We have these
25:59:07uh kind of some notebooks on the left.
25:59:08Then we can write our code right here.
25:59:10It looks very similar. But there is a
25:59:12big difference between running something
25:59:14in a notebook like we were in our
25:59:16workspace before and running something
25:59:18in an ETL pipeline. When you're just
25:59:20running your code, it's running the code
25:59:22as is. It's pretty simple. And if you
25:59:24did what we said earlier, which is you
25:59:26literally just take that notebook, you
25:59:27put it into a job, and you say run this
25:59:29and then run this, it's literally just
25:59:30going to take your code and run it. The
25:59:32issue with that though is it's not going
25:59:33to have any built-in data quality
25:59:35checks. we're going to have to manage
25:59:36basically all of the logic ourselves and
25:59:39it's not going to handle any lineage
25:59:40tracking or dependencies within your
25:59:42code. Now, this is where ETL pipelines
25:59:44come into play. An ETL pipeline is going
25:59:46to have things built into it like
25:59:47automatic incremental processing,
25:59:49built-in data quality checks, failure
25:59:51recovery, things like that that are
25:59:53extremely useful when you have really
25:59:56complex pipelines, which we aren't doing
25:59:58in this lesson. of course is very
25:59:59simple, but you have to think, you know,
26:00:01if you're creating a real ETL pipeline
26:00:03with a lot of dependencies, a lot of
26:00:04complexities to it. You absolutely are
26:00:06going to want to come in here. Now, when
26:00:08we write this out, we can't just write
26:00:10it as our regular code. And we can
26:00:12actually do that. Let's come back and
26:00:15let's go to all of our files. Let's go
26:00:17to bronze to silver transformation.
26:00:19We're going to move this just so we can
26:00:21visually see it. We're going to put this
26:00:23in our transformations. Then we're also
26:00:24going to take our silver to gold and
26:00:26we're going to move this to our
26:00:27transformations as well. So we're going
26:00:29to put this all in one place. And so now
26:00:31we have this silver to gold and we have
26:00:33the bronze. We don't actually need this
26:00:35uh file anymore. So we could just get
26:00:38rid of this. Now your UI might look
26:00:40slightly different. That's just because
26:00:41data bricks is always updating things,
26:00:43but you should still be able to follow
26:00:45along. But let's go ahead and this is
26:00:47our code. It's exactly how we wrote it
26:00:50before. Let's try to run this pipeline.
26:00:52It's going to try to run this and it
26:00:54should try to run that too. Let's just
26:00:55go ahead and run it and see what
26:00:56happens. All right, so we got this error
26:00:58down here that says pipelines are
26:01:00expected to have at least one table
26:01:01defined but no tables were found in your
26:01:04pipeline which might seem very
26:01:06counterintuitive because you're like
26:01:08we've created different data frames.
26:01:09We've been working with tables. So it
26:01:11should understand what it's doing. Now
26:01:13it is actually rewriting the code as we
26:01:15go. I think it's identified uh the issue
26:01:17already. And let me explain this even
26:01:19though uh it's starting to write it out
26:01:22already for us which is awesome. Thank
26:01:23you Genie Code. But here's what's
26:01:25happening when you're running code just
26:01:27in a notebook. It's just going line by
26:01:29line and running the code. But within
26:01:31this ETL process and just ignore that
26:01:34for a second cuz I'm just going to let
26:01:35it run within this ETL process. What
26:01:37it's using is something called an STP
26:01:39which is a Spark declarative pipeline.
26:01:42This is just a different construct and a
26:01:44different framework within the ETL
26:01:46pipeline. And so what it actually needs
26:01:48is something called a materialized view.
26:01:50It needs to kind of look at what the
26:01:51output is going to be or supposed to be.
26:01:53It's not just blindly running your code
26:01:55for you. It's doing a lot of heavy
26:01:57lifting with data quality checks and all
26:01:59these different things. Now it just went
26:02:00through uh and it fixed it for us. It is
26:02:03basically the same code and let's come
26:02:06up but it's creating these materialized
26:02:08views. So we have dp domaterialized view
26:02:11and it's kind of naming it and giving a
26:02:12little comment on what it is. It's doing
26:02:14the work for us and it's creating
26:02:16another materialized view where we use
26:02:18this insights goal and it's actually
26:02:20putting it all into one which is fine if
26:02:23that's what we want to do with uh this
26:02:25pipeline. But let's go ahead and accept
26:02:26this and let's try running this pipeline
26:02:29again. So now we have a little bit more
26:02:31information. We can come right down here
26:02:32and we can see it was trying to create
26:02:34these different materialized views and
26:02:36it was working. And so now this whole
26:02:39thing has run successfully. Let's
26:02:40actually rename this really quick. We're
26:02:42going to do bronze. Uh, I need to spell
26:02:44bronze, right? Bronze to silver to gold
26:02:48ETL pipeline. And let's save it like
26:02:50that. And we come back over here. We can
26:02:52go to our jobs and pipelines. We now
26:02:55have this pipeline right here. Of
26:02:57course, uh, it failed, but now it's
26:02:59running and it's working successfully.
26:03:00But now we have this pipeline that we
26:03:03have stored and we can actually start
26:03:04using this in, you know, automations
26:03:06where we can orchestrate these
26:03:08pipelines. Right here it says
26:03:09orchestrate notebooks, jobs, queries,
26:03:11and more. And there is a lot to that,
26:03:12and that's what we're covering in the
26:03:14next lesson. But if we open this up, we
26:03:16can actually see what's happening under
26:03:18the hood. We can see these are
26:03:19connected. We're doing this one and then
26:03:21this one. And we can see how it's
26:03:23running. And so there's a lot of things
26:03:26that this ETL pipeline is going to
26:03:27handle for us that we don't even have to
26:03:29worry about. That really is one of the
26:03:30biggest advantages of using an ETL
26:03:32pipeline instead of just running your
26:03:34notebooks. Although again, there are
26:03:36some advantages to just running your
26:03:38notebooks as is if it's a little bit of
26:03:40a simpler pipeline. I really hope you're
26:03:42able to follow along with this lesson
26:03:44because this is really cool stuff. You
26:03:46can also just come into here and we can
26:03:48create an ETL pipeline and you can
26:03:50create a pipeline with AI. So we can
26:03:52literally just come here and we can type
26:03:54in exactly what we want our code to look
26:03:56like and do within our data and it can
26:03:58build that out instead of starting with
26:03:59a notebook and then creating our ETL
26:04:01pipeline. you can just come right in
26:04:03here and start doing that process here.
26:04:05I will say though my personal workflow
26:04:07because I'm usually not doing super
26:04:09complex pipelines that are involving,
26:04:10you know, ton of different dependency
26:04:12chains and all these different things is
26:04:14I tend to like writing my code in
26:04:16notebooks. That's just what I'm used to.
26:04:18Uh but there are going to be lots of use
26:04:20cases where you're going to need to come
26:04:21in here and you can just start here
26:04:23instead of starting with a notebook.
26:04:25Now, in the last two lessons, we've been
26:04:26building out our ETL pipeline. We've
26:04:28been writing all of our code and getting
26:04:29everything set up. But once we actually
26:04:31have everything set up, then we need to
26:04:33automate this process so that we don't
26:04:35have to manually go in and run the code
26:04:37ourselves. Luckily, data bicks has this
26:04:39already built out for us. It is called a
26:04:41job. And so, we're going to jump into
26:04:42data bicks. We're going to create our
26:04:44own custom job and we're going to see
26:04:45all the small things that you need to do
26:04:47in order to create this automation. Now,
26:04:49in our last lesson, we built out this
26:04:51bronze to silver to gold ETL pipeline.
26:04:54And we're basically creating two
26:04:55separate tables. This S3_clean silver
26:04:58and then this insights gold. And that is
26:05:00our silver and our gold tables after
26:05:02they're transformed. And we find our
26:05:04business insights. Now, just for
26:05:06demonstration purposes, I also just kept
26:05:08our regular code in here as well. We
26:05:10have this bronze to silver. Then we have
26:05:12another notebook for silver to gold.
26:05:14Now, these are just regular notebooks
26:05:16and data bricks, but I do want to show
26:05:18you how you can use this within a job as
26:05:21well. But we have this bronze to silver
26:05:23transformation. You can see it in a
26:05:24pipeline. And then if we just go to our
26:05:26bronze to silver, this is just a regular
26:05:28notebook. Now, in order to create our
26:05:30job, let's come right down here. We're
26:05:32going to go to runs. We're going to come
26:05:34over to job. And this is orchestrate
26:05:35notebooks, pipelines, queries, and more.
26:05:39So, let's come in here. Now, this is a
26:05:41new UI for us. And what you can do here
26:05:43is you can orchestrate the different
26:05:45steps that you want within your job. If
26:05:47we click right down here, we can see all
26:05:50the things that we can do. We can create
26:05:52ingestion pipelines or we can use
26:05:53existing ones. We can come down here and
26:05:56we can run notebooks, Python files, SQL
26:05:58queries, SQL files and we have some more
26:06:00advanced things right down here like if
26:06:03else conditions or you can uh create
26:06:06triggers from another job and then we
26:06:08also have this ingestion and
26:06:10transformation and these are really
26:06:12useful because if you have an ingestion
26:06:13pipeline, an ETL pipeline or a database
26:06:16table sync then you can just use those
26:06:18that you've already created. Now we've
26:06:20created an ETL pipeline. Let's go ahead
26:06:22and click on this ETL pipeline. We're
26:06:24going to come down here and we're going
26:06:26to click on this bronze to silver gold
26:06:28ETL pipeline. Now, I'm just going to
26:06:30call this uh bronze to silver to gold.
26:06:35Keep it simple. And all we would need to
26:06:37do is create this task. Now, of course,
26:06:40that would be a little too simple,
26:06:42right? But this is as simple as it can
26:06:44get for any type of pipeline
26:06:46orchestration that you're trying to do.
26:06:47Oftent times when I'm creating entire
26:06:50pipelines and there's a lot of different
26:06:51steps to it, I package everything into
26:06:54an ETL pipeline and then I just place it
26:06:56in here. And then what I'll do is I'll
26:06:58come over here to schedules and
26:07:00triggers. Now we'll look at that in just
26:07:02a second really quick. We can also
26:07:04trigger a full refresh on this pipeline.
26:07:06So we can click on this. We can also add
26:07:08notifications if you want to send this
26:07:10notification when it kicks off or when
26:07:12it finishes. We can also look at
26:07:15retries. Now, this is really important
26:07:17because sometimes you are going to have
26:07:18things that fail just for a various
26:07:20number of reasons. Maybe you're trying
26:07:21to run this, but the data hasn't all
26:07:23imported yet, and so you're trying to
26:07:25run this transformation, but there's
26:07:26some connection issue, and that caused
26:07:28it to fail. You'd want to retry maybe an
26:07:31hour later or on a different day. You
26:07:33would want to attempt to try this. And
26:07:36so, you can come in here and you can
26:07:38say, "Okay, I want to try this a ton of
26:07:40times. Let's try it 30 total times." And
26:07:43every single time, we're going to wait
26:07:45maybe 30 or 40 minutes between each try
26:07:48and then it'll keep trying until it is
26:07:50successful. Again, with this, you can
26:07:52notify yourself and make sure that you
26:07:54know what's happening, especially if
26:07:55this is a really important pipeline
26:07:57within your company. It is important to
26:07:59have these things set up so you don't
26:08:01have to manually go in there and see it
26:08:03failed, you know, last night and just
26:08:05never got a notification. It never tried
26:08:07again. So, this would absolutely be
26:08:08something that you'd want to do. And
26:08:10then you have metric thresholds. You can
26:08:12set these, especially for something like
26:08:14a run duration. If you know this should
26:08:16take five minutes at most, you can set a
26:08:18timeout threshold or a warning threshold
26:08:20at maybe 30 minutes so that it isn't
26:08:22just going to keep running because
26:08:23sometimes it gets stuck in these loops
26:08:25and it keeps trying and it's going to
26:08:27run forever and it's going to cost a lot
26:08:28of money and you don't want that to
26:08:30happen. So, these are all really
26:08:31important things to think about when you
26:08:33are actually creating these jobs. Now,
26:08:35let's come back here to schedules and
26:08:37triggers for something like this. When
26:08:40you've done almost all the work in an
26:08:41ETL pipeline, you are going to want to
26:08:43schedule or trigger this most of the
26:08:45time. Now, for something like this
26:08:47pipeline, what we've done is we've
26:08:49extracted data out of an S3 bucket. What
26:08:51we would want to do is probably set a
26:08:53trigger for this. Now, what we need to
26:08:54do is we need to create this task first.
26:08:57So, that is saved in there. And then
26:08:59let's say this is our entire job. It's a
26:09:01very simple one. But now we can come in
26:09:03here and we can add a trigger. There are
26:09:05several different types of triggers.
26:09:07When we have a schedule, which is as
26:09:09simple as it sounds, we are just going
26:09:11to schedule this. Right now, it'll be
26:09:13active. You can pause it. We're just
26:09:15going to schedule this. And we'll say
26:09:17every one week. And so, every one week,
26:09:20we're going to save this. And this is
26:09:22going to run every week. So, that's
26:09:24super simple. Now, let's delete this.
26:09:26And let's add another trigger. We can
26:09:28also schedule it. We can go a little bit
26:09:30more advanced. And we can schedule it at
26:09:32a very specific day and time. Now, this
26:09:35is what I usually do because there are
26:09:37certain cadences and timing to things
26:09:39that I really like. For example, at a
26:09:41previous job that I used to work at, we
26:09:43wanted the data to be as fresh as
26:09:45possible because we actually had it
26:09:46refresh often, like every 10 minutes.
26:09:48And so, what we were doing was we were
26:09:50trying to run it as soon as we could in
26:09:52the morning to where it would still run,
26:09:54but it would give us the freshest set of
26:09:56data by about 8:30 in the morning. So,
26:09:58we would kick off this job at like 7:45
26:10:00so that the freshest data would be
26:10:02available by 8:30. This is more
26:10:04advanced. You don't have to do this, but
26:10:06this is a really useful thing to do. The
26:10:10next thing that you can do or the next
26:10:11type of trigger is a file arrival. So,
26:10:14if we click on file arrival, we're going
26:10:16to say when a file arrives at this
26:10:18location, kick off this job and run
26:10:21everything within it. Now, for our
26:10:22process, this would be like our S3
26:10:24bucket. If and we can go and look at our
26:10:26S3 bucket. If a new file gets dropped in
26:10:29here or this gets updated, then we may
26:10:31trigger this job and it will run. And of
26:10:34course, we have advanced settings as
26:10:36well where we can wait a minimum time
26:10:38between triggers because what if you're
26:10:40uploading a lot of documents at the same
26:10:41time? You don't want it to trigger 20
26:10:43times because you just dropped 20
26:10:45different files in there one at a time.
26:10:46You'd want to wait for all these files
26:10:48to get in there. So, that is absolutely
26:10:50an option. And if we go back, we also
26:10:53have a table update. So this would
26:10:56trigger when new data is updated on a
26:10:59table. Now for our use case, this may
26:11:01work because we have S3 data. We're
26:11:03bringing it into our bronze table. So I
26:11:06can come in here and I can say when this
26:11:07table and I would just specify that
26:11:09table name that we've been using. when
26:11:11this bronze table gets updated from that
26:11:14S3 bucket then kick off this job which
26:11:17of course this ETL pipeline takes that
26:11:20bronze data we transform all the data we
26:11:22create our gold tables and then we have
26:11:24all that data sitting there so this
26:11:25might be a really good use case we have
26:11:27some advanced options down here minimum
26:11:29time between triggers and wait after
26:11:31last change just like we did before
26:11:33because sometimes data gets updated
26:11:35continuously and so it might trigger it
26:11:37many times these are things that you
26:11:39should test and try out within in your
26:11:41pipelines just to make sure you get them
26:11:42right. Now, let's cancel out of this and
26:11:45let's actually get rid of this entirely.
26:11:48Let's actually come here and we're going
26:11:50to go back to our runs or sorry, back to
26:11:54our jobs. And I want to show you one
26:11:56more thing within here that might be
26:11:58really useful. Now, we just kind came
26:12:00down here and we pulled in uh this ETL
26:12:02pipeline, but let's actually pull in and
26:12:04run a notebook. So, we're going to
26:12:06specify our notebook. We're just going
26:12:08to do this is our bronze to silver and
26:12:10this is a notebook. It's within our
26:12:12workspace, not a git provider. And let's
26:12:14select our notebook. So, we're going to
26:12:16come in here. We're going to do bronze
26:12:17to silver. Let's confirm this. And
26:12:20you'll notice we have a lot of different
26:12:22options in here. Some similar, right? We
26:12:24have retries, we have notifications, and
26:12:26we have metric thresholds, but we also
26:12:28have parameters. These are parameters
26:12:29that you can pass down to the task.
26:12:31Because this is just a notebook, it
26:12:33doesn't have all that built-in stuff
26:12:35that we were talking about in the last
26:12:36lesson within the ETL pipeline. So you
26:12:38do need to configure this a little bit
26:12:40more within a job. So we can add these
26:12:43parameters where we create these kind of
26:12:45key value pairs that we pass into uh a
26:12:47notebook. But let's come in here. Let's
26:12:50create this task. And now we're going to
26:12:52add in another task. So let's come here.
26:12:54We're going to add in another notebook.
26:12:56And this is going to be
26:12:59our silver to gold. Now, these two
26:13:03tasks, and let's actually name this
26:13:06these two tasks that we've created,
26:13:07these two notebooks do the exact same
26:13:09thing as our pipeline. But I wanted to
26:13:12show you this because it does give us
26:13:14some more information when we're
26:13:15actually building out these jobs. So,
26:13:17we've specified our path. We have our
26:13:19computer serverless, but now we have
26:13:20something called a dependency or a
26:13:22dependency chain. This right here, this
26:13:25line is a dependency with what we have
26:13:28right now. This silver to gold is
26:13:30completely dependent on this bronze to
26:13:32silver. Which means if we get this data
26:13:35in and this bronze to silver does not
26:13:37run correctly, then this silver to gold
26:13:40is never going to run. And in this use
26:13:42case, that's perfectly fine because this
26:13:45relies heavily on this bronze to silver.
26:13:47But there are going to be use cases
26:13:49where that is not the case where we
26:13:51would not want that to be, you know, a
26:13:53dependency. We wouldn't have to rely on
26:13:55it. Or we also have an option right down
26:13:57here to run if dependencies. And we have
26:14:00a lot of different options. So right
26:14:02now, all succeeded means this has to run
26:14:05properly in order for this to run. But
26:14:08there are going to be cases when you
26:14:10create these chains or these dependency
26:14:12chains where you're like, it doesn't
26:14:14matter if this one runs. We just want it
26:14:16to run after this one runs, whether it
26:14:18fails or not. And so for that one, you
26:14:20can come in here and say at least one
26:14:22succeeded, none failed, all are done, at
26:14:25least one failed, or all failed. It
26:14:28doesn't matter. You can specify
26:14:30whichever option you need. For us, we
26:14:32would want to keep this all succeeded
26:14:34because if this one runs, we don't
26:14:36actually create the silver tables that
26:14:39are needed in order to run this one. So
26:14:41that is pretty important. We can come
26:14:42down here and we can create this task.
26:14:45And now we have this job that we've
26:14:48created and we can run it now or of
26:14:50course we could add in our trigger. Now
26:14:53typically with something like this it
26:14:55could go either way. You could have it
26:14:56on file arrival table update or a
26:14:58schedule. It really is just very
26:15:00dependent on your workflow and how you
26:15:02want this to trigger. For most of these
26:15:04you're going to have some type of
26:15:05trigger. Let's just set it on a
26:15:07schedule. And let's go to advanced. And
26:15:09we're going to set this for every week.
26:15:12And let's do this on a Monday. and let's
26:15:14do it at 7:45 because that's when I used
26:15:16to do our some ones at uh a previous
26:15:19job. So, I'm going to do at 7:45 every
26:15:21morning. Let's go ahead and schedule
26:15:23this. And now we've updated this job.
26:15:25And now we can also rename this. I'm
26:15:28going to call this our silver to gold
26:15:32job. So now if we go back to our jobs
26:15:34and pipelines, we have our silver to
26:15:37gold job right here. This was the
26:15:40pipeline that we built out in the last
26:15:42lesson. And this is going to be
26:15:44orchestrated and scheduled to run this
26:15:46pipeline. Well, actually we used uh the
26:15:48notebooks instead of the pipeline for
26:15:50that last example, but we're going to be
26:15:51running that code to actually create and
26:15:53update those tables. So that is how we
26:15:55create a job in data bricks. This is
26:15:58extremely extremely useful. Again, like
26:16:00we did just a little bit ago for our
26:16:02silver to gold job. And let's go into
26:16:05the tasks. If it's a really small
26:16:07transformation and maybe it's just for
26:16:09me, I'll just do it like this where I
26:16:11just have the notebooks. But if it's a
26:16:13larger transformation, especially if
26:16:15there's a lot of dependencies, if
26:16:16there's a lot of complexity, I will use
26:16:18an ETL pipeline. So get in here, mess
26:16:21around with this, try this out because
26:16:23this is super fun to play around with
26:16:24and kind of get all those dependency
26:16:26chains going and getting the ETL
26:16:28pipelines where they're triggering off
26:16:29of each other or when a file is updated.
26:16:31This is really cool stuff to mess around
26:16:33with and is awesome to use within data
26:16:35bricks. Now, if you haven't been
26:16:36following along in the past three videos
26:16:38in this series, we've covered several
26:16:40things. One, we've just learned about
26:16:42ingesting data. Then after that, we
26:16:44looked at ETL pipelines and then we
26:16:46looked at creating a job to orchestrate
26:16:48all these things and to kind of automate
26:16:50the process. In this video, we're going
26:16:52to be putting all of that together into
26:16:54one. We're going to add some things that
26:16:55we didn't cover in previous lessons to
26:16:57make it a little bit more advanced, but
26:16:59it's going to cover a lot of the same
26:17:01concepts. Let's not waste any time.
26:17:03Let's jump right onto my screen and get
26:17:04started. Now, before we actually jump
26:17:06into data bricks, what we're going to be
26:17:07working with is that same S3 bucket that
26:17:10we created earlier, but I created this
26:17:14transactions folder, and that is going
26:17:16to be an important piece of this
26:17:17process. It's something that we touched
26:17:19on in a previous video, but we actually
26:17:22going to be doing it in this lesson. So,
26:17:24we use this users_y.csv
26:17:26CSV in this bucket. But inside of this
26:17:29transactions, we have three separate
26:17:30transaction files. And we'll actually be
26:17:33adding another one later on to show how
26:17:35the entire process works. So, I'm going
26:17:38to have these and the other file down in
26:17:41the description. You can just download
26:17:42those from GitHub, but we will need
26:17:45those. So, we're just going to start off
26:17:46with these three, the 16, 1_13, and
26:17:501_20. Now, really quick, just to show
26:17:53you what data we're working with, this
26:17:55is our data. Let me actually zoom in
26:17:57just a little bit. Uh, the data itself
26:18:01is not as important for this specific
26:18:03project just because we're more focused
26:18:05on the process of building the pipeline
26:18:07within data bricks, but within the
26:18:10project, we will be cleaning this data a
26:18:12little bit because this is just a
26:18:14horrible column. Uh I think whoever you
26:18:16know was collecting this data just left
26:18:18this free text or something for people
26:18:20to just put whatever they wanted in
26:18:21there. Uh not a good not a good system
26:18:25but that is the kind of data that we're
26:18:27going to be working with. So let's come
26:18:28up here. Let's get out of this. We don't
26:18:31need to save it. Now let's come up to
26:18:34our data bricks. Now in our previous
26:18:36lesson this is what we built. We built
26:18:38this uh pipeline right here. Bronze to
26:18:40silver to gold ETL pipeline. And then in
26:18:43the very last lesson, we created the
26:18:45silver to gold job which basically
26:18:47scheduled this and automated this and it
26:18:50ran successfully and everything was
26:18:51great. Now what we're going to be doing
26:18:53is we're going to be doing it in a
26:18:55similar fashion but covering some new
26:18:57things. All you need to do and I
26:18:59actually have another tab for this cuz I
26:19:01don't want to have to keep going back
26:19:02and forth when we're building this out.
26:19:04But I created this end toend schema
26:19:06within our data engineering catalog. You
26:19:09don't have to do this. You can put this
26:19:11wherever you want. I just did. This is
26:19:13kind of where we'll be building things
26:19:15out. So, I'll just come back to this as
26:19:17we start adding in new tables, as we
26:19:18start creating this stuff. I'm going to
26:19:21come back to that. Now, this is where
26:19:23we'll be doing a lot of our work on this
26:19:25uh tab right here. So, let's come over
26:19:28to data ingestion. Let's go over to our
26:19:30Amazon S3. Now, if you haven't already,
26:19:34in a previous lesson, I think the second
26:19:36video, we connected to an S3 bucket. So,
26:19:39if you don't know how to do that, then
26:19:41come over here and do this. Now, we used
26:19:43it for the one time because all we used
26:19:45was this users dirty.csv.
26:19:48But in order to schedule this data
26:19:51ingestion, we're going to use a folder.
26:19:53So, we have this transactions folder
26:19:55right here. So, we're going to click on
26:19:57this. We're going to click on
26:19:58transactions and we have those three
26:20:00separate files in there. And we can
26:20:03schedule when we want to bring those in.
26:20:05Now, we can be very specific or pretty
26:20:07laid-back. Uh, so for example, if we
26:20:10want to do, you know, once a day, we can
26:20:12specify what time of day we want that.
26:20:14And that's similar to a job. So it's not
26:20:16that crazy. Now, what we're going to be
26:20:18actually doing is we're going to
26:20:19schedule this for basically every 30
26:20:21minutes. And what we're going to do is
26:20:22we're going to build this entire thing
26:20:23out. And what our trigger is going to be
26:20:26inside of our job is when a table gets
26:20:29updated. So then we're going to drop a
26:20:31file in our S3 bucket. And when this
26:20:33brings it in at that 30 minute point,
26:20:35it's then going to refresh, kick off the
26:20:37job, which runs our ETL pipeline. We
26:20:39should be able to do all this within 30
26:20:41minutes for sure. So, I'm going to say
26:20:43every 30 minutes, and we'll just set it
26:20:47at 0 minutes past the hour, which means
26:20:50at basically the top of the hour. Now,
26:20:53this is my time zone, but you can set it
26:20:55to whatever time zone you want. Now,
26:20:57let's go ahead and preview this table.
26:20:59It's going to start up our compute. Then
26:21:01it's going to give us uh basically what
26:21:04we need in order to create this table,
26:21:06which is our preview, and then where we
26:21:08want to place it along with the table
26:21:10name. Now, an important thing to note
26:21:12from just those three files is there's
26:21:14only 50 rows of data in each one. So, if
26:21:17we come down here, we got all the way up
26:21:19to 100. So, we at least know two of
26:21:21those files are coming in just from this
26:21:22preview. We're going to keep this as the
26:21:25transactions, but for the schema, we're
26:21:28going to add the end to end, which is
26:21:30the custom one that we created for this
26:21:32project. So, we have transactions right
26:21:34here. Let's go ahead and create the
26:21:36streaming table. So, now this table has
26:21:38been created. Let's just look at a
26:21:40sample of this data. It should show us
26:21:43enough to be confident all three got in.
26:21:45But then we can just also run a query
26:21:47and that's perfectly fine. In fact,
26:21:49instead of waiting, uh, never mind, we
26:21:51got them all in. I was going to say we
26:21:53don't have to wait on this. We could
26:21:54just run a query in like a notebook or a
26:21:57SQL editor, but we have all 150. So
26:21:59that's all three files. So now that we
26:22:01know we have all three of our files in
26:22:04cuz it's 50 each. It's going to be 150
26:22:06rows. Now that we know those are in, we
26:22:08can start building things out. Now that
26:22:10we know that it's all in there, what we
26:22:13can do is let's come over here to our
26:22:15jobs and pipelines. Now this is where we
26:22:17were before. We only had these two
26:22:20things. We had a pipeline and now we had
26:22:22a job. And now we have another pipeline.
26:22:24And we didn't build this ourselves. This
26:22:27was built automatically. And if we come
26:22:29in here, we can get a little bit of
26:22:31information on this. This is our
26:22:34streaming pipeline that we created to
26:22:36put into this endto-end transaction. So
26:22:39this is that streaming table that we
26:22:42created. And so we don't have to
26:22:43technically manage this. It's going to
26:22:45be managed by data bricks itself. And so
26:22:48this is just something to note that when
26:22:49we did that, we did create its own
26:22:52pipeline for this. Now what we need to
26:22:55do is we need to create an ETL pipeline.
26:22:58So let's come in here. We're going to
26:22:59click on the ETL pipeline. This new UI
26:23:02pops up right away. We don't have the
26:23:04options that we had before in previous
26:23:05lessons. Um but now what we're going to
26:23:08do is we're going to start building this
26:23:09out with Genie code. Now, I could
26:23:13absolutely just write all this out and
26:23:14this would be like an hour and a half
26:23:15video, or we can have Genie Code write
26:23:18it out, which I highly recommend trying
26:23:20it out and starting to use these tools
26:23:22because they really speed up your work.
26:23:24And if you already know how to program,
26:23:25if you know how to code, this is going
26:23:27to be a huge boost to your productivity.
26:23:30And so, what we're now going to do is
26:23:32use Genie Code right down here.
26:23:33Basically, tell it what we want to
26:23:35build. And we're going to do a few
26:23:37things. One, we want to build that
26:23:39bronze to silver, which is basically our
26:23:41raw data, which is that transactions
26:23:44table, to a silver table, which is where
26:23:46the data is cleaned, to then a gold
26:23:48table, which is what we would use for
26:23:50like a production level uh product or
26:23:53production level analysis or whatever
26:23:54that might be. So, we can come in here
26:23:57and we can use that at and it is
26:23:59prompting us to do that. And if we come
26:24:01in and we can say data engineering.end
26:24:06end to end. And I'll just put it like
26:24:09that. So, it's looking kind of at that
26:24:11schema. I'm just going to say uh for the
26:24:14transactions
26:24:16table, I want to create a bronze to
26:24:21silver
26:24:22transformation on this raw data.
26:24:26I want you to clean this data set. I'm
26:24:30just going to leave it really open-ended
26:24:32just to see what it does. Maybe it
26:24:34catches something outside of that
26:24:35column. I don't think it will, but uh
26:24:38let's just see what it does. Then we are
26:24:41going to create a silver to gold
26:24:45transformation.
26:24:47And you can do this in the same notebook
26:24:49or separate notebooks. It may also do
26:24:52that for you with Genie Code, but you
26:24:54can be really specific and it's honestly
26:24:56pretty great at what it does. And I want
26:24:58to track daily transactions
26:25:03in that gold table. So I'm going to give
26:25:06it just this to work on. It's going to
26:25:08take that. It's going to kind of create
26:25:10its logic. It's going to start writing
26:25:11everything out. Um I have found this is
26:25:14not just me saying this. I genuinely
26:25:16love working in this system because
26:25:18Genieode is very good at understanding
26:25:20context and what you're trying to do and
26:25:22working with tables and just everything.
26:25:25And so we're going to let this run for
26:25:27just a little bit. I'm going to come
26:25:28back. We'll take a look at what it said
26:25:30and then we'll commit some code to start
26:25:32going on the CTL pipeline. All right, so
26:25:34it just finished. I haven't even really
26:25:36reviewed this cuz it only took, you
26:25:38know, 30 seconds, but it took a look at
26:25:40the data. Then it came down here and
26:25:42gave a proposed pipeline architecture.
26:25:45So here's what we have. We have our
26:25:47bronze layer, which is just going to be
26:25:49our transactions. py. And this is just
26:25:52going to read in the data as is. So,
26:25:54it's going to recreate basically the raw
26:25:56data, which I'm totally fine with. It's
26:25:58not a big deal. Then for our silver
26:26:00layer, we have the silver
26:26:01transactions_clean. It's going to trim
26:26:04the white space, standardize
26:26:05capitalization, remove duplicate spaces,
26:26:07filter out null transaction IDs or
26:26:09negative quantities and amounts, and add
26:26:12data quality expectations. I think these
26:26:15are all perfectly reasonable things to
26:26:17do. Then, we have our gold layer.
26:26:19There's going to be transformations gold
26:26:21daily transactions summary.py. py. So
26:26:24there's three different files that it's
26:26:25going to create and it's going to
26:26:27aggregate some of this data into kind of
26:26:29these metrics right here. I think this
26:26:31all looks great. If there was something
26:26:33I wanted to change, I would just tell
26:26:34it, hey, let's do this instead. So let's
26:26:36just say go for it. Start writing the
26:26:39code,
26:26:42my friend. It really is my friend at
26:26:44this point. I've been using it a lot. So
26:26:46let's let this run. Let's watch the code
26:26:48and then we will commit everything. And
26:26:50then it probably will prompt us to do
26:26:52some type of dry run to make sure that
26:26:54there aren't any errors that were just
26:26:56missing. And then we will run the entire
26:26:58thing and start automating this with a
26:27:00job as well. It's still writing. That
26:27:03was like 10 seconds. I stopped talking,
26:27:04but it's still writing everything. It's
26:27:06going to start organizing this. It's
26:27:08going to start creating our py files or
26:27:10just our Python files. I just am reading
26:27:13it as is. uh but it's creating our
26:27:15Python files and then it's going to
26:27:17start writing the code in which we are
26:27:19then going to review approve and then
26:27:21run. You can see these things starting
26:27:23to pop up. So we have our code, we have
26:27:25our diff or you know if we had code that
26:27:28it took out it would also say the minus.
26:27:30Uh but we're just creating code right
26:27:32now. And so right here it's saying all
26:27:36right do we want to try dry running this
26:27:38pipeline? Do we want to just see if it
26:27:40works? Um, and of course we're going to
26:27:43do that in a second, but I'm going to go
26:27:44to each one just to kind of see what
26:27:46it's doing. It looks like this is our
26:27:48gold, and we're just using a group by
26:27:51for this. Uh, let's just see what it did
26:27:54for the data cleaning. So, it looks like
26:27:56it is going to drop some stuff in here,
26:27:59but we are looking at some regax
26:28:01replace, which is great. Some trimming
26:28:03and proper case uh for a few other
26:28:05stuff. And this looks perfectly uh good
26:28:08to me. I have no problem with what it's
26:28:10doing. Again, this is all subject to be
26:28:13altered. If you want to change this or
26:28:15have it do other things or fix the code
26:28:17yourself, you absolutely can do that.
26:28:18Now, all we're going to do is we're just
26:28:20going to accept this. And so, we're
26:28:22going to allow this and it's going to
26:28:24run a dry. So, we'll accept review next.
26:28:27We'll accept review next. And accept. I
26:28:29didn't have I could have done that a
26:28:30different way, but now we're going to
26:28:31try dry running this pipeline. Now, what
26:28:34this does is it is not going to actually
26:28:37run through and run your code. It's
26:28:38doing a dry run. It's basically testing
26:28:40are there any big errors that we need to
26:28:42fix before you actually implement this
26:28:44into you know whatever process you're
26:28:46doing so that you don't have issues
26:28:48right off the bat. It's going to run for
26:28:50just a little bit and then it'll tell us
26:28:52if there's any big issues. Um oftent
26:28:54times if you've never done this before
26:28:56you shouldn't have any big issues but
26:28:58you could get issues like oh this table
26:29:01uh you don't have the permissions for
26:29:02this table. Maybe you wrote something
26:29:04incorrectly or in this case uh you know
26:29:06Genie code wrote something incorrectly
26:29:08that is not going to create the
26:29:09materialized view properly or you're
26:29:11pulling from a table that doesn't exist
26:29:13anymore. So there's lots of issues that
26:29:15could arise but let's let this run. It
26:29:17shouldn't take very long and just like
26:29:19that we did encounter a small issue. Um
26:29:22it's actually going to run. It'll
26:29:24probably fix this very easily. Um, I am
26:29:27not exactly sure what the issue is here.
26:29:30Just glancing at it, but it looks like
26:29:31it's fixing that code for all of it. And
26:29:35let's go ahead and just accept that. And
26:29:37let's try dry running this one more
26:29:39time. Now it looks like everything is
26:29:42running properly. And this is really
26:29:44good. So what we can now do is we can
26:29:48rename this. So it's going to give us
26:29:50some feedback on that. But I'm going to
26:29:51rename this. And I'm going to say this
26:29:54is our end toend
26:29:56uh ETL pipeline.
26:30:00And that's what we're going to name it.
26:30:02So we have our endto-end ETL pipeline.
26:30:05And with this, if we come back here,
26:30:08obviously nothing has changed, right?
26:30:10This was just a dry run that we did. Now
26:30:14what we can do is we can actually run
26:30:16this pipeline and it will run
26:30:18everything. It's going to do all the
26:30:20transformations, all the things that we
26:30:22would want it to do and we should and we
26:30:24will do that in a little bit. Now, what
26:30:26we want to do is we want to automate
26:30:28this process. All we have to do is we're
26:30:31going to come back here to not data
26:30:33ingestion into runs and let's get rid of
26:30:36this. Now, we're going to create a job
26:30:39for this. So, we're going to come in
26:30:41here and we're going to say we want our
26:30:43pipeline. And if we come in here, we
26:30:45have our endto-end ETL pipeline. That's
26:30:47the one we want. We're just going to
26:30:49call this um end to end ETL pipeline.
26:30:53Keep it simple. Now, what we're going to
26:30:54do, and you can always come in here and
26:30:56add notifications and retries and metric
26:30:58thresholds, which we covered in the last
26:31:00lesson. Now, we're going to create this
26:31:02task, but now we're going to add this
26:31:04trigger right here. Now, this trigger is
26:31:06going to be a table update. So what we
26:31:09want it to do is when new data is
26:31:12actually updated and brought into that
26:31:14table, we want this job to kick off so
26:31:16that it runs our entire ETL process to
26:31:19clean the data and put that new data
26:31:21into our, you know, new tables that
26:31:23we're creating. Now what we want to do
26:31:26is we want to say this table when this
26:31:28table gets updated. So let's come up
26:31:30here and we're just going to copy this
26:31:32name to the clipboard and we're going to
26:31:34put it right down in here. We could also
26:31:35have typed it out. Um, either one's
26:31:37fine, but I just wanted to copy it. So,
26:31:39when this table gets updated by our S3
26:31:43process, which we're running every 30
26:31:44minutes, this is going to kick off the
26:31:47ETL pipeline, right? It's going to kick
26:31:49off
26:31:51this right here. So, now that we have
26:31:54that job updated and created, let's go
26:31:57back to our jobs and pipelines. And now
26:31:59we have a few new things in here. So,
26:32:01right here we have our endtoend ETL
26:32:03pipeline. I should have named this job.
26:32:06Let's actually come in here really
26:32:07quick. I'm just going to come up here.
26:32:09I'm going to rename this. I'm going to
26:32:10say job to run N2 to end pipeline. And
26:32:16let's rename this. So, we have our
26:32:19pipeline. We have our job to run the ETL
26:32:21pipeline. And we have our transactions.
26:32:24That's going to run 30 minutes uh on the
26:32:26minute. It looks like um it may have
26:32:29already run before. No, I think we're
26:32:32good. No, it did. It's already run
26:32:34twice. I think that's just because of
26:32:36when I set it to the zero time perfectly
26:32:38fine. Um, but what we're going to do now
26:32:41is we are going to just check that this
26:32:45end to end ETL pipeline is working
26:32:47properly. It's going to create all of
26:32:49our tables. We're going to then write a
26:32:51query just to show that the data looks
26:32:53good. And then we'll go drop our extra
26:32:56file in there. And then we'll wait to
26:32:58have it update and the ETL pipeline
26:33:00bring in the data. Then our job is going
26:33:02to trigger. And then it'll run our ETL
26:33:05pipeline to bring in and clean that new
26:33:06data as well. So let us run this
26:33:09pipeline.
26:33:11This is going to take just a little bit
26:33:13to actually run and then we'll go check
26:33:16the data in just a little bit. All
26:33:18right. So this looks like it worked
26:33:20properly. We have completed completed
26:33:22and completed. Let us come up here and
26:33:26let's go back and let's refresh this.
26:33:30And it is possible that I put it in the
26:33:34wrong place and it totally is. I
26:33:37absolutely forgot to change that in the
26:33:38ETL pipeline. It is pointed at the
26:33:41workspace default. Let's actually go
26:33:43back and you know this happens. We're
26:33:47going to edit this pipeline. So it is
26:33:48our default catalog that caused this
26:33:51issue. We have our default catalog and
26:33:53default schema as workspace and then
26:33:56default. Um you can change this. You
26:33:59don't have to, but you absolutely can.
26:34:00You also, if I'm being honest, I should
26:34:02have fixed this myself or caught uh this
26:34:04right away. I like to be explicit when
26:34:07I'm, you know, writing to places. I
26:34:10don't like to have defaults like this.
26:34:11So, I should have had it specified right
26:34:13here where we're writing. It should have
26:34:15been like, uh, you know, data
26:34:16engineering.end to end dot and then the,
26:34:20uh, table name just to be more explicit.
26:34:22And we should have done this in
26:34:23basically all the Python files within
26:34:25the ETL pipeline. Totally fine though.
26:34:28Not a massive deal, just you know,
26:34:30something to think about. Now, if we
26:34:33come back to this catalog and we look at
26:34:36this, we can go to let's go to the
26:34:39silver transactions clean. This is going
26:34:42to be our cleaned data. Let's just go
26:34:44ahead and run this real quick so we can
26:34:46look at that sample data. So now this is
26:34:48our clean data. This product name looks
26:34:52much better. Uh looks really good. It
26:34:55did a few other really small things in
26:34:57here, but this is the main one that
26:34:58we're looking at. If we go back to the
26:35:00bronze transactions, uh, and look at our
26:35:03sample data. So, this is our bronze
26:35:05table.
26:35:07This looks terrible. So, obviously, it
26:35:09did a really good job data cleaning it.
26:35:11Uh, and then we'll look at the gold
26:35:13daily transactions summary.
26:35:18And this is looking at the transactions
26:35:20just grouping by and then looking at a
26:35:22lot of our data. And this is great for
26:35:24like a gold table uh that we're going to
26:35:26be using for you know some metrics or
26:35:29whatever we want to use it for. So all
26:35:31of this looks really good. Now in our
26:35:33silver transactions clean in our sample
26:35:36data again we at least in the sample we
26:35:40only have 100. Let's go and run this. So
26:35:43let's actually create a notebook with
26:35:45this and let's run this query. So now we
26:35:48can see we have 150 rows and let's add
26:35:52code
26:35:54and let's copy this and instead of the
26:35:57transactions clean we'll say
26:36:00uh let's go see what that table's
26:36:02called. It's not gold customers.
26:36:05I should have kept it up over here.
26:36:07Let's go back to our catalog. See when I
26:36:09start when I start messing with my
26:36:10systems I start getting messed up. It's
26:36:13gold daily transaction summary. I could
26:36:15have gotten copied it somewhere else.
26:36:17Uh, but I'm going to put it right here
26:36:19so that we can look at this. And what
26:36:21we're going to do now is we're going to
26:36:23go drop in that other file into the S3
26:36:26bucket so that we can see when it gets
26:36:29updated and to make sure that the new
26:36:31data gets in there and gets cleaned. So,
26:36:34let's come over here. We're going to
26:36:36upload and let's click on add files.
26:36:39Now, we're going to come here. We have
26:36:41that 127. That's the new one that we
26:36:43didn't have before. Let's upload this
26:36:45and put this into our S3 bucket. And now
26:36:48we have 06, 13, 20, and 27 all in this
26:36:52S3 bucket. So now what we're going to do
26:36:55is we're just going to I'm going to
26:36:57literally just let this wait. Let's come
26:36:59over here and right here, this is going
26:37:03to kick off in probably like five
26:37:05minutes or so. I'm just going to let it
26:37:06run. This is going to kick off and then
26:37:09you will see that this job to run the
26:37:12end time pipeline will automatically
26:37:14kick off as well once that table is
26:37:17updated. So let's just be patient. Let's
26:37:19just wait. I'm going to skip you ahead
26:37:21and you will see this running in just a
26:37:23little bit. All right. Now you can see
26:37:25that this is kicked off. Looks like it
26:37:28is running. When this process is
26:37:30finished, it is going to update that
26:37:32table with the new data which is going
26:37:34to trigger this job right here based on
26:37:36this table update on data
26:37:38engineering.toend.transactions.
26:37:42And that should start any second here.
26:37:44And it looks like that is working. We
26:37:46can see this one running. And since it
26:37:48is literally running this pipeline, we
26:37:50can also see that this one is going to
26:37:51start running as well. It's just
26:37:53spinning up the compute so that it can
26:37:55run properly. Let's go ahead and let
26:37:57this run and then we're going to see and
26:37:59check in our queries if everything
26:38:02actually went through properly. All
26:38:03right, it looks like this uh job is
26:38:06still spinning, but the pipeline was
26:38:08kicked off successfully. It looks like
26:38:10it ran with no issues, which is exactly
26:38:13what we want. And now this job is done.
26:38:15So now our entire process is complete.
26:38:18And it is going to keep doing that every
26:38:2130 minutes. uh every single 30 minutes
26:38:24from now until I stop the job or I stop
26:38:27this pipeline from running, it is going
26:38:29to kick this off. It's going to kick off
26:38:31the job. It's going to kick off the
26:38:33pipeline every 30 minutes. Of course,
26:38:35I'm going to stop that cuz that's nuts
26:38:36to keep running. But let's come back.
26:38:39Let's go to our workspace. And I think
26:38:41it's this one. Let's go take a look.
26:38:44Yeah. So, now we have 150 rows. Let's go
26:38:46ahead and run this. We should see 200
26:38:48rows of data.
26:38:51And there we go. And let's just make
26:38:53sure it's all cleaned properly in that
26:38:56product uh name. Looks great. And let's
26:39:00come down here and let's just make sure
26:39:01that this gets updated. We have 21 rows,
26:39:03but more than that, it's about the data
26:39:05because this is aggregated. So, um let's
26:39:08go ahead and run this as well. And we
26:39:10have 28 rows. That's just another week's
26:39:12worth of data. And these numbers are
26:39:15actually look uh basically the same, but
26:39:17we have this new uh week's worth of data
26:39:20in here that we didn't have before. So
26:39:22that is the entire NTN project. It
26:39:24really brings everything together that
26:39:26we've been working with in the past
26:39:28several lessons into one final project.
26:39:30And I hope you were able to follow
26:39:32along. If you didn't follow along, you
26:39:34just watch this video to the end. I
26:39:36highly recommend using the free edition.
26:39:38I will have a link in the description.
26:39:40You can try all this out completely for
26:39:42free. You don't even have to enter a
26:39:43debit card or credit card, which I love.
26:39:45So, you can just use this and it is an
26:39:47amazing platform to try out. I highly
26:39:50recommend it. But with that being said,
26:39:52thank you guys so much for watching. I
26:39:54hope you like this video. I hope you
26:39:55learned something in this entire series.
26:39:57If you did, be sure to like and
26:39:59subscribe. I'll see you in the next
26:40:01lesson.
26:40:03[music]
26:40:09>> [music]
26:40:14>> What's going on everybody? Welcome back
26:40:15to another video. Today we're going to
26:40:17be setting up and installing R and R
26:40:19Studio.
26:40:23[music]
26:40:25Now, this is our very first lesson in
26:40:27this series, so we're just going to be
26:40:28installing and getting R and R Studio
26:40:30set up. In future lessons, we're going
26:40:32to be diving into the basics of R. We'll
26:40:33be grouping, aggregating, visualizing,
26:40:35cleaning, and a ton of other things
26:40:37using a lot of different packages that
26:40:39are very popular within R. By the end of
26:40:41this series, you should feel very
26:40:42comfortable using R and R Studio. If you
26:40:44haven't checked it out already, I have a
26:40:46full course called R for data analysis
26:40:48over on analystbuilder.com. I will leave
26:40:50a link in the description as well as a
26:40:51coupon code if you want to take that
26:40:53full course. It covers more topics, has
26:40:54practice problems throughout the course,
26:40:56as well as goes more in depth, and has
26:40:57more difficult projects. With all that
26:40:59being said, I'm super excited to get
26:41:01started on this series with you. Let's
26:41:02jump on my screen and get started. All
26:41:04right. So, let's get started in
26:41:05downloading R Studio. Now, R Studio is
26:41:09created by a company called Posit. Now,
26:41:11R is the programming language, but R
26:41:15Studio is the interface in which so many
26:41:17people interact with R. And so, we're
26:41:20going to be using R Studio throughout
26:41:21this entire series. What we need to do
26:41:23is first install R. And then next, we
26:41:26need to install R Studio. R is going to
26:41:28be installed almost the exact same way
26:41:30on every system. And then for installing
26:41:32R Studio, it's going to autopop populate
26:41:34this for Windows because it recognizes
26:41:36Windows as my machine. But if it
26:41:38doesn't, if it gets the wrong one, you
26:41:40go down here for uh Linux, Mac, Windows,
26:41:43whichever one that you want. So, let's
26:41:45get started by installing R on our
26:41:48machine. I went ahead and deleted R and
26:41:50R Studio from my computer. So, I'm
26:41:52starting fresh just like you. We're
26:41:54going to come right here. I'm going to
26:41:55do download R for Windows, but if you
26:41:57have a different type of machine, be
26:41:58sure to get the correct one. I'm going
26:42:00to download R for Windows and then I'm
26:42:02going to come up here and say install R
26:42:04for the first time and then we're going
26:42:05to get this package right here. So,
26:42:07let's go ahead and click on this and I'm
26:42:09going to go ahead and download it. Now
26:42:11that it's done downloading, I'm going to
26:42:12go ahead and open that.exe file. You may
26:42:14not be able to see it, but it just says,
26:42:16are you sure you want to download this?
26:42:18And I'm going to say yes. Now, we need
26:42:20to set up our language. I'm going to
26:42:21choose English, but go ahead and choose
26:42:23the correct language for you. I'm going
26:42:25to click okay. We're going to click
26:42:27next. And now we're just saying where
26:42:29it's actually going to be placed. So I
26:42:31by default it gets placed in my C drive
26:42:33under program files. That is where I
26:42:34want it. So I'm going to go ahead and
26:42:35click next. You can select your
26:42:37components. We want all of them uh just
26:42:39by default. And then it asks if you want
26:42:41to have this, you know, boot up on
26:42:43startup. I'm going to say no because I
26:42:44don't need it all the time when I'm, you
26:42:46know, restarting my computer. And then
26:42:48you can name this as well. I'm just
26:42:49going to keep it as R. Again, I don't
26:42:52really need a desktop shortcut or a
26:42:54quick launch shortcut. So, I'm just
26:42:56going to get rid of that. And now it's
26:42:58going to install on our computer. That
26:43:00took about 30 seconds. And we now have R
26:43:03on our computer, which means we can come
26:43:06back here and we can install R Studio
26:43:08Desktop for Windows. Again, make sure
26:43:10you have the right one for your
26:43:11operating system. I'm going to go ahead
26:43:13and click on this and we are going to
26:43:15save this. Now that it's done
26:43:16downloading, let's go ahead and open it
26:43:19up. Now, we're going to set up and
26:43:21install our R Studio. Let's go ahead and
26:43:23click next. We're just selecting our
26:43:25location. Make sure you have enough
26:43:27space required. It's very small, but
26:43:29that's happened to me in the past where
26:43:30I have no storage. Then it doesn't
26:43:32really work. So, let's go ahead and
26:43:34install this. It's going to install.
26:43:36Again, this should only take about a
26:43:38minute or so. Now that R Studio is
26:43:40completely set up, let's go ahead and
26:43:42click finish. And now, we're going to go
26:43:44look for it. So, we're going to come
26:43:45right over here. We're going to say R
26:43:47Studio, and we're going to open up R
26:43:49Studio. It says we need to choose our R
26:43:51installation because R Studio requires
26:43:53an existing installation of R. We're
26:43:56going to go ahead and choose the default
26:43:58one. We're going to click okay. And now
26:44:00R Studio from Bosit is popping up for
26:44:03us. And just like that, we are in our
26:44:05studio and now we have our user
26:44:08interface that we can take a look at. So
26:44:10really quickly, let's take a look at
26:44:12what we have here. Right here we have
26:44:14our console, our terminal, our
26:44:16background jobs. But this is our
26:44:18console. If you are looking for uh where
26:44:21you can write code, it's going to be
26:44:22right here. We just need to click on
26:44:24this R script and it pulls up our coding
26:44:27workspace right up here. Now, this is
26:44:28where we can uh start typing. We can
26:44:31bring things in like reading in a
26:44:34library or a package and we write all of
26:44:36our code in here. Down here is our
26:44:38console. So, when we execute code, this
26:44:40will be where we get error messages or
26:44:42we can see the output of our code. On
26:44:44this right hand side, we have our
26:44:46environment. Now, this is where if we uh
26:44:48pull in some type of data frame or if we
26:44:50create variables or lots of other things
26:44:52that we're going to be doing throughout
26:44:53this series, they will show up right
26:44:55over here. And lastly, in this section
26:44:57on the bottom, you can actually come in
26:44:59here and we can select a file path to
26:45:02give it. So, let's come in here. Let's
26:45:04go to our YouTube and uh this is going
26:45:07to be our R series. So, I'm going to
26:45:08open up our R series. We don't have any
26:45:11files in here just yet, but I'll have
26:45:13access to our files where we can then
26:45:15pull them in or read them in. And it's
26:45:17really great that R Studio has this
26:45:19because you use a lot of files when
26:45:20you're working within R Studio. This is
26:45:22absolutely something we will use quite a
26:45:23bit. We if we come right over here,
26:45:25actually, we can do Ctrl S and we can
26:45:27save this. Let's go down to our YouTube
26:45:30and I'm going to go to this R series and
26:45:32I'm just going to save this as first
26:45:35file. And if I save it, you'll see it
26:45:37saves as first file.R. Now, that's
26:45:40actually not a good naming convention. I
26:45:41should use an underscore there. But
26:45:43these are all things that we will get to
26:45:45uh within this series. Now that we have
26:45:47everything set up, we can keep going in
26:45:50this series moving forward. We're going
26:45:51to learn a ton of different things. But
26:45:53with that being said, I hope that this
26:45:55is helpful. I hope you were able to get
26:45:56everything set up, and I will see you in
26:45:58the [music] next lesson.
26:46:12Hello everybody. In this lesson, we're
26:46:13going be taking a look at the basics of
26:46:15using R Studio and R. Now, right down
26:46:19here in the last lesson, I showed you
26:46:20how to access uh kind of a file path
26:46:23here. And you can see what files are in
26:46:25here. And all you have to do is click on
26:46:26this. And then your file folder comes
26:46:28up. And then you can click into wherever
26:46:30you want. You're going to open it up.
26:46:31And these are some of the files that
26:46:32we're going to be using later on in this
26:46:35series. Over here on this lefth hand
26:46:36side, we have our console opened, but we
26:46:39can't interact with this data or write
26:46:42code yet because we don't have an R file
26:46:45open. So, what we're going to do is
26:46:46we're going to come right here. We're
26:46:47going to open up an R script. We can
26:46:49also open up a ton of different files.
26:46:51In fact, at the very end, we'll be
26:46:53opening up an R markdown file to create
26:46:55a dashboard for some data visualization.
26:46:57But you can open up lots of different
26:46:59files here if you want to do that. We
26:47:02can also open up these files over here.
26:47:05So if we want to, we can do that. But if
26:47:06we open it up right here, then we can
26:47:08name it and then it gets saved into that
26:47:10file path. So we're going to call this
26:47:12one the basics of R. And we're going to
26:47:15save that. So now we get this open right
26:47:18up here where we can actually write our
26:47:20code. Now I say write our code. We can
26:47:22actually write code down here in our
26:47:24console. And I'll show you that in a
26:47:26little bit when we start creating some
26:47:28variables and expressions and different
26:47:29things like that. But for me personally,
26:47:32most of my workflow is going to be up
26:47:33here. So I'm going to add some uh lines
26:47:36here so we have some room to work with.
26:47:39The other thing that I typically do or
26:47:41have set is a soft wrap long lines. Now
26:47:44if we don't have this, if I say I uh am
26:47:48writing something, this is going to be
26:47:50our code. Let me spell this right. And I
26:47:52add a bunch of dashes. You're going to
26:47:54note it just keeps going and it keeps
26:47:57going. Um, especially as we start
26:47:59bringing in code or we start bringing in
26:48:02files. Um, we're going to want something
26:48:04like this, which is a soft wrap, which
26:48:06will wrap it to the next line. And so,
26:48:09we still are on line two, but it is kind
26:48:12of all in our viewing area instead of
26:48:15being way over to the right, which I
26:48:17personally don't like. So, I'm going to
26:48:18get rid of this. Now, we've opened up a
26:48:21file. We now have access to write code
26:48:24in R. And what we're going to do is
26:48:26we're going to start with the very
26:48:28basics. And I'll kind of show you how
26:48:29you can write it up here and you can
26:48:30write it down here. Now, one thing to
26:48:33note is we have this little sweep button
26:48:34right here. This is our clear console.
26:48:36We can click this button. It's going to
26:48:38reset everything over here. And we also
26:48:40have a sweep button over here. We can
26:48:43clear out our environment. We'll be
26:48:45using both of those, but let's get
26:48:47started with the basics. So, we're going
26:48:49to take a look at variables.
26:48:52Now variables are where you're going to
26:48:54assign a value whether it is a string or
26:48:56it is a number to a variable which is
26:48:59then stored in R's memory. So let's call
26:49:02this one the num variable. We have to
26:49:05use an assignment operator. Now this
26:49:08assignment operator right here
26:49:10essentially says take this number. We'll
26:49:13do 42. Take this number and put it
26:49:15within this numbum variable. Now we want
26:49:19to run this code and we can do that by
26:49:21highlighting our code like this and
26:49:23doing controll enter. Now that we've ran
26:49:25it you can see over here it is store
26:49:27that as a value. We have our numvar.
26:49:30Then we have 42. Now that we have this
26:49:33what we can do is we can say print and
26:49:35we'll do num_var.
26:49:38If we print out you're going to notice
26:49:41down here in the console 42 is printed
26:49:44out. So we printed it out up here where
26:49:46we're actually writing our code into our
26:49:48console. But we can interact with our
26:49:49code as well down here. So we can do
26:49:52num_var
26:49:54and then we can do times 2. And if we
26:49:56run this, we get 84. It just doubles it.
26:49:59So we're able to write this. But notice
26:50:01if we come up here, let's do ctrls. I
26:50:04just saved this file. If we get rid of
26:50:07this, so let's get rid of our basics of
26:50:09R and we're going to clean this out. If
26:50:11I pull back up our basics of R, that
26:50:15multiplication times two is not there.
26:50:17So, we don't have access to the code
26:50:19that's written down here in our console.
26:50:21So, that is okay if you're just messing
26:50:24around and kind of looking at your code.
26:50:25Maybe we've pulled in a data set and
26:50:27we're looking at data. We don't really
26:50:28want to save it in our code. That's okay
26:50:31to write it down here. But anything you
26:50:33want to save, it is best to write it up
26:50:35here. Otherwise, when you save it and
26:50:37you exit out and you come back later,
26:50:39you won't have it. Now, I told you that
26:50:41this was numeric and we can actually
26:50:43check that by using this class right
26:50:46here. So, we're going to say we're going
26:50:48to pass through our variable. So, we
26:50:50have our number var. We're passing
26:50:51through this variable into this class
26:50:53function. And what it's going to do, it
26:50:56is going to output the data type of this
26:50:58right here. Now, of course, we're
26:51:00passing through our variable, but in the
26:51:03memory, it has this stored right here as
26:51:05the value in memory. So, we are looking
26:51:08at this right here. We're not actually
26:51:10looking at the text. So just something
26:51:11to note. Now we don't have to just store
26:51:14numbers like this. We can store a lot of
26:51:16other things including string. So if we
26:51:19want to say we'll call this a string
26:51:22variable. You don't have to do this uh
26:51:24but I'm going to do it. But we can put
26:51:25it in parenthesis and say I like R. And
26:51:29if we save this, you're going to see I
26:51:32like R here as our string value. So, we
26:51:35can also store strings as well as
26:51:38numbers. Now, all we're going to do for
26:51:40the rest of this lesson is take a look
26:51:42at a few different ways to store data
26:51:44inside of variables. Those are things
26:51:46like vectors and lists and maybe even a
26:51:49data frame. These are common ways that
26:51:51data is stored within R. And so, really,
26:51:53that's all we're going to take a look
26:51:54at. If you already know data types
26:51:56within R, this is a perfectly fine time
26:51:58to cut out. In the lessons after this,
26:52:00we're going to go into a lot of other
26:52:01things like operators and expressions,
26:52:03reading in files, sorting the data that
26:52:05we read into dataf frames. We're going
26:52:06to get a lot more advanced as we go, but
26:52:08again, this is the very basics. So,
26:52:10let's take a look at a vector. So, so
26:52:13far we've only stored something like a
26:52:15string or a number. That's just one
26:52:16value and one value, but we can actually
26:52:18store multiple values. So, let's call
26:52:21this a vector variable. And what we can
26:52:24do is we're going to do a c. Then we're
26:52:27going to open up our parenthesis. And
26:52:29this is going to say that this is a
26:52:30vector. So we can do 10, 20, 50, 100,
26:52:36and thousand. And then when we run this,
26:52:39you'll notice we have this right here.
26:52:41So it's going to say we're storing
26:52:43numbers. And this right here is an
26:52:46index. So we have one through five. So
26:52:49we have one, two, three, four, and five.
26:52:52So now we are storing multiple values
26:52:56within one variable. So vectors are good
26:52:59for storing sequences of the same data
26:53:01type. But if we want to store a lot of
26:53:02different stuff, it could be whatever we
26:53:04want. We can store that in a list. Now
26:53:06lists are very popular. So we're going
26:53:09to do um a list
26:53:12variable. And what we're going to do is
26:53:14we're going to name each of these
26:53:16variables that we're putting in here. So
26:53:17I'm going to say that the name is equal
26:53:19to Alex. And then we'll say age is equal
26:53:23to 30. And then we'll say scores is
26:53:27equal to we'll pass through a vector
26:53:29here. We'll say 90, 50, and
26:53:3424. Just like this. Let's go ahead and
26:53:37create this list. Now, we can just call
26:53:41this and we can just say we could even
26:53:43just say uh list var here. And it's
26:53:46going to pull in the name, the age, and
26:53:49the scores. Now, you'll notice we have
26:53:52this little shortcut right here. And
26:53:54this is saying that's what it's called.
26:53:56So, if I only want to call the name, I
26:54:00only want to retrieve that from this
26:54:02list, I can do that. And if I run it,
26:54:05it's only going to output just Alex,
26:54:08just the name. That's all it's going to
26:54:09do. There are other ways to do this as
26:54:12well. We could uh let's get rid of that.
26:54:14We could do a bracket and we could look
26:54:15at the index. So, if we did a one, let's
26:54:18go ahead and run this. The first index
26:54:21within R is Alex. So we're still pulling
26:54:24up the same one. We could also do it by
26:54:26the name just in a different way. We
26:54:28would need to do a double bracket and
26:54:30pass through name. And we can run this
26:54:33and it's still Alex. So all these ways
26:54:35work. I will say the one that I
26:54:36typically do and this is not just for
26:54:39lists. This is similar if we're working
26:54:41with data frames where we have columns
26:54:43and rows. You can pull specific columns
26:54:46by using this shortcut. And I do this
26:54:48all the time. So that is how I would do
26:54:50it. That's my personal preference. Now
26:54:52there are other data types within R, but
26:54:55the last I'm going to show you, and this
26:54:56is the one that we're going to be using
26:54:57in future lessons, is a dataf frame.
26:54:59Now, a dataf frame could be you just
26:55:01pulling in a CSV file that has columns
26:55:03and rows, which is what we are going to
26:55:05do. But I want to show you kind of what
26:55:07a dataf frame looks like. So if we come
26:55:10over here, we're going to say dataf
26:55:13frame, and we're going to pass through,
26:55:15and you're going to actually write
26:55:16data.frame. frame. So if we come right
26:55:19over here, it says the function
26:55:20data.frame creates dataf frames which is
26:55:22coupled correlations of variables which
26:55:24share many of the properties of
26:55:25matrices. Now matrices is another data
26:55:28type but we're not covering that one
26:55:29right now. But let's click on dataf
26:55:31frame and we can pass through different
26:55:34things like this data right here. Now
26:55:37when we pass through this data and let's
26:55:39actually take a look at this. This is
26:55:41what it actually looks like and this is
26:55:43just some of the uh information but it's
26:55:45storing it as data. And so what we're
26:55:47going to do is we're going to do
26:55:48something really similar. Let's say the
26:55:51name. This time we're going to pass
26:55:53through multiple values. So not just a
26:55:55list, it's essentially multiple lists.
26:55:57So we'll do a vector here. And I'm just
26:55:59going to make up some names. So we'll
26:56:01say Alex,
26:56:03Sally, and John. And I need to have that
26:56:07in quotes. And then we'll do a comma.
26:56:10We'll pass through our next one. So this
26:56:11will be age. And we'll do another
26:56:14vector. So we'll say is equal to uh
26:56:16we'll do 30
26:56:1950 and 99
26:56:21and then we'll pass through the scores.
26:56:24And in fact uh let's just keep the
26:56:26scores like this because it'll be
26:56:28simpler. Now let's go ahead and create
26:56:31our data frame. And our data frame is up
26:56:33here in data as well. You'll see it
26:56:35looks a little bit different. This says
26:56:36as a list of three, but this we have
26:56:38this little grid. Now let's actually
26:56:40click on this. It'll pull up this data
26:56:42frame. And now it looks like an Excel
26:56:44spreadsheet. It looks like a CSV file or
26:56:46a database or wherever you store
26:56:47structured data. We have our name, age,
26:56:50and scores. And here is kind of this
26:56:52index that it gives us. And you can see
26:56:54it's all pretty. It looks really nice.
26:56:56And so data frames are extremely popular
26:56:58within R to use, especially with
26:57:01structured data. And this is something
26:57:02that we are going to use when we start
26:57:04pulling in larger data sets. And that's
26:57:06something that as you kind of work with
26:57:08real data in a professional environment,
26:57:11you'll start working with CSVs and
26:57:12larger data sets and you'll have to do
26:57:14all these things with them. And so
26:57:15that's what we're building up to in this
26:57:17series. So I hope that this was helpful.
26:57:19I hope you learned a little bit more
26:57:20about how to use R, a little bit more
26:57:22about data types and variables. In the
26:57:24next lesson, we're going to be taking a
26:57:26look at operators and expressions.
26:57:30[music]
26:57:40Hello everybody. In this lesson, we're
26:57:41going to be taking a look at operators.
26:57:43Now, operators in R are just symbols or
26:57:46keywords that perform operations on
26:57:49variables or values if you haven't
26:57:51already put those values into a
26:57:52variable. Now, we're going to take a
26:57:54look at several different types of
26:57:55operators, but if you've been following
26:57:57along in this series, we've already used
26:57:59an operator before. it was something
26:58:02called an assignment operator. So if we
26:58:04have our variable and we go like this,
26:58:07this is an assignment operator. These
26:58:10are characters that say when we put a 42
26:58:14over here, this 42 is going to stay
26:58:16within this variable. So this already is
26:58:19something that we know. So that is an
26:58:21assignment operator. Now the next one
26:58:24that we're going to take a look at is an
26:58:26arithmetic
26:58:28and I spell this right? Operator. There
26:58:31are other ones as well like comparison
26:58:32operators and logical operators and
26:58:35we'll get to those in just a little bit.
26:58:37But let's start out with looking at
26:58:38arithmetic operators. These are
26:58:40operators that are specifically built
26:58:42and designed for math. And so let's
26:58:44declare two different variables just to
26:58:46start and then we'll take a look at
26:58:48these arithmetic operators. Let's just
26:58:50do the classic x and y. We'll say x is
26:58:5310 and we'll say y is three. Now let's
26:58:59go ahead and declare those. We're going
26:59:01to see in our environment right over
26:59:02here. We have our x and our y. Within
26:59:05these variables, we have the very basic
26:59:08ones. We're going to take our variables.
26:59:09We're going to say x + y. And this is
26:59:12going to be, of course, 13. Now, what we
26:59:16can do, by the way, is we can assign
26:59:19this to another variable. So, if we
26:59:21wanted to, we could say a is going to be
26:59:23assigned x + y. And if we run that, x +
26:59:27y is 13. So, a becomes 13. Now, we don't
26:59:31have to do that every time we do any
26:59:34type of operator or mathematical
26:59:36equation, but if we want to store that
26:59:38value, we can. And this may not look
26:59:41like it makes sense, but it does make
26:59:42sense because we've stored these values.
26:59:44We're adding them together, and then we
26:59:47are assigning them. So, that is just
26:59:49simple uh x + y, but we can also do x
26:59:53minus y. This is just subtraction. And
26:59:56we're going to get seven. So that's 10 -
26:59:583 is equal to 7. We also have
27:00:00multiplication which if we do x * 10 and
27:00:04that's the little star right here. So
27:00:06that is on my keyboard the shift 8. If
27:00:10we do x and sorry it's supposed to be x
27:00:14* y we're going to get 30. And so we're
27:00:17just multiplying them together. We have
27:00:18one more of the basic ones and then
27:00:20we'll start going to kind of the more
27:00:21complicated ones. But that's division.
27:00:24So, we're going to say x and then we're
27:00:26going to do a forward slash y. And this
27:00:29is going to be x / y. And that's 3.3333
27:00:33because it's 3 into 10. Now, the next
27:00:36one that we're going to look at is a
27:00:38little bit more challenging. This is an
27:00:39exponent or to the power of. So, it
27:00:41would be x and we're going to do this
27:00:43little carrot here. That's what it's
27:00:45called. It's called a carrot. X to the
27:00:47power of y. So, we're going to do 10 to
27:00:49the^ of 3. So, it' be 10 * 10 * 10. And
27:00:52so when we do that, we're going to get
27:00:541,00. So that one's really good. I'm
27:00:57sure you've seen that if you've taken
27:00:58algebra or, you know, other types of
27:01:00maths, you're going to recognize that
27:01:02pretty quickly. But this next one is
27:01:03most likely one that you haven't seen.
27:01:05This is called a modulo. And what it
27:01:07does is it takes the remainder of a
27:01:10division. So if we do x and we're going
27:01:12to do two percent signs of y. If we run
27:01:17this, we're going to do 10 / 3, which
27:01:21means three can go into 10, but we have
27:01:23a remainder of one. And so that's what a
27:01:26modulo is used for. Now, I will say
27:01:28you're going to use these a lot. They're
27:01:30fairly straightforward. They're not
27:01:32crazy confusing. This is kind of
27:01:34foundational math that most people
27:01:36should know. If you haven't, just messed
27:01:37around with those. Uh those are really
27:01:39good to know. Now, I'm not going to dive
27:01:41into it a lot, but have you ever heard
27:01:43of Hemdos? This is the order of
27:01:46operations within math. This is
27:01:48something you need to be aware of
27:01:50because if you do something like, and
27:01:52I'm just going to use numbers. I'm going
27:01:53to do five * 10. If we do it just like
27:01:57this and we run it, of course, that's
27:01:59going to be 50. But then, let's do
27:02:03divided by 2. And let's run this. Now,
27:02:06of course, that's going to be 25. But if
27:02:09I add plus six here and we run this,
27:02:13this is where it starts getting a little
27:02:14complicated, right? What's happening
27:02:16where and what's going to happen if I
27:02:19add a parentheses around all of this? So
27:02:22this is where PEMDOSS comes into play.
27:02:24PEMDOS stands for parenthesis, exponent,
27:02:26multiplication, division, addition,
27:02:28subtraction. So if we are looking at
27:02:31this, we have to start with the P, the
27:02:33parenthesis. So, we're going to do
27:02:35everything in this parenthesis first
27:02:37before we do anything outside of the
27:02:39parentheses, which is multiplying times
27:02:415. We would also do the exponent, which
27:02:44we don't have, multiplication, which is
27:02:46not in the parenthesis. And then
27:02:47division. So, we would divide 10 by 2,
27:02:49which is 5. Then, we would add 6, which
27:02:52is 11 * 5 is 55. Let's make sure that's
27:02:55correct. And there we go. If we did it
27:02:58in a different way, if we uh put the
27:03:00parentheses over here, and let's get rid
27:03:02of this real quick. If we put the
27:03:05parentheses over here, now we're going
27:03:07to multiply times 50, then divide by two
27:03:10and add six, and that's going to be 31
27:03:13because we're going to get 25 right
27:03:15here, and then add six to it. And so
27:03:17these parentheses are very important.
27:03:20You will see these in different types of
27:03:21equations and different types of math
27:03:23that you'll work with within R as it is
27:03:25very statistics heavy. So, you know,
27:03:26this is just kind of foundational
27:03:28arithmetic and maths that you need to
27:03:30know in order to work with these types
27:03:31of programming languages. The next one
27:03:33that we're going to take a look at, and
27:03:34let's give us some more room right here,
27:03:36and these are going to be comparison
27:03:39operators. Comparison operators allow
27:03:42you to evaluate different conditions.
27:03:44So, let's take a look at one. So, we
27:03:47have x and y here. We created those
27:03:49earlier. So, we're going to say x is
27:03:52greater than y. Now, when we run this,
27:03:55we're going to get a different output.
27:03:56We're no longer going to get a string or
27:03:59a number. Now, we're getting what's
27:04:01called a boolean value.
27:04:03Now, this boolean is either true or it
27:04:06is false. So, now if we say x is less
27:04:10than y and we run this, and I did an
27:04:13uppercase, whoops. Let's go ahead and
27:04:15run this. Now, we're going to get a
27:04:18false. So depending on whether it is a
27:04:21true condition, it evaluates to true or
27:04:23it evaluates to false, that's what our
27:04:26output is going to be. Now we can also
27:04:28add in to these ones an equal than or an
27:04:31equal uh as well. So it would say x is
27:04:33less than or equal to y. And if we run
27:04:36it, it's still going to be false, but
27:04:37you can add that in if you want that
27:04:40these uh to potentially be equal or you
27:04:42want to check for that. The other ones
27:04:44that you really need to know are x is
27:04:47equal to y. Now this is wrong uh because
27:04:52this is actually an assignment operator
27:04:54just like let's go back up. This is in
27:04:57certain conditions within R. We need to
27:04:59say equal equal. This is actually what
27:05:02you need to write. And so it may seem a
27:05:04little counterintuitive but that is the
27:05:07syntax that you need to use. Now x is
27:05:10not equal to y. One is 10, one is three.
27:05:12So of course that's going to be false.
27:05:14But if we said x is equal to x then this
27:05:18will evaluate to true. The last one that
27:05:21you need to know and this is the
27:05:22opposite of equal to is not equal to. So
27:05:25if we say x is not equal to y that's an
27:05:29exclamation point with an equal sign. If
27:05:31we say x is not equal to y we're
27:05:33checking are they not equal? And that is
27:05:36true. They're not equal. And so these
27:05:38are comparison operators that are very
27:05:40commonly used. You'll use these so often
27:05:43they'll just become second nature. And
27:05:45the last one that we're going to take a
27:05:46look at is logical operators. Now,
27:05:49logical operators, uh, you know, aside
27:05:52from, you know, the basics, logical
27:05:55operators are used all the time for
27:05:57everything. They are here to help
27:05:59determine and evaluate expressions to
27:06:01determine if they are true or if they
27:06:03are false or if multiple conditions are
27:06:06true or false. So, right up here, let's
27:06:08get this uh x is greater than y. We know
27:06:13that that is true, right? We know this
27:06:15is true. But what if I add I want that
27:06:18to be true. And this is where the
27:06:20logical operator comes in. So we want x
27:06:23and y that way evaluates to true. But we
27:06:26also want x to be equal to x. So we're
27:06:30going to put this right here.
27:06:32So now we're evaluating two separate
27:06:35conditions. And this one right here,
27:06:37this amperand I believe is what it's
27:06:39called, which is a shift uh let's see,
27:06:42seven on my keyboard. This amperand
27:06:44means that both this and this condition
27:06:47need to be met in order for it to
27:06:49evaluate to true. Let's go ahead and run
27:06:52this. So since both of these conditions
27:06:54is true, this is great. X is greater
27:06:57than Y and X is equal to X, this
27:06:59evaluates to true. But let's change
27:07:02this. We're going to change it to Y. We
27:07:04know this is false. So this is true but
27:07:07this is false. So are they both true? So
27:07:10it needs this and this need to be true.
27:07:13Since one of them is false, we will get
27:07:16an output of false. And so this is what
27:07:18a logical operator is. Now we're going
27:07:21to take the take this exact same thing
27:07:23and we're going to bring it down here
27:07:24and we're going to change this into a
27:07:27bar. and I call it a bar, but it's a
27:07:29shift and it's right above my enter key
27:07:31on my keyboard, right next to the
27:07:32backslash. Um, but this right here means
27:07:35or. So, this is and and this is or.
27:07:38These are the ones you're going to use
27:07:4099% of the time. So, you're going to say
27:07:43is this condition true or is this
27:07:46condition true? If either one of these
27:07:48conditions are true, it will be true
27:07:51down here. They don't both have to be
27:07:53true. So now if we run this we're going
27:07:56to get true x is greater than y this
27:07:59evaluates to true or this which
27:08:03evaluates to false since just one of
27:08:05them is true then the entire expression
27:08:08evaluates to true there is technically
27:08:10one more logical operator and I'm going
27:08:12to be honest I I don't think I ever use
27:08:14this one or if I do it's a very specific
27:08:17use cases. This is the not operator.
27:08:21This is going to say this. So let's run
27:08:23this. This is false. And we're going to
27:08:26run this with the not operator and it's
27:08:29going to change it to true. So it
27:08:30basically reverses the boolean value
27:08:33that is in the output. If it's true,
27:08:34changes it to false. If it's false, it
27:08:36changes it to true. It just flips the
27:08:38value. That's all this operator does. I
27:08:40think I've used this a few times in very
27:08:42specific use cases, but 99.9% of the
27:08:46time I'm using the and and the or
27:08:48logical operators. Those are the ones
27:08:49that are the most common to use. So this
27:08:51is the basics of using operators. Now in
27:08:54the next lesson, we're going to start
27:08:55working with our data set, how we can
27:08:57pull it in, how we can put it into a
27:08:59data frame, how we can write files and
27:09:01create files as well. If you haven't
27:09:03already, be sure to check out my full R
27:09:05for data analytics course on Analyst
27:09:07Builder. I'll leave a link in the
27:09:08description and a coupon code if you
27:09:10would like to take that course. Thank
27:09:12you guys so much for watching and I will
27:09:13see you in the next video.
27:09:17[music]
27:09:23>> [music]
27:09:27>> Hello everybody. In this lesson, we're
27:09:29going to see how we can read and write
27:09:31files within R. Now, as you can see down
27:09:34here, we have this parks and wreck data
27:09:36set. I will leave, you know, the GitHub
27:09:38below so you can go and you can get this
27:09:40exact file. It's just a little sample
27:09:42file that we are going to pull in. But
27:09:44what we can do is we can literally click
27:09:46on this and we can import this data set.
27:09:49Now in order to do that there are some
27:09:52packages that we need and we do want to
27:09:54install these. So let's go ahead and get
27:09:56those installed and then we'll continue.
27:09:58Now you can see right down here this is
27:10:00the code that they are writing to bring
27:10:02in this data set. And coincidentally
27:10:04enough in just a second we're going to
27:10:06write this exact same thing. And there
27:10:08are different ways to write it. We don't
27:10:10actually have to use this library right
27:10:13here. There are tons of other libraries
27:10:14or even default things within R that we
27:10:16can use but this is our data set and we
27:10:19can import this data set just like this
27:10:21and it's going to be saved into our
27:10:23memory into our data and now we have
27:10:25this data frame. So then if we come over
27:10:28here we can call in this parks and rack
27:10:31data set and I can just hit tab and I
27:10:34can run this and it's going to show all
27:10:36of this data and that is a very simple
27:10:39way to do it. Now, the only downside to
27:10:42doing it like this is if you are
27:10:44creating some type of automation. Let's
27:10:46say you're pulling in data from
27:10:47somewhere from the web and then you are
27:10:49creating a CSV file. You're pulling that
27:10:51data into here with that CSV file and
27:10:54you're manipulating it. You're changing
27:10:55it. Then you're writing another file
27:10:57after you've cleaned up that data for a
27:10:59specific purpose. Now, in that scenario,
27:11:01doing it like this is not going to work
27:11:03at all because you need to actually
27:11:05write out the code so it's saved to the
27:11:07file. And so we are not going to do it
27:11:10like that. I don't really recommend that
27:11:12unless there's just some one-off file
27:11:14that you just want to get in here and,
27:11:15you know, do whatever with. Let's go
27:11:17ahead and sweep this because uh we don't
27:11:19need it. Um what we can do is we can
27:11:23come in here. We can write very similar
27:11:25code. So I'm just going to show you how
27:11:26to do it uh without anything. So we're
27:11:28going to do read.csv.
27:11:31So read.csv. All we have to do is pass
27:11:33through the file name. There's a lot of
27:11:36other what are called parameters or
27:11:38arguments that you can pass through as
27:11:40well. So you know if we were uh writing
27:11:44it out again if we write out read
27:11:48CSV you can come in here and you can get
27:11:50additional help on this and look at all
27:11:52the different parameters that they have
27:11:54in here that you can pass through. Now
27:11:56what we need to do is pass through this
27:11:58file. So let's come down here. I have
27:12:01this uh parks and wreck data set. I'm
27:12:04going to click on it and I'm going to
27:12:05copy this as a path. You can do that
27:12:09with a shortcutt control shift C, but
27:12:11I'm just going to paste it in there. And
27:12:14if we try to run this, and actually
27:12:16let's really quick, let's declare this
27:12:18as our data frame. But if we try to run
27:12:21this,
27:12:23then
27:12:24it's not going to work. This backslash
27:12:27is actually a special character. What it
27:12:29does is it says that this Y, the letter
27:12:31right next to it, is a special
27:12:33character. Now, we can actually negate
27:12:35this by putting another backslash. And
27:12:39then what we're saying is is the special
27:12:40character is this right here. And so, it
27:12:43just keeps it. And we can Whoops. Let me
27:12:46put it in the right spot. So, we can put
27:12:48in a double slash. And that's perfectly
27:12:50acceptable. Let's go ahead and run it.
27:12:52And now we have our data frame right
27:12:55here. And we can click on it. We can
27:12:57open up this data frame and it's
27:12:59beautiful. We're doing uh some parks and
27:13:01wreck and this is what we're going to be
27:13:02using in the next several lessons uh
27:13:04this small parks and wreck data set but
27:13:06that is how we can read in a CSV file.
27:13:09Now there's lots of different types of
27:13:10file formats. There's JSON, Excel, etc.
27:13:13There's lots of different ones, but this
27:13:15is in a nutshell how you do it. You may
27:13:18just need, you know, a different way to
27:13:19read it in. But there's so many
27:13:21different file formats that you can read
27:13:23in within R. Just about anything you can
27:13:25imagine. Now when we have this data
27:13:28frame, we've created it. It's up here.
27:13:30It looks great. Now we can start messing
27:13:33with this dataf frame. We can use it. We
27:13:35can look at it. For example, we can use
27:13:37this right here, which says the head of
27:13:40the data frame is going to give us the
27:13:42first five rows of that data set. I
27:13:45guess it's giving us the first six here,
27:13:46but this is just a sample of the data.
27:13:49You can imagine because we're going to
27:13:51be working with larger data sets,
27:13:53especially as we get towards the end,
27:13:55that if you have a thousand, 10,000,
27:13:57100,000 rows, you don't want to look at
27:14:00all of that every single time. You may
27:14:02just want to print out just a few rows
27:14:04of the data. And that is how you can do
27:14:06this. We also have a few different
27:14:08things that we can do to kind of check
27:14:10our data that we just pulled in. This
27:14:13one right here, and let's see if we have
27:14:14it right here. This gives us the
27:14:16structure of our data frame. So if we
27:14:19come and open the parentheses and pass
27:14:20through our data frame and so what it's
27:14:22going to tell us is going to say here is
27:14:24the character the department the role
27:14:26these are our columns and it says this
27:14:28is a character column character
27:14:30character these are strings and then
27:14:32annual salary this is an integer and
27:14:34then dogs rescue with three legs that's
27:14:36also an integer. Another one that you
27:14:38might use and I do use these by the way
27:14:41because I often uh you know kind of want
27:14:44to see how the data is sitting. Another
27:14:46one we can use is summary. And this is
27:14:48going to give us some statistics of our
27:14:50data set. Now, more specifically, this
27:14:52is really good for our numeric columns,
27:14:55but it's not as good or useful for our
27:14:58character columns. But we have some min,
27:15:00first quarter median, mean, third
27:15:03quartile or quarter bolt, quartile max.
27:15:06Um, and the same for the other ones. So,
27:15:08it gives us just a little bit of
27:15:09information about our numeric columns
27:15:12and that can be really helpful. Now,
27:15:14like I said, there are different things
27:15:16when we're pulling in data. I'm going to
27:15:17pull this down. There are different
27:15:19things that we might want to use. For
27:15:22example, if we say header, the header
27:15:25says whether we're going to keep our
27:15:26headers in our file or not. So, if I say
27:15:29equals false, I'm going to do dataf
27:15:32frame 2 here. So, we're pulling this in
27:15:34and we're saying the header is equal to
27:15:36false. Now, if we run this, it's going
27:15:39to look a little bit different. So now
27:15:42this row as our headers which are our
27:15:44column names now they are actually part
27:15:47of the data and so if you don't have a
27:15:50header on your data it's just goes
27:15:52straight into the data on you know row
27:15:53one this would be something you need to
27:15:55write in there otherwise it's going to
27:15:57not understand that although it does
27:15:59have header as a default saying true
27:16:02because we didn't put it up here default
27:16:03is true you can specify that it is false
27:16:07another thing that you might need to use
27:16:10and let's go back here is SP.
27:16:13This stands for a separator. Now, by
27:16:16default, when you're pulling in a CSV,
27:16:18it's going to be a comma because that's
27:16:19what CSV stands for, a comma, separated
27:16:21value file. That's what a CSV file is.
27:16:24Now, we don't have to keep it as a
27:16:26comma. Let's say, for example, we have a
27:16:29space that's separating everything or we
27:16:31have a dash that's separating
27:16:32everything. It's whatever you want it to
27:16:34be. Now, by default, again, it is a
27:16:36comma. And if we run this and we go back
27:16:39up to our day frame two, it's going to
27:16:41be the exact same. But if we change it
27:16:43to something like a space, we're going
27:16:46to completely change how this file
27:16:48looks. Now, every space that you see
27:16:51within here is going to be a different
27:16:53column. It's going to separate it out
27:16:54like this. And this looks terrible. So,
27:16:56we definitely don't want to do that, but
27:16:58it is something that you should be aware
27:17:00of. That's a parameter that we can pass
27:17:02through. So, let's run this correctly.
27:17:05And let's actually put this back to the
27:17:07default. Let's run this. Let's say we
27:17:11have our data frame and we've done some
27:17:13things to it. We are happy with it.
27:17:15Maybe we've aggregated some of the data.
27:17:17And now we say, okay, I want to export
27:17:20this to a file. Instead of readcsv,
27:17:23we're going to write.csv.
27:17:27So write.csv right here is going to put
27:17:30all of this data into a file. So we're
27:17:32going to do this. And what we're going
27:17:34to do is we're going to pass through
27:17:35this exact same thing. I'm going to pass
27:17:38through this file path, but we have to
27:17:40specify what we're passing into that
27:17:42file path. So, we're going to say here,
27:17:44take dataf frame 2. And then here's
27:17:47where you're going to place it. Now, we
27:17:48don't want to call it the exact same
27:17:50thing. It will overwrite our previous
27:17:52file. We don't want that. So, we're
27:17:54going to say uh output. So, now we're
27:17:58taking this dataf frame 2, which is the
27:18:00exact same data. We haven't done
27:18:01anything to it, but you know, uh, we're
27:18:03taking this dataf frame 2, and now we're
27:18:05going to export that data set or that
27:18:07data frame into a file. Let's go ahead
27:18:10and run this. And there we go. So now we
27:18:14have our parks and rack data set. If we
27:18:16then wanted to read that in, and we can.
27:18:21Let's come right over here. We're going
27:18:23to do underscore output.
27:18:26And let's do this as dataf frame 3. Now
27:18:29let's run this. Now we have our dataf
27:18:32frame three and there is our data set.
27:18:35You'll notice we now have six variables
27:18:37instead of the five. That's because when
27:18:39we exported our data set, it brought
27:18:42along this index right here. What we can
27:18:45do is let's overwrite the that previous
27:18:47one. Let's put a comma here and we're
27:18:50going to do row.names.
27:18:53This says whether or not we want that
27:18:55index to be written with it or not. and
27:18:57we're going to say false. So, let's
27:18:59overwrite that file. And then let's read
27:19:02this in again. And let's go take a look
27:19:05at our dataf frame 3. It no longer has
27:19:07that extra column with the index. We got
27:19:10rid of that when we wrote it to the
27:19:11file. So, this is the basics of working
27:19:14with files. It of course goes more in
27:19:16depth. You can connect to APIs. You can
27:19:18connect to databases. You can connect to
27:19:21a lot of different file paths. But this
27:19:23is the basics of how you can read and
27:19:25write and create dataf frames with
27:19:27files. So, thank you guys so much for
27:19:29watching. In the next lesson, we're
27:19:30going to see how we can select and order
27:19:32our data that we pulled into a dataf
27:19:34frame.
27:19:36[music]
27:19:47Hello everybody. In this lesson, we're
27:19:48going to see how we can select and order
27:19:50our data that we're pulling in from a
27:19:52CSV that is in a data frame. Now, what
27:19:54we need to do is we have to first pull
27:19:56in our data set. If you haven't been
27:19:58following this series, then you can get
27:20:00this data set down below. It'll be in
27:20:02the GitHub. You can just download it,
27:20:04put it into file path. And if you don't
27:20:06know how to read in a file, check out my
27:20:07last lesson because that's how you
27:20:09actually work with files, which is right
27:20:11here. You can see how to write files and
27:20:13read in files. Now let's read in this
27:20:17data frame and let's open it up. So now
27:20:20we have columns like character,
27:20:21department, ro, annual salary, dogs
27:20:24rescued with three legs. Very specific
27:20:26column here. But let's start out with
27:20:28just seeing how we can select only
27:20:30specific columns. Then we'll look at
27:20:32only specific rows. And then what we're
27:20:35going to do at the end is we'll see how
27:20:36we can order different columns. So let's
27:20:38come back here. And what we're going to
27:20:40do is we're actually going to pull in a
27:20:42library here. And let's go over here to
27:20:45the library. And we want to pull in a
27:20:46library. Now, if we come over to the
27:20:48packages, we don't have a ton pulled in.
27:20:51Right? As we're going through here,
27:20:53there's a lot that will, you know, you
27:20:55might want to use that are not in here.
27:20:57And one specifically that we want is
27:21:00dlier. And that's how I pronounce it at
27:21:02least. Uh, but it's dpl.
27:21:07This package is specifically made and
27:21:10designed for people like data analyst,
27:21:12data scientist to manipulate and select
27:21:14and query and work with data. So we are
27:21:17going to go ahead and we're going to run
27:21:18this. Now it's saying there is no
27:21:20package called dlier. That's not true.
27:21:23There is a package called dlier. We just
27:21:25need to install it. So we're going to
27:21:27come up here. We'll go right above it.
27:21:28We're going to do install.packages.
27:21:33There we go. And then we're going to do
27:21:36dlier. I got to spell that right. I
27:21:38deleted everything that I had in R and
27:21:41started from scratch at the beginning of
27:21:42the series. So I need to install some
27:21:44packages as well. As you can see, we
27:21:47just installed dlier. And because of
27:21:49that, we have a lot of different options
27:21:50that we didn't have before. And
27:21:53specifically, we have this one right
27:21:54here. This is the grammar of data
27:21:56manipulation. If you click on this, you
27:21:59can come in here and you can see all
27:22:01that it can do. And it can do a lot of
27:22:03things. and we'll use a lot of these in
27:22:05this series. And so now that we have
27:22:07that, we're going to go ahead and we're
27:22:09going to uh pull in that library. We
27:22:11could also just click on this and it
27:22:15would be ready, but I still want to
27:22:17write it in the code. So now we've
27:22:19pulled in that library and we can start
27:22:21using things that are within that
27:22:23library. I'm going to come back here
27:22:25just to make it look nice again. Now one
27:22:27of the functions that's within dlier is
27:22:30select. Let's say we want to only bring
27:22:33in the character and the role column.
27:22:36Those are the only ones we want right
27:22:38now. We can do that by saying select and
27:22:42we're going to open this up. We're going
27:22:44to pass through our data frame. And now
27:22:46we can select what columns we want. So
27:22:48we're going to say character and roll.
27:22:52And let's go ahead and run this. Now,
27:22:55we're getting our output right now as
27:22:58just an output in the console, which is
27:23:01perfectly fine. What a lot of people
27:23:03will do is when they're running these
27:23:04select statements is they're going to
27:23:06assign this to its own data frame, which
27:23:09is fine. Here's what I will say, and I'm
27:23:12going to start doing it. But here's what
27:23:13I will say. Your data right here and
27:23:15what you're storing in your memory is
27:23:16going to exponentially increase. It's
27:23:18going to be kind of hard to work with
27:23:20all these different dataf frames, but
27:23:22I'm going to say dataf frame characters
27:23:25and we're going to assign this
27:23:27and then we're going to run it. So now
27:23:30I've created a new dataf frame with only
27:23:32uh two columns. Now we have character
27:23:35and ro. So we're able to just take the
27:23:39columns that we want. We don't have to
27:23:40take all of them. Now let's say we go
27:23:42back here and we actually want all these
27:23:45columns except for this last one. This
27:23:47last one we don't really need at all.
27:23:49And so what we're going to do is we're
27:23:51going to take that and we're going to
27:23:53just come down here and we'll do the
27:23:55same thing. We'll pass through our data
27:23:57frame and we're going to say minus which
27:23:59means we don't want this column. And
27:24:02let's see what that's called again. Dogs
27:24:03rescued with three legs. So I'll say
27:24:06dogs
27:24:07rescued
27:24:10with three legs. We'll see if uh I
27:24:16actually think this is capitalized. But
27:24:18now let's run this. And now you're going
27:24:20to see all of these columns except the
27:24:23one that we didn't want. Now this seems
27:24:25silly. This seems very, you know,
27:24:26simple, at least the way I'm looking at
27:24:28it because we're just getting rid of one
27:24:30column. But in the real world, when I
27:24:32have been working with data in the past,
27:24:34I'll get data from a client and every
27:24:36single time they send these columns that
27:24:39I just don't care about. I don't need.
27:24:41there's no reason they're in there and
27:24:42they take up space, they fill up your
27:24:44database or they take up your memory,
27:24:45whatever it is, and you just don't want
27:24:47to see it. This would be an example of
27:24:49one that I'm just like, okay, get rid of
27:24:50this column and then we can start
27:24:52looking at it. So then I would assign
27:24:54this to another data frame and then I
27:24:56would only work with that new data
27:24:57frame. I wouldn't work with the original
27:24:59data frame because that one has that
27:25:01dogs rescue with three lengths column
27:25:03that we never needed in the first place.
27:25:05We could also get this exact same output
27:25:07by doing something a little bit
27:25:09different. And what we would do is we
27:25:12would say character and then we do this
27:25:15little colon here and that means through
27:25:18and then I believe the last one let's
27:25:20get rid of this one is annual salary. So
27:25:23then I would do annual
27:25:26salary. And what this does is it says I
27:25:28want this column through this column.
27:25:31And then if we run this, it's again not
27:25:33going to include that last one because
27:25:35it is not in between character and
27:25:37annual salary. So that's how we're able
27:25:39to specify what columns we want. But I
27:25:42think more importantly is actually
27:25:44selecting what data we want. So what
27:25:46rows of data we actually want to keep in
27:25:49our output. So let's come down here and
27:25:51we're going to say filtering. Now
27:25:54filtering data is super important. You
27:25:56don't always want all the data or you
27:25:58want some subsection of the data. So
27:26:01what we can do is we're going to come
27:26:02right down here and we're going to say
27:26:04filter. And filter is going to allow us
27:26:06to create a specific condition. If it is
27:26:08met, then those rows will be returned.
27:26:11If it is not met, then it will not be
27:26:14returned. So again, we're going to pass
27:26:16through our data frame. But for this
27:26:18one, let's take a look where annual
27:26:20salary is greater than and let's say
27:26:2450,000. So we're going to do 501 23.
27:26:28So we're only going to filter on annual
27:26:30salary where it's greater than 50,000.
27:26:34Now if you look in our output, we have
27:26:35an annual salary here and none of them
27:26:37are going to be lower than 50,000. This
27:26:40is using our comparison operators here.
27:26:42So we have other comparison operators
27:26:44that we can use. For example, we could
27:26:48do right here. We want to say where the
27:26:51role is equal to and now we're looking
27:26:52at a string. We can just say they're a
27:26:55director. And if we run this, there's
27:26:58only going to be one person who's a
27:26:59director, and that's Ron Swanson. Now,
27:27:02you'll notice that we also have a
27:27:04different director. We have Ron Swanson,
27:27:05who's our director, but also Leslie Nope
27:27:07is technically a director, too, but it
27:27:10isn't a specific match here. Now, there
27:27:13is a specific function that is kind of
27:27:16like regular expression. It searches for
27:27:18a specific pattern. It's called grapple.
27:27:21And we can use that. We can say grapple.
27:27:25and we're going to pass through
27:27:26director. Now the only thing we have to
27:27:28do apart from that is we're also going
27:27:30to specify what column we're looking in
27:27:33which is RO and we'll close our
27:27:36parenthesis. So now we're looking for
27:27:38the word director in this RO column. If
27:27:42it's anywhere in there it's going to
27:27:43return it. So now if we run this it's
27:27:47not just director. Now we have deputy
27:27:49director as well because it was just
27:27:51searching for that keyword. And so if it
27:27:53contains director anywhere in this RO
27:27:56column, we are going to find it. Now
27:27:58when we're filtering data, we typically
27:28:00aren't just filtering on one thing or
27:28:02you know sometimes you are but sometimes
27:28:03we have multiple things. Let's go ahead
27:28:06and let's put this filter down here. I
27:28:09should have kept the uh previous one.
27:28:10I'll write it again. So we're going to
27:28:12do where the annual salary is greater
27:28:15than 50,000 and that's 5,000. And let's
27:28:19just run this. But we also only want to
27:28:22see where their department is parks. So
27:28:25what we're going to do is we're going to
27:28:27use a logical operator which is our
27:28:29amperand. We're going to say where the
27:28:31annual salary is greater than 50,000 and
27:28:35then we'll create another condition.
27:28:37We're going to say where their
27:28:38department is equal to
27:28:42parks. Now let's run this and you'll see
27:28:46now we have all the annual salaries
27:28:48greater than 50,000 and where the
27:28:50department is equal to parks. So we can
27:28:52have multiple conditions in here and we
27:28:55can either have an and or an or in order
27:28:58to specify what we're looking for. Now
27:29:00right now we're just filtering and
27:29:01before we were just selecting. Let's see
27:29:03how we can combine these. And there's
27:29:06going to be a little bit more advanced
27:29:07and we'll get to a lot of this in the
27:29:09next several lessons.
27:29:10What we're going to do is we're going to
27:29:12use something called a pipe operator. So
27:29:14we're going to take our dataf frame and
27:29:16we're going to build our pipe operator.
27:29:18This says take the data frame and then
27:29:22so what are we going to do next? We're
27:29:25going to select just this data set.
27:29:29So let's bring this back. So we're
27:29:32taking our data frame and then we're
27:29:34only selecting character through annual
27:29:37salary. So we don't have that last
27:29:39column. And then we're going to add this
27:29:41pipe operator as well. And then we'll
27:29:45say do this.
27:29:48And let's format a little better. This
27:29:50is kind of how you add multiple things
27:29:52on top of another. So we're going to
27:29:54take our data frame. We're going to say
27:29:56and then select only these columns. And
27:29:59then we're going to filter it down like
27:30:01this. So let's run this right here. And
27:30:04actually we're getting an error because
27:30:06uh we don't need to pass through these
27:30:10data frames anymore because we are
27:30:12already specifying at the very beginning
27:30:14take this data frame and then select
27:30:17just this data. If we do it if we keep
27:30:20passing through the data frame it kind
27:30:22of gets wonky because it's like yes I
27:30:24already know I'm supposed to be using
27:30:25the data frame. So it gets confused. But
27:30:27this right here is how we kind of chain
27:30:29these together. Now we only have these
27:30:31four columns and only with those
27:30:33conditions being met. Now the last thing
27:30:35that we're going to look at and I'm
27:30:37going to actually put this down here.
27:30:39I'm going to say uh this is our called
27:30:41our pipe operator. I'm going to put this
27:30:44down here. And up here I'm going to say
27:30:46ordering.
27:30:47And once we add ordering and see how we
27:30:49can order it, we're going to add it down
27:30:51here into kind of our chain is what
27:30:54we're going to call it. Now for ordering
27:30:55data, let's take this right here. We're
27:30:57going to look at our annual salary.
27:30:59Let's say we wanted to order it like
27:31:01this, lowest to highest. That's very
27:31:04easy to do when you're looking at the
27:31:06data in here, but there's a hund
27:31:08different reasons why you'd need to
27:31:09actually order it in your code. And so
27:31:11let's come over here and we're going to
27:31:13use arrange. Again, this is in within
27:31:15dlier even says it right here. This is
27:31:18how you order the rows of your data. So
27:31:21we're going to say arrange and what we
27:31:23need to do, of course, is pass through
27:31:24our data frame, but then we're going to
27:31:26specify what column do we want to order
27:31:28it on. We're going to say annual
27:31:31I need to spell right. I'm not spelling
27:31:34right at all. Annual salary. And then if
27:31:38we run this, you'll notice right up here
27:31:42that the annual salary is from lowest to
27:31:44highest. So by default it is ascending
27:31:48which means lowest to highest. But we
27:31:51can reverse that to do highest to
27:31:53lowest. Let's say uh let's do lower c
27:31:55lowerase actually. Uh let's pass through
27:31:58this annual salary. Give me one sec. So
27:32:02now we're saying take the annual salary
27:32:04but do it from highest to lowest. That's
27:32:06what descending means. It's going to be
27:32:08the exact same thing except the
27:32:10opposite. So we have lowest to highest
27:32:12here. So now let's go back to our pipe
27:32:15operator area where we've chained these
27:32:17all together. Now we have all this
27:32:18information, but I want to order it by
27:32:21annual salary right here. So, I'm going
27:32:23to take uh this and I'm going to say
27:32:28oops my pipe and I'm going to paste this
27:32:31in. But I have to get rid of this data
27:32:34frame again. So now I'm saying take our
27:32:36data frame, select these columns, filter
27:32:40on these conditions, and then arrange
27:32:43them with annual salary descending.
27:32:46Let's go ahead and run this. And there
27:32:48we go. So we have our output that we're
27:32:51looking for. and we filtered down by a
27:32:54ton of different stuff while selecting
27:32:55columns, rows, and ordering our data all
27:32:58in just one kind of easily readable
27:33:01code. And so that's how we select,
27:33:03filter, and order our data specifically
27:33:05using that dlier package. That is
27:33:07something that's very popular and very
27:33:09common to use within R. If you haven't
27:33:11already, be sure to check out my full R
27:33:12course on analybuilder.com. I will leave
27:33:14a link in the description with a coupon
27:33:16code if you are interested. Thank you
27:33:18guys so much for watching and I will see
27:33:20you in the next lesson.
27:33:24>> [music]
27:33:34>> Hello everybody. In this lesson, we're
27:33:35going to see how we can group and
27:33:37aggregate our data in R. Now, we're
27:33:40working with the exact same data set
27:33:41that we were in the previous lessons,
27:33:43but if you don't have it yet, then you
27:33:45can get this data set down below in the
27:33:47GitHub. You just have to download it and
27:33:48you'll be able to work with it along
27:33:50with me. Now, we also need this library
27:33:53dlier. Let's go ahead and run both of
27:33:55these. So, we've loaded our library and
27:33:58we've read in our file. Let's open up
27:34:01our file so we can take a look. Here we
27:34:03have our character. We have the
27:34:05department that they work in, their
27:34:06role, their annual salary, and dogs
27:34:09rescued with three legs. What we're
27:34:13going to be doing is we're going to be
27:34:14grouping on our data and then looking at
27:34:16aggregations. For example, we have a lot
27:34:19of people who work in the parks
27:34:21department. And let's say we want to
27:34:23know what's the average salary of
27:34:25someone who works at the parks
27:34:26department. Right now, just looking at
27:34:28this data, we don't know. We could kind
27:34:30of guess maybe it's, I don't know, like
27:34:3350ome thousand, but we don't know for
27:34:35certain. And so that's what group by and
27:34:38aggregations in general within R are
27:34:40used for. Let's go in here and let's see
27:34:43how we can write this. In the last
27:34:46lesson, we learned about the pipe
27:34:47operator, and we're going to be using
27:34:49that again. So, we're going to write it
27:34:50just like this. We're going to take our
27:34:52data frame, and then we're going to
27:34:54write our group by first. So, the group
27:34:56by says we want to take a specific
27:34:58column, put it all into one row, and
27:35:01then perform an aggregation on it. So,
27:35:02our group by is grouping the values
27:35:06right here in this department. So, let's
27:35:08say group, and we're going to do
27:35:09underscore by, and we're going to do
27:35:11this based off of the department. And
27:35:13then we'll use another pipe right here
27:35:16and we'll go down. Now we need to
27:35:18actually aggregate our data. Now we do
27:35:20this with summarize. Summarize allows us
27:35:24to use some median whatever it is and do
27:35:28it on whatever column we choose. Now it
27:35:30has to be except if we just use a count.
27:35:33Typical aggregations need to be on some
27:35:34type of numeric column. Now I'm going to
27:35:37show you the one where we don't have to
27:35:38do that. That would just be n. That's
27:35:41going to be account. And let's run this.
27:35:44So you're going to see right down here
27:35:45in our output we have city management,
27:35:47health, and parks. So we have two, one,
27:35:50and then seven. This is just getting a
27:35:53count. So we don't have to specify a
27:35:57column that's numeric because we're just
27:35:59counting the rows. We're not counting a
27:36:01specific column. You'll also notice that
27:36:04we have this n and then in parentheses
27:36:07for our column name. If we were to save
27:36:09this to a dataf frame, then it would
27:36:11have n with the uh parenthesis like
27:36:14that. We can change that by saying count
27:36:16is equal to. So we're going to assign
27:36:18these counts and sign it the name of
27:36:21count. So if you run this now, we have
27:36:24count right here. Now that is the most
27:36:27simple one. It's just a classic count.
27:36:30But let's take a look at let's say an
27:36:32average. So we're going to copy this.
27:36:35And now we're going to get the average
27:36:37of the salary. So we're going to say
27:36:40mean and mean means average uh within R.
27:36:44And then we're going to do annual
27:36:47salary. Now if we run this, we're going
27:36:51to take a look at the average salaries
27:36:53within each department. So within our
27:36:56parks departments, 54,571.
27:36:59Health only has one person as we know.
27:37:01So that's going to be 60,000. And then
27:37:04city management is 90,000. This is
27:37:07something that is really important when
27:37:08you're doing aggregations because right
27:37:10here, if you only had this information,
27:37:13you don't really know how many people
27:37:16are in each of these departments. Maybe
27:37:18it's only two people, maybe it's three,
27:37:19maybe it's 100, maybe it's 10,000. It is
27:37:22important to know when you're doing
27:37:23aggregations, especially with something
27:37:25like an average, how many numbers or how
27:37:28many data points are behind these
27:37:29numbers. we are actually able to add
27:37:33multiple. So if I put a comma there,
27:37:36then I go right down here and I say
27:37:38count of is equal to the n. I can run
27:37:42this. And in this aggregation, it
27:37:45doesn't change the department or the
27:37:46annual salary. But now we have an
27:37:48additional aggregation that we've added.
27:37:51And we can do that for as many as we'd
27:37:52like. We can do three, four, five, six,
27:37:55seven different types of aggregations.
27:37:56That would be perfectly acceptable. So,
27:37:59let's take a look at some other
27:38:00aggregations, and we're just going to
27:38:01add them in here. And we can always name
27:38:03them. This one, of course, says mean
27:38:04annual salary, but I could call this uh
27:38:07average
27:38:11salary is equal to,
27:38:14and I need to find that equal to. And
27:38:17then when we run this in just a second,
27:38:18it will work. But we have other types of
27:38:20aggregations. We can do the minimum of
27:38:24our annual salary. And if we run this,
27:38:26it'll give us the smallest salary within
27:38:30each department. So we have department
27:38:33average salary right here, the count,
27:38:35and then we have our minimum salary.
27:38:37Here's the minimum amount that someone
27:38:38makes in each of these. Conversely, we
27:38:41can also do the maximum salary. Let's go
27:38:43ahead and
27:38:46do the max. And we're just going to keep
27:38:48stacking these aggregations all on the
27:38:50department. But this is the most that
27:38:52somebody makes within that department.
27:38:54Now, we also have another interesting
27:38:56one, and this is one that I think gets
27:38:58overlooked a lot, but that's going to be
27:39:00median. Median gets the exact middle
27:39:02point of your aggregation. So, let's put
27:39:05in annual salary here. And I'll explain
27:39:08this in just a sec. I'm actually getting
27:39:10a few too many of these aggregations.
27:39:12Let's put this into a data frame. So,
27:39:13I'm going to say aggregation data frame.
27:39:16We're going to assign this. And let's
27:39:18run this right here. And let's open it
27:39:21up. So, here we have our median annual
27:39:25salary and this says 90,000. Let's go
27:39:28back and let's take a look. Let's just
27:39:31take a look at city management. This
27:39:34only has two values. If it had three
27:39:37values, it would take the one in the
27:39:39middle, but we only have two. So, what
27:39:42it does is it takes those two middle
27:39:44points. It takes the average of those
27:39:46two middle points, which is going to be
27:39:4790,000. Now, 60,000, we already know
27:39:50what happens there. But 52,000 is the
27:39:53one that's interesting to us. So let's
27:39:55order by the salary, then the
27:39:57department. So now we have 25,000 all
27:40:00the way up to 90. So we have 1 2 3 4 5 6
27:40:047. 7 is an odd number, which means it
27:40:06has a middle point, which is going to be
27:40:08four. So if we go 1 2 3 and four, this
27:40:12is the middle point of our data. If we
27:40:15go back to this data frame, 52,000 is
27:40:18our median right here. Now, as I've
27:40:20gotten more into data analysis and
27:40:22working with numbers, I find myself
27:40:24using median a lot because average has
27:40:27the possibility, it doesn't always, but
27:40:29it has a possibility of skewing some
27:40:30numbers. Let's say, for example, in our
27:40:33data frame, we had Ron Swanson and he's
27:40:36making $10 million or $100 million.
27:40:40If we went and aggregated and just
27:40:42looked at the average, the average would
27:40:44be like $20 million or something like
27:40:46that. I don't know. I'm just throwing
27:40:47out a number, but the average would be
27:40:49insanely high, but only one person
27:40:52actually makes that much in the
27:40:53department. If we took a look at the
27:40:55median, the median is still going to be
27:40:5852,000 because that's the middle point
27:41:00of the data. And so depending on what
27:41:02type of analysis you're doing, depending
27:41:04on what you're looking for, average and
27:41:06median can show you and give you
27:41:07different insights into the data. So,
27:41:09it's just one to be aware of and to use
27:41:11wisely. Let's go back and take a look at
27:41:14our data. And so that is how we group
27:41:17and aggregate our data. And again, you
27:41:19can do multiple aggregations. If you
27:41:21want to go even more in depth on
27:41:22grouping and aggregating data within R,
27:41:24I have a full R for data analytics
27:41:26course on analystbuilder.com. I will
27:41:28have a link in the description if you
27:41:29want to check it out, as well as a
27:41:31coupon code just in case you want to
27:41:32take it. With that being said, I hope
27:41:34that this was helpful. I hope that you
27:41:35learned something and I will see you in
27:41:36the next [music] lesson.
27:41:50Hello everybody. In this lesson, we're
27:41:51going to see how we can handle missing
27:41:53data within our data set. Now, in order
27:41:55to do this, we're going to be working
27:41:56with a new data set. It's called the
27:41:58messy data set. You can get that down in
27:42:00the GitHub. I'll have a link in the
27:42:02description. We're going to need a new
27:42:04library for this called TidyR. This is
27:42:06part of the tidyiverse and they have
27:42:07functions that are specifically created
27:42:09for cleaning up data. And so we're going
27:42:11to be using that in the next several
27:42:12lessons as we look at different things
27:42:14that we need to do in order to clean up
27:42:16data. If you haven't already installed
27:42:18it, you can just install it like this.
27:42:20It's install.packages and then tidyverse
27:42:23in quotes. And then you'll have access
27:42:25to tidyr.
27:42:27It'll look just like this. But I'm going
27:42:29to go ahead and get rid of this because
27:42:31I have it right down here for you. Now,
27:42:33let's go ahead and bring in our data
27:42:35set. We'll take a look at it. So, I'm
27:42:36just going to run everything really
27:42:37quickly, and we're going to take a look
27:42:40at our data set. So, going through our
27:42:41data really quickly, we have customer
27:42:43ID, customer name, email, transaction
27:42:46amount, transaction date, and category.
27:42:49Now, when I'm looking through this, I
27:42:51can just at a glimpse see very easily
27:42:54that there's several issues. One, we
27:42:56have some null data. So, there isn't any
27:42:59data in here. Now, these look a little
27:43:01bit different than this in the
27:43:03transaction amount, numeric versus
27:43:04string. We'll take a look at that in a
27:43:06little bit. We also have a transaction
27:43:08date, and these are all different types
27:43:10of formats, and that doesn't look good.
27:43:11So, we need to fix that up. We also have
27:43:13a category, and in this category, we
27:43:15have capital electronics, lowercase
27:43:17electronics, and maybe some different
27:43:19spellings as well in here. So, we need
27:43:21to clean these things up. These are all
27:43:23things that if I was working with this
27:43:25in a real data set, I would be taking a
27:43:27look at these, trying to standardize
27:43:29dates, as well as take a look at this
27:43:31data and see, do we want to do something
27:43:33with NLES or the blanks? Cuz sometimes
27:43:35we do, sometimes we don't. So, let's
27:43:37come right back over here. We're not
27:43:38going to get to everything in this
27:43:40lesson because, you know, they do have
27:43:42different use cases. We're just going to
27:43:43be looking at missing data. So, to get
27:43:47started, let's come right down here. And
27:43:49what we're going to do is we're going to
27:43:50check our null values within our data
27:43:53set. So what we're going to do is we're
27:43:54going to do call and then sums and we're
27:43:56going to take a look and it's basically
27:43:57just going to summarize uh we're going
27:43:59to do is na and then pass to our data
27:44:02frame. We're just going to summarize for
27:44:04each column which ones have no values or
27:44:07blank values. Let's go ahead and run
27:44:09this. Now you'll notice on here we have
27:44:11customer ID, customer name, email,
27:44:13transaction amount, but we have no
27:44:15blanks in these columns. But if we look
27:44:19back here, we know that in customer name
27:44:21and in email, we have blanks. Uh we see
27:44:24them, but it's only showing up for
27:44:27transaction amount. Now, that has to do
27:44:29with just how it was pulled in as a data
27:44:32set. Now, we can actually fix this very
27:44:35easily by coming in here and passing
27:44:37through a parameter called NA.
27:44:43This is going to take the blank fields
27:44:45and make them NA. So, we're just going
27:44:47to hit tab and we're going to pass
27:44:49through a vector here and we're just
27:44:50going to say if it's blank then, comma,
27:44:55make it NA. That's all we're doing. So,
27:44:57for these blank strings, if they're
27:44:59blank, we want to make them NA. Let's
27:45:02read this in again. We'll overwrite our
27:45:04previous data set. Now, let's look at
27:45:06our data frame. And now, you'll see that
27:45:08these are NA as well. This is just
27:45:10something that sometimes you have to do
27:45:12kind of somewhat manually. you have to
27:45:14specify you want that to be done. But
27:45:16now if we run this, you'll see we have
27:45:19one, one, and two. So this is a lot more
27:45:21accurate. Now, within this data,
27:45:24sometimes what you're going to want to
27:45:25do is just get rid of it. So for
27:45:27example, let's say we're creating an
27:45:29email list and we say we have to have
27:45:31this email available, otherwise this
27:45:33data is just not useful to us at all. So
27:45:36what we want to do is we want to
27:45:37actually get rid of this data. So, what
27:45:40we're going to do is let's come right
27:45:42down here and let's create we'll do
27:45:45dataf frame cleaned. We're just going to
27:45:46create a new dataf frame. But all we're
27:45:49going to do is we're going to drop that
27:45:52row if it is blank. So, we're going to
27:45:54say uh we can take the data frame and
27:45:57we'll use a pipe here. Then we're going
27:45:59to say drop na. And that's right here.
27:46:02And you can see it uses this tidy r. And
27:46:05it's just going to drop the rows where
27:46:06any column specified contains a missing
27:46:08value. So, we're going to come here and
27:46:10we're going to specify the email column.
27:46:12And if you remember, we have this email
27:46:14right here. So, let's go ahead. Let's
27:46:17run this. Let's take a look. You'll
27:46:19notice we have one less row. Let's take
27:46:21a look at the data frame cleaned. And
27:46:23we'll compare. So, now we only have 11
27:46:27rows of data compared to the 12 before.
27:46:30And now we don't have this customer ID
27:46:32109, which is uh Helen Carter. We have
27:46:36109 completely gone. And so we just got
27:46:39rid of that data. That is a perfectly
27:46:41acceptable thing to do depending on the
27:46:43use case for your data. Our use case was
27:46:45we're creating an email list. It's
27:46:47pretty hard to give an email list to
27:46:50somebody to email out if that person
27:46:51doesn't have an email. So we just got
27:46:54rid of it. And that is a perfectly
27:46:56acceptable thing to do. And I'm actually
27:46:58going to put up here. I'm going to say
27:46:59uh remove rows when no email is present.
27:47:06Now another thing that we could do is we
27:47:09could actually fill in this data. So for
27:47:12example we have transaction amount and
27:47:15let's say we need to use this
27:47:17transaction amount. For example in
27:47:19transaction amount let's assume that
27:47:21when it has na it's actually a zero. It
27:47:24means there was zero transaction amount
27:47:27paid here. Maybe they got a full
27:47:29discount but this would be contextual.
27:47:31It would be we know that when it says NA
27:47:33it's supposed to be a zero. So we just
27:47:35want to populate this with a zero. Now
27:47:37this makes a big difference when you
27:47:39start doing aggregations on it. Now we
27:47:41looked at aggregations and grouping.
27:47:43When you have a zero in here, that zero
27:47:44is going to be counted in that
27:47:46aggregation. And so if you just did 0
27:47:49and 99, then the average is going to be
27:47:52about a 50. If we have NA here, this
27:47:55does not count towards the aggregation.
27:47:57So it would just be 99.99.
27:48:00So let's go ahead and do that first
27:48:01option. Let's go through here and let's
27:48:04populate these with zeros. Let's say
27:48:06that's our use case for what we're
27:48:08doing. So, we need to take this column,
27:48:11this transaction amount. So, we're going
27:48:14to come right down here. We're going to
27:48:15say data frame. Then, we're going to do
27:48:17a dollar sign, and this is just a
27:48:18shortcut of specifying the column name.
27:48:20We're going to do transaction amount.
27:48:22And we just want to say when the
27:48:24transaction amount is null, then
27:48:27populate it with zero. So, we're going
27:48:29to create a bracket here. We're going to
27:48:31do is na and we're going to pass through
27:48:34this exact column. So we can just copy
27:48:36this. So we're saying when this column
27:48:38is blank then what are we going to do?
27:48:41We're going to populate it with a zero.
27:48:43And that's it. So let's go ahead and
27:48:45actually real quick let's do this on
27:48:47dataf frame cleaned. Glad I caught that.
27:48:49Otherwise we'd be doing that to our
27:48:51original dataf frame. So let's go ahead
27:48:53and run this and let's go back to our
27:48:56dataf frame cleaned. And you'll see 0.00
27:48:590 0 and 0.00.
27:49:02And that is fantastic. When we go to
27:49:04aggregate this, these zeros are going to
27:49:06be counted because now they're numbers.
27:49:08They're not null. Now, sometimes that's
27:49:10not what we want to do. Maybe we know
27:49:12that NA means it just didn't take the
27:49:15data incorrectly, but we want to count
27:49:17it as an actual sale. We know that it
27:49:20wasn't zero. It's a nonzero number.
27:49:22Maybe it was 50, maybe it was 100. We
27:49:24don't know. But we don't want to
27:49:26populate it with a zero because that's
27:49:27going to bring down our averages and we
27:49:29know that can't be correct. So what if
27:49:31we just wanted to populate it with the
27:49:33average value? We can definitely do that
27:49:36as well. Let's come back here and let's
27:49:39redo this really quick. Let's overwrite
27:49:42and we're going to go like this. It
27:49:45should bring us back to how it was
27:49:47before. So this is just the first option
27:49:50to populate
27:49:53null
27:49:55numeric values.
27:49:58Now we'll have our second option. So
27:50:00we're going to take this column again,
27:50:02this data frame cleaned. And in fact, we
27:50:05need this whole thing. So let's just
27:50:06copy this down.
27:50:08And we're going to populate it by going
27:50:11like this. And now we're going to say
27:50:13take the mean of the transaction amount.
27:50:16So we're going to say mean and then
27:50:18we're going to pass through
27:50:21just the transaction amount. So let's
27:50:22get rid of this and let's make sure we
27:50:26have our wrap on so we can see it all.
27:50:29So now we're taking the average of this
27:50:31column. Let's go ahead and run this and
27:50:35take a look. Now, it isn't populating
27:50:37it. And I have a feeling we need to pass
27:50:39through another parameter here,
27:50:43which is na.rm.
27:50:46It's basically going to say when you're
27:50:48taking this, make sure and check that
27:50:50it's the correct one. Otherwise, it may
27:50:52not evaluate it to NA when that is what
27:50:55we're looking for. So, we want to say
27:50:56that's equal to true. And we're going to
27:50:59run it just like this.
27:51:02And now we have these values populated.
27:51:04Now, if we go back at Charlie Brown, he
27:51:07had NA and so did Ian Brooks. So, let's
27:51:10come right here. We have 180.2656.
27:51:13180.2656.
27:51:15Now, when we run this, these people, Ian
27:51:18Brooks and Charlie Brown are going to be
27:51:20included in the aggregation, but they
27:51:22won't change the number if we're just
27:51:23looking at averages. Now, if we start
27:51:25doing other things and other types of
27:51:27aggregations, of course, it's going to
27:51:29change the output, but we're looking at
27:51:30specifically for averages. these people
27:51:32will be included and we can also run
27:51:34counts on them to know this is how many
27:51:36sales or counts we made. So that is very
27:51:38much an option, but you need to be
27:51:40really really confident that what you're
27:51:42filling in is accurate and useful for
27:51:44your analysis down the line. You don't
27:51:46want to put this in here and maybe just
27:51:48leave it in there because someone coming
27:51:50behind you is not going to understand
27:51:51that that's what that is. So you always
27:51:54want to keep an original data frame. You
27:51:56always want to kind of document what
27:51:58you're doing and why you're doing it
27:51:59because otherwise your analysis might
27:52:01not make sense down the line. Now, some
27:52:03other data that is missing is right
27:52:05here. This is a customer name. Now, we
27:52:08don't have this full customer name and
27:52:10maybe we don't have another data set or
27:52:12another part of our database where we
27:52:14can pull that in. And maybe we just want
27:52:16to pull in Emma. Maybe that's all we
27:52:18know and that's all we want to do. But,
27:52:21you know, people's emails aren't always
27:52:23accurate. So maybe we just want to fill
27:52:25this in with unknown. We know that the
27:52:28customer has a name. Everybody has a
27:52:30name. We just don't know it. And that
27:52:32isn't necessarily the most important
27:52:34part of the data. They can still reach
27:52:36out to this person via email and say,
27:52:38"Hey, customer or whatever it is." So
27:52:40we're going to put unknown here. And
27:52:42this would be something that we would
27:52:43have. And this is a small data set. But
27:52:46let's imagine we have a 100,000 rows of
27:52:47data. Maybe there's, you know, 200, 500,
27:52:50a thousand people with no customer name.
27:52:52we now can just specify that and so we
27:52:55keep it in the data set but it isn't
27:52:57just blank. And so let's see how we can
27:52:59do that. We're going to come here to
27:53:02dataf frame and let's actually pull it
27:53:04in just like this.
27:53:07Whoops. I'm just messing up here. Let's
27:53:09pull in this dataf frame cleaned. And
27:53:11now we're going to do customer name.
27:53:14Isn't that what it's called? Yeah,
27:53:15customer_enamed.
27:53:17And let's pull it in right here. Now
27:53:18we're going to do the same thing. We're
27:53:20going to say if it's blank. So, we're
27:53:21going to do is na and then we're going
27:53:24to push in here or or pass through in
27:53:27here our customer name just like we
27:53:29wrote it. If it's blank, then instead of
27:53:32a number, we're going to do unknown.
27:53:36And so that's it. We're checking if it's
27:53:38blank and we're passing through unknown
27:53:41if it's not. This again safeguards
27:53:43against other things you might do later
27:53:46on when working with this data set.
27:53:48Sometimes when you're working with nulls
27:53:49and you're putting it into a CSV or you
27:53:51put it into a data set, those get
27:53:52dropped. I know for I know for instance
27:53:56that if you put certain data that has
27:53:58blanks into something like my SQL, those
27:54:00rows just get dropped in some instances.
27:54:02And so you want to be careful about
27:54:04leaving things blank. Even though it's,
27:54:06you know, inconspicuous, it's just a
27:54:07customer name. We don't need it.
27:54:09Sometimes you just want to populate it
27:54:10just to make sure that if it's been put
27:54:12in other systems, it doesn't get dropped
27:54:14or deleted in some way. And so these are
27:54:16some of the ways that you can handle
27:54:18missing data within AR and within your
27:54:20data frame. We're going to call this one
27:54:22populating text or character
27:54:27columns. And so I hope that this was
27:54:29helpful. If you haven't checked it out
27:54:30already, I have a full course on R for
27:54:32data analytics. We go even more in depth
27:54:34into handling missing data as well as a
27:54:35bunch of other data cleaning techniques.
27:54:37I will leave a link in the description
27:54:38as well as a coupon code if you would
27:54:40like to check that out. But with that
27:54:41being said, thank you guys so much for
27:54:43watching. If you like this video, be
27:54:44sure to like and subscribe below and I
27:54:46will see you [music] in the next video.
27:54:48[snorts]
27:55:00Hello everybody. In this lesson, we're
27:55:01going to be taking a look at parsing and
27:55:03converting dates in an R dataf frame.
27:55:05We're going to be using this lubricate
27:55:07package right here. You can use library
27:55:09and lubricate to load in that package
27:55:12that we can use it. We're going to be
27:55:14using the exact same data set from our
27:55:16last lesson. If you haven't already
27:55:18gotten that from the GitHub, I'll have
27:55:20the GitHub below so you can go ahead and
27:55:21download it. And then we're going to
27:55:23import it exactly as we have it right
27:55:25here. This is what our data frame looks
27:55:27like. And we're going to be specifically
27:55:29looking at this transaction date. In
27:55:31here, you can see we have a ton of
27:55:33different formats. And that's not a good
27:55:35thing. uh that is a bad thing because if
27:55:37we want to use this, let's just say we
27:55:39wanted to use dates in order to create a
27:55:41line chart, so a time series plot or
27:55:44something like that. Well, if the data
27:55:46sits like this, it's not going to work.
27:55:48Or if you want to aggregate it on or if
27:55:50you want to parse it out or if you want
27:55:52to do anything, you can't really do that
27:55:54with how it sits right here. Now, this
27:55:56is a very exaggerated view of this. It
27:55:59normally doesn't look like this, but
27:56:01let's say, for example, you're pulling
27:56:03data from a database in two separate
27:56:05parts and you're putting into one output
27:56:06and you're outputting it to a CSV. I've
27:56:09seen it a 100 times where you have two
27:56:11different types of transaction dates.
27:56:12Ones like this and ones like this. And
27:56:15they both are technically dates and
27:56:17they're being stored, but they're being
27:56:18stored differently. And so, this is
27:56:19something that we want to clean up and
27:56:21we want to standardize so they're all
27:56:23looking the same in the same format. And
27:56:25Lubdate is very helpful with this. And
27:56:27what we're going to do is come right
27:56:28back here. We're going to run our code
27:56:32so that we pull in that data frame. And
27:56:34now we can start working with it. So
27:56:35we're going to specify. We're going to
27:56:37do data frame. The dollar sign is just a
27:56:39shortcut to specify our column. We're
27:56:41going to choose our transaction date.
27:56:43Now what we want to do is we want to
27:56:47first standardize all of them and then
27:56:50later we're going to parse them out into
27:56:53day, month, year, whatever we want
27:56:55because that can be very helpful as
27:56:56well. So the first thing that we're
27:56:58going to do is we're going to use parse
27:57:01date and let's see if it comes up right
27:57:03here. Parse date time. This is from the
27:57:05lubber date package as you can see right
27:57:07here and it basically is just a really
27:57:09helpful function to help standardize
27:57:11your dates. So let's go ahead and click
27:57:12on this and we need to pass through as
27:57:15one of the uh as one of our arguments
27:57:18you need to pass through what column
27:57:20we're actually working on. But now we
27:57:22need to pass through orders. So for
27:57:25parse date time this orders allows us to
27:57:27specify different formats that our dates
27:57:30are going to be in and then it
27:57:32standardizes them for us. So we're going
27:57:34to say orders here. We're going to say
27:57:36that's equal to. Now, let's make sure we
27:57:38have the soft wrap lines on uh just in
27:57:41case it gets long. Or we could just pull
27:57:43it down here. But what we need to do is
27:57:47we're going to pass through a vector and
27:57:50we just need to specify what different
27:57:52formats are we actually working with
27:57:54here because we have year, that's month
27:57:57and day. This one, and those are dashes,
27:57:59and then we have month, day, year with
27:58:02these forward slashes. And so we need to
27:58:05just kind of go through here and see
27:58:07what different formats we have and it'll
27:58:08standardize them. So let's do this first
27:58:11one. It's year, month, and day with
27:58:13dashes in between them. So we're going
27:58:15to do capital Y for the year, month, and
27:58:19day. And of course, this needs to be
27:58:21within quotes. Let's make sure we get
27:58:23that. Let me go back. There we go. But
27:58:26let's put a quote right here. And there
27:58:29we go. And now we can just do that and
27:58:31separate them by commas. Let's run this
27:58:33one actually
27:58:36and let's come in here and you'll see
27:58:39that now we have one and those are all
27:58:42standardized and all working correctly.
27:58:44We're getting a message down here saying
27:58:46seven failed to parse. This is just a
27:58:48message saying hey we did some of them
27:58:51correctly but there's a bunch that
27:58:53didn't work correctly and this is a very
27:58:54helpful hint for us. This is kind of a
27:58:56message that's saying hey you didn't get
27:58:58them all so be sure to go back. Now that
27:59:00isn't all of them. So, let's reread our
27:59:03data frame since we overwrote that. Uh,
27:59:06but let's look at our data frame and
27:59:07let's do some of these other formats as
27:59:09well. So, for that one, we're going to
27:59:12do a comma and then we're going to say
27:59:15month slash dayward slashyear and then
27:59:19we'll look at another one. So, let's
27:59:21come in here. We have another one with
27:59:23slashes or forward slashes, but this one
27:59:25is a different direction. We have year,
27:59:28month, and day. So, let's make sure we
27:59:30get this one as well. So we'll do year
27:59:32that's a forward slashmon and day. Now
27:59:36one thing and this is just I just know
27:59:38this is this right here isn't actually
27:59:41text. It is still a date although it has
27:59:44text in it. It's still being stored as a
27:59:46date properly but it has a different way
27:59:48it's displaying it. So on the back end
27:59:50it could actually look like this but
27:59:53it's displaying like this. So what we're
27:59:56going to do is let's run this one and
27:59:57see what happens. And we got one that
28:00:00failed to parse. So let's come back
28:00:02here. So this Grace Adams is the only
28:00:05one that failed to parse. Let's come up
28:00:08here. Let's run this again. And let's
28:00:11create a duplicate of this. We're just
28:00:12going to do dataf frame raw. And we're
28:00:15just going to pass through the data
28:00:16frame just so we have it. Um so we can
28:00:19compare. So we have our dataf frame raw.
28:00:22And if we go back the one that didn't
28:00:24work and let's run this one more time.
28:00:27Sorry about that. The one that didn't
28:00:29work was
28:00:33right here. So that's gonna be Grace
28:00:34Adams. Let's go through to Grace Adams.
28:00:37And that's because this is a different
28:00:39format than up here. Again, this is
28:00:41year, month, day. This is day, month,
28:00:45and year with dashes. So, let's come in
28:00:47here. We're going to say day- month
28:00:51dashy year. And we just need to rerun
28:00:54our data frame so that we can rerun this
28:00:56one one more time. Let's go ahead and
28:00:57run this. And now we're not getting any
28:01:00error message. So now if we go and we
28:01:02look at our data frame, this looks just
28:01:05beautiful. I mean, that is a thing of
28:01:07beauty. Let's compare it to how it was
28:01:08before. And I'm just going to go back
28:01:11and forth. You can see that this one is
28:01:14much more preferred, right? We now have
28:01:16everything in one format that we want.
28:01:18Now, after we get this, we can also
28:01:21parse these dates out. So let's get rid
28:01:24of this data frame raw. We don't need
28:01:26that anymore. Now we can take this
28:01:28transaction date and we can pull out the
28:01:31year, the month, and the day. And we can
28:01:33actually create new columns with this.
28:01:35So what we can do is we'll take data
28:01:37frame. But now we're not going to do
28:01:39transaction date. We're going to do
28:01:40transaction date
28:01:42let's do year. And then we're going to
28:01:46pass through. So for this new column,
28:01:48we're going to take this transaction
28:01:50date. But we only want the month. So
28:01:53let's just make this month. And I
28:01:55actually said year here. So let's
28:01:57actually do year. This year is going to
28:02:00extract just the year from here, which
28:02:04is 2024 for all of them, I believe. And
28:02:06it's going to make its own column. And
28:02:09we can do this exact same thing with the
28:02:12month.
28:02:14Whoops. And the day. So we can say month
28:02:18here. And then we have a function that's
28:02:20month that's going to extract just the
28:02:24month. And then of course we'll have one
28:02:26as well for day
28:02:29and we'll do day as well. So I'm going
28:02:31to run both of these.
28:02:34And if we come up here and go all the
28:02:36way to the right now we have this all
28:02:37broken out by the year, the month, and
28:02:40the day. Zuber Date makes it really easy
28:02:44to standardize and work with these dates
28:02:46because there's a lot of different
28:02:47reasons why you would want to actually
28:02:49parse these out. It just depends on what
28:02:51you're using it for in the future. And
28:02:53so, I hope that this was helpful. If you
28:02:56haven't already, I have a full course
28:02:57where we'll dive a lot more into the
28:02:59data cleaning process. So, if you want
28:03:01to check that out, I'll have a link down
28:03:02in the description as well as a coupon
28:03:04code if you would like to use that. With
28:03:06that being said, I hope you enjoyed this
28:03:07video. Thank you guys so much for
28:03:09watching. If you have not already, be
28:03:10sure to like and subscribe, and I will
28:03:12see you in the [music] next lesson.
28:03:26Hello everybody. In this lesson, we're
28:03:27going to see how we can remove
28:03:28duplicates from a data set. Now, if you
28:03:31have not been following along, you can
28:03:33get this messy data set down in the
28:03:34GitHub below. I'll have a link. You can
28:03:36just download the CSV and you are good
28:03:38to go. All we're going to need for this
28:03:40lesson is our dlier. Let's go ahead and
28:03:43run this whole thing. So, we get the
28:03:44library and the data frame. And let's
28:03:47open up our data frame. Now, at first
28:03:49glance, even if you've been using this
28:03:51data set in previous lessons when we
28:03:53were looking at parsing and converting
28:03:54dates as well as handling missing data,
28:03:57you may not have noticed that we have
28:03:58duplicates in here. We do. We have
28:04:00duplicates right down here. We have
28:04:02Alice Johnson and Alice Johnson. It just
28:04:05wasn't something that maybe I pointed
28:04:07out so maybe you just didn't notice. But
28:04:09we have another duplicate which is we
28:04:11have a customer ID here that's 104 and
28:04:13we have another customer ID that's 104.
28:04:15And typically within a data set like
28:04:17this, this is going to be our unique ID.
28:04:19This is our primary key. So we shouldn't
28:04:21have duplicates for a customer ID. And
28:04:24so there are a few things that we're
28:04:26going to be doing within this in order
28:04:28to make sure that we remove the correct
28:04:30duplicates from our data set. So let's
28:04:33come over here and the first thing that
28:04:35we are going to do is we want to check
28:04:37to see are there duplicate rows. So
28:04:41we're going to take our data frame and
28:04:43we'll just do it like this because it's
28:04:45a little easier to read. And we're going
28:04:46to do a distinct on it. So distinct,
28:04:50it's going to keep only unique or
28:04:52distinct rows from a data frame. And
28:04:55that's it. It's pretty simple. And I
28:04:57actually need to I don't think I wrote
28:04:59that right. Uh we're going to run it
28:05:01just like this. Keep it simple.
28:05:03And as we go up, you'll notice we still
28:05:06have the 104 and the 104 because it's
28:05:10looking for a distinct across all
28:05:12columns. So the only one that is the
28:05:14same across all columns is Alice
28:05:16Johnson. You'll see 101, Alice Johnson,
28:05:18Alice J. Same transaction amount, same
28:05:20transaction day, same category, same
28:05:23everything. But 104 David Lee is not the
28:05:27same as 104 Emma who made a different
28:05:29purchase. We got those confused. So the
28:05:32only true duplicate across all columns
28:05:34is going to be Alice Johnson. Now we can
28:05:37very easily remove the second Alice
28:05:40Johnson because we can come right here.
28:05:42We're just going to say dataf frame no
28:05:46spell duplicates right. If you guys have
28:05:49ever watched me, you know how terrible I
28:05:51am at uh
28:05:53at spelling. We're going to take our
28:05:55data frame. We're going to look at the
28:05:56distinct and that's going to get passed
28:05:58through into this new data frame. So
28:06:01let's run this. You'll see we have one
28:06:03less. And so now we have this saved.
28:06:06There's no more Alice Johnson. And so we
28:06:09are doing really good. But what about
28:06:13this 104 right here? Well, we can do it
28:06:15kind of the easy way, which is just
28:06:17saying, okay, we only want to keep one
28:06:19of these people. And so we can actually
28:06:22use this exact same syntax. And let's
28:06:26get it over here just so it looks nice.
28:06:28We'll do duplicates two here. And I'm
28:06:30going to say take the distinct but only
28:06:33of this one specific column customer ID.
28:06:36So we're going to pass through I need to
28:06:39spell this right. Customer ID. And when
28:06:41we run this, let's go ahead and run it.
28:06:46We're going to go like that. And now we
28:06:48don't have any duplicates in our
28:06:49customer ID. You're going to quickly
28:06:51notice though we don't have the rest of
28:06:54our data and that's not ideal. Let's
28:06:56come right here and we're going to use
28:06:59another argument which is keep all and
28:07:02we'll do keep all. And you can go back
28:07:04and check. It's just going to keep all
28:07:06the other columns. And so we just want
28:07:07to say true. By default it was false. So
28:07:10we'll run this again in duplicates 2. We
28:07:12now have all the columns and we got rid
28:07:16of that second one. Now, was that
28:07:18correct? I don't know. And there may be
28:07:22a better methodology to actually keep to
28:07:24actually determine which customer ID we
28:07:27want to keep because this one doesn't
28:07:28have a customer name. And if we go back
28:07:31to right here, this person made a
28:07:33purchase on 10 or the transaction date
28:07:35at least on 104 of 2024. And this person
28:07:39up here only made one on March, which is
28:07:4135, which is before it. So, you might
28:07:44want to incorporate some logic here to
28:07:47say we actually want to take the person
28:07:48with the most recent transaction date.
28:07:51And we can do that. And that's actually
28:07:53not super hard to do. So, let's come
28:07:56here. Let's put this right down here.
28:07:59We'll do we'll make this uh node
28:08:02duplicates three.
28:08:04We're going to take our data frame and
28:08:07let's tab over here. We still want to
28:08:10look at the customer ID and take the
28:08:12distinct customer ID. We just want to
28:08:14arrange it because it's taking it going
28:08:16top down. It's saying, "Okay, keep that
28:08:18one. If there's ever a duplicate down
28:08:20here, get rid of it." Which is what it
28:08:21did right here. It said, "Okay, keep
28:08:23this one. Oh, we have a duplicate. Get
28:08:25rid of it." So, what we want to do is we
28:08:27want to order this. And so, we're going
28:08:29to do an enter here. And let me just
28:08:32bring it over here so it looks nice.
28:08:34We'll do arrange. And we want to arrange
28:08:37this by that transaction date. We want
28:08:40to do it by the customer ID first and
28:08:42then the transaction date. But we want
28:08:45to do this descending from highest to
28:08:48lowest. That's the most recent. So we
28:08:50want to say take the most recent person.
28:08:53Then we'll add our pipe here. And now
28:08:56when we run this, we get an error
28:08:58because this is all caps. All right,
28:08:59let's try this again. I got excited and
28:09:02let's go look at this uh no duplicates
28:09:04three and unfortunately we're getting
28:09:06the same thing and that's because of a
28:09:09transaction date issue. Uh we're going
28:09:11to go back and if you haven't done this
28:09:14already or you haven't taken that lesson
28:09:16on parsing and converting dates you we
28:09:18need to do that. That is really
28:09:20important actually. So we're going to um
28:09:22clean this data up a little bit. I'm
28:09:23going to place it right here. So you'll
28:09:25have that in the code in the GitHub. But
28:09:27I'm going to take this. I'm going to run
28:09:29this. And if we go and look at our
28:09:33original data frame. Now we have
28:09:34standardized these dates. So now when we
28:09:38run this and we take a look at the
28:09:40duplicates three, now it's taking David
28:09:43Lee. It was not working properly because
28:09:46our dates were not properly cleaned. And
28:09:49so before they looked horrible. And I
28:09:52think we should be able to see this
28:09:54right here. These dates are all over the
28:09:56place and so it didn't know how to order
28:09:58them properly until we cleaned them.
28:10:01Now, I'm not going to go into the data
28:10:02cleaning process for the dates because
28:10:04that was our last lesson in this series.
28:10:05So, you can go and check that out if you
28:10:07want to. Again, I'll have this code in
28:10:09the GitHub so you can just copy and
28:10:11paste this if you'd like. But, it is
28:10:13worth noting that you can go learn how
28:10:14to do it in that last lesson. But then
28:10:16once we clean that up and we standardize
28:10:18them all properly, we order them from
28:10:21highest to lowest, which is going to
28:10:22look like this. So these are the most
28:10:25recent all the way down to the old ones.
28:10:28And then we kept the distinct customer
28:10:32ID. And so now our output looks a lot
28:10:35better. We now only have one Alice
28:10:36Johnson. And now we kept David Lee
28:10:38because he had the more recent
28:10:40transaction date. Now for this use case,
28:10:42that logic works. But you need to figure
28:10:44out the logic for your actual data set
28:10:46because maybe that doesn't make sense.
28:10:48Maybe you want the original person and
28:10:50this new person needs to get a new
28:10:52customer ID. And you can do that by
28:10:53maybe taking the max customer ID and
28:10:55adding one to it. I don't know. Depends
28:10:57on what you're doing. But that is how we
28:10:59can remove the duplicates and we can
28:11:01make sure at least follows some type of
28:11:02logic that we can actually document and
28:11:04hand off and people can understand. So I
28:11:07hope that you learned something. And if
28:11:09you want to dive into data cleaning and
28:11:10using R even more, I have a full course
28:11:12on my platform analyst builder. I will
28:11:14have a link in the description. But I
28:11:15hope that this was helpful. If it was,
28:11:18be sure to like and subscribe below and
28:11:20I will see you in the next video.
28:11:23>> [music]
28:11:33>> Hello everybody. In this lesson, we're
28:11:35going to be taking a look at data
28:11:36visualization and then creating a
28:11:38presentation. This is kind of like a
28:11:40little mini project and kind of shows
28:11:41you how to put all your visualizations
28:11:43into one output, which can be really
28:11:46helpful. Now, what we're going to do is
28:11:47work with a new data set here. We're
28:11:49going to be using this parks and wreckb
28:11:51budget.csv. This is a new data set for
28:11:53this series. So go ahead and get that
28:11:55down below in the GitHub and we're going
28:11:57to get going. The only thing that you
28:11:59need to do besides things that we've
28:12:01done in previous lessons is we're going
28:12:02to using a different library. It's
28:12:04called ggplot 2. This is a data
28:12:06visualization library specifically
28:12:08designed to make it easier to visualize
28:12:10your data. Let's go ahead and run this
28:12:14and let's take a look at this data
28:12:16frame. So in this data frame we have
28:12:18year, we have the department, then we
28:12:21have budget in thousands. And if you go
28:12:24down, you can see we have a lot of
28:12:25different departments. Sanitation,
28:12:27public works, city management,
28:12:29education, transportation. There's a lot
28:12:31of different departments. Now, what
28:12:33we're going to be working on is we're
28:12:34going to be taking a look at bar charts,
28:12:37and we're also going to take a look at
28:12:39line charts, some categorization with a
28:12:41bar chart, as well as some time series
28:12:44visualizations with a line chart. Once
28:12:47we get those, we are going to be putting
28:12:48this into an R markdown file, and we're
28:12:52going to kind of make it look pretty.
28:12:53We're going to make it look nice, and
28:12:54then this will be something you can hand
28:12:56off to somebody. So, let's get started
28:12:59and let's take a look at how we can get
28:13:02our very first bar chart. So, we have
28:13:04our data frame here and I'm just going
28:13:07to start by saying we do need to do a
28:13:09little bit of work to get it properly
28:13:11formatted for something like a bar
28:13:13chart. Bar charts are great when working
28:13:16aggregated data and we can easily
28:13:19aggregate this data because we have
28:13:21something like a department to then
28:13:22aggregate on something with the budget
28:13:24in thousands. We could also aggregate on
28:13:26the year and look at the total budget
28:13:28for the year, but I'm more interested to
28:13:31see how much money have we budgeted for
28:13:34in total for every single department.
28:13:37So, we can just use a sum here. If you
28:13:39want to change it up and you want to
28:13:40look at maybe the median or you want to
28:13:42look at the max or whatever it is, you
28:13:44can do that. But, we're going to do the
28:13:47sum of this parks and recck department
28:13:48to see how much money they budgeted for
28:13:50over all these years. So let's take our
28:13:54data frame. We're going to use our pipe
28:13:56operator and we're going to use some of
28:13:58the things that we've used throughout
28:14:00this entire series. If you haven't gone
28:14:01through the series, this is what we've
28:14:03covered so far. And then grouping and
28:14:05aggregating is one of them. So we're
28:14:07going to group on the department and
28:14:10then we'll add another pipe here.
28:14:14We're going to summarize this. And we
28:14:17want to do this on the budget. So we're
28:14:20going to do the sum
28:14:22of the budget in thousands. Now, we can
28:14:25name this as well. It isn't super
28:14:28important, but we'll call this total
28:14:30budget and we'll say that's equal to the
28:14:32sum of budget in thousands. Now, the
28:14:36next thing that we're going to do is
28:14:37where we actually start creating our
28:14:39visualization. And so, this is where it
28:14:40starts to get different than what we've
28:14:43just looked at with grouping and
28:14:44aggregating. Now, we're going to
28:14:45actually build out the visualization.
28:14:47Now, for the very basics, especially
28:14:49with using ggplot 2, is we need to just
28:14:51be able to kind of get something over
28:14:53here under our plots. So, we kind of
28:14:56want to create our very first layer. And
28:14:58that's how plots typically work is
28:15:00you're layering one thing on top of
28:15:02another. And then you're also adding in
28:15:05different adding in different colors or
28:15:07different variations of things that you
28:15:09want to change. Typically, when you're
28:15:11doing a plot, you're going to start off
28:15:13with ggplot. This ggplot is going to
28:15:16give us that first layer of what data
28:15:18are we actually working with. And the
28:15:19ones that we're working with are the
28:15:21department and our total budget. So
28:15:23we're going to come down here. We're
28:15:24going to say aes. Now AES is something
28:15:28that you use to specify the variables
28:15:30that are going to be mapped into the
28:15:31data. And you can look through it right
28:15:33here and read into that. But we're going
28:15:35to specify AES for our aesthetics. And
28:15:37we're going to pass through our X and
28:15:39our Y. So we're going to say X is equal
28:15:42to department. And I need to spell this
28:15:45right. And then we'll do a comma. And
28:15:47we'll say the Y is equal to, and let me
28:15:50make it look a little bit better.
28:15:52Total_budget.
28:15:54And we can keep it like that for now.
28:15:57But just knowing what's to come, we will
28:15:59come back and we'll edit this a little
28:16:01bit. But when we start getting to
28:16:04visualizations, we're no longer going to
28:16:05use these pipe operators. We're now just
28:16:07going to say plus. And that says, okay,
28:16:10we're adding to the next layer of our
28:16:13visualization. We can actually look at
28:16:15just this. Let's go ahead and run this.
28:16:18And we have down here our X and our Y,
28:16:22but we don't have the layer on top of
28:16:24it. What are we actually putting in this
28:16:25visualization? You'll also notice this
28:16:28looks horrible at the bottom. It's text
28:16:30over text, and it just does not look
28:16:32good. We'll fix that later on, trust me.
28:16:35But let's keep going. We want to add a
28:16:38bar chart here. So what we want to do is
28:16:40we're going to do geo m_bar
28:16:44and you can see it right down here. So
28:16:45within ggplot so within ggplot they have
28:16:48all these geometry visualizations is
28:16:51that's what that g stands for the g_bar
28:16:54and so we want to specify this is a bar
28:16:56chart and we can leave it just like
28:16:58this. Let's go ahead and run this. Now
28:16:59we're going to get an error here because
28:17:01we need to set up a specific parameter
28:17:04called stat or stat count. Now just
28:17:06knowing these if we come in here we have
28:17:08just the classic stat and then you can
28:17:11also do different options like stat
28:17:14count. I'm just going to use stat and
28:17:16we're going to say it's equal to
28:17:18identity. We're going to do it like this
28:17:20identity. And so this is just something
28:17:23that we have to pass through in order
28:17:24for it to actually kind of understand
28:17:26what we're working with here. So now
28:17:28we're actually getting a visualization.
28:17:30This looks really good, but we're going
28:17:32to have to clean this up quite a bit.
28:17:34make several changes to it. One thing we
28:17:36can just add really quickly is we're
28:17:38going to add a title. So we're going to
28:17:40do gg title and we just have to pass
28:17:42through a string. So we'll call this
28:17:44total budget
28:17:46by department. Just keep it super duper
28:17:50simple here. And so now we have a nice
28:17:52little title here. But the first thing
28:17:55that I want to fix is we want some
28:17:57colors in here. So, we're going to come
28:17:59back to our aesthetics and we're going
28:18:01to do a comma and we're going to say
28:18:02fill is equal to and we're going to do
28:18:05department. So, now it's going to give
28:18:07us a legend and each one of these colors
28:18:09will be a different department. Let's
28:18:11run this.
28:18:13And so, you don't have to do this. In
28:18:15fact, we could just make it a specific
28:18:17color. We could just say we could just
28:18:20say red. And we could do it like that.
28:18:24We'll make them all red. or it's kind of
28:18:26like a salmon almost, but a different
28:18:28color. We can just make it a specific
28:18:30color and that'd be perfectly fine. But
28:18:32let's just go back. I like this kind of
28:18:34like rainbow color here. It's nice. But
28:18:37at the very bottom, we're still having
28:18:38an issue with how this looks. Now,
28:18:42typically, if I'm working with kind of a
28:18:43simpler visualization, I would do a
28:18:46theme. And again, we're just adding a
28:18:48layer here. I would do this like minimal
28:18:50theme. And it would kind of get rid of
28:18:52some of these colors in the back. do a
28:18:55little bit of spacing and it looks nice.
28:18:56It's a minimal theme, but for this
28:19:00theme, we actually need to specify this
28:19:02right here, which is an element. So, we
28:19:05need to come back here. I'm going to get
28:19:07rid of this. I'm just going to have a
28:19:08base theme and we need to do axis.ext
28:19:12and we need this for the xaxis. So,
28:19:15along this x-axis, we need to work with
28:19:18this text. So, we're going to say is
28:19:20equal to. And now we need to pass
28:19:22through element
28:19:25text. And all we're going to do is
28:19:27adjust the angle. And we're just going
28:19:29to come in here. We're going to say the
28:19:30angle, not angel. The angle is equal to
28:19:3445. So now it's going to be at a 45
28:19:36degree angle. Let's go ahead and run
28:19:38this and just see how it looks.
28:19:40And this looks aund times better, but I
28:19:43think we need to adjust a little bit uh
28:19:45as well. So, we'll do the height
28:19:48adjustment and we'll say it's equal to
28:19:51one. Let's try this.
28:19:54And there we go. So, this looks a
28:19:57hundred times better. The only thing
28:19:59that I would change is I would want to
28:20:02order this from highest down to lowest.
28:20:05So, I could visually just see in order
28:20:07from highest to lowest which department
28:20:10is which. Now, we can do that. It's
28:20:12going to be right up here in our
28:20:13summarize. Now, we can do that. It's
28:20:16going to be right here in this X because
28:20:18we have our department right here. We
28:20:20can actually use something called a
28:20:22reorder
28:20:24and we pass through the department first
28:20:27but then we do a comma here and we say
28:20:31minus the total budget. That minus just
28:20:34means go from high to low. That's all
28:20:37that minus means. If we did it the other
28:20:38way, it'd be from low to high. Let's go
28:20:40ahead and run this. So now we see that
28:20:42public works takes about a hundred. I
28:20:44think that's a million dollars. Then we
28:20:46have recreation, park, sanitation. So
28:20:48that is visually you can see it going
28:20:50down. Again, we don't have to use this,
28:20:52but I just like the colors and I wanted
28:20:55to uh so let's keep going because I
28:20:58think we got a good gist of how we can
28:20:59create this visualization and we will
28:21:01add this to our output in a little bit.
28:21:04I might want to just show you what this
28:21:07R markdown looks like. We're going to
28:21:09call this uh presentation_final.
28:21:14Let's open this up real quick. It
28:21:16creates this output for an HTML
28:21:18document, which is perfectly fine. Um,
28:21:20I'm going to use a different one called
28:21:22flex dashboard. You don't have to use
28:21:24that, but that's the one I'm going to
28:21:25use in this lesson, but we'll be passing
28:21:27through this right here. And so, hold on
28:21:31to this. Don't change that because we
28:21:33will need it. Now, let's go down to our
28:21:35line chart. Now, line charts are a
28:21:37little bit different because in this
28:21:40we're just looking at as an in total.
28:21:42We're not looking at it over time, but
28:21:44now we're about to use this column right
28:21:47here. And now we have a specific year
28:21:50that we're going to look at over time
28:21:52for all of our data. I think we should
28:21:54do two separate visualizations. One
28:21:56where it's all the departments together
28:21:58and then one broken out by departments
28:22:00because those are two very different
28:22:01visualizations. And so what we're going
28:22:04to do is we can actually copy um I don't
28:22:08know quite a bit of this. Let's just
28:22:09copy this whole thing down because now
28:22:12we're just grouping on the year, but we
28:22:14still want that sum of total budget. And
28:22:17then down here in the X, we're not
28:22:20looking at the department anymore. Now
28:22:23we're looking at the year. So then our Y
28:22:25is our total budget, which because we're
28:22:28grouping on the year, we might want to
28:22:29call this our annual
28:22:32because this is our total budget. So now
28:22:34we should change this to annual budget.
28:22:37And we can still use the same fill at
28:22:40least for this visualization. It
28:22:41shouldn't matter at all. Um, but we will
28:22:44use it in the next one. We break it out
28:22:45by the department. The only thing that's
28:22:48really going to change is we have to go
28:22:50to GM and I'm going to say line here.
28:22:54So, this is going to create a line
28:22:56visualization. And let's just run it
28:22:58just like this. And apparently this
28:23:01actually does matter. Let's get rid of
28:23:02this really quick. And let's try running
28:23:05this one more time. And in essence, this
28:23:08is our visualization. It's very simple.
28:23:11Uh let's actually add in that minimal
28:23:14theme. So I'm going to do uh theme
28:23:17minimal. This is just the one that I
28:23:20think makes it look nicer. We can also
28:23:22add in little dots on the year so we can
28:23:24kind of see what this is supposed to
28:23:26look like. So I'm going to do gam point
28:23:29and this is just going to add dots on
28:23:32where the data goes. And you can see, so
28:23:34this is 2005 6 7 8 9 10. And it's just
28:23:38easier to see. And I think just for a
28:23:40quick line chart for the budget or the
28:23:44annual budget over the years, this is
28:23:46pretty good. But we could also add in a
28:23:48title here. So I'm going to say uh GG
28:23:52title and I'm going to say annual budget
28:23:57for all departments.
28:24:00And we'll do it just like this. And that
28:24:03looks great. Now, what if we want to
28:24:05break this out by the different
28:24:07departments? This is where it gets a
28:24:09little trickier. Not much. Let's copy
28:24:12this down. And we want to break this.
28:24:15I'm going to say break out by
28:24:18departments. That's just for the people
28:24:20who are looking at the code. Um, so now
28:24:22we're not really needing to group by
28:24:25this because it kind of already has this
28:24:27for us. Whoops. It kind of already has
28:24:29this for us. I just messed up all the
28:24:33data, but it has the year, the
28:24:35department, and the budget. So, we
28:24:37really shouldn't need to group this
28:24:39data. Let's pull this back. But in our
28:24:43ggplot, we do need the year. We do need
28:24:47the budget in thousands. So, we're going
28:24:49to do budget in thousands. And then
28:24:53we're going to need our color or our
28:24:55fill. So, I'll do color is equal to I'm
28:24:58going say department. So, let's go ahead
28:25:01and just try this.
28:25:05And there we go. Now, this is kind of
28:25:06all over the place. Maybe we want to get
28:25:07rid of these points real quick. Just see
28:25:09what this looks like.
28:25:11Maybe this looks a little bit better.
28:25:13But we have this department as our
28:25:15legend. And so now you can see you can
28:25:18just kind of follow one of these colors,
28:25:20whatever one you want to follow. And
28:25:21you'll be able to see that budget
28:25:23changing over the years. This is not the
28:25:25annual budget for all departments. Uh
28:25:28this is the angel budget per department.
28:25:33There we go. And we'll run it just like
28:25:35this. So this looks good. And we have
28:25:38you can even go back and look at you
28:25:40know previous ones you've done. And so
28:25:43you know we're holding a lot in memory
28:25:44just storing these. But
28:25:47these look really good to me. I think we
28:25:49are good to go. So what we need to do
28:25:52now is we want to group all of this.
28:25:55Now, what we want to do is we're going
28:25:56to change this a little bit and we're
28:25:57going to put all of our visualizations
28:25:59and essentially all of our data in here.
28:26:01Now, I'm going to change this to a flex
28:26:05dashboard.
28:26:06And I just find these easier to work
28:26:08with. Uh, so we're do flex
28:26:11dashboard here. Now, I'm going to save
28:26:14this. And when I save this on knit,
28:26:17which means we're going to actually
28:26:18process the data and we're going to
28:26:20create it. We're now knitting to a
28:26:21dashboard instead of before we were
28:26:23doing to an HTML document.
28:26:26Now, if you don't have this installed,
28:26:27which I completely deleted my R Studio
28:26:30and R in order to do this, I need to
28:26:32install this again. So, let's go ahead
28:26:34and do that. But while it's working on
28:26:36that, let's start moving over all of our
28:26:39stuff. So, I'm going to come up here and
28:26:42I'm going to get this right here. And
28:26:45what we need to do is put it in a little
28:26:46code block. and we're going to use
28:26:48little tick marks to specify where
28:26:52we're going to have our code. Now, I
28:26:54need to use what I like to call squiggly
28:26:56brackets. And I need to specify that
28:26:58this is our code because you don't have
28:27:00to use this uh just with R. You can use
28:27:03this with other programming languages.
28:27:05So, we want to specify this is our code.
28:27:07So, all we're doing is just getting in
28:27:09our data set. If we had other things
28:27:11where we were cleaning it up or
28:27:12transforming it or whatever it was, it'd
28:27:14be in this code block. Now, we want to
28:27:17put in our first one. This is our bar
28:27:19chart and we're going to come down here
28:27:22and we're going to do three with the R,
28:27:26of course. Oops.
28:27:28Let me fix that. We're going to post our
28:27:31code and then do three as well. And so
28:27:34now you'll notice it kind of has it
28:27:36highlighted. This is where our R code
28:27:38is. Now, if we try to just run this,
28:27:40let's save this and let's knit it. So I
28:27:43just clicked dit.
28:27:46It's going to start processing this as
28:27:47well as our visualization and it's going
28:27:49to give us an output. Now it just gives
28:27:51us our presentation final and our one
28:27:54visualization. But we haven't really
28:27:56added a lot to this. Let's add our other
28:27:59stuff as well. So now we're going to
28:28:01come down here and we can just copy
28:28:03this. Let's not get crazy. And we're
28:28:06going to do one, two, three. And then
28:28:07I'm going to do another one down here.
28:28:09One, two, three. So, let's go back and
28:28:11get our two other visualizations. We
28:28:13have this one, and I'll paste it right
28:28:15in here. And then we have this one. And
28:28:18I'll paste that one in the bottom one.
28:28:21And let's go ahead and save this as well
28:28:24as knit to flex dashboard, which is what
28:28:26we just did before.
28:28:33And taking a look at this, it looks like
28:28:35something went wrong because none of
28:28:37this is working properly. If we scroll
28:28:39up and down, this just doesn't look
28:28:41right.
28:28:44Let's go ahead and exit out of this.
28:28:46Let's go back up and let's give these
28:28:50titles really quick. We're going to do
28:28:51one, two, three. And I'm just going to
28:28:53give it the same title as we have up
28:28:55here.
28:28:58We don't need these uh quotes right
28:29:01here.
28:29:04We go down here. Let's give it three
28:29:08annual budget for all departments. And
28:29:11then lastly down here as well. And this
28:29:15is annual budget per department. Let's
28:29:19go ahead and run this
28:29:25and let's see what our output looks like
28:29:26this time.
28:29:29It looks quite a bit different. We have
28:29:31this right here. This uh side data that
28:29:34does not look correct. Uh we need to go
28:29:36back and we need to get rid of this.
28:29:39Let's go back to our visualization.
28:29:42Let's just run this and we'll run it
28:29:45just like that. You'll see in our output
28:29:48we're getting all these things and this
28:29:50is being put into our visualization or
28:29:52our output as well. We don't want this.
28:29:55Now, if I'm looking at this really
28:29:57quickly, I think we just didn't add in a
28:30:00space here. And so, it's like printing
28:30:02out these themes uh as we go. So, let me
28:30:06add in
28:30:08this space right here.
28:30:11And we'll add it in here as well. Now,
28:30:13let's try saving this. And let's knit
28:30:16it.
28:30:22And this looks good. Although it's
28:30:24really small in order to make it larger.
28:30:27Let's try something right here. Let's do
28:30:30a colon. We're going to say vertical
28:30:33layout. We'll do a colon and say screen.
28:30:38Just like this. Let's save this.
28:30:41And this should allow us to have them
28:30:43stacked on top of each other.
28:30:47And let's see. Knit to flex. So, let's
28:30:49just make sure it's working. I think
28:30:51this may need to be lowercase, which is
28:30:53why it's not working. Let's try this
28:30:55again.
28:30:56There we go. But this vertical scroll
28:30:59stacks them on top of each other and
28:31:00allows you to scroll down. And there we
28:31:03go. And this looks 100 times better than
28:31:06how we had it before because now it has
28:31:07a scroll wheel, allows these things to
28:31:09expand. Now, we can also publish this.
28:31:12We can say publish document. In order to
28:31:15do that, you have to have these
28:31:16connected. And RS Connect is is
28:31:18something that Posit created in order to
28:31:20publish these. We're just going to say
28:31:22yes. Uh I had this before, but you know,
28:31:24I deleted everything. So now I'm
28:31:26starting over with you guys. But let's
28:31:27go ahead and publish this. It looks like
28:31:30it's still working on that. Let's give
28:31:32it just a sec. It took about 1 minute.
28:31:35Says it is installed. Let's bring this
28:31:37back. Now we can p publish it to our
28:31:40pubs, which is a free service. You can
28:31:42also do Posit Connect and Posit Cloud
28:31:44which are paid services or at least in
28:31:46some way or shape or form. You'll have
28:31:47to pay for it. We're just going to use
28:31:49our pubs and we're going to go ahead and
28:31:51click publish. Now, all you have to do
28:31:52is create an account, share it, and
28:31:54you'll get a sharable link that you can
28:31:55then put on your resume, put on
28:31:57LinkedIn, put anywhere you want in order
28:31:58to actually share your project. So, I
28:32:01hope that this was helpful. If you
28:32:02haven't already, I have a full R course
28:32:04that goes a lot more in depth into the
28:32:05data visualization and presentation side
28:32:07of things. So, if you want to go ahead
28:32:09and check that out, I will leave a link
28:32:10in the description below. I'll also have
28:32:12a coupon code if you want that as well.
28:32:15But I really appreciate you watching and
28:32:17I hope that this whole series was really
28:32:19helpful. If you did find if you did like
28:32:21it, be sure to like and subscribe and I
28:32:23will see you in the next [music] lesson.
28:32:37What's going on everybody? My name is
28:32:38Alex Freeberg and today we're going to
28:32:40be walking through my top three tips on
28:32:42how to use LinkedIn to land a job.
28:32:44LinkedIn is a fantastic place to look
28:32:46for a job. It's its own little ecosystem
28:32:48where careerdriven people can connect
28:32:49and talk with one another and help each
28:32:51other find jobs. I personally have
28:32:53landed jobs through LinkedIn and so I
28:32:54know how effective it can be. Let's jump
28:32:56over to my screen and I'm going to show
28:32:58you my top three strategies that I have
28:33:00found to be the most successful to
28:33:01actually finding a job. So, I'm logged
28:33:03into my completely anonymous account
28:33:05here and I'm going to show you the very
28:33:06first tip, which is you shouldn't be
28:33:08just applying to a position. You should
28:33:09be actually reaching out to the
28:33:10recruiter. And I'm going to show you
28:33:12exactly how to do that. So, the first
28:33:13thing that we have to do is to actually
28:33:15find a job that we want to apply to. So,
28:33:17let's go to the job section right over
28:33:19here and let's search for data analyst.
28:33:25And let's do that in
28:33:28uh let's do Chicago
28:33:30cuz why not? Uh so it's going to search
28:33:33for data analyst positions in Chicago.
28:33:36Uh we have one right here. Let's see
28:33:38what it looks like cuz you know I don't
28:33:40want to apply to jobs that I'm not
28:33:42extremely qualified for. So this is a
28:33:45job that I want to apply for. And before
28:33:46I actually go and apply to the job, I
28:33:48want to see if I can reach out to a
28:33:49recruiter and talk to them beforehand.
28:33:51So let me show you how to do that. So,
28:33:53what we're going to do is actually click
28:33:54on the company right here. It's going to
28:33:56take us to basically their LinkedIn
28:33:57profile page for their entire company.
28:34:00And we're going to scroll down. We're
28:34:01going to go over to people.
28:34:03And then we're going to search for
28:34:05recruiter.
28:34:08So, if we scroll down all the way to the
28:34:10bottom, we can see that there are
28:34:11recruiters that actually work inhouse
28:34:13for this company. And so, now would be a
28:34:15time where I actually reach out to some
28:34:16of these recruiters and I say, "Hey, I
28:34:18see a job that I really like. I think
28:34:20I'm really qualified for and I would
28:34:21love to talk more about it with you. You
28:34:23can ask them things about the job to
28:34:25make sure that it is a good fit for you.
28:34:26And then I highly recommend you asking
28:34:28them what they think is the best way to
28:34:30apply for this job to make sure that
28:34:31your resume gets noticed and you get an
28:34:33interview. Since they are a recruiter
28:34:35who works at this company, they may be
28:34:36the one who's actually going to be
28:34:37looking at these résumés. And so they
28:34:39may give you a tip on the best way to
28:34:41actually apply. They may also just ask
28:34:43you to send them your resume directly so
28:34:45that they can look at it. or maybe later
28:34:46on down the line, this actually is a
28:34:48person who is reviewing résumés and so
28:34:50if they come across your resume, they
28:34:51may be able to put a face to the name
28:34:53and that may give you bonus points. I'm
28:34:55going to leave a template script in the
28:34:56description in case you don't know
28:34:57exactly what you want to say to this
28:34:59recruiter and it'll give you just a
28:35:00baseline of some of the things that you
28:35:02might want to say. Number two is to
28:35:03actually ask for a referral. Now, if you
28:35:05don't know what a referral is, it is
28:35:07where somebody who already works at the
28:35:08company can refer you to a specific job
28:35:11and it might get you a little bit higher
28:35:12on the list for interviews. So, I highly
28:35:14recommend reaching out to somebody who
28:35:15already works at that company and ask if
28:35:18they're willing to be a referral for
28:35:19you. I get people reaching out to me all
28:35:21the time asking to be a referral for
28:35:23them for my company. And nine times out
28:35:25of 10, I say yes. I always ask to see
28:35:27their resume first just to make sure
28:35:28that their resume aligns with the
28:35:30position at least a little bit. But
28:35:32there's basically no harm in me being a
28:35:34referral for somebody. In fact, I may
28:35:36actually get a bonus if that person ends
28:35:37up getting hired. And so, for the most
28:35:39part, there's almost no risk for the
28:35:41employee to actually being a referral.
28:35:43And so a lot of times they will say yes.
28:35:45Now let me show you how to do that. And
28:35:46it is very similar to finding a
28:35:48recruiter. So we're going to stay on
28:35:49this people section, but instead of
28:35:51searching for a recruiter, we're going
28:35:53to search for a job title that is
28:35:54similar to yours. So let's actually see
28:35:56if they do already have any data
28:35:58analysts. And if they do, that is the
28:36:00person that we're going to reach out to
28:36:01cuz that is the person we'll probably
28:36:02have the best connection with. So it
28:36:04looks like we have six employees. And
28:36:06let's scroll down. And so it looks like
28:36:08all these people have data related jobs.
28:36:10And so I would reach out to these people
28:36:12and say, "I saw an open data analyst
28:36:13position at your company. I would love
28:36:15to know more about your company as a
28:36:17whole." And then you can talk to them a
28:36:18little bit and then in the end your goal
28:36:20is to ask them for a referral. And if
28:36:22that happens, that is fantastic. And
28:36:24then you can go ahead and apply for the
28:36:25job and mark them as a referral for you.
28:36:27Now my third tip on how to get a job
28:36:29through LinkedIn is to actually have
28:36:30recruiters reach out to you. So let me
28:36:32show you how to do that. The first thing
28:36:34we're going to do is actually go over to
28:36:36my profile here and we'll click view
28:36:38profile.
28:36:40Now, there's a few things that we want
28:36:42to make sure that we have on here so
28:36:44that recruiters can reach out to us. The
28:36:46first thing that I want to do is to
28:36:47actually come to this section right
28:36:48here, which is show recruiters you're
28:36:50open to work. And when I click on this,
28:36:52I can actually choose some job titles
28:36:54and some locations where I actually want
28:36:55to apply and have recruiters reach out
28:36:57to me. And so, right now, I have data
28:36:59analyst. I have in the DFW area, which
28:37:02is where I live. I can also add titles
28:37:04like business analyst
28:37:06um and then maybe junior data analyst,
28:37:08entry- level data analyst or things like
28:37:10that that could potentially have
28:37:11recruiters reach out to me for positions
28:37:13that I'm interested in. And then you can
28:37:14say that you're immediately and actively
28:37:16applying. And you can also say that
28:37:18you're only looking for full-time
28:37:20positions or contract positions. And
28:37:22then you can actually add this to your
28:37:23profile. And I only want recruiters to
28:37:25see that because I do currently have a
28:37:27job at McDonald's. And so I don't want
28:37:29McDonald's firing me because I'm looking
28:37:31for employment elsewhere. So let's save
28:37:33that. And it looks like it was updated.
28:37:36And so now when recruiters are searching
28:37:37for candidates for a specific position,
28:37:39you will be on that list so that they
28:37:41can find you and reach out to you.
28:37:43Something else I should mention is on
28:37:44your profile page, I would try to have
28:37:46some type of professional photo so that
28:37:48you look really good. I would also try
28:37:50to include data analyst somewhere in
28:37:51your title. If you already have a data
28:37:53analyst job and you're looking for
28:37:54another one, you can just have your
28:37:55previous company. But if you're looking
28:37:57for a data analyst job, you can always
28:37:58put seeking data analyst position or
28:38:00something like that. Another thing I
28:38:02think is really important is having
28:38:04really good descriptions for your
28:38:05previous work. I don't currently have
28:38:07this, but I would go a little bit into
28:38:09the work that I actually do. Make sure
28:38:11that the experience matches kind of what
28:38:13you're looking for if you do have
28:38:14previous experience. If not, that's
28:38:16totally fine. The next section on your
28:38:18profile page that I would recommend
28:38:19looking at and updating is your skill
28:38:21section. And so you want to go in there
28:38:23and make sure that you have all of your
28:38:24relevant really data analyst heavy
28:38:26skills on there, specifically hard
28:38:28skills because soft skills aren't going
28:38:30to translate too much into this section.
28:38:32I would definitely stick to things like
28:38:34SQL, Python, Tableau, Excel, things that
28:38:37data analysts are going to use because
28:38:39this is where they're going to actually
28:38:40look and see if you have the skills that
28:38:41they are looking for for that position.
28:38:43When I was applying to jobs and only
28:38:45applying to job postings and not using
28:38:47any of these strategies, my success rate
28:38:49was 0.04, 04, which means out of 1,000
28:38:52applications that I filled out and sent
28:38:54my resume to, I only heard back from
28:38:56four of them to actually get an
28:38:57interview. But with these strategies, I
28:38:59was able to get that up to 10% and at my
28:39:01best, I was able to get that up to 15%.
28:39:03But that's because I was applying to a
28:39:05lot less positions, and I was targeting
28:39:06jobs that I really wanted to work for.
28:39:08And so, I put in more effort in order to
28:39:10contact people and work with recruiters
28:39:11in order to get that job. I genuinely
28:39:14hope that these strategies can be
28:39:15helpful for you, especially if you're
28:39:16trying to apply for jobs right now.
28:39:18Thank you guys so much for watching. I
28:39:20really appreciate it. If you like this
28:39:21video and got anything out of it at all,
28:39:23be sure to like and subscribe below and
28:39:25I'll see you in the next video. Hello
28:39:26everybody. Congratulations. If you are
28:39:28watching this, that means that you
28:39:30completed the data analyst boot camp. If
28:39:32you haven't, don't keep watching. This
28:39:33is only for people who have completed
28:39:35the data analyst boot camp playlist on
28:39:37my YouTube channel. Woo! All right. Now
28:39:39that we filtered those people out, I'm
28:39:41going to show you how you can download
28:39:42your certificate and your certification
28:39:44now that you've completed the data
28:39:46analyst boot camp. I will leave a link
28:39:47in the description, but let's go on to
28:39:49my screen. I'm going to show you how to
28:39:50actually access this and download your
28:39:52certification. All right, guys. Don't go
28:39:54around telling people this or sharing
28:39:56this. Uh, but this is our data analytics
28:39:58boot camp on the Alex the Analyst GitHub
28:40:01right up here. I will have this link in
28:40:03the description. What you can go ahead
28:40:04and do is you can come right here and
28:40:06you can download this. You'll just
28:40:08rightclick or click download and you
28:40:09just do something like save image as.
28:40:12Um, or you can come to this one. This is
28:40:14the one that I think is the the real
28:40:15money maker here. Uh this is the
28:40:17certificate of completion for the data
28:40:20analytics boot camp. I have my not
28:40:22signature but my name as well as u my
28:40:25position with a blank space right here
28:40:28to fill in your name. Feel free to put
28:40:30this on LinkedIn or Twitter or Instagram
28:40:32and tag me in that because I would love
28:40:33to just say congratulations because
28:40:35honestly it's a lot of work to go
28:40:37through all those videos and learn all
28:40:38of those skills. So congratulations. I
28:40:40hope that you learned something along
28:40:41this journey. A new skill, a new
28:40:43thought, a new idea and I'm proud of
28:40:45you. I'm proud of you for putting in the
28:40:46work. It's not easy, but you did it. And
28:40:48I hope that you came out on the other
28:40:51side better for it. So, congrats. I'll
28:40:53see you in the next video.
28:40:58[music]

About this transcript

This page contains the full transcript of 2026 FREE Data Analyst Bootcamp [24 Hours+] for FREE | SQL, Excel, Python, Power BI, GitHub, AWS by Alex The Analyst, generated from the public captions YouTube serves with the video. The transcript has 334,400 words across 45,258 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.