YT YouTube2Text

YouTube2Text → Transcripts → SQL Full Course for Beginners (30 Hours) – From Zero to Hero

SQL Full Course for Beginners (30 Hours) – From Zero to Hero — Transcript

by Data with Baraa · 363,984 words · 48,358 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Hello and welcome to this unique course
0:01to master SQL. My name is Barzalini and
0:04I lead big data projects at
0:05Mercedes-Benz over a decade of
0:07experience in SQL data engineering,
0:10building data warehouses and data
0:12analytics. Now, of course, the first
0:13question is what makes this course so
0:15special. Well, not only you will learn
0:17how to write SQL codes, but more
0:19important than that, you will learn how
0:21exactly SQL works behind the scenes. So
0:24I'm going to break complex concept in
0:26SQL using hundreds of animated visuals.
0:29This makes it really easier to
0:31understand SQL and as well it is more
0:33fun than just sharing my screen and I
0:35just show you code. Right. The second
0:37reason is this course is taught by me. I
0:39have industrial experience and I will be
0:42sharing with you everything that I know
0:43about SQL and how I use it in my real
0:46projects. So I will be sharing with you
0:47hundreds of best practices, tips and
0:50tricks and I'm going to show you my
0:51decision-m process in SQL. So by the end
0:54of this course, you will be ready to
0:56solve any complex task like I do using
0:58SQL. So now I designed this course to
1:01cover the basics like writing your first
1:03SQL query and then we're going to keep
1:05progressing in the course by covering
1:07advanced techniques in SQL like the
1:09window functions, stored procedures,
1:11indexes and even at the end we're going
1:13to build a data warehouse using SQL. And
1:16this course is suitable for anyone data
1:18engineers, data analyst, data scientist
1:20and even for students. And by the way
1:22the good news everything is for free
1:24from the start until the ends I will be
1:26sharing with you as well a lot of
1:28materials code presentations and
1:30animations and there are no hidden
1:32costs. So you don't have to pay for
1:34anything. But my friends in return I
1:36really appreciate it if you support the
1:37channel in order to grow. All right my
1:39friends I'm really excited about it. I
1:41don't know about you. If you are
1:42motivated join me learning SQL. This is
1:45going to be amazing. So let's
1:50go. All right. Now I'm going to show you
1:52the road map in order to learn
1:53everything about SQL starting from very
1:56basics and then advance step by step
1:58until we have very advanced topics. So
2:00now at the start we have to understand
2:02few stuff like what is SQL, why to learn
2:04it, what are databases and the types of
2:07databases and after the theory we're
2:09going to prepare your PC with data and
2:11the softwares. Now once we have
2:12everything then we can go to the next
2:14chapter. This is the basics how to query
2:16data using SQL and here we're going to
2:18cover the basic components in each SQL
2:21query like select from where those
2:24basics. Now once you understand how to
2:26query the data, how to get the data out
2:28of the database the next step we're
2:30going to go and learn how to define the
2:32structure of the database. How to create
2:34a new table add a new column remove
2:37column and as well how to drop a table.
2:39So with that you are defining new stuff
2:41in the database and then the next
2:43chapter you have to learn about the data
2:45manipulation. This time we're going to
2:46go inside the table and we're going to
2:48learn how to insert a new data, how to
2:50update the data and as well delete few
2:53rows from our database. So with that you
2:54have the basics how to query data, how
2:57to define the structure of your tables
2:58and how to manipulate your data. And I
3:01can say with that you cover the basics
3:02about SQL. Now after that we start with
3:05the intermediate phase where we're going
3:06to deep dive into topics like how to
3:08filter your data. Here we're going to
3:10learn about the comparison operators,
3:12logical operators, between and like. So
3:15all the operators that you can use in
3:17order to build a condition in order to
3:19filter your data. Then after that it's
3:21going to be very interesting topic. You
3:23have to learn how to combine them. And
3:24here we have two mechanism either using
3:26the join or using the set operators. And
3:29oh my god joining data. It's going to be
3:31very interesting topic. Here we're going
3:33to cover like a lot of stuff like we're
3:34going to start with the basic joins and
3:36then we go to advanced and then you have
3:38to learn how to choose the right join
3:40and after that you have to learn about
3:42the set operators and here you have like
3:44four methods union union all except
3:46intersects. So with that you learn how
3:48to combine multiple tables by combining
3:51the columns or the rows of your tables.
3:54So this is very important. Now moving on
3:56in our course. Now using SQL you can do
3:58a lot of stuff cleaning up the data a
4:01lot of data preparations and at the end
4:03you can do a lot of analytics and
4:05aggregations. So there are like two
4:07families of functions. The first one is
4:09the role level functions and here we
4:10have a lot of stuff you can transform
4:12your string values the numbers date and
4:15time and how to handle the nulls in SQL
4:17and at the end the amazing case
4:19statements. So all those stuffs are
4:21transformation for only one single
4:23value. We call it role level functions.
4:25And after you learn how to do data
4:26transformations, then you have to learn
4:28about how to do data analytics and
4:30aggregations using SQL functions. So
4:33we're going to start with very basics
4:34like the aggregate functions. And then
4:36we're going to deep dive into the window
4:38functions, analytical functions. And
4:40here we have like aggregates, ranking
4:42and value functions. Those are very
4:44important tool for any data analyst or
4:47data scientist doing analytics task in
4:49SQL. So I can say the rowle functions is
4:52for data engineers and the analytical
4:54functions are for data analysts. So at
4:56the chapter 8 we can say you have
4:58covered now the intermediate level and
5:00the last four chapters they will be the
5:02advanced stuff in SQL. So here there are
5:05a lot of techniques that you have to
5:06learn about SQL. So the first one is the
5:09subquery query inside another query and
5:11the very famous CTE common table
5:14expression. A lot of developers like
5:16this one and then you will learn about
5:17how to create views in the database.
5:20This technique if you learn it you're
5:21going to be really professional in SQL.
5:23Then we're going to learn how to create
5:24tables using select the temporal tables
5:27and then we're going to learn about the
5:28third procedures how to write a program
5:30in SQL and after that of course comes
5:33the triggers. So those are the advanced
5:35techniques that you have to learn in SQL
5:37in order to do advanced projects using
5:40SQL. So now once you learn all those
5:42concepts and you start writing a lot of
5:44SQL codes you will notice that some
5:46queries going to be really slow and for
5:48that you have to learn how to optimize
5:50the performance of your queries and here
5:52there are a lot of techniques. The most
5:53famous one is to create an index in the
5:55database or create a partition and at
5:58the end I will be sharing with you the
6:00top 10 best practices that I have
6:02learned in my projects on how to
6:04optimize the performance of your
6:05queries. So this is very important and
6:08then we're going to move to very
6:09interesting one. I will be sharing with
6:11you how I use AI like shy GBT or copilot
6:14as I'm using SQL in my projects. So here
6:17you have to learn how to write correct
6:19prompts to get assistance from AI as you
6:22are using SQL. And finally and my
6:24favorite one it will be about SQL
6:27projects. So my friends here you have to
6:28bring everything that you have learned
6:30about SQL in handon projects. With real
6:34projects you will get challenges and
6:36struggle and here going to happen the
6:37magic and the real learning and here
6:39there are three types of projects. The
6:41first one is data warehousing project.
6:43This is very data engineering focused
6:45project where you're going to learn how
6:46to build real data warehouse where
6:48you're going to take the data from the
6:49raw formats and then process it in
6:51different layers. Once you build it then
6:53you jump to another project. Here you're
6:55going to start exploring the data and
6:57start getting the first insights about
6:59the business. And the last project that
7:01you can do is the advanced data
7:03analytics project. So this is very
7:05important section where you do SQL
7:07projects. So my friends this is the road
7:09map on how to learn SQL. So as you can
7:12see it takes you step by step from
7:13basics to intermediate and you will end
7:16up having advanced topics and with that
7:18I can tell you you will learn everything
7:19about SQL. Okay. So now let's start with
7:21the first chapter the introduction to
7:23SQL and here we're going to cover few
7:24topics. So we have to understand first
7:26what is exactly SQL? Why we have to
7:28learn it? what are databases and the
7:31different SQL commands that we have in
7:32SQL. So it is the basics the theory
7:35about SQL. So what is exactly SQL? Let's
7:42go. So what is exactly SQL? Everything
7:45generate data and data is everywhere.
7:47Your first name is data your mobile and
7:50everything inside the mobile is data.
7:51Car is as well generating a lot of data.
7:54Bank, your finance statements,
7:55everything is data. And now of course
7:57the question is where do we store our
7:58data? Personally we store a lot of our
8:00data in like excels, spreadsheets in a
8:03text file. So you store a lot of your
8:05data in different files. Now how about
8:08companies? They have a lot of things
8:09that generate a lot of data that the
8:11products that they produce their
8:13customers as well generating a lot of
8:15data and sales informations and a lot of
8:17things. So companies generate massive
8:20amount of data. So now the big question
8:21is how they handle the data how they
8:23store it. Of course, they cannot go
8:25unused like simple files. They need
8:27something bigger, stronger and smarter.
8:29And here where the database comes in. So
8:31think about the database. It's like a
8:33container for storing data. But instead
8:35of just dumping files into folders, the
8:38database organized the data. So it is
8:40easy to access, to manage and to search.
8:43So a database simply it is a container
8:45that stores data. So now you might ask
8:48why we are using database. Can't we just
8:50use files like I do it personally? Well,
8:52let me tell you why we use databases.
8:54Imagine that someone asks the following
8:55question. Go and find the total spending
8:58in your data. So now, in order for Mike
9:00to find the total spending and the
9:02costs, he will be opening each of those
9:04files one by one, searching for the
9:06costs trying to combine the data and
9:09it's going to be very long and messy
9:11process. But now in the other side, if
9:13your data in database and you want to
9:14ask a question, it's going to be very
9:16easy. So all what you have to do is to
9:18talk to the database to ask a question
9:20and the database can answer your
9:22question with a result. And now comes of
9:24course the question how do we talk to a
9:26database? Well we use SQL. SQL is the
9:30language that you use in order to talk
9:33to the database. It stands for
9:35structured query language SQL. And here
9:38you have people that call it SQL like me
9:41and others that call it SQL. There is no
9:43right and wrong but if you follow me
9:45through the course I think you will
9:47start saying SQL. So by using SQL you
9:50can ask the database you can ask your
9:52data and the database going to answer
9:54your question by sending you a result.
9:56So this process is very easy simple and
9:58fast and this is way better than having
10:00your data stored in different files.
10:02Another reason why we use databases is
10:04that they can handle really huge amount
10:07of data. So sometimes we have like
10:09millions of data inside our database but
10:11in the other side if you are storing
10:12your data inside spreadsheets and you
10:14have like massive amount of data what
10:16can happen your spreadsheets going to
10:18just break they simply can't handle big
10:20data and another reason why we use
10:22databases is that it is just secure. It
10:25is safer to store important and critical
10:27data inside the database than just
10:29storing it in spreadsheets and files. So
10:31the databases are secure and you can
10:33control who is accessing what. So it is
10:36just more professional to store the data
10:38inside a database. All right my friends
10:40so far what we have learned most of the
10:42companies stores their data inside a
10:44container called a database and for you
10:47in order to ask questions and to talk to
10:49your database you have to speak the
10:51language of
10:55SQL. Now I'm going to show you how it
10:57looks like usually in companies. So we
10:59have our data inside the database and
11:01then you will have multiple people with
11:03multiple roles that are just writing
11:05different SQLs in order to talk to the
11:08data. But now not only employees and
11:09people interact with the database. You
11:11could build a website or an application
11:14that as well interacts with the database
11:16by sending different SQLs. And of
11:18course, depend on how many people are
11:19interacting with the application and the
11:21website, it might generate really
11:23massive amount of SQLs that sent to the
11:26database. And not only that, you might
11:28has as well tools in order to do data
11:30visualizations where you have like a
11:32dashboard or reports maybe created using
11:34PowerBI or Tableau and it is used by
11:37stakeholders and managers in order to
11:39make decisions and as well those tools
11:42will be connected to the database and
11:44creating SQLs. So now as you can see we
11:46have a lot of interactions with the
11:48database from people applications tools
11:51a lot of things are generating SQLs and
11:54interacting with the database but the
11:56database is just a container and storage
11:58right so we need something a software
12:00that manage all those requests and
12:02that's why we have something called
12:04database management system DPMS so it is
12:07a software that going to manage all
12:09those different requests to our database
12:11and it going to make the priority which
12:13SQL must be executed First, this
12:15software can as well manage the security
12:18whether the SQL is allowed to be
12:20executed in the first place. So my
12:22friends, the DPMS is the software that
12:24going to manage the database. And now we
12:26are not done yet. There is something
12:28missing. So we have our data, we have
12:30the software. What is missing here is
12:31the hardware. So in real companies, we
12:33cannot run that on our PC because first
12:36our PC is weak and as well it goes
12:38offline. That's why we need a server.
12:40server it is like very powerful PC and
12:43as well it lives 24/7 so it is always
12:46available and here we can decide whether
12:48we're going to have a server inside the
12:49company or we can use cloud services in
12:52order to run our database so my friends
12:54so far what we have learned the database
12:56it is container to store the data the
12:58SQL it is the language in order to talk
13:01to the database the DPMS it is the
13:03manager it manage the database and the
13:06server it is the physical machine where
13:08the database lives so this is how it
13:10looks
13:13Like and now my friends there are
13:16different types of databases. So let's
13:18see what do we have. The first and the
13:19most famous one it is the relational
13:22database. It is very simple. It is like
13:24spreadsheets call them table where we
13:26have columns and rows and then there is
13:28like a relationship between those tables
13:30to describe how they relate to each
13:32other and that's why we call it
13:33relational database. So if people hear a
13:35database they're going to think about
13:37this one. Now we have another type of
13:39databases called key value. This time
13:41the data is organized completely
13:43different where you have pairs of keys
13:45and values. Think about it. It's like a
13:47big dictionary where you have a word
13:49like the key and the definition of the
13:51word this is the value. And now moving
13:53on to the next one. This is as well
13:54important column based. So now instead
13:56of grouping the data by the rows this
13:59type of databases group the data into
14:01columns. That's why it's called column
14:03paste. And this is very advanced
14:05database in order to handle huge amount
14:07of data where the main purpose is to
14:09search for data. Moving on to another
14:11database called graph database. The main
14:13focus here is the relationship between
14:15objects. So the main idea here is how to
14:18connect my data points. And now finally
14:20we have the document database. The data
14:22is stored as entire documents where the
14:25structure of the data is not that
14:26important. What is more important is to
14:28fit everything in one page in one
14:30document. And now if you look to those
14:32five types, we can group the document,
14:34graph, column based, key value, all
14:36those databases called NoSQL databases
14:39and the relational database, SQL
14:42database. And in this course, we will be
14:44focusing of course on the relational
14:45database. And I'm sure you have heard
14:47about like the Microsoft SQL server, the
14:50MySQL, the
14:52possesses they are SQL relational
14:55database. And for the key value you have
14:57the radius the Amazon Dynamo DB and we
15:00have for the column paste we have the
15:01Cassandra and the red shift. For the
15:03graph database we have the Neo 4G and
15:06the very famous database the MongoDB as
15:08a document database. Now my friends for
15:11this course we're going to be focusing
15:12on the SQL relational databases because
15:15it is the most famous one and the most
15:18used one in companies and I will be
15:20focusing on the Microsoft SQL server. So
15:22those are the different types of
15:23databases.
15:28Now the databases are very structured
15:30and organized. It has the following
15:32hierarchy. The starting point is the
15:34server as we learned it is powerful PC
15:36and it is where the database lives and
15:38inside it we can have multiple
15:40databases. So maybe you have a database
15:42for the sales and another one for the
15:44HR. So the server can host multiple
15:47databases and as we learned a database
15:49is a container of your data. Now moving
15:51on to the next level. In each database
15:53we can have multiple schemas. A schema
15:56it is like category or you can call it a
15:58logical container that we can use it in
16:00order to group up related objects like
16:02let's say you have hundred of tables. So
16:04you can split all the tables that has to
16:06do with the orders in one schema and
16:08then another group of tables with the
16:10schema customers and so on. So it help
16:13you to organize your tables and your
16:14objects in the database. And now if you
16:16go inside schema you can have multiple
16:18objects like tables. So now of course
16:20the question is what is a table? It is
16:23like spreadsheet. It organize your data
16:25into columns. The column define the data
16:27that you store inside it. So you have
16:29one column about the customer ID.
16:31Another column about the names, the
16:33scores, the birthday. So each column is
16:35about one type of data and sometimes we
16:38call the columns as fields. Now the
16:39other thing that we have in tables is
16:41the rows or sometimes we call it
16:43records. It is where actually the data
16:45is stored. Now in this example each
16:47record represent one customer one
16:50person. So we have one record for Maria,
16:52John and Peter. Those we call them rows.
16:55Now in each table there is like one very
16:57important column called the primary key.
16:59It is always very important to have like
17:01one unique identifier for each customer
17:04for each row and we use it for different
17:06purposes in order to combine it with
17:08another table in order to identify
17:10quickly one customer. So it is unique.
17:13It's like fingerprint and there is no
17:14two customers having the same ID. Now
17:17the overlapping between the columns and
17:18the rows we have a single value a cell
17:21and each value each column stores
17:24specific data type. A data type it is
17:26like what kind of data we are storing
17:28like an integer 1 2 30 or a decimal
17:31where you have a decimal point 3.14. Now
17:34if you want to store characters we have
17:36different data types for that like you
17:37want to store the name or the
17:39description. So here we can use the char
17:42or the vchar. So you store inside them
17:44like the first name Maria or something.
17:46Now you might ask what is a char or
17:48vchar. So the char always a fixed one.
17:50So if you define it like five characters
17:52always it's going to go and reserve five
17:54characters from the space. But if you
17:56want things more dynamic then you go
17:58with the vchar. And now moving on we
18:00have another data types called the date
18:02and time. So if you want to store a date
18:04like the birth dates and if you want to
18:05store the time information you can use
18:07the time data type. So we call those
18:09stuff int, decimal, char, date, time.
18:12They are data types. So my friends, as
18:14you can see, SQL databases are very
18:16organized and
18:20structured. Okay. So now let's focus
18:22more about the SQL itself. We have in
18:24SQL different type of commands. So let's
18:27say that we have a database and this
18:28database is empty. So we have nothing
18:30inside it. Now, of course, the first
18:32thing that you have to do is to write an
18:34SQL with the command create in order to
18:37create brand new table in the database.
18:39So, once you executed the database going
18:41to go and build one, but this table is
18:43empty. So, we have nothing inside it. So
18:45now what you have done here is you have
18:47defined something new, right? And we
18:49call this type of commands the data
18:51definition language, the DDL. We have
18:53create to create something new, alter in
18:56order to edit something that already
18:57exists and drop in order to delete
19:00something. to drop for example a table.
19:02So this is the first family of commands.
19:04Now if you look at our table, it is
19:05empty. What do we need? We need data. So
19:07let's say that we have a website or an
19:09application. Now this application is
19:11generating a lot of data. Now in order
19:13for this application to move the data
19:15inside our new table, it must use the
19:18SQL command insert. So if you execute
19:20insert, you can add a new data inside
19:22your table. This type of commands we
19:25call it data manipulation language. And
19:27here we have three commands. insert in
19:29order to insert a new data, update in
19:31order to update an already existing data
19:33and delete in order to go and delete
19:35data from your table and that's why we
19:36call it data manipulation language
19:38because you are manipulating your data.
19:40So what do we have now? We have table,
19:42we have data inside the table. Now what
19:44we can do we can start asking questions.
19:46So let's say that you have analytical
19:48question about your data. Now all what
19:50you have to do is to write something
19:51called SQL query and inside it you use
19:54the command select but the whole thing
19:56we call it a query. So you send a query
19:58to the database, you have a question and
20:00the database can return for you the
20:02result, the data answering your query,
20:05your question and we call this type of
20:07activities using SQL, the data query
20:10language. And here we have only one and
20:12it is very famous. We have the select.
20:15We can use it in order to query our
20:17data. So those are the three different
20:19commands in SQL. And of course, we're
20:21going to learn all of them, but we will
20:22spend most of our time learning how to
20:25write the correct query for the correct
20:31answer. And now you might ask me, Barra,
20:33why we have to learn SQL? And if the
20:35time goes back, are you going to learn
20:37SQL again? Well, for sure, of course.
20:40And here are the top three reasons that
20:42I have. The first one, you have to learn
20:43it in order to talk to the data. You
20:45know, most of the companies stores their
20:47data in databases, and this is a
20:49standard way. This is how they do it.
20:50And if you want to work on the company
20:52in the data field and you want to talk
20:54to their data, then you have to use SQL.
20:56It's like you move to another country
20:58where they speak another language and
21:00you want to live there for a long time,
21:01you have to speak their language. The
21:03same thing here. If you want to work
21:04with data, you have to learn the
21:06language in order to speak to the
21:08database, the SQL. So this is for me the
21:10most important reason why we have to
21:11learn SQL and SQL it is in high demand.
21:14If you go now and check the job
21:15description of the software developer,
21:17data analyst, data engineer, data
21:19scientist, I promise you you will find
21:21there that they going to demand for SQL.
21:23So you will find they going to ask for
21:25SQL skills almost in each job
21:27description. So if you check for any
21:29data related jobs, you will find that
21:31they going to ask for SQL skills. Now
21:33another reason that I have is it is
21:35industry standard. So if you go and
21:37check multiple modern data platforms and
21:40tools like PowerBI, Tableau, Kafka,
21:42Spark, Synaps, you will understand that
21:45there will be always a section where you
21:47have to enter SQL code. So most of those
21:49vendors adopt SQL because it is the
21:52standard. It is widely used. It is like
21:54selling points that their tools are
21:56easy. So those are my top three reasons
21:58why SQL is still relevant and why you
22:01have to learn it. Okay, my friends. So
22:02with that we have now clear
22:03understanding what is an SQL why we need
22:06it what are databases and their
22:08different types why do we have DBMS
22:10servers and as well now you have
22:12understanding how things are very
22:14organized and structured inside the
22:16databases so that's all this is SQL all
22:19right so with that we have covered the
22:20basics about what is SQL and databases
22:23now in the next step we're going to go
22:25and set up our environments so that
22:27means we're going to prepare your PC
22:28with the data with the databases and all
22:30the tools that you need in order to
22:32learn
22:36SQL. Okay. So now go to the link in the
22:38description and you will land here in my
22:40newsletter website and you can subscribe
22:42if you want to get weekly news about my
22:45content. I make as well post about data
22:48and many other projects. So once you do
22:50that what we're going to do now we're
22:51going to go to the downloads over here
22:53and you will find here all the materials
22:55of different courses and the one that we
22:56want is SQL ultimate course. Let's go
22:58over here. Now once you do that you will
23:00land to this page where I have listed
23:02all the important links. So the first
23:04one and the most important one is to go
23:06and download the course materials. Here
23:08you can find everything code the slides
23:10the presentations the whole course or if
23:12you don't want that you can go to my get
23:14repository and there you will find
23:15exactly the same materials. So let's go
23:17and download everything. Okay. So now go
23:19and put the downloaded folder somewhere
23:21safe and let's go inside it. And here
23:24you can find three things. The first one
23:26is the data sets. Here if you go inside
23:28it you will find the data for the course
23:30the databases that we will be using in
23:32order to practice SQL. So everything is
23:35available here. Now the second folder
23:37you can find all the documentations. So
23:39that means all the visuals the
23:40presentation slides everything that I
23:43present during the course. It is
23:44available here as a documentation notes
23:46for you. Now moving on to the third one
23:48we have the scripts. So during the
23:50course we will be writing a lot of SQL
23:52codes and all those codes are here
23:54available. So that means those are all
23:56the codes that is used in the course.
23:58Okay. So with that you have now all the
24:00course materials. All right. So now the
24:02next step is that we have to go and
24:03download the SQL Server Express and you
24:05can find the link as well over here. So
24:07let's go there SQL Server Express. And
24:09now we're going to land on the Microsoft
24:11page where we can see the different
24:13offering from Microsoft where it's
24:14called server. So either we have it on
24:16the Azure or we can download it on the
24:18on premises. But we don't want those
24:20stuff. Just scroll down to see those two
24:22options. So the first option on the left
24:24side we have the developer edition. You
24:26will get all the features and services
24:28that Microsoft offers with the SQL
24:31server. It is as well free but the
24:33installation here is little bit
24:34complicated. But in the second option on
24:36the right side we have the express
24:38edition. Installation here going to be
24:40really fast and very easy. You will get
24:42as well all the stuff that you need for
24:44practicing SQL and learn SQL. So both of
24:46the options are free. It's just a matter
24:48of the installation. We will go now for
24:50the express edition. So go and click
24:53download now and it's very small file.
24:55So let's go and start it. And now the
24:57installation going to start. So we have
24:59basic, custom and download media. So
25:00download media means download now and
25:02later we're going to do the
25:03installation. Custom means we have more
25:05control on how to download and install
25:07the stuff. The basic is the easiest one
25:09and the quickest one. So let's go with
25:11the basics and click on that. And let's
25:13go and accept all those stuff. And now
25:16let's click on install. So now it's
25:18going to install the applications,
25:19drivers and so on. It may take a little
25:22bit
25:24time. So in order to do that, let's go
25:27and click on install SS SMS. So let's
25:30click on that and as well we can find
25:32the link over here. So let's go to SQL
25:34Server Management Studio. So let's click
25:36on that. You can find of course this
25:38link as well with the other links that I
25:40have collected. So now we are again at
25:41Microsoft page. Let's go scroll down and
25:44now we will see the following link free
25:46download for SQL Server Management
25:48Studio SS SMS. So let's go and click on
25:51that and then it's going to go and
25:52download it. Let's go and start it. So
25:55the first thing that we have to define
25:56the location. I will go with the default
25:58stuff. So let's click on
26:02install. Okay. Setup completed. We just
26:05installed SM SS SMS. So let's go and
26:09close it. So now let's go and start it.
26:11If you go to your menu over here, search
26:13for SQL Server and you will find it
26:15here. SQL Server Management Studio.
26:17Let's go and start it. Okay, so now
26:20we're going to get this window in order
26:21to connect to our server. So again, what
26:23is our server? It is the one we have
26:25installed at the first step, SQL Server
26:28Express. And that's why you're going to
26:29see in the server name, your PC name, of
26:32course, like it's not going to be my PC
26:34name. But here we have something called
26:36SQL Express. This is the server we just
26:38installed. So in the first option, we
26:40have database engines. We have reporting
26:42services. Those are different stuff from
26:44Microsoft. We're going to leave it as a
26:46database engine. And it should be like
26:48this. SQL Express. Now, how to access
26:51this database? We have the following
26:53stuff. We can do that using the window
26:55authentications or SQL server
26:56authentications. I'm going to say that
26:58let's stick with the window
26:59authentication. And the username going
27:01to be the PC name and as well the window
27:04user. If you don't have it for some
27:06reason those informations, you can go to
27:08your search search for
27:10cmd and then here you can say who am I?
27:15And with that you will get the PC name
27:17and as well the user that you are
27:19currently logged in. And this is exactly
27:21what I'm seeing over here. One more
27:22thing if you're having issue connecting
27:24to your database make sure to check the
27:26encryption. It should be mandatory and
27:28to click on the trust server
27:30certificates. So once you do that you
27:31will be able to connect. Okay. So with
27:33that we have the server we have the
27:35client. And now the last step we have to
27:37go and create the database. We want to
27:39insert our data. So now if you look to
27:41the object explorer and open the
27:43databases you can see that we don't have
27:44any database. So now let's do something
27:46about it. Go back to the course
27:48materials inside the data sets you will
27:50find the following. You will find we
27:52have here three folders MySQL postcress
27:55and SQL server. So if you want to follow
27:57with this course using different
27:59database like MySQL and Postgress you
28:01can find the exact same data for the
28:03database that you are using. But now in
28:05this course we are using the SQL server.
28:07So if you follow me with that go inside
28:10the SQL server folder and here you will
28:12find four files with different
28:13extensions. So what is going on here?
28:15Now for this course we have two
28:16databases. One that is very simple
28:19called my database and second one that
28:21has more tables called sales DB. And now
28:24in SQL server there are multiple ways on
28:26how to create databases. I will show you
28:28now two methods on how to create the
28:30database. Now the first option we want
28:32to create the database from a script.
28:34And if you look to those files, we have
28:36here two files with the extension SQL.
28:39Those are files with SQL code. So let's
28:41start with the first one, the init SQL
28:43server my
28:45database.SQL. Go inside it. And now here
28:47we have the SQL code. Copy everything.
28:49And now let's go back to our studio and
28:51then go to the menu and click on new
28:53query. And here in the middle you can
28:55paste the code. So now we have the code
28:57for the first database. And all what you
28:59have to do is to go and execute it. So
29:01once we executed you will see we will
29:03not get any error. And now on the left
29:05side we don't see yet our database
29:07because we have to refresh. So right
29:09click on the databases and click
29:10refresh. And now you can see it my
29:12database. So now let's see the content.
29:14Go extend it and then go extend the
29:16tables. And now you see here our two
29:18tables customers and orders. Inside
29:20those tables we can find our data. In
29:22order to see the data right click for
29:24example of the customers and let's go
29:26with the option select top 1,000 rows.
29:28Once you do that you can see now in the
29:30results we have here five customers.
29:33This is our data inside the table
29:34customers. So here again about the
29:36interface on the left side we have the
29:38object explorer where you can see the
29:40whole structure of the database from
29:41server to databases to tables. So you
29:44can see the whole structure on the top
29:46we have a menu with a lot of icons and
29:48then in the middle this place here we
29:50call it the SQL editor. We're going to
29:52go and write their SQL codes and then
29:55once you execute it at the bottom you
29:57will get the result and messages and
29:59below the SQL editor we have the output.
30:01So here you can see for example the data
30:03the results or different messages from
30:05the database. So the interface is very
30:07simple. Now we have to go and get our
30:09second database. So if you go back to
30:11our files you can find a second SQL file
30:14the initql server sales db.sql. Open
30:17that and let's go and copy everything
30:19here and let's go back to our studio.
30:22Same thing you have to go and create a
30:23new query then paste the whole code and
30:26this database is about the sales DB. So
30:29let's go and execute it and with that we
30:31will not get any errors and now we go to
30:33the left side and we do the same thing
30:35refresh and we can see the second
30:37database sales DB. Now we can go and
30:39explore it. So extend it go to the
30:41tables and here you can see five tables
30:43customers employees orders products. So
30:46here this is the intermediate database
30:48for our course. So now let's go and
30:49check our data. For example, let's go to
30:51the orders, right click on it and select
30:53top 10,00. And those are the orders of
30:56our database. Perfect. So everything is
30:58working. So those are the main two
31:00databases that we will be working
31:02through the whole course. And of course
31:03if you want to go and practice using
31:05another database, it's totally fine. For
31:07example, in Microsoft, there are a
31:09database called Adventure Works. It is
31:11really amazing. And I'm going to show
31:13you now how to import it. We can go over
31:14here the adventure works. So let's click
31:17on this link. So now we are again in
31:18Microsoft page. If you scroll down you
31:21can see here three different types of
31:22databases. The OLTB, data warehouse and
31:25lightweights. So they are like different
31:27databases. The OLTP is the most like
31:30complicated one. A lot of tables and
31:32transactions and so on. The data
31:34warehouse it is like really nice one in
31:36order to do data analyzes and stuff. The
31:38lightweight it is the simplest one. So
31:40let's go for example and get the data
31:42warehouse. So click on that and now as
31:44you can see the extension of this file
31:47isbak and now I'm going to show you the
31:49second way on how to create databases in
31:51SQL server. So now all what you have to
31:53do is to go to the following path. It
31:55really depends where you have installed
31:57the SQL server. So for me I have
31:59installed it in the program files
32:00Microsoft SQL Server MSSQL SQL Express
32:04then MSSQL backup. You have to go there.
32:07So here what you can do you can place
32:09all the files with the extension bak.
32:12For example, the adventure works that we
32:13just installed. This is a backup file
32:16for the database and we want to go and
32:18restore it and with that you are
32:20creating like a database. So this is the
32:22second method on how to create databases
32:24in SQL server by restoring the database.
32:27If for some reason the script didn't
32:29work for you. Now let me show you
32:30quickly how we can do that. Let's go
32:32back to our studio. Right click on the
32:34database and then here we have an option
32:36called restore database. Click on that.
32:38And now here we have two options under
32:40the source database and device. The
32:42default going to be database but we have
32:44to switch to a device because we want to
32:46import it from files. And then we go to
32:49these three dots. Click on that. And now
32:51we have to go to the option add. And now
32:53it's going to take you to the place
32:54where the SQL server creates backups. So
32:57here we can find our files and what we
32:59want you to create is the adventure
33:00works. Select that. Then okay, one more
33:03okay and one final okay. So now the
33:06database will be restored and it is
33:08successfully. So now on the left side we
33:10can see our third database. If you don't
33:12see it go and refresh of course and here
33:14you will find a lot of tables in the
33:16adventure works. And as usual we can go
33:18and explore the data by selecting top
33:20thousand rows. So my friends now you
33:22have three databases but of course our
33:24focus is only the first two that we have
33:25done my database and sales DB. And with
33:28that you have learned two ways on how to
33:30import databases into SQL server. So
33:33with that my friends we have prepared
33:34everything. We have the SQL Server
33:36Express running on your local PC. We
33:38have the studio the clients where we're
33:40going to use it in order to interact
33:42with the database and we have created
33:44our two databases that we will be using
33:47in order to practice SQL. So we are
33:49ready. All right my friends. So with
33:50that we are done with the first chapter.
33:52We have our introduction to SQL and now
33:54we're going to start learning the first
33:56thing in SQL and that is how to query
33:58our data. So let's go and start with
34:00that.
34:05Okay, so now we can understand exactly
34:07what is an SQL query. Now normally your
34:09data is inside the table and your table
34:11is inside the database and now you might
34:13have a question from the business like
34:15what is the total sales? What is the
34:17total number of customers? So any
34:19question that you have in your mind and
34:21you want to go and ask your data you
34:23want to go and retrieve data from the
34:25database and in order to do that you
34:27have to talk to the database using its
34:29language the SQL. So in order to do that
34:32you're going to go and write a query
34:34where you write inside the query
34:36something called select statement and
34:38with that you are asking the database
34:40for data. So once you execute your query
34:42the database going to go and fetch your
34:44data and then it prepares a result to be
34:47sent back to you. So with that you are
34:49asking the database a question by
34:51writing a query and the database going
34:53to process your query and answer your
34:55question by sending back data and with
34:57that we are like reading our data from
35:00the database and the queries will not
35:02modify anything will not change the data
35:04inside your tables or even change the
35:06structure of the database. So you use
35:08select statement only in order to read
35:11something from the database. You just
35:13want to retrieve data from the database.
35:16So this is what we mean with a
35:20query. And now my friends, each SQL
35:23query has usually different sections,
35:25different components. We call them
35:27clauses. And this is amazing because
35:29you're going to have enough tools to
35:31write a query that matches any question
35:33that you have about your data. So what
35:35we're going to do, we're going to cover
35:36all those clauses step by step in order
35:39to write any query that you need. So now
35:41we're going to start with two clauses
35:43that makes the simplest query in SQL.
35:45the select and from. So let's start with
35:51that. All right. So now it's really
35:53important for me that you understand how
35:55SQL works with the code with the
35:57queries. So now what I'm going to do,
35:58I'm going to show you on the right side
36:00the syntax of the query in SQL and then
36:03on the left side I'm going to show you
36:04exactly step by step how SQL going to go
36:07and execute your query. So now we have
36:09the table customers inside our database
36:11and we will start with the easiest form
36:13where we're going to select everything.
36:15Select the star. So the select star is
36:17going to go and retrieve all the columns
36:20from your table. So everything and the
36:22from clause it's going to tell SQL where
36:24to find your data. So with the select we
36:27select the columns that we want and the
36:29from you specify the table where your
36:31data come from. So the syntax going to
36:33be very simple. In each query we start
36:35always with the select. And now since we
36:37want all the columns we're going to
36:39write star and with that SQL going to
36:41understand I want to see everything. And
36:43then after that comes the keyword from.
36:45And now we want to tell SQL where the
36:46data come from. So we have to specify
36:48the table name. And that's it. This is
36:50all what you need to do. So once you
36:52execute it what's going to happen? SQL
36:54going to go and execute first the from
36:56clause. So it's going to go and retrieve
36:58all the data from the database to the
37:00results. And then in the next step going
37:02to go and check the select statement. So
37:04which columns we have to keep in the
37:06result since you are saying star then
37:08the SQL going to keep everything all the
37:10columns and with that you will see in
37:11the result everything all the columns
37:13and all the rows. So that's it. This is
37:16how it works. Now let's go back to scale
37:18in order to select few data from our
37:19database. Okay. So back to our studio.
37:21Let's go and start a new query and let's
37:24go and find our database just to expand
37:26it and our tables. Now it is very
37:28important to make sure that you are
37:30connected to the correct database. So go
37:32to the top left in the menu over here
37:34and make sure to select your database.
37:36So my database like this or we have a
37:38command for that called use and then
37:40just write the database name like this.
37:42So I'm telling SQL just use my database
37:45like this and with that SQL going to
37:47switch to your database. Now if you are
37:49learning any new programming language,
37:50it is very important to understand about
37:52the comments. So comments are like notes
37:54that you add to your code in order to
37:56understand what is going on. And of
37:58course the engine, the database will not
38:00go and execute it. it's going to go and
38:02ignore everything inside it. And there
38:04is like two ways on how to do that.
38:05Either you make inline comments by
38:07typing two dashes like this and then you
38:09write anything this is a comment. So now
38:13in SQL if you see it is green that means
38:15it is a comments. Now the other type you
38:18can have multiple line comments and in
38:20order to do that what you can do you can
38:21write slash and then start and then you
38:24can write anything this and then start a
38:26new line is a comment. So as you can see
38:29all the lines after the slash star it is
38:32getting green that means it is a comment
38:34and now let's say that you are at the
38:35end. So in order to close it you write
38:37again star and then slash and that you
38:40are telling SQL I'm done with my
38:41comments. So those are the two types of
38:43writing comments in SQL. Now back to our
38:45query. Let's say that we have the
38:47following task says retrieve all
38:49customer data. So I would like to see in
38:51the results all the data of my customers
38:53everything all the rows and all the
38:55columns. So currently our data is stored
38:57inside the table called customer and I
38:59need to see all the data in the output.
39:01In order to do that we're going to write
39:03a query and all our query start always
39:05with a select and since I need
39:07everything all the columns we write star
39:09and then a new line. Let's go and
39:10specify for SQL from where it's going to
39:13go and get the data. So it's going to be
39:15from and then we going to write the name
39:17of the table. It must be exactly like it
39:19is in the database. So it's called
39:21customers and you have to have it here
39:23as a customers. So that's it. Let's go
39:25and execute it. And now if you look to
39:27the results, you can see we have four
39:29columns and five rows. So with that you
39:32are seeing everything inside the table
39:34customers. You can see we have five
39:35customers and you can see all the
39:37columns about the customers. So this is
39:39very simple. We have ask question for
39:41the database using SQL query and the
39:44database should answer our question by
39:46returning our data in the results. All
39:48right. So now let's move to another
39:49task. I'm going to go and create a new
39:51query and this time we're going to
39:52retrieve all the order data. So that
39:54means I would like to see all the data
39:55inside the orders. So let's go and write
39:57a very simple query. We start as usual
40:00with select and since we want
40:01everything. So it is select star from
40:04our table orders. So that's it. Let's go
40:07and execute. And with that you can see
40:08in the output we have again four columns
40:11but this time we have only four rows. So
40:13that means in this table we have four
40:15orders and we can see all the data
40:17inside this table. So with that we can
40:19understand we have five customers inside
40:22our database and these customers did
40:24generate four orders. So as you can see
40:26we are now talking to our database and
40:28this is the simplest form of query in
40:34SQL. All right. So now let's move to the
40:36next step in our query where you say you
40:39know what I don't want to see all the
40:40columns from the database. I want to be
40:42more specific. So I would like to select
40:44exactly the columns that I need. So now
40:47we want to select few columns from the
40:49database where we select only the
40:51columns that we need instead of
40:53everything. Now about the syntax we're
40:54going to go and change a little thing.
40:56So instead of using star we're going to
40:58go and make a list of columns that we
41:00want to see in the output. So we're
41:02going to select column one column two
41:04and we're going to separate them using a
41:06comma. So we are just writing a list of
41:08columns exactly after the select. And
41:10for the from it's going to stay as it
41:12is. So from a table. Now if you execute
41:14this what going to happen as usual SQL
41:16going to start with the from. So it's
41:18going to go and get the data from the
41:19database and then the next step is going
41:21to go and check the select. So what
41:23going to happen? SQL going to go and
41:24keep only two columns like for example
41:27the name and the country and all the
41:29columns that are not mentioned in the
41:31select statements will be excluded. So
41:33SQL going to go and remove it from the
41:35results and keeps only the columns that
41:37we mentioned in our query. So this time
41:40instead of having four columns in the
41:41output we can have only two. So with
41:43that you are like filtering the columns
41:45and you are selecting exactly what you
41:47need. So now let's go back to SQL in
41:49order to practice this. All right. So
41:51now we have the following task and it
41:53says retrieve each customer's name,
41:55country and score. So that means I don't
41:58want to see everything from the table
42:00customers. I need only to see the three
42:02columns. So let's see how we can do
42:04that. As usual we start with select and
42:06I'm going to go with a star in order to
42:08see the whole table first from the table
42:09customers. So it's exactly like before.
42:12Let's go and execute it. And now I can
42:13see everything inside the table
42:15customers. But the task says I need only
42:17three columns. So now what we're going
42:18to do instead of the star, we're going
42:20to make a list of columns. So we start a
42:23new line and then we write the name of
42:25the first column. So the first name and
42:27a new line for the second column for the
42:30country and then again a comma and then
42:32we write a score. So with that we have
42:34the three columns. Now what I usually
42:36do, I go and select them and give it
42:38then a push using a tab. This just looks
42:40nicer and easier to read. So with that
42:42we have now between the select and from
42:44list of columns. Now there is like
42:46mistake that happens a lot where we go
42:48and type a comma after the last column.
42:50So if you do that and execute it you
42:52will get an error because SQL going to
42:54expect from you a column after the comma
42:56and since there is no column and
42:58immediately you have a from you will get
43:00an error. So there is no need for a
43:01comma after the last column. Now let's
43:03remove it and execute. And now that you
43:05can see in the output we don't have four
43:07columns we have only three. the first
43:09name, the country and the score. And by
43:11the way, they are ordered exactly like
43:14you selected in your query. So first we
43:16have the first name and then the country
43:19and then the last one the score. So that
43:20means if I go and now change the order.
43:23So let's get the country at the end and
43:25execute. You will see the country at the
43:28end. I'm going to go and put it back in
43:30between to match exactly like the task
43:32and remove the last comma. So execute
43:35again. And with that we have selected
43:37few columns from our table. So we are
43:39more specific to what we need. Okay. So
43:41that we have covered the two select and
43:43from next we're going to talk about the
43:45wear clause that you can use in order to
43:48filter your data. So let's
43:53go. So what is exactly where? We use
43:56where in order to filter our data based
43:58on a condition and any data that fulfill
44:01the condition going to stay in the
44:03output in the result and the data that
44:06don't meet the condition will be
44:08filtered out of the results. Condition
44:09could be anything like for example we
44:11say the score must be higher than 500 or
44:14you can say the country must be equal to
44:16Germany. So any condition that you have
44:18in your question. Now let's see the
44:19syntax in SQL. As usual we start with a
44:22select. We select the columns that we
44:24need. Then we write from where the data
44:26come from and then after the from we're
44:28going to write the where and exactly
44:30after that you specify your condition.
44:32So now let's see how SQL going to
44:33execute this. First SQL start as usual
44:36from the from. So it's going to go and
44:37get your data from the database and
44:39after that SQL going to go and execute
44:42the wear clause. So let's say that the
44:44condition should be higher than 500. And
44:46now what going to happen? SQL going to
44:48check each row whether it meets this
44:51condition or not. So for example for
44:52Maria she doesn't fulfill the condition
44:54because her score the 350 is not higher
44:58than 500. So she doesn't fulfill the
45:00condition and SQL going to go and remove
45:03completely this row this record from the
45:05results. Now SQL going to go to the
45:07second record. So Joan is fulfilling the
45:10condition. So he going to stay in the
45:11result. The same thing for George. Now
45:13moving on to the fourth one Martin. So
45:16this customer is not fulfilling the
45:18condition and SQL going to go and remove
45:20it from the results. The same things
45:22happen for the last customer. The score
45:23is zero and not fulfilling the
45:25condition. So that means if we apply
45:27this filter, SQL going to return only
45:29two customers out of five. So with that
45:32we are filtering the rows based on
45:34condition using the work clause. Now as
45:36you can see in the result we are getting
45:38all the columns but if you specify in
45:40the query like for example only two
45:42columns like the name and the country
45:44then SQL going to start removing as well
45:47the columns of the results. And this
45:49means in the output we will get only two
45:51columns and two rows. So with that you
45:53are filtering the columns and the rows
45:56of your results. So now let's go back to
45:58scale in order to practice this. All
45:59right. So let's have the following task
46:01and it says retrieve customers with a
46:04score not equal to zero. So now if you
46:06are looking to our task you see we have
46:08like here a condition. The condition
46:09says the score must not be equal to
46:12zero. So I don't want to see all the
46:13customers. I want to see only the
46:15customers thus fulfill this condition.
46:18So it's like we have to filter the data.
46:19So let's go and solve the task. Let's
46:21start as usual. Select star. There's no
46:23specifications about the columns from
46:25our table customers. Okay. So I'm going
46:28to start with this. Let's go and execute
46:30it. Now if you look at the result, you
46:31can see like almost all the customers
46:33are fulfilling the condition. Their
46:35scores are not equal to zero. Only one.
46:38The last customer his score is zero. So
46:40this customer does not fulfill our
46:42condition. Now let's go and build filter
46:44for that. So we're going to say where.
46:46And now there will be a section that is
46:48only focusing on how to build conditions
46:50and filtering in SQL. So don't worry a
46:52lot about the syntax of the conditions.
46:54We're going to cover that later of
46:55course but it is very simple. Now for
46:57the condition we need a column. So in
46:58which column is our condition based on
47:00it's going to be on the score. So we're
47:02going to write here score and since we
47:04are saying not equal there is like an
47:06operator in SQL called not equal and
47:08then we have to write a value after
47:10that. It's going to be a zero. So again
47:11the condition is like this. The score
47:13must not be equal to zero. It's very
47:16simple, right? And with that we have our
47:18condition and we are using the where in
47:20order to filter the data. So let's go
47:22and execute it. And now as you can see
47:23SQL did remove the last customer because
47:26he is not fulfilling this condition. And
47:28we have now only the rows that fulfill
47:31our condition. So as you can see it is
47:33very simple how to filter the data. All
47:35what you have to do is to write where
47:36clause after the from and then write a
47:39condition after that. Now let's have
47:40another task like for example it says
47:42retrieve customers from Germany. So I
47:45don't want to see all customers from
47:46different countries. I just want to see
47:48the customers that come from Germany. So
47:50that means we have a condition here.
47:52Country of the customer must be equal to
47:55Germany. So let's go and remove the
47:56current condition. It is not the one
47:58that we need and execute. If you are
48:00looking to the results, we have two
48:02customers that come from Germany and we
48:04are interested only to show those two
48:06customers. So let's go and make a filter
48:08for that. We're going to write where
48:10clause and after that we need a column.
48:12The column going to be the country. So
48:14we're going to write here country and
48:15this time the country must be equal to
48:18Germany. So we're going to write an
48:20equal operator. So we're going to write
48:22Germany like this exactly like the value
48:24inside our data. But now as you can see
48:26we are getting like an error here. And
48:28that's because in SQL if you want to
48:30write a value that contains characters
48:33then you have to put it between two
48:35single quotes. So at the start you put a
48:37single quote and as well at the end. And
48:39now as you can see the red line is away
48:41and the value now is red and that's
48:43because it is a string value. It is a
48:45value that contains characters and with
48:47that you will not get an error. So if
48:49your columns contains only numbers you
48:51can write it without single quotes. But
48:53if your values contains characters then
48:55you have to write it between two single
48:57quotes. Okay. So now back to our
48:58condition the country must be equal to
49:00Germany. Let's go and execute it. And it
49:03is working. So as you can see now we are
49:04seeing in the output only the customers
49:07does fulfill my condition where the
49:09country is equal to Germany. So this is
49:11exactly how we work with the wear clause
49:13in order to filter our data. So my
49:15friends this is how you filter your
49:17rows. And now let's say that I would
49:18like to filter the rows together with
49:20the columns. So I just want to keep the
49:22first name and the country and not
49:24interested to see the scores and the
49:25ids. So in order to do that we're going
49:27to go to the select and list the columns
49:29that we want to see. So the first name
49:32and after that a comma then the country
49:34and that's it. So let's go and give it a
49:37push and execute it. So we have two rows
49:39and two columns. So guys as you can see
49:42SQL is very simple. All right. So with
49:44that you have learned how to filter your
49:46data using the wear clause. Next we're
49:48going to talk about how to sort your
49:50data using the order by. So let's
49:56go. Okay. So what is exactly order by?
49:59You can use this type of clouds in order
50:01to sort your data. And of course, in
50:03order to sort your data, you have to
50:04decide on two mechanism. Either you want
50:06to sort your data ascending from the
50:09lowest value to the highest value or
50:11exactly the opposite way using
50:13descending from the highest value to the
50:15lowest. And the syntax kind of looks
50:17like this. So as usual, we start with
50:18the select and then from and after the
50:21from you can specify order by and with
50:23that you are telling SQL we have to sort
50:25the data and you have to specify two
50:27things. First you have to specify for
50:29SQL the column that should be used in
50:31order to sort the results. So for
50:33example you can say score and after the
50:36column name you have to specify the
50:37mechanism. So for example you say
50:39ascending from the lowest to the
50:41highest. And in SQL if you don't specify
50:43the mechanism the default going to be
50:45ascending. So you will not get an error
50:47if you don't specify anything after the
50:49column name. But my advice here is
50:51always to specify something after the
50:54column easier because it's just
50:56straightforward and easier to understand
50:58and if someone reads it can understand
51:00immediately it's going to be ascending
51:02because maybe not everyone knows what is
51:04the default in SQL. So always specify a
51:07value even if it's like easier to skip
51:09it and if you want to store the data
51:10from the highest to the lowest then you
51:12can specify descending. So as usual SQL
51:15going to go and start from the from it's
51:16going to go and grab your data from
51:18database. Then the second step is SQL
51:20going to go and sort the result. So the
51:22order by going to be executed and SQL
51:24going to see okay I'm going to sort it
51:25by the score and using the sending
51:28mechanism and still going to go and
51:29start like moving around your rows where
51:32the first row going to be the customer
51:34with the highest score and in this
51:36example John has the highest score the
51:38900. So John going to appear as a first
51:41row at the result and that's because his
51:43score and after that the second highest
51:45is going to be George with 750 and SQL
51:48going to go and keep sorting the data
51:50and then we have 500 then 350 and the
51:53last row going to be the customer with
51:55the lowest score the zero. So this is
51:57how SQL executes your order by. Now
52:00let's go back to scale in order to
52:01practice. All right. So now we have the
52:03firming task and it says retrieve all
52:05customers and sort the result by the
52:08highest score first. So now by looking
52:10at the task we need all the customers.
52:12So there is like no conditions or
52:14anything to filter but we have to sort
52:16the results. So let's go and do that.
52:18We're going to start as usual by
52:19selecting all the columns from the table
52:22customers. So now if you go and execute
52:24it you will get all your customers and
52:26you are now seeing the data exactly like
52:28stored in the database. And you can see
52:30the result is not sorted by the scores.
52:32So we have here a low score then high
52:34score then low and so on. Now the task
52:37says we have to sort the results. So we
52:39have to go and use the order by and now
52:42you have to understand from which column
52:43and we can get that from the task. So it
52:45says it should be sorted by the score.
52:48So we're going to go and define the
52:49score here. And the final thing that you
52:51have to define is the mechanism
52:52descending or ascending. And you can get
52:54it as well from the task. So we have to
52:56sort the data by the highest score
52:58first. So the highest first and then the
53:00lowest. So that means we're going to go
53:02and use the descending. So that's all.
53:04Let's go and execute it. Now as you can
53:06see in the results, the first customer
53:08has the highest score. Then we have the
53:10second one with the second highest until
53:12the last one with the lowest score.
53:14That's it. This is how you sort your
53:16data. And with that we have solved the
53:17task. Now let's do exactly the opposite.
53:20So we want to sort the results by the
53:22lowest score first. So that means we
53:24want to see first the customers with the
53:26lowest score like here in this example
53:28we should see the ID number five as the
53:30first because he has the lowest score
53:32the zero. Now in order to do that all
53:34what you have to do is to switch the
53:35mechanism instead of descending when you
53:37can use ascending. Let's go and execute
53:39it. And that's it. As you can see now we
53:41have the lowest score then the second
53:43lowest score until the last row. It's
53:46going to be the customer with the
53:47highest score. So the lowest score comes
53:50first. So it is very simple. This is how
53:52you sort your data using
53:56SQL. And now I'm going to show you one
53:58more thing that you can do with the
54:00order by. You can sort your data using
54:02multiple columns. And we call it nested
54:04sorting. So now let's take this very
54:06simple example where you want to sort
54:08your data using country. So we are
54:10saying order by the column country and
54:13the mechanism going to be ascending. So
54:15from the lowest to the highest. Now if
54:16you do that going to go and sort the
54:18data this time based on the country. So
54:20we're going to have like the first two
54:22customers from Germany. It is sorting it
54:24alphabetically. Then we have the UK and
54:26the last two going to be from USA. Now
54:28if you are checking the final results
54:29you might say you know what there is
54:31like something wrong. The data is not
54:34completely sorted correctly. So if you
54:36are looking to the first two customers
54:37that come from country Germany. You can
54:39see the scores are sorted in ascending
54:42way from the lowest to the highest. So
54:44first we have 350 then 500. Then UK it's
54:47fine because we have only one customer.
54:49Now if you look to the customers from
54:51USA you see that it is like sorted the
54:53way around. It is sorted descending from
54:56the highest to the lowest. So first we
54:58have the score 900 then zero. So there
55:00is like no clean way on how the data is
55:02sorted and the result is not really
55:04clean and this issue happens usually if
55:06you are sorting your data based in a
55:08column that has repetition like here the
55:10country we have twice Germany and twice
55:12USA. So now in order to refine the
55:15sorting and make it more correct, we can
55:17include in the sorting another column in
55:20this scenario for example the score. So
55:22we can make a list of columns in the
55:24order by and we can separate them using
55:26the comma. And of course you can have
55:27different mechanism for each column like
55:29for the country we are saying it is
55:31ascending but for the score we say you
55:33know what let's make it descending. It
55:34will not be only one for all columns. So
55:37now what can happen is we're going to
55:38start sorting the data for each section.
55:41So for the two customers from Germany
55:43the sorting going to be from the highest
55:44to the lowest. So it's going to go and
55:46switch the two customers. So Martin
55:48going to be first because he has higher
55:50score than Maria. And with that we are
55:52refining the scores based on the same
55:54value of course the country. Now for the
55:56UK nothing going to happen because we
55:58have only one value and for the USA as
56:00well nothing going to happen because it
56:02is already sorted in the correct way
56:04from the highest to the lowest. So as
56:06you can see if you are including a
56:07second column you are refining your
56:10sorting and as well my friends the order
56:12is very important. So this is how you
56:14can do nested sorting in SQL. Let's go
56:17back to our SQL and start practicing.
56:19All right so now we have the following
56:20task and it says retrieve all customers
56:23and sort the results by the country and
56:26then by the highest score. So again we
56:29need all customers. So select everything
56:31from customers table. And now the task
56:33says we have to sort the result by the
56:36country. So we're going to start with
56:37the order by and since it says by the
56:39country. We're going to go with the
56:40country and we're going to sort it
56:42alphabetically. So it's going to be
56:43ascending. So let's go execute it. Now
56:46you can see the data is sorted
56:47completely differently by the country.
56:49So we have first Germany, UK and then
56:51USA. But that's not all and says then by
56:55the highest score. So we have to go and
56:57include another column in the sorting
56:59and we can go and add that by adding a
57:01comma and then mention another column
57:03the score and now we have to specify the
57:05mechanism. It says by the highest score.
57:07So the highest must come first and with
57:09that we are using descending. Now what
57:11is the current situation in that? If you
57:13look to the results for example for
57:15those two customers we have 350 and then
57:18500. So that means the scores are sorted
57:21ascending right the same thing for USA.
57:23So from the lowest to the highest. Now
57:25if you go and do it like this what going
57:27to happen it's going to go and switch
57:29it. So you can see over here now for
57:31Germany first comes the highest the 500
57:33and then the 350 and for USA as well
57:36they switched. So we have the highest
57:38and then the lowest and with that we
57:40have solved the task. Now again the
57:41order of those columns are very
57:43important. So since the scores comes
57:45after the country we will not get the
57:47highest scores first at the results. So
57:49we will not get the 900 as a first row.
57:52And that's because the scores must be
57:54sorted after the country. So the country
57:56has more priority. Now if you go and
57:58flip that. So let's go over here and
58:01says sort first the score and then the
58:03country. So let's go and execute it.
58:05It's called has first to sort the
58:07scores. So with that you will get the
58:09900 first, right? And then the
58:11countries. And since there is like no
58:13duplicates in the scores, this makes no
58:15sense at all. So you can go and skip it.
58:17So nested sorting only makes sense if
58:19you have repetition in your results and
58:22you can use the help of a second column
58:25in order to make the sorting perfect. So
58:28that's it and with that of course we
58:29have solved the task. All right. So with
58:31that you have learned how to sort your
58:32data using order by. Now in the next
58:35step we're going to talk about how to
58:37aggregate and group up your data using
58:39group by and we're going to put it
58:41between the where and the order by
58:43because in the order of the query the
58:45group by comes between the where and the
58:47order by. So let's
58:52go. Okay. So what is exactly group by?
58:54It's going to go and combine the rows
58:56with the same value. So it's going to go
58:58and combine and smash press your rows to
59:01make it aggregated and more combined. So
59:03all what group by does it aggregates a
59:06column by another column. Like for
59:08example, if you want to find the total
59:10score by country. So you aggregate all
59:13the scores value for one country. If you
59:16have this kind of tasks, then you can
59:17use the group I. Let's see the syntax of
59:20that. We will start as usual with the
59:21select. And now what we want to see in
59:23the result is two columns. So we have to
59:25specify like a category like the
59:27country. This is the value that you want
59:29to group the data by. and another one
59:31where you are doing the aggregations. So
59:33for example you are saying I would like
59:35to see the total score. So we use the
59:37function sum in order to summarize the
59:39values of the score. After that as usual
59:41we use the from in order to select the
59:43data from specific table. And now comes
59:45the magic we use after the from group
59:47by. And now understands okay I have now
59:50to combine the data. I have to group up
59:52the data by something. And this time we
59:54are saying you have to group up the data
59:56by the country. So that means each value
59:58of the country must be presented in the
1:00:00output only once and for each country we
1:00:03want to see the aggregation and that is
1:00:06the total score. So let's see how is
1:00:08going to execute it. So it's going to
1:00:09first start with the from it's going to
1:00:10go and get the data from the database
1:00:12and then it's still going to execute the
1:00:13group by and now scale understand okay I
1:00:16have to group up now the data by the
1:00:17country and it understands it has to
1:00:20aggregate the scores for that. So it's
1:00:22going to go and identify the rows that
1:00:24are sharing the same value. Like for
1:00:26example here we have two rows for
1:00:28Germany and it's going to bring it to
1:00:29the results. So now we have two rows for
1:00:31the same country but since we are saying
1:00:34group by country SQL going to try and
1:00:36combine them smash them together in only
1:00:38one row. So each value of the country
1:00:40must exist at maximum once. We cannot
1:00:43leave it like this. So now what we going
1:00:45to do with the scores? We have two
1:00:46scores. Now SQL going to check the
1:00:48aggregate function. It is the
1:00:49summarization. So, and it's going to go
1:00:51and add those values 350 + 500. And with
1:00:54that, we're going to get the total score
1:00:56of 850. And with that, as you can see,
1:00:59scale is combining those two rows into
1:01:01one. So, in the output, Germany will
1:01:03exist only one. And about the scores, we
1:01:05will get the total score. And the same
1:01:07thing going to happen for the next
1:01:09value. In the country, we have the USA.
1:01:11We have it twice. So, we're going to get
1:01:13two rows. And scale going to combine
1:01:15those two rows in one because USA must
1:01:17exist only once. And with the scores we
1:01:20will have the total scores. So 900 plus
1:01:22zero we will get 900. And with that it's
1:01:25still converted those two rows into one.
1:01:27And for the last value in the countries
1:01:29we have the UK. It's going to stay as it
1:01:31is. There is no need to smash and
1:01:33combine anything because it's already
1:01:35one value. So my friends if you are
1:01:36looking to the output you can see we
1:01:38grouped the original data by the
1:01:41country. And that means we're going to
1:01:42get one row for each value inside the
1:01:45country column. So my friends the
1:01:47original data you have five rows in the
1:01:49output if you are using group by like
1:01:51this you will get only three rows. So
1:01:53this is exactly how the group by works.
1:01:56Let's go back to scale and practice.
1:01:58Okay. So we have the following task and
1:01:59it says find the total score for each
1:02:02country. So from reading this you can
1:02:04understand we have to do aggregations
1:02:06and we have to combine the data by a
1:02:08column. So now usually I start like
1:02:09this. I start selecting the columns that
1:02:12I need in order to solve this task. So
1:02:14what do we need? We need the country and
1:02:17score from our table customers. So let's
1:02:20start like this. Now you can see we have
1:02:21the countries and the scores. And the
1:02:23task says we have to group up the data
1:02:25by the country. So that means this is
1:02:27the column where we're going to do the
1:02:28group by and the total scores will be
1:02:31aggregated. So what we have to do? We're
1:02:32going to use the group by since it says
1:02:34for each country. We're going to use it
1:02:36over here. Group by country. And now we
1:02:38have to go and aggregate the scores. We
1:02:41cannot leave it like this. So we're
1:02:42going to say the sum of the score. So
1:02:44let's go and execute it. And with that,
1:02:47as you can see, we are getting the total
1:02:49scores for each country. So now instead
1:02:51of having five customers, we have only
1:02:54three rows now. And that's because the
1:02:56countries has three rows. And now if you
1:02:58check the result, you can see something
1:03:00weird. It says no column name. And
1:03:02that's because we have changed the
1:03:04scores. It's not anymore the original
1:03:06score. It is it is the total scores. We
1:03:09have summarized those values. So SQL
1:03:12don't know how we going to call it. So
1:03:14those values doesn't come directly from
1:03:16the database. It is manipulation that
1:03:18you have done here. Now in order to give
1:03:20a nice name for that we can go and add
1:03:22aliases. An alias it is only like a name
1:03:25that lives inside your query. So we can
1:03:27do it like this as and you can specify
1:03:29any name you want like for example total
1:03:32score. And now scale can understand okay
1:03:34this is the name for this column and if
1:03:36you go and execute it you will see the
1:03:38new name in the results. But you have to
1:03:40understand this name exists only in this
1:03:42query. You are not renaming anything
1:03:44inside your database and you cannot use
1:03:47it in any other queries. It is just
1:03:49something that is known inside this
1:03:51query and only for your results. And of
1:03:53course you can rename anything any
1:03:54column like for example here you can say
1:03:57this is the customer country and if you
1:04:00execute it you are just renaming the
1:04:02column in the output. So this is really
1:04:04nice in SQL. Okay. So now there is like
1:04:06one more thing about the group I the
1:04:08non-aggregated columns that you are
1:04:10adding in the select must be as well
1:04:12mentioned in the group I. So now for
1:04:14example let's say that okay I'm seeing
1:04:16now the countries the total scores I
1:04:18would like to see as well the first
1:04:20name. So you go over here and say you
1:04:22know what let's get the first name. So
1:04:24country first name the total scores and
1:04:26execute. You will get an error because
1:04:28it's going to tell you I need only the
1:04:30columns that you want to group the data
1:04:32by or should be aggregated. So now the
1:04:35first name it is not aggregated and as
1:04:37well not used for the group I. So it is
1:04:40just here to confuse SQL and it will not
1:04:42work. So if you bring a column either it
1:04:45should be in the aggregation or it
1:04:47should be part of the group I. So in
1:04:49order to fix this and you really want to
1:04:50see the first name you can go over here
1:04:52and say you know what let's add it to
1:04:54the group I and execute. This time it
1:04:57going to work because all the columns
1:04:59that are mentioned here is as well part
1:05:01of the group I. So now as you can see we
1:05:03have the countries the first name and
1:05:05the total scores and you can see again
1:05:07we have five rows we don't have three
1:05:09rows and that's because now you are
1:05:11combining the data by the country and as
1:05:14well the first name and now you can see
1:05:15in the output we are getting five rows
1:05:18we are not getting anymore the three
1:05:19rows the three countries and that's
1:05:21because SQL now grouping the data by two
1:05:24columns the combination of the country
1:05:26and the first name and those two columns
1:05:28gives five combinations and that means
1:05:30you will get five rows so that means you
1:05:32have to be really careful what you are
1:05:34defining in the group I and the number
1:05:36of the unique values that those columns
1:05:38are generating going to define the
1:05:41output the results. So if you go and
1:05:43remove the first name and from here as
1:05:45well you are grouping by only one column
1:05:47and this column has only three values
1:05:49and that's why you are getting three
1:05:51rows and with that of course we have
1:05:52solved the task and now let's extend the
1:05:54task and say find the total score and
1:05:57total number of customers for each
1:05:59country. So that means we need two
1:06:01aggregations. We have the total score
1:06:03and as well we need the total number of
1:06:05customers. So from reading this you can
1:06:07understand we still want to group up the
1:06:09data by the country but this time we
1:06:11need two type of aggregations. We need
1:06:14the total number of customers and the
1:06:16total scores. So we have almost
1:06:17everything but what is missing is the
1:06:19second aggregation. Now what you can do
1:06:21you can go over here and add another
1:06:23aggregate function called the count. And
1:06:26what we want to count is the number of
1:06:28customers. So we can go and add the ID
1:06:30over here and call it total customers.
1:06:33So now of course SQL going to So now if
1:06:34you go and execute it, you will get as
1:06:37well the total customers by the country.
1:06:39And now as you can see SSQL has no
1:06:41problem with the ID and that's because
1:06:42you are aggregating the ID. So SQL know
1:06:45what to do with it and how to combine
1:06:47it. So that means you don't have to
1:06:48mention the ID in the country because
1:06:51you are aggregating it. So that's all
1:06:53with that we have solved as well the
1:06:54task. All right. Right. So with this you
1:06:56have learned how to group up your data
1:06:58using the group eye. Next we're going to
1:07:00talk about another technique on how to
1:07:01filter your data but this time using the
1:07:04having clause. So let's
1:07:09go. All right. So what is exactly
1:07:11having? You can use it in order to
1:07:13filter your data but after the
1:07:15aggregation. So that means we can use
1:07:18the having only after using the group I.
1:07:21So let's see the syntax of that. So
1:07:23again like the previous example we are
1:07:25finding the total score by country. So
1:07:27we have our select from group I and now
1:07:30you say you know what I would like to
1:07:31filter the end results and in order to
1:07:33do that we use the having after the
1:07:36group I and now like the wear clause you
1:07:38have to specify a condition. So we have
1:07:40the following condition where we want to
1:07:42see in the results only the countries if
1:07:45their total score is higher than 800. So
1:07:48this going to be our condition. So now
1:07:50you might noticing something with the
1:07:51group by we are using the country the
1:07:54column where we are grouping the data by
1:07:55its value but with the having we are
1:07:58using the aggregated column the sum of
1:08:01the score. So this is how the syntax
1:08:03works and now let's see how is going to
1:08:05execute it. So as usual SQL start with
1:08:07the from we are getting our data and
1:08:09then the second step is going to go and
1:08:11aggregate the data by the country. So
1:08:13it's like before going to group the rows
1:08:16with the same value of the country. So
1:08:18we're going to have one row for each
1:08:20country and this is what going to happen
1:08:21if you use group I and with that we have
1:08:23now aggregated values right and after
1:08:26the group IQL going to go and execute
1:08:28the having. So having it is like a
1:08:30filter. Now we have a nice condition the
1:08:32total sale must be higher than 800 and
1:08:35SQL going to go and check the new
1:08:36results after the aggregation. So in
1:08:39Germany we have the total sales of 850.
1:08:41So it meets the condition and it going
1:08:43to stay in the results. The same thing
1:08:44for USA it is higher as well than 900s
1:08:47but for UK it is not meeting the
1:08:49condition 750 it is not higher than 800
1:08:53and SQL going to go and filter out this
1:08:55row so that means after applying the
1:08:57having we will get only two countries
1:08:59because they have values that is
1:09:01fulfilling the condition and that's it
1:09:03is what can happen if you are using
1:09:05having it is simply filtering the data
1:09:07but now you might be confused you say
1:09:09you know what we have used the wear
1:09:10clouds to filter the data so why we have
1:09:12in SQL another cloud how to filter my
1:09:15data. Can't we just use the where? Well,
1:09:17in SQL there are like different ways on
1:09:19how to filter your data based on the
1:09:21scenario. So now let's go and add both
1:09:23of the filters in my query. We are
1:09:25already using the having after the group
1:09:27I and now let's go and add the wear.
1:09:29Usually the wear comes between the from
1:09:31and the group I so directly after the
1:09:33from. And here we are saying the score
1:09:35must be higher than 400. So now we are
1:09:37filtering based on the scores twice,
1:09:39right? Once we are saying the score
1:09:41higher than 400 and by having we are
1:09:43saying the sum of score must be higher
1:09:45than 800. So what is the big difference?
1:09:48It is when the filter is happening. If
1:09:50you want to filter the data before the
1:09:52aggregation you want to filter the
1:09:54original data then you can go and use
1:09:57the wear clause. But if you want to
1:09:59filter the data after the aggregations
1:10:01after the group by then you can go and
1:10:03use having. So it's really all about
1:10:05when the filter is happening. So let's
1:10:07see how is still going to execute this.
1:10:09So as usual first the from going to be
1:10:11executed to get the data. Then after
1:10:13that the second step the wear going to
1:10:15be executed. This is our first filter.
1:10:17So SQL going to filter the data using
1:10:19where before doing any aggregations and
1:10:22based on our condition the first
1:10:24customer will be filtered out because
1:10:26score is less than 400 and the same
1:10:28thing for the last customer. Now after
1:10:30the applying the wear clouds we will get
1:10:32only three rows only three customers.
1:10:35And now next SQL going to go and execute
1:10:36the group by. So it's still going to go
1:10:38and group the data by the country. So
1:10:40now we have fewer data to be combined.
1:10:42So the values will not be summarized
1:10:44because we have only one row for each
1:10:46country. Now after the data is
1:10:47aggregated by the group by then SQL
1:10:50going to activate the second filter
1:10:52having. So the next step is going to
1:10:54execute the having and here SQL going to
1:10:56filter the new results based on the
1:10:58total scores and still going to check
1:11:00one by one. So, USA is meeting the
1:11:02condition. UK going to be filtered out
1:11:04because it is not higher than 800. And
1:11:06this time Germany as well will be
1:11:08filtered out because this time it is not
1:11:10fulfilling the condition. In the
1:11:12previous example without the wear, we
1:11:14had more scores for Germany. That's why
1:11:16it passed the test. But this time since
1:11:18we filtered a lot of customers using the
1:11:20wear, Germany will not have enough
1:11:22scores pass the second filter. So with
1:11:24that in the output we will get only one
1:11:26row and that's because we are filtering
1:11:28a lot of data. So it is very simple
1:11:30where going to be executed before the
1:11:32group by before the aggregations having
1:11:34going to be executed after the group by
1:11:36after the aggregations. So now let's go
1:11:38back to scale in order to practice.
1:11:40Okay. So now we have very interesting
1:11:42task find the average score for each
1:11:44country considering only customers with
1:11:46a score not equal to zero. So it sounds
1:11:49like condition and return only those
1:11:51countries with an average score greater
1:11:53than 430. So this is again another
1:11:56condition. So I know there is a lot of
1:11:58things that's going on. Let's do it step
1:12:00by step. Usually I start by doing a very
1:12:02simple select statement with the columns
1:12:04and data that I need. So let's start
1:12:06with a simple select. So what do we need
1:12:08over here? We need a score. We need a
1:12:10country. Again we need a score country.
1:12:13So all what we need is two columns. Now
1:12:15I'm going to go and select the ID just
1:12:17to see the customer ID. Then let's go
1:12:19and get the country score from our table
1:12:22customers. So let's go and query that.
1:12:23So now as you can see I start with the
1:12:25basics. Query the data and then build up
1:12:27on top of it the second step. Now what
1:12:29do we have in the task? We have to find
1:12:30the average score for each country. That
1:12:33means we have to do some aggregations.
1:12:35And here we have two conditions. The
1:12:36first condition says we need only the
1:12:39customers with a score not equal to
1:12:40zero. And the second one we need only
1:12:43the countries with an average score
1:12:44greater than 430. Now you have to decide
1:12:47for each condition whether you're going
1:12:49to use the where or having. Now for the
1:12:52first one we want to filter based on the
1:12:54scores. So that means we want to filter
1:12:56before the aggregations. It's not saying
1:12:59the average score. It's saying the score
1:13:00itself. So that means we can use for
1:13:03this a wear condition. Now about the
1:13:05second one it says countries with an
1:13:07average score greater than 430. That
1:13:09means we want to filter the data after
1:13:12aggregating the score. So that means for
1:13:14this condition we have to use the
1:13:16having. Now what I would like to do is
1:13:18to implement the first condition. It's
1:13:20very simple. We're going to say where
1:13:22after the from the score is not equal to
1:13:25zero. So let's go and execute it. And
1:13:27with that we don't have any customers
1:13:29where the scores is not equal to zero.
1:13:32So that we have solved this part. But
1:13:34now for the second condition first we
1:13:36have to do the aggregations. So we're
1:13:37going to start with the average score.
1:13:40We're going to go over here and say
1:13:41average and we're going to call it
1:13:43average score. Now we don't want to see
1:13:45only the average score. We want to see
1:13:47the average score for each country. So
1:13:49that means we have to aggregate by the
1:13:51country and for that we use the group I
1:13:54group by comes always after the wear
1:13:56clause. So group by and which column?
1:13:58It's going to be the country. So
1:14:00country. Now there is like an issue
1:14:02here. You cannot execute it like this.
1:14:04We have to go and get rid of the ID. We
1:14:06don't need it at all. So let's go and
1:14:08execute it. So with that we have the
1:14:10average score for each country and we
1:14:12have solved the first part. So that
1:14:14means the first and the second part they
1:14:17are completed. Now we're going to talk
1:14:18about the last part. The average score
1:14:20must be higher than 430. And for that
1:14:23we're going to use the having and having
1:14:25comes after the group by. Now we need to
1:14:27specify the condition. It must be the
1:14:29aggregated column. So we're going to
1:14:31take the average score from here and put
1:14:33it after the having and it should be
1:14:35greater than 430. So that's it. With
1:14:38that we have the last part as well.
1:14:40Let's go and execute it now. And with
1:14:42that my friends we have filtered the
1:14:44data after the aggregation. So this is
1:14:47how I decide between the where and
1:14:49having. It is very simple. All right. So
1:14:51with that you have learned how to filter
1:14:53the aggregated data using the having.
1:14:55And now next we're going to go back to
1:14:56the top where we can use there the
1:14:58keyword distinct exactly after the
1:15:00select. So let's go now and learn about
1:15:03the
1:15:06distinct. Okay. So what is exactly
1:15:09distinct? If you use it in SQL, it's
1:15:11going to go and remove duplicates in
1:15:12your data. Duplicates are like repeated
1:15:15values in your data and it's going to
1:15:16make sure that each value appears only
1:15:19once in the results. So it sounds very
1:15:21simple and as well the syntax is easy.
1:15:23So as usual we start always with a
1:15:25select but directly after the select we
1:15:28use the keyword distinct. So there is
1:15:30nothing between them and then the normal
1:15:32stuff we specify the columns and then
1:15:34the from in order to get the data from
1:15:36table. Let's say that I would like to
1:15:37get a list of unique values of the
1:15:39country. So the first thing that SQL
1:15:41going to do of course is to get the data
1:15:42from the database using the from. And
1:15:44now the second step is the select. So
1:15:47SQL going to execute it and going to
1:15:48select only one column the country. All
1:15:51other columns going to be excluded and
1:15:53removed from the results. And now SQL
1:15:55going to go to the third step. It's
1:15:56going to go and apply the distincts on
1:15:58the country values. So it acts like a
1:16:00filter where it going to make sure each
1:16:02value happens only once. So it's going
1:16:04to start with the first value Germany.
1:16:06Now it's going to look to the results.
1:16:07Do we have Germany? Well, we don't have
1:16:09anything yet. So that's why it's going
1:16:11to include it in the results. Then the
1:16:12next value is going to be USA. The same
1:16:14thing. We don't have USA in the results.
1:16:16So it's going to go and include it. And
1:16:18this happens as well for the UK. We
1:16:20don't have UK in the final results.
1:16:22That's why it's going to go as well
1:16:23included. Now comes Germany again. Now
1:16:26it's going to say wait, we have it
1:16:28already. So it will not go and add it
1:16:30again in the output because it must
1:16:32appear only once. So we will not have
1:16:34Germany twice. And as well for the last
1:16:36value the USA we have it already in the
1:16:38results that's why it will not appear
1:16:40again and with that we have removed the
1:16:42duplicates or the repetition inside our
1:16:45data. So each value is unique. Now let's
1:16:47go back to SQL. Okay that task is very
1:16:49simple. It says return unique list of
1:16:52all countries. So let's go and do that.
1:16:54It's going to be funny. So select and
1:16:56now let's get the column country from
1:16:58our table customers like this. Now you
1:17:01can see we have a list of all countries
1:17:02but the task says we need a unique list.
1:17:05So that means I cannot have here
1:17:06repetitions inside it. And with that
1:17:08we're going to use the very nice
1:17:09distinct. So if you do it like this
1:17:13let's go and execute. You will see there
1:17:15will be no duplicates in your results
1:17:16and all the values in the result going
1:17:18to be unique. So with that we have
1:17:19solved the task. It's it's very simple.
1:17:21Now there is like one thing about the
1:17:22distinct that I see a lot of people
1:17:24using it a lot in cases that it's not
1:17:27really necessary. So for example, let's
1:17:29go and get the ID. Now if you go and
1:17:31execute it, you can see here we have a
1:17:33list of all ids and there are no
1:17:35duplicates. But now if I go and remove
1:17:37the distinct and executed, we will get
1:17:39the same results because the ids are
1:17:42usually unique. So it really makes no
1:17:44sense to go and say distinct because as
1:17:46you can see the database has to go and
1:17:48make sure each value happens only once.
1:17:50So there's like extra work for the SQL
1:17:52and it is usually an expensive
1:17:54operation. So if your data is already
1:17:56unique, don't go and apply distincts.
1:17:58Only if you see repetitions and
1:18:00duplicates and you don't want to see
1:18:01that only in this scenario, go and apply
1:18:03the distinct. Don't go blindly for each
1:18:06query applying distinct just in case
1:18:08there is duplicates. This is usually bad
1:18:10practices. Okay. So that's all for
1:18:12distinct. Okay my friends. So with that
1:18:14you have learned how to remove the
1:18:15duplicates using the distinct. In the
1:18:17next step we're going to talk about
1:18:19another keyword that you can use
1:18:20together with the select. You can use
1:18:22top in order to limit your data. So now
1:18:24let's go and understand what this
1:18:29means. Okay. So what is exactly top or
1:18:32in other databases we call it limit. So
1:18:34it is again some kind of filtering in
1:18:36SQL. If you use it, it's going to go and
1:18:38restrict the number of rows returned in
1:18:41the results. So you have a control on
1:18:43how many rows you want to see in the
1:18:45results. The syntax is very simple as
1:18:47well. Directly after the selects you're
1:18:49going to use the keyword top and then
1:18:51you specify the number of rows you want
1:18:53to see in the results. So for example
1:18:55three and then only after that you
1:18:56specify the columns that you want and
1:18:58then from which table. Now let's see how
1:19:00going to execute it. So as usual the
1:19:02from going to be executed we will get
1:19:04our data and then the second step is
1:19:06going to go and select the columns. In
1:19:08this case all the columns going to stay
1:19:09and then after that it's going to
1:19:11execute that top. So how it works? It's
1:19:13very simple. For each row in database,
1:19:16we have a row number. It has nothing to
1:19:18do with your data with the ids. For
1:19:20example, here like in the current
1:19:21result, we have row number 1 2 3 4 5.
1:19:24Those numbers are not your actual data.
1:19:27It is something technical from the
1:19:28database. So it is not equal to the ids.
1:19:31For example, the ids is actually your
1:19:33content your data. So here we are not
1:19:35filtering based on the data based on the
1:19:37row numbers. So since here we have
1:19:39defined three SQL going to count. Okay.
1:19:41row number one 2 three and that's it. So
1:19:44it's going to make a cut and all the
1:19:46rows after number three they will be
1:19:48excluded from the results and you will
1:19:50get only the three rows at the results.
1:19:52So now as you can see this type of
1:19:54filtering is not based on a condition or
1:19:56something it's just based on the row
1:19:58numbers. So whatever results you have in
1:20:00your data it will go and make a cut at
1:20:03specific row. So let's go to scale and
1:20:05practice that. Okay. So now we have a
1:20:07very simple task. It says retrieve only
1:20:09three customers. So let's go and do
1:20:11that. We're going to go and select star
1:20:13from our table customers and execute it.
1:20:16Now as you can see in the output we have
1:20:17five customers. But the task says we
1:20:19want only three. And there is no
1:20:21specifications at all about any
1:20:23condition. So I don't have to go and
1:20:25make a work clause where we write a
1:20:26condition based on our data. We just
1:20:29want three customers. So we can do that
1:20:31very simply by just adding top exactly
1:20:34after the select and then specify the
1:20:36number of rows you want to see from the
1:20:38output. So select top three and then the
1:20:40star. Let's go and execute it. And with
1:20:42that we are getting three customers.
1:20:44That's it. It's very simple. All right.
1:20:46Now moving on to another task. It says
1:20:48retrieve the top three customers with
1:20:51the highest scores. Now of course this
1:20:53is like a mix between ordering the data
1:20:55and filtering the data. Right? So we
1:20:57usually sort the data by the scores from
1:20:59the highest to the lowest. But now it's
1:21:01like we are doing both together. So
1:21:03let's do it again step by step. I will
1:21:05just back to the select star from
1:21:07customers. Now what we can do we can go
1:21:09and sort the data by the score from the
1:21:11highest to the lowest using the order by
1:21:14so order by score and then descending.
1:21:16So let's go and execute it. And now you
1:21:18can see the first customer is with the
1:21:19highest score and then the second
1:21:21highest and so on. Now I think you
1:21:23already got it in order to get the top
1:21:26three customers with the highest scores.
1:21:28What you have to do is to just go over
1:21:30here and say top three and execute it.
1:21:33And with that you have now a really nice
1:21:35analyzis on your data. It's like a
1:21:37reports where we are finding the top
1:21:39customers with the highest score. So
1:21:41this is really amazing and very easy. So
1:21:43as you can see mixing the top with the
1:21:45sorting the data you can make top end
1:21:48analyzes or bottom end analyzers. So
1:21:50let's have this task retrieve the lowest
1:21:52two customers based on the score. So now
1:21:54we want to get the lowest scores in our
1:21:56table. And in order to do that is very
1:21:58simple. What we're going to do we're
1:22:00going to flip that. So we're going to
1:22:01sort our data based on the scores
1:22:02ascending from the lowest to the
1:22:04highest. And since we want only the
1:22:06lowest two customers, we're going to
1:22:07replace the three with a two and execute
1:22:09it. And with that, we're going to get at
1:22:10the lowest two customers. It is Peter
1:22:12and Maria. They have the lowest scores.
1:22:14Again, it's very easy. Okay, this is
1:22:16fun. Let's go to the next one. Get the
1:22:18two most recent orders. Well, this time
1:22:21we are speaking about another table.
1:22:23Let's go and select everything from the
1:22:24table orders like this. So now, as you
1:22:28can see, we have here four orders and we
1:22:30want the two most recent orders. So most
1:22:33recent means we have to deal with the
1:22:35order dates and we can build that by
1:22:37sorting the data by the order dates. So
1:22:40order by order dates and since we are
1:22:43saying the most recent orders so from
1:22:45the highest date to the lowest that
1:22:47means descending right let's go and
1:22:49execute it and as you can see based on
1:22:51our data and now we can look to our
1:22:52result this is the last order in our
1:22:54business based on the order age and this
1:22:57one is one of the earliest orders. So
1:22:59with that we have sorted the data and
1:23:01since we want the two most recent orders
1:23:03we go over here and say we go exactly
1:23:05after the select and say top two and
1:23:08execute and with that we have now the
1:23:10last two orders in our business. So as
1:23:13you can see combining the top with the
1:23:14order by you can do amazing analyszis.
1:23:17All right so this is how you limit your
1:23:19data using top and with that you have
1:23:20learned the basics everything that you
1:23:22can learn and with that you have learned
1:23:23all the clauses the sections that you
1:23:26can use in any query in SQL. Now next
1:23:29what we're going to do we're going to
1:23:30put everything together in one query in
1:23:33order to learn how SQL going to go and
1:23:35deal with all those clauses and how SQL
1:23:37going to go and execute it. So let's go
1:23:39and do
1:23:42that. Okay. So now I'm going to show you
1:23:45the coding order of a query compared to
1:23:48the execution order that happens in the
1:23:50database. So the coding order of a query
1:23:52starts always with a select and then
1:23:54exactly after that you can put a
1:23:56distinct and then after the distinct you
1:23:58can put a top. So this is the order of
1:24:00all those keywords and then you can go
1:24:02and select like few columns and after
1:24:04you specify the columns separated with a
1:24:06comma you tell SQL from which table your
1:24:08data come from using the from clause.
1:24:11Now after that if you want to filter the
1:24:13data before the aggregation you can use
1:24:14the where clause and this always comes
1:24:17directly after the from. And if you want
1:24:19to group the data then you have to do it
1:24:21after the wear clause using the group by
1:24:23and after the group buys comes the
1:24:25having if you want to filter the data.
1:24:27And the last thing that you can specify
1:24:29in query it is always the order by. So
1:24:32this is the order of all those
1:24:33components of the query. And if you
1:24:35don't follow this order you will get an
1:24:37error from the database. Now if you look
1:24:39to this query there are a lot of things
1:24:41that's going to filter your data. So
1:24:43let's check them one by one. The first
1:24:44thing that you can do is to filter the
1:24:46columns. If you don't want to see all
1:24:47the columns, you want to see only
1:24:49specific columns, you use the select and
1:24:51of course you must use it. So the
1:24:53columns that you specify will be shown
1:24:55in the results. So it's like filtering
1:24:56the columns. Now there is another type
1:24:58of filter where you filter out the
1:25:00duplicates if you want to see unique
1:25:02results and that's using the distinct.
1:25:04So this is another type of filter.
1:25:06Moving on, we can filter the result
1:25:08based on the row numbers. So we can
1:25:10limit the result using the top. But this
1:25:12type of filter doesn't need any
1:25:14conditions. It's purely based on the row
1:25:17number in the results. Now moving on, if
1:25:19you want to filter your data based on
1:25:20conditions based on your data, you can
1:25:22filter the rows before the aggregation
1:25:25using the wear clause. And the last type
1:25:28of filtering, you can filter your rows
1:25:30after the aggregation using the having.
1:25:32So as you can see, we have like five
1:25:34different types and how to filter the
1:25:36results in SQL. So now let's see the
1:25:38execution order. As we learned the first
1:25:41thing that's going to happen is that SQL
1:25:43going to execute the from clause. So SQL
1:25:45going to go and find your data in the
1:25:47database where all the next steps going
1:25:49to be paste on this data. Now the next
1:25:51step that is going to do is that it's
1:25:53going to go and filter the data using
1:25:55the wear clause. This has to be happen
1:25:57before anything else. So before any
1:25:59aggregations and so on we have to make
1:26:01scope of the data. So once SQL apply it
1:26:03maybe some of the rows going to be
1:26:05removed and once the data is filtered
1:26:07the third step SQL going to execute the
1:26:10group I so going to take the results and
1:26:12start combining the similar values in
1:26:15one row and start aggregating the data
1:26:17based on the aggregate function that you
1:26:18have specified. So now after the group
1:26:20by after aggregating the data what is
1:26:23going to do now it's going to go and
1:26:24apply the second type of filter the
1:26:26having. So based on the condition the
1:26:27SQL going to go and start removing few
1:26:30aggregated data away and keep the rest.
1:26:32Now moving on to the step number five.
1:26:34Finally it's going to go and execute the
1:26:37select distinct. So SQL going to go and
1:26:39start selecting the columns that we need
1:26:41to see in the results and remove the
1:26:42other stuff. And once the columns are
1:26:44selected SQL going to go and execute the
1:26:47order by. So SQL going to start sorting
1:26:49the data based on the column that you
1:26:51have specified and the mechanism as
1:26:53well. So the data will be sorted
1:26:55differently. And my friends the last
1:26:57step that going to happen in your query
1:26:59will be always the top statements. So
1:27:02based on the final final results SQL
1:27:04going to go and execute the top. So here
1:27:06we are saying top two that means we want
1:27:08to keep only the first two rows without
1:27:10any conditions. So SQL going to count
1:27:12okay row number one two and after that
1:27:14it's going to make cuts and remove
1:27:16anything after that. So this is the last
1:27:19filter that's going to happen and as
1:27:20well the last step. So now if you sit
1:27:22back and look at this the coding order
1:27:24is completely different than the
1:27:26execution order in the coding we have
1:27:27first to specify the select actually the
1:27:30select going to be executed just almost
1:27:32at the end. So at the step number five
1:27:34and once you understand how SQL execute
1:27:36your query you can understand how to
1:27:39build correct
1:27:43queries. So now the first thing that we
1:27:45have learned that we can go and have
1:27:47like one query right something like this
1:27:50select star from customers. Now this is
1:27:52one query and in the output we have one
1:27:54results but did you know that in SQL we
1:27:57can have like multiple queries and
1:27:59multiple results in one go. So we can do
1:28:02everything together like for example
1:28:03let's say I'm selecting as well the data
1:28:05from orders. So that means we have two
1:28:08queries and now if you go and execute
1:28:11what can happens you will get two result
1:28:13grids. The first result grid is for the
1:28:16first query and the second one is for
1:28:18the second query. So with that you can
1:28:20do multiple queries in the same window
1:28:23and with that the results can be
1:28:25splitted into multiple window depend how
1:28:27many queries you have and usually in SQL
1:28:29you might find that by the end of each
1:28:31query there is a semicolon like this. So
1:28:34at the end of the first query we have
1:28:35semicolon and for the second query we
1:28:37have as well at the end another
1:28:39semicolon. For the SQL server it is not
1:28:41a must but for other databases if you
1:28:43have multiple queries in one execution
1:28:46you must separate them with a semicolon
1:28:48and with that the database can
1:28:49understand okay this is the end of the
1:28:52first query and this is the end of the
1:28:54second query. So you have like
1:28:56separations between
1:29:01queries. Okay. Now moving on to another
1:29:03cool thing in SQL. Now what if we don't
1:29:06want to query the data inside our
1:29:08tables, we would like to show a static
1:29:10value from us from the one that is
1:29:12writing the query. And this is very
1:29:13practical. If you are like practicing
1:29:15and you want to check something using a
1:29:17value from you, not from the tables. So
1:29:20how we can do that? It is very simple.
1:29:22We're going to write select and then now
1:29:24after that instead of having a column
1:29:26name you can go and add any value like 1
1:29:292 3. So it is just a number and we do
1:29:32not specify after that any table. So we
1:29:34leave it like this. Select 1 2 3 and we
1:29:36don't need to use the from close. So now
1:29:39if you go and execute it you will get 1
1:29:412 3. So this is a static value. And of
1:29:43course you can go and rename the column
1:29:45like static number. So execute it again.
1:29:49So with that we have a static value. And
1:29:51you can go and add anything like string
1:29:53as well. So let's say hello as static
1:29:56for example string. So let's go and
1:29:58execute. Now we have two queries. The
1:30:00second one you can see our static value.
1:30:02Hello. So in queries we can add values
1:30:04from us. Not only selecting data from
1:30:07the queries but of course you can go and
1:30:08mix stuff. So we can have like in one
1:30:10query data from the database and static
1:30:13data from us. So let me show you what I
1:30:15mean. Let's go over here and say select
1:30:17and let's go and get for example the ID
1:30:20the first name from the table customers
1:30:23like this. So with that we can see we
1:30:24are getting data from the database. But
1:30:27now I can go and add something from me
1:30:29new customer and we can call it customer
1:30:32type. So now what is going on here? Two
1:30:34columns from the database and one column
1:30:37from us. It is the static one. So if you
1:30:39go and execute it, you can see for the
1:30:41ID and the first name those data comes
1:30:43from the database. But for each record
1:30:45we are always getting the same static
1:30:47value new customer, new customer and so
1:30:48on. So this piece of information comes
1:30:51from the query. It is not stored inside
1:30:53the database and those two informations
1:30:55come from the stored data inside the
1:30:57database. So this is really cool thing.
1:30:59You can add few informations from you
1:31:02and you can get the data from the
1:31:03database. This is the static
1:31:08values. Okay. One more cool thing that I
1:31:11want to show you that if you have a
1:31:12query like this you are selecting from
1:31:14table and filtering the data and now you
1:31:16would like not to execute the whole
1:31:17thing. You would like to execute only a
1:31:19part of this query. So now sometimes as
1:31:21you are writing a query, you don't want
1:31:23to execute the whole thing. You want to
1:31:25execute only part of the query. Like for
1:31:27example, I would like to see all the
1:31:28customers again in this query without
1:31:30this filter. So instead of removing it
1:31:33and then query and then again adding it,
1:31:35what you can do, you can highlight what
1:31:37you want without now the filter and
1:31:39execute. So without the database going
1:31:41to execute exactly what you highlighted.
1:31:43And now as you can see I'm getting all
1:31:44the customers without the filter. And if
1:31:46you don't highlight anything and
1:31:48execute, what's going to happen? It's
1:31:49still going to execute the whole thing
1:31:51inside the editor. And this is really
1:31:53nice if you want to query another table
1:31:55quickly in the same editor. Like we want
1:31:56to select everything from the orders
1:31:58just quickly. So you can highlight only
1:32:00this query and execute. And with that
1:32:02SQL is ignoring everything else and only
1:32:05executing what I'm highlighting. And
1:32:07this is really nice. It gives us like
1:32:08speed and dynamic. And you're going to
1:32:10find me doing that a lot in the course.
1:32:12So this is really nice. Okay. My
1:32:14friends. So with that we have learned
1:32:15the basics about SQL query. the basic
1:32:18components of the select statements and
1:32:20with that you can talk to our database
1:32:22in order to get data. Now in the next
1:32:24chapter we're going to learn how to
1:32:25define the structure of our database. So
1:32:27we're going to learn the data definition
1:32:29language DDL. So let's
1:32:34go. Okay. So usually if you have like an
1:32:37empty database what you want to do is to
1:32:39go and define the structure of your
1:32:41data. So one of the first things that we
1:32:43usually do is we go and create a new
1:32:45tables. So here we have a command called
1:32:47create and if you use it you can create
1:32:49a new object inside the database like
1:32:52for example a table. So once you execute
1:32:54it you're going to get brand new table
1:32:56and usually the table going to be empty
1:32:57without any data. So it is very simple.
1:33:00This is what the create command does.
1:33:02And now let's go to SQL in order to
1:33:04create a new table. So my friends we
1:33:06have the following task. Create a new
1:33:07table called persons with columns ID
1:33:11person name birth date and phone. Okay.
1:33:13So this time we will not start by select
1:33:16we will start with the command create
1:33:18table. So we are telling SQL to create a
1:33:20table and after that we have to define
1:33:22the name of the table. So in this task
1:33:24we have to call it persons. Now we have
1:33:26to go and open two parenthesis like this
1:33:28and in between we have to define the
1:33:31columns. So what do we need? First we
1:33:33need an ID. So this is the first column
1:33:35name. And next we have to define which
1:33:37data type for this column. It's going to
1:33:38be an int. So it is a number does not
1:33:41contain any characters. And now next we
1:33:43can define some constraints and we
1:33:44cannot have a person without an ID. So
1:33:46it should not be in null. So not null.
1:33:48This is the first column. So we have
1:33:50defined the name of the column, the data
1:33:52type and the constraint. Okay. So let's
1:33:54go to the second column and here we're
1:33:55going to have a comma and the next one
1:33:57name going to be person name. So this is
1:34:00the column name and the person name we
1:34:02can have. And now the data type for this
1:34:04column it going to be a varchar because
1:34:06the person name contains characters. So
1:34:08vchar. And now we have to define the
1:34:10length. So I'm going to go with 50
1:34:12characters. And now I would say this is
1:34:14a must. So each person should has a
1:34:16name. So we're going to say not null as
1:34:18well. So that we have the name, the type
1:34:21and the constraint. Now let's move to
1:34:22the third column. It's going to be birth
1:34:25date. Now which type of informations we
1:34:27have inside the birth date? So it's
1:34:29going to be a date, not a number, not
1:34:31characters. So we're going to go with
1:34:32the data date. And now about the
1:34:34constraint well depends. I would say in
1:34:37our application it is an optional
1:34:39because this is very personal
1:34:40information and maybe some persons will
1:34:42not provide their birth dates. So this
1:34:44is an optional and I will not say it is
1:34:46not null. So nulls are allowed. Now
1:34:48let's move on to the next one. It's
1:34:49going to be the phone. So now what is
1:34:51the data type of a phone? Well we have
1:34:53some types numbers we have characters
1:34:56special characters. So we could have
1:34:57anything. So that's why I'm going to go
1:34:59with the farchar. And here you can
1:35:00specify the length that you think it's
1:35:02okay. I'm going to go with 15. Now of
1:35:04course depend on the system that you are
1:35:06building. I would say the phones are
1:35:07very important in order to validate
1:35:09whether this is a real person. So we're
1:35:12going to say not null. So we are not
1:35:15allowing nulls in this field. Perfect.
1:35:17So with that we have covered all the
1:35:18columns that are required. We have
1:35:20defined the data types and as well the
1:35:22constraints. Now the last thing in each
1:35:24database table we should has a primary
1:35:27key in order to make sure this table has
1:35:29an integrity and maybe as well
1:35:31connectable to other tables. So now what
1:35:33we're going to do, we're going to go and
1:35:34add the primary key constraint, comma,
1:35:36for the last column. And then we're
1:35:38going to say constraint. Now we have to
1:35:40give a primary key name. This is only
1:35:42going to be visible for the database. So
1:35:44I'm going to call it PK for primary key.
1:35:46And here persons and then after that
1:35:48we're going to say primary key. And
1:35:51between two parentheses, we're going to
1:35:53go and pick which one is the primary
1:35:54key. And of course, it's going to be the
1:35:56ID. So we're going to go over here and
1:35:58say ID. So again, we are saying there is
1:36:00a new constraint. This is the name of
1:36:02it. It's only internal for the database.
1:36:04And then we are saying this one is a
1:36:06primary key on the field ID. So that's
1:36:08it with that. We have defined a primary
1:36:10key for our table. Let's go and execute
1:36:13it. So as you can see it is successful.
1:36:14Let's go and check our database for our
1:36:16new table. So if you don't see it
1:36:18already, you have to right click on the
1:36:20database and then go and refresh. So
1:36:22let's go to tables and now we have a
1:36:24brand new table called persons. So with
1:36:26that we have created our new table. Now
1:36:28of course for the DDL commands you will
1:36:30not get results or data. All what you're
1:36:33getting is a message from the database
1:36:35and the message says here the command
1:36:37completed successfully and then we have
1:36:39a date when this is completed. So that
1:36:41means the DDL command will never return
1:36:43data. It is changing the structure of
1:36:45your database. It's not about retrieving
1:36:48any data and so on. So this command did
1:36:50change something in our database and in
1:36:52this scenario it created a new table and
1:36:54that's why we call this data definition
1:36:57language DDL because we are defining the
1:37:00database. Now of course if you go and
1:37:01say select star from our new table
1:37:04persons. So let's go highlight it and
1:37:07then execute it. You will see we are
1:37:09getting of course the columns. So the
1:37:11ID, the person name, birth date, the
1:37:13phone but we don't have any rows that
1:37:16means our table is empty. Now what is
1:37:18very important to that you go and save
1:37:20those informations in an SQL script
1:37:23because maybe later you have to redefine
1:37:25this table but let's say that you have
1:37:27created different queries and you have
1:37:28lost the script and now I would like to
1:37:30see again the create statements for this
1:37:33table well there is trick for that if
1:37:35you go to the left side you see the
1:37:37persons right here right click on it and
1:37:39then you have here script table as and
1:37:41now we have here different options that
1:37:43you can run on the table and the first
1:37:45one says create two Then let's go to new
1:37:48query
1:37:49editor. So now what happened? The
1:37:51database did read the metadata
1:37:53informations about the person and
1:37:55created your DDL query with many extra
1:37:58stuff that we haven't done. But this is
1:37:59the template that the database uses. So
1:38:01now we can see a lot of stuff. But what
1:38:03is interesting is this create table. So
1:38:06we can see create table the schema DBU
1:38:09the default one then the persons and
1:38:11then we have our columns the data type
1:38:13and as well the constraints. So with
1:38:15that you got back your DDL statements
1:38:17and many other stuff about the table
1:38:19which is now not interesting. But now
1:38:21what I really need is to see the create
1:38:23statements about this table. So this is
1:38:25how you can get back your DL command.
1:38:27But of course what I recommend you is
1:38:29always put your code inside a get
1:38:31repository and always keep it up to
1:38:33date. So that always you can check your
1:38:35work and extend
1:38:39it. Okay. So now what else you can do
1:38:41with the structure of your database? If
1:38:43you have already a table, what you can
1:38:45do, you can go and edit and change the
1:38:48definition of the table. So for example,
1:38:50let's say I would like to add a new
1:38:51column. In order to do that, we can use
1:38:54the command alter. Alter means you want
1:38:56to edit the definition of your table and
1:38:59you want to change it like adding new
1:39:01column or maybe changing the data type
1:39:03and anything in the definition of the
1:39:05table. So the alter command, you can use
1:39:07it in order to change the definition of
1:39:08your table. And now let's go back to
1:39:10scale and try to change something. All
1:39:12right. Now the task says add a new
1:39:14column called email to the person's
1:39:16table. So it is very simple what you can
1:39:18do. We can use the alter table command.
1:39:20So we are not creating new table. We
1:39:23want to edit already existing table. So
1:39:25which table we want to modify it's going
1:39:27to be the persons. So we are telling SQL
1:39:29we want to change something in the table
1:39:31persons. And of course we have to tell
1:39:32SQL what we want to change. Are we
1:39:34removing a column? Are we adding column?
1:39:36In this scenario we want to add new
1:39:38column. So let's go and add the email
1:39:41information. So this is the column name
1:39:43and as you are creating a table you have
1:39:45to define column name the data type and
1:39:47the constraint. So now for the emails
1:39:49we're going to have like characters,
1:39:50numbers, special characters. So we're
1:39:52going to go with the varchar and about
1:39:54the length it's going to be let's say 50
1:39:56and I'm going to say each person has to
1:39:58has an email. So it's going to be not
1:40:00null. So with that we are adding
1:40:02completely a new column. So that's it.
1:40:04Let's go and execute it. Now again this
1:40:07is not a query. This is a DDL command
1:40:09and in the output we will not get data.
1:40:11We will get a message whether everything
1:40:13went correctly. So it says command
1:40:15completed successfully and the time when
1:40:17this is completed. Now we can go and do
1:40:19a simple query just to have a check to
1:40:21the table. So and now you can see we
1:40:23have our columns and at the end we have
1:40:25a new column called emails. This is very
1:40:27important. If you are adding new column
1:40:29it's going to be always at the end of
1:40:31the table. But now you might say you
1:40:33know what I would like to have the email
1:40:34like something in the middle maybe after
1:40:36the person name. Well, in order to do
1:40:38that, you have completely to delete and
1:40:40drop the table and create it from the
1:40:42scratch using create command which is
1:40:45might be bad if you have data inside the
1:40:47table. So if you are fine by adding your
1:40:49new column at the end, you can use the
1:40:50alter table. But if you say I would like
1:40:52it in the middle, then sadly you have to
1:40:54go and drop everything and start from
1:40:56the scratch. Okay. So now let's have
1:40:58another task and it says remove the
1:40:59column phone from the person's table. So
1:41:02now we're going to do exactly the
1:41:03opposite. We're going to go remove it
1:41:04completely with its data from the table.
1:41:07So we're going to still saying alter
1:41:09table persons. We are saying we want to
1:41:11edit the definition of the table
1:41:12persons. And now instead of adding we
1:41:15will be dropping a column. And then
1:41:18after that we have to specify as well
1:41:19the column name. It's going to be the
1:41:21phone. But we don't have to mention
1:41:23again the data type and the constraint.
1:41:26And that's because the database already
1:41:27knows those informations. So we need
1:41:29those informations if we are creating
1:41:31something new. That's why we can get rid
1:41:33of that. We just need the column name
1:41:35and the database is going to do the
1:41:36rest. So let's go and do that. Now you
1:41:39can see successful. And now let's go and
1:41:41check our table. And now as you can see
1:41:43we have the ID, person name, birth date,
1:41:45email, and we don't have the column
1:41:47phone. Be careful. If you are deleting
1:41:48column, you will be losing as well all
1:41:50the data inside this column. So as you
1:41:52can see, this is very simple. This is
1:41:54how we can edit the definition of our
1:41:56table by adding and removing columns.
1:42:03Okay, now moving on to the last one in
1:42:05this group of commands. So now so far
1:42:07what we have done, we have created
1:42:08something new in the database. We have
1:42:10changed the definition of something
1:42:12inside our database. And now the last
1:42:14one, you can go and drop something from
1:42:17the database. Let's say we have another
1:42:18table and we don't need it anymore. So
1:42:21we can go and use the drop command in
1:42:23order to remove the table completely
1:42:26from the database. And this means as
1:42:28well removing everything the table and
1:42:30the data inside it. So now let's go to
1:42:32SQL and let's drop something from our
1:42:34database. Okay. So now our task says
1:42:36delete the table persons from the
1:42:38database. This is the simplest form of
1:42:41command in SQL but yet the most risky
1:42:43one. So what we need? We have to delete
1:42:46and drop the whole table persons. We
1:42:48don't need it anymore. We're going to
1:42:49say drop table and then all what we have
1:42:52to do is to give the name of the table
1:42:54persons. So three words. You don't have
1:42:57to specify anything. Just destroy the
1:42:59table persons. Let's go and execute it.
1:43:01It is successful. So as you can see it
1:43:03is very simple. Now on the left side to
1:43:05your database go refresh and go to the
1:43:07tables and you will not see the table
1:43:09persons. So the drop command it is very
1:43:11simple but yet very risky. So if you
1:43:14compare now create table with a drop
1:43:16table you can see destroying things is
1:43:18way easier than building it. Those are
1:43:20the commands create alter drop. those
1:43:23commands we use in order to define the
1:43:26structure of our database the DDL
1:43:28commands that was very simple all right
1:43:30so that's all about the data definition
1:43:32language DDL and with that you have
1:43:34learned how to define new stuff in your
1:43:36database now moving on to the next one
1:43:38we're going to learn about the data
1:43:39manipulation language and here we're
1:43:41going to learn how to manipulate our
1:43:42data inside the database let's
1:43:48go all right so now what we're going to
1:43:50do we're going to go and modify and
1:43:51manipulate your data inside the
1:43:54database. So now sometimes what happens
1:43:56you have a table inside your database
1:43:58and the table is empty. You don't have
1:44:00any rows any data inside the table. Now
1:44:02in order to add your data to the table
1:44:04what you can do you can use the command
1:44:06insert. So insert going to go and add
1:44:09new rows to your table and of course not
1:44:12always the table must be empty to add
1:44:13your data. You can add new rows to
1:44:16already existing data and SQL going to
1:44:18go and append it at the end of the
1:44:20table. Now my friends in order to insert
1:44:22new data to the target table there are
1:44:24two methods. The first and the classical
1:44:26way in order to insert new data we can
1:44:29use the insert command and manually
1:44:31specifying the values that should be
1:44:33inserted to the table. So you're going
1:44:35to start specifying in the script the
1:44:37values and then they're going to be
1:44:38inserted as a new rows to the target
1:44:40table. So in this process you are
1:44:42manually inserting new values to the
1:44:44table using like an SQL scripts. So now
1:44:47we're going to focus on this scenario on
1:44:49how to insert data. All right. Now let's
1:44:50check quickly the syntax of the insert
1:44:52command. It start with the keyword
1:44:54insert into and after that we have to
1:44:56specify the table name. So where we want
1:44:58to insert and then we make a list of all
1:45:01columns that we want to insert. And then
1:45:03we specify list of columns where we're
1:45:05going to insert values into them. And
1:45:07after that we say values. And finally
1:45:09we're going to go now and specify the
1:45:11data that should be inserted to the
1:45:13table. and we make it as well as a list
1:45:15like we have done for the columns. Now
1:45:17in the insert statements specifying
1:45:19those columns it is totally optional. So
1:45:22if you don't specify the columns of the
1:45:23table then SQL going to expect you to
1:45:26insert values into each column because
1:45:29sometimes of course we don't want to
1:45:30insert value for each column. You can
1:45:32skip few columns of course but if you
1:45:34want to insert a value for each column
1:45:37either you go and specify them as a list
1:45:39or you can skip it. Now for the insert
1:45:41statements there is very important rule.
1:45:43The number of columns and values must
1:45:46match. So if you specify here three
1:45:48columns then you must insert as well
1:45:50exactly three values. So this must be
1:45:53matching. And one last thing about the
1:45:55syntax you can insert multiple values in
1:45:57one go. So for each row you can specify
1:46:00a list of values that must be inserted.
1:46:02So that's all about the syntax. Let's go
1:46:04back to SQL in order to practice insert
1:46:07command. Okay. So now let's go and
1:46:08insert a new customers. So it's very
1:46:10simple. It start with insert into. So we
1:46:13are saying we want to insert data into.
1:46:15So we have to go and specify the table
1:46:17name customers. Now after that we have
1:46:19to specify list of columns where we want
1:46:22to insert data into it. And what we can
1:46:24do we can go and check which columns do
1:46:25we have inside our table. So we can see
1:46:27we have ID, first name, country, score.
1:46:29And we can go and make a list of that.
1:46:31So we can say ID, first name, country
1:46:35and score. So we just have a list of all
1:46:38columns inside our table customers. Now
1:46:41what we need? We need the values. So
1:46:43which data should be inserted. So we can
1:46:45go and open two parenthesis. And now we
1:46:47have to specify an ID. We know the last
1:46:49customer was five. So we're going to go
1:46:51with the customer six. Now we have to
1:46:53give the name of the customer. Let's go
1:46:55for Anna. And then a country. Let's go
1:46:57for USA. And this customer has no
1:47:00scores. So what we can do? We can say
1:47:02null. So we don't know the score of this
1:47:05customer. nulls means nothing we don't
1:47:07know. So with that you can go and insert
1:47:09one row. But now let's say that I would
1:47:11like to go and insert like a second row
1:47:13one more customer. What we can do we can
1:47:15separate this with a comma and then we
1:47:18can go and repeat the whole thing again.
1:47:20So the ID is seven. The next one let's
1:47:22call this customer Sam and we don't know
1:47:25the country of this customer. So we're
1:47:27going to say it's null. But the score we
1:47:28know it already. It is 100. So as you
1:47:30can see we are adding a value for each
1:47:32of those columns. And if you don't know
1:47:34the answer then make it null. if the
1:47:36database allows it to be null. Some
1:47:38columns they are not allowed to be null
1:47:40like the primary key. So if you go and
1:47:41say over here null the database will not
1:47:43allow it. Well actually we can go and
1:47:45test it. Let's execute. And you can see
1:47:47you cannot insert the value null into
1:47:50the column ID. So this is not allowed.
1:47:52Going to have a seven. But for the other
1:47:53columns it is allowed. You can go and
1:47:55check the definition of the table. Now
1:47:57we go and execute. Now the output of the
1:48:00modifications command is going to always
1:48:02indicate what happens to the data. So it
1:48:04says two rows affected. Affected might
1:48:07be inserted, updated, deleted. So you're
1:48:09going to get a general statement from
1:48:11the database. But you are getting how
1:48:13many record is affected. So we got two
1:48:15because we have inserted two records. So
1:48:17now as you can see it's not like the
1:48:18query. We are not getting any data in
1:48:21the output. We are just getting a
1:48:22message. So this is a big difference
1:48:24between querying the data using the
1:48:26selects and modifying the data using
1:48:28inserts. We are doing now direct
1:48:30modifications to the data inside our
1:48:32database. Of course, if you want to see
1:48:34the data in the customers, what we can
1:48:36do, we can go and query the data, right?
1:48:39So, let's go and do that. Select star
1:48:40from customers. I would like to see the
1:48:42whole table. So, market and execute it.
1:48:45Now, you can see we have seven
1:48:47customers. So, we just manipulated our
1:48:49data. We have here Anna and Sam. This is
1:48:51how you can insert data to the database.
1:48:53Now, there's like few rules you have to
1:48:55be careful as you are inserting new data
1:48:57to your tables. You have to pay
1:48:58attention that the order of the columns
1:49:00that you have defined. insert is
1:49:03matching the values that you are
1:49:04inserting over here. Let's have an
1:49:06example. I'm going to go and remove this
1:49:08over here and let's say that we are
1:49:10inserting a new one number eight and now
1:49:12in the first name instead of the name of
1:49:14the customers we have inserted the
1:49:16country like USA and in the country we
1:49:19have inserted the name is just mistake
1:49:21and we are all human right? So let's
1:49:23have a name like this max. Now if you go
1:49:24and execute it the database can accept
1:49:27it because it is really hard for the
1:49:28database to understand that you have
1:49:30made here an error. Both of them are var
1:49:32and the database doesn't care about the
1:49:34content of the data as long as you are
1:49:36following the rules of the data type. So
1:49:37now if you go and select the data from
1:49:39the customers you can see now we have a
1:49:41customer called USA from the country
1:49:44max. So the SQL going to do it blindly
1:49:46like you insert the data as long as you
1:49:48are following the data type rules and
1:49:50the constraints. So for example, if you
1:49:52made this error over here and you say
1:49:54the id is max and let's say the first
1:49:57name is let's say nine and you execute
1:50:00it here the database is smart enough to
1:50:02say you know what there is something
1:50:04wrong the ID should not be strange so
1:50:06the database going to reject your
1:50:07inserts be careful of the order of your
1:50:09columns now let's go and query again our
1:50:11table now if you are in the insert
1:50:13commands defining all the columns
1:50:15exactly like the table so as you can see
1:50:17we have here complete match ID first
1:50:19name country score we have all the
1:50:21columns and as well the correct order
1:50:22there is like lazy way you can go and
1:50:24remove the whole thing over here and
1:50:26with that the database can understand
1:50:28okay we are inserting values to all of
1:50:30the columns so going to understand you
1:50:32are inserting something to each columns
1:50:34in the correct direction so let's go and
1:50:36do that correctly nine and here let's
1:50:39say we
1:50:41have from Germany so if you go and
1:50:44execute it it will be working even
1:50:46though we didn't define the columns and
1:50:48that's because the values that we are
1:50:50inserting as exactly the same number of
1:50:52columns of the table and following as
1:50:54well the rules. Now moving on to the
1:50:56next one, you can go and add only two
1:50:58columns in the definition. If you know
1:51:00already always the country and the score
1:51:02is null. We know only two informations,
1:51:05the ID and the name. Then you don't have
1:51:07always to go and say null null null and
1:51:08so on. We can go and skip that. Okay. So
1:51:11now let me show you what I mean. We're
1:51:12going to go after the table name and
1:51:14we're going to define only two columns,
1:51:15the ID and the first name. So that means
1:51:18we are telling SQL we want to insert
1:51:20only two columns. And now you have to be
1:51:22careful. If you define here two columns
1:51:24then the values should be as well two
1:51:26columns. So we're going to remove the
1:51:28country and the score. And we can go and
1:51:30add only two informations. So 10. And we
1:51:33can go and add here for example Sara. So
1:51:35if you go and execute it, it will be
1:51:37working. And now what is skill is doing
1:51:39with the other two columns. It's going
1:51:40to be nulls. So let's go and select
1:51:42again from our table. You can see here
1:51:44Sara has null in the country and as well
1:51:47in the score because we didn't define
1:51:48those informations. But be careful, you
1:51:50cannot here skip a column that is not
1:51:52allowed to be null. So you have always
1:51:54have in your list all the columns that
1:51:57are not null. So for example, I cannot
1:51:58go and insert only the first name. I
1:52:00will get an error because the database
1:52:02can try to insert a null in the ID and
1:52:04this is not allowed. So you can skip
1:52:06only nullable
1:52:11columns. All right, my friends. So that
1:52:13was the first method on how to insert
1:52:15data to your target table as you saw by
1:52:18typing manually the values inside an
1:52:20insert command using values. And now
1:52:23let's move to another methods. We're
1:52:25going to insert data but this time not
1:52:27manually. We're going to insert data
1:52:29using another table. So imagine we have
1:52:31the following scenario. We have an
1:52:32already existing table with data and
1:52:35this going to be the source table, the
1:52:36source of your data and we have another
1:52:38table. This table is empty and we want
1:52:41to insert a new data to this target
1:52:43table. Now what we can do, we can take
1:52:45the data from the source table and
1:52:47insert it into the target table without
1:52:49manually writing the script for the
1:52:52values. So we are moving the data from
1:52:54one table to another. Now in order to do
1:52:56that we need to do two steps. The first
1:52:58step we have to write an SQL query using
1:53:01select from and so on in order to select
1:53:04the data that we need from the source
1:53:06table. And once you do that you will get
1:53:09a results. So this is like you are doing
1:53:11a normal query. You right select and you
1:53:13will get an answer with the results. And
1:53:15now what we can do in the next step we
1:53:17can take this results and use an insert
1:53:20command in order to insert this results
1:53:23into the target table. And with that we
1:53:26have moved the data from the source
1:53:27table to the target table. So first
1:53:29write the query on the source table. And
1:53:32the second step use an insert to move
1:53:34this results to the target table. So
1:53:36let's go back to the scale in order to
1:53:37do that. So now we have the following
1:53:39task and it says insert data from the
1:53:41table customers into the table persons.
1:53:43So that means the source table is the
1:53:46customers and the target table is
1:53:48persons. Now how I usually do it that I
1:53:51keep my eye on the target table to
1:53:53understand the structure of this table
1:53:55and I start writing the query from the
1:53:58source table. If you go to the left
1:54:00side, we can see okay, we have here an
1:54:01ID. We have here person name, birth date
1:54:04and phone. And you can see only the
1:54:05birth date except nulls and the rest we
1:54:08have always to provide informations. So
1:54:10with that I have now understanding about
1:54:12the table persons. Now next I'm going to
1:54:14go and start writing the query from the
1:54:16source. So we start like this. Select
1:54:18star from our table customers just to
1:54:21have an overview of our table. Now the
1:54:23next step we're going to go and design a
1:54:25perfect result from this query that is
1:54:28matching the target table. So in the
1:54:30output we need ID and we have it from
1:54:32the customer from the original table.
1:54:34We're going to go and select ID. Okay.
1:54:36So now next we need a person name and
1:54:39here we have from the original table
1:54:40something called first name. So this is
1:54:42a perfect match. So we're going to go
1:54:44and select this table as a second
1:54:46column. So we have covered the first
1:54:48two. Then the third one is going to be
1:54:49the birth date. Well, my friends, we
1:54:51don't have birth dates, but the database
1:54:53can accept it as a null. So, I'm going
1:54:55to go and write a null because I don't
1:54:57have such information from the source
1:54:59table. And now the next one going to be
1:55:00the phone as well. We don't have phone
1:55:02informations. But we cannot have it as a
1:55:04null because it says here not null. So,
1:55:06what we're going to do, we're going to
1:55:07go and add a static value, a default
1:55:10value. So, we're going to have two
1:55:12single quotes and in between we're going
1:55:13to say unknown. Since it is var, it can
1:55:16accept this word. So, now let's go and
1:55:19just query. So we have the ID, we have
1:55:21the first name, the birth date is empty,
1:55:24and the phones is unknown. Now you might
1:55:26say, but the column name is not matching
1:55:29with the column name of the persons.
1:55:31Well, the database does not care about
1:55:32that. As long as the result of the data
1:55:35is matching the table, it can go and
1:55:37insert it. So the database will never
1:55:40compare the column names together. But
1:55:41if you like and go and add here like the
1:55:43aliases exactly like the target table it
1:55:46will not hurt but it has no effect on
1:55:49the results. All right. Okay. So now we
1:55:50have like query select and we have a
1:55:52results but this is not an insert. So
1:55:55how we going to insert the result of
1:55:57this into the table persons. Well for
1:55:59that we need the insert into command. So
1:56:03insert into and now we have to specify
1:56:05the target table going to be the
1:56:07persons. And of course you can go and
1:56:09list all the column names but if you
1:56:11have like exact match you can skip it
1:56:13but for me I would like always to add it
1:56:15just to make sure that we don't have any
1:56:18issue. So the ID, person name, birth
1:56:20date and the
1:56:22phone. So that's it. Let's go and
1:56:26execute. So it is working now. We can
1:56:28see 10 rows affected. Well that means 10
1:56:31rows are inserted from the table
1:56:33customers into the target persons. And
1:56:36now what we can do we can go and query
1:56:38the table persons just to check that
1:56:40everything is working perfectly. Select
1:56:42star from persons and let's go and
1:56:45execute. And with that you can see our
1:56:4710 persons that we have added from the
1:56:49customers. So with that we have moved
1:56:51the data from one table and inserted
1:56:54into another table. And as you can see
1:56:55it was very simple. First you have to
1:56:57write a query from the source table in
1:57:00order to collect the data that you need.
1:57:02and then you go and insert it into the
1:57:05target table. So this is really nice and
1:57:07easy and this is another way on how to
1:57:09insert data into your
1:57:14database. Okay, so with that we have
1:57:16learned how to insert data to our
1:57:18tables. Now let's say that I don't have
1:57:20something new. I don't have any rows to
1:57:22be added to my table but I have an
1:57:24update. I would like to go and change
1:57:26the content of the already existing
1:57:29rows. So what you can do? We can use the
1:57:31command updates in order to change the
1:57:34content of already existing rows. So
1:57:37again my friends insert going to go and
1:57:39insert completely new rows but update
1:57:42going to go and change the data of
1:57:44already existing row. Now let's have a
1:57:46look quickly to the syntax of the
1:57:48updates. It start with the keyword
1:57:49updates and then we have to specify the
1:57:51table name and after that we're going to
1:57:53use sit in order to specify what are the
1:57:56new values for the columns. So you have
1:57:58to write down for each column that you
1:57:59want to update a new value and you
1:58:01separate the columns of course using a
1:58:03comma. Now after that we have to specify
1:58:05as well a wear condition. So it's like
1:58:07the queries you say where and then you
1:58:09write a condition and if you don't do
1:58:11that and you don't use the wear clause
1:58:14what going to happen you will be end up
1:58:16updating all the rows inside your table.
1:58:19So that's why we need always the wear
1:58:20clause. All right. So that's all about
1:58:22the syntax. Let's go back to SQL in
1:58:24order to update our data. Okay. So let's
1:58:26have the following task and it says
1:58:28change the score of customer 6 to zero.
1:58:30So that means we have to go and modify
1:58:32the data of the customer ID equal to
1:58:34six. So now first I would like to go and
1:58:36have a look to our data. So select star
1:58:38from customers and now the task is
1:58:41targeting this customer over here and we
1:58:43would like to replace the null to zero.
1:58:46Now how we can go and update this
1:58:48information inside the table? We can use
1:58:50the update command. So what we going to
1:58:52do? We're going to start writing update
1:58:54and after that we have to specify the
1:58:56table name. So what we are updating? We
1:58:58are updating the customers and then
1:59:00we're going to tell the database to set
1:59:02the value of the score to a zero. So we
1:59:05would like to update and change the
1:59:06value from null to a zero. And now here
1:59:09comes something very risky. Don't
1:59:10execute this query yet. If you do that,
1:59:12what's going to happen? The database
1:59:14going to go to the table customers and
1:59:16replace all those values of all
1:59:18customers to zero. So it's going to go
1:59:20and update the whole table and this is
1:59:22of course very risky. That's why in the
1:59:24update command we have to give a wear
1:59:26condition a filter in order to target
1:59:29only specific row or the rows that you
1:59:31want really to modify. In this case we
1:59:33want to change only one row. So what we
1:59:36have to do is to go and specify the work
1:59:38condition like we have done in the
1:59:40select query. Nothing new, right? So
1:59:42we're going to say where the customer ID
1:59:45is equal to six. And with that SQL will
1:59:47not go and update everything. First it's
1:59:49going to filter the data and then
1:59:51updates. And now before I execute just
1:59:53to make sure I go and check which data
1:59:55going to be affected. So it's very
1:59:56simple you go and select star from table
1:59:59customers and then I go and take the
2:00:01exact where and put it in my query and
2:00:04then I select the whole thing and
2:00:06execute. And now if this query gives me
2:00:08the data that should be modified then
2:00:11I'm doing the update command correctly.
2:00:13And in this case we are targeting only
2:00:15one customer. This is the customer
2:00:17number six. And with that I feel really
2:00:19confident with my update. So what we can
2:00:21do since I'm going to use this later I'm
2:00:23going to put the whole thing in a
2:00:24comment and if I execute now only the
2:00:26update going to be executed. So let's go
2:00:29and do that. Now very important to check
2:00:31the message you can see one row is
2:00:33affected which is really good because if
2:00:35I see here 10 rows is affected that
2:00:38means everything is updated. Now let's
2:00:40go and check the data. I'm going to go
2:00:43and remove the wear here and check the
2:00:44whole table. Now you can see we still
2:00:46have the old scores only Anna has now
2:00:49score zero instead of null. So this is
2:00:51how I usually update the data. You have
2:00:53to do it very carefully. Now let's move
2:00:55to another task. It's going to say
2:00:57change the score of the customer number
2:00:5910 to zero and update the country to UK.
2:01:03So now this time we are targeting the
2:01:04user number 10. As you can see she
2:01:06doesn't have the country and score. And
2:01:08the task wants us to change the score to
2:01:11a zero and the country to UK. So now how
2:01:13we going to do it? We're going to use
2:01:14the exact same command but with
2:01:16different condition. So the ID this
2:01:18times is equal to 10 and the score is to
2:01:21zero. But now we have to change as well
2:01:23the country. Now if you want to do
2:01:25multiple updates, you're going to have
2:01:27here a comma after the score and the new
2:01:29line and let's say country equal and
2:01:32then we're going to add UK. So select
2:01:34the whole thing and let's go and
2:01:36execute. So again it is affecting only
2:01:39one row. This is really good. And if you
2:01:41go and check the table search for Sara,
2:01:43you can see in one update we have
2:01:45updated two columns the country and as
2:01:48well the score. So with that we have
2:01:50solved the task. It's very simple. Now
2:01:51moving on to the second task. It says
2:01:53update all customers with a null score
2:01:56by setting their score to a zero. So
2:01:59this time we are not speaking about one
2:02:01specific customer. We are talking about
2:02:03updating the data for a subset of
2:02:05customers. So now imagine you have like
2:02:07hundreds of customers and you are making
2:02:09one update command for each customer.
2:02:11It's going to be really wasting of time.
2:02:12Now instead of that we can specify a
2:02:15condition that targets multiple
2:02:17customers and we're going to do the
2:02:19update for those customers in one go. So
2:02:21now let's see how we're going to do it.
2:02:22We are talking only about replacing the
2:02:24nulls with a zero. So we don't need the
2:02:27country. So set score equal to zero. But
2:02:30now we will not be specific for the ids.
2:02:32Now we have to make a new condition.
2:02:34It's going to say like this where score
2:02:36is null. Now of course in the course we
2:02:38have a full dedicated chapter about the
2:02:40nulls and here all what we are doing is
2:02:42we are searching for scores that is
2:02:44equal to null. But we cannot write an
2:02:46equal we have to write it like this is
2:02:48null. Of course before we update
2:02:50anything we have to go and test it in a
2:02:52query. So select star from customers
2:02:55where score is null. Let's go and
2:02:56execute. Now as you can see we have two
2:02:59customers where the score is null. So
2:03:01that means this condition is targeting a
2:03:04subset of customers and we're going to
2:03:06do now the updates for multiple rows for
2:03:08this subset. So that means we can run
2:03:11this query. Let's go and execute it. Now
2:03:13you can see two rows are affected. So
2:03:16that means multiple rows got affected
2:03:19got updated. So now if you go and query
2:03:21our table customers you can see we don't
2:03:24have any nulls inside the scores and we
2:03:26have replaced all the nulls with a zero.
2:03:28And of course you can do the same thing.
2:03:30you can go and make an update command in
2:03:33order to replace all the nulls in the
2:03:35country to maybe something unknown or
2:03:37any default value that you want. So this
2:03:39is how you can update multiple rows in
2:03:41one
2:03:45go. All right my friends. So with that
2:03:47we have learned how to insert new rows
2:03:50to our tables and as well how to update
2:03:52the content of already existing row. Now
2:03:54the last thing or command that we can do
2:03:57to the data inside the table that we can
2:03:59go and remove rows from our table and we
2:04:02can do that using the command delete. So
2:04:04if you use delete SQL going to go and
2:04:07start removing already existing rows
2:04:09inside your table. All right. Now for
2:04:11the syntax of the delete it's going to
2:04:13be very simple. We're going to say
2:04:14delete from and then we're going to
2:04:16write the table name. And here comes
2:04:18something very important. We have to add
2:04:20a wear condition. And it's like the
2:04:22update. If you don't do that, if you
2:04:23don't include where condition, what
2:04:25going to happen? You will end up
2:04:27deleting all the rows inside the table.
2:04:29So the syntax is very simple. Let's go
2:04:31back to scale in order to delete some
2:04:33data. Okay. So now we have the following
2:04:35task. Delete all customers with an ID
2:04:38greater than five. So now we have to go
2:04:40and delete all the customers that we
2:04:41recently added. So how we going to do
2:04:43it? It's very simple. We're going to say
2:04:44delete from. So that means I want to
2:04:47delete something from a table. And we
2:04:49have to specify the table name. It's
2:04:51going to be the customers. So the syntax
2:04:52is very simple. Now my friends, this is
2:04:54more risky than updates because if you
2:04:57execute it like this, don't do that yet.
2:04:59Wait, what's going to happen? All the
2:05:01data of the customers going to be
2:05:02deleted. So you will get an empty table
2:05:05and we will not do that. So now we're
2:05:07going to do exactly like the update
2:05:08command. We're going to specify the work
2:05:10clause. So it says the ID should be
2:05:12greater than five. So that means ID
2:05:15higher than five. So with that we are
2:05:17defining a subset of the data that
2:05:19should be deleted, not everything. And
2:05:21if we check in the updates, we have here
2:05:22to do a double check before deleting
2:05:24anything. So again what we do, we select
2:05:27star from table customers and we're
2:05:30going to go and copy the work condition
2:05:32in order to test what going to be
2:05:34deleted. So it's going to be all the
2:05:35customers that is higher than five. And
2:05:37with that I'm making sure that my delete
2:05:40command is correct which is from what I
2:05:41see here is correct. So those five
2:05:44customers should be deleted. So now
2:05:45let's go and delete those customers. And
2:05:48now very important to read the message.
2:05:50It says five rows affected. So that
2:05:52means five customers got deleted. And
2:05:54this is better than 10 of course. So
2:05:56let's go and check what customers left.
2:05:59So we have 1 2 3 4 5. Those are the
2:06:01original customers. And everything else
2:06:03got deleted. And with that we have
2:06:05solved the task. And this is how we can
2:06:06delete data from tables. Be very
2:06:09careful. Always test before doing the
2:06:11delete command. Okay. So now we have the
2:06:13following task. And it says delete all
2:06:15data from table persons. So that means
2:06:18we have to go and drop everything from
2:06:20the table persons. But we don't want to
2:06:22delete the table. We just want to delete
2:06:24the data inside the table now. So now
2:06:26what we're going to do, we're going to
2:06:27write delete from. And now we have to
2:06:29specify the table persons. And if you
2:06:31execute it, what's going to happen? SQL
2:06:33going to go and drop all the data in the
2:06:35persons. But in SQL, we have more
2:06:37interesting command. If you want to
2:06:39delete everything from the table
2:06:40persons, we have that truncate.
2:06:42Truncate. It is exactly like delete from
2:06:44persons. It's going to go and make the
2:06:46whole table empty. But why I like to use
2:06:48truncate because it is way faster than
2:06:51deletes. If you have large tables, the
2:06:52delete command going to be really slow
2:06:54because with the delete there is like a
2:06:56lot of things happening behind the
2:06:57scenes. There is like logs and
2:06:59protocols. But if you are using trunk,
2:07:01the database going to skip all those
2:07:03extra stuff and it's going to be very
2:07:05fast. So if you want to delete all the
2:07:07data from table, you can do it like this
2:07:09if it's like small table. But what I
2:07:10usually do, I go and write truncate and
2:07:14then table. we're going to get the same
2:07:16effect and with that I'm saying reset
2:07:18everything make the table empty. So
2:07:20let's go and execute it and now with
2:07:21that you will not get the number of
2:07:23deleted rows and that's why it's
2:07:25truncate it's way faster. It is not
2:07:27protocoling anything it's not logging
2:07:29anything it just go and delete all the
2:07:31data without any extra steps. So this is
2:07:34how we can delete all the data from a
2:07:36table but the table still exists. Okay
2:07:38my friends, so with that you have
2:07:39learned the basics on how to manipulate
2:07:41your data inside the database the data
2:07:43manipulation language DML and with that
2:07:45I can tell you we have covered the
2:07:47basics of SQL. So with that we have
2:07:50covered the beginner level. Now in the
2:07:52next chapters we will be in the
2:07:53intermediate level and the first thing
2:07:55that you're going to learn in the
2:07:56intermediate level you will learn how to
2:07:58filter your data and we're going to
2:07:59cover many operators that you can use
2:08:01inside the workclass. So let's go.
2:08:08All right. So now let's have an overview
2:08:09about all different operators in SQL. So
2:08:12the first group of operators we have the
2:08:13comparison operators. They are the
2:08:15easiest one where all what we have to do
2:08:17is to compare two values and we have
2:08:19like six different variants and how to
2:08:20do that. Now to the next one we have the
2:08:23logical operators. We use it in order to
2:08:25combine multiple operators. And moving
2:08:27on to the next one we have the range
2:08:29operator. Here we have only one, the
2:08:31between. We're going to use it in order
2:08:32to check whether a value falls within a
2:08:35specific range. Now moving on to the
2:08:37next one, we have the membership
2:08:38operator. And here we have two things.
2:08:40We have the in operator or not in. Here
2:08:42all what you have to do is to check
2:08:44whether a value is in a list or not. And
2:08:46the last category that we have is the
2:08:48search operator. And here as well we
2:08:50have only one operator that like we use
2:08:52it in order to search for a specific
2:08:54thing in a text. So my friends, we're
2:08:56going to go through all those operators
2:08:58one by one. Okay. So now let's go and
2:09:00deep dive into the first category the
2:09:02comparison operators and we're going to
2:09:04cover all those stuff. So what is
2:09:05exactly comparison
2:09:10operator? Okay. So what is exactly
2:09:12comparison operators? It is very simple.
2:09:14We want to compare two things and there
2:09:16is a lot of things that we can compare
2:09:18in SQL. But the formula for that going
2:09:20to be always like this. So we have the
2:09:22first expression and then operator and
2:09:24then we have another expression and this
2:09:27going to form something called
2:09:28condition. So here we have a lot of
2:09:30variance. We can compare one column to
2:09:32another column. So for example, you can
2:09:34go and compare the first name with the
2:09:36last name. So both of the expressions
2:09:38are columns here. Another scenario, you
2:09:41want to compare a column with a value, a
2:09:43static value. Like for example, you say
2:09:46the first name must be equal to a value
2:09:49like John. So now we are comparing a
2:09:51column with a value. It's not anymore
2:09:53two columns. Now we have another
2:09:55scenario where we want to apply a
2:09:57function to a column and then compare
2:09:59the results to maybe a value. So for
2:10:01example, we apply the upper function to
2:10:03the first name and then this must be
2:10:06equal to a value like John with all the
2:10:08letters in the uppercase. And one more
2:10:10thing that you can compare you can write
2:10:12an expression in one of the sides like
2:10:15for example you can say if we multiply
2:10:17price with the quantity it must be equal
2:10:20to 1,000 for example. So here we have an
2:10:23expression. We have multiple columns
2:10:25included in one sides and the output of
2:10:27this expression must be equal to 1,000.
2:10:29And now the last one is going to be a
2:10:31little bit more advanced and we're going
2:10:32to cover that of course in other
2:10:33chapter. We can include a whole query
2:10:36the complete query to one of the sides
2:10:38and we call this a subquery. So in one
2:10:40of the sides you're going to write a
2:10:42whole query select from where whatever
2:10:45you want and you go and compare the
2:10:47result of this query to for example a
2:10:49value or a column. So as you can see in
2:10:51a scale we can compare a lot of things
2:10:53together. Either comparing the columns
2:10:55together or a column with a value or we
2:10:57use a function or an expression or even
2:11:00a whole query. So this is how we build
2:11:02conditions in SQL. Okay my friends. So
2:11:05let's see how the conditions works in
2:11:06SQL. So we have our data the name the
2:11:09country the score and let's say that we
2:11:11have built a condition where it says the
2:11:13country must be equal to the USA. So
2:11:15this is very simple comparison operator
2:11:17and this is the condition that we are
2:11:20using inside the work clause. So once
2:11:22you apply this filter to your data what
2:11:24going to happen? SQL going to go row by
2:11:26row evaluating whether it is meeting the
2:11:28condition. If it's not fulfilling the
2:11:30condition then SQL going to remove it
2:11:32from the results. But if it is
2:11:33fulfilling the condition it's going to
2:11:35keep it. So now we are comparing the
2:11:37values of column together with a static
2:11:39value the USA. So we're going to compare
2:11:41whatever value we get from the country
2:11:43together with the USA. So now let's see
2:11:45how is going to apply this filter to our
2:11:47data for the first customer Maria. Now
2:11:50you can see the value inside the country
2:11:51is Germany. So Isql now going to go and
2:11:54compare Germany to USA since it is not
2:11:57equal. Then is going to understand okay
2:11:59Maria is not fulfilling the condition.
2:12:02So it is false and is going to go and
2:12:04remove this customer from the results.
2:12:06So she is not fulfilling the condition.
2:12:08Moving on to the next one to Joan. Now S
2:12:10is going to take the value inside the
2:12:12country the USA it is equal to USA. So
2:12:15that means John is fulfilling the
2:12:17condition and Isl going to be happy
2:12:18about it. So it is true and this means
2:12:21is going to keep Joan in the final
2:12:23results. Now moving on to George the
2:12:25value is UK not equal to USA. He is not
2:12:27fulfilling the condition. Is going to go
2:12:29and remove him from the final result.
2:12:31Same thing for Martin. Germany is not
2:12:34equal to USA. Is going to remove this
2:12:36customer as well. And to the last one
2:12:38bit better you can see the value is USA.
2:12:40So USA equal USA. The condition is
2:12:42fulfilled. SQL is happy about it and
2:12:44going to leave the customer in the
2:12:45output. So now if you go and apply this
2:12:47condition using the comparison operator
2:12:50to your data only two customers going to
2:12:52be left in the output. This is exactly
2:12:54how the conditions and the comparison
2:12:56operators works in SQL. Okay. So now
2:12:58let's start with the first operator.
2:13:00It's very simple. We have the equal.
2:13:02It's going to checks if the two values
2:13:04are equal. That's very simple. Let's
2:13:06have an example. Okay. So now we have
2:13:07this task. It says retrieve all
2:13:09customers from Germany. So this is very
2:13:11basic. We're going to go and select and
2:13:12we're going to select all the columns
2:13:14since we don't have any specifications
2:13:15from the table customers. And if you go
2:13:17and execute it, you will get all the
2:13:19customers. But we don't need that only
2:13:21the customers that comes from Germany.
2:13:23So we have to go and apply a condition
2:13:25using the wear clause country equal to
2:13:28the value Germany. So make sure you are
2:13:31writing it exactly like in the database
2:13:33otherwise it will not work. So let's go
2:13:35and execute and with that we are getting
2:13:37only the customers from Germany. So it
2:13:39is very simple and this is why we use
2:13:41the equal operator. Okay. So now moving
2:13:43on to the next one again very simple. If
2:13:45you want to check if two values are not
2:13:48equal we can use the not equal operator.
2:13:50So let's have an example. Okay. So now
2:13:52we let's have the opposite task. It says
2:13:54retrieve all customers who are not from
2:13:57Germany. So this is very simple. We are
2:13:59saying here who are not they are not
2:14:02equal to Germany. So we can use the not
2:14:05equal operator in order to get these
2:14:07customers. So with that as you can see
2:14:09after executing we are getting all the
2:14:12customers country is not equal to
2:14:14Germany and there's like another way on
2:14:16how to do the not equal doing it like
2:14:18this we'll get the same results. All
2:14:19right my friends moving on to the next
2:14:21one. We can check if a value is greater
2:14:24than another value. So we use the
2:14:26greater operator. Let's have an example.
2:14:28Okay. So now the next task it says
2:14:30retrieve all customers with a score
2:14:33greater than 500. Now we want to filter
2:14:36the data based on the score. So we're
2:14:38going to say where score and now the
2:14:41task says greater than 500. We're going
2:14:44to use the operator greater than 500.
2:14:46It's very simple. So with that we will
2:14:48get only the customers where the score
2:14:51is higher than 500. So for example Maria
2:14:54it's not fulfilling the condition. The
2:14:56same thing for the Peter and as well for
2:14:58Martin it must be greater than 500. So
2:15:01if you go executed you will get only
2:15:03those two customers because they are
2:15:05greater than 500. Okay, moving on to the
2:15:09next one. This time we're going to check
2:15:10if a value is greater than or equal to
2:15:13another value. So it is like mix between
2:15:15the greater than and the equal. If one
2:15:17of them is fulfilled then the value
2:15:19going to meet the condition. So let's
2:15:21have an example for that. Now, if the
2:15:22task says retrieve all customers with a
2:15:25score of 500 or more, this time we're
2:15:28going to go and include the customers
2:15:30where their score is equal as well to
2:15:32500 or higher. So, we're going to have a
2:15:34similar condition based on the score and
2:15:36the 500's value, but this time we're
2:15:38going to say greater or equal to 500.
2:15:41So, if you go now and execute it, this
2:15:44time we're going to see the customer
2:15:45Martin with the score of 500. So, in
2:15:48this scenario, we're going to use
2:15:49greater or equal. All right. Right. So
2:15:51now let's keep moving. The next one is
2:15:53as well very simple. We're going to
2:15:54check this time if a value is less than
2:15:57another value. So we're going to use the
2:15:59less operator. Let's have an example.
2:16:01Now moving on to another simple task.
2:16:03Retrieve all customers with a score less
2:16:06than 500. So this time we want all the
2:16:08customers with a lower score. And we're
2:16:10going to use exactly the opposite. It's
2:16:12going to be the score is less than 500.
2:16:14And again here it is not equal, right?
2:16:16So if you go and execute, you will get
2:16:18all the customers with a low scores. he
2:16:21will not get to Martin because Martin is
2:16:23equal to 500. So with that we have
2:16:25solved the task. We have all the
2:16:26customers with the score less than 500.
2:16:29Okay my friends, now moving on to the
2:16:31last one. I think you already got it. So
2:16:33we're going to check whether a value is
2:16:35less than or equal to another value. So
2:16:38you can go and combine the less operator
2:16:40together with the equal and if one of
2:16:42them is fulfilled then the value going
2:16:44to meet the condition. So let's have an
2:16:46example for that. This time we are
2:16:48retrieving all customers with a score of
2:16:50500 or less. So the query going to be
2:16:52very similar but we are saying it is
2:16:55less or equal to 500. So we are
2:16:57including the value in our condition.
2:17:00And with that as you can see we still
2:17:01have our two customers where we have the
2:17:04score less than 500 but we have now as
2:17:06well Martin with a score of 500. Okay my
2:17:09friends. So with that we have covered
2:17:11the first group the comparison
2:17:13operators. Now we're going to move on to
2:17:14the next group. We're going to speak
2:17:16about the logical operators and here we
2:17:17have three and or not. So let's start
2:17:20with the first one. What is exactly and
2:17:26operator. Okay. So now what is the
2:17:28definition of the and it says all
2:17:31conditions must be true. So all the
2:17:33conditions that you have in the wear
2:17:34clause must be true in order to keep the
2:17:37row in the results. So let's understand
2:17:39what this means. things going to get
2:17:40more complicated where you can have not
2:17:43only one condition but you might have
2:17:45multiple conditions in your query. So
2:17:47here we're going to add a second
2:17:48condition where we're going to say not
2:17:50only the country must be equal to USA
2:17:53but also the score must be higher than
2:17:55500. So now you have two conditions and
2:17:58you have to put them in the wear clause.
2:18:00Now you have to combine those conditions
2:18:02using the logical operator and here we
2:18:04have two options two operators the and
2:18:06operator and the or operator. In this
2:18:09scenario, if you say and then SQL is
2:18:12very restrictive. Both of the conditions
2:18:14must be true in order to keep the row in
2:18:17the results. So now let's see how this
2:18:18going to work. Now for the first row and
2:18:20for the first condition you can see the
2:18:22country is Germany and it is not
2:18:24fulfilling the first condition. So this
2:18:27going to be false. And as well if you
2:18:28check the second condition for the first
2:18:30row you can see the score is 350. So
2:18:32that means this customer is as well not
2:18:34fulfilling even the second condition. So
2:18:37both of the conditions is false and it's
2:18:39going to go I remove this customer from
2:18:41the results. Now to the next one John
2:18:43you can see John is fulfilling the first
2:18:45condition because the country is equal
2:18:46to USA and as well fulfilling the second
2:18:49condition. His score is 900 and this is
2:18:52higher than 500. So now SQL going to be
2:18:54very happy about it because both of them
2:18:56is true and this is the only way in
2:18:58order to keep the row in the output
2:19:00because we are using the operator and so
2:19:03John going to stay in the output. Now
2:19:04moving on to George. He is not
2:19:06fulfilling the first condition. But now
2:19:08the second condition is fulfilled. His
2:19:10score is 750 and this is higher than
2:19:12500. So now it's like 50/50 right. In
2:19:15one side it's false but the other side
2:19:17is true. But this is not enough for the
2:19:19ant operator. Both of them should be
2:19:21true in order to keep the result in the
2:19:22output. That's why SQL going to remove
2:19:24this row. Now moving on to Martin. He is
2:19:26not fulfilling both of the conditions.
2:19:28So SQL going to go I remove it from the
2:19:30results. And now for the last one. Peter
2:19:33is fulfilling the first condition. the
2:19:35country is equal to USA but the second
2:19:37condition is sadly not fulfilled so we
2:19:39have the score zero not higher than 500
2:19:42again we have the same scenario it's
2:19:4350/50 and this is not enough for the ant
2:19:46operator that's why SQL going to go I
2:19:48remove it so as you can see if you use
2:19:50an and operator a lot of rows going to
2:19:52be removed if one of the condition is
2:19:55not met so the ant operator is very
2:19:57restrictive both of the conditions must
2:19:59be fulfilled to keep the row in the
2:20:01results so this is exactly how the and
2:20:04operator works. Okay. So now we have the
2:20:05following task. Retrieve all customers
2:20:07who are from USA and have a score
2:20:10greater than 500. So here we are like
2:20:12combining multiple conditions and let's
2:20:15go and do it step by step. So the first
2:20:16thing that we have to go and select the
2:20:18data from the correct table. So select
2:20:20star from customers and with that we are
2:20:22getting all the customers from the
2:20:24table. Now the first condition we need
2:20:26the customers that come from USA. So we
2:20:28need only those two customers and in
2:20:31order to do that as we learned we can go
2:20:32and use the wear clause and the
2:20:34condition going to be country equal to
2:20:37USA. So if you go and execute we will
2:20:39get those two customers. Nothing is new.
2:20:42We have used the compression operator
2:20:44equal. But we are not done yet. We have
2:20:46another condition from those two
2:20:48customers. We need only the customers
2:20:50where their score is higher than 500. So
2:20:53now by looking to those two customers
2:20:54you can see we see that the bitter here
2:20:57does not have a score higher than 500
2:20:59and we don't want to see that in the
2:21:01results. So now what we have to do we
2:21:02have to go and write a condition for
2:21:04this one over here. So this is based
2:21:06this time on the scores not on the
2:21:08country. So the score should be greater
2:21:11than 500. Now as you can see we have the
2:21:14first condition for the first one here
2:21:16and the second condition for the second
2:21:18requirement. Now the question how to
2:21:20connect those two conditions. So here we
2:21:22have two options and or and to be honest
2:21:25this is very simple the task says it
2:21:27customer should fulfill both of the
2:21:29conditions should be from USA and as
2:21:31well at the same time greater than 500.
2:21:34So it is very simple real and so with
2:21:37that we have connected both of those
2:21:39conditions and if you go and query it
2:21:41you will get only one customer that is
2:21:43fulfilling our conditions. So from all
2:21:46customers we have only one customer
2:21:48that's fulfilled this condition that
2:21:50comes from USA and at the same time the
2:21:52score of this customer is higher than
2:21:55500. So this is how we use the ant
2:21:57operator in order to connect two
2:21:59conditions. Okay my friends. So that's
2:22:01all for the ant operator. Let's speak
2:22:03now about the or
2:22:08operator. All right. Now the or operator
2:22:11it says at least one condition must be
2:22:13true. So it is less restrictive than the
2:22:16and it is enough to have one condition
2:22:18true in order to keep the row in the
2:22:20results. Let's understand exactly what
2:22:22this means. Okay. So now we have the
2:22:23same scenario. We have two conditions
2:22:25and in SQL you have to connect them
2:22:27either using the and operator or the or
2:22:30operator. In this scenario we're going
2:22:32to talk about the or operator. And as we
2:22:34said at least one of the conditions must
2:22:36be fulfilled in order to leave the
2:22:38record in the results. So let's see
2:22:40what's going to happen here. Now the
2:22:41first customer Maria she is not
2:22:44fulfilling the first condition and as
2:22:46well the second condition. So both of
2:22:48them is false and this is the only
2:22:50scenario where SQL going to remove the
2:22:52record from the results because it is
2:22:54not fulfilling the minimum at least one
2:22:57of them should be true. Both of them is
2:22:58false then SQL going to go and remove
2:23:00this row. Now moving on to the next one
2:23:02to John. John is from USA and has higher
2:23:05score than 500. Both of the conditions
2:23:08is green. So both of them is true and
2:23:10this is more than enough to keep the row
2:23:12in the output. That's why we will see
2:23:14John in the outputs. Now moving on to
2:23:16the third one, George. George is not
2:23:18fulfilling the first condition because
2:23:20UK is not equal to USA. But John this
2:23:22time is fulfilling the second condition.
2:23:25So we have here true and since we have
2:23:27at least one true, this is good enough
2:23:29to keep the record in the output. So you
2:23:31will see George in the results. Now
2:23:33moving on to Martin. He is not
2:23:35fulfilling the first condition as well
2:23:37not fulfilling the second condition.
2:23:39Both of them is false and this is not
2:23:41enough to keep the result in the output.
2:23:43So that's why it's still going to go and
2:23:45remove it. Now moving on to the last
2:23:46one. Peter he is fulfilling the first
2:23:48condition but not the second condition
2:23:50but still everything is fine because he
2:23:52is fulfilling at least one condition. So
2:23:55we have the minimum and it's still going
2:23:56to leave it in the output. So as you can
2:23:58see the or operator is not restrictive
2:24:00like the and operator. It's enough to
2:24:02have one true in order to keep the data
2:24:04in the output. And this is exactly how
2:24:06the or operator works. Now let's see the
2:24:08second task. Retrieve all customers who
2:24:11are either from USA or have a score
2:24:14greater than 500. So it is a very
2:24:16similar task. We have two conditions. So
2:24:18we need the customers that are either
2:24:20from USA. So it is based on this country
2:24:23equal to USA. And the second condition
2:24:25is the score is greater than 500. But
2:24:28this time we are very relaxed. either
2:24:30this condition is fulfilled or the
2:24:32second one. So instead of having and we
2:24:35will be using the operator or. So it is
2:24:37enough to fulfill one of those
2:24:38conditions. And if you go and execute
2:24:40now as you can see we are getting more
2:24:42results because it is easier to fulfill
2:24:44the conditions. So we can see those
2:24:46three customers either fulfilling the
2:24:48first condition or the second one. All
2:24:51right my friends. So that's all for the
2:24:52or operator and we're going to move to
2:24:54the last one in this group the not. So
2:24:56what do we mean with the not operator?
2:25:02Okay. So now what is this operator not?
2:25:04It is a reverse operator. It's going to
2:25:07go and exclude the matching values. So
2:25:10what this exactly means? Let's have a
2:25:11very simple example. All right. So now
2:25:13the net operator is not like the or and
2:25:16the ands. This operator will not go and
2:25:18combine two conditions. So you can use
2:25:21it with only one condition. And let's
2:25:22say that our current condition is like
2:25:24this. The country must be equal to USA.
2:25:26So this is like a comparison operator.
2:25:28And if you apply it to your data, as we
2:25:30learned, it's going to leave only two
2:25:31customers, John and Peter, because they
2:25:34fulfill the conditions and all other
2:25:35customers will be removed because they
2:25:38don't fulfill the condition. So nothing
2:25:39crazy so far. But now if you go and
2:25:41apply the not operator to the condition,
2:25:44what going to happen? You're going to
2:25:45reverse the whole truth. So you are
2:25:47saying if this condition is fulfilled,
2:25:50it must be removed from the final
2:25:52results. So it is switching everything.
2:25:54We want to see the customers that is not
2:25:57fulfilling the condition. So now let's
2:25:59see what can happen if you apply the not
2:26:01operator together with the condition. We
2:26:02can see that the first customer is not
2:26:04fulfilling the condition which is great
2:26:06thing. This is exactly what we want. We
2:26:08want the customer that is not fulfilling
2:26:10the condition. That's why going to be
2:26:12happy about it and SQL going to make it
2:26:14true and leave it in the output. So
2:26:17Maria is fulfilling the whole thing. She
2:26:19is not meeting the condition. So SQL
2:26:22going to leave it at the output. Now for
2:26:23the next one. So this customer is
2:26:25fulfilling the condition and that is not
2:26:27a good thing. So SQL going to go and
2:26:29this time remove John from the results
2:26:31because he is fulfilling the condition.
2:26:33And moving on to George. So George is
2:26:36not fulfilling the condition which is
2:26:38amazing. So that's why SQL going to keep
2:26:40this time George in the output. The same
2:26:42thing for Martin. Martin is not
2:26:44fulfilling the condition. So Isl going
2:26:46to keep the customer and better he is
2:26:48fulfilling the condition. So SQL going
2:26:50to go and remove this customer from the
2:26:52output. So as you can see we have
2:26:54reversed everything right. The not
2:26:56operator going to make the true false
2:26:58and the false true. Okay. So this is how
2:27:00it works. Now let's go back to SQL in
2:27:02order to practice. Okay. The next task
2:27:05it says retrieve all customers with a
2:27:07score not less than 500. So this sounds
2:27:11really funny. As usual we're going to go
2:27:12and select star from customers. And now
2:27:16we have to filter the data based on this
2:27:18condition. So the score is not less than
2:27:21500. Well, you can go and say well the
2:27:23score is higher, greater or equal to
2:27:27500, right? And with that it is not less
2:27:30than 500. So if you go and execute it,
2:27:32we just solve the task, right? We get
2:27:34all the customers that are not less than
2:27:36500. Or you can go and use the not
2:27:38operator to make things more funnier. So
2:27:41you go over here and say it is not and
2:27:44then you switch it. So you make like
2:27:46this. So the score is less than 500. But
2:27:49as we use here not then we twisted
2:27:51everything. So we are saying the score
2:27:54is not less than 500. And if you execute
2:27:56it you will get the exact same results.
2:27:59Convert the truth. If you remove it and
2:28:01execute you will get everything that is
2:28:03less than 500. But if you put the nut
2:28:06you will convert the whole logic. So if
2:28:08you go and execute you are not getting
2:28:10the scores that are less than 500. So
2:28:12this is really nice. This is how you use
2:28:14the nut operator. Okay my friends. So
2:28:16with that we have covered everything
2:28:17about the logical operators. Now we're
2:28:19going to move to the third group. We're
2:28:21going to talk about the range operator.
2:28:23And here we have only one the between.
2:28:25So what is exactly between
2:28:30operator? Okay. So what is between? It's
2:28:32going to go and check if a value falls
2:28:35within a specific range. So you have a
2:28:37range and you are checking whether your
2:28:39value is in the range or outside the
2:28:41range. So let's understand exactly what
2:28:43this means. Okay. So now in order to
2:28:45build a range you need two things. You
2:28:47need the lower boundary for the range
2:28:49and you need as well the upper boundary.
2:28:52Once you have two boundaries then you
2:28:54have a range and everything between
2:28:56those two boundaries going to be true
2:28:59and everything outside those boundaries
2:29:01going to be false. So now for example
2:29:03let's say that we have the lower
2:29:04boundary 100 and the upper boundary 500.
2:29:07And there is one thing that you have to
2:29:08understand about the between the
2:29:10boundaries are inclusive. So that means
2:29:12if a value is exactly 100 or exactly 500
2:29:16then it's going to considered as a true.
2:29:18So it is considered to be inside the
2:29:20range. Now if you apply this filter to
2:29:22our data where we say the score must be
2:29:25between 100 and 500 going to go and do
2:29:28the following. So for the first customer
2:29:30Maria is going to go and check whether
2:29:32her score is inside the boundaries. So
2:29:35as you can see 300 is between 100 and
2:29:37500. So she is in the green area and
2:29:40that's why Isque going to be happy about
2:29:41it and leave the customer in the
2:29:43outputs. Now moving on to John. John has
2:29:45900. As you can see 900 is greater than
2:29:48500. So this value is going to be
2:29:50outside the boundaries on the right side
2:29:52and this means the score of John is not
2:29:54in the range. That's why he is not
2:29:56fulfilling the condition and SQL going
2:29:58to go and remove this customer from the
2:30:00results. Now moving on to George 750.
2:30:03The same thing outside the range. SQL
2:30:05will not accept it and remove this
2:30:07customer from the final results. Now
2:30:09moving on to Martin his score is 500 and
2:30:12this is exactly at the boundary. So if
2:30:14it's like 5001 it's going to be outside.
2:30:17So since between is inclusive then SQL
2:30:20going to accept it and Martin considered
2:30:22to be in the range and fulfilling the
2:30:24condition. So SQL going to keep him in
2:30:25the final result. Now here are speaking
2:30:27about better he has zero score and this
2:30:30is less than 100. So in the left side
2:30:32not in the range. So not fulfilling the
2:30:34condition and SQL going to go and remove
2:30:36him. This is exactly how between works
2:30:38in SQL. It's very simple. Okay. So now
2:30:40we have the following task and it says
2:30:42retrieve all customers whose score falls
2:30:45in range between 100 and 500. So let's
2:30:48start as usual by selecting all data
2:30:51from customers and execute it. Now the
2:30:54task says everything. We need all
2:30:55customers in a range. So we have a lower
2:30:58value and a higher value. So in order to
2:31:00do that as usual we're going to use the
2:31:02where and then we're going to specify
2:31:04the column that we want to filter on. So
2:31:06it's going to be the score and since we
2:31:07have like two boundaries we can go and
2:31:09use the function between and we start
2:31:11with the first boundary the lowest
2:31:12boundary. So it is the 100 and 500 the
2:31:17high boundary the upper boundary. So
2:31:19between 100 and 500. So now let's go and
2:31:23execute it. And with that we get only
2:31:25those two customers because they are
2:31:27between this window. Now there is
2:31:29another way in how to solve this task by
2:31:31not using between. We can go and use the
2:31:34comparison operator together with a
2:31:36logical operator and. So let me show you
2:31:38how we can do that. I'm going to go and
2:31:40copy the whole thing. And now we're
2:31:41going to write two conditions. So first
2:31:43the score should be higher or equal to
2:31:46100 because the boundaries is inclusive
2:31:49and the other one the score is less or
2:31:52equal to 500. So this is the upper
2:31:55boundary. So with that we have the two
2:31:56conditions and we can go and connect
2:31:58them using the and operator. So it's
2:32:00like very similar to the between we have
2:32:02an and between the upper and the lower
2:32:04boundaries but we are using the
2:32:06comparison operators. So it is higher or
2:32:09equal to 100 and lower or equal to 500.
2:32:12If you go and run this query you will
2:32:14get exactly same results. Now if you ask
2:32:16me which method is my favorite I'm going
2:32:18to go with this method and I will skip
2:32:20the between because each time to be
2:32:22honest for me I forget about the between
2:32:24whether the boundaries are inclusive or
2:32:26exclusive. But if I read the script I am
2:32:29going to see exactly that those
2:32:30boundaries are inclusive because we have
2:32:32here the equals. So I really prefer
2:32:34using the compressor operator together
2:32:36with the and then using between. So it's
2:32:38up to you if you memorize it then go
2:32:40with the between. But for me I'm going
2:32:42to go with the compression operators.
2:32:44Okay my friends. So that's all about the
2:32:45between and the range operator. Now
2:32:47let's move to another group. We have the
2:32:49membership operator. So here we have
2:32:51like two. We have the in and the not in.
2:32:53So let's understand what this exactly
2:32:59means. Okay. So what is in operator?
2:33:01It's going to go and check if a value
2:33:03exist in a list. So you have a list of
2:33:06values and you are checking whether your
2:33:07value is a member of your list. So let's
2:33:10have very simple example in order to
2:33:11understand what this means. Okay. So now
2:33:13how this works exactly what you have to
2:33:15do is to go and make a list of values.
2:33:18So let's say that I have a list and
2:33:19there I have specified two values
2:33:21Germany and USA. So those two are the
2:33:24members of this list. Now if you use the
2:33:26n operator it's going to go and check
2:33:28the value of countries whether it is in
2:33:30the list or not. So let's do it one by
2:33:32one. For the first customer Maria her
2:33:34country is Germany and Germany is member
2:33:37of the list. So it's going to be happy
2:33:39and going to leave Maria in the final
2:33:41results. Now moving on to John. John
2:33:43comes from USA. USA is member of the
2:33:45list. So he is fulfilling as well the
2:33:47condition and you're going to see John
2:33:49in the final results. Now we come to
2:33:50George. George comes from UK and UK is
2:33:53not member of our list. And SQL going to
2:33:56go and remove this customer from the
2:33:58final results not fulfilling the
2:33:59condition. Now for the last two, Martin
2:34:01and Peter, their country is a member of
2:34:03the list and SQL going to go and leave
2:34:05those customers in the final results. So
2:34:07as you can see it's very simple. Or what
2:34:09you have to do is to define the members
2:34:10of a list and use the n operator and if
2:34:13the value is a member of this list it's
2:34:15going to be true otherwise it's going to
2:34:17be false. Now of course the other
2:34:18operator going to be exactly the
2:34:20opposite where we say not in the list.
2:34:23So we are searching for values that are
2:34:25not in this list. So as we are using not
2:34:27it's going to go and reverse completely
2:34:29the truth. And if you apply this you
2:34:31will get in the result only one
2:34:33customer. you will get George and the
2:34:35result because the country is UK and UK
2:34:38is not a member of the list. So if you
2:34:40use not together with the in operator
2:34:42you will get exactly the opposite
2:34:44effect. So this is how the in and the
2:34:46not in operator works in SQL. Let's go
2:34:48back to scale in order to practice that.
2:34:50Okay. So now we have this task and it
2:34:52says retrieve all customers from either
2:34:54Germany or USA. Okay. So let's try to
2:34:57solve this task. This going to be a
2:34:59little bit tricky. So select star from
2:35:02customers as usual and execute it. So
2:35:04now we need in the results only customer
2:35:06that comes either from Germany or USA.
2:35:09So that means this customer over here
2:35:11should be excluded from the result
2:35:12because he come from UK. So how we going
2:35:14to write it? It's going to be like this
2:35:16maybe. So the first one going to be the
2:35:19country is equal to Germany or the
2:35:22country is equal to USA right something
2:35:25like this. So if you go and execute it,
2:35:27you will get in the output only the
2:35:28customers that are either from Germany
2:35:30or USA. And with that we have solved the
2:35:32task, right? Well, there is another way
2:35:34in order to solve this task which is
2:35:36more clear and shorter using the n
2:35:38operator. So now how we going to do it?
2:35:40Let's go and get the whole thing in
2:35:43another query. And now instead of having
2:35:45equals and ors and so on, we're going to
2:35:48use the in operator and then we're going
2:35:50to have like two parentheses and then
2:35:51inside it we're going to have a list of
2:35:53values. So it's going to be the Germany
2:35:55and then the second value going to be
2:35:57USA like this. So we are saying country
2:36:00should be in this list Germany or USA
2:36:03and if it is like one of those values
2:36:05then the condition is fulfilled. So now
2:36:07if you go and execute this one over here
2:36:09you will get the exact same results. So
2:36:12my friends, if you notice that you are
2:36:13repeating yourself in the wear condition
2:36:15and you are just changing the value of
2:36:17the condition, it is based on the same
2:36:19column and you are connecting them using
2:36:21the or then there is something wrong and
2:36:23always think on this scenario to use the
2:36:25in operator because this can be really
2:36:28ugly once you have a lot of values. So
2:36:30imagine in our database we have a lot of
2:36:32countries and your query going to be
2:36:34like something like this. So you are
2:36:36keep repeating country equal or country
2:36:38equal and so on. Instead of that you're
2:36:40going to have a really nice list of
2:36:42countries in one go. So this is as you
2:36:44can see here it is easier to extend and
2:36:47as well has better performance. So as
2:36:49you can see we are repeating the same
2:36:51thing but we are just changing the value
2:36:53and we are connecting all those
2:36:54conditions using the or in this scenario
2:36:56go and use the in operator. All right my
2:36:59friends. So that's all for the
2:37:00membership operators. Now we're going to
2:37:02speak about the last one the search
2:37:04operator. And here we have only one the
2:37:06like. And each time we're going to say
2:37:07like, I'm going to remind you to like
2:37:09this course. So let's
2:37:13go. Okay. So now what is like operator?
2:37:16You can use it in order to search for a
2:37:19pattern in your text. So if you have
2:37:21like a text or characters and you are
2:37:23searching for a specific pattern inside
2:37:25the text. So let's have an example in
2:37:27order to understand exactly what this
2:37:28means. Okay. So now if you don't have
2:37:30yet cafe, go grab one because you have
2:37:33to focus for this one. Now what we have
2:37:34to do is to define a pattern in is
2:37:36scale. In order to build a pattern we
2:37:38have like two special characters. If you
2:37:40use a percentage you are saying
2:37:42anything. So I'm going to accept
2:37:44anything. So it could be no characters
2:37:46at all or only one character or many
2:37:48characters. So I'm saying anything. Now
2:37:50if you use an underscore you are
2:37:52expecting to have exactly one thing like
2:37:55one character or one number. So it is
2:37:57exactly one. I know this sounds
2:37:59complicated but with an example you can
2:38:00understand this. And I can tell you the
2:38:02percentage is way more famous than the
2:38:05underscore. I rarely really use the
2:38:07underscore. So now let's say that I
2:38:09build the pattern like this. I say the
2:38:11first character must be M and then
2:38:13percentage. So here I'm saying in my
2:38:15text the first character must be an M
2:38:18and after the first character I really
2:38:20don't care. It could be any character,
2:38:22any number whatever. So this is the
2:38:24pattern and now let's have few values in
2:38:26order to say whether it's true or false.
2:38:27So now if you have the value Mariam. So
2:38:29now you can see the first character is
2:38:31an M which is perfect. This is exactly
2:38:34our pattern. The first character must be
2:38:36an M. And then after the M we got like
2:38:38four characters. So whatever it is
2:38:40totally fine. We can say Maria is
2:38:42fulfilling our pattern. And this is
2:38:44exactly what we are searching for. This
2:38:47value is fulfilling the condition. Okay.
2:38:48Now moving on to the next value we have
2:38:50m a. So here again the first character
2:38:53is an M which is perfect. And after that
2:38:55we have only one character a. Well we
2:38:57have say percentage. So it could be
2:38:59anything one character multiple
2:39:00characters a number or whatever. So
2:39:02that's why this value can match our
2:39:04pattern and we will see it in the
2:39:06outputs. Now moving on to the next value
2:39:08we have only one m which is as well
2:39:10totally fine because we are saying the
2:39:12first character must be an M and then
2:39:14followed with anything. Now moving on to
2:39:16the last scenario we have Emma. Now this
2:39:18is a problematic because the first
2:39:20character is an E and in our pattern we
2:39:23say it must start with M. So we don't
2:39:25have that in this word. The first
2:39:27character is an E. That's why this value
2:39:29is not fulfilling our pattern and SQL
2:39:32going to remove this value from the
2:39:33final results. So this is exactly what
2:39:35going to happen if you have this pattern
2:39:37and those values. Now let's have another
2:39:39scenario where you say you know what it
2:39:41could start with anything but for me it
2:39:43is very important the last two
2:39:45characters it must be an I and N. So we
2:39:48could start with anything but the last
2:39:50two must be an I and N. So let's take
2:39:52this value Martin going to go and check
2:39:54immediately the last two characters. So
2:39:56you can see we have an I and N and the
2:39:59first part marks it is fine. It could be
2:40:01anything. So this value is fulfilling
2:40:03the condition because the last two
2:40:05characters is an I and N. Now moving on
2:40:07to the next one we have vin. So v i n
2:40:10the last two characters is as well
2:40:12exactly what we are searching for. It is
2:40:14fulfilling the condition and we have
2:40:15before it like only v. So we say
2:40:17anything with a percentage. Right? Now
2:40:19one more we have in. So it is as well
2:40:22fulfilling the condition because before
2:40:23it we don't have anything. So en is
2:40:26fulfilling as well the condition. The
2:40:28percentage is always saying anything.
2:40:30Now moving on to the last scenario we
2:40:31have Jasmine. They are not the last two
2:40:33characters. The last two characters is
2:40:36an N and E and this is not matching our
2:40:39pattern and this why this value is not
2:40:41fulfilling our pattern and you will not
2:40:43see it in the results. So with that you
2:40:44can understand how we can search for
2:40:46something in a text using the like
2:40:48operator. Let's keep going. Now let's
2:40:50say that I have a percentage at the
2:40:51start and percentage at the end and in
2:40:54between I have only one character an R.
2:40:56If you define it like this you are
2:40:58saying if there is an R anywhere it is
2:41:01good enough whether it's beginning or at
2:41:03the end or in between then the condition
2:41:06is fulfilled. So if you have Maria you
2:41:07can see we have an R in the middle. So
2:41:10in the left side we have two characters
2:41:11on the right side we have two characters
2:41:13doesn't matter the main thing we have an
2:41:15R somewhere. So this going to be
2:41:16fulfilling the condition. Now moving on
2:41:18to better we have an R at the end and
2:41:20that is totally fine cuz we say at the
2:41:22right side it could be anything. So we
2:41:24have an R somewhere that's why it's
2:41:26going to fulfill the condition. Now we
2:41:28have another case where we say Ryan we
2:41:30have an R at the start. So we don't have
2:41:32anything before and we have after that
2:41:34like three characters which is totally
2:41:36fine. So we don't really care about the
2:41:38position of the R. It is totally
2:41:40acceptable to have an R anywhere. And if
2:41:42you have only an R that is as well good
2:41:44enough. You don't have anything before.
2:41:46you don't have anything after and that's
2:41:48okay. But if you have a word like Alice,
2:41:50we don't have any R inside it. So that's
2:41:52why this is the only case where you say
2:41:54we don't have here an R and it's going
2:41:57to remove this value from the results.
2:41:59And this way of searching of something
2:42:00is very famous. You don't care about the
2:42:02words before this word and after the
2:42:04word, right? So if you are searching for
2:42:06any word, you're going to say percentage
2:42:08before and percentage after. Now I know
2:42:11that we want to practice with the
2:42:12underscore. So let's say that I have two
2:42:14underscores and then the character B and
2:42:16then a percentage. So here what I'm
2:42:18saying there should be something in the
2:42:20first position. There should be as well
2:42:22something in the second position. Then
2:42:24the third position should be the
2:42:26character B must be exactly at this
2:42:28position and after that it could be
2:42:30anything. So we really don't care. I
2:42:31know this is a little bit complicated.
2:42:33Let's have an example. So we have the
2:42:34value alert. Now we can see the first
2:42:36position we have something the A. Then
2:42:39the second position we have as well
2:42:40something the L. So so far we are good
2:42:42at the pattern and then the third
2:42:44position we have B. So we have complete
2:42:46match and the rest the ERT whatever. So
2:42:49with that Albert is matching our
2:42:51pattern. Moving on to the next one rope.
2:42:53You can see the first character we have
2:42:54something which is good. We have the R.
2:42:57Then the second character we have an O.
2:42:59So it's not empty. We have something.
2:43:01And then the third one we have exactly
2:43:03B. And after that we don't have anything
2:43:05which is fine. So again this value going
2:43:07to fulfill the condition. So moving on
2:43:08to the next one. So it start with an A.
2:43:11So we have something in the first
2:43:12position. The second position we have as
2:43:14well something the B. But now the third
2:43:16character it is a problem. It is not P.
2:43:18We have an E. So that's why it is not
2:43:20following our pattern. And is going to
2:43:22go and remove it. Now moving on to last
2:43:24example we have an A and an N. So in the
2:43:26first position we have something. The
2:43:27second one as well. But the third one we
2:43:30don't have anything. We don't have a B.
2:43:32So that's why it's going to be removed.
2:43:33So my friends I know that was a lot.
2:43:35This is exactly how you build a pattern
2:43:37for the like operator using the
2:43:39percentage and the underscore. But the
2:43:41percentage is more famous. So this is
2:43:43exactly how it works. Let's go back to
2:43:45scale in order to have some examples.
2:43:47All right, let's start with this task.
2:43:48Find all customers whose first name
2:43:51starts with a capital M. So let's go and
2:43:53start searching for those informations.
2:43:55We're going to start as usual. Select
2:43:56star from customers. And now we have to
2:43:58go and build the filter logic. So we're
2:44:01going to say where. Now we are searching
2:44:03something in the first name. So we're
2:44:05going to say first name. So that means
2:44:07it is very important to start with an M
2:44:09and then the rest it doesn't matter. So
2:44:11we're going to use the like operator in
2:44:13order to search. And we're going to have
2:44:15our single quotes and we're going to
2:44:16start with the M. And it doesn't matter
2:44:19what comes after that. So for us it is
2:44:21very important that the first character
2:44:23is an M. Let's go and execute it. And
2:44:26with that we got our two customers Maria
2:44:28and Martin. And both of them starts with
2:44:30an M. So with that we have solved the
2:44:32task. It is very simple. Now we have the
2:44:34following task. Find all customers whose
2:44:37first name ends with an N. So let's go
2:44:40first and select all the customers here.
2:44:42And we need all those customers where
2:44:44they are having an N at the end. So we
2:44:46have John and as well Martin. So how we
2:44:49going to do it? The same thing where
2:44:50first name like since we are searching
2:44:53but here we're going to change the
2:44:54expression. So it must ends with an N as
2:44:57a last character. So before that it
2:44:59doesn't matter whether it is the first
2:45:01character. So it could be anything but
2:45:03the last character of the word should be
2:45:05an N. So that's it. Let's go and
2:45:07execute. And with that we got John and
2:45:10Martin because the last character is an
2:45:12N. It is very simple, right? It is all
2:45:14about where we're going to place this
2:45:16percentage. Okay. So now we have the
2:45:17next task. Find all customers whose
2:45:20first name contains an R. So here we
2:45:22don't have like specifications whether
2:45:24it is at the start or at the end.
2:45:26Somewhere there should be an R. So if
2:45:28you go and execute first without any
2:45:30wear condition you can see here for
2:45:31example Maria we have in the middle
2:45:33somewhere an R George George as well
2:45:36Martin and Peter at the end. So we have
2:45:38a lot of names with an R. So how we can
2:45:40search for that? We're going to stick
2:45:41with the where first name like and here
2:45:44our character going to be an R and we're
2:45:46going to put before it and after it a
2:45:48percentage. So it doesn't matter what is
2:45:50before it or after it somewhere there
2:45:53should be an R. So let's go and execute
2:45:55it. And with that we got all our
2:45:57customers where somewhere we have an R.
2:46:00As you can see it is very simple. If you
2:46:01put it before and after then you are
2:46:03open for more results. And this is
2:46:05usually used a lot in order to search
2:46:07for a value inside your database. All
2:46:10right. Now we're going to move to a
2:46:11funny one. It kind of says find all
2:46:13customers whose first name has an R in
2:46:15the third position for some reason. I
2:46:17don't know why. So let's go and execute
2:46:19our customers here without any filter.
2:46:21So it is for us very important to find
2:46:23the customers where in the third
2:46:25position we have an R like here for
2:46:27example Maria the third character is an
2:46:29R which is okay but with Peter over here
2:46:31it is not the third character so it is
2:46:34not fulfilling the condition. So how we
2:46:35going to write that? It going to say
2:46:37like this where the first name like but
2:46:40we have to write it now from the start.
2:46:41So the first position going to be an
2:46:43underscore the second position going to
2:46:45be as well an underscore and now in the
2:46:47third position going to have an R. So
2:46:49with that we make sure the third
2:46:51position and an R and before it we have
2:46:53two positions and now afterward it
2:46:55doesn't matter what comes after that it
2:46:57could be nothing or characters. So if
2:46:59you go and execute it like this we will
2:47:01get Maria and Martin and we will not get
2:47:04Peter because the R is not in the third
2:47:06position. So now if you don't do it
2:47:08correctly with the underscores let's go
2:47:09and remove one of them and execute. You
2:47:12will get nothing because we don't have
2:47:14any first name where the second position
2:47:16is an R. So you have to be very careful
2:47:18with this. All right my friends. So this
2:47:20is how you search inside your values.
2:47:22And with that we have covered all
2:47:24different groups of operators that you
2:47:26can use inside a wear clause. So with
2:47:28that you have learned how to filter your
2:47:30data using multiple operators that you
2:47:32can use inside the wear clause. So you
2:47:35can filter anything now in SQL. Now we
2:47:37will move to very interesting topic. You
2:47:39will learn how to combine your data from
2:47:41multiple tables. And here we have two
2:47:43main methods. The first one is SQL joins
2:47:45and the second set operators. And they
2:47:48are really big topics. So we're going to
2:47:49first focus on the SQL joins. And here
2:47:52we have a lot of things to cover. So now
2:47:54we are talking about the core of SQL. So
2:47:56let's
2:48:00go. All right. So now we have two
2:48:02tables, table A and table B. And the big
2:48:05question here is how to combine those
2:48:08two tables. What do we want exactly? Do
2:48:10you want to combine the rows or the
2:48:12columns? And now if you say I would like
2:48:14to combine the columns then we are
2:48:16talking about joining tables. So we're
2:48:19going to use joins in SQL. So now let's
2:48:21say that we are joining the table A with
2:48:23the table B and we start from the table
2:48:25A. So SQL going to take the columns and
2:48:27the rows of the table A and SQL going to
2:48:30call it the left table because we
2:48:32started from there and then we join it
2:48:34with the table B and SQL going to call
2:48:36the second table as the right table. And
2:48:38here what's going to happen? and SQL
2:48:39going to take the columns and the rows
2:48:41from the right table and put it side by
2:48:43side with the columns and rows of the
2:48:46table A. So we are like combining the
2:48:48columns we are putting them side by
2:48:50side. And now if you say you know what I
2:48:52don't want to do that I would like to
2:48:53combine the rows both of the tables
2:48:55having the same columns. I just want to
2:48:57stack them. So we are now talking about
2:49:00another methods. It is called the set
2:49:02operators. So here there is like no left
2:49:04and right. So since we started with the
2:49:06table A, the SQL going to take the
2:49:08columns and the rows of the table A and
2:49:10put it in the results. And then it's
2:49:11going to go to the second table, table B
2:49:13and it's going to take only the rows and
2:49:15put it below the rows of the the table
2:49:17A. So we are putting the rows beneath
2:49:20each others. We are doing like
2:49:21appending. So that means as we are using
2:49:24the set operators, we are combining the
2:49:26rows. Our table going to be longer but
2:49:28with the joins we are combining the
2:49:30columns side by side and we are getting
2:49:32wider table. But now for each methods
2:49:34there are different types. So now for
2:49:36example in order to do the joints we
2:49:38have four very famous types. We can do
2:49:40an inner join, full join, left join,
2:49:42right join. But of course there are more
2:49:44than that but those are the basics. And
2:49:46for the set methods we have as well
2:49:48types. We have the union, union all
2:49:50except and intersect. And for each
2:49:53methods there are like different rules.
2:49:55In order to join the tables we have to
2:49:57define the key columns between the two
2:49:59tables. Don't worry we're going to learn
2:50:00about that later. This is the
2:50:02requirement in order to join tables and
2:50:04the requirement of combining tables
2:50:06using the set operators the tables in
2:50:08your query should has the exact same
2:50:10number of columns but here you don't
2:50:12need any like key in order to combine
2:50:14the tables. So guys if you look at this
2:50:16in order to combine two tables first you
2:50:19have to decide do I want to combine the
2:50:21columns or the rows. So first you have
2:50:22to decide in the methods and after that
2:50:25you have different types on how exactly
2:50:27you're going to go and combine the data
2:50:28and of course there are rules that you
2:50:30have to follow. Now, of course, we're
2:50:31going to go and cover everything in the
2:50:32course, but now in this section, we're
2:50:34going to learn how we're going to
2:50:35combine the tables using the SQL joins.
2:50:38So, we're going to go and dive into this
2:50:44word. All right. So, now what is exactly
2:50:46SQL joins? Now, let's say that we have
2:50:48two tables. On the left table, we have
2:50:50the customer name. So, we have four
2:50:52customers. And on the right table, we
2:50:53have the country informations about the
2:50:55customer. And now we would like to query
2:50:57both of those informations the names and
2:51:00the countries. Now in order to query
2:51:02those two tables in one query first we
2:51:05have to connect them. And in order to
2:51:06connect those two tables we need a key a
2:51:09column that exist on the left and on the
2:51:11right sides. And by looking to this the
2:51:14common column here is the ID of the
2:51:16customer. Now once we connect those ids
2:51:18together we will be able to query those
2:51:21tables together and SQL going to start
2:51:23matching those ids. So for the ID number
2:51:25one, we will get the name Maria and the
2:51:28country Germany. And the ID2 is
2:51:30connecting John to USA. And now you can
2:51:33see the ID3 is not connectable. So we
2:51:35cannot connect it to the right side. But
2:51:37for the ID4, we can use it in order to
2:51:40connect Martin to Germany. So this is
2:51:42exactly what happens if you join two
2:51:44tables. You connect those two tables
2:51:46using a common column, a key like the
2:51:48ID. And once we have matching value, we
2:51:51can connect the two rows together. So
2:51:53this is what we mean with SQL
2:51:58joins. Now you might ask why do we need
2:52:01actually joins? Well, the first and very
2:52:03important reason is to recombine your
2:52:05data. So now usually in databases the
2:52:07data about something like the customers
2:52:10could be spreaded into multiple tables.
2:52:12Like we could have table called
2:52:14customers, another one where we have the
2:52:16customer addresses and a third table
2:52:18where you can find the orders of the
2:52:20customers and maybe another one where
2:52:22you can find the reviews of the
2:52:24customers. So as you can see the data of
2:52:26the customers is spreaded into like four
2:52:28tables. Now how about I would like to
2:52:30see all the data about the customers in
2:52:32one results. So I would like to see the
2:52:34complete big picture about our
2:52:36customers. What we can do, we can go and
2:52:39connect those four tables using the SQL
2:52:42joins. And once we do that in one query,
2:52:45I will be able to combine all those
2:52:47tables in one big results. And this is
2:52:49the most important reason why we use SQL
2:52:52joins in order to combine all the data
2:52:54about specific topic in order to see the
2:52:57big picture. Now, another reason why we
2:52:59use SQL joins is to do data enrichment.
2:53:02It is where I want to get an extra data
2:53:04and extra information. So let's say that
2:53:06you are querying the table customers and
2:53:08this is your main table the master
2:53:10table. So you are able to see all the
2:53:12data that you need but sometimes what
2:53:14happens you would like to get an extra
2:53:16information from another table like for
2:53:18example the zip codes of the countries.
2:53:20So you would like the help of another
2:53:22table we call it a reference table or
2:53:24sometimes lookup table where there is
2:53:26like one extra information that you
2:53:28would like to add it to your master
2:53:30table to the primary source of your
2:53:32data. So now what we can do we can join
2:53:33those two tables in order to enhance our
2:53:36table. So we are getting one extra
2:53:38relevant informations for the customers
2:53:40and this process we call it data
2:53:42enrichments. I'm getting an extra data
2:53:44for my main table. So this is another
2:53:47reason why we use joins. All right. So
2:53:49now so far we have used joins in order
2:53:51to get the data from two tables. But now
2:53:54there is another use case for the SQL
2:53:55joins. We use it in order to check the
2:53:58existence of your data in another table
2:54:01or maybe as well the not existence. So
2:54:03let's say that I have a table called
2:54:04customers and I'm working with this
2:54:06table and doing queries. But now I would
2:54:08like to check something. I would like to
2:54:10check whether our customers did order
2:54:12something. Now in order to check that I
2:54:15need the help of another table for
2:54:17example the table orders. So that means
2:54:19I'm using the table orders only for my
2:54:22check. So I don't want to get any extra
2:54:24data from the orders in my final
2:54:26results. I'm just using the table orders
2:54:28and we call in this table a lookup. So
2:54:30now what we can do we can connect those
2:54:32two tables together. And now based on
2:54:34the existence of the customers inside
2:54:36the second table the orders either the
2:54:39customer going to stay in the final
2:54:40results or going to be removed. So that
2:54:42means I'm filtering the data based on
2:54:44the join. And of course I can check as
2:54:46well the net existence. I would like to
2:54:48see in the final results all the
2:54:50customers that didn't order anything. So
2:54:53it is the same scenario. So my friends,
2:54:55those are the main three reasons why you
2:54:57use SQL joins. First, if you want to
2:55:00combine the data from multiple tables in
2:55:02one big picture. So I use join in order
2:55:04to get the data from different tables.
2:55:06The second use case, you are working
2:55:08with one table but you would like to get
2:55:10an extra information from another table.
2:55:12So you are doing it like something
2:55:14called data enrichments. And in the
2:55:16third scenario, we don't want to combine
2:55:18the data. We want just to join it with
2:55:20another table in order to do a check to
2:55:22check the existence of your records in
2:55:24another table. So this is why we need
2:55:26joins in
2:55:31SQL. Now there is like a lot of
2:55:33different possibilities on how to join
2:55:35tables, how to join the data. Now in
2:55:37order to make it easy to understand,
2:55:39we're going to visuals as like two
2:55:41circles. So we have the table A and a
2:55:43table B. The table A is on the left
2:55:45side. We call it the left table. And the
2:55:48table B going to be on the right side
2:55:49and we call it the right table. The side
2:55:51of the tables is very important. Now if
2:55:53you combine those two circles, you will
2:55:55get three different possibilities. The
2:55:57circles going to overlap. And here
2:55:59exactly where we can have the matching
2:56:01data between the two tables. So the data
2:56:04is available on the left and on the
2:56:05right. Or another possibility you want
2:56:07to get all the data from one of the
2:56:09tables. So you can get all the rows from
2:56:12one circle. And the third possibility
2:56:14you want to get only the unmatching data
2:56:16from one table. So if something exists
2:56:19in one table but not in the other table
2:56:21then we call it unmatching data. So
2:56:23those are the three scenarios that you
2:56:25have to ask yourself once you are
2:56:27combining tables and this can generate a
2:56:29lot of join types. So here we have like
2:56:31basic SQL joins those are the classical
2:56:34one and here depends on the scenario
2:56:36whether you want only matching all or
2:56:38all the rows from either left or right
2:56:40and we have advanced SQL joins where we
2:56:43focus on the unmatching data. Now we're
2:56:45going to go and cover all those types
2:56:46one by one. So we're going to start
2:56:48first with the basics and the first
2:56:49option that you have is to get all the
2:56:52data without joining tables. So let's
2:56:54see what this means.
2:56:58So what do we mean with no join? Well,
2:57:01we want to returns the data from two
2:57:03tables without combining them. So
2:57:05actually this is not a joint type
2:57:07because we are not combining anything.
2:57:09We just want to query the data from two
2:57:11tables. So that means from the table A
2:57:13we want to see all the rows everything
2:57:15and from the table B we want to see
2:57:17everything as well all the rows. So that
2:57:19means we want to see two results and
2:57:21there is no need to combine them. So
2:57:23let's see the syntax of that. So all
2:57:25what you have to do is very simple.
2:57:26Select star from table A and then
2:57:29semicolon and then start another query.
2:57:31Select star from table B. So that's it.
2:57:34And of course since we are not combining
2:57:35the data there will be no join in the
2:57:38syntax. So that's it. Let's go to SQL in
2:57:40order to do that. Okay. So now we have
2:57:42the following task. It says retrieve all
2:57:44data from customers and orders in two
2:57:46different results. So that sounds that
2:57:49we don't have to go and combine the
2:57:50tables together. And all what we can do
2:57:52is the following. We can go and select
2:57:54the data from the first table like this
2:57:57and then we make another query for the
2:58:00second table the orders and we don't
2:58:03have to go and combine them in one big
2:58:05query. We just use a very simple select
2:58:07statements in order to retrieve the
2:58:09data. So if you go and execute it since
2:58:11you have two separate queries you will
2:58:12get two results and with that in one
2:58:14result you will get all the customers
2:58:16and in the other result you will get all
2:58:18the orders and the data is not combined
2:58:21at all. So this is how you query two
2:58:23tables without combining them. So with
2:58:25that we are getting all the data without
2:58:27joining the tables. Now we're going to
2:58:28start talking about the first type of
2:58:30join the inner join where we start
2:58:32combining the data from two tables. So
2:58:34let's
2:58:38go. Okay. So now what is exactly an
2:58:40inner join? So this type going to return
2:58:43only the matching rows from both tables.
2:58:46So that means we will see in the output
2:58:48only matching rows. So now what do we
2:58:50need from the left table? We want only
2:58:52the matching data. So we will not get
2:58:54the whole circle of A. We will get only
2:58:57where we have an overlapping with the
2:58:59table B. So we want to see the data from
2:59:01A only if it exists in the table B. And
2:59:04now what do we need from the table B?
2:59:06Exactly the same thing only the matching
2:59:08data. So that means I don't want to see
2:59:10all the data from B. I want to see only
2:59:12the data in B that has a match from the
2:59:15table A from the left side. And with
2:59:17that you will get only the matching data
2:59:19from both tables. Now let's see how we
2:59:21can write that in SQL. So it is a usual
2:59:24query and always we start with a select.
2:59:26So we select for example all the columns
2:59:28from and here we specify the table name.
2:59:30So it's going to be a. So so far nothing
2:59:32new. But now we want to add as well the
2:59:34table B in the same query. In order to
2:59:36do that we use the keyword join and then
2:59:39we say table B the name of the table.
2:59:42And since we have like different types
2:59:43of joins in SQL, you can specify the
2:59:46type of the join before the keyword
2:59:48join. And if you don't specify anything,
2:59:50the default type is inner join. But my
2:59:53friends, the best practices is always
2:59:55mention the type. I don't like to skip
2:59:57the defaults because in projects maybe
2:59:59not everyone is aware of the defaults.
3:00:02So don't skip that. Always specify the
3:00:04type. So now what we're going to do,
3:00:05we're going to put the keyword inner
3:00:07before the join. And with that SQL going
3:00:09to know how to deal with the rows
3:00:11between two tables. But still we are not
3:00:13done there. We have to tell SQL how to
3:00:16combine the tables. And with that we use
3:00:18the keyword on. And after that you
3:00:20specify the join condition. And as we
3:00:22learned in order to join two tables we
3:00:25have to find out a common column in
3:00:27order to match the data. Right? And
3:00:29usually in scale they are the keys or
3:00:31ids. So the condition can be like this.
3:00:33the key from the table A must be equal
3:00:36to the key from the table B. So this is
3:00:38the join condition and using this join
3:00:41SQL can go and start matching the data
3:00:43from the left table and the right table.
3:00:45And there is one thing that is very
3:00:47important while you are joining the
3:00:49tables you have to understand about the
3:00:51order of the tables in your query. Now
3:00:53in the inner join the order of the
3:00:55tables doesn't really matter. So whether
3:00:57you start from A or you start from B it
3:00:59doesn't matter because you will get the
3:01:01same results. Both of the tables has the
3:01:03same priority and it doesn't matter
3:01:05where we start whether we say from A
3:01:07join B or we say from B join A we will
3:01:10get the exact same results. So in the
3:01:12inner join you don't have to worry about
3:01:14the order of the tables. So that's all
3:01:16about the inner join. Now let's go back
3:01:18to scale in order to practice. Okay. So
3:01:20now we have the following task and it
3:01:22says all customers along with their
3:01:24orders but only for customers who have
3:01:26placed an order. So my friends that
3:01:28means we need the data from the
3:01:30customers and from the orders from two
3:01:32tables and we have to put everything in
3:01:34one results. That means we have to join
3:01:36two tables. Now let's go and do it step
3:01:38by step. So we're going to go and say
3:01:40select star from customers and then we
3:01:43have to go and join it with the orders.
3:01:45We're going to say join orders. Now you
3:01:47have to go and specify the join type. Is
3:01:50it inner, left, full and so on. Well
3:01:52that's depend on the task. It says we
3:01:55want all customers but only for
3:01:57customers who have placed an order. So
3:01:59there is like condition right here. We
3:02:01don't want to see everything from the
3:02:03customer. We just want to see only the
3:02:05matching data only if the customers has
3:02:08an order in the orders table. And for
3:02:10that we can go and use the inner join.
3:02:13Of course if you can leave it like this
3:02:14you will get the same effects but I'm
3:02:16going to go and specify it like this
3:02:17inner join just to make it clear. We are
3:02:19speaking about the inner join. And after
3:02:21that we have to go and specify the join
3:02:23condition. So we have to go and find a
3:02:25common column between the customers and
3:02:27the orders. So how I usually do it I go
3:02:29and explore both of the tables. So I'm
3:02:32going to go and select everything from
3:02:33customers and as well
3:02:35everything from the orders. So let's go
3:02:39and execute. Now we're going to start
3:02:40searching where do we have a common
3:02:42column between those two tables. So we
3:02:44have the from the first table first
3:02:46name, country score and you don't find
3:02:48any of those informations in the second
3:02:50table. The only one is the ID. So the ID
3:02:53of the customer and the ID of the
3:02:55customer you can find it in the orders
3:02:57the second column here. So this is the
3:02:59common column between those two tables.
3:03:01And usually in databases we create ids
3:03:04exactly for this in order to connect
3:03:05tables. So it's really rarely that we're
3:03:08going to use like a country or score or
3:03:10first name in order to join tables. We
3:03:12usually use the ids. So let's go back to
3:03:14our query and use those two columns. So
3:03:17it's going to be the ID from the
3:03:19customers equal to the customer ID. So
3:03:22that's it. With that we have the
3:03:23condition we have decided on the type
3:03:25and we can go and execute it. Now you
3:03:27can see we are getting only three
3:03:29customers. Right? If you don't apply the
3:03:31inner join we can see that we have five
3:03:34customers. So that means actually we
3:03:36have two customers without any orders
3:03:39any matching data from the other table.
3:03:41And as well you can see very nicely we
3:03:43have now not only the columns from the
3:03:45customers but as well all the columns
3:03:48from the orders side by side. So with
3:03:50that we have combined the data and as
3:03:52well with that we have solved the task
3:03:53but we will not leave our query like
3:03:55this because it is not really good
3:03:56practices. What we have to do is to go
3:03:58and select only the columns that really
3:04:00make sense in our query because in many
3:04:02cases in your tables you will have a lot
3:04:04of columns that is not needed like for
3:04:06example if you check here you see we
3:04:08have the customer ID here and as well
3:04:10the customer ID over here. So it's like
3:04:12repetition and it's enough to see it
3:04:14only once. So what you have to do is to
3:04:15go and pick few columns that we want.
3:04:18For example, I'm going to start with the
3:04:19ID maybe the first name and that's all
3:04:22from the first table. Let's go and get
3:04:24the order ID and I don't want the
3:04:26customer ID again. So from the second
3:04:28table I'll get add the sales. So let's
3:04:30go and execute it. And with that you can
3:04:32see very nicely the customer's name and
3:04:34their orders with the sales. And now
3:04:36comes something very important.
3:04:37Sometimes if you have two tables you
3:04:39might have columns that having the same
3:04:42names. Like imagine the order ID in the
3:04:44table orders it's called ID. So that
3:04:46means we have the same name in both
3:04:48tables and this kind of makes SQL very
3:04:50confused. And here you will get an error
3:04:52tells you I really don't know what do
3:04:54you mean with the ID. Is it from the
3:04:56table customers or from the orders? So
3:04:58we have to tell SQL exactly from which
3:05:00table did this column come from. So in
3:05:02SQL in order to do that what we do
3:05:04before the column name you write again
3:05:07the table name the customers and then
3:05:09you make a dot and now we are telling
3:05:11SQL this column the ID it comes from the
3:05:14table customers and SQL will not be
3:05:16confused about it and it's going to go
3:05:18and get the ID from the customers. And
3:05:20for the second id you can go over here
3:05:22and as well before it you say orders do
3:05:25id so that knows okay this ID come from
3:05:28the orders and the other one comes from
3:05:30the customers and it is always good
3:05:32practice especially if you are joining
3:05:34tables to always assign for each column
3:05:36a table because after a while if you
3:05:38open your query and you see okay the
3:05:40sales does the sales come from the
3:05:42customers or the orders and if you have
3:05:44a long list of columns it's going to be
3:05:46really confusing so that's why we
3:05:47consider it best practices if you always
3:05:50assign for each column the table name
3:05:52especially if you are doing joins. So
3:05:54it's going to be like this. But of
3:05:55course if you have like only one table
3:05:57it's clear that all the columns in the
3:05:59select comes from this table. But since
3:06:01here we are dealing with multiple tables
3:06:03it is good to show it like this. And of
3:06:04course here we don't have the ID. We
3:06:05have the order ID and the same thing for
3:06:08the join condition. So the ID from here
3:06:10comes from the customers and the
3:06:12customer ID come from the orders. So now
3:06:15it is clear for everyone which column
3:06:17come from which table. But now you might
3:06:19say you know what each time I have to
3:06:21write the customers this is very long
3:06:23name and sometimes in real projects
3:06:25you're going to see tables that has
3:06:26really long name and it's going to be
3:06:28really annoying to add it each time
3:06:30before each column right so instead of
3:06:32that we can go and assign aliases for
3:06:35the tables but only for the columns so
3:06:37usually we go over here and say as and
3:06:39maybe you can go and use only one
3:06:41character like the first character C.
3:06:43And now instead of saying customers you
3:06:45can go over here and say C. The same
3:06:46thing for the second column and as well
3:06:49over here. And you can use now the C in
3:06:51everywhere in your query. The same thing
3:06:53for the orders. You can go over here and
3:06:54say has O. And now instead of orders you
3:06:57say
3:06:59O on here. And now it is very easily to
3:07:02see those two columns comes from the C
3:07:05that means the customers and those two
3:07:06columns comes from the O the orders.
3:07:09Those are the best practices as you are
3:07:10joining tables together in SQL. And of
3:07:13course with that we have solved the
3:07:14task. And about the order of the tables,
3:07:15it doesn't matter where do you start. So
3:07:17for example, if you take the orders here
3:07:19and put it in the join and get the
3:07:22orders in the from. So I just switch the
3:07:24tables and execute it, you will get the
3:07:26exact same results. So if you are doing
3:07:28inner join between two tables, don't
3:07:30worry about the order of the tables.
3:07:32Okay. So now let's go and instant
3:07:33exactly how executed the inner join.
3:07:36Okay. So now again here we have our
3:07:38query. Then we have the two tables
3:07:40customers and orders. And here we have
3:07:42the ID where we are joining the data. So
3:07:44this is the ID from the table customers
3:07:46and this is the customer ID that we have
3:07:48in the orders. Now let's see how SQL can
3:07:50execute this. So we are saying I would
3:07:52like to see the ID and the first name.
3:07:54So we will get the ID, the first name
3:07:56from the table customers and we would
3:07:58like to get the order ID and as well the
3:08:02sales from the table orders. So our
3:08:04result going to focus on those four
3:08:06columns. Now the data should be joined
3:08:08between those two tables using the inner
3:08:09join and SQL going to start from the
3:08:11left table from the customers because we
3:08:13say from customers. So it's going to
3:08:16start matching the ID from the left
3:08:18table with the right table. So it's
3:08:19going to say okay is there a match from
3:08:21the first record from the first order?
3:08:24Well yes it is the same ID and then SQL
3:08:27going to say okay that condition is
3:08:28fulfilled and we are allowed to see the
3:08:30data. So the data will be presented in
3:08:33the output. So we're going to have the
3:08:35ID Maria and the order ID from Maria and
3:08:39the sales of this order. So there is a
3:08:41match. Then SQL going to go to the
3:08:43second record. Well, we don't have a
3:08:45match. The third we don't have match.
3:08:47And so on for the last one. So we have
3:08:49only one match for this ID. Then SQL
3:08:51going to go again to the customers and
3:08:53pick the second one and start matching
3:08:55again with the first order. Do we have a
3:08:57match? Well, no. Then it's going to go
3:08:59to the second. Well, now we have a
3:09:01match. So SQL going to be happy. the
3:09:03condition is fulfilled and we will see
3:09:05the results. So we're going to see the
3:09:07first name and as well the order
3:09:08information for this customer in the
3:09:10output. It's going to keep searching. So
3:09:13we don't have a match as well here. So
3:09:15that's it. Now for the third customer as
3:09:17well from the start there match no to
3:09:20the second to the third and here we have
3:09:21a match. So it's going to go and show
3:09:23this informations since there is a
3:09:26match. So the customer three George with
3:09:28the order from this customer order ID
3:09:31and the sales as well in the output. Now
3:09:33it's going to go and keep continuing the
3:09:35search. Well, we don't have any match.
3:09:37Then it's still going to go to the
3:09:39fourth customer and start matching. Do
3:09:41we have here an ID? Do we have here a
3:09:43match? Well, no. Then the second, third,
3:09:45and fourth. We don't have any order for
3:09:48this ID. There is no match at all. And
3:09:50since we are saying inner join then SQL
3:09:52will not allow to show the data of this
3:09:55customer in the results. There is no
3:09:57match and SQL going to totally ignore
3:09:59this customer. Then we're going to go to
3:10:01the last one and start as well matching
3:10:03this ID with the orders. Well, there is
3:10:05no match as well. SQL going to go and
3:10:07exclude this user from the results. So
3:10:10this is exactly how the inner join
3:10:12works. it start from the left side and
3:10:14start matching the data on the right
3:10:16side and only if there is match the
3:10:18result going to be presented in the
3:10:19output and this is exactly why we are
3:10:21getting this results and how the inner
3:10:23join works. So now if you look again to
3:10:25the reasons why we are joining tables we
3:10:27can say we can use the inner join in
3:10:29order to recombine the multiple tables
3:10:32into one big picture. So the first use
3:10:34case and as well we can use the inner
3:10:36join in order to filter the data. So
3:10:38since we are saying only the matching
3:10:40data that means we are filtering the
3:10:42data we are checking the existence of
3:10:44the records in another table. So you can
3:10:46use inner join either to combine data
3:10:49from multiple tables or you can use it
3:10:51as well only for filtering purposes only
3:10:53to check the existence of your rows. So
3:10:55this is usually the two use cases of
3:10:57inner. All right. So that's all about
3:10:59the first type the inner join. Next
3:11:01we're going to talk about the left join.
3:11:03So we're going to focus on the left
3:11:04side. So let's go.
3:11:09Okay. So now what is exactly left join?
3:11:11This type going to returns all the rows
3:11:14from the left table and only the
3:11:16matching from the right table. So now if
3:11:18you look again to our two circles A and
3:11:20B. What do we need from the left table?
3:11:22We want to see everything all the rows
3:11:25all the data. So that means we will get
3:11:27a full circle. And now from the right
3:11:29table we want to get only the matching
3:11:32data. So that means we don't want to see
3:11:34everything from the table B. We want to
3:11:36see only the records that has match to
3:11:38the table A. So that means my friends
3:11:41the left table has here more priority.
3:11:43This is the primary source of your data.
3:11:46The main source we cannot miss anything.
3:11:48This is very important. We want to see
3:11:49all the data. But from the table B, it
3:11:51is a secondary source of data and we are
3:11:54joining it only to get an additional
3:11:56data. So I don't want everything. I want
3:11:58only the data that has matched to the
3:12:00lift table. So this is what we mean with
3:12:02a lift join. Now if you look to the
3:12:04syntax it's going to be very similar to
3:12:05the inner join. So we start from the
3:12:07left table the A. Then we say left join
3:12:10the right table B and then the same
3:12:12condition using keys. So here we just
3:12:14switch the type. Instead of inner we
3:12:16have now left. But now here with the
3:12:18syntax we need to be very careful. The
3:12:20order of the tables now is very
3:12:22important. You have to start from the
3:12:24correct table. So you have to mention
3:12:26the left table exactly in the from
3:12:28clause and then you join it with the
3:12:31right table. So in the join you have to
3:12:33specify the right table. If you don't do
3:12:35it like this then you will not get all
3:12:37the data from a and you will not get the
3:12:39results that you are expecting. So this
3:12:41is what we mean with the left join.
3:12:42Let's go back to scale in order to
3:12:44practice. All right. So now we have the
3:12:46following task. It says get all
3:12:48customers along with their orders
3:12:50including those without orders. So again
3:12:53here we need the data from two tables
3:12:55the customers and orders and we want
3:12:57everything in one result. So that means
3:12:58we have to go and join the data. And now
3:13:00the task says includes those without
3:13:03orders. So that means I want to see
3:13:05everything the matching data and the
3:13:07unmatching data from the table
3:13:09customers. And by looking to our query
3:13:11this is not working because we are not
3:13:13getting everything right. We are getting
3:13:14only the customers that has match in the
3:13:17table orders. And this is not of course
3:13:19fulfilling the task. So now if you read
3:13:21the task you can understand the main
3:13:23table here is the customers. We are not
3:13:25speaking about to see all the orders and
3:13:28not missing any order and the orders
3:13:29here is only for additional
3:13:31informations. So now in order to not
3:13:32lose any data for the customers we make
3:13:35sure we start from the table customers.
3:13:37So that means now the customers on the
3:13:39left side and now after that instead of
3:13:41inner join this is not good thing for
3:13:43this task. We're going to say left join
3:13:45and with that we guarantee we will get
3:13:47all the data from the customers. Now we
3:13:49say left join orders and of course the
3:13:52condition going to stay like this. This
3:13:54is how we are connecting the two tables.
3:13:56So actually that's it. Let's go and
3:13:58execute it. And now by looking to the
3:14:00result you can see that we have now five
3:14:02customers even the customers that didn't
3:14:04place any orders. So you can see Martin
3:14:06and Peter they don't have any order ID.
3:14:09So that means they didn't order
3:14:11anything. And as you can see is showing
3:14:13us nulls when there is no match. So with
3:14:16that we have solved the task. Now my
3:14:17friends one more thing as I told you the
3:14:19order of the tables is very important
3:14:21because the customer is now the left
3:14:24table because you start from it and the
3:14:26second table the orders is the right
3:14:28table. Now if you go and switch them
3:14:30like this. So we start from the orders
3:14:32and then join it with the customers and
3:14:35you go execute it you will not get all
3:14:37the customers and of course the task is
3:14:39now not solved. So as you can see you
3:14:40are getting now completely different
3:14:42result if you go and switch the tables.
3:14:44So be careful where you start and how
3:14:46you join the tables in order to get the
3:14:48effects that you want. All right. So now
3:14:50I'm going to put everything back like
3:14:52before. Now let's go and understand how
3:14:54is exactly executed this query. Okay. So
3:14:57now again we have the data from
3:14:58customers and orders and this time we
3:15:00are doing the lift join. So now let's
3:15:02see how is going to do it. So going to
3:15:04say okay we need the ID and the first
3:15:05name and we will get that as well in the
3:15:08results and from the right table we need
3:15:09only those two informations the order ID
3:15:12and the sales in the output. So those
3:15:14are the columns that we need. So now SQL
3:15:16in the left join going to do it a little
3:15:18bit differently. It's going to start as
3:15:19well from the lift table from the
3:15:21customers. But this time going to go and
3:15:24immediately put the result in the output
3:15:27without like trying to match anything
3:15:29and to check whether the data exist or
3:15:30not because it doesn't matter not doing
3:15:33any validation whether the customer
3:15:35exist in the orders. Since it's lift
3:15:37join is still going to show all the data
3:15:38from the lift table. So there will be
3:15:41like no check. But now as a next step in
3:15:43order to get the order ID and the sales
3:15:45SQL will start searching. So SQL going
3:15:47to go over here and start searching
3:15:49where do we have a customer with this
3:15:51ID? Well, it's going to be the first
3:15:52order. We're going to get the order ID
3:15:54and as well the sales informations and
3:15:57we will see that in the output. So
3:15:59that's it for the first one. Now it's
3:16:01going to go to the second row and the
3:16:02same thing going to happen immediately.
3:16:04The SQL going to go and put the result
3:16:06in the output without checking anything.
3:16:09And then in order to get the order data,
3:16:11it will start searching for this ID. So
3:16:14we have it here in the second row. We
3:16:15have the order ID and the sales. And
3:16:18it's still going to put those results to
3:16:19the output. So the search for the third
3:16:22one immediately going to put everything
3:16:24in the output. And then start searching
3:16:27for orders with this ID. We have it over
3:16:30here. So this order belongs to the user
3:16:32ID number three. So far we are getting
3:16:34the same result as the inner joint. But
3:16:36we are not done yet. Now exactly count
3:16:38the difference this guy going to go and
3:16:40get Martin and put it immediately in the
3:16:42output and start searching for an order
3:16:44with this ID. So do we have any order
3:16:47with the ID number four? Well, we don't
3:16:49have anything this time. SQL of course
3:16:51will not go and exclude the ID number
3:16:54four. It's going to leave it. But in SQL
3:16:56if there is no match, we still have to
3:16:58have something in the output. So SQL
3:17:00going to go and say the output going to
3:17:03be null like this. We don't know it is
3:17:06unknown. And the same thing for the
3:17:08sales. So in the lift join if there is
3:17:09no match you will see nulls. The same
3:17:11thing for the next customer for better.
3:17:14So SQL will go and put the result
3:17:16immediately in the output and then start
3:17:18searching the orders. So do we have
3:17:20anything for the ID number five? We
3:17:22don't have anything. That's why SQL
3:17:25going to go and present nulls as well in
3:17:27the output. And that's why you saw nulls
3:17:29in the output because those customers
3:17:32don't have any orders. So this is
3:17:34exactly the effect of the lift join. you
3:17:35will get everything from the lift table
3:17:37and only the matching stuff on the right
3:17:39side and if there is something not
3:17:41matching you will get nulls. So that's
3:17:43it is this is how scale execute the left
3:17:45join okay so now back to this use cases
3:17:47of joins if I think about lift join I
3:17:50can use it in order to recombine data in
3:17:52order to build this big picture and as
3:17:54well in the second use case where we use
3:17:56it in order to get an extra information
3:17:58from another table. So we have a main
3:18:00table and secondary table. So we use it
3:18:03for both use cases and as well in the
3:18:05third use case only with a twist that
3:18:07we're going to learn later. So that's
3:18:08all about the left join. Now we have
3:18:10another type that is exactly the
3:18:12opposite of the lift join. We have the
3:18:14right join. So now let's understand what
3:18:17this
3:18:20means. Okay. So now what is exactly
3:18:23right join? This is the total opposite
3:18:25of the left join. So this tag going to
3:18:27returns all the rows from the right
3:18:29table and only the matching from the
3:18:32left table. So here the main table the
3:18:34main focus is the right table. So SQL
3:18:36going to get you all the rows everything
3:18:38from the table B the right table but
3:18:41from the left side we will get only the
3:18:43matching data. So that means in the left
3:18:45sides you will get only the data that
3:18:47has a match on the right side and with
3:18:50that the right table going to be the
3:18:52primary the main source of your data. So
3:18:54it is very important table but the lift
3:18:56table is not that important. You are
3:18:58just joining it in order to get
3:19:00additional data. So again about the
3:19:02syntax it's not that crazy. All what you
3:19:04have to do is to change the join type.
3:19:06So instead of left you say right join
3:19:09and again here the order of the tables
3:19:11is very important because the side here
3:19:13makes a difference. So we start from the
3:19:15left table A and then right join it to
3:19:18the table B. So it sounds very similar
3:19:20to the left join. We are just switching
3:19:22things. Now let's go back to scale. in
3:19:23order to practice. Okay my friends, so
3:19:25now we have the following task and it
3:19:27says get all customers along with their
3:19:29orders including orders without matching
3:19:33customers. So again we have the
3:19:35customers and the orders and we are
3:19:36doing the join but here the condition is
3:19:38different. We want to see all the orders
3:19:41even if they don't have a matching
3:19:43customer. So that means I would like to
3:19:45see everything from the table orders and
3:19:47the customers table here is only like
3:19:49supporting and helping. So the main
3:19:51table that we are focusing on is in the
3:19:53orders. We want to see everything and
3:19:55from the customers only the matching and
3:19:57if you are looking currently to the
3:19:59results you can see we are seeing only
3:20:01three orders right but in the original
3:20:04table if you go back over here you can
3:20:06see that we have four orders. So we are
3:20:08currently using this query not seeing
3:20:11all the orders. So now how we going to
3:20:12solve it? If you start from the table
3:20:14customers you can say you know what
3:20:16instead of left join we're going to say
3:20:18right join. And with that you're going
3:20:20to guarantee you will get everything
3:20:22from the table orders. But now the left
3:20:25table the customers is not that
3:20:26important and you will see the data of
3:20:28the customers only if there is a match.
3:20:30So doing the right join like this
3:20:32guaranteed to see everything whether
3:20:34there is match or no match. Now if you
3:20:36go and execute it you can see on the
3:20:38right side the order ID and the sales
3:20:40and we can see now all the orders and on
3:20:42the left side the ID and the first name.
3:20:45We are seeing only the customers if they
3:20:47did order something. And for the orders
3:20:49without a known customer, we are getting
3:20:51nulls. So with us, you have solved the
3:20:53task using the right join. So now my
3:20:55friends, you have to go and solve this
3:20:56task to get the exact same results. But
3:20:59you are allowed to use only the left
3:21:01join. So you are not allowed to use the
3:21:03right join. So now go pause the video,
3:21:05solve the task and meet you
3:21:08[Music]
3:21:11soon. Now my friends, in SQL there is
3:21:14always alternatives on how to solve a
3:21:16task. So now if you want to get all the
3:21:18data from B and only the matching from
3:21:20A, you can do it like we have done using
3:21:22the right join. But if you go and switch
3:21:24the sides and you make the table B as a
3:21:27left table and the table A as a right
3:21:29table, you can do that of course in SQL.
3:21:31But you have to switch the join type. So
3:21:33instead of right, we have to use left
3:21:35now since the B table now on the left
3:21:38side and as well you have to switch the
3:21:40order. So you start from the B table and
3:21:43then you say left join the A table. and
3:21:46of course the same join condition. And
3:21:47if you do that, you will get the exact
3:21:49same result as the left query. So if you
3:21:52just switch the tables and as well
3:21:54switch the join type, you can get the
3:21:56same results. And to be honest, my
3:21:58friends, I don't like the right join.
3:22:00It's just in the last 10 years, I always
3:22:02tend to start from a table and then use
3:22:04a left join. And from my point of view,
3:22:06the left join is way more famous than
3:22:09the right join. And I think I never used
3:22:11a query where I'm using a right join. So
3:22:14my advice for you always try to skip the
3:22:16right join and stick with the left join
3:22:18just get the order of the tables in the
3:22:20query correct and you will get the same
3:22:22results. So with that you know an
3:22:23alternative for the right join. Now all
3:22:25what you have to do is to go and switch
3:22:26the right to left. Uh this is not enough
3:22:29because if I go and execute it. So now
3:22:31all what I have to do is to go and
3:22:33switch the tables like this. So we start
3:22:36from the table orders because I want to
3:22:38see everything from the orders and then
3:22:40lift join it with the customers. And of
3:22:42course we don't have to change anything
3:22:43here. It doesn't matter the order
3:22:45because we have an equal operator here.
3:22:47What is very important here is where you
3:22:49start from which table and what is the
3:22:52table that you are joining with. So if
3:22:54you go and execute it, you will get the
3:22:56exact same results. So now I'm seeing
3:22:58all the orders. I'm not missing anything
3:23:00and only the matching customers. And I
3:23:03prefer this way solving this task
3:23:05instead of using the right join. All
3:23:06right. So that's all about the right
3:23:08join. Next we're going to combine
3:23:10everything. We're going to talk about
3:23:11the full join. So let's
3:23:16go. Okay. So now what is exactly a full
3:23:19join? If you use it, SQL returns
3:23:21everything all the rows from both
3:23:23tables. So now if you check again our
3:23:25circles from the left table, we want to
3:23:27get everything all the rows. So you will
3:23:30get the whole circle and as well from
3:23:32the right table you want to get
3:23:33everything all the rows the whole
3:23:35circle. So that you want to get
3:23:37everything the matching the unmatching
3:23:39all the data from left and right. Now
3:23:41let's check the syntax. It's going to be
3:23:43very simple. The joint type here going
3:23:44to be a full join. And the full join it
3:23:47is very similar to the inner join. You
3:23:49remember the order of the tables is not
3:23:51important at all. So there is here no
3:23:53main table and secondary table. Both of
3:23:55the tables are important and it doesn't
3:23:58matter in your query where you start.
3:23:59You can start from A full join B or you
3:24:02can start from B then full join A. you
3:24:04will get the exact same results. It
3:24:06sounds simple. Let's go to SQL and
3:24:08practice the full join. All right. So
3:24:10now we have the following task and it
3:24:11says get all customers and all orders
3:24:14even if there is no match. So now again
3:24:16we need the data from customers and
3:24:18orders. But now of course which type
3:24:20we're going to use? It says even if
3:24:22there is no match but it didn't say no
3:24:24match from orders or customers. So you
3:24:26can understand from this task we are not
3:24:28focusing only on the orders or the
3:24:30customers. Both of them are equally
3:24:32important and we need all the data. So
3:24:34that means we need all the data from
3:24:36left, all the data from right and we can
3:24:38go and use the full join. So now we have
3:24:40this query over here. We are starting
3:24:42from customers and then joining to
3:24:44orders. But now instead of having left,
3:24:46we're going to say full join. So now
3:24:48let's go and just execute it. Now if you
3:24:50are looking to the left side, you can
3:24:52see we are getting all the customers,
3:24:54right? So we have our five customers and
3:24:56if you are looking to the right, you can
3:24:58see all our orders. So with that we have
3:25:00everything from left and everything from
3:25:02right and the matching data is just side
3:25:05by side in the results and if there is
3:25:07no match we are getting nulls. So
3:25:08actually with that we have solved the
3:25:10task and again it doesn't matter how you
3:25:12start. You can start from the orders and
3:25:14then join it to the customers and you
3:25:16will get the exact same results. So you
3:25:19are getting exactly the same data. Now
3:25:20let's go and understand exactly how is
3:25:22executed the full join. Okay again we
3:25:25have the data of the customers and the
3:25:26orders and our full join. So now we're
3:25:28still going to identify those columns
3:25:30that we want to see in the results. So
3:25:32the ID and the first name, the order ID
3:25:34and the sales informations to the
3:25:36output. Now it's still going to start
3:25:38from the left table since it is started
3:25:39with the customers. It's still going to
3:25:41take simply everything from the left
3:25:43table and present it in the output.
3:25:45Since it is full join, we want to see
3:25:46all the data from the left side. And now
3:25:48start searching for matches from the
3:25:50right table. So let's start with the
3:25:52first customer. And as usual, we will
3:25:54get the order from the customer number
3:25:57one. And the same thing for the second
3:25:59customer, we have as well here match. So
3:26:01we will get as well. It's like that lift
3:26:03join. And for the third one, we have as
3:26:05well a match. And we're going to have it
3:26:07like this. And since we don't have
3:26:09orders for those two customers, we will
3:26:11get as well nulls in the outputs. So
3:26:14scale going to mark it with null. The
3:26:16same thing over here. And as well for
3:26:18the last customer. So we will get nulls
3:26:20for those two customers. And now of
3:26:22course SQL will not stop here otherwise
3:26:24we will get a left join effect. Now SQL
3:26:27going to start looking at the right side
3:26:29to find any order that is not in the
3:26:31output. So SQL going to see okay the
3:26:33first order is in the output. The second
3:26:35one is as well in the output. The third
3:26:37but the fourth one is not in the
3:26:39results. So SQL going to take this
3:26:41result and put it in the output. So this
3:26:44order has no match at all from the left
3:26:46side. And with that if you are looking
3:26:48to the right side you can see SQL going
3:26:50to be happy because we have all the
3:26:52orders from the right table. And of
3:26:54course SQL will not leave it like this.
3:26:56Instead of that SQL going to show nulls
3:26:58on the left side. So there is no ID and
3:27:00there is no first name. So this is
3:27:02exactly why we got this results. And
3:27:04this is how SQL executed the full join.
3:27:07Okay. Okay. So now if you are looking to
3:27:08the use cases I can say you can use the
3:27:10full join in order as well to recombine
3:27:13the data from multiple tables if you
3:27:15don't want to miss anything from all
3:27:17four tables all data the matching and
3:27:19unmatching data but I don't use it
3:27:21usually for data enrichment for the
3:27:23second use case and where we can use the
3:27:25full join is in the last use case as
3:27:27well but with a little twist that we're
3:27:29going to learn later. So this is mainly
3:27:31where we can use the full join. All
3:27:33right. So with that we have covered the
3:27:35basic types of joins inner, left, right
3:27:37and full join. Those are the classical
3:27:39joins on how to combine two
3:27:44tables. Now we're going to start talking
3:27:46about the advanced SQL joins. And now
3:27:48we're going to cover the first part the
3:27:50lift anti- join. So let's see what this
3:27:56means. Okay. So now what is exactly a
3:27:58lift anti- join? Now in this mechanism
3:28:00we want to return rows from the left
3:28:03side the left table that has no match in
3:28:06the right table. So now by looking to
3:28:07our two circles from the left table we
3:28:10want to see only the unmatching rows. So
3:28:13only rows that exist in table A but it
3:28:16don't exist in the table B. So if there
3:28:18is like matching data we don't want to
3:28:20see it. And now from the right table we
3:28:22don't want anything. We don't want any
3:28:24data. So that means the only source of
3:28:26your data going to be the left table.
3:28:28And from the right table we don't need
3:28:29any data. We are just joining the tables
3:28:31to do a check to filter the data. So now
3:28:34for the syntax this can be interesting.
3:28:36We don't have a special type called left
3:28:38anti- join. At least in the SQL server
3:28:40we still can create this effect. Since
3:28:42we are saying left we can use the type
3:28:44left join and then as usual the join
3:28:46condition with the keys. But now if you
3:28:48leave it like this you will get the
3:28:50effect of the lift join. And we don't
3:28:52want that because with the lift join you
3:28:53will get the complete circle from the
3:28:55lift table. But now in order to remove
3:28:57the matching data this overlapping in
3:28:59the middle what we can do we can use a
3:29:01filter and in order to filter the data
3:29:03we use the wear clause. So now in order
3:29:06to get rid of the matching data we can
3:29:08take the key from the right table and we
3:29:11say the key must be null. So if the key
3:29:14is null so that means there is no match
3:29:16on the right side. And if you do it like
3:29:18this you will get the effect of the left
3:29:21anti-join only the data in the left that
3:29:23has no match on the right. So now let's
3:29:26go in scale and create this effect.
3:29:28Okay. So now we have the following task
3:29:29and it says get all customers who
3:29:31haven't placed any order. So now by
3:29:33looking to this query clearly we are
3:29:35focusing on the table customers but we
3:29:37want to see the customers that didn't
3:29:39order anything. So they are in our
3:29:41database but the customers are inactive.
3:29:44Now there are like different ways on how
3:29:45to solve this task but we're going to
3:29:47solve it using the joins. Now let's go
3:29:49and start by just writing a very simple
3:29:51query where we are selecting everything
3:29:54from the table customers. Now you can
3:29:56see this is our five customers. And now
3:29:58I want to check which of those customers
3:30:00didn't order anything yet. Now since we
3:30:02are talking about the orders, we can go
3:30:04and join it with the table orders. So
3:30:06we're going to say lift join the table
3:30:08orders as all and then we're going to go
3:30:10and connect the tables using the ids
3:30:12with the customer ID. So now if you go
3:30:15and execute it now we are still seeing
3:30:17all the customers because we are using
3:30:19the lift join and now we can see the
3:30:21orders informations of each customer and
3:30:23you can see immediately those two
3:30:25customers didn't order anything because
3:30:28we are seeing here nulls right so they
3:30:30are empty there is no orders now we can
3:30:32use this information in order to filter
3:30:35the data I just want to see Martin and
3:30:37Peter so what you can do we can go and
3:30:39say where and all what you have to do is
3:30:42to take the key that we are using in
3:30:43order to join in the tables this is this
3:30:45one over here and say this must be null
3:30:48so is null so if you see it like this
3:30:51that means you want to see the data if
3:30:53the customer ID is null so let's go and
3:30:56execute it perfect now you are getting
3:30:58the customers who haven't order anything
3:31:01and this is exactly the effect that we
3:31:03wanted the left anti-join we are getting
3:31:05the data from the left side where there
3:31:08are no match on the right side so you
3:31:10have always to do it in two steps first
3:31:12join the data as you normally do using
3:31:14the classical joins the lift join and
3:31:17then the second step you go and use a
3:31:19filter using the wear clause if you do
3:31:21it like this you can check for not
3:31:23existence and with that we are getting
3:31:25the effect of the left anti-join so
3:31:27that's it okay so now if you are looking
3:31:29to this picture I think you already know
3:31:31where we use the lift anti- join we're
3:31:33going to use it only in the last use
3:31:34case where we are checking the existence
3:31:37so if you use the lift join together
3:31:38with the where you can check for the
3:31:40notexistence of your data in another
3:31:42table so This is exactly for this
3:31:44scenario. All right. So that's all about
3:31:46the left anti- join. Now we're going to
3:31:48speak about the exact opposite of that.
3:31:50We will cover the right anti- join. So
3:31:53it's going to be very similar but we are
3:31:55just switching sides. So let's
3:31:59go. Okay. So now what is exactly the
3:32:02right anti- join? Well, it is the
3:32:04opposite of the left anti- join. So we
3:32:06want to return the rows from the right
3:32:09table that has no match in the left
3:32:11table. So again if you are looking to
3:32:13our two circles. Now what is important
3:32:15is the right table. We want to see only
3:32:17the unmatching rows from the right
3:32:20table. So only the rows that exist in B
3:32:23but not in A. And from the left table we
3:32:25don't need anything. So no data is
3:32:27needed and that means the only source of
3:32:29data comes from the right table and you
3:32:32are using the left table as a filter as
3:32:34a lookup just in order to check the
3:32:36existence. So now the syntax of that
3:32:38going to be very similar to the left
3:32:39anti- join. So we don't have a special
3:32:41type called right anti-join. We have to
3:32:44use the classical one the right join.
3:32:46But if you do that you will get
3:32:47everything from the right table. And now
3:32:49in order to get rid of the matching data
3:32:51in the middle we use a filter. We use
3:32:53the wear clause where we say we are
3:32:55interested only on the unmatching data.
3:32:58So we take the key from the left table
3:33:00and we say the key from left is null.
3:33:02And if you do that you will get rid of
3:33:04any matching data. Is null means there
3:33:06is no match. And again here the same
3:33:08thing the order of the tables is very
3:33:10important since here we are talking
3:33:12about sides and you have to do it
3:33:14correctly. Okay. So now the task says
3:33:16get all orders without matching
3:33:19customers. So now it is exactly the
3:33:21opposite. We want to see all the orders
3:33:23that don't have a valid customer. So
3:33:25this is really bad scenario. You have in
3:33:27your business orders without a valid
3:33:29customers. So let's see how we can
3:33:30discover that using SQL joins. Now as
3:33:32you can see we are focusing completely
3:33:34on the orders. It's not the customers
3:33:36anymore. And we want to see only the
3:33:38orders where there is no match with the
3:33:39customers. So now again here we have two
3:33:41steps. The first step we're going to go
3:33:43and do the normal join. So using either
3:33:45the left or the right join. Now by
3:33:47looking to this query you can leave it
3:33:49like this where you can start from the
3:33:50customers. But if you want to fully
3:33:52focus on the orders you have to switch
3:33:54this from left to right. And with that
3:33:56you will get all the orders and only the
3:33:58matching customers. And let's go and
3:34:00remove this workloads from here first.
3:34:01So I'm just adding comments. And with
3:34:03that SQL going to totally ignore this
3:34:06line of code. So let's go and execute
3:34:08it. Now you can see we are getting all
3:34:09the orders right and data from customers
3:34:12only if there is a match. And now of
3:34:14course this is not the task. We don't
3:34:15want to see all the orders. We want to
3:34:17see only the orders where we don't have
3:34:19a match from the customers. So if you
3:34:21look to this those three orders they are
3:34:24okay. They are totally fine. We are
3:34:25finding customers for them. So they have
3:34:27valid customers. But this order here is
3:34:30really bad. So there is no valid
3:34:31customer for this order and now our task
3:34:34to show only this type of orders in the
3:34:36result. Now what we have to do we have
3:34:38to use the workclass in order to get
3:34:40exactly the effects. So this time we're
3:34:42going to say if the ID of the customer
3:34:44here. So here we're going to say the ID
3:34:46of the customer from the table customers
3:34:49must be null. So we're going to remove
3:34:50this here and take the key join from the
3:34:53customer and we are saying this ID must
3:34:56be null. So let's go and execute it.
3:34:58Perfect. With us we have solved the task
3:35:00and we are getting the effect of the
3:35:02right anti- join and we are getting now
3:35:04those orders that don't have any
3:35:06customers. So we have solved the task.
3:35:08Now my friends you have to go and solve
3:35:10this task without using the right join
3:35:13but still you have to get the same
3:35:14effects. You want to get exactly those
3:35:16orders without customers. So pause the
3:35:19video and go solve the task.
3:35:23[Music]
3:35:26Now again as you know me I don't like
3:35:28the right joins. We can create the same
3:35:30effects if you switch the sides of the
3:35:32table. So if you say the B table now on
3:35:34the left side and the A on the right
3:35:36side then we will get the same effect if
3:35:38you go and switch the type of join from
3:35:41right to left and you go just switch the
3:35:43tables. So you start from the B table
3:35:45since it's on the left side and then
3:35:47join it with the A. And we still say of
3:35:49course in our work condition where the
3:35:51data from A is null. So there is no
3:35:53match. So if you do this you will get
3:35:55the exact same results like the lift
3:35:57query by using the lift join and just
3:35:59switching the tables. So you will get
3:36:01the same results and with that you know
3:36:03that in scale we have always
3:36:04alternatives. I hope that you are done.
3:36:06So it's very simple what you're going to
3:36:07do. We're going to go and switch the
3:36:09joins and since the orders is the main
3:36:12table we're going to start first from
3:36:14the table orders. So we are putting it
3:36:16on the left side and then the right
3:36:18table going to be the customers. And of
3:36:20course the condition going to stay as it
3:36:21is. We want to see the orders where
3:36:23there is no customer. So we don't have
3:36:25to switch anything here or in the join
3:36:28key. So let's go and execute it. With
3:36:29that you are getting the same exact
3:36:31results. Since we are using here the
3:36:33star, it's always starts from the left
3:36:35table and show the data from the right
3:36:37table. But still the result is valid. We
3:36:39are getting this type of orders without
3:36:42matching customers. And I prefer this
3:36:44way. All right. So now with that we have
3:36:45the left, the right and now of course
3:36:47what is next? We will get the full. So
3:36:50let's speak about now the full anti-join
3:36:52in SQL. Let's
3:36:56go. Okay. So now what is exactly a full
3:36:59anti- join? Well, this time we don't
3:37:01have sides. We want to return only the
3:37:03rows that don't match in either tables.
3:37:06So what this means? If you are looking
3:37:08to the left circle, we want only the
3:37:10unmatching rows. So we don't want the
3:37:12whole circle. We want only the data that
3:37:14exist in A but it don't exist in B on
3:37:18the right table. Sounds like the left
3:37:19ant join but since we are saying full
3:37:21then you have to do the same thing on
3:37:23the right side as well. So on the right
3:37:24table we want only the unmatching rows.
3:37:27So we want to see in the result the data
3:37:29that is in B but don't have a match from
3:37:32A. So it's exactly the opposite. And if
3:37:34you look to this then that means we want
3:37:36to see only the unmatching data and this
3:37:38is exactly the opposite effect of the
3:37:41inner join. In the inner join we were
3:37:43interested only on the matching data
3:37:45only when there is like overlapping. But
3:37:47now with the full anti-join it is
3:37:48exactly the opposite. We don't want to
3:37:50see the matching data. We want to see
3:37:52everything else the unmatching data. So
3:37:54how we going to write this query? Again
3:37:56here we don't have a special type called
3:37:58full anti-join. We will use the help of
3:38:00the classical full join. So the basic
3:38:02one. So you start from a full join b and
3:38:05then the same key. But now what is
3:38:06interesting is about the where
3:38:08condition. Now we have like two
3:38:09conditions right? So now in order to get
3:38:11all data from A that has no match in B,
3:38:14you have to make a filter where you say
3:38:16the key from the B table must be null.
3:38:18And now since we want the exact same
3:38:20thing from the right table, we want all
3:38:22the data in B that has no match in A.
3:38:24You have to say as well the key from the
3:38:27A table must be null. So now we have
3:38:29here like two conditions. And in SQL if
3:38:31you have like two conditions in the work
3:38:33clause, you have here two options either
3:38:35use and operator or the over operator.
3:38:38So now the one that we're going to use
3:38:40here is the or operator. So either the
3:38:42key from right is empty or the key from
3:38:44left is empty. If you do it like this,
3:38:46you will get the effect of the full
3:38:49anti- join. And of course since here
3:38:50both sides are equal then the order of
3:38:53the tables as well here is not that
3:38:55important. So you can say from A full
3:38:57join B or from B full join A. It doesn't
3:39:00matter. So now let's go back to scale in
3:39:02order to create this effect. Okay.
3:39:04Instead we have the following task and
3:39:05it says find customers without orders
3:39:08and orders without customers. So if you
3:39:10are looking to this this means we want
3:39:12to see only the unmatching data from
3:39:14customers and as well from orders. There
3:39:17is no main table and secondary table.
3:39:19Both of them are equally important. So
3:39:21now since we are talking about the
3:39:22unmatching data and the anti-join we
3:39:24have to do it in two steps. The first
3:39:26step we're going to do the classical
3:39:27join and then we focus on the wear
3:39:29clause. So let me remove the wear clause
3:39:31to make it as a comment. Now since we
3:39:33want the data from left and right, we're
3:39:35going to go and use the full join. So
3:39:36let's go and execute it. Now you can see
3:39:38we are getting the effect of the full
3:39:40join. We are getting all the orders and
3:39:42as well all the customers. But now we
3:39:44are interested only on the strange cases
3:39:47where they are like orders without
3:39:49customers like this one here and as well
3:39:51customers without orders. So that means
3:39:53the first three rows they are not really
3:39:55interesting for us because it is boring.
3:39:57We have here matching data and this is
3:40:00totally fine but we are not focusing on
3:40:02that now. We are focusing only if there
3:40:04is like missing data from left or from
3:40:06right. As you notice I'm saying or and
3:40:09this is very important because we're
3:40:10going to use the or operator. So now
3:40:11let's focus on getting this scenario
3:40:13over here. We want to get an order
3:40:15without a customer. So that means the
3:40:17customer ID must be null. And we have it
3:40:20already here. So we are saying where the
3:40:22ID of the customer is null. So if I go
3:40:25and execute it, I will get only one
3:40:27records only this one over here. But as
3:40:29well I want to get the opposite
3:40:31scenario. So in this scenario, the
3:40:33customer ID must be null. So we're going
3:40:36to say or the customer
3:40:38ID in the orders is null or we can do it
3:40:42like side by side like this. Either the
3:40:44right side is null or the left side is
3:40:47null. So if you go and execute it, you
3:40:49will get the effect of the full
3:40:50anti-join. And with that we are finding
3:40:52the customers without orders and orders
3:40:55without customers. I think this is
3:40:57really fun and as well really easy. So
3:40:59this is how we do the full anti- join.
3:41:01All right. So now if you are looking to
3:41:02the use cases we use the full anti- join
3:41:04again exactly for the last use case in
3:41:07order to check the existence. So if you
3:41:09combine the full with the where you can
3:41:11check the existence or the notexistence
3:41:13of your data in another table. So this
3:41:16is exactly the scenario for that.
3:41:21Okay, my friends, now we have a bonus
3:41:23section where I'm going to challenge you
3:41:25to solve the following task without
3:41:27using an inner join. So, it says, "Get
3:41:29all customers along with their orders,
3:41:31but only for customers who have placed
3:41:34an order, but without using an inner
3:41:37join." So, pause the video now and go
3:41:39and solve this
3:41:40[Music]
3:41:42task. Okay, so now let's see how we're
3:41:44going to solve this. We want the
3:41:46customers, the orders, blah blah blah.
3:41:48But we want only the customers who have
3:41:50placed an order. Previously, we have
3:41:52used the inner join in order to solve
3:41:53this task. But this time, we are not
3:41:55allowed to use it. So, let's go and
3:41:57solve it. This is how I'm going to do
3:41:58it. Select star from table customers.
3:42:01Can't give it the alias. So, now I'm
3:42:03getting all the customers, but I am
3:42:05interested only the customers who have
3:42:07placed an order. So, as we know before
3:42:09there's like two customers didn't order
3:42:11anything, and we don't want to see them
3:42:13in the final results. Now how we will
3:42:15get that? Well, we can use the help of
3:42:17the table orders in order to check the
3:42:19existence of our customers there. And of
3:42:21course, I'm not allowed to use the inner
3:42:23join. So I'm going to go and use a left
3:42:24join with a table orders and then
3:42:27combine them as usual. Nothing new with
3:42:29the customer ID. So now let's go and
3:42:31execute it. As you can see, we are doing
3:42:33it step by step. You don't have to rush
3:42:35everything in one go. So you start
3:42:37simple, check the results and decide on
3:42:39the next step. So now by looking at
3:42:41these results I want to get those three
3:42:43customers because they have ordered
3:42:45something and we are seeing data about
3:42:47their orders and I don't want to get in
3:42:49the result the last two. So again we
3:42:51still can use the customer ID from the
3:42:53right table in order to decide which
3:42:56data going to stay in the result and
3:42:57which data should be filtered. We're
3:42:59going to go and use the wear clause and
3:43:01then the key from the orders and this
3:43:03time we're going to say is not null. I
3:43:06know we didn't learn yet about the not
3:43:08and the logical operators but using the
3:43:11not null it means there should be data
3:43:13inside the column it must not be null if
3:43:16you do it like this and execute you will
3:43:18get the exact effect as the inner join.
3:43:21So as you can see as you are joining the
3:43:22tables using the left join you can
3:43:25control what you want to see using the
3:43:27wear clouds using the filter and this is
3:43:29how you can solve this task without
3:43:30using an inner join. Okay, so with that
3:43:33we have covered all those three
3:43:34scenarios in order to find the
3:43:36unmatching data. Left, right, full and
3:43:38joints. Now we can speak about one crazy
3:43:41join. We call it the cross join. This
3:43:43one is totally different from all other
3:43:45types that we have learned. So let's
3:43:47understand exactly what is the cross
3:43:49join. Let's
3:43:53go. So now what is exactly a cross join?
3:43:56Now in some scenarios we want to combine
3:43:59every row from the left, every row from
3:44:02the right. So that means I want to see
3:44:04all the possible combinations from both
3:44:06tables. So we are doing something called
3:44:08like cartesian join. So now if you look
3:44:10at our two circles, we want everything
3:44:13from A and as well everything from B. So
3:44:16that means I want to see everything from
3:44:18A combined with everything with B. So in
3:44:20this example, we have two rows in A and
3:44:22three rows in B. If you do a cross join,
3:44:25you will get six possible combinations
3:44:27by just multiplying the number of rows
3:44:29between A and B. So be careful using the
3:44:31cross join. If you use it, you will get
3:44:33like crazy number of rows in the results
3:44:35and you're going to make the database
3:44:37really busy finding out the result for
3:44:39you. So now about the syntax, it's going
3:44:40to be the easiest. So you start as usual
3:44:42from one of those tables, the A for
3:44:44example, and then you say cross join B.
3:44:47So now my friends, if you look at this,
3:44:48you can see it's not like the previous
3:44:50joins that we have done. We have always
3:44:52before talked about unmatching rows,
3:44:54matching rows and so on. But here we
3:44:56don't care at all about whether the data
3:44:58is matching or not. I just want to see
3:45:00all the possible combinations
3:45:02everything. So since we don't care about
3:45:04matching the two tables, we don't have
3:45:06to specify any condition. So there is no
3:45:08need to use the keyword on because we
3:45:11don't need any condition. So that's it.
3:45:12You just say cross join B and the magic
3:45:15can happen. So this is a cross join.
3:45:16Let's go to SQL to try that. Okay. So
3:45:19now we have the following task. It says
3:45:20generate all possible combinations of
3:45:23customers and orders. So that means we
3:45:25want everything with everything using
3:45:27the cross join and this going to be very
3:45:29simple. So we're going to start with
3:45:31select star from whatever table. So you
3:45:33can start from the customers and then
3:45:35you say cross join orders. That's it.
3:45:39Very simple. Let's go and execute it. So
3:45:41now as you know we have five customers
3:45:42and four orders. And if you multiply
3:45:44them you will get in the results 20
3:45:46rows. So now we are getting everything
3:45:48with everything. even if the data is not
3:45:51matching at all. So you can see for
3:45:53example the orders here. So this is one
3:45:55order that belongs only to one customer
3:45:57the customer ID one. So it is an order
3:46:00from actually Maria but still we are
3:46:02seeing this same order with the other
3:46:04customers since we want to combine
3:46:06everything with everything. So there are
3:46:07no rules. The same thing for the next
3:46:09set. So this is the second order
3:46:11actually belongs to John but we are
3:46:13seeing this order with all customers. So
3:46:15that's it. This is how the cross join
3:46:18works. And now you might ask me why we
3:46:19have this. It makes no sense, right?
3:46:21Well, my friends, I rarely use it. But
3:46:23sometimes if I want to generate like
3:46:25test data or maybe if you have like for
3:46:27example table called colors and table
3:46:29called products and you would like to
3:46:30see all the combinations between the
3:46:33products and the colors. So in some
3:46:35scenarios it makes really sense to see
3:46:36all your products together with all the
3:46:38colors without any matching conditions
3:46:41or whatever. So there are like few
3:46:42scenarios for the cross join if you are
3:46:44like doing simulations or testing. So
3:46:47this is how we do the cross join. Okay.
3:46:49So that's all about the cross join. And
3:46:51with that we have covered the four
3:46:53advanced types of joins. Now if you look
3:46:55at this you might ask okay how I'm going
3:46:57to choose between all those types. So
3:46:59you might ask me okay bar how you do it?
3:47:01Well I'm going to show you now my
3:47:02decision tree that I usually follow in
3:47:05order to choose the correct type.
3:47:10So now if I'm combining two tables and I
3:47:13want to see in the results only the
3:47:15matching data between two tables then I
3:47:17go and use the inner join. We don't have
3:47:19any other type for that. So that's
3:47:21simple but now if I want to see
3:47:23everything all the data I don't want to
3:47:25miss anything after joining two tables
3:47:27then I take different path and here I
3:47:29ask myself is there like one side more
3:47:31important than the other am I interested
3:47:33in all data from one table from one side
3:47:36like here we have like a main table or a
3:47:39master table then I go and use the lift
3:47:41join but if I want to see all the data
3:47:43from all tables in my query everything
3:47:45so there is no one table more important
3:47:47than other then I go with the full join
3:47:50So this is another path and now the
3:47:52third path if I'm interested to see only
3:47:54the unmatching data. So I'm doing some
3:47:57kind of checkups and so on. And here
3:47:59again the same thing do I want to see
3:48:00the unmatching data from only one side.
3:48:02There is like one table that is
3:48:04important then I go and use the lift
3:48:06anti- join. So I want to see the
3:48:08unmatching data from one table and I'm
3:48:11using the other table only for the
3:48:12check. But in my query if both of the
3:48:15tables are important there is no main
3:48:17table and secondary table both are
3:48:19important then I go and use the full
3:48:21anti- join. So actually that's it. This
3:48:23is the decision tree that I follow
3:48:25usually as I'm writing a query. And you
3:48:27might ask me how about the right join.
3:48:30Well as you know me I don't have it at
3:48:31all in my decision tree. So I don't use
3:48:34it at all. Now by looking to this I can
3:48:36tell you if I check most of the queries
3:48:38that I write very often I use the left
3:48:41join. So I can tell you this is my
3:48:42favorite way on how to join tables. So
3:48:45let me show you exactly
3:48:49why. Usually I write queries in order to
3:48:52do data analyzes. So in data analytics
3:48:54you have always like starting points.
3:48:56You have like a topic that you are
3:48:58analyzing like the customer. So you have
3:49:00always like a master table. So I always
3:49:02start with the main table of my
3:49:04analysis. So in my query I start from
3:49:06this table from table A the main table.
3:49:09And then what happens? The data is not
3:49:11enough in this table. I need some extra
3:49:13data that comes from another table like
3:49:15the table B. So the table B is only here
3:49:18like an additional data to the master
3:49:20table. So I go and use the lift join in
3:49:23order to connect the table B and then I
3:49:25find another interesting information in
3:49:27another table in table C. So same things
3:49:29happens. I go and join the tables using
3:49:31the lift join and so on. So I keep
3:49:33connecting multiple tables to this main
3:49:36table in the middle. And my query going
3:49:38to look like this. always doing lift
3:49:40joins with multiple tables. Now, of
3:49:42course, you might say, "Yeah, but
3:49:43sometimes you would like to see only the
3:49:44matching data and so on. So, it makes
3:49:47sense only to use the inner join." Well,
3:49:49in order to do that, I can control
3:49:50everything that I want to see in the
3:49:52final results using the wear clause. So,
3:49:55in the wear clause, I define exactly
3:49:57what I want to see in the final result.
3:49:59So, with that, I get like more
3:50:00flexibility on whether I want to see the
3:50:02matching, unmatching data and so on like
3:50:04we done in the lift and join, right? So
3:50:06as I'm analyzing data I tend very
3:50:09frequently having this setup where I
3:50:11start from the main table and I lift
3:50:13join all other tables and with the word
3:50:15conditions I control the final results.
3:50:17So this is how I connect multiple tables
3:50:20together. So now if I want to visual
3:50:21this in like circles it's going to look
3:50:23like this. We have the circle A. So this
3:50:25is the master table the starting point.
3:50:27I want to see all the data from table A
3:50:30and I live join it then with another
3:50:32table B and from table B I want to see
3:50:34only the matching data. So it's like the
3:50:36lift join. Now what going to happen? I'm
3:50:38going to go and add another table. So
3:50:40another circle the circle C. And from
3:50:42the circle C, we want to see only the
3:50:44matching data. And of course you can
3:50:46keep adding circles to this. But it's
3:50:48going to be always the same thing. And
3:50:49in your circle going to has only the
3:50:51matching data. So now as we learned we
3:50:53can use joins in order to combine
3:50:55multiple tables to get a complete big
3:50:57picture about topic like the customers.
3:50:59I would like to see everything about the
3:51:01customers in the final results. So
3:51:02either you're going to do it like me
3:51:04where you start from the main table and
3:51:05then go and lift join all other tables
3:51:08or maybe you say you know what there is
3:51:10no main table about the customer's data
3:51:12all the tables are equally important
3:51:14then you can go and join all those
3:51:16tables using the inner join if you are
3:51:18interested only on the match data so
3:51:20what can happen if you have again those
3:51:22circles from the A you need only the
3:51:24matching data from B you need as well
3:51:26only matching data and as well from the
3:51:28third circle so you are interested only
3:51:30on the overlapping between all all three
3:51:32tables. So you will get only this
3:51:34section where you have overlapping
3:51:36between all three tables. So this is of
3:51:38course another way on how to join
3:51:40multiple tables. Okay. So now my friends
3:51:42let's go back to scale in order to
3:51:44practice how to join multiple tables.
3:51:46Okay. So now let's have a task. This
3:51:48going to be a little bit challenging. We
3:51:49will be doing multi- joins using the
3:51:52sales DB. Retrieve a list of all orders
3:51:54along with the related customer product
3:51:57and employee details. And for each order
3:52:00display the following. We want to see
3:52:01the order ID, the customer name, the
3:52:04product name, sales price, salesperson
3:52:07name. So there is a lot of things that
3:52:08is going on. And the first thing that
3:52:10you're going to notice it does now we
3:52:11are using different database. We will be
3:52:14not using the my database, we're going
3:52:16to go and use the sales DB. So this is
3:52:18the first thing that we have to do. So
3:52:20instead of using my database, so we say
3:52:23use sales DB and then execute it. We are
3:52:26now connected to the sales DB. So this
3:52:28is the first thing. So now if you are
3:52:30reading this task there are a lot of
3:52:31tables that are involved. We need the
3:52:33orders, we need the customers, products
3:52:36and employees. So there are like four
3:52:38tables needed in this task and we need
3:52:41different stuff from each table. So now
3:52:43how I think about it well it is mainly
3:52:45focusing on the table orders right? So
3:52:47we need all the orders we cannot miss
3:52:50any order here. So this sounds for me
3:52:52this is the main table and then it says
3:52:55along with that we need other
3:52:56informations. So that means the other
3:52:58tables are not that important like the
3:53:00orders. So this gives me feeling about
3:53:02what is the main table and this going to
3:53:04be my starting points. So let's start
3:53:06from that from the table orders. So
3:53:08select star from and here you have to
3:53:10pay attention that this database has
3:53:13always a schema. It's called if you look
3:53:15to the left side sales dot the table
3:53:17name. So we have to write that now in
3:53:19our query. So we're going to write it
3:53:21over here sales dot and then the table
3:53:23name orders. Let's go and execute it.
3:53:25Now I know this is the first time that
3:53:27you are querying this table. We have a
3:53:29lot of informations here and as well we
3:53:31have a lot of ids. Those ids going to
3:53:33help us of course on joining our data
3:53:36with the other tables. So what do we
3:53:37need from here? We need the order ID. So
3:53:39we have it over here. We're going to get
3:53:42the order ID. This time the naming
3:53:44convention is different. We don't have
3:53:46like underscores and comm. We have
3:53:48different type of namings. So be careful
3:53:50with that. So what else do we need? We
3:53:52need the sales. So if you go to the
3:53:53right side over here, we have column
3:53:55gold sales and we're going to go and
3:53:57include it to the results. Now all the
3:53:59other informations are actually not
3:54:01needed, but I need those ids in order to
3:54:04join it with the other tables. So now
3:54:06what I'm going to do, I'm going to go
3:54:07and give it an alias and all. So now I'm
3:54:09going to go and assign it for each
3:54:10column. This comes from the orders and
3:54:13as well the same thing for the sales. So
3:54:15that's it for now. And if I go and
3:54:17execute it, I will get the orders and
3:54:19the sales. All right, so that's all for
3:54:20the first table. Let's go now and see
3:54:22what do we need. We need the customer's
3:54:24name. Well, actually we don't have this
3:54:26piece of information in the orders. So
3:54:28all what you have to do is to go and
3:54:29explore in the other tables in order to
3:54:31find this column. So how I usually do I
3:54:34go and explore the tables like this. So
3:54:36I write a symbol select from each
3:54:39tables. So the customers. So now I go
3:54:42and repeat this for each table inside
3:54:44the database. So we have the customers,
3:54:47employees, we have an orders, the orders
3:54:52archive and as well the products. So now
3:54:56I start exploring the table. So if I go
3:54:58to the customers over here, we can see
3:55:00we have here five customers and we can
3:55:02see the names of the customers. So we
3:55:04see the first name and the last name and
3:55:06this is exactly what I need for my
3:55:08query. Now of course we have to go and
3:55:09connect this table with the orders. So
3:55:12we need a common column. Usually it's
3:55:13going to be the ID. So here we have the
3:55:15customer ID and if you go and query the
3:55:18orders you can find here as well the
3:55:20customer ID. Now if you are working in
3:55:21big projects you're going to have a lot
3:55:23of tables and exploring each one of them
3:55:25going to be really hard. So now of
3:55:26course if you have like in the project
3:55:28hundreds of tables it's going to be
3:55:29really hard to explore each table. So
3:55:31instead of that a good project a good
3:55:33database usually has an entity
3:55:35relationship model er model like the one
3:55:38that we have for the course. And here
3:55:40you can find easily the tables that you
3:55:42have inside your database and as well
3:55:44the relationship between them and this
3:55:46is very important especially if you want
3:55:49to join tables. So now by just looking
3:55:50quickly to this diagram I can understand
3:55:53okay there is an ID called customer ID
3:55:56inside the table orders and it is like a
3:55:59foreign key to the primary key the
3:56:01customer ID. So that means if I want to
3:56:03connect the orders with the customers I
3:56:05have to use that customer ID. So as you
3:56:07can see this is really nice
3:56:08documentations and I can quickly
3:56:09understand how to join the tables. So
3:56:11now back to our query. Now what I'm
3:56:13going to do I'm going to say lift join.
3:56:15So with that I guarantee all the orders
3:56:18going to be presented in the output and
3:56:19I will see always 10 orders. So now
3:56:21let's join it with the table customers
3:56:23sales dot customers and let's give it an
3:56:26alias like this. And now we're going to
3:56:28build the joining condition. So it's
3:56:30going to be the customer ID from the
3:56:32table orders equal to the customer ID
3:56:35from the table customers. So that SQL
3:56:37understand how to match the two tables.
3:56:40And now the two tables are connected and
3:56:42I can get the informations now from the
3:56:44customers. So see let's go and get the
3:56:46first name and as well the last
3:56:49name. So now let's go and execute it. So
3:56:52now as you can see we have customers for
3:56:54each order which is really nice. So with
3:56:56that we got the customer name and the
3:56:58order ID. Now the next one we need the
3:57:00product name. So either you're going to
3:57:02go here and start exploring. I think it
3:57:04is inside the table products. And here
3:57:06you can see we have the product. This is
3:57:09the name of the products. And if you
3:57:11check our ER diagram, you can see we can
3:57:13connect the table orders with the
3:57:15products using the product ID. So we
3:57:18have the product ID in the left and as
3:57:20well in the right. And now we can go and
3:57:22build this join as well over here. So
3:57:24again I go with a lift join. I don't
3:57:26want to lose anything from the table
3:57:27orders sales products and we give it an
3:57:31alias P. Now the condition for that here
3:57:34you have to be very focused. You want to
3:57:36get the product from the orders. So you
3:57:39say O dot product id equal to the
3:57:43product ID from the table products. So
3:57:46as you can see in the joins we are
3:57:47always joining with the table orders.
3:57:49Right? We are not trying to join for
3:57:51example the customers with the products.
3:57:53Always we are joining with the main
3:57:55table. So with that we have connected
3:57:57the third table and we can get the
3:57:59information that we need. So we need the
3:58:01products as I'm going to go and rename
3:58:03it products name. So let's go and
3:58:07execute it. And with that my friends I'm
3:58:09getting now the product informations
3:58:10from the table products. So we have the
3:58:13sales as well and we need the price. So
3:58:15if you go to the products you can see we
3:58:17have as well price information. I forgot
3:58:19about it. So let's go and get it as well
3:58:21from the same table. price. So let's go
3:58:24and execute it. And with that we have as
3:58:26well the prices. Now the last column it
3:58:28says we want to get the saleserson name.
3:58:31So the name of the employee right now if
3:58:33you go and explore as well we have here
3:58:35employees table and execute it. You can
3:58:37see we have here the name and the last
3:58:39name of the employees and we have an ID.
3:58:42So now we need this ID as well in the
3:58:44orders. So you can see we have the
3:58:46product ID, the customer ID. We already
3:58:48used those two. But we have here one
3:58:50more extra ID called the salesperson ID.
3:58:52Of course, it is not called employee ID.
3:58:54So here you might be a little bit
3:58:56skeptical about it. That's why we have
3:58:58to go and check again our ER diagram.
3:59:00And as you can see the employee ID from
3:59:02the employees, it is connected to the
3:59:05salesperson ID. So that I have better
3:59:07feeling about it and I understand. Okay,
3:59:09I can connect the orders with the
3:59:10employees using the salesperson ID. So
3:59:13let's go and do that. I'm going to say
3:59:15lift join. So as you can see I'm just
3:59:17doing left joins sales dot employees as
3:59:21e and the condition again very important
3:59:24always the first table is included in
3:59:26the join condition and here we're going
3:59:28to say the sales person ID is equal to
3:59:31the employee ID. So with that we have
3:59:34connected as well the employees and we
3:59:36will get as well the first name and the
3:59:39last name. So perfect that's it. Let's
3:59:42go and execute it. And as you can see
3:59:44guys, now we are getting the name of the
3:59:46salesperson. Now here comes an issue. As
3:59:48you are joining multiple tables and you
3:59:50are getting columns from different
3:59:52tables, what can happen? You might
3:59:53encounter this scenario where you have
3:59:55the same names in multiple tables. So
3:59:58now as you can see we have the first
3:59:59name last name from the employees and as
4:00:01well we have the first name last name
4:00:02from the customers and it's going to be
4:00:04really hard from the result to
4:00:05understand what are we talking about? Is
4:00:07it the customers? Is it the employee?
4:00:09That's why in this scenario if you have
4:00:10the same names we have to go and start
4:00:13giving aliases. So for the first one
4:00:15we're going to say customer first name
4:00:18and as well for the last name we're
4:00:20going to say customer last name. Same
4:00:22thing for the employee. So let's say
4:00:25employee first name or we can call it
4:00:28the saleserson whatever employee last
4:00:31name. So if you go and execute it now
4:00:34it's going to be more clear. Here we are
4:00:36talking about the name of the customer
4:00:38and here we are talking about the name
4:00:40of the employee. And again one more
4:00:42thing if you are not using aliases it's
4:00:43going to be an issue. So for example if
4:00:45you go over here and you don't use the
4:00:47table name before the column. So if I go
4:00:49and remove it and execute it you will
4:00:52see I'm getting an error. Now SQL can't
4:00:54understand what are you talking about.
4:00:56Is it the first name of the customer or
4:00:58from the employees because you are not
4:01:00specific about it. So you have to tell
4:01:02SQL to which table belong this column.
4:01:04It's very important to use a table name
4:01:06or the alias before the column name.
4:01:09Especially if you have the same column.
4:01:11So now we will not get an error. And
4:01:13with that we have solved the task. You
4:01:14have really to pay attention about the
4:01:17join keys. The condition you have to do
4:01:19it correctly cuz as you can see now we
4:01:21have a lot of tables and a lot of
4:01:22columns and sometimes happens an issue
4:01:24where you specify the wrong columns or
4:01:27the joins and the result can makes at
4:01:29all no sense. So always double check are
4:01:31you using the correct keys in order to
4:01:33join the tables. So with that you have
4:01:35solved the task and this is exactly how
4:01:37I join tables. I have always a starting
4:01:39point from an important table and
4:01:42everything else going to be left joined
4:01:44and in my results if I want to remove
4:01:46any scenario then I go and use the wear
4:01:49clause. So this is how I join multiple
4:01:51tables. Okay my friends. So with that
4:01:53you have learned now everything about
4:01:55how to join the tables in SQL and this
4:01:57is very important to understand. Now
4:01:59moving on to the second method on how to
4:02:01combine your data from multiple tables.
4:02:03We have the set operators. So we're
4:02:05going to go and cover how to combine the
4:02:07rows from multiple tables. So let's
4:02:12go. All right, my friends. So now as we
4:02:15learned before, in order to combine two
4:02:17tables we have two methods. If you want
4:02:19to combine the columns, we use the
4:02:21joins. And we have learned all those
4:02:23different types on how to combine data
4:02:25using join. So we have covered this
4:02:27section. But now if we want to combine
4:02:28the rows of two tables, we can use the
4:02:31set operators. And here we have four
4:02:33different types. We have union, union
4:02:35all, except and intersects. So now we're
4:02:38going to go and deep dive into this word
4:02:40on how to combine the rows of tables
4:02:42using the set operators. And now of
4:02:44course in this course we're going to
4:02:45cover everything. So let's
4:02:49go. All right. So now let's have a look
4:02:52to the syntax of the set operators.
4:02:54Okay. So now let's see that we have the
4:02:55following query. we are selecting the
4:02:57data from the customers. So this is our
4:02:59first query or our first select
4:03:01statements and we have another one which
4:03:03is very similar where we are selecting
4:03:05the informations from the employees and
4:03:08this is our second select statement. So
4:03:10now what we can do we can put between
4:03:12those two queries a set operators like
4:03:15for example the union. We can use of
4:03:17course any other set operators like the
4:03:18union all intersects except and so on.
4:03:22So as you can see the syntax is very
4:03:24simple. We have two different queries
4:03:26and we just put between them the set
4:03:28operator. So this is how the syntax of
4:03:30the set operators looks like. All right
4:03:32friends. So now we're going to talk
4:03:33about the rules of the set operators.
4:03:35And we're going to start with the rule
4:03:37number one the SQL clauses. In each
4:03:39individual select statements or query.
4:03:41We can use almost all the SQL clauses
4:03:44like where join group by having. But
4:03:47there is only one exception with the
4:03:49order by. Order by you can use it only
4:03:51once and only at the end of the entire
4:03:54query. So that means we cannot use order
4:03:57by in each select statements or in each
4:03:59query. We can use it only once and only
4:04:02at the ends of the entire query. All
4:04:04right. So about the syntax again here we
4:04:05have our two select statements and in
4:04:08between them we have the set operators.
4:04:10So now in each query we can go and use
4:04:12multiple stuff like the join where group
4:04:15by having. So we can make each query
4:04:18complex as we want. So everything is
4:04:21allowed but not the order by the order
4:04:24by must be always placed at the end of
4:04:26the entire query. So if you want to sort
4:04:28the result by the first name, you have
4:04:30to use the order by exactly at the end.
4:04:33So we are not allowed to use order by in
4:04:36each query. Okay. Moving on to the rule
4:04:38number two. The number of columns. The
4:04:40number of columns in each query must be
4:04:42the same. Okay. Okay. So now in order to
4:04:44understand this rule, let's have this
4:04:46very simple example. We're going to go
4:04:47and select the first name and the last
4:04:50name from the table sales customers. So
4:04:54this is our first query, our first
4:04:55select statements and let's say that I
4:04:58have another one and we want to select
4:05:00the first name last name but this time
4:05:02from another table, the employees. So
4:05:04with that we have our two queries and I
4:05:06would like now to go and combine them
4:05:08into one result. So we're going to go
4:05:10and use the set operator union. Let's go
4:05:13and execute it. So now as you can see in
4:05:14the result we will get the first name
4:05:16and last name from two tables the
4:05:18customers and employees. And it is
4:05:20working because we are fulfilling the
4:05:21rule where it says the number of columns
4:05:24must be the same in both queries. So how
4:05:26many columns do we have in the first
4:05:28query? We have two right and as well in
4:05:30the second query we have two columns. So
4:05:32that's why everything is working. So now
4:05:34let's go and break the rule by adding
4:05:36another column to the first query. So
4:05:38let's say that I would like to have the
4:05:40customer ID as well in the first query
4:05:42and with that as you can see in the
4:05:44first query we have three columns but in
4:05:46the second we have only two. So let's go
4:05:49and execute it. Now as you can see in
4:05:51the result we will get an error where it
4:05:52says if you are using union intersect
4:05:54and all those set operators you must
4:05:57have an equal number of columns between
4:05:59queries. So this is the rule you have to
4:06:02have the same number of columns in order
4:06:03to repair it. So I'm going to do I'm
4:06:05just going to remove the customer ID.
4:06:06Okay. So here again we have two columns
4:06:08and the second one as well two columns
4:06:10and everything going to be working.
4:06:12Okay. Moving on to the rule number
4:06:13three. The data types of columns in each
4:06:15query must match must be compatible in
4:06:19matching. In order to check that what
4:06:21we're going to do we're going to go to
4:06:22the object explorer to the left side.
4:06:24Let's go and browse the customers and
4:06:26the columns. And as you can see we have
4:06:28here the first name and last name with
4:06:30the same data type. We have the vchar.
4:06:32And if you go to the employees, you can
4:06:34see as well the first name, last name
4:06:36having varchar. So the first column is
4:06:38varchchar from the first query and as
4:06:40well for the employees and as well the
4:06:43last name from the customers having the
4:06:45same data type as the last name from
4:06:47employees. So the data type is matching.
4:06:50Now let's go and break this rule.
4:06:52Instead of having the first name, I
4:06:54would like to go and use the customer
4:06:56ID. So now let's check the customer ID
4:06:58on the left side. It is an int, an
4:07:00integer. But the first name is
4:07:03invarchar. So here we have a mismatch
4:07:05between data types. Let's go and try to
4:07:07execute it. So now we are getting an
4:07:09error where it says SQL is trying to
4:07:10convert the value Frank to an integer.
4:07:13So what this means the first query is
4:07:15always controlling everything the names
4:07:17and as well the data types. So here we
4:07:19have an integer and now scale is trying
4:07:22as well to convert the first name values
4:07:24to an integer and of course it will not
4:07:26work because we have here characters
4:07:28inside and it cannot convert characters
4:07:30to an integer. So we have a mismatch
4:07:32between data types between the customer
4:07:34ID and the first name and that's why we
4:07:36will get an error. The second column we
4:07:38don't have an issue because it is
4:07:40varchar in the first table and as well
4:07:42for the second table. So now in order to
4:07:44repair it either select a first name in
4:07:46the first query or we can go over here
4:07:48and say employee ID and with that if I
4:07:52execute it we will not get any errors
4:07:54because the employee ID is as well an
4:07:56integer and we have a match in the data
4:07:58types. So as you can see it's not enough
4:08:00to have the same number of columns. You
4:08:02have to have as well matching data types
4:08:04between those two queries. Okay, let's
4:08:06move to the next rule. Rule number four,
4:08:08the order of columns. The order of
4:08:10columns in each query must be as well
4:08:12the same. Okay, so let's understand what
4:08:14this means. Now we have here again the
4:08:16same example where we are selecting the
4:08:18ID and last name from customers and we
4:08:20are combining it using union with the
4:08:22employee ID and last name from the
4:08:24employees. And as you can see everything
4:08:25is working because we have the same
4:08:27number of columns and we have a matching
4:08:29data types. So now let's go and break
4:08:31it. What I'm going to do I'm just going
4:08:33to switch between those two columns. So
4:08:35first I'm selecting the last name and
4:08:37then the customer ID. So again I have
4:08:39the same number of columns and the ID is
4:08:42integer matching the ID of the employee
4:08:44and the last name having the same data
4:08:46type. So let's go and execute it. So
4:08:49here again SQL going to throw an error
4:08:51and says SQL is trying to convert the
4:08:52value go back to an integer. So it's
4:08:55like character to integer. It will not
4:08:56work. So what happened here? I have here
4:08:58the same informations. I have an ID and
4:09:00last name and ID and last name. Well,
4:09:02SQL doesn't work like this. SQL going to
4:09:04go and map the first column from the
4:09:06first query with the first column with
4:09:08the second query. So it's going to go
4:09:09and map last name to employee ID. And
4:09:12since they have different data types,
4:09:13SQL going to throw an error. So SQL
4:09:16doesn't understand or don't know how to
4:09:17map let's say the ID with the ID and
4:09:20since they have different data types SQL
4:09:21going to go and throw an error. So as
4:09:23you can see here we have the same
4:09:25informations between customers and
4:09:27employees but they don't have the same
4:09:29order. So SQL cannot go and map the
4:09:32informations because of the names of the
4:09:33columns. It's going to go and simply
4:09:35just mapping the columns like this. The
4:09:38first column from the first query with
4:09:39the first column from the second query.
4:09:42So as you can see in this rule you must
4:09:44have the same order of the columns.
4:09:46First the ID and then the last name and
4:09:50with that it's going to work again. All
4:09:52right moving on to the rule number five.
4:09:54The column aliases column names that we
4:09:56see in the output in the result is
4:09:58defined and determined by the column
4:10:00names of the first query the first
4:10:03select statements. So that means the
4:10:05first query is responsible of naming the
4:10:07columns in the output. Okay. So let's
4:10:10understand what this rule means. Again
4:10:11we have the same example. The customer
4:10:13ID, last name from customers, union,
4:10:15employee ID, last name from employees.
4:10:17So if you check closely the output, you
4:10:19can see that in the output we have the
4:10:21customer ID and not the employee ID.
4:10:24Even though we have the ids from the
4:10:26employee ID, but as you can see the
4:10:28first query is controlling the naming of
4:10:31the output. So since the first column
4:10:34called the customer ID, you will see it
4:10:36in the output as a customer ID. So the
4:10:38naming of the like the next queries will
4:10:41be totally ignored. So that's why if you
4:10:43want to give aliases to the output,
4:10:46you're going to go and do it only for
4:10:47the first query. So for example, I go
4:10:49over here and say instead of having
4:10:51customer ID, I would like to call it as
4:10:53an ID. So now if I go and execute it, as
4:10:56you can see in the output, we will get
4:10:58an ID. So I don't have to go and in each
4:11:00query give this alias. So I don't have
4:11:03to go over here and say yeah you are as
4:11:05well the ID because it's enough to
4:11:07define it from the first query. So
4:11:09there's no need to give the same names
4:11:12in the next queries. Let's take another
4:11:14example where we would like to have an
4:11:15alias for the last name. So I would like
4:11:17to have it like this last name and let's
4:11:21go and do it in the second query. So
4:11:24last name let's go and execute it. So
4:11:27now as you can see in the output, we
4:11:29still have last name and there's no
4:11:31underscore because this is totally
4:11:33ignored from SQL. This is not the first
4:11:36query. The first query says you are last
4:11:38name without underscore. So again if you
4:11:41want to do that we go over here. Let me
4:11:42just get it and put it in the first
4:11:45query. Let's go and execute it. So my
4:11:47friends, the first query is very
4:11:48important in order to give the names for
4:11:51the output. So if you want to do aliases
4:11:53and to rename stuff, do it only on the
4:11:55first query. And as well the first query
4:11:57controls the data types. All right. Now
4:11:59to the last rule matching the correct
4:12:02informations. If in your query you
4:12:03fulfill all other rules and you don't
4:12:05have an error in the SQL that doesn't
4:12:08mean that your result is accurate and
4:12:10correct. You are the only one that is
4:12:12responsible of mapping the informations
4:12:14between queries correctly because SQL
4:12:17doesn't understand the content and the
4:12:19informations of your tables of your
4:12:21queries. And if you don't match the
4:12:23informations correctly between the
4:12:24queries, you will get inaccurate and
4:12:27wrong results in the output. Okay. So
4:12:29now back to our example. Let's say I
4:12:30would like to get the first name and as
4:12:34well the last name from the customers
4:12:36and the same informations from the
4:12:38employees. Let's go and execute it. Now
4:12:40as you can see it's very nice where we
4:12:42are getting the first name, last name
4:12:43from both tables in one result and we
4:12:46are fulfilling all the requirements in
4:12:48SQL. Same numbers, same data types and
4:12:50so on. Now let's go and make incorrect
4:12:52results. So what I'm going to do, I'm
4:12:53just going to swap the first name and
4:12:56last name in the second query. So first
4:12:58last name and then the first name. So
4:13:01let's go and execute it. So now as you
4:13:03can see we will get results because we
4:13:04are fulfilling all other rules because
4:13:07we have the same number of columns and
4:13:09as well we have matching data types. So
4:13:11the first one is character the first
4:13:13name and the last name is as well
4:13:14character. So SQL will just present the
4:13:17result as you define it. But the result
4:13:20is completely wrong because now we have
4:13:22if you check the first column here the
4:13:24first name. So here we can see last
4:13:26names inside the first names. For
4:13:28example, Brown and Baker those are last
4:13:30names but we can see them inside the
4:13:32first name. And the same thing in the
4:13:34last name. We now we can see first names
4:13:36inside it. Mary, Carol, they are all
4:13:39first names. So as you can see the
4:13:41result has really bad data quality. We
4:13:42are now mixing stuff and it doesn't
4:13:44makes any sense. But SQL will not know
4:13:47that because SQL doesn't know the
4:13:49information the content of your data.
4:13:52It's just mapping the data types. So
4:13:54first name is varchchar the last name as
4:13:56well vchar. Everything is fine and you
4:13:59will get the results. So my friends you
4:14:01are responsible of having the same
4:14:03informations mapped between the two
4:14:05queries and not having an error from a
4:14:07skill doesn't mean that we have now
4:14:09correct results. So pay attention to the
4:14:11informations that you are mapping
4:14:13between the two queries. All right. So
4:14:15those are the rules of the set
4:14:17operators. So the first one is that the
4:14:19order by can only be used once at the
4:14:21end of the entire query and all queries
4:14:24must have the same number of columns,
4:14:26the matching data types, the same order
4:14:29of columns and the first query always
4:14:31control the names and the aliases of the
4:14:34result set and as well the data type.
4:14:36And the last rule is that make sure that
4:14:38you are mapping the correct informations
4:14:41to each others between queries. So those
4:14:43are the rules of the set
4:14:48operators. Okay. So what is union? Union
4:14:51going to go and return all distinct
4:14:53unique rows from both queries. So that
4:14:56means it's going to go and combine
4:14:57everything and all the rows going to be
4:14:59presented at the output. So since it
4:15:02says all distinct unique rows that means
4:15:05union going to go and remove all
4:15:07duplicates from the combined result set.
4:15:10So union going to make sure that each
4:15:12row going to appear only once. All
4:15:14right. So now let's have this very
4:15:15simple example. We have two sets of
4:15:17data. We have the customers where we
4:15:19have five customers with the first names
4:15:22and as well we have another set called
4:15:23employees and we have as well the first
4:15:25names of the employees and we have five
4:15:28employees. And now if you take a look to
4:15:29the first names you can see that we have
4:15:31the same persons as a customers and as
4:15:33well as employees. We have given and
4:15:36marry in both sets of data. So now how
4:15:38is k going to execute union it's going
4:15:40to go and return everyone from customers
4:15:43and everyone from the employees. But now
4:15:45since we have given and married twice in
4:15:47the output we're going to have them only
4:15:49once. So this is how the union works. It
4:15:51going to go and return everyone from two
4:15:53sets but without duplicates. All right.
4:15:55So now we have the following task and it
4:15:57says combine the data from employees and
4:15:59customers into one table. So that means
4:16:02in one table we want to combine all
4:16:04informations from employees and
4:16:05customers. So which informations do we
4:16:08need? This is the first question that I
4:16:09usually ask myself. So in order to do
4:16:11that first we have to explore the data.
4:16:13So select star from sales customers and
4:16:18then semicolon. Then I'm going to write
4:16:20another query select star from sales and
4:16:24employees and semicolon. So now why I'm
4:16:27using two different semicolons because
4:16:29I'm telling SQL we have now two separate
4:16:31queries. They have nothing to do with
4:16:32each others. And if you go and execute
4:16:35it like this. And now in the output you
4:16:36can see we got two result grids. The
4:16:39first result grid is for the first query
4:16:41and the second one for the second query.
4:16:44So they have nothing to do with each
4:16:45others. I just want to explore those two
4:16:47tables in order to understand how I'm
4:16:48going to map those informations. So now
4:16:50if we check those two tables you can see
4:16:52that both of them has ids. So we can map
4:16:55those informations right. Both of them
4:16:57has as well first name last name. So
4:17:00that means I can go and map the first
4:17:02name and last name together. Now in the
4:17:04customers we have country but we don't
4:17:07have this informations in the employee.
4:17:09So we have to go and ignore it. And we
4:17:11have as well here score where we don't
4:17:13have a score for the employees. That
4:17:15means I can go and map three
4:17:17informations between the customers and
4:17:19employees. Now of course we can go and
4:17:21think do we need really the ids because
4:17:23it doesn't make really any sense to have
4:17:24the ids in the tables. It's not anymore
4:17:27unique because we have here the custom
4:17:28ID one and employee one. So I think we
4:17:31can go and ignore it. So the only really
4:17:33two informations that is useful to map
4:17:35is the first name and last name. So now
4:17:37let's go and add those two informations.
4:17:39So we need the first name, last name and
4:17:42the same informations as well from the
4:17:44employees. But now we want everything to
4:17:46be in one query. That's why I'm going to
4:17:48go and remove the semicolons. And now we
4:17:51have to go and use set operators between
4:17:52those two queries. And now in order to
4:17:54combine the data we have two options
4:17:56either union or union all in this
4:17:58example it doesn't mention anything
4:18:00about duplicates and so on. I would like
4:18:02to go with the union in order to remove
4:18:04the duplicates if there is any. So
4:18:06that's it. Let's go and execute it. Now
4:18:09as you can see in the output we have
4:18:10only one result because we have only one
4:18:12big query. And now we have the first
4:18:14names and last names from the customers
4:18:16and employees. And now one more thing
4:18:18about the order of the queries. It
4:18:20doesn't matter whether we start with the
4:18:21employees or with the customers. we will
4:18:23get the exact same results but pay
4:18:25attention to the naming of the columns.
4:18:27Always the first query controls the
4:18:29names but since now they have the same
4:18:31naming so it should not be a problem. So
4:18:33if I go and switch those two tables and
4:18:36start it again we will get exact same
4:18:38results. So now let's understand how
4:18:40scale did combine the data using the
4:18:43union. Okay. So now we have here the
4:18:44results from the first query and the
4:18:46second query employees and customers and
4:18:48we are combining the data using union.
4:18:50The first step in SQL is that it's going
4:18:52to go and take the columns from the
4:18:54first query which is from the employees.
4:18:56So it's going to take the first name
4:18:58last name as a column name to the
4:19:00results. And now the next that is going
4:19:02to go and start combining the rows
4:19:04between those two tables. So first going
4:19:06to go and take the rows from employees
4:19:08and as well going to check whether there
4:19:10is duplicates in the data. So as you can
4:19:12see we don't have here any duplicates.
4:19:13So we're going to have the five
4:19:15employees. And now the next step is
4:19:17going to start adding rows from the
4:19:18second query from the customers very
4:19:21carefully without generating any
4:19:22duplicates. We don't have it in the
4:19:24output. That's why it's still going to
4:19:25go and add it to the result. Append it.
4:19:28And then the next customer we have Kevin
4:19:30Brown. As you can see, we have it
4:19:31already in the results. That's why will
4:19:33not go and add it to the result.
4:19:35Otherwise, it's going to go and generate
4:19:36duplicates. So it's still going to
4:19:38ignore this customer. The same thing for
4:19:40Mary. We have Mary as well in the
4:19:42results. So it's going to skip it. And
4:19:44then we're going to go to the mark. As
4:19:45you can see, we don't have mark in the
4:19:47results. That's why SQL going to go and
4:19:49take this customer and put it in the
4:19:51output. And then the last one, we have
4:19:53Anna. We don't have Anna in the results.
4:19:55That's why SQL can go and as well add it
4:19:57to the results. And now with this, SQL
4:19:59did combine the rows between those two
4:20:01tables. And we have here eight persons.
4:20:03So as you can see, SQL is combining the
4:20:05data, but very carefully not generating
4:20:07any duplicates. All right. So that's it.
4:20:09This is how the union operator works.
4:20:15Okay. So now union all union union all
4:20:17going to go and return all rows from
4:20:19both queries. So it's very similar to
4:20:21union. It going to go and combine all
4:20:23the rows and everything going to be
4:20:25presented in the combined result set.
4:20:27But the big difference to the union all
4:20:30will not remove any duplicates. It is
4:20:32the only set operators that doesn't
4:20:35remove duplicates and it going to show
4:20:37all the rows as it is. So if you have a
4:20:40row 10 times from the query, you will
4:20:42find it as well in the output 10 times.
4:20:44Now you might ask me when to use union
4:20:46and when to use union all. I'm going to
4:20:48say that there is one big difference
4:20:50between them is that union all has way
4:20:53better performance and it's faster than
4:20:55the union. And that's because union all
4:20:58doesn't perform additional steps like
4:21:00removing duplicates. So my friends that
4:21:02means if you know already that in my
4:21:04queries there is no duplicates. I know
4:21:06my tables. I know my queries. There's no
4:21:08duplicates. Don't use union and always
4:21:11use union all because you will get
4:21:13better performance. Another scenario for
4:21:15the union all is that I would like to
4:21:16see the duplicate. I'm doing data
4:21:18quality checks and I would like to see
4:21:19whether there is duplicate after I
4:21:22combine multiple queries. So in this
4:21:24situation I go and use as well the union
4:21:26all. Now we have again the same example.
4:21:28We have the customers and employees and
4:21:30we have as well the same persons Kevin
4:21:32and Mary as customers and as well as
4:21:34employees. So now if you want to combine
4:21:36the data using union all it going to
4:21:38return all rows including duplicates. So
4:21:41that means SQL going to go and execute
4:21:43union all like this it going to return
4:21:45everything from customers and everything
4:21:47from employees and Kevin and Mary going
4:21:50to be presented twice in the output. So
4:21:52as you can see union all is returning
4:21:54all the rows as it is from the two
4:21:57result sets and if there's duplicates in
4:21:59the sets we will get as well duplicate
4:22:01in the output. So Kevin going to be
4:22:02existing twice in the output and marry
4:22:05as well twice. So this is how the union
4:22:07all works. All right. So now we have
4:22:08very similar SQL task and it says
4:22:10combine the data from employees and
4:22:12customers into one table including
4:22:15duplicates. So it's exactly like the
4:22:17last task but this time in the task we
4:22:19are saying include duplicates. So we
4:22:21cannot go and use union. We have now to
4:22:23go and use union all. We will have the
4:22:25exact same query. So we are selecting
4:22:27the employees first last name and as
4:22:29well customers first last name. And now
4:22:31instead of using union, we're going to
4:22:33go and use union all. So all what we
4:22:35have to do is that to go over here and
4:22:36say union all. So now pay attention to
4:22:39this. As you can see in the union
4:22:40previously, we got eight records or
4:22:43eight persons from the output. So now
4:22:46let's go and execute it and check the
4:22:48results. Now as you can see we got now
4:22:4910 persons instead of eight. And that's
4:22:51because we have five customers and five
4:22:54employees and we have duplicates inside
4:22:56the data. We have two duplicates. Now if
4:22:57you check we have here Mary and as well
4:22:59over here we have Mary and same goes for
4:23:01given we have given over here and as
4:23:03well here. So we have duplicates inside
4:23:05the data and SQL just combine the two
4:23:07tables. Okay. So now we're going to
4:23:08understand how SQL execute union all in
4:23:10order to combine data. All right. Again
4:23:12we have the two results from queries. We
4:23:14have the employees and customers and SQL
4:23:16going to do the same steps. First going
4:23:17to go and get the column names from the
4:23:19first query and put it in the output.
4:23:21It's still going to go and take all the
4:23:23employees and put it in the output
4:23:25without checking anything. So that means
4:23:27if there is duplicates in the data, it's
4:23:29going to be presented as well in the
4:23:30output. It's very simple. Now it's going
4:23:33to go to the second step and as well
4:23:34take all the customers and append it
4:23:37into the output like this. So that's it.
4:23:40It's very fast. It's going to go and
4:23:42just combine all the rows from the
4:23:44employees and all the rows from the
4:23:45customers. And with that, we're going to
4:23:46get that 10 persons. And as you can see,
4:23:48we have duplicates in the data. So we
4:23:50have marry twice and given as well
4:23:52twice. And that's why union all is the
4:23:54fastest. It doesn't have any extra steps
4:23:56or checks. Just taking all rows from all
4:23:59queries and put it in the output. All
4:24:01right. So as you can see it's very
4:24:02simple, right? So that's all for the
4:24:04union
4:24:08all. Okay. So what is except sometime we
4:24:11call it minus in other databases but in
4:24:13SQL server we call it except. So it's
4:24:15going to go and return a distinct rows
4:24:17from the first query that are not found
4:24:20in the second query. So from this
4:24:22definition we can understand that the
4:24:24order of the queries can affect the
4:24:26final result. There is a first query and
4:24:29a second query. So it is the only set
4:24:31operator where you have to pay attention
4:24:33to the order of the queries. And as well
4:24:35it's like the others. It's going to go I
4:24:37remove the duplicates from the result
4:24:39set. All right. Again we have this very
4:24:40simple example. We have two sets, five
4:24:42customers, five employees and there is
4:24:44the same persons as a customer and as
4:24:47employees Kevin and Mary. So now we're
4:24:48going to go and combine those two sets
4:24:50using the excepts or sometime we call it
4:24:52minus. So it says it's going to return
4:24:54unique rows in the first table that are
4:24:56not in the second table. So what going
4:24:58to happen? What is the first table?
4:25:00Let's say the customers on the left
4:25:01side. So here we have five persons.
4:25:03Joseph, Mark, Anna, Kevin and Mary. So
4:25:06now the rule is we need the customers
4:25:08that are not employees. So it's safe for
4:25:11Joseph, Mark and Anna because they are
4:25:13not existing in the second set. That's
4:25:16why SQL going to return those three
4:25:18values. But now for the two customers
4:25:20given and marry here there is an issue.
4:25:22Given and marry they are members of the
4:25:24second set. The second table the
4:25:26employees. That's why SQL going to go
4:25:28and exclude them from the output because
4:25:31they are not fulfilling the rule. So in
4:25:33the output we will get only three
4:25:35customers and all the values from
4:25:37employees and the common values between
4:25:39customers and employees will be excluded
4:25:42from the output. So this is how the
4:25:45except works. All right. So let's have a
4:25:47very simple skill task and it says find
4:25:49the employees who are not customers at
4:25:52the same time. Okay. So let's see how
4:25:53we're going to solve that. We're going
4:25:55to stay with the same queries as usual.
4:25:56We have the employees and the customers
4:25:59but instead of having union all we're
4:26:00going to use the set operator except. So
4:26:03now since we are using except we have to
4:26:04make sure that the order of the queries
4:26:06are correct. So the first query is the
4:26:09employees which is correct because we
4:26:11have to find the employees who are not
4:26:14customers at the same time. So we are
4:26:15focusing on the employees. The first
4:26:17table is correct and the second table is
4:26:20customers. If the task says find the
4:26:22customers who are not employees at the
4:26:24same time then we have to go and switch
4:26:26it. We have first to query the
4:26:28customers. So now everything is correct.
4:26:30Let's go and execute it. And now in the
4:26:32output we see three employees who are
4:26:34not customers at the same time. So we
4:26:36have Carol, Frank and Michael. But as we
4:26:38know we have five employees Kevin and
4:26:40Mary. They are not here in the result
4:26:41because they are customers as well. So
4:26:44now let me show you what can happen if I
4:26:46just switch those informations. So we
4:26:48start with customers and then with
4:26:50employees. Let's go and execute it. As
4:26:52you can see, we're going to get
4:26:53completely different results. Now we are
4:26:55getting customers informations. And now
4:26:57in the output, we got three customers
4:26:59who are not employees at the same time.
4:27:01This is not what we want from this task.
4:27:03So if you do it like this, it's going to
4:27:05be incorrect. So pay always attention
4:27:07here to the order of that query. So now
4:27:09let's go and correct it. So we're going
4:27:10to have first employees and then
4:27:12customers. Let's execute it. And now
4:27:14let's go and understand how SQL execute
4:27:17the except operator. All right. So again
4:27:19we have the results from the two queries
4:27:20or from two tables and now we are doing
4:27:23except between them. So let's see how is
4:27:25going to execute it. It's going to take
4:27:26as usual first the names from the first
4:27:29query from the employees and put it in
4:27:31the output. And now SQL going to present
4:27:33data only from the first query in the
4:27:36output. And it going to go and use the
4:27:38customers only as a check. So SQL will
4:27:41not put any data or rows from the
4:27:43customers. It will just use the second
4:27:45query as a lookup in order to check the
4:27:47data. So, it's going to start with the
4:27:49first employee, Frankly. Do we have
4:27:52Frankly in the customers? Well, no, we
4:27:54don't have it. That's why it's going to
4:27:56accept it and put it in the output. And
4:27:58then in the next step, it's still going
4:28:00to go to the second employee and check.
4:28:02As you can see, we have it already in
4:28:03the customers. So, SQL going to go and
4:28:05ignore it. It's not allowed to be in the
4:28:07output. The same thing for Mary. We have
4:28:10it as well in the customers. That's why
4:28:12it will not be presented in the output.
4:28:14So Michael, we don't have a Michael in
4:28:17customers. That's why it can be
4:28:19presented in the output. And as well for
4:28:21Carol, the same thing. We don't have
4:28:23Carol as a customer and we're going to
4:28:25have it in the output. So as you can
4:28:28see, we will get data only from the
4:28:29first table and the second table only
4:28:31going to be used in order to check the
4:28:33informations from it. So we don't have
4:28:35in the output any customers, it's only
4:28:36employees. So now let's check quickly
4:28:38what going to happen if we switch the
4:28:39tables. So now we have the customers as
4:28:41the first table. SQL going to take the
4:28:43columns from the first table and it's
4:28:45going to start presenting the customers
4:28:47informations in the output and going to
4:28:49go and use the employees only as a
4:28:51lookup. So do we have Joseph? We don't
4:28:53have it in the employee. And then Kevin
4:28:56and Mary we have it already in the
4:28:58employees and Mark and Anna are not part
4:29:01of the employees that's why can go and
4:29:04present the results in the output like
4:29:06this. So now as you can see SQL is
4:29:07focusing on the table customers and we
4:29:09are getting data from the customers not
4:29:11from the employees. Employees is only as
4:29:13a check. So with that we understand the
4:29:15order of the queries is very important
4:29:17for the exceptions. We will get
4:29:18different results if we have different
4:29:20order. All right. So that's all for the
4:29:22except
4:29:26operator. Okay. So what is intersect?
4:29:29Intersect going to go and return only
4:29:31row that are common in both queries.
4:29:34It's something very similar to the inner
4:29:36join and as well here it's going to go
4:29:38and remove duplicates. So there will be
4:29:39no duplicates in the output. All right.
4:29:41Again we have this very simple example
4:29:42where we have five customers and five
4:29:44employees and now we're going to combine
4:29:46them using the intersect. So what
4:29:48intersect does it going to go and return
4:29:50common rows between two tables. So how
4:29:53SQL going to execute it? It's very
4:29:55simple. SQL going to go and search for
4:29:57the common values. So what are the
4:29:59common values? It's given and marry and
4:30:02SQL going to return only those two
4:30:04values given and marry and all others
4:30:06going to be excluded from the results.
4:30:09It's very simple, right? It's going to
4:30:10go and return only the common values and
4:30:12this is how the intersect works in SQL.
4:30:15Okay, let's have this simple task and it
4:30:16says find the employees who are also
4:30:19customers. So we're going to have the
4:30:20same queries employees and customers but
4:30:23instead of having except we're going to
4:30:24go and use intersect. Since we are
4:30:27finding the common informations between
4:30:29the employees and customers it's very
4:30:31simple and straightforward. Let's go and
4:30:33execute it. And with that we're going to
4:30:34get the Kevin and Mary. This is the two
4:30:37persons that are at the same time
4:30:39employees and customers. And of course
4:30:41here we don't have to pay attention to
4:30:42the order of the queries. It's going to
4:30:44be the same if we say find the customers
4:30:46who are also employees. So if you go and
4:30:48just switch for example the customers
4:30:50with employees you will see that we will
4:30:53get the exact same results. So it
4:30:55doesn't matter which query is first
4:30:56again pay attention to the first query
4:30:58that define the names. So now let's
4:31:00understand how is scale execute
4:31:01intersects behind the scenes. Okay again
4:31:04our two tables and now we are doing
4:31:05intersects. So as usual SQL going to go
4:31:08and take the columns from the first
4:31:09query and now we're going to go and find
4:31:11the common data between those two
4:31:13results. So it's going to do it row by
4:31:15row. So we have the employee Frank. Do
4:31:16we have it as a customer? No. So it will
4:31:19not be in the output. Given brown, we
4:31:21have it in the employees and as well as
4:31:23a customer over here. So that's why we
4:31:25will get it in the output. The same
4:31:27thing for Mary. So we have Mary as
4:31:29employee and as well as customer. So
4:31:31we're going to have it in the output.
4:31:33Michael and Carol, they are not
4:31:35customers. They are only employees.
4:31:37That's why we will not get it in the
4:31:38output. The same thing goes for the
4:31:40customers. Joseph, we don't have Mark.
4:31:42We don't have Anna because they are not
4:31:44employees. So with that we're going to
4:31:46get only the common informations between
4:31:48the two tables or two queries and it
4:31:50doesn't matter whether we start with
4:31:51customer or with employees we will get
4:31:53at the end the same information. All
4:31:55right so that's all it's very simple
4:31:57right this is how the intersect works in
4:31:59SQL.
4:32:04All right friends, so now we come to the
4:32:05part where I'm going to show you how I
4:32:07usually use the set operators in my
4:32:10projects for data analyszis or for data
4:32:12engineering. So here are the most
4:32:14important use cases for the set
4:32:16operators. All right, the first use case
4:32:18is combining similar tables before doing
4:32:20data analyzes. In some scenarios, we
4:32:23want to generate a report and we end up
4:32:25writing similar queries on top of
4:32:27similar tables and we go at the end and
4:32:29join all the results from the queries in
4:32:32order to present the final report. And
4:32:34now instead of doing that what we can do
4:32:36first we can go and combine all the
4:32:38similar informations into one table and
4:32:40then we can do on top of it a query a
4:32:42data analyzes in order to generate a
4:32:44report and we can do that using the
4:32:47union or union all. Let's have few
4:32:49examples. So let's say that we have four
4:32:51tables employees, customers, suppliers
4:32:53and students. So as you can see all of
4:32:55them are sharing the same informations.
4:32:57They hold data about persons. So now
4:33:00let's say that you are generating a
4:33:01report that requires all the individuals
4:33:04in the organization in the database. So
4:33:06what you're going to end up doing is
4:33:08writing SQL query for the employees,
4:33:10another one for customers and as well
4:33:13for the suppliers and the students. And
4:33:15then you're going to go and merge all
4:33:16the results from those queries into the
4:33:19final report. Now the issue with this
4:33:21setup is that you are having a lot of
4:33:23queries, a lot of similar queries. So
4:33:25you have it here four times. And now
4:33:27what might happen is that you go and
4:33:29change the logic of the first two
4:33:31queries and you forget later to do it
4:33:33for the other two and you will get
4:33:35really inconsistent data in the reports.
4:33:37So instead of that what we can do we can
4:33:39go and use the set operators in order to
4:33:42combine first all those tables in one
4:33:45big table. So what we're going to do
4:33:46we're going to go and use a union in
4:33:48order to combine those four tables into
4:33:51the table persons. So we're going to
4:33:53have it like this. So we will get all
4:33:54the rows from the employees and put it
4:33:56in the persons all the rows from the
4:33:58customers from the suppliers and as well
4:34:01from the students and put everything in
4:34:03one big table that holds all the
4:34:06informations about the individuals that
4:34:07we have inside our database. And now the
4:34:10next step after we combine the data now
4:34:12we write an SQL query in order to
4:34:14analyze this new big table and the
4:34:17result going to be presented in the
4:34:19reports. And now of course the advantage
4:34:21here is that we have only one SQL query
4:34:23for the data analyzers on top of this
4:34:26table instead of having it four times.
4:34:28And now if you go and change the logic
4:34:30of the SQL query, it going to be applied
4:34:32automatically on all the data that we
4:34:34have in the database. And we have done
4:34:36already this example where we have
4:34:37combined the data between the employees
4:34:39and customers. Another scenario where we
4:34:41have to combine data before doing any
4:34:42reporting. That's sometimes the database
4:34:45developers tend to divide a table one
4:34:47big table into multiple small tables in
4:34:50order to optimize the performance. For
4:34:52example, here splitting the orders by
4:34:54the year. We have orders 2022 2023. Now
4:34:58again here if you want to generate a
4:34:59report in order to analyze the orders
4:35:02over the years over the time either
4:35:04you're going to go and make a query for
4:35:06each of those tables or you're going to
4:35:08go first combining all those tables into
4:35:11one table called orders. So what we're
4:35:13going to do we're going to use a union
4:35:15between all those tables in order to
4:35:17generate one central table called the
4:35:19orders. So all the rows from the first
4:35:21table and all rows from the next table.
4:35:25next one and the last one. So, we're
4:35:27going to put everything in one big table
4:35:30and once we have the orders, we're going
4:35:31to go and write analytical skill query
4:35:34on top of the orders in order to
4:35:36generate the report. So, as you can see,
4:35:39it's very important step in order to
4:35:40prepare the data before doing data
4:35:42analyszis. Okay. So now let's have the
4:35:44following SQL task and it says the
4:35:46orders are stored in separate tables. We
4:35:49have the orders and orders archive. Now
4:35:51combine all orders data into one report
4:35:54without duplicates. Okay. So by looking
4:35:56to the task we have to combine two
4:35:58tables orders and orders archive. So
4:36:00either union or union all. But since the
4:36:02task says without duplicates that means
4:36:05we have to go with the union. But now
4:36:07before we combine any data we have first
4:36:09to understand the content of the orders
4:36:11and the orders archive in order to map
4:36:13the columns correctly. So first we have
4:36:15to go and explore the two tables. So
4:36:18let's start with selecting the data from
4:36:21orders everything semicolon and as well
4:36:24from the second table sales orders
4:36:28archive and as well semicolon. So let's
4:36:31go and execute it. So now in the output
4:36:33we get two results because we have two
4:36:35separate queries. The first result is
4:36:37for the orders and the second one is for
4:36:39the orders archive. Let me just make it
4:36:42a little bit bigger. And now as you can
4:36:44see we have almost identical tables. So
4:36:47as you can see we have the order ID,
4:36:48product ID, customer ID. So everything
4:36:51looks like identical and of course we
4:36:53can go and check that using the object
4:36:55explorer on the left side. So we have
4:36:57here the orders and those are the
4:37:00columns. And if you go to the orders
4:37:02archive, you can see that we have the
4:37:06exact same columns. So that means we can
4:37:08go and map all columns from orders with
4:37:10the all columns of orders archive. So
4:37:12let's go and do that. So I'm just going
4:37:14to remove all semicolons and then we're
4:37:16going to go and use the union. So now we
4:37:18have everything in one query. Let's go
4:37:21and execute it. Now we will get in the
4:37:23output one single results, one single
4:37:25table with all informations from orders
4:37:28and orders archive. So we have all
4:37:30orders now in one table and everything
4:37:33currently is matching. So with that we
4:37:35have solved the task. We have one result
4:37:36with all orders. We don't have any
4:37:38duplicates since we are using union and
4:37:41we have combined the data. But now we
4:37:42have one issue with that. This solution,
4:37:45this query is quick and dirty and
4:37:47actually it's not following the best
4:37:49practices. So now the best practices
4:37:50here is to list clearly all the columns
4:37:54in each query without using star. All
4:37:56right. So now let's go and do that. Now
4:37:57we need a list of all columns from the
4:38:00table orders and the table orders
4:38:01archive. And since we have a lot of
4:38:03columns, what we're going to do, we go
4:38:04to object explorer, right click on the
4:38:07table name, and then let's go select the
4:38:09top thousand rows. So let's click on
4:38:11that. And now we're going to get a very
4:38:13simple select statements where we have
4:38:15all the column names from the table
4:38:17orders. This is what I usually do if I
4:38:19need all the columns in the my select
4:38:21statements. So let's go and copy it and
4:38:23go back to our query. Then let's go
4:38:26replace the first star with those
4:38:29columns. And we're going to do the same
4:38:30thing as well for the orders archive
4:38:33since they have the same names. So let's
4:38:35go and do that as well. So let me just
4:38:37make this smaller in order to see the
4:38:39query. So now we have a select for the
4:38:42table orders with all columns and as
4:38:45well a select with all columns for the
4:38:47table orders archive. So let's go and
4:38:49execute it. And of course now we're
4:38:50going to go and get the same results.
4:38:52Now you might ask why we are doing this.
4:38:54Why didn't we stick with the star? It's
4:38:56quick. It's simple. Well for the
4:38:58following reason. So now currently the
4:39:00status is that everything is matching.
4:39:02We have 100% identical tables. But what
4:39:05happened with the time is that we do
4:39:07development in our solution and we might
4:39:09go and change the schema of the table
4:39:11orders. So we might rename stuff, we
4:39:14might add new columns or maybe switch
4:39:16the columns. So this means the table
4:39:19order with the time will not be anymore
4:39:21identical with the archive. And this is
4:39:23of course a problem if you are mapping
4:39:25the data blindly using the star. So now
4:39:28let me show you what I mean. Let's say
4:39:29that in this table we are developing the
4:39:32orders and we just switch those two
4:39:34columns in the schema for some reason.
4:39:37So now we have the product ID first and
4:39:39then the order ID. So let's go and
4:39:42execute it. Now if you are using star
4:39:44you will not notice this informations.
4:39:46But if you are using script you're going
4:39:47to see immediately that here we have
4:39:49first the order ID and then product ID.
4:39:52And here we have the opposite. So it's
4:39:54more clear listing the columns than
4:39:57using the star. And now as you can see
4:39:58in the output you can see that we have a
4:40:01problem that here we have order ids and
4:40:04then suddenly we have something like the
4:40:07product ID. So we're going to have
4:40:08incorrect data which leads to incorrect
4:40:10analyzes. So here the best practices to
4:40:12not use the star and to clearly list all
4:40:15the columns. Now one more technique that
4:40:17I usually use once I'm combining data is
4:40:19that I add the source of the data inside
4:40:22the query. So what I mean with that now
4:40:24you can see that we have here two orders
4:40:26with the order ID one they are not
4:40:29duplicates they are completely different
4:40:30informations and that's because they
4:40:32come from different tables. So what I
4:40:34usually do I go and add the source of
4:40:36each record it's really nice information
4:40:38for the analytics for the users to
4:40:40understand where these records come
4:40:42from. So how we going to do that? We're
4:40:44going to have for example on the first
4:40:46column the following word let's say
4:40:49orders and we're going to call it let's
4:40:51say that's source table and we're going
4:40:54to do the same thing as well in the
4:40:57second query. Right? So the source table
4:40:59here is not the orders it's the orders
4:41:02archive. So I'm just adding a static
4:41:04columns to my query in order to see the
4:41:07source of the table. So now we have here
4:41:09two different values. And let's go and
4:41:12execute it. And now you see we have
4:41:13created a new column called source table
4:41:15where it has only two values. We have
4:41:17the orders and the orders archive. Let's
4:41:19go and sort the data by the order ID. So
4:41:22order by order ID. So let's go and
4:41:25execute it. And now you can see it very
4:41:27clearly. The first order order ID one
4:41:30comes from the table orders and the
4:41:32second one comes from the orders
4:41:34archive. So this is really nice
4:41:35information that you can add to your
4:41:37data once you are combining multiple
4:41:38tables. So that's all about this use
4:41:40case on how to combine data between
4:41:42different
4:41:46tables. All right. Now we have another
4:41:48use case for the set operators. It's
4:41:50more for data engineers. We can use the
4:41:52except in order to find the delta
4:41:55between two batches of data. For
4:41:57example, data engineers build data
4:41:59pipelines in order to load daily new
4:42:02data from the source systems to a data
4:42:04warehouse or a data lake. Now, in those
4:42:06data pipelines, we have to build a logic
4:42:09in order to identify what are the new
4:42:11data that is generated from the source
4:42:13system in order to insert it in the data
4:42:16warehouse. One way to do it is to use
4:42:17the set operator except in order to
4:42:20compare the current data with the
4:42:22previous load. Let's have a very simple
4:42:24example. So in the day number one we
4:42:26have two customers one and two. So what
4:42:28going to happen in this day we're going
4:42:29to go and load those two customers into
4:42:31the data warehouse. So in the data
4:42:33warehouse we will get as well one and
4:42:35two. So this is for the first day
4:42:36nothing is crazy. We just load the data
4:42:38as it is. Now for the second day we will
4:42:41get the new data from the source system
4:42:42and it's going to look like this. So now
4:42:44if you check the second day you can see
4:42:45that we have again the customer number
4:42:47one we have already loaded to the data
4:42:49warehouse. So we have it as the previous
4:42:52day but we have a new customer ID number
4:42:55three. So now in order to load only the
4:42:57new data we don't need to load again the
4:42:59customer number one. What we can do? We
4:43:02can do an accept between the day number
4:43:05two with the previous load with the day
4:43:07number one. So now if we simply do an
4:43:09accept between those two sets we're
4:43:11going to go and identify the new data
4:43:14that is existing in the source system
4:43:16which is only the record number three.
4:43:18So now what going to happen if we do
4:43:19except between day two and day one we
4:43:22will get one record the new record that
4:43:24we're going to go and insert it inside
4:43:26our data warehouse. So as you can see
4:43:28this set operator except is very
4:43:30powerful in order to compare two sets
4:43:32and not only for data analysis we can
4:43:34use it as you can see for data
4:43:35engineering in order to identify what is
4:43:38the new data that is generated from the
4:43:40sources in order to insert it inside our
4:43:43data warehouse.
4:43:48Okay, one more use case for the set
4:43:49operators that I personally use a lot in
4:43:51my project is that if you are doing data
4:43:54migrations, you can use the accept in
4:43:56order to check the data quality and more
4:43:59specifically we can use it in order to
4:44:00check the data completeness. Okay, so we
4:44:03have the following scenario where we are
4:44:04doing data migrations between two
4:44:06databases. So let's say that we would
4:44:07like to move this table from database A
4:44:09to database B. So we're going to go and
4:44:12load the table to the new database. And
4:44:15now what is very important after you
4:44:16move the data is that to check whether
4:44:18all the records did move from database A
4:44:21to database B we are not missing
4:44:23anything even one record. So we want to
4:44:25do data completeness test and there are
4:44:27many methods on how to do this test. One
4:44:30of them is to use that set operator
4:44:32except. So how we going to do it? We're
4:44:33going to do an except between the table
4:44:36from database A and the table from
4:44:37database B in order to find any record
4:44:40that is still in database A which is not
4:44:43migrated to the database B. And of
4:44:45course the best result is that we will
4:44:47not get anything. The result should be
4:44:50empty. If we get an empty that means all
4:44:52the rows from database A exists in the
4:44:55database B. And now of course we are not
4:44:57done yet. We want to do the comparison
4:44:59but the way around. We want to find any
4:45:01new rows that is in database B that we
4:45:04don't find in database A. Those two
4:45:07tables must be identical. So now what
4:45:09we're going to do, we're going to do an
4:45:10except but the first table going to be
4:45:12from the database B. And then we're
4:45:14going to compare it with the database A.
4:45:15And we have the same expectation. The
4:45:18output should be as well empty. And now
4:45:20after doing the except twice for both
4:45:22sides and we are getting empty in the
4:45:24results. That means those two tables are
4:45:27identical and we are not missing
4:45:29anything. So this is another amazing use
4:45:31case for the set operators in order to
4:45:34improve the quality of your data
4:45:35migrations and in order to do data
4:45:37completeness
4:45:41test. Okay. So now let's have a quick
4:45:44summary about the set operators. So the
4:45:46set operator is going to go and combine
4:45:47the rows of multiple queries, multiple
4:45:49tables into one single result. And we
4:45:52have four different types of the asset
4:45:53operators. The first one is the union
4:45:56where it's going to go and combine all
4:45:57the rows but without including any
4:45:59duplicates. The second one we have the
4:46:01union all it's very similar. And the
4:46:03third one we have the except it's going
4:46:05to show all the rows from the first
4:46:06query that cannot be found in the second
4:46:09query. And the fourth one we have the
4:46:11intersect where it's going to show the
4:46:12common rows between two queries. And of
4:46:15course we have SQL rules in order to use
4:46:16the set operators. Both of the queries
4:46:18should have the same number of columns,
4:46:21the same data types and the order of
4:46:23columns. And the last rule, don't forget
4:46:25that the first query controls the
4:46:27aliases, the name of the columns and the
4:46:30data types of the entire result. And we
4:46:32have found amazing use cases for the set
4:46:34operators. Like for example, using union
4:46:36and union all in order to combine
4:46:39similar informations into one big table.
4:46:41Or we can go and use the amazing except
4:46:43operator in order to compare two
4:46:46different results in order to find the
4:46:48differences between them. And I usually
4:46:50use it in order to do data quality
4:46:52checks to test the data completeness.
4:46:54And another use case as a data engineer
4:46:56you can go and implement the except in
4:46:58your logic in your data pipelines in
4:47:01order to identify what are the new data
4:47:03that must be inserted in your system.
4:47:05Okay my friends. So with that we have
4:47:06learned all the set operators that we
4:47:08have inside SQL. And with that you have
4:47:10learned how to combine your data from
4:47:12multiple tables using SQL. So we are
4:47:14done with this chapter. Now we're going
4:47:16to go to the right side. So now we're
4:47:18going to start talking about the
4:47:19functions in SQL. And here we have two
4:47:21big families. The first one is the row
4:47:24level or the single value functions. And
4:47:26the second one we have the aggregate
4:47:27analytical functions. So let's start
4:47:29with the first one the rowle functions.
4:47:31And here we can group them into multiple
4:47:33categories. And we will start now with
4:47:35the string functions. But first let's
4:47:37understand what is exactly functions and
4:47:39why do we need them in SQL. So let's
4:47:45go. Okay. So what is exactly function
4:47:48and why we need it. Now again we have
4:47:49our data inside the table. Now there is
4:47:52like a lot of stuff that you can do with
4:47:53your data. So sometimes you have to
4:47:55change the values of your data like
4:47:57doing data manipulation or you want to
4:47:59do some aggregations and analyzes. So
4:48:01maybe you want to analyze your data and
4:48:03find insights and maybe build reports
4:48:06and sometimes you might find bad data
4:48:08inside your tables and you want to clean
4:48:09that up. So you want to do data
4:48:11cleansing and sometimes you have to do
4:48:13data transformations and data
4:48:15manipulation on our data in order to
4:48:17solve some SQL tasks and in SQL in order
4:48:20to solve those tasks we have functions.
4:48:22So again what is exactly a function? It
4:48:24is a built-in code block that accepts an
4:48:27input value. Then the function going to
4:48:29go and process this value and it going
4:48:31to return a result an output value. So
4:48:34you give an input value do some
4:48:36transformations and give an output. And
4:48:38we can group the functions into two big
4:48:40categories. The first one we call it
4:48:42single row functions. So you give the
4:48:44function only one value and at the
4:48:46return you will get as well one value.
4:48:48So the input for the function going to
4:48:49be only one single value like maria and
4:48:52the output of the function going to be
4:48:54as well single row value. So one value
4:48:56in one value out. And now the other
4:48:58category of functions we call it
4:49:00multirow functions. So for example if
4:49:02you have the function sum this function
4:49:04accept multiple rows multiple values
4:49:07like it gets 30 10 20 40 the function is
4:49:10then going to go and summarize all those
4:49:12rows and return in the output only one
4:49:14value. The summarization of all those
4:49:16values going to be 100. So the input is
4:49:18multiple rows and the output is one
4:49:21single value. So those are the two main
4:49:22categories of functions in scale.
4:49:28Now my friends you have to understand
4:49:30something about the functions that you
4:49:31can go and nest functions together. So
4:49:34you can use multiple functions together
4:49:36in order to manipulate one value. And
4:49:38this technique is not only in SQL in any
4:49:40programming language. So let's have this
4:49:42example. We have the function left. It's
4:49:44going to go and extract like few
4:49:45characters. Let's say two characters. So
4:49:47the input for this function let's say
4:49:49it's Maria. This value going to enter
4:49:50the function. The function is going to
4:49:52go and extract the first two characters.
4:49:54And in the output we will get only two
4:49:56characters m a. So this is one function.
4:49:58We have an input and output. Now you
4:50:00might say you know what we have multiple
4:50:02steps on this value. So the first step
4:50:04we want to extract the first two
4:50:05characters using the lift function. But
4:50:07we have a second step. So we want to
4:50:09transform this output into a lowercase
4:50:12characters. So we have another function
4:50:13lower and the input for this second
4:50:15function will be the output of the first
4:50:17function. So ma it is at the same time
4:50:20output and input for another function.
4:50:22So the lower function going to take this
4:50:24value and convert it into lowerase
4:50:26character. So it's like inside the
4:50:28factory the materials going to be
4:50:29processed into multiple stations and the
4:50:31output of one station going to be the
4:50:33input for the next station. And this is
4:50:35exactly what we can do with the
4:50:36functions. So now how we going to build
4:50:38that? The first step is to start with
4:50:39the first function. So this is simple
4:50:41one function. Now for the next step what
4:50:44you're going to do on the left side
4:50:45you're going to write lower and put the
4:50:47whole thing in parenthesis. So now the
4:50:49whole thing the first function going to
4:50:51be inside another function and with that
4:50:53you have nested one function in another
4:50:55and of course if you need a third
4:50:57function like for example the length
4:50:59what you're going to do you're going to
4:51:00put the whole thing again between two
4:51:02parentheses. So now that means the
4:51:03output of the lift going to go to the
4:51:05lower and the output of the lower going
4:51:07to go to the length. So it is very
4:51:09simple and the order of the execution
4:51:10for this will start always in the inner
4:51:13function. So the lift function going to
4:51:14be executed first and then the outside
4:51:16function the lower and the last function
4:51:18that's going to be executed is the
4:51:20length. This is how the nested functions
4:51:22works in SQL or in any programming
4:51:27language. Now my friends in SQL we have
4:51:30a lot of functions that's why we have to
4:51:32group them as well into subcategories.
4:51:34Like if you are talking about the single
4:51:36row functions, we have functions for the
4:51:38string values and as well for the
4:51:40numeric, the date and time and as well
4:51:42functions in order to handle the nulls.
4:51:44And if you are talking about the
4:51:45multirow functions, here we have
4:51:47basically two groups. The first one is
4:51:49the simple aggregate functions. Those
4:51:51are the basics in order to aggregate
4:51:52your data. And we have another advanced
4:51:54one. We call it the window functions or
4:51:57sometime we call it analytical
4:51:58functions. So now if I'm looking to
4:52:00those two groups and now my friends it
4:52:02is very important to understand those
4:52:03functions because using them you can do
4:52:05whatever you want with your data and if
4:52:07I'm looking to those two groups the
4:52:09single row functions those stuff here
4:52:11they are functions in order to
4:52:13manipulate and prepare the data for the
4:52:16second group. So if you are thinking
4:52:17about data engineers and data analysts
4:52:19the data engineers going to go and
4:52:21prepare the data in SQL using the single
4:52:23row functions. So you're going to use
4:52:25them in order to clean up, transform,
4:52:27manipulate your data in order to prepare
4:52:29it for the analyzes. And if you are data
4:52:32analyst, you will be mostly using the
4:52:33aggregate functions in almost every
4:52:35task. So I really see it like this. The
4:52:37single row functions for data engineers
4:52:39and multirow functions for data
4:52:41analysts. And my friends, what we're
4:52:43going to do in this course, we're going
4:52:44to visit each of those subgroups one by
4:52:47one, exploring the functions,
4:52:49understanding how they work and when
4:52:50we're going to use them. So let's start
4:52:52with the first group, the string
4:52:53functions. And here we're going to learn
4:52:55how to manipulate the string values. So
4:52:58let's
4:53:02go. Okay. So now since we have a lot of
4:53:04string functions, I'm going to go and
4:53:06divide them into categories based on the
4:53:08purpose. So for example, we have a group
4:53:10of functions that's going to go and
4:53:11manipulate the string values. So we have
4:53:13concatenation, upper, lower, replace,
4:53:15and so on. And another group where we
4:53:17have only one function. It is where we
4:53:19can do calculations on the string
4:53:20values. And the last group, it is all
4:53:22about how to extract something from a
4:53:25string value. And here we have three
4:53:26functions left, right, substring. So now
4:53:28let's go and start with the first group
4:53:30about the data manipulation. And the
4:53:32first function we have here
4:53:36concat. All right. So what is exactly
4:53:39concat or concatenation? It's going to
4:53:41go and combine multiple string values
4:53:43into one value. So if you have multiple
4:53:46things you can put everything in one
4:53:47value. So let's have a very simple
4:53:49example. Okay. So now let's say that you
4:53:50have one value called Michael. So here
4:53:52you have a first name and you have
4:53:54totally separated value for the last
4:53:56name another column where you have a
4:53:58value like Scott. And now you say you
4:54:00know what it makes no sense to have the
4:54:01first name separated from the last name.
4:54:03I would like to go and combine them in
4:54:05one value. So you can go and use the
4:54:07concat in order to combine those two
4:54:09values or multiple values into one
4:54:11single value like Michael Scott.
4:54:14I think that pretty much sums it up. So
4:54:17it is nicer to see the full name in one
4:54:19value instead of having like two columns
4:54:21for that. So that's it. This is why we
4:54:23need the concatenations. Now let's go
4:54:25back to scale in order to try that out.
4:54:27Okay. So now we have the following task.
4:54:29Show a list of customers first names
4:54:31together with their country in one
4:54:33column. So that means we have to make a
4:54:35list of customers and we have to combine
4:54:37two columns in one. So let's start
4:54:39writing the query. Select. We need the
4:54:41first name, the country from the table
4:54:44customers. So first let's go and execute
4:54:46this. Now as you can see we have list of
4:54:48customers but the issue here the first
4:54:50name and the countries those two
4:54:52informations are in different columns
4:54:54but the task says they should be in one
4:54:56column. So now in order to combine those
4:54:58two things we have to use the
4:54:59concatenate function. So concat. So I'm
4:55:02going to start with the first argument.
4:55:03It's going to be the first name and then
4:55:05the country like this. And we're going
4:55:07to give it a name. Let's call it like
4:55:09this name country. Now let's go ahead
4:55:12and execute it. Now in the output you
4:55:13can see we have a new column. It's
4:55:15called name country and we have both of
4:55:17the informations in one column. So we
4:55:19have Maria, Germany, join USA. But it
4:55:21doesn't really look good because there's
4:55:23like no spacing between them. Now we can
4:55:25go and make some separation between them
4:55:28by just adding one more thing in between
4:55:30like for example maybe a space. So now
4:55:33we are concatenating the first name
4:55:35together with a space this over here and
4:55:38then the country. So let's go and
4:55:39execute it. Now as you can see we have
4:55:41nice separations between the first name
4:55:43and the country. And of course you can
4:55:44go and add different separations like
4:55:46maybe my notes or underscore and you
4:55:48will get the same effect. So with that
4:55:50we have a list of customers where we
4:55:52have the first name together with the
4:55:54country in one column. As you can see
4:55:56it's very simple. This is how you
4:55:57combine two columns in one. It is really
4:56:00nice and easy transformation. Okay. So
4:56:01that's all about the concatenation in
4:56:03scale. Next we're going to talk about
4:56:05two functions. The upper and the
4:56:10lower. Okay. So what is upper function?
4:56:13It's going to go and converts all the
4:56:14characters of a string to an uppercase.
4:56:17It's going to make everything
4:56:18capitalized. And the lower function is
4:56:20exactly the opposite. It's going to go
4:56:21and convert everything to a lower case.
4:56:24So let's have very simple example for
4:56:26those two functions. Okay. So now we
4:56:27have like three values with different
4:56:29cases. The first one where you have only
4:56:31the first character capitalized and the
4:56:33rest is lowered and then the same value
4:56:35but everything is lowered and a third
4:56:37one where you have everything with an
4:56:39uppercase. Now if you go and apply the
4:56:41function upper to those three values
4:56:43what going to happen for the first value
4:56:45going to go and turn it into an
4:56:47uppercase. So everything going to be
4:56:48capitalized not only the first
4:56:50character. And now for the second value
4:56:52going to turn it as well to completely
4:56:54capitalized. So all the characters going
4:56:56to change. And for the last value it is
4:56:58already capitalized. So in the output
4:57:00you will get the same value. So actually
4:57:01nothing going to happen for that. So
4:57:03this is simply the uppercase. Now let's
4:57:05see what can happen if you use the lower
4:57:07case. For the first value only the first
4:57:09character going to be changed and then
4:57:10you will have everything in lower case.
4:57:12The second value it is already a
4:57:14lowerase value. So if you apply lower
4:57:16case nothing going to happen. You will
4:57:18get the same value. But for the last one
4:57:19everything here is capitalized and if
4:57:21you apply lower case all the characters
4:57:23going to convert to a lower case. So my
4:57:25friends this is very simple. Let's go
4:57:27back to your skill in order to practice
4:57:29that. Okay. So we have the following
4:57:30task and it says transform the
4:57:32customer's first name to lowerase. So
4:57:35now as you can see the first names here
4:57:37the first character is a capital the
4:57:39rest is lowerase. So now in this task we
4:57:41have to convert the whole thing into
4:57:43lower case. So let's go and do that.
4:57:45It's very simple. We're going to say
4:57:47lower first name and let's go and call
4:57:50it low name. So that's it. Let's go and
4:57:53execute it. Now if you go and compare
4:57:54the lower name with the first name, you
4:57:56can see all the characters now in the
4:57:59lower case. So that's it for the task.
4:58:01We have transformed the first name to
4:58:03lower case. All right. The next task is
4:58:05exactly the opposite. Transform the
4:58:06customer's first name to uppercase. So
4:58:09let's go and have a new column. We're
4:58:11going to say upper then the first
4:58:15name as app name. So that's it. It's
4:58:19very simple. Let's go and execute. Now
4:58:20you can see in the output we have a new
4:58:22column called up name and inside it we
4:58:24have the first name but now all the
4:58:26characters in upper case. So this is how
4:58:29you convert the case to lower or to
4:58:31upper in SQL. Okay. So that's all about
4:58:33the upper and the lower. Next we're
4:58:35going to talk about very interesting
4:58:36function. It is the
4:58:41trim. So the trim function going to go
4:58:44and remove the leading and trailing
4:58:46spaces in your string values. So it's
4:58:48going to go and get rid of the empty
4:58:50spaces at the start and at the end of a
4:58:52string value. Let's have very simple
4:58:54example. Okay. So now we're going to
4:58:55have different scenarios. The first one
4:58:56you can have like a value join where you
4:58:58don't have any spaces and this is the
4:59:00normal case. But sometimes you might
4:59:02have it like this where at the start you
4:59:04have a leading space. You have an empty
4:59:06space or sometimes we call it white
4:59:08space. In another scenario the space
4:59:10might be at the end of the word. So here
4:59:12we call it trailing space and in another
4:59:14scenario you might have both of them.
4:59:16This is really bad. where at the start
4:59:18you have the leading space and at the
4:59:20end you have the trailing space. And of
4:59:21course you might not have only one
4:59:23space, you might have multiple spaces
4:59:25depend on how long did the user press
4:59:27the space, right? So of course my
4:59:29friends spaces are really evil and this
4:59:31makes no sense to have it in your data.
4:59:33Now what you have to do is to do data
4:59:34cleansing. We have to clean up this miss
4:59:36and you have the best function in order
4:59:38to clean up the data. You have the trim.
4:59:40So if you apply trim for the first
4:59:42value, nothing going to happen because
4:59:44everything is clean and we don't have
4:59:45any spaces. Now if you apply it for the
4:59:47second case where you have a leading
4:59:49space if you do that SQL going to go and
4:59:52remove this space. The same thing for
4:59:54the trailing space. So if you have space
4:59:55at the end the trim function going to
4:59:57find it and clean that up. And if you
4:59:59have it at the start and at the end then
5:00:01it's as well no problem. It's going to
5:00:03go and clean that up. And as well the
5:00:05trim function can go and clean multiple
5:00:07spaces. So if you have like five spaces
5:00:0910 spaces at the end or at the start the
5:00:11trim function going to go and clean that
5:00:13up. So this is how the trim works. And
5:00:15now let's go back to our scale in order
5:00:16to find out whether we have any spaces.
5:00:18Okay. So now we have a very tricky and
5:00:21interesting task. It says find the
5:00:24customers whose first name contains
5:00:26leading or trailing spaces. So now by
5:00:29looking to those values we have to find
5:00:31any spaces inside the customer's name.
5:00:33Now by just looking to this results you
5:00:35will not find any white spaces because
5:00:38it's really hard to see especially if it
5:00:40is like trailing spaces. Now we have to
5:00:42write query order to detect any spaces
5:00:45in the names. So how we can do that?
5:00:48Okay. So now think about it a little bit
5:00:49and I can give you a hint. You can use
5:00:51the function trim in order to remove any
5:00:54white spaces and you have to use it
5:00:56inside a wear clause. So what we're
5:00:58going to do we're going to say where. So
5:00:59now we have to build a condition to
5:01:01detect any spaces. So if you are saying
5:01:04if the first name is not equal to itself
5:01:08first name after applying a trim. So
5:01:11after trimming the first name if it is
5:01:15not equal to the first name so that
5:01:17means there was spaces. So again what is
5:01:20going on here? Let's go for Maria. If
5:01:22Maria has no nulls if you trim this
5:01:24value nothing going to happen. The value
5:01:26going to stay exactly like before
5:01:28because there is no white spaces. But if
5:01:30in Maria there is any space inside it.
5:01:33Trimming the value will not be equal to
5:01:35the first name if it contains any
5:01:37spaces. So if the column is not equal to
5:01:40the same column after trimming it that
5:01:42means there is spaces. So let's go and
5:01:45execute it. And now we can see in the
5:01:47output we have one customer John where
5:01:49we have this situation. Now if you don't
5:01:51believe me or you don't follow me here
5:01:53we can have another easier check. So
5:01:55let's go and comment this out and let's
5:01:57have a look to our first names. Now we
5:01:59can go and calculate the length of the
5:02:02first name like we have done before. So
5:02:05length name and let's go and execute it.
5:02:08Now if you can see here Maria we have
5:02:09five characters but John we have here
5:02:12four characters but the length is five
5:02:14and that's because we have somewhere
5:02:16space and the space going to count as a
5:02:18character. So here there is like
5:02:20something wrong right and you can check
5:02:21the others as well everything is
5:02:23matching but only John we have here an
5:02:25issue and now in order to see this more
5:02:27clearly we're going to use two functions
5:02:29the trim and the length. So first let's
5:02:31go and trim the first
5:02:33name. And after trimming the values, I'm
5:02:36going to calculate the length. So we are
5:02:38nesting together the trim and the
5:02:41length. And I'm going to call it length.
5:02:43Trim name. So let's go and execute it.
5:02:47Now we can see the length before
5:02:49trimming any value. And we can see the
5:02:52length after trimming the values. So you
5:02:54can see over here that join before
5:02:56trimming is five and after trimming is
5:02:59four. So we have here an issue. Now we
5:03:01can make things more clear where we can
5:03:03go and subtract the length of the first
5:03:06name with the length of the first name.
5:03:09But first we trim the values. So here we
5:03:11can call it maybe a flag or something.
5:03:13So let's go and execute it. Now by
5:03:15looking to the flag it is really easy to
5:03:17now to see if we have a zero then
5:03:19everything is fine. We don't have any
5:03:21white spaces. But if we have higher than
5:03:23zero like here one then this is an
5:03:25indicator that we have a white space.
5:03:27Either you do it like this where the
5:03:29first name is not equal the first name
5:03:31after trimming or you use more
5:03:33complicated solution where you say where
5:03:36and I'm going to remove this from here
5:03:38the length of the first name is not
5:03:41equal to the length after trimming so
5:03:43not equal so if you go and execute it
5:03:46you will get exactly again join so this
5:03:48is how we detect any empty spaces inside
5:03:50our data using the trim function or
5:03:53maybe as well using the length but I
5:03:55really prefer the first solution it is
5:03:56way easier using one function. All
5:03:58right, so that's all about how to remove
5:04:00the empty spaces using the trim. Next,
5:04:02we're going to talk about very important
5:04:04function called
5:04:08replace. Now the replace function going
5:04:11to go and replace a specific character.
5:04:13So that means we have something old and
5:04:15we want to replace it with something
5:04:16new. Let's have a very simple example to
5:04:18understand it. All right. So now imagine
5:04:19we have a phone number where the data is
5:04:21splitted by a dash. Now let's say that I
5:04:24don't like to have the dash in my data.
5:04:25I would like to have slash like any
5:04:27other special character. Now in order to
5:04:29replace the dash, we can use the
5:04:31function replace. So we have to specify
5:04:33for SQL two things. The old value the
5:04:36dash with a new value the slash. So if
5:04:39you do that in the output it's going to
5:04:41go and remove all those dashes between
5:04:42the numbers and the replacement going to
5:04:44be the dash between them. So it's very
5:04:47simple, right? All what you are doing is
5:04:48replacing an old value with a new value
5:04:51and that's why we call it replace. But
5:04:53we can use this function as well in
5:04:54order to remove something not only we
5:04:57replace and you can do that by not
5:04:59specifying anything in the new value
5:05:01like just the single quotes and with
5:05:03that it's going to be nothing a blank.
5:05:05So now what's going to happen is still
5:05:06going to go and replace the dash with a
5:05:09blank and that means I'm just removing
5:05:11the dashes from the output. So if you do
5:05:13it you will remove the dash and you will
5:05:15get only numbers. So if the replacement
5:05:17going to be a blank then that means this
5:05:19function will be replacing any value
5:05:21that you specify. So this is exactly how
5:05:24it works and this is why we use the
5:05:26replace function in SQL. Now let's go
5:05:28back in order to practice. So let's do
5:05:30the same example. This time we're going
5:05:31to go and select from a static value. So
5:05:33we're going to get 1 2 3 4 5 6 7 8 9 0.
5:05:37So if you go and execute it, you can see
5:05:39we are getting the phone number. Now
5:05:41let's go and remove the dashes from this
5:05:43value. So let's have a new line and we
5:05:45start with replace. The first thing that
5:05:47you have to specify for SQL the value
5:05:49itself. So let's go and get the value.
5:05:51This is the first argument. The second
5:05:53argument going to be the old value. So
5:05:55the old value going to be the dash. And
5:05:57now the third argument will be the
5:06:00replacement. And since we want to remove
5:06:02it, we don't want to replace it with
5:06:04anything. We will have just single
5:06:05quotes and nothing between them. So
5:06:07there's no space between those single
5:06:09quotes. Now we can go and rename stuff
5:06:11like this is the phone. And this is a
5:06:13clean phone. Let's go and execute it.
5:06:16Now, as you can see in the output of the
5:06:18function, we don't have any dashes
5:06:20between the numbers. And you can go and
5:06:22test stuff. Like for example, I can go
5:06:25and add a slash and execute it. You will
5:06:27see slashes between them. So you can go
5:06:29and try multiple stuff. So this is one
5:06:31nice use case for the replace function.
5:06:33Now there is another use case for the
5:06:35replace function is that sometimes in my
5:06:37data file names going to be stored like
5:06:40for example, let's say reports.t txt and
5:06:43now let's say that I would like to
5:06:44change the file format from .txt to CSV.
5:06:48Now how we're going to do that we're
5:06:49going to go with a new line say replace
5:06:51and then the first argument going to be
5:06:53the value. So let's take our value from
5:06:56here and now what is the old value it's
5:06:58going to be the txt and I want to
5:07:00replace it with another format with
5:07:03another extension. So it's going to be
5:07:04the CSV. So we're going to say this is
5:07:07the new file name and this is the old
5:07:10file name. So let's go and execute it.
5:07:13And now as you can see in the output SQL
5:07:15did replace the txt with SCSV. This is
5:07:18as well where I use the replace function
5:07:20in my projects. So my friends the
5:07:22replace function is really fun and those
5:07:24are two nice use cases for the replace.
5:07:26All right. So that's all about the
5:07:28replace function in SQL and with that we
5:07:30have covered the whole datamations. Now
5:07:32in the next group we're going to talk
5:07:33about the calculations. And here we have
5:07:35only one function the
5:07:40length. Now the length function it's
5:07:42very simple. It's going to go and count
5:07:44how many characters you have in one
5:07:46value. So you are calculating the length
5:07:48of a value. Let's have very simple
5:07:50example to understand it. Okay. So now
5:07:52let's say that we have the value Maria.
5:07:53If you apply the length function for
5:07:55that what's going to happen? It's going
5:07:56to go and start counting how many
5:07:58characters we have inside this value. So
5:08:00the m is 1. a 2 3 4 5 in the output you
5:08:04will get the number five. So five is the
5:08:07length or the total number of characters
5:08:08in this value. Now let's say that you
5:08:10have a number like 350. If you go and
5:08:12apply the length function still is going
5:08:14to go and count how many digits do we
5:08:16have. The three is 1 5 2 3. So the total
5:08:19length for that going to be three. So
5:08:21you can apply it even for numbers and
5:08:23not only that you can go and apply it on
5:08:25a date value. So let's say that you have
5:08:26the following date 2026 1st 23. So SQL
5:08:30going to go and count each digit each
5:08:32character even the underscores not only
5:08:34the numbers underscore is as well a
5:08:36digit right? So the total length of this
5:08:38date it's going to be 10. So you can
5:08:40apply any data type to the links
5:08:42function and in the output you will get
5:08:44always a number. That's it. This is how
5:08:46you can count the number of characters
5:08:47in any value. Let's go back to scale in
5:08:49order to practice that. Okay. So now we
5:08:51have the task calculate the length of
5:08:53each customer's first name. So it is
5:08:56very simple. We're going to go and apply
5:08:57the function length len to the column
5:09:01first name and we're going to call it
5:09:03length name. So let's go and execute it.
5:09:06And with that as you can see we are
5:09:07getting in the output numbers and these
5:09:09numbers are the number of characters of
5:09:12each name of our customers. So this is
5:09:14how we calculate the length and that's
5:09:16it for this group. Now moving on to the
5:09:18next one. It's going to be very
5:09:19interesting. Now we're going to talk
5:09:20about how to extract something from a
5:09:22string value. And here we're going to
5:09:24cover now two functions the left and the
5:09:29right. Now the lift function going to go
5:09:32and extract specific number of
5:09:34characters from the start of a string
5:09:36value. So if you want to get few
5:09:38characters at the beginning of a value,
5:09:40you can use the lift. But now the right
5:09:42function is exactly the opposite. It's
5:09:44going to go and extract specific number
5:09:46of characters from the end of string
5:09:49value. So if you want few characters
5:09:51from the end of your value, you can use
5:09:53right. Now in order to apply the left or
5:09:55the right function, you have to give SQL
5:09:57two things. The value where you want to
5:09:59extract a part from it and the number of
5:10:01characters, how many characters you want
5:10:03to extract and this is the same for the
5:10:04left and the right. Now let's say that
5:10:06we have again this value Mariam. And now
5:10:08if the task says I would like to extract
5:10:10the first two characters and since we
5:10:12are talking about the starting position,
5:10:14we're going to use the lift function.
5:10:16And since it says two characters, we're
5:10:17going to go with the two. So it's going
5:10:19to start counting M is 1, A is two and
5:10:22after that it's going to stop and make a
5:10:23cut and it's going to go and return the
5:10:25two characters M A. So we are counting
5:10:27from the left side going to the right
5:10:29side. Right now if your task says
5:10:32extract the last two characters here we
5:10:34are talking about the end position of
5:10:36your value and for that we're going to
5:10:38use the right function since we are
5:10:40approaching from the right side and
5:10:42since we want only two characters the
5:10:43number of characters going to be two. So
5:10:45this time going to start counting from
5:10:46the right side moving to the left side.
5:10:49So A is one, I is two and that's it.
5:10:52Then SQL going to stop and extract only
5:10:54those two characters. I A. So if you
5:10:56want to extract data at the starting
5:10:58position, you use the left. But if you
5:10:59want to extract characters from the end
5:11:01position of your value, then you use the
5:11:04right function. Now let's go back to
5:11:05scaler in order to practice. Okay. So
5:11:07now we have the following task. Retrieve
5:11:09the first two characters of each first
5:11:11name. So we just need the first two
5:11:13characters. Since we are coming from the
5:11:15left side, we can go and use the
5:11:17function left. So it's very simple.
5:11:19First name and we need only two
5:11:22characters. So two. So we're going to
5:11:23call it first to character. Let's go
5:11:26ahead and execute it. And now you can
5:11:28see in the output we have two characters
5:11:30MA. Now with John we have only G because
5:11:32we have a leading space. Well, you can
5:11:35leave it like this or you can transform
5:11:36it. And then George we have G and so on.
5:11:39So with that we are getting the first
5:11:40three characters. Now in order to fix it
5:11:42for John what we're going to do we're
5:11:43going to say trim first and then apply
5:11:46the lift. So with that we are getting
5:11:48rid of all white spaces and then we
5:11:50apply the lift. So with that everything
5:11:52looks perfect. So for John we have jo.
5:11:54So this is how we can get the first two
5:11:56characters of a column. Now let's move
5:11:58to the next one. The task says retrieve
5:12:01the last two characters of each first
5:12:03name. So this time we need the last two.
5:12:05So we are coming from the right side. So
5:12:07we're going to do it like this. We're
5:12:08going to say
5:12:10write first name and then as well too.
5:12:14So last two character let's go and
5:12:17execute it. And now as you can see in
5:12:18the output we have new column where we
5:12:20have the last two characters from the
5:12:21first name. So we have here I a er and
5:12:25for John as well working and that's
5:12:26because we don't have any trailing
5:12:28spaces but if you have any trailing
5:12:30spaces then go and use that trim
5:12:32function. All right so that's all for
5:12:33the left and right and now we're going
5:12:35to go to the last function. we have the
5:12:40substring. So the substring going to go
5:12:42and extract a part of a string at a
5:12:45specified position. So this time we
5:12:47don't want something from the beginning
5:12:48or the end. We want something like in
5:12:51the middle. So we want to specify the
5:12:53starting position and we want to extract
5:12:55few characters from there. So let's have
5:12:57very simple example to understand it.
5:12:58Now in order to use the substring you
5:13:00need three things. The first one is the
5:13:02value itself where you want to extract a
5:13:05specific part from it and then you have
5:13:07to specify the starting position where
5:13:09SQL going to start extracting the
5:13:11characters that you want and as well SQL
5:13:13needs the links how many characters we
5:13:15have to extract. So now let's say that
5:13:17we have the following task after the
5:13:19second character extract two characters.
5:13:22So from reading this you can see we
5:13:24specified the starting position this is
5:13:26the second character and the length
5:13:28going to be the two characters. So let's
5:13:30have this example. Well, if you have
5:13:31Maria, so now we have to specify the
5:13:33starting position. Now we are saying
5:13:35after the second character. So the first
5:13:37character m is one. Then a is two. After
5:13:40two, we got the position number three,
5:13:42right? So starting from R. So that means
5:13:44we have to specify for SQL three because
5:13:47the starting position going to be number
5:13:49three. This is after the two. Now we
5:13:51want only two characters. So we want the
5:13:53R and the I. If you give this to SQL
5:13:55Maria starting position three and the
5:13:58length two, SQL can go and extract the
5:14:00two characters the R I. And this is
5:14:03exactly what we want. We want two
5:14:04characters after the second position,
5:14:06the second character. So with that, we
5:14:08didn't extract something from the left
5:14:10or from the right. We extracted at
5:14:12specific position. And this is exactly
5:14:14why we need the substring. Now let's
5:14:16make it a little bit more difficult
5:14:17where we're going to say after the
5:14:19second character extract everything all
5:14:22the characters. So not only RA I I would
5:14:24like RA I A. So now nothing's changed
5:14:26about the starting position. It's going
5:14:27to stay at three. But now if you are
5:14:29looking to this value and you want to
5:14:31extract everything starting from R. That
5:14:33means you have to specify the length of
5:14:35three. But this is not really good
5:14:37because let's have another value in the
5:14:39same column. So we have Martin. So the
5:14:41starting position going to be as well R.
5:14:43And now the lengths going to be
5:14:45different. So we have here four
5:14:47characters. So now the length is not
5:14:48anymore three. It is four. But you have
5:14:50to specify something at the end for SQL.
5:14:52You can go for four. That's fine for
5:14:54Maria as well. But if you have a lot of
5:14:56values, it's going to be really hard to
5:14:58specify exactly the correct length.
5:15:00That's why instead of specifying a
5:15:02static number like three or four, we can
5:15:05use another function. So now my friends,
5:15:07if you use the length function, you will
5:15:09get the total number of characters,
5:15:11right? So for Maria, you will get five.
5:15:13For Martin, you will get six. And those
5:15:15numbers are okay to use in the length
5:15:17because they are more than what we need.
5:15:19And that's totally fine. So if you are
5:15:21saying okay for Maria start from the
5:15:23third position and cut for me five
5:15:25characters SQL going to find only three
5:15:27but you will not get an error. So you
5:15:29are extracting more than you need and
5:15:31you will always get all the characters
5:15:32after the starting position. So this is
5:15:34a little trick that we use in order to
5:15:37make the links dynamic where we cannot
5:15:39find one value that we can use in all
5:15:41scenarios. And now let's go back to SQL
5:15:43in order to practice the substring.
5:15:46Okay. So now we have the following task
5:15:47and it says retrieve a list of customers
5:15:50first names after removing the first
5:15:52character. So now don't ask me why but
5:15:54for some reason we don't want to see the
5:15:56first character of the first names. We
5:15:58want to remove it. So how we can do
5:16:00that? We cannot use the left or the
5:16:01right. We have to go with the substring
5:16:04because it is little bit more
5:16:05complicated. So substring and let's go
5:16:08and get and the first argument going to
5:16:10be the value. So it comes from the first
5:16:12name and then the second argument is the
5:16:14starting position. So where we want to
5:16:16start since it is saying I want all the
5:16:18characters after the first character. So
5:16:21that means we will be starting from the
5:16:23position number two. So for example
5:16:25Maria here the first character M
5:16:27position number one and we want to start
5:16:29our substring from the position number
5:16:31two. So that was so that was the easy
5:16:33part. Now the next one the question is
5:16:36how much characters we want to leave. So
5:16:38do we leave here like four characters
5:16:41like in Maria we have four characters
5:16:43but in John we have only three then the
5:16:45next one is four and so on. So if you go
5:16:48for example with four and let's call it
5:16:51sub name. So we make it static. What can
5:16:54happen? It's going to work for some
5:16:56scenarios like Maria. We have here Ara
5:16:58and for better we are getting it. But
5:17:01for Martin it is not working. We are not
5:17:03getting the last N because it has like
5:17:06five characters after the first one. And
5:17:08by just looking to the result as you can
5:17:10see we have here one issue with John and
5:17:12that's because the first character is an
5:17:14empty string. So this is really
5:17:16annoying. So that's why we use the trim
5:17:18first just to get rid of all those white
5:17:21spaces. And now you can see it's working
5:17:23fine. So we are not getting the J. We
5:17:25have everything after the first
5:17:26character. So now instead of having this
5:17:28static what we're going to do we're
5:17:30going to make it variable. So we're
5:17:32going to go and use the length of the
5:17:35first name. So with that we make sure we
5:17:38have enough length to extract. And this
5:17:41can work for any value inside the first
5:17:43name even if the name is like 20
5:17:45characters. So let's go and execute. And
5:17:48now you can see for Martin it is now
5:17:50working. So we have here like five
5:17:51characters after the M. And here we have
5:17:53four characters after the M as well. And
5:17:56here we have three characters after the
5:17:57G. So it is working completely and it is
5:18:00full dynamic. So this is the trick by
5:18:03using the links together with the
5:18:04substring. And as you can see now we are
5:18:06using three functions in one go. We have
5:18:09the length, we have the trim and we have
5:18:11the substring. And this is what happens
5:18:12in scale. we use multiple functions
5:18:15together in order to solve like complex
5:18:17tasks. So this is how you can extract a
5:18:20substring from a string. All right. So
5:18:23that's all about the substring and with
5:18:24that we have covered a lot of very
5:18:26important string functions in SQL and
5:18:28now you have enough tools in order to
5:18:30manipulate the string values in your
5:18:32data. Okay my friends. So with that we
5:18:34have learned how to manipulate your
5:18:35string values inside SQL using the
5:18:37string functions. Now we will move to
5:18:39the second one. you will learn how to
5:18:41manipulate the numbers, the numeric
5:18:43values. So let's
5:18:48go. Okay. So now let's have this example
5:18:513.516. Now let's say that you want to
5:18:53apply the function round and you are
5:18:55using two decimal places. So what going
5:18:57to happen? It's going to go and keep
5:18:59only two digits after the decimal point.
5:19:01So five and one and the third digit
5:19:04after the decimal six. It will decide
5:19:06whether the number going to round up or
5:19:08stay as it is. And now since six is
5:19:10higher than five. So that means SQL
5:19:13going to go around the numbers up. So
5:19:15instead of having 51 we will get 52. And
5:19:18after that the third digit going to
5:19:20reset to zero. So in the out you will
5:19:22get
5:19:233.52. Now let's say that you have done
5:19:25round but only for one decimal place.
5:19:28Now it's still going to go and keep only
5:19:30one decimal place and that is the five.
5:19:32And the second digit this time going to
5:19:34decide whether we round up or not. And
5:19:36now since one is less than five, there
5:19:39is no need to round up and the five
5:19:40going to stay as it is. It will not turn
5:19:42to six. So there is no round up and the
5:19:45digits after the five going to reset to
5:19:47zero. So we're going to get 3.5. Now
5:19:49let's say that you say round zero. So
5:19:51that means I don't want to see any
5:19:53digits after the decimal point. So now
5:19:55SQL going to go and check the first
5:19:57digit after the decimal point, the five.
5:19:59This one going to decide whether the
5:20:01three going to turn to four or not. And
5:20:03now since we have five it is good enough
5:20:05to round the number because either five
5:20:07or above five going to round the
5:20:10numbers. So that's why it's going to be
5:20:12a round up and SQL going to return at
5:20:14the end four and all the digits after
5:20:16the decimal points going to be reset to
5:20:18zero. So this is exactly how the round
5:20:20function works in SQL. So now let's see
5:20:22how we can do that in SQL. Okay. So now
5:20:25let's go and practice about the number
5:20:26functions. So what we're going to do
5:20:27we're going to write SQL select but this
5:20:29time we will not select any data from
5:20:31the database. We going to practice using
5:20:33our static value like for example the
5:20:35value 3 dot 516. So let's go and execute
5:20:39it. So with that I have this decimal
5:20:41number. Now let's go and start
5:20:43practicing the round function. So now
5:20:45let's go and round this number
5:20:483.516 and this time we are rounding to
5:20:51decimals. So let's go and call it round
5:20:54two and let's go and execute it. So as
5:20:56you can see in the output we are
5:20:57rounding two decimal places and we have
5:21:00the two because as we learned the six
5:21:02going to go and round it up. Now let's
5:21:04go and do the same thing for one. So
5:21:07let's round one execute. And as you can
5:21:10see in the output we are rounding to one
5:21:12decimal. So we have the five and
5:21:14everything is zero. And we don't have
5:21:16six here because the one is lower than
5:21:18five and it will not round up the
5:21:20numbers. And let's and round by the
5:21:23zero. it is rounding it to an integer to
5:21:27the four and all the decimal digits are
5:21:29zero and we have four because we have
5:21:31five and five going to round up the
5:21:33number. So as you can see it is really
5:21:35nice and this is how we round numbers in
5:21:40SQL. Now there is another number
5:21:42function which is really cool called APS
5:21:45or the absolute what it going to do it's
5:21:47going to go and convert any negative
5:21:49number to a positive. So let me show you
5:21:51what I mean. Let's go and say we have
5:21:53like minus 10. So this is a negative
5:21:55number. But if I say APS, so the
5:21:57absolute of the minus 10, what I will
5:22:00get? I will get a positive number. So
5:22:02it's like giving us the absolute of any
5:22:04number or in other words, it is like
5:22:06converting the negative to a positive.
5:22:08And if the number is already positive,
5:22:11nothing going to happen. So if I say the
5:22:13absolute of the 10, I will get as well a
5:22:1510. So this is really nice and cool
5:22:17function that is really important in
5:22:19order to transform numbers in many
5:22:21scenarios like if you have mistakes on
5:22:23your database like let's say minus sales
5:22:26makes no sense to have sales that is
5:22:28minus. So in order to correct the data
5:22:29we can use the APS in order to convert
5:22:32all the negative numbers to a positive.
5:22:34So this is really nice cool and easy
5:22:36function to learn. All right my friends.
5:22:38So that's all for the numeric functions.
5:22:40We have covered two very simple
5:22:41functions and now in the next topic we
5:22:43have a lot of functions about how to
5:22:45manipulate the date and time in SQL. So
5:22:47let's
5:22:51go. So what is a date? If you take a
5:22:54look at calendar and you pick any date,
5:22:57for example, August 20th,
5:23:002025, this date could represent an event
5:23:03like a birth date. Happy birthday.
5:23:08Happy birthday.
5:23:10or a project deadline at your work and
5:23:13mainly it has three components. The
5:23:15first part is a fourdigit number
5:23:17indicating the year. Then the next
5:23:19component it is the month. So normally
5:23:22we represent the month with a number
5:23:24between 1 and 12. And the last component
5:23:26is the day. This is a number between 1
5:23:29and 31 depending on the month. Now in
5:23:31database we call this structure of those
5:23:33three components a date. So this is what
5:23:36we mean with dates in SQL. All right.
5:23:38All right. So now let's move to the next
5:23:39one. What is time? Time refers to a
5:23:42specific point within a day. Like for
5:23:44example, we have 18:00, 55 minutes, and
5:23:4745 seconds. So this structure has as
5:23:50well three components. The first one we
5:23:52call it the hours. It is as well a
5:23:54number between 0 and 23 indicating the
5:23:57hour of the day. Then the next one, it
5:23:59is the minutes. This is a number between
5:24:010 and 59. Moving on to the last
5:24:03component, we have the second. This is
5:24:05again the same thing a number between 0
5:24:08and 59. So now this structure with those
5:24:11three components we call it in databases
5:24:13and SQL a time. So this is what we mean
5:24:15with the time. Now to the last type if
5:24:18you go and combine both the date
5:24:20together with the time and you put them
5:24:22side by side you will get a new
5:24:25structure and a new name in the
5:24:26databases and we call it usually time
5:24:29stamp. This name is used in many
5:24:31databases like Oracle, Postgress and
5:24:33MySQL. But in the SQL server, we have
5:24:35another name for that. We call it date
5:24:37time. So again, it's very simple. The
5:24:39date time or time stamp has the date
5:24:41information together with the time
5:24:43information. So here in this example, we
5:24:45have six components from left to right
5:24:47and here we have like a hierarchy in
5:24:49this structure. So we start with the
5:24:50highest which is the year. Then we have
5:24:53the month, the day and then we continue
5:24:55to the hour, minutes and seconds. So
5:24:58those are the three different types
5:25:00about date and time informations in SQL.
5:25:02We have the date alone or the time alone
5:25:05or together in the date time. All right,
5:25:08let's explore now the data that we have
5:25:10inside our database searching for date
5:25:12and time informations. Now let's go to
5:25:14the table orders and if you go and
5:25:16expand it, you will find here two
5:25:18columns having the data type dates. So
5:25:20we have the order dates with the date
5:25:22and as well the shipping date with the
5:25:24data type dates. And if you check the
5:25:26last column, the creation date, this one
5:25:28is date time 2. So now let's go and
5:25:30query those informations in order to
5:25:32understand the structure. I'm just going
5:25:34to select the order ID, the order date,
5:25:38and the ship
5:25:39date and the creation
5:25:43time from sales orders and from is big.
5:25:47So let's go and execute it. Now if you
5:25:49go and check both order date and ship
5:25:51date, you can find that here we have
5:25:53only the structure or the informations
5:25:55about the date and we have nothing about
5:25:57the time. So again here we have a year,
5:25:59month and day and that's why they have
5:26:01the data type date. Now let's go and
5:26:03check the creation time. Not only we
5:26:05have the date information but as well we
5:26:07have the time information. So it start
5:26:09with the date information year, month,
5:26:11day and then we have hour, minute and
5:26:14seconds and then we have fractions of
5:26:16the seconds, milliseconds and so on. So
5:26:18this is how the date time or time stamp
5:26:20looks like in databases and this is how
5:26:22the date looks
5:26:27like. All right my friends now in SQL I
5:26:30can say that we have three different
5:26:32sources in order to query the dates. The
5:26:35first one is dates that are stored
5:26:37inside our database like we saw here in
5:26:40those columns like the order date,
5:26:41shipping date, creation time. All those
5:26:44are columns that holds this informations
5:26:46and they are stored inside our database.
5:26:49So this is the first source of dates
5:26:50that we can get inside our queries. Let
5:26:52me just remove those stuff and let's
5:26:54stick with the creation time. So let's
5:26:56just execute it. So those are date and
5:26:58time informations stored inside our
5:27:00database. The second type is a
5:27:02hard-coded date string that we can use
5:27:05inside our queries. Let me show you an
5:27:07example. So now if we go to a new line,
5:27:09I can go and define a date like this. So
5:27:132025 August 20th. So that in this string
5:27:17we have hardcoded a date that is static
5:27:20for all rows. Let me just call it
5:27:23hardcoded and let's go and execute it.
5:27:26Now we can see in the output we're going
5:27:27to get a static date for all rows. So
5:27:30this going to be the same for all rows
5:27:32inside our table. So this value is not
5:27:34stored inside our database. This value I
5:27:37just added to our query and hardcoded
5:27:39it. So sometimes in queries we define
5:27:41our dates that's going to be used maybe
5:27:43later in calculations and so on. Now the
5:27:46third source of getting dates inside our
5:27:48query is using the function get date.
5:27:51Get date is the first and the most
5:27:53important function that we use in SQL.
5:27:55It's going to go and return the current
5:27:57date and time at the moment of executing
5:28:00the query. So let's try that out. I'm
5:28:02going to go and get a new line. So get
5:28:04dates. It's very simple. It doesn't
5:28:06accept any values inside the function.
5:28:08So it's going to be empty. So let's call
5:28:10it today. All right. Let's go and
5:28:12execute it. And of course, we're going
5:28:14to get different results because the get
5:28:16date now is the date and the time that
5:28:18I'm recording this video. So currently
5:28:21it is July 18, 2024. And I'm recording
5:28:24this around 20 p.m. So as you can see,
5:28:26this going to be as well repeated for
5:28:28each row. We're going to get always the
5:28:30same value. So again, this depend on the
5:28:32execution of that query. So during the
5:28:34tutorial, you're going to learn a lot
5:28:36about the get date and we're going to
5:28:37use it in a lot of functions. So those
5:28:39are the three different sources of
5:28:40getting date information inside your
5:28:42query either from a column inside our
5:28:44database or hardcoded using a string.
5:28:47And the third one is using the get date
5:28:49in order to get the current date and
5:28:51time informations at the moment of the
5:28:54query
5:28:58execution. Nice. Now we have a clear
5:29:00understanding what is date and time in
5:29:02SQL. The next question is how to
5:29:04manipulate those informations using SQL
5:29:07functions. Okay. Now we have our date
5:29:09August 20th, 2025. One of the things
5:29:11that we can do with the date is we can
5:29:13go and extract different parts of the
5:29:16date. For example, we are interested
5:29:17only on the year. So we can go and
5:29:19extract only the year part. Or if you
5:29:21are interested in the month, you can go
5:29:23and extract the month and you will get
5:29:24August. And of course, we can go and
5:29:26extract the day and we will get the 20.
5:29:28So this is the first thing that we can
5:29:30do. We can extract the parts of the
5:29:32dates. Now another thing that we can do
5:29:33is we can go and change the date format.
5:29:36So instead of having like a small minus
5:29:39between those date parts, we can go and
5:29:41split them using slash. We can even
5:29:43start first with the month August then
5:29:4520 the day and then the year but having
5:29:48only the short form of the year 25 or we
5:29:50can go and change the format where we
5:29:52say we don't need any special character
5:29:54we just leave it as a space. So as you
5:29:56can see we are changing and manipulating
5:29:58the format of the date. Another category
5:30:00or task we can go and do date
5:30:02calculations. So we can go and take our
5:30:04date and add to it for example 3 years
5:30:07or we can go and find the differences
5:30:09between two dates like we are doing a
5:30:11subtraction or let's say minus and we
5:30:13will get for example 30 days. So we can
5:30:16go and add stuff subtract stuff or find
5:30:18differences between two dates. It's like
5:30:20we are doing calculations on the date.
5:30:21Now to the last thing that we can do
5:30:23with this date is we can go and test
5:30:25this date or validate it whether it is a
5:30:28real date that SQL understands. So we
5:30:31can put it on the test and at the output
5:30:33we're going to get true or false or zero
5:30:35and one. So as you can see here we have
5:30:36different ways or let's say categories
5:30:39on how to manipulate our dates in SQL.
5:30:41Now we're going to go and group up the
5:30:43different date and time functions under
5:30:45four categories. The first category and
5:30:47the most important one we have the part
5:30:49extraction and here we have around seven
5:30:51different functions that we can use in
5:30:54order to do this task. Another category
5:30:56we have the format and casting. And here
5:30:58we have three different functions.
5:31:00Underneath this category we have the
5:31:01format, convert and cast. And then the
5:31:04third category we have the calculations
5:31:06of the dates. We have two functions date
5:31:08add and date diff. And the last category
5:31:10the validation. We have here only one
5:31:12function called is dates. So as you can
5:31:14see we have a lot of scale functions. We
5:31:16have 13 date and time functions that
5:31:19we're going to cover in this tutorial on
5:31:20how to manipulate the date and time
5:31:22informations in SQL. And this is how we
5:31:24can group them into four different
5:31:26categories. Let's start now with the
5:31:28biggest category. We have the part
5:31:29extraction. We're going to cover all
5:31:31those seven functions in details on how
5:31:33to extract
5:31:37parts. All right friends, now we're
5:31:39going to cover three very easy quick
5:31:41functions in SQL to extract the parts of
5:31:43the dates. So they are very simple. The
5:31:45day function going to return a day from
5:31:48a date and in the same way the month
5:31:50going to return the month from a date
5:31:53and guess what the year going to return
5:31:56a year from a date. Okay. So now in
5:31:58order to understand how they work we
5:32:00have a date like this one 2025 August
5:32:0320th. Sometimes you are not interested
5:32:05in the whole date. You would like to get
5:32:07only a part from this date. So you go
5:32:09and use the function day in order to
5:32:12extract the two digit 20. Now in other
5:32:15scenario you might be interested in the
5:32:17month information. So you would like to
5:32:19get those two digits 08. So we can use
5:32:22the function month in order to extract
5:32:25the month information in order to get
5:32:27the August. So 08 and one more situation
5:32:30where you want to have only the year
5:32:32information. So you are interested in
5:32:34the four digits 2025. So you can go and
5:32:37use the function year in order to
5:32:39extract it. So in the output if you
5:32:41apply it you will get 2025. So it's very
5:32:44simple. This is how those three
5:32:45functions work. All right. Now let's
5:32:47check the syntax of those three
5:32:48functions. It's pretty easy. So we have
5:32:50it always like this. A keyword called
5:32:52day. This is the function name. And then
5:32:54it accept only one parameter. It is the
5:32:57date. The same things for the others. We
5:32:59have a function called month and it
5:33:01accept as well only one parameter the
5:33:03date and as well for the year the same
5:33:04thing. So the syntax is very
5:33:06straightforward. It accept only one
5:33:07value the date and we have the function
5:33:10name like the name of the part that we
5:33:12want to extract. All right. So now let's
5:33:14try out those functions. I will be
5:33:16working with the column creation time.
5:33:18So let's try for example extracting the
5:33:20year from the creation time using the
5:33:22year function. So it's going to be very
5:33:24simple. It's going to be year and then
5:33:27creation
5:33:28time like this. And let's call it year.
5:33:32That's it. Let's go and execute it. Now
5:33:34as you can see it's very simple. We have
5:33:35only one year 2025 from the creation
5:33:38time. So with that as you can see we got
5:33:40a new column where we have only the year
5:33:42informations inside it. And this
5:33:44information come from the creation date.
5:33:45So we have only 2025. Now let's go and
5:33:48do the same for the month. So we're
5:33:50going to have the same thing month
5:33:53creation time and let's call it month.
5:33:56So let's execute it. Now as you can see
5:33:58in the output we got as well the number
5:34:00of the month. So we have here January,
5:34:02February and March and those information
5:34:04as well are extracted from the creation
5:34:07time and the same thing using the day
5:34:09function. So let's go and use that. So
5:34:13creation time and we call it day. So now
5:34:17as you can see in the output we have the
5:34:19day part from the creation time. So here
5:34:22we have 1, 5, 10 and so on and all those
5:34:25informations come from the creation
5:34:26time. So as you can see those three
5:34:28functions are very simple and quick in
5:34:30order to extract parts from a date or
5:34:33date
5:34:33[Music]
5:34:37time. All right. So what is date part?
5:34:39Date part going to go and return
5:34:41specific part of the date as a number.
5:34:44All right. So now back to our example.
5:34:45We have learned how to extract the day,
5:34:47month and year. But of course now in a
5:34:50day we have more informations that we
5:34:52could extract. Not only those three we
5:34:54could extract for example the week right
5:34:57the quarter so all those informations
5:34:59are as well stored in this dates we
5:35:02cannot see it like as a value but inside
5:35:05the SQL you can extract the week and
5:35:07quarter but we don't have a function
5:35:09dedicated for those stuff because they
5:35:12are not commonly used like the year and
5:35:14month and day but still we can extract
5:35:16those information using the date parts
5:35:19for example we can say date part and we
5:35:22can specify the part as a week and with
5:35:24that SQL going to return for this
5:35:26example 34 and maybe in other situation
5:35:28you are interested in the quarter right
5:35:30so you can specify it like this date
5:35:33part quarter so we are interested in the
5:35:35part of quarter and in the output you
5:35:37will get three so this is exactly the
5:35:39power of the date part you can go and
5:35:42extract way more parts that is available
5:35:45in these dates and one more thing to
5:35:47notice about the date part year and day
5:35:50all of them are always generating the
5:35:52output an integer a number. So we have
5:35:55the for the quarter 3 for the week 34
5:35:58the day 20 2025 and so on. So all of
5:36:01those informations are integer. So
5:36:03integer is the data type of the output
5:36:06of these functions. Okay. So let's have
5:36:08a look to the syntax of the data part.
5:36:09It start with the function name date
5:36:12parts and it accept two parameters. The
5:36:14first one is the part that we want to
5:36:16extract. So we want to define what do we
5:36:18want. We want the month, the day, the
5:36:20year and so on. And the second parameter
5:36:23is the date itself. So let's have an
5:36:25example. We can say date part and we
5:36:27would like to extract the month from the
5:36:29order dates. So the part is the month
5:36:31and the order date is the date that we
5:36:33want to extract from. So with that we
5:36:35are specifying the part as a month. Now
5:36:37in SQL there is another way on how to
5:36:40specify the parts. We can go and use
5:36:42like an abbreviation of the month. So if
5:36:45you specify instead of month instead of
5:36:47writing the whole thing you write mm you
5:36:49will get the same results. So it's like
5:36:51abbreviation and shortcut in order to
5:36:53write scripts. But I rarely see that in
5:36:55the implementations. I always tend to
5:36:56write it completely like this month
5:36:58because it's more like standards if you
5:37:00are switching between different
5:37:01databases. So as you can see it's very
5:37:03simple. You have to give SQL two things
5:37:05which part you want to extract and the
5:37:07date that you want to extract from.
5:37:09Okay. So now we're going to go and
5:37:10extract different parts from the
5:37:11creation time using the date part. Let's
5:37:14start for example by extracting the year
5:37:16again. So let's go and do that. date
5:37:20parts and then we have to specify which
5:37:23part we need. So we're going to write
5:37:24year like this and then the next one
5:37:27going to be the value. So it's going to
5:37:29be the creation time. So let's call it
5:37:33year and let's say date parts. Let's go
5:37:36and execute it. So now at the output you
5:37:38can see we got as well again the years
5:37:41that is extracted from the creation
5:37:42time. So it's going to be identical to
5:37:44the year function. So there is no
5:37:46differences between them. Both of them
5:37:48are integer and it holds the year
5:37:50informations. Now we can go and try
5:37:52different parts. For example, let's copy
5:37:54the whole thing and let's extract for
5:37:57example the month. So you can go over
5:37:59here and change it to month and let's
5:38:02rename it
5:38:03execute. So at the output you see we got
5:38:06as well the months is identical as well
5:38:08to the function month. And the same
5:38:10thing for the day. So we are just
5:38:13changing the
5:38:14parts and in the output we are getting
5:38:17the parts. So here we have as well the
5:38:19days it is identical to the day
5:38:21function. So so far we don't have
5:38:22something new from the date part because
5:38:24we have it already from the other
5:38:26functions. But now we're going to go and
5:38:28extract other parts that are not year
5:38:30month and day. So for example let's go
5:38:33and get the hours. So we have the date
5:38:36part and here as a part you say hour and
5:38:40let's call it here as well hour. Let's
5:38:43go and execute it. Now you can see in
5:38:45the output we have a new dedicated
5:38:47column that shows only the information
5:38:49from the hour. So we have here 12 23 and
5:38:52so on. And those informations comes from
5:38:54the time and the same thing you can
5:38:56define minutes and so on. But now let's
5:38:59go and get something interesting like
5:39:00the quarter. So let's go and duplicate
5:39:05it and instead of hour let's get
5:39:07quarter. So this information it's not
5:39:09displayed in the creation time but SQL
5:39:12can go and extract it. So let's call it
5:39:15quarter and let's go and execute it. Now
5:39:18as you can see in the output we have one
5:39:20new field called quarter and inside it
5:39:22everywhere we have a one because all
5:39:24those dates are in the range of the
5:39:26quarter one. So as you can see this is
5:39:28amazing of course for reporting and
5:39:30analyzes. Let's go and have something
5:39:33else like the week day. So we are over
5:39:36here quarter and let's call it week day
5:39:40and rename as well this to week day. So
5:39:44let's go and execute it. All right. So
5:39:45now let's go and get something else like
5:39:47for example the week. So I just
5:39:50duplicated over here instead of quarter
5:39:52let's write week. So I would like to get
5:39:54the week number. So let's go and execute
5:39:58it. So now in the output as you can see
5:39:59we got a dedicated field that show us
5:40:01the week number from the creation time.
5:40:04So we can see this dates come from the
5:40:06week number one. Those two come from
5:40:08week number two and so on. So that's it.
5:40:11As you can see guys all those
5:40:12informations that you are getting from
5:40:14the date part are numbers. And now we
5:40:16can extract way more informations than
5:40:19only the year, month and day. And even
5:40:21if those informations are not displayed
5:40:23directly in the field itself like the
5:40:26quarter, weeks and so
5:40:27[Music]
5:40:30on. All right. So now we have very
5:40:32similar function to the date part. We
5:40:34have the date name. So the only
5:40:36difference here is that it returns the
5:40:38name of the date parts. All right. So
5:40:40now back to our example. We have learned
5:40:42we can extract different types of parts
5:40:44from one date. But we learned as well
5:40:47that all of them are numbers. How about
5:40:49we would like to extract the name of the
5:40:51month. So instead of eight, I would like
5:40:53to get the name of the month like
5:40:55August. Or instead of the 20, I would
5:40:57like to get the day name like here in
5:41:00this example, it going to be Wednesday.
5:41:02So in order to get the name of the
5:41:04parts, we have to use the function date
5:41:06name. So for example, if you use the
5:41:08function date name using the part month,
5:41:11you will not get eight in the output.
5:41:13You will get the full name of the month
5:41:15August. So as you can see we are getting
5:41:17a string a full name and as well the
5:41:19same thing if you use date name for the
5:41:21week day you will not get 20 like the
5:41:24day function you will get the name of
5:41:26the day Wednesday and as well here the
5:41:28output is string so as you can see it's
5:41:30very simple we are using the date name
5:41:32in order to get the name of the parts
5:41:35and the data type of the output here is
5:41:37a string it is not an integer so as you
5:41:39can see here we have different types of
5:41:41functions that all of them are doing the
5:41:43same job we are extracting ing parts
5:41:46from one date. Okay. So now by checking
5:41:48the data name syntax, it's going to be
5:41:51identical to the date part. So we are
5:41:53just switching the function name. It
5:41:56needs from you to define the part and as
5:41:58well the dates. The only difference here
5:42:00is that we are getting different data
5:42:01type at the output. So here we are
5:42:03getting a string instead of integer. All
5:42:05right. So now let's check the date name.
5:42:07It is very similar to the date part. So
5:42:09we're going to have it like this. We're
5:42:10going to work as well with the creation
5:42:12time. So we're going to say date name
5:42:14and then after that we have to define
5:42:16the parts. So let's go for example with
5:42:18the month and our field is as usual the
5:42:21creation time and let's call it month
5:42:25date
5:42:26name like this. So that's it. Let's go
5:42:29and execute it. Now if you go to the
5:42:31output over here you can see we have the
5:42:33month but this time we don't have
5:42:35numbers. We have the full name of the
5:42:37month. So we have January, February,
5:42:39March instead of having 1 2 3. So this
5:42:42is the big difference between the date
5:42:44name and date part. Date part you get
5:42:45numbers. Date name you get the name of
5:42:48the part. So let's do the same thing for
5:42:50the day. We would like to get the name
5:42:53of the day. So I'm just duplicating it.
5:42:55But now in order to get the full name of
5:42:57the day, we cannot go with the day.
5:42:59We're going to go with the week day as a
5:43:02part. So that's it. I will call it week
5:43:05day. So let's execute it. Now as you can
5:43:07see in the output, we have here a new
5:43:10column called week day. And inside it we
5:43:12have the name of the day instead of a
5:43:14number. So here we have Wednesday,
5:43:16Sunday, Friday and so on. So the full
5:43:18name of the day go of course with the
5:43:20day. Let's go and try that
5:43:23out. So this is the day of the month and
5:43:27of course the day of the month has no
5:43:29name and SQL of course going to return
5:43:31the numbers again. So you can see 1 5 10
5:43:3320 and so on. But still there is a
5:43:35difference between the day from the day
5:43:37name and the day from the date parts. In
5:43:40the date parts we are getting integers.
5:43:43So if you store this information in a
5:43:44new table it's going to be stored as an
5:43:46integer. But in the date that you are
5:43:49getting from the date name it is a
5:43:51number but still it can be stored as a
5:43:53string value. So the data type of those
5:43:56numbers is a string and the data types
5:43:58of the day from the date part is an
5:44:00integer. And the same thing can happen
5:44:02if you extract for example a year. So
5:44:04you don't have like a full text of the
5:44:06year. So let me just do it like this. So
5:44:10if we say a year, you will not get the
5:44:13name of the year. You're still getting
5:44:15the numbers, the digits, but the data
5:44:17type here is a string. So that's it.
5:44:19This is the difference between the date
5:44:21name and the date parts. For the month
5:44:23and weekday, you will get the full name.
5:44:25For the other stuff, you will get
5:44:26numbers but with the string data type.
5:44:29So the most important thing about the
5:44:30date name is to present easy to read and
5:44:34human readable informations to the
5:44:35users. So imagine you are building a
5:44:37report called sales by month and then
5:44:39you show to the user the muscles as
5:44:41numbers 1 2 3 until 12. This is of
5:44:43course okay but it is way more nicer if
5:44:46you present those informations as a full
5:44:48text. So you go with the date name in
5:44:50order to show instead of one you show
5:44:52January, February, March and the full
5:44:54name of the month. And this going to
5:44:56look way nicer in reporting for the
5:44:58users. So this is the core use case of
5:45:00the date
5:45:04name. So what is date trunk? Date trunk
5:45:07going to go and truncate the date to a
5:45:10specific part. So let's understand what
5:45:13this means. Okay. Now let's check the
5:45:15syntax of the date trunk. It's going to
5:45:16be exactly the same like date part and
5:45:18date name. So you have to define the
5:45:20part and the date that you want to
5:45:22extract apart from it. So the only thing
5:45:25that is different here we are giving
5:45:27different function name. So as you can
5:45:29see all those three functions like
5:45:31having the same structure you have to
5:45:33provide which part you want to extract
5:45:35like a month, day, week, hour, minutes
5:45:38and so on and the date or date and time
5:45:41that you want to extract a part from it
5:45:43and of course with the date trunk we are
5:45:44getting at the output date or date time.
5:45:47Okay. So now let's understand exactly
5:45:49how the date trunk works. We have the
5:45:51following date time and as we learned we
5:45:53have like a hierarchy where we start
5:45:55with the highest from the year then we
5:45:57move to the month, day, hours, minutes
5:46:00and seconds and by looking to this
5:46:02information it is very precise. We know
5:46:04exact second for this information right?
5:46:07So the level of details here is very
5:46:09high. We know the seconds of this event.
5:46:12So now the date going to allow us to
5:46:14change this level of details of this
5:46:16information by specifying the level of
5:46:18details. Let's take for example if we
5:46:20say the date trunk minutes. So we are
5:46:23saying we are interested only at the
5:46:25minutes level. We are not interesting
5:46:26with the seconds. So what can happen?
5:46:29Everything between the year and the
5:46:31minutes going to be kept. That means all
5:46:33those information will not be changed
5:46:35but only the seconds going to be
5:46:38reseted. We are not interested anymore
5:46:40with the seconds. This is very detailed
5:46:42for us. So it's going to go and reset
5:46:44the seconds to 0 0. So we are saying the
5:46:47minimum level is the minutes and we are
5:46:50not interested anything like before it
5:46:52the seconds let's say now we say you
5:46:54know what the minutes is very detailed I
5:46:56would like to be at the hours level so
5:46:58we specify for the date rank hour so
5:47:01here things changed we're going to keep
5:47:03the informations now between the year
5:47:05and the hours and anything after that
5:47:07going to be reseted so now minutes and
5:47:10seconds going to be in the range of the
5:47:12resets and SQL going to go and reset the
5:47:1555 to 0 0 so now the level of details is
5:47:19little bit lower now we know only the
5:47:21informations until the hours and we are
5:47:24not interested about the minutes and the
5:47:25seconds and I think you already get it
5:47:27if you say date trunk day what's going
5:47:30to happen it's going to keep everything
5:47:32between year and day and the whole time
5:47:35going to be resets so the hours and
5:47:38seconds all those information is going
5:47:40to reset to 0 0 so now by looking to
5:47:43this we don't know anything about the
5:47:45time we know only informations about the
5:47:47dates and now we can go one more step
5:47:49and we say you know what I'm not
5:47:50interested about the days I'm doing
5:47:52analyszis on the month level so what is
5:47:54here kept is only two informations year
5:47:57and month and everything below that the
5:48:00day and the time going to be reseted but
5:48:03this time SQL will not reset the date to
5:48:050 0 because there is no date called 0 0
5:48:08it start always with the first date so
5:48:10it's going to reset to 01 so the dates
5:48:13parts and the dates going to reset to 01
5:48:15one and the dates parts in the time
5:48:17going to reset to 0 0. So now we are at
5:48:20the level of the month. Now you can go
5:48:22to the last step and you say you know
5:48:24what I'm interested only on the years
5:48:26and I'm doing only analyzes at this
5:48:28level at the highest level. So you can
5:48:30go and say date trunk year and now
5:48:32what's going to happen going to keep
5:48:34only the year and everything below that
5:48:37going to be reseted. So between month
5:48:39and the seconds everything going to
5:48:41resets. So here is scale going to reset
5:48:42as well the August 2011. So the only
5:48:46value that is kept is the year and
5:48:48everything else is reseted. So this is
5:48:50the 1st of January and the time is
5:48:52completely reseted. So now we are at the
5:48:54lowest level of details. We know only
5:48:56information about the year and we don't
5:48:58care about any other parts. So as you
5:49:00can see the date trunk here is not
5:49:01really extracting a part here. Date
5:49:04trunk is like resetting stuff. So we are
5:49:07navigating through the hierarchy of the
5:49:09date and time and we are controlling at
5:49:11which level we are doing the analyszis.
5:49:13So as you can see at the end it's not
5:49:15very complicated once you understand how
5:49:16it works and it is very useful in
5:49:18analyzis. So this is how the date trunk
5:49:21works in SQL. Okay, let's have a few
5:49:23examples about the date rank together
5:49:24with the creation time. So as you can
5:49:26see the creation time the level of it is
5:49:28the seconds. So we have seconds
5:49:30information with the creation time. Now
5:49:32I would like to move it to the minutes.
5:49:35So let's go and do this date trunk and
5:49:38we're going to say let's tr it at the
5:49:41minutes level for the creation time. So
5:49:45let's call it minute date trunk. So
5:49:50let's go and execute it. Now if you go
5:49:52and check the output over here and
5:49:53compare it to the creation time, you can
5:49:55see here we have zeros at the seconds.
5:49:58So as you can see we have the seconds
5:50:00completely resetted compared to the
5:50:02creation time. Now let's say that I'm
5:50:04not interested in the time information
5:50:06inside the creation time. I would like
5:50:08only to get the date. So in order to do
5:50:10that, we can use the date trunk where we
5:50:12reset to the level of the day. So let's
5:50:15go and duplicate it. I'm going to put it
5:50:18over here and instead of minutes, let's
5:50:20say we have a day and let's go and check
5:50:23the output. Now if you go and check the
5:50:25result over here you can see all the
5:50:27time informations are reseted to zeros
5:50:30and we have here only information about
5:50:32the date. So we have year month and day
5:50:35and everything else is reset it to zero.
5:50:37Now of course we can go to the maximum
5:50:39where we say I just need the year. So I
5:50:42don't need anything else. So let's try
5:50:44that out. We're going to take date trunk
5:50:46and say year and let's call it year. So
5:50:51let's go and execute it. Now if you
5:50:53check the output over here you can see
5:50:54that everything is reseted beside the
5:50:57year. So we have only the year
5:50:58information but everything else is
5:51:00reseted to the first of January and the
5:51:03time is as well is reseted. So as you
5:51:05can see the output of the date trunk is
5:51:08always as a date time and it help us as
5:51:11well to navigate through the hierarchy
5:51:13of the day time and we can truncate at
5:51:15the level that we want. All right. So
5:51:17now we're going to check why data trunk
5:51:18is amazing function for data analyszis.
5:51:21So let's have this example. We are
5:51:23saying select
5:51:25creation time and we want to count the
5:51:29number of orders based on the creation
5:51:31time from our table sales orders and
5:51:34we're going to use the group by in order
5:51:37to group the data by the creation time.
5:51:39So let's go and execute it. Now as you
5:51:41can see we're going to get one
5:51:42everywhere because the level of details
5:51:45the granularity or the creation time is
5:51:47very high and that's because here we
5:51:49have the seconds and since our data is
5:51:51small we will not get like two orders at
5:51:53the same seconds. Now in data analytics
5:51:55you would like quickly to aggregate the
5:51:57data at different granularity like for
5:51:59example at the month level. So you can
5:52:01do that very quickly using the date
5:52:04trunk and you say you know what let's
5:52:06say at the month and let's call it
5:52:10creation and we're going to have the
5:52:12same thing for the group pie. So let's
5:52:14go and execute it. So now as you can see
5:52:16at the output we have only three rows we
5:52:18don't have like 10 rows and that's
5:52:20because we have three months. So that
5:52:22means we just rolled up to the month
5:52:24level instead of the seconds. And we can
5:52:26see now in the month of January we have
5:52:27four orders, February as well four and
5:52:30March we have only two. So now we are
5:52:32talking about different level of details
5:52:33in the output and granularity. And now
5:52:36you might say let's go and aggregate the
5:52:37data at different level at the year
5:52:39level. So you can just change over here
5:52:41the year and execute it. And with that
5:52:44now we are at the highest level of
5:52:46aggregations. We are at the year level
5:52:48and since in our data we have only 2025.
5:52:51So we will get the total number of
5:52:52orders inside the table and that is 10.
5:52:55And this is really amazing in data
5:52:56analytics. You can go and quickly change
5:52:58the granularity and the level of
5:53:00aggregation or details by simply
5:53:02defining the level inside the dates. So
5:53:05this is why the date rank is amazing. It
5:53:07allow us to do analyszis and
5:53:09aggregations by zooming in and zooming
5:53:15out. Okay. So now we're going to talk
5:53:17about the last function in the part
5:53:19extraction category. We have the end of
5:53:21the month. As the name says, it's going
5:53:23to go and return the last day of a
5:53:26month. So let's see how end of month
5:53:28works. This is very simple. So let's
5:53:29take our date 20th August 2025. If you
5:53:32go now and apply this function to it,
5:53:35what's going to happen? It's going to go
5:53:36and change only the day information. So
5:53:39instead of 20, it's going to go to the
5:53:41last day of the month. So it's going to
5:53:43go and change the 20 to 31. The last day
5:53:47of the month, August in 2025. Let's take
5:53:50another example is the 1st of February
5:53:522025. If you apply the end of the month,
5:53:55it's going to go and change the day from
5:53:58the 1st to 28. The last day of month
5:54:01February. So as you can see, it's very
5:54:03simple. Let's take another example where
5:54:04it is already the last day of the month.
5:54:08So we have 31 of March. If you apply the
5:54:11end of the month here, what can happen?
5:54:13Nothing going to happen. You're going to
5:54:14get in return the same value. So this is
5:54:17how it works. And as you can see always
5:54:18the output of the end of the month going
5:54:20to be as well a date. So this is how end
5:54:23of month work. It is very simple. All
5:54:25right. Now quickly about the syntax of
5:54:27the end of the month. It's going to have
5:54:28the exact same syntax like the day,
5:54:31month, year. It accepts only one
5:54:33parameter. It is the date. So we have to
5:54:35pass here a date in order to find out
5:54:38the end of the month. So let's go and
5:54:40find the end of the month of our
5:54:41creation time. So end of the month like
5:54:45this. And let's have our creation time.
5:54:49So let's see the end of month. Let's go
5:54:53and execute it. And now in the output
5:54:54you can see we have a new column a date
5:54:56column. And inside it we have values
5:54:58about the end of the month. So for
5:55:00example here we have January, January,
5:55:01January and so on. So you will see
5:55:03always here the end of January and the
5:55:05same thing for February and March. So
5:55:07that's it. This is really nice function
5:55:09in case you need the end of the month of
5:55:11each date. Maybe you're creating a
5:55:12report or analyzes where you need this
5:55:14information. And now you might ask me
5:55:16how about to get the first day of the
5:55:18month. Is there like any function for
5:55:19it? Well, no. But there is a trick in
5:55:22order to get the first day of the month
5:55:24using another function that we just
5:55:26learned. Think about it. How to get the
5:55:28days as one everywhere. So we have to
5:55:31get here the 1st of January, the 1st of
5:55:33February, and the 1st of March. So how
5:55:35we can do that? Well, using the date
5:55:38trunk. So let me show you how we're
5:55:39going to do this. So date
5:55:42trunk and we're going to reset at the
5:55:44level of month. So we don't need the
5:55:47days it going to reset to the first. So
5:55:50our field is creation time and this
5:55:53going to be the start of month. So let's
5:55:57go and execute it. So now as you can see
5:55:59in the output we have the start of month
5:56:01and you can see we have everywhere here
5:56:02a one since we reset it at the level of
5:56:05month and this going to give us the
5:56:06first day of the month. And now you
5:56:08might say you know what here we have a
5:56:10lot of zeros how to get it exactly like
5:56:11the end of the month and that's because
5:56:13the date rank give us date and time
5:56:15always. So that means we have to change
5:56:17the data type and that we're going to
5:56:18learn later using the cast function but
5:56:21we can go and do it right now. So we can
5:56:23say cast and we want to change the whole
5:56:26thing to date. And now that we change
5:56:28the data type from date time to date and
5:56:31in the output as you can see we have
5:56:32only the date information. So now it's
5:56:34really amazing that you got two dates.
5:56:35The first one is the start of the month
5:56:37and the second is the end of the month.
5:56:38And those information might be helpful
5:56:40if you are generating reporting and you
5:56:42need the start and the end of the
5:56:43[Music]
5:56:46month. So now we come to the part where
5:56:49we ask the question why do we need those
5:56:51parts? Why do we need to extract the
5:56:53date parts from a date? So let's have
5:56:56the following use cases. The first use
5:56:58case of extracting the part is doing
5:57:00data aggregations and reporting.
5:57:03Sometimes we are building like reports
5:57:04based on our data and sometimes we have
5:57:06to aggregate our data by a specific time
5:57:09unit like for example we are building a
5:57:11reports in order to show the sales by
5:57:13year. So we have different years and we
5:57:15are aggregating the data based on the
5:57:18year or you want to drill down to more
5:57:20details where you want to aggregate the
5:57:21data by the quarter. So in this report
5:57:24we are showing the sales by quarter Q1 2
5:57:273 4 or you decide to go in more details
5:57:30where you show a report says sales by
5:57:32month and then you start aggregating
5:57:34your data by the month. So you have
5:57:36January, February, March and so on. So
5:57:38as you can see we can use those
5:57:39different parts in order to aggregate
5:57:42the data based on it and these different
5:57:44parts can offer us different analyzes
5:57:46with different details. So now we have
5:57:48the following task and it says how many
5:57:50orders were placed each year. So that
5:57:53means we have to group up our data by
5:57:55the year and we have to count the number
5:57:56of orders. Let's go and solve it. So
5:57:59let's go with the select. And now what
5:58:00do we need? We need the order date. This
5:58:04going to indicate when the order is
5:58:05placed. So and we have to go and count
5:58:09the star. So this going to be number of
5:58:11orders. and from our table sales orders
5:58:16and we have to group up by the order
5:58:19dates. So that's it. Let's go a and
5:58:21execute it. So now in the output we are
5:58:23getting the number of orders but by the
5:58:26order date. So we are still not there.
5:58:28We have to have it as a year. So we
5:58:31don't need the whole date information.
5:58:33We need only the year information. So
5:58:35that means we have to go and extract the
5:58:37part year. In order to do that we can do
5:58:40it like this. So we can go with the year
5:58:43and we have it as well in the group I.
5:58:46So that's it. Let's go and execute it.
5:58:48And with that as you can see we got the
5:58:49number of orders for each year. And
5:58:51since in our data we have only 2025 we
5:58:53will get only one row. So with that the
5:58:55task is solved. We are now aggregating
5:58:57the data on the level of the year. Now
5:58:59let's have another task which is the
5:59:01same but only different parts. How many
5:59:03orders were placed each month. So we
5:59:06have to go and change it to a month.
5:59:08It's very simple. We're going to use the
5:59:10function month and as well in the group
5:59:13by. So let's go and execute it. And now
5:59:15as you can see in the output we don't
5:59:17have one row. Now we have three rows.
5:59:18And that's because we have three months
5:59:20inside our data. And for each month we
5:59:22will get the total number of orders. So
5:59:25for the January we have four, February
5:59:27we have four and March we have two
5:59:29orders. Now you might say you know what
5:59:31I don't want the months as a numbers. I
5:59:33would like to have the full name of the
5:59:34month. So in order to do that we're
5:59:36going to go and use the function date
5:59:38name. So let's go and use date name and
5:59:41then we have to specify the date part.
5:59:43It's going to be the month and the value
5:59:45going to be the order date and we have
5:59:47to have the same thing as well in the
5:59:49group I. So let's go and execute it. Now
5:59:52you can see in the output we are getting
5:59:53the full name of the month which is
5:59:55easier to read. So this is one of the
5:59:57use cases why we need to extract parts
5:59:59from a date in order to aggregate the
6:00:02data on a specific level.
6:00:07So now let's have the following task and
6:00:09it says show all orders that were placed
6:00:12during the month of February. So that
6:00:14means we don't need all the orders. We
6:00:16need only a subset of the orders based
6:00:18on the order dates. Now let's go and
6:00:20check the data. So select star first
6:00:23from sales orders and let's go and
6:00:27execute it. So now with that we have our
6:00:2810 orders. Now if you check the order
6:00:30date over here you can see that we have
6:00:32orders in January, February and March.
6:00:35Now we are interested only on the orders
6:00:37that were placed in February. So only
6:00:39these subsets. So that means we have now
6:00:41to filter the data based on the month
6:00:43information. So what we're going to do,
6:00:45we're going to have a wear clause. And
6:00:47now we don't need the whole order date.
6:00:49We need only the part month. So we're
6:00:51going to go with the month and order
6:00:54date and this going to be equal to two.
6:00:58Since the output going to be in number.
6:01:00So let's go and execute it. Now as you
6:01:02can see SQL did filter the data and in
6:01:04the output we have only the orders were
6:01:07placed in the month of February. So this
6:01:09is as well very common use case. Why do
6:01:12we need the parts? We use it in order to
6:01:14filter the data based on specific part
6:01:17of the dates. So as you can see it's
6:01:19very quick and easy. And here my
6:01:21recommendation is that if you are
6:01:22filtering the data always use the
6:01:24numbers. So always use a date function
6:01:26that gives you a number because it's
6:01:28always faster to search for integers
6:01:31instead of searching for a character or
6:01:34for string. So don't use the date name
6:01:36function in order to search or filter
6:01:38for the data. It's better to use the
6:01:40date part or month, year and day. Since
6:01:42you can work with numbers and numbers
6:01:44are always faster to retrieve data and
6:01:46to filter your
6:01:50informations. Okay. So now we have a lot
6:01:52of functions and I would like now to do
6:01:54a quick recap about the data type of
6:01:56their results. So as we learned we have
6:01:58functions like day, month, year, date
6:02:01bar and the output of all those
6:02:04functions going to be integer. It's
6:02:06going to be a number. Now we have
6:02:07another function the date time. If you
6:02:09use it the output of this function going
6:02:11to be a string because here we are
6:02:13extracting the name of the date part.
6:02:15And if you go and use the date trunk you
6:02:18will get in the output always date time
6:02:20two. So you are getting both the date
6:02:22and time. And the last function that we
6:02:24learned end of month if you use it in
6:02:26the results you will get the data type
6:02:28date. So this is really important to
6:02:30understand the data type of the output
6:02:32so that you don't get any unexpected
6:02:34results. All right. So now you might say
6:02:36you know what those are a lot of
6:02:37functions and like I'm saying they are
6:02:39doing the same stuff. We are extracting
6:02:41the parts of the dates. So now you might
6:02:43ask me how do you decide on when to use
6:02:46which function? This is how I usually do
6:02:47it. First I ask myself which part I want
6:02:50to extract. If I want to extract a date
6:02:52or a month then I ask the question do I
6:02:55need it as an integer as a number? If
6:02:58it's yes then I go and use the day
6:03:00function or the month function because
6:03:02they are quick and I will get exactly
6:03:04what I need. But now if I need the full
6:03:06name of the month or the day then I go
6:03:09with the function date name. Now moving
6:03:11back if I'm interested on the part year.
6:03:14So here we don't have a year name or
6:03:15something. I'm going to go immediately
6:03:17with the function year. But now let's
6:03:19say that I don't need the day, month or
6:03:21year. I'm interested in other parts like
6:03:24the week, the quarter and so on. Only
6:03:27for this scenario, I go with the
6:03:28function date part. So this is my
6:03:31decision process. This is how I decide
6:03:33when to use which SQL function in order
6:03:36to extract the parts of the
6:03:41dates. All right. All right. So now I
6:03:43have prepared for you here a list of all
6:03:45parts that we can use inside those three
6:03:47functions date part date name and date
6:03:49trunk. And you can see in this table the
6:03:51different outputs using those different
6:03:53three functions. So for example if you
6:03:55go and use the month with the date part
6:03:57you will get eight but for the date name
6:03:59you will get August and for the date
6:04:01trunk you will get truncated date time
6:04:04at the level of the month where you
6:04:06reset the days and times. So this is a
6:04:08full list of all examples you can go and
6:04:10check it. And one more thing that I have
6:04:11prepared for you in order to practice
6:04:13with all those different parts. I have
6:04:15made one big query with all different
6:04:17parts. So if you go and download the
6:04:19queries of this chapter, you will find
6:04:21the following files and let's go now and
6:04:23open all date parts. So we're going to
6:04:26go inside it and here we have a long
6:04:29query. So what we're going to do, we're
6:04:30going to select everything and copy it
6:04:32and let's go back to our scale and paste
6:04:34it. So let me just zoom out and then
6:04:38let's go and execute the whole thing. So
6:04:41now in my code I have just done a union
6:04:43for each possible part. For example for
6:04:46the year we have date part date name and
6:04:48date trunk and I'm using currently the
6:04:50get date. So we are manipulating this
6:04:52one and then the output can be presented
6:04:54over here. So you can see it like this.
6:04:56So if you use the part here for the date
6:04:58name you will get 2024. The same thing
6:05:01for the date name and this is for the
6:05:02date rank. And with that you have all
6:05:05possible parts that you can use in SQL
6:05:08in one query. So with that you can learn
6:05:10what are the outputs for different
6:05:12parts. All right. So with that we have
6:05:14learned all those functions on how to
6:05:16extract the parts of dates. All right.
6:05:18Moving to the second category. We're
6:05:20going to learn how to do formatting and
6:05:22casting for the date informations in SQL
6:05:24using three functions.
6:05:30So now before we deep dive to the
6:05:31formatting and casting I would like you
6:05:33to understand what is date format. So
6:05:35back to our example we have here the
6:05:37date and time informations and we
6:05:39understood there is components year
6:05:41month day and so on. Now if you check
6:05:43the date time there is combination of
6:05:45numbers and characters. For example the
6:05:472025 is a number but between the month
6:05:50and the year there is like a minus
6:05:52between them and this is a character. So
6:05:54now this is a very specific format and
6:05:57in SQL we can have a code for this
6:06:00format. So for example let's start with
6:06:01the year we have here four digits and we
6:06:04can represent it with 4 Y. So Y Y and we
6:06:08call those characters as format
6:06:11specifiers. So this is how we represent
6:06:14the year. Then between the year and the
6:06:16month there is like this small minus and
6:06:18then the month is two digits and we're
6:06:20going to represent it with two big M. So
6:06:23m M then between the month and the day
6:06:25there is a minus. So we have as well
6:06:27minus and then the day going to
6:06:28represented with two digits d and then
6:06:31we have like a space between the date
6:06:33and time and then we start with the
6:06:35date. So it start with the hour big h
6:06:37and big h because here we have the
6:06:39system of 24 and then we have double
6:06:41points small m small m. So as you can
6:06:44see here the formats are case sensitive.
6:06:46So there is a big difference between
6:06:48small m and a big m. So a small m
6:06:50indicates for a minute and big m
6:06:53indicates for a month. So as you can see
6:06:55here the case format is case sensitive.
6:06:58So two small m means minutes but two
6:07:01capital m means month. Then double point
6:07:04and small 2s. So now the whole code is
6:07:07called the date format. So this is the
6:07:10date format representation of this
6:07:12value. Now in the world there are
6:07:14different representations on how to
6:07:16represent a date. So for example in SQL
6:07:18we have the international standard
6:07:21ISO6801 and the date format is like we
6:07:23have learned first it start with the
6:07:25year. So four digit for the years minus
6:07:28two digit for the month minus two digit
6:07:31for the day. So year month day but in
6:07:34the USA we have different standards. So
6:07:36first it start with the month. So we
6:07:38have mm and then after that it is
6:07:41followed with the day. So we have then
6:07:43the day and after that at the end we
6:07:45have the year. So this is the sentence
6:07:47format that is used in USA and in Europe
6:07:49we have different representations of the
6:07:51day. So it start first with the small.
6:07:53So it starts with the day then the month
6:07:55and then the year. So this is exactly
6:07:58the opposite of the international
6:07:59standards. So as you can see we don't
6:08:01have one standard. We have different
6:08:02ways on how we represent dates. But in
6:08:05SQL the SQL server is following the
6:08:07format of the international standards.
6:08:10So SQL server start always with the year
6:08:12then month then day. So all dates that
6:08:15are used in our SQL database can be
6:08:17following this
6:08:21format. Okay. So after we understood
6:08:24what is date format, now let's talk
6:08:25about formatting and casting. So what is
6:08:28formatting? Is changing the format of
6:08:30value from one to another. So we are
6:08:32changing how the data looks like. So for
6:08:34example, we have our date. So it's
6:08:36following the international standards
6:08:37start with year, month, then day. Now we
6:08:40can go and change the format using the
6:08:42function format where we can go and
6:08:44define a different date format like it
6:08:46start with the month and then we have
6:08:48like slash instead of minus and then the
6:08:51day/ year. So in the outer we're going
6:08:53to get it like this and even the years
6:08:55is only two digits not four. So here we
6:08:57are providing for SQL the format that we
6:08:59would like to see the data with or you
6:09:01can go with other format where you have
6:09:03three big M and then four digits for the
6:09:05year and between them is just a space.
6:09:07So in the output you will get
6:09:09abbreviation of the month name and then
6:09:11space and the year. So this is one way
6:09:13on how to format data. But in the scale
6:09:15there is another function that help us
6:09:17to format data and that is convert. So
6:09:19here we provide not the format itself we
6:09:22provide style number. So for example the
6:09:24style number six. So it can show it like
6:09:26this day space and after that we have
6:09:28the abbreviation name of the month and
6:09:31then two digits of the year. Or if you
6:09:33use another style the 112 then you will
6:09:36get the year, month, day without any
6:09:38separation between them. And of course
6:09:40not only the date and time we can style
6:09:42we can style as well numbers and here we
6:09:44can use the function format in order to
6:09:45change the format of the number. So here
6:09:47if you're using the format of numeric
6:09:49values then the values will be separated
6:09:51with comma or if you use c for the
6:09:54currency then you will get the dollar
6:09:55sign or if you go and use p then you
6:09:58will get the percentage and at the end
6:09:59you have the percentage character. So as
6:10:01you can see we can as well change the
6:10:03format of the numbers but only the
6:10:05dates. So this is what we mean by
6:10:06formatting we are just changing how the
6:10:09value looks like. Now in the other hand
6:10:11the casting the casting can go and
6:10:13change the data type from one to
6:10:16another. So for example if we have the
6:10:18value 1 2 3 as a string we can go and
6:10:21convert it from the data type string to
6:10:23an integer. So in the output we will get
6:10:25as well 1 2 3 but as a number or we can
6:10:28go and change the data type from dates
6:10:30to a string. So in the output it is not
6:10:33anymore dates it is a string value or
6:10:36the way around we can change the data
6:10:38type from a string to a date. So as you
6:10:41can see we can change the data type from
6:10:43one to another and we can use that using
6:10:45two functions. The first one is and the
6:10:48most famous one is cast function or in
6:10:50SQL server we can use as well the
6:10:52convert function in order to change the
6:10:55data type. So this is what we mean with
6:10:57casting changing the data type from one
6:10:59to
6:11:03another. All right. So let's start with
6:11:05the first function the format. So what
6:11:07is format? As the name suggest it
6:11:09formats a date or time value. So it's
6:11:12like we are changing how the date and
6:11:14time looks. Okay. So let's check the
6:11:16syntax of the format and here it accepts
6:11:19two parameters and the third one is
6:11:21optional. So the first one we have to
6:11:23provide a value. It could be a date or a
6:11:25number. And the second one we have to
6:11:27provide the format. So here we are
6:11:28specifying the new look the new format
6:11:30for this value. Now the third one it is
6:11:33optional one. It is the culture. Culture
6:11:35means show me the value whether it's
6:11:37date, time or number. Show me this value
6:11:40in the style of a specific country or
6:11:43region. So each country each region has
6:11:45different format. So here we can go and
6:11:47change it to specific region format. But
6:11:50as I said it is optional. Let's have an
6:11:52example. So here we are saying go and
6:11:54format the order dates using the
6:11:56following format. So dd day then slash
6:12:00then we have the month then slash then
6:12:02the year. So going to go and format this
6:12:04with this new format. And as you can see
6:12:06here we didn't specify any culture since
6:12:07it's optional. Let's see another option
6:12:09where we can say you know what I would
6:12:11like to have the order date formatted
6:12:13with this format but we would like to go
6:12:15and add the style of Japan. So we are
6:12:17specifying here the code or the style of
6:12:20Japan. And of course we can go and use
6:12:22the format not only for the date but as
6:12:24well for formatting the numbers. So here
6:12:26we are specifying the value. The format
6:12:29is D. And as well we have activated the
6:12:31culture option. We are using the style
6:12:33of France. So this is the syntax of the
6:12:35format. Using this option is not really
6:12:37common. So I rarely see this format or
6:12:40someone using it. So the first example
6:12:42is the most used one in the projects
6:12:45where we have the culture as default or
6:12:47we are not using the culture at all. And
6:12:49of course if you don't specify anything
6:12:51is going to go and use the default
6:12:54culture which is enus. So this is all
6:12:57about the syntax of the format. All
6:12:58right. So now let's have a few examples
6:13:00using the format. So we're going to go
6:13:02and format the creation time. So we're
6:13:04going to do it like this. Format. And
6:13:06what we are formatting? We are
6:13:07formatting the creation time and now you
6:13:10can go and define any specifier you
6:13:12want. For example, let's say DD like
6:13:15this. So let's go and check the outputs.
6:13:18So execute it. Now if you are using DD,
6:13:20you will get the day information. So we
6:13:24can see if you're using this specifier,
6:13:26we are getting two digits about the day.
6:13:28So and as well we are getting the
6:13:29leading zero. So we are getting the 01
6:13:3105 and all those informations are the
6:13:34day information. Now let's go and try
6:13:35something else. adding one more D. So
6:13:39let's have it 3D and here as well. So
6:13:42let's go execute it. So now if you check
6:13:44the output, we are getting now the name
6:13:45of the day. It is not full. So we are
6:13:48getting like a short name of the day or
6:13:50abbreviated one. So this is sometime
6:13:51nice if you are creating like a calendar
6:13:53or something. Let's go and add one more
6:13:56D. So we're going to have 4 D. And let's
6:14:00go and check the result for this one.
6:14:02Now in the output we are getting the
6:14:04full name of the day. So it's really
6:14:06nice. Now we are getting full
6:14:08flexibility on how to format our day.
6:14:10Okay. So now let's keep playing. Let's
6:14:11get something else. I'm just going to go
6:14:13and duplicate everything and I will go
6:14:16with the month now. So this is 2 M, 3 M
6:14:21and 4 M. Let me do it like this. So
6:14:26let's go and execute it. Now as you can
6:14:28see we are getting the same stuff but
6:14:30for the month. So mm we will get the two
6:14:32digits and 3m we will get the
6:14:34abbreviated name of the month and for m
6:14:37we will get the full name of the month.
6:14:39So it's like we are extracting the date
6:14:41part from the format but of course we
6:14:43don't use it like this. We will go and
6:14:45write the whole format that we need for
6:14:48a date. So for example let's go and
6:14:50change this format to the USA format. So
6:14:53in order to do it so we're going to go
6:14:55over here. So let's say format again the
6:14:58creation time. And now we're going to
6:15:00write the format of USA. So it's going
6:15:02to be mm. Then after that then after the
6:15:05month we're going to have like minus
6:15:07then day and then after that we're going
6:15:09to get the year. So for time year and
6:15:12that's it. Let's call it USA format. So
6:15:15let's go and excuse it. And now you can
6:15:18see in the outut we got a new column
6:15:19where we see now the date information
6:15:21but as a USA standards. So it start with
6:15:24the month then the day and then
6:15:26afterward we got the year. And of course
6:15:28we can do the same thing in order to
6:15:29generate the standard format of Europe.
6:15:32So what we're going to do I'll just
6:15:34duplicate it. And now the format of that
6:15:37going to start with the day then the
6:15:39month and then the year. So now if you
6:15:42check the output you can see it start
6:15:43with day minus then we have the month
6:15:46then minus the year. So as you can see
6:15:48we are changing the format of the date
6:15:50from creation time to something new. All
6:15:52right. So now we have the following task
6:15:54and it says show creation time using the
6:15:57following format. Now we have a very
6:15:59weird format. So it start with the word
6:16:01day. Then after that we have the
6:16:03abbreviation of the day and then
6:16:05abbreviation of the month. This is the
6:16:07quarter informations. Then the year and
6:16:10after that we have the time and we're
6:16:12going to say whether it's PM or A.M. So
6:16:14it's little bit weird format that you
6:16:16don't see it everywhere but still we
6:16:18want to practice on how to construct
6:16:20such custom format. So let's do it step
6:16:22by step. I'm going to go over here and a
6:16:24new line. So the first one is like day.
6:16:27So we don't have any format for that.
6:16:29It's just like characters. So this one
6:16:32going to be static for all the format.
6:16:34So what we going to do? We're going to
6:16:35say with a string this is the day. So
6:16:39let's go and execute it. So with that we
6:16:41got a static value. Everywhere we have
6:16:43the word day. So that's it. And after
6:16:46that we have a space. So I'm going to go
6:16:47and include it after the day in the
6:16:49string. So we have a day then space and
6:16:52after that we need the abbreviation of
6:16:54the day name. So what we're going to do
6:16:56we're going to go first with the plus
6:16:57operator in order to concatenate the
6:17:00strings. So we need the format function
6:17:03for the creation time. And what do we
6:17:06need? We need the short name. So it's
6:17:09going to be three times the d. Let's go
6:17:11and execute it. Let me just say here
6:17:13custom formats. So now as you can see in
6:17:16the output we have here the day. Then
6:17:17afterward we have space and then the
6:17:20abbreviation of the name of the day. So
6:17:23it looks so far good. Now after that
6:17:25what do we need? We need space and then
6:17:27the abbreviation of the month. So we can
6:17:29go and add all those stuff together with
6:17:31the format here. So we don't have to
6:17:33create two formats. So space and the
6:17:36abbreviation of the month is 3 M. So
6:17:39let's go and test it. Great. So now as
6:17:41you can see we got the abbreviation of
6:17:43the month as well side by side. So we so
6:17:46far we have covered this part. Now we
6:17:48have to move to the second part. So we
6:17:50still need a space and then Q1. Well the
6:17:53Q going to be static. So we cannot go
6:17:55and extend this format. We have to start
6:17:57a new one. So what I'm going to do I'm
6:17:59just going to add a plus here and a new
6:18:02line. So what do we need? We need first
6:18:04a space between the month and the
6:18:06quarter. So let's go and add space and
6:18:09we need the Q as a static value like
6:18:12this. Let me just move it like this. And
6:18:14now after that we need this one like
6:18:16this right so now we need the quarter
6:18:18informations and we don't have format
6:18:20for that that's why we have to go and
6:18:22use the part extraction functions and
6:18:24the one that we're going to use since we
6:18:26are using string I will go with the date
6:18:28name so quarter and we are extracting
6:18:32from the creation time so let's go and
6:18:35test it so now in the output you can see
6:18:37we have everywhere a Q1 and that's
6:18:39because all of those dates are in Q1 all
6:18:41right so now we are so far halfway in
6:18:43our format Not. So now next what do we
6:18:46need? We need like a space and then the
6:18:48year information and then the time
6:18:50information. So now in order to go and
6:18:52get space we're going to do it very
6:18:53simply concatenate and we're going to
6:18:56have space. Now let's go to a new line
6:18:59and in order to get the year I will go
6:19:01with the format as well. So format and
6:19:03what do we have? We're going to have the
6:19:05creation time
6:19:07again. So how we going to format it now?
6:19:10What do we need? We need the year. So
6:19:11it's going to be four times the y and
6:19:13after that we have like space and then
6:19:15the time information. We still can't do
6:19:17that inside the format, right? So we're
6:19:19going to have space here. And then next
6:19:21what do we have? We have the hours. So
6:19:22it's going to be h the small h because
6:19:24here we are talking about the pm and am.
6:19:27It's not the 24hour system. And then
6:19:29after that what do we have? The points
6:19:31double points. Then the minutes going to
6:19:33be small 2 m. And then after that the
6:19:36seconds. So far this is exactly this
6:19:38part over here. And now what is missing
6:19:40a space and the PM the designator. So in
6:19:43order to do that we're going to have a
6:19:45space as well and then small 2 * tt. All
6:19:48right. So we are almost there. Let's go
6:19:51and execute it. Now you can see it is
6:19:53working. So we have the year then space
6:19:56the hours minutes and space and then we
6:20:00have the designator. So this is PM and
6:20:03this is A.M. which is correct. So that's
6:20:06it. We are done. This is how you can
6:20:07create those crazy formats in SQL using
6:20:11the help of format or maybe date name or
6:20:13maybe some static values like we just
6:20:15added here. So I think it's really fun
6:20:17formatting the dates in
6:20:21SQL. Now one use case for the format
6:20:24that I frequently use in my project is
6:20:26using it to format the date before doing
6:20:29aggregations. So it's like part
6:20:31extraction but here we have more
6:20:33customizations on how we represent the
6:20:35date at the reports. So we can show a
6:20:38report like sales by month where we
6:20:40display for example the date as
6:20:41abbreviation name of the month Jan and
6:20:44as well two digits for the year 25. So
6:20:46once we change the format like this and
6:20:48then do data aggregations we will have a
6:20:51nice report about the sales by month. So
6:20:53let's have a quick aggregations using
6:20:54the format. So, we're going to go and
6:20:56say select and now the order date and
6:20:59count the number of
6:21:01orders from our table sales orders and
6:21:06then group by. But now before we start
6:21:09using the order date, we have to go and
6:21:11format it. And then if you take the
6:21:12order date, let's go and execute it. So
6:21:15as you can see the level of details is
6:21:17very high and we have here 10 rows and
6:21:19for each day we have like one order. Now
6:21:21we learned we can go and use the date
6:21:23part in order to extract one part and
6:21:25then aggregate on it. So now instead of
6:21:27that we're going to go and use the
6:21:28format function. So let's go and change
6:21:30the format and it is the order dates.
6:21:33And our format going to be like this. So
6:21:35three big M and then two digits for the
6:21:38year. That's it. And let's call it order
6:21:41dates. And we need this as well for the
6:21:44order date over here for the group I and
6:21:46here a comma. So that's it. Let's go and
6:21:48execute it. So in the output as you can
6:21:50see over here we have three months and
6:21:52here we having the aggregation the
6:21:53number of orders for each month. So now
6:21:55it's like the date part but now we are
6:21:57customizing the format as we want. So we
6:22:00can use the format in order to change
6:22:02the granularity of the date in order to
6:22:04do that
6:22:08aggregations. Now I'm going to show you
6:22:10a real use case for the formatting in
6:22:12real projects. Now our data could be
6:22:14stored in different technologies like
6:22:16the data could be stored in CSV file or
6:22:18we can get our data using an API call or
6:22:21in very common scenario our data could
6:22:23be stored in database. So now what we
6:22:25usually do we go and extract the data
6:22:27from these different sources into one
6:22:29central storage. It could happen that
6:22:31you are getting different formats for
6:22:33the dates and of course this is a
6:22:34problem for analytics. You cannot
6:22:36present different formats for the dates.
6:22:37What we're going to do we're going to go
6:22:38and clean up the formats into one
6:22:41standard format. So that means we have
6:22:43to format the incoming data to new
6:22:45formats and once we have one standard
6:22:47format we can use it in analytics and
6:22:50reports. So this is very common use case
6:22:52in data preparation and in data cleanup
6:22:55by formatting different formats into one
6:22:57standard
6:23:01format. Now in SQL we have many
6:23:04different date and time specifiers and I
6:23:07said they are case sensitive and each
6:23:09one of them has a different meaning. So
6:23:11I prepared for you as well all possible
6:23:13specifiers that we can use with the
6:23:15formats. Not only that, if you go back
6:23:16to the queries that you can find in this
6:23:18chapter, you can find here date format.
6:23:21So all date formats. If you go inside
6:23:23it, you can go and copy the whole query
6:23:26and then go back to SQL then execute it.
6:23:29You can find here a live example because
6:23:31I'm manipulating now the get date. So
6:23:34you can find here a list of all possible
6:23:37date specifiers that you can use with
6:23:38the formats. So I would say go and
6:23:40practice with those different date
6:23:41formats in order to understand what is
6:23:44possible in SQL. So as we learned not
6:23:45only we can change the format of the
6:23:47date, we can change as well the format
6:23:49of the number using the function formats
6:23:51and those are the different possibility
6:23:53that you can use as a specifier for this
6:23:55format in order to change the format of
6:23:58the numbers and as well I have prepared
6:23:59all those different specifiers in one
6:24:02big query. So if you go inside it and
6:24:04copy it and then put it in SQL and
6:24:06execute it, you will find here all
6:24:08different possibilities that we have as
6:24:11a specifier to change the format of the
6:24:17numbers. All right. So what is convert?
6:24:19It's very simple. It's going to go and
6:24:21change the value to a different type and
6:24:23as well at the same time it helps
6:24:25formatting the value. Okay. So let's
6:24:27check the syntax of the convert and it
6:24:29looks like this. It start with the
6:24:31function converts and it accept two
6:24:33parameters the data type first since we
6:24:36can use this function in order to cast
6:24:38the data types. So you can use string
6:24:40integer dates and so on and then we have
6:24:43to specify the value. So which value
6:24:45should be casted. And the last parameter
6:24:47it is optional one where you define the
6:24:50style the format of the value. Let's
6:24:53have this very simple example. We are
6:24:54saying convert to the data type integer
6:24:57int and the value that should be
6:24:59converted is 1 2 3 as a string. So it's
6:25:02going to convert it to integer. We are
6:25:04saying convert to a vchart and the value
6:25:07that should be converted is the order
6:25:08date. So the order date should be a
6:25:10date. So we're going to convert it from
6:25:12date to v charts using the format or the
6:25:15style of 34. So here we are specifying a
6:25:19style a format for this value. And of
6:25:21course it is optional and if you are not
6:25:23using anything the default value that's
6:25:26going to be used is zero. So this is the
6:25:29syntax of the convert in SQL. All right.
6:25:31So now we're going to have few examples
6:25:32on how to work with the convert. So
6:25:34let's go and convert for example string
6:25:36to integer. So we're going to say for
6:25:38example convert. So what is the target
6:25:41data type? It's going to be the integer
6:25:43and the value. It's going to be like for
6:25:45example 1 2 3. So and let's call it like
6:25:48this string to integer and the function
6:25:51is convert. So now in the column name as
6:25:54you can see I'm using here brackets and
6:25:55that's because I'm using like empty
6:25:57spaces and so on and with that I will
6:25:59get more freedom on how to name things.
6:26:01So this is just the name. So this is no
6:26:04function or something. Let's go and
6:26:06excuse it. Now as you can see it's going
6:26:07to work. So we are converting from a
6:26:09string value to an integer and the
6:26:12output this 1 2 3 here is not string.
6:26:15This is the data type of integer. All
6:26:17right. So now let's have another example
6:26:18where we want to convert from string to
6:26:21date. So the target going to be the date
6:26:24and the value let's have this value as
6:26:27usual and we're going to go and call
6:26:30it string to date convert. Okay. So
6:26:35let's go and execute it. Now in the
6:26:37output we will get this information this
6:26:39string as a date. And with that we have
6:26:41converted the data type from string to
6:26:43dates. Now let's have another example
6:26:44where we want to convert the date time
6:26:46to a date. As you remember the creation
6:26:48time is a date time and we would like to
6:26:51have it as only date. So let's go and
6:26:53convert and we would like it to be as
6:26:56well date but this time it's going to be
6:26:58a column called creation time and let's
6:27:01give it the name. So we are converting
6:27:05date time to dates. But of course here
6:27:08we have to go and select. So from sales
6:27:11orders that's it. Let's go and execute
6:27:14it. Now, as you can see in the output,
6:27:15we got only date. I'm going to go and
6:27:17select the creation time in the query as
6:27:20well. So now, as you can see, the
6:27:22creation time was before a date time.
6:27:24So, we have the time information as
6:27:25well. But if you go and cast it using
6:27:28the convert and make it only date. So,
6:27:30SQL going to go and convert it to date
6:27:32and you're going to lose all the
6:27:33informations about the time. So, so far
6:27:35what we are doing here is just casting.
6:27:37So, we are changing the data type from
6:27:39one to another. But in the convert, we
6:27:41can do both. We can do casting and
6:27:43formatting. So let's see how we can do
6:27:45that. I will just get rid of those
6:27:47information at the start. So creation
6:27:49time. And now we're going to go and
6:27:51convert the date time of the creation
6:27:54time to a varchar to a string. And as
6:27:56well to give it the format of the USA
6:27:58standard format. So let's see how we can
6:28:00do that. We're going to start with
6:28:01convert. We are changing now to var. So
6:28:05this is the new data type and the value
6:28:08is the creation time. And now if I don't
6:28:10give it a style, it's going to stay with
6:28:11the standard format, but we would like
6:28:13to have the USA standards. So in order
6:28:16to do that, we're going to go and add
6:28:17the style of the format. So it's going
6:28:19to be 32. So that's it. Let's have a
6:28:23name like this. So
6:28:26USA standard and we are using the style
6:28:30of 32. Let's go with that. This is just
6:28:32a name again. So it's not a function.
6:28:34Let's go ahead and execute it. And now
6:28:36in the output we got a new field and the
6:28:38data type of this field is a varchar. So
6:28:40it's not a date or date time. And as you
6:28:42can see the date now is formatted using
6:28:45this style the 32 the US standard
6:28:47format. So it start with a month then a
6:28:50day and then a year. So now let's go and
6:28:52do the same thing in order to get the
6:28:54standard format in Europe. So I will
6:28:56just go and copy the whole thing. I will
6:28:59just change the style. So instead of 32
6:29:02we're going to go with the 34. And I
6:29:04will just change the name as well. So,
6:29:08so we are just changing the style. Let's
6:29:10go ahead and execute it. Now, as you can
6:29:11see, we got the same thing. We have as
6:29:13well a v jar and the format now is
6:29:15different. So, we have here the day,
6:29:18then the month, and then the year. So,
6:29:20this is how you work with the convert
6:29:21function. You can use it in order to do
6:29:23only casting or not only that, you can
6:29:26do casting and as well formatting. So,
6:29:28you have both things in one function.
6:29:34And now if you're talking about which
6:29:35styles are available, we have many
6:29:37styles that you can use inside the
6:29:39convert. So I have prepared for you a
6:29:41list of all styles that you can use with
6:29:43the convert. So we have styles only for
6:29:45the dates and another styles only for
6:29:47the time and styles for only date time.
6:29:50Now in the download folders you can find
6:29:52here one file called all culture
6:29:54formats. And here you can find one query
6:29:57that I have prepared where you can find
6:29:59inside it the different cultures and the
6:30:01examples. So let's go and copy it and
6:30:03let's go back to scale paste it and
6:30:05let's see the results. So now if you
6:30:07check the output we got the first column
6:30:09is the cultures that is used. So we have
6:30:11a lot of cultures like around 17s and
6:30:13you can see how the numbers are
6:30:14formatted or the date is formatted based
6:30:16on this culture. So it's really fun. You
6:30:18can check here for example how the
6:30:20format in Japan or Korea or France and
6:30:23the German one. If you scroll down, you
6:30:25can find the Arabic, the Russian and so
6:30:27on. So you can see the format of each
6:30:29dates is changing based on the culture.
6:30:32So I would say have fun. Go and try
6:30:34those different cultures formats in
6:30:36order to format your numbers or
6:30:40dates. So what is the cast function? It
6:30:43going to go and convert a value to a
6:30:46different data type. So it turns one
6:30:48data type to another. All right. So now
6:30:50let's check the syntax of the cast. I
6:30:52really like this one. It is not typical
6:30:54like format or syntax in SQL. So it says
6:30:57the cast is the function and then inside
6:31:00it we need two things but it's not
6:31:02separated like with the comma as we
6:31:03learned before with all other functions
6:31:05but this time is separated with the
6:31:07keyword as. So it's like the natural
6:31:09English you are saying cast the value as
6:31:12a data type. So you are casting the
6:31:14value to a new data type. So let's have
6:31:16this very simple example we have here
6:31:18cast the value 1 2 3 as integer. So
6:31:22previously it is string and it going to
6:31:24be converted to integer. So as you can
6:31:26see it's very simple. Now in this
6:31:27example we are saying cast this value
6:31:30this string value as a dates. So
6:31:32converted from string to dates. So as
6:31:34you can see with the cast we don't have
6:31:35here any option of formatting or styling
6:31:38the values. So it's only dedicated for
6:31:40casting the value from one data type to
6:31:43another one. So this is the syntax of
6:31:45the cast. It is very straightforward and
6:31:47really nice function. Okay. So now let's
6:31:49have a few examples about the cast. So
6:31:51let's go and convert a value from a
6:31:53string to integer. So it's very simple.
6:31:55We're going to say cast. So now we need
6:31:57the value. So let's go with the 1 2 3.
6:32:00So we have here a string. And then we're
6:32:01going to say as and then we have to
6:32:03define the data type. So the data type
6:32:05going to be integer. So that's it. So
6:32:07let's give it the name like this string
6:32:10to integer. Let's go and execute it. Now
6:32:13as you can see we got the value but with
6:32:14the data type integer. From string to
6:32:16integer. Now let's do the way around. We
6:32:19cast from integer to string. So we're
6:32:21going to say cast 1 2 3 as var jar and
6:32:26we're going to give it a
6:32:27name int to string. So let's go and
6:32:31execute it. Now in the output we have 1
6:32:332 3 but this time it has the data type
6:32:36varchar. Now let's go and work with the
6:32:38date. So we're going to go and convert a
6:32:40value a string value to a date. So our
6:32:44value going to be the usual one and we
6:32:48want it from string to date. So we're
6:32:50going to have the data type as date. So
6:32:52let's give it a name string to date.
6:32:56Let's go and execute it. Now we're going
6:32:58to have this value with the data type
6:33:00date. So that's it. Now let's say that I
6:33:02would like to have this value but as
6:33:04date time. So I will just copy the whole
6:33:07thing and go to a new line and say date
6:33:10time two. So the name of this going to
6:33:13be string to date time. Let's go and
6:33:16execute it. Now in the output as you can
6:33:17see we are getting not only the date but
6:33:20as well we are getting the time
6:33:21information. But now since we didn't
6:33:23provide SQL with any time information
6:33:25SQL going to go and show it as zeros.
6:33:27Now let's do one more casting where we
6:33:29change the data type from date time to
6:33:31date. So now we need our creation
6:33:33time but we have to get it from the
6:33:36tables. So from sales orders let's go
6:33:39and execute it. So now in the output you
6:33:41can see the creation time is a date
6:33:42time. We have the time information but
6:33:44we are not interested about the time
6:33:45information. I would like to have this
6:33:47field as a date. So it's very simple
6:33:49what we're going to do. We're going to
6:33:50say cast. Now the value is creation time
6:33:55and then the keyword as and we need it
6:33:57as a date. So we're going to give it the
6:33:59name date time to date. So let's go and
6:34:03execute it. Now as you can see in the
6:34:04output we got the creation time but only
6:34:07with the date information. We don't have
6:34:09anything about the time. So we get it as
6:34:11a date instead of date time. So that's
6:34:14it. This is amazing function SQL and
6:34:16it's very simple and we can use it only
6:34:18for casting. So only to change the data
6:34:20type from one to another. And we cannot
6:34:22use this function in order to change the
6:34:24format. So if you are casting you will
6:34:26get always the standard format from
6:34:32SQL. So now let's go and compare our
6:34:34functions side by side. So we have our
6:34:36three functions. cast, convert and
6:34:38format and we can do two things either
6:34:41casting or formatting. So by the casting
6:34:44for the first function cast we can
6:34:45change any type to any other type. So
6:34:48there is no restriction at all. The same
6:34:49thing for the converts the same thing we
6:34:52can convert anything to anything. But
6:34:53for the format we can change only to a
6:34:56string. So any data type like a date or
6:34:58number to a string value because the
6:35:01main thing for the format is not
6:35:03changing the data type. Now if you are
6:35:04talking about changing the format of the
6:35:06values, you cannot use the cast function
6:35:08in order to change the format. So the
6:35:10cast function is only for casting. It
6:35:12makes sense. Now about the convert, we
6:35:14can use it in order to change the format
6:35:17of the date and time. But we cannot use
6:35:19it in order to change the number
6:35:20formats. And for that we have a
6:35:22dedicated function called format. So we
6:35:25can use it to change the format of the
6:35:26date and time and as well the numbers.
6:35:28So those are the main differences
6:35:30between those three functions. All
6:35:31right. So with those three functions we
6:35:33have learned how to do formatting and
6:35:35casting on date informations. Now moving
6:35:38on to the third group we have the date
6:35:40calculations and here we have two
6:35:41functions on how to do date calculations
6:35:43or mathematical operations on the dates.
6:35:48If okay so now we're going to start with
6:35:50the first function the date add. So what
6:35:53is date add? Date add can allow us to
6:35:55add or subtract a specific time interval
6:35:59to or from a date. So let's understand
6:36:01how the date add work. So here again we
6:36:03have our date August 20th 2025. So now
6:36:06in some scenarios we would like to add
6:36:08years to our dates. So for example let's
6:36:11say I would like to add three years to
6:36:12our date. So we can do that using the
6:36:14date ad. So if you do that in the output
6:36:17you will get 2028 August 20th only the
6:36:20date part is changed and where we have
6:36:22added three years but in other scenarios
6:36:25you would like to go and add months. So
6:36:27for example let's go and add two months
6:36:29to the August. So in the output you will
6:36:31get 2025 10 20 with that we have added
6:36:34two months and of course we can go and
6:36:36add days to our dates. So for example
6:36:39we're going to go and add five days to
6:36:41our date. So in the output we'll get the
6:36:43same year 2025 the same month August but
6:36:46only the day will be changed to 25. So
6:36:49we have added five days to the original
6:36:51dates. And of course we can go and
6:36:53subtract dates even though that the
6:36:54function called date add. So for
6:36:56example, we can go and subtract three
6:36:59years from our dates and we will get So
6:37:01if you do that, you will get 2022 August
6:37:0420th or if you go and subtract two
6:37:06months from our dates. So it's going to
6:37:08stay the same year 2025. But this time
6:37:10instead of August, we will go back to
6:37:12June with the same date 20. And the same
6:37:14thing going to happen for the days if
6:37:16you go and subtract five days. So the
6:37:18same year 2025, the same month August,
6:37:21but only the days going to be instead of
6:37:2320, it's going to be 15. So as you can
6:37:25see with the date ad you can manipulate
6:37:27the years, the month and the days by
6:37:30subtracting or adding new intervals. So
6:37:33this is how the date ad works. All
6:37:35right. So now let's check the syntax of
6:37:36the date ad. And here things little bit
6:37:38more complicated. We have to provide
6:37:40three informations. The first one is a
6:37:42part. What do you want to add? Do you
6:37:44want to add years or months or days and
6:37:47so on. Then the second one is interval.
6:37:50So it's like how many days? How many
6:37:52years? How many months? And then the
6:37:54last one is the date. This is the date
6:37:56that we're going to be manipulating by
6:37:57adding or subtracting intervals. Let's
6:38:00check the following example. We are
6:38:02saying here date add. So what is the
6:38:04part here is a year. That means we want
6:38:06to manipulate only the year parts. Then
6:38:09the interval here is two. So it is
6:38:11positive. We want to add two years. So
6:38:14it's going to go to each order and start
6:38:15adding two years for each date value.
6:38:18Now let's check another example. Here we
6:38:20are saying date add month. So here we
6:38:22want to manipulate the month part. But
6:38:24here we are saying minus4 that means we
6:38:27want to go and subtract four months from
6:38:30each value in the order date. So as you
6:38:33can see the value of the interval
6:38:34whether it's positive or negative. We
6:38:36are controlling here the function
6:38:37whether it is subtraction or addition.
6:38:40So let's have few examples about the
6:38:41date add using our field order dates. So
6:38:44for example let's go and add two years
6:38:46for each date. So we can do it like this
6:38:49date adds. So we are adding years that's
6:38:52why we're going to go with the part year
6:38:53and how many years we are adding we are
6:38:54adding two years. So this is our
6:38:56interval and our field our value is the
6:39:00order date. So now in the output as you
6:39:02can see we got a date but this date is
6:39:04always 2 years higher than the order
6:39:06date. So everywhere you have see 2027.
6:39:09Now let's go and add maybe three months
6:39:12for each date. Just going to go and copy
6:39:14it and say a month. Let's change the
6:39:17interval to three and we're going to
6:39:19call it three
6:39:21months later. So now if you check the
6:39:23output over here we have a new date but
6:39:25now the difference between it and the
6:39:27order date we have here always three
6:39:29months more than the order dates. So for
6:39:31example here we have January but in the
6:39:34new one we have April and for the next
6:39:36one we have February and in the new
6:39:38field we have May. So as you can see we
6:39:40are adding months over here. So as you
6:39:42can see we are adding monthses to our
6:39:45original filled order date. Now let's
6:39:47say that I would like to go and subtract
6:39:4810 days. So let's go and do the same. So
6:39:52we're going to have the date add. Since
6:39:54we are talking about the days, it's
6:39:55going to be the day. We're going to
6:39:56subtract 10 days. So minus 10 for the
6:40:00order date. So let's call it 10 days
6:40:04before. Let's go and execute it. Now we
6:40:07got as well a new date. And this date
6:40:09has always 10 days before the order
6:40:12date. So for example, let's take the
6:40:14order number seven. In the order date we
6:40:16have 15, but in the new column we have
6:40:18five. So we have subtracted 10 days from
6:40:22the original filled order dates. So as
6:40:24you can see it's very simple to add or
6:40:26subtract days, year, months using the
6:40:29date
6:40:32add. All right. So what is date diff?
6:40:35diff stands for difference and date diff
6:40:38can going to can allow us to find the
6:40:40differences between two dates. All
6:40:42right. So let's understand how the date
6:40:44diff works in SQL. Now imagine we have
6:40:46two dates. We have the order date 2025
6:40:50August 20th and the shipping date is the
6:40:531st of February in the next year 2026.
6:40:56Now we might ask the question how many
6:40:58years have passed between the order date
6:41:01and the shipping date. So in order to
6:41:03answer this question we can use the
6:41:05function date diff and we can define the
6:41:07part year. If you do it like this it's
6:41:09going to subtract those two dates and it
6:41:11going to return one. So the date
6:41:13difference between those two dates is
6:41:15exactly one year. But now if the
6:41:17question is how many months are between
6:41:19the order date and the shipping dates.
6:41:21So here again we can go and use the date
6:41:23diff between the order date and the
6:41:24shipping date but we use the part month.
6:41:27If you do it like this in the output you
6:41:29will get three months. And now of course
6:41:31if the question is how many days are
6:41:34between the order date and the shipping
6:41:36dates. So here we can use the function
6:41:38date diff where we specify the day
6:41:40inside it and in the output you will get
6:41:4268. So this is how the date diff works.
6:41:45You go and subtract two different dates
6:41:47and you will get in the output a number
6:41:50how many years how many months how many
6:41:52days. So that's it. All right. Now to
6:41:54the syntax of the date diff. It accept
6:41:56here as well three parameters. So the
6:41:58first one is the parts as usual year,
6:42:01month, day. And then here we need two
6:42:03dates, not only one, we need two. So we
6:42:05need the starting dates and the ending
6:42:08dates. So that means here we have the
6:42:10youngest dates and the end date going to
6:42:12be the oldest dates. So for example,
6:42:14here we have date diff and we are saying
6:42:16find the differences in years between
6:42:19the order dates. This is the start date
6:42:22and the shipping dates. So which dates
6:42:25normally happen? First we have to order
6:42:26something. So we have the order date and
6:42:28once you order what can happen next is
6:42:30the shipping date. That's why the
6:42:32shipping date is as an end date. So we
6:42:35want to find the differences between
6:42:36them in years or of course if you want
6:42:38to find the differences between them in
6:42:40days we have to go and change the part
6:42:42from year to day. So as you can see the
6:42:44syntax is very simple and very logical
6:42:47right. All right let's have the
6:42:48following simple task and it says
6:42:49calculate the age of employees. So let's
6:42:52see how we can solve that. So we're
6:42:53going to go and select first all the
6:42:55informations from employees. So sales
6:42:58and employees. Okay, let's execute it.
6:43:01Now in the employees, we don't have any
6:43:03informations about the age, but we have
6:43:05the birthday. So we can go and transform
6:43:07this birthday to an age. And of course,
6:43:10how we calculate the age? We count how
6:43:12many years between this year and the
6:43:14birthday. So that means we have to go
6:43:16and use two functions the date diff and
6:43:19the get day in order to have the year of
6:43:22the current year. So that means we have
6:43:23to go and use the function date diff. So
6:43:25let's go and do that. I'm going to go
6:43:27first selecting only few informations.
6:43:30So employee ID and P date. So let's
6:43:34start with the date diff. So if we are
6:43:36talking about the age we are calculating
6:43:37how many years that's why we're going to
6:43:39say as a part going to be the year. So
6:43:42what is the starting date is the birth
6:43:44date of the person. So it's going to be
6:43:46the birth date. And now we need the end
6:43:48date. We don't have here anything about
6:43:50the end date. The end date going to be
6:43:51the current year. So in order to get the
6:43:54current year, we're going to go with the
6:43:55function get dates. And with that we are
6:43:58getting the current date information.
6:44:00And this is exactly what we want. So
6:44:02let's close it and let's go and call it
6:44:04an age. So it's very simple. We are
6:44:06counting how many years between the
6:44:08birth dates and the current dates. So
6:44:11let's go and execute it. So now we are
6:44:12getting the ages. As you can see the
6:44:14first person is 33, the second one is 52
6:44:17and so on. And now you might getting
6:44:19different values than I'm getting now.
6:44:22And that's maybe you are doing the
6:44:23course now in 2025 or 2026 and the
6:44:26employees going to be older than now.
6:44:28Now we are 2024 and I'm getting those
6:44:30ages. So this is how we calculate the
6:44:32age using the help of two functions. The
6:44:35date diff and the get date. Okay. Okay,
6:44:37so now we have another task for the day
6:44:39diff and it says find the average
6:44:41shipping duration in days for each
6:44:44month. So here we have a lot of
6:44:45informations. Let's do it step by step.
6:44:47Let's first find out the shipping
6:44:49durations in days. So let's go and
6:44:51select few informations from our table.
6:44:53So select order ID. We have the order
6:44:57date, ship
6:44:59date and I think that's it. So from
6:45:03sales orders. So let's go ahead and
6:45:06execute it. So now we have our 10
6:45:08orders. We have the order date and the
6:45:09shipping dates. Now we have to go and
6:45:11create a new field called shipping
6:45:13duration. So what is the shipping
6:45:15duration? It is the number of days
6:45:18between the order dates and the shipping
6:45:20dates. So how many days it took from the
6:45:23order placement until the day of the
6:45:26shipping. So that means we have two
6:45:27dates and we have to go and find the
6:45:29differences between them. We're going to
6:45:31go with the function date diff. So now
6:45:33since we are saying in days we have to
6:45:35go with the part day. So what is the
6:45:38start date? The start date is the order
6:45:40date. And what is the end date? It's
6:45:43going to be the shipping dates like
6:45:45this. So I'm going to call it day to
6:45:48ship like this. Let's go and execute it.
6:45:51So now by checking the result for
6:45:52example for the order one it is ordered
6:45:54at the 1st of January and it is shipped
6:45:57on 5th of January. So between those two
6:45:59dates we have around 4 days. So four is
6:46:02the shipping duration and if you go to
6:46:04the order number three the differences
6:46:06between the order date and the shipping
6:46:08date we have around 15 days. So with
6:46:10that we have solved this part shipping
6:46:12duration in days. But now the task says
6:46:15we have to find the average duration for
6:46:18each month. So that means we have to go
6:46:20and select for example the month of
6:46:21January and find the average duration.
6:46:24So we have to go and do a simple
6:46:25aggregation. We're going to go to the
6:46:27date if at the start and say average.
6:46:29And we're going to close it over here.
6:46:31And let's go and rename it average
6:46:34shipping. And now we have to aggregate
6:46:36by the month. So we don't need the whole
6:46:39order dates. We need the month of the
6:46:42order date. So like this. We don't need
6:46:44of course the order ID, but now we need
6:46:47to group up the data using this
6:46:49dimension, the month order dates. So
6:46:52that's it. Let's go and execute it. So
6:46:54now in the output you can see we have
6:46:56three months and for each month we have
6:46:58the average shipping durations in days.
6:47:01So for the first month it is around 7
6:47:04days for February is as well 7 days and
6:47:07for March we have less duration 5 days.
6:47:09So with that we have solved the task. As
6:47:11you can see the date diff is very strong
6:47:13function in order to do data analytics
6:47:16using the dates information. All right.
6:47:18Right. So now we have the following task
6:47:19and it says find the number of days
6:47:21between each order and the previous
6:47:24order. So there's a lot of stuff going
6:47:25on over here. Let's do it step by step.
6:47:28Let's start by selecting the basic
6:47:29stuff. So select order ID, order date
6:47:34from the table sales orders. Let's go
6:47:38and execute it. So we have our 10 orders
6:47:40and we have the current order dates. So
6:47:42now we have to find the differences
6:47:43between two dates. order dates, the
6:47:46current one and the previous order
6:47:48dates. So in our data, we have the
6:47:50current order dates, but we don't have
6:47:52the previous order date for each order.
6:47:54And in order to calculate the previous
6:47:56one, do you remember about the window
6:47:58functions? We can go and use the lag in
6:48:00order to access a value from a previous
6:48:03records. So let's go and do that. The
6:48:06order date, I'm just going to call it
6:48:07current order dates. And let's go and
6:48:10find the previous order dates. So we're
6:48:12going to go with the lag of the order
6:48:15date because we are interested in the
6:48:17value of the order date. Now over we
6:48:20have to sort the data. So we're going to
6:48:22sort it
6:48:24by the order date as well. So this is
6:48:27going to help us always to access the
6:48:30previous value of the order date. So
6:48:32we're going to call it
6:48:33previous order date. Let's go and
6:48:36execute it and let's check the result.
6:48:38For the first order, we don't have
6:48:39anything previously. So that's why we
6:48:41are getting a null. For the second
6:48:43record, the current order date is the
6:48:455th of January and the previous one is
6:48:48the 1st of January. And this value comes
6:48:50from the previous record, the previous
6:48:53order. Great. Amazing. So with that we
6:48:55have now the two dates, the current date
6:48:57and the previous one. And now we can go
6:48:59very simply finding the number of days
6:49:01between those two dates. And we can do
6:49:04using the amazing function date diff. So
6:49:07we are interested on the days that's why
6:49:09it's going to be the day. So what is the
6:49:12starting day? If you check those two
6:49:13dates, you can see that the previous
6:49:15order date is the starting date. So
6:49:18we're going to take the whole thing, the
6:49:20whole window function and put it over
6:49:22here. So I just moved my picture. So
6:49:24here is the previous order dates. And
6:49:26now the end date, what's going to be?
6:49:28It's going to be the current order date
6:49:30which is our order date like this. So
6:49:33again, we are finding the number of days
6:49:35between the previous dates and the
6:49:38current dates. So that's it. Let's close
6:49:41it. So I'm just going to call it number
6:49:43of
6:49:44days. So let's go and execute it. Now of
6:49:47course we have here null. So we will get
6:49:49as well null in the output. And now you
6:49:51can check over here how many days
6:49:53between those two dates. We have exactly
6:49:55four days. And as well for the next one
6:49:57we have around 5 days, 10 days and so
6:50:00on. So we have solved the task. We have
6:50:02now the number of days between each
6:50:04order and the previous order. So this
6:50:06type of analyszis is very important in
6:50:08the business. We call it time gap
6:50:10analyzes and we have done it using the
6:50:12help of the window function and as well
6:50:14the date function date diff. So date div
6:50:17function is amazing function to do data
6:50:20analyzes. All right. So with those two
6:50:21functions we have learned how to do
6:50:23mathematical operations on date
6:50:25informations or we can call it date
6:50:27calculations. Now moving on to the
6:50:28easiest and the last group, we have the
6:50:31date validation. And here we have only
6:50:33one function, the is
6:50:37date. Okay. So what is is date? So the
6:50:41is date is very simple. It's going to
6:50:42check whether a value is a date. So it
6:50:45going to return one if the string value
6:50:47is a valid date or zero if it is not a
6:50:51valid date. Okay. So let's check quickly
6:50:53the syntax of the is date. It's very
6:50:55simple. The keyword is date is the
6:50:57function name and it accepts only one
6:50:59value. So for example you can pass a
6:51:00string like this and you can ask SQL is
6:51:03it a date. So is date and the value and
6:51:06of course for this example you will get
6:51:07true or one. So as you can see we are
6:51:09passing here a string value and we are
6:51:11validating whether it is good enough to
6:51:14be a date or as well you can go and
6:51:15specify a number like here 2025. So is
6:51:19this value a date and of course SQL
6:51:21going to accept it and say yeah this is
6:51:22a year so you will get as well a one. So
6:51:24you can pass as well a number or
6:51:26integer. So you are just checking the
6:51:28values whether they are suitable enough
6:51:30to be a date. So that's all about the
6:51:32syntax of the is dates. Okay. So now
6:51:33let's have few examples. For example,
6:51:35let's go and select and we're going to
6:51:36say is date and we will check a value.
6:51:39So let's say this value is a string 1 2
6:51:423. Let's go and call it date. Check one.
6:51:46Let's go and execute it. Now in the
6:51:48output it's going to say no, it is not a
6:51:50date. And that's why we are getting the
6:51:52value zero which is correct because 1 2
6:51:543 is not a date. Let's pick another
6:51:57value. The same thing is dates. And now
6:51:59the value going to be the following. So
6:52:022025 August 20. So let's call it date
6:52:06check 2. And let's go and execute it.
6:52:09Now in the output we will get one. That
6:52:11means the value that we have provided is
6:52:14a date. And that's why we have a one in
6:52:16the output because ESKL is saying this
6:52:18is a date. Now let's have another
6:52:20example. We're going to take the whole
6:52:22thing. So this is a check three and
6:52:26remove this from here. But I would like
6:52:27to go and change the format. So let's
6:52:29say that we start with the day then
6:52:31month and then the year. Let's go and
6:52:34check. Now in the output you can see it
6:52:36is zero because SQL does not understand
6:52:38the formats. So we are not following the
6:52:40standard format of the database and
6:52:42scale and that's why going to say no
6:52:44this is not a date. This is like a
6:52:46string value. So this means only if the
6:52:49value is following the status format SQL
6:52:51going to understand this is a date. Now
6:52:53let's go and check another thing for
6:52:55example let's say is date and let's have
6:52:57only the year. So 2025 and let's give it
6:53:01the name date check for let's go and
6:53:04execute it. Now in the output we will
6:53:06get one. So that means is considering
6:53:08this value as a date. So that means
6:53:11Iskll is smart enough to understand okay
6:53:12we have provided a year information and
6:53:15is going to accept it and say okay maybe
6:53:17this is the 1st of January of 2025. Now
6:53:20let's go and do the same thing but for
6:53:22the month let's see whether SQL going to
6:53:24accept it. So check five and we have the
6:53:27month of August. Let's go and check now
6:53:30going to say no I don't understand this
6:53:32value this is zero. So that mean this
6:53:35value is provided is not a date. So by
6:53:37checking those results as you can see
6:53:39SQL understand only the standard formats
6:53:41and it allow you as well to check
6:53:43whether a year is a date. So this is how
6:53:45the is date works in SQL. And now you
6:53:47might ask well when I'm going to do this
6:53:49when I'm going to check whether the
6:53:51value is a date or not. Let me give you
6:53:52this following scenario. Now imagine
6:53:54that we have the following date. So we
6:53:56have four values as a string. And now if
6:53:58you check the data you can see that we
6:54:00are following the standard format but
6:54:02only one value has an issue. So we have
6:54:05here data quality problem. So now what
6:54:06we want to do, we want to go and cast
6:54:08this string value to a date. We don't
6:54:11want this to stay as a string value. We
6:54:13would like to have it in the final
6:54:15result as a date. So what we usually do
6:54:17is that we go and have like subquery on
6:54:20top of those values. So like this. So
6:54:22now what we're going to do, we're going
6:54:23to go and say we would like to go and
6:54:25cast the order dates as date. We don't
6:54:29want it as a string. And we're going to
6:54:31call it order dates from these values.
6:54:34So let me just make it like this and
6:54:37let's go and execute it. Now SQL going
6:54:39to give you an error and say well I
6:54:41cannot convert everything to a date
6:54:44because you have maybe corrupt data and
6:54:46this is of course because of this row.
6:54:48So SQL is not able to convert this
6:54:51string to a date. But of course now the
6:54:53example is very simple. We know that but
6:54:55if you have a huge table it's going to
6:54:56be really hard to identify those issues.
6:54:59But now still I would like to go and
6:55:01convert those value here. I don't want
6:55:03to get an error. And now if there is
6:55:04like some values like here that is
6:55:07corrupt and so on this value could be
6:55:09null. So how we can force SQL to convert
6:55:12the data type from string to date and
6:55:14not give us this error. And for this we
6:55:17can go and use the help of the function
6:55:19is date. Let me show you how I usually
6:55:20do it. So let's go and say let's check
6:55:23whether the order date is a date. So
6:55:26let's have it like this. And now before
6:55:28we go and execute, I'm going to make
6:55:30this as a comment because if I execute
6:55:32it like this, we will get an error. And
6:55:34let's go and get the order date in our
6:55:36select. So let's go and execute it. Now
6:55:38as you can see in the output, we have
6:55:40our string value. So they are not yet a
6:55:42date. And we have the result of our
6:55:44check. So as you can see the first row,
6:55:46we are getting a zero. So it's saying
6:55:48this value is not a date. But for all
6:55:51other values, we are getting one. So
6:55:53they are passing the check and they are
6:55:54dates. So now what we're going to do
6:55:56we're going to go and build a logic
6:55:57where we're going to say go and cast the
6:56:00value from string to date only if the
6:56:03flag or the check is equal to one. So
6:56:06that means we can go and use the help of
6:56:08the case when statement. Let me show you
6:56:10how we can do that. So let's do it step
6:56:11by step. We're going to say case win.
6:56:14Now we need the check. So is
6:56:16dates the order date. So if the output
6:56:20of this check is equal to one then you
6:56:23are allowed to do the casting. So let's
6:56:24go and get the cast as a result of this
6:56:28condition and if it's not equal to one
6:56:30then it could stay as a null. So let's
6:56:32have it as a null if it didn't pass the
6:56:34test. So end and we can call it new
6:56:36order dates. So now let's go and execute
6:56:39it. Now as you can see we are not
6:56:41getting error from SQL. So now if you
6:56:42check the output for the invalid dates
6:56:45we are getting a null. So we are not
6:56:47getting an SQL error. And now only if
6:56:49these string values are a valid dates
6:56:52it's allowed to be casted. So that you
6:56:54can go and cast a string value to a date
6:56:57even though that you have bad data
6:56:58quality and this is very important step
6:57:00in order to prepare the data before
6:57:02doing analyszis and it help us as well
6:57:04to find data quality issues. So for
6:57:06example we can go over here and say you
6:57:08know what let's go and search for all
6:57:10issues. So we're going to go and take
6:57:12the is dates. So let's go and get the
6:57:14check and I'm going to say let me see
6:57:16all string values that are invalid that
6:57:19are failing the test. So let me execute
6:57:21it. And with that we are getting this
6:57:22record. And now imagine we have a lot of
6:57:24data. So it's now it's really easy to
6:57:27identify those issues by just using the
6:57:29S dates. So this is as well amazing way
6:57:31in order to identify data quality
6:57:33issues. Now of course you might say you
6:57:35know what I don't want to see here null.
6:57:36Maybe let's get a dummy value. Well it's
6:57:39very easy. We can go over here and say
6:57:41else. So and we can go and get for
6:57:43example very large value something like
6:57:46this that is easy to identify. So now
6:57:48with that instead of getting nulls
6:57:50inside your data you can get such a
6:57:52dummy value. So now you understand the
6:57:54use case of the is dates and why this
6:57:56function is amazing doing data
6:58:02cleanup. All right. So with that we have
6:58:04covered 13 different date and time
6:58:07functions in SQL. So we have learned how
6:58:09to extract the date parts using seven
6:58:12different functions and we have learned
6:58:13as well when to use which one. So they
6:58:15are amazing in order to do data
6:58:17aggregations and as well filtering. And
6:58:19then we have learned how to change the
6:58:21date format from one to another and as
6:58:23well how to change the data types. And
6:58:25then we learned how to do mathematical
6:58:27operations on our dates. So how we can
6:58:30add or subtract days, years, months from
6:58:33a date or the amazing function the date
6:58:36diff where we can go and find the
6:58:38differences in days or years between two
6:58:40days. And the last one we can go and
6:58:42validate whether the values that we have
6:58:44are dates or not. So as we learned date
6:58:47functions are amazing functions in order
6:58:49to do data analyzes and reporting. All
6:58:51right my friends. So with that we have
6:58:53learned a lot of very important SQL
6:58:55functions and how to manipulate the date
6:58:57and time values in your database using
6:58:59SQL. Now in the next section we're going
6:59:01to start talking about the null
6:59:03functions in order to handle the nulls
6:59:05inside your tables. So let's go.
6:59:11So what are the nulls? Imagine you are
6:59:13filling out a forum and there will be
6:59:15usually like fields that are required
6:59:17and another fields that are optional. So
6:59:20what usually happens? We leave those
6:59:21optional fields unanswered. So we don't
6:59:24provide any values and we leave it
6:59:26empty. And now once we are done filling
6:59:28out the form and we click on register,
6:59:30the data will be inserted into database
6:59:33tables. So now what can happen? The
6:59:35fields where you have provided answers
6:59:36and values can be filled inside the
6:59:38table while the unanswered fields will
6:59:41have no value and this is what we call
6:59:44in SQL a null. So in databases a null
6:59:47means nothing unknown. It is not equal
6:59:50to anything. So it is not equal to zero
6:59:52or empty string or blank space. A null
6:59:55is simply nothing. It tells us there is
6:59:58no value and it is missing. It's like
7:00:01saying I don't know what this value is.
7:00:03So this is what a null means in
7:00:08SQL. All right friends, so now we're
7:00:11going to do a deep dive into special SQL
7:00:13functions on how to handle the nulls
7:00:16inside our data. Now in some scenarios
7:00:18we have nulls inside our tables and we
7:00:21would like to go and remove it and
7:00:23replace it with a new value like for
7:00:25example 40. And in order to do that in
7:00:27scale we have two functions. The first
7:00:29one called is a null and the second one
7:00:32called coales. But now let's say that we
7:00:35have another scenario where we have a
7:00:37value inside our table like the 40 and
7:00:39we want to go and make it as a null. So
7:00:41now we are doing the exact opposite. We
7:00:43are replacing the value with a null and
7:00:46for that we have the SQL function null
7:00:48if. So as you can see with those two
7:00:50scenarios we are replacing stuff. So
7:00:52from null to value or from value to
7:00:55null. So they are really helpful in
7:00:57order to manipulate the data inside our
7:00:59databases. Now moving on to another
7:01:01scenario where we don't want to
7:01:02manipulate anything. We want just to
7:01:04check. So we don't want to replace or
7:01:06convert anything. We want just to check
7:01:08in our database whether we have a null
7:01:11value. And for that we have a function
7:01:13called is a null. But between the is and
7:01:15null there is like space. It is
7:01:17different than the first function. So if
7:01:19you apply is null you're going to get a
7:01:21boolean true or false. For this scenario
7:01:23you will get true. Or the second option
7:01:25you can go and check whether the value
7:01:27is not null. So we can use is not null
7:01:29and for this example you can get false.
7:01:31So in the output we are getting a
7:01:33boolean true or false. So those keywords
7:01:35are really amazing in order to check
7:01:38whether we have nulls inside our data.
7:01:40So this is the big picture of all
7:01:41functions that we have in SQL in order
7:01:43to handle the nulls. So now let's go and
7:01:45understand those functions one by one.
7:01:52So let's start with the first function
7:01:53is null. Is null going to go and replace
7:01:56a null with a specific value. Now the
7:01:59syntax of the isnull is very simple.
7:02:00We're going to use the keyword is a null
7:02:03and it accepts two arguments. First the
7:02:05value and then the second the
7:02:07replacement value. So let's have an
7:02:09example. We can go and use the is null
7:02:11for the column called shipping address.
7:02:13So we are checking the nulls inside it.
7:02:15And if SQL encounters any null, it going
7:02:18to go and replace it with the value
7:02:20unknown. So this going to be like a
7:02:22default value for the nulls. So the
7:02:24first value is a column and the second
7:02:26value is like static. Always going to be
7:02:29the unknown if we find any nulls. Now of
7:02:32course in other scenarios we don't want
7:02:33to have it always like the unknown. We
7:02:36would like to use another column to help
7:02:38the first one. So let's have this
7:02:40scenario. So now with this syntax we are
7:02:42checking the values of the shipping
7:02:44address and if we find any nulls it's
7:02:47going to get the replacement from the
7:02:49billing address. So here in this example
7:02:51we have two columns. We don't have here
7:02:53any static value. We will get the values
7:02:55of the billing address only if the
7:02:58shipping address is null. So we are
7:03:00replacing the nulls using the help of
7:03:02other column. And in the first scenario
7:03:04we are replacing the nulls with a static
7:03:07value the default value. So let's have a
7:03:09very simple example in order to learn
7:03:11how this works. So what we are doing we
7:03:13are checking whether the value is null.
7:03:15If it's yes then we're going to go and
7:03:17get the value from the replacement and
7:03:20if the value is not null then show the
7:03:22value itself. So we have the following
7:03:24example. We are going to check the
7:03:25values from the shipping address and if
7:03:28there is nulls then go replace it with
7:03:30the default value na. So let's see how
7:03:32going to go and execute this very simple
7:03:34example. We have two orders. The first
7:03:36order we are checking the submit address
7:03:39is the value of this address is null.
7:03:42Well, no. We have a value a. So that's
7:03:44why it's scale going to go and return
7:03:46the same value. So in the outputs we
7:03:49will get a. So if it's not null, it's
7:03:52going to return the same value. So now
7:03:53it's going to move to the second order
7:03:55and here we have the shipment address as
7:03:57a null. So what going to happen here? If
7:03:59the value is null, then we going to get
7:04:01the replacement value. So what is the
7:04:04replacement value is the NA. So that's
7:04:07why in the output we will not get a null
7:04:09we will get the N A. So if you check the
7:04:12result what happens? We're going to get
7:04:14the addresses from the shipping address
7:04:16but only if we have a null we will get
7:04:18like default value. It's very important
7:04:20to understand if you are using the
7:04:22default value in the output you will
7:04:24never get a null. All right. So let's
7:04:26have another example for the second
7:04:27scenario where we are not using a
7:04:29default value we are using a column. So
7:04:32we have a supportive column that's going
7:04:33to be checked. So in this scenario we
7:04:35are saying is null shipping address and
7:04:37billing address. So we have two columns
7:04:39and of course the logic going to be the
7:04:41same right. So we are checking only
7:04:43once. Let's see how SQL going to execute
7:04:45this example. We have this time three
7:04:47orders and we have addresses from the
7:04:49shipments and as well from billing. So
7:04:51now SQL is always focusing on the
7:04:53shipping address since it is the first
7:04:55column. So we are not checking the
7:04:57billing address at all. So it start with
7:04:59the first order. Is it null? Well, no,
7:05:01we have the value A. So, we will get it
7:05:04as well in the output and SQL will not
7:05:07get anything from the billing address.
7:05:09So, we will get a. So, that's it for the
7:05:11first order. Now, it's still going to go
7:05:13to the second order. And this time,
7:05:14we're going to have a null. So, now in
7:05:16the rule, we are saying if the shipping
7:05:18address is a null, go get the value from
7:05:21the billing address. So, this time we're
7:05:23going to go to the replacement, right?
7:05:25So we will get the value C in the output
7:05:29because the shipping address is the
7:05:31null. Now let's move to the third row.
7:05:32As you can see here we have again null.
7:05:34So SQL going to go and get the value
7:05:37from the billing address. But here in
7:05:39this scenario the billing address is as
7:05:41well null. That's why we will get the
7:05:44value null in the output. So as you can
7:05:46see having the replacements values from
7:05:48a column there is no guarantee that
7:05:51there will be always a value like here
7:05:53in the third order it is a null that's
7:05:56why we will get null as well in the
7:05:58output. So if you think you are using is
7:06:00null to replace all the nulls by having
7:06:03two columns you might end up as well
7:06:05having a null in the output if the
7:06:07replacement having nulls. So if you want
7:06:09to make sure you don't get any nulls in
7:06:12the output you have to go and use a
7:06:14static value. So this is how SQL execute
7:06:17the
7:06:21isnull. All right. So what is coales?
7:06:24Coal is going to go and return the first
7:06:26null value from a list. All right. So
7:06:29now the syntax of the coales is way
7:06:31better than the is null. Here it accepts
7:06:34like a list of many values. So here for
7:06:36example we have value 1 2 3 you can add
7:06:39four five as much as you want. So we are
7:06:42creating here a list of values to be
7:06:44checked. So for example, we still can
7:06:46use it like the isnull where we have the
7:06:48shipping address where we replace the
7:06:50null with a static value the unknown or
7:06:53as we learned we can go and use two
7:06:54columns shipping address and the billing
7:06:56address. So so far it's like the same
7:06:58use cases as the is null but now of
7:07:00course the kalis is not only limited to
7:07:02two we can go and use three. So we are
7:07:05saying go check the shipping address if
7:07:07it's null then go check the billing
7:07:09address. If it is as well null then use
7:07:12at the end the default value the static
7:07:14one the unknown. So as you can see we
7:07:16can use more than two values with the
7:07:18coalis. Okay. So now let's understand
7:07:19the cowless and how this works. Now the
7:07:22workflow is something similar to the
7:07:24isnull. So in this example we have two
7:07:26columns shipping address and the billing
7:07:28address. It's going to consider it as a
7:07:30list and it's going to start checking
7:07:32from left to right. So it's going to
7:07:34check the first value from the shipping
7:07:35address whether it's null. If no, it's
7:07:38not null then we're going to go and get
7:07:40the value one. So we will get the value
7:07:42from the shipping address. And if yes,
7:07:44it is null then it's going to go and get
7:07:47the value two. So we're going to get the
7:07:49value from the shipping address. Now we
7:07:51have the similar data. We have three
7:07:52orders. Let's see how going to execute
7:07:54it. So it's going to start with the
7:07:55first row and it's going to focus on the
7:07:57shipping address. So here the value is
7:07:59not null. So we have it as an A. So
7:08:02that's why we will get the value one. So
7:08:05we will get the value from the shipping
7:08:07address and nothing else going to be
7:08:09checked. Now moving on to the second
7:08:11row. This time the shipping address is
7:08:13null. So it's going to go and get the
7:08:16value from the second column and it's
7:08:18going to be the C. Right? So in the
7:08:21output we will get C. Now to the last
7:08:23example, we have it as a null and it's
7:08:25going to go and get the value from the
7:08:27second column and this time we're going
7:08:29to get as well a null like the is null
7:08:32function. So at the results we are
7:08:34getting exactly the same result as
7:08:36isnull. So for this scenario it doesn't
7:08:38matter whether you use isnull or
7:08:40kowalis. So now of course we are still
7:08:41not happy with that because I don't want
7:08:43to see any nulls in the output and I
7:08:46will still need to use the billing
7:08:47address instead of any static values. So
7:08:50I would like to have everything the
7:08:51values from the billing address and as
7:08:53well I would like to have at the end a
7:08:55default value so that I don't have any
7:08:58nulls in the output. So how we going to
7:08:59solve it? So now we can use the power of
7:09:01the account list where we can include
7:09:03multiple values in one function. So what
7:09:06we're going to do we're going to have
7:09:07the shipping address first then the
7:09:09billing address and at the end we're
7:09:11going to have the default value. So we
7:09:13have now a list of three values and of
7:09:15course our workflow going to be a little
7:09:17bit bigger. So again here it's going to
7:09:19start from the left to the right. So
7:09:21first it's going to go and check the
7:09:22value one. If it is null then it's going
7:09:24to go as well checking the value two.
7:09:27And if the value two is as well null, we
7:09:29will get the last value. It's going to
7:09:31be the value three. So now let's run the
7:09:33example again using the new kalis. So
7:09:35the first thing we're going to go and
7:09:37check the first value which is the
7:09:38shipping address for the record number
7:09:40one. So now as you can see the value is
7:09:42not null. So we have here an a. So what
7:09:45going to happen? We're going to get the
7:09:46value a as well in the output. So that
7:09:49means this one going to be activated and
7:09:51we will not check anything else. So that
7:09:54means in the output it's going to be
7:09:56like this. and the first value is
7:09:58returned and everything else will be
7:10:00ignored. So, SQL will not check
7:10:02anything. So, as you can see, we are
7:10:04returning the first null value. So, now
7:10:08let's move to the second order. Now,
7:10:09we're going to check again the first
7:10:10value. Is it null? Well, yes. As you can
7:10:13see, we have here a null. So, that means
7:10:16we're going to go and activate this path
7:10:18over here on the right side. So, now SQL
7:10:20will not go blindly putting anything
7:10:22from the billing address in the results.
7:10:24First SQL has to check it. So SQL going
7:10:27to check it whether it's null or not.
7:10:29SQL going to go and return it as well in
7:10:31the output. And we have activated this
7:10:34path. So SQL is returning the value two
7:10:37which is the value from the billing
7:10:39address. So now let's move to the third
7:10:40order. SQL first going to go and check
7:10:43the shipping address. Is it null? Well
7:10:46yes it is null. So that's why SQL going
7:10:48to go and start checking the second
7:10:51value. So this time SQL will not return
7:10:54the billing address value since it's
7:10:56null. It's going to go and return the
7:10:58third value. And what is the third
7:11:00value? It is our static value the NA. So
7:11:03in the output we're going to get the NA
7:11:07our default value. So with that as you
7:11:09can see in the output we will not get
7:11:11any nulls. We are using the default
7:11:13value and as well multiple columns. So
7:11:15if you check the output, it's always the
7:11:17first priority to check the values from
7:11:20the first column, the shipping address.
7:11:22If it's null, then the second priority
7:11:24going to be the billing address. If it's
7:11:26null, then the last priority, it's going
7:11:29to be the default value. So as you can
7:11:30see, SQL is checking the values from
7:11:32left to right and it stops immediately
7:11:35once it encounters the first not null
7:11:38value and return it in the results. So
7:11:40this is how the cow works.
7:11:46All right. So now let's have a quick
7:11:47summary about the differences between
7:11:49the kowalis and isnull. So as we learned
7:11:51the isnull is limited only to two values
7:11:54where the kowalis is amazing where you
7:11:56can have a list of multiple values which
7:11:59is a great advantage compared to the
7:12:01isnull. Now if you are talking about the
7:12:02performance the isnull is faster than
7:12:05the kawalis. So if you want to optimize
7:12:07the performance of your query then go
7:12:09with the isnull. Now there is another
7:12:11problem with the isnull is that we have
7:12:14different keywords for different
7:12:15databases. So for Microsoft SQL server
7:12:18we use the isnull as we learned but in
7:12:21Oracle they have different
7:12:22implementations they use the NVL and
7:12:24other database like MySQL you have if
7:12:28null and all those three functions are
7:12:29doing the same but we have different
7:12:31implementations for different databases
7:12:33but in the other hand the cowis it is
7:12:35available in all different databases. So
7:12:37here we have like an agreement or
7:12:39standards between the databases of using
7:12:41the kowalis. So here again this is a
7:12:43great advantage for the kowalis because
7:12:45if you are writing like scripts and
7:12:47someday you want to migrate from one
7:12:49database to another. If you are using
7:12:50the kowalis you don't have to change
7:12:52anything but if you are using the isnull
7:12:54then you have to go and adjust your
7:12:56queries and scripts with the correct
7:12:58functions. That's why I tend always to
7:13:01use the kalis and avoid using the
7:13:03isnull. Only if it's really necessary
7:13:06that I have really bad performance, I go
7:13:08and try the isnull. But I usually stick
7:13:10with the kowalis. So that is my advice
7:13:13for you. Go with the kowalis and stick
7:13:15with the standard.
7:13:19Now the use cases of the kowalis and the
7:13:21isnull are very similar and we mainly
7:13:24use them in order to handle the null
7:13:27before doing any SQL task. For example,
7:13:30we can use them in order to handle the
7:13:32null before doing data aggregations. So
7:13:35let's understand what this means.
7:13:36Imagine that we have three sales. We
7:13:38have 15, 25, and a null. Now if you go
7:13:41and use an aggregate functions like the
7:13:43average, what's going to happen? SQL
7:13:45going to calculate it like this. 15 + 25
7:13:48divided by two and the average is going
7:13:50to be 20. So as you can see here SQL is
7:13:53including only the two values 15 and 25
7:13:56and ignores totally the null value. So
7:13:59in the calculations the null will not be
7:14:01included because if SQL does that the
7:14:04output going to be as well null. So the
7:14:06nulls are totally ignored. Now the same
7:14:08thing can happen with the other
7:14:09aggregate functions like the sum count
7:14:11if you are counting the sales min and
7:14:13max. There is only one exception about
7:14:16the aggregate function count. If you are
7:14:17using it with the star, SQL here is
7:14:20considering not the values. SQL going to
7:14:22consider the rows. That's why SQL going
7:14:24to go and include all those rows and
7:14:26find the output going to be three. Now
7:14:28in some scenarios, if your business
7:14:30understand the null as zero, then you're
7:14:32going to have a problem with the result
7:14:34of your analyzes if you don't handle the
7:14:36nulls. So what we have to do? We have to
7:14:38handle the null before doing the
7:14:39aggregations. So we have to go and
7:14:41replace a null with zero using either
7:14:43the isnar or the kowalis. So once you do
7:14:46that the calculation going to be changed
7:14:48for the average. So it's going to be 15
7:14:50+ 25 + 0 divided by 3 and the output
7:14:54this time going to be 13.3. So with that
7:14:57you're going to get more accurate
7:14:58results for the business if they
7:15:00understand nulls as zero. All right. So
7:15:02now we have the following example. It
7:15:03says find the average scores for the
7:15:05customers. So let's go and solve it. So
7:15:07we're going to go and select the
7:15:09customer ID, the score from table
7:15:14customers. So let's go and execute it.
7:15:16So as you can see, we have four
7:15:18customers with score and the last one
7:15:19doesn't have any score. So we have it as
7:15:21a null. Let's go and calculate the
7:15:23average for the score and I would like
7:15:27to have the window function in order to
7:15:29see the details as well. So this is
7:15:31average scores. So let's go and execute
7:15:35it. Now of course what is going on here?
7:15:37The four values going to be added to
7:15:38each others and divided by four and the
7:15:40null is totally ignored. Now of course
7:15:42the question is what the business
7:15:44understand with the null. If it is zero
7:15:47then we have inaccurate results. So
7:15:50let's go and fix it. Now this time we're
7:15:52going to say okay we're going to have
7:15:53the average but instead of score we're
7:15:55going to handle the nulls first. So we
7:15:57have to replace any nulls with zero. We
7:16:00can go and use the kowalis or the
7:16:02isnull. So I will go with the cabalis
7:16:04like this and score if you find any null
7:16:09make it zero. So that's it and as well I
7:16:12will go with the window function. So
7:16:14average scores let's call it two. Now
7:16:17let's go and execute it. Now as you can
7:16:19see in the output we got 500 and this is
7:16:22different than the previous average and
7:16:24that's because we have replaced the null
7:16:26with zero. Let's just go and display it
7:16:29in order to understand it. So I will
7:16:31copy it and put it here. So let's call
7:16:33it score two and execute it. So now SQL
7:16:36is going to summarize all those values
7:16:38and divided by five and that's why we
7:16:41are getting the 500. So if our business
7:16:44understand the null as a zero this
7:16:46average going to be more accurate after
7:16:49we handle the null. As you can see in
7:16:50some scenarios we have to handle the
7:16:52nulls before doing any data
7:16:54aggregations.
7:16:59All right, moving on to the next use
7:17:00case for the kowalis and isnull. We can
7:17:02use them in order to handle the nulls
7:17:05before doing any mathematical
7:17:07operations. So let's understand what
7:17:09this means using the plus operator. So
7:17:11if you do plus operator between two
7:17:12numbers like 1 + 5, you are summarizing
7:17:15the values and you will get six. And if
7:17:17you do the plus operator between string
7:17:19values like a + b. So now what we are
7:17:22doing, we are doing data concatenations
7:17:25and the output going to be a b. So now
7:17:27if you go and replace the one with a
7:17:29value like zero. So 0 + 5 we will get
7:17:32five. Nothing fancy about that. And for
7:17:34the strings if you go and replace a
7:17:36value with an empty string. So there is
7:17:39zero characters between the two quotes
7:17:41plus the B. So in the output you will
7:17:44get only B. So it's fine and nothing is
7:17:46critical. But now we come to the
7:17:48problem. If you use a null if you
7:17:49replace the one with null in the output
7:17:52you will get a null. because you are
7:17:54saying okay five plus something that I
7:17:56don't know so SQL says okay you are
7:17:58summarizing now a value with a no value
7:18:01it is unknown so I don't as well know
7:18:04what going to be the answer that's why
7:18:06going to say it's going to be null just
7:18:07don't know what is the answer and the
7:18:09same thing can happen with anything else
7:18:11like the string so if you're saying null
7:18:13plus b and here going to say the same
7:18:15thing the null is unknown and the answer
7:18:18going to be as well unknown so my
7:18:20friends this is very critical in the
7:18:22analyzes and working with data. So this
7:18:24means we have to handle the nulls before
7:18:26doing any mathematical operations. And
7:18:28this is not only for the plus operator,
7:18:31it's as well for the other operators
7:18:32like minus and so on. All right. So now
7:18:34let's have the following task. And it
7:18:36says display the full name of the
7:18:38customers in a single field by merging
7:18:41their first and last names and add 10
7:18:43bonus points for each customer's score.
7:18:46So let's go and solve it. We're going to
7:18:48select first the basic informations.
7:18:50Let's get the customer ID. What do we
7:18:52need? the first name, the last name and
7:18:55we need the scores. So that's it from
7:18:58sales customers. Let's go and execute
7:19:01it. Now the first task is that we have
7:19:03to generate a new field called full name
7:19:06where we have to go and merge or
7:19:08concatenate their first and last names.
7:19:10So let's go and do that. We need the
7:19:12first name plus and then let's have a
7:19:15space between the first and last name
7:19:19and then plus let's have the last name
7:19:22as
7:19:23full name. So let's go and execute it.
7:19:27Now if you check the result for the
7:19:29first customer it is working. So we have
7:19:31Joseph Goldenberg. The same thing for
7:19:33the second customer. But for the third
7:19:35customer we have here a problem.
7:19:37Customer doesn't have any last name but
7:19:40she has a first name. So we have here a
7:19:42Mary. So the full name here is
7:19:44completely null which is not correct.
7:19:46For this example we have at least to
7:19:48show the first name Mary even though
7:19:50that the last name is missing. So the
7:19:52result is not really accurate and that's
7:19:54because we are doing the plus operator
7:19:56between a null and marry. So that means
7:19:59we have to go and handle the nulls
7:20:01before doing any plus operator. So again
7:20:04here we can go with the cowless or the
7:20:06isnull. So let's go and create a new
7:20:08field using the cowless. So it's going
7:20:11to be the last name and now we have to
7:20:14define a new value. If it's null so we
7:20:17could have like something unknown or we
7:20:19could have like an empty string and we
7:20:21can do that using two quotes and between
7:20:23them there is nothing. So we are using
7:20:25an empty string. So let's go and check
7:20:28the results. Last name two. So let's go
7:20:33and execute it. Now we can see that the
7:20:35last name over here for marry it has an
7:20:39empty string and it is not anymore a
7:20:41null. So now SQL knows okay this is a
7:20:44string and there is no characters inside
7:20:46it. So with that SQL knows more
7:20:48informations and we can go and now
7:20:50concatenate those informations. So let's
7:20:52go and do that. We're going to take the
7:20:54whole thing and replace the last name
7:20:57with the kowalis. So let me just remove
7:21:00this last name over here and execute it.
7:21:03So now as you can see things looks
7:21:04better. Now we have in the full name for
7:21:06mari only the first name. And of course
7:21:09if you don't like it like this you would
7:21:10like to have another default value. You
7:21:12can go over here and say something like
7:21:14in a not available. So let's go and
7:21:16execute it. And with that you can see
7:21:18immediately uh there is here a missing
7:21:20last name. But it doesn't really look
7:21:22good. So I will just remove it and go
7:21:24with the empty string. We're going to go
7:21:26and execute it. So with that we have
7:21:28solved the first part of the task where
7:21:30we have the full names and we are not
7:21:32missing any informations from the first
7:21:34name and the last name. Now let's go to
7:21:35the second part of the task where we
7:21:37have to add 10 bonus points for each
7:21:39customer score. So we have to go and add
7:21:42a 10 for each score. So let's go and do
7:21:44it. I'm going to put it at the end. So
7:21:46score + 10 and let's give it the name
7:21:51score with bonus. So that's it. Let's go
7:21:54and execute it. So now in the output you
7:21:56can see it's very easy. We have added a
7:21:5810 for each score. So we have increased
7:22:01the score points for each customer. But
7:22:03now for the last customer Anna you can
7:22:05see over here she doesn't have a value
7:22:07in the scores and that's why didn't go
7:22:10and added 10. So we will get as well a
7:22:12null. And of course this might not be
7:22:14fair that the last customer is not
7:22:16getting any point even though that we
7:22:17have increased for all others. So that
7:22:19means we have to go and handle the null
7:22:21by replacing the null to zero. And only
7:22:24after that we're going to add a plus to
7:22:26it. So let's go and do that. I'm going
7:22:28to add a
7:22:30kalis if it is null then go and make it
7:22:34zero. And afterward go and add a 10
7:22:36points. So let's go and execute it. So
7:22:39now as you can see at the results
7:22:40everything now is fair where we have a
7:22:4310 bonus points for each customers even
7:22:45if the customer doesn't have any values
7:22:48in the scores like here Anna she has
7:22:50like null but still she is getting a 10
7:22:52points. So here again as you can see if
7:22:54you don't handle the nulls correctly
7:22:56before doing the mathematical operations
7:22:59you might get unexpected results. So be
7:23:01careful with the nulls and handle them
7:23:03correctly before adding anything.
7:23:09Okay, moving on to the next use case for
7:23:11the kowalis and is null. We can use them
7:23:13in order to handle the null before doing
7:23:16joins. This is little bit advanced use
7:23:18case but it's very important to
7:23:20understand it. So let's understand why
7:23:21this is important. Let's have for
7:23:23example two tables table A and table B.
7:23:26And in some scenarios we have to go and
7:23:28combine those two tables using the
7:23:29joins. And now in order to join two
7:23:31tables, we have to go and specify the
7:23:34keys between the table A and table B in
7:23:36order to join on it. So in this example,
7:23:38we have two keys in order to join the
7:23:40tables. Now here comes the special case.
7:23:43If those keys don't have any nulls
7:23:45inside it and all the data are filled,
7:23:47then your join going to work perfectly
7:23:49and you will get the expected results.
7:23:51And now you might have a special case
7:23:53where there are nulls inside the keys.
7:23:55So there are missing values and this is
7:23:57a big problem because in the output you
7:23:59will get unexpected results and some
7:24:01records will be totally missing. So in
7:24:04this scenario we have to handle the
7:24:05nulls inside the keys before doing the
7:24:08joins. Let's have a very simple example
7:24:10in order to understand this behavior.
7:24:12All right. So now let's have this very
7:24:13simple example where we have two tables
7:24:15and we want to combine them. So in the
7:24:16first table we have a year type orders
7:24:19and in the second table we have as well
7:24:21year type and we have sales. So now we
7:24:23would like to go and combine those two
7:24:25tables in order to have all informations
7:24:27in one result. Now we can go of course
7:24:29and use the inner join between the table
7:24:31one and table two and the keys for the
7:24:33joins here. As you can see we have the
7:24:36year in both of the tables and as well
7:24:38the type. So we're going to go and use
7:24:40both of those columns as a key for the
7:24:42join. So let's do it step by step how
7:24:45going to execute this. So we need the
7:24:46year type and the results. So it's going
7:24:49to go and take those two columns to the
7:24:51results and we need the orders and
7:24:53sales. So it's going to take as well the
7:24:56orders and the sales from the second
7:24:58table. So now let's start doing it row
7:25:00by row. So the first key going to be
7:25:02those two columns. So we have 2024 and
7:25:06the type A. So now it's going to start
7:25:08searching for those two informations in
7:25:10the second table. And as you can see we
7:25:12have here a match, right? So the first
7:25:15row is as well matching since it's inner
7:25:17join it going to present in the output
7:25:19only the matching rows from left and
7:25:22right. So in the outputs we're going to
7:25:24get the whole row from the table one and
7:25:27we will get the sales from the table
7:25:29two. All right. So that's all for the
7:25:31first row. Now let's move to the second
7:25:34row over here. So what are the values of
7:25:36the keys? We have 20 24 and null. So now
7:25:40if you check the matches on the right
7:25:42side you can see we have a match here
7:25:44right it is logical so it's as well 20
7:25:4724 and null so everything is matching
7:25:50and we should get it in the result right
7:25:52SQL cannot go and use the equal operator
7:25:55in order to join tables so even though
7:25:58that is logically it makes sense to have
7:26:00it at the output but still SQL cannot go
7:26:02and compare the nulls that's why this is
7:26:05a problem for this combination SQL will
7:26:08not find any matching So we will not get
7:26:10any informations for the combination of
7:26:122024 and null. So for us of course in
7:26:15the business this is missing
7:26:17informations and as well inaccurate
7:26:19results. So we're going to miss this row
7:26:22and it's still going to go and jump to
7:26:23the third row. So here what are the
7:26:25values of the key. We have 20 25 and B.
7:26:28Now it's going to go and search it in
7:26:30the second table and it's still going to
7:26:32find a match over here. So in the
7:26:34outputs we're going to get those values.
7:26:37The the orders going to be 50, the sales
7:26:40300. Now it's going to go to the last
7:26:42row and we have here again the same
7:26:44problem. We have here 2025 and null. And
7:26:48of course if you check the data you will
7:26:49say yes we have a matching over here but
7:26:52SQL would ignore it. So we have exactly
7:26:54the same situation and we will not find
7:26:57it at the results. So at the output we
7:26:59will get only two rows even though that
7:27:01those two tables are like identicals if
7:27:04you compare the keys. So with that we
7:27:06are losing data at the results and we
7:27:08are providing inaccurate results. So my
7:27:10friends if you have nulls inside your
7:27:13keys what can happen you will lose
7:27:15records at the output. So here it's very
7:27:18important to handle the nulls inside the
7:27:21keys before doing the joins. All right
7:27:24so now in order to fix it we're going to
7:27:25go and use either the kalis or the
7:27:27isnull in the join. So as you can see we
7:27:29are not using the type directly. We are
7:27:31handling it by replacing the null with
7:27:34an empty string. It doesn't matter which
7:27:36value you are using. The main thing is
7:27:38that you have a value and SQL can go and
7:27:41map it. So you could have it as empty
7:27:43string or a blank or any default value.
7:27:45But I usually go with the empty string
7:27:47since it's little bit faster than having
7:27:49any other characters. So now what going
7:27:51to happen is we're going to go
7:27:52everywhere and replace those nulls with
7:27:56an empty string. So now we don't have
7:27:58any nulls inside our keys and let's go
7:28:01and see what can happen. So we're going
7:28:02to start with the first row again. Here
7:28:04we have a matching from the right table
7:28:07and we're going to see the whole records
7:28:10in the outputs. So we will get as well
7:28:12the sales as 100. And now it's going to
7:28:14go to the second row over here. So this
7:28:17time we don't have a null. We have 2024
7:28:20and an empty string. So now it's going
7:28:22to go and search for a match and it's
7:28:24going to find it over here. we have as
7:28:25well 2024 and an empty string. So now
7:28:28what can happen in the outputs we're
7:28:30going to get a
7:28:32204 but here we will get a null. So we
7:28:35will not get an empty string we will get
7:28:37a null over here and that's because we
7:28:40are handling the null only on the join.
7:28:42So as you can see we have here the is
7:28:44null type on the join but we don't have
7:28:46it on the select. So in the select the
7:28:49type going to be like the original data
7:28:51and the original data was a null. We are
7:28:53just handling the null in the joints
7:28:55just in order to let SQL understand how
7:28:57to map and match the data. So in this
7:29:00example, I'm not changing the values in
7:29:02the select. So that's why we will get
7:29:03the original value. But the orders we
7:29:06will get it 40 and the sales going to be
7:29:0920. Now moving on to the third row. I
7:29:12think you already get it. So let's going
7:29:14to find the match and the sales going to
7:29:16be 300. All right. Now we're going to
7:29:19move to the last one. And here we have
7:29:20the same scenario. So we have 2025 and
7:29:23an empty string. So it's not null
7:29:25anymore. And SQL going to go and search
7:29:27for all those informations and it's
7:29:29going to find it over here. So SQL going
7:29:31to take this fields over here in the
7:29:34type in null not an empty string because
7:29:37in the select we didn't handle it. So
7:29:39the order going to be 60 and the sales
7:29:42going to be 200. So as you can see now
7:29:45the result is complete. We successfully
7:29:48combined both of those tables in one big
7:29:50results using joins but as well using
7:29:53the help of the isnull function in order
7:29:55to have a complete results and not miss
7:29:57any value. So my friends be very careful
7:30:00check always the keys whether they have
7:30:03nulls or not and if you find nulls go
7:30:05immediately and handle it so you don't
7:30:07lose any records in the results and you
7:30:10get accurate
7:30:14analyzes. All right, moving on to the
7:30:16next use case for the isnull. We can use
7:30:18it in order to handle the nulls before
7:30:22sorting the data. So imagine we have the
7:30:24following sales 15 25 and null. Now if
7:30:27you go and sort the data by the sales
7:30:29ascending from the lowest to the highest
7:30:32what can happen? SQL going to show the
7:30:34nulls at the start and that is not
7:30:37because the null is the lowest value
7:30:39because null has no value. But SQL show
7:30:42it like this. it's going to place it at
7:30:43the start and then below it we're going
7:30:46to have the lowest value. So it is the
7:30:4815 and at the end we're going to have
7:30:49the 25. Now if you are doing the exact
7:30:52opposite where we are sorting the data
7:30:54from the highest to the lowest using
7:30:56descending. So what going to happen is
7:30:59going to sort it like this. We're going
7:31:00to have 25 then 15 and the last thing
7:31:03that going to appear in the list going
7:31:05to be the null. So here SQL is showing
7:31:07the nulls at the end and that is again
7:31:10not because nulls are the lowest value
7:31:12it has no value but SQL do it like this
7:31:15show it at the end. So this is how SQL
7:31:17deals with the nulls if you are sorting
7:31:19the data. So in order to understand this
7:31:21use case let's have the following task.
7:31:23So the task says sort the customers from
7:31:26the lowest to the highest scores with
7:31:29nulls appearing last. All right. So
7:31:31let's solve it. This going to be very
7:31:33interesting one. So we need the customer
7:31:35informations. So let's go and select and
7:31:37we need the customer ID and the scores
7:31:41from sales customers and let's go and
7:31:45execute it. So we have a simple list of
7:31:47all customers and their scores. But now
7:31:49we have to go and sort the data from the
7:31:51lowest to the highest. So we're going to
7:31:53go and use the order by clause and we
7:31:55need the field score. And since it's
7:31:57lowest to the highest that means we need
7:31:59to have the ascending and in SQL it is a
7:32:02default. So we don't have to go and
7:32:03mention it. So let's go and execute it.
7:32:05So now as you can see in the results it
7:32:07start from the lowest to the highest and
7:32:09the first part of our task is solved.
7:32:12But now of course we have an issue right
7:32:13because we have a null and as we learned
7:32:15SQL going to put it at the first place
7:32:18on the list. But the task says with
7:32:20nulls appearing last. So we really don't
7:32:23want to see the nulls at the start. We
7:32:25don't worry about it. So we would like
7:32:26to have it at the end of the list. So
7:32:28that means we have to go and handle the
7:32:30nulls before sorting the data. And here
7:32:33we have two ways to do it. One way that
7:32:36is lazy and the other one is more
7:32:38professional. So let me show you first
7:32:39the lazy way. We're going to go and
7:32:41replace the null with a very big number.
7:32:44So for example, what we're going to do,
7:32:46we're going to go and use the kowalis
7:32:48and we're going to say okay score and
7:32:50then let's have a lot of number so that
7:32:53we have a really big score. I just want
7:32:55to select it in order to see the
7:32:56results. So as you can see it's a very
7:32:58big number here. So if you take this and
7:33:02replace the order by with the new score.
7:33:05So that's it. Let's go and execute it.
7:33:07So now if you check the results we have
7:33:09already solved the task. We have listed
7:33:10all the customers from the highest to
7:33:12the lowest and the nulls are at the end.
7:33:14So now the question why do we call this
7:33:16lazy or not professional and that's
7:33:18because we are defining a static value.
7:33:21And of course for this example it is
7:33:22working but we don't know later what's
7:33:24going to happen. Maybe things change
7:33:26where in this course you're going to get
7:33:27a higher value than this and then
7:33:29sorting the data will make no sense
7:33:31since the null going to be like in
7:33:33between values. So who knows your value
7:33:36might be a real value inside the data.
7:33:38Now let me show you the other way which
7:33:40is more professional in order to solve
7:33:42this task where we don't play with luck
7:33:44at all. So let's go and do that. Let me
7:33:46just move this little bit here. I'm
7:33:48going to go and create a new logic where
7:33:50we're going to say case when if the
7:33:52score is null then what's going to
7:33:54happen we want the value one otherwise
7:33:58the value going to be zero so end so we
7:34:01are just creating a flag with zero and
7:34:04one if the score is null then we're
7:34:06going to get the flag of one if we have
7:34:08a value for the score we will get zero
7:34:10so let's have it like this and I will
7:34:12just go and get rid of this kalis so
7:34:16let's go and execute it Now if you check
7:34:18our new nice flag you can see we have
7:34:20zeros everywhere where we have a value
7:34:22in the score but only once we have a
7:34:25null we will get the flag of one. So now
7:34:27once we got this what we're going to do
7:34:29we're going to go and sort our data
7:34:30based on this flag and the score even
7:34:33though the task is not mentioning
7:34:34anything about the flag but we are using
7:34:36it in order to force the nulls to be at
7:34:39the end of the result. Let me show you
7:34:41how we're going to do that. So let me
7:34:42just remove all this. So first we want
7:34:45to sort the data by our new flag in
7:34:48order to make sure that the nulls at the
7:34:50end. So we're going to have our flag and
7:34:52then afterward we sort the data by the
7:34:55score. So let's go and have the score.
7:34:58So again what we are doing first sort
7:35:00the data by the flag in order to push
7:35:02the nulls at the end. And now once all
7:35:04those values are equal to each others
7:35:07what's going to happen SQL going to go
7:35:08and sort the data by the score. So SQL
7:35:10going to use the scores in order to sort
7:35:12the data and both of them are ascending.
7:35:15Let's go and execute it. Now as you can
7:35:17see we're going to get exactly same
7:35:18results. The values from the lowest to
7:35:21the highest and the nulls are at the
7:35:23end. And as you can see with the order
7:35:25by we didn't use any static values or
7:35:28any big numbers. And of course we don't
7:35:30need the flag at the select. So we can
7:35:31go and remove it. So let's execute it.
7:35:33And with that we have solved the task.
7:35:35So as you can see we can use those nice
7:35:37functions like the cowis or the isnull
7:35:40in order to handle the nulls before
7:35:42sorting your
7:35:46data. So what is the function null if
7:35:49null if going to go and compare two
7:35:51values and it going to returns a null if
7:35:54they are equal otherwise if they are not
7:35:56equal it going to returns the first
7:35:59value. Okay. Okay. So now the syntax of
7:36:01the null if it accepts only two values
7:36:04value one and value two. So here again
7:36:06of course you can go and use a column
7:36:08with a static value like the unknown. So
7:36:10we are comparing the values between a
7:36:12column and a static value or you can go
7:36:15and compare two columns the shipping
7:36:17address and the billing address. So
7:36:19again here it accepts only two values.
7:36:21We cannot have it like the kalis where
7:36:23we have a list of multiple values. All
7:36:25right. So now let's understand exactly
7:36:27what do we mean with the null if. So the
7:36:29workflow going to be like this. SQL
7:36:30going to go and check two values the
7:36:32value one and the value two. And if they
7:36:35are equal then SQL going to go and
7:36:37return a null. But if the two values are
7:36:39not equal going to go and return the
7:36:42first value. So it is the one on the
7:36:44left side. So by checking the outcomes
7:36:46here we will never have a scenario where
7:36:48we're going to get the second value.
7:36:50That means the second value always used
7:36:52as a check. So we are checking against
7:36:55this value. So either we're going to get
7:36:56the value one or a null. Let's have this
7:36:59very simple example. We are saying null
7:37:00if price and we are checking whether
7:37:03it's equal to minus1. So we are saying
7:37:05if the price is equal to minus1 then go
7:37:08and replace it with a null because it is
7:37:10data quality issue that we have a price
7:37:12that is negative. It makes no sense for
7:37:14our business. And if it is minus1 then
7:37:16it means for us a null. We don't know
7:37:18the price of this product. So we will
7:37:20correct it using the null if. Let's
7:37:22check this very simple example. We have
7:37:24two orders. So SQL going to start with
7:37:25the first order and check the first
7:37:27value. So what is the first value? Is
7:37:29the price. So here we have a 90. SQL
7:37:32going to go and check is 90 equal to
7:37:34minus one. Well, no. That means it's
7:37:36going to go and execute this path. So
7:37:39that means in the output we will get the
7:37:42first value which is 90. So in the
7:37:44output we will get a 90. Now let's move
7:37:47to the second order. Here we have a
7:37:49minus one. So SQL going to check is
7:37:51minus one here equal to the minus one
7:37:53that we have in the null if well yes. So
7:37:56that means SQL going to go and execute
7:37:58this path where we were going to get the
7:38:00null value in the output and we're going
7:38:02to get it like this. So now if you
7:38:04compare the result from null if and the
7:38:05price you can see we don't have any more
7:38:08the minus one. And as you can see now we
7:38:10are doing exactly the opposite as
7:38:11kowalis and is null. We are replacing a
7:38:13real value with a null. Now moving on to
7:38:16the second example and this is very
7:38:18interesting one in the analytics where
7:38:20we can go and use two columns inside the
7:38:22null if. So in this example we are
7:38:24saying null if original price and
7:38:27discount price. So SQL have to go and
7:38:29compare the prices between those two
7:38:30columns and if they are equal it should
7:38:33return a null. And now you might say
7:38:34okay in this example why we are doing
7:38:36this? Well we can use it in order to
7:38:39highlight or flag special cases inside
7:38:41our data. And the special case here is
7:38:44if the original price is equal to the
7:38:47discount price and if those two prices
7:38:49are equals that means we have an issue
7:38:51in our program or something like went
7:38:53wrong as we are inserting data. So let's
7:38:55see what's going to happen for the first
7:38:57row we're going to go and compare the
7:38:59150 from the original price with the
7:39:01discount price. So they are not equal
7:39:04right. So that means going to go and
7:39:06return the original price the 150 in the
7:39:09output. So let's move to the second
7:39:11order. Here we have the original price
7:39:13250 and as well the discount price is
7:39:16250. So they are equal and if they are
7:39:18equal then we will get a null in the
7:39:21output. So as you can see again here we
7:39:23are not getting any values from the
7:39:25discount. We are using it only for a
7:39:26check. So with that we have a quick flag
7:39:28like using the nulls as flag in order to
7:39:31identify where we have equal values. So
7:39:34this is how the null if works.
7:39:39All right friends, here we have a very
7:39:41nice use case for the null if and that
7:39:43is preventing the error of dividing by
7:39:46zero. Let's see what this means. Okay,
7:39:48let's have the following task and it
7:39:50says find the sales price for each order
7:39:53by dividing the sales by quantity. So
7:39:56let's go and solve it. This should be
7:39:57very easy. So we need the order ID. We
7:40:02need the sales and the
7:40:05quantity from sales orders. Let's go and
7:40:10execute it. So now we have 10 orders.
7:40:12Those are the sales and the quantity. So
7:40:14now it's very easy to calculate the
7:40:16price. It's going to be the sales
7:40:18divided by quantity and we're going to
7:40:21call it price. So let's go and execute
7:40:24it. Now as you can see we got an error
7:40:26says divide by zero error encountered.
7:40:29So that means somewhere we have a zero
7:40:31for the quantity and this is a problem.
7:40:34Let's go and check the data again. So
7:40:35I'm just going to comment the whole
7:40:38thing and let's go and execute it. So
7:40:40now by checking the result yes we got
7:40:42for the order ID 10 here we have
7:40:44quantity zero. So it will not work if
7:40:46you divide by zero of course. So how we
7:40:48can solve it? We can use the magic of
7:40:50the null if where we're going to go and
7:40:52replace the zero with a null. So getting
7:40:55a null is way better than getting an
7:40:58error. Right? So let's go and do that.
7:41:00I'm just going to remove the comments.
7:41:01And here we're going to say null if if
7:41:04the quantity equal to the zero value. So
7:41:08that's it. Let's go and execute it. Now
7:41:10as you can see it is working. And with
7:41:12that we are making sure that we are not
7:41:15dividing by zero. And that's because we
7:41:17replace it with a null. And if you
7:41:19divide anything by null you will get a
7:41:21null. So if you check the result over
7:41:23here the order 10 we got the price of
7:41:25null which is correct and for the all
7:41:28other values everything is working
7:41:29because we have values and we didn't
7:41:31replace it with a null that's why we
7:41:33have values for the price and this is
7:41:36very common use case for the null if we
7:41:38can use it in order to prevent dividing
7:41:40by
7:41:44zero. All right so what is is null? It's
7:41:47going to return true if the value is
7:41:50null. So it is checking the value if
7:41:52it's null it's going to return true
7:41:54otherwise it's going to returns a false.
7:41:57Now the exact opposite if you go use the
7:41:59is not null. So if you use these
7:42:01keywords it's going to returns a true if
7:42:04the value is not null otherwise if it is
7:42:06null it's going to go and return a
7:42:08false. Okay. So the syntax for that is
7:42:10very simple. It start with a value or
7:42:13expression and then after that we're
7:42:15going to have the keyword is space null
7:42:17and the is not is exactly the same. So
7:42:20we have a value then afterwards we have
7:42:22the is not null. So we have the not
7:42:25operator after that and the is not is
7:42:27exactly the same. So we have a value
7:42:29then we have the is not the not operator
7:42:32then the null. So it's very simple.
7:42:34Let's have an example. We are checking
7:42:35whether the values of the shipping
7:42:37address is null. So we can have it like
7:42:39this. Shipping address is null or we can
7:42:42check the opposite whether it's not
7:42:43null. So the shipping address is not
7:42:45null. It's very easy. Okay. So now let's
7:42:47understand how this works. we are
7:42:49checking the value. So if the value is
7:42:52null then return a true if it is not
7:42:55null then we return a false. So as you
7:42:57can see it never returns the value
7:42:59itself or any nulls. So we are getting a
7:43:01boolean of true and false. So we are
7:43:04creating like a boolean flag in order to
7:43:06assist us with the checks. So we have
7:43:08this very simple example price is null
7:43:10and we have those two rows. So we are
7:43:12checking whether the price is null in
7:43:14the first order it is not null right
7:43:17that's why we will get a false in the
7:43:19output and the second order the value is
7:43:22null so it is correct that's why we will
7:43:25get true now of course if we go and use
7:43:27the is not null is going to be exact
7:43:29opposite so is the price not null well
7:43:33yes it's not null that's why you will
7:43:35get a true over here so now for the
7:43:37second check it is null right so the
7:43:40output going to be false we will get the
7:43:43exact opposite. So that's it. It's very
7:43:45simple how the isnull and is not null
7:43:50works. All right. One very obvious use
7:43:53case for is null and is not null is by
7:43:56searching for missing informations or
7:43:58searching for nulls. And maybe after
7:43:59that we can go and clean up our data by
7:44:02removing the nulls from our data set.
7:44:04Let's have the following task and it
7:44:06says identify the customers who has no
7:44:09scores. All right, let's go and solve
7:44:11it. This is very simple. So let's start
7:44:13by selecting star from sales customers.
7:44:17So we need everything. Let's go and
7:44:19execute it. Now as you can see we have
7:44:21our five customers. But the task says we
7:44:23have to have all the customers who have
7:44:25no score. So that means the result
7:44:27should return only the last record since
7:44:30the score of Anna is null. So let's go
7:44:33and have a wear clause. So where and now
7:44:36what do we need? We need the score. Then
7:44:38we don't use the equal, we use is null
7:44:42like this. So that's it. Let's go and
7:44:44execute it. And with that, as you can
7:44:45see, it's very simple. We have filtered
7:44:47our data and now we can see all the
7:44:49customers where the score is null. This
7:44:52is a very basic check to understand
7:44:54whether our data contains nulls. All
7:44:56right, moving on to the next task and it
7:44:58says show a list of all customers who
7:45:01have scores. So back to our example,
7:45:03this time we're going to do exactly the
7:45:05opposite. We want a list of all
7:45:06customers where we have a value in the
7:45:09scores. So what we're going to do, we're
7:45:10going to say where score is not null. So
7:45:14if you go and execute it, you can see
7:45:16we're going to get a clean list where
7:45:18all the customers have score. And with
7:45:20that, we get rid of all nulls inside the
7:45:22score field. And maybe this is helpful
7:45:24in order to do further analyzes.
7:45:30All right friends, now we come to very
7:45:32interesting use case for the isnull and
7:45:34that is by introducing a new type of
7:45:37joints between tables that's going to
7:45:39help us to find the unmatching rows
7:45:42between two tables. Let's have a quick
7:45:44recap about the joints in SQL in order
7:45:46to understand the new types. So
7:45:48basically we have two sets or let's say
7:45:50two tables the left and the right. And
7:45:53if you go and use an inner join what we
7:45:55are doing here we are finding only the
7:45:57matching rows between the left table and
7:45:59the right table. So at the result we
7:46:01will get only the matching rows. Now we
7:46:04have another type of joints called lift
7:46:06outer join. And if you use this type at
7:46:09the result you will get all the rows
7:46:11from the left table and as well only the
7:46:14matching rows from the right table. Now
7:46:16we have another type which is exactly
7:46:18the opposite the right join. And here
7:46:21we're going to get all the rows from the
7:46:22right table and only the matching
7:46:25informations from the left table. And
7:46:27now to the last type that we learned. We
7:46:29have the full join where we will get all
7:46:31the rows from the left and as well all
7:46:33the rows from the right. So we will not
7:46:35be missing anything. So those are the
7:46:37four basic joints that we have learned
7:46:39in SQL. But in SQL we have as well other
7:46:42types that are more advanced. But we
7:46:44don't have in SQL any keywords for that.
7:46:47So the first one called lift anti-join.
7:46:51So what we are saying here we need all
7:46:53the rows from the left table but this
7:46:55time without the matching rows. So all
7:46:59the informations that are matching with
7:47:01the right table we don't want to see it
7:47:03at the results. And as I said we don't
7:47:05have here an extra keyword for this type
7:47:07of join. But in order to get this effect
7:47:09we're going to go and combine the left
7:47:11join together with the isnull. And with
7:47:13that we're going to get all the data
7:47:15from the left side but without anything
7:47:17that is matching the right side. And
7:47:19this we call it left anti- join. And we
7:47:22have another advanced type for the
7:47:24joints called the right anti- join. This
7:47:26is exactly the opposite. So we are
7:47:28saying all the rows from the right table
7:47:31without having any matching rows from
7:47:34the left table. So all the informations
7:47:36on the right side that is not matching
7:47:38the left side. So again here we don't
7:47:39have a keyword for that. We're going to
7:47:41go and work with the right join plus and
7:47:44is null. So with that, as you can see,
7:47:46we have two new types of joins added to
7:47:48our four basic joins. Now this might be
7:47:50confusing. Let's have the following task
7:47:52in order to understand it. Show a list
7:47:54of all details for customers who have
7:47:57not placed any orders. All right. So
7:47:59let's see how we can create the effect
7:48:01of the left anti-join. So let's do it
7:48:03step by step. We need here two tables.
7:48:06We need the customers and as well the
7:48:08orders. So since we are focusing on the
7:48:10customers, the lift table going to be
7:48:12the customers. So let's go and do that.
7:48:15We're going to go and say select star
7:48:18from sales customers. This is our first
7:48:22table. So we are using the alias of C.
7:48:25So let's go and execute it. Now as you
7:48:27can see we got the list of all
7:48:28customers. So that we have all the
7:48:30details for our customers. But now we
7:48:32have to go and join it with the orders.
7:48:34So in order to do that let's have a new
7:48:37line. left join
7:48:40sales sales orders and let's have the
7:48:44lso and now we have to go and define the
7:48:46key for the join so on it's going to be
7:48:49the customer ID equal the customer ID in
7:48:53the order table so now if you go and
7:48:55execute it now what we're going to do
7:48:57we're going to go and show the order ID
7:49:00from the table orders so order ID just
7:49:03to see whether we have a match or not so
7:49:05let's have it like this and execute it
7:49:07Now let's go and check the results. As
7:49:09you can see those four columns comes
7:49:12from the table customers and only the
7:49:14last column come from the orders. So now
7:49:16what is interesting is to check the
7:49:18order ID whether we have nulls or not.
7:49:21So as you can see for the customer one
7:49:23we have everything matching. For the
7:49:25customer two as well we have orders the
7:49:27three as well for only the last one
7:49:30customer ID 5 we have here a null. So
7:49:33that means SQL was not able to find any
7:49:37order for this customer. So again what
7:49:39this means we have only one customer
7:49:41Anna where she doesn't have any order
7:49:45but all other customers they did have an
7:49:48order and that's because we have values
7:49:51from the right table. So once we have
7:49:53values that means we have matching but
7:49:55since here we have a null that means we
7:49:58don't have any matching. So now since
7:50:00the left anti- joint says we would like
7:50:02to have all the data from the left table
7:50:05without having any matching from the
7:50:07right table. So that means for this
7:50:09example we would like to get only this
7:50:11customer Anna. And this is exactly as
7:50:14well fulfilling our task. The task says
7:50:16list all details for customers who have
7:50:19not placed any order. All data from
7:50:21customers where we don't have matching
7:50:24from the orders. Now I think you already
7:50:26got it how to get this effect. We're
7:50:28going to go and filter the data like the
7:50:30following. So we're going to have the
7:50:32wear clause and now we need the column
7:50:34from the right table from the orders. So
7:50:36we're going to go with the customer ID
7:50:39comes from the orders. So we're going to
7:50:41say oh customer ID is null. And of
7:50:47course you can go with the order ID as
7:50:48well. You're going to get the same
7:50:49effect. But I would like always to use
7:50:51the key that we are using with the join.
7:50:54So let's go and execute it. And now as
7:50:56you can see we got the effect of the
7:50:58left anti join and with that as you can
7:51:00see we got the customer that we are
7:51:01aiming for. So here we have the data
7:51:03from the left side that is not matching
7:51:05the right side. So the customers who
7:51:07have not placed an order and with that
7:51:10we have solved the task. So as you can
7:51:11see we have implemented the left and
7:51:13join by combining the left join together
7:51:18with the is null. So this is the power
7:51:20of playing with the nulls in SQL.
7:51:26Now my friends, there is something that
7:51:28is really confuses a lot of developers
7:51:30or anyone that is working with data in
7:51:33databases and SQL and that is the
7:51:36differences between nulls, empty string
7:51:39and blank spaces. So the nulls as we
7:51:41learned we are saying I don't know what
7:51:44the value is it is unknown. But now in
7:51:46the other hand the empty string you are
7:51:48saying I know the value it is nothing.
7:51:52So the empty string is a string value
7:51:55which has a zero characters. This is
7:51:57totally different than the nulls. The
7:51:59nulls we don't know anything about it.
7:52:01So now sometimes maybe happens to you as
7:52:03you are filling a forum and you come to
7:52:05one field you go and by mistake hit a
7:52:08space bar and with that you are entering
7:52:10space into the field and you just jump
7:52:12to the next field without entering any
7:52:14other values. So we have now like a
7:52:17space character inside the field. This
7:52:19is really evil in databases because once
7:52:21the user enter a blank space, it's going
7:52:24to go and store it as a value inside the
7:52:26database and it's going to take storage.
7:52:28So it could be one space or many spaces
7:52:30depends on how long you press the space
7:52:33bar. So the blank space is a string but
7:52:36the size is not zero like the empty
7:52:38string. We're going to have a size of
7:52:40how many spaces you have entered. So
7:52:42here it's not like the null. We know the
7:52:44value it is string and the character of
7:52:46that going to be space. Okay. So let's
7:52:48see those three scenarios inside scale.
7:52:50Now I have like a dummy data using the
7:52:52city statements. Don't worry about it.
7:52:54I'm going to teach you all those stuff
7:52:55in the next tutorials. So now we have
7:52:57here like four rows. The first one with
7:53:00a value a. The next one with null. The
7:53:02third one with empty string. So as you
7:53:05can see there is nothing between those
7:53:07two quotes. And the last one we have a
7:53:09space between those two quotes. Now
7:53:11let's go and query this temporal table.
7:53:14So select star from orders and execute.
7:53:19So now by looking to the values of the
7:53:21categories you can find all the
7:53:23scenarios now. So now the first scenario
7:53:25is the easiest one where we have a
7:53:27normal value. We have here an a. But the
7:53:29other three rows we don't have normal
7:53:31values. We have like empty stuff. So the
7:53:34first one going to be the null. So we
7:53:36don't have a value. This is the special
7:53:38marker from SQL. It says null. So there
7:53:41is no value. And the other two they are
7:53:43really confusing. As you can see it's
7:53:45really hard by just looking to the data
7:53:47or to the results whether it is an empty
7:53:50string or a blank space. And this confus
7:53:53a lot of developers or anyone working
7:53:54with data seeing those results. It's
7:53:56really hard to detect the data quality
7:53:58issues by just looking at the results.
7:54:01So now in this scenario what I do I go
7:54:03and calculate the length of each value
7:54:06inside my column. So let's go and do
7:54:08that. Now we're going to go in the SQL
7:54:11server. We're going to go and use the
7:54:12function data length and our field going
7:54:15to be the category. So let's call it
7:54:18category length. So let's go execute it.
7:54:22And now let's check the result. The
7:54:23first one since we have only one
7:54:25character, the length of that is going
7:54:26to be one which is correct. And now to
7:54:28the next row we have the category null.
7:54:30We don't know the value and as well we
7:54:33don't know the length of the value,
7:54:34right? So that's why we will get a null.
7:54:36So now by moving to the next one as you
7:54:38can see those two looking really exactly
7:54:40the same. But now with the help of the
7:54:43length or the data length function we
7:54:46can see that the third row or the third
7:54:48category value has the length of zero.
7:54:51That means it is an empty string and we
7:54:54don't have any characters over here that
7:54:56is hidden. So with that we are sure this
7:54:58is an empty string. But now let's move
7:55:00to the last one. Here it is very tricky
7:55:02and evil. we have a hidden space inside
7:55:05this value and we can understand that by
7:55:08the length of this field. So as you can
7:55:10see we have here a one that means we
7:55:12have here one hidden space inside this
7:55:15value and it is not empty string. So
7:55:18that means I have here only one space
7:55:20let's go and give it another space and
7:55:23calculate the length. So as you can see
7:55:24we have two spaces and that's why the
7:55:26length is two. So don't count on your
7:55:29eyes in order to understand the spaces.
7:55:31go and calculate the length in order to
7:55:34be very precise. So now let's go and
7:55:36compare the three scenarios side by
7:55:37side. So let's start with the first one
7:55:39about the representations in the table.
7:55:41The null we're going to see it as a null
7:55:43inside the table. The empty string going
7:55:45to be like two quotes and nothing
7:55:47between them. And the blank space it's
7:55:49as well two quotes and between them one
7:55:52or many spaces. And now if you are
7:55:54talking about the meaning the null means
7:55:56unknown. We don't know the value. The
7:55:59empty string it is known but it is
7:56:02nothing it is empty value. And the third
7:56:04one blank spaces it is as well known and
7:56:07the spaces are the value. And now if you
7:56:09are talking about the data types since
7:56:11the null is no value. So we don't have a
7:56:14data type for this and it is like a
7:56:16special marker in the SQL. And now the
7:56:19empty string has a data type. It is a
7:56:21string and the size of this string going
7:56:24to be zero since we have zero characters
7:56:26inside the empty string. Moving on to
7:56:28the blank spaces, it is a string since a
7:56:30space is a character and it's going to
7:56:32be the size of one or many. And now if
7:56:35we are talking about the storage, the
7:56:36null is the best. They don't consume or
7:56:39occupy a lot of storage. While the empty
7:56:41string and the blank spaces, they occupy
7:56:43here storage and memory and they waste
7:56:46the space. So if you are worried about
7:56:48the storage, the best option here is a
7:56:50null. Now talking about the performance,
7:56:52you will get the best performance if you
7:56:55are using nulls. Now the empty string is
7:56:57as well fast but it is not that fast
7:56:59like the nulls. Now the worst option
7:57:01here is the blank spaces it is slow. So
7:57:04again if the speed is important for you
7:57:06you have to have those scenarios as a
7:57:08null. So now if you are talking about
7:57:09the comparison and you are searching for
7:57:11those values if you want to search for
7:57:13the null you have to go and use is null.
7:57:16But in the other hand if you want to
7:57:18search for the empty string and the
7:57:19blank spaces you have to go and use the
7:57:21operator equal. So that's all those are
7:57:24the main differences between the null
7:57:26empty string and blank spaces.
7:57:31Now you might ask you know what why do I
7:57:34have to understand the differences
7:57:35between all those stuff the nulls empty
7:57:37strings and the blanks everything's like
7:57:39empty so why do I care well in new
7:57:42projects I'm going to promise you that
7:57:44you will be working with sources and
7:57:46data that has bad data quality and you
7:57:48might encounter all those three
7:57:50scenarios in your data and now if you
7:57:52don't do any data preparations like
7:57:54cleaning up the data handling those
7:57:56three scenarios and bringing standards
7:57:58to your data and you jump immediately to
7:58:01the analyzes without doing all those
7:58:02stuff, you will end up providing
7:58:04inaccurate results in your reports and
7:58:07analyzes which leads to wrong decisions.
7:58:10So preparing your data before doing any
7:58:12analyszis by cleaning up the data,
7:58:14handling those three scenarios and as
7:58:16well bringing standards is very
7:58:18important step before doing any
7:58:21analyszis. So this is how we're going to
7:58:23do it together with the stakeholders and
7:58:25the users of your reports and analyzes.
7:58:28You have to define a clear data
7:58:30policies. It's like rules and you have
7:58:33to commit yourself during the
7:58:34implementations by following those
7:58:36rules. And here we have three different
7:58:38options. The first one you can go and
7:58:40define the data policies like this. Only
7:58:42use nulls and empty string but avoid
7:58:45using blank spaces. In my project I
7:58:48cannot imagine that there is a scenario
7:58:51where we need blank spaces. They are
7:58:53just evil. Just go get rid of them. All
7:58:55right. Right. So with this policy, we
7:58:57have to go and get rid of all blank
7:58:59spaces inside our data. And in order to
7:59:01do that, we have a wonderful function in
7:59:02SQL called trim. The trim function in
7:59:05SQL going to go and remove the spaces
7:59:07from a string from the left side and as
7:59:10well from the right side. So all the
7:59:12leading spaces and the trailing spaces
7:59:14going to be removed. So now if we go and
7:59:16apply the trim function on that
7:59:17category, what's going to happen? All
7:59:19the blank spaces going to be removed and
7:59:21it going to be turned into empty string.
7:59:24So let's go and do that. It's very
7:59:26simple. So we're going to use the trim
7:59:28function and we're going to apply it on
7:59:30the
7:59:31category. Let's go and call it policy
7:59:34one. So let's go and execute it. So now
7:59:37by just comparing the policy one with
7:59:39the category. You see like it's
7:59:41identical but it's not. Now in order to
7:59:43have a better feeling about this we can
7:59:45go and test it using the data length.
7:59:47Now let's go again and use the data
7:59:49length function. So we're going to use
7:59:52it for the whole results and as well I'm
7:59:54going to go and use it for the category
7:59:56in order to just compare it. So without
7:59:59the
8:00:03trim so like this. Let's go and execute
8:00:06it. Now if you go and check the result
8:00:08as you can see here again we have the
8:00:09length of two because here we have two
8:00:11spaces but with the policy one we have
8:00:14zero. So those two values after applying
8:00:17the trim function they have the length
8:00:20of zero and with that we don't have
8:00:22blank spaces. So that means now we are
8:00:24sure after applying the trim we have
8:00:27either a null or empty string. So let me
8:00:29just get rid of all those informations.
8:00:32Now I am sure both of them are empty
8:00:34string. So as you can see it's very
8:00:36simple using only one SQL function you
8:00:39are cleaning up the data and bringing
8:00:41standards. All right moving on to the
8:00:43option two. You can define your data
8:00:45policies like this. Only use nulls and
8:00:47avoid both empty strings and as well
8:00:50blank spaces. So that means in our
8:00:52business we don't have anything
8:00:54meaningful for the empty string and the
8:00:56blank spaces. We can go and use only the
8:00:58nulls. Okay. So now let's go and
8:01:00implement this rule. We have to go and
8:01:01convert a value to a null. So the value
8:01:05going to be empty string to a null. And
8:01:07as we learned we can go and use the null
8:01:09if function in order to get nulls
8:01:11instead of values. So let's go and apply
8:01:13this policy. But now here we have two
8:01:15values the empty string and spaces. Now
8:01:18instead of having two rules for that I'm
8:01:21going to convert first the blank spaces
8:01:23to an empty string like we have done
8:01:26here. So I'm going to take the result of
8:01:28this function first as a first step and
8:01:31afterwards we're going to go and use the
8:01:33null if. So we're going to say null if
8:01:37for the result of the trim if if you
8:01:40find any empty strings convert it to
8:01:43null. So that's it policy 2. So as you
8:01:46can see in the result we have converted
8:01:48those empty spaces and planks to a null.
8:01:51So with that we are getting three nulls
8:01:53and of course we're going to get the
8:01:54value a. And now if you compare those
8:01:56three columns side by side you're going
8:01:57to see the bully C2 is really easier to
8:02:00understand compared to the previous
8:02:02ones. Right? So now if you compare the
8:02:04policy two now to the policy one, you
8:02:05can see it's easier to understand and
8:02:08it's easier as well to handle. So again
8:02:10it's very easy to do data cleanup with
8:02:12only two functions we have now like
8:02:14standards inside our data. And now
8:02:16moving on to the last option, we can
8:02:18define our data policy like this. Use
8:02:20only a default value unknown and avoid
8:02:24using anything else like nulls, empty
8:02:26strings and blank spaces. So that means
8:02:29in the analyzes and reports we want to
8:02:31see the value unknown and we have to
8:02:33handle all those three informations and
8:02:35convert them to unknown. So now in order
8:02:38to implement the policy three we have to
8:02:39go and convert a null with a value a
8:02:42default value and here we have two
8:02:44options either use the is null or we can
8:02:46go and use the kalis and I will go with
8:02:49the kowalis so
8:02:51kowalis and I'm going to use directly
8:02:54the category. So if you find any null
8:02:57replace it with the default value
8:02:59unknown and let's call it policy 3. So
8:03:02let's go and execute it. So now if you
8:03:04check the result over here you see that
8:03:06we got it only once correct. So we
8:03:08replaced the null with the unknown but
8:03:10we still have like empty spaces and
8:03:12blanks and that's because we rushed
8:03:14using the qualis and we skipped the
8:03:16other steps. So as you can see preparing
8:03:18the data you have to do it slowly step
8:03:20by step. So first we have to go and
8:03:22convert everything to a null like the
8:03:24policy 2. And after that the last step
8:03:27we're going to go and use the default
8:03:28value. So that means instead of using
8:03:30the category we have to go and get the
8:03:32result of the policy 2. So let's go and
8:03:35copy it and replace the category with
8:03:37those two steps and let's go and execute
8:03:40it. So now as you can see we have the
8:03:42default value for all those three
8:03:44scenarios. First we have to trim the
8:03:46data in order to remove all the blank
8:03:48spaces. The second step, we're going to
8:03:50go and replace all the empty strings
8:03:53with a null. And with that, we're going
8:03:55to get a null for all those three
8:03:57scenarios. And finally, we're going to
8:03:59go and replace the nulls with a default
8:04:02value, the unknown. So, that's it for
8:04:04the three policies. And this is the
8:04:06different ways in order to clean up the
8:04:08data and bring standards before doing
8:04:10analyszis. And now you might ask me,
8:04:12okay, which one should I use in my
8:04:13project? Like if I want to suggest
8:04:15something for the users, which one
8:04:17should I use? Well, it really depends on
8:04:19the business, but I tried always to
8:04:21avoid this one, the policy one, because
8:04:23it's always confusing and I have always
8:04:25explained for the users. So now we are
8:04:27left with the two and three. Well, I use
8:04:29both of them in different scenarios. I
8:04:31normally go with the policy 2 because it
8:04:34takes less storage and as well the
8:04:36performance of your queries afterward
8:04:38going to be really good. So if I'm doing
8:04:40data preparations in my ETL before
8:04:42inserting it inside a table, I go with
8:04:45the policy too. But in other hand, if
8:04:47I'm doing a preparation step before
8:04:50showing it in a report like in Tableau
8:04:52or PowerBI. So if it is like one of the
8:04:54last steps before showing the data to
8:04:57the users, I go with the policy 3
8:04:59because if you present a null inside a
8:05:01report, it's going to be really hard to
8:05:03read. So having like a word like
8:05:05unknown, it's easier to understand.
8:05:07Okay, we have here missing data. So
8:05:09again if the data preparations is
8:05:11exactly before I present the data to the
8:05:13users I go with the policy 3 where I use
8:05:16default values but if I'm using a data
8:05:19preparations before inserting it in the
8:05:21database I go with the policy 2 because
8:05:24it's going to optimize the storage and
8:05:26it's going to be really bad if you go
8:05:28with the policy 3 because it's really
8:05:30bad to store the whole world each time
8:05:32there is no value like the unknown. it's
8:05:34gonna take a lot of space and as well
8:05:36you're going to get bad performance as
8:05:37you are building the queries. That's why
8:05:39I tend to store the data using nulls. If
8:05:42you present it to the users go and show
8:05:44it as a default value. So as you can see
8:05:46it's very important to understand the
8:05:47differences between the nulls empty
8:05:49strings and blanks and how to prepare
8:05:51the data by cleaning up the data and
8:05:53bringing standards and policies before
8:05:55doing any analyszis. So with this we
8:05:57have cleared up the confusion between
8:05:59those scenarios and if you encounter it
8:06:01in your projects you know how to deal
8:06:03with
8:06:06it. All right. So now let's have quick
8:06:09summary about the nulls. Nulls are
8:06:11special markers in SQL in order to say
8:06:14there is no value. It is missing. It is
8:06:16unknown. So nulls are not equal to zero
8:06:19or empty string or any blank spaces. And
8:06:22using nulls inside our database is going
8:06:24to save some storage and as well provide
8:06:27a strong performance in your queries.
8:06:29And in scale we have different functions
8:06:30in order to handle the nulls. So now if
8:06:32you want to replace a null with the
8:06:34value we can go either with the function
8:06:36kowalis or is null or if you want to do
8:06:39the opposite where you want to replace a
8:06:41value with null you can go use the
8:06:43function null if or in other cases we
8:06:45want only to check whether there is
8:06:47nulls or not we can use the is null or
8:06:50is not null. And we have learned as well
8:06:52that we have to treat the nulls
8:06:54especially before doing any tasks. So
8:06:56that means we have to handle the nulls
8:06:58before doing for example data
8:07:00aggregations like average, sum, max, min
8:07:03and so on. And we have to handle the
8:07:05nulls as well before doing any
8:07:06mathematical operations like using the
8:07:08plus operator to concatenate two
8:07:11strings. And in some scenarios as we
8:07:13learned we have to handle the nulls as
8:07:14well before doing joins. And in other
8:07:17cases we have as well to handle the
8:07:18nulls before sorting the data. And we
8:07:21have learned as well by combining the
8:07:23joins and the isnull we introduce new
8:07:26types of joins like as we learned the
8:07:28left anti- join and the right anti-join
8:07:31where we exclude the matching rows using
8:07:33the isnull and we can use the null
8:07:35functions in order to provide standards
8:07:38and data policies in our data like using
8:07:41the nulls or using a default values like
8:07:43the unknown. All right my friends. So
8:07:45with that you have learned how to handle
8:07:46the nulls inside your data and now we're
8:07:48going to move to a very special topic
8:07:50called the case statements. This is very
8:07:52important tool in order to do data
8:07:54transformations. So let's
8:07:59go case statements. It can allow you to
8:08:02build a conditional logic in your SQL
8:08:04query by evaluating a list of conditions
8:08:07one by one and return a value when the
8:08:10first condition is met. So now let's
8:08:12understand the syntax of the case
8:08:13statements and what this
8:08:18means. Okay. So now let's see the syntax
8:08:20step by step. It start with the keyword
8:08:23case. This case indicates now we are
8:08:25starting a logic a conditional logic in
8:08:27SQL. It's like programming languages as
8:08:29you start with the if else. So the if is
8:08:32the keyword of a logic and the whole
8:08:34logic as well ends with another keyword
8:08:36called end. So once SQL sees the end. So
8:08:39this is the end of the conditional
8:08:41logic. So the case is the start and the
8:08:43end is the end. So now what we're going
8:08:45to have in between is the conditional
8:08:47logics right. So the conditional logic
8:08:49start with the keyword when. Now we are
8:08:51telling SQL we have a condition to be
8:08:53evaluated and then we're going to go and
8:08:55specify that conditional logic. So now
8:08:58we have to tell SQL what can happen if
8:09:00this condition is fulfilled. So now we
8:09:03have to use another keyword called then.
8:09:05So now we are telling SQL show this
8:09:07results if the condition is true. So as
8:09:10you can see it's very simple. It's like
8:09:11the natural language, right? It's like
8:09:13in English when the condition one is met
8:09:16then show the results. It's very logic,
8:09:18right? And now of course we can go and
8:09:19add a second condition inside our case
8:09:22statements. So we're going to have the
8:09:23same setup. When condition two if this
8:09:26is true then show the result number two.
8:09:28We specify the keyword when then we have
8:09:30a second condition. And if this
8:09:32condition is true then we tell SQL to
8:09:35show another results. And of course it's
8:09:37very important to understand in the
8:09:38syntax of that SQL going to go and
8:09:40process the conditions from the top to
8:09:42the bottom. So the first most important
8:09:45condition should be at the start. So SQL
8:09:47going to first check this condition. If
8:09:50it fails and it's not true then it going
8:09:52to go and jump to the second condition.
8:09:54So the order of the conditions is very
8:09:56important in your logic. And now of
8:09:58course we can go and add multiple
8:09:59conditions depend on the logic using the
8:10:02keyword when. And now once we are done
8:10:04defining all the conditions we can go
8:10:06and specify an else keyword. The else
8:10:09can introduce the default value and it
8:10:11is optional. You can go and skip it. So
8:10:13the value of the else or the default
8:10:15going to be used only if all the
8:10:18condition failed. So that means all our
8:10:20conditions are not true and nothing is
8:10:22fulfilled then SQL going to go and use
8:10:24the value from the else. So it is the
8:10:27default value that's going to be used if
8:10:29all conditions are false. So those are
8:10:31the keywords that you must use inside
8:10:33each case statement. So we have case
8:10:35when then and only the else is an
8:10:39optional. So you can go and use it or
8:10:41skip it. So this is the main structure
8:10:43and the syntax of each case
8:10:48statement. Now let's have a very simple
8:10:50example in order to understand how SQL
8:10:53execute the case statements behind the
8:10:55scenes. All right, let's have this very
8:10:56simple example where we have only one
8:10:58condition. So as you can see in the
8:11:00syntax, it starts with case and end and
8:11:03then we have only one condition and we
8:11:05are evaluating here the sales. So the
8:11:07condition says if the sales is higher
8:11:10than 50 then show as a result the value
8:11:13of high. So it's very simple only one
8:11:15condition and on the right side we have
8:11:16here a flowchart in order to understand
8:11:18how the logic is executed. And now what
8:11:20we're going to do, we're going to go and
8:11:21evaluate those four sales through this
8:11:24logic and see what the output going to
8:11:26be with the case statement. So let's do
8:11:28it one by one. Let's start with the
8:11:30first sales. It is 60. So here we're
8:11:32going to go and check is 60 higher than
8:11:3550. Well, yes. That means this sales is
8:11:38meeting this condition and we will get
8:11:41true and we're going to get in the
8:11:42output the value of high. So here we're
8:11:45going to get the value high in the
8:11:47output. So that means the first sales is
8:11:50fulfilling the requirement the condition
8:11:52and SQL going to give us the value from
8:11:54this condition. All right. So now SQL
8:11:55going to go to the next value and we're
8:11:57going to start evaluating the 30. Now
8:11:59we're going to ask the same question the
8:12:00same condition is 30 higher than 50.
8:12:03Well no. So that means in the output for
8:12:06this condition we will get false. So we
8:12:09will take the path of the false. Now if
8:12:11you take the path of the false we will
8:12:13not get any value. Right? So that means
8:12:15the output going to be a null. So the
8:12:18output for the 30 is null. And that's
8:12:21because we didn't define in our logic
8:12:23anything about the default option. So we
8:12:25don't have here an else. And this is
8:12:27what going to happen. If you don't use
8:12:28else, you will get a null in the output
8:12:31for the case statement. So now let's
8:12:32move to the next one. It's going to be
8:12:34the same thing. So 15 is smaller than
8:12:3750. So it's not fulfilling the
8:12:39condition. And as well we're going to
8:12:41get a null. And for the last one since
8:12:43it's null we will get as well a null
8:12:45since it will not fulfill the condition.
8:12:47So now after evaluating all those sales
8:12:49only the first sales is fulfilling that
8:12:52condition and that's why we have only
8:12:54one value the high. All right. So now
8:12:55let's keep moving and adding stuff to
8:12:57our case statements. Now we are adding a
8:12:59second condition. So it says after
8:13:02checking the sales whether it's higher
8:13:03than 50 and it fails check again the
8:13:06sales whether it's higher than 20. If
8:13:08yes then show the value of medium. So
8:13:11now in our workflow we are adding a
8:13:13second condition to be checked if the
8:13:16first one is false. So now let's go and
8:13:18evaluate our sales again and check the
8:13:20output the first one the 60. So as you
8:13:22can see the 60 is higher than 50. So we
8:13:25are fulfilling the first requirement
8:13:27that's why we will get the value of
8:13:29high. So it's same like before. So here
8:13:31we're going to get high in the output.
8:13:33And now here very important to
8:13:35understand one thing is that SQL didn't
8:13:37evaluate here in this scenario the
8:13:40second condition. So SQL didn't waste
8:13:42any time by checking the other
8:13:43condition. It skipped everything once it
8:13:45get a true from one condition. So this
8:13:48is exactly how SQL process the case win.
8:13:50It going to check each conditions from
8:13:52top to down and once it finds a true
8:13:56it's going to stop everything
8:13:57immediately and throw the value from
8:13:58this condition and it will not evaluate
8:14:01any other conditions. So now it's going
8:14:03to go and jump to the next value. We are
8:14:05at the value of 30. So let's evaluate
8:14:07the conditions. Is 30 higher than 50?
8:14:10Well, it's not. So it's false. So now
8:14:12what can happen is going to go and jump
8:14:14to the next condition and start
8:14:15evaluating the second one whether it's
8:14:17true or false. So now we're going to
8:14:19check here. Is 30 higher than 20? Well,
8:14:21yes. So it's going to be fulfilled and
8:14:24we will get the value of medium. So it's
8:14:27going to stop everything and show in the
8:14:28output for this value the medium. So
8:14:32we're going to get medium here. So in
8:14:34this scenario, we have evaluated both of
8:14:36the conditions that we have in the case
8:14:38statement. Now it's going to go to the
8:14:40third one. We have 15. Is 15 higher than
8:14:4250? Will no. So we will get to false for
8:14:45the first condition. Then it's going to
8:14:47go and jump to the second condition and
8:14:48check it. Is 15 higher than 20? Will as
8:14:52well no. So now what going to happen?
8:14:54The false going to be activated over
8:14:55here. And we will not get any value as a
8:14:58return. So we will get the value of null
8:15:01in the output. And now for the last one
8:15:03we have null. We will get as well null
8:15:05because it will not fulfill any of those
8:15:08conditions and that's because we didn't
8:15:10define an else in the case statement. So
8:15:12if we define these conditions like this,
8:15:14we will get the category medium for the
8:15:1730. And this is how SQL evaluate
8:15:19multiple conditions in the case
8:15:21statement. All right. Now we're going to
8:15:22go to the final form of our case
8:15:24statements and we're going to go and add
8:15:26an else. So we're going to have a
8:15:28default value. So we are seeing here if
8:15:29the sales is not higher than 50 or
8:15:31higher than 20 then show a default value
8:15:34as low. So that means any sales that is
8:15:37equal or smaller than 20 going to get
8:15:39the value of low. And now very
8:15:41interesting if you check the workflow
8:15:42over here you can see that we have now a
8:15:45value for each path. So for the first
8:15:47condition we're going to get high for
8:15:48the second one medium. And if nothing is
8:15:51fulfilled we're going to get always the
8:15:53value of low. So there is no way in this
8:15:55chart to get any nulls. Right? So let's
8:15:58go and evaluate again our values. I
8:16:00think you already get it. The 60 is
8:16:02fulfilling the first requirement and SQL
8:16:04going to stop everything immediately and
8:16:06just show the value of high. So on the
8:16:09right side over here nothing going to be
8:16:10evaluated because the first condition is
8:16:13true. So here in the output we're going
8:16:15to get the value of high. So nothing
8:16:17changed like the two previous examples.
8:16:19Now it's going to go to the next value.
8:16:21We have the 30. So we're going to
8:16:23evaluate the first one. It's going to be
8:16:24false. The next one it's higher than 20.
8:16:28It is true. And that's why is still
8:16:30going to show the value of medium. And
8:16:32this is as well. We had it in the
8:16:34previous example. So medium. So now
8:16:37scale going to move to the next one. And
8:16:38here things going to get interesting. So
8:16:40the value of 15. We're going to evaluate
8:16:42the first condition. Is it higher than
8:16:4450? Well, no. Is it higher than 20?
8:16:46Well, no. So now we are in scenario
8:16:48where none of those conditions are true.
8:16:51So that's why SQL going to go and
8:16:53execute the else. So if you check our
8:16:56chart it's going to be false and we will
8:16:58get the value of low. So in the output
8:17:01we will not get this time a null because
8:17:04we have else we will get the value of
8:17:06low. The same thing now for the null.
8:17:08Null will not fulfill the first
8:17:10condition as well the second condition
8:17:12and that's why we will get as well the
8:17:15value from the else. So here in the
8:17:17output we will get as well the value of
8:17:19low. So now as you can see if you use an
8:17:21else inside the case statements you will
8:17:24make sure that there will be no nulls in
8:17:26the output. So that you have learned the
8:17:28different options that we have inside
8:17:30the case statements and how skill
8:17:32execute the case behind the
8:17:37scenes. All right friends so now we come
8:17:40to the part where I'm going to show you
8:17:41the most useful use cases of the case
8:17:44statements that I usually use in my
8:17:46projects. So let's start. The main
8:17:48purpose of the case statement is to do
8:17:50data transformations. And data
8:17:52transformations is very important
8:17:53process in each data project. And one
8:17:56very important task in data
8:17:57transformations is that we can generate
8:17:59new informations. We can go and create
8:18:01new columns based on the existing data
8:18:04that we have in the database using the
8:18:06case statements and this of course can
8:18:08help us deriving new informations for
8:18:10our analyzes without modifying the
8:18:12source database only for analytics. So
8:18:15my friends, the main purpose of the case
8:18:17statement is to do data transformations
8:18:20by creating and generating new columns.
8:18:22So now let's start with the first use
8:18:24case and the most important and famous
8:18:26one is we use case statement in order to
8:18:29categorize the data. This means we are
8:18:31going to group up the data into
8:18:33different categories based on certain
8:18:35conditions. And now you might ask why
8:18:36this use case is important. Well,
8:18:39classifying and grouping data is
8:18:41fundamental in data analysis and
8:18:43reporting because it makes the data
8:18:45easier to understand and as well to
8:18:47track. But what's more important, it
8:18:49going to help us aggregating the data
8:18:51based on the categories. All right. So
8:18:53now let's have the following task. And
8:18:54it says generate a report showing total
8:18:58sales for each of the following
8:18:59categories. category high if the sales
8:19:01is over 50. Category medium if the sales
8:19:04is between 20 and 50 and low if the
8:19:07sales is 20 or less and sort the
8:19:10categories from the highest sales to the
8:19:12lowest. Okay, so let's do it step by
8:19:14step. And now before we do any data
8:19:15aggregations, we have to go and create a
8:19:17new column called categories because we
8:19:19don't have it in the database. So now
8:19:21let's start with very simple select
8:19:23statements. So select what do we need?
8:19:25Let's take the order ID, the sales and
8:19:28that's it for now. So from sales orders
8:19:32let's go and execute it. And now we have
8:19:34our 10 orders and we have to go and now
8:19:36create a new column called categories.
8:19:39And we're going to do that using the
8:19:40case statements. So let's take a new
8:19:43line and we start with case and then
8:19:45again a new line in order to define the
8:19:47first condition using the when. So the
8:19:49first condition is the high where sales
8:19:52is over 50. So it's very simple. So when
8:19:55the sales is higher than 50, what can
8:19:58happen if this is true? We want to show
8:20:00the value high. So this is the first
8:20:03condition. And then let's move to the
8:20:05second one. If the sales is higher than
8:20:0920, that means it's less than 50 and
8:20:12higher than 20, then we want to see the
8:20:15value medium. And now for the last
8:20:17category, the low, we don't have to go
8:20:19and create a condition for that because
8:20:21if those two fails, then that means that
8:20:23the sales either equal to 20 or less. So
8:20:25what we're going to do, we're going to
8:20:26just do a simple else and show the value
8:20:29low like this. Let me make this a little
8:20:32bit smaller. Now what is missing in our
8:20:35case is of course the end. Without it,
8:20:37you're going to get an error. So end and
8:20:39let's give it a name
8:20:40category. So we are ready. Let's go and
8:20:43execute it. So now let's check randomly
8:20:45stuff. So as you can see here we have
8:20:46the sales of 50 it is low which is
8:20:49correct and then we have here 60 it's
8:20:51above 50 and we have the category high
8:20:53and now if you check the order number
8:20:54six we have the order 50 it's medium
8:20:57because it is not higher than 50 it is
8:21:00between 50 and 20. So now as you can see
8:21:02we have now classified our orders using
8:21:05the category. Now the next step that
8:21:07we're going to go and aggregate the
8:21:08data. So how we going to do that? We
8:21:10will use a subquery. So let's do it like
8:21:13this. So we're going to go and select
8:21:14and of course we're going to group up
8:21:15the data by the category. So we're going
8:21:17to go and select that category and we
8:21:20need the total sales. That means we're
8:21:22going to go and use the function sum for
8:21:24the sales and we're going to call it
8:21:26total sales. So now we have to nest the
8:21:29queries together. So from this is our
8:21:33query like this and then we have to
8:21:36close it and group by. So we are
8:21:39grouping by the category. Okay. So with
8:21:42that we are now aggregating the sales by
8:21:45that category. It's very simple. Let's
8:21:47go and execute it. So now in the result
8:21:49we have only three categories. We don't
8:21:50have the 10 orders because now we are
8:21:52doing data aggregations. So now the
8:21:54granularity now on the level of
8:21:56category. So now we can see the total
8:21:58sales for the high is 2010. The low we
8:22:01have 65 and the medium we have 105. And
8:22:04of course we are not done yet because in
8:22:05the task it says sort the categories
8:22:08from the highest sales to the lowest.
8:22:10That means we have to go and use an
8:22:12order by statement at the end and we're
8:22:14going to sort the data by the sales from
8:22:16the highest to the lowest. That means
8:22:18descending. So that's it. Let's go and
8:22:20execute it. And now with that we have
8:22:22our reports. Now we are showing the
8:22:24total sales by the categories and the
8:22:25data is sorted from the highest to the
8:22:27lowest. So the highest category is high
8:22:30then medium and then the last one is
8:22:32low. So my friends as you can see with
8:22:34the help of the case win we have created
8:22:36the new informations from our data we
8:22:38have the category and then we have
8:22:39created insights or report based on this
8:22:42new informations where we have
8:22:44aggregated our data using this new
8:22:46information. So the use case of
8:22:48categorizing data using case statements
8:22:49is fundamental and very important in
8:22:52each data
8:22:56project. Okay. Okay. So now one more
8:22:58thing before we jump to the next use
8:22:59case is that there is one rule to follow
8:23:01if you are using case statements and
8:23:03that is the data types of the result
8:23:06must be matching. So what this means if
8:23:08we check again our example over here we
8:23:10can see that the result of each
8:23:12condition is a string. So as you can see
8:23:14we have here high, medium and low and
8:23:16all of those informations are following
8:23:18the same data type. So it is correct. So
8:23:20now if I go and break this rule for
8:23:22example after this then let's have the
8:23:25value two. So now we have a number and
8:23:27we have characters. So let's go and
8:23:29execute it. And now of course we're
8:23:30going to get an error because now SSQL
8:23:32is trying to convert the value low to an
8:23:35integer which is incorrect. So the data
8:23:37types of the output of the result must
8:23:40be matching and that's not only include
8:23:42the value after the then but also the
8:23:45value after the else because this value
8:23:48is as well part of the output. So let's
8:23:50have here again medium. And now let's go
8:23:52and change this to let's say one. So
8:23:55let's go and excuse it again. Isl going
8:23:57to throw an error because this is an
8:23:59integer in number and the others are
8:24:01string characters. So this is the rule
8:24:03of using the case statement. The data
8:24:05types after then and after else must be
8:24:07matching. And if you ask me whether
8:24:09there is restriction about where you can
8:24:11use the case statement in which clauses
8:24:13you can use it everywhere in select, in
8:24:15joins, from, where, group by, order by,
8:24:19everywhere. So there are no restrictions
8:24:21and we have only this one rule.
8:24:27Okay friends, another use case for the
8:24:28case statement. We can use it in order
8:24:30to map values. So we can use the case
8:24:33statement in order to transform the data
8:24:35from one form to another in order to
8:24:37make it more readable and more usable
8:24:39for analytics. One scenario of mapping
8:24:41values is that sometimes the database
8:24:43developers stores the data and values
8:24:46inside the database as codes and as
8:24:48flags. So for example, the status of the
8:24:50order could be stored as one and zero
8:24:53instead of having inactive and active.
8:24:56And this is one technique in order to
8:24:57optimize the performance of the database
8:24:59for the application because one and zero
8:25:01is way faster than storing the whole
8:25:04string. But in data analyzes, we usually
8:25:06generate a report to be read by human by
8:25:08persons. And now instead of showing the
8:25:11data as zero and one, it's going to be
8:25:12more nicer and readable if you show the
8:25:15data as active and inactive. So for
8:25:17these scenarios, we're going to go and
8:25:18use the case statement in order to
8:25:20translate those cryptical and technical
8:25:23values into readable terms. Otherwise,
8:25:26each one going to consume your report.
8:25:27Going to ask you what do you mean with
8:25:29the zero and one. Let's have the
8:25:30following task and it says retrieve
8:25:32employee details with gender displayed
8:25:35as full text. Okay. So now let's go and
8:25:37solve it. First we're going to go and
8:25:38explore few informations. So let's go
8:25:41and show the employee
8:25:42ID and let's take the first name, last
8:25:46name and we need the gender
8:25:48informations. So gender from sales
8:25:52employees. So that's it. Let's go and
8:25:55execute it. So now as you can see in the
8:25:57result we got our five employees and now
8:25:59the gender informations are stored as
8:26:01only one character F and M. And of
8:26:04course it's easy to understand that the
8:26:05F is female and M is male. but we would
8:26:08like to show it in the report as a full
8:26:10text. So, female and male instead of
8:26:13those abbreviations. So, in order to do
8:26:15that, we're going to go and use the case
8:26:17statement in order to do the mapping
8:26:19between the old value and the new value.
8:26:21So, let's go and create a new column
8:26:23using the case. So, we're going to have
8:26:25here two conditions because we have two
8:26:26values. Let's start with the first one.
8:26:28So, we're going to have a new line and
8:26:30when. So when the gender equals to f
8:26:34ladies first then female and now for the
8:26:38second value it's going to be exactly
8:26:39the same when gender equal to m then
8:26:44we're going to have male be careful for
8:26:47the case sensitivity of the values. So
8:26:49of course we will not end this without
8:26:51an else. So else then we can have the
8:26:53default value. We're going to have the
8:26:55default value not available. It's better
8:26:57than having nulls. So what we are
8:27:00missing is the end. So we're going to
8:27:02have an end over here and we're going to
8:27:04call you gender full text. So that's it.
8:27:08Let's go and execute it. Now if you
8:27:10check the results, we have now done the
8:27:12mapping between the old format of the
8:27:14value with the new format. So instead of
8:27:16m we have males and females. And of
8:27:18course we don't have here any nulls.
8:27:20That's why we don't have a not available
8:27:22in the data. But if you have huge data
8:27:24of course you can have somewhere a null
8:27:26and then you will get this default
8:27:28value. So this is how you can do mapping
8:27:30between values very easily using the
8:27:32case statements. Okay let's have another
8:27:34task for the mapping use case and the
8:27:36task says retrieve employee details with
8:27:39abbreviated country code. Sometimes as
8:27:41we are generating reports maybe using
8:27:42PowerBI or Tableau we don't have enough
8:27:45spaces in order to use the full name of
8:27:48values. So what do we need? We need
8:27:50abbreviations. we need short form of the
8:27:52values and we can go and use in SQL the
8:27:55case statement in order to map the full
8:27:57value to an abbreviated value. So it's
8:28:00like the previous example but the way
8:28:02around. All right. So now let's go and
8:28:03solve it. We're going to go and select
8:28:04few details like the customer ID. Let's
8:28:07take the first name, last name and what
8:28:10do we need? We need the country
8:28:12information from sales customers. So
8:28:15that's it. Let's go and execute it. And
8:28:18now as you can see we get our five
8:28:19customers and we have the country
8:28:21informations as a full name. Now of
8:28:23course for the report we need
8:28:24abbreviated values from this. So we're
8:28:26going to go and map those full names of
8:28:28the countries to a short form. But in
8:28:30real project you might get big tables
8:28:32where you have thousands and millions of
8:28:34records. So you cannot just check it
8:28:36like this. So how I usually do it I go
8:28:38and retrieve a distinct list of all
8:28:40values from one column. So I usually go
8:28:42and have a separate query for that. So
8:28:45we're going to have select distinct
8:28:48country from the table sales customers.
8:28:51It's just for me to see all the possible
8:28:53values inside the database. So now you
8:28:55see the second result over here. We have
8:28:56only two values Germany and USA. And
8:28:58then I can go and map the data
8:29:00correctly. So always if you are mapping
8:29:02data using the case win you have to
8:29:04understand all the possible values that
8:29:06you have inside the table. So let's go
8:29:08and generate this new informations.
8:29:10Let's start with case and then you line
8:29:13when country equal to the first value.
8:29:16It's going to be Germany. Make sure you
8:29:18write it exactly like in the database.
8:29:20The first character is capital and the
8:29:22rest is small. So what happened? We're
8:29:25going to have the abbreviation of
8:29:27Germany. It's going to be de. All right.
8:29:29So this is for the first value. And then
8:29:30let's move to the second one. It's going
8:29:32to be country equal to USA. It's already
8:29:36abbreviated but maybe we can get only
8:29:39two characters.
8:29:40So us like this. And now let's go and
8:29:43add an else. It's optional but in case
8:29:45that we have nulls in the data or we get
8:29:47a new value. So else it's not available.
8:29:50So na. So that's it. And never forget
8:29:53about the end. So end. And the name
8:29:56going to be country abbreviation. So
8:29:59that's it. Let me just get rid of the
8:30:01other query. So the mapping is correct.
8:30:04Let's go and execute it. And now if you
8:30:06check the results, we got a new column
8:30:08called country abbreviation. And as you
8:30:10can see now the mapping is working. Here
8:30:12we have Germany and we have here DE and
8:30:14for the USA we have US. So with that we
8:30:17have solved the task and we done the
8:30:18mapping correctly between the old value
8:30:20and the new
8:30:24value. All right friends, now there is
8:30:26special case for the syntax of the case
8:30:28statements if you are using it for
8:30:31mapping values. So now let's go and
8:30:32check it. So now let's say that we have
8:30:34a lot of different distinct values
8:30:35inside the country not only to values
8:30:37you have a lot of values and if you are
8:30:39mapping the values using the case when
8:30:41you're going to end up always writing
8:30:42the same thing country equal Germany
8:30:45country equal India country equal United
8:30:47States and so on. So we are always using
8:30:49the column country. So the conditions
8:30:51over here using always one column and
8:30:54it's always the operator is equal. So
8:30:57now only for this scenario we have
8:30:59another syntax for the case statements
8:31:01and it looks like this. We start with
8:31:03the keyword case but after that
8:31:05immediately we're going to use the
8:31:07column that we want to evaluate and here
8:31:09you can use only one column you cannot
8:31:11use multiple columns. So now we are
8:31:13telling SQL we are now evaluating one
8:31:15column the country and then for each
8:31:18condition we have the following stuff we
8:31:19say when Germany that means when country
8:31:22is equal to Germany then de so as you
8:31:25can see here we don't have here the
8:31:26whole condition we have only a possible
8:31:29value that we can see inside the
8:31:31country. So we are saying is the value
8:31:34country if it's true then show de the
8:31:36next one is it India then en United
8:31:39States US and so on. So we call this
8:31:41syntax a quick form of the case
8:31:43statements and on the left side we call
8:31:45it full form of the case statements and
8:31:47of course the restriction and limitation
8:31:50using the quick format is that you can
8:31:52use only one column and it's only for
8:31:55the equal operator. So that means only
8:31:57for these scenarios you can go and use
8:31:59the quick format. If things get a little
8:32:01bit complicated where you have to mix
8:32:03and make complex logic, you cannot use
8:32:05the quick formats. So I would say if you
8:32:06are sure that the logic will not get
8:32:08complicated and you can stay always with
8:32:10the same column, you can go with the
8:32:12quick format. But I would recommend
8:32:13always to go with the full format
8:32:15because for one simple reason if you add
8:32:17one small logic you have to go and
8:32:19rewrite the whole case statements back
8:32:22to the full format in order to add any
8:32:24small logic. But of course there is
8:32:26nothing wrong using the quick form in
8:32:28order to do the case statements if the
8:32:30logic can stay static and you are sure
8:32:32we are using only one column and we are
8:32:34just doing mapping. There is no any
8:32:36extra logic. Okay. So now let's try this
8:32:37quick format for the case statement for
8:32:39the previous example. So I will just go
8:32:41and copy everything to a new column. So
8:32:44I'm just going to rename it to two. And
8:32:46now how we going to do it? So it's going
8:32:49to be case but this time we're going to
8:32:51write country and then inside the wind
8:32:54we will have only the values. So no need
8:32:57for the condition. So it's going to be
8:32:59like this. Let me scroll up. So that's
8:33:01it. As you can see it's smaller and
8:33:03quicker than writing the whole condition
8:33:05each time. So now let's go and execute
8:33:08this. And as you can see in the result
8:33:09we're going to get identical values. So
8:33:11now you know one more trick in the case
8:33:13statement.
8:33:18All right, moving on to the next use
8:33:19case for the case statements. We can use
8:33:21it in order to handle nulls. Handling
8:33:23nulls means replace a null with a value.
8:33:26And as we learned before with the window
8:33:27aggregate functions, sometimes nulls
8:33:29leads to incorrect calculations and
8:33:32results which leads to wrong decision-m.
8:33:34We're going to have later a dedicated
8:33:36chapter on how to handle nulls in SQL.
8:33:38But now we're going to learn how to
8:33:39handle nulls using case statements. So
8:33:42now let's have the following task and it
8:33:43says find the average scores of
8:33:46customers and treat nulls as zero and
8:33:48additionally provide details such as
8:33:50customer ID and the last name. Okay. So
8:33:52now let's solve it step by step and
8:33:54again we have here details and as well
8:33:56we have to do aggregations that means we
8:33:57have to go and use the window functions
8:33:59and we don't have to forget that we have
8:34:01to treat the null so we have to handle
8:34:02it. So now let's go and start with very
8:34:05simple uh select. So select
8:34:08customer ID we need the last name and as
8:34:12well we need the scores. So from sales
8:34:16customers let's go and execute it. So as
8:34:19usual we have our five customers and the
8:34:21scores. And here we have a null. Now
8:34:22we're going to go and write the window
8:34:24function but without handling the nulls
8:34:26just in order to see the differences. So
8:34:28we need the average function for what
8:34:31for the scores. Do we have to now
8:34:33partition the data? Well no. So we're
8:34:35going to leave it as empty. We need the
8:34:37average score of all customers. So
8:34:39that's it. Let's go and give it a name
8:34:42and then execute it. I think I have here
8:34:44mistake. So it is a score not scores. So
8:34:46and now as you can see we have the
8:34:48average of 625. And as you learned
8:34:50before SQL going to go and summarize all
8:34:53those four values and divide it by four.
8:34:55But our business understand the nulls as
8:34:57zero not as missing information. So we
8:34:59have to go and handle the null. Let's go
8:35:02and create a new column for the scores.
8:35:05But this time we're going to go and use
8:35:07the case statements. It's going to be
8:35:09very simple. So we're going to say when
8:35:12the score is null. So in SQL we don't
8:35:15write equal null, we say is null. So
8:35:18with that we are replacing the nulls
8:35:20with zero. Right? So now otherwise what
8:35:23can happen? So if it's not null so we
8:35:25need the score as it is. We should not
8:35:27manipulate anything. So the default
8:35:30value is the score itself if the score
8:35:32is not null. So now let's go and end it
8:35:35and let's call it score clean. So let's
8:35:39go and execute it. Now if you check the
8:35:41result over here, it's like almost
8:35:43identical as the score. So we don't have
8:35:45any new values for the scores but only
8:35:48the nulls now are zero and all other
8:35:50values they are not affected. So we
8:35:52didn't touch it. We didn't transform it
8:35:54at all. So this is what do we mean with
8:35:56handling nulls replacing nulls with
8:35:58another value. So now in order to finish
8:35:59the task we have to do the average for
8:36:01the score clean and not for the original
8:36:04score. So how we going to do it? Let's
8:36:05go and copy the whole case statements.
8:36:07I'm just going to do it in another
8:36:09column. So let's have an average and
8:36:12inside it we have the case statements
8:36:14like this. Let me just sort it like
8:36:17this. And now what is missing is the
8:36:19over and it's going to be empty. So
8:36:23average customer let's call it clean. So
8:36:26this is the logic. Let me just make
8:36:28everything smaller. So now as you can
8:36:30see it's exactly like the previous one
8:36:32but instead of using the original score
8:36:35now we are using the column that we have
8:36:38created. But of course we don't need the
8:36:39alias over here. So we have to remove
8:36:41it. So it start with case and end. So
8:36:44let's go and execute it. And now you can
8:36:46see in the output we got a new value for
8:36:49the average and it is more accurate for
8:36:51the business. So now we have 500.
8:36:53Previously we had
8:36:54625. So as you can see you have to
8:36:56understand what the nulls means in your
8:36:58business and handle it correctly.
8:37:00Otherwise you will get wrong results. So
8:37:02that's it. We use case statements in
8:37:04order to handle the nulls inside our
8:37:07data.
8:37:10Conditional aggregations means we're
8:37:12going to go and apply an aggregate
8:37:13function in SQL like some average count
8:37:16but this time only on a subset of data
8:37:19that meet specific conditions. This
8:37:22technique is amazing in order to do deep
8:37:24dive analyzes or target analyzes on a
8:37:27specific subset of the data. So now
8:37:29let's have the following SQL task in
8:37:30order to understand this use case. The
8:37:32task says count how many times each
8:37:34customer has made an order with sales
8:37:37greater than 30. All right. So, as
8:37:39usual, we can do it step by step. So,
8:37:41what do we need? We need the orders. So,
8:37:43let's get the order ID and as well,
8:37:45let's get the customer ID like this and
8:37:49the sales from sales orders. Let's go
8:37:53and execute it. So now, what else I'm
8:37:55going to do with I'm going to go and
8:37:57order the data by customer ID. So, let's
8:38:00execute it again. Okay. So, now the task
8:38:02sounds easy, but it's a little bit
8:38:04tricky. We have to count the number of
8:38:06orders for each customer where the sales
8:38:08is higher than 30. Let's have an
8:38:10example. For example, this customer
8:38:12number one. So the total number of
8:38:13orders is three orders, right? But we
8:38:16have to count only the orders where the
8:38:17sales is higher than 30. And in this
8:38:20example, we have only one order where
8:38:22the sales is higher than 30. So it's
8:38:24only the order number four. So the count
8:38:26for the customer ID number one should be
8:38:28one. Now let's check another customer.
8:38:30For example, the two. And as you can
8:38:32see, we have three orders, but none of
8:38:34them have the sales higher than 30. So
8:38:37the count should be zero here. So how we
8:38:39going to do that? We have to go and flag
8:38:41each row whether it's higher than 30 or
8:38:44not. So if it's higher than 30, it gets
8:38:47the flag of one. If it's less than 30 or
8:38:50equal to 30, it's going to get zero. And
8:38:52then we're going to go and summarize all
8:38:53those flags in order to get the count.
8:38:56So let's do it step by step. Let's first
8:38:58create the flag. So we're going to go
8:39:00and use case and then our condition is
8:39:02very easy. We're going to say when. So
8:39:04what is the condition? Sales greater
8:39:06than 30. So sales is higher than 30.
8:39:10Then what can happen? We're going to
8:39:11flag it with the one because later we're
8:39:14going to go and summarize the one. And
8:39:15now else if it's not higher than 30,
8:39:18equal to 30 or less. So it's going to
8:39:20get zero. All right. So now let's go and
8:39:22end it. So let's say sales flag. Now
8:39:26let's go and execute it and check the
8:39:28results. All right. So now if you check
8:39:29the results we got now a very nice flag
8:39:31in order to see which orders has sales
8:39:34higher than 30. So now for example let's
8:39:36take the customer ID number one. As you
8:39:38can see only the order number four has
8:39:40sales higher than 30 and it's flagged
8:39:42with one and all others are zero. Now
8:39:45let's take the customer ID number three.
8:39:47And as you can see we have now two
8:39:49orders where the sales is higher than
8:39:5130. And as you can see we have the one
8:39:53twice. And now we can use this flag in
8:39:55order to do the aggregation. So now if
8:39:57you go and summarize the flag for the
8:39:59customer id number three we will get two
8:40:02and this is the count of orders where
8:40:04the sales is higher than 30 right and
8:40:06let's take another example the customer
8:40:08ID number two we have everywhere zero
8:40:10and if we summarize those values we will
8:40:12get zero which is the count of orders
8:40:14where the sales is higher than 30 which
8:40:15is correct so now as you can see first
8:40:17we have built an extra column in order
8:40:19to help us doing the aggregation and now
8:40:21in the next step we're going to go and
8:40:23aggregate this column so let's go and do
8:40:25that we don't need all those
8:40:26informations the order ID we need the
8:40:29customer ID because it is the
8:40:31granularity for the aggregation and
8:40:33let's remove the order by and now let's
8:40:36go and group up the data by customer ID
8:40:40but of course we need the aggregate
8:40:42function so how we going to do it we're
8:40:44going to go and summarize the whole flag
8:40:46so and now of course we're going to go
8:40:48and rename this since now it is an
8:40:50aggregated column so we're going to call
8:40:52it total orders so now let's go and
8:40:55execute it. So now let's go and check
8:40:57the result. As you can see, now we have
8:40:58our four customers. And for the customer
8:41:00ID number one, we got only one order
8:41:03higher than 30. The second one has no
8:41:05orders higher than 30. The third we have
8:41:08two and one. And with that, we have
8:41:10solved the task. Now I would like to add
8:41:12one more thing to our query in order to
8:41:13see the normal aggregations, not the
8:41:16conditional aggregations. So usually we
8:41:18go and count for example the star in
8:41:22order to get the total orders. And let's
8:41:24rename the previous one to high sales.
8:41:27So let's go and execute it. So we are
8:41:29just now doing aggregations without any
8:41:32conditions. And now we can see how many
8:41:34orders did each customer. So we can see
8:41:36that the customer ID number one did
8:41:38order three times but only one order
8:41:40higher than 30. So this is a normal
8:41:42aggregation and this is a conditional
8:41:44aggregations using the case
8:41:49statements. All right friends. So now
8:41:51let's do a recap about the case
8:41:53statements. Case statement can go and
8:41:54evaluate a list of conditions one by one
8:41:57and return a value once the first
8:41:59condition is met. And if you are talking
8:42:01about the rules of using the case
8:42:02statements, we have only one where the
8:42:04data types of each condition after the
8:42:07then and else must be matching. And now
8:42:09if we talk about the use cases of the
8:42:11case statements, the main use case is to
8:42:13do data transformations and especially
8:42:15by creating new columns and deriving new
8:42:18informations. So as we saw there are
8:42:20amazing use cases for the case
8:42:22statements. For example, we can use it
8:42:23in order to categorize our data. As we
8:42:25learned, we can go and create a new
8:42:27groups of data then to be aggregated for
8:42:29our reports. And then we saw another use
8:42:31case is mapping values. We can use the
8:42:34case statement in order to help us
8:42:36mapping the cryptical technical values
8:42:38that is stored in databases to new
8:42:41values which is more readable and more
8:42:43friendly to be used. And the next use
8:42:45case that we have learned is handling
8:42:46the nulls. We can use the case statement
8:42:49in order to replace the nulls with value
8:42:51to make our aggregations more accurate.
8:42:54And the last use case that we have
8:42:55learned and I think the most used one in
8:42:57my project is doing conditional
8:42:59aggregations where we can aggregate a
8:43:01subset of data that meets specific
8:43:03conditions in order to do focus and
8:43:06target analyszis. Okay my friends. So
8:43:08with that we have covered all the topics
8:43:09and all the functions in order to
8:43:11transform single value in SQL the role
8:43:14level functions that was very important
8:43:16especially for data engineers. So we are
8:43:19done with this chapter. Now we are
8:43:20moving to very interesting chapter.
8:43:22Finally we're going to talk about data
8:43:24analytics in SQL and we will be covering
8:43:26now the aggregate and the analytical
8:43:28functions that we have in SQL. So first
8:43:30we're going to start with the basics. So
8:43:32we will learn simple functions on how to
8:43:34aggregate your data. So let's go.
8:43:40Hey my friends. So now we're going to
8:43:42talk about the aggregate functions in
8:43:44SQL. They are amazing if you are a data
8:43:46analyst or data scientist where we
8:43:48usually use them in order to uncover
8:43:50insights about our data. So the
8:43:51aggregate functions they accept multiple
8:43:54rows as an input and the output of the
8:43:56aggregate function usually is one single
8:43:58value. So now we're going to go and
8:44:00cover first the basic aggregate
8:44:02functions in SQL. So let's go. So now in
8:44:04our database we have four orders and we
8:44:06have the sales informations for each one
8:44:08of them. So now one question that comes
8:44:10in our mind what is the total number of
8:44:12orders in our business. So how many
8:44:14orders do we have? Now in order to do
8:44:16that we use the function count because
8:44:18what it does it's going to go and count
8:44:20the number of rows inside our table. So
8:44:22if you apply the count function on this
8:44:24data SQL going to go and start counting
8:44:26how many rows do we have. So the total
8:44:28number is four and in the output we will
8:44:30get four. So as you can see we don't
8:44:32really care about the content of the
8:44:33tables. Scale is just counting how many
8:44:36rows. So the number is not based on the
8:44:38sales or formations or the orders. So
8:44:40this is how the count function works.
8:44:42Now we have another question and we say
8:44:44I would like to find the total sales in
8:44:46our data in our business. So that means
8:44:48we have to go and summarize all those
8:44:50sales that we have in the order and for
8:44:52that we have the sum function. So if you
8:44:54go and apply the sum function, it's
8:44:56going to go and summarize all the sales
8:44:58and return at the end the total sales.
8:45:00In this example, it's going to be 80.
8:45:01So, as you can see, the aggregate
8:45:03function accept multiple rows, multiple
8:45:05values, and the output going to be one
8:45:07single value, the aggregated value. Now,
8:45:10moving on, I would like to understand
8:45:11what is the average sales in our
8:45:13business. So, it sounds simple. In order
8:45:15to do that, we're going to use the
8:45:16average function. So, if you apply it on
8:45:19the sales, it's going to go and
8:45:20summarize all those values and divide it
8:45:22by the number of values. So, you will
8:45:24get the average of 20. Now comes
8:45:26interesting question where you want to
8:45:28find what is the highest sales in my
8:45:30data. So for that we can use the
8:45:32function max. So once you apply it it's
8:45:34going to go and start searching for the
8:45:36highest value inside our table. So this
8:45:38time we are not really aggregating the
8:45:40data into something new. It's like
8:45:43searching for the highest value between
8:45:44multiple values. So in this example we
8:45:47will get the 35 as the highest sales.
8:45:49Now of course if you want to see the
8:45:51lowest sales inside your business you
8:45:53can use the min function. And if you go
8:45:55and apply it as well, the same thing is
8:45:57going to go and start searching for the
8:45:58lowest value in the sales. And in this
8:46:01example, it's going to be the 10. So as
8:46:02you can see guys, the aggregate
8:46:03functions is very simple but yet very
8:46:05powerful. So it is really useful for
8:46:07insights in order to understand how well
8:46:09your business is performing. So now
8:46:10let's go to SQL in order to try those
8:46:12functions. Okay. So now we're going to
8:46:14go and analyze the orders table inside
8:46:16our database by doing very simple
8:46:18aggregations. So let's start with the
8:46:20first task. It says find the total
8:46:22number of orders. So this time we are
8:46:24targeting the table orders. So let's
8:46:28just start with the select. So now we
8:46:29can see we have like four orders. And
8:46:31now we would like to have like one
8:46:32number. What we can do? We can go and
8:46:34say count star as total number of
8:46:38orders. So let's go and execute it. And
8:46:41with that we got one number. It is the
8:46:43four. This is the total number of
8:46:44orders. Now let's move to the second
8:46:46task. It says find the total sales of
8:46:49all orders. So this time we have to
8:46:51summarize all the sales values in one
8:46:54big value. So how to do it? We're going
8:46:56to use the function sum and this time we
8:46:59are targeting the sales and we're going
8:47:01to go and call it total sales. So let's
8:47:04go and execute it. And with that we have
8:47:0680 as the total number of sales. So all
8:47:09the sales values are summarized in one
8:47:11big value. So as you can see now we are
8:47:12exploring the business right? We are
8:47:14understanding how many sales, how many
8:47:16orders. So this is really the basics of
8:47:18analytics in SQL. Now let's go to the
8:47:20second task. Let's find the average
8:47:22sales of all orders. So we're going to
8:47:24have average this time the sales as
8:47:29average sales. Again very simple. Let's
8:47:31go and execute it. Now the total sales
8:47:33is 80 but the average sales is 20. So
8:47:36all the values of the sales is
8:47:37summarized and then divided by the
8:47:39number of orders. So 80 divided by four.
8:47:42And with that SQL finding the 20 as an
8:47:45average. Now let's go and get
8:47:46interesting stuff. Let's go and find the
8:47:48highest sales of all orders. So what is
8:47:51the highest sales that happens in our
8:47:53business? In order to do that, we can
8:47:54use the function max sales as highest
8:47:59sales. Very nice. Let's go and execute.
8:48:02So the highest sales in the database is
8:48:0535. And now I think you already know
8:48:07what is the next task. Find the lowest
8:48:09sales of all orders. So this is exactly
8:48:12the opposite. So we're going to go and
8:48:13use the min sales as lowest sales. So
8:48:18let's go and execute. The lowest sales
8:48:20in our business was 10. So my friends,
8:48:22as you can see, the aggregate functions
8:48:24are really amazing. And if you use it
8:48:26like this, you will get like the big
8:48:28numbers about our business. But now
8:48:29don't forget about the aggregate
8:48:31functions. If you combine it with a
8:48:33group by then you will be breaking those
8:48:36big numbers into something like you are
8:48:39aggregating by the customer ID maybe by
8:48:42a date by a country. So anything you
8:48:45specify with the group by it going to
8:48:47breaks those big numbers into smaller
8:48:49number based on the column that you are
8:48:51using. For example let's go with the
8:48:52customer ID over here and let's put it
8:48:55at the start as well. And now if you go
8:48:58and execute it. So now as you can see in
8:49:00the output all those numbers are not
8:49:02anymore like big numbers. We drill down
8:49:04to more details based on the column that
8:49:07we have specified. So now we have for
8:49:09each customer the total number of
8:49:11orders, the total number of sales, the
8:49:13average sales, the highest sales or the
8:49:15lowest sales. Of course the data is very
8:49:17small and those numbers can be more
8:49:19interesting if you have bigger data. So
8:49:20if you combine the aggregate functions
8:49:22together with the group by, you will
8:49:24break those big numbers into more
8:49:26details based on the column that you are
8:49:28grouping by. So now what you can do, you
8:49:30can go and apply those functions as well
8:49:32for the customers. There we have a score
8:49:35and you can go and find the average
8:49:37score, the highest score, the lowest
8:49:39score and then you can group up the data
8:49:41by the country for example. So pause the
8:49:44video and do some aggregations on the
8:49:46table customers.
8:49:48[Music]
8:49:52All right my friends. So with that you
8:49:54have learned the basics on how to
8:49:55aggregate your data using SQL. Now we're
8:49:58going to move to more advanced way on
8:49:59how to aggregate your data. We will
8:50:01start talking about the window functions
8:50:03the analytical functions. So first we're
8:50:05going to start talking about what is
8:50:07exactly window functions and we're going
8:50:08to cover the basics about this topic. So
8:50:10let's go.
8:50:15window functions or sometimes we call
8:50:17them analytical functions. They are very
8:50:20important functions in SQL. Everyone
8:50:22must know them especially if you are
8:50:24doing data analyszis. Each time I write
8:50:26SQL script in order to do data
8:50:28analytics, I end up using them. So as
8:50:30usual, we're going to go and now
8:50:31understand the concept behind them and
8:50:34then we're going to start practicing. So
8:50:36let's
8:50:39go. Okay guys, so now let's start with
8:50:41the first question. What are SQL window
8:50:44functions? They are functions that
8:50:46allows you to do calculations like
8:50:48aggregations but on top of subset of
8:50:51data without losing the level of details
8:50:53of the rows. So it is something very
8:50:56similar to the group pi but here we have
8:50:57special case you don't lose the level of
8:51:00details. So now in order to understand
8:51:02the definition let's have a very simple
8:51:04example. Okay. So now let's understand
8:51:05how SQL works with the group by clouds.
8:51:08Let's say that we have the very simple
8:51:10example. We have four orders. two orders
8:51:12for the cabs and two order for the
8:51:14gloves. And let's say that I would like
8:51:16to see the total sales for each
8:51:18products. So now if we decided to use
8:51:20the group by what SQL going to do going
8:51:22to take the first two orders for the
8:51:25caps and put it in one row. So in the
8:51:27output we're going to have only one row
8:51:29for the caps with the total sales of 40.
8:51:32And the same thing going to happen for
8:51:33the gloves. So we're going to take the
8:51:34two rows of the gloves from the input
8:51:36and in the output we're going to have
8:51:38only one row for the gloves. So that
8:51:40means the number of rows going to be
8:51:42depending on the number of products we
8:51:44have on our data. We have two products,
8:51:46we get two rows. So that means SQL is
8:51:49really like smashing or squeezing the
8:51:51results in the output. And this is
8:51:53exactly what the grouper does to our
8:51:55data. It aggregate the rows, aggregate
8:51:57the data into different level of
8:51:59details. So now on the left side we see
8:52:02four rows. On the right side we have two
8:52:04rows and with that we are losing some
8:52:06details in the results. But still we
8:52:08have solved the tasks. So now let's see
8:52:10what going to happen if you use window
8:52:12function in SQL. Okay. So now we have
8:52:13the same data and as with the same task
8:52:15we have to find the total sales for each
8:52:17product. Now if you use window function
8:52:20SQL going to do the following. It going
8:52:22to go and execute each rows individually
8:52:25from each others. So what going to
8:52:27happen it start with the first row the
8:52:28order ID one. In the output we're going
8:52:31to get as well the same stuff the order
8:52:33ID one the same row but we will get the
8:52:35total sales for the caps. So here the
8:52:38total sales is going to be 10 30 we will
8:52:40get 40. Then it's going to jump to the
8:52:42second row and it's going to process it
8:52:44as well. So in the output we will get
8:52:46the order ID two the product caps and as
8:52:48well we have the same aggregation since
8:52:50we are talking about the same product.
8:52:52So we will get 40. Then it's going to go
8:52:54to the third order and here we have the
8:52:55gloves. So in the output again we have
8:52:57the order ID 3 the product gloves and
8:53:00the total sales this time going to be 5
8:53:02+ 20 so we will get 25 then it goes to
8:53:05the last row to the order ID number four
8:53:08in the output we're going to get four
8:53:09gloves and as well 25. So now we can
8:53:12notice that if you use the window
8:53:14function you will not lose the level of
8:53:16details of your data. So we are doing
8:53:18something called rowle calculations. So
8:53:21if in input data we have four orders in
8:53:23the output we're going to get four
8:53:25orders and as well we will get our
8:53:27aggregations correctly. So now if you
8:53:29compare both of the methods side by side
8:53:31we can see that we are solving the same
8:53:33task. So we are finding the total sales
8:53:35for each products but with the group by
8:53:38we are smashing squeezing the results
8:53:40from four orders into two rows one row
8:53:44for each order. So that means with the
8:53:46group by the granularity is changing
8:53:48right in the input the order ID is
8:53:50controlling the level of details but in
8:53:52the output of the group by the product
8:53:54is controlling the level of details. So
8:53:56we have different granularity but in the
8:53:58other hand with the window functions we
8:54:00are still able to do aggregations but we
8:54:02are not losing the level of details. So
8:54:04the granularity of the input going to be
8:54:07the same like the output in the results.
8:54:09So this is exactly the main difference
8:54:11between the group eye and the window
8:54:12function. If you want just to do simple
8:54:15aggregations, then go with the group by.
8:54:17But if you care about the level of
8:54:19details and you need to add more details
8:54:21to your results, then you can go with
8:54:23the window function where you can do
8:54:25aggregations plus having more details.
8:54:27And now if you go and compare the
8:54:29functions between the window and the
8:54:30group by, we can find that both of them
8:54:32has exactly the same functions for the
8:54:34aggregations. So we have the count, sum,
8:54:36average, min, max. And here comes
8:54:38another difference between the window
8:54:40and the group by. The group I has only
8:54:42the aggregate functions. So that's it.
8:54:44But in the window functions, we have way
8:54:46more functions to use for analytics. So
8:54:49for example, we have the ranking
8:54:50functions. And we have here another
8:54:52group of functions for the value or we
8:54:54call it analytical functions. So that
8:54:56means in the SQL window, we have a lot
8:54:58of functions. We can cover a lot of
8:55:00analytical use cases and advanced
8:55:02complex stuff. But with the group by we,
8:55:05we have only the aggregate functions
8:55:06only for simple use cases. So this is
8:55:09another difference between the group by
8:55:10and the window. Group by use it if you
8:55:13have simple analyzes, simple
8:55:15aggregations, window functions, we're
8:55:17going to use it for more advanced data
8:55:18analyszis where we're going to cover a
8:55:20lot of use cases. All right guys, so now
8:55:22we're going to have few tasks in order
8:55:23to understand one thing. Why do we need
8:55:26scale window functions and why in some
8:55:28scenarios group is not enough and we
8:55:31have to use scale window functions. So
8:55:33let's go. All right. So let's start with
8:55:34very simple task. It's going to say find
8:55:37the total sales across all orders. So we
8:55:40need one value with the total sales.
8:55:42Let's see how we can do that. First make
8:55:44sure that you are using the database. So
8:55:46use sales database in case you have
8:55:48closed the clients so that we don't get
8:55:51any errors. So now we're going to start
8:55:53with the first thing. We're going to go
8:55:54and select the sales. You're going to
8:55:56find it in the table sales orders. So
8:55:59now let's just query the data. And as
8:56:01you can see we have 10 orders with 10
8:56:03sales. We didn't aggregate anything yet.
8:56:06So we have the row data now. So now in
8:56:07order to solve the task, we're going to
8:56:09use the function sum. So sum of sales
8:56:12and we're going to give it new name
8:56:14total sales. We don't have to use any
8:56:16group by because we don't have to group
8:56:18up anything. So that's it. Let's go and
8:56:21execute that. And as you can see SQL
8:56:22going to return one value 380. This is
8:56:25the total sales that we have inside our
8:56:27data. And this is the highest level of
8:56:30aggregations. So with that we have
8:56:32solved the task. We have the total sales
8:56:34across all orders. We don't have to
8:56:36group up anything. Let's move to the
8:56:38next example. Let's say that in the next
8:56:40task, this time we want to find the
8:56:42total sales but for each products. So
8:56:45not for the all orders, for each
8:56:47products we want to find the total
8:56:49sales. So this time we don't need only
8:56:51one value. We need one value for each
8:56:54product. In order to do that now, we're
8:56:56going to go and use the group by
8:56:58function. And we're going to group up by
8:57:00the product ID. and group up need as
8:57:02well the dimension in the selection. So
8:57:05we can do it like this. So that's it.
8:57:07Let's go and execute the query. Now as
8:57:09you can see in the results we don't have
8:57:10one value. We don't have the highest
8:57:12aggregations. This time we are drilling
8:57:15down to the next level of details. So
8:57:17the level of details here is the product
8:57:19ID. We have one row for each product. So
8:57:22for the first product we have 140. The
8:57:25next one 105 and so on. So as you can
8:57:27see we are now splitting the data at the
8:57:30level of product ID and we went from 10
8:57:33orders now in the results we have four
8:57:35orders and that's because we have four
8:57:37products. So the number of rows at the
8:57:40output going to be defined by the
8:57:42dimension the product ID and with we
8:57:44have solved the task we have the total
8:57:46sales for each product. All right guys
8:57:48so let's keep progressing in our
8:57:49examples. Now the next one going to be a
8:57:52little bit advanced where we have the
8:57:53same aggregation. Find the total sales
8:57:56for each product. Additionally, provide
8:57:58details such order ID and the order
8:58:01date. So, as you can see, we have
8:58:03already solved the first part. We are
8:58:05finding the total sales for each
8:58:06product. Now, we just have to add some
8:58:08additional informations like the order
8:58:10ID and the order date. So, let's go over
8:58:12here and just add it in our select. So,
8:58:14order ID and let's have the order date.
8:58:17So, let's go and execute that. Just
8:58:19going to make it a little bit bigger.
8:58:21So, let's go. But now as you can see SQL
8:58:23will not be happy going to throw an
8:58:24error and says the stuff that you are
8:58:26adding to your select are not included
8:58:29in the group by. So as you can see in
8:58:31the group buy we have only one dimension
8:58:32or one field called the product ID. But
8:58:35in our selection we have three
8:58:36dimensions the order ID, the order date
8:58:39and the product ID. So there is no
8:58:41matching between the select and group by
8:58:43and SQL will not allow it. And now you
8:58:45might say you know what let's add
8:58:47everything to the group by. So with that
8:58:49we're going to get our aggregation and
8:58:50as well we're going to get our details.
8:58:52So let's try that. I'm just going to
8:58:53zoom out a little bit and instead of
8:58:55having the product ID let's add
8:58:57everything. So the order ID, order date
8:59:00and the product ID. So now we have
8:59:02matching and SQL should not throw any
8:59:04error. Let's go and execute it. So now
8:59:06let's check whether we have solved the
8:59:08task. The task has two parts right. We
8:59:10have to do the aggregations and to
8:59:12provide details. So as you can see we
8:59:14have solved the second part. We have the
8:59:16details, order ID and order dates. But
8:59:18now the first part finding the total
8:59:20sales for each product is destroyed
8:59:22because if you check the results, we
8:59:24have the product ID 101. It has the
8:59:26total sales of 10. But in the third
8:59:28order, we have it as a 20 for the same
8:59:31product. So actually the data is not
8:59:33aggregated and that's because we are
8:59:34aggregating at different levels and we
8:59:36have included way more stuff that we
8:59:39don't need for the aggregations. We are
8:59:40aggregating at the order ID level. So as
8:59:43you can see now we are hitting the
8:59:45limits of group by. We cannot provide
8:59:47aggregations and as well provide
8:59:49additional informations from our data.
8:59:51You have to pick one. That's why we have
8:59:53to go to the second option where we can
8:59:55use the window functions. So let's do
8:59:57that. I'm just going to get rid of the
8:59:58group by parts and as well all the
9:00:01fields. Let's back to the root. So now
9:00:03we have the sum of sales and if execute
9:00:05this I'm going to get one value. So we
9:00:07are at the highest level of
9:00:08aggregations. So now we need to use the
9:00:10window function. I'm just going to
9:00:12remove the name. And now we're going to
9:00:14tell SQL this is a window functions
9:00:17using over after the aggregations or the
9:00:19functions tells SQL we are talking about
9:00:22window functions. So let's just execute
9:00:23it like this. And with that we got 10
9:00:26rows and that's because we have 10
9:00:28orders and for each row we have exactly
9:00:30the same value. So we have the total
9:00:32sales of all orders for each row. So as
9:00:35you can see SQL understands this is a
9:00:37window function and SQL should not like
9:00:40group all the data in one row. It should
9:00:42keep exactly the same rows or same
9:00:44number of rows like the input. So with
9:00:46that we have the window function but we
9:00:48have to split the data by the products.
9:00:51So now we're going to use the keyword
9:00:52partition by it's like the group by but
9:00:54another wording products ID the same
9:00:57dimension. So with that we have the
9:00:59total sales by products as a name. So
9:01:02let's go and execute this. So now as you
9:01:04can see in the output we still have the
9:01:06same number of rows. We have 10 orders.
9:01:08We have 10 rows but the result did
9:01:10change because now we are aggregating
9:01:12the data at the level of product ID. In
9:01:14order to understand the results we have
9:01:16to add more informations to our select.
9:01:18So now let's add the same dimension.
9:01:20It's going to be the product ID. I'm
9:01:22just going to add it at the front over
9:01:23here. So let's select. And as you can
9:01:26see now it makes more sense. We have
9:01:28those products and they have always the
9:01:31exact same uh sales. and as well for the
9:01:34next product and so on. And now here
9:01:35comes the magic of the window function.
9:01:37We can add more informations to our
9:01:39select statement without having any
9:01:41errors. So now we need additional
9:01:43informations like the order ID. So we
9:01:45can go over here and say order ID, order
9:01:48date, any type of column you can add it
9:01:51to your select and let's go and execute.
9:01:53So as you can see now we got the result
9:01:55even though that those three dimensions
9:01:57in the select are not part of the window
9:02:00aggregation. So with that we have solved
9:02:02the tasks. We have additional
9:02:04informations. We have the order ID, the
9:02:06order date and as well the first part of
9:02:08the task to find the total sales for
9:02:10each products. So each of those values
9:02:13are the total sales for each product.
9:02:15And with that we have solved the tasks
9:02:17and this is exactly why we need window
9:02:19functions. In real projects things get
9:02:21really complicated. You are doing
9:02:23different tasks in one query. So you are
9:02:25doing aggregations, you are doing some
9:02:27other stuff. So just focusing on the
9:02:30aggregations is not going to be enough.
9:02:31You have always to add additional
9:02:33informations to your query. So as you
9:02:35can see we use group eye to do simple
9:02:36analyszis but as things get complicated
9:02:39in the analytics we use the window
9:02:41functions in order to show the
9:02:43aggregations and as well add additional
9:02:45informations. So as you can see we use
9:02:47group eye to do simple analyszis but as
9:02:49things get complicated in the analytics
9:02:52we use the window functions in order to
9:02:54show the aggregations and as well add
9:02:56additional informations.
9:03:01All right everyone. So now we're going
9:03:02to go and deep dive into the syntax of
9:03:04the SQL window functions. We're going to
9:03:06cover everything each part of the syntax
9:03:08for you to understand how to use them.
9:03:11So let's go. All right. So let's start
9:03:12first by understanding the basic
9:03:14components or the basic parts of each
9:03:16window syntax. Mainly we have two parts.
9:03:19The first part going to be the window
9:03:21function. We have like sum, average and
9:03:23so on. The second main part is going to
9:03:25be the over close. And inside the overlo
9:03:28we have three different parts. The first
9:03:30one going to be the partition close. The
9:03:32second order close and the last one we
9:03:35have the frame close. And those are all
9:03:37components that you can use inside the
9:03:39window function. So two main parts
9:03:41window function and the offer close. And
9:03:43inside the over we have partition order
9:03:45and frame. Let's go more in details. So
9:03:47for example we have the following window
9:03:49function. So as you can see we have a
9:03:51lot of stuff going on here. We're going
9:03:53to understand them step by step
9:03:54component by component. Let's start from
9:03:56the left from the first one. So what do
9:03:58we have over here? We have a function
9:04:00window function. So what is a window
9:04:02function? Like here we have the average.
9:04:04It's like any other function in SQL. You
9:04:06can use it in order to do calculations
9:04:08on top of the window. So the first thing
9:04:10to do or to define in a window is to
9:04:13define the function of the window. And
9:04:15as we learned before, we have a long
9:04:17list of many window functions available
9:04:19in SQL. And we group them into three
9:04:21groups. The first one we have the
9:04:23aggregate functions. So we have the
9:04:25count, sum, average, max. All those
9:04:27functions we have them as well for the
9:04:29group by. So those are used for the
9:04:32aggregations. The second group of
9:04:34functions we have the ranking functions.
9:04:37So we have the row number, rank, entile
9:04:39and so on. So we can use those groups in
9:04:41order to give a rank for our data. The
9:04:43last group we call it value or sometimes
9:04:46analytics functions. So here we have
9:04:48very important functions like the lead,
9:04:49lag, first value and the last value in
9:04:52order to access a specific value and of
9:04:54course we're going to go and learn all
9:04:56of them one by one understanding the
9:04:58concepts some examples and as well for
9:05:00you to understand when to use them for
9:05:02that analyzers. All right so now let's
9:05:04keep moving understanding the other
9:05:05parts of the window syntax. Now inside
9:05:08the function average we have here a
9:05:10field name or column name called sales.
9:05:12This called a function expression. It's
9:05:15like a value, a parameter, argument that
9:05:17we can pass it to the function. And here
9:05:19we can use multiple different stuff. For
9:05:22example, depend of the function of
9:05:23course. So here it could be empty like
9:05:25here in the ranking. It doesn't allow to
9:05:27use an expression. So it should be
9:05:29always empty. Or we can use a column
9:05:31like in the example we use the sales. So
9:05:34we use the column name as an argument or
9:05:36an expression. For the average we are
9:05:39finding the average of sales or we could
9:05:41use a number. So here in the intile we
9:05:43are allowed only to use numbers or we
9:05:46could have multiple stuff. For example
9:05:48in the lead we can have sales then
9:05:50numbers and so on. So things get
9:05:52complicated. Don't worry about it. I'm
9:05:53going to explain that. So here we have
9:05:55multiple stuff or we can have a whole
9:05:58conditional logic. So for example here
9:06:00we have the case win so on inside the
9:06:03sum. So the whole thing over here holds
9:06:06an expression for the sum. So as you can
9:06:08see we can build here a complex logic
9:06:10and the output of this logic can be
9:06:12passed to the function sum. So that
9:06:14means as an expression for the function
9:06:16we can use different stuff of course
9:06:18depends whether the function allows it
9:06:20or not. All right. So now let's have a
9:06:21quick overview in order to understand
9:06:23which data types are allowed in the
9:06:25expressions for those functions. Let's
9:06:27see the aggregate functions. As you can
9:06:29see the count function accept any data
9:06:31type but the others like the sum,
9:06:34average, min, max, they allow only
9:06:36numerical data types. All right. So now
9:06:38let's move to the rank function. The
9:06:40expressions it's pretty easy. It should
9:06:42be empty. It doesn't allow any argument
9:06:44or anything inside those functions. So
9:06:46as you can see all of them are empty but
9:06:49only one that accept numerical values
9:06:51which is the end tile. You have to
9:06:53define a numeric value. And now moving
9:06:55on to the last type we have the value
9:06:57functions. they accept any data types
9:06:59inside the expressions. So as you can
9:07:01see each functions has its own
9:07:03specifications and you have to be
9:07:05careful which data type you are using in
9:07:07the expressions. Okay. So now let's keep
9:07:09moving to the next one. We have a very
9:07:11important part in the window syntax. So
9:07:13so far what do we have? We have a
9:07:14function. We have an expression. It's
9:07:16like usual stuff. We have done that
9:07:18before using the group by. Now we have
9:07:20to tell SQL that we are dealing with the
9:07:23window function. It's not a normal one.
9:07:25In order to do that we have to specify
9:07:27the keyword over. So the second main
9:07:30part in the syntax is the over clause
9:07:32and we use it in order to define a
9:07:34window and inside it we can define
9:07:36multiple stuff like the partition pie
9:07:38the order by the frame but all those
9:07:41stuff are optional. We can skip it and
9:07:43leave it empty. So the main task of the
9:07:46over it tells first SQL we are dealing
9:07:48with the window function here and as
9:07:50well you can use it in order to define a
9:07:53window of your data. So now we're going
9:07:55to go and cover everything inside the
9:07:56over clause and we're going to start
9:07:58with the first one the partition
9:08:03pi. All right. So now we're going to
9:08:05learn how to define a window inside the
9:08:07overlaus. The first part that we can
9:08:10define is the partition pi. So for
9:08:12example here we have partition pi
9:08:14category. We have to define the
9:08:15dimension. It's very similar to the
9:08:17group by another wording. So the first
9:08:19part is going to be the partition
9:08:20clause. What it going to do? It's going
9:08:23to divide the entire data sets into
9:08:26groups or you can call it windows
9:08:28partitions. So here we tell SQL how to
9:08:31divide our data. And here we have two
9:08:33options. Let me just show you. So if we
9:08:36don't use anything so we have it empty.
9:08:38You see over and partition by is not
9:08:41used. What going to happen? SQL going to
9:08:44use the entire data in order to do the
9:08:46calculations. So the whole data the
9:08:49entire data going to be counted as one
9:08:51window. So we are telling SQL don't
9:08:53divide anything leave it as it is. The
9:08:56second option that we have is to divide
9:08:57the data by partition pi. So we define
9:09:00the window like this partition pi
9:09:02products for example. So SQL going to go
9:09:04and divide the entire data into
9:09:06different windows. For example here two
9:09:08windows. And here this time the
9:09:10calculation the sum of sales will not
9:09:12apply on the entire data set. This time
9:09:15it going to be applied on the different
9:09:17windows individually. So we're going to
9:09:19find the sum of sales for window one
9:09:22separately from the total sales of
9:09:24window 2. All right. So now we have this
9:09:25very simple example. We have here three
9:09:27fields. The month, product, sales. They
9:09:30are really easy informations. And now we
9:09:32have the following SQL window function.
9:09:34So we have sum of sales and inside the
9:09:38overlo we are not using anything. So we
9:09:41are not using partition by. So how SQL
9:09:43going to define the window. Now SQL
9:09:45going to say okay I don't have to divide
9:09:47anything. The entire data set is one
9:09:50window. So SQL going to go over here and
9:09:53say the whole thing is one window. So
9:09:56there is no partitions, there is
9:09:57nothing. We have only one window. So the
9:10:00entire data going to be aggregated. So
9:10:02this is what happen if you don't use
9:10:04partition pi and you leave the
9:10:06overclos. The entire data is one window.
9:10:09All right. So now let's move to the next
9:10:10example. We don't want to have only one
9:10:12window. We would like to have multiple
9:10:13windows. So we have to divide the data
9:10:16by something. So in the over clause
9:10:18we're going to define the window like
9:10:19the following partition by month. So
9:10:22it's not empty. We are now dividing the
9:10:25data by the field month. So the values
9:10:28inside this column going to divide the
9:10:30data sets. So here we have two months
9:10:33January and February. So what's going to
9:10:35do? SQL going to go and divide the data
9:10:38into two sets. The first window going to
9:10:40be this one of January. So we have the
9:10:43first window going to make it smaller
9:10:45and the second window going to be the
9:10:47February. So it's going to be two
9:10:49windows inside our data and the
9:10:51calculation going to be happening on
9:10:53each window separately. So here as you
9:10:55can see we are using the month in order
9:10:57to divide our data sets into two
9:11:00windows. One window for January and
9:11:03another window for the February. So now
9:11:05let's have a quick overview about the
9:11:06options that we have with the partition
9:11:08p. The first option as we learned we can
9:11:10just skip it. So without partition by
9:11:12for example here total sales across all
9:11:15rows and here we don't find anything
9:11:16inside the SQL. The second option we can
9:11:19use one field one column for example
9:11:22partition by product. So we are using
9:11:24one dimension but we can go and mix
9:11:27stuff. We can use multiple columns or
9:11:29multiple dimensions in the partition by
9:11:31for example here partition by product
9:11:33and order status. So here with the
9:11:35partition by we can define a list of
9:11:37dimensions that could be used in order
9:11:39to divide our data. So in this example
9:11:41we are saying find the total sales for
9:11:44each combination of products and order
9:11:46status. So those are the different
9:11:48options on how to work with the
9:11:49partition pi. So now let's have this
9:11:51overview again for all functions. The
9:11:54partition pi for all those functions is
9:11:56optional. So if you don't use the
9:11:58partition pi in all those functions you
9:12:00will not get any errors. So now let's go
9:12:02back to SQL in order to start practicing
9:12:04with this clause. Okay. So now we have
9:12:06the following task. Find the total sales
9:12:08across all orders. And we have to
9:12:10provide additional informations like the
9:12:12order ID and the order date. So let's go
9:12:15and solve it step by step. First I would
9:12:17like to provide the details. So I'm
9:12:19going to select the order ID and the
9:12:21order dates from the table sales orders.
9:12:26And next we're going to work with the
9:12:27aggregations. So we need to find the
9:12:28total sales across all orders. Again
9:12:31since we have here details and
9:12:32aggregations we cannot use group by. We
9:12:35have to use the window function. So
9:12:36we're going to go use the function sum
9:12:39for sales. And now we have to tell SQL
9:12:42we are working with window functions.
9:12:44That's why we're going to use the over
9:12:45close. And now the next step we have to
9:12:47think about defining the window. So
9:12:50let's check the task. It says total
9:12:51sales across all orders. So that means
9:12:54we don't have to partition or divide the
9:12:57data sets into like chunks or
9:12:59partitions. We have to leave it as it is
9:13:01like the whole data going to be one
9:13:03window. And that's why we don't use
9:13:05partition pi inside the definition.
9:13:07We're going to leave it empty. Let's go
9:13:08now and give it a name. It's going to be
9:13:10the total sales. Let's go and execute
9:13:12this. And now at the results, as you can
9:13:14see, we have all the orders, all the
9:13:16details, and as well, we have the total
9:13:18sales across all orders. So with that,
9:13:21we have solved the tasks. We have the
9:13:22total sales and as well some details
9:13:24about the order. All right. So now let's
9:13:26move to the next task. It's going to be
9:13:28very similar. So it says find the total
9:13:30sales for each product. And we have to
9:13:32provide additional informations like the
9:13:34order ID and the order dates. So it's
9:13:36going to be very similar task but this
9:13:38time we have to divide the entire data
9:13:42into windows and that's going to be by
9:13:44the product. Since we are saying total
9:13:46sales for each product. So this time we
9:13:48have to go and divide the data. So we're
9:13:51going to define the window like this
9:13:52partition by and we can use the
9:13:55dimension product ID. Let's go and
9:13:58execute this. So now you can see in the
9:14:00total sales we don't have anymore the
9:14:02total sales of the whole data but they
9:14:04are divided but in order to understand
9:14:06the results let's go and include the
9:14:08product ID in the results. So product ID
9:14:11and execute. So now by looking to the
9:14:13results you can see that the data is
9:14:15divided into four windows. Let's see
9:14:18them. It's going to be by the product
9:14:19ID. So this dimension going to be
9:14:21controlling the partition. So the first
9:14:23window going to be the product ID 101.
9:14:26So we have the total sales for this
9:14:27product 140 and the next window going to
9:14:30be 102. The third one 104 and the last
9:14:34window it's going to be only one row the
9:14:36105 and the total sales of 60. So with
9:14:39that we have solved the task. We have
9:14:40the total sales for each product and as
9:14:43well we have some details. Now I would
9:14:44like to show you the dynamic of the
9:14:46window function. We can add multiple
9:14:49aggregations on multiple levels. Let me
9:14:51show you what I mean. Let's say we stay
9:14:53with the same example but we're going to
9:14:55find the total sales across all orders
9:14:57and as well the total sales for each
9:15:00products. So what we can do we can do
9:15:03the window functions on different levels
9:15:05by for example here removing the whole
9:15:07definition. So here we have the total
9:15:08sales for the entire data for the first
9:15:11task and the next one going to be the
9:15:12total sales but divided by the product
9:15:15ID. Let's here rename it by
9:15:17products. Let's go and execute this. And
9:15:21now you know what I'm going to go and
9:15:22add the sales as well just to explain
9:15:24the flexibility of the window function.
9:15:26So let's go add the sales and execute it
9:15:28again. And now by looking to the results
9:15:30you can see we have the sales
9:15:32informations three time but with
9:15:34different granularities. The first sales
9:15:37the sales it sales without any
9:15:38aggregations. It is the highest level of
9:15:41details of the sales and we're going to
9:15:42have the sales for each order. The next
9:15:45one the total sales with the window
9:15:47function. Here we have the highest level
9:15:48of aggregation. So we have the total
9:15:50sales of all orders and the last one the
9:15:53total sales by product it's something
9:15:54like in the middle we are aggregating on
9:15:57a window and the window going to be the
9:15:59product ID. So as you can see we have
9:16:01different granularities of the
9:16:02aggregations and this is exactly the
9:16:04flexibility that we have with the window
9:16:06function. We can do all those stuff in
9:16:08one query. Okay. So now let's keep
9:16:10moving and adding stuff to our task.
9:16:12It's going to say find the total sales
9:16:14for each combination of the products and
9:16:16the order status. So this time we have
9:16:19to divide the data not only by the
9:16:20product as as well with another
9:16:23dimension the order status. So now let's
9:16:25see how we can do that. I'm going to
9:16:27just show the dimension order status and
9:16:29the results and we're going to add the
9:16:32following thing. So sum sales over since
9:16:37it's a window function and let's go now
9:16:39and define the window partition by. So
9:16:41we have again the product ID but not
9:16:43only this dimension as well the order
9:16:46status and let's go and call it sales by
9:16:50products and status. Let me just rename
9:16:53those stuff. Okay. So let's go and
9:16:56execute. All right. So now let's check
9:16:57the results. It is the last aggregation
9:16:59over here. And as you can see here the
9:17:01aggregation has different granularities
9:17:03as the previous one. And we have more
9:17:06details. This time we are splitting the
9:17:08data by two dimensions. So the first
9:17:10window going to be the product ID with
9:17:12the order status it's going to be only
9:17:14those two rows. So we have the order ID
9:17:16101 and the order status delivered. So
9:17:19the total sales of this going to be 10 +
9:17:2220 and we're going to have 30. The next
9:17:24window going to be the same product but
9:17:27with different status. So it's going to
9:17:29be the 101 shipped and we're going to go
9:17:31and summarize those two values and we're
9:17:33going to have 110. The next product and
9:17:36order status going to be the 102 and we
9:17:38have it only once. So 102 delivered it's
9:17:41only once. So it's going to be the same
9:17:43value. The next partition or window it's
9:17:45going to be two rows. 102 with the
9:17:48shipped is going to be those two things
9:17:5160 + 15 we're going to get 75. So as you
9:17:54can see here the product ID and the
9:17:56order status they are controlling how
9:17:59many windows we're gonna get. So we get
9:18:00here around like six windows. With the
9:18:02product ID we got only four windows and
9:18:05without using anything inside the
9:18:07overlause we will get only one window.
9:18:10So this is how the partition by
9:18:15works. All right. So that was the first
9:18:17part of the window definition within the
9:18:19overclo. Let's move to the next part. We
9:18:21have the order by. For example, we can
9:18:23use order by order date. It's just a
9:18:25field. So the order close is very
9:18:28important in order to sort your data
9:18:30within a window. So the order by is very
9:18:32important as well for many functions. So
9:18:34by just checking the overview over here
9:18:36for the aggregate functions it is
9:18:38optional. So you could just leave it or
9:18:39add it. But for the rank function and as
9:18:42well for the value functions they are a
9:18:45must. So if you want to use those
9:18:47functions you must use the order clause
9:18:50because it makes no sense for example if
9:18:52you are ranking the data without sorting
9:18:54your data first. Okay guys. So now back
9:18:56to our very simple example and we have
9:18:57the following query. So the function
9:18:59this time going to be rank. So we have
9:19:01to rank the data and the definition of
9:19:03the window going to be partition by
9:19:05month. So that means we divide the data
9:19:07by the months. So we have it over here.
9:19:10And then the second part going to be
9:19:12order by sales descending. So we have to
9:19:14sort each window by descending order.
9:19:17That means we start with the highest
9:19:18value and we end up by the lowest value.
9:19:21So let's see how SQL going to go and
9:19:22execute this. So first partition by
9:19:25month. So it's going to divide the data
9:19:26into two partitions because we have two
9:19:28values by the month. So let's see how
9:19:31this going to look like. So one window
9:19:33for January and another window for
9:19:36February. All right. So now SQL going to
9:19:38go to the second part and execute order
9:19:40by sales descending. So what's going to
9:19:42happen? SQL going to go for each window
9:19:44separately and start sorting the data
9:19:46from the highest to the lowest without
9:19:48checking the other window. So in those
9:19:50three values, the highest one is this
9:19:52one. So it's going to be on top. Let me
9:19:54just sort it. This is going to be the
9:19:56lowest. You're going to be in the
9:19:58middle. So SQL going to sort this window
9:20:01separately from the next one. And then
9:20:02once it's done, it's going to go to the
9:20:04second one. So the highest value going
9:20:06to be this one. You are the lowest. Let
9:20:09me just do it like this. So SQL going to
9:20:12sort it like this. The highest one is
9:20:1470. The next one is 40. And the last one
9:20:16is five. So with that SQL done with the
9:20:18definition of the window. So it's
9:20:20splitted by the month. And each window
9:20:23is sorted by the cells. The next step is
9:20:26going to go and rank those values. So
9:20:28it's really simple. In the output, it's
9:20:30going to rank the data like this. So the
9:20:32first one going to be this value. The
9:20:34next one going to be two and the third
9:20:36one going to be three. So as you can
9:20:38see, SQL is sorting only this window and
9:20:40it's going to go and repeat the same
9:20:42stuff for the second window. So each
9:20:44rank is separately from the others. So
9:20:46as you can see, it's very simple. This
9:20:47is how SQL executes partition by
9:20:50together with the order buy for the rank
9:20:52function. All right. So now let's have a
9:20:53quick task for the order by. It says
9:20:55rank each order based on their sales
9:20:57from the highest to the lowest. And we
9:21:00have to provide additional informations
9:21:01like order ID and order date. So let's
9:21:03see how we can write the query. So we
9:21:05have the basic stuff order ID, order
9:21:07date and sales. And now we're going to
9:21:09go and rank the data using window
9:21:11function. So we're going to use the
9:21:12function rank and then we're going to
9:21:15tell SQL this is a window function and
9:21:18inside it we have now to provide the
9:21:20definition of the window. So now by
9:21:21checking the task you can see that we
9:21:23don't have to divide the data. So we
9:21:25don't have to use partition by we have
9:21:27just to use rank and with rank we have
9:21:29to use the order by it is must. So we're
9:21:31going to use order by the field going to
9:21:34be the sales and from the highest to the
9:21:36lowest. So let's just call it rank sales
9:21:39and let's go and execute this. And as
9:21:41you can see our results going to be
9:21:43sorted from the highest to the lowest.
9:21:45So you can see the sales 90 at the top
9:21:47and the lowest going to be the 10. And
9:21:49as well we have a rank. So for the top
9:21:51rank it's going to be one and the lowest
9:21:52rank going to be 10. So as you can see
9:21:54we just quickly create a rank in SQL.
9:21:57It's very simple. The whole thing is one
9:21:59window since we are not using partition
9:22:01pi. And of course if you want to have
9:22:03ascending so from the lowest to the
9:22:06highest you can just remove it because
9:22:08optionally going to be ascending. So
9:22:10let's go and execute the query. So now
9:22:12we can see the orders are sorted the way
9:22:14around. So we start with the lowest and
9:22:16end up with the highest. And of course
9:22:17you're going to get the same results if
9:22:19you go over here and add ascending. So
9:22:22if you execute you see we got exactly
9:22:24the same results. So this is how you use
9:22:26the order by inside the window
9:22:28definition.
9:22:32Okay guys, so with that we have covered
9:22:34the second part of the window
9:22:35definition. Now we're going to go to the
9:22:37last part to the most advanced part of
9:22:39window and we have the following stuff.
9:22:41So we have rows unbounded proceeding. We
9:22:45call this frame close or window frame.
9:22:47So what we are doing over here that we
9:22:49are defining a subset of rows within
9:22:52each window that is relevant for the
9:22:54calculation. Totally understand if this
9:22:56is confusing at the start or complex. It
9:22:58was for me as well. So what we're going
9:23:00to do we're going to deep dive into the
9:23:01concept in order to understand how this
9:23:03works and we're going to do it step by
9:23:05step. So don't worry about it. All
9:23:06right. So now let's understand what is
9:23:08going on with the frame close from the
9:23:10basics. So now if you do aggregations
9:23:12and you don't use window function you're
9:23:14going to consider the entire data or
9:23:16rows inside the table. But what we can
9:23:18do we can go and divide the data using
9:23:20partition pi to a window. So for example
9:23:23here we have window one and window two.
9:23:25Now if you go and do aggregations all
9:23:27the rows in the window one going to be
9:23:29aggregated and then scale going to go to
9:23:31the window two and aggregate all the
9:23:33rows. What we can do in scale is that we
9:23:35can say you know what I don't want all
9:23:37rows inside the window. I want a subset
9:23:40of rows inside the window. So what we
9:23:42are doing over here is that we have
9:23:44those two windows but we specify a scope
9:23:47or we specify subset of data from each
9:23:50window to be involved in the
9:23:51aggregations. And of course not only
9:23:53aggregations we can do ranking other
9:23:55stuff. So I mean calculations. So here
9:23:57like we have a window inside a window.
9:23:59So we are defining a scope of rows. Not
9:24:02all rows should be involved in the
9:24:04calculation but only specific subset of
9:24:07data. And we can do that using the frame
9:24:09clause. So again the partition by you
9:24:12can use it in order to divide the entire
9:24:14data set into multiple windows. And now
9:24:17for the frame close. If you don't want
9:24:18to consider all the rows within each
9:24:20window in the calculation, you want to
9:24:23focus and specify only a subset of data
9:24:25within each window. Then you going to go
9:24:27and use the frame close. All right. So
9:24:29now let's go and understand the syntax
9:24:31of the frame close. Let's have the
9:24:33following example. We are saying the
9:24:34window function is the average of sales
9:24:37and then we define the window. So we
9:24:38have the first partition by categories,
9:24:41order by order dates and then we have
9:24:43the frame close. It's going to be the
9:24:45following rows between current row and
9:24:48unbounded preceding. This is the frame
9:24:50types and we have two types. We have the
9:24:52rows and groups. Then we have like
9:24:54between and range. So the first range
9:24:57going to be the frame boundary, the
9:24:59lower value. And here it accepts three
9:25:02types of keywords like the current row
9:25:05or number of preceding or the unbounded
9:25:07preceding. And then we have another
9:25:09frame boundary. It's going to be the
9:25:11higher values and it accepts the
9:25:12following stuff. We can use the current
9:25:14row in following or unbounded following.
9:25:17So as you can see we are defining like
9:25:19boundary or a range from low value to
9:25:21higher value. So now we have some rules.
9:25:24We cannot use the frame clause without
9:25:26order by. So order by must be exist in
9:25:29the definition in order to use frame
9:25:32clause. And the second rule it says
9:25:34lower boundary must be before the higher
9:25:36boundary. So always we start with the
9:25:38lower boundary and we end up having the
9:25:40higher boundary. You cannot switch that.
9:25:43Okay. So now we have a very simple
9:25:44example. We have the month and the sales
9:25:46and the following query. Sum of sales.
9:25:48This is the window function. And the
9:25:50definition of the window going to be
9:25:51order by month. We are not using
9:25:53partition by just in order to make our
9:25:54life easier. And the frame close going
9:25:57to be defined like this. Rows between
9:25:59current row and the two following. So
9:26:01now let's see how SQL going to execute
9:26:03this. The first definition order by
9:26:05month. As you can see the months are
9:26:06sorted already. So now SQL going to work
9:26:09with the frame definition current row
9:26:11and the two following. So SQL going to
9:26:13process this row by row. So it's going
9:26:15to start with the first row and it's
9:26:17going to be our current row as here in
9:26:19the SQL. So this is our current row and
9:26:22we say the range until two rows, two
9:26:24following rows. So it's going to be
9:26:25February and March. So that means the
9:26:28pointer going to be over here for the
9:26:30two following. So with this we have the
9:26:32frame boundaries and SQL have the
9:26:34following scope for the first row. So we
9:26:36have three rows and the summarization of
9:26:39those three rows going to be around 70.
9:26:42So we will get for the first row 70
9:26:44because the scope is not all rows but
9:26:46only this subset of data. Okay. So with
9:26:49that scale is done with the first row
9:26:50it's going to jump to the second row. So
9:26:52the pointer going to be the current row
9:26:54at the February and the second two
9:26:56following going to be at April. So with
9:26:58that as you can see we are sliding down
9:27:01in the subset of data or in the window
9:27:03and with that we have a new scope a new
9:27:05subset and the summarization of all
9:27:06those values going to be 45. So that's
9:27:09it. I think you get it already. It's
9:27:11going to go to the next one. The pointer
9:27:13going to be on March and the two
9:27:14following going to be on June and it's
9:27:16going to slide like this. We have those
9:27:18three rows in the scope and the
9:27:19summarization of that going to be 105.
9:27:22So now things gets interesting for the
9:27:24next row. So the pointer for the current
9:27:26row going to be April but the two
9:27:28following going to be like after the end
9:27:30of the table or something like that. So
9:27:32as we slide down as you can see the
9:27:34scope now or the subset of the frame
9:27:35going to be only two rows and the output
9:27:38going to be 75. And finally if you go to
9:27:40the last row it's going to be the
9:27:42current row and we're going to have only
9:27:44one row for this subset because the two
9:27:46following is just outside of the table
9:27:48and we're going to get the same value as
9:27:50the summarization. So as you can see
9:27:51that's it. It's very simple right? So
9:27:53the frame we use it in order to scope
9:27:56which rows are involved in the
9:27:58calculations. So all you have to do is
9:28:00to define the boundaries of the frame,
9:28:01the lower and the upper boundary. Let's
9:28:04see what other options do we have with
9:28:05the frames. Okay. So here we have the
9:28:07same example but we redefine the
9:28:09boundaries of the frame like this. Rows
9:28:11between current row this is the first
9:28:13boundary and unbounded following. This
9:28:16means that we are targeting always the
9:28:18last record in the window or in the
9:28:20table. So unbounded following going to
9:28:22be always static and it's going to be in
9:28:24this example pointing to June. And now
9:28:26it's still going to go row by row and
9:28:28the current row going to be like the
9:28:30start January and then February. I'm
9:28:32just going to take this example the
9:28:34pointer is on February and the subsets
9:28:36or the frame going to be those four
9:28:38rows. So it's going to be February,
9:28:40March, April, June. So it's going to be
9:28:41four rows and the total aggregation of
9:28:44that going to be 115. So you can do it
9:28:47like this. And previously it was like
9:28:50flexible more flexible it was two
9:28:52following but this time we have
9:28:54unbounded following that means always
9:28:56the boundary going to be the last one.
9:28:58So as we are moving with the records
9:29:00over here the boundary is going to be
9:29:03smaller smaller and like this and the
9:29:06last one they going to be both in the
9:29:08same record. So the current record going
9:29:11to be as well the unbounded following.
9:29:13Okay let's see the next one. The
9:29:14definition of the window going to be the
9:29:16following rows between one proceeding
9:29:18and the current row. So here is the way
9:29:21around one proceeding is lower than the
9:29:23current row. So let's see how SQL going
9:29:25to execute this. Let's say that we are
9:29:28currently at March. So this is the
9:29:30current row and we are saying between
9:29:32one proceeding. So that means one row
9:29:34before the current row. So the frame
9:29:37going to be like this and we have only
9:29:39two rows. So the value going to be the
9:29:41summarization of those two rows and it's
9:29:43going to be 40. So that means we are
9:29:45always targeting the rows before the
9:29:48current row. Okay. So now let's keep
9:29:49going with the other options in order to
9:29:51understand everything about the frame.
9:29:52So we redefine like this rows between
9:29:55unbounded preceding and the current row.
9:29:57So unbounded preceding going to be the
9:29:59first row in the table or in the window.
9:30:01So it's going to be static like this.
9:30:03It's going to be the first one January.
9:30:04And let's say that we are at this
9:30:07current row in March. So the window or
9:30:10the subset going to look like this.
9:30:12Those three rows and the total of that
9:30:14going to be 60. So now as SQL is
9:30:16proceeding to the next one, it's going
9:30:18to fix the first boundary. So it's going
9:30:20to be always pointing to January and the
9:30:22subset going to be a little bit bigger
9:30:24until we reach the last one. And with
9:30:27that we're going to have the subsets the
9:30:30whole rows. So with that we get really
9:30:32great flexibility on how to define the
9:30:34subset and how the subset is shifting
9:30:37through the window. Okay, so now we are
9:30:38just having fun. So we are just playing
9:30:40around with the boundaries. We don't
9:30:41have always to use the current row. So
9:30:43we can use for example here in this
9:30:45definition row is between one proceeding
9:30:48and one following. So we don't include
9:30:50at all the current row in the
9:30:51boundaries. So let's say again our
9:30:54current row going to be in March. So one
9:30:56preceding going to be February and one
9:30:59following going to be April. So with
9:31:01that our frame going to be the three
9:31:03rows. And let me get it like this. And
9:31:06the aggregation of this going to be
9:31:08around 45. So with that as you can see
9:31:10the boundary is going to be one
9:31:11proceeding and one following. So it
9:31:13should not be always the current row.
9:31:15All right. So now I think you already
9:31:16get it. What going to be the last
9:31:17option? We're going to have everything.
9:31:19So the definition of the frame going to
9:31:21be rows between unbounded preceding and
9:31:23unbounded following. What we're going to
9:31:25have over here. The unbounded preceding
9:31:27going to be January and the unbounded
9:31:29following going to be June. And now the
9:31:32frame going to be everything all the
9:31:34rows. And it doesn't matter where are we
9:31:37with the current row, it's going to be
9:31:38always a fixed subsets. So it's going to
9:31:41be always everything. So if we are over
9:31:43here or February or March, we're going
9:31:45to be considering all rows and the total
9:31:48sales of that going to be 135. So we
9:31:52will get the exact same results for
9:31:54everything for all rows. So with that I
9:31:57think it's not that complicated, right?
9:31:59We just have to provide the boundaries
9:32:01and then the calculation going to be
9:32:02depending on the frame on the subset of
9:32:05data. Okay guys, so now let's go back to
9:32:07SQL and start practicing in order to
9:32:09understand how the frame work. So let's
9:32:11go and define a window like this. So sum
9:32:14of sales and the window definition like
9:32:17this. We going to divide the data by
9:32:19order status and let's say we're going
9:32:22to sort it by order date. And let's
9:32:26define a frame like this. rows between
9:32:29current row and two following. Let's
9:32:33give it a name total sales. So let's go
9:32:36and execute it. So now let's look to the
9:32:38data. You see that SQL going to divide
9:32:40our results into two sections, two
9:32:42windows delivered and shipped. And you
9:32:45can see that the data is sorted by the
9:32:47order date. So as you can see over here
9:32:49for example in this status delivered we
9:32:52can see that 1 of January 10 and so on.
9:32:56And then the third part we have defined
9:32:57a frame in each window. So for example,
9:33:00let's take the first one. So this is the
9:33:02current row. So we say the frame is
9:33:04between the current row and the two
9:33:06following orders. So that means the
9:33:08scope going to be like this. So 10 + 20
9:33:1225 it's going to be 55. And now what is
9:33:14interesting as well to check here is the
9:33:16last record of each window. So now let's
9:33:19take this window over here and the last
9:33:21record going to be number seven. So this
9:33:23order and let's say this is the current
9:33:25record. So we set the frame between
9:33:28current record and the two following.
9:33:30But since it is the last record of this
9:33:33window, it will not go and consider the
9:33:35next two orders because those two orders
9:33:38are outside of the window and that's why
9:33:40we have here 30 and SQL doesn't go and
9:33:43summarize all those value. So we have it
9:33:4530 and there is nothing after that.
9:33:48That's why we will get 30. So as you can
9:33:51see the frame going to be calculated
9:33:52within one window. So it will not
9:33:55consider anything outside of the window.
9:33:57So this is how the frame works within
9:34:00partitions. So now I would like to show
9:34:01you as well a few stuff about the
9:34:03frames. We can use shortcuts but we can
9:34:05use them only with the proceeding. So
9:34:08for example let's say I'm going to
9:34:09change the definition like this to
9:34:12proceedings and current row. So let's go
9:34:15and execute it and we will get those
9:34:17results. So now if you want to check the
9:34:18results quickly, let's take for example
9:34:20this order over here and we are always
9:34:23summarizing the values of the two
9:34:25previous orders. So that means those
9:34:28three orders going to be involved in the
9:34:29frame and the output going to be 55. So
9:34:32now there is a shortcut for SQL but only
9:34:34for the proceeding where we can remove
9:34:37the range. So we can go and remove
9:34:39everything and we can leave it like this
9:34:42rows to proceeding and if you go and
9:34:44execute it we will get exact results. So
9:34:46this is a quick way or a shortcut on how
9:34:48to define a window but it only works
9:34:50with the proceeding. So for example, if
9:34:52I go over here and say for example
9:34:55unbounded it's going to work. So we will
9:34:57get the results between the unbounded
9:35:00proceeding and the current row. But if
9:35:02you go over here and you say you know
9:35:03what let's have the unbounded following
9:35:05SQL going to say there's an error. And
9:35:07the same thing if you remove the
9:35:08unbounded let's say for example one
9:35:10following SQL will not like it. So you
9:35:12can use the shortcut only with the
9:35:15proceeding. And one last thing about the
9:35:17frames it does there is a default frame.
9:35:20So if you don't use any frame and you
9:35:22use order by what can happen SQL going
9:35:24to use a default frame. So if you check
9:35:27the result you will notice that for this
9:35:29window over here those values are not
9:35:31like the whole values of the sales.
9:35:33There is like frame there is hidden
9:35:35frame and the default frame in SQL going
9:35:37to be like this rows between unbounded
9:35:41preceding and current row. So this is
9:35:44the default frame if you use order by.
9:35:47So now if you go and just execute it you
9:35:49will see that we will get the exact
9:35:51results. So be careful once you use
9:35:53order by with the aggregate functions
9:35:55there will be a hidden frame or a
9:35:57default frame like this between the
9:35:59unbounded proceeding and the current
9:36:01row. So that means there are three ways
9:36:02in order to do this scenario framework
9:36:04between unbounded proceding and current
9:36:06row. Either write it like this or you
9:36:08can go and have a shortcut like this.
9:36:11Let me just execute it. So we'll get the
9:36:13same result or just remove it
9:36:15completely. We will get as well the same
9:36:17results. Now again the hidden frame or
9:36:19the default frame is only working with
9:36:21the order by. So if you go for example
9:36:23here and remove the order by let's see
9:36:26the results. The whole window will be
9:36:28aggregated. So again let me just select
9:36:30it. So you can see that SQL going to
9:36:32consider all the rows in the
9:36:34aggregations and we will get the total
9:36:36sales for the whole window. So there
9:36:38will be no frame defined only it's going
9:36:40to be present once you use order by. All
9:36:42right friends so with the frame closed
9:36:44we have now covered all the components
9:36:46on how to define a window inside an
9:36:49overclo and with that we have covered
9:36:51everything about the syntax of the
9:36:54window functions.
9:36:59Okay guys, so now we're going to go and
9:37:00understand the rules or let's say the
9:37:02limitations of window functions. So
9:37:04let's learn what you are not allowed to
9:37:06do while using window functions. Okay,
9:37:08so the first rule that you are allowed
9:37:10to use the window function only in the
9:37:13select close and as well in the order by
9:37:16clause. So here we have again the same
9:37:17example where we finding the total sales
9:37:19by the order status. So as you can see
9:37:21we used the window function in the
9:37:23select clause and we didn't get any
9:37:25error right. So now we can go and use it
9:37:28as well in the order by. So let's say
9:37:30order by and let's go and copy
9:37:32everything but not the name in the order
9:37:35by. So if I go and execute this there
9:37:37will be no errors and SQL going to allow
9:37:39it. And as you can see the result didn't
9:37:41change. So let's go and sort it for
9:37:42example descending. So I'm going to
9:37:44write here descending and let's execute.
9:37:47Now we have the total sales with the
9:37:49highest values then the lowest values.
9:37:51So having this rule that we can use it
9:37:52only in select and order by that means
9:37:54we cannot use window functions in order
9:37:57to filter data. So let me show you for
9:38:00example instead of order by let's have
9:38:03clause where the total sales let's say
9:38:06bigger than 100. So let's go and execute
9:38:08this. And as you can see XQL going to
9:38:10say no you are not allowed to do that.
9:38:12You can do that only for select and
9:38:13order by. We are not allowed to use it
9:38:16for filtering data using the wear clause
9:38:18and as well you are not allowed to use
9:38:20it in the group by. So if I go and do a
9:38:22group by and as well remove the
9:38:24condition over here. So if you execute
9:38:26it you're going to get the same error.
9:38:28You are not allowed to use the window
9:38:29function in the group by. So only with
9:38:32the order by or as well in the select
9:38:35clause. Okay. So now to the second rule.
9:38:37You cannot use window functions inside
9:38:39another window function. So that means
9:38:41you cannot go and nest window functions
9:38:44together. Let me show you what I mean
9:38:45with that. So let's remove the group
9:38:47pie. Now everything should be working.
9:38:50Let's take and copy the whole window
9:38:51function over here and let's just nest
9:38:53it. So instead of sales, we're going to
9:38:56have now window function inside another
9:38:58window function. So as you can see this
9:39:00is the inner window function and the
9:39:03rest the outside is the outside window
9:39:06function. So if I go and execute this
9:39:08you will see that scale going to tell us
9:39:10you cannot use the window function in
9:39:12the context of another window function.
9:39:14So we cannot do nesting using window
9:39:17functions. So as you can see this is
9:39:18another limitation for those functions.
9:39:20All right moving to the third rule or
9:39:22let's say an info the window function
9:39:24will be executed after filtering the
9:39:27data with the work clause. Let's have an
9:39:29example. So okay so now let's say that I
9:39:31would like to have the same
9:39:31informations. the total sales for each
9:39:33status but only for two products 101 and
9:39:36102. So let's go and do that. We're
9:39:39going to use the wear clause and then
9:39:41we're going to say product ID in we're
9:39:43going to specify 101 and 102. So let's
9:39:47go and execute this. Now you can see we
9:39:49still have two partitions. So one for
9:39:51the delivered and one for the shipped
9:39:53but the total sales is reduced because
9:39:55we are only focusing on two products and
9:39:58we filtered the whole data sets. So how
9:40:00SQL works? First the workflow is going
9:40:02to be executed and then the window
9:40:05functions going to be calculated. So
9:40:07that means first filtering and then
9:40:08aggregations. Okay guys, now we're going
9:40:10to move to the last rule to the most
9:40:11interesting one and it says the
9:40:13following. You are allowed to use the
9:40:15window function together with the group
9:40:17by clause only if you use the same
9:40:19columns. So let me explain what do I
9:40:21mean but first some coffee.
9:40:26Let's have the following task and it
9:40:28says rank the customers based on their
9:40:30total sales. So now it sounds really
9:40:33easy but if you check it you have here
9:40:35two calculations. The first one you have
9:40:37to rank the customers and the second
9:40:39calculation is an aggregation. You have
9:40:42to find the total sales for each
9:40:43customers. Okay. So now I'm going to
9:40:45show you step by step how I usually
9:40:46solve those tasks. So for now let's
9:40:49check the total sales. It is an
9:40:51aggregation right? So we can use the sum
9:40:53function and this function is available
9:40:55in both group pi and as well in the
9:40:57window function. So for now I'm going to
9:40:59go with the group by and that's because
9:41:01the task is very simple. We don't have
9:41:03to show any other details. Right? So
9:41:05it's all about aggregations. So why not
9:41:07using the group by and now to the first
9:41:09part where we have to rank the
9:41:11customers. We cannot use the rank
9:41:13function with the group by right. Groupy
9:41:15uses only aggregations. So here we are
9:41:17forced to use the window function. So
9:41:19that means for the rank I'm going to use
9:41:21window function. For the total sales I'm
9:41:23going to use a group by. So now let's do
9:41:25it step by step. So first we have to
9:41:27find the total sales for each customer
9:41:30using group by. It's very simple. So I'm
9:41:32just going to remove all those stuff in
9:41:34our select statements. We need the
9:41:36customer ID and then we don't need a
9:41:39window function over here. And then
9:41:41after the from we're going to have a
9:41:43group by customer ID. So now I'm just
9:41:45grouping the customers and finding the
9:41:47sum of all sales. Let's go and execute
9:41:50this. So now as you can see in the
9:41:51results we have four customers and
9:41:53that's why we have four rows and as well
9:41:55we have the total sales. So let's say
9:41:57the half of the tasks is already solved.
9:42:00Right now what is missing that we need a
9:42:03rank. So let's go and build that. The
9:42:05second step we're going to use the rank
9:42:08function and we can define a window for
9:42:09that. So over and inside it will not
9:42:12partition the data at all because it's
9:42:13already like grouped up. So what we're
9:42:16going to do over order by the rank
9:42:18function always needs an order by don't
9:42:20worry about it we can talk about it
9:42:21later. So now we are ranking the data
9:42:23based on the total sales that means the
9:42:25sum of sales. So what we're going to do
9:42:27let's just go and copy this and put it
9:42:30after the order buy. And now we have to
9:42:31decide whether ascending or descending.
9:42:33It's going to be descending. So the
9:42:34highest sales first and then the lowest
9:42:36sales. So now as you can see we have now
9:42:38a rank customers and we have a window
9:42:41function now together with the group by.
9:42:44So now let's go and execute this and see
9:42:46whether SQL going to allow it. So let's
9:42:48run it and as you can see SQL runs it
9:42:50and we will get the rank for each
9:42:52customers. So the customer three has the
9:42:55highest total sale. Then the customer
9:42:57number one and the last one going to be
9:42:59customer number two with the lowest
9:43:00total sales. All right. So we solve the
9:43:03tasks. We have now ranked the customers
9:43:05based on their total sales. So as you
9:43:07can see SQL allows you to use window
9:43:09function together with the group by but
9:43:11only with one rule. Anything that you
9:43:13are using inside the window function
9:43:15should be part of the group I. So for
9:43:18example, we fulfilled the rule because
9:43:19we are using the sum of sales and the
9:43:21sum of sales is a part of the group I
9:43:24right. So now if I go I just break the
9:43:26rule by not using the sum just using the
9:43:28sales. So if I just remove the sum and
9:43:31use only the sales, SQL will not allow
9:43:33it because the sales is not part of the
9:43:36group I. So as you can see SQL is very
9:43:38strict with this. If you want to use
9:43:39everything in one query without using
9:43:42like subqueries and so on, you have to
9:43:44use the exact same columns. So for
9:43:46example, if I go over here instead of
9:43:48sales, I use the customer ID. So since
9:43:50the customer ID is a part of the group
9:43:53by, SQL can allows it. So be careful
9:43:55using window function together with the
9:43:57group by. As long as you are using the
9:43:59same columns, nothing going to go wrong
9:44:01and SQL going to allows it. Okay, so now
9:44:03I'm just going to go and fix this and
9:44:06let's run it. So now as you can see it's
9:44:08really easy if you follow those steps.
9:44:10First build the query using group by. So
9:44:12don't you think about the window
9:44:13function just build the group by and
9:44:15then the next step the last one you go
9:44:18and define and build the window
9:44:19function. So with that you can solve
9:44:21really nice analytical use cases with a
9:44:23simple one query without having you to
9:44:25build like some queries and so on. You
9:44:27can go and use group by together with
9:44:29the window functions. All right guys so
9:44:31those are the four rules for the SQL
9:44:33window functions.
9:44:37All right friends, so now let's have a
9:44:39quick recap about the SQL window
9:44:41functions. Let's start with the
9:44:42definition. It will go and perform
9:44:44calculations like aggregations on top of
9:44:47subset of data without losing the level
9:44:49of details. So that means we can do
9:44:51aggregations and at the same time we are
9:44:54not losing the details. Now, of course,
9:44:56there is a lot of similarity between the
9:44:57window function and the group by. But
9:44:59the main difference is that window
9:45:01functions are very powerful and dynamic
9:45:04compared to the group by. We have way
9:45:06more functions than the group by. Right?
9:45:08But now if you are doing data analyzes
9:45:10and you have an advanced use case, then
9:45:12you have to go and use window function.
9:45:14It's more suitable for complex and
9:45:16advanced data analyzes. But in the other
9:45:19hand if you have a simple question
9:45:21simple data analyzes then you can go and
9:45:23use the aggregate functions using the
9:45:25group by and of course you can go and
9:45:27use them in the same query in the same
9:45:29select you can go and mix the group by
9:45:32together with the window function with
9:45:34only one rule you have to use the same
9:45:36columns and of course the first step is
9:45:38to do the group by and then later you do
9:45:40the window function in the same query.
9:45:42And now to the next point about the
9:45:44window components we have two main
9:45:46components. The first one is the window
9:45:49function and the second part is the
9:45:51window definition using the over close.
9:45:53And inside the overlo we can define
9:45:55three things. If you want to divide the
9:45:57data to create windows you can use the
9:46:00partition by the second section we have
9:46:02the order by in order to sort your data.
9:46:05And the last part you can go and specify
9:46:07a subset of data like a frame within
9:46:10each window. Now let's move to the last
9:46:12part. We have rules for the SQL window
9:46:14functions. So the first thing is that if
9:46:16you have two window functions or
9:46:18multiple window functions, you cannot go
9:46:20and nest them together. You have to go
9:46:22and use multiple subqueries. The next
9:46:25point is that you can use the window
9:46:27function only in the select and the
9:46:29order by clause. So for example, you
9:46:32cannot use the window together with the
9:46:33wear clause in order to filter the data.
9:46:36Talking about filtering data, how SQL
9:46:38going to go and execute the window
9:46:40function? It's always after SQL filter
9:46:43the data. All right. So those are the
9:46:44basic stuff about the SQL window
9:46:46function. So with that we have learned
9:46:47the basics about the window functions in
9:46:49SQL. And next we're going to start
9:46:51talking about the functions. So the
9:46:53first group is the window aggregate
9:46:54functions. And here we're going to learn
9:46:56how to summarize our data for a specific
9:46:58group of rows. So let's
9:47:02go. Okay guys, let's say that in our
9:47:05data we have the following informations.
9:47:07We have the months and the sales. Now if
9:47:10you apply any aggregate functions in SQL
9:47:12what going to happen SQL going to go
9:47:14through all rows of the window or the
9:47:16entire data and start aggregating the
9:47:18data. So that means in the result in the
9:47:20output SQL going to give you one single
9:47:23aggregated value. SQL going to go and
9:47:26summarize all those values and in the
9:47:28output you're going to find for example
9:47:29here the total sales it's going to be
9:47:32175 or you can use the average or count
9:47:34the data and so on. So the aggregate
9:47:36functions going to deliver at the end
9:47:39one aggregated value for a window or for
9:47:42the entire data. Okay. So now let's have
9:47:43a quick overview of the syntax of all
9:47:45aggregate functions. Most of them follow
9:47:48the same rule. So first as usual we have
9:47:50to define the function name. And in this
9:47:52example we have the average. Then to the
9:47:53next part we have to define inside it as
9:47:56well the expression. We cannot leave it
9:47:58empty. So here we are using the sales
9:48:00and the second rule for all functions
9:48:02beside the count. The data type of this
9:48:05field should be a number. And this of
9:48:07course makes sense, right? So we cannot
9:48:09find the average of the first name of
9:48:11customers or something like that. So we
9:48:13have to define a number. Then next we
9:48:14have to define the frame. So we have the
9:48:16partition pi and it is optional. So you
9:48:19could use it or leave it depends. And
9:48:21then the next one we have the order by.
9:48:22It is as well optional. It is not a must
9:48:25or required. So you could use it or
9:48:27leave it. That mean the whole definition
9:48:29of the window could be empty for the
9:48:31aggregate functions. Let's have a look
9:48:32to all functions. So we have the count,
9:48:35sum, average, min, max. And as you can
9:48:37see, only the count accepts all data
9:48:39types as an expression or arguments. All
9:48:42others require you to have a number as a
9:48:44data type. And for all functions, the
9:48:46partition by is optional. The same for
9:48:48order by and frame. So everything is
9:48:50optional over here. So now what we're
9:48:52going to do with that, we're going to go
9:48:53and deep dive into each of those
9:48:54functions in order to understand how
9:48:56they work, what are the use cases, and
9:48:59of course, we're going to practice in
9:49:00SQL. So we're going to start with the
9:49:02first one with the function
9:49:07count. Okay. So what is the count
9:49:09function? It's really simple. It's going
9:49:10to return the number of rows within each
9:49:13window. So it's going to help you to
9:49:14understand how many rows do you have
9:49:16within each subset of data. So now let's
9:49:18go and understand how SQL works with
9:49:21this function. All right guys, so now we
9:49:22have again this very simple example for
9:49:24the orders and we have the following
9:49:26informations. We have the products and
9:49:27sales and now we want to solve very
9:49:29simple task. How many orders do we have
9:49:32within each products? So in order to
9:49:34solve it, we can use the function count
9:49:36like the following. So we can say count
9:49:38and then we pass for it an argument or
9:49:41expression the star. So with that we are
9:49:43telling SQL go and count how many rows
9:49:45do we have in our table. But we have a
9:49:48window definition like this over
9:49:50partition by products. So now what SQL
9:49:52going to do? We're going to go and
9:49:53divide the data sets into two
9:49:55partitions. We're going to have one
9:49:56partition for the caps and another one
9:49:59for the gloves. So with that we have
9:50:01prepared our data into windows and we
9:50:03are ready to do aggregations. So how
9:50:04many rows do we have within each window?
9:50:06It's going to be three. So for this
9:50:07window it's going to be three rows and
9:50:10as well for the next window we have as
9:50:11well three rows. So we're going to have
9:50:13three three and three. It's very simple
9:50:16right guys? We are just finding the
9:50:18number of rows within each window. But
9:50:20now with the aggregate functions we have
9:50:21to be very careful with the null values
9:50:24for the count star. As you can see over
9:50:26here we are not specifying anything
9:50:28about the sales. So we are just saying
9:50:30find me the number of rows. So that
9:50:32means SQL will just count the nulls as
9:50:34one row. So that means if we are using
9:50:36the star as an argument for the function
9:50:38count the null will not affect anything.
9:50:41So whether we have nulls or nots we are
9:50:43just counting how many rows do we have
9:50:45inside our data. But in some scenarios,
9:50:47we should be ignoring the nulls in our
9:50:49account. For example, let's say that I
9:50:51would like to count how many sales do we
9:50:53have within each product. That means if
9:50:56we have nulls, it should not be counted.
9:50:58So now in order to achieve this task,
9:51:00what we're going to do, we're going to
9:51:02use instead of a star over here, we're
9:51:04going to have the filled sales. So now
9:51:06with this, we are telling SQL, don't
9:51:08just count blindly how many rows do we
9:51:10have within each window. You should be
9:51:12very careful with the values. Find how
9:51:15many cells do we have within each
9:51:17window. So now let's see what's going to
9:51:19happen. For the first window we have
9:51:21three cells. So we have three values. So
9:51:23the number of rows is correct. But for
9:51:25the next one, how many cells do we have?
9:51:27We have two. So we have this sale and
9:51:30then the 70. But the last one is null.
9:51:32So it will not be counted. It will be
9:51:34ignored. That's why we're going to get
9:51:36in the output the value two. We have two
9:51:39sales. So as you can see the result did
9:51:41change and we are now more sensitive to
9:51:43the null values. So be careful what you
9:51:45are specifying for the count. If you are
9:51:47using a column name like this it will
9:51:49ignore the nulls. But if you have a star
9:51:52it just going to go and find how many
9:51:54rows do we have within each partition.
9:51:56Okay. So now if you go and compare the
9:51:57results side by side you can see that if
9:51:59you specify a column within the count
9:52:01function it's going to be sensitive with
9:52:03the nulls. So it's going to ignore it
9:52:05and will not use it within the
9:52:07aggregations. That's why we have here
9:52:08only two rows. But if you go and use the
9:52:11star within the count function, what
9:52:13going to happen? SQL just going to go
9:52:15and count it. So we're going to find the
9:52:17number of rows that we have inside our
9:52:18table. And there is one more way in
9:52:20order to do the same thing here on the
9:52:22left side. You can use instead of star
9:52:24you can use one. So you might find it
9:52:26somewhere that people are using count
9:52:29one and then the same window function
9:52:31and we will get exactly the same result.
9:52:33So the nulls will be counted and will
9:52:35not be ignored. So now you might ask me
9:52:37which one should I use the one or the
9:52:38star? Well, I would say it doesn't
9:52:41matter right we are getting the same
9:52:42results and if you are thinking about
9:52:43the performance I hardly find any
9:52:46differences between them so you can go
9:52:47and try both of them and stick with the
9:52:49one that is giving you like more better
9:52:52performance. Now we have special case
9:52:53for the count function compared to all
9:52:55other aggregate functions it allows any
9:52:57data type. So that means we can use
9:53:00numbers we can use characters dates and
9:53:02so on. So that means we can go and
9:53:03specify something like the product for
9:53:05the count instead of sales. So we can go
9:53:07over here and say product and it's going
9:53:09to go and count how many rows do we have
9:53:10for the product. So it's going to be
9:53:12three over here. And since here we don't
9:53:14have any nulls, it's going to go and
9:53:16count it like this. So we have three
9:53:17rows and be careful here. We are not
9:53:19counting the unique rows. We are just
9:53:21counting the rows that we have inside
9:53:23our data. So this will not be counted as
9:53:25one and this as well would not be one.
9:53:28So we have three times the caps. That's
9:53:30why we have here three. Okay. Okay. So
9:53:31now we have this very simple example.
9:53:33Find the total number of orders. This is
9:53:35very simple task in order to find how
9:53:37many rows, how many records do we have
9:53:40inside the table orders. So let's go and
9:53:43solve it. So let's start by selecting
9:53:45just star from the table orders without
9:53:48anything like this. So as you can see we
9:53:50have 10 orders. It's very simple. It's
9:53:52very easy as well. But now let's say
9:53:54that you have thousands or millions of
9:53:56rows. You cannot do it like this by just
9:53:57checking the rows. What you're going to
9:53:59do? We're going to go and use the
9:54:00function count. So we can go over here
9:54:02and say counts star and then let's give
9:54:05it a name total orders. So let's go and
9:54:09execute it. So now as you can see we got
9:54:10only one record, one value. We don't see
9:54:13any other details. We got the 10 orders.
9:54:16So this is the total number of orders.
9:54:18This is very helpful in order to
9:54:19understand the content of your data. So
9:54:21this we call it overall analyzes or
9:54:24let's say having the big numbers about
9:54:26your business. For example, how many
9:54:28orders do we have? how many customers,
9:54:30products, employees and so on. So having
9:54:32those big numbers going to help us to
9:54:34track our business to understand how
9:54:36well we are doing with the orders and
9:54:39with the customers and so on. So this is
9:54:41the basics of reporting. Now let's go
9:54:43and extend our task by saying provide
9:54:46details such as the order ID and the
9:54:48order dates. So let's go and do that. So
9:54:51select order ID, order dates. And now of
9:54:54course we cannot do it like this. So let
9:54:56me just execute it. we will get an error
9:54:58because here we have different level of
9:55:00details in our select. So in order to
9:55:02solve this what we going to do we're
9:55:04going to use the over clause and with
9:55:07that we are telling SQL this is a window
9:55:09function. So now let's go and execute
9:55:11it. So with that you can see with that
9:55:12we have solved the task we have details
9:55:14we have the order ID order dates. So
9:55:16this is the highest level of details
9:55:19since we have the order ID and as well
9:55:21we have the highest level of
9:55:22aggregations. we have the total number
9:55:24of orders in the entire table orders. So
9:55:27now let's keep going and add more stuff
9:55:29to our task. Let's say that we want to
9:55:31find the total number of orders but for
9:55:34each customers. So that means this time
9:55:36we have to go and divide our data by the
9:55:39customers. So let's go and do that.
9:55:41We're going to use as well a window
9:55:42function. So count star over we have to
9:55:45divide the data using partition by and
9:55:48we're going to use the field customer
9:55:50ID. So let's call it orders by customers
9:55:54and I would like to see as well the
9:55:55customer informations in the query.
9:55:57That's why I'm going to go and add it.
9:55:59All right. So that's all. Let's go and
9:56:00execute it. Now as we learned before
9:56:02that SQL first going to go and divide
9:56:04the data. So that means we have four
9:56:06customers. We're going to get four
9:56:07windows. The first window going to be
9:56:09for the customer ID number one. And as
9:56:11you can see we have three rows. That's
9:56:13why we have here three orders. And the
9:56:15same thing for the customer two. We have
9:56:17three orders. customer three three
9:56:19orders but only the last customer the
9:56:22customer ID number four we have only one
9:56:24row and one order. So now if you go and
9:56:25look to the total orders and the orders
9:56:27by customers you can see now we are not
9:56:29doing the overall analyzes we are doing
9:56:32like comparison between different
9:56:34categories and of course in this example
9:56:35the category is the customers and with
9:56:38that we can understand as well the
9:56:39behavior of our customers. So you can
9:56:42see that we have three customers that
9:56:44has exactly the same amount of orders.
9:56:47So they are very similar but we have one
9:56:49extreme which is the customer ID number
9:56:51four. This customer has only one order.
9:56:54So this is the only customer that has
9:56:56different behavior than all other
9:56:58customers. So you see with very simple
9:57:00query we are able now to analyze our
9:57:02business and understand the behavior of
9:57:04our customers. So if you divide the data
9:57:07by partition by and using count you can
9:57:10go and now compare stuff together. All
9:57:12right. So now let's keep moving. Next
9:57:13we're going to understand the special
9:57:15cases that we have with the function
9:57:17count. So now we have this very simple
9:57:19task. It says find the total number of
9:57:21customers and additionally we have to
9:57:23provide all customers details. So I
9:57:25think it's very easy to solve. What
9:57:27we're going to do we're going to go and
9:57:28select star since we need all details
9:57:30from customers from sales customers. So
9:57:34let's just have a look. So we have five
9:57:36customers and the function is count star
9:57:39over and we don't have to divide the
9:57:41data since we have to find the total
9:57:43number of customers for the entire table
9:57:45and it's going to be total customers. So
9:57:47nothing new that's it we have five
9:57:50customers and now as we learned before
9:57:52if you are passing the star to the count
9:57:54function what you are telling to escale
9:57:56is that just go and count how many rows
9:57:58do we have inside the table customers.
9:58:00So SQL just going to go and start
9:58:02counting and going to say we have five
9:58:04customers, five rows. So it doesn't
9:58:06matter whether we have nulls inside our
9:58:08data like in the last name or the score.
9:58:10It's just going to count the number of
9:58:12rows. So now let's say that we have the
9:58:13following task. It's going to say find
9:58:16the total number of scores for
9:58:18customers. So what do we need with this
9:58:19task is to find out how many scores
9:58:21inside our data. So as you can see we
9:58:23have around four scores but the last
9:58:26customer doesn't have any score. So we
9:58:28have it as a null. So the result should
9:58:30be four. We cannot go now and use the
9:58:32star for it because we're going to get
9:58:34five. We have to go and count the
9:58:35scores. So let's see how we're going to
9:58:37do that. We're going to count as well.
9:58:39But this time the score and the
9:58:42definition of the window going to be
9:58:43empty. So total scores and let's go and
9:58:46execute this. So now we can see in the
9:58:48results we got four scores which is very
9:58:50correct because SQL did ignore the null
9:58:53and SQL now focusing only on one column.
9:58:56So focusing on those values the nulls
9:58:58will not be counted. This is really
9:59:00great in order to check the quality of
9:59:01your data. So let's say that you are not
9:59:03expecting any nulls inside your data. So
9:59:06instead of going manually through the
9:59:07whole records what you can do you can go
9:59:09and find the total number of customers
9:59:11like this and then you can go and count
9:59:14the total number of scores and you can
9:59:16see there is a difference. So by just
9:59:18checking the data I can say you know
9:59:20what we have one null without checking
9:59:23every record in our data. So with that
9:59:25we can check the quality of our data and
9:59:27understand very quickly how many nulls
9:59:30do we have in the field score and you
9:59:32can do the same stuff for example for
9:59:33the first name show it to you. So I'm
9:59:36just going to go and copy this and let's
9:59:37say first name or let's say country
9:59:40actually. So I will go with the country.
9:59:42So let's go with the country total
9:59:45countries. So let's go and execute this.
9:59:48So now if you check the result you can
9:59:49see we have five rows with the
9:59:51countries. So SQL going to go and focus
9:59:53on the countries and it will not find
9:59:54any nulls. So we have here complete
9:59:57data. We don't have any nulls because
9:59:59the total number of customers is equal
10:00:01to the total number of values within the
10:00:03country. And I can immediately find okay
10:00:05the data quality of the country is very
10:00:08good. All right. So now one more thing
10:00:09about the count function that we have
10:00:10learned before. We can use either star
10:00:13or one in order to count how many rows
10:00:15do we have. So let's just try it. I'm
10:00:17just going to go and duplicate it. And
10:00:19instead of having a star, let's have a
10:00:21one. Just going to give it a name here.
10:00:23It's going to be one and you are star.
10:00:25So let's go and execute it. So now if
10:00:27you check the output, we got exactly
10:00:30identical results. So there is no
10:00:32difference between those two queries.
10:00:34It's up to you. You can try it and check
10:00:36the performance. I usually go with the
10:00:38star instead of one. Okay. So now we're
10:00:39going to talk about a very important use
10:00:41case for the SQL window function count
10:00:43that I frequently use in my real
10:00:45projects. The data that we use for data
10:00:47analyzes has usually bad data quality.
10:00:51And if we don't find those data quality
10:00:53issues and we don't clean it before
10:00:54doing the analyzes, what going to
10:00:56happen? We're going to deliver bad
10:00:58results, bad analyzes which going to
10:01:01lead to bad decisions. And one very
10:01:03common data quality issue that you might
10:01:05encounter in your project or on your
10:01:06data is that having duplicates.
10:01:08Duplicates are really bad for doing data
10:01:10analyszis. So now in order to discover
10:01:13or let's say identify the duplicates in
10:01:15our data, we can go and use the SQL
10:01:18window function count. So now let's go
10:01:20and have some examples. Okay. So now the
10:01:22task says check whether the table orders
10:01:24contains any duplicate rows. So how we
10:01:27going to do that? By checking now the
10:01:28table orders over here. We can see that
10:01:31there are many orders. But how to find
10:01:33out the duplicates? Well, the first step
10:01:35is to understand what is the primary key
10:01:38of the table orders. So what we usually
10:01:40do we go and check the data model if
10:01:42there is one. So for example for this
10:01:44course we have the following data model
10:01:46and we can see that it is defined that
10:01:48the order ID is the primary key for the
10:01:50orders. The product ID is primary key
10:01:52for the products. So that means for our
10:01:55table the orders we have the order ID as
10:01:57the primary key and it should be unique.
10:01:59It should not contain any duplicates. So
10:02:01now let's go to our data and check the
10:02:04order ID. By just looking at the data
10:02:06you can see that we don't have any
10:02:08duplicates. Rightes all of them are
10:02:09unique. So we have 1 2 3 4 and so on.
10:02:12But of course in real projects you
10:02:13cannot do it like this. You have to go
10:02:15and build a query in order to find out
10:02:17whether the primary key is unique. But
10:02:20now you might say the primary keys are
10:02:22usually unique because we can define it
10:02:23in the DDL in the rules of building the
10:02:26table. Well that's true. If you have it
10:02:29like this then you don't have to find
10:02:31any duplicates. But usually in data
10:02:33analyzes we export a lot of files and a
10:02:35lot of data inside an extra database and
10:02:37we don't build such a rules. So now in
10:02:39order to check the quality of the
10:02:41primary keys that you get from the
10:02:42source we can use the count function. So
10:02:45let's go and build it. I'm just going to
10:02:46select the order ID first as a detail.
10:02:49And now we're going to do the following.
10:02:50So count and then star. And let's go and
10:02:53define the window. So it's going to be
10:02:56partition by and here the field going to
10:02:58be the primary key. So the order ID I'm
10:03:01checking now the quality of this field.
10:03:04This should not contain any duplicates.
10:03:06And now we're going to go and give it a
10:03:07name check primary key. So now my
10:03:10expectation is that the result of this
10:03:13should be at maximum one. That means we
10:03:15have one row for each primary key. And
10:03:17that means as well it is unique. So if
10:03:19we get anything more than one then it
10:03:21means we have duplicates. Let's go and
10:03:23run the query. And as you can see in the
10:03:25results we get for each primary key one.
10:03:28So that's great. That means we don't
10:03:29have any duplicates inside our data and
10:03:32the primary key is unique. So that means
10:03:34the table orders is clean and we don't
10:03:36have any duplicates inside it. Now let's
10:03:39check our database. We have here another
10:03:41table called orders archive. Let's go
10:03:44and check the table. So first I'm just
10:03:46going to go and select the data. So
10:03:48select from orders archive. So sales do
10:03:51orders archive. Let's check the results.
10:03:55And here we can see that we have exactly
10:03:57the same structure as the table orders.
10:04:00So now let's go and check whether the
10:04:01data quality is well clean. So now what
10:04:03we're going to do, we're going to use
10:04:04exactly the same query as before, but
10:04:07instead of using the table orders, we're
10:04:09going to take the orders archive. So
10:04:11that's it. Let's go and execute it. So
10:04:14now by checking the data, you can see
10:04:15that we don't have everywhere one.
10:04:17Sometimes we have two rows for the same
10:04:20primary key, which is really bad. So we
10:04:22have here for the order ID four we have
10:04:24two orders with the same order ID and as
10:04:27well for this order id six we have three
10:04:30orders that means those stuff are
10:04:32duplicates and they are against our data
10:04:35model. So now what else we can do is
10:04:36that to generate a list specifically for
10:04:38the data quality issue where we have
10:04:40duplicates. So anything that has one we
10:04:43are not interested in it. In order to do
10:04:45that we're going to use the subquery. So
10:04:48let's say select star from and then
10:04:51we're going to use the first query as a
10:04:53subquery and we're going to say in our
10:04:55filter where the check primary key is
10:04:58higher than one. So that means I need
10:05:00only the order ids where we have
10:05:02duplicates. So let's go and execute
10:05:04this. Now I have a list with the primary
10:05:06keys where we have duplicates. So we
10:05:08have the order ID 4 and as well the
10:05:10order ID six. So guys, as you can see,
10:05:12the window count function is wonderful
10:05:14in order to find data quality issues
10:05:16like the duplicates. All right guys, so
10:05:18those are the four most important use
10:05:20cases in the SQL window function count.
10:05:22So the first one we can use it in order
10:05:24to do overall analyzes or we can use it
10:05:27in order to do category analyzes like we
10:05:29have done the analyzes on the customer
10:05:31behavior or another use case we can use
10:05:33it in order to check the nulls inside
10:05:35our data. And the last use case we can
10:05:38use it in order to identify or discover
10:05:40the data quality issue duplicates in our
10:05:43data. So now let's go and check the next
10:05:45function. We have the
10:05:46[Music]
10:05:49sum. All right. So now let's understand
10:05:52what is the sum function. It's very
10:05:54simple. It's going to return the sum of
10:05:56all values within each window. So now
10:05:58let's go and understand how SQL works
10:06:00with this function. All right. So this
10:06:02is very easy and we are using the same
10:06:04simple example and now we would like to
10:06:06find the total sales for each products.
10:06:08So we can define like this sum of sales
10:06:11since we are finding the total sales and
10:06:13then we define the window like this over
10:06:16partition by products. So as we learned
10:06:18SQL going to go first and divide our
10:06:20data into two windows. So one window for
10:06:22the caps and another window for the
10:06:25gloves right. So now after SQL define
10:06:27the windows it's going to go and starts
10:06:28aggregating the data. So the sum of
10:06:31sales that means for the first window we
10:06:33have the three sales and it's going to
10:06:34go and just simply summarize all those
10:06:37values. So we are adding 20 + 10 + 5 and
10:06:40we will get the result 35. So in the
10:06:43outputs we will get everywhere 35. So
10:06:47that's it for the first window and as
10:06:48you can see SQL going to go aggregate
10:06:50the data within each window separately.
10:06:53So that means as we are aggregating the
10:06:54data for the caps will not check
10:06:57anything with the gloves. So they are
10:06:59completely separated. So now it's going
10:07:01to go for the next window. And here we
10:07:03have two values and a null. So again
10:07:05here the null will just be ignored. So
10:07:07what we going to have? We're going to
10:07:08have 30 + 70 and the total sales for
10:07:12that going to be 100. So as you can see
10:07:14it is very simple, right? So 100 100 and
10:07:18so guys that's it. It's really simple.
10:07:19We don't have here like a lot of special
10:07:21cases like the count function. It's only
10:07:24that it ignores the null in the
10:07:25calculation and as well the requirement
10:07:28here it allows only integers or let's
10:07:30say numbers. So we cannot go and say sum
10:07:33the products since the products are not
10:07:35numbers they are characters. So you can
10:07:37only use numbers for the sum function.
10:07:40Let's go now and have some tasks and
10:07:42some use cases in order to practice in
10:07:43SQL. find the total sales across all
10:07:46orders and as well find the total sales
10:07:48for each product and additionally we
10:07:50have to provide some details like the
10:07:51order ID and the order dates. So let's
10:07:54go and do that. Select order ID, order
10:07:56date and let's get as well the sales.
10:07:58And now we have to find the total sales
10:08:00across all orders. That means we're
10:08:02going to use the window function sum
10:08:04sales and the definition of the window
10:08:06going to be empty since we don't have to
10:08:08divide the data. So that's it. total
10:08:11sales and we have to select the table
10:08:13sales orders. So that's it. Let's go and
10:08:16execute it. So with that as you can see
10:08:18we got all the details that we need and
10:08:20as well the total sales the
10:08:22summarization of all those sales in one
10:08:25field. So with that we have our overall
10:08:28analyzes one big number for our
10:08:29reporting. We know how much sales we did
10:08:32made in the entire business. So now
10:08:34let's go for the next task. It says
10:08:36total sales for each product. I think
10:08:38you know already what we're going to do.
10:08:39So sum of sales and we're going to do it
10:08:42like this. Partition
10:08:44by product ID. So that's it. We're going
10:08:48to call it sales by products. And with
10:08:51that we are dividing the data by the
10:08:53product. So let's go and execute it. So
10:08:55as you can see we don't have the product
10:08:56information. So let's go and add the
10:08:58product ID in the query just in order to
10:09:01analyze the results. So we can see from
10:09:03the data that the winner is the product
10:09:05ID 101. So as you can see we have here
10:09:08the highest sales if you compare it with
10:09:10the other products and the lowest one
10:09:12going to be the products ID 105. So as
10:09:15you can see we can use the window
10:09:16function sum together with the partition
10:09:18by in order to compare stuff to do
10:09:21comparison between the products in order
10:09:22to understand the performance for
10:09:24example of the products. So it's really
10:09:26great analyzes for the performance. All
10:09:28right. Now we're going to move to very
10:09:30interesting use case for the aggregate
10:09:31functions not only for the sum but as
10:09:33well for the others. It is the
10:09:35comparison analyzes. Okay. Okay, so
10:09:37let's understand quickly what is the
10:09:38comparison use cases. So it's going to
10:09:40go and compare the current value. For
10:09:42example, let's say we are currently at
10:09:44the month of March and the sales is 30.
10:09:47So we're going to compare this value,
10:09:49the current sales with an aggregated
10:09:51value. For example, let's say the total
10:09:53sales using the sum function. So what
10:09:55happen if you compare the current value
10:09:57with the total sales? You are comparing
10:09:59here or doing analyszis called part to
10:10:01whole analyszis where it's going to help
10:10:03us to understand how important was the
10:10:05sales in this month compared to the
10:10:08total sales or we can go and compare it
10:10:10to the best months to the highest value.
10:10:12For example, the highest value is June
10:10:14and we can go and compare this month
10:10:16with the best months of the year or to
10:10:18the lowest month in the year or we can
10:10:20go and compare the sales of the current
10:10:22month with the average in order to
10:10:24understand are we above the typical
10:10:26sales or below the average. And this is
10:10:28very important analysis in order to
10:10:30study and understand the performance of
10:10:33the current data. All right, let's have
10:10:34an example in order to understand the
10:10:36use case. Find the percentage
10:10:38contribution of each product sales to
10:10:40the total sales. So let's go and solve
10:10:42it step by step. What we're going to do,
10:10:44we're going to go and let's select the
10:10:46order ID and as well let's take the
10:10:49product ID and the sales just like this
10:10:53from sales orders. So let's go and
10:10:56execute it. Okay. Okay. So now as you
10:10:57can see in the results we got the first
10:10:59part of the equation. We have the sales.
10:11:01So nothing like a crazy over here. Now
10:11:04we need the total sales over all data.
10:11:07So what we're going to do we're going to
10:11:09have the sum of sales and the definition
10:11:12going to be empty. So this is the total
10:11:15sales. Let's go and execute it. So now
10:11:18we have everything for the equation. We
10:11:19have the sales and as well the total
10:11:21sales and that is enough in order to
10:11:23find the percentage of the contribution.
10:11:25So the calculation for that is going to
10:11:27be very simple. We're going to divide
10:11:28the sales by the total sales. So it's
10:11:32really simple. Let's go and do that.
10:11:33It's going to be the sales divided by
10:11:36the total sales. So we're going to go
10:11:37and copy the whole window function over
10:11:40here. And then we're going to multiply
10:11:41it with 100. So that's it. Let's go and
10:11:44execute it. So now you notice that in
10:11:46the output we got zeros. This is because
10:11:48of the data type. So now if we go to our
10:11:50table over here on the left side you can
10:11:52see that the orders has the data type of
10:11:56integer. So if you divide integers you
10:11:58will not get a float or decimal number.
10:12:00You have to go and change the data type.
10:12:02So now what we're going to do we're
10:12:03going to go and change the data type for
10:12:05one of them. So it's enough for the
10:12:06sales over here. So we're going to use
10:12:08the following statement. So cast sales
10:12:11as floats. So that's it. I'm just
10:12:13converting the integer to floats. So
10:12:15that's it. Let me just give it a name.
10:12:17So it's going to be percentage of total.
10:12:20So that's it. Let's go and execute it.
10:12:23So now in the output, you can see we got
10:12:24now the percentage of the total or let's
10:12:27say percentage of contribution. So now
10:12:29what we're going to do with that, we're
10:12:30going to go and round those numbers
10:12:31because we have a lot of decimals. In
10:12:33order to do that, we're going to use the
10:12:35round function like this. Then we're
10:12:37going to have two decimals. And let's go
10:12:40and execute it. So now, as you can see,
10:12:42it is really easier to read because we
10:12:44have only two decimals. And we can find
10:12:46immediately that the order rate is the
10:12:48highest contributor to the total. So
10:12:51this is what we call part to whole
10:12:52analyszis where we find the percentage
10:12:55of total. It is very common analyzes in
10:12:57order to understand the performance of
10:13:00each order compared to the total. So
10:13:02this is an example how the window
10:13:04function is helping us here to compare
10:13:06the current value with an aggregated
10:13:08value. All right everyone. So that's all
10:13:10for the window function sum. Next we're
10:13:12going to talk about the average
10:13:13function.
10:13:17All right. So now let's understand what
10:13:19is an average function. As the name
10:13:21says, it's going to find the average of
10:13:23values within each window. So now let's
10:13:25go and understand how SQL works with the
10:13:28average. All right. So now back to our
10:13:29very simple example and the task says
10:13:31find the average sales for each product.
10:13:33So it's really easy. We're going to use
10:13:35the average then pass to it the column
10:13:37sales and we define the window like this
10:13:39partition by products. So the first
10:13:41thing that SQL going to go is to define
10:13:43the window. So it's going to divide our
10:13:45data into two partitions. One for the
10:13:47caps and one for the gloves. And now I
10:13:49hope that everyone knows how to
10:13:50calculate the average. So as you know
10:13:52that it's going to go and summarize all
10:13:54the values and divide it by the number
10:13:56of rows. So it's going to go and
10:13:57summarize 20 + 10 + 5 and divide it on
10:14:01three rows and the output going to be
10:14:0311. So we're going to get it for each
10:14:05row. So as you can see SQL just ignored
10:14:08everything in the next window. We are
10:14:10focusing only on the caps. Now it's
10:14:12going to go to the second window and
10:14:14start doing the same aggregations. But
10:14:15here we have the special case of null.
10:14:17So the null is going to be ignored in
10:14:19the calculations and we're going to have
10:14:20it like this. It's going to say you know
10:14:22what 30 + 70 and we are just including
10:14:25two rows. So it's going to be divided by
10:14:27two and the average going to be 50. So
10:14:29we will get the result 50 for each row
10:14:32and we are completely ignoring the
10:14:34nulls. But now we might be in scenario
10:14:36where your users understand the business
10:14:37like this. If we find a null in the
10:14:40sales it means a zero. So there is no
10:14:42sales and it is actually a zero. But we
10:14:44store it in the database as a null. So
10:14:46that means the average that we have
10:14:47provided is not really correct. We have
10:14:49to divide by three. So that means first
10:14:52we have to handle the nulls before doing
10:14:54the aggregations before finding the
10:14:56average. Now we're going to have a whole
10:14:58chapter on how to handle nulls in SQL.
10:15:00What are the different functions? But
10:15:02for now we're going to go with the
10:15:03functions qualisk. Okay. So now what
10:15:05we're going to do, we will not use the
10:15:06sales as it is. First we're going to
10:15:08handle the nulls. So that means we're
10:15:10going to use the qualisk sales and
10:15:13replace it with zeros. So as you can see
10:15:15we are not using immediately the sales
10:15:17we are handling it first and then we're
10:15:19going to find the average. So SQL going
10:15:21to go over here and if it finds any null
10:15:24going to go and replace it with zero and
10:15:26that's going to have then an effect on
10:15:28our average over here. So it going to be
10:15:3030 + 7 + 70 but now plus 0. And now we
10:15:33have three rows. So instead of dividing
10:15:36by two, it's going to go and divide it
10:15:37by three and the total result going to
10:15:39be like this 33. So that means we're
10:15:41going to have in the output 33 for each
10:15:44row and with that we are now fulfilling
10:15:47the expectation from the business. If
10:15:49you have a null it's going to be handled
10:15:50as zero and the result going to be more
10:15:52accurate. You see right it is very
10:15:54tricky. If you are doing data analyszis
10:15:56and aggregations be very careful with
10:15:58the nulls. understand them, understand
10:16:01what they mean for the business, handle
10:16:03them correctly in order to get correct
10:16:05results in your analysis. So now let's
10:16:08go back in order to practice SQL using
10:16:10some tasks and use cases. Okay, so let's
10:16:13start with the basics. We have the
10:16:14following task. Find the average sales
10:16:16across all orders and as well find the
10:16:18average sales for each product. And
10:16:20don't forget the details. So now let's
10:16:22go and solve it step by step. So select
10:16:24order ID, order date, and let's get the
10:16:28sales as well. And let's go and find the
10:16:30average sales. So it's going to be a
10:16:31window function. And we have the sales
10:16:33inside it. The usual stuff. The window
10:16:35going to be empty. So average sales,
10:16:38we're going to call it the table going
10:16:40to be sales orders. So that's it. Let's
10:16:44go and execute it. Oh, we have to select
10:16:46everything of course. So what SQL did in
10:16:48the output, it going to go and summarize
10:16:50all those values and then divide it by
10:16:5210. So with that we have the average
10:16:54sales of 38. Very easy. So this is again
10:16:57what we call an overall analyzis. Let's
10:16:59move to the next one. Find the average
10:17:01sales for each products. So again we're
10:17:03going to go and build the window
10:17:05function like this. Average sales over
10:17:09and we're going to divide it by product
10:17:11ID. And we're going to call it average
10:17:13sales by products. And we're going to go
10:17:16and add the product ID in the query. So
10:17:19that's it. Let's go and execute. And we
10:17:22missed something here. So it is the
10:17:24partition by going to execute again. So
10:17:27with that we have the following data. So
10:17:28now SQL going to go and divide the data.
10:17:30So for example for this products we have
10:17:32those four orders. So what going to
10:17:34happen is still going to go and
10:17:35summarize the four values and then
10:17:38divide it by four. That's why we have
10:17:39here 35. The same thing for the next
10:17:42order. It's going to divide it by three.
10:17:44And the last one is just going to divide
10:17:46it by one. That's why we have 60. So as
10:17:48you can see the aggregation can done
10:17:50separately for each window and this is
10:17:52as well very nice way in order to
10:17:54compare the averages between the
10:17:56different products. Okay. So now let's
10:17:58have an example in order to learn how to
10:18:00deal with the nulls. Let's say that we
10:18:01have the following task. Find the
10:18:03average scores of customers and show as
10:18:06well additional informations like the
10:18:08customer ID and the last name. So let's
10:18:10go and solve this. We are now targeting
10:18:13the table customers. So let's just
10:18:16select it first.
10:18:18like this. And now let's go and include
10:18:20the customer ID and the last name. And
10:18:24let's have as well the score. But this
10:18:26time we're going to go and find the
10:18:27average score. So it's going to be the
10:18:29average score. And since we don't
10:18:31partition the data, we're going to leave
10:18:33the definition like this and it's going
10:18:35to be the average score. So that's it.
10:18:39Let's go and execute it. So now as you
10:18:40can see, we have the average score of
10:18:42625. SQL is going to go and summarize
10:18:45the four values and divide it by four.
10:18:48But here we have a null. So now we have
10:18:50to understand the business or ask about
10:18:51it what the null means in the scores of
10:18:54the customers. Is it zero or is it
10:18:56something empty? If it's zero then the
10:18:59average that we have is wrong because it
10:19:00should be divided by five and not four.
10:19:03So let's say it's zero that means we
10:19:06have to go and handle the nulls. So what
10:19:07we're going to do now we're going to go
10:19:08and use the function kalis. So qualis
10:19:12and for the score and replace the null
10:19:15with zero. So you are the customer
10:19:17score. Let's go and execute this. So now
10:19:19as you can see if there is a value it's
10:19:21going to be exactly the same value but
10:19:23only if we have null it's going to be
10:19:24replaced with zero. So now let's go and
10:19:26correct the average. I'm just going to
10:19:28do it like this. So let's go and copy
10:19:31the whole thing. But now instead of
10:19:33using the score we're going to use the
10:19:35score that is handled with nulls. So I'm
10:19:37just going to go and replace it like
10:19:39this. So here without nulls. So let's go
10:19:43and execute it. So now as you can see we
10:19:45are getting more valid result at the
10:19:47output compared to the previous one. And
10:19:49this is only for the case if the null
10:19:51means zero. So guys as you see be very
10:19:54careful with the nulls especially if you
10:19:55are doing aggregations and handle it
10:19:57correctly before doing any aggregations
10:19:59like the average. All right. Moving on
10:20:01to the last use case. We have the
10:20:03comparison analyzes and the task says
10:20:05find all orders where the sales are
10:20:08higher than the average sales across all
10:20:10orders. So that means we have to go and
10:20:13compare the current sales with the
10:20:15aggregated value and this time the
10:20:17average of sales. So now let's go and do
10:20:18it step by step. So what we're going to
10:20:20do we're going to go and select of
10:20:22course the order ID. What do we need the
10:20:24let's take the product ID and we need
10:20:26the current sales. So it's going to be
10:20:28the sales as it is and that's it for
10:20:31now. So from sales orders. So that's it.
10:20:35Let's go and execute it. So now by
10:20:37checking the result, you can see that we
10:20:39got the first part of the equation,
10:20:40right? We have the sales for each order.
10:20:43Now we need the second part, the average
10:20:45sales across all orders. In order to do
10:20:47that, we're going to go and use the
10:20:48window function average sales and we're
10:20:51going to use over since across all
10:20:53orders that means it's going to be
10:20:54empty. So let's give it a name average
10:20:56sales. So let's go and execute it. So
10:20:59now in the output we got the average
10:21:01sales. So it's going to be 38. So now we
10:21:03need all the orders that are higher than
10:21:06the average. So as you can see for
10:21:07example the order one is not higher but
10:21:10the order for is higher than the
10:21:12average. So in order to filter the data
10:21:14we cannot use the window function in the
10:21:17wear close. Right? So what we're going
10:21:19to do sadly we're going to go and use
10:21:21the subquery. So it's going to be like
10:21:23this. select star from and then we're
10:21:27going to define the condition outside
10:21:30the subquery. So it's going to be where
10:21:32the sales is higher than the average
10:21:35sales. So that's it. Let's go and
10:21:38execute it. And now as you can see it's
10:21:40very simple. We got all the orders that
10:21:42are higher than the average. Right? So
10:21:44you can see all those sales are higher
10:21:46than the average. It would be nice if we
10:21:48can do all those stuff in the first
10:21:50query. But since we cannot do that, we
10:21:53need to use the subqueries in order to
10:21:55filter the data afterward. So that we
10:21:58can understand the importance of the
10:21:59comparison analyszis. For example, here
10:22:01we are finding or evaluating the data
10:22:04whether they are above the average or
10:22:06below the average. And this is very
10:22:08important in the business analyzes. All
10:22:10right, everyone. So that's all for the
10:22:11window function average. Next, we're
10:22:13going to talk about two very interesting
10:22:15functions, the min and max.
10:22:20All right guys, so what is min and max
10:22:22functions? They are very simple but yet
10:22:25very powerful functions for analytics.
10:22:27So the min simply is the function that
10:22:29can return the minimum or let's say the
10:22:31lowest value within a window where the
10:22:34max it's exactly the opposite. It's
10:22:36going to find the maximum value or the
10:22:38highest value within a window. So now
10:22:41let's go and understand how SQL works
10:22:43with these functions. All right. So now
10:22:44we have the same data and we have two
10:22:46tasks. First we have to find the lowest
10:22:49sales for each product. And the second
10:22:51one side by side we would like to find
10:22:53the highest sales for each product. So
10:22:55we're going to go and use the min max.
10:22:57And as you can see the syntax is very
10:22:58simple. Min the sales and then the
10:23:00partition going to be by the products.
10:23:02And here as well the same stuff but
10:23:04having the max. Okay. So now let's see
10:23:06how going to execute the first query. As
10:23:08usual first it's going to prepare the
10:23:09data. So it's going to split the data
10:23:11into two windows. One for the caps and
10:23:13another one for the gloves. And after
10:23:15that it's going to search for the lowest
10:23:17sales within each window separately. So
10:23:20for the first window we have the
10:23:21following values 20 10 and five. And of
10:23:24course the lowest value going to be the
10:23:26five. So that's why SQL going to find it
10:23:28over here. And everywhere for this
10:23:30window it's going to be the value five.
10:23:32So we have it as the lowest sales for
10:23:34the product caps. So now it's going to
10:23:36jump to the next window for the gloves
10:23:37and start searching the values. So as
10:23:39you can see we have 30 70 and null. Null
10:23:42will be ignored. So null will not be
10:23:44considered as the lowest value. So SQL
10:23:46going to find the lowest sales with the
10:23:4830. So it's going to be actually the
10:23:50first row within this window and the
10:23:52value the output going to be 30 for each
10:23:55row. So that's it. It's very simple,
10:23:56right? Now let's move to the next one.
10:23:58We have the same stuff but using max. So
10:24:00the data is partitions and for the first
10:24:02partition what is the highest value?
10:24:04It's going to be the first row, right?
10:24:06The 20. So SQL going to find it and in
10:24:08the output we will get the highest sales
10:24:1120 for this window and then it's going
10:24:12to go to the second window and search
10:24:14for the highest value. So here we have
10:24:16two values 30 and 70 and it's going to
10:24:18be the 70 right. So it's going to point
10:24:20it over here and in the output we will
10:24:22get everywhere 70. So guys it's really
10:24:24simple right now let's back to our
10:24:26scenario in the average where in our
10:24:28business we understand nulls as zero in
10:24:30the sales. So that means first we have
10:24:32to handle the nulls and replace it with
10:24:34zero and then we're going to go and
10:24:36search for the value. So what's going to
10:24:38happen? We're going to go and replace
10:24:40nulls with zero. For the max nothing
10:24:42going to change the highest value going
10:24:44to be 70 and we're going to get the same
10:24:45output. But for the min now we have new
10:24:49lowest value. So it's not anymore the
10:24:5130. It's actually the zero. So SQL going
10:24:54to go over here and replace the 30 with
10:24:56nulls. So nulls is the lowest sales for
10:25:00the product gloves. So again guys, the
10:25:02nulls are very tricky and those
10:25:04functions are really sensitive with the
10:25:05nulls. Understand what the nulls means
10:25:08and handle it correctly so that you get
10:25:10correct results in the output. So that's
10:25:12it. Let's go back to SQL to have some
10:25:14tasks and use cases in order to practice
10:25:16SQL. All right everyone, let's start
10:25:18with the basic stuff. find the highest
10:25:20and lowest sales of all orders and as
10:25:22well find the highest and lowest sales
10:25:24for each product and we have to provide
10:25:26additional informations. So let's go and
10:25:29solve it. Select order ID order and
10:25:33let's take as well the product ID. Now
10:25:34let's find the highest sales of all
10:25:36orders. It going to be the max function
10:25:39for the sales and the window function
10:25:42going to be empty since of all orders.
10:25:44So you are the highest sales. Let's go
10:25:47for the lowest sales of all orders. It's
10:25:48going to be exactly the opposite. The
10:25:50main function for sales over then we
10:25:53have the lowest sales. So I'm just going
10:25:56to make it bigger capital. So let's
10:25:58select the table sales orders. So I
10:26:02think that's it. Let's have as well the
10:26:04sales actually. All right. So now let's
10:26:06go and execute it. So now this is very
10:26:08simple, right? This is the wholesales.
10:26:10What is the highest sales? We have the
10:26:1390 of the order eight. So, as you can
10:26:15see, we have now the highest sales, the
10:26:1790, and the lowest sales is the 10. The
10:26:20first order is the lowest. So, it's very
10:26:22easy. Now, we're going to go and repeat
10:26:24the same stuff for the product. So, we
10:26:26have go and partition the data by the
10:26:28product ID. So, what I'm going to do,
10:26:29I'm just going to go and copy paste
10:26:31stuff around. So, the first one going to
10:26:33be partition by the product ID. So,
10:26:36highest sales by product. And the next
10:26:39one going to be the same stuff. Copy
10:26:41paste by the product. So that's it.
10:26:45Let's go and execute it. So now again
10:26:48the data going to be partitioned and
10:26:49divided by the product. So for the first
10:26:52window what is the highest sales? It's
10:26:54going to be the 90 and the lowest sales
10:26:55is going to be the 10. So it's exactly
10:26:57like the overall rights now let's go to
10:26:59the second window over here. We can see
10:27:01that the lowest or the highest sales is
10:27:03the 60 the first one and the lowest this
10:27:05time is 15. And this is great in order
10:27:08to see that the SQL going to execute
10:27:10each of those functions for each window
10:27:12separately. So let's go to the last
10:27:14window. It's funny one. So the sales is
10:27:1660 and we have only one row. So it's
10:27:18going to be the highest and as well the
10:27:19lowest sales. So with that as you can
10:27:21see we can define a range for each
10:27:22product and the range are different from
10:27:24each product to another one. For
10:27:26example, for this product 101 the range
10:27:28from 10 until 90. But for the second
10:27:31product we have it between 15 and 60.
10:27:33Okay guys, let's move to the next one
10:27:35which is one of my favorites in the
10:27:36window function where we filter the data
10:27:39using the minmax functions. Let's have
10:27:41the following task. It says show the
10:27:43employees who have the highest salaries.
10:27:46So this sounds very simple but we can
10:27:48use the help of window functions in
10:27:50order to solve it. So now we are working
10:27:52with the table employees. Let's just
10:27:54select the data. So select from sales
10:27:58employees. So that's it. Let's go and
10:28:01execute it. So now we have five
10:28:03employees and we have those different
10:28:04salaries. Let's go and find the highest
10:28:08salary. So max salary and let's use the
10:28:11window function over but we don't
10:28:13partition the data at all. So it's going
10:28:15to be like this highest salary. So let's
10:28:18go and execute it. And now by checking
10:28:21the results we got a new column called
10:28:22highest salary and inside it we have the
10:28:2590k. So if you check those five salaries
10:28:27you can see that the highest is from the
10:28:30employee Michael. But still the task is
10:28:32not solved. We have to show only the
10:28:34employees who have the highest salaries.
10:28:36So we have somehow to filter the data
10:28:38and only show this employee. So in order
10:28:41to do that we have to use the subqueries
10:28:44since we cannot use the window function
10:28:46in the wear clause. So what we're going
10:28:48to do select star from and then our
10:28:52first query going to be the inner query.
10:28:55So we have the following condition. It's
10:28:57going to be the
10:28:59salary should be equal to the highest
10:29:02salary. So it's very simple. So with
10:29:04that we are comparing the salaries with
10:29:06the highest salaries. If there is a
10:29:08match the data going to be presented. So
10:29:10let's go and execute that. And that's
10:29:12it. As you can see we got the employee
10:29:14with the highest salary. But if there
10:29:16are like multiple employees with the
10:29:18same salary of 90k of course we're going
10:29:20to get it in the results. I think
10:29:22Michael going to need a new job. Right.
10:29:24This is the worst.
10:29:30So this is another use case for the
10:29:32window functions minmax. All right. So
10:29:34now we come to the use case of the
10:29:36comparison analyzers where we want to
10:29:38compare the current sales with the
10:29:40highest and the lowest value. So we have
10:29:42the following task. It says find the
10:29:44deviation of each sales from the minimum
10:29:47and the maximum sales amount. So now as
10:29:49you can see this is our sales. This is
10:29:50the highest and this is the lowest. So
10:29:52now we just have to go and subtract the
10:29:54data from each others in order to get
10:29:56the deviation. So it's very simple.
10:29:58Let's get the first deviation where
10:30:00we're going to go and subtract the sales
10:30:02with the lowest value. So it's going to
10:30:05be like this. So now what we are doing
10:30:06over here, we are subtracting the sales
10:30:08from the lowest sales of all records. So
10:30:12we're going to go and call you
10:30:14deviation from min. So let's go and
10:30:17execute it. So now we can see from those
10:30:19values how far is the current value from
10:30:23the extreme. The extreme here is the
10:30:25lowest value. So this is a really great
10:30:27way on to analyze the extremes in your
10:30:29data. So now as we are near to the
10:30:31extreme the value going to be low. So as
10:30:34you can see here we have a zero. This is
10:30:35the lowest because we have it exactly as
10:30:38the extreme. So actually this is our
10:30:41value. So the 10. Now the next one is
10:30:43little bit far away from the extreme
10:30:45which is 15. So we have it here as a
10:30:47five. So this is not far away from our
10:30:49extreme value. And then if you check
10:30:51this value over here we have it 80. So
10:30:53the distance is very far away from our
10:30:56extreme value the lowest sales. So this
10:30:58is really nice analyszis in order to
10:31:00analyze and evaluate the sales of your
10:31:03data. Now of course we can go and
10:31:05evaluate our data with an another
10:31:07extreme which is the highest sales. So
10:31:10in order to do that we're going to first
10:31:12say let's get the highest sorry this one
10:31:15the highest sales and subtract it from
10:31:18the sales. So you are the deviation from
10:31:21the max. So let's go and execute it. So
10:31:24now we can see in the output we're going
10:31:26to get exactly the opposite distances.
10:31:28So the order number one is the farthest
10:31:31from the extreme. So as you can see we
10:31:33have the value of 80 and the order eight
10:31:36is the identical one. So that's why we
10:31:38have the distance of zero. So now we can
10:31:40see as well very quickly which data
10:31:42points are the nearest to the extreme to
10:31:44the highest sales. So as you can see
10:31:46guys using the window function min and
10:31:48max it is very powerful in order to
10:31:50understand and evaluate your data points
10:31:53to the
10:31:54[Music]
10:31:57extremes. All right everyone so now
10:31:59we're going to focus on very important
10:32:00use case. One of the must know use cases
10:32:03for data aggregations is doing running
10:32:05total and rolling total. These two
10:32:08concepts are very important for data
10:32:10analyszis and doing reporting that you
10:32:12must know. The key use case for those
10:32:14two concept is to do tracking. For
10:32:17example, we can go and track the current
10:32:19total sales with the target sales in our
10:32:22business. And as well, it's great in
10:32:24order to do historical analyszis for the
10:32:26trends. Okay. So now the question is
10:32:28what is running a rolling total. They
10:32:31are basically very similar. They're
10:32:32going to go and aggregate a sequence of
10:32:34members and the aggregation going to get
10:32:36updated each time we add a new member to
10:32:38the sequence. A sequence could be like a
10:32:41time sequence. That's why we call this
10:32:43type an analyzes over time. So now we
10:32:46still have the question, what is the
10:32:47difference between the running and the
10:32:49rolling totals. The running total going
10:32:51to go and aggregate everything from the
10:32:53beginning until the current data point
10:32:56without dropping off any old data. Where
10:32:58on the other hand in the rolling total
10:33:00it going to go and focus on a specific
10:33:02time window like the last 30 days or the
10:33:05last two monthses and each time we add a
10:33:07new member or a new data point to the
10:33:09window we will be dropping off the
10:33:12oldest data point in the window and with
10:33:14this we're going to get the effect of
10:33:16rolling or let's say shifting window
10:33:18okay I totally understand if this might
10:33:20be complicated now let's go and have
10:33:22very simple example in order to
10:33:23understand this concept and as well how
10:33:25we can solve it using SQL all right guys
10:33:27so now We have very simple example. We
10:33:29have the months and sales and we have it
10:33:31twice because I want to show you side by
10:33:33side how SQL works with the running
10:33:35total and the rolling total. So now what
10:33:37is the task on the left side? We want to
10:33:39find the running total of sales for each
10:33:42month and on the right side we would
10:33:44like to find three month rolling total
10:33:47of the sales for each month. So they
10:33:49sound very similar but on the right side
10:33:51we have only fixed window. So now how we
10:33:54can solve this using SQL. On the left
10:33:56side we can use sum of sales. So we want
10:33:58to go and aggregate all the sales using
10:34:00the sum function. And the definition for
10:34:02the window going to be like this order
10:34:04by month and of course you can go and do
10:34:06anything like you can have here an
10:34:07average. And if you use an average with
10:34:09order by you will get the running
10:34:11average or the running max or the
10:34:13running count and so on. So that means
10:34:16always if you go and mix an aggregate
10:34:18function together with an order by you
10:34:20will generate an effect of running
10:34:22total. Now on the right side we can have
10:34:24the same stuff. So we can have an
10:34:26aggregate function together with order
10:34:27by. So sum of sales, order by month. So
10:34:31far we have everything like the left
10:34:32side, right? But now you might ask why
10:34:34is going to go and generate this effect
10:34:36the running total. We didn't here
10:34:38specify like crazy stuff, right? It's
10:34:40all about the definition of the frame
10:34:42close. So now do you remember if you use
10:34:44an order by and you don't specify a
10:34:46frame close you will get like hidden or
10:34:49let's say default frame close and it's
10:34:51going to look like this rows between
10:34:53unbounded preceding and current row. And
10:34:55what was the definition of the running
10:34:57total? It's going to go and aggregate
10:34:59all the data from the very first
10:35:01beginning well the unbounded proceeding
10:35:03until the current position the current
10:35:06row without dropping off any old
10:35:08members. So that means the definition of
10:35:09the running total going to be the exact
10:35:12definition of the default frame clause.
10:35:14That's why it's going to go and generate
10:35:16the effect of the running total. Now
10:35:18let's go to the right side the rolling
10:35:20total. Here again we have the same stuff
10:35:22right. We're going to go and aggregate
10:35:24the data using the sum function and
10:35:26we're going to go and sort the data
10:35:27order by month. So with that we are as
10:35:29well generating the effect of running
10:35:32total. So each time you use order by
10:35:34with aggregate function. So now in the
10:35:35running total we want always to specify
10:35:37a frame. So here in this example three
10:35:39months. So that means if we are getting
10:35:41a new month we don't want to include the
10:35:43latest months. We want always to be
10:35:45fixed window. Now in order to have this
10:35:48fixed window effect we have to go and
10:35:50redefine the frame close because if you
10:35:52leave it as a default like the running
10:35:54total the frame going to keep extending.
10:35:56You will see this effect in the example.
10:35:58So now we define it like this rows
10:36:00between two preceding and current row.
10:36:03So the total number of rows going to be
10:36:05included in each window going to be
10:36:07maximum of three months. So now I know
10:36:10you might saying bar what you are
10:36:12talking about you didn't get anything.
10:36:14It's total normal you will understand it
10:36:16only with an example. So in order to do
10:36:19this let's start with the left side. So
10:36:20first going to go and sort the data. So
10:36:22everything is sorted from the smallest
10:36:24month until the highest one. So from
10:36:26January until July everything is good.
10:36:28And now it's going to go and start
10:36:30working with the frame. So the frame
10:36:32says unbounded proceeding. So this going
10:36:34to be static. It's going to be always
10:36:36pointing to January. This is the
10:36:39unbounded proceeding. The first row in
10:36:41the data set. And now of course we are
10:36:43starting from top to bottom. The current
10:36:45row going to be pointing as well to
10:36:47January. So the frame going to look like
10:36:49this. It's going to be only one row and
10:36:52the total sale of this row going to be
10:36:5420. So that's why we're going to have in
10:36:55the output 20. So now let's move to the
10:36:57right side. The current row going to be
10:36:59as well January. And what is the two
10:37:01proceeding? We don't have it yet. So
10:37:03it's going to be pointing maybe
10:37:04somewhere here before the table. So
10:37:06again, what is the frame? It's going to
10:37:08be as well one row. So in the output, we
10:37:10will get exactly the same result 20. So
10:37:13so far there is no differences between
10:37:15the running total and the rolling total.
10:37:17But let's keep going. Now we're going to
10:37:19go to the next row over here. So what
10:37:21can happen to our frame? It going to go
10:37:23and extend, right? So we're going to
10:37:24have now two months in this frame. And
10:37:27what is the total sales over here? It's
10:37:28going to be 30. So we added a new
10:37:30member. You can calculate it like this.
10:37:32Either go and calculate all the sales
10:37:34within the frame or you can go and say
10:37:37this is the previous aggregated value
10:37:39plus the new member. So the previous one
10:37:41was 20. The new member is 10. We will
10:37:44get 30. Both of them is correct. So now
10:37:46let's move to the right side. What's
10:37:48going to happen? We're going to be as
10:37:49well at February. The two preceding is
10:37:51still like pointing somewhere outside.
10:37:53And here the window going to go and
10:37:55extend like this. We have two months and
10:37:58the same aggregation going to happen. So
10:37:59we have 30. So so far nothing crazy
10:38:02right. Let's go to the next month March.
10:38:04The frame going to be extended. So we
10:38:06have now three months. And the
10:38:08aggregation going to be either here 60
10:38:10or 30 + 30. We will get the running
10:38:13total of 60. And now on the right side
10:38:15what going to happen? We're going to
10:38:16point as well to March. And this time
10:38:18the two proceeding going to be pointing
10:38:19to January. And this is the first time
10:38:22we are getting the whole fixed frame.
10:38:24Right? So we have here three muscles in
10:38:26this frame. So what is the total of
10:38:28that? It's going to be 60. Okay. So now
10:38:30you say, okay, we're still getting the
10:38:31same results. There's no difference. I'm
10:38:33going to say wait for it. It's going to
10:38:34be the next one. So as we go to April,
10:38:36the effect here is that the frame going
10:38:38to get extended to four months because
10:38:41always we start from the first month
10:38:43until the current month without dropping
10:38:45any member outside. So what is the total
10:38:47of this? It's going to be 65. Sorry,
10:38:50like this. So now on the right side,
10:38:52what going to happen? We're going to go
10:38:53and add a new member. the April but we
10:38:55are at the maximum sides of the window
10:38:57we have only three and that's because
10:38:59the two preceding going to shift as well
10:39:01down over here so the boundary going to
10:39:04be from February until April and with
10:39:06that we are dropping off January and now
10:39:08you're going to see the effect it is
10:39:10sliding it is rolling or shifting from
10:39:13top to bottom and that's because the
10:39:15boundaries as well shifting so you can
10:39:17see now the effect of the rolling total
10:39:20the newest member going to be added the
10:39:22oldest member going to be But we are
10:39:24allowed only to have three muscles. So
10:39:26what is the total of this? It's going to
10:39:28be 45. So this times we are not
10:39:30aggregating this value the 60 together
10:39:32with the five. We are aggregating the
10:39:35values within the window. So now let's
10:39:37keep going. Now we are at June. What can
10:39:40happen on the left side? The frame going
10:39:41to get bigger. And with that we will get
10:39:43the result of 135. So the frame is
10:39:47getting really bigger. But on the right
10:39:48side it's going to has a fixed frame. So
10:39:51we are just sliding, shifting and
10:39:53rolling. So with that we are adding new
10:39:56member. Another member is leaving the
10:39:57oldest one. And the total over here
10:39:59going to be 105. And now we're going to
10:40:01go to the last row. We will have
10:40:03everything for the ring total. So the
10:40:05whole data set is going to be
10:40:07aggregated. So this is the maximum what
10:40:09we're going to get. It's going to be
10:40:11around 175. But on the right side it
10:40:15just going to keep shifting until we
10:40:16reach the last record. the window the
10:40:19frame going to be as well shifting like
10:40:20this. So the total of this going to be
10:40:22105. Okay guys so you see it's very
10:40:25simple the running total it's always
10:40:27consider everything from the starting
10:40:30position until the current row without
10:40:32dropping any member. The rolling total
10:40:35it's always drop the oldest member in
10:40:37order to add something new and the
10:40:39window is keep shifting. So the running
10:40:42total is very great in order to do
10:40:43tracking like for example budget
10:40:45tracking or we track for example the
10:40:48current total sales with a target or
10:40:50something like that. So always we are
10:40:52considering the whole data sets but with
10:40:54the rolling total we always do here
10:40:56focused analyzes. We are always
10:40:58interested with the window of 3 months.
10:41:01So they might sound very similar but
10:41:03they have completely different scope for
10:41:04analyzes but both of them are doing
10:41:07aggregations over time. So they're going
10:41:09to help us to do analyzes over time like
10:41:11checking whether our business is growing
10:41:13over time or declining. So guys as you
10:41:15can see using very simple SQLs using the
10:41:17window functions we can do really great
10:41:20analysis on our data. So those stuff are
10:41:22really fundamental of data analyzes or
10:41:25doing reporting for our business. So
10:41:27window functions are really powerful for
10:41:29data
10:41:33analytics. Okay. Okay. So now we have
10:41:35the following task and it says calculate
10:41:37the moving average of sales for each
10:41:39products over the time. So now we have
10:41:42here something called moving average. It
10:41:44is very similar to the running total. In
10:41:47the running total we used count and sum
10:41:49and so on. But here we're going to go
10:41:51and use the function average and instead
10:41:53of calling it running average we call it
10:41:56moving average. So let's go and solve
10:41:58the task. Let's start always by
10:42:00selecting the usual stuff. So let's get
10:42:02the order ID. Let's get the product ID
10:42:05and I would say since it's over the time
10:42:07I will get the order date as well and
10:42:09the last one the sales from our table
10:42:12sales orders. So that's it. Let's go and
10:42:17execute it. So now we got our 10 orders
10:42:19with the products order date and sales.
10:42:21Let's start building our window function
10:42:23step by step. So which function do we
10:42:26need? We need the average. This is the
10:42:27easiest one. It says moving average. So
10:42:30that's it. We need the sales. So it's
10:42:32going to be the average of sales. Let's
10:42:34go and define the window. So now do we
10:42:36have to divide the data, partition the
10:42:38data? Well, yes. It says for each
10:42:41product that means we're going to go and
10:42:43use the partition by clause by the
10:42:45product ID. So now I would say that's it
10:42:48for the first step. So average by
10:42:51product. So let's go and execute it. So
10:42:54now if you check the result, you can see
10:42:55that we got our windows. So the first
10:42:57one for the product 101 and the total
10:43:00average of the sales going to be 35. So
10:43:03we have like aggregated one value for
10:43:06each window. The same thing for the next
10:43:08product and for the next and so on. So
10:43:10we don't have any progress over time or
10:43:12something like moving average all the
10:43:14time. Right? We don't have this effect.
10:43:16We have just one average for each
10:43:18window. So now in order to have the
10:43:19effect of the moving average, it's going
10:43:21to be like the running total. We have to
10:43:23use the aggregate function together with
10:43:25the order by. So I'm just going to make
10:43:28it in the new column. I'm just going to
10:43:30copy everything like here. And now what
10:43:33we going to do? Order by. And since it's
10:43:36over the time, we're going to go and use
10:43:38the order dates. Order dates. And we're
10:43:42going to have it ascending because it's
10:43:44overtime. Over time always like start
10:43:46with the earliest dates, end up with the
10:43:48latest dates. So from the lowest to the
10:43:50highest, we're going to leave it like
10:43:52this. So let's call it moving average.
10:43:55So now let's go and execute it. And we
10:43:58got here an extra comma because of the
10:44:00copy paste. So let's execute it again.
10:44:02All right. So now let's check the
10:44:03results. Let's take the first window
10:44:05over here. And you can see we have on
10:44:07the moving average like a progress. So
10:44:10it start with 10 15 14 35. So there is
10:44:14like moving average. We don't have one
10:44:16solid number for the average. We have
10:44:19different values. So now how SQL going
10:44:21to solve this? It's really simple. It's
10:44:22going to start row by row. So the first
10:44:24row what is the average of 10? It's
10:44:26going to be 10. Then moving on to the
10:44:28next one it's going to be 10 + 20
10:44:30divided by 2 you will get 15. So now
10:44:33moving to the third one all those three
10:44:35values going to be summarized divided by
10:44:37three you will get 40. And now to the
10:44:40last row in the window it's going to be
10:44:42summarizing all those four values
10:44:44divided by four and you will get 35. And
10:44:47this is exactly the same value in the
10:44:49previous column. You have here the
10:44:51average byproducts. We don't have order
10:44:52by you got as well 35 exactly like this
10:44:56last row and that's because we have the
10:44:58same calculation. It is summarizing all
10:45:00those four values dividing it by four.
10:45:02But now it's interesting the next value.
10:45:05So as you can see the next value it
10:45:06comes from another window. So you see
10:45:09here we have 15 for the product 102 but
10:45:12the average going to be as well 15. So
10:45:15scale is not considering the old values
10:45:17from the other window. So SQL going to
10:45:20calculate each window separately. So
10:45:22it's again here this is the first value
10:45:24of this window 15 the average 15 then
10:45:27the same stuff right. So summarizing
10:45:29those values divided by two and so on.
10:45:31And this we call in data analyzes this
10:45:34last field over here we call it a moving
10:45:36average and you can implement it very
10:45:38simply using an average function
10:45:40together with the order by. All right,
10:45:42let's move to the next task and it says
10:45:44calculate the moving average of sales
10:45:46for each product over time including
10:45:49only the next order. So as you can see
10:45:52the first part we have already done it
10:45:53right. We have the moving average and
10:45:55divided by partition by for the products
10:45:58but here we have more specifications. It
10:46:00says including only the next order. That
10:46:03means we are talking about the current
10:46:05order and as well the next order. So
10:46:08here we have like a fixed frame or fixed
10:46:10window. So we don't need the whole
10:46:12average of the window. We need only
10:46:14maximum two orders in each calculation.
10:46:17So how we going to do that? We can have
10:46:19our custom frame close inside our window
10:46:22function. So that means we cannot leave
10:46:24it as a default. We have to specify it.
10:46:27So let's go and do that. I will just
10:46:29copy the old definition of the window
10:46:31because we have the exact stuff. So we
10:46:33have the average sales over partition by
10:46:36product ID order by date. So this is the
10:46:39first part. So now we would like to have
10:46:41this fixed window. So we're going to go
10:46:43now and define our frame close. I'm just
10:46:46going to zoom out a little bit. It's
10:46:48going to be rows between. So we have now
10:46:50the boundaries of the frame. It says
10:46:52including the next order. So we're going
10:46:54to go and use the following. So the
10:46:56first boundary going to be the current
10:46:58row. And since it's next order, so it's
10:47:00going to be one following. So that is
10:47:03our frame including only the next order.
10:47:06And we have it like this one following.
10:47:09Let's call it yeah rolling average. So
10:47:12that's it. Let's go and execute. So now
10:47:14let's go and check the result. You can
10:47:16see the moving average has completely
10:47:18different values as the rolling average.
10:47:20So let's go and understand why. We're
10:47:22going to do it row by row. Let's take
10:47:24the first row over here. So the sales
10:47:26here is 10 and the rolling average is
10:47:2815. So why is that? Because in the
10:47:31calculation we are considering the next
10:47:33value. So 10 + 20 divided by 2 you will
10:47:37get 15. So that means the SQL defined
10:47:40the frame like this those two rows for
10:47:42this calculation for the first row. So
10:47:44now moving on to the second row. SQL
10:47:47going to include as well the third one
10:47:49right the next one. But since the window
10:47:52is only two orders it's going to go and
10:47:55drop the first row. So the next frame
10:47:57going to be like this. And as you can
10:48:00see it's going to be 20 + 19 divided by
10:48:022. You will get 55. So now you can see
10:48:05the effect of the rolling average.
10:48:07Right? So now for the next one going to
10:48:09be exact same. So we are at the third
10:48:11row. It's going to go and include the
10:48:12next one and we're going to get the same
10:48:14value because 19 + 20 divid by two you
10:48:17will get 55. Now interesting to the last
10:48:20row in the window over here. It will not
10:48:22go and consider the next value because
10:48:25it is outside of the window. So it's
10:48:27going to be 20 and it's going to stay as
10:48:30well 20. So that's it. All right guys.
10:48:32So with that we have learned about the
10:48:34moving average, the rolling average and
10:48:36those amazing concepts using the window
10:48:39function. All right. Now we're going to
10:48:40have a quick overview of the different
10:48:41use cases in the aggregate functions and
10:48:44how the definition of the window going
10:48:46to change the whole use case. So now the
10:48:48first use case is finding the overall
10:48:50total. And here if you don't define
10:48:53anything in the window if you leave it
10:48:54empty what going to happen you are doing
10:48:56here overall analyzes. So you're going
10:48:59to go and aggregate the whole data sets
10:49:02and then provide this aggregation for
10:49:04each row. So this is what happen if you
10:49:06leave it empty. You don't define
10:49:07anything. You are aggregating the whole
10:49:10data sets. Now moving to the next step,
10:49:12we can do analysis called total pair
10:49:14groups. So what we're going to do, we
10:49:17will add partition by to the definition
10:49:19of the window. So by adding for example
10:49:21here partition by products, what can
10:49:23happen? The data going to be splitted
10:49:25into two categories or two groups and
10:49:27the aggregation going to be done for
10:49:29each window separately. This is of
10:49:31course a great analysis in order to go
10:49:32and compare different products like here
10:49:35the caps and gloves. So this is helpful
10:49:37in order to compare categories. So you
10:49:39can do this analysis total pair groups
10:49:41if you use the partition by. Now if you
10:49:44go and use the order by you're going to
10:49:46land in the third use case. As we
10:49:48learned we will be doing running total.
10:49:50So as you can see here in the output we
10:49:52are building a cumulative value for the
10:49:54sales and this going to help us in order
10:49:56to do progress over time analyzes in
10:49:59order to understand the performance of
10:50:00our business. And now moving on to the
10:50:02last use case the final phase of the
10:50:04window function with the aggregation.
10:50:06Here you have the aggregate function
10:50:08together with the order by with
10:50:10customized fixed window. And of course
10:50:12we can use it in order to help us
10:50:14building progress over time in specific
10:50:16fixed window. And of course you can use
10:50:18those use cases you will get the same
10:50:20effect if you use the other functions
10:50:22not only the sum you can use average
10:50:24count max so all aggregate functions. So
10:50:26guys as you can see the window function
10:50:28in scale is very important in order to
10:50:30do data analytics by just like changing
10:50:33the part of the window you are
10:50:34generating a whole new use case for data
10:50:40analytics. All right friends so now
10:50:42let's do a quick recap about the window
10:50:44aggregate functions. So what they do
10:50:46they're going to go and aggregate a set
10:50:48of values and return a single aggregated
10:50:51value for each row. So it's very similar
10:50:53to the groupy but here we don't lose
10:50:55details. Now to the next point what are
10:50:57the rules for the syntax about the
10:51:00expressions they all expect a number in
10:51:02the expression. So you have to pass a
10:51:04number like sales or any integer but
10:51:06only for the count you can go and use
10:51:08any data type. And the things for the
10:51:10aggregate functions are very simple.
10:51:12Everything is optional inside the
10:51:13definition of the overclouds or the
10:51:15definition of the window. So you can go
10:51:17and use partition by order by frames or
10:51:19not or just leave everything empty. So
10:51:21everything is optional. So now as we
10:51:23learned we have a lot of use cases for
10:51:25the aggregate functions and they are
10:51:26really amazing for analytics. So the
10:51:28first one the simplest one you can do
10:51:30overall analyzes if you just leave the
10:51:32window function empty. So you will get
10:51:34one big number about your business. And
10:51:36the next use case we can do total bear
10:51:38groups analyzes. As you've learned, we
10:51:40can use partition by in order to compare
10:51:42categories with each others like
10:51:44comparing the products or customers and
10:51:45so on. Moving on to the next one, we can
10:51:47do partto-hole analyszis. We can go and
10:51:49compare the performance of each data
10:51:51point with the overall. So you can for
10:51:53example compare the sales to the total
10:51:55sales in the window or to the all data
10:51:58sets. And we have many comparison
10:52:00analyzes. We can go and compare the
10:52:01current value with the average or we can
10:52:04compare them to the extreme to the
10:52:05highest sales to the lowest sales and so
10:52:07on. And another use case, we can go and
10:52:09identify data quality issues in our
10:52:11data. So we can go for example and
10:52:13identify duplicates using the count
10:52:15function. Moving on to the next use
10:52:17case, we have the outlier detection. We
10:52:18can go and find out which data points
10:52:21are above the average and below the
10:52:23average and so on. Then the next one we
10:52:25have the running total. As we learned,
10:52:27it is great tool in order to track the
10:52:29progress or to monitor the performance
10:52:31of our business over the time. Or if you
10:52:34want to be more specific, you can go and
10:52:36use the rolling total in order to have
10:52:39like a specific window and only track
10:52:41this window like three months or
10:52:43something like that. And the last use
10:52:44case, we can go and calculate the moving
10:52:46average of our data. So it's really
10:52:48amazing how order by and aggregate
10:52:50functions can open for you a door for
10:52:52amazing or advanced analyzers. So guys,
10:52:55as you can see, we have a lot of use
10:52:56cases for the window aggregate functions
10:52:58in the world of data analytics. All
10:53:00right. Right. So with that we have
10:53:01covered the aggregate window functions
10:53:03and in the next step it's going to be
10:53:04very important. We will learn how to
10:53:06rank our data using window functions. So
10:53:09let's
10:53:12go. All right. So now let's say that we
10:53:14have the following data. We have
10:53:16products and their sales. If you want
10:53:18now to go and rank your products first
10:53:21you have to sort the data based on
10:53:22something like for example ranking the
10:53:25products based on their sales. So that
10:53:27means SQL first is going to go and start
10:53:29sorting your data from the highest to
10:53:31the lowest. So sorting the data is
10:53:33always the first thing SQL has to do
10:53:36before ranking anything. Now in order to
10:53:39rank our data we have two methods. The
10:53:41first method we call it the integer
10:53:43based ranking. So that means SQL going
10:53:45to go and assign for each row an integer
10:53:48a whole number based on the position of
10:53:51the row. So now by looking to the
10:53:52example the first row we have the
10:53:54product E with the sales 70 it's going
10:53:57to be rank number one then the next row
10:54:00the product B with 30 sales we will get
10:54:02the rank number two then the next one
10:54:04going to be three four and the last one
10:54:06going to be five. So that means SQL here
10:54:08is assigning an integer for each row
10:54:11based on their position in the sorted
10:54:13list. So this method we call it integer
10:54:16based ranking. Now let's go to the
10:54:18second method we have the
10:54:19percentagebased ranking. So in this
10:54:21methods going to go first and calculate
10:54:24the relative position of the row
10:54:26compared to all others and then assign a
10:54:29percentage for each row. So here in the
10:54:31output is going to start assigning
10:54:33percentages instead of integer and we're
10:54:36going to have a scale from 0 to one. So
10:54:38now if you go and compare both of the
10:54:40methods you can see that on the left
10:54:41side on the integer base ranking we have
10:54:44discrete distinct values. So it starts
10:54:47from 1 then 2 3 and end up in this
10:54:50example by five. So it really depends on
10:54:52how many rows do we have in the results.
10:54:55So it could be five, it could be 500, 5
10:54:57million and so on. But in the right side
10:55:00we have always the same scale from one
10:55:02to zero. So between 0 and one we have
10:55:05infinite number of data points and this
10:55:07scale we call it a normalized scale or
10:55:10we call it continuous scale continuous
10:55:13values. So now the question is when to
10:55:15use which method. So for example for the
10:55:17percentage based ranking it is great to
10:55:19answer such questions find the top 20%
10:55:22products based on their sales. So this
10:55:24method is a great way in order to
10:55:26understand the contributions of data
10:55:28values to the overall total and we call
10:55:31this kind of analyszis a distribution
10:55:34analyszis where in the other hand in the
10:55:36integer based ranking we can answer
10:55:38questions like find the top three
10:55:40products. So with this question we are
10:55:42not interesting about the contributions
10:55:44of each product to the overall total. We
10:55:46are just interested in the position of
10:55:49the value within a list. So this is as
10:55:51well very commonly used analyzes and
10:55:53reporting. We call it top button in
10:55:56analyzers. So now let's group up our
10:55:57ranking functions based on those two
10:55:59methods. For the first group in the
10:56:01integer based ranking we have four
10:56:03functions. Row number rank d rank and
10:56:06inile. But in the other hand we have
10:56:08only two functions that generate
10:56:10percentage based ranking. We have the
10:56:12cumid list and as well the percentile.
10:56:15So now that was an introduction an
10:56:17overview of those methods and how we
10:56:19group up those ranking functions. Next
10:56:21we're going to go and learn about the
10:56:23syntax of the ranking functions. Most of
10:56:25them follow the same rules. So for
10:56:27example we start always with the
10:56:28function name. So we have here the rank.
10:56:31But as you can see we don't use any
10:56:33expressions. So they don't allow you to
10:56:35use any argument inside it. It must be
10:56:38empty. So this is the first rule using
10:56:40rank functions. Then about the
10:56:42definition of the window as usual the
10:56:44partition by it is an optional thing.
10:56:46You can use it or leave it. And now to
10:56:48the second part we have the order by it
10:56:50is as well required. So you must order
10:56:53the data or sort your data in order to
10:56:55do ranking. So you cannot leave it
10:56:57empty. So that means for the definition
10:56:59of the window at least we should have an
10:57:02order by for example here sales. So we
10:57:05cannot leave it empty. All right. So the
10:57:07two requirements you cannot use any
10:57:09expressions for those functions and as
10:57:11well you have to sort your data using
10:57:13order by. Okay. So now let's have an
10:57:15overview of all functions. So as you can
10:57:16see all those functions are ranking
10:57:18functions and almost all of them don't
10:57:21allow to use any expressions inside
10:57:23them. Beside this function here we have
10:57:26the end tile. it accepts a number inside
10:57:28it. So that means you cannot use it
10:57:30empty. You should use a number inside
10:57:32it. All others must be empty. So now for
10:57:35the partition by all of them are
10:57:36optional and for the order by all of
10:57:39them are required. So you must use order
10:57:42by and the frame clause they are not
10:57:45allowed to use in the ranking functions.
10:57:47So you cannot change the definition of
10:57:49the frame inside the window function. So
10:57:51now what we're going to do as usual,
10:57:53we're going to go and deep dive into all
10:57:54of those functions in order to
10:57:56understand when to use them and what are
10:57:58the use cases and as well practice in
10:58:01SQL. So we're going to start with the
10:58:02first one, the row
10:58:07number. All right. So what is a row
10:58:09number in SQL? The row number function
10:58:11going to go and assign for each row a
10:58:13unique number as a rank and it doesn't
10:58:16care at all about the ties. That means
10:58:18if you have two rows sharing the same
10:58:21value, they will not share the same
10:58:23rank. Okay. So now we have very simple
10:58:25example. We have a list of all sales and
10:58:28we have the following query. So it's
10:58:30going to start with the ranking function
10:58:32row number. It doesn't accept any
10:58:33argument inside it. And the definition
10:58:35of the window going to be like this
10:58:37order by sales disk. So that means we're
10:58:40going to go and sort the data descending
10:58:43from the highest to the lowest. So SQL
10:58:45going to go and do the following. The
10:58:47highest going to be the 100. The lowest
10:58:50going to be the 20. And here we have
10:58:52twice the 80. So now once SQL done
10:58:55sorting the data, what's going to
10:58:56happen? It's going to start assigning a
10:58:58rank. So the row number going to go and
10:59:00assign a unique number for each row. So
10:59:03that means it's going to start with the
10:59:04first one. The 100 going to be the rank
10:59:07number one. The next one going to be
10:59:09rank number two. The 80 going to be rank
10:59:11number three. And the 54. And then the
10:59:14last one going to be five. And now if
10:59:15you check the output you can see that
10:59:17all those numbers are unique. We don't
10:59:19have any repetitions. So 1 2 3 4 5
10:59:22there's no repetitions. They are unique
10:59:24distinct value. And as well there are no
10:59:27skipping of ranking. So that means we
10:59:29have here 1 2 3 there is no jumping to 6
10:59:317 or something. They are clear sequence
10:59:34of distinct value and there are no gaps.
10:59:37But still there is something special in
10:59:38our data. We can see that in the sales
10:59:41we have the same value twice. So we have
10:59:43two rows with the same sales. As you can
10:59:46see in the row number they will get
10:59:48distinct values. So they will not share
10:59:51the same ranking. So that means row
10:59:53number does not handle the ties. If you
10:59:56have multiple rows sharing the same
10:59:58values they will not share the same
11:00:00rank. They going to have a distinct rank
11:00:03different ranks. So this is how the row
11:00:05number works in SQL. It generates unique
11:00:07ranks for each row. It does not handle
11:00:09the ties and as well it doesn't leave
11:00:12any gaps. So there is no skipping of
11:00:14ranking. So now let's go to SQL in order
11:00:16to have few examples and use cases. All
11:00:18right. So now we have the following
11:00:19task. It's very simple. Rank the orders
11:00:21based on their sales from the highest to
11:00:24the lowest. So now this is very easy.
11:00:25We're going to go and select first the
11:00:27data. So order ID, product ID. Let's
11:00:30take the sales as well and select the
11:00:33table. So it's going to be sales orders.
11:00:35Let's go and execute it. So with that we
11:00:38got all our orders. What we're going to
11:00:39do now is to assign for each row a rank.
11:00:42So that means we need a column here that
11:00:45contains the rank for each row. So in
11:00:47order to do that we're going to go and
11:00:48use the window function row number. It
11:00:51doesn't accept any argument inside it.
11:00:53So should be empty. And then we have to
11:00:54define the window. So as we learned in
11:00:57the ranking functions we cannot leave it
11:00:59empty. We have to sort the data using
11:01:00order by. So order by is a must. We
11:01:03don't have to use any partition by. So
11:01:05we're going to rank all the data that we
11:01:07have inside the table. So how to sort
11:01:08the data? It says it should be based on
11:01:10their sales from highest to lowest. That
11:01:12means we order by sales since from
11:01:15highest to lowest we have to use the
11:01:17descending. And now we're going to go
11:01:18and give it a name sales rank and let's
11:01:21say row since we are using the row
11:01:23number. So that's it. It's very simple.
11:01:26Let's go and execute it. So now let's
11:01:28have a look to the results. Before SQL
11:01:30did sort the data by the order ID since
11:01:32we didn't define anything. But since now
11:01:34we are order by sale descending SQL went
11:01:37and sorted the data by the sales from
11:01:39the highest to the lowest and start
11:01:41assigning a rank or let's say an integer
11:01:44unique integer for each row. So now the
11:01:46highest order going to be the order
11:01:48number eight. We have the sales of 90.
11:01:50This is the highest one. So as you can
11:01:51see we have 1 2 3 4 5 until 10. So now
11:01:54by checking the results you can see that
11:01:56the ranking here is unique. So there is
11:01:59no duplicates over here and as well
11:02:01there is no skipping or gaps. So we have
11:02:03everything between 1 and 10 even though
11:02:05that we have in our data a couple of
11:02:07sales that sharing the same value. So
11:02:09for example we have those two orders you
11:02:11can see both of them has the 60 at the
11:02:13sales but they don't share the same
11:02:15ranking. Right? So we have here as well
11:02:18the 9 and three they share the same
11:02:20value 20 but they don't share the same
11:02:22ranking. So with that we have solved the
11:02:24task. It's very simple. We have now a
11:02:26rank based on the sales from highest to
11:02:28the lowest.
11:02:33All right. So what is a rank function in
11:02:35SQL? The rank function going to go and
11:02:36assign for each row a number a rank and
11:02:39this time it going to go and handle the
11:02:41ties. So that means if in your data you
11:02:44have two rows having the same values
11:02:46they going to share the same ranking.
11:02:49One thing about the ranking function is
11:02:50that it's going to go and leave gaps in
11:02:52the ranking. So there is possibility of
11:02:55skipping ranks. In order to understand
11:02:57how the rank function works in SQL,
11:02:59we're going to have a very simple
11:03:00example. All right. So again with the
11:03:02same data but with different function.
11:03:04So our window looks like this. It start
11:03:06with the function rank doesn't accept
11:03:08any argument inside it. Then we have the
11:03:10window like this. Order by sales
11:03:12descending from the highest to the
11:03:14lowest. And our data is already sorted
11:03:16like that. So now how is scale going to
11:03:18go and assign the ranks. The first row
11:03:19going to be the highest rank. So the
11:03:21value 100 is going to be one. Then the
11:03:23second one going to be two. But now for
11:03:25the third one, as you can see, we have
11:03:27here two values that are the same. So we
11:03:29have a tie and this time SQL going to go
11:03:32and as well let them to share the same
11:03:35rank. So both of them going to be the
11:03:37rank two. So it's not like the row
11:03:39number where we have over here three.
11:03:41This time we have two because we have a
11:03:43tie. So having same values means they
11:03:46going to share the same rank. And now
11:03:48moving to the next value going to be
11:03:49tricky one because if you check over
11:03:51here you can see that the next rank
11:03:54should be like the three right? So we
11:03:55have one two and then the next value
11:03:57that generated in the rank should be
11:03:59three but going to say you know what
11:04:02this value position going to be number
11:04:04four. So as you can see 1 2 3 four. So
11:04:07actually the position number here is
11:04:08four and going to go and give it the
11:04:11rank of four. So with that SQL going to
11:04:14be leaving a gap in the ranking. You can
11:04:16see we are skipping the rank number
11:04:18three and this always happen once you
11:04:20have a tie where you are sharing the
11:04:22same ranking. So for the next one it's
11:04:24going to be easy. It's going to be the
11:04:25row number five. So now by looking to
11:04:27the output of the rank function you can
11:04:29see that we don't have a unique ranking.
11:04:31Here we have shared ranking in case of
11:04:34the ties. So it handles the ties but
11:04:36here we have gaps in the ranks. So we
11:04:39are skipping ranks. When I think about
11:04:41the rank function I think about the
11:04:43Olympics. If two athletes tie for the
11:04:46gold medal, the first place, there will
11:04:49be no silver medal for the second place,
11:04:51the next medal going to be given to the
11:04:53bronze to the third place. All right. So
11:04:55now let's go in SQL in order to practice
11:04:57the rank function. All right. Now we're
11:04:59going to go and solve the same task but
11:05:00using the rank function. So what we're
11:05:03going to do, we're going to stay with
11:05:04the same example over here and we're
11:05:05going to rank the order based on their
11:05:07sales from highest to lowest but this
11:05:08time using the rank function. So we use
11:05:12the rank and everything inside is going
11:05:14to be empty and then our window going to
11:05:16be exactly the same as before. So over
11:05:19order by sales and disk. So let's give
11:05:23it a name sales rank. Yeah, let's give
11:05:26it a rank. So that's it. As you can see
11:05:28the syntax is very simple and very
11:05:29similar to the row number. We just
11:05:31changed the function. So now let's go
11:05:33and execute this in order to check the
11:05:35results. So now let's go and check the
11:05:37results by looking to the new rank. If
11:05:39you go and compare it with the old rank,
11:05:41we can see that we are sharing some
11:05:43ranking, right? We have here the two
11:05:44twice. So the rank number two, we have
11:05:46it twice because we have over here the
11:05:49same value. So 60 60 we have it here two
11:05:51and two. But if you compare it to the
11:05:54row number, you can see that it is not
11:05:56sharing the same ranking. So this is one
11:05:58difference. And as well here the same
11:06:00thing. They have the same value. The
11:06:01sales is 20. So we have it twice the
11:06:04rank number seven. And here we have it
11:06:06as different values. And the next value
11:06:08as you can see we are skipping the rank.
11:06:11So there is gap there is no rank of
11:06:13eight. So you can see that this is the
11:06:15row number nine and that's why it get
11:06:17the nine. The same thing I believe over
11:06:20here. So now if you check those two
11:06:21ranks the next one should be three. But
11:06:23since it is in the row number four it's
11:06:26going to get the rank four. So by
11:06:27checking the results we can see that
11:06:29sharing the same ranks and as well we
11:06:31have gaps. So this is how the rank
11:06:33works.
11:06:37All right. So what is a dense rank? It
11:06:40is very similar to the ranking function.
11:06:41It's going to go and assign for each row
11:06:43a number rank and it as well handles the
11:06:46ties. So same values they going to share
11:06:48the same ranking but this time it
11:06:51doesn't leave any gaps like the rank
11:06:53function. So the d rank it will not
11:06:56leave any gaps. It will not skip any
11:06:58ranking. So in order to understand this
11:07:00we're going to have a very simple
11:07:01example. So let's go. All right. So
11:07:03again the same data but with different
11:07:04function. We have this time the rank
11:07:06function dense rank and the window going
11:07:09to be the same order by sales descending
11:07:11from the highest to the lowest. So now
11:07:13the data is as well sorted already.
11:07:15Let's see how SQL going to go and assign
11:07:17the ranks as usual. The first row going
11:07:19to be the rank number one the second as
11:07:22well but again here we have the same
11:07:24values. So we have same values and it's
11:07:26like the rank it's going to go and share
11:07:28the same rank. So both of them going to
11:07:30has the rank number two. And now you
11:07:33might say, well this is very similar to
11:07:35the rank function. So why do we have
11:07:36dense rank? I'm going to say wait for
11:07:38it. We're going to have the difference
11:07:40in the next value. So it's going to come
11:07:42over here. This value is exactly after
11:07:44the tie. In rank SQL went and took the
11:07:47position number. So the row number it
11:07:49was four, right? So 1 2 3 4. But this
11:07:52time with the dense rank SQL will not
11:07:54leave gaps in ranking. So there will be
11:07:56no skipping the next rank in the
11:07:59sequence going to be three. So that's
11:08:01why we're going to have the rank three
11:08:03for this value. So as you can see there
11:08:05is no gap. We have one, we have two and
11:08:07three. So we are not skipping, we are
11:08:09not leaving any gaps. And the last one
11:08:11going to be four. So this is exactly the
11:08:14difference between the dense rank and
11:08:15the rank. So now by checking the output
11:08:17of the dense rank, you can see that we
11:08:20don't have unique ranks. We have here
11:08:23shared ranks. As you can see, we have
11:08:24here repetition. So, it handles the ties
11:08:27and as well it doesn't leave any gaps.
11:08:30It doesn't skip anything in the ranking.
11:08:32Okay, so that's it. Now, let's go back
11:08:34to SQL to practice the dense rank. All
11:08:36right, so now we have the same task.
11:08:37Rank the orders based on their sales
11:08:39from highest to lowest. So, we're going
11:08:41to do the same stuff, but this time
11:08:42using the function dense rank. So, dense
11:08:46rank is going to be empty. And then
11:08:47we're going to define it like all others
11:08:49over order by sales disk. And then we're
11:08:53going to give it the name of sales rank
11:08:55dense. And that's it. So as you can see
11:08:57all of those functions having the exact
11:09:00syntax, right? So let's go and execute
11:09:02it. Okay. So now let's go and check the
11:09:04results. We got our newest rank using
11:09:06the dens. And by just checking the
11:09:08results, you can see that it handles the
11:09:10tie. We have two twice, right? So let's
11:09:13check the example over here. We have the
11:09:15sales 60 twice. That's why they are
11:09:17sharing the same ranking in the dense
11:09:19and as well in the normal rank. But now
11:09:21what is interesting is the value after
11:09:24the tie. So as you can see over here
11:09:26with the dense rank we have three. So we
11:09:28didn't skip any ranking. We don't have
11:09:30any gap 1 2 and then three. But with the
11:09:33rank it's just focus on the position
11:09:36number. So it is the row number four.
11:09:38That's why it's four. With that we have
11:09:40a gap. So as you can see now we don't
11:09:42have any gaps in the dense rank. So we
11:09:44have three four five. And now we have
11:09:46over here the same two values. So we
11:09:49have sales of 2020 and they share the
11:09:51six twice. So as you can see there is
11:09:53difference now between the dense and the
11:09:55rank. So here we have seven seven but
11:09:57here we are at the rank 66. So that's
11:09:59why we have differences between them
11:10:01because we skipped before in the rank
11:10:03number three. Now the other stuff you
11:10:04can see we have seven and eight. So now
11:10:06if you compare those three ranking you
11:10:08can see that they all start with the
11:10:10rank number one but they didn't all end
11:10:13with the same ranking. So the row number
11:10:15and the rank they really focus on the
11:10:18position number or the row number of the
11:10:20orders. So you can see over here it is
11:10:23the row number 10. That's why we have
11:10:24here 10 and 10. So the scale is from 1
11:10:27to 10. And that is exactly the same for
11:10:30the row number from 1 to 10. But with
11:10:32the d over here we have it from 1 to 8
11:10:35and that's because we shared the same
11:10:37ranking and with that we wasted let's
11:10:40say few ranks. So the scale is different
11:10:42from the two others. And that's because
11:10:44we have ties twice. This is one tie and
11:10:47as well we have over here one tie.
11:10:49That's why we are missing over here two
11:10:51ranks. So this is how the dense ranks
11:10:53works. And you can go and compare now
11:10:55all three togethers in order to
11:10:57understand how those ranks are
11:11:02working. All right. So now let's quickly
11:11:04compare the three functions side by
11:11:05side. Let's start with the first point
11:11:07about the uniqueness of the rank. And if
11:11:09you compare those three you can see that
11:11:11only the row number generates unique
11:11:13distinct rank. So this going to be
11:11:16unique rank and the two others we have
11:11:18duplicates or let's say shared ranks.
11:11:20Okay. So now the second point whether
11:11:22the function handles the ties and the
11:11:25only one that doesn't handle the ties is
11:11:26the row number. So this one doesn't
11:11:29handle the ties and the two others
11:11:31handles the ties since they offer the
11:11:33shared rank. And now we have the last
11:11:35point about leaving gaps or skipping
11:11:37ranking. So now if you check the row
11:11:39number and the dense rank you can see
11:11:41there will be no skipping. So there is
11:11:44no gaps for the row number and as well
11:11:47for the dense rank only for the rank
11:11:50function the middle one we are skipping
11:11:52ranks and we are leaving gaps. So that's
11:11:54it guys. This is the differences between
11:11:56those three functions. I tend usually to
11:11:58work with the row number more often than
11:12:00that to others.
11:12:05All right guys, so now I had a look to
11:12:07those three functions and I checked my
11:12:09projects real projects and I found out
11:12:11that there are many use cases for the
11:12:14function row number compared to the
11:12:16other functions dense rank and rank. So
11:12:19now what we're going to do I'm going to
11:12:20show you a few use cases for the rank
11:12:22number that I usually use in my real
11:12:24projects in order for you to understand
11:12:26how important is the row number
11:12:28function. So let's go to SQL. All right.
11:12:30So now let's start with the first use
11:12:31case and we have the task of find the
11:12:33top highest sales for each product. So
11:12:36this is very classic in reporting or
11:12:39data analyzes. We call this top end
11:12:41analyzes. So here the managers or
11:12:44decision makers they would like to have
11:12:46the best performers or the best success
11:12:48in our data. So for example the top
11:12:50highest five customers or the top five
11:12:54products or categories and so on. So
11:12:56this is very important analyzis in order
11:12:58to focus on the best products or on to
11:13:01the most important customers and so on
11:13:03and this is as I said very classic and
11:13:05very important in order to make
11:13:06decisions in the business. So now let's
11:13:09see how we can solve this. So we're
11:13:10going to start with the usual stuff.
11:13:11Let's first select the data. So select
11:13:14order ID. Let's take as well the product
11:13:16ID and the sales from sales orders. So
11:13:21let's go and execute this. And now as we
11:13:23know that for each product we have
11:13:24multiple orders and we have multiple
11:13:26sales but we are interested only in the
11:13:28highest sales for each product. So we
11:13:30have to go and create a rank. In order
11:13:32to do that we're going to use the row
11:13:34function row number and we have to
11:13:37define the window now. So do we need
11:13:38partition by check the query. So it says
11:13:42for each product that means we have to
11:13:44divide the data by the product ID. So
11:13:46let's go and use the partition by
11:13:49products ID. And now we must use the
11:13:52order by. So order by. And now how to
11:13:55sort the data by the sales, right? And
11:13:57it is from the highest to the lowest. So
11:13:59let's go sales. And we have here
11:14:02descending. So from highest to lowest.
11:14:04Let's go and give it a name. So you're
11:14:06going to be rank by products. So let's
11:14:09go and execute this. And now by looking
11:14:12to the result, you can see that SQL did
11:14:14divide the data by the product ID. So we
11:14:16have here like around four windows. The
11:14:19first one over here you can see that the
11:14:20rank starts from one end with four. So
11:14:23the highest rank can be the order number
11:14:25eight with the sales of 90 and then it
11:14:28goes to the four. Now as you can see
11:14:31that the second window we have a new
11:14:33ranking. So it resets the first going to
11:14:35be uh the order number 10 and the last
11:14:38one going to be order number two. So as
11:14:40you can see each window has its own
11:14:42ranking and as well the last one we have
11:14:44it only as one row. So now of course in
11:14:47the task we have to return the highest.
11:14:49So we are not interested in the others.
11:14:51We have to return this row this row as
11:14:53well and this one and this one. So as
11:14:55you can see we have to return everything
11:14:57that has the rank one. We are not
11:14:59interested in the rank 2 3 4 and so on.
11:15:02So we would like to have the highest. So
11:15:04now in order to filter the data what
11:15:05we're going to do we're going to go and
11:15:07use subqueries. So select star from and
11:15:12then we're going to have the following
11:15:13condition. So where and we're going to
11:15:15say rank by product equals to one. So we
11:15:20are interested only on the rank number
11:15:22one. So let's go and execute it. And
11:15:24with that since we have four products in
11:15:26our data, we're going to have only four
11:15:27rows and we have the highest sales. So
11:15:30as you can see we have only number one
11:15:31over here. And those sales are the
11:15:33highest for each product. And with that
11:15:36we have solved the tasks by finding the
11:15:38top end analyzers.
11:15:44Okay, moving on to the next use case. We
11:15:46have the following task and it says find
11:15:48the lowest two customers based on their
11:15:50total sales. So now we have the exact
11:15:53opposite use case. We call it button in
11:15:56analyzes. So now in this example in the
11:15:58business the decision makers want to
11:16:00optimize the costs want to cut costs and
11:16:03with that they have to analyze the
11:16:05lowest performers in the products or the
11:16:08lowest performance in the employees in
11:16:10order to cut costs. So now with this
11:16:12analysis the decision makers are not
11:16:14focusing on the best successful stuff.
11:16:16We are focusing on the lowest stuff the
11:16:19lowest performers. So now let's solve
11:16:21this tasks. So now if you check the
11:16:22question we have multiple stuff right we
11:16:24have the total sales and as well we have
11:16:27to find the lowest two customers. So we
11:16:29have ranking and as well aggregations
11:16:31remember we can do stuff together with
11:16:33the group I. So now let's do it step by
11:16:35step. First let's select the data right.
11:16:37So what do we need? Order ID customer ID
11:16:40and we need the sales from sales orders.
11:16:45So let's go and execute this. So now if
11:16:47you check the customers over here we
11:16:49have around four customers and they have
11:16:51multiple sales. Now we would like to
11:16:53have the total sales for each customers
11:16:56in order to find the lowest two. So
11:16:58let's start first with the aggregations.
11:17:00So what we going to do? We're going to
11:17:01go and aggregate the sales. So the sum
11:17:03of sales and let's call it total sales.
11:17:06And now in order to do the group by we
11:17:08have to have only the customer. So group
11:17:11by and we have the customer ID. So it is
11:17:14very simple group by statements. Let's
11:17:17go and execute this. So now by checking
11:17:18the result we can see that SQL did
11:17:20aggregate the data. We have four rows
11:17:22and that's because we have four
11:17:23customers and we have their total sales.
11:17:26So we have solved the first part of the
11:17:28task. We have the total sales for each
11:17:30customers. Now let's move to the second
11:17:32part. It says lowest two customers. That
11:17:35means we have to use the ranking
11:17:37functions in order to rank those
11:17:39customers. So we are not interested in
11:17:41all customers. We are interested only in
11:17:43the lowest two. So in order to do that
11:17:45now we're going to go and use the window
11:17:47function row number. So and then over.
11:17:51Now do we have to partition the data?
11:17:53Well no we don't have to do that. We
11:17:55have now to sort the data. So order by.
11:17:58So this time we're going to go and use
11:18:00the aggregations in the order by. So the
11:18:02sum of sales and we want to have it
11:18:04sorted from the lowest to the highest.
11:18:07So I'm just going to go and use the
11:18:09defaults. So it is ascending. Now let's
11:18:11call it rank customers. So that's it.
11:18:15Again here the rule is that if you are
11:18:17using a window function together with
11:18:19the group by function, you have to use
11:18:21only columns that is used in the group
11:18:23by. So this should be working. Let's go
11:18:25and execute it. So now as you can see in
11:18:27the results, we got an extra column for
11:18:29the rank. So now the lowest customer
11:18:31going to be the customer number two. The
11:18:33second one going to be four with the 90
11:18:35total sales. And the highest customer
11:18:38with the sales is going to be the last
11:18:39one, the 125 customer number three. So
11:18:42now we have almost everything but the
11:18:44list should contain only the last two.
11:18:47So in order to do that to filter the
11:18:48data, we're going to go and use
11:18:50subquery. So select star
11:18:53from and then we have to define the
11:18:55condition where rank customers it should
11:18:59be smaller or equal to two. Right? So
11:19:01with that we will get the first two. So
11:19:03let's go and execute this. And with that
11:19:06we got the lowest two customers based on
11:19:08their total sales. So customer number ID
11:19:10you two and the four. So that's it. We
11:19:13have solved the task and now we have
11:19:14done button in
11:19:19analyzes. Okay let's keep moving to the
11:19:21next use case and we have the following
11:19:23task. It says assign unique ids to the
11:19:26rows of the table orders archive. So now
11:19:30guys we might be in situation where you
11:19:31have a table without any primary key and
11:19:34you would like to create an ID for each
11:19:36row. So in order to do that we can use
11:19:38the function row number in order to
11:19:40generate unique identifier ids for each
11:19:44row inside our table if we don't have
11:19:45one. And generating such ID for each
11:19:47row. It's very important to do stuff
11:19:49like importing data, exporting data,
11:19:52maybe joining tables as well using this
11:19:53ID or let's say optimizing the
11:19:56performance of query using the ID. So
11:19:58now let's see how we can generate that
11:20:00using row number. Okay. So now let's
11:20:02first select the table order archives in
11:20:04order to understand the content. So
11:20:06select star from sales orders archive.
11:20:11So let's go and execute. So now by
11:20:13checking the result you can see that we
11:20:14have 10 orders and we have repetitions
11:20:17in the order ID over here. So it is not
11:20:19really primary key. As you can see over
11:20:21here we have twice the ID four and here
11:20:24we have three times the ID6. So now what
11:20:26we're going to do we're going to go and
11:20:27generate unique identifier for each row.
11:20:30So in order to do that what we're going
11:20:32to do going to go over here and say row
11:20:34number and then we're going to define
11:20:37the window function. We don't partition
11:20:39the data at all but we have to sort the
11:20:41data by the order ID. So order by order
11:20:46ID or you can use something else as well
11:20:48using the order date or something
11:20:50doesn't matter. So let's add to it order
11:20:53data as well and let's call it unique
11:20:57ID. Let's go and execute this. Now by
11:21:00checking the data you can see that we
11:21:02have a new ID over here that comes from
11:21:05the row number and we have like a unique
11:21:08identifier. As you can see we have 10
11:21:10rows and with that we have as well 10
11:21:12different distinct unique ids. So with
11:21:14this as you can see we have solved the
11:21:16task and we have now a unique identifier
11:21:18an ID for the table orders archive. So
11:21:21now having this ID we can do many stuff
11:21:23like joining tables or doing something
11:21:25special and important called pagenating.
11:21:28Imagine we have like a huge table and we
11:21:30would like to retrieve the data. So now
11:21:32in order to not have all the data in one
11:21:35go we can go and divide the data by the
11:21:37primary ID or by unique identifier. For
11:21:39example, we can make a page from 1 until
11:21:42100,000 and then the second page starts
11:21:45from 100K to 200ks. So now by dividing
11:21:48the data, we can maybe improve exporting
11:21:50or importing data or we can have faster
11:21:53retrieval for the users. We don't want
11:21:54to have the whole data in one go in one
11:21:57page. So it has a lot of benefits using
11:21:59pagionating and we can do that only if
11:22:01we have a nice ID like
11:22:06this. All right. Right. Today I'm going
11:22:08to show you the last use case for the
11:22:09function row number that I usually use
11:22:11in my real projects. So sometimes if you
11:22:14are doing data analyszis you're going to
11:22:15find out that there are data quality
11:22:17issues especially with the duplicates.
11:22:19So what I usually use I use the raw
11:22:22number in order to identify the
11:22:23duplicates. Not only that I can use it
11:22:26in order to delete the duplicates. So we
11:22:28can use it in order to do data
11:22:30cleansing. And this is essential task
11:22:32for each data engineer not only data
11:22:35analysts in order to prepare and clean
11:22:37up the data before doing data analyzes.
11:22:40So let's have the following task.
11:22:41Identify duplicate rows in the table
11:22:43orders archive and return a clean result
11:22:46without any duplicates. So not only we
11:22:48have to identify the duplicates, we have
11:22:50to return no duplicates in our results.
11:22:53So let's see how we can do this. Let's
11:22:54first select the data. So select star
11:22:56from sales orders archive. So let's go
11:23:01and execute. So now by looking to the
11:23:02data you can see that we have
11:23:03duplicates. We have an issue. So the
11:23:05order ID number four is twice in our
11:23:07database. It doesn't make sense, right?
11:23:09It should be only one. So which one is
11:23:11the correct one? If you check the data
11:23:13over here, you can see that this order
11:23:15is shipped and then delivered. So it
11:23:17looks like the last one is the correct
11:23:19one. So how we can do that? If you just
11:23:22scroll to the right, you can see that we
11:23:24have a creation time. And we usually use
11:23:26such a time stamp in order to identify
11:23:29what was the last valid like order. And
11:23:32here we can see immediately that this
11:23:34order time is higher than the previous
11:23:36one. Which means this is the more up to
11:23:38date, right? The more current. So what
11:23:41we're going to do, we're going to go and
11:23:43rank our data for each order ID and sort
11:23:46the data by the creation time in order
11:23:48to find the last inserted or created row
11:23:51for this order. So let's see how we can
11:23:53do that. What we going to do? We're
11:23:55going to go over here and say let's have
11:23:57a row number and then over and what
11:24:01we're going to do, we're going to
11:24:02partition by the primary key. So
11:24:04partition by order ID and as we said we
11:24:07have to order the data by this time stab
11:24:09at the end. So partition by or order by
11:24:13creation time and descending. So we want
11:24:16the highest then the lowest. So that's
11:24:19it. Let's call it Rn and execute the
11:24:22query. So now by checking the data if
11:24:24everything is clean and we don't have
11:24:25duplicates everything should be one
11:24:28because maximum for each primary key we
11:24:30should has one row. So but you can see
11:24:32over here we have here two and we have
11:24:34here three two. So that means this is
11:24:36indicator that we have duplicates inside
11:24:38our data. So now by checking one by one
11:24:40as you can see the order ID is only one.
11:24:43So we have the rank one the second one
11:24:45as well we have the rank one but here we
11:24:47have the issue. So as you can see we
11:24:49have now two ranks for the order ID
11:24:51four. So now which one is the correct in
11:24:54our logic? We say it is the last row
11:24:56that is inserted inside our data and
11:24:58this is rank number one. So if you
11:25:00scroll to the right side you can see
11:25:01that the creation time here is higher
11:25:04than the second one. So with that we
11:25:06have identified what we want. We want
11:25:08the last inserted row for each ID. And
11:25:12now let's check this over here. So here
11:25:14we have it three times. So it says the
11:25:17first one is the highest creation date.
11:25:19So if you go to the right side and now
11:25:21by comparing those time stamps you can
11:25:22see that this record the first one is
11:25:26the la latest one that is inserted
11:25:28inside our data. So as you can see this
11:25:31one is the one that we need the other
11:25:33two we don't need it because it is old
11:25:35informations. So now everything that
11:25:37doesn't has the rank number one is not
11:25:39valid. It's something old and it's
11:25:41actually bad data quality. So we want to
11:25:44remove it or not to select it. So now in
11:25:46order to have a clean data what we going
11:25:48to do we're going to go and select the
11:25:50following as sub select. So select star
11:25:52from the table and now we are interested
11:25:56only with the rank number one. We don't
11:25:59need anything else. So let's go and
11:26:00execute. And now if you check the
11:26:02results you can check the order ID over
11:26:03here. It is unique. We don't have any
11:26:05duplicates. Right? 1 2 3 4 5 6 7. There
11:26:08is no duplicates at all. And we have now
11:26:11only the latest inserted data inside the
11:26:13orders. and we don't have any duplicates
11:26:16or data quality issue. So now of course
11:26:18now we can go with this results in order
11:26:19to do for the analyzes and this is
11:26:21exactly what data engineers usually do
11:26:23clean up the data and prepare the data
11:26:25before doing any data analyzes. And of
11:26:28course if you want to communicate those
11:26:30data quality issues to the source of the
11:26:32data let's say you are not the owner of
11:26:33those informations. You can generate a
11:26:35list of all bad data quality issues and
11:26:38you can send it to the source system and
11:26:39tell them to clean it up from the
11:26:41sources. So now in order to select the
11:26:43bad data what we're going to do is we
11:26:45can just change here the condition and
11:26:47say if it is higher than one then you
11:26:50are like bad data. So let's go and
11:26:52execute this. And now with this we have
11:26:54in the results all records that
11:26:56shouldn't exist in the data in the first
11:26:58place. So we can go and export it and
11:27:00communicate it to the source and tell
11:27:01them check here you have something wrong
11:27:03in your system and those information
11:27:05should not be inserted in the data. So
11:27:07everyone it is very strong right? It is
11:27:09very powerful. I use it a lot in my
11:27:11projects. There are many use cases for
11:27:13the row number function in SQL. We can
11:27:15do it in order to find the top end
11:27:18analyzes, the bottom end analyzes, the
11:27:20best performance, worst performance and
11:27:22as well we can assign unique ids to do
11:27:24benating or we can use it in order to
11:27:26discover data quality issues to clean up
11:27:29our data. So it is amazing function in
11:27:31SQL and you're going to use it a lot. So
11:27:33that's it for the three functions ro
11:27:34number, rank and dense rank. Now we're
11:27:36going to talk about the
11:27:40inile. Okay. So what is inile? Intile in
11:27:44SQL is very simple. It's going to go and
11:27:45divide your rows, your data into
11:27:48specific number of almost equal groups
11:27:51or sometimes we call them packets. So
11:27:53now in order to understand this and how
11:27:55it scale works with this function, we're
11:27:57going to have a very simple example. So
11:27:58let's go. Okay, we have the following
11:28:00setup. We have four rows for sales and
11:28:02we would like to divide it into two
11:28:04groups or into two buckets. So in order
11:28:07to do that we can use the entile
11:28:09function. It has different syntax than
11:28:11the other ranking functions. So it
11:28:13starts with entile then we must define a
11:28:16number. So we cannot leave it empty like
11:28:18the other ranking. So here we have two
11:28:20buckets then over and here again we have
11:28:23to sort the data. So it is must order by
11:28:26sales descending from the highest to the
11:28:28lowest. So now as usual SQL going to go
11:28:30and sort the data. We have it already
11:28:32sorted in this example. Then it going to
11:28:34start assigning each of those rows into
11:28:36buckets. But SQL first has to calculate
11:28:40the bucket size. So how many rows we can
11:28:43like insert inside each bucket. So the
11:28:45calculation is very simple. It says the
11:28:47bucket size equals to the number of rows
11:28:51divided by the number of buckets. So
11:28:53what is the number of rows here? We have
11:28:55four rows, right? So we have four over
11:28:58here. Then the number of buckets we
11:29:00define it in the syntax of the query. So
11:29:02here we defined two buckets. We need two
11:29:04groups. So that means we are dividing
11:29:07four by two. And the size of the bucket
11:29:09going to be two. So now with this SQL is
11:29:11ready and going to start assigning each
11:29:13row to a bucket. So it's going to start
11:29:15on the top. The first one going to be in
11:29:16the bucket number one. Then go to the
11:29:19next one. It's going to say okay we
11:29:20still have enough space in the bucket.
11:29:22Right? So it's going to sign as well to
11:29:24one. But with this we reach the maximum
11:29:27number of rows within each bucket. So
11:29:29the next row going to be assigned to
11:29:31another bucket. So it's going to be two
11:29:33and the last one going to be as well
11:29:35too. So as you can see it's very simple.
11:29:37We have just assigned our sales based on
11:29:39the sorting of course into two buckets.
11:29:41These two sales belongs to the bucket
11:29:43number one and the other two belongs to
11:29:45the bucket number two. Very easy. So
11:29:47that was very straightforward because we
11:29:49are dividing even numbers and we got
11:29:51perfectly sized buckets. But now what
11:29:53going to happen if we have an odd
11:29:55number? So we have here five instead of
11:29:57four. So the bucket size going to be
11:30:00dividing five by two. We're going to get
11:30:022.5. And now of course SQL will not go
11:30:04and divide like two half for each
11:30:06bucket. Then we are splitting this into
11:30:08two packets. Of course this will not be
11:30:10working. We should has now a bucket with
11:30:12three and another bucket with two. So
11:30:14now the rule in SQL make it very clear.
11:30:17It says larger groups comes first then
11:30:20smaller. So that means if we have here
11:30:22an even number like this, the larger
11:30:24group going to be the first group. So
11:30:26that's going to look like this. It's
11:30:28going to like reset everything. So let's
11:30:32see what's going to happen. The first
11:30:33one going to be one. The second one has
11:30:35bill one. The third one going to be as
11:30:37well one. So it going to has a larger
11:30:40package than the second one. Then the
11:30:43rest going to be two. So as you can see
11:30:45the larger group comes first then the
11:30:48smaller. And this is how a scale going
11:30:50to work. if you have odd numbers. So you
11:30:52don't have here perfectly sized buckets.
11:30:54You have approximately or roughly
11:30:56equally sized buckets. So this is how
11:30:58the intel works. Now let's go back to
11:31:00scale in order to practice this
11:31:01function. Okay. So now let's have some
11:31:03fun working with this function. So we
11:31:05just going to select something like
11:31:07order ID sales from sales orders. So
11:31:13let's go and execute it. And with that
11:31:15we got our 10 rows. Now let's say that I
11:31:17would like to create only one bucket
11:31:19from the data. So entile and only one
11:31:23bucket over partition let's say not
11:31:26partition by let's take order by sales
11:31:29descending. So that's it. I'm going to
11:31:32call it one bucket. So let's go and
11:31:35execute it. As usual it's still going to
11:31:37go and sort the data and then calculate
11:31:38the bucket. It's going to be 10 rows
11:31:41divided by one. So the size of the
11:31:43bucket going to be 10. So that's why
11:31:45you're going to see everywhere here as
11:31:47one because all those rows going to fit
11:31:49into one bucket. So this is very simple.
11:31:52We have only one bucket. Let's go and
11:31:54now have two buckets. So I'm just going
11:31:56to copy and paste. And instead of one,
11:31:59we're going to have two and let's call
11:32:00it two buckets. So let's go and execute
11:32:04this. So now here again, what is the
11:32:06size of the buckets? It is 10 divided by
11:32:08two. So we will get perfectly grouped
11:32:11buckets. So the first bucket going to be
11:32:13five rows and the second one going to be
11:32:15the next five rows. So it is very
11:32:17perfect. Let's go to the next one. Let's
11:32:20have three buckets. So three. So let's
11:32:24go and execute. So now what going to
11:32:26happen is going to go and divide 10 by
11:32:28three in order to get the size of the
11:32:30bucket. And it's going to be 3.3. So it
11:32:33is decimal and we will not get perfectly
11:32:35sized buckets. So again the larger group
11:32:38comes first then the smaller. So as you
11:32:41can see we have to fit then in the first
11:32:43group four in order to get the others
11:32:45with three. So that's why the first
11:32:47bucket is going to be the biggest one.
11:32:49So four rows into the first bucket. Then
11:32:51the second three rows going to be in the
11:32:53bucket two. And as well the last one
11:32:55going to be bucket three. So as you can
11:32:57see the largest group is going to be the
11:32:59first bucket. So now let's keep playing
11:33:01with the data. Let's go and take now
11:33:04four. We would like to have four
11:33:05buckets. Now things going to get
11:33:08interesting. So now by checking the
11:33:09result it's going to be interesting. SQL
11:33:11going to divide 10 by four and we will
11:33:13get something like 2.5. So again we will
11:33:16not get perfectly sized groups. So SQL
11:33:18has to fit now 10 rows into four groups.
11:33:22So the first three rows going to be fit
11:33:24in the bucket number one and as well the
11:33:27second three rows like this going to be
11:33:29in the bucket number two. And then you
11:33:32can see over here we have two buckets
11:33:34with a size of two. And with that we can
11:33:36fit 10 into four groups. And again you
11:33:38can see the larger groups comes first
11:33:41like this one and then the second and
11:33:43the smallers comes later. Okay. So this
11:33:46is how the inter works in SQL. And now
11:33:47you might say you know what why do I
11:33:49need buckets in the first place. So what
11:33:51is the use
11:33:55case? There is two use cases for the
11:33:58intel function in my projects. In one
11:34:00hands if I am data analyst I'm going to
11:34:02use the intel function in order to
11:34:04segment my data. In the other hand, if
11:34:06I'm data engineer, I'm going to use the
11:34:08intel function in order to do ETL
11:34:10processing and as well to do load
11:34:13balancing. So now let's start with the
11:34:14first use case as a data analyst where
11:34:17you want to do segmentations with the
11:34:18entire function. Segmentations is very
11:34:21nice way in order to understand your
11:34:22data. So you can go and segment your
11:34:25data into different buckets or groups
11:34:27like for example doing segmentations for
11:34:29the customers. So you can go and group
11:34:32up your customers depend on their
11:34:33behavior like the total sales or the
11:34:36total number of orders. So with that you
11:34:38can make like for example VIB section
11:34:41and then the medium and then the low. So
11:34:43now in order to understand the
11:34:44segmentation use case let's have the
11:34:46following task. Okay. The task says
11:34:48segment all orders into three categories
11:34:51high medium and low sales. So in order
11:34:54to solve this let's do the basic stuff
11:34:57right. So select order ID. Let's take
11:34:59the sales from our table sales orders
11:35:03and let's go and execute it. So as usual
11:35:06we got our 10 sales. So now if you check
11:35:08the task it says we need three
11:35:10categories. So that means we need three
11:35:12buckets right and it says high, medium
11:35:15and low sales. So that means we are
11:35:18dividing by the sales. So let's go and
11:35:20do it step by step. So we're going to
11:35:22use inile since we need to segment the
11:35:24data. Three categories means three
11:35:26buckets. And then let's define the
11:35:29window over we don't have to divide the
11:35:31data by partition by we just need to
11:35:33sort it first by the sales. So it's
11:35:36going to be by sales and let's take
11:35:38discrete since we want to sort it from
11:35:40the highest to the lowest. So that's it.
11:35:42Let's say you are our buckets. So let's
11:35:46go and execute this. So now if you check
11:35:48the data you can see that they are
11:35:49segmented into three buckets. So the
11:35:52first bucket going to contain all orders
11:35:54with the high sales. Then the second one
11:35:57going to be all sales with the medium.
11:35:59And then the last one going to be all
11:36:01sales with the low sales. So as you can
11:36:03see we have already categorized our data
11:36:06into three groups. But now as you can
11:36:08see we have numbers and maybe the user
11:36:10is expecting to have those text high,
11:36:12medium, low. So that means what we're
11:36:14going to do now we're going to go and
11:36:15translate those numbers into text into
11:36:17words. And of course we cannot do that
11:36:19inside the window function. We're going
11:36:21to use data transformation using the
11:36:23case when statements. Don't worry about
11:36:25it. We're going to have complete
11:36:26dedicated section explaining the case
11:36:28when. So for now just follow me in order
11:36:31to see how this works. We're going to go
11:36:33and use subquery. So it's going to be
11:36:35select and let's take the star for
11:36:38everything and then let's have the
11:36:39following logic. Case when buckets equal
11:36:43to one then it is high the sales is
11:36:46high. So we are just mapping the numbers
11:36:49into text. So otherwise case when the
11:36:52brackets equal to two then we are
11:36:54targeting the medium medium and then the
11:36:58last group packets equal to three then
11:37:02those sales are low. So let's call it
11:37:05end it and let's call it sales
11:37:08segmentations. So that's it. Let me just
11:37:10make it a little bit smaller in order
11:37:12for you to see it. And all right so from
11:37:17and then we have our subquery like this.
11:37:20So as you can see we just mapped the
11:37:22numbers into text. We are just doing
11:37:25translations. So let's go and execute
11:37:27it. And now by checking the results we
11:37:29got our three categories for the users.
11:37:32So the first category going to be the
11:37:33high sales. The second one going to be
11:37:35the medium sales and the third one going
11:37:37to be the low sales. So guys you see
11:37:39Intel is very powerful in order to
11:37:41segment our data. So now you can go and
11:37:43segment stuff like the customers by
11:37:46their total sales or the products by
11:37:48prices, employees by their salaries and
11:37:51so
11:37:54on. All right. So this is the first use
11:37:56case for the Intel function as a data
11:37:58analyst where you go and segment your
11:38:00data in order to understand the
11:38:01behavior. Now in the other hand, if you
11:38:04are data engineer, you can use Intel
11:38:06function in order to do load balancing
11:38:08in your ETL. So now I'm just going to
11:38:10explain it in very simple sketch. All
11:38:12right. So now we have the following
11:38:13scenario where we have two databases and
11:38:15we would like to move one big table from
11:38:18the database A to database B. So in this
11:38:21case I'm doing something called full
11:38:22load. That means I'm loading all the
11:38:24rows from one database to another. So if
11:38:27you do it in one go what could happen is
11:38:30that it could take long time. So it
11:38:32could take hours or even sometimes days
11:38:34and maybe at the end you will get maybe
11:38:36some network errors because you have
11:38:38stressed the networks between those two
11:38:40databases and everything going to break
11:38:41and you're going to lose the data and
11:38:43you have to start again. So now instead
11:38:44of loading this table in one go what we
11:38:47can do we can go and split it into
11:38:49fractions or let's say packets. So we
11:38:51can split this table for example into
11:38:53four small tables using the function
11:38:56entile. So now after we split this big
11:38:58table into small tables, we're going to
11:39:00go and start moving those small tables
11:39:02one after another and with that we are
11:39:05not stressing the networks and it's
11:39:06going to succeed. So now after loading
11:39:08everything at the end in the target
11:39:10database we're going to have those small
11:39:12tables and of course we can go and use
11:39:14the union in order to merge them in
11:39:16order to have again the big table that
11:39:18we have it in the original database. So
11:39:20this is very common use case for the
11:39:22entile in order to split the load and to
11:39:24balance the processing of extracting
11:39:27data. All right. So now we have the
11:39:28following SQL task. It says in order to
11:39:30export the data divide the orders into
11:39:33two groups. So let's go and do that.
11:39:35First we're going to select everything
11:39:37from the table just in order to see the
11:39:39data sales orders. So let's go and
11:39:43execute it. So now we got our 10 orders
11:39:45and what we have to do is that to go and
11:39:47split it into two groups. In order to do
11:39:49that we can use the entile function. Two
11:39:53groups means two buckets. So let's
11:39:55define the window. So here we don't have
11:39:57to partition the data using partition by
11:39:59but we have to specify the order by. So
11:40:02now which column we're going to use in
11:40:04order to sort the data. Of course here
11:40:06there is no rule like you can go and
11:40:07split the data by sales or by the order
11:40:10status by date by anything you want. But
11:40:12we usually go and use the primary key.
11:40:15It's just systematic, better, and more
11:40:17clean, especially if you have a sequence
11:40:19of numbers in the order ID. So you can
11:40:21export the first range of the orders,
11:40:23then you can go to the next group and so
11:40:25on. So let's go with the order ID and
11:40:28let's give it a name buckets. So that's
11:40:31it. Let's go and hit execute. Now, as
11:40:33you can see, it's very simple. We got
11:40:35our two groups. So this is the first
11:40:37batch of of the data and this is the
11:40:40second batch of data. So now we can go
11:40:42and select the first batch and export
11:40:44it, import it in the next system. And
11:40:46then after that we go with the second
11:40:48batch. And of course if you still suffer
11:40:49from the size of those packets, you can
11:40:51go and split it to more smaller size. So
11:40:54you can go over here and make it four.
11:40:56So with that we're going to get smaller
11:40:58buckets and it might be easier to export
11:41:01the data. So this is really great use
11:41:02case for the entile function. All right
11:41:04everyone. So with this you have learned
11:41:06the two use cases for the entile
11:41:07function that I usually follow in my
11:41:09projects. So as a data analyst you can
11:41:11use it in order to do segmentations and
11:41:13as a data engineer you can use it in
11:41:15order to do load balancing of the
11:41:20ETL. Okay everyone so with that we have
11:41:23covered everything about the integer
11:41:24based ranking functions. Now we're going
11:41:26to talk about the second methods. We
11:41:28have the percentagebased ranking
11:41:30functions and here we have two functions
11:41:32the cubist and as well the percentile.
11:41:35So now let's have a quick recap. So with
11:41:37the percentage based ranking SQL going
11:41:39to go and calculate a relative position
11:41:42as a percentage and assign it for each
11:41:44row. So the output going to be a
11:41:46continuous normalized scale from 0 to
11:41:50one. And this is really amazing in order
11:41:52to do distribution analyszis. So those
11:41:54functions going to consider in their
11:41:56calculation the overall total the whole
11:41:59size of the data set which can help us
11:42:01in order to find out the contribution of
11:42:03each value to the overall total. And now
11:42:06in SQL in order to generate the
11:42:08percentage we have two different
11:42:10formulas. So in one hand we have the
11:42:12function cumist and in the other hand we
11:42:15have the percent rank. So that means we
11:42:17have two different functions with
11:42:18different formulas in order to generate
11:42:20and calculate the percentage. So now
11:42:23let's start with the first function the
11:42:28cumist. All right everyone. So now let's
11:42:30start with the first function. We have
11:42:32the dis and it stands for
11:42:34commumulative distribution. It's going
11:42:36to go and focus or calculate the
11:42:38distribution of your data points within
11:42:40a window. So what this means in order to
11:42:42understand it, we're going to go and
11:42:44have very simple example to understand
11:42:46how SQL works with this function. So
11:42:48let's go. All right. Again we have our
11:42:49very simple example of the sales and we
11:42:52have the following query. So dist
11:42:55then we don't give any argument inside
11:42:56it. So it's going to be empty and the
11:42:58window going to be like usual order by
11:43:00sales descending from the highest to the
11:43:02lowest and the order by is must. So the
11:43:05first step is SQL going to go and sort
11:43:06the data. We have it already sorted from
11:43:08the highest to the lowest. So now the
11:43:10next step is that SQL going to go and
11:43:11start calculating the percentage for
11:43:13each row. And we have a very simple
11:43:16formula. It says the cumist equals to
11:43:18the position number of the value divided
11:43:21by the number of rows. So now the next
11:43:24step is still going to go and start
11:43:25calculate the percentage for each row.
11:43:27And we have this very simple formula. It
11:43:30says the cubist equals to the position
11:43:32number of the value divided by the
11:43:35number of rows. It's very simple. Let's
11:43:37do it step by step. So SQL going to
11:43:39start with the first value in our list.
11:43:41So it going to be calculated like this.
11:43:43So what is the position number of the
11:43:45first value? It's going to be one,
11:43:47right? So this is the first value in our
11:43:50list. And what is the total number of
11:43:52rows? We have five rows, right? So 1 2 3
11:43:544 5. So we're going to divide one by
11:43:57five. And the result going to be 0.2. So
11:44:01this going to be the first value for the
11:44:02first row. Okay. So now SQL going to go
11:44:04to the next row. And this time we're
11:44:06going to get a special case. As you can
11:44:08see, we have the 80 twice. So we have
11:44:10here a tie. So now first we need the
11:44:12position number. As you can see, we are
11:44:14at the position number two, right? But
11:44:17since we have the 80 multiple times, SQL
11:44:20going to go and take the last position
11:44:22that we see the value 80 and the last
11:44:24position going to be the record number
11:44:27three. So that's why SQL going to say
11:44:29for this record it's going to be the
11:44:31position number three and not two and
11:44:34then it's going to go and divide it by
11:44:35five and we will get the value of 0.6.
11:44:39So this is the most confusing thing with
11:44:40this function. So if SQL finds a tie, it
11:44:43will completely ignore the current
11:44:45position number. So we don't have two.
11:44:47It going to go and take the last
11:44:49position number for the same value. And
11:44:52the last in our list going to be the
11:44:54record number three. So that's why we
11:44:56have three over here. Okay. So now let's
11:44:58keep moving. Let's go to the third row.
11:45:00And as you can see, we are again in the
11:45:02tie. But this time, this is the last
11:45:04time we see 80. So next we don't have
11:45:0680. So what's going to happen? We're
11:45:08going to have exact same result. So it's
11:45:10going to be 3 divided by 5. So as you
11:45:13can see if we have a tie they going to
11:45:15share the same percentage. So that means
11:45:17with the cube list if you have same
11:45:19values they going to share the same
11:45:21rank. So let's keep moving to the fourth
11:45:24one. So now what is the position number
11:45:26of the 50 we are at the record four. So
11:45:29position number four divided by five we
11:45:32will get 0 comma 8. Okay. So now let's
11:45:35move to the last one and it is the
11:45:37easiest one. So which position do we
11:45:39have over here? It is the position
11:45:41number five. It's the last one. And the
11:45:43number of rows is five. That's why we
11:45:45will get one. So guys, that's it. This
11:45:47is how the cumulative distribution
11:45:48works. Once you understand the formula,
11:45:51it's going to be very easy in order to
11:45:52understand the output. So as you can
11:45:54see, calculating the percentage always
11:45:56depends on the total size of our data
11:45:58sets. You can see here the number of
11:46:00rows. So with this we're going to get an
11:46:02output that help us in order to
11:46:03understand the distribution of our data
11:46:05points within the data
11:46:10sets. All right everyone. So now we're
11:46:12going to go and focus on the second
11:46:13function that generate percentage as a
11:46:16rank. We have the percent rank. So the
11:46:18percent rank going to go and focus on
11:46:20generating the relative position of each
11:46:23row within a window. So in order to
11:46:26understand what this means, we can have
11:46:27a very simple example in order to
11:46:29understand how scale works with this
11:46:31function. So let's go. Okay, again we
11:46:33have those sales very simple example and
11:46:35the syntax going to be like this percent
11:46:37rank and inside it we don't use any
11:46:39arguments and the window going to be
11:46:41like this order by it is a must sales
11:46:44descending from the highest to the
11:46:46lowest the first step that is going to
11:46:48do is that it's going to go and sort the
11:46:49data from the highest to the lowest and
11:46:51we have it already like this and next
11:46:53SQL going to go and start calculate the
11:46:54percentage which is very similar to the
11:46:57cumulative distribution but this time
11:46:59it's going to be like this position
11:47:01number then we subtract it from one and
11:47:04as well divided by the number of rows
11:47:07subtracted from one. So it's like exact
11:47:10formula but we are only subtracting here
11:47:12once for both numbers. Okay. So now
11:47:14let's go through all rows step by step
11:47:16and see the output. So it's still going
11:47:18to start with the first row right. So
11:47:20what is the position number of the first
11:47:22row? It's going to be one. Then we have
11:47:24to subtract it by one. That's why we
11:47:26will get zero. Now what is the total
11:47:28number of rows? So we have here five
11:47:30rows and it is subtracted by one that's
11:47:32why we're going to get four. So now 0
11:47:35divided by any value the output going to
11:47:37be a zero. So that's why for the first
11:47:39value we will get a zero. All right. So
11:47:41now let's move to the second row over
11:47:43here. And here we have our special case
11:47:45where we have a tie. So we have two
11:47:47sales sharing the same value 80. So now
11:47:50for the percent rank SQL gonna have
11:47:51different behavior than the cumist.
11:47:54Remember in the list SQL did search
11:47:56for the last position of the shared
11:47:58value. So it was the position number
11:48:00three since this is the last time we see
11:48:0380. But now with the person rank is
11:48:05still going to stick with the first
11:48:06occurrence of the shared value. So now
11:48:08by checking those two 80s what is the
11:48:10first occurrence? It is the record
11:48:12number two. So that's why we have
11:48:14position number two subtracted by one we
11:48:16will get one. And here the same going to
11:48:18be number of totals we have five
11:48:21subtract by one we have four. So now if
11:48:22you divide one by four we will get the
11:48:24result of 0 comma 25. So this is the
11:48:28percentage of this value. So now let's
11:48:30go to the second row. Here we have again
11:48:31the tie. So scale going to stick with
11:48:34the position number two the first
11:48:35occurrence. So it's going to be the same
11:48:38two subtracted by one we will get one.
11:48:40And as well the total number of rows
11:48:42five subtract by one we will have four.
11:48:44That's why we will get the same exact
11:48:46results. So here as you can see with the
11:48:48percent rank it's like the list the
11:48:50shared value going to share as well the
11:48:52same percentage rank. Now let's move to
11:48:54the fourth one. So we have the value 50.
11:48:57So what is the position of this? It's
11:48:58going to be the record number four.
11:49:00Subtract it by one we will get three.
11:49:03And if you divide three by four you will
11:49:05get
11:49:060.75. And now moving to the last value
11:49:09over here it's going to be easy. So what
11:49:11is the position number of the 30? It is
11:49:13five. Five subtracted by one it's going
11:49:15to be four. And as well we're going to
11:49:17have four as well here for the total
11:49:19numbers subtracted by one. So if you
11:49:21divide four by four you will get one. So
11:49:23that's it guys. This is how the percent
11:49:25rank works. It always has the scale from
11:49:280 to one. So it's always like this.
11:49:30Doesn't matter which values do we have
11:49:32inside and it's going to has like
11:49:34continuous scale. And again here if you
11:49:36have a tie they're going to go and share
11:49:37the same percentage rank. Okay guys. So
11:49:39now if you go and compare those two
11:49:41functions you're going to see that they
11:49:42are really similar to each others. The
11:49:43output of both functions we are
11:49:45generating percentage based ranking and
11:49:48both of them as well handling the ties
11:49:50perfectly. So they share the same
11:49:52percentage rank. If you check the syntax
11:49:54they are very similar. And now by
11:49:55checking the formulas of both of them we
11:49:57are always considering the overall size
11:50:00of the data sets. So here the size is
11:50:03considered in the calculation to help us
11:50:05finding the relative position of each
11:50:08value to the overall and this is very
11:50:10important in the analyszis in order to
11:50:12measure the contribution of each value
11:50:14to the overall. So now about the use
11:50:16cases if you want to focus on the
11:50:18distribution of your data points go with
11:50:20the cumulative distribution but if you
11:50:22want to focus on the relative position
11:50:24of each rows then go with the percent
11:50:26rank. All right. So now there is one
11:50:28more difference between the and the
11:50:30percent rank and that's if you check the
11:50:32formulas. You can see that the is
11:50:35more inclusive. We always consider the
11:50:38position number of the current row. But
11:50:40with the person rank we don't consider
11:50:42the current row. We like skip it or make
11:50:45it exclusive. So we say for the person
11:50:47rank it is more exclusive and the
11:50:50cumulative distribution it is more
11:50:52inclusive. So now if you ask me the hard
11:50:53question which one to use, I'm going to
11:50:55say if you want to be more inclusive, go
11:50:58with the commutive distribution. If you
11:51:00want to be more exclusive with the
11:51:01current row, go with the person rank. So
11:51:04they are very similar to each others. So
11:51:06if you want to calculate the
11:51:07distribution of your data, go with the
11:51:08cumulative distribution. If you want to
11:51:10find the relative position of each row,
11:51:13then go with the percent rank. All
11:51:15right. So now we have the following task
11:51:16that says find the products that fall
11:51:19within the highest 40% of the prices.
11:51:22Let's go and solve this. Now we are
11:51:23targeting the table products and I will
11:51:26just select like two columns products
11:51:28price from sales products. So that's it.
11:51:33Let's go and execute this. So now as you
11:51:36can see we got five products and their
11:51:37prices. And the task says find the
11:51:40highest 40%. So we have to find and
11:51:42generate a percentage rank. In order to
11:51:45do that we have the two functions cumist
11:51:47and the percent rank. I will go this
11:51:49time with the list. So let's go and
11:51:52do that. So list and then let's go
11:51:55and define the window like this. It's
11:51:58going to be order by we are targeting
11:52:00now the prices right? So order by the
11:52:03price from the highest to the lowest and
11:52:05let's go give it a name this rank. So
11:52:08let's go and execute this. So now with
11:52:10that SQL going to go and generate for us
11:52:12a percentage ranking using the formula
11:52:14that we just learned before. So now in
11:52:16the output we are getting all the
11:52:17products but the task says we have to
11:52:19get only the products that are in the
11:52:21highest 40%. So that means the first row
11:52:24the second row and that's it. So those
11:52:27rows are in the highest 40% the rest are
11:52:31below that. So in order to do that to
11:52:32filter the data we're going to use the
11:52:34subquery. So select star from and then
11:52:38we have our sub query like this and then
11:52:41our filter going to be this rank smaller
11:52:45or equal to 0.4. So this is our
11:52:49threshold in order to get the data. So
11:52:51let's go and execute this. And now as
11:52:53you can see we got the top products the
11:52:55top 40%. Now of course you can go and
11:52:58format the percentage. We can do that
11:53:00like this. So let's take the test
11:53:05rank multiply it with 100. So let's go
11:53:08and execute this. So as you can see we
11:53:09got 20 and 40%. We can go and add to it
11:53:13as well the percentage character right.
11:53:15So we can go and say concat and we're
11:53:18going to add the character after that
11:53:21like this and let's call it test rank
11:53:25percentage. So that's it. Let's go and
11:53:27execute it. So that we have solved the
11:53:29task. We have the products that fall
11:53:31within the highest 40%. Now, of course,
11:53:33you can go and try the percent rank. So,
11:53:36it's very simple. We just have to go and
11:53:38switch the cumulative distribution with
11:53:40the function percent bank. So, let's go
11:53:44and execute it. Now, as you can see, we
11:53:46will get the exact same results. So,
11:53:48we're still getting the gloves and caps
11:53:50as the highest products within the 40%
11:53:53of the price. So, guys, that's it. It's
11:53:54very simple, right?
11:53:59All right friends, so now let's have a
11:54:01quick recap for the window ranking
11:54:02functions. So what they're going to do,
11:54:04they're going to go and assign a rank
11:54:05for each row within a window. And we
11:54:08have two types of ranking, right? The
11:54:10first one is the integer based ranking.
11:54:12It's going to go and assign a number an
11:54:14integer for each row. And here we have
11:54:16four functions. Row number, rank, dense
11:54:19rank, and in tile. And the second type
11:54:21of ranking, we have the percentage based
11:54:23ranking. So scale fair is going to go
11:54:25and calculate a rank and then assign it
11:54:27for each row. And here we have two types
11:54:29of formula or functions. So we have the
11:54:31cube dist the cumulative distribution
11:54:34and the second one we have the percent
11:54:35rank. And now to the next point if we
11:54:37are talking about the rules of the
11:54:38syntax. So the expression should be
11:54:41empty. We should not pass any argument
11:54:42to the functions. We must use order by
11:54:45in order to sort our data. So it is
11:54:47required and the frame clause are not
11:54:50allowed to use. So you cannot go and
11:54:51customize a frame within the window
11:54:54function. And as we learned there are
11:54:56many use cases for the ranking
11:54:57functions. For example, we have the top
11:54:59end analyzes the button end analyzes in
11:55:02order to identify our top performers or
11:55:04the worst performers in our business.
11:55:07Another use case using the row number we
11:55:09can identify and remove duplicates in
11:55:11our data. So we can use it in order to
11:55:14find data quality issues and as well to
11:55:16improve the quality. And another use
11:55:18case if our table don't have a clean
11:55:20primary key we can go and generate
11:55:22unique ids using the row number in order
11:55:24to do as well by generating one more use
11:55:26case it was the data segmentations you
11:55:29can use the intel in order to segment
11:55:31your customers your products employees
11:55:33and so on and another use case we can do
11:55:35data distribution analysis as we learned
11:55:37we can use the cubeist in order to
11:55:39understand the data distributions of our
11:55:42data points compared to the overall and
11:55:44the last use case it's more for data
11:55:46engineering we can use the intel
11:55:47function in order to equalize the
11:55:49loading process of our ETLs. So as you
11:55:51can see there are many use cases for the
11:55:53ranking functions. Okay, so that's all
11:55:55about how to rank your data using the
11:55:57window functions and now we're going to
11:55:59cover the last group. We will learn
11:56:00about the value window functions. How to
11:56:03access another records. So let's
11:56:08go. All right everyone. So now we have
11:56:10this very simple example. We have the
11:56:12months and the sales. Now we can use the
11:56:15value functions in order to access a
11:56:17value from another row. So in order to
11:56:20understand it let's say that SQL now
11:56:22processing the months and we are
11:56:24currently at the month of March. So now
11:56:26for example I would like to access the
11:56:28value from the previous month from
11:56:30February. So in order to do that we can
11:56:32use the lag function in order to get the
11:56:35value of 10. So with that we have in the
11:56:37same row the current sales of the month
11:56:40March and as well the sales from the
11:56:42previous month the February. And maybe
11:56:45in other cases I would like to get the
11:56:47sales of the next month from April. In
11:56:50order to do that we can use the function
11:56:52lead and we will get at the same row the
11:56:55value five. So now I can very quickly
11:56:57compare the current month with the
11:56:59previous month and as well with the next
11:57:02month. And now in the other cases you
11:57:03might be interested in the first month
11:57:06of your list. So it's going to be here
11:57:07January. So in order to get the sales of
11:57:10the first month you can use the function
11:57:13first value. So we're going to get at
11:57:15the same row 20. And now for the last
11:57:17option I think you already get it. We
11:57:19can go and get the value of sales of the
11:57:22last month. So here we can get the July.
11:57:24So for that we're going to use the
11:57:25function last value and we will get the
11:57:28value of 40. So this is exactly the
11:57:30purpose of the value functions or
11:57:32analytical functions. We can access a
11:57:35value from another rows. And here is
11:57:37really important to understand as well
11:57:38the value functions is like the ranking
11:57:40functions. We have to use the order by
11:57:43in order to sort the data in order to
11:57:45understand what is the first row and the
11:57:47last row. In this example, the data is
11:57:49sorted by the month. So guys, the access
11:57:51functions are really important for
11:57:52analytics. You can use it in order to
11:57:54access a value from other rows in order
11:57:57to do comparison. All right. Right. So
11:57:58now let's have a quick overview of the
11:58:00syntax and the rules for the value
11:58:02functions. So here we have four
11:58:04functions lead, lag, first value and
11:58:06last value. So as you can see we can
11:58:08group them into two groups. So we have
11:58:10the lead and lag. They are very similar
11:58:12to each others. Especially with the
11:58:13syntax we can use three things or three
11:58:16arguments inside it. Expression offset
11:58:18default for both of them. For the first
11:58:21value we can use only an expression. So
11:58:23that means we have to pass a value for
11:58:26those functions. You cannot leave it
11:58:27empty. So now about the expression data
11:58:29type, you can use any field with any
11:58:31data type. There is no restrictions
11:58:32about only for example using numbers.
11:58:35Any data type is allowed. Now about the
11:58:37definition of the window. The partition
11:58:39by as usual is optional like any other
11:58:41group. The order by here is a must. You
11:58:44must define an order by. It's like the
11:58:46ranking. So here you cannot leave it
11:58:48empty. And now we come to the last one.
11:58:50We have the frame clause. There are
11:58:52really different stuff over here. So for
11:58:54the first two functions lead and lag you
11:58:56are not allowed to define any frame. So
11:58:58you are not allowed to define any subset
11:59:00of data. It's very similar to the
11:59:02ranking. So you must use order by but
11:59:05you cannot define the frame of the
11:59:07window. But for the other two functions
11:59:09the first value and the last value they
11:59:11are optional. You can go and use them.
11:59:13And for the last value it is recommended
11:59:16to define frame close. Don't worry about
11:59:18it. We're going to have enough examples
11:59:19in order to understand. So as you can
11:59:21see those functions has different
11:59:22requirements. So there is no generic
11:59:24rule for all of them. But one thing that
11:59:26they all agree on that you must use
11:59:29order by. So now as usual what we're
11:59:31going to do we're going to go and deep
11:59:32dive into those functions. We're going
11:59:34to address first the two functions lead
11:59:36and lag because they are very similar to
11:59:38each others. We're going to understand
11:59:40the use cases when to use them and of
11:59:42course we're going to practice in SQL.
11:59:44So let's go.
11:59:48lead and lag functions. The lead
11:59:50function can allow you to access a value
11:59:52from the next row within a window where
11:59:56the lack function is exactly the
11:59:57opposite. It's going to allow you to
11:59:59access a value from a previous row
12:00:02within a window. It sounds very easy,
12:00:04right? So let's understand how is SQL
12:00:05going to execute those functions. Okay.
12:00:07So now let's have a quick overview of
12:00:08the syntax for both of the functions
12:00:10lead and lag. We have here very simple
12:00:12example for the lead function. So as
12:00:15usual we start with the function name.
12:00:16It's going to be the lead. And now after
12:00:18that we're going to go and pass the
12:00:20arguments. And as you can see we have
12:00:21here multiple stuff. So let's do it step
12:00:24by step. So the first thing is that
12:00:25we're going to go and specify an
12:00:26expression. And the data type could be
12:00:28any data type. It could be a number like
12:00:31here the sales. It could be a character
12:00:32like names or dates or anything. So this
12:00:36is required. We have to specify an
12:00:39expression. We cannot leave it empty.
12:00:41And we can use any data type. Now moving
12:00:43on to the next one. We have here a
12:00:44number. So what is this? This is the
12:00:47offset and this offset is optional. So
12:00:49you can go and skip it. So what offsets
12:00:51means? What we are doing over here? We
12:00:53are specifying for SQL the number of
12:00:56rows forward or backward from the
12:00:59current row. So here in this example we
12:01:00are specifying the offset as two using
12:01:02the lead. And with that we are telling
12:01:04SQL go jump to the next two rows and get
12:01:06me the value. And if you are using lag
12:01:08it means you are telling SQL go back two
12:01:11rows up and get me the value. So here
12:01:13you are telling SQL how many rows it
12:01:15needs to jump and if you don't specify
12:01:17anything like leave it empty SQL going
12:01:19to go and use a one. So the default of
12:01:22this with the offsets going to be one if
12:01:23you don't specify anything. All right
12:01:25moving on to the last one and to the
12:01:26third one. This is as well optional. You
12:01:28can go and leave it empty. So here it is
12:01:31the default value. Now what happens with
12:01:33those functions is that sometimes SQL
12:01:35jump to the next two rows or something
12:01:37like that and SQL doesn't find anything.
12:01:39So there is no more rows available to
12:01:41access and with that SQL going to go and
12:01:44return a null. So that means if SQL goes
12:01:46to the next rows or go to the previous
12:01:48rows and doesn't find anything SQL as a
12:01:51default going to go and return a null.
12:01:53So if you don't specify anything over
12:01:55here in those scenarios you will have a
12:01:57null values as a return from the whole
12:02:00function. But in some scenarios you
12:02:01don't want to have a null you would like
12:02:03to have a value. So here you are
12:02:05defining the default value. So it should
12:02:08not be a null, it should be a 10. So
12:02:10scale if you don't find anything return
12:02:12a 10. Don't return a null. So again
12:02:15guys, the default values, the offsets,
12:02:17all those informations are optional for
12:02:18you in order to configure it. But you
12:02:20should know the default if you don't use
12:02:22anything for the offset is going to be
12:02:24one for the default value going to be
12:02:26null. But you must specify an
12:02:28expression. So here you cannot leave it
12:02:30empty. All right. So that's all about
12:02:32the arguments that you can pass to the
12:02:34lead or lag functions. Then the next
12:02:36stuff are the standard stuff. So we have
12:02:38the overclos then we have the partition
12:02:40by as usual partition by is optional.
12:02:43And then to the order by those functions
12:02:45it's like the rank functions. It
12:02:47requires you to sort the data. So it is
12:02:50a must to sort the data otherwise will
12:02:52not know what is the next row what are
12:02:54the previous rows. So we have to sort
12:02:56the data. It is required. You cannot
12:02:58skip this. So it is not optional. All
12:03:00right. So the syntax is not crazy right?
12:03:02We have the usual stuff but only we can
12:03:04go and configure the default value and
12:03:06the offsets. Okay guys, now we have very
12:03:07simple example. We have months and sales
12:03:10and we're going to go and understand how
12:03:11the SQL works for both of the functions
12:03:14lead and lag side by side. So now in the
12:03:16first example we are interested in the
12:03:18sales of the next month. So in order to
12:03:21do that we're going to use the lead
12:03:22function. So lead and then we're going
12:03:25to specify the argument. It is the
12:03:26sales. We want the value of sales and
12:03:28then we define the window like this
12:03:31order by month. So it's going to be
12:03:33ascending. And now in the right side
12:03:34we're going to be interested in the
12:03:36sales of the previous months. So in
12:03:38order to do that we're going to use the
12:03:40lag function. So it's going to be very
12:03:42similar to the lead. We have lag and
12:03:44then the sales since we are interested
12:03:45in the sales and we're going to sort the
12:03:47data by the month. So now let's see how
12:03:49going to do it step by step and side by
12:03:51side. So going to start with the first.
12:03:53So now let's see how skill going to
12:03:55process those informations side by side
12:03:57and row by row. So it's going to start
12:03:59with the first row over here. What is
12:04:02the next month of January? It is
12:04:04February and we are interested in the
12:04:07sales of this row. So SQL going to take
12:04:10the value from the next row and we're
12:04:12going to have the value of 10. So now by
12:04:14looking through the January we can see
12:04:16the sales of the next month of February
12:04:18in the same row. So now let's check the
12:04:20right side over here. Now we are
12:04:22interested in the previous month. So
12:04:24what is the previous month of the first
12:04:26row? It will be nothing. Right? So we
12:04:28cannot point it with anything. That's
12:04:31why going to say this is null. There is
12:04:33no previous month for the current row.
12:04:36And we're going to have it as a null.
12:04:37Okay. So now it's going to go to the
12:04:39next row. We are at February. What is
12:04:41the next month? It's going to be March.
12:04:43And it's going to point to it. So we
12:04:45will get the 30 as the sales of the next
12:04:48month of March. And on the right side,
12:04:51what is the previous month of February?
12:04:54It's going to be January, right? So,
12:04:56it's going to get the value the sales of
12:04:58the previous month. And here we will get
12:05:0120. So, as you can see, it's very
12:05:02simple. On the lead, we are always
12:05:04checking the next values. On the lag, we
12:05:06are always checking the previous value.
12:05:08So, let's keep going. We are currently
12:05:10at March. What is the next month? It's
12:05:12going to be April. So, it's going to go
12:05:14and point to it like this. and we will
12:05:18get the sales of the next month April.
12:05:21For the March on the right side, what is
12:05:22the previous month? It is February.
12:05:25Right? So, it's going to go and point to
12:05:27February. So, we will get the sales of
12:05:3010. And now, interesting to the last row
12:05:32over here. You can see that we are at
12:05:34April. What is the next month of April?
12:05:37There is nothing because we are at the
12:05:39end of our table, right? So, since there
12:05:41is no month after that, we will get a
12:05:44null in the output. But for the lag, we
12:05:47still have a previous month for April.
12:05:48So what is the previous month? It is
12:05:50March. And we will get the sales of the
12:05:54March. So it's going to be 30. So that's
12:05:56it guys. It's really simple, right? It's
12:05:58just like they are doing the opposite
12:05:59things. So now if you check those values
12:06:01side by side, you can see that with the
12:06:03lead, we will always get a value for the
12:06:06first row, but for the last row, it can
12:06:08be always empty because there is no next
12:06:11value. We are at the end of the table.
12:06:13But if you check the lag for the first
12:06:15value, we will always get a null because
12:06:18there is no previous value or previous
12:06:20record from the first row. And for the
12:06:22last record, as you can see, we're
12:06:24always going to get a value because we
12:06:26will have a previous value. Okay, let's
12:06:28move on in order to understand how scale
12:06:29this time works with the offsets and the
12:06:31default value. So now we have the same
12:06:33data, but we have different task. So now
12:06:35on the left side, we would like to get
12:06:36the sales of two months ahead. So it's
12:06:39not the next month, it's going to be two
12:06:41months. And we would like to tell SQL if
12:06:43you don't find any value don't return
12:06:45null return for us is zero. So this is
12:06:47going to be our default. Now if you
12:06:49check the syntax it's going to be exact
12:06:51like before but we are adding now an
12:06:53offset of two because we are interested
12:06:55in two months ahead and we are
12:06:58specifying here a default value zero. So
12:07:00if you don't find anything put zero
12:07:02don't put null. Now on the right side we
12:07:04have the exact opposite. We are
12:07:06interested in the sales of two months
12:07:08ago. So we are not interested in the
12:07:10direct previous month we need the sales
12:07:12of two months ago. And here the same
12:07:14thing if you don't find anything don't
12:07:16return null give us a zero. So as you
12:07:18can see we have the same syntax but
12:07:19using the function lag. So now let's
12:07:21understand how going to execute this
12:07:23step by step and side by side. So going
12:07:25to start with the first month January.
12:07:27So now SQL going to ask what is the
12:07:28sales of two months ahead. So we are at
12:07:32January. It will not be February it's
12:07:34going to be the month of March. So it's
12:07:36going to go and point it like this and
12:07:38we will get the value of 30. So 30 is
12:07:41the sales of two months ahead. And now
12:07:44on the right side we are as well at
12:07:45January. It's going to ask the question
12:07:47what is the sales of two months ago. So
12:07:50we don't have any previous data. Right?
12:07:52So we will not get anything. It's going
12:07:54to return null but it's going to check
12:07:56do we have a default value? Well yes. So
12:07:59this time HQL will not return null. It's
12:08:01going to return the default value. And
12:08:03this time it's going to be zero. All
12:08:04right. All right. So now let's go to the
12:08:06next value. We are currently at
12:08:07February. What is the sales of two
12:08:10months ahead? So it will not be March,
12:08:12it's going to be April. So it's going to
12:08:14go and point it like this and we will
12:08:16get the value of five. So now on the
12:08:18right side we are currently at February.
12:08:20Now the question is what is the sales of
12:08:22two months ago? We have history. We have
12:08:24the previous month but we don't have two
12:08:26months in the history. That's why we
12:08:29will still get zero as the output with
12:08:31the default value. Okay. Okay. So now
12:08:33let's keep going to the next value. We
12:08:34are currently at March. SQL going to ask
12:08:37what is the sales of the two months
12:08:38ahead. We have only one month after that
12:08:41but we don't have two months. That's why
12:08:43SQL will not find anything and it's
12:08:45going to return null but it's going to
12:08:47go and use the default. So here we're
12:08:48going to go and get the value of zero.
12:08:50There is no more data available in the
12:08:52table. But now on the right side we are
12:08:54currently at March and we are asking
12:08:56what is the sales of two months ago. So
12:08:59now we have enough history in the past
12:09:01and it's going to get the value of 20.
12:09:04All right. So now let's go to the last
12:09:05month over here in our table. April.
12:09:07What is the sales of two months ahead?
12:09:10We don't have any data. So it's going to
12:09:12be zero as well. But now on the right
12:09:13side, we are currently at April. What is
12:09:16sales of two months ago? We have enough
12:09:18history. That's why SQL going to get and
12:09:20point it like this. So we will get the
12:09:23February going to be 10. So that's it.
12:09:25This is how SQL works with the lead and
12:09:27lag using offsets and as well default
12:09:30value. Let's go back in SQL in order to
12:09:32practice those two
12:09:36functions. Okay, so now we have the
12:09:38following task and it says analyze the
12:09:40month over month performance by finding
12:09:42the percentage change in sales between
12:09:45the current and the previous month. So
12:09:46that means we have to go and compare the
12:09:48current month with the previous month.
12:09:50So the main use case for the lead unlock
12:09:52is to do comparison analyszis and we
12:09:55have a very common use case it's called
12:09:57time series analyzes. So it is the
12:10:00method of analyzing our business our
12:10:02data in order to understand the patterns
12:10:05and trends over the time. And one of the
12:10:08most important and classical question
12:10:10that you're going to get from the
12:10:11decision makers or business is to do
12:10:14year-over-year analyszis or month over
12:10:16month analyszis. So the year-over-year
12:10:18analysis is going to help us in order to
12:10:20understand the overall growth or decline
12:10:23in the performance of our business over
12:10:26the years over the time. But in the
12:10:27other hand, we have month- over-month
12:10:29analyszis in order to do shortterm
12:10:32trends analyzes and as well discover the
12:10:34patterns in the seasonality. So the main
12:10:36focus is to understand the performance
12:10:38of our business over the time. So now
12:10:40let's go back to it in order to solve
12:10:42the task. Okay guys, so now let's go and
12:10:44do it step by step. Now what is the
12:10:46first step? Before we go and compare
12:10:48things together, we have to collect the
12:10:49data. We have to do the calculations
12:10:51first. So we have to find out first the
12:10:53total sales for the current month and
12:10:55then the total sales for the previous
12:10:57month. And after that we can go and
12:10:59compare them. So now let's start with
12:11:00the easy stuff. We have to find out the
12:11:03current sales for the current month. So
12:11:05in order to do that, let's just do very
12:11:07simple select. So what do we need? We
12:11:09need let's take the order ID. Let's take
12:11:11the order date because inside it we have
12:11:14the month. Uh let's go and collect the
12:11:16sales. So that's it for now from sales
12:11:21orders. So let's go and execute this. So
12:11:24now in the result we got the usual
12:11:25stuff. We have 10 orders, sales and
12:11:28order dates. But the order date is on
12:11:30the level of the days and we are not
12:11:32interested on the whole date. We would
12:11:34like to get only the month in order to
12:11:36calculate the total sales for the month.
12:11:38Now we're going to go and use a function
12:11:39in order to extract the month from a
12:11:41date. Don't worry about it. We're going
12:11:42to have a dedicated chapter in order to
12:11:45show you how to deal with the dates
12:11:46format in SQL. So now what we're going
12:11:48to do, we will use a very simple
12:11:51function called month and order dates.
12:11:54And let's call it order month. So that's
12:11:57it. Let's go and execute it. Now, as you
12:11:59can see, we got a new field where we
12:12:01have only the month of informations. So
12:12:03here we have January, February, and
12:12:04March. So now the next step is that we
12:12:06want to find the total sales for each
12:12:08month. So what we're going to do, we're
12:12:10going to go and use group by. So, let's
12:12:12do that. We're going to go and say we
12:12:15want the sum of sales. I'm just going to
12:12:18call it current month sales. And let's
12:12:22go and get rid of all those
12:12:23informations. We're going to go and
12:12:25group by the month, right? So, group by
12:12:28and let's have the month. So, that's it.
12:12:31Let's go and execute it. So, it's very
12:12:33simple, right? We got now the three
12:12:35months and the total sales of the
12:12:37current month. So now with that we got
12:12:39the first information that we need in
12:12:40order to do the comparison. We have for
12:12:42each row the total sales for the current
12:12:45month. So now the next thing that we're
12:12:46going to do is to find out the total
12:12:49sales for the previous month like side
12:12:51by side in the same row. And in order to
12:12:53do that we have learned we can go and
12:12:55use the lag function. So we're going to
12:12:57go and integrate the lag window function
12:12:59in the same group by. So we're going to
12:13:01do it like this. So lag we are now
12:13:04interested in the previous month. So
12:13:06that's why we're going to go and get the
12:13:08sum of sales as an expression inside it.
12:13:11And after that we're going to define the
12:13:12window. It's going to be like this over
12:13:14and order by is a must. So we're going
12:13:16to go and sort the data by the month.
12:13:19Right? So let's go and do it. And with
12:13:21that we have defined the previous month
12:13:24sales. So you are the previous month
12:13:27sales. So now let's go and execute it in
12:13:29order to see the results. All right. So
12:13:31now let's check the results. The first
12:13:33row what is the previous month? There is
12:13:35no previous month. We are at the first
12:13:37record and the first month that's why we
12:13:39have null. Now let's go to February.
12:13:41What is the sales of the previous month
12:13:44from January? It is 105. So this is
12:13:47correct. And now to the last value to
12:13:49the March. What is the sales of
12:13:51February? The previous month it is 195.
12:13:54So with that we got the two
12:13:56informations. We have the current month
12:13:58and as well the previous month. So guys
12:14:00as you can see it's magic right? It's
12:14:02very simple. we can go and use the lead
12:14:04and lag functions in order to access
12:14:07another values from another rows without
12:14:09doing any complicated joins and so on.
12:14:12Okay. So now what is the next step?
12:14:13We're going to go and subtract the total
12:14:15sales from the current month with the
12:14:17previous month. So in order to do that
12:14:19we're going to go and use a sub query
12:14:21like this. So select star from and we're
12:14:26going to have it like this as subquery.
12:14:28And now the calculation is very simple.
12:14:30Let me just move this a little bit down.
12:14:33So it is the current month subtracted
12:14:36from the previous month and let's go and
12:14:39call it month over month change. So
12:14:43that's it. Let's go and execute this. So
12:14:45now let's go and check the results for
12:14:47the first month. You can see that we
12:14:49don't have any value and that is correct
12:14:50because the previous month is empty. So
12:14:53there is no change. And now moving on to
12:14:55the February. You can see over here we
12:14:57got plus 90. That means we have here
12:15:00improvement in the performance of our
12:15:02sales. Now moving on to the last one.
12:15:04It's really bad. We have decline in our
12:15:06performance. We can see that we have
12:15:08minus 115. So that means the current
12:15:11month is doing really bad compared to
12:15:13the previous month. So the March is
12:15:15really bad month. Okay. So now as you
12:15:16can see in the output we got the
12:15:18absolute numbers but the task says find
12:15:20the percentage change. So we have to
12:15:22convert this to a percentage and we can
12:15:24do it like this. It's very simple. Let's
12:15:26do it in a new column. Just going to
12:15:28zoom out a little bit. So, it's going to
12:15:31be the change the differences divided by
12:15:34the previous month sales. And then let's
12:15:36go and multiply it with 100 in order to
12:15:39get the percentage. So, like this. And
12:15:41now, as you can see, we got zeros. And
12:15:43that's because those numbers are
12:15:45integer. So, we have to go and cast one
12:15:47of those values. Just going to do it for
12:15:49the first. So, cast and float. So,
12:15:53that's it. Let's go and execute it
12:15:55again. Now the result looks better. We
12:15:57have the percentages but we have a lot
12:15:59of decimals. So let's go and round the
12:16:02number to let's say one decimal. So only
12:16:05one and let's give it a name. So you are
12:16:07month over month percentage. So let's
12:16:11execute. So now as you can see things
12:16:12get better. And with that we have
12:16:14calculated the percentage change in
12:16:16sales between the current and the
12:16:19previous months. And this is how we do
12:16:21month overmonth analyszis.
12:16:26All right. So now we have another use
12:16:28case for the lead and lag function. We
12:16:30can use them in order to do customer
12:16:31retention analyzes. It's all about
12:16:33measuring the customer behavior and
12:16:35loyalty. So we are helping the business
12:16:37and decision makers to build strong
12:16:40relationship with the loyal customers
12:16:42and for them as well to focus on their
12:16:44needs. So now let's see how we can use
12:16:46lead and lag function in order to do
12:16:48customer retention analyszis. So let's
12:16:50go. All right. Right. So now we have the
12:16:51following task and it says in order to
12:16:53analyze customer loyalty rank customers
12:16:56based on the average days between their
12:16:58orders. So there is a lot of things
12:17:00going on over here. Let's do it step by
12:17:02step. And I would like always to start
12:17:03with a very simple select. So let's go
12:17:06select informations like the order ID.
12:17:09Let's get the customer ID and as well
12:17:12since we want the days we would like to
12:17:14have the date. So order dates from the
12:17:17table sales orders and let's go and sort
12:17:21the data. So order by customer ID and
12:17:25order dates. So that's it. Let's go and
12:17:28execute. So now as usual we got our 10
12:17:30orders, the customers and when they did
12:17:32order. So now let's check the task.
12:17:35Let's solve this over here. Days between
12:17:37their orders. So we have to find how
12:17:39many days are between two orders. For
12:17:42example, if we check the customer number
12:17:44one over here, he did order around 10
12:17:47January and the second order is like
12:17:50after 10 days 20 January. So we have to
12:17:53go and subtract those two dates. Now in
12:17:56order to subtract those informations and
12:17:58do calculations, we have to have
12:17:59everything in the same row. So for
12:18:01example, if we are at the first row over
12:18:03here, I would like to have as well one
12:18:05column about the next order. So the date
12:18:08of the next order. So we have to access
12:18:11a value from another row. Of course, we
12:18:13can go and do joins, but we have lead
12:18:16and lag functions. And for this
12:18:17scenario, we're going to go and use the
12:18:19lead window function. So let's go and do
12:18:21that. I'm going to go and call the order
12:18:23date over here as a current order. And
12:18:26let's go and calculate the lead. So we I
12:18:30would like to get the next order date.
12:18:32So I would like to get this value over
12:18:34here in the same row. That's why we this
12:18:37time we're going to get the order dates.
12:18:39And now let's go and define the window.
12:18:42Now we have to go and partition the data
12:18:44because we are analyzing each customers
12:18:46separately, right? So that's why we have
12:18:48to partition that by the customer ID.
12:18:51And of course in order to do the lead,
12:18:53we have to use the order by. So let's go
12:18:55and define that as well. Order by and
12:18:58it's going to be by the order date. So
12:19:00now we have to give it a name. The order
12:19:02date here is the current order. This
12:19:04going to be the next order. So next
12:19:07order. Let me zoom out a little bit and
12:19:10make this smaller. So let's go and
12:19:12execute it. So now as you can see in the
12:19:14output we got a new column called next
12:19:16order. And with that we got the current
12:19:19order, the current row and as well the
12:19:22value from the next row. So what is the
12:19:24next row? It's going to be the 20
12:19:25January. The same thing of course for
12:19:27the next row. Over here we have the
12:19:29current order date and the next order
12:19:32date. So this value going to be exactly
12:19:34as the next one over here 15 of February
12:19:37and then since we are working with
12:19:39window since this is the whole window
12:19:40over here the last order for this
12:19:43customer it's 15 of the February there
12:19:46is no next order so this going to be
12:19:48null the same thing if you check the
12:19:50other customers you're going to see
12:19:52always the last order don't have any
12:19:54next order so looks like everything is
12:19:57fine and for the last customer he has
12:19:59only one order so now with this we got
12:20:01all the informations for our
12:20:03calculations. So we have the current
12:20:05order and the next order in the same
12:20:07row. Now we can go and subtract them in
12:20:10order to get the days between those two
12:20:12orders. And now in order to subtract
12:20:14date we has to use the function date
12:20:16div. Don't worry about those functions.
12:20:18We're going to explain all those stuff
12:20:19in the next chapters. So now just follow
12:20:21me with those steps. What we're going to
12:20:23do, we're going to go and subtract this
12:20:25date the order date with the whole thing
12:20:27over here. Right? So the whole thing
12:20:29here is the next order. So let's do it
12:20:31in a new line and it's going to be very
12:20:34simple. So date diff we are finding the
12:20:37differences between two dates. So the
12:20:39syntax going to be like this. First we
12:20:41have to define what we are talking
12:20:42about. Are they days, months, years and
12:20:44so on. So we have to tell SQL find me
12:20:46the differences in days. Now we have to
12:20:49specify two days. So the first one going
12:20:51to be the order date. This is the
12:20:53current date and the second date going
12:20:55to be the whole thing from here. So
12:20:58let's take it and put it side by side
12:21:00and this calculation going to give us
12:21:02number of days. So we're going to call
12:21:04this days until next order. All right.
12:21:08So now let's go and execute the whole
12:21:10thing. So now let's check the result. As
12:21:11you can see over here we got 10. So this
12:21:13is 10 days between those two dates and
12:21:16the next one we have around 26 days.
12:21:19Here we have a null because we don't
12:21:20have here a date and for the next one we
12:21:23have 31 days. So we have a whole month
12:21:25over here. So everything is working
12:21:27perfectly and with that we have solved
12:21:30only this part days between their
12:21:32orders. So guys you see right this is
12:21:34the magic of the lead and lag function.
12:21:36We can very easily access any
12:21:38information you need in the same row in
12:21:40order to do such a important analyzis
12:21:42and with very simple query. We are not
12:21:44doing any crazy stuff like joining and
12:21:46stuff. We are just specifying the lead
12:21:48function. So now we got all the
12:21:50informations that we need. Next we're
12:21:52going to go and calculate the average of
12:21:53those days. So in order to do that we
12:21:56have to go and use a subquery. So let me
12:21:58just zoom out. So let's go and select
12:22:01star just prepare the subquery. So the
12:22:05whole thing going to be a subquery. I'm
12:22:06going just get rid of the order by it's
12:22:08not now necessary. So let's me just put
12:22:11it like this and shift it. So now what
12:22:14do we need? We need the average of the
12:22:16days. So we need the average of this
12:22:17value. So what can we do? We're going to
12:22:19go and use a group by. So customer ID
12:22:22since we have to find the average for
12:22:24each customers and we're going to get
12:22:26this value and say average days until
12:22:29the next order and we're going to call
12:22:30it average days. So and we have here to
12:22:34group by. So group by customer ID. So
12:22:38like this just make this a little bit
12:22:41smaller and zoom in here. So that's it.
12:22:43Now we are just doing a very simple
12:22:45average and group I statements. So let's
12:22:47go and execute it. Now as you can see
12:22:49it's going to go and aggregate the data.
12:22:51So we have now only four customers and
12:22:53for each customer we have the average
12:22:56days between their orders. So now what
12:22:58is missing in our task? If you check
12:23:00over here it says rank the customers
12:23:02based on this average. So we have to go
12:23:05and use the rank function. So here again
12:23:07another window function that we have to
12:23:09go and use. We're going to do it
12:23:10together with the group by. So let me
12:23:12just make this a little bit smaller and
12:23:15then let's do it over here. So I'm just
12:23:17going to go with the rank function. Then
12:23:19we're going to define the window like
12:23:21this over order by and then we're going
12:23:23to go and sort the data by the average
12:23:25days. So that means we're going to go
12:23:27and get this calculation over here and
12:23:29put it as order by it's going to be
12:23:31ascending. So we are focusing on the
12:23:33lowest average days. So that's it. Let's
12:23:36call it rank average. So now let's go
12:23:39and execute this. So now by checking the
12:23:41result, you can see now we have a
12:23:43ranking for the average. And here skill
12:23:46says that the number one customer or the
12:23:49number one loyal customer is the
12:23:51customer number four which is not really
12:23:52correct because the number four we don't
12:23:54have a lot of informations about this
12:23:56customer he or she did order only once.
12:23:59So either now you go and like filter the
12:24:01data and remove this customer where you
12:24:03say if the average is null then don't
12:24:05put it in the rank or we can go and
12:24:07replace this value with a very huge
12:24:09value in order to make it at the end of
12:24:11our list. For example, we can go over
12:24:13here and replace the null with
12:24:16qualisk like this. And we say if the
12:24:19average is null, then let's say give me
12:24:22a crazy number like this very huge one.
12:24:25So that's it. Let's go and execute. And
12:24:27now as you can see this customer going
12:24:29to be at the end of our list. And now we
12:24:31can see that the most loyal customer is
12:24:33number one. And then the other two
12:24:35customers are in the rank two. Here we
12:24:38are sharing the same rank since we have
12:24:39the same average. So guys with that we
12:24:41have solved the task and we have ranked
12:24:42the customers based on the average days
12:24:45between their orders. So we have now a
12:24:47really nice rank and we can understand
12:24:49now the behavior of the customers and
12:24:51maybe we have to go and focus on the
12:24:53customer number one and understand her
12:24:55or her needs. And of course the function
12:24:56that helped us here in order to do such
12:24:58a customer retention analyszis is the
12:25:01lead function in order to find the next
12:25:03order to calculate the days. So this is
12:25:05how you use lead functions to do such a
12:25:07use case.
12:25:12the first value and the last value
12:25:14functions. I think the name says
12:25:16everything, right? So the first value
12:25:17going to allow you to access a value
12:25:19from the first row within a window where
12:25:22the last value exactly the opposite. It
12:25:24going to allow you to access a value
12:25:26from the last row within a window. Easy,
12:25:29right? So now let's understand how SQL
12:25:31execute those functions. Okay. So now as
12:25:33usual, we have this very simple example.
12:25:34we have the months and sales and we have
12:25:36it twice because we would like now to go
12:25:38and compare side by side the two
12:25:40functions first value and last value. So
12:25:42now for the left sides we would like to
12:25:44get the sales of the first month and on
12:25:47the right sides we would like to get the
12:25:49sales of the last month. So now for the
12:25:51first task we can go and use the first
12:25:53value. It's very simple. So the first
12:25:55value function then the argument going
12:25:56to be sales since we want the sales and
12:25:59then the window going to be defined like
12:26:00this order by month because we want to
12:26:03get the first month. So as usual we must
12:26:06use order by now on the right side in
12:26:08order to get the sales of the last
12:26:09months we can go and use the last value
12:26:11right so the same things last value
12:26:13sales over order by month. So as you can
12:26:16see on the left and right we don't use
12:26:18any frame definition but the default
12:26:20going to be used from this. All right.
12:26:21So now let's see how SQL going to
12:26:22process both of those queries side by
12:26:25side. So the first step is SQL going to
12:26:26go and sort the data. They are already
12:26:28sorted from the lowest to the highest.
12:26:30And then the next step is going to start
12:26:32row by row finding the first value on
12:26:34the left side. So what is the unbounded
12:26:36proceeding? It's going to be static and
12:26:39always pointing to January. So this is
12:26:42always going to be the unpounded
12:26:43proceeding. We have it in both sides
12:26:45like this. And what is the current row?
12:26:48It's going to be at the start the first
12:26:50row. And on the right side the same
12:26:52things over here. So the window
12:26:54definition going to be is only one row
12:26:57right. So what is the first value in
12:27:00this window? It is 20. Right? The same
12:27:03things on the right side. What is the
12:27:05last value in this window? It is as well
12:27:0820. So we will get exactly same results.
12:27:10Now let's move to the second row. So
12:27:12it's going to be pointing to February.
12:27:14And the frame definition going to be
12:27:16here extended like this. So what is the
12:27:19first value in this frame? It's going to
12:27:21be as well 20. Right? So in the output
12:27:24we're going to get 20. And now in the
12:27:25right side the current row going to be
12:27:27as well pointing to February and the
12:27:30window going to go get extended. So now
12:27:32what is the last value of this frame?
12:27:35It's going to be 10. Now let's keep
12:27:37going. We're going to go to the March
12:27:39and the window going to get extended.
12:27:41What is the first value? It's always
12:27:44going to be the same. So 20 on the right
12:27:46side window going to get extended. What
12:27:49is the last value? It's going to be 30.
12:27:52So as you can see the default definition
12:27:53is always having the static start always
12:27:56the same start of the subset and as we
12:27:58are moving with the current row the
12:28:00frame going to get extended. So now
12:28:02moving to the last one and with that we
12:28:04will get the whole data set inside the
12:28:07frame and the first cell is going to be
12:28:0920 on the right side. the same things
12:28:12going to get extended like this and this
12:28:14time the last one going to be April and
12:28:16five. So now if you go and compare them
12:28:18side by side you see that on the left
12:28:20side the task is solved and everything
12:28:22is working correctly right. So we have
12:28:24for each row always the sales of the
12:28:26first row and what is the first row it
12:28:29is January. So we have everywhere a 20
12:28:31which is correct. But now if you check
12:28:33the right side you can see there is
12:28:35something wrong right? We are getting
12:28:37not the last value. We should always get
12:28:39April right? We should have here
12:28:40everywhere a five. So we have here
12:28:43exactly the same result as the sales. So
12:28:45it's really useless to use it like this,
12:28:47right? And that's of course because SQL
12:28:49is using the default definition of the
12:28:51window frame. Last value is the only
12:28:53function from all window functions that
12:28:55you cannot use the default frame
12:28:58definition. You have to go and customize
12:29:00the frame definition in order to get the
12:29:02effect of the last value. For the first
12:29:04value, everything is working. If you're
12:29:05using a default frame, if you are not
12:29:07specifying anything, but for the last
12:29:09value, you will not get the effect
12:29:10correctly without customizing the frame
12:29:13window. So my friends, you can go and
12:29:15use the first value function like all
12:29:17other window functions without defining
12:29:19a frame. You can go with the default and
12:29:21you will get the effect of the first
12:29:22value, but the last value you have to go
12:29:24and define a frame. So let's see how we
12:29:26can solve that. All right. So now in
12:29:28order to solve this, we going to define
12:29:29the frame like this. It's going to be
12:29:31the rows between the current row and the
12:29:34unbounded following. So we just switch
12:29:36things around. So now let's see how this
12:29:38going to work. Now of course it's going
12:29:40to go and sort the data and so on. Now
12:29:42it's still going to have a pointer to
12:29:44the unbounded following. So it's going
12:29:46to point always to the last row in our
12:29:48data set and then it's going to proceed
12:29:50step by step. So the first row going to
12:29:52be like this and the frame going to be
12:29:54the whole thing, right? So from the
12:29:56current row until the unbounded
12:29:58following. So what is the last value the
12:30:01last row? It's going to be the five,
12:30:03right? The April. So we will get in the
12:30:04output five. Now let's proceed to the
12:30:07next value. The frame going to be
12:30:09shorter and smaller. And what is the
12:30:11last value? It's going to be as well the
12:30:13five. Right? So now we jump to the next
12:30:15one. And the frame going to be like
12:30:17this. What is the last value? As well
12:30:19five. And then we will get the last
12:30:21value like this. Current row is equal to
12:30:24the unbounded following. We have only
12:30:25one row and it's going to be as well
12:30:27five. So as you can see it's very simple
12:30:29just fix the frame clause and you will
12:30:31get the last value working as expected.
12:30:34So this is how SQL going to go and do
12:30:36it. Now let's go back to SQL and start
12:30:38practicing. All right. So now we have
12:30:39the following task. It says find the
12:30:42lowest and highest sales for each
12:30:44product. So now let's see how we can do
12:30:46this. As usual we're going to start with
12:30:47very simple select statement. So select
12:30:50order ID. We need the product ID and as
12:30:54well their sales. So let's select the
12:30:56table sales orders. So that's it. Let's
12:31:00go and select this. Now in the output we
12:31:02got our orders, products and sales. So
12:31:05now let's start with the first part of
12:31:06the task. Find the lowest sales for each
12:31:09product. So in order to do that, we can
12:31:11use the first value function. So let's
12:31:14go and do that. First value. Then what
12:31:17we are talking about, we have to give an
12:31:18expression. We need the lowest and
12:31:21highest sales. So let's go and have the
12:31:23sales inside it. And now we have to
12:31:25define the window. So over since we are
12:31:28saying for each product that means we
12:31:29have to go and make windows. So we have
12:31:32to divide the data using partition by
12:31:34products ID. And then we must use an
12:31:37order by right. So we have to go and
12:31:39sort the data by the sales. Since the
12:31:42first value should be the lowest value,
12:31:45we have to do it as ascending from the
12:31:47lowest sales to the highest sales. So
12:31:49we're just going to leave it like this
12:31:50as a default and we're going to call it
12:31:53lowest sales. So let's go and execute
12:31:57this. So now let's go and check our
12:31:58results. First going to go and partition
12:32:00the data by the product ID. So as you
12:32:02can see we got now here four windows.
12:32:05Then sort the data by the sales. So the
12:32:07data are sorted from the lowest to the
12:32:09highest from 10 to 90. So now what is
12:32:12the first value of the sales? It is the
12:32:14first row, right? So it's going to be
12:32:1610. That's why we have everywhere a 10.
12:32:19Let's check another one. Let's take this
12:32:21one here. So this window has two rows
12:32:23and it is sorted the lowest sales or
12:32:26let's say the first value is 25. So with
12:32:29that we have solved the first part of
12:32:31the task finding the lowest sales for
12:32:33each product. Let's go to the next one.
12:32:35We have to find out the highest sales
12:32:36for each product. So let's go and use
12:32:39the last value for this. So let's have a
12:32:41new line. We're going to have a last
12:32:44value again the
12:32:47sales. Then we're going to go and define
12:32:49the window. So it's going to be the
12:32:50exact same window. We have to partition
12:32:52the data by the product ID and order the
12:32:54data by sales. So let's go and just copy
12:32:58the previous one and let's call it for
12:33:00now highest sales. So let's go and
12:33:04execute it. So now if you check the
12:33:06results, you will see our issue over
12:33:07here again. Right? We are not getting
12:33:09the highest sales for this window. The
12:33:11highest sales is 90. But as you can see,
12:33:13we are getting the exact same sales. And
12:33:15we have explained that in the previous
12:33:16example. So in order to fix this, we're
12:33:19going to go and add for it the frame. So
12:33:22rows between current row and the
12:33:26unbounded following.
12:33:28So now let's go and execute this. So now
12:33:32let's check the result. As you can see
12:33:34over here, we got the highest sales
12:33:36correctly. So for this window, the
12:33:37highest ones is 90. and as well for this
12:33:40window the 60 and so on. So with that
12:33:42you have solved both of the tasks the
12:33:44lowest and the highest sales. But now I
12:33:46would like to show you my honest opinion
12:33:48about this tasks. I will not go and use
12:33:51the last value to find the highest
12:33:53sales. So let me show you how I usually
12:33:55do it. I'm going to go and use the first
12:33:57value in order to find the last value.
12:34:00So now let me show you what I mean.
12:34:01Let's go and add a new row. I will just
12:34:04take the whole thing from the lowest
12:34:06sales. But what I'm going to do, I'm
12:34:09just going to go and change the order.
12:34:10So that means we will not go and sort
12:34:12the data like this ascending from the
12:34:15lowest sales to the highest sales. We're
12:34:16going to go and switch it. So we're
12:34:19going to go and sort the data from the
12:34:20highest sales to the lowest sales. And
12:34:23with that, the first value going to be
12:34:25the highest sales. So let me just rename
12:34:28it highest sales. Let's give it like
12:34:31two. So let's go and execute this. And
12:34:34now you can see over here we got the
12:34:36exact same results because we sorted the
12:34:38data differently and we get the first
12:34:40value. So this is going to give you the
12:34:42exact same effect like the last value.
12:34:45And as you can see I don't have to
12:34:46define now any window or something like
12:34:48that. I can stick with the default frame
12:34:51but just twisting the order by. So this
12:34:54is how you can do it as well using only
12:34:56the first value. So now just for the
12:34:58sake of this task there's as well
12:35:00another possibility in how to solve
12:35:01this. You can go and use the minmax
12:35:04functions. So let me just take the same
12:35:07and have a new one the lowest sales. We
12:35:10can go and say you know what let's get
12:35:13the min. So we are saying find me the
12:35:15minimum sales and we don't have to go
12:35:18and sort anything. So we can go and just
12:35:20divide it like this. So let's give it
12:35:22another ID. Let's go and execute it. So
12:35:25as you can see we got the exact same
12:35:27results like the other two highest
12:35:29sales. So as you can see we can solve
12:35:31this task using three different
12:35:32functions. Either go and use the last
12:35:35value but you have to define the frame
12:35:37or you can go and use the first value
12:35:39where you switch or flip the order by or
12:35:42simply just using the max function in
12:35:44order to get the highest sales. So guys
12:35:47as you can see we can use the first
12:35:48value and the last value in order to
12:35:50find out the extremes like here in this
12:35:52example the lowest and the highest
12:35:54sales. So there is like similarity
12:35:56between those two functions and as well
12:35:57the min and max. And of course what
12:35:59we're going to do with this value over
12:36:01here we can go and compare it with the
12:36:03current sales. So for example we can go
12:36:05and extend our task where we say find
12:36:07the difference in sales between the
12:36:09current and the lowest sales. So in
12:36:12order to do that let me just clean up
12:36:13all those stuff and let's stick with the
12:36:16first value and the highest value like
12:36:18this. So we have to compare now the
12:36:20current sales which is this field over
12:36:23here. the sales the original one with
12:36:25the lowest sales with the whole thing
12:36:27from here. So let's go and do that. So
12:36:29we're going to have a new line and we're
12:36:30going to say just simply subtract the
12:36:33sales from the lowest sales like this.
12:36:36And let's give it a name sales
12:36:39difference. So that's it. Let's go and
12:36:42execute it. Now as you can see the
12:36:44result in one row I'm comparing the
12:36:46current sales which is 90 with the
12:36:48lowest sales from this product. It's
12:36:50going to be the 10. So with that we're
12:36:52going to get the distance let's say
12:36:54between those two informations and it
12:36:56going to be 80. So now for the next one
12:36:58the distance between this value and the
12:37:00lowest value is shorter. So we are near
12:37:03the lowest value. So as you can see over
12:37:05here we can now compare the sales
12:37:08between the current sales and one
12:37:10extreme in order to find the distances
12:37:12between two values. So this is again
12:37:15very important analysis in order to do
12:37:17comparison analyszis.
12:37:22All right friends, so now let's do a
12:37:24quick recap about the value functions or
12:37:25we call them sometimes analytical
12:37:27functions. So what they do, they're
12:37:29going to go and allow you to access a
12:37:30specific value from another row. This
12:37:33going to help you in order to do complex
12:37:34calculations with very simple SQL
12:37:37without having you joining tables
12:37:39together or doing self joins. And for
12:37:41the value functions we have four types
12:37:43or let's say for functions the first one
12:37:45allows you to access the previous value
12:37:47like the previous month using the lag
12:37:50function. The next one it allows you to
12:37:52access the next values the next month
12:37:55using the lead function. Then we have
12:37:57another one it allows you to access the
12:38:00first value in a subset using the first
12:38:02value function. And another option we
12:38:04can go and access the last value in a
12:38:07subset using the last value function.
12:38:09Moving on to the next one, we have the
12:38:11rules of the syntax. So about the first
12:38:13point, it is the expressions. We can go
12:38:15and use any data type. It could be a
12:38:17number, string, a date, anything. Now in
12:38:19order to perform those functions, we
12:38:21have to go and sort the data by the
12:38:22order by. So order by is required. It is
12:38:25a must. Then for the frame, you are
12:38:28allowed to use it. So it is an optional
12:38:30thing. I would say always leave it empty
12:38:32for the frame. But only for the last
12:38:33value, you have to go and customize
12:38:35otherwise it will not work. Now to the
12:38:37next point, we have the use cases. We
12:38:39have simply very important use cases for
12:38:41the value functions in data analytics.
12:38:43So what we can do? We can do time series
12:38:45analyszis. As we learned, we can do
12:38:47month overmonth analyzes and
12:38:49yearover-year analyzes. Those analyszis
12:38:52are classical and it's always the first
12:38:54question in that analyszis in order to
12:38:56measure are we growing with the business
12:38:58or are we declining? How the performance
12:39:00between the current year and the
12:39:02previous year. So as you can see we are
12:39:03doing always comparison using those
12:39:06window functions. The next use case is
12:39:08as well about the time we can do time
12:39:10gap analyzes as we analyzed the customer
12:39:13behavior the customer retention where we
12:39:15have calculated the average days between
12:39:17two orders and the last use case it's as
12:39:20well about comparison comparison
12:39:21analyzes we can go and use the value
12:39:24functions in order to compare the
12:39:25current value with extreme like
12:39:27comparing the current sales with the
12:39:29highest sales or to the lowest sales. So
12:39:32my friends those analyzers are essential
12:39:34in data analyzers you will be countering
12:39:37them in each company in each business
12:39:39you have to answer those questions and
12:39:41you can do that very easily using the
12:39:43SQL window functions all right my
12:39:45friends so that's all about the window
12:39:47value functions and with that we have
12:39:49covered everything about how to
12:39:50aggregate your data using SQL and those
12:39:53are very important tools on how to do
12:39:55data analytics in SQL especially if you
12:39:58are a data scientist and data analyst.
12:40:00So with that we are done with this
12:40:01chapter and I can tell you with that we
12:40:03have covered the intermediate level. So
12:40:05we have learned how to filter the data,
12:40:07how to combine the data and as well the
12:40:09most important functions in SQL. Now
12:40:11we're going to go to the third and last
12:40:13level we will cover now the advanced
12:40:15level. So the first level going to be
12:40:17about the advanced SQL techniques. So
12:40:19now if you go inside it and in SQL there
12:40:21are like different techniques in order
12:40:23to organize our complex projects. So
12:40:26first I'm going to explain for you what
12:40:27is exactly I'm talking about what is
12:40:29complex queries and why we have it and
12:40:31then we're going to start with the first
12:40:32topic the subqueries. So let's
12:40:38go. Normally in projects we have a
12:40:40database and we have a person that is
12:40:42responsible for the database the
12:40:44database administrator that take cares
12:40:47of the database structure. And now in
12:40:49very simple scenario we're going to have
12:40:50a user that is writing queries in order
12:40:53to retrieve data from the database. So
12:40:55he or she going to write an SQL query
12:40:57and then this query going to be sent to
12:40:59the database where it's going to execute
12:41:00it and then the database going to return
12:41:03the results. So at the end our user
12:41:05going to see the result of the query
12:41:07that he wrote. So this is a very
12:41:09simplified scenario on how we use a
12:41:13database. But my friends in the real
12:41:15world things are totally different.
12:41:17Things in real projects get very
12:41:19complicated like this. So for example,
12:41:21you have a financial analyst that is
12:41:24writing a huge block of SQL query that
12:41:27is very complex and there will be like
12:41:30another user that have different role
12:41:32like a risk manager that is as well
12:41:34writing a very complex query and from
12:41:38different departments from different
12:41:39projects for different tasks. You will
12:41:42have a lot of analysts that are writing
12:41:45many complex queries. So all those
12:41:47analysts and managers have a direct
12:41:50access to your database and they are
12:41:52executing a complex analytical queries
12:41:54in order to generate maybe a report or
12:41:57something. Now not only those guys are
12:41:59doing analyszis on your database you
12:42:01will have as well our friend the data
12:42:03engineer that is saying you know what
12:42:05I'm building a data warehouse and I
12:42:07would like to extract your data. So that
12:42:09data engineer going to go and write an
12:42:11extract query in order to extract the
12:42:13data from the database. And then he has
12:42:16a different script for the
12:42:17transformations in order to manipulate,
12:42:20filter, clean up, aggregate your data.
12:42:22And then a third script in order to
12:42:25collect the result of the
12:42:26transformations and load it in another
12:42:29database called data warehouse. A data
12:42:32warehouse is like special database that
12:42:34collect data from different sources and
12:42:36integrate it in one place. in order to
12:42:38do analytics and reporting. And now at
12:42:40the end of this chain, you will have a
12:42:42data analyst and she writes as well
12:42:44queries in order to analyze the data in
12:42:47the data warehouse. Or you might have a
12:42:49different query in order to prepare the
12:42:51data before inserting it to a tool like
12:42:53PowerBI in order to generate
12:42:56visualizations and reports. So we call
12:42:58this a data warehouse system or a
12:43:00business intelligence system that
12:43:02extract and extract from your data and
12:43:05manipulate it and transform it for
12:43:07analyzes. Now not only we have a data
12:43:09engineer and data analyst accessing your
12:43:11database and doing queries, we have as
12:43:13well our friend the data scientist. So
12:43:15now our data scientist as well has a
12:43:17direct access to your database. So he
12:43:20might write like different queries in
12:43:22order to extract the data and as well to
12:43:24manipulate the data that are needed in
12:43:26order to develop a model and doing
12:43:28machine learning and AI. And now one
12:43:30more scenario that I see in many
12:43:32projects where the result of the data
12:43:34analyst going to be used in another
12:43:36query in order to prepare the results
12:43:38for data visualizations PowerBI or in
12:43:41order to export like a Excel list. So as
12:43:43you can see we have a lot of people with
12:43:45different roles that want to access your
12:43:48database and do analyzes on top of it
12:43:50and that's because everyone want to
12:43:52answer questions based on the data and
12:43:54now if I look to this I still think this
12:43:56is a simplified version and how things
12:43:59works in the data projects and I can
12:44:01tell you in real projects things are way
12:44:03more complicated than this so now if you
12:44:05sit back and look to this we will find
12:44:08many challenges and problems for example
12:44:10all those people are not talking to each
12:44:12others And each one of them are creating
12:44:14like their own query. But if you go and
12:44:17take all those queries and compare them
12:44:19side by side, you will find in the
12:44:21scripts and queries logic that is keep
12:44:23repeating. So the queries from the
12:44:25analyst or the data scientists and data
12:44:28engineers, they might contain a
12:44:30redundant logic. And of course the issue
12:44:33of this we have the same effort
12:44:35repeating over and over and maybe not
12:44:38everyone is getting the logic
12:44:39implemented correctly because not all of
12:44:41them having the right skills in SQL. So
12:44:44this is a big issue in this setup. And
12:44:46now we have another challenge having
12:44:48this scenario. If you don't optimize it
12:44:51you will have a performance issue
12:44:53everywhere. So the data warehouse or the
12:44:55data engineer scripts might take like 5
12:44:57hours and the query from the analyst
12:45:00might take like 40 minutes and before
12:45:02inserting the data to reports we might
12:45:04have 30 minutes and 1 hour there 30
12:45:06minutes there and everyone else is as
12:45:08well suffering from bad performance on
12:45:10their queries and the performance
12:45:12everywhere is really bad. So if everyone
12:45:14is writing big complex queries don't
12:45:16expect that they will have a good
12:45:18performance. Now to the third challenge
12:45:20that I observed in many projects and
12:45:22that is the complexity. Now behind the
12:45:24original database you might have a data
12:45:26model that is prepared and optimized
12:45:29only for one application. So you will
12:45:31have in the data model a lot of tables
12:45:32and all those tables have different
12:45:34relationship between them and of course
12:45:36only the developers and the experts of
12:45:39this database understand the physical
12:45:41data model behind this database. And now
12:45:43if you give access to all those analysts
12:45:46they will have a lot of questions
12:45:48because first they have to understand
12:45:50the data model before writing any query.
12:45:53So that means a lot of data workers are
12:45:55keep asking our expert from this
12:45:58database questions. So for example how
12:46:00to connect the table A with the table B
12:46:02and where do I find my columns? What
12:46:04this table means? I'm getting bad result
12:46:06in my query because your data is really
12:46:08corrupt. So the developers of the
12:46:10database will get a lot of questions
12:46:12from the analyst and they have to
12:46:14explain over and over their data model
12:46:16so that the users are able to write
12:46:19those complex queries. So that means all
12:46:21those users are stressing the database
12:46:23team by many questions and as well the
12:46:26users are writing very complex queries.
12:46:28So the complexity is a really big
12:46:30challenge. Now as well by looking to
12:46:32this picture you will find a lot of
12:46:34errors from those queries to the
12:46:35database and this might cause a lot of
12:46:39database stress. So keep executing
12:46:41repeatedly a big complex queries going
12:46:44to makes really big stress for the
12:46:46database and it going to bring the
12:46:48database down. And the last challenge of
12:46:50this picture is that the data security.
12:46:52So if you leave it like this by giving
12:46:54the users a direct access to your
12:46:56database tables you might have a problem
12:46:59because it might be okay for like some
12:47:01data engineers and so on but you don't
12:47:03want to give for each data analyst a
12:47:05full access to the database tables. So
12:47:08you have to protect your tables the
12:47:10columns the rows everything. So you
12:47:12cannot leave it like this where everyone
12:47:14having a direct access to the physical
12:47:16database tables. Now enough talking
12:47:18about challenges problems and issues.
12:47:20Let's be solutionoriented. So what are
12:47:23the solutions of those issues? Of
12:47:25course, there are many solutions, but
12:47:26we're going to focus now on five
12:47:28techniques. We can go and use sub
12:47:30queries or CTE, common table
12:47:32expressions. We can introduce views to
12:47:34our database or temporary tables or we
12:47:37can go and use the technique of the CTAs
12:47:40carrier table as select. So this is
12:47:43exactly why we have to understand those
12:47:45five techniques in order to solve all
12:47:47those issues that we might face in our
12:47:49data
12:47:53projects. All right friends, so now
12:47:55after we understood the importance of
12:47:57those five techniques, let's take a
12:48:00quick and simplified look to the
12:48:01database architecture because I want you
12:48:03to understand what happens behind the
12:48:06scenes and how the database execute the
12:48:09queries from these five techniques. So
12:48:12by understanding this architecture you
12:48:14will understand how things works. So
12:48:16let's go. For each story there are two
12:48:18sides. We have the server side and the
12:48:20client side. In the client side it's
12:48:22like for example you you are writing an
12:48:24SQL query for a specific purpose. Now in
12:48:27the server side we have many things. So
12:48:30the server is where the database lives
12:48:32and it has many components like the
12:48:34database engine. The database engine is
12:48:37the brain of the database that handles
12:48:40different operations like storing,
12:48:42retrieving and managing data in the
12:48:45database. So each time you execute a
12:48:47query, the database engine going to take
12:48:49care of it. And now in the database we
12:48:51have very important component that is
12:48:53the storage and the two main types of
12:48:55storage in a database are disk storage
12:48:58and cache. The disk storage is like a
12:49:01long-term memory where the data is
12:49:03stored permanently. So it's like the
12:49:05disk at your PC. It stores the data
12:49:08permanently even if you turn off the
12:49:10system. And one important feature of the
12:49:12disk is that it can stores a lot of
12:49:14data. But the disadvantage of the disk
12:49:16storage is that it is slow. So it is
12:49:19slow to write and to read. Now in the
12:49:21other hand we have the cache is a
12:49:23short-term memory where the data is
12:49:25stored temporary. It's like the RAMs at
12:49:27your PC. It holds the most frequently
12:49:30used data. So the database can access it
12:49:33quickly in order to retrieve data. And
12:49:35the big advantage of the cache is that
12:49:37it is fast. So it is very fast for the
12:49:39database to retrieve data from the cache
12:49:41compared to the disk. But the
12:49:43disadvantage of the cache, the data is
12:49:45stored there only for short period. So
12:49:47it's like tradeoff between the speed and
12:49:49how much data you can store and how
12:49:51long. Now let's talk about the disk
12:49:53storage. This is very important in
12:49:55databases. There are typically three
12:49:57types of storage areas. There we have
12:49:59the user data, the system catalog and
12:50:02the temporary data and each storage type
12:50:05has a different purpose. So what is user
12:50:07data storage? It is the main content of
12:50:10the database. So it stores the actual
12:50:13data all the informations that are
12:50:15relevant for the users. So it's stored
12:50:18there all the important data that the
12:50:20users cares about. So this is the
12:50:22storage where the users are interacting
12:50:24all the time. So where do we find the
12:50:26user data? If you go to our database
12:50:28sales DB and then you go to the tables
12:50:30now we find all these tables that we are
12:50:32already used the customers employees
12:50:34orders and so on those tables are the
12:50:37user data. So now if I go and say select
12:50:41from sales orders and all those
12:50:44informations that we are seeing now are
12:50:46the users data. So this is what we users
12:50:49actually care about. All other stuff
12:50:51that we see inside databases as a user
12:50:54we don't care about it. We care only
12:50:55about our data. But in the database, we
12:50:58don't have only the user data. We have
12:51:00many other informations. So this is what
12:51:02we mean with the user data
12:51:07storage. Now what is system catalog?
12:51:10This is the internal storage for the
12:51:12database for its own information. So
12:51:15it's like a blueprint that keeps
12:51:17tracking everything about the database
12:51:20itself. So that means the main purpose
12:51:22of the system catalog is that it holds
12:51:25the metadata informations about the
12:51:28database. So what is a metadata?
12:51:30Metadata is data about data. Now let's
12:51:33understand what this means. What we have
12:51:35done so far is that we have created a
12:51:37table called customers and we have
12:51:39defined inside it like multiple columns
12:51:41like the customer ID, first name, last
12:51:43name and then we have inserted our data
12:51:45inside this table. So we have inserted
12:51:48five customers. So those informations
12:51:50are my data. I have created those
12:51:52informations and stored it inside the
12:51:54database. That's why we call it the user
12:51:57data. So nothing so far is new. So now
12:51:59what happens behind the scenes is that
12:52:01the database server will not only store
12:52:04the user data that you have provided but
12:52:06also it's going to go and store a
12:52:08different type of data inside the
12:52:10database and this data is the metadata.
12:52:13So the database server going to store
12:52:15the metadata of the customer's table and
12:52:18it going to look like this. There is
12:52:20like a table name, there is a column
12:52:22names and those are the column names
12:52:24that you have defined inside your
12:52:26database and those are the column names
12:52:29that you have defined as you are
12:52:31creating the table customers and it's
12:52:33going to store as well additional
12:52:35informations like which data type like
12:52:37the customer ID is int and the last name
12:52:39is v charts and many other informations
12:52:41like the length of the column and
12:52:44whether the column is nullable or not.
12:52:46So as you can see in the metadata we are
12:52:48having a description a data about the
12:52:52structure of the customers and in the
12:52:54metadata we can find a lot of
12:52:56informations about not only the tables
12:52:57and columns but as well about the
12:52:59schemas and the database. So you can
12:53:02find a full catalog about the structure
12:53:04of your database. Basic table the
12:53:06customers table it contains data about
12:53:09the actual data. So it stores data about
12:53:12the customers. But the metadata of the
12:53:14customers table contains data about
12:53:17data. So in the databases each table
12:53:20that you are using in order to store
12:53:21your data has a table twin that
12:53:24describes the structure of your data. So
12:53:27this is what we mean with a system
12:53:29catalog or a metadata. And now you might
12:53:31ask where I can find all those system
12:53:33catalog and metadata inside our client
12:53:36here. Well, you cannot navigate through
12:53:38those informations in the object
12:53:40explorer like we used to do for the user
12:53:42data. But you can find those
12:53:43informations in a special hidden schema
12:53:47called the information schema. The
12:53:49information schema in SQL server is a
12:53:52systemdefined schema that contains a set
12:53:55of built-in views that help us to find
12:53:58information about our database like
12:54:00tables, columns, and other objects. So
12:54:03let's go and explore it. We're going to
12:54:05go and say select star from
12:54:09information schema. And then let's have
12:54:11a dot. And now we get from SQL a list of
12:54:14all views that are available in order to
12:54:16browse the metadata of our database. So
12:54:19for example, you can see here tables.
12:54:21You can see informations about the views
12:54:24and as well about the columns. So let's
12:54:26go and select the columns and let's go
12:54:29and execute it. And now in the output we
12:54:31can find informations about the schema
12:54:33about the table names like for example
12:54:36here the customers. Let me just go and
12:54:38select this table. And then we find all
12:54:41the columns inside this table how they
12:54:44are sorted. So we have here the order of
12:54:46each column and as well the data type
12:54:49and the size of each column and many
12:54:52other stuff. So as you can see we got
12:54:54here all the informations all the
12:54:56metadata of each table and as well for
12:55:00each column inside the table. So with
12:55:02that you can check which tables does
12:55:04exist in your database. For example I
12:55:05find here like something called test
12:55:07two. So maybe I was trying to test
12:55:09something. I can go now and clean up
12:55:11stuff right and this is exactly why the
12:55:13database maintain such a catalog. It
12:55:16helps the database to quickly find the
12:55:18structure of each table and of each
12:55:20column. and it helps me as well as a
12:55:22user to browse the catalog of the
12:55:24database. So for example I can go over
12:55:26here and say okay let's get a distinct
12:55:29table name. So with that I will get a
12:55:32list of everything that I have inside
12:55:35the database. So we have the customers
12:55:37employees and some tests that I have
12:55:39done. So metadata are
12:55:43awesome. Now we come to the third
12:55:45storage that temporary data storage. It
12:55:48is a temporary space used by the
12:55:50database for short-term task like
12:55:53processing a query or sorting data. And
12:55:55once these tasks are done, what going to
12:55:58happen? The database going to go and
12:55:59clean up the storage. And now of course
12:56:01the question is where we can find these
12:56:02temporary tables that is using the
12:56:05temporary storage in the disk. Well
12:56:07actually if you go to the object
12:56:08explorer you will not find it inside our
12:56:11database sales DB but you will find it
12:56:14inside the system databases. Now since
12:56:16we are working locally we have the full
12:56:18access to everything inside the SQL
12:56:20server. But in real projects if you are
12:56:22just a user or let's say developer you
12:56:25will not have access to the system
12:56:26databases only for the database
12:56:29administrators. But now we are working
12:56:30on the local copy. So let's go to the
12:56:33system database and here you have a
12:56:35special database from the SQL server
12:56:37called temp DB. And if you go inside it
12:56:41we will find here tables and temporary
12:56:43tables. So this is exactly where you can
12:56:45find all the temporal tables that you
12:56:47are generating. Now currently we didn't
12:56:49create any temporary tables that's why
12:56:52it's empty. But once you start creating
12:56:54temporary tables you will find those
12:56:56tables underneath this folder. We will
12:56:58learn about the temporary tables in the
12:57:00next
12:57:04sections. So these are the main
12:57:06component of the database architecture.
12:57:09So now let's have an example. Now we
12:57:10have a table called orders that is
12:57:12stored inside the user storage and the
12:57:15metadata of this table is stored in the
12:57:16catalog. So now let's say that you are
12:57:18at the client side and you write a
12:57:20simple select query in order to select
12:57:22the data of the orders. So now that
12:57:24query is sent to the server in order to
12:57:26be executed and the database engine
12:57:28going to take the query in order to
12:57:29process it. So first the database engine
12:57:32going to check whether we have the data
12:57:33in the cache because if the data is
12:57:35stored in the cache then things going to
12:57:37be really fast and the database engine
12:57:38can solve the task quickly but in this
12:57:40scenario we don't have the orders
12:57:42informations in the cache that's why the
12:57:44database engine going to say okay it's
12:57:46not in the cache let's check the disk so
12:57:48it will find the orders information in
12:57:50the disk and the query going to be
12:57:51executed then the result of this query
12:57:54going to be sent back to the client side
12:57:56where at the end in return you will see
12:57:58in the output the result of the table
12:58:01orders. So this is how the SQL database
12:58:03execute very simple select
12:58:09query query is a query inside another
12:58:13query. So what this means let's have a
12:58:16sketch to understand it. So so far what
12:58:18we have learned we have different
12:58:19database tables like the orders
12:58:21customers and so on and we write a
12:58:23simple SQL queries like select from
12:58:26where. So the SQL going to retrieve data
12:58:28from the database tables and in the
12:58:30output we will get some kind of results.
12:58:32So this is so far what you have done. We
12:58:34have done very simple queries. Now in
12:58:36our query we can have things little bit
12:58:38different. So we could have another
12:58:40query that is inside our query where we
12:58:42do the same things like select from
12:58:44where. So we have now a query inside our
12:58:47query and we call this embedded query we
12:58:50call it a sub query and the original
12:58:52query the first one where we have select
12:58:54from we call it main query. So now if
12:58:56you execute the whole query what going
12:58:58to happen SQL first going to go and
12:58:59select the subquery and then it's going
12:59:01to execute it. So it's going to go and
12:59:03select and retrieve data from our
12:59:05database tables and the result of the
12:59:07subquery will not be sent to the users
12:59:09to us. So we cannot see it. What can
12:59:12happen? the result can stay inside the
12:59:14query as an intermediate results and
12:59:17then now our main query can go and start
12:59:20interacting with this intermediate
12:59:22result from the subquery. So the main
12:59:24query going to do some kind of
12:59:25operations on top of this intermediate
12:59:27results and use it for filtering or
12:59:29joining or any purpose and still the
12:59:31main query can go and query the original
12:59:34database tables. So now the main query
12:59:36has two sources for data. The original
12:59:38database tables and as well the result
12:59:41from another query. So now by looking to
12:59:44this you can see the subquery is a query
12:59:46inside the main query and it play a role
12:59:49of supporter. So it supports the main
12:59:52query with data and the main job of the
12:59:55main query is of course to get all those
12:59:57data and to show us at the end the final
13:00:00results. Now there is now two things
13:00:02about this intermediate results that we
13:00:04got as a result from the subquery. Once
13:00:06the execution of the query is completely
13:00:09done, what can happen is going to go and
13:00:11destroy this intermediate result. So
13:00:13it's going to totally drop it. So we
13:00:15will not find it anywhere. It's
13:00:17completely lost. Now the other thing
13:00:19about the intermediate results is that
13:00:21imagine you are making another query
13:00:23that is completely outside of the first
13:00:25query. We are selecting few tables from
13:00:27our database. Now you might say you know
13:00:29what is it possible to access the
13:00:32intermediate results from the first
13:00:34query. So now we are talking about
13:00:36completely external query you cannot do
13:00:38that. The intermediate result of the
13:00:40subquery is only locally known from the
13:00:44main query itself and it is not globally
13:00:46available for any other query. So the
13:00:48subquery can be used only from the main
13:00:51query.
13:00:55So with that we have understood what are
13:00:57subqueries and now you might ask me why
13:01:00do we need them in the first place? Why
13:01:02sub queries are important? Let's have
13:01:04the following sketch. Now in our complex
13:01:06task we might have to do several stuffs
13:01:08in our query. Like for example the first
13:01:10step we have to go and join tables in
13:01:12order to prepare the data and then the
13:01:14outcome of the joins should be filtered.
13:01:16So this going to be our step two. And
13:01:18then on top of that in the step three we
13:01:20have to go and do transformations like
13:01:22maybe handling the nulls or creating new
13:01:25columns and many other stuff. And the
13:01:27last step we want to go and do data
13:01:29aggregations like summarizing the data
13:01:31or finding average. Now if you go
13:01:32immediately and start writing the SQL
13:01:34query without having a plan what can
13:01:36happen you're going to end up having a
13:01:39long complex SQL query and it's going to
13:01:41be really hard to write and as well to
13:01:44understand and read. And now what we can
13:01:45do instead of that we're going to go and
13:01:47divide our task based on those steps. So
13:01:50we're going to write one query section
13:01:52for each step. For example, for joining
13:01:53tables we're going to have one query for
13:01:55filtering another one transformation
13:01:57another one and for the aggregation
13:01:59we're going to have the last query. So
13:02:00now since each step is like a
13:02:03preparation for the next step we can go
13:02:05and say each of those queries is a
13:02:07subquery. So for step one, step two,
13:02:10step three, we have sub queries and they
13:02:13are all doing like calculations and
13:02:15preparations for the last step to the
13:02:17aggregations and we call the last step
13:02:19the main query and of course the whole
13:02:21thing can exist in one single query. So
13:02:23if you want to visual this like you have
13:02:25a subquery in circle and then this
13:02:28circle belongs to a bigger circle called
13:02:30the main query. By the way, sometimes we
13:02:32call the main query as the outer query
13:02:35and the subquery we can call it an inner
13:02:37query. And of course, we can have many
13:02:40subqueries and many small circles inside
13:02:42each others to form something called
13:02:44nested queries. So this is the main
13:02:46purpose of using subqueries in our
13:02:49scripts and queries. It's going to help
13:02:51us to reduce the complexity and going to
13:02:53make it easier to read and we can have
13:02:56like a flow logical flow inside our
13:02:58queries.
13:03:03Now for the sub queries there are many
13:03:05different types and categories. So now
13:03:07what we're going to do I'm going to show
13:03:08you an overview of all those types and
13:03:10categories and then later we're going to
13:03:12deep dive into each of those types. So
13:03:14first of all if you are thinking about
13:03:16the dependencies between the subquery
13:03:18and the main query. There is mainly two
13:03:20types of subqueries. We have the
13:03:23non-correlated subquery. That means the
13:03:26subquery is independent from the main
13:03:28query. And the second type is the
13:03:30correlated subquery. It's exactly the
13:03:33opposite. The subquery gonna depend on
13:03:35the main query. Of course, we can
13:03:37explain all those stuff in details.
13:03:39Don't worry about it. So, this is the
13:03:40first group. Now, there is another group
13:03:42on how to group up the subqueries
13:03:45depending on the result type. So, I mean
13:03:48with this that the subquery has
13:03:50different output and results. For
13:03:52example, we have scalar subquery. It
13:03:54returns only one single value. or
13:03:56another type it's called the row
13:03:58subquery. It's going to return multiple
13:04:01rows and the final type called the table
13:04:03subquery. It is a subquery that returns
13:04:06multiple rows and as well multiple
13:04:08columns. Now we come to the third way
13:04:10and the last way on how to categorize
13:04:12the subqueries and this time based on
13:04:14the location and the clauses. So we are
13:04:17describing here where the subquery going
13:04:19to be used within the main query. So we
13:04:22can use it in different locations and
13:04:24clauses like the select clause or we can
13:04:27use it in the from clause and this is
13:04:29the most common type for the subqueries
13:04:31or we can use it before joining tables
13:04:33and we can use it in order to filter the
13:04:35data in the work clause and in the work
13:04:38clause as we learned there are two
13:04:39different sets of operators. We can use
13:04:42the subgrade together with the
13:04:43comparison operators the less, greater,
13:04:46equal and so on. Or we can use it with
13:04:48the logical operators like the in, any,
13:04:52all and exists. So now those are the
13:04:55different types and categories for the
13:04:57subqueries and we're going to now deep
13:04:59dive into all of them. So now let's go
13:05:01and start with the easiest category, the
13:05:03result types of the subqueries.
13:05:10Now we have different types of
13:05:11subqueries based on the results. So this
13:05:14means the amount of data that the
13:05:16subquery going to return. So the first
13:05:18type is the scalar subquery. So it is a
13:05:21subquery that it's going to return only
13:05:24one single value like for example the
13:05:26value three. Let's have an example for
13:05:28the scalar subquery. So in this query
13:05:30for example if you are saying select
13:05:32star you will get all columns all the
13:05:34rows from one table. But for the scalar
13:05:36subquery we need only one value. So how
13:05:39we usually get it is by doing some
13:05:41aggregations. For example, if you go and
13:05:43say let's get the average of sales. So
13:05:46let's execute it. And with that in the
13:05:49output we have only one value with a 38.
13:05:51We call such a query as a scalar query.
13:05:54So it has only one row and only one
13:05:57column. So this is a scalar query. All
13:05:59right. So now to the second type we have
13:06:02the row subquery. So it is a subquery
13:06:05that going to return multiple rows and a
13:06:08single column. So we're going to have
13:06:09like values 1 2 3. So it is only one
13:06:12column with multiple rows. Let's have an
13:06:14example for the row query. As you can
13:06:16see now we are saying select star from
13:06:18the table orders and now we are getting
13:06:21multiple rows and multiple columns. But
13:06:24for the row queries we need only one
13:06:26column. So you can go over here for
13:06:27example say customer ID. And if you go
13:06:31and execute it. So now if you check the
13:06:32output we have a single column and as
13:06:35well multiple rows. So we have like a
13:06:37list of values and this is what we call
13:06:40row query. All right. So now to the last
13:06:42type we have the table sub query. It's
13:06:45going to go and return multiple rows and
13:06:48as well multiple columns like any
13:06:50regular tables. So this subquery going
13:06:52to return a lot of values. Okay. So
13:06:54let's see an example of that table
13:06:56query. So if you check our example here,
13:06:58select star from orders, we got here
13:07:01multiple rows and as well multiple
13:07:04columns and of course we can go and
13:07:06select multiple columns like for example
13:07:08the order ID and the order dates. So if
13:07:12we execute it here in the output we have
13:07:15multiple columns we have two columns and
13:07:17as well multiple rows that's why this
13:07:19kind of query is as well a table query.
13:07:22All right. So with that we have learned
13:07:23the different types of subqueries based
13:07:25on the result type. Now we're going to
13:07:27go and learn how to use the subqueries
13:07:30in different locations in our query. So
13:07:32we're going to start with how to use
13:07:34subquery in the from
13:07:36[Music]
13:07:39clause. Okay. So we typically use the
13:07:42subqueries in the from clause in order
13:07:45to create temporary result sets that act
13:07:48as a table for the main query. So it's
13:07:51like in some scenarios we cannot use the
13:07:54tables directly from the database. We
13:07:56have to prepare it somehow before we do
13:07:59our actual query. Okay. So let's check
13:08:01the syntax of the sub query inside the
13:08:03from clause. So we start with the usual
13:08:05stuff where we go and say select and few
13:08:08columns that we want to retrieve and
13:08:10then we say okay from usually after the
13:08:12from comes the table name from our
13:08:14database that we want to query. But this
13:08:17time instead of writing the table name,
13:08:19we're going to have another SQL query.
13:08:22So that means we don't define the table
13:08:24name, we define another select
13:08:26statements where we have as well again
13:08:28select a column from specific table and
13:08:31then maybe we have a filter. And in
13:08:33order now to tell SQL this is a
13:08:35subquery, we have to use the
13:08:37parenthesis. So we're going to have the
13:08:38parenthesis at the start and at the end.
13:08:41This is a subquery. This is not the main
13:08:43query. And after the parenthesis, we can
13:08:45go and define the alias for the results
13:08:48that we're going to get from this
13:08:49subquery. In many databases, this alias
13:08:52is an optional, but for the SQL server,
13:08:54we have to go and specify an alias. So,
13:08:57it is a must in SQL server. So, again,
13:08:59we call this a subquery and the outer
13:09:03query we call it a main query. So, this
13:09:05is the syntax of the subquery in the
13:09:07from clause. Okay. Okay, so now we have
13:09:09the following task and it says find the
13:09:11products that have a price higher than
13:09:14the average price of all products. So
13:09:16we're going to do it step by step and
13:09:18here we have two steps. The first one is
13:09:20that we have to go and calculate the
13:09:22average price of all products and the
13:09:24second step we're going to use this
13:09:25value in order to filter the table
13:09:28products in order to find the prices
13:09:30that is higher than this average price.
13:09:33So let's start with the first step where
13:09:34we're going to find the average price.
13:09:36I'm going to select the following
13:09:38informations. So product ID,
13:09:42price from the table sales products. So
13:09:47let's go and execute it. So now we have
13:09:49the product and as well the prices and
13:09:51we need this price here in order to
13:09:53compare it with the average price. So
13:09:56that means we need this price and as
13:09:58well side by side we need the average
13:10:00price. So that means we need
13:10:02aggregations and details and that's why
13:10:04we're going to go with the window
13:10:05function average. So let's go and do
13:10:07this. This is very simple. So it's going
13:10:09to be the average
13:10:11price and we don't want to partition the
13:10:14data. So it's going to be an over empty
13:10:16and this going to be the average price
13:10:19like this. So let's go and execute it.
13:10:21And with that we have calculated the
13:10:23average price. So now we have all the
13:10:24informations in the first step. We have
13:10:26the average price, we have the price and
13:10:28as well the products. So now the next
13:10:30step is that we have to go and filter
13:10:32the data to find out all the products
13:10:35where the price is higher than the
13:10:37average. That means we will do this step
13:10:40based on those information that we have
13:10:42now. So that means we have to go and use
13:10:44the logic of subquery and main query.
13:10:47Since this is the first step to prepare
13:10:49the data, we're going to use this as a
13:10:52subquery. So we're going to call this a
13:10:54sub query like this. And we have to go
13:10:58and use it in the main query. So how we
13:11:01going to do that? We have to go and
13:11:02write the main query. So it's going to
13:11:04be I'm going to start over here. Select
13:11:07and then I will take all the columns
13:11:10from. So this is the main query. Let me
13:11:14just make this a little bit smaller. And
13:11:17what we're going to do now so now the
13:11:18main query going to get the data from
13:11:20the sub query. So the whole thing going
13:11:23to be used inside the from close. So now
13:11:26in order to put the subquery inside the
13:11:28main query we have to go and use the
13:11:30parenthesis. So we're going to have it
13:11:32at the start and as well at the end and
13:11:34what we usually do we go and add like a
13:11:37tab in order to understand okay this is
13:11:40the subquery and then this is the main
13:11:42query. So now one more thing that we
13:11:44have to add for the whole subquery in
13:11:46the SQL server that we have to give it
13:11:48an alias. So you can go and give it any
13:11:50name that you would like. I usually go
13:11:52with only one character with the T. It
13:11:55stands for table. So you can use
13:11:56anything that you want. But we have in
13:11:57SQL Server to give an alias for the
13:12:00subquery. So now what we are saying, we
13:12:02are saying select everything from the
13:12:04subquery. If you go over here and
13:12:06execute it, you will get the exact same
13:12:09results because the main query is doing
13:12:11nothing. It's saying just select
13:12:12everything from the subquery. But now in
13:12:15order to solve the task, we are not
13:12:16interested with all products. We are
13:12:18interested only the products where the
13:12:19price is higher than the average. That's
13:12:22why we have to go and use the where
13:12:24clause. So we're going to say where the
13:12:27price is higher than the average price.
13:12:32So this filtering is done in the main
13:12:34query. It's not inside the subquery. So
13:12:37now that means in the main query we are
13:12:39doing something. Let's go and execute
13:12:41it. And with that we saw the task. We
13:12:43are getting now two products where the
13:12:45price is higher than the average price.
13:12:47So as you can see it's very simple. If
13:12:49the task has multiple steps then we can
13:12:51do that using multiple sub queries until
13:12:54we have the main query and we can learn
13:12:56from this that the subquery is here is
13:12:58only to support the main query. So we
13:13:01are preparing here that all the data
13:13:03that we need in order to have the final
13:13:05result for the main query. So for this
13:13:07task we cannot go immediately
13:13:09calculating the results we have first.
13:13:12So for this kind of task we cannot
13:13:13immediately like put everything in one
13:13:15select query. We have first to prepare
13:13:18the data in one subquery and then pass
13:13:20the values for the main query. And this
13:13:23is what we mean with the table subquery.
13:13:26And here one quick tip for you. If you
13:13:27would like to see the intermediate
13:13:29results that we are getting from the
13:13:31subquery, you can go and highlight the
13:13:33subquery itself without the parenthesis.
13:13:36So we are just highlighting the
13:13:38subquery. You can go now and execute it.
13:13:41And with that SQL will not go and
13:13:43execute everything. SQL going to execute
13:13:45only what you are highlighting. So this
13:13:47is really nice way in order to see the
13:13:49results of the subquery as you are like
13:13:52debugging or searching for errors. You
13:13:54can go and see the intermediate results
13:13:56that is used from the main query. And of
13:13:58course if you deselect and not highlight
13:14:00anything and execute SQL going to go and
13:14:03execute everything the whole query. So
13:14:05this is how we use the table sub query
13:14:08inside the from close. All right. Right.
13:14:10So let's have another task and it says
13:14:12rank the customers based on their total
13:14:15amount of sales. So again if you check
13:14:17here we have like two steps. First we
13:14:20have to find the total amount of sales
13:14:22and then after that we have to go and
13:14:25rank the customers. So again we have
13:14:27like two steps and we can use the
13:14:29subqueries in order to solve it. So
13:14:31let's start with the first step where
13:14:32we're going to find the total amount of
13:14:34sales. So let's go and select the
13:14:36customer ID and as well the
13:14:39sales from the table sales
13:14:44orders. Let's go and execute it. So now
13:14:47in the output we have like multiple
13:14:48customers and their sales. We have to go
13:14:50and now find the total amount of sales
13:14:53for each customer. That means we have to
13:14:55go and use the group by. So we're going
13:14:57to go and summarize the
13:15:00sales. So total sales and then group up
13:15:05the data by the customer ID. So like
13:15:08this. Let's go and execute it. Now as
13:15:11you can see in the output we have four
13:15:12customers and we have the total sales
13:15:14for each customer. And with that we have
13:15:16solved the first step. We have the total
13:15:18amount of sales for each customer and we
13:15:20have now prepared the data for the next
13:15:23query in order to rank the customers. So
13:15:26now I think you already getting how
13:15:28important are the subqueries in order to
13:15:30do stepby-step analyszis. So this is our
13:15:33subquery. Now we need the main query. So
13:15:37I will start preparing it. So main query
13:15:39like this. And let's go first and select
13:15:42everything. So select star from let me
13:15:46just make this a little bit bigger like
13:15:48this. And now we have to go and convert
13:15:50this query to a subquery. So we need the
13:15:52parenthesis. So the starting and the
13:15:55ending and for the SQL server I'm going
13:15:57to give it an alias and I would like to
13:15:59push everything to the right side. So
13:16:02let's go and execute it. Perfect. So it
13:16:04is working with that the subquery is
13:16:07passing the data in the from clause to
13:16:09the main query. Now of course the main
13:16:11query is now is useless. It's just like
13:16:13selecting the data. We have to go and
13:16:15calculate the rank and for that we have
13:16:17a very nice window function. So we're
13:16:20going to go and use the rank. So it
13:16:22doesn't need any parameters over we have
13:16:25to sort the data order by. So we have to
13:16:28go and sort the data by the total sales
13:16:30descending from the highest to the
13:16:32lowest. So we're going to go with the
13:16:34total sales and descending. So now as
13:16:37you can see we are using the total sales
13:16:39that we have already prepared in the
13:16:41subquery. So without preparing first the
13:16:43data we will not be able to rank the
13:16:46customers in the main query. So that's
13:16:48it. Let's go and execute it. And with
13:16:50that SQL sorted our data and we have a
13:16:52nice ranking based on the data that we
13:16:55had from the subquery. So this is the
13:16:58highest customer with the sales and then
13:17:00the customer number one and so on. So
13:17:02again in this task we have like multiple
13:17:04steps and we use the power of the
13:17:06subqueries in order to do it step by
13:17:08step. So that's all on how to use the
13:17:10subquery inside the from close. Okay. So
13:17:13now let's see quickly how SQL executed
13:17:15our query. So we have here our query and
13:17:17we are quering the table orders. So the
13:17:19first step is that SQL going to go and
13:17:21identify the subquery and then it going
13:17:24to go and execute it. So SQL going to go
13:17:26and execute the subquery part where we
13:17:28are aggregating the data based on the
13:17:30customer ID. So once the subquery is
13:17:32executed the next step is that the
13:17:35result going to be introduced as an
13:17:37intermediate results. So these results
13:17:39we will not see it in the output. It's
13:17:41going to be like temporarily saved in
13:17:43the memory. So now the next step is that
13:17:46SQL going to go to the main query and
13:17:48it's going to execute it based on the
13:17:50intermediate results. So that means the
13:17:52main query will not go back to the
13:17:54original table. It's going to go and
13:17:56query the intermediate results. So here
13:17:58what SQL going to do going to go and
13:18:00rank the intermediate results by
13:18:02introducing a new column where we see
13:18:04the ranks 1 2 3 4 and the output of the
13:18:06main query going to be the final
13:18:08results. So as you can see it's very
13:18:09simple. First SQL is executing the
13:18:11subquery and the result of the subquery
13:18:14going to be used in the main query and
13:18:16once the main query is executed we will
13:18:19get the final results. So the subquery
13:18:21here is only supporting the main query.
13:18:24So those are the steps that SQL uses in
13:18:26order to execute the
13:18:32subqueries. So now let's understand how
13:18:34the database server execute the
13:18:36subqueries behind the scenes. Let's go.
13:18:39So now let's say that you are data
13:18:40analyst and you are writing a query at
13:18:42the client side where you have a
13:18:44subquery inside the main query. So once
13:18:46you go and execute it what's going to
13:18:48happen the database engine going to go
13:18:50and identify the subquery and in this
13:18:53situation the database going to execute
13:18:55first the subquery. So here subquery is
13:18:57like selecting and retrieving data from
13:18:59the table orders. So that means the
13:19:01database has to retrieve the data from
13:19:04the disk storage from the user data. So
13:19:07now once the subquery is executed the
13:19:09result the intermediate results going to
13:19:11be stored in the cache. So this means
13:19:14the result of the subquery is temporary
13:19:16and as well very fast to retrieve. And
13:19:19now once the database engine is done
13:19:21with the subquery it going to go and
13:19:23start executing the main query. So let's
13:19:25see in this scenario it's completely
13:19:27depending on the result of the subquery.
13:19:29So that means the main query going to go
13:19:31and interact with the cache storage. So
13:19:34this means now the data going to be
13:19:35retrieved very fast from the result of
13:19:37the subquery. Once it's done, it's going
13:19:39to forward the result to the database
13:19:41engine and the database engine going to
13:19:43forward the results to the client side.
13:19:45And at your side, you will find the
13:19:47final result. And of course, once
13:19:49everything is executed, the database
13:19:51engine going to go and clean up the
13:19:53cache. So the subquery results going to
13:19:55be destroyed and removed completely from
13:19:57the cache in order to have a free space
13:19:59for other queries. So this is how the
13:20:01database server execute the subqueries
13:20:04behind the
13:20:08scenes. All right. So now we're going to
13:20:10talk about how to use the subquery in
13:20:12the select clause. So now we typically
13:20:14use the subqueries in the select clause
13:20:17to aggregate the data side by side with
13:20:19the columns of the main query. Okay. So
13:20:22let's check the syntax of the subquery
13:20:24in the select clause. So we start with
13:20:26the simple stuff where we say okay let's
13:20:28go and select a column that we want to
13:20:30retrieve from specific table. So nothing
13:20:33new we are just quering a table. And now
13:20:35what we can do in this query is that not
13:20:37only we can go and select the columns
13:20:39from specific table we can go and insert
13:20:42here inside the select another query
13:20:45like a full query like select from and
13:20:48where. So again it's like query inside
13:20:50another query and we call this of course
13:20:52a subquery. In order to tell SQL this is
13:20:55a subquery we go and add the
13:20:57parenthesis. So with SQL going to
13:20:59understand huh this is a subquery and
13:21:01the result of this query going to be
13:21:03used in the select. So we can handle it
13:21:06like any other column. We can go and
13:21:07give it like an alias. It is here
13:21:09optional and not m to add an alias. So
13:21:12this inner query we call it a subquery
13:21:14and the outer query going to be the main
13:21:16query. So this is how you put a subquery
13:21:18in the select clause. But there is one
13:21:21rule for this query that the result of
13:21:24this subquery must be a scalar query.
13:21:27That means the result must be a single
13:21:29value because otherwise it will not
13:21:31work. SQL here is expecting only one
13:21:34value. So this is how we use the
13:21:36subquery inside the select clause. All
13:21:38right, let's have the following task and
13:21:39it says show the product ids, product
13:21:42names, prices and the total number of
13:21:45orders. So now if we check the task
13:21:47there is like two parts. The first part
13:21:49is that we are showing the details about
13:21:51the products and the second part that we
13:21:53have to go and calculate the total
13:21:55number of orders. So let's see what
13:21:56we're going to do. First let's go and
13:21:58solve this simple part here where we
13:22:00have the product ID, product names and
13:22:02prices. So we're going to go and select
13:22:05the product ID and the product and then
13:22:08the price from the table sales products.
13:22:13Let's go and execute it. So with that we
13:22:15have solved the first part of the task.
13:22:17We have the details about the products.
13:22:20Now we go and solve the second part. We
13:22:22have to go and calculate the total
13:22:23number of orders. Now this information
13:22:25come from different table from the
13:22:27products. We cannot calculate it from
13:22:29products. We have to go and query the
13:22:30orders. So now what am I going to do?
13:22:32I'm going to go and calculate this part
13:22:34in separate query. Instead of having it
13:22:37here inside the products. So let's have
13:22:39a semicolon in order to have a second
13:22:42query. So we're going to go and select
13:22:44the total number of orders. That means
13:22:46we can go simply do account star from
13:22:49the table sales orders. Let me just make
13:22:53it a little bit bigger. So we're going
13:22:55to call it total orders and a semicolon
13:22:59as well. So now if you just execute the
13:23:01whole thing, you will get here like two
13:23:03parts in the results. First you have the
13:23:05details of the products and the second
13:23:07part we have now the total number of
13:23:09orders. We have 10 orders. But now with
13:23:12that we have like two different queries
13:23:14like separated from each others and we
13:23:16have two different results. But in the
13:23:18task we have to show all those
13:23:19informations in one result. So now what
13:23:22we can do we can put one query inside
13:23:24another query. So now if you check the
13:23:26second query the total orders you can
13:23:28see we have only single value. So we
13:23:31have a scalar query scalar subquery.
13:23:34That's why we can go with this as a
13:23:36[Music]
13:23:37subquery like this. And I'm going to go
13:23:40and put everything in one line in order
13:23:42to see it. So let's remove the
13:23:44semicolons. We don't need it. And now
13:23:46what we're going to do, we're going to
13:23:47go and take the whole thing and put it
13:23:50inside the main query. So this is the
13:23:54main query. And now think about it as
13:23:57new column. So I will put the query
13:24:00here. So it is just one new column in
13:24:02our select. But in order to have it as a
13:24:05subquery, we have to use the parenthesis
13:24:08at the start and at the end. And of
13:24:10course, we have to go and give it a
13:24:11name. So I'm going to go and use the
13:24:13same name over here. So it's going to be
13:24:16as total orders. So with that, the setup
13:24:19for the subquery is ready and it is
13:24:21inside the select clause in the main
13:24:25query. Let's go and execute it. Now, as
13:24:27you can see, we have everything
13:24:28together. We have the three informations
13:24:30the product details and as well side by
13:24:33side with the total orders and since it
13:24:35is always the same value it going to go
13:24:37and be repeated for each row. So this is
13:24:39what we call scalar sub query inside the
13:24:42select clause and here again very
13:24:44important to understand if you are using
13:24:46a subquery inside the select clause only
13:24:50the scalar subquery is allowed. So for
13:24:53example instead of having one value from
13:24:54the aggregation we can go and use the
13:24:57order ID. So let's see what going to
13:25:00happen. We will get an error. It going
13:25:02to says subquery is returning more than
13:25:04one value and this is not allowed
13:25:06because we are using the subquery in the
13:25:08select clause. So that's why we have to
13:25:11have only one value and by using the
13:25:13aggregation you will get one value. So
13:25:15let's repair it. And it's working. And
13:25:17now again if you would like only to see
13:25:19the results from the subquery what you
13:25:21can do you can go and highlight the
13:25:23subquery like this without the
13:25:25parenthesis of course and you go and
13:25:27execute it and with that you can see in
13:25:29the output the 10 this is the
13:25:31intermediate results that's going to be
13:25:33passed to the main query and if you want
13:25:35the whole thing to be executed just like
13:25:38unmark it and execute and with that
13:25:40everything can be executed the subquery
13:25:42and the main query. So this is the
13:25:43scalar subquery in the select clause.
13:25:46Okay, so now let's see quickly how SQL
13:25:48executed this query step by step. So
13:25:50this is our original query and we need
13:25:52two tables from our database for it. So
13:25:54the first step is that SQL going to go
13:25:56and identify the subquery and it's going
13:25:59to go and execute it. So this is the
13:26:01first step. So the query is targeting
13:26:03the orders table and we are just simply
13:26:05doing a count. So in the output we will
13:26:08get an intermediate results where we are
13:26:10counting the number of rows of the
13:26:12orders. Now the next step is that SQ is
13:26:14going to go and pass this value to the
13:26:17main query. So this is the second step
13:26:19and if you go and pass this value to the
13:26:21main query, it's going to look like
13:26:23this. So you are saying product ID,
13:26:25products and the tin. So after SQL
13:26:27prepared the main query, SQL going to go
13:26:30and execute it. So this time we are
13:26:32targeting the products and in the output
13:26:34we will get all the informations from
13:26:36the products without any filter because
13:26:37here we don't have any work clouds and
13:26:39the final results we will get it like
13:26:42this. So we will have the product ID,
13:26:43the product and the total that we got it
13:26:46from the subquery. So as you can see
13:26:47here the subquery here is a scalar
13:26:49subquery where we have only one single
13:26:52value. So again it's very simple always
13:26:54SQL starts with the subquery and then
13:26:57it's going to go and pass the values to
13:26:59the main query and at the end the main
13:27:01query going to be executed and we will
13:27:03get the final result from it. So this is
13:27:05how SQL executed our query.
13:27:12All right, next we're going to talk
13:27:13about how to use the subquery in the
13:27:16join clause. All right, so now as we are
13:27:18joining tables in SQL, sometimes we have
13:27:20to go and prepare the data before doing
13:27:23the join to dynamically create a result
13:27:25sets for joining with another table. So
13:27:28again here we cannot join tables
13:27:30directly. We have to do a preparation
13:27:32step before doing the joins. Okay, let's
13:27:34have the following task and it says show
13:27:37all customer details and find the total
13:27:40orders of each customer. Now, of course,
13:27:42in SQL, you don't have only one
13:27:43solution, you have multiple solutions.
13:27:45But I would like to solve this task
13:27:47using the subquery. So, now if you check
13:27:49the task, we have like two parts. The
13:27:51first part we have to show all the
13:27:53customer details. And the second part,
13:27:54we have like here an aggregation find
13:27:57the total orders of each customer. So,
13:27:59now let's solve those different parts
13:28:01using two different queries. Let's start
13:28:03with the easiest one. Show all customer
13:28:05details. So I think this is very simple.
13:28:07So select star from sales customers. So
13:28:13let's go and execute it. So in the
13:28:15output we have all the details about the
13:28:16customers and we have solved the first
13:28:18part. Very simple. Now let's go and
13:28:20solve the second part. We have defined
13:28:23the total number of orders of each
13:28:25customer. That means let me just have a
13:28:28semicolon over here. We have to go to
13:28:30the table orders. So let's go and select
13:28:33first the order ID, customer
13:28:36ID from the table sales orders like
13:28:40this. So I will just highlight the
13:28:42second query and execute it. Now in the
13:28:44output we have 10 orders and we have the
13:28:46different customers. Now in order to
13:28:48find the total orders for each customer
13:28:49we have to go and use the group pie. In
13:28:51order to do that it's very simple. We're
13:28:54going to go over here and say so count
13:28:57let's go with the star and then we're
13:28:59going to go and group up the data by the
13:29:01customer ID. I will go and call this
13:29:04total orders. So let's go and execute
13:29:08only these parts and with that we have
13:29:10four customers and we have the total
13:29:12number of orders. So with that we have
13:29:14solved the second part of the task. So
13:29:17now what I'm going to do, I'm going to
13:29:18go and execute both of those queries
13:29:20using the semicolon separately like
13:29:23this. I will just make this a little bit
13:29:25bigger. So let's go and execute it. Now
13:29:27in the output we have the two results,
13:29:29all details about the customers and the
13:29:32total number of orders for each
13:29:34customer. So now what we want to do is
13:29:35to go and combine those two results in
13:29:38one. And in order to do that we can use
13:29:41the joins. So now we have to think about
13:29:43what is the first query, what is the
13:29:45second query. Since the first query
13:29:47returns all the customers that we have
13:29:49in the database, I would like to have
13:29:51this as the left table and since in the
13:29:54second query we have only four
13:29:55customers, I would like to have it then
13:29:57as the right table and I will go with
13:30:00the left join so that I don't miss any
13:30:03customer because if I do the inner join,
13:30:05I will lose the customer number five. So
13:30:07let's go and do that. So this is the
13:30:09first query in the main query. So I'm
13:30:11going to call this main query.
13:30:15And now I'm going to give this as well
13:30:17an alias like the C. And now we're going
13:30:19to go and join this table from the
13:30:22database together with the results the
13:30:24output of this query. So that means
13:30:27we're going to do it like this. Left
13:30:29join and now we're going to join with a
13:30:32sub query. So we will have our
13:30:34parenthesis. I will just put here few
13:30:36spaces so that it's clear it is a
13:30:39subquery and we need for this an alias.
13:30:41So let's go and say for example the O.
13:30:44So with that we are joining a table with
13:30:46the result of a sub query. And now of
13:30:49course what is missing is joining the
13:30:51tables using a key. Now if you check the
13:30:53two results you can see in both queries
13:30:55we have the customer ID. That's why
13:30:58we're going to join with the customer
13:31:00ID. So on then the customer
13:31:04ID with the customer ID from the sub
13:31:08query like this. So we have everything
13:31:12and let's go and execute it. Now as you
13:31:14can see in the output we have all the
13:31:16details about the customer and as well
13:31:18together with the total number of orders
13:31:21for each customer together with the
13:31:23total number of orders for each customer
13:31:25and as you can see we didn't miss any
13:31:27customer. So we have all the customers
13:31:29from the database and we can see that
13:31:31Anna doesn't have any orders. Now you
13:31:34might say you know what we have here the
13:31:35customer ID twice. So what I'm going to
13:31:37do I will select all the columns from
13:31:40the customers but from the subquery I'm
13:31:43interested only on the total orders. So
13:31:47like this let's go and execute it. Let's
13:31:49make this a little bit smaller. So now
13:31:51the results are really clean. We have
13:31:53all details from the customers and as
13:31:55well the total orders of each customer.
13:31:58And of course as we learned if you would
13:31:59like to check the results from only the
13:32:01subquery you go and highlight it and
13:32:03execute it. So as you can see you can
13:32:05put the subqueries almost everywhere and
13:32:07this is how we use subqueries inside
13:32:13joins. Okay. So now we're going to focus
13:32:15on how to use the subquery in the wear
13:32:18clause. So now in this scale as we
13:32:20learned we can go and filter the tables
13:32:22using the wear clause by using like
13:32:24static values. But now in real data
13:32:27projects we're going to go and filter
13:32:28the data based on like complex logic. So
13:32:31now in order to prepare this complex
13:32:33logic we go and use the sub queries in
13:32:36order to make like dynamic filtering for
13:32:39our main tables. And now in order to
13:32:41filter data using the wear clause we
13:32:43have to go and use operators and we can
13:32:45split it into like two groups. We have
13:32:47the comparison operators and another
13:32:49sets we can call it logical operators or
13:32:52sometime we call it subqueries
13:32:53operators. So now first we're going to
13:32:55talk about the comparison operators. So
13:32:57there are operators that we can use in
13:32:59order to compare two values in order to
13:33:01help us filtering the data based on
13:33:04specific condition. And now in SQL
13:33:06basics we have learned that we have
13:33:07different comparison operators and they
13:33:09are very simple. So in order to compare
13:33:11two values we have operator like the
13:33:13equal we have as well not equal the
13:33:15opposite. So we have greater than less
13:33:18than and as well we have greater than or
13:33:21equal to and the last one we have less
13:33:23than or equal to. So they are very
13:33:25simple. Now instead of comparing two
13:33:27values, we're going to go and compare a
13:33:29value with the result of subquery using
13:33:32the comparison operators. All right,
13:33:34let's check the syntax of the subquery
13:33:36inside the wear clause using the
13:33:38comparison operators. So we start with
13:33:40the standard stuff where we say select
13:33:42few columns that we want to retrieve and
13:33:44we want to get the data directly from
13:33:46specific table in our database and now
13:33:48we come to the where condition where we
13:33:50want to filter the table. So we say
13:33:52where and then we select specific column
13:33:55from the table one. Now since we are
13:33:57talking about the comparison operators
13:33:59we can go with operator for example
13:34:00equal and usually we go and specify here
13:34:03like static value like a number or
13:34:05string but instead of having a static
13:34:07value what we can do we can get the
13:34:10value from another select statements
13:34:12another query like here for saying
13:34:14select a column from table two and with
13:34:17a filter. So now whatever comes from
13:34:19this subquery going to be used in order
13:34:22to filter the table number one. And of
13:34:24course we are telling SQL this is a
13:34:26subquery by defining the parenthesis at
13:34:28the start and at the ends and the outer
13:34:30query going to be the main query. So as
13:34:32you can see we are using the subquery in
13:34:35order to filter the main query. And here
13:34:37in SQL if you're using subquery with the
13:34:39comparison operators we have a rule the
13:34:42subquery must be a scalar subquery. So
13:34:45only one single value. So that's all
13:34:47about how to use the subquery in the
13:34:49wear clause using the comparison
13:34:51operators. All right. So now we have
13:34:53again the same task and it says find the
13:34:56products that have a price higher than
13:34:58the average price of all products. We
13:35:01have solved this task already using the
13:35:02subquery inside the from clause. But now
13:35:05we're going to go and solve it again
13:35:07using the subquery but this time inside
13:35:09the wear clause. So let's do it step by
13:35:11step. Let's go and get the informations
13:35:14that we need. So we need the product ID,
13:35:17we need the price from the table sales
13:35:21products. So let's go and execute it. So
13:35:23now we got the list of all products. But
13:35:26we have to go and filter those
13:35:28informations using the column price. So
13:35:31with that in the result, we got all the
13:35:32products, but we don't need all the
13:35:34products. We need only the products
13:35:36where the price is higher than the
13:35:38average. That means we have to go and
13:35:40filter the table based on the values of
13:35:42the price. So now in order to do that
13:35:44what we're going to do we're going to
13:35:45use the wear clause and we have to go
13:35:48and filter the data based on the price
13:35:50and since we need higher than we're
13:35:52going to go and use the compressor
13:35:53operator higher than now next we need
13:35:55the value average price. So how we going
13:35:58to do it? We don't have the average
13:36:00price like out of the box in the table
13:36:02products. We have to go and calculate
13:36:04it. That's why we're going to go and
13:36:06write another query where we're going to
13:36:08go and find the average
13:36:11price from the table sales products like
13:36:16this. So now let's go and highlight it
13:36:18and then execute it. And with that we
13:36:21got now the average price of our
13:36:23products. And as you can see in the
13:36:24output we have only one single value. So
13:36:27this is a scalar query. So now what we
13:36:30need? We need this value in order to be
13:36:32used in order to filter the first query.
13:36:35So that's why the first query is the
13:36:38main query bigger. The second one is the
13:36:41subquery that going to support the main
13:36:44query in order to filter the data. So
13:36:46now what we're going to do, we're going
13:36:48to take the subquery and use it in the
13:36:51wear clause. And now of course we have
13:36:52to tell SQL this is a subquery. That's
13:36:54why we have to put it inside two
13:36:56parenthesis. So with that we have the
13:36:58sub query inside the wear clause in
13:37:01order to filter the main query. So let's
13:37:03go and execute it. And now as you can
13:37:05see in the output we have now only two
13:37:07products where the price is higher than
13:37:10the average price. So with that we have
13:37:12solved the task but this time using the
13:37:14subquery in the wear clouds in order to
13:37:17filter the main query. And of course in
13:37:18order to see this value in our select
13:37:21since it is scalar sub query we can as
13:37:23well go over here and put it in our
13:37:26select just in order to see the value.
13:37:28So average price. So let's go and
13:37:31execute it. And with that we can see as
13:37:32well in our results the average price.
13:37:35So this is how we use the subquery in
13:37:36the workcloud using the comparison
13:37:38operator. Okay. So let's see quickly how
13:37:40is going to execute our query step by
13:37:42step. So as usual first is going to go
13:37:44and identify the subquery. It's going to
13:37:47be our select average price and so on.
13:37:49And now the next step SQL going to go
13:37:51and execute our sub query. So it is
13:37:53based on the products and since we are
13:37:55doing aggregations without group by at
13:37:57the output we will get only one value.
13:38:00So the average going to be 20. This
13:38:02value is start intermediately in the
13:38:04memory. So we will not see it in the
13:38:05output. SQL going to go and pass this
13:38:08value to the main query. So the main
13:38:10query going to look like this. We are
13:38:12selecting few columns from the table and
13:38:15we are filtering the data based on the
13:38:16price that is higher than the value 20
13:38:20that we got it from the subquery. So now
13:38:22once SQL have everything for the main
13:38:24query SQL going to go and execute it. So
13:38:26SQL going to go to the products and only
13:38:28select the products where the price is
13:38:31higher than 20. So it's only those two
13:38:33rows and in the output we will get the
13:38:35final results the two products as well.
13:38:38So product ID and product price. So
13:38:40that's it. It's very simple. This is how
13:38:41SQL executed our query. So as usual
13:38:43first starting with the subquery passing
13:38:46the value to the main query and at the
13:38:48end so the main query going to be
13:38:49executed with the informations from the
13:38:51subquery and we will get at the end the
13:38:53final results. So that's
13:38:55[Music]
13:38:58it. All right. So now we're going to
13:39:00talk about the second group of operators
13:39:02and we're going to start with the in
13:39:04operator. So what is in operator? As we
13:39:07learned before in the comparison
13:39:08operators, we can go and filter the data
13:39:10based on only one single value. But now
13:39:13in some scenarios, we have to go and
13:39:14filter the data based on multiple
13:39:17values, not only one. In this case, we
13:39:19can go and use the n operator. So if you
13:39:21go and use the n operator, it's going to
13:39:23go and check whether the value matches
13:39:25any value from a list. So a list of
13:39:28multiple values. If it matches any of
13:39:30them, so we will get a true. Okay. Okay.
13:39:32So now let's have a quick look to the
13:39:34syntax of the sub query using the in
13:39:37operator. So we start with the classic
13:39:39stuff where we say okay we would like to
13:39:40retrieve the column one column two from
13:39:43the table one and we want to filter the
13:39:45data based on the column from the table
13:39:48one. Now after specifying the column
13:39:50we're going to use the in operator and
13:39:52after that we can go and specify static
13:39:54values but since we are talking about
13:39:56the subqueries the values going to come
13:39:58from another query. So here we have
13:40:01another select statements from table two
13:40:03and we filter the data for this query.
13:40:06And now the result of this subquery
13:40:08going to be used in order to filter the
13:40:10data using the in operator. And now the
13:40:13big difference between the in operator
13:40:15and the comparison operators that the
13:40:17subquery is allowed to have multiple
13:40:20rows. So there is no rule about having
13:40:22like one single value scalar subquery.
13:40:25We can have in the result a list of
13:40:28multiple values. So this is the syntax
13:40:30of the subquery using the in operator.
13:40:33All right, let's practice using this
13:40:35task. It says show the details of orders
13:40:38made by customers in Germany. So let's
13:40:40see how we can solve this task. First it
13:40:42needs the details of orders. So as we
13:40:46know we have the
13:40:47table sales orders. So let's go and
13:40:50execute it. So in the output we have all
13:40:53orders and with all details. But for the
13:40:55task we don't need all the orders. We
13:40:57need only the orders that made by
13:40:59customers from Germany. So now if you
13:41:01check the table orders, you don't find
13:41:03any informations about the countries,
13:41:05right? So we have to go and get it from
13:41:07another table. And as we know, we can
13:41:09find these informations in the table
13:41:11customers. So let's build another query.
13:41:14So let's say select star from sales
13:41:19customers like this. So let's go and
13:41:22execute only the second query like this.
13:41:25Now, as you can see in the customers, we
13:41:27have the country column, and this is
13:41:29exactly what we need. So, now let's make
13:41:30a list of all customers from Germany.
13:41:33So, we don't need all customers. We need
13:41:35only the one that come from Germany.
13:41:36That's why we're going to go and use the
13:41:38work clause and we say country equal to
13:41:41the value Germany like this. So, let's
13:41:45go and execute it again and check the
13:41:48results. Now, in the output, we have our
13:41:50German customers number one and number
13:41:52four. So now we're going to go and use
13:41:54this information in order to filter the
13:41:56table orders. So let's go back to the
13:41:58table orders over here. And here we have
13:42:00the customer ID informations. And as we
13:42:02can see we need the orders where the
13:42:04customer is either one or four. Now in
13:42:07order to filter that we're going to go
13:42:09to the first query and use the work
13:42:11clause like this and say the customer
13:42:15ID. So now since we have like two values
13:42:17one on four we can go and use the
13:42:20operator in. So let's go and use the in
13:42:22and let's go and build the list. So
13:42:25let's go and have the one and four. So
13:42:28let's go and execute it. Now we can see
13:42:30the results. We have the orders but only
13:42:33from the customers one and four. So with
13:42:36that we have solved the task. We have
13:42:37the details of orders made by customers
13:42:39in Germany. Right? And now of course
13:42:41this is really bad solution because what
13:42:43about if we get like in the future new
13:42:45customer you don't want to go and keep
13:42:48adding here like values and so on for
13:42:50each time you have a new customer. We
13:42:52want to make the values for this list to
13:42:55be dynamic. So we don't need a static
13:42:57value we need like dynamic values and we
13:43:00can use the subqueries in order to
13:43:02retrieve those informations. Right? And
13:43:04we have it already in the second query.
13:43:06So let's go back to the second query
13:43:08over here. We need only those two values
13:43:10one and four. That's why we're going to
13:43:12go to the query and say okay let's
13:43:15retrieve the customer ID. So let's go
13:43:18and execute it again. And with that we
13:43:20have with a one and four exactly like we
13:43:22have it here in the first query. And of
13:43:24course in the future if there's like
13:43:25another customer that come from Germany
13:43:27this list going to be little bit longer.
13:43:29So this query going to always retrieve
13:43:31all the customer ids that have the
13:43:34country equal to Germany. So now what
13:43:36we're going to do, we're going to take
13:43:38this as a sub query. Let's go and get
13:43:41everything from it and now put it
13:43:43instead of those static values. So of
13:43:45course we're going to go now and put few
13:43:47spaces to the right side in order to
13:43:49understand this is subquery and of
13:43:52course here we don't use any aliases. So
13:43:54now what we are doing the results from
13:43:56this subquery going to be used in order
13:43:59to filter our main query. So let me just
13:44:02call it main
13:44:04query like this and make this smaller.
13:44:08So let's go and execute it. And now we
13:44:10are getting the same results. We are
13:44:12getting all the orders from only the
13:44:14customers one and four where they come
13:44:16from Germany. And this informations come
13:44:18dynamically from the subquery and we
13:44:21don't have to worry about new customers
13:44:22from Germany. It's going to be added
13:44:24here automatically. And this query going
13:44:26to always return all the orders from
13:44:28Germany. So this is the power of the
13:44:30subquery together with the in operator
13:44:32if you are having like multiple values
13:44:34multiple rows. So we have solved the
13:44:36task. All right. Now one more thing.
13:44:38Let's say that the task is exactly the
13:44:39opposite. It says show the details of
13:44:42orders made by customers who don't come
13:44:45from Germany. So now here there's like
13:44:47two ways in order to do it. Either you
13:44:49go to the subquery and you say you know
13:44:51what the country should not be equal to
13:44:54Germany. So if you go and execute it,
13:44:57you will get all the customers ids that
13:44:59are not from Germany. And if you execute
13:45:01the whole thing, you will get all the
13:45:03orders where the customers are not from
13:45:04Germany. So either you do that or you
13:45:07stay with the equal to Germany, but you
13:45:09go and convert the whole logic by using
13:45:12the operator not. So now we are saying
13:45:14the customer ID should not be equal to
13:45:17one of those values. So it should not be
13:45:20equal to one or four. And for that we
13:45:22are using the notin operator. So let's
13:45:25go and execute it. So now with that we
13:45:27are getting all the orders where the
13:45:29customers don't come from Germany by
13:45:31just using notin operator. So that's all
13:45:34about the notin and the in operators.
13:45:37All right. So now let's see step by step
13:45:38how is execute our query. So we are
13:45:41targeting two tables the customers and
13:45:43the orders. So the first step is that
13:45:44SQL going to go and identify the
13:45:46subquery and it's going to go and
13:45:48execute it. So the subquery here is
13:45:50filtering the data based on the country.
13:45:53So the query going to be executed and in
13:45:55the output we will get only two rows. So
13:45:57it is one column with multiple rows.
13:45:59This is the row subquery and this is our
13:46:02intermediate results where it's going to
13:46:03be passed to the main query. So our main
13:46:06query going to look like this. We are
13:46:07selecting few informations from the
13:46:09orders and we are filtering the table
13:46:11orders based on the customer ID where we
13:46:14are saying the customer ID must be one
13:46:16of those values one or four. So the
13:46:18subquery here is supporting the main
13:46:21query with the informations for the
13:46:23filter. Now once SQL have everything
13:46:25going to go and execute our main query
13:46:28and this going to be like the following.
13:46:30So we will start with the first row and
13:46:32here the customer ID is equal to two. So
13:46:34the value two is not equal to 1 or four.
13:46:37That's why this row will be excluded
13:46:39from the final results. Now let's move
13:46:42to the second row. We have here the
13:46:43value three and the value three is not
13:46:46equal to one of those values. That's why
13:46:48this value going to be as well failing.
13:46:51So we will not have it at the output.
13:46:53And then it's still going to go to the
13:46:54next one. Now this time the customer ID
13:46:56is one and it is equal to one of those
13:46:59values. It's equal to one. So we have a
13:47:01match. That's why this row will be
13:47:03included to the results. And the same
13:47:05thing for the next row because we have
13:47:07the customer ID one and so on. Now after
13:47:10SQL checking all those customer ids
13:47:12whether they are in the list one or four
13:47:14we will get the final results where we
13:47:16have here all the orders where the
13:47:18customer ID either one or four. So this
13:47:21is how SQL executed the in operator
13:47:24using the
13:47:28subqueries. Okay. So now moving on to
13:47:30the any operator. So we can go and use
13:47:32the any operator in order to compare a
13:47:34value if it matches any value from a
13:47:38list. So that means we can go and use it
13:47:40in order to check whether a condition is
13:47:42true for at least one of the values in a
13:47:45list. Okay. So now let's check quickly
13:47:47the syntax of the subquery using the any
13:47:49and all operators. So as we learned
13:47:51before we can go and use a subquery
13:47:53inside the wear clause in order to
13:47:55filter the main query using like the
13:47:57comparison operators like here less
13:47:59than. Now the syntax of the any operator
13:48:02is that you're going to go and use the
13:48:04comparison operator and after that
13:48:07immediately you use the keyword any. And
13:48:09for the all operator going to be exactly
13:48:11the same where you're going to go and
13:48:13put after the comparison operator the
13:48:15keyword all. So the syntax is very
13:48:17simple. We just add those keywords. So
13:48:20let's practice using the following task.
13:48:22Find female employees whose salaries are
13:48:25greater than the salaries of any male
13:48:28employee. So that means we want to go
13:48:30and compare the salaries between the
13:48:31male and female and specifically we are
13:48:34searching for female employees whose
13:48:36salary is greater than at least one male
13:48:39employee. So let's solve it step by
13:48:41step. Let's go and start selecting few
13:48:43informations like for example the
13:48:45employee
13:48:46ID and first
13:48:49name, gender, salary from the table
13:48:53sales employees. So let's go and execute
13:48:57it. So now we have like five employees.
13:48:59Three of them are male and two are
13:49:01female. So now since we want to compare
13:49:03the data between male and female let's
13:49:05go and create two queries. The first one
13:49:07is filtering the data based on the
13:49:10gender. So the first one is for the
13:49:12female. So and we can go and remove this
13:49:15information over here. Let me just make
13:49:17this little bit smaller and zoom out.
13:49:20And the second query it's going to be
13:49:23the exact opposite. Let's go and get
13:49:25employee informations for the male. So
13:49:28let's go and execute it. Now the first
13:49:30results are the female employees and the
13:49:32second one are So now for the first
13:49:34result is for the female employees and
13:49:36the second one is for the male
13:49:38employees. So now what do we need in the
13:49:40output? We need the female employees.
13:49:42That means this is going to be our main
13:49:46query. So we are focusing on the female
13:49:49employees and we are using the male
13:49:51employees only as a filter and what we
13:49:54need we need only the salary
13:49:55informations that's why we can prepare
13:49:58it like this. I will just put everything
13:50:01in one line to make it clear. So this
13:50:04going to be our sub query. So now we're
13:50:06going to go and work with the main query
13:50:08where we're going to add one more filter
13:50:10where we're going to filter the data
13:50:12based on the salary. Right? So we're
13:50:14going to say if the salary is greater
13:50:17than and now we need the values from the
13:50:20subquery right so this is our subquery
13:50:22we're going to put it like this and
13:50:24don't forget about the parenthesis at
13:50:26the start and at the ends and I would
13:50:28like still to have those two uh queries
13:50:31so let's go ahead execute it and now we
13:50:34will get an error and that's because our
13:50:36sub query is returning multiple rows and
13:50:39this is not acceptable we are using the
13:50:41comparison operator and SQL expect from
13:50:44the subquery to have scalar subquery. So
13:50:47only one single value. But now in order
13:50:49to solve this issue, we can go and use
13:50:52the logical operators either all or any.
13:50:54So now since we are saying it's enough
13:50:56for the salary of the female employee to
13:50:58be higher than at least one male
13:51:00employee, we will go with the operator
13:51:03any. So let's go after the comparison
13:51:06operator and have the keyword any. And
13:51:08let's go and execute it again. And now
13:51:10as you can see in the output we got only
13:51:12one female employee where her salary is
13:51:15higher to one of those male employees.
13:51:18So let me just go and get the first name
13:51:20as well from the
13:51:22second query just to have it like this.
13:51:24So now if you go and compare the salary
13:51:26of Mary it is not higher than Michael
13:51:29but it is higher than Frank and Kevin.
13:51:32And since we are using the any operator
13:51:34it's enough for Mary to have salary
13:51:36higher to one of those values. In this
13:51:38case, it's higher than both Frank and
13:51:40Kevin. And the condition is fulfilled.
13:51:42That's why we are getting the marry. And
13:51:44the other female, let me just check. Do
13:51:47we have else? So, we have Carol is
13:51:49salary is less than all the salaries of
13:51:51the male employees. So, it must be at
13:51:53least higher than one of the male
13:51:56employees. So, with that, we have solved
13:51:57the task, right? All right. So, now we
13:51:59have another operator that is similar.
13:52:01We call it the all operator. We can go
13:52:04and use it in order to compare a value
13:52:06if it matches all values in a list. So
13:52:09that means we can go and use it if we
13:52:11need to check whether a condition is
13:52:13true every value in a list. I know that
13:52:16might sound a little bit complicated but
13:52:18don't worry about it. We can have
13:52:19examples. Now let's say that our task
13:52:21says find female employees whose salary
13:52:24are greater than the salaries of all
13:52:28male employees. So that means now the
13:52:30condition is more restrictive. Mary
13:52:32should now has a salary higher than
13:52:35every male employee. So it should be
13:52:37higher to all those values that we have
13:52:40from the male employees. And of course
13:52:42in this scenario it's not because we
13:52:44have Michael. Mary has less salaries
13:52:47than Michael. And this is a problem
13:52:49because Mary should has higher salary
13:52:52than everyone. So let's go and try it.
13:52:54If I go and write here all and let's go
13:52:57and execute it, you will see we will not
13:52:59find any results that fulfill this
13:53:01requirement. So we don't have any female
13:53:03employee who her salary is higher than
13:53:06all male employees and that's because we
13:53:08have a very small data sets. So this is
13:53:10how we use all and any operators in our
13:53:13subqueries in SQL. All right. So with
13:53:15that we have covered almost everything
13:53:17about how to use the subqueries in
13:53:20different locations and clauses. But we
13:53:22didn't talk about the exist operator and
13:53:24that's because I would like you to
13:53:25understand a very important concept in
13:53:27the subqueries where we have two
13:53:29different types of the subqueries based
13:53:31on the dependencies the non-correlated
13:53:34and correlated subqueries. And after
13:53:36that we're going to go back to the exist
13:53:41operator. All right friends. So now we
13:53:44come to the part where it is a little
13:53:45bit complicated about the subqueries.
13:53:47Now we're going to talk about the
13:53:49dependencies between the subquery and
13:53:51the main query. So far all the examples
13:53:54and the subqueries that we have learned
13:53:56where a noncorrelated subquery. A
13:53:59non-correlated subquery means a subquery
13:54:02that can run independently from the main
13:54:05query. So that means the subquery is
13:54:07like standalone query. But in the other
13:54:09hand we have the exact opposite type of
13:54:12the subquery. We have the correlated
13:54:14subquery. A correlated subquery is a
13:54:17subquery that relies on values from the
13:54:20main query for each row it processes. So
13:54:24that means the subquery here is
13:54:26completely depending on the main query.
13:54:29So I know this might be a little bit
13:54:30confusing. That's why we can have the
13:54:32following very simple sketch in order to
13:54:34exactly understand how this works. So as
13:54:37usual we have a database tables and now
13:54:40this time going to go and start
13:54:42executing the main query first. This is
13:54:45the first thing happens. So the main
13:54:47query going to go and query the database
13:54:50in order to get results and SQL going to
13:54:53process the results row by row. So now
13:54:55what going to happen? The main query
13:54:57going to go and pass the first row
13:54:59informations to the sub query. So now
13:55:02the subquery going to get the data from
13:55:04the main query. So SQL going to execute
13:55:06the subquery. So here the subquery going
13:55:09to return a value like for example one.
13:55:11So here it's very important to
13:55:13understand that now the SQL or the main
13:55:15query going to check is there a result
13:55:18from the sub query in this example yes
13:55:20we have a results. So here SQL is
13:55:22checking the output for the subquery for
13:55:26the first row. So if there is a result
13:55:28SQL going to go and return the row in
13:55:31the final result. So this is the whole
13:55:34iteration happened only for the first
13:55:36row. So we're going to process the whole
13:55:38thing again from the start for the
13:55:40second row. So the main query going to
13:55:42get the second row from the database and
13:55:44it going to pass it to the subquery.
13:55:47Once the subquery gets this new
13:55:49informations, SQL going to go and
13:55:51execute the subquery once again. So now
13:55:54let's say that after executing the
13:55:56subquery, there were no results. So the
13:55:58subquery is not returning anything after
13:56:01the execution. So now what can happen?
13:56:04SQL and the main query going to check
13:56:06okay there is no result from the sub
13:56:08query and this means this row should be
13:56:11excluded and not presented in the
13:56:13output. So we will not see this row at
13:56:15the output. So as you can see SQL is
13:56:18executing the subquery once again for
13:56:21the second row. So this will keep
13:56:23happening as long as we have row. For
13:56:25example, we have another row. The main
13:56:27query going to pass it to the subquery.
13:56:29The subquery going to be executed for
13:56:31the third time and the result of the
13:56:33subquery is going to be one. So the same
13:56:35thing going to happen. SQL going to
13:56:37check it. Okay, we have a value. So this
13:56:39row is allowed to be in the final
13:56:41results and so on. The cycle going to
13:56:43keep repeating for each row that's going
13:56:46to be retrieved from the main query and
13:56:48once we have processed all the rows, the
13:56:51final result going to be presented in
13:56:53the output. So what we have understood
13:56:54so far the correlated subqueries is
13:56:57always depending on the main query and
13:57:00the subquery going to be executed for
13:57:03each row that we're going to get from
13:57:05the main query. So in this example we
13:57:07have four rows and the subquery is
13:57:09executed four times. So this is how the
13:57:12correlated subquery works. It's a little
13:57:14bit more complicated than the
13:57:16non-correlated subquery. The
13:57:18non-correlated subqueries are really
13:57:20straightforward. So first the subquery
13:57:22going to go and execute the database
13:57:25only once and the output of the subquery
13:57:27going to be like an intermediate results
13:57:30that going to be used from the main
13:57:32query. So the main query going to go and
13:57:34query the intermediate results and in
13:57:36the output we're going to get the final
13:57:38results. So as you can see in the
13:57:40execution of the non-correlated subquery
13:57:42it is straightforward. There's no
13:57:44iterations everything going to be
13:57:46executed only once. So now if you
13:57:48compare them side by side you can see
13:57:50that with the non-correlated subquery it
13:57:52is completely independent from the main
13:57:55query. So that means the subquery going
13:57:57to be executed only once and after that
13:58:00SQL going to go and as well execute the
13:58:02main query only once using the result
13:58:05from the subquery. But on the left side
13:58:07the subquery is going to be executed
13:58:09multiple times and it is completely
13:58:11depending on the main query and there is
13:58:14like an iteration for each row that's
13:58:16going to be retrieved from the main
13:58:18query. So the process going to be
13:58:20cycling until all the rows are processed
13:58:23and this is exactly how the correlated
13:58:25and the non-correlated subqueries work
13:58:28in SQL. All right. So now let's have the
13:58:30following task and it says show all
13:58:31customer details and find the total
13:58:34orders of each customer. We have already
13:58:36solved this task and you know in scale
13:58:38we don't have only one query in order to
13:58:40solve something. We have multiple ways
13:58:42in order to do it. So we solved this
13:58:44task before using the subqueries and the
13:58:47joins. Now we're going to go and solve
13:58:48this task using subquery in the select
13:58:51clause and as well using the correlated
13:58:54subqueries. So again let's do it step by
13:58:57step. It's very simple. First we need
13:58:59all the customer details. So as we
13:59:02learned select star from sales
13:59:05customers. So if you execute it you will
13:59:07get all the details of all customers.
13:59:09Now we need to find the total number of
13:59:12orders of each customer. Now before we
13:59:14have solved this using a simple query
13:59:16where we have used the count function
13:59:19together with a group I but this time
13:59:21we're going to do it little bit
13:59:22different. So let's go and write query
13:59:25saying select count star from the table
13:59:30sales orders. So now let's go and
13:59:34execute it. With that we have the total
13:59:36number of orders. So let's go and take
13:59:38this sub query and use it in the select.
13:59:42So we are using it as a scalar subquery.
13:59:45So let's just put it over here. And this
13:59:47is the main query. And in order to make
13:59:50this as a subquery, what we're going to
13:59:52do, we're going to have the parenthesis
13:59:54and we're going to say the total sales.
13:59:56So now let's go and execute it. So now
13:59:58as you can see, we have here all the
14:00:00details about the customers and we have
14:00:01the total sales. But we have one issue.
14:00:03We don't need just the total order. We
14:00:05need the total orders for each customer.
14:00:08So each customer has different total
14:00:10orders. So we cannot have like the
14:00:12following setup. We cannot say group by
14:00:15customer ID. And then you have like here
14:00:18the customer ID and so on. So if you go
14:00:21and execute it, you will get a problem.
14:00:23And that's because if you go and execute
14:00:25this subquery over here, you will get
14:00:27like multiple rows and multiple columns.
14:00:30So you have like a table query. And this
14:00:32type of subquery is not allowed to be
14:00:34used in the select clause, right? We
14:00:36have to have only scalar
14:00:38subquery. So that's why we cannot do
14:00:41that. So we have to go and remove all
14:00:43those stuff.
14:00:45But we can go and solve it using the
14:00:47correlated subqueries. So now the
14:00:49subquery is completely independent from
14:00:52the main query. So in order to correlate
14:00:55it, what we're going to do, we're going
14:00:56to go and connect it. So I'm going to
14:00:57give aliases for the tables and I'm
14:01:00going to say where the customer ID equal
14:01:04to the customer ID from the main query
14:01:07from the customers. So again we are
14:01:09connecting the customer ID from the
14:01:11orders in the subquery with the customer
14:01:14ID from the table customers that comes
14:01:17from the main query. So now we are
14:01:19saying okay execute this only for a
14:01:22specific customer not for the whole
14:01:24table. So let's go and execute it. So
14:01:27now in the output we have the total
14:01:29sales for each customer and we don't
14:01:31have here like the total sales in the
14:01:33whole table orders and that's because
14:01:35what is happening for each row the
14:01:37subquery going to be executed. So for
14:01:39the customer number one this query going
14:01:42to be executed like this count the total
14:01:44number of orders where the customer ID
14:01:48equal to the one. So let me just show
14:01:50you what this means. If I go and remove
14:01:52this from here and just put the number
14:01:54one. So if I go and execute this, you
14:01:57will see the customer ID one has three
14:02:00orders. And let's just put it back and
14:02:03execute. And the same thing going to
14:02:04happen for each customer. So for each
14:02:06customer, for each row, this subquery
14:02:09going to be executed and it can be
14:02:11filtered with the customer ID that comes
14:02:13from the main query. So this is another
14:02:15way in how to solve this task using the
14:02:17correlated subqueries. So now let's
14:02:19summarize and understand quickly what
14:02:21are the differences between the
14:02:22non-correlated and the correlated
14:02:24subqueries. So now if you are talking
14:02:25about the definition the non-correlated
14:02:28subquery are subqueries that are
14:02:30independent of the main query but in the
14:02:33other hand the correlated subqueries are
14:02:35dependent of the main query. And now if
14:02:37you're talking about the execution the
14:02:39non-correlated subquery is going to be
14:02:41executed only once and then the results
14:02:44going to be used by the main query but
14:02:46by the correlated subqueries the
14:02:48subquery going to be executed for each
14:02:50row that we have from the main query.
14:02:52And as we learned for the non-correlated
14:02:54subqueries we can execute it on its own.
14:02:57So we can go and select it and execute
14:02:59it. But the correlated subqueries we
14:03:01cannot execute it on its own. So we have
14:03:03to execute always the whole thing. And
14:03:06if you are talking about which one is
14:03:07easier, I think it's clear that the
14:03:09noncorrelated subqueries are easier to
14:03:12write and to read. And in the other
14:03:13hand, the correlated subqueries are
14:03:15harder to read and as well it's complex.
14:03:18Now, if you're talking about the
14:03:19performance of the database since the
14:03:21correlated subqueries can be executed
14:03:23only once, this of course going to lead
14:03:25you to have better performance because
14:03:27things are really straightforward and
14:03:28not complicated. But in the other hand
14:03:30with the correlated subqueries there is
14:03:32more effort because SQL has to check a
14:03:35lot of stuff and the subquery going to
14:03:36be executed many times. So the
14:03:38noncorrelated subqueries are faster. We
14:03:40use the noncorrelated subqueries in
14:03:42order to do static comparison. So the
14:03:46value that we are getting from the
14:03:47subquery is executed only once and we
14:03:49will get only one static value in order
14:03:52to use it for filtering and so on. But
14:03:54in the other hand we use correlated
14:03:55subqueries in order to do rowby row
14:03:57comparison. And since we don't have here
14:03:59a static value each time the subquery
14:04:01going to run we're going to have
14:04:02different results. This going to add
14:04:04more dynamic to the filters and we don't
14:04:06have a static value. So those are the
14:04:08big differences between the
14:04:09non-correlated and the correlated
14:04:10subqueries. All right. So now after we
14:04:13understood the concept of the two types
14:04:15correlated and non-correlated subqueries
14:04:17we're going to go now and cover the last
14:04:20operator for the subqueries. We have the
14:04:22exists. So what is exist
14:04:27operator? All right. So now we're going
14:04:29to talk about a very interesting
14:04:31operator function in SQL the exists. So
14:04:34now in some scenarios as you are
14:04:35querying the data from one table you
14:04:38would need to go and check whether the
14:04:39rows of this table exist in another
14:04:42table. So that means you are checking
14:04:44like the existence of your rows in
14:04:46different table. And exactly in this
14:04:48scenario we go and use subqueries
14:04:51together with the operator exists. So
14:04:53the exist operator is very simple. It
14:04:55just simply check whether the subquery
14:04:58returns any results any rows. All right.
14:05:00So now let's understand the syntax of
14:05:02the correlated subqueries using the
14:05:04exist operator. This can be a little bit
14:05:07complicated but we're going to do it
14:05:08step by step. Don't worry about it. So
14:05:10let's start with the easy stuff. In the
14:05:12main query we're going to go and write a
14:05:14simple select. We are selecting few
14:05:16columns from the table two. And now we
14:05:18don't need all the data from table two.
14:05:20We want to filter the table using the
14:05:22wear clause. Now what we're going to do
14:05:24after the wear clause, we're going to
14:05:25write immediately another keyword called
14:05:27exists. So we don't specify any column
14:05:30before the exist like we have done in
14:05:32the comparison operator or the in
14:05:33operator. We don't need that because we
14:05:35are not filtering based on a value. We
14:05:37are filtering based on the logic. That's
14:05:39why we have the word exist immediately.
14:05:42And now directly after they exist, we're
14:05:44going to go and define the subquery like
14:05:46this. So we're going to start saying
14:05:47select one from the table number one.
14:05:50Well, it is not like a must or
14:05:52something. But it is very commonly used
14:05:54to specify here a one. We are not using
14:05:56the subquery in order to retrieve
14:05:58informations from the table one. We are
14:06:00just testing whether the subquery going
14:06:03to return a value or not. And we don't
14:06:05care about the returned value. It could
14:06:07be one, it could be column, it could be
14:06:09anything. So we don't care about the
14:06:12data that is retrieved. We are just care
14:06:14whether the subquery is returning
14:06:16anything. So that's why we go and write
14:06:18any value like here a one. So now we are
14:06:20not done yet. This subquery is not yet
14:06:23connected to the main query. We have
14:06:25somehow to go and connect them together.
14:06:27And we can do that using the wear clause
14:06:30where we go and connect the ID from the
14:06:32table one from the subquery with the ID
14:06:35from the outer query from the main
14:06:38query. And with that we are building
14:06:39like a relationship between the subquery
14:06:43and the main query. So with that the
14:06:45subquery is now depending on the values
14:06:48from the main query because here we have
14:06:50the table 2 id. So the ids from the main
14:06:53query going to filter the subquery. So
14:06:57this is the syntax of correlated sub
14:06:59queries using the exist where we are
14:07:01making the subquery depending totally on
14:07:04the main query. So let's understand how
14:07:06exist works. So now for each row that we
14:07:09have from the main query, it's going to
14:07:11trigger and cause an execution of the
14:07:14subquery. This subquery going to help us
14:07:16to evaluate this row. So we are testing
14:07:18this row. Now if the subquery doesn't
14:07:21return anything, so there is no results,
14:07:23what can happen? The row that we are
14:07:25evaluating from the main query will be
14:07:28excluded from the final results. But now
14:07:30in the other hand if the subquery is
14:07:33returning a value so we have like some
14:07:35kind of results then this row that we
14:07:37are evaluating going to be included in
14:07:40the final results. So the subquery is
14:07:42used in order to do a test. Do we have a
14:07:45results or we don't and based on this
14:07:47SQL either going to include or exclude
14:07:50the row from the final results. So this
14:07:52is the logic behind the exist in SQL.
14:07:55All right. So now we're going to go and
14:07:56solve the same task using the exists. So
14:07:58the task says show the details of orders
14:08:01made by the customers in Germany. So we
14:08:04have already solved this task using the
14:08:05in operator and the subquery. Now we're
14:08:08going to go and solve it using the
14:08:09exists. So again we're going to have the
14:08:11same logical steps that we have done
14:08:12before. So first we're going to go and
14:08:14select all the details from the table
14:08:17sales orders. So let's execute it. And
14:08:20with that we have all the orders and all
14:08:22the details. But of course we don't need
14:08:25all those informations. We need only the
14:08:27orders that's made by customers from
14:08:29Germany. So that is the first query.
14:08:31Let's go and construct the second query.
14:08:34We're going to say select star from
14:08:37sales customers. But we don't need all
14:08:40the customers. We need only the
14:08:41customers from country equal to the
14:08:45value Germany. So let's go and execute
14:08:49it. So now we have all customers that
14:08:52come from Germany. Now we have to go and
14:08:54put those two queries together in order
14:08:57to get the final results. So as we
14:08:59learned before the second query going to
14:09:01be our subquery. So it's going to be
14:09:03supporting the first query in order to
14:09:05filter the data. So the first query
14:09:07going to be our main query. Let me just
14:09:10make this smaller and the text as well.
14:09:13Now we don't need all the orders, right?
14:09:16We need only the orders where the
14:09:17customer come from Germany. So we need
14:09:19the work clause. So now we can have the
14:09:21filter logic like this. Show the order
14:09:24details only if the customer ID exist
14:09:27from the subquery. And now we have to go
14:09:30and put our subquery. So our subquery
14:09:32going to be this one over here. So let's
14:09:36just move it to the right side. And in
14:09:39order to have it as a subquery, we have
14:09:40to close the parenthesis. And now since
14:09:43exist is correlated subquery, we cannot
14:09:45have it like this. we have to go and
14:09:47connect the subquery together with the
14:09:50main query. So now the subquery is
14:09:52currently independent from the main
14:09:54query because we want to check each
14:09:56order information from the order table
14:09:58to check whether the customer exist in
14:10:01the sub query. We're going to go and add
14:10:03the condition like the following. And
14:10:06now it's like the joins we have to go
14:10:07and connect the customer ids together.
14:10:10So we're going to go over here and give
14:10:11it like an alias and as well for the
14:10:13subquery. And now we're going to say
14:10:15customer ID from the orders should be
14:10:19equal to the customer
14:10:21ID from the subquery the table customers
14:10:25like this. So again this customer ID
14:10:28come from the subquery and this customer
14:10:30ID comes from the main query. So now
14:10:33since we are using the subquery only in
14:10:36order to test the existence of the
14:10:38customer. So if the subquery returns
14:10:40anything or not, it doesn't matter what
14:10:42you are selecting in the subquery. So so
14:10:45you can go with the star or a column or
14:10:47any static value. But for some reason
14:10:50all the SQL developers decided to go
14:10:52with the static value one. And of course
14:10:54you can go and add like a column like
14:10:56the customer ID but it's like
14:10:58unnecessary step for the SQL in order to
14:11:00retrieve the information from the
14:11:02customer ID. So it's going to be way
14:11:04faster for SQL if you say okay select
14:11:06one. So let's stick with the best
14:11:08practices. Use the one value if you are
14:11:11working with exist. So this is our sub
14:11:14query and I think we have everything.
14:11:17Let's go and execute it. Now as you can
14:11:19see in the output we got all the orders
14:11:21where the customers come from Germany.
14:11:23Now of course if you want to go and try
14:11:25another value and execute you will get
14:11:28exactly the same results. So it doesn't
14:11:30matter which value you are using. So
14:11:32with that we have solved the task this
14:11:33time using the exists. Now if the task
14:11:35says show the details of orders made by
14:11:38customers that don't come from Germany
14:11:40it's going to be very simple. We're
14:11:42going to go and use the operator not
14:11:44before the exist. So where not exists.
14:11:47So now we are flipping the whole logic
14:11:49and we are saying there should be no
14:11:51matching with the subquery. So now if
14:11:54you go and execute it you will get all
14:11:56the orders where the customers don't
14:11:59come from Germany by simply using the
14:12:01not logic. And there is one more thing
14:12:03that is annoying about the correlated
14:12:05subqueries. If you compare to the
14:12:06non-correlated subqueries as we learned
14:12:09before, let me go back to the n
14:12:11operator. Now this is a non-correlated
14:12:13subquery. And if I go and select only
14:12:16the subquery, I can go and execute it
14:12:18independently. So I can go and check the
14:12:21intermediate results and like validate
14:12:23my query. But the problem with the
14:12:25correlated subquery, I cannot go and
14:12:27highlight the subquery and then go and
14:12:29execute it. And that's because in the
14:12:31syntax of the subquery we are adding a
14:12:34column that is outside our subquery that
14:12:36come from the main query. So this piece
14:12:39of information currently for the SQL is
14:12:41unknown and that's why we are getting
14:12:43this error because SQL saying okay I
14:12:45don't know where this column come from.
14:12:47So this is little bit annoying using the
14:12:49correlated subqueries you cannot go and
14:12:51test the intermediate results. But how I
14:12:53usually do it I go and test like an
14:12:55intermediate result for only one row. So
14:12:57for example, I'm going to go and pick
14:12:58like a customer here. For example, two.
14:13:01So I'm going to go and say okay, the
14:13:04customer ID should be equal to two. So
14:13:06let me just remove this from here. I got
14:13:09this value from the main query. So if I
14:13:12go now and execute it, I can see here.
14:13:14Okay, the subquery is not returning
14:13:16anything because there is no such a
14:13:18value. So with that, I'm just testing
14:13:20like one row. And of course in order to
14:13:22make this working I have to go and add
14:13:24as well the column from the main query.
14:13:28So this is why correlated subqueries are
14:13:30a little bit more hard to understand
14:13:32compared to the non-correlated because
14:13:34we cannot go and test the intermediate
14:13:36results like we can do there. So this is
14:13:38another way on how to solve this task
14:13:40using a correlated subqueries with the
14:13:43operator exists. Okay. So now let's see
14:13:45step by step how SQL executed the
14:13:47correlated subqueries using the exists
14:13:50operator. So now this time SQL will not
14:13:52start with the subquery. SQL going to go
14:13:55and start immediately with the main
14:13:56query. SQL first going to identify the
14:13:59main query and it going to go and
14:14:01execute it. But it's going to executed
14:14:03row by row. So the first row going to be
14:14:05the first customer. So now SQL going to
14:14:07go and put the first customer under the
14:14:10test. So now the next step is that SQL
14:14:12going to go and pass the value of the
14:14:14customer ID from the main query to the
14:14:17subquery. So we are doing now exactly
14:14:19the opposite. So now what going to
14:14:21happen? SQL going to prepare the
14:14:22subquery with the following information.
14:14:24So we are saying the customer ID equal
14:14:27to one and then SQL going to go and
14:14:29execute it. So now once SQL executed
14:14:31this query, we will get the result of
14:14:33one and that's because we have here
14:14:35multiple times where the customer ID is
14:14:38equal to one. So there is rows in the
14:14:41order table where the customer ID equal
14:14:43to one. So now what going to happen? the
14:14:45row from the main query going to pass
14:14:47the test and this customer going to be
14:14:50included in the final results. So now
14:14:52the next step with that is going to go
14:14:53and start testing the second customer.
14:14:56So we're going to put this customer
14:14:58under the test. Now we're going to go
14:15:00and pass the value to the subquery. So
14:15:02here we're going to have the value of
14:15:03two and then SQL going to go and execute
14:15:05this query and of course we will get a
14:15:07result because we have here multiple
14:15:09times where the customer ID equal to
14:15:11two. So that's why in the output of this
14:15:13subquery we will get one. So now it's
14:15:15still going to say great we have a value
14:15:17from the subquery that's why it is safe
14:15:20to show this customer in the output. And
14:15:22now it's still going to go to the next
14:15:24row and so on. So for the next two
14:15:26customers the same things going to
14:15:28happen. All of those customers will have
14:15:31a value from the subquery and that's why
14:15:33they are all like passing the test. So
14:15:36we will have it in the output. Now skill
14:15:38going to go to the last row from the
14:15:40table customers. So we have the Anna and
14:15:43we're going to put Anna to the test. So
14:15:45now what going to happen? SQL going to
14:15:46go and pass the value five to the
14:15:49subquery and SQL going to go and execute
14:15:51this query to the table orders. Now once
14:15:54SQL execute this query there will be
14:15:56nothing returned and that's because we
14:15:59don't have here in the table orders a
14:16:01customer ID equal to five. And now SQL
14:16:03going to say well we are not getting any
14:16:05results from the subquery. That's why
14:16:08this customer going to fail and SQL will
14:16:11not show it at the output. So it will be
14:16:14completely removed. So the customer Anna
14:16:16is excluded because the subquery is not
14:16:18returning anything. Customer ID number
14:16:20five Anna does not exist in the table
14:16:24orders. So it's going to fail the test
14:16:25and we will have in the final results
14:16:28only for customers. So this is exactly
14:16:30the purpose of the exist. we are
14:16:32checking and testing the existence of
14:16:34our rows from another table from another
14:16:37query. So this is how SQL executes the
14:16:40correlated subqueries using the operator
14:16:43[Music]
14:16:47exists. All right friends, so with that
14:16:49you have covered everything about the
14:16:51subqueries, all the different categories
14:16:53and types of the subqueries and now
14:16:55we're going to do a quick recap about
14:16:56the subqueries. So as we learned
14:16:58subqueries is just simply a query inside
14:17:01another query. And we use the subqueries
14:17:04in order to break down a complex queries
14:17:07into smaller, simpler, easy to manage
14:17:10pieces that makes everything easier to
14:17:12develop and as well to read. And as we
14:17:14learned there are like many different
14:17:16use cases for the subqueries. So we use
14:17:18subqueries in order to create temporary
14:17:21result sets to be used later from
14:17:23another query. And we learned that we
14:17:25can use the subqueries in order to
14:17:27prepare the data before joining the
14:17:30tables. And another very important use
14:17:32case for the subquery is that we can use
14:17:34it in order to filter our data using a
14:17:37dynamic and as well complex filter
14:17:40logics. And as we learned, we can go and
14:17:42use the correlated subqueries using the
14:17:44exist operator in order to check the
14:17:47existence of data and rows from another
14:17:50tables. and as well using the correlated
14:17:52subqueries help us to do rowby row
14:17:55comparison. All right my friends, so
14:17:56with that we have covered an important
14:17:58technique on how to nest your queries in
14:18:00SQL. Now in the next step we're going to
14:18:02talk about one of the most famous
14:18:04technique on how to do multi steps in
14:18:06SQL the city common table expression. So
14:18:09let's go.
14:18:14A city common table expression is a
14:18:17temporary named result set like a
14:18:19virtual table that could be used
14:18:21multiple times within your query to
14:18:24simplify and organize complex query. So
14:18:27let's understand what this means using
14:18:29the following sketch. So we have our
14:18:31database tables like orders, customers
14:18:33and so on. And in very simple scenario
14:18:35we write a simple SQL in order to query
14:18:38and retrieve the data from the database
14:18:40and then in the output we will get the
14:18:42result of the query. So this is the
14:18:44simplest version of querying data. Now
14:18:46things get complicated in our project
14:18:48and we could have the following
14:18:50technique in our query. So we still have
14:18:52this section where we are saying select
14:18:54from. But now inside our query we can
14:18:57write another query like for example
14:18:59select from where which is completely
14:19:02nothing to do with the first query and
14:19:04we can give this new query inside our
14:19:06query a name CTE and we can call this
14:19:09query a CTE query common table
14:19:12expression. And the first query outside
14:19:14this CDE we call it a main query. Now if
14:19:17you check this we have like a query
14:19:19inside another query. So now let's see
14:19:21what is going to do with this. The first
14:19:23thing is going to go and execute the
14:19:25city query. So the city query going to
14:19:28be executed and we're going to go and
14:19:30retrieve few informations from our
14:19:31database tables. Now the output going to
14:19:34be available only in the query and the
14:19:36output going to have the shape of like a
14:19:38table like for example the sales. So now
14:19:40the sales table and the orders tables
14:19:43both of them are tables but one is
14:19:45stored in the database and the other one
14:19:48is an intermediate virtual table. So now
14:19:51what can happen in the main query we can
14:19:53go and start querying the sales table
14:19:56the result from the CTE as any other
14:19:59normal table like we do to the database
14:20:01tables. So the main query going to go
14:20:03and retrieve few informations and maybe
14:20:05do some manipulations on top of the
14:20:07sales table or let's say the CTE results
14:20:10and of course the main query as well can
14:20:12go and say you know what let's go and
14:20:13query as well few tables from the
14:20:16database. So the main query has two
14:20:18sources of tables. Either get it
14:20:20directly from the database or get it
14:20:22from the table that is created inside
14:20:25the query and then once everything is
14:20:27done the final results of the main query
14:20:29going to be presented for the user as a
14:20:32final result. So as you can see the CTA
14:20:34query has one task where it generates
14:20:37like a table that lives inside our query
14:20:40and we can go and use it as we want. So
14:20:42now this intermediate table that is
14:20:44created from the city has two features.
14:20:46First this table will not live long. So
14:20:49once the query ends what going to happen
14:20:51is going to go and destroy this table.
14:20:54So it will not be available afterward
14:20:56and we are not able to query it anymore.
14:20:58So SQL is doing here like a cleanup and
14:21:00the second character about this let's
14:21:02imagine that we have another side query
14:21:05and it's retrieving tables directly from
14:21:07the database tables. Now if you say
14:21:09let's go and join those tables as well
14:21:12with the sales from the first query well
14:21:14it will not be working because SQL going
14:21:16to say I don't know what you are talking
14:21:18about and that's because the sales is
14:21:21only locally available for the main
14:21:24query in the same query. So that means
14:21:28it's not globally available like the
14:21:30database tables for any query. It is
14:21:32dedicated only for the main query within
14:21:35the same query. And now you might tell
14:21:37me bar wait I have heard this story
14:21:40before right? So this is an identical
14:21:42story to the one that you have told us
14:21:44about the subqueries. So what is exactly
14:21:46the difference between the subquery and
14:21:48the CTE? Well, you are totally right.
14:21:50The story is identical between the
14:21:52subqueries and the CTE but still there
14:21:54are differences between them. So let me
14:21:56show you few differences. Now let's put
14:21:58them side by side. We have on the left
14:22:00side the subqueries on the right side we
14:22:02have the CTE. So now if you look on how
14:22:04we wrote the CT and the subqueries you
14:22:07can see that on the subquery we are
14:22:08writing it from bottom to top. So first
14:22:11we have this inner query the subquery
14:22:14and then on top of it we have the main
14:22:16query. But now on the other hand the CTE
14:22:18we are writing it from top to bottom. So
14:22:20first we write this inner query the CTE
14:22:23query and then beneath it we're going to
14:22:25go and write the main query. So this is
14:22:27the first difference between them on the
14:22:30way we write the query. So if I'm
14:22:32thinking about subqueries, I start from
14:22:34bottom to top. If I'm thinking about
14:22:35CTE, I think from top to bottom. But
14:22:38still you say, you know what, I don't
14:22:39care how we write it. They are doing the
14:22:41same thing. The subquery is introducing
14:22:44an intermediate result that is used
14:22:46later from the main query. And the same
14:22:49thing for the CTE. It present like
14:22:51intermediate table that is used as well
14:22:53from the main query. Now let me tell you
14:22:55the big differences between them is that
14:22:57in the subquery the result can be used
14:23:00only once. So you cannot have another
14:23:02place in your main query where you go
14:23:05and reuse the result from the subquery.
14:23:07So you can use it maximum only in one
14:23:10position and only once. But in the other
14:23:12hand with the city technique, you can
14:23:14think about the sales table as a virtual
14:23:17table and not only you can use it in one
14:23:19place in the main query, you can go and
14:23:22use it in many other places. So you can
14:23:25go and join it again. So that means I'm
14:23:27using the output from the CTE query in
14:23:30two different places in the main query
14:23:32or maybe from three different places. So
14:23:34you can have another place where you go
14:23:36as well and query the sales table that
14:23:39is only available in our query. So this
14:23:41is the main and the most important
14:23:43difference between the subquery and the
14:23:44CTE. It's from the name common table
14:23:46expression. We think about the result of
14:23:48the CTE as a table. So we can go and
14:23:52select it. We can go and join it with
14:23:54any other table. So it is like a hidden
14:23:56virtual table lives inside our query.
14:23:59But the subqueries it's totally
14:24:01different. It's a result only for one
14:24:03position in the main query and it's used
14:24:06only once. So that means if you want the
14:24:08subquery in two three different places,
14:24:10you have to go and write the subquery
14:24:12three different times. So now you
14:24:14understand why do we have CTE and why do
14:24:17we have
14:24:21subqueries. All right. So with that you
14:24:23have understood what is CTE. Now the
14:24:25question is why do we need CTE in the
14:24:27first place? What is the main purpose of
14:24:30the CTE? Let's go back to the sketch.
14:24:32Now let's say in our complex SQL task we
14:24:35have to do the following step. Step one
14:24:37we have to go and join the tables
14:24:38together in order to prepare all the
14:24:40data that we need for the next step. And
14:24:42now in the second step we have to go and
14:24:44aggregate the data. Maybe we are doing
14:24:46summarizations. Now in our task we have
14:24:48to do as well different types of
14:24:50aggregations based on different data.
14:24:52And now what might happen is that we
14:24:54have to go and join again the same
14:24:56tables in order to prepare the data and
14:24:58perform different type of aggregations
14:25:00like for example the average which going
14:25:02to be in the last step. Now we have
14:25:04learned before we can go and use the
14:25:06subqueries in order to make this logical
14:25:08flow. So for step one, step two, step
14:25:11three, we will have subqueries and the
14:25:13final step going to be in the main
14:25:15query. But now if we keep doing this
14:25:17we're gonna have a problem and that is
14:25:19we are repeating the same step more than
14:25:22once. So we are joining the table twice
14:25:24in step number one and three for
14:25:26different purposes which cause us to
14:25:28have two different subqueries that looks
14:25:31exactly the same and this is exactly the
14:25:34weak point of the subqueries. It might
14:25:36introduce redundancies. So that means
14:25:39the subqueries alone will not help you
14:25:41to eliminate all the duplicates in your
14:25:44code. But still we have different
14:25:45techniques in order to solve this issue.
14:25:47So what we going to do? We're going to
14:25:49have only one step in order to join the
14:25:52tables. And then this data going to be
14:25:53used in the step two in order to
14:25:55aggregate the data. And then we don't
14:25:57need the step three of joining again the
14:25:59data. We're going to reuse the step one.
14:26:02And we're going to use the same data for
14:26:04the step four which is aggregating the
14:26:06data using average. And we can do this
14:26:09with the help of the amazing CTE. So now
14:26:11if you compare the steps in the
14:26:13subqueries with the steps with the CTE
14:26:15you can see with the CTE we are reducing
14:26:18the number of steps which can lead to
14:26:20reduce the size of the query. So now
14:26:22again here in subquery we think about
14:26:24the steps from bottom to top but in the
14:26:27city it's the way around we think from
14:26:30top to bottom. So that means the first
14:26:31step on the top it's going to be joining
14:26:33the tables and then below it going to be
14:26:35step two and step three. And of course
14:26:37since we are repeating the join we're
14:26:38going to put it in CTE and then we can
14:26:40use it twice in different places in the
14:26:43main query. So as you can see there are
14:26:45a lot of benefits of the CTE. It's like
14:26:47the subqueries. We are breaking down
14:26:49complex queries into smaller pieces that
14:26:52are easier to write manage understand
14:26:55and as well we have like a logical flow
14:26:57from step one to three but with one more
14:26:59benefit that we reduce the redundancies
14:27:03of our code. So we don't have to join
14:27:05the tables twice. Now I'm going to show
14:27:06you a simple example how the CTE makes
14:27:09our life easier in our query. We might
14:27:11have to do different stuff like for
14:27:13example we have to go and find the top
14:27:15customers. So we can put this in one CTE
14:27:18and we might need as well to calculate
14:27:20what are the top products and we can put
14:27:22as well this in another city. So you
14:27:25don't have to put everything in one big
14:27:26city. Then you can have the same issue
14:27:28of having complex query. And let's say
14:27:30that we have as well to find and
14:27:31calculate the daily revenue. And for
14:27:34this as well, we have to put it in one
14:27:35CTE. Now once we have all those parts,
14:27:38we can put everything together in the
14:27:40main query. So now if you look to this
14:27:42structure, you can see it's really easy
14:27:44to understand this code. It's easy to
14:27:47read. So CTE improves the readability of
14:27:50our queries. So that means your code is
14:27:52divided into clear sections making it
14:27:55easier to understand what each part
14:27:57does. Now if you keep looking to this we
14:27:59have another advantage of the CTE
14:28:02introduces modularity. So that means it
14:28:04breaks your code into smaller manageable
14:28:07parts. So this means instead of writing
14:28:10one huge complex query you break it down
14:28:13into smaller chunks using CTE. Each city
14:28:16is like self-contained and handles
14:28:19specific part of the problem and then
14:28:21you can combine them all together in the
14:28:23final query. It's like we are putting
14:28:25together a puzzle piece by piece. And
14:28:27now one very important advantage of the
14:28:29CTE is the reusability. So that means we
14:28:32can have a result set that is used
14:28:34multiple times inside our query. So that
14:28:37means you write the logic the code only
14:28:39once and then use it in different places
14:28:41inside your query. This is very
14:28:43important. Not only you are wasting time
14:28:45writing the same stuff over and over,
14:28:47but also it reduces the errors and
14:28:49mistakes that you might do if you are
14:28:51repeating the same code. Especially if
14:28:53later you want to go and change the
14:28:54logic then you have to go and visit each
14:28:57time you have done this logic and then
14:28:59do the changes and you might forget some
14:29:01places. That's why the CTE is amazing.
14:29:04You can write the logic once and then
14:29:06you go and reuse it in different places.
14:29:08So these are the advantages of using
14:29:10this technique the CTE inside your
14:29:13[Music]
14:29:16queries. So again you are at the client
14:29:19side and you are data analyst. You are
14:29:21writing a query where you are defining a
14:29:23CTE called details and inside it you
14:29:25have some logic and now in the main
14:29:27query you are selecting the data from
14:29:29the orders and as well you are joining
14:29:31it with the details with the CTE
14:29:33multiple times using multiple
14:29:35conditions. Now once you go and execute
14:29:37this query the database engine going to
14:29:39read the query and say aha we have here
14:29:41a CTE and it has the main priority. So
14:29:44that means it going to go and execute
14:29:46the CTE first. And now let's say that in
14:29:48the city you are retrieving data from
14:29:50the table orders and the table orders of
14:29:52course in the disk storage inside the
14:29:54user data. And now once the city is
14:29:57completely executed the database engine
14:29:59going to go and place the results in the
14:30:01cache and it's going to name this result
14:30:03as details. It's like a table name. So
14:30:06the database engine is done with the
14:30:07CTE. It's going to go now and grab the
14:30:09main query and it's going to start
14:30:11executing it step by step. So the first
14:30:13step is that to get the data from the
14:30:15orders. So since the orders exist in the
14:30:17disk storage, it going to go and
14:30:19retrieve it from there. Now the database
14:30:21engine going to check the details. Okay,
14:30:22we have it in the cache. That means we
14:30:25don't have to search for it in the disk
14:30:26storage and it going to start retrieving
14:30:28the data from the details with high
14:30:30speed. And now it's going to go to the
14:30:32second step as well joining the data
14:30:34with the details. So again the database
14:30:36engine going to go to the cache and
14:30:37going to see the table details and
14:30:39retrieve the data based maybe in
14:30:40different conditions. And then to the
14:30:42third time as well we are joining to the
14:30:44details and we're going to get the data
14:30:45from the cache. So as you can see from
14:30:47the main query we are using the result
14:30:50from the CTE multiple times in different
14:30:52places and the retrieval of all those
14:30:54informations is happening in high speed.
14:30:57So this is one big benefit of using the
14:30:59CTE is to utilize using the high-speed
14:31:02memory of the cache. So that means
14:31:04retrieving the data from the cache from
14:31:06the details is way faster than
14:31:08retrieving the data from the disk
14:31:10storage from the orders. Now once the
14:31:12main query is completely executed the
14:31:14result going to be returned to the
14:31:16database engine and then it's going to
14:31:18send it back to the client side and we
14:31:20will see the results in the output. So
14:31:22that's it. It's amazing right? This is
14:31:24how the database server execute the
14:31:26amazing technique the CTE behind the
14:31:29scenes.
14:31:33All right. So now for the CTE, we don't
14:31:35have only one CTE. We have different
14:31:37types of CTE. So mainly there are like
14:31:40two types of CTE. We have the
14:31:42nonrecursive CTE and recursive CTE. And
14:31:45we can say for the nonrecursive CTE, we
14:31:48have two subtypes. The first type is the
14:31:51standalone CTE and the second one is the
14:31:54nested CTE. And now what we're going to
14:31:56do, we're going to deep dive into each
14:31:57type. And we will start with the easiest
14:32:00form of the CTE, the standalone CTE. It
14:32:03is the simplest
14:32:08form. So what is standalone CTE? It is a
14:32:11CTE query that is defined and used
14:32:14independently in the query. So that
14:32:16means it is self-contained and it
14:32:18doesn't depend on anything. It doesn't
14:32:20depend on any other CTE or queries. So
14:32:24that means we can run the standalone
14:32:26query independently from anything inside
14:32:28our query. So let's understand what this
14:32:30means. We have our CTE. It's going to go
14:32:32and query the database tables and in the
14:32:35output we will get an intermediate
14:32:37results and then the output can be used
14:32:39from the main query. So the main query
14:32:41going to query the intermediate results
14:32:43and present in the output the final
14:32:45results. So now if you check our CTE, it
14:32:47is completely independent from anything
14:32:49else. So it simply query the database
14:32:52and it has one output. So since this CTE
14:32:55is independent from anything else we
14:32:57call it a standalone CTE. Now if you
14:33:00compare this CT with the main query you
14:33:02can see that the main query cannot be
14:33:04executed alone. And that's because it
14:33:06needs the result from the first query.
14:33:09So we cannot say the main query is
14:33:11independent cannot be executed alone. It
14:33:14always depend on the city query. So that
14:33:17means city first need to be executed
14:33:19then the main query can be executed. So
14:33:21this is what we mean with the standalone
14:33:24city. It doesn't depend on anything
14:33:26else. So now we can understand the
14:33:28syntax of the CTE. So we have a very
14:33:30simple query select from where. So it is
14:33:32a very simple select statement. Now in
14:33:35order to put it inside a CTE we can go
14:33:38and use the with clause. So it starts
14:33:41with the keyword with then the CTE name.
14:33:44It's like a table name and then we have
14:33:46the keyword as in order to say this CTE
14:33:49is defined like the following. So this
14:33:52is the definition of the CTE and it has
14:33:54two parenthesis the starting and the
14:33:56ending. So with this you are telling a
14:33:58scale okay now we are talking about CTE
14:34:01and it has a name. So if you are using a
14:34:03query inside with clause we call this a
14:34:06CTE query it is where you define the
14:34:08CTE. Now of course we don't want only to
14:34:11define a CTE. We want to use it. So
14:34:13outside of this definition we can go and
14:34:15use it like this. So we are saying
14:34:17select from the CTE name. So that means
14:34:21we want to select the data from the
14:34:23result of the CTE. And here it's very
14:34:25important to use exactly the same name
14:34:27as you define it in the width clause. So
14:34:30if you leave it like this, we can call
14:34:32this the main query. It is the place
14:34:34where we use the CTE. So this is the
14:34:37syntax of a very simple CTE in SQL.
14:34:40Okay. So now what we're going to do,
14:34:42we're going to have like a task that's
14:34:44going to keep progressing through this
14:34:45section. So we're going to start with
14:34:47the first step and we will keep adding
14:34:49steps as we progress in the CTE. So now
14:34:51the first step in this task says find
14:34:54the total sales per customer. And now of
14:34:56course since we have only one step, it
14:34:58makes no sense to use the CTE. But we
14:35:00will use it since we know that there
14:35:02will be different steps later. So let's
14:35:05start doing that. Now before I use any
14:35:07CTE, I would like just to write our
14:35:09query first. So we need the total sales
14:35:11for each customers. It's very simple. So
14:35:13we're going to go and select and what do
14:35:15we need? Let's go and get the customer
14:35:17ID and we need to do aggregations on the
14:35:21sales. So summarize the sales and we're
14:35:24going to call it total sales from the
14:35:28table. And now since this is our first
14:35:30query, we have to get the data from our
14:35:32database. So we don't have any other
14:35:34option. Our data going to be in the
14:35:35sales orders. So let's go and get it.
14:35:39And don't forget to group by for the
14:35:41aggregation. We are grouping by the
14:35:43customer ID. That's it. Let's go and
14:35:46execute it. And as you can see in the
14:35:47output, nothing is fancy. We are just
14:35:49aggregating the sales by the customers.
14:35:51So with that, we have solved the task.
14:35:53But now I would like to put my query in
14:35:55a CTE. And that's because later we're
14:35:57going to add more steps. So let's put
14:35:59our query in a city. And in order to do
14:36:02that, we're going to start with the with
14:36:04keyword. And now we have to define the
14:36:07name of the CD. So I'm going to call it
14:36:09city
14:36:10total sales like this. And then
14:36:13afterward we're going to say as and then
14:36:15we have to go and add the parenthesis at
14:36:17the start and as well at the end. And
14:36:20with that we are telling SQL this query
14:36:22is a CTE query. So that means the SQL
14:36:26should store the result of this query in
14:36:29a cache in memory to be used later in
14:36:31the main query. our CTE and of course
14:36:34what is missing is the main query and
14:36:36you have to do it exactly after the
14:36:38definition of the CTE. I will just make
14:36:40here a small comment about the main
14:36:42query. Uh let me just make this smaller
14:36:45like this. And now we have to go and
14:36:47have a very simple select
14:36:49statements from. And now I would like to
14:36:51get more details from the customers
14:36:53table. So I will just go now to the
14:36:57customers. So now we are not querying
14:37:00the CTE right? We are just querying the
14:37:02database table that we have and I would
14:37:04like to get from the customer the
14:37:06customer ID and the first name and let's
14:37:11go and get as well the last
14:37:13name. So now if we go and query this
14:37:17what happens in the output we are
14:37:18getting the data actually completely
14:37:21from the database table the customers
14:37:24and of course we are not using at all
14:37:26the CTE inside our main query. Of
14:37:28course, we can do that, but it's just
14:37:29waste of like space in the memory
14:37:31because SQL did execute this and stored
14:37:34it in the database memory. And of
14:37:36course, we would like to use the city in
14:37:38our main query. So, let's go and do
14:37:40that. So, let's go and do a join, but
14:37:43this time we're going to join the data
14:37:45from the CTE. So, let's go and get the
14:37:47name and I will just call it CTS. So
14:37:51what we are doing now we are joining the
14:37:53physical table the customers with the
14:37:56virtual table that we have created with
14:37:58the CTE that exist only in our query and
14:38:02of course not only we are joining the
14:38:04tables we would like to get the
14:38:06informations from the CTE. So CTS and we
14:38:09need only the total sales. So total
14:38:12sales. So that means those three columns
14:38:16comes from our database table customers
14:38:18and only this column the total sales
14:38:20comes from our CTE. So let's go and
14:38:22execute the whole thing. Now as you can
14:38:24see in the output everything is working.
14:38:26We have the three columns from the table
14:38:28customers and we have the total sales
14:38:31for each customer and this total sales
14:38:33comes from our city. Now as you can see
14:38:35the last customer has a null over here
14:38:38and that's because in the table orders
14:38:40we don't have the customer five. And now
14:38:42you might say you know what I would like
14:38:43to see the intermediate result from the
14:38:45CTE because what we are seeing now in
14:38:47the output is the final result from the
14:38:50main query. So now what we can do in
14:38:52order to see the result of the CTE we're
14:38:55going to mark the query in the CTE of
14:38:57course without any parenthesis or the
14:38:59width. So just the query and execute it.
14:39:02And with that you can see in the output
14:39:04the intermediate results that we are
14:39:06passing to the main query. And as you
14:39:08can see we don't have here customer
14:39:10number five. That's why in the final
14:39:12results we are getting null and that's
14:39:14of course because we are using the lift
14:39:16join. So if I execute the whole thing
14:39:19you can see we are getting the customer
14:39:20five over here with the null. So as you
14:39:22can see is very simple. We just treat it
14:39:24as any normal database table. But this
14:39:26table is created from our query that we
14:39:29have defined in the city over here. Now
14:39:31of course in the city you can use any
14:39:34kind of clauses like select from join
14:39:37group by having everything that you want
14:39:39window functions all aggregate functions
14:39:41but there is only one restriction you
14:39:44cannot go and use the order by clause so
14:39:47you cannot sort the data in the city so
14:39:50let's go and try it out let's go and say
14:39:53order by and let's say I want to sort by
14:39:56the order ID for example so let's go and
14:39:58execute it you can see here SQL is
14:40:00saying Okay, I cannot do it for you
14:40:03because order by is not allowed in many
14:40:05things. So you cannot use it in views,
14:40:07in sub queries, in comment table
14:40:10expressions, the CTE over here. So it is
14:40:12not allowed. You cannot use order by in
14:40:15the CTE. But of course you can go and
14:40:17sort the data in the main query. So if
14:40:20you go over here and say order by
14:40:23customer ID. So if we execute it, it's
14:40:27going to be working. So in the main
14:40:29query you can use order by but in the
14:40:31CTE this is the only thing that you
14:40:33cannot use inside the city. So that's
14:40:35it. This is our first CTE in this
14:40:38section. All right. So this is the
14:40:40simplest form of the CTE the standalone.
14:40:42Now we can have not only one CTE, we can
14:40:45have multiple
14:40:50CTE. So it's going to look like this. We
14:40:52have our database and this time we don't
14:40:54have only one CTE. We have multiple
14:40:57CTEes in our query and each CTE is going
14:41:01directly to the database and it will
14:41:03query the database in order to prepare
14:41:06the intermediate results. So in this
14:41:08example four CDEs is going to the
14:41:10database and preparing four different
14:41:13intermediate results and of course SQL
14:41:15going to execute it from the top to the
14:41:16bottom. So first the CD 1 then 2 3 four
14:41:19but they have nothing to do with each
14:41:21others. So now once we have all the four
14:41:23intermediate results the main query
14:41:25going to go and retrieve all those
14:41:26informations and do some magic in order
14:41:28to prepare the final result for the end
14:41:31user. So now by looking to this sketch
14:41:33you can understand all those CTE are
14:41:35independent from each others. So there
14:41:37is no nesting or something. Each CTE is
14:41:40self-contained and it could be executed
14:41:42on its own without depending on any
14:41:45other results from any other CTE or any
14:41:48other query. So it goes directly to the
14:41:50database and get the data. So that's why
14:41:53all of them are standalone CDs. And
14:41:55since we have multiple CDs, then it is
14:41:57standalone multiple CDs. That's it. It's
14:42:00simple. So now let's check the syntax of
14:42:02the multiple standalone cities. So we're
14:42:04going to start writing our first city.
14:42:06So it start with the with clause and
14:42:08then we have the city name and then the
14:42:10logic of our city. So nothing new. This
14:42:12is how we define the city. And then in
14:42:15order to use it, we're going to have our
14:42:16main query where we select from our new
14:42:19city and we make sure we are using the
14:42:21name of our city. So nothing new. Now in
14:42:24order to add another city to our query,
14:42:27what we're going to do, we're going to
14:42:28go after the definition of the city. And
14:42:31below it, we're going to go and start
14:42:33defining the city too. But this time, as
14:42:36you can see, we are not using the width
14:42:38clause. We are using a comma. So that
14:42:40means only the first city going to be
14:42:43using the with clause in order to tell
14:42:45SQL we are talking about CTE. All the
14:42:48other CDEs you're going to separate it
14:42:50using the comma. So the syntax going to
14:42:52be comma instead of with then the name
14:42:55of the CTE and then we're going to say
14:42:57as the following definition. So we're
14:42:59going to write here the query of the
14:43:01second CTE. So now of course if you want
14:43:03to go and add more CTE you go and use
14:43:06the comma below it and as well you
14:43:08define the third city. So you can have
14:43:10as much cities as you want and always
14:43:13separate it with comma but only the
14:43:15first city start with the width. And of
14:43:17course in the main query we can go and
14:43:18use the results from the city 2 where we
14:43:21are for example here joining the data
14:43:23between the city 1 and city 2. So as you
14:43:25can see in the main query here we are
14:43:27like collecting the data from these
14:43:28different cities in order to do the
14:43:31final step in the main query. It start
14:43:34with the width. So SQL understands okay
14:43:36now we are talking about CTE and once
14:43:38SQL sees after the parenthesis a comma
14:43:40SQL can understands okay now we are
14:43:42talking about another city and now if
14:43:44you don't go and use a comma after the
14:43:46parenthesis SQL can understands okay we
14:43:49don't have any more CDEs the next query
14:43:51it's about the main query so this is how
14:43:54you create multiple standalone CTE all
14:43:56right so now back to our task where we
14:43:58are creating like a report step by step
14:44:00so now we have in the task a second step
14:44:02where it says find the last order date
14:44:06for each customer. So now we have to go
14:44:07and add one more information about our
14:44:09customer. So when the last time the
14:44:12customer did order. So how we going to
14:44:13do it? Now we have to add this to our
14:44:16query. And I would like to use as well
14:44:18the CTE in order to have this logic. So
14:44:20as we learned from the first task, this
14:44:22is the first step in order to find the
14:44:25total sales for each customer. And here
14:44:27we have the main query. Now I would like
14:44:29to put now in between another CTE. And
14:44:32as we learned from the syntax, we have
14:44:34to go and add a comma. We cannot go and
14:44:36use the width again. And we have to give
14:44:38it a name. So let's call it CTE and last
14:44:42order. So latex and we have to define
14:44:44it. So as and then double parenthesis.
14:44:48And now in between we have to go and add
14:44:49our logic. So now we have to focus only
14:44:51in this logic. So forget about the other
14:44:54CTE and the main query. So we have to
14:44:57find the last order date for each
14:44:59customer. So we're going to go and query
14:45:01again the table orders. So what do we
14:45:04need? We need the customer ID. We need
14:45:06the order
14:45:08date from our table sales orders. So
14:45:13that's it for now. Let's just select it
14:45:16and execute it. And now with that you
14:45:18can see all the customers and as well
14:45:20all the orders. But we would like to
14:45:22have the highest order for each
14:45:24customer. And we can go and use our
14:45:26aggregate function, the max function. So
14:45:29what we're going to do it's like here at
14:45:30the top. So we have to go and use the
14:45:33function max and group up by the
14:45:36customer ID. So group up the customer
14:45:41ID. Uh let me just shift it like this.
14:45:44And let's give it the name last order.
14:45:47So like this. And as you can see I'm
14:45:49just selecting now only my query. I'm
14:45:52not selecting everything. And I keep
14:45:54executing in order just to check the
14:45:56results before we integrate it in the
14:45:59main query. So now as you can see we
14:46:01have for each customer one row and we
14:46:03have as well the highest order for each
14:46:05customer. So with that we have solved
14:46:07this subtask. So as you can see it's
14:46:09really easy to extend. I'm just making
14:46:11like another box and I'm adding inside
14:46:14it the business logic that I want and
14:46:16this going to solve one problem from the
14:46:18whole task. So you feel now exactly the
14:46:21power of the CTE. We are making complex
14:46:23logic but still it's easy to add. Now
14:46:26imagine you are not doing this. You are
14:46:27always extending one big query. It's
14:46:30going to be really hard to extend and
14:46:31that's why a lot of SQL developers
14:46:34really love using CTE and they like use
14:46:37it in each query or in each task that
14:46:39they have. So we have solved this task
14:46:42and we have to go now integrated in the
14:46:44main query. It's going to be very
14:46:45simple. So we're going to get over here
14:46:48and we will go and just add another
14:46:50join. So we're going to join it with the
14:46:52city and as you can see SQL now is
14:46:54offering it as a table even though it is
14:46:57not a physical table that exists in our
14:46:59database. It only lives inside our data
14:47:02but still SQL treat it as a table. And
14:47:04this is exactly what we are doing. We
14:47:06treat those informations as table. So
14:47:08city the last order and I will call it
14:47:11CL. And then of course we have to go and
14:47:14do the same condition like here. So the
14:47:18CLLO customer ID should be equal to the
14:47:21customer ID from the first table, the
14:47:23customers. And of course we have to go
14:47:25and add this new information to the main
14:47:27query. So
14:47:28CL the last order. So now what we're
14:47:31going to do, we're going to go and
14:47:33execute the whole thing. So we have now
14:47:35two CDs and as well our main query. So
14:47:37let's go and execute it. Now again let's
14:47:39check the data. The first three columns
14:47:41comes from the physical table customers.
14:47:44The fourth one, the total sales comes
14:47:46from our first city over here. So from
14:47:50here and the last order comes from our
14:47:52new city that we just defined the city
14:47:55number two. So as you can see guys,
14:47:56everything feels like organized and
14:47:58structures and we have like flow and of
14:48:01course those cities are standalone
14:48:03cities. So we can go always and select
14:48:06the city and execute it separately. It
14:48:08doesn't need anything else from outside
14:48:11this query. It just needs the tables
14:48:13inside your database. So guys again here
14:48:15pay attention if you want to add more
14:48:17CDs use the comma. You cannot go and use
14:48:19for example here I another width. So if
14:48:21I execute it I will get an error. So you
14:48:24have to separate it with this comma. And
14:48:25another mistake that I do frequently
14:48:27that I forget and go add here like to
14:48:30the last CTE a comma and this happens to
14:48:33me if I'm using a lot of CDEs. So if I
14:48:36go and do it like this, I will get as
14:48:38well an error because the main query
14:48:41doesn't need a comma. So the last city
14:48:43should not has a comma after the
14:48:44parenthesis. So I just removed it and
14:48:47execute. So guys with us we have now
14:48:49multiple cities inside our
14:48:54query. All right. So now what is a
14:48:56nested CTE? It is a city inside another
14:49:00city. So it's kind of like subqueries, a
14:49:02query inside another query. So not only
14:49:05a main query can use the result of CTE
14:49:08another CTE can use the result from a
14:49:10CTE and of course the nested CTE is like
14:49:12a main query is depend on other query
14:49:15that means you cannot go and select it
14:49:17and run it independently from the query.
14:49:20So always you have to run the CTE inside
14:49:22it first before seeing the result of the
14:49:25nested CTE. Okay. So now let's
14:49:27understand what this means. Again we
14:49:28have our database and we have a city
14:49:31query that goes directly to the database
14:49:33and queries the data from there and in
14:49:36the output we will get the intermediate
14:49:38results. And now in this scenario this
14:49:40time we will not have only one
14:49:41intermediate results because we have
14:49:43many different steps. We need another
14:49:46intermediate results before everything
14:49:47is prepared for the main query. So that
14:49:50means we have another step that's going
14:49:52to be built up on top of the first
14:49:55intermediate results. So that means we
14:49:57can have another CTE that's going to be
14:50:00quering the results from the first CTE
14:50:02and build on top of it another
14:50:04intermediate result. So as you can see
14:50:06here we have CTE1 and CTE2 and that
14:50:10means now we have like two intermediate
14:50:12results. And now of course we can go and
14:50:14add CTE 3 4 and so on. But now let's say
14:50:17that the CTE2 going to prepare the final
14:50:20intermediate result for the main query.
14:50:22So now the main query going to go and
14:50:24query the second intermedator results
14:50:26and it's going to do the final step
14:50:28where the final result can be presented
14:50:30for the user and of course if it is
14:50:32needed the main query can access not
14:50:35only the second intermediate result from
14:50:37the second CTE but also the first
14:50:40intermediate result from the CTE1. Now
14:50:42we call the first CTE a standalone CTE
14:50:45because it doesn't depend on any
14:50:47intermediate results. It goes directly
14:50:49to the database and gets the data. But
14:50:52now since the second city is completely
14:50:54depending on the city one. So this time
14:50:57we're going to call this CTE a nested
14:50:59CTE because we cannot go and execute it
14:51:02on its own. It always depends on the
14:51:04city one. And of course the main city is
14:51:06depending on everything. So as you can
14:51:07see we're using the CTE we're going to
14:51:09go and build like a chain. So this is
14:51:11what we mean with the standalone city
14:51:14and nested city. Okay. So now let's
14:51:16understand the syntax of the nested
14:51:17city. So we start as usual with the
14:51:20definition of the first city using the
14:51:22with clause and then the name of the
14:51:24city and the definition of the city. So
14:51:26here it's nothing new. Now we go and
14:51:29define the second city as we learned
14:51:31using the comma then the name of the CTE
14:51:34and the definition. So this is our CTE
14:51:36number two. So now the second CTE is
14:51:39depending on the results of the first
14:51:41CTE. So how we going to do it? It's very
14:51:43simple. Now for the CTE number two,
14:51:46we're going to select the data from the
14:51:48CTE number one. And with that, we are
14:51:50making the second city depending on the
14:51:53first one. So this means the second CTE
14:51:55is getting the data from the first one
14:51:57and it's querying the data in order to
14:51:59do the second step. And with that we are
14:52:02nesting one CTE in another. And the CTE2
14:52:06is completely depending on the first
14:52:08one. So again we call the first CTE as a
14:52:10standalone CTE because it doesn't depend
14:52:13on anything. We can execute it on its
14:52:16own and it just need the data directly
14:52:18from the database. But the second city
14:52:20since is completely depending on the
14:52:22city number one we call it a nested
14:52:25city. So they are very similar. We are
14:52:27just selecting the data from the city
14:52:29number one. And now comes our main
14:52:31query. And of course it's going to go
14:52:33and use the data from the second step.
14:52:36So it's going to go and select the data
14:52:37from the city number two. But it's still
14:52:40of course it's not a rule. It can go and
14:52:42access the data and select the data from
14:52:44the city number one. So this is how we
14:52:46can create a nested city in SQL. All
14:52:49right guys, back to our project where we
14:52:51are creating a report about the
14:52:52customers and we would like to add one
14:52:54more step. So the task is rank the
14:52:57customers based on total sales per
14:53:00customer. So this is one more step
14:53:02inside our projects and we would like to
14:53:04go and use as well the CTEs in order to
14:53:07implement this step. So now what do we
14:53:09need? We need to rank the customers
14:53:11based on total sales for each customer.
14:53:13So here like we have two steps. First we
14:53:16have to calculate the total sales per
14:53:18customer and then we have to go and rank
14:53:20it based on this information and of
14:53:22course the sales are stores inside the
14:53:24orders. So now let's go and start
14:53:26implementing the CDE. So we're going to
14:53:29have a comma and we're going to call it
14:53:31CTE customer
14:53:34rank as and then we're going to go have
14:53:36the parenthesis and inside it we're
14:53:39going to develop now the logic. So first
14:53:41we have to go and aggregate the data by
14:53:43the total sales. So select customer ID
14:53:47and then sum the
14:53:50sales from the table sales orders and
14:53:54then of course group
14:53:56by the customer id. And now I can hear
14:54:00you even telling me bar we have already
14:54:02done this. We have already this logic.
14:54:04So why we are repeating? If we go to the
14:54:06first CTE you can see we have already
14:54:09done that. And you are totally right. We
14:54:11have already the logic. So it makes no
14:54:13sense to repeat it again. And if we do
14:54:15this then we didn't understood the power
14:54:18of the city. So we don't have to repeat
14:54:20the same logic and we can reuse the city
14:54:24inside another city. So now we don't
14:54:26need all those stuff. We can go and
14:54:28focus immediately with ranking the
14:54:30customers. So first let me just select
14:54:32the data from the first city. So I'm
14:54:35going to go and select. So what do we
14:54:37have? We have customer
14:54:39ID and we have total
14:54:42sales. And we're going to select it this
14:54:44time not from any physical table. We're
14:54:47going to select our city. So like this.
14:54:50And now what we're going to do, we're
14:54:51going to go and select the whole thing
14:54:53and execute it. Well, this is the issue
14:54:56of nesting cities. Sadly, this CTE is
14:55:00completely depending on the first city.
14:55:02So we cannot go and execute it on its
14:55:04own. And this is of course very annoying
14:55:07because each time I execute the query by
14:55:10the end of the query SQL gonna go and
14:55:12destroy all the CTE. So in the memory we
14:55:16will not find the CT and that's why once
14:55:19I executed it SQL don't know anything
14:55:22about this city. And in order now to see
14:55:25the result of this we have always to
14:55:27execute as well with it the city that
14:55:30I'm using. So what I usually do I go
14:55:33over here and make everything in comment
14:55:36in the main query and now I can go and
14:55:39execute the whole thing and now I will
14:55:41see in the output the outcome of this
14:55:43nested city. So this is the big
14:55:46difference between the standalone cities
14:55:48like here and the nested. So now let's
14:55:50go back to our task. We have to rank
14:55:52those sales based on the total sales. So
14:55:55we can go and use the rank function from
14:55:58the window function. So rank over and
14:56:02now we don't have to partition the data.
14:56:04We just want to sort the data by the
14:56:07total sales
14:56:10descending. So like this the highest
14:56:12sales going to get the rank number one.
14:56:14So let's go and give it the name as
14:56:17customer rank. Now as you can see we
14:56:19have a really nice rank beside those
14:56:21informations. Customer three has the
14:56:23highest sales and customer two has the
14:56:26lowest total sales. So with that, as you
14:56:28can see, we didn't repeat ourself. We
14:56:30just reused another CTE in our current
14:56:33city. And this is exactly why this
14:56:35technique is very amazing in order to
14:56:38reduce redundancies and to reduce the
14:56:40complexity of the whole query. So nested
14:56:43are annoying to execute, but they reduce
14:56:46the redundancies of our code. Now we are
14:56:48done with our logic. We tested
14:56:50everything. So what we're going to do,
14:56:51we're going to go and integrate it in
14:56:53our main query. So let me just remove
14:56:56the comments from here and let's go and
14:56:58add it in the main query. So we will do
14:57:01the same thing. We're going to go and do
14:57:02a left join with the last city that we
14:57:05just created. So let me just call it
14:57:08CCR and the same conditions. We are
14:57:11always joining on the customer ID. But
14:57:13don't forget to rename the alias. So it
14:57:17is CCR customer ID equal to the customer
14:57:21ID from the first table. And of course
14:57:23we have to go and select the new
14:57:25information. So CCR dot customer rank.
14:57:30And now let's go and execute the whole
14:57:32thing. Now as you can see in the results
14:57:34those three columns comes from the
14:57:36customers table. The total sales comes
14:57:39from the first city. The last order from
14:57:42the second city and the customer rank
14:57:44comes from our nested city that we just
14:57:47created. So guys, it is not a simple
14:57:50task creating such a reports because it
14:57:53involves different aggregations and
14:57:55different functions, but our work is
14:57:57organized. As you can see, it's very
14:57:59simple. We have step one, step two, step
14:58:01three, and the main query. And it's
14:58:03really easy to add more components to
14:58:05our query. Now, I would like really to
14:58:08keep practicing using those nested
14:58:10queries. So, we have the following task.
14:58:11We would like to add one more step in
14:58:13our report. segment the customers based
14:58:16on their total sales. So I would like to
14:58:19implement this as well using CTE. So
14:58:22let's go and solve it. We want to go and
14:58:24add a new CTE. It's going to be CTE
14:58:27customer
14:58:29segments as and then we have to go and
14:58:32define our logic. Now if you check our
14:58:35task, it has two parts. We have to find
14:58:37the total sales and then we have to
14:58:40segment the customers based on this
14:58:41information. So it is something very
14:58:44similar to what we have done in the step
14:58:46three. So that means we don't have to go
14:58:49and calculate again the total sales. We
14:58:50have to go and use as well our amazing
14:58:53first city. So let's go and do it. What
14:58:56do we need? We need the customer ID like
14:59:00this. And let's do basic segmentations
14:59:03using the case win. So let's say case
14:59:05when the total sales if it's higher than
14:59:10100 then let's say the customer going to
14:59:13belong to the group high and let's go
14:59:16and add another category. If it's not
14:59:19higher than 100 if it is higher than
14:59:2450 then the customer going to belong to
14:59:27medium. And if the total sales is less
14:59:30or equal to 50. So what's going to
14:59:33happen? We're going to say else the
14:59:35customer belong to the low category. So
14:59:38that's it. We're going to have an end
14:59:40and let's call it customer
14:59:43segments. All right. But of course we
14:59:45have to go and select it from a table
14:59:47and it's going to be our city. So total
14:59:51sales and let's put it in our new city.
14:59:54And I would like to test it before like
14:59:56putting it inside our main query. That's
14:59:58why I will put everything in comments in
15:00:01my main query since it is a nested city
15:00:04sadly. And we will just go and select
15:00:06our new nested city like we have done
15:00:10before. So let's go and execute it. Now
15:00:13as you can see in the output we have two
15:00:15customers with the category high and two
15:00:18customers with the medium. But in order
15:00:20to make sure that everything working
15:00:22perfectly, I would like to go and add
15:00:23the total sales just to see the numbers.
15:00:26So let's go and execute it. Well, you
15:00:28can see everything is correct. So those
15:00:30customers having higher than 100 in the
15:00:33total sales and those two having higher
15:00:35than 50. But let's go and change stuff
15:00:37around. I would like to have it like 80
15:00:39as a medium just in order to have a low.
15:00:42So with that the customer number two
15:00:44having a lower sales than 80. That's why
15:00:46we are getting the segment low.
15:00:48Everything is done and we have segmented
15:00:49the users into different categories. So
15:00:52I don't need to test anymore. Let's go
15:00:54integrate it in our main query. So we're
15:00:57going to do the same things over here.
15:00:58We're going to say lift join and we're
15:01:01going to get our new CTE. So CCS and we
15:01:07have to do the join condition. Don't
15:01:11forget to change it. And we have to
15:01:13select our new nice information. It's
15:01:15going to be the customer segments. And
15:01:17now we can go and execute the whole
15:01:20thing. So we have now like four
15:01:22different cities and one main query. And
15:01:25now we can see in the output we got all
15:01:26three informations from the table
15:01:28customers. The first city, the second,
15:01:30third and this is our new column that we
15:01:33just created. So again we have done this
15:01:35using a necessityd like this. Let me
15:01:39just add
15:01:41it and it was really easy to extend and
15:01:44to add to our report. All right guys, so
15:01:46with us we have done like a many
15:01:47projects where we have analyzed the
15:01:49customer information based on different
15:01:52aspects from our data and we have done
15:01:54it like step by step and now you have
15:01:56like a feeling on how to write complex
15:01:58SQL queries using the help of the CTE
15:02:01and we have done it like step by step.
15:02:03So as you can see if you go through the
15:02:05scripts you can understand okay it is
15:02:07divided into multiple steps and each
15:02:09block is responsible for one specific
15:02:12problem of the whole report and this is
15:02:14exactly the power of the CTE it
15:02:16introduce modularity. So each CTE is
15:02:20self-contained and talk about one issue
15:02:23and this is amazing way on how to
15:02:25organize your project using SQL and how
15:02:27to structure your work.
15:02:33All right, my friends. So, now let's
15:02:34have a little break in order to have a
15:02:35real talk about the city. But first,
15:02:37some
15:02:39coffee. And now I can say that I'm
15:02:41working with SQL since really long long
15:02:43time ago, over 15 years. And I can say
15:02:46as well, I have met a lot of SQL
15:02:48developers in different projects. And if
15:02:50there is one thing that all those SQL
15:02:52developers love is the CTE, they love
15:02:55using it everywhere. like each time they
15:02:57write a query they going to be writing
15:02:59SQL CTE and of course it's fine it's not
15:03:02a bad thing but the problem with that
15:03:04they overuse it of course not all of
15:03:06them but a lot of SQL developers overuse
15:03:09using the CTE of course the CTE is very
15:03:12powerful but with power comes great
15:03:14responsibility
15:03:16remember with great power comes great
15:03:20responsibility so my advice for you
15:03:22especially if you are new to the CTS try
15:03:25to not add a new CTE each time you are
15:03:28doing something new and I saw it a lot
15:03:30like for each new calculation for each
15:03:32new column they jump immediately and
15:03:34create a new CT and what happens at the
15:03:36end we can have like massive number of
15:03:39CTE inside one query and the developer
15:03:41thinks now everything is organized and
15:03:43easy to read but believe me it's exactly
15:03:46the opposite if you open any code and
15:03:48you have a lot of CDEs and especially if
15:03:51they are necessities it is impossible to
15:03:53understand what is going on even if the
15:03:56developer like describe each CTE and the
15:03:58task of the CTE, it's going to be really
15:04:00hard to understand and as well to read.
15:04:02If everything is like nested and you
15:04:04have like I don't know 20 cities in one
15:04:06query. So it's going to be impossible to
15:04:08read and to understand and as well
15:04:10you're going to be using a lot of memory
15:04:12and you might get bad performance. So my
15:04:15advice for you try always as you are
15:04:17creating new CDs to think about how
15:04:19about to merge two CDEs in one. So it is
15:04:22really always important to rethink and
15:04:24refactor your CDEs in order to merge it
15:04:27into one and to reduce the number of
15:04:29CTE. But now if you ask me how many CTEs
15:04:31are okay in one query, well I don't have
15:04:34a magic number for that. But normally I
15:04:36tend to say between three and five CTE
15:04:39it's fine. So it's going to be easy to
15:04:41understand and to read and so on. But
15:04:43once you get more than five CTE then you
15:04:46have to rethink your code. Maybe you
15:04:48have to create another complete query so
15:04:51you don't have to put everything in one
15:04:52query. So this is my advice for you. Try
15:04:55to not overuse the CTEs in your
15:04:57projects. Not for each step always
15:04:59refactor the CTE, consolidate them and
15:05:02try to not have more than five CTEs in
15:05:04one query. So that's my advice for you.
15:05:06Be responsible using the CTE. And let's
15:05:09go back to our course.
15:05:14So with that we have learned the
15:05:16standalone CTE and the NIST CDE and both
15:05:18of them belongs to a type called
15:05:21nonrecursive CTE. So what is a
15:05:23non-recursive CDE? It means it is a city
15:05:26that is executed only once. So there is
15:05:29no repetitions or looping or anything.
15:05:31So the SQL going to execute it in one go
15:05:33and that's it. But in the other hand the
15:05:35recursive city is exactly the opposite.
15:05:38So a recursive city it is a
15:05:40selfreferencering query that repeatedly
15:05:43processing the data until a certain
15:05:46condition is met and we usually use the
15:05:48recursive city if we have like
15:05:50hierarchical structure and we want to
15:05:52navigate and travel through the
15:05:54hierarchy. I know this might be
15:05:56confusing but don't worry about it.
15:05:57We're going to have very simple
15:05:58examples. Now again we have our tables
15:06:00in the database and we have a CTE. Now
15:06:03the query of the CTE going to be
15:06:04executed for the first time and in the
15:06:07results we're going to have the initial
15:06:08data from the CTE but it is not
15:06:11everything yet. Now this intermediate
15:06:13result is not ready yet for the main
15:06:15query but instead of that it's going to
15:06:18go back to the CTE and CTE going to
15:06:21check whether the current results is
15:06:23meeting a specific condition. So now if
15:06:26the check says no it's not meeting the
15:06:29condition what's going to happen the
15:06:31city query going to be executed for the
15:06:33second time. So as you can see we are
15:06:35looping through the CTE. Now the result
15:06:38of the second iteration the second
15:06:40execution will be added to the
15:06:43intermediate result. So now the
15:06:44intermediate result has more data and
15:06:47again before we can use it from the main
15:06:49query it going to be checked from the
15:06:51CTE. Does the result fulfill the
15:06:54condition? If it's still no, then go and
15:06:57execute the CTE again. So we're going to
15:06:59have a third iteration and a new data
15:07:01going to be added to the intermediate
15:07:03result. So this is our third iteration.
15:07:06Now it's going to be checked again from
15:07:07the CTE. Did we fulfill the condition?
15:07:10If the answer is yes, then the loop
15:07:13going to break and everything else. So
15:07:15there will be no fourth iteration of the
15:07:16CTE. So with that, the CTE says okay,
15:07:19I'm done. This is the final result of
15:07:22the intermediate result. then the loop
15:07:24going to break and everything ends and
15:07:27the city will not be executed for the
15:07:29first time and now the city going to say
15:07:31okay I'm done now my intermediate result
15:07:34is ready to be used from the main query
15:07:36and now nothing new happens the main
15:07:38query going to go and retrieve the data
15:07:40from the intermediate results and do
15:07:42some magic in order to prepare the final
15:07:44results so that means there will be no
15:07:46iterations or looping inside the main
15:07:48query the looping going to be happen
15:07:50only in the CTE and that's why we call
15:07:52it recursive CTE. So now if you compare
15:07:55it with the other types, all other types
15:07:57are always in one direction and all the
15:07:59CTE is going to be executed only once
15:08:02but the recursive CTE going to be keep
15:08:04looping until the condition is met and
15:08:07only then it's going to forward the data
15:08:09to the main query. And normally we use
15:08:11the recursive CTE if you are navigating
15:08:14through hierarchical structure. So if
15:08:16you have in your data like hierarchal
15:08:17structures, you can go and use the
15:08:19recursive CTE in order to navigate
15:08:21through it. So this is the recursive
15:08:24city. Okay. So now let's check the
15:08:25syntax of the recursive CTE. It is a
15:08:27little bit complicated but we're going
15:08:28to do it step by step. So what do we
15:08:30have? We have a query and we would like
15:08:32to put it in a city. So we're going to
15:08:34have the usual stuff with clause the
15:08:37name of the city and as and then the
15:08:40query. So this is the definition of our
15:08:42city. But now if you leave it like this
15:08:44SQL going to execute it only once. But
15:08:46we would like to make a loop iteration.
15:08:48So in order to do that we have to go and
15:08:50define a second select statement inside
15:08:53our CTE like this. So we are selecting
15:08:56the data and here we have to define a
15:08:58breaking condition. So here in the
15:09:00second query we are defining a condition
15:09:02in order to break the loop otherwise
15:09:05it's going to loop for infinite or the
15:09:07system going to break. You could use it
15:09:08in the wear clause or you can use it
15:09:10even in an inner join because both of
15:09:12them are filtering the data and you can
15:09:14use it in order to break the condition.
15:09:16All right. So now still there is
15:09:18something missing. How we going to make
15:09:19like things looping? Well, we have to
15:09:21reference this CTE to itself. So what we
15:09:24going to do? We're going to say the
15:09:26second query going to select the data
15:09:28from the same CTE. So that means we have
15:09:31now a query that is quering itself. And
15:09:33this is of course what we want. We want
15:09:35to make iterations and we want to make a
15:09:37loop. That's why we have to go and
15:09:39reference it to itself. And now in SQL
15:09:41you cannot have it like this. You cannot
15:09:43have like two select statements in one
15:09:45query. you have to connect it somehow.
15:09:47That's why we can go and use the union
15:09:49all or union depend if you want to have
15:09:51duplicates or not. So now we call the
15:09:53first query the anchor query. The anchor
15:09:56query going to be the first query that
15:09:58interacts with the database and provide
15:10:00us the initial intermediate results. So
15:10:03it is the starting point of the
15:10:05iteration and we can say it is the first
15:10:08step in the process. So this going to be
15:10:10executed only once and it going to
15:10:12provide us the initial step the first
15:10:14step in the process. Now we call the
15:10:16second step as a recursive query and we
15:10:19call it like this because this query
15:10:21going to be executed multiple times and
15:10:24it will keep repeating and add data to
15:10:26the intermediate results until the
15:10:29condition is met or let's say there will
15:10:31be no more data that is available to be
15:10:34processed. So this is the syntax of the
15:10:36city query for the main query nothing is
15:10:38changed. So we have to go and use the
15:10:41city name in the main query. So this is
15:10:44the syntax of the recursive city. So
15:10:47think about it like this. SQL going to
15:10:48go and execute the anchor query only
15:10:51once and then after that going to go
15:10:53through the recursive query and keep
15:10:56looping and looping and iterating until
15:10:59a certain condition is met and then SQL
15:11:02going to go out from the CTE. So this is
15:11:04actually what we mean with the anchor
15:11:05and recursive queries. All right. Right.
15:11:07So now let's have a simple task in order
15:11:09to understand the recursive city. So the
15:11:11task says generate a sequence of numbers
15:11:14from 1 to 20. So now let's do it step by
15:11:17step. So that means we have to create a
15:11:19loop from 1 to 20 and after 20 the loop
15:11:22should stop. So let's go and do it. Now
15:11:25the first step of the recursive CTE is
15:11:27to build the anchor query. So the anchor
15:11:30query is responsible for the first
15:11:32iteration. So that means the first row
15:11:34of the output. So what is the first
15:11:36value between 1 and 20? It is the one.
15:11:39So let's go and write a query that
15:11:41generate the value one. So select and
15:11:45we're going to say one as I'm going to
15:11:47give it the name my number. So that's
15:11:51it. Let's go and execute it. Now you can
15:11:53see in the output we have the first
15:11:55member of our sequence. And this is
15:11:57exactly the task of the anchor query. It
15:12:00retrieves the first step in the
15:12:02iteration. So let's go and call
15:12:05it
15:12:07anchor query. Now the next step with
15:12:10that we have to go and build the
15:12:11iteration. So we need a CTE. So I will
15:12:14build now the city. So we're going to
15:12:16say with we're going to call it series
15:12:19and then we're going to put everything
15:12:21in parenthesis and then we're going to
15:12:23go to the main query. So this is the
15:12:25main query and we will go and select
15:12:28everything from the Sirius the city. So
15:12:32let's go and execute it just to make
15:12:34sure that everything is working fine. So
15:12:36we didn't create any loop or anything.
15:12:38We have just created a city on top on
15:12:41the anchor query and we just call it
15:12:43from the main query. So now we come to
15:12:45the second step of building the
15:12:47recursive city. We have to build the
15:12:49recursive query. So let's do it. I will
15:12:52just make this little bit smaller. And
15:12:53now before we start writing the query,
15:12:55we have to go and use union
15:12:58all in order to go and connect the
15:13:00anchor query with the recursive query.
15:13:03And let me say this is the
15:13:05recursive query. So how we going to
15:13:08build it? Let's go and start with the
15:13:10select. And now next what I usually do I
15:13:12just make sure that we are making a
15:13:14recursive city. So I go with selecting
15:13:16from and then we're going to use the
15:13:19name of the current city so that we are
15:13:21referencing the city to itself in order
15:13:23to make the city recursive and to do the
15:13:26looping. Now here comes the tricky part.
15:13:29So we need to create like the sequence.
15:13:32Now what is the current value? The
15:13:33current value is one. Right? Now what do
15:13:35we need? We need the second value in the
15:13:37sequence which is two. So we can do it
15:13:40by 1 + 1. So if you do it like this you
15:13:43will get the output two. But actually
15:13:45what we are doing here we are always
15:13:47taking the current value and we are
15:13:49saying plus one in order to generate the
15:13:52next value. So in order to do that
15:13:54instead of saying one we're going to
15:13:56take the my number the current value and
15:13:58we're going to add to it plus one in
15:14:01order to generate the second value in
15:14:03the sequence. So that means my number
15:14:05always holds the current value and we do
15:14:09the operation + one in order to generate
15:14:11the next sequence. So having it like
15:14:14this what we are doing we are generating
15:14:16the sequence of numbers. Now if you go
15:14:18and execute it like this let me just
15:14:20execute it what will happen it going to
15:14:23breaks because SQL will not allow it and
15:14:25SQL set it to 100 iterations. So more
15:14:28than 100 SQL going to break the query so
15:14:30that we don't have infinite number of
15:14:33looping. So this is bad because we
15:14:35didn't define the breaking mechanism of
15:14:37the looping. So now we have to define as
15:14:39well in the recursive query how the loop
15:14:42going to ends and we usually use a
15:14:44condition. For example, we can go and
15:14:45use the wear clause and we can say okay
15:14:48keep looping and keep generating but
15:14:50always check whether the value of the my
15:14:53number is less than 20. And you might
15:14:58ask okay it should be less or equal to
15:15:0020 right? Well no because if you are
15:15:03making less and equal to 20 what going
15:15:05to happen once the my number is equal to
15:15:0720 you are allowing one more iterations
15:15:10where you will get in the output 21. So
15:15:13that's why we are making it with 20. So
15:15:15now let's go and execute it and let's
15:15:18check the sequence. It start with 1 2 3
15:15:204 5 and until we reach the 20. So with
15:15:23that we have solved the task. Again here
15:15:25it's not that hard right? We are just
15:15:28providing the initial step and then we
15:15:30are providing the loop where we are
15:15:32defining inside it how the loop going to
15:15:34ends. Now there is one more thing that
15:15:36you can do with the recursive CTE is to
15:15:38define the limit of iterations. So for
15:15:41example in your code if you say okay if
15:15:43this iterates more than 10 times then
15:15:46the SQL should breaks and stops. So you
15:15:49can define for the SQL the maximum
15:15:51number of recursions. So how we can do
15:15:54that? We can do that in the main query.
15:15:56So if you go over here and say option
15:15:59then two parenthesis and then max
15:16:02recursion and after that you can define
15:16:05the limit. So for example let's go with
15:16:07the 10. Now of course we are iterating
15:16:09in our code now more than 20 but here we
15:16:12are making the rule it should not
15:16:13iterate more than 10. So let's go and
15:16:16execute it. So now we can see that our
15:16:17SQL breaks and it says the maximum
15:16:20recursion is 10. So as you can see now
15:16:22in the output we are getting the error
15:16:24of having more than 10 iterations which
15:16:26is not allowed. So with that you can
15:16:28control how many recursions you can
15:16:30have. Let's say that you would like to
15:16:31have like thousand iteration. So if you
15:16:34go over here and say you know what I
15:16:35would like to have a sequence of 1,000.
15:16:37If you let me just comment this out. So
15:16:40if you execute it you will get an error
15:16:42because the default is 100. But of
15:16:45course you can go and increase the
15:16:47maximum recursion. For example let's go
15:16:49with 5,000s. in the output it will work
15:16:52and you will get a sequence of 1,000. So
15:16:54with this you can control how many
15:16:56iterations are allowed in your query. So
15:16:58that you have like a control on it.
15:17:00Okay. So now we can understand step by
15:17:02step how SQL executed the recursive
15:17:04query. And here we have like flow
15:17:06diagram in order to understand the
15:17:08process the steps of executing the
15:17:10recursive query. So let's go and do it.
15:17:12Now in the start we have the first step
15:17:14is to run the anchor query. So our
15:17:17anchor query is just a select for the
15:17:19value one. So in the output we will get
15:17:22the value one in my number and as you
15:17:24can see the anchor query going to be
15:17:26executed only once. So there is no
15:17:29iterations or anything. SQL executed
15:17:31once and then goes to the next step. So
15:17:34what is the next step? It's going to
15:17:35execute the recursive query. So it's
15:17:38going to go over here and now what going
15:17:39to happen? We will get the current value
15:17:41of my number. The current value is one.
15:17:44and then we're going to add to it a one.
15:17:46So 1 + 1 we will get from the recursive
15:17:49query the two which is added to our
15:17:52results. Now it's going to check the
15:17:54condition is my number now smaller than
15:17:5820. Well yes it's smaller than 20 and
15:18:01what's going to happen since it's true
15:18:03is going to go and reexecute the
15:18:06recursive query. So now we are doing the
15:18:08second iteration. So again it's going to
15:18:10go to the recursive query and going to
15:18:12say okay what is the current value of my
15:18:14number? It is two. So 2 + 1 the second
15:18:18iteration will give us the value three.
15:18:21So as you can see each time the
15:18:22recursive query is executed it is adding
15:18:25more values to our result. So the same
15:18:28question can be asked is now my number
15:18:31smaller than 20. Well yes it is smaller.
15:18:34Well what can happen is still going to
15:18:36reexecute the recursive query. So SQL
15:18:39going to keep looping and iterating and
15:18:41adding values to the output until we
15:18:45reach the value 20. So now SQL going to
15:18:48ask is 20 my number now smaller than 20.
15:18:52Well no. So it is false and what's going
15:18:55to happen the chain will break and we
15:18:57will not loop anymore. So it's going to
15:19:00be the end of the city and this going to
15:19:02be the final results that's going to be
15:19:04used from the main query. So this is how
15:19:07SQL executed this recursive CD. Okay. So
15:19:10now let's have another task for the
15:19:11recursive CD. This time it's going to be
15:19:13a little bit more advanced. So the task
15:19:15says show the employee hierarchy by
15:19:18displaying each employees level within
15:19:20the organization. So that means we have
15:19:22to show for each employee for each row a
15:19:25level that tells us the hierarchy of the
15:19:27employee. So first let's go and explore
15:19:30the table employees. So let's go and
15:19:32select everything prompt sales
15:19:36employees. Okay, let's go execute it. So
15:19:39now by looking to the results we have
15:19:40like few informations about the
15:19:42employee. We have information about
15:19:43which department the gender salaries but
15:19:46here we have the key. It is the manager
15:19:48ID. So this is like self referencing to
15:19:51the same table. So for example the first
15:19:54employee the value is null. That means
15:19:56this employee has no manager which makes
15:19:59this employee like the big boss, the
15:20:01CEO. Then now by looking to the next two
15:20:04employees, they have a manager ID one.
15:20:06So who is the manager of those two? It's
15:20:09going to be the first row, the manager
15:20:11ID number one. So the manager ID number
15:20:13one is the post of those two employees.
15:20:16And then for the fourth one, we can see
15:20:18the manager ID number two. So the
15:20:20manager of Michael is actually Kevin,
15:20:23the second row. And for Carol the
15:20:25manager ID is three. That means Mary is
15:20:28the manager of Carol. And this is
15:20:30exactly what we can do with the
15:20:32recursive CTE. We can use such
15:20:34informations in order to create like a
15:20:36loop. So let's go and do it step by
15:20:38step. First we're going to start with
15:20:40the anchor query as usual. So this is
15:20:44the anchor query and here the first step
15:20:47or the first record going to be the
15:20:49highest manager which is the CEO, right?
15:20:52The first record. So in order to select
15:20:55now the only the first record what we
15:20:57can say we can say where manager id is
15:21:03null. So let's go and execute it. And
15:21:06with that we have now the first row and
15:21:08we can use this as the first step in our
15:21:10iteration. So now let's go and pick few
15:21:11informations in the select like the
15:21:13employee ID and the first name and as
15:21:17well let's go and get the manager ID.
15:21:19And now we have to start creating the
15:21:21levels. Right? So this is the first
15:21:24level. So I'm going to have the value
15:21:25one as let's have it like level. So our
15:21:29CEO has the level number one. So let's
15:21:32go and execute it. So now as you can see
15:21:34Frank is the CEO and he is in the level
15:21:38number one. So this is our anchor query.
15:21:40Now we have to do the iteration right.
15:21:43So we have to go and start creating the
15:21:44city. So let's call it with CD employee
15:21:50hierarchy and then as and then this is
15:21:54the definition of our CD. So let me just
15:21:57make it like this. And of course what do
15:21:59we need? We need the main query.
15:22:02So main query we will select
15:22:07everything from our new city like this.
15:22:11So let's go and test it. All right. So
15:22:14now we have prepared the CTE and the
15:22:16main query and of course the next step
15:22:17with that we're going to go and build
15:22:19the recursive query but first we need
15:22:21the union all in order to connect the
15:22:25two queries and recursive query and now
15:22:29we can start building the logic. So now
15:22:31we want to find all the employees where
15:22:33their manager is the employee ID number
15:22:36one right because they going to have the
15:22:38second level in the hierarchy. So what
15:22:40we're going to do, we're going to go and
15:22:41select and we need the same stuff. So we
15:22:44would like to get the employee ID, the
15:22:46first name, and the manager ID. And we
15:22:48need the level. So this going to be the
15:22:50level number two. It's not correct yet.
15:22:52I'm just want to show what this means
15:22:55because we need to get the employee ID
15:22:57and the first name and so on. We cannot
15:22:59get it yet from the CT because in the
15:23:01city we have only one employee. So we
15:23:03still have to go to the database and
15:23:05grab the next employees. So now I will
15:23:08give this as an alias like E and I will
15:23:11select it as well from those employees.
15:23:13So so far we are not doing any recursive
15:23:16yet right in the recursive query we're
15:23:18still querying the database but now we
15:23:21don't need all the employees from this
15:23:23table we need all the employees where
15:23:25the manager ID equal to one right now.
15:23:28Of course, in order to get those
15:23:29employees where the manager equal to
15:23:32one. So we can do it with the workclouds
15:23:34for example and say manager ID equal to
15:23:38one. Let me just select this and query
15:23:40it. Now we will get those two employees
15:23:43where their manager is the CEO the top
15:23:45manager. But of course we cannot do it
15:23:47like this. What we're going to do we're
15:23:49going to join this table with our
15:23:51current CTE in order to make a loop. So
15:23:54let me show you what I mean. We will
15:23:55remove this. We're going to use the
15:23:57inner join and we're going to reference
15:24:01it from the CTE and let's give this the
15:24:04name C H and we connect it like this. So
15:24:08on we're going to say the manager ID of
15:24:11the employee should be equal to the
15:24:14employee ID. So the employee ID at the
15:24:17start going to be the number one. So
15:24:19it's going to be like
15:24:20this employee ID. Now we are connecting
15:24:24the manager ID with the employee ID and
15:24:27we are as well reusing the CD inside
15:24:30itself in order to make the iterations
15:24:32and here we don't need the work clause
15:24:34because the inner join going to filter
15:24:36the data automatically as we learned the
15:24:38inner join going to show only the
15:24:40matching rows from the left and to right
15:24:42so that mean there will be filtering. So
15:24:44we are almost there but of course we
15:24:46don't want to show it as a two. What
15:24:48we're going to do, we're going to show
15:24:49it like this. Level + one. So the
15:24:52current level is one. The second
15:24:54iteration going to be two. And the third
15:24:56iteration going to be three. So I think
15:24:58we have everything for our iteration.
15:25:01Let me just check and make this smaller.
15:25:04Now again we have here our anchor query.
15:25:06This is only for the top level manager.
15:25:08And then here we are just connecting the
15:25:11managers with the employees. And we are
15:25:13reusing the CTE in order to make the
15:25:16effect of the loop. And as well we are
15:25:18using the inner join in order to break
15:25:20the loop once there are no more rows to
15:25:23process. So let's go and execute it. Now
15:25:26let's check the output. This is our top
15:25:28manager. So level one. This information
15:25:30comes from the anchor query. Then the
15:25:32second iteration it is the employees
15:25:35where the manager ID equal to one. So
15:25:38it's going to be those two employees. So
15:25:40those employees in our hierarchy are the
15:25:43second level in our organization. And
15:25:45then we're going to search for employees
15:25:47where their manager ID is equal to
15:25:50either two or three. And this is going
15:25:52to be those two employees, Carol and
15:25:54Miracle. And now to the third iteration,
15:25:57we're going to search for all employees
15:25:58where their manager ID equal to either
15:26:01two or three. And now to the third
15:26:03iteration, we're going to search for all
15:26:05employees where their manager ID equal
15:26:07to either two or three. And this going
15:26:10to result having those two employees
15:26:12because their manager ID is equal to
15:26:14three or two and they're going to get
15:26:16the level of three. And then after that
15:26:18SQL going to try to search for employees
15:26:20where their manager ID equal to five and
15:26:23four and SQL will not find anything and
15:26:25that's why it kind of breaks. So with
15:26:28that we have solved the task. All right.
15:26:29I totally understand if this is
15:26:31complicated but now we're going to do it
15:26:33step by step in order to understand how
15:26:35SQL executed this and why we have done
15:26:38it in this way. So again we have our
15:26:39flow diagram. We start by running the
15:26:42anchor query then the recursive query
15:26:45and then we have a check. If the check
15:26:47fails we iterate otherwise we end. So
15:26:50let's do it step by step. Here we have
15:26:51the table employees and beneath it we
15:26:54have the result of the city. So the
15:26:56first step it says we run the anchor
15:26:58query and we run it only once. So it's
15:27:00going to go to the anchor query and
15:27:02start executing it. So here we are
15:27:04selecting from the table employees but
15:27:06we are making a filter on the manager
15:27:08ID. So the manager ID should be null. So
15:27:11that means we will get the record of
15:27:14Frank and Frank going to be at the
15:27:16output and we are saying the level of
15:27:19this employee is one. So we will have
15:27:21here at the level one. So this is the
15:27:24output of the anchor query and that's
15:27:26it. This will never be executed. Now we
15:27:28go to the next step. Now we will run the
15:27:31recursive query. So what's going to
15:27:33happen in the recursive query we are
15:27:35saying okay I would like to select as
15:27:36well data from the employees and join it
15:27:39with the city results but the join
15:27:41should be an inner join so only the
15:27:43matching data between the CTE and the
15:27:45employees and now comes the join
15:27:48condition and this is the key for this
15:27:50iteration we are saying the manager ID
15:27:52of the employee should be matching to
15:27:55the employee ID from the CTE. So SQL
15:27:58going to go and join the table with the
15:28:00CTE. So now we have here only employee
15:28:02number ID one. So it's still going to do
15:28:04it step by step searching for any
15:28:06matches. So for the first one we don't
15:28:09have a match because the manager ID is
15:28:11not equal to one. So that's why it will
15:28:13not be included in the result. The
15:28:16second row here the manager ID is equal
15:28:18to one and this is a match with the
15:28:20employee ID. So SQL going to take it and
15:28:23put it at the output. Not only that, SQL
15:28:26going to increase the level. So we have
15:28:28here the current value is one. So level
15:28:31+ one. What can happen? We will get the
15:28:33value two. We are still in the same
15:28:35iteration. We are not iterating yet. So
15:28:37this is the first iteration of the
15:28:40recursive query. So until the whole join
15:28:43is done to the next row, we have a match
15:28:46as well because the manager ID is equal
15:28:48to one. And we're going to have the same
15:28:51thing. The level going to be as well too
15:28:53because the value of the level didn't
15:28:54change. It's still the current value is
15:28:56equal to one. And this going to keep
15:28:58going. So two, three, we don't have any
15:29:00matches. And with that, SQL is done
15:29:03executing the recursive query. All
15:29:05right. So now the SQL going to say,
15:29:07okay, did we process everything? Well,
15:29:10no. We still have missing output. We
15:29:12still have missing employees. That's why
15:29:14we didn't fulfill the condition. And
15:29:16we're going to run this again. So now in
15:29:18the second iteration, it's going to join
15:29:20as well again the city result with the
15:29:23employees by matching the manager ID and
15:29:26the employee ID. But this time it's
15:29:27going to focus only on those two ids. So
15:29:30the two and three. So SQL going to go
15:29:32and find any matching where the major ID
15:29:35equal to two or three. So it's going to
15:29:38do it step by step. The first one is
15:29:40not. The second one is as well not. The
15:29:43third one is not because the manager ID
15:29:45is one. But now to the employee number
15:29:49four we have a match. So it's still
15:29:51going to take this one and put it in the
15:29:54output like this. And now in this
15:29:56iteration what is the current level? It
15:29:58is two but we add to it one that's why
15:30:01we will get in the output three. And
15:30:04then SQL keep going. So we have here the
15:30:06employee number five and the manager ID
15:30:09is equal to three. So what happens? SQL
15:30:11takes it as well and put it in the
15:30:14output as the result of the CTE and as
15:30:17well the current level is two + one.
15:30:20We're going to have as well three. So
15:30:22with that SQL done joining the tables
15:30:25and going to ask again did we process
15:30:27all employees? Well yes it's true that
15:30:30means we don't have to do any more
15:30:32iterations because if you do any
15:30:34iterations SQL will not find anything.
15:30:37So for example if you go over here let
15:30:39me just remove this and let's say we are
15:30:41joining with the four and five. So what
15:30:43can happen isql going to search in the
15:30:45manager's ID for four and five and it
15:30:48will not find anything. So that means we
15:30:50will not be adding anything to the CTE.
15:30:52That's why SQL stops. So we have a
15:30:54complete results and we have now all the
15:30:56data from the employees in the output
15:30:58and this results going to be passed to
15:31:00the main query. So this is why we have
15:31:02done it like this and this is how
15:31:04executed this recursive query. I would
15:31:07like to visual for you what this means
15:31:09the level or the structure of the
15:31:11organization. So the hierarchy looks
15:31:13like this. The level one the top manager
15:31:15is Frank. So this is the level number
15:31:19one. And then we go to the level number
15:31:21two. So we have those two employees. So
15:31:24we have Kevin. So this is the level
15:31:26number one. And then we have two
15:31:28employees Kevin and Mary at the level
15:31:31two. So they work together and their
15:31:35boss is Frank. So it's going to look
15:31:38like this. And they are at the level
15:31:41two. We have then Michael that directly
15:31:44reports to who? To Kevin because here we
15:31:47have the employee ID two and two. So we
15:31:50have one employee here and as well Carol
15:31:52is as well at the level three and she
15:31:55reports to Mary and both Michael and
15:31:58Carol are at the level three. So this is
15:32:01what we mean with the level. It can help
15:32:03us to identify which employee at which
15:32:05level in the organization. If you have
15:32:08like hierarchy in your data and you can
15:32:10see in one table things are referencing
15:32:13each others like here the manager ID is
15:32:16actually the employee ID. So it's like
15:32:18we are referencing those ID to each
15:32:20others. This means there is hierarchy
15:32:23and there is a structure in this table
15:32:25and you can use the recursive city in
15:32:27order to build those levels and to
15:32:30navigate as well through the hierarchy.
15:32:33All right. So that's all for the
15:32:34recursive city and with that we have
15:32:35covered all the different types of
15:32:38cities that we have in
15:32:42SQL. So now let's have a quick recap. So
15:32:45we have learned that the CTE the common
15:32:47table expression is a temporary named
15:32:50result like a virtual table that could
15:32:52be used from different places in the
15:32:54query and we have a lot of advantages
15:32:56for the CTE. The main one is it breaks
15:32:59the complexity of query into small
15:33:01multiple pieces which makes our query
15:33:04much easier to read and as well to
15:33:06understand. So it improves readability.
15:33:08Another advantage of the city is that
15:33:10those small multiple pieces they are
15:33:13really easy to manage and to develop. So
15:33:15those pieces are like self-contained
15:33:17which makes our queries more modular. So
15:33:20it introduces modularity inside our
15:33:23queries. And we also learned that the
15:33:24CTE help us to reduce the redundancy
15:33:27inside our queries where it makes the
15:33:30result of one query usable in multiple
15:33:33places inside our query. So it makes our
15:33:36code smaller and reduce redundancy. And
15:33:38one more advantage of the city is that
15:33:40it help us to do looping and iterating
15:33:43in SQL by using the recursive CTE. And
15:33:46we have understood as well that we can
15:33:48treat the CTE result as any other
15:33:51physical table inside our database. So
15:33:53we can treat it and handle it like any
15:33:55other tables. Only one exception that
15:33:57this table lives only in one query. So
15:34:00we cannot query the CTE from an external
15:34:03query. Now we have learned that the
15:34:05result of the CTE could be used from the
15:34:07main query. This is the classical one.
15:34:10But not only we can use it in the main
15:34:12query but also we can use it in another
15:34:15CTE query which leads to having nested
15:34:18cities. And of course we have learned as
15:34:20well we can use the result of the CTE
15:34:23within itself which makes the CTE
15:34:25recursive and allows for looping and
15:34:27iterating. And I can only keep
15:34:29recommending to not use more than five
15:34:32CTEs in one query. Otherwise you're
15:34:34going to get the exact opposite and
15:34:36benefits from cdes where your code going
15:34:38to be really hard to understand and to
15:34:40read and even to extend. Okay my friends
15:34:42with that we have covered this amazing
15:34:44and very important technique in SQL the
15:34:46common table expressions the city. Now
15:34:49in the next step we're going to talk
15:34:50about a new type of objects that you can
15:34:52use in databases. We don't have only
15:34:54tables we have as well views. And views
15:34:57are amazing in order to give you dynamic
15:34:59and flexibility in your project. So
15:35:01let's talk about
15:35:06views. Now a view is not like a query
15:35:09that we can use in SQL. It is an object
15:35:11that we can find in the database. So
15:35:14before we jump immediately to the view,
15:35:15I would like to give you the big
15:35:17picture, the whole structure of the
15:35:19database. So let's go. We have like
15:35:22hierarchy structure and the highest
15:35:24level of this hierarchy is the SQL
15:35:26server. The SQL server manages multiple
15:35:29databases. It's like the control center
15:35:31that keep everything running and
15:35:33accessible. Now inside the SQL server,
15:35:36we have multiple databases. So a
15:35:38database is collection of informations
15:35:40that are stored in structured way. It's
15:35:43where all your data is kept and
15:35:45organized in different tables and
15:35:47objects. And each database is separated
15:35:49from others and it has its own data. Now
15:35:52inside each database we can find
15:35:54multiple schemas. A schema is like a
15:35:57logical way on how you group up related
15:35:59objects like tables and views together
15:36:02within a database. Like for example, if
15:36:04you have a database called sales, we can
15:36:06group up different tables about the
15:36:08orders underneath the schema orders. And
15:36:11maybe we have like multiple views and
15:36:13tables about the customers where we can
15:36:15put it in the schema customers. So if
15:36:17you find like multiple tables and views
15:36:19that are describing the same object, the
15:36:22same topic, we put them all together
15:36:24underneath one schema. So again, a
15:36:26database could be like the sales
15:36:28database and the HR database. They are
15:36:30completely different types of data. And
15:36:33underneath the sales, we can have like
15:36:35different sections. We have the sections
15:36:36about the orders and sections about the
15:36:38customers. And now moving on, what we
15:36:40can find inside the schema, we can find
15:36:42tables. A table is where actually your
15:36:45data is stored. It contains multiple
15:36:47columns and rows. So it is where the
15:36:49data physically lives. And now inside
15:36:51the schemas, we have another type of
15:36:53object. We call it view. And of course
15:36:56in this section, we are focusing on the
15:36:57views. So a view is like a virtual table
15:37:00that has a structure and everything but
15:37:03inside it we don't have any data. So the
15:37:06view does not store any data and in
15:37:08order to see the data we have to execute
15:37:10the query behind the view and only after
15:37:13that we're going to see some data but it
15:37:15is not like the tables it doesn't store
15:37:16the data permanently. Now inside the
15:37:18tables we can define multiple stuff like
15:37:21columns and as well keys and the same
15:37:23thing for the views. Inside the views we
15:37:25can define multiple columns and one last
15:37:28level for each column we have like a
15:37:30name and a data type. So as you can see
15:37:32the databases are really organized and
15:37:34we have like hierarchy where the top
15:37:36node is the SQL server and the lowest
15:37:39node is the columns and rows. So this is
15:37:42what we call the database structure. Now
15:37:44in order for you to build and manage
15:37:46this structure we have set of commands
15:37:49we call it DDL the shortcut of data
15:37:52definition language. So the detail is a
15:37:55set of commands that allow us to define
15:37:58and manage the structure of the
15:38:00database. So we have commands like
15:38:02create where it help us to create
15:38:05databases, schemas, tables, views.
15:38:07Another command called alter. Of course
15:38:09after you create something you would
15:38:11like maybe later to do changes and
15:38:13updates and of course we have the drop
15:38:14in order to remove any database object
15:38:17like dropping a schema, dropping a
15:38:19database, tables, views. So as you can
15:38:21see the DDL commands can help us to
15:38:23manage the database structure. So from
15:38:25this picture we have understood that we
15:38:27can create views inside schemas in the
15:38:30database. So now if you check the client
15:38:32and the object explorer you can find the
15:38:35exact hierarchy. So it start with the
15:38:37SQL server. This is our local server
15:38:39that's run at our machine and then we
15:38:41can find inside it multiple databases
15:38:44and one of them is our sales DB that you
15:38:46have installed together with other
15:38:47database like the adventure works. So
15:38:49now if you go to the sales DB over here
15:38:52you can go and drill to the next level
15:38:54and now we can find here a lot of
15:38:55objects and one of them that you know we
15:38:57have tables and views and now you might
15:38:59say okay but between the database and
15:39:01tables we have schemas so where are the
15:39:04schemas well actually if you go inside
15:39:06the tables you're going to find our
15:39:08tables customers employees and so on but
15:39:10before it we have a name called sales
15:39:13doc customers and you can find it
15:39:15everywhere sales doc customers sales do
15:39:17employees and so on the sales is the
15:39:20schema that bring all those tables
15:39:22together underneath one logical schema.
15:39:25So we have a database called sales DB.
15:39:28We have a schema called sales and we
15:39:30have a table called customers. And now
15:39:32if you would like to see all the schemas
15:39:34inside this database, what you can do?
15:39:36You can go to the securities over here
15:39:37and then here we have like a folder
15:39:39called schemas. If you go over there,
15:39:41you will find the list of all schemas
15:39:43that we have in this database. You might
15:39:45say, but we didn't create all those
15:39:46stuff. If we have only the sales that we
15:39:48know. Well, as you create a database in
15:39:51SQL server, you will get a lot of other
15:39:53system default schemas that the server
15:39:56can create. One of them is the
15:39:58information schema where it holds a lot
15:40:00of views about the catalog and the
15:40:02metadata where you can find the list of
15:40:04columns, tables, views and so on. So
15:40:06here we have only one schema that we
15:40:08have created for the user. It is the
15:40:09sales. So let's go back. Now if you go
15:40:12inside one of those tables you will find
15:40:14here multiple stuff like we have
15:40:16columns, keys, constraints and so on.
15:40:19And if you go to the columns you will
15:40:21end up at the lowest level of the
15:40:23hierarchy. And here we have the columns
15:40:25like the customer ID and we have some
15:40:27extra informations like the data type
15:40:29length and so on. So this is the
15:40:31structure and the hierarchy of
15:40:32databases.
15:40:37Now I would like you to understand a
15:40:39fundamental concept on the database in
15:40:41order to understand the views the
15:40:43three-level architecture of the
15:40:44database. This architecture can describe
15:40:47the different levels of data
15:40:49abstractions in a database. So let's see
15:40:52what this means. So the architecture is
15:40:54divided into three levels. The first
15:40:56level is the physical level. Then we
15:40:58have the logical level and the third one
15:41:00is the view level. Now let's understand
15:41:03each level what it means. So now the
15:41:05physical level it is the lowest level of
15:41:07the database where the actual data is
15:41:10stored in a physical storage and usually
15:41:12who has access to this layer are the
15:41:15database administrators because they are
15:41:17the experts and they have to manage the
15:41:19access and the security of this layer
15:41:21because they are the expert that have to
15:41:24manage a lot of stuff like optimizing
15:41:25the performance making sure that
15:41:27everything is secure and managing the
15:41:30backup and recovery and to do all the
15:41:32configurations and many other tasks. So
15:41:34at the physical layer we have to deal
15:41:36with a lot of stuff like the data files,
15:41:39partitions, logs, cataloges, blocks and
15:41:42caches and many other stuff that each
15:41:44database needs in order to store your
15:41:46data. So as you can see this layer is
15:41:49very complicated and you need to be
15:41:51really an expert of databases in order
15:41:54to be able to manage all those stuff. So
15:41:56we call this layer a physical layer or
15:41:59sometimes we call it an internal layer.
15:42:01So now let's move to the next level. we
15:42:03have the logical level. So the logical
15:42:05layer it is less complicated than the
15:42:07physical layer. Here at this level you
15:42:09have to deal on how to organize your
15:42:11data and normally we have here like an
15:42:14application developer or we have like
15:42:16data engineers that access the logical
15:42:19level in order to define the structure
15:42:21of your data. So those developers can
15:42:23focus on how to structure your data
15:42:25rather than how the data is exactly
15:42:28storing the data physically at the
15:42:29storage. So they don't have to deal with
15:42:31all those details. they leave it for the
15:42:33database administrator and they can
15:42:35focus only on how to structure the data.
15:42:38That's why we need for this kind of role
15:42:41an abstraction level for them which is
15:42:44the logical level. So now what actually
15:42:46the developers are doing at this level?
15:42:48Well, they are like creating tables and
15:42:51defining the relationships between those
15:42:53tables or they can go and define views.
15:42:56they can create indexes on the tables in
15:42:58order to optimize the performance of the
15:43:00tables or maybe they are creating stored
15:43:03procedures and functions and some other
15:43:05codes in order to manage those tables.
15:43:08So as you can see they are building the
15:43:09data model they are structuring your
15:43:11data but they don't care at all where
15:43:14are those data stored physically in the
15:43:17database. So as you can see here things
15:43:19are less complicated than the physical
15:43:21layer and it is perfect abstraction for
15:43:23developers to build projects. So we call
15:43:25this the logical layer or sometimes we
15:43:28call it the conceptual layer. Okay. So
15:43:30now moving on to another level of
15:43:32abstraction. We have the view level. So
15:43:35the view level is the highest level of
15:43:38abstraction in the database and it is
15:43:40what the end users and applications can
15:43:43access and can see. So for example, you
15:43:46could have like one view for business
15:43:48analyst. So you prepare and customize a
15:43:51views that are suitable only for the
15:43:54business analyst and you might say you
15:43:55know what let's prepare another set of
15:43:58views that are suitable for data
15:44:00visualizations and reporting like you
15:44:03can go and connect for example a PowerBI
15:44:05in order to create dashboards. So they
15:44:08are fully customized and prepared views
15:44:10in order to be connected with the
15:44:13PowerBI reports and you can keep doing
15:44:15that by creating multiple set of views
15:44:18that are suitable for specific purpose
15:44:21and use case. So as you can see at this
15:44:23level we are exposing our data for
15:44:26multiple users and multiple
15:44:27applications. So now the question is
15:44:29what do we have to deal at the view
15:44:31level? Well, you have their only views
15:44:33that holds only the relevant
15:44:35informations for the use case or users.
15:44:38So the users at this level have only
15:44:40views. They don't have to deal with the
15:44:42tables, indexes, store procedures, any
15:44:45files, logs, partitions or anything.
15:44:48This is the highest level of abstraction
15:44:51because the focus of this layer is to
15:44:53make it friendly for the end users and
15:44:55easy to consume. So we call this layer
15:44:57the view layer or sometimes we call it
15:45:00an external layer. So this is the
15:45:02three-level architecture of the
15:45:04databases or we call it the three
15:45:06abstraction levels of the database. So
15:45:08the physical layer has the highest
15:45:10complexity, the lowest abstraction and
15:45:13the view layer has the highest
15:45:15abstraction. So this is one more reason
15:45:17why the views are very important concept
15:45:19in SQL
15:45:21[Music]
15:45:24databases. Okay. So with that we have
15:45:26enough fundamentals in order to start
15:45:28talking about the views. So the question
15:45:29is what are views? A view is a virtual
15:45:33table in SQL that is based on the result
15:45:36of a query without actually storing the
15:45:39data in the database. So in short this
15:45:41means views are stored or persisted SQL
15:45:46query in the database. So let's
15:45:48understand what this exactly means. Now
15:45:50so far what you have learned we have
15:45:51like database table and all what you
15:45:53have done we create a select query in
15:45:55order to retrieve the data from this
15:45:57table. So once we execute our query we
15:45:59will get the result back. Now if you are
15:46:01talking about views they have as well
15:46:04like the structure of the table but
15:46:06without any data inside it. And for each
15:46:08view there is like a query attached to
15:46:11it. So there is no data but we have like
15:46:13a query in order to get data. We call
15:46:16the normal table as a physical table and
15:46:18the view we call it a virtual table. So
15:46:21now how exactly we're going to get the
15:46:22data. So now if you go and write query
15:46:25by selecting data from the view not from
15:46:27the table from the view what going to
15:46:29happen SQL going to go and trigger the
15:46:32queue that is attached to the view and
15:46:34this query is responsible to query the
15:46:37physical table and then the result going
15:46:40to fill the structure of the view and we
15:46:42will get back of course the results. So
15:46:44we are directly querying a view but
15:46:46actually we are indirectly querying a
15:46:49physical table. So the view is like
15:46:51between us and the data. So that means
15:46:53my real data is stored inside the
15:46:55database tables and the views are like
15:46:58an abstraction layer between me and my
15:47:00real data. And of course the data will
15:47:03not be stored inside the view. Each time
15:47:05I'm querying the view what's going to
15:47:07happen the SQL query behind the view
15:47:09going to be executed again. So it's
15:47:11going to go and retrieve the data and
15:47:13get it back to the view and then I will
15:47:14see it in the output. So this is what we
15:47:17mean with SQL
15:47:21view. So now let's have a quick
15:47:23comparison between tables and views.
15:47:25Tables stores the actual data physically
15:47:28at a database. So the tables where the
15:47:30data is persisted with in the other hand
15:47:32the views they are virtual tables and
15:47:35they do not store any data inside the
15:47:37database but they present the data from
15:47:39the underlying tables. So that means
15:47:41views don't persist any data physically.
15:47:44Now the tables are hard to maintain and
15:47:46as well hard to change. So it needs a
15:47:49lot of efforts in order to do any change
15:47:51like adding columns and moving columns
15:47:53always requires a lot of efforts for the
15:47:55migration especially if you have large
15:47:57tables. But in the other hand the views
15:48:00are way easier to maintain and very
15:48:02flexible to change. All what you have to
15:48:04do is only to change the query of the
15:48:06view. So that means you can very quickly
15:48:08change stuff in the views compared to
15:48:10the tables. But if you are talking about
15:48:12performance, tables are faster than
15:48:14views. For example, if you go and do a
15:48:16simple select on the table, you will get
15:48:18the data back as soon as the database
15:48:21fetches the data. But if you are
15:48:22selecting something from the view, it is
15:48:24actually two queries. The query that
15:48:27comes from the user and as well the
15:48:29second query is the view query. and the
15:48:31query of the view could be very
15:48:33complicated in order to extract the data
15:48:36from the underlying table. So selecting
15:48:38something from the view is always slower
15:48:40than selecting something from a table.
15:48:42Now if you have a table you can read
15:48:44from the table and as well you can write
15:48:46to a table but the views are read only
15:48:49as the name says it is only a view. You
15:48:51cannot go and write something to the
15:48:52database using the view. Okay. So those
15:48:55are the big differences between views
15:48:57and tables.
15:49:02All right. So with that we have a clear
15:49:03understanding what are views. But now
15:49:05you might ask me why do we need views?
15:49:08That's why now what we're going to do
15:49:09we're going to deep dive into multiple
15:49:11scenarios and use cases that you might
15:49:13encounter in your SQL projects. So let's
15:49:16start with the first use case. The first
15:49:18use case and the core reason why we use
15:49:20views in our data projects is to store
15:49:23central logic from a complex query in
15:49:26the database so that everyone can access
15:49:28it and with that we improve reusability
15:49:31between multiple queries and we reduce
15:49:33as well the complexity of the overall
15:49:36projects. So let's understand what this
15:49:37means. So now in our project we have
15:49:39like two tables in the database orders
15:49:41and customers and we have learned
15:49:43previously that if we have like a
15:49:45complex query we can go and use the
15:49:47city. So for example in our city we are
15:49:49joining tables and doing some
15:49:51aggregations using the sum and the city
15:49:53going to store the data in an
15:49:54intermediate results and then we have
15:49:56the main query. For example we are doing
15:49:58the step two where we are ranking the
15:50:01data. So the whole thing is in one query
15:50:03and let's say that a financial analyst
15:50:05was doing this type of analyszis. Now
15:50:08what could happen is that you might have
15:50:10another user for example a budget
15:50:12analyst where he is doing exactly the
15:50:14same first step. So he has as well a
15:50:17city query where first the data are
15:50:19joined and then aggregated using the
15:50:21sum. But the last step in the main query
15:50:23he's not doing ranking he's just doing
15:50:25like max and min. And not only that, we
15:50:28have a third user, the risk analyst,
15:50:30where as well doing the same initial
15:50:33step using the CTE, joining the tables
15:50:35and doing the summarization. But here
15:50:37the risk analyst in this scenario, he's
15:50:40just comparing the data at the last step
15:50:42in the main query. So now if you sit
15:50:43back and look to this, you can see all
15:50:45those three data workers, all of them
15:50:47are doing the same first step. So all of
15:50:49them are doing the same CTE. They are
15:50:51joining the data and then doing
15:50:53summarization. And of course this is a
15:50:55complete waste of time that each one of
15:50:57them has to create first the city from
15:50:59the scratch in order to do some
15:51:01analyszis. So it is complete redundancy
15:51:04and makes no sense. So this is exactly
15:51:06the disadvantage of only using cities in
15:51:09the projects. Now what we can do instead
15:51:11of that those three data workers going
15:51:13to decide to say you know what let's put
15:51:15the first step as view in the database.
15:51:18So instead of using CTE each time we're
15:51:21going to take this script and put it in
15:51:23the database. So we have now a central
15:51:26logic that is stored in the database
15:51:28where everyone can use it. So we have
15:51:30this query this logic only once and
15:51:33everyone can benefit from it. So now the
15:51:36financial analyst instead of going
15:51:38directly to the physical tables they can
15:51:40go to the view. So thus means she needs
15:51:43only to write one script the rank
15:51:45script. Same thing goes for the budget
15:51:47analyst. he has only to write the query
15:51:49for the max and min and as well for the
15:51:51risk analyst he just need to compare the
15:51:53data. So as you can see all those
15:51:55queries are reduced and they can only
15:51:57focus on the analyzes. So this is
15:51:59exactly the magic of views in data
15:52:02analytics. This logic this knowledge can
15:52:04be centralized in the database and this
15:52:07is way faster and better than having
15:52:09this logic written each time someone
15:52:11want to do any analyzes. So this is why
15:52:14we need views in data projects.
15:52:20So now if you compare views with CTE,
15:52:22the CTE are used in order to reduce the
15:52:25redundancy within one single query. So
15:52:28it improves the reusability within one
15:52:31query. Where in the other hand in the
15:52:33views we are reducing the redundancies
15:52:36from multiple queries. So we are
15:52:38reducing the complexity of the whole
15:52:40project. So the views are improving the
15:52:43reusability in multiple queries. Now
15:52:46think about it like this. We use views
15:52:48in order to persist a logic in the
15:52:51database. So the logic is so important
15:52:54that we want to persist it in the
15:52:55database. It's like in the tables we
15:52:57persist data but with the views we are
15:52:59persisting logic. But in the other hand
15:53:02in the CTE the logic is not persisted.
15:53:05It is temporary and going to be
15:53:06calculated only on the fly within the
15:53:09scope of one query. So this logic is
15:53:12important only in this scenario and it
15:53:14is not important for any other queries.
15:53:17That's why it makes no sense to persist
15:53:19it using the views. So you have to
15:53:21decide is this logic is very important
15:53:24then take it away from the city and put
15:53:26it in the view. But if you think you
15:53:28know what this logic is not really
15:53:30important and only important in this one
15:53:32query then stay with the city because
15:53:34creating views always needs some extra
15:53:37steps in order to maintain the view. You
15:53:40have to create the view. You have to
15:53:41drop the view if you don't need it. But
15:53:43the CTE, there is almost no maintenance
15:53:45for it. The database going to do
15:53:47automatically the cleanup once the query
15:53:49is done. So there is no extra activity
15:53:52to drop a city or something. That's why
15:53:55CTE is easier to use than views. So
15:53:57those are the big difference between the
15:53:59views and
15:54:03cities. Okay. So now let's check quickly
15:54:05the syntax of a view. So now we have a
15:54:08query like select from where. So this is
15:54:10a query a simple select statement. But
15:54:13now in order to create a view an object
15:54:15in database we have to go and use a DDL
15:54:18command create. So we're going to say
15:54:20create view cuz we want to create a view
15:54:23then the name of the view and then it's
15:54:25like the CTE we say as and then double
15:54:28parenthesis. So as you can see it's very
15:54:30simple and we call this a DDL command
15:54:33where we are telling the database go and
15:54:35create a view and the logic of the view
15:54:37comes from this query. So it's very
15:54:39simple. This is how you can create views
15:54:41in database. Okay. So now let's have the
15:54:43following task and it says find the
15:54:45running total of sales for each month.
15:54:47I'm going to start this task by solving
15:54:49it using the CTE. So first I'm going to
15:54:51go and do few aggregations on the top of
15:54:53the month. So let's go and select. So
15:54:56now what do we need? We need the order
15:54:58dates but we need it as a month. I'm
15:55:00going to go and use the date truncate
15:55:03like this and say okay I would like to
15:55:05have the date as the granularity of
15:55:08month. So let's go and call it order
15:55:12month. And now after that we're going to
15:55:13do a few aggregations like for example
15:55:15let's go and get the sum of sales and
15:55:19we're going to call it total sales. And
15:55:21that's it for the start. So now let's go
15:55:23and call it from the table sales orders
15:55:26and group by and we are grouping up by
15:55:30by the month. So something like this.
15:55:32Let's go and execute it. And now for
15:55:34this we get for each month the total
15:55:36sales. And now the next step that we
15:55:37have to go and calculate the running
15:55:39total for the sales. This is of course
15:55:41not the running total. So that means
15:55:43either we can go and use subqueries. So
15:55:45this means this is our first step and we
15:55:47need a second step. So either use
15:55:48queries or cities. I will go with the
15:55:50city over here. So I'm going to say with
15:55:53city and
15:55:56monthly summary and we're going to
15:55:59define it like this. And now what we're
15:56:01going to do, we're going to go and
15:56:02define the main query. So the main query
15:56:04going to be simple. So select and let's
15:56:07go and get the order month. And now we
15:56:10have to build the running total. So
15:56:12we're going to go and use the window
15:56:13function. So sum total sales. And then
15:56:17we're going to say over we don't have to
15:56:19partition the data. We will just sort it
15:56:22by the order
15:56:24month and we can leave it ascending. So
15:56:27this is the running
15:56:30total and we have to go and select of
15:56:33course our CTE from here. So let's go
15:56:36and execute it and with that we are
15:56:38getting the running total. Of course we
15:56:39can go and add the total sales in the
15:56:41output in order to understand the
15:56:43results. So here in the output we are
15:56:44just building accumulative sales. So for
15:56:47this scope everything is fine. and we
15:56:49are using the CTE. But now imagine that
15:56:51this logic is important for multiple
15:56:54queries. So it's really nice to have
15:56:56such a report where we are aggregating
15:56:58the data at the level of the month and
15:57:00this could be used from different users
15:57:02and different queries. So now we say how
15:57:04about to put this logic in one view so
15:57:07that everyone can access it and we don't
15:57:09have to repeat the same aggregations
15:57:11over and over. And now before we put it
15:57:13in view, someone comes and say how about
15:57:15to add one more aggregation so that not
15:57:17only the total sales we can add. So now
15:57:19before we put it as view maybe some
15:57:21other user says you know what we would
15:57:23like to have one more aggregation not
15:57:24only the total sales let's make the
15:57:26scope a little bit bigger so that
15:57:28everyone can believe it. So for example
15:57:29we can go over here and say you know
15:57:31what let's go and add the total number
15:57:33of orders. So we can go over here and
15:57:35say counts and let's get the order ID
15:57:38and say this is the total orders or
15:57:41maybe some other says let's get the
15:57:43quantities as well. So we can go and
15:57:46summarize the quantity like this and we
15:57:50call it total
15:57:51quantities. So with that we are like
15:57:54doing a lot of aggregations on the month
15:57:56level. Let's go and execute only the
15:57:58CTE. So now we have really nice report
15:58:00that is based on the months and can be
15:58:03used from many different queries. So now
15:58:05what we're going to do, we're going to
15:58:06take this and put it in a view. Let's go
15:58:08and select only this logic and create a
15:58:11new query. And now what we're going to
15:58:13do, we're going to put our query here
15:58:14and we have to create now the DDL in
15:58:16order to create a view. So it's going to
15:58:19be like this. Create view. Let's give it
15:58:21the name maybe starts with the V
15:58:23underscore and this going to be the
15:58:25monthly summary. So this is the name of
15:58:28the view and as then we put everything
15:58:32in parenthesis. It's like you are
15:58:34building a CTE. So we have here our
15:58:36logic and here is our DDL query in order
15:58:40to create the view. So now let's go and
15:58:42execute it. Now as you can see in the
15:58:44output it says only that the command is
15:58:46completed because this is not a select
15:58:48query. This is a DDL command. So the SQL
15:58:51going to tell you okay either I created
15:58:53it successfully or not. So now the
15:58:55question is where do I find now my view?
15:58:58Well, if you go to the object explorer,
15:59:00you can see over here underneath our
15:59:02database sales DB, we have here
15:59:05something called tables where we are
15:59:06used to query those tables. But beneath
15:59:09it, we have as well our views. So if you
15:59:12check the views and expand it, now we
15:59:14are not seeing M view because we just
15:59:16created the view here. So go over here
15:59:18and refresh. And once you do that, you
15:59:21will see the newly created view. So this
15:59:24is the one that we just created. So now
15:59:26what we can do, we can go and create a
15:59:27new query and let's go and just query
15:59:30the view. So select star from so v month
15:59:35monthly summary. Let's go and execute
15:59:39it. And now as you can see we are
15:59:40getting now the result of the view and
15:59:42I'm accessing now this logic from
15:59:45completely external query. So now I can
15:59:47think about the view as any other table
15:59:49that we have in the database. And again
15:59:51the big differences between the views
15:59:53and the tables. The tables has data has
15:59:56actual data and everything there is
15:59:58persisted but the view is just an
16:00:00abstraction for me and behind it there
16:00:03is like a query that goes to the table
16:00:05and query the tables in order to present
16:00:08the results. But for me I don't care
16:00:09about all those details. I can go
16:00:11immediately to the query over here and
16:00:13start querying. So now in order to
16:00:15create the total running sales I don't
16:00:17have to create the CTE and sub queries.
16:00:20I just go and get for example our main
16:00:23query. Let's go back over here. So now
16:00:26instead of using the CTE I can go
16:00:28directly and access the view. So as you
16:00:32can see now my query is very simple. I'm
16:00:34doing immediately the step two without
16:00:37having to prepare the data first. So if
16:00:39I go and execute it I will get exact
16:00:41results. And now if you compare the
16:00:43query on top of the view like this with
16:00:45the city query you can see that the CTE
16:00:48has more steps and it is like little bit
16:00:51more complicated than the query on top
16:00:53of the view and this is exactly the
16:00:55benefit of the view. We reduce the
16:00:57complexity and it is very easy to
16:00:59consume from the point of view of users.
16:01:01So this is how you can put your logic in
16:01:04central place using views and with that
16:01:07we have learned how we create a view.
16:01:09Now one more thing about the schemas. If
16:01:11you check our tables over here, they
16:01:13have all one schema. So we have sales
16:01:15dot customers, sales do employees,
16:01:18orders and so on. Our new view has the
16:01:20schema of DBO. If you create any object
16:01:23whether it's table or view and you don't
16:01:26specify a schema in a default schema
16:01:28called DBO. And now let's go back to our
16:01:30DDL scripts. So as you can see over
16:01:32here, we didn't specify any schema. We
16:01:34just said okay, this is the view name.
16:01:37And now in order to put our view in the
16:01:39correct schema we don't want it to be in
16:01:41the defaults. You have to go and specify
16:01:43the schema name in the DDL. And now in
16:01:45order to do that we go to the name of
16:01:47the view and we write the schema name
16:01:50and then separated with a dot. So the
16:01:52first one is the schema name and the
16:01:54second one is the view name. So now
16:01:57let's go and execute it. Now if you
16:01:59check over here you don't see anything
16:02:00new. But if you refresh you will find
16:02:03another view in the correct schema. So
16:02:05we have sales dot vmon monthly summary
16:02:08and this is exactly what we want. So
16:02:10this is how you can assign a view or
16:02:12even a table to the correct schema if
16:02:15you don't want to use the default one
16:02:16the view. All right. So now the next
16:02:18step is that you say you know what I
16:02:19would like to clean up. I don't need
16:02:20those two views in my database. So how
16:02:23to delete a view? We can go and use the
16:02:25command drop. It is very simple. If you
16:02:27go and create a new query and you say
16:02:30drop and then you say what you want to
16:02:32drop. you want to drop a view and then
16:02:33you have to specify the name and schema
16:02:36of the view. But now since it is the
16:02:38default schema DBU, I don't have to
16:02:40write it down. So we can start
16:02:42immediately with the view name. So V
16:02:45monthly summary. So that's it. It's very
16:02:48simple. So now we go and execute it. It
16:02:51says it's completed but as you can see
16:02:53nothing has changed. We go and refresh.
16:02:55And now we can see that the database did
16:02:57remove the view with the schema DBU. So
16:03:00it's very simple. This is how you can
16:03:02drop a view in SQL. Okay. So now to the
16:03:05next step. Let's go back to our DDL of
16:03:07creating the view sales monthly summary.
16:03:09And now you say you know what I would
16:03:11like to change the logic inside the
16:03:13view. So how we can update this content?
16:03:16How I can update my query? If you say
16:03:18let's go and for example delete this
16:03:20column. I need only three columns. So
16:03:23and you go execute it. The database say
16:03:25I cannot do it for you because we have
16:03:27already such a view. So SQL will not go
16:03:30and replace stuff going to say no we
16:03:33have the same name and I cannot do
16:03:35anything for it. So how we can update
16:03:37the view? Well in other databases like
16:03:39ocris for example it's very simple. You
16:03:42can go over here and say create or
16:03:45replace view. So it's like you are
16:03:47telling the database create this view or
16:03:49if it already exists then replace it and
16:03:52you will not get error in the postcress.
16:03:54But in the SQL server it is little bit
16:03:57more complicated. we don't have this
16:03:58command. So here you have two ways.
16:04:00Either you go over here and say you know
16:04:02what let's first drop the view. So you
16:04:06go with the same name over
16:04:08here and then what you're going to do
16:04:10you're going to go and mark the drop
16:04:12view. So if you execute it like this the
16:04:14view going to be dropped and then we
16:04:16recreate the view like this. So what we
16:04:18have done we destroy the view and then
16:04:20we recreate it using the new logic. Or
16:04:22you say you know what I would like to
16:04:24have everything in one go like I don't
16:04:26want to do it in two steps. I would like
16:04:27to have everything in one command and
16:04:29for that you have to use in SQL server
16:04:31the TSQL the transacts SQL it is like an
16:04:35extension for SQL only in SQL server
16:04:37well it's like programming where you can
16:04:39go and add variables or you can all go
16:04:41and add checks we will not do a deep
16:04:43dive in this language but I would like
16:04:44to show you how to do it for the views
16:04:46so just follow me with that I'm going to
16:04:48go and replace the whole thing and then
16:04:51we're going to say if and now we are
16:04:53checking the system catalog if the
16:04:55object ID
16:04:57And now we go and specify the view name.
16:05:00So let's go and copy the whole thing
16:05:01with the schema as well. And then we're
16:05:03going to say for SQL this is a view. So
16:05:07if this object exists so we are saying
16:05:09is not null. So that means it exist in
16:05:12the catalog then what SQL should do?
16:05:15Should drop this view. So we're going to
16:05:17say drop view and it's like we have done
16:05:19it first and then semicolon and then we
16:05:23say for scale go and with that we are
16:05:25saying for SQL the tscale is done. So
16:05:28the logic is done and after that we have
16:05:31the DDL for our view. So again what we
16:05:33are doing we are checking before
16:05:36creating the view whether the view
16:05:38exist. If it exist then we are telling
16:05:40the scale go and drop it and if it
16:05:42doesn't exist that means we haven't
16:05:44created this view yet. it is completely
16:05:47brand new view then this step going to
16:05:49be skipped so that there is nothing to
16:05:51drop. So now if you go and execute the
16:05:53whole thing it will work and of course
16:05:54if you go and refresh over here you
16:05:56still see the view. So SQL did destroy
16:06:00the table first and then recreated. So
16:06:02if you execute it again. So this is how
16:06:04you replace your logic in view in SQL
16:06:07server. And with that we have learned
16:06:09all possible scenarios. How to create a
16:06:11view, how to drop a view and how to
16:06:14update the logic of a
16:06:19view. Now back to our database
16:06:21architecture and let's understand how
16:06:23the database executes views. So now
16:06:26let's say that the data engineer is
16:06:28creating view called top end. So the
16:06:30query going to be sent to the database
16:06:32engine and once the database engine
16:06:34understand this is a view this is not a
16:06:36table. So now the database engine going
16:06:38to go to the disk storage and to the
16:06:40catalog and it will stores not only the
16:06:43metadata about the view also the SQL
16:06:47that is responsible for the view. So
16:06:49it's going to take the SQL statements
16:06:50that you have defined in the create view
16:06:53and place it as well in the catalog. So
16:06:55if you compare to the tables we have in
16:06:57tables only metadata but in the views we
16:07:00have both the metadata and as well the
16:07:02query of the view and as well you can
16:07:04see that the database engine will not
16:07:07create a table in the user's data. So
16:07:10there is nowhere data stored inside the
16:07:12disk or the cache. So the actual data
16:07:15the physical data will not be stored
16:07:16anywhere. We are storing only metadata
16:07:19and the query inside the system catalog.
16:07:22So now we tell our data analyst okay we
16:07:24have a new view and the data analyst can
16:07:26go and write a query in order to
16:07:28retrieve the data from the view. So he
16:07:30going to say and say select from the
16:07:32view and execute it. The database engine
16:07:34going to take it and understand okay now
16:07:36we are talking about view. So the
16:07:37database first has to retrieve not the
16:07:40data going to retrieve the query from
16:07:43the catalog in order to understand what
16:07:45do we have now to execute. Then the
16:07:47database going to execute the query of
16:07:49the view first and the data for this
16:07:52query comes from a physical table called
16:07:54orders. So now the database engine is
16:07:56querying the order to retrieve the data
16:07:58so that we have a data for the end user
16:08:01and then it's going to be executed and
16:08:02the result going to be sent back to the
16:08:05data analyst. So as you can see there is
16:08:06like two queries. The SQL engine first
16:08:09has to execute the query from the view
16:08:12and only after that the database engine
16:08:15can execute the query that comes from
16:08:17the user. So actually the data comes
16:08:19always from a physical table but we are
16:08:21not providing the data analyst an access
16:08:23to the table. We are just providing an
16:08:26access to the view. So this can happen
16:08:28each time an end user selecting data
16:08:30from the view. Always the database
16:08:32engine going to grab the query from the
16:08:34catalog, execute it first in order to
16:08:36get the data and then execute what the
16:08:39end user wants. And now if the data
16:08:40engineer says no, let's go and drop the
16:08:42view. So she writes a query in order to
16:08:45drop the view. And the database engine
16:08:46going to go to the system catalog and
16:08:48delete both the metadata and the query.
16:08:51So as you can see, if you are dropping a
16:08:52view, you are not losing the actual
16:08:55data. So there will be no user data lost
16:08:57at all. So don't worry about it. What
16:09:00you are losing is only the query and the
16:09:02metadata about your view. It's only if
16:09:04you drop a physical table like the
16:09:07orders, you will lose your data. So
16:09:09dropping views is not that bad like
16:09:11dropping a database table. So this is
16:09:13how the database works with the views
16:09:15behind the scenes.
16:09:20Now moving on to the second scenario to
16:09:22the next use case of using views in
16:09:24projects is that we use views in order
16:09:26to hide complexity and to improve
16:09:29abstraction. In many scenarios we work
16:09:31with a very large and complex databases
16:09:34and we can use views in order to reduce
16:09:37the complexity and make things easier
16:09:39for the users. So let's understand what
16:09:41this means. Now I'm going to explain for
16:09:42you a scenario that happens almost in
16:09:45each project. Like if you get an access
16:09:46to a database where you want to do
16:09:48analyzes, you will be in scenario and
16:09:50this can happen a lot where you're going
16:09:52to find a large database where the
16:09:54tables are very complex to understand.
16:09:56They have a lot of columns. They have
16:09:58like technical and cryptical names and
16:10:00how tables are connected to each others
16:10:02and relationship between them. It's
16:10:04almost impossible to understand. then
16:10:06you have to be deeply involved with the
16:10:08data models with documentations and with
16:10:11experts until you understand how to
16:10:14query this database. So if you are not a
16:10:16developer and from end user perspective
16:10:19it can be nightmare where you are trying
16:10:21to do multiple joins in order to make
16:10:23simple analyzes and of course from the
16:10:25database perspective this data model is
16:10:28good enough for one application but if
16:10:30you are opening your database for
16:10:32multiple data analyszis projects this
16:10:34can be a nightmare because you have to
16:10:36go and explain for each user how to
16:10:38query the data. So what we usually do
16:10:41instead of giving a direct access to
16:10:43such technical and hard to understand
16:10:46data model we go as developers creating
16:10:49multiple views since we are the expert
16:10:51of the data model and these new views
16:10:53going to be an abstraction of the
16:10:56complexity that I have in my database
16:10:58and we have to make sure that those
16:11:00views are providing objects that are
16:11:03friendly. So they have like a full
16:11:05English name that makes sense and as
16:11:08well the columns are friendly and we try
16:11:10to not offer a lot of views so the user
16:11:12don't have to do all the joins. So we
16:11:14provide like few views that are friendly
16:11:18and has a lot of informations that the
16:11:20users needs for the analyzes. So with
16:11:22that the users can have an access to
16:11:24something more friendly and easy to
16:11:26consume and then they can write simple
16:11:28queries in order to do analyzes on top
16:11:30of these friendly views. And this is
16:11:33what we can give a name like we are
16:11:35providing a data product from my complex
16:11:39physical database. So here again how
16:11:41important are the views to provide an
16:11:44abstraction and easy to consume objects
16:11:46for the users and with that I can hide
16:11:49all my complexity and the script of the
16:11:52view going to be developed from the
16:11:54experts and only once so that the users
16:11:57don't have to understand or to write
16:11:59these complex SQL joins and with that
16:12:02you can make your data projects way
16:12:04easier than before. So this is another
16:12:06important use case for the views where
16:12:08we can use it in order to provide
16:12:10abstraction and as well easy and
16:12:12friendly objects for the end users.
16:12:14Okay. So now let's have the following
16:12:15task and it says provide view that
16:12:18combines details from orders, products,
16:12:22customers and employees. So now instead
16:12:24of having all those tables from our
16:12:26database, we have to provide one
16:12:29combined view that has everything well
16:12:32almost everything. So now let's see how
16:12:33we can create such a view. So let's
16:12:35start first by the table orders. I'm
16:12:37going to go and
16:12:39select first star from sales orders and
16:12:44let's go and execute it. This is the
16:12:46central table that connects everything.
16:12:47You can see here we have the order ID,
16:12:49product ID, sales, customers and so on.
16:12:52So it is a great start point. So now
16:12:54we're going to go and be picky about the
16:12:55columns. I would not show all the
16:12:57columns but I would say let's go and
16:12:59show for example the order ID. This is
16:13:01essential. It's nice to have a unique
16:13:04identifier. Now the product ID, I will
16:13:06not show it but I will just list it over
16:13:08here. The same for the customer ID,
16:13:11saleserson ID. Those stuff I would like
16:13:14to replace later. So I will just make it
16:13:17as comment so I don't forget about it
16:13:19because it makes no sense to show the
16:13:21product ID and customer ids and so on.
16:13:23We would like to show the details about
16:13:26each object because instead of having
16:13:28the product ID, I would like to show for
16:13:30example the product name itself and some
16:13:32other informations from the table
16:13:34products. And with that we are reducing
16:13:36the complexity. So now what else we can
16:13:38get from the table orders? We can go and
16:13:40get the order date. I will put it here.
16:13:42And maybe we can go and get stuff like
16:13:45sales and
16:13:46quantity. So like this. Of course, we
16:13:50can go and put all the columns. But for
16:13:52now, I will go with those informations.
16:13:54Now, it's important since we're going to
16:13:55have a lot of tables. Let's go and make
16:13:57sure we are using aliases. So, now we're
16:14:00going to have the O for each of those
16:14:03columns. All right. Fine. So, now we
16:14:05have four details from the table orders.
16:14:08Now, what is next? We have the product
16:14:10ID. So, let's go and get the
16:14:11informations from the products. What
16:14:14we're going to do, we're going to use a
16:14:15left join just to make sure to not miss
16:14:17any order. If you go with the inner
16:14:20join, you might miss some orders. So I
16:14:22will not do that. So let's join it with
16:14:24the products like this.
16:14:27And so now we have to go and join the
16:14:30tables. So we can use the keys product
16:14:34ID equal the order product ID. All
16:14:38right. So now the question is which
16:14:40informations we want to show for the
16:14:42users. Let's go to the table orders. So
16:14:45we have the product and category and the
16:14:47price. I would say let's go and get the
16:14:49product and category. That's enough. So
16:14:51now instead of the ID I'm going to have
16:14:54it like this. So it's going to be the
16:14:56product and the
16:14:58category. Now let's go and test it. I'm
16:15:00going to execute it. Now as you can see
16:15:02we don't have a product ID. We have the
16:15:04product name which is more friendly. So
16:15:06we have now those two columns from the
16:15:07orders and those two from the products
16:15:10and the last two as well from the
16:15:11orders. So it looks really nice and
16:15:13friendly and with that the user don't
16:15:15need extra table called products. We
16:15:18have everything in one. Now let's go and
16:15:19do the same for the customers. So let's
16:15:21go
16:15:22and do the same thing. So let's join
16:15:26sales customers see and as well join
16:15:30them using the key customer ID equal to
16:15:33the customer ID. Now we have to go and
16:15:36grab a few columns from the customers.
16:15:38Let's go and check. So we have a first
16:15:40name, last name and country and score. I
16:15:42would say I would go with the names and
16:15:44the countries but instead of having
16:15:46first name and last name I'm going to
16:15:48put everything in one. So we have to go
16:15:50and concatenate the informations. So
16:15:53we're going to get the first
16:15:54name then plus then empty between the
16:15:58first name and the last name and then
16:16:01the last name like
16:16:03this. Now we will not call it a name.
16:16:06We're going to go and call it the
16:16:07customer name because later we're going
16:16:09to have as well an employee name. All
16:16:11right. So next we want to get the
16:16:14country and we have to say this is the
16:16:16country from the customers. So we're
16:16:18going to call it customer country and
16:16:21that's it. Let's go and execute it. Now
16:16:23we can see we have again our orders
16:16:26products and now we have the
16:16:27informations from that customer. But
16:16:29here we have issue that we have some
16:16:31nulls and that's because there is no
16:16:32last name. So what we're going to do,
16:16:34we're going to go and handle the nulls
16:16:36for the last name and as well for the
16:16:38first name. So we're going to use the
16:16:40kowalis. If the last name is null then
16:16:44make an empty string and the same thing
16:16:47for the first name. So first name. All
16:16:50right. So now let's go and execute it.
16:16:52So with that we are getting as well the
16:16:55first name if the last name is missing
16:16:57or if the first name is missing we can
16:16:59get the last name. So looks good. So it
16:17:01looks good with that. We have the
16:17:02customer's details. The last thing we
16:17:04have to go and get the employees. So the
16:17:07employee here is called salesperson ID
16:17:09which we can connect it directly to the
16:17:11table employees. So if you go to the
16:17:13employees over here, which columns do we
16:17:15need? We have the first name, last name,
16:17:17department and so on. I would say let's
16:17:19go get the names and the departments. So
16:17:21first let's go and join it. So lift join
16:17:25sales
16:17:27employees and we're going to join it
16:17:29using the employee ID. and we're going
16:17:31to join it with the sales person ID that
16:17:36comes from the order table. So now
16:17:38instead of the person ID we're going to
16:17:40have as well the same thing. So I will
16:17:42just go and copy paste this. So instead
16:17:45of the alias we're going to have E and
16:17:47as well E over here and we're going to
16:17:49call it sales name and as well what we
16:17:52going to have we're going to have the
16:17:54department. So
16:17:55department and that's it. Let's go and
16:17:59execute it. So now we have a lot of
16:18:01informations in our view. So we have the
16:18:03first columns from the orders then from
16:18:06the products and here we have from
16:18:08customers and those two from the
16:18:10employees and the last two again from
16:18:12the orders. So that we have combined now
16:18:14all the relevant informations from
16:18:16multiple tables in our database in only
16:18:19one view. This result is relative big
16:18:21but still we have all the informations
16:18:23in one and it is more friendly for the
16:18:26users in order to consume our data
16:18:28instead of going and joining like all
16:18:30those four tables together. So now the
16:18:32next step we're going to put the result
16:18:34of this query in view in our database so
16:18:37that our end users can start consuming
16:18:39it. So how we going to do it? This is
16:18:41our combined query and now we're going
16:18:45to write the DDL for it. So create view
16:18:47and now we're going to give it the name
16:18:49order details and then as and we're
16:18:53going to put the whole thing in two
16:18:54parenthesis. So at the start and at the
16:18:57end and of course don't forget the
16:18:59schema. So our schema is sales sales dot
16:19:04then we have the view name just in order
16:19:06to have it in the correct schema and not
16:19:08in dbo. So everything is ready. Let's go
16:19:11ahead and execute it. So now let's go
16:19:13and check our database. So if you go and
16:19:15refresh, you will find our second view
16:19:18order details. So now let's go and test
16:19:20it. We're going to say select star
16:19:24from sales v order details. Let's go and
16:19:30execute it. And with that we are getting
16:19:32now a combined view that are showing all
16:19:35important informations from the
16:19:36database. So this is what the users can
16:19:38see. And with that the users don't care
16:19:41about how many tables do we have in the
16:19:43tables and how to join all those tables.
16:19:46We have only one view and we can start
16:19:48working on it. This is a very common use
16:19:50case for the
16:19:54views. Okay. Moving on to the next
16:19:56scenario to the next use case. We use
16:19:58SQL views in order to implement security
16:20:01and to protect our data in the database.
16:20:03In many scenarios, we have sensitive
16:20:05informations in our data and we cannot
16:20:07go and share it with everyone. So one of
16:20:09the best practices is to create views in
16:20:12order to protect your data before
16:20:14sharing it with the users. So let's
16:20:16understand what this means. So now let's
16:20:18understand first the scenario without
16:20:20views only tables. So now let's say that
16:20:22you have the table orders four columns
16:20:24and three rows and then you have like
16:20:26for example a manager that has an access
16:20:28directly to the database and start
16:20:30writing some queries in order to
16:20:32retrieve data. But in your project you
16:20:34have multiple people that has an access
16:20:36to your database like for example a data
16:20:38analyst and as well she is writing a
16:20:40script in order to retrieve data from
16:20:42the orders and as well you have maybe a
16:20:44students that has an access to your
16:20:45database and querying the data like any
16:20:48other role like a manager and data
16:20:49analyst. So as you can see you have now
16:20:51different rules in your project and all
16:20:53of them having the same rights by
16:20:56accessing directly your table. So a
16:20:58manager or data analyst or a student
16:21:01they are seeing the whole table all rows
16:21:04and all columns. And of course in the
16:21:06real projects this is a big problem.
16:21:08Sometimes the data are sensitive and you
16:21:10cannot give an access for everyone. And
16:21:12of course if you are using only tables
16:21:14this going to be a nightmare because you
16:21:15can go and create multiple tables but
16:21:17it's going to be really hard to make all
16:21:19those tables in sync. But instead of
16:21:21that we have views. So what you can do
16:21:23you can go and remove all accesses to
16:21:25the physical table but instead you can
16:21:28go and create multiple views for each
16:21:31role. For example you can go and create
16:21:33a view called orders managers and maybe
16:21:36you can give all the data and all the
16:21:37columns because the managers are allowed
16:21:40to see let's say sensitive data but
16:21:42still it's nice to create a view maybe
16:21:43you change your mind later and you go
16:21:45and remove something. Now let's say that
16:21:47for the data analyst you want to offer
16:21:49all the data but there is only one
16:21:51column that is very sensitive. So what
16:21:53you can do you can go and create another
16:21:55view called orders analyst. So in the
16:21:57view only three columns are available
16:21:59ABC and then you give access to all data
16:22:02analyst and with that you have protected
16:22:05this sensitive information. So we call
16:22:07this column level security. And now we
16:22:09come to our poor students. And here we
16:22:11create another view where we are not
16:22:13only protecting the column D but also we
16:22:16are protecting few rows like for example
16:22:19the row number three because we want to
16:22:21offer only few informations to the
16:22:23students. So we are protecting the
16:22:24columns and as well the rows and for
16:22:26that we can create another dedicated
16:22:28view called for example orders students
16:22:31and we can offer it to the students and
16:22:33with that we are doing column level
16:22:35security and as well row level security.
16:22:38So we are offering multiple views very
16:22:40easily without having to worry how to
16:22:42load the data from one table to another.
16:22:45So creating those views are really easy
16:22:47and provide us a perfect tool in order
16:22:50to manage the security of our data. So
16:22:53this is one very common use case of
16:22:55using views in data projects. All right.
16:22:57So now let's have the following task and
16:22:59it says provide a view for EU sales team
16:23:02that combines details from all tables
16:23:04and excludes data related to the USA. So
16:23:08the first part of the task is similar to
16:23:10what we have already done but we cannot
16:23:12offer all data for the user. So this
16:23:15time we are providing a view that is
16:23:17specifically created for a team the
16:23:19sales team. So the first part we have
16:23:21already done it where we are combining
16:23:23all details in one view. But the problem
16:23:25with the view that we have created that
16:23:27it shows all data. But now the
16:23:29requirement change we cannot show all
16:23:30data. We have to go and exclude the USA
16:23:33data from our details. So let's see how
16:23:35we can do that. It's very simple. We're
16:23:37going to go and grab the same query. We
16:23:38will not repeat that. So we have as well
16:23:41here joining tables and prepare
16:23:43everything. But instead of showing all
16:23:46data, what we're going to do, we're
16:23:47going to go and filter the data based on
16:23:49the customer country. So it's very
16:23:51simple. At the ends we will have a work
16:23:54clause where the C country is not equal
16:23:59to
16:24:01USA. So we have now a filter. Let's go
16:24:04and execute it. And with that, as you
16:24:06can see in the output, we are getting
16:24:08the orders that are not from USA. And
16:24:11with that we are protecting the data of
16:24:12the USA and the EU sales teams can
16:24:16access only their data. So it looks nice
16:24:18and protected. And with that we are
16:24:20doing now role level security. That
16:24:22means we are hiding now all the orders
16:24:24all the rows that are not allowed to be
16:24:26seen and consumed from this group of
16:24:29users. So now what is the next step? It
16:24:31is very simple. We're going to go and
16:24:32put everything in one view. So with that
16:24:34we have the query ready and we can go
16:24:36and create the new view. So we're going
16:24:39to call it create view. Then we need the
16:24:42schema and the name going to be almost
16:24:45the same. So order details but EU. And
16:24:49then we have to have as punch
16:24:51parenthesis like this. So everything is
16:24:54ready. Let's go and execute it. And now
16:24:56we can go and refresh in order to see
16:24:58our new view. If you still don't see it,
16:25:00you can go to the views over here and
16:25:02refresh as well to the folder. So with
16:25:04that I can see we have our new view.
16:25:06Now, of course, the next step we go and
16:25:08test it. So, let's create a new query.
16:25:11Select star
16:25:13from sales and v order details EU. So,
16:25:19let's test it. And with that, as you can
16:25:21see, we are getting the combined view
16:25:23only for the data that is relevant for
16:25:25the EU sales team. So, I'm not seeing
16:25:28here any USA records. So, with that, we
16:25:30are providing view that protects few
16:25:33rows like the orders from USA. So as you
16:25:35can see views are really great in order
16:25:38to provide security to our data whether
16:25:40we are protecting the columns or the
16:25:42rows. For example in our view we can say
16:25:44not only I want to remove the USA orders
16:25:48but let's say the department information
16:25:50is sensitive information and I would
16:25:52like to hide it from the view. So you
16:25:53can just simply remove it from the
16:25:55select and with that you are doing
16:25:56column level security. So now I have two
16:25:58options that I can provide to the users.
16:26:01The first option doesn't has any like
16:26:03role level security. It is the first
16:26:05view the order details. We don't have
16:26:07there any filters. So it's going to show
16:26:09all the orders. So here we give access
16:26:11only to people that are allowed to see
16:26:12all data. And we have another option the
16:26:15details with the EU. It doesn't show all
16:26:17data. It shows only a subset that is
16:26:19relevant for the EU team. So now it's
16:26:21really easy to control the security of
16:26:23my data using the views. And this is
16:26:26very important use case for the
16:26:28[Music]
16:26:31views. Okay. Okay, so moving on to the
16:26:33next use case for the views, we can use
16:26:35it in order to have more dynamic and
16:26:37flexibility in our projects. So let's
16:26:40understand what this means. If you have
16:26:41a table and you have multiple users
16:26:43accessing this table, now what can
16:26:45happen? you might change your mind about
16:26:47the design and the data model of your
16:26:49database where you can say you know what
16:26:51instead of having one table I'm going to
16:26:53go and split it into two tables or maybe
16:26:55another decision you say you know what
16:26:57I'm going to go and rename a table or in
16:26:59another day you decide you know what
16:27:00let's go and rename few columns or maybe
16:27:03add a column remove column so you are
16:27:05doing changes to your physical data
16:27:06model and you are changing stuff in the
16:27:08tables you know what's going to happen
16:27:10all those users that are accessing the
16:27:12tables going to scream because all of
16:27:15them having a complex SQL queries and
16:27:18your small changes at the tables are
16:27:20breaking everything in their queries and
16:27:22what this means this means escalations
16:27:25and you don't have anymore the freedom
16:27:27to change anything in your database
16:27:29without talking before to 100 people
16:27:32before doing any change. So we don't do
16:27:34that instead of that we use views. So
16:27:36what's going to happen? You create a
16:27:38view and you tell the users, okay, take
16:27:40this view and consume it and leave me
16:27:42alone. And now you have again your
16:27:44freedom to do any changes you want. So
16:27:46you go to your tables and do splitting,
16:27:48renaming and changing everything you
16:27:50want as long as you are updating the
16:27:53query between the table and the view to
16:27:56make sure that the users are not
16:27:58noticing any change. So for example, if
16:28:00you go and split the table into two
16:28:01tables, then you have to put in the view
16:28:03a join or union in order to reconstruct
16:28:07the same structure that the users are
16:28:09used to. And if you would like to rename
16:28:11something in your database, like instead
16:28:13of ID, you are now calling it a key. All
16:28:15what you have to do now is to go to the
16:28:17query of the view and rename it back
16:28:20from a key to an ID. So no one going to
16:28:22notice that you are doing changes to the
16:28:24physical tables. So using views and
16:28:27offering it to users is a gamecher for
16:28:29you because giving the users views kind
16:28:32of gives you more freedom dynamic and
16:28:34flexibility to change anything in your
16:28:36data model and the tables without
16:28:39getting any headache. So this is amazing
16:28:41use case for the
16:28:45views. Okay, moving on. We have a lot of
16:28:47use cases for the views. They are just
16:28:49amazing. So the next one is we can use
16:28:52views in order to introduce a second
16:28:54version of my data model in another
16:28:56language. So we could offer multiple
16:28:58languages to the users. Let's understand
16:29:00what this means. So now we have the
16:29:02following scenario. We have again our
16:29:03table orders where the data is persisted
16:29:06and everything in English and of course
16:29:08what happens sometimes you have like
16:29:10international team that are accessing
16:29:13your data. So you have team in USA and
16:29:15maybe you have team from Germany that as
16:29:17well are end users that want to access
16:29:19the data. Of course it depend on the
16:29:22number of users that are using your
16:29:23database. But if you have a lot of users
16:29:26that come from Germany and as well from
16:29:28India, it might make sense that you go
16:29:30and translate your data and the table
16:29:33structure into another language. So for
16:29:36example, instead of giving access to the
16:29:37table orders, we can create another view
16:29:41called bishong. That's the order in
16:29:43German. But not only you are giving a
16:29:45new name for the object, you could go as
16:29:48well and rename all the columns inside
16:29:50the view. Then the German users going to
16:29:52access the German view and it's going to
16:29:55be for them easier to understand the
16:29:57content of your database. The same thing
16:29:59for the Indian team. And for the Indian
16:30:01users, you can go and provide a view in
16:30:04Hindi. I'm not sure whether I'm
16:30:05pronouncing the word correct, but this
16:30:08is the first word that I said in Hindi.
16:30:10I don't promise that I'm going to learn
16:30:12the Hindi language because it's enough
16:30:14to learn Germany. So I'm trying as well
16:30:16to write this word Adish. I hope it is
16:30:20correct. And to be honest, it is really
16:30:22interesting how you write this word in
16:30:24Hindi. So now back to the topic. As you
16:30:26can see now we are using like the views
16:30:29in order to provide a translation for
16:30:31our database by just giving a new name
16:30:34for the views and as well for the
16:30:35columns. So this is another nice use
16:30:37case that I usually use as well in my
16:30:39projects in order to provide multi-
16:30:41languages for the data model that I have
16:30:43and I can do that with the power of
16:30:49views. Now we come to my favorite use
16:30:52case for the views and that I personally
16:30:54recommend in each project that we can
16:30:56use views as a virtual data ms in a data
16:31:00warehouse. So now why this is my
16:31:02favorite? Because I'm specialist in data
16:31:03warehouses and data leaks and this topic
16:31:06is very important decision in each
16:31:08project like this. So let's understand
16:31:10what this means. So now a classical data
16:31:12warehouse architecture based on the
16:31:14approach of enmon is going to look like
16:31:16this. We have multiple source systems
16:31:18where our data are spreaded and now we
16:31:20would like to go and extract all our
16:31:22data from these multiple sources and put
16:31:25it in one big database called data
16:31:28warehouse. And there will be a lot of
16:31:29operations on this central database like
16:31:32the data going to be first cleaned and
16:31:34then maybe integrated together and maybe
16:31:36we are building there some historical
16:31:38data. So we're going to be doing
16:31:40multiple steps in order to prepare the
16:31:42data for complex reporting and analyzes.
16:31:44And what we usually do in the data
16:31:46warehouse, we're going to store all
16:31:48those informations as a physical table.
16:31:50Now once we have built the data
16:31:52warehouse, what's going to happen? We're
16:31:53going to have multiple use cases that
16:31:56would like to access the data warehouse
16:31:58in order maybe to do some different
16:32:00reporting. Now, it's going to be very
16:32:02complex if we connect immediately like a
16:32:04reporting engine like PowerBI directly
16:32:06to the data warehouse. But instead of
16:32:08this, we try to split the data warehouse
16:32:11into multiple subsets like we can split
16:32:13it after topic or domain or departments
16:32:16and we call those subsets as data marts.
16:32:19So a data mart is always specific for a
16:32:22use case that's focus on one topic like
16:32:25for example we could have a dedicated
16:32:26mart for the sales and another data m
16:32:29which is dedicated only for finance
16:32:31topics but both of them comes from our
16:32:34data warehouse. Then the last layer
16:32:36going to be like for example the
16:32:37reporting and dashboarding maybe you
16:32:39have something like powerbi where you
16:32:40are creating a dashboard one data m like
16:32:43the sales or and as well maybe few stuff
16:32:45from other marts. But now the big
16:32:47question here in the data mart is how
16:32:49should I store the data? Should I store
16:32:51the data using tables or should I use
16:32:54views? And now the best practice says if
16:32:56you are building data marts then use
16:32:58views. And we call this virtual data
16:33:01marts. And there are many reasons why
16:33:03using views at a data mart it's way
16:33:05better than using tables. Like for
16:33:07example, it is more dynamic and quicker
16:33:10to change them cuz usually at the data
16:33:12mart you are building a lot of business
16:33:13logics and you want to have some
16:33:15flexibility and speed and the
16:33:17maintenance efforts is very simplified.
16:33:19No need to build any ETLs or data loads
16:33:22from the data warehouse to the data
16:33:24parts and this makes the data warehouse
16:33:26as a real single point of truth for your
16:33:29data. And once you start copying data
16:33:31from one layer to another layer, it's
16:33:33going to be really hard to maintain and
16:33:35chaotic and you have to have really
16:33:37restrict monitoring and data quality. So
16:33:39that's why using views you're going to
16:33:41always reflect the status of the data
16:33:43warehouse and this can help you of
16:33:45course with the data consistency which
16:33:47is a critical point in each data
16:33:49warehouse project. So there are many
16:33:51reasons why we build virtual data mart
16:33:54and we go with the views in this layer.
16:33:56So as you can see how the views are
16:33:58playing a very important role in
16:34:01building a data warehouse. So this is
16:34:03another amazing and very important use
16:34:05case of using views in your data
16:34:11projects. All right friends, so now
16:34:13let's have a quick recap about views. So
16:34:15we have learned that views are a virtual
16:34:17table that is based on the result of a
16:34:20query without actually storing any data
16:34:23in the database. So we use views in
16:34:25order to presist a complex SQL logic and
16:34:28query in the database. And we have
16:34:30learned that in some scenarios views are
16:34:33better than CTE because it improves the
16:34:35reusability and reduce the complexity in
16:34:39multiple queries which reduce the
16:34:41complexity of the whole projects where
16:34:43the CTE only improves the reusability in
16:34:45one query. And we have learned that as
16:34:47well the views in some scenarios are
16:34:49better than tables. We have learned that
16:34:51they are very flexible and easier to
16:34:53maintain since they don't store any data
16:34:56and it's really fast and easy to change
16:34:58stuff in the view compared to the
16:35:00tables. But as well we have learned that
16:35:02the tables are faster than views. Now
16:35:04there are like endless use cases for the
16:35:06views. But from my experience in
16:35:08projects I have choose for you the best
16:35:10use cases for the views. The first use
16:35:11case is if we find like a common
16:35:14repeated logic in SQL queries, we can go
16:35:18and store this logic in view in the
16:35:20database so that the users don't have to
16:35:22keep repeating the logic over and over.
16:35:24So we use views in order to have a
16:35:26central business logic. Another use case
16:35:29is to hide the complexity of your
16:35:31physical data model and to offer for the
16:35:33users and high abstracted layer. So you
16:35:36provide for the user something very
16:35:37friendly and you hide all the complex
16:35:40technical data model that you have in
16:35:41the database because not everyone is
16:35:43expert with your data model. One more
16:35:45use case we can use views in order to
16:35:48implement security and to protect our
16:35:50sensitive data in the database. So we
16:35:52can offer multiple views in order to
16:35:54protect columns or rows in a table.
16:35:56Another use case we have learned that we
16:35:58can use views in order to have more
16:36:00dynamic and flexibility for your
16:36:03database where we offer the users a
16:36:05table view and then you have the freedom
16:36:07to change stuff at your physical data
16:36:09model without affecting all users. And
16:36:12another nice use case for the views we
16:36:14can offer multiple languages from our
16:36:16data model. And the last use case we
16:36:18have learned how views play an important
16:36:21role in a data warehouse system. So
16:36:23views are amazing. All right my friends.
16:36:25So with that we have learned everything
16:36:26about this new objects the views in
16:36:29databases. This is amazing for
16:36:30flexibility and dynamic in your
16:36:32projects. Now in the next one we're
16:36:34going to learn how to create tables
16:36:35based on query and we will learn about
16:36:38the temporary tables. So let's
16:36:43go. Okay. So now first let's have a look
16:36:46again to the database structure. We have
16:36:48learned that in each SQL server there
16:36:50are multiple databases and in each
16:36:52database there are multiple schemas. And
16:36:54now inside each schema we can define
16:36:57multiple objects like we can define
16:36:59tables and views. And now we will be
16:37:01focusing on the object table. And we
16:37:03have learned as well we can use the
16:37:05language DDL data definition language
16:37:07which is a set of SQL commands in order
16:37:10to define this database structure. So we
16:37:12can use the SQL command create in order
16:37:14to define a new table or alter in order
16:37:17to update the structure or drop in order
16:37:20to drop the whole table. So a table is
16:37:23an object in the database structure and
16:37:25we have learned as well there is three
16:37:27levels of the database architecture and
16:37:29we have understood that at the logical
16:37:31level the middle one the conceptual
16:37:33level we deal as application developer
16:37:36or data engineer with the tables. So we
16:37:39define tables and relationship between
16:37:41them. So if you are an end user or a
16:37:43business analyst it's going to be little
16:37:45bit more hard to work with the tables.
16:37:47You have to be a developer or a data
16:37:49engineer. But working with tables is way
16:37:52easier than working with the complexity
16:37:54of the database at the physical level.
16:37:57So you don't have to be a database
16:37:58expert or administrator to work with
16:38:01tables. So the difficulty here is like
16:38:03in the middle. The abstraction is not
16:38:05that low but as well not that high. So
16:38:07now let's answer the question, what are
16:38:09tables? A database table is a structured
16:38:12collection of data. It's like a simple
16:38:14grid or spreadsheet that you might find
16:38:16in Excel. So it has different columns
16:38:18like each column represent a field like
16:38:20the ID, name, country and the table has
16:38:23as well multiple rows and each row
16:38:26represent a record or an entry of the
16:38:28data. So for example if this table is
16:38:30about the employees then each record
16:38:33each row is one employee. Now the
16:38:36intersect between the rows and columns
16:38:38we call it a cell and a cell is a single
16:38:41piece of data. Now the whole table going
16:38:43to be stored physically in the database
16:38:45as database files. So they are in the
16:38:48database like multiple files that are
16:38:51holding the informations about the table
16:38:53and those files are stored physically in
16:38:56that disk storage of the database. So
16:38:58that means your data inside the tables
16:39:00are not stored like a spreadsheet like
16:39:02an Excel but they are stored in special
16:39:05database files that usual developers and
16:39:08end users don't have access to those
16:39:10files. So tables again it's like an
16:39:12abstraction and representation for the
16:39:14actual data that are in the files. So
16:39:16actually each time you are querying the
16:39:18database table the database has to go to
16:39:20those files and fetch the data for you.
16:39:22All right. So this is what we mean with
16:39:24database
16:39:28tables. Okay. So now we have like
16:39:31different types of tables in SQL. We
16:39:33have tables that stays forever. We call
16:39:35it permanent tables. So they stay as
16:39:38long as you don't drop them. And you
16:39:40have another type of tables they called
16:39:42the temporary tables. And those tables
16:39:45going to be deleted and dropped once the
16:39:47session ends. So now we're going to
16:39:48focus first on the first type, the
16:39:50permanent tables. And there are two ways
16:39:52on how to create them. The first way is
16:39:55the classical way where you create table
16:39:57from the scratch and then you go and
16:39:59insert your data. So we call it create
16:40:01insert and the other way called create
16:40:05table as select. It's going to create as
16:40:07well the table but based on SQL query.
16:40:10So let's understand the differences
16:40:12between
16:40:15them. The create insert method is the
16:40:18classical way on how we define and
16:40:20create tables in SQL where first we have
16:40:23to go and create the table and define
16:40:25the structure and after that we insert
16:40:27our data into the database table where
16:40:29the other method the CTAs create table
16:40:32as select. And this one going to create
16:40:34a new table as well but this time based
16:40:37on the result of SQL query. So let's
16:40:40understand what this means. Okay. So now
16:40:42to the first method create insert. So
16:40:44here we have two steps. The first step
16:40:46is we have a DDL statements where we use
16:40:49the command create. So once we execute
16:40:51the first step what's going to happen
16:40:52the database engine going to go and
16:40:54create for us an empty table. It is a
16:40:57brand new table where we can hold our
16:40:59data. So with that we have defined the
16:41:01structure of our table but it's still an
16:41:04empty table. So now in the next step we
16:41:06have to go and insert our data inside
16:41:08this new table. So our data can come
16:41:10from multiple sources like a CSV file or
16:41:13maybe completely from another database
16:41:15where we are doing migration or maybe
16:41:17you are inserting manually your data or
16:41:19maybe it come from an application or you
16:41:21are doing data migration from one
16:41:24database to another. So at the end once
16:41:26you execute insert what's going to
16:41:27happen your data going to be inserted in
16:41:30this new table. So in this method we
16:41:32have like two steps. First we define the
16:41:35structure of the table and the second
16:41:37step we take care of inserting our data
16:41:40inside the table. And now this new table
16:41:42and your data going to be persisted
16:41:44permanently. Now let's check the other
16:41:46method the CTIS. Here it's only one step
16:41:49where you define a query and once you
16:41:51execute this query what going to happen
16:41:53the database has to retrieve the data
16:41:56from another table. So it might retrieve
16:41:58data from our new table that we just
16:42:00created using create insert. So once the
16:42:02query is executed we will get a result.
16:42:05So now what the database going to do
16:42:07going to create a new brand table but
16:42:10this time the definition and the data of
16:42:12this new table it doesn't come from any
16:42:15definition that we specify. it comes
16:42:17from the result of the query. So
16:42:20whatever structure that we have in the
16:42:22results, it going to be reflected in our
16:42:24new table. So again the definition and
16:42:27the data that we see in this new table
16:42:29comes one one to one from the result of
16:42:32our query. So in this type we don't have
16:42:34to define anything or to insert any
16:42:37data. We are just writing a query and
16:42:39the output of this query going to define
16:42:41the table. But in this method as you can
16:42:43see it always needs a database table in
16:42:46order to execute the query. But the
16:42:47create insert method we are creating
16:42:50something from the scratch. So these are
16:42:52the two different ways on how you create
16:42:54tables in SQL and the differences
16:42:56between
16:43:00them. Okay. So now you might ask you
16:43:02know what the CTAs are very similar to
16:43:05the views. We have a query and the
16:43:07output of this query going to be like an
16:43:08object in the database. So what are the
16:43:11differences between them? Let's check
16:43:13this. Now let's say that in our database
16:43:15we have a table that has three columns
16:43:17A, B, C. And now what we can do, we can
16:43:19go and create view based on a query. So
16:43:21you create the DDL statement in order to
16:43:23create the view in the database. And of
16:43:25course the database going to go and
16:43:26store the query in the database and it's
16:43:29going to be empty. So there will be no
16:43:31data because views does not store any
16:43:33data and the query of the view will not
16:43:35be yet executed. But now in the other
16:43:37hand if you go and create a table using
16:43:39CTIS. So here again we have a query
16:43:42attached to the object to the table. So
16:43:44here what happens the database has to
16:43:46execute the query in order to understand
16:43:49the structure and as well the data that
16:43:52should be inserted inside the table. So
16:43:54our SQL query going to be executed and
16:43:57the result of the query going to be
16:43:59inserted inside the table. So that means
16:44:01this new table is storing already the
16:44:04result of the query. So now this is the
16:44:06first differences between the table and
16:44:08view. As you create view the query will
16:44:10not be executed and we don't have
16:44:12anything about the result of the query
16:44:15where in the CTIS we have already result
16:44:18of the query stored inside the table and
16:44:20everything is prepared. So now let's see
16:44:22what's going to happen once the user
16:44:24selects something from the view. So now
16:44:26the database going to go for the first
16:44:28time executing the query of the view in
16:44:31order to fetch the data from the
16:44:33original table and then presented as a
16:44:36result for the user. But now in the
16:44:37other hand if the user go and query the
16:44:40table that is created from the CTIS. So
16:44:42now what can happen? SQL will not
16:44:44execute again the query of the CTIS
16:44:47because the database already done that
16:44:49and prepared everything. So that means
16:44:51we are not querying anything from the
16:44:53original table and the data can be
16:44:55directly fetched from the new table. So
16:44:58the user is going to get immediately the
16:44:59result from our table that is created
16:45:01from the CDIS. So here comes the second
16:45:04difference between the tables and views.
16:45:06The views are slower than CTIS and
16:45:09that's because the database has here an
16:45:11extra task. It must execute the query of
16:45:14the view in order to get the data. But
16:45:16in the CTIS the query going to be faster
16:45:19than the view because we have already
16:45:21executed everything and prepared it for
16:45:23the user. So that's why tables from CTIS
16:45:26are way faster than views. And now there
16:45:28is another difference and perspective
16:45:30about this which is from my point of
16:45:32view is more important than the
16:45:34performance. So now let's say that in
16:45:36the next day we are doing data updates
16:45:38on the original table like we are doing
16:45:40updates on the column C and as well in
16:45:43the column P. So now let's see what this
16:45:44means for the user if they are using
16:45:46views. So the user in the next day is
16:45:48executing again the same query and again
16:45:50here the database has to execute the
16:45:52query of the view in order to fetch the
16:45:55data from the original table. So that
16:45:57means today in the views we are getting
16:45:59different data than yesterday because we
16:46:02have a new data and new updates and the
16:46:04user in the result going to see as well
16:46:06the new updates and the fresh data. So
16:46:08the user is seeing exactly the status of
16:46:12the data in the original tables. But now
16:46:14let's see what going to happen if the
16:46:15user go and query the table from the
16:46:18CATS. So in the table of the CATS, we
16:46:21are still having the data from
16:46:23yesterday. All those new updates from
16:46:25the original data will not be reflected
16:46:27in this new table because once the user
16:46:29selects something from this table, the
16:46:31database will not go and query or fetch
16:46:34the new changes from the original table
16:46:36because we have already prepared the
16:46:38data from yesterday. So that means our
16:46:40user now is getting old data from the
16:46:43CTAs table and the only way to get new
16:46:46fresh data from the CTIS is to reexecute
16:46:50the CTIS query. And of course this is
16:46:52another step and it is harder to
16:46:55maintain the table from the CTAs and
16:46:58this is a big difference for the users
16:47:00between the views and the tables from
16:47:02the CTAs. Now think about views you are
16:47:04ordering a pizza at restaurants. So
16:47:07every time you are quering the view you
16:47:09are placing an order the chef going to
16:47:11go and make a pizza from the scratch
16:47:13using the freshest ingredients. So that
16:47:15means you are always getting a fresh hot
16:47:18pizza. And think about the CTS as like a
16:47:21frozen pizza from a grocery store. The
16:47:23pizza was prepared earlier and stored in
16:47:26the freezer. And if you want to eat it,
16:47:27you have to go and heat it up in the
16:47:29oven. But it's still not like a fresh
16:47:31pizza that is made on the spot and from
16:47:33the scratch. Now I made myself hungry
16:47:35because I love pizza. So I think I'm
16:47:37going to go for a quick break.
16:47:40[Music]
16:47:51Okay, so now let's check quickly the
16:47:52syntax of those two methods. The first
16:47:54one is create insert. So first step we
16:47:56have to go and create a table using a
16:47:58DDL statements. So we use the command
16:48:00create and then we have to tell SQL are
16:48:03we creating a table or view. In this
16:48:05scenario we are creating a table and
16:48:07then we specify the name of the table.
16:48:09Then after that we have two parenthesis
16:48:11and inside them we make a list of all
16:48:13columns that we need inside this table.
16:48:15So we have two columns the ID and the
16:48:17name. And after that we are defining the
16:48:19data type of those columns and maybe as
16:48:21well the length. There are a lot of
16:48:23options that we can add to this syntax
16:48:25but now we are just checking the
16:48:27simplest form of creating a table. Now
16:48:29the next step is that we need an insert
16:48:31statement. So we are saying insert into
16:48:34our new table the following values. we
16:48:36are inserting the id number one and the
16:48:39value for the name going to be frank. So
16:48:41this is a classical way on creating new
16:48:43table and inserting data to it. Now
16:48:45let's move to the second method the cas.
16:48:47Now this time we have an SQL query like
16:48:50select from where and some extra logic.
16:48:53So this is our query and then we're
16:48:55going to go and put our query inside a
16:48:58DDL statement. It's like we have done it
16:49:00in the views. It's exactly like we have
16:49:02done it in the views but this time
16:49:05instead of saying view we're going to
16:49:06say table. So again we have the create
16:49:08command and we are creating a table then
16:49:11the name of the table and then we say as
16:49:14and then we have two parenthesis and
16:49:16inside them we have our query and this
16:49:18is where the name come from create table
16:49:22as select cas. So it is very simple in
16:49:26one statement you have everything you
16:49:28are creating a new table and as well you
16:49:31are inserting the data that comes from
16:49:33this query. Now this syntax is used in
16:49:36databases like MySQL, Postgress and
16:49:39Oracle. But in MySQL we have like a
16:49:41shorter way on how to do it. Again we
16:49:44have our query select from where. But
16:49:47now in SQL server we can insert a
16:49:49command between the select and from like
16:49:52this. So we are saying select the
16:49:55following columns into new table. So we
16:49:59have this keyword into then the table
16:50:01name and then you continue after that
16:50:03with your query from where aggregations
16:50:05and so on. So here it's like the DDL is
16:50:08inside your query itself but in the
16:50:10other databases you can have like the
16:50:12query is separated from the DDL
16:50:14statements. Personally, I prefer this
16:50:16syntax than having this into because if
16:50:19you have like big complex query, this
16:50:21can be really hard to see and to miss
16:50:23the column selection. So, this is the
16:50:25syntax of creating a new table from a
16:50:28query the CTAs in different
16:50:30[Music]
16:50:33databases. Okay. So, now we're going to
16:50:35check the scenarios and use cases where
16:50:37it makes sense to use. So, let's start
16:50:40with the first one. Now we have learned
16:50:42before it makes sense to have a complex
16:50:44logic stored inside the database so that
16:50:47our end users don't have to keep
16:50:49repeating the same logic over and over
16:50:51and it's as well maybe complicated for
16:50:53some users. So that's why we have used
16:50:55views and the result of the view going
16:50:57to be used from our users. So everything
16:50:59can stay easy and friendly to consume
16:51:01for our users. But now what might happen
16:51:04is that the logic of the view could be
16:51:06very complicated and needs a lot of time
16:51:09to be executed from the database. So it
16:51:12takes really long time until we get the
16:51:14intermediate result from the database.
16:51:16So that means if it's going to takes 30
16:51:18minutes then each users has to wait 30
16:51:21minutes until the query is executed and
16:51:23none of your users going to be happy
16:51:25with this situation. In this scenario,
16:51:27if this happens, you have to try maybe
16:51:29to optimize the query. But if you cannot
16:51:32do anything about that, you have to
16:51:34switch the view to CTAs table. So now
16:51:37what you have to do, you have to take
16:51:38the same logic and then put it
16:51:41in so that the intermediate results are
16:51:44stored in a table. And of course at the
16:51:46moment of creating the table, it will
16:51:48take 30 minutes. It will take long time
16:51:50because it is the same query and the
16:51:52database going to need the time until
16:51:54creating the intermittent results. But
16:51:56the big advantage is that once
16:51:58everything is prepared maybe at the
16:52:00night at the morning once your users are
16:52:03like online and start querying the data
16:52:06they have everything prepared. So the
16:52:08user is going to go and start selecting
16:52:10and analyzing the intermediate result
16:52:12but this time using the table that you
16:52:15have created from the CTAs and the
16:52:17response time going to be for all users
16:52:19again normal and fast. So if you have a
16:52:21scenario where your views are very slow
16:52:24you have to go and prepare the data at
16:52:25the night using the CTIS and prepare the
16:52:28tables to be analyzed from the end
16:52:30users. So this is the most common use
16:52:32case for the CTIS and this scenario
16:52:35happens a lot in projects where you
16:52:37decide to go instead of views to go with
16:52:40the CTIS in order to have persistence
16:52:42data and you gain performance. Okay, so
16:52:44finally back to SQL let's go and create
16:52:47a table using now we're going to go and
16:52:49create a table that shows the total
16:52:51number of orders for each month. Let's
16:52:53go and do it. So first what do we need?
16:52:55We need a query. So let's write it.
16:52:58select. I'm going to go with the date
16:53:00name in order to get the name of the
16:53:02month from our order dates and we're
16:53:06going to call it order month. And then
16:53:09we're going to go and aggregate the data
16:53:11by counting the order ID for total
16:53:16orders from our table sales orders. Uh
16:53:20don't forget to group by our month. So
16:53:24something like this. Let's go and
16:53:27execute it. So the result is very
16:53:28simple. We have the order month and the
16:53:30total orders. So we have two columns and
16:53:33three rows. So we have our query and of
16:53:35course we didn't create anything yet.
16:53:37Now in SQL server in order to create a
16:53:39table from the query what we're going to
16:53:42do exactly before the from we're going
16:53:45to write into and now we have to specify
16:53:47the schema and the table name. I'm going
16:53:49to stay with the schema sales and I'm
16:53:51going to call it monthly
16:53:54orders like this. So that means we have
16:53:57our query and the DDL is exactly between
16:54:00the from and select. So now if I go and
16:54:02execute this what going to happen we
16:54:05will not see here the result of the
16:54:06query. We're going to get here like
16:54:08three rows affected because this is a
16:54:10DDL statement. It is not anymore a query
16:54:13and the database is telling us I have
16:54:14created now a table with three rows. So
16:54:17now if you check our tables we don't see
16:54:18it yet. Let's go and refresh and check
16:54:20again the tables. Now we can see our
16:54:22table here sales monthly orders. Now of
16:54:25course we have to go and check whether
16:54:27everything is fine. So let's go and
16:54:29select the rows from our new table sales
16:54:33monthly
16:54:35orders. So let's go select it first and
16:54:39execute. And now we can see again the
16:54:42result of our query. But we are not
16:54:44writing here the query. We are just
16:54:45selecting it from the table. So our data
16:54:47is stored in our table. And we can go
16:54:50and check the structure of this table.
16:54:52So if you go to the columns you can see
16:54:54we have here the order month and the
16:54:56total orders and those informations
16:54:58comes from our query. So SQL is saying
16:55:00here the order month is a var which is
16:55:02correct because here we have the names
16:55:04of the month. So SQL is able to define
16:55:07the data type of the table from our
16:55:09query and the second column the total
16:55:11orders it is an integer and that's
16:55:13because we have here numbers. So as you
16:55:15can see SQL is defining the structure of
16:55:17the table based on the result of our
16:55:20query over here. And of course the data
16:55:22inside the table comes as well from the
16:55:24query. And the result of this table
16:55:26going to stay like this as long as you
16:55:28don't change anything. So if you go and
16:55:30close this and open it after one year
16:55:32it's going to show exact same results.
16:55:34So it's going to live in the database as
16:55:36long as you don't drop this table. But
16:55:38if things change in the table orders,
16:55:41this table will not be updated
16:55:42automatically like we have learned in
16:55:44the views. So now if you want to say you
16:55:46know what I would like to go and drop
16:55:47this table well it is very simple just
16:55:50go and say drop table and the table name
16:55:53over here. So make sure you select it
16:55:55and execute it. And now if you go over
16:55:57here and refresh. So let's check the
16:55:59tables. You can see here the table is
16:56:05dropped. And now if you say you know
16:56:07what let's go and refresh the table that
16:56:10come from the CTAs every day so that we
16:56:13always get refresh data inside this
16:56:15table. So now let's go and execute again
16:56:18our CIS. And with that if we go and
16:56:20refresh we're going to find again our
16:56:22table inside it. Now if you go and
16:56:24execute it one more time in order to
16:56:26refresh the data of the table what you
16:56:28going to get? You're going to get an
16:56:29error. The database going to tell you we
16:56:31have already this table so we cannot
16:56:32recreate it. So now the question is how
16:56:35we can update the the content of this
16:56:37table. Well, we have to go and drop it
16:56:39first and then recreate it. And if you
16:56:41want to put everything in one statement,
16:56:43we have to go and use the TSQL. It is
16:56:46transacts SQL. It's like extension where
16:56:48you can do some programming inside SQL.
16:56:51So in order to do that, what we're going
16:56:53to do, we're going to go at the start
16:56:54over here and we're going to make an if
16:56:57logic. So we're going to go and search
16:56:58for the objects. So we're going to say
16:57:01if the object ID and now we have to go
16:57:03and specify the name of this object
16:57:06together with the schema. Make sure to
16:57:07select everything sales monthly order
16:57:10and put it inside here. And then we have
16:57:12to define the type of this object. And
16:57:15here we're going to go with you. It is
16:57:17userdefined table. So we are saying if
16:57:21the object sales monthly orders is not
16:57:24null. So that means it exist. So what
16:57:27you want to do? we have to go and drop
16:57:29it. I'm going to take the statement from
16:57:31here and then we're going to put it
16:57:33after the if over here. So we are saying
16:57:36if this table exist then drop the table
16:57:39otherwise don't do anything because we
16:57:41don't have any new table and the query
16:57:43going to work and at the end of the TSQL
16:57:46we have go in order to say the TSQL is
16:57:49done and then our usual query after all
16:57:51that. So let's go and execute the whole
16:57:53thing and as you can see it is working.
16:57:56So what happens? The database did find
16:57:58this table and drop it and then executed
16:58:01our query. So if you keep executing
16:58:03this, you are just refreshing the
16:58:05content of this table. So this is how we
16:58:07work with the CTAs in
16:58:13SQL. All right, moving on to another
16:58:15common use case for the CTAs that I
16:58:17usually use as well in my projects. We
16:58:19use CDS in order to create a persistent
16:58:22snapshot of the data at specific time in
16:58:25order to analyze data quality issue. So
16:58:28let's understand what this means. Now in
16:58:29some scenarios you have like a table and
16:58:31you are analyzing an issue. So there is
16:58:33like a data quality issue at your data
16:58:35and you are analyzing this scenario in
16:58:37order to understand why it happens. But
16:58:39the problem is that at the same time
16:58:41there will be updates on the table and
16:58:44your data is changing. So there will be
16:58:46updates maybe on some fields or you are
16:58:48getting new records and everything is
16:58:50getting mixed up and you will not be
16:58:52able to analyze the scenario where the
16:58:54data quality issue happened. So now it's
16:58:57almost impossible to find the ro cause
16:58:59of your issue. But instead of that what
16:59:02we do if we have like an issue of the
16:59:03data we go and create a fixed persisted
16:59:06snapshot of the data in a separate table
16:59:09using CTS so that we make sure nothing
16:59:11is changing and everything is fixed. And
16:59:14with that I can keep doing my analysis
16:59:16on the same data without the worry that
16:59:19data are getting changed. So this is
16:59:21another way why we use CTS in projects
16:59:23to make sure that we have snapshot of
16:59:25the data to ensure that our analyzes are
16:59:28done on the same scenario that caused
16:59:30the buck and going to be used as a
16:59:32foundation for finding the problem and
16:59:35fixing
16:59:39it. All right, moving on to another use
16:59:41case of the CTAs. We can use it in order
16:59:44to create our data m to make it physical
16:59:47data m instead of virtual data ms using
16:59:50views. So let's understand what this
16:59:52means now. As we learned before, if you
16:59:53have a data warehouse system, our data
16:59:56warehouse layer going to store the data
16:59:57inside tables. But for the second layer,
17:00:00the data m, we can go and use views in
17:00:02order to have dynamic and flexibility in
17:00:04order to generate multiple data ms. And
17:00:07we called it the virtual layer. But now
17:00:10in some scenarios if things get
17:00:12complicated your data m and reports
17:00:14going to be slow because there for each
17:00:16action you are generating a query. So
17:00:19the powerbi reports and dashboards are
17:00:21creating queries in your data marts and
17:00:23your data marts have always to go to the
17:00:25data warehouse in order to retrieve the
17:00:28data for the reports and the whole thing
17:00:30could take minutes or maybe sometimes
17:00:32hours. So in these scenarios we cannot
17:00:34stay using views because they are
17:00:36slowing everything down. But instead of
17:00:38that we have to convert our data mart to
17:00:41a physical layer. That means instead of
17:00:43using views we have to go and use
17:00:45tables. And one very common way in order
17:00:47to generate the tables of the data marts
17:00:49on daily basis is to use queries between
17:00:53the data warehouse layer and the data
17:00:55mart layer. It's still going to take
17:00:56maybe 30 minutes. That's why you can go
17:00:58and prepare the data at the night. But
17:01:00at the reporting layer where things and
17:01:02the performance really matters, the
17:01:04performance going to be better because
17:01:05the response time from the tables is way
17:01:08faster than views and the reports don't
17:01:10have always to waste time waiting for
17:01:12the data marts to get data from the
17:01:14warehouse. So this is another use case
17:01:16where you use CTAs where the views at
17:01:18the data marts are slow and we have to
17:01:20go and replace them with stables using
17:01:22CTAs to speed up things. But still my
17:01:25recommendation here is that start first
17:01:27with the views. So create a virtual data
17:01:29mart using views because the
17:01:31implementation going to be very dynamic
17:01:33and fast and you are always getting
17:01:35fresh data from the warehouse but maybe
17:01:37later if you notice okay some data ms
17:01:39and models are complex then maybe go and
17:01:42replace few marts from views to tables
17:01:45using cis. So this is another use case
17:01:47for the and it is nice workaround for
17:01:50your data warehouse system. All right
17:01:52friends, so with that we have covered
17:01:53now the first type of the tables that we
17:01:55have in databases. The permanent tables
17:01:58where you create a table and it's going
17:01:59to live forever until you go and drop
17:02:01it. Now we're going to talk about
17:02:03another type of tables in databases. We
17:02:05have the temporary tables. So let's
17:02:07understand what are temporary
17:02:12tables. So temporary tables or sometimes
17:02:15you call them as a shortcut temp tables.
17:02:18They store intermediate results in a
17:02:20temporary storage in the database during
17:02:22a session and the database automatically
17:02:25drop these tables after the session
17:02:28ends. So let's understand what this
17:02:30means. Now we have learned in the CIS we
17:02:32could use a query in order to retrieve
17:02:35data from one table and then it puts the
17:02:37intermediate results in brand new table
17:02:40in the database. So with that we are
17:02:42creating another table based on a query.
17:02:44The same thing for the temporary tables.
17:02:47We have as well a query that goes and
17:02:49retrieves the data from a table and as
17:02:51well the database going to go and create
17:02:53new brand table in the database that has
17:02:55the structure and the data from the
17:02:58result of the query. So it is exactly at
17:03:00the CTIS. What is the difference here?
17:03:02Well, it is about the lifetime of the
17:03:05table. Now the database tables that you
17:03:07have created using create insert or CTIS
17:03:10those tables going to stay permanent and
17:03:12they're going to live in the database as
17:03:14long as you don't drop them. So even if
17:03:17the system is completely offline the
17:03:19data going to stay at the database once
17:03:21it is online again but the temporary
17:03:23tables going to get deleted and dropped
17:03:26from the database automatically once the
17:03:29session ends. So what session means like
17:03:32once you open the client and you connect
17:03:34to the database and you are start doing
17:03:36queries we call the time between
17:03:38connecting ourself to the database and
17:03:40disconnecting from the database we call
17:03:43this a session. So that means once you
17:03:45close the client and you disconnect from
17:03:46the database and maybe shut down your PC
17:03:48and do something else. What going to
17:03:49happen? The database going to go and
17:03:51destroy and delete all the temporary
17:03:54tables that you have created during the
17:03:56session. So that mean the table going to
17:03:58live as long as you have a session and
17:04:00you can access during this time the
17:04:02table as you are accessing any other
17:04:04permanent table. So this is what we mean
17:04:06with temporary tables or sometimes we
17:04:08call it as a shortcut temp
17:04:10[Music]
17:04:13tables. Okay. So now let's check the
17:04:16easiest syntax ever. So for the
17:04:18temporary table the syntax going to look
17:04:19like this. you're going to have like a
17:04:21query select from where and as we
17:04:23learned in the CTIS if you go and say
17:04:26into then the table name it's going to
17:04:28go and create a physical new table but
17:04:30now if you want it as a temporary table
17:04:32what you going to do you're going to
17:04:34just put hash before the name of the
17:04:36table then SQL can understand okay now
17:04:38we are talking about temporary table and
17:04:40the database going to store it in that
17:04:42temporary storage so it is very simple
17:04:45this is the syntax of that temporary
17:04:47tables so so far we have learned that we
17:04:49have a database called sales DB and
17:04:51inside it we can find the tables that we
17:04:53have created the customers, employees,
17:04:56orders and so on. Those are our tables
17:04:58and they are always there like if you go
17:05:00and close everything and then start it
17:05:02or in the next day you're going to find
17:05:03always those tables with the same data.
17:05:06So they're going to exist as long as we
17:05:07are not dropping them. Now the question
17:05:10is where do we find the temporary
17:05:12tables? Well, as we learned, if you go
17:05:14over here at the system databases, you
17:05:16will find multiple databases from the
17:05:19SQL server and normally only the
17:05:21database administrator has an access to
17:05:23this and one of those databases called
17:05:25temp DB, temporary database. So, let's
17:05:28go inside it. Now, we can find multiple
17:05:30objects and one of them we can find here
17:05:32the temporary tables. And now, of
17:05:34course, we don't have anything inside it
17:05:36because we didn't create anything. So,
17:05:38let's go and create one. We have already
17:05:40an open session and active session with
17:05:41the SQL server. As you can see here, we
17:05:43are connected to the database and we can
17:05:45start creating temporal tables. So now
17:05:47what is the plan? I would like now to do
17:05:49few modifications on the table orders.
17:05:52But I will not do it directly at the
17:05:54table orders. I would like to take a
17:05:56copy from the sales DB and create from
17:05:59it a temporary table. So let's go and do
17:06:01that. What do we need first? We need a
17:06:04query. So I would like to select
17:06:06everything all the columns all the rows
17:06:08from the table orders. So from sales
17:06:12orders. So this is my query. Now so far
17:06:16nothing is created. We have only select
17:06:17statements. But now in order to create a
17:06:20temporary table what we're going to do
17:06:21we're going to put a statement between
17:06:23the select and from. So exactly before
17:06:26the from go over here and say into then
17:06:29in order to make sure it is a temporary
17:06:31table we use hash and then the table
17:06:33name. So we're going to call it orders.
17:06:35So that's it. We have our query and in
17:06:38between we have the into and make sure
17:06:40you are using hash in order to be a
17:06:42temporary table. So let's go and execute
17:06:44it. And now we can see that 10 rows are
17:06:46affected and we don't have any error.
17:06:48And now of course we cannot see it yet
17:06:50because we have to go and refresh the
17:06:52object explorer. So let's go and do
17:06:53that. And now let's expand it. And now
17:06:56we can see our temporary tables. As you
17:06:58can see it is at the schema dbo because
17:07:01we haven't defined any schema. And this
17:07:03is the default one from the database. So
17:07:05nice. Now we have the table and let's go
17:07:07and check few stuff. So let's go and
17:07:09select the table itself. So select star
17:07:12from and make sure to say hash orders.
17:07:16Let's go and select it. And now we are
17:07:18getting the data from the temporary
17:07:20table and not from the original table.
17:07:22The orders in the database sales DB. So
17:07:25all those informations comes from the
17:07:27temporary table. Now, of course, you can
17:07:29do whatever you want to this temporary
17:07:30table because it's not that important
17:07:32and it's anyway going to get deleted.
17:07:34So, let's say that I would like to
17:07:36delete all the orders where the order
17:07:39status equal to delivered. So, let's go
17:07:41and do that. What we're going to do
17:07:43delete from our hash orders. So, make
17:07:47sure we are selecting the temporary
17:07:49table and then where we're going to say
17:07:51the
17:07:53order status equal to what I say
17:07:57delivered. Yeah, delivered. So delivered
17:08:00like this. Let's go and execute it.
17:08:02Okay, with that it says five rows are
17:08:04affected. Let's go and select it again.
17:08:07So
17:08:08select from
17:08:11orders and let's check that. So as you
17:08:14can see now we don't have all orders. We
17:08:16have only the orders where the status
17:08:18equal to shipped. So all delivered
17:08:19orders are removed. And now we can do
17:08:22whatever we want to this copy. We can
17:08:23analyze it. We can modify it. We can go
17:08:26and insert a new data. So we can do
17:08:28whatever manipulation we want on this
17:08:30copy. And now if you say, you know what,
17:08:31I like this result and I would like to
17:08:33have it not only during the session.
17:08:36Maybe I'm going to need it for tomorrow
17:08:37or something. So now what we're going to
17:08:39do, we're going to do the exact
17:08:40opposite. We're going to now store the
17:08:42result of the temporary table back to
17:08:44our database so that we don't lose this
17:08:47intermediate result. So in order to do
17:08:49that, we're going to say into and then
17:08:52make sure to specify the sales dot
17:08:54because we want to select the correct
17:08:56schema and then let's say it is orders
17:08:59and I'm going to call it test like this.
17:09:02So let's go and execute it. So it says
17:09:04five rows are affected. Now we have to
17:09:07see those informations in the sales DB.
17:09:09We still don't have this table over
17:09:10here. So right click on the DB and then
17:09:13refresh it. So let's go again to the
17:09:15tables. And now you can see we have our
17:09:17new table orders test. So it is amazing
17:09:20right? What we have done is we have took
17:09:22a copy from the original table orders to
17:09:25a temporary space. We have done some
17:09:28modifications and play with the data and
17:09:30we have done some analyzes and then the
17:09:32end result of our temporary table. We
17:09:35have loaded back to another new table
17:09:38called orders test in order maybe in the
17:09:40next day to keep working on it. So it is
17:09:43really nice way to do changes in place
17:09:45where you say you know what it is
17:09:47temporary and whatever mistakes you
17:09:48makes it's okay it is like playground.
17:09:51So now we still have an active session
17:09:53with the database and our temporary
17:09:55table going to be always here. Now let's
17:09:57see what going to happen if we end our
17:09:58session. So in order to do that let's go
17:10:01and just close everything. So I will
17:10:04just close and we'll not store anything.
17:10:07So with that we have now ended the
17:10:09session. Let's go and start it again and
17:10:11see whether we still have the temporary
17:10:13table. So we have now again to connect
17:10:15to the SQL server and now we have
17:10:18another session. So that means the old
17:10:20session is already lost. Let's go to the
17:10:22databases to the system databases to the
17:10:25temp DB and let's go to the temporary
17:10:27tables. As you can see the database
17:10:30already cleaned up everything and this
17:10:32space is again empty for any new
17:10:34temporary table that I'm going to
17:10:35create. So as you can see once you close
17:10:38the session everything going to get
17:10:39lost. Now let's go back to our sales DB
17:10:41over here to the tables. We can see the
17:10:44table that we have created orders test
17:10:46it is still living here and still has
17:10:48like the data that we have created. So
17:10:50this is how things works with the
17:10:52temporary tables in
17:10:56SQL. Now let's see how the database
17:10:59server executed that temporary SQL. So
17:11:02now let's say that you are as a data
17:11:04analyst. You have created a query and
17:11:05then you say into in a temporary table.
17:11:07Now the database engine going to
17:11:09identify the query and first it's going
17:11:11to go and execute the query and then
17:11:13it's going to go and execute it and
17:11:15maybe we're going to get the data from
17:11:16the table orders and after the query is
17:11:19executed the database engine now has the
17:11:21results. Now two things can happen.
17:11:23First the database engine going to go
17:11:25and store the metadata informations in
17:11:28the system catalog. And now the second
17:11:30thing the database engine going to
17:11:31create a table but this time not in the
17:11:34users but in the temporary storage in
17:11:37the disk. So the table going to live
17:11:38there for a short time. And now what you
17:11:40can do you can write multiple SQL
17:11:42queries that are doing maybe multiple
17:11:44analysis on top of this table. So each
17:11:46time you select something the database
17:11:48engine has to go to the temporary
17:11:50storage and fetch the data from there.
17:11:53And now once you are finished and let's
17:11:54say you close your client the session
17:11:57between you and the database going to
17:11:58ends and now the database going to
17:12:00understand okay there is no more
17:12:02connection to this user and it going to
17:12:04go and clean up now the temporary
17:12:07storage with any tables that are created
17:12:10from this session. So that means the
17:12:12database is automatically cleaning up
17:12:14the storage maybe for other sessions. So
17:12:16this is how the database engine works
17:12:18with the temporary tables.
17:12:24So now the question is why do we need
17:12:26temporary tables? Let's see the
17:12:27following scenario. Now let's say that
17:12:29in our source database we have a table
17:12:31called orders and now we would like to
17:12:33go and load the table in our data
17:12:34warehouse. We have to do several
17:12:36transformations in order to prepare the
17:12:38data for the analyzes in the data
17:12:40warehouse. So maybe you have one query
17:12:41to remove the duplicates and another one
17:12:43to handle the nulls and maybe you are
17:12:45doing filtering and cleaning up and the
17:12:48last step you would like to aggregate
17:12:49the data. And now of course those
17:12:51queries those transformations want to
17:12:53change the content of the table orders
17:12:56and there is no scenario where you can
17:12:58do that directly on the source database
17:13:00and of course this is not allowed.
17:13:02That's why in data warehousing we have
17:13:04to go and get our own copy of the data
17:13:06and then on top of this data we can do
17:13:08our transformations. Now one way to do
17:13:10this using the temporary tables. So you
17:13:13have one script in order to extract the
17:13:15data from the table orders and put it in
17:13:17temporary table as an intermediate
17:13:19results and then you come with the
17:13:21transformations and all those queries
17:13:23and they start manipulating and changing
17:13:25the data of this extra copy in the
17:13:28temporary table and the last step you
17:13:30have the load where you go and load the
17:13:32final version of the intermediate
17:13:34results in the database. This is if you
17:13:36would like to do the whole ETL before
17:13:39inserting the data to the database. So
17:13:41now the orders table and the final table
17:13:43in the data warehouse both of them are
17:13:45tables. So they are permanent tables and
17:13:48they will stay there as long as we don't
17:13:50drop them. So they are very important
17:13:52tables. But now for the intermediate
17:13:54results it is not that important. It is
17:13:56just an intermediate step that we have
17:13:58done in order to have our extra copy of
17:14:01the data to manipulate it and so on in
17:14:03order to prepare it to be inserted in
17:14:05the data warehouse. So after we loaded
17:14:08it in the data warehouse, this copy of
17:14:10the data is not anymore important. It
17:14:12shouldn't stay like for a long time.
17:14:14That's why in this scenario, maybe we
17:14:16can go and use the temporary tables
17:14:18instead of normal tables for the
17:14:20intermediate results. And that's because
17:14:22only of one advantage is that the
17:14:24database going to go and do an automatic
17:14:27clean up after the host session ends. So
17:14:30it comes out of the box automatically
17:14:31from the database. So that means I don't
17:14:34have to deal with the dropping mechanism
17:14:36of this table for the next load. If
17:14:38there is like something wrong in the
17:14:40data warehouse, you would like always to
17:14:41check the copy where the transformations
17:14:44are done in order to debug and find
17:14:46issues. So I don't normally use
17:14:47temporary tables in these scenarios, I
17:14:49use just normal tables. But for other
17:14:52small projects, maybe this makes sense.
17:14:53So this is one use case on when to use
17:14:56the temporary tables in your projects.
17:14:58We use it in order to store intermediate
17:15:00results temporary until we are done with
17:15:03the session and then once we are done
17:15:04the database can go and drop that
17:15:07temporary
17:15:11table. All right guys, now a quick talk
17:15:13about the temporary tables. To be
17:15:15honest, I never use this in my projects.
17:15:17If I need an intermediate results in one
17:15:20query, I can go and use the CTEs. And if
17:15:23my intermediate results is very
17:15:25important then I put it in either view
17:15:27or CTIS but it is nice technique to
17:15:30learn maybe you can utilize it in one of
17:15:32your
17:15:36projects. All right guys so now let's
17:15:38have a quick summary about tables.
17:15:39Tables in database are like spreadsheet
17:15:42or grid that contains columns and rows
17:15:44and your actual data are stored in these
17:15:47tables. And we have learned there are
17:15:48two types of tables. We have permanent
17:15:50tables and temporary tables. Permanent
17:15:53tables lives in the database forever as
17:15:55long as you don't drop them. But in the
17:15:57other hand that temporary tables they
17:15:59have short lifetime. They will be
17:16:01dropped from the database once you end
17:16:03the session. Now we have learned as well
17:16:05there are two methods on how to create
17:16:07tables in databases. The first method is
17:16:10create insert. This method involves two
17:16:12steps. The first one is defining and
17:16:15creating the table and the second step
17:16:17is by inserting the data inside this new
17:16:20table. So you are creating something
17:16:21from the scratch. And the second method
17:16:23we call it CTAs. It create as well brand
17:16:27new table but based on the result of a
17:16:30query. So this type is done with only
17:16:32one step but it always needs another
17:16:35existing table. And we have learned as
17:16:36well the difference between tables and
17:16:38views where the main advantage of using
17:16:40tables created from CTIS is that to
17:16:43ensure the performance is fast enough at
17:16:46the end of the users or your reporting
17:16:48system. So we use CIS instead of views
17:16:52if the logic of the view is very complex
17:16:54and takes a lot of time to be executed
17:16:56in the database. And one more nice use
17:16:58case for the CIS is that we can go and
17:17:01persist a snapshot of the data in order
17:17:03to analyze a bug and data quality issue
17:17:06and to ensure that we have the exact
17:17:08data in order to find a solution for the
17:17:10bug and the issue. Now we have learned
17:17:12as well that we can use temporary tables
17:17:14in order to store intermediate results
17:17:16in a temporary storage and the main
17:17:18advantage of the temporary table is the
17:17:21database automatically drops all that
17:17:23temporary tables when the session ends
17:17:26and that's because for you the
17:17:28intermediate results are not that
17:17:29important to live long
17:17:34time. Hey my friends. So we have learned
17:17:37that in real data projects if you have a
17:17:40database there will be a lot of
17:17:42analytical use cases that want to access
17:17:44your data and do analytics. And what
17:17:46going to happen? They're going to write
17:17:48complex queries because in many
17:17:50scenarios they are doing complex
17:17:52analyzes. And if you don't do anything
17:17:54about it in your projects, you're going
17:17:55to face a lot of challenges like
17:17:57complexity and a lot of redundancy of
17:18:00the same complex logic but from multiple
17:18:02users and maybe performance and security
17:18:05issues. And we have learned we have five
17:18:07amazing techniques in order to solve
17:18:10those problems. We have learned the
17:18:11subqueries and cities and as well how to
17:18:14create objects like views, CTAs and
17:18:17temporary tables. So now what we're
17:18:19going to do, we're going to go and
17:18:20compare them side by side in order to
17:18:22have a big picture about the advantages
17:18:24and the disadvantages of each method. So
17:18:27let's go and compare them. Okay. So now
17:18:29we have our five methods and the first
17:18:31criteria that I would like to compare
17:18:32them is the storage type. We have
17:18:35learned that if you are using subqueries
17:18:37and CTE, what can happen? and the
17:18:39database going to put the result of
17:18:41those two techniques in the memory in
17:18:43the cache so that later the main query
17:18:46has a fast access to those intermediate
17:18:48results. But in the other hand if you
17:18:50are using temporary tables or tables
17:18:52from CDS the new created table can be
17:18:55stored inside the disk storage. And now
17:18:57for the views as we understood there
17:18:59will be no data storage and that means
17:19:02we are not using any storage from the
17:19:04database. Now if you are talking about
17:19:06the lifetime so that means how long the
17:19:08object going to live or persist in the
17:19:10database. Now our three techniques sub
17:19:12queries CTE and temporary tables all of
17:19:15them going to live a short time in the
17:19:17database. So all of them are temporary.
17:19:20But now if you are talking about
17:19:21creating objects using CIS and views
17:19:24those two going to be permanent. So that
17:19:26means they're going to live in the
17:19:27database as long as you don't drop them.
17:19:30Now we're going to compare them with
17:19:31something similar is when the database
17:19:34going to go and drop or delete those
17:19:36objects. Now we have learned that the
17:19:38subqueries and the cities have a short
17:19:40time. They going to live only during the
17:19:43execution of the query. So once the
17:19:45query ends the database going to go to
17:19:47the cache and delete everything. But for
17:19:50the temporary tables they live little
17:19:52bit longer as long as you are in the
17:19:54session. But once you end the session,
17:19:56the database as well going to go and
17:19:58drop and delete your table. Now for the
17:20:00objects that comes from the CIS and
17:20:03views as we learned they are persistent
17:20:05and permanent and the database can only
17:20:07delete them if you ask the database to
17:20:09do that by using the DDL command drop.
17:20:12So the database will not delete anything
17:20:14for these two. So now the next one is
17:20:16the query scope like how we can access
17:20:19those objects. Now for the subquery and
17:20:21the CTE the scope is here very small. It
17:20:24is accessed only from one single query.
17:20:26The query itself where you write the
17:20:28city and subquery. So you cannot access
17:20:31it from external queries. But we have
17:20:33learned that the temporary tables cis
17:20:35and views you can access all those
17:20:38objects from multiple queries. So that
17:20:40means you can access those objects from
17:20:43multiple external queries. Now the next
17:20:45one if you are thinking about the
17:20:47reusability if you look to the
17:20:49subqueries they are very limited. the
17:20:51subquery going to be used only in one
17:20:53query and only in one place. So if you
17:20:56need it in multiple places, you have to
17:20:58go and repeat the same logic. So
17:21:00subqueries are the worst with their
17:21:02reusability. But now if you are talking
17:21:04about the CTE, it is little bit better.
17:21:06You still can access it only from one
17:21:08single query but you can access it in
17:21:10the same query from multiple places. So
17:21:13you can access it multiple times from
17:21:15different joins and you don't have to
17:21:17repeat the same logics over and over.
17:21:19But still it is limited because you have
17:21:21only one query that is using the logic.
17:21:24Now if you think about the temporary
17:21:25tables I could say the reusability here
17:21:28is medium and that's because you can
17:21:30access the data by multiple queries but
17:21:33only during this session. So once the
17:21:35session is ended you cannot access it
17:21:38anymore which means you have to recreate
17:21:40it in order to reuse it again. So it is
17:21:42more reusable than the city and the
17:21:44subqueries but not that good like the
17:21:46CTAs and views. Those techniques can
17:21:49offer the highest reusability for you.
17:21:52So they are always there for multiple
17:21:54users from multiple queries. So it can
17:21:57eliminate a lot of redundancies and you
17:21:59have to do the job only once. Now moving
17:22:01into the next one. If you are thinking
17:22:03about the intermediate result of those
17:22:04techniques, the question is how fresh is
17:22:07the data? Is the data from these objects
17:22:10always up to date? Now for the
17:22:11subqueries and the cities they are
17:22:14always up to date because the SQL is
17:22:16executing the logic on the fly and
17:22:18storing the data in the memory and
17:22:20immediately after that going to come the
17:22:22main query and get the data. So always
17:22:24the intermediate results in the memory
17:22:26are up to date. But now if you think
17:22:28about that temporary tables and the CTIS
17:22:30the query is only executed once and if
17:22:33there is like any update and changes on
17:22:35the original table you will not find
17:22:38those changes in those objects and
17:22:40that's because SQL executed once and
17:22:42that's all. So if you query those tables
17:22:44there is no guarantee that the data are
17:22:46up to date. So if you want fresh data
17:22:49you have always to drop the table and
17:22:51create it again from the query. Now if
17:22:53you are talking about the views they are
17:22:55amazing they are always up to date
17:22:57because views does not store any data.
17:22:59So each time you ask the views for data
17:23:02what's going to happen the database
17:23:03going to go to the original table and
17:23:05fetch the data to the view. So your data
17:23:07are always fresh and up to date. So this
17:23:09is a big picture about the behavior of
17:23:11those advanced techniques that you can
17:23:13use in SQL projects. And if you ask my
17:23:16opinion my favorite is going to be the
17:23:17views in the first place. Then in the
17:23:20second in my list is the city. They are
17:23:22amazing, but don't use more than five
17:23:24CTEs in one query. Otherwise, it's going
17:23:26to be really annoying and hard to read.
17:23:28And then I'm going to say in the third
17:23:29place, the sub queries. And then the
17:23:32CDIS. I use CIS if the views are slow.
17:23:35If that's a scenario, I'm jump to the
17:23:37CDIS and create a permanent physical
17:23:39tables from my query. And the last one
17:23:41that I rarely use is the temporary
17:23:43tables. So, this is how I rank those
17:23:46techniques in my skill projects.
17:23:52Now I would like to show you as well a
17:23:53big picture on how things works in my
17:23:55projects in order to see all those
17:23:57different techniques and possibilities
17:23:59that you can use. It's like a big
17:24:01picture and recap. So story time. So you
17:24:04have a database and things starts where
17:24:06you have a database administrator or
17:24:08let's say a data engineer that is
17:24:10creating a new table from the scratch.
17:24:12So he going to write a DDL statement in
17:24:14order to create one physical table at
17:24:16our database. And now our database table
17:24:18is empty. That's why in the second step
17:24:21he going to go and write an insert
17:24:23statement in order to fill our new table
17:24:25with data. Now once we have a table
17:24:27we're going to give the access maybe to
17:24:29a data scientist or data analyst in
17:24:31order to start writing SQL queries. So
17:24:33now the first thing that could happen
17:24:35that the logic is complex and she has to
17:24:37do that in two steps. So the first step
17:24:40is a query that prepares the data in
17:24:42order to execute the second step. So
17:24:45that's why she going to go and use the
17:24:46subquery and the main query going to go
17:24:48and retrieve the data from the
17:24:50intermediate results in order to prepare
17:24:52the final results for the analyst. Now
17:24:54what could happen is that there will be
17:24:56an SQL logic in the query where it keep
17:24:59repeating the scripts. So now instead of
17:25:01writing another subquery for that she
17:25:03going to go and put this logic in CTE
17:25:05and now she going to go to the main
17:25:07query and use the result of the CTE in
17:25:10multiple places in the same query. So
17:25:12all those stuff the sub queries and the
17:25:14city queries the main queries all those
17:25:16stuff happens in one single query and
17:25:19now what could happen is that she is
17:25:21writing an amazing code. So instead of
17:25:23using it only in her query what's going
17:25:26to happen she going to go and persist
17:25:28this logic in the database. So she going
17:25:30to put it as a view in the database so
17:25:32that all other users and analysts can
17:25:35benefit from this logic and they don't
17:25:37have to write it again. So instead
17:25:39they're going to go and query the view
17:25:41and this going to makes the life easier.
17:25:43And of course our data analyst can as
17:25:45well use this view in the main query.
17:25:47And now one more thing she has as well
17:25:49another logic that is really complex and
17:25:52as well everyone can benefit from it.
17:25:54But the issue this query is very slow.
17:25:56So now she has to decide do I put it in
17:25:59view or do I create a new table based on
17:26:01the query using CTAs. Now of course
17:26:03because of the performance and the view
17:26:05takes around 30 minutes to be executed.
17:26:07She decided to execute the query using
17:26:09the CTIS where she generate a physical
17:26:12table so that all other analysts as well
17:26:14can access this new table in order to
17:26:17reuse the results and of course she can
17:26:19use it in her main query and with that
17:26:22now you have experience how things works
17:26:24in real projects. It is not simple
17:26:26select query from table it is like this
17:26:29people are creating subquery CTE views
17:26:32temporary tables CTAs for different
17:26:35purposes. All right my friends. So
17:26:37that's all about the CTIS and the
17:26:39temporary tables. And with that we have
17:26:40learned all the techniques on how to
17:26:42organize our complex projects. Now next
17:26:44we're going to start talking about
17:26:45something completely different. We're
17:26:47going to talk about the stored
17:26:49procedures on how to put our code inside
17:26:51the database. This is all about that
17:26:53programmability and how to add stuff
17:26:55like parameters, variables, error
17:26:58handling. So it's like programming. So
17:27:00let's go. So let's uncover this word of
17:27:02the s procedures and let's go.
17:27:08Now think about store procedures like
17:27:10this. Every time you go to a coffee
17:27:12shop, you say, "I would like a large
17:27:14coffee with a coconut milk, no sugar,
17:27:17and extra whipped cream." And you repeat
17:27:19this over and over each time you go to
17:27:22this coffee shop. And now, if you are
17:27:24working with stored procedures, it's
17:27:25going to be like this. Whenever you go
17:27:27to the coffee shop, you just say, "Give
17:27:29me my usual." and the barista know
17:27:31exactly what you mean behind that and
17:27:33you will get exactly your order without
17:27:35specifying and repeating everything word
17:27:37by word and this is exactly what's going
17:27:39to happen if you work with stored
17:27:40procedures so let's have some coffee
17:27:44right all right so now we can continue
17:27:47all right so now let's start again from
17:27:48the scratch we have always these two
17:27:50sides we have the client side and the
17:27:52server side of the database and what we
17:27:54have learned we have like a database and
17:27:55you as a user you can go and create like
17:27:58different SQL statements Like for
17:28:00example, you can create like an SQL
17:28:02select statements in order to retrieve
17:28:04data from the database or another SQL
17:28:07statements where you are inserting data
17:28:09to the database and another one let's
17:28:11say that you are updating the content of
17:28:13your tables and so on. So you have like
17:28:16different statements in order to
17:28:17interact with the database. Now let's
17:28:19say that what you are doing is not only
17:28:21one time job you are keep repeating
17:28:23those steps over and over. So you are
17:28:26always like doing an insert then an
17:28:28update and then a select and you keep
17:28:30repeating that day after day. So now
17:28:33imagine that you are doing something
17:28:34crazy where you go in vacation but the
17:28:37job should be done. So what you do you
17:28:38hand over all those select statements to
17:28:41your colleagues and they have to do it
17:28:43every day as well as you are gone. So
17:28:45you go and give them all those SQL
17:28:47scripts and you tell them okay you have
17:28:49to execute the first query then the
17:28:51second query and then the third query.
17:28:53This is of course not a good way on how
17:28:55to do things because of course there
17:28:57will be some human errors where like the
17:29:00execution of the script is not correct
17:29:02like first updating then inserting and
17:29:04things can go wrong and that's exactly
17:29:07why we have stored procedures in SQL. So
17:29:10what we can do we can put all those SQL
17:29:13statements together in one frame in one
17:29:16program and we call it start procedure.
17:29:19And now once you do that all your SQL
17:29:21statements will not stay at the client
17:29:23side they will be stored now in the
17:29:26server side of the database. So that
17:29:28means in store procedures we are storing
17:29:30our SQL statements inside the database.
17:29:33So you don't have to go and hand over
17:29:35your SQL statements to your colleagues.
17:29:37And now all what you have to do in order
17:29:39to interact with your SQL statements is
17:29:42to go and execute the store procedure.
17:29:44So you write very simple command called
17:29:47execute SP for example. So with that you
17:29:50are calling your stored procedure that
17:29:52is stored inside the server. And once
17:29:54you execute this what can happen the
17:29:56database going to go to the stored
17:29:57procedure and start executing all the
17:30:00SQL statements that you have inside the
17:30:03store procedure and it's going to do it
17:30:05exactly in the order that you have
17:30:06defined. So from top to bottom. So now
17:30:09once the database went through all your
17:30:11SQL statements, it's going to return
17:30:13back to the user the data that we have
17:30:15from the selects. And with that things
17:30:17are really easy and you can tell your
17:30:19colleagues okay just execute this third
17:30:21procedure and the rest can be done from
17:30:24the database. So with that you minimize
17:30:26the human errors and you make sure that
17:30:28everything can be executed as you wish
17:30:30and as well as you are back from your
17:30:32vacation things are easier. You have to
17:30:34just go and execute the third procedure.
17:30:36So this is what we mean with start
17:30:38procedure. You can store inside it
17:30:40multiple SQL statements in specific
17:30:42order and you can save it inside the
17:30:44database and each time you need your SQL
17:30:46statements you can go and simply execute
17:30:52them. So now let's have a quick
17:30:54comparison between a normal query normal
17:30:56SQL statements compared to a stored
17:30:58procedure. So a normal SQL query you
17:31:01have like select from where and so on.
17:31:04This is like one-time transaction. You
17:31:06are asking the database for one thing
17:31:08and the database is answering. So it is
17:31:11like one-time request. But now in the
17:31:13other hand in the stored procedures you
17:31:15have multiple SQL statements and once
17:31:17you execute the stored procedure there
17:31:19will be many interactions with the
17:31:21database in one go. So that means you
17:31:23will have multiple transactions that is
17:31:25happening in your store procedure. So an
17:31:28SQL query it is like a simple request.
17:31:30You need one thing and you are getting
17:31:32it. But on the other hand in the start
17:31:34procedure it is like a program. As you
17:31:37are writing a code in any programming
17:31:39languages it is more than one request it
17:31:41has a lot of stuff like for example you
17:31:44can go and build looping logic where we
17:31:46go and iterate through something or you
17:31:49can go and build a control flow where
17:31:51you have a logic like the FL statements.
17:31:54So there are like different paths in
17:31:56your code and as well in programming we
17:31:58have like parameters and variables in
17:32:01order to make our code dynamic and
17:32:03flexible and as well we can build error
17:32:05handling on our code in order to
17:32:08customize what can happen if there is
17:32:09like an issue. So the store procedure it
17:32:12is like having a code like for example
17:32:14in Python. So that means you can do more
17:32:16complicated stuff compared to a simple
17:32:19query where you have only like one
17:32:21request. So in the stored procedures you
17:32:23are doing like programming and coding
17:32:25and it is more advanced than only just
17:32:28having a query. So that means if you are
17:32:30working with stored procedures things
17:32:32going to get more complicated and
17:32:33advanced but of course you will get a
17:32:35lot of flexibility and reusability
17:32:38compared to a simple
17:32:39[Music]
17:32:42query. So now there is like another
17:32:45alternative to stored procedures. Well,
17:32:47you can go and put all your SQL
17:32:49statements in a Python code and things
17:32:52can work as well. So, either you put
17:32:53your SQL statements inside the stored
17:32:56procedure or in a Python code. But now
17:32:58the big question is what are the
17:33:00differences between them? Well, there is
17:33:01like a disadvantage if you having Python
17:33:03in different server because you have to
17:33:05go and build a connection between your
17:33:08server and the database server and
17:33:10connection means always networking and
17:33:12you might get slightly worse
17:33:14performance. So this is one advantage
17:33:16for the start procedure. Another
17:33:18advantage for the search procedure that
17:33:20all the scripts that you're going to
17:33:21store inside the store procedure in the
17:33:23database going to be pre-ompiled. So
17:33:26pre-ompiled means the SQL database
17:33:28servers knows already about your SQL
17:33:31statements and there was already a check
17:33:33whether all the syntaxes are correct and
17:33:35the database as well going to be
17:33:37preparing everything to execute the
17:33:39stored procedure like maybe preparing
17:33:41the execution plans and a lot of stuff.
17:33:43So if you store your skill statements
17:33:45inside store procedure in the database,
17:33:47it is very close to the database and the
17:33:49database knows everything about your
17:33:51scripts and it is ready to execute it.
17:33:53But if you put all your SQL statements
17:33:55outside of the database, of course, the
17:33:57database has no chance to understand
17:34:00what is coming. So it cannot go and
17:34:02compile anything until Python sends the
17:34:05code to database. So this is another
17:34:07advantage for the stored procedure. But
17:34:09now if you build your SQL statements in
17:34:12Python, you will get a lot of
17:34:13advantages. Like for example, you can go
17:34:15and build very flexible Python codes
17:34:18where you can use Python features
17:34:20together with the SQL and with that you
17:34:22open the door of many possibilities and
17:34:24flexibility. Another thing with Python,
17:34:26you can make great version control. So
17:34:29everything is integrated in Python
17:34:31tools. And one more advantage is that if
17:34:33you have a complex requirement in your
17:34:35projects, it's going to be really hard
17:34:36to implement it in stored procedures.
17:34:38it's going to cost you a lot of lines of
17:34:40code and things going to be not
17:34:42comfortable. But if you are implementing
17:34:44a complex logic in Python, things going
17:34:47to be way easier. So with Python, you
17:34:49can implement complex logics very easily
17:34:52compared to the stored procedure. So
17:34:54those are the big differences between
17:34:55the stored procedure and Python. Now I
17:34:57have to be honest with you about having
17:34:59your code in store procedure or in
17:35:01Python. Well, if you are working
17:35:03together in a data project, I will never
17:35:05recommend you to use stored procedure if
17:35:07you have the possibility to have your
17:35:09code in Python. And that's because I saw
17:35:11a lot of projects using stored procedure
17:35:13and most of them ends in chaos. It is
17:35:16really hard to debug. It is really hard
17:35:18to test. It's like catastrophic. So
17:35:20really don't use in your projects any
17:35:23store procedures. Especially if you have
17:35:25like a big project and you have a lot of
17:35:27data and tables and so on. You can
17:35:29manage everything perfectly using
17:35:31Python. Especially if you have platform
17:35:33like data bricks or snowflakes then of
17:35:36course the best way to control your data
17:35:38projects is using Python. But of course
17:35:40if you don't have this possibility and
17:35:42you have only a database server and you
17:35:44can only work with this then you don't
17:35:46have any other option. You have to work
17:35:47with the store procedures. But if you
17:35:49have this possibility to put your
17:35:51project inside Python and to run your
17:35:53scripts from there, then it is way
17:35:55better than having stored procedure.
17:35:56Well, this is my opinion. I'm just
17:35:58talking about working in projects in big
17:36:00projects. But if you have like small
17:36:02projects, few tables and so on, then
17:36:04it's fine to stay with the store
17:36:05procedure. But never build a big project
17:36:07using stored procedures because I tell
17:36:10you it will never work. So try to always
17:36:13to think about to have the right
17:36:14platform in order to run your projects.
17:36:17And now I'm thinking about it. Maybe I
17:36:19should have put this tip at the end of
17:36:20the video, not in the middle. So
17:36:22whatever. If you still want to learn
17:36:24store procedures, we're going to
17:36:25continue on that. And I'm going to have
17:36:27like a really nice example about how to
17:36:29build store procedures step by step like
17:36:31having a mini projects. So why not
17:36:33learning both of them. So let's
17:36:38go. Okay. So now let's have a quick look
17:36:40to the syntax of the store procedure. It
17:36:42is very simple. So it has always two
17:36:44parts. First we have to define the start
17:36:46procedure. So we can do it like this.
17:36:48Create procedure. Then we have to define
17:36:50the procedure name and then we say as
17:36:52and then we have begin and end. It's
17:36:54very important for SQL to understand
17:36:56when that definition starts and when it
17:36:58ends. And then between the begin and end
17:37:00we're going to have a set of SQL
17:37:02statements. So here you can insert
17:37:04whatever you want. Insert update queries
17:37:06anything. And once you have defined the
17:37:08sort procedure the next step is that
17:37:10we're going to go and execute it. So the
17:37:12syntax is very simple. We're going to
17:37:13say execute and then the procedure name.
17:37:15So that's it with that SSQL going to go
17:37:17to the S procedure and start executing
17:37:19all the SQL statements that you have in
17:37:21the definition. So this is the syntax of
17:37:23the S procedure. As I said it is very
17:37:25simple. All right guys. So now let's do
17:37:27it step by step. The first step is that
17:37:29we're going to go and write a query. So
17:37:31let's say that we have a very simple
17:37:32task and it says for US customers find
17:37:35the total number of customers and the
17:37:38average score. So let's go and do it.
17:37:40It's very simple. So select count star
17:37:44total customers and then the average of
17:37:47scores as average score from our table
17:37:52sales customers and then since it says
17:37:55US customers we have to go and filter
17:37:57the data based on the column country is
17:38:00equal to USA. So that's it. This is our
17:38:03query. Let's go and execute it. So we
17:38:05have a very quick nice report about the
17:38:07total number of customers and the
17:38:08average score. So now let's say that I
17:38:10have a weekly meeting and I have to
17:38:12represent this reports over and over. So
17:38:15that means I have to go and execute this
17:38:17query like frequently in weekly basis in
17:38:20order to get the data for the reports.
17:38:22So now what this means I have to go and
17:38:24save this query in order to use it later
17:38:27that each time I have to rewrite it. So
17:38:29that means I have to store this text
17:38:30somewhere that I don't go and rewrite
17:38:33the query over and over. So what I
17:38:35usually do, let's go and we copy the
17:38:37whole query and then we create a new
17:38:40text and let's say it's going to be my
17:38:41weekly query and it's going to be SQL.
17:38:45So I'm going to go and edit it and here
17:38:48I'm going to save my query and each time
17:38:50I need this query I have to go and copy
17:38:52it, go back to my SQL and then I'm going
17:38:55to go and paste it in order to execute
17:38:58it. So either going to write it each
17:38:59time or copy and paste it. Well, we
17:39:02don't have to do that. we have start
17:39:04procedures. So that means we're going to
17:39:06go to the step two where we're going to
17:39:07turn this query into a store procedure.
17:39:10So let's do that. It's very simple. So
17:39:12we're going to say create procedure. And
17:39:15now we have to go and give it a name. So
17:39:16it's going to be get customer summary.
17:39:19And then after that we're going to say
17:39:21as and then we need the begin and end.
17:39:23And in between we're going to put our
17:39:25query. So let's go and copy our query
17:39:28and just put it in between. So that's
17:39:31it. Let's go and execute it. And with
17:39:33that we have created our store
17:39:35procedure. And now in order to see our
17:39:37store procedure we can go to the object
17:39:39explorer to our database sales DB. And
17:39:41then here we have a folder called
17:39:43programmability. So let's go inside it.
17:39:45And here we have a lot of stuff like
17:39:47functions, triggers and we have stored
17:39:49procedures. So let's go inside it. And
17:39:51we can see over here this is our new
17:39:53created stored procedure. So we are
17:39:56almost there. The next step is that
17:39:58we're going to go and call our store
17:39:59procedure. And this is the easiest part.
17:40:02So it's going to be execute the stored
17:40:04procedure. And the syntax is very
17:40:05simple. So execute and then the name of
17:40:08the stored procedure. So get customer
17:40:11summary. So let's go and execute it. And
17:40:14with that as you can see we get the
17:40:16result of our query. So as you can see
17:40:18it is very simple. In just few steps we
17:40:21created a store procedure. And then in
17:40:23the future you don't need the whole
17:40:24thing. You just go and execute the store
17:40:27procedure. I don't have to store the
17:40:29query locally at my PC or to copy and
17:40:32paste anything. If I want this report
17:40:35now, I just have to execute the store
17:40:37procedure like this and I will get the
17:40:43results. Okay. So now let's keep moving.
17:40:45Now we're going to talk about the
17:40:47parameters inside stored procedures. So
17:40:49what is a parameter? It is like a
17:40:52placeholder where you can pass in
17:40:53information from you into the store
17:40:56procedure while running it and using
17:40:58parameters in store procedure it's going
17:41:00to make it flexible reusable and
17:41:03dynamic. So let's understand what this
17:41:05means. Let's say that you got a new
17:41:07task. So it says for German customers
17:41:10find the total number of customers and
17:41:12the average score. So that means now we
17:41:14have like to generate two reports one
17:41:16for USA and one for Germany. And in both
17:41:18of them you are doing the same
17:41:19aggregation. And again we have to go and
17:41:21start writing the query. It's going to
17:41:23be very similar to the one that we have
17:41:24in the previous example. So we are doing
17:41:26the same stuff same aggregations but the
17:41:29only change here is that we're going to
17:41:31use another value to filter the data. So
17:41:33instead of USA we're going to go and say
17:41:35here Germany. So let's go and execute
17:41:38this one over here. And with that we can
17:41:39see we have total number of customers
17:41:41too. So this is the report that we have
17:41:43to provide like in weekly basis. And
17:41:45again in order not to go and copy paste
17:41:47stuff we're going to go and create a
17:41:49store procedure for that. At the end
17:41:51we're going to have an end. But now of
17:41:53course we cannot have like the same
17:41:54names we're going to go and say here
17:41:56Germany. So let's go and execute it. And
17:42:00the next step we have to go and execute
17:42:02the store procedure. So like this. Let's
17:42:06go and execute it. And the whole logic
17:42:08now stored inside the database. Let's go
17:42:10and refresh on the explorer over here.
17:42:12And you can see now we have two stored
17:42:14procedures. But now you have to feel
17:42:16there is something wrong. Always in
17:42:18programming and coding. If you find
17:42:20yourself repeating the same task over
17:42:22and over then there is always a smarter
17:42:25way on how to optimize that. Repeating
17:42:28stuff in coding is always bad thing. So
17:42:30now clearly we are repeating the same
17:42:32query in two different store procedure.
17:42:34And now if you compare them you see it's
17:42:36because of the value. So we have here
17:42:38the value for the filter once Germany
17:42:40and one USA. And those values are static
17:42:43values. So it's always going to stay
17:42:45inside the store procedure as USA. But
17:42:47instead of that we can replace those
17:42:49static values with a parameter. And then
17:42:52you decide as you are executing the
17:42:54stored procedure for which country you
17:42:57want to execute the store procedure. So
17:42:59let's go and do that. I'm just going to
17:43:01remove everything from here and focus
17:43:03only on the first store procedure. Now
17:43:05what we're going to do after giving the
17:43:07name of our store procedure we have to
17:43:10define our parameter. So it start with
17:43:13at and with that SQL understandhuh now
17:43:15we are talking about parameters and we
17:43:17need now the name of the parameter. So
17:43:19it's going to be country. It could be
17:43:21any name that you want and after that we
17:43:23have to define for SQL the data type.
17:43:25It's like when you are creating a table
17:43:27and you define columns you assign a data
17:43:29type for each column. The same thing
17:43:31here you have to assign as well a data
17:43:33type for each parameter. So we're going
17:43:35to use the data type in var and for the
17:43:38countries it's enough to have the length
17:43:39of 50. So with that we are telling SQL
17:43:42for this third procedure we can pass an
17:43:45information to the store procedure and
17:43:47this information and value going to be
17:43:49used inside this parameter. So now after
17:43:51we defined this parameter over here we
17:43:54can go and use it anywhere inside our
17:43:56query. And of course we want to go and
17:43:58use it instead of this static value. So
17:44:00now we're going to remove this static
17:44:02value and instead we're going to have
17:44:04the parameter. So now we are saying
17:44:06you're going to filter the table based
17:44:08on the value that comes from the user
17:44:10and not anymore static with a USA. And
17:44:12as I said you can use this parameter
17:44:15everywhere like even here in the select
17:44:16statements. So it is a value that could
17:44:18be used everywhere in your query. So
17:44:20that's it. We have defined our new
17:44:22parameter and we have used this
17:44:25parameter in our query. So now we have
17:44:26to go and update the store procedure. We
17:44:29cannot leave it as create. Instead of
17:44:31that, we're going to say alter. So we
17:44:32are saying alter procedure and with the
17:44:35new informations. Let's go and execute
17:44:37it. And now we have to go and execute
17:44:40it. So now what we're going to do, we're
17:44:41going to say execute get customer
17:44:43summary. But now our store procedure is
17:44:45expecting a value from you from the
17:44:47input. So we're going to do it exactly
17:44:48like we done in the name over here. So
17:44:51we're going to say the parameter country
17:44:53is equal
17:44:54to Germany. So that means the value of
17:44:58this parameter come from me come from
17:45:00the input and this information going to
17:45:02be passed to my query to the store
17:45:04procedure. So let's go and execute it.
17:45:06And with that as you can see we are
17:45:08getting the report of customers for
17:45:10Germany. And now if you say okay let's
17:45:12go and generate the report for USA. All
17:45:14what you have to do is replace the
17:45:17parameter. So in the value instead of
17:45:19Germany we're going to say USA. So let's
17:45:22go and execute it. Great. Now we are
17:45:24getting as well the report for us
17:45:26customers. So that seems my friends for
17:45:29those two reports I just need one store
17:45:31procedure and with the help of the
17:45:32parameter I made my store procedure now
17:45:35more flexible and professional. So this
17:45:37is exactly the power of the parameters
17:45:40it makes everything reusable and
17:45:43dynamic. And now of course we don't need
17:45:45the store procedure for Germany. So what
17:45:47we can do we can go and drop it. So
17:45:49we're going to say drop procedure and it
17:45:53was like this Germany. So we don't need
17:45:56this store procedure and we're going to
17:45:58stay with only one dynamic store
17:46:00procedure. So this is how to use
17:46:02parameters in store procedure and why
17:46:04it's important. Okay. So now to the next
17:46:06step is that we can go and add default
17:46:08values for the parameters. So let's say
17:46:11that I execute very frequently this
17:46:13report where I say the country equal to
17:46:14USA and I don't want each time to define
17:46:17the parameter value equal to USA. So if
17:46:20you are using a value very frequently
17:46:22you can add it as a default inside the
17:46:24definition of the store procedure and it
17:46:26is very simple. So if you go to the
17:46:28definition again over here after the
17:46:30parameter and you say equal to USA. So
17:46:34now it's very important to understand
17:46:35that the country will not be always
17:46:38equal to USA. It is just you are saying
17:46:40if I don't get from the user any value
17:46:43then as a default I'm going to go and
17:46:45use the USA. So let's go and again
17:46:48change the definition of our stored
17:46:50procedure using alter. So execute and
17:46:52now we can go to our store procedure and
17:46:55I can skip the whole thing over here and
17:46:57execute it. So now as a default I'm
17:46:59getting the report of USA without
17:47:02passing an information to the store
17:47:03procedure because I know it is as a
17:47:05default USA. But if you need it as a
17:47:08Germany of course you have to go and
17:47:10define it. So you say execute the store
17:47:12procedure where the country equal to
17:47:14Germany. So if you execute it like this
17:47:16SQL still going to use your value. So
17:47:18the value that comes as an input from
17:47:20the user has more priority of course as
17:47:22the defaults. And with that we are
17:47:24getting the Germany reports. So as you
17:47:26can see it's really nice right using
17:47:27parameters in store
17:47:32procedure. All right moving on to the
17:47:34next step. Now we can work with multiple
17:47:36queries inside one stored procedure. And
17:47:39this is what we have learned at the
17:47:40start. We can have multiple SQL
17:47:42statements in one stored procedure. And
17:47:44now we have a new report and query to
17:47:46generate. It says find the total number
17:47:49of orders and the total sales. So let's
17:47:51do it quickly. We can write it like
17:47:53this.
17:47:54Select counts order ID. This is the
17:47:58total orders and then the sum of sales.
17:48:03Total sales from our table sales orders.
17:48:07And of course we are always creating a
17:48:09report based on specific country. So
17:48:11that means we have to go and join it
17:48:12with the customers table in order to
17:48:15filter the data. So on customer ID equal
17:48:19to the customer id. And now we're going
17:48:22to go and filter the data. So country
17:48:25equal to USA. So something like this.
17:48:28Let's go and execute it. And with that
17:48:31for the US customers, we have six orders
17:48:33and the total sales 180. And of course,
17:48:36the same thing we're going to do for
17:48:37Germany. So now, of course, we will not
17:48:39go and create an extra store procedure.
17:48:41For this, we're going to go and put
17:48:42everything in one store procedure. So
17:48:44let's go and copy the whole thing and
17:48:47put it here inside. So after the first
17:48:50report we're going to have the second
17:48:51report and now the best practice here if
17:48:53you have multiple queries in store
17:48:55procedure go and add at the end of each
17:48:58query a semicolon. It is just easier to
17:49:00understand how now this is the end of
17:49:02this query especially if you have like a
17:49:04big complex queries where you have CTE
17:49:07union and so on. It's going to be really
17:49:09hard to understand that we are talking
17:49:11now about completely new query but it is
17:49:13not like something the database requires
17:49:15it but it's just easier to read. So just
17:49:18add semicolons at the end of each query.
17:49:21So now let's go and execute the whole
17:49:23thing in order to change the definition
17:49:25of our query. And one more thing of
17:49:27course don't forget we don't need static
17:49:29values over here. We're going to go and
17:49:30add our nice
17:49:32parameters. So add country. So I think
17:49:36with that we have everything is ready to
17:49:38be executed. So let's go and change the
17:49:40definition of our store procedure. And
17:49:42now let's go and start with the defaults
17:49:44where the country equal to USA. So let's
17:49:47go and execute it. And now in the output
17:49:49as you can see we have two results. And
17:49:51that's because we have two queries. So
17:49:53the first report is for the first query
17:49:55and the second one for the new one that
17:49:57we just created. And the same thing if
17:49:59you go and execute the store procedure
17:50:01for Germany we will get as well two
17:50:03results. And here we can see we have
17:50:04four orders and 200 of total sales for
17:50:07Germany. So as you can see it's very
17:50:09simple. You can go now and add multiple
17:50:11SQL statements not only queries you can
17:50:14go and update you can do an insert
17:50:16delete any kind of SQL statements you
17:50:19can just go and add it inside your
17:50:21program. And as usual SQL going to
17:50:23execute it from the top to the bottom.
17:50:25So since this is the first SQL statement
17:50:27it's going to execute it first and then
17:50:29after that it's going to go to the next
17:50:31one. So this is how you can add multiple
17:50:33SQL statements to your store
17:50:39procedure. All right everyone. So now
17:50:40we're going to talk about the variables.
17:50:42So what is a variable? It is like a
17:50:44placeholder where you store inside it a
17:50:46value in order to use it later inside
17:50:49your stored procedure. So that means
17:50:51variable holds like a value inside the
17:50:53memory and you can reuse it everywhere
17:50:55you want inside your stored procedure
17:50:57but it's not like the parameters.
17:50:59Parameters are something like outside
17:51:01the store procedure. It's an input from
17:51:03the one that is executing the store
17:51:05procedure and the store procedure has to
17:51:07adapt with the parameter. But a variable
17:51:10it's something that lives inside the
17:51:12store procedure and we use it as a
17:51:14developers in order to make our code
17:51:15dynamic and to move a value from one
17:51:17place to another. So let's have a very
17:51:19simple example now. Let's say that we
17:51:21don't want our report here about the
17:51:23total customers as a query. So I don't
17:51:25want it as a result in the output. Let's
17:51:27say I'm generating a report always like
17:51:29this. We are saying the total customers
17:51:31from Germany equal to two and the
17:51:33average score from Germany is equal to
17:51:35425. So I need it as a text not as a
17:51:38table like here. So in order to do that
17:51:40we can use the TSQL print in order to
17:51:43give a message after executing the store
17:51:45procedure. So the syntax of print is
17:51:47very simple. So we can go over here and
17:51:50say print and then we have single quotes
17:51:53and let's go and get the whole message
17:51:54from here without the comments and then
17:51:57the semicolon and we can repeat that for
17:52:01the second message. So for the average
17:52:04score and we put it over here as well a
17:52:07semicolon. Now if you do it like this
17:52:10this message going to be always static.
17:52:12So we will have always like two for the
17:52:14total customers and the average score
17:52:16going to always be like this even though
17:52:18that the data is changing. So we cannot
17:52:20have it static like this. We have to
17:52:22make it dynamic and especially if we are
17:52:24calling this function for USA. So we
17:52:26cannot have it here as a Germany. So
17:52:28let's see how we can make this dynamic.
17:52:30Now let's start with the easy stuff.
17:52:32Instead of the Germany over here we can
17:52:34go and put our parameter right. So
17:52:36instead of this so we're going to say at
17:52:39country but now the problem is it is
17:52:41part of the whole string we cannot do
17:52:43that so we're going to stop the text and
17:52:46you can see the coloring is changing and
17:52:47then have a plus in order to have
17:52:49concatenations. So this text comes first
17:52:52then the value from the country and then
17:52:55we're going to have as well the double
17:52:56point as a static text and again a
17:52:59concatenation and then we have the two
17:53:00we can talk about later. So let's do the
17:53:02same stuff over here. So we're going to
17:53:04say plus add country caring is not
17:53:07changing because of this code. So let me
17:53:09just remove it and then afterward plus
17:53:12make it static again plus and remove the
17:53:15final quotes. So with that in the
17:53:17message we have now dynamic where we get
17:53:19the value of the country from the
17:53:21parameter. And now we come to the
17:53:23interesting part. We have here an issue
17:53:25those two values they come from this
17:53:27query. And of course we cannot use a
17:53:29parameter for that. We have to use now
17:53:31the variables. Now in order to make a
17:53:33variables we have three steps. The first
17:53:35step is that we have to tell SQL about
17:53:38our new variable. So SQL can prepare and
17:53:40make like placeholder for it in the
17:53:42memory. So we have to tell and prepare
17:53:44it with our new variables. Now usually
17:53:47we do all the declarations of our
17:53:49variables at the start of the store
17:53:51procedure immediately after begin. So
17:53:53that means we're going to go over here
17:53:54and say declare and now after that it's
17:53:58like the parameters. It's very simple.
17:53:59So at total customers. So this is the
17:54:03name of the variable. And after that we
17:54:04have to define the data type. Of course
17:54:06you have to understand the data type
17:54:08from the query. Since we are saying
17:54:10count star then the output going to be
17:54:12an integer. That's why we're going to
17:54:13write it like this. So integer. And now
17:54:15we need another one for the average. So
17:54:18what we're going to do we're going to
17:54:19make a comma. Now we are declaring
17:54:21another variable. So at average score
17:54:24and the data type of this one going to
17:54:26be float because we have an average. So
17:54:29that's it for the first step. We are
17:54:30telling SQL we have two variables and
17:54:32SQL going to go and create an empty
17:54:34placeholder. So now in the second step
17:54:36we have to give our variables a value.
17:54:39So where we going to get the values?
17:54:40We're going to get it from the query. So
17:54:42let's do that. Now let's start with the
17:54:44first column. As you can see we have
17:54:45here the count star. And as we learned
17:54:47anything that we write on the right
17:54:49side, it going to be like an alias for
17:54:51the column. But in SQL if you go and
17:54:53write something before it, it going to
17:54:55be the variable. So we can do it like
17:54:58this. at total customers and then equal.
17:55:02So now we are saying whatever value this
17:55:04query returns it should be stored inside
17:55:06my new variable so that I'm assigning
17:55:09values to my variable. But here there is
17:55:12one thing that we cannot have any more
17:55:13aliases because our query will not
17:55:16return any results. Our query have now
17:55:19only one task to assign values to my
17:55:21variables. So that's why we cannot have
17:55:23it like this. We have to remove the
17:55:24alias. And the same thing we're going to
17:55:25do it for the average. So at average
17:55:29score equal to the average score and we
17:55:32have to remove the alias. So that's it.
17:55:34Now our query having different purpose.
17:55:36It is not for returning result. It is to
17:55:39assign values to our variables. So now
17:55:42we have values in the next step we have
17:55:43to go and use it. And we can use our
17:55:45variables everywhere inside our store
17:55:47procedure. So it could be in the print,
17:55:49it could be in the next query. So in any
17:55:51select statements in any place.
17:55:53Sometimes we use variables in order to
17:55:55pass an information from one query to
17:55:58another one. But in this example, we
17:55:59want to use our variables inside the
17:56:02prints. So it is very simple. We now
17:56:03we're going to go and replace the static
17:56:06number and it's like the parameter.
17:56:07We're going to say at total customers
17:56:11and the same thing for the average at
17:56:14average score. So that's it. It's very
17:56:17simple. So again the step one we have to
17:56:19declare them to define it for SQL and
17:56:21with that we're going to get an empty
17:56:22variable. The second step we have to add
17:56:25values to those variables and the last
17:56:27step we have to go and use those
17:56:28variables. So it makes sense right now
17:56:31if you check our message over here you
17:56:33can see that everything is dynamic and
17:56:35we don't have any static values but
17:56:37there is one more thing that's in the
17:56:39print everything should be as a string.
17:56:41So we cannot have dates numbers floats
17:56:43and so on. So that's why you have to
17:56:45make check if you're adding any
17:56:46parameter and variables all of them
17:56:48should be string. So the country it is
17:56:51okay because we have the data type of
17:56:53varchar but the total number and the
17:56:55average score this is not really good
17:56:58because they have different data type
17:56:59and we have to go and now cast those
17:57:01data types to another one. So we're
17:57:04going to say cast and we're going to say
17:57:05here as invar so that we don't get any
17:57:09errors from SQL. So cast as well here as
17:57:12in
17:57:14vchar like this. All right. So I think
17:57:17we are ready. Let's go and change the
17:57:20definition of our store procedure in
17:57:21order to test. So let's go and execute.
17:57:23Perfect. And now let's go and test. So
17:57:26let's start with the defaults where we
17:57:27have the parameter as USA. So now as you
17:57:30can see we are getting one result and
17:57:32this is from the second query. So the
17:57:34first query is not returning anything
17:57:36anymore in the output. But if you go to
17:57:38the messages over here, you can see we
17:57:40have a new message. It says total
17:57:42customers from USA is equal to three and
17:57:45the average score from USA is equal to
17:57:48825. And this is exactly what we wanted
17:57:51for our reports. Now let's go and
17:57:53execute the parameter equal to Germany.
17:57:56Again, we have only one result. And in
17:57:58the messages, we're going to get total
17:57:59customers from Germany is equal to two
17:58:01and the average score from Germany is
17:58:03equal to 425. So this is exactly how we
17:58:06work with the variables. We use it in
17:58:08order to hold one information in one
17:58:10place in order to reuse it later in
17:58:13different place. So that's it for
17:58:15[Music]
17:58:18variables. All right everyone. Now we're
17:58:20going to talk about how to control the
17:58:21flow in your store procedure and we're
17:58:24going to learn how to do that using the
17:58:25if else statements. So now let's have
17:58:27the following scenario. Now if you check
17:58:29our query over here we are doing the
17:58:31average of score and if you check the
17:58:33data you can see that in the scores we
17:58:36have nulls and nulls are really bad for
17:58:38aggregations. So we usually have to
17:58:40clean up our data before doing any
17:58:42aggregations. And in this scenario we
17:58:45can understand null as a zero. And how
17:58:47we going to clean up and handle the
17:58:48data? We're going to go and make an
17:58:50update on our table where we say if
17:58:52there is like a null then make it as a
17:58:55zero. And we will do this as a pre-step
17:58:57inside our store procedure. So that
17:58:59means first we have to clean up the data
17:59:02and then afterward we're going to
17:59:03generate the reports. And this is what
17:59:05we usually do inside SQL projects. So
17:59:08the logic going to be very simple. We
17:59:10have to check first do we have nulls
17:59:11inside the score. If the answer is yes
17:59:14then we have to go and update the null
17:59:16values to zero. But if the answer is no,
17:59:18we don't have any values then we can
17:59:20skip everything. So now we're going to
17:59:22go and build this logic inside our store
17:59:24procedure in order to clean up and
17:59:26prepare the data. So let's go. Okay. So
17:59:28now this part we're going to call it
17:59:31generating
17:59:33reports and we're going to have another
17:59:35part called prepare and clean up data.
17:59:40So now let's prepare first the structure
17:59:42of the if statements. So the syntax
17:59:44going to look like this. So if and then
17:59:46begin and end. So this is the block of
17:59:50the if and we're going to do the same
17:59:51thing for the else. So we have else and
17:59:54we have begin and end. Let me just
17:59:56separate them. So now how this works? We
17:59:58have to create a condition. If the
18:00:00condition is met then the if statement
18:00:03going to be executed. But if the
18:00:04condition is not fulfilled and we have
18:00:06false then the else statement going to
18:00:08be executed. So what is the condition?
18:00:10We have to check whether there is null
18:00:12inside the scores. So let's write a very
18:00:15simple query. It's going to say select
18:00:17one from sales
18:00:21customers where score is null and always
18:00:25we have to check the country equal to
18:00:28let's say USA. So let's go and execute
18:00:31this one over here. So now we are
18:00:32getting in the output a results. If we
18:00:34are getting a results that means
18:00:36somewhere there are nulls. But if you go
18:00:39for example and say here Germany and
18:00:41execute the same query in the output you
18:00:44see that we don't have any results. That
18:00:46means for the German customers we don't
18:00:48have any nulls in their scores. So if
18:00:51this query returns something we have
18:00:53nulls. If it didn't return anything then
18:00:55there is no nulls. And we're going to
18:00:57use exactly this query as a condition.
18:01:00So we're going to take our check and say
18:01:02if exists and then two parenthesis and
18:01:05then we put our query. So what we are
18:01:07saying if exist if this query return
18:01:10anything then go and execute the next
18:01:12block and if it is not exist that means
18:01:15it is not returning anything then go and
18:01:17execute the second block. So it's a
18:01:19logic right it's very simple now of
18:01:20course instead of having a static value
18:01:22over here we can use our parameter so at
18:01:25country and now we have to tell SQL what
18:01:27to do if it exists. So in between we can
18:01:31have like an update statement. So update
18:01:33sales customers and we're going to set
18:01:36the score equal to zero. But very
18:01:39important we have to go and use where
18:01:41condition otherwise it going to go and
18:01:43update everything. The score is
18:01:47null and the country equal to our
18:01:50parameter country. So with that we are
18:01:53updating exactly the nulls for specific
18:01:55country. And let's have a semicolon at
18:01:57the end. And at the start maybe I'm
18:02:00going to say just to have a nice message
18:02:01in the output print and we can have a
18:02:04message updating null scores to zero and
18:02:09as well a semicolon at the end. So if
18:02:11there is any nulls then execute the
18:02:13whole thing print the message and update
18:02:15the table. So now the next step is that
18:02:17we're going to go and tell SQL what can
18:02:19happen if the condition is not
18:02:20fulfilled. That means we don't have any
18:02:22nulls. Well we don't have to update the
18:02:24table at all because we don't have to
18:02:26clean up anything. But I'm going to go
18:02:28and make print over here. So print and
18:02:31we're going to give the message no null
18:02:34scores found. And at the last end I'm
18:02:38going to go and put a semicolon. So
18:02:40that's it. This is our logic. We are
18:02:42checking our condition and then we
18:02:44execute if the condition is met where we
18:02:46update the table with zero instead of
18:02:48null and if the condition is not met
18:02:50then don't do anything. Just print a
18:02:52message. Now you might say you know what
18:02:54why you are doing this? we just can use
18:02:56this update statements and we don't need
18:02:57the whole if else statements. So why we
18:02:59are checking in the first place? I can
18:03:01like each time I run this store
18:03:02procedure I go and update all the nulls
18:03:05if they exist to a zero. Well, this is
18:03:07not really professional because you are
18:03:09wasting resources. So each time you run
18:03:11an update statement like this. So
18:03:14imagine that you have a big table and
18:03:15each time you run your store procedure,
18:03:17SQL have to go and check whether there
18:03:19is any nulls and so on. And this is of
18:03:22course consume resources. It's way
18:03:24better if you go and check first whether
18:03:26it's really needed. So that's why we are
18:03:28doing this logic. Now as you can see our
18:03:30store procedure is getting bigger and
18:03:32bigger. So we have like two parts. The
18:03:34first part is preparing and cleaning up
18:03:37the data. And the second part we are
18:03:39generating reports. Let's go and update
18:03:41the whole thing and execute it. And now
18:03:44we have to do it step by step. So let's
18:03:46check our query over here. And you can
18:03:47see we have here null for USA customers.
18:03:50So let's go first execute it for the USA
18:03:53as a defaults. And now let's go and
18:03:55check the messages. It's saying updating
18:03:57null scores to zero. That means the
18:04:00first block is executed because SQL did
18:04:03find a customer with a null. And with
18:04:06that the average of scores going to be
18:04:08different than previously. So we have
18:04:10now more accurate average in our
18:04:13reports. So if you go and check our
18:04:15query again, you can see now we have a
18:04:16zero instead of null. Let's go and
18:04:19execute it for Germany like this. And
18:04:21let's go and check the messages. It says
18:04:23no null scores found. And that is
18:04:25correct because for Germany we don't
18:04:28have any nulls. So with that we have
18:04:30created a control flow using the FL
18:04:33statements. And as you can see we are
18:04:34not doing any more like simple queries.
18:04:36We are creating like a mini program. And
18:04:39now it's like an ETL where first we
18:04:41prepare the data and second we generate
18:04:43reports. And you can imagine a real
18:04:45project how big those stored procedures
18:04:47going to get where you have a lot of
18:04:49tables and a lot of things to
18:04:54do. Okay. So now we're going to talk
18:04:56about the error handling in store
18:04:58procedure. Error handling it is like
18:05:00essential things to do while programming
18:05:02because it gives you the control on what
18:05:04can happen once you have an error. And
18:05:07there's a lot of things that you can do
18:05:08like maybe deleting data, printing a
18:05:11very structured like message or maybe
18:05:13doing some logging and so on. So you
18:05:15have a full control on what to do if
18:05:18there is an error and of course we can
18:05:19do that in the store procedure. So now
18:05:21let's check the quickly the syntax. It
18:05:23is usually has two parts. The first part
18:05:26is the try part. So the syntax is like
18:05:28this begin try end try. So you are
18:05:31defining the boundaries of the try and
18:05:33in between you going to have all your
18:05:35SQL statements and your code and the
18:05:37second part going to be the catch parts.
18:05:39So you say begin catch and end catch. So
18:05:42you are defining the boundaries and then
18:05:44in between you can tell SQL what to do
18:05:46if there is like an error. So what is
18:05:48try and catch? Like the word it says try
18:05:51it's like you are attempt to do
18:05:52something that might fail. So you are
18:05:54telling SQL try to execute this code. So
18:05:57the SQL going to go and try to execute
18:06:00your codes. And if any error happens
18:06:02while executing your codes, the SQL
18:06:04going to jump to the second block and
18:06:06start doing whatever you have defined in
18:06:08the catch. But if there is no errors at
18:06:11all, this part will not be executed. So
18:06:14the catch is like your backup plan. If
18:06:16something goes wrong here, then go to
18:06:18the plan B and do something. So let's
18:06:20see the workflow of the try catch. So
18:06:22first the SQL going to go and execute
18:06:24the try and then it going to check is
18:06:26there any error. If we don't have any
18:06:28error then everything ends and that's
18:06:30it. But while execution if the SQL face
18:06:33any error what going to happen it going
18:06:35to go and execute the catch. So as you
18:06:37can see the workflow is very simple and
18:06:39this is what we mean with try and catch.
18:06:41So let's go back to SQL to have some
18:06:43example. All right. So now back to our
18:06:45store procedure. Let's go and introduce
18:06:47an error inside our code. So let's go
18:06:49over here and maybe in our query we're
18:06:52going to go and divide by zero which is
18:06:55of course a problem. So we have this
18:06:57error over here and let's go and update
18:07:00the logic of our store procedure. And
18:07:03now if you go and execute it. So let's
18:07:05go and do that. We will get an error
18:07:07saying yeah you cannot divide by zero.
18:07:10But now what I would like to do I would
18:07:11like to have something else where we
18:07:12have customized message when error
18:07:14happens. So I would like to have the
18:07:16control on which information should be
18:07:18displayed if there is an issue. And in
18:07:20order to do that we have to use the try
18:07:22and catch. So it's going to be very
18:07:23simple. Now this is my whole code. So
18:07:26the whole thing from preparing to
18:07:28generate the report the whole thing is
18:07:30my code and we have to put the whole
18:07:32thing in a try. So how to do that?
18:07:34Exactly after the first begin we're
18:07:37going to have another begin but for the
18:07:39try. And now what we're going to do,
18:07:40we're going to go to the last end over
18:07:42here and have an end try. So with that
18:07:46we put now the whole code inside the
18:07:48try. And after that we're going to
18:07:50introduce the catch. So begin catch and
18:07:54end catch. And now in between we have to
18:07:56tell SQL what can happen if we encounter
18:07:59an error. And here we can do many stuff
18:08:01but I would like now to focus on
18:08:03customizing the error message. Let's
18:08:05start with the first one. So I'm going
18:08:06to say print let's say an error
18:08:11accord. This is the first thing. Then on
18:08:14the next line I'm going to print more
18:08:16informations. And now we're going to say
18:08:17the error message. So error message
18:08:21double point space. And now we can go
18:08:24and use some predefined functions from
18:08:26SQL like for example the error
18:08:29message. This function going to return
18:08:31the description of the error like the
18:08:33one we have here divide by zero error
18:08:35encountered and we can go and keep
18:08:37adding stuff the way that we need like
18:08:41maybe the error
18:08:43number. So we can have it like this and
18:08:45for that we have as well a function
18:08:46called error number and I think we have
18:08:49to cast this one because it is a number
18:08:51and in the messages we have to have only
18:08:54vchar. So this going to be as int var
18:08:58like this and we can keep adding stuff
18:09:00to our message like for example let's
18:09:03take the error line and for that we have
18:09:05as well a function so it's going to be
18:09:07the error
18:09:09line like this and we have to cast
18:09:12it because it is as well a number and as
18:09:16well what is really important is the
18:09:19name of the stored procedure. So error
18:09:23procedure and we have a function for
18:09:25that error procedure like this. It's
18:09:29going to be a string. So that's why I
18:09:31don't have to cast it. So now with that
18:09:32we have defined for SQL what to do if
18:09:35there is like an error in our code. So
18:09:37let's go and execute the whole thing.
18:09:40And now let's go and execute our stored
18:09:42procedure. So let's go and do that. So
18:09:44now as you can see in the output we are
18:09:46not getting any results and it is not
18:09:48giving an error. But if you go to the
18:09:50messages, you will see a very nice
18:09:52message. So it says an error is
18:09:54occurred. The error message is divided
18:09:56by zero and we have the error number in
18:09:59which line and as well the stored
18:10:01procedure name. So as you can see it's
18:10:02amazing. This is how we use the try and
18:10:05catch in order to have more options on
18:10:07to control what can happen if there is
18:10:09an
18:10:13error. Now the next step what I'm going
18:10:15to do, we have to go and organize our
18:10:17store procedure. As you can see,
18:10:18everything is getting bigger. So now
18:10:19what we usually do, we use tab in order
18:10:21to make spaces between each section. So
18:10:24now the first section is between the
18:10:26first begin and the last end. So we have
18:10:29to go and mark everything and hit once a
18:10:32tab. So now it is easier to read. Now
18:10:35the whole thing is our codes. So now the
18:10:37next level is the block of the try. So
18:10:39the whole thing over here is the try. So
18:10:43let's go and do that. I'm just going to
18:10:45mark everything until here and then hit
18:10:48tab. So now we can see it better, right?
18:10:50And the same thing for the catch. I
18:10:52think I have already done that. So it's
18:10:54already pushed. Now we go to the next
18:10:55level. So between this begin and end,
18:10:58everything is pushed. So this looks
18:11:00nice. The same thing over here. It's
18:11:02pushed as well. And then we don't have
18:11:03here any begin and end. So it looks
18:11:06okay. And the same thing over here. So
18:11:08all our begin and end is now sorted
18:11:11correctly. Now the next step is that we
18:11:12can go and improve the comments a little
18:11:14bit. So we can split our code into
18:11:16multiple sections. So what we're going
18:11:18to do, we're going to go over here and
18:11:19say this is step one. And what I like to
18:11:22do is to go and add separation using the
18:11:25equals or any special character that you
18:11:27like and as well here. So with that we
18:11:31have the first step. We are preparing
18:11:32the data. And then let's go and copy the
18:11:34whole thing and go over here and say
18:11:37this is the step two. And we're going to
18:11:39say this is
18:11:41generating summary reports and something
18:11:45like this. And of course below that we
18:11:47can say what is this report about. So
18:11:51calculate total
18:11:53customers and average score for
18:11:58specific country. And as well we can go
18:12:01over here and add as well a comment.
18:12:04calculate total number of orders and
18:12:09total sales for specific country. And of
18:12:14course we have to go and remove this
18:12:16error over here otherwise we'll get an
18:12:18error and we can go and add something
18:12:20about the catch where we can say like
18:12:21this again few
18:12:23comments we're going to say error
18:12:28handling. So let's go and execute it
18:12:31again in order to make sure we have the
18:12:33newest version. And with that we are
18:12:35done. We have a really nice stored
18:12:36procedure with multiple steps and we
18:12:38have it professional where we have error
18:12:40handling inside it and everything looks
18:12:42well organized and easy to read. So this
18:12:45is how we build stored procedures. All
18:12:47right my friends. So that's all about
18:12:48the store procedures. That was an
18:12:50amazing feature in SQL to add
18:12:52programmability in SQL. Now in the next
18:12:54step we're going to cover quickly the
18:12:56topic of the triggers. So let's
18:13:02go. All right. So previously we have
18:13:04understood that we can put all our SQL
18:13:07statements in one stored procedure and
18:13:09you have to go and manually execute the
18:13:12store procedure. So that means in order
18:13:14to trigger the start procedure, you have
18:13:16manually to execute it and this is of
18:13:19course a problem. How about to do that
18:13:21automatically? So triggers in SQL they
18:13:24are special stored procedure that
18:13:26automatically runs or let's say fired in
18:13:29response to a specific event that
18:13:31happens on a table. So what this exactly
18:13:34means? So now let's say that we have a
18:13:36table in our database and now something
18:13:38could happen to this table like
18:13:40inserting data, deleting, updating data,
18:13:43all those stuff that is happening we
18:13:45call them events. And now what we can do
18:13:47we can go and attach like a trigger on
18:13:50top of this table and each time an event
18:13:52happened like insert update delete
18:13:54something else going to be triggered
18:13:56like maybe going and inserting data
18:13:58somewhere else in another table or doing
18:14:01a check whether we are allowed to delete
18:14:03the data in the first place or maybe
18:14:05sending a warning message or something.
18:14:07So based on any changes to the table we
18:14:10can trigger another events and we can do
18:14:12that using the SQL triggers and for the
18:14:15SQL triggers we have like multiple types
18:14:17like the DML triggers and this type of
18:14:20trigger going to respond once we have
18:14:22like insert update delete statements.
18:14:25Another type of triggers we have the DDL
18:14:27triggers like you can make a trigger to
18:14:29respond to any schema changes like
18:14:32creating altering or dropping a table or
18:14:35even view by the way not only tables.
18:14:37And the third type of triggers we have
18:14:38the login trigger. So the trigger can
18:14:40respond to login events. Now in this
18:14:43tutorial we're going to focus on the DML
18:14:45triggers the insert update delete. And
18:14:47for the DML triggers we have two types.
18:14:50We have after triggers and as well we
18:14:52have instead of triggers. So as the name
18:14:55suggest if you use after so it can be
18:14:58executed after the event and the other
18:15:00type that instead of it's something that
18:15:02cannot wait until everything happens. So
18:15:05this time the trigger going to be
18:15:06executed during the event not after it.
18:15:10So now in order to understand all of
18:15:11this we're going to have really nice use
18:15:17case. And now the use case is about
18:15:19maintaining an audit logs. So what we
18:15:22mean with that? Let's have for example
18:15:23the table employees. The employee data
18:15:26are usually very sensitive informations
18:15:28because there we can see which employees
18:15:31are added, the salary updates, the
18:15:33employee terminations and this makes the
18:15:35table very important because we would
18:15:38like to track all those changes that is
18:15:40happening to this table. So each time we
18:15:42are inserting, updating, deleting, we
18:15:44would like to maintain a log about all
18:15:46those changes in order to analyze it
18:15:48later. It is of course very important
18:15:50such a logs for the compliance and the
18:15:53auditors and in case there is like a
18:15:55problem we can go to the logs to
18:15:57understand when this happened who made
18:15:59the changes and what exactly changed and
18:16:02now in order to maintain logs we're
18:16:04going to use the power of triggers. So
18:16:06what we're going to do we're going to go
18:16:07and attach like a trigger on the table
18:16:10employees and each time we insert new
18:16:13data to the employees we are triggering
18:16:16another events. So what can happen this
18:16:18new employee going to be inserted in the
18:16:21audit logs in order to have a record
18:16:23about this activity in the logs. So that
18:16:26means each time you are inserting data
18:16:28to the table employees you are
18:16:30automatically inserting data inside the
18:16:32logs and this is really amazing use case
18:16:34for the triggers. So let's go and
18:16:36implement it. Okay. So now let's check
18:16:38quickly the syntax of the triggers. So
18:16:40we start with the usuals create trigger
18:16:42then the trigger name and then we have
18:16:43to specify on which table this trigger
18:16:46going to be built in. So now we are
18:16:48attaching like a trigger on top of one
18:16:50table and after that we have to define
18:16:52for SQL when this trigger going to
18:16:55happen. So what is actually triggering
18:16:57the trigger and here you can define
18:16:59after or instead then you have to define
18:17:02the operator. So first you have to
18:17:04define like after or instead of and then
18:17:06we have to define the operation. So
18:17:09insert, update, delete or one of them.
18:17:11And with that you are telling SQL when
18:17:13exactly this should happen. And now
18:17:14after that we have to tell SQL what
18:17:17going to happen if the trigger is
18:17:19triggered. So here we have like begin
18:17:21and end. And then we have like several
18:17:23skill statements that's going to
18:17:24describe what's going to happen once we
18:17:26have the trigger. So that's it. As you
18:17:28can see the syntax is very simple. Okay.
18:17:30So now let's do it step by step. First I
18:17:32would like to create a table where we're
18:17:34going to store the logs information. So
18:17:36it's going to be very simple table.
18:17:38We're going to say create table. Then
18:17:40we're going to call it sales employee
18:17:43logs and we're going to have the
18:17:44following columns inside it. So let's
18:17:47start with the primary key. It's going
18:17:48to be the log ID and the data type int
18:17:50and then we're going to have like a
18:17:51sequence. So we're going to have
18:17:53identity and this is the primary key.
18:17:56Let's go to the next one. It's going to
18:17:57be the employee ID and the data type
18:18:00going to be ins. The next one is going
18:18:01to be the log message. So let's have it
18:18:04as a vchar and I'm going to have it like
18:18:07255 and then to the next one we're going
18:18:09to have the lock dates and then we're
18:18:11going to have like let's say a date or a
18:18:13date time. So that's it. Let's go and
18:18:15execute it and with that we have a new
18:18:17table inside our database. Now the next
18:18:19step is that we're going to go and
18:18:21create our trigger. So we're going to
18:18:23say create trigger and I'm going to call
18:18:26it like this trg. This is just a prefix
18:18:28to indicate this is a trigger. And I'm
18:18:30just going to call it after insert
18:18:33employee. And now we have to define the
18:18:36table. So it's going to be on sales
18:18:39employee. So now with that we are saying
18:18:41we have now a trigger on the table
18:18:43employees. And now we have to define the
18:18:45logic. So we're going to use after
18:18:47insert. So that means after we insert
18:18:50any record to the table employees the
18:18:52following things should happen. So we're
18:18:54going to say as and then begin and end
18:18:58and in between we can have our logic. So
18:19:00what can happen after a new record is
18:19:02inserted to the employees. We're going
18:19:04to go and insert a new record to the
18:19:06employee logs. So we're going to have
18:19:08insert into sales employee
18:19:12logs and we're going to have here the
18:19:14three columns employee
18:19:16ID the log
18:19:18message and the log dates. So now which
18:19:22value is going to be inserted? it going
18:19:23to be like from a query. So we're going
18:19:25to say select and we're going to say as
18:19:27well employee ID and for the log message
18:19:30we can have customized one like let's
18:19:32say new employee added and it's going to
18:19:37be equal to the employee ID. So in order
18:19:40to have the employee ID it's going to be
18:19:41like
18:19:42this. So that's it. Now to the next one
18:19:46we need the log date. It's going to be
18:19:48get date. And now you might say okay but
18:19:51where this employee ID is coming from?
18:19:54Well, it going to come from the table
18:19:56from inserted. So what is actually
18:19:58inserted? It is like special virtual
18:20:00table that holds all the new inserted
18:20:03data to our table employees. So anything
18:20:06we are inserting inside the employees
18:20:08will be available inside this table. And
18:20:11of course this is only available during
18:20:14the execution of this trigger. So you
18:20:16cannot go now outside of this query and
18:20:18start querying the table inserted
18:20:20because you will not find anything. This
18:20:21is only like a virtual table that
18:20:23contains anything that you are doing to
18:20:26the table employees and you find a lot
18:20:28of informations like the salary, the age
18:20:30and so on. So that's it for the
18:20:32inserted. Now we have to make sure that
18:20:35in our message we have everything as a
18:20:37string because the employee ID is an
18:20:39integer. So we have to cast it. So cast
18:20:42and then we're going to say as far char
18:20:46like this otherwise we'll get an error.
18:20:48So I think we have our trigger ready. We
18:20:50have a new trigger on the table
18:20:52employees. And now the first question is
18:20:55when this trigger going to happen? Well
18:20:57it can happen after inserting data to
18:21:00the employees. And then the second
18:21:01question what's going to happen? Well,
18:21:04once we have this event, the whole thing
18:21:06here going to be executed where we are
18:21:08saying insert to the logs, the employee
18:21:11ID, the message and as well the date
18:21:13when this happens. And we can get all
18:21:15those informations from the table, the
18:21:17virtual table inserted. So I think we
18:21:20are ready. Let's go and execute it. And
18:21:22now if you go to the object explorer to
18:21:24our database, let's go to our table
18:21:26employees and then to the triggers. So
18:21:29if you refresh over here you can see our
18:21:31new trigger that we just created. So
18:21:34with that we have to find our trigger
18:21:35and we are ready. Now the next step is
18:21:37that we're going to go and trigger our
18:21:39trigger. So let's go and do that. Let's
18:21:41have a new query. But first I'm going to
18:21:43have a look to our
18:21:45logs. So sales employee logs. So let's
18:21:50query this one. And as you can see our
18:21:52logs is empty because we didn't insert
18:21:54anything to the table employees. Let's
18:21:57go and do that. Let's trigger our
18:21:58trigger. So what we're going to do,
18:21:59we're going to say insert into sales
18:22:02employees and we're going to have the
18:22:04following values. So we are at the
18:22:06counter, I think six. Let's have the
18:22:09first name
18:22:10Maria. The last name an then we're going
18:22:14to have the position. It's going to be
18:22:15the HR for example. The birth date,
18:22:18let's pick something. I don't know. We
18:22:21have a female
18:22:23here. And the salary. Let's go and get
18:22:26this salary and the hierarchy it can be
18:22:29for example three. So let's go and
18:22:31execute it. And with that as you can see
18:22:33we have inserted a new data to the
18:22:34employees. Let's check now the logs. So
18:22:38let's query it. So we have here nice log
18:22:40about the employee number six. And we
18:22:42have here nice message and when this did
18:22:45happen. Of course you can go and insert
18:22:47another employee let's say seven with
18:22:50the same data. So let's do that and
18:22:52check the logs. And with that we have
18:22:55another log for the new employee. So
18:22:57this is really amazing use case in order
18:22:58to maintain a log to your data and you
18:23:01can go and make like some analyzes on
18:23:03how many inserted happens and of course
18:23:06not only on the insert you can have it
18:23:08on the update delete. So as you can see
18:23:10it is very simple. This is how we create
18:23:12the triggers in SQL. All right my
18:23:13friends. So that's all about the
18:23:14triggers with that with with that we
18:23:17have covered now with that we have
18:23:19covered now all the concepts and topics
18:23:21that you have to learn about SQL. Now in
18:23:23the next chapter it's going to be about
18:23:25the performance. So as you start writing
18:23:27queries and so on you will start
18:23:29noticing some queries are really slow.
18:23:31Now what we're going to do in this
18:23:32chapter we're going to learn different
18:23:34techniques on how to optimize the
18:23:35performance. And the first and the very
18:23:37famous one is to go and build indexes in
18:23:40databases. So let's understand what this
18:23:46means. So what is an index? An index is
18:23:49a data structure that provides a quick
18:23:52access to the rows to improve the speed
18:23:55of your queries. So an index is like a
18:23:58guide for your database in order to
18:24:00speed up the process of searching for
18:24:02data especially if you have like big
18:24:04tables. So now in order to understand
18:24:06what are indexes, imagine you have huge
18:24:08book and you want to find a specific
18:24:10topic or a chapter. Instead of flipping
18:24:13each single page in order to find the
18:24:15topic that you are searching for, you
18:24:16would use the index at the back of the
18:24:18book in order to jump straight to the
18:24:21right page. And that's exactly what
18:24:23index does but for your data. Another
18:24:26analogy that I use in order to
18:24:27understand indexes is think about the
18:24:30indexes as a big hotel. Now let's say
18:24:33that in the hotel we don't have any
18:24:34guide and you would like to find the
18:24:36room number let's say 5001. Now what you
18:24:40going to do? You're going to go and
18:24:41search for your room floor by floor and
18:24:44checking each room until you find your
18:24:46room. But instead of that, thankfully
18:24:48hotels have a numbering system. And you
18:24:51can ask for a map from the reception in
18:24:53order to understand in which building in
18:24:55which floor you can find your room. So
18:24:58by just following the map and maybe some
18:25:00signs, it's going to be very quickly to
18:25:02locate and find your room in such a big
18:25:04hotel. And that's exactly what each
18:25:06database needs. It needs an index in
18:25:08order to help the database finding and
18:25:11locating the right data without having
18:25:13to scan
18:25:18everything. And now let's say that you
18:25:20ask me, you know what, I have this big
18:25:22table and I would like to speed up the
18:25:24queries using indexes. And my first
18:25:27question going to be, what are you
18:25:28exactly doing with this table? Are you
18:25:30using this table to search for text or
18:25:33are you doing like complex analyszis
18:25:35with this table? And the reason why I'm
18:25:37asking this is that we have different
18:25:40indexes in databases for different
18:25:42purposes. So now let's have a quick look
18:25:43to the different types of indexes that
18:25:46we have in database. I divide the
18:25:48indexes in databases into three
18:25:50categories. The first one is by the
18:25:52structure how the database is organizing
18:25:55and referencing the data. And here we
18:25:57have two types. The clustered index and
18:26:00the non-clustered index. Those are very
18:26:02important to understand. Now we have
18:26:04another category for the indexes. We can
18:26:07divide them by the storage. And in this
18:26:09category we are talking about how the
18:26:11data is stored physically in the
18:26:14database. So we have two types. We have
18:26:16the row store index and the column store
18:26:19index. And the third type is the
18:26:21functions and here we have two types. We
18:26:24have the unique index and the filtered
18:26:26index. Now each index type has its own
18:26:29strings but as well there is always a
18:26:32tradeoff. Some might improve their read
18:26:34performance. The other one might improve
18:26:36the insert and update operations. So
18:26:39it's all about choosing the right type
18:26:41of index for the job. So now what we're
18:26:43going to do, we're going to go and deep
18:26:44dive into each of those types in order
18:26:46to understand how they work and how we
18:26:49can create them. And we will start with
18:26:50the first category, the structure. We
18:26:53have the clustered index and the
18:26:54nclustered index.
18:27:00Now before we dive into how the indexes
18:27:02works in databases, let's understand
18:27:04first what happens to the database
18:27:07tables if you don't use any index. When
18:27:10you create a new table in your database
18:27:12like for example the customers table
18:27:14where you have let's say 20 customers
18:27:16inside this table. What you're going to
18:27:18see at the client side is like
18:27:20spreadsheets like a table with rows and
18:27:23columns but behind the scenes the
18:27:24database store it a bit differently.
18:27:27It's going to store the data in a data
18:27:29file on the disk and inside this file
18:27:32the data can be stored inside blocks
18:27:34called pages. So it's not like rows and
18:27:37columns that are stored inside data
18:27:39files and inside the data files we have
18:27:42pages. So what is a page? A page is the
18:27:46unit of data storage in a database and
18:27:49it is a fixed size of 8 kilobyt where
18:27:52the SQL database can store anything
18:27:55inside it. It can store inside it the
18:27:57rows of your tables or columns metadata
18:28:00indexes and every time you are
18:28:02interacting with your data the SQL is
18:28:05reading and writing to those pages. So
18:28:08as you can see the SQL is not storing
18:28:10the data inside like rows and columns.
18:28:13So if you are running a query the SQL is
18:28:15not like selecting a specific column it
18:28:18always fetch a data page in order to
18:28:21read the rows inside this page. And the
18:28:24main two types that we're going to learn
18:28:25is the data page and the index page. So
18:28:28how the data page looks like it is
18:28:30divided into multiple sections. The
18:28:32first section is the page header where
18:28:34the database can store key informations
18:28:36about the metadata like the page ID and
18:28:39it has the following format. It start
18:28:41with the file ID like one and then we
18:28:44have a unique number for each page. So
18:28:46for example 150. So the page header is a
18:28:49fixed size of 96 bytes. Now to the next
18:28:51section, we're going to have a variable
18:28:53size. This is where your data row is
18:28:56going to be stored. So your actual data
18:28:58and row is going to be stored in this
18:28:59section. And the SQL going to try and
18:29:01fits as many rows as it can in one
18:29:04single page. And this of course depends
18:29:06on the size of each row. So if you have
18:29:09like a large table where the rows are
18:29:11really big, so SQL can fit only few rows
18:29:14in one single page. And now moving on to
18:29:16the last section in the data page, we
18:29:19have the offset array. This is like a
18:29:21quick index for the rows stored inside
18:29:24this page. It keeps track of where each
18:29:27rows begins so that the SQL can easily
18:29:30locate a specific row without having SQL
18:29:33like scanning the entire page in order
18:29:35to find a row. So this is the structure
18:29:37of the data page and this is exactly how
18:29:40the SQL stores data inside the
18:29:43databases. So now back to our example
18:29:44where we have the customers table and 20
18:29:47rows. So let's see how SQL going to be
18:29:49creating those pages. Now if you are not
18:29:51using any index in this table. So now
18:29:53what going to happen? SQL going to
18:29:55insert the data inside those pages as
18:29:57you are inserting the data inside the
18:30:00customers. So maybe first you are
18:30:01inserting the customers like 12 5 6 7
18:30:05and SQL going to insert it to the data
18:30:07pages exactly like that. So that means
18:30:10SQL is just inserting the data as you
18:30:12insert it to the table. So let's say
18:30:14each data page is like fitting only five
18:30:17rows. So after we insert five customers,
18:30:19SQL going to go and create another data
18:30:21page for the next rows. So in the next
18:30:24page, the SQL going to insert the next
18:30:26five customers. And once it's full, it's
18:30:29going to create another data page in
18:30:30order to start adding the next customer
18:30:33until we have like for example four
18:30:35pages for that 20 customers. So now if
18:30:38you check the customers inside those
18:30:39four pages you see that they are not
18:30:42sorted at all and that's because in this
18:30:44scenario we are not using any index. So
18:30:47we call this structure as a heap
18:30:49structure. So a heap table is a table
18:30:52without a clustered index. That means
18:30:54the rows are stored randomly without any
18:30:57particular order. This is not a really
18:31:00bad because it's going to be very quick
18:31:02to insert data inside this table. But of
18:31:05course finding something from this table
18:31:07going to be very slow. So this is the
18:31:09first tradeoff. You have a very fast
18:31:12writes but a very bad reads. Think about
18:31:15it like you are throwing all your papers
18:31:17in a drawer without organizing them. So
18:31:20you can toss things very quickly in this
18:31:22drawer. But if you want to search for
18:31:24specific paper later, it's going to be
18:31:26very long process until you find it
18:31:28because nothing's in order. So now let's
18:31:31see how the SQL going to handle if you
18:31:33read something from this table. Let's
18:31:35say that you are searching for the
18:31:36customer with the ID 14. So now SQL has
18:31:39totally no idea where to find this
18:31:41customer. So SQL going to start fetching
18:31:43each data page and start scanning each
18:31:46row. So it's going to start with the
18:31:47first data page and start scanning.
18:31:49Well, SQL will not find 14 here. So SQL
18:31:52going to go to the next page and start
18:31:54scanning as well. Searching for the ID
18:31:5714 and nothing going to be found. The
18:31:59same thing for the third page as well.
18:32:00SQL will not find 14. So SQL going to go
18:32:03to the last data page and there after
18:32:05scanning four rows in this data page
18:32:08finally SQL going to find the customer
18:32:10number 14 and it's going to return it
18:32:12for the clients. So as you can see in
18:32:14order to find one customer SQL did read
18:32:18four different pages and scanned like 19
18:32:21rows in order to find the customer and
18:32:23this process we call it full table scan.
18:32:27So the full table scans means SQL is
18:32:29scanning the entire table page by page
18:32:32and row by row in order to find specific
18:32:35row. And of course for this table maybe
18:32:37it's not a big deal. But if you have
18:32:39like a big table where you have like
18:32:41hundred of thousands or maybe millions
18:32:43of rows searching through the heap
18:32:45structure going to be very painful and
18:32:47slow in order to locate one row. And
18:32:49here exactly why we need indexes in SQL
18:32:52databases. So let's understand the first
18:32:54type of indexes the clustered
18:33:00index. All right. So now let's
18:33:02understand what can happen if you create
18:33:04clustered index in your table. So say
18:33:06you create a clustered index on the ID
18:33:09column of the customers. So the first
18:33:11thing that's going to happen SQL going
18:33:12to physically sort all the data based on
18:33:16the column ID. So the rows going to
18:33:18rearranged in each data page from the
18:33:21lowest to the highest. So in the first
18:33:23page we're going to have the first
18:33:25customer ID number one then 2 3 4 5
18:33:28until we reach in the last page the last
18:33:31customer number 20. So as you can see
18:33:33the first page has the lowest value and
18:33:36the last page has the highest value. So
18:33:39that's not all. The next step is that
18:33:41SQL going to go and start structuring
18:33:44and building the B tree. So what is a B
18:33:48tree? A B tree short for balance tree.
18:33:50It is hierarchal structure that store
18:33:53the data as a tree upside
18:33:56[Music]
18:34:02down. It start with the root the root
18:34:05node and then it keep branching out
18:34:07until we reach eventually the leaves.
18:34:10Between the leaf nodes and the root
18:34:12nodes we call this section the
18:34:14intermediate nodes. So it could be like
18:34:16one level or multiple levels between the
18:34:18root and the leaves. And once SQL
18:34:21construct the B tree, it's going to be
18:34:23very easy for SQL to navigate through
18:34:25the B tree in order to find specific
18:34:27information. So let's see how SQL is
18:34:30building the B tree for the clustered
18:34:32index. Now very important to understand
18:34:34that the leaves the leaf nodes and the B
18:34:37tree for the clustered index contain the
18:34:40actual data the data pages. So all your
18:34:43nice sorted data pages and your data is
18:34:46stored at the leaf level. Then after
18:34:48that SQL going to start building the
18:34:50intermediate nodes and here the database
18:34:52going to use different type of pages. We
18:34:56have the index page. So in the index
18:34:58page we cannot find the actual data the
18:35:01entire rows but instead the index page
18:35:03stores a key value that contain a
18:35:06pointer to another index page or to a
18:35:09data page. So for example we have here
18:35:11the value one the key and then the value
18:35:14going to be the ID of the data page. So
18:35:16here we don't have like the whole row
18:35:18about the data we have here only a
18:35:21pointer to another data page. So here we
18:35:23are telling the scale if you are
18:35:25searching for ids between 1 and five you
18:35:28can locate it at the data page ID
18:35:321.100 and then we can store in this
18:35:34index page another pointer where we can
18:35:36tell SQL if you are searching between 6
18:35:38and 10 then you can locate it at the
18:35:41second data page. So this is the
18:35:44structure of the index page it contains
18:35:45only pointers to another page and the
18:35:48same thing for the second two pages. The
18:35:51SQL going to create another index page
18:35:53where it's going to says if you are
18:35:54searching for IDs between 11 and 15, you
18:35:57can find it at the third page 1 double
18:36:00point 10002. And for the last group
18:36:02between 16 and 20, we have another
18:36:04pointer to the last page to the page
18:36:06number
18:36:08one3. So as you can see inside those
18:36:10index pages, we have like a pointer for
18:36:13each group of ids for each cluster. So
18:36:16for the group of customers between 1 and
18:36:18five we have one pointer and for the
18:36:21second group between six and 10 we have
18:36:22another pointer. So that means we don't
18:36:25have here a pointer for each row. We
18:36:28have a pointer for each group for each
18:36:30cluster. That's why we call it clustered
18:36:33index. And now once SQL is done building
18:36:35the intermediate nodes, SQL going to go
18:36:37and build the last node, the root node
18:36:40where it says if you are searching for
18:36:42customers between 1 and 10, then go to
18:36:46the index page with the ID
18:36:481.200. So that means the route node here
18:36:51is pointing to another index page, not
18:36:54directly to the data page. And the same
18:36:56thing, we need another pointer for the
18:36:59second index page. So the customers
18:37:01between 11 and 20 go to the index page
18:37:04with the ID
18:37:061.201 and this is exactly what going to
18:37:08happen if you create a clustered index
18:37:11in SQL. First it going to go and
18:37:14physically sort all your data in the
18:37:16databases. So if it's from the first
18:37:18time sorted randomly SQL has to arrange
18:37:22everything and sort the data from the
18:37:24scratch. And then it's going to go and
18:37:25build this structure where you have in
18:37:27the root node and index page in the
18:37:30intermediate nodes the index pages but
18:37:32at the leaf level at the leaves we have
18:37:36the actual data the data pages. So now
18:37:38let's see what going to happen if you
18:37:40query the table where you search for the
18:37:42ID number 14. So it's going to check
18:37:44which pointer to use since 14 is in the
18:37:47group between 11 and 20. It's going to
18:37:49go and use the second pointer to the
18:37:52index page with the ID one double point
18:37:542011. And here the SQL going to open
18:37:56this index page and check the pointers.
18:37:59So since 14 is between 11 and 15 it
18:38:03going to go and use the pointer to the
18:38:04data page one point 102 and with that
18:38:08SQL located the correct data page the
18:38:11third page and now SQL going to open
18:38:13this data page and find the customer ID
18:38:17number 14. So as you can see it was very
18:38:19fast for SQL to locate the correct data
18:38:21page with only three jumps from the root
18:38:24node to the intermediate node. The SQL
18:38:26were able to find fast the correct data
18:38:28page. And here SQL needs only to read
18:38:31one data page instead of reading as we
18:38:33saw in the heap structure four different
18:38:36data pages. And of course you might say
18:38:37but still here we are reading like three
18:38:39pages. Well, reading an index page is
18:38:42very fast compared to the data page
18:38:44because reading a data page is always
18:38:46slower than reading an index page. So,
18:38:49as you can see, this P3 structure, the
18:38:51clustered index structure did help the
18:38:53SQL and the database to locate the right
18:38:56data in the right
18:38:59[Music]
18:39:02databases. And this is exactly how that
18:39:05clustered index works in the SQL
18:39:07database.
18:39:12All right. So now we're going to move to
18:39:13the second type and we're going to
18:39:14understand how exactly SQL build and
18:39:17create the nonclustered index. So let's
18:39:20go. So now we are back to the heap
18:39:22structure where our table don't have any
18:39:24index and our data are stored randomly
18:39:26inside the data pages. And now if you go
18:39:28and create a non-clustered index on the
18:39:31customer ID, what can happen? And here's
18:39:33the big difference that SQL will not
18:39:36touch or change anything on the physical
18:39:40actual data on the databases. So the
18:39:42database is going to stay as it is and
18:39:45nothing going to be changed and the SQL
18:39:47start immediately building the B
18:39:50structure. So it's going to start
18:39:51immediately building an index page and
18:39:53this index page is a little bit
18:39:55different than the one that we have
18:39:56learned previously. So since it's index
18:39:58page, it's going to store pointers. But
18:40:00this time SQL going to store in the key
18:40:02the customer ID. So one is the customer
18:40:05ID and now the value the pointer it will
18:40:08not be the data page ID. We will be more
18:40:11specific. So we're going to have like an
18:40:12address where exactly the row is stored.
18:40:16So it's going to start with the file ID,
18:40:18the page number because the customer ID
18:40:21one is stored in the page
18:40:24one2. But SQL gonna go add as well the
18:40:28offset number of the row where exactly
18:40:30in the page we can find this ID and the
18:40:33whole thing we can call it an air ID the
18:40:36row identifier. So now let's see quickly
18:40:38how the index page is pointing exactly
18:40:41to the row inside the data page. So the
18:40:43first part of the row identifier is
18:40:46mapping to the data page ID and then
18:40:49from the 96 it's going to take us to the
18:40:51offset and that's exactly the location
18:40:54of the row number one. So 96 is the part
18:40:57where we're going to start finding the
18:40:59row number one and that's going to takes
18:41:01us exactly to the place where we can
18:41:04read the information about the row ID
18:41:07number one. So this is how the index
18:41:09page is locating the exact place of the
18:41:13rows. So SQL going to go and continue
18:41:15and assign for each customer ID a
18:41:18pointer to the exact location. So as you
18:41:21can see now in the index page we don't
18:41:22have like a pointer for each group of
18:41:25customers like we have learned in the
18:41:27clusters index. We have now a pointer
18:41:30for each ID and this type of index page
18:41:33we call it roator page. So now SQL going
18:41:36to go and continue and map a pointer for
18:41:39each customer ID that we have inside our
18:41:41table. So we will have multiple index
18:41:43pages pointing to our data page. So as
18:41:46you can see we have a lot of pointers
18:41:48and the data inside the index page is of
18:41:51course sorted but inside the data pages
18:41:54it left as it is. And now those index
18:41:57pages that has the row identifier going
18:42:00to be stored at the leaf level of the B
18:42:03tree. So at the leaf level we don't have
18:42:05the actual data the data pages we have
18:42:08index pages where we have pointers then
18:42:11to the actual data and then it's going
18:42:13to go and start building the
18:42:14intermediate nodes. It's exactly like
18:42:16the clustered index where it's going to
18:42:18point to another index page. So between
18:42:21one and five customers it's going to be
18:42:23in the index page number 200. So the
18:42:26next step is going to go and build the
18:42:27intermediate nodes. It's going to be
18:42:29exactly like the clustered index.
18:42:31Nothing going to be changed. is like the
18:42:33same structure. So it is an index page
18:42:35pointing to another index page but this
18:42:38time for a group of customers and then
18:42:40we're going to have as well the root
18:42:41node. So again we call this structure as
18:42:44a B tree structure where they point to
18:42:47another databases but the databases are
18:42:49not part of the B tree. So now let's say
18:42:52if we are searching for the customer ID
18:42:55number 14, what's going to happen? It's
18:42:57going to start again from the root node
18:42:59and then it's going to find the pointer
18:43:01to the intermediate node and then jump
18:43:02to the next step to the intermediate
18:43:04node and then it's going to find the
18:43:05pointer to the index page between 11 and
18:43:0815 and then it's going to go and scan
18:43:11this index page and find okay for the
18:43:13customer ID number 14 we have the
18:43:15following address. So it's going to go
18:43:17and locate the exact database and as
18:43:20well the exact place of the row. So it
18:43:23can go and jump immediately to the row
18:43:25without scanning anything else. So here
18:43:27this time with the nclustered index the
18:43:30SQL did read three different index
18:43:32pages. And finally the one data page in
18:43:34order to find the data. So if you
18:43:37compare to the clustered index you can
18:43:38see that we have here one extra layer
18:43:41one extra index page to be scanned in
18:43:44order to find the right place of the
18:43:46row. And this is how SQL creates the B
18:43:48tree for the nonclustered index and how
18:43:51it scans it in order to find the
18:43:57information. All right. So now when I
18:43:59think about the clustered index and the
18:44:00non-clustered index, I think about a
18:44:03book. You can think of the clustered
18:44:05index like the table of contents at the
18:44:08front of the table. So the table of
18:44:10contents kind of tells you where to find
18:44:12each chapter and the chapters are
18:44:15exactly sorted like the table of
18:44:17contents and this is exactly what the
18:44:19clustered index does. But now in the
18:44:21other hand think about the nclustered
18:44:23index as the index that you can find at
18:44:26the end of the book. The index of the
18:44:28book is a very detailed list of topics,
18:44:32terms and keywords where it points
18:44:35exactly to the location where you can
18:44:37find it in the book. And the content and
18:44:39the topic of the book is not sorted like
18:44:42the index of the book. And this is
18:44:44exactly what the noncluster index does.
18:44:46It is coexisting with the data. It is an
18:44:49extra list where it can point exactly
18:44:51where we can find the data inside our
18:44:54table. All right. Right. So now let's
18:44:55put those two indexes side by side to
18:44:57understand the differences between them.
18:44:59So the structure of the cluster the
18:45:00index is a B tree where it start with
18:45:03the root node where we have an index
18:45:05page. This index page is pointing to the
18:45:08intermediate nodes where we have as well
18:45:10index pages and those index pages are
18:45:13pointing to the actual data to the data
18:45:15pages. So at the leave level of the
18:45:18clustered index we have the data pages
18:45:20the actual data. What's special about
18:45:22the clustered index is that it
18:45:24physically sort the data inside those
18:45:27pages. So everything here is physically
18:45:29rearranged and sorted. Now if you are
18:45:32talking about the nclustlustered index
18:45:34we have as well a bit tree. So the same
18:45:36thing at the root node we have an index
18:45:38page pointing to an intermediate index
18:45:41page but this time the intermediate
18:45:43nodes are pointing to another index
18:45:46page. They are not pointing like the
18:45:48clustered index to a data page. they are
18:45:50pointing to index page. So now if you
18:45:53check this structure you can see that at
18:45:54the leaf level for the clustered index
18:45:57we have the actual data the data pages
18:45:59but on the other side at the leaf level
18:46:01for the nclustered index we don't have
18:46:03the actual data we have index pages but
18:46:06those index pages are pointing to the
18:46:09actual data to the data pages but the
18:46:12big difference of that the data pages
18:46:14are not part of the B3 the B3 of the
18:46:17nclustlustered index is just a separate
18:46:19structure that does not involve any
18:46:22data. So we have only index pages and it
18:46:25just points to the data pages without
18:46:27changing anything physically with your
18:46:30data. But in reality what happen is that
18:46:33you can have those two types of indexes
18:46:35the clustered and the nclustered indexes
18:46:38in one table. So one can happen the leaf
18:46:41level of the nclustered index going to
18:46:44be pointing to the data pages of the
18:46:46clustered index because those index
18:46:48pages don't care whether those pages are
18:46:51sorted or not. It's just going to go and
18:46:54point to the correct page and to the
18:46:56correct row. So that means we have now
18:46:58like two different B3 structures that
18:47:01are pointing to the data. And here there
18:47:03is like one thing that you have to
18:47:04understand that that you can create only
18:47:08one clustered index on a table. And this
18:47:11rule really makes sense because you can
18:47:12sort the data only in one way in SQL.
18:47:15And that's of course makes sense because
18:47:17you can sort the data physically only
18:47:19once. And that's why in SQL databases
18:47:21you are allowed to create only one
18:47:24clustered index because physically the
18:47:26data can be sorted only in one way. But
18:47:29in the other hand in the non-clustered
18:47:30index you can create as many
18:47:33nonclustered index you need. So you can
18:47:35create three four and all of them are
18:47:37pointing to the same data pages because
18:47:40in the B tree of the non-clustered index
18:47:42you don't store any data pages. We store
18:47:44only pointers to the data and you could
18:47:46have like multiple pointers. So this is
18:47:48the most important and the main
18:47:50difference between those two indexes.
18:47:52Now if you put it side by side, we have
18:47:54learned that the clustered index going
18:47:56to go and physically sorts and stores
18:47:59the rows at the B tree. But the
18:48:01nclustered index is going to go and
18:48:03create a separate p structure with
18:48:06pointers to the actual data. And by the
18:48:08way, the clustered index we call it the
18:48:11main index that we could use in each
18:48:14table. So the clustered index is the
18:48:16main one, the most important one that
18:48:18you can go and use in each table in your
18:48:21database. Now as we learned if you are
18:48:22talking about the number of indexes you
18:48:25can create maximum one index for each
18:48:27table but for the nclustered index there
18:48:30is no limitations you can go and create
18:48:32multiple indexes for each table. And now
18:48:35if you go and compare them about the
18:48:37read performance how fast we can get
18:48:39data using clustered index. Well it is
18:48:42faster than the nclustlustered index.
18:48:44And that's because in the nonclass and
18:48:46index we have this extra layer at the
18:48:49leaf node from the B tree and because of
18:48:52this having extra layer that means SQL
18:48:54has to do extra job in order to find the
18:48:56data that's why clustered index is
18:48:59faster than the nonclustered index but
18:49:01now in the other hand if we are talking
18:49:03about the right performance how fast we
18:49:06can insert data to the tables well
18:49:09writing data to a table with a clustered
18:49:11index is slower than the nclustered
18:49:14index. And that's because as you are
18:49:16inserting data to the table, SQL has
18:49:18always to check the databases is
18:49:21everything sorted correctly and if not
18:49:23SQL has to go and start physically
18:49:26sorting the data again in order to have
18:49:28the correct order. So there is a lot of
18:49:30stress in order to sort the data with
18:49:32the clustered index. But in the other
18:49:34hand in the non-clustered index we don't
18:49:36have this. So the physical data going to
18:49:38stay as it is. We are just creating nice
18:49:41new pointers. So if you are writing to a
18:49:43table where you have a clustered index,
18:49:45it's going to be slower than writing to
18:49:47a table where you have nclustered index.
18:49:49And of course the fastest way to write
18:49:52data to a table is to not have indexes
18:49:55at all. So a heap structure. So SQL just
18:49:58go and start inserting data inside those
18:50:01databases without creating any extra
18:50:03structures. So as you can see it's like
18:50:05always a tradeoff. You can read fast but
18:50:08you're going to write slower. So you
18:50:10cannot have like everything. Now we are
18:50:12talking about the storage efficiency.
18:50:14The clustered index going to be better
18:50:16with the storage than the nonclustered
18:50:18index and that's because of the same
18:50:20reason with the nonstructured index. We
18:50:22have this extra layer of index pages and
18:50:25index pages needs storage and that's why
18:50:27they can waste more storage than the
18:50:29clustered index. Now if you're talking
18:50:31about the use cases when to use
18:50:33clustered index. Well, if you have like
18:50:35a column this column has to have few
18:50:37criteria in order to be good candidate
18:50:39for the clustered index. First, it's
18:50:41going to be good if the values inside
18:50:43the columns are unique. And second, and
18:50:45it is way more important than that, the
18:50:48values of this column should not change
18:50:50a lot because if this column having a
18:50:53lot of update operators and the data is
18:50:55keep changing, that means each time SQL
18:50:58going to go and start sorting the data
18:51:00again left and right. So having a column
18:51:02that is frequently changing, it's not
18:51:05good for clustered index. And that's why
18:51:08the primary keys of tables are a perfect
18:51:11candidate because first they are unique
18:51:13and second we will never go and update a
18:51:15primary key value. We always append a
18:51:18new primary key value and that's why
18:51:20primary keys are perfect for clustered
18:51:23index. And one more thing where I go and
18:51:25use clustered index is that to optimize
18:51:27the performance of a range query. If you
18:51:30are quering the data between one value
18:51:32and another one clusters index works
18:51:34really well. Now in the other hand if we
18:51:36are talking about the non-clustered
18:51:37index we could use it on coms that are
18:51:40used in the search conditions or if you
18:51:42are joining tables without using the
18:51:44primary keys then you can go and apply
18:51:46the nclustered index in order to have
18:51:48faster joins or you can go and use it to
18:51:50optimize the performance if you are
18:51:52searching for an exact value exact
18:51:55match. So those are the main and
18:51:57important differences between the
18:51:59clustered and the nclustered indexes.
18:52:05All right. So now before we go to SQL
18:52:07and start practicing, I would like to
18:52:09show you the syntax of the index. So
18:52:12it's very very simple. It start with
18:52:14create and then we can define whether it
18:52:17is clustered or nonclustered and then
18:52:19the keyword index. But this section is
18:52:22optional. So if you don't define
18:52:24anything, the default going to be the
18:52:26nonclustered. So if you say create index
18:52:29the SQL server going to go and create
18:52:31nclustered index. Then after that we
18:52:33have to go and define the name of the
18:52:35index and then we have to tell SQL which
18:52:38table we have to create the index in on
18:52:40table name and then we can go and define
18:52:44one column or multiple columns for the
18:52:46index and we call an index with multiple
18:52:49columns as composite index. So for
18:52:51example we can go and create a clustered
18:52:53index using this command create
18:52:56clustered index the index name and then
18:52:58we specify the table and the ID. So we
18:53:01are saying create clustered index based
18:53:04on this column the ID from the table
18:53:06customers. And if you want to create a
18:53:08nclustered index you say create
18:53:10nclustered index and the same thing. So
18:53:12so far we are using one column in the
18:53:14index but we can go and create a
18:53:16composite index with multiple columns
18:53:18like the following example. So we can
18:53:20say create an index and as you can see
18:53:22we skipped here defining the type and
18:53:25that's because the default going to be
18:53:27nonclustered index. And now here we are
18:53:29specifying two columns the last name and
18:53:31the first name. And as you can see we
18:53:33specifying as well for SQL how to sort
18:53:36the data. So we are saying last name
18:53:38should be sorted inside the data page
18:53:41ascending lowest to the highest but the
18:53:44first name should be the way around from
18:53:46the highest to the lowest. So you can
18:53:48control how the data going to be sorted
18:53:50physically in the data page. So as you
18:53:52can see it is very simple. This is the
18:53:54syntax for creating index in SQL. All
18:53:57right. So back to SQL and the first
18:53:59question is where do we find indexes in
18:54:01the database? Well you can go and
18:54:03explore it. If you go to the object
18:54:05explorer over here and check any tables
18:54:06from our sales DB for example the
18:54:08customers and here you have a folder
18:54:10called indexes. So if you expand it you
18:54:13will find here an index. I didn't create
18:54:16any of those indexes in the database.
18:54:18But in SQL server, if you define any of
18:54:20the columns as a primary key, the SQL
18:54:23server going to go by default creating a
18:54:26clustered index for the primary key
18:54:29because it makes always sense to create
18:54:30a clustered index on the primary key. So
18:54:33this one is created as a default and as
18:54:35you can see at the start we have like a
18:54:36key primary key customer and then it is
18:54:39clustered. Now I would like to start
18:54:40from the scratch. That's why I would
18:54:42like to go and create a new table
18:54:44without any indexes. So what we're going
18:54:46to do, we're going to go and load the
18:54:48table customers into a new table. So how
18:54:50we going to do that? We're going to go
18:54:51and say select star from sales
18:54:56customers and before the from we're
18:54:58going to say into a new table. So it's
18:55:01going to be TB customers. So like this.
18:55:05Let's go ahead and execute it. So now if
18:55:07you go to the left side and refresh the
18:55:09tables you can find we have now a new
18:55:11table called DB customers. Now let's go
18:55:14and check whether we have any indexes
18:55:15inside it. So indexes it is empty. So we
18:55:19don't have anything no clustered index
18:55:21or anything else. And this table has the
18:55:23structure of heap structure. So the data
18:55:27are inserted there randomly. It is not
18:55:29sorted. And if I go over here and for
18:55:31example, let's say I'm going to select
18:55:34from this new
18:55:35table where customer ID equal one and I
18:55:40execute it. The SQL server did a full
18:55:43scan on the table in order to find this
18:55:46customer ID. So our new table DB
18:55:48customers is heap cluster. But let's go
18:55:52and change that. What we're going to do,
18:55:53we're going to go and create a new
18:55:55clustered index. So we're going to say
18:55:57create
18:55:59clustered index and then we're going to
18:56:01go and give it a name for the index. We
18:56:04usually follow the following index. So
18:56:06we have index as prefix and then after
18:56:08that we specify the table name. So DB
18:56:11customers and then the key for the
18:56:14index. So the column that we are using
18:56:16in order to index the table. This is
18:56:18important to stick with the same naming
18:56:20convention for the index name because
18:56:22later as you are monitoring your
18:56:24indexes, it's going to be really easy to
18:56:25understand. Okay, this index is for the
18:56:28table DB customers and we are using the
18:56:30customer ID to index. So now after that
18:56:33we're going to go specify on which table
18:56:35we are doing the index. So on sales DB
18:56:40customers and then we're going to
18:56:42specify the column name. So we are
18:56:44saying build for me a clustered index
18:56:46based on the customer ID. So now let's
18:56:49go and execute it. So as you can see
18:56:51it's very fast because we have only five
18:56:53rows. So the database just switched all
18:56:55the data pages very fast. Now let's go
18:56:58and check our new index. So let's go and
18:57:00refresh and let's go inside it. And now
18:57:02we can see that we have our new index
18:57:04clustered index based on the customer
18:57:06ID. Now as we learned we cannot create
18:57:08multiple clustered index. But let's go
18:57:10and test that. So I will just take the
18:57:13whole thing and let's say I would like
18:57:15to create a class index based on the
18:57:17first
18:57:18name as well
18:57:22here. So let's go and execute it. So as
18:57:25you can see saying you cannot create
18:57:28more than one clustered index on this
18:57:30table. That means we can create only one
18:57:32clustered index. And let's say that
18:57:35after you created the index you chose
18:57:37the wrong column and you would like to
18:57:39change it to the first name. So what
18:57:42we're going to do, we have to go and
18:57:43drop the index. So we say drop index and
18:57:48then you need the index name. It was
18:57:50this one. And then you have to specify
18:57:52which table. So it's going to be sales
18:57:55DB
18:57:57customers like this. So if I do it like
18:58:01this and let's go and refresh again. You
18:58:03can see that we don't have any indexes
18:58:05anymore and the table is packed as a hip
18:58:08structure. And now you can go and create
18:58:10the correct clustered index for this
18:58:12table. But to be honest, I'm going to
18:58:14stick with the customer ID. So I will
18:58:16not create a clustered index on the
18:58:18first name because the first name of
18:58:19course is not unique. You can have like
18:58:21maybe multiple customers having the same
18:58:23name. And as well updates could happen
18:58:26on the first name and that's going to be
18:58:27very expensive. So that means I'm going
18:58:29to stick with my index on the customer
18:58:32ID. Let's go and execute it. And now I
18:58:34have again my index on my table. Now
18:58:38let's say that that I have the following
18:58:40select statements from our tables. So
18:58:44customers and I'm searching for the last
18:58:48name where let's say we are searching
18:58:50for brown. So let's go and execute it.
18:58:53So let's say that we are getting more
18:58:54and more customers and our table is
18:58:56getting bigger and I frequently use this
18:58:59query. So I'm searching for specific
18:59:01customers using the last name. So what
18:59:04we can do, we can go and create a
18:59:06nonclustered index for the last name in
18:59:08order to improve the performance of this
18:59:10query. So let's go and create that. So
18:59:12we're going to say create
18:59:14nonclustered index. And now we're going
18:59:17to give it the name using the naming
18:59:18convention. So DB customers and we're
18:59:22going to use the last name for this
18:59:23index. So on
18:59:26sales DB customers and we will use the
18:59:30column last name for the index. So let's
18:59:33go and execute it. And now if you go to
18:59:36our indexes and refresh, we will find
18:59:39our new index over here. And as you can
18:59:42see, it says it is nonclustered and as
18:59:45well non-unique. We will talk about the
18:59:47uniqueness later. So as you can see,
18:59:48it's very easy. We have just created a
18:59:50uncclustered index on the last name. And
18:59:52now as we learned, we can go and create
18:59:54multiple nonclustered index on the same
18:59:57table. Let's say for example, now we our
19:00:00query looks like this. We are searching
19:00:01for the first name using for example the
19:00:04value Anna. And now this query happens a
19:00:07lot and maybe slow. So we can go and
19:00:09create new nonclustered index. So let me
19:00:12just have it like this. And for the
19:00:14nonclustered index you don't have to
19:00:16specify always like nonclustered index.
19:00:19As default it's going to be
19:00:20nonclustered. So we can skip that. And
19:00:23here let's call it first name. And the
19:00:26column that we are using is the first
19:00:29name. So let's go and create this index.
19:00:33And now let's go and refresh our
19:00:35indexes. And as you can see, SQL did
19:00:37create a nonclustered index for the
19:00:40first name. So if you don't specify the
19:00:42type of the index, it's going to be as a
19:00:44default nonclustered
19:00:49index. All right. So now let's talk
19:00:51about the composite index. It is an
19:00:53index that has multiple columns inside
19:00:56the same index. So far we have used only
19:00:59one column in the index but we can go
19:01:02and specify multiple columns and that's
19:01:04because sometimes our wear conditions
19:01:07are complicated and based on multiple
19:01:09columns. So for example let's say that
19:01:11we are searching for country equal to
19:01:15USA and at the same time we are saying
19:01:18the score should be higher than 500. So
19:01:21that means in this condition we are
19:01:23using two columns and we would like to
19:01:25speed up this query. So how we going to
19:01:27do it? So we're going to go and create
19:01:29let's say an index and give it a name DB
19:01:33customers and let's say country score on
19:01:39sales DB customers. And now it is very
19:01:43important to do the following thing. Now
19:01:45we have to go and define a list of
19:01:48columns that we want to be included in
19:01:50this index. And it is very crucial and
19:01:53important that you get the same order as
19:01:56your query. So your query start with the
19:01:59country and then the score. You have to
19:02:02do it the same thing in the index. So
19:02:04the first column it's going to be the
19:02:06country and then the score. So it must
19:02:09be the same order as your query. So
19:02:12let's go and create this index. And if
19:02:15you go to the indexes over here, you can
19:02:16see that we have created our new index.
19:02:19So now once you create such a index and
19:02:22your table going to be like always
19:02:24updating this index you have to be
19:02:25committed and responsible. So in your
19:02:28queries if you want to filter the data
19:02:29using country and score always start
19:02:32with the country then the score in order
19:02:34to be able to use the index optimizer.
19:02:38So if you do it like this the index
19:02:40going to be working but if you go and
19:02:42query the way around. So you start with
19:02:45the score and then the country the SQL
19:02:48will not be using your index. So either
19:02:51you adjust your queries or you have to
19:02:54go and recreate the index based on this
19:02:57switch. So be very careful with the
19:02:59composite indexes. The order is very
19:03:02crucial. So you're going to have it
19:03:04exactly like the query. And now you
19:03:07might say you know what now we have like
19:03:08a nice index for those two columns. What
19:03:11going to happen if I go and use in my
19:03:14query only one of them like for example
19:03:16the country. So now the question is if I
19:03:18go and execute this query is the SQL is
19:03:22using this index even though that I
19:03:24don't have the score. Well yes because
19:03:27it follows the leftmost prefix rule. So
19:03:30this means SQL can use the index if you
19:03:33are using always the lift columns. So
19:03:36here in our index country is on the left
19:03:39that's why it is working over here. But
19:03:42if you go and skip the lift column it
19:03:44will not work. So if you go over here
19:03:46for example and say let's go and select
19:03:49only the
19:03:51score and it is like higher than 500.
19:03:55What we have done, we have skipped the
19:03:57country in this query and that's why it
19:04:00will not be working. So as long as you
19:04:02are including the left columns, it will
19:04:05work even though it is only one column.
19:04:07So in this scenario, the first query
19:04:09going to use the index, the second one
19:04:12will not be using it. So now let me give
19:04:13you a very simple example in order to
19:04:15understand how this works. So let's say
19:04:17that we have an index using four columns
19:04:20A, B, C, D. Now in your query if you go
19:04:22and target the column A the index going
19:04:24to be used. Now the same thing going to
19:04:26happen if you go and use A and P. So if
19:04:29you're using those two columns you will
19:04:31be using the index. So those are where
19:04:34the index will be used. So now let's
19:04:38have the scenarios where the index wants
19:04:42be used. So for example if you go and
19:04:44just jump immediately to the column B.
19:04:47So you are not using the left column the
19:04:49A that's why you will not be using the
19:04:52index and as well in your query if you
19:04:54are using A and you are skipping the P.
19:04:56So you have A and then C you will not be
19:05:00using the index. So you have always to
19:05:02use always the lift columns. So here if
19:05:05you are using A B C you will be using
19:05:08the index. And let's see here you are
19:05:09using A B and then you jump and skip to
19:05:12the D you will not be using the index.
19:05:14So this is what we mean with the
19:05:15leftmost prefix rule by using the
19:05:18composite index. So if you're using
19:05:20multiple columns inside one index, be
19:05:22careful with the order of the columns
19:05:24that you are defining. All right. So
19:05:26that's all for this category, clustered
19:05:28and uncclustered index. Now we're going
19:05:30to move to the second category where we
19:05:32talk about the indexes by the storage,
19:05:35the row store and the column store.
19:05:41So now let's say that we have a table we
19:05:43have multiple rows and multiple columns.
19:05:45Now if we use a row store index this is
19:05:48the classical one. What going to happen?
19:05:50Our table going to be splitted into
19:05:52multiple rows. And as we learned each
19:05:54group of rows going to be stored inside
19:05:56a data page. So that means we are
19:05:58organizing the data row by row which
19:06:01means all the columns for each row going
19:06:04to be stored together. This is the
19:06:06traditional way on how the databases
19:06:09organize their data where the
19:06:11informations are stored row by row. But
19:06:13now in the other side if you use column
19:06:15store index the SQL going to go and
19:06:17split your table into multiple separate
19:06:19columns and then SQL going to go and
19:06:21store the values of one column together
19:06:24in data page. So that means if you go
19:06:26and open a data page you will find only
19:06:29the values of one column. You will not
19:06:31find the entire row. So if it's like the
19:06:34first name you will see only the first
19:06:36name informations you will not see the
19:06:37last name information in this data page.
19:06:39So if you compare them the row store
19:06:41index stores the data row by row the
19:06:44column store index stores the data
19:06:46column by column. So this is a very high
19:06:48level representation on how the column
19:06:50store index is stored. As you know me we
19:06:53go in details in order to understand
19:06:55exactly how SQL works with the column
19:06:58store index. So let's go.
19:07:03All right. So now let's say that we have
19:07:05a table for the customers. We have three
19:07:07columns ID, name and status. And as well
19:07:10we have around 2 million rows, 2 million
19:07:12customers. And as we learned as a
19:07:14default, the table going to be built as
19:07:16a heap structure where the rows are
19:07:18stored row by row inside data pages. But
19:07:20now we go and create a column store
19:07:23index on top of this table. So now once
19:07:25you do that SQL going to go through a
19:07:28process in order to build the column
19:07:30store. So the first step is SQL going to
19:07:32go and divide the data the rows into row
19:07:35groups. Now in SQL server each row group
19:07:38can contain around like 1 million row.
19:07:40So in this example our table going to be
19:07:42splitted into two row groups. The first
19:07:45one million row in one group and the
19:07:47second one in another row group. Now you
19:07:49might ask me we are talking about
19:07:50columns. Why we are splitting the rows?
19:07:53Well, this is just a pre-step in order
19:07:55just to optimize the performance and to
19:07:57do parallel processing. And of course,
19:07:59the data will not be stored like this
19:08:01because we have the second step. Now, in
19:08:03the next step, SQL going to go and
19:08:05segment the columns. So now, SQL will go
19:08:08for each row group and start splitting
19:08:10the data by the columns. And that's why
19:08:13we call it a column store because we are
19:08:15separating the columns from each others.
19:08:17So that means we have one segment for
19:08:19the ID, another one for the name and a
19:08:22third one for the status. And this can
19:08:24happen for each row group. And now it's
19:08:26going to move to the third step in this
19:08:28process. We have the data compression.
19:08:31And this is the most important step in
19:08:33this process because it is the reason
19:08:36why column store is very fast compared
19:08:38to the ro store. So in this process
19:08:41there are like different techniques on
19:08:42how to do data compression and the most
19:08:45famous one is that it's going to go and
19:08:47create like a dictionary. Let's take for
19:08:49example the column status the status of
19:08:52the customer whether it is active or
19:08:54inactive. So the word active and
19:08:56inactive going to be repeated like 2
19:08:59million times because we have 2 million
19:09:01customers and since it is like string it
19:09:03is like taking a lot of space and
19:09:05storage. But now instead of that we're
19:09:07going to go and compress the data. So
19:09:09first it's going to go and create a
19:09:10dictionary by replacing the value active
19:09:14and inactive into smaller values like
19:09:16one and two. So we have like a mapping
19:09:19between the long value to a small value.
19:09:22And after that SQL going to store like a
19:09:25data stream where we have like only two
19:09:27values one two one two. So we're going
19:09:29to have like a big stream of 2 million
19:09:31rows. So it's going to go and do this
19:09:33for each column and with that the size
19:09:35of each column going to be changed
19:09:37depends of course on how much different
19:09:39values you have in each column. So this
19:09:41step is very important in order to
19:09:43reduce the size of the data and as well
19:09:45to increase the performance. So now once
19:09:48everything is organized and compressed,
19:09:50SQL going to go and start storing the
19:09:53results in databases. But TSQL will not
19:09:56use the standard databases that we have
19:09:58learned previously. But instead going to
19:10:00use a special database called LOB large
19:10:04object page. So now let's quickly
19:10:06compare the structure of the normal
19:10:07database that we have learned in the row
19:10:09store with the new one, the column
19:10:11store, the LOB data page. So as usual
19:10:14each page has a header. This is same as
19:10:16any data page. But the next section is
19:10:18going to be the segment header. It has
19:10:20like metadata informations about the
19:10:23column segment that is stored in this
19:10:25page. Like we have the segment ID, the
19:10:28row group ID, the column ID and it has
19:10:31as well very important information the
19:10:33ID to the dictionary page. So the
19:10:35dictionary page is as well a type of
19:10:38pages in SQL. It has as well a header
19:10:41but inside it we have like a mapping. So
19:10:43it maps the original value, the long
19:10:45one, the inactive to the smaller version
19:10:48of this value, for example, one. And
19:10:50that's all for the dictionary page. It
19:10:52has the mapping between the original
19:10:54values and the smaller values. And
19:10:56beneath the segment header, we can have
19:10:58now the important place where our data
19:11:00can be stored. We have the data stream.
19:11:03So it is like sequence of ids from the
19:11:05dictionary that represents the values of
19:11:08the columns side by side. And of course,
19:11:11we cannot fit the whole 1 million rows
19:11:13inside this data stream. We're going to
19:11:15have like multiple LOP databases. So
19:11:18this is how exactly the SQL stores your
19:11:21data. If you decided to go with the
19:11:23column store, so let's go back to the
19:11:25process. So back to the process. As you
19:11:27can see, SQL is storing the data as LO
19:11:30data storage. So this is the last step
19:11:32and with that SQL did convert your table
19:11:35into a column store. So now we cannot
19:11:37just create a column store without
19:11:39defining whether it is clustered index
19:11:42or non-clustered index. So let's start
19:11:44with the first one the clustered column
19:11:46store index. So if you create such a
19:11:48index SQL of course will not be building
19:11:51a B3 structure. SQL going to use exactly
19:11:54this structure the column store
19:11:56structure. So as we learned the cluster
19:11:58index is a complete makeover of your
19:12:00table. when you apply it then SQL going
19:12:02to format everything column-wise and it
19:12:05is fully replacing the old row based
19:12:08table structure that we have at the
19:12:09start. So once you apply the clustered
19:12:12column store index it will not leave
19:12:14anything behind and your table going to
19:12:16be completely structured as a column
19:12:18store and one more thing which is makes
19:12:20sense of course all the columns from the
19:12:23original table going to be converted to
19:12:24a column store. So it is not leaving
19:12:27anything behind it. But in the other
19:12:29hand, if you are using non-clustered
19:12:31column store index, as we learned, it is
19:12:34like a companion to your existing table.
19:12:37So it coexist with the table and it will
19:12:39not replace anything. So the column
19:12:42store index can be an additional thing
19:12:44that is stored beside your table. So
19:12:47that means the original table will not
19:12:49be deleted at all like the clustered
19:12:51column store index. The first one is in
19:12:54the old row based storage. the regular
19:12:56table, the first one, and your data
19:12:58going to be as well stored in a separate
19:13:01structure in the column store index. And
19:13:03of course, in the non-clustered column
19:13:05store index, since we are creating an
19:13:08extra index outside of your original
19:13:10table, you can go and define which
19:13:12column should be included in this
19:13:13process. It must not be all the columns.
19:13:16You can go for example with only the
19:13:17status. So that means you build a column
19:13:20store index only for one column for the
19:13:22status of the customers. So this is what
19:13:24we mean with the clustered column store
19:13:26index and the nclustered column store
19:13:32index. All right friends, so now you
19:13:35might ask me why we are doing all those
19:13:36stuff. Why I would split my data by the
19:13:39columns? Well, it's all because of
19:13:42analytics. Because in analytics we have
19:13:44like big complex query where we have a
19:13:46lot of data aggregations and stuff on
19:13:48big tables. And the roster index is
19:13:51perfectly designed in order to improve
19:13:54the performance of such big queries. And
19:13:56that's why SQL databases like SQL server
19:13:59and as well BI tools like Tableau and
19:14:01PowerBI did adopt this methods in order
19:14:05to offer fast platform for data
19:14:07analyzes. So now let's understand
19:14:09exactly why the column store index is
19:14:12way faster for data analyzes than the
19:14:14row store index. So let's go. So again
19:14:16we have the customers tables and let's
19:14:18say we have like five customers where we
19:14:20have ID, name and status and as we
19:14:22learned before if we are using roster
19:14:24index the data can be stored in multiple
19:14:26databases and in each database we're
19:14:28going to have the whole record the whole
19:14:30information about one customer. So for
19:14:33this example we're going to have like
19:14:34three databases but if you are using the
19:14:36column store index it's going to be
19:14:38stored little bit differently. So the
19:14:40first column the id going to be stored
19:14:41in one data page and here the SQL will
19:14:43not go and build a dictionary because
19:14:45the ids are already short. So we're
19:14:47going to have like one data stream with
19:14:49all ids and now for the next column name
19:14:51is going to be stored in separate data
19:14:53page where we're going to have an extra
19:14:55dictionary page where each name going to
19:14:57be mapped to one small value. So the
19:14:59data going to be compressed and we're
19:15:00going to save storage. Now the database
19:15:02going to create for the third column the
19:15:04status one more data page and the
19:15:06dictionary here going to be very small.
19:15:08So for active we're going to have one
19:15:09and for the inactive we're going to have
19:15:11two and in the data stream we will be
19:15:13storing only the ids of the dictionary.
19:15:15So now let's understand why the column
19:15:17store is faster. Let's have the
19:15:19following query. We want to find the
19:15:21total number of customers that are
19:15:23active. So we have the query select
19:15:26count star from customers and we're
19:15:28going to filter the data by the status
19:15:30where it is equal to active. So now if
19:15:32we query the table with the row store
19:15:34what can happen? SQL have first to go
19:15:36and collect the data. So it's going to
19:15:38go to the first data page and collect
19:15:39the first two customer then to the
19:15:41second to the third and so on. And as
19:15:43you can see SQL here is reading
19:15:45everything the whole row the ID the name
19:15:48the status even though that for the
19:15:50query we actually we don't need all
19:15:52those informations we just need to count
19:15:54how many customers we need with the
19:15:56status active but still cannot go and
19:15:59selectively only reading the status has
19:16:03to read the whole record. So after SQL
19:16:05has all the data it's going to go and
19:16:07filter the data. So it's going to go and
19:16:08remove the inactive rows and then SQL
19:16:11going to do the aggregate operation and
19:16:13with that we're going to get three rows.
19:16:16So that's why the total count of active
19:16:18customers going to be three. But now
19:16:19let's see how SQL going to query the
19:16:22column store. So SQL first have to
19:16:23analyze okay which columns do I need
19:16:26actually for this query. Well, we need
19:16:28only the status. So SQL will not go and
19:16:31open all three data pages and read it.
19:16:33SQL will target only one data page the
19:16:36database where we have the column
19:16:37status. So it's going to take this very
19:16:39simple data stream and then it's going
19:16:41to go and understand the dictionary and
19:16:43it going to go and remove all the values
19:16:45where it is equal to two. So without in
19:16:47the output we have only three values and
19:16:49SQL going to go and do a very quick
19:16:51count for those values. So in the output
19:16:53we will get as well three total number
19:16:55of active customers. So now if you
19:16:57compare this intermediate result sets
19:17:00from the row store and the column store
19:17:02you can see that in the row store we
19:17:04have fetched and retrieved a lot of
19:17:06unnecessary informations for this query
19:17:08and this of course going to make the
19:17:10speed of the query very slow but in the
19:17:13column store reads exactly what it needs
19:17:15for this aggregation and we didn't read
19:17:18any extra informations about the names
19:17:20of the customers the ids it didn't like
19:17:23open any extra data pages it exactly
19:17:25gets the data that it needs for the
19:17:28aggregation and that's exactly why the
19:17:30performance of queries where we have
19:17:32aggregations and data analyzes is going
19:17:35to be very fast if you are using column
19:17:37store compared to the row store. So
19:17:40that's why we use column store for big
19:17:42data and data analytics. All right. So
19:17:44now let's summarize the differences
19:17:46between the row store and the column
19:17:47store indexes side by side. So let's
19:17:50start by the definition. The row store
19:17:52going to go and organize and store the
19:17:53data row by row. It is really nice
19:17:55method if you need a lot of columns in
19:17:57one row. But in the other hand, the
19:17:59column store index going to go and store
19:18:01the data and organize it column by
19:18:03column which is really great if you're
19:18:05focusing on specific column. Now if you
19:18:07are talking about the storage
19:18:08efficiency, the row store index going to
19:18:10take more space compared to the column
19:18:13store index and that's because as we
19:18:14learned the column store going to go and
19:18:16compress the data which going to save a
19:18:18lot of storage if you have large tables.
19:18:20Now to the next point which is more
19:18:22important about the performance. The
19:18:24read and write optimizations we can say
19:18:27for the row store things are more
19:18:28balanced. So you will get a decent speed
19:18:31for both write and read operations but
19:18:33things in the column store is different.
19:18:36It is fast for reading especially if you
19:18:38are doing data analytics but writing
19:18:40data like inserting and updating it is
19:18:43slower because as we learned there are
19:18:45like multiple steps until the data is
19:18:48written in the pages. So in one hand you
19:18:50are optimizing the speed of your
19:18:52analytical queries but in the other hand
19:18:55changing data it is slower than the
19:18:57roster index. Now let's talk about the
19:18:59next point input and output efficiency.
19:19:01Well the roster index it's not really
19:19:03good because you are retrieving a lot of
19:19:05columns. So a lot of data should be read
19:19:08from the disk storage in order to answer
19:19:10your queries. But in the other hand for
19:19:12the column store it is lower and that's
19:19:15because it targets exactly the data and
19:19:18columns that is needed for the query. So
19:19:20there will be generally less data that
19:19:22is read from the disk storage and of
19:19:24course that's why we are getting fast
19:19:26read performance. So now if you are
19:19:28thinking which systems are best for ro
19:19:30store index well the roster index is
19:19:32very suitable for the OLTB systems
19:19:35online transactional systems like
19:19:37banking and commerce systems where the
19:19:40full records are accessed very
19:19:42frequently but in the other hand the
19:19:44column store index is great for OLAP.
19:19:46All app systems are online analytical
19:19:49processing where you have like data
19:19:51warehouses, data league, business
19:19:52intelligence. You are building reports
19:19:54and analyzes. You have large data sets
19:19:57and very complicated aggregated queries.
19:20:00So if you have such a project then the
19:20:01column store index is the way to go. So
19:20:04that means the use case for the row
19:20:06store index if you have high frequency
19:20:08transactions where the system has to
19:20:10quickly access records and the use case
19:20:13for the column store is big data
19:20:15analytics where the SQL has to scan
19:20:17large data sets. So those are the main
19:20:19differences between the row store index
19:20:22and the column store
19:20:27index. All right. So now let's check the
19:20:29syntax of the column store index. Well,
19:20:31it is really easy what we're going to
19:20:32do. we can just put a column store
19:20:34keyword between the clustered or
19:20:37nonclustered and the index. So once you
19:20:40specify that then you are telling SQL
19:20:42you want to create a column store index
19:20:44and the rest is going to stay as it is.
19:20:46Now if you want to create row column
19:20:48store then you don't have to specify
19:20:50anything. There is no keyword for the
19:20:52row store. So as we learned before we
19:20:54can go and create a nonclustered index
19:20:56and cluster on the index and both of
19:20:57those syntax is going to tell SQL we are
19:21:00creating row store index but if you go
19:21:03and use the column store keyword then
19:21:06you are telling SQL that you want to
19:21:07create either clustered or nclustered
19:21:10column store index and here there is
19:21:11like a syntax rule if you are creating a
19:21:14clustered column index then you must not
19:21:17specify anything for the columns. So you
19:21:19cannot go and specify anything like an
19:21:21ID or country or any columns over here
19:21:24because it makes no sense once you say
19:21:26cluster column store then all the
19:21:28columns going to be included in the new
19:21:31structure. So this is the syntax of the
19:21:34column store index. All right. So back
19:21:36to scale let's check how we can create
19:21:37column store index. Now if you check our
19:21:39table here DB customers that we have
19:21:41created previously and we go to the
19:21:43indexes you can see that we have created
19:21:46few indexes and one of them is the
19:21:48clustered index. This one is a row store
19:21:52index. So our table is splitted by the
19:21:55rows. Now let's go and change that.
19:21:57Let's make our table splitted by the
19:21:59columns using the column store. So we're
19:22:01going to say
19:22:02create
19:22:04clustered column store index and we're
19:22:08going to give it the name index DB
19:22:11customers and it's going to be on the
19:22:13table sales DB customers and here if you
19:22:18go and specify a column it's going to be
19:22:20a mistake. So let's go and check that.
19:22:23So if you go and execute it says it
19:22:25fails because key lists or the columns
19:22:28is not allowed. So we cannot have this.
19:22:30So let's remove it. And now we have the
19:22:32correct syntax. Let's execute it again.
19:22:35We will get another error because it
19:22:37says in one table you cannot have more
19:22:40than one clustered index. We have
19:22:42already one. You have to decide do you
19:22:44want to split your table by columns or
19:22:46by rows. That's why we have to go and
19:22:48drop the previous index. So we're going
19:22:50to do it like this. Drop index. And I
19:22:54need the name of the index like this.
19:22:56And then we have to specify the table
19:23:01name. So that's it. Let's drop the
19:23:04index. Now if you refresh, we cannot see
19:23:07anymore our clustered index and our
19:23:09query should be working. So let's do
19:23:11that. Now let's check the indexes again.
19:23:14And now as you can see, we got a new
19:23:17clustered index, but this time it is
19:23:19column store. Now you can see at the
19:23:21start we have like an icon. This looks
19:23:23like a bar chart or like analytics and
19:23:26reports and that's because the main
19:23:27purpose of creating com store is to have
19:23:30a bar chart. So now of course we cannot
19:23:32go and create multiple clustered column
19:23:34index. We can have maximum only one. So
19:23:37now if you say you know what let's go
19:23:38and create for the first name another
19:23:41index but this time it's going to be a
19:23:42column store. So if I go and copy the
19:23:45whole thing over here and let's say it
19:23:47is none clustered column index and let's
19:23:51call it for example first
19:23:53name and we define over here the first
19:23:58name. So that's it. Let's go and execute
19:24:01it. You will see that we will get an
19:24:03error where SQL tells us you cannot
19:24:06create multiple column store indexes.
19:24:09That means you can create only one
19:24:11column store index for each table and
19:24:14you have to decide whether it is a
19:24:16clustered or non-clustered and you
19:24:18cannot create like the row store
19:24:20multiple non-clustered index. So you are
19:24:23allowed only with one column store index
19:24:26but this limitation is only here in the
19:24:28SQL server. In other databases I know
19:24:30that is allowed to use multiple column
19:24:32store indexes like in the Azure SQL
19:24:36server you can do that. So now in order
19:24:37to practice and you would like to create
19:24:39a nonclustered column store index, you
19:24:42can drop the first one and you can go
19:24:43and create the one that you need as a
19:24:46nclustered index. So actually let's go
19:24:48and do that. Let's drop the first one.
19:24:50So drop index and this is our index on
19:24:55this table. Let's do that. And once you
19:24:58execute the nonclustered column store
19:25:00index is going to work. And if you
19:25:02refresh over here, you will see that we
19:25:04have a non-clustered column store index
19:25:07for the first name. Okay. So now as we
19:25:09learned that the column store going to
19:25:11go and compress the data and the storage
19:25:13that is needed for the entire table
19:25:15going to be less than the row store. So
19:25:18let's see whether that is really true.
19:25:20Now in order to check this I will not do
19:25:22that in the database sales DB because
19:25:24everything here is already small. We're
19:25:26going to go and use another database. We
19:25:28have the adventure works DW2022 and if
19:25:32you have a newer version that's okay. So
19:25:34now what is the plan? We're going to go
19:25:36and create three identical copies of one
19:25:38table and we're going to have different
19:25:40structures. So the first one going to be
19:25:41the heap structure. The second one going
19:25:43to be row store structure and the third
19:25:46one going to be column store structure
19:25:48and then we're going to go and compare
19:25:49the storage of those three. So now we
19:25:52have to go and pick one of those tables.
19:25:54We need one big table. So for example
19:25:56the fact internet sales. So let's see
19:25:58how we can do that. Let's start with the
19:25:59heap structure. We're going to say
19:26:01select star into a new table. So it's
19:26:05going to be the
19:26:06fact internet sales and underscore hp
19:26:11for the heap. And we're going to get it
19:26:13from the table fact internet sales. So
19:26:18like this. And here it's very important
19:26:19if you are switching databases you have
19:26:21to go and use the database. So it's
19:26:23going to be use adventure work DW 2022.
19:26:27So execute this at the starts to make
19:26:30sure that you are switching to the new
19:26:32database. And now let's go and execute
19:26:34our heap structure. So with that we have
19:26:36created heap table as you can see 60,000
19:26:39rows. And since we didn't define any
19:26:41clustered index this table going to be
19:26:44heap structure. Now let's go and create
19:26:46another table where we use clustered row
19:26:49store index. So what we're going to do,
19:26:51we're going to copy the whole thing over
19:26:53here and we're going to call this row
19:26:55store and we're going to go of course
19:26:57change the name to RS but still we are
19:26:59targeting the same table. So let's go
19:27:02and execute this at the start. But now
19:27:04in order to make it as clustered row
19:27:06store we have to go and create an index.
19:27:08So it going to be like this create
19:27:12clustered index. We don't have to
19:27:14specify the row store because it is as a
19:27:17default. It's going to be ro store. So
19:27:18let's call it
19:27:20index facts
19:27:23internet sales RS and then the primary
19:27:27key. So B key and now we need the
19:27:31table fact internet sales RS and now we
19:27:36need the columns the primary key well
19:27:39actually I don't know what is the
19:27:40primary key so let's go and check that
19:27:42so it is a composite primary keys so
19:27:45it's going to be the sales order number
19:27:49and sales order line number like this.
19:27:53So let's go and execute this. And with
19:27:56that we have clustered row index. I'm
19:27:58going to go and check what do we have
19:28:00over here. So let's go and refresh
19:28:01everything. So we have now two tables
19:28:04the heap and the row store. So let's
19:28:07extend it and check the indexes. And as
19:28:09you can see we have the clustered index.
19:28:12Now we need the third table. It's going
19:28:14to be the column store index. I'm just
19:28:16going to go and copy the whole thing
19:28:17over here. So this is the column store
19:28:22going to be here CS and CS and of course
19:28:27we don't need any columns for the column
19:28:29store and don't forget to add the column
19:28:32store keyword. So create cluster column
19:28:35store index and we have to rename as
19:28:37well over here. So let's go and execute
19:28:40our new stuff. So we create first the
19:28:42table and then we convert it to a column
19:28:45store index. So let's go and do that and
19:28:48we have to go and refresh and check our
19:28:50tables. So this is our third table and
19:28:53let's go and check the indexes and we
19:28:55have clustered column store. All right.
19:28:57So now we are done. We have our three
19:28:59different tables. Now let's go and check
19:29:01the stoages of those three tables. So
19:29:04now let's go and check our first table
19:29:05the heap table. So right click on it and
19:29:08go to the properties. And now we can see
19:29:10here a lot of informations about our
19:29:12table. But we are interested on the
19:29:13storage. So click here on the page for
19:29:16the storage. And now we can see here few
19:29:18informations about the storage and one
19:29:20of them is the data space. It is around
19:29:229 MB and the index space is almost
19:29:25nothing. So we don't have anything over
19:29:27here. So this is the storage of the heap
19:29:30structure. We don't have any indexes.
19:29:31Let's go now to the row store. So we're
19:29:34going to go to the RS and properties.
19:29:37Then let's go to the storage. And now as
19:29:39you can see the data space is exactly
19:29:41the same. And that's because whether it
19:29:43is heap or row store index, we're going
19:29:45to store the data in data pages as rows.
19:29:48So the size of the data itself will not
19:29:50change. It will be sorted differently.
19:29:52But what changed here is the size of the
19:29:55index. Now we are consuming more storage
19:29:58for the index. So that means the overall
19:30:01storage of the table with a cluster draw
19:30:03store index it is more than the heap
19:30:06structure. Let's go and check now our
19:30:08column store index. So to the CS and
19:30:11let's go to the properties. And now it
19:30:13is interesting to see whether our table
19:30:15is getting smaller. So let's go to the
19:30:17storage. And as you can see the data
19:30:19space is around 1 mgabyte compared to
19:30:22the 9 mgabyte. I know those are small
19:30:25numbers but still it is massively
19:30:28reduced space because everything is
19:30:30compressed and of course we are not
19:30:32using any index spaces because we don't
19:30:35have this B3 structure in the column
19:30:38store. So as you can see if you compare
19:30:39to the others it is the winner. This
19:30:42table that is using the column store is
19:30:44consuming way less storage than the
19:30:47others. So now if you want to rank it
19:30:49based on the storage the best one is the
19:30:52column store index table. Then the next
19:30:55one is the table with the he structure
19:30:57and the worst one is the table with the
19:31:01row store clustered index. So that's
19:31:03true. column store index is consuming
19:31:06less space than the other type of
19:31:12indexes. All right. So now what is
19:31:14unique index? Unique index is a special
19:31:16type of indexes that going to make sure
19:31:19no duplicates in your data. And there
19:31:22are a couple of reasons why is it
19:31:23important to have a unique index. The
19:31:25first one and the most obvious reason is
19:31:28to have data integrity. So the unique
19:31:30index going to go and enforce uniqueness
19:31:32in your data and that is very helpful.
19:31:35For example, if you have a column like
19:31:37an email address or a product ID. Having
19:31:41duplicate in such a columns can mess up
19:31:43your data very badly. So having a unique
19:31:46index on a column like an email going to
19:31:48make sure there are no sneaky duplicates
19:31:50inside your data. And the second
19:31:52important reason why unique index is
19:31:55important is to improve the performance.
19:31:57So for example, if you are searching for
19:31:59specific email, the SQL going to start
19:32:01searching for the email value and once
19:32:04the SQL find the value, the SQL will
19:32:06stop searching because we are sure that
19:32:08there is no duplicates in the data. So
19:32:10with that you are improving the
19:32:12performance of your queries. So if you
19:32:14are creating an index and you know this
19:32:16column is unique then make sure to make
19:32:18the index as unique index. So now if you
19:32:21have a look again to our clustered index
19:32:23where we have the B structure if you
19:32:25make this index as unique then you are
19:32:28giving an extra task for the SQL that's
19:32:31going to go and make sure that all those
19:32:34ids of the customer going to be unique.
19:32:37So SQL has to guarantee that there are
19:32:40no duplicates at all inside your data in
19:32:42the databases. So now since we are
19:32:45giving SQL an extra task to prove the
19:32:48uniqueness of the data building the
19:32:50clustered index going to be little bit
19:32:52slower. So that means inserting new data
19:32:55writing data going to be slower as the
19:32:58normal clustered index. But now if you
19:33:00are talking about the read performance
19:33:02the performance of our query it's going
19:33:04to be optimized a little bit faster than
19:33:06a normal clustered index. So again this
19:33:09tradeoff we are making writing data
19:33:12slower but we are gaining more speed on
19:33:15the query performance. So this is what
19:33:17we mean with unique index. Okay. So
19:33:20let's keep extending the syntax of the
19:33:22index. So now in order to tell whether
19:33:23it is unique or not we can specify it
19:33:26exactly at the start. So we say create
19:33:29unique is just before the clustered or
19:33:31nonclustered and then afterward the cl
19:33:33store and nothing changed for the rest.
19:33:35So we can specify this keyword to TSQL,
19:33:38it should be unique. And if you don't
19:33:40write anything before the clustered
19:33:42index, it's going to be not unique. So
19:33:45for example, this one says create an
19:33:47index. So we didn't specify anything
19:33:50here, duplicates are allowed in the
19:33:52index. But if you go and specify a
19:33:54unique index, then the duplicates are
19:33:57not allowed. So it is very simple. Okay.
19:33:59So now let's go and create unique
19:34:01cluster. Now let's go and target the
19:34:03table products. Let's go and first
19:34:05select the data from the table. So sales
19:34:10products and execute it. Now let's see
19:34:12that I'm going to go and create a unique
19:34:14index on the column category. Let's go
19:34:17and try it. So create unique
19:34:21nonclustered index and let's give it the
19:34:24name index products
19:34:29category on the table sales products and
19:34:33we are targeting the column category. So
19:34:37let's go and execute it. Now we will get
19:34:40an error because the category has
19:34:43duplicates. So if you go and query again
19:34:45our table, you can see we have here
19:34:47duplicate values and the SQL cannot go
19:34:50and create unique index for this table.
19:34:53It's too late. But you still can create
19:34:55this index if the table is empty and SQL
19:34:57will not allow you to insert any
19:34:59duplicates about the categories. And of
19:35:02course it makes no sense to have unique
19:35:04index on the categories because of
19:35:06course we're going to get duplicates
19:35:07here. But maybe you say, you know what,
19:35:09my products are unique. The product name
19:35:12should be unique and we are not allowed
19:35:14to have in this table two products with
19:35:16the same name. So if you have such a
19:35:18rule at your business, you can go and
19:35:20define a unique index for the products.
19:35:22So let's go and do that. Now we're going
19:35:24to go and replace the category with the
19:35:27products and the same thing over here.
19:35:30So we are targeting the column products.
19:35:32Let's go and execute it. As you can see
19:35:35now it is working because we don't have
19:35:37any duplicates inside the table
19:35:39products. And if you go and check the
19:35:41indexes over here, we can see our new
19:35:44index. And as you can see at the start
19:35:46here, it says it is unique non-clustered
19:35:48index. Now let's go and try the data
19:35:51integrity. Are we allowed not to add any
19:35:54duplicate to this table? So let's go and
19:35:56try that out. Let's have an insert
19:35:57statement. Let's say insert into sales
19:36:01products. And I would like only to
19:36:03insert the product ID and the product
19:36:08name. and we're going to insert two
19:36:11values. Values, let's say we're going to
19:36:14have a new ID 106, but we're going to go
19:36:16and insert duplicate for the product
19:36:19name. So, we're going to say caps. We
19:36:22have already a product called caps over
19:36:24here. So, we are now inserting
19:36:26duplicates. Let's go and try it. Now,
19:36:28you will get an error saying you cannot
19:36:30insert duplicates to this table because
19:36:33we have unique index. So as you can see
19:36:35this index is now helping us and
19:36:38improving the quality of my table. So
19:36:40this is how we work with the unique
19:36:42index in
19:36:46SQL. Okay. So now what is a filtered
19:36:49index? A filtered index is a regular
19:36:51index but with a twist. It only includes
19:36:55rows that meet specific condition. So
19:36:58let's understand what this means. So
19:36:59again we have our nonclustered index and
19:37:02the B3 structure. So now at the leaf
19:37:04nodes we will get only the ids the data
19:37:08that fulfill a specific condition. So
19:37:10for example if we are saying we want
19:37:12only the active customers this is the
19:37:14condition. So that means on the leaf
19:37:16nodes we will have only the customer ids
19:37:19that are active and any inactive
19:37:22customer will not be included at all at
19:37:25the data page and at the nodes. So that
19:37:27means our B structure going to be little
19:37:30bit smaller as usual because we have
19:37:32less data included in the structure. So
19:37:35our index going to be smaller than the
19:37:37regular nclustered index. So now the
19:37:40question is why is it important to have
19:37:42a filtered index? Well the biggest
19:37:44benefit is we going to have targeted
19:37:47optimizations. So for example if our
19:37:49analyzes always focuses on the active
19:37:52users and the inactive users are totally
19:37:54unrelevant. So that means having only
19:37:57relevant subset of data in the index
19:38:00going to make the whole index much
19:38:02smaller which leads to faster
19:38:05performance. So it's going to be faster
19:38:07to query this filtered B3 structure. So
19:38:11that means we are doing targeted
19:38:13optimizations and we are improving the
19:38:15query performance. Now the second
19:38:18benefit if you think about the storage
19:38:20since the size of the B structure going
19:38:22to be smaller that means we're going to
19:38:24need less storage space in order to
19:38:26store the index which is great thing if
19:38:29you have large tables in your database.
19:38:32So the filter the index going to make
19:38:34the structure of the index smaller which
19:38:36going to improve the speed and the
19:38:38performance and as well reduce the
19:38:40storage that is needed for your index.
19:38:43Okay. So now let's check the syntax of
19:38:44the filtered index. It's very simple.
19:38:46It's like any query you can go and add
19:38:49at the end of creating the index the
19:38:52wear clause and then the condition as
19:38:54you are doing in any select statements.
19:38:56But the SQL server is very restrictive
19:38:59using this type of index. So you cannot
19:39:01use filtered index on a clustered index.
19:39:05So it is only allowed for the nclustered
19:39:08index because it makes no sense. If you
19:39:11create a clustered index, the entire
19:39:13table should be reorganized and ordered.
19:39:16So it will not work for only subset of
19:39:19data and as well you cannot create a
19:39:21filtered index on a column store. So it
19:39:24is only allowed if you are using row
19:39:27store but you can go and combine the
19:39:29unique index together with the filtered
19:39:31index. There's no restrictions. So it's
19:39:33going to be like this. Create unique
19:39:35nonclustered index on the table and then
19:39:38you specify the wear condition. So this
19:39:40is the syntax of the filtered index and
19:39:42we have these restrictions. All right.
19:39:44So now let's say that we have the
19:39:45following query where we are selecting
19:39:47data from customers but always in our
19:39:49program or in our report we are
19:39:52selecting only the customers from USA.
19:39:55So we have the following condition. It
19:39:57says where country equal to USA and
19:40:01execute. So this is the basics of many
19:40:04queries that we have in our project and
19:40:06we are always filtering the customers
19:40:08based on the country. So in one query we
19:40:10are finding maybe the top customers and
19:40:13another query we are finding the average
19:40:15of scores and so on. But we are always
19:40:17filtering the data like this where
19:40:19country equal to USA. So now since we
19:40:22are using this column a lot and our
19:40:24table may be getting like million of
19:40:26records we can go and create
19:40:28nonclustered index on this column. So
19:40:30the usual way we go over here and say
19:40:32create
19:40:34nonclustered index and we call it like
19:40:37this index
19:40:39customers country and then it's going to
19:40:42be on the
19:40:43table sales customers and we select the
19:40:47column country like this. So if you do
19:40:50it like this SQL going to go and create
19:40:52a nclustered index for all customers not
19:40:55only from USA but for everything. So
19:40:58even if the customers come from Germany
19:41:00which is not really necessary because in
19:41:02our project we only focus on the
19:41:05customers from USA. So instead of that
19:41:08we can go and include the wear condition
19:41:11inside our cluster. So it's very simple
19:41:13we're going to go and say where country
19:41:16equal to USA exactly like our query. So
19:41:20now the index that's going to be built
19:41:22it will be focused and targeted only for
19:41:25subset of data only for the data that
19:41:27fulfill this condition. So now let's go
19:41:30and create our filtered index and it is
19:41:33working. Let's go and check our indexes
19:41:35on the customers. So let's go to the
19:41:37indexes over here and refresh. Now we
19:41:40can see our index over here. It says it
19:41:42is not unique because we didn't define
19:41:44anything at the start. So duplicates are
19:41:46allowed of course which is what we
19:41:48defined here. And as well it is
19:41:50filtered. So it doesn't contain all the
19:41:53rows from your table. It contains only
19:41:55the rows that fulfill our condition. So
19:41:58that means now if I go and execute this
19:42:00query, the index going to be used
19:42:03because the rows of this query is
19:42:05included in the index. But if I go over
19:42:07here and say Germany and execute the
19:42:10query, it's going to be slower because
19:42:12all those rows inside the query is not
19:42:16part of our index. So this index will
19:42:18not be used at all in order to improve
19:42:21the query. So this is how we work with
19:42:23the filtered index in
19:42:28SQL. All right. So now we're going to
19:42:30summarize and talk quickly about how to
19:42:32use the right index. So when to use
19:42:34which type? Let's start with the first
19:42:36one. We have the heap structure. So as
19:42:38we learned it is a table without any
19:42:41index. So in which scenario we don't
19:42:43have to use any indexes in case you want
19:42:46to have fast inserts. So if you want to
19:42:48have a fast write performance then don't
19:42:51take any index. So you stay with the
19:42:53default with the he structure of your
19:42:55table and we usually use it in not very
19:42:57important tables like the staging tables
19:43:00or temporary tables where we want to
19:43:02insert the data fast and then get rid of
19:43:05the data later. So here there is no need
19:43:07to utilize any index. Now if you are
19:43:09talking about the clustered index, we
19:43:12usually use the clustered index for
19:43:14primary keys. It is even a default from
19:43:16the database. If you create any primary
19:43:18keys, then SQL going to go and create a
19:43:20clustered index. So this is the main
19:43:23usage of the clustered index, you use it
19:43:25in the primary keys. And if there's like
19:43:27no primary key in your table, then you
19:43:29can go and pick another column where
19:43:31sorting the data is important like for
19:43:33example a date column. So it could be a
19:43:35good candidate for your clustered index.
19:43:37Now moving on to another type we have
19:43:39the column store index. So when I said
19:43:42here clustered index I mean clustered
19:43:44row store index of course. But now the
19:43:46question is when do we use the column
19:43:48store index. If you have like big
19:43:50complex analytical queries where you are
19:43:52aggregating a lot of data doing data
19:43:54aggregations then go for the column
19:43:56store index because it going to give you
19:43:58amazing performance. And as well if you
19:44:00are struggling with the size of tables
19:44:02so if you have a super large table you
19:44:05can go and use the column store index
19:44:07because it can go and compress the data
19:44:09and reduce the size of the whole table.
19:44:11So for those scenarios we use the column
19:44:13store index. So again for the row store
19:44:16clustered index we use it usually for
19:44:18the old TB systems where you have a lot
19:44:20of transactions and so on but for the
19:44:23column store we use it usually for the
19:44:25OLAP systems where you have a data
19:44:26warehouse reporting system business
19:44:28intelligence and so on. Now moving on to
19:44:30another type we have the nonclustered
19:44:32index. We usually use this index for non
19:44:35primary key columns. So that means the
19:44:37rest of the columns of your tables could
19:44:39be candidate for the nonclustered index.
19:44:41And there are a lot of reasons why you
19:44:43would do that. For example, for the
19:44:45foreign keys or using it on the columns
19:44:47that are used in order to join two
19:44:50tables and another place where you can
19:44:52use the nonclustered index for the
19:44:54columns that are used for the work
19:44:56clause. So there are like many scenarios
19:44:58where we can use the nonclustered index
19:45:00but not for the primary keys. Now moving
19:45:02on to another type, we have the filtered
19:45:04index. We use it in order to target a
19:45:07subset of data. So if in our query and
19:45:09analyzes we are only focusing on a
19:45:12subset of data all time, it makes no
19:45:14sense to have one big index for all
19:45:16data, we can use the filtered index to
19:45:18have focused index. And of course if the
19:45:21size of the index is a problem then you
19:45:24can use a filtered index in order to
19:45:26reduce the overall size of the storage
19:45:28of the index. And then to the last type
19:45:30we have the unique index. you can go and
19:45:33use the unique index in order to ensure
19:45:35data integrity of your table and as well
19:45:37it might prove slightly the performance
19:45:39of your query and that's because SQL has
19:45:42less task to do if the index is unique
19:45:44once SQL finds a match it going to skip
19:45:46the search so this is a quick summary
19:45:48and guide on when to use which index
19:45:51type that usually help me finding the
19:45:54right
19:45:58index all right friends so now let's say
19:46:00that you have created your index ES in
19:46:02your database and your query is
19:46:04optimized and you have fast performance
19:46:07but the job is not done yet.
19:46:10No
19:46:11god no god please no no no no
19:46:17because over the time the indexes get
19:46:19fragmented outdated unused and this
19:46:22going to lead to a poor performance in
19:46:24your queries and as well going to
19:46:26increase the storage costs and the
19:46:29overall performance of your database
19:46:31going to drop down. So indexes like
19:46:33having a car it need maintenance. So you
19:46:36need to change the oil and the tire of
19:46:38the car. And the same thing goes for the
19:46:40indexes. You have to maintain them. They
19:46:42need attention to keep everything
19:46:44running smoothly. So now I'm going to
19:46:46show you how I manage, maintain, and
19:46:49monitor the indexes of my SQL projects.
19:46:51So let's
19:46:55go. The first and the most important
19:46:58task is to monitor the usage of your
19:47:00indexes. So of course the first question
19:47:02we have to ask ourself over the time are
19:47:05we using really the indexes that we have
19:47:07created are they really helping
19:47:09improving the speed of my queries or was
19:47:12it just a good idea at the start of the
19:47:14project and later no one used those
19:47:16indexes. This is very crucial because if
19:47:19you are having an unused index you are
19:47:21consuming unnecessary storage space and
19:47:24as well the right performance in the
19:47:26tables going to be slow which is
19:47:28completely unnecessary if you are not
19:47:30using the index. So now our task is to
19:47:33find out the usage of each index that
19:47:36you have in the projects. So let's see
19:47:37how we can do that. So now previously we
19:47:39have created like multiple indexes on
19:47:41the table DB customers. So if you go to
19:47:44the DB customers and to the indexes, you
19:47:46can see that we have four indexes. Now
19:47:48we can go and show those informations by
19:47:50using a special stored procedures from
19:47:52the SQL server called SP help index.
19:47:55Let's go and do that. So SP help index.
19:47:59So it is a system stored procedure that
19:48:02comes with the database. So this stored
19:48:04procedure needs only one value and that
19:48:06is the table name. So we have it over
19:48:09here sales DB customers. Let's go and
19:48:12query it. So we have four indexes. Then
19:48:14we have a nice description of the index.
19:48:16So it says it is nonclustered index and
19:48:19whether it is column store. And it say
19:48:21where it is located. So it says it
19:48:23located on primary. Primary is the name
19:48:26of the file group where the data is
19:48:28stored. And as a default it can be
19:48:29stored as primary. And now the next
19:48:31information we have the index keys. It
19:48:33is nice information to understand which
19:48:35keys are used or which columns are used
19:48:37for the index. So the first one you can
19:48:40see we have two columns that means it is
19:48:41a composite index and of course for the
19:48:44column store we don't have any columns
19:48:46and then we have the first name last
19:48:47name. So this is a really nice quick
19:48:49store procedure in order to see
19:48:50information about our index. Okay. So
19:48:52now let's focus on our task on how to
19:48:54monitor the usage of the indexes. Now in
19:48:57databases we have a lot of schemas and
19:48:59tables that protocol the metadata of our
19:49:03database. And in SQL Server, we have a
19:49:05special schema called CIS where you can
19:49:07find a lot of metadata information about
19:49:10the SQL server. Metadata like the
19:49:12description of the tables, views,
19:49:14columns and as well the indexes. So now
19:49:17let's check what we can find inside the
19:49:19table indexes. So let's going to do it.
19:49:21Select star from CIS. This is the schema
19:49:24name. And then as you can see we have a
19:49:26list of many informations but we want to
19:49:29focus on the indexes. Now let's go and
19:49:32execute it. Now we get a huge list of
19:49:34all indexes that we have and a lot of
19:49:37informations for each index. We don't
19:49:40have to go and understand now each
19:49:41column. But I'm going to go and select
19:49:43the main important informations from
19:49:45this table. So what do we need? The
19:49:47object ID. This is the table ID. So the
19:49:50object
19:49:51ID and we have the name. It is the index
19:49:55name. And then here we have a nice
19:49:58information whether it is clustered or
19:50:00nonclustered. So let's go and select it
19:50:03type disk as so let's call it index type
19:50:06and we can go and check whether it is
19:50:08primary key or not. So let's get this
19:50:10information as well is primary key. I
19:50:13will go and just rename it is primary
19:50:16key and what else do we need whether it
19:50:19is unique. So it is as well nice
19:50:20information to have. So is
19:50:23[Music]
19:50:26unique. So of course you can go and grab
19:50:28a lot of stuff. It depends really on
19:50:30what you are monitoring. So for example,
19:50:32I'm going to go and check whether it is
19:50:33disabled or not. So is
19:50:36disabled and I'll just rename
19:50:40it. So with that I have like focus
19:50:43monitoring. I don't have to have all
19:50:44those informations. So let's go and
19:50:46execute. But now I would like to go and
19:50:47change few stuff like for example I
19:50:49don't want the object ID. I would like
19:50:51to have the full name of the table. And
19:50:53as well there is a lot of indexes that
19:50:56is unrelevant for my database. So now in
19:50:58order to do that we have to go and get
19:51:00the informations from another metadata
19:51:02table. So let's go and call this index
19:51:04and let's go and join it with another
19:51:07metadata table. It's called tables. So
19:51:11tbl and we're going to go and join it
19:51:13using the so the index object
19:51:17ID equal to the table object ID. And now
19:51:22if you like to see the content of this
19:51:24table we can go and create separately.
19:51:26So select star from our new table. So
19:51:30let's see the content of this table. So
19:51:32you can see we have the name which is
19:51:34the table name. And I think that's all
19:51:35what we need. We have a lot of other
19:51:37informations about the table. Well, I
19:51:39just need the table name. So let's go
19:51:41and do it at the start. tbl name as
19:51:45table
19:51:46name and I don't need anymore the object
19:51:49ID. But of course we have to go and use
19:51:51the alias for each of those informations
19:51:53in order to understand those
19:51:54informations comes from the index. So
19:51:56let's go and do that. All right. So my
19:51:59query is ready. Let's go and execute it
19:52:01again. So now as you can see we are
19:52:02getting the table name and the list is
19:52:04very short because it is only focusing
19:52:06on the tables that you have in the
19:52:08database. And this filter happens
19:52:10because of the inner join. But one more
19:52:12thing I would like to go and sort the
19:52:13data. So I'm going to say order by I
19:52:15would like to sort it by the table name
19:52:18and then the index name. All right. So
19:52:21now let's go and check for example the
19:52:23table customers. You can see that we
19:52:25have two non-clustered index and one of
19:52:27them is column store index. Those two we
19:52:29have created from the previous tutorial
19:52:31and we have an index on the primary key
19:52:33as you can see here is primary key equal
19:52:35to one and this is as well unique. So
19:52:37with that we have a really nice list of
19:52:39all indexes that we have in our
19:52:41database. But we are not there yet
19:52:43because our task is how to monitor the
19:52:47usage of the index. Now in order to get
19:52:49the usage for each of those indexes, we
19:52:52have to go to a special view called
19:52:54dynamic management view. And there the
19:52:56SQL server going to provide a lot of
19:52:58statistics about the usage for that
19:53:00index. And we can find it as well in the
19:53:01same schema. So let's go and query this
19:53:04table. So it's going to be select star
19:53:06from. So the same schema says
19:53:09dodm
19:53:11db_ind index usage stats. So let's go
19:53:15and explore this table and execute it.
19:53:18Now in those statistics we can find the
19:53:20usage of two indexes the index number
19:53:23three and one. And we can see there are
19:53:25like three usage informations of the
19:53:27index number one. And next we have like
19:53:29user seeks user scans and user lookups.
19:53:32So this is how many times the index is
19:53:34used as seeks or scans or lookups. We
19:53:37will understand those informations as we
19:53:39learn about the execution plan. And here
19:53:41we have a very nice information about
19:53:43how many time our index got updated. So
19:53:46as you can see here is zero because I
19:53:48didn't add any new data after creating
19:53:50the index. But of course all those
19:53:52numbers might be different at your site
19:53:54because it depends whether you are doing
19:53:55more queries and practicing. And you can
19:53:57find here more informations about when
19:54:00was exactly the last usage of those
19:54:02indexes and many many nice informations.
19:54:05So now let's go and integrate this view
19:54:08with our query. So now what I'm going to
19:54:10do, I'm going to do a lift join because
19:54:12if I do an inner join, I will only find
19:54:15the used indexes. But I don't want that
19:54:18because I want to see a full build of
19:54:20all my indexes in the database. So left
19:54:22join and we're going to go and get our
19:54:26view and call it S. And then we have to
19:54:28join it on the keys. So S on. So I would
19:54:31say let's go and grab the object ID
19:54:33equal to the index object ID. And of
19:54:37course we have to join on the index ID.
19:54:40So it's going to be the index ID equal
19:54:42to the index ID like this. Now we have
19:54:46to go and select few informations from
19:54:48this view. So I'm going to go and select
19:54:50like all those number of usage. So s
19:54:54let's get the user
19:54:57seeks as the user
19:55:00scans and the
19:55:02lookups and maybe as well the user
19:55:05updates and it is really nice
19:55:07information to understand when it was
19:55:09the last time used. So last user
19:55:12seek and the
19:55:14last user scan. Let me just correct it
19:55:17over here. And actually I can go and put
19:55:19those two dates in one date because if
19:55:23it's like the last seek it's going to be
19:55:24null over here or the opposite. And now
19:55:27what we can do we can go and put those
19:55:28two together actually in one column
19:55:30because when we have a value over here
19:55:32it's going to be null and vice versa. So
19:55:34we can do that using the null function
19:55:37kowalis like this and we can get this
19:55:40over here and we can call the whole
19:55:42thing last
19:55:44update. So like this and maybe I'm going
19:55:46to go and rename all those
19:55:48[Music]
19:55:54stuff. All right. So now we are done.
19:55:57Let's go and execute it. Okay. So let's
19:55:59go and check our new report over here.
19:56:01So this is our query and let's start
19:56:04with the first table for example the
19:56:05customers and go to the right side. And
19:56:07now we can see that we have three
19:56:09indexes and from these two indexes we
19:56:11have only one index that is not used at
19:56:14all. So we can see over here that the
19:56:17nclustered index on the country is not
19:56:20being used and that's because we have
19:56:22another index about the country that
19:56:24comes from the column store. So it could
19:56:26be like this that you are quering the
19:56:27table using the country but the SQL
19:56:31saying I would like to go and use this
19:56:32index instead of the first one. So we
19:56:34can say okay this one is not really
19:56:37useful maybe we can go and drop it right
19:56:39and for the rest you can see okay this
19:56:41column store index is used twice and the
19:56:43next one is once again the numbers at
19:56:45your side might be different and if we
19:56:47have a look to all other tables we have
19:56:49a lot of nulls so that means all those
19:56:51indexes that you have created on the DB
19:56:54customers let me check only one is used
19:56:56but now you might say you know what I've
19:56:58used the index but why I'm not seeing
19:57:00here any numbers about it well that's
19:57:02because those numbers will not live
19:57:04forever and we are using now the express
19:57:07edition locally at our PC. So each time
19:57:09you shut down your PC and you close the
19:57:11client the database going to shut down
19:57:13as well and those statistics going to be
19:57:15lost because they are in the memory. But
19:57:17in real projects the numbers going to be
19:57:19totally different than here and of
19:57:20course you're going to get realistic
19:57:22numbers. Now let's try to target one of
19:57:24those not used indexes. So for example
19:57:27let's go with this index. It is not
19:57:29clustered index on the product. So let's
19:57:31go and query that. Currently it is
19:57:33completely not used. So if I go and
19:57:35select it. So select star from sales
19:57:40products where
19:57:43product equal to caps. So with that we
19:57:46have used the index I think. Let's go
19:57:49back and query again and let's go to our
19:57:52index and check whether it is used. Well
19:57:54it is correct. So our query did use this
19:57:57index and we can see here it is used
19:57:59once. And now you can go and analyze in
19:58:01your project all the indexes that you
19:58:03have on your tables and you can see
19:58:05whether you are really using it with
19:58:07your queries or not. And if you are not
19:58:09using the query of course you have to
19:58:11make a decision about it. Maybe if you
19:58:13are working a team to ask about it who
19:58:15did create it and why. Maybe there is
19:58:17like one task in the database that is
19:58:20not frequently used. Maybe it's
19:58:22something that is run like once a month
19:58:24or something like that. So the index is
19:58:25needed but not that frequently. But
19:58:27still now we have like insights about
19:58:29what is going on with those indexes and
19:58:31whether we need them or not. And if you
19:58:34don't need them, go and drop them. All
19:58:36right, my friends. So here is the secret
19:58:38that 90% of SQL developers don't do
19:58:41that's going to make you in 1 minute the
19:58:43hero of the projects. So once I join a
19:58:46project and after saying hello to
19:58:47everyone, I open the database of the
19:58:50project and do one query. I checked the
19:58:52usage of the indexes of the projects and
19:58:55I can tell you after working 15 years
19:58:57with SQL that 90% of indexes created in
19:59:01projects are totally untouched and
19:59:03unused. So I collect all unused indexes
19:59:06and discuss it with the team. And if we
19:59:09don't find real usage for those indexes,
19:59:11we go and drop them. So after dropping
19:59:14all those unused indexes, you have done
19:59:16two great things for the projects.
19:59:19First, you have saved a lot of storage
19:59:21in the database. And second, which is
19:59:23way more important, you have improved
19:59:25and optimized the right performance on
19:59:28the database. So in your first day with
19:59:30one query, you have optimized the
19:59:33performance of the database. You have
19:59:34save storage and you're going to shine
19:59:36like an expert in your project. So if
19:59:38you haven't done that, do that
19:59:43now. All right. And now moving on to the
19:59:46next one. As we learned, identifying an
19:59:48unused index is an important task. But
19:59:51in the other hand, identifying a missing
19:59:54index is as well very important to
19:59:57improve the performance of your queries.
19:59:59So in SQL server, you can get
20:00:01recommendations from the database itself
20:00:03about missing indexes for your query. So
20:00:06let's see where we can find those
20:00:08recommendations. All right. So now let's
20:00:09say that you are doing multiple queries
20:00:11and you are doing analyszis and so on.
20:00:13For example, I have this query over
20:00:15here. It is query on the database
20:00:17adventure works DW and I'm joining just
20:00:20two tables the fact with the dimension
20:00:22and then filtering the data based on the
20:00:25colors and as well on the date key where
20:00:28I have like a range over here. So once I
20:00:30executed I got the following
20:00:32informations. It could be any query that
20:00:34you are doing while practicing and
20:00:35analyzing and so on. So now if you have
20:00:37like slow query and so on you can go and
20:00:39check the recommendations from the
20:00:41database about missing indexes. So in
20:00:43order to do that we can go and check
20:00:45again the metadata from the database
20:00:47system to see the recommendations about
20:00:49the missing indexes. So let's go and do
20:00:51that. So we're going to go and select
20:00:53from and now we have to go and target
20:00:55the dynamic management views and it is
20:00:57like this dm
20:01:00db
20:01:02missing index details. So let's go and
20:01:06explore the content over here. And don't
20:01:08forget that those informations going to
20:01:10be inside the cache of the server and if
20:01:13there's like a restart or something in
20:01:15the server you will lose all those
20:01:16informations. So now from my query there
20:01:19is few suggestions and recommendations
20:01:21from the database. Let's go and check
20:01:22it. So we can see here there are four
20:01:24recommendations about missing indexes
20:01:27from the database. So now let's go and
20:01:29check the first recommendation over
20:01:30here. You can go and check the table
20:01:32name from the object ID or you can find
20:01:34it here in the statements. So here the
20:01:36database is suggesting an index for the
20:01:38table dimension product and it is
20:01:41recommending us to make an index for the
20:01:43column color and that's because if you
20:01:45check our query we have like here a
20:01:47filter the wear condition where we are
20:01:48seeing the color equal to black and
20:01:50since we don't have an index on the
20:01:52color SQL is just suggesting to use an
20:01:54index for the color and of course in
20:01:56this situation we can go and use an
20:01:58uncclustered index. Now after that we
20:02:00have three recommendations for the same
20:02:02table fact internet sales. So for
20:02:04example here it is suggesting to make an
20:02:06index on the order date K because we are
20:02:09using it in the filter over here and as
20:02:11well suggesting to make an index for the
20:02:14product key since we are using it for
20:02:16the join. So this is really nice report
20:02:19about missing indexes in the database
20:02:21and it could assist you to find out
20:02:23things that you didn't thought about.
20:02:25But here my recommendation is evaluate
20:02:27those informations very carefully. Don't
20:02:30go and create like an index for each
20:02:32suggestions from the database. You still
20:02:34have to think about it. Is it really
20:02:36necessary? Do we really use this query
20:02:38very frequently and so on? So don't go
20:02:41blindly creating indexes for each
20:02:43recommendations from the database. So
20:02:45this is really nice tool and assistant
20:02:47for you in order to make a good strategy
20:02:49for your indexing. So this is how you
20:02:51find the recommendations of missing
20:02:53indexes from SQL
20:02:59database. Okay. Okay. So now to the next
20:03:01step, we have to go and monitor the
20:03:03duplicates in indexing. If you are
20:03:06working in team with multiple developers
20:03:08and you are working parallely in order
20:03:10to optimize the performance of the
20:03:12queries, what might happen is that
20:03:14different developers creating different
20:03:17indexes for the same column in the same
20:03:20table. But of course, this must not
20:03:21happen if you have a clean and solid
20:03:23review process in the project. But we
20:03:26are human and those things could happen.
20:03:28So that's why you have to monitor
20:03:30whether there are like duplicates. So
20:03:32the mission is to find whether there is
20:03:34a column that is involved in multiple
20:03:36indexes. So let's see how we can monitor
20:03:38that in SQL. Okay. So now it's very
20:03:40simple in order to find the duplicates
20:03:42of indexes inside your database. So we
20:03:45have learned before that we can find the
20:03:46list of all indexes in this table
20:03:48indexes in the system schema and then we
20:03:51join it with the tables in order to get
20:03:53the table name and then we have another
20:03:55table in order to find the columns that
20:03:58are involved in the index. Those
20:04:00informations we can find it inside the
20:04:02index columns and now in order to get
20:04:04the full name of the columns we're going
20:04:06to go and join it with the columns
20:04:08table. So it's very simple and makes
20:04:10sense. Let's go and execute the whole
20:04:12query. Now as you can see it is sorted
20:04:14by the table name and the column name
20:04:16and that's because we can find then
20:04:18easier the duplicate. So let's go and
20:04:20check the first table. So the country is
20:04:22part of this index where we have the
20:04:25column store nonclustered and again the
20:04:27country is involved in another index
20:04:29where we have the customer's country and
20:04:32this is a row store nonclustered index.
20:04:35So this is of course bad thing. We have
20:04:36to go and decide now do we want it as a
20:04:38column store or row store. And if we
20:04:41check as well this table, we can find
20:04:43the first name in two different clusters
20:04:46the same story. And that's because we
20:04:47were practicing and creating those
20:04:49indexes. And that's it. But now if you
20:04:51have like large schema and a lot of
20:04:53indexes, I would go and make like a flag
20:04:55in order to understand whether we have a
20:04:57duplicate or not. And that's by
20:04:58calculating the number of rows of unique
20:05:01table name and index name. And we can do
20:05:03that very easily using the window
20:05:05functions. So let's have new row. And
20:05:07we're going to go and use the function
20:05:09count since we want to find the number
20:05:11of rows over. Then we're going to go and
20:05:14partition
20:05:15by we need the table name and as well
20:05:19the column name. Our expectation of this
20:05:22column should be one. If we have more
20:05:24than one then there is an issue and that
20:05:27means the column is inside two different
20:05:30indexes. And now let's go and sort it by
20:05:32the column name and descending. So let's
20:05:36go and execute it. And now we have here
20:05:38a nice flag where we can see how many
20:05:40rows we have for a specific column in a
20:05:42table. So if it's one like those
20:05:44columns, they are fine. Those columns
20:05:47are involved only once in one index. But
20:05:49for the first four rows, we have here an
20:05:52issue because we count here two columns.
20:05:54That means we have two indexes for the
20:05:56same column. So as you can see the query
20:05:58is very simple and with that we have a
20:05:59nice report about the duplicates of
20:06:02indexes inside our database.
20:06:08Okay, one more thing in order to
20:06:10maintain our indexes is by updating the
20:06:12statistics. The database engines usually
20:06:15use statistics in order to understand
20:06:18which index should be used for our
20:06:20query. And if these statistics are not
20:06:22up to date, SQL going to make wrong
20:06:24decisions. So let's understand what this
20:06:26means. Now let's say that you have
20:06:28created a table and you start inserting
20:06:30data to this new table. Now the database
20:06:33engine going to go and create your new
20:06:34table and insert the data. Behind the
20:06:36scenes the database engine going to go
20:06:38and create for your new table
20:06:40statistics. It's like metadata
20:06:42informations about your data and that's
20:06:45like a report or insights about your
20:06:47table where you can find a lot of
20:06:49informations like the number of rows
20:06:52that distribution of values in a column
20:06:55and as well we can find the number of
20:06:57distinct values and histogram and
20:07:00patterns and many other informations
20:07:02about your table. So now of course the
20:07:04question is why do we have those
20:07:06informations in the database? Now
20:07:08imagine that you are doing select from
20:07:10where what going to happen the database
20:07:12engine has to go and create an execution
20:07:15plan. We're going to learn about this
20:07:17later in details. It is just a road map
20:07:19on how to execute this query. So here
20:07:22for example in order to load the data
20:07:24from the table there are like different
20:07:26ways on how to do it. So there is like a
20:07:28table scan, index scan, index seek. So
20:07:31that means the database engine has here
20:07:33three different ways on how to do it.
20:07:36And now in order for the database to
20:07:38decide which way to use, it's going to
20:07:41go and read the statistics of the table.
20:07:43So it's going to go and collect
20:07:45informations. Okay, how many rows do we
20:07:47have? Are the informations are unique?
20:07:49How is the distribution of the data and
20:07:51so on. And now based on those statistics
20:07:53and numbers, the database can now make a
20:07:56good decision about which methods to use
20:07:59in order to load the data. So for
20:08:01example, here the index scan is the best
20:08:03way to load our table. So this is
20:08:06exactly why the database needs the
20:08:08statistics in order to make the correct
20:08:11decision and to use the correct index.
20:08:13So now you might ask okay this is
20:08:15something internal for the database why
20:08:17do we have to care about it? Well there
20:08:19is an issue. Now for example in our
20:08:21table we have 50 rows and let's say that
20:08:23in the next day you went and inserted to
20:08:26this table like around 1 million row.
20:08:28Now the issue that could happen is that
20:08:30the statistics will not get updated
20:08:33about this table and the statistics can
20:08:35still say that we have only 50 rows. So
20:08:38that means the statistics of this table
20:08:40is now outdated. And the big issue that
20:08:43once you query this table, the SQL
20:08:45engine don't know at all about the 1
20:08:48million row that you have inserted in
20:08:50the table because it's going to go and
20:08:52ask the statistics and it's going to
20:08:54answer with only 50 rows and the
20:08:56database going to say okay this is very
20:08:58small table and let's maybe skip an
20:09:00index or something. So that means the
20:09:02database going to make wrong decisions
20:09:04because the statistics are outdated. And
20:09:08now your task is to monitor those
20:09:10statistics and to keep updating them. So
20:09:13let's see how we can do that. Okay. So
20:09:15now the first thing that we have to do
20:09:16is to find out whether our statistics
20:09:18are up to date or outdated. In order to
20:09:21do that we have as well to access the
20:09:23metadata about our database. And for
20:09:25that as well we have tables and dynamic
20:09:27management functions in the system
20:09:30schema where we can find a lot of
20:09:32details about the statistics. And in
20:09:34order to monitor the statistics, I have
20:09:36prepared a query like this. So here I'm
20:09:38using a table called stats uh where here
20:09:41you're going to get a list of all
20:09:42statistics inside our database and the
20:09:45name of the statistics and then I'm
20:09:47joining it with the tables in order to
20:09:48get the table name and what is very
20:09:51important is the dynamic management
20:09:53function. So here we're going to get
20:09:54very important informations like the
20:09:56last updates and the number of rows and
20:09:59the number of modifications. So let's go
20:10:02and query it. So here we can see
20:10:04informations like the table name, the
20:10:06statistics name and now it's very
20:10:08important when the last time the
20:10:10statistics get updated. So now let's go
20:10:12and check our table DB customers. We can
20:10:15see here the statistics name and what is
20:10:18very important is the last update. So
20:10:20this tells us how old is the statistics.
20:10:23So for me it is like 4 days. And then we
20:10:25can find the total number of rows in
20:10:27this table. And now what is very
20:10:29important is the number of modifications
20:10:31that have been done on the table. So
20:10:34after updating the statistics on the
20:10:3619th of October, there were around 15
20:10:39rows that got modificated. This could be
20:10:41an insert, update, delete. So any
20:10:44operation of the table considered to be
20:10:46a modification. So that you can see
20:10:48there were a lot of modifications. So
20:10:50these statistics should be updated. So
20:10:52now for the table customers, you can see
20:10:54that the statistics are up to date. So
20:10:57we have here zero as a modifications and
20:10:59there will be no need to update the
20:11:01statistics. So this is how you can go
20:11:03and check the statistics informations
20:11:05inside your database in order to make a
20:11:07decision should I update the statistics
20:11:09or not. So now let's say that I would
20:11:11like to go and update the statistics of
20:11:13our table DB customers. Now as you can
20:11:16see we have here multiple statistics. So
20:11:18over here we have this statistics on
20:11:20this table and as well we have the
20:11:22statistics on the index. So as you can
20:11:24see we have here multiple statistics in
20:11:26one table. One for the table itself and
20:11:29one for each index that we have in this
20:11:31table. So now let's say that I would
20:11:33like to go and update the statistics
20:11:35only for one. I don't want to update
20:11:37everything in this table only for one
20:11:39statistics. Let's go and do that. So
20:11:42it's going to be very simple update
20:11:44statistics. And then we have to go and
20:11:46mention the name. So it's going to be
20:11:47sales DB customers. And then we have to
20:11:50specify the name of the statistics. So
20:11:52let's go and get this over here and
20:11:55let's go and execute it. So it was very
20:11:58fast. Let's go and reexecute our query
20:12:01and check the data. So now let's go and
20:12:04find it. It was exactly this one. And as
20:12:07you can see it just got updated and the
20:12:09number of rows is five and the number of
20:12:11notifications is zero. So we have now an
20:12:14upto-date statistics for this table. But
20:12:17let's say that I would like to go and
20:12:19update the rest but I don't want to do
20:12:20it one by one. So what we can do we can
20:12:23just copy the same thing over here but
20:12:26we don't specify any name of the
20:12:28statistic. So we are saying update
20:12:30statistics and then only the table name.
20:12:33So let's go and execute it. So now what
20:12:35going to happen is still going to go and
20:12:36update all the statistics that belongs
20:12:38to this table. So let's go and check our
20:12:41query again. Now you can see everything
20:12:43disappeared and the DB customer is
20:12:46completely up to date with no
20:12:48modifications problem. So this is how
20:12:50you can go and update your table and you
20:12:53can do then for the rest as well. But
20:12:55now there is like one more thing where
20:12:57you can go and update the statistics of
20:12:59the whole database. But beware this
20:13:03might take really long time and we're
20:13:05going to do that by executing a special
20:13:07store procedure. So execute SP update
20:13:11stats. This one over here. Let's go and
20:13:15do that. And now it is done. And we have
20:13:17here a pretty long log. It was fast
20:13:20because we don't have a big database. It
20:13:22is very small database. So it's not
20:13:23compared to any real databases. So now
20:13:26we can see over here that SQL is going
20:13:29through everything that you have in the
20:13:31database and trying to update the
20:13:32statistics. So in many situations it's
20:13:35going to be not necessary because there
20:13:36is nothing to update. There were no
20:13:38modifications and so on. That's why the
20:13:40database is smart enough to say no it is
20:13:42not required and it go and skip it. So
20:13:44now how I usually do it in my project is
20:13:46that I have like a job on the weekend
20:13:49where it's going to go and update the
20:13:51whole database statistics. So with that
20:13:54I make sure all my tables and indexes
20:13:57having up to-date statistics. Of course
20:14:00if you have small database you can run
20:14:01this like every day but if this takes
20:14:04long time then you can schedule it in
20:14:06the weekend. And as well if I know in
20:14:08the project that there will be in one
20:14:10day a lot of new incoming data. So we
20:14:13are doing some kind of data migrations.
20:14:15So I go and update the statistics after
20:14:17the data migration is done just to make
20:14:19sure we have up-to-date statistics. So
20:14:21this is how we monitor and update the
20:14:23statistics of the
20:14:24[Music]
20:14:28database. Okay. Okay, so now moving on
20:14:30to the final task that I usually do in
20:14:32order to monitor and manage the indexes
20:14:34is to monitor the index fragmentations.
20:14:37Over the time as your data is inserted,
20:14:40updated, deleted into your tables,
20:14:42indexes can become fragmented. So what
20:14:45is fragmentation? It means like there is
20:14:48unused spaces in your databases and the
20:14:50database is not filling them or your
20:14:53data is not anymore sorted correctly in
20:14:55the index and this of course leads to
20:14:58inefficient use of the storage and as
20:15:00well going to slow down your
20:15:07queries and in SQL in order to get
20:15:10everything organized again we have two
20:15:12methods the first method is reorganize
20:15:15so it's going to go and def fragment the
20:15:17leaf level of the index in order to get
20:15:20it organized and sorted again with the
20:15:22logical order. So it is very light
20:15:25operation and it will not block the user
20:15:27from using your table. And the second
20:15:29method called rebuild this is
20:15:31heavyweight operation. It going to go
20:15:34and drop the whole index and recreate it
20:15:37from the scratch. And this means of
20:15:39course not only the data going to get
20:15:40sorted again but as well the
20:15:43fragmentations inside your databases and
20:15:45the index going to be eliminated. So
20:15:47let's see how we can do that in SQL.
20:15:49Okay. So now back to our database and
20:15:51the first question that you have to ask
20:15:53do we have an issue with the
20:15:54fragmentations in our indexes. So we
20:15:57have to check the health of our indexes
20:15:59in the database. And in order to do
20:16:01that, we have again to go to the system
20:16:04metadata that we have and we're going to
20:16:06check their dynamic management
20:16:08functions. So there is like a special
20:16:10functions in order to get an answer in
20:16:12the SQL server. Let's go and do that. So
20:16:14we're going to go and select star from
20:16:16the function. So it is sis dot so it's
20:16:19going to be sis dot dm db index physical
20:16:24states this one. And this is a function
20:16:26that we have to pass few parameters. We
20:16:28will not go in details just follow me
20:16:30with this. So we have to give it the DB
20:16:32ID and a null another null and a third
20:16:37null and the last one going to be
20:16:39limited. So we have to do it like this.
20:16:41So let's go and query it. Now what do we
20:16:43find? We have the object ID. We have the
20:16:45index ID and few other informations but
20:16:48the most important one is the average
20:16:50fragmentation in percent. So this
20:16:52columns gives us the degree of the
20:16:54fragmentations in a word index. If it is
20:16:57zero then it is perfect. We have no
20:16:59fragmentation in the index and our index
20:17:02is very healthy. But if it is like 100
20:17:05then that means it is completely out of
20:17:07order and we have to do something about
20:17:09it. And now you might say you know what
20:17:11I don't know which object it does and
20:17:13which index. Well you have to go and
20:17:15join few tables like the cy.ts and
20:17:18cis.index in order to get those
20:17:20informations. So we have to go and do
20:17:22that like we have done at the first
20:17:23query. So okay so offline I have done
20:17:25that. So I joined with the tables and
20:17:27the indexes and I'm sorting the data by
20:17:30the average fragmentations and
20:17:32percentage descending in order to get
20:17:34the problems at the start because we are
20:17:36interested where we have high
20:17:37percentage. So let's go and execute
20:17:39this. And now since it is practicing
20:17:41database I didn't insert any data and so
20:17:44on. But in real projects you will get
20:17:45here different numbers. And here is my
20:17:48recommendations about the percentage. If
20:17:50the fragmentation is between like zero
20:17:52and 10 that means everything is like
20:17:54okay and you don't have to do anything
20:17:56about it. But if the percentage is
20:17:58between like 10 and 30 then here we have
20:18:01to do something about it. So here I
20:18:03recommend to use the reorganize method
20:18:05in order to sort the data again
20:18:07correctly. But if you have more than 30%
20:18:10then here my recommendation is to go and
20:18:12rebuild the whole index because not only
20:18:15the data is in wrong order but as well
20:18:18there is a new spaces in your data page
20:18:20in the index. So you have to do
20:18:21something about it. So now let's go and
20:18:23imagine one of those indexes for example
20:18:25this one over here has fragmentation of
20:18:2815%. So now what we have to do is to go
20:18:30and reorganize this index. Let's see how
20:18:32we can do that. So let's go over here
20:18:35and say the following. alter index and
20:18:38then we need the index name. So let's go
20:18:40and get it from here and then you have
20:18:42to mention the table name where the
20:18:44index exists. So we have it from the
20:18:46customers. So from sales customers so
20:18:49now we are editing the index and we have
20:18:51to tell SQL what to do now. So we just
20:18:53want to reorganize the index. So you go
20:18:55and use the keyword reorganize. So
20:18:58reorganize and that's it. This is very
20:19:01simple. So let's go and do that. And as
20:19:03you can see it is completed and it was
20:19:05very fast because we have small
20:19:06database. But sometimes it take little
20:19:08more time if you have a big index and
20:19:10big table. So after reorganizing you can
20:19:12go and again check the table over here
20:19:14and see the results and it should be
20:19:17like here is zero. Now let's see that we
20:19:19have another index where the
20:19:20fragmentation around like 50%. So let's
20:19:23go and copy it and this time instead of
20:19:26reorganize we're going to do rebuild. So
20:19:29I'm going to take the whole thing and
20:19:32this time we're going to go and rebuild
20:19:34this index over here on the same table
20:19:36and instead of reorganize we're going to
20:19:38say rebuild. So let's go and execute
20:19:41that. And with that SQL did drop the
20:19:44whole index and create it from the
20:19:46scratch. And this is usually takes more
20:19:48time than reorganize of course. And the
20:19:51next step of course is to go and check
20:19:52again the fragmentations and so on. So
20:19:55that's all about how to make your index
20:19:57healthy and remove the fragmentations
20:20:00from your index. All right, my friends.
20:20:03So as you can see, improving the
20:20:04performance of your queries doesn't end
20:20:06by creating them. It's all about staying
20:20:09proactive. So monitor the usage of the
20:20:12indexes, check whether there are any
20:20:14missing indexes, and always make sure
20:20:16the statistics of the database are up to
20:20:19date and keep your eyes on the
20:20:20fragmentations and make sure you have
20:20:22healthy indexes. So with that you have
20:20:25learned how I manage and monitor the
20:20:27indexes once I create them and I really
20:20:29recommend you to follow those
20:20:34steps. All right friends, so now let's
20:20:36say that you have a large complex
20:20:38analytical SQL query and it involves a
20:20:42lot of joins and aggregations and so on
20:20:44but it is slow and of course you want to
20:20:47go and optimize the performance of your
20:20:48query by maybe using indexes. And now
20:20:51the big question is where exactly I'm
20:20:54going to go build this index on which
20:20:55table on which columns. So that means
20:20:58you have to understand where exactly the
20:20:59problem is. Is it by joining tables or
20:21:02sorting data or by the aggregations? Now
20:21:05in order to answer all those questions
20:21:08we have something called execution plan.
20:21:10So what is that? The execution plan
20:21:13going to show you how the database
20:21:15exactly process your query step by step.
20:21:19And this is what we need. It's going to
20:21:21show us where exactly we have a
20:21:23performance issue. So in other words,
20:21:25the execution plan it's like your window
20:21:28on how the SQL database thinks and once
20:21:31you understand that then you're going to
20:21:33make a right decision on building an
20:21:35index. So let's understand exactly what
20:21:37this means. Okay. So now let's imagine
20:21:39that you are doing a query like
20:21:41selecting from table and then joining
20:21:43the data with another table. So now once
20:21:45you execute this query the database
20:21:47engine will not go immediately and start
20:21:49fetching data from the disk but instead
20:21:51of that first the SQL has to make a
20:21:54plan. So it's like you are planning a
20:21:56trip where you check the Google map in
20:21:58order to find the best route in order to
20:22:01reach the destination and the execution
20:22:03plan is exactly the same thing. The
20:22:05database has first to plan how to
20:22:08execute your query and it's going to
20:22:09build this plan step by step based on
20:22:12your query and as well the statistics.
20:22:15So the first step for example how to get
20:22:17the data from the tables and there are
20:22:19like multiple ways like scan index or
20:22:22full table scan and then after that it
20:22:24need to decide which type of joins going
20:22:26to be done like is it hash join or a
20:22:28loop join and then at the end of this
20:22:30plan it's going to be the select
20:22:32statements. So once the execution plan
20:22:34is ready the database engine going to
20:22:36start implementing the steps. So it's
20:22:38going to go and start reading your
20:22:40tables for example from the disk and
20:22:41then after that it's going to join the
20:22:43tables and then select the columns and
20:22:45send at the end the results to the end
20:22:47user. And now once everything is done
20:22:50the database engine going to do one more
20:22:51thing where it's going to go and take
20:22:53this execution plan and store it at the
20:22:56cache. And that's because the database
20:22:58engine can go and reuse this plan if we
20:23:02have a similar query. So for example, if
20:23:04you go and execute the same query again,
20:23:06the database engine here going to
20:23:08understand ah this is the same query. We
20:23:10have already built an execution plan for
20:23:13that. So it going to go and check the
20:23:15cache and it is way faster to get it
20:23:17immediately from the cache instead of
20:23:19building it. So in this scenario, the
20:23:20database engine doesn't have to make any
20:23:22decisions or something like that. going
20:23:24to go and get the plan from the cache
20:23:26and start immediately by executing the
20:23:29plan. And of course, the database engine
20:23:31will not hide the execution plan from
20:23:33the users. You can go and check it
20:23:35because you can go and check how the
20:23:37database loaded the data, how they are
20:23:39joined and so on. And then you can make
20:23:41a correct decision on how to optimize
20:23:43your query maybe by adding indexes. So
20:23:46let's go back to SQL and see how we can
20:23:48do that.
20:23:52Okay, so now we're going to work with
20:23:54the database Adventure Works DW2022. And
20:23:57now we're going to go to our tables and
20:23:59we're going to focus on the fact fact
20:24:02reseller sales. Now let's go and check
20:24:04the type of this table. So if you go
20:24:06inside it and go to the indexes, you can
20:24:08see that we have an index on the primary
20:24:11key. So we have a clustered roster
20:24:14index. So that means the data is
20:24:16structured in this P tree. So now what
20:24:18we're going to do, we're going to go and
20:24:18create a mirror of this table but
20:24:20without any indexes. So it's going to be
20:24:23very simple. Select star from our fact
20:24:26reseller sales and we're going to insert
20:24:29it in a new table. So
20:24:32into fact
20:24:34reseller sales and I'm going to call it
20:24:37HP for heap. So let's go and execute it.
20:24:40And now you can see we have inserted in
20:24:41the new table around 60,000 rows. So now
20:24:44we can go and refresh our tables in
20:24:46order to find our new table. So it is
20:24:48over here factory seller sales and if
20:24:51you check the indexes you will not find
20:24:53any. So that means it is a heap table.
20:24:55Now let's go and do a very simple query
20:24:57on top of our new table. So select star
20:25:01from the factory seller HP like this. So
20:25:04let's go and execute it and we got the
20:25:07results. So now the question is I would
20:25:09like to see the execution plan of this
20:25:11query. Now in order to see the execution
20:25:13plan we're going to go to the toolbar
20:25:15over here and we have three things. The
20:25:18first one is says display estimated
20:25:21execution plan and we have another one
20:25:24says include actual execution plan and a
20:25:27third one says include live query
20:25:30statistics. So now the question is what
20:25:32are the differences between them? Let's
20:25:33start with the first one displayed
20:25:35estimated execution plan. So here what's
20:25:37going to happen? SQL going to go and
20:25:39guess the execution plan without
20:25:42executing the query. So it's just an
20:25:44estimation. So this is only a guess an
20:25:46estimation. The second one is the actual
20:25:49one. So this going to show you the
20:25:51execution plan that is used in order to
20:25:54process your query. So after executing
20:25:57your query, SQL going to show for you
20:25:59which plan is used. So that means the
20:26:01estimated plan it is something before
20:26:04executing your query and the actual plan
20:26:07is something after executing your query.
20:26:10And the third one is while executing the
20:26:13query. So you're going to get a realtime
20:26:15execution of your query and you can see
20:26:17how your execution plan is working. So
20:26:20now we can go and try that. Let's go and
20:26:22activate the estimated execution plan.
20:26:24Now we can see over here we have a new
20:26:26output where you can see like few boxes.
20:26:29So this is an estimated execution plan
20:26:31without executing your query. But now if
20:26:34you go over here and switch it to the
20:26:36actual execution plan nothing going to
20:26:38happen because first you have to execute
20:26:40your query. So let's go and do that. So
20:26:42once we have executed we got the result
20:26:45the messages and here we have a new tab
20:26:47called execution plan. So if you go over
20:26:49here you will find the real execution
20:26:52plan that is used to process your query.
20:26:54And let's go and try the third one. And
20:26:57let's go and
20:26:58execute. It was pretty fast because the
20:27:01query is very fast. But here we can see
20:27:03how the data and the plan is working
20:27:06during the execution. So this is the
20:27:08live execution plan. And of course we
20:27:10have the last one which is the current
20:27:13execution plan. So those are the
20:27:15differences between those stuff. Now you
20:27:17might ask why do we have this estimated
20:27:20and actual execution plans? Well, it is
20:27:23really nice tool to understand whether
20:27:25everything like is healthy at your
20:27:27database because if the guessing is
20:27:29something else at the actual execution
20:27:32plan that means this is an indicator
20:27:34that something is wrong at the
20:27:36statistics or the index at your
20:27:38database. So if they are matching the
20:27:40estimated and the actual then everything
20:27:42looks good. But now we're going to focus
20:27:44only on one type of those execution
20:27:46plans. We're going to stick with the
20:27:48actual execution plan. So now what we're
20:27:50going to do, we're going to go and open
20:27:52two queries side by side and one going
20:27:54to be from the clustered index and
20:27:56another one is from the heap structure.
20:27:59So it's going to be like one to one.
20:28:00Let's go and query both of them. And now
20:28:03let's go and try to read the execution
20:28:05plan. But make sure that you are
20:28:07activating the actual execution plan. So
20:28:10we have here now two plans. So now we
20:28:13are at the he table and we don't have
20:28:15any indexes. So now the question is how
20:28:17to read this execution plan? Well, now
20:28:20the plan is very simple because we have
20:28:21a very simple query but we read it from
20:28:24the right to the left. So the first
20:28:26operation is the table scan and then we
20:28:29have here a very small arrow to the next
20:28:31one where we have the select. So from
20:28:33right to left. So now of course the
20:28:35first operator is how to read your data
20:28:37inside the table and here we have
20:28:39different types of scans and one of them
20:28:41is the table scan. So table scan
20:28:43actually is scanning the entire table.
20:28:46So it's going to go and scan all the
20:28:48rows inside your tables in order to
20:28:50execute this query. Now if you go and
20:28:52mouse hover on the table scan, you will
20:28:55find a lot of details about what is
20:28:58happening during loading the data or
20:29:00scanning the table. But it is little bit
20:29:02annoying better than that. If you go
20:29:04right click on it and then go to
20:29:06properties, you will get in the right
20:29:08side the same details but it is easier
20:29:10to read. So the first thing that we have
20:29:12to read is the number of rows that has
20:29:15been read. So we can see that we have
20:29:17read all the rows inside the table which
20:29:19is not really good and we have another
20:29:21important informations about the
20:29:22resources and the cost. So we have the
20:29:24CPU cost and the input output costs and
20:29:27what is interesting is the logical
20:29:29operator the table scan and we can see
20:29:32some nice informations about the
20:29:33storage. It says it is row store. Now
20:29:35let's go and check the execution plan of
20:29:38this other table where we have a
20:29:40clustered index. So let's go to the
20:29:42execution plan. And now you can see that
20:29:44we have on the right side something
20:29:46else. We don't have table scan. We have
20:29:48something called clustered index scan.
20:29:51It is either scanning the entire table
20:29:53again or only a range or a part of the
20:29:57index. And of course in the details we
20:29:59can see whether it read all the
20:30:00informations or not. Now if you go and
20:30:03check the number of rows again the whole
20:30:05index is read in order to get this
20:30:07results. So again we have here the total
20:30:09number of rows inside our table. And as
20:30:11well you can see over here the logical
20:30:13operation it is clustered index scan. So
20:30:16it is not table scan. Now of course we
20:30:18have to go and check the CPU and the
20:30:20input output costs whether we are
20:30:21consuming the same efforts or not. So we
20:30:24can go and compare stuff. So here we
20:30:26have like
20:30:270.07. And if you go over here you can
20:30:30see we didn't gain like a lot of
20:30:32information having an index on this
20:30:34table. And that's of course logical
20:30:35because this query is not using any
20:30:38indexes. It is just like selecting
20:30:40everything from the whole
20:30:44table. So now let's go and extend it
20:30:47where we're going to sort the data by
20:30:48the primary key sales order number. So
20:30:53let's go and get this one and as well
20:30:55for the heap structure. So let's go and
20:30:58execute it and check the execution plan
20:31:01and the same thing for our cluster
20:31:04table. Now let's check first the heap
20:31:07structure. As you can see here, we have
20:31:08like two steps. First, it's going to go
20:31:11and scan the whole table and then we
20:31:14have sort operator in order to go and
20:31:17sort all the data in order to present it
20:31:19in the output. And at the end, we have
20:31:21the select which is not really
20:31:23important. So here we have like two
20:31:24operators. But now if you go to our
20:31:26clustered index, you can see that we
20:31:29have only like two steps. There is no
20:31:30sort step, right? And that's because the
20:31:33clustered index is only sorted and SQL
20:31:37don't have to go and sort the data
20:31:39again. So it doesn't have to go and sort
20:31:41anything. The data is already sorted. So
20:31:43this is the first win that you have if
20:31:46you have an index. So everything is
20:31:48already sorted and if you have an order
20:31:50by on this column then SQL don't have to
20:31:53do it during the query. So now if you
20:31:55want to go and compare the cost you can
20:31:56see here we still have the same cost for
20:31:58the CPU and the input output in the h
20:32:01structure without any index we have here
20:32:04like double cost. The first cost is for
20:32:06the table scan. It is the exact same
20:32:09amount of CPU and input output like the
20:32:12clustered but as well on top of it we
20:32:14have high cost for sorting the data. So
20:32:17we are consuming more CPU and input
20:32:19output. And if you summarize those cost
20:32:21of course this query going to be slower
20:32:24and bad compared to the clustered index.
20:32:28So with that in the execution plan you
20:32:30can understand exactly the benefit of
20:32:32your index. And one more thing about
20:32:34this plan if you go over here. So if you
20:32:36go to the objects and let me just extend
20:32:39it like this. You can see the name of
20:32:42the index that has been used for your
20:32:44query. So it says the index is B key for
20:32:48primary key. And then we have the whole
20:32:50thing. So now if you go to our table on
20:32:52the left side, check the indexes, it
20:32:54going to be exactly this index. So in
20:32:57the execution plan you can find as well
20:33:00which index has been used in your query.
20:33:02And this is very important to check. If
20:33:04you create a new index then run your
20:33:07query and check whether the database is
20:33:09using your new created index. And if not
20:33:12then you are making the wrong decisions
20:33:14about your index. So each time you
20:33:15create a new index, make sure to check
20:33:17whether in the execution plan the
20:33:19database is using your new
20:33:21[Music]
20:33:24index. Okay, so now let's keep going.
20:33:27Now instead of using the primary key,
20:33:29I'm going to go and filter the data
20:33:30based on one of those columns that we
20:33:32have in this table. So let me check the
20:33:34results and let's take for example the
20:33:37carrier tracking number. So carrier
20:33:40tracking number and let's go and pick a
20:33:43value. the first one here like this and
20:33:46let's do the same thing for the heap
20:33:48table and execute it. And now in the
20:33:51execution plan you see we still have a
20:33:53table scan and on this table let's see
20:33:56the execution plan with the clustered
20:33:58index. Now let's say that I would like
20:34:00to go and create a nclustered index for
20:34:03this column. So let's go and do it. So
20:34:05create
20:34:07nonclustered index and I'm going to call
20:34:09it index fact reseller and then the
20:34:13column name. So on our table fact
20:34:18reseller and the column going to be
20:34:20carrier tracking number. So I'm going to
20:34:22take it from here and let's go and
20:34:25create it. Now let's see whether our
20:34:27query going to use this index. So let's
20:34:29go and execute it and let's go to the
20:34:31execution plan. Now things looks
20:34:34completely different than before. So
20:34:36what is going on? We can see that we
20:34:38have now something new. We don't have a
20:34:40clustered index. We have something
20:34:42called index seek. Index seek is an
20:34:45amazing sign in your execution plan
20:34:47because it tells us that SQL server did
20:34:50find a way to use the index in order to
20:34:53find the exact data that we need without
20:34:55scanning a lot of stuff. So that means
20:34:58now we have like three types of scans.
20:35:00We have the table scan where the SQL
20:35:02going to go and scan the whole table and
20:35:04this can happen in the heap structure
20:35:06and the second one we have the index
20:35:09scan and here we don't know whether it
20:35:11is scanning the whole index or a part of
20:35:13the index and the last one we have the
20:35:16index seek where the database is able to
20:35:19find directly the data without scanning
20:35:21a lot of stuff. So the worst type is the
20:35:24table scan. Then we have the index scan
20:35:27and the best one is the index seek. So
20:35:30if you check here the details you can
20:35:32see the number of rows that has been
20:35:34read is only 12. This is amazing. Let's
20:35:37go and check the heap scan over here. So
20:35:40to the execution plan and if you go over
20:35:42here you can see that we are reading
20:35:45around 60,000 rows in order to get 12.
20:35:49But with the index we are reading only
20:35:5212 in order to get 12 and this is
20:35:54amazing and very fast of course and of
20:35:57course the cost of this is very very
20:35:59small. So if you check the CPU and the
20:36:01input output you can see those numbers
20:36:03are nothing and of course if you go to
20:36:05the object over here you can see which
20:36:07index has been used and this is exactly
20:36:09the index that we have just created. So
20:36:12that means it was a really good decision
20:36:15to create this index and the SQL was
20:36:17very happy about it and used it in order
20:36:19to fast find our data. So now let's go
20:36:22and check the rest of the plan. And now
20:36:24you can see over here we have key
20:36:25lookup. The key lookup is an operation
20:36:28that we need in order to get the rest of
20:36:30the columns because from this index we
20:36:33are getting the data of only one column
20:36:35the carrier tracking number. But since
20:36:37in our query we are saying select star
20:36:40that means we have a lot of columns and
20:36:42those columns are not part of the index.
20:36:44So in this index is called don't know
20:36:46anything about the rest. That's why has
20:36:49to go and search for the other columns
20:36:52and of course it is called a lookup not
20:36:54a scan or something like that and that's
20:36:56why we have here as well only 12 rows
20:36:59but from this step we will get the rest
20:37:02of the columns. So and now the next step
20:37:05is that SQL going to go and join those
20:37:07two informations. So we have from the
20:37:10first one the carrier tracking number
20:37:12and the second one we have the rest of
20:37:14course SQL has to go and merge all those
20:37:16stuff in one in order to have it as a
20:37:18results. And now this operation called a
20:37:20nested loops. Behind the scenes there
20:37:23are different types of joins not the one
20:37:25that we know the inner lift and so on
20:37:27but there is another types of joints. We
20:37:29have the nested loop. We have the merge
20:37:32join and the hash join. The nested loop
20:37:34is very good for small stuff. If you
20:37:37have large tables, then the merge and
20:37:39the hash joints are way better than the
20:37:42nested loop. So that means if you are
20:37:44getting here a lot of data from the
20:37:46index and the lookups and you seek is
20:37:49using a nested loop, this is not good.
20:37:52But for now it is okay because we are
20:37:54getting only 12 rows and the operation
20:37:56going to be fast enough. And now one
20:37:58more thing that we can see inside our
20:37:59execution plan is the cost in
20:38:01percentage. So from checking this plan
20:38:03you can see the select is almost costing
20:38:06nothing. The cost of the nested loop is
20:38:09as well like 0%. And then we have like
20:38:116% of the index seek. That's because it
20:38:14is pretty fast and the most expensive
20:38:17operation that done in our query is the
20:38:20key lookups of course because it's going
20:38:22to go and get all the columns. And now
20:38:25if you go and compare to the heap
20:38:27structure even though that the execution
20:38:29plan of the heap structure looks very
20:38:31small doesn't mean that is faster than
20:38:34the indexes that we have. Still if you
20:38:36go and add up all those numbers it is
20:38:39way way faster than the heap structure.
20:38:42Now I would like to show you one more
20:38:43thing. If you want to get rid of this
20:38:45key lookup and in your query you have
20:38:48only selecting the carrier tracking
20:38:50number. Let's go and execute it and go
20:38:52to the execution plan. As you can see
20:38:54there is no need for the lookup because
20:38:57we have only one column and this data we
20:38:59can get it completely from our index. So
20:39:02as you can see it is interesting to
20:39:04understand how SQL is working with your
20:39:06table and with your index and this is
20:39:08how to validate whether you are making
20:39:11correct decisions about your
20:39:16indexes. Okay. So now let's go and add
20:39:18more stuff where we are doing
20:39:20aggregations joins and so on. Let's
20:39:22extend our query. So I'm going to go and
20:39:24join it with another dimension like for
20:39:26example the dim products and the join
20:39:31going to be on the product key. So
20:39:33product key and equal to as well product
20:39:37key. Now after that we're going to go
20:39:39and aggregate few stuff. So we're going
20:39:40to aggregate by the product name. So I'm
20:39:43going to take the product name. So it's
20:39:45going to be the English product name and
20:39:48let's go and call it product name. And
20:39:51let's go and aggregate the sales. So sum
20:39:55and we're going to get it from the fact
20:39:56table. It's going to be sales amount. So
20:40:00as
20:40:01total sales and of course we have to go
20:40:04and do group by and not French name.
20:40:08It's going to be the English
20:40:09name. So let's group up by the product
20:40:13name. And that's it. Let's go and
20:40:15execute it. Now we have a nice list of
20:40:17products and total sales. But let's go
20:40:19and check the execution plan. And oh my
20:40:22god, we have a lot of stuff. So let's
20:40:24start from the right side. So let's do
20:40:27it quickly from the right to the left.
20:40:28So the first thing is that it's going to
20:40:30go and get the data from the fact. So it
20:40:33is using the clustered index. And then
20:40:35after that it's going to go and do a
20:40:37hashmatch for the aggregation. And after
20:40:40that it's going to go and sort the data
20:40:42because it is doing later a merge join.
20:40:45So all those steps are preparing the
20:40:47fact table. And then we have another
20:40:49cluster scan for the dimension. So it
20:40:52going to go and as well select the
20:40:54informations from the dimension. And we
20:40:55have here like not a lot of rows. So it
20:40:58is very small table 600 rows. And now of
20:41:00course the result of the cluster scan is
20:41:03as well sorted right and of course as we
20:41:05learned the cluster the index going to
20:41:06go and sort the data. So we have here a
20:41:09sorted output together with another
20:41:11sorted output. So we have like two data
20:41:14sets that are sorted and SQL here
20:41:16decided to go with the merge join which
20:41:18is a good join in order to join two
20:41:20sorted data sets. It is way faster than
20:41:23joining using the nested loop. So
20:41:25everything is fine and then the data
20:41:27going to be sorted and presented at the
20:41:29output. And now if you are checking this
20:41:31plan you can see the most expensive
20:41:33thing happened at the fact table. So
20:41:3671% of the total cost happened in this
20:41:39step. Now let's say that the query is
20:41:41slow and I would like to go and optimize
20:41:43it. We have learned that if you are
20:41:45doing aggregations on big tables then
20:41:48the column store index is a good idea.
20:41:52So let's go and find whether that is
20:41:54true. So I'm going to go to our other
20:41:56table. So our sales table was with the
20:41:59heap structure. And now you say you know
20:42:01what let's go and convert this he
20:42:04structure to a column store. So let's go
20:42:06and do that. So we're going to say
20:42:07create
20:42:09clustered column store index and we're
20:42:13going to call it index and then the
20:42:14whole name fact
20:42:17reseller sales HP and we don't have to
20:42:20specify any columns. So it's going to be
20:42:23our table on and that's it. Let's go and
20:42:26execute it. So now our table is not
20:42:29anymore heap structure. It should be a
20:42:32column store. So if you go and check the
20:42:34informations we can see we have like
20:42:36clustered column stored index on it. So
20:42:39now let's go and do the same query and
20:42:41check whether we have a better
20:42:42performance. Let's go and execute it.
20:42:44And of course you have to go and
20:42:45activate the execution plan. So I'm
20:42:47going to and now let's go and check from
20:42:50the right again. So this is our fact
20:42:53table and as you can see already it is
20:42:56costing only 6%. Interesting. So let's
20:42:59go and compare what happened to our fact
20:43:00table. First of all, we can see that the
20:43:02physical operation is a column store
20:43:05index scan. And if you go to the objects
20:43:07over here, you can see that the SQL did
20:43:10use the column store. And that is of
20:43:12course going to happen because the whole
20:43:13data is stored only in the index. So
20:43:16there is no way around it. So it can go
20:43:18and of course and use the index. But now
20:43:19what is interesting maybe we have to go
20:43:21and compare the CPU costs. So if we
20:43:24check over here, it is like
20:43:270,000.67 almost the same thing for the
20:43:29input output. Let's go to the previous
20:43:32plan where we don't have a column store
20:43:34and check our facts. So as you can see
20:43:36here it is way more expensive reading
20:43:39the fact table than the column store and
20:43:42as well we have reduced the input output
20:43:44costs. So as you can see we went from
20:43:4671% of total cost for the fact table to
20:43:51only 6%. And the resources that is used
20:43:54to execute the query it is way less than
20:43:58a normal clustered res store. And this
20:44:00is exactly the power of this index, the
20:44:02column store index. You can use it in
20:44:05big tables like the fact tables like we
20:44:07are doing here in this query, you will
20:44:09be getting amazing performance for this
20:44:12scenario. So of course you can go and
20:44:14compare the execution plan by moving
20:44:16left and right. So as you can see if I
20:44:18click over here and I just switch to the
20:44:20other tab, I can like quickly compare
20:44:22the numbers. But there is another way on
20:44:24how to compare execution plans and that
20:44:27is if you go to the execution plan and
20:44:29right click on it then go to save
20:44:31execution plan as and then you have to
20:44:34go and give it a name for example query
20:44:36pro store. So let's go and save it and
20:44:40then you can go to the second query
20:44:42where we have the row store and then
20:44:43right click on the execution plan and
20:44:46say compare show plan. So once you click
20:44:49on that then you have to go and select
20:44:51the one that you want to compare with.
20:44:53So open and now on top you have your
20:44:56query and at the bottom you have the
20:44:58execution plan that you have saved and
20:45:00then you have here a lot of informations
20:45:03where they compare both of the execution
20:45:06plan and with that you can go in more
20:45:08details in order to understand which
20:45:10plan is better. All right friends so as
20:45:12you can see having the execution plan is
20:45:14is amazing. We can see how the SQL is
20:45:17working behind the scenes and we can
20:45:19understand how SQL is processing my
20:45:21query step by step. How much resources
20:45:24it is consuming, whether my indexes are
20:45:26useful or useless and I can go and
20:45:29experiment stuff. I can go and add like
20:45:31an index then test and check whether I
20:45:33gained like few performance or not. And
20:45:35we can go and compare like multiple
20:45:37execution plans before and after until
20:45:40you get the right index for the right
20:45:42table and the right column. So the
20:45:44execution plan are amazing in order to
20:45:47help us understanding whether our
20:45:50indexing strategy is correct or
20:45:56not. All right friends, so so far we
20:45:58have learned that the SQL server going
20:46:00to make its own decisions on how to
20:46:03execute your queries and the SQL make
20:46:05those plans based on the statistics. But
20:46:07sometimes the plan that you are getting
20:46:09from the database might be not the best
20:46:12one for your query and there could be
20:46:14many reasons why this could happen.
20:46:16Maybe the statistics are not up to date
20:46:19or you have a lot of indexes and the
20:46:21database engine get confused and here
20:46:24exactly where we need the SQL hints. So
20:46:27you can use the SQL hints in order to
20:46:30command to force the SQL database on how
20:46:34exactly your SQL query should be
20:46:36executed. So you can intervene and
20:46:38change the steps in the execution plan.
20:46:41So let's see how we can do that. All
20:46:43right. So now let's have a very simple
20:46:45query. We are just joining the table
20:46:46orders with the customers and we are
20:46:48showing like few columns. Now if you go
20:46:50and execute it and we go and check the
20:46:53execution plan, we can see in this plan
20:46:55that it is using the clustered index in
20:46:57order to read the data from the orders
20:46:59and the customers and then it is using
20:47:01the nested loop in order to do the
20:47:03joins. Now let's say that our tables are
20:47:06really big but still the SQL is using
20:47:08the nested loops and of course this is
20:47:10not good for large tables and maybe the
20:47:13SQL was confused with the indexes and
20:47:15statistics and so on and it decided to
20:47:18use the nested loops. So now in order to
20:47:20force the SQL to use another type of
20:47:23join, we can go and give a hint in our
20:47:25query for the SQL to use different types
20:47:28for the join. So let's go and do that.
20:47:31We're going to go at the end of our
20:47:32query and we're going to say option and
20:47:36inside it we're going to say use the
20:47:38hash join like this. So that's it. This
20:47:42is our query and at the end we are
20:47:44giving the database a hint for the
20:47:46execution plan. So let's go and try that
20:47:48out. So let's check the execution plan.
20:47:50And now as you can see is using
20:47:53different type of join. So with that we
20:47:55are intervening in the execution plan
20:47:58and we are making choices. So with that
20:48:00we have changed the technicality on how
20:48:03the SQL is joining those two tables. All
20:48:05right. So now let's go and change
20:48:07something else like for example instead
20:48:08of having index scan I would like to
20:48:12have an index seek. So if you have the
20:48:14right index in your table, you can go
20:48:16and tell SQL how to read your data in
20:48:20the table. So let's go and do that.
20:48:22Currently here we have an index scan on
20:48:25the table customers. So we can go over
20:48:27here near the table and we're going to
20:48:29say with and inside it we're going to
20:48:31say for SQL force seek. So we are
20:48:34forcing SQL to use the seek index. So we
20:48:37can use those keywords near the table in
20:48:41order to specify for SQL how to load the
20:48:43data. If you are not specifying anything
20:48:45like here with the orders, we don't have
20:48:47here any hints. That means we are
20:48:49counting on the execution plan that is
20:48:52generated from the SQL. But if you don't
20:48:55want the recommendations, you can go and
20:48:57specify which one should be used. So now
20:49:00let's go and execute it. Now we got an
20:49:02error because the SQL is not able to
20:49:05process what we are asking for and I
20:49:08think maybe we are using the force
20:49:09command and as well the hash join. Let
20:49:11me just uncomment this and let's go and
20:49:13give it another try and now it is
20:49:16working. So let's go to the execution
20:49:17plan. So you can see we got again the
20:49:19nested loop. And now if you go to the
20:49:21customers table you can see now it is
20:49:24using the index seek. So it is not using
20:49:26anymore the index scan. So as you can
20:49:29see again we are intervening and forcing
20:49:31SQL to use the method that might be
20:49:34better for our query. Now if you are
20:49:36creating a lot of indexes in one table
20:49:38and the SQL is still not targeting the
20:49:41right index. So if you check the object
20:49:43you can see it is targeting specific
20:49:45index. But if you have a better index
20:49:48than that you can give a hint for the
20:49:50SQL to use a specific index. And we can
20:49:53do that like this. If you go over here
20:49:56and remove the force seek and you say
20:49:58use index and then we have to go and
20:50:01specify the index name. So let's go and
20:50:04get again the primary key over here. Now
20:50:06I'm telling SQL you have to go and use
20:50:08this index in order to scan the table
20:50:11customers. So let's go and try this out.
20:50:13And if you go to the execution plan you
20:50:16can see it is as well targeting this
20:50:18index. So not only you can force SQL for
20:50:21a specific type of loading or joining,
20:50:25you can force SQL to use a specific
20:50:27index that you created. All right
20:50:29friends, so as you can see, SQL hands
20:50:31are very powerful, but we have to be
20:50:33very careful with them because I really
20:50:35had a bad experience using them in my
20:50:38projects. So here are my recommendations
20:50:40and what happens. So what could happen
20:50:41is that you are optimizing the
20:50:43performance in the development database
20:50:46and you start using the hints and the
20:50:48speed was really good and once you roll
20:50:50that out to another database the
20:50:53production database this hint will not
20:50:56be working correctly. The same hint that
20:50:58you are using might not improve the
20:51:00performance and one reason is that
20:51:02sometimes the productive database has
20:51:05like large data compared to the
20:51:07development database. So you have really
20:51:09to test the hint in each database that
20:51:12you have. So if your hint is working in
20:51:15one environment that doesn't mean it
20:51:17going to work in the other one. So
20:51:19always make sure to test. And the second
20:51:21recommendation is that don't use the
20:51:23hint as a permanent fix for your
20:51:25queries. So what this means? Let's say
20:51:27that you are working in the project and
20:51:29one of your queries are very slow. Now,
20:51:31if it's not clear why the execution plan
20:51:33is really bad, you can go and use the
20:51:36hints as a workaround in order to speed
20:51:38up your query again, but it's still as a
20:51:41workaround temporary. You still have to
20:51:43invest and spend time in order to
20:51:45analyze what was the road cause. So
20:51:48maybe it is an old statistics or you
20:51:50have wrong indexing and so on. So use
20:51:52hints only to work around and speed up
20:51:55your queries, but don't use it as a
20:51:57permanent fix. So friends, SQL hints are
20:52:00really amazing in order to control the
20:52:02execution plan, but use it very
20:52:04carefully and only if there is like an
20:52:06emergency.
20:52:11All right friends, so now for each SQL
20:52:14data project, we have to make sure that
20:52:16we create a clear guidance about the
20:52:19index strategy and everyone in the team
20:52:22has to commit and follow the strategy in
20:52:24order to make sure that each index that
20:52:27is created in the project to fulfill a
20:52:29purpose and that's because without a
20:52:31clear strategy about the indexing, I'm
20:52:33going to promise you there will be a lot
20:52:35of redundancy, unused indexes, uh waste
20:52:38of storage and the whole system of your
20:52:41project is going to be slow and bad. So
20:52:43now what we're going to do, I'm going to
20:52:44show you my indexing strategy that I
20:52:46usually follow in my projects. But I'm
20:52:48going to tell you from now there is like
20:52:49not one strategy that can fit any
20:52:52project and any scenario. That's why the
20:52:54team of each project should brainstorm
20:52:56in order to make their own strategy. So
20:52:59now let's have a look to my indexing
20:53:01strategy.
20:53:05And now if I have to pick only one
20:53:07recommendation from me to you in this
20:53:10indexing tutorial, I'm going to have
20:53:12this advice for you. Avoid overindexing.
20:53:15Overindexing is the biggest mistake and
20:53:18trap that a lot of developers do where
20:53:21they think adding more indexes. That
20:53:23sounds like we are speeding up things
20:53:25and our queries can be fast. But I have
20:53:28to tell you this exactly lead to the
20:53:30opposite. And here's why. As we learned,
20:53:33each time you add a new data to your
20:53:35table, your index has to get updated,
20:53:37sorted, rearranged. That means having
20:53:40too many indexes, what's going to
20:53:42happen? Your insert, update, delete
20:53:44operations going to be slow. And this
20:53:46means your database is slower and not
20:53:49faster. And one more very important
20:53:51reason why overindexing is bad is you
20:53:54make the database confused while
20:53:56creating the execution plan. As we
20:53:58learned, the SQL database has to create
20:54:01the best execution plan for your query.
20:54:04And if you have a lot of indexes in your
20:54:06database, it's going to make the process
20:54:08of creating an execution plan
20:54:10complicated for the database, which
20:54:12makes it of course for database harder
20:54:14to choose the best path and index. And
20:54:17as well, you open the door for bad
20:54:19execution plans. And this means it's
20:54:21going to slow the query because first
20:54:23the database has to create the execution
20:54:25plan before executing your query. So
20:54:27again it has a bad effect for the
20:54:29performance and as well there is another
20:54:31bad thing. It can make it harder for the
20:54:33database to decide what is the best
20:54:36execution plan for a query and having
20:54:38too many indexes might make the SQL
20:54:41database choosing a really bad execution
20:54:44plan. So overindexing confuse the
20:54:46execution plan and as well makes the
20:54:49query slower. So that's why I call this
20:54:51a golden rule and you have to commit to
20:54:53it. Just avoid overindexing because it
20:54:56is double-edged sword and exactly you
20:54:58have to have the mindset of less is
20:55:01more. So having a few effective indexes
20:55:04is way better than having a lot of
20:55:06indexes. So keep it in mind and write it
20:55:09in your development guideline for the
20:55:11team with big statement avoid
20:55:13overindexing. So this is the first
20:55:15statement in your indexing strategy. So
20:55:17now let's check the
20:55:18[Music]
20:55:21rest. All right. So now we can split the
20:55:24indexing strategy into four phases and
20:55:27each phase has multiple steps. So now
20:55:29the first step is we're going to go and
20:55:30create an initial indexing strategy. So
20:55:33now once you start a new SQL project you
20:55:36have to define the objectives of the
20:55:38projects very clearly. So that means we
20:55:40have to make it clear what we are
20:55:41focusing on what we want to achieve and
20:55:43in order to define the goal of your
20:55:46indexing strategy you have to understand
20:55:48your system. We have mainly two types of
20:55:51databases. In one hand we have OLAB
20:55:54databases. It stands for online
20:55:56analytical processing. The purpose of
20:55:58this database is for data analytics and
20:56:00an example for that is the data
20:56:02warehouse. So in data warehousing we go
20:56:05and extract the data from multiple
20:56:07sources and then we prepare it and
20:56:09transform it and put it in one big
20:56:11storage and we call this process an ETL
20:56:13process. And then the front end we have
20:56:16like reports and dashboards where the
20:56:18data is summarized and aggregated and
20:56:20presented for the end user. And these
20:56:22reports could be used from users in
20:56:24order to analyze and have insights about
20:56:26the data. And now in order to generate
20:56:28those reports there will be like heavy
20:56:31reading on the data warehouse database.
20:56:34So that means there will be huge queries
20:56:36that's going to access the database in
20:56:38order to aggregate and prepare the data
20:56:40for the visualization. But now in the
20:56:42other hand we have the OLTP systems
20:56:45online transactional processing. It is
20:56:47like an e-commerce finance banking where
20:56:50you have at the back end a database
20:56:52where the data is stored and on the
20:56:54front end we have like an applications
20:56:56for the end users. So now as the users
20:56:58are interacting with the app this can
20:57:00cause write operations on the database.
20:57:03So inserting new data or changing data
20:57:05and as well there will be read
20:57:07operations on the database in order to
20:57:09show the data in the app. So we have
20:57:11both write and read. So now of course we
20:57:13have to ask ourself what is the goal
20:57:15what do we want to achieve and here
20:57:17mainly there is like two strategy either
20:57:19you want to improve the read performance
20:57:22or the right performance. Now if you are
20:57:24looking to the OLAP system here it's
20:57:26really you have to understand the
20:57:27project where is the struggle sometimes
20:57:29it could be like the ATL process itself
20:57:32it's slow and mainly the ATL is writing
20:57:34data from the sources in the data
20:57:36warehouse and maybe you have scenario
20:57:38where it takes like every day 10 hours
20:57:40and 10 hours is of course a problem
20:57:42because you cannot wait so long in order
20:57:45to get a new data fresh data to the
20:57:47report every day. So you can make the
20:57:49goal of the project is to optimize the
20:57:52right performance. You want to speed up
20:57:54the ETL. But actually most of those
20:57:56projects having another issue. Well, it
20:57:59is the read operation on the database
20:58:01because data warehouses normally have
20:58:03really big data sets and at the front
20:58:06end the reports generate large complex
20:58:08queries on the database. So that means
20:58:11the rate process going to be the pain
20:58:13point in each OLAP system. So normally
20:58:16the big goal in each OLAP system going
20:58:18to be how to optimize the read
20:58:21performance. But now in the right hand
20:58:23with the OLTB we have different nature
20:58:25of database and scenario. What going to
20:58:28happen? You will not have like big
20:58:29queries from the apps. You're going to
20:58:31have like many query many transactions
20:58:34happening between the application and
20:58:36the database. So you're going to have
20:58:38like massive amount of read and write
20:58:41transactions. So the whole time we are
20:58:43reading, writing, reading, writing and
20:58:45so on. But with the OL app we have like
20:58:47something bigger and slower because in
20:58:49the ATL we usually run it only once.
20:58:52That means we are writing only once new
20:58:54data to the database and this happen
20:58:56usually at the night but on the
20:58:58transactional systems you have a lot of
20:59:00readrs all time. Again depend on the
20:59:02project but usually the main pain point
20:59:05in the OLTP is the right operation. So
20:59:08it could be like this. If you are
20:59:10building OTP system, the main goal is to
20:59:13optimize the right performance. Now of
20:59:15course the question is how to do that?
20:59:17How we going to optimize that? Well,
20:59:19again we have to understand the nature
20:59:20of the database. What do we have in the
20:59:22OLAP systems is usually like a data
20:59:25model where you have a very big fact
20:59:28tables and around the fact we have like
20:59:30multiple dimensions that are connected
20:59:32to the facts. So those fact tables are
20:59:34really big tables in the database and
20:59:37each time they are used in order to
20:59:39build a report and the report going to
20:59:41be using all time those facts in order
20:59:44to prepare the data for the
20:59:45visualizations and a lot of aggregations
20:59:48query going to be done on the facts and
20:59:50now of course you have to answer now the
20:59:52question which type of index should we
20:59:54use in this scenario. Well we have a
20:59:56perfect one called a column store index.
20:59:59So the best practice here is and you can
21:00:01make it as a strategy for the whole
21:00:03project that we make all fact tables as
21:00:06a column store index because this is
21:00:08what we are doing in the OLAP. We are
21:00:10aggregating large data sets but now the
21:00:13data model and the scenario is
21:00:15completely different at the right side
21:00:16here. We're going to have like a lot of
21:00:18tables and they have like different
21:00:20sizes and so on and there are like a lot
21:00:23of relationship between all those
21:00:25tables. So it is completely connected.
21:00:27So you have a lot of like primary keys
21:00:29and foreign keys relationships between
21:00:31them and normally those tables are
21:00:33completely normalized table. So they are
21:00:36like small pieces but on the left side
21:00:38we have denormalized tables as a facts.
21:00:41So here is like one strategy that we can
21:00:43follow in the indexing of the ALTB is
21:00:45that we create clustered index for each
21:00:48primary key of our tables. This of
21:00:51course can improve a lot of stuff like
21:00:52searching, sorting and as well joining
21:00:55tables together. But of course since we
21:00:57are focusing on optimizing the right
21:00:59performance on the OLTP you have to be
21:01:01more sensitive by adding new indexes
21:01:04compared to the OLAP because each index
21:01:06you add it could be a reason why the
21:01:09data is written very slowly. So in the
21:01:12OLTB you have to be way more careful
21:01:14adding indexes. So now as you can see
21:01:16you have to understand the nature of
21:01:18your project. You have to understand
21:01:20what is the main issue. Once you
21:01:22understand your project, you can go and
21:01:24define like a goal for optimizing the
21:01:27system. So either read or write or maybe
21:01:30both of them and with that you are
21:01:32making like the initial strategy of
21:01:34indexing your
21:01:35[Music]
21:01:39system. All right. So with that we have
21:01:41an initial strategy for our indexing and
21:01:44we have a rough plan. Now in the next
21:01:47phase we have usage patterns indexing.
21:01:50So now we're going to do a deep dive
21:01:51into our project. And the first thing
21:01:53that we have to do is that we have to
21:01:55identify the frequently used tables and
21:01:57columns. So that means you have to go
21:01:59and check the queries used in your
21:02:01project in order to understand okay what
21:02:03is the most important table that is used
21:02:06in many queries. Like for example here
21:02:08we have the fact internet sales. It is
21:02:11used like in many many queries in our
21:02:13scripts. So here you are like developing
21:02:15a feeling about what are the most
21:02:18important frequently used tables and not
21:02:20only that you can go and check how we
21:02:22are filtering the data on those queries.
21:02:24So for example we have over here we are
21:02:26filtering by the order date key is this
21:02:29kind of filtering is used like in
21:02:31multiple queries. So as you can see we
21:02:32have like here a couple of queries where
21:02:34we are doing always the same where we
21:02:36are filtering the data by the dates. So
21:02:39with that we understand there is like a
21:02:40pattern inside our projects where this
21:02:43column is used mainly on filtering and
21:02:45as well for aggregating. So that means
21:02:47you do a deep dive in order to
21:02:49understand what are the most and
21:02:51frequently used tables and columns
21:02:53inside your scripts. And now of course
21:02:55what I usually do I go and use the help
21:02:57of the AI and IBT where I give it my
21:02:59code and then ask questions about it.
21:03:02For example, this prompt, it says, "Anal
21:03:03analyze the following SQL queries and
21:03:05generate a report on table and column
21:03:07usage statistics. And for each table,
21:03:09provide the total number of times the
21:03:11table is used across all queries. A
21:03:14breakdown for each column in the table
21:03:16showing the number of times each column
21:03:18appears. And I would like to see as well
21:03:20the primary usage of each column,
21:03:22filtering, joining, grouping, and so on.
21:03:24And in the output, as you can see, we
21:03:25got like nice statistics about my
21:03:27scripts. So as you can see the most used
21:03:30fact table is fact internet sales. It is
21:03:33like 13 times used in the projects and
21:03:35then we can see like statistics about
21:03:37each column that is inside these facts.
21:03:39So most of the time is the sales is used
21:03:42for aggregating and as we saw the order
21:03:45date key is used like five times for
21:03:47filtering and the other keys is used for
21:03:50joining tables. So as you can see it's
21:03:52amazing right now we can identify which
21:03:55tables are important which columns as
21:03:57well are important and we can like based
21:03:59on those informations maybe derive our
21:04:02indexing for our database. So with that
21:04:04we have identified our frequently used
21:04:06tables and columns and now the next step
21:04:09we have to go and choose the right index
21:04:12type and as we learned before we have
21:04:14multiple types of indexes and that's
21:04:16really depend on the usage and the
21:04:18scenario. So for examples, if your
21:04:20columns are primary keys, then go with
21:04:21the clustered index. And if you are
21:04:23using columns that are not primary key
21:04:25where you are doing joining filtering,
21:04:27then think about the non-clustered
21:04:29index. And of course, if the table is
21:04:31very big, as we said, you can go and use
21:04:32the column store index. And if you are
21:04:34targeting always like a subset of data
21:04:37only like one year informations, then
21:04:39you can think about the filtered index.
21:04:41And the last one, if you have like a
21:04:43unique column where you don't have any
21:04:44duplicates, then you can go and apply a
21:04:46unique index. So it depends on the
21:04:48scenario and the usages. You have to
21:04:50choose the right index. And of course
21:04:52the last step in this phase is that you
21:04:54have to go and test your index whether
21:04:56everything is working
21:05:01fine. So that's all for the phase two.
21:05:04Then we go to phase three scenario-based
21:05:06indexing. So here we have to tackle and
21:05:08focus on specific issues to specific
21:05:12pain points. So that means we have first
21:05:14to identify the slow queries. So it
21:05:17could be reported from users or the team
21:05:19is doing like analyzing on the logs and
21:05:22to understand which queries are causing
21:05:24like performance issues. And now once
21:05:27you get a list of slow queries then you
21:05:29have to analyze them one by one and it
21:05:32is time to dig into the execution plans.
21:05:35So as we learn we can check how SQL is
21:05:37implementing our queries and start
21:05:39looking for areas for example where the
21:05:41SQL is doing a full scan of the tables
21:05:44or maybe using expensive operations like
21:05:47nested loop joins and so on. So once you
21:05:49understood where is exactly the pain
21:05:52point the next step is that you have to
21:05:54go and choose the right index. So which
21:05:56type of indexes we're going to use in
21:05:58order to optimize the query. And once
21:06:00you go and create the index, the last
21:06:03step is that you have to go and test it.
21:06:05So you're going to run again the
21:06:06execution plan in order to make sure
21:06:08that your query is using the index that
21:06:11you have just created. So that means you
21:06:13have to go and compare the execution
21:06:15plans before and after. And if you see
21:06:17that there is no benefit, then something
21:06:20is wrong. That means you have to go and
21:06:22investigate more and analyze the
21:06:23execution query and maybe choose a
21:06:26better index way. And you have to do
21:06:28this process for each slow query until
21:06:31you get all your queries fast. But of
21:06:33course, don't forget indexing is not the
21:06:35only methods on how to optimize the
21:06:37speed of queries. So as you can see
21:06:40through these three phases, we went from
21:06:42a very generic methods on how to index
21:06:45our system to something very specific
21:06:48and scenario based. So as you can see as
21:06:50we moving in the phases, we are doing
21:06:52more deep dive into our projects.
21:06:59All right. So now moving to the last
21:07:00phase, we have the monitoring and
21:07:02maintenance of our indexes. As we
21:07:05learned, the job doesn't stop by just
21:07:07creating and implementing indexes. We
21:07:09have to be responsible by keeping eye on
21:07:12the health of our indexes. And here the
21:07:14databases offers a lot of statistics and
21:07:17metadata about your data that you could
21:07:19use in this phase. So the first step is
21:07:22to monitor the usage of the indexes. And
21:07:25as we learned, we can use the dynamic
21:07:27management views or functions that we
21:07:30can find in the system schema where we
21:07:32can see the number of usage of each
21:07:34index and when the last time our queries
21:07:37did use the indexes. So with that we can
21:07:40go and find out all those indexes that
21:07:42we have created and never been used in
21:07:45our projects. And now the next step is
21:07:46that we can go and monitor the missing
21:07:48indexes. So here we can go and check
21:07:51what are the recommendations from the
21:07:52database where the database is reporting
21:07:56missing indexes from the execution plan
21:07:58and again we can go and use those
21:08:00dynamic management views or functions in
21:08:02order to see more details and as well we
21:08:05can go and monitor whether we have
21:08:07duplicates in the indexing. It happens a
21:08:09lot if you have like a lot of developers
21:08:11in your team. So it could be that they
21:08:13are working parallelly to optimize the
21:08:15performance of slow queries and then go
21:08:18and create multiple indexes for the same
21:08:20column. So this is something that we can
21:08:22go and check whether we have duplicates
21:08:24in our indexes and if you have
21:08:25duplicates then you have to go and find
21:08:27how you can go and consolidate them.
21:08:29Then the next step we have to go and
21:08:31update the statistics. So as we learned
21:08:34statistics are very important for the
21:08:36execution plan because the database
21:08:38engine use those informations to decide
21:08:41the best execution plan for your query
21:08:43and if the statistics are old then the
21:08:45database going to make wrong decisions
21:08:47about how to execute your query which
21:08:50might lead to bad performance. So here
21:08:52again we have like special functions in
21:08:54order to monitor the statistics but here
21:08:56my recommendation that each weekend have
21:08:59a job that go and create all the
21:09:01statistics of your database. And the
21:09:03last step we don't have to forget about
21:09:04monitoring the fragmentations as we
21:09:06learned over the time as you are doing
21:09:08modifications on the tables. What could
21:09:10happen the order of the databases could
21:09:12get wrong or there are like free spaces
21:09:15on the database that are not used. So we
21:09:17have like fragmentations in the index
21:09:20and the same thing we have to monitor
21:09:21the fragmentations of each tables and
21:09:24here if the percentage is between 0 and
21:09:2610 then there is no issue but if the
21:09:28fragmentation is between 10 and 30 then
21:09:30we have to go and reorganize the index
21:09:33and if it's more than 30 then this is
21:09:35alerting you have to go and rebuild the
21:09:38whole index and usually for the
21:09:40monitoring I go and build like automated
21:09:43dashboard in PowerBI or Tableau where I
21:09:45go and extract all those metad data and
21:09:47create a nice dashboards in order to
21:09:50monitor the health of the database or
21:09:51you can go and buy some other tools that
21:09:54are advanced in order to do those
21:09:59stuff. All right. So this is my indexing
21:10:02strategy that I usually follow in my
21:10:03projects. And as you can see, each phase
21:10:05builds upon the previous one. Moving
21:10:08from a general strategy to more
21:10:10targeted, refined, specific strategy
21:10:13where we define first the goal of the
21:10:15indexing strategy of the projects. And
21:10:17as we move with the phases, we're going
21:10:19to be targeting more specific scenarios.
21:10:22And this cycle keep repeating. It's not
21:10:24only one time. So you have to keep
21:10:26discussing is the goal still suitable
21:10:28for the projects. You have to keep
21:10:30analyzing the frequently used tables and
21:10:33columns and keep searching and finding
21:10:35those slow queries and always keep an
21:10:38eye monitoring the indexes and of course
21:10:41I can only keep repeating this avoid
21:10:44overindexing. All right my friends so
21:10:46that's all about the indexes that was a
21:10:48lot of informations and a lot of
21:10:50technique. So now you know everything
21:10:51about indexing in SQL. Now in the next
21:10:53one there is another important
21:10:55techniques on how to optimize the
21:10:57performance. So we're going to talk
21:10:58about the partitions. So how to divide
21:11:00our data in order to optimize the
21:11:02performance. So let's
21:11:07go. All right. So what is SQL
21:11:09partitioning? It's a technique in order
21:11:11to divide a large table into small
21:11:14pieces and each piece we call it a
21:11:16partition. Well, this sounds like we are
21:11:18dividing one big table into smaller
21:11:20tables but it's not like that. We are
21:11:22just dividing one table into smaller
21:11:25partitions. So we going to see it in the
21:11:27database still as one solid table but
21:11:30behind the scenes it is splitted into
21:11:32multiple partitions. So now let's go and
21:11:34understand what this means. Okay. So now
21:11:36let's say that you have a table at your
21:11:38database and over the time this table is
21:11:41getting bigger and bigger where you have
21:11:43like hundreds of millions of rows. Now
21:11:45once you have such a big table what's
21:11:46going to happen everything going to be
21:11:48slow. So for example, if you are reading
21:11:50the table and the execution plan is
21:11:52doing full scan of the table, this can
21:11:55take SQL long time until all the rows
21:11:57are fetched. And if you decide to make
21:11:59like an index for this table, what's
21:12:02going to happen? SQL going to go and
21:12:03build a very big B tree index where
21:12:06there are a lot of branches and files
21:12:08and so on. And having a big index is not
21:12:11always a good thing because if you do
21:12:13operations like delete rows, update rows
21:12:16or inserting rows, these operations
21:12:19going to need long time to process. So
21:12:21having a big index doesn't mean that you
21:12:24can have a good performance for your big
21:12:26table. So that means having a big table
21:12:28is a problematic because everything
21:12:30going to be slow. So now what we can do
21:12:32in order to optimize the performance of
21:12:35this big table? Well, we can use SQL
21:12:38partitioning and in order to do that, we
21:12:40have to understand the behavior and the
21:12:43transactions that are happening on our
21:12:45table and what usually happen with that
21:12:47the table grows over the time. So, you
21:12:50can have like subset of data that
21:12:51belongs to 2023 and another one that is
21:12:54created and updated in 2024 and then you
21:12:57have something like more current in
21:12:592025. So that means we have like in our
21:13:02table old data and as well new data and
21:13:05we usually interact with the new data
21:13:07more often than the old data. So maybe
21:13:10for example for 2023 there is like only
21:13:12one read transaction and for the data in
21:13:152024 we have done like two reads and one
21:13:18rights. So it is little bit more than
21:13:202023 but for the new data for the
21:13:23current year there will be heavy
21:13:25transactions. So we're going to have a
21:13:27lot of reads a lot of rights. We are
21:13:29updating, inserting, reading. So a lot
21:13:31of things are going on for the new data.
21:13:33So that means we are accessing
21:13:35frequently the big table only to
21:13:37interact with the new data and we rarely
21:13:40need the old data. So what we can do, we
21:13:42can go and divide this big table and we
21:13:45usually divide it by like a date. So
21:13:48that means we can go and split this
21:13:49table by the year and we put each year
21:13:52in one partition. So at the end we're
21:13:54going to have like three partitions. And
21:13:55now it's really important to understand
21:13:57that that those are three partitions.
21:13:59They are not three tables. So that means
21:14:01at the client side the users can see
21:14:04only one table but behind the scenes we
21:14:07have like three partitions. Now let's
21:14:08say that you have a query in order to
21:14:10read the data from 2025. And now what
21:14:13going to happen? SQL will not go and
21:14:14scan all the data from the table. It's
21:14:16going to go and only target one
21:14:18partition the 2025. So that means SQL is
21:14:22only scanning the relevant informations
21:14:24the relevant partition and not the
21:14:26entire table. And now we have another
21:14:28benefits of having partitions. Let's say
21:14:30that you're using a modern database and
21:14:33normally they support parallel
21:14:34processing. So if you have the
21:14:36infrastructure for that what can happen
21:14:37the database engine can process each
21:14:40partition independently and parallelly.
21:14:43So whether you are reading or writing
21:14:45data. So what's going to happen? SQL
21:14:47going to process your queries parallelly
21:14:49which of course can reduce the overall
21:14:52execution time. So that means if you
21:14:54have a modern infrastructure like maybe
21:14:56for example the Azure Synapse and so on
21:14:58go with the partitions because the
21:15:00partition then could be stored in
21:15:02different servers and this helps of
21:15:04course the SQL engine to use all the
21:15:07resources at once. So that means
21:15:09partitions allow scalability and as well
21:15:12parallel processing. partitions going to
21:15:14make the indexing more efficient. So
21:15:17instead of having one very big index for
21:15:20the whole table, if you put an index on
21:15:22a partition table, what's going to
21:15:23happen? Each partition going to get its
21:15:26own index, which means the size of the
21:15:28indexes going to be smaller. And of
21:15:30course, this helps a lot with searching
21:15:33for data or as well extending the index
21:15:35itself. So for example, if you are
21:15:37inserting data to the partition 2025,
21:15:40the SQL will not go and change anything
21:15:42on the other indexes, it's going to go
21:15:44and only change the index of the
21:15:46partition 2025. So that you can see the
21:15:48power of the partitioning. It improves
21:15:51significantly the performance of your
21:15:53table whether you are reading or writing
21:15:55data to this big table. So this is what
21:15:58we mean with partitioning and why we
21:15:59need
21:16:03it. All right, friends. So now we're
21:16:05going to go to the process of creating
21:16:07partitions in SQL. At the start it might
21:16:10sounds a little bit complicated but
21:16:11we're going to do it step by step and I
21:16:13have a sketch for that. So we have like
21:16:15four steps because we have in the
21:16:17database like multiple layers. So let's
21:16:19see how we can do that. Let's go. So the
21:16:22first step is that we're going to go and
21:16:23define the partition function. So what
21:16:25is that? We're going to go and define
21:16:27here in the function the logic on how to
21:16:30divide the table into partitions. And
21:16:32this can be based on the partition key.
21:16:35So that means we need a column in order
21:16:37to define the logic. And we usually use
21:16:39columns with the dates like for example
21:16:41the order dates or in other scenarios we
21:16:44can use the region or country and so on.
21:16:46But the most famous one is the dates and
21:16:48that's because our tables like get
21:16:51bigger over the time and there are like
21:16:53multiple types of functions. We're going
21:16:55to focus on the range function. So how
21:16:57it going to work? We're going to have
21:16:58like a range of dates and then we have
21:17:00to define like boundary values and let's
21:17:03say that I would like to make a
21:17:05partition for each year and in order to
21:17:08do that we have to define the partition
21:17:10boundary. So it is like a value the
21:17:12boundary of the years could be like the
21:17:14first day of the year or the last day of
21:17:16the year. So here in this example we're
21:17:18going to take for the boundary the last
21:17:20day of the year. So the last day of
21:17:222023, 2024 and 2025. So we call those
21:17:26values the boundary of our function. Now
21:17:30between the boundaries we going to have
21:17:31our partitions. So for examples all the
21:17:34rows for 2025 and earlier years is going
21:17:38to be the partition one. So between the
21:17:40boundary and everything before is one
21:17:42partition and after that between the two
21:17:44boundaries we have partition two. So
21:17:46this partition going to be for all rows
21:17:48of 2024. And then we have another
21:17:51section the partition three where we
21:17:53have all rows of 2025 and then between
21:17:56the last boundary and everything onwards
21:17:59is going to be partition 4 and here
21:18:00we're going to have all the rows from
21:18:022026 onward. So with that we have now a
21:18:05logic we are telling SQL how to divide
21:18:08our data into multiple partitions and
21:18:10here there is like two methods the left
21:18:12and the right. So what are those two
21:18:15methods? So again we have our boundary
21:18:17and now the big question to which
21:18:19partition does this boundary belongs to
21:18:22is it partition one or partition two and
21:18:24that's why we have those two methods. If
21:18:26you say it is left that mean the
21:18:28boundary belongs to the partition number
21:18:30one. But in the other hand if you say it
21:18:32is right then the boundary going to be
21:18:34part and belongs to the partition number
21:18:37two. So you have to decide whether the
21:18:39boundaries belongs to the left partition
21:18:40or to the right partition. And with that
21:18:42in the partition one, we're going to
21:18:44have all the rows of 2023 including the
21:18:47last day of 2023 because in the
21:18:50partition 2 we only focus on 2024. So
21:18:53it's just the boundary belongs to the
21:18:56left partition. It's very simple. Now
21:18:57let's go and implement that in SQL. So
21:19:00let's do it. The syntax is very simple.
21:19:02We're going to say create partition
21:19:05function and then we have to give it a
21:19:07name. So it's going to be
21:19:09partition by year since we are dividing
21:19:12the data by the year. And after that we
21:19:14have to define the data type. So we are
21:19:17splitting the data by a date. So it's
21:19:19going to be date. And after that we have
21:19:21to define the partition function type.
21:19:24So in our example we are using the
21:19:26range. And now we have to define whether
21:19:28it is left or right. We're going to
21:19:31stick with the left. And now comes the
21:19:33very important step. We have to define
21:19:35the boundaries. So we're going to say
21:19:38for
21:19:39values and we're going to enter here
21:19:41three boundaries like in our example for
21:19:43each year we're going to define a date.
21:19:45So
21:19:472023 and the last day of the year. Same
21:19:50goes for
21:19:522024 and for the last one
21:19:562025. So with that we have defined the
21:19:59logic the range we have defined the
21:20:02boundaries and we tell SQL the
21:20:04boundaries are a date. So let's go and
21:20:06execute our function. Okay, so that's
21:20:09it. As you can see, it's very simple. We
21:20:11just created a function that split the
21:20:14data by the date using the range lift.
21:20:16And of course, this function is not yet
21:20:18attached to any tables or anything. It
21:20:21is just a logic that is stored in the
21:20:23database. All right. So now since our
21:20:24partition function is stored inside the
21:20:26database, we will have metadata about
21:20:29those functions stored in the system
21:20:31schema. So we have there a dedicated
21:20:33table called partition functions and
21:20:35there we're going to find informations
21:20:37about all functions that we have inside
21:20:39our database. So let's go and execute
21:20:41it. And as you can see we find now our
21:20:43new created partition function. So
21:20:46partition by year it is a range and it
21:20:48has an ID and so on. And I really
21:20:50recommend you to check it before
21:20:52creating any new partition function.
21:20:54Maybe you have already one in the
21:20:59projects. Okay. Okay. So now let's check
21:21:01the next step in our process. We're
21:21:03going to go and build now the file
21:21:05groups. So what is a file group? It is
21:21:08like a logical container of one or more
21:21:11data files. So it's very simple. It's
21:21:13like folders. We're going to go and
21:21:15create now like multiple folders. So
21:21:17later we can insert inside them files.
21:21:20And this is really nice because it gives
21:21:22us like freedom and flexibility where we
21:21:25can go and decide how the data files are
21:21:27organized for each partition. So what we
21:21:30usually do, we go and create for each
21:21:32partition a file group. So we're going
21:21:34to have like four folders or four file
21:21:36groups for 2023, 2024 and so on. So now
21:21:39let's go back to SQL in order to do
21:21:41that. All right. So now let's go and
21:21:43create those file groups. The syntax is
21:21:45very simple. So it's going to say alter
21:21:47database. And now we have to tell the
21:21:49database where these file groups should
21:21:51be stored in which database. So I'm
21:21:53going to stay with the sales DB. And
21:21:55then we have to tell okay add file group
21:21:59and after that we have to define the
21:22:01name of the file group. So the first one
21:22:02going to be for
21:22:032023. So the syntax is very simple.
21:22:06Let's go and do it for the other years.
21:22:09So we need
21:22:112024 5 and six. Okay. So that's all. We
21:22:14can just select everything and execute.
21:22:17So as you can see it's very simple. We
21:22:18have just created four file groups and
21:22:21they are empty. So we don't have
21:22:23anything inside those containers. Now
21:22:25let's say that you have made mistake
21:22:26with the namings and so on and you would
21:22:28like to drop one of them. So the syntax
21:22:30is as well very easy. So it's going to
21:22:33say alter database sales DB and instead
21:22:35of add you're going to say remove. So
21:22:38once you execute this file group will be
21:22:41dropped but we need it. So let's go and
21:22:44recreate it. Now as usual after creating
21:22:46stuff let's check whether everything is
21:22:48created correctly and whether we have
21:22:51any duplicate or anything wrong. So with
21:22:53that we have as well a file group table
21:22:56inside the system schema and let's go
21:22:58and execute it. So I'm just filtering
21:23:00with the type FG for file group. So
21:23:03let's execute it. And now we can see in
21:23:05our database we have four file groups.
21:23:08Now four of those file groups we just
21:23:10created it right. So we have the 2023 24
21:23:13and so on. But we have something called
21:23:16primary file group. This is the default
21:23:18file group that is created for each
21:23:21database. So it is a container for all
21:23:23data files in your database. And as you
21:23:25can see we have here a flag saying it is
21:23:27a default. So it's default and we have
21:23:29it one and for the rest they are not the
21:23:31defaults. So this is really nice to see
21:23:33all the file groups inside your database
21:23:35to check that you don't have duplicate
21:23:37and so
21:23:41on. Okay. Now moving on to the third
21:23:43step where things going to get more
21:23:45physically. So so far we have like a
21:23:47function the file group and all those
21:23:48stuff are logical stuff. We don't have
21:23:50data yet. In order to have data, we have
21:23:53to go and create data files. So, as we
21:23:55learned before, data files going to
21:23:57contain our actual data and they're
21:23:59going to be stored physically in the
21:24:00database. So, you can go and assign for
21:24:02each file group like one or multiple
21:24:05data files. And the file format here is
21:24:08MDF. It is secondary data files. We have
21:24:11like primary and secondary. But in the
21:24:13partitions, we usually go with this
21:24:15format, the NDF. So again the file
21:24:17groups are illogical containers and the
21:24:19data files are physical files where our
21:24:22actual data going to be stored inside
21:24:23it. So now let's go back to SQL in order
21:24:25to create some data files. Okay. So now
21:24:27we're going to come to the little bit
21:24:29annoying part where we're going to go
21:24:30and create files. But the syntax is as
21:24:32well very simple. So we're going to say
21:24:34the same things alter database and our
21:24:37database is sales DB. And then this time
21:24:39we're going to say add file. And now we
21:24:42have to give SQL not only the name but
21:24:44the physical place of the files. So
21:24:46let's do it step by step. We're going to
21:24:48open new two parenthesis. So first we
21:24:50have to define for SQL the logical name.
21:24:53It is not the file name. It is the
21:24:55logical name of the file. So let's give
21:24:57it a name for example B 2023 and then
21:25:00comma. So this is the logical name. And
21:25:04now the next one is we're going to give
21:25:05the physical name of the file together
21:25:08with the path. So we're going to say
21:25:09file name equal and now we have to
21:25:12define for SQL the complete path of the
21:25:15file in SQL server there is like a
21:25:17default path where the data going to be
21:25:19stored and I'm going to go and use the
21:25:21same path and the path really depends on
21:25:24the version and as well the type of the
21:25:26SQL server that you are using. So for
21:25:28the current version that I'm using for
21:25:30this tutorial we can find it over here
21:25:32in this path. So if you go to the C then
21:25:34program files Microsoft SQL Server MSSQL
21:25:38and the version for me is 16 SQL Express
21:25:41and then inside MSSQL data and so on. So
21:25:44we're going to go inside this folder and
21:25:46now we can see over here all the
21:25:48database files. So we can see for
21:25:50example here the sales DB the sales DB
21:25:53logs and we have here the adventure
21:25:55works and so on. So you're going to see
21:25:56all the files of your database. And what
21:25:58we're going to do, we're going to put as
21:25:59well our partitions files inside the
21:26:02default folder. But for real project,
21:26:04you have to ask the database
21:26:05administrators about the exact location
21:26:08where you can put your partitions. So
21:26:10let's go back to SQL and I'm going to
21:26:12put this path over here. And then we
21:26:14have to specify the file name. So it's
21:26:17going to be P 2023 dot. And now we have
21:26:20to specify the file name. So, NDF and
21:26:23with that we have now a complete path
21:26:25with the file name. So, we are almost
21:26:27there but we are not done yet. We have
21:26:29to tell SQL where to put this file in
21:26:31which container in which file group. So,
21:26:34we're going to go over here and we're
21:26:35going to say to file group and here make
21:26:39sure to select the correct one. So, FG
21:26:422023. All right. So, that's all. Let's
21:26:45go and execute it. So, let's do it. And
21:26:48with that we have created a file inside
21:26:50a file group. I will not be creating
21:26:52like multiple files inside one file
21:26:54group. It's going to be like one to one.
21:26:56So now what we're going to do we're
21:26:57going to go and create the other files
21:26:58for each file group for each year. So we
21:27:01just have to copy and paste and just
21:27:03change the names. So for
21:27:062024 going to be like
21:27:08this. So that's it. And the same thing
21:27:11for 2025.
21:27:17And for the last one 20
21:27:2426 and we can go and select now
21:27:27everything and execute it. So that's it
21:27:30with that we have created now four
21:27:32different files and we have mapped as
21:27:34well each file to the correct file group
21:27:37and I usually don't create like a lot of
21:27:39files. I just create like one for each
21:27:41year or maybe for bunch of years. So you
21:27:44don't have to go and make for each day
21:27:46like partition or something like that.
21:27:48Okay. As usual after creating stuff we
21:27:50have to go and check the metadata. Now I
21:27:52have here prepared a query where we
21:27:54query the file groups together with the
21:27:56files. So all the data informations
21:27:58could be found inside the table master
21:28:00files and then we join those tables and
21:28:02select our database. So let's go and
21:28:05query this one. And now we're going to
21:28:07get a list of all files inside your
21:28:09database. So we see over here we have
21:28:12the primary for the database itself and
21:28:15you can see the path of the file and as
21:28:17well the size of it and we can see over
21:28:19here we have four files and the file
21:28:21group that is assigned to and the
21:28:23complete path of each file and you can
21:28:26monitor over here of course how the size
21:28:28of each file is growing over the time.
21:28:31Maybe one of them is getting like really
21:28:33big and then you can think about let's
21:28:35go and split it to multiple files. So
21:28:37that's it about how to create data
21:28:39files.
21:28:43All right. So now we're going to move to
21:28:45the last step where we're going to go
21:28:46and define the function scheme. Now if
21:28:48you have a look to this picture, you see
21:28:50that there is something missing. From
21:28:52one side, we have defined how to divide
21:28:54our data into multiple partitions. And
21:28:57from the other side, we have repaired
21:28:58all the files and the file groups and so
21:29:01on. And now what is missing is the
21:29:03connection. How to connect those
21:29:05partitions to the file groups. And we
21:29:08can do that by using the partition
21:29:09scheme. So all what we are doing now is
21:29:12just defining which partition belongs to
21:29:14which file group. So for example, we're
21:29:16going to go and map the partition one to
21:29:18the file group 2023. And with that all
21:29:21the data of 2023 and earlier going to go
21:29:24to the file group 2023. And of course we
21:29:28have to go and map each partition to a
21:29:30file group. If you don't do that, you
21:29:32will get error in SQL. And once we build
21:29:34the partition scheme then we can have
21:29:36all the component ready in order to have
21:29:39partition table. So now let's have a
21:29:41quick summarize. The partition function
21:29:44going to decide on how to split your
21:29:46data into multiple partitions. The
21:29:48partition scheme going to go and map the
21:29:50partitions to a file group. And the file
21:29:53groups are like folders in order to
21:29:55organize your files. And each file group
21:29:57has one or more data files where your
21:30:00actual data going to be stored
21:30:01physically. add these files at the
21:30:04start. It might be confusing, but now as
21:30:06you understand each layer, then it's
21:30:08going to make it easier for you to build
21:30:09partitions. So now let's go back to SQL
21:30:12in order to build the partition scheme.
21:30:14Okay, so now we have the easiest part
21:30:16where we're going to connect everything
21:30:17together. So the syntax as well very
21:30:19simple. It's going to say create
21:30:21partition scheme and now we have to give
21:30:24it a name. So let's go with like scheme
21:30:28partition by year. And now we have to
21:30:31map the partition function with the file
21:30:33groups. So first we're going to say as
21:30:35and then we define here the partition
21:30:38function. So as partition and now we
21:30:41need the partition function that we have
21:30:43created. So as
21:30:44partition by year and then after that
21:30:48we're going to map it to the file
21:30:49groups. And here it is very important to
21:30:52map it in the correct order. So the
21:30:54order is very important. So the first
21:30:57one was file group 2023. The second one
21:31:012024 and we have 2025 and the last one
21:31:072026. So again the order is very
21:31:10important and as well it's going to be a
21:31:11little bit tricky. So sometimes as you
21:31:13are creating like the functions maybe
21:31:15you make mistake that you don't know how
21:31:17much partitions are going to create like
21:31:19in our example we have three boundaries
21:31:21and SQL going to create four partitions.
21:31:23So it happens sometimes that you think
21:31:25okay I have three boundaries and then
21:31:28I'm going to get three partitions which
21:31:30is not really correct. So for example
21:31:32let me just remove one of those and
21:31:34let's say I have only three five groups
21:31:37and let's go and execute this one over
21:31:39here. Now we are getting error. It says
21:31:41the partition function generates more
21:31:44partitions than the five groups. And
21:31:46that is really correct because our
21:31:48definition of the logic can split the
21:31:50data into four partitions. And now we
21:31:52are giving SQL only three five groups
21:31:55which is not correct. So we have to go
21:31:56and add the plus one. And one more thing
21:31:59SQL will not go and check whether you
21:32:01are mapping things correctly to the five
21:32:03groups because it doesn't really care
21:32:05about the naming of those five groups.
21:32:07So for example, if you go and put this
21:32:09one at the end, what's going to happen?
21:32:11It's going to be a big problem. So all
21:32:13the years of 2023 going to be stored
21:32:15inside 2024, 2024 going to be in 2025.
21:32:19So everything going to be mixed and the
21:32:21skill can do it like you tell it. So
21:32:23that's why make sure you have the
21:32:25correct sorts. So that's it. Let's go
21:32:27and create our scheme. So it is working.
21:32:30This is very simple. We just map now the
21:32:33partitions to the five groups. And as
21:32:35usual we check things after creating and
21:32:38I have prepared here like really nice
21:32:40query from the metadata in order to see
21:32:42the whole thing the functions the file
21:32:44groups the schemes you can of course add
21:32:46to it the data files but I'm just going
21:32:49to stick with this over here. So again
21:32:51in SQL server we have a dedicated table
21:32:53for the partition schemes. Then I'm just
21:32:55joining it with the functions and then
21:32:57with the destination data spaces in
21:32:59order to get the partition number and
21:33:01the file groups. So let's go and execute
21:33:03it. And now we can see very nicely the
21:33:05scheme that we have created and the
21:33:07function name of the partition. And then
21:33:09we can see the partition number and the
21:33:12file group name. So we can see how
21:33:14things are mapped together. So if you
21:33:16get it like this then so far everything
21:33:18is
21:33:22good. All right. So so far what you have
21:33:24done we have prepared all the layers. So
21:33:27we have the setup is ready to be used in
21:33:30any table. So we have the functions, the
21:33:32files, the file groups and schema and
21:33:34everything is ready. But still we are
21:33:36not using it. The logic just exist and
21:33:38the files are empty. So now what we're
21:33:40going to do we're going to go and create
21:33:41a table but not a normal one a partition
21:33:44table. So let's go and do that. It's
21:33:46very simple as well. So create table and
21:33:48we have to give it a name. So let's get
21:33:50it as well in the schema sales orders
21:33:53and I'm just going to give it the name
21:33:54partitions. So now we have just to
21:33:56define like few columns inside this
21:33:58table. So let's get an order ID and data
21:34:03type int. And let's go and get an order
21:34:05date. We call it dates with the data
21:34:08type dates. And maybe just one more
21:34:10called sales and a data type in. So this
21:34:13is very normal table that we create in
21:34:15databases. But it's still not yet
21:34:17partitioned. Now in order to use
21:34:19everything that we have defined, we're
21:34:21going to go do the following. We're
21:34:22going to say on and now we have to tell
21:34:25SQL only the name of the partition
21:34:28scheme. So everything else is like
21:34:30connected and mapped together because
21:34:32the scheme is mapping the function with
21:34:34the file groups. The file groups are
21:34:36mapped to the data files and everything
21:34:38is like connected together. And here in
21:34:40the table we have just to give the name
21:34:42of the scheme. So the name of the
21:34:43partition scheme is scheme
21:34:46partition by year. And now it's very
21:34:49important to give a column. And since
21:34:52the whole logic and the function is
21:34:54based on a date, we cannot go and
21:34:56specify here for example the order ID or
21:34:58sales because it makes no sense. We're
21:35:00going to go and pick the order date and
21:35:02put it over here. And with that, we have
21:35:04created a partition table. So now what
21:35:06we're going to do, we're going to go and
21:35:07start inserting that out of our table.
21:35:09So let's go and do that. We're going to
21:35:10say insert into sales order
21:35:15partitioned and we're going to pick
21:35:17values like this. So one and then let's
21:35:20get any dates like 2023 like for example
21:35:24my the mid of the month and the sales
21:35:26could be anything like let's say 100. So
21:35:30let's go and execute this and let's go
21:35:32query our
21:35:34table. So it is this one over
21:35:38here. All right. So now we have one
21:35:40record inside our partition table. And
21:35:42now the big question is in which
21:35:44partition in which data file did SQL
21:35:47store this record. So we have to test
21:35:49whether everything is working fine. So
21:35:51in order to do that I have prepared as
21:35:53well a query. So we are again asking the
21:35:55table partitions with the destination
21:35:57data spaces where we're going to get the
21:35:59number of rows in each partition and
21:36:01then we have the file group and we are
21:36:03focusing on our table orders partitions.
21:36:06So let's go and execute this one. And
21:36:07now we can see very easily we have the
21:36:09four partitions. our new record is
21:36:11inserted in the correct place in 2023
21:36:15file group and in the correct partition.
21:36:17So with that we make sure our function
21:36:20and the whole logic that we have built
21:36:22is working correctly. So now let's go
21:36:24and add more records. I'm just going to
21:36:26go and duplicate it. Record number two.
21:36:28And I'm just going to pick a date in
21:36:312024. And this one going to be like 20.
21:36:35Let's just change the value. So 50.
21:36:38Let's go and execute it.
21:36:40And now we have a second row inside our
21:36:42table. And again the big question is
21:36:45whether it is working. So let's go and
21:36:47execute this again. And now we can see
21:36:49our record is inserted in the partition
21:36:512 in the file group 2024 which is
21:36:55correct. Now let's go and check the
21:36:57boundaries whether it is working
21:36:59correctly. So I'm going to go and here
21:37:01in the third row I'm going to say the
21:37:04last day of 2025. So it's going to be
21:37:07month 12 and the last day. So 20. Let's
21:37:12go and insert it and check our table. So
21:37:15we have a new record. And now let's go
21:37:18and check. My expectation here that this
21:37:21row is going to be inserted in the file
21:37:23group
21:37:242025. So let's go and execute. And that
21:37:27is correct. As you can see the record is
21:37:30inserted in the correct partition. And
21:37:32this is really important to test the
21:37:33boundaries whether they are working
21:37:35correctly because it's a little bit
21:37:36tricky. You have this range left right
21:37:39and boundaries and so on. So you can do
21:37:41it like this to check whether the
21:37:42expectation of your logic is working
21:37:44correctly. And the last one I'm just
21:37:46going to do it very fast. So let's do it
21:37:492026. And I'm going to pick the first
21:37:51day of this
21:37:54year. So let's go and insert it. And now
21:37:57what is the expectation? I think it is
21:37:59pretty simple. So let's go and query.
21:38:01And the first day of this year is
21:38:02inserted in the partition number four.
21:38:04So I can say everything is working
21:38:06correctly. If you get it like this then
21:38:08you have created successfully a
21:38:10partition table and you have prepared
21:38:12all the layers of this partition
21:38:14correctly. I know this is a lot of work
21:38:16but to be honest it is fun because for
21:38:19the first time in database you feel like
21:38:20you are controlling stuff. Usually in
21:38:22database everything like behind the
21:38:24scenes and you don't know exactly where
21:38:26the files are stored of your tables and
21:38:28so on. There is a lot of abstraction in
21:38:30databases but here like we are getting
21:38:33deep in databases and we are controlling
21:38:35and managing all those files which is
21:38:37sometimes it's nice to have this freedom
21:38:39and flexibility. All right one quick
21:38:41thing that I would like to show you that
21:38:42if you go to the database in the
21:38:44explorer then let's go to the storage
21:38:46over here. So let's expand it and here
21:38:48you can find easily informations about
21:38:50the partitions. So over here we can find
21:38:52our partition scheme and as well the
21:38:54partition function that we have created.
21:38:56it is just a quick access instead of
21:38:58like querying the
21:39:02metadata. So now let's have a quick
21:39:04summarize how everything is connected
21:39:06together. So we have a table and then we
21:39:08specify for scale that is connected to a
21:39:11partition scheme and in the partition
21:39:13scheme we have everything connected. It
21:39:15is linked to a specific partition
21:39:17function and there we have the
21:39:19partitions and at the same time it is
21:39:21connected to file groups and the file
21:39:24groups are connected to the data files.
21:39:26So as you can see all those layers and
21:39:28elements are connected together. Now
21:39:30let's see how this works. So we have
21:39:32inserted the last day of 2025 and now
21:39:34the first thing that's going to happen
21:39:36the partition function going to decide
21:39:38to which partition it belongs. So as you
21:39:40can see it is a boundary value and since
21:39:42we have defined it as a lift it going to
21:39:45target the left partition the partition
21:39:47three and then the partition scheme
21:39:49going to connect it to the right file
21:39:51group and in this scenario it's going to
21:39:52be the file group 2025 and we have here
21:39:55only one file so it going to as well go
21:39:57to the correct data file and in this
21:39:59file the SQL going to store this row so
21:40:02it is pretty
21:40:06easy and now we come to very important
21:40:08part where we can understand how the
21:40:10partitions are really improving the
21:40:12performance of my query and of course we
21:40:14can do that by checking the execution
21:40:16plan. So now in order to compare like
21:40:18the behavior with and without the
21:40:20partition what we have to do is to
21:40:22create a mirror table without partition.
21:40:24So we have our table here the
21:40:25partitioned one what I'm just going to
21:40:27do I will go over here and say into and
21:40:29we're going to call it sales
21:40:32orders no partition. So we are taking
21:40:35the data and the structure from the
21:40:38orders partitions and of course it will
21:40:40not be partitioned. So let's go and
21:40:42execute it. Now if you go over here we
21:40:45can see that we have two tables. We have
21:40:47the no partition and the partitioned
21:40:49one. So now what we're going to do we're
21:40:50going to write a query on both tables
21:40:52and then compare the execution plan. So
21:40:54first let's start with the no partition.
21:40:56also from and and now in order to see
21:40:59the effect of the partition what we're
21:41:00going to do we're going to say where
21:41:03order dates equal to and now we're just
21:41:05going to pick a value like 2026 the 1st
21:41:08of January so let's go and query it and
21:41:11we're going to do the same thing a new
21:41:13query but this time for the partitions
21:41:17so now in order to see the execution
21:41:20plan make sure to activate it so we go
21:41:22to the action bar over here and we're
21:41:24going to say include the actual
21:41:25execution plan. So let's click on it and
21:41:28execute. And with that we have here an
21:41:30execution plan. And let's do the same
21:41:32thing for the no partitions. So execute
21:41:35and we have here execution plan. So now
21:41:37let's check what we have in execution
21:41:39plan. We're going to focus on this one
21:41:40over here. So right click on it and then
21:41:42go to properties. And now we can see a
21:41:44lot of details about the execution plan.
21:41:46But what is interesting is the number of
21:41:48rows. So as you can see we are reading
21:41:50four rows. That means the whole table.
21:41:52And of course we have here the CPU and
21:41:54the other costs. Now let's go and check
21:41:56the partition. So let's click over here.
21:41:59So now if you check over here, you can
21:42:00see that the total number of rows is
21:42:03one. So SQL didn't read all four rows.
21:42:06It reads only row and that's because we
21:42:08have in this partition only one row. And
21:42:10as you can see the number of partitions
21:42:12that is used is as well only one. So as
21:42:14you can see using partition we have
21:42:16reduced the number of rows that is
21:42:18retrieved from the files. Now let's go
21:42:20and retrieve like two data from two
21:42:23different partitions and check the
21:42:25execution plan. So let's target 2025 the
21:42:28last day of the year like this. So let's
21:42:32go and execute it. And the same thing
21:42:34for the other
21:42:37query. So let's check the without
21:42:40partition. We still we are reading like
21:42:42four rows. But now if you go to the
21:42:44other one, if you check the execution
21:42:46plan and check the table scan, you can
21:42:49see we are reading only two rows and
21:42:51this time the number of partitions that
21:42:53are involved in this query is two and
21:42:56that's because we have partition for
21:42:572025 and 2026. So as you can see it's
21:43:00worth the efforts. We have optimized our
21:43:02queries and this has a great impact on
21:43:05big tables. The number of resources and
21:43:07the number of reads going to be reduced
21:43:09massively. All right my friends. So
21:43:11that's all about the partitions in SQL.
21:43:13It is amazing and you can use it as well
21:43:14not only in databases but as well in
21:43:17many other data platforms and tools
21:43:19where you always can divide your data in
21:43:21order to optimize the performance. Now
21:43:22in the next step what I have prepared
21:43:24for you after 15 years working in real
21:43:26projects using SQL. I have a lot of best
21:43:29practices and tips for you. So I have
21:43:31collected everything that I know and now
21:43:33I'm going to show you the best practices
21:43:34and tips and tricks that I can give you
21:43:36in order to optimize the performance in
21:43:38SQL. So let's go.
21:43:44And now before we deep dive into the 30
21:43:47best practices, I'm going to give you
21:43:49the golden rule. The SQL optimizer
21:43:51responds differently for different sizes
21:43:53of tables. So that means if you have
21:43:55small and medium tables like hundred of
21:43:57thousands, you might not notice any
21:43:59performance differences if you are
21:44:00following the best practices. And that's
21:44:02because the size of the data is small.
21:44:04But if you have like million or hundred
21:44:06of millions of records in tables, you
21:44:08will immediately notice how things can
21:44:10be faster if you follow the best
21:44:12practices. And here is my golden rule.
21:44:14If you get any best practice from me or
21:44:17let's say you are reading something in
21:44:18the internet, always you have to test
21:44:20using the execution plan. So for
21:44:22example, if you have like two queries
21:44:24are returning the same result of the
21:44:25data, I'm going to recommend you here to
21:44:27check the execution plan. And if you
21:44:29notice there is no differences between
21:44:31them in the execution plan then pick the
21:44:34one that you see it is easier to read
21:44:36and to understand because sometimes if
21:44:38you are following the best practices for
21:44:40the performance your query might be like
21:44:42little bit more complicated. So always
21:44:44write the query to be understandable and
21:44:47only optimize it if you notice it is
21:44:49slow. So the golden rule here is always
21:44:52test. If you find you are optimizing the
21:44:54performance with the new query then pick
21:44:56that and if there is no gain in the
21:44:58performance then focus on making your
21:45:00queries readable. So this is the golden
21:45:03rule always test test test using
21:45:05execution plan. So let's deep dive into
21:45:07best practices and we're going to start
21:45:09by optimizing the performance of our
21:45:15queries. All right let's start with the
21:45:17easy stuff. The first step is select
21:45:19only what you need. What I usually see
21:45:21in many queries is that the developers
21:45:23just go and select all the columns from
21:45:25one table and I can tell you I cannot
21:45:27think of one scenario where you need all
21:45:28the columns of one table in one query.
21:45:31So for sure in the result we will get
21:45:33like unnecessary columns and of course
21:45:35reading unnecessary informations going
21:45:37to make your query slower. So this is
21:45:40usually a bad practice. Don't use select
21:45:42star but instead of that go list all the
21:45:44columns that you need for your query. So
21:45:47make sure that you only select what you
21:45:49need. Don't go and select all the
21:45:50columns from one table and with that you
21:45:52don't risk reading unnecessary
21:45:54informations from the database. So
21:45:56always make sure that you select exactly
21:45:58what you need for a query don't go with
21:46:00a star. Okay. Tip number two avoid
21:46:03unnecessary distinct and order by. I
21:46:05have noticed that many developers as
21:46:07they are writing a lot of queries they
21:46:09tend by default adding always distinct
21:46:12and order by for each query. And as we
21:46:15review the code and discuss it with the
21:46:16developer, we see that we really don't
21:46:19need to remove any duplicates in the
21:46:21query because there are no duplicates
21:46:22and it was only a habit to remove the
21:46:25duplicates using distincts. And the same
21:46:26thing for the order by in many
21:46:28situations there is no need to sort the
21:46:30data at all. And those operations, the
21:46:32distinct removing the duplicate and
21:46:34sorting the data, they are very
21:46:36expensive operations in your execution
21:46:38plan. So they're going to take a lot of
21:46:39resources and slow down your query. So
21:46:42this considered as a bad practice if you
21:46:44always go and use distinct even though
21:46:46it's not needed or you are using the
21:46:48order by in order to sort the data when
21:46:50it is not necessary. So the best
21:46:52practice here is to avoid them. Don't
21:46:54use distinct or order by only if it is
21:46:56necessary. Okay. The next one for
21:46:59exploration purposes limit the rows. So
21:47:02sometimes especially if you are working
21:47:03with a new database you would like to
21:47:05explore the tables just to have a quick
21:47:07peek in order to see the content of the
21:47:09tables. And if your database has a lot
21:47:11of big tables with millions of rows and
21:47:13so on, you will be consuming a lot of
21:47:15resources. If you just select the data
21:47:17like this. So now imagine that the
21:47:19orders has like 100 million. As you run
21:47:21this query, the database has to fetch
21:47:24all the 100 million for you. And usually
21:47:26for exploration, it's enough to see like
21:47:2810 rows and that's going to be enough.
21:47:30That's why it is considered as a bad
21:47:32practice if you are exploring the tables
21:47:34to not have a limit or top. So a good
21:47:37practice would be to say select top 10
21:47:39and then have the same query. So if you
21:47:41go over here you will get only 10 rows
21:47:44and the database will not fetch 100
21:47:46million. It can fetch only 10 rows. And
21:47:48now if you are exploring a lot of tables
21:47:51you will not consume a lot of resource
21:47:53from the database. So if you are
21:47:55exploring always limit the number of
21:47:57rows that you are
21:48:01retrieving. All right. Right. So now
21:48:03we're going to talk about how to
21:48:04optimize the filtering in SQL. So the
21:48:07tip here is to create an uncclustered
21:48:09index on frequently used columns in wear
21:48:12clause. So now of course you have to
21:48:13check your queries and so on. And if you
21:48:15see that you are frequently filtering
21:48:17the data using the order status then it
21:48:20makes sense to create a non-clustered
21:48:22index for this column in order to
21:48:24improve the performance of your query.
21:48:26So for this situation I'm going to go
21:48:27and create then a nonclustered index for
21:48:30the table sales order for the order
21:48:32status. So once you create it then you
21:48:36improving now the performance of your
21:48:38query. Okay. The next one is avoid
21:48:40applying functions to columns in the
21:48:42works. So in many cases what we usually
21:48:45do is that we go and transform the
21:48:47columns before like filtering the data.
21:48:49Like for example here I'm applying the
21:48:51function lower on the order status
21:48:53because I'm searching for the value
21:48:55delivered and I'm not sure about the
21:48:56values in the table whether they have
21:48:58like a camel case or uppercase or
21:49:01anything but in order to make sure that
21:49:03I'm going to find the value I'm going to
21:49:05go and say lower the order status and
21:49:07then give here a lower value and of
21:49:10course it's going to work. So if we go
21:49:11and search for it and as you can see we
21:49:13have here the status delivered and the
21:49:15value is different than the one I used
21:49:16because here we have like a capital
21:49:18first character but here we have a
21:49:20problem we have an index on the order
21:49:22status and now if you use any functions
21:49:24like for example here the lower the SQL
21:49:27will not use the index so that means the
21:49:30whole index is now useless and the SQL
21:49:33is not using it and that's why we
21:49:35consider it as a bad practice to use
21:49:37functions for the wear clause and
21:49:40Instead of that the good practice is
21:49:42that to not use any function and to
21:49:44write exactly the value that is used
21:49:46inside your data and with that the SQL
21:49:48going to be happy and use the index that
21:49:51you have created. Okay, let's have
21:49:52another example about this rule and here
21:49:54we are selecting all the customers where
21:49:56the first name start with the A. So with
21:49:58that we can go and use the function
21:50:00substring in order to get the first
21:50:03character of the first name and once you
21:50:05match it with a then you will get the
21:50:07result and here we have Anna. And this
21:50:08is again bad if you have an index on the
21:50:10first name and that's because we are
21:50:12applying a function on the column. So
21:50:15this considered to be a bad practice and
21:50:17instead of that we can go and use the
21:50:18help of the like. So we can go and
21:50:21search for this pattern where it start
21:50:22with the A and then we have a white
21:50:24card. We don't care about the rest. So
21:50:26it must start with a. So if you go and
21:50:28execute it you will get the same
21:50:30results. So try as much as you can to
21:50:32avoid the functions in the wear clouds
21:50:34in order to hit and get the index
21:50:37working. And in many scenarios, we have
21:50:38a workaround in order to use the
21:50:40function without transformations. So try
21:50:43your best to avoid using functions if
21:50:45your columns having an index. All right,
21:50:47one more example that you see a lot on
21:50:49queries that you filter by the year. So
21:50:52we are searching for the orders that
21:50:54happens in 2025 and we usually go and
21:50:56use the year order dates. And now if you
21:50:59have an index on the order dates, this
21:51:01again will not be working because you
21:51:03are using a function year. So this
21:51:05considered to be a bad practice. Instead
21:51:07of using the year function, you can go
21:51:09and use between. So we don't apply a
21:51:12function on the order date and we say
21:51:13the order date is between the boundaries
21:51:16of the year. Of course, now our query is
21:51:18not looking really cool and easy like
21:51:20the first one. But still with the second
21:51:22one, we are hitting the index. So again
21:51:25while you are filtering, try to not use
21:51:27functions on the columns because it is
21:51:29really waste if you have an index and
21:51:31you are not using it. and most of the
21:51:33cases you have like a workound for your
21:51:35function. So those are the three
21:51:36examples that I wanted to show you about
21:51:38this tip. All right, moving on to a
21:51:40similar one. It says avoid leading wild
21:51:42cards as they prevent index usage. So
21:51:45this is a similar one. Let's say for
21:51:47example I'm searching for the word gold
21:51:49inside the last name. And here we have
21:51:51to be careful what we are searching for.
21:51:53Should the gold exist somewhere in the
21:51:56last name or only we are searching for
21:51:58the last name that start with gold? If
21:52:01it's like that we are searching only the
21:52:03last name that starts with gold then we
21:52:05are doing it here wrong. And in SQL if
21:52:07you're using the leading wild card then
21:52:10the SQL will not be using the index. But
21:52:12if you are using the wild card at the
21:52:14end and the trailing this one is fine
21:52:16and will not avoid using the index. So
21:52:19this considered as a bad practice
21:52:21because you will not be hitting the
21:52:23index. Better than that to not use the
21:52:25white card as a leading and if that's
21:52:27enough for your search then with that
21:52:29you are hitting and using the index.
21:52:32Okay, moving on to the next one. It says
21:52:34use in instead of multiple or or
21:52:37operator is very evil for performance
21:52:40and try to avoid using it. It really
21:52:42kills your performance whether it is in
21:52:44the filters or joins and so on. So now
21:52:46we want to show the orders where the
21:52:47customers is equal to one or two or
21:52:50three. And of course this is considered
21:52:52to be bad practice and hard to read and
21:52:54so on. Please don't do that. Instead we
21:52:56have the in operator and we are saying
21:52:58if the customer is one of those values
21:53:01then show the orders. So if you go and
21:53:03run it you will get the exact results
21:53:06and it's not only looks nicer than the
21:53:08first query but it has as well a better
21:53:11performance. So if you find out writing
21:53:13a lot of ors think about the inoperator.
21:53:16So those are the best practices for
21:53:17filtering data to improve the
21:53:19performance.
21:53:24Okay, so now we're going to focus on how
21:53:26to optimize joining tables in SQL. So
21:53:28the first tip here is to understand the
21:53:31speed of joins and to use inner join
21:53:34when it's possible. Well, as we learned
21:53:35before, we have like different types of
21:53:37joins. We have the inner, left, right,
21:53:39and outer join. And if we talk about the
21:53:41performance, the best performance you
21:53:43will get from the inner join. And that's
21:53:45because SQL going to work only on the
21:53:47matching rows. That means the effort and
21:53:49the processing time is better than the
21:53:52other joins. Now in the next one in
21:53:54ranking we have the left and right
21:53:56joins. They are slightly slower than the
21:53:58inner join because usually they process
21:54:01more data and more rows than the inner
21:54:03join because SQL will work not only with
21:54:06the matching rows as well with the
21:54:08unmatching rows. So for right and left
21:54:10SQL has to do more stuff than the inner
21:54:12join. And now the worst type of joins we
21:54:15have the outer join. And that and that's
21:54:17because this type works with the biggest
21:54:20number of rows compared to the other
21:54:21types. It's going to present unmatching
21:54:23rows from the left and from the right
21:54:25tables. So that means SQL has a lot of
21:54:28to-do and that's why this join has the
21:54:30worst performance. So here my advice is
21:54:33always try to use the inner join if it's
21:54:36enough to work with the matching rows
21:54:38and if the matching rows is not enough
21:54:40then go with the lift join maybe. But
21:54:42try your best always to bring the inner
21:54:44join instead of lift join. But don't
21:54:46forget inner join filters the data.
21:54:49Okay. The next one it says use explicit
21:54:51join the unzi join instead of implicit
21:54:55join. Well it is considered as a bad
21:54:57practice if you join tables like this
21:54:59the implicit join or the nonzi join.
21:55:02It's better to use the normal modern
21:55:04join where you use the inner join for
21:55:06example. about the performance. There is
21:55:08like no differences between them. And
21:55:10for this scenario, it's very simple. But
21:55:12if you have like a complex query, then
21:55:14joining table like this might be very
21:55:16confusing and really hard to read and as
21:55:19well complex to optimize. That's why the
21:55:21best practice says go with the normal
21:55:23inner join. So go with the anzi join
21:55:26instead of the nonzi join. Okay. To the
21:55:29next tip. Make sure to index the columns
21:55:31used in the on clause. So we have to go
21:55:34and make sure that both of those columns
21:55:36has an index because indexes speed up
21:55:39the lookup process. Without an index,
21:55:41the SQL might go and do a full table
21:55:44scan. Without an index on those columns,
21:55:46the database might go and scan the
21:55:48entire tables in order to find a match.
21:55:51And that is really slow if you have big
21:55:53tables. So now if you go to the
21:55:54customers over here and then to the
21:55:56indexes, we can see that we have an
21:55:59index, a clustered index for the
21:56:01customer ID. But if you check the
21:56:03customer ID in the orders, we don't have
21:56:06an index for that. So this one doesn't
21:56:08have an index. So in order to fix that,
21:56:10we're going to go and create an
21:56:11uncclustered index on the table orders
21:56:14for the customer's ID since it is a
21:56:16foreign key. So once we do that, we have
21:56:19now an index for both of those columns
21:56:21and with that our join going to be
21:56:23faster. Okay. So now we come to a tip
21:56:25where we say really it depends on there
21:56:28is like not one clear way on how to do
21:56:30it. But let's say if you have a big
21:56:32tables, it is better to filter data
21:56:34before joining. And here we have like
21:56:36three different scenarios that going to
21:56:38deliver the same results. But of course
21:56:40the question is which one is the best
21:56:41for performance. So now let's have a
21:56:43look to them. What we are doing here we
21:56:45are just joining two tables and then we
21:56:47are filtering the result based on the
21:56:49order status that comes from the orders.
21:56:52So in the first query what we are doing
21:56:53we are first joining tables and at the
21:56:56ends we are using where clause in order
21:56:58to filter the data. So by looking to
21:57:00this we are just filtering the data
21:57:01after joining the tables. But there is
21:57:03another way on how to do it. You can go
21:57:05and join the tables but on the join
21:57:08condition you can go and add this order
21:57:11status equals to delivered. So we are
21:57:13matching the data by the customer ID and
21:57:15at the same time we are filtering the
21:57:17data by the order status since we are
21:57:20using the inner join. So the filtering
21:57:22is happening during the join or you can
21:57:24do it like this where we have here more
21:57:26stuff to be added where we don't join
21:57:28the table directly with the orders. We
21:57:30first prepare the table orders before
21:57:33joining it with the customers. And here
21:57:35our preparation is we are just selecting
21:57:37the columns that we need and we are
21:57:39already filtering the data before doing
21:57:42the join using the subquery. But if you
21:57:44run all those queries you will get the
21:57:46exact same results. And of course there
21:57:48is another way on how to do it. you can
21:57:50go and prepare the data not in subquery
21:57:53you can go and use a CTE and then join
21:57:55the result of the CTE with the table
21:57:57customers. So now about the performance
21:57:59if your query is like small not that
21:58:02complex and as well you don't have a big
21:58:04data inside your tables all those three
21:58:07queries going to deliver the same
21:58:09performance. I know it might sounds
21:58:11weird because here we are like filtering
21:58:13after joining or here we are filtering
21:58:15during the join. Normally in databases
21:58:18the SQL optimizers are now very smart
21:58:20can understand that there is a filter
21:58:22here and decide on the best execution
21:58:25plan for you. So actually wherever you
21:58:27put your filter after, during or before
21:58:30the SQL is smart enough to do it
21:58:32correctly. So if you don't have complex
21:58:34query and you don't have like big
21:58:36tables, go with the one that suits you.
21:58:38And I really recommend you to go with
21:58:40the first one because it's logical and
21:58:42easier to understand. But if you have
21:58:44big tables and complex queries, the best
21:58:46practices says try always to prepare the
21:58:49data before joining it. So try to
21:58:51isolate and abstract the pre-step in a
21:58:54subquery or in a CTE before joining it
21:58:57with any other tables. And in many
21:58:59scenarios in my project where I have a
21:59:01big table, this did help where the
21:59:03execution plan was better if I isolate
21:59:05and prepare the data before joining it.
21:59:07So if you have small or medium tables,
21:59:10go with the normal way, use the wear
21:59:11clause. But if you have complex big
21:59:13tables, prepare the data in subquery or
21:59:16CTE and then join it with the tables.
21:59:19Okay. And now moving on to tip number
21:59:2112. It is similar to the previous one
21:59:24but this time it says aggregate data
21:59:26before joining tables and again it is
21:59:28special case to improve the performance
21:59:30of big tables. So now we have the
21:59:32following scenario where we are joining
21:59:34the orders and the customers and we are
21:59:36aggregating the data by the customer ID
21:59:39but we are just joining the table
21:59:40customers because we need the first
21:59:42name. So as a result we have the
21:59:44customer ID, the first name and the
21:59:46order count. So the standard way is to
21:59:48join the tables and then do a group by
21:59:51in order to summarize the data. Now if
21:59:53you look to this query, we actually
21:59:54don't need the join in order to do the
21:59:56aggregations. We can do first the
21:59:59aggregation like preparing the orders
22:00:01with the aggregated data and then join
22:00:04the result with the customers in order
22:00:06to get the first name. So again we
22:00:09prepare first and then we do the join
22:00:11and we can do that using either the
22:00:13subqueries or using the CTE. So in this
22:00:16scenario first we are doing the group by
22:00:18we are aggregating the data and the
22:00:20result of this is joined with the
22:00:22customers tables in order to get the
22:00:23first name. Now of course there are like
22:00:25many ways on how to do it like for
22:00:26example as well using the correlated
22:00:28queries where we can go and use the
22:00:31subquery in the select statements and
22:00:33then use the where condition over here
22:00:35to make the correlated query. Now all
22:00:37those three going to deliver the same
22:00:39results but the question here again
22:00:41which one has the best performance?
22:00:43Well, I can go immediately and tell you
22:00:45that correlated subqueries are the worst
22:00:48one. Always avoid using correlated
22:00:51subqueries. They has really bad
22:00:52performance. And that's because SQL
22:00:55going to go and do the aggregations for
22:00:57each customer individually. So it's
22:01:00going to go like for each row and doing
22:01:01aggregation then to the next row and so
22:01:03on. So it takes long time. So this is
22:01:06bad practices. Don't use it. Now we are
22:01:08left again with the first option and the
22:01:11second option. And here my tip going to
22:01:13be like the previous one. I'm going to
22:01:14say if you have small to medium size of
22:01:17tables then go with this one because it
22:01:19is easier to read and to understand and
22:01:22you will gain exactly the same
22:01:23performance as this subquery. But if
22:01:26your tables are big the best practices
22:01:28is to prepare first the data to group up
22:01:31the data to filter the data and to
22:01:33isolate it in a subquery or a CTE before
22:01:36joining it with the final table in the
22:01:39final query. But again here only for big
22:01:41tables and always test check the
22:01:43execution plan whether you are really
22:01:45getting any benefits from it. All right.
22:01:47So if you have big tables try to prepare
22:01:49the data first in city subquery and then
22:01:51join. Okay moving on to the next tip. It
22:01:54says use union instead of or operator in
22:01:57joins. So what this means sometime let's
22:02:00say that you are joining two tables the
22:02:02customers and the orders. And now about
22:02:04the join key, you can see over here it
22:02:06says the customer ID should be equal to
22:02:08the customer ID from the orders or the
22:02:11customer ID should be equal to the
22:02:12saleserson's ID. If one of these two
22:02:15conditions is fulfilled, then we have a
22:02:17match. And I can tell you the or
22:02:19operator over here is a performance
22:02:21killer. It has really bad performance.
22:02:24So try to avoid it. Don't use ore in the
22:02:26joins. It has a lot of problems like it
22:02:29avoid indexes, it create like loop joins
22:02:32and so on. That's why we consider it as
22:02:35a bad practice. And now in order to get
22:02:37the same results, we can go and split
22:02:40the joins. So we can go and have two
22:02:42queries. The first query is joining the
22:02:44data based on the customer ID and the
22:02:47second query based on the saleserson and
22:02:50then we go and merge those two results
22:02:52using the union. It sounds like bigger
22:02:54and too much for the SQL but with this
22:02:57you will get better performance than
22:02:59using this simple or operator. So again
22:03:02if you have big tables try to avoid
22:03:04using or and instead of that go and use
22:03:07union. Okay the next tip says check for
22:03:10nested loops and use SQL hints. Now
22:03:13imagine that we have like big tables and
22:03:15we are joining tables. So now if you are
22:03:17checking the execution plan you have to
22:03:19check always the join type. So for
22:03:21example here it is using the nested
22:03:23loops which is of course is okay because
22:03:26we have small tables but if you have big
22:03:28tables and still SQL is using for some
22:03:30reason the nested loops then this is
22:03:33alerting. So in order to change this
22:03:35what we can do we can go and use the SQL
22:03:37hints in order to force SQL to use the
22:03:40hash join. Hash join is really good if
22:03:42you have a big table like for example
22:03:44the orders that is joins with a small
22:03:47table like the customers. So now what we
22:03:49can do at the end we can write over here
22:03:50option hash join. So let's go and
22:03:53execute it and let's check the execution
22:03:55plan and with that we have forced SQL to
22:03:58use the hash join or hash match. Again
22:04:01you have here really to evaluate your
22:04:02tables. If you have like small tables
22:04:05don't bother with that. But if you have
22:04:07big tables and SQL still doing the
22:04:09nested loops, nested loops are usually
22:04:11very slow because you have a lot of
22:04:13iterations and so on and with the hash
22:04:15join that small table going to be stored
22:04:17in the memory and then you have really a
22:04:19quick matching between the two tables.
22:04:22So those are all the best practices and
22:04:24tips on how to optimize joining tables
22:04:26in SQL. All right, so now we're going to
22:04:28talk about union and here is the best
22:04:29practices. It says use union all instead
22:04:32of using union if duplicates are
22:04:35acceptable. So it's very simple. If the
22:04:37duplicates are acceptable or let's say
22:04:39that there is no duplicates then don't
22:04:42go with the union because it needs more
22:04:44time to be executed. SQL has to go and
22:04:46check row by row whether we have
22:04:48duplicates or not and this usually takes
22:04:51longer time than using the union all. So
22:04:54if duplicates are acceptable or you
22:04:55don't have any duplicates in your data
22:04:57go with the union all just have to go
22:05:00and merge all the data without checking
22:05:02anything and the performance going to be
22:05:04faster. All right, the next one is
22:05:05little bit tricky. So it says use union
22:05:08all together with the distinct instead
22:05:10of using union if the duplicates are not
22:05:13acceptable. So you want to remove the
22:05:15duplicates. So we have learned that in
22:05:17order to do that we're going to go and
22:05:18use the union. It's going to go and
22:05:20merge the data and as well remove the
22:05:22duplicates which is really okay to use
22:05:24it if you have like smaller data or
22:05:26medium. But let's say that you have like
22:05:28millions of row which is really okay if
22:05:30you have like medium and small tables.
22:05:32But again here if you have huge tables
22:05:34big tables hundreds of millions the best
22:05:37practice says go with the union all and
22:05:40afterwards use a distincts. So in the
22:05:43sub query we are using union all but in
22:05:46order to remove the duplicates we use
22:05:48the distincts. But again here you have
22:05:50to test it to check the execution plan.
22:05:52If you are getting benefit then go with
22:05:54this version. But if your data is not
22:05:56really big you have hundred of
22:05:57thousands. So go just with the normal
22:06:00union. the code is smaller and you will
22:06:02get the same effects but only for large
22:06:04tables you can go with this best
22:06:06practice. So that's all what I have for
22:06:08you for the
22:06:09[Music]
22:06:12union. Okay. So now let's talk about
22:06:14aggregations and here the tip says use
22:06:17column store index for aggregations on
22:06:19large tables like for example fact
22:06:21tables and that's because column store
22:06:23index going to compress the data. So the
22:06:25size of the data going to be smaller and
22:06:27as well the aggregation is super fast
22:06:30because we are selecting only the
22:06:32relevant informations only the relevant
22:06:34columns. So it makes it a perfect setup
22:06:37for aggregating large tables. And now
22:06:39let's say that we have hundreds of
22:06:40millions of orders and we have this
22:06:42query over here. So the best practice
22:06:44says convert this table to a clustered
22:06:47column store index. So if you go and
22:06:49create this clustered index over here,
22:06:52the whole table going to have amazing
22:06:54performance for aggregations like this.
22:06:56All right. So to the next one, it says
22:06:59pre-agregate data and store it in a new
22:07:02table for reporting. So let's say that
22:07:04we have like a big query where we are
22:07:06aggregating the data and so on. And this
22:07:08query takes really long time. Let's say
22:07:10like 5 minutes or something like that.
22:07:12But now the problem with that I would
22:07:14like to show the results as a report
22:07:16maybe to my manager or let's say during
22:07:18a meeting it's going to be really bad if
22:07:20everyone have to wait until the query is
22:07:23done. So the best practice here if you
22:07:25have like a query that runs very slow
22:07:28what you can do you can go and store the
22:07:30results in a table. So if I go over here
22:07:33and say into sales summary what going to
22:07:36happen going to store the result inside
22:07:39this table. So let's go and execute it.
22:07:41And now with that we have a nice table
22:07:43where everything is prepared. So all
22:07:44that you have to do is to go and query
22:07:47this table. And of course it's going to
22:07:48be very fast because it's only select
22:07:50statements. And with that you have like
22:07:52prepared and pre-agregated the data to
22:07:55have like fast reports. So don't forget
22:07:58about this. If you have a big query you
22:08:00can insert the result of this query in a
22:08:02new table in order later to use it for
22:08:04reporting. But one thing that you have
22:08:06to make sure that you have always to
22:08:08update this table. So if we have new
22:08:10orders, it will not be presented inside
22:08:12the sales summary. You have to go and
22:08:15run this query again in order to get new
22:08:17data inside the sales summary. So those
22:08:19are the tips on how to improve the
22:08:21performance of your aggregations in
22:08:27SQL. So now what is happening here? I
22:08:29would like to show the orders but only
22:08:31from customers from USA. So if you check
22:08:34this query over here, we are joining the
22:08:36tables order and customers but mainly we
22:08:39are showing only the orders information
22:08:41and that means we are using the
22:08:43customers only to filter the table
22:08:45orders and there are like multiple ways
22:08:47on how to do this task. So it's not only
22:08:50the joins you can go and use the exist
22:08:52as a subquery and as well you can go and
22:08:55use the in operator in the subquery. And
22:08:58now comes the old but gold question.
22:09:02Which one is better? Should we join or
22:09:04use exist or in? And oh my god, if you
22:09:07go to the forums, you will see people
22:09:09fighting about which one is the best.
22:09:14Clean tech. Come on, do that again. Do
22:09:16that again. I dare you.
22:09:19Okay,
22:09:21bring it. Oh, you can't say you can't
22:09:24say one point. Two point. Now, about the
22:09:27best practices, everyone agrees that's
22:09:30don't go and use the in operator. So
22:09:32this is the bad practice. So
22:09:36bad practice avoid it. Don't use it. And
22:09:39of course I'm always speaking about big
22:09:41tables, okay? Not small tables. So we
22:09:44don't go and use this in order to filter
22:09:47one table based on the result of another
22:09:49table. So don't use any operator in this
22:09:52scenario. Now here comes the conflicts.
22:09:54We have join and exist. Well, about the
22:09:57performance of those two, they are very
22:09:59similar for medium tables. like I'm
22:10:01speaking about hundred or thousand and
22:10:03so on. But still you have to test it.
22:10:05You have to go and compare the execution
22:10:07plans and if you are getting like
22:10:09identical results and both of them are
22:10:12having the same speed then I prefer to
22:10:15go with the join and that's because to
22:10:17be honest it is easier to write than
22:10:19writing that exists. So I'm going to say
22:10:21from my point of view this is best
22:10:24practice if the
22:10:27performance
22:10:29equal to exist. But now what happens for
22:10:32me is that sometimes I get better
22:10:34performance using exists. So I'm going
22:10:37to say from my point of view the best
22:10:39practice
22:10:40here. And now you might ask why we are
22:10:43getting with the exist better
22:10:44performance than in the inner join. And
22:10:46that's because SSQL has only to check
22:10:48the existence of data from the subquery.
22:10:51But in the other hand with the inner
22:10:52join SQL has to go and start doing
22:10:55matching between two tables. So it can
22:10:57go and evaluate all matching records and
22:11:00so on. It is not evaluating whether it
22:11:03exist or not. And as well sometimes SQL
22:11:05has to deal with more rows because you
22:11:08might introduce duplicates as you are
22:11:10joining tables. And this will not happen
22:11:12using exists. So for some scenarios if
22:11:14you are using exist you might get better
22:11:17performance than using join but everyone
22:11:20agrees to not use the end operator. Okay
22:11:22the next tip is to avoid redundant logic
22:11:25in your query. This happens a lot if you
22:11:28have a lot of sub queries and if you
22:11:30analyze it you might find sometimes
22:11:32there is like redundancy. So for example
22:11:34this query I would like to have like a
22:11:36tag for each employee whether the salary
22:11:39is above the average or below the
22:11:40average. So now we might do it like
22:11:42this. we say okay let's get the data for
22:11:44employees where the salary is higher
22:11:46than the average and you go and
22:11:48calculate the average in a subquery. So
22:11:50if it's higher then you write here above
22:11:53average and now we say okay let's go for
22:11:55the below average. So we do a union all
22:11:57and the condition going to be salary is
22:11:59less than the average. And now by
22:12:01checking this you see that there's a
22:12:03problem. First of all we are querying
22:12:06the employees like four times. We have 1
22:12:082 3 4. So we are scanning the table
22:12:12employees four times and as well we have
22:12:14the same logic over here. So we are
22:12:16calculating the average of salary at
22:12:18twice. So this is of course I can say a
22:12:21bad practice and there is like many ways
22:12:23on how to do it better than that. For
22:12:25example, you can go and put this
22:12:26subquery in CTE and then use it multiple
22:12:29times. But there is like better solution
22:12:31using the window function. So if you
22:12:33check this, it is very simple. Let's me
22:12:34execute it. We are reading the table
22:12:36employees only once and then we are
22:12:39using the case statements. If the salary
22:12:41is higher than the window function. So
22:12:43we are calculating the average on top of
22:12:45the whole table employees. If it's
22:12:48higher then write above average. If it's
22:12:50lower then below average. So as you can
22:12:52see it is easier to read and it is
22:12:54smaller and the performance here is way
22:12:56better than reading four times the
22:12:58employees and repeating the same logic.
22:13:00So here you have always to look to your
22:13:02queries and if you see that you are
22:13:04repeating the same things over and over
22:13:06then you are writing a bad query. Think
22:13:08about alternatives like CTE window
22:13:11functions and I'm sure you will find a
22:13:12better way than reading the table
22:13:15several times or repeating the same
22:13:17logic several times. So as you can see
22:13:18optimizing the queries is not always
22:13:20about using indexes and partitions. It's
22:13:23all about using best practices. All
22:13:25right guys, so with that we have covered
22:13:27a lot of best practices on how to
22:13:29optimize the performance of your query.
22:13:31And as you can see it's not always
22:13:33creating indexes, right? In many
22:13:34scenarios it's about how you write the
22:13:40query. And now in the next section I'm
22:13:42going to show you the best practices on
22:13:45how to create tables. So the best
22:13:47practices of DDL data definition
22:13:50language. If you have a poor definition
22:13:52of your tables, this has a great impact
22:13:54on the performance of your queries. All
22:13:56right. So now we have here like a DDL in
22:13:58order to create a table customer info
22:14:01and it is not really following best
22:14:02practices. So let's go through it one by
22:14:05one. The first tip is try to avoid the
22:14:07data types varchar and text if it's
22:14:10possible. The vchart and text they are
22:14:13like one of the worst data types for
22:14:15performance because they consume a lot
22:14:17of resources whatever you do like for
22:14:19example if you are sorting the data by a
22:14:21column that is var or text it is very
22:14:23expensive operation the same thing if
22:14:25you go like and create an index on top
22:14:27of such a column it's going to be as
22:14:28well expensive and they cause a lot of
22:14:30problems with the data fragmentations
22:14:32and many issues. So try as much as you
22:14:35can to skip those data type if it's
22:14:37possible. So now let's go and review all
22:14:39those columns in order to see whether we
22:14:40can change something about it because it
22:14:43has a lot of bar charts. So the first
22:14:44one over here we have is var because it
22:14:46is the first name. Well, it is okay. Now
22:14:49moving on to the next one. We have the
22:14:50last name as a text which is not really
22:14:52good because text is worse than vchar.
22:14:55So it's better to use var than a text.
22:14:57So here we have to fix it. So var and
22:15:00I'm going to go with the links 50. Now
22:15:02moving on to the countries. So the
22:15:04country is going to be vartar. We cannot
22:15:06change that. that contain characters. So
22:15:08the next one is the score of the
22:15:10customer. H here we can do something
22:15:12about it because scores are only
22:15:14numbers. So that's why we can go and
22:15:16skip this one. So let's remove it and
22:15:19say you are integer and with that we
22:15:21have avoided using the varchar. And the
22:15:23same thing goes for the birthday. The
22:15:25birthday is a date and here we have it
22:15:27as a vchar. Well this is not really good
22:15:29and we can skip that by having this
22:15:31column as a date. So date is way better
22:15:34than having a vchar. All right. And the
22:15:36next one is integer. So with that we
22:15:38have fixed few stuff. So we have fixed
22:15:40the score and the birthday. And with
22:15:42that we have saved some storage. If we
22:15:44have an index on the score it's going to
22:15:45be way better than having a var. And if
22:15:48you are filtering the data based on the
22:15:50birthday it's going to be faster. So
22:15:51again try your best to avoid the vchar
22:15:54and the text. I have seen in many
22:15:55projects that a lot of developers tend
22:15:57to use the vchar and I understand it is
22:16:00easier to make everything as a vchar
22:16:02than deciding whether it is an integer,
22:16:04date, float and so on because you can
22:16:06fit everything in the vchar and text but
22:16:09this is lazy. Take time to understand
22:16:11the content of this column and try to
22:16:13assign it to the correct data type
22:16:15because this has really impact on the
22:16:17performance. Okay, to the next one it
22:16:19says avoid using max or overly large
22:16:22lengths. So now we have to keep our eyes
22:16:24on the links of each data type
22:16:26especially the bar charts. Not only it
22:16:28going to waste like a lot of storage.
22:16:30It's also going to like mislead the SQL
22:16:33by creating large indexes which is
22:16:36totally unnecessary because the data
22:16:38itself is small but because you have
22:16:40defined like a large length SQL going to
22:16:43check those informations and make
22:16:44decision to make a big index and large
22:16:47indexes are always problematic because
22:16:49they're going to slow everything down by
22:16:50sorting the data by retrieving data by
22:16:53updating the index. So it is really bad
22:16:55practices if you go blindly and define
22:16:58everywhere max or
22:17:00255. Again give it a chance to think
22:17:03about each column and predict a length
22:17:05for it. So for example if you check over
22:17:07here we are saying first name v chart
22:17:09max. Well most of the first names are
22:17:11short. So we don't need like the maximum
22:17:14size of a v chart to fit a first name.
22:17:17So here we can go easily instead of max
22:17:19with the 50. And the same thing goes for
22:17:22the column country. We don't need 255
22:17:24characters for the country name. We can
22:17:27go with something more realistic like
22:17:29around 50. I think you can even go
22:17:31smaller, but it's fine to have 50. So,
22:17:33the best practice here is to analyze
22:17:35your data and to predict the size of
22:17:38each column. And don't be lazy by just
22:17:40defining max everywhere. I know it's
22:17:42faster, but it's bad for performance.
22:17:44Okay. What do you have else? Use the
22:17:47constraint nutnull as much as possible.
22:17:50The nutnull is amazing. It has a lot of
22:17:52advantages. Of course, the biggest
22:17:53advantage is that's the data integrity
22:17:55of your table. So with that, you make
22:17:57sure no nulls are inserted in specific
22:18:00column. But it is as well good practices
22:18:02to use it for improving the performance
22:18:05because if you are creating an index,
22:18:06you're going to get a better index
22:18:08performance since SQL knows there is no
22:18:10nulls inside my tree inside the index.
22:18:13And in the other side, if you are
22:18:14writing query, we tend to use a filter
22:18:17where we say a specific column should
22:18:19not be null. But if you make sure that
22:18:21in the DDL it is not null then you can
22:18:23skip this filter and with that you are
22:18:25reducing the size of your query. So what
22:18:27we're going to do we're going to go
22:18:28through all the columns and decide
22:18:29whether it is not null and null. So for
22:18:31example the first name and the last name
22:18:33they should not be null. So that's why
22:18:35I'm going to say not null and the same
22:18:37thing for the last name not null. For
22:18:39the customer ID we're going to talk
22:18:40about it soon because we're going to
22:18:41convert it to primary key and primary
22:18:43keys are usually not null. So now for
22:18:45the country we make have it in the
22:18:47business that it should not be null. So
22:18:49we go and make a constraint about it.
22:18:51Now about the total purchases and
22:18:53scores. If it is new customer, maybe we
22:18:55can have a null inside our data. So
22:18:57we're going to leave it empty. And I
22:18:58think birthday is going to be usually
22:19:00optional. So we're going to leave it as
22:19:01well. And whether the customer is
22:19:03employee or not. This could be as well a
22:19:06null. So with that we have found out
22:19:08like three columns where we can have a
22:19:10constraint about the not null. And if we
22:19:11go and create like an index on the
22:19:14country, it's going to be a better
22:19:15index. Okay. Moving on to the next one.
22:19:17It says make sure that all your tables
22:19:19inside the database have a clustered
22:19:22primary key and as well it can help you
22:19:24building the relationship between tables
22:19:26where you have primary keys and foreign
22:19:27keys and you can join tables then very
22:19:30easily and as well a primary key has
22:19:32importance for the performance and
22:19:34incale server the default going to be a
22:19:36clustered index which is really good to
22:19:38have an index on the primary key because
22:19:40sometimes you are doing like an update
22:19:42operations or delete operations it's
22:19:44going to help up by the lookups of
22:19:46joining tables. So there are a lot of
22:19:48performance benefits of having a primary
22:19:50key and make sure that all your tables
22:19:52having a primary key. So as you can see
22:19:54the issue of our table we don't have a
22:19:56primary key and our primary key going to
22:19:58be the customer ID. So let's go and do
22:19:59that primary key and as I said as a
22:20:03default it can be clustered but I'm
22:20:05going to write it down in case if you
22:20:07are working with different databases
22:20:08make sure it is clustered. Okay moving
22:20:11on to the next one. It's not only about
22:20:12the primary key we have to take care of
22:20:14our foreign keys. So the best practice
22:20:16says create non-clustered index for the
22:20:20foreign keys if they are frequently
22:20:22used. The foreign keys are usually
22:20:24important in order to connect and join
22:20:26two tables and usually we frequently use
22:20:28it and not only that we use it sometimes
22:20:30in order to filter the data and if you
22:20:33create a nonclustered index for that it
22:20:35can improve the speed. So what we can do
22:20:37it's very simple we're going to go and
22:20:38create a nclustered index on our table
22:20:41customers info for the foreign key
22:20:44employee ID. So how to do it is very
22:20:46simple. We're going to go and say create
22:20:48nonclustered index on our table the
22:20:50customer's info on our foreign key the
22:20:52employee ID. But again make sure that
22:20:54this is an important foreign key that is
22:20:57used frequently from your queries. All
22:20:59right friends so as you can see there
22:21:01are a lot of best practices on how to
22:21:02improve and optimize the DDL. Having a
22:21:05healthy DDL can improve the performance
22:21:07of your queries. Now in the next section
22:21:09I'm going to show you the best practices
22:21:11and tips and tricks about indexing. So
22:21:13let's go.
22:21:18All right, the fifth best practices and
22:21:20the most important one is avoid
22:21:22overindexing because too many index is
22:21:24going to slow down the insert, update,
22:21:26delete operations and it's going to
22:21:28confuse as well the execution plan about
22:21:30choosing the right index and the
22:21:32performance of the whole system going to
22:21:33go down. And another tip is to monitor
22:21:35the usage of the indexes and I can tell
22:21:38you 90% of the indexes that is being
22:21:41created usually are not used at all. So
22:21:43they are taking a lot of space slowing
22:21:46down everything. So go and drop those
22:21:48unused indexes in your system. The next
22:21:51best practice is to have a regular job
22:21:53like maybe a weekly job. So first you
22:21:56have to update the statistics regularly
22:21:58as you are inserting new data and
22:22:00modifying data inside your database. The
22:22:02statistics and the metadata of your
22:22:04tables might get outdated and this is
22:22:06really bad because you will not get an
22:22:08optimal execution plan for your queries
22:22:10and this can slow down your queries of
22:22:12course. So regularly make sure that all
22:22:15the statistics are updated in order to
22:22:17have an optimal execution plan. And what
22:22:19else we can do in this weekly job is
22:22:21that we can go and rebuild and
22:22:23reorganize our indexes. And that is to
22:22:26make sure that we are preventing data
22:22:28fragmentations in our indexes. Data
22:22:31fragmentations in your indexes is really
22:22:33bad because there will be a lot of
22:22:34unused spaces. The order of your
22:22:36clustered index will not be correct. So
22:22:39make sure that at least weekly you are
22:22:41rebuilding and reorganizing all your
22:22:43indexes. So those are the best practices
22:22:45of improving the performance and
22:22:47optimizing your indexing. If you are
22:22:49struggling with very large tables in
22:22:51your projects like having fact tables,
22:22:54then go and use SQL partitioning in
22:22:57order to divide these tables into
22:22:59smaller pieces which can improve the
22:23:02performance whether you are reading data
22:23:04from the table or writing data. And of
22:23:06course you can go and mix things where
22:23:09you can go and apply a column store
22:23:11index on this partition table then you
22:23:14will get the best performance if you are
22:23:17having large
22:23:21tables. All right friends so that's all
22:23:24those are the best practices tips and
22:23:26tricks that I've collected in the many
22:23:28years working with SQL. And now my final
22:23:31thought about this is that try always to
22:23:33focus on making clear queries. Make it
22:23:36like easy to read and easy to understand
22:23:38and try to optimize the performance only
22:23:41if it's needed. So if you have like
22:23:42small database don't worry a lot about
22:23:44the performance because the SQL
22:23:46optimizer going to pick the best plan
22:23:48for you and focus only on having simple
22:23:50queries and if there is like performance
22:23:53problem always test using the execution
22:23:55plan. It should be your judge. So if you
22:23:57are applying any index or you are
22:23:59rewriting your queries always compare
22:24:02before and after using the execution
22:24:04plan. And if you are gaining more
22:24:05performance then adopt the new query or
22:24:08the new index. All right my friends. So
22:24:10that's all the tips and tricks best
22:24:11practices that I have for you in order
22:24:13to optimize the performance. And with
22:24:15that we have covered now everything
22:24:16about this chapter the performance
22:24:17optimization. Now in the next chapter
22:24:19I'm going to show you how I use AI in
22:24:22order to assist me while I'm using SQL.
22:24:24So let's
22:24:28go. All right. Right. So now I would
22:24:30like to share something important with
22:24:32you especially as a future developer
22:24:34that is working with AI. One of the best
22:24:36ways in order to truly build skill and
22:24:39to grow as a developer is by working on
22:24:41complex task and issue on your own. So
22:24:44when you are stuck on complex task and
22:24:46you are pushing yourself to find a
22:24:48solution for it and you are writing your
22:24:50code in yourself here the magic happens
22:24:53and the real learning can happen. And if
22:24:56you jump too quickly and ask the AI for
22:24:58a solution, what you are doing, you are
22:25:00skipping an essential step in order to
22:25:02become an expert. And more important
22:25:04than that, you won't develop skills in
22:25:07order to understand when and where the
22:25:09AI was wrong. So my recommendation here
22:25:12is to have a discipline. Always try to
22:25:14solve the task on your own and only turn
22:25:16to AI if you don't have any more ideas
22:25:18on how to solve the task. So that's my
22:25:21opinion and my advice for you.
22:25:27So quickly what is shippet? It is an AI
22:25:29program that is developed by open AI
22:25:32that is trained to understand questions
22:25:34and provide humanlike answers. So what
22:25:37GPT stands for? The G stands for
22:25:39generative. So that means the data model
22:25:41can generate a new content new text and
22:25:45P stands for pre-trained. The data model
22:25:47is already trained on huge amount of
22:25:50data. And the T stands for transformer.
22:25:52It is type of neural network
22:25:54architecture that processes your
22:25:56sentences in the prompts in order to
22:25:58understand the context behind it very
22:26:01fast and accurate. And in the other hand
22:26:03we have the GitHub copilot. It is
22:26:05developed by the GitHub and as well
22:26:07using the same data models from the open
22:26:10AAI. So that means both shad and copilot
22:26:14both of them are using the same language
22:26:16model that is developed from OpenAI. So
22:26:19the GitHub copilot did train on tons of
22:26:22codes that is available in GitHub. So
22:26:24how it works as you are writing a code
22:26:26in the code editor like for example
22:26:28visual studio it going to provide
22:26:30realtime suggestions as you are writing
22:26:33and typing your code. So now if we
22:26:35compare those two shad and the copilot
22:26:37we can say that the shajibet is a
22:26:39standalone application where you can
22:26:42interact with it using a website or an
22:26:44app where you go and start a
22:26:45conversation with the AI where in the
22:26:48other hand the copilot is directly
22:26:50integrated in your code editor like for
22:26:52example the visual studio code this is
22:26:55way better than shibility because you
22:26:57have realtime interaction with the AI
22:26:59this is a great advantage for the
22:27:01copilot because everything in one place
22:27:03so with the copilot pilot you are
22:27:05getting realtime assistant during your
22:27:07coding. So the main purpose of the ship
22:27:10is to have a conversation with the AI
22:27:12for any topic that you like not limited
22:27:15only for software developments but in
22:27:18the other hand a copilot focuses only on
22:27:21assisting the software development where
22:27:23you as a developer as you are writing
22:27:25your code you are getting auto
22:27:27completion of the code or maybe a block
22:27:29of code as a suggestion. So these are
22:27:31the key differences between shad and
22:27:34copilot.
22:27:38Now if you are doing software
22:27:40developments or you are working with
22:27:42data projects and of course it depends
22:27:44on your role in the projects there will
22:27:46be many different types of tasks and
22:27:48activities that should be done in the
22:27:50project like there will be a lot of
22:27:52brainstormings about new ideas and
22:27:54coding solutions debugging generating
22:27:56documentations discussing the different
22:27:58types of architecture doing road cause
22:28:01analyzes. So the spectrum of activities
22:28:03and tasks in each projects usually is
22:28:06very huge. And of course we can go and
22:28:08use the help of different AI tools to
22:28:10assist us with those tasks and
22:28:12activities and there is like not one AI
22:28:14tool that can cover all those stuff. I
22:28:17tend to jump between co-pilots and
22:28:18something like Shajbet. Okay. So now I'm
22:28:20going to go and map those different
22:28:21tasks to either sht or copilot. So now
22:28:24let's focus on the shibbet. The first
22:28:26one is brainstorming and ideas. So now
22:28:29if we have in our project a big task or
22:28:31let's say a big issue that we want to
22:28:33find solution for it. I tend to use of
22:28:35course tools like shad in order to have
22:28:38a discussion about the topic in order to
22:28:41explore and discuss multiple ideas and
22:28:43then start evaluating all those ideas.
22:28:45The next one where I found myself using
22:28:47shbt is doing the project planning. So
22:28:49it is as well something high level. You
22:28:51can go and discuss with the shaj GBT
22:28:53about the design of your projects and
22:28:55you can as well discuss the milestones
22:28:58the road map of the projects. The next
22:29:00thing that I find myself using shajbt is
22:29:02for learning knowledge and research. If
22:29:04you are working with big data projects
22:29:06you will be overwhelmed with the amount
22:29:08of cloud services and AI analytics
22:29:11tools. So and of course you can go and
22:29:13learn new stuff gather informations and
22:29:15knowledge using shajibb. Okay, moving on
22:29:18to the next task. We have generating
22:29:19documentations. Writing documentations
22:29:21is always painful process and consumes a
22:29:24lot of time and I tend to use tools like
22:29:26shibbit in order to generate those
22:29:28documentations. But of course, I always
22:29:30review the documentations and make it
22:29:32short. Okay, moving on to another topic
22:29:34where I use shadet is that to discuss
22:29:36architecture. Of course, if you are
22:29:37starting new projects, they will be like
22:29:39different types of architecture in order
22:29:41to implement the projects. And of
22:29:43course, you can discuss with the
22:29:44shajibility about the different types of
22:29:46architecture and if you give the
22:29:47specifications about your projects then
22:29:49you can discuss with the shajibility
22:29:51which architecture is suitable for the
22:29:53project. And another task that I find
22:29:55myself always like researching is
22:29:58exploring the best practices, tips and
22:30:00tricks. So you can have a discussion
22:30:01with the SHP about the recommendations,
22:30:04what are the best practices, what are
22:30:06the common pitfalls in order to make
22:30:08sure that your code and your solution is
22:30:10always up to date with the best
22:30:12practices. And one more thing, if
22:30:14there's like in the projects a very
22:30:16complex task, then I tend to have a
22:30:18discussion with a tool like Shajibet in
22:30:20order to break this complex task into
22:30:22small pieces and start finding the
22:30:24solution for each piece. And now in the
22:30:26other hand, I'm using copilot in order
22:30:28to solve different type of tasks. So
22:30:30here where I get my hand dirty in the
22:30:32code. So while I'm coding I'm using
22:30:34alltime co-pilot in order to assist me
22:30:37because it provide directly inline
22:30:39suggestions and help me to code faster
22:30:41and reduce the human error that I might
22:30:43make. So while I'm writing a code or
22:30:45debugging I tend to use copilot and I
22:30:48don't find myself going to shy GBT to
22:30:50ask about code or syntax. We can do it
22:30:53directly in the copilot. And one task
22:30:55that is very famous in any software
22:30:57developments we have the refactoring. So
22:30:59if you have like a code that is slow and
22:31:01bad designs and you want to refactor the
22:31:03whole codes, you can do it directly in
22:31:05your code together with the copilot in
22:31:06order to find optimizations. And I use
22:31:09as well copilot in order to add inline
22:31:11comments. So I don't find myself going
22:31:13to ship and asking to add comments to my
22:31:15codes. You can do it directly in your
22:31:18code using cilot. And of course if
22:31:20everything is working perfectly, I have
22:31:22the best practices, the good
22:31:23performance, I have the comments, it's
22:31:24still you have to maintain nice style
22:31:26and format of your code. And of course
22:31:28now we can do that directly using the
22:31:30copilot. We don't have to go and jump to
22:31:31shajbt in order to style and format your
22:31:33code. And as you can see I'm currently
22:31:35using both of them for different types
22:31:37of tasks. So again if I have the feeling
22:31:39that I have to discuss something I go to
22:31:41shbt. But once the idea is very clear
22:31:44and I know the solution then I start
22:31:47using copilot in order to write the code
22:31:49and with the help of the copilot I can
22:31:52deliver clean and professional code. So
22:31:54this is how I currently use both
22:31:56Shajbuty and
22:32:01Copilot. Okay friends, so now what we're
22:32:03going to do, I'm going to show you a
22:32:04quick guide about the GitHub copilot in
22:32:07the Visual Studio Code. Once you create
22:32:09a profile and connect it to your Visual
22:32:11Studio, you will get a new icon for the
22:32:13copilot. So once you go there, you can
22:32:15see quickly the status and as well you
22:32:18can go and disable the copilot. So if
22:32:20you have it like this, that's means your
22:32:22co-pilot is active. So now once you have
22:32:24everything up and running, what you have
22:32:26to do is very simple. Just go and start
22:32:28writing your code. So start typing any
22:32:31select statements. And now you can see
22:32:33that we have a gray text. This gray text
22:32:36called the ghost text. It is an auto
22:32:38completion from the copilot. And now it
22:32:40says select star from table. And now as
22:32:43you can see as I mouse hover on it, we
22:32:46can see that I can go and switch between
22:32:48different suggestions. So here we have
22:32:50like three suggestions. One, two, three.
22:32:52And I'm going to go with the third one.
22:32:54So now here as it says if you want to
22:32:56accept the suggestion all what you have
22:32:58to do is to press tab. So let's go and
22:33:00do it. So you are accepting the whole
22:33:02thing. But now if you say you know what
22:33:04I'm going to accept only part of the
22:33:06code. So let's go again and write
22:33:08select. So this time we're going to be
22:33:10selective. In order to do that hold
22:33:12control and then with the right arrow
22:33:14and with that we are accepting part of
22:33:17the ghost not everything. But of course
22:33:19if you are accepting the whole thing
22:33:20just go with the tab. And now there is
22:33:22another way in order to trigger the
22:33:24ghost text and that's by defining first
22:33:27a comments. For example we want to
22:33:28select the top three customers based on
22:33:30the score. So now once you start writing
22:33:32the query the co-pilot going to go and
22:33:34write a query that is relevant for the
22:33:37comments. So now as you can see we are
22:33:38getting top three from customers because
22:33:41we want the top three customers and here
22:33:42we have like two suggestions like over
22:33:45here we have the order buy or without
22:33:46it. So I will go with order by and hit a
22:33:48tap. And now here another suggestion
22:33:50which is correct. In order to solve the
22:33:51data from the highest to the lowest. All
22:33:53right moving on to the next one. As we
22:33:55learned in SQL in order to solve a task
22:33:58there could be like multiple solutions
22:34:00and multiple variants of queries that
22:34:02solving the same task. So let's say that
22:34:04we have this task rank customers based
22:34:06on their total order sales. So what you
22:34:08can do if you start writing the query we
22:34:10are getting now the ghost text. But now
22:34:12what we can do we can go and hit ct
22:34:13controll enter. So now what happens on
22:34:15the right side you will get different
22:34:17suggestions and here we have like nine
22:34:19suggestions on how to solve this task in
22:34:21scale. So now what you have to do is to
22:34:23go through all those suggestions and
22:34:25pick one. For example I can go with the
22:34:27suggestion number three and say accept
22:34:29suggestion and you will get it in your
22:34:31code editor. So this is what we mean
22:34:32with the copilot autocomp completion and
22:34:35integrating the AI directly as you are
22:34:38developing and writing a code. Now in
22:34:39the co-pilot, not only using the ghost
22:34:41text and the autoco compilation, we can
22:34:43go and interact with the AI using inline
22:34:46shots. So it's something like shimity.
22:34:48Now in order to trigger the shot, what
22:34:50you're going to do, you're going to go
22:34:51and hit control I and then you're going
22:34:53to get a place in order to ask the
22:34:55copilot any question like for example
22:34:57join the query with the
22:35:00table sales orders. So let's go and hit
22:35:04it.
22:35:06And now as you can see we got a full
22:35:08query where the customers is joined with
22:35:10the orders and it is totally correct how
22:35:13the table are joins. So that means
22:35:14copilot knows already all the tables
22:35:17that I have in the database and as well
22:35:20the columns and how to join them. This
22:35:22is amazing. So if you like it you go and
22:35:24accept it of course and this is way
22:35:26faster than having shajibbd because in
22:35:28shajibity you have to introduce your
22:35:30database your columns and stuff before
22:35:32even asking anything. This is exactly
22:35:35the power of copilot. Now what else we
22:35:38can do with that? We can go and
22:35:39highlight part of our codes and then
22:35:41start again the shots and here we can
22:35:43say replace this column with an
22:35:46aggregation of the sales. So let's go
22:35:50and hit okay. Now as you can see it
22:35:52replaced it with an aggregate function.
22:35:53And one thing that is very important the
22:35:55code is not changed yet. So it is
22:35:58highlighted and showing you a suggestion
22:36:00and now you have to accept it or discard
22:36:02it. If you discard, nothing going to
22:36:04change in your codes. But once you say
22:36:06accept, it's going to go and replace
22:36:07your original codes. So if you go and do
22:36:10that, now your code is replaced with the
22:36:12AI suggestion. Okay. Another thing about
22:36:14the copilot, it's try to fix issues that
22:36:17you have in your codes. So for example,
22:36:19we have here an error. If you go and
22:36:20mouse hover it, you can see a menu from
22:36:22the copilot in order to view the error
22:36:24or to fix it. And another way to do
22:36:26that, if you right click on it, you go
22:36:28to the copilot. And here you can see we
22:36:30can explain or fix. So if you go and
22:36:33explain, you will get another window
22:36:34where you get an explanation about the
22:36:36issue in your code. And once you
22:36:38understand it, you can go and ask the
22:36:39copilot in order to fix it. So let's go
22:36:42over here and go to
22:36:44fix. And with that, the copilot did fix
22:36:46the issue. It was all about the order of
22:36:49the select statements. So first you have
22:36:50to do the group by then order by. So it
22:36:52helps you to find issues and to fix it
22:36:54as well. And now, as you might already
22:36:56noticed, as we are writing the code and
22:36:58interacting with the Visual Studio, you
22:37:00will often get a sparkle, this little
22:37:03yellow sparkle on the left side. So, you
22:37:05will see this icon each time the copilot
22:37:07thinks it can help. So, if you go and
22:37:09click on it, you will get a menu of
22:37:11different stuff that the copilot can do
22:37:13for you, like fixing, explaining,
22:37:15modifying, and so on. Well, my friends,
22:37:17that's it. This is the copilot, and it
22:37:19is very simple, but yet very powerful
22:37:22for developers. And of course, not only
22:37:24for SQL, for anything like for Python
22:37:26and so on. Everything is integrated in
22:37:28one place. I don't have to jump to
22:37:29Shajibbet and ask stuff. It is live and
22:37:32I can do it directly as I'm writing my
22:37:34code. So that's all for Copilot. All
22:37:36right friends. So now let's switch to
22:37:38Shajibet. So let's start first by
22:37:40understanding the structure and the
22:37:42basic components of Shajbet prompts.
22:37:49So the first component and the most
22:37:50important one we have the tasks. You
22:37:53have to be very clear by defining what
22:37:55the AI should do and without having a
22:37:57clear tasks the AI will not understand
22:38:00what to do. So this is mandatory in each
22:38:03prompt and then after that you have to
22:38:05provide some context. So you give some
22:38:07background informations like for example
22:38:09you say I am students or I am a data
22:38:11engineer and so on. And another
22:38:13components we have to add
22:38:15specifications. So in the task you give
22:38:17the main task what the AI should do but
22:38:20with the specifications you go in
22:38:22details like for example which topic
22:38:24should be added or maybe excluded the
22:38:26number of word counts. So here you are
22:38:29specifying a lot of wishes and small
22:38:31details and specifications in order to
22:38:33get an answer that meet your
22:38:35expectations. So both of the context and
22:38:37specifications they are important. And
22:38:39then after that we have some nice to
22:38:41have components like for example
22:38:43specifying a rule. So here you give the
22:38:45AI a role like for example you tell it
22:38:48to act as an expert as a teacher
22:38:51interviewer. So you are setting the AI
22:38:53to play a role and the last component
22:38:56that you can add as as well the tone.
22:38:58Here you are defining like the voice of
22:39:00the answer in order just to make the
22:39:02answer like more friendly and easy to
22:39:04read and engaging. So the role and the
22:39:06tone they are nice to have and if you go
22:39:08and use all those components you will
22:39:10get a better results from the AI. So
22:39:13let's take for example the following
22:39:15prompts explain SQL window functions. So
22:39:18this is very simple and very short and
22:39:20here we have only one component the
22:39:22task. So here you are not giving any
22:39:24context whether it is for data analytics
22:39:27or for data engineering. So you leave it
22:39:29up to the AI and maybe the answer that
22:39:31you will get will not meet the
22:39:33expectation that you have. And now if
22:39:35you want to shape it in the way that you
22:39:37want you have to add more components
22:39:39like for example this prompt you are
22:39:40saying you are a senior SQL expert. So
22:39:43here we are defining the rule for the
22:39:46AI. So the AI should act now as an SQL
22:39:49expert. And then the next section we are
22:39:51adding a context to the prompts. So we
22:39:53are saying I'm data analyst working on
22:39:55SQL projects using SQL server. So now
22:39:58the answer that you will get from the AI
22:40:00going to use the syntax of the SQL
22:40:02server and focus on the topic of
22:40:03analytics. That's why the context is
22:40:05very important and then we go specify in
22:40:08the prompt the task the main task. So we
22:40:10say explain the concept of SQL window
22:40:12functions and do the following. And now
22:40:14we go and give more fine details about
22:40:17what the AI should provide. We are
22:40:19saying explain each window function and
22:40:22show the syntax. describe why they are
22:40:24important and when to use them and list
22:40:26the top three use cases. So you are now
22:40:29specifying what you are expecting from
22:40:31the AI and after that of course it is
22:40:33nice to have we specify the tone of the
22:40:36explanation. So we say the tone should
22:40:38be conversational and direct as if you
22:40:41are speaking to me onetoone so that it
22:40:43is not like you are reading a document
22:40:45you are reading something that is
22:40:47engaging. So I know this prompt is
22:40:49really big but still you will get way
22:40:51better results than only saying explain
22:40:54the concepts. So those are the main
22:40:56components that I usually use if I'm
22:40:58starting like a conversation and a
22:41:00discussion with the
22:41:05shajuti. Okay. Next I'm going to show
22:41:07you the frequently used prompts that I
22:41:09use in my projects. Now little bit
22:41:11awareness about using shajib in
22:41:13companies. If you are working in new
22:41:15company, make sure to ask about the
22:41:17rules of using Shia Gibbt because some
22:41:19companies offer their own chatbots for
22:41:22few security reasons. So make sure
22:41:24always to check with the rules before
22:41:26jumping immediately to sht. All right.
22:41:28So let's start with the first prompts.
22:41:30We can use shad in order to solve an SQL
22:41:33task that you have in the project. So
22:41:35let's see this prompts. It start first
22:41:37with the context. So I'm telling that I
22:41:40have an SQL server database and we have
22:41:42like two tables. So now I have to
22:41:44explain for shad the database that I
22:41:46have. So I'm saying we have a table
22:41:48called orders and we have the following
22:41:50columns and we have another table called
22:41:52customers and here are the columns for
22:41:53the customers. So that I gave shy a
22:41:56context about the tables that I have in
22:41:58my database and as well I was precise
22:42:01about the database. It is SQL server.
22:42:03Now after we have the context the next
22:42:05step is that I'm going to tell SQL what
22:42:07to do. So I'm telling the AI do the
22:42:09following. write a query to rank
22:42:11customers based on their sales and then
22:42:13I'm detailing what I'm expecting to have
22:42:15at the output. So the result should
22:42:17include customer ID, full name, country,
22:42:19total sales and so on. And here I'm
22:42:21adding like more tasks. It's not enough
22:42:23to have a query. I would like as well to
22:42:24have a comments. So I'm saying include
22:42:27comments but avoid commenting on obvious
22:42:29parts because if you tell just include
22:42:32comments, you will get a lot of
22:42:34unnecessary comments. Now of course in
22:42:36square there is like not one solution
22:42:37for a task. there is always like
22:42:40different variants on how to achieve the
22:42:42same task. So usually I would like to
22:42:44understand what are my options. That's
22:42:45why I'm telling Shaji write three
22:42:48different versions of the query to
22:42:50achieve this task and then I would like
22:42:51to evaluate each of those versions and
22:42:54that's why I'm giving the task for the
22:42:55AI to evaluate those versions and to
22:42:58focus on two things. It is easy to read
22:43:00and as well has good performance. Okay.
22:43:03So let's see what shajivity going to
22:43:04give us the results. So we can see the
22:43:06first solution over here where shadivity
22:43:08is using the CTE. So we can see in the
22:43:11CT over here that the table first are
22:43:13joined and then we have like a group by
22:43:15in order to aggregate the sales. In the
22:43:18step two we can see over here we have
22:43:20the rank window function in order to
22:43:22rank the sales. So of course you can do
22:43:24that. Let's check the version number two
22:43:27over here. So they I used the subquery
22:43:30and it is as well a nice solution where
22:43:33the shad first prepared the data. So
22:43:36first done the aggregation before
22:43:38joining the data. Let's get the last
22:43:40solution over here. So we have here
22:43:42single query using window function which
22:43:45is as you can see it is the smallest
22:43:47one. We don't have CTE we don't have any
22:43:49sub queries. So first it is joining the
22:43:52tables and doing together the group by
22:43:54together with the window function and
22:43:56after that we get an evaluation from the
22:43:59AI where where as you can see it focus
22:44:01on two things the readability and the
22:44:03performance. So it is saying with the
22:44:05CTE the readability is really high
22:44:08compared to the sub query and to the
22:44:10last version where you have the group by
22:44:12together with the window function. So I
22:44:13totally agree with the shajibbity the
22:44:15first version was the best one for the
22:44:17readability. Now checking the
22:44:18performance. You can see the performance
22:44:20is moderate. The second one, the
22:44:22subquery is good. And the last one is
22:44:25the best for the performance. But of
22:44:27course, always test with the execution
22:44:28plan. So as you can see, there is like a
22:44:30trade-off between the readability and
22:44:32the performance. If the priority is
22:44:34readability, then go with the version
22:44:35one. But if the priority is the
22:44:37performance, then go with the version
22:44:39three. As you can see, we got three
22:44:41solutions for our one task. And you can
22:44:43now evaluate which one you want to use.
22:44:46And this is really amazing, right? All
22:44:48right, moving on to the next one that I
22:44:49frequently use. We have
22:44:52impromptability. As you are creating an
22:44:54SQL query for a complex task, you might
22:44:57end up writing a lot of CTE, sub
22:44:59queries. You might end up having a lot
22:45:01of joins, sub queries, CTE, hundreds of
22:45:05lines, and you might lose the big
22:45:07picture. So what I always do, I give the
22:45:09query to the SHBT and ask it to optimize
22:45:12it in order to be more readable and to
22:45:14find any redundancy in my query in order
22:45:17to consolidate it. So now let's check
22:45:19the prompt. It says the following SQL
22:45:21server query is long and hard to
22:45:23understand. And then we're going to give
22:45:24the AI tasks. So the first task is to
22:45:27improve its readability and the next one
22:45:29is to detect any redundancy in the code
22:45:32in order to remove it and to consolidate
22:45:34the query. So to make our query compact
22:45:36and small and of course to include some
22:45:38comments and not to comment the obvious
22:45:40parts and now always if there is like
22:45:42some optimizations there should be a
22:45:44learning process. So I'm asking now the
22:45:46AI to explain each improvement to
22:45:49understand the reasons behind it so that
22:45:51next time I'm writing the queries I can
22:45:53avoid those mistakes and of course you
22:45:55have to go and give the query to the AI.
22:45:57All right. So now let's check the answer
22:45:59from the ship for my prompt. So as you
22:46:02can see we have a really long query and
22:46:04here we have now from the result the
22:46:06improved query. So we can see that we
22:46:09have only one city. Well that is crazy.
22:46:12We had before like five six cities and
22:46:14we can see here that the team managed to
22:46:17put everything in one city and then do
22:46:20all the aggregations and the window
22:46:21function and then we have here the final
22:46:24select. Well this is huge improvement to
22:46:26the previous query. Let's check here the
22:46:28explanation. So it says it consolidated
22:46:30the cities so combined all the cities
22:46:33into one and many other stuff like there
22:46:36were a lot of unnecessary joins and so
22:46:38on. And here a small improvement where
22:46:40it uses the concat instead of the plus
22:46:43because concat is standards for multiple
22:46:46databases. And here we have a final
22:46:48benefits. So we have shorter query
22:46:50instead of five CDs we have only one and
22:46:53combining the logic you can reduce the
22:46:55number of scans of the tables which is
22:46:57correct. So as you can see it is the
22:46:59magic of the AI. It found the issues in
22:47:01my code, improved the readability and
22:47:03reduced all the redundancy and
22:47:05unnecessary joints and so on in the SQL
22:47:08script. Okay, moving on to the next
22:47:10prompt. It is about optimizing the
22:47:12performance of my query. And if you are
22:47:14working in big projects where you have
22:47:16like millions of data in your tables, it
22:47:18can be an issue if you are writing
22:47:19queries that are not following the best
22:47:21practices for performance. So that's why
22:47:24I go and double check with the AI
22:47:26whether my script is following the best
22:47:28practices for the performance. So as
22:47:30usual in the prompt we have to go and
22:47:31give the context. So the following SQL
22:47:34server query is slow and then we start
22:47:36giving the AI some tasks. So propose
22:47:38optimizations to improve its performance
22:47:40and provide me then the improved SQL
22:47:43query and I would like always to
22:47:45understand the reason why it's better to
22:47:47write it in another way so that by the
22:47:49next time I improve while I'm writing
22:47:51the query. So explain each improvement
22:47:54to understand the reasoning behind it
22:47:56and then at the end we go and give our
22:47:58query. Okay. So now let's write the
22:48:00prompts on the following query over
22:48:02here. So on this query we have a lot of
22:48:04bad practices like for example doing
22:48:06aggregations using correlated subquery.
22:48:09We are using a lot of functions inside
22:48:11the work clause which is not really good
22:48:13for indexing and we are using a lot of
22:48:15or operators and here we have again a
22:48:18subquery. So let's check whether shad
22:48:20going to find all those bad practices.
22:48:23So let's check the results from the
22:48:24shad. And as you can see now we have an
22:48:26optimized query. It is little bit longer
22:48:29but I think we have here better
22:48:30practices. So we have here a lot of
22:48:32changes. Let's check what did. So first
22:48:35it replaced the lower in the query. It
22:48:37says it's not really good to use
22:48:39functions in the works so that the index
22:48:43can work. So it replaced the lower with
22:48:46the order status without the function.
22:48:48the next one. So it is avoiding the
22:48:50correlated subquery. So instead of that
22:48:53it is using a lift join. So it is
22:48:55joining the table normally without doing
22:48:57any correlated queries and as well it is
22:49:00avoiding the function year in the works
22:49:02and instead of that it is using the
22:49:04range using between and the next one it
22:49:07is using exist better than in which is
22:49:09better for the performance of course. So
22:49:11as you can see you can use the AI in
22:49:13order to optimize the performance of
22:49:14your query and to convert it to a script
22:49:17that is following the best practices. Of
22:49:20course my recommendations always don't
22:49:22go blindly with all changes that is
22:49:24suggested from the shajibity. Always
22:49:26take each recommendation one by one.
22:49:28Test it and evaluate it using your
22:49:30knowledge. Okay to the next one. It is
22:49:32interesting one. We can use
22:49:35[Music]
22:49:37impromptution plan. So now the execution
22:49:39plans usually are advanced. So you need
22:49:42a lot of knowhow and experience in order
22:49:44to understand and read the execution
22:49:46plan and if you have a big query it's
22:49:48going to be really nightmare in order to
22:49:50understand the flow and where is exactly
22:49:52the issue. But now we are not alone. We
22:49:54have assistant the AI in order to help
22:49:57us understanding this complex stuff. So
22:49:59what we can do we can take a screenshot
22:50:01of the execution plan and upload it to
22:50:03Shajib and we say the image is execution
22:50:06plan of SQL server query and now we give
22:50:09the following task to say describe the
22:50:11execution plan step by step after that
22:50:13I'm going to tell SQL to identify the
22:50:15performance bottlenecks and where is
22:50:17exactly the issue what makes my query
22:50:20slow this is of course the hardest part
22:50:22of reading an execution plan and once it
22:50:25identify the performance issues I'm
22:50:27going to ask it to suggest ways to
22:50:28improve improve the performance and
22:50:30optimize the execution plan. So first
22:50:33understand the execution plan identify
22:50:35the issues and how to optimize it. Okay.
22:50:37So now after uploading the photo and
22:50:39asking the AI we have the following
22:50:41results. So now we can see a detailed
22:50:44explanation about the execution plan and
22:50:46there is like a lot of details. I will
22:50:48not go through everything. So we start
22:50:50with the table scans then the cluster
22:50:52scan and the nested loops. So we have
22:50:54several nested loops and then the
22:50:56aggregation and the final step. So that
22:50:58now we have like a nice explanation what
22:51:00is SQL is doing behind the scenes for my
22:51:03query and you don't have to be an expert
22:51:05understanding the execution plan. You
22:51:07can ask the AI about it. Now what is
22:51:09very important is to understand where
22:51:11are the bottlenecks what are the
22:51:13problems. So let's see what's we have
22:51:15here. So let's say the first one we have
22:51:17a table scan which is really bad. That
22:51:19means this table the orders archive does
22:51:22not has any index. So it says the table
22:51:24scan indicates a lake of useful index on
22:51:26the table which forces the engine to
22:51:28scan the whole table or rows. And now
22:51:31what is very important is the nested
22:51:33loops in the joins. This is really bad
22:51:36if you have big tables. So here it's
22:51:38saying it's fine if you have like small
22:51:40data sets but it going to be really
22:51:42problematic if you have many rows. So as
22:51:45you can see we are getting more
22:51:46knowledge about the issues that we have
22:51:48from our execution plan. And the last
22:51:50step it is the suggestions. So the first
22:51:53one and the most obvious one is to add
22:51:55an index to the orders archive. The
22:51:57nonclustered index. Well, if there's no
22:51:59index at all, I would go first with a
22:52:01clustered index, not immediately with a
22:52:03nonclustered index. And then some other
22:52:05best practices, but I think this one is
22:52:08very relevant is to change the join
22:52:10type. So you can use the hints in order
22:52:12to use a merge join or a hash join. So
22:52:15now we understand how it works, where
22:52:17are the issues and what the suggestions
22:52:19to fix it. All right, the next prompt is
22:52:21about debugging. As you are writing a
22:52:23complex SQL query, you might get from
22:52:25the database an error when you execute
22:52:27it and sometimes it is challenging to
22:52:30find the root cause of the issue. So we
22:52:32have the following prompts. First the
22:52:33context is going to say the following
22:52:35SQL server query causing this error.
22:52:38Then we can paste the error message that
22:52:40we are getting and then we ask the AI to
22:52:42do the following stuff. First explain
22:52:44the error message. So I would like to
22:52:45have better understanding of the error.
22:52:47And then we ask the AI to find the root
22:52:50cause of the issue from my scripts. And
22:52:52after finding the problem and the issue,
22:52:54we're going to ask the AI to suggest how
22:52:56to fix it. And of course, we have to
22:52:58give in the prompt as well our SQL
22:53:00query. All right. So now I have the
22:53:02following query and if I execute it, I'm
22:53:04getting the following error. It says the
22:53:06column sales.order dot sales in invalid
22:53:10in the select list because it is not
22:53:12contained in the aggregations and so on.
22:53:14So I'm not really understanding what's
22:53:15going on. Let's ask the AI about it. So
22:53:18let's check what shity did answer. When
22:53:20you are using group by every column in
22:53:22the select must be used in the group by
22:53:25as well. And it says in your query you
22:53:27are selecting few columns which is this
22:53:29one is valid. The other two as well
22:53:31valid but we have one inside the rank
22:53:34function. It is invalid. Okay. So now we
22:53:37can see here more details about the root
22:53:39cause. It is saying when you are using
22:53:41window function like the rank it doesn't
22:53:43directly work with the aggregate
22:53:45functions. So here it's indicate clearly
22:53:47that the sales inside the rank function
22:53:50is the issue. So let's see the fix over
22:53:53here. So since we don't have here sales
22:53:55at all you cannot have here sales in the
22:53:57partition. That's why the fix here is to
22:54:00use the sum of sales because we have it
22:54:02in the select. And here you have as well
22:54:04a nice explanation about the fix. So you
22:54:07can see here we have an explanation
22:54:09about the error message the road cause
22:54:11it's pointing exactly where there's the
22:54:13issue suggesting a fix and explaining
22:54:16the fix and this is exactly the steps
22:54:18that you have to do if you are debugging
22:54:20a code all right moving on to the next
22:54:22prompt we can use AI to explain the
22:54:24result that I'm getting from SQL well
22:54:26sometimes you might have an SQL query
22:54:29that you have in the project and you are
22:54:30not understanding why you are getting
22:54:32specific results so as usual we start
22:54:34with the context we tell the AI I didn't
22:54:37understand the result of the following
22:54:38SQL server query and then we ask the AI
22:54:41to do the following. First break down
22:54:43how SQL processes the query step by step
22:54:47and as well I would like to get an
22:54:48explanation for each stage and how the
22:54:51result is formed. So as you can see here
22:54:53I don't need any optimizations. I don't
22:54:55need in the output any query. I just
22:54:58need an explanation and then at the end
22:55:00you're going to go and paste your query.
22:55:02Okay. So now we have the following
22:55:03query. We have a recursive CTE where we
22:55:05are generating like numbers between 1
22:55:08and 20. Can tell you recursive CTE are
22:55:10usually like complicated to understand.
22:55:13So now maybe we are having hard time
22:55:14understanding the result of this query.
22:55:16After asking the AI about it, we got the
22:55:19explanation first about the query
22:55:21structure. So it says you are using the
22:55:23CTE with the main query. Well, okay. But
22:55:25what is very interesting is to
22:55:27understand step by step how SQL executed
22:55:30this query. So it tells the step one
22:55:32it's going to go and execute the anchor
22:55:33query and that's why we will get first
22:55:35the one and then the next step the
22:55:37recursive query going to be executed for
22:55:39the first time. So it is saying okay we
22:55:42are adding one to the current value. So
22:55:44as you can see 1 + 1 we will get two and
22:55:47then in the iteration two we will get 2
22:55:49+ 1 3 and it will keep repeating this
22:55:53process until we get all the result from
22:55:551 to 20. And then as well we have here
22:55:57an explanation about the termination of
22:55:59the recursive query. So it's saying the
22:56:01filter is the way out of the loop. So
22:56:04once we reach the 20 it will stop. And
22:56:07then a few informations about the main
22:56:09query and with that you will get a deep
22:56:11knowledge about how works and why you
22:56:14are seeing those results. This is really
22:56:16amazing use case for the GBT. All right
22:56:19friends. So now we're going to talk
22:56:20about my favorite prompts. So we can use
22:56:23the AI to style and format my code. So
22:56:26now once you are done writing a complex
22:56:28query to solve a task and everything is
22:56:30correct and optimized as well for the
22:56:33performance. Now it's time to go and
22:56:35review your code in order to style and
22:56:37format your script. So we have the
22:56:39following prompt. It says the following
22:56:41SQL server query is hard to understand.
22:56:44So now we ask the AI to do the
22:56:45following. Restyle the code to make it
22:56:47easier to read. And the next task for AI
22:56:50is to align all the columns aliases.
22:56:52Sometimes if you are using any tool to
22:56:54style and format your code, you will
22:56:56find that it is bringing a lot of new
22:56:58lines. So I tell he AI, keep it compact,
22:57:01do not introduce unnecessary new lines.
22:57:04And the last task for the AI is to make
22:57:06sure it is following the best practices.
22:57:09And of course, what do we need at the
22:57:10end? Our query. Okay, so now we have the
22:57:12following query. And as you can see, we
22:57:15have very annoying query where it is
22:57:17really hard to read and that's because
22:57:19the format and the styling of the query
22:57:21is really bad. I don't want to speak
22:57:23about the alignment and so on. But as
22:57:25you can see, we have here lower cases,
22:57:27we have here uppercase sometimes for the
22:57:29keywords. And of course, if you are
22:57:31developing and writing codes and you are
22:57:32delivering something like this, it is
22:57:34really not nice. So let's see how shipy
22:57:36can fix it. Okay. So now after executing
22:57:38the prompts, as you can see, now my
22:57:41query looks way nicer. So first of all
22:57:43all the keywords are uppercase and then
22:57:46you can see our CTE are really nice to
22:57:48read. We have here enough spacing. The
22:57:50alignment of everything looks really
22:57:52nice and the case is very clear and the
22:57:55main query over here is as well easy to
22:57:59read. So they done wonderful job styling
22:58:01and formatting my code and here you have
22:58:03like explanation what did change. So
22:58:06first it is saying okay all the keywords
22:58:08are capitalized the alignment of the
22:58:10aliases and the columns and so on. So
22:58:12with that we got a really nice style
22:58:14formatted query that we can share with
22:58:16others. Okay, moving on to the next one.
22:58:19We can use AI in order to generate
22:58:21documentations and as well to add
22:58:23comments to my code. Creating
22:58:25documentations and adding comments to
22:58:27code is usually something very annoying
22:58:29for the developers. And sadly I see a
22:58:32lot of developers that they tend to not
22:58:34add any comments or anything to their
22:58:36code. And of course, this is really bad
22:58:38because you are not thinking about other
22:58:40developers that are reading your code.
22:58:43No
22:58:45god, no god, please no.
22:58:50And since this process is annoying and
22:58:53takes time, we can use the help of AI to
22:58:56improve the speed of creating those
22:58:57stuff. So let's check the following
22:58:59prompt. It says the following SQL server
22:59:01query lakes comments and documentation.
22:59:04So we are saying first insert a leading
22:59:07comment at the start of the query
22:59:08describing its overall purpose. So this
22:59:12is what we usually do. We add at the
22:59:14start a short description about the
22:59:16following code and then it should go and
22:59:18add comments only where clarifications
22:59:21is necessary and very important it
22:59:23should avoid obvious statements. So it's
22:59:25like indexing don't over commenting your
22:59:28code and usually if you are creating
22:59:30query for data analytics it's really
22:59:32good to explain the business rules and
22:59:34transformations that you are doing
22:59:36inside your query and maybe another
22:59:38documentations describing how the query
22:59:41works. So for now we are asking to add
22:59:43comments and documentations and of
22:59:44course you have to go and add your
22:59:46query. Okay. So now I just used this
22:59:48prompt to one of my queries. Let's go
22:59:50and check the results. Now the first
22:59:52comment is the most important one
22:59:54because it gives the overall purpose of
22:59:56the whole query. So let's see what it's
22:59:58saying. It's saying this query identify
23:00:00customers based on their total salaries
23:00:02and provide list of customers with their
23:00:05total sales and their assigned segments.
23:00:08So we have here like customer
23:00:09segmentations. We have high value,
23:00:11medium value and low value. So with this
23:00:13comment we have the overall purpose of
23:00:15the query and then we have the inline
23:00:18comments like here. So it says it's
23:00:20calculate the total sales for each
23:00:21customer for the first CTE and now for
23:00:23the second CTE we have here a full
23:00:25description how the segment is built and
23:00:28this is built of course from the
23:00:30business rule of the customer segments.
23:00:32So it say the high values for total
23:00:34sales above like 100 and between and so
23:00:37on. Well this case win is really easy.
23:00:40So actually you can read it from the
23:00:42case win. But if you have like complex
23:00:44queries, it's really nice to have the
23:00:45full text of the case win and then add
23:00:48the main query. You can see here the
23:00:50final output and the inline comments. So
23:00:53as you can see it's really nice comments
23:00:55inside our codes. And now the next one
23:00:57we have like a document about the
23:00:59business rule. And I totally agree with
23:01:01the AI that the business rule is here
23:01:03about the customer segmentations. So we
23:01:06have here again very nice like short
23:01:08documentations about the business rules
23:01:10that we have and then we have another
23:01:13document about how the query is working.
23:01:15Well I think this is too much for small
23:01:18query. We can go and ask the shibility
23:01:20to make the documentation like shorter.
23:01:23So as you can see we have a full
23:01:24documentation about our query about our
23:01:27business rules and we have really nice
23:01:29comments in our code. All right. Now
23:01:31moving on to the next prompts. It is
23:01:33very important to improve the whole
23:01:35project, the whole database. So what
23:01:37we're going to do, we're going to go and
23:01:38take our DDL scripts and give it to the
23:01:41AI and start asking AI to optimize our
23:01:44database DDL. So here there is a lot of
23:01:47things that you can optimize with the
23:01:48database. So let's check this prompts.
23:01:50It's going to say the following SQL
23:01:52server DDL script has to be optimized
23:01:54and we ask the following task from the
23:01:56AI. The first one is to check the
23:01:58naming. So if you have a database where
23:02:00you have a lot of tables and columns and
23:02:02so on, you should be always working with
23:02:04a specific naming convention. So here
23:02:07just to make sure that the naming that
23:02:09you are using is correct. Then what is
23:02:11very important in DDLs is the data type.
23:02:14Data types plays very crucial role in
23:02:17optimizing your queries. So we are
23:02:19telling the AI to check the data types
23:02:21and whether they are optimized as well.
23:02:23And now the next point is about the data
23:02:26integrity. So if you are building a
23:02:27relational database, you will have a lot
23:02:29of primary keys and foreign keys and you
23:02:31can tell the AI to check the integrity
23:02:33of all those keys. The next point is
23:02:35about indexes. Here you can tell the AI
23:02:38to check the overall indexing that you
23:02:40are using in the DDL scripts just to
23:02:42make sure that you are not missing
23:02:43anything and as well to check whether we
23:02:46have duplicates. So it is really great
23:02:48check and the last check is that to
23:02:50check the normalizations of the table to
23:02:52check the data model and whether there
23:02:54is like any suggestions about splitting
23:02:56tables and normalizing tables or they
23:02:59are like some weird redundancy. Okay. So
23:03:01now what we're going to do we're going
23:03:02to let the chat activity to optimize the
23:03:05DDL of the sales DB. So now we have here
23:03:07the DDL of the customers employees
23:03:09orders and so on. And after running it
23:03:12we have the following results. So now we
23:03:14have here again the DDL but optimized
23:03:16one. And here the AI is adding comment
23:03:19about the changes. So here it added the
23:03:21auto incremental for the primary key.
23:03:23And here for example a check that is not
23:03:26a negative score and for the employees.
23:03:29Here another check to make sure that the
23:03:31birthday is not something in the future.
23:03:33So all those constraints in order to
23:03:35make sure that the quality of the table
23:03:37is good. And here for the gender it is
23:03:39restricting the valid values that could
23:03:42be used inside this column and many
23:03:44other stuff. And at the end we have like
23:03:47the key changes. So about the naming
23:03:49it's saying that we have to stick with
23:03:51one naming convention. So here it did
23:03:53understand that we are using the bascal
23:03:55case and for those two columns we have
23:03:57an issue like for example this product
23:03:59it should called product name. And for
23:04:01the data types I don't want to go in all
23:04:02details. So here for example it says
23:04:04don't use the int use a decimal for the
23:04:06price and sales for the integrity saying
23:04:09go and add foreign keys. I think for the
23:04:11orders we don't have any foreign keys
23:04:14that is used in the DDL. So the sht did
23:04:16go and add all the foreign keys in the
23:04:19DDL. So that was good. And now about the
23:04:22indexing it says since we have primary
23:04:24keys we will get automatically the
23:04:26clustered indexing and the foreign keys
23:04:28should get as well an index in order to
23:04:31improve the queries and so on. So as you
23:04:33can see there is a lot of optimizations
23:04:35that could be done in our DDL. So now if
23:04:38you are working on the project and you
23:04:39have a DDL go ask the AI what could we
23:04:42optimize I'm sure you will find
23:04:44something and this is very critical
23:04:46because having a solid and optimized DDL
23:04:49improves of course the speed of the
23:04:51queries. All right so now we come to
23:04:53very useful use case of using AI for
23:04:55your SQL projects and that is by using
23:04:57AI to generate test data sets. It is
23:05:00always really nice to have small data
23:05:03sets in order to test the logic of your
23:05:05query. Sometimes you are building a
23:05:07logic that does not exist yet in your
23:05:09database and of course if you are not
23:05:11able to test the scenario that you are
23:05:13developing it can be really bad and it
23:05:15is always very painful process in order
23:05:17to generate a data sets for your code
23:05:19but of course now it is easier because
23:05:21we have the help of AI. So let's check
23:05:24the following prompt. It says I need the
23:05:26data sets for testing the following SQL
23:05:28server DDL. And now next we have to
23:05:30specify for the AI different tasks. The
23:05:32first one is we have to define the shape
23:05:35of the data sets. So how do you want the
23:05:37output? Do you want it as an insert
23:05:39statements or do you want it as an excel
23:05:41or a file and so on. Now the next
23:05:43specifications I would like always to
23:05:44have a data set that is realistic. So I
23:05:47would like to always to have a data set
23:05:48that is relevant and realistic not to
23:05:51get dummy word data. So again he's like
23:05:54only configurations about the data set.
23:05:56The next configuration is that I would
23:05:58like to have small data sets. Of course,
23:05:59you can go and specify for charge the
23:06:02exact size of your data sets. You can
23:06:04say I would like to have like 100,000
23:06:06rows or millions of rows and so on. So
23:06:08you can define the size that you want.
23:06:10For me, I would like to have like small
23:06:12data sets. And now what is very
23:06:14important that if you have multiple
23:06:16tables in your DDL and those table have
23:06:19primary keys and foreign keys, the data
23:06:21set should be correct. So the AI should
23:06:23generate keys that is joinable. So if
23:06:26you go and join data together, you will
23:06:28not get weird results. And of course,
23:06:30you can go and keep adding
23:06:31specifications whether you want to have
23:06:33nulls or no nulls inside your data set.
23:06:36So here for example, I'm saying don't
23:06:38introduce any null values. And of course
23:06:40at the end you have to go and give the
23:06:41DDL for the AI. It could be one table or
23:06:44the whole database. So you could
23:06:46generate a data set for one table or
23:06:48hundreds of tables. Okay. So now I'm
23:06:50asking the SHT to create test data sets
23:06:53for two tables. the employees and the
23:06:55orders. Let's check the results. So now
23:06:58we can see very small nice insert
23:07:00statements for the table employees. So
23:07:02we have over here like five employees
23:07:04with the different informations. And now
23:07:06for the table orders we have a lot of
23:07:08columns. So as you can see we have four
23:07:10orders. And what is very important is
23:07:12that the salesperson ID comes from the
23:07:15table employees. So as you can see we
23:07:16have two and one where we have it
23:07:18already in the employees. and the rest
23:07:20of the informations we have like here
23:07:22fake addresses and stuff. So with that
23:07:25we have a very nice test data sets in
23:07:27order to be inserted to our database to
23:07:29test our queries. Of course we can go
23:07:31and ask maybe to extend it maybe instead
23:07:33of only four orders we can go with 20
23:07:35orders and so on. So we can go and
23:07:37change the size of it and here we have
23:07:39some notes about the data itself. So it
23:07:42is really amazing we are now generating
23:07:44this data using our DLS. All right. So
23:07:46now we have the following query and of
23:07:48course we are using the SQL server and
23:07:50let's say that you are migrating from
23:07:51SQL server to MySQL. So let's ask
23:07:54Shajbet to convert my code to MySQL. All
23:07:57right. So after running it as we can see
23:07:59now we have the same query but in MySQL.
23:08:02So instead of the isnull we are using
23:08:04Kawalis and here we are using the
23:08:06concatenation instead of the plus
23:08:08operator and instead of the get date in
23:08:11MySQL we use the now function. And the
23:08:13last thing we are using here top 10 but
23:08:16in my scale we use limit 10. And here we
23:08:19have really nice explanation about the
23:08:21transition. So as you can see it is
23:08:23amazing and if you are working on
23:08:25companies and in projects this might
23:08:27happen that there is like decision to
23:08:29start migrating from one database to
23:08:31another database and then your project
23:08:33going to get a big task of migrating the
23:08:35data migrating the DDLs and the queries
23:08:38and everything and I really recommend
23:08:40using the shad in order to help with the
23:08:42migration otherwise this big task might
23:08:45take really long time. So as you can see
23:08:47this is really amazing how shad can
23:08:49improve the speed of your
23:08:54projects. Okay. Now in the next section
23:08:56I'm going to show you the prompts that
23:08:58you can use as a student or if you are
23:09:00learning any new programming language.
23:09:02Okay. So the first thing that you can do
23:09:04with Shajibet is that you can ask it to
23:09:06generate an SQL course. So you can ask
23:09:09the shajibet to guide you step by step
23:09:11in your journey learning any programming
23:09:14language and you want to do it
23:09:15completely onetoone with the AI. So
23:09:17first it is very important in creating a
23:09:20course is that to give enough context.
23:09:22So in this example it is very short I'm
23:09:24saying create an SQL course with a
23:09:26detailed road map and agenda. But of
23:09:29course you can go and give more
23:09:31specifications. You can tell about your
23:09:33current knowledge. You can specify which
23:09:35database type you would like to work
23:09:37with MySQL SQL server. So the more
23:09:39context and details you give for the AI,
23:09:42the better results you're going to get.
23:09:43And then you go and configure your
23:09:45course. So you can say for example start
23:09:47with SQL fundamentals and advance to
23:09:50complex topics. And as well we can say
23:09:52make it beginner friendly and it is
23:09:55important if it is the first time you
23:09:56are learning about the topic. And now we
23:09:58have to shape the focus of the course
23:10:00like I'm saying here include topics that
23:10:02is relevant for data analytics because
23:10:05SQL is widely used in different topics
23:10:07for data engineering data analytics and
23:10:10it's really important in each course to
23:10:12focus on use cases. So we are saying
23:10:14focus on real world data analytics use
23:10:17cases and scenarios and of course you
23:10:19can go and add more details about your
23:10:21course. Okay. So now I just asked the
23:10:23shivity in order to make this course. So
23:10:25now let's see the road map and the
23:10:27structure of our course. So let's start
23:10:29with the phase one with the SQL
23:10:31fundamentals. So it start with the basic
23:10:33select where and so on. Then the next
23:10:35section we are talking about order by
23:10:37group by and insert update delete. So
23:10:39the basic stuff. Now in the road map you
23:10:41get the phase two intermediate SQL. So
23:10:44here we are talking about inner joins
23:10:45few functions about the text the date
23:10:48and the case statements and views. And
23:10:50now to the phase three we have the
23:10:52advanced SQL for analytics. So we have
23:10:54the window functions, the CTE and data
23:10:57cleaning using the null functions and
23:10:59few transformations. Then we go to the
23:11:02phase number four. Here in your road map
23:11:04you start talking about real world use
23:11:06cases. And here you have like multiple
23:11:08projects. So as you can see this is
23:11:10really solid road map in order to learn
23:11:13SQL. And now in the next step what you
23:11:15can do you can start deep diving into
23:11:17each of those chapters until SQL to
23:11:19start okay with the phase number one
23:11:21with the week one to give more details.
23:11:24All right. So now the next one once you
23:11:25have the agenda and the road map
23:11:27learning the SQL now you can go and
23:11:29focus on specific chapter specific SQL
23:11:32concepts. So in this prompt we are
23:11:33saying the context first I want detailed
23:11:36explanation about SQL window functions
23:11:38and now after that we are specifying for
23:11:40the AI the exact structure of the
23:11:43explanation. So first it should explain
23:11:45what are the window functions and maybe
23:11:47as well to give an analogy in order to
23:11:49understand exactly what is window
23:11:51functions and after that it should
23:11:53explain why we need them and when to use
23:11:55the window functions. So once you
23:11:57understand the basics then you can start
23:11:59learning about the syntax of the window
23:12:01functions and it should provide as well
23:12:03few simple examples and at the end the
23:12:06AI should show you the best or the most
23:12:08frequently use cases used for the SQL
23:12:11window functions. So this is the pattern
23:12:13that I like in order to learn something
23:12:15new. All right. So now let's see how the
23:12:16AI going to explain the SQL window
23:12:18functions. So as you can see it start
23:12:19with the big title understanding SQL
23:12:21with the functions. So we have here a
23:12:24quick definition and then we have here
23:12:25an analogy and the analogy about like a
23:12:28teacher grading students. Well that's
23:12:31nice because we have the rank function.
23:12:33So you have here a nice analogy about
23:12:34the window function and then we
23:12:36understand why do we need the window
23:12:38functions. Well I totally agree in order
23:12:40to have row level details with the
23:12:42aggregations. So you can do aggregations
23:12:45while maintaining the raw level details
23:12:47and as well you can do complex
23:12:49calculations because you cannot do
23:12:50everything with a group I there's
23:12:52functions that only work with the window
23:12:54and then we have some explanation when
23:12:56to use them. So we see here for example
23:12:58the syntax of the window function. So it
23:13:01divided to a function partition order by
23:13:03over and here few explanation about
23:13:05that. Then we have few simple examples
23:13:08with queries. So explaining the
23:13:10different functions but not all of them.
23:13:12Of course, you can go and ask the
23:13:14schedule to extend the examples for all
23:13:17functions. And now we can see the top
23:13:19three use cases for the window
23:13:20functions. So we use it in order to rank
23:13:23the data and as well to build the
23:13:25running totals and the moving average.
23:13:27And at the end we have a summary. So as
23:13:29you can see we have wonderful
23:13:30explanation about the concept of the SQL
23:13:33window functions. Okay, moving on to the
23:13:35next one. And this one I use it very
23:13:36frequently in my projects. There is like
23:13:38in programming always different concepts
23:13:41that are very close to each others and
23:13:43sometimes it is confusing and naturally
23:13:45clear what are the big differences
23:13:47between them. So here I have for you a
23:13:49prompt in order to compare different SQL
23:13:51concepts. So now the prompt says I want
23:13:53to understand the differences between
23:13:55SQL window functions and the group by.
23:13:58So both of them are used usually to
23:14:00aggregate data in SQL and I would like
23:14:02to understand more what are the
23:14:03differences between them. So we define
23:14:05for the AI the following task. Explain
23:14:07the key differences between the two
23:14:09concepts and then it's really important
23:14:10to understand when to use what. So
23:14:13describe when to use each concept with
23:14:15examples and it's really nice to
23:14:17understand as well the advantages and
23:14:19the disadvantages of each concept and at
23:14:21the end you would like maybe to get a
23:14:23quick summarization about the
23:14:25differences between those two functions
23:14:26side by side in one table. Okay. So now
23:14:29let's see how the share GBD can compare
23:14:31those two concepts. So first we have
23:14:32really nice table in order to see the
23:14:34differences between those two. So for
23:14:36example the output granularity it says
23:14:38the wind function provides calculation
23:14:41at the rowle details where the group by
23:14:43provides aggregated results at the group
23:14:46level detail and if you are talking
23:14:47about the functions it allow ranking
23:14:49running total moving average and the
23:14:52group by it allows only the basic
23:14:53aggregations like sum average count. So
23:14:56this is really nice overview for the
23:14:57differences. Then we have when to use
23:15:00which concepts. So it's telling the
23:15:02window function it is used if you want
23:15:04role level details together with the
23:15:06aggregations and here you have like a
23:15:08nice example for the group by it says
23:15:11you can use it for example when
23:15:12summarizing data into categories like
23:15:15here grouping up the data by the region
23:15:17and then after that we have like pros
23:15:19and cons for each concept. So the
23:15:21advantage of the window function we get
23:15:23all the rows and for the group I it is
23:15:25like easier to understand and to use.
23:15:28For the disadvantage of the window
23:15:29function it is more complex. For the
23:15:31group I the disadvantage is it removes
23:15:34the details about the rows and at the
23:15:36end we have like sideby-side comparison
23:15:38between those two concepts. So as you
23:15:40can see we have really nice full
23:15:41detailed comparison between those two
23:15:43SQL concepts. Practicing SQL with the
23:15:46AI. Well, it is really not enough to
23:15:49just read about something or maybe to
23:15:51follow and watch a course in order to
23:15:54learn something. You have always to
23:15:55practice. And of course, it is really
23:15:57hard to find a materials in order to
23:15:59practice a new programming language. So,
23:16:02we can do it like this. We give a rule
23:16:04act as an SQL trainer and then a context
23:16:06where we say and help me practice SQL
23:16:09window functions and then we go and
23:16:11configure this training this practice by
23:16:13doing the following. We tell it to make
23:16:15it interactive practicing. So the AI
23:16:17provide a task and you give a solution.
23:16:20And what else is important is that it
23:16:21provides you a simple data set and of
23:16:24course you can specify which data set
23:16:26you want. Is it industrial data set or
23:16:28healthcare or anything you want and then
23:16:30we tell the AI give SQL task that
23:16:33gradually increase in difficulty. So we
23:16:35start with the basics until getting
23:16:37advanced tasks. And you can tell the AI
23:16:39to act as an SQL server and show the
23:16:42results of your query. So you would like
23:16:43to get as a result not only the correct
23:16:46solution or feedback you want to see the
23:16:48result of the query that you gives and
23:16:50then finally the AI should go and review
23:16:52your queries provide a feedback and
23:16:54suggest improvements okay so now let's
23:16:57start practicing I gave the prompt to
23:16:58shity and now we have simple data sets
23:17:02so it is very simple we have the sales
23:17:04ID employee region sales dates and
23:17:06amounts and then we have the first task
23:17:08so it says write a query to rank
23:17:11employees by their total sales. So here
23:17:14you have like an example output and now
23:17:16it says your turn. So the shad is
23:17:18waiting for your answer. Okay. So now I
23:17:21just prepared a query for it. Let's see
23:17:23what can happen once I post it. Oh no, I
23:17:26got some errors in the query. So let's
23:17:28see what we have. So it says error in
23:17:30the aggregations. You should use the
23:17:32amount instead of sales. And it says
23:17:35unnecessary partition by in the rank and
23:17:37so on. So let's check the correct query.
23:17:40So we have here the group pi and then we
23:17:42have to do the window function without
23:17:45using partition pi. So that was a
23:17:47mistake and the result of this query
23:17:49going to be this one. And here I have
23:17:51really nice feedback about the first
23:17:53task. So now it ask me about the next
23:17:55task. So I'm going to say yes. So now we
23:17:57have this task number two about the
23:17:59running total. We have a task and we
23:18:00have the data and we have now to write
23:18:03query in order to solve the task. So my
23:18:05friends it is nice right interactive and
23:18:08not only SQL you can go and practice any
23:18:10programming language. Now moving on to
23:18:12the last prompt you can use AI in order
23:18:14to prepare you for SQL interview. So
23:18:17let's say that you are invited to an
23:18:18interview and you would like to prepare
23:18:20yourself for it. So you can do a quick
23:18:22preparation together with the AI. So you
23:18:24can say the following act as interviewer
23:18:26and prepare me for SQL interview. And
23:18:29now you can go and configure the
23:18:30interview where you can say ask common
23:18:32SQL interview questions and make it
23:18:35interactive. So it provide a question
23:18:37and then wait for you to answer and then
23:18:39you can say gradually progress to
23:18:41advanced topics. So from basics to
23:18:43advanced and it is very important that
23:18:45it evaluates your answer and give you a
23:18:47feedback. So it is a really great way to
23:18:50prepare for interviews and I really
23:18:51recommended to do it and you can prepare
23:18:53yourself not only for an SQL interview,
23:18:55you can prepare yourself for an SQL
23:18:57exam. Okay. Okay. So now let's prepare
23:18:59for an ISQL interview. And here we have
23:19:01the first question. Shibility says what
23:19:03is the difference between where and
23:19:05having. So now it is waiting for an
23:19:07answer. We can say where filters data
23:19:12before
23:19:13aggregation and
23:19:16having filters data after aggregation.
23:19:21So let's check the answer. So here it is
23:19:23giving me an example of a very solid
23:19:26answer. But in general I have answered
23:19:28correctly. So it says the answer is
23:19:30correct. But the feedback says here
23:19:32maybe the interviewer like needs more
23:19:34details not only one sentence about the
23:19:36differences. So here it is like
23:19:38encouraging me to speak more and to give
23:19:40more details as an answer but still the
23:19:43answer is correct. So now let's go to
23:19:45the next question. What we have here can
23:19:47you explain the differences between
23:19:48inner join and left join. So I hope you
23:19:51know the answer but as you can see it is
23:19:53very interactive and nice and I think
23:19:55those questions are really relevant. So
23:19:56if I'm interviewing someone I'm going to
23:19:58go and ask this question. What is the
23:20:01difference between where and having and
23:20:02as well the differences between the
23:20:04joint types. So this is amazing right? I
23:20:07really recommend you if you have like an
23:20:08interview go and prepare yourself using
23:20:11shajbt and you can go and practice and
23:20:13prepare yourself before going to the
23:20:15interview. All right. So with that you
23:20:16have learned how I use AI in order to
23:20:18assist me while I'm coding using SQL.
23:20:21And now my friends we come to the most
23:20:23important chapter from the whole course.
23:20:25You have now learned a lot of things
23:20:27about SQL. A lot of advanced techniques,
23:20:29a lot of functions, how to transform
23:20:31data, how to aggregate data. But now
23:20:33what you have to do is to take
23:20:34everything and to apply it in SQL
23:20:37projects. And those projects are not
23:20:39only like easy projects. I bought
23:20:41projects for you that is very similar to
23:20:43the real project that I do in the
23:20:44industry. So you will not learn only
23:20:46like how to do project in SQL but as
23:20:49well what are the main steps and how we
23:20:51implement projects in real world. And
23:20:53here I have for you three projects data
23:20:55warehousing data exploration and
23:20:57advanced data analytics. We're going to
23:20:58start with the first one the data
23:21:00warehousing projects. This one can be
23:21:02amazing. So let's go and deep dive in
23:21:08that. All right my friends. So now if
23:21:10you want to do data analytics projects
23:21:12using SQL we have three different types.
23:21:14The first type of projects you can do
23:21:16data warehousing. It's all about how to
23:21:18organize, structure and prepare your
23:21:21data for data analyszis. It is the
23:21:23foundations of any data analytics
23:21:25projects. And in the next step, you can
23:21:27do exploratory data analyzes, EDA. And
23:21:30all what you have to do is to understand
23:21:31and cover insights about our data sets.
23:21:34In this kind of project, you can learn
23:21:36how to ask the right questions and how
23:21:38to find the answer using SQL by just
23:21:41using basic SQL skills. Now moving on to
23:21:43the last stage where you can do advanced
23:21:46analytics projects where you're going to
23:21:47use advanced SQL techniques in order to
23:21:50answer business questions like finding
23:21:52trends over time, comparing the
23:21:54performance, segmenting your data into
23:21:56different sections and as well generate
23:21:58reports for your stakeholders. So here
23:22:00you will be solving real business
23:22:02questions using advanced SQL techniques.
23:22:04Now what we're going to do, we're going
23:22:05to start with the first type of projects
23:22:07SQL data warehousing where you will gain
23:22:09the following skills. So first you will
23:22:11learn how to do ETL ELT processing using
23:22:14SQL in order to prepare the data. You
23:22:16will learn as well how to build data
23:22:17architecture, how to do data
23:22:19integrations where we're going to merge
23:22:21multiple sources together and as well
23:22:22how to do data load and data modeling.
23:22:25So if I got you interested, grab your
23:22:26coffee and let's jump to the
23:22:31projects. All right, my friends. So now
23:22:33before we deep dive into the tools and
23:22:35the cool stuff, we have first to have
23:22:37good understanding about what is exactly
23:22:40data warehouse why the companies try to
23:22:42build such a data management system. So
23:22:44now the question is what is a data
23:22:46warehouse? I will just use the
23:22:47definition of the father of the data
23:22:49warehouse bill in a data warehouse is
23:22:52subject-oriented integrated time variant
23:22:55and nonvolatile collection of data
23:22:57designed to support the management's
23:22:59decision-making process. Okay, I I know
23:23:01that might be confusing.
23:23:03Subject-oriented it means that the
23:23:04warehouses always focus on a business
23:23:06area like the sales, customers, finance
23:23:09and so on. Integrated because it goes
23:23:11and integrate multiple source systems.
23:23:13Usually you build a warehouse not only
23:23:15for one source but for multiple sources.
23:23:18Time variance it means you can keep
23:23:20historical data inside the data
23:23:22warehouse. Nonvolatile it means once the
23:23:24data enter the data warehouse it is not
23:23:26deleted or modified. So this is how
23:23:29build inmon defined data warehouse.
23:23:31Okay. So now I'm going to show you the
23:23:32scenario where your company don't have a
23:23:34real data management. So now let's say
23:23:36that you have one system and you have
23:23:38like one data analyst has to go to this
23:23:40system and start collecting and
23:23:42extracting the data and then he going to
23:23:44spend days and sometimes weeks
23:23:46transforming the raw data into something
23:23:48meaningful. Then once they have the
23:23:50reports they're going to go and share
23:23:52it. And this data analyst is sharing the
23:23:54report using an Excel. And then you have
23:23:56like another source of data and you have
23:23:58another data analyst that she is doing
23:24:00maybe the same steps collecting the data
23:24:02spending a lot of time transforming the
23:24:04data and then share at the end like a
23:24:06report and this time she is sharing the
23:24:08data using PowerPoint and a third system
23:24:10and the same story but this time he is
23:24:12sharing the data using maybe PowerBI. So
23:24:15now if the company works like this then
23:24:17there is a lot of issues. First this
23:24:19process it take two way long. I saw a
23:24:22lot of scenarios where sometimes it
23:24:23takes weeks and even months until the
23:24:26employee manually generating those
23:24:28reports. And of course, what can happen
23:24:29for the users? They are consuming
23:24:31multiple reports with multiple state of
23:24:34the data. One report is 40 days old,
23:24:36another one 10 days and a third one is
23:24:39like 5 days. So it's going to be really
23:24:41hard to make a real decision based on
23:24:43this structure. A manual process is
23:24:45always slow and stressful and the more
23:24:47employees you involved in the process
23:24:49the more you open the door for human
23:24:51errors and errors of course in reports
23:24:53leads to bad decisions and another issue
23:24:56of course is handling the big data. If
23:24:59one of your sources generating like
23:25:01massive amount of data then the data
23:25:03analyst going to struggle collecting the
23:25:05data and maybe in some scenarios it will
23:25:07not be anymore possible to get the data.
23:25:09So the whole process can breaks and you
23:25:11cannot generate anymore fresh data for
23:25:14specific reports. And one last very big
23:25:16issue with that. If one of your
23:25:17stakeholders asks for an integrated
23:25:20report from multiple sources, well good
23:25:23luck with that because merging all those
23:25:25data manually is very chaotic,
23:25:27time-conuming and full of risk. So this
23:25:29is just a picture. If a company is
23:25:31working without a proper data
23:25:33management, without a data leak, data
23:25:36warehouse, data lake houses. So in order
23:25:38to make real and good decisions, you
23:25:40need data management. So now let's talk
23:25:42about the scenario of a data warehouse.
23:25:44So the first thing that's going to
23:25:46happen is that you will not have your
23:25:48data team collecting manually the data.
23:25:51You're going to have a very important
23:25:52component called ETL. ETL stands for
23:25:55extract, transform and load. It is a
23:25:58process that you do in order to extract
23:26:00the data from the sources and then apply
23:26:02multiple transformations on those
23:26:04sources and at the end it loads the data
23:26:06to the data warehouse and this one going
23:26:08to be the single point of truth for
23:26:10analyzes and reporting and it is called
23:26:13data warehouse. So now what can happen
23:26:15all your reports going to be consuming
23:26:18this single point of truth. So with that
23:26:20you create your multiple reports and as
23:26:22well you can create integrated reports
23:26:25from multiple sources not only from one
23:26:27single source. So now by looking to the
23:26:29right side it looks already organized
23:26:31right and the whole process is
23:26:33completely automated. There is no more
23:26:35manual steps which of course it reduces
23:26:38the human error and as well it is pretty
23:26:40fast. So usually you can load the data
23:26:42from the sources until the reports in
23:26:45matter of hours or sometimes in minutes.
23:26:48So there is no need to wait like weeks
23:26:50and months in order to refresh anything.
23:26:52And of course the big advantage is that
23:26:54the data warehouse itself it is
23:26:56completely integrated. So that means it
23:26:58goes and bring all those sources
23:27:00together in one place which makes it
23:27:02really easier for reporting and not only
23:27:05integrated you can build in the data
23:27:07warehouse as well history. So we have
23:27:09now the possibility to access historical
23:27:11data and what is also amazing is that
23:27:14all those reports having the same data
23:27:16status. So all those reports can have
23:27:18the same status maybe sometimes one day
23:27:20old or something. And of course if you
23:27:22have a modern data warehouse in cloud
23:27:24platforms you can really easily handle
23:27:27any big data sources. So no need to
23:27:29panic if one of your sources is
23:27:31delivering massive amount of data. And
23:27:33of course in order to build the data
23:27:35warehouse you need different types of
23:27:36developers. So usually the one that
23:27:38builds the ETL component and the data
23:27:41warehouse is the data engineer. So they
23:27:44are the one that is accessing the
23:27:45sources, scripting the ATLs and building
23:27:48the database for the data warehouse. And
23:27:50now for the other part, the one that is
23:27:52responsible for that is the data
23:27:54analyst. They are the one that is
23:27:56consuming the data warehouse, building
23:27:58different data models and reports and
23:28:01sharing it with the stakeholders. So
23:28:03they are usually contacting the
23:28:04stakeholders, understanding the
23:28:06requirements and building multiple
23:28:08reports based on the data warehouse. So
23:28:10now if you have a look to those two
23:28:11scenarios, this is exactly why we need
23:28:14data management. Your data team is not
23:28:17wasting time and fighting with the data.
23:28:20They are now more organized and more
23:28:22focused and with like a data warehouse
23:28:24and you are delivering professional and
23:28:26fresh reports that your company can
23:28:29count on in order to make good and fast
23:28:32decisions. So this is why you need a
23:28:34data management like a data warehouse.
23:28:36Think about data warehouse as a busy
23:28:38restaurant. Every day different
23:28:39suppliers bring in fresh ingredients,
23:28:42vegetables, spices, meat, you name it.
23:28:44They don't just use it immediately and
23:28:46throw everything in one pot, right? They
23:28:48clean it, shop it, and organize
23:28:50everything and store each ingredients in
23:28:53the right place, fridge or freezer. So,
23:28:55this is the preparing phase. And when
23:28:57the order comes in, they quickly grab
23:28:59the prepared ingredients and create a
23:29:02perfect dish and then serve it to the
23:29:04customers of the restaurant. And this
23:29:06process is exactly like the data
23:29:07warehouse process. It is like the
23:29:09kitchen where the raw ingredients, your
23:29:11data are cleaned, sorted and stored. And
23:29:14when you need a report or analyzes, it
23:29:16is ready to serve up exactly like what
23:29:19you
23:29:22need. Okay. So now we're going to zoom
23:29:24in and focus on the component ETL. If
23:29:27you are building such a project, you're
23:29:28going to spend almost 90% just building
23:29:31this component, the ETL. So it is the
23:29:34core element of the data warehouse and I
23:29:36want you to have a clear understanding
23:29:38what is exactly an ETL. So our data
23:29:41exist in a source system. And now what
23:29:43we want to do is is to get our data from
23:29:45the source and move it to the target.
23:29:48Source and target could be like database
23:29:49tables. So now the first step that we
23:29:51have to do is to specify which data we
23:29:54have to load from the source. Of course
23:29:56we can say that we want to load
23:29:58everything but let's say that we are
23:29:59doing incremental loads. So we're going
23:30:01to go and specify a subset of the data
23:30:03from the source in order to prepare it
23:30:05and load it later to the target. So this
23:30:07step in the ATL process we call it
23:30:09extract. We are just identifying the
23:30:12data that we need. We pull it out and we
23:30:14don't change anything. It's going to be
23:30:16like one to one like the source system.
23:30:18So the extract has only one task to
23:30:20identify the data that we have to pull
23:30:23out from the source and to not change
23:30:25anything. So we will not manipulate the
23:30:26data at all. It can stay as it is. So
23:30:29this is the first step in the ETL
23:30:31process, the extract. Now moving on to
23:30:33the stage number two. We're going to
23:30:35take this extract data and we will do
23:30:37some manipulations, transformations and
23:30:40we're going to change the shape of those
23:30:42data. And this process is really heavy
23:30:45working. We can do a lot of stuff like
23:30:47data cleansing, data integration and a
23:30:49lot of formatting and data
23:30:51normalizations. So a lot of stuff we can
23:30:53do in this step. So this is the second
23:30:55step in the ETL process, the
23:30:56transformation. We're going to take the
23:30:58original data and reshape it, transform
23:31:01it into exactly the format that we need
23:31:04into a new format and shapes that we
23:31:06need for analyzes and reporting. Now,
23:31:08finally, we get to the last step in the
23:31:10ATL process. We have the load. So, in
23:31:12this step, we're going to take this new
23:31:14data and we're going to insert it into
23:31:16the target. So, it is very simple. We're
23:31:18going to take this prepared data from
23:31:20the transformation step and we're going
23:31:22to move it into its final destination,
23:31:24the target like for example data
23:31:26warehouse. So that's ETL in a nutshell.
23:31:29First extract the raw data, then
23:31:31transform it into something meaningful
23:31:33and finally load it to a target where
23:31:35it's going to make a difference. So
23:31:37that's it. This is what we mean with the
23:31:39ETL process. Now in real projects, we
23:31:41don't have like only source and targets.
23:31:43Our data architecture going to have like
23:31:45multiple layers depend on your design
23:31:48whether you are building a warehouse or
23:31:49a data lake or a data warehouse. And
23:31:52usually there are like different ways on
23:31:53how to load the data between all those
23:31:55layers. And in order now to load the
23:31:57data from one layer to another one there
23:31:59are like multiple ways on how to use the
23:32:01ATL process. So usually if you are
23:32:03loading the data from the source to the
23:32:05layer number one like only extract the
23:32:07data from the source and load it
23:32:09directly to the layer number one without
23:32:11doing any transformations because I want
23:32:13to see the data as it is in the first
23:32:16layer. And now between the layer number
23:32:17one and the layer number two you might
23:32:19go and use the full ETL. So we're going
23:32:22to extract from the layer one, transform
23:32:24it and then load it to the layer number
23:32:26two. So with that we are using the whole
23:32:28process the ATL. And now between layer
23:32:30two and layer three we can do only
23:32:32transformation and then load. So we
23:32:34don't have to deal with how to extract
23:32:36the data because it is maybe using the
23:32:38same technology and we are taking all
23:32:40data from layer 2 to layer three. So we
23:32:43transform the whole layer 2 and then
23:32:44load it to layer three. And now between
23:32:47three and four you can use only the LM.
23:32:49So maybe it's something like duplicating
23:32:51and replicating the data and then you
23:32:54are doing the transformation. So you
23:32:56load to the new layer and then transform
23:32:58it. Of course, this is not a real
23:32:59scenario. I'm just showing you that in
23:33:01order to move from source to a target,
23:33:03you don't have always to use a complete
23:33:06ETL. Depend on the design of your data
23:33:08architecture. You might use only few
23:33:10components from the ETL. Okay. So this
23:33:13is how ETL looks like in real projects.
23:33:16Okay. So now I would like to show you an
23:33:17overview of the different techniques and
23:33:19methods in the ETLs. We have wide range
23:33:22of possibilities where you have to make
23:33:24decisions on which one you want to apply
23:33:26to your projects. So let's start first
23:33:28with the extraction. The first thing
23:33:30that I want to show you is we have
23:33:31different methods of extraction. Either
23:33:33you are going to the source system and
23:33:35pulling the data from the source or the
23:33:37source system is pushing the data to the
23:33:40data warehouse. So those are the two
23:33:41main methods on how to extract data. And
23:33:44then we have in the extraction two
23:33:46types. We have a full extraction
23:33:48everything all the records from tables
23:33:50and every day we load all the data to
23:33:52the data warehouse or we make more
23:33:54smarter one where we say we're going to
23:33:55do an incremental extraction where every
23:33:57day we're going to identify only the new
23:34:00changing data. So we don't have to load
23:34:02the whole thing only the new data we go
23:34:04extract it and then load it to the data
23:34:06warehouse. And in data extraction we
23:34:08have different techniques. The first one
23:34:09is like manually where someone has to
23:34:11access a source system and extract the
23:34:14data manually or we connect ourselves to
23:34:16a database and we have then a query in
23:34:18order to extract the data or we have a
23:34:20file that we have to parse it to the
23:34:22data warehouse or another technique is
23:34:24to connect ourself to API and do their
23:34:27calls in order to extract the data or if
23:34:29the data is available in streaming like
23:34:31in CFKA we can do eventbased streaming
23:34:34in order to extract the data. Another
23:34:36way is to use the change data capture
23:34:39CDC is as well something very similar to
23:34:41streaming or another way is by using web
23:34:44scrabbing where you have a code that
23:34:46going to run and extract all the
23:34:47informations from the web. So those are
23:34:49the different techniques and types that
23:34:51we have in the extraction. Now if you
23:34:53are talking on the transformation there
23:34:55are wide range of different
23:34:57transformations that we can do on our
23:34:59data like for example doing data
23:35:01enrichment where we add values to our
23:35:03data sets or we do a data integration
23:35:05where we have multiple sources and we
23:35:07bring everything to one data model or we
23:35:09derive new columns based on already
23:35:12existing one. Another type of data
23:35:14transformations we have the data
23:35:15normalization. So the sources has values
23:35:18that are like a code and you go and map
23:35:20it to more friendly values for the
23:35:22analyzers which is more easier to
23:35:24understand and to use. Another
23:35:26transformations we have the business
23:35:28rules and logic depend on the business
23:35:30you can define different criterias in
23:35:32order to build like new columns. And
23:35:34what belongs to transformations is the
23:35:36data aggregation. So here we aggregate
23:35:38the data to a different granularity and
23:35:41then we have type of transformation
23:35:42called data cleansing. There are many
23:35:44different ways on how to clean our data.
23:35:47For example, removing the duplicates,
23:35:49doing data filtering, handling the
23:35:51missing data, handling invalid values or
23:35:54removing unwanted spaces, casting the
23:35:56data types and detecting the outliers
23:35:59and many more. So we have different
23:36:01types of data cleansing that we can do
23:36:03in our data warehouse and this is very
23:36:06important transformation. So as you can
23:36:07see we have different types of
23:36:09transformations that we can do in our
23:36:11data warehouse. Now moving on to the
23:36:13load. So what do we have over here? We
23:36:15have different processing types. So
23:36:17either we are doing patch processing or
23:36:20stream processing. Patch processing
23:36:22means we are loading the data warehouse
23:36:24in one big patch of data that's going to
23:36:26run and load the data warehouse. So it
23:36:29is only one time job in order to refresh
23:36:31the content of the data warehouse and as
23:36:33well the reports. So that means we are
23:36:35scheduling the data warehouse in order
23:36:37to load it in the day once or twice. And
23:36:39the other type we have the stream
23:36:41processing. So this means if there is
23:36:43like a change in the source system,
23:36:44we're going to process this change as
23:36:46soon as possible. So we're going to
23:36:48process it through all the layers of the
23:36:50data warehouse once something changes
23:36:52from the source system. So we are
23:36:53streaming the data in order to have real
23:36:56time data warehouse which is very
23:36:58challenging things to do in data
23:36:59warehousing. And if you are talking
23:37:01about the loads we have two methods
23:37:03either we are doing a full load or
23:37:05incremental load. It's the same thing as
23:37:07extraction right? So for the full load
23:37:09in databases there are like different
23:37:11methods on how to do it like for example
23:37:13we truncate and then insert that means
23:37:15we make the table completely empty and
23:37:17then we insert everything from the
23:37:19scratch or another one you are doing an
23:37:21update insert we call it upsert. So we
23:37:24can go and update all the records and
23:37:26then insert the new one and another way
23:37:28is to drop create and insert. So that
23:37:30means we drop the whole table and then
23:37:32we create it from scratch and then we
23:37:34insert the data. It is very similar to
23:37:36the truncate but here we are as well
23:37:38removing and dropping the whole table.
23:37:40So those are the different methods of
23:37:42full loads. The incremental load we can
23:37:44use as well the upserts. So update and
23:37:46insert. So we're going to do an update
23:37:48or insert statements to our tables. Or
23:37:50if the source is something like a log,
23:37:52we can do only insert. So we can go and
23:37:54append the data always to the table
23:37:56without having to update anything.
23:37:58Another way to do incremental load is to
23:38:00do a merge. And here it is very similar
23:38:02to the upsert but as well with a delete.
23:38:04So update, insert, delete. So those are
23:38:07the different methods on how to load the
23:38:09data to your tables. And one more thing
23:38:10in data warehousing, we have something
23:38:12called slowly changing dimensions. So
23:38:14here it's all about the historicizations
23:38:16of your table. And there are many
23:38:18different ways on how to handle the
23:38:20historiizations in your table. The first
23:38:22type is sedd0. We say there is
23:38:24notoriizations and nothing should be
23:38:26changed at all. So that means you are
23:38:28not going to update anything. The second
23:38:30one which is more famous, it is the sedd
23:38:33one. you are doing an overwrite. So that
23:38:35means you are updating the records with
23:38:38the new informations from the source
23:38:39system by overwriting the old value. So
23:38:41we are doing something like the upsert.
23:38:43So update and insert but you are losing
23:38:46of course history. Another one we have
23:38:47the sedd2 and here you want to add
23:38:49historiizations to your table. So what
23:38:51we do each change that we get from the
23:38:53source system that means we are
23:38:55inserting new records and we are not
23:38:57going to overwrite or delete the old
23:38:59data. we are just going to make it
23:39:01inactive and the new record going to be
23:39:03active one. So there are different
23:39:05methods on how to do historiizations as
23:39:07well while you are loading the data to
23:39:10the data warehouse. All right. So those
23:39:12are the different types and techniques
23:39:13that you might encounter in data
23:39:15management projects. So now what I'm
23:39:16going to show you quickly which of those
23:39:18types we will be using in our projects.
23:39:20So now if we are talking about the
23:39:21extraction over here we will be doing a
23:39:24pull extraction and about the full or
23:39:26incremental it's going to be a full
23:39:28extraction. And about the technique we
23:39:30are going to be parsing files to the
23:39:32data warehouse. And now about the data
23:39:35transformations. Well, this one we will
23:39:37cover everything all those types of
23:39:39transformations that I'm showing you now
23:39:41is going to be part of the project
23:39:43because I believe in each data project
23:39:45you will be facing those
23:39:46transformations. Now if you have a look
23:39:48to the load our project going to be
23:39:50patch processing and about the load
23:39:52methods we will be doing a full load
23:39:54since we have full extraction and it's
23:39:56going to be truncate and inserts. And
23:39:59now about the historiizations we will be
23:40:01doing the sedd one. So that means we
23:40:04will be updating the content of the data
23:40:06warehouse. So those are the different
23:40:07techniques and types that we will be
23:40:09using in our ETL process for this
23:40:11project. All right. So with that we have
23:40:13now clear understanding what is a data
23:40:15warehouse and we are done with the
23:40:17theory parts. So now the next step we're
23:40:19going to start with the projects. The
23:40:20first thing that we have to do is to
23:40:22prepare our environment to develop the
23:40:24projects. So let's start with
23:40:29that. All right. So now we go to the
23:40:32link in the description and from there
23:40:34we're going to go to the downloads and
23:40:35you can find all the materials of all
23:40:37courses and projects. But the one that
23:40:39we need now is the SQL data warehouse
23:40:41projects. So let's go to the link and
23:40:43here we have bunch of links that we need
23:40:45for the projects. But the most important
23:40:47one to get all data and files is this
23:40:49one download all project files. So let's
23:40:52go and do that. And after you do that
23:40:54you're going to get a zip file where you
23:40:56have there a lot of stuff. So let's go
23:40:58and extract it. And now inside it if you
23:41:00go over here you will find the
23:41:01repository structure from git. And the
23:41:04most important one here is the data
23:41:05sets. So you have two sources the CRM
23:41:08and the ARP. And in each one of them
23:41:10there are three CSV files. So those are
23:41:13the data set for the projects. For the
23:41:15other stuffs don't worry about it. We
23:41:17will be explaining that during the
23:41:18project. So go and get the data and put
23:41:21it somewhere at your PC where you don't
23:41:22lose it. Okay. So now what else do we
23:41:24have? We have here a link to the get
23:41:26repository. So this is the link to my
23:41:28repository that I have created through
23:41:30the projects. So you can go and access
23:41:32it. But don't worry about it. We're
23:41:33going to explain the whole structure
23:41:34during the projects and you will be
23:41:36creating your own repository. And as
23:41:38well we have the link to the notion.
23:41:40Here we are doing the project
23:41:42management. Here you're going to find
23:41:43the main steps the main phases of the
23:41:45SQL projects that we will do and as well
23:41:48all the task that we will be doing
23:41:49together during the projects. And now we
23:41:52have links to the project tools. So if
23:41:54you don't have it already go and
23:41:55download the SQL server express. So it's
23:41:58like a server that's going to run
23:41:59locally at your PC where your database
23:42:01going to live. Another one that you have
23:42:03to download is the SQL Server Management
23:42:05Studio. It is just a client in order to
23:42:07interact with the database and there
23:42:09we're going to run all our queries and
23:42:11then link to the GitHub and as well link
23:42:13to the draw AO if you don't have it
23:42:15already go and download it. It is free
23:42:17and amazing tool in order to draw
23:42:19diagrams. So through the projects we
23:42:21will be drawing data models the data
23:42:24architecture a data lineage. So a lot of
23:42:26stuff we'll be doing using this tool. So
23:42:29go and download it. And the last thing
23:42:30it is nice to have you have a link to
23:42:32the notion where you can go and create
23:42:34of course free accounts if you want to
23:42:36build the project plan and as well
23:42:38follow me by creating the project steps
23:42:41and the projects tasks. Okay. So that's
23:42:43all those are all the links for the
23:42:45projects. So go and download all those
23:42:47stuff create the accounts and once you
23:42:49are ready then we continue with the
23:42:54projects. All right. So now I hope that
23:42:56you have downloaded all the tools and
23:42:58created the accounts. Now it's time to
23:43:00move to very important step that almost
23:43:02all people skip while doing projects and
23:43:05that is by creating the project plan and
23:43:08for that we will be using the tool
23:43:10notion. Notion is of course a free tool
23:43:12and it can help you to organize your
23:43:14ideas, your plans and resources all in
23:43:17one place. I use it very intensively for
23:43:20my private projects like for example
23:43:21creating this course and I can tell you
23:43:23creating a project plan is the key to
23:43:25success. Creating a data warehouse
23:43:27project is usually very complex. And
23:43:29according to Gartner reports, over 50%
23:43:32of data warehouse projects fail. In my
23:43:35opinion about any complex project, the
23:43:37key to success is to have a clear
23:43:40project plan. So now at this phase of
23:43:41the project, we're going to go and
23:43:43create a rough project plan because at
23:43:45the moment we don't have yet clear
23:43:47understanding about the data
23:43:49architecture. So let's go. Okay. So now
23:43:51let's create a new page and let's call
23:43:52it data warehouse projects. The first
23:43:54thing is that we have to go and create
23:43:56the main phases and stages of the
23:43:59projects and for that we need a table.
23:44:00So in order to do that hit slash and
23:44:03then type database in line and then
23:44:05let's go and call it something like data
23:44:08warehouse epics and we're going to go
23:44:10and hide it because I don't like it. And
23:44:12then on the table we can go and rename
23:44:14it like for example projects epics
23:44:17something like that. And now what we're
23:44:18going to do we're going to go and list
23:44:20all the big task of the project. So an
23:44:22epic is usually like a large task that
23:44:24needs a lot of efforts in order to solve
23:44:26it. So you can call it epics, stages,
23:44:29phases of the project, whatever you
23:44:30want. So we're going to go and list our
23:44:32project steps. So let's start with the
23:44:35requirements analyzes and then designing
23:44:38data
23:44:39architecture and another one we have the
23:44:42project
23:44:44initialization. So those are the three
23:44:46big task in the project first. And now
23:44:48what do we need? We need another table
23:44:50for the small chunks of the tasks, the
23:44:52subtasks and we're going to do the same
23:44:54thing. So we're going to go and hit
23:44:55slash and we're going to search for the
23:44:56table in line and we're going to do the
23:44:58same thing. So first we're going to call
23:45:00it data warehouse tasks and then we're
23:45:03going to hide it and over here we're
23:45:05going to rename it and say this is the
23:45:07project tasks. So now what we're going
23:45:09to do, we're going to go to the plus
23:45:10icon over here and then search for
23:45:12relation. This one over here with the
23:45:14arrow. And now we're going to search for
23:45:16the name of the first table. So we
23:45:18called it data warehouse eix. So let's
23:45:21go and click it and we're going to say
23:45:22as well two-way relation. So let's go
23:45:25and add the relation. So with that we
23:45:27got a field in the new table called data
23:45:29warehouse eix. This comes from this
23:45:31table and as well we have here data
23:45:33warehouse tasks that comes from the
23:45:35below table. So as you can see we have
23:45:37linked them together. Now what I'm going
23:45:38to do I'm going to take this to the left
23:45:40side and then what we're going to do
23:45:41we're going to go and select one of
23:45:43those epics. Like for example let's take
23:45:45design the data architecture. And now
23:45:47what we're going to do, we're going to
23:45:48go and break down this epic into
23:45:50multiple tasks. Like for example, choose
23:45:53data management approach. And then we
23:45:57have another task. What we're going to
23:45:58do, we're going to go and select as well
23:45:59the same epic. So maybe the next step is
23:46:02brainstorm and design the layers. And
23:46:06then let's go to another epic for
23:46:08example the project initialization. And
23:46:10we say over here for example create get
23:46:13repo prepare the structure. we can go
23:46:16and make another one in the same epic.
23:46:19Let's say we're going to go and create
23:46:20the database and the schemas. So, as you
23:46:23can see, I'm just defining the subtasks
23:46:25of those epics. So, now what we're going
23:46:27to do, we're going to go and add a
23:46:28checkbox in order to understand whether
23:46:30we have done the task or not. So, we go
23:46:32to the plus and search for check. We
23:46:34need a checkbox. And what we're going to
23:46:36do, we're going to make it really small
23:46:39like this. And with that, each time we
23:46:41are done with the task, we're going to
23:46:42go and click on it just to make sure
23:46:44that we have done the task. Now, there
23:46:46is one more thing that is not really
23:46:47working nice and that is here. We're
23:46:49going to have like a long list of tasks
23:46:51and it's really annoying. So, what we're
23:46:53going to do, we're going to go to the
23:46:54plus over here and let's search for roll
23:46:56up. So, let's go and select it. So, now
23:46:58what we're going to do, we have to go
23:46:59and select the relationship. It's going
23:47:01to be the data warehouse task. And after
23:47:03that, we're going to go to the property
23:47:04and make it as a checkbox. So, now as
23:47:06you can see in the first table, we are
23:47:08saying how many tasks is closed. But I
23:47:10don't want to show it like this. What we
23:47:12can do, we're going to go to the
23:47:12calculation and to the percent and then
23:47:15percent checked. And with that, we can
23:47:17see the progress of our project. And now
23:47:19instead of the numbers, we can have
23:47:21really nice bar. Great. So as well, we
23:47:23can go and give it a name like progress.
23:47:25So that's it. And we can go and hide the
23:47:27data warehouse tasks. And now with that,
23:47:29we have really nice progress bar for
23:47:31each epic. And if we close all the tasks
23:47:33of this epic, we can see that we have
23:47:35reached 100%. So this is the main
23:47:37structure. Now we can go and add some
23:47:39cosmetics and rename stuff in order to
23:47:41make things looks nicer. Like for
23:47:43example, if I go to the tasks over here,
23:47:45I can go and call it tasks and as well
23:47:48go and change the icon to something like
23:47:50this. And if you'd like to have an icon
23:47:52for all those epics, what you're going
23:47:54to do, we're going to go to the epic for
23:47:55example design data architecture. And
23:47:57then if you hover on top of the title,
23:47:59you can see add an icon. And you can go
23:48:01and pick any icon that you want. So for
23:48:04example, this one. And now as you can
23:48:05see, we have defined it here in the top.
23:48:07And the icon going to be as well in the
23:48:09below table. Okay. So now one more thing
23:48:11that we can do for the project tasks is
23:48:13that we can go and group them by the
23:48:15epics. So if you go to the three dots
23:48:17and then we go to groups and then we can
23:48:19group up by the epics. As you can see
23:48:22now we have like a section for each epic
23:48:24and you can go and sort the epics if you
23:48:26want. If you go over here sort then
23:48:28manual and you can go over here and
23:48:30start sorting the epics as you want. And
23:48:33with that you can expand and minimize
23:48:35each task. if you don't want to see
23:48:37always all tasks in one go. So this is
23:48:40really nice way in order to build like
23:48:41data management for your projects. Of
23:48:43course, in companies, we use
23:48:45professional tools in order to do
23:48:46projects like for example Gyra. But for
23:48:49private personal projects that I do, I
23:48:51always do it like this and I really
23:48:53recommend you to do it not only for this
23:48:55project, for any project that you are
23:48:56doing. Cuz if you see the whole project
23:48:58in one go, you can see the big picture
23:49:00and closing tasks and doing it like
23:49:02this. These small things going to makes
23:49:04you really satisfied and keeps you
23:49:06motivated to finish the whole project
23:49:08and makes you proud. Okay friends, so
23:49:10now I just went and added few icons, a
23:49:13renamed stuff and as well more tasks for
23:49:15each epic and this going to be our
23:49:17starting point in the project and once
23:49:19we have more informations we're going to
23:49:20go and add more details on how exactly
23:49:23we're going to build the data warehouse.
23:49:25So at the start we're going to go and
23:49:26analyze and understand the requirements
23:49:28and only after that we're going to start
23:49:30designing the data architecture and here
23:49:33we have three tasks. First we have to
23:49:35choose the data management approach and
23:49:37after that we're going to do
23:49:38brainstorming and designing the layers
23:49:40of the data warehouse and at the end
23:49:42we're going to go and draw a data
23:49:44architecture. So with that we have clear
23:49:46understanding how the data architecture
23:49:48looks like and after that we're going to
23:49:50go to the next epic where we're going to
23:49:52start preparing our projects. So once we
23:49:54have clear understanding of the data
23:49:56architecture the first task here is to
23:49:58go and create detailed project tasks. So
23:50:01we're going to go and add more AP and
23:50:02more tasks. And once we are done then
23:50:04we're going to go and create the naming
23:50:06conventions for the project just to make
23:50:08sure that we have rules and standards in
23:50:10the whole project. And next we're going
23:50:12to go and create a repository in the git
23:50:14and we're going to prepare as well the
23:50:15structure of the repository so that we
23:50:17always commit our work there. And then
23:50:19we're going to start with the first
23:50:21script where we're going to create a
23:50:22database and schemas. So my friends this
23:50:25is the initial plan for the project. Now
23:50:27let's start with the first epic. We have
23:50:29the requirements
23:50:34analyzes. Now analyzing the requirement,
23:50:36it is very important to understand which
23:50:39type of data warehouse you're going to
23:50:40go and build because there is like not
23:50:42only one standard on how to build it.
23:50:44And if you go blindly implementing the
23:50:46data warehouse, you might be doing a lot
23:50:48of stuff that is totally unnecessary and
23:50:50you will be burning a lot of time. So
23:50:53that's why you have to sit with the
23:50:54stakeholders with the department and
23:50:56understand what we exactly have to build
23:50:58and depend on the requirements you
23:51:00design the shape of the data warehouse.
23:51:03So now let's go and analyze the
23:51:04requirement of this project. Now the
23:51:06whole project is splitted into two main
23:51:08sections. The first section we have to
23:51:10go and build a data warehouse. So this
23:51:12is a data engineering task and we will
23:51:14go and develop ETLs and data warehouse.
23:51:17And once we have done that we have to go
23:51:20and build analytics and reporting
23:51:22business intelligence. So we're going to
23:51:24do data analyszis. But now first we will
23:51:26be focusing on the first part building
23:51:28the data warehouse. So what do we have
23:51:30here? The statement is very simple. It
23:51:32says develop a modern data warehouse
23:51:35using SQL server to consolidate sales
23:51:38data enabling analytical reporting and
23:51:41informed decision making. So this is the
23:51:43main statements and then we have
23:51:45specifications. The first one is about
23:51:47the data sources. It says import data
23:51:49from two source systems ERB and CRM and
23:51:52they are provided as CSV files. And now
23:51:55the second task is talking about the
23:51:57data quality. We have to clean and fix
23:51:59data quality issues before we do the
23:52:02data analyzers because let's be real
23:52:04there is no raw data that is perfect is
23:52:06always messy and we have to clean that
23:52:08up. Now the next task is talking about
23:52:10the integration. So it says we have to
23:52:12go and combine both of the sources into
23:52:15one single userfriendly data model that
23:52:18is designed for analytics and reporting.
23:52:21So that means we have to go and merge
23:52:22those two sources into one single data
23:52:25model. And now we have here another
23:52:26specifications. It says focus on the
23:52:29latest data sets. So there is no need
23:52:31for historiization. So that means we
23:52:33don't have to go and build histories in
23:52:34the database. And the final requirement
23:52:36is talking about the documentation. So
23:52:38it says provide clear documentations of
23:52:40the data model. So that means the last
23:52:42product of the data warehouse to support
23:52:45the business users and the analytical
23:52:47teams. So that means we have to generate
23:52:49a manual that's going to help the users
23:52:51that makes lives easier for the
23:52:53consumers of our data. So as you can see
23:52:55maybe this is very generic requirements
23:52:57but it has a lot of informations already
23:52:59for you. So it's saying that we have to
23:53:01use the platform SQL server. We have two
23:53:04source systems using the CSV files and
23:53:06it sounds that we really have a bad data
23:53:08quality in the sources and as well it
23:53:10wants us to focus on building completely
23:53:13new data model that is designed for
23:53:16reporting and it says we don't have to
23:53:18do historiization and it is expected
23:53:20from us to generate documentations of
23:53:22the system. So these are the
23:53:24requirements for the data engineering
23:53:26part where we're going to go and build a
23:53:28data warehouse that fulfill these
23:53:30requirements. All right. Right. So with
23:53:31that we have analyzed the requirements
23:53:33and as well we have closed the first
23:53:35easiest ebick. So we are done with this.
23:53:38Let's go and close it. And now let's
23:53:39open another one. Here we have to design
23:53:42the data architecture and the first task
23:53:44is to choose data management approach.
23:53:47So let's
23:53:50go. Now designing the data architecture
23:53:53it is exactly like building a house. So
23:53:56before construction starts, an
23:53:57architect's going to go and design a
23:54:00plan, a blueprint for the house. How the
23:54:02rooms will be connected, how to make the
23:54:04house functional, safe and wonderful.
23:54:07And without this blueprint from the
23:54:08architects, the builders might create
23:54:10something unstable, inefficient or maybe
23:54:13unlivable. The same goes for data
23:54:15projects. A data architect is like a
23:54:16house architecture. They design how your
23:54:19data will flow, integrate and be
23:54:21accessed. So as data architects we make
23:54:23sure that the data warehouse is not only
23:54:25functioning but also scalable and easy
23:54:27to maintain. And this is exactly what we
23:54:29will do now. We will play the role of
23:54:31the data architect and we will start
23:54:33brainstorming and designing the
23:54:35architecture of the data warehouse. So
23:54:37now I'm going to show you a sketch in
23:54:39order to understand what are the
23:54:40different approaches in order to design
23:54:42a data architecture. And this phase of
23:54:44the projects usually is very exciting
23:54:46for me because this is my main role in
23:54:48data projects. I am a data architect and
23:54:51I discuss a lot of different projects
23:54:53where we try to find out the best design
23:54:55for the projects. All right. So now
23:54:56let's
23:55:00go. Now the first step of building a
23:55:02data architecture is to make a very
23:55:04important decision to choose between
23:55:07four major types. The first approach is
23:55:09to build a data warehouse. It is very
23:55:11suitable if you have only structured
23:55:13data and your business want to build
23:55:15solid foundations for reporting and
23:55:18business intelligence. And another
23:55:20approach is to build a data leak. This
23:55:22one is way more flexible than a data
23:55:24warehouse where you can store not only
23:55:26structured data but as well semi and
23:55:29unstructured data. We usually use this
23:55:31approach if you have mixed types of data
23:55:33like database tables, logs, images,
23:55:35videos and your business want to focus
23:55:37not only on reporting but as well on
23:55:40advanced analytics or machine learning
23:55:42but it's not that organized like a data
23:55:44warehouse and data leaks if it's too
23:55:46much unorganized and turns into data
23:55:49swamp and this is where we need the next
23:55:51approach. So the next one we can go and
23:55:53build data lakehouse. So it is like a
23:55:56mix between data warehouse and data
23:55:58lake. You get the flexibility of having
23:56:00different types of data from the data
23:56:02lake but you still want to structure and
23:56:04organize your data like we do in the
23:56:06data warehouse. So you mix those two
23:56:07words into one and this is a very modern
23:56:10way on how to build that architecture
23:56:12and this is currently my favorite way of
23:56:14building data management system. Now the
23:56:16last and very recent approach is to
23:56:18build data mesh. So this is a little bit
23:56:20different. Instead of having centralized
23:56:22data management system the idea now in
23:56:24the data mesh is to make it
23:56:26decentralized. You cannot have like one
23:56:28centralized data management system
23:56:29because always if you say centralized
23:56:31then it means bottleneck. So instead you
23:56:34have multiple departments and multiple
23:56:35domains where each one of them is
23:56:37building a data product and sharing it
23:56:39with others. So now you have to go and
23:56:41pick one of those approaches and in this
23:56:43project we will be focusing on the data
23:56:45warehouse. So now the question is how to
23:56:47build the data warehouse. Well there is
23:56:49as well four different approaches on how
23:56:51to build it. The first one is the
23:56:53enimmon approach. So again you have your
23:56:55sources and the first layer you start
23:56:57with the staging where the row data is
23:56:59landing and then the next layer you
23:57:01organize your data in something called
23:57:03enterprise data warehouse where you go
23:57:06and model the data using the third
23:57:08normal format. It's about like how to
23:57:10structure and normalize your tables. So
23:57:12you are building a new integrated data
23:57:15model from the multiple sources. And
23:57:17then we go to the third layer. It's
23:57:18called the data marts where you go and
23:57:20take like small subset of the data
23:57:22warehouse and you design it in a way
23:57:25that is ready to be consumed from
23:57:27reporting and it focus on only one topic
23:57:29like for example the customers sales or
23:57:32products and after that you go and
23:57:34connect your BI tool like PowerBI or
23:57:36Tableau to the data marts. So with that
23:57:38you have three layers to prepare the
23:57:40data before reporting. Now moving on to
23:57:42the next one we have the Kimple
23:57:44approach. He says you know what building
23:57:46this enterprise data warehouse it is
23:57:48wasting a lot of time. So what we can do
23:57:50we can jump immediately from the stage
23:57:53layer to the final data because building
23:57:55this enterprise data warehouse it is a
23:57:58big struggle and usually waste a lot of
23:58:00time. So he always want you to focus and
23:58:02building the data ms quickly as
23:58:04possible. So it is faster approach than
23:58:06in but with the time you might get chaos
23:58:09in the data MS cuz you are not always
23:58:10focusing in the big picture and you
23:58:12might be repeating same transformations
23:58:14and integrations in different data ms.
23:58:17So there is like trade-off between the
23:58:19speed and consistent data warehouse. Now
23:58:22moving on to the third approach we have
23:58:23the data vault. So we still have the
23:58:25stage and the data marts but it says we
23:58:28still need this central data warehouse
23:58:30in the middle but this middle layer
23:58:32we're going to bring more standards and
23:58:34rules. So it tells you to split this
23:58:36middle layer into two layers the row
23:58:38vault and the business vault. In the row
23:58:41vault you have the original data but in
23:58:43the business vault you have all the
23:58:44business rules and transformations that
23:58:46prepares the data for the data marks. So
23:58:49that vault it is very similar to the
23:58:50inmon but it brings more standards and
23:58:53rules to the middle layer. Now I'm going
23:58:56to go and add a fourth one that I'm
23:58:58going to call it medallion architecture
23:59:00and this one is my favorite one because
23:59:02it is very easy to understand and to
23:59:04build. So it says you're going to go and
23:59:06build three layers bronze, silver and
23:59:08gold. The bronze layer it is very
23:59:10similar to the stage but we have
23:59:12understood with the time that the stage
23:59:14layer is very important because having
23:59:16the original data as it is it going to
23:59:18helps a lot by traceability and finding
23:59:20issues. Then the next layer we have the
23:59:22silver layer. It is where we do
23:59:24transformations data cleansing but we
23:59:26don't apply yet any business rules. Now
23:59:28moving on to the last layer the gold
23:59:30layer. It is as well very similar to the
23:59:32data marts but there we can build
23:59:34different type of objects not only for
23:59:36reporting but as well for machine
23:59:38learning for AI and for many different
23:59:41purposes. So they are like business
23:59:43ready objects that you want to share as
23:59:45a data products. So those are the four
23:59:47approaches that you can use in order to
23:59:49build a data warehouse. So again if you
23:59:52are building a data architecture you
23:59:54have to specify which approach you want
23:59:56to follow. So at the start we said we
23:59:57want to build a data warehouse and then
23:59:59we have to decide between those four
24:00:00approaches on how to build a data
24:00:02warehouse and in this project we will be
24:00:04using the medallion architecture. So
24:00:06this is a very important question that
24:00:08you have to answer as the first step of
24:00:10building a data architecture. All right.
24:00:12So with that we have decided on the
24:00:13approach. So we can go and mark it as
24:00:16done. The next step we're going to go
24:00:17and design the layers of the data
24:00:20warehouse.
24:00:24Now there is like not 100% standard way
24:00:28and rules for each layer. What you have
24:00:30to do as a data architects you have to
24:00:32define exactly what is the purpose of
24:00:35each layer. So we start with the bronze
24:00:36layer. So we say it's going to store row
24:00:39and unprocessed data as it is from the
24:00:41sources. And why we are doing that it is
24:00:43for traceability and debugging. If you
24:00:46have a layer where you are keeping the
24:00:47raw data, it is very important to have
24:00:49the data as it is from the sources
24:00:51because we can go always back to the
24:00:53bronze layer and investigate the data of
24:00:55specific source if something goes wrong.
24:00:58So the main objective is to have raw
24:01:01untouched data that's going to helps you
24:01:03as a data engineer by analyzing the root
24:01:05cause of issues. Now moving on to the
24:01:07server layer. It is the layer where
24:01:09we're going to store clean and
24:01:10standardized data and this is the place
24:01:13where we're going to do basic
24:01:14transformations in order to prepare the
24:01:16data for the final layer. Now for the go
24:01:19layer it's going to contain business
24:01:20ready data. So the main goal here is to
24:01:23provide data that could be consumed by
24:01:25business users and analysts in order to
24:01:27build reporting and analytics. So with
24:01:29that we have defined the main goal for
24:01:31each layer. Now next what I would like
24:01:33to do is to define the object types and
24:01:36since we are talking about a data
24:01:37warehouse in database we have here
24:01:39generally two types either a table or a
24:01:42view. So we are going for the bronze
24:01:43layer and the silver layer with tables
24:01:45but for the gold layer we are going with
24:01:47the views. So the best practice says for
24:01:50the last layer in your data warehouse
24:01:52make it virtual using views. It going to
24:01:54gives you a lot of dynamic and of course
24:01:56speed in order to build it since we
24:01:58don't have to make a load process for
24:01:59it. And now the next step is that we're
24:02:01going to go and define the load method.
24:02:03So in this project I have decided to go
24:02:04with the full load using the method of
24:02:06truncating and inserting. It is just
24:02:08faster and way easier. So we're going to
24:02:10say for the bronze layer we're going to
24:02:11go with the full load. And you have to
24:02:13specify as well for the silver layer as
24:02:15well. We're going to go with the full
24:02:16load. And of course for the views we
24:02:18don't need any load process. So each
24:02:20time you decide to go with tables you
24:02:22have to define the load methods with our
24:02:24full load, incremental loads and so on.
24:02:25Now we come to the very interesting part
24:02:27the data transformations. Now for the
24:02:29bronze layer, it is the easiest one
24:02:31about this topic because we don't have
24:02:33any transformations. We have to commit
24:02:35ourself to not touch the data, do not
24:02:38manipulate it, don't change anything. So
24:02:40it's going to stay as it is. If it comes
24:02:42bad, it's going to stay bad in the
24:02:43bronze layer. And now we come to the
24:02:45silver layer where we have the heavy
24:02:46lifting. As we committed in the
24:02:48objective, we have to make clean and
24:02:50standardized data. And for that we have
24:02:52different types of transformations. So
24:02:54we have to do data cleansing, data
24:02:56standardizations, data normalizations.
24:02:58We have to go and derive new columns and
24:03:01data enrichment. So there are like bunch
24:03:03of transformations that we have to do in
24:03:06order to prepare the data. Our focus
24:03:07here is to transform the data to make it
24:03:10clean and following standards and try to
24:03:12push all business transformations to the
24:03:14next layer. So that means in the god
24:03:16layer we will be focusing on business
24:03:19transformations that is needed for the
24:03:21consumers for the use cases. So what we
24:03:23do here we do data integrations between
24:03:25source system we do data aggregations we
24:03:28apply a lot of business logics and rules
24:03:30and we build a data model that is ready
24:03:32for for example business intelligence.
24:03:34So here we do a lot of business
24:03:36transformations and in the silver layer
24:03:38we do basic data transformations. So it
24:03:40is really here very important to make
24:03:43the fine decisions what type of
24:03:45transformations to be done in each layer
24:03:47and make sure that you commit to those
24:03:49rules. Now the next aspect is about the
24:03:51data modeling in the bronze layer and
24:03:53the silver layer. We will not break the
24:03:55data model that comes from the source
24:03:57system. So if the source system deliver
24:03:59five tables, we're going to have here
24:04:00like five tables and as well in the
24:04:02silver layer. We will not go and
24:04:04denormalize or normalize or like make
24:04:06something new, we're going to leave it
24:04:08exactly like it comes from the source
24:04:09system because what we're going to do,
24:04:11we're going to build the data model in
24:04:13the gold layer. And here you have to
24:04:14define which data model you want to
24:04:16follow. Are you following the star
24:04:18schema, the snowflake or are you just
24:04:20making aggregated objects? So you have
24:04:22to go and make a list of all data models
24:04:24types that you're going to follow in the
24:04:26gold layer. And at the end, what you can
24:04:27specify in each layer is the target
24:04:29audience. And this is of course very
24:04:31important decision. In the bronze layer,
24:04:33you don't want to give access to any end
24:04:35user. It is really important to make
24:04:36sure that only data engineers access the
24:04:39bronze layer. It makes no sense for data
24:04:41analysts or data scientists to go to the
24:04:44bad data because you have a better
24:04:46version for that in the silver layer. So
24:04:48in the silver layer of course the data
24:04:50engineers have to have an access to it
24:04:51and as well the data analysts and the
24:04:54data scientists and so on but still you
24:04:55don't give it to any business user that
24:04:58can't deal with the raw data model from
24:05:00the sources because for the business
24:05:02users you're going to get a better layer
24:05:04for them and that is the go layer. So in
24:05:06the gold layer it is suitable for the
24:05:09data analyst and as well the business
24:05:11users because usually the business users
24:05:13don't have a deep knowledge on the
24:05:14technicality of the server layer. So if
24:05:17you are designing multiple layers you
24:05:18have to discuss all those topics and
24:05:20make clear decision for each layer. All
24:05:23right my friends. So now before we
24:05:24proceed with the design I want to tell
24:05:26you a secret principle concept that each
24:05:29data architect must know and that is the
24:05:32separation of concerns. So what is that?
24:05:34As you are designing an architecture,
24:05:36you have to make sure to break down the
24:05:38complex system into smaller independent
24:05:41parts and each part is responsible for a
24:05:44specific task. And here comes the magic.
24:05:47The component of your architecture must
24:05:49not be duplicated. So you cannot have
24:05:51two parts are doing the same thing. So
24:05:53the idea here is to not mix everything.
24:05:56And this is one of the biggest mistakes
24:05:58in any big projects and I have shown
24:06:00that almost everywhere. So a good data
24:06:03architects follow this concept this
24:06:05principle. So for example if you are
24:06:07looking to our data architecture we have
24:06:09already done that. So we have defined
24:06:11unique set of tasks for each layer. So
24:06:14for example we have said in the server
24:06:16layer we do data cleansing but in the
24:06:18gold layer we do business
24:06:20transformations and with that you will
24:06:22not be allowing to do any business
24:06:24transformations. In the server layer and
24:06:26the same thing goes for the gold layer.
24:06:28You don't do in the gold layer any data
24:06:29cleansing. So each layer has its own
24:06:32unique tasks and the same thing goes for
24:06:34the bronze layer and the silver layer.
24:06:36You do not allow to load data from the
24:06:38source systems directly to the silver
24:06:40layer because we have decided the
24:06:42landing layer. The first layer is the
24:06:45bronze layer otherwise you will have
24:06:46like set of source systems that are
24:06:48loaded first to the bronze layer and
24:06:50another set is skipping the layer and
24:06:52going to the silver and with that we
24:06:54have overlapping. You are doing data
24:06:56ingestion in two different layers. So my
24:06:59friends, if you have this mindset,
24:07:01separation of concerns, I promise you,
24:07:03you're going to be a top data architect.
24:07:05So think about it. All right, my
24:07:06friends. So with that, we have designed
24:07:08the layers of the data warehouse. We can
24:07:10go ahead close it. The next step, we're
24:07:12going to go to DYO and start drawing the
24:07:14data
24:07:18architecture. So there is like no one
24:07:20standard on how to build a data
24:07:22architecture. You can add your style and
24:07:24the way that you want. So now the first
24:07:26thing that we have to show in that
24:07:27architecture is the different layers
24:07:29that we have. The first layer is the
24:07:31source system layer. So let's go and
24:07:33take a box like this and make it a
24:07:35little bit bigger. And I'm just going to
24:07:37go and make the design. So I'm going to
24:07:38remove the fill and make the line dotted
24:07:41one. And after that I'm going to go and
24:07:42change maybe the color to something like
24:07:44this gray. So now we have like a
24:07:46container for the first layer. And then
24:07:48we have to go and add like a text on top
24:07:50of it. So what I'm going to do, I'm
24:07:51going to take another box. Let's go and
24:07:53type inside it sources. And now I'm
24:07:55going to go and style it. So I'm going
24:07:56to go to the text and make it maybe 24.
24:07:59And then remove the lines like this.
24:08:02Make it a little bit smaller and put it
24:08:04on top. So this is the first layer. This
24:08:06is where the data come from. And then
24:08:08the data going to go inside a data
24:08:10warehouse. So I'm just going to go and
24:08:12duplicate this one. This one is the data
24:08:18warehouse. All right. So now the third
24:08:20layer what it going to be? It's going to
24:08:22be the consumers. who will be consuming
24:08:24this data warehouse. So I'm going to put
24:08:27another box and say this is the consume
24:08:30layer. Okay. So those are the three
24:08:32containers. Now inside the data
24:08:34warehouse, we have decided to build it
24:08:36using the medallion architecture. So
24:08:38we're going to have three layers inside
24:08:40the warehouse. So I'm going to take
24:08:41again another box. I'm going to call
24:08:44this one. This is the bronze layer. And
24:08:47now we have to go and put a design for
24:08:49it. So I'm going to go with this color
24:08:50over here. And then the text and maybe
24:08:52something like 20. And then make it a
24:08:55little bit smaller and just put it here.
24:08:57And beneath that we're going to have the
24:09:00component. So this is just a title of a
24:09:02container. So I'm going to have it like
24:09:03this. Remove the text from inside it.
24:09:06And remove the filling. So this
24:09:09container is for the bronze layer. Let's
24:09:11go and duplicate it for the next one. So
24:09:14this one going to be the silver layer.
24:09:17And of course, we can go and change the
24:09:19coloring to gray because it is silver.
24:09:21And as well the lines and remove the
24:09:23filling. Great. And now maybe I'm going
24:09:25to make the font as bold. All right. Now
24:09:28the third layer going to be the gold
24:09:31layer. And we have to go and pick a
24:09:33color for that. So style and here we
24:09:36have like something like yellow. The
24:09:38same thing for the container. I remove
24:09:40the filling. So with that we are showing
24:09:42now the different layers inside our data
24:09:44warehouse. Now those containers are
24:09:46empty. What we're going to do, we're
24:09:47going to go inside each one of them and
24:09:48start adding contents. So now in the
24:09:50sources, it is very important to make it
24:09:52clear what are the different types of
24:09:54source systems that you are connecting
24:09:56to the data warehouse because in real
24:09:58project there are like multiple types.
24:09:59You might have a database, API, files,
24:10:02cafka and here it's important to show
24:10:04those different types. In other projects
24:10:06we have folders and inside those folders
24:10:08we have CSV files. So now what you have
24:10:09to do we have to make it clear in this
24:10:11layer that the input for our project is
24:10:14CSV file. So it really depend how you
24:10:16want to show that. I'm going to go over
24:10:18here and say maybe folder and then I'm
24:10:20going to go and take the folder and put
24:10:21it here inside and then maybe search for
24:10:24file more results and go pick one of
24:10:26those icons. For example, I'm going to
24:10:28go with this one over here. So I'm going
24:10:30to make it smaller and add it on top of
24:10:32the folder. So with that we make it
24:10:34clear for everyone seeing the
24:10:35architecture that the sources is not a
24:10:38database is not an API it is a file
24:10:41inside the folder. So now very important
24:10:42here to show is the source systems. What
24:10:45are the sources that is involved in the
24:10:47project. So here what we're going to do
24:10:48we're going to go and give it a name.
24:10:49For example we have one source called
24:10:51CRM like this and maybe make the icon
24:10:54and we have another source called ERP.
24:10:56So we're going to go and duplicate it
24:10:58put it over here and then rename it ERP.
24:11:01So now it is for everyone clear. We have
24:11:02two sources for this project and the
24:11:04technology is used is simply a file. So
24:11:06now what we can do as well we can go and
24:11:08add some descriptions inside this box to
24:11:10make it more clear. So what I'm going to
24:11:12do, I'm going to take a line because I
24:11:14want to split the description from the
24:11:16icons something like this and make it
24:11:18gray. And then below it, we're going to
24:11:19go and add some text and we're going to
24:11:21say is CSV file. And the next point and
24:11:25we can say the interface is simply files
24:11:29in folder. And of course you can go and
24:11:31add any specifications and explanation
24:11:34about the sources. If it is a database,
24:11:36you can say the type of the database and
24:11:37so on. So that we made it in the data
24:11:39architecture clear what are the sources
24:11:41of our data warehouse. And now the next
24:11:43step what we're going to do we're going
24:11:44to go and design the content of the
24:11:46bronze silver and gold. So I'm going to
24:11:48start by adding like an icon in each
24:11:50container. It is to show about that we
24:11:52are talking about database. So what
24:11:54we're going to do we're going to go and
24:11:55search for database and then more
24:11:58result. More results. I'm going to go
24:12:00with this icon over here. So let's go
24:12:03and make it bigger. Something like this.
24:12:05Maybe change the color of dots. So,
24:12:07we're going to have the bronze and as
24:12:09well here the silver and the gold. So,
24:12:12now what we can do, we're going to go
24:12:13and add some arrows between those
24:12:15layers. So, we're going to go over here.
24:12:17So, we can go and search for arrow and
24:12:18maybe go and pick one of those. Let's go
24:12:21and put it here. And we can go and pick
24:12:23a color for that. Maybe something like
24:12:25this. And adjust it. So, now we're going
24:12:27to have this nice arrow between all the
24:12:29layers just to explain the direction of
24:12:31our architecture, right? So we can read
24:12:33it from left to right and as well
24:12:36between the go layer and the consume.
24:12:38Okay. So now what I'm going to do next
24:12:40we're going to go and add one statement
24:12:42about each layer the main objective. So
24:12:44let's go and grab a text and put it
24:12:46beneath the database and we're going to
24:12:48say for example for the bronze layer
24:12:49it's going to be the row data. Maybe
24:12:51make the text bigger so you are the row
24:12:54data. And then the next one in the
24:12:56silver you are clean standard data. And
24:13:01then the last one for the gold we can
24:13:03say business
24:13:05ready data. So with that we make the
24:13:08objective clear for each layer. Now
24:13:10below all those icons what we're going
24:13:11to do we're going to have a separator
24:13:13again like this. Make it like colored.
24:13:16And beneath it we're going to add the
24:13:17most important specifications of this
24:13:19layer. So let's go and add those
24:13:21separators in each layer. Okay. So now
24:13:24we need a text below it. Let's take this
24:13:26one here. So what is the object type of
24:13:29the bronze layer? That's going to be a
24:13:31table and we can go and add the load
24:13:34methods. We say this is patch
24:13:36processing. Since we are not doing
24:13:38streaming, we can say it is a full load.
24:13:41We are not doing incremental load. So we
24:13:43can say here trank and insert. And then
24:13:48we add one more section maybe about the
24:13:50transformations. So we can say no
24:13:53transformations. And one more about the
24:13:55data model. We're going to say none as
24:13:58is. And now what I'm going to do I'm
24:14:00going to go and add those specifications
24:14:01as well for the silver and gold. So here
24:14:03what we have discussed the object type
24:14:05the load process the
24:14:07transformations and whether we are
24:14:08breaking the data model or not the same
24:14:11thing for the gold layer. So I can say
24:14:13with that we have really nice layering
24:14:15of the data warehouse and what we are
24:14:16left is with the consumers over here you
24:14:19can go and add the different use cases
24:14:20and tools that can access your data
24:14:22warehouse like for example I'm adding
24:14:24here business intelligence and reporting
24:14:26maybe using PowerBI or Tableau or you
24:14:29can say you can access my data warehouse
24:14:30in order to do at analyzes using the SQL
24:14:33queries and this is what we're going to
24:14:35focus on the projects after we build the
24:14:37data warehouse and as well you can offer
24:14:39it for machine learning purposes and of
24:14:41course it It's really nice to add some
24:14:43icons in your architecture and usually I
24:14:44use this nice websites called flat icon.
24:14:47It has really amazing icons that you can
24:14:49go and use it in your architecture. Now,
24:14:51of course, we can go and keep adding
24:14:53icons and stuff to explain the data
24:14:55architecture and as well the system.
24:14:56Like for example, it is very important
24:14:58here to say which tools you are using in
24:15:00order to build this data warehouse. Is
24:15:02it in the cloud? Are using Azure datab
24:15:04bricks or maybe snowflake? So we're
24:15:06going to go and add for our project the
24:15:08icon of SQL server since we are building
24:15:11this data warehouse completely in the
24:15:13SQL server. So for now I'm really happy
24:15:14about it. As you can see we have now a
24:15:16plan right. All right guys so with that
24:15:18we have designed the data architecture
24:15:20using the doyo and with that we have
24:15:22done the last step in this epic and now
24:15:24with that we have a design for the data
24:15:26architecture and we can say we have
24:15:28closed this epic. Now let's go to the
24:15:30next one. We will start doing the first
24:15:32step to prepare our project. And the
24:15:34first task here is to create a detailed
24:15:36project
24:15:40plan. All right, my friends. So now it's
24:15:43clear for us that we have three layers
24:15:44and we have to go and build them. So
24:15:46that means our big epics going to be
24:15:48after the layers. So here I have added
24:15:50three more epics. So we have build
24:15:53bronze layer, build silver layer and
24:15:55gold layer. And after that I went and
24:15:57start defining all the different tasks
24:16:00that we have to follow in the projects.
24:16:02So at the start we will be analyzing
24:16:04then coding and after that we're going
24:16:06to go and do testing and once everything
24:16:08is ready we're going to go and document
24:16:10stuff and at the end we have to commit
24:16:12our work in the get repo. All those
24:16:14epics are following the same like
24:16:16pattern in the tasks. So as you can see
24:16:18now we have a very detailed project
24:16:20structure and now things are more
24:16:22cleared for us how we're going to build
24:16:24the data warehouse. So with that we are
24:16:26done from this task and now the next
24:16:29task we have to go and define the naming
24:16:31convention of the
24:16:36projects. All right. So now at this
24:16:38phase of the projects we usually define
24:16:40the naming conventions. So what is that?
24:16:42It is set of rules that you define for
24:16:45naming everything in the projects
24:16:47whether it is a database, schema,
24:16:49tables, stored procedures, folders,
24:16:52anything. And if you don't do that at
24:16:54the early phase of the projects, I
24:16:56promise you chaos can happen because
24:16:58what going to happen? You will have
24:16:59different developers in your projects
24:17:01and each of those developers have their
24:17:03own style of course. So one developer
24:17:05might name a table dimension customers
24:17:08where everything is lowerase and between
24:17:10them underscore and you have another
24:17:12developer creating another table called
24:17:14dimension products but using the camel
24:17:16case. So there is no separation between
24:17:18the words and the first character is
24:17:20capitalized and maybe another one using
24:17:22some prefixes like
24:17:24dim categories. So we have here like a
24:17:26shortcut of the dimension. So as you can
24:17:28see there are different designs and
24:17:30styles and if you leave the door open
24:17:32what can happen in the middle of the
24:17:33project you will notice okay everything
24:17:35looks inconsistent and you can define a
24:17:38big task to go and rename everything
24:17:41following a specific rule. So instead of
24:17:43wasting all this time at this phase you
24:17:46go and define the naming conventions and
24:17:48let's go and do that. So we usually
24:17:50start with a very important decision and
24:17:52that is which naming convention we going
24:17:55to follow in the whole project. So you
24:17:56have different cases like the camel
24:17:58case, the Pascal case, the kebab case,
24:18:02and the snake case. And for this
24:18:04project, we're going to go with the
24:18:05snake case where all the letters of a
24:18:08word going to be lowercased. And the
24:18:10separation between words going to be an
24:18:13underscore. For example, a table name
24:18:15called customer info. Customer is
24:18:17lowercased. Info is as well lowercased.
24:18:19And between them an underscore. So this
24:18:21is always the first thing that you have
24:18:23to decide for your data projects. The
24:18:25second thing is to decide the language.
24:18:28So for example, I work in Germany and
24:18:30there is always like a decision that we
24:18:31have to make whether we use Germany or
24:18:34English. So we have to decide for our
24:18:35project which language we're going to
24:18:37use. And a very important general rule
24:18:40is that avoid reserved words. So don't
24:18:43use a square reserved word as an object
24:18:45name like for example table. Don't give
24:18:47a table name as a table. So those are
24:18:50the general principles. So those are the
24:18:52general rules that you have to follow in
24:18:54the whole project. This applies for
24:18:56everything for tables, columns, stored
24:18:58procedures, any names that you are
24:19:00giving in your scripts. Now moving on,
24:19:02we have specifications for the table
24:19:04names. And here we have different set of
24:19:06rules for each layer. So here the rule
24:19:09says source system underscore entity. So
24:19:12we are saying all the tables in the
24:19:14bronze layer should start first with the
24:19:16source system name like for example CRM
24:19:19or ARB and after that we have an
24:19:21underscore and then at the end we have
24:19:23the entity name or the table name. So
24:19:25for example we have this table name CRM.
24:19:28So that means this table comes from the
24:19:31source system CRM and then we have the
24:19:34table name the entity name customer
24:19:35info. So this is the rule that we're
24:19:37going to follow in naming all tables in
24:19:39the bronze layer. Then moving on to the
24:19:41silver layer, it is exactly like the
24:19:43bronze because we are not going to
24:19:45rename anything. We are not going to
24:19:47build any new data model. So the naming
24:19:50going to be one one to one like the
24:19:52bronze. So it is exactly the same rules
24:19:54as the bronze. But if we go to the gold
24:19:56here, since we are building new data
24:19:58model, we have to go and rename things.
24:20:01And since as well we are integrating
24:20:02multiple sources together, we will not
24:20:05be using the source system name in the
24:20:07tables because inside one table you
24:20:09could have multiple sources. So the rule
24:20:11says all the names must be meaningful
24:20:13business aligned names for the tables
24:20:16starting with the category prefix. So
24:20:18here the rule says it start with
24:20:20category then underscore and then
24:20:22entity. Now what is category? We have in
24:20:25the code layer different types of
24:20:27tables. So we could build a table called
24:20:29a fact table. Another one could be a
24:20:31dimension. A third type could be an
24:20:33aggregation or a report. So we have
24:20:36different types of tables and we can
24:20:38specify those types as a prefix at the
24:20:41start. So for example we are saying here
24:20:44effect sales. So the category is fact
24:20:47and the table name called sales. And
24:20:49here I just made like a table with
24:20:51different type of patterns. So we could
24:20:53have a dimension. So we say it start
24:20:55with the dim underscore for example
24:20:57dimim customers or products. And then we
24:21:00have another type called fact table. So
24:21:02it start with fact underscore or
24:21:04aggregated table where we have the first
24:21:06three characters like aggregating the
24:21:08customers or the sales monthly. So as
24:21:11you can see as you are creating a naming
24:21:13convention you have first to make it
24:21:15clear what is the rule describe each
24:21:17part of the rule and start giving
24:21:19examples. So with that we make it clear
24:21:21for the whole team which names they
24:21:23should follow. So we talked here about
24:21:25the table naming convention. Then you
24:21:27can as well go and make naming
24:21:29convention for the columns. Like for
24:21:30example in the code layer we're going to
24:21:32go and have surrogate keys. So we can
24:21:34define it like this. The surrogate key
24:21:36should start with a table name and then
24:21:38underscore a key. Like for example we
24:21:40can call it customer key. It is a
24:21:43surrogate key in the dimension
24:21:45customers. The same thing for technical
24:21:47columns. As a data engineer, we might
24:21:49add our own columns to the tables that
24:21:51don't come from the source system. And
24:21:53those columns are the technical columns
24:21:55or sometimes we call them metadata
24:21:57columns. Now, in order to separate them
24:21:59from the original columns that comes
24:22:02from the source system, we can have like
24:22:04a prefix for that. Like for example, the
24:22:06rule says if you are building any
24:22:08technical or metadata columns, the
24:22:10column should start with DWH underscore
24:22:13and then the column name. For example,
24:22:15if you want the metadata load dates, we
24:22:17can have
24:22:18DWH load dates. So with that, if anyone
24:22:21sees that column starts with DWH, we
24:22:24understand this data comes from a data
24:22:26engineer. And we can keep adding rules
24:22:28like for example the store procedure
24:22:30over here. If you are making an ETL
24:22:32script, then it should start with the
24:22:34prefix load underscore and then the
24:22:36layer. For example, the store procedure
24:22:38that is responsible for loading the
24:22:39bronze going to be called load bronze.
24:22:43and for the silver load underscore
24:22:45silver. So those are currently the rules
24:22:47for the start procedure. So this is how
24:22:49I do it usually in my projects. All
24:22:50right my friends. So with that we have a
24:22:52solid naming conventions for our
24:22:54projects. So this is done and now the
24:22:56next step is that we're going to go to
24:22:57git and you will create a brand new
24:22:59repository and we're going to prepare
24:23:01its structure. So let's
24:23:06go. All right. Right. So now we come to
24:23:08as well important step in any projects
24:23:10and that's by creating the G repository.
24:23:12So if you are new to Git, don't worry
24:23:14about it. It is simpler than it sounds.
24:23:16So it's all about to have a safe place
24:23:18where you can put your codes that you
24:23:20are developing and you will have the
24:23:21possibility to track everything happens
24:23:23to the codes and as well you can use it
24:23:25in order to collaborate with your team
24:23:27and if something goes wrong you can
24:23:29always roll back. And the best part here
24:23:31once you are done with the project you
24:23:32can share your repository as a part of
24:23:34your portfolio and it is really amazing
24:23:36thing if you are applying for a job by
24:23:38showcasing your skills that you have
24:23:40built a data warehouse by using well
24:23:42doumented get repository. So now let's
24:23:44go and create the repository of the
24:23:46project. Now we are at the overview of
24:23:48our account. So the first thing that we
24:23:50have to do is to go to the repositories
24:23:52over here and then we're going to go to
24:23:54this green button and click on new. The
24:23:56first thing that we have to do is to
24:23:57give the repository name. So let's call
24:24:00it SQL data warehouse project and then
24:24:04here we can go and give it a
24:24:05description. So for example I'm saying
24:24:08building a modern data warehouse with
24:24:09SQL server. Now the next option whether
24:24:11you want to make it public and private.
24:24:13I'm going to leave it as a public and
24:24:15then let's go and add here a readme
24:24:17file. And then here about the license we
24:24:20can go over here and select the MIT. MIT
24:24:23license gives everyone the freedom of
24:24:25using and modifying your code. Okay. So
24:24:27I think I'm happy with the setup. Let's
24:24:29go and create the repository. And with
24:24:32that we have our brand new repository.
24:24:34Now the next step that I usually do is
24:24:36to create the structure of the
24:24:38repository. And usually I always follow
24:24:40the same patterns in any projects. So
24:24:42here we need few folders in order to put
24:24:44our files right. So what I usually do I
24:24:47go over here to add file create a new
24:24:49file and I start creating the structure
24:24:51over here. So the first thing is that we
24:24:53need data sets then slash and with that
24:24:56the repository going to understand this
24:24:58is a folder not a file and then you can
24:25:00go and add anything like here
24:25:03placeholder just an empty file this just
24:25:06going to help me to create the folders
24:25:07so let's go and commit so commit the
24:25:10changes and now if you go back to the
24:25:12main projects you can see now we have a
24:25:14folder called data sets so I'm going to
24:25:16go and keep creating stuff so I will go
24:25:19and create the documents placeholder
24:25:22commit the changes and then I'm going to
24:25:24go and create the scripts
24:25:28placeholder and the final one what I
24:25:31usually add is the
24:25:33tests something like
24:25:35this. So that as you can see now we have
24:25:38the main folders of our repository. Now
24:25:41what I usually do the next that I'm
24:25:42going to go and edit the main readme. So
24:25:45you can see it over here as well. So
24:25:46what we're going to do, we're going to
24:25:47go inside the readme and then we're
24:25:49going to go to the edit button here and
24:25:51we're going to start writing the main
24:25:53information about our project. This is
24:25:55really depend on your style. So you can
24:25:56go and add whatever you want. This is
24:25:59the main page of your repository. And
24:26:02now as you can see the file name here is
24:26:04MD. It stands for markdown. It is just
24:26:07an easy and friendly format in order to
24:26:10write a text. So if you have like
24:26:11documentations, you are writing a text.
24:26:13It is a really nice format in order to
24:26:15organize it, structure it and it is very
24:26:18friendly. So what I'm going to do at the
24:26:20start I'm going to give a few
24:26:21description about the project. So we
24:26:23have the main title and then we have
24:26:25like a welcome message and what this
24:26:27repository is about. And in the next
24:26:29section maybe we can start with the
24:26:30project requirements and then maybe at
24:26:33the end you can say a few words about
24:26:35the licensing and few words about you.
24:26:38So as you can see it's like the homepage
24:26:39of the project and the repository. So
24:26:41once you are done we're going to go and
24:26:43commit the changes. And now if you go to
24:26:45the main page of the repository you can
24:26:48see always the folder and files at the
24:26:50start and then below it we're going to
24:26:52see the informations from the readme. So
24:26:54again here we have the welcome statement
24:26:56and then the projects requirements and
24:26:59at the end we have the licensing and
24:27:00about me. So my friends that's it. We
24:27:03have now a repository and we have now
24:27:05the main structure of the project and
24:27:07through the projects as we are building
24:27:09the data warehouse we're going to go and
24:27:11commit all our work in this repository.
24:27:13Nice, right? All right. So with that we
24:27:16have now your repository ready and as we
24:27:18go in the project we will be adding
24:27:20stuff to it. So this step is done and
24:27:23now the last step finally we're going to
24:27:25go to the SQL server and we're going to
24:27:27write our first script where we're going
24:27:28to create a database and schemas.
24:27:35All right. Now the first step is we have
24:27:36to go and create a brand new database.
24:27:39So now in order to do that first we have
24:27:41to switch to the database master. So you
24:27:43can do it like this. Use master and
24:27:46semicolon. And if you go and execute it
24:27:48now we are switched to the master
24:27:49database. It is a system database in SQL
24:27:52server where you can go and create other
24:27:54databases. And you can see here from the
24:27:55toolbar that we are now logged into the
24:27:58master database. Now the next step we
24:28:00have to go and create our new database.
24:28:02So we're going to say create database
24:28:04and you can call it whatever you want.
24:28:06So I'm going to go with data warehouse
24:28:09semicolon. Let's go and execute it. And
24:28:12with that we have created our database.
24:28:14Let's go and check it from the object
24:28:15explorer. Let's go and refresh. And you
24:28:17can see our new data warehouse. This is
24:28:19our new database. Awesome. Right now to
24:28:22the next step we're going to go and
24:28:23switch to the new database. So we're
24:28:25going to say use data
24:28:28warehouse and semicolon. So let's go and
24:28:31switch to it. And you can see now we are
24:28:32logged into the data warehouse database.
24:28:35And now we can go and start building
24:28:36stuff inside this data warehouse. So now
24:28:39the first step that I usually do is I go
24:28:41and start creating the schemas. So what
24:28:44is schema? Think about it. It's like a
24:28:46folder or a container that helps you to
24:28:48keep things organized. So now as we
24:28:50decided in the architecture we have
24:28:52three layers, bronze, silver, gold. And
24:28:54now we're going to go and create for
24:28:55each layer a schema. So let's go and do
24:28:58that. We're going to start with the
24:28:59first one. Create schema. And the first
24:29:02one is bronze. So let's do it like this.
24:29:04And a semicolon. Let's go and create the
24:29:07first schema. Nice. So we have new
24:29:09schema. Let's go to our database. And
24:29:12then in order to check the schemas, we
24:29:13go to the security and then to the
24:29:15schemas over here. And as you can see,
24:29:17we have the bronze. And if you don't
24:29:19find it, you have to go and refresh the
24:29:21whole schemas. and then you will find
24:29:23the new schema. Great. So now we have
24:29:24the first schema. Now what we're going
24:29:26to do, we're going to go and create the
24:29:28others two. So I'm just going to go and
24:29:29duplicate it. So the next one going to
24:29:31be the silver and the third one going to
24:29:33be the gold. So let's go and execute
24:29:36those two together. We will get an error
24:29:38and that's because we are not having the
24:29:40go in between. So after each command,
24:29:43let's have a go. And now if I highlight
24:29:46the silver and gold and then execute, it
24:29:49will be working. the go in SQL it is
24:29:51like separator. So it tells SQL first
24:29:54execute completely the first command
24:29:56before go to the next one. So it is just
24:29:58separator. Now let's go to our schemas
24:30:00refresh and now we can see as well we
24:30:02have the gold and the silver. So with
24:30:04that we have now a database. We have the
24:30:06three layers and we can start developing
24:30:09each layer
24:30:14individually. Okay. So now let's go and
24:30:16commit our work in the git. So now since
24:30:18it is a script and code we're going to
24:30:20go to the folder scripts over here and
24:30:22then we're going to go and add a new
24:30:23file let's call it in it
24:30:26database.sql and now we're going to go
24:30:28and paste our code over here. So now I
24:30:30have done few modifications like for
24:30:33example before we create the database we
24:30:35have to check whether the database
24:30:37exists. This is an important step if you
24:30:39are recreating the database otherwise if
24:30:41you don't do that you will get an error
24:30:43where it's going to say the database
24:30:45already exists. So first it is checking
24:30:47whether the database exists then it
24:30:49drops it. I have added few comments like
24:30:52here we are saying creating the data
24:30:53warehouse creating the schemas and now
24:30:56we have a very important step. We have
24:30:58to go and add a header comment at the
24:31:00start of each script. To be honest after
24:31:023 months from now you will not be
24:31:04remembering all the details of this
24:31:06script. And adding a comment like this
24:31:08it is like a sticky note for you later
24:31:10once you visit this script again. And it
24:31:13is as well very important for the other
24:31:14developers in the team because each time
24:31:16you open the scripts the first question
24:31:18going to be what is the purpose of this
24:31:20script because if you or anyone in the
24:31:23team open the file the first question
24:31:25going to be what is the purpose of this
24:31:27scripts why we are doing this stuff. So
24:31:30as you can see here we have a comment
24:31:31saying this script creates a new data
24:31:33warehouse after checking if it already
24:31:36exists. If the database exists, it's
24:31:38going to drop it and recreate it. And
24:31:40additionally, it's going to go and
24:31:41create three schemas, bronze, silver,
24:31:43gold. So that it gives clarity what this
24:31:46script is about. And it makes everyone
24:31:48life easier. Now, the second reason why
24:31:50this is very important to add is that
24:31:52you can add warnings and especially for
24:31:55this script, it is very important to add
24:31:57these notes because if you run this
24:31:59script, what's going to happen? It's
24:32:00going to go and destroy the whole
24:32:02database. Imagine someone open this
24:32:04script and run it. Imagine an admin open
24:32:07this script and run it in your database.
24:32:09Everything going to be destroyed and all
24:32:11the data will be lost and this can be a
24:32:14disaster if you don't have any backup.
24:32:16So with that we have nice header
24:32:17comments and we have added few comments
24:32:19in our code and now we are ready to
24:32:22commit our code. So let's go and commit
24:32:24it. And now we have our script in the
24:32:27git as well. And of course if you are
24:32:29doing any modifications make sure to
24:32:31update the changes in the git. Okay my
24:32:33friends. So with that we have an empty
24:32:35database and schemas and we are done
24:32:38with this task and as well we are done
24:32:39with the whole epic. So we have
24:32:41completed the project initialization and
24:32:43now we're going to go to the interesting
24:32:45stuff. We will go and build the bronze
24:32:47layer. So now the first task is to
24:32:49analyze the source systems. So let's
24:32:54go. All right. So now the big question
24:32:56is how to build the bronze layer. So
24:32:59first thing first we do analyzing. As
24:33:01you are developing anything, you don't
24:33:03immediately start writing a code. So
24:33:05before we start coding the bronze layer,
24:33:07what we usually do is we have to
24:33:09understand the source system. So what I
24:33:11usually do, I make an interview with the
24:33:13source system experts and ask them many
24:33:15many questions in order to understand
24:33:18the nature of the source system that I'm
24:33:20connecting to the data warehouse. And
24:33:22once you know the source systems, then
24:33:24we can start coding. And the main focus
24:33:26here is to do the data ingestion. So
24:33:29that means we have to find a way on how
24:33:31to load the data from the source into
24:33:33the data warehouse. So it's like we are
24:33:35building a bridge between the source and
24:33:38our target system the data warehouse.
24:33:39And once we have the code ready, the
24:33:41next step is we have to do data
24:33:43validation. So here comes the quality
24:33:45control. It is very important in the
24:33:47bronze layer to check the data
24:33:48completeness. So that means we have to
24:33:50compare the number of records between
24:33:53the source system and the bronze layer
24:33:56just to make sure we are not losing any
24:33:57data in between. And another check that
24:33:59we will be doing is the schema checks
24:34:01and that's to make sure that the data is
24:34:03placed on the right position. And
24:34:05finally we don't have to forget about
24:34:07documentation and committing our work in
24:34:09the G. So this is the process that we're
24:34:11going to follow to build the bronze
24:34:16layer. All right my friends. So now
24:34:19before connecting any source systems to
24:34:20our data warehouse, we have to make very
24:34:22important step is to understand the
24:34:25sources. So how I usually do it, I set
24:34:27up a meeting with the source systems
24:34:29expert in order to interview them to ask
24:34:31them a lot of stuff about the source.
24:34:33And gaining this knowledge is very
24:34:35important because asking the right
24:34:36question will help you to design the
24:34:39correct scripts in order to extract the
24:34:41data and to avoid a lot of mistakes and
24:34:43challenges. And now I'm going to show
24:34:44you the most common questions that I
24:34:46usually ask before connecting anything.
24:34:48Okay. So we start first by understanding
24:34:50the business context and the ownership.
24:34:53So I would like to understand the story
24:34:54behind the data. I would like to
24:34:56understand who is responsible for the
24:34:57data, which IT departments and so on.
24:35:00And then it's nice to understand as well
24:35:02what business process it supports. Does
24:35:04it support the customer transactions,
24:35:06the supply chain, logistics or maybe
24:35:09finance reporting. So with that you can
24:35:11understand the importance of your data.
24:35:12And then I ask about the system and data
24:35:15documentation. So having documentations
24:35:17from the source is your learning
24:35:19materials about your data. And it's
24:35:20going to saves you a lot of time later
24:35:23when you are working and designing maybe
24:35:25new data models. And as well I would
24:35:27like always to understand the data model
24:35:29for the source system. And if they have
24:35:31like descriptions of the columns and the
24:35:33tables, it's going to be nice to have
24:35:34the data catalog. This can helps me a
24:35:36lot in the data warehouse. How I'm going
24:35:38to go and join the tables together. So
24:35:40with that you get a solid foundations
24:35:42about the business context, the
24:35:44processes and the ownership of the data.
24:35:46And now in the next step we're going to
24:35:47start talking about the technicality. So
24:35:49I would like to understand the
24:35:50architecture and as well the technology
24:35:52stack. So the first question that I
24:35:54usually ask is how the source system is
24:35:56storing the data. Do we have the data on
24:35:58the on-prem like in SQL server, Oracle
24:36:01or is it in the cloud like Azure, AWS
24:36:04and so on. And then once we understand
24:36:06that then we can discuss what are the
24:36:08integration capabilities like how I'm
24:36:11going to go and get the data. Do the
24:36:12source system offer APIs maybe cafka or
24:36:15they have only like file extractions or
24:36:18they're going to give you like a direct
24:36:19connection to the database. So once you
24:36:21understand the technology that you're
24:36:23going to use in order to extract the
24:36:24data then we're going to deep dive into
24:36:26more technical questions and here we're
24:36:28going to understand how to extract the
24:36:30data from the source system and then
24:36:32load it into the data warehouse. So the
24:36:34first things that we have to discuss
24:36:35with the experts can we do an
24:36:37incremental load or a full load and then
24:36:40after that we're going to discuss the
24:36:41data scope the historicizations do we
24:36:44need all data do we need only maybe 10
24:36:46years of the data are there histories
24:36:48already in the source system or should
24:36:50we build it in the data warehouse and so
24:36:52on and then we're going to go and
24:36:54discuss what is the expected size of the
24:36:57extracts are we talking here about
24:36:58megabytes gigabytes terabytes and this
24:37:01is very important to understand whether
24:37:03we have the right tools and platform to
24:37:06connect that source system and then I
24:37:08try to understand whether there are any
24:37:09data volume limitations like if you have
24:37:11some old source systems they might
24:37:13struggle a lot with performance and so
24:37:15on. So if you have like an ETL that is
24:37:17extracting large amount of data you
24:37:19might bring the performance down of the
24:37:21source system. So that's why you have to
24:37:23try to understand whether there are any
24:37:25limitations for your extracts and as
24:37:27well other aspects that might impact the
24:37:30performance of the source system. This
24:37:31is very important. If they give you an
24:37:33access to the database, you have to be
24:37:35responsible that you are not bringing
24:37:37the performance of the database down.
24:37:39And of course, very important question
24:37:40is to ask about the authentication and
24:37:43the authorization like how you going to
24:37:44go and access the data in the source
24:37:46system. Do you need any tokens, keys,
24:37:48password and so on. So those are the
24:37:50questions that you have to ask if you
24:37:52are connecting a new source system to
24:37:54the data warehouse. And once you have
24:37:56the answers for those questions, you can
24:37:58proceed with the next steps to connect
24:38:00the sources to the data warehouse. All
24:38:02right, my friends. So with that, you
24:38:04have learned how to analyze a new source
24:38:06systems that you want to connect to your
24:38:08data warehouse. So this step is done and
24:38:10now we're going to go back to coding
24:38:12where we're going to write scripts in
24:38:13order to do the data ingestion from the
24:38:16CSV files to the pros
24:38:20layer. And let's have a quick look again
24:38:23to our bronze layer specifications. So
24:38:25we just have to load the data from the
24:38:27sources to the data warehouse. We're
24:38:29going to build tables in the bronze
24:38:31layer. We are doing a full load. So that
24:38:33means we are truncating and then
24:38:34inserting the data. There will be no
24:38:36data transformations at all in the
24:38:38bronze layer. And as well we will not be
24:38:40creating any data model. So this is the
24:38:42specifications of the bronze layer. All
24:38:45right. Right now in order to create the
24:38:46DDL script for the bronze layer creating
24:38:48the tables of the bronze we have to
24:38:50understand the metadata the structure
24:38:52the schema of the incoming data and here
24:38:55either you ask the technical experts
24:38:57from the source system about these
24:38:58informations or you can go and explore
24:39:01the incoming data and try to define the
24:39:04structure of your tables. So now what
24:39:05we're going to do we're going to start
24:39:06with the first source system the CRM. So
24:39:09let's go inside it and we're going to
24:39:11start with the first table the customer
24:39:12info. Now if you open the file and check
24:39:14the data inside it, you see we have a
24:39:16header information and that is very good
24:39:18because now we have the names of the
24:39:21columns that are coming from the source
24:39:22and from the content you can define of
24:39:24course the data types. So let's go and
24:39:26do that. First we're going to say create
24:39:28table and then we have to define the
24:39:31layer. It's going to be the bronze. And
24:39:32now very important we have to follow the
24:39:34naming convention. So we start with the
24:39:36name of the source system. It is CRM
24:39:38underscore and then after that the table
24:39:40name from the source system. So it's
24:39:42going to be the cost underscore info. So
24:39:45this is the name of our first table in
24:39:47the bronze layer. Then the next step we
24:39:48have to go and define of course the
24:39:50columns. And here again the column names
24:39:52in the bronze layer going to be one to
24:39:54one exactly like the source system. So
24:39:56the first one going to be the ID and I
24:39:59will go with the data type integer. Then
24:40:01the next one going to be the key invar
24:40:04char and the length I will go with 50.
24:40:10[Music]
24:40:13And the last one going to be the create
24:40:16date. It's going to be date. So with
24:40:18that we have covered all the columns
24:40:20available from the source system. So
24:40:22let's go and check. And yes the last one
24:40:24is the create date. So that's it for the
24:40:26first table. Now a semicolon of course
24:40:28at the end. Let's go and execute it. And
24:40:30now we're going to go to the object
24:40:32explorer over here. Refresh. And we can
24:40:34see the first table inside our data
24:40:36warehouse. Amazing right? So now next
24:40:38what you have to do is to go and create
24:40:41a DDL statement for each file for those
24:40:44two systems. So for the CRM we need
24:40:46three DDLs and as well for the other
24:40:49system the ERP we have as well to create
24:40:52three DDLs for the three files. So at
24:40:54the end we're going to have in the
24:40:55bronze layer six tables six DTLs. So now
24:40:59pause the video go create those DDLs. I
24:41:01will be doing the same as well and we
24:41:03will see you soon.
24:41:06[Music]
24:41:09All right. So now I hope you have
24:41:10created all those details. I'm going to
24:41:12show you what I have just created. So
24:41:13the second table in the source CRM we
24:41:16have the product informations and the
24:41:18third one is the sales details. Then we
24:41:21go to the second system and here we make
24:41:23sure that we are following the naming
24:41:24convention. So first the source system
24:41:26ERB and then the table name. So the
24:41:29second system was really easy. You can
24:41:31see we have only here like two columns
24:41:32and for the customers like only three
24:41:35and for the categories only four
24:41:36columns. All right. So after defining
24:41:38those stuff of course we have to go and
24:41:40execute them. So let's go and do that.
24:41:43And then we go to the object explorer
24:41:44over here. Refresh the tables. And with
24:41:47that you can see we have six empty
24:41:49tables in the bronze layer. And with
24:41:51that we have all the tables from the two
24:41:53source systems inside our database. But
24:41:56still we don't have any data. And you
24:41:58can see our naming convention is really
24:41:59nice. You see the first three tables
24:42:01comes from the CRM source system and
24:42:04then the other three comes from the ERB.
24:42:06So we can see in the bronze layer the
24:42:08things are really splitted nicely and
24:42:10you can identify quickly which table
24:42:12belong to which source system. Now there
24:42:14is something else that I usually add to
24:42:16the DDL script is to check whether the
24:42:19table exists before creating. So for
24:42:21example, let's say that you are renaming
24:42:23or you would like to change the data
24:42:25type of specific field. If you just go
24:42:27and run this query, you will get an
24:42:29error because the database going to say
24:42:31we have already this table. So in other
24:42:33databases you can say create or replace
24:42:35table. But in the SQL server you have to
24:42:37go and build a TSQL logic. So it is very
24:42:40simple. First we have to go and check
24:42:41whether the object exists in the
24:42:43database. So we say if object ID and
24:42:47then we have to go and specify the table
24:42:48name. So let's go and copy the whole
24:42:51thing over here and make sure you get
24:42:54exactly the same name as the table name.
24:42:56So there you see like space. I'm just
24:42:57going to go and remove it. And then
24:42:59we're going to go and define the object
24:43:00type. So it's going to be the U. It
24:43:02stands for user. It is the user defined
24:43:04tables. So if this table is not null. So
24:43:08that means the database did find this
24:43:10object in the database. So what's going
24:43:12to happen? We say go and drop the table.
24:43:16So the whole thing again and semicolon.
24:43:18So again if the table exist in the
24:43:21database is not null then go and drop
24:43:23the table and after that go and create
24:43:26it. So now if you go and highlight the
24:43:28whole thing and then execute it it will
24:43:30be working. So first drop the table if
24:43:33it exist then go and create the table
24:43:36from scratch. Now what you have to do is
24:43:37to go and add this check before creating
24:43:40any table inside our database. So it's
24:43:42going to be the same thing for the next
24:43:44table and so on. I went and added all
24:43:47those checks for each table and what can
24:43:50happen if I go and execute the whole
24:43:52thing it going to work. So with that I'm
24:43:54recreating all the tables in the bronze
24:43:57layer from the
24:44:01scratch. Now the methods that we're
24:44:03going to use in order to load the data
24:44:05from the source to the data warehouse is
24:44:07the bulk inserts. Pulk insert is a
24:44:09method of loading massive amount of data
24:44:12very quickly from files like CSV files
24:44:15or maybe a text file directly into a
24:44:18database. It is not like the classical
24:44:20normal inserts where it's going to go
24:44:22and insert the data row by row but
24:44:24instead the bulk insert is one operation
24:44:27that's going to load all the data in one
24:44:29go into the database and that's what
24:44:31makes it very fast. So let's go and use
24:44:33this method. Okay. Okay, so now let's
24:44:34start writing the script in order to
24:44:36load the first table in the source CRM.
24:44:39So we're going to go and load the table
24:44:41customer info from the CSV file to the
24:44:44database table. So the syntax is very
24:44:46simple. We're going to start with saying
24:44:48bulk insert. So with that SQL understand
24:44:50we are doing not a normal insert, we are
24:44:52doing a bulk insert and then we have to
24:44:54go and specify the table name. So it is
24:44:57bronze dot CRM cost info. So now we have
24:45:01to specify the full location of the file
24:45:04that we are trying to load in this
24:45:06table. So now what we have to do is to
24:45:08go and get the path where the file is
24:45:10stored. So I'm going to go and copy the
24:45:12whole path and then add it to the bulk
24:45:14insert exactly like where the data
24:45:17exists. So for me it is in CSQL data
24:45:20warehouse project data set in the source
24:45:23CRM. And then I have to specify the file
24:45:25name. So it's going to be like cost
24:45:27info. CSV. You have to get it exactly
24:45:31like the path of your files otherwise it
24:45:33will not be working. So after the path
24:45:35now we come to the with clause. Now we
24:45:38have to tell the SQL server how to
24:45:39handle our file. So here comes the
24:45:42specifications. There is a lot of stuff
24:45:44that we can define. So let's start with
24:45:46the very important one is the row
24:45:48header. Now if you check the content of
24:45:50our files you can see always the first
24:45:53row includes the header information of
24:45:55the file. So those informations are
24:45:58actually not the data. It's just the
24:46:00column names. The actual data starts
24:46:02from the second row and we have to tell
24:46:05the database about this information. So
24:46:07we're going to say first row is actually
24:46:10the second row. So with that we are
24:46:12telling SQL to skip the first row in the
24:46:15file. We don't need to load those
24:46:16informations because we have already
24:46:18defined the structure of our table. So
24:46:21this is the first specifications. The
24:46:23next one which is as well very important
24:46:25in loading any CSV file is the separator
24:46:28between fields. The delimiter between
24:46:30fields. So it's really depend on the
24:46:32file structure that you are getting from
24:46:33the source. As you can see all those
24:46:35values are splitted with a comma and we
24:46:38call this comma as a file separator or a
24:46:41delimter and I saw a lot of different
24:46:42CSVs like sometime they use a semicolon
24:46:45or a pipe or special character like a
24:46:47hash and so on. So you have to
24:46:49understand how the values are splitted
24:46:51and in this file it's splitted by the
24:46:53comma and we have to tell SQL about this
24:46:55info. It's very important. So we're
24:46:57going to say filled terminator and then
24:46:59we're going to say it is the comma and
24:47:02basically those two informations are
24:47:03very important for SQL in order to be
24:47:05able to read your CSV file. Now there
24:47:08are like many different options that you
24:47:10can go and add. For example, tape lock.
24:47:13It is an option in order to improve the
24:47:15performance where you are locking the
24:47:17entire table during loading it. So as
24:47:20SQL is loading the data to this table,
24:47:22it going to go and lock the whole table.
24:47:25So that's it for now. I'm just going to
24:47:26go and add the semicolon and let's go
24:47:28and insert the data from the file inside
24:47:30our bronze table. Let's execute it. And
24:47:32now we can see SQL did insert around
24:47:3480,000 rows inside our table. So it is
24:47:37working. We just loaded the file into
24:47:38our database. But now it is not enough
24:47:41to just write this script. you have to
24:47:43test the quality of your bronze table
24:47:45especially if you are working with
24:47:47files. So let's go and just do a simple
24:47:49select. So from our new
24:47:53table and let's run it. So now the first
24:47:56thing that I check is do we have data
24:47:58like in each column? Well yes as you can
24:48:00see we have data and the second thing is
24:48:03do we have the data in the correct
24:48:05column. This is very critical as you are
24:48:07loading the data from a file to a
24:48:09database. Do we have the data in the
24:48:11correct column? So for example, here we
24:48:12have the first name which of course
24:48:14makes sense and here we have the last
24:48:15name. But what could happen and this
24:48:17mistakes happens a lot is that you find
24:48:20the first name informations inside the
24:48:22key and as well you see the last name
24:48:24inside the first name and the status
24:48:27inside the last name. So there is like
24:48:29shifting of the data and this data
24:48:31engineering mistake is very common if
24:48:33you are working with CSV files and there
24:48:35are like different reasons why it
24:48:36happens. Maybe the definition of your
24:48:38table is wrong or the field separator is
24:48:41wrong. Maybe it's not a comma, it's
24:48:43something else or the separator is a bad
24:48:45separator because sometimes maybe in the
24:48:47keys or in the first name there is a
24:48:50comma and the SQL is not able to split
24:48:52the data correctly. So the quality of
24:48:54the CSV file is not really good and
24:48:56there are many different reasons why you
24:48:58are not getting the data in the correct
24:49:00column. But for now everything looks
24:49:02fine for us. And the next step is that
24:49:04I'll go and count the rows inside this
24:49:07table. So let's go and select that. So
24:49:09we can see we have
24:49:1218,493. And now what we can do, we can
24:49:14go to our CSV file and check how many
24:49:16rows do we have inside this file. And as
24:49:18you can see we have
24:49:2118,494. We are almost there. There is
24:49:23like one extra row inside the file. And
24:49:25that's because of the header. the first
24:49:28header information is not loaded inside
24:49:30our table and that's why always in our
24:49:32tables we're going to have one less row
24:49:34than the original files. So everything
24:49:37looks nice and we have done this step
24:49:38correctly. Now if I go and run it again
24:49:41what's going to happen we will get
24:49:43duplicates inside the bronze layer. So
24:49:46now we have loaded the file like twice
24:49:48inside the same table which is not
24:49:50really correct. The method that we have
24:49:52discussed is first to make the table
24:49:54empty and then load truncate and then
24:49:57insert. In order to do that before the
24:50:00bulk inserts, what we're going to do,
24:50:01we're going to say truncate table and
24:50:04then we're going to have our
24:50:06table and that's it with a semicolon. So
24:50:09now what we are doing is first we are
24:50:12making the table empty and then we start
24:50:14loading from the scratch. We are loading
24:50:16the whole content of the file inside the
24:50:19table and this is what we call full
24:50:20load. So now let's go and mark
24:50:22everything together and execute. And
24:50:24again if you go and check the content of
24:50:27the table you can see we have only
24:50:2918,000 rows. Let's go and run it again.
24:50:32The count of the bronze layer you can
24:50:34see we still have the 18,000. So each
24:50:36time you run this script now we are
24:50:39refreshing the table customer info from
24:50:41the file into the database table. So we
24:50:44are refreshing the bronze layer table.
24:50:46So that means if there's like now any
24:50:47changes in the file, it will be loaded
24:50:50to the table. So this is how we do a
24:50:52full load in the bronze layer by
24:50:54truncating the table and then doing the
24:50:57inserts. And now of course what we have
24:50:58to do is to pause the video and go and
24:51:00write the same script for all six files.
24:51:04So let's go and do
24:51:08[Music]
24:51:09that. Okay, back. So I hope that you
24:51:12have as well written all those scripts.
24:51:14So I have the three tables in order to
24:51:16load the first source system and then
24:51:18three sections in order to load the
24:51:20second source system. And as I'm writing
24:51:22those scripts, make sure to have the
24:51:23correct path. So for the second source
24:51:26system, you have to go and change the
24:51:27path for the other folder. And as well,
24:51:29don't forget the table name on the
24:51:30bronze layer is different from the file
24:51:32name because we start always with the
24:51:34source system name with the files. We
24:51:37don't have that. So now I think I have
24:51:38everything is ready. So let's go and
24:51:41execute the whole thing. Perfect.
24:51:43Awesome. So everything is working. Let
24:51:45me check the messages. So we can see
24:51:47from the message how many rows are
24:51:49inserted in each table. And now of
24:51:51course the task is to go through each
24:51:53table and check the
24:51:58content. So that means now we have
24:52:00really nice script in order to load the
24:52:03bronze layer. And we will use this
24:52:05script in daily basis. every day we have
24:52:07to run it in order to get a new content
24:52:10to the data warehouse. And as we learned
24:52:12before, if you have like a script of SQL
24:52:14that is frequently used, what we can do,
24:52:17we can go and create a stored procedure
24:52:20from those scripts. So let's go and do
24:52:22that. It's going to be very simple.
24:52:23We're going to go over here and say
24:52:25create or alter procedure. And now we
24:52:28have to define the name of the S
24:52:30procedure. I'm going to go and put it in
24:52:31the schema bronze because it belongs to
24:52:34the bronze layer. So then we're going to
24:52:36go and follow the naming convention. The
24:52:38source procedure start with load
24:52:39underscore and then the bronze layer. So
24:52:42that's it about the name and then very
24:52:44important we have to define the begin
24:52:45and as well the end of our skill
24:52:47statements. So here is the begin and
24:52:50let's go to the end and say this is the
24:52:53end. And then let's go highlight
24:52:55everything in between and give it one
24:52:57push with tab. So with that it is easier
24:52:59to read. So now next what we're going to
24:53:01do we're going to go and execute it. So
24:53:02let's go and create this store
24:53:04procedure. And now if you want to go and
24:53:05check your store procedure, you go to
24:53:07the database and then we have here a
24:53:08folder called programmability. And then
24:53:10inside it we have start procedure. So if
24:53:12you go and refresh, you will see our new
24:53:14stored procedure. Let's go and test it.
24:53:16So I'm going to go and have a new query.
24:53:18And what we're going to do, we're going
24:53:19to say execute
24:53:22bronze.load bronze. So let's go and
24:53:25execute it. And with that, we have just
24:53:27loaded completely the bronze layer. So
24:53:29as you can see SQL did go and insert all
24:53:32the data from the files to the bronze
24:53:34layer. It is way easier than each time
24:53:36running those scripts of course. All
24:53:38right. So now the next step is that as
24:53:39you can see the output message it is
24:53:42really not having a lot of informations.
24:53:44The message of your ETL sold procedure
24:53:47it will not be really clear. So that's
24:53:49why if you are writing an ETL script
24:53:51always take care of the messaging of
24:53:53your code. So let me show you a nice
24:53:55design. Let's go back to our store
24:53:57procedure. So now what we can do we can
24:53:58go and divide the message based on our
24:54:01code. So now we can start with the
24:54:03message for example over here let's say
24:54:04print and we say what we are doing with
24:54:07this store procedure we are loading the
24:54:09bronze liar. So this is the main message
24:54:12the most important one and we can go and
24:54:14play with the separators like this. So
24:54:16we can say print and now we can go and
24:54:18add some nice separators like for
24:54:20example the equals at the start and at
24:54:23the end just to have like a section. So
24:54:25this is just a nice message at the
24:54:26start. So now by looking to our code we
24:54:29can see that our code is splitted into
24:54:31two sections. The first section we are
24:54:33loading all the tables from the source
24:54:35system CRM and the second section is
24:54:37loading the tables from the ERP. So we
24:54:39can split the prints by the source
24:54:41system. So let's go and do that. So
24:54:43we're going to say print and we're going
24:54:45to say loading CRM tables. This is for
24:54:49the first section. And then we can go
24:54:51and add some nice separators like the
24:54:53one. Let's take the minus. And of
24:54:56course, don't forget to add semicolons
24:54:58like me. So, we're going to have
24:55:00semicolon for each prints. Same thing
24:55:03over here. I will go and copy the whole
24:55:05thing because we're going to have it at
24:55:06the start and as well at the ends. Let's
24:55:09go copy the whole thing for the second
24:55:11section. So, for the ERP, it starts over
24:55:13here. And we're going to have it like
24:55:15this. And we're going to call it loading
24:55:17ERP. So, with that in the output, we can
24:55:20see nice separation between loading each
24:55:22source system. Now we go to the next
24:55:24step where we go and add like a print
24:55:26for each action. So for example here we
24:55:29are truncating the table. So we say
24:55:31print and now what we can do we can go
24:55:33and add two arrows and we say what we
24:55:36are doing. So we are truncating the
24:55:38table and then we can go and add the
24:55:40table name in the message as well. So
24:55:42this is the first action that we are
24:55:43doing and we can go and add another
24:55:45print for inserting the data. So we can
24:55:48say inserting data into and then we have
24:55:53the table name. So with that in the
24:55:54output we can understand what SQL is
24:55:56doing. So let's go and repeat this for
24:55:58all other tables. Okay. So I just added
24:56:01all those prints and don't forget the
24:56:03semicolon at the end. So I would say
24:56:05let's go and execute it and check the
24:56:07output. So let's go and do that and then
24:56:09maybe at the start just to have quick
24:56:11output execute our stored procedure like
24:56:14this. So let's see now if you check the
24:56:17output you can see things are more
24:56:19organized than before. So at the start
24:56:21we are reading okay we are loading the
24:56:23bronze layer. Now first we are loading
24:56:25the source system CRM and then the
24:56:27second section is for the ERP and we can
24:56:29see the actions. So we are truncating
24:56:31inserting truncating inserting for each
24:56:33table and as well the same thing for the
24:56:35second source. So as you can see it is
24:56:37nice and cosmetic but it's very
24:56:39important as you are debugging any
24:56:41errors. And speaking of errors, we have
24:56:43to go and handle the errors in our store
24:56:46procedure. So let's go and do that. It's
24:56:48going to be the first thing that we do.
24:56:50We say begin try and then we go to the
24:56:52end of our script and we say before the
24:56:55last end we say end try and then the
24:56:58next thing we have to add the catch. So
24:57:01we're going to say begin catch and end
24:57:04catch. So now first let's go and
24:57:06organize our code. I'm going to take the
24:57:08whole codes and give it one more push
24:57:11and as well the begin try. So it is more
24:57:14organized and as you know the try and
24:57:16catch going to go and execute the try
24:57:19and if there is like any errors during
24:57:22executing this script the second section
24:57:24going to be executed. So the catch will
24:57:26be executed only if the SQL failed to
24:57:30run the try. So now what we have to do
24:57:32is to go and define for SQL what to do
24:57:34if there's like an error in your code.
24:57:36And here we can do multiple stuff like
24:57:38maybe creating a logging tables and add
24:57:40the messages inside this table or we can
24:57:43go and add some nice messaging to the
24:57:45output like for example we can go and
24:57:47add like a section again over here. So
24:57:50again some equals and we can go and
24:57:53repeat it over here and then add some
24:57:54content in between. So we can start with
24:57:57something like to say error
24:58:00accord during loading bronze layer and
24:58:04then we can go and add many stuff like
24:58:06for example we can go and add the error
24:58:10message and here we can go and call the
24:58:12function
24:58:13error message and we can go and add as
24:58:17well for example the error number. So
24:58:20error number and of course the output of
24:58:22this going to be a number but the error
24:58:24message here is a text. So we have to go
24:58:26and change the data type. So we're going
24:58:28to do a cast as invar like this and then
24:58:33there is like many functions that you
24:58:34can add to the output like for example
24:58:37the error state and so on. So you can
24:58:39design what can happen if there is an
24:58:41error in the ETL. Now what else is very
24:58:43important in each ATL process is to add
24:58:46the duration of each like step. So for
24:58:49example, I would like to understand how
24:58:51long it takes to load this table over
24:58:53here. But looking to the output, I don't
24:58:55have any informations how long is taking
24:58:57to load my tables. And this is very
24:58:59important because as you are building
24:59:01like a big data warehouse, the ETL
24:59:04process going to take long time and you
24:59:05would like to understand where is the
24:59:07issue, where is the bottleneck, which
24:59:09table is consuming a lot of time to be
24:59:11loaded. So that's why we have to add
24:59:13those informations as well to the output
24:59:16or even maybe to protocol it in a table.
24:59:18So let's go and add as well this step.
24:59:20So we're going to go to the start and
24:59:21now in order to calculate the duration
24:59:23you need the starting time and the end
24:59:25time. So we have to understand when we
24:59:28start loaded and when we ended loading
24:59:30the table. So now the first thing is we
24:59:32have to go and declare the variables. So
24:59:35we're going to say declare and then
24:59:36let's make one called start time and the
24:59:39data type of this going to be the date
24:59:40time. I need exactly the second when it
24:59:43started and then another one for the end
24:59:45time. So another variable end time and
24:59:49as well the same thing date time. So
24:59:51with that we have declared the variables
24:59:53and the next step is to go and use them.
24:59:55So now let's go to the first table to
24:59:57the customer info and at the start we're
24:59:59going to say set start
25:00:01time equal to get date. So we will get
25:00:05the exact time when we start loading
25:00:08this table. And then let's go and copy
25:00:10the whole thing and go to the end of
25:00:12loading over here. So we're going to say
25:00:14set this time the end time equal as well
25:00:17to the get dates. So with that now we
25:00:20have the values of when we start loading
25:00:22this table and when we completed loading
25:00:25the table. And now the next step is we
25:00:26have to go and print the duration those
25:00:29informations. So over here we can go and
25:00:31say print and we can go and have as
25:00:33again the same design. So two arrows and
25:00:35we can say very simply load duration and
25:00:38then double points and a space. And now
25:00:41what we have to do is to calculate the
25:00:43duration and we can do that using the
25:00:45date and time function date diff in
25:00:47order to find the interval between two
25:00:50dates. So we're going to say plus over
25:00:52here and then use date diff. And here we
25:00:54have to define three arguments. First
25:00:56one is the unit. So here you can define
25:00:58second, minute, hours and so on. So
25:01:00we're going to go with the second and
25:01:01then we're going to define the start of
25:01:03the interval. It's going to be the start
25:01:04time. And then the last argument it
25:01:06going to be the end of the boundary.
25:01:08It's going to be the end time. And now
25:01:10of course the output of this going to be
25:01:11a number that's why we have to go and
25:01:13cast it. So we're going to say cast as
25:01:16invar and then we're going to close it
25:01:18like this and maybe at the end we're
25:01:20going to say
25:01:21plus space seconds in order to have a
25:01:24nice message. So again what we have done
25:01:26we have declared the two variables and
25:01:28we are using them at the start we are
25:01:30getting the current date and time and at
25:01:33the end of loading the table we are
25:01:34getting the current date and time and
25:01:36then we are finding the differences
25:01:38between them in order to get the load
25:01:40duration and in this case we are just
25:01:42printing this information and now we can
25:01:44go of course and add some nice separator
25:01:46between each table so I'm going to go
25:01:48and do it like this just few minuses not
25:01:51a lot of stuff so now what we have to do
25:01:53is to go and add this mechanism for each
25:01:56table in order to measure the speed of
25:01:59the ETL for each one of
25:02:03[Music]
25:02:05them. Okay. So now I have added all
25:02:08those configurations for each table and
25:02:11let's go and run the whole thing now. So
25:02:14let's go and edit the store procedure
25:02:16this and we're going to go and run it.
25:02:19So let's go and execute. So now as you
25:02:21can see we have here one more info about
25:02:23the load durations and it is everywhere
25:02:25I can see we have zero seconds and
25:02:28that's because it is super fast of
25:02:30loading those informations we are doing
25:02:32everything locally at PC so loading the
25:02:34data from files to database going to be
25:02:36mega fast but of course in real projects
25:02:38you have like different servers and
25:02:40networking between them and you have
25:02:42millions of rows in the tables of course
25:02:44the duration going to be not like 0
25:02:46seconds things going to be slower and
25:02:48now you can see easily how long it takes
25:02:51to load each of your tables. And now of
25:02:53course what is very interesting is to
25:02:55understand how long it takes to load the
25:02:58whole bronze layer. So now your task is
25:03:00as well to print at the end informations
25:03:03about the whole patch. How long it took
25:03:05to load the bronze
25:03:07[Music]
25:03:11layer. Okay, I hope we are done. Now I
25:03:14have done it like this. We have to
25:03:16define two new variables. So the start
25:03:19time of the batch and the end time of
25:03:21the batch. And the first step in the
25:03:23start procedure is to get the date and
25:03:25time informations for the first
25:03:26variable. And exactly at the end the
25:03:29last thing that we do in the start
25:03:31procedure, we're going to go and get the
25:03:33date and time informations for the end
25:03:36time. So we say again set get date for
25:03:38the patch and time. And then all what we
25:03:40have to do is to go and print a message.
25:03:42So we are saying loading bronze layer is
25:03:45completed and then we are printing total
25:03:47load duration and the same thing with a
25:03:49date difference between the patch start
25:03:51time and the end time and we are
25:03:53calculating the seconds and so on. So
25:03:55now what we have to do is to go and
25:03:56execute the whole thing. So let's go and
25:03:59refresh the definition of the start
25:04:01procedure and then let's go and execute
25:04:03it. So in the output we have to go to
25:04:05the last message and we can see loading
25:04:08bronze layer is completed and the total
25:04:10load duration is as well 0 seconds
25:04:13because the execution time is less than
25:04:151 second. So with that you are getting
25:04:17now a feeling about how to build an ETL
25:04:19process. So as you can see the data
25:04:21engineering is not all about how to load
25:04:24the data. It's how to engineer the whole
25:04:26pipeline. how to measure the speed of
25:04:28loading the data. What can happen if
25:04:30there is like an error and to print each
25:04:32step in your ETL process and make
25:04:35everything organized and cleared in the
25:04:37output and maybe in the logging just to
25:04:39make debugging and optimizing the
25:04:41performance way easier. And there's like
25:04:43a lot of things that we can add. We can
25:04:45add the quality measures and stuff. So
25:04:48we can add many stuff to our ETL script
25:04:50to make our data warehouse professional.
25:04:53All right, my friends. So with that we
25:04:54have developed a code in order to load
25:04:56the bronze layer and we have tested that
25:04:58as well. And now in the next step we're
25:05:00going to go back to draw because we want
25:05:02to draw a diagram about the data flow.
25:05:04So let's
25:05:08go. So now what is a data flow diagram?
25:05:11We're going to draw a simple visual in
25:05:12order to map the flow of your data where
25:05:15it come from and where it ends up. So we
25:05:18want just to make clear how the data
25:05:20flows through different layers of your
25:05:22projects. And that's help us to create
25:05:24something called the data lineage. And
25:05:26this is really nice especially if you
25:05:28are analyzing an issue. So if you have
25:05:30like multiple layers and you don't have
25:05:32a real data lineage or flow, it's going
25:05:34to be really hard to analyze the scripts
25:05:36in order to understand the origin of the
25:05:39data and having this diagram going to
25:05:41improve the process of finding issues.
25:05:43So now let's go and create one. Okay. So
25:05:45now back to draw and we're going to go
25:05:47and build the flow diagram. So we're
25:05:48going to start first with the source
25:05:51system. So, let's build the layer. I'm
25:05:53going to go and remove the fill dot it.
25:05:55And then we're going to go and add like
25:05:57a box saying sources and we're going to
25:06:00put it over here. Increase the size 24
25:06:04and as well without any lines. Now, what
25:06:07do we have inside the sources? We have
25:06:09like folder and files. So, let's go and
25:06:12search for a folder icon. I'm going to
25:06:14go and take this one over here and say
25:06:16you are the CRM. And we can as well
25:06:18increase the size. And we have another
25:06:20source. We have the
25:06:23ERP. Okay. So, this is the first layer.
25:06:26Let's go and now have the bronze layer.
25:06:29So, we're going to go and grab another
25:06:31box. And we're going to go and make the
25:06:33coloring like this. And instead of auto,
25:06:35maybe take the hatch, maybe something
25:06:37like this, whatever, you know. So,
25:06:40rounded. And then we can go and put on
25:06:42top of it like the title. So, we can say
25:06:46you are the bronze layer. and increase
25:06:49as well the size of the font. So now
25:06:51what we're going to do, we're going to
25:06:52go and add boxes for each table that we
25:06:55have in the bronze layer. So for
25:06:56example, we have the sales details. We
25:06:58can go and make it a little bit smaller.
25:07:00So maybe 16 and not bold. And we have
25:07:04other two tables from the CRM. We have
25:07:07the customer info and as well the
25:07:10product info. So those are the three
25:07:12tables that comes from the CRM. And now
25:07:15what we're going to do, we're going to
25:07:16go and connect now the source CRM with
25:07:20those three tables. So what we're going
25:07:21to do, we're going to go to the folder
25:07:22and start making arrows from the folder
25:07:26to the bronze layer like this. And now
25:07:28we have to do the same thing for the ERP
25:07:31source. So as you can see the data flow
25:07:33diagram shows us in one picture the data
25:07:36lineage between the two layers. So here
25:07:38we can see easily those three tables
25:07:40actually comes from the CRM and as well
25:07:42those three tables in the bronze layer
25:07:44are coming from the ERP. I understand if
25:07:46we have like a lot of tables it's going
25:07:48to be a huge mess. But if you have like
25:07:50small or medium data warehouse building
25:07:53those diagrams going to make things
25:07:54really easier to understand how
25:07:56everything is flowing from the sources
25:07:59into the different layers in your data
25:08:01warehouse. All right. So with that we
25:08:03have the first version of the data flow.
25:08:05So this step is done and the final step
25:08:07is to commit our code in the get
25:08:13repo. Okay. So now let's go and commit
25:08:15our work. Since it is scripts, we're
25:08:17going to go to the folder scripts. And
25:08:18here we're going to have like script for
25:08:20the bronze, silver, and gold. That's why
25:08:22maybe it makes sense to create a folder
25:08:24for each layer. So let's go and start
25:08:26creating the bronze folder. So I'm going
25:08:28to go and create a new file. And then
25:08:30I'm going to say bronze slash. And then
25:08:32we can have the DDL script of the bronze
25:08:35layer SQL. So now I'm going to go and
25:08:37paste the DDL codes that we have
25:08:39created. So those six tables and as
25:08:42usual at the start we have a comment
25:08:44where we are explaining the purpose of
25:08:45this script. So we are saying this
25:08:47scripts creates tables in the bronze
25:08:49schema. And by running this scripts you
25:08:51are redefining the DDL structure of the
25:08:53bronze tables. So let's have it like
25:08:55that. And I'm going to go and commit the
25:08:58changes. All right. So now as you can
25:09:00see inside the scripts we have a folder
25:09:02called bronze and inside it we have the
25:09:05DDL script for the bronze layer and as
25:09:07well in the bronze layer we're going to
25:09:08go and put our start procedure. So we're
25:09:10going to go and create a new file let's
25:09:12call it proc load bronze dossql and then
25:09:17let's go and paste our script and as
25:09:19usual I have put it at the start an
25:09:21explanation about the store procedure.
25:09:23So we are saying this third procedure
25:09:25going to go and load the data from the
25:09:26CSV files into the bronze schema. So it
25:09:29going to go and truncate first the
25:09:31tables and then do a bulk insert. And
25:09:33about the parameters, this source
25:09:35procedure does not accept any parameter
25:09:37or return any values. And here a quick
25:09:39example how to execute it. All right. So
25:09:41I think I'm happy with that. So let's go
25:09:44and commit it. All right. My friends, so
25:09:47with that we have committed our code
25:09:49into the g. And with that we are done
25:09:51building the bronze layer. So the whole
25:09:53op is done. Now we're going to go to the
25:09:55next one. This one going to be more
25:09:57advanced than the bronze layer because
25:09:59there will be a lot of struggle with
25:10:01cleaning the data and so on. So we're
25:10:02going to start with the first task where
25:10:04we're going to analyze and explore the
25:10:06data in the source systems. So let's
25:10:11go. Okay. So now we're going to start
25:10:13with the big question. How to build the
25:10:15server layer? What is the process? Okay.
25:10:17As usual, first things first, we have to
25:10:19analyze. And now the task before
25:10:21building anything in the server layer we
25:10:23have to go and explore the data in order
25:10:26to understand the content of our sources
25:10:28once we have it what we're going to do
25:10:29we will be starting coding and here the
25:10:32transformation that we're going to do is
25:10:33data cleansing this is usually process
25:10:35that take really long time and I usually
25:10:37do it in three steps the first step is
25:10:40to check first the data quality issues
25:10:42that we have in the bronze layer so
25:10:44before writing any data transformations
25:10:46first we have to understand what are the
25:10:48issues and only then I start writing
25:10:50think data transformations in order to
25:10:52fix all those quality issues that we
25:10:54have in the bronze and the last step
25:10:56once I have clean results what we're
25:10:58going to do we're going to go and insert
25:11:00it into the server layer and those are
25:11:02the three faces that we will be doing as
25:11:04we are writing the code for the silver
25:11:06layer and the third step once we have
25:11:07all the data in the server layer we have
25:11:09to make sure that the data is now
25:11:11correct and we don't have any quality
25:11:13issues anymore and if you find any
25:11:15issues of course what you going to do
25:11:16we're going to go back to coding we're
25:11:18going to do the data cleansing and again
25:11:19object. So it is like a cycle between
25:11:21validating and coding. Once the quality
25:11:24of the silver layer is good, we cannot
25:11:26skip the last phase where we're going to
25:11:28document and commit our work in the G.
25:11:30And here we're going to have two new
25:11:31documentations. We're going to build the
25:11:33data flow diagram and as well the data
25:11:36integration diagram after we understood
25:11:38the relationship between the sources
25:11:40from the first step. So this is the
25:11:42process and this is how we're going to
25:11:43build the server layer.
25:11:49All right. So now exploring the data in
25:11:51the bronze layer. So why it is very
25:11:52important? Because understanding the
25:11:54data it is the key to make smart
25:11:56decisions in the server layer. It was
25:11:58not the focus in the bronze layer to
25:12:00understand the content of the data at
25:12:02all. We focus only how to get the data
25:12:04to the data warehouse. So that's why we
25:12:06have now to take a moment in order to
25:12:09explore and understand the tables and as
25:12:11well how to connect them. what are the
25:12:13relationship between these tables and it
25:12:15is very important as you are learning
25:12:17about the new source system is to create
25:12:19like some kind of documentation. So now
25:12:21let's go and explore the sources. Okay.
25:12:24So now let's go and explore them one by
25:12:26one. We can start with the first one
25:12:27from the CRM. We have the customer info.
25:12:30So right click on it and say select top
25:12:32thousand rows. And this is of course
25:12:33important if you have like a lot of
25:12:35data. Don't go and explore millions of
25:12:37rows. Always limit your query. So for
25:12:39example here we are using the top
25:12:40thousands just to make sure that you are
25:12:42not impacting the system with your
25:12:43queries. So now let's have a look to the
25:12:45content of this table. So we can see
25:12:46that we have here customer informations.
25:12:49So we have an ID, we have a key for the
25:12:51customer, we have first name, last name,
25:12:53marital status, gender and the creation
25:12:56date of the customer. So simply this is
25:12:58a table for the customer information and
25:13:00a lot of details for the customers. And
25:13:02here we have like two identifiers. one
25:13:05it is like technical ID and another one
25:13:07it's like the customer number so maybe
25:13:10we can use either the ID or the key in
25:13:12order to join it with other tables so
25:13:14now what I usually do is to go and draw
25:13:16like data model or let's say integration
25:13:19model just to document and visual what I
25:13:21am understanding because if you don't do
25:13:23that you're going to forget it after a
25:13:24while so now we go and search for a
25:13:26shape let's search for a table and I'm
25:13:28going to go and pick this one over here
25:13:30so here we can go and change the style
25:13:32for example we can make it rounded or
25:13:34you can go make it sketch and so on. And
25:13:37we can go and change the color. I'm
25:13:39going to make it blue. Then go to the
25:13:41text. Make sure to select the whole
25:13:42thing. And let's make it bigger. 26. And
25:13:46then what I'm going to do for those
25:13:48items, I'm just going to select them and
25:13:50go to our range and maybe make it 40.
25:13:53Something like this. So now what we're
25:13:55going to do, we're going to just go and
25:13:56put the table name. So this is the one
25:13:58that we are now learning about. And what
25:14:01I'm going to do, I'm just going to go
25:14:02and put here the primary key. I will not
25:14:04go and list all the informations. So the
25:14:06primary key was the ID. And I will go
25:14:09and remove all those stuff. I don't need
25:14:11it. Now, as you can see, the table name
25:14:12is not really friendly. So I can go and
25:14:14bring a text and put it here on top and
25:14:17say this is the customer information.
25:14:20Just to make it friendly and to not
25:14:22forget about it. And as well going to
25:14:24increase the size to maybe 20 something
25:14:27like this. Okay. With that, we have our
25:14:29first table. and we're going to go and
25:14:31keep exploring. So let's move to the
25:14:33second one. We're going to take the
25:14:34product information, right click on it
25:14:37and select the top thousand rows. I will
25:14:39just put it below the previous query.
25:14:41Query it. Now by looking to this table
25:14:43we can see we have product informations.
25:14:46So we have here a primary key for the
25:14:48product and then we have like key or
25:14:50let's say product number and after that
25:14:52we have the full name of the product the
25:14:54product costs and then we have the
25:14:56product line and then we have like start
25:14:58and end. Well this is interesting to
25:15:01understand why we have start and ends.
25:15:03Let's have a look for example for those
25:15:05three rows all of those three having the
25:15:08same key but they have different ids. So
25:15:10it is the same product but with
25:15:13different costs. So for 2011 we have the
25:15:16cost of 12. Then 2012 we have 14 and for
25:15:20the last year 2013 we have 13. So it's
25:15:23like we have like a history for the
25:15:25changes. So this table not only holding
25:15:27the current informations of the product
25:15:30but also history informations of the
25:15:32product and that's why we have those to
25:15:34date start and end. Now let's go back
25:15:36and draw this information over here. So
25:15:38I'm just going to go and duplicate it.
25:15:40So the name of this table going to be
25:15:42the BRD info and let's go and give it
25:15:44like a short description current and
25:15:47history products information something
25:15:51like this just to not forget that we
25:15:53have history in this table and here we
25:15:54have as well the PRD ID and there is
25:15:57like nothing that we can use in order to
25:15:59join those two tables we don't have like
25:16:02a customer ID here or in the other table
25:16:04we don't have any product ID okay so
25:16:06that's it for this table let's jump to
25:16:07the third table and the last one in the
25:16:09CRM M. So let's go and select. I just
25:16:12made the other queries as well short. So
25:16:14let's go and execute. So what do we have
25:16:15over here? We have a lot of informations
25:16:17about the order, the sales and a lot of
25:16:20measures. Order number. We have the
25:16:22product key. So this is something that
25:16:24we can use in order to join it with the
25:16:26product table. We have the customer ID.
25:16:28We don't have the customer key. So here
25:16:30we have like ID and here we have key. So
25:16:33there's like two different ways on how
25:16:35to join tables. And then we have here
25:16:37like dates. the order date, the shipping
25:16:40date, the due date and then we have the
25:16:42sales amount, the quantity and the
25:16:44price. So this is like an event table.
25:16:46It is transactional table about the
25:16:48orders and sales and it is great table
25:16:51in order to connect the customers with
25:16:53the products and as well with the
25:16:55orders. So let's document this new
25:16:57information that we have. So the table
25:17:00name is the sales details. So we can go
25:17:02and describe it like this.
25:17:05Transactional records about sales and
25:17:10orders. And now we have to go and
25:17:12describe how we can connect this table
25:17:14to the other two. So we are not using
25:17:16the product ID. We are using the
25:17:18products key. And now we need a new
25:17:21column over here. So you can hold
25:17:22control and enter or you can go over
25:17:24here and add a new row. And the other
25:17:27row going to be the customer ID. So now
25:17:29for the customer ID it is easy. we can
25:17:31go and grab an arrow in order to connect
25:17:33those two tables. But for the product
25:17:34key, we are not using the ID. So that's
25:17:37why I'm just going to go and remove this
25:17:39one and say product key. Let's have
25:17:41again a check. So this is a product key.
25:17:44It's not the product ID. And if we go
25:17:46and check the old table, the products
25:17:47info, you can see we are using this key
25:17:50and not the primary key. So what we're
25:17:52going to do now, we will just go and
25:17:54link it like this. And maybe switch
25:17:56those two tables. So I will put the
25:17:58customers below. Just perfect. It looks
25:18:01nice. Okay. So, let's keep moving. Let's
25:18:03go now to the other source system. We
25:18:05have the ARP and the first one is ARB
25:18:07cost and we have this cryptical name.
25:18:10Let's go and select the data. So, now
25:18:12here it's small table and we have only
25:18:14three informations. So, we have here
25:18:16something called CD and then we have
25:18:18something I think this is the birthday
25:18:20and the gender information. So, we have
25:18:22here male, female and so on. So, it
25:18:24looks again like the customer
25:18:26informations but here we have like extra
25:18:27data about the birthday. And now if you
25:18:30go and compare it to the customer table
25:18:32that we have from the other source
25:18:33system. Let's go and query it. You can
25:18:35see the new table from the ARB don't
25:18:37have ids. It has actually the customer
25:18:41number or the key. So we can go and join
25:18:43those two tables using the customer key.
25:18:46Let's go and document this information.
25:18:48So I will just go and copy paste and put
25:18:50it here on the right side. I will just
25:18:51go and change the color now since we are
25:18:53now talking about different source
25:18:55system. And here the table name going to
25:18:57be this one. and the key called C ID.
25:19:01Now, in order to join this table with
25:19:03the customer info, we cannot join it
25:19:05with the customer ID. We need the
25:19:06customer key. That's why here we have to
25:19:08go and add a new row. So, ctrl enter and
25:19:10we're going to say customer key. And
25:19:13then we have to go and make a nice arrow
25:19:15between those two keys. So, we're going
25:19:17to go and give it a description,
25:19:20customer information. And here we have
25:19:23the birth date. Okay. So, now let's keep
25:19:26going. We're going to go to the next
25:19:28one. We have the ERP location. Let's go
25:19:32and query this table. So, what do we
25:19:34have over here? We have the CD again.
25:19:36And as you can see, we have country
25:19:38informations. And this is of course
25:19:40again the customer number. And we have
25:19:42only this information, the country. So,
25:19:44let's go and document this information.
25:19:46This is the customer location. Table
25:19:48name going to be like this. And we still
25:19:50have the same ID. So, we have here still
25:19:53the customer ID and we can go and join
25:19:55it using the customer key. And we have
25:19:57to give it the description location of
25:20:00customers and we can say here the
25:20:02country. Okay. So now let's go to the
25:20:05last table and explore it. We have the
25:20:07ERP ex catalog. So let's go and query
25:20:12those informations. So what do we have
25:20:14here? We have like an ID, a category, a
25:20:17subcategory and the maintenance. Here we
25:20:19have like either yes and no. So by
25:20:21looking to this table we have all the
25:20:23categories and the subcategories of the
25:20:26products and here we have like special
25:20:28identifier for those informations. Now
25:20:30the question is how to join it. So I
25:20:33would like to join it actually with the
25:20:34product informations. So let's go and
25:20:36check those two tables together. Okay.
25:20:38So in the product we don't have any ID
25:20:40for the categories but we have these
25:20:42informations actually in the product
25:20:45key. So the first five characters of the
25:20:47product key is actually the category ID.
25:20:50So we can use this information over here
25:20:53in order to join it with the categories.
25:20:55So we can go and describe this
25:20:57information like this and then we have
25:20:59to go and give it a name. And then here
25:21:01we have the ID and the ID could be
25:21:03joined using the product key. So that
25:21:06means for the product information we
25:21:08don't need at all the product ID the
25:21:10primary key. All what we need is the
25:21:13product key or the product number. And
25:21:15what I would like to do is like to group
25:21:16those informations in a box. So, let's
25:21:20go grab like any boxes here on the left
25:21:22side and make it bigger and then make
25:21:26the edges a little bit smaller. Let's
25:21:28remove the fill and the line. I will
25:21:31make a dotted line. And then let's grab
25:21:33another box over here and say this is
25:21:36the CRM. And we can go and increase the
25:21:38size maybe something like 40 smaller 35
25:21:42bold and change the color to blue and
25:21:44just place it here on top of this box.
25:21:46So with that we can understand all those
25:21:48tables belongs to the source system CRM
25:21:51and we can do the same stuff for the
25:21:53right side as well. Now of course we
25:21:55have to go and add the description here.
25:21:58So it's going to be the products
25:22:00categories. All right. So with that we
25:22:02have now a clear understanding how the
25:22:04tables are connected to each others. We
25:22:07understand now the content of each table
25:22:09and of course it can help us to clean up
25:22:11the data in the silver layer in order to
25:22:13prepare it. So as you can see it is very
25:22:16important to take time understanding the
25:22:18structure of the tables the relationship
25:22:20between them before start writing any
25:22:22code. All right. So with that we have
25:22:24now clear understanding about the
25:22:25sources and with that we have as well
25:22:27created a data integration in the draw.
25:22:30So with that we have more understanding
25:22:32about how to connect the sources. And
25:22:34now in the next two task we will go back
25:22:36to SQL where we're going to start
25:22:38checking the quality and as well doing a
25:22:40lot of data transformations. So let's
25:22:42go.
25:22:46Okay, so now let's have a quick look to
25:22:48the specifications of the server layer.
25:22:50So the main objective to have clean and
25:22:52standardized data. We have to prepare
25:22:54the data before going to the gold layer.
25:22:57And we will be building tables inside
25:22:59the silver layer. And the way of loading
25:23:01the data from the bronze to the silver
25:23:03is a full load. So that means we're
25:23:05going to truncate and then insert. And
25:23:06here we're going to have a lot of data
25:23:08transformations. So we're going to clean
25:23:09the data. We're going to bring
25:23:11normalizations, standardizations. We're
25:23:13going to derive new columns. We will be
25:23:15doing as well data enrichments. So a lot
25:23:18of things to be done in the data
25:23:19transformation. But we will not be
25:23:21building any new data model. So those
25:23:23are the specifications and we have to
25:23:25commit ourself to this scope. Okay. So
25:23:27now building the DDL script for the
25:23:28silver layer going to be way easier than
25:23:31the bronze because the definition and
25:23:33the structure of each table in the
25:23:34silver going to be identical to the
25:23:36bronze layer. We are not doing anything
25:23:39new. So all what you have to do is to
25:23:40take the DDL script from the bronze
25:23:42layer and just go and search and replace
25:23:45for the schema. I'm just using the
25:23:46Notepad++ for the scripts. So I'm going
25:23:49to go over here and say replace the
25:23:51bronze dots with silver dots and I'm
25:23:53going to go and replace all. So with
25:23:55that now all the DDL is targeting the
25:23:58schema silver layer which is exactly
25:24:01what we need. All right. Now before we
25:24:03execute our new DDL script for the
25:24:05silver, we have to talk about something
25:24:07called the metadata columns. They are
25:24:09additional columns or fields that the
25:24:11data engineers add to each table that
25:24:14don't come directly from the source
25:24:16systems. But the data engineers use it
25:24:18in order to provide extra informations
25:24:21for each record. Like we can add a
25:24:23column called create date is when the
25:24:25record was loaded or an update date when
25:24:28the record got updated or we can add the
25:24:31source system in order to understand the
25:24:33origin of the data that we have or
25:24:36sometimes we can add the file location
25:24:38in order to understand the lineage from
25:24:40which file the data come from. Those are
25:24:43great tool if you have data issue in
25:24:45your data warehouse if there is like
25:24:46corrupt data and so on. This can help
25:24:49you to track exactly where this issue
25:24:52happens and when. And as well it is
25:24:54great in order to understand whether I
25:24:56have gap in my data especially if you
25:24:57are doing incremental loads. It is like
25:24:59putting labels on everything and you
25:25:02will thank yourself later when you start
25:25:04using them in hard times as you have an
25:25:06issue in your data warehouse. So now
25:25:08back to our DDL scripts and all what you
25:25:10have to do is to go and do the
25:25:11following. So for example for the first
25:25:13table I will go and add at the end one
25:25:16more extra column. So it start with the
25:25:18prefix TWW as we have defined in the
25:25:21naming convention and then underscore
25:25:23let's have the create date and the data
25:25:26type going to be date time 2 and now
25:25:28what we can do is we can go and add a
25:25:30default value for it. I want the
25:25:32database to generate these informations
25:25:34automatically. We don't have to specify
25:25:36that in any scripts. So which value?
25:25:38It's going to be the get date. So each
25:25:41record going to be inserted in this
25:25:42table will get automatically a value
25:25:45from the current date and time. So now
25:25:47as you can see the naming convention it
25:25:49is very important. All those columns
25:25:51comes from the source system and only
25:25:53this one column comes from the data
25:25:55engineer of the data warehouse. Okay. So
25:25:57that's it. Let's go and repeat the same
25:25:59thing for all other tables. So I will
25:26:01just go and add this piece of
25:26:03information for each
25:26:06DDL. All right. So I think that's it.
25:26:09All what you have to do is now to go and
25:26:11execute the whole DDL script for the
25:26:13silver layer. Let's go and do that. All
25:26:15right, perfect. There's no errors. Let's
25:26:17go and refresh the tables on the object
25:26:19explorer. And with that, as you can see,
25:26:21we have six tables for the silver layer.
25:26:23It is identical to the bronze layer, but
25:26:25we have one extra column for the
25:26:31metadata. All right. All right. So now
25:26:33in the server layer before we start
25:26:35writing any data transformations and
25:26:37cleansing we have first to detect the
25:26:39quality issues in the bronze without
25:26:41knowing the issues we cannot find
25:26:43solution right we will explore first the
25:26:45quality issues only then we start
25:26:47writing the transformation scripts. So
25:26:50let's
25:26:56go. Okay. Okay. So now what we're going
25:26:58to do, we're going to go through all the
25:26:59tables over the bronze layer, clean up
25:27:01the data, and then insert it to the
25:27:04server layer. So let's start with the
25:27:05first table, the first bronze table from
25:27:08the source CRM. So we're going to go to
25:27:09the bronze CRM customer info. So let's
25:27:13go and query the data over here. Now, of
25:27:15course, before writing any data
25:27:17transformations, we have to go and
25:27:19detect and identify the quality issues
25:27:22of this table. So usually I start with
25:27:24the first check where we go and check
25:27:26the primary key. So we have to go and
25:27:28check whether there are nulls inside the
25:27:30primary key and whether there are
25:27:31duplicates. So now in order to detect
25:27:34the duplicates in the primary key what
25:27:35we have to do is to go and aggregate the
25:27:38primary key. If we find any value in the
25:27:40primary key that exist more than once
25:27:42that means it is not unique and we have
25:27:44duplicates in the table. So let's go and
25:27:46write query for that. So what we're
25:27:48going to do, we're going to go with the
25:27:50customer ID and then we're going to go
25:27:51and count and then we have to group up
25:27:54the data. So group by based on the
25:27:56primary key and of course we don't need
25:27:58all the results. We need only where we
25:28:00have an issue. So we're going to say
25:28:02having
25:28:04count higher than one. So we are
25:28:06interested in the values where the count
25:28:09is higher than one. So let's go and
25:28:11execute it. Now as you can see we have
25:28:13issue in this table. we have duplicates
25:28:15because all those ids exist more than
25:28:18one in the table which is completely
25:28:20wrong. We should have the primary key
25:28:21unique and you can see as well we have
25:28:23three records where the primary key is
25:28:25empty which is as well a bad thing. Now
25:28:28there is an issue here. If we have only
25:28:30one null it will not be here at the
25:28:32result. So what I'm going to do I'm
25:28:34going to go over here and say or the
25:28:36primary key is null just in case if we
25:28:39have only one null I'm still interested
25:28:41to see the results. So if I go and run
25:28:44it again, we'll get the same results. So
25:28:46this is equality check that you can do
25:28:47on the table. And as you can see, it is
25:28:49not meeting the expectation. So that
25:28:51means we have to do something about it.
25:28:53So let's go and create a new query. So
25:28:56here what we're going to do, we can
25:28:57start writing the query that is doing
25:28:59the data transformation and the data
25:29:01cleansing. So let's start again by
25:29:03selecting the
25:29:07data and execute it again. So now what I
25:29:10usually do I go and focus on the issue.
25:29:13So for example let's go and take one of
25:29:15those values and I focus on it before
25:29:17start writing the transformation. So
25:29:19we're going to say where customer ID
25:29:21equal to this value. All right. So now
25:29:24as you can see we have here the issue
25:29:25where the ID exist three times but
25:29:28actually we are interested only on one
25:29:30of them. So the question is how to pick
25:29:32one of those. Usually we search for a
25:29:34time stamp or date value to help us. So
25:29:36if you check the creation date over here
25:29:38we can understand that this record this
25:29:41one over here is the newest one and the
25:29:44previous two are older than it. So that
25:29:46means if I have to go and pick one of
25:29:48those values I would like to get the
25:29:50latest one because it holds the most
25:29:53fresh information. So what we have to do
25:29:55is we have to go and rank all those
25:29:57values based on the create dates and
25:30:00only pick the highest one. So that means
25:30:03we need a racking function and for that
25:30:05in scale we have the amazing window
25:30:07functions. So let's go and do that. We
25:30:10will use the function row number over
25:30:14and then partition by and here we have
25:30:17to divide the table by the customer ID.
25:30:20So we're going to divide it by the
25:30:22customer ID and in order now to rank
25:30:24those rows we have to sort the data by
25:30:26something. So order by and as we
25:30:28discussed we want to sort the data by
25:30:30the creation date. So create
25:30:33date and we're going to sort it
25:30:35descending. So the highest first then
25:30:37the lowest. So let's go and do that. And
25:30:39now we're going to go and give it a name
25:30:41flag last. So now let's go and execute
25:30:44it. Now the data is sorted by the
25:30:46creation date. And you can see over here
25:30:48that this record is the number one. Then
25:30:51the one that is older is two and the
25:30:53oldest one is three. Of course we are
25:30:55interested in the rank number one. Now
25:30:57let's go and remove the filter and check
25:30:59everything. So now if you have a look to
25:31:01the table you can see that on the flag
25:31:03we have everywhere like one and that's
25:31:06because the those primary keys exist
25:31:08only one but sometimes we will not have
25:31:10one we'll have two three and so on. If
25:31:12there's like duplicates we can go of
25:31:13course and do a double check. So let's
25:31:15go over here and say select
25:31:18star from this query we can say where
25:31:21flag last is in equal to one. So let's
25:31:24go and query it. And now we can see all
25:31:26the data that we don't need because they
25:31:28are causing duplicates in the primary
25:31:30key and they have like an old status. So
25:31:32what we're going to do we're going to
25:31:34say equal to one. And with that we
25:31:36guarantee that our primary key is unique
25:31:38and each value exist only once. So if I
25:31:40go and query it like this you will see
25:31:42we will not find any duplicate inside
25:31:44our table. And we can go and check that
25:31:46of course. So let's go and check this
25:31:48primary key. And we're going to say and
25:31:50customer ID equal to this value. And you
25:31:53can see it exists now only once and we
25:31:56are getting the freshest data from this
25:31:58primary key. So with that we have
25:31:59defined like transformation in order to
25:32:02remove any duplicates. Okay. So now
25:32:04moving on to the next one. As you can
25:32:06see in our table we have a lot of values
25:32:08where they are like string values. Now
25:32:11for these string values we have to check
25:32:13the unwanted spaces. So now let's go and
25:32:16write a query that's going to detect
25:32:17those unwanted spaces. So we're going to
25:32:20say select this column the first name
25:32:24from our table bronze customer
25:32:27information. So let's go and query it.
25:32:29Now by just looking to the data it's
25:32:31going to be really hard to find those
25:32:33unwanted spaces especially if they are
25:32:35at the end of the word. But there is a
25:32:37very easy way in order to detect those
25:32:39issues. So what we're going to do we're
25:32:41going to do a filter. So now we're going
25:32:42to say the first name is not equal to
25:32:46the first name after trimming the
25:32:48values. So if you use the function trim,
25:32:50what it going to do? It's going to go
25:32:51and remove all the leading and trailing
25:32:54spaces. So the first name. So if this
25:32:57value is not equal to the first name
25:33:00after trimming it, then we have an
25:33:02issue. So it is very simple. Let's go
25:33:04and execute it. So now in the result, we
25:33:06will get a list of all first names where
25:33:08we have spaces either at the start or at
25:33:11the end. So again the expectation here
25:33:15is no results. And the same thing we can
25:33:18go and check something else like for
25:33:19example the last name. So let's go and
25:33:22do that over here and here. Let's go and
25:33:25execute it. We see in the results we
25:33:27have as well 17 customers where they
25:33:30have like space in their last name which
25:33:32is not really good. And we can go and
25:33:34keep checking all the string values that
25:33:37we have inside the table. So for example
25:33:38the gender. So let's go and check
25:33:41that and execute. Now as you can see we
25:33:44don't have any results. That means the
25:33:46quality of the gender is better and we
25:33:48don't have any unwanted spaces. So now
25:33:51we have to go and write transformation
25:33:53in order to clean up those two columns.
25:33:55Now what I'm going to do, I'm just going
25:33:56to go and list all the columns in the
25:33:58query instead of the star. All right. So
25:34:01now I have a list of all the columns
25:34:02that I need. And now what we have to do
25:34:04is to go to those two columns and start
25:34:06removing the unwanted spaces. So we will
25:34:09just use the trim. It's very
25:34:11simple. And give it a name, of course,
25:34:14the same name. And we will trim as well
25:34:17the last name. So let's go and query
25:34:19this. And with that we have cleaned up
25:34:21those two columns from any unwanted
25:34:23spaces. Okay. So now moving on we have
25:34:25those two informations. We have the
25:34:27maritalial status and as well the
25:34:29gender. If you check the values inside
25:34:31those two columns as you can see we have
25:34:33here low cardality. So we have limited
25:34:35numbers of possible values that is used
25:34:37inside those two columns. So what we
25:34:40usually do is to go and check the data
25:34:42consistency inside those two columns. So
25:34:45it's very simple what we're going to do.
25:34:47We're going to do the following. We're
25:34:48going to say
25:34:50distinct and we're going to check the
25:34:52values. Let's go and do that. And now as
25:34:54you can see we have only three possible
25:34:56values either null, f or m which is
25:34:59okay. We can stay like this of course.
25:35:01But we can make a rule in our project
25:35:03where we can say we will not be working
25:35:05with data abbreviations. We will go and
25:35:07use only friendly full names. So instead
25:35:10of having an F, we're going to have like
25:35:12a full word female. And instead of m
25:35:15we're going to have like male and we
25:35:16make it as a rule for the whole project.
25:35:18So each time we find the gender
25:35:20informations we try to give the full
25:35:22name of it. So let's go and map those
25:35:24two values to a friendly one. So we're
25:35:26going to go to the gender over here and
25:35:28say case when and we're going to say the
25:35:31gender is equal to f then make it a
25:35:36female. And when it
25:35:39is equal to
25:35:41m then map it to male. And now we have
25:35:46to make decision about the nulls. As you
25:35:48can see over here we have nulls. So do
25:35:50we want to leave it as a null or we want
25:35:52to use always the value unknown. So with
25:35:55that we are replacing the missing values
25:35:58with a standard default value or you can
25:36:00leave it as null. But let's say in our
25:36:02project that we are replacing all the
25:36:04missing value with a default value. So
25:36:06let's go and do that. We're going to say
25:36:09else I'm going to go with the NA not
25:36:12available or you can go with the unknown
25:36:14of course. So that's for the gender
25:36:16information like this. And we can go and
25:36:19remove the old one. And now there is one
25:36:21thing that I usually do in this case
25:36:22where sometimes what happens currently
25:36:24we are getting the capital F and the
25:36:26capital M but maybe in the time
25:36:28something change and you will get like
25:36:30lower M and lower F. So just to make
25:36:32sure in those cases we still are able to
25:36:34map those values to the correct value.
25:36:37What we're going to do we're going to
25:36:38just use the function upper just to make
25:36:41sure that if you get any lowerase values
25:36:44we are able to catch it. So the same
25:36:47thing over here as well. And now one
25:36:49more thing that you can add as well. Of
25:36:51course, if you are not trusting the data
25:36:53because we saw some unwanted spaces in
25:36:55the first name and the last name, you
25:36:57might not trust that in the future. You
25:36:58will get here as well unwanted spaces.
25:37:01You can go and make sure to trim
25:37:04everything just to make sure that you
25:37:07are catching all those cases. So that's
25:37:10it for now. Let's go and execute. Now,
25:37:12as you can see, we don't have an M and
25:37:14an F. We have a full word, male and
25:37:17female. And if we don't have a value, we
25:37:19don't have a null, we are getting here
25:37:21not available. Now we can go and do the
25:37:23same stuff for the maritial status. You
25:37:25can see as well we have only three
25:37:27possibilities. The s null and an M. We
25:37:30can go and do the same stuff. So I will
25:37:32just go and copy everything from here.
25:37:34And I will go and use the marital status
25:37:37and just remove this one from here. And
25:37:39now what are the possible values? We
25:37:41have the S. So it's going to be single.
25:37:44We have an M for married. And we have as
25:37:47well a null and with that we are getting
25:37:50the not available. So with that we are
25:37:52making as well data standardizations for
25:37:54this column. So let's go and execute it.
25:37:57Now as you can see we don't have those
25:37:59short values. We have a full friendly
25:38:01value for the status and as well for the
25:38:04gender. And at the same time we are
25:38:06handling the nulls inside those two
25:38:08columns. So with that we are done with
25:38:10those two columns. And now we can go to
25:38:11the last one that create date. For this
25:38:13type of informations, we make sure that
25:38:15this column is a real date and not as a
25:38:18string or varchar. And as we defined it
25:38:20in the data type, it is a date which is
25:38:22completely correct. So nothing to do
25:38:25with this column. And now the next step
25:38:26is that we're going to go and write the
25:38:28insert statement. So how we going to do
25:38:30it? We're going to go to the start over
25:38:31here and say insert into
25:38:36silverm customer info. Now we have to go
25:38:38and specify all the columns that should
25:38:41be inserted. So we're going to go and
25:38:43type it. So something like this. And
25:38:45then we have the query over here. Let's
25:38:47go and execute it. So let's do that. So
25:38:50with that we have inserted clean data
25:38:52inside the silver table. So now what
25:38:54we're going to do we're going to go and
25:38:55take all the queries that we have used
25:38:58in order to check the quality of the
25:38:59bronze and let's go and take it to
25:39:01another query and instead of having
25:39:03bronze we're going to say silver. So
25:39:05this is about the primary key. Let's go
25:39:08and execute it. Perfect. We don't have
25:39:10any results. So we don't have any
25:39:12duplicates. The same thing for the next
25:39:14one. So the silver and it was for the
25:39:18first name. So let's go and check the
25:39:21first name and run it. As you can see
25:39:23there is no results. It is perfect. We
25:39:25don't have any issues. You can of course
25:39:27go and check the last
25:39:30name and run it again. We don't have any
25:39:33results over here. And now we can go and
25:39:35check those low cardality columns like
25:39:38for
25:39:39example the gender. Let's go and execute
25:39:42it. So as you can see we have the not
25:39:43available or the unknown male and
25:39:46female. So perfect and you can go and
25:39:48have a final look to the table to the
25:39:50silver customer info. Let's go and check
25:39:53that. So now we can have a look to all
25:39:54those columns. As you can see everything
25:39:56looks perfect and you can see it is
25:39:58working this metadata information that
25:40:00we have added to the table definition.
25:40:03Now it says when we have inserted all
25:40:05those records to the table which is
25:40:08really amazing information to have a
25:40:09track and audit. Okay. So now by looking
25:40:11to this script we have done different
25:40:13types of data transformations. The first
25:40:15one is with the first name and the last
25:40:17name. Here we have done trimming
25:40:19removing unwanted spaces. This is one of
25:40:22the types of data cleansing. So we
25:40:24remove unnecessary spaces or unwanted
25:40:26characters to ensure data consistency.
25:40:29Now moving on to the next
25:40:30transformation. we have this case when
25:40:32so what we have done here is data
25:40:34normalization or we call it sometimes
25:40:37data standardization so this
25:40:38transformation is type of data cleansing
25:40:41where we're going to map coded values to
25:40:43meaningful user friendly description and
25:40:46we have done the same transformation as
25:40:48well to the gender another type of
25:40:50transformation that we have done as well
25:40:52in the same case when is that we have
25:40:54handled the missing values so instead of
25:40:57nulls we going to have not available so
25:40:59handling missing data is as type of data
25:41:02cleansing where we are filling the
25:41:03blanks by adding for example a default
25:41:06value. So instead of having an empty
25:41:08string or a null we're going to have a
25:41:10default value like the not available or
25:41:12unknown. Another type of data and
25:41:13transformations that we have done in
25:41:15this script is we have removed the
25:41:17duplicates. So removing duplicates is as
25:41:19well type of data cleansing where we
25:41:21ensure only one record for each primary
25:41:24key by identifying and retaining only
25:41:27the most relevant row to ensure there is
25:41:30no duplicates inside our data and as we
25:41:32are removing the duplicates of course we
25:41:34are doing data filtering. So those are
25:41:36the different types of data
25:41:38transformations that we have done in
25:41:40this script.
25:41:44All right, moving on to the second table
25:41:46in the bronze layer from the CRM. We
25:41:48have the product info. And of course, as
25:41:50usual, before we start writing any
25:41:52transformations, we have to search for
25:41:54data quality issues. And we start with
25:41:56the first one, we have to check the
25:41:57primary key. So we have to check whether
25:41:59we have duplicates or nulls inside this
25:42:01key. So what we have to do, we have to
25:42:03group up the data by the primary key or
25:42:05check whether we have nulls. So let's go
25:42:07and execute it. So as you can see,
25:42:09everything is safe. We don't have
25:42:10duplicates or nulls in the primary key.
25:42:12Now moving on to the next one, we have
25:42:14the product key. Here we have in this
25:42:16column a lot of informations. So now
25:42:18what we have to do is to go and split
25:42:20this string into two informations. So we
25:42:22are deriving new two columns. So now
25:42:24let's start with the first one is the
25:42:26category ID. The first five characters
25:42:29they are actually the category ID and we
25:42:31can go and use the substring function in
25:42:33order to extract part of a string. It
25:42:36needs three arguments. The first one
25:42:38going to be the column that we want to
25:42:40extract from. And then we have to define
25:42:43the position where to extract. And since
25:42:45the first part is on the left side, we
25:42:47going to start from the first position.
25:42:49And then we have to specify the length.
25:42:51So how many characters we want to
25:42:52extract, we need five characters. So 1 2
25:42:553 4 5. So that's it for the category ID.
25:42:59Category ID. Let's go and execute it.
25:43:02Now, as you can see, we have a new
25:43:03column called the category ID. and it
25:43:05contains the first part of the string
25:43:07and in our database from the other
25:43:09source system we have as well the
25:43:11category ID. Now we can go and double
25:43:13check just in order to make sure that we
25:43:16can join data together. So we're going
25:43:17to go and check the ID from the bronze
25:43:20table ERP and this canopy from the
25:43:24category. So in this table we have the
25:43:26category ids and you can see over here
25:43:29those are the ids of the category and in
25:43:31the code layer we have to go and join
25:43:33those two tables. But here we still have
25:43:35an issue. We have here an underscore
25:43:38between the category and the
25:43:39subcategory. But in our table we have
25:43:42actually a minus. So we have to replace
25:43:45that with an underscore in order to have
25:43:47matching informations between those two
25:43:49tables. Otherwise we will not be able to
25:43:50join the tables. So we're going to use
25:43:52the function
25:43:53replace. And what we are replacing? We
25:43:56are replacing the minus with an
25:43:59underscore something like this. And if
25:44:01you go now and execute it, we will get
25:44:04an underscore exactly like the other
25:44:06table. And of course we can go and check
25:44:08whether everything is matching by having
25:44:10very simple query where we say this new
25:44:13information not in. And then we have
25:44:17this nice subquery. So we are trying to
25:44:19find any category ID that is not
25:44:22available in the second table. So let's
25:44:24go and execute it. Now as you can see we
25:44:26have only one category that is not
25:44:28matching. We are not finding it in this
25:44:30table which is maybe correct. So if you
25:44:32go over here you will not find this
25:44:35category. I just make it a little bit
25:44:36bigger. So we are not finding this one
25:44:38category from this table which is fine.
25:44:41So our check is okay. Okay. So that we
25:44:43have the first part. Now we have to go
25:44:45and extract the second part and we're
25:44:47going to do the same thing. So we're
25:44:48going to use the substring and the three
25:44:50argument the product key but this time
25:44:53we will not start cutting from the first
25:44:54position we have to be in the middle. So
25:44:561 2 3 4 5 6 7. So we start from the
25:45:01position number seven. And now we have
25:45:03to define the length how many characters
25:45:06to be extracted. But if you look over
25:45:07here you can see that we have different
25:45:10length of the product keys. It is not
25:45:12fixed like the category ID. So we cannot
25:45:15go and here specify number. We have to
25:45:17make something dynamic and there is
25:45:18trick in order to do that. We're going
25:45:20to go and use the length of the whole
25:45:22column. With that we make sure that we
25:45:24are always getting enough characters to
25:45:26be extracted and we will not be losing
25:45:28any informations. So we will make it
25:45:31dynamic like this. We will not have it
25:45:33as a fixed length and with that we have
25:45:35the product key. So let's go and execute
25:45:38it. As you can see we are now extracting
25:45:41the second part from this string. Now
25:45:44why we need the product key? We need it
25:45:46in order to join it with another table
25:45:48called sales details. So let's go and
25:45:51check the sales details. So let me just
25:45:54check the column name. It is SLS product
25:45:57key. So from bronze
25:46:01CRM sales. Let's go and check the data
25:46:04over here. And it looks wonderful. So
25:46:08actually we can go and join those
25:46:09informations together. But of course
25:46:11we're going to go and check that. So
25:46:12we're going to say where and we're going
25:46:14to take our new column and we're going
25:46:15to say not in the sub query just to make
25:46:19sure that we are not missing anything.
25:46:21So let's go and execute. So it looks
25:46:23like we have a lot of products that
25:46:26don't have any orders. Well, I don't
25:46:28have a nice feelings about it. Let's go
25:46:30and try something like this one here.
25:46:33And we say where sld key like this value
25:46:39over here. So I'll just cut the last
25:46:42three just to search inside this table.
25:46:44So we really don't have such a keys. Let
25:46:47me just cut the second one. So let's go
25:46:50and search for it. We don't have it as
25:46:52well. So anything that starts with the F
25:46:55key, we don't have any order with the
25:46:57product where it starts with the F key.
25:46:59So let's go and remove it. But still we
25:47:02are able to join the tables, right? So
25:47:04if I go and say in instead of not in. So
25:47:07with that you are able to match all
25:47:09those products. So that means everything
25:47:11is fine. Actually it's just products
25:47:14that don't have any orders. So with that
25:47:16I'm happy with this transformation. Now
25:47:19moving on to the next one. We have here
25:47:21the name of the product. We can go and
25:47:23check whether there is unwanted spaces.
25:47:26So let's go to our quality checks. Make
25:47:28sure to use the same table and we're
25:47:31going to use the product name and check
25:47:33whether we find any unmatching after
25:47:36trimming. So let's go and do it. Well,
25:47:38it looks really fine. So we don't have
25:47:40to trim anything. This column is safe.
25:47:42Now moving on to the next one. We have
25:47:44the costs. So here we have numbers and
25:47:47we have to check the quality of the
25:47:48numbers. So what we can do? We can check
25:47:50whether we have nulls or negative
25:47:52numbers. So negative costs or negative
25:47:55prices which is not realistic depend on
25:47:58the business of course. So let's say in
25:47:59our business we don't have any negative
25:48:02costs. So it's going to be like this.
25:48:04Let's go and check whether it's
25:48:06something less than zero or whether we
25:48:09have costs that is null. So let's go and
25:48:14check those informations. Well, as you
25:48:15can see, we don't have any negative
25:48:17values, but we have nulls. So we can go
25:48:19and handle that by replacing the null
25:48:21with a zero. Of course, if the business
25:48:24allow that. So in SQL server, in order
25:48:26to replace the null with a zero, we have
25:48:28a very nice function called is null. So
25:48:32we are saying if it is null then replace
25:48:35this value with a zero. It is very
25:48:37simple like this and we give it a name
25:48:40of course. So let's go and execute it.
25:48:42And as you can see we don't have any
25:48:44more nulls. We have zero which is better
25:48:46for the calculations if you are later
25:48:48doing any aggregate functions like the
25:48:50average. Now moving on to the next one
25:48:52we have the product line. This is again
25:48:55abbreviation of something and the
25:48:57cardinality is low. So let's go and
25:48:58check all possible values inside this
25:49:01column. So we're just going to use the
25:49:03distinct going to be BRD line. So let's
25:49:06go and execute it. And as you can see
25:49:08the possible values are null M R ST. And
25:49:11again those are abbreviations but in our
25:49:13data warehouse we have decided to give
25:49:16full nice names. So we have to go and
25:49:18replace those codes those abbreviations
25:49:21with a friendly value. And of course in
25:49:23order to get those informations I
25:49:24usually go and ask the expert from the
25:49:27source system or an expert from the
25:49:29process. So let's start building our
25:49:31case win. And then let's use the upper
25:49:34and as well the trim just to make sure
25:49:36that we are having all the cases. So the
25:49:39BRD
25:49:40line is equal to so let's start with the
25:49:44first value the M. Then we will get the
25:49:48friendly value it's going to be
25:49:50mountain. then to the next one. So I
25:49:52will just copy and paste here. If it is
25:49:55an R then it is road and another one for
25:50:00let me check what do we have here? We
25:50:03have M R and then S. The S stands for
25:50:06other sales and we have the T. So let's
25:50:10go and get the T. So the T stands for
25:50:14touring. We have at the end an else for
25:50:17unknown not available. So we don't need
25:50:20any nulls. So that's it. And we're going
25:50:22to name it as before. So product line.
25:50:24So let's remove the old one. And let's
25:50:27execute it. And as you can see, we don't
25:50:30have here anymore those shortcuts and
25:50:32the abbreviations. We have now full
25:50:34friendly value. But I will go and have
25:50:37here like capital O. It looks nicer. So
25:50:40that we have nice friendly value. Now by
25:50:42looking to this case when as you can see
25:50:43it is always like we are mapping one
25:50:45value to another value and we are
25:50:47repeating all time upper time upper time
25:50:50and so on. We have here a quick form in
25:50:52the case when if it is just a simple
25:50:54mapping. So the syntax is very simple we
25:50:57say case and then we have the column. So
25:51:00we are evaluating this value over here
25:51:02and then we just say when without the
25:51:05equal so if it is an M then make it
25:51:08mountain. the same thing for the next
25:51:10one and so so with that we have the
25:51:13functions only once and we don't have to
25:51:15go and keep repeating the same function
25:51:16over and over and this one only if you
25:51:18are mapping values but if you have
25:51:20complex conditions you cannot do it like
25:51:22this but for now I'm going to stay with
25:51:24the quick form of the case when it looks
25:51:26nicer and shorter so let's go and
25:51:28execute it we will get the same results
25:51:30okay so now back to our table let's go
25:51:31to the last two columns we have the
25:51:33start and end date so it's like defining
25:51:36an interval we have start and end so
25:51:38Let's go and check the quality of the
25:51:39start and end dates. We're going to go
25:51:41and say select star from our bronze
25:51:45table. And now we're going to go and
25:51:47search it like this. We are searching
25:51:48for the end date that is smaller than
25:51:52the start. So we are key to start dates.
25:51:56So let's go and query this. So you can
25:51:58see the start is always like after the
25:52:01end which makes no sense at all. So we
25:52:03have here data issue with those two
25:52:05dates. So now for this kind of data
25:52:07transformations what I usually do is I
25:52:09go and grab few examples and put it in
25:52:11Excel and try to think about how I'm
25:52:13going to go and fix it. So here I took
25:52:15like two products this one and this one
25:52:17over here. And for that we have like
25:52:18three rows for each one of them. And we
25:52:20have this situation over here. So the
25:52:22question now how we going to go and fix
25:52:24it? I will go and make like a copy of
25:52:26one solution where we're going to say
25:52:28it's very simple. Let's go and switch
25:52:29the start date with the end date. So if
25:52:32I go and grab the end date and put it at
25:52:35the start, things going to look way
25:52:37nicer, right? So we have the start is
25:52:39always younger than the end. But my
25:52:41friends, the data now makes no sense
25:52:43because we say it start from 2007 and
25:52:46ends by 2011 the price was 12. But
25:52:50between 2008 and 2012, we have 14. which
25:52:54is not really good because if you take
25:52:55for example the year 2010 for 2010 it
25:52:59was 12 and at the same time 14. So it is
25:53:02really bad to have an overlapping
25:53:04between those two dates. It should start
25:53:06from 2007 and end with 11 and then start
25:53:09Feb from 12 and end with something else.
25:53:12There should be no overlapping between
25:53:14years. So it's not enough to say the
25:53:17start should be always smaller than the
25:53:19ends but as well the end of the first
25:53:22history should be younger than the start
25:53:25of the next records. This is as well a
25:53:27rule in order to have no overlapping.
25:53:30This one has no start but has already an
25:53:32end which is not really okay because we
25:53:35have always to have a start. Each new
25:53:37record in historiizations has to has a
25:53:40start. So for this record over here this
25:53:42is as well wrong. And of course it is
25:53:44okay to have the start without an end.
25:53:47So in this scenario it's fine because
25:53:49this indicate this is the current
25:53:52informations about the costs. So again
25:53:54this solution is not working at all. So
25:53:56now for the solution two what we can say
25:53:58let's go and ignore completely the end
25:54:01date and we take only the start date. So
25:54:03let's go and paste it over here. But now
25:54:05we go and rebuild the end date
25:54:07completely from the start date following
25:54:09the rules that we have defined. So the
25:54:11rule says the end of date of the current
25:54:14records comes from the start date from
25:54:17the next records. So here this end date
25:54:20comes from this value over here from the
25:54:22next record. So that means we take the
25:54:24next start date and put it at the end
25:54:27date for the previous records. So with
25:54:29that as you can see it is working the
25:54:30end date is higher than the start date.
25:54:34And as well we are making sure this date
25:54:36is not overlapping with the next record.
25:54:38But as well in order to make it way
25:54:40nicer we can subtract it with one. So we
25:54:43can take the previous day like this. So
25:54:45with that we are making sure the end
25:54:47date is smaller than the next start. And
25:54:50now for the next record this one over
25:54:53here the end date going to come from the
25:54:55next start date. So we will take this
25:54:57one for here and put it as an end date
25:54:59and subtract it with one. So we will get
25:55:02the previous day. So now if you compare
25:55:04those two you can see it's still higher
25:55:07than the start. And if you compare it
25:55:08with the next record this one over here
25:55:11it is still smaller than the next one.
25:55:13So there is no overlapping. And now for
25:55:15the last record since we don't have here
25:55:17any informations it will be a null which
25:55:20is totally fine. So as you can see I'm
25:55:22really happy with this scenario over
25:55:24here. Of course you can go and validate
25:55:25this with an expert from the source
25:55:27system. But let's say I have done that
25:55:29and they approved it and now I can go
25:55:30and clean up the data using this new
25:55:33logic. So this is how I usually
25:55:34brainstorm about fixing an issues. If I
25:55:37have like a complex stuff, I go and use
25:55:39Excel and then discuss it with the
25:55:40expert using this example. It's way
25:55:42better than showing a database queries
25:55:45and so on. It just makes things easier
25:55:47to explain and as well to discuss. So
25:55:49now how I usually do it, I usually go
25:55:51and make a focus on only the columns
25:55:53that I need and take only one two
25:55:55scenarios while I'm building the logic
25:55:57and once everything is ready I go and
25:55:59integrate it in the query. So now I'm
25:56:01focusing only on these columns and only
25:56:03for these products. So now let's go and
25:56:06build our logic. Now in SQL if you are
25:56:08at specific record and you want to
25:56:10access another information from another
25:56:12records and for that we have two amazing
25:56:14window functions. We have the lead and
25:56:17log. In this scenario, we want to access
25:56:19the next records. That's why we have to
25:56:21go with the function leads. So, let's go
25:56:23and build it lead. And then what do we
25:56:25need? We need the lead of the
25:56:27start date. So, we want the start date
25:56:30of the next record. And then we say over
25:56:33and we have to partition the data. So,
25:56:36the window going to be focusing on only
25:56:38one product which is the product key and
25:56:41not the product ID. So, we are dividing
25:56:43the data by product key. And of course,
25:56:45we have to go and sort the data. So
25:56:47order by and we are sorting the data by
25:56:49the start
25:56:50date and ascending. So from the lowest
25:56:53to the highest and let's go and give it
25:56:55another name. So as let's say test for
25:56:58example just to test the data. So let's
25:57:01go and execute. And I think I missed
25:57:03something here. It is partition by. So
25:57:05let's go and execute again. And now
25:57:07let's go and check the results for the
25:57:09first partition over here. So the start
25:57:10is 2011 and the end is 2012. And this
25:57:14information came from the next record.
25:57:17So this data is moved to the previous
25:57:19record over here. And the same thing for
25:57:21this record. So the end date comes from
25:57:24the next record. So our logic is
25:57:26working. And the last record over here
25:57:28is null because we are at the end of the
25:57:30window and there is no next data. That's
25:57:32why we will get null and this is perfect
25:57:34of course. So it looks really awesome.
25:57:36But what is missing is we have to go and
25:57:39get the previous day. And we can do that
25:57:41very simply using minus one. we are just
25:57:43subtracting one day. So we have no
25:57:45overlapping between those two dates and
25:57:47the same thing for those two dates. So
25:57:49as you can see we have just built a
25:57:51perfect end date which is way better
25:57:53than the original data that we got from
25:57:55the source system. Now let's take this
25:57:57one over here and put it inside our
25:57:59query. So we don't need the end date, we
25:58:02need our new end date. Let's just remove
25:58:05that test and execute. Now it looks
25:58:08perfect. All right. Now we are not done
25:58:10yet with those two dates. Actually we
25:58:12are saying all time dates because we
25:58:14don't have here any informations about
25:58:16the time always zero. So it makes no
25:58:18sense to have these informations inside
25:58:21our data. So what we can do we can do a
25:58:23very simple cast and we make this column
25:58:26as a date instead of date time. So this
25:58:29is for the first one and as well for the
25:58:31next one as date. So let's try that out.
25:58:34And as you can see it is nicer. We don't
25:58:36have the time informations. Of course,
25:58:38we can tell the source systems about all
25:58:40those issues. But since they don't
25:58:42provide a time, it makes no sense to
25:58:44have date and time. Okay, so it was a
25:58:46long run, but we have now a cleaned
25:58:49product informations. And this is way
25:58:51nicer than the original product
25:58:53information that we got from the source
25:58:54CRM. So if you grab the DDL of the
25:58:56server table, you can see that we don't
25:58:58have a category ID. So we have product
25:59:00ID and product key. And as well those
25:59:03two columns, we just changed the data
25:59:04type. So it's date time here but we have
25:59:06changed that to a date. So that means we
25:59:09have to go and do few modifications to
25:59:11the DDL. So what we're going to do we're
25:59:12going to go over here and say category
25:59:14ID and I will be using the same data
25:59:16type for the start and the end. This
25:59:18time going to be date and not date and
25:59:20time. So that's it for now. Let's go and
25:59:22execute it in order to repair the DDL.
25:59:25And this is what happen in the silver
25:59:27layer. Sometimes we have to adjust the
25:59:29metadata if the quality of the data
25:59:31types and so on is not good or we are
25:59:33building new derived informations in
25:59:35order later to integrate the data. So it
25:59:38will be like very close to the bronze
25:59:40layer but with few modifications. So
25:59:42make sure to update your DTL scripts.
25:59:45And now the next step is that we're
25:59:46going to go and insert the data into the
25:59:48table. And now the next we're going to
25:59:49go and insert the result of this query
25:59:52that is cleaning up the bronze table
25:59:54into the silver table. So as we done it
25:59:57before insert into silver the product
26:00:00info and then we have to go and list all
26:00:02the columns. I've just prepared those
26:00:04columns. So with that we can go and now
26:00:07run our query in order to insert the
26:00:09data. So now as you can see this did
26:00:11insert the data and the very important
26:00:13step is now to check the quality of the
26:00:15silver table. So we go back to our data
26:00:17quality checks and we go switch to the
26:00:20silver. So let's check the primary key.
26:00:22There is no issues and we can go and
26:00:24check for example here the trims there
26:00:27is as well no issue and now let's go and
26:00:29check the costs it should not be
26:00:31negative or null which is perfect let's
26:00:34go and check the data standardizations
26:00:36as you can see they are friendly and we
26:00:38don't have any nulls and now very
26:00:40interesting the order of the dates so
26:00:43let's go and check that as you can see
26:00:45we don't have any issues and finally
26:00:47what I do I go and have a final look to
26:00:50the silver table and As we can see
26:00:53everything is inserted correctly in the
26:00:55correct columns. So all those columns
26:00:57comes from the source system and the
26:00:59last one is automatically generated from
26:01:01the DDL indicate when we loaded this
26:01:04table. Now let's sit back and have a
26:01:06look to our script. What are the
26:01:07different types of data transformations
26:01:09that we have done here is for example
26:01:10over here the category ID and the
26:01:12product key we have derived new columns.
26:01:15So it is when we create a new column
26:01:17based on calculations or transformations
26:01:20of an existing one. So sometimes we need
26:01:22columns only for analytics and we cannot
26:01:25each time go to the source system and
26:01:26ask them to create it. So instead of
26:01:28that we derive our own columns that we
26:01:31need for the analytics. Another
26:01:33transformation we have is the is null
26:01:35over here. So we are handling here
26:01:37missing information. Instead of null
26:01:39we're going to have a zero. And one more
26:01:40transformation we have over here for the
26:01:42product line. We have done here data
26:01:45normalization. Instead of having a code
26:01:47value we have a friendly value. And as
26:01:49well we have handled the missing data.
26:01:51For example, over here instead of having
26:01:53a null, we're going to have not
26:01:55available. All right, moving on to
26:01:56another data transformation. We have
26:01:58done data type casting. So we are
26:02:00converting the data type from one to
26:02:02another. And this considered as well to
26:02:04be a data transformation. And now moving
26:02:06on to the last one. We are doing as well
26:02:08data type casting. But what's more
26:02:10important, we are doing data enrichment.
26:02:12This type of transformation, it's all
26:02:14about adding a value to your data. So we
26:02:17are adding new relevant data to our data
26:02:20sets. So those are the different types
26:02:21of data transformations that we have
26:02:24done for this
26:02:28table. Okay. So let's keep going. We
26:02:30have the sales details and this is the
26:02:32last table in the CRM. So what do we
26:02:33have over here? We have the order number
26:02:35and this is a string. Of course we can
26:02:36go and check whether we have an issue
26:02:38with the unwanted spaces. So we can
26:02:40search whether we're going to find
26:02:42something. So we can say trim and
26:02:45something like this. and let's go and
26:02:46execute it. So we can see that we don't
26:02:48have any unwanted spaces. That means we
26:02:50don't have to transform this column. So
26:02:51we can leave it as it is. Now the next
26:02:53two columns they are like keys and ids
26:02:56in order to connect it with the other
26:02:58tables. As we learned before we are
26:02:59using the product key in order to
26:03:02connect it with the product informations
26:03:04and we are connecting the customer ID
26:03:06with the customer ID from the customer
26:03:07info. So that means we have to go and
26:03:09check whether everything is working
26:03:10perfectly. So we can go and check the
26:03:12integrity of those columns where we say
26:03:14the product key not in and then we make
26:03:17a subquery and this time we can work
26:03:19with the silver layer right so we can
26:03:21say the product key from silver dot
26:03:25product info so let's go and query this
26:03:27and as you can see we are not getting
26:03:28any issue that means all the product
26:03:30keys from the sales details can be used
26:03:33and connected with the product info the
26:03:35same thing we can go and check the
26:03:36integrity of the customer ID and we can
26:03:39use not the product we and go to the
26:03:41customer info and the name was CST ID.
26:03:45So let's go and query that and the same
26:03:47thing we don't have here any issues. So
26:03:49that means we can go and connect the
26:03:50sales with the customers using the
26:03:52customer ID and we don't have to do any
26:03:54transformations for it. So things looks
26:03:56really nice for those three columns. Now
26:03:58we come to the challenging one. We have
26:04:00here the dates. Now those dates are not
26:04:03actual dates. They are integer. So those
26:04:05are numbers and we don't want to have it
26:04:07like this. We would like to clean that
26:04:09up. we have to change the data type from
26:04:11integer to a dates. Now if you want to
26:04:13convert an integer to a date, we have to
26:04:15be careful with the values that we have
26:04:17inside each of those columns. So now
26:04:19let's check the quality for example of
26:04:21the order dates. Let's say where order
26:04:23dates is less than zero for example
26:04:26something negative. Well, we don't have
26:04:28any negative values which is good. Let's
26:04:29go and check whether we have any zeros.
26:04:32Well, this is bad. So we have here a lot
26:04:34of zeros. Now what we can do? We can
26:04:36replace those informations with a null.
26:04:38We can use of course the null if
26:04:40function like this. We can say null if
26:04:42and if it is zero then make it null. So
26:04:45let's execute it. And as you can see now
26:04:47all those informations are null. Now
26:04:49let's go and check again the data. So
26:04:51now this integer has the year's
26:04:53information at the start then the months
26:04:55and then the day. So here we have to
26:04:57have like 1 2 3 4 5. So the length of
26:05:00each number should be h. And if the
26:05:02length is less than eight or higher than
26:05:04eight then we have an issue. Let's go
26:05:06and check that. So we're going to say or
26:05:09length sales order is not equal to h
26:05:12that means less or higher. Let's go and
26:05:14execute it. Now let's go and check the
26:05:16results over here. And those two
26:05:18informations they don't look like a
26:05:20date. So we cannot go and make from
26:05:22these informations a real date. They are
26:05:24just bad data quality. And of course you
26:05:26can go and check the boundaries of a
26:05:28date. Like for example it should not be
26:05:31higher than for example let's go and get
26:05:33this value 2050 and then any for the
26:05:36month and the date. So let's go and
26:05:37execute it. And if we just remove those
26:05:39informations just to make sure. So we
26:05:41don't have any date that is outside of
26:05:43the boundaries that you have in your
26:05:45business. Or you go for example and say
26:05:47the boundary should be not less than
26:05:49depend when your business started. Maybe
26:05:51something like this. We are getting of
26:05:53course those values because they are
26:05:55less than null. But if you have values
26:05:57around this dates you will get it as
26:05:59well in the query. So we can go and add
26:06:01the rests. So all those checks like
26:06:03validate the column that has a date
26:06:05informations and it has the data type
26:06:07integer. So again what are the issues
26:06:09over here? We have zeros and sometimes
26:06:11we have like strange numbers that cannot
26:06:14be converted to a dates. So let's go and
26:06:16fix that in our query. So we can say
26:06:18case when the sales order the order
26:06:21dates is equal to zero or of the order
26:06:25date is not equal to 8 then null. Right?
26:06:28We don't want to deal with those values.
26:06:30they are just wrong and they they are
26:06:31not real dates otherwise we say else
26:06:34it's going to be the order date. Now
26:06:36what we're going to do we're going to go
26:06:37and convert this to a date. We don't
26:06:39want this as an integer. So how we can
26:06:41do that? We can go and cast it first to
26:06:43a varchar because we cannot cast from
26:06:46integer to date in SQL server. First you
26:06:49have to convert it to a varchchar and
26:06:51then from varchchar you go to a date.
26:06:53Well this is how we do it in SQL server.
26:06:55So we cast it first to a varchar and
26:06:58then we cast it to a date like this.
26:07:01That's it. So we have end and we are
26:07:03using the same column
26:07:05name. So this is how we transform an
26:07:08integer to a date. So let's go and query
26:07:11this. And as you can see the order date
26:07:13now is a real date. It is not a number.
26:07:16So we can go and get rid of the old
26:07:18column. Now we have to go and do the
26:07:19same stuff for the shipping dates. So,
26:07:21we can go over here and replace
26:07:24everything with the shipping date and
26:07:26let's go and query. Well, as you can
26:07:27see, the shipping date is perfect. We
26:07:29don't have any issue with this column.
26:07:31But still, I don't like that we found a
26:07:32lot of issues with the order date. So,
26:07:34what we're going to do just in case this
26:07:36happens for the shipping date in the
26:07:38future, I will go and apply the same
26:07:39rules to the shipping dates. Oh, let's
26:07:42take the shipping
26:07:43date like this. And if you don't want to
26:07:46apply it now, you have always to build
26:07:48like quality checks that runs every day
26:07:50in order to detect those issues. And
26:07:53once you detect it, then you can go and
26:07:55do the transformations. But for now, I'm
26:07:56going to apply it right away. So that is
26:07:58for the shipping date. Now we go to the
26:08:00due date and we will do the same test.
26:08:04Let's go and execute it. And as well, it
26:08:06is perfect. So still, I'm going to apply
26:08:09the same rules. So let's get the due
26:08:11date everywhere here in the query. Just
26:08:13make sure you don't miss anything here.
26:08:15So let's go and execute now. Perfect. As
26:08:18you can see, we have the order date,
26:08:20shipping date, and due date. And all of
26:08:21them are date and don't have any wrong
26:08:24data inside those columns. Now, still
26:08:26there is one more check that we can do
26:08:27and it's that the order date should be
26:08:30always smaller than the shipping date or
26:08:32the due date because it makes no sense,
26:08:34right? If you are delivering an item
26:08:36without an order. So first the order
26:08:38should happen then we are shipping the
26:08:40items. So there is like an order of
26:08:42those dates and we can go and check
26:08:43that. So we are checking now for invalid
26:08:45date orders where we can say the order
26:08:48date is higher than the shipping date or
26:08:52we are searching as well for an order
26:08:54where the order date is higher than the
26:08:56due date. So we can have it like this
26:08:58due date. So let's go and check. Well,
26:09:01that's really good. We don't have such a
26:09:03mistake on the data and the quality
26:09:05looks good. So the order date is always
26:09:07smaller than the shipping date or the
26:09:10due date. So we don't have to do any
26:09:12transformations or cleanup. Okay
26:09:13friends, now moving on to the last three
26:09:15columns. We have the sales, quantity and
26:09:18the price. All those informations are
26:09:20connected to each others. So we have a
26:09:21business rule or calculation. It says
26:09:24the sales must be equal to quantity
26:09:26multiplied by the price. And all sales
26:09:28quantity and price informations must be
26:09:31positive numbers. So it's not allowed to
26:09:33be negative, zero or null. So those are
26:09:36the business rules and we have to check
26:09:38the data consistency in our table. Does
26:09:40all those three informations following
26:09:42our rules? So we're going to start first
26:09:44with our rule, right? So we're going to
26:09:46say if the sales is not equal to
26:09:50quantity multiplied by the price. So we
26:09:53are searching where the result is not
26:09:55matching our expectation. And as well we
26:09:58can go and check other stuff like the
26:09:59nulls. So for example we can say or
26:10:01sales is null or quantity is
26:10:06null and the last one for the price and
26:10:10as well we can go and check whether they
26:10:12are negative numbers or zero. So we can
26:10:14go over here and say less or equal to
26:10:17zero and apply it for the other columns
26:10:19as well. So with that we are checking
26:10:22the calculation and as well we are
26:10:24checking whether we have null, zero or
26:10:25negative numbers. Let's go and check our
26:10:28informations. I'm going to have here
26:10:29extinct. So let's go and query it. And
26:10:32of course we have here bad data. But we
26:10:35can go and sort the data by the sales
26:10:38quantity and the price. So let's do it.
26:10:41Now by looking to the data we can see in
26:10:43the sales we have nulls. We have
26:10:45negative numbers and zeros. So we have
26:10:48all bad combinations and as well we have
26:10:50here bad calculations. So as you can see
26:10:52the price here is 50, the quantity is
26:10:54one but the sales is two which is not
26:10:56correct. And here we have as well wrong
26:10:58calculations. Here we have to have a 10
26:11:00and here nine or maybe the price is
26:11:02wrong. And by looking to the quantity
26:11:04now you can see we don't have any nulls.
26:11:06We don't have any zeros or negative
26:11:08numbers. So the quantity looks better
26:11:10than the sales. And if you look to the
26:11:12prices we have nulls we have negatives
26:11:14and yeah we don't have zeros. So that
26:11:17means the quality of the sales and the
26:11:18price is wrong. The calculation is not
26:11:21working and we have these scenarios. Now
26:11:23of course how I do it here I don't go
26:11:24and try now to transform everything on
26:11:26my own. I usually go and talk to an
26:11:29expert maybe someone from the business
26:11:31or from the source system and I show
26:11:32those scenarios and discuss and usually
26:11:34there is like two answers either they
26:11:36going to tell me you know what I will
26:11:38fix it in my source so I have to live
26:11:40with it there is incoming bad data and
26:11:42the bad data going to be presented in
26:11:44the warehouse until the source system
26:11:46clean up those issues. And the other
26:11:48answer you might get you know what we
26:11:49don't have the budget and those data are
26:11:51really old and we are not going to do
26:11:53anything. So here you have to decide
26:11:55either you leave it as it is or you say
26:11:57you know what let's go and improve the
26:11:59quality of the data. But here you have
26:12:00to ask for the experts to support you
26:12:03solving these issues because it really
26:12:05depend on the rules. Different rules
26:12:07makes different transformations. So now
26:12:09let's say that we have the following
26:12:11rules. If the sales informations are
26:12:13null or negative or zero, then use the
26:12:16calculation the formula by multiplying
26:12:18the quality with the price. And now if
26:12:20the prices are wrong, for example, we
26:12:22have here a null or zero, then go and
26:12:24calculate it from the sales and the
26:12:26quantity. And if you have a price that
26:12:28is a minus like minus 21, a negative
26:12:31number, then you have to go and convert
26:12:33it to a 21. So from a negative to a
26:12:36positive without any calculations. So
26:12:38those are the rules and now we're going
26:12:39to go and build the transformations.
26:12:41based on those rules. So let's do it
26:12:43step by step. I will go over here and
26:12:45we're going to start building the new
26:12:46sales. So what is the rule says case
26:12:49when of course as usual if the
26:12:52sales is null or let's say the sales is
26:12:56negative number or equal to zero or
26:12:59another scenario we have a sales
26:13:00information but it is not following the
26:13:03calculation. So we have wrong
26:13:04information in the sales. So we're going
26:13:06to say the sales is not equal to the
26:13:09quantity multiplied by the price. But of
26:13:13course we will not leave the price like
26:13:14this by using the function APS. The
26:13:17absolute is going to go and convert
26:13:19everything from negative to a positive.
26:13:21Then what we have to do is to go and use
26:13:23the calculation. So it going to be the
26:13:26quantity multiplied by the price. So
26:13:29that means we are not using the value
26:13:31that's come from the source system. We
26:13:33are recalculating it. Now let's say the
26:13:35sales is correct and not one of those
26:13:37scenarios. So we're going to say else.
26:13:39We will go with the sales as it is that
26:13:41comes from the source because it is
26:13:43correct. It's really nice. Let's go and
26:13:45say an end and give it the same name. I
26:13:47will go and rename the old one here as
26:13:50an old value and the same for the price.
26:13:53The quantity will not touch it because
26:13:56it is correct. So like this. And now
26:13:58let's go and transform the prices. So
26:14:00again as usual we go with case when. So
26:14:03what are the scenarios? The price is
26:14:06null or the price is less or equal to
26:14:10zero. Then what we going to do? We're
26:14:12going to do the calculation. So it's
26:14:14going to be the sales divided by the
26:14:16quantity the SLS quantity. But here we
26:14:20have to make sure that we are not
26:14:21dividing by zero. Currently we don't
26:14:23have any zeros in the quantity but you
26:14:24don't know in the future you might get a
26:14:26zero and the whole code going to break.
26:14:28So what you have to do is to go and say
26:14:30if you get any zero replace it with a
26:14:33null. So null if if it is zero then make
26:14:36it null. So that's it. Now if the price
26:14:38is not null and the price is not
26:14:40negative or equal to zero then
26:14:42everything is fine and that's why we're
26:14:44going to have now the else it going to
26:14:46be the price as it is from the source
26:14:48system. So that's it. We're going to say
26:14:50end as price. So I'm totally happy with
26:14:53that. Let's go and execute it and check
26:14:55of course. So those are the old
26:14:57informations and those are the new
26:14:59transformed cleaned up informations. So
26:15:00here previously we have a null but now
26:15:02we have two. So two multiplied with one
26:15:05we are getting two. So the sales is here
26:15:07correct. Now moving on to the next one
26:15:09we have in the sales 40 but the price is
26:15:12two. So two multiplied with one we
26:15:14should get two. So the new sales is
26:15:16correct. It is two and not 40. Now to
26:15:18the next one over here the old sales is
26:15:20zero. But if you go and multiply the
26:15:22four with the quantity you will get
26:15:24four. So the sales here is not correct.
26:15:26That's why in the new sales we have it
26:15:28correct as a four. And let's go and get
26:15:30a minus. So in this case we have a minus
26:15:32which is not correct. So we are getting
26:15:34the price multiplied with one. We should
26:15:36get here a nine. And this sales here is
26:15:39correct. Now let's go and get a scenario
26:15:41where the price is null like this here.
26:15:44So we don't have here a price but we
26:15:45calculated from the sales and the
26:15:47quantity. So we divided the 10 by two
26:15:49and we have five. So the new price is
26:15:52better. And the same thing for the
26:15:54minuses. So we have here minus 21 and in
26:15:56the output we have 21 which is correct.
26:15:58So for now I don't see any scenario
26:16:01where the data is wrong. So everything
26:16:03looks better than before. And with that
26:16:05we have applied the business rules from
26:16:07the experts and we have cleaned up the
26:16:09data in the data warehouse. And this is
26:16:12way better than before because we are
26:16:13presenting now better data for analyszis
26:16:16and reporting but it is challenging and
26:16:18you have exactly to understand the
26:16:20business. So now what we're going to do
26:16:21we're going to go and copy those
26:16:22informations and integrate it in our
26:16:24query. So instead of sales we're going
26:16:27to get our new calculation and instead
26:16:30of the price we will get our correct
26:16:32calculation and here I'm missing the
26:16:35end. Let's go and run the whole thing
26:16:37again. So with that we have as well now
26:16:40cleaned sales quantity and price and it
26:16:43is following our business rules. So with
26:16:45that we are done cleaning up the sales
26:16:47details. The next step we're going to go
26:16:48and insert it to the sales details. But
26:16:51we have to go and check again the DDL.
26:16:53So now all what you have to do is to
26:16:54compare those results with the DDL. So
26:16:57the first one is the order number. It's
26:16:58fine. The product key, the customer ID,
26:17:01but here we have an issue. All those
26:17:02informations now are date and not an
26:17:04integer. So we have to go and change the
26:17:07data type. And with that we have better
26:17:09data type than before. Then the sales
26:17:11quantity price it is correct. Let's go
26:17:13and drop the table and create it from
26:17:15scratch again. And don't forget to
26:17:17update your DDL script. So that's it for
26:17:20this. And we're going to go now and
26:17:21insert the results into our silver table
26:17:24sales details. And we have to go and
26:17:26list now all the columns. I have already
26:17:28prepared the list of all the columns. So
26:17:31make sure that you have the correct
26:17:32order of the columns. So let's go now
26:17:34and insert the data. And with that and
26:17:37with that we can see that the SQL did
26:17:39insert data to our sales details. But
26:17:41now very important is to check the
26:17:43health of the silver table. So what
26:17:45we're going to do instead here of
26:17:46bronze, we're going to go and switch it
26:17:48to silver. So let's check over here. So
26:17:51here always the order is smaller than
26:17:53the shipping and the due date, which is
26:17:55really nice. But now I'm very interested
26:17:58on the calculations. So here we're going
26:17:59to switch it from bronze to silver. And
26:18:01I'm going to go and get rid of all those
26:18:03calculations because we don't need it
26:18:06this. And now let's see whether we have
26:18:08any issue. Well, perfect. Our data is
26:18:11following the business rules. We don't
26:18:12have any nulls, negative values, zeros.
26:18:14Now as usual the last step the final
26:18:17check we will just have a final look to
26:18:19the table. So we have the order number
26:18:21the product key the customer ID those
26:18:23three dates we have the sales quantity
26:18:26and the price and of course we have our
26:18:29metadata column. Everything is perfect.
26:18:31So now by looking to our code what are
26:18:33the different types of data
26:18:34transformation that we are doing. So in
26:18:36those three columns we are doing the
26:18:38following. So at the start we are
26:18:39handling invalid data and this is as
26:18:42well type of transformation and as well
26:18:44at the same time we are doing data type
26:18:47casting. So we are changing it to more
26:18:49correct data type. And if you are
26:18:51looking to the sales over here then what
26:18:53we are doing over here is we are
26:18:55handling the missing data and as well
26:18:57the invalid data by deriving the column
26:19:00from already existing one. And it is as
26:19:03well very similar for the price. We are
26:19:05handling as well the invalid data by
26:19:08deriving it from specific calculation
26:19:10over here. So those are the different
26:19:12types of data transformations that you
26:19:14have done in these
26:19:18scripts. All right. Now let's keep
26:19:20moving to the next system. We have the
26:19:22customer AZ2. So here we have like only
26:19:25three columns and let's start with the
26:19:27ID first. So here again we have the
26:19:29customer's informations and if we go and
26:19:31check again our model you can see that
26:19:34we can connect this table with the CRM
26:19:36table customer info using the customer
26:19:39key. So that means we have to go and
26:19:40make sure that we can go and connect
26:19:42those two tables. So let's go and check
26:19:46the other table. We can go and check of
26:19:47course the server layer. So let's query
26:19:50it and we can query both of the tables.
26:19:53Now we can see there is here like extra
26:19:55characters that are not included in the
26:19:57customer key from the CRM. So let's go
26:20:00and search for example for this customer
26:20:02over here where C ID like so we are
26:20:08searching for customer has similar ID.
26:20:10Now as you can see we are finding this
26:20:12customer but the issue is that we have
26:20:14those three characters NAS. There is no
26:20:17specifications or explanation why we
26:20:19have the NAS. So actually what we have
26:20:21to do is to go and remove those
26:20:22informations. We don't need it. So let's
26:20:24again check the data. So it looks like
26:20:27the old data have an NAS at the start
26:20:29and then afterward we have new data
26:20:31without those three characters. So we
26:20:33have to clean up those ids in order to
26:20:35be able to connect it with other tables.
26:20:38So we're going to do it like this. We're
26:20:39going to start with the case when since
26:20:41we have like two scenarios in our data.
26:20:43So if the C ID is like the three
26:20:47characters in as so if the ID start with
26:20:50those three characters then we're going
26:20:52to go and apply transformation function
26:20:54otherwise it's going to stay like it is.
26:20:58So that's it. So now we have to go and
26:21:00build the transformation. So we're going
26:21:02to use substring and then we have to
26:21:05define the string. It's going to be the
26:21:07CD and then we have to define the
26:21:09position where it start cutting or
26:21:11extracting. So we can say 1 2 3 and then
26:21:15four. So we have to define the position
26:21:17number four. And then we have to define
26:21:19the string how many characters should be
26:21:21extracted. I will make it dynamic. So I
26:21:24will go with the length. I will not go
26:21:26and count how much. So we're going to
26:21:27say the C ID. So it looks good. If it's
26:21:30like NAS then go and extract from the CD
26:21:34at the position number four the rest of
26:21:36the characters. So let's go and execute
26:21:38it. And I'm missing here a comma again
26:21:41where we don't have any NAS at the
26:21:43start. And if you scroll down you can
26:21:46see those as well are not affected. So
26:21:48with that we have now a nice ID to be
26:21:51joined with other table. Of course we
26:21:53can go and test it like this where then
26:21:55we take the whole thing the whole
26:21:57transformation and say not in we remove
26:22:00of course the alias name we don't need
26:22:02it. And then we make very simple
26:22:04substring select
26:22:06distinct CST key the customer key from
26:22:10the silver table can be silver CRM cost
26:22:14info. So that's it. So let's go and
26:22:17check. So as you can see it is working
26:22:19fine. So we are not able to find any
26:22:21unmatching data between the customer
26:22:23info from ERB and the CRM. But of course
26:22:26after the transformation if you don't
26:22:28use the transformation. So if I just
26:22:30remove it like this, we will find a lot
26:22:32of unmatching data. So this means our
26:22:34transformation is working perfectly and
26:22:36we can go and remove the original value.
26:22:38So that's it for the first column. Okay.
26:22:41Now moving on to the next field, we have
26:22:43the birthday of the customers. So the
26:22:45first thing to do is to check the data
26:22:46type. It is a date. So it's fine. It is
26:22:49not an integer or a string. So we don't
26:22:51have to convert anything. But still
26:22:53there is something to check with the
26:22:55birth date. So we can check whether we
26:22:57have something out of range. So for
26:22:58example, we can go and check whether we
26:23:00have really old dates at the birth
26:23:02dates. So let's take 19, 100, and let's
26:23:05say 24 and we can take the first date of
26:23:08the month. So let's go and check that.
26:23:10Well, it looks like that we have
26:23:12customers that are older than 100 year.
26:23:14Well, I don't know. Maybe this is
26:23:16correct, but it sounds of course strange
26:23:18to do the business. Of course.
26:23:20Hey, this is Creed and he is in charge
26:23:24of something. That is correct. Say hi to
26:23:27the kids. Hi kids. Yay. And then we can
26:23:30go and check the other boundary where it
26:23:32is almost impossible to have a customer
26:23:35that the birthday is in the future. So
26:23:37we can say birth date is higher than the
26:23:41current date like this. So let's go and
26:23:44query this information. Well, it will
26:23:45not work because we have to have like an
26:23:47or between them. And now if we check the
26:23:49list over here, we have dates that are
26:23:52invalid for the birth dates. So all
26:23:54those dates they are all per day in the
26:23:57future and this is totally unacceptable.
26:23:59So this is an indicator for bad data
26:24:01quality. Of course you can go and report
26:24:03it to the source system in order to
26:24:04correct it. So here it's up to you what
26:24:06to do with those dates. Either leave it
26:24:08as it is as a bad data or we can go and
26:24:11clean that up by replacing all those
26:24:13dates with a null or maybe replacing
26:24:15only the one that is extreme where it is
26:24:18100% is incorrect. So let's go and write
26:24:21the transformation for that. As usual,
26:24:22we're going to start with case when
26:24:24birth date is larger than the current
26:24:28date and time then null. Otherwise, we
26:24:32can have an else where we have the birth
26:24:34date as it is and then we have an end as
26:24:38birth date. So, let's go and execute it.
26:24:41And with that, we should not get any
26:24:43customer where the birthday in the
26:24:45future. So, that's it for the birth
26:24:47date. Now, let's move to the next one.
26:24:49We have the gender. Now again the gender
26:24:51informations is low cardalities. So we
26:24:53have to go and check all the possible
26:24:55values inside this column. So in order
26:24:57to check all the possible values we're
26:24:59going to use select distinct gen from
26:25:02our table. So let's go and execute it.
26:25:04And now the data doesn't look really
26:25:06good. So we have here a null, we have an
26:25:08f, we have here an empty string, we have
26:25:11male, female, and again we have the M.
26:25:14So this is not really good. And what
26:25:15we're going to do, we're going to go and
26:25:16clean up all those informations in order
26:25:18to have only three values. Male, female,
26:25:21and not available. So, we're going to do
26:25:23it like this. We're going to say case
26:25:25when and now we're going to go and trim
26:25:26the values just to make sure there is
26:25:28like no empty spaces. And as well, I'm
26:25:31going to go and use the upper function
26:25:32just to make sure that in the future if
26:25:34we get any lower cases and so on, we are
26:25:37covering all the different scenarios. So
26:25:40case this is in F or let's say
26:25:44female then make it as female and we can
26:25:48go and do the same thing for the male
26:25:51like this. So if it is an M or a male
26:25:54make sure it is capital letters because
26:25:56here we are using the upper then it is a
26:25:58male otherwise all other scenarios it
26:26:01should be not available. So whether it
26:26:03is an empty string or nulls and so on.
26:26:06So we have to have an end of course as
26:26:09gen. So now let's go and test it and
26:26:10check whether we have covered
26:26:11everything. So you can see the M is now
26:26:13male. The empty is not available. The F
26:26:16is female. The empty string or maybe
26:26:19spaces here is not available. Female
26:26:21going to stay as it is. And the same for
26:26:23the male. So with that we are covering
26:26:25all the scenarios and we are following
26:26:27our standards in the project. So I'm
26:26:29going to go and cut this and put it in
26:26:31our original query over here. So let's
26:26:34go and execute the whole thing. And with
26:26:36that we have cleaned up all those three
26:26:39columns. Now the question is did we
26:26:40change anything in the DDL? Well we
26:26:42didn't change anything. We didn't
26:26:44introduce any new column or change any
26:26:46data type. So that means the next step
26:26:48is we're going to go and insert it in
26:26:50the server layer. So as usual we're
26:26:52going to say here insert into silver ERP
26:26:56the customer and then we're going to go
26:26:58and list all the column names. So C ID
26:27:01birth date and the gender. All right. So
26:27:04let's go and execute it. And with that
26:27:06we can see it inserted all the data. And
26:27:08of course the very important step as the
26:27:10next is to check the data quality. So
26:27:12let's go back to our query over here and
26:27:15change it from bronze to silver. So
26:27:17let's go and check the silver layer.
26:27:19Well of course we are getting those very
26:27:21old customers but we didn't change that.
26:27:24We only change the birthday that is in
26:27:26the future and we don't see it here in
26:27:28the results. So that means everything is
26:27:29clean. So for the next one, let's go and
26:27:32check the different genders. And as you
26:27:34can see, we have only those three
26:27:35values. And of course, we can go and
26:27:37take a final look to our table. So you
26:27:40can see the C ID here, the birth date,
26:27:42the gender, and then we see our metadata
26:27:44column. And everything looks amazing. So
26:27:47that's it. What are the different types
26:27:49of data transformations that we have
26:27:50done? First with the ID, what we have
26:27:52done, we have handled invalid values. So
26:27:55we have removed this part where it is
26:27:57not needed. And the same thing goes for
26:27:59the birth dates. We have handled as well
26:28:02invalid values. And then for the last
26:28:04one, for the gender, we have done data
26:28:06normalizations by mapping the code to
26:28:08more friendly value. And as well, we
26:28:10have handled the missing values. So
26:28:12those are the types that we have done in
26:28:14this
26:28:18code. Okay. Moving on to the second
26:28:20table, we have the location
26:28:22informations. So we have ERP location
26:28:25A101. So now here the task is easy
26:28:27because we have only two columns and if
26:28:29you go and check the integration model
26:28:30we can find our table over here. So we
26:28:33can go and connect it together with the
26:28:35customer info from the other system
26:28:36using a CID with the customer key. So
26:28:40those two informations must be matching
26:28:42in order to join the tables. So that
26:28:44means we have to go and check the data.
26:28:46So let's go and select the data CST key
26:28:50from let's go and get the silver data
26:28:52customer info. So let's go. Now if you
26:28:54go and check the result you can see over
26:28:56here that we have an issue with the CI
26:28:59ID there is like a minus between the
26:29:02characters and the numbers but the
26:29:03customer ID the customer number we don't
26:29:05have anything that splits the characters
26:29:08with the numbers. So if you go and join
26:29:10those two informations it will not be
26:29:11working. So what we have to do we have
26:29:13to go and get rid of this minus because
26:29:15it is totally unnecessary. So let's go
26:29:17and fix that. It's going to be very
26:29:19simple. So what we're going to do we're
26:29:20going to say CI ID. So we're going to go
26:29:22and search for the minus and replace it
26:29:25with nothing. It's very simple like
26:29:27this. So let's go and query it again.
26:29:30And with that things looks very similar
26:29:31to each others. And as well we can go
26:29:33and query it. So we're going to say
26:29:35where our transformation is not in then
26:29:39we can go and use this as a subquery
26:29:41like this. So let's go and execute it.
26:29:44And as you can see we are not finding
26:29:46any unmatching data now. So that means
26:29:48our transformation is working. And with
26:29:50that we can go and connect those two
26:29:52tables together. So if I take the
26:29:54transformation away you can see that we
26:29:56will find a lot of unmatching data. So
26:29:58the transformation is okay. We're going
26:30:00to stay with it. And now let's speak
26:30:02about the countries. Now we have here
26:30:04multiple values and so on. What I'm
26:30:06going to do this is low cardinality and
26:30:08we have to go and check all possible
26:30:10values inside this column. So that means
26:30:12we are checking whether the data is
26:30:14consistent. So we can do it like this.
26:30:17distinct the
26:30:19country from our table. I'm just going
26:30:21to go and copy it like this. And as
26:30:23well, I'm going to go and sort the data
26:30:25by the country. So, let's go and check
26:30:28the informations. Now, you can see we
26:30:30have a null. We have an empty string,
26:30:32which is really bad. And then we have a
26:30:34full name of country and then we have as
26:30:36well an abbreviation of the countries.
26:30:39Well, this is a mix. This is not really
26:30:41good because sometimes we have DE and
26:30:43sometimes we have Germany and then we
26:30:45have the United Kingdom and then for the
26:30:47United States we have like three
26:30:49versions of the same information which
26:30:51is as well not really good. So the
26:30:53quality of the country is not really
26:30:55good. So let's go and work on the
26:30:57transformation. As usual we're going to
26:30:58start with the case win. If trim
26:31:01country is equal to D, then we're going
26:31:06to transform it to Germany. And the next
26:31:09one it's going to be about the USA. So
26:31:11if trim country is in. So now let's go
26:31:15and get those two values the US and the
26:31:17USA. So US and USA then it's going to be
26:31:23the United States states. So with us we
26:31:26have covered as well those three cases.
26:31:29Now we have to talk about the null and
26:31:30the empty string. So we're going to say
26:31:32when trim country is equal to empty
26:31:36string or country is null then it's
26:31:41going to be not available otherwise I
26:31:44would like to get the country as it is.
26:31:45So trim country just to make sure that
26:31:48we don't have any leading or trailing
26:31:50spaces. So that's it. Let's go and say
26:31:53this is the country. So it is working
26:31:56and the country information is
26:31:58transformed. And now what I'm going to
26:31:59do, I'm going to take the whole new
26:32:00transformation and compare it to the old
26:32:03one. Let me just call this as old
26:32:07country and let's go and query it. So
26:32:09now we can check those values state as
26:32:11before. So nothing did change. The DE is
26:32:14now Germany. The empty string is not
26:32:16available. The null the same thing and
26:32:18the United Kingdom stayed as like it's
26:32:21like before. And now we have one value
26:32:23for all those information. So it's only
26:32:26the United States. So it looks perfect.
26:32:28And with that we have cleaned as well
26:32:30the second column. So with that we have
26:32:32now clean results. And now the question
26:32:34did we change anything in the DDL? Well
26:32:36we haven't changed anything. Both of
26:32:38them are varchar. So we can go now
26:32:40immediately and insert it into our
26:32:42table. So insert into silver customer
26:32:46location. And here we have to specify
26:32:48the columns. It's very simple the ID and
26:32:50the country. So let's go and execute it.
26:32:53And as you can see we got now inserted
26:32:55all those values. Of course, as a next,
26:32:57we go and double check those
26:32:59informations. I would just go and remove
26:33:01all those stuff as well here. And
26:33:03instead of bronze, let's go with the
26:33:05silver. So, as you can see, all the
26:33:08values of the country looks good. And
26:33:10let's have a final look to the table.
26:33:12So, like this. So, we have the ids
26:33:14without the separator. We have the
26:33:16countries and as well our metadata
26:33:19information. So, with that, we have
26:33:20cleaned up the data for the location.
26:33:22Okay. So now what are the different
26:33:23types of data transformation that we
26:33:25have done here is first we have handled
26:33:27invalid values. So we have removed the
26:33:30minus with an empty string and for the
26:33:32country we have done data normalization.
26:33:35So we have replaced codes with friendly
26:33:37values and as well at the same time we
26:33:40have handled missing values by replacing
26:33:42the empty string and null with not
26:33:44available. And one more thing of course
26:33:46we have removed the unwanted spaces. So
26:33:49those are the different types of
26:33:50transformation that we have done for
26:33:52this
26:33:55table. Okay guys, now keep the energy
26:33:58up, keep the spirit up. We have to go
26:34:00and clean up the last table in the
26:34:02bronze layer. And of course, we cannot
26:34:04go and skip anything. We have to check
26:34:06the quality and to detect all the
26:34:08errors. So now we have a table about the
26:34:10categories for the products. And here we
26:34:12have like four columns. Let's go and
26:34:14start with the first one, the ID. As you
26:34:16can see in our integration model, we can
26:34:18connect this table together with the
26:34:20product info from the CRM using the
26:34:22product key. And as you remember in the
26:34:24silver layer, we have created an extra
26:34:26column for that in the product info. So
26:34:29if you go and select those data, you can
26:34:31see we have a column called category ID
26:34:34and this one is exactly matching the ID
26:34:36that we have in this table and we have
26:34:38done the testing. So this ID is ready to
26:34:41be used together with the other table.
26:34:43So there is nothing to do over here. And
26:34:46now for the next columns they are
26:34:47string. And of course we can go and
26:34:49check whether there are any unwanted
26:34:51spaces. So we are checking for the
26:34:52unwanted spaces. So let's go and check
26:34:56select start from and we're going to go
26:34:58and get the same table like this here.
26:35:00And first we are checking the category.
26:35:02So the category is not equal to the
26:35:05category after trimming the unwanted
26:35:07spaces. So let's go and execute it. And
26:35:10as you can see we don't have any
26:35:12results. So there are no unwanted
26:35:14spaces. Let's go and check the other
26:35:15column. For example, the subcategory,
26:35:18the next one. So let's get the
26:35:20subcategory and run the query as well.
26:35:23We don't have anything. So that means we
26:35:25don't have unwanted spaces for the
26:35:27subcategory. Let's go now and check the
26:35:29last column. So I will just copy and
26:35:31paste. Now let's get the maintenance and
26:35:34let's go and execute. And as well, no
26:35:36results. Perfect. We don't have any
26:35:38unwanted spaces inside this table. So
26:35:41now the next step is that we're going to
26:35:42go and check the data standardizations
26:35:45because all those columns has low
26:35:47cardinality. So what we can do we can
26:35:49say
26:35:49select distinct let's get the cats
26:35:53category from our table. I'll just copy
26:35:56and paste it and check all values. So as
26:35:58you can see we have the accessories,
26:36:00bikes, clothing and components.
26:36:02Everything looks perfect. We don't have
26:36:04to change anything in this column. Let's
26:36:06go and check the subcategory. And if you
26:36:08scroll down, all values are friendly and
26:36:11nice as well. Nothing to change here.
26:36:14And let's go and check the last column,
26:36:15the maintenance. Perfect. We have only
26:36:18two values, yes and no. We don't have
26:36:20any nulls. So my friends, that's means
26:36:22this table has really nice data quality
26:36:25and we don't have to clean up anything.
26:36:27But still, we have to follow our
26:36:29process. We have to go and load it from
26:36:31the bronze to the silver even if we
26:36:33didn't transform anything. So our job is
26:36:35really easy. Here we're going to go and
26:36:37say insert into silver dot ERP px and so
26:36:42on. And we're going to go and define the
26:36:44columns. So it's going to be the ID, the
26:36:46category,
26:36:48subcategory, maintenance. So that's it.
26:36:50Let's go and insert the data. Now, as
26:36:52usual, what we're going to do, we're
26:36:53going to go and check the data. So
26:36:56silver
26:36:58ERP. Let's have a look. All right. So we
26:37:01can see the ids are here, the
26:37:02categories, the subcategories, the
26:37:04maintenance and we have our meta column.
26:37:07So everything is inserted correctly. All
26:37:10right. So now I have all those queries
26:37:12and the insert statements for all six
26:37:14tables. And now what is important before
26:37:17inserting any data, we have to make sure
26:37:19that we are truncating and emptying the
26:37:22table because if you run this query
26:37:23twice, what's going to happen? You will
26:37:25be inserting duplicates. So first
26:37:27truncate the data and then do a full
26:37:30load insert all data. So we're going to
26:37:32have one step before it's like the
26:37:34bronze layer. We're going to say
26:37:35truncate table and then we will be
26:37:37truncating the silver customer info and
26:37:39only after that we have to go and insert
26:37:42the data. And of course we can go and
26:37:43give this nice information at the start.
26:37:46So first we are truncating the table and
26:37:48then inserting. So if I go and run the
26:37:50whole thing. So let's go and do it. It
26:37:52will be working. So if I can run it
26:37:54again, we will not have any duplicates.
26:37:56So we have to go and add this step
26:37:58before each insert. So let's go and do
26:38:00that. All right. So I'm done with all
26:38:03tables. So now let's go and run
26:38:06everything. So let's go and execute it.
26:38:08And we can see in the messaging
26:38:10everything working perfectly. So with
26:38:12that we made all tables empty. And then
26:38:13we inserted the
26:38:18data. So perfect. With that we have a
26:38:20nice script that loads the silver layer.
26:38:23But of course like the front layer,
26:38:25we're going to put everything in one
26:38:26stored procedure. So let's go and do
26:38:28that. We'll go to the beginning over
26:38:30here and say create or alter procedure
26:38:34and we're going to put it in the schema
26:38:36silver and using the naming convention
26:38:38load silver and we're going to go over
26:38:40here and say begin and take the whole
26:38:42code end it is long one and give it one
26:38:45push with a tab and then at the end
26:38:47we're going to say edge. Perfect. So we
26:38:50have our stored procedure but we forgot
26:38:52here the ass with that we will not have
26:38:54any error. Let's go and execute it. So
26:38:56the stored procedure is created. If you
26:38:58go to the programmability and you will
26:39:00find two procedures load bronze and load
26:39:03silver. So now let's go and try it out.
26:39:05All what you have to do is now only to
26:39:07execute the silver load silver. So let's
26:39:11execute the start procedure and with
26:39:13that we will get the same results. This
26:39:15third procedure now is responsible of
26:39:18loading the whole silver layer. Now of
26:39:20course the messaging here is not really
26:39:23good because we have learned in the
26:39:24bronze layer we can go and add many
26:39:26stuff like handling the error doing nice
26:39:29messaging catching the duration time. So
26:39:32now your task is to pause the video take
26:39:34this start procedure and go and
26:39:36transform it to be very similar to the
26:39:38bronze layer with the same messaging and
26:39:41all the add-ons that we have added. So
26:39:43pause the video now. I will do it as
26:39:44well offline and I will see you
26:39:49[Music]
26:39:51soon. Okay. So I hope you are done and I
26:39:54can show you the results. It's like the
26:39:56bronze layer. We have defined at the
26:39:57start few variables in order to catch
26:40:00the duration. So we have the start time,
26:40:02the end time, patch start time and patch
26:40:04end time. And then we are printing a lot
26:40:06of stuff in order to have like nice
26:40:08messaging in the output. So at the start
26:40:10we are saying loading the server layer
26:40:12and then we start splitting by the
26:40:14source system. So loading the CRM tables
26:40:17and I'm going to show you only one table
26:40:18for now. So we are setting the timer. So
26:40:21we are saying start time get the date
26:40:23and time informations to it. Then we are
26:40:25doing the usual. We are truncating the
26:40:27table and then we are inserting the new
26:40:29informations after cleaning it up. And
26:40:32we have this nice message. We will say
26:40:34load duration where we are finding the
26:40:36differences between the start time and
26:40:38the end time using the function date
26:40:40diff. And we want to show the result in
26:40:42the seconds. So we are just printing how
26:40:45long it took to load this table. And
26:40:47we're going to go and repeat this
26:40:48process for all the tables. And of
26:40:50course we are putting everything in try
26:40:52and catch. So the SQL going to go and
26:40:55try to execute the try part. And if
26:40:57there are any issues the SQL going to go
26:41:00and execute the catch. And here we are
26:41:02just printing few information like the
26:41:04error message the error number and the
26:41:06error states. And we are following
26:41:07exactly the same standard at the bronze
26:41:10layer. So let's go and execute the whole
26:41:13thing. And with that we have updated the
26:41:15definition of the third procedure. Let's
26:41:17go now and execute it. So execute silver
26:41:20dot load silver. So let's go and do
26:41:23that. It went very fast like fewer than
26:41:251 seconds again because we are working
26:41:27on local machine loading the server
26:41:29layer loading the CRM tables and we can
26:41:32see this nice messaging. So it start
26:41:34with truncating the table inserting the
26:41:36data and we are getting the load
26:41:38duration for this table and you will see
26:41:40that everything is below 1 second and
26:41:42that's because in real projects you will
26:41:44get of course more than 1 second. So at
26:41:47the end we have load duration of the
26:41:49whole silver layer. And now I have one
26:41:51more thing for you. Let's say that you
26:41:53are changing the design of this store
26:41:55procedure for the server layer. You are
26:41:57adding different types of messaging or
26:41:59maybe you're creating logs and so on. So
26:42:01now all those new ideas and redesigns
26:42:04that you are doing for the silver layer,
26:42:05you have always to think about bringing
26:42:08the same changes as well in the other
26:42:10store procedure for the pros layer. So
26:42:12always try to keep your codes following
26:42:14the same standards. Don't have like one
26:42:16idea in one store procedure and an old
26:42:19idea in another one. Always try to
26:42:21maintain those scripts and to keep them
26:42:23all up to date following the same
26:42:25standards. Otherwise, it can be really
26:42:27hard for other developers to understand
26:42:29the cause. I know that needs a lot of
26:42:31work and commitments, but this is your
26:42:33job to make everything following the
26:42:35best practices and following the same
26:42:37naming convention and standards that you
26:42:39put for your projects. So guys, now we
26:42:42have very nice two ETL scripts. One that
26:42:45loads the bronze layer and another one
26:42:47for the server layer. So now our data
26:42:48warehouse is very simple. All what you
26:42:50have to do is to run first the bronze
26:42:53layer and with that we are taking all
26:42:54the data from the CSV files from the
26:42:57source and we put it inside our data
26:42:59warehouse in the bronze layer and with
26:43:01that we are refreshing the whole bronze
26:43:03layer. Once it's done the next step is
26:43:06to run the store procedure of the server
26:43:08layer. So once you execute it you are
26:43:10taking now all the data from the bronze
26:43:12layer transforming it cleaning it up and
26:43:15then loading it to the server layer. And
26:43:18as you can see the concept is very
26:43:19simple. We are just moving the data from
26:43:21one layer another layer with different
26:43:23tasks. All right guys, so as you can see
26:43:25in the server layer we have done a lot
26:43:27of data transformations and we have
26:43:29covered all the types that we have in
26:43:31the data cleansing. So we remove
26:43:33duplicates, data filtering, handling
26:43:35missing data, invalid data, unwanted
26:43:37spaces, casting the data types and so
26:43:39on. And as well we have derived new
26:43:41columns, we have done data enrichment
26:43:44and we have normalized a lot of data. So
26:43:46now of course what we have not done yet
26:43:48business rules and logic data
26:43:50aggregations and data integration. This
26:43:52is for the next layer. All right my
26:43:54friends. So finally we are done cleaning
26:43:56up the data and checking the quality of
26:43:58our data. So we can go and close those
26:44:01two steps. And now to the next step we
26:44:03have to go and extend the data flow
26:44:05diagram. So let's
26:44:09go. Okay. So now let's go and extend our
26:44:12data flow for the silver layer. So, what
26:44:15I'm going to do, I'm just going to go
26:44:16and copy the whole thing and put it side
26:44:19by side to the bronze layer. And let's
26:44:22call it silver layer. And the table name
26:44:25is going to stay as before because we
26:44:26have like one to one like the bronze
26:44:29layer. But what we're going to do, we're
26:44:30going to go and change the coloring. So,
26:44:31I'm going to go and mark everything and
26:44:33make it gray like silver. And of course,
26:44:36what is very important is to make the
26:44:38lineage. So, I'm going to go now from
26:44:40the bronze and take an arrow and put it
26:44:43to the silver table. And now with that
26:44:45we have like a lineage between three
26:44:47layers and you are checking this table
26:44:48the customer info you can understand aha
26:44:51this comes from the bronze layer from
26:44:53the customer info and as well this comes
26:44:55from the source system CRM so now we can
26:44:58see the lineage between different layers
26:45:01and without looking to any scripts and
26:45:03so on in one picture you can understand
26:45:05the whole projects so I don't have to
26:45:07explain a lot of stuff by just looking
26:45:09to this picture you can understand how
26:45:11the data is flowing between sources is
26:45:14bronze layer, silver layer, and to the
26:45:16gold layer, of course, later. So, as you
26:45:18can see, it looks really nice and clean.
26:45:20All right. So, with that, we have
26:45:21updated the data flow. Next, we're going
26:45:24to go and commit our work in the G repo.
26:45:26So, let's
26:45:30go. Okay. So, now let's go and commit
26:45:32our scripts. We're going to go to the
26:45:34folder scripts. And here we have a
26:45:36server layer. If you don't have it, of
26:45:38course, you can go and create it. So,
26:45:39first we're going to go and put the DDL
26:45:41scripts for the server layer. So let's
26:45:43go and I will paste the code over here.
26:45:46And as usual, we have this commit as the
26:45:48header explaining the purpose of this
26:45:50script. So let's go and commit our work.
26:45:53And we're going to do the same thing for
26:45:55the store procedure that loads the
26:45:57server layer. So I'm going to go over
26:45:59here. I have already filed for that. So
26:46:01let's go and paste that. So we have here
26:46:04our stored procedures. And as usual at
26:46:06the start, we have as well. So this
26:46:08script is doing the ATL process where we
26:46:10load the data from bronze into silver.
26:46:13So the action is to truncate the table
26:46:15first and then insert transformed cleans
26:46:17data from bronze to silver. There are no
26:46:20parameters at all. And this is how you
26:46:22can use the source procedure. Okay. So
26:46:24we're going to go and commit our work.
26:46:27And now one more thing that we want to
26:46:29commit in our project all those queries
26:46:31that you have built to check the quality
26:46:33of the server layer. So this time we
26:46:35will not put it in the scripts. We're
26:46:36going to go to the tests and here we're
26:46:38going to go and make a new file called
26:46:40quality checks silver and inside it
26:46:42we're going to go and paste all the
26:46:44queries that we have filled. I just here
26:46:46reorganize them by the tables. So here
26:46:49we can see all the checks that we have
26:46:51done during the course and at the header
26:46:54we have here nice comments. So here we
26:46:56are just saying that this script is
26:46:57going to check the quality of the server
26:46:59layer and we are checking for nulls,
26:47:01duplicates, unwanted spaces, invalid
26:47:03date range and so on. So that each time
26:47:06you come up with a new quality check,
26:47:08I'm going to recommend you to share it
26:47:09with the project and with other team in
26:47:11order to make it part of multiple checks
26:47:14that you do after running the ATL. So
26:47:17that's it. I'm going to go and put those
26:47:18checks in our repo and in case I come up
26:47:22with new check, I'm going to go and
26:47:23update it. Perfect. So now we have our
26:47:25code in our repository. All right. So
26:47:28with that, our code is saved and we are
26:47:30done with the whole epic. So we have
26:47:32built the silver layer. Now let's go and
26:47:35minimize it. And now we come to my
26:47:37favorite layer, the code layer. So we're
26:47:39going to go and build it. The first step
26:47:41as usual, we have to analyze. And this
26:47:43time we're going to explore the business
26:47:44objects. So let's
26:47:48go. All right. So now we come to the big
26:47:51question. How we going to build the gold
26:47:52layer? As usual, we start with
26:47:54analyzing. So now what we're going to do
26:47:56here is to explore and understand what
26:47:58are the main business objects that are
26:48:00hidden inside our source system. So as
26:48:02you can see we have two sources six
26:48:04files and here we have to identify what
26:48:06are the business objects. Once we have
26:48:07this understanding then we can start
26:48:09coding and here the main transformation
26:48:11that we are doing is data integration.
26:48:13And here usually I split it into three
26:48:15steps. The first one we're going to go
26:48:17and build those business objects that we
26:48:19have identified. And after we have a
26:48:21business objects we have to look at it
26:48:23and decide what is the type of this
26:48:25table. Is it a dimension? Is it a fact?
26:48:28Or is it like maybe a flat table? So
26:48:30what type of table that we have built
26:48:32and the last step is of course we have
26:48:34now to rename all the columns into
26:48:36something friendly and easy to
26:48:38understand so that our consumers don't
26:48:40struggle with technical names. So once
26:48:41we have all those steps what we're going
26:48:43to do it's time to validate what we have
26:48:44created. So what we have to do the new
26:48:46data model that we have created it
26:48:48should be connectable and we have to
26:48:50check that the data integration is done
26:48:52correctly and once everything is fine we
26:48:54cannot skip the last step. we have to
26:48:56document and as well commit our work in
26:48:58the g. And here we will be introducing a
26:49:00new type of documentations. So we're
26:49:02going to have a diagram about the data
26:49:04model. We're going to build a data
26:49:06dictionary where we're going to describe
26:49:07the data model. And of course we're
26:49:09going to extend the data flow diagram.
26:49:11So this is our process. Those are the
26:49:13main steps that we will do in order to
26:49:15build the code
26:49:19layer. Okay. So what is exactly data
26:49:22moduling? Usually the source system
26:49:23going to deliver for you row data
26:49:25unorganized messy not very useful in its
26:49:29current states. But now the data
26:49:31modeling is the process of taking this
26:49:33row data and then organize it and
26:49:35structure it in meaningful way. So what
26:49:38we are doing we are putting the data in
26:49:40new friendly and easy to understand
26:49:43objects like customers, orders,
26:49:45products. Each one of them is focused on
26:49:47specific information and what is very
26:49:49important is we're going to describe the
26:49:51relationship between those objects. So
26:49:54by connecting them using lines. So what
26:49:56you have built on the right side we call
26:49:58it logical data model. If you compare to
26:50:00the left side you can see the data model
26:50:02makes it really easy to understand our
26:50:03data and the relationship the processes
26:50:06behind them. Now in data modeling we
26:50:07have three different stages or let's say
26:50:09three different ways on how to draw a
26:50:11data model. The first stage is the
26:50:13conceptual data model. Here the focus is
26:50:16only on the entity. So we have
26:50:18customers, orders, products and we don't
26:50:20go in details at all. So we don't
26:50:22specify any columns or attributes inside
26:50:24those boxes. We just want to focus what
26:50:26are the entities that we have and as
26:50:28well the relationship between them. So
26:50:30the conceptual data model don't focus at
26:50:33all on the details. It just gives the
26:50:35big picture. So the second data model
26:50:37that we can build is the logical data
26:50:39model. And here we start specifying what
26:50:41are the different columns that we can
26:50:43find in each entity like we have the
26:50:45customer ID the first name last name and
26:50:48so on and we still draw the relationship
26:50:50between those entities and as well we
26:50:51make it clear which columns are the
26:50:53primary key and so on. So as you can see
26:50:55we have here more details but one thing
26:50:57we don't describe a lot of details for
26:50:59each column and we are not worry how
26:51:01exactly we going to store those tables
26:51:03in the database. The third and last
26:51:05stage we have the physical data model.
26:51:08This is where everything gets ready
26:51:09before creating it in the database. So
26:51:12here you have to add all the technical
26:51:13details like adding for each column the
26:51:16data types and the length of each data
26:51:18type and many other database techniques
26:51:21and details. So again if you look to the
26:51:23conceptual data model it gives us the
26:51:25big picture and in the logical data
26:51:26model we dive into details of what data
26:51:29we need and the physical layer model
26:51:31prepares everything for the
26:51:33implementation in the database. And to
26:51:35be honest in my projects I only draw the
26:51:37conceptual and the logical data model
26:51:39because drawing and building the
26:51:41physical data model needs a lot of
26:51:43efforts and time and there are many
26:51:45tools like in data bricks they
26:51:46automatically generate those models. So
26:51:49in this project what we're going to do
26:51:50we're going to draw the logical data
26:51:52model for the gold
26:51:57layer. All right. It's now for analytics
26:51:59and especially for data warehousing and
26:52:01business intelligence. We need a special
26:52:03data model that is optimized for
26:52:05reporting and analytics and it should be
26:52:08flexible, scalable and as well easy to
26:52:10understand. And for that we have two
26:52:12special data models. The first type of
26:52:14data model we have the star schema. It
26:52:16has a central fact table in the middle
26:52:18and surrounded by dimensions. The fact
26:52:20table contains transactions, events, and
26:52:23the dimensions contains descriptive
26:52:25informations. And the relationship
26:52:26between the fact table in the middle and
26:52:28the dimensions around it forms like a
26:52:31star shape. And that's why we call it
26:52:33star schema. And we have another data
26:52:34model called snowflake schema. It looks
26:52:37very similar to the star schema. So we
26:52:39have again the fact in the middle and
26:52:41surrounded by dimensions. But the big
26:52:43difference is that we break the
26:52:45dimensions into smaller subdimensions.
26:52:48And the shape of this data model as you
26:52:50are extending the dimensions it's going
26:52:52to looks like a snowflake. So now if you
26:52:54compare them side by side you can see
26:52:55that the star schema looks easier right?
26:52:58So it is usually easy to understand easy
26:53:00to query it is really perfect for
26:53:02analyzers but it has one issue with the
26:53:05dimension might contain duplicates and
26:53:07your dimensions get bigger with the
26:53:09time. Now if you compare it to the
26:53:10snowflake you can see the schema is more
26:53:13complex. You saw you need a lot of
26:53:15knowledge and efforts in order to query
26:53:17something from the snowflake. But the
26:53:19main advantage here comes with the
26:53:20normalization as you are breaking those
26:53:22redundancies in small tables. You can
26:53:24optimize the storage. But to be honest,
26:53:26who care about the storage? So for this
26:53:28project, I have chose to use the star
26:53:30schema because it is very commonly used.
26:53:32Perfect for reporting like for example
26:53:34if you're using PowerBI and we don't
26:53:37have to worry about the storage. So
26:53:38that's why we're going to adopt this
26:53:40model to build our gold layer.
26:53:46Okay. So now one more thing about those
26:53:47data models is that they contain two
26:53:50types of tables fact and dimensions. So
26:53:52when I say this is a fact table or a
26:53:54dimension table well the dimension
26:53:56contains descriptive informations or
26:53:58like categories that gives some context
26:54:00to your data. For example a product info
26:54:02you have product name, category,
26:54:04subcategories and so on. This is like a
26:54:06table that is describing the products
26:54:09and this we call it dimension. But in
26:54:11the other hand we have facts. They are
26:54:13events like transactions. They contain
26:54:15three important informations. First you
26:54:18have multiple ids from multiple
26:54:20dimensions. Then we have like date
26:54:22informations like when the transaction
26:54:24or the event did happen. And the third
26:54:27type of information you're going to have
26:54:28like measures and numbers. So if you see
26:54:30those three types of data in one table,
26:54:33then this is a fact. So if you have a
26:54:35table that answers how much or how many,
26:54:38then this is a fact. But if you have a
26:54:40table that answers who, what, where,
26:54:43then this is a dimension table. So this
26:54:45is what dimension and fact
26:54:50tables. All right my friends. So so far
26:54:52in the bronze layer and in the silver
26:54:54layer we didn't discuss anything about
26:54:56the business. So the bronze and silver
26:54:58were very technical. We are focusing on
26:55:00data ingestion. We are focusing on
26:55:02cleaning up the data quality of the
26:55:04data. But still the tables are very
26:55:06oriented to the source system. Now comes
26:55:09the fun part in the god layer where
26:55:11we're going to go and break the whole
26:55:12data model of the sources. So we're
26:55:14going to create something completely new
26:55:16to our business that is easy to consume
26:55:19for business reporting and analyzes. And
26:55:22here it is very important to have a
26:55:23clear understanding of the business and
26:55:25the processes. And if you don't know it
26:55:27already at this phase you have really to
26:55:29invest time by meeting maybe process
26:55:31experts, the domain experts in order to
26:55:34have clear understanding what we are
26:55:36talking about in the data. So now what
26:55:38we're going to do, we're going to try to
26:55:39detect what are the business objects
26:55:41that are hidden in the source systems.
26:55:44So now let's go and explore that. All
26:55:46right. Now in order to build a new data
26:55:48model, I have to understand first the
26:55:50original data model. What are the main
26:55:52business objects that we have? How
26:55:53things are related to each others? And
26:55:55this is very important process in
26:55:57building a new model. So now what I
26:55:59usually do, I start giving labels to all
26:56:02those tables. So if you go to the shapes
26:56:03over here, let's go and search for
26:56:05label. And if we go to more icons, I'm
26:56:08going to go and take this label over
26:56:09here. So, drag and drop it. And then I'm
26:56:11going to go and increase maybe the size
26:56:13of the font. So, let's go with 20 and
26:56:16bold. Just make it a little bit bigger.
26:56:18So, now by looking to this data model,
26:56:20we can see that we have product
26:56:22informations in the CRM and as well in
26:56:24the ARP. And then we have like customer
26:56:26informations and transactional table.
26:56:29So, now let's focus on the product. So,
26:56:31the product information is over here. We
26:56:33have here the current and the history
26:56:35product informations and here we have
26:56:37the categories that's belong to the
26:56:39products. So in our data model we have
26:56:41something called products. So let's go
26:56:42and create this label. It's going to be
26:56:44the product and let's go and give it a
26:56:47color to the style. Let's pick for
26:56:49example the red one. Now let's go and
26:56:52move this label and put it beneath this
26:56:54table over here. And with that I have
26:56:56like a label saying this table belongs
26:56:59to the objects called products. Now I'm
26:57:01going to do the same thing for the other
26:57:03table over here. So I'm going to go and
26:57:05tag this table to the product as well.
26:57:07So that I can see easily which tables
26:57:09from the sources does has informations
26:57:11about the product business object. All
26:57:14right. Now moving on, we have here a
26:57:15table called customer information. So we
26:57:18have a lot of information about the
26:57:19customer. We have as well in the ARP
26:57:21customer information where we have the
26:57:22birthday and the country. So those three
26:57:25tables has to do with the object
26:57:27customer. So that means we're going to
26:57:28go and label it like that. So let's call
26:57:30it customer and I'm going to go and pick
26:57:33different color for that. Let's go with
26:57:35the green. So I will tag this table like
26:57:38this. And the same thing for the other
26:57:40tables. So copy tag the second table and
26:57:44the third table. Now it is very easily
26:57:46for me to see which table to belong to
26:57:49which business objects. And now we have
26:57:51the final table over here and only one
26:57:53table about the sales and orders. In the
26:57:55arb we don't have any informations about
26:57:57that. So this one going to be easy.
26:57:59Let's call it sales. And let's move it
26:58:03over here. And as well maybe change the
26:58:05color of that to for example this color
26:58:07over here. Now this step is very
26:58:09important by building any data model in
26:58:11the gold layer. It gives you a big
26:58:13picture about the things that you are
26:58:14going to module. So now the next step is
26:58:16that we're going to go and build those
26:58:18objects step by step. So let's start
26:58:19with the first objects with our
26:58:21customers. So here we have three tables
26:58:23and we're going to start with the CRM.
26:58:25So let's start with this table over
26:58:26here. All right. So with that we know
26:58:28what are our business objects and this
26:58:30task is done and now in the next step
26:58:32we're going to go back to scale and
26:58:34start doing data integrations and
26:58:36building completely new data model. So
26:58:39let's go and do
26:58:43that. Now let's have a quick look to the
26:58:45good layer specifications. So this is
26:58:47the final stage. We're going to provide
26:58:49data to be consumed by reporting and
26:58:51analytics. And this time we will not be
26:58:53building tables. We will be using views.
26:58:56So that means we will not be having like
26:58:58stored procedure or any load process to
26:59:00the code layer. All what we are doing is
26:59:02only data transformation and the focus
26:59:04of the data transformation going to be
26:59:06data integration, aggregation, business
26:59:08logic and so on. And this time we're
26:59:10going to introduce a new data model. We
26:59:12will be doing star schema. So those are
26:59:15the specifications for the gold layer
26:59:16and this is our scope. So this time we
26:59:19make sure that we are selecting data
26:59:21from the silver layer not from the
26:59:23bronze because the bronze has bad data
26:59:25quality and the silver is everything is
26:59:27prepared and cleaned up. In order to
26:59:29build the good layer going to be
26:59:30targeting the server layer. So let's
26:59:33start with select star from and we're
26:59:36going to go to the silver CRM customer
26:59:39info. So let's go and hit execute. And
26:59:41now we're going to go and select the
26:59:42columns that we need to be presented in
26:59:45the go layer. So let's start selecting
26:59:47the columns that we want. So we have the
26:59:49ID, the key, the first
26:59:55name. I will not go and get the metadata
26:59:58information. This only belongs to the
27:00:00silver. Perfect. The next step is that
27:00:01I'm going to go and give this table an
27:00:03alias. So let's go and call it CI. And
27:00:06I'm going to make sure that we are
27:00:07selecting from this alias because later
27:00:10we're going to go and join this table
27:00:12with other tables. So something like
27:00:14this. So we're going to go with those
27:00:15columns. Now let's move to the second
27:00:17table. Let's go and get the birthday
27:00:19information. So now we're going to jump
27:00:20to the other system and we have to join
27:00:22the data by the CID together with the
27:00:25customer key. So now we have to go and
27:00:27join the data with another table. And
27:00:29here I try to avoid using the inner join
27:00:32because if the other table doesn't have
27:00:34all the information about the customers,
27:00:36I might lose customers. So always start
27:00:38with the master table and if you join it
27:00:41with any other table in order to get
27:00:43informations try always to avoid inner
27:00:45join because the other source might not
27:00:48have all the customers and if you do
27:00:49inner join you might lose customers. So
27:00:51I tend to start from the master table
27:00:53and then everything else is about the
27:00:56lift join. So I'm going to say lift join
27:00:58silver ERP customer a12. So let's give
27:01:01it the alias ca. And now we have to join
27:01:04the tables. So it's going to be by CE
27:01:06from the first table. It's going to be
27:01:08the customer key equal to CA and we have
27:01:12the CI ID. Now of course we're going to
27:01:14get matching data because we checked the
27:01:16server layer. But if we haven't prepared
27:01:18the data in the server layer, we have to
27:01:20do here preparation step in order to
27:01:22join the tables. But we don't have to do
27:01:23that because that was a pre-step in the
27:01:26server layer. So now you can see the
27:01:27systematic that we have in this bronze,
27:01:30silver, gold. So now after joining the
27:01:32tables we have to go and pick the
27:01:33information that we need from the second
27:01:35table which is the birth date. So B date
27:01:38dates and as well from this table there
27:01:40is another nice information it is the
27:01:43gender information. So that's all what
27:01:46we need from the second table. Let's go
27:01:48and check the third table. So the third
27:01:50table is about the location information
27:01:52the countries and as well we connect the
27:01:54tables by the CID with the key. So let's
27:01:57go and do that. We're going to say as
27:01:59well left join silver ERP location and
27:02:03I'm going to give it the name LA and
27:02:05then we have to join Y the keys the same
27:02:07thing it's going to be CI customer key
27:02:10equal to LA CI ID again we have prepared
27:02:14those ids and keys in the server layer
27:02:16so the join should be working now we
27:02:18have to go and pick the data from the
27:02:20second table so what do we have over
27:02:22here we have the ID the country and the
27:02:24metadata information so let's go and
27:02:26just get the country Perfect. So now
27:02:29with that we have joined all the three
27:02:31tables and we have picked all the
27:02:33columns that we want in this object. So
27:02:35again by looking over here we have
27:02:37joined this table with this one and this
27:02:39one. So with that we have collected all
27:02:41the customer informations that we have
27:02:43from the two source systems. Okay. So
27:02:46now let's go and query in order to make
27:02:47sure that we have everything correct and
27:02:49in order to understand that your joints
27:02:51are correct you have to keep your eye in
27:02:54those three columns. So if you are
27:02:55seeing that you are getting data that
27:02:58means you are doing the the joints
27:03:00correctly but if you are seeing a lot of
27:03:02nulls or no data at all that means your
27:03:05joints are incorrect but now it looks
27:03:07for me it is working and another check
27:03:10that I do is that if your first table
27:03:12has no duplicates what could happen is
27:03:14that after doing multiple joins you
27:03:16might now start getting duplicates
27:03:18because the relationship between those
27:03:20tables is not clear one to one you might
27:03:23get like one to many relationship ship
27:03:24or many to many relationships. So now
27:03:26the check that I usually do at this
27:03:28stage is that I have to make sure that I
27:03:30don't have duplicates from their
27:03:32results. So we don't have like multiple
27:03:34rows for the same customer. So in order
27:03:36to do that, we go and do a quick group
27:03:39by. So we're going to group by the data
27:03:41by the customer ID and then we do the
27:03:44count from this subquery. So this is the
27:03:48whole subquery and then after that we're
27:03:51going to go and say group by the
27:03:54customer ID and then we say having
27:03:58count higher than one. So this query
27:04:02actually try to find out whether we have
27:04:04any duplicates in the primary key. So
27:04:07let's go and execute it. We don't have
27:04:09any duplicates and that means after
27:04:11joining all those tables with the
27:04:13customer info those tables didn't cause
27:04:16any issues and didn't duplicate my data.
27:04:18So this is very important check to make
27:04:21sure that you are in the right way. All
27:04:23right. So that means everything is fine
27:04:25about the duplicates. We don't have to
27:04:27worry about it. Now we have here an
27:04:29integration issue. So let's go and
27:04:30execute it again. And now if you look to
27:04:32the data we have two sources for the
27:04:34gender informations. one comes from the
27:04:36CRM and another one come from the ERP.
27:04:39So now the question is what we're going
27:04:40to do with this? Well, we have to do
27:04:42data integration. So let me show you how
27:04:44I do it. First I go and have a new query
27:04:47and then I'm going to go and remove all
27:04:49other stuff and I'm going to leave only
27:04:51those two informations and use it
27:04:53distinct just to focus on the
27:04:56integration and let's go and execute it
27:04:58and maybe as well to do an order by. So
27:05:00let's do one and two. Let's go and
27:05:02execute it again. So now here we have
27:05:04all the scenarios and we can see
27:05:06sometimes there is a matching. So from
27:05:08the first table we have female and the
27:05:10other table we have as well female but
27:05:12sometimes we have an issue like those
27:05:13two tables are giving different
27:05:15informations and the same thing over
27:05:17here. So this is as well an issue
27:05:18different informations. Another scenario
27:05:20where we have a data from the first
27:05:22table like here we have the female but
27:05:24in the other table we have not
27:05:26available. Well this is not a problem.
27:05:28So we can get it from the first table
27:05:30but we have as well the exact opposite
27:05:32scenario where from the first table the
27:05:34data is not available but it is
27:05:36available from the second table. And now
27:05:38here you might wonder why I'm getting a
27:05:40null over here. We did handle all the
27:05:42missing data in the silver layer and we
27:05:44replace everything with not available.
27:05:46So why we are still in getting a null?
27:05:48This null doesn't come directly from the
27:05:50tables. It just come because of joining
27:05:53tables. So that means there are
27:05:55customers in the CRM table that is not
27:05:58available in the ARB table and if there
27:06:01is like no match what going to happen we
27:06:03will get a null from SQL. So this null
27:06:06means there was no match and that's why
27:06:08we are getting this null. It is not
27:06:10coming from the content of the tables
27:06:12and this is of course an issue. But now
27:06:14the big issue what can happen for those
27:06:16two scenarios here we have the data but
27:06:19they are different. And here again we
27:06:20have to ask the experts about it. What
27:06:23is the master here? Is it the CRM system
27:06:26or the ARP? And let's say from their
27:06:28answer going to say the master data for
27:06:30the customer information is the CRM. So
27:06:33that means the CRM informations are more
27:06:36accurate than the ERP information and
27:06:39this is only about the customers of
27:06:40course. So for this scenario where we
27:06:42have female and male then the correct
27:06:44information is the female from the first
27:06:46source system. The same goes over here
27:06:48and here we have like male and female
27:06:50then the correct one is the male because
27:06:53this source system is the master. Okay.
27:06:55So now let's go and build this business
27:06:57rule. We're going to start as usual with
27:06:59the case win. So the first very
27:07:01important rule is if we have a data in
27:07:03the gender information from the CRM
27:07:06system from the master then go and use
27:07:08it. So we're going to go and check the
27:07:10gender information from the CRM table.
27:07:12So customer gender is not equal to not
27:07:15available. So that means we have a value
27:07:18male or female. Let me just have here a
27:07:20comma like this. Then what's going to
27:07:22happen? Go and use it. So we're going to
27:07:24use the value from the master. CRM is
27:07:28the master for gender info. Now
27:07:32otherwise that means it is not available
27:07:35from the CRM table. Then go and use and
27:07:39grab the information from the second
27:07:40table. So we're going to say CA gender.
27:07:44But now we have to be careful with this
27:07:46null over here. We have to convert it to
27:07:47not available as well. So we're going to
27:07:49use the
27:07:50kis. So if this is a null then go and
27:07:54use the not available like this. So
27:07:57that's it. Let's have an end. And let me
27:07:59just push this over here. So let's go
27:08:01and call it new gen for now. Let's go
27:08:04and execute it and let's go and check
27:08:06the different scenarios. All those
27:08:08values over here we have data from the
27:08:11CRM system and this is as well
27:08:13represented in the new column. But now
27:08:15for the second part we don't have data
27:08:18from the first system. So we are trying
27:08:19to get it from the second system. So for
27:08:22the first one is not available and then
27:08:24we try to get it from the second source
27:08:26system. So now we are activating the
27:08:27else. Well it is null and with that the
27:08:30kalis is activated and we are replacing
27:08:32the null with not available. For the
27:08:35second scenario as well, the first
27:08:36search system don't have the gender
27:08:39information. That's why we are grabbing
27:08:40it from the second. So with that we have
27:08:43a female. And then the third one the
27:08:45same thing we don't have information but
27:08:46we get it from the second source system.
27:08:48We have the male and the last one it is
27:08:50not available in both source systems.
27:08:53That's why we are getting not available.
27:08:55So with that as you can see we have a
27:08:56perfect new column where we are
27:08:58integrating two different source system
27:09:00in one. And this is exactly what we call
27:09:03data integration. This piece of
27:09:05information, it is way better than the
27:09:07source CRM and as well the source ARP.
27:09:11It is more rich and has more
27:09:13information. And this is exactly why we
27:09:15try to get data from different source
27:09:17system in order to get rich information
27:09:19in the data warehouse. So with that we
27:09:21have a nice logic and as you can see
27:09:23it's way easier to separate it in
27:09:25separate query in order first to build
27:09:27the logic and then take it to the
27:09:29original query. So what I'm going to do,
27:09:31I'm just going to go and copy everything
27:09:32from here and go back to our query. I'm
27:09:35going to go and delete those
27:09:36informations the gender and I will put
27:09:39our new logic over here. So a comma and
27:09:42let's go and execute. So with that we
27:09:44have our new nice column. Now with that
27:09:46we have very nice objects. We don't have
27:09:48duplicates and we have integrated data
27:09:50together. So we took three tables and we
27:09:52put it in one object. Now the next step
27:09:54is that we're going to go and give nice
27:09:57friendly names. The rule in the gold
27:09:59layer that to use friendly names and not
27:10:01to follow the names that we get from the
27:10:03source system and we have to make sure
27:10:05that we are following the rules by the
27:10:07naming conventions. So we are following
27:10:09the snake case. So let's go and do it
27:10:11step by step. For the first one let's go
27:10:13and call it the customer ID. And then
27:10:15the next one I will get rid of using
27:10:18keys and so on. I'm going to go and call
27:10:19it customer number because those are
27:10:22customer numbers. Then for the next one,
27:10:25we're going to call it first name
27:10:28without using any prefixes. And the next
27:10:31one last name and we have here marital
27:10:35status. So I will be using the exact
27:10:38name but without the prefix. And here we
27:10:41just going to call it gender. And this
27:10:42one we're going to call it career date.
27:10:46And this one birth date. And the last
27:10:49one going to be the country. So let's go
27:10:52and execute it. Now as you can see the
27:10:54names are really friendly. So we have
27:10:56customer ID, customer numbers, first
27:10:58name, last name, material status,
27:11:00gender. So as you can see the names are
27:11:02really nice and really easy to
27:11:04understand. Now the next step I'm going
27:11:06to think about the order of those
27:11:07columns. So the first two it makes sense
27:11:09to have it together. The first name,
27:11:11last name, then I think the country is
27:11:13very important information. So I'm going
27:11:14to go and get it from here and put it
27:11:16exactly after the last name is just
27:11:19nicer. So let's go and execute it again.
27:11:21So the first name, last name, country.
27:11:23It's always nice to group up relevant
27:11:25columns together, right? So we have here
27:11:27the status of the gender and so on. And
27:11:29then we have the career date and the
27:11:30birth date. I think I'm going to go and
27:11:32switch the birth date with the career
27:11:34date. It's more important than the
27:11:36career dates like this. And here not
27:11:38forget the comma. So execute again. So
27:11:41it looks wonderful. Now comes a very
27:11:43important decision about these objects.
27:11:45Is it a fact table or a dimension? Well,
27:11:48as we learned, dimensions hold
27:11:50descriptive informations about an
27:11:52object. And as you can see, we have here
27:11:54a descriptions about the customers. So
27:11:56all those columns are describing the
27:11:59customer information. And we don't have
27:12:00here like transactions and events. And
27:12:03we don't have like measures and so on.
27:12:04So we cannot say this object is a fact.
27:12:07It is clearly a dimension. So that's why
27:12:10we're going to go and call this object
27:12:12the dimension customer. Now there is one
27:12:14thing that if you are creating a new
27:12:16dimension you need always a primary key
27:12:18for the dimension. Of course we can go
27:12:20over here and depend on the primary key
27:12:22that we get from the source system but
27:12:24sometimes you can have like dimensions
27:12:26where you don't have like a primary key
27:12:28that you can count on. So what we have
27:12:30to do is to go and generate a new
27:12:33primary key in the data warehouse. And
27:12:35those primary keys we call it surrogate
27:12:37keys. Srogate keys are system generated
27:12:40unique identifier that is assigned to
27:12:42each records to make the record unique.
27:12:45It is not a business key. It has no
27:12:47meaning and no one in the business knows
27:12:50about it. We only use it in order to
27:12:52connect our data model. And in this way
27:12:54we have more control on how to connect
27:12:56our data model and we don't have to
27:12:58depend always on the source system. And
27:13:01there are different ways on how to
27:13:02generate surrogate keys like defining it
27:13:05in the DDL or maybe using the window
27:13:07function row number in this data
27:13:09warehouse. I'm going to go with a simple
27:13:11solution where we're going to go and use
27:13:12the window function. So now in order to
27:13:15generate a surrogate key for this
27:13:17dimension what we're going to do it is
27:13:19very simple. So we're going to say row
27:13:21number over and here we have to order by
27:13:25something. You can order by the create
27:13:27date or the customer ID or the customer
27:13:30number. whatever you want but in this
27:13:32example I'm going to go and order by the
27:13:34customer ID. So we have to follow the
27:13:37naming convention that all surrogate
27:13:39keys with a key at the end as a suffix.
27:13:41So now let's go and query those
27:13:43informations. And as you can see at the
27:13:45start we have a customer key and this is
27:13:47a sequence. We don't have here of course
27:13:49any duplicates. And now this target key
27:13:52is generated in the data warehouse and
27:13:54we're going to use this key in order to
27:13:56connect the data model. So now with that
27:13:58our query is ready and the last step is
27:14:01that we're going to go and create the
27:14:03object and as we decided all the objects
27:14:05in the gold layer going to be virtual
27:14:07one. So that means we're going to go and
27:14:09create a view. So we're going to say
27:14:11create view gold dot dim. So follow the
27:14:15naming convention stand for the
27:14:17dimension and we're going to have the
27:14:18customers and then after that we have
27:14:20ass. So with that everything is ready.
27:14:23Let's go and execute it. It was
27:14:25successful. Let's go to the views now
27:14:27and you can see our first objects. So we
27:14:30have the dimension customers in the gold
27:14:32layer. Now as you know me in the next
27:14:34step that we're going to go and check
27:14:36the quality of this new objects. So
27:14:38let's go and have a new query. So select
27:14:41star from our view temp customers. And
27:14:45now we have to make sure that everything
27:14:47in the right position like this. And now
27:14:49we can do different checks like the
27:14:51uniqueness and so on. But I'm worried
27:14:54about the gender information. So let's
27:14:56go and have a distinct of all values. So
27:14:59as you can see it is working perfectly.
27:15:00We have only female, male and not
27:15:03available. So that's it with that. We
27:15:05have our first new
27:15:10dimension. Okay friends. So now let's go
27:15:12and build the second object. We have the
27:15:15products. So as you can see product
27:15:17information is available in both source
27:15:19systems. As usual, we're going to start
27:15:21with the CRM informations and then we're
27:15:23going to go and join it with the other
27:15:24table in order to get the category
27:15:26informations. So those are the columns
27:15:28that we want from this table. Now we
27:15:30come here to a big decision about this
27:15:33objects. This object contains historical
27:15:35informations and as well the current
27:15:37informations. Now of course depend on
27:15:38the requirement whether you have to do
27:15:40analyszis on the historical
27:15:42informations. But if you don't have such
27:15:43a requirements we can go and stay with
27:15:46only the current informations of the
27:15:48products. So we don't have to include
27:15:49all the history in the objects and it is
27:15:51anyway as we learned from the model over
27:15:53here we are not using the primary key we
27:15:56are using the product key. So now what
27:15:58we have to do is to filter out the
27:16:00historical data and to stay only with
27:16:02the current data. So we're going to have
27:16:04here a wear condition. And now in order
27:16:06to select the current data what we're
27:16:08going to do we're going to go and target
27:16:10the end dates. If the end date is null
27:16:12that means it is a current data. Let's
27:16:14take this example over here. So you can
27:16:16see here we have three records for the
27:16:18same product key and for the first two
27:16:21records we have here an information in
27:16:23the end dates because it is historical
27:16:25informations but the last record over
27:16:28here we have it as a null and that's
27:16:29because this is the current information
27:16:32it is open and it's not closed yet. So
27:16:34in order to select only the current
27:16:36informations it is very simple we can
27:16:38say brd in dates is null. So if you go
27:16:42now and execute it, you will get only
27:16:44the current products. You will not have
27:16:46any history. And of course we can go and
27:16:48add comment to it. Filter out all
27:16:51historical data. And this means of
27:16:54course we don't need the end date in our
27:16:56selection of course because it is always
27:16:58a null. So with that we have only the
27:17:01current data. Now the next step is that
27:17:03we have to go and join it with the
27:17:05product categories from the ERP. And
27:17:07we're going to use here the ID. So as
27:17:10usual the master information is the CRM
27:17:13and everything else going to be
27:17:14secondary. That's why I use the lift
27:17:17join just to make sure I'm not losing
27:17:19I'm not filtering any data because if
27:17:21there is no match then we lose data. So
27:17:24lift join silver ERP and the category.
27:17:27So let's call it PC. And now what we're
27:17:30going to do we're going to go and join
27:17:31it using the key. So en from the CRM we
27:17:35have the category ID equal to PC ID. And
27:17:39now we have to go and pick columns from
27:17:40the second table. So it's going to be
27:17:43the PC. We have the category very
27:17:45important PC. We have the
27:17:48subcategory and we can go and get the
27:17:50maintenance. So something like this.
27:17:53Let's go and query. And with that we
27:17:55have all those columns comes from the
27:17:58first table and those three comes from
27:18:00the second. So with that we have
27:18:01collected all the product informations
27:18:04from the two source systems. Now the
27:18:06next step is we have to go and check the
27:18:08quality of these results. And of course
27:18:10what is very important is to check the
27:18:13uniqueness. So what we're going to do
27:18:14we're going to go and have the following
27:18:16query. I want to make sure that the
27:18:19product key is
27:18:21unique because we're going to use it
27:18:23later in order to join the table with
27:18:25the sales. So
27:18:27from and then we have to have group by
27:18:30product key and we're going to say
27:18:32having
27:18:33counts higher than one. So let's go and
27:18:37check. Perfect. We don't have any
27:18:38duplicates. The second table didn't
27:18:41cause any duplicates for our join. And
27:18:43as well this means we don't have
27:18:45historical data and each product is only
27:18:47one records and we don't have any
27:18:49duplicates. So I'm really happy about
27:18:52that. So let's go and query again. Now,
27:18:54of course, the next step, do we have
27:18:56anything to integrate together? Do we
27:18:58have the same information twice? Well,
27:19:00we don't have that. The next step is
27:19:02that we're going to go and group up the
27:19:04relevant informations together. So, I'm
27:19:06going to say the product ID, then the
27:19:09product key, and the product name are
27:19:11together. So, all those three
27:19:13informations are together. And after
27:19:15that, we can put all the category
27:19:17informations together. So, we're going
27:19:18to have the category ID, the category
27:19:21itself, the subcategory. Let me just
27:19:23query and see the results. So we have
27:19:25the product ID key name and then we have
27:19:28the category ID name and the subcategory
27:19:31and then maybe as well to put the
27:19:33maintenance after the subcategory like
27:19:36this and I think the product cost and
27:19:38the line can start could stay at the
27:19:40end. So let me just check. So those
27:19:42three four informations about the
27:19:44category and then we have the cost line
27:19:46and the start date. I'm really happy
27:19:48with that. The next step we're going to
27:19:49go and give nice names, friendly names
27:19:52for those columns. So let's start with
27:19:54the first one. This is the product ID.
27:19:57The next one going to be the product
27:19:59number. We need the key for the
27:20:01surrogate key later. And then we have
27:20:04the product name. And after that we have
27:20:07the category ID and the category. And
27:20:11this is the subcategory. And then the
27:20:14next one going to stay as it is. I don't
27:20:16have to rename it. The next one going to
27:20:18be the cost and the product line and the
27:20:22last one going to be the start stage. So
27:20:25let's go and execute it. Now we can see
27:20:27very nicely in the output all those
27:20:29friendly names for the columns and it
27:20:32looks way nicer than before. I don't
27:20:34have even to describe those informations
27:20:36the name describe it. So perfect. Now
27:20:38the next big decision is what do we have
27:20:40here? Do we have a fact or dimension?
27:20:42What do you think? Well, as you can see
27:20:44here again, we have a lot of
27:20:46descriptions about the products. So all
27:20:48those informations are describing the
27:20:51business object products. We don't have
27:20:53like here transactions, events, a lot of
27:20:56different keys and ids. So we don't have
27:20:58really here facts. We have a dimension.
27:21:01Each row is exactly describing one
27:21:03object, describing one product. That's
27:21:06why this is a dimension. Okay. So now
27:21:08since this is a dimension, we have to go
27:21:10and create a primary key for it. Well,
27:21:12actually the surrogate key and as we
27:21:14have done it for the customers, we're
27:21:16going to go and use the window function
27:21:18row number in order to generate it over
27:21:21and then we have to sort the data. I
27:21:22will go with the start date. So let's go
27:21:25with the start dates and as well the
27:21:28product key and we're going to give it a
27:21:31name products key like this. So let's go
27:21:35and execute it. With that, we have now
27:21:37generated a primary key for each product
27:21:40and we're going to be using it in order
27:21:42to connect our data model. All right.
27:21:44Now, the next step with that, we're
27:21:45going to go and build the view. So,
27:21:47we're going to say create view. We're
27:21:50going to say gold and dimension products
27:21:53and then us. So, let's go and create our
27:21:56object. And now, if you go and refresh
27:21:58the views, you will see our second
27:22:00object, the second dimension. So, we
27:22:02have here in the gold layer the
27:22:04dimension products. And as usual, we're
27:22:06going to go and have a look to this view
27:22:09just to make sure that everything is
27:22:10fine. So dem products. So let's execute
27:22:14it. And by looking to the data
27:22:16everything looks nice. So with that we
27:22:18have now two
27:22:23dimensions. All right friends. So with
27:22:25that we have covered a lot of stuff. So
27:22:27we have covered the customers and the
27:22:29products and we are left with only one
27:22:31table where we have the transactions the
27:22:33sales and for the sales information we
27:22:36have only data from the CRM. We don't
27:22:37have anything from the ERP. So let's go
27:22:39and build it. Okay. So now I have all
27:22:41those informations and now of course we
27:22:43have only one table. We don't have to do
27:22:45any integrations and so on. And now we
27:22:47have to answer the big question. Do we
27:22:48have here a dimension or a fact? Well by
27:22:51looking to those details we can see
27:22:53transactions. We can see events. We have
27:22:55a lot of dates, informations. We have as
27:22:58well a lot of measures and metrics and
27:23:00as well we have a lot of ids. So it is
27:23:03connecting multiple dimensions. And this
27:23:05is exactly a perfect setup for effect.
27:23:08So we're going to go and use those
27:23:09informations as a facts. And of course
27:23:11as we learned a fact is connecting
27:23:13multiple dimensions. We have to present
27:23:16in this fact the surrogate keys that
27:23:18comes from the dimensions. So those two
27:23:20informations the product key and the
27:23:23customer ID those informations comes
27:23:25from the source system and as we learned
27:23:27we want to connect our data model using
27:23:29the surrogate keys. So what we're going
27:23:31to do we're going to replace those two
27:23:32informations with the surrogate keys
27:23:34that we have generated and in order to
27:23:36do that we have to go and join now the
27:23:39two dimensions in order to get the
27:23:42surrogate key and we call this process
27:23:44of course data lookup. So we are joining
27:23:46the tables in order only to get one
27:23:48information. So let's go and do that. We
27:23:50will go with a lift join of course not
27:23:52to lose any transaction. So first we're
27:23:55going to go and join it with the product
27:23:56key. Now of course in the silver layer
27:23:58we don't have any surrogate keys. We
27:24:01have it in the gold layer. So that means
27:24:03for the fact table we're going to be
27:24:05joining the silver layer together with
27:24:07the gold layer. So, gold dots and then
27:24:09the dimension products and I'm going to
27:24:12just call it PR. And we're going to join
27:24:15the SD using the product key together
27:24:18with the product
27:24:20number from the dimension. And now the
27:24:23only information that we need from the
27:24:25dimension is the key, the surrogate key.
27:24:28So, we're going to go over here and say
27:24:30product key. And what I'm going to do,
27:24:32I'm going to go and remove this
27:24:34information from here because we don't
27:24:36need it. We don't need the original
27:24:37product key from the source system. We
27:24:39need the surrogate key that we have
27:24:41generated in our own in this data
27:24:42warehouse. So the same thing going to
27:24:44happen as well for the customer. So gold
27:24:48dimension customer again we are doing
27:24:51here a lookup in order to get the
27:24:53information on SD. So we are joining
27:24:56using this ID over here equal to the
27:25:01customer ID because this is a customer
27:25:03ID. And what we're going to do the same
27:25:06thing we need the surrogate key the
27:25:09customer key and we're going to delete
27:25:11the ID because we don't need it. Now we
27:25:12have the surrogate key. So now let's go
27:25:14and execute it. And now with that we
27:25:17have in our fact table the two keys from
27:25:19the dimensions. And now this can help us
27:25:21to connect the data model to connect the
27:25:24facts with the dimensions. So this is
27:25:26very necessary step building the fact
27:25:28table. You have to put the surrogate
27:25:30keys from the dimensions in the facts.
27:25:32So that was actually the hardest part
27:25:34building the facts. Now the next step
27:25:36all what you have to do is to go and
27:25:37give friendly names. So we're going to
27:25:39go over here and say order number. Then
27:25:42the surrogate keys are already friendly.
27:25:44So we're going to go over here and say
27:25:46this is the order date. And the next one
27:25:49going to be shipping date. And then the
27:25:53next one due age and the sales going to
27:25:56be I'm going to say sales
27:25:59amount the
27:26:01quantity and the final one is the price.
27:26:05So now let's go and execute it and look
27:26:07to the results. So now as you can see
27:26:09the columns looks very friendly and now
27:26:11about the order of the columns we use
27:26:13the following schema. So first in the
27:26:15fact table we have all the surrogate
27:26:17keys from the dimensions. Then second we
27:26:19have all the dates and at the end you
27:26:22group up all the measures and the
27:26:24metrics at the end of the fact. So
27:26:26that's it for the query for the facts.
27:26:28Now we can go and build it. So we're
27:26:30going to say
27:26:31create view gold in the gold layer and
27:26:35this time we're going to use the fact
27:26:37underscore and we're going to go and
27:26:38call it sales and then don't forget
27:26:40about the ass. So that's it. Let's go
27:26:42and create it. Perfect. Now we can see
27:26:45the fact. So with that we have three
27:26:47objects in the go there. We have two
27:26:49dimensions and one facts. And now of
27:26:51course the next step with that we're
27:26:52going to go and check the quality of the
27:26:55view. So let's have a simple
27:26:58select fact sales. So let's execute it.
27:27:02Now by checking the result you can see
27:27:03it is exactly like the result from the
27:27:06query and everything looks nice. Okay.
27:27:08So now one more trick that I usually do
27:27:10after building effect is try to connect
27:27:13the whole data model in order to find
27:27:14any issues. So let's go and do that. We
27:27:16will do just simple lift join with the
27:27:19dimensions. So gold dimension customers
27:27:23see and we will use the
27:27:28keys and then we're going to say where
27:27:31customer key is null. So there is no
27:27:33matching. So let's go and execute it.
27:27:36And with that as you can see in the
27:27:37results we are not getting anything that
27:27:39means everything is matching perfectly
27:27:42and we can do as well the same thing
27:27:44with the products. So left join called
27:27:48then products p on product key and then
27:27:53we connect it with the fact product key
27:27:56and then we going go and check the
27:27:58product key from the dimension like
27:28:00this. So we are checking whether we can
27:28:02connect the fact together with the
27:28:04dimension products. Let's go and check
27:28:06and as you can see as well we are not
27:28:08getting anything and this is all right.
27:28:09So with that we have now SQL codes that
27:28:12is tested and as well creating the gold
27:28:15layer. Now in the next step as you know
27:28:17in our requirements we have to make
27:28:19clear documentations for the end users
27:28:21in order to use our data model. So let's
27:28:23go and draw a data model of the star
27:28:28schema. So let's go and draw our data
27:28:31model. Let's go and search for a table.
27:28:34And now what I'm going to do, I'm going
27:28:35to go and take this one where I can say
27:28:37what is the primary key and what is the
27:28:40foreign key. And I'm going to go and
27:28:41change a little bit the design. So it's
27:28:43going to be rounded. And let's say I'm
27:28:45going to go and change to this color.
27:28:46And maybe go to the size, make it 16.
27:28:49And then I'm going to go and select all
27:28:51the columns and make it as well 16 just
27:28:54to increase the size. And then go to our
27:28:57range and we can go and increase it 39.
27:29:00So now let's go and zoom in a little bit
27:29:02for the first table. Let's go and call
27:29:04it gold dimension customers and make it
27:29:08a little bit bigger like this. And now
27:29:10we're going to go and define here the
27:29:11primary key. It is the customer key. And
27:29:14what else we're going to do? We're going
27:29:15to go and list all the columns in the
27:29:16dimension. It is a little bit annoying
27:29:18but the result is going to be awesome.
27:29:19So what do we have? The customer ID. We
27:29:21have the customer number and then we
27:29:25have the first name. Now in case you
27:29:28want a new rows so you can hold control
27:29:31and enter and you can go and add the
27:29:33other columns. So now pause the video
27:29:35and then go and create the two
27:29:36dimensions the customers and the
27:29:38products and add all the columns that
27:29:40you have built in the
27:29:43[Music]
27:29:45view. Welcome back. So now I have those
27:29:48two dimensions. The third one going to
27:29:50be the fact table. Now for the fact
27:29:52table I'm going to go with different
27:29:54color. for example, the blue and I'm
27:29:56going to go and put it in the middle.
27:29:58Something like this. So, we're going to
27:29:59say gold fact sales and here for that we
27:30:04don't have primary key. So, we're going
27:30:05to go and delete it. And I have to go
27:30:07and add all the columns of the facts.
27:30:09So, order number, products key, customer
27:30:13key. Okay. All right. Perfect. Now, what
27:30:15we can do, we can go and add the foreign
27:30:17key information. So, the product key is
27:30:19a foreign key for the products. So,
27:30:21we're going to say FK1. And the customer
27:30:23key going to be the foreign key for the
27:30:25customers. So FK2 and of course you can
27:30:27go and increase the spacing for that.
27:30:30Okay. So now after we have the tables
27:30:31the next step in data modeling is to go
27:30:34and describe the relationship between
27:30:35these tables. This is of course very
27:30:37important for reporting and analytics in
27:30:39order to understand how I'm going to go
27:30:41and use the data model. And we have
27:30:43different types of relationships. We
27:30:44have one to one, one to many. And in
27:30:46star schema data model the relationship
27:30:48between the dimension and the fact is
27:30:51one to many. And that's because in the
27:30:52table customers we have for a specific
27:30:55customer only one record describing the
27:30:57customer but in the fact table the
27:30:59customer might exist in multiple records
27:31:01and that's because customers can order
27:31:04multiple times. So that's why in fact it
27:31:06is many and in the dimension side it is
27:31:09one. Now in order to see all those
27:31:10relationships we're going to go to the
27:31:12menu to the left side and as you can see
27:31:13we have here entity relations and now we
27:31:16have different types of arrows. So for
27:31:18example we have zero to many, one to
27:31:20many, one to one and many different
27:31:22types of relations. So now which one we
27:31:24going to take? We're going to go and
27:31:26pick this one. So it says one mandatory.
27:31:28So that means the customer must exist in
27:31:31the dimension table. Too many but it is
27:31:33optional. So here we have three
27:31:34scenarios. The customer didn't order
27:31:36anything or the customer did order only
27:31:38once or the customer did order many
27:31:41things. So that's why in the fact table
27:31:43it is optional. So we're going to take
27:31:45this one and place it over here. So
27:31:47we're going to go and connect this part
27:31:50to the customer dimension and the many
27:31:52parts to the facts. Well actually we
27:31:55have to do it on the customers. So with
27:31:57that we are describing the relationship
27:31:59between the dimensions and fact with one
27:32:01to many. One is mandatory for the
27:32:03customer dimension and many is optional
27:32:06to the facts. So we have the same story
27:32:07as well for the products. So the many
27:32:10part to the facts and the one goes to
27:32:14the products. So it's going to look like
27:32:15this. Each time you are connecting new
27:32:17dimension to the fact table, it is
27:32:19usually one to many relationship. So you
27:32:22can go and add anything you want to this
27:32:24model like for example a text like
27:32:26explaining something. For example, if
27:32:27you have some complicated calculations
27:32:29and so on, you can go and write this
27:32:31information over here. So for example,
27:32:33we can say over here sales calculation,
27:32:36we can make it a little bit smaller. So
27:32:38let's go with 18. So we can go and write
27:32:41here the formula for that. So sales
27:32:43equal quantity multiplied with the price
27:32:47and make this little bit bigger. So it
27:32:50is really nice info that we can add it
27:32:53to the data model and even we can go and
27:32:55link it to the column. So we can go and
27:32:57take this arrow for example put it like
27:32:59this and link it to the column and with
27:33:02that you have as well nice explanation
27:33:04about the business rule or the
27:33:05calculation. So you can go and add any
27:33:07descriptions that you want to the data
27:33:09model. Just to make it clear for anyone
27:33:11that is using your data model. So with
27:33:13that you don't have only like three
27:33:15tables in the database. You have as well
27:33:17like some kind of documentations and
27:33:19explanation. In one click we can see how
27:33:21the data model is built and how you can
27:33:23connect the tables together. It is
27:33:26amazing really for all users of your
27:33:27data model. All right. So now with that
27:33:29we have really nice data model. And now
27:33:32in the next step we're going to go and
27:33:33create quickly a data catalog.
27:33:39All right, great. So with that we have a
27:33:41data model and we can say we have
27:33:42something called a data products and we
27:33:44will be sharing this data product with
27:33:47different types of users and there is
27:33:48something that every data products
27:33:51absolutely needs and that is the data
27:33:53catalog. It is a document that can
27:33:55describe everything about your data
27:33:57model. columns, the tables, maybe the
27:34:00relationship between the tables as well.
27:34:02And with that, you make your data
27:34:03product clear for everyone. And it's
27:34:05going to be for them way easier to
27:34:06derive more insights and reports from
27:34:09your data product. And what is the most
27:34:11important one? It is time-saving because
27:34:14if you don't do that, what's going to
27:34:15happen? Each consumer, each user of your
27:34:17data product will keep asking you the
27:34:19same questions about what do you mean
27:34:21with this column? What is this table?
27:34:23How to connect the table A with the
27:34:24table P? and you will keep repeating
27:34:26yourself and explaining stuff. So
27:34:28instead of that you prepare a data
27:34:30catalog, a data model and you deliver
27:34:32everything together to the users and
27:34:34with that you are saving a lot of time
27:34:36and stress. I know it is annoying to
27:34:38create a data catalog but it is
27:34:40investments and best practices. So now
27:34:42let's go and create one. Okay. So now in
27:34:44order to do that I have created a new
27:34:45file called data catalog in the folder
27:34:48documents. And here what we're going to
27:34:49do is very straightforward. We're going
27:34:50to make a section for each table in the
27:34:52code layer. So for example we have here
27:34:54the table dimension customers. What you
27:34:56have to do first is to describe this
27:34:58table. So we are saying it stores
27:35:00details about the customers with the
27:35:02demographics and geographics data. So
27:35:04you give a short description for the
27:35:06table and then after that you're going
27:35:07to go and list all your columns inside
27:35:09this table and maybe as well the data
27:35:11type. But what is way important is the
27:35:13description for each column. So you give
27:35:15a very short description like for
27:35:17example here the gender of the customer.
27:35:19And now one of the best practices of
27:35:21describing a column is to give examples
27:35:23because you can understand quickly the
27:35:25purpose of the columns by just seeing an
27:35:27example. Right? So here we are saying we
27:35:29can find inside the male, female and not
27:35:32available. So with that the consumer of
27:35:33your table can immediately understand uh
27:35:35it will not be an M or an F. It's going
27:35:38to be a full friendly value without
27:35:40having them to go and query the content
27:35:41of the table. They can understand
27:35:43quickly the purpose of that column. So
27:35:45with that we have a full description for
27:35:46all the columns of our dimension. The
27:35:49same thing we're going to do for the
27:35:50products. So again, a description for
27:35:52the table and as well a description for
27:35:54each column and the same thing for the
27:35:56facts. So that's it. With that you have
27:35:58like a data catalog for your data
27:36:00products at the code layer. And with
27:36:02that the business user or the data
27:36:04analyst have better and clear
27:36:05understanding of the content of your
27:36:07code layer. All right my friends. So
27:36:09that's all for the data catalog. In the
27:36:11next step we're going to go back to DO
27:36:13where we're going to finalize the data
27:36:15flow diagram. So let's go.
27:36:21Okay. So now we're going to go and
27:36:22extend our data flow diagram, but this
27:36:25time for the gold layer. So now let's go
27:36:27and copy the whole thing from the silver
27:36:29layer and put it over here side by side.
27:36:31And of course we're going to go and
27:36:33change the coloring to the gold. And now
27:36:35we're going to go and rename stuff. So
27:36:38this is the gold layer. But now of
27:36:40course we cannot leave those tables like
27:36:42this. We have completely new data model.
27:36:44So what do we have over here? We have
27:36:46the fact sales, we have dimension
27:36:50customers, and as well we have dimension
27:36:53products. So now what I'm going to do,
27:36:55I'm going to go and remove all those
27:36:56stuff. We have only three tables. And
27:36:58let's go and put those three tables
27:37:00somewhere here in the center. So now
27:37:02what you have to do is to go and start
27:37:03connecting those stuff. I'm going to go
27:37:05with this arrow over here, direct
27:37:07connection, and start connecting stuff.
27:37:10So the sales details goes to the fact
27:37:12table. Maybe put the fact table over
27:37:14here. And then we have the dimension
27:37:16customer. This comes from the CRM
27:37:19customer info. And we have two tables
27:37:22from the ERP. It comes from this table
27:37:24as well. And the location from the ERP.
27:37:27Now the same thing goes for the
27:37:30products. It comes from the product info
27:37:32and comes from the categories from the
27:37:35ERP. Now, as you can see here, we have
27:37:37cross arrows. So what you can do, we can
27:37:38go and select everything and we can say
27:37:40line jumps with a gap. And this makes it
27:37:43a little bit like better in the visual
27:37:45for the arrows. So now for example if
27:37:47someone asks you where the data come
27:37:49from for the dimension products you can
27:37:51open this diagram and tell them okay
27:37:53this comes from the server layer. We
27:37:55have like two tables. The product info
27:37:57from the CRM and as well the categories
27:38:00from the ERP and those several tables
27:38:02comes from the bronze layer and you can
27:38:04see the product info comes from the CRM
27:38:06and the category comes from the ERP. So
27:38:09it is very simple. We have just created
27:38:11a full data lineage for our data
27:38:12warehouse from the sources into the
27:38:15different layers in our data warehouse
27:38:16and data lineage is this really amazing
27:38:18documentation that can help not only
27:38:20your users but as well the developers.
27:38:22All right. So with that we have very
27:38:23nice data flow diagram and a data
27:38:25lineage. All right. So we have completed
27:38:27the data flow. It's really feel like
27:38:29progress like achievements as we are
27:38:31clicking through all those tasks. And
27:38:33now we come to the last task in building
27:38:35the data warehouse where we're going to
27:38:37go and commit our work in the get
27:38:42repo. Okay. So now let's put our scripts
27:38:44in the project. So we're going to go to
27:38:46the scripts over here. We have here
27:38:48bronze silver but we don't have a gold.
27:38:49So let's go and create a new file. We're
27:38:51going to have gold/ and then we're going
27:38:53to say ddl gold.sql. So now we're going
27:38:56to go and paste our views. So we have
27:38:59here our three views. And as usual at
27:39:01the start we can describe the purpose of
27:39:03the views. So we are saying create gold
27:39:05views. This script can go and create
27:39:07views for the code layer and the code
27:39:09layer represent the final dimension and
27:39:11fact tables. The star schema each view
27:39:13perform transformations and combination
27:39:15data from the server layer to produce
27:39:17business ready data sets and those views
27:39:19can be used for analytics and reporting.
27:39:21So that's it. Let's go and commit it.
27:39:24Okay. So with that as you can see we
27:39:26have the bronze the silver. So we have
27:39:28all our ETLs and scripts in the
27:39:31repository. And now as well for the code
27:39:33layer, we're going to go and add all
27:39:35those quality checks that we have used
27:39:37in order to validate the dimensions and
27:39:39facts. So we're going to go to the test
27:39:41over here and we're going to go and
27:39:42create a new file. It's going to be
27:39:44quality checks gold and the file type is
27:39:47SQL. So now let's go and paste our
27:39:50quality checks. So we have the check for
27:39:51the fact, the two dimensions and as well
27:39:54an explanation about the script. So we
27:39:56are validating the integrity and the
27:39:58accuracy of the go layer. And here we
27:40:00are checking the uniqueness of the
27:40:01surrogate keys and whether we are able
27:40:03to connect the data model. So let's put
27:40:05that as well in our git and commit the
27:40:08changes. And in case we come up with a
27:40:09new quality checks, we're going to go
27:40:11and add it to our script here. So those
27:40:13checks are really important if you are
27:40:15modifying the ATLs or you want to make
27:40:17sure that after each those script should
27:40:19run and so on. It is like a quality gate
27:40:22to make sure that everything is fine in
27:40:24the gold layer. Perfect. So now we have
27:40:26our code in our repository. Okay
27:40:28friends. So now what you have to do is
27:40:30to go and finalize the get repo. So for
27:40:32example all the documentations that we
27:40:34have created during the projects we can
27:40:37go and upload them in the docs. So for
27:40:38example you can see here the data
27:40:40architecture the data flow data
27:40:42integration data model and so on. So
27:40:44that each time you edit those pages you
27:40:46can commit your work and you have like a
27:40:48version of that. And another thing that
27:40:50you can do is that you go to the readme
27:40:53like for example over here I have added
27:40:54the project overview some important
27:40:56links and as well the data architecture
27:40:59and a little description of the
27:41:00architecture of course and of course
27:41:02don't forget to add few words about
27:41:04yourself and important profiles in the
27:41:06different social medias. All right my
27:41:08friends. So with that we have committed
27:41:09our work and as well closed the last
27:41:12epic building the god layer and with
27:41:14that we have completed all the phases of
27:41:17building a data warehouse. Everything is
27:41:19100% and this feels really nice. All
27:41:22right my friends. So with that we have
27:41:23covered the first type of SQL projects
27:41:26that data warehousing projects. This is
27:41:28usually a very complex project that you
27:41:30can get involved in a company and this
27:41:32is really amazing project if you are
27:41:33planning to be a data engineer. But of
27:41:35course, if you are a data analyst, you
27:41:37might end up as well building
27:41:38warehouses. So now we have everything
27:41:40prepared for the second type of projects
27:41:42in SQL. We will deep dive now into the
27:41:44exploratory data analyzers. So let's
27:41:50go. And now here we're going to cover
27:41:52the second type of projects where we're
27:41:54going to use our basic SQL skills in
27:41:56order to do something called data
27:41:58profiling where we're going to try to
27:41:59understand all the aspects of our data
27:42:02sets using simple aggregations like the
27:42:04sum, average, count and as well we will
27:42:06be using techniques like some
27:42:08[Music]
27:42:11queries. All right my friends. So the
27:42:13first step in any data project is that
27:42:15we need data sets. If you have done the
27:42:17previous project where we have built the
27:42:19SQL data warehouse, then you have
27:42:21everything the data and the database. So
27:42:23you don't have to worry about it. But if
27:42:25you skip that, which I don't recommend,
27:42:27I still have prepared for you the files
27:42:29and the database. So let's get the data
27:42:31and create our database. All right. So
27:42:32now if you go to the link in the
27:42:34description, we're going to go to the
27:42:35downloads. And of course, you can
27:42:37subscribe to my newsletter. And then
27:42:39here we have the SQL course materials.
27:42:41And here we have a link for data
27:42:42analytics projects. Let's go to the
27:42:44link. And now here you have some
27:42:45important links like downloading the
27:42:47server the management studio where we're
27:42:49going to write our SQLs and as well
27:42:51there is a link to the g repository and
27:42:53as well what is very important is to
27:42:55download all the project files. So click
27:42:57on that and download all the files. Now
27:42:59extract the file and put it somewhere
27:43:01safe at your PC and now inside it you
27:43:03can find all the scripts and the data
27:43:04sets. Now there is like three ways on
27:43:06how to create the database in SQL
27:43:08server. So the first one is by executing
27:43:11scripts. If you go to the scripts over
27:43:12here, the first one we have a file
27:43:14called init database. Just go inside it
27:43:17and copy the whole thing and then let's
27:43:19go to SQL server. Now make a new query
27:43:22and make sure you switch to the master
27:43:25database and then paste the whole code.
27:43:27So now what you are doing here is we are
27:43:29creating a new database. We are creating
27:43:31a schema and then three very important
27:43:34tables that we're going to use in our
27:43:36data analyzes. Now there is like only
27:43:38one thing that you have to change in
27:43:39this script and that is the path of the
27:43:42files. And once you have done that just
27:43:44go and execute the whole script. And now
27:43:47as you can see everything is done and
27:43:48there is like data inserted. Now if you
27:43:50go to the left side to the database and
27:43:52refresh you can find a new database
27:43:55called data warehouse analytics. And if
27:43:57you go inside the tables you will find
27:44:00our three tables customer products and
27:44:02sales. So this is one way on how to
27:44:04create the database. The second methods
27:44:06is to go to the databases over here.
27:44:08Right click on it and say new database.
27:44:11And for example, let's call it data
27:44:13warehouse analytics. I'm going to call
27:44:15it two because I have already one. And
27:44:17then click okay. And with that you have
27:44:19a new database. So what we're going to
27:44:21do now, we're going to right click on it
27:44:23and then go to tasks and then import
27:44:26flat file. And now what we're going to
27:44:27do, we're going to go and import the CSV
27:44:30files to our new database. So we can go
27:44:32next and then you have to go and locate
27:44:34your files. I have them somewhere over
27:44:37here. So data set CSV files and we have
27:44:40to focus on the gold tables. So I'm
27:44:43going to go and select this one and then
27:44:44next. Now I'm just getting an overview
27:44:46of my data. So next. Now just to make
27:44:49sure that you are not getting any error,
27:44:51I'm going to go and allow nulls and
27:44:53that's all. So next and finish. So
27:44:56perfect. The data has been inserted. Now
27:44:58let's go to our database tables. And as
27:45:01you can see, we have here our new table.
27:45:03So you have to go and repeat this three
27:45:05times in order to import the data. Well,
27:45:07you can use this method if the first
27:45:09method didn't work. But I really
27:45:10recommend you to use the script in order
27:45:12to create the database. The third way is
27:45:14to go and restore the database itself.
27:45:17Now how we're going to do it? We're
27:45:18going to go again to the data sets and
27:45:20as you can see we have here a database
27:45:22backup. So as you can see we have here a
27:45:24PAK file. So now what you have to do is
27:45:26to go and copy that and then we're going
27:45:28to go to the database location. So it
27:45:30really depend where you have installed
27:45:32the SQL server. So currently I have it
27:45:34here program files Microsoft SQL server
27:45:36and then the express MSSQL backup and
27:45:40you have to place the file over here. So
27:45:42I have it here data warehouse analytics
27:45:44backup. And now all what you have to do
27:45:45is to right click on the database and
27:45:47then say restore database and then we're
27:45:50going to go to the device three points
27:45:52and we're going to say add. And now you
27:45:54can see our database data warehouse
27:45:56analytics. Once we say okay and then
27:45:58okay and now since I have it already I
27:46:01will get an error but once I click okay
27:46:03the whole database can be restored
27:46:05without running any scripts. So those
27:46:07are the three ways on how to create the
27:46:10database of the projects and if you have
27:46:11built with me the data warehouse
27:46:13projects before you don't have to do it
27:46:14because we have built that together. So
27:46:17pause the video and get the data for the
27:46:23projects. All right my friends. So we're
27:46:25going to start with a secret, a little
27:46:27trick that I usually do by analyzing any
27:46:30data sets. So let's start with little
27:46:31coffee before we start. H this is really
27:46:35hot. Okay. So the secret says as I'm
27:46:37looking to any data sets in any
27:46:39projects, I see the data always divided
27:46:41between dimensions and measures.
27:46:45What truth? You take the blue pill, you
27:46:48take the red pill. All I'm offering is
27:46:51the truth. Nothing more.
27:46:54If you see your data like me as
27:46:55dimensions and measures, you can
27:46:58generate like endless amount of insights
27:47:01from any projects from any data sets and
27:47:03you will find me through the projects
27:47:05that I'm always speaking about measures
27:47:07and dimensions. So I'm going to show you
27:47:09how I usually do it. So now usually by
27:47:11looking to any data sets in any
27:47:12projects. So you have like multiple
27:47:14columns and rows here I see the data
27:47:16always splitted into two categories
27:47:18either a dimension or a measure. And now
27:47:21of course the question is here is my
27:47:23column a dimension or a measure? Well in
27:47:26order to assign it to one of those
27:47:27categories you have to ask the first
27:47:30question is it a numeric value? If it's
27:47:32not so you have like string or date or
27:47:35any other data type then it is a
27:47:36dimension and if it is yes in numeric
27:47:39then you have to ask the second question
27:47:42does it make sense to aggregate it. So
27:47:44if the answer for both questions is yes,
27:47:46it is numeric and it makes sense to
27:47:49aggregate it then it is a measure
27:47:51otherwise it is a dimension. Now let's
27:47:53practice and have some examples. So now
27:47:55by looking to the values of the column
27:47:57category you can see all the values are
27:47:59characters. So it is not numeric that
27:48:02means this column is a dimension. So it
27:48:04is very simple. Let's take another
27:48:06column. We have the sales amount. So now
27:48:08as you can see the values are numeric
27:48:09and as well it makes sense to aggregate
27:48:12those values. we can get the total sales
27:48:14or the average sales and so on. So it
27:48:16fulfill both of the conditions. It is
27:48:18numeric and it makes sense to aggregate
27:48:20it. That's why we say sales is a
27:48:22measure. Now if you're checking the
27:48:24values of the product name, you can see
27:48:26that all of them are characters and
27:48:28names. So it is not numeric. That means
27:48:30the product is a dimension. Moving on to
27:48:32the next one, we have the quantity. The
27:48:34values are numeric and as well it makes
27:48:36sense to aggregate it. Can summarize all
27:48:38those values to have the total quantity.
27:48:40So quantity is a measure. Now if you're
27:48:42looking to the values of the birth dates
27:48:44you can see this is a date information
27:48:46it is not numeric so that means it is a
27:48:49dimension right but if you calculate the
27:48:51age from the birth dates age of the
27:48:54customer going to be in numeric and it
27:48:56makes sense to aggregate it for example
27:48:58finding the average age of customers. So
27:49:01if we derive a numeric value from a
27:49:04dimension then we can use it as a
27:49:06measure. So age is measure and now we
27:49:08come to something really tricky. This is
27:49:10the ID. So for example if you are
27:49:12checking the customer ID you can see all
27:49:14those values are numeric. So the first
27:49:17condition is fulfilled. Now the very
27:49:19important question does it make sense to
27:49:21aggregate the ids? Well those ids are
27:49:24unique identifier for a customer and if
27:49:26you find like the average of that it is
27:49:28not like helpful right I cannot think of
27:49:31one use case of aggregating the customer
27:49:34ID like having the average of all those
27:49:36ids or summarizing the ids. So it makes
27:49:39no sense to aggregate it. That's why we
27:49:42can consider the ID of a customer as a
27:49:44dimension not as a measure. So as you
27:49:46can see it is very simple. If it is
27:49:48numeric and it makes sense to aggregate
27:49:50then it is measure otherwise it is a
27:49:52dimension. And this is the foundations
27:49:55of any data analytics. If you see your
27:49:57data as dimensions and measures you can
27:50:00generate a lot of use cases and insights
27:50:03from your data sets. Now I totally
27:50:05understand if you are still confused
27:50:06about dimensions and measures and you
27:50:08might be asking why do I need measures
27:50:10and dimensions. Well if you are doing
27:50:12any type of data analysis or you are
27:50:14exploring any data sets you will be end
27:50:16up always like grouping up the data by
27:50:18something like you are grouping the data
27:50:20by countries or grouping the data by for
27:50:22example products or categories. So we
27:50:25need dimensions to group up our data and
27:50:28in the other sides you will be asking
27:50:30questions like how much how many what is
27:50:32the total of something. So you always
27:50:34need to aggregate or calculate something
27:50:36right and for that you need the measure.
27:50:38So we need the measures in order to
27:50:40answer the question how many and how
27:50:42much and we need the dimensions in order
27:50:44to group up the data by something. So
27:50:47that's why almost in any type of data
27:50:49analyzes you need dimensions and
27:50:51measures and this going to be more clear
27:50:53as we progress in the projects. All
27:50:54right. So now I'm going to walk you
27:50:56through the project road map and I have
27:50:58split that into six steps. So we're
27:51:00going to do different types of
27:51:01explorations like the database
27:51:03dimensions, measures, dates and we're
27:51:05going to do some basics analyszis like
27:51:07the magnitude and the ranking. So let's
27:51:09start with the first step in our
27:51:10projects. We're going to do database
27:51:16exploration. So let's say that you have
27:51:18joined a team and you got an access to a
27:51:20database. The first thing that I usually
27:51:22do is that I explore the structure of
27:51:24the database just to have basic
27:51:26understandings about the database
27:51:28tables, the views, columns. Are we
27:51:30talking about like 10 tables, hundreds
27:51:32of tables? So it is just a few queries
27:51:35in order to say hello to the database.
27:51:37So now let's go to SQL and explore the
27:51:39database of our projects. So now how we
27:51:41going to do it? Either you go to the
27:51:43left side over here and start clicking
27:51:44the objects of your database and explore
27:51:47the tables, views, columns and so on. Or
27:51:49a better way that I usually do it that I
27:51:51explore the database using a query. So
27:51:53what we can do, we can go and select
27:51:55data from the system tables because the
27:51:58database stores metadata informations
27:52:01about our tables and objects. So we're
27:52:03going to target an information schema.
27:52:06This is an internal schema in the
27:52:07database where we have like multiple
27:52:09tables and views to explore the metadata
27:52:12and the structure of our database. So
27:52:14for example, we can go with the tables.
27:52:16So let's go and create it. And with that
27:52:19you have a list of tables and with that
27:52:22you can see multiple informations like a
27:52:24catalog, the schema and the table names
27:52:27and you can see over here the object
27:52:28type whether it is a table or a view. If
27:52:31you done the data warehouse project with
27:52:32me then you will find a lot of tables.
27:52:35But if you are just doing the data
27:52:37analyzes you will see only those three
27:52:40tables. So customers, products and
27:52:42sales. So with that we can see in our
27:52:44database there are like around 15 tables
27:52:47or three tables. Now in the output you
27:52:49can see the database name the schema and
27:52:52a list of all tables and of course don't
27:52:54forget that you are using the database
27:52:57that we created. So with that we have a
27:52:59nice quick list with all tables inside
27:53:01our database. Now the next step we can
27:53:03go and drill down and check what are the
27:53:05columns that we have inside our
27:53:07database. And for that we can as well
27:53:08target the same schema. So select star
27:53:11from information schema and it is very
27:53:14simple. So we're going to go to the
27:53:15table columns. So let's go and execute
27:53:18it. And now we will see a lot of
27:53:20informations over here. So we can see in
27:53:22our database we have around 101 columns.
27:53:26So that we can see all the columns
27:53:28available in our database. And what I
27:53:30usually do with that I go and select the
27:53:33columns only for specific table. So we
27:53:35can say where are table name
27:53:38equal let's get for example the
27:53:40dimension customers. So let's query the
27:53:43whole thing and with that we can see we
27:53:46have 10 columns inside this dimension
27:53:48and this is how the columns are sorted
27:53:50inside our table or view and we can see
27:53:52all the metadata informations about each
27:53:55column. So now as you can see we are now
27:53:57exploring the structure of our database
27:54:00and this is really helpful to get an
27:54:02overview of the database and the
27:54:03projects. Are we talking about like 20
27:54:06tables or hundreds of tables? And we can
27:54:08quickly see the naming of the columns,
27:54:10the tables. This is really important to
27:54:13get a feeling about the projects and
27:54:15sets the foundations for exploring the
27:54:18data inside those tables. All right
27:54:20friends, so with that we have done the
27:54:21first step. We have explored the
27:54:23database structure and now we can start
27:54:25diving into the actual data. The first
27:54:27thing that we can explore is the
27:54:29dimensions.
27:54:34Okay. So what we going to do with the
27:54:35dimension exploration? All what we have
27:54:36to do is to go and identify the unique
27:54:39values of each dimension that we have
27:54:41inside our database. This can help us to
27:54:43understand what are the categories,
27:54:45which countries, what are the product
27:54:47types that we have inside our database
27:54:49and we have a very simple formula for
27:54:51that. So all what you need is the SQL
27:54:53keyword distinct together with any
27:54:55dimension in your data set like distinct
27:54:57country, distinct category. So for
27:54:59example if you are checking any column
27:55:00that is dimension you can see a lot of
27:55:02values and repeating stuff but now once
27:55:04you say distinct column what going to
27:55:06happen you will get a list of all unique
27:55:08values and with that you can understand
27:55:10quickly I have three different types so
27:55:12I have a bc and this as well going to
27:55:14help you to understand the granularity
27:55:16of your dimension does the dimension has
27:55:19like three values or 100 value so it is
27:55:21very simple let's go and analyze our
27:55:23dimensions okay so now let's explore the
27:55:25dimension values inside our database so
27:55:27let's start with the first table the
27:55:29customers and if you check those columns
27:55:31we have to find an interesting dimension
27:55:33like for example the country. So now
27:55:34what we can do we can go and explore all
27:55:36the countries our customers come from.
27:55:39So let's go and do that. It is very
27:55:41simple. Select distinct and then we have
27:55:43our column the dimension country from
27:55:45our table customers. So let's go and
27:55:48execute it. And with that we can see in
27:55:50the result we have six countries. This
27:55:52is really nice in order to understand
27:55:54the geographical spread. So we have
27:55:56customers for our business that comes
27:55:58from six different countries. Germany,
27:56:00United States, France, Canada and so on.
27:56:02So now with that we have like the first
27:56:04little insights about our business. Now
27:56:06let's jump to another table the
27:56:08products. So what we have to do is to
27:56:10explore all the categories inside our
27:56:12business the major divisions. So we're
27:56:14going to say select distinct category
27:56:17from our table products. So let's go and
27:56:20execute it. Now in the output you can
27:56:22see we have four categories. We have the
27:56:24accessories, bikes, clothing and
27:56:26components. This is like giving us an
27:56:28overview of the product range. What are
27:56:30the major divisions inside our business?
27:56:33Now the next one I'm digging deeper in
27:56:35this information. So not only I want to
27:56:37see the categories, I would like as well
27:56:39to see the
27:56:40subcategories. I'm not starting a new
27:56:42query because there is of course
27:56:44relationship between the category and
27:56:45the subcategory. Let's go now and
27:56:47execute it. Now you can see in the
27:56:48output our categories are now splitted
27:56:51into more specific groups. So for
27:56:54example the bikes over here we have
27:56:56mountain bikes, road bikes and so on. So
27:56:58as you can see the subcategories has
27:57:00more details about the products than the
27:57:03category. And now in order to get the
27:57:05full picture we going to bring now the
27:57:07product name. So with that we're going
27:57:10to get a big picture in one shot. So now
27:57:13you can see the whole hierarchy of our
27:57:15products. And of course it is more
27:57:17interesting if you go and sort the data
27:57:19by those three informations. So let me
27:57:22just execute it again. So now if you go
27:57:23and explore our data for example we have
27:57:25here the category accessories and we
27:57:27have a subcategory inside it called
27:57:29lights. And in this subcategory we have
27:57:31three different products. And if you
27:57:33scroll to the end of our table you can
27:57:35see that we have around
27:57:37295 products. So you can see the
27:57:40granularity of the product name is
27:57:42different than the category and the
27:57:43subcategory. And all those three
27:57:45informations are related to each others.
27:57:47So now as you can see after exploring
27:57:49those dimensions we have now better
27:57:51understanding on how the data is
27:57:54organized and this can help us by the
27:57:55analyzes if you are aggregating by the
27:57:58category you will get only four rows. If
27:58:00you are aggregating by the products you
27:58:02will get hundreds of rows. So this is
27:58:04how we explore the dimensions of our
27:58:07database. Okay. So now with that we have
27:58:09a clear picture about the dimensions
27:58:10inside our data sets. And now in the
27:58:12next step we're going to deep dive into
27:58:13one special type of dimensions. We have
27:58:16the dates. So we're going to explore the
27:58:18date
27:58:21columns. Okay. So now what we going to
27:58:23do with the date exploration? We're
27:58:25going to go and explore the boundaries
27:58:27of the dates that we have in the data
27:58:29sets. What is the earliest and the
27:58:31latest dates in my data? We're going to
27:58:33understand the time span. Do we have in
27:58:35our business 2 years or like 10 years?
27:58:38And this is of course very important to
27:58:40understand in order later to make
27:58:41different types of time analyzes. Now
27:58:44the formula for that is very simple. All
27:58:46what we need is the min and max
27:58:48functions in order to get the earliest
27:58:50and the latest dates. And of course
27:58:51we're going to apply that on date
27:58:53columns, date dimensions. So for
27:58:55example, we're going to have like min
27:58:56order date, max create date, min birth
27:58:59date. So any date that you have in your
27:59:01data set. And here if you look to any
27:59:02date column inside your data, you will
27:59:04find multiple values. But what is
27:59:06interesting is to understand what is the
27:59:08earliest date like here for example 2018
27:59:11and what is the latest date for example
27:59:142028 and with that we can understand aha
27:59:17we have like time span of 10 years using
27:59:20the date diff function. So now let's go
27:59:22and apply our new formula on our date
27:59:24columns. All right. So now let's search
27:59:26for date informations inside our
27:59:28database. And usually you're going to
27:59:30find a lot in the facts. So let's go to
27:59:32the fact cells. And here we have like
27:59:33multiple dates. the order date, shipping
27:59:35date and due dates. Now let's go and
27:59:37explore the boundaries of the order
27:59:39date. So we have the following task.
27:59:40Find the date of the first and last
27:59:43order. So how we going to do that? We're
27:59:45going to say select and we are targeting
27:59:47the order date from our table sales. So
27:59:51let's go and execute it. And now we can
27:59:53see we have a lot of values inside our
27:59:56database. So now in order to find the
27:59:58first dates, what we're going to do,
27:59:59we're going to go and use the function
28:00:01min in order to get the minimum order
28:00:04dates. So we're going to go and call it
28:00:06first order dates. So let's go and
28:00:09execute it. So now we can see the date
28:00:11of the first order. It is in December
28:00:132010. Now let's go and find the date of
28:00:16the last order. So we're going to have
28:00:18this time the max order date. Uh let's
28:00:22go and call it last order date. So let's
28:00:26go and explore now the other boundary
28:00:28and with that we can see in January 2014
28:00:31it is the date of the last order in our
28:00:34system. So with that we have explored
28:00:36the boundaries of the order dates the
28:00:38first and the last and of course we can
28:00:40now understand very quickly that we have
28:00:43four years of sales inside our business
28:00:45but we can go and calculate it. So now
28:00:47the task says how many years of sales
28:00:49are available. Now in order to find the
28:00:52years between those two dates, we have
28:00:54another scale function. It's called date
28:00:56diff. And now we have to go and subtract
28:00:58two dates. Now this function need three
28:01:00arguments. The first one you have to
28:01:02specify whether it is a year, month and
28:01:04day. And we start with the smallest
28:01:06date. So it's going to be the min order
28:01:09dates. And then the last argument is
28:01:11going to be the latest or the highest
28:01:13date. And it's going to be the max order
28:01:15dates. And we can go and call it order
28:01:18range in years. Okay. So let's go and
28:01:21execute it. And with that you can see in
28:01:23the output we have four years. Of course
28:01:25if you want to go and check the months
28:01:27you can go over here and say month and
28:01:29execute. So between those two dates we
28:01:31have 37 months. And of course now we
28:01:34have to go and rename it. So with that
28:01:36we have explored the dimension order
28:01:39dates. But what is more interesting is
28:01:41to check the customers and here we have
28:01:44the birth date. So now what we can do,
28:01:46we can go and find the youngest and the
28:01:49oldest customer. So let's go and do
28:01:51that. We're going to say select
28:01:53minates and with that we are getting the
28:01:56oldest birth date and we will get now
28:01:59the max birth date and with that we will
28:02:02get the youngest birth date from our
28:02:06table customers. So let's go and explore
28:02:09that. Now we can see the birth date of
28:02:11the oldest customer. I hope he or she is
28:02:14still alive. So it is more than 100
28:02:16years and the youngest customer is
28:02:19around like 40 years. So we don't have
28:02:22really young customers inside our
28:02:23business. And of course if you don't
28:02:25want to see the birth dates, you want to
28:02:27see the age, what you have to do is
28:02:29actually very simple. You're going to
28:02:30use as well diff and we want the year
28:02:33and then we're going to say min birth
28:02:36date with the current date and time. And
28:02:39for that we have a function called get a
28:02:41date and we're going to call it oldest
28:02:44age. So if you go ahead and execute this
28:02:47one over here you can see the age of the
28:02:49oldest customer it is 109. Of course you
28:02:52can do the same thing for the youngest.
28:02:54If you just replace this with max and
28:02:57here we have the youngest age. So let's
28:03:00go and execute it. It is 39. So my
28:03:03friends this is how we explore the
28:03:05boundaries of a date and by finding the
28:03:08first date and the last date and the
28:03:10years between them we are having now
28:03:12more understanding of the time span of
28:03:14our business and that's going to help us
28:03:16later by making different type of
28:03:18complex analyzers. So this is how we
28:03:20explore the dates. All right. So with
28:03:22that we have now a clear picture about
28:03:24the scope of our projects and the date
28:03:26range inside our data sets. Now in the
28:03:28next step, we're going to go and explore
28:03:30the second type of data, the
28:03:35measures. All right. So now what is
28:03:37exactly exploring the measures? What
28:03:39we're going to do is to calculate and
28:03:41find out the key metrics of our
28:03:44business, the big numbers, the highest
28:03:46level of aggregations of our data. And
28:03:48the formula for that is very simple.
28:03:50We're going to go and use the aggregate
28:03:52functions in SQL like the sum, average,
28:03:55count for any measure inside our data
28:03:57sets. So for example, we're going to
28:03:59find the total sales by summarizing the
28:04:01sales value, finding the average price,
28:04:04finding the sum of quantity in order to
28:04:06have a big number about all sold items.
28:04:09So always an aggregate function together
28:04:11with a measure. So for example, if you
28:04:12have a column where you have a lot of
28:04:14values and you go and summarize all
28:04:16those values, you will get for example
28:04:18240. So this is a key metric. This is
28:04:21the highest level of aggregations and
28:04:23the value is not splitted at all. So for
28:04:25example, we say this is the total
28:04:27revenue of our business. And this is
28:04:29exactly what we mean by exploring the
28:04:31measures. We will get those big numbers.
28:04:33So now let's go and apply those
28:04:34aggregate functions to the measures that
28:04:36we have inside our data set. Okay. So
28:04:38now we're going to go and spotlight on
28:04:40the big numbers that matters the most of
28:04:42our business. So now based on those
28:04:44three tables, I have collected here the
28:04:47following questions. So let's go and
28:04:48solve them one by one. The first one is
28:04:50find the total sales. So we're going to
28:04:53go and summarize by using the sum
28:04:55function for the sales amount as total
28:04:59sales from our table fact sales. So
28:05:03let's go and execute it. So this is the
28:05:06total amount of sales in our business.
28:05:08It is around 29 millions. So this is the
28:05:12business total revenue. Now we can go to
28:05:14the second one. It says show how many
28:05:17items are sold. So this time we need
28:05:19another column but from the same table
28:05:22from the fact sales. So the question is
28:05:24how many items that means we want the
28:05:26quantity and we're going to stay with
28:05:28the same function. So we are summarizing
28:05:30all the values of the quantity and we
28:05:32can call it total quantity. Let's go and
28:05:36explore that. So we can see our business
28:05:38did sold around 60,000 items and these
28:05:4260,000 items did generate around 30
28:05:44million. So let's keep going. The next
28:05:46question, find the average selling
28:05:48price. So that means we are targeting
28:05:50the same table. And here we have the
28:05:52price informations. So we're going to
28:05:54say the price. This time the aggregate
28:05:56function going to be the average. And
28:05:57we're going to call it average price. So
28:06:00let's go and execute it. So the average
28:06:02price in our business is 486. So that
28:06:05means our business is selling like
28:06:07expensive items. Now let's go to the
28:06:09next question. It says find the total
28:06:12number of orders. And for that we're
28:06:14going to go and use the function count
28:06:16and we can count the order numbers. So
28:06:20order number total orders let's go and
28:06:24execute it. So it says we have 60,000
28:06:27orders. And now as you are working with
28:06:28the count function what I usually do I
28:06:31try to count the same thing but using a
28:06:34distinct. So distinct order number. So,
28:06:37what I'm trying to do here is first
28:06:40eliminate any duplicates in the order
28:06:42number and then count it. I don't want
28:06:44to count the same order twice inside our
28:06:47sales. So, let's go and execute that.
28:06:50Now, as you can see, we have only 27,000
28:06:53orders out of 60,000. So, that means the
28:06:57same order is repeating in our database.
28:07:00Let's have actually a look. So, select
28:07:02star from our table and let's go and
28:07:06have a look. Now as you can see from the
28:07:07first order over here you can see the
28:07:09same order is repeated three times and
28:07:12that's because this customer did order
28:07:14three things in the same order. So now
28:07:17of course what is the definition of
28:07:19order? Usually the whole thing is one
28:07:21order. That's why in order to get an
28:07:23accurate number of orders you have to go
28:07:25and use a distinct in order to eliminate
28:07:28first all duplicates and then count how
28:07:30many orders we have. So in this scenario
28:07:32I'm going to say in our business we have
28:07:34around 27,000 orders. So that's why it
28:07:37is little bit tricky using the count
28:07:39function. Always try to compare the
28:07:41numbers before and after using distinct.
28:07:44So let's keep going to the next one. It
28:07:46says find the total number of products.
28:07:48So it is very simple. We're going to say
28:07:50select count and we're going to say
28:07:52product key as total products from the
28:07:57table gold products. So let's go and
28:08:00execute it. So as you can see we have
28:08:02295 and if you go and make it distinct
28:08:06just to check you will get the same
28:08:09number. So that means there is no
28:08:10duplicates and of course you can go and
28:08:12count the product name instead. The
28:08:14names of the product is unique. So
28:08:15that's why we are as well getting the
28:08:18same numbers. So that's it. Let's
28:08:20continue find the total number of
28:08:22customers. So the same thing select
28:08:25count and you can go with a customer key
28:08:27for example from called a dimension
28:08:30customers and I'm going to call it as
28:08:33total customers. So let's go and execute
28:08:36it. So we can see in our system we have
28:08:3818,000 registered customers. Now the
28:08:41next one it says find the total number
28:08:43of customers that has placed an order.
28:08:45So that means having a customer inside
28:08:48our database doesn't mean that this
28:08:50customer did already placed an order.
28:08:52Maybe we have customer that just
28:08:54registered and didn't order anything. So
28:08:56what we're going to do, we're going to
28:08:57take the same query, but instead of
28:08:59targeting the customers table, we're
28:09:01going to target our fact the sales. So
28:09:04let's go and execute it. So now, as you
28:09:06can see, we are getting 16,000, which
28:09:08makes no sense because one customer
28:09:10might order multiple stuff. So what
28:09:12we're going to do, we're going to say
28:09:14distinct and let's query it again. So
28:09:16now it is more correct. We are getting
28:09:18around 18,000 customers. Now we can go
28:09:21and compare them one by one. So as you
28:09:23can see we are getting the same numbers.
28:09:25So that means all our registered
28:09:26customers did already placed an order
28:09:29because the numbers are matching. So it
28:09:31is very simple. We are just using an
28:09:33aggregate functions and that we are
28:09:35getting those key values. But what I
28:09:37usually do is that I collect all those
28:09:39measures in one query in order to have
28:09:41an overview of all key numbers in our
28:09:44business. So instead of me querying each
28:09:46one of them individually, I combine them
28:09:49in one go. So now what we're going to
28:09:50do, we're going to generate a report
28:09:52that shows all key metrics of our
28:09:54business. So how I usually do it, I'm
28:09:56going to go and get the first query for
28:09:57the total sales and put it over here.
28:09:59And now I'm going to build only two
28:10:01columns. The first one is the name of
28:10:03the measure and the second one is the
28:10:05value of the measure. So let me show you
28:10:07what I mean. Now this one over here, I
28:10:09will not call it total sales. I'm going
28:10:11to make it like generic. So I'm going to
28:10:13say measure value. And before it we're
28:10:16going to make another column from a
28:10:18static string value is the total sales
28:10:21and we're going to call it measure name
28:10:23like this. So let's go and just execute
28:10:26this one over here. So the measure is
28:10:28total sales. So it is not anymore like
28:10:31the column name. It is now a value in
28:10:33the output and the measure value is like
28:10:35around 30 millions. Now what I'm going
28:10:37to do I'm going to go and add another
28:10:39measure as a second row. And in order to
28:10:41do that, we're going to use the union
28:10:43all and then copy the whole thing over
28:10:45here and say total quantity and we're
28:10:48going to change the measure to quantity.
28:10:51So now let's select both of them and
28:10:53query. And now as you can see we have
28:10:55now the two big numbers in one query. So
28:10:58the total sales and the total quantity.
28:11:00So now what we can do we can go and
28:11:01collect all those big numbers and
28:11:03measures and put it in one query. So
28:11:06with that we have the average price, the
28:11:07total number of orders, product,
28:11:09customers and as well you can go and
28:11:11target different tables because SQL
28:11:13cares here only about the number of
28:11:15columns and the data type of columns
28:11:17must be matching. So now let's go and
28:11:20query this and now in single query we
28:11:23can see the big numbers the key metrics
28:11:26of our business. We can see the total
28:11:28sales, total quantity, average price and
28:11:30so on. This is a super report where you
28:11:32can generate it for any business where
28:11:34you have in one go the full big picture
28:11:37about the business. So this is how I
28:11:39generally do if I'm exploring a new
28:11:40database. I put all those big numbers
28:11:43and measures in one query to have better
28:11:46understanding about the business. All
28:11:47right my friends. So with that we have
28:11:49now a clear understanding about the
28:11:51dimensions and as well the measures of
28:11:53our data sets. Now in the next step
28:11:55we're going to go and start combining
28:11:57stuff together in order to generate
28:11:58insights. And we're going to focus now
28:12:00in a very basic analyszis. It is the
28:12:02magnitude
28:12:07analyzis. Okay. So now what is exactly a
28:12:09magnitude analyszis? It's all about
28:12:11comparing the measure values across
28:12:13different categories and dimensions. And
28:12:15this can help us of course to understand
28:12:17the importance of different categories.
28:12:20Now the formula for that going to be
28:12:21interesting. So now this time we will be
28:12:23mixing stuff together. So first we have
28:12:25to go and aggregate a specific measure
28:12:27and then we say by dimension. We need
28:12:30here the dimension in order to split the
28:12:32measure. It sounds complicated but it is
28:12:34very simple and basics. So for example
28:12:36we can say the total sales by country,
28:12:39the total quantity by category, the
28:12:41average price by products, the total
28:12:44orders by customer and if you follow
28:12:46this formula you will be generating
28:12:48endless amount of insights by just
28:12:50combining any measure with any
28:12:51dimension. You can call it it is a new
28:12:53insight. So it's going to look like like
28:12:55this. If you have one measure that is
28:12:57like for example 600 and if you put now
28:12:59this measure together with dimension
28:13:01what's going to happen this 600 is going
28:13:03to be splitted by the dimension values.
28:13:05So A going to have like 200, B going to
28:13:07have 300 and C 100. And now with that we
28:13:10can go and compare those categories
28:13:12right. So we can see now that category B
28:13:15has the highest measure and the C has
28:13:17the lowest. And this help us to compare
28:13:19the values of the measure. what is the
28:13:20best category and what is the worst
28:13:22category. So this is very basics
28:13:24analyszis. So let's go and apply this
28:13:26formula on our data sets. Okay. So now
28:13:29let's go and break all our measures by
28:13:31dimensions. So here I have prepared few
28:13:33interesting examples where first we're
28:13:35going to break the total number of
28:13:37customers. As we learned we have 18,000
28:13:39by the countries. So the measure is
28:13:42total customers and the dimension going
28:13:44to be the countries. So let's go and
28:13:45write the query for that. So we're going
28:13:47to select. So the first thing that we're
28:13:48going to go and add is the dimension. So
28:13:50it's going to be the country. And then
28:13:52we need the measure. It's going to be
28:13:54the count of the customer key. So this
28:13:58will give us the total customers. And we
28:14:01need to select our table. So it's going
28:14:03to be the dimension customers. And of
28:14:05course we have to go and group up the
28:14:07data by the countries. So group up
28:14:10country. So let's go and execute it. And
28:14:13with that you see again the list of
28:14:15countries. So we have our six countries
28:14:18and then the total customers for each
28:14:20country. So that we can see the
28:14:22distribution of customers by the
28:14:24country. But what we usually do is that
28:14:26we go and sort the data by the measure
28:14:28the total customers like this. And we're
28:14:31going to sort it by descending. So with
28:14:33that we will get first the countries
28:14:35with the highest customers. So let's go
28:14:38and execute it. So now we can see in the
28:14:40results the highest number of customers
28:14:42come from United States then Australia,
28:14:44United Kingdom
28:14:46337 customers without the country
28:14:50informations it is not available. So
28:14:52that's it right it is very simple. So
28:14:54with that we have splitted the total
28:14:56number of customers by a dimension the
28:14:58country. Now of course we can go and
28:15:00split the data by different type of
28:15:02dimension. So for the next one we are
28:15:04saying find the total customers by
28:15:07gender. So here's the same thing. We
28:15:08have the same measure that to other
28:15:10customers but we are splitting the data
28:15:12by different type of dimension. So just
28:15:14copy and paste and now instead of
28:15:16countries we just going to switch it to
28:15:18gender and over here and that's it. So
28:15:21let's go and execute. So now as you can
28:15:23see the granularity of the gender over
28:15:26here is different than the countries. We
28:15:27have here only three values and we can
28:15:30see it is almost splitted evenly between
28:15:32male customers and female customers. And
28:15:34of course this going to help us to
28:15:36understand the demography of our
28:15:38customers. And as you can see it was
28:15:40very simple. We just switch the
28:15:42dimension. So you can go and split as
28:15:44well by the marital status and so on.
28:15:46Now let's go and split the total
28:15:49products by the category. Well actually
28:15:51the query is going to be very simple as
28:15:53well. So select and here we're going to
28:15:54have the same aggregate function the
28:15:56count products key as total
28:16:00products from our table gold dimension
28:16:04products and then we're going to group
28:16:05up by the dimension the category and
28:16:08we're going to order by as well the same
28:16:10thing total products distinct from the
28:16:13highest to the lowest. So let's go and
28:16:15execute it. And with that we can see how
28:16:17many products do we have in each of
28:16:19those categories. And we can see the
28:16:21biggest category the components and
28:16:23after that the pikes. And this is
28:16:25interesting that we have seven products
28:16:27where we have nulls where they don't
28:16:29belong to any category. This is really
28:16:31nice. Let's go to the next one. What do
28:16:33we have over here? What is the average
28:16:36costs in each category? So this is like
28:16:38different style of question but at the
28:16:40ends we're going to have the same thing.
28:16:41We have over here the average costs.
28:16:43This is the measure and the category is
28:16:46our dimension. It's like we are saying
28:16:48find average costs by category. So what
28:16:51we're going to do, we're going to go and
28:16:52copy the same query and the dimension is
28:16:54the same. So the categories but the
28:16:57measure is different. We are not talking
28:16:59about the total products. We are going
28:17:00to say average and here we're going to
28:17:02have the column costs and let's go and
28:17:04rename it average costs. So that's it as
28:17:07well for the order by we have to use the
28:17:10new measure. So let's go and execute it.
28:17:12So now we can see the most expensive
28:17:14category is the bikes costs a lot
28:17:17compared to the accessories of course.
28:17:19So you can see the accessories is only
28:17:2113 and the bikes is 900. So this is as
28:17:24well gives us insights about how
28:17:26expensive each category is and as you
28:17:28can see it is always the same templates.
28:17:31We are splitting specific measure by a
28:17:33dimension. So let's keep going to the
28:17:35next one. It says what is the total
28:17:38revenue generated for each category. So
28:17:40again here the question is find the
28:17:43total revenue by category. So the total
28:17:46revenue here is the measure and the
28:17:48category again is the dimension. So now
28:17:50the total revenue comes from the fact
28:17:52and the category comes this time from
28:17:54the dimension. So that means we have to
28:17:56go and join tables right. So how we
28:17:58going to do it? Let's go and start with
28:17:59the select star from and I would like
28:18:02always to start from the fact table. So
28:18:05fact sales f and then we're going to go
28:18:07and join it with the dimension and
28:18:09usually I go with the left join in order
28:18:10to not lose anything because if you use
28:18:13an inner join you might lose in the fact
28:18:15few orders and few sales I don't want
28:18:17that. So lift join with the dimension
28:18:20this one going to be the products and
28:18:22the key for that going to be very simple
28:18:24going to be the product key and the same
28:18:26thing for the facts. So with that we
28:18:28join the fact table with the dimension.
28:18:30So now we have to go and pick what do we
28:18:32need? We need from the fact the sales
28:18:34right. So sales amount and we need from
28:18:37the products the category and we want to
28:18:40group up the data by the category. So so
28:18:43this part is done. What is missing is of
28:18:46course the aggregations. So we are
28:18:48aggregating actually the sales. So sum
28:18:51sales and we can call it total revenue.
28:18:54So like this. And of course we can go
28:18:57and order the data by the total revenue
28:18:59by our measure and distinct from highest
28:19:02to the lowest. So as you can see it is
28:19:03exactly like the previous one. But here
28:19:05the data doesn't come from only one
28:19:07table. Here it comes from two tables. So
28:19:09the measure come from the facts and the
28:19:11dimension come from the dimension
28:19:13products. And this is classic right? The
28:19:15dimension has all those descriptions and
28:19:18details about the products like the
28:19:20categories. And the fact table has all
28:19:22those measures and dates that we use in
28:19:25order to calculate our measures. So
28:19:27that's it. Let's go and execute it. Now,
28:19:29as you can see in the output, the
28:19:31category bikes is bringing the most of
28:19:33revenue. So here it's like in millions
28:19:3628 millions of sales and the accessories
28:19:39and the closing is not really bringing a
28:19:41lot of like revenue. Both of them are
28:19:43below like 1 million. So with that you
28:19:45can understand our business is making a
28:19:47lot of money selling bikes, right? So my
28:19:50friends as we are exploring the data we
28:19:52are understanding more and more about
28:19:54our business right so let's keep going
28:19:56to the next one we have here the
28:19:57question what is the total revenue
28:19:59generated by each customer so now we
28:20:01want to find out the top spender right
28:20:04select star and as well we start from
28:20:06the fact table and this time we're going
28:20:10to lift join it with the customers right
28:20:12so the dimension customers and we're
28:20:15going to go join the data so we're going
28:20:17to use the customer key for the join And
28:20:19what we're going to do, we're going to
28:20:20go and get maybe the customer key. And
28:20:23let's go and get as well the first name,
28:20:25maybe few details about the customer and
28:20:28as well the last name. So those are the
28:20:30columns that we want from the customers.
28:20:32And now what do we need? We need the
28:20:34aggregation. So it's going to be the
28:20:35same thing. Sales amount as total
28:20:38revenue. And we have to go and group up
28:20:41the data by all those three
28:20:43informations. So we're going to go and
28:20:45copy paste. And at the end as usual,
28:20:47we're going to order by the measure
28:20:49total revenue descending. So that's it.
28:20:52It is exactly like previous one but with
28:20:54different dimensions. So let's go and
28:20:56query it. And now we get a full list of
28:20:59all our customers, the 18,000s. And we
28:21:02can see the total revenue for each
28:21:04customer. So we can see Nicole and
28:21:06Caitlyn, they are our top spenders and
28:21:09the most royal customers that generated
28:21:12sales and revenue for our business. This
28:21:14is really cool. Right now let's go to
28:21:16the next one. It says what is the
28:21:18distribution of sold items across
28:21:21countries. It is like finding the total
28:21:23quantity by countries. So it is very
28:21:26simple. I'm going to go and take the
28:21:27same query because countries comes from
28:21:29the dimension customers and the sold
28:21:32items the quantity come from the sales.
28:21:34So we are doing the same joints but with
28:21:36different dimensions and measures. So
28:21:38what do we need from the customers is
28:21:40only the country and the measure going
28:21:42to be the quantity. And here we're going
28:21:44to go and say total sold items and we
28:21:47have to change the group by to the
28:21:50countries and sorting the data by the
28:21:52new measure. That's it. And with that we
28:21:54are generating new reports by just
28:21:57changing the dimensions and measures. So
28:21:59again this is very interesting to
28:22:00understand which country is generating
28:22:02like good business for us. So my friends
28:22:04as you might already noticed if in the
28:22:06dimension we have like small number of
28:22:09unique values like in the countries we
28:22:11have here only seven values in the
28:22:12gender we have only three we call those
28:22:14dimensions low cardality dimensions
28:22:17because we have low number of values
28:22:19inside it and in the result we will get
28:22:21only here for example seven rows but if
28:22:23our dimension is high cardality like by
28:22:27the customers we have 18,000 unique
28:22:30customers then our measure going to be
28:22:32splitted by those 18,000 and in the
28:22:35results we will get exactly the same
28:22:37number of customers. So the number of
28:22:39rows and results really depends on the
28:22:41cardality of the dimension. So as you
28:22:43can see we can generate a lot of
28:22:45different reports by only following this
28:22:47formula dividing the measure by a
28:22:49dimension. So we just generated eight
28:22:52different insights and reports by only
28:22:54few measures and dimensions. So now what
28:22:56you can do you can pause the video and
28:22:57try different dimensions and measures in
28:23:00order to have more insights about our
28:23:02business. Okay. So as you can see this
28:23:04is the basics analyszis that we can do
28:23:06in any data set or any domain where we
28:23:08are aggregating a measure by dimension.
28:23:10Now in the next and last step in our
28:23:12projects we will be doing ranking
28:23:18analyszis. Okay. So what is ranking
28:23:20analyszis? It is very basic. We're going
28:23:22to go and order the value of our
28:23:24dimension based on a measure in order to
28:23:27identify the top performers and as well
28:23:29the bottom performers. And the formula
28:23:31for that is going to be the following.
28:23:33So this time we're going to be ranking
28:23:34the dimensions by an aggregated measure.
28:23:37So for example, we're going to rank the
28:23:38countries by the total sale or we're
28:23:40going to find the top five products by
28:23:42the sold item, the quantity or the
28:23:45bottom three customers by total orders.
28:23:47So it's like the magnitude analyzes.
28:23:49We're going to have like an ordered list
28:23:51of dimensions value. For example, from
28:23:53the highest to the lowest in order to
28:23:55identify quickly the top performers. And
28:23:57of course we can go and filter the data
28:23:59by saying I would like to have only the
28:24:00top two categories. And with that you
28:24:03are removing all other dimensions that
28:24:05are not on the top two. And in SQL we
28:24:07can use for that the keyword top or we
28:24:10can use the ranking window functions
28:24:12like rank, dense rank, row number and so
28:24:15on. So let's go and apply our formula in
28:24:17order to rank our data set. Okay. So now
28:24:20let's check our data. We're going to
28:24:22start with the first question. Which
28:24:23five products generate the highest
28:24:25revenue? So we are searching for the
28:24:27best performing products in our
28:24:29business. So of course the first
28:24:30question what is the dimension and
28:24:32measure that we have in this question.
28:24:34Well the revenue that means we need the
28:24:36sales from the facts and the products
28:24:38that means we need the dimension
28:24:40products. Now in order to write this
28:24:41query it's going to be very simple. So
28:24:43we can use as well the group by I will
28:24:45not write it from the scratch. So I'm
28:24:47just going to take this query over here
28:24:49where we aggregated the total sales by
28:24:51the category. Now what I have to do is
28:24:53just to change the dimension. So instead
28:24:56of the category we need the product name
28:24:59and we are aggregating now the data by
28:25:01the product name because we need the top
28:25:03five products right. So the revenue is
28:25:06the sales amount and with that we have
28:25:08like almost everything is ready. So
28:25:09let's go and execute it. And now we can
28:25:11see we have a list of all products in
28:25:13our business and as well we can see the
28:25:16total revenue. But the task says here we
28:25:18need the top five. So we don't need all
28:25:21the products from our database. We have
28:25:23to go and select only this subset. Now
28:25:25in order to do that in SQL server, it's
28:25:27very simple. We're going to go over here
28:25:28and say top five and SQL going to go and
28:25:31return only the first five rows from the
28:25:34results. So let's go and execute it. And
28:25:36as you can see now in the results, we
28:25:38have only five products with the highest
28:25:40sales. And that's it. With that, we have
28:25:42solved the task and we can see the top
28:25:44five products and all of them are pikes.
28:25:47Now let's go and check the other sides.
28:25:49We want to find the five worst
28:25:51performing products by the same measure,
28:25:53the sales. And this is very simple. So
28:25:56what we're going to do, we're going to
28:25:57go and take the same query over here.
28:25:59And now what we're going to do, we're
28:26:00going to go and sort the data from the
28:26:02lowest to the highest. So instead of
28:26:05descending, we're going to remove it.
28:26:07And with that, SQL going to use the
28:26:09ascending. So let's go and execute it.
28:26:11And with that, as you can see, we are
28:26:13getting the worst five performing
28:26:15products by just sorting the data
28:26:17differently. So it is very simple right
28:26:19and with that we can see our five best
28:26:21sellers and the five worst sellers. And
28:26:24now what we can do we can go and just
28:26:25change the dimension and generate
28:26:28different reports like instead of the
28:26:29product name let's go and check the
28:26:31subcategories what are the best
28:26:33subcategories of our data. So I just
28:26:35change the dimension let's go and query.
28:26:37So with that we can see the best
28:26:39subcategories we have in our business
28:26:41and the same thing if you want to go and
28:26:42check the worst performing
28:26:44subcategories. So generating reports is
28:26:46very simple and now my friends in SQL
28:26:48there is like two ways on how to create
28:26:50ranking. We have a simple one where we
28:26:53are using the group by clouds together
28:26:55with the keyword top. But if you are
28:26:57generating a reports where it's things
28:26:59are more complex and you need more
28:27:01flexibility, you should use the window
28:27:03functions. So let me show you how I can
28:27:05solve this task using the window
28:27:07function. So now I'm going to go and
28:27:08take almost the same query. Let's put it
28:27:11over here. I'm going to get rid of the
28:27:13top five. And let's see, we are still
28:27:15speaking about the products name as well
28:27:18with a group I. But now what we're going
28:27:20to do, we're going to go and generate a
28:27:21rank. So we can go and use for example
28:27:23the row number. And in scale there's
28:27:25like different types of window functions
28:27:27for ranking. One of them is the row
28:27:29number or the rank and then we're going
28:27:31to say over. Now we're going to go and
28:27:33sort the data. It's like we have done in
28:27:35the previous one. We have to sort the
28:27:37data by the total revenue and the total
28:27:39revenue is the sum of sales and
28:27:41descending and we're going to call this
28:27:43rank products. So let's go and execute
28:27:47it. Now as you can see we have created a
28:27:49new column where we have like a rank. So
28:27:52we have for each products like one rank
28:27:55until the last
28:27:56products 130. So now what we are
28:27:59interested is to go and select the top
28:28:01five. Right now in order to do that we
28:28:04need a second step. That's why we're
28:28:05going to go and use the subquery. So
28:28:07we're going to say select star from and
28:28:09then we're going to put the whole thing
28:28:11in a subquery something like that. And
28:28:14all what you have to do is to use the
28:28:16new flag that we have created in order
28:28:18to filter the data. So we're going to
28:28:20say where the rank products is smaller
28:28:22or equal to five. And with that we
28:28:25should get only the top five products.
28:28:28So let's go and execute it. And as you
28:28:30can see we are getting the same results.
28:28:32Now, of course, with the window
28:28:33function, it is more complicated than
28:28:35the first one. But with the window
28:28:36function, we get more flexibility on
28:28:38selecting more columns or adding more
28:28:41different types of aggregations and
28:28:42details on the query. And as well, we
28:28:44can go and use different types of
28:28:46ranking functions that handles the tice
28:28:49differently. So, if the task is very
28:28:51simple like this, I'm going to go with
28:28:53the simple group pie. But if you are
28:28:55generating like complex reports, I'm
28:28:57going to go with the window function. So
28:28:59now what you can do, you can go and rank
28:29:00the data by different dimensions and
28:29:02measures. For example, find the top 10
28:29:04customers who have generated the highest
28:29:06revenue. And as well, you can go and
28:29:08find the three customers with the fewest
28:29:11orders placed. So again, we can go and
28:29:14reuse the previous queries that we have
28:29:16generated. So this query generates the
28:29:18customers and their total sales. And all
28:29:20what you have to do is to say top 10 and
28:29:23then rerun the query. And with that, we
28:29:25are getting the top 10 customers. and
28:29:27about the lowest three customers. All
28:29:30what we have to do is to go and replace
28:29:32the measure. So we are counting the
28:29:36unique number of orders. So we're going
28:29:39to say total orders and as well go
28:29:42change the order by not descending
28:29:44ascending. And we need the top three. So
28:29:47let's go and execute it. So we can see
28:29:50the three customers that did order only
28:29:52once and they are the three customers
28:29:54with the fewest orders. So as you can
28:29:56see by just switching the dimensions and
28:29:58measures we are generating completely
28:30:00new important insights and as you can
28:30:02see as we are exploring the data we are
28:30:04understanding what are the best products
28:30:06what are the top customers that are
28:30:08usually very important for reporting.
28:30:11All right my friends so with that we
28:30:12have covered the last step in our
28:30:15projects how to rank our data and with
28:30:17that we have covered all the steps of
28:30:19the project road map. We have done a lot
28:30:21of explorations for the database,
28:30:22dimensions, measures. We have combined
28:30:25the dimensions and measures in order to
28:30:27do magnitude and ranking analyszis.
28:30:29Okay, my friends. So that's all about
28:30:31the EDA projects. And now in the next
28:30:33one, we will do the last type of
28:30:35projects, the advanced data analytics.
28:30:37So let's
28:30:42go. And now the type that we're going to
28:30:44cover is advanced analytics projects
28:30:46using SQL where we're going to write
28:30:48complex SQL queries to answer real
28:30:50business questions. So we're going to
28:30:52use the advanced window functions, the
28:30:54CTE subqueries and we're going to go and
28:30:56script two big queries in order to
28:30:59generate two reports. So with this type
28:31:01of project, you will learn how to solve
28:31:03real business questions using advanced
28:31:05techniques. All right. So for this
28:31:06project as well, we have a road map
28:31:08where we're going to progress through
28:31:10different type of steps and analyzes. So
28:31:12we're going to do many stuff like change
28:31:13over time, cumulative analyszis,
28:31:15performance, data segmentations and at
28:31:18the end reporting and all using SQL. So
28:31:20let's start with the first step in the
28:31:21road map. We going to analyze the change
28:31:24over time. So let's
28:31:28go. Okay. So now what is change over
28:31:31time? It is a technique in order to
28:31:33analyze how a measure evolves over the
28:31:35time. And this is very important in
28:31:37order to track the trends and as well to
28:31:40identify seasonality of your data. And
28:31:42the formula for that is very simple.
28:31:44We're going to go and aggregate a
28:31:45measure but this time based on a date
28:31:48dimension. For example, the total sales
28:31:50by a year, the average cost by the
28:31:53month. So if you combine any aggregated
28:31:55measure together with a date column or
28:31:57dimension, then all what you are doing
28:31:59is you are analyzing the change over
28:32:01time. So for example, we're going to go
28:32:03and break our measure this time for
28:32:04example by the years. And with that we
28:32:06can track immediately how our business
28:32:09is doing over the time over the years.
28:32:11So for example, we can see here the best
28:32:13year was 2024 and then we have really
28:32:15hard decline in our business in 2025 and
28:32:18then slightly it's going up in 2026. So
28:32:21with that we can quickly analyze the
28:32:23trends of our business. So now let's go
28:32:25and check the trends and the changes
28:32:27over time in our business. Okay. So now
28:32:29let's analyze the trends and changes
28:32:31over time in our data and in order to do
28:32:33this kind of analyzes usually we target
28:32:36the fact table because there usually we
28:32:38have our measures and as well dates. So
28:32:40we have the order date, shipping date
28:32:42and due date. Now what we can do we can
28:32:44go and analyze there the sales
28:32:46performance over time. So as we learned
28:32:48all what we need is a metric and a date.
28:32:51Let's go for example and select the
28:32:54order date and as well one of those
28:32:56measures sales amount from our fact
28:33:00table. So let's go and query it. And we
28:33:04can go and order the data by the order
28:33:06dates ascending. So let's go and
28:33:09execute. And as you can see we have
28:33:11nulls in our data. What we can do? We
28:33:13can go and filter those data out. We
28:33:15don't need it. So we're going to say
28:33:17where order date is not null. So let's
28:33:19go and execute it again. All right. So
28:33:22that we don't have those orders. Now, as
28:33:24you can see, we have sales over time,
28:33:26right? We have a date and we have a
28:33:28measure. So this looks really good. But
28:33:31now what we're going to do, we're going
28:33:31to go and aggregate the data by the
28:33:33sales amount. So let's go and say sum.
28:33:37And we're going to call it total sales.
28:33:39And then we group up the data by the
28:33:42order dates. So let's go and execute it.
28:33:45And with that, as you can see, for each
28:33:47day, we have the total sales. So now the
28:33:49granularity of our data is the day and
28:33:52we can say of course now we are
28:33:54analyzing the sales over time but
28:33:56usually we don't aggregate the data on
28:33:58the day level we want to have higher
28:34:00aggregations for example let's go to the
28:34:02years and now in order to change the
28:34:05dimension date here from a day to a year
28:34:07we have to use date functions and there
28:34:10are a lot of date functions in order to
28:34:12extract that date part and now in order
28:34:14just to get the year we have a quick
28:34:16function called year and it going to
28:34:18convert convert our date to year. So
28:34:21let's call it order year and of course
28:34:23we have to go and group up the data by
28:34:26the year and as well sort it by the
28:34:28year. So let's go and execute. Now we
28:34:30are at the year level and we have only
28:34:33five years. So that means we have
28:34:35changed the aggregation from the day to
28:34:37year and now it is very easily to
28:34:39analyze the performance of our business
28:34:42over the years. So the first year was
28:34:44the lowest and you can see 2013 is the
28:34:47best year in our business and then it is
28:34:50declined massively in 2014. And of
28:34:53course we can go and add more measures
28:34:55to our data not only the total sales.
28:34:57For example, let's go and calculate the
28:34:59total number of customers. So we can say
28:35:02count distinct customer key as total
28:35:06customers. So let's go and execute it.
28:35:09And with that we can check are we
28:35:10gaining like customers over the time if
28:35:13there are any trends that we can see and
28:35:15we can go and keep extending stuff like
28:35:17we can go and add the total number of
28:35:19quantities. So summarize quantity as
28:35:23total quantity. So let's go and execute
28:35:26and with that we have really nice
28:35:27picture in order to understand is the
28:35:30revenue increasing or decreasing over
28:35:32the time what is the best year the worst
28:35:35year are we gaining customers over time
28:35:37if there any like trends that we can
28:35:39spot now by looking to the result you
28:35:41can see this gives us highlevel
28:35:43long-term view of your data and of
28:35:45course it helps for strategic decisions
28:35:48and now what we can do we can go and
28:35:50drill down to the months so we can go
28:35:52and aggregate the data by the month
28:35:54regardless list the years in order to
28:35:56give us an idea how each month is
28:35:58performing on average. So all what we
28:36:00have to do is to switch the function
28:36:02from year to a month like this. And of
28:36:04course for the group by and the order by
28:36:06let's go and execute and of course in
28:36:09the output we will get all the months
28:36:11and guess what which month is the best
28:36:13for sales is of course December because
28:36:16you have all those Christmas and stuff
28:36:18and the worst months as you can see is
28:36:20February. So with that we are
28:36:22understanding the seasonality of our
28:36:24business and the trends patterns of our
28:36:26business. And as you are not including
28:36:28the year in our analyzes you are
28:36:30aggregating all the data from all years.
28:36:33Now what we can do we can make it more
28:36:35specific for each year where you go and
28:36:37add the year informations to our query.
28:36:40So we can have both a year and months.
28:36:42Let me just change this to a month. And
28:36:45of course we have to go and add it to
28:36:47the group by and the order by. So let's
28:36:50go and execute and with that we are
28:36:51aggregating the data of a month of
28:36:53specific year. So now we have all the
28:36:55months of all years and now if you want
28:36:58to focus on only one year what you can
28:37:00do you can go and filter the data by the
28:37:02order year and with that you can see how
28:37:04the data is evolving over time. Now of
28:37:06course in SQL we can go and format the
28:37:08date differently. So instead of using
28:37:11the year and the month in separate
28:37:13columns what we can do we can use the
28:37:15date trunk function. So instead of here
28:37:17we're going to say date trunk and if you
28:37:19want the granularity of your date at the
28:37:21month level we're going to say month and
28:37:23then the date and with that you will get
28:37:27both the year and the date and let's
28:37:30call it order date like this. So let's
28:37:32go and execute. Now in the output we
28:37:34will get exactly the same result as
28:37:36before but instead of having like two
28:37:38columns for the year and the month we
28:37:40have everything in one and because we
28:37:42saved the month that means it's still
28:37:44going to go and remove all the days. So
28:37:46as you can see it always starts with the
28:37:48one. So the first day of the month and
28:37:50with that you will get one row for each
28:37:53month for each year. And if you want to
28:37:55change that quickly to a year just you
28:37:57go and change the date parts to a year
28:38:00and you will get the granularity of the
28:38:01year. Now if you don't like this format
28:38:03and you would like to have your specific
28:38:05format what you can do you can go and
28:38:08use the format function. So format the
28:38:11first argument is going to be the date
28:38:13and then you go and do your format that
28:38:15you want. So for example it start with
28:38:17the years and let's say I would like to
28:38:19have the abbreviation of the month name.
28:38:22So something like this and of course
28:38:24group by and order by. So let's go and
28:38:27execute it. And with that we got our
28:38:29format the year minus then the
28:38:31abbreviation of the month. But you have
28:38:32to be careful which function you are
28:38:34using because the format you will get in
28:38:36the output a string. And as you can see
28:38:38you cannot sort it correctly. So the
28:38:41data here is sorted by the year but not
28:38:43by the month. But if you are using date
28:38:45trunk you can see the data is correctly
28:38:48sorted. So if we switch it to a month it
28:38:51will be as well. Okay. So everything is
28:38:53sorted correctly because the output here
28:38:55is a date and SQL going to sort the date
28:38:58correctly. It is not string. And if you
28:39:00are using the year and the month the
28:39:02output here going to be an integer and
28:39:05sorting an integer is not a problem. So
28:39:07of course you can go and pick the one
28:39:08that you like. So that's it. Let's go
28:39:11and execute it. And now you can go and
28:39:13keep analyzing by finding another date
28:39:15in our data set and another measure. So
28:39:18as you can see it is very simple. Okay.
28:39:20So that's all about how to analyze the
28:39:22trends and the change over time. Now in
28:39:24the next step we're going to do some
28:39:25kind of advanced aggregations by doing
28:39:27cumulative
28:39:32analyszis. Okay. So what is cumulative
28:39:34analyszis? It is aggregating the data
28:39:37progressively over the time and this is
28:39:39very important technique in order to
28:39:41understand how our business is growing
28:39:44over the time. So how our business is
28:39:46progressing over the time whether it is
28:39:48growing or declining it is very
28:39:50interesting analyszis. So the formula
28:39:52going to be very similar to the changes
28:39:54over time but instead of having a simple
28:39:56aggregations on the measure we're going
28:39:57to aggregate our measure but this time
28:39:59cumulative. So we are like adding stuff
28:40:01on top of each others and the data again
28:40:03can split it by the date dimension cuz
28:40:06we want to track the progress over the
28:40:08time. For example, we can find the
28:40:10running total of sales or the moving
28:40:12average of sales by a month. So now
28:40:15let's have again our simple example
28:40:16where our sales is splitted by the
28:40:18years. Now this is the classic change
28:40:20over time. But in order now to make it
28:40:22cumulative what can happen? We're going
28:40:24to take the measure and add to it. For
28:40:26example, 2024 we have 300. And now for
28:40:292025, we're going to add the 300
28:40:31together with the 100 in order to make
28:40:34it cumulative. So for 2025, we're going
28:40:36to have 400. And the same thing for
28:40:382026, we're going to go and add the 400
28:40:41together with the 200. And with that, we
28:40:43will get 600. So as you can see, we are
28:40:45keep adding the values in order to
28:40:47generate something called cumulative
28:40:49value. Now for this type of analysis, we
28:40:51use in SQL the aggregate window
28:40:53functions. in order to find out the
28:40:55cumulative values. So now let's go and
28:40:57apply our formula in order to find
28:40:59whether our business is growing or
28:41:01declining. So let's go. Okay, so now we
28:41:02have to analyze the following. We're
28:41:04going to calculate the total sales for
28:41:06each month and as well the running total
28:41:08of sales over time in order to analyze
28:41:10the trends. So let's see how we're going
28:41:12to do that. Let's start with the easy
28:41:14stuff where we're going to calculate the
28:41:15total sales for each month. So we are
28:41:17calculating the changes over time and we
28:41:18have already done that. So all what we
28:41:20need is a date and a measure. Our date
28:41:22going to be the order date and the
28:41:24measure going to be the sales amount
28:41:26from our fact
28:41:27table. So let's query this. And now we
28:41:30want to find the total sales for each
28:41:32month. That means we're going to change
28:41:33the granularity of the order date from a
28:41:35day to a month. And I usually like using
28:41:38the date rank for this kind of tasks.
28:41:40And the granularity going to be the
28:41:42month. So this is the order
28:41:44dates. And now for the sales we're going
28:41:47to use aggregate function sum sales as
28:41:49total sales. And of course we have to go
28:41:52and group up the data by the
28:41:54date. So let's go and execute it. So as
28:41:59you can see we have now the total sales
28:42:00for each month. And don't forget to get
28:42:03rid of the nulls. So where we can say
28:42:05where order date is not null. Now it
28:42:09looks better. We don't have nulls. And
28:42:10of course we can go and order the data
28:42:12by our date. Now our measure is just
28:42:15aggregated for each month individually.
28:42:17Right? But we don't want that. We want
28:42:19to have like a running total. So we'd
28:42:22like to have like commumulative metric.
28:42:24In order to do that, we have to use
28:42:26window function. So let's go and do
28:42:27that. We will use a subquery for that.
28:42:30In order just to make it simple. So what
28:42:32we need? We need the order date and
28:42:34let's say the total sales and here we
28:42:36have to have our window
28:42:39function. Then we're going to put the
28:42:41rest in a subquery. And of course we can
28:42:44go and get rid of the order by because
28:42:46anyway our data going to be sorted using
28:42:48the window function. So now let's start
28:42:49writing our window function. We will
28:42:52have the sum of total sales. So we want
28:42:56to summarize those new values. And we're
28:42:58going to build a window function like
28:43:00this over. We don't have to go and
28:43:01partition anything. So we can go
28:43:03immediately and say order by our new
28:43:06order date that we have calculated. And
28:43:08we want it to be ascending. So actually
28:43:10that's it. So as running total sales. So
28:43:15let's try that out. Now if you look to
28:43:16the result you can see that all those
28:43:19values are cumulative and it is working
28:43:21like this. The first total sales is
28:43:23equal to the total sales because
28:43:25previously we don't have anything. Now
28:43:27for the next row what going to happen is
28:43:29going to go and add this value to the
28:43:32previous one. And with that we get the
28:43:34running total value. Now moving on to
28:43:36the third row is going to go and add all
28:43:38those three values together. And of
28:43:40course this going to give us the running
28:43:42total for this month and so on. So as
28:43:44SQL is moving through the window it is
28:43:48always adding the current value to all
28:43:50previous values. And this is because of
28:43:52the default frame of the window. The
28:43:54frame going to be between the unbounded
28:43:57preceding and the current row. So that
28:43:59means for example if we are at this row
28:44:01over here current total sales for this
28:44:03month is this one and the unbounded
28:44:05preceding is all the values before this
28:44:08month. So that means we are getting all
28:44:10the previous values together with the
28:44:12current value and with that we will get
28:44:15the effect of the running total sales.
28:44:17And now of course as you can see it is
28:44:19going through all the years. Right now
28:44:22we can go and limit the running total
28:44:24for only one year. So for each new year
28:44:27it has to reset and start from the
28:44:29scratch. So that means we are
28:44:30partitioning the data. For each year we
28:44:33would like to have partition. For the
28:44:34first year, it's going to be 2010. It is
28:44:36one row. And for the 2011, we're going
28:44:39to get the whole partition over here.
28:44:41So, in order to partition our window,
28:44:43it's very simple. We're going to go and
28:44:44say partition by the order date. That's
28:44:49it. Let's go and execute it. Now, let's
28:44:51go and check for the first partition for
28:44:532010. You can see the running total is
28:44:55the same as the first month. But since
28:44:57we have only one month, that's it for
28:44:59this year. Now, as we go to the next
28:45:01year, as you can see, it resets. So you
28:45:04can see the running total sales for
28:45:062011. It is exactly as January. It is
28:45:09not adding up now the value of the
28:45:11current value with the previous one
28:45:13because the previous one is outside of
28:45:15the window. So as you can see we are
28:45:17getting running total for the whole year
28:45:19and once we hit a new year it is going
28:45:22to reset. So it is working and this is
28:45:24how you can create cumulative values in
28:45:26SQL. And of course if you would like to
28:45:28change the granularity of our data it is
28:45:30very simple. All what you have to do is
28:45:32to go over here and say instead of month
28:45:34we're going to make it as a year. And of
28:45:36course don't forget to change as well
28:45:38the group by. So let's go ahead and
28:45:40execute. And with that we are creating
28:45:42cumulative values for each year. But of
28:45:45course it makes no sense to partition by
28:45:47the years. Let's go and remove it and
28:45:50execute it again. And with that you are
28:45:52creating the running total sales the
28:45:54cumulative metric over the years. So as
28:45:56you can see it is very simple. Now we
28:45:58can go and add like another measure and
28:46:01another aggregation like for example
28:46:03instead of finding the running total we
28:46:05can find the moving average. So let's
28:46:07for example go and get the moving
28:46:09average of the price. So first we have
28:46:12to calculate the average of the price as
28:46:15average price. And now what we have to
28:46:16do is to go and make another window
28:46:19function over here where we are saying
28:46:22average the average price and we're
28:46:25going to go and call it moving
28:46:27average. That's it. So let's go and
28:46:30execute it. And with that you are
28:46:32getting the moving average price of our
28:46:34sales. All right. So now you might still
28:46:36asking what is really different between
28:46:38using a normal aggregation and
28:46:40cumulative aggregation. Well, we usually
28:46:43use normal aggregations in order to
28:46:45check the performance of each individual
28:46:47row. Like if I want to see how each year
28:46:49is performing, I'm going to go and do a
28:46:51normal aggregation. But if you want to
28:46:53see a progression and you want to
28:46:55understand how your business is growing,
28:46:57you have to go and use cumulative
28:46:59aggregations because you can see easily
28:47:01here the progress of your business over
28:47:04the years. So there is like a difference
28:47:06between using cumulative value and
28:47:08normal aggregation. All right. So with
28:47:10that you have done with the cumulative
28:47:12analyszis and you have learned all
28:47:14different types of aggregations. Now the
28:47:16next step in our road map we're going to
28:47:18do performance
28:47:22analyszis. Okay. So what is performance
28:47:24analyszis? It is the process of
28:47:26comparing the current value with a
28:47:29target value to compare the performance
28:47:31of specific category and this can help
28:47:33us in order to measure the success to
28:47:35compare the performance. So the formula
28:47:37for that is very simple. We're going to
28:47:39find the difference between the current
28:47:41measure and the target measure by
28:47:43subtracting them. Like for example, we
28:47:44can go and compare the current sale with
28:47:46the average sale or the current year
28:47:49sales with the previous year sales or
28:47:51the current sales with the lowest sales
28:47:54or maybe the highest sales. So as you
28:47:56can see we are always comparing the
28:47:57current measure together with a target
28:48:00with something else. So for example, we
28:48:01have here again a measure that is
28:48:02splitted by three categories. So those
28:48:04values are the current values. Now if
28:48:07you have a target like for example the
28:48:08average. Now as you can see for each row
28:48:10we have like the 200. Now what we can do
28:48:12once we have those two things in one row
28:48:14we can go and simply subtract them. So
28:48:16for the A the current value is exactly
28:48:19equal to the average. Both of them is
28:48:20200 and the difference between them is
28:48:23zero. So this product is performing as
28:48:25an average. Now for the next one we have
28:48:27300 and the target is 200. So the
28:48:29differences between them is 100. That
28:48:31means this category is performing very
28:48:34well. So this is a good performer. Now
28:48:36for the last one we will get minus 100.
28:48:38So that means it is below the average.
28:48:41So it is not performing very well. And
28:48:43for this type of analysis we usually use
28:48:45window functions like the aggregate
28:48:46window functions, the sum, average, max,
28:48:49min or the value window functions like
28:48:51lead and lag. So now let's go back to
28:48:53SQL and apply this formula in order to
28:48:56measure the performance of our business.
28:48:58So let's go. All right my friends. So
28:48:59now we have the following task. analyze
28:49:01the yearly performance of products by
28:49:04comparing their sales to both the
28:49:06average sales performance of the
28:49:08products and the previous year sales.
28:49:10Okay, this sounds a little bit
28:49:12complicated and serious. Let's have some
28:49:13coffee before we
28:49:15start. Okay, so what do we have over
28:49:18here? So it is talking about the yearly
28:49:20performance of products. So that means
28:49:23we need the order date as a dimension
28:49:25and as well the product and the measure
28:49:27that is used over here is the sales. So
28:49:30let's do it step by step. So we need
28:49:32things from our fact table. So fact
28:49:35sales and we need the product. So I'm
28:49:37going to go and get it from the
28:49:39dimension product in order to have a
28:49:41nice name. So we have to join the data
28:49:44by the product key and I'm going to go
28:49:47and change the alias to P. So product
28:49:51key. Okay. So with that we have our two
28:49:53tables. Now let's go and select our
28:49:54columns. So we need the order date. We
28:49:57need the product name and we need our
28:50:00measure. So it's going to be the sales
28:50:02amount. All right. So now let's go and
28:50:04query those informations. Now we have to
28:50:06analyze the yearly performance. That
28:50:08means we don't need the day. The
28:50:10granularity is the years. So that's why
28:50:12let's go and convert it using year
28:50:15function. And we're going to call it
28:50:17order year. And of course we have to go
28:50:19and aggregate then the sales. And I'm
28:50:21going to call it current sales. And of
28:50:24course we have to group up the data by
28:50:26the date, the year and as well by the
28:50:30product name. So that's it. Let's go and
28:50:34execute it. And of course I'm going to
28:50:35go and get rid of all those nulls. So
28:50:38where order date is not null. All right.
28:50:42So with that we have solved the first
28:50:44part. So we have the yearly performance
28:50:46of the product. Now in the task we have
28:50:49to compare this value the current sales
28:50:51to the average sales performance of the
28:50:54products. So that means we need the
28:50:55average and as well the previous year
28:50:58sales. So that means we have to compare
28:51:00each value to the previous year for the
28:51:02same product of course. So that means
28:51:04things are getting a little bit more
28:51:06complicated and with that we need the
28:51:08help of the window functions. Let's do
28:51:09it one by one. Let's focus on the
28:51:11average sales. So now what we're going
28:51:13to do based on those values based on
28:51:15this results we will do a new
28:51:17calculations and aggregations. And now
28:51:19in order to do that either we use a
28:51:22subquery or a city. I'm going to go with
28:51:24a city because it looks nicer. So with
28:51:27yearly product sales this is the new
28:51:30name that we are giving for this
28:51:32results. And now what we're going to do
28:51:34we're going to build queries on top of
28:51:35these results. So first of all I will
28:51:38just select everything from this table.
28:51:40yearly product sales just to test. So it
28:51:42is working. Now I'm selecting data from
28:51:45our city. So now the next step I'm going
28:51:47to go and list all the columns that I
28:51:49want in my results. So the order date,
28:51:52the product
28:51:53name, the current sales. This is just
28:51:57nicer in order to have control on which
28:51:59columns you want to present at the end
28:52:01results. Now the next step, I'm going to
28:52:02go and order the data by first the
28:52:05product
28:52:06name and then the order year. And with
28:52:09that we can have better understanding of
28:52:11the results. So we can see this product
28:52:13has three years of sales and those are
28:52:15the current sales for each year. So now
28:52:17we have to go and calculate the average
28:52:20of those three sales. So in order to do
28:52:22that we're going to use the
28:52:24average current sales over we have to
28:52:29decide now how to partition the data.
28:52:31Since we are focusing on the products we
28:52:33have to partition the results by the
28:52:35product name. So we're going to say
28:52:36partition
28:52:37by product name and we don't have to
28:52:40sort the data because we are using the
28:52:42average. So it doesn't matter how the
28:52:44data is sorted. So let's call it average
28:52:47sales. So let's go ahead and execute it.
28:52:49And now if you are looking to the
28:52:51results for this product the average
28:52:53sales of all those three values is
28:52:5513,000. So now as you can see for each
28:52:58row we have the current sales and side
28:53:01by side with the average sales and the
28:53:03same thing for the next product as well.
28:53:05So now since we have both of the
28:53:07informations on the same row current
28:53:09sales and the average the change the
28:53:11difference between the current value and
28:53:14the average value. So all what we have
28:53:15to do is to go and subtract right. So
28:53:17we're going to say the current sales
28:53:20subtracted by the average sales and
28:53:23we're going to call
28:53:24it the difference in average. So let's
28:53:28go and execute it. And now as you can
28:53:30see we are getting now the comparison.
28:53:32we have the differences between the
28:53:34current and the average and of course
28:53:36what I like to do is to make a flag or
28:53:38like indicator whether we are above the
28:53:41average below the average or at the
28:53:42average so in order to do that we're
28:53:44going to go and use the case when
28:53:46statement so if the difference is higher
28:53:50than zero then we are above the average
28:53:53right above average oh let's have an
28:53:55abbreviation for that and if we are
28:53:58below zero that means we are below the
28:54:00average right so below
28:54:03then below average and if it is exactly
28:54:06zero else then it is average. So that's
28:54:10it. Let's end it and I'm going to call
28:54:12it average change. So let's go and
28:54:16execute it. Now if you focus again on
28:54:18one of the products you can see the
28:54:20current sales of this product in 2012 it
28:54:23is below the average. It is really low.
28:54:25And for the next year for 2013 it is
28:54:29above the average. It was really nice
28:54:31year for these products and the last
28:54:33year 2014 it was again below the
28:54:35average. So with that we have really
28:54:37nice flag in order to see quickly
28:54:39whether we are above or below the
28:54:41average and it is interesting to see
28:54:43whether we have zeros. So yeah sometimes
28:54:46it is exactly like the average and here
28:54:48we have like a zero. It's not below or
28:54:50above. So with that we are comparing the
28:54:53performance of the sales of each
28:54:55products with the average. And as you
28:54:57can see it is really simple. Yeah. using
28:54:59the window functions. So let's go and
28:55:01check again our task. We have compared
28:55:03the current sales to the average sales
28:55:05performance. Now we have to compare it
28:55:07as well with the previous year sales. So
28:55:10let's go back to our example over here.
28:55:12This time we have to compare the current
28:55:14sales not with the average but with the
28:55:17previous year. So we don't have to write
28:55:19like another CTE or query. We can
28:55:21continue with the same results. So now
28:55:23all what you have to do is to access the
28:55:26previous year. And in order to do that,
28:55:27we have amazing window function called
28:55:30lag. So let's do it step by step. So now
28:55:32we're going to go and create a new
28:55:33column that's called lag. I want to
28:55:36access the previous value of what the
28:55:38current sales, right? So current sales
28:55:42and
28:55:42over we still have to partition the data
28:55:45by the product name because we focus on
28:55:47the products. So partition by product
28:55:50name. But now in order to access the
28:55:53previous value that means we have to
28:55:55sort the data and we're going to sort it
28:55:57by the years. We need the previous year.
28:55:59So we're going to say order by order
28:56:02year and we're going to sort it
28:56:03ascending from the lowest to the
28:56:05highest. So we're going to leave it like
28:56:06this. And with that this window function
28:56:08going to give us the previous year sales
28:56:11of the products. So I'm just going to
28:56:13call it previous year sales like this.
28:56:15And I think here we have something
28:56:17wrong. Okay. So let's go a and execute
28:56:19it and let's go and focus on one of
28:56:21those products. So now for the first
28:56:22year of this product, the previous year
28:56:25was null, right? So we don't have any
28:56:27data from the previous year. But for the
28:56:292013, we have a previous year of 2012.
28:56:32So that's why now we are getting the
28:56:35previous value of the sales based on the
28:56:37years. And the same thing for the last
28:56:39year over here. You can see we are
28:56:41getting the previous sales. So it is
28:56:43working. And for the next window, same
28:56:45thing for the first year. we will get
28:56:47null and the previous sales we will get
28:56:49it from the previous year. So with that
28:56:51we have now the previous sales and if
28:56:53you check this over here we have in the
28:56:56same row now the current sales of the
28:56:58current year and as well the sales of
28:57:01the previous year. Now what we have to
28:57:03do the same thing we have to go and
28:57:05subtract those two informations in order
28:57:07to compare them. Right? So we're going
28:57:09to go and do the same thing. So we will
28:57:11get the current sales minus the whole
28:57:14thing the whole window function and
28:57:16we're going to call it previous year. So
28:57:19difference of the previous year and with
28:57:21that we are calculating the differences
28:57:23between them. So for this year for this
28:57:25product as you can see the difference
28:57:27here is really big between the current
28:57:29sales and the previous year. Now of
28:57:31course what we can do we can go and make
28:57:32as well a flag or an indicator. I'm
28:57:35going to go and copy the whole thing
28:57:37from the previous average but we have to
28:57:39go and get the right function this and
28:57:42the same over here and now it is not
28:57:44above or below the average I'm going to
28:57:46say it is increasing or decreasing right
28:57:49so increase or decrease and we're going
28:57:52to call it previous year change and
28:57:54instead of average we can say no change
28:57:57so let's go and execute it and I'm
28:58:00having here an extra comma let's go and
28:58:02execute it so again let's go and focus
28:58:05of one of those products. For the first
28:58:07year of this product, there is no change
28:58:09because there is no previous year. For
28:58:11the next year of this product, we have
28:58:13an increase, right? Because the current
28:58:15sales is way higher than the previous
28:58:18year. And now by going to the last year
28:58:19of this product, we have a decrease
28:58:21because the current sales is less than
28:58:24the previous year. So my friends, we
28:58:26call this type of analyszis year over
28:58:30year analyszis. And if you want to
28:58:32calculate the month over month analyzes,
28:58:34it's very simple. All what you have to
28:58:36do is to go and change the function from
28:58:38year to a month and with that you are
28:58:40extracting the month part. And the
28:58:42difference between analyzing the months
28:58:44and years is of course the scope.
28:58:46Year-over-year is good for long-term
28:58:49trends analyzes where on the other hand
28:58:51the month over month it is shortterm
28:58:53trends analyzes. You are just focusing
28:58:56on the seasonality of your data. So this
28:58:58is how we analyze the performance of our
28:59:01business by comparing the current
28:59:03measure with a target measure and you
28:59:05can go and use different dimensions and
28:59:07stuff. So instead of the sales you can
28:59:09check the quantity instead of products
28:59:11you can check the customers and you can
28:59:13go and compare the current information
28:59:15not only with the average or the
28:59:17previous year you can compare it with
28:59:19the lowest sales and the highest sales
28:59:21and it can open the door for many
28:59:23different insights. But we are always
28:59:25using the same methods using the window
28:59:28functions. We compare the current value
28:59:30with another value in our data sets. So
28:59:33this is how we do performance
28:59:34comparison. All right. So that you have
28:59:36learned how to analyze the performance
28:59:38of our business. Now in the next step
28:59:40we're going to do partto-hole analyszis.
28:59:43So let's
28:59:46go. Okay. So now what is exactly part to
28:59:49whole analyszis? Well, we use it in
28:59:51order to find out the proportion of a
28:59:53part relative to the whole. Well, here
28:59:55we're going to analyze how an individual
28:59:57category is contributing to the overall
29:00:00in order to understand what is the most
29:00:03impacting category to the overall
29:00:05business. So now for the formula, it is
29:00:07very simple. You have to go and pick one
29:00:09of your measures divided by the total of
29:00:12the measure and then multiply it by 100
29:00:14in order to find the percentage by a
29:00:16specific dimension. Like for example, if
29:00:18you take the sales, so you divide the
29:00:20sales by the total sales, multiplied by
29:00:23100 by the category or if you take the
29:00:25quantity divided by the total quantity
29:00:28and then find the percentage by a
29:00:30country. So for example, again we have
29:00:31our measure splitted by categories. But
29:00:33now instead of having this number, what
29:00:35we're going to do, we're going to
29:00:36calculate the percentage. So for the
29:00:38first one, we're going to take the 200
29:00:40divided by 600 multiply it by 100. So
29:00:43we're going to get the percentage 33. So
29:00:45once we do that for the all categories,
29:00:47it's going to be now very easy to see
29:00:49that the category P it is contributing
29:00:51to the overall number by 50%. Which
29:00:54makes it of course a top performer. So
29:00:56you can visual in your head as like a
29:00:58pie chart and you can see how each part
29:01:01is contributing to the whole pie chart
29:01:03and with that it can help us to
29:01:04understand the importance of each
29:01:06category to our business. So now let's
29:01:08go and apply this formula to our
29:01:10measures in order to understand the
29:01:12importance of our categories. So let's
29:01:14go. Okay. So now let's do part hole
29:01:16analyszis. All what we need one
29:01:17dimension and one measure. So for
29:01:19example we have the following task. It
29:01:21is very simple. Which categories
29:01:22contribute the most to the overall
29:01:25sales. So now let's go and do it step by
29:01:27step. So first we're going to go and
29:01:29collect the informations. So we need the
29:01:31category. We need the sales amount and
29:01:34those informations come as usual from
29:01:37the fact sales and from our dimension
29:01:41the product. Right? So we have quickly
29:01:44to go and connect them using the product
29:01:47key. Okay. So that's all what we need
29:01:50for our query. So let's go and select.
29:01:52So we have here the categories and the
29:01:54sales amount. So now the first thing we
29:01:56have to calculate the total sales for
29:01:58each category. So let's go and do that.
29:02:00It is very simple. So sum total sales
29:02:04and we are grouping up the data by the
29:02:07category. So this is basics. Right now
29:02:09we have the total sales for each of
29:02:11those categories. Now in order to
29:02:13calculate the percentage we need two
29:02:14measures the total sales for each
29:02:16category and we have it here already and
29:02:19as well side by side we need the total
29:02:21sales across all categories. So the big
29:02:24number without any dimension but now as
29:02:26you look to the result you can see the
29:02:28granularity here is that category. Now
29:02:31we need the total sales again by
29:02:33different granularity. And in order to
29:02:35mix those stuff together we use the
29:02:37window functions. So now how we going to
29:02:39do it? either you go over here and start
29:02:41writing your window function. And of
29:02:42course, you can do it together with the
29:02:44group by or you can do it as a second
29:02:46step in your query using either a CTE or
29:02:49a subquery. So I'm going to go with the
29:02:51CTE just to make it clear. So category
29:02:55sales like this. So now let's start
29:02:58again selecting the same information. So
29:03:00category total sales from our table
29:03:03category or CTE sales. So let's go and
29:03:07execute it. So now we have the same
29:03:09results and now we're going to go and
29:03:10build our window function like this. So
29:03:12we're going to say the sum we want to
29:03:14aggregate all those values right to get
29:03:16the total sales over the whole data
29:03:19sets. So we're going to say sum total
29:03:21sales. And now in order to get the big
29:03:24number we're going to say over and
29:03:26inside it we will not define anything
29:03:28because we don't want to partition the
29:03:30data. We don't want to introduce any
29:03:32dimension. We just want the big number.
29:03:34And with that we will get the overall
29:03:38sales. So let's go and execute it. Now
29:03:41as you can see this is the total sales
29:03:43by the category. So the total sales is
29:03:46splitted by the categories. And this is
29:03:48the overall sales of all orders of
29:03:51everything the highest number. Now since
29:03:53we have them side by side what we can do
29:03:55we can very easily calculate the path to
29:03:58whole or the percentage. So let's start
29:04:01doing that. We need the total sales and
29:04:03we want to go and divide it by the
29:04:05overall sales. So we're going to take
29:04:07our window function and put it over
29:04:08here. So let's go and multiply it now
29:04:11with 100. I'm going to go and call it
29:04:14percentage of total. So let's go and
29:04:18execute it. Now as you can see we are
29:04:19getting zeros and that's because the
29:04:21total sales is not float. So what we
29:04:24have to do is to go and cast it to
29:04:25something like a decimal. So floats like
29:04:29this. So let's go and reexecute it. And
29:04:31now, as you can see, we are getting now
29:04:32the percentages, but we have a lot of
29:04:35numbers after the comma. So, we're going
29:04:37to go and round the numbers now. So,
29:04:39let's go to the start round and then go
29:04:42to the end, comma, and let's have like
29:04:44two decimals. So, let's go and execute
29:04:47it again. Now, looks perfect. Now, what
29:04:49we can do, we can go and add like a
29:04:51percentage. And with that, we are
29:04:52converting the whole thing to a string.
29:04:54So, we're going to do concatenation. So,
29:04:56concat at the start and go to the end.
29:05:00And let's add the percentage character.
29:05:02And as well we can go and order the data
29:05:05by the total sales descending. So let's
29:05:09go and execute it. So now by looking to
29:05:11the result you can see the category
29:05:13bikes is dominating. So it is
29:05:15overwhelming top performing the
29:05:17categories. It is making 69% of the
29:05:21total sales of our business. So this
29:05:23means my friends most of the business
29:05:25revenue comes from the bikes. And as you
29:05:28can see the accessories and clothing
29:05:30they are really minor contributors to
29:05:32our business which is not really good
29:05:35and this is actually dangerous thing. If
29:05:37you have like one category dominating
29:05:39your whole business you are over relying
29:05:41on only one category in your business
29:05:43and if this fails this category then the
29:05:46whole business is going to fail. So by
29:05:48looking to this either the business has
29:05:49to decide removing all those products by
29:05:52those two categories or to focus more on
29:05:56bringing more revenue for the products
29:05:58that are inside those two categories. So
29:06:00as you can see guys those insights are
29:06:03really amazing for the business and
29:06:04helps the managers and the decision
29:06:06makers to understand what is going on
29:06:08quickly and make very critical
29:06:11decisions. And now you can see as well
29:06:12from the results perfectly why the part
29:06:15to whole analyszis is very important
29:06:17because by just looking to those numbers
29:06:19it's going to be really hard to
29:06:20understand the importance of the
29:06:23categories. But seeing the data as a
29:06:25percentage how each category is
29:06:27contributing to the whole sales of the
29:06:29business makes it easier to understand
29:06:32which category is underperforming or top
29:06:35performing. And now you have a very
29:06:36simple formula where you can go and
29:06:38change the metrics. For example, instead
29:06:40of total sales, you can go and change
29:06:42the aggregations to total number of
29:06:44orders or the total number of customers.
29:06:47So you can go and bring any type of
29:06:49measures and bring it to this analyszis
29:06:51and you're going to generate completely
29:06:53new view for the decision makers in
29:06:56order to develop a new strategy for the
29:06:58business. It was very interesting. Now
29:06:59in the next step, we're going to do my
29:07:01favorite topic where we're going to
29:07:03start doing data segmentations using
29:07:04SQL. So let's go.
29:07:10Okay. So now what is data segmentations?
29:07:12What we're going to do here is we're
29:07:13going to go and group up the data based
29:07:15on specific range. So that means we're
29:07:17going to go and create a new categories
29:07:19and then go and aggregate the data based
29:07:21on the new category. And the formula for
29:07:23that going to be very interesting. So
29:07:25it's going to be this time we're going
29:07:26to have a measure by a measure not by
29:07:29dimension. So you have to go and pick
29:07:31two different measures and convert one
29:07:33of those measures to a range or to a
29:07:35group and then aggregate the data by
29:07:36this measure. So for example, we're
29:07:38going to go and calculate the total
29:07:40number of products by the sales range or
29:07:43the total number of customers by the age
29:07:45group. So as you can see we have two
29:07:46measures and we are trying to combine
29:07:48them together in order to create new
29:07:50insights. Let's have the following
29:07:51example. So here for example we have
29:07:53like two measures and now the first step
29:07:55is that we're going to take one of those
29:07:56measures and convert it to a dimension.
29:07:59converted to a category. For example,
29:08:01we're going to say if the values are
29:08:03like equal or below 100, it will be
29:08:05converted to a category called low. And
29:08:08between 100 and 200, it's going to be
29:08:10assigned to a new category called
29:08:12medium. And everything above 200, it's
29:08:15going to be large. So, as you can see
29:08:16what we are doing, we are taking one
29:08:18measure and based on the range of this
29:08:20measure, we are building a new
29:08:22categories, new dimension. And now the
29:08:24final step is the easiest one. We're
29:08:26going to go and aggregate another
29:08:27measure based on the new category. So
29:08:29we're going to have seven for low, six
29:08:31for medium, and 15 for large. So with
29:08:33that, as you can see, we are creating
29:08:35new categories or segments based on a
29:08:37measure. And then we are aggregating
29:08:39another measure based of this new
29:08:42segments. And in SQL, in order to create
29:08:44those new categories and segments, we
29:08:46use the amazing case when statements
29:08:49because it's going to help us to define
29:08:50the rules and based on the range, it's
29:08:52going to go and create a new category
29:08:54and labels. So now let's go and apply
29:08:56this formula on our data set in order to
29:08:59segment our data. So let's go. Okay. So
29:09:01now let's go and segment our data and
29:09:03all what we need is two measures. So now
29:09:05we have the following task and it says
29:09:07segment products into cost ranges and
29:09:10count how many products fall into each
29:09:12segment. So now by looking to this task
29:09:14we have two measures. First the costs
29:09:16and as well the second one is the total
29:09:19number of products. And of course we
29:09:21have to go and segment one of those two
29:09:22measures. And in this task we are
29:09:24segmenting the costs. So we have to
29:09:27focus now on taking this measure and
29:09:29convert it to a dimension. So now all
29:09:31those informations are available in the
29:09:33table products. So now let's go and
29:09:35select few columns. We're going to get
29:09:36the product key and let's get the
29:09:40product name and the costs. That's all
29:09:43what we need. So let's execute it. Now
29:09:45as you can see this is our measure the
29:09:47costs. Now we have to go and convert
29:09:49this measure to dimension. And in order
29:09:52to do that, we use the case win
29:09:54statements. We always use the case win
29:09:56statement in order to create new
29:09:57categories. So let's go and do that.
29:10:00Case win. Let's start with the first
29:10:01range. Let's say it is below 100. So all
29:10:04the costs that are below 100. We're
29:10:07going to label it with a new value. It's
29:10:09going to be below 100. So now let's go
29:10:12to the next range. We are saying when
29:10:14costs now between 100 and 500. So all
29:10:19costs between this range. They will get
29:10:21the label 100 and 500. So this is very
29:10:25simple. Let's go and get another range.
29:10:27For example, between 500 and 1,000. Then
29:10:33it's going to get a label between 500
29:10:35and 1,000. And now it depend how many
29:10:37categories and segments you want to
29:10:39create. Each row of this case when each
29:10:42condition will be creating like a new
29:10:44value for your dimension. So I'm going
29:10:46to stop with that. I'm going to say at
29:10:47the end else. So if the cost is not
29:10:49fulfilling any of those, it's going to
29:10:51be above 1,000. Right? So that's it.
29:10:55Let's give it a name. It's going to be
29:10:57cost range. So now let's go and execute
29:10:59it. Now let's go and check the result.
29:11:02For example, the cost here is zero. It
29:11:03is below 100, which is correct. This
29:11:06value is above 1,000. This is between
29:11:08500 and 1,000. And this is between 100
29:11:11and 500. So everything looks correct.
29:11:14Nice. So with that we are done with the
29:11:16first step where we have converted one
29:11:18measure into a dimension. So with that
29:11:20we have now our segments. The next step
29:11:23with that we're going to go and
29:11:25aggregate the data based on this a new
29:11:27dimension. So either you do it in one go
29:11:29or what I usually do I put everything in
29:11:31one city or a subquery and I'm going to
29:11:34call it products
29:11:36segments as based on this results I'm
29:11:40going to go and aggregate the data. So
29:11:41this is my temporary results and now
29:11:44we're going to go and just aggregate the
29:11:45data like this. So let's get first our
29:11:47dimension cost range and then we need
29:11:50our measure. So it's going to be count
29:11:52product
29:11:53key as total products from our city. It
29:11:58was the product segments and then group
29:12:01by our new dimension. That's it. It's
29:12:04very simple. Let's go and execute it
29:12:06now. Now you can see in the output we
29:12:08have our segmented measure and we can
29:12:10see the total numbers in each of those
29:12:13segment and range and of course we can
29:12:14go and order the data by our aggregation
29:12:17the total products. Let's go and execute
29:12:19it maybe descending. So now as you can
29:12:21see we have a lot of products that are
29:12:24not costing a lot. It is below 100.
29:12:26After that between 100 500 and the
29:12:29lowest number of products is in the
29:12:32range that is above 1,000. So we don't
29:12:34have a lot of products that are costing
29:12:36a lot and that's because maybe we have a
29:12:38lot of accessories in the business. So
29:12:40my friends this is very powerful. If
29:12:42your dimensions in the data set is not
29:12:45enough to create insights you can take
29:12:47one of your measures convert it to a
29:12:49dimension using case win and then
29:12:51aggregate your other measures based on
29:12:54this new dimension. So we are deriving
29:12:56new informations and as I told you by
29:12:58just following this concept measures and
29:13:00dimensions you can generate endless
29:13:03amount of reports even if your business
29:13:05or your data set is small. Okay my
29:13:08friends so now let's go and segment
29:13:09something else. So this time it's going
29:13:11to be a little bit more complicated. So
29:13:13we have the following task and it says
29:13:15group customers into three segments
29:13:17based on their spending behavior. So we
29:13:19have the VIB customers. They are the
29:13:21customers with at least 12 months of
29:13:24history and spending more than 5,000.
29:13:27And the second category we have the
29:13:29regular customers. They have at least as
29:13:31well 12 months of history but they spend
29:13:34like less than 5,000. And the last
29:13:36category we have the new customers.
29:13:38Their lifespan is less than 12 months.
29:13:41And we have to find the total number of
29:13:42customers by each group. So now here we
29:13:44have a lot of measures and stuff. So the
29:13:46first one is the total number of
29:13:48customers. This is going to be the final
29:13:50aggregation that we're going to do. But
29:13:52what is interesting, we're going to
29:13:53build the segments and this time is
29:13:55based on different columns. So first it
29:13:58is based on a measure the total number
29:14:00of months for each customer and as well
29:14:03the total spending, the total number of
29:14:05sales. So we have the sales, we have the
29:14:08total number of months and as well the
29:14:10total number of customers. So now we're
29:14:11going to do it step by step. Don't you
29:14:13worry about it. So now what I usually
29:14:14do, I start collecting all the data that
29:14:16I need. So what do we need? We need a
29:14:19customer key. In order to do the
29:14:21aggregation for the total number of
29:14:22customers, we need as well the sales
29:14:24amount right for the spending. And now
29:14:27in order to calculate those number of
29:14:29months, we need a date. And for that, we
29:14:32have to calculate the lifespan of a
29:14:34customer. And usually we create it using
29:14:36the order date. I'm going to show you
29:14:38how we're going to do it. So we need the
29:14:39order date. And of course, we have to
29:14:41select our table. So let's start with
29:14:43the fact table. So fact sales and we're
29:14:46going to join it with the
29:14:48customers. So our dimension customers
29:14:51and the key for that it is the customer
29:14:54key as well for the customers. And here
29:14:56we have to specify which column come
29:14:58from which table. So the first one from
29:15:00the customers, the sales from the fact
29:15:02and the order date from the fact as
29:15:04well. So now let's go and execute. Now
29:15:07we can see we have our customers, the
29:15:09sales and the order dates. So now the
29:15:11sales going to help us in order to
29:15:12specify the range of spending. But now
29:15:15what is interesting we have to calculate
29:15:17the lifespan. So now in order to get the
29:15:19lifespan we have to find out the first
29:15:21order and the last order of each
29:15:24customer. So how many months is between
29:15:26the first order and the last order. So
29:15:28in order to do that we need the min
29:15:31function for the order dates. So this is
29:15:34the first order and the max in order to
29:15:37get the last order. Right.
29:15:41And since we are using min and max, we
29:15:43have to go and group up the data. And we
29:15:45need to do that anyway in order to get
29:15:47the total spending. So for the sales
29:15:50amount, we're going to have the sum in
29:15:52order to have the total spend total
29:15:56spending. And we don't need the order
29:15:58age. And the dimension where we're going
29:15:59to group up the data is by the customer
29:16:02key. So let's go and execute it. So now
29:16:04in the results we have a list of all our
29:16:06customers and as well the total spending
29:16:09for each customer and we have the first
29:16:11order date and the last order dates. Now
29:16:13in order to calculate how many months
29:16:16between the first order and the last
29:16:18order we can go and use the function
29:16:20date diff in order to get a new measure.
29:16:23So let's go and do that date diff. And
29:16:25now since we need the number of months
29:16:27we're going to use the month and then
29:16:29the second argument going to be the
29:16:31first order. So order date and the
29:16:35second one going to be the latest. So
29:16:36max order date and we're going to call
29:16:39this lifpan. So let's go and query and
29:16:43let's have a look to our results. You
29:16:45can see for this customer 712 between
29:16:48the first order and the last order we
29:16:50have 11 muscles and for this customer
29:16:53over here we have zero because the first
29:16:55order and the last order is in the same
29:16:57month and maybe there is only one order.
29:16:59So with that we have the lifespan and as
29:17:01you can see guys we have derived a new
29:17:04measure from the dimension order age in
29:17:07order later to derive from this new
29:17:09measure a new dimension the segments. So
29:17:12we are converting a dimension to a
29:17:14measure and then from a measure to a new
29:17:16dimension and this is usually what we do
29:17:18in analyzes and in SQL. So now do we
29:17:21have all the informations for the logic?
29:17:23So we have the lifespan. So we have the
29:17:25total number of monsters, we have the
29:17:27total spending and I think we are ready
29:17:29to start building our segments. So now
29:17:31what we're going to do, we're going to
29:17:32create the segments based on these
29:17:34results that we have prepared. So this
29:17:36result is the intermediate result before
29:17:38the final one. Now either you're going
29:17:40to put it in a CTE or subquery. Well, I
29:17:43usually go and use the CTE. It is nicer.
29:17:47So with customer
29:17:49spending and I'm going to put the whole
29:17:52thing in ECT and we can start writing a
29:17:54new query from the scratch based on the
29:17:56inter results. So let's go and select
29:17:58again the customer key. I'm going to get
29:18:01the total spending and the lifpan. So we
29:18:05don't actually need the first and the
29:18:07last order and we're going to get all
29:18:09those informations from our new city. So
29:18:12let's go and execute. And now let's
29:18:14start building the segments. And as
29:18:16usual, we're going to go and use the
29:18:17case win statements. It is just amazing
29:18:20statements in order to derive and build
29:18:23new columns. So now what do we have for
29:18:25the first category? So they are the
29:18:27customers over 12 months and spending
29:18:29more than 5,000. So now we're going to
29:18:31say if the laugh span is higher than 12
29:18:35and the total spending is higher than
29:18:395,000 then we have our VIB customers. So
29:18:44this is the first label. Let's go to the
29:18:47second one. If the lifespan as well I
29:18:49think more than 12. So let's go and
29:18:51check. Well, it is at least 12. I have
29:18:53here mistake. So it's going to be larger
29:18:56or equal. So now it is more correct. So
29:18:58the customers that has at least 12
29:19:00months but they spend like 5,000 or
29:19:02less. So that means it's going to stay
29:19:04the same condition but the total
29:19:06spending will be less or equal 5,000s
29:19:11and they are the regular customers. So
29:19:14they will get this label. Now if it is
29:19:17not fulfilling those two conditions what
29:19:19this means this means this is a new
29:19:22customer right. So they will get this
29:19:24label. Let's go and have an end and
29:19:26let's call it customer segments. So
29:19:30let's go and execute it. Now let's have
29:19:32a look for this customer 712. So the
29:19:35total spending is less than 5,000. So
29:19:38this customer is not a VIB and as well
29:19:41the lifespan is less than 12. So that
29:19:44means for us it is a new customer. Now
29:19:46the next one we have a VIB. So this
29:19:49customer has a history at least 12
29:19:51months. So we have here 16 months and as
29:19:53well the total spending more than 5,000.
29:19:56That's why this customer is a VIB. But
29:19:58now let's go and search for a regular
29:20:01customer
29:20:032349. So this customer spent less than
29:20:065,000. So we are fulfilling this
29:20:08condition over here and as well this
29:20:10customer has at least 12 months of
29:20:13history that's why we have a regular. So
29:20:15now as you can see we have derived a new
29:20:18dimension from two measures the lifespan
29:20:21and the total spending. Now of course
29:20:23the last step what is going to be we
29:20:25have to go and find the total number of
29:20:27customers for each of those categories.
29:20:29So now what we're going to do we're
29:20:30going to remove all those stuff and
29:20:32we're going to start with our new
29:20:33dimension and then comes the aggregation
29:20:36count customer key. So as total
29:20:40customers and then we have to group up
29:20:42the data by our new dimension. So this
29:20:46going to be really annoying if I'm going
29:20:47to take this here and put it in the
29:20:49group I because this means each time I'm
29:20:52changing the logic I have to take care
29:20:54of that twice. One in the select
29:20:56statement and the second one in the
29:20:58group I. So now actually instead of that
29:21:00what I'm going to do I changed my mind.
29:21:02I'm going to still having the
29:21:04aggregation in the second step. So we
29:21:06need the customer key we have the
29:21:08definition of our customer segments. And
29:21:10now I'm going to go and use the subquery
29:21:13where I put the aggregation as a second
29:21:15step. So my friends that means this is
29:21:18again a second intermediate results. You
29:21:21can of course put it in a second city.
29:21:23So that means this is the first
29:21:25intermediate results where we have
29:21:27created the lifespan and the total
29:21:28spending and the second intermediate
29:21:30result is creating the customer segments
29:21:33and the third step and the last one is
29:21:35by doing the final aggregation. So we're
29:21:37going to do it like this. Select our
29:21:40dimension customer segments. Then we're
29:21:43going to go and count the customer key
29:21:46from our sub query. So this is our
29:21:49subquery and don't forget to group by
29:21:52our dimension customer segments. I think
29:21:54I have it wrong. All right. So this is
29:21:56the subquery and this is the final step
29:21:58where we are aggregating everything. I'm
29:22:00going to go and order the data by the
29:22:03total customers like this. So now let's
29:22:07go and execute the whole thing. Well
29:22:10descending not ascending. Okay. Okay. So
29:22:12now we can see from our results the
29:22:14highest number of our customers belong
29:22:17to the category new. So we have 14,000
29:22:20customers that are new in our business.
29:22:22And then the second category we have the
29:22:24regular customers. So we have around
29:22:262,000 customers. And in VIB we have a
29:22:29lot of VIB customers. So we have
29:22:321,655 VIB customers in our business. So
29:22:34with that my friends, we have done data
29:22:36segmentation. It is amazing. We have
29:22:38segmented our customers based on their
29:22:41spending behavior and as you can see all
29:22:43those informations are totally derived
29:22:46from the our data and this help us to
29:22:48have a deep understanding of the
29:22:50behavior of our customers and of course
29:22:52this can help as well making smart
29:22:58decisions. All right my friends so with
29:23:00that we have covered the five different
29:23:02types of data analytics thus we can do
29:23:05using SQL. Now what I usually do as the
29:23:07last tip in my project is that I try to
29:23:09collect all the different types of
29:23:11explorations and analyzes that I have
29:23:13done in my data sets so that I can put
29:23:15everything in one for example view or
29:23:17table and then offer it to other users
29:23:20and with that it going to help the other
29:23:21users or stakeholders to make a quick
29:23:24analyszis for decision- making. So now
29:23:26what we're going to do, we're going to
29:23:26have like some kind of requirements
29:23:28where we're going to bring a lot of
29:23:29different analyzes in one big script in
29:23:32order to have insights about one object
29:23:34like for example the customers. So I'm
29:23:36going to show you the requirement of
29:23:38this reports and we're going to analyze
29:23:39it and start writing the scripts. So
29:23:41let's go. Okay friends. So now let's
29:23:43create a customer report and here are
29:23:45the requirements for the report. So now
29:23:48we have like a general statement. It
29:23:49says this report should consolidate key
29:23:52customer metrics and behaviors. So it
29:23:54says first we have to gather all the
29:23:56details about the customers like names,
29:23:59age, transaction details and then we
29:24:01have to segment the customers into
29:24:03categories VIB, regular and new and as
29:24:05well by the age groups and we have to
29:24:07provide as well aggregations like the
29:24:10total order, total sales, quantity,
29:24:12products and so on. And we have to
29:24:14generate important KPIs like the
29:24:16recency, the average order value, the
29:24:19average monthly spends. So we have a lot
29:24:22of things and we're going to do it step
29:24:23by step. All right. Now I'm going to
29:24:24take you step by step in the process of
29:24:26building a complex query that I usually
29:24:29use in order to build a report. Now the
29:24:31first thing that I usually do is I start
29:24:33selecting the data from the database and
29:24:36I usually start with the fact table. So
29:24:38this is my starting point and then
29:24:39usually I join it with the dimensions
29:24:41and here I use lift join and after that
29:24:44I think about how to filter the data
29:24:46because usually we don't need all the
29:24:48data that is available in the database
29:24:50and of course in the result I will not
29:24:52be selecting all the columns. I'm going
29:24:54to be selecting only the relevant
29:24:55columns that I need for my reports. So
29:24:58since we have like complex query we will
29:25:00be dividing the process into multiple
29:25:02steps and I usually call this step the
29:25:04base data and this going to be the
29:25:06foundation the scope for the next steps
29:25:08and since we have like multiple steps
29:25:10I'm going to put this in a CTE so we
29:25:12have this as an intermediate results and
29:25:14what we're going to do in this step as
29:25:16well we're going to do few
29:25:17transformations like maybe calculating
29:25:19and deriving new columns maybe
29:25:21formatting the date so some basic
29:25:23transformations so now let's go and
29:25:24build this results for our report so the
29:25:27first step is retrieving the core
29:25:29columns from the tables. So let's go and
29:25:31do it together. So we need of course our
29:25:33fact table facts and we need our
29:25:38dimension gold customer and as usual
29:25:40we're going to go and connect them. All
29:25:42right. Okay. So this is the basic and
29:25:45now what we're going to do we're going
29:25:46to go and retrieve all the columns that
29:25:48we need for our reports. So let's start
29:25:50picking stuff. So order number let's get
29:25:53the product key the order date sales
29:25:58amount quantity and I think that's all
29:26:01from the facts let's go and get few
29:26:03informations from the customers so let's
29:26:06get the customer key the customer number
29:26:10the first name and as well the last name
29:26:13and what else we can go and get the
29:26:15birth dates because we have to create
29:26:16the age groups so birth dates let's go
29:26:20and query. So I think those are all the
29:26:23columns that we need in order to do the
29:26:25next steps. And now before we go and
29:26:27proceed with the aggregations, what
29:26:29we're going to do, we're going to think
29:26:30about filtering the data. As I recall,
29:26:32we have some orders where the order date
29:26:34is null. So I'm going to go and remove
29:26:37those stuff. So order date is not null.
29:26:40So that means in the first query the
29:26:42base query not only I'm selecting the
29:26:44columns that I need for the reports also
29:26:47I'm defining the scope of the data sets
29:26:49by filtering the data. So you can as
29:26:51well make the scope here only one year
29:26:53or something. Now what else we can do is
29:26:54to think about all those columns and
29:26:56whether we can do any type of
29:26:58transformations in order to prepare them
29:27:00for the aggregations. Like for example
29:27:02I'm going to go and say you know what
29:27:03instead of first and last name I'm going
29:27:05to put them together in one. So it's
29:27:07going to be the customer name. It's
29:27:08better than having like two columns. So,
29:27:10let's go and do it. We're going to say
29:27:12concat and then we're going to start
29:27:14with the first name and we're going to
29:27:16have a separator between them. You can
29:27:18have like a minus or a white space like
29:27:21this and after that the last name. So,
29:27:24let's call it customer name. And we can
29:27:27go and get rid of those two columns. So,
29:27:29let's go and execute. And with that, you
29:27:31have everything in one column. Now,
29:27:32another thing that we can prepare that
29:27:35we don't need the birth date. We
29:27:37actually need for our reports the age
29:27:39groups. So that means we have to go and
29:27:41calculate the age. So let's go and
29:27:43transform it. So date diff we want it in
29:27:46years, the birth date and the current
29:27:49date from system and we're going to call
29:27:50it age. So let's execute again. Perfect.
29:27:54So with that we have all the data that
29:27:56we need for our reports. Let's go and
29:27:58put everything in one city. So I'm going
29:28:00to call it with query as and put
29:28:03everything in this city. And I'm going
29:28:05to go and put this comment over here
29:28:08inside the city. Perfect. And now we're
29:28:10going to go and write a query from the
29:28:13scratch. Paste on our intermediate
29:28:15results. So base is query. It's execute.
29:28:18All right. So now by looking to our
29:28:20report with that we have the important
29:28:22columns. Right. So now in the next step
29:28:24we're going to do aggregations on top of
29:28:26these intermediate results. So here
29:28:28we're going to do all the aggregations
29:28:29that is needed for the report and we're
29:28:31going to put everything again in CTE as
29:28:34an intermediate results which makes
29:28:36everything a modular and easy to read.
29:28:38So now let's go and do the necessary
29:28:40aggregations on the result that we have
29:28:42previously prepared. So that's why this
29:28:44is very important as a second step in
29:28:47our report. Always tend to make a
29:28:49separated CTE only for aggregations. So
29:28:52let's go and do that. I'm going to go
29:28:54and select again all the customer
29:28:56informations like the customer key
29:28:58number, age. So I'm just going to copy
29:29:00and paste and put it over here. And we
29:29:03just need the column
29:29:05names. So the key number, name, and age.
29:29:09Now after that, we're going to start
29:29:10doing aggregations. So what do you want
29:29:12to aggregate is first, for example, the
29:29:14total number of orders. So we're going
29:29:16to go and count distinct order
29:29:19number as total orders. So this is one
29:29:24aggregation. We can go and summarize all
29:29:26those sales
29:29:27amounts as
29:29:30total sales and the quantities as well.
29:29:33So sum quantity as total quantity and as
29:29:38well we can go and count how many
29:29:40products did our customer order. So the
29:29:43products key as total
29:29:47products. So what I'm doing now I'm just
29:29:50looking to our intermediate results and
29:29:51try to figure out what we can aggregate
29:29:54for example it makes no sense to
29:29:55aggregate for example the ages right so
29:29:58from the order number we have total
29:29:59orders total product sales amount
29:30:01quantity and from the right side we
29:30:03cannot aggregate anything and that's
29:30:05because they are the details of the
29:30:06customers but from the fact table we can
29:30:09do a lot of aggregations so now what we
29:30:10can do with the order date over here we
29:30:12can for example find the last order
29:30:15dates from our customer which is really
29:30:17nice information. So we can say max
29:30:20order date as last order and of course
29:30:23we can go and calculate the lifespan and
29:30:26that we're going to need it as you
29:30:27remember in order to categorize our
29:30:30customer. So I will just copy and paste
29:30:32it from the previous query is the date
29:30:34diff month between the first order from
29:30:37the customer and the last order of the
29:30:39customer. So and we call this lifespan.
29:30:42Okay. So we derived two measures or
29:30:44aggregations from the order date. Now I
29:30:47think we have done everything possible
29:30:49and what is missing of course is to have
29:30:51a group by because we are doing
29:30:53aggregations and we are grouping by the
29:30:56customer details. So going to be
29:30:58customer key, customer number, name and
29:31:01age. So I think we have everything for
29:31:03our aggregations. Let's go and execute
29:31:06it. A list of all customers and we have
29:31:09few details about the customers and now
29:31:11we have a lot of measures. So the total
29:31:13order, total sales, total quantity,
29:31:15products, the last order and the
29:31:17lifespan. And with that we have covered
29:31:20this part over here where we have
29:31:22provided aggregations on the customer
29:31:24level. So we have the details and we
29:31:27have the aggregations. All right. So
29:31:28with that we have now all the
29:31:30preparations that is required to build
29:31:32the final results. So it really depend
29:31:35on the scenario. If it's possible we can
29:31:37take all the data from one city or if
29:31:39it's needed we can get it from multiple
29:31:41cities. But in our scenario, we're going
29:31:42to take it from the second city, the
29:31:44aggregations, and we're going to prepare
29:31:46the final results. So here we're going
29:31:48to bring everything together and we
29:31:49might introduce final transformations
29:31:52that is needed for the reports. So let's
29:31:54go and write the query for the final
29:31:56results. Now we can go and start
29:31:58segmenting our customer and as well
29:32:01creating the KPIs. So let's go to the
29:32:03third step. I'm going to go and put this
29:32:05in a CTE. So let's call it customer
29:32:08aggregation.
29:32:10And now based on these results, we will
29:32:13write the final query. So I like always
29:32:16to put a comment about the steps. So the
29:32:18first city is the base query where we
29:32:20just joined the data and prepared it.
29:32:22And then the second query is for the
29:32:24aggregations. And the final one is for
29:32:26the final results. So let's go and start
29:32:29writing our final query. We will start
29:32:31with select. And I'm going to go and
29:32:32list again all the customer
29:32:34informations. So I'm going to go and get
29:32:36again same things. We have the customer
29:32:39key, customer number, name, age and so
29:32:41on. And now after that we need to create
29:32:43the age categories. And now after that
29:32:46I'm going to go and get all those
29:32:47measures as well from our query. But of
29:32:50course without the calculations I just
29:32:52need the names of
29:32:56it. So with that we have everything from
29:32:59our previous CTE. So the customer
29:33:02aggregation. Okay. So let's just test
29:33:05it. Now everything is working. So now
29:33:06what we have to do? We have to create
29:33:08few categories age category and as well
29:33:10the segments of the customers right for
29:33:12segmenting the customers we have already
29:33:14done the query so I will just copy and
29:33:16paste it from the previous analyszis it
29:33:18looks like this if the lifespan is at
29:33:20least like 12 months and the sales above
29:33:235,000 then a less or equal 5,000 then
29:33:26regular otherwise it is a new customer
29:33:28so this is our first segment but the
29:33:31second segment about the ages we're
29:33:32going to go and build it now and again
29:33:34how we going to do it when so if the age
29:33:36for example example less than 20 then
29:33:39the customer is under 20. Let's make
29:33:42another range where we say if the
29:33:44customer age is
29:33:46between 20 and let's say
29:33:4929 then we have the second range and we
29:33:53can keep repeating the same thing for
29:33:55the second one. It really depend how
29:33:57many categories you want to build. So 30
29:34:00and 39 I belong to this group. Now the
29:34:04next one let's have the 40s as well
29:34:06right so 40 49 same thing over here and
29:34:11now else let's say 50 and above right
29:34:14and above so let's go and end it as age
29:34:19group I just want to sort it little bit
29:34:22like this okay now it looks nice so with
29:34:24that again we have turned a measure into
29:34:27a dimension and let's go and execute it
29:34:29now so now by checking the results we
29:34:32have the details of the customers and
29:34:33Now we have a new category. So as you
29:34:36can see it is working. 54 it is above
29:34:3950. This is in the range between 40 and
29:34:4249. We have here 67 above 50. I believe
29:34:45we don't have any customer that is below
29:34:4820. Right? Or even between 20 and 30.
29:34:50Okay. So with that we have created our
29:34:53two categories and by looking to the
29:34:56reports you see we can segment the
29:34:58customers now into categories. The VIB,
29:35:00regular, new and the age group. And with
29:35:02that we have covered all those three
29:35:04requirements and we come now to the last
29:35:06requirements. We have to calculate the
29:35:08following KPIs. Now the first one it is
29:35:10an easy one. It is the recency. How many
29:35:12months since the last order we have
29:35:14calculated over here the last order for
29:35:16the customer. It is this one. And now in
29:35:19order to find the recency it is very
29:35:20simple. So all we have to do is to take
29:35:22this over here. I will just put it maybe
29:35:24after the segmentation. And all what you
29:35:26have to do is to use the date diff as
29:35:28usual. So month is the last order date
29:35:32and the get date. So as you can see we
29:35:35are using this setup like in many
29:35:37analyzes right we always find the
29:35:39differences between a date from our data
29:35:41sets and the current date and time and
29:35:43with that we will get the recency. So
29:35:47let's go and execute it. Now you can see
29:35:48how many months since the last order of
29:35:51the customer and of course you can go
29:35:53and test it using the last order date.
29:35:55And this is really important in order to
29:35:56understand whether the customer is still
29:35:58active or inactive. Okay, so this is for
29:36:01the first easy KPI. Now let's go to the
29:36:03second one. It says calculate the
29:36:05average order value. So how we going to
29:36:08do this? Let's go back over here. Now in
29:36:10order to compute the average order
29:36:12value, we have to divide the total sales
29:36:15by the total orders. So how many revenue
29:36:18did the customer generate? And we divide
29:36:20it by the total number of orders and
29:36:22after that we have to find the average.
29:36:24So it is very simple. Let's go and write
29:36:26that. We're going to go to the end of
29:36:27our table where we're going to put our
29:36:29KPI and I'm going to say here compute
29:36:32average order value. So as a shortcut
29:36:35AVO. So we say total sales divided by
29:36:39total orders. And let's call it average
29:36:42order value. So let's go and execute it.
29:36:45And if you go to the last over here, you
29:36:47can see the average order value of our
29:36:50customers. But now if you are dividing
29:36:51numbers together you have to be careful
29:36:53that you are not dividing by zero
29:36:55otherwise you will get an error. So
29:36:56imagine that a customer has a zero
29:36:58didn't order anything you might get an
29:37:01error. In our scenario, we don't have
29:37:02that because we are starting from the
29:37:04order table or the fact table. But
29:37:06still, I like to make sure this never
29:37:08happens. And for that, I usually go and
29:37:10use the case when statements. Very
29:37:12simple one. If the total orders is equal
29:37:14to zero, then make it zero. Otherwise,
29:37:18do the calculation that we talked about.
29:37:20So like this. And at the ends, we will
29:37:23add an end. So that's it. And with that,
29:37:26I make sure we will never divide by
29:37:28zero. So that's it. It was simple,
29:37:30right? Let's go to the last KBI the
29:37:33average monthly spend. So how we will
29:37:36calculate that
29:37:38compute average monthly spend. So now
29:37:42since we are speaking about the spending
29:37:44that means we need the total sales.
29:37:46Right? So how much sales did the
29:37:48customer generate totally and then we
29:37:51divide it by the number of months and
29:37:53with that we will get the average
29:37:55monthly spend. Right? So that means we
29:37:57can divide the total sales by the
29:38:00lifespan as we calculated it is the
29:38:02period where the customer has been
29:38:04active from the starts until the end.
29:38:06Okay. So now let's do it step by step.
29:38:08First we have to be careful that we are
29:38:10not dividing by zero and I believe in
29:38:12the lifespan we have zeros. So what
29:38:14we're going to say as usual case when
29:38:16lifespan is equal to zero then this time
29:38:19we will not make it zero the customer
29:38:21exist only for one month. So what we can
29:38:23do we can get the total sales of the
29:38:25customer and we don't have to divide it
29:38:27by the month in order to find the
29:38:29average because the average is equal to
29:38:31the current total
29:38:32sales. So with that we make sure we are
29:38:35not dividing by zero otherwise we're
29:38:37going to have our calculation. So total
29:38:39sales divided by life span. So the total
29:38:43sale divided by the months and with that
29:38:45we will get the average monthly spend.
29:38:47So and and ass and we're going to call
29:38:50it average monthly spend. Perfect. So
29:38:54let's go and try that out. Let's go to
29:38:56the right side. And with that we have
29:38:58our third KPI and we have the average
29:39:01monthly spends. And with that guys, we
29:39:03have now full reports about the
29:39:06customers and we have covered all the
29:39:08requirements. All right. So with that we
29:39:09have the final results and we have
29:39:11fulfilled the requirements. So what
29:39:13we're going to do, we're going to take
29:39:14the whole query and put it in the
29:39:16database as a view. And once we have the
29:39:18view, the report in the database, we can
29:39:20share it with the others. Now the other
29:39:22data analyst in the team can go and
29:39:24maybe create a dashboard in order to
29:39:26visual data using API tool like Tableau
29:39:29or PowerBI. But in this scenario, the
29:39:31user can go and connect your view the
29:39:33last prepared data to the dashboard. And
29:39:36with that the user can quickly generate
29:39:38insights without doing a lot of steps in
29:39:40order to prepare the data for the
29:39:42visualizations. And of course the data
29:39:44analyst can go and connect the
29:39:45dimensions and facts. But having this
29:39:47one solid view it's going to be like way
29:39:50easier to consume. And of course the
29:39:51data analyst can as well write a query
29:39:54on top of your view in order to generate
29:39:56a quick insights. So as you can see
29:39:57using only SQL you are covering a lot of
29:40:00complex steps in order to make the data
29:40:03ready for reporting and analyzes and
29:40:05this is what usually happened in real
29:40:07projects. We're going to go and put the
29:40:09query in the database so that the others
29:40:12can use it. So what we're going to do
29:40:14very simple create review and we're
29:40:16going to put it in a good layer and
29:40:17we're going to call it report customers
29:40:20and then ask like this and let's go and
29:40:23execute it. It is successful. Now if you
29:40:25go to our database and check the views
29:40:27you will find a new view called gold
29:40:29report customers. Now all what you have
29:40:31to do is to go and have a simple select.
29:40:34So codes reports customers and you will
29:40:38get an amazing report about the
29:40:41customers. This kind of reporting it is
29:40:43very important because you are giving a
29:40:45full picture 360° view of all your
29:40:48customers. So you have details,
29:40:50categories, measures everything in one
29:40:52go and it going to makes life easier.
29:40:54Now for any user of this view to quickly
29:40:56understand the data and generate maybe
29:40:58insights based in this one view that can
29:41:01helps of course your customers. So I
29:41:02just want to show you now what this
29:41:04means. If a user using your reports so
29:41:06either in SQL or maybe they're going to
29:41:08go and connect it to PowerBI or Tableau
29:41:10they can generate immediately insights.
29:41:12So for example, if they go and say count
29:41:15customer number so as total customers
29:41:19and then they're going to go and take
29:41:21any dimension for example the age group.
29:41:24So something like this and then group by
29:41:27the age group. Put just put it here
29:41:30first. And then they're going to go and
29:41:31add any other measure. For example, the
29:41:34total
29:41:36sales and any other measure that you
29:41:38have in this view and then execute and
29:41:41quickly they can do analyszis on top of
29:41:43your view without having them to go to
29:41:45their fact and dimensions. So this is
29:41:48like one extra prepared layer the data
29:41:50model that you have built. And if you
29:41:52don't want to group it by the ages, you
29:41:54can go and have the customer segments
29:41:56and it will be working. So quickly they
29:41:58can analyze the new derived informations
29:42:01that you have prepared in your reports.
29:42:03So guys, this is amazing reports about
29:42:06the
29:42:11customers. And now what you're going to
29:42:13do, you're going to go and prepare the
29:42:15second report where you have to build
29:42:17complete insights about the products of
29:42:19the business. It is very similar to the
29:42:22customers. So we want to generate a
29:42:24report for the products. You have to
29:42:26provide details like the product name,
29:42:27category, subcategory and the costs. You
29:42:30have to segment the products by the
29:42:32revenue. So you can have categories like
29:42:34high, medium and low. And then you have
29:42:35to provide the basic aggregations at the
29:42:38level of the products and then calculate
29:42:40few KPIs. So as you can see it is very
29:42:42similar to the customers. And now what
29:42:44you have to do you have to pause the
29:42:46video follow the same step at the
29:42:48customers where we join the tables car
29:42:50create aggregations and put everything
29:42:52like in CTE and at the end once you are
29:42:55done create the view where you have the
29:42:57report about the products. So I'm going
29:42:59to go now and do it offline and I will
29:43:01see you
29:43:02[Music]
29:43:06soon. Okay my friends I hope you are
29:43:08done with the reports. I'm going to show
29:43:10you quickly how I've done it. So I've
29:43:12just created a new view called report
29:43:14products and then we start with the base
29:43:16query where we have joined the fact
29:43:18table with the dimension products and
29:43:20collected all the columns that we need
29:43:23for the reports and we put everything in
29:43:25the first city. So this is the first
29:43:27step and there was from my side no need
29:43:30for any transformations over here. So we
29:43:32go now to the second step and here we
29:43:34have to put all the different types of
29:43:36aggregations in one go. So we calculate
29:43:38the lifespan, the last sales order,
29:43:41total orders, total customers, sales
29:43:43quantity and as well I have created the
29:43:45average selling price of the products.
29:43:48It is very simple. We are dividing the
29:43:50sales amount by the quantity. So this is
29:43:52the basic aggregations about the
29:43:54products and finally we have the final
29:43:56query. So we start with selecting the
29:43:58basic informations about the products.
29:44:00So we have the key, name, category and
29:44:02then we have here the recency and we
29:44:04have our new segments. This one is very
29:44:07easy for the products. So we are saying
29:44:08if the total sales is higher than 50,000
29:44:11then this is a high performer and if
29:44:14it's like between 50 and 10k then this
29:44:16is a mid-range otherwise it is low
29:44:19performer. So the segmentations of the
29:44:21products is very simple and after that
29:44:23we have like all our measures that we
29:44:26aggregated in the CTE and now we come to
29:44:28the two KBIS. It is very similar to the
29:44:30customers. So the first one the average
29:44:32order revenue it is simply dividing the
29:44:34sales by the total orders and you have
29:44:36to take care of the zeros of course and
29:44:39the average monthly revenue we divide
29:44:41the total sales by the lifespan of the
29:44:43products and of course if the lifespan
29:44:45is zero so it is only one month then it
29:44:48is the total sales and with that you
29:44:50generate the average monthly revenue. So
29:44:52as you can see it is very similar to the
29:44:54customers but still the focus here is
29:44:56the products. Now of course we put this
29:44:58query in view. So we have the report
29:45:01products side by side by the report
29:45:03customers and now we have really amazing
29:45:06report about the products where we have
29:45:07everything. So we have a lot of details
29:45:09about the customers. We have as well a
29:45:11dimension in order to segment our
29:45:14products and we have a lot of measures
29:45:16that are really important about each
29:45:18products. So we have the total number of
29:45:20orders sales, how many customers did
29:45:22order the products, the average price,
29:45:24the average revenue and the monthly
29:45:26average revenue. And this gives you
29:45:28really deep insights about each product
29:45:30of your business. And of course, this is
29:45:32very helpful in order to compare the
29:45:34products, right? And now, of course,
29:45:36this is core analyzis that you're going
29:45:38to need it a lot in your business.
29:45:39That's why we offer it as a view. So, I
29:45:42think we have now two amazing reports
29:45:44about our
29:45:48data. All right, my friends. So, now
29:45:50don't forget to put all your work in the
29:45:52Git repository in order to share it with
29:45:54others as a successful project. So as
29:45:57usual we have the data sets,
29:45:58documentations and as well the scripts
29:46:00that you have done through this projects
29:46:02and here I'm putting everything
29:46:04together. So we have all the activity of
29:46:06the exploration as well with the
29:46:08advanced analyszis that we have done. So
29:46:10we have the change over time, the
29:46:12cumulative analyszis, performance, data
29:46:14segmentations, part tool analyszis and
29:46:16as well our two new reports. So I
29:46:19recommend you if you haven't done that
29:46:20yet go and create now a repository put
29:46:23all your work there to make sure that
29:46:25everyone can access and see your work
29:46:26and my friends don't forget to add nice
29:46:29commenting on your code and formatting
29:46:31and styling your code should be perfect.
29:46:34So if you haven't done that yet go and
29:46:36do it now. All right my friends so with
29:46:38that we have done the last step in our
29:46:40road map. We have created two solid
29:46:42reporting for our users. And with that,
29:46:44we have completed all the steps of our
29:46:46advanced analytics projects. And with
29:46:48this project and the previous projects,
29:46:49you can see now the full picture on how
29:46:51to do data analytics on any data sets
29:46:54using SQL. So starting by the first step
29:46:57where we have explored the database and
29:46:59end up having a very solid reports where
29:47:02we have consolidated everything in one
29:47:04view and with that we have now really
29:47:05great understanding about the business,
29:47:08about our data. And now what you can do,
29:47:10you can go and grab any data sets in the
29:47:12internet and you can go through all
29:47:13these faces again and I promise you at
29:47:16the end you will have a full picture and
29:47:18understanding of the business and this
29:47:20is what I exactly do in each project if
29:47:22I want to understand any type of data
29:47:24sets. All right my friends. So with that
29:47:25we have covered the last type of SQL
29:47:28projects the advanced data analytics.
29:47:30And with that we have now three solid
29:47:32projects using SQL and they are very
29:47:35similar to real world projects in the
29:47:36industry especially if you want to be a
29:47:38data engineer or a data analyst. And my
29:47:40friends we have covered the last chapter
29:47:43in our course. So this is the advanced
29:47:45level in SQL. And those are all the
29:47:47chapters that I have designed for you to
29:47:50take you from the basics to intermediate
29:47:52and then to the advanced topics. My
29:47:54friend, you made it. Congrats. You
29:47:56should be really proud of yourself. And
29:47:58now with that, I can say that I have
29:47:59shared everything that I know about SQL
29:48:02and you can now solve any complex task
29:48:04using SQL like I do in my real projects.
29:48:07And I hope that you have enjoyed the
29:48:08journey. And if you do and you want me
29:48:10to create more free courses like this,
29:48:12make sure to support the channel by
29:48:14subscribing, liking, and commenting.
29:48:16This of course going to make the channel
29:48:17grow, reach the others, and as well
29:48:19motivates me to make more content like
29:48:21this. So nothing left to say. Thank you
29:48:24so much for watching and I will see you
29:48:25in the next course.

About this transcript

This page contains the full transcript of SQL Full Course for Beginners (30 Hours) – From Zero to Hero by Data with Baraa, generated from the public captions YouTube serves with the video. The transcript has 363,984 words across 48,358 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.