YT YouTube2Text

YouTube2Text → Transcripts → Data Modeling in Power BI Full Course for Beginners (5 Hours) – From Zero to Hero

Data Modeling in Power BI Full Course for Beginners (5 Hours) – From Zero to Hero — Transcript

by Data with Baraa · 67,962 words · 9,104 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Hey friends, so now we're going to speak
0:01about the most important chapter in the
0:03whole PowerBI, the data moduling.
0:05Because if data moduling is weak in your
0:07project, everything else going to suffer
0:09and fail. So in this chapter, we're
0:11going to do a full deep dive into data
0:13moduling in PowerBI, but the right way.
0:16>> There's a right way to do this.
0:18>> So I'm going to do it with you carefully
0:20step by step as usual with the
0:21fundamentals using sketches and then we
0:23deep dive as we progress. So now let's
0:26get started. All right. Right. So now
0:27let's have a look to the main faces the
0:29main components in PowerBI. So we start
0:32by connecting the data from the sources
0:34and after that we transform it using
0:36power query and then we build the data
0:39model and on top of it we write DAX
0:41calculations. Then the next step we
0:43start building visuals report and
0:45dashboard and share it with the others.
0:47So now by looking to this we have like
0:49clear separation between like backend
0:51work and front-end work. But now what
0:54happened in real PowerBI projects in
0:56many companies is that people try to
0:58skip things in the back end and jump
1:00immediately to the front- end things
1:02because building visuals are more fun
1:04right colors, charts, dashboards and it
1:07is the thing that the end users can see
1:09from you. So they try to speed up the
1:11backend process and especially one thing
1:13they skip the data moduling. So they
1:16might do some transformations in Power
1:18Query, build some DAX but the data model
1:20step it is totally skipped and this is
1:22my friends where many PowerBI projects
1:25start to fail over the time. So
1:27everything in the projects going to
1:28start be heavy, expensive, slow and hard
1:30to maintain and I can tell you many
1:32stories about this at work. I got
1:34invited many times to save PowerBI
1:36projects that are suffering and failing.
1:39So they ask me to review the project and
1:41suggest fixes in order to save it. And I
1:43can tell you after reviewing and
1:45auditing the whole project almost all of
1:47them are doing the same mistake. They
1:49are skipping the data moduling. They are
1:51really huge projects but they rush it.
1:53So they go immediately from the sources
1:55and start building visualizations and
1:57dashboard and over the time they start
1:59failing. So in many cases we just build
2:01the data model and save the project. But
2:03there were some projects where it was
2:05too late and they died over the time. So
2:07data modeling is very important and I
2:09understand why this happens because data
2:11modeling can feel hard and many courses
2:13out there they just show you simple star
2:16schema and they move on to the next
2:17steps but with me we will not do that
2:20>> I will not be beat
2:22up
2:23>> so now let's start with the mindset and
2:25understand why data modeling is so
2:27important and if you skip it everything
2:28going to fail let's go
2:30>> let's get the party started
2:31>> let's get the party started
2:32>> okay so it start like this someone going
2:34to ask you for a new report and you
2:36going to Start connecting the data and
2:38create one big flat table with all
2:41columns that are needed. Then you start
2:43adding some DAX calculations. Build the
2:45report and then share it with the user.
2:48So that's it. And this might takes like
2:50maybe one or to two weeks. At the start
2:52you're going to feel that everything
2:53looks fine and you're going to feel that
2:54you are fast and flexible. But PowerBI
2:57projects don't stay small. The data
2:59going to grow. You're going to get more
3:00requests to extend their report. You're
3:02going to get more users are using it.
3:04and you're going to start feeling things
3:06are getting like out of control because
3:08this one table my friend it is not a
3:11data model and it going to cause a lot
3:13of issues. So now the first issue is all
3:15about the size. Things might work if you
3:17have like 10,000 rows you won't feel the
3:19problem but if the sources are like
3:21providing 100k or million 10 million
3:25rows you're going to feel the issue.
3:26This one table going to get bigger very
3:29fast because the same information like
3:31the product name, customer name is
3:33repeated over and over which going to
3:35make the whole PowerBI heavy for no good
3:38reason. Instead of storing the product
3:40name only once, you are storing it
3:42thousands or millions of times. So the
3:45size of your one table going to grow
3:47very fast. And this going to lead my
3:49friend to many other issues like the
3:51performance. And here two things going
3:53to get slow. First performance issue is
3:55the load process from the source to
3:57powerbi. So refreshing the model going
4:00to move quickly from few minutes into
4:02half hour or hour. So it's going to be
4:04painful to bring a new data to the
4:05powerbi. And the other performance issue
4:07is the report performance. PowerBI going
4:10to need more time to scan the one big
4:12table to build the visualizations, the
4:14filters and to react to the interaction
4:17from the end user. So your report going
4:19to be slow your users going to suffer
4:21and I can tell you in real projects if
4:23the report doesn't open in like two or 3
4:25seconds your users going to say you know
4:27what the whole project is slow and
4:29they're going to start complain about
4:31it. So performance issue going to be
4:32everywhere. Now we come to another big
4:34issue the ducks. I met a lot of people
4:36in projects that think ducks is
4:38complicated or they are bad at ducks but
4:40actually this is not the problem. The
4:42one big table forcing them to write
4:44complicated formulas and calculations in
4:46DAX. The rule is very simple. Bad data
4:48model going to lead to bad complicated
4:51ducks because of not having a data model
4:53and having one table. A simple measure
4:55that should take only one line becomes
4:5710 lines because you have to twist
4:59things in order to get the right
5:01results. So having no model going to be
5:03the issue for the size, the performance
5:05and ducks. Now what is the solution is
5:07of course to build a data model. So that
5:09means we stop putting everything in one
5:11big table. We split the data into
5:14meaningful tables and connect them with
5:16the relationships. This going to solve
5:18all those issues because it going to
5:20make the model smaller. So the same
5:22product information or the customer
5:24information will not keep repeating
5:26because of the sales. So the text is not
5:28repeated for no reason million of times.
5:30The size going to stay small and this of
5:33course means you're going to have good
5:34performance. So you can load a new data
5:36to PowerBI. you can refresh faster and
5:39as well the performance of your reports
5:41going to be faster. Like for example, if
5:43you have a slicer in the report, PowerBI
5:45doesn't have to go and scan one big
5:47table. Instead, it's going to go and
5:49read one small dimension table which can
5:51be way faster to build the visual for
5:54your users and as well because of having
5:56relationships between those tables,
5:58ducks can become easy. So the data model
6:01and the relationships going to do big
6:02part of the work and your calculations
6:04on ducks can become smaller and easier.
6:07So those are the big wins and the first
6:09reasons why we need a data model. But
6:11there is another problem. Now the thing
6:13is if you skip building solid data
6:15models and you go with the one table
6:17this going to lead over the time to very
6:19bad habit or let's say bad system design
6:22that going to be the main reason of why
6:24your project is meant to fail. And by
6:27the way I saw this in many projects at
6:29companies. Now the thing is you will not
6:31build like only one report and that's it
6:33at companies. You will get requirements
6:35to build another report, another
6:37dashboard. And now since you don't have
6:39a data model and you just build one
6:41tailored table. This going to lead you
6:44to build another one table for the new
6:46requirements for the new reports. So
6:48you're going to do the whole process
6:49from the scratch. Connecting data,
6:51building one table, building one
6:52dashboard, and then sharing it with a
6:54user. And this what this design going to
6:56continue. So you're going to get a new
6:58requirement for new reports. And you're
6:59going to do the same thing over and
7:01over. One table for one report. And my
7:04friends if your project start looking
7:06like this one table one report then I
7:08can tell you this going to be the
7:10beginning of the end of your project.
7:12Now let me tell you why. Now for you the
7:14one that is building the project this
7:16going to be a nightmare to maintain. So
7:18let's say that you found an issue in the
7:20transformations or you want to change
7:22something because you got a new
7:23requirement. Now what going to happen?
7:25You're going to go to all PowerBI files
7:27and start editing this one table one by
7:30one. And you're going to try your best,
7:32of course, to do the same thing over and
7:34over to have the sync between those
7:37transformations. But I can tell you this
7:38is a lost battle because one day you're
7:41going to forget to update one of them.
7:43So one small change going to be a lot of
7:45work for you. It's going to become days
7:47or weeks until you finish. So really
7:49hard to maintain. And this going to lead
7:51as well to another issue. your
7:53calculations and transformations going
7:55to start be different from one report to
7:57another. And this is a big issue not
7:59only for you but as well for the
8:01business for the users because now your
8:03reports are showing different numbers.
8:05In one report you show the sales as 1.2
8:07million and another report could show
8:09the sales 1.4 million. At the end it is
8:12the same thing. It is a sales but you're
8:14going to calculate it differently in all
8:16those files. And I can tell you when
8:18your users going to start seeing that
8:20the numbers are not matching, they will
8:22stop trusting the whole PowerBI project.
8:24They will not see that the data model is
8:26wrong. They're going to see okay the
8:28whole PowerBI tool or let's say the
8:30whole project is wrong. And this is
8:32exactly the turning point. This is how
8:34projects starts to fail to die where
8:37people stop using the dashboard and they
8:39go back to their old tools using Excel.
8:42Now of course the question is what is
8:43the solution? Of course, building a
8:45solid data model. Now, the thing is one
8:48big table is usually built and tailored
8:50around one reports. But once you start
8:52thinking in models, you go and start
8:54building things around the business. So
8:57you got to say, okay, we need a sales,
8:59we need products, customers, dates and
9:01connect them together. You are not
9:03tailoring anything for a report. You are
9:05actually building a solid model for the
9:07business that can answer analytical
9:09questions. So naturally you're going to
9:11start reusing this model for multiple
9:14reports because usually the next reports
9:17are for the same business. So what I'm
9:19trying to say here is once you start
9:20building data models you start thinking
9:23how this model can serve multiple
9:26reports and this is a big win if your
9:28project start looking like this. First
9:30of all it is easier to maintain. You're
9:32going to be very fast. So if you found
9:34any issue or you want to change the
9:36logic of something you go to one place
9:38and do that and your change or your fix
9:41going to affect all the reports so it is
9:43easy to maintain and not only that this
9:45going to automatically force standards
9:47and consistency. So you're going to have
9:49less chance of having different numbers
9:52in different reports. So that means your
9:54projects going to have better chance to
9:56survive and be solid for many other use
9:59cases. So this is I'm going to say a
10:02bigger reason why we need data moduling
10:04in PowerBI projects. And now we come to
10:06some reality check or let's say some
10:08people going to push back on this and
10:10going to say hey this going to looks
10:11really good on paper but in real
10:13projects it will never work because
10:15different reports have different
10:17requirements and if everything depends
10:19on one shared model then changes one
10:21thing going to become really scary. So
10:23you're going to feel like I might break
10:25something if I change the calculation
10:27because all reports are using it at
10:29once. Well, I'm going to be honest. This
10:31is totally fair. This is what's going to
10:33happen. If you build one giant shared
10:36data model for everything, we will not
10:38do that. Of course, you will not build
10:40for one company, one model. You will be
10:42creating shared model by subject area,
10:45by domain. One model for the sales
10:47reports, one model for the finance,
10:49another one for the HR report. So,
10:51you're going to get like a group of
10:52reports that are using one model. Things
10:55going to looks like this. For each
10:56domain, you're going to have multiple
10:58reports and one model. And if your
11:00project is getting bigger and bigger,
11:01you're going to have multiple models. So
11:03that means we will be in the middle. So
11:05we will not go and create like one table
11:07for each report. And at the same time,
11:09we will not try to force one model for
11:12the whole company. Now let's have a
11:14recap why data modeling is important
11:16because bad modules going to create
11:18technical problems, very big PowerBI
11:20files, slow reports, hard to refresh and
11:24complex ducks. and as well bad moduling
11:26going to also create project problems,
11:29repeated work, hard maintenance,
11:31different KBI numbers and users losing
11:34trust. So I hope with this you have now
11:36enough arguments why you have to take
11:38time to do data moduling. Maybe you have
11:40to go and now convince one of your
11:41managers or your project. If you skip
11:43it, your report might work for now, but
11:45over the time you have high risks of
11:47failing. [music]
11:51So now what is data modeling in PowerBI?
11:55Now first of all the data model it is a
11:57structure that describes the business
11:59data and how the pieces are actually
12:01related to each others. What this
12:03exactly means? Now data modeling it is a
12:06process that looks like this. You're
12:07going to have a look to your data. It
12:09could be chaotic inside files, inside
12:11databases or in one big file. And inside
12:14it you're going to have like mix of
12:15business objects, sales, products,
12:18customers, dates. Now you're going to
12:19look at it and then decide what is the
12:22best structure for it. And here you
12:24decide for two reasons. First for the
12:26business. So you're going to go and
12:28start organizing the data into clear
12:30business objects. Like you're going to
12:32say okay all the sales data going to go
12:34in one place. Then you split and take
12:37all the products in another table. Same
12:39things for customers and dates. So you
12:42start splitting your data and put it in
12:44places based on the business itself. Now
12:47the second reason why you want to bring
12:49a structure for it, it is for the tool
12:51that you are using like for example
12:53PowerBI. We want to shape the data in a
12:55way that PowerBI can work efficiently
12:58and correctly so that you get nice
13:00response and fast performance. That
13:02means our goal is to look to the
13:04original data and think about how to
13:07reshape it for those two goals. Split it
13:09into tables based on the business and
13:12split it in order to get good
13:14performance for the tool that you are
13:15using. We are not splitting things
13:17randomly. We have a plan.
13:19>> My plan a man pan.
13:22>> Now the next step in data modeling is to
13:25decide how to connect those tables with
13:27relationships. Relationship it is as
13:29well a description that can explain how
13:32those tables relate to each others. And
13:34the final third component in data
13:36modeling we define the calculation on
13:39top of the model like the total sales
13:41the average profits the total number of
13:43orders number of customers all those
13:45calculations and in PowerBI we do those
13:48calculations using DAX. So that means
13:50the model is not just like one table or
13:52one object. It is three parts that work
13:55together. Tables to store the data
13:57relationships in order to connect those
14:00tables and calculations in order to
14:02answer business questions. All right. So
14:04this is the data moduling and now let's
14:06go and focus on tables because in data
14:09modeling tables can have different
14:11shapes different roles depend on the
14:13moduling technique. In techniques like
14:15for example the data vault the tables
14:17can have the role of hubs links
14:19satellites bridge. So different like
14:22let's say styling and in operational
14:24systems the tables might take the shape
14:26of an object or let's say play the role
14:29of relationships and link but in PowerBI
14:32things a little bit easier because we
14:34are building for analytical reports and
14:36the most important moduling style or
14:39technique it is the dimensional
14:41moduling. So in this technique the
14:43tables might take one of those two
14:44shapes fact tables and the dimension
14:47tables. And now let's deep dive into
14:49those two and start with the first one
14:51the fact table. So what is that? A fact
14:53table stores business events,
14:56transactions or activity that we can use
14:59in order to calculate something. So now
15:01what this exactly means? Let's have this
15:03example. So now here we have a sales
15:05table. Now what we do? We have a deep
15:07look to the data in order to understand
15:09the behavior or what this table is
15:11trying to tell us. Now first of all we
15:13can see the table is recording something
15:15that happened to the business and what
15:17happened here a sale did happen. So
15:20there is like an event. The second thing
15:22that you're going to see it might
15:23include a lot of ids those going to be
15:26like a link to another thing another
15:28table. Now another pattern you're going
15:31to find informations about the date and
15:33time like an order date an invoice date
15:36transaction date. So you're going to
15:37find informations about when this event
15:40did happen. And the last pattern that
15:42you're going to find here, we have
15:44numbers like quantity, sales, discount,
15:47cost or profit. All those numbers going
15:49to help us to calculate something. So if
15:52you see all those together, so an event,
15:55ids to something else, information about
15:57the date and time and a lot of numbers,
16:00then my friend you are looking at a fact
16:03table. This table is just recording the
16:05business activity. So this is the first
16:07type, the fact tables. Now let's talk
16:09about the second type. We have the
16:11dimensions. A dimension table stores
16:14descriptive informations that gives the
16:16context to the facts. Of course, we have
16:18to understand what this means. Look at
16:20this table. It is storing the products
16:22informations. And you can see each one
16:24row is actually representing one
16:27product. So that means the row here is
16:29not like an event or like we are
16:31recording something happened. It is just
16:34describing a business object like the
16:36product. Now the second thing that
16:38you're going to notice that we have a
16:40lot of attributes informations that is
16:42describing this product like the product
16:44name, the category, the brand, the
16:47color, the status. So they are just
16:49adding more context to the product. And
16:52now by looking to this you can
16:53understand those columns are really
16:55important in reports. So we could use
16:57one of those columns in order to do data
17:00filtering in order to focus only on
17:01small subset or we could use one of
17:03those informations to make a group of
17:06numbers in our reports. Like for
17:08example, we can take the categories and
17:10we can build something like the sales by
17:12category. So that means this information
17:14the category going to help us to group
17:17up the sales number and this is of
17:19course the core thing in building
17:21visualizations and reports. So if you
17:23have a table like this where we are
17:25describing one business object like the
17:28products or maybe the customers and we
17:31have a lot of columns that are adding
17:33more and more context and they look
17:35amazing in order to build a filter or to
17:37use it in order to group the numbers
17:39then my friends you are looking at a
17:41dimension table. So this is exactly what
17:43we mean with the second type the
17:45dimensions. All right so now let's
17:46compare those two quickly side by side.
17:49The fact tables record what happens.
17:52Dimension tables describe what happened.
17:55The core of the facts are usually dates
17:58and numbers in order to do calculations.
18:00The core of the dimensions, they are
18:02usually the labels that we use to
18:05analyze those numbers. Like for example,
18:08sales by category. The sales are numbers
18:10that comes from the fact. But the
18:12category, it is the label, the context,
18:15it comes from the dimension. So the fact
18:18answers what happened. It is like an
18:20event and transaction. A dimension
18:23answers who, what, where and when. So it
18:26is the description, it is the context.
18:29So actually that's it. Once you
18:30understand those two then data modeling
18:32become much easier to understand and as
18:35well to build. Okay. So now by looking
18:36back to our data model with that we have
18:39the two main tables facts and
18:40dimensions. And now of course the
18:42question is how to connect them. This of
18:44course going to lead us to a very famous
18:47shape called the star schema. A star
18:49schema means we have a fact table that
18:52sits in the middle and we have all those
18:54dimension tables that connect around it.
18:57So the fact sales is in the middle and
19:00around it we have a products, customers,
19:02date, store and look at this, it is
19:05forming the shape of the star, right?
19:07That's why we call it star schema. And
19:10this is our target in data modeling. We
19:12want to get this shape. And my friends,
19:14PowerBI just love star schema. It works
19:17very well with this shape and of course
19:19you might ask why. Well, there are many
19:22reasons like for example it gives
19:24PowerBI a clean way to filter the data.
19:26So all those dimensions around the fact
19:29they're going to go and filter the
19:31central table the fact that holds the
19:33numbers and of course all those
19:34relationships going to help PowerBI to
19:36understand how the filter should move.
19:39So for example if you are filtering the
19:41products this has an effect on the
19:44numbers like for example the sales. Same
19:46thing if you filter the customers this
19:48going to automatically filter the sales
19:50and for all other dimensions. And now
19:52because of filter logic is so simple DAX
19:55going to become as well much easier to
19:57write. So all what you have to write now
19:59is just like for example the total sales
20:01is equal to sum of sales. We don't have
20:04to write any complex DAX calculations
20:06because we have a clean data model that
20:08is supporting it. The model already know
20:11how products customers and dates are
20:13connected to the sales. So you don't
20:15have to repeat it in the calculation.
20:17And not only that as well, it's going to
20:19make building the visuals much easier
20:21for you and as well for PowerBI. So
20:23you're going to have this mental mode in
20:25your head to say, okay, if I'm building
20:27the visuals, I need numbers and you go
20:29immediately to the middle to the fact
20:31and if you want to group it up by
20:33something, then you look at the
20:35dimensions to get a label. Same thing if
20:37you want to build a slicer, you usually
20:39go and look to the dimensions. So that
20:41means star schema gonna gives you a
20:43clean starting point to build any
20:45visual. So actually that's it. This is
20:47the star schema.
20:48>> So shoot for the stars. Okay. All right.
20:51>> Fact in the middle dimensions around it.
20:53Relationships from the dimensions to the
20:55facts. The facts going to hold the
20:57activities the numbers the dimension
21:00going to give the context and the
21:01filters. So this is exactly why star
21:03schema is amazing not only for PowerBI
21:06but as well for other BI tools like for
21:08example Tableau. And now the thing is we
21:10don't have only one methods on how to
21:12module the data for PowerBI but as well
21:14we have another thing that you might
21:16hear snowflake schema and the galaxy
21:19schema. So what are those stuff in real
21:21projects what might happen that one of
21:24your dimensions in the star schema start
21:26getting bigger and bigger over the time.
21:29Not only you are getting more data and
21:31rows but as well you are adding more
21:33columns over the time. So your dimension
21:35gets really big and here you decide to
21:37go and split it into two dimensions. So
21:40now for example instead of having one
21:42dimension for the products you're going
21:44to go and split it into two dimensions
21:46one for the products and one for the
21:48category. And now what you do after
21:50splitting usually you go and connect
21:52them and build relationship between
21:53them. So that means the relationship is
21:56not only between the dimension and fact
21:59but as well you can have it between one
22:01dimension and another dimension. And
22:03this might happen to another dimension
22:05and so on and you start splitting and
22:07over the time this star schema going to
22:10looks like a snowflake schema. So it
22:13looks like snowflake. Now if you work
22:14with those two and compare them side by
22:16side you're going to understand that the
22:18snowflake schema it is more complex. You
22:20have more relationships to manage. You
22:23have to write a ducks that is suitable
22:25for the snowflake. And now PowerBI might
22:28get a little bit slower because it's
22:30going to go and connect a lot of tables
22:32until it reach the fact table. But of
22:35course on the other hand the snowflake
22:36going to help you reducing the size of
22:39the model. So if your PowerBI file is
22:42really big one, you can go and split it
22:44and use the snowflake in order to reduce
22:46the size of it. So I worked with those
22:48two and I can tell you I try most of the
22:50time to avoid using the snowflake and
22:52try as a default to use the star schema
22:55because it is just simple and only in
22:58extreme cases I go and split a dimension
23:00into two and okay we call it now
23:03snowflake. So snowflake is the second
23:05type of data modeling but actually we
23:07have a third one that called galaxy.
23:09Sounds fancy right?
23:10>> I say chiao cuz I'm fancy from
23:13Tallahassee. So what this means now this
23:15time the dimension is not a problem but
23:17the fact in many scenarios in real
23:20projects you will not have like only one
23:22perfect fact that's going to serve all
23:24the reports you might end up having more
23:27than one fact like for example you have
23:29one for the sales and another one for
23:31the budget. Now the thing is you need
23:33both in order to build one big dashboard
23:35and those two facts are sharing the same
23:38dimensions like for example fact sales
23:41is using the dimension product and same
23:43thing going to happen the fact budget
23:45going to need as well the dimension
23:47products. So with this we have the
23:49product as a shared dimension between
23:51two facts. So actually that's it. If
23:53your model going to look like this then
23:55my friends you have a galaxy. But here
23:58be very careful. This is very common
24:00mistake that done in data moduling. We
24:03never connect the fact tables together.
24:05They going to get connected through a
24:07shared dimension. Of course, this is
24:09very important topic and at the same
24:11time very advanced. We're going to leave
24:12it later in the chapters. But now for
24:14you just have the sounding that we have
24:16star schema. This is the most common and
24:19default data modeling that we do in PI
24:22and analytics. Then we have a snowflake.
24:24If you have an issue with big dimensions
24:26and you want to split and normalize, by
24:29the way, if I say normalize, that means
24:30we are just splitting big tables into
24:32smaller tables. And the third one, the
24:34galaxy, if you have multiple facts that
24:37are sharing the same dimensions. Now,
24:39just quick clarification about the terms
24:41that you might hear in PowerBI projects
24:43at work. You're going to hear people
24:44saying data set, semantic model, data
24:47model. At the end, it is all the same
24:49data set. It is just an old terminology
24:52used by Microsoft PowerBI and now they
24:54are calling it semantic model. The data
24:56model it's like people like me I use it
24:58as official term. It is the process not
25:01only in PowerBI but as well in any like
25:03platform that you are building that has
25:05to do with data. So if you hear all
25:07those terms don't worry about it. Just
25:09keep this in your head. The report is
25:11the visual layer and the model is what
25:14sits behind the report and they can call
25:16it whatever they want. Okay. So now
25:18let's do a quick recap for everything
25:20that we have learned today. A data model
25:22it is the way that we structure our data
25:24for two reasons. In order to build
25:26business objects and as well to make a
25:28clear structure for the tool that we are
25:30using like PowerBI it is made of three
25:33components tables relationships and
25:36calculations on top of those tables. And
25:38now since PowerBI is a BI tool
25:41analytical tool then the tables usually
25:43have two important roles. Either it's
25:46going to be a fact table or a dimension.
25:48Fact table going to record what happens.
25:50So they are events. Dimensions they
25:53going to describe what happened. So it
25:55is the context and it going to help us
25:57to filter and group the numbers. And
25:59when we connect the dimensions around
26:02the fact then we're going to get the
26:03shape of the star schema. This is the
26:06foundations of building PowerBI models.
26:08So everything going to be easy for
26:10PowerBI in order to filter data and as
26:12well to write a DAX calculations and if
26:15one of your dimensions are getting
26:17bigger we split it into two dimensions
26:20then we connect them together and with
26:21that you're going to get the snowflake
26:23schema or in other scenario if you have
26:25more than one fact that are sharing the
26:28same dimension then you have a galaxy.
26:30All right, my friends. So, with this,
26:31I'm really happy that now you have a
26:34solid foundations about data modeling in
26:36PowerBI in order to start really
26:38practicing. And this is exactly what
26:40we're going to do in the next video.
26:41We're going to take a row table build
26:43full star schema end to end in order to
26:45practice all those concepts.
26:51Hey friends, [music] so now we're going
26:52to speak about the technical part of
26:54data modeling in PowerBI. It will not be
26:56easy. So, I need your full focus. Now
26:58the thing is PowerBI is very advanced in
27:00data moduling and as well very
27:02sensitive. That means not only making a
27:05bad decision in big topic like how to
27:07module the data in star schema,
27:09snowflake and so on could lead to bad
27:11performance and bad results as well a
27:14tiny small mistake that you decide in
27:16the data model like for example the
27:18cardinality on a relationship between
27:20two tables could lead to bad numbers and
27:23bad results. So that's why it is not an
27:25easy topic to learn and to be honest to
27:27teach. You have to do a lot of projects
27:28in order to understand the different
27:30scenarios and why we do those decisions.
27:32Now I'm going to do my best to bring all
27:34the scenarios that I had in my projects
27:36as a scenarios in this course. So now
27:38let's get started.
27:40>> Going speed. Here we go.
27:43>> All right. So now let's start with the
27:44foundations in PowerBI. We have all
27:46those different layers. And we're going
27:47to focus now into two layers together.
27:49The module and the visual layer. So
27:52let's zoom in. The visual layer it is
27:54the place where you're going to go and
27:55build your report. So usually you have
27:57like maybe two tables and you pick one
27:59of them and start drag and drop to the
28:01visual to build maybe a table or a
28:03charts a card and PowerBI going to just
28:05show the visual. But now what going to
28:07happen if you go to another table and
28:10drag and drop inside the same visual.
28:13Now PowerBI going to say okay but wait
28:14how I'm going to go and combine those
28:16two different data from two different
28:18tables in one visual. So PowerBI need
28:21from you more infos in order to combine
28:23the data and we cannot define the logic
28:26in the visual layer. We have to do it in
28:28the engine in the previous layer the
28:31module layer. There we're going to
28:32define everything the shape of the
28:34tables how the tables are related to
28:36each others using relationships the
28:38cardinality the filtering the
28:40calculations how the data going to move
28:42from one table to another define the
28:45security rules. So PowerBI going to find
28:47all the rules, all the description about
28:50how everything is related to each others
28:53and how PowerBI going to go and deal
28:55with those two informations in the
28:56visual. And based of all those
28:58informations in the module layer,
29:00PowerBI going to have all the knowledge
29:02in order to calculate the numbers and
29:04plot charts in the visual layer. And
29:07here is the most important thing to
29:09understand. If the numbers are wrong in
29:11the visual layer, it is highly likely
29:13because the rules, the shapes, the
29:16relationships on the module layers are
29:18wrong. So bad model going to define bad
29:21rules that going to lead to bad numbers.
29:24And of course, my friends, don't expect
29:25if you make mistakes on the module layer
29:27that you're going to find big error
29:29message in the visual in the reports.
29:31No, mostly the report going to function.
29:34You're going to see numbers, your
29:35visuals are working, everything is okay.
29:37Only if you or your users make zoom in
29:40and start really checking the numbers,
29:42they might start seeing those wrong
29:44numbers. So mostly the mistakes going to
29:47lead to silent errors that it is really
29:49hard to find. But don't worry about it.
29:51We're going to start with the
29:52foundations. Now it always start like
29:54this. In my data I have two tables left
29:56and right. And I'm going to need both of
29:58them in my reports. And here my friends,
30:01you have two options, not only one.
30:02Either you're going to go and combine
30:04those two tables together into one
30:07single table like we have done in power
30:09query by merging the tables into one.
30:12They are physically now one table. Or
30:14the other option that you say you know
30:16what let's just leave them as it is. So
30:18we will not go to power query and do
30:19anything. We will just like put a line a
30:22relationship between those two
30:24physically they're going to start
30:25separated from each others but they have
30:27a relationship between them. So this is
30:29the first decision that you have to make
30:30either in barquery you are merging or
30:33building relationships in the module
30:35layer. Now of course we are focusing on
30:37the relationships. So what are they? A
30:39relationship it is like a contract
30:41between two tables in order for you to
30:43define the rules that going to tell
30:45PowerBI how to filter and how to combine
30:48the data from those two tables into one
30:51visual in the front end. And now you
30:52might say wait you said relationships we
30:55don't combine data only merging in power
30:57query we combine. So now we are
30:59combining actually in relationships. So
31:01what is going on? What is going on? Now
31:04let's zoom out again the three layers
31:06power query module layer and visual
31:09layer. Now if you merge two tables it's
31:11going to happen at the beginning in the
31:13start in the power query. So everything
31:15going to be merged together the columns
31:17the rows into one single table. And of
31:19course in the next layers the module the
31:21visual you're going to see only one
31:22single table. So that means combining
31:25the data happens in very early stage in
31:28the power query. But now in the other
31:29side if you go to the module layer and
31:31build a relationship between those two
31:33tables they will not get combined. They
31:35going to stay separated. So that means
31:38in module and as well in visual layer
31:40you're going to see the original two
31:41tables. But only if you go and build a
31:44chart a visual in the visual layer and
31:46you start picking data from left and
31:48right only there the data going to get
31:50combined and of course not all columns
31:52not all rows this depends on what you
31:55picked and what filters you are using.
31:56Now of course you're going to say okay
31:58when we combine in power query and when
32:00we build relationships the rule is very
32:02simple. If two tables are describing the
32:04same row like for example you have
32:06customer info and customer emails
32:08separated in two different tables then
32:10no need to build a relationships just go
32:12and combine them at early stage in the
32:15power query. But if you have two
32:16different tables like for example the
32:18sales and the customers it would be bad
32:21idea to go and combine them together
32:23instead build a relationship between
32:24them. Those are the two options and the
32:26first decision that you have to make.
32:27But now we have more things. Now in
32:29order to build a relationship between
32:30two tables, you have to define for
32:32PowerBI three things. First you have to
32:34define the keys. They are common columns
32:37between two tables like ids, keys or
32:40whatever. So that PowerBI know how to
32:42match the rows between those two tables.
32:45Without it, PowerBI has no idea how to
32:47combine the rows. Another thing you have
32:48to define the cardality. Yes, a fancy
32:51name. So you have to describe the values
32:53inside those keys. And here we have two
32:55options that we have to define for each
32:57side. It is very simple. One means the
33:00values are unique. There is no
33:02repetitions. And other value many means
33:05the values are repeating. They are not
33:07unique. And one value can appear
33:09multiple times. So that means we have
33:10two options, two sides. That means we
33:12have four cardinality options. Yes, I
33:14can do math.
33:15>> Hey Michael, what's 394
33:18* 5,912?
33:21>> Let's see. We have one to one, one to
33:24many, many to one and the evil and the
33:26one, the only many to many relationship.
33:29This is going to be the real sufferer.
33:31Okay. So let's have very simple example.
33:33We start with the onetoone relationship.
33:35So here we have two tables about the
33:36customers. One has the names, the other
33:38has the tiers and they are connected
33:40using the customer ID. Now let's go and
33:42check the values inside the keys. We can
33:43see on the left side the keys are
33:45unique. For each customer we have
33:47exactly one row. That's why the
33:49relationship on the left side is one.
33:51Now let's go and check the right side.
33:53We have the exact same thing. Values of
33:55the customer ID is unique. There are no
33:57duplicates. And as well on the right
33:59side is going to be one. So that's it.
34:00This is the one to one relationship. Now
34:02let's have an example about the one to
34:04many relationship. So on the left side
34:06we have the customers again. And on the
34:08right sides we have the orders. Now same
34:10thing. We're going to check the values
34:11inside the keys. On the customers we
34:13don't have duplicates. So they are all
34:15unique. But on the right side we have
34:17duplicates. So for the first customer,
34:19Maria did place two orders. So on the
34:22left side it is one. On the right side
34:24it is many. It exist more than once.
34:27Let's try out Joan. Same thing. On the
34:29left side one. On the right side many.
34:32But of course this is not must. Like for
34:33example Anna she ordered only once. But
34:36since we have duplicates from other
34:37customers this relationship going to be
34:39one to many. Now the many to one it's
34:42exactly the same but we just like flip.
34:44So if the dimension now on the right
34:45side we say it is many to one
34:47relationship. So the same thing we are
34:49just like flipping the sides nothing
34:50new. Okay. Now we come to the many to
34:52many relationship and here we have two
34:54tables. On the left side we have the
34:55customer segments and on the right side
34:57we have the orders the sales. Same thing
34:59they are connected using the customer
35:01ID. But now if you have a look to this
35:03you can see the customer ID is not
35:06anymore unique on the left side. So a
35:08customer could belong to multiple
35:10segments and on the right side the fact
35:12it was normal because a customer could
35:14order multiple things. So that means if
35:16I click to one of those customers same
35:18effect they go and like order multiple
35:20things but now here is the issue one
35:22order belongs now to two rows on the
35:25left side because the customer belong to
35:27two segments. So that means the left
35:29side is not anymore one it is many and
35:32with that we have many to many
35:35relationship. So we have to define the
35:37rules on that cardality. Another thing
35:39that you have to define is the filter
35:41direction. Now in reporting we filter
35:43data like crazy and we have to define
35:46the direction of how the filter going to
35:48move from one table to another and
35:50whether it is allowed and in order to
35:53control the traffic of the filters we
35:55have as well some options. So now again
35:57we have this simple example where we
35:58have two tables the sales and the
36:00segments and for each table we have like
36:02an aggregation for the segments we have
36:04the total number of segments and here we
36:05have the total number of sales. Now one
36:07option to define the direction of the
36:09filters to say it is single filter from
36:12the left to the right. So for example if
36:14I go and filter the data for the
36:15customer ID you can see the sales is
36:18already filtered with the total but as
36:20well the right side is filtered. So we
36:22have here only two segments. Same thing
36:24if I go and filter the data by another
36:26customer you can see it always has an
36:28effect on the right side. So whatever
36:30I'm filtering on the left the filter is
36:33going to the right side. But now if I go
36:35and start filtering from the right side,
36:37you can see okay the table is filtered.
36:39We have only two segments. But this
36:41filter has no effect on the sales. I'm
36:44still seeing the total number of sales.
36:46So the filter is not traveling from the
36:49right side to the left side. Same thing
36:51if I go and select the other customers.
36:54You can see the sales table is not
36:55affected. Now of course we can do the
36:57opposite where we can say single filter
36:59from segments to the sales. So this time
37:02if I filter on the segments it's going
37:04to filter the sales. But now of course
37:05if I filter the sales there's no effect
37:08on the right side. Now if you say you
37:09know what let's have it free for all
37:11everything filter everything. For that
37:13we're going to use the both. So that
37:15means if I'm filtering on the left side
37:17the data going to get filtered on the
37:19right side. And the same thing if I'm
37:20filtering on the right side the data are
37:23getting filtered on the left side. So
37:25wherever I'm filtering it going to go
37:27everywhere. So as you can see my friends
37:29with this we can control the behavior of
37:31the whole data model how the data is
37:33filtered and this has a great effect on
37:35the reports. So those are the three
37:37rules that we have to define on the
37:39relationships the keys cardalities and
37:42the filter direction. And guess what we
37:44will not do this only between two
37:46tables. We have to do it for the whole
37:48data model. Each time you connect two
37:50tables you have to define those. And if
37:52you do it wrong you're going to get
37:54wrong results. All right my friends. So
37:56now enough with the sketches and the
37:58theory. Let's go back to PowerBI and
38:00practice. [music]
38:04All right, new [music] start blank
38:06report and we are on the model view. Of
38:08course, what do we need? We need data.
38:10So we're going to go and connect a CSV
38:12files. Here I have prepared some facts
38:14and dimensions. We're going to start
38:15with the fact sales. So let's go and
38:17load it. As usual, we're going to get a
38:19preview. Okay, load. Now you can see we
38:22have something new. We have a card
38:23inside the model. This is the fact sales
38:26and inside it we have a lot of columns.
38:28You can go and expand it if you want but
38:30if you want to hide all those details
38:32you can go and minimize it. So this is
38:33our fact. Now if you click on it you're
38:35going to find here on the right side
38:36some options like for example in the
38:38properties you can go and rename your
38:40fact. You can go and add a description.
38:42We have some synonyms. So we can call it
38:44fact sales sales facts and so on. This
38:47is important for the PowerBI Q&A. If the
38:49user like ask okay give me the sales.
38:51PowerBI knows okay the sales means the
38:54fact table. Same thing for the row label
38:56as well for the Q&A and the last one you
38:59can go and define the key column but for
39:00now we will not define anything. So as
39:02you can see some few metadata about our
39:05new table. Now if you go to the advanced
39:07this is all about the load modes since
39:09it is a file it is an import. Now on the
39:11right side in the data pane we have here
39:13two tabs the tables. It is a nice way to
39:16see the tables and the columns inside
39:17it. And if you go to the model, this is
39:19another way on how to present the data
39:21model for us. Like for example, we can
39:23go to the tables over here and we're
39:24going to see like a list of all our
39:26tables. In our model, we have a lot of
39:28things. So we have tables,
39:29relationships, we have measures. So here
39:31PowerBI tried to group the different
39:33things inside our model by topics. With
39:36that, you get a nice overview of the
39:38whole data model. So this is the
39:40interface about modeling in PowerBI. It
39:42is really advanced compared to other
39:43tools. Now we don't have yet a data
39:45model. We have only the fact sales.
39:47That's why we're going to go and connect
39:49for it another table. Usually, we go and
39:51connect a dimension. So, let's go and do
39:53that. I'm going to go and pick the
39:55dimension customers. All right, let's go
39:57and load it. Okay, once it's done, we
39:59have a new card inside our data model
40:02for the dimension customer and we can
40:04move it around, but we keep the fact
40:06table in the center. So, we have a new
40:08dimension and you can see we have
40:10already a relationship between those two
40:12tables. So this nice line means those
40:15two tables are connected and PowerBI yes
40:17do it automatically. Now as a default if
40:19you add new table inside the model
40:21PowerBI going to go and investigate all
40:24the data inside this new table
40:26everything the values the uniqueness and
40:28as well the column names and once it
40:30finds a possibility to connect this new
40:33table to the whole model it going to go
40:35and do that. So it is a suggestion from
40:37PowerBI that is based on the data. It's
40:39not like a random guess from PowerBI. Uh
40:42this is of course a nice thing if you
40:43have small data model but once the model
40:46gets bigger and bigger this might be
40:48annoying because PowerBI always try to
40:50connected to all other tables and what's
40:53happen you're going to end up like
40:54cleaning up all those relationships and
40:56start over from the scratch. So if you
40:58have a small model leave it as it is but
41:01if you have a big data model you can go
41:03to the options and deactivate it. So now
41:05in order to learn how to create a
41:06relationship, we're going to go and
41:07delete this one over here. So if you
41:09just select it and hit delete, you can
41:11go and remove it. Now in order to
41:13recreate it, you have like multiple
41:14options. Either you're going to go to
41:16the manage relationships over here in
41:18the ribon. So if you click on it, you
41:20can go over here and start relationship
41:21between two tables or what I find like
41:24cooler if I go and let me close this.
41:26For example, here customer ID, drag and
41:28drop it on top of the other column, the
41:30customer ID on the fact then release.
41:33With that you're going to create a
41:34relationship. So now what do we have in
41:35this window? You're going to read it
41:37from top to bottom. It's very simple. It
41:39says okay from table dimension customer.
41:41And then you have a nice preview of the
41:43data to understand uh the values inside
41:45the columns to the table fact sales.
41:48Same thing we have a preview. So now
41:50what we are doing over here is we are
41:51just connecting the tables using the
41:53keys. So you can see here it's
41:54highlighted customer ID and customer ID.
41:57This is of course correct because I
41:58dragged and dropped on the correct
42:00columns. But if it was wrong, you can go
42:01and select another column by just like
42:03clicking on the column name. So this I'm
42:05saying the key is the customer number
42:07from the dimension and the customer ID
42:09from the fact. But of course this is not
42:11correct. So I'm going to go and switch
42:12it back. So this is the first thing that
42:14we define. Another thing we define the
42:15cardality and here we have four options.
42:17Now PowerBI going to target one to many
42:20relationship and this is because PowerBI
42:22knows the data and it selects the
42:24correct one. So this selection is based
42:27on the data. But PowerBI has no idea
42:29whether this is correct for the business
42:31or not. Only you and the business users
42:34know the relationship between business
42:36objects in the model. So this is not a
42:38suggestion about how to connect tables.
42:41This is something based on the data that
42:43we have inside PowerBI and PowerBI here
42:46says okay based on all the values that
42:48we see inside the customers this is
42:50unique that's why it is one. But the
42:52customer ID inside the fact you can see
42:54here we have one one it is not unique
42:56that's why we have the money. Now of
42:58course you can go and force another
42:59cardality if you want but PowerBI will
43:02not allow it can say no no we have here
43:04many so you cannot do that. So we're
43:06going to go to the one to many. Now of
43:08course if we start from the table fact
43:10and then dimension is going to be many
43:12to one. It's the same thing of course.
43:14So this is about the cardity. Now the
43:16last one the filter direction. We can
43:18see here we have two options both and
43:20single. So if you leave it as both and
43:22then let's go and check. So let me save
43:24it. So now look at this. We have like an
43:26icon on the relationship. It says both
43:28direction. And here comes something
43:29really important to understand. The best
43:31practices in the star schema is like
43:34this. The filter should move from the
43:36dimension to the fact and not both
43:39directions because this going to create
43:41something called ambiguity in filtering.
43:44That means it going to has like a
43:45different effect on another table. We're
43:47going to have an example. Don't worry
43:48about it for now. Keep it in mind the
43:51filter direction is always from
43:53dimensions to the center to the facts.
43:55So now in order to make this more
43:57healthy and to follow the best practices
43:58we're going to go to the relationship
43:59over here. So either double click on it
44:01or actually we can find it on the right
44:03side. In the properties we see now
44:05informations about the relationship.
44:06Same thing but is like more compacted.
44:08Now the cross filter going to be single
44:11and then apply changes. Now this looks
44:13perfect. The filter direction is from
44:15the dimension to the fact. And I can
44:18tell you if your model looks like this
44:20then you are following the best
44:21practices. On the dimension side we have
44:23the cardality one. On the fact side we
44:26have the star. So it is one to many. And
44:28the direction is from dimension to
44:30facts. Once I see this then I'm going to
44:32be happy in the model. So this is how
44:34you create a relationship between those
44:35two tables. And of course you can see
44:37the relationship on the right side over
44:39here under the group relationships. We
44:42have the new one or you go to manage
44:44relationships over here. and you're
44:45going to have like a nice overview of
44:47all relations in your data model. All
44:49right. Now, before we continue, another
44:51thing that I usually do once I import a
44:53new data, I check whether the ids has
44:55this icon. Like for example, here the
44:57order ID has like an aggregation sign.
45:00Same thing for the product ID, the store
45:02ID. Now this icons means PowerBI can go
45:05and summarize aggregate this column. And
45:08usually we aggregate numbers, right? But
45:10there is only one case where it doesn't
45:11make sense and that is for the ids. We
45:14don't go and find the average of ids or
45:16go and summarize all the values or
45:18something like that. Let me just show
45:20you what I mean. If you go to the report
45:22view and let's go to the fact sales then
45:24grab the order ID over here. So what
45:27this tells us the sum of order ID 10k
45:30but this is not useful information. The
45:32only thing that we want to do with the
45:33order ID is to count it. How many orders
45:36do we have? But we don't need the sum or
45:38the average. And this auto aggregation
45:40is really annoying thing because
45:41sometimes I would like to see the row
45:43data in the front end and always Barbie
45:45I go and aggregate it. That's why what I
45:47do I go to those ids on the facts or in
45:50the dimensions. I click on it and then
45:52go over here and say don't summarize. So
45:55don't do that. And with that PowerBI
45:57will not do that anymore. So let's go
45:59and try. I'm going to go and drag the
46:01order ID in the middle. Look at this.
46:02Now I see the IDs and this is perfect.
46:05So usually I go and do this for all ids
46:08in the facts so that I don't have this
46:10issue. It is again luxury thing that we
46:12do but it is important because usually
46:14as we are building the data model we
46:16have always to go to the report view and
46:18verify whether everything is correct.
46:20All right. So now what we're going to do
46:21next is that as we do data modeling we
46:24would like always to understand the
46:25behavior on the reports. So we test
46:27things in order to understand whether
46:29this is exactly what we expect. And I
46:31usually do it like this. After doing few
46:33things in the model, I go back to the
46:35report view and start testing. Like for
46:37example, I'm going to go and add one
46:38visual only for the facts and another
46:41one only for the customers. So here for
46:43example, I go usually with the table and
46:45I'm going to go and add an order ID,
46:47maybe a customer ID, line total. So only
46:50few columns and then I add a card for
46:53the fact table. So the total sales and
46:55after that for each table I make like an
46:57aggregation in order to see how things
46:59change like the total sales. So same
47:02thing I'm going to go and duplicate it
47:03but this time it's going to be only from
47:06the customer dimension. So the customer
47:08ID first name last name. So that's it.
47:11And as well a card this time it's going
47:13to be the total number of customers. As
47:15you can see it's like I'm building the
47:16model again but on the front end. Now
47:19let's go and test the filtering. Right.
47:21So now if you just click one of those
47:22values you're going to see the table
47:24customer is filtering actually the fact
47:27table. Now it's all about the customer
47:29ID one. Same thing if I click on Anna,
47:32I'm filtering the whole fact table. So
47:34that means anything that I'm filtering
47:36on the right side, it has a direct
47:38effect on the facts. Now let's deselect
47:40and do the opposite. We start selecting
47:42things from the facts and see whether it
47:44has an effect on the customers. Like for
47:46example, the first order. If I click on
47:48it, look at this. Nothing happened to
47:50the right side. I still have five
47:52customers. So it doesn't matter what I'm
47:54like filtering from the fact it has no
47:56effect on the right side. So that's
47:58because it is in one direction. Now
48:00let's go and test the both. If I go over
48:02here and select the relationship say
48:05both apply. Now let's test the report.
48:09It is already filtering the customers.
48:11So now since I'm filtering on the order
48:13ID 8 has the customer three. This has an
48:16effect on the total number of customers
48:18and the visual. And same thing if I go
48:20and start like clicking around it has a
48:23full the filter is moving from the fact
48:26back to the dimension. And of course
48:27this is not a good thing. Now of course
48:29you might say why it's not good. It is
48:31correct. Like here we are just selecting
48:33or filtering one order. It is good
48:35information that we have only one
48:37customer for this one order. Well this
48:39is okay if you have one fact one
48:40dimension. But once you start getting a
48:43lot of dimensions they going to start
48:45filtering each others and it's going to
48:47be full cows. Of course to understand
48:48this effect we have to go and add
48:50another dimension. And this is exactly
48:53what we're going to do now. We're going
48:54to go and extend the data model with a
48:56new dimension. So let's go and get my
48:58friends the dimension products. So let's
49:01go and load it. Then say load. Now we
49:04got a new dimension. It is here the
49:06dimension product. So I will just make
49:08the fact in the middle surrounded with
49:10the dimensions. But now why PowerBI
49:13didn't create a relationship between
49:15those two tables. Now let's go and
49:17create the relationship by drag and drop
49:19the product ID on the product ID. And
49:21now we can see here we have an issue.
49:23This relationship has cardinality many
49:25to many. So here PowerBI is saying there
49:27is something issue with the product ID.
49:29Now by looking to the examples you can
49:31see one two three like we cannot see any
49:33issue here. At the end it is just a
49:35preview like only few rows. But our goal
49:38is of course to connect it with the
49:39product ID. Now let's go and try to
49:41force it to be one to many like we have
49:44done for the customer. So one from the
49:46product and many to the facts. It says
49:49no. It even doesn't allow us to save it.
49:51It says the product ID in the dimension
49:53product contain duplicates the value
49:56six. So this is not really unique my
49:58friends. So what we're going to do we're
50:00going to switch it back to menu to many
50:01in order for PowerBI to allow us to save
50:03it. And then we have to dig into this
50:06issue. Now let's go and save this. Look
50:08at this. Between the products and the
50:10facts we have many to many. And again
50:13this is not healthy. We would like it to
50:15be like the customers one to many. So
50:17now let's go and check the issue with
50:19the products. And we can do that by
50:21going to the table view. So here we have
50:23the customers. Let's go to the products.
50:25It says the product ID6. And yes, we
50:27have duplicates. The same row exists
50:30twice. And this makes of course no
50:32sense. And that's why PowerBI was not
50:34happy about this because it breaks the
50:37rule. So my friends PowerBI was good
50:40enough to detect the issues and it
50:42doesn't go and connect things randomly.
50:45If it sees something wrong, it will not
50:46go and create a relationship. Now, how
50:49we going to fix this? It is very simple.
50:51We're going to go to the Power Query and
50:52clean the data before moduling it. So,
50:54we're going to go to transform data.
50:56Then go inside and make sure to check
50:58the dimension products. And what we have
51:00to do is to just remove duplicates over
51:03here on the repon. Remove duplicates.
51:05And now look at this. The product ID 6
51:07is only once inside our data. And all
51:10the values are now unique. Actually,
51:12that's it. It should fix the issue. So,
51:14let's go and save it. Now let's go and
51:16try to change the cardality again. So
51:19let's try the one to many. And yes, we
51:22are not getting now any more issues.
51:23PowerBI is happy. The product ID is
51:26clean, unique, pointing of course to
51:28many in the fact. Let's go and save it.
51:30So now things looks way better. The
51:32dimension has one, the fact has many.
51:34Same things for the customers. Now of
51:36course the filter direction, both of
51:38them are not that good. Just going to go
51:40and switch it to single. Same thing for
51:42the customer. Now look at this. This is
51:44a perfect data model. It's very
51:46satisfying. All the filters are pointing
51:48from the dimensions to the middle and we
51:51start seeing the stars around the facts
51:53and everything else with a one
51:55cardality. So we have to keep pushing to
51:57get exactly this picture. And of course
51:59the next step is just the luxury things.
52:01You might go and add some maybe metadata
52:04by defining the key. So here it is the
52:06product ID in order to have this small
52:08icon. And now since we have multiple
52:10dimensions, I can show you exactly why
52:13it is bad to have both directions as a
52:15filter between the dimensions and facts.
52:17So again, I'm going to go and switch
52:19them quickly. So here both as well. And
52:21as usual, in order to understand it,
52:23we're going to go to the report level.
52:24And now since we have a third table, I'm
52:26going to go and just add it quickly. So
52:28let me just make those things small in
52:31order to test and fit the products. So
52:34this is our fact, this is the customer
52:36and we can go and add now the products
52:38over here. So I'm going to go and get
52:40rid of those columns and only pick
52:42things from the products. So the product
52:44ID, name, and I'm going to say that's
52:46it. And the last thing I'm going to go
52:48and add card for it. So this time we're
52:50going to go and count the total number
52:52of products. So now those are the
52:54totals. Six products, five customers,
52:57and around 6K of sales. All right. So
52:59now let's start filtering. And we're
53:01going to do that by filtering the
53:03products. Like for example, I would like
53:05to see the total sales for the product
53:07number one. So I can see here we have
53:082K. But look at this. The customers get
53:12filtered as well. Now I can see only two
53:15customers. And here of course comes the
53:16big question. Should we as well filter
53:19the customers based on the products?
53:22Well, I can tell you most of the cases
53:24we don't need this effect. It is too
53:26extreme to filter as well another
53:28dimension like the customers because
53:30usually this is how the users works on
53:32dashboard. They have the dimensions in
53:34order to filter the major event, the
53:37major measure that we have in the report
53:40which is the total sales. And now if the
53:42filter is filtering another filter, they
53:45going to lose immediately the big
53:47picture. They're going to lose
53:48immediately the other possibilities that
53:50we have in the dashboards. So they're
53:51going to look to the right side and say,
53:52"Oh, we have only two customers." But
53:55they will not think about yeah I'm
53:56seeing two customers because I'm
53:58filtering using the products. This is
54:01too deep and too much. Same thing if
54:03they go to another products they're
54:04going to see wa we have only one
54:06customer in our business. So they will
54:08not as well think inside your data model
54:10and they know ah everything is connected
54:12and so on. They usually use the
54:14dimensions to filter one thing and
54:17imagine you have a lot of dimensions
54:19this going to be catastrophic. filtering
54:21only one small thing going to filter
54:23deeply the whole dashboard and of course
54:25you're going to get the same effect if
54:26you go to the customers over here we go
54:28and use for example Anna they're going
54:30to see okay this is the total sales of
54:32Anna but it is filtering as well the
54:35products now what is way better than
54:37this what we're going to do we're going
54:38to go back to the model and yeah we're
54:40going to go and change things back to
54:42single same thing over here now let's go
54:44back to our reports so now let's try
54:47again we're going to go and select the
54:49product one And now yes I see the total
54:52sales for this product but look at this
54:54the customers going to stay as it is. So
54:56the effect it is not that overwhelming
54:58for the users. Now if they go to the
55:01customers and they say okay let's go and
55:03filter Anna only the fact table going to
55:05change. We still see the six products on
55:08the left side. So the effect is not that
55:10big. So my friends as a default if
55:12you're using star schema don't go and
55:13use the both directions for everything.
55:16This going to be very overwhelming and
55:18very deep for users. Usually all what
55:20they do they want to do analyzes on one
55:23main measure like for example the total
55:25sales and they need your dimensions for
55:27filtering. But if the dimensions starts
55:29filtering each others they going to lose
55:31quickly the big picture of the dashboard
55:33and it's going to make filtering one
55:35thing like an overreaction in the report
55:37by wiping out all other data. That's why
55:40as a default use one single direction
55:43but in extreme cases or if the users
55:45really want to see this effect in some
55:47reports then go and use both directions
55:50but limit it between maybe two tables
55:52only. So this is the best practices for
55:54filter directions in PowerBI. And now
55:56you know why.
55:57>> Thank you. I got it. Now I know. [music]
56:05>> Okay. So now let's go and add another
56:07dimension to our data model. This time
56:09we're going to go and pick the store.
56:11So, dimension store. All right. It is
56:13very cute, very small data model. Let's
56:15go and load it. All right. So, we got it
56:17over here. Let's go and grab it near to
56:19the fact and bring the customers above
56:22something like this. First of all, we
56:24have something good. So, the
56:25relationship between the store and the
56:27fact is one to many. And the filter is
56:30pointing to the right direction. So, if
56:32I go over here, I'm going to say okay,
56:34it is using the store ID as well in the
56:36dimension. So everything looks nice but
56:39now we have this weird thing. We have
56:41like a relationship between the
56:42dimension store and the dimension
56:44customer and it is using this dotted
56:47relationship. It's not the straight line
56:49that we used to see between the fact and
56:51the dimension. So what is going on here?
56:53Let's go and open it. Aha. PowerBI is
56:55able to build a relationship between the
56:57store and the customers using the city.
57:00Well, this is very usual thing that you
57:01have the same information in multiple
57:03dimensions and probably I thought yeah
57:05why not to go and connect them using the
57:07city. This looks nice but it is not a
57:10normal relationship right it was dotted
57:12and you can see over here it says make
57:14this relationship active and it is
57:16deactivated that means we have in
57:18powerbi two type of relationships active
57:21relationship and inactive relationship
57:23so here it is inactive let's go and
57:25activate it and see what can happen say
57:27no there are ambiguous path between fact
57:30sales dimension store fact sales
57:32dimension customers dimension store and
57:35fact sales dimension store I know a lot
57:38things here but simply PowerBI is just
57:40whining about saying we have a loop
57:43relationship so the path is like cycling
57:45between the tables and I cannot go and
57:48activate it otherwise it's going to be
57:50for me unclear how I'm going to answer
57:52your questions and you can see it over
57:54here the loop the cycle so customers and
57:57fact then we have fact and I mission
57:59store and then back so we have a
58:01relationship between the customers and
58:03the fact another one between the store
58:05and the fact and a third one which is
58:07closing the cycle between the store and
58:10the customer and this is exactly the
58:12cycle. Now of course the big question is
58:14why PowerBI doesn't allow that. So what
58:17is the big issue? All right let's have
58:19this very simple example. We have
58:20exactly the same setup here. We have the
58:22customers the stores and they are both
58:24connected to the sales but at the same
58:26time they are connected to each others
58:28using the city. Now let's say that I
58:29would like to see the total sales for
58:31one of our customers. So if I go and
58:33filter for example Maria, I expect that
58:36the filter going to go to the fact sales
58:37like this and the total of sales going
58:39to be 1,500. So this is our
58:41expectations. But the thing is PowerBI
58:43has another path to the sales. So it can
58:45go to the store and then from the store
58:47it can go to the sales. Now because of
58:49this path, PowerBI going to filter the
58:51sales differently and get 2K for this
58:54customer. Now, of course, this is going
58:55to be a big issue for PowerBI because it
58:58has two different results for the same
59:00action and PowerBI cannot decide for us
59:03which one is the correct one and this is
59:05exactly what we mean with ambiguous.
59:07PowerBI don't know the correct path
59:10between two tables. That's why it
59:12prevent us from having this scenario and
59:15PowerBI force us to decide only on one
59:17direction, one path between two tables.
59:20Now, of course, we have the same
59:21scenario if you look to the stores. If
59:23you say, you know what, let's go and
59:24filter by the downtown. It's going to go
59:26to the sales over here and get 2K. But
59:28as well, it has another path through the
59:31customers. So, it can go to the
59:32customers and from customers go to the
59:34sales. Same scenario, same issue. So,
59:36this is ambiguous and PowerBI will not
59:38allow that. Instead, PowerBI says, okay,
59:40I'm going to leave for you the
59:42relationship between those two tables,
59:43but it is inactive. I will not go and
59:46use it. I'm going to go and use only the
59:48active relationships. So if you go for
59:50example and filter Maria PowerBI has one
59:53clear active path between those two
59:55tables and it going to get only one
59:57results. Same thing if you go to the
59:58downtown over here it has as well one
1:00:01clear path. This is the loop and why
1:00:03PowerBI was not allowing us to have
1:00:05ambiguous path different options for the
1:00:08same action. Okay. So with this now we
1:00:10have understanded why PowerBI did make
1:00:13this inactive between the customers and
1:00:15store and now let's say that okay you
1:00:17decided that this is the correct path
1:00:19between those two tables and you want to
1:00:21activate it. Now what you can do you can
1:00:23go for example yeah it's wrong but you
1:00:25can go between the customers and sales
1:00:27open the relationship and make it
1:00:29inactive. So now it is dotted and then
1:00:32go back to the link between customers
1:00:34and stores and make it active which is
1:00:37of course again not correct but this is
1:00:39the way on how to switch between active
1:00:42and inactive relationships. Now of
1:00:43course I'm going to get it back because
1:00:45this is not correct. So now what we're
1:00:47going to do with this inactive
1:00:49relationship you have here my friend two
1:00:51options. Either you're going to say you
1:00:52know what I would never use this
1:00:54relationship between those two tables.
1:00:56It is totally wrong. So that means you
1:00:58go and just delete this relationship.
1:01:01But the other option is that we still
1:01:03actually my friends could use this
1:01:05relationship inside our reports. Now the
1:01:07big question is how you going to use the
1:01:09inactive and at the same time active
1:01:11relationship in your projects. Now the
1:01:13thing is the active relationship going
1:01:15to be always used automatically from
1:01:17PowerBI. So if you are using any
1:01:19calculations and filtering in the
1:01:21charts, PowerBI always going to use the
1:01:23active relationship. But if you have
1:01:26special cases where you say, you know
1:01:27what, I need this inactive relationship.
1:01:30You can go and use ducks. There is a du
1:01:33function called use relationship. It's
1:01:35going to turns on an inactive
1:01:37relationship specifically for one
1:01:39calculation. So that the measure the
1:01:42calculation going to go through an
1:01:44inactive relationship instead of the
1:01:46active one. So that means in one report
1:01:48you can let PowerBI use all time the
1:01:51active relationship. But for specific
1:01:53cases, you're going to create an extra
1:01:55calculation, an extra measure that use
1:01:58the inactive relationship. So that's why
1:02:00my friends, if you have an inactive
1:02:02relationship between two tables, either
1:02:04drop it or you're going to use DAX in
1:02:06order to use this path for specific
1:02:08calculations. So those are our two
1:02:10options. But now I'm going to say for
1:02:12this scenario I don't need it. So I'm
1:02:14going to go and delete it. So yes, now
1:02:16again everything looks nice. All the
1:02:19dimensions are pointing toward the
1:02:21facts. The cardalities around the fact
1:02:24are the many, the star and the cardality
1:02:26on the dimensions are one. So far
1:02:28everything is healthy like plants.
1:02:34All right, let's [music] keep going and
1:02:36connect more things to our data model.
1:02:38Now this time we're going to go and
1:02:39connect another customer dimension, the
1:02:42loyalty. So let's go and do that. Again,
1:02:44a very simple dimension. All right. Now
1:02:46you can see that PowerBI did connect our
1:02:49new dimension to another dimension to
1:02:51the customers using a onetoone
1:02:54relationship. So let's go and open it.
1:02:56It says okay we are connecting using the
1:02:58customer ID which is correct. And it
1:03:00says as well the customer ID in the new
1:03:03dimension is unique. So we don't have
1:03:05here duplicates. Of course same thing in
1:03:07our original customer dimension.
1:03:09Everything is unique here. And that's
1:03:11why PowerBI did decide to go with the
1:03:13one to one relationship. So now the
1:03:15question is should we panic? Should we
1:03:18leave it like this? This onetoone
1:03:20relationships are actually a harmless
1:03:22relationship and you will not do a lot
1:03:24of mistakes. If you leave it as it is,
1:03:27it will not go and lead to wrong numbers
1:03:29or any issues in the front end. The only
1:03:32side effect that you have to think of is
1:03:34that PowerBI has to go and join those
1:03:36two tables in order to present the
1:03:38results. So it is an extra effort for
1:03:40PowerBI to travel from one table to
1:03:43another. but it has no effect on the
1:03:46correctness on the report. And here you
1:03:48have actually two options on how to do
1:03:49it. Either you're going to leave it as
1:03:51it is and here you could say you know
1:03:52what it makes no sense to have them like
1:03:55splitted. I can go and join those two
1:03:57tables together and merge them into one
1:03:59dimension. So with that I reduce the
1:04:02number of travels for PowerBI and maybe
1:04:04speed up things a little bit. So very
1:04:06famous solution is to just go and merge
1:04:08them. Don't leave them separated. So for
1:04:10example here the names and the tier
1:04:12going to go together inside the
1:04:13dimension customers. So I'm going to say
1:04:15this is the default solution that I
1:04:17usually do in my projects. But now in
1:04:19projects sometimes we decide to split
1:04:22things using the onetoone relationships
1:04:24for some reasons like for example your
1:04:27dimension customer over the time gets
1:04:29bigger and bigger. you have a lot of
1:04:30columns about the customers and in order
1:04:32for you to manage things easily you say
1:04:35you know what I'm going to put all the
1:04:36descriptions about the customers in one
1:04:38table and then group another set of
1:04:40columns in another table maybe some date
1:04:43informations about the last login the
1:04:46first sign up so they are like group of
1:04:48columns about the same topic and then
1:04:50connect those two tables using onetoone
1:04:52relationship or it could be for security
1:04:55reasons where you split some sensitive
1:04:57data into another table so that you
1:04:59could hide this type of informations and
1:05:02you show only the informations that are
1:05:04allowed. So that's we have like split
1:05:05between insensitive and sensitive
1:05:07informations and we connect them using
1:05:09the one to one relationships. But again
1:05:11those are special edge cases where we
1:05:14have to go and split things. But as a
1:05:16default if you see one to one
1:05:17relationship you can go and merge them
1:05:19into one dimension. And this is exactly
1:05:21what we're going to do now. We're going
1:05:22to go and merge those two tables
1:05:24together. And how we going to do it?
1:05:26We're going to use the power query. So
1:05:28we're going to go to transform data and
1:05:30I'm going to go and make this bigger. So
1:05:32let's go to the dimension customer and
1:05:34then we're going to go and say merge
1:05:35queries. This is the first table. We
1:05:38need the second table the loyalty. Now
1:05:40we have both of them. Of course we have
1:05:42to connect them using the customer ID.
1:05:44And here be careful. You have to decide
1:05:47about the join. I usually always use
1:05:49lift join. Inner join is dangerous
1:05:51because it's going to go and filter the
1:05:53data. And by looking to this I can say
1:05:56the customer has more weight than the
1:05:58customer loyalty because it has the main
1:06:01informations about the customers and I
1:06:03would not like to lose anything. So you
1:06:06start with the main table and then the
1:06:08secondary table and make sure to use the
1:06:10lift join. So actually that's it. Let's
1:06:12go and hit okay. Now we have it here as
1:06:14a table. The next step we're going to go
1:06:16and expand it. So we have to select the
1:06:18columns that we need. Well, we need the
1:06:19tiers. We need the points, but we don't
1:06:21need the customer ID because we have it
1:06:23already as primary key. So, actually,
1:06:25that's it. Let's hit okay. Now, look at
1:06:27this. We have the tiers and as well the
1:06:29points. The next step, I'm going to go
1:06:30and just rename them to be more
1:06:32friendly. So, we're going to go over
1:06:34here and say rename. This going to be
1:06:36tier and the other one going to be
1:06:38points. So, this looks really nice. Now,
1:06:42in our model, we don't need anymore this
1:06:44dimension because it is already merged
1:06:46into the customers. So what we can do we
1:06:48can right click on it and deactivate
1:06:51enable load. Yes. Continue. So with that
1:06:53we deactivate this table to be loaded
1:06:55inside our model. And with that we have
1:06:58everything inside the dimension
1:06:59customers. So this is how we merge. It
1:07:01is like preparation step before the data
1:07:04modeling. So I'm going to go and close
1:07:06it. Make sure to save it as well. Now
1:07:08look at this. The dimension did
1:07:10disappear but we have its informations
1:07:13inside the customers. Again, it is not
1:07:15that dramatic if you just leave it, but
1:07:18it is more clean to bring everything
1:07:20into one dimension to have the shape of
1:07:23the star schema and you don't have to
1:07:25deal with a lot of tables as you are
1:07:26building your reports. So, the dimension
1:07:29customer did absorb another dimension
1:07:32that has the relationship one to one.
1:07:38Now, my [music] friends, dimensions
1:07:39don't always live in dimension tables.
1:07:42Sometimes they sneak into fact tables as
1:07:45descriptive informations. Our job is to
1:07:47spot those sneaky dimensions inside the
1:07:50facts and pull them out into real
1:07:53dimensions. So let's go and find them.
1:07:55>> Well, let's see.
1:07:56>> So now what we're going to do, we're
1:07:57going to go and inspect our fact sales
1:08:00to see whether we have dimensions inside
1:08:02this table. Let's go to the table view
1:08:04in order to inspect. Now, as we learned
1:08:06at the start, fact tables going to
1:08:08contain a lot of ids, dates, and
1:08:11numbers. So let's see whether this is
1:08:13the case. Yes, we have a lot of ids at
1:08:15the starts. We have the dates. We have
1:08:17the numbers but as well we have a lot of
1:08:20text over here. This is a big alarm. Now
1:08:23usually text and string columns they
1:08:26used in order to describe something and
1:08:28this is exactly the job of the
1:08:30dimensions not the fact. So that means
1:08:33my friends all those columns are
1:08:35actually good candidates for dimensions
1:08:37and it makes no sense to leave them
1:08:40inside our facts. And this happens a lot
1:08:42in real projects where you think okay
1:08:44this table has an event it is
1:08:47transactions we have dates and stuff but
1:08:49still it has as well same time a lot of
1:08:52description so that's why it's time to
1:08:54extract all those dimensions from the
1:08:56facts and make our fact clean and as
1:08:58well we got to get some useful
1:09:00dimensions so of course the question is
1:09:02what we going to do with all those
1:09:03informations do we have here one
1:09:04dimension two three well this depends on
1:09:07the relationship between those columns
1:09:10so now let's see What do we have over
1:09:11here? We have promotion code. So it is a
1:09:13code that is used maybe for some sales.
1:09:16We have the order channel in store,
1:09:18online, phone, the order status. We have
1:09:21information about the payment methods,
1:09:23credit card, PayPal and so on. The card
1:09:25network and the card type. Now your main
1:09:28job is to spot the columns that relate
1:09:31to each others. And from looking at this
1:09:33I can say the payment methods, the card
1:09:36network and the card type. All those
1:09:38three columns are actually related to
1:09:40each others and they are perfect in
1:09:42order to make a dimension and this
1:09:44dimension of course going to be the
1:09:46payment dimension the order status and
1:09:48the order channel promotion code they
1:09:50are not related to the payment methods.
1:09:53So that's why I would not go and connect
1:09:55those stuff to our new dimension. Now of
1:09:58course the big question is how we going
1:10:00to split? Well again that means we need
1:10:02to transform the data and for that we
1:10:04have the power query. So let's go there.
1:10:07We don't use power query only to merge
1:10:10things as well to split things. So now
1:10:12there are like multiple ways on how to
1:10:14do it. I usually do it like this. Now
1:10:16the thing is we don't have the dimension
1:10:18yet inside our data model. So either
1:10:20you're going to go and get it from the
1:10:21CSV file or you can go and duplicate the
1:10:24fact sales which I think it is way
1:10:26easier than just like loading the data
1:10:29again. So I'm going to go and duplicate
1:10:30the fact sales and then I'm going to
1:10:32give it the name dimension. It's going
1:10:34to be the payments. So this is duplicate
1:10:37from our fact sales and of course I
1:10:39don't need all those columns right for
1:10:41the payments I just need the last three.
1:10:43So what we do we're going to go and
1:10:45select everything and say go and remove
1:10:47the columns with that I got only the
1:10:49three columns that I need for the
1:10:51payments. Now comes something really
1:10:53important to do those informations
1:10:54usually going to have a lot of
1:10:56duplicates. So look at the first two
1:10:58rows they are just the same. So what
1:10:59we're going to do we're going to go over
1:11:00here and say remove the duplicates. So
1:11:03with that each row is unique and this is
1:11:06very important for the dimensions. In
1:11:07order to get one to many relationship
1:11:10with the facts. Now look at this what we
1:11:13are missing. We need the ID for the
1:11:15dimension in order to connect it with
1:11:17the facts. We don't connect things using
1:11:19text. And in order to generate an ID
1:11:21it's going to be very simple. We're
1:11:22going to go and add a column. And then
1:11:24we add an index column. It's going to be
1:11:26a sequence of number for each row. So
1:11:28we're going to go over here and say okay
1:11:30start from one. And with that we have a
1:11:32nice ID. I'll just going to go bring it
1:11:34to the start. And then what we're going
1:11:35to do, we're going to go and rename it
1:11:37like all others. So we're going to say
1:11:39payment ID. Now look at this. If you go
1:11:41and compare it to any other dimensions
1:11:43that we have, this is exactly a nice
1:11:45dimension that we could use in our model
1:11:47to do filtering and all other stuff.
1:11:49Again, very important that we have an ID
1:11:51and the data is unique. We don't need
1:11:54many to many relationships. So this is
1:11:56how you build a dimension out of facts.
1:11:59But we are not done yet. We have to
1:12:01connect now the dimension to our facts.
1:12:04So if you go to the fact sales and you
1:12:06can see that in the fact we are always
1:12:07connecting the dimensions using ids and
1:12:10we need the same thing actually for the
1:12:12payments. So actually we don't need all
1:12:14those informations in the fact we just
1:12:16need the payment ID. Now how we going to
1:12:18do it? This is going to sounds a little
1:12:19bit weird but we have to merge now the
1:12:22fact sales with the dimension payments
1:12:25just in order to get the payment ID. I
1:12:27know this sounds strange that you are
1:12:29splitting and then combining but only
1:12:31with this you're going to get a clean
1:12:33model later in the next layers so that
1:12:35it is easy to connect and as well easy
1:12:37to use later in the reports. So now what
1:12:40you going to do we have to go and merge
1:12:42things. So the first is going to be the
1:12:43fact sales. The second is going to be
1:12:45the new dimension the payments. Of
1:12:48course the question is how you going to
1:12:49go and connect those two tables. We're
1:12:51going to use the three columns. So we
1:12:54need the payment methods the card
1:12:56network and the card type. Same thing
1:12:58from the first table from the fact we're
1:13:00going to use those three columns. So we
1:13:02have to connect all those columns in
1:13:03order to get the payment ID. And of
1:13:05course it's going to work because they
1:13:06are actually the same columns. Again
1:13:08we're going to use only the left outer
1:13:10join. This looks good. Let's go and hit
1:13:12okay. Now we got the whole dimension in
1:13:14the facts. We don't need that. We just
1:13:16need only one thing, the payment ID. So
1:13:18let's go and select it and say okay.
1:13:20Look at this. We have it already here.
1:13:22Let's go and rename it. So this going to
1:13:24be payment ID. And what we're going to
1:13:26do, we're going to go and put it side by
1:13:28side by all other columns, right? It
1:13:30deserve now its place. So I'm just going
1:13:32to go and put it at the end. Now look at
1:13:34this. It looks clean. We have customer
1:13:36ID, product ID, store ID, and the
1:13:38payment ID. The last step that we have
1:13:40to do is to clean up. So we don't need
1:13:43anymore those three columns. I'm going
1:13:44to go and delete them. So right click
1:13:47and remove. So it's going to be over
1:13:49here. Now look at this. Our fact is
1:13:51clean. And with that I'm going to say we
1:13:53successfully extracted a dimension out
1:13:56of facts. We have of course other
1:13:57informations but we have now less
1:13:59descriptions in the fact. Now the last
1:14:01step. We have of course still some text
1:14:03but we're going to do that as a second
1:14:04step. Now let's go and save and close.
1:14:07Then apply all the changes. All right.
1:14:09So with this we got a new dimension our
1:14:11dim payments and as well PowerBI did
1:14:13connect it correctly. So things looks
1:14:15really nice. We have one to many
1:14:17relationship and as well the direction
1:14:19is correct. You can see we are getting
1:14:21the shape of the star right. If I just
1:14:23move things like this fact in the middle
1:14:25and surrounded by dimensions and
1:14:28everything is pointing to [music] our
1:14:29facts.
1:14:33So now of course we are not done yet. We
1:14:35still have some descriptive informations
1:14:37inside our facts. So the promotion code,
1:14:39the order channel and the order status.
1:14:42Now the big question is how we going to
1:14:44extract those informations. Do they
1:14:46relate to each others? Now if you look
1:14:48closely, you would not have the feeling
1:14:50those informations actually relate to
1:14:52each other. So the promotion code has
1:14:54nothing to do with the order channel.
1:14:55Same thing for the order status. So they
1:14:58are actually three different piece of
1:15:00informations in the fact and all three
1:15:02together will not form a dimension and
1:15:05they are not related enough to go and
1:15:07bring them together into one dimension.
1:15:09So now how are we going to solve this?
1:15:10You're going to find in some projects
1:15:12that people say you know what for each
1:15:14column we're going to go and create a
1:15:15dimension. So they start splitting each
1:15:17of those columns inside the fact and
1:15:19build a dimension and create for it an
1:15:21ID. And over the time you're going to
1:15:23find in their star schema a lot of many
1:15:25dimensions around the facts. One
1:15:27dimension, one column and so on. And I
1:15:30think this is really overreacting. So
1:15:32this going to be a lot of work if you
1:15:34have a lot of columns inside the facts.
1:15:36And this is usually the thing in
1:15:37projects. Fact tables are usually big
1:15:39tables and they have a lot of status
1:15:41columns. So how we going to solve this?
1:15:43We don't want it inside the fact, right?
1:15:45Well, the solution for this something
1:15:47called a junk dimension. A junk
1:15:49dimension. It is a small dimension that
1:15:51bundles several unrelated low-level
1:15:55flags. So all those junk or garbage,
1:15:57let's say columns that are actually not
1:15:59related to each others and take space
1:16:01inside the facts, we're going to go and
1:16:03bundle them into a junk table, a junk
1:16:06dimension. And each time you find a new
1:16:08flag that is unrelated for anything,
1:16:11you're going to go and put it in this
1:16:12junk dimension. So this is the best
1:16:14practices in dimensional modeling or
1:16:17building a star schema to put all
1:16:19unrelated informations into one
1:16:21dimension. So now let's go and see how
1:16:23we're going to do it. So that means my
1:16:24friends all those three columns going to
1:16:26go into a new dimension. So how we going
1:16:28to do it? So same thing as we done with
1:16:30the payments. We're going to go to the
1:16:31power query. We're going to repeat the
1:16:33same steps. So we're going to go and
1:16:34duplicate the fact table and we're going
1:16:37to give it a name. So here the official
1:16:38name people going to go and call it like
1:16:40dimension junk or I usually go and use a
1:16:43friendly name. So dimension flags well
1:16:46it doesn't matter which one you're going
1:16:47to use. It is just the idea. So what
1:16:49we're going to do same thing we don't
1:16:50need all those informations. We just
1:16:52need the unrelated columns. So remove
1:16:55the columns. And now comes something
1:16:57really important to do with that. So we
1:16:59have here a lot of nulls inside the junk
1:17:02dimension and we have to handle it
1:17:04otherwise the join or the merge with the
1:17:06fact sales will not work. So this is
1:17:08very critical. What we're going to do
1:17:10we're going to go and right click on it
1:17:11and say replace values. So instead of
1:17:14nulls we're going to say unknown or you
1:17:16can say okay. So any string value that
1:17:18you want like unknown or not available.
1:17:20Let's go and hit okay. So this looks way
1:17:22better. Now only then we're going to go
1:17:24and remove the duplicate. So, make sure
1:17:26to select everything and then go over
1:17:28here, remove duplicates. Nice. Now,
1:17:30things are unique. The last step, we're
1:17:32going to go and add a column. So, we're
1:17:34going to add an index that starts from
1:17:36one and then move it to the starts.
1:17:38We're going to call you the flag ID. So,
1:17:41this is our junk dimension. Everything
1:17:43is ready. We have a lot of unnecessary
1:17:45informations to gather, but it is not
1:17:47that big. Now, what we're going to do,
1:17:48we're going to go to the fact sales and
1:17:50connect the junk to the facts. But we
1:17:53have to be very careful since we're
1:17:55going to use all those three columns in
1:17:56the joins. They must match. And now the
1:17:59promotion code will not work, right?
1:18:01Because here we have nulls and in the
1:18:02junk we have the unknown. That's why we
1:18:04have to handle this as well. So we're
1:18:06going to go and say okay, replace
1:18:08values. If you are null, then you are
1:18:10unknown. Let's hit okay. Nice. Now we're
1:18:12going to go and connect things using the
1:18:15merge. So we're going to go to the right
1:18:16side and select those three columns and
1:18:19connect it to our junk dimension. Same
1:18:22thing we just need the flag ID in the
1:18:24dimension, right? So left join and let's
1:18:26hit okay. So with this we got the
1:18:28dimension as a table but all what we
1:18:30need is the flag ID. So I'm going to go
1:18:32and deselect everything and say flag ID.
1:18:35And look at this. It is working. So
1:18:37before I move it, I'm going to go and
1:18:39call it flag ID. And this junk as well
1:18:42deserve its place near the other
1:18:44dimensions. So I'm going to go and move
1:18:46it to the left side. But I'm going to go
1:18:48and put it at the end. So look at this.
1:18:50We have the flag ID, the junk, and on
1:18:53the right side, now we're going to go
1:18:54and remove all those stuff. So, right
1:18:56click on it, remove the columns. Now,
1:18:59look to our facts. We have over here a
1:19:02lot of ids. Then, we have the dates and
1:19:05we have numbers and all the junk now
1:19:07inside a junk dimension. So, this looks
1:19:10really clean. Now, let's go and close
1:19:12and save. Everything is loaded
1:19:14correctly. Let's go to the model. Now,
1:19:16we have our junk dimension. I'm going to
1:19:18put it here on top. and you go down and
1:19:21we have to make sure this is one to many
1:19:24relationship and the correct direction.
1:19:26So my friends, I'm really happy with the
1:19:28result so far. With this we have
1:19:30extracted a new two dimensions out of
1:19:32the fact. So we have a really solid
1:19:35dimension called the payment. All the
1:19:36columns relate to each others. But as
1:19:38well we have a lot of junk dimensions, a
1:19:41lot of unrelated flags. We bundled
1:19:43everything together into one dimension
1:19:45and connected back to the fact sales. So
1:19:48this is really amazing. I always keep my
1:19:50eyes on the directions of the filters,
1:19:51the cardalities just to make sure that
1:19:53we have a clean star schema in order to
1:19:56do analytics.
1:20:00All right friends, [music] so now we're
1:20:01going to speak about dimensions that
1:20:03they are kind of special. So they play
1:20:05different roles in different scenarios.
1:20:07That's why we call them the roleplaying
1:20:09dimensions.
1:20:10>> What um what do you think of role play?
1:20:13>> So now let's understand what this means.
1:20:15Okay. Okay. So now in order to
1:20:16understand it, we're going to have this
1:20:17very simple scenario where we have two
1:20:19dimensions. The salespersons and as well
1:20:21the managers. Both of them are connected
1:20:24to the fact sales. And now with this
1:20:26setup, I can go and find out the total
1:20:28sales for example per person. If I go to
1:20:30the Anna here, I can see that she made
1:20:32two orders and the total sales is around
1:20:351,500. Same thing if I go to other
1:20:37salesperson like Carlos Dana with that
1:20:40I'm able to find out the total sales and
1:20:42the total number of orders. But now if I
1:20:44go to the other dimension to the
1:20:46managers and I go and select the team of
1:20:48pop, I'm going to see the total number
1:20:50of orders and as well the total number
1:20:52of sales for the whole team and his team
1:20:54has only two persons. So the one and the
1:20:57two, so Anna and Carlos and same thing
1:20:59if I go and check the other manager, I'm
1:21:01going to see as well the total sales for
1:21:03that team. So far nothing is crazy,
1:21:05right? It is working. But now what could
1:21:07happen in real business that one of the
1:21:09managers like for example pop he can go
1:21:11and sometime close some sales. So that
1:21:14means if I look to pop as a salesperson
1:21:16I can go over here and see okay he
1:21:18closed two sales and I can see the total
1:21:20amount of sales. And if I look at pop as
1:21:23a manager I'm going to see now the total
1:21:25number of sales and orders for the whole
1:21:27team. So that means I can look at pop in
1:21:29two directions. One as a salesperson and
1:21:32another as a manager. So so far there is
1:21:34no big problem. But the issue is that
1:21:36the design of the data model. We have
1:21:38the same person in two dimensions. So
1:21:41that means we have double informations
1:21:42and this is not really good because for
1:21:44example now if maybe pop now change his
1:21:46name or address or whatever we have
1:21:48information inside the dimension. I have
1:21:50to go now and update both of the
1:21:52dimensions. Of course here it's a very
1:21:54simple example but over the time this
1:21:55can be really hard to maintain and to
1:21:58manage and as well this is a bad design
1:22:00if I make like a dimension for each
1:22:02role. Maybe next day I'm going to get
1:22:04like a new role like an account owner.
1:22:06Then we're going to have like a third
1:22:07dimension about the persons and then
1:22:09we're going to have maybe pop in all
1:22:11three dimensions. So that means you're
1:22:12going to lose control quickly if you
1:22:14start splitting the dimensions by the
1:22:16role. Especially if they are identical.
1:22:18You can see here we have the same
1:22:19columns. We have ID, name, team, ID,
1:22:22name, team. So they have the same
1:22:23schema. The only difference is the role.
1:22:25So now of course what is the solution?
1:22:27It is very obvious. We're going to go
1:22:28and bring all those dimensions that are
1:22:31identical and we're going to bring them
1:22:33into one dimension that going to play
1:22:35different roles. Okay. So now how it
1:22:37going to looks like in the modeling
1:22:39we're going to go and combine those two
1:22:41dimensions into one generic dimension
1:22:44called employee. So before it was
1:22:46splitted by the role but now we have it
1:22:48generic. And of course in the future if
1:22:50we had like an HR dimension we can go
1:22:52and always fit it inside the employee so
1:22:55that we have like single point of truth
1:22:57for people inside the company. Now of
1:22:59course each person going to be one row
1:23:01inside the dimension. And now if we do
1:23:03this of course we have to be careful not
1:23:05to lose information from the module.
1:23:07Like for example before we had a manager
1:23:09we had a salespersons. Those
1:23:11informations were the table name and
1:23:13they are of course important in the
1:23:15model. But now instead of having them in
1:23:17the names, we're going to go and put
1:23:19them as a values inside a new column
1:23:21called a ro. So that we have one
1:23:23complete dimension that is ready to be
1:23:26reused multiple times inside the facts.
1:23:28Now by looking to this you can see that
1:23:30our dimension going to play two roles
1:23:32right salesperson and manager. And for
1:23:35that we have to go and create two
1:23:37relationships. One for the salesperson
1:23:39and one for the manager. And of course
1:23:40in PowerBI we cannot have both of them
1:23:42active. one of them going to be active
1:23:44and the rest going to be inactive. So
1:23:46for now, for example, the saleserson
1:23:48going to be the active relationship and
1:23:50the manager going to be the inactive. So
1:23:52now let's have an example like before
1:23:53let's see the total sales for Anna. We
1:23:56can see we're still going to get the
1:23:57total sales the two orders. Same thing
1:23:59if I go to Carlos to Diana and as well
1:24:02to our manager Bob. So by looking at
1:24:04this our dimension now is playing the
1:24:06role of the salesperson. But now let's
1:24:08say that I would like now to switch the
1:24:10roles and I would like to see the total
1:24:12sales for the whole team for the
1:24:14managers. So that means we have to
1:24:16switch roles for this dimension and we
1:24:18can do that by using the other
1:24:20relationship the manager one. Of course
1:24:22we can do that using ducks. And now the
1:24:24dimension employee is playing this role.
1:24:26So if I go to pop I will not see the
1:24:28sales of pop. I'm going to see the sales
1:24:31of the whole team. Same thing if I go to
1:24:33the other manager I'm going to see as
1:24:35well the sales of the whole team. Now if
1:24:37I go to Anna, Carlos, Diana, I will not
1:24:40see anything because they are not
1:24:41managers. They don't have teams. So as
1:24:44you can see, I'm using the same table
1:24:45but from two different perspectives. One
1:24:48as a salesperson and another one as a
1:24:50manager. Now imagine you have a third
1:24:52role. All what you have to do is to go
1:24:54and create a new relationship between
1:24:57those two tables and you use that
1:24:59relationship in order to play another
1:25:01role for this dimension. So that's why
1:25:04my friends we call the dimension
1:25:06employee is roleplaying dimension. It is
1:25:09just a dimension that is playing
1:25:11different roles for the same fact table
1:25:14by using multiple relationship to it and
1:25:17only using one active relationship at a
1:25:20time and of course we switch roles by
1:25:22using ducks function called the user
1:25:24relationship. So this is what we mean
1:25:26with the role playinging dimension. Now
1:25:28let's go to PowerBI and see how we're
1:25:29going to do that. Okay. Okay. So, as
1:25:31usual, we're going to go and connect our
1:25:32dimensions. This time, we're going to go
1:25:34and get to the managers and as well the
1:25:37salespersons. So, this one over here.
1:25:41Load. Now, at the start, if you are not
1:25:43investigating about those new
1:25:45dimensions, you're going to see, okay,
1:25:46we have here a manager ID. So, I'm going
1:25:48to go and connect it with the fact.
1:25:50Looks good. One to many. So, I'm not
1:25:52hurting the star schema. Same thing for
1:25:54the salespersons. I have an ID for it in
1:25:57the fact. So I'm going to go and connect
1:25:58it as well. One to many. So at the start
1:26:02things looks healthy, right? But now if
1:26:04you go and investigate the data itself,
1:26:06we're going to see that the dimension
1:26:08manager has like an ID, name, email,
1:26:10team. And it going to look identical to
1:26:13the salesperson. We have as well ID,
1:26:15name, email, team, and so on. So this
1:26:18going to be like an indicator for you to
1:26:19say, you know what, let's go and make
1:26:21from this roleplaying dimension. So of
1:26:24course now the question is how we going
1:26:26to merge those two into one dimension.
1:26:28As usual we have to do transformations
1:26:31using power query. So let's go to
1:26:33transform data. Okay. So now we are
1:26:35inside and we see here the manager and
1:26:37the person. Of course in order to make a
1:26:39union between those two tables we have
1:26:42to stack them and for that we have the
1:26:44append queries. We are not merging. So
1:26:47we are just like stacking the tables on
1:26:50top of each others. But now if we do
1:26:51this immediately we're going to lose a
1:26:53very important information to understand
1:26:55where the data come from. So for example
1:26:58here is Anna is a saleserson or a
1:27:00manager. That's why before we append we
1:27:03have to go and add a column. That's
1:27:05going to help us to understand the
1:27:06source of each row. So in order to add a
1:27:09new column we're going to go over here
1:27:10and say add column. We're going to go
1:27:12and say custom column. So we're going to
1:27:14give it a name. It's going to be the
1:27:16role. And now we have to define what is
1:27:18the value for each row. This is going to
1:27:20be a static value. So we will not go and
1:27:22select any other available columns. It's
1:27:24going to be always a same static value
1:27:27the saleserson. So actually that's it.
1:27:29This is the column name and this is the
1:27:31static value. Let's go and say okay. And
1:27:33now with this we can see for each row we
1:27:35have the same value saleserson. Now we
1:27:37have to go and do the same thing for the
1:27:39managers so that we don't lose this
1:27:41important information. Again a custom
1:27:43column. It's going to be the role and a
1:27:45static value. So you're going to be the
1:27:47manager. That's it. Let's go and hit
1:27:50okay. Look at this. We have for each
1:27:51person now the role of manager. Okay. So
1:27:55with this we prepared the two dimensions
1:27:57now to get appended. I'm going to go to
1:27:59the saleserson then home and we're going
1:28:02to go and say append. So we are
1:28:03appending two tables. Append to the
1:28:06table manager. All right. Now look at
1:28:08this. Each person has the role of
1:28:11manager. So with this we have now the
1:28:13role for the two tables. But I think
1:28:15that we're going to do with that before
1:28:16we append. We have to make sure that all
1:28:18column names are the same. So here we
1:28:21have name, email, team, and so on. The
1:28:24only difference is actually the ID. So
1:28:26what we're going to do is that we're
1:28:27going to go and rename it. So instead of
1:28:30salesperson ID, it's going to be an ID.
1:28:32And same thing for the managers going to
1:28:35go and rename it to ID. So as you can
1:28:37see, I'm trying to bring those two
1:28:39tables to become really identical. only
1:28:41after preparation we go and merge the
1:28:44two dimensions into one. Now we are
1:28:46ready. So I'm going to go to the
1:28:47saleserson and from here I'm going to go
1:28:49and append the second table the manager.
1:28:52All right we are appending two tables.
1:28:53The second table going to be the
1:28:55dimension manager. Let's go and hit
1:28:57okay. Look at this. The main thing that
1:28:59we are not getting any nulls which is
1:29:01perfect. So we have here now 10 persons.
1:29:03The names the email the team and very
1:29:05important of course the role. Now the
1:29:08last step that we're going to do is that
1:29:09we're going to go and rename this
1:29:10because this is not anymore the
1:29:12saleserson. This is a generic one the
1:29:14role playing dimension. So I'm going to
1:29:16go and call it dim employee and of
1:29:19course we don't need the manager. So I'm
1:29:21going to go and disable it. Don't enable
1:29:23load so that we don't see it in our
1:29:25model. Okay. So this is our role playing
1:29:28dimension. Let's go and save and close
1:29:30everything. Now look at this. We got
1:29:32only one dimension for the employee and
1:29:34not two. And of course now comes the
1:29:36very important question how to connect
1:29:38the role playing dimension to our facts.
1:29:40So again we have two roles that means
1:29:42two relationships. The first one I think
1:29:45it is working. So we have between the
1:29:47saleserson from the fact and the ID and
1:29:50this is an active one but we need the
1:29:52second one. So what we're going to do
1:29:53we're going to grab the ID and now put
1:29:55it on the manager ID. So yes this is one
1:29:58to many relationship but of course we
1:30:00cannot activate it because we have
1:30:01already one active relationship to our
1:30:04facts. So look at this. This is exactly
1:30:07the role playinging dimension. You have
1:30:09two relationships. One of them is active
1:30:10and the other is inactive. All right. So
1:30:12with this we are done with the data
1:30:14modeling. Now of course comes the big
1:30:15question. How we going to go and use
1:30:17those two relationships at the same
1:30:19time. So let's go to the report view in
1:30:21order to make an example. At the start
1:30:24let's go and see the total sales for
1:30:26each employee. So, we're going to go
1:30:28grab the name, the role, and of course,
1:30:30we need the total sales. And that's
1:30:32means we're going to go to the fact
1:30:33table and grab the line total [snorts]
1:30:35and drop it over here. Now, of course,
1:30:37the question is which relationship did
1:30:39PowerBI uses in order to connect the
1:30:41employee and the sales. This is my
1:30:44friends, the active one. So, that means
1:30:46it is connecting the ID with the
1:30:48saleserson and not the manager ID.
1:30:51Manager ID is inactive. And with this we
1:30:53got a really nice total sales for each
1:30:56person. So far nothing crazy right? This
1:30:58is like the default setup in PowerBI.
1:31:00But now of course comes the question how
1:31:02we going to show side by side as well
1:31:05the total sales for each team not
1:31:08[clears throat] for each individual
1:31:09person. So that means I would like to
1:31:10see the line total but using the
1:31:13relationship manager ID to the dimension
1:31:16employee. Now, of course, in order to do
1:31:18that, we have to go and create an extra
1:31:20calculation in the fact in order to use
1:31:23this relationship. So, we're going to go
1:31:24and write a DAX. So, how to do it? Right
1:31:26click on the fact and say new measure.
1:31:28And then we get here a new window. In
1:31:30order to write the calculation, I'm
1:31:32going to go and expand it. So, now we're
1:31:33going to do it step by step. It is not
1:31:35that hard. So, first of all, I'm going
1:31:36to call it sales by team. So, this is
1:31:39the name of the new field. So, now of
1:31:41course the question is what we are
1:31:43calculating? we are aggregating the line
1:31:45of total. So very simple, we're going to
1:31:47say sum and then we're going to go and
1:31:49find the table. So it's going to be the
1:31:51fact sales and we are searching for line
1:31:53total. So this you're going to get the
1:31:55total sales. But of course, if you leave
1:31:57it like this, then you're going to get
1:31:59the same effect like before. PowerBI
1:32:01going to go and use the active
1:32:03relationship between the fact and the
1:32:04dimension. But no, we don't want that.
1:32:07We want PowerBI to use the inactive
1:32:09relationship. And for that we have the
1:32:11nice function use relationship so that I
1:32:14force PowerBI to use it the way that I
1:32:16want. So here we just have to define the
1:32:18keys between the two tables. So it is
1:32:20between the fact sales and this time we
1:32:23need the manager ID and the manager ID
1:32:26goes to the dimension employee and here
1:32:29we have only the ID. So it is very
1:32:32simple. We are just saying okay go and
1:32:34calculate the total sales but use the
1:32:36relationship. Then we define the keys.
1:32:38So the manager ID to the employee ID.
1:32:41Now let's go and get out. Now look at
1:32:43this. We have new extra column inside
1:32:46our fact called sales by team. It has as
1:32:49well this calculation icon because we
1:32:50just calculated. It has as well this
1:32:52calculation item means we are
1:32:54calculating this column. And now by
1:32:56going to this you have now the two
1:32:57options. Either you want the total sales
1:32:59using the active relationship by just
1:33:02dragging the line total like we have
1:33:04done at the start or if you want the
1:33:05same measure but using another inactive
1:33:08relationship. We're going to go and grab
1:33:09our calculation and put it in the
1:33:11visual. Look at this. I'm able now to
1:33:13see everything side by side. I can see
1:33:15over here for example pop. You can see
1:33:17in one go his total sales as a
1:33:20salesperson and the total sales as a
1:33:22manager for his whole team. Now for Anna
1:33:25of course she has no team. So that's why
1:33:27we don't see any number. For the other
1:33:29managers, they are not making any sales.
1:33:31That's why you're going to see here
1:33:32nulls. But you're going to see their
1:33:34total sales for their themes. As you can
1:33:36see, my friends, in one go, in one
1:33:38visual, we are able to use multiple
1:33:40relationships between the two tables,
1:33:42the fact and the dimension. And this is
1:33:44exactly the power of the roleplaying
1:33:46dimension. I don't have here like two,
1:33:48three, four tables. I have everything in
1:33:51one table in one go. And in the future
1:33:53if I got like an access to another role
1:33:56I'm going to go and integrate it inside
1:33:58the employee table so that I reduce the
1:34:01number of dimensions around the fact and
1:34:03I'm going to be very flexible using the
1:34:05relationships in order to do the
1:34:07aggregations. Okay. So now let's do a
1:34:09quick recap. Role playing dimension. It
1:34:11is one dimension that plays multiple
1:34:13roles in the same fact. You're going to
1:34:15use it when you see you have multiple
1:34:17entities that have the same structure,
1:34:19the same schema. And you're going to
1:34:21have the fact that is pointing using
1:34:23multiple ids to the same [music] entity.
1:34:29All right friends. So now finally we're
1:34:31going to start talking about a concept
1:34:32that is not that easy. The fact tables.
1:34:35This one confuses a lot of people and I
1:34:37think it get people lost in data
1:34:39modeling once they start working with
1:34:41facts. But the good news is that there
1:34:43is like one trick one very simple
1:34:45concept. If you understand it then
1:34:47suddenly working with fact tables going
1:34:49to feel easy. And here I'm talking about
1:34:52understanding the grain in the facts. So
1:34:54now of course the big question is what
1:34:56is a grain? The grain is the level of
1:34:58details that one row in your fact table
1:35:01represents. So now of course in order to
1:35:04understand what this exactly means we're
1:35:06going to have very simple example. All
1:35:07right. So in this example we have three
1:35:09fact tables and at the start if you look
1:35:11at this inside your data model you're
1:35:13going to get the feeling they are all
1:35:14like the same. The total number of sales
1:35:17is like identical and we are speaking
1:35:19about the same business. So they are
1:35:21very similar fact tables. But now my
1:35:23friends if you zoom in and start looking
1:35:25to the details you're going to
1:35:27understand those three fact tables are
1:35:29completely different from each others.
1:35:31And that's because of one thing the
1:35:33grain is different. Each table has
1:35:35different level of details. Okay, let me
1:35:37show you what I mean. Let's start with
1:35:38the first table. So here we have the
1:35:40fact sales. Now you have to ask this
1:35:42question. What does one row in this
1:35:45table represent? So we have to try to
1:35:47explain it in one sentence. So now you
1:35:49can see that this one row is actually
1:35:52part of like one order. So here we can
1:35:54see that we ordered three products in
1:35:57the same order number, right? That means
1:35:59each row is representing one line of one
1:36:02order. And this is my friend the grain
1:36:04of this fact table. Each row represent a
1:36:06line. Same thing here. For example, in
1:36:08this order, the one row representing one
1:36:10line of order. So this is the grain for
1:36:13the first table. Let's go to the second
1:36:14one. Again, we have to ask the same
1:36:16question. What does one row represent?
1:36:18Now, if you go and pick one row and say
1:36:21loud, each row here is representing the
1:36:24whole order. It doesn't matter how many
1:36:26products did we order. It could be one
1:36:28product, 10 products. It's still going
1:36:30to be only one row. So my friends the
1:36:32grain in this fact table going to be the
1:36:35whole order. It is not anymore like
1:36:37detailed like the fact sales right. So
1:36:39it has like less details than the first
1:36:42facts. Same thing if I go to another row
1:36:44like for example over here. This one is
1:36:46representing the order. But if I go to
1:36:48the other fact you can see it contains
1:36:50two products. This is the grain of this
1:36:53table. Now let's go to the last one.
1:36:55Same thing. Ask the same question. What
1:36:57does one row represent in this fact? Now
1:36:59if you go and check each row you can see
1:37:01we don't have anymore the orders like
1:37:03each row is representing one month. We
1:37:06have January, February and March. So
1:37:08that means each row here is representing
1:37:10a month of sales right? So the grain for
1:37:12this fact table going to be the month
1:37:14and of course here it is highly
1:37:15aggregated compared to the other facts.
1:37:18Other facts contains more details. So
1:37:20that means my friends all those three
1:37:22facts have different grains and
1:37:24different level of details. The first
1:37:26one fact sales is the most detailed one.
1:37:28One row represent one line of order and
1:37:31in the second fact we are losing those
1:37:33details. So the fact order the grain
1:37:35here is the whole order and the third
1:37:37fact sales monthly. The level of detail
1:37:40here is really low. So we have here high
1:37:42aggregation and the grain here is month
1:37:45of sales. We don't see any details about
1:37:47the orders or the line of orders. So
1:37:50this is what we mean with the grain. It
1:37:52is not that hard. Right?
1:37:53>> That's what she said. And from now on,
1:37:55each time you see a fact table in your
1:37:57model, you're going to ask this one
1:37:59simple question and you're going to say
1:38:00it loud. What does each row represent in
1:38:03my facts? And once you answer this
1:38:05question, then everything going to be
1:38:07simple for you. You're going to
1:38:08understand your fact. You're going to
1:38:09understand how to connect it inside your
1:38:11data model. But what is way more
1:38:13important, you're going to write correct
1:38:16DAX calculations on top of your fact. So
1:38:18my friends, understanding the grain is
1:38:20the first thing that you're going to do
1:38:22once you start data moduling. And as
1:38:24well once you understand it then you
1:38:25have really solid foundations otherwise
1:38:28things might go wrong. So now let me
1:38:29show you an example why exactly it is
1:38:32critical for calculations. Okay again
1:38:34here we have one fact table the sales
1:38:36and it looks simple right and now if I
1:38:38go and start asking you business
1:38:39questions for this fact and you jump
1:38:41immediately to the calculations without
1:38:44understanding the grain of this fact you
1:38:46might start doing wrong calculations.
1:38:48Like for example if I tell you go and
1:38:50find the total number of orders. Now you
1:38:52might say okay this is really simple I'm
1:38:54going to go and count the order ids
1:38:56inside the fact and you tell me okay we
1:38:58have six orders in the table my friend
1:39:00that was really quick look again do we
1:39:02have really six orders well no right we
1:39:05have here only three orders so my
1:39:07friends without like taking step back
1:39:09and understanding the grain of your fact
1:39:11table even if it's like simple like this
1:39:13you're going to do wrong calculations so
1:39:16what one row represent in this fact it
1:39:18is a line of order so that means if I
1:39:21want to find the total the number of
1:39:22orders in this very detailed fact table.
1:39:25I have to go and count the order ID but
1:39:28using distinct counts. So I'm counting
1:39:30one ID only once and with that you're
1:39:32going to get three orders. Now of course
1:39:34in some calculations you're going to be
1:39:35lucky like for example if you go over
1:39:37here and I tell you find the total sales
1:39:39inside our business. You're going to go
1:39:41and make the sum of this column and
1:39:43you're going to get it correct. All
1:39:44right. So now let's move to the third
1:39:46one. This going to be tricky one. If I
1:39:48ask you to find the total cost of
1:39:50shipping, if you don't understand the
1:39:51grains, you're going to go immediately
1:39:53to the shipping and say, okay, the sum
1:39:55of all those values and this is wrong,
1:39:57my friends. Because look at this, the
1:39:59same cost is like repeating for each
1:40:01line, right? And this of course makes
1:40:03sense because the cost of shipping going
1:40:05to be for the whole order, right? For
1:40:07all products. So that means if you go
1:40:09blindly and start like aggregating all
1:40:10those values and you tell me, okay, the
1:40:12total cost of shipping is 450. This is
1:40:14wrong. You still have to be careful with
1:40:16the grain and it is way more correct if
1:40:19you say okay I'm going to go and find
1:40:20the highest shipping cost in the whole
1:40:23order in case we got like wrong numbers
1:40:25or different pricing you're going to go
1:40:26with the highest one and with that the
1:40:28total cost of shipping is 200 and here
1:40:31comes really something important to
1:40:32understand that in one table we have
1:40:35different columns with different grains
1:40:37and the grain of the line of sales is
1:40:40actually the line of order and not the
1:40:42whole order and if you go and compare it
1:40:44to the shipping Here we have different
1:40:46grain in this column. It is again the
1:40:48order itself and not the line of order.
1:40:50So that means not only you have to
1:40:52understand what each one row is
1:40:55representing but as well you have to
1:40:56understand your numbers at which level
1:40:59they are. It could be as the same level
1:41:01as the table itself like the line of
1:41:03sales or it could be a different
1:41:05aggregated grain like we had in the
1:41:08shipping and the order ID. So my
1:41:10friends, as you can see, even with a
1:41:11very simple fact table, you might do
1:41:14wrong calculations if you don't
1:41:15understand the grain of your table and
1:41:17as well the grain of each column.
1:41:19Otherwise, you're going to be delivering
1:41:21wrong numbers to the business and they
1:41:23might do wrong decisions. [music]
1:41:28All right friends, so now we come to the
1:41:30most confusing topic in data modeling.
1:41:32How to work with multiple facts. Here I
1:41:34can tell you from my projects I saw a
1:41:36lot of weird things a lot of mistakes
1:41:38that happens in data modeling once you
1:41:41start working with facts. So what
1:41:42happened that people see like two facts
1:41:44and they might noticed okay they are
1:41:46sharing a common column and usually
1:41:48comes the idea why not to go and build a
1:41:51relationship between them. I would like
1:41:52to combine those two. Maybe others going
1:41:54to say you know what let's go and merge
1:41:56those two facts together or maybe go and
1:41:58append those two facts. So you have
1:42:01different opinions and different designs
1:42:02and at the end all what you want to do
1:42:04is I would like to do some analyszis
1:42:06like colorations in order to compare the
1:42:09numbers between those two facts. So at
1:42:11the start this sounds smart but once you
1:42:13start like combining building
1:42:15relationships you're going to understand
1:42:17this going to be the main mistake that
1:42:19you do in data modeling that going to
1:42:20generate a lot of issues and wrong
1:42:22numbers in the reports. And this is
1:42:25exactly what we're going to do now. I'm
1:42:26going to show you multiple scenarios,
1:42:28multiple fact tables, show you the bad
1:42:30data modeling design and of course I'm
1:42:32going to show you the solutions, the
1:42:33best practices. So now let's get
1:42:35started. All right. So now let's start
1:42:36with this very simple scenario. We have
1:42:38two fact tables. Fact sales US and fact
1:42:41sales EU. Now before we do anything, we
1:42:43have to understand those two facts and
1:42:45we start with the very important
1:42:46question. What is the grain? What each
1:42:49row is representing? On the left side,
1:42:51each row is actually representing one
1:42:53order. So we have 1 2 3 4. If you go to
1:42:55the right side, it is the same thing.
1:42:57Each row is representing actually an
1:42:59order as well. So this is the first
1:43:01thing that we understood from those two
1:43:03facts. They have the same grain. It is
1:43:05the order. So this is the first thing
1:43:07that we have understood. Now the second
1:43:08thing is to check what we are talking
1:43:10about inside each table. What is the
1:43:12event on the left side? You can see the
1:43:14event is actually a sales is happening.
1:43:16So we have an order ID, customer, a
1:43:18product, and the total amount of sales.
1:43:20So it is an event about the sales. Now
1:43:22let's go and check the right side. Well,
1:43:24it is the same event as well. Sales are
1:43:27happening here. Order ID, customer,
1:43:29product and total amount of sales. So
1:43:31thus means those two fact tables are
1:43:33actually not only having the same grain
1:43:35but as well they are identical. They are
1:43:37talking about the same event. They have
1:43:39as well the same structure, the same
1:43:41shape. They are just splitted into two
1:43:43tables. And this happens a lot in real
1:43:45world because source systems usually
1:43:47tend to split things into multiple
1:43:49tables. So here they split by the
1:43:51region, by countries, maybe by dates as
1:43:54well. Very classical one, they split
1:43:55tables by years, but at the end it is
1:43:58the same table, but it's splited into
1:44:00multiple tables in order to make things
1:44:02more manageable or maybe for security
1:44:04reasons. So now this is the setup. And
1:44:06now of course the big question is if I
1:44:07want to go and build a reports where I
1:44:10need data from the left and from the
1:44:11right. So what you're going to do in the
1:44:13data modeling in order to build this
1:44:15visual either you're going to go and
1:44:17build a relationship or you're going to
1:44:18go and combine those two tables into one
1:44:21by using merge or append. So you have to
1:44:23decide on what to do with those two
1:44:25facts inside the data model. Now let's
1:44:27start with the first one about the
1:44:28relationships. It makes no sense to
1:44:30build a relationship between those two
1:44:31because they are identical tables and
1:44:33usually we build a relationship between
1:44:35two different things like sales and
1:44:38customers orders and products. So that
1:44:40means it makes no sense to build a
1:44:42relationship between those two facts.
1:44:44What is left is to go and combine them
1:44:45into one. And here we have the two
1:44:46options, merge or append. Now if you go
1:44:49and merge those two, it will not work
1:44:50because yes, you have a key like the
1:44:52order ID. But actually there is no match
1:44:55between them. Here they start with the
1:44:561,001 and here they start with the
1:44:582,000s. So if you merge those two, you
1:45:01will not get any result. What is way
1:45:03better is to go and append those two
1:45:05together. So you go and stack them on
1:45:07top of each others. And with this you're
1:45:08going to get all the orders inside your
1:45:10business. So with this you got all the
1:45:12eight orders and things looks healthy
1:45:14right? But there is like small mistake
1:45:16that we have in the module or in this
1:45:18one table. We don't know anymore which
1:45:20order comes from where. So we lost this
1:45:22small information in the table name.
1:45:24From here we can see the orders come
1:45:26from the US and on the right side we can
1:45:28see they come from the EU. That's why in
1:45:30best practices if you go and append two
1:45:32tables go and create a new column like
1:45:34for example here the source in order to
1:45:36know where those sales come from. So
1:45:39with this after combining the two facts
1:45:41into one we will not lose any
1:45:43information. And now of course the last
1:45:44question is can we use now this fact
1:45:46table in order to do aggregations? Well
1:45:48yes if you go and find the total amount
1:45:50of sales you're going to get a correct
1:45:52number. You will not get anything
1:45:53corrupted. So this my friends the best
1:45:55way to react on this scenario. If you
1:45:57have two tables having the same grain,
1:45:59having the same event, the same shape so
1:46:01they are completely identical, go and
1:46:04append them together into one single
1:46:06facts. But of course, don't forget to
1:46:07add this new column in order to
1:46:09understand the source of each row. Okay,
1:46:11so that was an easy scenario. Now let's
1:46:13go to another one where we have as well
1:46:14two fact tables. This time we have the
1:46:17fact sales and on the right side we have
1:46:18the fact revenue. Same thing before we
1:46:20do anything we have to understand the
1:46:21content of those two facts. So what is
1:46:24the grain on the left side? Each row is
1:46:26representing one order. On the right
1:46:28side, same thing. Each row is
1:46:29representing as well one order. So again
1:46:32we have two fact tables with the same
1:46:33grain. Now the second question are they
1:46:35are dealing with the same events? Well
1:46:37on the left side like the previous
1:46:39scenario we are dealing with the sales.
1:46:41So we have an order ID, customer and the
1:46:43total sales. But now on the right side
1:46:45this time we have different event. Yes
1:46:47we have the order ID but we are talking
1:46:49about the source of the revenue and as
1:46:51well the total number of the revenue. So
1:46:53this is the setup. We have the same
1:46:54grain. We have different events but we
1:46:56are talking about the same thing about
1:46:58the orders. Now of course same question.
1:47:00What we going to do if we need to build
1:47:02a reports and we need informations from
1:47:04those two fact sales. Are we going to go
1:47:06connect them using a relationship or go
1:47:08and combine them into one fact using
1:47:10merge or append. So now let's start with
1:47:12the relationship. If you go and connect
1:47:14them you're going to get the cardality
1:47:16one to one. Well usually if we have one
1:47:18to one relationship it makes no sense to
1:47:20keep them splitted. It's way better if
1:47:22you just combine them into one table.
1:47:24But of course, always under one
1:47:26condition if they have the same grain.
1:47:29So what are the other options? We're
1:47:30going to go and combine those two facts
1:47:32tables into one single fact. Here we
1:47:34have two options either append or merge.
1:47:36Now I found a lot of projects where they
1:47:38go and actually append those two facts
1:47:41together. So they look like almost
1:47:42similar those two fact tables. You have
1:47:45an ID, you have maybe something that is
1:47:47describing the ID and then the measure
1:47:49the total sales. So they are not
1:47:51identical but they have the same pattern
1:47:53that's why they go and like append them
1:47:55together. So if you look at this you're
1:47:56going to say wow this looks nice right I
1:47:58have all the orders here I have the
1:48:00measures and I can see the grain so the
1:48:03measure type that I have in the amount
1:48:05so things looks clean and with this I
1:48:07can go and do multiple like analyzes and
1:48:09I have some dynamic well you can go and
1:48:11do that of course but you have to be
1:48:13very careful each time you use this fact
1:48:15table like for example if you don't
1:48:16explain this for the others and someone
1:48:18goes and start like doing a sum of the
1:48:21amount they're going to go and get like
1:48:223,500 and this makes no sense Because
1:48:26here those numbers they have different
1:48:27meanings for each row. In one row this
1:48:30number is a sales but in another row
1:48:32this number is revenue. So if you go
1:48:34blindly and do a total sales on this
1:48:36fact or maybe an average for the sales
1:48:38you're going to get of course wrong
1:48:40numbers in the reports and you have
1:48:42always to be careful that you do the
1:48:44aggregations based on the grain. And
1:48:46this of course makes everything
1:48:47complicated and open the door for many
1:48:50mistakes. That's why I never recommend
1:48:52to append it like this. Instead, we go
1:48:54and merge them. So, you're going to say,
1:48:56you know what, they are sharing onetoone
1:48:58relationship. I'm going to go and just
1:49:00put everything side by side. So, I have
1:49:02the order ID, I have the customer, the
1:49:04source, the sales, and the revenue. And
1:49:06with that, my measures are actually not
1:49:08mixing together. And it's still very
1:49:10easy to analyze. Like for example, if I
1:49:12need to find the total sales, I'm just
1:49:14going to go to this column over here and
1:49:16find the total sales. Same thing for the
1:49:18revenue. So yes, I have wider table and
1:49:20I have more columns inside my data
1:49:22model, but doing aggregations and stuff
1:49:25on the fact it is way easier and safe
1:49:28compared to the append. So my friends,
1:49:29in this scenario, if you have two fact
1:49:31tables that have the same grain like
1:49:33this, they have one to one
1:49:35relationships. They are not identical.
1:49:37So they have like different events or
1:49:39different columns. Then just go and
1:49:40combine them, merge them together into
1:49:42one single fact table. And don't use
1:49:44append if they are not exactly
1:49:47identical. Okay. So let's keep moving
1:49:48now to the third scenario. Things going
1:49:50to get a little bit more advanced. So
1:49:51again we have two fact tables. The fact
1:49:53sales and the fact budget. We have to
1:49:55start with the first question. What is
1:49:56the grain? So what each row is
1:49:58representing in each fact. You can see
1:50:00that in the fact sales is actually each
1:50:02row is one order. So it is very simple.
1:50:05But now if you go to the fact budget you
1:50:07can see that each row is actually a
1:50:09month. It is the planned budget for
1:50:11specific product in a month. So for
1:50:13example for the product laptop we can
1:50:15see that in January we have so much
1:50:17budget and for February we have another
1:50:19planned budget for it. So that means my
1:50:21friend the grain is different between
1:50:23those two tables they have completely
1:50:25different level of details. Now the next
1:50:27question are we talking about the same
1:50:29events the same structure well they are
1:50:31totally different from each others. The
1:50:33fact sales here the event is sales is
1:50:35happening but on the right side planning
1:50:37is happening. So for each month for each
1:50:40product we have a planned budget as
1:50:41well. The structure it is totally
1:50:43different. Here we don't have any ids.
1:50:44On the left side we have an order ID. So
1:50:47those two fact tables they are
1:50:48completely different. They don't have
1:50:50the same events. They are not sharing
1:50:52the grain. They are not sharing the
1:50:53event or the structure. And now of
1:50:55course the question is if a business
1:50:57user ask you to go and compare the sales
1:51:00versus the budget by the product. So you
1:51:02need data from those two facts into one
1:51:05visual. What you have to do? So again we
1:51:07have the same options. Either we're
1:51:08going to build a relationship between
1:51:09them or merge those two facts together
1:51:11into one single fact. So now let's start
1:51:13with combining everything together. Here
1:51:15you have the two options either append
1:51:17or merge. Well, it makes no sense to
1:51:19append. Please don't do that unless they
1:51:21are totally identical and they have the
1:51:23same grain. But now a mistake that might
1:51:25happen is that you can see you know what
1:51:27we have a product on the left side and a
1:51:29product on the right side. Actually we
1:51:31can go and merge them together using
1:51:33this key. So if you go and do that
1:51:34you're going to get something like this.
1:51:36So you have the order ID, you have the
1:51:37month, you have all the products and
1:51:39then you have the two measures, the
1:51:41sales and the budget. So with this you
1:51:42have everything like side by side and
1:51:44you can go and compare them right. But
1:51:46my friends those numbers going to makes
1:51:48no sense at all once you start
1:51:50aggregating. So if you go and give this
1:51:51to your business users they're going to
1:51:53say well actually we didn't plan so much
1:51:55budget. We don't have like 9,000. We
1:51:57have planned only 6K. So actually your
1:52:00numbers are wrong and PowerBI has maybe
1:52:02an issue. You're going to go to your
1:52:03data model and check well actually we
1:52:05have correct numbers we don't have like
1:52:06wrong data inside PowerBI it's only the
1:52:09decision that you have made to merge
1:52:11those two together into one table
1:52:13because my friend the relationship
1:52:14between those two fact tables are
1:52:16actually money to money like for example
1:52:17on the left side the laptop if you
1:52:19combine it with a budget you're going to
1:52:20get actually two plan budget for this
1:52:22product so you get the January and
1:52:24February and you're going to have them
1:52:26as two rows inside the new table and
1:52:28same thing for example for the order
1:52:30number three as well It going to connect
1:52:32to two plan budget the January and
1:52:35February. So with this you're going to
1:52:36get the same budget inside the combined
1:52:38table twice. One for the first order and
1:52:41another one for the second order. And of
1:52:42course this is really bad if you combine
1:52:45two tables that has many to many
1:52:47relationship. It will just cause
1:52:48duplicates in the new combined table. So
1:52:50no, you cannot go and combine those two
1:52:53tables together. They don't have a key
1:52:55between them and they have different
1:52:56grains. So what are the other options?
1:52:58Well, actually, we can go and build a
1:53:00relationship between them, right? So,
1:53:02you're going to start something like
1:53:03this. You're going to leave them
1:53:04separated so we don't merge. And you're
1:53:06going to start picking maybe the product
1:53:07ID from the fact sales in order to build
1:53:09the visual and then put the sales for
1:53:12each product. So, everything so far is
1:53:13clean. Right now, what is missing is of
1:53:15course the budget. So, what you're going
1:53:16to do, you're going to go and relate
1:53:18those two tables together using the
1:53:20product. Again, you see here the money
1:53:21to many relationship between those two.
1:53:23And then you're going to go and grab the
1:53:25budget and you put it on the visual. So
1:53:27now if you go and give this as well to
1:53:29the users, they're going to as well see
1:53:31bad numbers in the visual. They're going
1:53:32to say okay from one side you are
1:53:34getting the total sales correct not like
1:53:36the previous one as you merge the two
1:53:38tables. But my friend that total budget
1:53:40makes no sense. We didn't plan so much.
1:53:43So again your report is wrong, your data
1:53:45is wrong and they will not trust it. So
1:53:46again you're going to go back to the
1:53:47data model and you start adding those
1:53:49numbers. Well it makes no sense. The
1:53:51data is correct. It's all because you
1:53:53are connecting directly two fact tables
1:53:55together. And this can only cause
1:53:57issues. From one hand, you are totally
1:53:59depending on the products from the fact
1:54:01sales. And this is a big issue because
1:54:03in the fact sales you can see only the
1:54:05products where sales happens. But maybe
1:54:08on the budget you are planning for the
1:54:10future. So you have a new product and
1:54:11you have a budget for it. But no orders
1:54:14yet happened to this new product. So for
1:54:16example here the monitor we have a
1:54:18budget for it but still we don't have
1:54:19any sales for it. Now of course this
1:54:21product now is missing from the report
1:54:23because you are building the access for
1:54:25the visual based on the products from
1:54:27the fact sales. So that means you cannot
1:54:30count on the products from the fact
1:54:31sales. Same thing if you say you know
1:54:33what let's go and use the products on
1:54:34the right side as well. You will not get
1:54:36the full big picture like for example
1:54:38here the mouse we have an order for it
1:54:40but we don't have a planned budget for
1:54:42it. So it doesn't matter where you start
1:54:44building the visual it will not work if
1:54:46you are connecting two facts together.
1:54:48And as well same thing since we have
1:54:49many to many relationship we're going to
1:54:51get this effect where the budget going
1:54:52to fan out and you're going to get here
1:54:54duplicates inside your visual and this
1:54:56is of course what is causing the big
1:54:58number of the budget. So actually my
1:55:00friends we cannot append we cannot
1:55:01combine we cannot build a relationship
1:55:04between those two facts. We have only
1:55:06one solution to make this clean and
1:55:08that's by introducing something else. So
1:55:11what we're going to do we're going to
1:55:12get rid of this bad relationship between
1:55:14those two facts and we're going to
1:55:15introduce a new dimension in the middle.
1:55:17So now you're going to hear a lot of
1:55:19different names about this dimension
1:55:20here in the middle. Some people call it
1:55:22a shared dimension, others call it
1:55:24confirmed dimension or you might hear
1:55:27about the bridge table between two fact
1:55:29tables. You can call it whatever you
1:55:31want. At the end it is the same concept.
1:55:33We're going to build one dimension where
1:55:34it has the single point of truth. So it
1:55:37is complete and it has all the products
1:55:39in the business. So here should be
1:55:41nothing missing. We have the laptop, we
1:55:43have the phone, we have the monitor and
1:55:45as well we have the mouse. So all the
1:55:47products in the business and after that
1:55:49we connect our facts to this single
1:55:51dimension and if you do it like this on
1:55:53one side you're going to get money to
1:55:54one relationship between the sales and
1:55:56the dimension and same thing on the
1:55:58other side between the budget and the
1:55:59dimension you're going to get one to
1:56:00many. And now once you start building
1:56:02the visual things going to be way easy
1:56:04for the axis you start always from the
1:56:06dimension in order to make sure that all
1:56:09the products are inside my visual. So I
1:56:12have all my four products and the
1:56:13measures going to come from each fact.
1:56:16So the total sales going to come from
1:56:17the fact sales and the budget going to
1:56:19come from the budget. So with this first
1:56:20of all you make sure that nothing is
1:56:22missing right? So we have here the
1:56:24monitor. We don't have sales for it but
1:56:26we have a budget for it and we have as
1:56:28well the mouse where we have a total
1:56:29sales for it but no budget. So with this
1:56:32I'm getting completeness in my visual.
1:56:34Nothing is missing and as well the
1:56:36numbers going to be correct. There is no
1:56:37more fan out effect like for the laptop.
1:56:40things are connecting to my dimension
1:56:42using the many to one relationship and
1:56:44this will not cause any duplicate effect
1:56:46in the visual. So this solution is a
1:56:49clear win for me. I have the complete
1:56:50picture. Nothing is missing and as well
1:56:52my numbers are correct. My friends, if
1:56:54you see two fact tables, two tables that
1:56:56they have different grains, then you
1:56:58have to be very careful. You might see
1:57:00things that could be related to each
1:57:02others or could be combined. Otherwise,
1:57:04you're going to deliver only wrong
1:57:05numbers. Leave them separated and
1:57:07connect things only using a shared
1:57:10dimension. Okay. So now to the last
1:57:11scenario. This one going to be a little
1:57:12bit tricky. So I need your full focus.
1:57:14This is advanced. Again we have two fact
1:57:16tables. So we have the sales, we have
1:57:18the budget. We will not go through
1:57:19everything. They have different grains.
1:57:21They have different events. But the
1:57:22thing is they don't connect using the
1:57:25products. They connect only using a
1:57:27higher level. So we have the category.
1:57:29So this is a setup. And now the business
1:57:31user has the same question. I would like
1:57:33you to go and compare sales versus
1:57:35budget but by the product. So again if
1:57:37you go and build a direct relationship
1:57:39between those two you're going to get of
1:57:41course a wrong budget. We said you don't
1:57:43do that. So instead what we're going to
1:57:45do we're going to go and introduce the
1:57:46shared dimension in between and the
1:57:48shared dimension this time going to be
1:57:50the category because they are not
1:57:51sharing products right? They are sharing
1:57:53category and after that you go and
1:57:55connect your facts to the new dimension
1:57:57category. Now the thing is we're going
1:57:59to still have an issue in the visual
1:58:00even if we introduce the shared
1:58:02dimension between because the user is
1:58:04asking us to compare those two measures
1:58:06by the products and the products my
1:58:08friends it's not an information that
1:58:10exist in the budget right the budget
1:58:12knows only about the categories and the
1:58:14categories as I said it is a higher
1:58:16aggregated level than the products so
1:58:19that means with the products and the
1:58:21fact sales we have more details than the
1:58:23budget and here we have very important
1:58:25rule to understand we can show a measure
1:58:27only at the grain that it was recorded
1:58:30at or a higher level so more summarized.
1:58:33So that means we can never show the
1:58:34measure at a lower level than the grain
1:58:37that is recorded at like for example the
1:58:39budget was only recorded at the category
1:58:42level. So we can show only the numbers
1:58:43at this level or we can roll up if we
1:58:46remove the categories in order to have
1:58:47the total budget. But we never can go
1:58:50down that it cannot go to the budget by
1:58:52the products because it is not recorded
1:58:54at this level. And if you use any tool
1:58:56it going to go only and fake the
1:58:58numbers. It will not work. So the rule
1:59:00is very simple. If you have two fact
1:59:01tables, you can compare them only at the
1:59:04highest level they share. So that means
1:59:06if you want to do really this
1:59:07comparison, we have to do it at the
1:59:10category level. So yes, we compare the
1:59:12sales versus budget but by the category.
1:59:15And don't misunderstand me if you want
1:59:16to build a visual only for the fact
1:59:18sales. Of course, you can go and use the
1:59:20products, but we are talking only about
1:59:22connecting two facts into one visual.
1:59:25Here you have to be careful which one is
1:59:27the highest and only that you can use. I
1:59:29know this one was an advanced one, but
1:59:31once you know those rules, then
1:59:32everything going to be simple. Back to
1:59:34PowerBI to practice. Again, we are at
1:59:36our data model. So far, we have only
1:59:38dimensions around one fact. the fact
1:59:41sales. Now, what we're going to do,
1:59:42we're going to go and grab another fact
1:59:43to the data model and we're going to go
1:59:45and connect it. So, get a new data CSV
1:59:48and this time we're going to go and
1:59:49connect the fact budget. So, let's go
1:59:52and do this. Okay. So, go and load it.
1:59:54All right. So, with this we got our new
1:59:56fact inside our model. And the thing is
1:59:58I'm going to go and remove the
2:00:00relationship that PowerBI automatically
2:00:02created between this fact and other
2:00:04things. So, let's go and remove it
2:00:06because now we are dealing with the
2:00:07facts and we have to be careful. So
2:00:09first we have to understand our facts
2:00:12not only the budget but as well the fact
2:00:14sales. Now in order in PowerBI to
2:00:16understand the grains and all those
2:00:17stuff we're going to go to other tab. So
2:00:20we're going to go to that table view and
2:00:21let's go to the fact sales. And now of
2:00:24course we're going to ask the classical
2:00:25question what each row here is
2:00:27representing. So we can see that the
2:00:29order ID, 101 we can see it twice. So we
2:00:33have two rows for the same ID. So a row
2:00:35is not representing the whole order. row
2:00:37is a line of order. This is a really
2:00:40detailed one. Now let's go and check the
2:00:42budget. So now look at this. This is
2:00:44highly aggregated. So each row is
2:00:46representing actually a month. The plant
2:00:48budget and the plant quantity for each
2:00:50product for one month. The fact sales
2:00:52has a lot of details and the fact budget
2:00:55is highly aggregated. That means those
2:00:57two facts have different grains and of
2:01:00course we cannot go and combine them.
2:01:01And of course we will not relate them
2:01:03directly to each others just because we
2:01:05have here a product ID and as well in
2:01:07the budget we have a product ID. So we
2:01:09will not do this where we're going to
2:01:11grab the product ID from the fact budget
2:01:13to the fact sales. Look at this. PowerBI
2:01:16says many to many relationship and as
2:01:18well we are getting here like a warning
2:01:20from PowerBI says this is many to many
2:01:22both of the product ids are not
2:01:24containing unique values and it's like
2:01:26[snorts] be careful of this behavior. So
2:01:29if I go and save it, we are doing now
2:01:31bad practices by connecting the fact
2:01:34tables together. So my friends, we will
2:01:36not do that. So we're going to remove
2:01:37it. Instead, we need a dimension in
2:01:39between a bridge, a conform dimension.
2:01:42And this is going to be our friend, the
2:01:44dimension product. So let's go and grab
2:01:46it over here and just put it nicely
2:01:48between those two facts. So you can see
2:01:50it is already connected to the fact
2:01:52sales. So we don't have to deal with
2:01:54this side. But we have to go and connect
2:01:56it to the fact pageto. Grab the product
2:01:58ID and put it on the dimension product.
2:02:01Look at this. Looks nice. We have many
2:02:02to one relationship. And that's because
2:02:04the product ID in the dimension products
2:02:06is unique. So, looks good. Let's go and
2:02:09save it. And actually, that's it. Look
2:02:11at this. We have a nice shared dimension
2:02:14between two facts. So, one to many
2:02:15relationship, many to one. And this
2:02:17dimension is just like shared between
2:02:20those two facts. So, this is how we
2:02:22connect two facts together. And now we
2:02:24can go and use the values of those two
2:02:26facts inside our visuals and reports.
2:02:28Let's go and try this out. Now, as we
2:02:30learned in order to build a visual, we
2:02:32always start from the shared dimension
2:02:34which is the dimension products. So, it
2:02:36is over here. I'm going to go and pick
2:02:38the product name and only then you start
2:02:40picking the measures from the facts. So
2:02:42now let's go and grab let's say the
2:02:44total sales from the fact sales. So I'm
2:02:46going to go and drop it here. So this is
2:02:47the first fact. Now let's go and compare
2:02:49it by another measure from another fact,
2:02:51the budget. So we're going to go and
2:02:53grab the budget sales and drop it on the
2:02:55view. Look at this. Everything is
2:02:57working. And now I'm able to do sales
2:02:59versus budget analyzes. Now of course in
2:03:02order to test whether you are missing
2:03:03any numbers, what we can do is that you
2:03:05can just go and duplicate this view like
2:03:08twice. One left, one right. And now we
2:03:10can make one view dedicated only for one
2:03:13fact. So that means I'm going to go over
2:03:14here and remove the budget. This going
2:03:16to be only for the sales. And if the
2:03:19numbers are exactly the same as the
2:03:21combined one over here, then everything
2:03:23is healthy. Same thing on the right
2:03:25side. This time this going to be
2:03:27dedicated only for one fact, the budget.
2:03:29So that means I'm going to go and remove
2:03:31any value from the other facts. Now look
2:03:33at this. The total budget is exactly
2:03:36like the combined one. And this means
2:03:38everything is working perfectly. So this
2:03:40is how I test if I'm combining two facts
2:03:43tables together. You make one dedicated
2:03:44view for each and see whether the
2:03:46numbers are matching. And now of course
2:03:48you can go and check other stuff like by
2:03:50grabbing other measures to our view and
2:03:52see whether everything working fine. And
2:03:54now one more thing that we could
2:03:56practice that we could actually connect
2:03:58multiple dimensions as a shared
2:04:00dimension. So it must not be always like
2:04:02only one we can go and add more. So that
2:04:05means if we go and check the facts and
2:04:07see something in common. So here we have
2:04:09the month of the budget. So this is a
2:04:11date. And now if we go to the fact sales
2:04:13we can see as well let's say the order
2:04:15date. This is not a month but as well it
2:04:18is a date and we could like use a shared
2:04:20dimension called the dimension date in
2:04:22between. Let me show you how to do it.
2:04:24So let's go back to the model view. Now
2:04:26of course we need a dimension date. We
2:04:27don't have it yet inside our model.
2:04:29We're going to go and connect something
2:04:30new. So over here you can see we have a
2:04:33dimension date. Let's go and load it and
2:04:35let's go and hit load. All right. So
2:04:36with this we got a new dimension and I'm
2:04:39going to put it just in between those
2:04:41two facts. Now of course in order to
2:04:43make a share dimension we have to go and
2:04:45connect both facts to this new
2:04:47dimension. So for example let's go and
2:04:49grab the order date and connected to the
2:04:52date inside the dimension. So as you can
2:04:54see they are using the same informations
2:04:56and we have many to one relationship. So
2:04:58perfect. Let's save it. So this is one
2:05:00side. Now let's go and connect the other
2:05:01side. So budget month it goes as well to
2:05:04the same key to the date. Same story we
2:05:07have here money to one relationship
2:05:08which is active and this is nice. So now
2:05:11with this we have two shared dimensions
2:05:13between our two facts and this of course
2:05:15open the door for more analytics between
2:05:18the two measures inside the different
2:05:19facts. Now we can compare things by the
2:05:21dates. Let's go and try this out to the
2:05:23report view. As usual we start with the
2:05:25dimension. So the new one is the
2:05:27dimension date. Let's go and drop it in
2:05:29the visual. Now here very important to
2:05:31understand the level of details the
2:05:33grains. You can see the dimension is
2:05:35based on the days. So for each day we
2:05:37have one row. So we have to keep this in
2:05:39our mind. Now let's go and grab the
2:05:41measures. It's going to be the again the
2:05:43total sales. Now look at this and
2:05:44understand we have for few days a total
2:05:47sales. So in January we have two sales
2:05:49and in February we have two days of
2:05:51sales. With this we can understand that
2:05:53the level of details of the dates inside
2:05:55the fact sales is as well the day and
2:05:57you can check as well that in the table
2:05:59view you can see the order date always
2:06:01include like a day. So far everything is
2:06:04matching. Now let's go to the budget. We
2:06:06need as well the planned budget for the
2:06:08sales and drop it on the same view. Now
2:06:10the first thing that we have to check is
2:06:12of course the total. So the total budget
2:06:14is matching over here and as well for
2:06:16the sales. We don't have to worry about
2:06:18now wrong numbers. But now if you look
2:06:20to the table over here you can see there
2:06:22is something weird right? You can see
2:06:24that the numbers are not really side by
2:06:26side. There is like some shifting. Now
2:06:28if you look closer to the budget you can
2:06:30see it's always at the first day of the
2:06:32month. So the 1st of January, the 1st of
2:06:35February and so on. So we see a budget
2:06:38only on the first day and for the total
2:06:40sales we see it in different days.
2:06:42Sometimes the 5th, sometimes 12th, third
2:06:44and so on. So we are not really able to
2:06:47compare the two measures side by side.
2:06:49And this is my friends because we have
2:06:52different level of details and grain for
2:06:54the budget. So we can go and check this
2:06:56over here. Let's switch to the fact
2:06:58budget. We see it like the 1st of
2:06:59January. This is only the format but the
2:07:01level here is a month and not a day. And
2:07:04what we have learned before if you have
2:07:06different level of details, if you have
2:07:08different grains, you have always to
2:07:10work on the highest level. Otherwise,
2:07:12your chart kind of looks exactly weird
2:07:14like this. So in order to compare now
2:07:16the budget and the total sales, we have
2:07:18to bring them together at the same level
2:07:20where both of them can understand the
2:07:22level. And this is my friend not the
2:07:24day. This is the month. That's why I'm
2:07:26going to go to the date over here and
2:07:28just get rid of the day. And with this,
2:07:30we have like a roll up to the month. And
2:07:32look at this. Now we see things side by
2:07:35side. So the budget is near the total
2:07:37sales. That's because both of the fact
2:07:39tables can work at that same level at
2:07:41the month. And now things are more
2:07:43healthy and we can go and compare the
2:07:45two measures side by side. As you can
2:07:47see my friends, not only you have to
2:07:49build a healthy data model, you have as
2:07:51well to understand the grain of
2:07:53everything. the grain of the dimensions,
2:07:55the grain of the facts. Without that,
2:07:57you're going to build weird reports and
2:08:00in the worst case you will be reporting
2:08:02wrong numbers. So now let's recap. So if
2:08:04two fact tables have the same grain, the
2:08:07same fact, the same structure, the same
2:08:08event, then go and just append those two
2:08:11facts together. But if you have a
2:08:12scenario where the two facts yes have
2:08:14the same grain but they have different
2:08:15shapes and you can build a onetoone
2:08:18relationship between those two facts
2:08:19then just go and combine them into one
2:08:21fact by merging them together. But now
2:08:24in the other scenario if they have
2:08:25different grains different shapes
2:08:27different events. Never combine those
2:08:29two facts and as well never relate them
2:08:32directly to each others. Always build in
2:08:34between a shared or bridge dimension and
2:08:37connect your facts to it. And at the
2:08:39visual level if you are comparing two
2:08:41facts only work at the level that both
2:08:44facts understand. So always pick the
2:08:46highest level that both facts can
2:08:48understand. So my friends once you
2:08:50understand those rules then it's going
2:08:51to be really easy to work with facts in
2:08:53[music] data modeling.
2:08:58All right friends now we're going to
2:08:59talk about something really important
2:09:00for data modeling and as well for real
2:09:02projects. The security. The thing is in
2:09:05real projects it is not enough to have a
2:09:07model that gives correct numbers but
2:09:09also you have to control who is allowed
2:09:11to see which data. So that means you
2:09:12have to make it secure and this is
2:09:14deeply connected into data moduling. If
2:09:17you have clean data model then it's
2:09:18going to be really easy to implement the
2:09:20security requirement and it's going to
2:09:21be safe. But if you have a data model
2:09:23that is messy chaotic the security going
2:09:26to be really hard to implement and your
2:09:28data going to leak to the wrong users.
2:09:30And this is exactly what we're going to
2:09:32do now. And we start as usual with the
2:09:33foundations. So what is security in data
2:09:36moduling? Let's go. Okay. So we have a
2:09:38table and by default this table has no
2:09:41security. So that means if we are
2:09:42building a report on top of it, everyone
2:09:45all the users going to be able to see
2:09:46all the data of this table. But now
2:09:48let's say that we are a real project and
2:09:50we have to go and apply security on top
2:09:52of this table. Here we're going to have
2:09:53like three options. The first scenario
2:09:55if the whole table is actually contains
2:09:58a lot of sensitive informations like the
2:10:00salaries, the contacts, the finance data
2:10:03and some users are actually not allowed
2:10:05to access this table at all. So that
2:10:07means what we have to do in this
2:10:08scenario we have to go and hide the
2:10:10whole table everything all the rows and
2:10:12all the columns and for them this table
2:10:14is basically doesn't exist. This is what
2:10:16we call the table level security. Now
2:10:19the second scenario maybe not that
2:10:21extreme where you say you know what we
2:10:23have a table like employees and it
2:10:25contains a lot of data that are actually
2:10:27not sensitive but we have like one two
2:10:29columns that are really sensitive like
2:10:31for example the salary. So we will not
2:10:33go and hide the whole table for the
2:10:35users. What we can do we can go and
2:10:37protect and hide only one specific
2:10:39column. So that means the user is going
2:10:41to be able to see all the data but only
2:10:43this column going to be hidden for them.
2:10:45This is what we call the column level
2:10:48security. Now to the third scenario
2:10:50which is the most common one in many
2:10:52projects where we can say actually the
2:10:54table is totally okay. All the columns
2:10:56are not sensitive but now it is not
2:10:58allowed to show all the rows for all the
2:11:00users. Like for example a salesperson
2:11:02can only see the region or maybe the
2:11:05managers are allowed only to see their
2:11:06own departments. So that means the whole
2:11:09table can be visible. All the columns
2:11:10are allowed but the users will not see
2:11:13all the rows. They are allowed to see
2:11:14only specific rows. And this is what we
2:11:17call the very famous rowle security and
2:11:20you're going to hear a lot of people are
2:11:22saying RLS. So those are the three
2:11:24levels of security or hiding things in
2:11:27data modeling. The table level security
2:11:29you want to hide the whole table. The
2:11:31column level security you want to hide
2:11:32only specific sensitive columns. And the
2:11:35third one is the role level security if
2:11:37you want to go and hide specific rows.
2:11:39Now of course the most common technique
2:11:40that is used almost in every project is
2:11:42the role level security. And this is
2:11:44exactly what we're going to deep dive
2:11:46now. So we have to understand why and
2:11:48what. Let's go. Okay. So now we have
2:11:49this following setup where we have a
2:11:51table about the sales and a report in
2:11:53top of it. Now this sales table going to
2:11:55contain all the sales. So for all
2:11:57departments or regions. Now the thing is
2:12:00in companies we have different people
2:12:01with different roles. Like for example,
2:12:03we might have like a director that is
2:12:05actually allowed to see all the data and
2:12:08we might have another person like for
2:12:09example a manager that is allowed only
2:12:12to see the sales for specific
2:12:14department. So this means not all the
2:12:16rows only specific one and we might have
2:12:18a third one. So we have a salesperson
2:12:20that is allowed only to see the data
2:12:22based on the region that is assigned to
2:12:24it like maybe only Europe. So that means
2:12:26we have three different users with three
2:12:28different security requirements. Now, of
2:12:30course, because of this, you cannot go
2:12:32and just share the report with everyone.
2:12:34You have to be careful with the security
2:12:36requirements. Now, one of the ideas is
2:12:38to go and do this where you're going to
2:12:39say, "Okay, I'm going to go and
2:12:40duplicate my report into three files.
2:12:42I'm going to build a filter specifically
2:12:45tailored for each role and then start
2:12:47sharing those reports based on the
2:12:49role." So, you are making like
2:12:51role-based reports. But of course I
2:12:53think it's obvious this is really bad
2:12:55solution because you have the same
2:12:56report the same work over and over and
2:12:59all those duplicates is just because of
2:13:01one simple filter and of course if you
2:13:03want to change one thing it going to be
2:13:05really nightmare you have to go through
2:13:06all the files and start implementing the
2:13:08same change again and again. So this
2:13:10going to be really nightmare to maintain
2:13:12and it will not work. Now we have
2:13:13smarter and better solution that instead
2:13:16of having all those duplicates we're
2:13:18going to go back to one model and one
2:13:19reports. But now what we're going to do,
2:13:21we're going to go and add a rowle
2:13:23security component on top of the data
2:13:25model. And it is smart enough to let
2:13:27PowerBI understand who is looking at the
2:13:30reports and based on the information
2:13:32based on the role PowerBI going to go
2:13:34and filter the data model specially for
2:13:37that person and this is exactly the best
2:13:39solution is to implement the relevant
2:13:41security so that you stick with the one
2:13:43project without having you to duplicate
2:13:45things. Now of course the question is
2:13:46how we going to go and build this smart
2:13:48component in the data model? how this
2:13:50works and how to build it. Now, of
2:13:52course, implementing role security
2:13:53depend on the tech stack that you are
2:13:55using. One solution that you might find
2:13:58rarely in projects where they build a
2:14:00live connection between PowerBI and
2:14:03underneath like database or maybe like
2:14:05data bricks and they decide to build the
2:14:07whole role security in the back end. So
2:14:10they build the whole filtering logic in
2:14:12the database and PowerBI has only to go
2:14:14and pass the user information to the
2:14:17back end and of course the backend
2:14:18system going to return only the rows the
2:14:21user is allowed to see. So that means
2:14:23the whole logic going to be built
2:14:24outside of PowerBI and to be honest this
2:14:27is something really complicated to do
2:14:29because now you have two systems the
2:14:31testing going to be really hard and we
2:14:33rarely use the live connection between
2:14:35PowerBI and the backend system. Most of
2:14:37the times we import the data to PowerBI.
2:14:39So that's why the second option is to go
2:14:42and build the role of it security inside
2:14:44PowerBI and this is the most common
2:14:46option that we actually build and
2:14:47implements in real projects and this is
2:14:49exactly what we're going to learn. So
2:14:50now how we're going to build this RLS
2:14:52inside PowerBI and how it works. So now
2:14:54we have again this setup where we have a
2:14:55table in the model and this table has a
2:14:58column called region where you have the
2:14:59values US and EU. After that we have a
2:15:02report and then three users are using
2:15:04this report and accessing it. Now if you
2:15:06leave it like this then we don't have
2:15:07security that means everyone is seeing
2:15:09everything. But now if we cannot leave
2:15:11it like this and we have to go and
2:15:12implement the role level security then
2:15:14of course we cannot go and build some
2:15:16random filters inside PowerBI. We need
2:15:18more tools. So the first component or
2:15:21tool that we're going to use for role
2:15:22level security is the role and it is a
2:15:24tool that we're going to use in order to
2:15:26filter the model. So it's very simple.
2:15:28It has a name and a value. Now the value
2:15:31could be either static or a ducks
2:15:33calculation if you have expression. Now
2:15:35if you decide for the static values then
2:15:37you're going to have like two roles
2:15:38based on the regions one for EU and
2:15:41another one for US. So this is the rule
2:15:43and with this PowerBI knows how to
2:15:45filter the data model based on the role.
2:15:48But that's not enough of course because
2:15:49PowerBI is missing another important
2:15:52information the mapping between the
2:15:54users and the role. PowerBI doesn't know
2:15:56whether Maria has the role like us or U.
2:15:59So we don't have yet this mapping. And
2:16:01for this you have two options. either
2:16:03you're going to leave it like this
2:16:04static where you go and publish the
2:16:06report to PowerBI service and from there
2:16:09you start manually mapping which user
2:16:12belong to which role and now this
2:16:14PowerBI has everything the two
2:16:15informations the mapping between the
2:16:17user and the role and the mapping
2:16:19between the role and the data model. So
2:16:22everything is linked. Now if a user like
2:16:24Maria go and access the reports, what
2:16:26going to happen? PowerBI first has to go
2:16:27and check the role and say ahuh she
2:16:29belongs to the role EU and then go to
2:16:31the other mapping. Okay, the role EU has
2:16:34the static value EU and after that it's
2:16:37going to go to the data model and filter
2:16:38the data based on this static value. So
2:16:41this Maria going to see only EU data and
2:16:44this is my friends what we call the
2:16:46static role security in PowerBI. Now by
2:16:49looking into this the role of security
2:16:51has actually two problems. The first
2:16:53issue we are creating like a role for
2:16:55each value inside our data. So here we
2:16:57have like two values in the region we
2:16:59have create two roles. But if the
2:17:01security concept is based like on
2:17:02another column on another values where
2:17:04you have 10 and 30 you have to go and
2:17:06create like 30 rules and if you keep
2:17:08like getting new values you have to go
2:17:10and keep expanding and maybe you're
2:17:12going to go and miss something. And the
2:17:13second issue, each time we get a new
2:17:15user, we have always to go to the
2:17:16PowerBI service. We have to assign those
2:17:19new users to the different roles. To be
2:17:21honest, I have never done this in
2:17:22projects because I had like 3,000 users
2:17:24and I don't have the time to go and
2:17:26assign manually each user to each role.
2:17:28That's why we have another option where
2:17:30things are more dynamic and we call it
2:17:33dynamic role level security where we're
2:17:35going to say we will not do those manual
2:17:36things but instead we're going to add a
2:17:38new table to the data model. This is a
2:17:41special table only for security where it
2:17:43has two informations which user is
2:17:45allowed to see which region. So we're
2:17:47going to see like information for each
2:17:49user like Maria going to see you and so
2:17:51on. Now of course the big question is
2:17:53where this data come from. Now if you
2:17:55are unlucky you have to go and create it
2:17:57manually. So you can have your own Excel
2:17:59where you are mapping the users to the
2:18:01regions. Again it is kind of manual
2:18:03work. Or if you are lucky enough and
2:18:05this is something that really happens a
2:18:07lot in projects where you're going to
2:18:08find this kind of information stored
2:18:10somewhere in your source systems and
2:18:12this is actually what I usually do. I go
2:18:14and load the security informations like
2:18:16any other tables inside my model. So
2:18:18with that I don't have to map anything
2:18:19manually and everything going to be
2:18:20automatic. Now so far we have just like
2:18:22table and data and PowerBI has no idea
2:18:25how to do the role of a security. So now
2:18:28what do we want? We want just to connect
2:18:30the users to this security table and we
2:18:32can do that by doing two steps. First
2:18:34PowerBI has to know who are the users
2:18:37that are looking at the reports and for
2:18:39that we have a nice function called user
2:18:41principle name. It's going to return the
2:18:43current user that is opening the
2:18:45reports. So usually going to get the
2:18:46email address. So with this PowerBI
2:18:48knows okay who is looking at the report.
2:18:50Now in order to map this value that is
2:18:53returned from this function to the
2:18:55security table we have to go and create
2:18:57a role. But now of course we will not go
2:18:58and create a role for each value inside
2:19:01the table. We're going to make things
2:19:02more dynamic by creating only one role
2:19:04and we're going to use some DAX
2:19:06calculation in order to just map the
2:19:08value from this function to the security
2:19:11table. So those are the component and
2:19:12now let's understand what going to
2:19:14happen if Maria go and open the reports.
2:19:16Barbie going to go and execute the
2:19:17function and understand okay who is that
2:19:19it's going to get the value. Okay Maria
2:19:21then Barbie going to go to the role in
2:19:22order to understand how to filter and
2:19:24where to go with this information. Then
2:19:26it's going to say okay we have to go and
2:19:28filter the security table and this is
2:19:30very important to understand it is
2:19:32filtering the security table and not
2:19:33going directly to the whole data model
2:19:35and filtering the tables. So that means
2:19:37only the security table going to get
2:19:39filtered to Maria and now what's going
2:19:41to happen this filter going to travel to
2:19:44our data model. So it is very important
2:19:46to have a correct relationship between
2:19:48the security table and your data model.
2:19:51So this is how it works. This is the
2:19:53dynamic role level security. And now
2:19:54let's go and compare the static with the
2:19:56dynamic side by side. In the static RLS,
2:19:59we have to go and create multiple roles
2:20:01based on the values inside our tables.
2:20:03But for the dynamic RLS, we have usually
2:20:05to go and create only one role with a
2:20:08DAX calculation. Now for the static, the
2:20:10users are assigned to the roles inside
2:20:12the PowerBI service. But for the
2:20:14dynamic, the users are mapped inside a
2:20:17security table inside the model in
2:20:19PowerBI desktop. Now for the static if
2:20:21the access change we have to go and
2:20:23change the assignment in the PowerBI
2:20:25service. But with the dynamic if access
2:20:27change we just change the data inside
2:20:29the security table. And here the
2:20:30recommendation is if you have like small
2:20:32projects, small data and only few users
2:20:34and then go and use the static because
2:20:36it is more straightforward and it is
2:20:38easier to implement. But if you have
2:20:40like more data, you have a lot of users,
2:20:42you have bigger projects, then the best
2:20:43practices says go and use the dynamic
2:20:45role of the security. It is yeah at the
2:20:47start not that easy to build but you are
2:20:49investing for the future so things going
2:20:51to be way faster and dynamic once you
2:20:53get a lot of users using your reports
2:20:56and now based on this we will be
2:20:57focusing on the dynamic role the
2:20:59security because it is the method that
2:21:00we usually implement in real projects at
2:21:03companies. So now since this is
2:21:04confusing topic and you have to pay
2:21:06attention to the filters and directions
2:21:08we're gonna have this very simple
2:21:09example where we have a fact table fact
2:21:11sales and two dimensions and now we have
2:21:13to pay attention to the numbers. So here
2:21:14we have the total sales of 10K. We have
2:21:16five customers and five products. So a
2:21:19very classical star schema. Now of
2:21:21course anyone that is using this data
2:21:22model going to see exactly those
2:21:24numbers. Nothing is filtered unless they
2:21:26go and make a filter slicer or something
2:21:29to interact with the report. But if they
2:21:31don't filter anything, they're going to
2:21:33see the whole thing. Now of course we
2:21:34want to make this secure by implementing
2:21:36the dynamic RLS. And we start with the
2:21:39first step where we're going to go and
2:21:40add a security table. Now this table is
2:21:42just mapping between the email address
2:21:44of the users and the region and we have
2:21:46a relationship between this security
2:21:48table to the fact sales since we have as
2:21:50well here the region. But now look at
2:21:52this even if we have the security table
2:21:54everyone is able to see the 10k. So
2:21:56adding a security table and build the
2:21:58relationship doesn't mean the data going
2:22:00to get filtered automatically. For that
2:22:02we have to go and add more steps like
2:22:03for example a rule where it going to
2:22:05check the user principle name. Now let's
2:22:07say that after adding the role we have a
2:22:09user called Alice and she going to go
2:22:11and now open the report and start
2:22:13looking at the data. Now suddenly she
2:22:16will not see the whole 10k as a default.
2:22:18She going to see only 4k even without
2:22:21filtering any data in the report and
2:22:23that's all because of the security role
2:22:25and as well the security table that is
2:22:27connected to our fact. But now we have
2:22:29to pay attention to this. You can see
2:22:31that she is able to see the five
2:22:33customers and the five products and
2:22:35that's all because of the filter
2:22:36direction. We are not filtering from the
2:22:39fact to the dimension. It is the
2:22:40opposite from dimension to fact. So that
2:22:43means it's going to stop over here and
2:22:44will not carry out to the other
2:22:46dimensions. But now if you say you know
2:22:48what I would like as well to filter the
2:22:50dimension customers based on the
2:22:52security then what you have to do is to
2:22:54just go and change the filter direction
2:22:56to both. Now look at this. The security
2:22:58table is filtering the fact and the fact
2:23:00is actually carrying the filter to the
2:23:02dimension customer. But with that Alice
2:23:04going to see 4K and as well only two
2:23:07customers not anymore the five but the
2:23:09dimension product is not affected
2:23:10because here we have one direction but
2:23:12if you say let's go and filter the whole
2:23:14data model then as well you have to go
2:23:16and add like both and with this actually
2:23:18you are filtering the whole data model.
2:23:20So that means my friend you have to pay
2:23:22attention to the filter direction in the
2:23:24relationships because it's going to
2:23:26define the behavior of the security of
2:23:28the whole data model. If you do it wrong
2:23:30then the whole security will not work
2:23:32even if you have the best setup like you
2:23:34have the roles you have the table but
2:23:36the relationship is wrong. Like for
2:23:38example if you just switch it like this
2:23:40where you say the filter direction is
2:23:42from the fact to the security. Now what
2:23:44going to happen? The user going to see
2:23:45the 10k because the data is not
2:23:47traveling from the security table to the
2:23:49fact. So whoever like go and look to the
2:23:52table all of them going to see the 10k.
2:23:54So that means the filter is not
2:23:56traveling the whole setup going to be
2:23:58wrong and you're going to have like
2:23:59security leak that's why the definition
2:24:01of the direction going to be very
2:24:03critical topic. And now for example if
2:24:05you go and switch to pop has different
2:24:07scenario where he is mapped to South
2:24:08America and for that we have like two
2:24:10orders that means pop can see only 3k
2:24:13two customers and two products. Now if
2:24:15you go to Carla, she has the region
2:24:18east. She can see only three orders and
2:24:20she can see three customers, three
2:24:22products. Now of course something that
2:24:23might happen a lot where you have a user
2:24:25that is not actually inside your
2:24:27security table. What going to happen?
2:24:29This user will not see anything. They
2:24:31will see total sales of zero, no
2:24:33customers, no product. Since Dan has no
2:24:35record inside our security table, he
2:24:37will be completely blind and will not
2:24:38see anything. So this is exactly how the
2:24:40dynamic role level security works. And
2:24:42as you can see it is something that is
2:24:44completely depending on the data model
2:24:47and how things are connected. That's why
2:24:49building the relevant security it is
2:24:51part of data moduling. If you do it
2:24:53wrong then you will not be able to
2:24:54connect things nicely like now we are
2:24:56doing. So now back to our data model. Of
2:24:58course here we have only star schema. So
2:25:00we have dimensions facts connected to
2:25:02each others but we don't have security.
2:25:03Trust means everyone going to see
2:25:05everything. Now how we going to do it?
2:25:07Of course we have to do it step by step
2:25:08and we don't have to rush it. Now I'm
2:25:10going to show you how to do the static.
2:25:11It's very simple. So now the first step
2:25:13we have to understand the security
2:25:15requirement in our project. So we cannot
2:25:17go and apply like some random filters on
2:25:19top of our data based on our like gut
2:25:21feeling. If you don't know about that
2:25:23you have to ask your team your
2:25:24department and usually there is like a
2:25:26security officer that understand those
2:25:28topics. Now let's say after some
2:25:30discussions I understood we have always
2:25:32to secure our data based on the regions.
2:25:34So this is very important step because
2:25:36now I know my goal. I want to secure
2:25:38this data model based on the region and
2:25:40not based on the city or the country or
2:25:43some specific departments and so on. Now
2:25:45the next step is that we have to
2:25:47understand where are the regions inside
2:25:49our data model. So that means I start
2:25:51going to the table view and start
2:25:53investigating all my dimensions and
2:25:54facts to understand where I can find the
2:25:56region informations. Now for example
2:25:58here we can see it in the dimension
2:25:59store. This is very good and in other
2:26:01tables like the facts the fact sales we
2:26:04don't see this information. So that I
2:26:06located the dimension store. This going
2:26:08to be a very good starting point. So now
2:26:09that means based on the stores I will be
2:26:12able to use it in order to secure the
2:26:14fact table in the middle. So this I know
2:26:16exactly my target tables and this is
2:26:18very important in order to build the
2:26:20relevant security. Now the third step
2:26:22before I start doing anything I'm going
2:26:24to go to the report view and then make
2:26:26like a special view only for testing.
2:26:28You have always to understand the big
2:26:30numbers of your data model and
2:26:32understand the behavior of the role
2:26:33level security on top of it. Otherwise
2:26:35you don't know whether you have done
2:26:36things correctly or not and you might
2:26:38have some security leaks or maybe are
2:26:40filtering unnecessary things. So I have
2:26:42to memorize it in my head. The total
2:26:44sales in the whole data set is 5500 five
2:26:47customers six products. So that means I
2:26:49know the numbers. I know which table is
2:26:51going to be affected and filtered and I
2:26:53know how to build the roles. Now since
2:26:54we are building the static one you have
2:26:56always to understand the values. So as
2:26:58you can see here the regions we have
2:26:59two. That means we need two roles and
2:27:02those are the values that we're going to
2:27:03use in order to filter the data model.
2:27:05So Europe and USA you have exactly to
2:27:07use those values as it is. So now how we
2:27:10going to build it? We're going to go
2:27:10back to the model view. Now in order to
2:27:12create the roles we're going to go to
2:27:14the rep points over here and we have
2:27:15manage roles. It is very simple. We're
2:27:17going to go from left to right. So we're
2:27:18going to start by creating the roles.
2:27:20Now the first one we're going to go and
2:27:22create the role for EU. And another one
2:27:25we're going to go and create the role of
2:27:27US. So now the next step we have to tell
2:27:29PowerBI which table must be filtered and
2:27:32as we checked the dimension store should
2:27:34be filtered because only here we have
2:27:35the region informations of course if you
2:27:37have multiple tables then go and select
2:27:39multiple tables but now we have only one
2:27:40and now the next step we have to go and
2:27:42create the filter condition and here we
2:27:44have two options either we're going to
2:27:45go and click around if we are creating
2:27:47the static one or we're going to switch
2:27:48to DAX in order to create the dynamic
2:27:51RLS but now we are creating only the
2:27:53static so I'm going to go and say new
2:27:54rule it's just a very simple filter
2:27:56condition so we're going to go and
2:27:57select the column. It's going to be the
2:27:59region. And now the value, we're going
2:28:00to write it exactly like the value
2:28:02inside our data set. So since this is
2:28:04Europe, we're going to write it like
2:28:05this. And it is case sensitive. Now if
2:28:07you just go and read it from left to
2:28:08right, we are creating a new role in
2:28:11order to filter the dimension store. And
2:28:13this is the filter condition. Region
2:28:15must be equal to Europe. So that's it.
2:28:17We have to go and do it again for the
2:28:19US. Yes, we are filtering the dimension
2:28:21store. And then we can go and create the
2:28:23condition. Same thing to the region. And
2:28:25now the value going to be USA. So
2:28:27actually it is very easy. That's it.
2:28:29With that we have created the two roles
2:28:31and now we can go and save it. So that's
2:28:33it and then go and close it. Now of
2:28:35course you might ask okay what happened
2:28:36now. Go to the report view and check.
2:28:39You can see nothing changed. So yes we
2:28:41have created the roles but PowerBI is
2:28:43not filtering anything yet because in
2:28:45order to do that we have to go and
2:28:46publish the report to the PowerBI
2:28:48service and then start assigning the
2:28:50different people to the different roles
2:28:51and then it going to work. But now of
2:28:53course the question is how we going to
2:28:54test it? Are we going to go and create
2:28:56like a dummy user or go and ask someone
2:28:58to test for us? Well, of course, no. We
2:29:00have something smarter in PowerBI. All
2:29:02what you have to do to test is just to
2:29:04go back to the model view and then we
2:29:05have in the report as well something
2:29:07side by side called view as. So, it is
2:29:09very cool. We can go and view the report
2:29:12as a specific role. Like for example, I
2:29:14would like to view the report as the
2:29:15role you. And this is of course very
2:29:17cool because now I can test the report
2:29:19as a person that has this specific role.
2:29:22So now let's go and do that and say
2:29:23okay. Now you're going to see this nice
2:29:25warning. It says now viewing as the role
2:29:28AU. So very cool. But now of course in
2:29:30order to know whether this works. We
2:29:32have always to test by going back to the
2:29:34report view. And look at this. So you
2:29:36can see in the fact we are seeing only
2:29:38the orders with the region Europe not
2:29:40from USA. You can see as well the total
2:29:43sale did change. So that means my friend
2:29:45this role is working. Now let's go and
2:29:47test the opposite. So I'm going to say
2:29:48again okay views and then we're going to
2:29:50go and switch to US and then hit okay.
2:29:53Now it says viewing as RO US. Let's go
2:29:56and test. Look at this. Now we are
2:29:58seeing the orders only from the region
2:30:00USA. So as you can see it is working.
2:30:02And now as you might noticed the total
2:30:04products and the total customers they
2:30:05are not changing and that's all because
2:30:07of the star schema relationships that we
2:30:09have created. You can see the filter
2:30:11directions are always from the
2:30:13dimensions to the facts. So that means
2:30:14the filters will not reach the
2:30:16dimensions. Now, of course, if you want
2:30:17to test, we can go and try this out by
2:30:20going, for example, to the relationship
2:30:22between the fact sales and the
2:30:23dimension. Now, of course, we're going
2:30:24to go to the direction instead of
2:30:26single, we're going to go and switch it
2:30:27to both. Now, here we have another
2:30:29option. It says apply security filter in
2:30:31both directions. Well, yes, this is
2:30:33exactly what we want. So, that's it.
2:30:35Let's go and save. Now, we can see the
2:30:37filter is in both directions. So, the
2:30:39filtering should reach now the products.
2:30:41Let's go and test. Look at this. We
2:30:42moved from six products to five
2:30:44products. So that means not only the
2:30:46fact table is getting filtered by the
2:30:48role of the security but as well our
2:30:50dimension. So as you can see my friends
2:30:52this is the power of the relationships
2:30:54and the filter directions. We can
2:30:55control which tables must be secured
2:30:57based on the security requirements. So
2:30:59actually you have it. This is how we
2:31:01create a static role security. Now of
2:31:04course let's move to that dynamic one.
2:31:06So now how we going to do it? First of
2:31:07all I'm going to go and remove all those
2:31:08stuff. So we don't need you. Same thing
2:31:11for you. And then reset everything back.
2:31:14Now before I start creating any roles,
2:31:16we actually need a security table. So I
2:31:18need a mapping between a user and their
2:31:21regions. Now there is like two options
2:31:23for that again. Either you load it from
2:31:25outside or if it's like only few users,
2:31:27you can go and enter data for that. But
2:31:29for this I have prepared something for
2:31:31you. So if you go and connect a new CSV,
2:31:33there is here a table called security.
2:31:35Let's go and connect it. So now look at
2:31:37this. We have user emails and the
2:31:39regions. Now we can see that the first
2:31:41row is actually not the header. So
2:31:42[clears throat] that's why I'm going to
2:31:43go and transform the data quickly. And
2:31:45all what you have to do is to just use
2:31:46first row as a headers. All right. This
2:31:49looks way nicer. Now let's go and close
2:31:51and apply. All right. So with this we
2:31:52got our new security table. Of course
2:31:54the next step we have to go and connect
2:31:56it to our data model. So we're going to
2:31:58go through the dimensions. We're going
2:31:59to connect the regions per region. Okay.
2:32:01So now here we have like many to many
2:32:03relationship which is totally fine. What
2:32:05is very important is the filter
2:32:07direction. So we have to pick this. The
2:32:09security filters the dimension store.
2:32:11This is very critical to do it correctly
2:32:14otherwise it will not reach our data
2:32:16model. So actually that's it. Let's go
2:32:17and save it again. Look at this. The
2:32:20arrow is moving from the security to the
2:32:22dimension and the dimension is moving to
2:32:24the facts. This is very important one.
2:32:26All right. So now if you go of course to
2:32:28the report view, you're going to see all
2:32:29the data. PowerBI don't know yet how to
2:32:31use the security table. So nothing
2:32:33actually going to happen. That's why
2:32:35we're going to go back to the model
2:32:36view. And of course comes the magic.
2:32:38We're going to go and create a new role.
2:32:40So again manage roles this is not static
2:32:42we are working on the dynamic we will
2:32:44not go and create like a role for each
2:32:47region it is enough to have only one. So
2:32:49let's go and create one and let's go and
2:32:51call it regional access. Now of course
2:32:54the question is which table should be
2:32:56filtered based on this role. It's again
2:32:58the same thing the dimension store. And
2:33:00now since the dynamic it is little bit
2:33:02more advanced we will not go and create
2:33:03here a new rule. It is not a simple
2:33:05filter. We're going to go and switch to
2:33:07DAX in order to write the filter
2:33:09expression for PowerBI to understand how
2:33:11to filter the dimension. Okay. So now
2:33:13let's do it step by step. We just want
2:33:15to create the filter condition. And the
2:33:17first question is what we are actually
2:33:19filtering. We are filtering the regions
2:33:21inside the dimension store. So that
2:33:23means I'm going to go and write it like
2:33:24this. The region inside the dimension
2:33:26store should be equal to the following
2:33:28value. Now if you are just creating like
2:33:30a static RLS, it's going to be like
2:33:32this. So equal to Europe or USA. But
2:33:34actually we will not do that. We want to
2:33:36make things dynamic. Now the first thing
2:33:38that we have to decide is actually we
2:33:40will not go and use the equal operator.
2:33:42We're going to go with the in. And
2:33:43that's because one user might have
2:33:45multiple regions. So that means we might
2:33:47have a list and in order to filter using
2:33:50a list we have to use an n operator not
2:33:52an equal. Equal is only for one value.
2:33:54Now the regions should be filtered based
2:33:56on the other table. The security. So
2:33:58that means we have to get the values of
2:34:00the security in the filter and we can do
2:34:03that using the function calculate table.
2:34:05So we're going to tell PowerBI go and
2:34:07grab the values from another table based
2:34:10on a filter. So that means we're going
2:34:11to go and use the values and actually we
2:34:13need the security region. Perfect. With
2:34:16that we are getting the regions from the
2:34:17security but of course I don't need all
2:34:19the regions. I need only the regions of
2:34:22specific user that is actually opening
2:34:24the reports. That's why we have to go
2:34:25and build a filter. And of course comes
2:34:28now the magic the most important thing
2:34:29we're going to say filter the emails
2:34:31inside the security but based on the
2:34:34email of the person that is actually
2:34:36looking at the reports and for that we
2:34:38have the very nice function user
2:34:40principle name. This function will just
2:34:42return the email address of the current
2:34:44user. So with that we will not get all
2:34:45the regions from the security only for
2:34:47the one that is looking at the report.
2:34:49And actually that's it. So now let's
2:34:51read it again. filter the regions inside
2:34:53the dimension store based of the values
2:34:56of the security regions but of course
2:34:59not all the regions only for the regions
2:35:01that are equal to the email address of
2:35:04the current user so this is exactly how
2:35:06we build a dynamic role level security
2:35:08now we can go and verify the DAX looks
2:35:10good we're going to go and save it and
2:35:12actually that's it we have only one role
2:35:14we are filtering one dimension and we
2:35:16have a dynamic flexible expression so
2:35:18let's go and close it of course we have
2:35:20to test if you go to the report view.
2:35:22You can see nothing is changed. We still
2:35:23have all the orders. Of course, the
2:35:25effect is going to happen once you
2:35:26publish to PowerBI service. Again, we
2:35:28will not go and ask a user to test for
2:35:30us. We can test it directly in the
2:35:32desktop. So, we just need like an email
2:35:34address to test. I'm going to go to the
2:35:35security and go and pick Maria. I'm just
2:35:38go and copy the value. Now, in order to
2:35:40test, we're going to go to the model
2:35:41view and then view us. Now, again, we're
2:35:43going to go and activate the regional
2:35:45access and then say okay. Now, as usual,
2:35:48we're going to see this warning. And now
2:35:49we are viewing as the regional access.
2:35:51Of course, we have to test. If you go to
2:35:52the report view, we will not see
2:35:54anything. Well, what is going on? Now
2:35:56the thing is PowerBI is trying to get my
2:35:59email address that is currently looking
2:36:01to the report and try to filter the data
2:36:03based on that. And since I don't have my
2:36:05email address inside my data, this is
2:36:08just like fake one. I will not find any
2:36:10match over here. And it going to say
2:36:11okay, you are not allowed to see
2:36:13anything. That's why I cannot see
2:36:15anything in the fact table, which is a
2:36:16good thing of course. But in order to
2:36:18really test this is not a bad effect.
2:36:20We're going to go to the view as and
2:36:22this time we're going to say other user
2:36:23not my user. So I would like to see the
2:36:25report as Maria is looking to the
2:36:27report. So we're going to go over here
2:36:29and put her email. So that I'm going to
2:36:31see exactly how she can see the report
2:36:33and say okay. Now you can see now
2:36:34viewing as a regional access and the
2:36:37email address. So Maria now let's go and
2:36:39test to the report view. Look at this.
2:36:41Now we can see only Europe data and
2:36:43that's because Maria has access only to
2:36:45the region Europe. So this is perfect.
2:36:48Now let's go and test another user. So
2:36:50we're going to say okay not Maria this
2:36:53going to be John and then okay. You can
2:36:55see now we are viewing as John. Let's
2:36:57see what he can see. Well John has
2:36:58access only to region USA. That's why
2:37:01the report is now showing only USA
2:37:03region. So this is exactly the dynamic
2:37:05the report the data model going to be
2:37:07filtered exactly for specific user and
2:37:09if the user doesn't exist like the one
2:37:11that I had at the start then the user
2:37:13will not see anything. Now let's go and
2:37:15test the fake one that we had. So I had
2:37:17another one called bar which has of
2:37:19course all the access of all regions.
2:37:21Now we can see okay we are viewing the
2:37:23data as bar Gmail. Now let's go and
2:37:25test. You can see now I'm able to see
2:37:27multiple regions which is totally okay.
2:37:29So even if a user like me with multiple
2:37:32access to multiple regions the role
2:37:34level security is working because we are
2:37:36using the in operator and actually my
2:37:38friends that's it. This is how we create
2:37:40a dynamic role level security in PowerBI
2:37:43by using a very simple expression index.
2:37:46All right friends, so now let's do a
2:37:47quick recap. Security is a very
2:37:49important part in data moduling. Do the
2:37:51data moduling correctly. Then it's going
2:37:53to be really easy to build a security on
2:37:55top of it. Security have three types.
2:37:57Table level, column level, and rowle
2:37:59security. Of course, the most common one
2:38:01that is used in project is the rowle
2:38:03security RLS. It's all about how to
2:38:05filter the rows of your tables to show
2:38:07only the allowed rows. And there are
2:38:09actually two ways on how to build the
2:38:11row level security. We have the static
2:38:14RLS to filter only the rows based on a
2:38:17static value. It works great if you have
2:38:19a small projects with few users. But if
2:38:21your project is growing over the time,
2:38:23then you have to switch to more flexible
2:38:25way by building a dynamic RLS where
2:38:28you're going to filter the data based on
2:38:30users not static values. And in order to
2:38:33do that, we need some extra stuff like
2:38:35for example a security table where it
2:38:37going to map the users to specific
2:38:39values of your data model. Then you have
2:38:42to be very careful how to connect this
2:38:44security table to your data model by
2:38:46using the right direction of the
2:38:48filters. Otherwise, nothing going to
2:38:49happen. And the last step, we're going
2:38:51to go and write a very simple DAX
2:38:53expression in order to let PowerBI know
2:38:55how to filter your data model based on
2:38:57the security values. Of course in real
2:38:59projects we don't rush and build the
2:39:01role securities. We have always to do it
2:39:03step by step and slowly. So first we
2:39:05have to understand the security
2:39:07requirement in our domain. Usually we go
2:39:09and ask a security officer or someone
2:39:11that knows about those topics. We don't
2:39:13go and randomly apply security stuff
2:39:15inside our data model that is totally
2:39:17maybe unnecessary. So don't follow your
2:39:19guts about this. Always ask and make
2:39:21sure that you are implementing what is
2:39:23needed. Once you understand the
2:39:24requirements then understand where you
2:39:26can apply it inside your data model.
2:39:28Don't try to force it in all tables
2:39:30unless it's needed. And of course, not
2:39:32only that as well, your filter should be
2:39:34able to reach those tables and
2:39:36dimensions. But again, we secure only
2:39:38the things that is required. We don't
2:39:40have to go and like secure every single
2:39:42table inside the data model. So don't
2:39:44make your life hard if it's not
2:39:46required. And what is very important, we
2:39:48always have to test. That means you have
2:39:50to know the big numbers of your data
2:39:51model by heart and you have always to
2:39:53test the effect of the RLS. So actually
2:39:56that's it. This is the role of security
2:39:57in PowerBI. I know that was not an easy
2:40:00one. This is an advanced topic. But once
2:40:02you understand it, you can go and apply
2:40:03it [music] to any project.
2:40:08Hey friends. So now we're going to do
2:40:09something amazing. We're going to go and
2:40:11build data model projects in PowerBI
2:40:13exactly like I do in my real projects.
2:40:15The thing is I have been involved in
2:40:17many PowerBI projects at my work that
2:40:19had bad struggle with the performance
2:40:21and as well delivering wrong numbers.
2:40:23But after investigating it was not the
2:40:25dashboard was bad or ducks. Almost each
2:40:27time it was the bad model underneath the
2:40:30reports. So that's why I have prepared
2:40:32for you for this project a nightmare
2:40:34data set that looks exactly like the one
2:40:36that I have seen in companies. Full
2:40:38chaos, bad shapes, bad data models and
2:40:40as well bad relationships. So now what
2:40:42I'm going to do, I'm going to walk you
2:40:43through this data model step by step
2:40:45exactly like I do in my real project. So
2:40:47that at the end we're going to get a
2:40:49healthy data model that is ready for
2:40:51reporting. And yes, this is the best way
2:40:53to learn data modeling by doing
2:40:55projects, not just like watching
2:40:57disconnected videos. So that at the end,
2:40:59you develop like a feeling for the whole
2:41:01process end to end. And very important,
2:41:03you don't just watch me doing the
2:41:05things, you always work with me. So for
2:41:07each step, you pause the video, you do
2:41:09it at your end, and then you continue.
2:41:11All right, my friends. So this is going
2:41:12to be fun project to do. And now let's
2:41:14get started.
2:41:15>> Battle stations, everybody. Let's go,
2:41:16go, go, go, go, go. Okay, so now before
2:41:19we jump in, I'm going to show you the
2:41:20project faces for the data moduling. So
2:41:22first we have to prepare. We're going to
2:41:24open the mess. We're going to set up the
2:41:25workspaces in power query and then we're
2:41:28going to start investigating. So we're
2:41:30going to start exploring and have basic
2:41:32understanding of the tables. What is the
2:41:34business? What are the entities? Which
2:41:36one are good candidates for dimensions?
2:41:38Which are for facts. So we're going to
2:41:40develop like a big picture in our head.
2:41:42The second phase we start building. So
2:41:44we start building the dimensions. We're
2:41:46going to find like a group of tables
2:41:47that relate to each others and turn them
2:41:50into clean dimension. So we keep
2:41:52iterating until we build all the
2:41:54dimensions for the data model. Then the
2:41:56third step we're going to start building
2:41:57the facts and connect the dimensions to
2:41:59them. So we take all those event tables
2:42:02and start building the fact. And after
2:42:03that we connect all the dimensions to
2:42:05our new facts and start testing of
2:42:08course so that the numbers never break
2:42:09silently in our data model. And then the
2:42:11last phase is going to be the polishing
2:42:13phase. We're going to go and add
2:42:14security to our data model and then we
2:42:16do like final test to validate the whole
2:42:18data model. So this is the whole project
2:42:20those are the phases that we're going to
2:42:22go through and as well before we start
2:42:23like building anything we have to keep
2:42:25in our mind those rules we're going to
2:42:27follow them from the start to the end no
2:42:29matter which step we are in. So the
2:42:32first rule or goal is to build star
2:42:34schema fact table in the middle
2:42:36surrounded by dimensions and we never
2:42:38connect fact table together. The second
2:42:41rule we have always to understand before
2:42:43do any changes specially we have to
2:42:45understand always first the grain of
2:42:47each table. We have always to say it
2:42:49loud in order to understand the grain
2:42:51otherwise we're going to build wrong
2:42:52dimensions and facts. Now the third rule
2:42:54every info every column inside our data
2:42:56model must earn its place. So that means
2:42:59if we find any column that is not really
2:43:01needed for analytics and reporting we
2:43:04have to go and drop it. And that's
2:43:05because of two reasons. First of all,
2:43:07they are wasting space inside our
2:43:09PowerBI file. If you have a lot of
2:43:10unnecessary columns, your PowerBI file
2:43:13going to be really heavy and hard to
2:43:14refresh and hard to move. So that's why
2:43:16we remove them to have some space. And
2:43:18the second and more important, they
2:43:20going to confuse you during building the
2:43:22visuals and reports. So if we find
2:43:24anything unnecessary, we just go and
2:43:26drop it. We can always go and bring it
2:43:28back if we need it. Rule number four, we
2:43:30always have to protect the numbers. We
2:43:32must know by heart the totals of our
2:43:34data model so that each time we do like
2:43:36a change or we connect something new. We
2:43:38go and check those numbers to make sure
2:43:40that they don't break. And now we come
2:43:42to the final rule or let's say big step
2:43:45we have to make sure that we are
2:43:46following standards that we have to
2:43:48define before starting the projects. And
2:43:50of course those standards we can define
2:43:52them from one project to another depends
2:43:54on the requirements. But for this
2:43:56project the standards going to be like
2:43:57this. First of all we have to define the
2:43:59language. Yes, this sounds like simple
2:44:01but I saw a lot of projects where they
2:44:02are mixing languages in the data model.
2:44:05So yes, the language going to be here
2:44:06English. Then we have to follow the same
2:44:08naming convention for the table and the
2:44:10columns. And of course there are a lot
2:44:12of naming conventions but for these
2:44:14projects we're going to follow the snake
2:44:15case. All letters, all characters are
2:44:17lowered and they are separated with
2:44:19underscore. Now as well you have to
2:44:21define the prefixes and the suffixes.
2:44:23Like for example in the table names if
2:44:25we have a fact table we can have the
2:44:26prefix fact core. Like for example fact
2:44:30sales and if it is dimension is going to
2:44:32be dim shortcut for dimension like only
2:44:34three letters underscore then the
2:44:36dimension name like for example dim
2:44:38customer this is really important
2:44:40because I saw a lot of confusion if a
2:44:42lot of people are working on the same
2:44:43powerbi project one going to call it d
2:44:46customer and another developer going to
2:44:48call it dimension customer so we have to
2:44:51set up the standard for our projects
2:44:53before we start building now about the
2:44:55columns we as well could bring some
2:44:57standards like For example, the
2:44:58surrogate keys, anything that we create
2:45:00inside the data model going to ends with
2:45:02the suffix underscore key like for
2:45:04example customer key, product key. So
2:45:07once you see in the data model, okay, it
2:45:09ends with the underscore key that means
2:45:11we created that. It is not a natural key
2:45:14that cames from the source like for
2:45:15example the customer underscore ID. This
2:45:18is something comes from the source file
2:45:19and the customer underscore key it is a
2:45:22surrogate key that we just created in
2:45:24the model. So that you have understand
2:45:25the different types of your keys in the
2:45:27model. Now the last rule we have to make
2:45:29everything friendly. So that means we
2:45:31have to give for everything the table
2:45:33name, the column name, even some values
2:45:35inside our tables friendly meaningful
2:45:38name. So that once you start building
2:45:39the reports the values can be friendly
2:45:41and everyone can follow the reports. We
2:45:44don't bring any cryptical shortcut
2:45:46technical names that we inherited from
2:45:49the source systems. So this is the
2:45:50setup. Those are the faces, the rules,
2:45:52the mindsets that we're going to follow
2:45:54through the whole project. Now, let's
2:45:56get started with our PowerBI project.
2:45:58All right, friends. So, now I'm really
2:45:59glad to present you the nightmare data
2:46:02set that you're going to use for this
2:46:03project. Have a look to this beautiful
2:46:05chaos because my friends, there is high
2:46:06chance if you join a company, join
2:46:08PowerBI projects that their data model
2:46:10looks exactly like this. I have seen
2:46:12this in many projects where the data
2:46:14model makes no sense. You cannot see
2:46:16like a star schema here. Many to many
2:46:18relationships between tables. The filter
2:46:21direction is like everywhere and that's
2:46:23all because people skip the data
2:46:25modeling step. They just connect the
2:46:27source systems to PowerBI and jump
2:46:29immediately to the visuals to the report
2:46:31dashboards. They just want to build
2:46:33fancy reports for the end users. But of
2:46:35course, after a while once they start
2:46:37sharing the dashboards, the users can
2:46:39understand all the numbers makes no
2:46:40sense. they usually come back to this
2:46:42model and start fixing the data model in
2:46:45order to bring better numbers. So my
2:46:47friends what I have done for the
2:46:48preparations I brought all the scenarios
2:46:49from my project and build this nightmare
2:46:51data set in order to cover all the
2:46:53scenarios that you might face in the
2:46:55projects. So our goal is to turn this
2:46:58into a healthy star schema bring correct
2:47:00numbers and as well has good
2:47:02performance. [music]
2:47:06Okay. So now we're going to start with
2:47:07the phase number one the preparation.
2:47:09That means first we have to have good
2:47:11understanding about the content of those
2:47:13tables. So what is the business? What
2:47:15are the entities? Which tables are good
2:47:17candidates for dimensions? Which are for
2:47:19facts. So basic understanding about what
2:47:21we have and we don't have like to deep
2:47:23dive into each small detail just to
2:47:25develop a feeling about the business
2:47:27about the data and after that we're
2:47:28going to do like few setups in the power
2:47:30query as a preparation for the project.
2:47:32So now let's start let's understand what
2:47:34you have here and I will not go through
2:47:35all the relationships because we're
2:47:37going to break the data model and anyway
2:47:39we have like to build the relationships
2:47:40again. So just basic understanding of
2:47:42the tables and we're going to do that by
2:47:44going to the table view. So we're going
2:47:46to start from the top to the addresses.
2:47:47So what do we have here? We have an
2:47:49address ID, street, city name. So it's
2:47:51all about the geographic, the addresses
2:47:53and maybe for the customers or
2:47:55something. So this is for the first
2:47:56table. Let's go to the second one. The
2:47:58campaign log. Okay, this one is a little
2:48:00bit bigger. So here we have the
2:48:01[clears throat] campaign name, the
2:48:02channel, start date, end date, a budget,
2:48:04impressions, click and spend. So it's
2:48:06all about our marketing campaign day by
2:48:08day. So this looks like an event and
2:48:10this is really good candidate for a
2:48:12fact. So we have numbers, we have dates.
2:48:14Now let's go to the third one. We have
2:48:15the campaign SKUs. This one looks
2:48:17compacted. So we have the campaign name
2:48:20again. And for each campaign, we have
2:48:21like a list. And I think those are the
2:48:24products that we have in the business.
2:48:25So that means this table shows us what
2:48:27are the products that were involved in
2:48:30the campaign. And of course this is not
2:48:32easy one to consume because the list is
2:48:34usually bad for reporting and we can see
2:48:36as well the column names are not correct
2:48:39and we can find them in the first row
2:48:41inside our table. Of course we have to
2:48:42clean those up. Now let's go to another
2:48:44table the cities. So what do we have
2:48:46here? We have again the cities and as
2:48:48well the regions. So this is again a
2:48:50part of the informations about the
2:48:52geography, the locations, the addresses.
2:48:54This is of course a descriptive
2:48:56informations and we're going to need
2:48:57those stuff as a dimensions. Now let's
2:48:59go to the next one. We have the custom
2:49:01master. So it looks like this is our
2:49:03customer table. So we have customer ID,
2:49:06customer name, segments, the address ID.
2:49:08So this is the first connection to our
2:49:10addresses. Then we have like people, the
2:49:12account manager, payment terms, some
2:49:14hash keys and source ids. Those stuff
2:49:17comes from the sources. But now look at
2:49:19this. The customer name is not actually
2:49:21like a person. They are companies. Like
2:49:23here for example we have the Atlas
2:49:25industries. So that means this is a very
2:49:27important table for the customers and
2:49:29our business is B2B. So that's all for
2:49:31this one. Let's go to the next one. We
2:49:33have the customer contacts. So what do
2:49:35we have here? We have again the customer
2:49:36ID, the contact name, the emails is
2:49:39primary. Now you can see for each
2:49:41customer like for each company we have
2:49:43multiple contacts which makes sense of
2:49:45course because our customers are
2:49:46companies and usually we have to deal
2:49:48with multiple people in the same
2:49:49company. So this is all about who we
2:49:51talk inside each company. Let's go to
2:49:53the next one. We have a dimension
2:49:55underscore order. So this one looks
2:49:57really garbage. We have only one column
2:50:00and no context to it. So we have some
2:50:02kind of ids that actually we cannot
2:50:04relate to anything so far. So I think
2:50:06this is junk. This is something that we
2:50:08could drop. So we just faced our first
2:50:10garbage inside the data model. Let's go
2:50:12to the next one. We have the exchange
2:50:13rate. So what do we have here? We have
2:50:15the currency, the rate and as well the
2:50:17date. So this is as well information
2:50:19about our business. Let's keep going to
2:50:21the inventory. So now what do we see
2:50:23here? We see the product names and for
2:50:25each month we have like numbers. So this
2:50:27all about the inventory. So that means
2:50:30how much stock we are holding month by
2:50:32month. And here we have like a whole
2:50:34year. So for each month we have one
2:50:36column. All right. Let's move to the
2:50:38next one. We have the invoice lines. So
2:50:40here we have like an invoice line, an
2:50:41invoice ID, product and amount. So that
2:50:44means we have invoices in our system and
2:50:46those are the details about each invoice
2:50:48and that brings us to the next table the
2:50:51invoices and here we see like the master
2:50:53table for the invoices. So we have the
2:50:55invoice ID the order ID the customer
2:50:58name the invoice date and the amount as
2:51:00well. So this looks an event a good
2:51:02candidate for a fact table. So it is
2:51:04connecting to other things like the
2:51:05invoice ID the order ID the customer
2:51:07name to the customer table. We have a
2:51:09date and numbers. And if we need more
2:51:11details about the invoices, we have the
2:51:13other table that we just checked, the
2:51:14invoice lines. So we just puted another
2:51:17fact. Now let's move to the next one. We
2:51:19have the order line items. So again
2:51:21sounds similar to the invoices. We have
2:51:23the details about each order. So for
2:51:25each order we have like lines and order
2:51:27ID. And this time we are connecting to
2:51:29the product names. So finally we see
2:51:31here the products and then we have here
2:51:33some numbers like the quantity, the
2:51:34price, the cost, discount and total
2:51:37sales. So those are the details for each
2:51:39line. Now we're going to go to the
2:51:41orders. So we have here two orders where
2:51:43the orders are splitted by the year. So
2:51:45for 2025 we have here a lot of things.
2:51:48So here we have an order ID, a legacy
2:51:50reference, customer name, customer city,
2:51:52region. So again we have like some
2:51:54repetitions inside our data set to which
2:51:56city we're going to ship to which city
2:51:57we're going to pill. So we have here
2:51:59different cities. Then we have the date,
2:52:01the order date, the order channel and a
2:52:04lot of columns in this table. So again
2:52:06this looks like a big table for a fact
2:52:09because here we have an events each line
2:52:11is about an order but as well at the
2:52:13same time we have a lot of context
2:52:15inside these facts. So this is a
2:52:17gigantic one for the 2025 and I think
2:52:19the 2026 looks exactly the same. So they
2:52:23are just getting splitted. All right. So
2:52:25this is our third fact I believe but
2:52:27inside it we have a lot of dimensions
2:52:29that we could extract. Now let's go to
2:52:31the next one. We have the payments. So
2:52:32this one is a small one. We have the
2:52:34payment ID, the invoice ID, customer
2:52:36name, date and amount. Again, looks like
2:52:38transactions events. So this sounds like
2:52:40a fact table as well. And now to the
2:52:43next one which is really important, the
2:52:45products. Finally, we have a dimension.
2:52:47So what do you have over here? We have
2:52:48the product code, the name, the brand,
2:52:50the subcategory, the price and many
2:52:53other descriptions for the products. So
2:52:55this table holds the context for the
2:52:58business products which is a classical
2:53:00one for a dimension. Let's go to the
2:53:02next one. We have the regions. Well,
2:53:04again we have informations about the
2:53:05region and as you can see we are
2:53:07operating globally. North America,
2:53:09Europe, Asia. So our business is
2:53:11everywhere. Now let's go to the next
2:53:12one. We have the sales targets. So we
2:53:15have a row for each month and as well
2:53:17the target revenue. So it looks like
2:53:19someone planned the budget the revenue
2:53:21for each month. And this is a classical
2:53:23use case where we're going to compare
2:53:25the target to the actual sales that we
2:53:27have in the transactions. So for sure
2:53:29we're going to need this. Now let's go
2:53:30to the next one. We have the security.
2:53:32This is really important one. So here we
2:53:34can see the user email and the region.
2:53:36So we can see each employee which region
2:53:39they are responsible to. And of course
2:53:41this is a perfect table that we're going
2:53:42to use in order to build the role level
2:53:44security. So we have to keep our eyes on
2:53:46it. Now let's go to the next one. We
2:53:48have the sheet one. Weird name right? I
2:53:50think this is like something wrong here.
2:53:52So we have the shipment ID, order ID,
2:53:54mode, ship day, delivery date. So all
2:53:57informations about the delivery. So we
2:53:59can see for each step how and when
2:54:01orders get out. Okay, interesting one.
2:54:03But we have to rename the table. Now
2:54:05let's go to the next one. We have the
2:54:06shipments. Well, those two tables are
2:54:08identical. And yes, I think the sheet
2:54:11one is an import problem. And this is a
2:54:13classical thing that happens inside the
2:54:15data model that you have like two tables
2:54:17that are identical and you have to get
2:54:19rid one of them. So this is not really
2:54:21good. Now let's go to the subcategories.
2:54:23So very simple one we have here like
2:54:25mapping between the category and the
2:54:27subcategory. Of course, this is going to
2:54:28help us in order to organize our
2:54:30products later. Now, let's go to the
2:54:32last table, the user details. So, we can
2:54:34see here for each user we have like
2:54:37credit limits and a phone number. Well,
2:54:39of course, this is confusing which user
2:54:41we are talking about. I think this going
2:54:43to be for the customers. If you go and
2:54:45check the customer master, well, yes, we
2:54:47have like the same patterns in the ID.
2:54:50So, those are more details for the
2:54:52customers. So, that means in the data
2:54:53sets we are using like two names.
2:54:55sometimes users, sometimes like
2:54:57customers. Of course, we have to deal
2:54:58with that. All right, my friends. So
2:55:00that's it. We just visited all the
2:55:02tables. We developed like a feeling
2:55:04about the business. What are the
2:55:05entities, what are good candidate for
2:55:07dimensions? Of course, here the big
2:55:09dimensions going to be the customers and
2:55:10the products. And as well, we have a lot
2:55:12of events happening in the business. So
2:55:14we have a lot of facts. We have
2:55:16invoices. We have orders. We have
2:55:18payments. And as well, we have
2:55:19shipments. So campaign. So a lot of
2:55:22events are happening in the business.
2:55:23And this is going to be really
2:55:24interesting on how to make a data model
2:55:27out of this and how to connect things
2:55:29together. So as you can see my friends
2:55:30this is really important step. We have
2:55:32to get a feeling about the data before
2:55:34we change anything. The more you explore
2:55:36the more you analyze you build this
2:55:38understanding about the data about the
2:55:40domain and it's going to be easier for
2:55:42you later to understand the grain and
2:55:44how to build the dimensions and facts.
2:55:46The more you invest in this phase the
2:55:48less errors and mistakes you're going to
2:55:50do in the data moduling. Now, of course,
2:55:52I have done this quickly because I have
2:55:54built these data sets. But I really
2:55:55recommend you to pause the video and go
2:55:57through them one by one, understand its
2:55:59contents. If you have done this already,
2:56:01now we're going to go and do a last step
2:56:03in this phase where we're going to
2:56:04prepare the power query because we're
2:56:05going to work a lot with the power query
2:56:07in order to shape the data model. Not
2:56:09everything going to be like connected.
2:56:11We have sometimes to merge, append, and
2:56:13as well clear some quality issues. So
2:56:15now let's go to power query. As usual,
2:56:17we go to transform data. Now of course
2:56:18in this phase we will not go now and
2:56:20transform or change any table. All what
2:56:22we want is to just organize it because
2:56:24the thing is as you start like building
2:56:26the data model you will be creating new
2:56:28tables references and quickly the
2:56:30queries over here going to go in chaos.
2:56:32So you will not remember which tables
2:56:34are the source table which tables are
2:56:36the new one that you just created. So
2:56:38this going to get out of hand quickly
2:56:40especially if you have like big model
2:56:42like this. Now of course there is like
2:56:43no best practices how to organize your
2:56:45project but I'm going to show you how I
2:56:47usually do it. So everyone does it in
2:56:48their own way on their own style but
2:56:50this is how I like to do it. So I start
2:56:52by creating a new group. So new one and
2:56:55we're going to give it a name. So I like
2:56:56like numbering the folders just to sort
2:56:58it on the left side. So the first one
2:57:00going to be the stage. So the stage
2:57:02going to be the folder that going to
2:57:03contain all the row queries as it is
2:57:06from the source systems. So that mean is
2:57:07my friends all those queries that we see
2:57:09now on the left side going to be inside
2:57:11this folder. So that I know okay all the
2:57:14row stuff are living inside this folder.
2:57:16with that we don't mix it with new
2:57:18tables that we generate. So let's go and
2:57:20create it. Now you can see PowerBI put
2:57:22everything under other queries which is
2:57:24a good thing. Now I'm going to go and
2:57:26select everything and drag and drop them
2:57:28on the new folder. So this my new folder
2:57:30the stage contains all the row sources.
2:57:33So this is the first folder. We're going
2:57:34to go and create another one and we're
2:57:36going to call it two dimensions. Now,
2:57:39this is going to be an empty folder for
2:57:40now because we're going to put inside it
2:57:42the new dimensions that we create from
2:57:44the stage like the customers, the
2:57:46products and so on. So, that means if I
2:57:47want to go and do something for the
2:57:48dimensions, I go immediately inside this
2:57:50folder and I'm going to see it. Now,
2:57:52usually after I build the dimensions,
2:57:53I'm going to go and build the facts.
2:57:55That's why the third one going to be the
2:57:57fact folder. So, same thing, any fact
2:58:00that I build from the stage, it going to
2:58:01land inside the folder facts. Of course,
2:58:03you can go and put them together like
2:58:05maybe call it the data model where the
2:58:07prefix is going to decide for you
2:58:08whether it's fact or dimension. But I
2:58:10like to split it like this in order to
2:58:12have like a quick access. Now we're
2:58:13going to go and create a last one. So a
2:58:15new group. We're going to call this for
2:58:17support. Now in our data model, we're
2:58:19going to have some tables that don't
2:58:21belong to dimensions and facts. Like for
2:58:23example, the security stuff or maybe
2:58:25some like data that we insert. So
2:58:27anything that don't belong to dimensions
2:58:29and facts going to go to this folder,
2:58:31the support. Now you might ask okay why
2:58:33do we have the support if we have like
2:58:34another folder called others right we
2:58:36could use it as well but the other
2:58:38queries has different job so in case we
2:58:40connected any new tables any new sources
2:58:42to our data model it going to land
2:58:44inside the other queries they are like
2:58:46tables that are in my to-do list that I
2:58:48have to work on so my friends this is
2:58:50how I usually organize my power query in
2:58:53order to build the data model out of the
2:58:55row tables now look at this we have 23
2:58:57tables inside the stage and nothing
2:59:00inside other tables of course As we
2:59:02progress in the projects, we're going to
2:59:03take all those tables and put them
2:59:04inside the different folders. We have to
2:59:06grab all those stuff and start building
2:59:08the data model. All right, friends. So,
2:59:10that's what's for the phase one. We have
2:59:11now basic understanding about the
2:59:13content, the tables, and we have
2:59:14prepared our projects to start building.
2:59:16Finally, we're going to kick off the
2:59:18phase two where we start building the
2:59:20dimensions.
2:59:24This is how we're going to do it. So,
2:59:25we're going to have like a loop. We're
2:59:27going to go through each dimension where
2:59:29we're going to grab a group of tables
2:59:30that describes the same thing. So the
2:59:33same entity. So all tables that have the
2:59:35same context for something like a
2:59:37customer. So what we're going to do,
2:59:38we're going to go and pull them
2:59:39together, reshape them into one clean
2:59:42dimension if it's possible. And we're
2:59:44going to do whatever it takes. So we're
2:59:46going to split, merge, append, fix, drop
2:59:49until we get the shape of a dimension.
2:59:51And of course before we move on to the
2:59:53next one we have to polish the dimension
2:59:55to make sure that we are following the
2:59:57standards that we agreed on the starts.
2:59:59So the snake case names the table names
3:00:02the prefixes the suffixes everything.
3:00:04Once it is polished then we go to the
3:00:06next dimension. So my friends this is
3:00:08the whole loop. We pick tables build
3:00:10clean it to the standards and then we
3:00:12move to the next one. But I'm going to
3:00:13warn you from now you might find
3:00:14yourself that you are repeating the same
3:00:16steps over and over for each dimension.
3:00:18And guess what? This is the part of the
3:00:20job. This is the nature of things how we
3:00:22build data modules. And with each
3:00:23iterations, of course, you're going to
3:00:25be faster. So now let's start building
3:00:27our dimensions.
3:00:28>> Let's go build an airport.
3:00:30>> Okay. So now let's start with the first
3:00:31dimension. And here we have to go and
3:00:33pick like one topic or entity. And I'm
3:00:35going to start with the big elephant in
3:00:36the data model. We have the dimension
3:00:38customer. So we're going to go and build
3:00:40it. Now how we going to do it? We go and
3:00:41pick all the tables that are related to
3:00:43the customers and turn it into one
3:00:45dimension. So now based on what we have
3:00:47learned we have a lot of tables but
3:00:49obviously the biggest table that's going
3:00:50to gives us context about the customers
3:00:52is the customer master. So now what I
3:00:54usually do is that in this scenario
3:00:56since we have like a lot of chaos over
3:00:58here it's like a basel game. I'm going
3:01:00to go and grab all those tables that are
3:01:02related to the same topic and move them
3:01:04to one section in the model. So for
3:01:06example here the table customer master
3:01:08I'm going to go and grab it over here
3:01:09just in order to get like feeling of
3:01:11progress over the time. Now what is the
3:01:13other table that actually gives us more
3:01:15context about the customers? We had the
3:01:17contacts. So it is over here the
3:01:19customer contacts and at the end we
3:01:21found like a table the user details
3:01:23where we get like phones and emails as
3:01:25well. This one gives us some context
3:01:27about the customers. Now there is a
3:01:29topic that actually could be relevant
3:01:31that the addresses. Each customer has an
3:01:33address. Of course here the question is
3:01:35whether we make like an extra dimension
3:01:37for the addresses or we merge everything
3:01:39inside the customer. So I'm going to say
3:01:41let's go and group them as well together
3:01:43with the customers. So this is the
3:01:45addresses over here and for the
3:01:46addresses we saw as well some other
3:01:48related tables for the locations. So we
3:01:50had like the cities it is over here and
3:01:53as well we had the regions. So this
3:01:56could be as well interesting. So now by
3:01:57looking to this we have like three
3:01:59tables about the customers and three
3:02:01tables about the addresses. So now I
3:02:03know okay on the top right I have the
3:02:05dimension customers and my goal is to go
3:02:07and turn all those tables into one
3:02:09single dimension. So as you can see our
3:02:11data model is getting like maybe slowly
3:02:14less chaotic but we are not there yet we
3:02:16didn't start. So now in order to change
3:02:17the shape of the tables of course we
3:02:19have to go to the power query. Now in
3:02:21this scenario where we have a lot of
3:02:22tables I usually start with the
3:02:24foundations with the biggest table
3:02:26that's going to gives us a lot of
3:02:28context about the customer and the one
3:02:30table that let's say has a completeness
3:02:32like has all the customer and I'm going
3:02:34to say this is very obvious it's going
3:02:35to be our customer master here we have a
3:02:37lot of columns about the customers and
3:02:39this is very good starting point so now
3:02:41before we do anything we will not go and
3:02:43change now directly the shape of this
3:02:46table we have to go and make a reference
3:02:48out of it so that I can work on a copy
3:02:50let's say and leave the row data as it
3:02:52is. Maybe later we're going to need as
3:02:54well this table for something else.
3:02:56That's why I usually go and say okay
3:02:58let's go and build a reference out of
3:03:00this. So we have it over here and then
3:03:02drag and drop it to the dimensions. All
3:03:04right. So now let's minimize this. The
3:03:06next step we have to go and rename it.
3:03:08And here we have to be really careful
3:03:09with our rules. So we said we have a
3:03:12prefix for each dimension called dim and
3:03:14we are using the snake case. So
3:03:16everything should be lowered. Actually
3:03:18that's it. This is our first dimension.
3:03:19Now let's have a look. What do we have
3:03:21here? We have a lot of context about the
3:03:23customers. And how many customers do we
3:03:25have? We have 61 customers. And this is
3:03:28really interesting. The last row is 999.
3:03:30And it sounds like a test customer. So
3:03:33this is like a dummy information that
3:03:35came from the source system. And I'm
3:03:37going to say we have to get rid of it.
3:03:38So I'm going to go and filter this data
3:03:40out. And we can do that very simply by
3:03:42going to the filter over here and search
3:03:44for the ID. And then all what you have
3:03:46to do is to just unmark it. That's it.
3:03:48So now by looking to this we have 60
3:03:51customers inside this table. This is the
3:03:52foundations. We have to keep this in our
3:03:54mind because as we start adding more and
3:03:56more things to this table we have to
3:03:58make sure that everything stay unique
3:04:00and we stay with the 60 customers. So
3:04:02now we have the foundations. Now let's
3:04:04go and look to the other tables. So we
3:04:06have the customer contacts. So over here
3:04:08we have the ID, the name, the email and
3:04:11is primary. So we have here more context
3:04:13about the customers. And of course it
3:04:15makes sense to go and put those
3:04:16informations side by side with the other
3:04:18context. So with the dimension customer
3:04:20and how we do that we go and merge the
3:04:22two tables. So it is very simple. We're
3:04:24going to go to the dimension customers
3:04:26and say okay let's merge the queries
3:04:28together. So dimension customer and the
3:04:30next one it's going to be the customer
3:04:32contacts. The ID going to be the
3:04:33customer ID and the left join. Make sure
3:04:35it is always left. If you use inner you
3:04:38might lose data from the main dimension.
3:04:40We don't want that. So okay. Now we get
3:04:42it like a new column over here. we have
3:04:43to expand and select exactly what do we
3:04:46need. So we need the name, the email and
3:04:48is primary. So with this we got our
3:04:50three columns side by side with the
3:04:52other context. Now it is very important
3:04:54once you add once you merge new data
3:04:56inside your table you have to make sure
3:04:58that everything stay correct. That means
3:05:00the data is not fanning out or let's say
3:05:02getting duplicated. Now if you go and
3:05:04check here we had 60 customers. But now
3:05:07as I scroll down, we have way more than
3:05:0960 customers. And that's because the
3:05:11same ID is keep repeating over and over.
3:05:14And of course, that's all because the
3:05:16new merge table that we have attached to
3:05:18this table. So that was a really bad
3:05:20merge between those two tables. And this
3:05:23is exactly what you have to be careful.
3:05:25But merging is not destroying your
3:05:27dimension or maybe your fact. Now of
3:05:29course the question is why did this
3:05:30happen? Because my friends, we are
3:05:32merging two tables with different
3:05:34grains. So again the customer contacts
3:05:36each row doesn't represent one customer
3:05:38each row is representing one contact for
3:05:41the customer. So here for example for
3:05:43the second customer we have four
3:05:45contacts. So the grain is different from
3:05:47the customer master. Here each row is
3:05:49really representing only one customer.
3:05:52And once you merge different grains you
3:05:54get this bad effect. So of course the
3:05:56question is how we going to solve this?
3:05:58Well there is like different ways on how
3:06:00to do it. You can say you know what we
3:06:02can go and convert this table into the
3:06:04grain of customer. So that means each
3:06:06row represent one customer and what do
3:06:08we do with the contacts? We make like a
3:06:10list of contacts. It's like we are
3:06:12grouping up or let's say aggregating
3:06:13this table to the level of the customer.
3:06:15And of course we can do that if you go
3:06:17to the transform and then group by the
3:06:19columns by the customer ID. Another
3:06:21solution that you might say you know
3:06:22what this is the contact of the
3:06:24customers and we can make another
3:06:25dimension out of it and then make a
3:06:27relationship between this new dimension
3:06:29and the customer or a third solution
3:06:31that you might say you know what let's
3:06:33not go and add all the contacts inside
3:06:35the customer dimension let's just add
3:06:37the primary contact so for example for
3:06:39the second customer we have here like
3:06:41four people only one of them Sara is
3:06:44actually the primary contact for this
3:06:46customer and if that is okay for the
3:06:47analyzes and the reports then we can go
3:06:49and merge actually this table to the
3:06:52customers again. But of course we have
3:06:53to make sure that we really have like
3:06:55one primary contact for each customer.
3:06:57We can do that by filtering the data for
3:06:59example over here and say show me only
3:07:02the primary contact. So of course this
3:07:04dimension is very small. You can go and
3:07:06check like by eyes and see with this I
3:07:08have exactly 60 customers and I don't
3:07:10have like any duplicates in the primary
3:07:12key. So now again my grain over here is
3:07:15one row is one customer and the
3:07:17relationship between the customer
3:07:18contacts and the dimension customer is
3:07:20again one to one and this is very safe
3:07:22in order to do a merge. So now since we
3:07:24filter the data like this let's go and
3:07:26check our dimension customer. Now let's
3:07:29see again we have here a clean dimension
3:07:32where we have only 60 customers and on
3:07:34the right side of course we have our
3:07:36primary contacts for each customer. So
3:07:38with this finally it is safe to merge
3:07:40those two tables into one. Now of course
3:07:42we are just checking by looking right
3:07:44whether there's duplicates or not. There
3:07:46is like smarter way on how to do it that
3:07:48you go to the primary key the customer
3:07:50ID and you group up the data by the
3:07:52rows. So you say here group by and you
3:07:54just say okay count and here the
3:07:56operation going to be count rows. So if
3:07:58you go and say okay of course the data
3:08:00going to get grouped up by the rows and
3:08:02if you see it like this where the count
3:08:04is everywhere is one then my friend you
3:08:06don't have any duplicates and you are
3:08:08okay. Of course, you can go and check
3:08:09for here. And you can see, okay, we have
3:08:11only one value, the one. So, this is
3:08:12something that I do each time I merge
3:08:14two tables together to check quickly
3:08:16whether we have duplicates or not. But
3:08:17of course, this is only for checking.
3:08:19So, I'm going to go and remove it. All
3:08:20right. So, that's what the first lesson.
3:08:22Be careful for the grains. If you are
3:08:23merging things together now, let's go
3:08:25and check the third table. We had the
3:08:27user details. Right now, again, we have
3:08:29more context to the dimension customer.
3:08:32We have the limits and the phone. That
3:08:33means we have to go and merge those
3:08:35informations to our new dimension. But
3:08:37this time we have to be careful. Do we
3:08:38have here the same grain? Well, by
3:08:40looking to this, I'm going to say we
3:08:42don't have here any duplicates. Each row
3:08:45representing one customer. So this one
3:08:47going to be really easy. I'm going to go
3:08:48to the table customers. Same thing.
3:08:50We're going to go home and then merge.
3:08:52Find our table was at the end. So user
3:08:55details. Now things here getting
3:08:57interesting about the entity. We have
3:08:59here the user ID. But in other table
3:09:02they call it user ID. We have of course
3:09:04to talk about this because we have now
3:09:05like two different names for the same
3:09:07entity right but first let's go and
3:09:09merge things together and of course we
3:09:11have to expand. So we just need the new
3:09:12informations credit limit and the phone.
3:09:15And again before we celebrate having
3:09:16more data inside our dimension we have
3:09:18to go and check whether everything is
3:09:20unique and we don't have duplicates. So
3:09:22looks really good. Now before we
3:09:24continue with the other tables and
3:09:25building the dimension I still don't
3:09:27like it that we found out inside our
3:09:29data we have like two names for the same
3:09:31entity. We had the customer and as well
3:09:33the user. So of course what is the
3:09:35through how we going to call this
3:09:36dimension customer or dimension user.
3:09:39This happens a lot of course by the way
3:09:40in the companies where you have
3:09:42different departments calling the same
3:09:44entity different names. In one
3:09:45department they call all the customers
3:09:47as users and other is customer. So of
3:09:49course this is always a dilemma as you
3:09:51are building a data model especially if
3:09:53you are connecting like multiple sources
3:09:55inside one model inside your projects.
3:09:57Now of course one idea could be let's go
3:09:59and ask the business users who are using
3:10:01the report which name they want to use.
3:10:04Now the thing is if you are asking a
3:10:05business user from specific department
3:10:08they might say you know what we use
3:10:09customers then I want to see it in the
3:10:11report as well as customers. But if you
3:10:13go and ask another department another
3:10:15business they might say use the user. So
3:10:18still it is dilemma and they will not
3:10:20really help you in order to decide for
3:10:21your data model. You cannot go and use
3:10:23both of them. That's why you have to
3:10:25have like feeling about the data model.
3:10:27If you have a lot of tables are using
3:10:29the customer as an info, then go with
3:10:32the customer. Like here in our example,
3:10:33most of our tables are using this term
3:10:36customer and only like one table is
3:10:38using the user. That's why I'm going to
3:10:40decide for our data model to go with the
3:10:42customer as an entity name and let's say
3:10:45ignore the user as a term. So now let's
3:10:47go back. So that means we're going to
3:10:48stick with the name dimension customer.
3:10:50We will not rename anything. Now let's
3:10:52go back to the other three tables. Now
3:10:54the next batch of tables, they are all
3:10:56to do with the addresses. Look at this.
3:10:58We have here the address ID. And now
3:11:00here my friend, you have two options.
3:11:01Either you're going to go and build a
3:11:03new dimension called addresses and then
3:11:05connected to the dimension customer. And
3:11:08with that you are actually building
3:11:09remember the snowflake schema one
3:11:12dimension connect and branch to another
3:11:14dimension or we're going to stick with
3:11:16our goal where we are building the star
3:11:18schema. So we always try to fit
3:11:20everything together into one dimension
3:11:22and only in extreme cases we might go
3:11:25and branch. So of course I'm going to
3:11:26stick with the goal. We are building
3:11:28star schema. That means all the address
3:11:30informations going to go inside the
3:11:32dimension customer. And don't worry, we
3:11:33can always go and split if things goes
3:11:35wrong. So that means we have now to go
3:11:37and merge more and more things to the
3:11:39dimension customer. And I would like to
3:11:40show you one trick that I usually use in
3:11:42order to feel safe before merging things
3:11:45together. Yes, you can go and count
3:11:46things after merging, but usually I go
3:11:48to the module layer. So, first save.
3:11:51Yeah. Okay. Close and apply before going
3:11:53to the model view. Now, look at this.
3:11:55This is our new dimension that we are
3:11:57actually building all the time. And you
3:11:58can see that PowerBI is always trying to
3:12:01connect a new table to the whole data
3:12:03model. Of course, we have to remove all
3:12:04those stuff and start from a scratch.
3:12:06But don't worry about it. Now, now the
3:12:07thing is if I have like a new table that
3:12:09I would like to merge it to the table
3:12:10customers, I go and check the cardality.
3:12:13Like for example, we want to merge the
3:12:14addresses with the dimension customer.
3:12:16Right? Now you can see we have already
3:12:17like a relationship between those two.
3:12:19Now if I see the cardality is one to one
3:12:22relationship between those two tables,
3:12:24then this is really safe to merge those
3:12:26two tables together because PowerBI
3:12:28automatically check the values inside
3:12:30those two tables and detect the
3:12:32cardality. And of course this is a great
3:12:34indicator for me to say nothing going to
3:12:36go wrong if I go and like merge those
3:12:38two tables together. So this is a good
3:12:40thing. And by the way about the
3:12:42cardality if the relationship is one to
3:12:44many so from the addresses is one and in
3:12:47the dimension customer is many it is
3:12:49still safe. So this is another way to
3:12:51check before let's say merging two
3:12:53tables together. Let's go back to power
3:12:54query. So now same thing let's go to the
3:12:57dimension customer. We're going to go
3:12:58and merge and here we're going to search
3:13:00for the addresses. This time the ID
3:13:03going to be the address ID as well. Here
3:13:05address ID always left join never inner
3:13:08join. If you have one missing address
3:13:10then you are removing a whole customer.
3:13:12So this is not safe. We always use lift
3:13:14join and same things. So we're going to
3:13:16go and expand. We just need the street
3:13:18and the city name. So look at this.
3:13:20Looks nice. Now of course we have to go
3:13:22and test. So do we have 60 customers?
3:13:24Well yes. So merging the addresses
3:13:27didn't cause any harm any issues. Now
3:13:29what else do we have? We have the cities
3:13:31right? So let's go over here. Now what
3:13:33do we have here? We have the city name
3:13:34and the region name. But here we have
3:13:36equality issue. You can see that the
3:13:38first row is actually the column names
3:13:40and not the column one and column two.
3:13:42Now in order to fix this, this is very
3:13:43simple. We have here in the ribon use
3:13:45first row as headers. So probably are
3:13:47going to go and promote the first header
3:13:49as the column name for the table. So
3:13:51this is very simple. Look at this. We
3:13:52have now the city name and the region
3:13:54name. Now again what we're going to do
3:13:55we're going to go and merge this inside
3:13:57our dimension customer. Again as I told
3:13:59you you can go and check whether we have
3:14:01a one relationship at this table. So we
3:14:04have it over here the cities. Let's go
3:14:06and grab it. I'm going to go and remove
3:14:07this relationship and then try to
3:14:10connect it to the dimension customer.
3:14:12This of course only for test. So I'm
3:14:13going to take the city name to the city
3:14:15name. All right. Look at this one to
3:14:17many relationship. So at the side of the
3:14:19cities we have one. So I'm going to get
3:14:21good feeling about it in order to merge
3:14:22those two tables together. We don't have
3:14:24many. This will not duplicate my
3:14:26customers. So that means we're going to
3:14:28go back to transform data. So again we
3:14:30merge. And as you can see I'm merging
3:14:32all the time and I will keep doing that.
3:14:34So let's go and connect our cities. So
3:14:36city name and the city name is somewhere
3:14:39over here. So let's go and connect left
3:14:41join. Now what do we need from this
3:14:43table is actually only the region name.
3:14:45We have already city name. So we
3:14:47actually don't need it twice inside our
3:14:49dimension. And we're going to go and
3:14:50check we have 60 customers. All right.
3:14:53Now to the last table. We have the
3:14:54regions. Well here we have only one
3:14:57column. We don't have like a lot of
3:14:58context that's going to help us for the
3:15:00dimension customer. And the region
3:15:02information we have it already. So it
3:15:04makes no sense to merge this table to
3:15:06the dimension customers. This
3:15:08information is already in our dimension.
3:15:10So as you can see we have here the
3:15:11regions. So actually this makes no sense
3:15:14to merge inside our dimension customer.
3:15:16It will not add any value or add any
3:15:18context. And this is of course one of
3:15:20the things that happens all the time
3:15:21inside the data models. You're going to
3:15:23find the same piece of information
3:15:25multiple times in multiple tables. So
3:15:27the region exists here and as well in
3:15:29the cities and maybe in another table
3:15:31that we didn't discover yet. So here we
3:15:33have to be very careful we are not like
3:15:35repeating the same data over and over
3:15:37otherwise we are confusing the data
3:15:38model. So my friends we went through the
3:15:40six tables and we have now one gigantic
3:15:43table that holds all the context about
3:15:45the customers. So that was big step. We
3:15:48have compiled now six tables into one
3:15:50dimension customers. But we are not done
3:15:52yet. We will not go to the second
3:15:53dimension until we do my favorite step.
3:15:56We have to do the housekeeping. So we
3:15:57have to clean up, organize and make sure
3:15:59that we are following the standards. So
3:16:01we polish everything in order to make
3:16:03the dimension is ready to get used in
3:16:06the data model. So let's do that. All
3:16:08right. So now the first step that we
3:16:09have to do is that we have to get rid of
3:16:11unnecessary informations unnecessary
3:16:13columns inside the dimension. Remember
3:16:15the rule each info each data should earn
3:16:17its place. So that means if there is
3:16:19something unnecessary we have to remove
3:16:21it. Otherwise it's going to take space.
3:16:23It going to lead to bad performance and
3:16:25as well it going to lead to confusion as
3:16:27you start building the reports. So that
3:16:29means my friends now we're going to go
3:16:30through all the columns inside our new
3:16:32dimension and think about it should we
3:16:34leave it or should we drop it. So the
3:16:36customer ID for sure it's going to stay
3:16:38the customer name the segments all those
3:16:40informations are important for filtering
3:16:42and slicing and building visuals. Now to
3:16:45the address ID well my friends I believe
3:16:48we don't need it. I can't imagine that
3:16:50this going to be necessary for reporting
3:16:52only for connecting things. And we have
3:16:54decided to put everything together into
3:16:56one dimension. We are not making
3:16:57snowflake that's why I'm going to go and
3:17:00drop it. So remove the account manager.
3:17:02Okay the payment terms going to stay.
3:17:04Now the hash keys well my friends those
3:17:06stuffs comes from the source system and
3:17:08I will never imagine that we're going to
3:17:10need this for reporting. That's why
3:17:12we're going to go remove. By the way
3:17:13this takes a lot of space inside the
3:17:15data model. I remember in one project we
3:17:17had like for each table one hash key
3:17:20where the refresh time for the data
3:17:21model was really horrible. It took a lot
3:17:23of time and all because those hash keys
3:17:25they are string values and they are
3:17:27really big and if you have millions of
3:17:29rows imagine how much space this takes.
3:17:31So the fix was very simple to just go
3:17:33and drop all those hash keys and
3:17:35suddenly we saved around I think 20% of
3:17:37the data set. So now let's go and remove
3:17:39the hash key. Then we have source ID.
3:17:41This is as well some garbage from the
3:17:43source system. We would never use this.
3:17:45Now here we have the primary contacts.
3:17:47Okay we have their email. Now whether it
3:17:50is primary well this going to be always
3:17:51true, right? We use this as a filter. So
3:17:54that means actually this is like totally
3:17:56useless information and we would never
3:17:58use this. So let's go and remove it.
3:18:01Credit limits going to stay the phone,
3:18:03the street, the city name and the
3:18:05region. So not that bad, right? We just
3:18:07dropped like around four columns. All
3:18:09right. So this is the first step that
3:18:11we're going to do in order to clean up
3:18:13our dimension. Now comes the naming
3:18:15convention. As you can see, we are not
3:18:17following the standards. Only the table
3:18:19name is actually using the camel case.
3:18:21But all the columns are actually
3:18:23following another case. This we call it
3:18:25the Pascal case where we don't have any
3:18:27separations and the first letter is
3:18:29capitalized. Now of course since we have
3:18:32committed to the snake case we have to
3:18:34go and rename all the columns or you can
3:18:36say you know what since everything is
3:18:37like Pascal case I'm going to go and
3:18:39change the rules but of course you have
3:18:40to stick to it and we're going to follow
3:18:42the Pascal case. Well this is something
3:18:44that you decide but I have decided for
3:18:46the snake case and I'm going to go and
3:18:47now rename everything to the standards
3:18:49but of course I'm going to speed up. So
3:18:51I'll just do it for the first column for
3:18:53you. So this going to be the customer ID
3:18:55and this going to be customer name right
3:18:58like this. So now see you soon. All
3:19:00right. So now to the last column. As you
3:19:02can see here we have the cities dot
3:19:04region name. We don't need the cities
3:19:06dot because this is just comes from the
3:19:08original table the cities. We're just
3:19:10going to go and call it region. That's
3:19:12it. So I'm going to say everything is
3:19:14clean inside our dimensions. The names
3:19:16are friendly. So the table name, the
3:19:18column names and as well the values
3:19:21inside the table. This is really
3:19:22important. If you see something critical
3:19:24technical, we have to do something about
3:19:26it. Now I'm going to say our dimension
3:19:28customer is ready to use. But now of
3:19:30course before we close and go to the
3:19:32second dimension, we have the last last
3:19:34step where we're going to go and hide
3:19:35all the original tables so that we don't
3:19:37see them anymore inside the model. So
3:19:39we're going to go through them one by
3:19:41one. The addresses we don't need it
3:19:42anymore. So we'll deactivate the enable
3:19:44load. So yes, same thing for the
3:19:47customer master. So we have to do the
3:19:49whole thing for all six tables, the
3:19:51cities. So deactivated the contract as
3:19:54well here. The regions, well actually we
3:19:56didn't merge it, but we don't need it.
3:19:58We have this information already in the
3:20:00dimension customer. And the last one,
3:20:01the user details. So right click and
3:20:04disable. Okay. So with this, we have
3:20:06consolidated a lot of things into one
3:20:08dimension. Let's go and close and apply.
3:20:14Well, look at this. Everything gone and
3:20:16I have only the dimension customer. Now,
3:20:18I'm going to go and clean up those
3:20:20relationships. I don't need them. So,
3:20:22remove, remove, and last remove. So, now
3:20:25comes the most satisfying like thing
3:20:27that I'm going to do. We're going to
3:20:28zoom out and look at this. Our data
3:20:30model get reduced a little bit. And now
3:20:32we have here one good solid dimension.
3:20:34So, my friends, this is the first
3:20:36dimension. And I'm going to say this is
3:20:37the biggest dimension that we have
3:20:38inside this row data model. We have of
3:20:41course to keep iterating on this. And
3:20:42the next big dimension of course in our
3:20:45business the products. And now again the
3:20:47same iteration we start by collecting
3:20:48all the tables that has to do with the
3:20:50products. We had very big table called
3:20:53products where we have a lot of context
3:20:55about it. So I'm going to drag it over
3:20:56here. And the second table that actually
3:20:58has to do with the products that was the
3:21:00subcategories. I don't find it. Well
3:21:02yeah it's over here. So this one going
3:21:04to organize the products. And I think by
3:21:06looking to this actually that's it. So
3:21:08we have only two tables to do with the
3:21:10products. We're going to go and try to
3:21:12merge everything into a new dimension
3:21:14called dem product. So let's go and do
3:21:16that. We're going to go again to the
3:21:18power query. All right. So now our two
3:21:20tables are again the products and the
3:21:22subcategories. Now of course we have to
3:21:24start one of them as the foundations.
3:21:26It's going to be the products of course
3:21:27because here we have a lot of context.
3:21:29This is the master table for our
3:21:31dimension. So as usual so what we're
3:21:33going to do same thing. We're going to
3:21:34go and create a reference out of it.
3:21:35Drag and drop it to the dimensions. Let
3:21:38me just make this smaller and we're
3:21:40going to go and rename it. Same thing.
3:21:41The prefix is dimension and the name is
3:21:44product. Okay. So now let's have a quick
3:21:45look to our dimension products. As you
3:21:47can see, we have a lot of columns
3:21:49describing as well. We have a lot of
3:21:51things that we're going to clean up at
3:21:52the end. But of course, we have to
3:21:53understand how many products do we have.
3:21:55Well, same thing. We have 60 products.
3:21:58And the last one, we have as well a
3:22:00dummy data. So it says don't use test
3:22:02blah blah blah. So that means this is
3:22:04something from the source. We're going
3:22:05to go and remove it quickly. So uncheck
3:22:07it. And other thing that's let's make
3:22:09sure that we don't have duplicate inside
3:22:11the products. So we can do a quick group
3:22:14I group by. So count rows. Okay. Now we
3:22:17check this. Do we have any twos? Well,
3:22:20only one. So this is clean. We don't
3:22:22have any duplicates inside our products.
3:22:24But now you might notice that we don't
3:22:26have actually a product ID. We have a
3:22:28product code. So that means the source
3:22:30table is not providing like a unique
3:22:32identifier in order to connect it to the
3:22:34data model. it is using the code as like
3:22:36a business key. This is of course a
3:22:38tricky thing. So now we will not clean
3:22:40anything yet. Let's go and check the
3:22:42subcategories. So the subcategories is
3:22:44over here. Now the first issue with that
3:22:46we don't have the column name. So what
3:22:48we're going to do we're going to use the
3:22:49first row as headers. And the second
3:22:51issue actually here we are merging two
3:22:53informations in one column. We have the
3:22:55category and the subcategory side by
3:22:57side and they are separated with a pipe.
3:23:00Now of course we have the subcategories
3:23:02already in the dimension products. So we
3:23:04have it over here but what is missing
3:23:06which is very important is the category.
3:23:08So that means we need the category in
3:23:10order to add more context to this
3:23:11dimension and we have somehow to grab it
3:23:14from this column. Now of course we
3:23:15cannot go and merge this directly to the
3:23:17product because we don't have keys. The
3:23:19keys are actually like piece of
3:23:21information inside each value. That's
3:23:23why we have to go and split this into
3:23:25two columns. One category and another
3:23:27one subcategory. So let's go and do
3:23:29that. So here we have a nice function
3:23:31called split columns by delimiter. So
3:23:33here PowerBI get it correctly. We are
3:23:35splitting by custom and the separator is
3:23:38the pipe. So actually everything is
3:23:40ready. Let's go and hit okay. Look at
3:23:42this. Now we have the categories and the
3:23:44subcategories. I'm going to go and
3:23:45rename things because here it says one
3:23:47and two. PowerBI has no idea about the
3:23:49content. So this going to be the
3:23:51category and the other one going to be
3:23:53the subcategory. Now we are not there
3:23:55yet because if you look at this you see
3:23:57that all the subcategories all the
3:23:59values are lowercased but if you check
3:24:01our dimension over here you see the key
3:24:04is not ready yet. We have it
3:24:05capitalized. So the first letter is
3:24:07actually an uppercase and this will not
3:24:09work. So if I go and merge now things
3:24:11together powerbi will not find any
3:24:13match. So as you can see we have always
3:24:14like to prepare the data before merging.
3:24:16The keys must work otherwise you're
3:24:18going to get only nulls after merging.
3:24:20So now either we're going to make this
3:24:21lowerased or we go over there and we
3:24:23make the key as a capitalize. I'm going
3:24:25to change it over here in the reports.
3:24:27It looks way nicer if you have things
3:24:29capitalized. So how we going to do it?
3:24:30We're going to go to transform and here
3:24:32we have a nice function called
3:24:33capitalize each word. So let's do it.
3:24:36Now our key is ready for the merge. So
3:24:38everything now is simple. We're going to
3:24:39go to the dimension product. As usual we
3:24:42merge. We go and pick our table. So it
3:24:44is the subcategories over here. And now
3:24:47all we have to do is to just select
3:24:48things. So subcategory with the
3:24:50subcategory and left join. So let's go
3:24:53and set. Okay. All right. We have it
3:24:56here as a table. We have to expand.
3:24:57Well, we don't need the subcategory. We
3:24:59have it already. We just need the
3:25:00category. So okay, now it is working. As
3:25:03you can see, we have now the categories
3:25:05inside our table. All right. So now with
3:25:07this, I'm going to say that our
3:25:09dimension now hold all the context about
3:25:11the products in one table. We don't have
3:25:13anything else from the other tables. Now
3:25:15we're going to come back to this topic
3:25:16where I said we don't have like an ID
3:25:18from the source system. We just have
3:25:20like a code and a name. So it's not like
3:25:22the dimension customer we had here like
3:25:24an ID. Now here we have like two options
3:25:26either we're going to leave it as it is
3:25:28or we're going to go and create a
3:25:29surrogate key something that we create
3:25:31inside the model not from the business
3:25:33and we're going to use this ID in order
3:25:35to connect it to the facts in the
3:25:37future. So I'm going to say for now I'm
3:25:38going to go and add it. So we're going
3:25:40to go and add column and we just go and
3:25:41add an index column and we start from
3:25:43one. So we have it over here. I'm going
3:25:45to go and rename it. So this going to be
3:25:47not the product ID. This going to be the
3:25:50product underscore key. We agreed in the
3:25:52rules if you are creating any keys from
3:25:54us. We always have the suffix underscore
3:25:57key for it. So that I know which ids
3:25:59come from the source and which one that
3:26:01I created. So it looks like this and we
3:26:03have like 60 products. Now I'm going to
3:26:05say we are done with merging and
3:26:07creating things. Let's go to the last
3:26:08step where we start like cleaning up and
3:26:10bringing standards to this dimension. So
3:26:12that means again we're going to go
3:26:13through all the columns one by one and
3:26:15say whether they going to stay or leave.
3:26:17So the code of course going to stay the
3:26:19name, the brand, the subcategory, the
3:26:21prices, the supplier. We could use it
3:26:23for slicer. Now comes something that I
3:26:26saw a lot in data models where people
3:26:28just leave such information here. So
3:26:30those are the big text the descriptions.
3:26:33Now this is of course a huge text. It is
3:26:36describing each product. And of course,
3:26:38we have to ask ourselves, do we really
3:26:40going to go and use such a description
3:26:41inside our reports? In most cases, no,
3:26:44we are not building like an operative
3:26:46system. We are not building an interface
3:26:48in order to show the descriptions of the
3:26:50products. We are building analytics and
3:26:52reports. And in most cases, we don't go
3:26:54and use product descriptions and long
3:26:56text in the visuals. But of course in
3:26:58the other hand I saw a lot of people
3:27:00that are using PowerBI in order to build
3:27:02some kind of application let's say to
3:27:04see all the data. Well there's like some
3:27:06projects that are misusing PowerBI in
3:27:08order to build interfaces but not BI
3:27:11right. So in most cases I refuse to have
3:27:13such informations inside the data model.
3:27:16That means we go and drop it. So remove
3:27:19now same thing we have here hash key the
3:27:20source ID all those informations came as
3:27:23garbage let's say from the source system
3:27:25and we don't need it here. Of course,
3:27:26there are maybe very important
3:27:27informations in the source systems, but
3:27:29here we don't need it. So, let's go and
3:27:31remove it. Now, here we have our
3:27:32category and the product ID. Now, before
3:27:34we start like renaming everything, I
3:27:36would like to go and sort the columns.
3:27:38So, I would like to see the product key
3:27:40at the start as like primary key and the
3:27:42category. I would like to have it side
3:27:44by side with the subcategory. Well, I'm
3:27:46going to say subcategory and then
3:27:48category. So, that like we have
3:27:50hierarchy, right? The product then
3:27:52subcategory and then category. I like to
3:27:55have things ordered so it makes sense as
3:27:56you are building visuals and reports.
3:27:58And now I'm going to say we have all
3:28:00useful informations and we can go now
3:28:02and start renaming things again. I'm
3:28:04going to do it now offline in order not
3:28:05to waste time. So see you soon.
3:28:10All right. So this is the last column
3:28:12and with that we are following now the
3:28:13same naming convention. Sometimes I
3:28:15remove few things like for example the
3:28:17subcategory name. I call it now
3:28:19subcategory. So there is no need for the
3:28:21name inside the column name. Now there
3:28:24is one more thing by looking to the
3:28:26values you can see that the subcategory
3:28:27is actually capitalized. Same thing for
3:28:29the product name only the category is
3:28:32everything here lowercased and this of
3:28:33course because it comes from another
3:28:35table where everything is lowercased. So
3:28:37now in order to like to bring standards
3:28:39to the values I'm going to go and apply
3:28:40the same function. So transform and
3:28:43capitalize each word. So now things
3:28:45looks like more harmonized everything
3:28:47capitalized and maybe we can add it as a
3:28:49rule inside our project where we say all
3:28:51the text values inside our data model is
3:28:53capitalized so that people once they
3:28:55start building reports they don't have
3:28:57to go back to the data model and start
3:28:59like adding more functions and
3:29:00transformations. All right so now we
3:29:02have really nice dimension as well. The
3:29:04last step that we're going to do we will
3:29:05just go and deactivate the row tables.
3:29:08So the products we will go and
3:29:09deactivate it. Yes. Same thing for the
3:29:12subcategory as well. It is deactivated.
3:29:14All right. So, we are done. Let's go and
3:29:16make sure that we save and apply. All
3:29:18right. Now, this is our new dimension
3:29:21over here. Let me just zoom in. We're
3:29:22going to go and remove the
3:29:24relationships. We don't need them. And
3:29:25for this as well. So, now look at this.
3:29:28Now, we have clean two dimensions are
3:29:30waiting here on the side. And our
3:29:32original data model is getting cleaned
3:29:34up step by step.
3:29:39Okay. So, now what is next? What we
3:29:41going to do? Do we have like a third
3:29:42dimension that we're going to go and
3:29:43build? Of course, there will be a lot of
3:29:45dimensions, but here we have like
3:29:46something that we have to decide on.
3:29:48Now, the products and the customers,
3:29:50they were like standalone and very
3:29:52obvious dimensions where we had like a
3:29:55master table for them. So, we have like
3:29:57table products and another table called
3:29:58the customer master. So, that's why it
3:30:00was really easy to make those dimensions
3:30:03out of those row tables. But now the
3:30:05thing is in all other tables that are
3:30:07remaining in the data model, most of
3:30:09them are events. So we don't have like a
3:30:11master table for the third dimension.
3:30:13Actually most of the dimensions going to
3:30:15get like extracted out of the event. As
3:30:17we said before in many facts there are
3:30:19always like hidden sneaky dimensions
3:30:21that we extract out of it. This is
3:30:23exactly now the scenario because of this
3:30:25now we have like two ways on how to
3:30:27continue. Either you're going to go
3:30:28through all the events one by one and
3:30:30start like extracting all the dimensions
3:30:32and once you are done once you have all
3:30:34the dimensions then you start building
3:30:36the facts or the other way that I prefer
3:30:39that at the same time we take like one
3:30:41event table and we split it into fact
3:30:44and dimensions. So at the same time we
3:30:46are building the fact and the dimensions
3:30:48out of the row tables and we don't have
3:30:50like to redisuss it and brainstorm it
3:30:52again. So either you go all the way by
3:30:54extracting all the dimensions and only
3:30:57after that you build the facts or at the
3:30:59same time you are building the
3:31:00dimensions and facts. Well I tend to the
3:31:03second option. So now let's go and do
3:31:04that. All right. So now by looking back
3:31:06to our data model we have actually a lot
3:31:08of events. So we have the orders and
3:31:09sales, we have the invoices, the
3:31:12payments, the campaigns and of course
3:31:14it's going to be really hard to think of
3:31:16everything at the same time. That's why
3:31:18we have to go and pick one event, go
3:31:20deep inside it and start splitting it
3:31:22into facts and dimensions. So that means
3:31:24we have to go and pick an event. I'm
3:31:25going to say we're going to start with
3:31:27the orders and sales. This is the most
3:31:29important one and I think this is a
3:31:30little bit complicated. So now let's go
3:31:32and collect all the related tables like
3:31:34we have done in the dimensions together
3:31:36in one place. So we have the orders 2026
3:31:39and as well we have the same table for
3:31:41the 2025. What else do we have? We have
3:31:43as well the lines for the orders. So
3:31:45this one over here, the sales target is
3:31:48going to be the budget. So I'm going to
3:31:49say it's not really same event. It is
3:31:52like the planned budget for each month.
3:31:54We're going to think about it. Maybe
3:31:55we're going to consider it. So I'm going
3:31:56to say this is more than enough. We have
3:31:58a lot of things. We have the orders and
3:31:59the lines. And here we have to have full
3:32:01focus because we start talking about
3:32:02facts. Facts are very sensitive topic.
3:32:05So again we're going to go to the Power
3:32:07Query. Let's go inside. Now the first
3:32:08thing before we start touching anything,
3:32:10let's have some understanding more about
3:32:12our tables. So let's start with those
3:32:14two orders. So 2025 and 2026. Now again
3:32:17I'm going to ask the question of the
3:32:18grain what each row is representing.
3:32:21Well you can see over here each row is
3:32:23actually one complete order. So we have
3:32:25all the informations about the orders.
3:32:27We have the customer name, city, region
3:32:29looks like dimension, right? But of
3:32:31course we have here the events, the
3:32:32order date, the channel of the order,
3:32:34the status, priority, total sales for
3:32:37the order and some big notes. Now let's
3:32:39go to the other table. The orders 2025.
3:32:42This is very identical table. Also
3:32:44actually here we have a split from the
3:32:45source system. Here as well we have the
3:32:47same grain. Each row is actually one
3:32:49order but again the source system is
3:32:51splitting the data maybe for some
3:32:53reasons like archive and so on. And of
3:32:55course we have to do something about it.
3:32:56We cannot leave it splitted like this.
3:32:58It just confused the data model. Now
3:33:00let's go to the third one the order line
3:33:02items. So now again the same question.
3:33:04What each row here is representing? Well
3:33:06I'm going to say each row is actually a
3:33:08line of order. So one line is not the
3:33:11whole order like we had it in the other
3:33:12tables. It is one position about the
3:33:15order and of course we can go and check
3:33:16that by aggregating the order ID. So
3:33:19let's just group by and say okay this is
3:33:21the count of rows. Well by looking at
3:33:23this you can see here we have a lot of
3:33:24duplicates. That means for each order we
3:33:27might have like multiple lines like here
3:33:29for the first order we have three lines.
3:33:31All right. So I will just go and remove
3:33:32it. We are just testing here. So that
3:33:34means this has like more details another
3:33:37grain than the orders. Now my friends
3:33:39here we have like a very classical
3:33:41scenario with the order lines and the
3:33:43orders. This is something that we call
3:33:45from the transactional systems from the
3:33:47sources the headers and the details. All
3:33:49right friends now I can say we come to
3:33:51the most complicated and confusing part
3:33:54in the whole data modeling project.
3:33:56That's why if you don't have cave yet go
3:33:57and grab one.
3:34:02Okay. So now we're going to talk about
3:34:04the header and the details. This is very
3:34:06tricky and confusing part where a lot of
3:34:08people make mistakes in the data
3:34:10modeling for reporting and analytics but
3:34:12yet one of the best practices and the
3:34:14common way on how to store transactions
3:34:17in OLTB sources. So it is a very common
3:34:20way on how to module the sources and
3:34:22you're going to find it a lot in many
3:34:24companies. So my friends, if you
3:34:25understand this concept and you know how
3:34:27to deal with it in the data modeling for
3:34:29business intelligence and analytics,
3:34:31then my friends, your life going to be
3:34:32really easy working at companies because
3:34:35you're going to face this data model
3:34:36over and over. So now in order to
3:34:38understand this, let's start with very
3:34:40simple example. I don't know, look
3:34:41around you. Maybe you find an order or
3:34:43an invoice.
3:34:44>> I found it. If not, go and check online
3:34:46your stuff. So now here I have an order.
3:34:48And if I look at this, this is one
3:34:51business event. I have done an order.
3:34:53And this business event actually has
3:34:55like two sections a header and details.
3:34:57The header it is like the top level
3:34:59informations in the order. Like for
3:35:01example I have the address, the
3:35:03customer, the company, the order number,
3:35:05the order state, the order dates. So all
3:35:07those informations we have it like only
3:35:09once for each order right and those are
3:35:12my friends the header. Now if you look
3:35:13below it depends if you ordered multiple
3:35:15things you're going to find like a table
3:35:17with more details. Like for example,
3:35:18you're going to find the list of all
3:35:20products that you have ordered in this
3:35:22one order and other things like the
3:35:23quantity, the price and as well the
3:35:25final costs. And my friends, this setup,
3:35:28the header and the details, you're going
3:35:29to find it in many events in many
3:35:31companies. So you're going to find it in
3:35:33orders, in invoices, in shipment, in
3:35:35delivery. Even in manufacturing, you're
3:35:37going to have the bill of materials, the
3:35:38production order, or if you have like
3:35:40ticketing system, you're going to find
3:35:42it as well in tickets. all those
3:35:43transactions and many systems going to
3:35:45store those informations into header and
3:35:48details. Now of course the big question
3:35:49is how we going to deal with those stuff
3:35:51inside our data model for business
3:35:53intelligence. Now one of the biggest
3:35:54mistake that people do is that they look
3:35:56at the header and details as two
3:35:58different things. So they say okay the
3:36:00header looks like a fact. So they go and
3:36:02make like for example fact orders and
3:36:04they see okay the details as well looks
3:36:06as fact but this is like different
3:36:09grains. That's why they're going to go
3:36:10and make another fact for that. the fact
3:36:13order line. So they make out all of this
3:36:15like two facts. And now of course in
3:36:16reporting you're going to need those two
3:36:18tables in one visual in one report. So
3:36:21that means they need two data from the
3:36:23facts. And here people come in this trap
3:36:25where they're going to say okay well
3:36:27actually we have the order key in the
3:36:29header and the details. This is actually
3:36:30what happens in transaction systems.
3:36:32They link the order with the header
3:36:34using the key. And they're going to say
3:36:36okay this is a perfect match. This is
3:36:37the common column. I'm going to go and
3:36:39connect those two facts together. Well,
3:36:41yes, this might work, but this can
3:36:42introduce only issues in the reporting.
3:36:45This has first of all, bad performance,
3:36:47can lead to bad numbers and as well this
3:36:49is very confusing once you are building
3:36:51the reports. So, this is one of the
3:36:53biggest mistakes that people do in data
3:36:54modeling. And now, of course, the
3:36:56question is how we going to do it? How
3:36:58we going to model this? Well, the rule
3:37:00is always the same. The fact table going
3:37:02to sit always at the level of things
3:37:04that you are measuring. So, ask yourself
3:37:06where the numbers come from. So if you
3:37:07look to the header and the details
3:37:09you're going to see that actually all
3:37:10the numbers are stored in the details.
3:37:12The header is just like giving us some
3:37:14context and maybe like maximum one or
3:37:17two numbers but the core numbers the
3:37:19core measures are in the details. So
3:37:21that means it is always better to build
3:37:23a fact table that comes from the finest
3:37:26details from the lowest level because my
3:37:29friend we can always roll up and
3:37:30aggregate the data pack to the header.
3:37:32So that means in most cases we're going
3:37:34to go and create the core fact table
3:37:37from the details and we're going to the
3:37:39header in order to get more context for
3:37:42our new facts. So either we're going to
3:37:44use it in order to generate a new
3:37:45dimensions from the header or we're
3:37:47going to use it as well to connect the
3:37:49data model to our new fact from the
3:37:52details. So this is the very common
3:37:54pattern. We make fact from the details
3:37:56and we use the help of the headers in
3:37:58order to generate dimensions and to
3:38:00connect it to our fact. And I can say
3:38:02this going to work like 90% of the time.
3:38:04Sometimes you have some edge cases where
3:38:06people are storing a lot of numbers in
3:38:08the headers. But in most cases this is
3:38:10going to work if you follow this
3:38:12pattern. Okay. So now with this I have
3:38:13prepared you with the foundations. Now
3:38:15let's go to PowerBI and try this out.
3:38:17All right. So now what we're going to do
3:38:18with those three tables I'm going to say
3:38:19let's start with the easy and obvious
3:38:21thing. We have to go and combine those
3:38:23two orders into one single table. So
3:38:25again if you have two tables have the
3:38:27same grain and as well they are
3:38:29identical. They have the same shape. we
3:38:30go and append them. So how we going to
3:38:32do it? It's going to be very simple.
3:38:33We're going to go over here and say
3:38:34append queries and I'm going to go and
3:38:36create a new one so that we don't lose
3:38:38let's say the row tables. So now here it
3:38:40says two tables and the first table. I'm
3:38:42going to start with the oldest. So first
3:38:442025 and then 2026. All right. Let's hit
3:38:47okay. So now we have a new table over
3:38:49here. Let's go and rename it. So this
3:38:51going to be the orders. Now of course
3:38:52this going to be like an intermediate
3:38:54table. We don't have like fact and
3:38:55dimension. We are just like merging
3:38:57things. Of course, if you want, you can
3:38:59go and make another order for all the
3:39:01temporary things, but I'm going to leave
3:39:03it now in this stage. Now, of course, we
3:39:04are not done. Let's have a look to the
3:39:06append whether we have any strange
3:39:08things here. We have some nulls and if I
3:39:10go over here, the source file has as
3:39:13well some nulls. So, actually there is
3:39:15like something wrong and as well the
3:39:17legacy reference. So, that means they
3:39:19are not completely identical. Let's have
3:39:21a look again. We have here the legacy
3:39:22reference and in 2026 we don't have it.
3:39:25So actually with the new orders we don't
3:39:27have this legacy column. So let's go a
3:39:29and remove it. It says as well legacy.
3:39:31So I think we don't need it for
3:39:32reporting. And let's go and check the
3:39:34resource file. So over here we have it
3:39:36in 2025 but we don't have it I think for
3:39:392026. So again it is a legacy thing. We
3:39:42go and remove it from our new table. So
3:39:45you go out remove. Now here we have a
3:39:48lot of nulls. The gift message. If I go
3:39:50over here you can see actually we don't
3:39:52have any value. This is garbage column.
3:39:54So it is just taking space. Let's go and
3:39:56remove it as well. So as you can see, we
3:39:57are cleaning up. And since we are
3:39:59talking about cleaning, we have here the
3:40:00order notes as well those big text. Come
3:40:02on, we don't need those stuff in
3:40:04analytics and reporting. So I'm going to
3:40:06go and remove it as well. So I think now
3:40:08we have like complete match between
3:40:10those two tables. We don't have any more
3:40:12nulls. So they are completely identical.
3:40:15Of course you might say but we lost an
3:40:17information which record come from which
3:40:19table. So the 2025 and 2026. Well,
3:40:22actually we don't need this information.
3:40:23We have it already in the order date. So
3:40:25it will be just like an extra
3:40:27information that we have already. But if
3:40:28it was like another information like the
3:40:30country or something. I would go and add
3:40:32the source. All right. So now again our
3:40:34orders is the header information. As you
3:40:36can see we have here a lot of context
3:40:38for each order. So very good candidates
3:40:40for dimensions and the order line items.
3:40:43Here we have all the numbers. So this is
3:40:46the detail table for the orders. And
3:40:48this means my friends this going to be
3:40:50our core fact table for the topic sales
3:40:52and orders. So now where we're going to
3:40:54start either we create the fact table
3:40:56now or we start extracting the
3:40:58dimensions from the orders. I'm going to
3:41:00say let's go to the orders and start
3:41:01like creating new dimensions. So what do
3:41:03we have here? We have the customer name
3:41:05the customer city. Well we don't create
3:41:07now another dimension called customers.
3:41:09So we're going to leave it like here.
3:41:10Same thing for the region. Those cities
3:41:12going to stay over here. The date as
3:41:14well. Now here we have something
3:41:16interesting. We have the order channel
3:41:18where we have like some numbers. Those
3:41:20are technical ids. They are not
3:41:21something that we're going to use in
3:41:22order to aggregate. We have as well
3:41:24status and priority. All three
3:41:26informations are actually good
3:41:27candidates for dimensions because we can
3:41:29use them in order like to filter and
3:41:31slice. Now, of course, the question is
3:41:32what is the topic? Are they like related
3:41:34to each others? Could we make like one
3:41:36dimensions out of this? Well, they are
3:41:38actually not related to each other. You
3:41:39can see here the order channel, the
3:41:41status, the priority. They are just like
3:41:43confirmations for the order but they are
3:41:45not like connected together in order to
3:41:47make like good dimension. And of course
3:41:49we will not go and make like one
3:41:50dimensions for each column. This is not
3:41:52good in this scenario. What we're going
3:41:53to do my friends, we're going to build
3:41:54the junk dimension. We're going to put
3:41:56them all together in one and then
3:41:58reference it back in the fact table
3:42:00later. So how we going to build a junk
3:42:01dimension? We're going to go and make
3:42:03another reference out of this. So we're
3:42:05going to say reference and this going to
3:42:06be a dimension. So I'm going to drag and
3:42:08drop it to the dimensions. Let me just
3:42:10minimize this. All right. So right click
3:42:12on it, rename. Well, I'm not going to
3:42:14call it again junk. So this going to be
3:42:16the order flags. Maybe you're going to
3:42:18have another junk from another events.
3:42:20That's why we will not make it like a
3:42:22super generic. Now how we build this?
3:42:24It's very simple. We're going to go and
3:42:25remove all the unnecessary stuff and say
3:42:27okay, remove. We don't need the order
3:42:29total. This is a number. We don't need
3:42:31it for dimension. Now the last thing
3:42:32what we're going to do, we're going to
3:42:33go and remove all the duplicates from
3:42:35all three. So select all and say remove
3:42:37duplicates. Very nice. We have only 14
3:42:39rows. Now, of course, what is missing?
3:42:41We need like a key for this dimension.
3:42:43So, we're going to go and add a column,
3:42:44an index column that starts from one.
3:42:47Then we're going to call it so rename
3:42:49the flag key because this is a surrogate
3:42:52key. We just created this in our model
3:42:55and go to the top. Okay. So, this looks
3:42:57already good. Now, we are not done yet.
3:42:59There is something that I really don't
3:43:01like those order channels. So, as you
3:43:03can see those numbers now, this can
3:43:05confuse a lot. If you just bring it to
3:43:06the reporting, people might go and
3:43:08aggregate or find an average of this.
3:43:10Those are ID for something that we don't
3:43:13know and we cannot leave it like this.
3:43:15We have to make something meaningful of
3:43:16this. So now at work in order to
3:43:18understand what those numbers are, we
3:43:20actually go and read documentations or
3:43:22ask the people responsible for the
3:43:23source system. Maybe ask an expert from
3:43:25the business or from the processes. So
3:43:27at the end you have to find the mapping
3:43:29between those numbers and the real
3:43:31values so that we can show it in the
3:43:33reports. Now let's say that I got it
3:43:34from somewhere those informations. Now
3:43:36the question is how we going to add it
3:43:38to this data model because those
3:43:39informations are not coming from the
3:43:41source. From the source we are getting
3:43:42only those ids. So that means this is
3:43:44new information that I'm adding to this
3:43:46data model and it doesn't exist from the
3:43:49sources. And how we can do that my
3:43:51friends we go and add a data inside our
3:43:53data model by going to the home rippon
3:43:56and say enter data. So this is going to
3:43:58be very simple. We are just creating a
3:43:59table from scratch like an Excel. So the
3:44:02first column going to be the channel
3:44:03code and we need another one where we
3:44:06have the full name. So the channel name.
3:44:10Okay. So it is just like mapping between
3:44:12code and the name. So we make sure that
3:44:14we cover everything from our data. We
3:44:16have the 10 and this stands for online
3:44:19store. Then we have 20. This is the
3:44:22retail partner. Then the 30 this is the
3:44:26channel of wholesale. And 40 is a filled
3:44:30sale. So we make sure that we have
3:44:32covered everything. So we have here
3:44:33actually only four. Okay. So this is the
3:44:35translation the mapping between the
3:44:37cryptical code and the friendly name.
3:44:40All right. So now let's go and hit okay.
3:44:42Well now I have it here as a table. So
3:44:43I'm going to go and rename it channels
3:44:46and then maybe bring it back to the
3:44:48stage because this is something like
3:44:49intermediate, right? So now what we're
3:44:51going to do, we're going to go back to
3:44:52our order flags and we have to replace
3:44:54those numbers with the new values. And
3:44:56we can do that by merging queries. So
3:44:58let's go and find our new table
3:45:00channels. All right. So this is the key
3:45:02and left join. So let's hit okay. All
3:45:04right. Okay. So let's go and expand.
3:45:07Remove the code. We just need the name.
3:45:09And I'm going to move it over here. Of
3:45:11course you can now decide whether you're
3:45:12going to leave the code together with
3:45:14the name. But I'm going to say actually
3:45:16the codes are just like confusing. So
3:45:19I'm going to go and remove it. So now my
3:45:20friends this is what we call data
3:45:22enhancement, data enrichment. And this
3:45:24is very typical things that we do in
3:45:26data modeling because in source systems
3:45:28we're going to get a lot of cryptical
3:45:30things that we cannot just go and
3:45:32present it in the reports. So we have to
3:45:34add from our site something new to the
3:45:36data model and with this comes a really
3:45:38great responsibility because this is
3:45:40very typical what could happen that
3:45:42initially going to go and start mapping
3:45:44everything manually. So everything looks
3:45:45perfect but now suddenly the source
3:45:47systems are generating let's say new ids
3:45:49new values that we didn't map yet. So a
3:45:52typical problem happens that initially
3:45:54we do things correct but afterward we
3:45:56forget about it or maybe we just go and
3:45:58switch to another project and your
3:46:00reports can show only nulls over the
3:46:02time because we don't have values for
3:46:04that. So adding data manually from us
3:46:07the humans to the data model it is
3:46:09something critical and we have to be
3:46:10careful for it. So if you go to another
3:46:13project and you hand this data model to
3:46:15someone else, make sure to inform them
3:46:16that they have to check from the source
3:46:18systems. If there is something new, they
3:46:19have to go and extend this manual data
3:46:22that we added to the data model or what
3:46:24is way better and perfect. If you say,
3:46:26you know what, this mapping should
3:46:28happen in the source system and you are
3:46:30responsible for it and provide it with
3:46:32all other files. This is of course the
3:46:34best option because then we don't have
3:46:36any more the responsibility to touch the
3:46:38data inside the data model. everything
3:46:40going to be like fully automated. Okay.
3:46:42So now back to our junk dimension. The
3:46:44last step is just to polish. So that
3:46:46mean we will just rename things the
3:46:48channel and here we have a status. So
3:46:50only the S and the priority.
3:46:54Okay. So our junk dimension is ready to
3:46:57go. Now let's go back to our orders.
3:46:59Let's see. Okay. Again we say those
3:47:01information is going to stay and that
3:47:02means we just extracted one dimension
3:47:04out of the order which is totally okay.
3:47:07Now let's start with our fact. This is
3:47:08going to be really interesting. Let's go
3:47:10to the order line. This has the lowest
3:47:12details. It makes sense. So, let's go
3:47:14first and make a reference out of it.
3:47:17This is going to be our first fact
3:47:18inside the data model. Let me just uh
3:47:20minimize all those stuff. And then let's
3:47:23go and give it a name. I'm going to call
3:47:24it the fact sales. All right. So, this
3:47:27is the foundations for our fact. We have
3:47:29enough numbers inside it. But now the
3:47:31thing is we need the dimensions. So, we
3:47:34have already here the product name, but
3:47:35we need the customers. We need the junk
3:47:37dimension. We need the order date. So we
3:47:40need all those stuff from the header to
3:47:42be included as well in our new facts. So
3:47:44that means now we're going to start like
3:47:46combining things to our new fact. And
3:47:48here you have to be very very careful.
3:47:49If you do one wrong merge, you're going
3:47:51to destroy the whole facts. So now we
3:47:53come to the rule that we have to protect
3:47:55our numbers. So that means any changes
3:47:57that we do to the fact table, we still
3:47:59have to see the same numbers. And if we
3:48:02see something else, we have of course to
3:48:03fix before merging it to our fact. So
3:48:06now let's try this out. and I'm going to
3:48:08show you how I usually do it. Now, of
3:48:09course, I will not go and start like
3:48:11scrolling down because usually the event
3:48:13tables are really big. So, instead I'm
3:48:15going to go and close and apply. So,
3:48:16just save it. All right. So, I think
3:48:18somewhere is our new facts. But we don't
3:48:20deal now with the data model. Let's go
3:48:22to the report view. And now we're going
3:48:24to go and pick like one important
3:48:25measure from our fact in order to keep
3:48:28tracking it for each change. So if I go
3:48:30to the fact sales over here, we have
3:48:32here some options like for example the
3:48:34line total, the quantity, the cost, the
3:48:36price. I'm going to go with the main
3:48:38measure, this going to be the line
3:48:40total. So let's drag and drop it to our
3:48:42card. So my friends, this is the number
3:48:44and we have to protect this. Anything
3:48:46that we combine, we add to this fact
3:48:49table, it always should result exactly
3:48:51to this same number. If something
3:48:52change, then we have an issue. So we
3:48:55have to know this number by heart until
3:48:57the end of the project. This number
3:48:59should be correct. Okay. So now we have
3:49:01the number. Now let's go back to power
3:49:03query and start adding things. So back
3:49:05to the fact sales. Now what we're going
3:49:07to do in order to bring the other
3:49:09dimensions to our facts, we're going to
3:49:11go and add the header. So we do that by
3:49:14doing a merge query. We're going to go
3:49:15to the orders that we have prepared. It
3:49:18is over here. And how we going to
3:49:19combine it? By the order ID. And of
3:49:21course very important the left outer
3:49:23join. Okay. Now of course we have to
3:49:25expand. And here we have a lot of
3:49:27things. I will not now judge everything.
3:49:29I'm just going to go and remove the
3:49:30order ID that we just used because we
3:49:32have it already. And let's go and add
3:49:34all the things. We can always remove
3:49:36later. So now we have everything from
3:49:38the header, the order. And of course,
3:49:40now the big question is, do we have any
3:49:42changes for the numbers? So let's go and
3:49:44close and apply and wait second until
3:49:46it's loaded. So keep focusing on the
3:49:48number. Any changes? Well, nothing
3:49:51changed. This is perfect. We still have
3:49:53the total sales. Of course, there are
3:49:54like different methods on how to test.
3:49:56Like for example, you can go and build
3:49:57the relationship between them and see
3:49:59whether it is one to many. So you can do
3:50:01like different ways of testing before
3:50:03merging. This is one of them. So that
3:50:05was successful. Now let's go back to the
3:50:07power query. Let's go back to the fact
3:50:09sales. Okay. So with this we have done
3:50:11the big merge between the details and
3:50:12the sales safely. And now we can go to
3:50:15the next steps. Like for example, we
3:50:16have to go and build the keys inside the
3:50:19fact. You know each fact has a lot of
3:50:20keys in order to connect things to the
3:50:22dimensions. And our fact is not ready
3:50:25yet. So now instead of having the
3:50:26product names, we're going to go and
3:50:28have the product key. Same thing for the
3:50:30customer. So let's do that. Let's bring
3:50:32the ids and the keys and we can start
3:50:33with the customer. Now what we going to
3:50:35do, we call it something like lookup. So
3:50:37that means I'm going to go and merge
3:50:39this to the dimension only to get the
3:50:41ids. And again, since we are merging, we
3:50:43have to be very careful. So we're going
3:50:45to go and merge a query again, but this
3:50:47time we are merging with the dimensions.
3:50:49So we're going to go over here and find
3:50:50the dimension customer. So now this is
3:50:53our target to get the customer ID. But
3:50:55of course we cannot now connect using
3:50:57ids. We have to do something scary.
3:50:58We're going to go and use the text the
3:51:01customer name and this can open the door
3:51:03for many mistakes. So we have really to
3:51:05check after that. So customer name
3:51:06customer name join. This is my friends a
3:51:10lookup. So let's go and hit okay. Now
3:51:12what we going to do? How we going to
3:51:13expand? Of course we will not go and
3:51:15bring now all the columns from the
3:51:16dimension. We just want one thing the
3:51:18customer ID. So this one we need and hit
3:51:21okay. Okay. So now what I like to do
3:51:22next is to just bring the ID on the
3:51:25starts so that we have a real fact
3:51:27table. So it's going to be somewhere
3:51:28here. Now of course before we do
3:51:30anything else we have to go and test. So
3:51:32I'm going to go and close and apply and
3:51:34keep my eye on this number.
3:51:37Well nothing changed. That was a
3:51:39successful merge between those two
3:51:41tables and there is no risk about it.
3:51:43Now let's go and do the same thing for
3:51:45the products. So let's go back to the
3:51:48fact sales. Same thing we have here the
3:51:50product name. We need the product ID or
3:51:52key. So merge. And now same thing. We're
3:51:54going to go to that dimension product
3:51:56over here. We join my friends using the
3:51:59product name. So it's not the product
3:52:00code which is really scary. So let's go
3:52:03and hit okay. And we just need a lookup.
3:52:05So that means we need the product key.
3:52:07Perfect. So now let's go as well bring
3:52:09it to the starts side by side with the
3:52:11other keys somewhere here. Now before we
3:52:13touch anything, we have to go and test.
3:52:15Let's close and apply. Keep our eye on
3:52:18the numbers. Did the numbers just
3:52:19changed? Now we have way more total
3:52:22sales than before. Oh no, that was not
3:52:24successful merge with the products. We
3:52:26have something wrong and we have to
3:52:28protect the original number. So yeah, we
3:52:30have to do something. So let's go back
3:52:32to power query and see what is going on.
3:52:34So let's go back to the fact sales.
3:52:37Well, what we can do, we can go and
3:52:39check up actually which order have an
3:52:41issue. So if you select those two and
3:52:43say okay, let's go and group by. I
3:52:45imagine those two should be unique. So
3:52:47let's go and count the rows for that. So
3:52:49now we see a lot of ones but we see two
3:52:52right. So here we have an issue. You can
3:52:53see those orders are fanning out. So we
3:52:56are getting like duplicates. And let's
3:52:57go and check something like for example
3:52:59the order number 25. And then go and
3:53:01remove this test. So now what we going
3:53:03to do? We're going to go and search for
3:53:04this order. It was the 25. All right.
3:53:07Look at this. We have the same order.
3:53:09The product name is identical but they
3:53:12have different keys which is really bad.
3:53:14Of course, the rest going to be like
3:53:16identical. It is getting like fanning
3:53:18out. So, this product has an issue. The
3:53:20same product name but different keys.
3:53:22And this is a typical issue. If we are
3:53:24joining using like names and text,
3:53:26things can go quickly wrong. So, that
3:53:29means we have to go and check the
3:53:30dimension products. So, it was the
3:53:32kitchen something, right? So, let's just
3:53:35search for it. This is the first one.
3:53:38And look at this. We have duplicates
3:53:41actually inside the Damian products. So
3:53:43we have different codes for the same
3:53:45product and if you scroll to the right
3:53:48side you can see one of them is actually
3:53:50have more data than the other. So here
3:53:52we have like brand and the other row
3:53:54doesn't have brand. Same thing for the
3:53:56price and the supplier. Of course this
3:53:58is a data quality issue that came from
3:54:00the source system and here we have like
3:54:02two options. Either we're going to go to
3:54:03the source system and tell them hey guys
3:54:05you have like wrong stuff you have the
3:54:06same product twice. Go and clean up the
3:54:08data and then everything going to be
3:54:10fine here. or you're going to go and
3:54:11manually start removing things. But of
3:54:14course, this is no guarantee that later
3:54:16we will not get any like quality issues.
3:54:18Now, if the source system don't want to
3:54:20clean the data and they going to leave
3:54:21it as it is, then either you're going to
3:54:23go and now like clean things manually or
3:54:25what is way better, we usually build
3:54:26like a data warehouse where we build
3:54:29pipelines and quality gates in order to
3:54:31detect and fix those issues. But now for
3:54:34this project, since we don't have like a
3:54:35lot of products, we're going to go and
3:54:36manually remove those bad rows. So here
3:54:38we have to decide which version should
3:54:40stay and of course it's going to be the
3:54:42one with more infos. So that means with
3:54:44the nulls should be dropped out of our
3:54:47dimension. Now of course we have to
3:54:48detect how many other like scenarios
3:54:51like this. So I will just remove this
3:54:52filter select all. Now how we going to
3:54:55do it? We're going to go and just group
3:54:56by the product names. So we're going to
3:54:58go and say based on the product name go
3:55:00and count the rows and change the filter
3:55:03to two. You can see we have the issue
3:55:05with two products the kitchen and the
3:55:07audio. So it is not that a big deal what
3:55:09we're going to do. We're going to go
3:55:11remove this test. Of course, now the
3:55:12question is how we going to remove those
3:55:13two rows. I will not do it like as the
3:55:15last step inside our dimension. I'm
3:55:17going to go way back. Maybe at this step
3:55:19filtering the rows as we removed the one
3:55:22bad row where the product key is equal
3:55:24to Z zed. Now of course the question is
3:55:26where are those stuff? Well, it is the
3:55:28last two rows. You can see here we don't
3:55:30have brands for them. We don't have a
3:55:32unit price. And now I remember we had
3:55:34like a really nice column for the source
3:55:36ID. So you can see here we have like a
3:55:38source ID and row identifier for each
3:55:40one of them and we can use it in order
3:55:42to filter the data out. So this is one
3:55:44bad ID. I'm going to go and copy the
3:55:46value and then use it in order to filter
3:55:48the data. So I will check this out. Same
3:55:52thing for the last row. So I'm going to
3:55:54go and copy this as well and as well use
3:55:56it in order to filter the data out. All
3:55:58right. So as you can see now in our
3:56:00filter we have three things. We are
3:56:02filtering the data based on the product
3:56:04code and based on the source ID. So now
3:56:07let's go and check again. We have to
3:56:09group up the data by the product name.
3:56:11So group by insert and product name. Now
3:56:15let's check the filter. Yes, we have
3:56:17only one. So that means my friends, our
3:56:19product names are actually unique. But
3:56:21be careful now to just remove the test.
3:56:23All right. Now we go to the last step to
3:56:26make sure that everything is working
3:56:27correctly. Nice. So now let's go and
3:56:29save and check again our numbers. So
3:56:33come on. Come on. Yes, we got back our
3:56:35number with that we protected the number
3:56:37and the fact and it was successful merge
3:56:40to the product. So we can now go and
3:56:42continue. Let's go back to the power
3:56:44query. So where is our fact table? So
3:56:47with this now we got the two big
3:56:48dimensions the product key, the customer
3:56:51ID. Now what is left is actually the
3:56:53junk, right? So we have those three
3:56:55columns and we have to connect the junk
3:56:57to it. Now I see already that we have an
3:56:59issue for the lookup. We need all those
3:57:01three in order to join it with the
3:57:03dimension. But we just removed the order
3:57:05channel, right? So if I go now and
3:57:07merge, it will not work. That means I'm
3:57:09going to take it back and going to go to
3:57:11the junk and I'm going to go and bring
3:57:13the ID back. So I believe this was the
3:57:16step. So let's go and bring it back.
3:57:18Yes. So we need those stuff. I'm going
3:57:21to go and rename it. This is the channel
3:57:24code. Now let's go back to the fact
3:57:26sales and let's do the merge with the
3:57:28junk dimension. So we need it. It is
3:57:31over here the order flags. Now we're
3:57:33going to go and mark all those stuff but
3:57:34not the channel itself because it has no
3:57:36idea about it. It is something that we
3:57:38added. Then we're going to go to the
3:57:39right side and select those three. So
3:57:42now things should work and in the
3:57:44correct order by the way. So the order
3:57:46channel, the status and the priority.
3:57:48Same thing from the dimension. Left
3:57:50join. Okay. Now the next step we're
3:57:52going to go and expand. Since it is only
3:57:54a lookup, we just need the key. So okay.
3:57:57Yes, we got the junk flag. Uh let's go
3:58:00and do it side by side with the others.
3:58:02And of course the big question, do we
3:58:04have issues with our number? So I'm
3:58:06going to go and close and apply. Well,
3:58:08look at this. It is safe. So our number
3:58:10is protected. Let's go back. I know we
3:58:12are jumping between but this is part of
3:58:14the job. We have to do things like
3:58:16carefully. All right. So with this now
3:58:17we got all the keys, all the ids inside
3:58:20our fact and it is ready to get
3:58:22connected to the dimensions. And now
3:58:24we're going to go to the next phase
3:58:25where I like to go and drop things and
3:58:27polish the facts in order to follow the
3:58:29standards and the rules that we have
3:58:31added at the beginning. So now we're
3:58:32going to start with the first thing. Do
3:58:34we really need all those columns inside
3:58:35our facts? So let's do it step by step.
3:58:37The line ID, we need the order ID as
3:58:39well in order to count how many orders
3:58:41inside the fact. The product name, we
3:58:43don't need it anymore. We just used it
3:58:45in order to build the keys. So we're
3:58:47going to go and remove it. Then we have
3:58:49the keys. That's of course what you
3:58:51need. And then we have the numbers. We
3:58:53need all that stuff in the facts. Now we
3:58:55come to something that is really
3:58:56important to understand. Now the thing
3:58:58is in transactional source systems they
3:59:00usually store the same informations over
3:59:03and over in different tables just in
3:59:05order to optimize the performance of
3:59:07their UI and applications. And this is
3:59:09of course a big issue for us because
3:59:11then we're going to have like the same
3:59:12information everywhere inside our data
3:59:14model. And this is going to make a big
3:59:16confusion on how to build the reports.
3:59:18So that means my friends you have to
3:59:20protect the data model by having like
3:59:22one single point of truth for each info.
3:59:24So that means if I have the customer
3:59:26city inside the dimension customer I
3:59:29don't need it somewhere else. This going
3:59:30to be the dimension customer is the
3:59:32single point of truth for the cities of
3:59:34the customers. So I don't have to leave
3:59:36it in like another dimension or to leave
3:59:38it in the facts. So always try to reduce
3:59:40the noise that we inherited from the
3:59:42source systems and make a decision where
3:59:45you going to store what but only once. I
3:59:47said I was only going to say it once.
3:59:49>> So that means my friends, we will not
3:59:51leave the customer city inside our
3:59:53facts. We have it already inside our
3:59:55dimension. Look at this. We have here
3:59:58the city of our customers. So that means
4:00:00we don't need this extra noise that we
4:00:02have in the facts. We will just go and
4:00:04remove it. By the way, this is one of
4:00:06the things that I have noticed a lot.
4:00:07People going to say, you know what, we
4:00:09might need it. So you really have to
4:00:10take a hard decision and start removing
4:00:12things that you don't need and keep it
4:00:14only in one place. Same thing as well
4:00:16for the region name. It as well belongs
4:00:18to the customer dimension. So let's go
4:00:21remove. And now here's something
4:00:22interesting to what we said. We have
4:00:24here the ship to city and pill to city.
4:00:27This is different information than we
4:00:29had in the dimension customer. This
4:00:31belongs to the transaction to the order.
4:00:33So for each order we have the city where
4:00:35it is shipped to. And here the customers
4:00:38might change from one order to another
4:00:40to which city or to which location the
4:00:42order should be delivered. So the city
4:00:44here belongs to the transaction to the
4:00:46event. It is not like master data of the
4:00:48customer. So that means this has
4:00:50different meaning than the cities that
4:00:52we have inside the dimension. It is
4:00:54something here transactional and could
4:00:56change from one order to another. So
4:00:58that's why I will not go and remove
4:01:00those informations because this belongs
4:01:02to the event to the transaction. So it's
4:01:04always really important to understand
4:01:05the business to understand the domain
4:01:07how things works. So let's keep going.
4:01:09The order date of course we need it. Now
4:01:11we have a lot of things that we don't
4:01:12need anymore because they were only here
4:01:15in order to do a lookup. So I'm going to
4:01:17go and remove those stuff that were
4:01:18belonging to the junk. And now we come
4:01:20to something really interesting. The
4:01:22total sales in the header. So as well we
4:01:24have to decide because now we have two
4:01:26things. Look at this. We have the total
4:01:28sales for the whole order. And here we
4:01:30have like the sales for each item each
4:01:33line in the order. Now of course if
4:01:34everything is clean the sum of the line
4:01:37of totals should be exactly the same
4:01:39value that we have here from the header.
4:01:41So that means the grain between those
4:01:43two are different. And now here usually
4:01:45I decide on the following where I say we
4:01:47go and remove any pre-agregated values
4:01:50because to be honest I don't trust it.
4:01:52If I need the total sales for each order
4:01:54actually I just go and summarize those
4:01:56values. So I have the data already
4:01:58available. I don't want to have like
4:02:00pre-agregated things inside the data
4:02:02model that's maybe outdated or maybe the
4:02:05calculation is not anymore correct.
4:02:07That's why I tend to remove any
4:02:09pre-agregated things that I inherit from
4:02:11the source system. Especially if I have
4:02:14the same information from another
4:02:16detailed measure. So that's why the
4:02:18order total that we got from the header,
4:02:19it is something unnecessary and I'm
4:02:21going to go and remove it. I can always
4:02:23get it back if it's needed. All right,
4:02:24my friends. So with this we have removed
4:02:26a lot of things from our fact table and
4:02:28now things looks really good. Now I
4:02:30would like always to order my fact like
4:02:32this. We have always at the start all
4:02:34the IDs and then all the dates and at
4:02:37the end all the measures all the
4:02:39numbers. It is not something that's very
4:02:41important but I like to order it like
4:02:42this. So that's why I'm going to go and
4:02:44grab all the context to the starts and
4:02:47just leave the numbers at the end. So
4:02:48now look at this. We have ids
4:02:50everywhere. Then we have the cities. We
4:02:52have to do something about it. After
4:02:53that we have the dates. We have only one
4:02:56and at the end we have the bundle of all
4:02:58measures and numbers that we have in the
4:03:00fact. So we are almost there but to be
4:03:02honest I'm still not happy about those
4:03:04two. So we have the ship to city and the
4:03:06pill to city. Now of course I can leave
4:03:08those informations like this but we
4:03:10could build a dimension for it. So
4:03:12that's we have like ids instead of text
4:03:15here which can reduce of course the size
4:03:16of the fact table and at the same time
4:03:18maybe I need more context to those
4:03:20cities like for example the region or
4:03:22the country and so on. So it's always
4:03:25better to convert those contextes to
4:03:27dimensions. So that means we have
4:03:29actually to construct a new dimension
4:03:31called dimension geo or locations or
4:03:34whatever you want to call it. Now of
4:03:36course the question is what is the
4:03:37source for this new dimension? Should we
4:03:39get it from the fact or do we have like
4:03:41some row tables that can help us with
4:03:43that? Well, remember we had really nice
4:03:45row table called cities. This is
4:03:47perfect. This is already like waiting
4:03:49for us to build a dimension for it. We
4:03:51just skipped it. As you can see, we
4:03:52cannot like think of everything at the
4:03:54start. As we progress in the data model,
4:03:56we're going to understand, okay, we need
4:03:57more things and we always can go back to
4:03:59the row data and make new dimensions.
4:04:01So, how we're going to do it? This going
4:04:02to be very simple. We're going to go and
4:04:04create a reference and we're going to
4:04:05drag this reference to the dimensions.
4:04:07We're going to go and give it a name dem
4:04:09go. Now we have to make sure that this
4:04:11has no duplicates. I'm not really sure
4:04:13whether it has or not. So I'm going to
4:04:15go and remove duplicates. Well, nothing
4:04:17changed, but I'm going to leave it as
4:04:18step. Maybe things changed. Of course,
4:04:20what is missing this dimension? We need
4:04:22an ID, right? So we're going to go and
4:04:23add a new index that starts from one and
4:04:26bring it to the top. So let's go and
4:04:28rename it. This going to be geo key,
4:04:31surrogate key. And the last thing that
4:04:32I'm going to do just rename things. So
4:04:34this is the city and this is the region.
4:04:38Perfect. Now we have new dimension. Of
4:04:40course, what is the next step? We're
4:04:42going to go and connect it to our fact
4:04:43sales. So over here using the city name.
4:04:46But my friends, we're going to go and
4:04:47connect it twice because here we have
4:04:49like two cities, right? They are
4:04:51different. The city for the shipment and
4:04:52another one for the pill. So let's go
4:04:54and do that. We're going to do it
4:04:55quickly. We're going to go and merge and
4:04:57then grab our new dimension. It's going
4:04:59to be of course the city. So and then
4:05:02hit okay. Expand. We need only the key.
4:05:05All right. And then we're going to merge
4:05:07again for the other column. So same
4:05:09dimension. We need the city, but this
4:05:11time for the pill. All right. Expand as
4:05:13well. We need the key. Now, before we
4:05:16forget, we're going to go and rename
4:05:17things. So we're going to call it sheep
4:05:19to city key. And the other one bill to
4:05:24city key. All right. So now let's go and
4:05:26grab them to the other keys. So we can
4:05:29put it over here. And now we're going to
4:05:31go and remove those two cities. We have
4:05:33ids. So remove. Perfect. Now let's have
4:05:36a final look. Look at this. We just have
4:05:39numbers. Now the last step that I'm
4:05:40going to do is that I'm going to go and
4:05:42just rename all the columns to follow
4:05:43our standards. The snake case. All
4:05:45right. So now I just went through all
4:05:46the columns and everything is renamed.
4:05:48Look at this. Everything looks perfect.
4:05:50We have only numbers and date. And this
4:05:52is exactly how the fact table supposed
4:05:54to be. If you get it, my friend like
4:05:56this then congrats. We just built a
4:05:58perfect core fact table for our data
4:06:00model out of this complicated setup, the
4:06:03header and the details. Now I'm going to
4:06:04say the fact is ready. And now we're
4:06:06going to go to the satisfying part where
4:06:08we're going to go and deactivate some
4:06:10row tables and then we're going to go
4:06:11ahead and connect the dimensions to our
4:06:13fact. Let's go and do that. I'm just go
4:06:15and deactivate now the orders both of
4:06:18them and as well the order line and then
4:06:20close and apply. All right. So now what
4:06:23we're going to do, let's go and make
4:06:24like two words on the left side. It's
4:06:27going to be this chaos. But on the right
4:06:28side, we're going to have our new clean
4:06:30data model, the star schema. So this is
4:06:33our fact. And now let's go and bring our
4:06:35dimensions. So product, the geo,
4:06:37customers, and the junk dimension. So
4:06:40let's form the star and make sure we
4:06:43don't have any garbage like in our
4:06:45model. So now let's go and zoom in. We
4:06:47still have to do it carefully how we
4:06:48connect the dimensions to the fact, but
4:06:50the pattern it's always the same. So
4:06:52we're going to grab the customer ID and
4:06:54put it on the fact here. It's very
4:06:56important that you see one to many
4:06:58relationship and the filter is single.
4:07:00So never both. Okay. So that's it. Let's
4:07:02save. This looks good. One to many and
4:07:05the arrow going to the fact. Same thing
4:07:07for the product key. So drag and drop.
4:07:09One to many single. Save. And for the
4:07:12junk. So we need flag key is over here.
4:07:15One to many single. And now comes
4:07:17something of course really interesting.
4:07:19We have only one dimension. But we have
4:07:21to connect it twice to our fact table.
4:07:23one for the billing and another one for
4:07:25the shipment. Well, we're going to do
4:07:26that. So, the geo key goes to the
4:07:29shipment. One to many single. This going
4:07:31to be the active one and the second one
4:07:33going to be the billing. So, drag and
4:07:35drop one to many and single. So, save.
4:07:38So, with us, my friend, our dimension go
4:07:40is actually a role-playing dimension. We
4:07:43have two relationships to the fact
4:07:44table. One is active that's going to be
4:07:46used all the time and another one is
4:07:48inactive. That's we have to use ducks in
4:07:51order to use this relationship of
4:07:52course. But now you might say okay this
4:07:54is like way too more complicated than
4:07:56before. If we had like both of them
4:07:58inside a fact table I can just go and
4:08:00use them inside my report. Well this is
4:08:03a fair point. If you never need like
4:08:05more geo informations but if you start
4:08:07like needing the regions the countries
4:08:09and this geo is like getting bigger and
4:08:11bigger then it makes no sense to leave
4:08:13it inside the fact and as well it makes
4:08:16no sense to go and duplicate the
4:08:17dimension twice. Those are all bad
4:08:19designs. So now my friends, look at
4:08:22this. If I zoom out, you can see clearly
4:08:24the two words. On the left side we have
4:08:26the full chaos and on the right side we
4:08:28start having the shape of our star
4:08:30schema our goal and things looks way
4:08:33more better than before. So this exactly
4:08:35my friends how we migrate from bad data
4:08:38model to a new better model that's going
4:08:40to deliver only correct numbers, good
4:08:42performance and as well reduce any
4:08:44confusion. Okay. Okay. So now we have to
4:08:46keep going but this time I'm going to go
4:08:47and pick something that is little bit
4:08:49easy so that we have break. I will not
4:08:52pick something complicated. We're going
4:08:53to go this time with the inventories.
4:08:55This is an easy one. So let's go and do
4:08:58something about it. We're going to go
4:08:59and enter the power query. So what do we
4:09:02have here the inventories? Oh yes we
4:09:04have here the product names and for each
4:09:06month we have the total amount of items
4:09:08that is planned. Well of course the
4:09:10first question is is it a fact or
4:09:11dimension? Well, since we have numbers,
4:09:13we have dates, this is going to be a
4:09:15fact. So, before we do anything, we're
4:09:17going to go and make a reference, then
4:09:18move it to our facts. This is our second
4:09:21fact. I'm going to go and call it fact
4:09:23inventory. And now we can start
4:09:24preparing the fact. Now, the first easy
4:09:26issue with that, the first column is not
4:09:28promoted as a header. So, the column
4:09:30names are weird. That's why we're going
4:09:32to go and say use first row as header.
4:09:34Easy win. Now, by looking to this, you
4:09:36can see it is really nice table for us
4:09:38for humans to read. But this is very bad
4:09:40for data moduling because the date
4:09:43information is stored as a column name.
4:09:45We cannot use this of course in
4:09:47reporting. That's why we have to convert
4:09:49this from human ready to data module and
4:09:51reporting. And how we can do that using
4:09:53unpivot by turning the columns into
4:09:56rows. It is very simple what we can do.
4:09:58We can go to transform then select all
4:10:00the columns. And here we have this nice
4:10:02function unpivot. Look at this. Like
4:10:04magic. Now all the column names are
4:10:06values inside my table. And this looks
4:10:09already like a fact table. And now after
4:10:11that I think this is the data type text
4:10:13right. So we can see here text we have
4:10:15to go and convert this to a real date.
4:10:17Then it's going to be easier to use in
4:10:19the reporting. And same thing that we
4:10:20have done before we have to go and get
4:10:22the product key. So we need a lookup.
4:10:24And how we going to do it? As usual we
4:10:26go and merge with the dimension. So
4:10:28dimension product connect the names and
4:10:31say okay. Now after that we're going to
4:10:33go and get only the key bring it to the
4:10:36start and remove the product name. We
4:10:38don't need it anymore. So remove and of
4:10:40course my friends you have to go and
4:10:42check things. Now I'm doing things
4:10:43quickly but still we have to go to the
4:10:45report make a card go and calculate all
4:10:48the values after merging. Are we getting
4:10:50like any duplicates? But now in order to
4:10:51make things quicker I will not do that
4:10:53but you have to do it at your side. I
4:10:55would never trust that everything is
4:10:56okay now. So now the last step in our
4:10:58new fact I'm going to go and rename
4:11:00things. So this is the product key and
4:11:02this going to be I'm going to call it
4:11:04for now month and instead of value I'm
4:11:06going to call it units. All right
4:11:08friends. So actually that's it. With
4:11:09this we got our second fact and it is
4:11:11perfect. It has an ID. It has a date and
4:11:14as well a measure. Now before we go and
4:11:16start connecting things I'm just going
4:11:17to go and like deactivate so the
4:11:19inventory will not be loaded. And
4:11:21actually we have here few things that we
4:11:22didn't deactivate before. So we have
4:11:24here the orders and as well the
4:11:26channels. So you we don't need you.
4:11:29Okay. Okay, so now let's close and
4:11:30apply. You can see the chaos is getting
4:11:32reduced by the time. And here we got our
4:11:34new fact table. Let me just zoom in and
4:11:37have a look. Probably I done it
4:11:38correctly. I wanted to do that. But as
4:11:40you can see, we never connect the fact
4:11:42tables together. So there's no direct
4:11:44lines using the product key. We connect
4:11:47the fact tables always using the shared
4:11:49dimension. And this can be the dimension
4:11:51products. Of course, a quick look. It is
4:11:53one to many relationship. And the
4:11:55direction is correct. As you can see, if
4:11:56you do things correctly, PowerBI going
4:11:58to love it and it's going to build it
4:12:00for you correctly. So my friends, look
4:12:01at this. We are moving from star schema
4:12:04and now we are like building the galaxy,
4:12:06right? Which is of course still star
4:12:08schema. So my friends, that was a quick
4:12:10win. All right, friends. So that was the
4:12:12inventory. It was a little break. Now
4:12:14let's go back to something more
4:12:16advanced. Now we have to go and pick
4:12:17something from our data model. So we
4:12:19have invoices and stuff. I'm going to
4:12:21say let's go and solve the campaign. So
4:12:24we have the campaign log and another one
4:12:26the campaign is cues. So it's time to
4:12:28solve this puzzle. As usual, we're going
4:12:30to go inside Power Query. So let's go
4:12:32and do that. Okay. So let's go to the
4:12:34campaign log. So again, what do you have
4:12:36here? The campaign name, the channel,
4:12:38the start date, and the end date of the
4:12:39campaign, the budget date, and some nice
4:12:43numbers. So clearly this looks like a
4:12:45nice fact table. But still inside it, we
4:12:48have some context about this fact. So
4:12:50the context is my friend, the campaign
4:12:53itself. So it looks like a clear split
4:12:55between one dimension and one fact,
4:12:57right? And let's have a quick look to
4:12:58the other one. The campaign is cues. We
4:13:00have here the campaign name and the
4:13:02product names as a list. So this sounds
4:13:05like the connection to the rest of our
4:13:07star schema, the products, the product
4:13:09code. So I think I got the full big
4:13:11picture. Now let's go and start building
4:13:13the dimension, the new one. All right.
4:13:14So as usual, we're going to go and
4:13:16create a reference out of the row table.
4:13:18Now we're going to start with the
4:13:19dimension. So that's why we're going to
4:13:20bring it to the dimension folder and
4:13:22rename it. So this is going to be the
4:13:23dimension campaign. All right. So now
4:13:26which columns going to stay and which
4:13:27going to leave? Well my friends we have
4:13:29here clear split. So all those columns
4:13:32are actually transactional columns that
4:13:34belongs to the fact and the rest going
4:13:35to be the dimension. So let's get rid of
4:13:38those. Now of course as usual we're
4:13:40going to go and remove all the
4:13:41duplicates from those columns. So remove
4:13:44with this we got our six campaign that
4:13:47we have inside our data model. And now
4:13:48by looking to this you might say but
4:13:50wait a second what is going on here
4:13:53actually the fact always contains like
4:13:55dates and numbers and this dimension
4:13:57looks like a fact. So why we are calling
4:14:00this a dimension campaign? Well totally
4:14:02fair point at the start I was as well
4:14:04confused about this topic but my friend
4:14:06the dimensions and facts are not defined
4:14:08by the data type of the columns. Think
4:14:10about it a dimension is just a context a
4:14:13description for an event and a fact
4:14:15table it is the event itself. It is
4:14:17transaction. It is something that keep
4:14:19happening every day. So something that
4:14:21grows fast. Right? So now by looking to
4:14:23this my friends, this is not
4:14:25transactional. This is not an event.
4:14:27Yes, it has a start date and end date.
4:14:29But this is actually describing the
4:14:31campaign. It is more context, more info
4:14:34for the campaign itself. And the number
4:14:36budget over here, it is a fixed value
4:14:38that as well describe how big is our
4:14:41budget for this campaign. So yes, it
4:14:43looks like an event, but this is not
4:14:45transactional. Those are just describing
4:14:47the campaign. Now in order to understand
4:14:49what is a real transaction, just go back
4:14:52to the original table. Look at this what
4:14:54each row here is representing. So for
4:14:56the first row for example, we have the
4:14:57campaign name, the channel, the start
4:14:59and ends, the budget. But here we have a
4:15:02real transaction with the dates. So we
4:15:04can see every day what is happening. So
4:15:06in the first day what happens? We spent
4:15:08so much, we got so much clicks and
4:15:11impressions. What happens in the next
4:15:12day for the same campaign? We spent less
4:15:15but we got more clicks and so on. So my
4:15:17friends all those columns are actually
4:15:19transactional columns. Every day we are
4:15:21like getting something new about the
4:15:23campaign. And if you go to the left side
4:15:25you're going to see those informations
4:15:27are really static. So the budget will
4:15:29not change the start and end will not
4:15:31change as well. And because my friends
4:15:33those informations are describing the
4:15:36event itself. So the left side is not
4:15:38changing so much. That's why it is a
4:15:40dimension. But on the right side there
4:15:42is a lot of transactions and movement
4:15:44and that's why my friends those four
4:15:46columns are effects. I hope this makes
4:15:48sense but this is how we define
4:15:50dimension and effect not only based on
4:15:53the data types but as well what is going
4:15:55on how the transaction is happening and
4:15:57by understanding the business itself. So
4:15:59that's why I decided on the split based
4:16:01actually on the data. Now let's go back
4:16:03to our dimension. We are not done yet.
4:16:05Of course what is missing? We need an
4:16:07ID. So start from one, move it to the
4:16:09left side. And actually that's it. The
4:16:11last thing that we have to do is to just
4:16:12go and rename things. So I will just do
4:16:14it quickly.
4:16:16This is the key. This is the name.
4:16:20This is the channel start date.
4:16:27All right. So I can say our dimension is
4:16:29ready. Now let's go and create our fact
4:16:32table. So same thing we're going to go
4:16:33and create a reference. As you can see
4:16:35now our work is repetitive, right? This
4:16:37is exactly how things works. So this
4:16:40going to be the fact campaign but I'm
4:16:42going to call it spend since we have the
4:16:44spending there. All right. So now what
4:16:46we're going to do actually we don't need
4:16:48all those affirmations. We have them
4:16:49already in the dimension. That means we
4:16:51need now to do a lookup in order to get
4:16:53the campaign key. So as usual we go and
4:16:56merge queries with our new dimension the
4:16:58campaign. We join with the campaign
4:17:00name. It is unique in our dimension. And
4:17:02then say okay. Now, since this is a
4:17:04lookup, we just need the key. We don't
4:17:06need all other informations. Drag it to
4:17:08the start because we have always the
4:17:10keys at the start. And get rid of all
4:17:12those things. Those are in our
4:17:14dimension. So, now what is left? We have
4:17:16the date, impression, click spent. Looks
4:17:18nice. The last step, we're going to go
4:17:19and just rename things. So, I'm going to
4:17:21do it quickly. Campaign key. And I'm
4:17:25going to make everything lowered.
4:17:30All right, my friends. So with this now
4:17:32we got our third fact inside our data
4:17:34model. Now let's go to the last table we
4:17:36have here still the campaign skues. Now
4:17:39before we start working on this we're
4:17:40going to go and create a reference out
4:17:42of it. So reference now I'm going to go
4:17:44and put it in the facts but I'm going to
4:17:46explain myself later. Okay just stay
4:17:48with me. Now the first issue with that
4:17:50we have here we don't have a column
4:17:51names. So that's why I'm going to go and
4:17:53promote it as a header. Now what do you
4:17:55have here? We have like a list of
4:17:57product codes. Uh this is of course a
4:17:59nice thing but I cannot use it in
4:18:00reporting right I cannot go and connect
4:18:02it now to our model. So that's why we
4:18:04have to do something about it and we
4:18:06call moving a list into rows and
4:18:08explode. So we have to explode it into
4:18:10multiple rows and we can do that easily
4:18:13by going to the column and say split
4:18:16columns by delimiter because we have
4:18:18here a comma. Let me just move it a
4:18:19little bit. So the delimiter is actually
4:18:21a comma. Now the thing is I will not go
4:18:23and split by columns. I don't want each
4:18:25value to go in separate columns. This
4:18:27makes no sense. This time we are
4:18:29splitting by row. So if you go to the
4:18:31advanced settings, you can see here we
4:18:32split by columns or rows. I'm going to
4:18:34go with the rows. All right. So actually
4:18:36that's it. Let's go and hit okay. Look
4:18:37at this. Now each product code is now a
4:18:40new row. But now before we celebrate
4:18:42this, we have equality issue. If you see
4:18:44like some values are shifted little bit
4:18:46to the right side. You can understand
4:18:48this if you go back to the original
4:18:49steps at the start. You can see the
4:18:51values are yes splitted by comma but
4:18:53there's like after that a space. This is
4:18:55just like a cosmetic thing for us in
4:18:57order to read the list easier. But of
4:18:58course, this is causing an issue after
4:19:00we explode. So you see the first value
4:19:02has no space, but all other values are
4:19:04like moving to the right side because of
4:19:06the white space at the start. So how to
4:19:08solve this? It is very simple. We're
4:19:09going to right click on it and then go
4:19:10to transform and here we have the nice
4:19:12trim. Look at this. Everything moved to
4:19:14the left side and we don't have any more
4:19:16empty space because my friends, we have
4:19:18always to take care of the values that
4:19:19are used for the merge later. All right.
4:19:22So now what we going to do with this, my
4:19:23friends? Well, this is not like a
4:19:25dimension because it is like two
4:19:27different entities, right? The campaign
4:19:28and the products. And as well, you can
4:19:30see it in our dimensions. We have a
4:19:32dimension for the product. We have
4:19:33dimension for the campaign. And this is
4:19:35like just mapping between them. So, I
4:19:37cannot go and put it in the dimension
4:19:38product or in the dimension campaign.
4:19:40And of course, it makes no sense to
4:19:42create like a third dimension called
4:19:44campaign product mapping because this is
4:19:46not one entity. So, now in order to
4:19:48understand more what is this, let's go
4:19:50and convert those values to keys. as we
4:19:53have done like in the facts. So now
4:19:54instead of the campaign name, I'm going
4:19:56to go and make a lookup and grab it from
4:19:58the dimension. So campaign and we
4:20:01connect by the campaign name. All right.
4:20:03Now let's go and grab the key. We're
4:20:05going to do the same thing as well for
4:20:07the products. So let's go and merge
4:20:09dimension product. This time we are
4:20:11connecting using the product code, not
4:20:13the product name. All right. Now let's
4:20:16go and get the key, put them side by
4:20:18side and get rid of those two
4:20:20informations. So remove columns. Now the
4:20:22last thing I'm just going to go and
4:20:24replace the names. So this is the
4:20:25campaign name and the other one is the
4:20:28product key. Now look at this my
4:20:30friends. If you have table that contains
4:20:32a lot of keys and you have the feeling
4:20:34this is like mapping or a bridge between
4:20:36multiple dimensions. This table has no
4:20:39numbers just like collection of keys.
4:20:41Then my friend welcome to a new type of
4:20:43facts. We call it the factless fact. A
4:20:46factless fact it is like a classical
4:20:48fact table. It has a lot of ids in order
4:20:51to connect multiple dimensions. It has
4:20:53as well dates. But the only difference
4:20:56is that it doesn't contain any numbers.
4:20:59So no measures. So that means with the
4:21:01factless facts, we are not measuring
4:21:03anything. So we are not interested about
4:21:04the how much. We are only tracking
4:21:06whether something happened or not. So it
4:21:09is very simple. If your fact has no
4:21:10measures, no numbers, then this is a
4:21:12factless fact. So in this example, it is
4:21:15little bit weird. Okay? It is not that
4:21:17like perfect fact that we usually have.
4:21:19We don't have a date only the keys. And
4:21:21as well we cannot turn this into
4:21:22dimension because this is not describing
4:21:25actually anything. If this keeps like
4:21:27really weird inside your data model, you
4:21:29can go and just get rid of it. But for
4:21:30now, we just build it. That's why I'm
4:21:32going to go and call it the fact
4:21:34promotion coverage. Okay. So this is our
4:21:36weird fact. But of course, we have way
4:21:38better fact here that you can do a lot
4:21:40of analyzes using it. And we have our
4:21:43new dimension. So now next what we're
4:21:44going to do with that we're going to go
4:21:45and deactivate things. We don't need
4:21:47them anymore. So disable load. Yes. Same
4:21:50thing for the campaign is cues.
4:21:53All right. So now let's go and close and
4:21:55apply. All right. So now we have here
4:21:57less things on the chaos word. We are
4:21:59moving to the right side. And here we
4:22:01have our new stuff. So this is our
4:22:03factless facts. Let me zoom in. This is
4:22:06our new facts. The campaign spend. It is
4:22:09connected correctly to the dimension
4:22:11campaign. So one to many relationship
4:22:13and the arrow is correct and the
4:22:16dimension campaign it is a shared
4:22:17dimension to the factless facts which is
4:22:20just playing the role as a bridge
4:22:21between the product and the campaign.
4:22:23Look at this things are getting
4:22:25connected. Yeah slowly. So I will just
4:22:27move things a little bit closer. So now
4:22:30by looking at this you can see our
4:22:31shared dimension the product is shared
4:22:33now by three facts. So it is a really
4:22:35very important one of course and this is
4:22:37of course a typical thing because in
4:22:38each business the products are the most
4:22:40important things. So we are
4:22:42transitioning right? All right. Now my
4:22:44friends, we come as well to a point
4:22:45where it's going to get a little bit
4:22:46heavy. But I'm going to explain it step
4:22:48by step. So that means if you are
4:22:50watching and you didn't take break, go
4:22:52take fresh air, make a cafe and then we
4:22:54continue. This is going to be a little
4:22:56bit tricky, but we're going to see.
4:23:00Okay, let's go. So now as I said, we
4:23:01have now some events that we're going to
4:23:04tackle in one go. Of course, as usual,
4:23:06we're going to go back to the Power
4:23:07Query in order to understand the content
4:23:09and reshape things later. So now what
4:23:11else that we have inside our table? We
4:23:13have the invoices. So you can see we
4:23:14have an invoice ID, an order ID, the
4:23:17customer name, invoice date, amount. So
4:23:19this is as well transactional. This is
4:23:21an event. Now if you go to the payments,
4:23:23you're going to have the same feeling.
4:23:24We have the payment ID, the invoice ID,
4:23:26customer payment day, and the amount.
4:23:29Now if you go to the shipments, again,
4:23:31very close shipment ID, order ID, and so
4:23:34on. So my friends, all of those things
4:23:36are events and somehow they are all
4:23:38connected together. Now in order to
4:23:39understand how to deal with this, we
4:23:41have first to understand the process. So
4:23:43the process is very simple like any
4:23:45order. We have a customer which is the
4:23:47company. They go and place an order. So
4:23:50this is the first step. After that we're
4:23:52going to go and ship the products to the
4:23:54customer. So this is the second step.
4:23:57And of course it going to follow up with
4:23:58a delivery. So this going to be the
4:24:00third step. So we're going to deliver.
4:24:02And after that we're going to issue an
4:24:04invoice. So this is number four. And the
4:24:07last step my friends the payment is
4:24:09going to happen. So pay this is step
4:24:12number five. So actually this is the
4:24:14whole process of ordering something from
4:24:16our business. Now of course each step in
4:24:18our process is actually a transaction.
4:24:20So it has like a date. It has a number.
4:24:23So how much the total sales for this
4:24:25order. Same thing for the shipment. The
4:24:27delivery had as well date invoice and as
4:24:31well we have like some numbers on top of
4:24:32it and how much the customer did pay. So
4:24:35each one of them has their own table and
4:24:37transactions. But everything is actually
4:24:39my friends are connected back to the
4:24:41order. This is the main trigger for the
4:24:44whole event. Now of course what are our
4:24:46options in order to build this process
4:24:47inside our data model? Either we're
4:24:49going to go like the source system where
4:24:51we're going to go and build a table for
4:24:53each step. So that means we're going to
4:24:54go and build like one fact table for
4:24:57each step. With that we're going to get
4:24:59like five fact tables. We have already
4:25:01one for the orders but for the rest we
4:25:02have to go and build them. Well, this
4:25:04sounds smart and reasonable things to
4:25:06do. But now, if you have like a deeper
4:25:08look to each of those steps, you're
4:25:09going to find that we are talking about
4:25:11the same money over and over. So, the
4:25:14same number should be in the orders and
4:25:16as well in the invoice and I hope in the
4:25:18payments. Now, of course, the big
4:25:19question is, do the business users
4:25:21really want all those numbers in our
4:25:23reports? Are there many use cases for
4:25:25all those numbers? Well, I'm going to
4:25:27say no. Usually the business user is
4:25:29going to ask us okay find me the total
4:25:31number of sales for the whole business
4:25:33or maybe by the region or by the
4:25:35products and for this my friends we have
4:25:37one single point of truth for that it
4:25:39going to be the orders so I'm going to
4:25:41go and grab the number the money from
4:25:44the order so I would not go and start
4:25:45like showing all the numbers from each
4:25:47steps because at the end it is the same
4:25:49money so that means we will not get like
4:25:51a lot of use cases for those numbers
4:25:53that we're going to have inside those
4:25:55facts but actually by looking to this
4:25:57we're going to get better questions from
4:25:58the business. They're going to ask
4:26:00questions about the whole process
4:26:01itself. So they're going to start asking
4:26:03questions actually not about the numbers
4:26:05but about the dates. So those
4:26:08informations are way more interesting
4:26:10than the numbers like for example how
4:26:12long it takes from an order to delivery
4:26:14or how long it takes from an order to
4:26:17the payment. So there will be a lot of
4:26:18questions how long from the delivery to
4:26:20invoice or invoice payment. So that
4:26:23means usually the business and
4:26:24especially the process expert they going
4:26:26to ask a lot of questions about the
4:26:28dates. So that means what is actually
4:26:30interesting from all those tables are
4:26:32actually the dates for the business
4:26:34intelligence. And now what we could do
4:26:36instead of having like a fact for each
4:26:38step, we're going to go and make like
4:26:40one fact table that's going to put side
4:26:43by side all those dates inside this fact
4:26:46table. And once we have everything in
4:26:47one fact table, then we can answer many
4:26:50questions about the process. And of
4:26:52course I'm not inventing here something
4:26:54in business intelligence in data
4:26:55moduling. This has a famous name. This
4:26:58is my friend a type of fact table called
4:27:00accumulating snapshot. So each row going
4:27:02to represent like one event one order
4:27:05that going to hold all the milestone
4:27:07dates of the steps in our process. So
4:27:10this is the accumulating snapshot fact
4:27:12that we're going to go now and build.
4:27:13Okay. So now since we have now special
4:27:15type of fact, it's going to be as well
4:27:17special how to build it. It will not be
4:27:19like the others. Now since we're going
4:27:21to have like the whole process in one
4:27:23row, a lot of tables going to be
4:27:25involved in building this fact. So this
4:27:27going to be something special. Now how
4:27:29to build it? We usually start with the
4:27:30spine and usually it is the first step
4:27:33in the process. It is the order. So that
4:27:36means my friends this orders that we
4:27:38build at the start where we combine
4:27:40things together. This going to be the
4:27:41starting point for our new facts. That
4:27:43means I'm going to go and create a
4:27:45reference out of it. Then we're going to
4:27:47go and drop it in the facts. So we can
4:27:50go and call it for example fact order
4:27:53process or fulfillment whatever. All
4:27:56right. So with this we have now good
4:27:57starting point. This is the spine of our
4:28:00fact. Each row is one order. And now the
4:28:02question is which columns going to stay
4:28:04and which going to leave. Now again what
4:28:05is our purpose? We want to show for each
4:28:07order for each customer all the steps
4:28:10dates for the whole process. So that
4:28:12means I don't actually need a lot of
4:28:14things over here. I just need the order
4:28:16dates. So that means I'm going to go and
4:28:18remove all those stuff. Same thing for
4:28:20the junks over here. So let's just and
4:28:22go and remove. Actually that's all what
4:28:24I need. So order ID, customer name, and
4:28:26order date. This is the first step in
4:28:29the process. Now of course instead of
4:28:30having the customer name in order to
4:28:32connect it to our data model, we're
4:28:34going to go and make a lookup. So we're
4:28:36going to go and join it with the
4:28:37dimension customer. Customer name.
4:28:39Customer name. And then we're going to
4:28:41go and get our key.
4:28:44Perfect. go to the left side and remove
4:28:47the customer name. Perfect. Now we have
4:28:49the first step. Now we have to go and
4:28:51get the other steps of our process. What
4:28:53is the next one? We have the shipments.
4:28:56So we have the shipment ID, the order
4:28:58ID, the mode, the date and the delivery
4:29:00date. So actually we have here two dates
4:29:02of the process which is exactly what do
4:29:04we need. Now about the mode, we can
4:29:06include it or not. But for now I will
4:29:08not include those stuff inside our fact.
4:29:10So I'm going to go back to that order
4:29:11process and then merge the shipment with
4:29:14our table. So it is somewhere over here
4:29:17shipments. We join by the order ID and
4:29:20hit okay. Now we have to go and expand.
4:29:23We don't need the ids. We have it
4:29:24already. We don't need the mode. I just
4:29:26need the dates. Perfect. Look at this.
4:29:28We have the shipment date and the
4:29:30delivery. Now what is the next step? We
4:29:32need actually the invoice. So let's have
4:29:34a look to the invoices. So the invoice
4:29:36ID again the order ID. This is our key.
4:29:38We have again the customer name. We
4:29:40don't need it. We have it already. We
4:29:41just need the invoice dates. The amount
4:29:43is going to be like the total sales. So
4:29:45I'm just collecting the dates. Let's go
4:29:47back to our process and same thing.
4:29:49We're going to go and merge invoices.
4:29:51Order ID. Order ID. Okay. Then we have
4:29:53to expand. So just give me the date. So
4:29:56now we are getting closer. Now the last
4:29:58step in our process going to be the
4:30:00payment, right? So over here, well, we
4:30:02don't have the order ID. We have an
4:30:04invoice ID. So we're going to join using
4:30:06the invoice ID. We have it. We don't
4:30:07need the customer name, just give me the
4:30:09date. So let's go back to the process.
4:30:12Now, of course, I need the invoice ID.
4:30:14That's why I'm going to go some steps
4:30:15back in order to expand correctly. So I
4:30:18need the invoice and as well the dates
4:30:21otherwise I cannot join it. So actually
4:30:23now we have the ID and now we can do the
4:30:25merge. So we need the payments over
4:30:28here. This time the invoice ID with the
4:30:30invoice ID. Let's hit okay. Then the
4:30:33last expansion we need the payment date.
4:30:36Now I can go and get rid of the invoice
4:30:39ID. I don't need it anymore. So remove.
4:30:41So now do we have everything for our
4:30:43process? We have the order, the
4:30:44shipment, delivery, invoice, and
4:30:46payment. And this looks exactly like our
4:30:48business process. So this is my friends
4:30:50the fact that called accumulating
4:30:52snapshots for one event. You're going to
4:30:54get all the dates of the process. Of
4:30:56course, I'm doing things very quickly. I
4:30:58usually do and test a lot. Each merge I
4:31:00check whether we have duplicates or the
4:31:02numbers are correct. But now in order to
4:31:04make things a little bit quicker, I'm
4:31:05just merging like crazy. So this looks
4:31:08really good. Now what we're going to do
4:31:09actually with the other tables, I'm
4:31:11going to say we're going to go and
4:31:12deactivate them because we have now the
4:31:15most important aspect of all those
4:31:16tables. And of course if in the future
4:31:18we got like more specific things and
4:31:20more specific questions about for
4:31:22example the invoices, then we might go
4:31:24activate it again and build like special
4:31:26purpose fact for it. But actually we got
4:31:28the best of this table. So we connected
4:31:30is with the order we have already the
4:31:32customers the invoice dates and the
4:31:34amount is not that important we have it
4:31:36in the fact sales. So that's why my
4:31:38friends I'm going to go and deactivate
4:31:39it from the model. Same things for the
4:31:42others. So let's go to the payments. The
4:31:44same money is just repeating over and
4:31:46over. So deactivate. Same thing for the
4:31:49shipment over here. Well we can go and
4:31:51include the mode if it's really that
4:31:53important. But again if we get like some
4:31:55specific use case for the shipment we
4:31:57can go and activate it and add this
4:31:59column. This will not be now necessary
4:32:01for our data model. So we are
4:32:02deactivating a lot of things. Now by
4:32:05looking to this we still have the
4:32:06invoice lines. This is exactly the same
4:32:08setup like we had at the start. The
4:32:10invoice line is the details of the
4:32:12invoice and the invoices is the header.
4:32:14Again if there is like special needs,
4:32:16special use case in order to analyze the
4:32:18invoices. We can go and always build a
4:32:20fact tables out of those stuff. But for
4:32:22now I don't see it as a core thing
4:32:24inside our data model because again the
4:32:26same money is repeating. So I'm going to
4:32:28go and deactivate this as well. All
4:32:30right. Now we have deactivated almost
4:32:32like everything. So let's go and save
4:32:34and see what do we have and look to our
4:32:36data model. So this should get really
4:32:38empty.
4:32:41Oh wow. Look, we still have like four
4:32:45tables out of the chaos. And of course
4:32:48everything went to the new data model.
4:32:50Now of course you might think now I'm
4:32:52trying to rush up the project in order
4:32:54to close things. But to be honest, this
4:32:56is exactly the way how I work. I don't
4:32:58go and push everything inside the data
4:33:00model in order to have everything just
4:33:02in case someone ask for something. I try
4:33:05to push really hard on the data model
4:33:07only to leave what is necessary. And I
4:33:09can tell you after working in many
4:33:11projects, this exactly the process. I
4:33:13understand the business and try to bring
4:33:15the core facts and the core dimensions
4:33:17to the data model. And if there is like
4:33:19one or two customers are asking very
4:33:21specific questions about like some weird
4:33:24tables, I would not go and destroy the
4:33:26data model to be honest. I'm going to go
4:33:27and build another PowerBI model just for
4:33:30their purposes. So it's going to be
4:33:31something quick just to make their
4:33:33analysis and that's it and try to
4:33:35protect the core data model from those
4:33:38edge use cases. So this is my style on
4:33:40how I do data modeling. But there are a
4:33:42lot of people that go and include
4:33:44everything inside the data model just in
4:33:46case. Well, if you do it like this, then
4:33:47you are including a lot of things that
4:33:49will never be used in reporting and this
4:33:52is really waste of space performance and
4:33:54as well confused a lot in reporting. All
4:33:56right, so now back to our data model.
4:33:58Now we have like a new things. So we
4:34:00have to be as well careful that the new
4:34:02core data model is not getting like full
4:34:04of chaos. So we have the order process.
4:34:07This is the accumulating snapshot type
4:34:09and yes it is connected with the
4:34:11customers and that's it. So yes, we are
4:34:13getting a lot of facts but we are making
4:34:14sure none of the facts are connecting to
4:34:17each others. They always go through a
4:34:19shared dimension. Our shared dimension
4:34:21now is the customers and the products
4:34:23and always the relationships are
4:34:25healthy. So we have here one to many
4:34:27relationship and the direction is from
4:34:29the dimension to the facts. So this is
4:34:31not chaotic. This is very healthy. Okay.
4:34:33So now we are getting very close to end
4:34:36the migration. Look at this sneaky table
4:34:38at the ends. Let's go ahead and put it
4:34:39here. So what do we have here? We have
4:34:41five tables yet to go. Now let's go to
4:34:43power query and start like investigating
4:34:45those stuff. Now I'm going to start with
4:34:46the very obvious one. Remember the sheet
4:34:49one. It is a complete duplicate of the
4:34:51shipment. So they are like identical
4:34:53tables. Think this is an import problem
4:34:55that happens maybe from the source
4:34:56system or in the Excel. Of course
4:34:58happens a lot in real projects where you
4:35:00import stuff that are actually not
4:35:02necessary inside the model. So what you
4:35:04going to do? We're going to go and just
4:35:05delete it. Not enable. So we remove it
4:35:08completely physically from our model. It
4:35:10is not something that we're going to
4:35:12need later. Okay, so let's keep going.
4:35:13We have the exchange rate. Well, I'm
4:35:15thinking about this. We don't have like
4:35:17any relevance for this inside our data
4:35:19model. It is something transactional
4:35:20maybe important for the interfaces in
4:35:22the source system. But for reporting,
4:35:24why do we need those informations? And
4:35:26actually, I cannot go and connect it to
4:35:27our data model. That's why I'm going to
4:35:29say this is a standalone table that
4:35:32actually will not help us at all. So,
4:35:34I'm going to go actually and deactivate
4:35:35it. I will not like physically remove
4:35:37it. Maybe there is like use case for it.
4:35:39We cannot connect this anywhere. So
4:35:41deactivate. Now what else we have here?
4:35:43The sales target. So here we have the
4:35:45target revenue and the period. So
4:35:47someone is like actually planning the
4:35:49target for each month. Well, this is
4:35:51like mini transactional table. This is
4:35:53not a dimension. That's why I'm going to
4:35:55say let's go and make small fact out of
4:35:57this. It will not hurt. So we're going
4:35:59to go and bring it to the facts. Rename
4:36:01it of course. So rename fact. I'm going
4:36:04to stay call it the sales targets. And
4:36:06what else we could do? We just go and
4:36:08maybe rename things to follow our
4:36:09standards. So the date and the target
4:36:12revenue. So actually that's it. It is a
4:36:14mini one that don't have really any like
4:36:16IDs to customers or dimensions. Only has
4:36:19a date and a measure. So that's it.
4:36:21Let's go and deactivate this one.
4:36:22Disable load. Now what else do we have?
4:36:25We have the security. And as well we
4:36:27have a dimension order. This is of
4:36:30course completely garbage table. So here
4:36:32we have just like a list of all orders
4:36:34and this makes no sense. We have already
4:36:36better tables for this. So I'm going to
4:36:38say I'm going to go and remove this
4:36:40completely physically from our data
4:36:41model. So delete we will not deactivate.
4:36:44Perfect. Now the last thing that is left
4:36:46is the security. Of course my friends we
4:36:48cannot leave the data model as it is. We
4:36:51have to make it secure following the
4:36:52standards. And here we have a very
4:36:54helpful table to let us know which user
4:36:56is allowed to see which region. But of
4:36:58course we're going to do this in the
4:36:59last phase. So now what I'm going to say
4:37:01actually we have covered everything. So
4:37:03let's go and close and apply and have a
4:37:05look. Yes, it is empty. I know we have
4:37:07the security. We can do this later. I'm
4:37:09going to bring it near to our data model
4:37:11here. And my friends, I can say the
4:37:13nightmare data set just got cleaned up.
4:37:16So, we have here a healthy star schema.
4:37:19Just enjoy this view. Have a look to
4:37:21this. We have a healthy data model. I
4:37:24think my camera gave up. So, back now.
4:37:26Look at this my friends. We don't have
4:37:28anything anymore from the old spider
4:37:30web, the nightmare data set. Everything
4:37:32is migrated to a clean star schema of
4:37:35course a galaxy one. So everything is
4:37:37connected using the best practices and
4:37:40we have here only one fact table that
4:37:42stands alone and the security table. So
4:37:44just sit back and look at this. This is
4:37:46really satisfying. But of course we are
4:37:47not done yet. We still have the [music]
4:37:49last phase.
4:37:54All right friends. Now the next phase
4:37:55it's all about polishing the data model.
4:37:57So we're going to finalize our data
4:37:59model. We're going to make it
4:38:00trustworthy before handing it over for
4:38:02the next phases. So, we're going to do
4:38:03things like for example, we're going to
4:38:04go and check again our standards, our
4:38:06rules and check whether all the tables
4:38:08and the columns are following it like
4:38:10the snake case, like the prefixes,
4:38:12suffixes, the keys and so on. The second
4:38:14step that I usually do is that I go and
4:38:16create the dimension date. It is a
4:38:18special dimension that actually going to
4:38:20be needed almost in all our facts. So,
4:38:23we have to create it and connect it in
4:38:24the data model. This is one of the
4:38:26things that I do at and after that we're
4:38:28going to build like a collection of
4:38:30measures. So we're going to make like
4:38:31set of very obvious measures that's
4:38:33going to be needed on top of this model.
4:38:35And after that we're going to make sure
4:38:36that our data model is secure. That
4:38:38means we're going to apply on this model
4:38:40the role level security is like almost
4:38:42standards in any project. And after that
4:38:44we do the final testing. So we go to the
4:38:47report level and we go and check only
4:38:48the big numbers and whether the
4:38:50dimensions are working correctly to make
4:38:52sure that everything is correct before
4:38:54handing over this data model for the
4:38:56others. Usually the last step is I go
4:38:58and make a meeting and talk to the
4:38:59others, show them the start schema and
4:39:01make a guide on how to use it for
4:39:03reporting. All right, my friends. Now
4:39:05let's deep dive in this phase.
4:39:06>> Okay, let's do it.
4:39:08>> Okay, so now let's start with the first
4:39:09step where we're going to go and check
4:39:10whether our data model is following the
4:39:12standards and rules. I'm pretty sure
4:39:14that I have forgotten something during
4:39:16the building phase. We are at end human
4:39:18and we might forget things. So now how I
4:39:20usually do it is that I go to the table
4:39:21view over here and have a quick check on
4:39:23the right side. So I can see that
4:39:25everything is sorted correctly using the
4:39:27prefix. So we have the dimensions and
4:39:29the facts and the last one is the
4:39:31security. So by looking to this I can
4:39:32say all the names of our tables are
4:39:34following the standards. Then we go just
4:39:36quickly through all the tables one by
4:39:38one just to have a quick look. Well
4:39:40everything looks so far fine. Well,
4:39:43seems that I have followed all the
4:39:45rules. Well, not everywhere. [laughter]
4:39:48So, I have to fix here the naming
4:39:49conventions for this fact table. So, we
4:39:52have to do it quickly to go to the power
4:39:54query. Okay. The fact order. So,
4:39:57customer ID, order dates,
4:40:02ship date.
4:40:04Okay, looks good. So, I'm going to go
4:40:06and close and apply. All right, so we
4:40:08are following the standards over here.
4:40:10Let's go and keep going. Looks good as
4:40:13well. Good. The security going to be at
4:40:14the end. Okay. So that means we are
4:40:16following all the rules for the table
4:40:18names and as well for the columns. Now
4:40:20there is another thing that I do that I
4:40:22go and check the date format. Well, you
4:40:24can see here that in our dates we are
4:40:26using the full name, the full month.
4:40:28This looks really weird in reporting. If
4:40:30you have a lot of dates, it's going to
4:40:31take a lot of space in the report level.
4:40:33So what I usually do is that I decide on
4:40:35one like smaller format and I stick to
4:40:37it for the whole data model. So now what
4:40:39I'm going to do, I'm going to go and
4:40:41first decide on the formats. So if you
4:40:42go over here to the column tools, you
4:40:44can see that we have format for the
4:40:46dates and in PowerBI we have a lot of
4:40:48formats. I'm going to say the standards
4:40:49is going to be this one. So the year,
4:40:51then the month and day depend of course
4:40:54on the region. But this one is compacted
4:40:56and small. Look at this. It is not
4:40:57taking so much space. So that means this
4:41:00is going to be the standards on how to
4:41:02show the format inside the data model.
4:41:04Now we're going to do very annoying task
4:41:06where I'm going to go through the whole
4:41:08data model and make sure that all the
4:41:09dates are using this format. Now of
4:41:11course you can pause the video and do
4:41:13it. I'm going to now fast forward just
4:41:14to make sure this video don't explode.
4:41:16So I'm going to do this pretty fast. So
4:41:19let's go.
4:41:29So that was the last fact in our data
4:41:31model and with that we have now the same
4:41:32format for the whole data model. Now
4:41:34there is another thing that I usually do
4:41:36in this phase as well since we are now
4:41:37going through all tables. When PowerBI
4:41:39sees a column as a number it going to
4:41:41make it automatically like aggregatable.
4:41:43So that means if you use it inside your
4:41:45report PowerBI going to automatically
4:41:47like summarize all the values. This is
4:41:49of course nice thing but sometimes we
4:41:51have numbers where it makes no sense to
4:41:53summarize. So it's really nice if you
4:41:55clean this in the data model before
4:41:57going to the next phase so that people
4:41:59don't make any mistakes. So now what
4:42:00we're going to do, we're going to go
4:42:01through all the tables one by one and
4:42:03always keep your eye on this icon. That
4:42:05means this column could be aggregated.
4:42:08For example, here the budget can see
4:42:09here the summarizations it is sum. If
4:42:11you don't want it to get summarized,
4:42:13you're going to go over here and say
4:42:14don't summarize. So now you have to make
4:42:16decisions almost for all your numbers.
4:42:18I'm going to say for the budget it makes
4:42:19sense of course to make maybe an
4:42:21average. So I'm going to leave it as it
4:42:23is. For the customers, well I'm going to
4:42:25say maybe makes sense to make an average
4:42:27out of this. So this is okay. So let's
4:42:29go through all the dimensions. Same
4:42:31things for the products. So that was the
4:42:33last table, the last column and with
4:42:35that all dates inside my data model is
4:42:37following the same format. All right. So
4:42:39that's what the first step. We made sure
4:42:40that we are following the rules and the
4:42:42standards on the format and everything.
4:42:44Now we're going to go to the next step
4:42:46that I usually do with that. I go and
4:42:47create the dimension date. There are
4:42:50many different ways on how to create
4:42:51this. I usually do that in the back end.
4:42:54So if I have a database or data bricks,
4:42:56I go and create the dimension there. But
4:42:58since now we are just doing PowerBI, I'm
4:43:00going to show you a way on how to create
4:43:02it directly in your data model. Uh there
4:43:04is a really nice way. I'm going to say
4:43:05very smart way actually from PowerBI.
4:43:07Now I will not go and give you like a
4:43:09CSV file or text with all dates inside
4:43:12it and say hey this is your dimension
4:43:14but this time we're going to go and
4:43:15create it by getting inspired from the
4:43:17data model. So how we going to do it?
4:43:19You're going to go over here to the home
4:43:21and here in the ribon we have something
4:43:22called new table. So what we are doing
4:43:24here we are like creating a new table
4:43:27inside the data model and we're going to
4:43:29go and call it since this is the
4:43:31dimension dimension dates. Now there is
4:43:33a very cool function in PowerPI called
4:43:35calendar auto. So read this. It returns
4:43:38a table with one column of dates
4:43:40calculated from the model automatically
4:43:42and this exactly what it does. It go and
4:43:44scan the whole data model understand the
4:43:47range of our business. So understand
4:43:49what is the earliest date and what is
4:43:51the latest date and just create one
4:43:53column based on that. Very very cool. So
4:43:56actually that's it. We don't have to
4:43:57give it any arguments and hit enter.
4:43:59Amazing things. So look at this. The
4:44:01earliest date is the 1st of January 2025
4:44:04and the latest date is the end of 2026.
4:44:07And of course this my friends going to
4:44:09expand automatically as the business
4:44:11grow. So that means if the data sets
4:44:13like grow over the time and you have
4:44:142027, you don't have to deal with the
4:44:17dimension date because it going to grow
4:44:19automatically. So suddenly you're going
4:44:20to have as well in the dimension date
4:44:222027. I can tell you this is magic
4:44:24really. In my old days as I done
4:44:26business intelligence with other tools
4:44:28that was one of the biggest struggles
4:44:30where I had like a manual table and I
4:44:32have to always add a new years and stuff
4:44:34like that. And I can tell you many
4:44:36people forget to update their dimension
4:44:38dates and of course this cause a lot of
4:44:40problems in the report because almost on
4:44:42each report you are doing like analyzes
4:44:44over the time and you will be using this
4:44:46dimension. So this is one of the most
4:44:48important dimension compared to the
4:44:50others and if you don't make sure it is
4:44:52always up to date it's going to be
4:44:54catastrophic. But now we don't have to
4:44:55deal with all this chaos. PowerBI did
4:44:57solve this and everything now is
4:44:59automatic. Just some trauma from my old
4:45:01days. All right. So now what do we have?
4:45:02We have only like one column right now.
4:45:04What you can do, you can go of course
4:45:06and change the format to our format. So
4:45:08we need only the date. We don't need the
4:45:10hours and time. And of course the next
4:45:12step if you want to go and expand this
4:45:14dimension to add more columns like for
4:45:16example the year, the month, the day,
4:45:18the full name of the day, the full name
4:45:20of the month. So you can add a lot of
4:45:22things to this dimension by just going
4:45:24and adding like a new column. Like for
4:45:26example, if I would like to get only the
4:45:28year, I'm going to do something like
4:45:29this year equal and then use the
4:45:31function year and specify here the
4:45:33column. So this going to be our
4:45:35dimension date and that's it. So with
4:45:38that we have the column called year and
4:45:40I can do the same thing. So I'm going to
4:45:42go and duplicate it. So that we have the
4:45:44years. We can go and create another one
4:45:46for the month. Same story. So month
4:45:48equal the function month in order to
4:45:50extract this part our column. So my
4:45:52friends, we have a lot of date functions
4:45:54that we could add in order to expand
4:45:56this dimension, but I'm going to leave
4:45:57it like this. Now, of course, the big
4:45:59question is how we going to connect this
4:46:01new dimension to our data model. So we
4:46:03have it over here and there are like
4:46:05many places where we could connect this.
4:46:07Almost all our facts has dates right
4:46:10now. The thing is we have a lot of fact
4:46:12tables inside our data model that have a
4:46:14lot of date fields and this dimension
4:46:17date can play a very important role in
4:46:19order to make the bridge between all
4:46:21those facts. This going to be shared
4:46:23dimension that going to help us in order
4:46:25to compare all those numbers from
4:46:27different facts. So this is going to be
4:46:28our biggest shared dimension in the
4:46:31whole model. So how we do it? It's very
4:46:33simple. We're going to go through the
4:46:34facts one by one and start connecting it
4:46:37to our dimension. So here we have the
4:46:39fact sales target. We grab the date and
4:46:41connect it to our dimension date. Okay.
4:46:44Same thing. We go to the next one. We
4:46:46have as well date. So we connected our
4:46:48dimension. All right. So now we have
4:46:49like a shared dimension between those
4:46:51two. And we can go and compare now the
4:46:53numbers between those two facts. Now we
4:46:55have here this factless fact. It is a
4:46:57little bit weird. We don't have a date.
4:46:58So that's why we will not connect
4:46:59anything here. Let's go to the next one.
4:47:01Here we have a month as well. We can go
4:47:03and connect it to our dimension. Let's
4:47:05go. And of course the most important one
4:47:07over here we have the fact sales. So we
4:47:10have the order date. We go and connect
4:47:12it and say save. And the last fact table
4:47:15here we have a lot of dates. So for
4:47:17example I'm going to go and grab the
4:47:18order date to the dimension. So now look
4:47:21at this. Our dimension date is actually
4:47:23a big shared dimension between a lot of
4:47:25facts. Now this is amazing because now
4:47:28we can do a lot of analyzes side by side
4:47:30from all those facts. Let me just show
4:47:32you an example. Let's go to the report
4:47:34view. Don't know how we're going to do
4:47:35it now. Very important as usual in order
4:47:37to build the visual we start first from
4:47:40the dimension. So the dimension date
4:47:42this is going to build the axis for our
4:47:44visual. And then after that we start
4:47:46picking the measures from the different
4:47:47facts like for example the fact sales.
4:47:50We can go and now grab the total sales
4:47:52and put it in the view. Perfect. Now we
4:47:54can go to another one. For example the
4:47:56fact sales target. This one going to be
4:47:58on the month level. So the visual going
4:48:00to looks weird but we can fix it if we
4:48:02just remove the day from our date so
4:48:04that everything is on the month level.
4:48:07Now we can go to a third fact like for
4:48:09example the inventories. Let's go and
4:48:11drop the stocks or the units. This is
4:48:14exactly my friends the power of the
4:48:15dimension dates. We can go now and visit
4:48:18all the facts and compare things side by
4:48:20side very easily without losing the
4:48:22numbers. Like you can see here the total
4:48:24sales is always protected and you can do
4:48:27now a lot of correlations and
4:48:29comparison. So this is the main job of
4:48:31the dimension date inside our data
4:48:33model. Now of course what else we can do
4:48:35with the dimension date? We can use it
4:48:36in order to do roleplaying dimension.
4:48:39Like for example we have here a huge
4:48:40fact table with a lot of dates. Right?
4:48:43So we have the order date, the invoice
4:48:45date, the pay date. Now we can go and
4:48:47connect our new dimension multiple times
4:48:49to this fact. Let me show you how to do
4:48:51it. So let me just bring this fact here.
4:48:53And now we just grab the date to the
4:48:56different columns. Like for example here
4:48:58the pay date of course this going to be
4:49:00inactive. Then one more time for the
4:49:02ship date and another one for the
4:49:04invoice date. And I think the last one
4:49:06going to be the delivery date. So look
4:49:08at this. Our dimension date is playing
4:49:10five different roles for this fact. But
4:49:12of course you're going to do this only
4:49:14if you really need it. Otherwise this
4:49:16going to looks really ugly. So don't go
4:49:18randomly like connecting the date for
4:49:20every column. Sometimes we don't really
4:49:22need it like for example if I'm just
4:49:24like analyzing the sales I don't really
4:49:27always to go and get the dimension date
4:49:29so actually I can go and build the
4:49:31visual completely based on the fact
4:49:33sales I have there the order date I have
4:49:35everything so if your analyzis is based
4:49:37only on one fact table then we don't
4:49:39really need the dimension date but only
4:49:41if you are like comparing multiple fact
4:49:44then we go to the dimension of course
4:49:46and with that my friends means we don't
4:49:48have to go crazy and connect the
4:49:49dimension date every single single date
4:49:51column that we have in the data model
4:49:53only if it's needed and if you create
4:49:55something like this then you have to go
4:49:56and start like building DAX calculations
4:49:59for each inactive relationship so that
4:50:01you can use it actually in your reports
4:50:03but if it is not needed then don't go
4:50:06and create all those relations now our
4:50:08dimension date looks crazy and this is
4:50:10usually how this dimension looks like at
4:50:13the end so my friends this is the topic
4:50:14of the dimension date I usually do it at
4:50:16the end just to have a big picture first
4:50:18about all my fact tables and only then
4:50:21at the end I go and connect the
4:50:23dimension date. Okay. So now we come to
4:50:24very important step in data moduling
4:50:26that a lot of people actually skip at
4:50:28this phase. We have to go and create the
4:50:29measures. Now the thing is yes we have a
4:50:31good star schema we have good structure
4:50:34everything is connected correctly the
4:50:35filter direction and all those stuff.
4:50:37But now if you go and hand this over to
4:50:39your project team and everyone start
4:50:41like building on top of it reports and
4:50:43visuals. Now what could happen is that
4:50:45multiple people going to go and start
4:50:46creating multiple versions of
4:50:48calculations that actually are
4:50:50calculating the same thing over and over
4:50:52and nobody going to know which one is
4:50:54the right one for the data model. Now
4:50:56another thing that might happen is that
4:50:58people that are working with the data
4:50:59model they might not yet develop like
4:51:02understanding of the grains of your
4:51:04facts. So for example they might go to
4:51:05the fact sales and say you know what I
4:51:07have to go and calculate the total
4:51:09number of orders. So they go and create
4:51:10calculation where they're going to count
4:51:12the order ids which is totally wrong.
4:51:14The green there is the line of order. So
4:51:17in order to do it correctly they have to
4:51:19do count distinct on the order ID. That
4:51:21means this going to lead to wrong number
4:51:23even if the data model is perfect. So
4:51:25that means in this phase our job is not
4:51:27only to structure and build a perfect
4:51:29star schema but as well we have to go
4:51:31and prepare the measures side by side
4:51:34with our data model just to make sure
4:51:36that we reduce the confusion. And of
4:51:38course I'm not saying here we have to go
4:51:39and now create all the measures for
4:51:42everything that we have in the data
4:51:43model. We just have to go and create the
4:51:45core measurements but think that we are
4:51:47sure it's going to be reused over and
4:51:49over in many reports. And by the way not
4:51:51everyone is working in teams. If you are
4:51:53solo data analyst this is as well good
4:51:55for you. I'm pretty sure over the time
4:51:56you're going to have like multiple
4:51:57PowerBI projects and you're going to
4:51:59forget about the grains and the data
4:52:00model. So you protect yourself making
4:52:03different versions of calculations and
4:52:05as well making wrong calculations. So my
4:52:07friends this is exactly what we have to
4:52:08do now. We have to go and create a
4:52:10bundle of measures. All right. So now
4:52:12how we going to do it? There's like two
4:52:13ways. Either you're going to go to each
4:52:15fact and inside them you're going to
4:52:16start create the individual measures or
4:52:19there is like another way that I like to
4:52:20do and many others as well is to go and
4:52:23bundle let's say the major measurements
4:52:25inside one fake table. Let me show you
4:52:28how to do it. So we're going to go over
4:52:29here and say okay new table. Now I'm
4:52:31going to call it underscore in order to
4:52:33have it at the top. Then measures equal
4:52:36nothing. With this we got like an empty
4:52:38table here on the top. And now inside it
4:52:40we're going to go and start creating all
4:52:42the measures of the data model. So
4:52:44that's during creating the reports
4:52:45you're going to have it all in one place
4:52:47and it's going to be easy to go and drag
4:52:48and drop stuff. And if you want to
4:52:50update your measures, you go directly
4:52:51here and start updating. So now this is
4:52:53our new measure that we have. Of course
4:52:55we will not connect it to anything. I'm
4:52:57just going to go and move it here side
4:52:58by side with the security so that we can
4:53:01see okay this is my star schema and
4:53:02those are the let's say the add-ons on
4:53:04top of it. Now let's go to the maybe
4:53:07report view and create a new tab. I'll
4:53:09just start now and create like few major
4:53:11measures inside our data model. So let's
4:53:14start with the fact sales like for
4:53:15example here it's very obvious that's
4:53:17one of the major measures it is the
4:53:19total sales right. So how we going to do
4:53:21it? We're going to go to the measures
4:53:22over here and right click on it and say
4:53:24new measure. Then we just create like
4:53:26very simple DAX calculations. So for
4:53:27example total sales equal to the sum of
4:53:31the fact table sales and now you pick
4:53:34the column. So line total and actually
4:53:36that's it. Now what we do actually after
4:53:39that we decide like on the format of
4:53:41this measure like for example I'm going
4:53:42to have the thousand like separator.
4:53:44Actually I don't want it as a decimal
4:53:46number. So I just want it always to be a
4:53:48whole number so that everything is
4:53:50formatted and is ready in order to be
4:53:52used in my reports. So now by just drag
4:53:54and drop from this measure I'm going to
4:53:55get the total sales without worrying
4:53:58about how this is calculated. So that
4:54:00was an obvious thing. Then we have to
4:54:02keep thinking okay what could we
4:54:03measure? Well very important the total
4:54:06number of orders. So we're going to go
4:54:07and create a new measure total orders
4:54:10equal. Now we want to count how many
4:54:12orders that we have in the business. So
4:54:14we're going to say okay count but again
4:54:16since the grain is line of total we're
4:54:17going to go and use the distinct count
4:54:20and we're going to say okay fact sales
4:54:22the order ID. So actually that's it. Now
4:54:24of course we can think about how we're
4:54:26going to format it. So it's going to be
4:54:27as well the thousand's going to be
4:54:29separated and it is a whole number which
4:54:31is correct. So if I go drag and drop
4:54:33same thing I got that total number of
4:54:35orders which is correct. So as you can
4:54:37see this is the whole job. We're going
4:54:38to go through all the facts one by one
4:54:39and we start creating those measures.
4:54:42But we have just to focus on the main
4:54:43things that we are sure that's going to
4:54:45be used in many reports. Like for
4:54:47example, we can go and count the total
4:54:49number of active customers. So there is
4:54:51like a difference between all the
4:54:52customers that we have inside the data
4:54:54model. For that we can go and count it
4:54:56from the dimension customer. But if you
4:54:58want to know how many customers are
4:55:00actually ordering things, then you have
4:55:02to go to the fact sales. So we can go
4:55:04over here and say okay new measure total
4:55:06active customers equal as well distinct
4:55:10count because we're going to go to the
4:55:12fact table. So fact sales and here we
4:55:15need the customer ID. So this is the
4:55:18count from the fact. Now we can go and
4:55:20create another one that is actually
4:55:22counting from the dimension. So base
4:55:25maybe total customers equal. This time
4:55:28we can just go and use the count for the
4:55:30dimension because we don't have
4:55:31duplicates. dimension customer and here
4:55:34we pick as well the ID over here same
4:55:36thing if you have a lot of customers you
4:55:38can go and start like separating things
4:55:39with a th00and now what else we can do
4:55:42we can go to another fact like for
4:55:43example here we have the fact order
4:55:45process we can go and calculate the
4:55:47average number of days from the first
4:55:49step to the last step so now how we
4:55:51going to do this measure first we have
4:55:53to go and create maybe a column let me
4:55:55just go back over here to the table view
4:55:57so look at this we have all the steps
4:55:58and I would like to go and calculate how
4:56:00many days that we have between in the
4:56:02first step the order date and the last
4:56:04step the pay date and with that we're
4:56:05going to get like a new measure for this
4:56:07fact and in top of it we're going to go
4:56:09and create a new calculation. So let's
4:56:11go and do that. First of all we need a
4:56:12new column because we need calculation
4:56:14on the row level. So for example this
4:56:17going to be order to pay. Now we just
4:56:19want to find the differences between two
4:56:21dates. So there is a function called
4:56:23date diff. The first date is the order
4:56:25date. The second one is the last one. So
4:56:27it's going to be the pay date. And what
4:56:29is the difference? We are calculating
4:56:30how many days not month or year. So this
4:56:33is the part that we are interesting in.
4:56:35And actually that's it. So look at this.
4:56:37Now we are getting the total number of
4:56:39days that it took from the order to the
4:56:41payment. Now the next step is very
4:56:43simple. We're going to go and create a
4:56:44new measure in our collection where
4:56:46we're going to say like for example
4:56:48average order to pay equal average. Then
4:56:52we just go and pick our new measure. So
4:56:54it is the order to pay. So look at this.
4:56:56Now we have another measure based on
4:56:58another fact. But everything now
4:57:00together. So now if you are building a
4:57:01report, all what you have to do is to
4:57:03just go and open this one and start
4:57:04browsing all the measures of this data
4:57:07model. But of course again you don't
4:57:08have to go and sit and think about each
4:57:10single detail. Just create the obvious
4:57:13one. Okay. Now I'm going to say for now
4:57:14I'm going to stick with this. This is a
4:57:16good collection of measures that we can
4:57:18hand over together with the data model.
4:57:25[music] All right friends, now we are
4:57:26getting very close. We have now the last
4:57:28step in the last phase. We have to make
4:57:30sure that our data model is secure. So
4:57:32that means we have to go and apply the
4:57:34role level security mechanism on top of
4:57:36our data model. Let's go and do that.
4:57:38Okay. So now again we have our star
4:57:40schema. We have those measures and the
4:57:43last add-on that we have to add is the
4:57:45security. Now first of all to make sure
4:57:47that the security is ready we have to go
4:57:49back to the power query and make sure
4:57:51that everything there is ready and
4:57:53clean. Okay. So this going to be our
4:57:55last table inside the stage the
4:57:57security. Now quickly by looking to this
4:57:59you can see the header informations are
4:58:01not correct. So this should be the
4:58:03columns and yes we still have to follow
4:58:05the naming convention. So I'm going to
4:58:06go and rename things. This is the user
4:58:09email and this is my friend the region.
4:58:12All right. Now the last step is that I'm
4:58:13going to go and grab it actually to the
4:58:15other supports. So with that I don't
4:58:17have anything more in the stage that I
4:58:19have to work on. Look at this. We have
4:58:20here the dimensions. We have the facts
4:58:22and the support going to be the security
4:58:24table since it doesn't belong to those
4:58:26two. All right, let's close and apply.
4:58:28All right, now look at this. It start
4:58:30connects to our data model. But I have
4:58:32to make sure that it is correct. So
4:58:34let's go and just remove it. So we
4:58:36didn't need this relationship. And about
4:58:38this one is actually correct. So now
4:58:40again let's first understand the
4:58:42security that we have here. So let's
4:58:43just talk about it. This is the
4:58:45security. Look at this. Here we have the
4:58:47user email addresses. Those are the
4:58:49employees of our business. and it looks
4:58:51like they are assigned to specific
4:58:52regions. Now, of course, in order to
4:58:54build this, we have to talk to the
4:58:56security officers of our company and
4:58:58understand how the security works. They
4:59:00might say, you know what, each employee
4:59:02in our company is allowed to see only
4:59:04specific region. Like for example, Nora
4:59:06can see only the North America. She is
4:59:08not allowed to see the sales that are
4:59:10happening in Europe or Middle East. Same
4:59:13thing for hands. Hans can see only
4:59:14Europe sales. So, we have to understand
4:59:16the security requirements before we
4:59:18start like connecting things. So we
4:59:20cannot go blindly like making a security
4:59:22concept for the data model and only
4:59:24after understanding then we can start
4:59:26connecting things together and then of
4:59:28course the next step is to think okay
4:59:30where are the region informations in
4:59:32which facts in which dimensions we can
4:59:34see that of course it is very obvious if
4:59:36you go to the dimension customers you're
4:59:38going to find over here the region
4:59:39information. So this is a very good
4:59:41starting point. Of course the question
4:59:43is how we going to connect this security
4:59:45to our data model and we usually do it
4:59:48on top of dimensions because dimensions
4:59:50anyway go and filter the fact tables and
4:59:52the security table going to be filtering
4:59:54this dimension. Now it's very obvious
4:59:56that we have the dimension customer
4:59:58where we have the region over here. So
5:00:00this is a perfect candidate where we're
5:00:01going to connect our security on top of
5:00:03this so that the user can see only
5:00:06specific customers and this will be
5:00:08filtering as well our facts. So this is
5:00:11one way on how to create the security.
5:00:13There is another way on how to do it is
5:00:15to go to the geographic dimension as
5:00:17well. Here we have regions and this one
5:00:19is connecting as well directly to our
5:00:21fact which is really as well good
5:00:23candidate that we could use in order to
5:00:25filter our data model based on the
5:00:27security. Now how we going to decide on
5:00:28this? Usually I do it like this. I go
5:00:30back to the data model and look to those
5:00:32two dimensions. So I can see here for
5:00:34example let me zoom in. This is the geo
5:00:36dimension and you can see it is
5:00:38connected to the fact sales. So now if I
5:00:40put the security on top of this it going
5:00:42to go and filter only one fact table.
5:00:45But now the other one the dimension
5:00:47customer it is actually connected to two
5:00:49facts. The fact sales and the fact order
5:00:52process. So if I connect the security on
5:00:54top of this, I'm going to benefit in
5:00:55order to secure actually two facts,
5:00:57which is of course a nice thing. So I'm
5:00:59going to go and decide actually on
5:01:01connecting the security on top of the
5:01:03dimension customers just because it
5:01:05secures more facts for me. But again
5:01:07based on the requirements we have to do
5:01:09those stuff and now I saw like some
5:01:11weird scenarios where people go and say
5:01:13you know what I want to secure the whole
5:01:15data model without any reason. So they
5:01:17start like forcing somehow the data
5:01:19model to use the security table on top
5:01:21of each facts. So they start like doing
5:01:24crazy things like by merging the facts
5:01:26just in order to get the region column
5:01:29inside each fact and they destroy the
5:01:31grain because of this just to add the
5:01:33security at the end. Maybe this is
5:01:35totally unnecessary. So always
5:01:37understand the security requirements and
5:01:39only apply it when it's needed. You
5:01:40don't have to secure every single
5:01:42dimension and fact. Like for example, is
5:01:44it really bad to show the dimension
5:01:46products for all our employees? I think
5:01:48it is no harm if they see the whole
5:01:50data. So that means for our project,
5:01:52let's say that we're going to apply the
5:01:53security on top of those two facts, the
5:01:56sales and the orders. So now how we
5:01:58going to do it? It's going to be very
5:01:58simple. We're going to go and build a
5:02:00relationship between the security and
5:02:02the dimension. So we're going to go over
5:02:03here and say, okay, region by region,
5:02:06one to many relationship, and the filter
5:02:08is single direction. We have to make
5:02:10sure it is going in the right direction,
5:02:12of course. Well, yes. Look at this. The
5:02:14filter direction of the security is
5:02:16going to the dimension and the filter
5:02:18direction of the customers is going to
5:02:20both the fact sales and the fact order
5:02:22process. So this looks perfect for
5:02:24security. Of course the next step is
5:02:26that we're going to go and start
5:02:27creating the manage rules. Otherwise
5:02:30this will not work. So it's going to be
5:02:31very simple. We're going to go and
5:02:32create a new role. Let's go and call it
5:02:34regional access. And of course in which
5:02:36table we are building the security. It's
5:02:38going to be on the dimension customer.
5:02:40and we're going to write a DAX
5:02:41calculation in order to secure it. So
5:02:43we're going to go and switch to DAX.
5:02:45Okay. So this is going to be very
5:02:46simple. We are just writing the DAX
5:02:48filter expression where we're going to
5:02:49say okay I want to show only the rows
5:02:52where the region is matching the user
5:02:54region. So that means it has to go and
5:02:56find up what is the region for our
5:02:59current user. So it is inside the
5:03:01security table the region. then it
5:03:03should match the email address that we
5:03:05have in the security with the user
5:03:07whoever is opening the PowerBI reports
5:03:10and for that we have a nice function the
5:03:12user principle name. So very simple
5:03:14thing so it is not that hard we are
5:03:16saying just check each row and ask this
5:03:19question does the region belong to the
5:03:21person that is looking at the report if
5:03:24yes then keep the row otherwise go and
5:03:26drop it. So it is just a filter
5:03:28expression that going to help us to
5:03:29filter the rows based on the region. Of
5:03:32course actually that's it. Let's go and
5:03:34save it. Very important that we are
5:03:35getting green. Now let's go and close
5:03:37it. All right. So with this we have
5:03:38applied the role of the security on top
5:03:40of our data model. But of course we
5:03:42don't trust until we test. So how we
5:03:44going to test it? I need one email like
5:03:46one of the employees. So for example
5:03:48Nora I'm going to go and copy its value.
5:03:50Then go back to the model. And now we
5:03:52can try to view the model as Nora is
5:03:54opening the report and seeing it. How to
5:03:56do it? We can go over here and say view
5:03:59as then we have to specify the rule that
5:04:01we just created. So the regional access
5:04:03and we want to see it as another user
5:04:06not as par because otherwise it's going
5:04:07to take the informations of my users but
5:04:09we want to see it as Nura. Okay. So
5:04:12let's set okay. Then you can see we're
5:04:13going to get like a warning. The
5:04:15original access is activated and we are
5:04:17viewing the object as Nora email. So now
5:04:19let's go and test. Look at this. We are
5:04:22already seeing like less details than
5:04:24before. So the total sales is not
5:04:26anymore the total number of sales of the
5:04:28whole data model and we are seeing only
5:04:30the total sales that is based on their
5:04:33Nora's region. Of course we can test
5:04:34this as well by going to the dimension
5:04:37customer and let's go and grab the
5:04:39region. So you can see we are seeing
5:04:40only the North America. So my friends
5:04:42that means the dimension is working. The
5:04:44fact table is as well working and of
5:04:46course the last test. Let's go and
5:04:48remove this effect. So if I stop
5:04:50viewing, look at this. Suddenly I'm
5:04:52going to see the total sale for
5:04:54everything and as well I'm going to see
5:04:55all the regions. Of course now you can
5:04:57go and pick another email and test that.
5:04:59But for me it is already working and we
5:05:01have applied the roll security on top of
5:05:04our data model. All right friends, oh my
5:05:06god I can say that we are done. We made
5:05:08it. Just sit back, look at this and
5:05:10enjoy it because this is awesome. And
5:05:13remember at the very beginning I showed
5:05:14you this beautiful chaos and the
5:05:16nightmare data model where we had the
5:05:18many to many relationships everywhere.
5:05:20The filters are going from one table to
5:05:22another just in all directions. No star
5:05:25schema nothing makes sense full chaos.
5:05:28But now look back to this. We have clean
5:05:30healthy star schema or let's say galaxy
5:05:32where we have fact tables and surrounded
5:05:34with dimensions. Every relationship and
5:05:37every filter is going to the right
5:05:38direction. The numbers are protected and
5:05:41everything is friendly and easy to read.
5:05:43And I can tell you this one was not an
5:05:45easy one. That was a really hard project
5:05:47and complicated. And if you stayed until
5:05:49the end and you build exactly this data
5:05:51model with me, then my friend, you
5:05:53should be proud of yourself. And I can
5:05:54say welcome to the club with this, you
5:05:56went with me through the journey exactly
5:05:58like I do in my projects in my
5:06:00companies. And I can tell you over all
5:06:02projects I just met like only few people
5:06:04that have this kind of skills where they
5:06:07can turn a chaos data model into a clean
5:06:09one because most people actually don't
5:06:11know how to do data modeling and they
5:06:13skip it. But my friends, we are not one
5:06:15of them. We are the expert that can
5:06:17build a real data model and this is
5:06:19really a rare skill that many companies
5:06:21need. So what I really recommend you is
5:06:23to go and make this project as your
5:06:25portfolio project. Of course feel free
5:06:27to use it as your project. So go and
5:06:29share it as well with the others in
5:06:30LinkedIn and you can tag me if you want.
5:06:32But what is way more important than
5:06:34this, if you are searching for job and
5:06:35you got interviewed, make sure to talk
5:06:37about this project because this is
5:06:39really rare. It going to makes you
5:06:40quickly stand out the crowd. And if you
5:06:42got hired or you are already at the
5:06:44company and you joined a new project
5:06:46where they have exactly the data model,
5:06:48the chaos that I showed you at the
5:06:49start, what's going to happen this time?
5:06:51You will not panic. You're going to look
5:06:52at it and smile because you're going to
5:06:54remember exactly this project and you
5:06:56know what to do. So now before we close
5:06:58this project I would like to go through
5:07:00all the most important points that we
5:07:02have learned during the data modeling.
5:07:04So let's recap the first point before
5:07:06you build anything always take time to
5:07:09analyze explore and understand. You have
5:07:11to understand the business the processes
5:07:13the data. The more you invest in
5:07:15exploring the data and understanding the
5:07:17business the easier things going to be
5:07:19during building the data model. Then the
5:07:21second point, always understand the
5:07:23grain of your tables, dimensions and
5:07:25facts and say it loudly what each row is
5:07:27representing. Once you know that you
5:07:29will not make mistakes as you are
5:07:31connecting tables. Third point, always
5:07:33set rules and standards before you start
5:07:35the projects and always follow them
5:07:37until the end. All the naming
5:07:38conventions, the prefixes and so on.
5:07:40Rule number four, always remove the
5:07:42unnecessary data from the data model.
5:07:45This is very important. Always ask, do
5:07:47we really need those informations for
5:07:49reporting? Is it really necessary? Never
5:07:51be the person that's going to leave
5:07:53things just in case. And the next one,
5:07:55our goal is always to build the star
5:07:57schema. All BI projects love that.
5:07:59You're just going to make your life
5:08:00easier with the calculations. And of
5:08:02course, always avoid connecting fact
5:08:04tables together. Just use a shared
5:08:05dimension in between. And the filter
5:08:07directions are always from the
5:08:09dimensions to the facts. So my friends,
5:08:11that's it. Yes, this is the whole
5:08:12project. I really enjoyed making this
5:08:14project. For a while, I thought that I'm
5:08:16working at a company. I'm not recording
5:08:18a video and I'm talking to a colleague.
5:08:20I really had a lot of fun making this
5:08:22project. I really enjoyed it. Now it's
5:08:24time for you. So you have to go and
5:08:25practice on your own. Maybe go and grab
5:08:27any data model or your own data. Try to
5:08:30make a healthy data model out of it. So
5:08:32that's it my friends. I hope you enjoyed
5:08:34this project with me. If yes, then
5:08:36support the channel by subscribing,
5:08:37liking, and commenting. Share it with
5:08:39the others in order to reach nice people
5:08:41like you. And I'm going to say thank you
5:08:43so much for watching and I'm going to
5:08:44see you in the next project. Bye-bye. At
5:08:46you later, warehouse. See you on the
5:08:49flippity flip.

About this transcript

This page contains the full transcript of Data Modeling in Power BI Full Course for Beginners (5 Hours) – From Zero to Hero by Data with Baraa, generated from the public captions YouTube serves with the video. The transcript has 67,962 words across 9,104 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.