YT YouTube2Text

YouTube2Text → Transcripts → Databricks Tutorial (From Zero to Hero) | Azure Databricks Masterclass

Databricks Tutorial (From Zero to Hero) | Azure Databricks Masterclass — Transcript

by Ansh Lamba · 40,026 words · 5,660 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00this 4 hours of datab Bri master class
0:04will make you a tab Bri data engineer
0:07even if you do not have any prior
0:09experience because in this master class
0:12you will learn about spark cluster spark
0:16architecture data Brak fundamentals
0:18magic commands data brakes utilities
0:20datab brakes file system reading
0:22external data from cloud data storage
0:25secret Scopes ipark Transformations
0:27loading data incrementally using
0:30autoloader and after covering all these
0:34areas you will learn something special
0:37which is required to crack the
0:38interviews I am talking about Delta Lake
0:41you will learn Delta Lake in detail and
0:44learn about Delta log external Delta
0:47table versus manage Delta table
0:49tombstoning of files data versioning
0:52time travel Delta table optimization
0:55techniques such as zorder by vacuum
0:57command and much more wow that sounds
1:01like this is your onstop solution to
1:04become a datab brakes data engineer but
1:08you should only watch this video if you
1:10are dedicated enough to learn everything
1:13with excitement if your answer is yes so
1:17let's get started with this amazing data
1:20breakes master class so welcome welcome
1:24welcome welcome to data breaks Master
1:28Class
1:30this session is really really really
1:33interesting because I know that demand
1:35for data Brak is growing ridly and
1:39companies are going after like data Brak
1:41developers and they should because data
1:45Brak is like one of those Technologies
1:47which are in demand plus which are
1:49powerful enough and which are strongly
1:52inclined towards future they are
1:56continuously adding so many updates that
2:00you need to be updated enough to just
2:03cope with the pace so I can say that if
2:06you just see data breaks few years back
2:08it was totally totally different but now
2:12now now the scenario is totally changed
2:16there are so many updates in data brakes
2:19let me give you a hint so earlier we
2:22were using mounting if you want to just
2:26um read any data from the external
2:28location but now we use Unity catalogs
2:31now we use Delta lakes and in Delta
2:33Lakes we are seeing updates on a daily
2:37basis trust me like there are so so so
2:40many updates in the Delta Lake and it's
2:43very important to stay updated when I
2:46just look for the courses available in
2:48data breakes even the paid courses they
2:51are really really good the only thing
2:53that was missing was information was not
2:56updated information was not as per the
3:00latest Trend so here I am with terab
3:04brick master class and I will be
3:06covering all the latest
3:09information yes all the latest trends
3:13that are going with the data bricks so
3:16I'm just trying to fill that Gap and I
3:19hope that you will love the session and
3:22trust me like I have like read books
3:25related to data breaks I have read books
3:28related to Delta l and let me just show
3:31you as well so this book so this book is
3:35a gym so this is like specially related
3:38to Delta Lake and this this book I think
3:40is recently published so there's like so
3:44so so much of knowledge that is missing
3:47right now in the online platform so I
3:50thought like it's my duty for my
3:52community to just pass on this knowledge
3:56so that you can also be competitive
3:58enough and TR trust me if you want to
4:00crack interviews in today's scenario you
4:03need to be updated you need to be like
4:06you you need to be like someone who
4:08knows like all the latest trends and
4:10technology that are being used who is
4:13like well vered with all the recent
4:15updates so now you have found the best
4:19video which will be covering each and
4:21everything from scratch yes this master
4:24class will get will be covering
4:27everything from scratch so if you do not
4:28have any prior EXP experience any PR
4:30knowledge do not need to worry because I
4:32will explain you each and everything
4:34from scratch and we'll be covering all
4:37the latest updates and just a hint in
4:41this particular master class we will be
4:42covering Delta l in detail which is the
4:46most in demand technology right now and
4:49we will see how we can just work with
4:51Delta Lake and I will just try
4:54[Music]
4:55to pass on the knowledge that I have
4:58gained so far so do not need to worry
5:00you do not need to pay for any of the
5:02resources everything is available for
5:04free you just need to be focused you
5:07just need to just take out your
5:08notebooks pen pencils everything and
5:11just start taking notes because this is
5:13the onstop solution that you were
5:16looking for and now you have got it so
5:20let's get started with our Tera breaks
5:22master
5:24class so first of all let's discuss
5:26about the prequests so there are not
5:28much prequest obviously you just need a
5:31laptop or PC with stable internet
5:33connection so that you can just enjoy
5:35this video so that you can just learn
5:38from this video next data breakes
5:41account
5:43oops oops oops oops so if you do not
5:46have datab Bri account do not need to
5:48worry because I am here to help you how
5:51you can just set up a free datab Brix
5:53account and this will not be a databas
5:57community account no no no no no no as I
6:01just told you because this
6:03video is ah level up for like as
6:08compared to other videos so we will be
6:10just creating cloud account because that
6:13is required if you want to just crack
6:16interviews because if you want to show
6:18that you are experienced enough then you
6:20should know all the uh handson and you
6:23should know all the workings that we
6:26have within the cloud account so even if
6:28you do not have any cloud account we
6:30will create everything from scratch do
6:32not need to worry next next next next
6:35the main thing excitement to learn data
6:37braks because data braks is like the
6:41Hulk I I call it as Hulk because
6:44everyone is running towards data brakes
6:48why the main advantage of using data
6:51Brak is it is independent of any Cloud
6:53environment you can just use data braks
6:56with any Cloud environment and that is
6:57the best thing if you using aure you can
7:00use data Bri if you are using gcp that
7:02is Google Cloud platform you can use
7:04data briak if you are using AWS you can
7:06use data if you're using AWS you can use
7:08datax yeah so this is like Cloud
7:12independent so that's why all the three
7:15major Cloud players can integrate datab
7:19B and that's the biggest advantage of
7:21this and that's why every company is
7:24just relying on data breaks because
7:26obviously when you transform the data
7:28when you want to just work with the big
7:29data sets data brakes spark clusters are
7:33your only choice did I just say spark
7:35clusters don't need to worry I will tell
7:37you everything that I'm saying right now
7:41so what is a spark cluster what is a
7:44spark cluster so basically when we talk
7:46about data braks data braks we are
7:48actually talking about spark clusters
7:51okay because the founder of data BRS the
7:55creator of data brakes is the same that
7:57we were creator of Apache spot
8:00so now let's cover this
8:04topic what is a spark cluster what what
8:07is a SW cluster what is a cluster so the
8:11thing is let me give you an example so
8:12that you can just get this knowledge
8:15like very easily instead of just making
8:17it complicated let me just uh give you
8:20an
8:21example you must have worked in your
8:24computer lab maybe in your school or in
8:27your college so you would have seen so
8:30many computers in that particular room
8:33right let's say you saw like 30
8:36computers right let let's say you just
8:38saw 30 computers in one room right now
8:42if I just ask you to combine all the 30
8:47computers together and treat those 30
8:51computers as one computer can you
8:54imagine the power of that one computer
8:57would be the power of processing the how
9:01how fast it can process the data can you
9:04imagine obviously it is much more than
9:08one single computer even if I just say
9:13that you can just take one computer
9:16right and you can just increase its Ram
9:19its Rome its storage everything there is
9:22a limit to do
9:25it there is a limit but there is no
9:29limit when you combine computers now you
9:32would say I only have 30 computers so
9:35what you can just include another
9:37computer lab of 30 computers now you
9:39have 60 computers then you can include
9:42another computer lab of 30 computers now
9:44you have 90 computers so there is no
9:47limit when you horizontally scale your
9:52machines and in spark language we call
9:55machines as nodes let me type it for you
9:58and start taking notes start taking
10:01notes right
10:03now so we call them as nodes instead of
10:08machines like you can just call them
10:10machines as well but in spark language
10:11you call them as
10:13nodes so just imagine now we have 90
10:18machines now we have 90 nodes and we are
10:22treating them at treating them as like
10:24only one single machine now just imagine
10:27its power that is is spk cluster so
10:29basically when we combine machines when
10:33we combine nodes and we connect them
10:36together it becomes a cluster so as you
10:40can see let's say we have machine one or
10:42let's say we have computer lab one which
10:45have 30 computers we have computer lab
10:47two having 30 computers we have computer
10:49lab three 30 computers now this cluster
10:53is
10:54having 90 nodes 90 computers 90 90
11:00machines and these 90 machines are being
11:04treated as one single machine because
11:08these are
11:09interconnected now these M okay okay
11:13okay okay your question is valid why do
11:15we need to just combine so many machines
11:17because we can distribute our work just
11:20imagine I just got a task I just got a
11:23task to let's say process 1 GB of data
11:26this is an a hypothetical situation so
11:28don't about that let's see I just got
11:31okay let's take it as one
11:34TB one TB of data right one TB of data
11:38one
11:39terabyte so now I can just ask these 90
11:44machines to distribute the data to
11:48distribute some some data some part of
11:49data that's called like subset of data
11:51or let's say that's also called as like
11:53data partitioning so these like this one
11:56TB of
11:57data can be distributed among 90
12:01machines and these 90 machines can work
12:04in parallel and can deliver the
12:08results that is why we need parallel
12:13processing that's why we need
12:16clusters and that's why we need a par
12:19spark got it I hope like now you know
12:22the concept of spark cluster what is a
12:24spark cluster what is a spark cluster
12:27now after this I am going to cover one
12:30very important topic not just for the
12:33sake of learning but if you want to
12:36crack interviews this is the area that
12:38you should focus on
12:40definitely this is none other than spark
12:43architecture I know this topic is a
12:48little bit complicated but I will tell
12:51you like let me tell you like I I will
12:54just make this topic so so so easy to
12:57understand trust me just just be with me
12:58for like next few minutes and then you
13:01will be like okay now we have understood
13:03each and everything related to spark
13:05architecture just trust me
13:08now we don't know what is this okay now
13:12we don't know what is this now we don't
13:13know what is this what is this we just
13:15know so far that cluster is a group of
13:19machines cluster is a group of machines
13:22we just know this part okay you know
13:25this part very good so in this example
13:28we talked about that we have let's say
13:30three machines machine one Machine 2
13:33machine 3 and our cluster is just having
13:36three machines
13:38right okay we just have three machines
13:42so our cluster is having just three
13:44machines let's understand the spark
13:46architecture with this example so let's
13:49say we have just three machines so first
13:51of all obviously when I say I want to
13:54process one TB of data right
13:59and let's say this is
14:01you and you want to process this one TB
14:05of data this is
14:09you don't mind my drawing okay this is
14:13you right so now you have this is a
14:16cluster this is a cluster so let's call
14:20it as cluster because you just know
14:22cluster you don't know what is going
14:23inside this right so this is your
14:25cluster so now every cluster has a
14:29cluster manager so this cluster manager
14:33can be spark Standalone cluster manager
14:36or the external one as well so this this
14:39has a cluster manager Every Spark
14:41cluster has a cluster manager and
14:43obviously as name suggest it manages
14:45everything let me tell you let me tell
14:46you wait wait wait wait wait so now what
14:49it will do first of all it will use one
14:54machine one node to create driver
14:58application to create driver program
15:00this one first of all this is the first
15:03thing it will treat one machine as
15:07driver machine right just take notes
15:11just take notes so one machine is like
15:14driver machine or driver program right
15:17so this is our our driver program now
15:21this is done like one machine is
15:23occupied and we have two machines left
15:25right okay now what will what will
15:28happen now when we submit the codee to
15:31Cluster manager like to Cluster
15:35right it first goes to Cluster manager
15:39yes it first goes to Cluster manager
15:42then it creates this driver program this
15:45is done this you understood right this
15:47you understood now this driver
15:52program this driver program will divide
15:56your data processing in small small
16:00tasks it will just give the directions
16:04like what actually need to happen right
16:08let me repeat it will break your data
16:11processing command in small small task
16:14and will give instructions what actually
16:18need to happen so it has some
16:21information it has some information
16:25right it has some information now this
16:28information will go to the cluster
16:33manager okay now this information will
16:35go to the cluster manager now this
16:39information has also the information
16:43related to number of machines required
16:46number of machines required to actually
16:49execute the code and those machines are
16:52called as worker nodes called as worker
16:56nodes right got it so let's say this
17:00driver program said that I need both the
17:02machines I need this machine as well I
17:05need this machine as well okay now it
17:08will create worker node on both the
17:11machines okay worker
17:14one worker two okay worker one worker
17:17two so it will create worker node on
17:20both the machines now this
17:23information will be going to these
17:27worker nodes worker one and worker to
17:30and these worker nodes has something
17:32called
17:33executors as you can see
17:36executor executor so in data breaks we
17:39just have one executor per work worker
17:41node so don't worry about that so now
17:45this information will go to worker one
17:47and worker two right now cluster manager
17:51responsibility is done e says you gave
17:55me one TB of data I created driver
17:57program for you driver program gave me
18:00information to create two worker notes I
18:02have created that manager's
18:04responsibility is done manager's task is
18:06done now these two worker nodes will
18:10actually process the data will actually
18:12do the hard work then they will return
18:15this result back to the driver program
18:19yes now there's no intervention of this
18:22cluster manager is out now these two are
18:25communicating worker nodes is
18:26communicating with driver program this
18:28worker node is communicating with driver
18:29program and obviously the whole cluster
18:32is now synced with each other and we get
18:37the processed data out of
18:39it
18:41understood this is the easiest way this
18:44is the easiest example I could have
18:47taken so if you have understood some of
18:50this part it's really good because when
18:52I was understanding this part I've
18:53watched I think videos like so many
18:56times I just read books so many times
18:59because it is really confusing but try
19:00to grasp it rewatch this part again and
19:04trust me you will just grasp this part I
19:06know you have understood like at least
19:0750% of it I am damn sure I damn sure
19:10about it but just rewatch this part so
19:13to summarize let me summarize
19:15it this is you you want your Spar
19:18cluster to process your data right so
19:21you have a cluster manager this cluster
19:23manager will create one driver program
19:26right so this driver program will see
19:28your code okay you want to do this you
19:30want to do this so it will break that
19:32processing thing into small small task
19:34and it will provide the information to
19:36Cluster manager and then that
19:39information will also be written in this
19:40information page that we need two worker
19:42notes it will create two worker notes he
19:44will say okay my task is done now you
19:46can communicate now these worker nodes
19:48and dver program will communicate with
19:50each other one thing to note driver
19:53program will not do any kind of
19:55execution no it will not process your
19:58data it will just give the directions it
20:00will just orchestrate the task that's it
20:03that's it it will not process it will
20:07not do the hard work hard work will be
20:09done by worker nodes that's it simple
20:13and they do not need to use their brain
20:16no they will get the instructions they
20:18do it simple simple they do not need to
20:21think hey can we do this can we do this
20:23no everything will be instructed by
20:27driver program like what task should be
20:29done first and like what should be the
20:32orchestration flow everything will be
20:34provided by driver program this
20:37one simple this was all about spark
20:40architecture if you haven't understand
20:41100% of it it is fine that means you are
20:44learning and now it's time to just
20:46rewatch this part again and once you
20:47have rewatched it now let's start with
20:50now un please tell us like what is data
20:54so the thing is we just talked about
20:56clusters right now the best thing is you
20:59do you do not need to manage these
21:01cluster no data breaks is actually doing
21:06all the hard work for us it is so good
21:08it is managing this whole
21:11cluster for us it is kind of a
21:14management layer for our spark clusters
21:18yes that's why we use data brakes
21:20nowadays we do not need to manage
21:22clusters we do not need to do anything
21:24everything will be done by data brakes
21:27everything will be done by data break
21:28that's why we love datab brakes and
21:30that's why data brakes is the like OG it
21:33is like the
21:35OG so now data is managing our cluster
21:38that's a good news right yeah that's a
21:40good news because we do not need to do
21:41anything and we just need to configure
21:43our cluster don't worry I will just show
21:45you how you can do it so that was all
21:47about data brakes now you have
21:48understood what is actually data brakes
21:49so data brakes will manage our clusters
21:52data breaks will actually take care of
21:54all the Clusters everything right so
21:59now finally it's time to talk about
22:04account creation so there are basically
22:06two ways to create account one is
22:08Community Edition what is Community
22:10Edition so basically this fre this is
22:12the kind of free account that is
22:13provided by datab break itself when you
22:15want to learn something when you just
22:17want to uh have a have have an exposure
22:20of datab Brak right so this is a kind of
22:22communication account uh you can also
22:25create this account but the thing is if
22:27you want to crack interviews if you want
22:30to show that you are experienced in data
22:32breaks you need to show that you are
22:35well vered with working with data bricks
22:39along with the cloud storage because
22:40obviously in the real world we do not
22:42know we do not need to use Community
22:44Edition in the real world we do not use
22:46Community Edition accounts no we do not
22:48use this we use cloud accounts obviously
22:53because company data organizations data
22:56is not available on Community Edition it
22:58is stored in the data Lakes it is stored
23:01in the storage accounts of cloud
23:03providers such as AWS Azure gcp so in
23:07the real world we work with datab brakes
23:10and we connect our datab BR account with
23:12Cloud providers that's why in this
23:14particular video I will show you how we
23:17can create data account which is linked
23:20to a cloud provider and then we will
23:23learn each and everything linked to that
23:26cloud provider or let's say cloud
23:28storage account so that you can actually
23:30have that exposure you can actually have
23:33that experience how things are working
23:36with Cloud because it is totally
23:38different like not different but yeah
23:41like you cannot use storage paths which
23:42is like provided by the cloud and you
23:45actually cannot see the data because
23:48that is not shown to you when you use
23:50Community Edition right so there are
23:52many things that are missing in
23:53Community Edition but I will show you
23:56how you can just create your free uh
23:58cloud account cloud data account and
24:01definitely you will learn a lot trust me
24:03this can be the game changer for you
24:05because you will feel confident while
24:06sitting in the interviews and you can
24:07just tell them that I am experienced
24:09because I know like how to work with
24:11Cloud accounts because obviously when
24:14you sit in the interview and you will
24:15say I have only worked with Community
24:17editions obviously they will not
24:19consider you but now they will consider
24:23you because you are watching my video
24:24and I'm just providing the best quality
24:26of education so so for this particular
24:30video we will use Azure I love Azure so
24:33we will be using Azure it doesn't matter
24:36because you can just use any cloud
24:37provider because we are learning data
24:39breaks we are not learning Azure AWS or
24:41gcp we are learning data breaks along
24:44with any cloud storage right and as I
24:48have mentioned that data brakes is
24:50independent of cloud so when you learn
24:52data braks connected with any Cloud
24:54platform that means you have learned
24:56data brakes with all the platforms this
24:58is the best thing about data breaks and
25:01why I have picked data breaks with Azure
25:03because you need to configure so much
25:07when you work with Azure you need to
25:09just use keywords you need to just use
25:12service principle you need to just uh
25:14configure let's say a secret Scopes in
25:17data when you actually want to access
25:20data stored in the data Lake and that's
25:23what recruiters want you to know that's
25:27what they are looking for you should be
25:29well wored about the all the
25:32configurations that are required to
25:33access the data so that's why I have
25:35picked Microsoft Azure and it will give
25:38you like a lot of knowledge a lot of
25:40skills trust me so how this flow will
25:43work let me just tell you so first of
25:45all we will just create a free Azure
25:48account right and once we have that free
25:51Azure account we can just create a data
25:53braks resource within that don't mind my
25:56handwriting because I know it's really
25:58good so first of all we will create a
26:01your free account and then we will
26:02create data bre free account so without
26:05wasting any time let's get started with
26:08account creation and yes let's get
26:11finally started with datab brakes as
26:13well let's
26:16see to create in order to create free
26:20your account so first of all what we
26:22will do we will just go to Google uh
26:24let's
26:25see yeah just write this aure free
26:28account and then you should just click
26:30on the first link that you see and here
26:33here you just need to click on try aure
26:35for free do not click on this one pay as
26:37you go because we want to create a free
26:39account so just click on this and then
26:42it will just ask you to put your email
26:45ID so if you do not have any Microsoft
26:47email accounts you can just click on
26:49create one and if you already have one
26:52just put that email ID and then sign in
26:55with your email ID
26:58so after
26:59login you will just see this page so you
27:03just need to fill this form you just
27:05need to put your all the details name
27:08and your email address phone number
27:09everything and as you can see here on
27:12this side that you get popular free
27:14services for 12 months that's great and
27:1755 services that are always free and the
27:20best thing is you will get 200 credits
27:24US dollars to access this account so
27:26basically aure provides you $200 so that
27:29you can just use their services and it
27:31is only available for 30 days and I
27:33think that is enough to learn this
27:35lecture to just gain knowledge from this
27:37lecture and that is fine and the best
27:40thing you just need to click on sign up
27:43and that's all that's all just click on
27:45the sign up button and that's all after
27:48clicking on this it will just ask you to
27:50put your C details and all do not afraid
27:52to put those details because it will
27:54just ask you to confirm that you are the
27:56one who will be using their services
27:58and they will not charge you after 30
28:00days because they will just
28:01automatically remove those services that
28:04you were using and you will not get
28:05charged just trust aure so you do not
28:08need to worry just put those details and
28:10you are good to go so once you have
28:13created your aure account it's time to
28:15actually go to a your portal and
28:18actually see how we can create datab
28:21brakes workspace within a short data
28:23break yes let's see that so in order to
28:25access your portal account you just need
28:27to go on Google and just write portal.
28:29azure.com and it will just take you to
28:32the Microsoft Azure portal account so
28:35are you excited to see your first aure
28:37portal account wow this is our aure
28:41portal account do not need to worry if
28:43you see these services and not in your
28:46account because these are like my
28:48previous resources that I have created
28:50so far so do not need to worry now it
28:52should be fine right now I have just
28:54changed my tab from recent to favorites
28:57so
28:58this is the homepage of aure portal and
29:02this is your aure account where you just
29:06create all the resources where you just
29:08manage your your portal account
29:10everything is done here if you are new
29:14to a your so this is your first homepage
29:17congratulations like I was really really
29:18excited I still remember those days like
29:21when I was just creating my first
29:23account and I just landed on this page
29:26for the first time I was really excited
29:27and I love love the UI like it is so so
29:30so good so as I just said like if you
29:33just see different resources or
29:35different Services here do not need to
29:37worry because it just shows the services
29:39that you use that that you have used
29:42before so do not need to worry do not
29:44need to worry so first of all first of
29:46all first of all first of all before
29:49just creating a your data break or let's
29:51say data break service I just want to
29:53tell you like some of the basics that
29:55you should know about as y like it is
29:57like very very Basics so first of all I
30:00will just tell you what is this Resource
30:01Group so basically whenever we want to
30:06store our services what are Services
30:08services are like whenever you create
30:10your databas that is a service whenever
30:12you create a storage account that is a
30:13service whenever you create anything in
30:15your that is a service so we need to put
30:17those services in a folder and that
30:20folder is also called as Resource Group
30:23that's it that that is a kind of fancy
30:25term that we use in a y but that is just
30:26a folder do not need to work
30:29so we need to create two resources for
30:32this master class what are these two
30:33resources first of all first of all we
30:37will
30:38create data Lake obviously nowadays we
30:42work with data Lake a lot and data lake
30:45is the future obviously Delta lake is
30:47the future but Delta L is built on top
30:49of data l so do not need to worry
30:51actually Del I will just tell you each
30:52and everything about Delta L but the
30:54storage the core storage is data L and
30:58and I will also show you how you can
30:59create data l in aure so do not need to
31:01do not need to worry bro first is data L
31:04and second thing that we need is
31:05obviously data brakes data breakes
31:08account or let's say data brakes
31:10workspace these are the two things that
31:11we want to create right now in order to
31:15access or let's say in order to uh
31:19proceed with the further steps so these
31:21are the two things so first of all let's
31:23create a data link and a quick question
31:27for you if you want to store your data
31:28like where you will be storing it we
31:30just discussed yes in the resource Group
31:34so we have to create a resource Group so
31:37actually we need to create first a
31:39resource Group then these two services
31:41so how you can create a resource Group
31:42just go on this search Tab and then just
31:45write let's say Resource Group it should
31:47pop up yeah here we have click on
31:49Resource Group and don't mind my
31:51previous resource groups just click on
31:53plus create button here in this corner
31:56just click on this plus create
31:59and click on create Resource Group so
32:01then we need to just name our Resource
32:02Group so you can just give it a name
32:04anything like I will just give like data
32:08breaks Master
32:12Class simple and I will just add RG in
32:16the beginning as a prefix so region you
32:20can just pick any region don't need to
32:21worry you can just pick any region yep
32:24so now just click on review plus create
32:27simple so now it is validated just click
32:30on create for one more time and your
32:33group is ready how you can search your
32:35resource Group so just go to your home
32:38Tab and then if you see this button you
32:40can click on it either you can just
32:42write your resource Group like here
32:44resource groups and then it should just
32:47show you the name yeah mine is here so
32:51as we all all as we all can see that
32:54this is empty why because we do not have
32:56anything we do not not have anything
32:58that's why it is empty right so first of
33:01all let's create storage account which
33:03is which is also known as datal not
33:05really not really so there is a tip that
33:09I will show you while creating these
33:10storage account so by default when we
33:13create a storage account in a j it
33:15creates The Blob storage but by that tip
33:18you can just convert your blob storage
33:19into a data l so let me show you click
33:23on this plus create by the way this is
33:24also another interview question so you
33:26can just note it down so I will be
33:29sharing like lots of interview questions
33:30as well so do not need to worry just
33:32take notes
33:33and be happy and be excited so first of
33:37all I will just write storage
33:40account so and just be careful by
33:44picking the resources because we just
33:46need to pick the services provided by
33:47Microsoft because this is a kind of
33:49marketplace where external providers
33:51have already put some of the resources
33:54or Services as third party services but
33:56we are just using Microsoft on click on
33:58this and click on Create and then we
34:02need to pick our Resource Group and as
34:04we have already created one and as we
34:07have already picked that Resource Group
34:09so it is already filled very good now we
34:12need to use this box to give a storage
34:15account name first of all the account
34:17name that I will be using here you
34:18cannot use it no I'm not just ordering
34:21you that's the rule because every
34:23storage account name should be unique
34:25throughout the Azure Network so you can
34:28just uh pick a a unique name or you can
34:30just follow my tip so what I will do I
34:33will just say data
34:36Lake and I will just write
34:39unch yeah it should be
34:41unique we do not have much on available
34:44on this
34:45Earth so it is fine so you can also
34:48write like data leg and then your name
34:50anything right so it's up to you then we
34:54need to pick the region region is fine
34:57okay then we have two options standard
34:59versus premium just pick standard now
35:01one important thing in case of
35:03redundancy you need to pick lrs now what
35:06is
35:07lrs so basically we have data rency
35:10options available in a lrs means locally
35:12R storage where data is replicated
35:15within the same data
35:17center do you really need to worry about
35:19this right now no so I'm just giving you
35:22this knowledge for your sake so we have
35:24like four different types of rency
35:25options lrs GRS and then zrs zzz Z zrs
35:31and gzrs so the most expensive one is
35:34like
35:36gzrs and the cheapest one is lrs so I
35:39will just pick lrs obviously this is a
35:41free account and obviously we have like
35:43us $200 but you should be like aware of
35:46the money that you are spending because
35:48obviously once you land as a data
35:50engineer and any organization you should
35:52be worry about their cost as well right
35:54so just click on next instead of click
35:56on review plus create no no no no if I
35:58just click on this button it will create
36:01the blob storage and we do not want to
36:02do that do that we need to create a data
36:05lake so just click on next and here is
36:09that thing so when you just check this
36:13box when you just check this box you are
36:16actually saying as your that please
36:19create a data L for us now what is so
36:23magical in this box that when we check
36:26it it will create a data l so the thing
36:27is in data L we can store hierarchial
36:31folders hierarchial let's say hierarch
36:35name space means you can just store
36:36hierarchial folders so let's say let me
36:39just show you let's say you have a
36:41container this is your container right
36:44and within this container you want to
36:46create a folder and within this folder
36:48you want to create another folder within
36:50this folder you need to create another
36:51folder then you need to save your files
36:53you can do this and this is called Data
36:56L when you can just store folders within
36:58folders and then you can just save your
36:59files but in case of blob storage you
37:02cannot do these steps you will just get
37:06the opportunity to store data within the
37:08containers only that's it that's it now
37:12everything is good now you can just
37:13click on review plus create that was
37:15really an important interview question
37:17just note it down and note all the
37:19questions
37:20because in the future we will be
37:23discussing lots of lots of lots of
37:24questions so not all the equation all
37:27the question like take notes of all the
37:28questions so it is deploying our storage
37:31account as you can see deployment is in
37:33progress and it should take just few
37:34seconds and then we will be having our
37:36full storage account ready and after
37:38this we will be creating our data Pricks
37:43account okay let me just refresh
37:51it okay this is done this is done this
37:54is done so now we can just go to our
37:56resource so if you want to go to your
37:59resource just click on it okay so this
38:01is your storage account okay do not feel
38:05overwhelmed do not feel overwhelmed
38:07because this is just kind of overview or
38:09you can say summary of our resource so
38:11we are just concerned with this thing
38:14containers this is our data Lake we also
38:16get three more services within the
38:17storage account but we are not using
38:19that one like file shares cues tables so
38:21file share is like a kind of onetop
38:23solution when you want to share files
38:25with all the team members and then cues
38:28is used for like storing messages when
38:31we are streaming something and then
38:32tables is for like no SQL or you can say
38:35store data in the form of key value
38:37pairs so yeah but we are concerned with
38:39data L that is continuous click on it
38:41now we want to create a container why
38:44why why why because within this
38:46container we will create a
38:48folder simple and we can actually create
38:51two containers two containers why
38:54because one container will get treated
38:56as source and second container will get
38:59treated as the destination or let's say
39:03will get treated as the solving layer
39:05because I will show you to create tables
39:08Delta tables everything so we will
39:10create two containers one is like you
39:12can say source and second one will be as
39:15destination and within these containers
39:17we will just read and write our data so
39:19first of all let's create a container
39:22okay so I will call it as
39:25Source oh yes perfect source so now I
39:30have a container called Source I will
39:31just create another one called
39:34destination
39:37okay so source and destination two
39:39folders are two containers are created
39:41so now within this Source we have
39:43nothing we have nothing but we should
39:46have some data so that's why I will just
39:50upload one CSV file here right now and
39:53do not need to worry I will just provide
39:54the link of this data set in my just
39:57description of this video and you can
39:59just download this data set and you can
40:01also upload this data so I will show you
40:02how you can do that the step is very
40:05simple we just need to click on this
40:07upload
40:08button that's it and then we will be
40:11having some data available in data lake
40:14so now I know you are getting a why what
40:17we are trying to do we have our real
40:20data sitting in a your data link and now
40:24we will read this data we will work with
40:26this data in data bricks and yes we will
40:29follow all these steps how we can
40:31actually do this yes that's why I picked
40:34a short that's why I pick the short so
40:36in order to upload let me just upload
40:37the
40:40data so as you can see that I have
40:42uploaded sales. CSU file and you can
40:44also upload it and I will just provide
40:46the link in the description so once you
40:48have uploaded the file just click on
40:50this upload button that's it see our
40:52data is uploaded our data is uploaded if
40:56you want to have a look with the file
40:59within aure you can do it just click on
41:00that file and then click on this edit
41:03button so this is the data do not worry
41:06this is a comma separated values file
41:08that is called CSV which is separated by
41:10comma if you just click on preview
41:12button you will actually see the data in
41:15the structured
41:18format
41:21see so now you can just see this data so
41:24this is the kind of preview that are
41:26that is available in sure so our
41:29groundwork is ready we have dat ready
41:31within our aure Network so now it's time
41:34to actually create the aure databas
41:36workspace let's do that so in order to
41:38create aure databas workspace you simply
41:40need to go on homepage go there then you
41:44can just click on resource groups just
41:46select your resource Group that you
41:49created just now so now then you can
41:53just click on create button because we
41:55want to put our data account datab break
41:58workspace within this Resource Group
42:00right so simply search datab
42:03brakes oops datab brakes search it then
42:08you should select this one as your data
42:11break which is provided by Microsoft
42:13just remember this click on this one and
42:15then click on create button
42:18simple okay then same steps same steps
42:22we just need to name our workspace so I
42:23will just name it as data
42:27bricks
42:30master databas master let's let's call
42:33it databas master so now the thing is
42:36pricing deer which one you should pick
42:38so basically you have two options
42:41premium and dral why I'm not saying
42:42standard because standard doesn't
42:44provide you all the uh functionalities
42:47so basically you can either pick premium
42:50because you have like 200 cards and if
42:52you do not have 200 cards you have
42:54already spent you can just create trial
42:55premium which is just the trial version
42:57of the premium available for 14 days it
43:00anything should work fine so for this I
43:03I will just pick premium so you can also
43:05pick premium so here is the thing why we
43:07need to put this manage Resource Group
43:09name
43:11again good question right so the thing
43:13is this manage Resource Group name is
43:16not for us then it is for data PRS
43:20remember we talked about data braks
43:22manages our clusters everything it
43:24manages so it will put all those
43:27resources all those virtual machines
43:30within this Resource Group and we should
43:32not touch this Resource Group because
43:34this is for aure data this is for aure
43:37data bre
43:38only simple so we will just name it
43:40let's say RG uh RG
43:45managed DB
43:48Master
43:50simple BB Master Class like you can name
43:53it anything because we will not be using
43:55it don't worry don't worry don't worry
43:56don't worry don't worry so now we are
43:59good to just create this just click on
44:00review plus create and then you can just
44:03click on create button for one more time
44:05and now it will just take few more
44:07seconds and your data workspace will be
44:09ready and let me tell you you are just
44:12few steps away to actually get started
44:15with aure data braks or data braks with
44:18Azure so once it is deployed I will just
44:21show you the user interface I will just
44:24give you a walk through with the data
44:26works space and then obviously step by
44:28step we will be learning each and
44:30everything required for your data bricks
44:32and yes Delta Lake in detail because you
44:36cannot expect working with data bricks
44:39without Delta L without Delta L you
44:42cannot expect no no no no no so it will
44:45take just few more seconds and after
44:47that we are good to explore our databas
44:50workspace I know you really excited I
44:52know I know I was really excited when I
44:55was learning this so we can just just
44:57check the status here as well so just
44:59click on Resource Group select your
45:01resource
45:03Group and
45:04then here is our data breakes Master
45:07Class ready it is ready when you just
45:10click on
45:13it so here this button is missing which
45:16is launch workspace that means it is
45:19taking some few more seconds to created
45:21once it is ready we can just click on it
45:23so just wait for a few more just just
45:25wait just take a chill Pell
45:28just take a chill pill I'm just
45:29refreshing my page as you can see
45:31deployment is in progress so yeah we
45:33know that and we are waiting so you just
45:36need to wait for a few more seconds and
45:38once it is done we will just kickart our
45:40datab journey
45:42so it is deployed so now I'm really
45:46excited to hit this button that is
45:48launch workspace because this will take
45:50us to datab break workspace so without
45:53wasting a time let's click on this
45:57so this is your first aure data breaks
46:01workspace so you're finally working with
46:05Cloud oh oh oh okay okay okay okay okay
46:09so this is your your data breakes
46:13homepage how is it if your if this is
46:16your first data break workspace
46:18obviously it is like full
46:20of questions and at the same time full
46:23of excitement as well so do not need to
46:26worry because is in the coming
46:28sessions or let's say in the next few
46:32more minutes you will be well wor with
46:35each and everything available in data do
46:38not need to worry at all so let me give
46:42you a walkth through first of all what
46:44is this what is this left pain so it is
46:49everything so just kidding so this is
46:51your workspace so basically workspace
46:56you can just create different different
46:58workspaces within datab Brak workspace
47:00let's say you want to
47:01store um you you want to keep uh data
47:07let let me just click on it and let me
47:09just show you because that will be
47:10better for you to learn instead of just
47:13giving you theoretical knowledge so this
47:15is your let's say datab Bri workspace
47:18within this datab bre workspace you can
47:20click on this button this there's a drop
47:22down and you can actually create
47:25workspaces
47:27and workspace is nothing but a kind of
47:29you can say repository where you can
47:31just actually store the
47:33notebooks simple yeah and how you can
47:36create it it is very simple let me show
47:37you simply click on this workspace
47:39button and it will just show you repos
47:41shared users just click on this drop
47:43down you will see nothing but just click
47:46on this create button and then you have
47:49these options available but as I just
47:53said that first we will create folders
47:55because folders are really really handy
47:57whenever we want to do something so we
48:00can also create notebook we can also
48:02create dashboard we can also like create
48:04so many things directly within the
48:06workspace but it is a recommended
48:08approach that we we should first create
48:10the folder and then within that folder
48:11we should create these resources so
48:14first of all I will just click on folder
48:16and I will name it as data brakes
48:19workspace or let's say master
48:22class data breaks Master Class simple um
48:26it's to long but yeah it's fine click on
48:29create button simple now you can see
48:32that if workspace is created data bricks
48:34master class now you can create
48:36notebooks and different different stuff
48:38within this particular folder yes we
48:40will create it let me just give you a
48:42work through with all the things as well
48:45then we have recent stab so recent tab
48:47is recent tab is like really handy when
48:49you want to open something that you have
48:52recently opened so this will just show
48:54you everything and for now if I just
48:56click on it uh it will just show me
48:58nothing because I haven't opened
48:59anything then within the catalog tab so
49:04basically when we just click on this
49:06catalog button it shows us like all the
49:08data catalog all the uh let's
49:12say data basically data catalog is like
49:16one stop solution for your all the
49:19metadata metadata is like data about
49:21data so all those things like data types
49:23table schema database everything Under
49:27One Roof that is called like data
49:28catalog so you can just monitor each and
49:30everything from this tab if I just click
49:33on workflow so workflows
49:35is a feature that allows us to
49:38orchestrate our notebooks within data
49:40break so we can just orchestrate we can
49:42just run our notebooks after like in a
49:45sequence or in a flow using datab break
49:47workflows then then then then the very
49:51important thing which is this compute
49:53button what is this compute button so
49:55basically we talked about cluster right
49:58so now this
50:00compute button or let's say let's this
50:03compute tab this create compute button
50:07will allow us to create a cluster yes
50:10this will allow us to create a cluster
50:12and we just need to configure it and
50:14everything will be managed by data brex
50:16itself that's it this is so so so good
50:19so we will do it just just hold on hold
50:21on hold on then we have like tabs such
50:23as CQ data engineering machine learning
50:25so these are some of the pack packages
50:27that we have available under every tab
50:29so if you are like more inclined towards
50:31data engineering then it will just show
50:33us these things like data inje it will
50:34show us the connectors and in case of
50:37machine learnings it will just give us
50:38some of the these model buildings and
50:41these let's say machine learning model
50:43building capabilities so it has give you
50:46a customized tab as well so you can just
50:49focus on these Services align to your
50:52role simple then we have Marketplace so
50:56this Market places like third party uh
50:58connectors or let's say third party
51:00services that you can use within datax
51:03then at the end we have partner connect
51:05this is very handy because whatever data
51:08whatever data lak house you will be
51:09building within data brakes you can
51:11actually connect with the third party
51:13Solutions let's say I will click on part
51:15partner connect and I will see like bi
51:17tools such as this cross this such as
51:21you can say powerbi table click view so
51:24you can actually connect with these
51:26third party Solutions and at the end I I
51:29think I can just show you like how you
51:30can just connect this once we have like
51:32everything set up so you can just
51:34efficiently create uh create your powerb
51:37dashboards T dashboards connecting your
51:39data within the data bricks so
51:41everything is available Under One Roof
51:43and that's why data brakes is very much
51:45in demand yes that is the sole reason so
51:50now I want to create cluster I want to
51:52create cluster so I will simply go to
51:54compute Tab and I will just click on
51:56this compute but I will create create a
51:58cluster so to create a cluster simply
52:00click on this compute button and just
52:02need to click on create compute this
52:04step is very important and this is not a
52:06rocket science but still like you should
52:09be aware of this step as well so first
52:12of all we will pick the policy first
52:15first of all first of all you can just
52:16rename your cluster but I just like to
52:18keep this name like an lambas cluster
52:21you can just like name it anything but I
52:23do like this thing but you can just
52:24click on this pencil button you can just
52:26change it as well do not need to worry
52:29about anything till the time you are on
52:32my channel so click on this policy I
52:34will keep it as unrestricted so here are
52:39the thing like multi- node and single
52:41node for this particular master class we
52:44do not need multi nodes because we are
52:47not processing large amount of data but
52:49in real world you will just click on
52:50Multi node but for now you can just
52:51click on single node this will not
52:54affect your learning because we are not
52:55managing any clust I told you data
52:57brakes manages everything for us click
52:59on single node then access mode is
53:02single user single user access done then
53:05databas run time okay let's pick one
53:07data breas run time so I can just pick
53:09this 14.3 LTS that is for long time
53:11support and I have like worked with this
53:14run time like so many times so it is
53:16good so then we have like Photon
53:18acceleration there's no need to check
53:20this then node type node type we will
53:23just keep the basic node with in which
53:25we have like four course so I will pick
53:27DS3 V2 14gb memory that is enough now
53:30what is this terminate after button so
53:32basically we do not want our cluster to
53:34run for 2 hours that is for like 120
53:36Minutes like if our cluster is inactive
53:39for 120 minutes and after that it will
53:42be disabled automatically but we need to
53:44cut this time we can just say only 30
53:47minutes of inactivity is enough if I'm
53:50inactive for like 30
53:52minutes BR what you are doing brow yeah
53:56so so if I'm just inactive for like 30
53:58minutes it can disable my cluster to
54:00save my cost obviously for now we do not
54:02need to worry about cost but in real
54:04world you should be aware of this step
54:05as well because at the end you can make
54:08a lot of impact if you are saving a lot
54:10of
54:12money for the organization so tags we
54:15are good now it's simple it's done just
54:17click on create compute done it will
54:20take like few minutes to create our
54:21cluster so as you can see it is rotating
54:26so once it is done we will see one green
54:28button that is also called as online
54:31button but yeah in this world in spark
54:33world we have like cluster is ready that
54:36means it is green so once it is green
54:39like meanwhile we can do one thing we
54:41can just create our notebook we can
54:43create our notebook yeah so how we can
54:45create it simply go to
54:47workspace and just select your workspace
54:50by clicking on this button and just
54:52click on your workspace that you have
54:54created and then just click on this
54:57create button click on this create
54:58button and this time we want to create a
55:01notebook okay
55:03yeah so this will open a canvas in which
55:08we will be seeing notebook and if this
55:10is your first notebook then you will
55:12have lots of questions so yeah that's
55:14good that's good so wow this is our
55:17notebook this is our notebook this is
55:19our notebook don't worry don't worry
55:22don't worry I'm here to answer your all
55:23the questions so first thing first just
55:26rename your notebook because you will
55:28forget it when you will be just coding
55:31it and after some time you will see Hey
55:33where's my notebook and I don't know and
55:35then you will you would need to just
55:37check all the notebooks manually and I
55:39don't know I I don't want that so first
55:41of all just click on this
55:43and just press backspace and then you
55:47can just give any name let's say I want
55:49to give
55:53tutorial simple hit enter now your
55:56notbook is named as tutorial simple
55:59simple simple so first of all what is
56:00this file button so basically datab
56:03braks allows us so many options like we
56:05can just create new notebook obviously
56:07we can import a notebook as well yes if
56:09you already have a notebook saved okay
56:13so what does it mean so basically I will
56:15just provide the link of this notebook
56:18of my notebook as well so if you want to
56:20refer it you can simply click on import
56:22and that's it you can have my notebook
56:25in your workspace C then you have like
56:28share schedule CH defa sell language we
56:32will just talk about it don't worry then
56:33we have clone if you want to clone your
56:35notebook then rename obviously we
56:37there's a there's I think better way to
56:40do it directly from here then export
56:42what is what does this mean export now
56:45when I will be uploading my notebook in
56:47my GitHub repository I will use this
56:49option I will use this option so I can
56:52export my notebook into my personal
56:56computer and I can just share it with
56:58anyone I can even export the whole
57:00folder let me just show you let me just
57:02click on it so now as you can see DBC
57:05archive source
57:08file python notebook
57:11HTML so you can create anything you want
57:14so DBC archive means I can just download
57:18this DBC archive so whenever you want to
57:20import a notebook you can just click on
57:22this and it will open it in the datab
57:25briak workspace
57:26right and this IPython notebook you can
57:29just uh open this notebook in any IDE
57:33let's say Jupiter or anything but DB
57:36kive is really good let me just click on
57:37this import button then you will know
57:39when you click on import then obviously
57:42when you will click on browse you can
57:44either pick DRC or any I python notebook
57:46do not need to worry I will just provide
57:48the link and you can just click on
57:49import and you can just download my
57:51notebook from GitHub repository and then
57:53you can just upload using my notebook
57:55and and obviously using your PC simple
57:59so this was all about
58:01file now what is this edit button so
58:04basically like delete cell insert cell
58:06select cell and do not need to worry
58:08because we do not use this edit tab to
58:12run the cell to delete it we just use
58:14the shortcuts and I will tell you what
58:15are those shortcuts now this view but
58:18button is very important for some so the
58:21best feature of this button is like
58:23editor theme not editor theme workspace
58:25theme if if you want to use dark theme
58:27you can just click on it and you will
58:29have dark theme but I do like dark theme
58:32but while working with data Bri I like
58:34White theme it's up to you if you want
58:36to keep it dark you can keep it dark but
58:38I really like light theme while working
58:40with data I don't know why but yeah it
58:43is what it is simple and then this was
58:46all about like some some of the basic
58:48things that you should see in the
58:49notebooks and now now now now what is
58:52this connect button so basically
58:54whenever you run your notebooks you
58:56should connect your notebooks to a
58:58cluster because obviously whatever code
58:59you will be writing you need some
59:01computation to actually access your code
59:04simply click on this connect button and
59:06it will show all the available clusters
59:08right now so as you can see my cluster
59:10is also ready it it is showing green and
59:12I can simply click on it and it will
59:14connect my cluster connect my notebook
59:16with the cluster simple now I can just
59:18write anything I can just run anything
59:21and everything will be run by this
59:22cluster these machines in this case this
59:24machine because there is single LOE
59:27simple simple so this was an overview of
59:31data Pricks notebook now it's time to
59:36create chapters yeah chapters because it
59:39will be really handy when you will be
59:40creating chapters so what is the chapter
59:42number one okay let me show you so the
59:45chapter number one is Magic commands and
59:49why we call it as like magic commands
59:52what is the magic Behind These commands
59:54and what are these commands
59:58so basically as we all know as we just
1:00:01talked about that we can just use
1:00:02multiple languages within databas
1:00:05notebook we didn't talk about it I know
1:00:08so the thing is we can use multiple
1:00:10languages within the databas notebook
1:00:13yes you can use Python you can use SQL
1:00:17you can use R I personally hate
1:00:21R you can use MD what is MD MD is also
1:00:25called called as markdown which is very
1:00:27handy when you want to create headings
1:00:29subheadings titles within those notebook
1:00:31cells then FS file system there are like
1:00:34much more magic commands so why we call
1:00:36them as magic command so basically let's
1:00:39say this is your cell we just saw like
1:00:41how cell looks like let's say in this
1:00:44cell you want to use Python okay and
1:00:49then you want to use SQL you can use
1:00:51this magic command you can use this
1:00:54magic command
1:00:56you can just switch among different
1:00:58different languages like
1:01:00this like this so you do not need to
1:01:03change the default language every time
1:01:06we just saw default language right now
1:01:07we are just going to cover that as well
1:01:09so there's a default language written
1:01:11here for all these CS by default it keep
1:01:13it as python because ppar is a very
1:01:15popular uh API right now and everyone
1:01:18prefers writing P code in pypar which is
1:01:21python API of spark so it shows python
1:01:24by default so by default def it will
1:01:26execute our code in Python language but
1:01:29let's say in this said you want to
1:01:30execute a code in SQL you can instead of
1:01:32changing your default language every
1:01:34time you can just use magic command and
1:01:36it will do magic for you it will just
1:01:39treat it as your SQL
1:01:41command simple let's cover these magic
1:01:44commands first let's let's let's see
1:01:47so so I'm in my databas notebook so the
1:01:51default language I was talking about is
1:01:52this one python it is already written
1:01:55python in your scenario as well it is
1:01:56python now someone among like someone
1:01:59from you like will say hey if I just
1:02:01want to change my default language yeah
1:02:03you can do that how you can just click
1:02:05on this file
1:02:07button and here you can see this change
1:02:10default cell language this one and when
1:02:14you click on it it will show you python
1:02:16equal Scala R so some of you like will
1:02:18prefer doing coding is Scala I'm not a
1:02:21fan of Scala I just like python mode so
1:02:24it's up to you so by default it gives us
1:02:26the option as python so if you want to
1:02:27change it you can just change it from
1:02:28here so then it will just show here
1:02:30Scala or r or SQL if you want to change
1:02:33it so if I just write anything let's say
1:02:35I want to write any python code
1:02:37print hello sir then if I run it now
1:02:41another shortcut key if you want to run
1:02:43the cell there are two ways so basically
1:02:45you can just run it from here as well
1:02:47like by going to edit and then just run
1:02:49cell every time so other way of doing it
1:02:52just click on shift plus enter just just
1:02:54press shift plus enter
1:02:57yeah it it ran yeah it is running see it
1:03:01is running yeah and it print it it gave
1:03:03me the output as well hello sir simple
1:03:06and you saw by default it ran my code AS
1:03:10under like Python language now if I just
1:03:14run something like let's say I want to
1:03:16run a SQL command so I can just type
1:03:18percentage SQL and default language is
1:03:21still python now it is changed because
1:03:23it recognized it because I wrot magic
1:03:25command this is this is the magic so if
1:03:28I just type select
1:03:30hello
1:03:32sir that's it if I run it it will return
1:03:36me a table like hello sir column name
1:03:38and value is hello sir simple see this
1:03:40is this is the magic this is the magic
1:03:42that it does this is the magic now I
1:03:45know you are really excited to uh try MD
1:03:48what is MD so MD is a markdown first of
1:03:50all let me write percentage MD so if I
1:03:53just write anything if I want to create
1:03:55a heading it gave me so many options
1:03:56like heading one heading two heading
1:03:58three bardic everything if I just want
1:04:00to make a heading let's say H1 it will
1:04:03add one hashtag and if I just write
1:04:05anything let's say data
1:04:09bricks
1:04:11data bricks master
1:04:14class and the same way you can run it
1:04:17Shi plus enter see you can just create a
1:04:20heading and under this heading you can
1:04:22just create sales and I know now you
1:04:25know why it is important because it
1:04:28promotes readability of the notebook you
1:04:31would know which cell is doing what it
1:04:34is like adding comment but with better
1:04:37UI you can even add text something like
1:04:40this like percentage MD wait wait wait
1:04:42percentage
1:04:44MD and then you can just say I want to
1:04:47make any text bold without making a
1:04:50bigger heading so let's
1:04:51say data reading let's say so it will
1:04:56create a title simple and if you want to
1:05:00create a heading of different size so
1:05:02just click on it and let me hit enter
1:05:04let's say heading two I will say data
1:05:08bricks it will create a heading of a
1:05:10little bit smaller size see heading
1:05:13to you can do anything like you want and
1:05:16it's up to you like how you want to
1:05:18create your notebooks how you want to
1:05:20keep the readability of your notebook
1:05:23and obviously more the readability
1:05:25better the performance because whenever
1:05:27you will be revisiting your notebooks
1:05:29you can easily debug your code so it's
1:05:31not about the performance of The
1:05:34Notebook but it's like your uh your
1:05:36personal performance so let me just
1:05:38remove it so this was all about magic
1:05:41commands let me just run it because I
1:05:43want to keep it yeah that was all about
1:05:45magic commands so now I want you to
1:05:49create your Force data frame using your
1:05:51own customized data yeah you can do that
1:05:54so obviously we create data frame on top
1:05:56of the data setting in the data lake so
1:05:58what do I mean let's say this is our
1:05:59data
1:06:01Lake and this is our data Lake container
1:06:05obviously we create data
1:06:09frame on top of this data but you can
1:06:13also create data frame like with
1:06:15customized data with the data that you
1:06:17want to put and for that you do not need
1:06:20to read any data and how you can create
1:06:22it it is very simple first of all you
1:06:24need to create one data it like simple
1:06:26data a list of tles or list of record
1:06:30simple I will create a list and I will
1:06:32say like one and then name one is my ID
1:06:36AA is my name and then I want to put
1:06:39their marks let's say 30 I know it's
1:06:42less but still let's keep it oh so datab
1:06:45Brak has all so this that is the reason
1:06:47I love datab Brak so it monitors our
1:06:50code continuously as we are writing so
1:06:53it has identified a pattern that I'm
1:06:55using this kind of pattern so it has
1:06:57given me suggestion see that I can use
1:07:00these things I love data so I will just
1:07:03write two and I can just click on Tab
1:07:06button to just fill the data no I don't
1:07:09want more data I just want three records
1:07:11thank you so much for your help so now I
1:07:13can just Define schema so how you can
1:07:15Define schema schema basically is like
1:07:18the column names and the data types
1:07:19simple so in our scenario we have three
1:07:22columns like one is ID
1:07:25then we have name and then their Mark
1:07:29simple so I will Define I will call it
1:07:31as my
1:07:32data my
1:07:35data and I will call it as my
1:07:39schema simple and how we can do it just
1:07:42use a string and let's say ID own he's
1:07:45so smart but not 100% because it has
1:07:48suggested me a column with age and by
1:07:50the way it is not age it is
1:07:52marks obviously how does it know so
1:07:55simple so this is our groundwork for it
1:07:59and we just need to use a function
1:08:02called create data frame let me write it
1:08:04for you it's create data frame this
1:08:08function helps us to create our own data
1:08:10frame using our customized data so I can
1:08:13simply say DF equals to spark dot create
1:08:17data frame see it gave me suggestions
1:08:20okay okay it gave me some more
1:08:21suggestions okay okay we do not need
1:08:23your suggestion let me let me let me
1:08:24provide the knowledge okay my
1:08:26data comma schema equals to my
1:08:32schema simple so as you can see let me
1:08:36just make it a little bit
1:08:38zoomed not this much yeah this much is
1:08:40an yeah so as you can see that I have
1:08:44provided the schema here and my data
1:08:48here now I just want to run it and how
1:08:51you can run it shift plus enter I won
1:08:53repeat this because I have already
1:08:54repeated this so many times so you
1:08:56should remember it so if you want to
1:08:58display your data how you can display
1:09:00your data so there's a function called
1:09:03display display and then you can just
1:09:06write your DF you can just encapsulate
1:09:08your DF inside this or there's another
1:09:10way called DF dot
1:09:14display the suggested way is this
1:09:18but I don't know I like using this it's
1:09:22up to you I'm just kidding so yeah that
1:09:24I'm not kidding about this justed I
1:09:25suggested as the previous one because it
1:09:26is very handy when you want to build
1:09:28visualizations yes yeah you you you can
1:09:31just build visualizations as well I will
1:09:33show you don't worry bro don't worry
1:09:34just just stay calm just stay here so I
1:09:37can just run this command and then you
1:09:40should see your data frame this is your
1:09:41first data frame that you have created
1:09:43in data breaks 1 a a 30 2 BB 40 3
1:09:50cc50 simple simple simple simple this is
1:09:52your like data frame like columns
1:09:55records and everything this is your
1:09:56customized data frame that you have
1:09:58created using your own data and now it's
1:10:02time to jump on the chapter 2 in which
1:10:05we'll be covering much more than the
1:10:07basics and yes let's start with that now
1:10:12let's discuss this chapter so this is a
1:10:15part of that chapter so dbfs what is
1:10:18this dbfs this is really really
1:10:20important concept so basically dbfs
1:10:24stands for
1:10:29data
1:10:31breaks file system
1:10:34okay what's that that sounds fancy but
1:10:37what's that datab braak file system so
1:10:40basically let me give you a scenario so
1:10:44our data is in the Raw container as we
1:10:47know right and I'm telling you how you
1:10:50can access it but before that just think
1:10:52so this is in a container right this is
1:10:55in a container and this is the data
1:10:59L this is the oh my hand
1:11:04writing this is data
1:11:08Lake right this is data lake so if you
1:11:12want to access this data if you want to
1:11:15access this data you need to provide the
1:11:18URL you need to
1:11:21provide
1:11:23URL simple see
1:11:26to access this data because obviously
1:11:28you need to give the URL where the data
1:11:30is residing right in which container and
1:11:32which storage account so instead of
1:11:35providing that URL again and again dbfs
1:11:39that's datab Brak file system is
1:11:42basically a distributed file system
1:11:45which acts as an abstraction layer which
1:11:48acts as an abstraction layer what does
1:11:51it mean so it will create an abstraction
1:11:58layer like this
1:12:03see like this now this is an abstraction
1:12:08layer on top of the data L this is my
1:12:11abstraction
1:12:14layer see now this will create an
1:12:18abstraction layer so what does it mean
1:12:21instead of providing URL we can use file
1:12:26semantics let me repeat we can
1:12:30use file
1:12:33semantics what are file semantics so
1:12:37file semantics are similar to file
1:12:38system so instead of writing URL you can
1:12:44write something like this file
1:12:49store then your container let's say raw
1:12:53and then your let's file. CSV
1:12:56anything so instead of writing URLs you
1:12:59can use these semantics you can use
1:13:02these semantics that's called
1:13:05dbfs so you do not need to provide the
1:13:08full URS actually you can point your
1:13:10data to this thing but this is just an
1:13:12abstraction layer this is just an
1:13:14abstraction layer the actual data is
1:13:17residing in data Lake but it will act as
1:13:20it is residing in this particular folder
1:13:22in your data bricks notebook
1:13:25this is very handy because you do not
1:13:27need to provide URLs again and again and
1:13:29obviously you do not need to
1:13:32uh I think devote much time to find data
1:13:35as well because you can just go to file
1:13:36store folder and you can just get to
1:13:38know like where data is
1:13:39residing simple this is dbfs data break
1:13:43file system or distributed file
1:13:46system yes this was all about dbfs this
1:13:50is really important and yes when we are
1:13:53just reading data and we can and we want
1:13:56to just use the file systems or you can
1:13:58say folders We can just use file system
1:14:00dbfs it's your personal choice if you
1:14:02want to use uh like my responsibility is
1:14:05to just give you all the information I
1:14:07personally like using URLs I don't know
1:14:09why but yeah you have the uh option
1:14:13available to use these file semantics to
1:14:16read the data from the data bricks or
1:14:19let's say read the data from the data
1:14:21bricks but that data is residing in the
1:14:23data like actually it is residing in
1:14:25data
1:14:26Lake understood I hope you are just
1:14:29taking notes of these
1:14:31things yes yes yes just take notes and
1:14:35that's where we have another concept
1:14:37called mounting so what does that mean
1:14:40so we just Mount the external location
1:14:42and we we create the dbfs on top of it
1:14:45so that we can just use the same things
1:14:47do not worry we will discuss everything
1:14:49in detail so this was all about dbfs so
1:14:52now it's time to cover another another
1:14:55thing which is accessing the data from
1:14:59data breaks and data is in our data Lake
1:15:04okay this this is the major step and
1:15:06this is the step I think that you should
1:15:09be aware of if you want to be an outlier
1:15:12in the
1:15:13crowd yes because that is the sole point
1:15:17I have picked as your because you need
1:15:19to be proficient enough to actually
1:15:22perform the step so now let me just EXP
1:15:24explain you this with the simplest
1:15:26example or this with like how with the
1:15:29let's say simplest analogy so this is
1:15:32your datab Bri workpace right and we
1:15:33have a notebook created in this datab
1:15:35workspace and you want to read the data
1:15:37sitting in this data L and we have
1:15:39already created the container right how
1:15:42you can read the data sitting in this
1:15:45data Lake
1:15:46how you will say I want to read I know
1:15:49but this data is in
1:15:53Azure this data is in aure data Lake
1:15:56this data is secured not everyone can
1:15:59access it it is not available for
1:16:02everyone how this data briak workspace
1:16:05can read it let me tell you this is
1:16:09exactly same as we follow the steps in
1:16:12real life how let me tell you so let's
1:16:15say if you want to enter into a college
1:16:19bro from where we have included College
1:16:21just hold on because that's the simplest
1:16:23analogy so
1:16:25if you enter into a
1:16:28college not enter you want to enter like
1:16:31you want to just uh enter into your
1:16:33college or let's say to attend your
1:16:36class so this is your college and this
1:16:39is
1:16:40you this is you
1:16:44right and this college is not a public
1:16:48park similarly this data leg is not a
1:16:51public like data available for public
1:16:54this is
1:16:55secured and here they have a security
1:16:58guard
1:17:00right it's fun man but it's really
1:17:03helpful to for you to understand it and
1:17:06they have a security guard they have a
1:17:08stick as well in the hand
1:17:11okay so here they have a security guard
1:17:14you try
1:17:17to go there enter there and it stops you
1:17:22it stops you why he says
1:17:25show me your ID
1:17:27card he says show me your ID card and
1:17:29you say I do not have this ID card let's
1:17:34say you say this you do not have this ID
1:17:37card so it will ask you just go back
1:17:41just go
1:17:43back then similarly data braks will go
1:17:46to this data Lake data L will say hey
1:17:50show me your ID card show me something
1:17:52that you are eligible to read my data
1:17:55he will say I do not have
1:17:58anything he will say okay then go back
1:18:03then go back but but but this time he
1:18:09says I have this ID card this is a let's
1:18:13suppose this is an ID card he says hey
1:18:15this time I have an ID card then it will
1:18:18say okay you can read my data you are
1:18:21welcome understood so how we can get
1:18:24give this particular ID
1:18:26card to our datab break so we will
1:18:29include something called service
1:18:32principle
1:18:35oops service principle so we will
1:18:39include something called as service
1:18:41principle which is a kind of an
1:18:43application which is kind of an
1:18:49application which have the access to
1:18:52read this
1:18:53data which
1:18:55have the access to read this data this
1:18:58data this data this has the access this
1:19:00is an our
1:19:01application and how we will give this
1:19:03access I will show you there's a way to
1:19:05do that so once this has the eligibility
1:19:09or access to read this data we will use
1:19:11this key in our datab break workspace
1:19:14here and then we can access this data
1:19:17magic that's not magic these are like
1:19:19the credentials that you need to follow
1:19:21this is the thing that makes you an
1:19:23outlier because not not everyone is
1:19:25aware of how to perform this step and
1:19:28once you know that means you are
1:19:30learning a lot and you are on the path
1:19:34which is going towards be an
1:19:38outlier simple so let me just show you
1:19:40steps are really really easy really easy
1:19:42so there are like two steps involved
1:19:44first of all we will create this ID or
1:19:47application then we will allow the exess
1:19:51to read the data from the data Lake step
1:19:53number one is done step number one is
1:19:55done then in the step two this datab
1:19:58bricks
1:19:59workspace will import this credential
1:20:03that's it simple done and let me show
1:20:05you how you can do it everything can be
1:20:07done in aure data brakes so let me just
1:20:09go to my a datab briak like like in aure
1:20:12datab Bri and then you can just go to
1:20:15your aure portal click on it and just
1:20:19click on the home button so first of all
1:20:21we will go to Microsoft entra ID so simp
1:20:25you can just search it entra ID yep
1:20:28click on it and then after clicking on
1:20:31it just go to app registrations Simple
1:20:35app registrations now it will just show
1:20:38owned applications we do not have
1:20:39anything so just click on new
1:20:42registration simple and in this new
1:20:45registration you just need to name your
1:20:48application that's it you just need to
1:20:49name let's say I can put
1:20:51unch service principal let's say
1:20:57service
1:20:59principle simple and then you can just
1:21:01click on register button so now once it
1:21:04is done you can just go to all
1:21:06applications and you can just find your
1:21:08application that you have created so
1:21:10this is the one so just click on it it
1:21:12will give you the overview like
1:21:14everything so here are the things that
1:21:16we need to take care of what are those
1:21:19things let me tell you so first of all
1:21:21we need to copy these two informations
1:21:23application ID and object ID these two
1:21:26applications that's it so I will just
1:21:30use my all sorry just application ID and
1:21:32tenant ID these two applications this
1:21:35one and this this one like these two
1:21:38application ID and tenant ID you can
1:21:40just store it in your notebook or let's
1:21:42say in the notepad for now and then when
1:21:44we will be just jumping in the databas
1:21:46notebook we can just use this that's it
1:21:49so once you have copied this like you
1:21:52can just copy to clipboard and just
1:21:53paste it in the
1:21:55uh notepad simple so there's one more
1:21:58thing that you need to do you just need
1:22:00to go to certificates and secrets this
1:22:02one yep because we need to create the
1:22:04secret as well just click on it and when
1:22:07you click on it you just need to create
1:22:09a new secret client secret let's see
1:22:11anything like I already have some let me
1:22:13delete it
1:22:14quickly or just keep it like like I can
1:22:17just create a new one as well so once
1:22:18you click on it you can just name it
1:22:20anything like or provide description
1:22:22like let's say master class
1:22:25simple and it will be expired in 6
1:22:28months so click on ADD then when it will
1:22:31be added it's a very important step it
1:22:34will show the value instantly copy it
1:22:37because it will get disappeared after
1:22:38that and you have to create a new one
1:22:40then just click on this copy to
1:22:42clipboard that's it once you have click
1:22:45on once you have clicked on Co copied
1:22:47let me just go to the datab brakes
1:22:49workspace this one and just go to the
1:22:53recent step Tab and in the recent tab
1:22:56you should see your tutorial notebook
1:22:58just click on
1:22:59it simple and here we will use these
1:23:04values and as I have mentioned that you
1:23:07have you can just copy this information
1:23:09so I will directly copy it in my databas
1:23:12notebook instead of just copying it to
1:23:13the notepad but if it is in your notepad
1:23:16it's good so first of all I will just
1:23:18create a markdown because that's a kind
1:23:19of different heading so we know like how
1:23:21to create it percentage MD then I want
1:23:24want to keep it as H3 and I will say uh
1:23:28Access
1:23:29Data Access Data
1:23:32simple simple simple so I will just
1:23:35paste it here let's say secret
1:23:40is this one and let me just grab other
1:23:44two values as well I can just go to my
1:23:46service principal and this is my
1:23:47application
1:23:49ID so I can call it as app ID equals
1:23:53this and this is my tenant ID this is
1:23:57this 10
1:24:03ID
1:24:05simple yep simple so we have noted down
1:24:09all the values now we need to complete
1:24:11our first step we have created the
1:24:13application we have stored all the
1:24:15information of that application now we
1:24:17need to assign this application to
1:24:19access the data in data link how we can
1:24:21do that just simply go to a your tab and
1:24:23now you need to click on your home
1:24:25button now you need to click on your
1:24:28resource Group now you need to select
1:24:30your data storage account then click on
1:24:34containers and here just STP click on
1:24:38access control IAM click on this see
1:24:42these are the steps are missing these
1:24:45are the steps missing when we work with
1:24:47only databas Community Edition and then
1:24:51people say that they have they know how
1:24:54to work with data Bri but still they
1:24:56cannot clear interviews because they do
1:24:58not have end to end experience how to
1:25:00work with the cloud that's where
1:25:03recruiters just grab them like just stop
1:25:07here we know that you do not know this
1:25:09so that's why you should practice these
1:25:11things as well that's why I have picked
1:25:13a your just for you so that you can
1:25:15learn and you can Ace your interviews
1:25:17and I would love to hear from you all
1:25:20when you will say hey we have just uh
1:25:23qualified this interview View and we got
1:25:25selected I would be the happiest
1:25:27one so once you click on access control
1:25:30you just need to click on plus add
1:25:32because we are defining one new Access
1:25:35Control one new role click on plus add
1:25:38role assignment then you need to search
1:25:41a role called storage blob contributor
1:25:44storage
1:25:46blob data contributor this one we will
1:25:49just pick this one simply just search it
1:25:52again storage blob
1:25:55data contributor click on it and after
1:25:57clicking on it click on next then it
1:26:00will ask us which service principal you
1:26:03should provide the access we will say we
1:26:05want to select members okay and then we
1:26:08will search our service principal name
1:26:10so I will write my storage service
1:26:12principle so orang service principle
1:26:14yeah this one simply click on it
1:26:16obviously yours will be different so I
1:26:18will click on select and then you can
1:26:21just click on review plus assign so I
1:26:23will click on it
1:26:26and click on it for one more
1:26:28time and it will assign the role the
1:26:31role is assigned for but sometimes like
1:26:33it takes some time so do not need to
1:26:35worry now our first step is done and do
1:26:37you know second step is already done by
1:26:39datab Bri itself or let's say by
1:26:41Microsoft itself because obviously these
1:26:43things are available in documentation so
1:26:45I will simply jump on my datab notebook
1:26:48and now my this application ID D and ID
1:26:51secret that means this app my app app
1:26:55has the access it has the access now the
1:26:58Second Step was we need to import this
1:27:01in data braks and to do that we do not
1:27:04need to do anything we simply need to go
1:27:06on Google and let we can just write
1:27:08something as Access Data
1:27:11Lake using data bricks yeah click on
1:27:17it so you can just click on the
1:27:19Microsoft
1:27:20documentation simple so it will show you
1:27:23how you can connect you do not need to
1:27:25read this you do not need to read this
1:27:26we just need this code and the code is
1:27:29here this one simply copy do not read
1:27:32anything I have explained all the things
1:27:34in the easiest way see all these things
1:27:37are explained by your friend so do not
1:27:40need to worry so we need to just create
1:27:42another cell another code cell so I will
1:27:46just paste that code here and then we
1:27:49need to just replace these values these
1:27:52ones with our our tenant ID see these
1:27:57ones these
1:28:00ones why because this is a generic code
1:28:03and you can see the variables here
1:28:05storage account application ID we just
1:28:08need to provide this that's it our
1:28:10second step is already done by datab and
1:28:12Microsoft so thank you so much so first
1:28:15of all just remove this thing we do not
1:28:17need
1:28:19this then what we can do first of all we
1:28:22will remove the storage account variable
1:28:24storage account parameter that they have
1:28:26added so we will write our own storage
1:28:29account name and in my case it is Data
1:28:32Lake hch in your scenario your storage
1:28:34account name should be different and
1:28:37whatever you have picked right so I will
1:28:39just replace in all the
1:28:41places data Lake unch and
1:28:44[Music]
1:28:46then data Lake un and then here as well
1:28:50data l
1:28:54dat okay now second thing that we need
1:28:57to remove is this application ID this
1:28:59application ID and we have our own
1:29:01application ID so I will copy it and I
1:29:04will paste it
1:29:06here okay now we need to remove this
1:29:09service credential in case of service
1:29:11credential we need to put our secret
1:29:14just follow these steps carefully these
1:29:15are really really
1:29:17important so simply add double codes and
1:29:20then remove it simple then the last
1:29:23thing is direct ID this directory ID so
1:29:26for directory ID we will use tenant ID
1:29:29and we will just remove this and we will
1:29:32put it
1:29:33here
1:29:35simple just do these steps carefully you
1:29:38can just rewatch this part if you just
1:29:40make any mistake regarding double codes
1:29:42or anything just reat this part and
1:29:44carefully see what things I have
1:29:46selected see now I do not need to do
1:29:48anything I just need to run this let me
1:29:51run this it is done it is done
1:29:54now our this step is done this one this
1:29:58step is done now our data brakes is
1:30:01allowed to read this data it is allowed
1:30:05it is allowed so now I will just show
1:30:08you
1:30:10an
1:30:12important factor important thing within
1:30:15the data bricks that's called utilities
1:30:19that's called datab bre utilities I will
1:30:22write it for you
1:30:24that's called Data brakes
1:30:29utilities data
1:30:32brakes
1:30:35utilities so now it's time to discuss
1:30:38this because now we have all the exes
1:30:40with the data so now we can just run
1:30:43this and now we can just test this and
1:30:45now we can actually learn it practically
1:30:49so what are these utilities so we have
1:30:52some of the utilities available within
1:30:54data Pricks and very famous utility
1:30:59is
1:31:01is
1:31:04DB utils do FS what is
1:31:08this with this particular
1:31:11utilities we can get to know what are
1:31:14the files stored within the particular
1:31:19location let me repeat this particular
1:31:22datab breakes utility
1:31:24help us to know the files available
1:31:27within the particular location so if you
1:31:30want to see how many folders or how many
1:31:34files do we have under let's say Source
1:31:38container we can do this yes we can do
1:31:42this so this utility is really really
1:31:45handy so this is our first utility just
1:31:48take notes because we are discussing all
1:31:50the utilities one by one
1:31:55this is our first U let's do this so I
1:31:59will just write it here
1:32:01text and I will create a heading three
1:32:03and I will called
1:32:06DB utilities don't worry I will just
1:32:08provide this notebook as well that's why
1:32:10I'm creating so many headline oh not
1:32:13headlines like headings titles and uh
1:32:17subheadings so that you can easily
1:32:19navigate through it but still you should
1:32:21also create it so first of all our first
1:32:24database utility is let me create
1:32:26another heading and I will make it just
1:32:29bold and I want to call it as dbutils
1:32:36FS simple and I will run this let's
1:32:39discuss it so now we know that our
1:32:44source container remember we created One
1:32:46Source container and we uploaded one
1:32:47file I want to confirm that do we have
1:32:51that file in that container do we have
1:32:53some data in that container how we can
1:32:55do it we simply need to write dbutils
1:32:58do
1:33:00fs. LS now we need to give the location
1:33:07so there's a syntax that we follow
1:33:09within datab braks that syntax is really
1:33:11really easy to learn but if you do not
1:33:13need to learn if you do not want to
1:33:15learn you can just uh copy paste it from
1:33:17the documentation it is really really
1:33:19handy and available available there as
1:33:21well but as I have like loaded so many
1:33:24times so I remember that syntax and it's
1:33:26always good to remember that let me just
1:33:28zoom in a little bit and the syntax is
1:33:31oh they have given us the syntax you do
1:33:33not need to remember you do not need to
1:33:34go to any documentation data Brak is
1:33:36there for you he is just giving the
1:33:39suggestions see wow man such such a
1:33:42great tool so syntax is really really
1:33:44easy so what we need to do first of all
1:33:46we need to write a b FSS simple it is
1:33:49written on the screen as well but still
1:33:50I'm just explaining you as well because
1:33:52I want you to know each and everything
1:33:55the reasoning how and why and what
1:33:58behind everything so AB FSS is like your
1:34:01blob storage then colon then SL slash
1:34:06then we need to write the container name
1:34:09container name not storage account name
1:34:10container name our container name is
1:34:12Source put it then add the rate then we
1:34:16need to put the storage account name
1:34:18storage account name in my scenario it
1:34:21is Data Lake unch
1:34:24then rest of the things are same like
1:34:26DFS core do
1:34:30windows net simple and then I will write
1:34:33slash to know how many things do we have
1:34:35under this now I will simply run this
1:34:39now I should see my file oh okay we have
1:34:42an error so this error is related
1:34:46to that it is unable to identify this
1:34:50particular location so there are
1:34:52basically two reasons behind it either
1:34:55you have put wrong destination or let
1:34:58let's say wrong URL or your service
1:35:02credential has some errors that's why
1:35:05I tested this first and then I wanted to
1:35:09show you this because in the real life
1:35:11we do face these errors that our data is
1:35:15not accessible our data is not readable
1:35:17so what is the reason behind it let me
1:35:19show you this was intentional and let me
1:35:21show you how we can get rid of of this
1:35:24error because this is the real time
1:35:25example this is the real time scenario
1:35:27so did you find the error so that there
1:35:29was like no error so I intentionally
1:35:32what I did so I did not replace this
1:35:36with my storage account why did I do
1:35:38that so the thing is uh you should be
1:35:40aware of this tip and there are like
1:35:43many reasons behind the error that you
1:35:45can face like this because when I was
1:35:47learning and and I was facing this issue
1:35:49I didn't find anything I was like
1:35:51everything is right why why I'm not a to
1:35:53read the data so the thing is you just
1:35:55need to replace this variable as well
1:35:56and just make sure that you have
1:35:57replaced all the variables that's it and
1:36:00then I will write data Lake un here as
1:36:02well and when you write it I hope it
1:36:04should run fine so simply run it and
1:36:07then here I know that you know like what
1:36:11you should see exactly exactly exactly
1:36:14now it is giving us the file which is at
1:36:18this location abfs say Source at the
1:36:21data a and then
1:36:24it is giving us the name as well that is
1:36:26sales. CSV that's perfect as you know
1:36:30that this particular utility is used to
1:36:33know the files that we have under
1:36:35particular location now you can see
1:36:37sales. CSV file if we would have
1:36:39multiple files it would have given us
1:36:41multiple files if we would have multiple
1:36:43folders it would have given us multiple
1:36:45folders this utility is really handy and
1:36:49especially especially I personally use
1:36:51it to test my connections that
1:36:53everything is running fine like we can
1:36:55access the
1:36:56data how was it how was it good okay so
1:37:02now the second utility is it's dbutils
1:37:07do widgets widgets widgets widgets what
1:37:10is this what is this we need to know
1:37:12yeah I'm just telling you just just hold
1:37:14on let me just create a markdown for my
1:37:17widgets so it is DB dot vets simple
1:37:23simple what it does so in the real world
1:37:26we need to make our databas notebook
1:37:29parameterized in which we need user to
1:37:32give us the input and according to that
1:37:34we use that
1:37:36input did you get it let me repeat so
1:37:40let's say I want to create a parameter
1:37:42and I want user to provide me the input
1:37:45for this particular parameter how we can
1:37:48use it there's a function there's a
1:37:50utility called dbutils do widgets so to
1:37:54use it we simply need to write dbutils
1:37:56do
1:37:58widgets do
1:38:01text now there are many options like
1:38:04text and I think drop down multiple but
1:38:07for now I'm just explaining you with an
1:38:08example called text so what it will do I
1:38:10just need to give the parameter name
1:38:13let's say I want to name it as pcore
1:38:16name I want user to give me the name and
1:38:20then I can add comma and then I can
1:38:22Define the default default value for it
1:38:24let's say default value is unch let's
1:38:26say right then I can just run it and I
1:38:31just need to encode it because it's a
1:38:34name so I will just run this now let's
1:38:37see the magic did you see did you see
1:38:39what happened it has created a box for
1:38:43us it has created a box for us this is
1:38:46not a box this is a parameter so
1:38:49whenever we will be running this
1:38:51notebook user has the ability to give us
1:38:55the value and it if user doesn't provide
1:38:57us anything it will take unch it will
1:39:00take
1:39:02unch not un okay so like this is just an
1:39:06example so let me just show you how it
1:39:10works okay an you have told us that we
1:39:14can just Define the parameter but if
1:39:16what if user has defined here let's
1:39:19say what's your name what what should I
1:39:21write let me put my last name lamba so
1:39:25let's say user has provided lamba now
1:39:27this value is changed this value is
1:39:29changed now this is lamba right how I
1:39:32can pull this value so there is a
1:39:34function called dbutils do widgets.
1:39:37getet so instead of using text we will
1:39:41be using dotg instead of using tech text
1:39:45we will using doget so how we can just
1:39:47do it let's say I will write dbutils
1:39:50do widgets doget and then I just need to
1:39:54Define my parameter name which is p name
1:39:56simple and I can store this value in a
1:39:58variable called V like any variable
1:40:01simple now I will run this now I will
1:40:04run this this is done now I can use this
1:40:07variable anywhere anywhere in the
1:40:09notebook see I can just print it as well
1:40:11if I want to print it I will simply
1:40:12write where and I will run it it will
1:40:14give me lamba see I can use this
1:40:18variable anywhere in my notebook and I
1:40:20can make my notebook parameterized so
1:40:23interview question if you want to make
1:40:25your notebooks parameterized how you can
1:40:28how you can use it you can use db.
1:40:31widgets this is the kind of utility that
1:40:33we have within data
1:40:36bricks was it good was it good was it
1:40:39good yes now there's another utility
1:40:42that we really really important it's a
1:40:46little bit complex but this can be the
1:40:50game changer yeah because this is this
1:40:52is really important to know and I know
1:40:55this this utility is little bit hard to
1:40:57grab but once you understood it once you
1:41:01like uh understand the whole
1:41:04architecture like how things are working
1:41:06it is really really easy to remember and
1:41:08easy to implement as well and in the
1:41:11real world scenarios you cannot imagine
1:41:14working with data braks without this
1:41:16utility trust me so if this is so so so
1:41:21important you need to be double focused
1:41:23time and just turn off your all the
1:41:25notifications because this is like the
1:41:28next few minutes need lots of focus so
1:41:32what's that uity like we need to know
1:41:35the name so the name is let me make it
1:41:39bold so dbutils do
1:41:43Secrets as the name suggests as the name
1:41:46suggests that it is very handy when we
1:41:49are working with secret so the thing is
1:41:52in the real world situation ation in the
1:41:53reable scenarios when we are just
1:41:56working with different different
1:41:57environments such as Dev QA prod so we
1:42:01do not want to hardcode these values
1:42:03such as this one such as this secret
1:42:07obviously we do not want to show this
1:42:08value we as developers can see this but
1:42:10let's say we do not want to show this to
1:42:12admins we do not want to show this to
1:42:13other developers or Junior developers
1:42:15anymore so in that particular scenario
1:42:18we need to remove this value from here
1:42:22and we need to do something so that we
1:42:25can grab the value as well and we do not
1:42:27want to show the value as well so how we
1:42:29can do it so let me share you the
1:42:31architecture like how things work in the
1:42:34real world so when we working with Azure
1:42:37this is the reason I have picked Azure
1:42:39I'm saying it again and again because
1:42:40these things are really really important
1:42:44so what we do in a we have a resource
1:42:47called keyal keyal keyal key e y v a u l
1:42:51t let me write it for you so we
1:42:55have a resource called
1:42:59keyart Simple yeah we have a resource
1:43:02called keyal in Azure so in Azure we
1:43:06store this keyal that that is a resource
1:43:09and within this eal we store our secret
1:43:14we store our
1:43:15secret
1:43:19okay we store our secret so in data
1:43:22breaks
1:43:23we
1:43:24create a secret
1:43:28scope we create secret scope what does
1:43:31it mean so in data brakes we create
1:43:35something called as secret scope which
1:43:38links data breaks to this particular
1:43:42Secret by not hardcoding anything by
1:43:45directly connecting to this keyw this is
1:43:48so powerful that means your secret will
1:43:51be pulled directly from the key W
1:43:53instead of this notebook see this is so
1:43:56powerful and this is so so so important
1:43:59if you are working with the real world
1:44:00scenarios so how we can actually do it
1:44:03it is very simple just go to your Azure
1:44:05portal and you just need to Search
1:44:07keywords as your resource so it should
1:44:09pop up here simple so now obviously you
1:44:13do not have any key created so just
1:44:14click on create button from here or from
1:44:16here anything it's up to you so now we
1:44:19need to just pick the resource Group we
1:44:21can just pick our Resource Group just
1:44:23make sure you are picking the same
1:44:24Resource Group right now we need to just
1:44:26name it now I can just say um Master
1:44:31Class key
1:44:33Vault Master Class key or let's say
1:44:36vault as well Master Class key Vault
1:44:39unch okay okay okay I know the name is
1:44:42really really long but can't help sorry
1:44:45so now it is done now we can go to next
1:44:50and one thing to remember one thing to
1:44:52remember you need to pick Vault access
1:44:55policy instead of a your rbag just pick
1:44:58this one VA access
1:45:00policy and then you can click on review
1:45:03plus create
1:45:05simple and click on create button for
1:45:07one more
1:45:10time and it should just create that
1:45:12keyword it will take just like few
1:45:15seconds
1:45:17simple so actually it is creating that
1:45:20key wal keyal is really really handy
1:45:23when you want to store the secrets or
1:45:25like let's say any sensitive information
1:45:28any sensitive information right so it is
1:45:30done we can just go click on go to
1:45:32Resource and now we can create something
1:45:35called as Secrets as mentioned in the
1:45:37architecture right now click on
1:45:39secrets we will create one and and and
1:45:43hold on hold on hold on if if you want
1:45:46to create Secrets if you want to do
1:45:48anything you just need to first of all
1:45:51go to here Access Control I am first of
1:45:53all go here and you need to assign
1:45:56yourself as keyal admin because you need
1:45:59to be an admin administrator to create
1:46:02any secret go here and click on plus add
1:46:06add Ro assignment and then you will pick
1:46:08a role called
1:46:10keyal admin see keyal administrator
1:46:14simple click on next then you need to
1:46:17pick the users okay then you can just
1:46:19pick your email ID click on select click
1:46:22on review Plus sign simple click on R
1:46:24plus sign for one more
1:46:26time done it is added now we can create
1:46:29the secrets now we can just click on
1:46:30this button Secrets click on it and
1:46:33click on generate plus import now we
1:46:35will just name our secret let's say I
1:46:37want to call it as app
1:46:39Secret app secret because we are just
1:46:42storing the application ID secret that
1:46:44is service service principle I'm just
1:46:46showing an example okay and then secret
1:46:48value I can just grab it from
1:46:51here uh uh uh
1:46:54okay I can just grab this value from
1:46:57here and this is my value done and now I
1:47:02can just click on create button so now
1:47:05what we have done we have stored our
1:47:07secret we have stored our secret our one
1:47:11like half part is done now we need to
1:47:14link our databas account to this
1:47:16particular Secret by creating secret
1:47:18scope how you can create secret scope in
1:47:21data brakes so there is a tip for you
1:47:23just go to Home tab and then just search
1:47:27your datab breaks workspace by simply
1:47:29searching datab breaks like in the name
1:47:32of your datab workspace then copy this
1:47:34URL copy this URL which is the URL of
1:47:37your databas workspace like paste it in
1:47:39your tab Google tab or any tab then just
1:47:43focus on this area you need to add
1:47:47hashtag
1:47:49Secrets then
1:47:51slash create create
1:47:54scope this is the thing that you need to
1:47:56add after you after your url # secret SL
1:48:01create scope C in lower case S in
1:48:05uppercase simply hit
1:48:09enter okay so now we can create the
1:48:12scope now if I want to create a scope I
1:48:14need to name it I can
1:48:16say
1:48:18uncope simple like anything and manage
1:48:21principle you can pick any like all
1:48:23workspace users so leave it like this
1:48:25now we need to provide DNS name and
1:48:27resource ID in order to provide the DNS
1:48:29name and resource ID we need to go to
1:48:32our Azure keyw and we need to grab DNS
1:48:36name and resource ID from there because
1:48:39we are linking our data bre secret scope
1:48:41with the Azure keybo so just go here
1:48:44like Home tab and click on keywords and
1:48:48then click on your keyal resource and
1:48:51then you need to just go to the proper
1:48:54simple and here you have everything the
1:48:56VA URI is your DNS
1:48:59name VA URI is your DNS name and your
1:49:03resource ID is your resource ID
1:49:06simple simple simple now click on
1:49:09create so it should show that it is
1:49:11created yes it is created now everything
1:49:13is done now everything is done now I
1:49:15will show you how you can use it simply
1:49:18go to your databas workspace now we have
1:49:21some functions available in dbutils do
1:49:24secrets so now it will show us some like
1:49:27Secrets let's say I want uh let's say
1:49:30let's say I forgot like what secret I
1:49:33have stored in a your keyboard so it can
1:49:35show us that as well we can write
1:49:37something known as DB u. Secrets dot
1:49:42list and then I can just enter my scope
1:49:46and my scope
1:49:48is uncope and I will just run it
1:49:54I will just run it and
1:49:57then it should show me that hey anch you
1:50:02have a key called app Secret in my
1:50:05keyboard and that's correct that's
1:50:07correct that's what I want and if I want
1:50:11to access the value of this key I can
1:50:13use it what I can do I can just run a
1:50:15command called
1:50:17dbut do secrets doget doget and this
1:50:22time I need to say scope equals to un
1:50:25scope and scope equals to un
1:50:29scope and key equals
1:50:32to that name and name is app hyphen
1:50:38Secret app hyphen secret that's all
1:50:42that's all oops I have used single code
1:50:46yep perfect that's all that's all when I
1:50:49run this see see the output it will say
1:50:53redacted that means now we can use this
1:50:56value when I write this thing instead of
1:50:59that secret it will give me the value
1:51:02redacted means here that your value is
1:51:04pulled and is secured it is not visible
1:51:08but it is secured and it is pulled
1:51:10safely now you can use this anywhere so
1:51:14now I can just use this
1:51:16code instead of hard coding this thing
1:51:20let me just show you instead of hard
1:51:22coding this thing this secret I can just
1:51:24use this thing this is the power of db.
1:51:29secret command this is my one of the
1:51:31favorite utilities and yes it is little
1:51:33bit complex but not really it was so
1:51:36simple it was so simple you just need to
1:51:37watch this part at least two to three
1:51:39times and please watch it for me because
1:51:42you will gain a lot of knowledge and
1:51:44just practice it just practice it for
1:51:45like one to two times this is so so so
1:51:49handy I know you love this part I know
1:51:52it because you have gained so much of
1:51:53knowledge after this thing I know it was
1:51:56difficult but if you do difficult things
1:51:58then then then only you will be
1:52:00competitive enough right you need to do
1:52:03difficult things you need to do
1:52:04difficult things so now we have
1:52:07successfully covered all the DB
1:52:09utilities now it's time to cover the
1:52:12next chapter of our databas master class
1:52:16which is data reading from your cloud
1:52:19storage let's cover that quickly so
1:52:23let me create a markdown percentage MD
1:52:27and I want to use heading one
1:52:31Y data
1:52:33rating so now we going to
1:52:36discuss spark reader API do not get
1:52:40afraid of this name this is just a name
1:52:42but this is a function or this is very
1:52:45handy when we need to read data like
1:52:48which is residing in the data L and as
1:52:50we just discussed that we always create
1:52:52data for on top of it so how we can do
1:52:54it we will simply create a data frame
1:52:55called DF let's say
1:52:58sales right and we will write spark do H
1:53:03datab breaks we know that you know a lot
1:53:06of things do not recommend us right now
1:53:08because I'm teaching them I'm teaching
1:53:11my community and they should know each
1:53:13and everything just hold on this is
1:53:16suggesting us like everything so let me
1:53:19tell you so first of all we need to
1:53:20write like we need to write we need to
1:53:24write spark dot
1:53:28read we need to write spark. read this
1:53:31is our spark reader API when we read
1:53:35this when we write this that means we
1:53:37are trying to read some data so then we
1:53:39will specify what format of data we need
1:53:42to read in our case it is
1:53:47CSV it is CSV simple yes what is the
1:53:52next thing that we need to decide first
1:53:53of all here's a tip for you if you do
1:53:55not want to write your all the code in
1:53:57one line you can simply add backslash
1:54:00and then hit enter then write your code
1:54:02here because it is a kind of python
1:54:04function that you use with multi-line
1:54:07commands and you can use it here as well
1:54:08so don't need to worry now bro you
1:54:12are so next thing next thing next thing
1:54:15next thing yeah it has already provide
1:54:17the spoiler so next thing obviously we
1:54:20need to put the header because we need
1:54:23our data with header not without header
1:54:27we need with header we need to treat the
1:54:28first line as header and how we can do
1:54:30it this this application has already
1:54:34shown to you so it is do
1:54:37option then you need to write header
1:54:41then comma then write true what it will
1:54:44do it will make your header true that
1:54:46means your first line or first record
1:54:49will be treated as header simple very
1:54:51good
1:54:53then next thing is infer schema what is
1:54:56this infer schema let me tell you Pro
1:54:59just just just just hold
1:55:03on so what is that so it is called infer
1:55:08schema what is this infer schema so
1:55:11basically whenever we read data from
1:55:13spark we do not want to Define schema
1:55:17every time so infer schema function or
1:55:19let's say this API helps us to get the
1:55:22schema for free not for free so instead
1:55:25of manually defining it again and again
1:55:27spark reads some records from our data
1:55:31and decides the best schema for it that
1:55:34means columns and their data types the
1:55:36best possible combination it
1:55:38automatically picks simple it is really
1:55:41powerful and it saves a lot of time I
1:55:43use it like most of the time so you can
1:55:46also use it so how we can use it do
1:55:49option then we will write infer schema
1:55:52just make sure that you are using the
1:55:54same case sensitive uh letters so do not
1:55:58need to worry just write I small and S
1:56:00capital then add comma then write true
1:56:03we will make it true as well
1:56:06then back
1:56:09slash why it showing us red here I don't
1:56:11know ah we we will see okay that's all
1:56:15now that last thing is we need to define
1:56:18the URL we need to define the URL where
1:56:21our data is setting it is sitting in
1:56:23this URL and we have already uh seen
1:56:26this we need to write load and we will
1:56:29write AB
1:56:31bfss back SL back like sorry single
1:56:34slash like front slash front slash then
1:56:36we need to write our container name
1:56:39container name is
1:56:41Source atate my storage account name
1:56:45data Lake
1:56:46on dot DFS do core
1:56:53do windows.net
1:56:57simple so we can Define sales. CSV as
1:57:01well but even if we do not Define it
1:57:03will automatically pick it so database
1:57:05we do not want to use sales. CSV it will
1:57:08automatically pick it okay so this was
1:57:11all about data reading so when I just
1:57:14run this it should just read my data as
1:57:17you can see our data is successfully
1:57:20loaded not loaded but yeah it has read
1:57:22the data so now you already know how we
1:57:25can display the data frame we can use DF
1:57:27do
1:57:29display simple yes so now it will show
1:57:32us DF is not defined oh because we are
1:57:35using DF Sales instead of DF okay thank
1:57:38you so
1:57:40much so it will just give us the data
1:57:42frame so this is our data frame that we
1:57:44are currently working with so this is
1:57:46our data frame so you have successfully
1:57:49read the data sitting in data Lake and
1:57:54you have already learned all the
1:57:55utilities everything so congratulations
1:57:59you have learned a lot in the world of
1:58:01data bricks but lots of stuff are still
1:58:04pending so we will be covering but still
1:58:06you should feel like you should you
1:58:08should you should feel good bro because
1:58:10you have learned so much and just few
1:58:12minutes back you didn't know anything
1:58:14but now you are feeling so much so much
1:58:16like so confident so that was all about
1:58:19data reading how you can just read the
1:58:21data and let me tell you even if you
1:58:23have like different different data file
1:58:25formats let's say you have CSV let's say
1:58:27you have Json let's say you have pocket
1:58:29let's say you have Delta file format you
1:58:30can read any any any file format like
1:58:34this so that was all about data reading
1:58:39and now here I have a gift for you
1:58:42what's that so if you want to learn p
1:58:47park only PP Park let me write it for
1:58:49you what do I mean if you want to
1:58:56learn ppar what is ppar basically ppar
1:58:59is a library that is used to transform
1:59:02our data right if you want to just apply
1:59:05some transformation I will just show you
1:59:06some of the functions but not all the
1:59:08functions but yeah some of the functions
1:59:10I will just show you how you can do it
1:59:12and if you want to learn ppar in detail
1:59:15in detail means genuinely in detail you
1:59:17can just click on the video that I have
1:59:20created and this is a six hours long
1:59:22video dedicated towards p park only so
1:59:26this is a full course of p park in which
1:59:28I have covered each and everything
1:59:30regarding ppar from scratch and
1:59:33definitely if you want to become a ppar
1:59:35developer which is obviously a
1:59:37requirement if you are working with data
1:59:38brakes so you should just watch that
1:59:40video and do not need to worry you can
1:59:42just click on the link just click on the
1:59:44like button so that it will go to your
1:59:46liked videos or you can just save it by
1:59:48any way so just save that video and you
1:59:52can still watch this video because this
1:59:54video is independent of that one I'm
1:59:56just giving you a gift that is a free
1:59:58course that you can do without paying
2:00:01anything so after watching this databas
2:00:04video you should obviously watch that
2:00:06video because you will feel much more
2:00:08confident because you will know all the
2:00:09functions available in ppar so let me
2:00:12show you some of the functions here so
2:00:14that you will get the idea and do not
2:00:16need to worry these functions I will
2:00:18just tell from scratch but yeah we will
2:00:20be covering we will be covering just
2:00:21like two to three questions or let's say
2:00:23two to three functions because we are
2:00:24learning data breaks only but if you
2:00:27want to learn P spark in detail
2:00:29definitely you can just watch this video
2:00:30so without wasting in time let me just
2:00:32show you some of the powerful ppar
2:00:34transformation that we have so let's
2:00:38say I want to I want to create a list
2:00:44what do you mean create a list so as you
2:00:46can see that we have dairy we have soft
2:00:49drinks we have this we have fruits and
2:00:52vegetables we have household we have
2:00:54baking Goods if what if I do have a
2:00:57requirement in which I want to separate
2:01:00these values by comma and I want to use
2:01:04this space as a delimeter that means
2:01:06whenever I see a space I should treat
2:01:08that value as a different value and I
2:01:11want to store all the values in a list
2:01:14for each record this is my requirement
2:01:17this can be a requirement so if we want
2:01:19to do this we can just do it so I'm I'm
2:01:21just showing you two to three
2:01:22transformation functions within ppar
2:01:25that you can learn and definitely that
2:01:26complete video will teach you each and
2:01:28everything do not need to worry and this
2:01:29function obviously I will just tell you
2:01:31from scratch so you can still watch and
2:01:34this video is independent of that one do
2:01:35not need to worry but when you watch
2:01:38both the videos you become a pro ppar
2:01:42developer and obviously a datab brakes
2:01:45developer and both goes hand in hand
2:01:48five spark data break goes hand in hand
2:01:50so it's good it's ideal to watch both
2:01:54the videos so that's why I have created
2:01:56both the videos for free and obviously
2:01:59I'm just I'm just helping you helping
2:02:00you to grow so
2:02:04let's let's mention it markdown okay and
2:02:09let me just write heading three and I
2:02:11will call it as bpark
2:02:16Transformations Transformations okay
2:02:19perfect bpar Transformations let me show
2:02:21you transform number one so if I want
2:02:26to let me keep it as DF Sales because
2:02:28I'm not saving this I'm just showing you
2:02:30so DF Sales we have a function called
2:02:33with column dot with column so I will
2:02:35write DF do with column so what we can
2:02:38do with this particular function we can
2:02:41either transform the current column or
2:02:44we can create a new column so what if I
2:02:46want to transform the current column
2:02:50because I want to create a list yes
2:02:52so if I want to transform the current
2:02:53column I will just put the same column
2:02:55name and what is my column name here
2:02:58it's item type okay so I will simply
2:03:00write
2:03:02item type okay then I will uh add a
2:03:06comma then I need to put the
2:03:08transformation that I want to apply the
2:03:10transformation I want to apply is split
2:03:13we have a function called
2:03:17split we have a function called split so
2:03:20what it will do it will split my values
2:03:23based on the space and it will store
2:03:25every single value whenever it sees a
2:03:28space so I will write
2:03:32split okay and then I will write column
2:03:35so column is basically an object that we
2:03:37use with the column name so I just need
2:03:39to write item
2:03:43type okay column item
2:03:46type and
2:03:48then this Auto suggest I need to turn
2:03:51this off man so I have successfully
2:03:54given this item type right so now we
2:04:00need to also tell split function that
2:04:02what should be the delimeter what should
2:04:04be the uh delimeter on which it should
2:04:06create a list in our scenario it is
2:04:09space so I will just simply write column
2:04:12and this then it's
2:04:15space simple then I can just close this
2:04:17function simple that was all about this
2:04:21split function I can
2:04:23just close this function as well and
2:04:27then I can say do display then I can say
2:04:31dot display so when I run this command I
2:04:34should see the results as desired but
2:04:40before that I will just import some
2:04:42libraries yes if we want to perform
2:04:44Transformations we need to import some
2:04:46libraries and what are those libraries
2:04:48let me just show you simply add a code
2:04:49cell above above to this cell and I will
2:04:52I will simply write
2:04:53from pyspark.sql
2:04:56import estx and basically I want to
2:04:59import all the
2:05:00functions so simply I will write
2:05:02functions and there's another library
2:05:04from
2:05:06pyspark.sql do types so it will import
2:05:10all the data types so I will first run
2:05:12this then I can just run this and you
2:05:14will see my data frame column is
2:05:17transformed I'm sure yeah it is
2:05:19transformed see now it has created a
2:05:22list now it has created a
2:05:25list based on every single value this
2:05:29was transformation number one and what
2:05:31is transformation number two let me show
2:05:32you what if I want to add one column
2:05:37which will write one one one one one one
2:05:40or let's say yes yes yes yes yes yes in
2:05:42every record this is my requirement this
2:05:45is a kind of flag column that I want to
2:05:47create if I want to create it is very
2:05:50simple so so I can simply write here DF
2:05:56Sales and why I'm using this
2:05:57transformation because this will also
2:05:59use our our that particular uh utility
2:06:04that we have created that is tb. widget
2:06:08yes it will use that particular widget
2:06:11as well so how we can do it as you
2:06:13remember that we have stored that value
2:06:16which is where if I just print
2:06:19this oh where is not defined let me just
2:06:22rerun
2:06:23it where is that utility yeah this one
2:06:27if we run this it returns lamba you
2:06:31remember it right if we want to give the
2:06:34name like if if I want to just put lamba
2:06:38as like in in in front of like every
2:06:40record in the each record I can use this
2:06:43variable I can use this
2:06:46variable yes I can use this variable in
2:06:49my column and how we can do it let me
2:06:51just show you
2:06:52you let me just show
2:06:55you so when we just run this you should
2:06:58see lamba so how we can do it so we can
2:07:00simply write DF Sales dot with
2:07:04column and this time we are creating a
2:07:06new column so I can just name it
2:07:08anything let me call it as
2:07:10flag and then I will
2:07:12say lit there's a function called lit
2:07:16lit lit lit function helps us to provide
2:07:20the literals or let's say constants to
2:07:22our records so I will say let and I will
2:07:24write where simple now let me just write
2:07:29dot display
2:07:31command now I should see that column now
2:07:34I should see that
2:07:35column yes I have that column with lamba
2:07:39mentioned in the each record you can
2:07:42give like any
2:07:43value this is the transformation number
2:07:47two here is the transformation number
2:07:49three do not worry I have discussed each
2:07:51and every transformation in detail and
2:07:53from scratch in that video so you can
2:07:55definitely watch it after watching this
2:07:56video because you need to just focus on
2:07:58this part first because we are just
2:08:00trying to hit that Delta Lake and that
2:08:02is really really important so here is
2:08:05the transformation number three what's
2:08:07that let me just show
2:08:09you what
2:08:11if as you can
2:08:14see this
2:08:16particular number
2:08:201.2 let's say I want to change its data
2:08:24type I want to cast this that's called
2:08:26type casting what if I want to change
2:08:28the type of this data type of this
2:08:30particular column I want to make it
2:08:32string that's my requirement maybe
2:08:34that's a kind of joining key that we
2:08:36have in the data frame so I want to make
2:08:38it string how we can do it let me show
2:08:40you so for that we will simply write DF
2:08:46Sales dowith column and as you know that
2:08:49we are just transforming the current
2:08:50column so we will write the same name
2:08:52that is item
2:08:56visibility simple and then we have a
2:08:59function called first of all we will
2:09:00write the column name using call object
2:09:03item
2:09:05visibility item
2:09:09visibility oops
2:09:11visibility we have a function called dot
2:09:15cost dot cast it will help us to cast
2:09:19the current data type of the column so
2:09:22if we want to convert into string type
2:09:24we need to call the string API that is
2:09:26called string
2:09:29type string
2:09:31type and then when I just write do
2:09:35display it should return me that column
2:09:38with string data type and how we can
2:09:41confirm it as you can see I can see ABC
2:09:44instead of 1 23 or 1.2 something that is
2:09:47fla so its data type is also changed so
2:09:52these are some of the P spark powerful
2:09:55Transformations that we have and you
2:09:57should know all the Transformations but
2:10:00for this particular Master Class these
2:10:02Transformations are enough and obviously
2:10:05after watching this master class you
2:10:06should watch that pbar Master Class as
2:10:08well that is pbar full course but for
2:10:10now it is enough because we are focusing
2:10:14more on data brecks right yes and
2:10:18promise me you will watch that video
2:10:20trust me you will be like having so much
2:10:23of professional skills after that so
2:10:26that is all about our data reading data
2:10:29U is now it's finally time to work with
2:10:33Delta Lake and for that you should know
2:10:37the fundamentals of Delta lake so let me
2:10:38tell you quickly what is a Delta Lake
2:10:41what is a Delta format and why there's
2:10:43so much craze of Delta Lake let me show
2:10:46you each and everything so finally it's
2:10:48time to discuss this OG Delta lake so
2:10:52why we have so much of craze of Delta
2:10:55Lake like why why why by the way I'm a
2:10:58fan of Delta l so let's talk about it so
2:11:00basically Delta lake is not a file
2:11:02format no it is a file format but in an
2:11:07indirect way not a direct way like we do
2:11:10not have something called dot Delta no
2:11:12no no no no so don't worry everything
2:11:15will be cleared in just few minutes
2:11:16let's say let's create a hypothetical
2:11:19situation and you will understand the
2:11:21whole concept very easily with that okay
2:11:25so let's say you have a source you have
2:11:30a source folder right let's call it as
2:11:33Source okay and in this particular
2:11:36Source you have a file in Pocket format
2:11:40so if you have not heard about pocket
2:11:42file format pocket file format is a file
2:11:45format like like like we have CSV file
2:11:48format GSM file format fet file format
2:11:51so basically pocket file format is one
2:11:53of the file formats available so this is
2:11:55one of the file format so what's special
2:11:57with this particular file format this is
2:11:59a columnar based file format which is
2:12:01designed which is efficient for faster
2:12:04reads which is efficient for Big Data so
2:12:07everyone is aware of pocket file formats
2:12:09for now right I for those who are who
2:12:12are not aware of I have just G I have
2:12:14just given a quick overview of that
2:12:16paret file format so pocket file format
2:12:18is one of the file formats available and
2:12:21it is a columnar based file format and
2:12:23it is efficient enough for like faster
2:12:25reads let's say I have a data in Pocket
2:12:29file format and it's called Data do
2:12:34pocket simple y simple so this is
2:12:39residing in this folder now I have a
2:12:42destination folder
2:12:45right let's say destination folder
2:12:48simple now when I am moving the data
2:12:51from source to destination I want to
2:12:54change its file format I want to store
2:12:56this data
2:13:00in Delta file format but you just said
2:13:03like Delta is not a file format yeah
2:13:05wait wait hold on hold on I want to be I
2:13:08want data to be in Delta format in
2:13:11destination what will happen the same
2:13:15data. pocket file
2:13:18the
2:13:19same data
2:13:21dot pocket file will be written in this
2:13:26destination bro are you making a fool of
2:13:29us if it is the same why we are calling
2:13:31it Delta bro because along with this
2:13:35file format we will get something called
2:13:39as transaction log yes and another
2:13:44folder don't worry I will show you
2:13:46everything practically this is the
2:13:48fundamental you should be aware of we
2:13:51will get something called as Delta
2:13:55log SL transaction
2:13:58log simple you will get something called
2:14:01as Delta log SL transaction log okay
2:14:06what is this and why do we need it
2:14:10basically this transaction this Delta
2:14:13log is so powerful it's so so so
2:14:16powerful it will store each and every
2:14:20information of the data within this
2:14:22folder within this folder whereas in
2:14:26Pocket file format every information is
2:14:30stored within the file itself so let's
2:14:33say if we have multiple files let's say
2:14:35we have one 2 3 4 5 up to 100 so if it
2:14:42needs to read its metadata if it needs
2:14:46to read its schema it needs to go to
2:14:49Every individual files again again and
2:14:51again and it uses so much of computation
2:14:54bro just hear this carefully because
2:14:58this is really really important and I
2:15:02myself have has like read so many so
2:15:05many books to just conclude this
2:15:07information so please be focused now so
2:15:11if it needs to read all the pocket files
2:15:15it needs to read the metadata of
2:15:17individual files again and again just
2:15:21imagine if we have thousands of files it
2:15:23will be reading metadata of individual
2:15:25files but in case of transaction log in
2:15:29case of Delta log it do not need to read
2:15:33the
2:15:34Met I'm so thirsty I'm just speaking
2:15:37non-stop so it will read data not from
2:15:41the file it will not read the metadata
2:15:44from the file it will read metadata from
2:15:47this transaction log which will save a
2:15:50lot of lot of time and let me tell you
2:15:54this is just one of the powerful
2:15:56features available in Delta L because we
2:15:58are just covering all the features so
2:16:00this is just the basic one this is the
2:16:03basic
2:16:04one so can you imagine how powerful it
2:16:08can be with Delta Lake Delta format so
2:16:11actually file is same it is in paret
2:16:15format but we get Delta log we get
2:16:18transaction log on top of of
2:16:22it understood now I will show you
2:16:26practically how you can create a Delta
2:16:30log while Transforming Our like
2:16:32transferring the data because in our
2:16:34scenario our data is in CSV format we
2:16:38will store this data in destination
2:16:40folder in Delta format I will show you
2:16:43the pocket file I will show you the
2:16:46Delta log as well and then when you
2:16:49actually see it with your eyes you will
2:16:51believe it and then we will start
2:16:55working with all the functionalities of
2:16:58Delta Lake in detail but this is the
2:17:00basic one let's uncover this let's
2:17:03uncover this so we are into our datab
2:17:06bre notebook so now I want to create a
2:17:09separate notebook for Delta l so let's
2:17:12do it so simply go to file and click on
2:17:15new notebook and we will name it as
2:17:18Delta L let's name it okay it's created
2:17:21let's name it as Delta
2:17:23L simple now I would also like to show
2:17:28you one important magic command which is
2:17:32percentage run so you would have noticed
2:17:36one thing let me just go to workspace to
2:17:38or yeah just let's go to works let me
2:17:40just show you one thing so if we go to
2:17:43our workspace which is data breakes
2:17:46Master Class if I want let me just
2:17:48refresh because I should show I should
2:17:50see two
2:17:51here oh I got it so in order to create
2:17:56notebook in this particular folder we
2:17:58need to just create here let's say
2:18:00create one create notebook so we will
2:18:02just create Delta Lake in our this
2:18:04particular
2:18:05workspace because we want to store our
2:18:08Delta Lake notebook within the same
2:18:10workspace because I want to show you one
2:18:12important command okay let's say we are
2:18:14in this workspace right let me refresh
2:18:17now it should show me Delta Lake
2:18:21okay I'm not interested in this one
2:18:23which is created in home I can just
2:18:24delete it move to trash because it is in
2:18:28home it is not available in any kind of
2:18:31workspace but this is available in
2:18:32workspace C so in this particular
2:18:36tutorial notebook we have configured
2:18:40that uh service principle to access data
2:18:43in the data Lake right so whenever we
2:18:46create a new notebook we have to do it
2:18:48again there are two options you can also
2:18:51copy and paste it obviously you can copy
2:18:53and paste it but what if you have
2:18:56something in your code let's say you
2:18:57want to use all those things that you
2:19:00have created in this notebook in this
2:19:02particular new
2:19:03notebook what if so we have a command
2:19:08called percentage run we have a command
2:19:12called percentage run so what it will do
2:19:16it will just run our previous or not
2:19:20previous let's say we can just run any
2:19:22any notebook so it will just run the
2:19:26notebook uh mentioned in our location
2:19:30and it will just import that notebook
2:19:33for us so let's say in this Delta Lake
2:19:36notebook it will import this notebook as
2:19:38well this is really really powerful
2:19:40command and I love to use it so let me
2:19:43show you how you can use it it is very
2:19:44very simple just one line of code you
2:19:46just need to take care of this thing
2:19:47that is your workspace name that is data
2:19:49breaks Master Class and your uh notebook
2:19:52name so we will simply go to this
2:19:54notebook and we have already renamed it
2:19:56second thing we need to attach it to our
2:19:58cluster simple then we will write this
2:20:01command and we will write percentage run
2:20:04and then we need to say first of all
2:20:07slash datab
2:20:11bricks master class and then you need to
2:20:15give the notebook name simple when you
2:20:17run
2:20:18it that means your whole notebook is
2:20:22imported you do not need to perform
2:20:24anything like anything means anything
2:20:26see you do not need to perform any
2:20:28utilities you do not need to perform
2:20:29anything anything so everything is done
2:20:31you do not need to configure anything
2:20:32again so we are good to read the data or
2:20:35anything we want to do so this was
2:20:36really really handy I wanted to show you
2:20:38so so one more important thing that we
2:20:41do not need to read any data frame again
2:20:44we do not need to just read dfdore CS
2:20:47again because otherwise we would have
2:20:48read it again so this this is very
2:20:50powerful command so let me just create a
2:20:52heading here and I will
2:20:56say heading one and I will say Delta
2:21:00Lake OH Delta Delta Lake okay so first
2:21:04of all as I promised that first I will
2:21:05just write my data which is DF Sales to
2:21:08destination container and I will show
2:21:10you the Delta log and pocket files yes
2:21:13let's do that how we can do it there's a
2:21:15command slash there's a data frame
2:21:18writer API which is called DF
2:21:22WR simple DF do write and in our case it
2:21:25is dfdore sales good catch so dfdore
2:21:30says. WR then again the same step we
2:21:32need to specify the format of the data
2:21:35and in this scenario it is Delta it is
2:21:38Delta simple then we need to specify the
2:21:41mode now what what is this mode so
2:21:43basically we have four types of modes
2:21:46let me quickly discuss that as well
2:21:47because that is really really important
2:21:49so when we write the data we get four
2:21:52modes number
2:21:54one number one is
2:21:56upend okay number two is
2:22:01overwrite number three
2:22:04is uh
2:22:07error number four is
2:22:11ignore simple so what does append do so
2:22:16basically whenever we want to save our
2:22:19data in this container which is our
2:22:20let's say destination container and if
2:22:23there is an already file sitting there
2:22:25any any file so what it will do it will
2:22:28add this file to that particular
2:22:31container simple this is this is very
2:22:33very calm function this is very Cal like
2:22:36it will not create any Kos or it will
2:22:39simply write our data and it will just
2:22:42add that data to the existing data but
2:22:45this overwrite command will go here and
2:22:49will say
2:22:52who the hell are you sitting here who
2:22:55the hell are you just go from here it
2:22:58will just overwrite the data it will
2:23:00just remove this file and it will save
2:23:03this file and it will overwrite it
2:23:05simple now what is this error function
2:23:08like error mode so if this file is
2:23:11already here then it
2:23:14will not write anything it will go and
2:23:17check if some file is there it will not
2:23:19write anything it will throw an error it
2:23:22will say hey there is an
2:23:26error that means there's file sitting
2:23:29there so if under any circumstance you
2:23:33do not want to add any data if any data
2:23:35is there you can use this mode then the
2:23:38last one it is very very very good so
2:23:41what it will do it will go here it will
2:23:45see hey if there is any file setting
2:23:48there obviously I will will not write
2:23:51anything but at the same time I will not
2:23:53throw any error as well no I will simply
2:23:57go and check if any file is there that
2:23:59means no file need to be written plus do
2:24:02not need to throw any error do not need
2:24:04to disturb anywhere and anyone so this
2:24:07was like all about the four modes So
2:24:10currently we can use any mode so let
2:24:12let's use upend mode for now okay simple
2:24:16then we need to say path this is the
2:24:19most important step we need need to say
2:24:21path so we can just use path using
2:24:23option function then path and within
2:24:27this path we need to define the path so
2:24:29what is the path AB FSS then our
2:24:33container name is
2:24:35destination at theate storage account
2:24:38which is Delta
2:24:41lunch. DFS
2:24:43doc. windows.net
2:24:47simple and let's create a folder yes
2:24:49let's I want to create a folder within
2:24:51this container then I want to say do sa
2:24:55when I run this command when I run this
2:24:58command I will show you Delta log
2:25:00created and data is saved in the pocket
2:25:02format let me run it let me run
2:25:07it okay I can see an error what's the
2:25:11error storage account invalid
2:25:14configuration H oh I just mistakenly
2:25:18what Delta Lake an it's data Lake
2:25:22data Lake oops data Lake an let's see if
2:25:27it is right or wrong because I don't
2:25:29remember the
2:25:32name yeah it should be
2:25:34fine yeah it's the name dataon yeah so
2:25:38now it is writing our data in the Delta
2:25:40format and yes I can see that it has
2:25:43successfully return the data now I am
2:25:47very excited to check my destination
2:25:49container I should see one folder called
2:25:52sales and within that I should see Delta
2:25:54log and pocket file do you want to see
2:25:56that let's see it simply go to
2:25:59your account storage account and click
2:26:02on destination oh
2:26:04perfect I can see sales folder I can see
2:26:07sales folder let me just increase the
2:26:09size of it okay you can see sales folder
2:26:11in your case very good click on it yes
2:26:15yes yes yes this is your Delta log that
2:26:18we were talking about this is is the
2:26:20Delta log this is the pocket file name
2:26:23do not worry it is automatically decided
2:26:25when we create the folder and you do not
2:26:26need to worry about file name because we
2:26:29read our data using folder names not
2:26:31file names so do not need to worry about
2:26:32that this is pocket format as you can
2:26:35check the properties here click on this
2:26:37and click on properties so this is the
2:26:41pocket for you can just check the data
2:26:43type I think from here or you can just
2:26:44go to edit tab you should see data is
2:26:47encoded so this is encoded data you
2:26:49cannot understand it
2:26:51but spark can understand do not need to
2:26:53worry so the main point here is we have
2:26:57the pocket file on the top of it we have
2:27:01Delta log do you really want to check
2:27:03what is inside this Delta log let me
2:27:05show you we have Json and CRC files see
2:27:10we have something called Json and CRC
2:27:12file let me just expand it we have
2:27:1600. Json
2:27:1800. CRC
2:27:20this is not of our use do not need to
2:27:23worry CRC stands for I think cyclic
2:27:25redundant check files and the main thing
2:27:29is this one I was saying that all the
2:27:32metadata all the instructions are
2:27:34written in this Delta log that mean
2:27:36everything is written within Json file
2:27:38let me open it and let me click on edit
2:27:41I will show you everything is written
2:27:43within it everything see metadata what
2:27:46what do we have inside metadata we have
2:27:48ID okay we have schema string as well we
2:27:55have all the other fields see item fat
2:28:00content everything all the column names
2:28:02all the data types every information is
2:28:05written within this Delta log so let's
2:28:08suppose let's suppose we have here
2:28:12multiple files like one 2 3 up to like
2:28:17thousands up to thousands let's
2:28:21see so it will read data from these
2:28:24files yes but instead of reading data
2:28:27from the metadata of individual file it
2:28:30will straightforwardly go to this Delta
2:28:32log and it will read data from here this
2:28:36is your Delta leg this is your Delta
2:28:39format this is your Delta
2:28:43Data so this is all about Delta Lake all
2:28:48about no we have just started this topic
2:28:51I'm going to cover each and everything
2:28:53within this topic I know that would
2:28:56require a lot of time but still I'm
2:28:58ready to give it I'm ready to devot it
2:29:00just for you so that you can learn you
2:29:01do not need to go here and there and you
2:29:03do not need to just suffer I will help
2:29:06you to learn and grasp the whole
2:29:09content
2:29:10so we have successfully loaded the data
2:29:13into the destination container using
2:29:15spark writer API and in the Delta Lake
2:29:18format now we are excited to learn more
2:29:22about Delta tables Delta tables because
2:29:27in the real world we create tables yes
2:29:29we work with data frames as well but at
2:29:31the same time we work with Delta tables
2:29:34more because at the end we need to serve
2:29:37Delta tables to the data analyst or
2:29:39let's say bi managers bi developers to
2:29:42create reports on top of it how we can
2:29:44create Delta tables let me show you but
2:29:46before that I will tell you about one
2:29:50important concept that is called
2:29:53external tables versus managed tables
2:29:56and yes that means external Delta tables
2:29:59and managed Delta tables before covering
2:30:02Delta tables we will cover an important
2:30:05concept called external Delta tables
2:30:07versus managed Delta tables and trust me
2:30:10this is your hot topic for interview so
2:30:13let me cover it for you like this so
2:30:16let's understand first the manage tables
2:30:19manage dat tables or manage tables in
2:30:22data so let's say I created one table
2:30:27any table any random table like create
2:30:30table table name and columns anything
2:30:32like I just created this random table
2:30:35right so this is my random table okay I
2:30:40have like let's say three columns ID
2:30:42name and marks right this is my random
2:30:46table if I create map manage tables if I
2:30:51create manage table first of all I want
2:30:55to create a database obviously I need a
2:30:58database under which I can register my
2:31:00table so first of all I will create a
2:31:04database oops database okay I will
2:31:08create one
2:31:09database within this database I will
2:31:12create something called as
2:31:16table and I will name it as table one
2:31:20simple yes I will name it as table
2:31:23one within this database right within
2:31:27this
2:31:28database what will happen what will
2:31:31happen what will happen just guess just
2:31:33just just make some guess so this is a
2:31:37meta store this can be your hive meta
2:31:40store this can be your Unity meta store
2:31:42So Meta store is basically the
2:31:45repository where we store all the uh
2:31:48definitions database definition
2:31:50schema definitions table definitions all
2:31:53the definitions all the metadata right
2:31:56so this is the meta
2:31:58store okay when we create any table in
2:32:02data bricks manage table let's talk
2:32:04about manage table when we create any
2:32:06manage table what will happen let's say
2:32:09this is the table one right so in this
2:32:12particular meta store in this particular
2:32:15meta store our database definition will
2:32:18be stored
2:32:20let's say our database name is sales
2:32:22like sales DB right let's let's say
2:32:24let's call it as sales DB okay and
2:32:28within this we have a table
2:32:30called table
2:32:33one so this information will be stored
2:32:37here yes this information will be stored
2:32:40here that under this
2:32:43database we have a
2:32:45table but but but but but but this is
2:32:50our cloud storage this is our data Lake
2:32:52this is our your cloud storage or maybe
2:32:54awss 3 or anything this is the cloud
2:32:57storage actually the data brakes
2:33:01workspace have one default storage
2:33:03account as you remember we created
2:33:05manage Resource Group in the beginning
2:33:08for just for the uh datab Brak workspace
2:33:10and I told you that datab braks manages
2:33:12it and we do not touch it this is the
2:33:15part of that Resource Group in that
2:33:17particular Resource Group it will create
2:33:19one storage account as
2:33:21well and when we say store our manage
2:33:26Delta table store our managed table it
2:33:30will store the metad data in The Meta
2:33:33store and the actual data that means
2:33:37records ID marks um like names
2:33:41everything will be stored in
2:33:44this actual data Lake yes data will get
2:33:50stored in cloud storage which is the
2:33:53default account which is the default
2:33:56storage account or default data Lake
2:33:58which is given to datab Brees
2:34:01workspace interesting yeah it is
2:34:04interesting because we just store the
2:34:06metadata in meta store it can be your
2:34:09hive metast store I will show you don't
2:34:10worry I will show you everything I'm
2:34:11just giving you the concepts I just
2:34:13giving you the knowledge right now so
2:34:17this is the meta store and act data is
2:34:20residing in the data
2:34:23Lake yes so if we want to that is the
2:34:30difference between manage and external
2:34:31as well just listen it carefully if we
2:34:34want
2:34:35to delete this table let's say I want to
2:34:38drop this table after creating it so and
2:34:42as we know that this is the real data
2:34:43sitting inside
2:34:44this data bricks will delete the table
2:34:49definition obviously and this will
2:34:52delete the data as well because this is
2:34:55managed by the data breaks that's why it
2:34:58is called managed table managed Delta
2:35:04table got it this is the managed table
2:35:08in data bricks and let me tell you the
2:35:12external table now and it is so easy for
2:35:15you now to grasp the concept of external
2:35:17table I'm so sure you will just grasp it
2:35:19like this
2:35:41so in case of external Delta tables I
2:35:45know you have noticed one thing that our
2:35:48cloud storage is change from default to
2:35:51our own now in this particular
2:35:55scenario in this particular scenario we
2:35:59have we have here default storage as
2:36:02well yeah we have let me write it for
2:36:04you we have because it is attached to
2:36:07the datab brakes workspace yes so I will
2:36:10just write
2:36:12Cloud
2:36:14default we have Cloud default as well
2:36:18and at the same time we we have external
2:36:20storage account as well obviously we
2:36:23have created storage account but listen
2:36:25it carefully let's say this time we have
2:36:28data we have table right and we
2:36:32have the database let's say let's keep
2:36:35the database name same which is sales
2:36:39DB and table name is table
2:36:43one
2:36:45okay okay
2:36:47so when we Define when we create
2:36:50external table obviously this
2:36:53information will be stored in the meta
2:36:56store let me write it for you bro please
2:36:59make notes bro bro bro I'm repeating it
2:37:03again and again and bro means boys and
2:37:06girls both men and ladies both so just
2:37:10create your notes on your own just take
2:37:13your not take notes on your own please
2:37:16I'm repeating this again and again
2:37:17because this knowledge is something that
2:37:19you you need to revise again and again
2:37:21so please please please take notes
2:37:23please take notes so this data will be
2:37:27stored in the meta store which data Only
2:37:30The Meta data so we will simply
2:37:35write sales DB
2:37:39and table one perfect table one this
2:37:43definition is here but but in the
2:37:47previous scenario our data was residing
2:37:50in this default storage but this time
2:37:53our data will be residing here in our
2:37:56storage account okay but why you are so
2:37:59happy with this information because let
2:38:01me tell you the answer so this is the
2:38:04data that is sitting here let me just
2:38:05create the data okay now if I want to
2:38:09drop this table if I want to delete this
2:38:12whole table it will obviously remove the
2:38:16metadata from The Meta store because we
2:38:17do not need it but but but but but but
2:38:21this data will not get deleted because
2:38:24this is my own storage account data
2:38:27breaks is not allowed to drop data from
2:38:30my storage account this is the power of
2:38:33external table this is the reason that
2:38:35we most of the times deal with external
2:38:38tables in the organization because we do
2:38:40not want to drop the data yes in some
2:38:42scenarios we want but in most of the
2:38:44scenarios we do not want so this is the
2:38:47external table it will not not drop the
2:38:50data from my storage account I know you
2:38:52really excited to actually see these
2:38:54Concepts live in Practical yes I'm just
2:38:58about to show you all these things
2:39:00practically and let's get started with
2:39:03this concept manage Delta tables versus
2:39:06external Delta tables let's see so if we
2:39:10just go to our datab notebook and here I
2:39:14will just add one heading or let's say a
2:39:18markdown percentage MD and let's create
2:39:21heading to so here we will write
2:39:25managed
2:39:26versus external Delta
2:39:30tables I'm so so so thirsty right now
2:39:35so I just want you to learn I just want
2:39:38you to learn so that's why like I I just
2:39:41want to devote my all the time to just
2:39:44mention all the things that are relevant
2:39:46for this particular topic data brakes
2:39:48obviously Delta leg is like one of the
2:39:50most in demand tool in demand technology
2:39:54in demand file format it is like
2:39:56everything so first of all let's create
2:39:58one manage table so now you will ask me
2:40:00like how we can create manage table
2:40:02syntax is really really easy it is
2:40:05similar to SQL let me show you how you
2:40:07can create so let's create our manage
2:40:09table so I will simply write one more
2:40:13markdown and I will make it bold and I
2:40:16will call it as manage table
2:40:19perfect okay let me write the code so
2:40:22first of all you can uh switch the
2:40:25default language from here and you know
2:40:26the magic command now now you're smart
2:40:28you can simply write poage SQL hit enter
2:40:31now you can just write your uh SQL code
2:40:33because obviously we use SQL to create
2:40:36uh let's say Delta tables you can also
2:40:38use uh data frame API as well there's no
2:40:40need to just say okay okay we just want
2:40:43to use SQL no you can use data frame API
2:40:45as well but Mo but most of the times we
2:40:47use SQL and it is really easy to grasp
2:40:49the concept once you know the concept
2:40:51then you can just play with every
2:40:52possible scenario but it is really
2:40:55important for the understanding you
2:40:56learn it from SQL point of view why
2:40:59because you are already familiar with
2:41:00SQL obviously you would have like
2:41:02created tables in SQL right so that's
2:41:04why just telling you for your benefits
2:41:09okay let's create it create and oh okay
2:41:13okay first of all we need to create
2:41:15database you are smart you are smart
2:41:17first of all we need to create database
2:41:18so first of all I just add one cell here
2:41:21and I will make markdown for my database
2:41:24make it bold so I
2:41:26will say
2:41:29database
2:41:31database and done and I will simply
2:41:34write percentage SQL let's first create
2:41:36database
2:41:39man so what's this index for creating
2:41:41database it is as similar to SQL
2:41:44create
2:41:46database and then database name will be
2:41:48sales DB let's keep it same so that you
2:41:50can learn and you can relate let's run
2:41:52it let's run it okay it's done it's done
2:41:57okay now we have successfully created
2:41:58database and now we know our database
2:42:01will get stored in the meta store right
2:42:04right now we will create manage table
2:42:07first okay
2:42:09create table table name will be
2:42:13obviously starting with sales DB because
2:42:15this is our database now what table name
2:42:17you need to select let's say I want to
2:42:22keep it as man table that is manage
2:42:24table right man
2:42:26table man table man table short and like
2:42:30to the point to the point then once you
2:42:33write this now what we need to do we
2:42:35need to define the columns within it
2:42:37okay I will simply hit enter and open
2:42:39the braces I will simply create three
2:42:42columns ID data is very smart he knows
2:42:45like I love ID columns next will be my
2:42:47name not my name
2:42:49name String and then let's say marks
2:42:53right marks will be int okay now we need
2:42:57to Define delta we will write using
2:43:02Delta why because we are creating Delta
2:43:04tables now here's the thing bro just
2:43:08hold on so here's the thing if you want
2:43:11to create external table using any
2:43:13different file format let's say pocket
2:43:15you can create that if you want to
2:43:17create external table like manage table
2:43:19with CSV you can create with CSV but
2:43:21nowadays we most of the times create Del
2:43:24uh Delta tables so that's why we use
2:43:26Delta and data brakes has also adapted
2:43:29this so it has seen that most of the
2:43:31times people do create tables using
2:43:33Delta format so it has created this as
2:43:37default table format so if we do not
2:43:40Define anything it will automatically
2:43:41create Delta table format but being an
2:43:44efficient data engineer you should
2:43:45always write using Delta because it
2:43:49promotes readability because you are
2:43:51writing code for yourself as well as for
2:43:54others as well so that they can read and
2:43:56understand so here see it is suggesting
2:43:58me pocket bro we do not work with pocket
2:44:01we are Delta
2:44:02peeps so it is done it is done it has
2:44:05created our Delta table I will run
2:44:10it whoa sales DV okay I just used a
2:44:13different spelling like SES
2:44:16TV okay okay my my bad okay okay I
2:44:19accept it bro okay now it is creating a
2:44:21Delta table wow it is created we have
2:44:23created our first Delta table we want to
2:44:25query it wait hold on let just insert
2:44:28some data in this so that you can query
2:44:30it and actually feel
2:44:32it okay so how we can just insert the
2:44:35data it is same as SQL exact same that's
2:44:38why I picked like SQL API to just
2:44:40explain you these Concepts so that you
2:44:41do not need to suffer of the concept
2:44:43that you already know you will only
2:44:45focus on the concept that are new right
2:44:48right
2:44:50okay we will write insert into and then
2:44:53obviously sales DB dot table name table
2:44:57name is man
2:44:59table then same way we need to just
2:45:02insert the values like one and name will
2:45:04be AA then marks will be let's say 30
2:45:08who is this guy with 30 marks
2:45:10man okay let's two and then BB let's say
2:45:15then
2:45:1633 then oh
2:45:19I ran it by mistake wait wait wait I ran
2:45:23it by mistake I just hit like shift plus
2:45:24enter instead of enter wait wait wait
2:45:26wait wait hold on hold on hold on okay
2:45:29just hold on
2:45:33so
2:45:34[Music]
2:45:36uhuh okay perfect so now I will say
2:45:39three do
2:45:42CC and then let's say
2:45:4735 okay okay perfect and
2:45:51now why is throwing error here why why
2:45:54why let me run it
2:45:58now near
2:46:01one oh I didn't add values keyword oh
2:46:05bro see it also suggests as the uh right
2:46:08thing I I love data brakes I love data
2:46:10brakes so instead of accept and we can
2:46:13use accept and run command as well but I
2:46:15will reject it and I will hardcode it so
2:46:17you need to just write values
2:46:20my bad because I was just discussing
2:46:22something unimportant which was like who
2:46:25scored like marks 30 so I want to insert
2:46:28one more value let's say four and then
2:46:31DD and then 40 who scored 40
2:46:37mons okay let's not be judgmental so I
2:46:41will just run this command and it should
2:46:43just insert the values in my table now I
2:46:46am excited to see the output how you can
2:46:49query this table you can use select
2:46:52statement as you use in SQL yes exactly
2:46:55same so I would like to query this I
2:46:58would like to query this first of all I
2:47:01will write percentage SQL then I will
2:47:03say select
2:47:05ax oops from
2:47:09sales
2:47:10db.
2:47:12manable I want to run this I will run
2:47:14this and you will see your Delta table
2:47:17you will see your Delta table
2:47:20wow wow wow wow and this is your managed
2:47:24table this is your managed table yes
2:47:29this is not your external table this is
2:47:31your managed
2:47:33table right okay so now here is the
2:47:37thing this data is stored in this
2:47:41catalog if you go to catalog I will show
2:47:43you the meta store just click on
2:47:46it okay and then when you you just click
2:47:49on this so now you can see sales DB now
2:47:53you can see sales DB this is your
2:47:55definition this is your repeat repeat
2:47:58with me this is your only definition
2:48:00when I click on it I should see the
2:48:02table okay I should see the table this
2:48:06is our meta store this is also called as
2:48:09hi meta store and nowadays you you would
2:48:12also see like Unity meta
2:48:13store this is the meta store where is
2:48:16the data written where is the actual ual
2:48:19data actual data is stored in the data
2:48:22Lake SL storage account of this data
2:48:26bricks workspace which was created at
2:48:29the time of this workspace
2:48:32creation simple that means we do not
2:48:35have any control over that data do you
2:48:38want to test it let me show you bro let
2:48:41me go to this recents and let me just go
2:48:44to Delta Lake
2:48:46and now if if I
2:48:50just if I just drop this table I want to
2:48:54drop this table I want to drop this
2:48:56table this will drop the data as well
2:49:00because I do not have any control data
2:49:02briak is managing the data for me let me
2:49:04show
2:49:05you let me
2:49:08write percentage
2:49:10SQL
2:49:12drop
2:49:15table uh sales DB man table let me drop
2:49:20it let me drop it it is dropped it is
2:49:25dropped now if I just run this command
2:49:27select srix from it it should not run it
2:49:30should not
2:49:31run it should not run it will throw an
2:49:34error that there's not table because it
2:49:36is dropped and data is also dropped
2:49:37because I do not have any control over
2:49:39that data but but but but oh man I'm so
2:49:44tired ah
2:49:47but here's the thing when we create
2:49:50external table when we create external
2:49:54table our data will not get deleted
2:49:57let's create external table let's create
2:49:59it and how you can create external table
2:50:01it is same same you just need to add one
2:50:03line of code it is same let me show you
2:50:06so let's create external table I need
2:50:09water I will definitely have full glass
2:50:12of water after this this this session
2:50:14like this uh external table okay so let
2:50:18me just add add a heading so percentage
2:50:20MD then we will write something called
2:50:22as external table let me make it
2:50:25bold external table okay perfect
2:50:30external table let me run it so now I
2:50:34will create external table first of all
2:50:35I will write percentage SQL then I will
2:50:38say create
2:50:39table sales DB under the same database
2:50:43obviously sales DB dot external table
2:50:47instead of man table it not now this
2:50:48time is external table now I will Define
2:50:52the same thing ID integer right
2:50:58name and
2:51:00string then we have
2:51:03marks not
2:51:04marks I'm not Elon
2:51:07Musk marks and then we have int by the
2:51:12way I love Elon Musk I love I love that
2:51:15math so now everything is same obviously
2:51:18we need to Define delta using Delta
2:51:21what's that what's after this I know
2:51:23like this is the same step we have done
2:51:25for manage table when you write one more
2:51:28line it will become external table what
2:51:30was the fundamental in external table we
2:51:33stored the data in our own storage
2:51:36account what does it mean it means we
2:51:39need to provide the location where we
2:51:41need to save our data so we will simply
2:51:43write location and then I will write the
2:51:46location AB FSS
2:51:49and this time I want to save it at
2:51:52destination and obviously my storage
2:51:55account name which is datal
2:51:58lunch.
2:52:00DFS doc. windows.net
2:52:03and then I will create a folder called
2:52:06sales
2:52:08DB then within that I will call this at
2:52:11as external table external table let's
2:52:14let's do it okay once it is done we want
2:52:18to run it and you will see the magic I I
2:52:20will show you the magic not the magic it
2:52:22is just like the functionality of
2:52:25external table now it is creating
2:52:27external table it has created external
2:52:29table right now I want to insert the
2:52:33data as we have inserted the data for
2:52:35this table so I will just copy it from
2:52:36here and I will insert the same values
2:52:39here within our external
2:52:41table okay and obviously we're inserting
2:52:45the values in external table
2:52:48okay let's insert
2:52:51it values are inserted values are
2:52:54inserted
2:52:56now I want to query
2:52:58this percentage
2:53:03SQL
2:53:06select Ax
2:53:09from csdb do external table do external
2:53:14table let's run this
2:53:18perfect perfect perfect perfect perfect
2:53:21and if you want to see this data this
2:53:24table we need to just go to the catalog
2:53:27then within the high meta store we
2:53:28should see this table external table now
2:53:30it is now it is external table okay this
2:53:34is here we know but an you told us that
2:53:38our data is residing in our data L yeah
2:53:40I'm not lying I'm not
2:53:43lying so H I need to stretch so if you
2:53:48want to see the data I can just show you
2:53:50okay so if you just go to your storage
2:53:53account just go to your storage account
2:53:55and just select your destination
2:53:57container within that container you will
2:53:59see a new folder created it's called
2:54:02sales DB it's called sales
2:54:05DB what is this this is our database
2:54:09when you click in this like when you
2:54:11click on this folder you will see
2:54:13another folder in this folder external
2:54:16table this is our table when you just
2:54:18click on this table you will see the
2:54:22Delta Theta this is your Delta
2:54:25log this is your actual data actual
2:54:29actual physical data which is in our
2:54:31control now I will show you the command
2:54:34I will just run the drop command let me
2:54:36just show
2:54:38it so when you just run the drop command
2:54:42now when you just run the drop command
2:54:44let me just run the drop command drop
2:54:49table and what was the schema oh ssdb do
2:54:54external
2:54:55table now when you just do this oh I
2:54:58just provided the wrong
2:55:00spelling table Yeah I was a little bit
2:55:04more excited so now what T like like our
2:55:06table is dropped right if I just go to
2:55:08my catalog or not even go to catalog I
2:55:10can just query it when I just query it
2:55:14it should not give me the table because
2:55:16it is dropped from the meta store as
2:55:18well but you will see the data residing
2:55:20here let me just open the destination
2:55:22table again I will still see the sales
2:55:24DB I will still see the external table I
2:55:26will still see the data and Delta log
2:55:29bro you are just making us a fool just
2:55:31refresh it okay let me click on refresh
2:55:34it is still here it is in our control
2:55:36that's called external table external
2:55:38Delta
2:55:41table was it helpful I think so so this
2:55:45is all about manage versus external
2:55:48Delta
2:55:49tables now we will level up our skills
2:55:55how let me tell you let me tell you let
2:55:58me tell you so now what we will be doing
2:56:01we will discuss the functionalities of
2:56:03Delta L what you can do with Delta
2:56:06tables now you are clear with manage
2:56:08versus external tables now obviously you
2:56:11will just work with external table why
2:56:13because you will see the results at the
2:56:15same time because you can look at the
2:56:16data but in case of manage table you can
2:56:19so we will carry forward our lessons
2:56:21with external tabl so that you can grasp
2:56:23the knowledge so that you can just
2:56:25compare the data now now now now it's
2:56:28time to discuss the functionalities of
2:56:31Delta tables and before that let me
2:56:34drink some water I'll be come back in
2:56:37just a few minutes all good all good all
2:56:39good yeah I just made my protein shake
2:56:43that was really really
2:56:46required no worries that will get
2:56:48compensated when you will show lots of
2:56:51lots of love so that's what I want so
2:56:55now we will discuss Delta table's
2:56:57functionalities and we will cover all
2:57:00all all the functionalities and you will
2:57:02be amazed by the knowledge that you will
2:57:04get right now so I was also amazed when
2:57:07I got to know okay this is like this is
2:57:10the thing like how things are working
2:57:12let me give you a hint let me give you a
2:57:14hint okay let's not give you a hint let
2:57:17let me show you okay let me just write a
2:57:19heading percentage MD and we'll call it
2:57:23as first of all it would be heading
2:57:26three or let's say heading two okay
2:57:29let's say heading two and we will call
2:57:31it as Delta
2:57:32tables don't worry I will provide this
2:57:34notebook so that you can just keep it
2:57:36for reference and you can just give like
2:57:38make notes out of it Delta tables
2:57:43functionalities functionalities
2:57:50okay so what we will be discussing in
2:57:54this almost everything almost everything
2:57:57okay first of all let's perform like I
2:58:00will I will perform like all the DML
2:58:02operations that we perform like delete
2:58:04update insert then alter table like
2:58:07everything and we will learn everything
2:58:10one by one and I will also tell you a
2:58:12concept called tombstoning don't worry
2:58:14don't worry don't worry I will just
2:58:16discuss everything first of all let's
2:58:17perform one function that is insert
2:58:20right percentage MD and I will make it
2:58:22bold and let's call it as insert right
2:58:26perfect so I want to insert some more
2:58:29data in my Delta table and I want to see
2:58:31the uh like how it will write my data
2:58:35right like how it will store my data
2:58:36behind the scenes so first of all I need
2:58:39to recreate the data so for that I can
2:58:43just copy the command that is what
2:58:44create table so that it can create the
2:58:47data again
2:58:50right or I can just rerun it from here
2:58:52as well there's no need to worry about
2:58:55it and I can just remove this drop
2:58:58command where it is here I can just
2:59:00remove this drop command and I can just
2:59:02rerun those steps and before that I can
2:59:04just delete that folder as well like
2:59:07which is external table because I'm just
2:59:09recreating this table so do not need to
2:59:11worry just delete this table because I
2:59:14just wanted to show you that data
2:59:16resides there so I'm just recreating
2:59:17this table
2:59:18this one let me run
2:59:21this
2:59:30mhm it is creating our Delta table again
2:59:33with the same
2:59:38location uh uh
2:59:41uh do it fast bro do it fast
3:00:06H okay yeah sometimes like I just
3:00:09restarted my cluster because I just I
3:00:11just went to make my protein shake so
3:00:14that's why so whenever we just run the
3:00:17very first after that like after
3:00:19restarting the cluster it takes some
3:00:20time but no need to worry no need to
3:00:21worry so obviously I will just insert
3:00:23this data
3:00:25yeah data is inserted perfect now I will
3:00:29just see this data yeah I want to see it
3:00:37[Music]
3:00:52H why it is
3:00:54waiting let me just rerun
3:00:57it yeah it is
3:01:00done what problem has occurred oh I just
3:01:02reran it
3:01:04okay let me wait until it has done
3:01:25let's canel it reload page let's reload
3:01:28the page make like giving some fishy
3:01:32errors let's reload the whole page so
3:01:34now it is
3:01:36refreshed I don't know it was like due
3:01:39to maybe internet connection or maybe
3:01:40internal error I don't know so I just
3:01:41refreshed it so now it is working fine
3:01:43so as we can see that our data is there
3:01:46and it has recreated the external TB on
3:01:47top of it that's good that's good so now
3:01:50we want to perform a function or let's
3:01:53say a DML command that is insert so I
3:01:57will just copy this code insert
3:01:59obviously I will update the values don't
3:02:01worry so now I want to insert more
3:02:04records can it allow us to do that
3:02:07obviously bro you can so let's say I
3:02:10want to insert
3:02:13five four six 54 wow wow wow wow wow
3:02:17punch hats off man you have done
3:02:20Bachelor of Science in mathematics and
3:02:21you are saying four after five very very
3:02:25well done very well done okay very well
3:02:28done
3:02:29so basically we will now insert the
3:02:33values inside the table so it can allow
3:02:36us but the whole intention is totally
3:02:39different and I will tell you what's
3:02:41that let me just first run this and it
3:02:44ran successfully as expected and now I
3:02:47want to quer quer the data select ASX
3:02:50from sales DB do external table perfect
3:02:54I will query
3:02:56it obviously I can see four new records
3:02:595 6 7 8
3:03:00yes like can you imagine like you know
3:03:04that we have parket
3:03:07files behind this table right behind
3:03:10this external tables so what just
3:03:14happened with that pocket file let me
3:03:16show you when you just check
3:03:20the data L and you just go to your
3:03:23container which is
3:03:25destination and within the destination
3:03:27just go to your ssdb which is your
3:03:29database external table is your table
3:03:31now a new file is added see now new file
3:03:35is added new file is added part 00
3:03:40hyphen e a44
3:03:43e so it has inserted a new pocket file
3:03:48right and do you want to know what has
3:03:50just happened in Delta log let me tell
3:03:52you a new Json file is inserted which is
3:03:56called this
3:03:59one which is this one
3:04:02Jason yes so what it has done let me
3:04:06just open it for
3:04:07you if you go to edit you will see it
3:04:11said add part
3:04:150000 ea4 44 that is our latest data in
3:04:21Json 1 you will see something called
3:04:26as something just got into my eyes
3:04:31wait
3:04:33bro what was that man yeah it's fixed
3:04:39so now this is the pocket file which was
3:04:43initially added when we just inserted
3:04:45values 1 2 3 4 so this is that thing
3:04:50that means that means how it reads our
3:04:54data how it reads our data so this is
3:04:58our table EXT table folder right so how
3:05:02it Delta table read our data so it will
3:05:06not go to these partitions
3:05:09no it will go to Delta log it will read
3:05:14all the Json files Json 0000 that
3:05:17consist our metadata 01 had our initial
3:05:21load then adjacent 2 had the incremental
3:05:24load that is 5678 then after all those
3:05:28logs it will read the data accordingly
3:05:32this Insight this knowledge will get
3:05:34more clear when I will show you the next
3:05:37thing which is the delete command which
3:05:41is the delete
3:05:42command and we will also learn the
3:05:45concept called tomb stoning let me just
3:05:47just write it
3:05:48here tomb
3:05:53stoning tomb stoning oops yep
3:05:57tombstoning we will also learn this Con
3:06:00concept tombstoning let me show you so
3:06:04our insert is done let me show you the
3:06:08delete percentage MD and let me make it
3:06:13bold delete let's say let's say I want
3:06:18to delete this record which is record
3:06:20number eight let's say I want to delete
3:06:23this record okay let's delete it let's
3:06:26delete it syntax is simple right we need
3:06:28to just write delete oops let me first
3:06:31add percentage SQL then we need we can
3:06:34say delete from table which is sales db.
3:06:38external table let me tell you what just
3:06:40happened so first of all to understand
3:06:42what just happened let's go to the Delta
3:06:44log okay here's the Delta log
3:06:48first of all in the Delta log Json 0000
3:06:51this is created this was created when we
3:06:54defined the table when we didn't have
3:06:55any data okay let's skip this now in
3:06:58Json 1 what we have what we have we have
3:07:03the information that add this partition
3:07:06what is this partition just remember the
3:07:07name 42b just remember this name
3:07:1142b add this partition okay okay and we
3:07:15just know we just got to know that
3:07:17whenever we read the data in Delta
3:07:19tables it first goes to to the Delta log
3:07:22it read all the logs and then it read
3:07:24the partitions it do not like it does
3:07:27not care if there are like 300
3:07:29partitions or 3,000 partitions or like
3:07:31three lakh poket files no it will only
3:07:34read the Delta log whatever it is
3:07:37written in the Delta log it will read
3:07:39only those pocket files okay it says add
3:07:43part 42b remember this 42b okay we added
3:07:46it in Json 2 it
3:07:49says it in Json 2 it says add c92 as
3:07:54well 42b c92 both the partitions were
3:07:57headed yeah yeah we we are fine till
3:08:00till now okay what just happened then I
3:08:05gave the command which is
3:08:08delete which is delete command here now
3:08:13what pocket or let's say what Delta
3:08:15table has done for us it went to all the
3:08:19partitions it went here and let me go to
3:08:22the folder sales TB external table so
3:08:25these three partitions and we know that
3:08:28this is the first partition 1 2 3 4 this
3:08:30is the second partition 5 6 78 now we
3:08:35know that this particular partition had
3:08:39the ID equals to
3:08:418 right we added 56 78 in the second
3:08:45iteration and this partition of or let's
3:08:47say this pocket file whenever I say
3:08:49partition it's it's called part it's
3:08:51called pocket file right so in this
3:08:53pocket file it had ID equals to
3:08:578 what it has done it has copied this
3:09:01data this whole pocket file and it has
3:09:06created a new pocket file and it has
3:09:09removed that particular record from this
3:09:13pocket
3:09:14file
3:09:16wow it has not deleted the record from
3:09:20this file no I will tell you why don't
3:09:24worry just hold on hold on it has not
3:09:26deleted the record from this partition
3:09:29it has created a new Partition in which
3:09:33it copied all the data and it removed
3:09:37that particular record it removed that
3:09:40particular ID that set and it created
3:09:42the new
3:09:43Partition now in the Delta log it will
3:09:47say read this
3:09:51partition remove this
3:09:53partition read this
3:09:55partition that means that means just
3:09:59take notes bro this is really important
3:10:02that means
3:10:03add and then
3:10:05remove and then add simple and let me
3:10:10just show you let me just confirm you
3:10:12let's go to the let me just expand this
3:10:17Json number four let me show you it will
3:10:21say
3:10:23see remove
3:10:26this remove
3:10:28this and read this now why it has said
3:10:34like remove this as well because it has
3:10:35combined all the data but the point is
3:10:38it has said removed this because this
3:10:41had the ID equals to
3:10:448 it had ID equal to8 so it removed it
3:10:49so it removed this and it removed this
3:10:52as well because this parket file just
3:10:54had like four record so it said let me
3:10:57just recreate one pocket file which is
3:10:59the new one e7d and let me combine the
3:11:02four uh records of the previous pocket
3:11:04file and just three from this new one
3:11:08because obviously Delta tables just
3:11:10optimize the partitions for us so that
3:11:12is a different story let's not go in
3:11:14that way but it remov this
3:11:17it remov this that that's called Soft
3:11:20deletion soft
3:11:23deletion it has not actually removed the
3:11:26files files are there all the three
3:11:28files are there but it has given the
3:11:31instructions that do not read this data
3:11:33that means remove this partition remove
3:11:35this partition and read only this
3:11:38partition read only this partition
3:11:41because in this partition it has three
3:11:43records that is 567 and 1 2 3 4 as as
3:11:47well this is called tombstoning that
3:11:51means it has tombstoned these two
3:11:53partition tombstones mean like yeah yeah
3:11:56those ghost modes so this has ghost put
3:12:00the put these two partitions in the
3:12:02ghost mode or it has blacklisted these
3:12:05two
3:12:06partitions and let me drink some water
3:12:14man it has put these two parties in The
3:12:18Blacklist and it has asked just read
3:12:21this partition just read this partition
3:12:24only just read this partition
3:12:27only now now now now when it will read
3:12:33the data now when it will read the data
3:12:37let's go to destination sales TB
3:12:38external table now it has all the three
3:12:41partitions and I just mentioned it
3:12:43doesn't care how many partitions are
3:12:45there it doesn't care so it will
3:12:48straightforwardly go to Delta Lake and
3:12:49Delta lake has specifically mentioned
3:12:51that do not read this because it has
3:12:54combined that data do not read this
3:12:56because obviously it has deleted some
3:12:57data in that one so it has created one
3:12:59new Partition which has data from this
3:13:02one as well and obviously the data from
3:13:06this partition as well see this is
3:13:09called
3:13:10tombstoning tombstoning in Delta Lake
3:13:13just take notes just take notes this is
3:13:17the thing that you should
3:13:19know so when you perform delete that
3:13:22doesn't mean that it has physically
3:13:23deleted data no but why why why it it it
3:13:29just didn't delete the data let me tell
3:13:31you why let me tell you why now here is
3:13:35the concept
3:13:37called here is the concept called Data
3:13:43versioning data versioning let me make
3:13:45it bold
3:13:48data
3:13:50versioning what is this data versioning
3:13:52so why it
3:13:54didn't just change the partitions why I
3:13:57I will I will tell you just wait hold on
3:14:00what is data versioning if I want to
3:14:02keep a track of all the changes happened
3:14:05in our table we can do that how I will
3:14:08just WR uh run the command called
3:14:12describe oh man who will put percentages
3:14:15equal me me me me sorry
3:14:19describe
3:14:22table describe
3:14:24history and then table name sales db.
3:14:27exe
3:14:29table
3:14:32okay we don't want your suggestions bro
3:14:34it is showing no suggestions when I run
3:14:37this command when I run this command it
3:14:40will show me all the versions of my
3:14:43table all the versions that means all
3:14:46the changes that have happened so far
3:14:49all the changes so as you can see
3:14:53that the here are the
3:14:56operations
3:14:59see so
3:15:01now
3:15:03in0 in
3:15:06zero we created our table we created our
3:15:11table that means operation is create
3:15:15table right
3:15:18okay so in the version one we inserted
3:15:23the data it says
3:15:25right in the version two it says right
3:15:31in version three in version three it
3:15:37says
3:15:38delete it says
3:15:41delete that's where we deleted the
3:15:44record that's our version number three
3:15:47and in the version number four it has
3:15:49optimized the table and that we know and
3:15:52that we have already talked about like
3:15:54it has just combine the partitions we
3:15:56will talk about optimizations in just
3:15:58few more minutes so do not worry about
3:15:59that we will discuss that part later so
3:16:01just forget about optimize command let's
3:16:03just imagine that we have three three
3:16:05versions just just imagine that we just
3:16:07have three versions that's it it will be
3:16:09good for your understanding just trust
3:16:10me okay so now we have all the history
3:16:15why do we need to keep the history why
3:16:19let me tell you why let's say I just
3:16:23missed like I deleted this data which is
3:16:26this like ID equals to wa by
3:16:29mistake by mistake I want to get this
3:16:32data back how we can do that how we can
3:16:35just roll back our table you would have
3:16:38like heard about roll backs in SQL
3:16:41databases we can just do roll backs here
3:16:43as well we call it as time travel you
3:16:47time travel exactly let me show you let
3:16:50me show you it's it's really really good
3:16:52let me show you if you have deleted this
3:16:54data by mistake and now you want to go
3:16:57back to the previous state of the table
3:16:59that's why Delta table that's why Delta
3:17:03table didn't didn't didn't delete it
3:17:07didn't delete these
3:17:09partitions didn't the lead these
3:17:12partitions got it because it knows that
3:17:17we can use time travel to go back in
3:17:20time yeah it's similar to doron's time
3:17:23travel machine but it is much more
3:17:25powerful than that so if I want to do
3:17:29that I can how let me show you let me
3:17:32show
3:17:33you I will just write percentage MD bro
3:17:36T trust me this knowledge is really
3:17:39really really powerful once you know all
3:17:43these things and then when you will just
3:17:46sit in the interviews trust
3:17:48me trust me that you will make an impact
3:17:52trust me because all these things are
3:17:55really really
3:17:56new all these things are really really
3:17:59new so you have really good chances to
3:18:03be select to be selected and to actually
3:18:07make an impact trust me so I will simply
3:18:11write percentage MD and now I will
3:18:14simply make it bold and I will write it
3:18:17as time travel time travel if we want to
3:18:21time travel how we can do it let me just
3:18:23show you so as you know that we have
3:18:25version 0 1 2 3 right if I want to go to
3:18:31version two which was this state which
3:18:34was right State when we just added the
3:18:37data I want to go to that particular
3:18:39version how we can do it we can use
3:18:42something called
3:18:44as first of all percentages
3:18:48SQL oops so we can use a command called
3:18:53restore restore table
3:18:55name restore table table name sales DB
3:18:59and here you can
3:19:02see I want to restore my table two
3:19:05version as of two two version as of two
3:19:08let me just run it two version as of two
3:19:12I haven't do I haven't done anything I
3:19:14haven't done anything I have just change
3:19:17the version of the table now you will
3:19:19see ID equals to 8 visible in the table
3:19:23that's not magic That's The Power of
3:19:26Delta tables That's The Power of Delta
3:19:29time travel now let me just query my
3:19:32table again percentage SQL
3:19:38oops select
3:19:41ASX
3:19:43from sales db.
3:19:47EXT table now you will see ID equals to
3:19:508 as well see see you have successfully
3:19:54learned how you can perform time travel
3:19:56so now if you get any question in the
3:19:58interview hey you have mistakenly
3:20:00deleted all the data all the data you
3:20:02truncated all the data just say I don't
3:20:06care because I can just get the data
3:20:08back by using time travel by using
3:20:11restore command by using versioning
3:20:13simple and your recruiter will be like
3:20:16okay this boy this girl has really
3:20:19skills like has really some amazing
3:20:24skills and just tell them like you watch
3:20:26my
3:20:31videos that's really really
3:20:34powerful so now you know how you can
3:20:37perform time Travel how you can perform
3:20:40time travel let's
3:20:43say you have performed time travel okay
3:20:46now you will ask me bro is there any way
3:20:50that we can actually delete the data
3:20:53from the folder let's
3:20:56say let me just refresh
3:20:59it okay now you will see no partition is
3:21:02created because we haven't done anything
3:21:04we just went to the previous version
3:21:06that's it now you will say hey bro do we
3:21:08have anything by which we can just uh
3:21:12actually delete like physically delete
3:21:15these pocket files
3:21:17one option is physically go there and
3:21:19delete it no don't you dare do it you
3:21:22will break the whole table just trust me
3:21:25do not do it how we can do it then so
3:21:28there's a command called vacuum
3:21:31command I'm just sharing all the
3:21:34information don't worry don't worry
3:21:35don't worry don't worry bro now you will
3:21:37learn each and
3:21:39everything related to data breaks and
3:21:42Delta link today you will become a pro
3:21:45developer not today like obviously you
3:21:47are just learning in pieces that's how
3:21:49we learn but at the end of this session
3:21:53you will be a pro so we have a command
3:21:56called vacuum what it will do it will
3:21:59just check all the partitions available
3:22:02for the table and we have just discussed
3:22:05that like let let me just take return
3:22:08the version 2 three so that you can just
3:22:10relate so let's imagine now recently our
3:22:14uh our table is not time travel to
3:22:17version two it is still at version three
3:22:20that is the latest one like in which we
3:22:22do not have ID equals to8 so in that
3:22:24scenario you know that we do not use
3:22:28these two
3:22:29partitions right we do not use these two
3:22:32partitions because we just use this
3:22:34partition we have seen this come on man
3:22:37we have seen this we do not use these
3:22:39two partitions because we have Rewritten
3:22:41this partition and Rewritten this
3:22:43partition as well and we have removed
3:22:45the ID equals to8 right when I run
3:22:49vacuum command on top of this table it
3:22:53will physically remove these partitions
3:22:55yes it will physically remove these
3:22:58partitions like this one and this one
3:23:01just a disclaimer do not try this in
3:23:05production do not do not try this in
3:23:08production but still you should no like
3:23:11you can you can be asked like how you
3:23:12can just physically delete you can how
3:23:14you can hard delete those files you can
3:23:15use a command called vacuum command so
3:23:19obviously you should not use it but
3:23:21sometimes in some scenarios you have to
3:23:22delete it there are some guidelines such
3:23:24as gdpr in which customer has the right
3:23:27to get their information deleted from
3:23:29company's database so under that
3:23:31scenario they have to delete it that's
3:23:33why we have a command called vacuum
3:23:34command so if we want to delete it we
3:23:38can just run vacuum command it will
3:23:40physically delete these two partitions
3:23:42yes trust me it will physically delete
3:23:44these two partitions like it will just
3:23:46do
3:23:47T it will just remove these two
3:23:49partitions that's it let me show you let
3:23:51me just show
3:23:53you obviously we will not uh use vacuum
3:23:56command for now like I will just show
3:23:57you the uh code because I do not want to
3:24:00like delete my files because what what
3:24:02is the reason because we lose the power
3:24:04of time travel then we cannot perform
3:24:06time travel why we do not have files how
3:24:09it can go to previous pocket file if it
3:24:10doesn't have any pocket file use your
3:24:13common sense I know it's not very common
3:24:15but my data Comm is very very smart so I
3:24:18can assume that you have some common
3:24:20sense so I will just show you how you
3:24:23can perform uh vacuum command but do not
3:24:25run it do not run it if you run it do
3:24:27not do not come to me that you because
3:24:28of you we have uh lost our data it's up
3:24:31to you so my task is just show you
3:24:33everything so percentage
3:24:36MD because obviously we will just run
3:24:38back command I'm just kidding but before
3:24:40that I just want to show you the
3:24:41optimization commands as well because
3:24:43being an efficient data engineer you
3:24:45should know how how to optimize the
3:24:47Delta tables that's
3:24:50the thing that is not everyone aware of
3:24:53how and why why is
3:24:56like not everyone is updated because you
3:25:00need to go through documentation you
3:25:02need to read books like
3:25:03this I love this book I I love this book
3:25:07so you need to go through these books or
3:25:10you can just watch my videos and you can
3:25:14just take my sessions so that that I can
3:25:16just convey I can just transfer that
3:25:19information that knowledge with you all
3:25:21so you have like both the ways and I
3:25:23think you would choose the second option
3:25:25no I'm I'm not kidding bro just come on
3:25:27like I'm just passing all the
3:25:28information like like like like
3:25:31this H so now I will just show you the
3:25:35vacuum
3:25:37command uh I will like to make it
3:25:40bold vacuum okay so if you want to run
3:25:45vacuum command we simp simply first of
3:25:46all we will write personent SQL so then
3:25:48we will write
3:25:50vacuum right and then we will write
3:25:52table name that's it that's it it will
3:25:54delete the files now here are some
3:25:56points that you need to take that you
3:25:58need to note just take your pencil or
3:26:00pen just write those points first of all
3:26:03when we run this vacuum
3:26:05command nothing going to happen why let
3:26:08me run it take it nothing nothing will
3:26:12happen nothing will happen I'm I'm sure
3:26:14I'm sure bro trust me Trust me trust me
3:26:17trust me I don't break
3:26:19trust so you can trust me so it it is
3:26:23just running everything that is required
3:26:25and still I will see the partition I
3:26:27will I I will let you know like why it
3:26:29didn't delete anything because you just
3:26:31mentioned that it physically deletes the
3:26:33data I will show you and I will tell you
3:26:36the reason as well so when I go to this
3:26:40external table and when I just refresh
3:26:42it I still see like all the partitions
3:26:45or all the pocket files see I'm just
3:26:47refreshing it again and again now what
3:26:49the thing is the default timestamp or
3:26:53let's say the default time frame to
3:26:55actually delete the data is 7 days that
3:26:58means even if your parket files are not
3:27:02being used by the tables but if those
3:27:05party pocket files are less than 7 days
3:27:08old that means they will be there that
3:27:10is by
3:27:11default that is by
3:27:14default so this P these pocket files are
3:27:17just created so these are not seven days
3:27:19old but if I run the same command after
3:27:21seven days these two pocket files these
3:27:24two like oh oh oh oh oh oh like this one
3:27:27and this one will be deleted but now
3:27:31these are not 7 days old so it is fine
3:27:33it is fine what if I want another inter
3:27:36question what if I want to drop or
3:27:39delete the data right now right now I
3:27:41don't care about the uh let's say
3:27:45Default Time there are two ways way
3:27:48number one you can set the default time
3:27:50frame you can just change the default
3:27:52time frame I can just keep it like maybe
3:27:54I can enhance it obviously you can ask
3:27:56me to just enhance the time frame I can
3:27:57just keep it like for Interval 365 days
3:28:00or there's one more command that I can
3:28:01add within this vacuum I can say
3:28:05retain zero
3:28:07hours now if I run it it will delete it
3:28:11it will delete it now I will not run it
3:28:13so I I can just make a separate
3:28:16command for it let's call it as retain
3:28:19zero
3:28:22hours okay percentage
3:28:29MD percentage MD and I want to make it
3:28:33bold vacuum retain zero
3:28:37hours perfect and here I can just paste
3:28:39it so if I run this it will just
3:28:42definitely instantly drop these uh
3:28:45pocket files and I will show you at the
3:28:47end when we will be just finishing our
3:28:49session so I will just show you how you
3:28:50can do it so do not worry you you will
3:28:52get everything that you want right now
3:28:54everything so let me just correct it
3:28:57because we are just explaining vacuum
3:28:58command here yeah perfect so this was
3:29:01all about vacuum command how we can just
3:29:04drop the data and all now we want to
3:29:07know how we can optimize our tables how
3:29:09we can optimize our tables so let me
3:29:12tell you how you can do the
3:29:14optimization okay that's a really
3:29:16interesting topic and that is the topic
3:29:18that everyone is looking for because
3:29:19everyone has data everyone knows how to
3:29:21create Delta tables but not everyone
3:29:23knows how to optimize those tables to
3:29:25improve the performance and that's where
3:29:27you can be an outlier in the crowd yep
3:29:32let let me tell you let me tell you I'm
3:29:33here to tell you everything so first of
3:29:35all I will create a new vacuum not
3:29:38vacuum command like vacuum command is
3:29:39like in my head right now so I will
3:29:43first of all tell you let's see the
3:29:45headings like what we I have given
3:29:47before okay data versioning
3:29:51delete Delta functionality oh yeah it's
3:29:54fine it's fine let me make it
3:29:57bold
3:29:59and I will call it
3:30:01as optimization command so okay I can
3:30:04just create a heading
3:30:06then okay
3:30:08H3 H3 H3 I will say table Delta
3:30:14tables Delta table
3:30:18optimization if I want to optimize my
3:30:20table how I can optimize my table how
3:30:24how how how I will show you the live
3:30:26demo as well don't need to worry yeah
3:30:29yeah I will show you
3:30:33everything
3:30:36so let's say we have a table called
3:30:38external table and if I want to improve
3:30:40the performance first of all you can use
3:30:43optimize command so first of all we will
3:30:45write here optimize
3:30:48percentage MD and then I can make it
3:30:50bold and I can call it as
3:30:55optimize optimize
3:30:58okay when we R run when we Russ when we
3:31:02run optimize command on top of our table
3:31:05it optimizes the table but I know but
3:31:08what exactly it happens like what
3:31:11exactly it does to our tables so that it
3:31:13is optimized so I will tell you let me
3:31:15first show you the
3:31:17result select T tricks
3:31:20from sales db. external table and I will
3:31:24just show you the time it is taking to
3:31:26read the data so it took 0.9 in seconds
3:31:29right now okay now I will just add a
3:31:32cell here and I will write
3:31:36optimize oh let me just add
3:31:40SQL percentage SQL optimize and then I
3:31:43will write sales dbx table let me just
3:31:47run it let me just optimize my table and
3:31:50now I will just show you the time it has
3:31:54taken
3:31:55okay
3:31:57so
3:31:59now it has taken 0.93 seconds now we
3:32:03will say hey earlier like it took 0.99
3:32:06second and now it it took just 0.93
3:32:08second obviously this is a very small
3:32:11data set very small just eight records
3:32:14you can actually see its powers or its
3:32:17results when you use it with the bigger
3:32:19table like bigger data sets you will see
3:32:23that it is almost almost 30 to 40%
3:32:26faster when you use optimized command
3:32:28the next command is z order
3:32:32by and I will show you what exactly it
3:32:34does don't worry I I'm just showing you
3:32:36the commands right now so percentage
3:32:41MD Now I'm
3:32:43hungry literally bro I'm just
3:32:46continuous speaking continuous I can't
3:32:48speak I am continuously speaking since
3:32:51the
3:32:55morning no worries no worries no worries
3:32:59no
3:33:01worries
3:33:03so now I will show you the other command
3:33:07which is z order
3:33:11by Z okay these sounds these sound cool
3:33:15like we you also want to know like what
3:33:16is happening behind the scenes I will
3:33:18tell you just just hold on so Z order by
3:33:21command let me just run it and how we
3:33:24can run it obviously first of all we
3:33:26will write percentage SQL and this is a
3:33:28kind of addition to uh optimize command
3:33:32Okay in in terms of architecture as well
3:33:34and in terms of code as well so first of
3:33:36all we will write
3:33:37optimize and then table
3:33:39name and then we will write z order by
3:33:43and then we need to just Define a maybe
3:33:45any Colum let's say ID let's say ID and
3:33:49I think I need to use single code I
3:33:50don't know we can just test it or
3:33:52without it is fine yeah let's let's run
3:33:56it okay and now we can just
3:34:00see
3:34:02select estx from CS db. external
3:34:07table okay now it took wow now you can
3:34:11see the difference 0.37 seconds when we
3:34:14use Z order by command
3:34:16and it it is I think almost almost
3:34:191/3 it took like 1/3 of the total time
3:34:22can you imagine one3 of the total time
3:34:26wow so what it has done like what
3:34:29exactly it has done I will tell you let
3:34:31me first show you the folder like how it
3:34:34looks like right now now you will see
3:34:37another folder like another pocket file
3:34:39yeah this was
3:34:41expected and that means when we just
3:34:44check the Delta log it will
3:34:46say remove all the partitions like all
3:34:50the pocket files remove all the pocket
3:34:52files all the bucket files and just take
3:34:56one but an we
3:34:58didn't I'm just taking my name like for
3:35:01the first time like obviously I haven't
3:35:03taken my name anch it sounds weird and
3:35:06good as well I don't know so you will
3:35:08say
3:35:09anch why we are reading this partition
3:35:11because why it has Rewritten the
3:35:13partition now we haven't deleted
3:35:15anything now now we haven't updated
3:35:16anything now we haven't done anything
3:35:19why you will get to know everything with
3:35:21the architecture behind the uh optimize
3:35:24command and Z order by command let me
3:35:25just show you the Delta log first to
3:35:27confirm my
3:35:28statement so I will simply go
3:35:32here I will check the latest log in
3:35:34which it will
3:35:37say I will go to edit see it has said
3:35:40remove this
3:35:42one remove this one which was for the
3:35:45two initial files and add this one which
3:35:48is 3 f2c and I should see this as well
3:35:52in my containers 3 f2c just remember
3:35:55it 3 f2c which is this one okay only one
3:36:01uh uh only one pocket file so obviously
3:36:05instead of deleting this one because I
3:36:07didn't know the name that means the
3:36:09whole like whole intent is removing all
3:36:12the other partition removing all the
3:36:14other partition that that is the intent
3:36:15I thought like this is the latest one
3:36:16but three of two is the latest one I
3:36:18didn't know the name the whole intent is
3:36:20removing all the partitions do not get
3:36:22me wrong the whole intent is removing
3:36:25all the partitions now we will know why
3:36:27it happened and we we are really excited
3:36:30don't worry I I will tell you so what
3:36:34exactly
3:36:36happened let me just create a
3:36:40blank sheet for you all so that you can
3:36:43understand well what exactly happen let
3:36:46me tell you do you know do do you need
3:36:49this PP I can just give it to you as
3:36:52well so what exactly happened jokes
3:36:54apart let's be serious okay this is our
3:36:58partition one this is our partition two
3:37:01these are the only two partitions which
3:37:02were in use right yeah third one was
3:37:05removed when we just used version like
3:37:07when we just restored our table so that
3:37:08means the third partition was not in use
3:37:10because we didn't refer it right so we
3:37:13we just had like these two partitions
3:37:15okay okay let me write it as
3:37:18well like partition one or let's say
3:37:21pocket one like it's up to
3:37:23you partition two okay so these two
3:37:27partitions yeah these two partitions now
3:37:30when we
3:37:32ran optimize
3:37:36command optimize command let let me
3:37:38create another partition so that you can
3:37:40just understand better so let's say this
3:37:42is our partition zero let's say
3:37:46partition zero because you will
3:37:48understand better let trust me trust me
3:37:50trust me so this is your partition
3:37:54four you do not need to take notes right
3:37:57now because it's high time to understand
3:37:58the concept first and in case of taking
3:38:00notes you can just rewatch this part do
3:38:02not do not uh get yourself distracted
3:38:06right now so listen carefully okay so
3:38:09these were the four partitions which
3:38:11were on the table right okay this is
3:38:15this is s
3:38:16now we wanted to optimize this data how
3:38:19we can optimize it first of all you
3:38:21should know how
3:38:23optimization is defined in case of Delta
3:38:25tables so it is always listen listen
3:38:28listen to me carefully it is always
3:38:32better to read from lesser partitions
3:38:36what did I
3:38:37say it is always better to read from
3:38:42lesser partitions even if they are
3:38:44bigger in size that means that means if
3:38:48we have four partitions okay let's say
3:38:52of uh let's say 1
3:38:55GB right and then 1
3:38:58[Music]
3:38:59GB
3:39:01then 1 GB then of 1 GB if I want to read
3:39:074 GB of data distributed among four
3:39:10partitions versus
3:39:13versus I just have
3:39:16two partition partitions of 2 GB
3:39:20each of 2
3:39:22GB partition
3:39:24one 2 GB
3:39:28okay partition
3:39:31two 2 GB okay it is always better to
3:39:35read from these two partition this is
3:39:37the fundamental and this is the
3:39:39definition of optimization in the world
3:39:42of uh partitions or in the world of
3:39:44Delta Lake okay this is the thing that
3:39:46you should know okay now when we run
3:39:49this optimized command what it does what
3:39:51it does it just combine all these small
3:39:55small partitions into bigger partitions
3:39:59so these four partitions get
3:40:01combined these four partitions get
3:40:04combined to create only two partitions
3:40:07to improve the efficiency to improve the
3:40:10performance to improve the
3:40:13reability this is the functional ity of
3:40:16optimize command this is the
3:40:17functionality of optimize command okay
3:40:22okay now when we ran Z order by on top
3:40:26of this optimize command as well what
3:40:28exactly happened let me tell
3:40:30you when we ran Z order
3:40:37by Z order by on top of this optimize
3:40:42command as well so obviously it first
3:40:46created two big partitions okay that's
3:40:48fine but but but but but but the Z order
3:40:52by command sorted the data based on ID
3:40:56column ID we mentioned ID in our Z order
3:41:00by we can put any column but it is
3:41:02always recommended to use that
3:41:04particular column in Z order by command
3:41:06which will be using in we Clause right
3:41:08to just filter out the data it will sort
3:41:12this data it will sort this data in
3:41:16let's say sending or descending order so
3:41:20in in both the partitions in both the
3:41:22partitions
3:41:24yes it in both the partitions it doesn't
3:41:28care how data was distributed among
3:41:30these partition it doesn't care let's
3:41:32say in this we we had like eight records
3:41:351 2 3 4 5 6 7 8 perfect let's take that
3:41:37example so that you can completely
3:41:39understand this concept and after this
3:41:41session you will feel like we have we we
3:41:45know everything that we need to
3:41:46know ah so let's do it let's do it let's
3:41:51do it so let's say we have here three
3:41:57comma 7 IDs right here we have 1 comma
3:42:038 okay here we have 2 comma 5 here we
3:42:10have what's left 4A 6 okay now it will
3:42:16what it will do first of all it will
3:42:17optimize the data optimize this data
3:42:19into two partitions so at the same time
3:42:23when we apply Z order by it will write
3:42:25here let me use green
3:42:281 1 2 3 4 and here in this partition it
3:42:34will say 5 6 7 8 Simple it sorts that
3:42:40data as well and yes this is a kind of
3:42:43indexing that we have in SQL traditional
3:42:46databases we use indexing to improve the
3:42:48query performance we have Z order by
3:42:50command in place of uh
3:42:54indexing it will first create big
3:42:56partitions and it will sort that data as
3:43:00well within that
3:43:01partition what is the biggest advantage
3:43:04of it what is the biggest advantage of
3:43:05it it's called Data skipping it's called
3:43:08Data skipping let me write it for
3:43:12you bro I'm telling you I I had to refer
3:43:15so many resources just to gain knowledge
3:43:17regarding this and now you are getting
3:43:19everything under one video you should be
3:43:21focused
3:43:23enough it's called Data skipping what it
3:43:26will do let's say I applied a select
3:43:28statement select estx from my table
3:43:32where ID equals to 3 where ID equals to
3:43:383 if I am not using Z order by command
3:43:42it has to it it has to enter each and
3:43:47every single partition because it
3:43:49doesn't know if three is here or here
3:43:53but when we use theorder by when we use
3:43:56indexing it knows I have found three
3:43:59here and this data is sorted in the
3:44:01ascending order that means we do not
3:44:03need to read this partition we do not
3:44:06need to read this partition this
3:44:11one understood that was all all about
3:44:18optimization that was all about
3:44:21optimization Z order by optimize
3:44:25command I have tried my best to explain
3:44:29you these Concepts and trust me these
3:44:31concepts are really really hard to find
3:44:34these the like the resources to actually
3:44:38get the information regarding these
3:44:40things are really hard to find but now
3:44:42you have me so you do not need to worry
3:44:46so this was all about Z order by command
3:44:49and optimize
3:44:51command got
3:44:52it okay that's good now I'm happy that
3:44:55you got the concept now I'm really happy
3:44:58so now you know what exactly happened it
3:45:02has combined all the partitions into one
3:45:06and it has sorted the data as well and
3:45:07now you have seen the performance as
3:45:09well it is improved by see almost almost
3:45:14it is reduced to one third that is a big
3:45:16big big achievement earli it was taking
3:45:18almost 1 second now it is taking 0.37
3:45:21second now just replace it with hours
3:45:24like 1 hour and 37 minutes not oh sorry
3:45:27s sorry 90 minutes and 37 minutes see
3:45:311.5 hour only 30 minutes it is equalent
3:45:35to that now you will feel the
3:45:38difference one day or just 0.3
3:45:42days got it
3:45:45this is the power of optimization
3:45:47techniques that we have within Delta
3:45:49Lake you should know you should be aware
3:45:52of these
3:45:54techniques I know now you have learned
3:45:57all these
3:46:00techniques right happy now you have
3:46:04everything in place for Delta Lake for
3:46:08data breakes everything that's not the
3:46:10end of this session we have something
3:46:12special to cover that as well what's
3:46:14that
3:46:15so after covering everything within
3:46:18Delta Le and like there's like so many
3:46:20stuff obviously now you have that base
3:46:22ready to explore to learn now you can
3:46:25try something new now if you will be
3:46:28just reading some random documentation
3:46:30regarding any update you will feel
3:46:31connected you will feel connected
3:46:33because now you know the basics now your
3:46:36Basics are
3:46:37clear that is the sole reason behind
3:46:40this
3:46:41video behind this session
3:46:46right and this is all about Delta Lake
3:46:51now let's cover one more important thing
3:46:55what is that so when you just see the
3:46:57left hand side which is
3:47:00called where is
3:47:03that
3:47:05workflows now we need to cover workflows
3:47:07as well because this is really really
3:47:09important and we will just conclude our
3:47:12session with this workflow session
3:47:14because we need I need to show like how
3:47:16you can orchestrate your work within
3:47:18data bricks how you can orchestrate your
3:47:20notebook so for this particular session
3:47:23we will create two notebooks notebook
3:47:24one notebook two and we will just put
3:47:26some random code and I will show you how
3:47:28you can orchestrate it how you can
3:47:30perform the whole data
3:47:32upload and I am really happy I know that
3:47:36I have been talking since the morning
3:47:37but now I am feeling happy because now I
3:47:39know that I have successfully shared all
3:47:41the knowledge that is required to be a
3:47:44Pro Data brakes developer and obviously
3:47:47I have provide the resources as well
3:47:48from where you can learn ppar as well
3:47:50and you promis me that you will learn
3:47:53ppar as well and just learn the ppar as
3:47:56well after watching this last session
3:47:58and just see yourself in the mirror and
3:48:01just say good job just say
3:48:05this so now without wasting in time
3:48:07let's cover this workflow session as
3:48:11well so we need to cover this topic as
3:48:15well by the way how's my new history so
3:48:18I just went to Saloon to have a haircut
3:48:20and I was sitting there and I just
3:48:21realized that we need to just cover this
3:48:24topic as well this is really really
3:48:27really really really important why as
3:48:31the name suggest incremental loading
3:48:33with auto loader obviously in the real
3:48:36world we just deal with data and we do
3:48:40not like do the full load every time we
3:48:43load the data increment ly that's why we
3:48:46need to understand how we can perform
3:48:48incremental loading in data brakes and
3:48:51do you know what you can perform
3:48:52incremental loading like this why and
3:48:56how because of this because of
3:49:01this super cool function that is
3:49:05autoloader what is this and what we can
3:49:08expect in this particular function it's
3:49:11not basically a function you can say
3:49:12it's a kind of framework or it's a kind
3:49:15of architecture that data mix performs
3:49:17so let me just give you a scenario so
3:49:21let's say you have a source
3:49:25okay you have a source and you have
3:49:28let's say pocket files let's say let's
3:49:30say pocket files like you have pocket
3:49:32file
3:49:33one only one okay
3:49:36then you want to
3:49:39just load this data to this container
3:49:44let's say a destination folder anything
3:49:46right this is your
3:49:48destination by the way how is my looking
3:49:51I'm seriously asking bro is it looking
3:49:54good see I'm so so so possessive for you
3:49:57I was just having a haircut and I I was
3:49:59just thinking about you guys like how
3:50:01you can learn this topic as well see how
3:50:04possessive possessive I am for you for
3:50:07your
3:50:09learning I I love my data fam love you
3:50:12love you love you by the way I have I'm
3:50:15like wearing pink color now
3:50:17so so okay okay okay jokes apart let's
3:50:20focus okay so let's say you want to just
3:50:22transfer this file to this destination
3:50:24and this will be
3:50:26P1 then let's say after 1 hour or let's
3:50:29say next day you have another file
3:50:33P2 now now you just want to
3:50:38transfer only this data you do not want
3:50:42to read this data again because you have
3:50:44already loaded this data in your
3:50:46destination why you would be just
3:50:48reading this data again and again
3:50:50obviously you will not read it then we
3:50:53will use autoloader which will only pick
3:50:57this data only pick this data and it
3:50:59will just write it here which is
3:51:04P2
3:51:06yes see how powerful it is so how we can
3:51:11actually perform this so basically we
3:51:13use something called
3:51:15as just write it for
3:51:18you streaming data frame what is
3:51:21streaming data frame streaming data
3:51:23frame is basically like similar to the
3:51:26like standard or the normal data frame
3:51:29that we create in a PES part this is
3:51:32unbounded data frame that means it will
3:51:35be a continuous data frame based on a
3:51:38stream as we just mentioned that our
3:51:41source will be having data continuously
3:51:44right let's say today it will be one
3:51:46file let's say tomorrow it will be
3:51:47second file or or or it will have one
3:51:51file now after after 2 minutes it will
3:51:54be having second file or it will be
3:51:57having one file now and after a few
3:51:59seconds it will be having second file so
3:52:01it is continuous form of data it is
3:52:03continuous stream of data so we create
3:52:06something called as streaming data frame
3:52:09for these types of situations and for
3:52:11autol loader we have to have to create
3:52:14streaming data frame so that it can just
3:52:16continuously incrementally load the data
3:52:19into our into our
3:52:21destination if this seems to be
3:52:24difficult to implement the answer is no
3:52:26it is very very simple let me show you
3:52:28how you can do it and for that we will
3:52:30first create a dummy source and dummy
3:52:33destination and we will show you like
3:52:36how you can actually perform out order
3:52:37let's do it so first of all we will
3:52:40simply add a heading and we will simply
3:52:42write
3:52:44uh uh uh Auto
3:52:49loader simple perfect and now let's
3:52:52prepare the scenario for that so I will
3:52:54just go to my data rate and I will just
3:52:57create a container and I will call it
3:52:59as uh Al
3:53:02Source
3:53:04okay Al source autol loer source
3:53:09simple what happened man okay we are not
3:53:11using
3:53:13uppercase here Source
3:53:17simple
3:53:19perfect and now let's also create
3:53:24Al
3:53:26uh Sy or destination yeah Al destination
3:53:30so these are like basically two
3:53:31containers that we will involve in our
3:53:33scenarios so simply create we have
3:53:35created two scenario like two containers
3:53:38so now I will upload some data in my Al
3:53:41source and I have some dummy pocket file
3:53:45so I will just upload that and let me
3:53:48just upload that
3:53:49one I will simply click on upload and
3:53:52then browse for files as you can see I
3:53:54have uploaded this paret file don't
3:53:56worry I will just provide the link to
3:53:57this paret file and I will just upload
3:53:59this in my GitHub repository you can
3:54:01download it do not need to worry bro
3:54:04just click on upload and now this file
3:54:07is uploaded right let's go to our datab
3:54:11breaks so first of all as I just
3:54:13mentioned that we need to create
3:54:14something called as streaming data frame
3:54:17it is very very very similar to The
3:54:19Standard data frame very similar even
3:54:21the syntax let me show you so let me
3:54:25just add a heading not a heading just
3:54:27title I will call it
3:54:30as let's make it bold
3:54:33streaming data
3:54:37frame right okay so now I will simply
3:54:41say DF equals
3:54:44par. read now instead of using read I
3:54:47will use read stream simple simple
3:54:50simple then I need to specify the
3:54:54format now you will ask what is the
3:54:56format
3:54:57pocket no the format is let me show
3:55:01you cloud files why yes because when we
3:55:05use aut loader we just use a generic
3:55:10file format
3:55:14then then I was really thirsty sorry bro
3:55:17so then we just Define the file format
3:55:21of our Cloud file in our scenario it is
3:55:23parket now we will Define it and our
3:55:26lovely data breaks has already
3:55:27recommended us thank you so much by the
3:55:29way we already know so still thank you
3:55:32to save our time okay so once it is done
3:55:36then we need to define something called
3:55:39as schema location what's that let me
3:55:42tell you why we need it he is very smart
3:55:45he's very he's not letting me to explain
3:55:47he's just he's a spoiler man just
3:55:49kidding so let me just show you like
3:55:51what do I
3:55:53mean so basically we just discussed that
3:55:56we have a source right yeah we have a
3:55:58source we have P1 we have P2 so
3:56:03basically we need to store the schema of
3:56:06each file so that our destination table
3:56:10which will be using right stream method
3:56:13can actually use that
3:56:18schema to write the data in the
3:56:20destination folder simple and for this
3:56:23thing we store the schema in a specific
3:56:26location and we call it as checkpoint
3:56:29yeah it's a kind of checkpoint so let's
3:56:32create that as well so now you
3:56:34understood right why we need to store
3:56:35the schema because it will be referred
3:56:38by the read uh sorry right stream method
3:56:41right right right right okay so now we
3:56:44we will simply say dot
3:56:45option cloud files
3:56:49oops cloud
3:56:52files dot schema
3:56:56location okay let's use this okay so now
3:57:00it has given us some recommendations so
3:57:02first of all my source is Al Source
3:57:06obviously and then it's not sales bro
3:57:08come on man come on so I will just
3:57:11create a folder called checkpoint inside
3:57:13my source
3:57:15inside my source or should yeah yeah I
3:57:19can just create it like anywhere so I
3:57:20will just create a checkpoint inside my
3:57:22source or what I can do I can actually
3:57:25store this checkpoint in my sync it will
3:57:27be better it will be better yeah like
3:57:31it's not like any kind of rule that you
3:57:33need to follow but obviously it makes
3:57:36sense because it will be read by your
3:57:38right string method and right stream
3:57:40method will be pointing to the
3:57:42destination and it makes since as you
3:57:45can obviously U maintain that data
3:57:47easily so let's not put it in the source
3:57:51let's put it in the sync okay so our
3:57:53sync is Al
3:57:56destination perfect yeah perfect okay so
3:57:59this is done what's
3:58:01left then you need to Simply
3:58:05write dot
3:58:07load and we will just give the
3:58:10location and in our scenario it is Al so
3:58:14Source spelling is right right yeah okay
3:58:18and it is do we have created any folder
3:58:22inside that do we no no no no no no no
3:58:25no no no no simple only container bro
3:58:28only container we we do not need to put
3:58:29sales bro what are you doing man we do
3:58:32not need
3:58:33sales okay so let me just run it what
3:58:37will happen nothing why I will tell you
3:58:41so we have successfully created DF so
3:58:43obviously nothing will be executed
3:58:45because of lazy valuation of spark so
3:58:49now we will create the writing stream
3:58:50and when we just uh write right stream
3:58:55then it will just run this cell as well
3:58:57so what will happen in our right stream
3:58:59so we will simply say DF do WR and as
3:59:02you guessed it now we need to write
3:59:05right stream instead of right right
3:59:07right right right right right right so
3:59:09many
3:59:10rights okay now do
3:59:12format now let's say I want to just save
3:59:15it in the Delta format I love Delta so
3:59:19you can just save it in any format
3:59:20doesn't matter doesn't
3:59:22matter as we are just dealing
3:59:24extensively with Delta lake so it's nice
3:59:26to just use Delta lake so we will just
3:59:29simply write DF do right stream. format
3:59:31Delta then we need to specify something
3:59:34special what's that another checkpoint
3:59:36why why why why why why I will tell you
3:59:40then we can say okay he's very smart
3:59:43Okay what is is checkpoint location so
3:59:45basically basically when let let me just
3:59:48draw it for you here basically this is
3:59:51our
3:59:52source this is our
3:59:54destination when it has read
3:59:58P1 it has read P1 okay this is our
4:00:02streaming query this whole thing is our
4:00:04streaming
4:00:05query okay let me just write for you if
4:00:08you are not taking notes if you're not
4:00:11crafting your
4:00:12notes don't to me don't talk to
4:00:18me this is our streaming query
4:00:22perfect our streaming query has
4:00:25successfully read this file okay and
4:00:29this file is here now okay
4:00:33now it needs to store this
4:00:36information at a specific location that
4:00:41this file is already read
4:00:44streaming query needs to save the
4:00:48current state of the
4:00:51stream got it obviously that's how it
4:00:55will just got get to know that this file
4:00:58is already read now it just needs to
4:01:00pick
4:01:01P2 simple bro so now we'll just give the
4:01:05checkpoint location oh we have already
4:01:07given that's that's really nice now
4:01:09we'll just put dot trigger obviously it
4:01:13uh it is a stream so we need to just
4:01:15tell that always look for the records
4:01:18after let's say 1 minute or let's say 30
4:01:22seconds or let's say 10 seconds because
4:01:26when we just create stream we want to
4:01:29give the trigger okay let's say if we do
4:01:32not give any value to the trigger or if
4:01:34we do not write trigger it will still
4:01:37give the default value to the trigger
4:01:39what is the default value I read it
4:01:42somewhere if I'm not wrong it is
4:01:44somewhere around 0.5 milliseconds so
4:01:47even if we do not write anything it will
4:01:50just take the default value that is 0.5
4:01:53milliseconds it is very very very quick
4:01:55so we will just say let's say processing
4:01:59time equals to
4:02:01oops equals to 10 seconds oh he's very
4:02:05he's just reading my mind bro now I will
4:02:07not give 10 I will give five
4:02:11seconds okay simple now we need to just
4:02:15write dot
4:02:17start do start and obviously we need to
4:02:20give the location where it needs to
4:02:21store the data in the destination in our
4:02:23scenario Al destination very right very
4:02:26right wow very
4:02:28good data simple I will create a data
4:02:33folder yeah okay do you want me to start
4:02:36it do you want me to start the stream
4:02:38because when I just started you will see
4:02:40some graphs as well it is very very very
4:02:42nice so I have just started
4:02:44it okay as you can see stream
4:02:47initialization when I just click on it
4:02:49you will see
4:02:51graph o so basically it shows the
4:02:54current state of the streaming query and
4:02:58you will see some line graphs and all so
4:03:02and it is continuous streaming right it
4:03:04is continuous data what do I mean I mean
4:03:08see the line graph and do you know what
4:03:11is this raw data this is called QP
4:03:14UPL yeah query progress log just save it
4:03:17just save this information this is
4:03:19called query progress
4:03:21log yes and this is your dashboard where
4:03:24you see the values that data is coming
4:03:26see it just process this data this is
4:03:30your processing rate this is your this
4:03:31is your processing rate this is your
4:03:32input rate and this is like batch
4:03:34duration and now it will be flat because
4:03:36obviously we didn't do anything we
4:03:38didn't do anything we just uploaded one
4:03:40file it copied do you want to see the
4:03:42result this stream is really like
4:03:44working
4:03:45continuously this will not get stopped
4:03:48until and unless I stop it so it will be
4:03:51running continuously continuously so I
4:03:54will just take you to our data L to just
4:03:57show you something so as you can see
4:03:59this is our source container and we have
4:04:02only one file right it is automatically
4:04:04copied to Al destination wow and we Sav
4:04:08the data in the data folder wow it has
4:04:12copied this partition now I will do
4:04:16something else I will go to my
4:04:19containers and I will add this file I
4:04:22will only upload this file I will not
4:04:25touch anything in this notebook and you
4:04:27will see data will be copied
4:04:29automatically and incrementally that
4:04:32means this file will not get picked this
4:04:35time only the new file will get will get
4:04:37picked let me show you I will simply
4:04:39click on upload and browse file I have
4:04:41added this file now I will click on
4:04:44upload button and you will see that it
4:04:47will get copied obviously you need to
4:04:49just wait for 10 seconds maybe it's like
4:04:51in progress of 10 seconds so we will
4:04:53just wait for 10 seconds 10 9 8 7 6 5 4
4:04:593 2 1
4:05:02Boom come on man I a child
4:05:08yeah I'm a child so you will just see El
4:05:12destination will have two files now only
4:05:15two files not one not three only two and
4:05:18that's the result of our incremental
4:05:20loading let me show you in the data
4:05:23folder
4:05:25boom boom boom boom I told you I told
4:05:29you bro I told you so now it has
4:05:32successfully loaded this data now as you
4:05:36can see another Spike because we
4:05:37uploaded another file so you can just
4:05:39continuously add the data it will
4:05:41continuously just transfer this data now
4:05:44just a tip bro please click on this
4:05:47interrupt button because cluster is
4:05:51running and it is very very expensive so
4:05:54just click on this interrupt button so
4:05:55that you can just interrupt the query
4:05:57and your cluster is turned off and you
4:06:00are fine obviously if you're using free
4:06:02account it doesn't matter but obviously
4:06:03if you're working in a company you
4:06:05should know these things that you cannot
4:06:06let your query running like overnight
4:06:09obviously in the real world scenario
4:06:10like we let our query to run because we
4:06:13are streaming data but if you are if you
4:06:15do not want to intentionally run it then
4:06:19just stop it right away after doing some
4:06:21testing after learning just stop it
4:06:22right
4:06:23away so this was all about autoloader
4:06:28and trust me I was having a haircut and
4:06:32I just got to know that I want to
4:06:33discuss this area as well I love you so
4:06:38much and you know this and I know that
4:06:41you love me more so
4:06:44this was all about
4:06:46autoloader so are you excited to cover
4:06:50our last session which is workflows by
4:06:53the way this is really really important
4:06:55so we will just have a look at this
4:06:58session as well so just a prequest for
4:07:01this one we will just create as I just
4:07:03mentioned like we will create two
4:07:04notebooks some random notebooks and we
4:07:06will
4:07:07just uh create very very basic let's say
4:07:11statements or code for that because we
4:07:13just need to learn work about workflows
4:07:14instead of just coding everything yeah
4:07:17so I will simply go to my workspace and
4:07:19I will select my workspace which is this
4:07:21data brakes master class and then I will
4:07:23create two notebooks first of all
4:07:25notebook
4:07:26one and I will name it as notebook one
4:07:29let's keep it simple bro so let's say
4:07:32use print command that's it so print we
4:07:36can say
4:07:38hellan simple yeah right and obviously
4:07:42this is done
4:07:44and then we can again go to workspace
4:07:46select your workspace select your folder
4:07:49then we will create Notebook
4:07:522 okay
4:07:55perfect notebook two okay and here we
4:07:57will
4:08:00say we love UNL obviously you do
4:08:04obviously you do I know I know I know I
4:08:06I just read your
4:08:07mind so I'm just kidding don't need to
4:08:11worry but I know you do so
4:08:14these are like two notebooks and now
4:08:16let's learn workflows what if I want to
4:08:20I I will just give you an overview so
4:08:23this is my notebook one right this is my
4:08:26notebook two why do we use workflows
4:08:29workflows are basically used for
4:08:32orchestration let me type it here it is
4:08:35used for
4:08:38orchestration when we need to
4:08:40orchestrate our work let's say this is
4:08:42our notebook one as we just mentioned
4:08:44let's say one and this is our notebook
4:08:47two so what if we want to run the second
4:08:52notebook after one and similarly let's
4:08:54say we have multiple notebooks so we
4:08:56want to create a flow we want to create
4:08:58a job we want to create a data pipeline
4:09:02so if we want to do that we can simply
4:09:04go to workflows and we have this option
4:09:06available and data p and we can just
4:09:08learn how we can just create workflow
4:09:11using our notebooks and here is your
4:09:13friend will tell you each and everything
4:09:15so let me go to
4:09:16workflows click on it and then you will
4:09:19see something at the corner create job
4:09:22you just need to click on it because
4:09:24create job job is similar to pipeline or
4:09:26let's say workflow you can use any name
4:09:29okay so this is the page that we see do
4:09:31not need to worry do not need to worry
4:09:33do not need to worry first of all let's
4:09:35see some of the things here in the right
4:09:37hand side job ID which is the ID of this
4:09:40particular pipeline then obviously
4:09:42create orang lamba run as orang lamba
4:09:45tags we can just add it if you want to
4:09:47add some description we can add it
4:09:50lineage no lineage information for this
4:09:52job because there's no dependency so
4:09:54this was all about the job details then
4:09:56we have schedule and triggers if we want
4:09:58to schedule our job let's say it should
4:10:01run at 2 p.m every day we can do that we
4:10:05can add a trigger if we have a
4:10:10parameter here if we have a parameter we
4:10:13can just Define the parameters as well
4:10:16if we want
4:10:18notifications if we want notifications
4:10:20we can also get the
4:10:22notifications yes datab brakes provides
4:10:26us all the important users all the
4:10:30important features within the workflows
4:10:33that are required to orchestrate the
4:10:35pipelines let me just click on this
4:10:37arrow button and let's hide it first of
4:10:39all we need to name our pipeline
4:10:40obviously if we are just importing any
4:10:42activity if we are just running any
4:10:44activity we should just name it right so
4:10:46let's say I want to name it as
4:10:48run notebook one this is my activity
4:10:52this is a kind of activity if you are
4:10:54familiar with the your data Factory this
4:10:55is a kind of
4:10:57activity
4:10:59simple
4:11:00then we need to Define we need to just
4:11:03Define which notebook we need to run we
4:11:05need to just go to this folder and we
4:11:07will pick notebook one because we want
4:11:09to run notebook one in the first step
4:11:11then we need to pick the cluster we will
4:11:13just pick the pick our own cluster that
4:11:14we have created okay and it's all done
4:11:18we do not need to provide any parameters
4:11:20because we do not have any parameters
4:11:21within this notebook this is a very
4:11:22simple notebook that we have then I will
4:11:26I
4:11:27will I will click on
4:11:29it wait wait wait this card changes no I
4:11:33don't want to cancel it okay let me
4:11:35click on this canvas okay
4:11:39now if you just see here dependent
4:11:42libraries
4:11:43if your notebook has some dependent
4:11:45libraries still you can just mention it
4:11:48here as well so it's very handy it's
4:11:51really handy and again the parameters if
4:11:54you have any parameters you can just
4:11:55Define it here in the notebooks but as I
4:11:58mentioned we do not have anything then
4:12:00this is this is very important retries
4:12:03if due to any reason our notebook or
4:12:06let's say our activity fails so it can
4:12:09retry let's say two times three times
4:12:11four times we can Define it here as well
4:12:14okay and obviously this is the UI in
4:12:17which you do not need to code anything
4:12:19but if you click on Json it will show
4:12:21you the code like which parameter you
4:12:23need to refine because obviously for
4:12:25parameter we need to give key value
4:12:26pairs right but UI is better UI is
4:12:29better UI is better okay let me drag it
4:12:33here so this is my notebook one this is
4:12:36my task
4:12:37one and if I just click on this and this
4:12:41like this is my canvas that we I can
4:12:43just um keep my mouse here I can rotate
4:12:46everything right so so once you are
4:12:49satisfied with all your configurations
4:12:52you can just click anywhere on the
4:12:53canvas and just click on Save and
4:12:54continue so it will just save your work
4:12:57here now if you want to include the
4:13:00other activity let's say Notebook 2 you
4:13:02can click on add task and I can pick
4:13:05notebook so it has just linked The
4:13:07Notebook after the previous notebook now
4:13:08we just need to do the configuration
4:13:10same way that we have done for Notebook
4:13:12one so I will name it as run Notebook
4:13:162 my
4:13:19activity is run Notebook 2 then I can
4:13:22just pick the same cluster and then the
4:13:24important step we need to just pick The
4:13:25Notebook 2 that it needs to run okay
4:13:29yeah everything is fine yeah and okay
4:13:32okay okay one important thing I found so
4:13:35now you can see it is dependent on the
4:13:37notebook
4:13:38one because we have selected to run
4:13:41notebook two after notebook one so it is
4:13:43dependent on Notebook one so this
4:13:45was really an important step that I
4:13:48wanted to show so once it is done yeah
4:13:51now we can click on create task now we
4:13:53can click on create task so it will
4:13:55create the task for like all these two
4:13:57activities now now it is
4:14:00sorted now we can run our pipeline that
4:14:04we have created using workflows in data
4:14:06braks do you want to run it let me run
4:14:08it run
4:14:10now so it it triggered our pipeline and
4:14:13let me click on view run to just monitor
4:14:15our pipelines see it is running our this
4:14:18activity for now and once it is done it
4:14:20will run this activity okay this is done
4:14:22this is this ran successfully you can
4:14:23see a small tick and now this is running
4:14:26this activity so I also want to see tick
4:14:28here as well okay okay perfect perfect
4:14:32perfect we have successfully created our
4:14:35Pipeline and ran it as well and these
4:14:38are all the details that you can see the
4:14:40job ID job run ID and and it was
4:14:43launched manually yes I clicked on Rand
4:14:45button started ended everything is
4:14:48written here everything is written here
4:14:51see everything is written here every
4:14:54every everything so this was all about
4:14:57the
4:14:59workflows see and when I just click on
4:15:01this notebook I can also see the output
4:15:03see this is the
4:15:07output this is very very very powerful
4:15:10let me go to this run and let me click
4:15:13on the second notebook I should see we
4:15:16love Anam I wanted to see
4:15:18this see so this was all about the
4:15:21workflows that we have created within
4:15:23data breaks now I believe that you have
4:15:28a strong knowledge and data breaks now
4:15:32you have a strong base now you are all
4:15:35set to explore the world of data breaks
4:15:37and now you can just build some projects
4:15:41now you can just do so many so many
4:15:43stuff and here is a tip for you what's
4:15:46that I I knew it because after watching
4:15:49the session you would love to just
4:15:51explore more and more stuff so here's a
4:15:54tip for you I have created endtoend your
4:15:58data engineering project and so far I
4:16:01have created two projects so in both the
4:16:04projects I have used data braks in which
4:16:07I have used p park and which I have used
4:16:09everything that you would be
4:16:13inclined to learn you would love to
4:16:16learn and if you want to check those
4:16:19projects just click on the video link in
4:16:22the description and just put your
4:16:25projects in your resume so that you can
4:16:27highlight your skills within data bricks
4:16:29and one small request if you love my
4:16:32dedication if you love my hard work and
4:16:35if you want me to continue this content
4:16:38continue the these insightful videos
4:16:41insightful sessions so please subscribe
4:16:43to this Channel and share this Channel
4:16:46with all the data Enthusiast and like
4:16:48this video and please please please drop
4:16:51a comment because this will really
4:16:53motivate me a lot and till then bye-bye
4:16:58take care and happy learning see you

About this transcript

This page contains the full transcript of Databricks Tutorial (From Zero to Hero) | Azure Databricks Masterclass by Ansh Lamba, generated from the public captions YouTube serves with the video. The transcript has 40,026 words across 5,660 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.