YT YouTube2Text

YouTube2Text → Transcripts → Databricks Was HARD Until I Learned These 21 CONCEPTS (Databricks Ansh Lamba)

Databricks Was HARD Until I Learned These 21 CONCEPTS (Databricks Ansh Lamba) — Transcript

by Ansh Lamba · 42,215 words · 6,967 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Data-bricks is that single technology
0:02right now, which is the most in demand
0:04in the field of data engineering. But it
0:07has so many components and features that
0:10it can be overwhelming to grasp all at
0:13once. That's why I have created this
0:15four plus hours long tutorial covering
0:1821 plus key concepts with practical
0:21hands-on examples to help you understand
0:24and master Data-bricks effectively. This
0:26is completely beginner friendly and
0:29designed to take you from zero to a pro
0:32level in Data-bricks and will cover
0:34everything including Apache Spark and
0:37Data-bricks architecture, Data-bricks
0:38volumes, Delta tables, Lake flow
0:41connect, Lake flow jobs, Lake flow
0:43declarative pipelines, Spark streaming
0:45with auto loader, declarative automation
0:48bundles, Data-bricks SQL warehouse,
0:50Data-bricks lake base and much more. By
0:53the end of this tutorial, you will have
0:55a solid understanding of Data-bricks and
0:58you will be able to confidently work
1:01with it. And everything is explained
1:03with amazing hand-drawn doodles with
1:06real-world examples and scenarios. So
1:09simply hit the subscribe button if you
1:10want to stay ahead in the field of data
1:13engineering and let's get started with
1:15this amazing masterclass. So now first
1:18of all, we need to understand the very
1:20first concept because only then you will
1:22be able to learn any other concept. And
1:25the concept is very simple.
1:27What is
1:28Apache Spark?
1:30Cool.
1:32What is Apache Spark? So Apache Spark is
1:34first of all very very simple. Nowadays
1:36everyone knows about it,
1:37but still let's talk about it a little.
1:40So let's say you want to process your
1:43data. Okay. You would have like
1:45processed your data in your early days
1:47of
1:48any you can say
1:51profession within the data domain. Okay.
1:54So what what was the you can say default
1:57practice? Obviously you will just bring
1:59one you will perform the
2:00um processing, cleaning, anything on
2:03that, right? Perfect, everything was
2:05nice, but with the rise of data,
2:08this single machine or your system or
2:11any virtual machine that you were using
2:14was not enough. Really? Yeah, this was
2:18not working at all.
2:19Then what was the solution? The solution
2:21was very simple.
2:22We needed to bring multiple multiple
2:26machines together.
2:28Let say 1 2 3 4 5 depending upon the
2:31size of the data, right? So when we
2:33wanted to introduce this kind of
2:36distributed computing,
2:39then we got introduced to Apache Spark.
2:42Now I know there are so many you can say
2:45frameworks like Hadoop, MapReduce, which
2:47was introduced even before Apache Spark,
2:50but it was not very very you can say
2:52efficient to process your data because
2:54it keeps on writing the data on the
2:56disk, but Apache Spark started
2:58processing the data in the memory
3:00itself, which was very very fast. And
3:01this was not the only change. Then they
3:03just started
3:05building let say your you can say SQL
3:07engine, which was building DAGs for you,
3:10which was like optimizing your code. And
3:12with the course of time, they introduced
3:13something called as C++ executors. So
3:16there was like continuous improvement
3:18going on within Apache Spark. That is
3:20why currently Apache Spark is one of the
3:23most trusted big data processing
3:26framework we use, right? Very good. This
3:29is your Apache Spark. Now very well
3:31done. Then what is this Databricks? If
3:33we have Apache Spark, everything is
3:35good. Now why do we need Databricks? So
3:38basically
3:39we need to
3:40use a kind of cluster. Cluster is what?
3:43An interconnection
3:45of these machines. These machines need
3:47to talk to each other.
3:49This requires networking. This requires
3:52a lot of things like operating system,
3:53context switching, everything, right?
3:55Because we perform multiprocessing
3:56within each machine and then we run our
3:59processes, driver process, executor
4:01process. There are like so many
4:02processes running.
4:03And everything needs to be communicated
4:06nicely, with fault tolerance, with
4:08everything.
4:09So,
4:10in order to handle this particular
4:13distributed system,
4:15it is not very easy. Okay?
4:18And Databricks, the team who built
4:21Apache Spark, yes,
4:23they
4:24figured out that okay, we have just
4:27created this framework called Apache
4:29Spark and we have provided this
4:31framework to the world as a research
4:33project. Now it's time to provide the
4:35real solution as well because I know
4:37Apache Spark can be used by the big big
4:38organizations and they can handle this
4:40distributed system, but there are so
4:42many companies who cannot handle all of
4:44these things, right? So they built
4:46something called as Databricks, which
4:49was, not is, which was nothing but a
4:52kind of managed solution which built on
4:56top of your
4:58um Apache Spark. This was the story in
5:01the initial days. Okay?
5:03In the initial days, this was the
5:05Databricks
5:06where you were not supposed to handle
5:10the cluster. You do not need to um you
5:13were not need to um you can say required
5:16to um manage this cluster, um you can
5:19say handle the um communication between
5:22the machines, nothing. You were only
5:25responsible to pick the right cluster
5:26size. Let's say you just want four
5:28machines with um 16 GB of RAM on each
5:31machine, like this. That is it.
5:34Everything was handled by Databricks.
5:36And not just this, running your Spark
5:38jobs, basically submitting your Spark
5:40jobs, everything was being handled by
5:42Databricks.
5:43And not just this, they even um
5:46introduced the notebooks. Okay? Because
5:49earlier we were not submitting the
5:51notebooks to our Spark clusters. We were
5:53submitting the Python scripts. Obviously
5:56you are just writing the Spark code in
5:57that, but these were like raw files. But
6:00data professionals like to just code in
6:02the notebook. So they even introduced
6:03notebooks that you were just um able to
6:06submit directly to the cluster.
6:08Make sense? This is your Databricks.
6:10This is your platform. This is the
6:12technology right now. But then with the
6:14course of time they introduced so so so
6:17many things as I just mentioned like
6:18optimization and then C++ based
6:20executors and so on.
6:22And now we will be just looking at all
6:24of those those concepts but do not
6:26worry. This was the basic idea behind
6:28Databricks.
6:29Even today if someone asks you like what
6:31is Databricks you can tell that this is
6:33a managed Spark platform with
6:36these these these these features and
6:38these these these features will be like
6:40very soon um displayed in front in front
6:43of you. Make sense? But this is the
6:45Databricks. This is the initial version
6:47of Databricks.
6:48Right? But then um we all know the
6:50story. Now it is like it has become the
6:53one stop solution for your AI and data
6:55solution. So now it is like completely
6:57different. But this is your Databricks.
6:59Make sense? Because a lot of people are
7:00still confused like what is the
7:02difference between Spark and Databricks?
7:04There's no difference. One complements
7:06the other one. Simple.
7:07Apache Spark is the framework.
7:10Databricks is the solution built on top
7:12of Apache Spark.
7:13Simple. Very good.
7:15So this is the concept. Now
7:17the next concept that you need to
7:19understand within the Databricks or I
7:20can say the very first concept within
7:23the Databricks
7:24is nothing but
7:26Unity
7:28Catalog.
7:32Okay? Unity Catalog or not just like
7:34what is Unity Catalog? What is the Unity
7:36Catalog?
7:37And
7:38Unity Meta Store because these two
7:41concepts
7:42both go hand in hand.
7:44So that is why let's try to discuss both
7:47together. Now what is this Unity Catalog
7:49and Unity Meta Store? Because the story
7:52of Databricks is totally changed after
7:55Unity Catalog. Databricks who like have
7:58used Databricks before Unity Catalog
8:00era, they would know like it was totally
8:02different. Okay? But now with Unity
8:05Catalog we get the centralized
8:07governance and everything. So let's talk
8:08about like what exactly is Unity
8:09catalog. And before even talking about
8:12Unity catalog, you need to understand
8:13Unity meta store. Because all of these
8:16things work in a hierarchy.
8:19Okay? Work in a hierarchy. So now what
8:21is this hierarchy? Hierarchy goes like
8:23this. First of all,
8:25you will get something called as Unity
8:27meta store.
8:30Unity meta store. Now
8:34forget about Unity,
8:36forget about the word Unity.
8:38What is a meta store? You tell me.
8:40Meta store is a location where you put
8:42all of the metadata.
8:44Simple.
8:45Right?
8:46But what metadata?
8:49We all know like metadata is the
8:52information, is the data about data.
8:55But here meta store means the
8:58information about
9:00your catalogs,
9:02your schemas, your tables, everything.
9:07Everything.
9:08And you need to
9:10you need to
9:12put everything within this Unity meta
9:14store.
9:15Makes sense? There's a centralized
9:17location.
9:18Centralized location.
9:20Cool? Just understand this concept first
9:22of all. This is a centralized
9:25you can say location.
9:26Within the meta store,
9:28we get a lot of things. A lot. Not just
9:31Unity catalog, but a lot of things.
9:35Makes sense? Very good.
9:36So when we have a catalog
9:40within this
9:42Unity meta store,
9:44that is considered as
9:47Unity catalog.
9:49But we do not
9:51refer this thing as um Unity catalog as
9:55a different thing and Unity meta store
9:56as a different thing. So whenever we
9:57talk about Unity catalog, we are talking
10:00about the catalog which is created
10:03which is created inside a Unity meta
10:06store.
10:07A lot of people get confused like
10:09um Unity catalog, Unity metastore,
10:11catalog under Unity metastore.
10:13All are interrelated.
10:16Cool? So this is the catalog created
10:18within the Unity metastore. Now you will
10:21ask me what is this catalog?
10:23Have you ever used database?
10:26Ever?
10:27Any database like Postgres, MySQL, MS
10:31SQL.
10:32So we create something called as
10:34database there, right? So catalog is
10:37equivalent to database in terms of
10:39hierarchy.
10:41Within the database we create something
10:43called as schema.
10:46Right? So you get schema as well
10:49inside this.
10:51Cool?
10:52Very good. Schema.
10:53Within the schema
10:55what do we create?
10:56You will say it's very easy. We create
10:58tables.
10:59Very good. We create views.
11:02Very good. We create functions. Very
11:04good. Yeah. We have all of these things
11:06here.
11:07All of these things.
11:09Functions.
11:11We have views.
11:14Simple. So we have all of these things
11:17here as well.
11:19Very good.
11:21So this is your hierarchy.
11:23Now what is the advantage? What is the
11:25advantage here?
11:27You get a centralized location for your
11:31entire data governance. What do I mean
11:33by
11:35one-stop solution or basically
11:36centralized location for data
11:37governance?
11:38So let say you just want to query any
11:41table.
11:42Right?
11:43You will now use
11:45three level
11:47naming space
11:48to query your table. You will first of
11:51all write your catalog name,
11:53then schema name, then your table name.
11:56That way
11:58you will know from where I am fetching
12:01the table, from which catalog, from
12:03which schema.
12:05Make sense?
12:06And it's not just about tables, it's
12:09about views, it's about functions and
12:12not just bro.
12:14It's also about now models.
12:17Oh,
12:19that is why Databricks is becoming even
12:21more popular in this AI era.
12:23You can even govern all of the models as
12:26well.
12:27Everything.
12:29And there's one more thing, but do not
12:32worry about that because that is our
12:33next concept. It is called volume.
12:38Oh, okay.
12:39Just for now, I can just give you a
12:41little bit of hint.
12:43You can even govern the raw files. Can
12:46you imagine raw files? Let's say you
12:49have some CSV files and you want to dump
12:51those files let's say in the cloud
12:53storage and you want to govern those
12:55files as well. You can just track the
12:57lineage and govern those files via Unity
12:59Catalog via three-level namespace.
13:03But do not worry about volume. We will
13:04discuss this as a next concept. Simple.
13:07Very good.
13:08And within this Unity Metastore,
13:11catalog is not the only thing.
13:13Okay. We have a lot of things. Just a
13:17few. Let me just name them. Let's say
13:19external location.
13:22External location is used whenever you
13:24want to connect your Databricks with
13:27you can say third-party cloud provider.
13:29Let's say your AWS anything because
13:31obviously Databricks is an independent
13:34platform and you can just connect it via
13:38Azure AWS anything. So you do that thing
13:42using external location. Like whenever
13:44you want to access the data. So
13:45anything.
13:46Right?
13:48You can even create the credentials.
13:52You can even create the external
13:53metadata. You can even create the
13:56recipients if you just want to share the
13:57data. You can even just do the clean
13:59rooms. They're like so many things which
14:01fall under this Unity Metastore.
14:04Unity Metastore. Okay.
14:06Let's say
14:07clean room. Clean room is basically the
14:09area where you if you have worked on
14:14that particular kind of project. Where
14:16what you can do?
14:18You or your organization can share some
14:20information, some data with outside
14:24um or you can say with any other
14:26organization and they can just
14:27collaborate on that particular thing
14:28together.
14:29Okay, good. So this is the concept of
14:32Unity Catalog and Unity Meta Store.
14:34When Databricks introduced Unity
14:37Catalog,
14:38the demand of Databricks actually
14:41started growing because organizations
14:43saw the benefit of it.
14:45Where they are not
14:47responsible to manage all of these
14:50things manually. We or you can say any
14:51organization get the all of these
14:53features by default.
14:55Make sense?
14:57Very good.
14:58This is the concept of Unity Catalog or
15:00basically Unity Catalog and Unity Meta
15:02Store. Whenever anyone asks you Unity
15:04Catalog, Unity Meta Store even if and
15:06not in like
15:08if if even anyone is not asking you
15:10anything still for your understanding as
15:12well, Unity Catalog and Unity Meta Store
15:14go hand in hand. But
15:17we just use the term Unity Catalog as
15:19the whole.
15:20Make sense? Okay. So even if you are
15:22just reading some documentation, if you
15:24just find Unity Catalog, you will just
15:26find this thing as well.
15:27Make sense?
15:28So what is Unity Catalog? Unity Catalog
15:31is the unified governance layer.
15:34Okay, simple one-liner answer. Unity
15:37Catalog is the unified data governance
15:39or basically governance layer for AI and
15:41data solutions.
15:43Simple, very good.
15:44So now let's do one thing. Let's also
15:47see all of these things and practice as
15:49well so that you can grasp the most out
15:52of all of these concepts. So now if you
15:54already have one Databricks account, you
15:56can just open that account. If you do
15:57not have one, do not worry. You do not
15:59even need to deploy a paid version
16:01because we get one free Databricks
16:04workspace directly from Databricks just
16:06for the learning purpose. So you can
16:09just create one free account there.
16:11Simply search Databricks free edition in
16:12your browser, hit enter.
16:15Click on this link and here
16:18simply expand this thing and you can
16:20just This is your Oh yeah, this is very
16:22important. So first of all, if you are
16:24just trying or you can say you you are
16:27just creating the Databricks account for
16:28the very first time, make sure that you
16:30are not creating account using the 14
16:32days free trial because you need to get
16:34the free edition. These are two things.
16:36One is like your free edition, second
16:38one is different. So click on get free
16:40edition and make sure you get something
16:43like this, free forever learning and
16:45blah blah blah. Then create your account
16:47because a lot of people come late, hey I
16:48only got 14 days trial. No, because you
16:51created the wrong account. So make sure
16:53that you have the account here and I
16:54already have one account here. So I will
16:56directly jump on to that particular
16:58account. So let me just open that.
17:00So this is my Databricks workspace and
17:02as you can see that this is the
17:04homepage, nothing special. So let me
17:06just take you to that location where I
17:08have all of the Unity catalogs created
17:10and I will also show you the Unity Meta
17:12Store. Do not worry. So as we know that
17:14we are using the Databricks free
17:16edition, we do not have full control
17:18over the Unity Meta Store, but we have
17:20some control over Unity catalog. Okay?
17:23So first of all, let's click on the
17:24catalog tab.
17:25And here you will see all of the things
17:29that you have for your catalogs. This is
17:31basically called catalog explorer. Okay?
17:34So first thing first, if you are using
17:36the Databricks for the first time, you
17:38might see only these two. One is
17:40workspace, second is system.
17:42These catalogs are created by me. Okay?
17:45So you can just totally ignore. If you
17:47have your own Unity catalogs, you will
17:49see that as well. And if you want to
17:51create a new one, it's very easy. What
17:53you can do, simply click on this plus
17:54button. And here you will see create a
17:56catalog option. Cool? So this catalog is
18:00the same catalog that we have recently
18:02discussed and this is nothing but
18:03basically your Unity catalog. Basically
18:05the catalog created under Unity Meta
18:07Store.
18:08Perfect. Now okay, this is the Unity
18:10catalog, but where is the Unity Meta
18:12Store? So if you want to access the
18:14Unity Meta Store, we need to go to the
18:16Databricks admin console. And Databricks
18:19admin console is not in our control when
18:21we are using Databricks free edition,
18:23okay? But we can still check like to
18:25which unity metastore my catalog is
18:27attached. So what I can do? I can just
18:29show you my let's say any catalog if I
18:31just create in front of you or even I
18:33can just show you this as well. Let's
18:34say asset bundles, okay?
18:36If I click on this catalog,
18:38I can click on the details and here
18:41these are like all of the basic basic
18:42details that I have for the catalog,
18:44some permissions, some policies and
18:47workspaces as well. Cool? This is like
18:49some some normal basic um your
18:52introduction.
18:54Then
18:55you can click on this settings button.
18:57Make sure that this catalog is selected,
18:59okay?
19:00Then click on this button and then you
19:02will see something called as metastore.
19:04Click here.
19:06This is the metastore which is created
19:09automatically by Databricks, okay? And
19:12it is hosted on AWS. I didn't select
19:15this thing. This is automatically
19:16created for me. So whenever I create any
19:19unity catalog or basically any catalog,
19:22this will automatically be
19:25attached under this metastore.
19:27Makes sense? That is why my unity
19:30catalog is enabled for this particular
19:32workspace,
19:33for this particular Databricks account.
19:36Cool? We can even go to account console,
19:38but we do not have permission to edit or
19:40to make some changes because obviously
19:42it is managed by the Databricks, okay?
19:45Then we have some permissions on this
19:47metastore and then we have some policies
19:50and obviously no allowed charge is there
19:52as I just mentioned that this is the
19:54free edition. So this is the way we can
19:55just actually explore the Databricks
19:58free edition, okay? And we'll be like
20:00keep on referring this Databricks free
20:02edition whenever you want to understand
20:04any concept related to it. Simple? Cool.
20:07So this was all about unity catalog and
20:08unity metastore. Now let's discuss the
20:11another concept which is called
20:13Databricks volume. So I already have the
20:15heading. Okay. Cool.
20:17So now let's talk about Databricks
20:18volumes. What are Databricks volumes and
20:20why every organization is like currently
20:23using and is like literally mad about
20:26Databricks volumes. Let me just tell
20:28you.
20:29In a nutshell, in a nutshell, Databricks
20:31volume is the way to govern the files.
20:35Just an example, the files.
20:38Make sense a little bit at least.
20:41So let say
20:42you have some files, okay?
20:44Let's first of all let's say you are
20:45using any cloud account. It can be AWS,
20:48it can be ADLS Gen 2 by Azure.
20:52And you have some let's say files.
20:55Any files. It can be CSV file, it can be
20:58Excel file.
20:59Let's say CSV file.
21:02Let's bring CSV.
21:04CSV,
21:05Parquet,
21:07JSON, any files.
21:09Cool.
21:10Now you tell me can you govern these
21:13files? Obviously no because these are
21:15sitting in cloud. These are not sitting
21:17in Databricks. But with Databricks
21:20volumes you can govern these files.
21:24Really? Yes. What you can do, what
21:26Databricks says
21:28Databricks says that you can register
21:31these files,
21:33okay?
21:34Under a special concept or basically
21:35special feature called
21:37volume.
21:38Okay?
21:39Once these files are attached or not
21:43files, you first of all register the
21:45folder itself and within that folder you
21:47can just save anything. Okay?
21:49So what you will do? You will connect
21:51this folder
21:52with this volume and then once it is
21:56connected then any user
21:59any user can query something like this.
22:01Let's say
22:02select a tricks from this file.
22:05Let's say like this using three level
22:07namespace obviously because you are
22:09governing this file using the catalog
22:11name, schema name and file name.
22:13Wow.
22:14Yes, you can do this thing.
22:17That means you can access the files
22:21same way you access the tables.
22:25This is one of the biggest advantage of
22:27data bricks volume and in the rise of
22:31the era where we are literally dumping
22:34the images um media content or you can
22:38say anything any file any unstructured
22:40data
22:42governing those things
22:44is actually very important and we can
22:45govern everything using the volumes.
22:48Makes sense? So that is why you see a
22:50major major rise of volumes and because
22:53we have created the free edition as well
22:55of our data bricks um free edition. So
22:58let's try to see that as well right now.
23:01So this is our data bricks okay and here
23:04if you just want to create the data
23:06bricks volumes it's very easy simply go
23:08to catalog and let's try to create a
23:11catalog
23:12we might be using it so let's say create
23:14a catalog and let say
23:17demo okay
23:18then simply say create. So this is the
23:20catalog that I have created within just
23:22a few seconds configure um nothing to
23:25configure so you can even ignore this
23:28thing so let's say next and save.
23:31So this is my demo catalog and I only
23:34have two schemas default and information
23:36schema. If I want to let's say create
23:39the volume inside this default schema I
23:41can click on create button and I will
23:43see something called as volume.
23:45Makes sense? Very good. Let me just
23:47click on volume and I can just type any
23:49name let's say
23:52my volume. Now I have two options
23:54managed volume or external volume. What
23:56is the difference? When you pick the
23:57managed volume
23:58whatever files you will be dumping okay
24:01will be just stored in the default
24:03storage location. In external volume you
24:06can just connect your third party cloud
24:08service providers and you can just
24:10access those files and basically govern
24:12those files easily.
24:13Makes sense? Good. So let's say manage
24:15volume and that's it and I you can even
24:17read this thing this is written here
24:19files are stored in location managed by
24:21Unity catalog okay files are stored in
24:24an external location that you configure.
24:27Cool. Now you can ask me which one is
24:28popular. So,
24:30whenever I want to govern my files which
24:33are not stored in the Unity catalogs
24:36location, then obviously I will just go
24:38with the external volume. But if I do
24:40not have like much stop and I can just
24:42directly access it via the Unity catalog
24:43location, I will just go with managed
24:45one. Simple.
24:47And whenever I'm just uploading some
24:48data
24:49on the fly, then obviously managed
24:51volume is the um go-to choice.
24:54Click on create
24:56and this is created. See?
24:58So now under this schema, I have this
25:00volume. The same way I have tables, I
25:03have views, everything. So this is my
25:05volume and now in this volume, which is
25:08the name of my volume, my volume, I can
25:10upload any data. I can just click on
25:12these three dots. I can say upload to
25:13volume and I can even create multiple
25:16directories.
25:18Makes sense? So let's say if I just
25:19upload any file, this file will be
25:22uploaded directly to my volume and I can
25:24access it the same way I access my
25:27table.
25:28Okay? This is the concept of volume.
25:31It's not very difficult, but this is new
25:35and let me tell you all of the modern
25:37data platforms are trying to implement
25:39this thing, Databricks volumes with
25:41obviously different different names. So
25:42yeah, this concept is very cool. I
25:44personally love it a lot. So now let's
25:46talk about another concept
25:48which is very very important
25:51in terms of modern Databricks.
25:54So it is called Databricks architecture,
25:57but
25:59not the traditional architecture,
26:01the
26:03modern architecture, which is also
26:05called as serverless architecture
26:07because
26:08Databricks is changed, so you need to
26:09understand that as well.
26:11First of all,
26:12let me tell you there are two types of
26:14architecture that we have. Okay? And
26:16which one we follow? We follow actually
26:18both, so you need to understand both.
26:20Okay?
26:22Architecture has two major things.
26:24The first one is
26:26control plane,
26:29second one is compute plane.
26:33Control plane and compute plane.
26:35What is control plane? Control plane is
26:38nothing but basically your um UI.
26:43It includes all of the backend services
26:45that Databricks manages. Your
26:48um profile section, your Databricks um
26:52UI, the entire UI.
26:54Okay?
26:56Everything
26:57which is managed by Databricks.
26:59Everything.
27:01And here everything means the backend
27:03services only.
27:04Backend services means like all of the
27:06APIs, all of the things that you just
27:08use your mouse. See?
27:10So all of this thing is managed. This is
27:12a kind of application.
27:14This thing is managed by Databricks. So
27:16all of these things come under control
27:18plane.
27:19Makes sense? Whatever we see only in the
27:21Databricks, not the cloud.
27:24Okay?
27:25This entire application comes under
27:27control plane.
27:29Simple? Second thing, which is the most
27:31important thing, compute plane.
27:33Because whatever we are doing in
27:34Databricks, we are using compute
27:38to process the data.
27:40Right?
27:41So compute plane is the area where my
27:43data will be processed
27:45or where your data will be processed.
27:48Where here where what is very important,
27:50where it will be processed.
27:53So here the compute actually creates
27:57that distinction. Like whether your
27:59architecture is your classic
28:02architecture or serverless architecture.
28:04Because we have two types of compute.
28:07Classic compute and serverless compute.
28:10Oh.
28:11That is why we have two different
28:13architectures just because of this
28:15compute. Because control is control.
28:17Control is same. See? This application
28:19is same, right? This UI is same.
28:22There's no change in that. Okay? They
28:25will not say, hey, you need to host this
28:26thing in your cloud. No.
28:28So this is same. The major difference
28:31that we see is on the architect the
28:33architect in the architecture we see on
28:34the compute side. Okay, so let's talk
28:36about the very first one which is my
28:38classic type or you can say classic.
28:43Okay, what do we have inside this thing?
28:45So we have control plane. Let me first
28:48of all draw the
28:49control plane which is the
28:52sure sure thing. Okay.
28:54So let's say this is my Databricks
28:56account
28:57because this location is very important
28:59like where each component will reside.
29:01This is the core thing. So first of all
29:03I will be having my Databricks account
29:05that you just saw.
29:06Within that I have control plane.
29:09Okay.
29:11This is my control plane.
29:14Makes sense? Within the control plane I
29:16have the application, I have the Unity
29:19Catalog, I have the
29:21everything that I just see on the UI.
29:23Okay. Here you can just even write this
29:26thing. Let's say
29:28this particular thing. Okay.
29:31Databricks
29:32Databricks application that we see or I
29:35can just pick this one. This is better.
29:37See, this is my control plane.
29:40Cool.
29:41So that you can just understand it
29:43obviously.
29:44So this is my control plane.
29:45Now within my Databricks account
29:48I have the cloud account as well where I
29:52will be having my what? Resources.
29:55Right? Okay.
29:56So now I will be having my cloud
29:58account. Okay or even my resources as
30:01well. Not a big deal. So let's say cloud
30:02account.
30:04So let's say this is my cloud account.
30:08So in this cloud account
30:10I will be having first of all my
30:12resources.
30:13It can be inside cloud or outside cloud.
30:15So let's say my resources, some
30:17databases
30:21anything. Right? So let's say this
30:23database is there or let's create a
30:24different one here for resources.
30:27It will be easier for you. So let say
30:29this is these are my resources. Okay.
30:34Cool. These are my resources. Database
30:37one, two, three, blah, blah, blah, some
30:38Kafka, anything. Cool.
30:41Okay. Cool.
30:43And these are my resources. This can be
30:45in cloud. This can be on premise as
30:47well. That's why I just created a
30:48separate one. But now the main thing is
30:50all of the compute
30:52that I am just creating within this
30:53cloud will be or basically within this
30:56Databricks account will be in the cloud.
30:58Compute means my machines,
31:00my VMs
31:02that I'll be just using to process.
31:03Okay. These are my compute.
31:07Oh.
31:08This will reside on my cloud account.
31:11Cool.
31:12Very good.
31:13Along with that, I will also have one
31:14more thing, workspace storage bucket
31:18or you can say workspace storage
31:19account. So whatever we just create in
31:21the workspace in the Databricks, we also
31:23get one storage account as well.
31:26Makes sense?
31:29Understand this thing. This is very,
31:30very important. I know this is like a
31:32little bit complex. That is why I'm just
31:33covering this concept under this
31:35particular resource.
31:37So this is everything,
31:38right? This is my traditional
31:41traditional architecture. This is
31:43connected to this one
31:44and obviously compute is connected to
31:46this one
31:47and that's it. And there is one more
31:49thing
31:49which is compute plane. You will say
31:51Anshul, the compute plane is here.
31:53We just created this compute plane here.
31:56There's one more compute
31:59because when serverless compute was
32:01newly added, okay, then we got compute
32:04in the workspace as well, basically in
32:06the Databricks
32:08account when it was newly added,
32:10obviously.
32:11And here I can just connect this compute
32:13with the resources because compute needs
32:15to talk to the resource, right? And
32:17obviously this will also be connected.
32:18This is your classic example.
32:21But then
32:22in the serverless
32:24you can now, I think, imagine what would
32:26have happened.
32:28I think so.
32:30So first of all, let me just write this
32:31is my serverless compute.
32:33Okay.
32:34Yeah, you will just understand
32:35serverless compute. So now what will
32:36happen in this particular thing?
32:38This piece will remain as it is.
32:41As it is, because I need to have
32:43serverless compute, I need to have my
32:45control plane as it is. But
32:47I do not even need this cloud thing now.
32:49I just need my resources. That is it.
32:53That is it. I do not need anything else.
32:57I just need these resources and that's
32:59it.
33:00I hope it makes sense.
33:03This is the entire game.
33:04Why?
33:05Because here I do not need cloud
33:08compute. I will only use the serverless
33:10compute.
33:11That is the whole story.
33:14That is the whole story. Now
33:17these resources
33:19can reside in your cloud account as
33:20well, can reside outside as well.
33:23Simple thing, obviously. So you can even
33:25write here cloud
33:29or not. It doesn't matter.
33:31It doesn't matter.
33:33Because
33:34you can use cloud data as well and you
33:36can even use some, you can say obviously
33:37on-prem data as well. But usually we
33:39connect our data with the cloud.
33:42But you do not need this particular
33:43compute. Oh, okay, compute is not
33:46required. What about this um workspace
33:48storage account?
33:50Because obviously we need to store our
33:51data. So whenever we use this kind of
33:54architecture, we connect our workspace
33:57with the storage account, default
33:58storage account.
34:01This is the difference. If you
34:02understood this um understood this
34:04particular difference, you can actually
34:07understand the new Databricks.
34:09But now
34:11most of the organizations are using
34:13both. Really yes, because they built a
34:16lot of solutions using classic compute.
34:19They are doing the migration from
34:21classic compute to serverless compute
34:24and they are slowly moving towards
34:26serverless compute because serverless
34:27compute solves a lot of problems. So
34:29what is the difference between
34:30serverless and
34:31um or serverless compute and classic
34:32compute? So see,
34:34in the classic compute you need to
34:36decide the number of machines first of
34:38all.
34:39You need to decide like how scaling will
34:43take place like um on [clears throat]
34:45what load the
34:47new machine should be added, on what
34:49load the machine should be reduced,
34:51everything will be decided by you. But
34:53in serverless compute you do not need to
34:55do anything. You simply attach the
34:56compute.
34:57DataBricks will automatically
35:00scale up and scale down your resources.
35:04This is the biggest advantage. You do
35:05not even need to take care of the
35:08decisions
35:10that you take during you can say scale
35:12up, scale down scenarios.
35:14This is the major advantage and
35:16obviously when you do not have that
35:18particular boilerplate stuff in your
35:20plate you can just focus on different
35:22things, right?
35:23This is the major advantage of using
35:25serverless compute.
35:27That is why a lot of lot a lot of teams
35:30are like moving towards
35:33your serverless compute.
35:34I hope this architecture is crystal
35:36clear and within this architecture we
35:37have discussed compute concept as well.
35:40We have just discussed so many things so
35:41because this concept is very clear um
35:43important that you need to clear. Okay?
35:45Very good. So so so so so it's Ansh
35:47Lamba. So now I just want to remind you
35:50that please hit the subscribe button
35:51because it takes a lot of efforts to
35:54create these kinds of masterclasses and
35:56providing it for free. So quickly hit
35:58the subscribe button and let's continue
36:00with the masterclass. So the next
36:02concept that we need to learn, we need
36:05to understand, we need to master is
36:06Delta Tables and in the modern data
36:09engineering Delta Table or you can say
36:12similar formats are non-negotiable. Ansh
36:15Lamba, what do you mean by similar
36:16formats? So
36:18Delta Table
36:19is basically you can say the bigger
36:22umbrella of something called as open
36:25table format. Okay?
36:28Delta Tables go hand in hand with
36:31DataBricks but Delta Tables or basically
36:33Delta Lake is not the only solution we
36:36have. We have Apache Iceberg as well. So
36:39let me just show you. So one is
36:42Delta Lake or basically
36:44see first of all, do not get confused
36:46between a data Lake and Delta Lake.
36:48Delta Lake is nothing but just the name
36:51we use for Delta tables. Okay? That is
36:53why I use the word Delta table. But
36:55wherever you see Delta Lake, it is a
36:58Delta table. Okay? Do not worry. Do not
37:00get confused. So this is my Delta table.
37:03The second thing is
37:05Apache
37:07Iceberg. Okay?
37:09Let's use this one. Cool. So these are
37:11the two most popular solutions we have.
37:13Okay? But whenever we just talk about
37:15Databricks, we talk more about Delta
37:18tables because Delta Lake or basically
37:20Delta tables were created by the same
37:22team. Okay? So that is why.
37:24So what what is a Delta table? What is a
37:26Delta table? So
37:27before Delta table, before you can say
37:30the concept of Lakehouse,
37:32which was introduced I think couple of
37:34years back.
37:35So what we were doing or what data
37:37engineers were doing, what every data
37:39engineer was doing. So let's say I have
37:43parquet file, which is a very popular
37:45file
37:46for big data processing. Right? Very
37:47good.
37:49We were building external tables on top
37:51of it. Okay?
37:53Tables.
37:55Just tables. So we were creating parquet
37:57table.
37:58Like we have the file, right? And we
38:00were just registering the metadata of
38:01that file and we were just storing the
38:04metadata under a metastore and that was
38:06our table. Simple.
38:08But what Delta table or basically Delta
38:11Lake provided to us is they said like,
38:14hey, you cannot do this thing. Why?
38:17Why?
38:18Why why why we cannot do this thing? You
38:20can. No one can stop you, but this does
38:23not give you features like acid,
38:27atomicity, consistency, isolation,
38:29durability.
38:31These are the you can say
38:32non-negotiables whenever you want to
38:34work with structured data.
38:37You do not get something called as
38:38commits and rollbacks. Okay?
38:41Or we also
38:43call it as you can say um time traveling
38:47and Delta Lake, okay?
38:48So all of these features were not there.
38:50We didn't have indexes, we didn't have
38:52you can say some kind of optimization
38:55that we just apply on top of our data
38:57for better search results. So we had
38:59nothing. We had nothing. We just had the
39:02raw parquet files and that's it.
39:04Then this team created this thing. So
39:07they said we will keep your data in the
39:09parquet files for sure because parquet
39:11is one of the best solutions out there
39:13for big data um storage, okay?
39:16They said we will just keep this thing
39:17as it is, but on top of this thing, on
39:20top of these files, we will create a
39:22folder,
39:23a special folder called Delta Log.
39:26Okay?
39:28Delta Log. This Delta Log is the heart
39:32of this entire Delta Lake because this
39:34stores all of the transactions, all of
39:37the information about transactions, all
39:40of the metadata.
39:42And because of this particular Delta
39:44Log, we also get an amazing feature
39:46which is called
39:48schema on write.
39:51What is schema on write? Schema on write
39:53means schema will be tested and
39:56validated before it writes your data.
39:59But with basic raw parquet files, basic
40:02raw data lake, we do not get this
40:04feature. So we get everything when we
40:07add this Delta Log on top of it. And
40:09within this Delta Log, if I just open
40:11this Delta Log folder, what I see?
40:13I see JSON files. Basically all of the
40:16transactions will be written in the JSON
40:18files.
40:20There will be one more type of file, CRC
40:21files. This is not directly related to
40:23your Delta Lake, but it is a kind of you
40:25can say another type of file that we
40:27get. It is called I think cyclic
40:29redundant checkpoint file, something
40:30like that, okay? So this is for that
40:32particular purpose. The major file is
40:34JSON file where we store our all of the
40:36metadata, all of the schema, all of the
40:39transactions, data which is added, data
40:42which is deleted, everything within this
40:43particular JSON file, okay? And we keep
40:46on adding JSON files as our transactions
40:48grow, simple.
40:50Simple? Very good. So this is the
40:51concept of Delta Table. So let's try to
40:53see this thing practically as well.
40:55That is how you learn by doing it. And
40:58that is why I created your free
41:00Databricks account. Okay? Cool. So now
41:02what I will do
41:04I will create a new schema. Let's try to
41:07or let's use default schema. It's fine.
41:09Let's use default schema. Click on this
41:10default schema
41:12and click on create. And this time do
41:14not pick volume, pick table.
41:16Okay?
41:17And here it will ask me that you can
41:19even dump your
41:21local file
41:23which is one way. Second, you can even
41:26pick data from the volume. The third is
41:28you can even connect your Databricks to
41:30AWS S3 to upload the data. But just for
41:33the simplicity
41:35I will just upload this file. It's
41:37called orders. And do not worry. I will
41:39just provide all of the data that I'm
41:41just using with this tutorial in my
41:44GitHub repo link. Do not need to worry.
41:45But
41:47you can use any file. Okay? Here I've
41:49just used orders file. You can just use
41:50any file. Okay? And as you can see this
41:53preview is here. And I can see the data
41:55looks fine. Perfect.
41:57And then it will simply ask me to pick
41:59the catalog name. I want to create this
42:01table in the demo catalog.
42:03And schema name I want to pick is
42:05default. I can even create a new schema
42:07right here as well if I want to. But I
42:09want to keep
42:11it under default. I'm good with that.
42:12Okay? Cool. Then I can click on create
42:14table.
42:16So do you know what will happen?
42:18Now my CSV file that I uploaded is of
42:22CSV format. Right?
42:24This will be automatically converted
42:26into a Delta table.
42:28Automatically. Yes.
42:29Automatically. And the data type, like
42:33basically the file type
42:35will be of Parquet because in the Delta
42:37Lake we always store our data in the
42:39Parquet file along with the
42:42that Delta log.
42:43Make sense? Very good. So this is my
42:46data. I think this is ready and I can
42:48see under the tables I have orders
42:49table. And if I just scroll down I can
42:51see the data source is delta.
42:54I told you, right? Because this is a
42:57delta table. This is not a CSV table.
43:00Very good. And this is a managed table
43:02because I am not storing my data in a
43:05separate cloud account. I am literally
43:07dumping the data within the Databricks.
43:09Cool?
43:10Very good.
43:11That is all.
43:12That is all. Now if you just want to
43:14query this data, you can even come here,
43:16click on sample data just to see the
43:18sample of your entire file. It will just
43:21show you like this.
43:22If you want to query it using SQL, you
43:24can even go to SQL editor or you can
43:26even go to notebook and you can just
43:28query this thing. It's very simple. You
43:30can, let say, click on workspace. Let's
43:32try to do it and let me just delete all
43:35the stuff that I have here.
43:38What what do you want, bro? Do you want
43:41Perform actions for bulk action. Right
43:43click on the items. Wow.
43:46They keep on just changing this UI
43:47stuff. I don't know why.
43:49So now let me just create a new folder
43:50where I'll be just putting everything
43:52that I'm just using here. So let say
43:55um
43:57demo. Okay.
43:59And I will also attach one special file
44:02which is called
44:05dot
44:06DBC file.
44:08You cannot open this file by literally
44:10double clicking on it. A lot of people
44:11say, um Ansh, you uploaded dot DBC file.
44:14I am not able to open that file because
44:16that file will be opened only by
44:19Databricks. So how you can just um take
44:21advantage of that file?
44:22So you can come here
44:24and then you can just click on, let say,
44:27upload. Um
44:30where is my folder first of all? Okay,
44:31this is my folder, demo folder.
44:33And then you can just click on these
44:34three dots. Then you can just click on
44:35import and then you can add that file.
44:38What will what will happen after that?
44:40You will see all of the resources like
44:42notebooks, scripts, anything that I will
44:44have in that file will automatically be
44:47imported here. Okay? But do not try to
44:48double click on it and open it.
44:50So let me just create a new notebook
44:52here. Click on new notebook and that's
44:54it.
44:55And let me just rename it. Let me say
44:58query.
45:00One query and let's say if I just want
45:02to query the data, I will simply select
45:04asterisk from because I can just use SQL
45:06as well here. I will simply say um demo
45:12demo dot default dot orders.
45:14Remember I have to use three level
45:17namespace, right? Catalog name, schema
45:19name and then table name.
45:21Simple.
45:22You can just run like this
45:24and it will just show me all of the
45:25things and it will automatically use the
45:28serverless compute as you can see here.
45:30It has automatically picked this thing
45:31because with the Databricks free edition
45:33we always get one free serverless
45:35compute that we can use to run our
45:37queries.
45:38Cool. You do not need to create one
45:40because it is already there so you can
45:41just take advantage of it. So I think
45:44the task is done and I can see the data
45:46is here. That's how you can just query
45:47it,
45:48right? And if you just want to see much
45:51more details about um delta table, you
45:54can simply write
45:56describe
45:58and then table name. Let's say
46:00demo
46:02dot
46:04default dot orders.
46:08And let me just write percentage SQL.
46:12Cool. So this will simply describe the
46:14table and that's it. That this is my
46:16table name, this is everything and I can
46:17even say describe
46:20um
46:21detail and then it will get me some more
46:25information about like okay what um like
46:27how it was created, some description,
46:29everything. Okay?
46:31I can even run one more command which is
46:34history. It will tell me like what
46:36actually happened with all of the
46:38versions. So it will simply display like
46:39version zero table created. If I add
46:42more data into it, it will create
46:43version one. If I just let say delete
46:45anything, it will create version two. So
46:47that's how it keeps on creating the
46:49versions within my table which helps me
46:51to perform time traveling on top of my
46:53data.
46:54Oh, yeah. So you never you never lose
46:58any version until and unless you
47:00intentionally want to, but you can
47:02anytime time travel back to any specific
47:05version of the table.
47:07Makes sense? That is the biggest
47:08advantage of the Delta table.
47:10I hope it makes sense.
47:12Very good. So this is all about Delta
47:14tables and I hope this concept is
47:16crystal crystal
47:18crystal clear. Okay.
47:20So what is the next concept that we need
47:23to understand? That is a very good
47:24point.
47:25So what should be covered? Let me just
47:27see my list. So the next topic, next
47:30concept that we are just trying to
47:32understand is Lake Flow Connect.
47:34And you know what? This is so important.
47:37Why? Why so important? And first of all,
47:39what is this Lake Flow Connect?
47:41Lake Flow Connect is the way to ingest
47:45your data to
47:47Databricks. Not to basically,
47:50you can say yeah, kind of to because now
47:52you can even store the data as well in
47:53the Databricks because Databricks has
47:55their own managed storage accounts. So
47:57let's say you have
47:59data in
48:01PostgreSQL database.
48:02Just for an example.
48:04Let's say you have the data in the S3
48:06bucket. Just for an example. Let's say
48:09you have data in any kind of message
48:11queue, let's say Kafka.
48:14Just for an example. So you have so many
48:17sources, right? So many sources.
48:19How will you ingest the data in the
48:21Databricks?
48:23That is the point. That is the point.
48:26Now when Databricks, I told you
48:29initially Databricks was only about
48:31managed Apache Spark. So how people were
48:34ingesting the data?
48:35People were ingesting the data using the
48:37Spark APIs, basically Spark connectors,
48:39raw Apache Spark connectors.
48:41People still use that option. For
48:43example, if they want to ingest the data
48:45using PostgreSQL, they create the JDBC
48:47connection.
48:48If they want to pull the data from AWS
48:50S3, they still use the, you can say
48:52Spark API to read the data. If they just
48:55want to pull the data from Kafka, they
48:57still just create that particular
48:58connection. But
49:00but
49:01with the course of time, Databricks also
49:04created some kind of managed connectors
49:06that you can leverage along with the
49:09code.
49:10Databricks is not saying that you do not
49:11need to use code. You can still use
49:13code, but Databricks created an umbrella
49:16of lake flow connect
49:18where you have your code and your
49:20managed connectors as well.
49:22Why? Because see,
49:24if you have used any good integration
49:27tool, and let me just take an example of
49:29Azure ecosystem. So Azure has a very
49:31good data integration tool which is
49:34called Azure Data Factory.
49:36They provide you managed services to
49:38integrate the data from hundreds of
49:40sources.
49:41Data bricks in the beginning was not
49:44very mature enough to provide this
49:45service. So Databricks actually welcomed
49:49one service called Five Tran.
49:52I do not have logo for that. So Five
49:54Tran was very very active on Databricks
49:56platform where Five Tran Five Tran is
49:58another very popular data integration
50:00service where you can connect to the
50:02hundreds of sources, right? So Five Tran
50:05was acting as a kind of layer for that.
50:07But now Databricks has created their own
50:10layer as well and they have introduced
50:12so many connectors and they are keep on
50:13adding more and more and more.
50:15I know you are really really excited. So
50:16let me just give you a glimpse of it.
50:18Then I will just go deeper inside this
50:20topic because this is very very very
50:22important. So let me just show you.
50:25So this is my Databricks account
50:27and here you can go to let's say data
50:30ingestion. Here if you do not see this
50:32data engineering tab, do not worry.
50:34Simply click on plus new and here you
50:36will just see more and then you will
50:38just find the data engineering here as
50:39well. Do not worry.
50:41So let me just click on data engineering
50:42and then data ingestion.
50:44See?
50:45So here I have some files option
50:48which we have already seen how to create
50:50a table using file. We can upload the
50:52file. We used this thing just now,
50:54right? So this is one of the connectors.
50:56We can even upload data directly to the
50:58volume, another kind of connector. But
51:00I'm talking about these data bricks
51:02connectors. See, I have so many
51:05connectors available that I can use.
51:07Some are in preview, some are generally
51:10available and now they are just keep on
51:12adding more and more and more. All of
51:14the major or most popular, see Apache
51:17Kafka is already there now. So all of
51:20the popular
51:21sources are there
51:23that you can just use to pull your data.
51:25Wow, the list is really really big now
51:26because last time when I just saw it was
51:28like
51:29uh I would say half of it. Obviously
51:31like there were so many in preview, but
51:32they have removed the preview tag from
51:35so many services. Earlier I remember
51:37only 1 2 3 4 5 6 or 7 8 services were
51:42without preview, but now they are just
51:44testing it and they are just improving
51:46it. So all of these services, all of
51:48these connectors are there.
51:50Makes sense? And now if I just scroll
51:51down, these are the community and the
51:54custom connectors. These are either
51:56developed by community or maybe
51:58third-party services like FiveTran,
52:00right? So these are uh FiveTran
52:02connectors are here, partner connectors.
52:04See, FiveTran integration. Earlier
52:06FiveTran had a dedicated tab here. So
52:09let me just click on it and you will see
52:11a long list, but now obviously they have
52:13just
52:14deprioritized it because now they have
52:16just their own thing. So let me just go
52:18here, data integration integration.
52:20Perfect. So these are all of the
52:22services. Now you do not need to write
52:24the code from scratch to pull the data
52:26from these sources and you can directly
52:28start loading the data from these
52:30services. S3, foreign catalog, open
52:32sharing, Salesforce, SQL Server,
52:34everything is there. Let me just show
52:36you one simple example that you can use.
52:37Okay, let's say SQL Server.
52:40So here I already have some connection
52:42because I was just working with these
52:44this particular data bricks integration
52:45to create my projects, right? Data
52:47engineering projects for obviously
52:48YouTube. So I just created one with AWS
52:52and one with agent database. So yeah. So
52:54what you can do, let me just show you
52:56one quickly. So you can just create a
52:57connection and what you need to do.
52:59First of all, it will ask you how you
53:01want to authenticate yourself.
53:04I usually pick username and password
53:05whenever I'm just building the quick
53:07tutorial. So you can here pick the
53:08connection name. Let's say test.
53:11Host is obviously the server where your
53:14SQL server is hosted. So you can simply
53:17type here, let's say XYZ, username,
53:20password, right? And port usually
53:23whenever you are just connecting to SQL
53:24server, it is like 1433. Then create
53:27connection or let's say test connection.
53:29You can say advanced settings and then
53:30create connection.
53:31It will create the connection for you.
53:33See test. It is there.
53:35That is so so so easy.
53:38Then what you can do with that?
53:40You can
53:41start ingesting your data directly.
53:44Really? Yeah.
53:45You can just um click on data ingestion,
53:48which is here, right? And then let's say
53:50if I just want to use AWS connection
53:53because I know this is like the legit
53:55one. So here you can see the URL for
53:57that, host for that, and everything is
53:59there.
54:00Oh, and now if I just want to create the
54:02ingestion pipeline with that, ingestion
54:04pipeline means if I want to ingest the
54:05data, I can directly click here
54:07and then it will simply ask me what type
54:10of ingestion you want to perform. These
54:12are some advanced top that I will be
54:14just discussing right now. Okay? So
54:16let's try to talk about it. So we got
54:19some taste of Lake Flow Connect. Lake
54:21Flow Connect is the area which we use to
54:24ingest the data within our data bricks.
54:26Simple, very good.
54:28Now
54:29let's talk about some inner areas which
54:31comes under Lake Flow Connect, but
54:33I like to talk about it independently
54:35because these are like very detailed
54:36ones.
54:37First of all, we have database
54:39connectors.
54:42Okay.
54:43What are database connectors? As we just
54:45saw, we have popular popular databases
54:47like MySQL, MS SQL, PostgreSQL,
54:51anything.
54:52Now the beauty of these connectors is
54:54why I just created a different category
54:56of concept for this because you need to
54:58understand one concept. Let's say let
55:01let's pick um MS SQL server.
55:04Do we have MS SQL server? Uh yeah.
55:08Let's say this is SQL database.
55:09So whenever we perform any data
55:11engineering task on top of a database,
55:14we perform one thing which is
55:17non-negotiable.
55:19This is incremental loading.
55:23Okay.
55:24Incremental loading.
55:27Or you can say incremental ingestion. If
55:28we want to perform incremental loading
55:30or incremental ingestion,
55:32we have to
55:34take care of the CDC manually. CDC means
55:37change data capture. Because what
55:39happens is let's say you are just keep
55:41on incrementally loading the data, but
55:43now
55:45there are
55:46not only the you can say
55:48new data,
55:49some previous data can also be updated.
55:53So you need to make sure that your data
55:55is fully updated.
55:57Either it is an append
55:59or update, anything, I want fully
56:02updated data.
56:03If let's say I have new record with ID
56:05let's say 10, I want 10 here
56:08immediately.
56:09If I have an ID
56:11nine which was already there, but now it
56:14is updated. So let's say green, this is
56:16updated. I want this updated here as
56:19well.
56:20So all of these things require a lot of
56:23manual and boilerplate code and I can
56:26say that it would require at least
56:27hundreds of lines of code at least,
56:29okay? Because you need to write a lot of
56:31try-except statements whenever you are
56:33building this thing. Hundreds of lines
56:35of code. But now
56:37with the help of database connectors,
56:38let's say if I'm just picking database
56:40connector, I automatically get the
56:42feature called change data capture.
56:44I can literally pick this thing
56:46and whenever I am ingesting the data, if
56:49my data source is changed, let say if I
56:51get a new get some new data, if my
56:53existing data is updated, I will get all
56:55of those changes. Wow.
56:57What is this query-based capture? So
57:00very good question. Let me just tell you
57:01what is this thing.
57:03CDC is clear? Very good.
57:05This query-based capture is nothing but
57:07the lightweight version of CDC.
57:10Why saying this thing? Because see,
57:12whenever I'm just ingesting the data
57:14with the changes, I have to enable CDC
57:17on the source tables and in order to
57:19perform this thing, you need to involve
57:22a DBA because in bigger organizations,
57:24in any organization, you will see that
57:26databases are governed and owned by the
57:29DBAs. Data engineers cannot directly
57:32change anything on the source side. No,
57:34it is not allowed.
57:36I don't think so it is allowed. It is
57:37allowed, but
57:39in smaller organizations, in startups,
57:41obviously one person wears so many hats,
57:43so that person can change it. But it is
57:46a mess. You first raise a ticket, then
57:48the ticket will go to DBA. That DBA will
57:50first change the thing in QA, you will
57:52test it, then it will do that thing in
57:54production. So it takes a lot of time.
57:56So this query-based capture is what we
57:58data engineers have been doing since the
58:00start because I am not also not a big
58:02fan of change data capture. I always
58:04prefer this thing. So what we do here,
58:07we pick a CT, change tracking column.
58:12That change tracking column can be a
58:13date. In most of the scenarios, it could
58:15be a date column. So what we do, we
58:17simply take that date column and we keep
58:20on adding or appending the new data
58:24according to that last updated date. And
58:26obviously we handle the update update of
58:28that data as well.
58:30If we see any kind of changes happening,
58:33we use the same change tracking column
58:35there as well. But in order to perform
58:37this thing, we have to write a lot of
58:38PySpark code. But that is fine. That's
58:39how we were just doing the stuff. But
58:41now Data Bricks has introduced this this
58:44QBC, query-based capture as well, and it
58:47does exactly the same thing. When this
58:49feature was newly launched, I was just
58:51like looking at some
58:52internal things. Obviously, not the
58:54entire script is
58:55publicly available. But when I was just
58:57looking at the behavior, I was like,
58:59wow, this is the exact thing that we
59:00data engineers have been doing. So this
59:02is the managed solution for it.
59:04Makes sense? So in short, you will get
59:06the most updated data every time you run
59:08the pipeline. Simple.
59:10This is all about the database
59:12connectors. I hope it is clear. I hope
59:14it makes sense.
59:16Very good.
59:17The second thing that we have here is
59:19SAS connectors. SAS connectors is very
59:21very important
59:23if you are working with
59:26a domain where they have data coming
59:29from so many sources.
59:31It can be a SharePoint. Do we have the
59:34SharePoint logo?
59:36No? No. Let's use this one. So let say
59:39you have the SharePoint, you have Teams,
59:42you have Outlook, you have so many those
59:44those things, right? Let me just use
59:46this particular
59:47Databricks.
59:48With this here I can see that particular
59:50thing. So let me just go to Data
59:53Injection.
59:55Cool.
59:56And if I go to SQL Server, just to
59:57delete this thing that I created for
59:59you.
1:00:00Perfect. So if I just go to Data
1:00:01Injection, all of these Salesforce,
1:00:03Workday, SAP, Google, right? These are
1:00:06all the SAS connectors.
1:00:07And to be honest,
1:00:09those non-technical managers use data
1:00:12from these sources for sure. And
1:00:14especially if you're working within the
1:00:15finance domain, you will ingest the data
1:00:18from these sources. And
1:00:20if you need to find the code to connect
1:00:23to these things, it is a mess because
1:00:25obviously they keep on changing the
1:00:27connectors, they keep on changing the
1:00:29APIs. But now you do not need to worry
1:00:31about it.
1:00:32Cool?
1:00:33Very good.
1:00:34Then third thing,
1:00:36my personal favorite,
1:00:37file connectors.
1:00:39This is the thing that a lot of
1:00:41developers, lot of data engineers
1:00:43currently work with. What is this file
1:00:45connectors? So let say you have the data
1:00:48sitting in S3.
1:00:49You have data sitting in let say ADLS
1:00:52Gen2,
1:00:53any cloud storage usually. So you use
1:00:56file connectors.
1:00:57Okay. Basically you you just want to
1:00:59read the data from the object storage
1:01:01and you just want to
1:01:03leverage the Databricks.
1:01:05Now I know your question. Do we have the
1:01:07managed service for this as well? The
1:01:09answer is yes or no. What what is this
1:01:11answer yes or no? Let let just explain.
1:01:14So the answer there is no
1:01:17there is yes okay okay wait.
1:01:19Let me frame the answer.
1:01:21So there the best way to perform the
1:01:24data ingestion from the file connectors
1:01:26is using
1:01:28PySpark code
1:01:30that is my recommendation. Okay.
1:01:33Let let me just come on to the point
1:01:34don't worry.
1:01:35PySpark code is the best way to fetch
1:01:37the data from the file connectors. First
1:01:39thing
1:01:40first thing.
1:01:41Second thing do we have no code solution
1:01:44available where we can just directly run
1:01:46the pipelines and it will just simply
1:01:48fetch the data. Yes, we have that as
1:01:49well. Let me show you. If you just go
1:01:51here in the data bricks you will find it
1:01:53let's say Amazon S3
1:01:55there will be like ADLS Gen 2 as well.
1:01:57Let me just find it for you.
1:01:59Like you can just
1:02:01write it in the search bar
1:02:04ADLS
1:02:06Azure Data Lake
1:02:08they do not have it for ADLS Gen 2 for
1:02:10now because I I told you coding is the
1:02:12best one for file connectors. Okay, but
1:02:14you can just pick let's say this one
1:02:17Amazon S3 and Data Bricks simple.
1:02:19What it will do? It will simply fetch
1:02:21the data from the S3 and it will load
1:02:23the data into your staging table. That's
1:02:26it. Why I am saying that it is
1:02:29not very very good as compared to
1:02:30PySpark code. It is my personal
1:02:32experience first of all.
1:02:33When you write the PySpark code to
1:02:35connect to the files, connect to the
1:02:36object storage, you can perform a lot of
1:02:39schema modification, you can do some
1:02:41filtering as well before loading the
1:02:43data in the staging layer. So it is my
1:02:45personal recommendation.
1:02:47And now let me come on to the point.
1:02:48Point when I say PySpark code, I'm not
1:02:50saying that you need to write the raw
1:02:53code again and again. Or really yes.
1:02:55Here we get a feature within the data
1:02:57bricks that where you write PySpark code
1:02:59for sure, but
1:03:01they provide you a lot of features where
1:03:03you do not need to write the code for
1:03:07every small thing and for the complex
1:03:10things.
1:03:11What do I mean? So let's say
1:03:13you have AWS S3 right and you want to
1:03:17perform let's
1:03:19data ingestion for the files.
1:03:21Let's say you have some parquet files,
1:03:23so you have one file, two file, three
1:03:24file.
1:03:25So you will keep on adding these files
1:03:27within your source, right?
1:03:29Keep on adding.
1:03:31Now, if you want to incrementally load
1:03:34the files,
1:03:35you have to write the logic for that.
1:03:36But Databricks says,
1:03:38I know that you love coding. Let me just
1:03:41give you an option to perform the
1:03:43coding, but with the managed approach.
1:03:45Here Databricks will say, I have
1:03:48copy into
1:03:50command which will incrementally load
1:03:53the data, incrementally.
1:03:56Which is one of the managed solutions
1:03:58with code, not without code, with code.
1:04:01Simple. So that's how we just ingest the
1:04:03data using this thing. So let's say if I
1:04:05have if if we have like a lot of data.
1:04:08Do not worry. We will just talk about
1:04:10how to ingest this thing, okay? In the
1:04:12next concept, which is our streaming
1:04:14connectors, basically how to perform
1:04:15streaming, okay? Which is very very
1:04:17important. So do not worry about that
1:04:19about that for now because you first
1:04:20need to understand the concept.
1:04:22Cool? So this is all about the
1:04:25connectors or you can say Lake Flow
1:04:26Connect that we have.
1:04:28Makes sense?
1:04:29Now let me just talk about a very
1:04:33important concept in the world of
1:04:34streaming because I just used the word
1:04:36streaming.
1:04:37It's called auto
1:04:40loader.
1:04:41What is this auto loader?
1:04:43What is this auto loader?
1:04:45So now
1:04:46with the rise of data,
1:04:48managers, non-technical people, all
1:04:51those leaders,
1:04:53everyone wants to take data-driven
1:04:56decisions as soon as possible, as soon
1:04:58as data arrives in the cloud.
1:05:00They want to see that data in the
1:05:02dashboard. And I don't know what they do
1:05:04with that data, but they want to see the
1:05:05dashboard immediately refreshed, okay?
1:05:08Immediately.
1:05:09Immediately. Data is dropped at let's
1:05:11say
1:05:127:30 p.m. They want to see that data at
1:05:147:30 p.m.
1:05:16Okay?
1:05:17And they they won't do anything like
1:05:19until next day of 7:30 p.m., they want
1:05:21to see the data in their dashboard being
1:05:23refreshed. So, now you being a
1:05:25developer, we cannot argue with them.
1:05:26Obviously, you will say, okay, let me
1:05:28just provide the data to you, bro, and
1:05:29then do whatever you want to do. So,
1:05:31auto loader, this is one of the
1:05:35best things that they have added. So,
1:05:36let's say
1:05:38let's take this scenario. This scenario
1:05:39is a very good scenario.
1:05:41So, let's say you have the AWS S3 and
1:05:44you have data,
1:05:45okay, with these files.
1:05:47Now, you want to
1:05:50fetch the data and you want to
1:05:53use this data as soon as possible. So,
1:05:55what you will
1:05:56usually perform? What you will perform?
1:05:59You can perform, let's say,
1:06:01um streaming,
1:06:04Spark streaming. So, here we just
1:06:05perform something called a streaming.
1:06:07So, whenever we are just performing
1:06:08streaming, we have to take care of the
1:06:10idempotency. We have to take care of the
1:06:12checkpoints. We have to take care of all
1:06:14of the offsets, everything, right?
1:06:17But Spark structured streaming
1:06:20I know is already very good, but it is
1:06:22not managed.
1:06:24We do not have built-in capabilities to,
1:06:27let say, efficiently manage the
1:06:29end-to-end solution. But
1:06:32with auto loader, auto loader is
1:06:34basically a kind of managed Spark
1:06:36structured streaming solution for you
1:06:38created by Databricks.
1:06:40So, they say that you can use
1:06:42auto loader to stream your files, let's
1:06:45say from here
1:06:46to here.
1:06:48Okay.
1:06:49If let's say a new file arrives
1:06:53a new file arrives
1:06:54this will be automatically
1:06:58be like go here
1:07:00with idempotency, with all the
1:07:01checkpoints, with even the schema
1:07:04evolution as well.
1:07:07Here Databricks provides you
1:07:10some concepts called schema evolution
1:07:13and we get two to three options. First
1:07:15of all, if you want to update the
1:07:17destination table with the new schema,
1:07:18you can do that.
1:07:20If you want to, let say
1:07:22do not want if you do not want to
1:07:23include the new schema or if you do not
1:07:25want to update the schema, you can keep
1:07:27on just having that particular same
1:07:30schema after the
1:07:32the after this new injection as well.
1:07:34All of these things come under auto
1:07:36loader.
1:07:38Auto loader is very very important and
1:07:40you need to understand this thing deep
1:07:43um into an example.
1:07:45Makes sense? I hope that you got some
1:07:47taste of like auto loader like what
1:07:48exactly is auto loader.
1:07:50I hope so.
1:07:52Concept is very simple.
1:07:54Just a kind of streaming um managed
1:07:57streaming solution for you and that's
1:07:58it.
1:07:59That is it. Nothing else, nothing else.
1:08:02And yes, it is like mostly used with um
1:08:05files, mostly, but yes, you can just use
1:08:08it with multiple sources as well. This
1:08:10is like
1:08:11my personal experience. I have just used
1:08:13mostly with the file. So that's what I'm
1:08:14just telling you.
1:08:17Makes sense?
1:08:18I hope it makes sense. Now how we can
1:08:19just use this particular um option. So
1:08:23you can either use PySpark code as well
1:08:25and you can even use Spark SQL as well.
1:08:27Both the options are there. So you are
1:08:28good with that. So I personally use
1:08:31PySpark code more than Spark SQL code
1:08:33whenever whenever I'm just doing
1:08:34anything. So this is like up to you if
1:08:36you just want to use anything.
1:08:38Cool.
1:08:40Now I know that you also want to see
1:08:41like how it will work. So let me take an
1:08:43example. Let's try to perform one very
1:08:45simple example with this auto loader so
1:08:46that you also understand this thing.
1:08:48Let's see. So now because we will be
1:08:49just performing a lot of examples and we
1:08:52also need to see this example as well,
1:08:54data injection, data connections,
1:08:56everything. So I would need one cloud
1:08:58account for that and the best one is
1:09:01like which can be easily created. You
1:09:03can just create AWS
1:09:05free account because you get I think six
1:09:07months of free access for that. So you
1:09:09can just click here. If you already have
1:09:11cloud account, very good. If you do not
1:09:13have, click on create account and make
1:09:15sure that you have the AWS cloud
1:09:17account. I already have one cloud
1:09:19account that I just use for the
1:09:20tutorial. So let me just quickly sign
1:09:22in. If you already have one, let's say
1:09:24you just created this thing with this
1:09:25particular thing.
1:09:26Then you can click on sign in to console
1:09:28and then it will take you to the sign in
1:09:30page. Let me just do this for me.
1:09:32So as you can see this is the sign in
1:09:34page and for the first time
1:09:36you will not have or you will not
1:09:38remember your account ID or alias. In
1:09:40that particular scenario you can just
1:09:42click on sign in using root user email.
1:09:45Then you just need to put your
1:09:49root user which is the email ID that you
1:09:51just picked to create your account and
1:09:52then password and that's it. But if you
1:09:54are if you have one IAM user, IAM user
1:09:57is nothing but basically the cloud user
1:09:59that you create within your parent
1:10:01account. I already have that, but if you
1:10:03do not have that, simply click on root
1:10:04user. Okay?
1:10:06So let me just log in into my AWS
1:10:07account and I hope that you will also
1:10:09log in nicely and
1:10:11I think everyone should know how to log
1:10:13in to AWS account or any cloud account.
1:10:16Simple, create the account and then
1:10:18click on sign in, put the ID and
1:10:19password and that's it. So let me just
1:10:21put my credentials and let me just take
1:10:23you to my AWS account and let's try to
1:10:25proceed then. Okay?
1:10:26So I'm here in my AWS account and in
1:10:29your case you will also see something
1:10:31like that. And if you already know how
1:10:33to use AWS then it's so easy. If you do
1:10:35not know, nothing to worry because we
1:10:37are not doing anything special with AWS,
1:10:39nothing. So what we need to do? We just
1:10:41need to create one S3 account which is a
1:10:43data lake and that's it. Okay?
1:10:45How you can just create one? Simply go
1:10:47to search bar and search S3.
1:10:49And click on S3.
1:10:51And it will open up
1:10:53oh wow, this nice UI with a lot of
1:10:57unnecessary stuff. Click on create
1:10:58bucket.
1:10:59And that's it. So here are some things,
1:11:01general purpose or directory. You can
1:11:03simply say general purpose and bucket
1:11:05name, let's pick the bucket name as
1:11:10data bricks demo. Okay?
1:11:12Then account regional namespace.
1:11:16I will simply go with global namespace.
1:11:17This is like you can just pick the same
1:11:19name within the
1:11:21like across the accounts, but I am just
1:11:23good with that data bricks demo.
1:11:25And ACL is disabled. Yeah, that is fine.
1:11:29And then
1:11:31bucket versioning. I don't need bucket
1:11:33versioning. This is just for the if you
1:11:35just want to, let's say
1:11:37perform the time traveling, but that's
1:11:38this is fine. And I think everything is
1:11:40good. I will simply say create bucket.
1:11:43This name is already taken. Wow. I will
1:11:44simply say Onsh. Onsh is not
1:11:47like Onsh is available. So
1:11:51so this is my bucket, as you can see,
1:11:52Databricks demo Onsh, and this is my
1:11:55empty bucket, empty S3 bucket, empty
1:11:57data. Like I do not have anything,
1:11:58right? So what I will do, I will create
1:12:00one folder within this
1:12:02and I will simply say
1:12:04raw. Raw means let's say raw data
1:12:07and click on create folder. So this is
1:12:09my folder name which is empty. Now I
1:12:11will upload some data. I'll click on
1:12:13upload and what I will do, I will simply
1:12:16upload the same orders file that I just
1:12:17uploaded before. Okay? So I have just
1:12:21dragged it here. It is called orders.csv
1:12:23and it is good. Cool.
1:12:25Let's say upload.
1:12:27This data is uploaded. Now the point is
1:12:30this data
1:12:32this data resides in AWS, right? This
1:12:35this is my AWS.
1:12:37So AWS is different entity. Databricks
1:12:40is a different entity. How these two
1:12:42entities will talk to each other?
1:12:44This is the concept of external
1:12:46location. Remember we just talked about
1:12:48external location which comes under
1:12:49Unity Metastore in the beginning. So
1:12:51this is the exact thing that we need to
1:12:54configure now.
1:12:55That is why I have just added a lot of
1:12:57practical demos within this tutorial so
1:12:58that you actually understand the
1:13:00concepts by doing it. Okay? I know this
1:13:03is a concepts video, but still I want
1:13:05you to understand it. So let me just
1:13:07open my Databricks. So how you can just
1:13:09create this connection? There are so
1:13:11many ways you can directly create the
1:13:12connection from here as well, but let me
1:13:14just show you the recommended approach.
1:13:16Click on catalog
1:13:17and here then click on plus and here you
1:13:20will see something called as external
1:13:22location. Okay? You can click on this as
1:13:24well. You can even create the credential
1:13:26as well. Both are good, but I will
1:13:27simply go with create an external
1:13:29location. Okay?
1:13:30So let me just click on plus button and
1:13:33click on create an external location.
1:13:35So now it will ask me, do you want the
1:13:38manual setup or you want to set up your
1:13:41external location automatically? So
1:13:43here's the thing, whenever you are
1:13:45working in AWS, you can just go with AWS
1:13:47automatic setup. It is actually very
1:13:49good and it saves a lot of time.
1:13:51But with Azure, we do not have this
1:13:53option. Whenever I work with Azure, I
1:13:56always create the credential manually,
1:13:58which is called database
1:14:00Databricks connector ID from the cloud
1:14:03account and then I wrap that
1:14:06you can say credential Databricks
1:14:08connector into a credential within the
1:14:10Databricks and then I just create the
1:14:12external location. So this is like
1:14:13multi-step process. Let me just explain
1:14:15this thing. First of all, you create the
1:14:17credential. And what is that credential?
1:14:19Credential is nothing but the Databricks
1:14:21connector.
1:14:23Databricks
1:14:25Wow.
1:14:26Databricks
1:14:27connector. Okay. That Databricks
1:14:29connector get wrapped inside something
1:14:32called as credential.
1:14:35Cool. And that credential will be used
1:14:37to create your external location.
1:14:41And that external location is used to
1:14:43access your data. But all of these three
1:14:46steps are automatically handled by
1:14:48Databricks if you select this thing in
1:14:50AWS only. Okay. So I will simply say
1:14:52next.
1:14:54Then it will say what should be the
1:14:55external location name. I will simply
1:14:57say
1:14:58S3
1:15:00in just you can just pick any name. Now
1:15:03it wants the bucket name. So I will
1:15:05simply say S3, which is a kind of
1:15:07protocol that we use and then the bucket
1:15:09name. The bucket name is I think let me
1:15:12just check.
1:15:13The bucket name is
1:15:16Databricks demo
1:15:18So I just need to provide that.
1:15:21Databricks demo
1:15:24punch. Cool.
1:15:26And log in to AWS because it needs to
1:15:29verify that okay, you are the authorized
1:15:31one to access this Databricks this
1:15:33particular data lake obviously. So let
1:15:35me just click here.
1:15:37And IAM storage role. So what it will
1:15:39do? It will automatically create this
1:15:41role as well, IAM storage role because
1:15:44obviously for in order to access your
1:15:47data from the data lake, you need the
1:15:49right
1:15:51role as well, right? Which is the
1:15:53concept of role. So whenever you want to
1:15:55access any resource from the AWS, we
1:15:58always create the roles, right? Let me
1:16:00just show you that as well so that you
1:16:02understand what's going on. Nothing is a
1:16:04black box for you. So you simply go to
1:16:06IAM
1:16:08in the AWS and here you go to roles
1:16:11here.
1:16:12And if you just want to let say read S3
1:16:15bucket,
1:16:16you can say I have so many roles data
1:16:18bricks because I just keep on just using
1:16:20that data bricks S3 automatic way of
1:16:22this thing. So let's say S3
1:16:26read
1:16:28where is that role? I think there's no
1:16:30role for that. That is fine. But if you
1:16:32just want to create a new role, let's
1:16:34say I just want to read the S3. So I
1:16:36simply say read S3 and I can just create
1:16:39a role because it doesn't exist. Then it
1:16:41will say
1:16:43trusted entity type AWS service,
1:16:45account, web entity and blah blah blah
1:16:48blah. I will simply say AWS service that
1:16:49is fine because I will be accessing this
1:16:52thing through a kind of AWS service.
1:16:54Then service or use case I want to use
1:16:57S3.
1:16:58Simply say S3 and then click on next.
1:17:01And here I just need to pick the right
1:17:05you can say
1:17:07role, not role basically the access or
1:17:10basically the permissions.
1:17:11I can say S3 access if I have or let say
1:17:14read data lake something like that I can
1:17:17just search it if I have that. So here I
1:17:20have S3 Amazon S3 full access. I can
1:17:23just pick that as well. If I have S3
1:17:26files full access if you just want to
1:17:28access files read only. So this way I
1:17:30have all of these things. This is the
1:17:32manual way of doing it, but this data
1:17:34bricks will do everything for me. So let
1:17:36me just click on initiate and let's wait
1:17:39for it. So it has automatically opened
1:17:41this window for me. Let me just read it
1:17:43and let's say allow access. So this will
1:17:46automatically created this thing for me.
1:17:48But but but you need to wait for it
1:17:50because it takes some time to create
1:17:52this thing and in the back end it can
1:17:54take I would say a few minutes because
1:17:56it will create some functions for you as
1:17:59well so that you can easily access the
1:18:01data. And let me just show you if we can
1:18:04see that particular progress. So let me
1:18:05just go to IAM or not even IAM.
1:18:09Let's go to S3 directly.
1:18:12And here I have this data
1:18:14bricks demo on very good. So this is my
1:18:17raw folder. Everything is good on this
1:18:18side. So let's wait for a moment and
1:18:21then I can just show you if we have any
1:18:23kind of progress on the back end files.
1:18:25Okay. So a few minutes are passed. So
1:18:28now let me just go to Databricks and if
1:18:30I just go to let's say
1:18:32plus icon, create an external location.
1:18:34You can even click here or you can click
1:18:35on go on or connect as well. Click on
1:18:37external locations. This way you will
1:18:38see all of the external locations.
1:18:41So I can see that I created this
1:18:43external location called S3 ingest and
1:18:46this external location is here. That
1:18:48means it is created.
1:18:50If you have not waited for at least a
1:18:52few minutes, you will only see pending
1:18:54locations instead of existing locations.
1:18:57So in pending locations, you will see
1:18:59the message that please wait for a few
1:19:01minutes because some back end working is
1:19:03being done. Once it is done, you will
1:19:05see the existing locations. Otherwise
1:19:07you will only see pending locations. So
1:19:09make sure that you are waiting for the
1:19:12right amount of time.
1:19:13If I just go to credentials, it has
1:19:15created this credential automatically
1:19:16for me. Databricks UC blah blah blah. So
1:19:19this is the
1:19:20credential automatically created for me.
1:19:22I didn't create anything. Cool. What is
1:19:24the advantage now? Because this external
1:19:26location now exist, I can read the data
1:19:29from the
1:19:31external location. Really? Let's test
1:19:32it.
1:19:33Simply go to workspace, click on demo
1:19:35and let's create a new notebook. Let's
1:19:38call it as
1:19:40second
1:19:43S3. Okay. So now let me just create the
1:19:45data frame quickly. So let's say DF
1:19:47equals spark dot read dot format format
1:19:52is CSV.
1:19:53Then
1:19:55let's say dot option
1:19:57header
1:19:58true
1:20:00and then let's say
1:20:01dot load load. Yeah.
1:20:04Let's provide the location. Location
1:20:06will be like your bucket location.
1:20:08Databricks
1:20:10demo on
1:20:11then raw.
1:20:13Then if I just want to display the data
1:20:14frame, I can simply say display DF.
1:20:16This will read the data
1:20:18and then it will display the data as
1:20:20well. Now because my external location
1:20:22exists
1:20:23if I will know if I would not have
1:20:25created that external location, this
1:20:27read
1:20:28would have thrown the error that I do
1:20:30not have permission to read the data
1:20:32from this S3 lake because this data this
1:20:35this data lake is totally different
1:20:37thing, right? But now because our
1:20:39Databricks is attached to it, now we can
1:20:41just read this thing. I hope it makes
1:20:42sense. That's how you can just perform
1:20:44the data ingestion. Okay. This is the
1:20:46very basic, very standard thing that we
1:20:48just perform. Cool. So I hope you
1:20:51understand this thing and now
1:20:53we will see how we can just perform the
1:20:56streaming with the auto loader. Okay,
1:20:57let's see. So now for that, let's
1:20:59prepare our first of all the data
1:21:00source. So I will simply go to S3
1:21:04uh let's click on this bucket and let's
1:21:08create a new folder.
1:21:10Let's say
1:21:11um stream. Okay, stream, simple.
1:21:15And within this, I will
1:21:17upload some data and I have prepared
1:21:20some data for you as well that you can
1:21:21also use. I will just drop this file as
1:21:23well. So this is a kind of JSON file
1:21:27where we have got the day one audits.
1:21:29Okay.
1:21:30And we will ingest this data using the
1:21:32auto loader and then we will add the
1:21:35another day data
1:21:37and then we will run our pipeline again.
1:21:39And it will not load the data which is
1:21:42already loaded. Okay? It will only load
1:21:44the new data, which is called like
1:21:45streaming. Okay?
1:21:47So it will handle all of the things,
1:21:48idempotency.
1:21:50Idempotency is like exactly once
1:21:52behavior. We'll just talk about it. So
1:21:53let's try to see how we can just set up
1:21:55this thing.
1:21:56So let me just go to Databricks
1:21:58workspace and let me just create a new
1:22:00file here.
1:22:02And let me just call it as
1:22:06three
1:22:08auto
1:22:10loader.
1:22:11Now
1:22:12if you just want to create this auto
1:22:15loader, you can say injection, you first
1:22:17of all need to understand that we work
1:22:19with something called as
1:22:21streaming.
1:22:23So here I will also be using streaming
1:22:25instead of Spark. So if I just show you
1:22:27the second notebook, you can see that I
1:22:29have just used spark.read here.
1:22:31But when we just perform the streaming,
1:22:33we use read stream. Okay? This is the
1:22:35very fundamental of Apache Spark. So
1:22:37this is the first thing that you need to
1:22:38consider.
1:22:40Second thing is whenever we work with
1:22:42the streaming concept, we also create
1:22:44something called as checkpoint location.
1:22:47Oh, really? Yeah, checkpoint location.
1:22:49So we will just create that as well. So
1:22:50you do not need to worry.
1:22:51So let me just show you. First of all, I
1:22:53will read
1:22:54the data which is here.
1:22:56df equals spark.readStream.
1:23:02Okay? readStream.format.
1:23:04Here, what will be the format? You will
1:23:07say JSON. No. Here the format will
1:23:09become cloud files. Why? Because in auto
1:23:12loader, we define all of the formats at
1:23:15as cloud files.
1:23:16Then we define what is the format of
1:23:19that cloud files, which is nothing but
1:23:21JSON.
1:23:22Okay? This is one thing that we always
1:23:23perform. So let's say
1:23:25cloud files
1:23:26and then I will simply go to next line
1:23:28and then here I can write option
1:23:32and within the option I can write cloud
1:23:34files.format is JSON.
1:23:38I hope it makes sense. Very good.
1:23:40Now we need to provide the checkpoint
1:23:45Okay, what is that checkpoint location?
1:23:46Basically,
1:23:49checkpoint is the folder
1:23:51which Databricks auto loader will manage
1:23:55for us where it will write all of
1:23:58things. Just you you you just tell me.
1:23:59Let me just show you the diagram.
1:24:02You tell me
1:24:03how Databricks will know that
1:24:06these four files are already processed
1:24:10and this is the new file. How? How
1:24:12Databricks knows this thing? So it know
1:24:15it needs to maintain that particular
1:24:17information somewhere. So that
1:24:19information is nothing but a special
1:24:21folder
1:24:22that it maintains. It's called
1:24:24checkpoint location.
1:24:27Okay, and within that it has something
1:24:29called as RocksDB where you just see all
1:24:31of that stuff created. Checkpoint
1:24:33location. Because in that checkpoint all
1:24:36of this information will be written
1:24:38for streaming
1:24:39and only then this file will be loaded
1:24:41which is the new file.
1:24:42Right? Okay, I hope it makes sense. So
1:24:45now let's quickly create our checkpoint
1:24:48location as well within here. Now a very
1:24:50good question that you can ask is where
1:24:52should we create our checkpoint
1:24:54location?
1:24:55That's a very good thing. You can even
1:24:56create
1:24:58checkpoint location anywhere, but
1:24:59usually how we create is we create
1:25:02something called as first of all the
1:25:04output folder. So let's
1:25:06quickly create the output folder.
1:25:10Or you can just stream output.
1:25:13Stream
1:25:15output. Cool.
1:25:17Click on create folder. Within the
1:25:19stream output folder, we create two
1:25:21folders. One is checkpoint,
1:25:24right? And second one is data.
1:25:27Create folder and then click on data.
1:25:31So now I have two folders. So one folder
1:25:33I will use it for the streaming data.
1:25:35One will be for checkpoint.
1:25:37Make sense? Very good. I can click here
1:25:39and I can just copy the S3 URI if I want
1:25:41to and let me just click on actions.
1:25:45Copy S3 URI.
1:25:47Oh man, me just go here
1:25:50and go here, go here, buckets and this
1:25:53is the bucket stream output.
1:25:56Okay, so this is the checkpoint. Okay,
1:25:57makes sense. I can just copy this
1:25:58particular thing. So
1:26:00cool. Let me just go to this particular
1:26:02thing. So here I will write
1:26:05dot option
1:26:08and here I will simply write the schema
1:26:09location. Cloud files dot schema
1:26:12location and then I need to provide the
1:26:14checkpoint location, right? So here let
1:26:17me just provide the checkpoint location
1:26:18F
1:26:20and I will simply create the checkpoint
1:26:22path. That is it. That is it.
1:26:25Now let's create the variable for it
1:26:27because we do not have the variable. So
1:26:29I'll simply say checkpoint
1:26:32path equals
1:26:34this is the one streaming output and
1:26:36then within that checkpoint.
1:26:38Cool. So this is my checkpoint path that
1:26:40I have just created.
1:26:41Now you will say Anshul, what is this
1:26:43schema location? So whenever we just
1:26:46work with auto loader,
1:26:48it tracks the schema of the data
1:26:51somewhere. Somewhere where? In the
1:26:53checkpoint as well. So this is like
1:26:55purely end-to-end managed Spark
1:26:58structured streaming service available
1:27:00whenever we use the auto loader.
1:27:02Make sense? Very good. So my schema
1:27:04location is already good.
1:27:07Very good.
1:27:08After that, I can just provide the mode,
1:27:11the rescue mode or basically the schema
1:27:14evolution mode for my files.
1:27:16Okay. By default, if I do not provide
1:27:18this thing, this is not mandatory first
1:27:20of all. By default, it will just use the
1:27:23um add columns. Basically, whenever you
1:27:25just add anything new within your source
1:27:28data, it will simply update that thing.
1:27:30But usually we do not add this thing. I
1:27:32always like to have rescue column
1:27:35because in the rescue it will not break
1:27:37my pipeline. It will simply store all of
1:27:39the new columns within the rescue
1:27:41columns. Okay. Very good. Then I just
1:27:43want to load it and I want to load this
1:27:46data from here.
1:27:48Let's say let me copy this path.
1:27:51I think it is just stream.
1:27:54Perfect.
1:27:55So this is my streaming read location,
1:27:59right?
1:28:00That is it. But now I also want to just
1:28:03write this thing somewhere else, right?
1:28:05You can say this instead of display I
1:28:07just want to write it somewhere.
1:28:09So now if I just want to write it, I
1:28:11have two options. I can either
1:28:13create a table as well if I want to. But
1:28:16if I just want to write it to a location
1:28:18that is also available, that is also
1:28:19fine. And I just like to stream to a
1:28:21folder instead of creating a table. So
1:28:23okay.
1:28:24So let's try to do that. df.
1:28:27write stream. So similarly when we have
1:28:28read stream, we also have write stream.
1:28:30Okay. And I want to create a delta table
1:28:33instead of like creating a CSV or
1:28:35parquet, I just want to create a delta
1:28:37streaming table. Okay. So I will simply
1:28:38say delta.
1:28:40And then I will say dot option.
1:28:43And in the option I can pick checkpoint
1:28:45path. Perfect. And this is my checkpoint
1:28:48path. Cool.
1:28:50And then I can also say
1:28:53trigger mode because this is very
1:28:54important whenever we are using the
1:28:56serverless compute. So you cannot have
1:29:00you can say your continuous mode with
1:29:02serverless compute because serverless
1:29:03compute cannot be used for continuous
1:29:06running of pipeline. Here we all only
1:29:08use once equals to true or available now
1:29:10equals to true. Okay. So you can even
1:29:12use once equals to true or available now
1:29:14equals to true. It's fine.
1:29:16Then I will simply say dot start.
1:29:18And in the start you just need to define
1:29:19where you want to push the data. That is
1:29:21it. So I'll simply say I want to push
1:29:24the data to stream output and then
1:29:27data.
1:29:28Cool. I hope it makes sense.
1:29:31Okay.
1:29:32Let's try to run this cell. Let's see
1:29:34what do we get.
1:29:37And I can also see C performance.
1:29:40And let's wait. Okay. I can see stream
1:29:42initializing. That means Spark structure
1:29:44streaming is initialized. Okay.
1:29:46It will also show me some graph,
1:29:48basically some charts. I I it. Very
1:29:51good. So I think this is done.
1:29:53And if I just go to my S3,
1:29:55if I just go to my data tab, perfect. I
1:29:58can see my delta data being loaded. And
1:30:02as I just mentioned in the very
1:30:03beginning that how delta tables are
1:30:05created. So this is my parquet file and
1:30:08this is my delta log.
1:30:09This is the parquet file. Everyone knows
1:30:11this thing, but this is the delta log.
1:30:12Remember I told you that we have two
1:30:14types of files, JSON and CRC files. So
1:30:16this is your delta log.
1:30:19Wow. This is your delta log.
1:30:22That means this data is fine. This data
1:30:25is amazing.
1:30:26Now I can even query this data. Really?
1:30:29Yes. Let me just show you. Let's say if
1:30:31I want to query this data,
1:30:33DF equals spark dot read dot format and
1:30:36this time because I'm reading delta
1:30:38table delta
1:30:40dot load
1:30:42and this is my
1:30:44path.
1:30:45This is my path, right?
1:30:46Then I can say
1:30:48display DF. Let's see what do we get.
1:30:51I should see that data nicely created in
1:30:54my delta table.
1:30:55Cool.
1:30:56So I have this rescue data and I have
1:30:59this key value and everything, order ID,
1:31:01timestamp, custom data. Obviously
1:31:03because all of the columns were like
1:31:05created under this thing and I didn't
1:31:06pass any kind of schema here and that is
1:31:09why I didn't see anything. I can even
1:31:11infer the schema if I want to, but I
1:31:13didn't write that statement. That is why
1:31:15I'm just seeing this thing. Otherwise if
1:31:17you just want to infer the schema, you
1:31:18can also infer the schema. Nothing is
1:31:20like you can say wrong in that
1:31:22particular thing. Okay. But yeah,
1:31:23because I didn't write any infer schema
1:31:25because I didn't want to infer the
1:31:27schema, that is why I'm seeing this
1:31:29thing.
1:31:30Okay. But if you just want to, you can
1:31:31even write that thing.
1:31:33Let's say dot option
1:31:36cloud files
1:31:38dot infer schema um
1:31:42infer schema and then just make it true.
1:31:45Let me also check the documentation. Um
1:31:49schema let me just write autoloader
1:31:53infer schema perfect.
1:31:56So so so so so where is that auto
1:31:59loader? This is amazing browser.
1:32:02So configure schema inference and
1:32:03evolution very good. So we have
1:32:07schema location okay. I want infer
1:32:09schema.
1:32:11Where is that infer schema?
1:32:14Let me let me check schema hints I want
1:32:16schema hints and
1:32:20let me just search it for you. Let's use
1:32:22the same one that we just used it. It's
1:32:23not a big deal. Um so
1:32:27now
1:32:28the point that I just want to make here
1:32:29is
1:32:31we have just loaded this data. That is
1:32:33fine. That is fine and you can like
1:32:35usually what we do before just dumping
1:32:37this data into into the delta lake we
1:32:38also obviously
1:32:40deserialize this JSON and then we just
1:32:42perform some data cleaning to convert
1:32:45this JSON into nice nice looking columns
1:32:47but that is a different thing.
1:32:49Now the main thing that I just want to
1:32:50make is here you can just see that I
1:32:53have order ID 1001 right cool. Now now
1:32:58if I just upload
1:33:00another file here in the stream what
1:33:03will happen then? Let me just upload it.
1:33:06Let me just upload day two JSON.
1:33:10So now I have day two JSON as well. Let
1:33:13me just upload. Do you know what will
1:33:14happen?
1:33:15This time
1:33:17because of checkpoint folder
1:33:20this will only load this new file which
1:33:22is day two dot JSON and it will not
1:33:24touch the previous file.
1:33:26Let see if it works right because we
1:33:28want to test it. So let me I will not do
1:33:30anything. I will simply run this cell
1:33:32again without changing anything. Again
1:33:35the stream will initialize
1:33:37and after some time I should see the
1:33:39results and that's it. Very good.
1:33:42Now if I just run this cell for one more
1:33:44time
1:33:45what do I see let's see.
1:33:47Perfect I only have a new order 1002. I
1:33:50do not have a duplicate order 1001
1:33:53again.
1:33:54Oh but how it happened? It due to this
1:33:58checkpoint location. And where is that
1:34:00checkpoint location?
1:34:02The checkpoint location is here in the
1:34:04stream output
1:34:06in the checkpoint folder. Wow, we have
1:34:08so many folders, right? But not all the
1:34:11folders are for checkpoint. One folder
1:34:13is also for schema location. Okay, so
1:34:15this folder has the schemas. This
1:34:18metadata has some metadata information
1:34:19and obviously some offsets, some
1:34:21sources, everything. So in the sources
1:34:23we have this option
1:34:24and then in the offsets we have the
1:34:27offsets available. If I go to metadata,
1:34:30I will see some metadata information for
1:34:32this particular stream.
1:34:34And if I go to commits, I will see some
1:34:36commits and if I just go to schemas,
1:34:38then obviously some schema information
1:34:40is there. All the files are visible to
1:34:41you. Do you know why? Because you own
1:34:43this external location. That is why it
1:34:46is available to you.
1:34:48Oh, makes sense.
1:34:50Now
1:34:51let me also show you the folder that I
1:34:52just um
1:34:54took name in the beginning, Rocks DB,
1:34:56because Rocks DB is one of the major
1:34:58folder. Obviously every folder is like
1:35:00important, but Rocks DB is that folder
1:35:02which holds the state of your stream.
1:35:05So let me just go to sources, click on
1:35:07the zero and within that you will see
1:35:08Rocks DB. Rocks DB will have all of the
1:35:11information like, okay, this data was
1:35:12processed, this data is new and
1:35:14everything.
1:35:15Everything is being managed by Data
1:35:17Bricks just for you
1:35:19using Auto Loader.
1:35:20This is the power of Auto Loader.
1:35:23Now because we use the trigger type as
1:35:25once equals to true, that means whenever
1:35:27the stream was completed, it turned off
1:35:30the pipeline.
1:35:32But if you want to keep on running this
1:35:34pipeline, what you can do? You can
1:35:35simply pick a new trigger type, let's
1:35:37say processing time equals to 10
1:35:39seconds, 5 seconds, 3 seconds. And now I
1:35:42think Data Bricks has even um um you can
1:35:44say initialize something called as
1:35:46real-time mode. So it depends like how
1:35:48frequently you want to run the pipeline.
1:35:50Because we are using the free edition,
1:35:51obviously they will not provide us the
1:35:53compute that we can just run 24/7,
1:35:55obviously. But if you just want to, you
1:35:57can just pick the trigger type here
1:35:59well.
1:36:00This, let me just tell you, this is
1:36:02actually one of the like complex
1:36:04concepts. If you understood it very
1:36:06well, very well done. If not, that is
1:36:08fine. You can just rewatch this part.
1:36:11Why? Because that's how you learn.
1:36:13That's how you learn, right?
1:36:14So this is one of the end-to-end example
1:36:17of how streaming works, how data
1:36:19ingestion works, how Databricks lake
1:36:21flow connect works with examples. I hope
1:36:24this entire part is crystal clear.
1:36:27I hope so. Very well done. So now the
1:36:29next concept that you need to learn is
1:36:32DB utils. What is what what are DB
1:36:34utils? And what is this DB utils
1:36:37module or concept that you need to
1:36:38learn? So first of all,
1:36:40DB utils is basically a package of
1:36:42modules that Databricks has created for
1:36:45you or for all of the developers. So
1:36:48let's say we all are using Databricks
1:36:51and obviously we need to perform so many
1:36:54stuff and we can do everything on our
1:36:56own. We have APIs, we have SDKs, we have
1:36:59literally everything, right?
1:37:01But sometimes we need some helper
1:37:03functions, helper modules so that we can
1:37:06efficiently work with Databricks. And
1:37:09here the core thing is using the code.
1:37:13Let's say we just want to perform
1:37:15everything using our code within our
1:37:17notebooks, anything.
1:37:18Okay.
1:37:19So these are some of the utilities that
1:37:21are created by the Databricks.
1:37:23Have you ever worked with something
1:37:25called as OS module in Python? Now
1:37:28everyone knows that if we want to work
1:37:30with let's say file management or
1:37:32anything within the programming
1:37:34language, we can do it via so many ways.
1:37:37But when we just use these packages,
1:37:39these modules like OS,
1:37:41we see our work
1:37:43efficiency is boosted, right? Similarly,
1:37:46if we just talk about Databricks, we
1:37:48also have the similar
1:37:50component within Databricks within DB
1:37:52utils. It's called
1:37:54DB utils dot FS. Okay. FS means like the
1:37:58short form for file systems. So it is
1:37:59very similar to that. So similarly,
1:38:02similar to this, we have so many other
1:38:05utilities like dbutils.fs,
1:38:08then dbutils.widget,
1:38:10dbutils.
1:38:13um jobs,
1:38:15so many stuff. And I think they have
1:38:16recently introduced one new one as well.
1:38:19It is in experimental zone. It's called
1:38:21data.
1:38:22Okay.
1:38:23So these are so many dbutils that we
1:38:25have. And just to tell you, 80% of the
1:38:29time you will be interacting with this
1:38:31particular module because this is so so
1:38:34so efficient. If you want to let say
1:38:36list the files of a of any particular
1:38:40bucket, of any particular folder, how
1:38:42will you do that? Obviously, there are
1:38:44multiple ways. You can just run a loop,
1:38:46you can just send the API request to the
1:38:48folder to list the components directly
1:38:50via the SDK. But
1:38:52when you connect
1:38:54the cloud with Databricks, you literally
1:38:56can use dbutils module to directly fetch
1:39:00the details of all the content that we
1:39:02have within any folder.
1:39:04This is just a very simple example of
1:39:06it.
1:39:07Cool. So if you are a Databricks
1:39:09developer and if you are just trying to
1:39:11become one, this package, this module is
1:39:14non-negotiable for you. Okay. So now
1:39:16let's try to actually see it with some
1:39:19hands-on examples. So let me just bring
1:39:21my Databricks workspace here so that you
1:39:25can also see these things. So let's go
1:39:27to our demo folder that we just created
1:39:30and let's create a new notebook
1:39:32and let's name it as Did I create
1:39:35notebook? Yeah, yeah, perfect. So this
1:39:37is my notebook.
1:39:38Perfect. And here I can say
1:39:43db
1:39:44utils. Cool. So this is my dbutils
1:39:46notebook. And as you know that we have
1:39:50already connected our Databricks
1:39:52workspace to our S3 bucket, right? That
1:39:54means we can access it via this
1:39:56Databricks. We all know that. But let's
1:39:58see how we can just use the dbutils
1:40:00module. Okay.
1:40:01So first of all, you do not even need to
1:40:03import dbutils like this. Like you can
1:40:05write this thing, but by default, when
1:40:07you just run the Databricks notebooks,
1:40:09it is automatically
1:40:11imported here. So that is why you will
1:40:13see something like
1:40:14module not found error, no module called
1:40:17DBUtils.
1:40:18So do not just try to import it, okay?
1:40:21Because I have seen like people import
1:40:23from DBUtils import this, import DBUtils
1:40:25something blah blah blah. So you do not
1:40:27even need to import it like this. It is
1:40:29already
1:40:30integrated within your Databricks
1:40:31workspace.
1:40:32So first of all, if you want to see all
1:40:34of the options available within DBUtils,
1:40:37you first need to run this command
1:40:39DBUtils.
1:40:41help.
1:40:42Then what will happen?
1:40:43DB
1:40:45Utils DB
1:40:48Utils. help.
1:40:50So it is saying bound method and let me
1:40:51just write this DBUtils. help. So
1:40:54perfect. So these are all of the
1:40:56packages, mini mini packages or you can
1:40:59say mini mini modules created within the
1:41:01DBUtils.
1:41:02So we have credentials, we have data, we
1:41:05have See, I told you that this is the
1:41:07new one and it is in the experimental
1:41:08phase.
1:41:09Then FS, this is the most important one
1:41:12that you need to
1:41:14learn. Then we have jobs. This is used
1:41:17whenever we want to work with the
1:41:19Databricks jobs and do not worry, we
1:41:20will just talk about Databricks jobs as
1:41:22well. That is obviously a very important
1:41:24area for orchestration.
1:41:26Then we have some library meta. This is
1:41:28also new one. This is like recently
1:41:30added. Then we have Notebook Utils, this
1:41:32one, DBUtils. notebook.
1:41:35The previous method or you can say to
1:41:37use this thing was DBUtils, but Notebook
1:41:40Utils, but Notebook Utils like is there
1:41:42I think since so long, so it is still in
1:41:44the experimental, but we have this thing
1:41:46whenever we just want to fetch anything
1:41:47from the notebook, we just use this
1:41:49thing. Then we have preview, then we
1:41:51have secret. Secret is very important.
1:41:52This is used when you are working with
1:41:54let's say key vaults, when you are
1:41:55working with the secrets and you do not
1:41:57want to expose the sensitive data within
1:42:00the notebooks. So what you do? You do
1:42:02not hard code any API credential or any
1:42:05kind of key vault. You simply create the
1:42:07secret for that. Cool. And then finally
1:42:10you have widgets.
1:42:12This is basically used for the
1:42:13parameterization. So whenever you want
1:42:15to see the list of DB utils, always run
1:42:17dbutils.help first of all. Cool.
1:42:19Now let's try to see some more examples.
1:42:21So let's say now I want to
1:42:24um explore dbutils.fs command. So what I
1:42:26can do? I can simply say dbutils
1:42:29.fs and if I just want to learn more
1:42:32about this FS module, FS um you can say
1:42:35component within DB utils module, then I
1:42:37can simply say help. And I will see all
1:42:40of the options available that I have
1:42:42within this particular module.
1:42:44So first of all, I have this particular
1:42:45information. Cool. Then I have some
1:42:48commands that I can just use. CP.
1:42:50CP is a short form for copy. If I want
1:42:53to move the data, or not just move
1:42:55basically, if I just want to copy the
1:42:57data from one place to another,
1:42:59I do not even need to write any code for
1:43:02let say moving data from one place to
1:43:04another. No ETL is required. You can
1:43:07literally use CP method and the data
1:43:08will be moved. Okay? Moved means like it
1:43:10will be copied.
1:43:12Then head if you just want to return
1:43:14some top records, then LS.
1:43:17LS is um you can say most of the time
1:43:19used with the list comprehensions. If
1:43:21you just want to see what do I have
1:43:23within my folder, um I want to fetch
1:43:26only the parquet files out of so many
1:43:28files. So these this particular
1:43:29component is used. Then mkdirs, which is
1:43:31very similar to the one that we just use
1:43:34within the Windows or any particular OS
1:43:36to create the directories. Then mount.
1:43:39Mount was used very frequently in the
1:43:42early days when we didn't have the Unity
1:43:44catalog. But when we have Unity catalog,
1:43:46we hardly use mount. But yeah, you can
1:43:48still use mount. Okay?
1:43:49Then we have move. If you want to move
1:43:51the data, then we have put. If you want
1:43:52to create the file, then we have remove.
1:43:55Let say if you want to remove any folder
1:43:56and all of these things. Okay? So this
1:43:58is very similar to OS module that we
1:44:01have within the Python. Cool. So let's
1:44:03say I want to I know that I have this
1:44:06folder, right? Um which folder? Um this
1:44:08one. So I have this
1:44:11Let's say I want to list what do I have
1:44:13inside this folder. So what I can do
1:44:15dbutils. fs. ls and then I can just
1:44:19provide the path.
1:44:20And let's see what do I get.
1:44:22It will return everything within the
1:44:24list. So this is the list and in the
1:44:26list I will have the
1:44:28you can say different different
1:44:29components. First of all file info.
1:44:31File info is this. So I have the path.
1:44:34Okay. I have the path. This is the path
1:44:36and name is metadata. Metadata means
1:44:39this is the component that I have within
1:44:41that. Okay. Then size and then this.
1:44:43Then I have file info. Then I have
1:44:45schemas folder. Then I have this folder
1:44:47commits offset sources. Very good.
1:44:50That's how you can just
1:44:53identify anything that you have within
1:44:54the folder. So how we use it? Let me
1:44:56show you.
1:44:58So we can let say
1:45:00create the path like this and then I can
1:45:03just run the loop
1:45:05in this particular thing and then I can
1:45:06say print or not even print. Let's use
1:45:09the list comprehension. So I can say
1:45:13my list equals
1:45:15and then I want to run i. let's say path
1:45:19for i in
1:45:22there.
1:45:23Cool. And then if I just print my list
1:45:26then I should see only the path. So
1:45:28that's how we just create the automation
1:45:31whenever you want to let say
1:45:33push the data to a specific location or
1:45:36pull the data from a specific location.
1:45:38We just use these kinds of list
1:45:39comprehensions and this way you have
1:45:41already seen that I have literally
1:45:43fetched all of the file paths. I can
1:45:45even just write some conditions here.
1:45:46Let's say if
1:45:49i.
1:45:50file path or i. path
1:45:52equals equals or let's say anything. If
1:45:55if I just let say I want to fetch only
1:45:57the metadata path. Okay. Because it can
1:45:59be the condition. Right? So let's I just
1:46:01want to
1:46:02write this thing checkpoint and then
1:46:04after that
1:46:05not
1:46:06committed I will simply say schemas.
1:46:11You can even perform some, you can say
1:46:15I do not have this thing. Wow. So, let
1:46:17me just write this thing.
1:46:21Uh
1:46:23Checkpoint schemas.
1:46:27Obviously, I will not see this thing.
1:46:28So, let me just comment it out.
1:46:31Just to see what do I have. So, I have
1:46:34the schemas. Okay. Got it. So, I have
1:46:36one more slash
1:46:40schemas and then slash.
1:46:42So, this way you can perform some, you
1:46:44can say character regex to fetch only
1:46:47the specific folders, only the specific
1:46:49files from the folder. So, this is the
1:46:52automated way to perform anything
1:46:53related to that. Makes sense? Very good.
1:46:56So, now let's try to use more utilities.
1:46:58Let see what more folders do we have. I
1:47:01think we also have the data, right? So,
1:47:04let's do one thing. Let's try to copy
1:47:05this data
1:47:07to a different one. Okay. And I will
1:47:09simply use db utils dot
1:47:12fs dot mkdirs because I want to create
1:47:15the folder. And I will create the
1:47:17folder, let's say stream output. And I
1:47:19will say um db util data. db utils data.
1:47:23So, this will create the folder like
1:47:24this. Cool. My folder is created. Now
1:47:27what I will do? I will simply say db
1:47:29utils dot fs dot cp. cp means copy. And
1:47:34what I want to copy? I will simply write
1:47:36this path. So, I want to copy
1:47:39this particular folder
1:47:41to
1:47:42this particular folder. And this folder
1:47:44is db utils
1:47:47data. Makes sense? Cool. Um uh I was
1:47:51just like expecting this particular
1:47:53error because we have the directory
1:47:55within this. Like we have the delta log
1:47:57as well. So, that is why it is saying
1:47:58that cannot copy directory
1:48:01unless recurse is set to true. So, I can
1:48:03literally write here um recurse is true.
1:48:07Then it can just keep on copying
1:48:09everything, everything, everything to
1:48:10the new folder.
1:48:11So, perfect. It is done. Are you
1:48:13serious? Yes.
1:48:14Yes, I'm serious. I can even show you.
1:48:17So DBUTL obviously, you can just go to
1:48:18the S3 account and you can just confirm
1:48:20this thing, but I do not need to,
1:48:21because I have everything everything I
1:48:24have access to everything within my
1:48:25Databricks itself. So I will simply say
1:48:26DBUTL.
1:48:28fs.ls
1:48:30and here I will simply write
1:48:32DBUTL data folder and let's see what do
1:48:35I have inside this folder. Perfect. I
1:48:36have three things.
1:48:38First parquet file, second parquet file
1:48:40and third is this delta log and that I
1:48:42know this thing.
1:48:43That means I
1:48:45do not even need to write any ETL code,
1:48:48any kind of script, nothing. I can
1:48:50literally perform this CP.
1:48:52This is efficient when you just want to
1:48:54perform one time copy or you are just
1:48:57literally dumping the data from one
1:48:59location to another. This is a very
1:49:01useful command and to be honest, when
1:49:03you are just building the automations,
1:49:05you might need to move some data as
1:49:07well. Let's say you just process some
1:49:09data and let me just tell you one thing.
1:49:11All of these managed thing like this
1:49:14auto loader and all of these things, if
1:49:16you just try to think about it like how
1:49:17these things are built, under the hood
1:49:20they are literally using these things.
1:49:21How? See, let's say you just process
1:49:24some data, right? And you do not want to
1:49:26process the same data. You can literally
1:49:28move the existing copied data to a
1:49:30different folder using the move command
1:49:32and then obviously the target folder
1:49:33will be empty. So you can just compare
1:49:35what files are copied or not and that's
1:49:37it. So these are some internal things
1:49:38that you can even build by yourself, but
1:49:40obviously because these things are
1:49:42already built with
1:49:43considering all of the edge cases, you
1:49:45do not even need to worry about all of
1:49:47the edge cases, any failures, nothing
1:49:49because everything is tested. So then
1:49:52you just use these managed tools. But it
1:49:54is not like you cannot build by your by
1:49:56your own, right? Cool. So that's how you
1:49:59can just use DBUTL module and I hope
1:50:01this module will be very very helpful
1:50:03for you and now I want to just tell you
1:50:05a little bit more about one more thing
1:50:08which is DBUTL.
1:50:11So now let's say I want to create the
1:50:12parameters for my notebook. I'm not
1:50:15talking about variables. I'm talking
1:50:16about parameters. Parameters and
1:50:18variables are two different things. We
1:50:20create variables if we want to, let's
1:50:23say use dynamic
1:50:26text, dynamic numbers, anything within
1:50:28my code.
1:50:29We use parameters
1:50:31and we change the value for the
1:50:33parameters at the runtime.
1:50:35Makes sense? So let's say I have the let
1:50:38me just show you the example. Let's say
1:50:40I'm just doing something within my
1:50:41notebook and I have something like this.
1:50:43Let's say
1:50:45print
1:50:47data is loaded from or let say
1:50:52data is loaded for
1:50:55date. Okay, I do not know for which
1:50:58date. Let's say in this notebook I have
1:51:00just built the pipeline to fetch the
1:51:01data from somewhere and then load it
1:51:04somewhere. Cool? And every day I need to
1:51:07run this notebook on a daily basis.
1:51:09Obviously, then what I can do, every day
1:51:12I can provide a new date parameter so
1:51:14that every time I am fetching the data,
1:51:17it will only load the data greater than
1:51:19that particular date. That means I can
1:51:21supply the last load date.
1:51:23Makes sense? So like I can even write
1:51:25last load date. And this is a very
1:51:27classic and very traditional example how
1:51:30we used to perform the incremental data
1:51:32loading.
1:51:33Makes sense? So you can just imagine
1:51:35that I have just written some code to
1:51:37fetch the data from SQL. I can even
1:51:39write the query for you if you just want
1:51:40to. So let's say
1:51:43select asterisk from let say from your
1:51:46orders table
1:51:47where
1:51:50order date
1:51:53greater than
1:51:54last load date. Simple. I'm just
1:51:55printing it because I do not have the
1:51:57SQL connection right now. So I am just
1:51:59printing this thing. But you can imagine
1:52:00I'm I am not printing it printing this
1:52:02thing. I am literally executing this
1:52:04thing, right? So now what will happen?
1:52:06Every day if I just run this particular
1:52:08notebook, every time it will fetch all
1:52:11of the orders greater than last load
1:52:13date. So let's say today is seventh of
1:52:17today seventh or yeah, today is seventh
1:52:18of November. So now what I can do? Today
1:52:21is 7th of November. So let's imagine
1:52:24that I have loaded the data
1:52:26for 1st of November, okay? Till 5th of
1:52:30November.
1:52:31So till 5th of November my data is
1:52:33loaded, okay? Now today on 7th I want to
1:52:38load the data greater than what?
1:52:41Greater than 5th
1:52:42obviously so that I can just get all of
1:52:45the data after 5th. I will get the data
1:52:46for 6th and um 7th as well, not the
1:52:49entire day but yeah for
1:52:51few hours. So this is the date, this 5th
1:52:53November should be passed every time to
1:52:56this particular notebook.
1:52:57And now let's say once this particular
1:53:00notebook is notebook run is completed,
1:53:03then let's say next day or let's say on
1:53:05Monday you want to rerun this particular
1:53:08notebook, then what date you will
1:53:09provide? Let's say 7th November.
1:53:12Makes sense? So every time this date is
1:53:14being changed, right? This particular
1:53:16thing needs to be addressed here and how
1:53:18we can just address this thing? We just
1:53:20create the widgets.
1:53:21So I will simply say DB utils dot
1:53:25widgets dot
1:53:27set
1:53:28and here I can literally set any name,
1:53:31let's say last load date and I just
1:53:32provided a default value because we need
1:53:34to provide default value which is a good
1:53:36practice. You can even ignore the
1:53:37default value, it's fine but I usually
1:53:39just try to keep it. And what we
1:53:41provide? 1900 01 01. Why do we just
1:53:45provide this date? This is called
1:53:47initial load. This is called full load.
1:53:49Let's say we want to load all the data
1:53:52and when we do not have any kind of
1:53:53date, it will literally load all of the
1:53:55data that we have so far. So if I just
1:53:57run this thing
1:53:58um set widgets dot
1:54:02set what? Widgets dot set or get?
1:54:06I think widgets dot set it was widgets
1:54:09dot set or get because get is used to
1:54:10get the get the thing.
1:54:12Widgets dot set
1:54:15data text.
1:54:19Oh, it's not set, it's text, sorry.
1:54:21Because I was just confused between set
1:54:22and get. So it is text.
1:54:25So now what will happen? This will
1:54:27create this particular box for me. And
1:54:29now what I can do, I can literally
1:54:31override this parameter within my
1:54:33notebook as well when I am running this
1:54:35within my development environment. when
1:54:37we just do it in the you can say
1:54:39orchestration in the form of job, we
1:54:41override this parameter directly via the
1:54:43Databricks jobs. And we will just talk
1:54:45about Databricks jobs very very soon. So
1:54:46do not worry about that. First you need
1:54:48to understand that this state now can be
1:54:50overridden. And now how we use this
1:54:52thing, let's say this is there, but how
1:54:54we use it? Let me show you. So we use a
1:54:57special kind of module and that's where
1:54:59I was just confused. dbutils.widgets.get
1:55:03and here I just need to provide the name
1:55:05of it.
1:55:06That is it. And I can store this value
1:55:09in the variable
1:55:11like this. Oh. So now if I run this
1:55:14thing,
1:55:16this will get the print statement like
1:55:17this. Last load date wow. Let me just
1:55:19try a test string here.
1:55:22Perfect.
1:55:24Um last load date is not found. Very
1:55:26good because we had last load
1:55:28variable.
1:55:30Cool. Now you can see that it is
1:55:31printing 19000101.
1:55:33Now if I just override this thing, let's
1:55:35say
1:55:36um
1:55:372026
1:55:3911 05
1:55:41if I just
1:55:43run this thing now, it will show me this
1:55:44date. So we override this date using
1:55:47this particular module like this.
1:55:50Makes sense? And when you are just
1:55:51developing this thing in the development
1:55:52environment, you just do it directly via
1:55:54this notebook. But when you are
1:55:57orchestrating this notebook, you are
1:55:58done with the development and you know
1:55:59you just need to orchestrate it, you
1:56:01override this parameter directly via the
1:56:04jobs.
1:56:05We will just look at it. Do not worry.
1:56:06So this is the way we just use the
1:56:08dbutils command.
1:56:09I hope this makes you um you can say
1:56:13aware about this module very well so
1:56:15that you can just explore more and you
1:56:16can just learn more. But to be honest,
1:56:1880% of the time we just use FS and
1:56:21widgets commands, but obviously
1:56:23sometimes using secrets as well. So
1:56:24yeah, that's it. Makes sense? Very good.
1:56:27So now let's talk about this amazing
1:56:29thing and you might feel like is it
1:56:32related to Databricks only? Um not
1:56:34actually because this is a kind of
1:56:35design thing that we just built. But you
1:56:37know what? Databricks was one of the
1:56:40earliest, you can say, creators of
1:56:43medallion architecture because they just
1:56:45came up with the lakehouse architecture
1:56:47and this medallion thing was also be
1:56:49like very popular into that particular
1:56:51um structure.
1:56:52Do not worry about lakehouse, okay?
1:56:54Because lakehouse is nothing but when
1:56:56you just put your data in the data lake
1:56:57and you just query the data using your
1:56:59um you can say engine on top of it. So
1:57:02it is totally fine. You do not need to
1:57:03worry about lakehouse, data lake,
1:57:05warehouse. These are the architecture
1:57:07things. We are talking about Databricks
1:57:10and medallion architecture is the thing
1:57:12which is not actually a kind of
1:57:15data architecture.
1:57:17This is a kind of strategy that we use
1:57:20to build our final data architecture.
1:57:22Let me just make it clear.
1:57:24So first of all,
1:57:25when we talk about data architecture, we
1:57:27talk about lambda architecture, kappa
1:57:29architecture, lakehouse architecture,
1:57:31data mesh, data fabric. These are the
1:57:32pure data architecture things.
1:57:34Medallion architecture is the journey of
1:57:37the data. Let me just tell you. So for
1:57:39example, let's say
1:57:40um I want to use Databricks, right?
1:57:43for the entire data lifecycle. Cool.
1:57:46Very good.
1:57:47Now obviously there can be some sources.
1:57:49Um sources can be let say S3, which is
1:57:52very popular one, data lake,
1:57:54database,
1:57:56database, um database, okay?
1:58:00Then you can also have maybe Kafka. Like
1:58:02you can just imagine anything. Let's say
1:58:03Event Hub. Let's bring some Azure stuff
1:58:06as well. So let's say Event Hub. These
1:58:08are some of the very popular popular
1:58:10sources that we data engineers deal with
1:58:12a lot of times, right? Simple.
1:58:15And the end goal is to
1:58:18store this data or you can say
1:58:21not store, basically represent. We want
1:58:24to build the presentation layer, the
1:58:26final layer, the serving layer
1:58:28into the Databricks.
1:58:30Now obviously, Databricks does not hold
1:58:32your data. It will be attached to the
1:58:34data like under the hood. That is like
1:58:36non-negotiable. We know that thing. But
1:58:38you need to understand this thing that
1:58:39at the end we will be using Databricks
1:58:42to just build reports, let's say. I want
1:58:45to build reports, let's say
1:58:48Power BI, okay, um Tableau, blah blah
1:58:51blah. I don't know who is using Tableau
1:58:53nowadays. Angela, a lot of people are
1:58:55using. Yeah, yeah. I don't know why they
1:58:56are still using because
1:58:59outdated stuff. So Power BI is there,
1:59:01right? You can have, let say, some data
1:59:03scientists, data science people, AI
1:59:05engineers, anything because AI is also
1:59:07built on top of the final data. Let's
1:59:08say we have some agents as well, right?
1:59:10Nowadays
1:59:11agents are the most popular stakeholders
1:59:14for all the data. So let's say agents
1:59:16are using this data. Perfect, makes
1:59:18sense.
1:59:19Now you think, you think that it is just
1:59:23point A to point B journey. It is not
1:59:26the reality. There are some, you can
1:59:28say, checkpoints in the between. There
1:59:30are some, you can say,
1:59:33small small
1:59:35destinations that you just need to
1:59:37actually see. So what are these
1:59:39destinations? Let me just tell you. The
1:59:40first of all, you will see the very
1:59:43first thing which is nothing but
1:59:46a layer. This is the layer
1:59:49that we use
1:59:50to literally dump our data in as it is
1:59:54form. What do I mean? If I'm just
1:59:56ingesting the data from AWS S3,
1:59:58I will have one layer, I will have one
2:00:00stop where I will literally create the
2:00:03replica of the data. Why do I do that?
2:00:05So that I also own the data which is
2:00:08available on the source side.
2:00:11Cool. We just use it and we just call
2:00:13this thing as
2:00:15bronze layer. Okay? Bronze.
2:00:17Then we perform some cleaning. We
2:00:21perform some transformation. We perform
2:00:23some, you can say,
2:00:26some kind of manipulation, right? This
2:00:29is called transformation and this layer
2:00:30is called silver layer.
2:00:32Here the data will be cleaned, first of
2:00:34all. Data will be deduplicated.
2:00:37Data will be
2:00:38joined. Okay? Data will be created in a
2:00:41kind of a separate structure that we
2:00:44want to use it, our organizations want
2:00:46to use. In some scenarios, we might
2:00:48create the entities as well, like
2:00:50totally different entities out of the
2:00:52data that we have the bronze layer.
2:00:54In short, data will be cleaned.
2:00:57Simple? Very good.
2:00:58The third thing that we have is the
2:01:00final thing, which is the gold layer.
2:01:02Okay?
2:01:03In the gold layer, data will be
2:01:05available
2:01:07in the
2:01:08final, you can say
2:01:12layer or you can say in the final
2:01:13structure. Like this data is the final
2:01:15one.
2:01:16Here you create the models, dimensional
2:01:18data models, fact tables,
2:01:20dimension tables. If you are working
2:01:22with data vault, you will create
2:01:23satellite links, all of these things.
2:01:26Make sense? This is the gold layer. In
2:01:28some scenarios, you might also create
2:01:29OBD, which is one big table.
2:01:31So in short, gold layer is just to serve
2:01:33our data. So in short, we can say that
2:01:35we connect our data to this gold layer.
2:01:38Oh, yes. But we also have B and S as
2:01:41well.
2:01:43Okay? Bronze, silver, gold. This is the
2:01:45entire medallion architecture.
2:01:47It is not a structure, it is not an
2:01:49architecture, it is not a design. It is
2:01:51just a journey of the data which says
2:01:54that
2:01:55data will be much more clean and
2:01:58reliable as it moves forward the line.
2:02:01Simple. Because you can less rely on
2:02:03this data because it is literally raw
2:02:04data. You can rely more on this data
2:02:06because it is at least clean,
2:02:08deduplicated and blah blah blah. You can
2:02:10most rely on this data because this is
2:02:12the final data that we are representing.
2:02:15I hope it makes sense. This is the
2:02:16medallion architecture.
2:02:18Okay?
2:02:19Now let's try to see this thing with the
2:02:21help of an example. We will create three
2:02:23notebooks. The first one, I will simply
2:02:25ingest the data. In the second one, we
2:02:27might perform some transformation,
2:02:28simple transformation, because obviously
2:02:30we're not mastering PySpark here.
2:02:32PySpark, we know that we need to master
2:02:34PySpark for Databricks. And if you just
2:02:36want to learn PySpark, you can literally
2:02:38refer to a dedicated PySpark resource.
2:02:39And I also have created the PySpark
2:02:41course. You can literally go through
2:02:42that particular one. And I hope that you
2:02:44might have some PySpark experience.
2:02:46Only then you are just learning
2:02:47Databricks, right? So we will be just
2:02:49using PySpark and then I will just show
2:02:51you that how we can just serve our
2:02:53table. We can even create the aggregated
2:02:54views. The
2:02:57and you can say
2:02:59the motive here is not to just show you
2:03:01how you can just master PySpark. The
2:03:02motive is how you can just learn the
2:03:04Medallion architecture. Like what is the
2:03:06intention behind it? Okay, so we'll just
2:03:08perform very very simple simple you can
2:03:09say transformations. And then
2:03:12the next topic is directly linked to it,
2:03:14which is Databricks jobs. Then we will
2:03:16see how we can orchestrate our
2:03:18notebooks, orchestrate our resources,
2:03:19everything. Cool. So let's try to
2:03:21perform this thing. And let me just
2:03:23bring my Databricks workspace.
2:03:25So this is my Databricks workspace.
2:03:27Let me just create a new notebook. Let
2:03:29me just click on this folder. Let me
2:03:31just say create and new notebook.
2:03:34Perfect. So this notebook will be called
2:03:36as fifth
2:03:38bronze.
2:03:39I can even create everything within the
2:03:41same notebook, but I want you to feel
2:03:43the real world taste as well. So that is
2:03:45why I have just created this new
2:03:46notebook. And in this notebook, first of
2:03:48all, I will ingest the data. And we know
2:03:50that we have data in the orders table,
2:03:52if you remember.
2:03:54You I think we just dumped some data in
2:03:57the S3, right? Let me just open my AWS
2:04:00account as well in parallel so that we
2:04:01can just see what do we have.
2:04:03So I have opened my AWS account and let
2:04:06me just click on S3
2:04:08just to see what do we have. So we have
2:04:09this particular lake and we have this
2:04:11raw data, right?
2:04:13So what we're going to do, we will open
2:04:14this raw table, okay, or basically raw
2:04:16folder. And then we will let me just
2:04:20dump more data if I have more data. Let
2:04:22me just see it here.
2:04:25So
2:04:27yeah, I think I have more data.
2:04:29So let me just dump at least one to two
2:04:31more files. And yeah, so let me just
2:04:34upload.
2:04:35Um, I have products as well.
2:04:38So let's upload products.
2:04:41And should we upload order items as
2:04:44well? Order items, yes. And then we can
2:04:46also
2:04:48import employees. Perfect. So I have
2:04:51uploaded these three files as well. And
2:04:53now all of this data is in my S3.
2:04:56Now you can imagine that this is our
2:04:58source and not imagine like this is our
2:05:00source and within the source we have
2:05:02these these files. Okay? And this is
2:05:03sitting in the S3. This is our source.
2:05:06Now
2:05:07this is a kind of you can say OLTP style
2:05:10data where we have orders, order items,
2:05:12products, employees, all of these
2:05:13things, right? So now what I will do in
2:05:15the very first notebook, I will ingest
2:05:18this data.
2:05:19Where? In my um bronze layer, right? And
2:05:23in order to create the bronze layer, I
2:05:25can also have the dedicated schemas for
2:05:28that like bronze, silver, gold. I can
2:05:30just have this thing as well if I want
2:05:32to. So what I will do, I will simply say
2:05:34create schema
2:05:36if not exists.
2:05:39And the schema name will be um catalog
2:05:41name. What is my catalog name? Let me
2:05:43just check. So catalog name is demo.
2:05:46I'll simply say demo dot
2:05:48bronze. Let me just run this thing.
2:05:50So here I will truly create the schema.
2:05:52That's it. Cool.
2:05:54Very good. So once the schema is here,
2:05:56then what I will do, I will simply
2:05:57ingest the data one by one. And what I
2:06:00need to do? Nothing.
2:06:02Simply dump the data in as it is form,
2:06:05in as it is. Yes, you can only do one
2:06:08thing which is like very obvious. You
2:06:09can just change the format of the data
2:06:11because what we do usually, we do not
2:06:14want to keep our data in the CSV format.
2:06:16We want to have the data at least in the
2:06:18delta format, right? So what you can do,
2:06:20you do not need to transform anything,
2:06:22but you can ingest the data in the delta
2:06:25tables or basically in the delta format.
2:06:28Makes sense? How you can just do that?
2:06:29Let me just show you very basics of
2:06:31PySpark. So first of all, you will just
2:06:33perform this thing. DF equals spark dot
2:06:36read dot format format is CSV, right?
2:06:40And here you will simply write your
2:06:41option.
2:06:43Option is header
2:06:45true, then one more
2:06:48which is inverse schema because by
2:06:50default we do not get these schema in a
2:06:52CSV file. CSV is just a text file. I'll
2:06:54simply say true and
2:06:56finally I'll simply say load. Now this
2:06:59thing is very very
2:07:00good for you as well being a developer.
2:07:01So now as you just saw that we have
2:07:03multiple files in my S3 bucket. I do not
2:07:07like to ingest the data one by one
2:07:09duplicating my code. No.
2:07:12I always promote modular coding. You
2:07:14need to be a programmer where you can
2:07:16just literally
2:07:17perform the things
2:07:19efficiently. So what I will do? I will
2:07:20literally I will literally run the loop.
2:07:22I'll simply say for I in
2:07:26db utils dot fl dot ls. Remember I just
2:07:29talked about this thing. You can
2:07:30literally fetch everything. So here I
2:07:32will get the path of this file.
2:07:35I'll simply say copy URI
2:07:37copy S3 URI.
2:07:38Why do we have this button if it is
2:07:40grayed out?
2:07:42Bro, S3 AWS what are you doing?
2:07:46So this is the folder. Cool. So I want
2:07:49to run the loop over here and this is my
2:07:53you can say df. So now what I will do? I
2:07:55will literally have this particular I
2:07:58and just to show you first of all we
2:08:01need to see what do we have it here. So
2:08:03I'll simply say print I
2:08:05just to see what exact thing exact keys
2:08:07that we have within this loop within
2:08:08this list. So we have the entire path.
2:08:11See?
2:08:12So that means I can use I dot path if I
2:08:15just write I dot path
2:08:18to see the data frames.
2:08:19Right? So what I will do? I will simply
2:08:21write the code here and here I will
2:08:24write I dot path. Now it will happen it
2:08:27will load the data frame one by one of
2:08:29each file. Wow.
2:08:31But I do not just want to load the data
2:08:33frame. I also want to create the delta
2:08:35table. So I will say
2:08:37df dot
2:08:39write dot format, right? And here I can
2:08:42do?
2:08:43I can say I want to just save in the
2:08:45data format.
2:08:47And here I want to perform option. Um,
2:08:51no. I just want to perform directly mode
2:08:53and I want to perform let say
2:08:55append.
2:08:56Append.
2:08:57I can perform append. I can even perform
2:08:59overwrite, but let say append.
2:09:01Cool. Then I will simply say dot save as
2:09:03table.
2:09:05What?
2:09:07[laughter]
2:09:08Save as table and then
2:09:10I will simply say demo dot and bronze
2:09:14dot. Now here is a tricky thing.
2:09:17Now if you just look at this thing, I
2:09:19have I have
2:09:21this table name. Now obviously I cannot
2:09:23pick the um you can say static name for
2:09:27all the tables because it will literally
2:09:29create the bug because I cannot have the
2:09:32table name same for all the tables. So
2:09:34how I can just dynamically fetch the
2:09:36names out of it? So I can do a lot of
2:09:38things here. So the best thing I think
2:09:40is we can just use um
2:09:43employees dot csv. Do we have the file
2:09:46name here? So I would have to use I
2:09:48think regex.
2:09:49So
2:09:50um yeah. So I think employees dot csv.
2:09:56So let me just see if we get the file
2:09:58name separately as well because if we
2:10:00have the file name
2:10:02then I can literally ignore regex,
2:10:04otherwise I have to use it. So we have
2:10:06file info, we have file path. Yes, we
2:10:08have name. So we have the name because I
2:10:10was thinking like we get a lot of
2:10:11metadata. So I can use here i dot name
2:10:15dot
2:10:17i dot name. But here is one thing. I
2:10:19cannot directly use i dot name because
2:10:21in the name I have employees dot csv as
2:10:23well. So here I would need to remove
2:10:26this dot csv from my i dot name because
2:10:29I do not want to include dot csv. So
2:10:31what I can do here I can literally
2:10:33perform this thing.
2:10:36So let say
2:10:38table name
2:10:40equals i dot name. Okay? And then can
2:10:44say table name
2:10:46equals
2:10:48table name. Or let's say first of all
2:10:50file name because this is the
2:10:52file name
2:10:53just to make it like more readable. And
2:10:55then table name
2:10:57equals file name
2:11:00dot split and I want to split it on dot
2:11:04and then I just want to fetch the zeroth
2:11:06index which is the very first index.
2:11:07And then I can write here
2:11:10table name. I hope it makes sense.
2:11:13Nothing fancy, just basics. And now if I
2:11:16just run this particular code, now what
2:11:18it should do it should create all of the
2:11:20tables dynamically
2:11:22in the loop. I do not need to hard code
2:11:24that particular same code again and
2:11:26again.
2:11:28This will just do my work. Okay. I hope
2:11:30so if there are no little little small
2:11:32small bugs. And let's wait. Let's wait
2:11:35for it to complete and let's see our all
2:11:37the bronze tables if we have it ready.
2:11:39And if we have it ready then boom.
2:11:45I think four five. Okay. So now I think
2:11:48all of the tasks are completed. Cool. So
2:11:50this says it is completed. So now if I
2:11:54just
2:11:55run this thing.
2:11:57So I should have used override but it is
2:12:00fine. It is fine. It is fine. It is
2:12:02fine. So if I just show you my tables,
2:12:04let me just expand this thing. So I have
2:12:06this bronze schema.
2:12:07And within that I have these tables. See
2:12:09employees, order items, orders,
2:12:10products. All of these tables are
2:12:12created using this particular modular
2:12:14code. I hope it makes sense. Simple.
2:12:16And I can even query anything. Let's say
2:12:18select asterisk from
2:12:21demo. bronze. orders.
2:12:25Cool. So this should just display data
2:12:27because it is in the delta format.
2:12:29And all of these tables are not CSV
2:12:31tables. These are the delta tables.
2:12:34Cool. And yes, we can directly create
2:12:37the table using dot save as table
2:12:39method. We do not even need to write
2:12:40create table table name. No. When we are
2:12:43using PySpark we just use this method.
2:12:45Cool. So this is done. My bronze
2:12:48notebook is completed and I cannot and I
2:12:51should not perform any kind of
2:12:53transformation. Now I will simply go to
2:12:54my new notebook.
2:12:56And this note will be silver. So let's
2:12:58say oh not a file. I don't want to
2:13:01create a new file because I'm not
2:13:03writing any py code.
2:13:05You can even create py file as well.
2:13:07Same thing but
2:13:09notebooks are very good for PySpark
2:13:11developers.
2:13:12So let's say
2:13:13six silver.
2:13:15Cool. So this is my
2:13:17notebook, silver notebook. So what I
2:13:19need to do now? I need to perform the
2:13:21transformation on the independent table.
2:13:24So what we do? We always try to perform
2:13:26some transformation. And here as I just
2:13:28mentioned, we are not mastering PySpark.
2:13:30So we will not be performing a lot of
2:13:31PySpark transformations, right? We will
2:13:33simply simulate that we are transforming
2:13:35our data and that's it.
2:13:36So first of all, I will
2:13:38read DF orders and then I'll simply say
2:13:41spark dot read dot table because this
2:13:44time we have the delta table.
2:13:46I will simply say three level namespace
2:13:48first of all. Demo dot
2:13:50bronze dot
2:13:53orders.
2:13:54Cool. And then I can say display
2:13:56DF orders.
2:14:01Cool. Let's wait.
2:14:03And this should display the orders.
2:14:09Perfect.
2:14:12So this particular DF is ready. Cool.
2:14:14Now if you just want to
2:14:16perform anything, any transformation,
2:14:18you can just perform. And because we are
2:14:20simulating this thing, so you can just
2:14:21imagine that we have performed a lot of
2:14:24transformations according to our use
2:14:26case.
2:14:27Simple.
2:14:28Very good.
2:14:29Now what I will do? I will literally
2:14:32perform this thing.
2:14:34I will literally load all of the other
2:14:36tables as well. So I'll simply say DF
2:14:39order items
2:14:42equals let say
2:14:44yeah perfect and then let's load
2:14:47DF products.
2:14:50We have products, we do not have
2:14:51customers and all.
2:14:53I think we have one more table which is
2:14:56employees. Cool.
2:14:58So let's say employees.
2:15:03So we have all of the tables loaded.
2:15:06And let me just check one of these.
2:15:08Let's say ordered
2:15:10items.
2:15:13So we have just loaded all of the data
2:15:15frames. Order items is not defined. Wow.
2:15:18TF order items.
2:15:23So cool. So I have all of this
2:15:25information available. Now what I will
2:15:26do, as I just mentioned, you can just
2:15:28imagine that we have performed a lot of
2:15:30transformations. So here I will
2:15:32literally at the end, after performing
2:15:34all the transformations because this
2:15:35data is already cleaned, now I will
2:15:37literally join these tables. Okay? So
2:15:39how I can just join these tables? So
2:15:40let's say DF join.
2:15:42And I can write the code to join all of
2:15:45these tables. Let's say DF orders
2:15:49dot join then order items. And I want to
2:15:52join it on the order ID.
2:15:54Then let's say order ID and then I want
2:15:57to perform inner join, but ideally I
2:15:59should just perform left join, but it's
2:16:00fine.
2:16:01Cool. So that's how you can just perform
2:16:03join. And if you just see
2:16:05it is DF join, you will
2:16:08see the table.
2:16:11And now let's wait. So this is the join
2:16:13data set where we have order and order
2:16:15items.
2:16:15So similarly you can just join this
2:16:18particular Do we have order items?
2:16:22DF order items.
2:16:26I think Yeah, yeah. We have all of
2:16:28things. So this is the way we can just
2:16:30perform all of the joins. But now let me
2:16:32just write the code to join all of the
2:16:35data frames together. And here you can
2:16:36even write the SQL query if you want to.
2:16:38So it is fine. Okay?
2:16:40So this is the code that I have just
2:16:41used to join all of my data frames.
2:16:44Nothing special, very fundamentals of
2:16:46PySpark where we literally join the
2:16:49like the same thing I have just written
2:16:51multiple times and I literally delegated
2:16:53this task to Jenny and Jenny just wrote
2:16:55this particular code for me. Same
2:16:57boilerplate stuff because you do not
2:16:58need to write the boilerplate code again
2:17:01and again repeatedly, right? Especially
2:17:03in the AI era. So this is the code. Let
2:17:05me just run it for you just to show you
2:17:07again the results. Obviously I know like
2:17:09this is working fine but still.
2:17:11So this is the end data frame which is
2:17:13cleaned, which is joined and this is now
2:17:15serving as a kind of one big white table
2:17:19that we just use in the silver layer.
2:17:21Cool, very good. So this is all about
2:17:23the silver layer that we have just
2:17:24created. Now let's create the final
2:17:27notebook which is our gold notebook,
2:17:29right? So let's name it as
2:17:32seven gold. And here what we do in the
2:17:35gold notebook, we perform dimensional
2:17:37modeling, we create fact tables,
2:17:39dimension tables, we create aggregated
2:17:40tables. It depends like what
2:17:42stakeholders want. So let's imagine that
2:17:44they want some aggregated views or
2:17:46aggregated tables that we can just
2:17:47create in the gold layer. Okay?
2:17:49So first of all
2:17:51you need to
2:17:52read data frame.
2:17:55So spark dot read dot
2:17:57format or let's say table.
2:18:00Because we have a table. Did we create
2:18:02the table? I think no because it
2:18:03literally joined the data frame. So
2:18:05let me just write the table quickly.
2:18:08So DF join
2:18:10dot write dot format.
2:18:13Format is let's say delta.
2:18:15And then mode overwrite. Yeah, let's use
2:18:18overwrite then dot save
2:18:20as
2:18:22table.
2:18:24And table will be let's say demo silver
2:18:26orders.
2:18:27Orders OBT.
2:18:30Cool. Nice strong this time.
2:18:32And I'm just using overwrite that makes
2:18:34sense. Demo dot silver cannot be found.
2:18:36Yes, because we didn't create the
2:18:38schema. So let me just quickly create
2:18:40the schema here.
2:18:42Create
2:18:43schema if
2:18:45not exists.
2:18:48It's demo dot silver.
2:18:50Cool.
2:18:52That's okay. It's done. So now let me
2:18:54just run this particular set.
2:18:58Let's wait, let's wait, let's wait.
2:19:02Hm cool so this is done and I'm also
2:19:03planning to mark here as overwrite
2:19:07instead of append
2:19:08because we might run this thing multiple
2:19:11times. So let me just run it one more
2:19:12time.
2:19:14And this is
2:19:16why why it is append?
2:19:18I just changed it to overwrite.
2:19:22Oh
2:19:23no worries at all. I can simply write
2:19:25overwrite here.
2:19:30Cool and I can just confirm this thing
2:19:32if this is
2:19:35yeah
2:19:36select asterisks from
2:19:39demo.bronze.
2:19:42orders or I can write count here
2:19:47just to confirm the mess that we have
2:19:48created.
2:19:50Okay 10,000 very good. So if I just run
2:19:52this thing again and if I just again
2:19:54count the things I should see 10,000 if
2:19:56everything is fine otherwise something
2:19:57is wrong.
2:19:59So so so
2:20:02cool
2:20:06yes perfect everything is fine. I can
2:20:07see 10,000. So that means our table are
2:20:09good. Tables are good because we might
2:20:11run it multiple times via a job so it is
2:20:13fine. So this is done this is created.
2:20:15Now let me just try it
2:20:17silver um
2:20:20demo
2:20:22dot
2:20:25silver
2:20:27then orders OBT. Cool.
2:20:31So this is my data frame and let me just
2:20:33quickly create create schema if not
2:20:37exists
2:20:40demo.
2:20:42gold.
2:20:44So this is display
2:20:47DF
2:20:48and let's see what do we have in the
2:20:50data frame.
2:20:52And let's try to create one aggregated
2:20:54view and the very simple aggregated view
2:20:55that you can just expect from any
2:20:58stakeholder is so let's take a real
2:21:00world requirement that you as a
2:21:02developer or you as a data engineer
2:21:04would receive most probably. Let's say
2:21:07the store or store owner, store manager
2:21:10or let's say any analytics manager,
2:21:11anyone who is just trying to figure out
2:21:13some KPIs to help the store grow because
2:21:16there can be like multiple stores for
2:21:18that particular store chain. As you can
2:21:20see we also have like multiple stores,
2:21:21store ID, blah blah blah.
2:21:23So let's say
2:21:24manager wants to see which store is
2:21:26performing good. First of all on the
2:21:28basis of number of orders, second is on
2:21:30the basis of number of customers. So
2:21:32that
2:21:34manager or analytics manager or anyone
2:21:36who is performing the analytics can
2:21:37better determine what things do they
2:21:39need. Do they need more staff because
2:21:42the orders are more or do they need some
2:21:44more you can say people on the floor
2:21:46because there are more number of
2:21:47customers so that they can help or they
2:21:48can just assist the customers better,
2:21:50right? So let's try to build this
2:21:52particular kind of view. So how you can
2:21:54just create that or let's say aggregated
2:21:55table as well. It's not a big deal.
2:21:57First of all let's write the code for
2:21:59that. So let's say
2:22:02DF join, not join, let's say DF
2:22:04aggregated.
2:22:06Cool and then I will simply say DF dot
2:22:09group by and I want to group by on store
2:22:12ID again fundamental supply spark.
2:22:15And after that
2:22:17I will simply say dot agg and then let's
2:22:19use count, count on order ID.
2:22:24Cool and then I will simply agg as as
2:22:26agg
2:22:27as this as total orders, cool.
2:22:30And let's import the functions quickly
2:22:32from PySpark dot SQL dot functions
2:22:35import as F.
2:22:38Cool. So we have loaded all the
2:22:40functions and let's also perform one
2:22:41more you can say aggregation which is
2:22:45let's go on the next line. Let's say
2:22:48count of customer ID
2:22:52total dot total customers
2:22:54and let's use here comma. Cool.
2:22:58I think it looks good so let's see what
2:23:00do we have displayed df.
2:23:04Uh, let see our aggregated version. So
2:23:06that looks good. Just to make it
2:23:09look like better, let's me just sort
2:23:12this particular data. Let's say df.
2:23:15Sort and then let's sort on order ID,
2:23:18store ID.
2:23:20Cool.
2:23:21So perfect. We have all of these things.
2:23:23We have total orders.
2:23:25Why we are seeing the same numbers? Why
2:23:28why why? So each store has these many
2:23:31total orders or these many customer IDs.
2:23:34Why why why? Oh, I see.
2:23:36Because obviously we need to apply the
2:23:38distinct as well because here we have
2:23:41the customer ID as just the count and we
2:23:43need the distinct count. But order is
2:23:45fine because I think order ID is the you
2:23:47can say primary key, but still we can
2:23:49just use the distinct key word.
2:23:52So let's use count
2:23:55distinct.
2:23:57Perfect here as well. Count distinct.
2:24:01Perfect.
2:24:03Now let's wait. Let's wait. Let's wait.
2:24:05Let wait and now now it is good. I was
2:24:08like why we have like exact total number
2:24:10of orders as exact total number of
2:24:11customers. So this is the aggregated
2:24:13version of our data frame. That means we
2:24:15can literally see the performance of
2:24:17each store directly within our view.
2:24:19Perfect basically the data frame.
2:24:21Now it's our choice if we want to create
2:24:22the view, we can even create the view.
2:24:24If we want to create the tables, we can
2:24:25even create the tables.
2:24:27So now let's finally try to create the
2:24:28gold table, aggregated table for us. So
2:24:31I'll simply say df.
2:24:34or df agg. write.
2:24:36format
2:24:39and let's
2:24:39write the write it in the delta format.
2:24:43Then let's say dot mode mode let's say
2:24:46overwrite.
2:24:47Perfect.
2:24:49And we do not need to overwrite schema
2:24:50because schema will not evolve so it's
2:24:52fine. So let's say
2:24:54dot save as table.
2:24:57And let's say it as demo. gold.
2:25:01dot
2:25:02orders 80. Cool. So this is my thought
2:25:05data frame, which is just the aggregated
2:25:07portion of it. Simple.
2:25:08So we can even see it if you want to.
2:25:11Let's say select a from demo . gold .
2:25:17orders
2:25:19So this is our final data which will be
2:25:22used to
2:25:23which will be used for the data
2:25:24reporting, analytics, anything, right?
2:25:27And we have all the information for each
2:25:28store.
2:25:29Now, if anyone wants to analyze this
2:25:31data, now let's talk about this thing as
2:25:33well. This is also the part of
2:25:34Databricks concept, on the fly concept.
2:25:36So if let's say you just want to allow
2:25:39some managers to use this notebook, do
2:25:41not worry, they will not edit your
2:25:42notebook, but they can do what? They can
2:25:44just use the visualizations which are
2:25:46created within the notebook itself. How
2:25:48we can just do that? Let me show you.
2:25:50Click on this plus button. Here you will
2:25:52see visualization.
2:25:53Here you can use the
2:25:55use like literally anything, like if you
2:25:58just want to use the
2:26:00data which is there, which is let's say
2:26:01for
2:26:03these particular stores. So if you just
2:26:04want to pick the store for like one and
2:26:06then pick the total number of orders,
2:26:09then total number of customers,
2:26:10anything. Okay. So this is your source
2:26:12of data.
2:26:13Then you can pick any particular, let's
2:26:15say bar chart, pie chart, histogram,
2:26:18anything. It depends like what exactly
2:26:19you want to do. Let's say bar chart.
2:26:22You can just pick the column. Let say I
2:26:23just want to pick X as store ID. Then Y
2:26:27columns is let's say
2:26:29total number of orders. So this way I
2:26:31will see okay, store number one as
2:26:33403 then 366 and like this. And then I
2:26:37can also pick some count, same thing
2:26:39because you already have the data.
2:26:41So you can just pick the count, okay,
2:26:42like sum.
2:26:44So this is the way you can just build
2:26:46the visualization directly within the
2:26:47notebook and you can even save it.
2:26:49Oh wow. Yes, you can even save it. You
2:26:51can even edit these things. You can even
2:26:52change the color, blah blah blah.
2:26:54Everything is here. So you can just pick
2:26:56it. Let say you just want this color,
2:26:57you can just pick this color. If you
2:26:59want to pick let's say pie chart, this
2:27:00is the pie chart according to the store
2:27:02IDs. Obviously we do not pick pie chart
2:27:04if we let say more than 10 values. That
2:27:06that is like kind of standard rule, but
2:27:09yeah, you have the options.
2:27:10So you can even pick let say histogram,
2:27:12you can even pick let say heat map, you
2:27:15can even pick let say
2:27:16bubble,
2:27:17which is not a very good thing, but
2:27:19yeah, you have all of these things let
2:27:20say cohort, everything.
2:27:22Cool? But the very simple and the very
2:27:24basic one is your
2:27:27bar chart. So this way you can even
2:27:28perform or build the visualization
2:27:30directly within the notebook as well.
2:27:31And you can even save it. Just click on
2:27:33save, it will be appearing here and
2:27:34that's it because by default you just
2:27:36get the table.
2:27:38Cool? This is another feature. So this
2:27:40is all about your medallion
2:27:41architecture. Now your medallion
2:27:43architecture, medallion notebooks are
2:27:45there. Now we need to understand the
2:27:47next concept to completely a kind of you
2:27:50can say run this particular thing in
2:27:51production. Okay? Let's see. And for
2:27:53that we need to understand lake flow
2:27:55jobs. And this is yes the one of the
2:27:58most important areas for medallion
2:28:00architecture. This is not the part of
2:28:02medallion architecture first of all, but
2:28:05this actually completes the medallion
2:28:07architecture. Why? Because have you ever
2:28:10heard the word called or term called
2:28:13data orchestration?
2:28:16If yes, then data
2:28:18orchestration in Databricks
2:28:20is achieved via lake flow jobs. Oh, as
2:28:23the name suggest, this is responsible to
2:28:25build the data jobs. And we always build
2:28:28jobs for the orchestration.
2:28:30What is orchestration? When you want to
2:28:33run the specific components of your um
2:28:36you can say
2:28:38medallion architecture, anything of your
2:28:40code, of your solution in a specific
2:28:42order. Makes sense?
2:28:44So data orchestration is specifically
2:28:46done via lake flow jobs. Those
2:28:48components, those things, those um you
2:28:52can say order is decided by lake flow
2:28:53jobs. Lake flow job is a you can say new
2:28:56feature relatively.
2:28:58It was not there before. So what we were
2:29:00using before when we didn't have the
2:29:02lake flow jobs, we were using
2:29:04the data factory
2:29:07via Azure if you are if you were using
2:29:10Databricks via Azure which was or which
2:29:13is the most popular way okay like using
2:29:16Databricks with Azure. So this was like
2:29:18one solution. The other solution was
2:29:22obviously the airflow
2:29:25Apache airflow where you was triggering
2:29:27the jobs manually via Apache airflow
2:29:31orchestrating the notebooks directly via
2:29:33the Apache airflow. So these were like
2:29:35two very very popular frameworks
2:29:37solutions available for data
2:29:39orchestration for our Databricks
2:29:40notebooks. But then Databricks created
2:29:43its own lake flow jobs because obviously
2:29:46Databricks is a complete AI and data
2:29:48solution. So obviously they do not want
2:29:50the users to go on a different platform
2:29:52just to perform the orchestration. So
2:29:54they built this orchestration solution
2:29:57within the Databricks.
2:29:58That makes sense. That makes sense.
2:30:00So this is nothing but just the
2:30:02orchestration okay like how you run the
2:30:03pieces, how you schedule the pieces, how
2:30:06you even let say monitor the jobs
2:30:08because obviously if you want to monitor
2:30:10the things like how the things are
2:30:11failing, how the things are going, how
2:30:13you are getting the alerts during the
2:30:15midnight. So all of these things are
2:30:17directly available now in Databricks
2:30:20directly.
2:30:21Let me just show you quickly within my
2:30:23Databricks as well.
2:30:25So let's go to home and not even home
2:30:28let's say
2:30:30click on the menu button click on home
2:30:32from here.
2:30:33So here
2:30:34if you do not see this option jobs and
2:30:35pipelines click on plus new you will
2:30:37just find it here okay so do not worry.
2:30:39So let's say jobs and pipelines.
2:30:41So this is the area that we can just see
2:30:44and I already have one job so do not
2:30:46need to worry. So you can just ignore
2:30:48this thing. So this is the area where we
2:30:50create jobs and pipelines. Now let me
2:30:53just tell you jobs are different,
2:30:55pipelines are different. We will just
2:30:56talk about pipelines very very soon but
2:30:59currently we are just talking about jobs
2:31:01lake flow jobs.
2:31:02So lake flow jobs means when you are
2:31:05orchestrating the things right. So here
2:31:07if you just want to create a job you can
2:31:09click on jobs here, or you can just
2:31:10click here as well, or you can even
2:31:12click on create and then job. There are
2:31:13multiple options. Let me just click
2:31:15here, job.
2:31:17So you will see this kind of interface.
2:31:19Nothing fancy. This is very very simple.
2:31:21The first thing is you first need to
2:31:23rename your job. Let say
2:31:26I am just renaming it as medallion
2:31:29job. This is my job name.
2:31:31Then it asks us to pick the tasks
2:31:35because obviously within the job
2:31:38we need to pick the task,
2:31:40right? We need to orchestrate the pieces
2:31:42and in our the pieces are notebooks, but
2:31:44it can be Python files, it can be SQL
2:31:46scripts, it can be anything, but in our
2:31:48case it is just notebooks and in most of
2:31:49the cases it will be just notebooks,
2:31:50okay?
2:31:51So let me just add it and you can
2:31:53directly click here as well, but let me
2:31:54just show you the long list. We have all
2:31:56of these tasks available. We have DBT
2:31:58tasks available natively in with
2:32:00directly in the Databricks because
2:32:01obviously DBT is well integrated with
2:32:03Databricks nowadays. So they have just
2:32:05created this thing as well and this
2:32:08Databricks Lakeflow jobs is not a simple
2:32:10orchestration. You can even run the
2:32:12control flow, if else condition, for
2:32:14loops, everything,
2:32:16everything. This is like very well
2:32:18written for Databricks, very well
2:32:20written. So you can just perform all of
2:32:22these things and yes, AI things are also
2:32:24possible now, document parsing,
2:32:25classification.
2:32:27By the way, classification is like very
2:32:29similar to your JEF model that you can
2:32:31just perform, okay?
2:32:33So all of these things are available
2:32:35directly within the um Lakeflow jobs,
2:32:37okay? So job is just like orchestration,
2:32:39similar to Apache Airflow. Very straight
2:32:42forward one liner definition,
2:32:44alternative of Apache Airflow, simple,
2:32:46simple.
2:32:47So let me just click on the notebook
2:32:49and here I can just name it. Let say I
2:32:52want to name it as bronze
2:32:54and here I can pick the notebook as
2:32:56type, source is this one, path. Here I
2:32:58need to just pick the um let me just
2:33:01bring it. So it is I think in the
2:33:05recents, no. Let me just go here.
2:33:09Where is my folder?
2:33:10Where is my folder?
2:33:12Um my folder is I think in the repos
2:33:15recents.
2:33:16Yeah, let me just open it from recents.
2:33:18So this is the notebook that I just want
2:33:20to pick. So it is here, users and this
2:33:24email and the demo and bronze. So this
2:33:26is my path of this notebook and I want
2:33:28to use the compute as serverless compute
2:33:30because it will automatically scale up
2:33:31as per the load. Then environment and
2:33:34libraries. If I want to add any specific
2:33:36environment to my data bricks, I can do
2:33:38that. But I do not have any specific
2:33:40environment, so that is fine. And if I
2:33:42want to add the libraries, I do not have
2:33:43any libraries, so that is also good. And
2:33:46then if you have some environment
2:33:47variables, no, I do not have. So you can
2:33:49simply ignore. If you have some
2:33:50parameters, remember the widgets part,
2:33:52you can just override it here as well,
2:33:54but we do not have it. So simply say
2:33:55create task.
2:33:57Now this task is completed, which is
2:33:58bronze. Now I want to add another task.
2:34:00Let's say I want to run another
2:34:03notebook, which is
2:34:04silver.
2:34:05And what is the path of this one? So the
2:34:07path of this one is let's say this
2:34:09silver.
2:34:10And everything else is same. Simply say
2:34:12create task. And after that, I want to
2:34:14create another notebook task, which is
2:34:16for gold layer.
2:34:20Simple. And then let me just pick the
2:34:22notebook.
2:34:23The notebook is there.
2:34:25Cool.
2:34:26So this job is created. Now what will
2:34:28happen? I can literally
2:34:31click on save.
2:34:33Um, I think it is automatically saved,
2:34:34so it is fine. I can even say run now if
2:34:36I want to run. I can even schedule it.
2:34:38Let's say if I want to run this
2:34:40particular job every day at 8:00 a.m.,
2:34:429:00 a.m. or let's say 1:00 a.m., 2:00
2:34:44a.m. just to disturb the developers. So
2:34:47I can even schedule this particular
2:34:50job
2:34:51anytime. Okay? I can even provide the
2:34:53job parameters, which will be just just
2:34:56just overridden for this particular job.
2:34:58I can even swap the compute, but I do
2:35:00not want to. It is fine. And some health
2:35:02checks are also there and this is good.
2:35:04Cool. And some permissions, obviously
2:35:05I'm just running it as an owner, but if
2:35:07I want to just run it via service
2:35:08principal, I can also do that. Not a big
2:35:10deal. Cool. So I can even go to jobs and
2:35:13pipelines now and if I just refresh, I
2:35:15should see that job created. See,
2:35:18medallion job is here. Currently, we do
2:35:19not have any kind of runs. So, if you
2:35:21just want to run it, you can just click
2:35:22on this particular run now button and
2:35:24this will start running my job. You can
2:35:25click on view run from here just to see
2:35:28how it performs as as you can see it is
2:35:30just now running the first task. Once
2:35:32the first task is completed, then only
2:35:34it will move to the second task. Once
2:35:35the second task is completed, then it
2:35:37will move to the third task.
2:35:39This is called orchestration.
2:35:41This is called dependency between the
2:35:43task within the orchestration. Simple.
2:35:47Okay? With the right example, with the
2:35:49right demo.
2:35:50Straightforward, simple. So now let's
2:35:52wait for it to complete and once it is
2:35:54done, I will just show you the results.
2:35:56So I can see that this job is completed
2:35:58and our task bronze, silver, gold all
2:36:01are completed and it took around 1
2:36:03minute 25 seconds and that is good.
2:36:06I can also see it in the form of
2:36:07timeline if you just want to compare.
2:36:10I also have the option of list as well,
2:36:11but graph is the best.
2:36:13This is everything about this particular
2:36:15job and you can even see all of the runs
2:36:17that you will have in your case as well
2:36:19and again the task as well if you just
2:36:21want to edit this then. You can edit
2:36:23this job anytime. Just click on these
2:36:25three dots and edit as YAML if you have
2:36:27the YAML file, otherwise you can even
2:36:29keep on adding the tasks here as well.
2:36:31You have both the options.
2:36:33Cool.
2:36:34Very good. So this is all about jobs and
2:36:36pipelines. Now yes, jobs is not
2:36:39something which is very very complex,
2:36:41but when you are just building the
2:36:43complex dependency structure for your
2:36:45orchestration, then yes, you need to
2:36:47master jobs in detail and the only way
2:36:50to master it by building more and more
2:36:52jobs. Simple. Simple. Very
2:36:54straightforward.
2:36:55Build more and more. Just think about
2:36:57some things like okay, let's have this
2:36:58orchestration,
2:37:00build some projects and then just build
2:37:02this thing. This is all about the lake
2:37:04flow jobs available in the Databricks.
2:37:07I hope it makes sense. Very good.
2:37:10So now let's talk about another concept
2:37:12which is becoming the most important
2:37:14concept in Databricks and I personally
2:37:15love it a It's called lake
2:37:19pipelines.
2:37:21A lot of people can be confused between
2:37:24pipeline and a job, but both are totally
2:37:27different concepts, totally different
2:37:29concepts. Okay?
2:37:31Yes, there are some you can say
2:37:34siding part between both, but both are
2:37:37totally different. First of all, let's
2:37:38try to understand what is lake flow
2:37:39pipeline. Then I will compare it with
2:37:41the job as well. Okay?
2:37:43So first of all, lake flow pipeline
2:37:45was
2:37:47earlier known as DLT, Delta Live Tables.
2:37:52If you have worked with DLT, you would
2:37:54know that this is the very popular
2:37:55framework back then as well. And now
2:37:56with LDP, they have just improved a lot
2:37:58of things and they have just changed a
2:38:00lot of code syntaxes as well. Breaking
2:38:02changes, yes, breaking changes.
2:38:04So yeah, this is all about the history
2:38:06of it. So now it is referred to as LDB,
2:38:10lake flow declarative pipelines, but
2:38:11usually we just call it as lake flow
2:38:13pipelines and it it it works very well
2:38:16with the jargons.
2:38:17So lake flow pipelines is created by
2:38:20Databricks.
2:38:22Okay? Directly,
2:38:23directly.
2:38:25Listen to me. Okay? It is created by
2:38:27Databricks.
2:38:28Then
2:38:29in Apache Spark 4.1, I guess, or 4.0. I
2:38:34think 4.1.
2:38:35They contributed
2:38:37this particular thing to open source
2:38:40Apache Spark as well. And this became
2:38:43even bigger thing because they
2:38:45contributed it to the open source Apache
2:38:47Spark.
2:38:49That means lake flow pipelines is now
2:38:52not like just integrated within the
2:38:55Databricks itself. It can be or it is
2:38:57now integrated across so many different
2:39:00different platforms who are managing
2:39:02data who are managing Apache Spark.
2:39:05But in Apache Spark, it is not called as
2:39:07lake flow pipelines or lake flow
2:39:09declarative pipelines. It is called as
2:39:11SDP,
2:39:13Spark declarative pipelines because it
2:39:15is on Spark side. When it is on the lake
2:39:18flow side on Databricks side, it is
2:39:20called lake flow declarative pipelines.
2:39:21Okay, remember these two terms.
2:39:23Very good. So now, what is a lake flow
2:39:26pipeline? What is a lake flow pipeline?
2:39:29So when lake flow pipeline was not
2:39:31there, we developers
2:39:35were writing all of the code by
2:39:38ourselves.
2:39:39And we were writing
2:39:41imperative code.
2:39:43What is this imperative code?
2:39:45Imperative code means you have to just
2:39:47take care of
2:39:49each step, small step, big step, read,
2:39:52write, transformation, everything. That
2:39:54is called imperative coding or
2:39:56imperative style of building pipelines.
2:39:58Cool? Or basically code any solution.
2:40:01But after this particular lake flow
2:40:03pipelines, we got something called as
2:40:08declarative
2:40:10declarative approach. What is
2:40:11declarative approach?
2:40:13Declarative approach says you just need
2:40:16to declare
2:40:18what you want
2:40:19without worrying
2:40:22how it will be achieved.
2:40:24You need to just focus on this
2:40:27what.
2:40:28That's it.
2:40:29Do not focus on
2:40:31how.
2:40:33This is the simplest definition for
2:40:35declarative approach.
2:40:37Here you just define
2:40:39how your end table or end data frame
2:40:43data set should look like. These are
2:40:46transformations you need and that is it.
2:40:48Everything will be managed by
2:40:51LDP or basically declarative pipeline.
2:40:54Makes sense?
2:40:55Not just this, like just the
2:40:57transformation,
2:40:58it also takes care of the
2:41:01execution order as well.
2:41:03Dependencies. If you want to let say run
2:41:06one piece of code, let's say this is a
2:41:09task number one.
2:41:11Okay, let me just draw it. Let's say
2:41:13this is a task number one.
2:41:15And you want to run another task after
2:41:17this.
2:41:18Let's say you want to run this task.
2:41:21And then you want run
2:41:23another task.
2:41:25Right?
2:41:26You do not even need to define this
2:41:28dependency. Okay? You
2:41:31Let's say you have some parallel task as
2:41:33well which do not have any kind of
2:41:34dependency. That will run in parallel
2:41:36automatically.
2:41:37Wow.
2:41:38So everything will be taken care by
2:41:42your LDP. Everything.
2:41:46Not just this. Not just this.
2:41:49With the course of time, Data Bricks
2:41:51added some amazing amazing components,
2:41:54amazing amazing packages within the LDP.
2:41:57Now
2:41:58let me just give you a very quick
2:41:59example.
2:42:01There's a concept if you know in data
2:42:02engineering, it's called SCD
2:42:05type two.
2:42:07Slowly changing dimension type two.
2:42:09Where we build the dimensions of type
2:42:11two. And it takes at least at least
2:42:14I would say 80 to 100 lines of code to
2:42:18apply or to implement the logic for SCD
2:42:21in PySpark. At least 80 to 100 lines of
2:42:23code. Easily.
2:42:26With LDP,
2:42:27we can literally use something called as
2:42:29Auto CDC. And that's it.
2:42:32Auto CDC is basically a decorator which
2:42:34is created for us. We just need to
2:42:36define that we want SCD type two and
2:42:38some parameters and that's it.
2:42:39That will automatically handle
2:42:41everything.
2:42:43This will automatically handle
2:42:44incremental injection as well.
2:42:47Wow. You do not even need to write the
2:42:48logic for incremental data loading. So
2:42:50they keep on keep on adding these kinds
2:42:53of features so that you developers can
2:42:55easily build all of these things.
2:42:59Now one more thing.
2:43:00In this AI era,
2:43:03LDP is even more popular. Why?
2:43:06Because just think about it. Let's say
2:43:08you are integrating AI LLM within your
2:43:11solution and you're using the LLM to
2:43:13generate the code.
2:43:15Now LLM will generate hundreds of lines
2:43:18of code for traditional PySpark coding.
2:43:20And now same model, same LLM will
2:43:23generate almost almost let say 20 to 30%
2:43:26of the code code because obviously the
2:43:29number of lines of code in lake flow
2:43:30pipelines is like very very less.
2:43:32So you are directly saving 50 to 70% of
2:43:36the tokens cost as well for your
2:43:38organization. Just think about it.
2:43:41So that is why it is even more popular
2:43:43in this AI era lake flow declarative
2:43:44pipelines because it generates less um
2:43:48codes less tokens.
2:43:50Make sense?
2:43:52This is an amazing framework which is
2:43:54added and now I can say that since it is
2:43:56added
2:43:58all of the organizations who are using
2:44:00Databricks are directly or indirectly
2:44:03integrating this particular framework in
2:44:05their solutions.
2:44:06If they are building something new, they
2:44:08are trying it, they are testing it, they
2:44:09are
2:44:10um relying on it. If they already had
2:44:13some solution, they are just comparing
2:44:14it with this particular thing.
2:44:16Obviously they need to just test a lot
2:44:17of things, they need to just um look at
2:44:19some benchmarks, everything, but yeah,
2:44:22organizations have started touching this
2:44:24particular framework.
2:44:25Make sense?
2:44:26So this is all about it and the best way
2:44:29to learn LDP is just to build some lake
2:44:31flow declarative pipelines and let me
2:44:33just tell you I also have a dedicated
2:44:34video just on the lake flow pipelines or
2:44:37I think Databricks pipelines. You can
2:44:39just search it. I can just show you if I
2:44:41I can just find it. Let me just bring
2:44:42the
2:44:44Let me just write Databricks um
2:44:47pipelines on Slamba.
2:44:50Um I just created this video. Where's
2:44:52that?
2:44:53Do we have um this browser by the way is
2:44:56like very very good. That's why I am
2:44:58just seeing this particular thing. But
2:44:59if you just search this thing on
2:45:00YouTube, you you will just find the
2:45:01video. I'm just so sure about it. So you
2:45:04can just learn about lake flow pipelines
2:45:06in detail if you want to because it is
2:45:08that particular skill that
2:45:10is not attained by everyone right now,
2:45:13all of the data engineers as well. But
2:45:15if you have, then obviously it is a kind
2:45:17of outlier that you can just add in your
2:45:18resume and it will help you a lot a lot.
2:45:22Okay.
2:45:22Let's try to create a very simple
2:45:24pipeline in Databricks so that you can
2:45:26also learn how it works and it's very
2:45:27very easy. So let me just open my data
2:45:31bricks. This is my data bricks. So let's
2:45:33try to see it. Where is my home button?
2:45:36Perfect. So how you can just create the
2:45:37pipeline? First of all, click here jobs
2:45:39and pipelines and then click on ETL
2:45:40pipeline or click on create and then ETL
2:45:42pipeline. Both are same thing. Let me
2:45:45just click here. And yes, one more
2:45:46thing, very small thing. The major
2:45:48advantage of this particular lake flow
2:45:50pipeline is you can either use SQL or
2:45:52Python. You have both the options. Both
2:45:54the options. If you are more into SQL,
2:45:57go with SQL. If you are more into
2:45:58PySpark, then Python is the way for you
2:46:01because in Python you literally write
2:46:02PySpark code. Okay.
2:46:04So let me just click here and let's try
2:46:06to see how it works. So this will open a
2:46:08completely different interface because
2:46:10when you want to build the lake flow
2:46:13declarative pipeline, you build the
2:46:14entire package. Okay. So yes, it is a
2:46:17kind of different thing which is a new
2:46:19thing, but do not worry. I am here with
2:46:21you. Do not worry at all. Let me just
2:46:23give you a very quick overview of this
2:46:25entire thing.
2:46:26And I love this particular interface to
2:46:29be honest because they have recently
2:46:30added this thing. Earlier we used to
2:46:32write all of the lake flow pipelines
2:46:34code in the notebook when the DLT era
2:46:36was there. But then they introduced lake
2:46:38flow and then they also introduced this
2:46:40thing and this is amazing. This is
2:46:42amazing.
2:46:43And yeah, so first of all, you have to
2:46:45rename this pipeline. Okay. Let me just
2:46:47do it for you. Let's say
2:46:49um first LDP. Cool.
2:46:53Let's say rename.
2:46:54So this pipeline is renamed and perfect.
2:46:57First LDP is here. Now the thing is by
2:46:59default this pipeline will be created in
2:47:01the root folder. Okay. Now you will say
2:47:04what's the big deal? I can easily move
2:47:06the pipeline directly from the home
2:47:07button. You should never do it. Why?
2:47:10Because see
2:47:11whenever you just change the um you can
2:47:13say location of your pipeline, it will
2:47:15fail because it works on the YAML, like
2:47:19where you have stored the pipeline.
2:47:20Everything is written here. So you
2:47:22cannot touch directly these things from
2:47:24the root location. You have to do
2:47:25everything from here. So let me show you
2:47:27how you do it. Just click on these three
2:47:28dots. If you just want to move this
2:47:30particular pipeline, click on move. And
2:47:32this is also written in their official
2:47:34documentation that you should always
2:47:36move the stuff from this interface.
2:47:39And then you can just go to demo because
2:47:40see, by default it created at the root
2:47:43location. So I just want to move this
2:47:44inside this demo folder. So I'll click
2:47:46on this demo and then I will click on
2:47:48move.
2:47:49So now this particular pipeline is moved
2:47:51to my demo folder. That's how it works.
2:47:54Cool. Very good.
2:47:55Now that is all about the root location
2:47:58of this particular pipeline. Now
2:48:01whatever we need to create in this
2:48:03particular lake flow pipeline,
2:48:05we can write all of the things here in
2:48:08the transformations folder. See, by
2:48:10default we get this particular empty
2:48:13file which is called my transformation
2:48:14dot py, but I can just rename it or I
2:48:16can even delete it. Not a big deal. Let
2:48:18me just move to trash.
2:48:20So now you can just create your Python
2:48:24file, right? You can even create mix
2:48:28match of Python and SQL both. Let's say
2:48:30you just want to create some pieces in
2:48:31Python, some pieces in SQL, both works.
2:48:34Third thing, you can even
2:48:37segregate your code in multiple files.
2:48:39You do not need to write everything
2:48:40within one single file. No.
2:48:43All the files created, listen to me
2:48:45carefully, all the files created under
2:48:47this transformations folder will be
2:48:49considered as the components of your
2:48:52pipeline.
2:48:53So if you have 10 files, all of the 10
2:48:56files will contribute towards the same
2:48:58LDP, same pipeline. Remember? Very good.
2:49:02So now let me just create one Python
2:49:04file. Let me just click on test and it
2:49:06will just create this test particular
2:49:08file. So let me just save it as let's
2:49:11say
2:49:13um
2:49:15LDP code dot py.
2:49:19Makes sense? So let's wait. Let's wait.
2:49:21Let wait. Let wait. And this is my
2:49:23transformation folder.
2:49:25And where is this file created? Let me
2:49:26just create it inside this thing. Create
2:49:28file. Perfect. Not a Python
2:49:31I Python notebook. This is a Python
2:49:33file. Okay? So you have to create the
2:49:34Python file. You can even create the
2:49:36notebook as well, but we usually create
2:49:37the Python files.
2:49:39So simply say
2:49:40LDP code dot py.
2:49:42You can even create multiple multiple
2:49:44files like LDP code blah blah blah.
2:49:47Everything works. Okay. So let me just
2:49:49show you if you want to let say ingest
2:49:52or blah blah blah. So this is my ingest
2:49:53dot py. So now if you just want to
2:49:56ingest the data,
2:49:57you can have ingestion done from
2:50:00directly from the S3 bucket as well. You
2:50:01can use auto loader for that.
2:50:04You can even ingest the data from the
2:50:05delta tables. Everything works fine.
2:50:07It's not a big deal. But let's say we
2:50:09just want to ingest the data from the
2:50:11bronze table because we already have
2:50:12that bronze table within our delta lake
2:50:15within our catalog and we can just reuse
2:50:17that particular thing. We do not need to
2:50:19again create the new table or new
2:50:21source. Okay.
2:50:22So how you can just import this thing?
2:50:24There are multiple ways and let me just
2:50:25tell you the previous method still work.
2:50:27If you just want to import the DLT, it
2:50:29still works. Okay. It's not like it they
2:50:31have literally
2:50:34reduced the use case of this thing. No.
2:50:36But the modern way is this one from
2:50:38PySpark dot
2:50:40pipelines or I think from PySpark
2:50:44import
2:50:46pipelines as
2:50:48DP DP or LDP. It's up to you, but I just
2:50:51use DP because we used to import
2:50:54import DLT as DT or something. So it's
2:50:57fine. Let's say import pipelines as DP.
2:51:00Now there are some decorators. There are
2:51:02some functions that we can use within
2:51:04this particular
2:51:06package. So we have three major things
2:51:09within this. The very first is
2:51:12materialized views.
2:51:15Okay. The second thing we have is
2:51:17streaming tables.
2:51:20Streaming
2:51:23tables.
2:51:24Third thing we have is normal views.
2:51:29Cool.
2:51:30So if you just want to create the
2:51:34tables. Okay. Where you can just perform
2:51:36batch processing.
2:51:37You can simply go with mat views,
2:51:39materialized views. If you want to
2:51:41perform streaming, if you want to build
2:51:43the streaming tables, then you should go
2:51:44with the streaming tables. If you just
2:51:46want to create the views, then obviously
2:51:47you should just go with the views. So if
2:51:49you just want to create the normal batch
2:51:51processing, do not go with streaming
2:51:52tables, okay? Simply go with the mat
2:51:54views. Mat views behave like you can say
2:51:56incremental data injection
2:51:58for your batch tables.
2:52:00Makes sense?
2:52:01Very good. So how you can just use it?
2:52:03There are decorators. Let's say I'll
2:52:04simply say dp dot um you can even write
2:52:08dp dot um streaming, something like
2:52:10that. So it's not like you have to have
2:52:12to um write this dp here. Let me just
2:52:15tell you I'm just writing dp here just
2:52:17because I am importing this package as
2:52:20dp. If you are importing this package as
2:52:22let say sdp, you have to write sdp. Just
2:52:25make sure this thing. This is like a
2:52:26very small thing but
2:52:27okay, like this. So it's up to you how
2:52:29you are importing this thing.
2:52:31I hope it makes sense. Now if let's say
2:52:33I'm just I I just want to create the
2:52:34materialized view. I'll simply say dp
2:52:36dot materialized
2:52:39materialized view.
2:52:41I'm not getting any recommendation but
2:52:42it's fine. So this is the thing, at the
2:52:44rate dp dot materialized view.
2:52:46Then I need to define my function.
2:52:49What function? Function will have all of
2:52:52my code. For example, let's say I want
2:52:54to create the table as LDP
2:52:57um bronze.
2:53:00Let's say LDP bronze.
2:53:02So here in the LDP bronze I will
2:53:04literally write
2:53:06this thing and then I can just read the
2:53:07table normally spark dot read dot table
2:53:12and the table name will be let say demo
2:53:15dot
2:53:16bronze
2:53:19dot orders.
2:53:20Cool, this is my table name
2:53:22and I will simply say return here. What
2:53:23it will do? It will literally take my
2:53:25transformations here. I'm not performing
2:53:27any transformations. I'm literally
2:53:29reading this table and I'm returning
2:53:31this dataframe. Whatever thing I return
2:53:34will become my dataset. Oh really? Yeah,
2:53:36will become my dataset, simple.
2:53:38Whatever thing I'm returning, it will
2:53:39become my table.
2:53:41So if I just write let say DF something
2:53:43something, it will become something
2:53:44something. So
2:53:45it is my returned DF. So this DF will
2:53:47become my data frame.
2:53:49I hope it makes sense.
2:53:50Now
2:53:52you need to understand how materialized
2:53:53view works because if you understood
2:53:55this thing, your entire concept
2:53:58of you can say LDP will be crystal
2:54:01clear. Why? Because you will be able to
2:54:02understand how things are working.
2:54:04And if you do not understand this thing,
2:54:05you will feel like this thing is working
2:54:08in a black box. No, we need to open this
2:54:10box. Let's try to open this box. So
2:54:12first of all,
2:54:13what is a materialized view? View is at
2:54:16the end of the day is just a view. And
2:54:19view is what? Let's say this is my view.
2:54:21View is just holding the query and
2:54:24that's it, right? What is the query
2:54:26here?
2:54:27Query is this one, like whatever things
2:54:29I'm writing here.
2:54:30I'm reading the table.
2:54:32If I want to perform some
2:54:33transformations, that will become the
2:54:34part of query as well, but this is my
2:54:36query for now, right? So this is the
2:54:38query that my view is referring.
2:54:41Now materialized view is different from
2:54:44a traditional view in this manner
2:54:46because a materialized view
2:54:49let say this is materialized view.
2:54:51Materialized view will cache the result
2:54:54as well in its memory.
2:54:57Oh, cache the results.
2:55:01Oh.
2:55:03It will work on the query. It will run
2:55:05the query, but this time it will cache
2:55:07the result as well. And every time it
2:55:09will run the query, it will not
2:55:11reprocess the entire query again and
2:55:13again. It will only process the new
2:55:15data.
2:55:17Only process the new data. So how it
2:55:19does know how how it knows like um it is
2:55:23the new data and it is not the new data?
2:55:24Because in the view we need to rerun the
2:55:26same query again and again, simple and
2:55:27sorted. But in the materialized view,
2:55:30okay, we just cache the data. But this
2:55:33time let's say I am just reading this
2:55:34table and all of the data orders rows
2:55:37will come here.
2:55:39Perfect. 10,000 rows will come here.
2:55:41If I rerun this particular pipeline,
2:55:44it will not process any record if I do
2:55:46not have any new data. Let's imagine we
2:55:48do not have any new data.
2:55:49Then it will only process zero records.
2:55:51Why? Because there is no new data and it
2:55:54will keep on using the same cached
2:55:56results which will be much faster than
2:55:58rerunning the entire query.
2:56:00If I have let's say 10 new records,
2:56:02let's say 10 new records, it will not
2:56:05process all the rows again. It will only
2:56:07process the 10 records. Why? Because it
2:56:10maintains that state
2:56:12internally. You do not have access to
2:56:14it. Do not worry. I will just show you
2:56:15the documentation as well where
2:56:17everything is written here because when
2:56:18you have that particular confidence
2:56:20whatever I'm saying, I know you trust
2:56:21me. I know but I still want to show you
2:56:23the documentation
2:56:25because it is a good practice to refer
2:56:27it whenever you want to, especially for
2:56:28the new things where you do not have
2:56:30much sources available. So when you have
2:56:32less sources available, you should rely
2:56:33on the official documentation and that's
2:56:35it. Okay? I know you rely on me. I know
2:56:37you trust me. I know I know but still
2:56:39I just want you to have a good practice
2:56:41and that's it.
2:56:43Cool. So that's how it works and let me
2:56:44just bring and let me just try to search
2:56:46if I just find it because they keep on
2:56:47updating the page.
2:56:50Materialized views. Yeah, I think I got
2:56:52it. They updated it on October 15, 2026.
2:56:54Very good. Very good. Keep on updating
2:56:56every day
2:56:57because we will just keep on coming and
2:56:58by the way, they they didn't update
2:56:59every single page. They just mention
2:57:01like what area was updated. So first of
2:57:03all, this is the official documentation,
2:57:05okay, for LDP and especially for the
2:57:07materialized views. First of all, let's
2:57:08try to read it what is written here.
2:57:10They are saying
2:57:12um like standard views materialized
2:57:14views are the results of a query that
2:57:15you access them the same way you would
2:57:17like a table. Unlike standard
2:57:19Yeah, I think this is the same thing
2:57:20which is written here. So that is fine.
2:57:21Now let me just show you the main thing.
2:57:23How they compute this thing.
2:57:26First of all, it is written that a
2:57:27materialized view is a declarative
2:57:30pipeline object. We know that. It
2:57:31includes a query. We know this as well.
2:57:34We have a flow. Flow is nothing but just
2:57:36the pipeline that we are creating and
2:57:38the cached results for fast access.
2:57:41A materialized view tracks changes in
2:57:43the upstream data. Upstream data means a
2:57:45source data. It will keep on tracking
2:57:47the changes.
2:57:49And whenever we trigger our pipeline, it
2:57:50will incrementally process the changed
2:57:53data, not the entire data.
2:57:56good.
2:57:57now
2:57:59more thing important written here in the
2:58:00virtual lock condition, they are saying
2:58:02they are often incremental. Anshul
2:58:04Lamba, what do you mean?
2:58:06You said that they are always processing
2:58:08the new data. Yes.
2:58:10So whenever they want to compute the
2:58:11data, the data will be incremental for
2:58:14sure, but the way it is processed is
2:58:16different or can be different.
2:58:17Because Databricks is managing the how
2:58:20part. So let's say if you want to
2:58:22incrementally ingest the data. Okay? You
2:58:24want to incrementally ingest the data
2:58:26and your table has, let's say, only 1000
2:58:28records. Okay? Only 1000 records. And
2:58:30you got only 10 new records.
2:58:32Now obviously it makes sense to
2:58:35reprocess the entire table because the
2:58:37number of records are so less.
2:58:39And when you will be just performing the
2:58:40incremental data ingestion, it might
2:58:42take more compute. So it is just a
2:58:44balance. It is just a pick between these
2:58:45two approaches. But you do not need to
2:58:47worry about this thing.
2:58:48Makes sense? So that is why they are
2:58:50saying they are often incremental. Most
2:58:52of the times it will be incremental, but
2:58:54sometimes it can be full load as well.
2:58:56But your number of records will be in
2:58:58sync. But the processing nature will be
2:59:01different.
2:59:02Makes sense? Now the main thing,
2:59:05how they know that the data is changed?
2:59:07Again,
2:59:09it will track the source and all of the
2:59:11information of the source will be
2:59:13written in this hidden table
2:59:16which is created inside this catalog,
2:59:17system.information_schema.tables.
2:59:21Oh oh.
2:59:22And these are not visible in catalog
2:59:23explorer explorer, but it was visible
2:59:26earlier then
2:59:27when we were just working with Delta
2:59:29live tables, it was visible. And you can
2:59:31even see the Databricks internal catalog
2:59:33if you have access to this thing because
2:59:35it stores all of these things. See,
2:59:37Databricks uses Unity Catalog to store
2:59:38metadata. We know that. And including
2:59:41the query and additional system view for
2:59:42incremental updates, Databricks
2:59:44materializes the cached data in the
2:59:45cloud storage. Yes, because whenever we
2:59:47are just storing the cache, it needs to
2:59:49materialize it somewhere. So it keeps on
2:59:52maintaining everything within the this
2:59:54particular catalog, which is Databricks
2:59:55stores some backing data in Databricks
2:59:57internal catalog. You might have access
2:59:59to this particular catalog. You might
3:00:01not have access to this particular
3:00:02catalog depending upon your permissions
3:00:03and the Databricks version you are
3:00:05using.
3:00:07If you're using a your hosted one, you
3:00:08might see it. If you're using free
3:00:10edition, I am not sure. And I am not
3:00:12even sure if you see this and you can
3:00:14access it or not because you are using
3:00:15the free edition. So you can just check
3:00:17it. But
3:00:18you should not
3:00:19play with these catalogs because these
3:00:21are just maintaining your
3:00:23checkpoints. These are maintaining your
3:00:25last load information, everything. And
3:00:27do you know what? Earlier everyone was
3:00:29having the access to these particular
3:00:31catalogs, these particular tables. Then
3:00:32people started
3:00:34touching those tables and playing around
3:00:35it. And then they were just literally
3:00:38breaking the code and then they were
3:00:39just complaining Databricks, hey, my
3:00:41database is not synced, blah blah blah.
3:00:44Then they just made it like private.
3:00:45Okay. So this is the way I hope now you
3:00:47understood how materialized views are
3:00:50created and how materialized views are
3:00:52actually
3:00:53used. I hope it makes sense. Okay. So
3:00:55this is my materialized table. Very
3:00:57good. So let's say I have this
3:00:59materialized table and similarly I can
3:01:01just create the streaming table as well.
3:01:03Um streaming table is same thing that we
3:01:05have just um saw in the Spark structured
3:01:08streaming phase. So it is just like a
3:01:10table
3:01:11created on Spark structured streaming
3:01:12table and that's it. Simple, very
3:01:14simple.
3:01:15Materialized view is different, is new.
3:01:17That is why I'm just telling now talking
3:01:18about this thing in detail. Okay.
3:01:20Very simple, very good. And you do not
3:01:22use
3:01:23you do not use the streaming table for
3:01:25all of the sources. You can only use the
3:01:28streaming tables for the streaming
3:01:29sources like Kafka, like Event Hub, like
3:01:32Delta streaming tables, all of these
3:01:34things. Okay. But we are just focusing
3:01:36on the materialized views here. And if
3:01:37you just want to learn
3:01:39like LDP in detail, obviously I have
3:01:41just already mentioned the source for
3:01:44for this particular uh topic. Okay.
3:01:46So now let's try to complete this
3:01:49particular LDP. Let's say I just want to
3:01:50create another materialized view and
3:01:52this time I will be just creating the
3:01:54silver task and I will read
3:01:57from where? I will read it from the from
3:01:59this table, LDP bronze. And the good
3:02:01thing is I do not even need to refer it
3:02:04like this demo.silver.orders. I can
3:02:06literally say
3:02:07LDP bronze.
3:02:09Now how it will know that this LDP
3:02:11bronze is already created here? This is
3:02:13called dependency.
3:02:15This is called dependency in the LDP. It
3:02:17will automatically fetch that do I have
3:02:20this table created here or not? Because
3:02:22whatever I am writing here
3:02:24it will automatically
3:02:26populate the fields called catalog and
3:02:28schema directly from here. Whatever I am
3:02:30just mentioning here. I can click here
3:02:32on the workspace. Workspace is my demo
3:02:35which is the catalog. Schema I will
3:02:37create a new schema for it and I will
3:02:38simply say LDP. Okay? And let's say
3:02:40create. So every time I am just writing
3:02:43LDP bronze here, it will automatically
3:02:46automatically
3:02:48it will automatically change this
3:02:51particular
3:02:52thing to this catalog and then this
3:02:54particular schema. Okay?
3:02:56And I saw one thing file or notebook not
3:02:58found. Source code is this one. Let me
3:02:59just see configure paths
3:03:02and
3:03:03let's see the bronze. Let's see demo. We
3:03:07have first LDP. We have transformations.
3:03:08Let's say select.
3:03:10Perfect.
3:03:11So
3:03:12this is done. It was like saying like
3:03:15why we do not have that particular LDP.
3:03:17So now I have just simply picked it. So
3:03:19it is fine.
3:03:20Cool cool cool. Very good. So what is
3:03:22this thing tests? So I think it just
3:03:24created this particular folder. Let me
3:03:25just move it to trash and we have files
3:03:28external to pipeline folder.
3:03:30And perfect I think yeah perfect. So
3:03:32this is my silver layer. This is my
3:03:34bronze layer and let's create the some
3:03:37transformations as well. Let's say I
3:03:38just want to add one column. So I simply
3:03:40say df equals df dot with column.
3:03:44And here I will create the column let
3:03:46say
3:03:47current
3:03:48timestamp. Okay?
3:03:50And here I can just pick, let say, um,
3:03:53get now? Not get now. Let's say current
3:03:56timestamp.
3:03:59Cool.
3:04:00Very good.
3:04:01So we can just use this thing.
3:04:04And I can simply say return DF. This is
3:04:06my silver table.
3:04:07Now if I just want to create the gold
3:04:09table,
3:04:11I simply say LDP silver and let's
3:04:13perform some aggregation. Can simply say
3:04:15order, price and total. Perfect. Do we
3:04:18have price? Um, I doubt. So let's use
3:04:20count.
3:04:22Or I will say
3:04:23order ID.
3:04:26And not order ID, let's say
3:04:28customers.
3:04:30Simple and let's perform this
3:04:31aggregation on store
3:04:33as we performed earlier. So let's say
3:04:35customer
3:04:37ID and then let's say total customers.
3:04:40I hope it makes sense. So this is our
3:04:41simple pipeline, three step pipeline
3:04:43that we are just creating for this
3:04:44particular task. Now
3:04:46in LDB, one of the biggest advantage
3:04:49that we get is um, you can say dry run.
3:04:52So if before even running the pipeline,
3:04:54we can even perform dry run. Dry run
3:04:55will literally test
3:04:57and will assess our code. If everything
3:05:00is fine, nothing is broken. If yes, it
3:05:02will tell us beforehand. And if
3:05:04everything is fine, it will give us the
3:05:06pipeline graph just to show that
3:05:08dependency.
3:05:09And I do not even need to perform any
3:05:11kind of dependency. It will
3:05:12automatically build the dependency graph
3:05:14on the basis of the dependency. Just
3:05:16look at it.
3:05:17This is the independent task. So it will
3:05:20create it at first. Then this is
3:05:22dependent on this particular one, as you
3:05:23can see.
3:05:24Then this is dependent on this
3:05:26particular one. So that's how it just
3:05:28keeps on creating this particular
3:05:29dependency graph.
3:05:30Now it is saying current timestamp is
3:05:31not defined. This is not depend. Yes,
3:05:33because we didn't import the packages.
3:05:35So let me just import it
3:05:37from PySpark. SQL. functions
3:05:41import as DF.
3:05:43Perfect.
3:05:44So now let's perform the
3:05:46dry run again. Let's see if it works or
3:05:48not.
3:05:51Let's wait for a bit.
3:05:53And yeah, you can also perform the tried
3:05:54on if you want to. I think this is good.
3:05:57I can click on pipeline graph just to
3:05:58see it. So
3:06:01perfect. I have this source delta, which
3:06:04is my source table, as you can see.
3:06:07Cool.
3:06:08And this is my materialized view, which
3:06:10is incremental, as you can see. Then I
3:06:12have this full recompute
3:06:14for LDP silver, and then I have
3:06:16materialized view incremental, as you
3:06:18can see.
3:06:19Wow. So this is the entire lineage
3:06:21automatically created using my LDP. I
3:06:23didn't touch anything. I literally
3:06:25didn't touch anything.
3:06:27That's how it works.
3:06:29You can even click on this I button if
3:06:30you just want to.
3:06:32And if you can just read it, you can
3:06:33just do it like this. So let's do one
3:06:35thing. Let's try to now run this
3:06:37particular pipeline and click on run
3:06:39pipeline. This will run this pipeline
3:06:41within this canvas.
3:06:43And you will see all of the number of
3:06:44records processed, all of the things
3:06:47processed, everything, all of the
3:06:49information will be here within this
3:06:50particular canvas. So let's wait, and I
3:06:52think this is running this particular
3:06:53thing. So let's click on pipeline graph
3:06:56just to see it. So I can see that output
3:06:59records are 10,000. That means it
3:07:01ingested 10,000
3:07:03records,
3:07:04and it
3:07:05processed or transformed 10,000 records,
3:07:08and at the end I got 25 records because
3:07:10I only have the, you can say obviously,
3:07:1325 store IDs. Simple.
3:07:15Now the good thing is this is done. This
3:07:17is ingested. If I just go to, let's say
3:07:19my catalog, I can just duplicate this
3:07:22tab.
3:07:23So if I just duplicate this tab, let me
3:07:25just show you
3:07:26catalog.
3:07:28And I should see the schema LDP. Yes, I
3:07:30have. And all of these things are here.
3:07:32LDP bronze, LDP gold, LDP silver.
3:07:35All of the three tables are created with
3:07:36data.
3:07:37And I also have information schema, I
3:07:39told you. And within the information
3:07:40schema, we get all of these things, and
3:07:43we do not have access to it, as we just
3:07:44read. Simple. Very good.
3:07:47So you cannot see that catalog here that
3:07:49you just were seeing database internals.
3:07:51But if you are just using Azure hosted
3:07:53or hosted Databricks workspace, I think
3:07:55you might see that. Good.
3:07:57Very good.
3:07:58Now the good thing is, if I just go to
3:08:00pipelines, if I now click on run
3:08:02pipeline for one more time, do you know
3:08:03what will happen?
3:08:05Obviously it will run my pipeline, but
3:08:07this time it will not process or I can
3:08:10say it should not process any new
3:08:13records because there are no new
3:08:14records. That is why you are seeing no
3:08:17change.
3:08:18Output records is this one. Let me just
3:08:20refresh. No change, full recompute and
3:08:23then no change. Let's wait.
3:08:25Incremental.
3:08:28Let's wait. Let's wait.
3:08:31For it to complete. Yeah.
3:08:33Obviously it will show 25 just as the
3:08:36output records because we have the
3:08:37output as 20 10,000.
3:08:40But it says no change. Why no change?
3:08:43Why no change? You tell me.
3:08:45Because we do not have any kind of
3:08:48change on the source side.
3:08:51We do not have any kind of change.
3:08:53We just read the data from the orders
3:08:56from the bronze schema and we returned
3:08:57it.
3:08:58This is our bronze materialized view.
3:09:00This is called LDP bronze.
3:09:02And it has output records at 10,000 with
3:09:04with no change. We do not have any kind
3:09:06of change. And if we just go to
3:09:07performance, we might see this thing
3:09:09here as well. Rows read zero.
3:09:12Read zero.
3:09:14Zero. See, LDP bronze is zero.
3:09:17Oh, see.
3:09:19This read
3:09:2110,000 rows read and this read 20,000.
3:09:24This one.
3:09:26The very first task row like very first
3:09:28task read only zero rows. Why? Because
3:09:31source was not changed.
3:09:35Makes sense?
3:09:36You can even monitor this thing. If you
3:09:38just go to let's say here
3:09:40and if you just go to pipelines, you
3:09:41will see forced LDP.
3:09:43And this is an ETL pipeline. That is why
3:09:45you are seeing this ETL pipeline. If I
3:09:47click here,
3:09:48you will see a tab called runs and these
3:09:50are all the runs here.
3:09:52And all of the same information is here
3:09:54as well. And if you just want to edit
3:09:55it, you can click on pencil icon. You
3:09:57will land on that particular page where
3:09:59we are just doing all of this stuff.
3:10:01Make sense? But everything is
3:10:03visible here as well.
3:10:06Wow, everything.
3:10:08Make sense? Even if you just want to
3:10:09schedule this pipeline so that it can
3:10:11just run let's say every single day, you
3:10:14can just schedule this. Let's say run
3:10:15pipeline
3:10:17again just to run it and if you just
3:10:18want to schedule it you can just click
3:10:19on this particular button this one
3:10:21schedule and it will schedule it simple.
3:10:24Cool and because we just ran it let's
3:10:26see
3:10:27and we should see the exact same stuff
3:10:29because it is no changed then fully
3:10:31compute and then this will be like
3:10:33incremental
3:10:34because this is just an aggregation.
3:10:36Now why it just picked full recompute
3:10:38because I told you often times you will
3:10:40see is see it as incremental but
3:10:42sometimes if the processing is a
3:10:44processing compute is better with full
3:10:47full recompute instead of incremental it
3:10:48will simply go with recompute
3:10:50full recompute instead of incremental
3:10:52simple.
3:10:53Make sense? Very good.
3:10:55So this is all about LDB. Now
3:10:58what is the difference between this LDB
3:11:01and this particular job?
3:11:04This is a very good question.
3:11:05Now the thing is pipeline is built for a
3:11:08specific purpose
3:11:11specific you can say task.
3:11:13But
3:11:14Data Bricks job or basically
3:11:17any kind of orchestration job
3:11:19performs a lot of similar similar tasks
3:11:22bro
3:11:23a lot of tasks.
3:11:25For example let's say this is your one
3:11:27LDB.
3:11:28Okay?
3:11:30After this LDB or let say along with
3:11:32this LDB you should have one more LDB,
3:11:34you should have one more LDB and once
3:11:37all of these LDBs are completed you
3:11:39should run one more LDB
3:11:42but after completing all of these LDBs
3:11:45that means all of these pipelines
3:11:47and after running all of these pipelines
3:11:49you should run let's say
3:11:54three
3:11:55PySpark notebooks let's say
3:11:57one two three.
3:12:00Now you tell me
3:12:02can you handle this
3:12:04without a job? No. Here you have to use
3:12:08lake flow job where you can even
3:12:10orchestrate the LDP pipeline. Oh.
3:12:15You can even orchestrate this thing as
3:12:17well. So that is why if you just go to
3:12:19Databricks jobs
3:12:21if you just click on job and let's say
3:12:23if you have this medallion job and if
3:12:25you just want to go to tasks and if you
3:12:27want to add that lake flow job after
3:12:29this gold you can click on add task and
3:12:32you will have a task called ETL
3:12:34pipeline. You can even scroll down and
3:12:36you will find ETL pipeline. Here you can
3:12:38just say
3:12:39LDP and pipeline. You need to pick the
3:12:42pipeline.
3:12:43Pipeline is this one and that's it. Save
3:12:45task.
3:12:46Wow. This way you can even orchestrate
3:12:50the pipeline as well.
3:12:52So job is like the orchestration.
3:12:56Pipeline is achieving a task
3:13:00or you can better
3:13:01phrase it like
3:13:03pipeline is like you can say a kind of
3:13:06small job
3:13:08for a specific task.
3:13:10Just for the understanding, okay? Just
3:13:12for the understanding I'm just giving
3:13:13you this definition.
3:13:14It is a kind of flow
3:13:17for that particular task only.
3:13:20That is why
3:13:21they both are interconnected but both
3:13:24are not same.
3:13:25Job is for orchestration. Pipeline is
3:13:28the execution order for that particular
3:13:31task only.
3:13:33You should not build 100 things within
3:13:35one single pipeline. No.
3:13:37Why? It will be a nightmare to debug any
3:13:41pipeline if you see any error at 2:00
3:13:43a.m.
3:13:43Because most of the pipelines of the
3:13:47organizations run during the midnight.
3:13:49If you see any error who will fix that?
3:13:51Who will fix that? And how you will fix
3:13:53that? It's not a piece of cake. But when
3:13:55you have small small pipelines for
3:13:57dedicated tasks you can easily debug it.
3:14:00And in order to orchestrate those small
3:14:01small pieces you have the job.
3:14:04I hope now it makes sense. Now I hope
3:14:06all the dots are connected.
3:14:09With example, obviously because I wanted
3:14:11to just show you everything with
3:14:12examples.
3:14:14Make sense?
3:14:15good.
3:14:16This is all about lake flows, jobs, lake
3:14:18flow pipelines, and let me just tell you
3:14:22your 80% of the concepts are crystal
3:14:24clear, really yeah, because the the
3:14:26these are the
3:14:27major things that you need to
3:14:29understand.
3:14:30These are the major things.
3:14:33Cool. Let's see what topic we need to
3:14:35cover next.
3:14:36So now let's talk about this topic DAB.
3:14:41Data bricks asset bundles or declarative
3:14:43automation bundles because recently they
3:14:45just change the name and they just pick
3:14:47the name as declarative automation
3:14:49bundles. What is this thing? What is
3:14:51this thing?
3:14:52So this is nothing but a very useful
3:14:54package to deploy your data bricks
3:14:57packages.
3:14:59Data bricks modules, data bricks
3:15:01pipelines, data bricks, everything.
3:15:03Because whatever we are building so far,
3:15:05whatever we have built so far,
3:15:07pipelines, jobs, notebooks, everything,
3:15:10you are doing it within your workspace
3:15:12or you can say within your
3:15:14account
3:15:15workspace.
3:15:17So
3:15:18this is good,
3:15:19but this is just the one side of it. You
3:15:22also need to learn how to deploy it to
3:15:25multiple platforms.
3:15:27Let's say this is your data bricks dev,
3:15:29you need to deploy it to QA
3:15:33and then ultimately to
3:15:35prod as well.
3:15:36Oh, so in order to deploy this thing,
3:15:39you need a framework.
3:15:41There are multiple ways to be honest.
3:15:43You can use Terraform, you can use so
3:15:46many ways, right? So many ways, there
3:15:47are so many ways, Azure Bicep, so many
3:15:50ways. But data bricks created a package
3:15:52called asset bundles or automation
3:15:54bundles
3:15:55where they say that we will make your
3:15:57deployment much, much, much easier and
3:15:59very, very simple.
3:16:01Very simple.
3:16:03Okay.
3:16:04What is the simple thing? So what we
3:16:06Whatever you build here, we create a
3:16:09kind of package.
3:16:13Oh, we create a package
3:16:15and then we deploy this bundle, this
3:16:17package. We just called as bundle.
3:16:19That's why it is called as automation
3:16:21bundle or asset bundle. So, we create
3:16:23this package, okay? And we deploy it to
3:16:25QA
3:16:26and same package will ultimately go to
3:16:28production as well.
3:16:30Cool? This is the concept.
3:16:32But how to bundle it? How to bundle it?
3:16:35There are multiple ways. There are
3:16:36multiple ways. So, what we do? Let's say
3:16:38you have built notebooks, you have um
3:16:41created jobs, you have created
3:16:43pipelines.
3:16:44Everything that you have created,
3:16:46everything
3:16:48behind the scenes, behind the UI, you
3:16:51will see something called as YAML.
3:16:55YAML. And YAML, you should know that
3:16:58this is the standard for DevOps, okay?
3:17:01Every DevOps software, technology,
3:17:03framework uses YAML and not just DevOps.
3:17:06YAML is becoming standard for literally
3:17:07everything. Okay? Even Docker. Docker
3:17:10compose file is full of YAML. So, YAML
3:17:11is literally everywhere.
3:17:13And now in the world of AI,
3:17:16it is the first choice. Why? Because
3:17:17when you just write the JSON code or
3:17:19JSON syntax, you waste a lot of tokens
3:17:22by writing curly braces and blah blah
3:17:24double quotes.
3:17:25In YAML, we do not have curly braces, we
3:17:27do not have quotes. So, it saves a lot
3:17:28of tokens because it's all about tokens
3:17:30now. Literally, it's all about tokens.
3:17:33So, YAML is just a good way to replace
3:17:35JSON. Um it is used for configurations,
3:17:38it is used for um you can say defining
3:17:40hierarchies, key-value pairs, but
3:17:42without those curly braces and without
3:17:43those um double quotes. Simple? This is
3:17:46the best answer that I can just give
3:17:48what is YAML. Similar thing, but we we
3:17:50just use it like like it it we just
3:17:52write it like this. Let's say we just
3:17:53have a key, then key will have a colon,
3:17:55then let's say this is the value.
3:17:57Simple. No curly braces, no double
3:17:59quotes, nothing. Simple.
3:18:02Okay? So, this is YAML. So, whatever we
3:18:03are creating, jobs, pipelines, um
3:18:07notebooks, everything,
3:18:08under the hood,
3:18:09these are just YAML files. So what we
3:18:12do?
3:18:13We simply say that we will provide you
3:18:15one parent YAML file. Listen to me
3:18:17carefully. Parent YAML file. So this
3:18:19will be my parent YAML file. Let's say.
3:18:23And within this parent YAML file, I will
3:18:25have small small
3:18:28connecting YAML files.
3:18:30Let's say this YAML file will be for
3:18:32job.
3:18:33This YAML file will be for pipelines.
3:18:37And blah blah blah. So I will connect
3:18:39these these or I will include these
3:18:41these
3:18:43YAML
3:18:45to this parent YAML file. And I will
3:18:47create a package out of it.
3:18:49Okay?
3:18:51I will create a package, basically a
3:18:52bundle out of it. And that bundle is
3:18:54nothing but this one. And it is also
3:18:56called as dot bundle. When you just
3:18:58deploy it, it gets deployed with this
3:18:59particular extension, dot bundle.
3:19:01And this dot bundle will go to QA and
3:19:04will ultimately go to production. This
3:19:07is your automation bundle.
3:19:09Oh, yes. This is your automation bundle.
3:19:11Let me show you.
3:19:13If you just go to your Databricks, if I
3:19:15just click on let say pipelines, if I
3:19:16click on anything, let say job,
3:19:19you will see in the three dots, edit as
3:19:21YAML. So this is the YAML written for
3:19:24this particular job. Oh. So this UI is
3:19:27just for you.
3:19:28But for machine,
3:19:30for Databricks internal code, for
3:19:32backend, this is the thing they want.
3:19:34Simple.
3:19:36Simple.
3:19:37This is the YAML. So they have
3:19:39resources, they have jobs, job name,
3:19:41medallion task, each task, and where is
3:19:44the notebook for each thing, everything.
3:19:46So this is the job. This is just the UI.
3:19:49This is just the UI. But the resource is
3:19:51this one.
3:19:52So what we do? We bundle this job,
3:19:55everything, and then when we deploy this
3:19:58bundle to QA, this exact job will be
3:20:01deployed to QA. So when that dot bundle
3:20:04folder will be taken to the QA
3:20:07workspace, will have this job. Why?
3:20:09Because we have this particular
3:20:12job created
3:20:14under dot bundle.
3:20:15Oh, because Databricks understands this.
3:20:20Make sense? Very good. How we can just
3:20:22create this thing? Let me show you. So
3:20:24you can literally go to workspace,
3:20:26right? And you can let's say create a
3:20:29new folder. Let me just remove this
3:20:31forced LDB as well from here.
3:20:33Let's say move to trash. So if you just
3:20:35want to create this thing, you can
3:20:37create one new folder within this as
3:20:39well.
3:20:40So let's say I want to create bundle.
3:20:43Open a git folder. We do not want to
3:20:45create git folder. It's fine. So let's
3:20:46say folder.
3:20:48So we will just talk about git folder
3:20:49don't worry. It's very simple. So let
3:20:52say this is my
3:20:53dab folder. Let's say it's called as
3:20:55dab.
3:20:56Okay. And this is the folder I want to
3:20:58use for all of the YAML, all of the
3:20:59deployments, everything, right? So how
3:21:01you can just create this thing? You need
3:21:02to open one terminal, web terminal. Yes,
3:21:05we get the web terminal as well within
3:21:06the Databricks. In order to open it, you
3:21:08need to have at least one notebook
3:21:10because only then it will open it
3:21:11because it needs to use a compute. So
3:21:13just for that purpose, I'm just creating
3:21:14a demo notebook which is not linked to
3:21:16any kind of deployment, but just for
3:21:19using the compute. Let me just close all
3:21:21the
3:21:22unused files. Cool. So when we just get
3:21:25into this particular notebook, I will
3:21:27see one terminal here on the bottom
3:21:29side. Okay. Click on this logs and this
3:21:32terminal will be
3:21:33open and it will simply say requires
3:21:35cluster to be attached. So simply click
3:21:36on serverless compute and it will attach
3:21:38this compute.
3:21:40Now it will open the terminal.
3:21:42And this is my terminal. You can even
3:21:43click on this button to open it in the
3:21:46new window like this.
3:21:49So this is my terminal and now I can
3:21:51just literally write anything I want. So
3:21:53here I will first of all initialize the
3:21:55bundle.
3:21:56You can write
3:21:57Databricks bundle init. Hit enter. It
3:22:00will simply say installing the CLI
3:22:02because behind the scenes we first need
3:22:04to install the CLI. So in the web
3:22:06terminal, it will automatically install
3:22:07the CLI. Okay. Then it will simply say
3:22:09do you want default Python? Simply say
3:22:11yes. And unique name for the project,
3:22:13let's say
3:22:15dab
3:22:16project. Include a job that runs a
3:22:18notebook. Yeah, because this will just
3:22:19show you how the things works. Simply
3:22:21create a demo notebook, um demo job.
3:22:23Yes. Include ETL, yes.
3:22:26Include a sample Python package.
3:22:28We'll
3:22:29Yeah, just keep it. Fine.
3:22:31Use serverless compute, yes. Serverless
3:22:32compute and now default catalog for any
3:22:34table, literally say demo.
3:22:37Um yes.
3:22:40After answering these simple simple
3:22:42questions, your package will be
3:22:44initialized, your bundle will be
3:22:45initialized, and I think it has done the
3:22:46work.
3:22:47Let me just check and let me just open
3:22:50it.
3:22:51Demo dab project. Perfect. As I can see,
3:22:54everything is here. Let me just close
3:22:55this terminal. It is not required now.
3:22:57So within the dab project, like as you
3:22:59can as you can see, inside the dab I
3:23:01have this demo notebook which I created
3:23:02just for this particular purpose to open
3:23:05this terminal, but I can just close it
3:23:06now. I can even delete it. It's not
3:23:07required.
3:23:09Within that particular folder, it
3:23:10created this particular folder for me.
3:23:13Wow.
3:23:14Yes. It created this folder for me. And
3:23:16within this folder, I have some things.
3:23:18First of all, fixtures, ignore this
3:23:19thing. Resources, I will just talk about
3:23:21this thing. Source and tests. These are
3:23:24demo demo things.
3:23:25Okay. Ignore everything because the main
3:23:27thing here is databricks.yml file. This
3:23:30is the major parent YAML file that I
3:23:33just talked about. Okay. This is the
3:23:34parent YAML file. Okay.
3:23:37Within this parent YAML file, we get
3:23:38some things. First of all, the project
3:23:40name, which is this one, dab project. It
3:23:42says include resources.yml. Remember I
3:23:46told you that we attach the small small
3:23:48components YAML files to it. So it
3:23:50simply says this is my parent YAML and
3:23:53anything which is created inside the
3:23:55resources folder, like any YAML file,
3:23:57should also be the part of this parent
3:23:59YAML file. and let me just open this
3:24:01YAML file, like resources folder, you
3:24:03will see two YAML files. First of all,
3:24:05sample job and this is a sample ETL. If
3:24:07I open this particular file, you will
3:24:09see the
3:24:10sample job, sample job YAML. See?
3:24:14Oh, wow.
3:24:15Everything is here.
3:24:17And if I just open the ETL pipeline,
3:24:18this is the YAML for the ETL. Again,
3:24:20sample.
3:24:22Oh, so these two YAML file
3:24:24are now the part of this parent YAML
3:24:27file.
3:24:28Wow. And then we have some artifacts. If
3:24:30you just want to include some, you can
3:24:32say internal artifacts like Python wheel
3:24:34file. So it has already created that
3:24:35thing. Python wheel, type wheel, UV
3:24:36build wheel. That's it.
3:24:39But you can just ignore it. It's not a
3:24:40big deal. It is only
3:24:42it is only required when you have like
3:24:44some packages within your source folder
3:24:46and you just want to include it within
3:24:48your
3:24:49Python build if you just want to build
3:24:51it during the deployment. So it is for
3:24:53that particular purpose. But usually if
3:24:54you just want to use the wheel, you like
3:24:57it's my personal opinion. You can just
3:24:59put the wheel file directly instead of
3:25:00rebuilding it. But you can just do it as
3:25:02well. It's not a big deal. Okay.
3:25:04And then inside the source folder we
3:25:05have the project. We have this main
3:25:07taxes or TY sample thing, DAP project
3:25:10ETL, some project ETL. But now here's
3:25:12the thing.
3:25:14When you have
3:25:15first of all the
3:25:18ETL file like
3:25:20you can say job forget about ETL just
3:25:22the job.
3:25:23If you just talk about job, if I just go
3:25:25here in the resources and if I just go
3:25:27to sample job,
3:25:28you will see that we have sample
3:25:30notebook as well. Oh, because in the job
3:25:32we are orchestrating the components, but
3:25:34we also need to include the component as
3:25:36well within this.
3:25:38So it says
3:25:39here is the component. So this is the
3:25:41thing that you need to remember whenever
3:25:42you are deploying the stuff, whenever
3:25:44you are deploying the package, you need
3:25:45to include YAML. Yes, but you also need
3:25:48to include the dependent like dependent
3:25:49files.
3:25:50I know it is like very common thing, but
3:25:52I still want to tell you
3:25:54that you need to include notebooks,
3:25:57scripts, Python files, everything within
3:25:59the
3:26:00bundle. Now it is not required that you
3:26:03have to have to create this source
3:26:04folder, but it is a good practice just
3:26:06to put all of the modules within the
3:26:08source folder.
3:26:09Cool. Very good.
3:26:11Similar to ETL. Now whenever we just
3:26:13create the ETL, we put all of the files
3:26:15inside transformations folder I told
3:26:17you. So see it is the same thing here as
3:26:19well. If I open this thing inside the
3:26:21transformation, I have these two files.
3:26:24And if I just open the ETL YAML for it,
3:26:26you will find all of the things here.
3:26:28See, I have root path here and I want to
3:26:31include everything that I have inside
3:26:33the transformations folder.
3:26:36That means these are the YAML files.
3:26:38These are just the building block. This
3:26:40is just the blueprint, but the raw
3:26:42material is sitting inside the source
3:26:44folder.
3:26:46And that should be the part of the
3:26:47deployment package as well. I hope it
3:26:49makes sense.
3:26:51So that's how we just create the
3:26:52package. And if you just want to deploy
3:26:53it, we can simply write bundle deploy
3:26:55and that's it. It will simply create a
3:26:56dot bundle folder and it will be there
3:26:58in your workspace. That is obviously now
3:27:00some people say that hey, do data
3:27:03engineers need to know about these
3:27:04things? This is just a DevOps things.
3:27:06No.
3:27:07You need to know because see, there are
3:27:08two things.
3:27:09Um let me just tell you.
3:27:11First of all,
3:27:12data brick CICD
3:27:16for
3:27:17bundle.
3:27:19This part needs to be mastered by data
3:27:21engineers because you will be the one
3:27:24who know what is inside transformations
3:27:27and what should be there inside
3:27:28transformations folder. How you can just
3:27:30expect this thing from a DevOps engineer
3:27:32that they will know what is LDP? How how
3:27:35how how do they know?
3:27:37You need to master this thing. You need
3:27:39to understand how bundle works. Now then
3:27:42where DevOps team come in? Let me just
3:27:44tell you that as well. Once you created
3:27:46the dot bundle
3:27:48folder, okay, let's say once you just
3:27:51created this bundle, now this bundle
3:27:54needs to go to QA,
3:27:56needs to go to prod.
3:27:59And this bundle creation should also be
3:28:01automated because there can be like
3:28:02let's say developer one created this
3:28:04bundle, developer two created this
3:28:06bundle. So all the quality checks here,
3:28:08let's say some test cases should also
3:28:10run. So this part, this part, this part
3:28:14will be handled by DevOps team. They
3:28:16will just create the pipeline CICD
3:28:18pipeline in Azure DevOps, let's say in
3:28:20most of the cases in Azure DevOps. So
3:28:22they will just automate this thing. They
3:28:24will just write the job there, let's say
3:28:26trigger
3:28:28trigger this job on this particular
3:28:30thing and if anyone creates a pull
3:28:31request, simply create a bundle folder,
3:28:34then just take that dot bundle folder,
3:28:36deploy it to QA, run the test cases. If
3:28:39everything is fine, you can even include
3:28:41some senior folks in the pipeline and
3:28:43then approve it, then go to production.
3:28:45This entire automation will be handled
3:28:47by the DevOps team
3:28:49or in today's era, a senior data
3:28:51engineer can also handle this thing.
3:28:54But this is the DevOps part. This is not
3:28:56the DevOps part. This is not the DevOps
3:28:58part. This is the data engineering
3:28:59development part. This is not a DevOps
3:29:02part
3:29:03because you you literally like start
3:29:05shouting,
3:29:06CI CD is all about DevOps, why do we
3:29:08need to learn it? No.
3:29:09How can you expect this thing from a
3:29:11DevOps person to know about LDP,
3:29:15materialized views, Spark structured
3:29:17streaming, ETL? How how can you just
3:29:19expect this thing? No.
3:29:21You need to perform this thing.
3:29:24Make sense? Make sense? Very good.
3:29:26So this is all about asset bundles and I
3:29:28hope this will give you a very clear
3:29:29understanding of it. And yeah, this is
3:29:31all.
3:29:32Nothing fancy, nothing difficult as
3:29:35well. So this was all about DAPP.
3:29:37Now let's talk about the Git folders
3:29:40because we just talked about it. So
3:29:41let's say
3:29:43Git folder. So let's say Databricks
3:29:46Git folders.
3:29:47So what is Databricks Git folder? So
3:29:49let's say on the same Databricks
3:29:52workspace,
3:29:54multiple developers
3:29:56multiple developers are working. Let's
3:29:58say this developer is working, this
3:29:59developer is working, multiple
3:30:00developers are working, right?
3:30:02This decent developer is also working.
3:30:06So now these all are working, right? All
3:30:08can create conflict. All can just mess
3:30:11up with each other's development,
3:30:12obviously, because there's just one
3:30:13workspace. So the solution is simply
3:30:17involve
3:30:19one
3:30:20version control or basically a system
3:30:23which can isolate each other's
3:30:25development. So here we involve Git.
3:30:29Here you can just use GitHub, Azure
3:30:31DevOps, any repository if you want to,
3:30:33but software will be Git.
3:30:35So here you include Git and everyone
3:30:38creates their own branch using Git and
3:30:40this Data Bricks is also connected to
3:30:43this Git.
3:30:44So what will happen? So everything will
3:30:46be
3:30:48connected to this Git and this Git will
3:30:49be connected to Data Bricks. Now
3:30:51everyone is contributing to Data Bricks,
3:30:52yes, but via Git.
3:30:54Cool?
3:30:56Let me show you how you can just create
3:30:57it. It's very simple. Simply go to home
3:30:59and then go to workspace and then click
3:31:02on create or let's go to demo
3:31:04first of all and then click on create,
3:31:06click on Git folder. So this is the Git
3:31:07folder, but whenever you want to create
3:31:09a Git folder, you need to create the
3:31:10GitHub repository or Git repository.
3:31:13You can just connect it via GitHub, you
3:31:15can also connect it via Azure DevOps. It
3:31:17is not a big deal and in most of the
3:31:18scenarios you will get this thing done
3:31:22on your workspace by a senior folk, but
3:31:24if you know this thing, it's good.
3:31:26It's really good.
3:31:28I hope it makes sense. Okay, okay, okay,
3:31:30okay.
3:31:31So this is all about Git and this is a
3:31:33very fundamental thing. I hope that you
3:31:35all would already know, but
3:31:37like you would know about Git, but Data
3:31:38Bricks Git folder is a different thing,
3:31:40right? Like now you also know that as
3:31:41well.
3:31:42So this is all about Data Bricks Git
3:31:44folders.
3:31:46Makes sense?
3:31:47Now let's talk about Data Bricks SQL.
3:31:49Now what is this thing? Data Bricks SQL.
3:31:52Do we just need to write SQL? It is Data
3:31:53Bricks SQL? No. It is totally a
3:31:56different thing. Really?
3:31:59Now it is [clears throat] also called as
3:32:00SQL warehouse, Data Bricks SQL
3:32:01warehouse. But what is this thing? Data
3:32:02Bricks SQL, Data Bricks SQL warehouse.
3:32:05So basically whatever we have done being
3:32:08a data engineer
3:32:09in Data Bricks, we used all purpose
3:32:10compute, we used serverless SQL compute,
3:32:13we use all of these things, right?
3:32:15These computes are very good
3:32:18whenever you want to perform data
3:32:19engineering task, ingestion,
3:32:21transformation, storage, blah blah blah.
3:32:24But when we want our bricks
3:32:29to be connected
3:32:30to
3:32:31tools like Power BI, any reporting tool,
3:32:35we do not
3:32:37connect those tools via all purpose
3:32:40compute or the compute that we use for
3:32:42data engineering. No.
3:32:44We have a special type of compute called
3:32:45SQL warehouse.
3:32:47Oh, because these are the optimized
3:32:49compute to perform the SQL workloads or
3:32:51basically reporting workloads.
3:32:55This area comes under Databricks SQL.
3:32:58This is one thing. Second,
3:33:01obviously you have built amazing amazing
3:33:02things. Now, who will use them?
3:33:06Stakeholders. Not stakeholders only. You
3:33:08will also use that data. You also might
3:33:10need to query that data. You might also
3:33:12need to look at some you can say might
3:33:14numbers anything.
3:33:16So how will you just take care of all of
3:33:18these things? Let me just show you. So
3:33:19let me show you
3:33:21because this area has a dedicated space
3:33:23within the data bricks. So simply come
3:33:25to data bricks and then oh to SQL. So if
3:33:28it is like collapsed, you can just
3:33:29expand it or you can even go to plus
3:33:31new.
3:33:32So this is the SQL area where they have
3:33:34put all of the things. SQL warehouses,
3:33:36alerts, SQL editor as well because they
3:33:38have just created this integrate SQL
3:33:40editor where you can write the queries
3:33:42and you will get the results back.
3:33:44You can even store the queries and
3:33:47very popular option dashboards is also
3:33:49here. You can even create the
3:33:51dashboards. You do not even need to go
3:33:52to Power BI. And if you just take my
3:33:54advice,
3:33:55I think in the very very very
3:33:59not very, I would say I do not know the
3:34:02exact timing, but you will see a change
3:34:05that
3:34:06people might not use the traditional
3:34:08dashboards. Now people be using the
3:34:10integrated dashboard. That's why
3:34:13Databricks dashboards
3:34:14are becoming more popular.
3:34:16Dashboards within the AI chatbots are
3:34:19becoming more popular, but I think that
3:34:21will take a lot of time because not
3:34:23every organization would
3:34:24trust it blindly to just hand over all
3:34:26the data and then just ask the
3:34:27questions. But integrated dashboards
3:34:30like Databricks dashboards will become
3:34:31very popular.
3:34:33Power BI will obviously will be there
3:34:34because Power BI has been there since so
3:34:36long and organization has created like
3:34:38thousands of dashboards and it is not a
3:34:40piece of cake to migrate all of them. It
3:34:42takes years, okay? It takes years, but
3:34:45Power BI will be there, but you will see
3:34:46a rise of these kinds of dashboards as
3:34:48well. Okay?
3:34:49So first of all, let's click on the SQL
3:34:50warehouse. Let's see what do we have. So
3:34:52by default in the free edition, we get
3:34:54serverless starter warehouse and this is
3:34:56a type of compute that we use to connect
3:35:00to SQL workloads, to connect to
3:35:02reporting workloads, everything. Okay?
3:35:04You can even create one
3:35:06if you have the hosted version. If you
3:35:07are on free version, you can even they
3:35:09like you can just only use this one and
3:35:11that's it.
3:35:12Cool, very good. You can just turn it
3:35:14on. It is also serverless, so it will be
3:35:16just automatically scaling up and down
3:35:18depending upon the workloads and the
3:35:19size is 2x small, which is the cluster
3:35:21size. Okay? We have 2x small, x small,
3:35:24and then small, large, extra large,
3:35:26these kinds of sizes.
3:35:27Right? So this is my compute. Let's say
3:35:29if I want to look at my data, I can go
3:35:31to SQL editor.
3:35:33Cool. And let me just open the query
3:35:36and this is a new tab.
3:35:38Here you can just open the catalog on
3:35:40the left. Here you can let's say click
3:35:43on SQL query
3:35:44and you can just query anything the same
3:35:46way you query your SQL um database.
3:35:50Let's say select asterisks from
3:35:53orders. And I can just decide what
3:35:57I have within my workspace. So let's say
3:35:59demo and then let's say bronze. Okay?
3:36:02So if I just click on run all,
3:36:04it will simply run this session and this
3:36:06will be my
3:36:08just SQL workload area.
3:36:11No data injection, nothing I can perform
3:36:13because this is obviously a SQL workload
3:36:15area, but here we just perform some
3:36:17quick analysis, ad hoc analysis and all
3:36:18of these things.
3:36:19Cool.
3:36:20If you have written some special queries
3:36:23that you just want to save it, you can
3:36:24even save it using this particular
3:36:25button.
3:36:26And that is not everything.
3:36:29Databricks has recently created
3:36:31something called as parameterized
3:36:32queries. So let say you just want to
3:36:35perform parameterization as well within
3:36:37your query. You can perform let's say
3:36:39where um store ID
3:36:41okay
3:36:42equals
3:36:44colon let's say this um store ID.
3:36:47Okay? You can just add it like this as
3:36:49well or you can even use this UI because
3:36:52I just created it. You can just click
3:36:53here as well add parameter and parameter
3:36:56one can be anything and you can even
3:36:58rename it let's say
3:37:00param store.
3:37:02Either way you can directly code it
3:37:03because I remember this particular colon
3:37:05thing so yeah. So now what happened?
3:37:07This has become my query
3:37:10parameterized query. Wow.
3:37:12Wow. Literally wow. So I can just remove
3:37:14this thing parameter store. Now what
3:37:16will happen? I can just literally
3:37:17provide some value let's say 10. Now I
3:37:19can click on run all.
3:37:21Now this will just show me all of the
3:37:24things for just store ID equals to 10.
3:37:27Let's say I just want to see any other
3:37:29thing let's say 11. If I let's say click
3:37:31on run all, I will see only the store ID
3:37:33equals to 11. What is the advantage of
3:37:35it?
3:37:36Let's say you are a technical person
3:37:39and you have created a very long query
3:37:42let's say 50 lines of query.
3:37:45Now you ran it once or let's say your
3:37:47stakeholder ran it once. Cool. Now next
3:37:50day they want to check the same result
3:37:52but for a different store
3:37:55right? Now it is so risky to change the
3:37:58things directly within the query because
3:38:00this is a very simple query and in real
3:38:02world we write like sub queries, sub
3:38:04queries within sub query, sub queries
3:38:06within sub queries and you know that we
3:38:08use CTEs, we use sub queries so it
3:38:10becomes a mess when you have 50 to 60 to
3:38:1270 lines of queries right?
3:38:15So you cannot miss any single thing
3:38:17otherwise result will be wrong. But
3:38:19let's say you created one query, you
3:38:21tested it end to end and everything is
3:38:23working fine. Now what you can do you
3:38:25can create parameters within your
3:38:26queries and you can just change those
3:38:28parameters and you can just get the
3:38:30dynamic results every time whenever you
3:38:32are just running that query.
3:38:34It will help the technical people as
3:38:35well because you do not need to touch
3:38:37the query again and again. It will be a
3:38:39gold for non-technical people because
3:38:42they do not need to touch the query
3:38:44because obviously if they touch the
3:38:46query they will break it.
3:38:48So what you can do? You can literally
3:38:49save this query and you can just only
3:38:51allow the read access and they can only
3:38:53change the parameters and that's it.
3:38:55That way they will not break the code.
3:38:57They will only put something garbage in
3:38:59the parameters at the in in the worst
3:39:01case scenario and that's it. That will
3:39:02not do anything harmful to your code, to
3:39:05your resources, right?
3:39:08This way you can perform the like I
3:39:10really like love this feature like
3:39:11parameterization of queries.
3:39:13Simple
3:39:14because earlier people were just
3:39:16creating the functions, the procedures
3:39:18just for performing all of these things.
3:39:20But now you do not need to create that.
3:39:22Simply save the queries,
3:39:23parameterization and that's it.
3:39:25Cool.
3:39:27This is like very important and then
3:39:29whatever queries you save it will be
3:39:31saved under queries but I do not want to
3:39:32save it so I'm not saving it and that's
3:39:34it.
3:39:34And if you just want to create the
3:39:36alerts as well you can even create the
3:39:37alerts. Let's say
3:39:39you want to notify your manager if
3:39:42any customer has just purchased
3:39:44something more than xyz amount, right?
3:39:47So you can even create the alerts that
3:39:49way by clicking here alerts. So all of
3:39:52these things come under Databricks SQL.
3:39:53Databricks SQL is not just like I am
3:39:56using SQL in Databricks, it is
3:39:57Databricks SQL. I am using SQL in
3:39:59Databricks and I know how to use SQL. I
3:40:02am a Databricks developer because I am
3:40:04using SQL and Databricks also use SQL.
3:40:06What?
3:40:08What? What is this?
3:40:11Bro, yes, SQL is the base for Databricks
3:40:15as well but
3:40:18do you write this query in your SQL? No.
3:40:22Then it is different.
3:40:25Yes, SQL is a fundamental but who will
3:40:28learn the technology?
3:40:30Who?
3:40:31No, no, no, no. We know SQL, we know
3:40:33everything. We know the entire data
3:40:35engineering because we know SQL.
3:40:37Very good, very good, very good But do
3:40:39not worry, because if you are my data
3:40:41fam, it's my responsibility to tell you
3:40:43what is actually required to become a
3:40:46data engineer, what is actually required
3:40:49to become an AI data engineer or
3:40:51anything which is relevant into this
3:40:53domain. Make sense? Make sense?
3:40:56This is all about Databricks SQL. Let's
3:40:58jump onto the next topic. Now let's talk
3:41:00about Databricks Genie. And Databricks
3:41:03Genie is like so so so popular and it is
3:41:06actually very good. Let me just see if I
3:41:08have the logo for it.
3:41:11Databricks. No.
3:41:13It's fine.
3:41:14So yeah, Databricks Genie. What is
3:41:15Databricks Genie? Databricks Genie is
3:41:16nothing but the AI agent built by
3:41:19Databricks
3:41:20for Databricks,
3:41:22inside Databricks. Oh really? Yeah.
3:41:25All of these things.
3:41:26And this is not just another AI agent.
3:41:28It is actually agentic AI agent. What
3:41:31what is the difference? See,
3:41:33one AI agent can be the one
3:41:36which can answer your questions.
3:41:38That is an AI agent.
3:41:40But when that AI agent can also perform
3:41:43the tasks in the form of workflows, in
3:41:46the form of like multi-step tasks, then
3:41:48that agent is not just an AI agent. It
3:41:50is an agentic AI agent
3:41:52that can perform the tool calls,
3:41:53everything. Okay. I know at the end of
3:41:56the day it is still an agent, but it
3:41:57matters a lot because now it is able to
3:42:00perform the agentic things, agentic
3:42:01things.
3:42:03Hmm. That is a Databricks Genie.
3:42:05And Databricks Genie has actually
3:42:08boosted the
3:42:10efficiency,
3:42:11speed, productivity to the next level.
3:42:14Really? Yeah.
3:42:15And you also get Databricks Genie for
3:42:16free, like in the free edition. Let me
3:42:18just show you.
3:42:19Let me just open this particular thing.
3:42:22And if you just want to open Genie,
3:42:23there are like multiple areas. You can
3:42:25even go to dedicated notebook. You will
3:42:27find Genie there. Let me just show you
3:42:28first of all that thing. So if you just
3:42:30let say go to demo. Why do I have
3:42:32drafts? Oh, because I just I didn't save
3:42:34the queries. Let me just drop it.
3:42:36I can just drop this as well. Perfect.
3:42:38So if I just open this one and if have
3:42:40let say any notebook, let say
3:42:43Okay?
3:42:44And if I'm just performing any kind of
3:42:46data injection, data transformation,
3:42:49data loading, anything, and if I want
3:42:51any help, okay?
3:42:52If I want any help in the dedicated
3:42:54cell, I can open this Jenny right away.
3:42:57This is a kind of inline code help by
3:43:00Jenny.
3:43:01Oh, yes.
3:43:03If I do not want an inline code help, I
3:43:06want the help for the entire notebook.
3:43:09I might need to ask something else. Then
3:43:11I can click on this button here.
3:43:14This will open the interactive AI agent
3:43:17session, and it will take this notebook
3:43:20as a context.
3:43:21It doesn't mean that it cannot take
3:43:23context of any other notebook, but this
3:43:24will be the primary source of the
3:43:26context.
3:43:27Makes sense? So, let's say
3:43:29I want to take some help. Okay? Let me
3:43:31just click here. Jenny. So, this is the
3:43:34new chat for me.
3:43:36I can just ask anything. I can say let
3:43:38say um I want to
3:43:42fetch
3:43:44only or let say I want to
3:43:48This is my orders, right? Demo default
3:43:49orders. Yeah.
3:43:50I want to fetch
3:43:53orders
3:43:54only from store
3:43:58ID equals
3:43:5910. Let's say if I hit enter, just
3:44:02natural language and that's it.
3:44:04What it will do? It will first of all
3:44:06look at your current
3:44:07notebook, which is one query, right? It
3:44:10has said that I will add a SQL cell to
3:44:12fetch those orders and run it. And it
3:44:14has edited my query notebook, and it is
3:44:16simply saying connected to compute
3:44:17because it is verifying that
3:44:20is the query right or wrong that it has
3:44:22written. Okay? And I would also see one
3:44:25cell if it has created for me for now.
3:44:26Yeah. So, this is the cell created by
3:44:29Jenny. It is also running it for me just
3:44:31to confirm it is fine or not. Once it is
3:44:33there, then it will give me the final
3:44:35answer. So, let's wait.
3:44:37And I didn't write this query. Jenny has
3:44:39written it for me. Okay? I just used the
3:44:41natural language.
3:44:42This is very similar to having Claude
3:44:45code within your Databricks.
3:44:48Even better than that because Claude
3:44:50code doesn't have all of the internal
3:44:51knowledge of Databricks this has. All of
3:44:54the package information this has, right?
3:44:56So if you are good with all of the
3:44:58changes, you can accept it. If you are
3:44:59not happy, you can simply say reject
3:45:01all. Oh, yes.
3:45:03It's totally in your control. It's not
3:45:05like you ask something and now it is not
3:45:06in your control. If I want to accept it,
3:45:08simply say accept all and that's it.
3:45:10That is done. If you want to ask
3:45:11something more, you can just ask more.
3:45:13If you want to attach something else,
3:45:15you can click on plus button. You can
3:45:16just add some files, connectors,
3:45:18anything and even add some context.
3:45:20If you want to initialize another
3:45:22session, new chat, you can click here
3:45:24and all of the previous things will be
3:45:25gone because you do not need to keep
3:45:27everything. And if you just want to see
3:45:29the history, all of the chat history
3:45:31that you have so far within your
3:45:32workspace, everything will be here.
3:45:34I will be having so many, so that is why
3:45:36I'm not opening it.
3:45:37Obviously you need to just delegate all
3:45:39of the tasks to AI now.
3:45:41Makes sense? Cool. But you need to have
3:45:43the overlook. You need to have to you
3:45:45you you need to monitor what it is
3:45:47doing. You need to accept it, right? It
3:45:49will be you who will be just accepting
3:45:51the changes. Makes sense? Cool. That is
3:45:53why knowledge is important, but yeah,
3:45:55you also need to delegate the repetitive
3:45:58boilerplate code to AI now. Cool. So
3:46:01this is Jenny code. Like the very simple
3:46:03use case of Jenny when you have the
3:46:05inline code help, when you have the code
3:46:07help within your session.
3:46:09There is a dedicated space for Jenny as
3:46:12well. Let me just show you.
3:46:13If you click on this particular Jenny,
3:46:15this is Jenny code, like that is also
3:46:17Jenny code, but this is a dedicated
3:46:19space for it. Let me just open it.
3:46:22And this will just simply open this
3:46:23particular window right here.
3:46:25What is the benefit of it?
3:46:26You can literally ask anything
3:46:28within this particular Databricks
3:46:30workspace. Oh, really? Yeah. Let's say
3:46:32um I want you to
3:46:35summarize
3:46:37about
3:46:39what I have
3:46:42inside
3:46:44my
3:46:46demo catalog.
3:46:49I just asked any random question related
3:46:51to this particular workspace. I just
3:46:53simply said that please check my catalog
3:46:55and that's it.
3:46:56It will automatically read the catalog.
3:46:58It will read the schema. All of the
3:46:59schemas are read automatically. It will
3:47:02list the columns, everything. I am
3:47:04expecting a brief summary of what I have
3:47:07inside my catalog.
3:47:09This is so useful. Let's say you are
3:47:12deployed deployed. Like you are let's
3:47:15say
3:47:18on a project and you do not have any
3:47:20knowledge about that project, then
3:47:21obviously you need some context. So the
3:47:23in in the traditional days you would be
3:47:25just looking at each thing by yourself
3:47:27and you would need to just
3:47:29understand what is going on in this
3:47:30catalog. But now you can literally ask
3:47:33all of these things from the AI, from
3:47:35the GenAI code. Okay, just explain me
3:47:37this thing. Okay, okay, how this thing
3:47:38is happening, everything.
3:47:39See, it has just given me a very quick
3:47:42summary
3:47:43and I have bronze schema. I have four
3:47:45row managed table. These are the table
3:47:46names. Then I have silver. I have one
3:47:48white table, one gold, one aggregated
3:47:50table, one LDP schema, one default and
3:47:52information schema, simple. In short,
3:47:54raw detail in bronze, everything
3:47:58everything. I can literally ask anything
3:48:00from this GenAI code here. And if I just
3:48:02want to expand it, I can click on this
3:48:04particular button. It will simply open
3:48:05this particular
3:48:08dedicated space.
3:48:10Simple.
3:48:11Simple, simple, simple.
3:48:13Very good. And these are like some of my
3:48:15previous sessions and I just told you
3:48:16that I just used GenAI code to join the
3:48:18data frames. So this is also stored
3:48:20here. I can even remove it. It's not
3:48:21something like you can just not ignore
3:48:24remove. And one thing which is so good,
3:48:27schedules. This is actually inspired by
3:48:30obviously the popular frameworks that we
3:48:31have. You can even schedule any chat or
3:48:34anything that you want to. Let's say you
3:48:35just want to ingest something. Okay, and
3:48:37you just asked it from GenAI code and
3:48:39you do not want to schedule it every
3:48:41morning. So you can even do that.
3:48:44Makes sense? And if you just want to
3:48:45delete anything, you can simply delete
3:48:46it. Let's say delete this.
3:48:48Delete this. You just delete it. Like
3:48:50it's very easy.
3:48:51So everything is here in front of you.
3:48:53This is Jenny code.
3:48:56Make sense?
3:48:57Now Jenny code can be integrated within
3:48:59the dashboards, within the you can say
3:49:02anything because Jenny code can perform
3:49:04the task for you.
3:49:05So now that means you do not even need
3:49:09to integrate Databricks to your local
3:49:11system to work with cloud code because
3:49:13you do not even need cloud code. You
3:49:15have Jenny code
3:49:17for you.
3:49:18Databricks has provided a better cloud
3:49:21code for you because Jenny code
3:49:23understands the internal things of
3:49:24Databricks. They have created it. They
3:49:26have just trained it and obviously they
3:49:28would have just trained it on some
3:49:29things that you even or no one knows
3:49:31because these are these would be the
3:49:33internals of Databricks. So this is very
3:49:34very good.
3:49:35The Jenny code actually understands your
3:49:37workspace, how to change the things, how
3:49:39to fix the things, everything.
3:49:42Of and
3:49:43the best thing
3:49:45if you just talk about the tokens cost,
3:49:47everything, everything is optimized
3:49:50here.
3:49:51Make sense? Because obviously when you
3:49:53are just using cloud code, you do not
3:49:55even need to
3:49:57worry about
3:49:58how you will just how how you will be
3:49:59just telling everything to cloud code
3:50:01because when you are just using cloud
3:50:02code,
3:50:03you need to first of all attach the
3:50:04MCPs, you first of all need to attach
3:50:06the skills, you first of all need to
3:50:08connect your Databricks and then just
3:50:09share some credentials which can be
3:50:11risky. But here everything is going to
3:50:14Databricks.
3:50:15Simple. So it is secure as well as
3:50:17compared to other things. Obviously you
3:50:19can just use anything, but I'm just
3:50:20telling you the comparison.
3:50:23This is all about Jenny code and I hope
3:50:25that you that you would love and you
3:50:26would just use Jenny code more and more.
3:50:28Okay? And let's see the next topic. So
3:50:31now let's talk about the Databricks
3:50:33visual prep which is one of the most
3:50:35recent additions within the Databricks
3:50:38workspace that we have bought.
3:50:40Databricks visual prep is like no code
3:50:45ETL or basically no code
3:50:48pipelines.
3:50:49No code pipelines where you do even need
3:50:51to write any code.
3:50:54If you have used something called as
3:50:56Azure data factory,
3:50:57if you have used something called as AWS
3:50:59glue, it is very similar to that but for
3:51:02data bricks.
3:51:04Because there are so many people who do
3:51:05not like to write code and let me just
3:51:07tell you first of all,
3:51:09if you are building the complex, the
3:51:11real world end to end data engineering
3:51:13pipelines, you still use all of the
3:51:15stuff that we have discussed.
3:51:17This area is for those you can say mini
3:51:20mini pipelines or you can say some add
3:51:22hoc pipelines that you can just build or
3:51:25some people can build who do not
3:51:26understand the code very well.
3:51:28So understand the difference.
3:51:30Data bricks visual prep is not the
3:51:31replacement for what we have just
3:51:33discussed so far. No. This is just for
3:51:35like mini mini add hoc quick pipelines.
3:51:37Okay.
3:51:38This is for that particular thing but it
3:51:39has a great potential. It has a great
3:51:41potential but I'm just telling you the
3:51:43difference. Okay. If you have like very
3:51:45very complex pipelines, you want a very
3:51:47you can say dedicated support for
3:51:50customization everything, then obviously
3:51:52you can just go with code but if you do
3:51:53not want to write the code from scratch
3:51:55everything, then you can just go with no
3:51:56code pipelines. And yes, the good thing
3:51:58is it is even available within the free
3:52:00edition. So let me show you. So if you
3:52:02just go here and you will just see under
3:52:04data engineering, it's called visual
3:52:06data prep. Click here.
3:52:08And here you can just click on plus new
3:52:10because it is also called as lake flow
3:52:12designer. This is another name if you
3:52:14want to know. Let me just write it here.
3:52:17Lake flow designer.
3:52:23Cool. And it is also called a data
3:52:24bricks visual prep.
3:52:26Cool.
3:52:27So if you just want to create plus new
3:52:31and it will just open the canvas. This
3:52:33canvas will be a little bit similar to
3:52:37AWS glue if you have used. Okay. And if
3:52:39you have not used, then do not worry.
3:52:42This is the you can say
3:52:44canvas. So what we do here, we have
3:52:46multiple options. If you want to let say
3:52:49use any source,
3:52:50you can just pick source, import data
3:52:52into designer. It will
3:52:54automatically import it.
3:52:55But sometimes you have the
3:52:58local files.
3:52:59You want to manually enter the data.
3:53:01You can click here, enter data, and you
3:53:02can just manually feed it, and you can
3:53:04even drop the local files from here.
3:53:06And you have all of the connectors here
3:53:08as well that we just saw in the lake
3:53:09flow connect. So you have a very good
3:53:11list of um you can say connectors. Let
3:53:13say I want to pick the source. And here
3:53:15I want to pick let say orders data
3:53:17because I already have this table. So
3:53:19this will be picked automatically and
3:53:20this will connect this thing here and
3:53:22this is the canvas. I can do a lot of
3:53:24stuff.
3:53:25So this is my orders table. Let's wait
3:53:26for it to load because it will run that
3:53:28particular
3:53:29query to show you the preview and
3:53:31everything. Cool. So this is the preview
3:53:33I can see.
3:53:34Let me have some space and this is my
3:53:37output.
3:53:38Now let's say I want to filter some
3:53:41records.
3:53:42I can do that, right?
3:53:44So what I can do? I I will simply click
3:53:46here in the filter and I will put what
3:53:50condition I want.
3:53:51Let's say
3:53:52first condition, first condition is
3:53:53let's say store ID
3:53:55is equal to let's say 10. I can even
3:53:58click on drop down just to see what
3:53:59values I have. Let's say one. Okay. You
3:54:01will say why are you just picking again
3:54:02again 10?
3:54:04So this is one way. If you want to put
3:54:06some customization using code, you can
3:54:08click here as well and here you can just
3:54:10write the expression. Either way it will
3:54:11work.
3:54:12Click on apply
3:54:14and it will apply this transformation
3:54:16without writing any code just by
3:54:18selecting the things and I can just see
3:54:20it here. See?
3:54:22I have this particular thing. So let me
3:54:23just click here, just here and that's
3:54:26it. So this is my preview.
3:54:29Now let's look at the preview. I have
3:54:31this particular preview and I only have
3:54:3336 rows
3:54:34in the output. Why?
3:54:36Because
3:54:37out of 1,000 rows, out of 1,000 rows and
3:54:40obviously these are like just 4,000 rows
3:54:42out of 10,000 rows,
3:54:44I just have 36 records for store equals
3:54:47to one.
3:54:48Did I write any kind of code? No.
3:54:50Everything is automatically done.
3:54:52Wow.
3:54:53This way you can perform all of the
3:54:55transformations select, limit, sort, you
3:54:58can even combine, join, you can even
3:55:02perform
3:55:03AI functions as well, like all of the
3:55:05things like collect not collect like
3:55:07parse information everything, all of the
3:55:09AI functions that Databricks has,
3:55:11data parsing, everything.
3:55:14Then you can even perform some
3:55:17notification as well.
3:55:19This is amazing. If you just want to
3:55:21send any notification to yourself, to
3:55:23your let's say email, to your Slack,
3:55:25Messenger, anything, so you can just do
3:55:27that as well.
3:55:28And the most important thing,
3:55:32if let's say
3:55:34obviously you can just perform drag and
3:55:35drop. It is amazing, it is very
3:55:37convenient. If let's say
3:55:39you do not even need to perform drag and
3:55:41drop, then what you can do? You can ask
3:55:43this particular thing. This is nothing
3:55:45but a kind of Databricks Gen code. You
3:55:48can simply say
3:55:50I want to change my filter
3:55:53and I want
3:55:55to filter
3:55:56store ID
3:55:57equals
3:55:5910.
3:56:00Now this will look at my
3:56:04workflow and it will
3:56:05enable auto proof. Yes, enable.
3:56:08It will automatically change everything.
3:56:09You can even ask it to build more and
3:56:11more transformation, more and more
3:56:12steps, or you can even ask it to change
3:56:14the existing step or anything.
3:56:17And let's say allow because obviously it
3:56:19would need to
3:56:21just ask you before changing it. So it
3:56:23is just equal it is changing this
3:56:24particular thing.
3:56:25And we will just see the result very
3:56:27very soon once it is done.
3:56:29So I can see that it is changed. Now I
3:56:30can see all of the store IDs are 10.
3:56:33Wow. [snorts] Now you do not even need
3:56:34to perform drag and drop. You can rely
3:56:36on this particular thing. But remember
3:56:39this is just for the you can say
3:56:42mini mini pipelines, mini mini ad hoc
3:56:44pipelines that you can just build. And
3:56:46anyone can build, but those who know how
3:56:48to build the things, not anyone, like
3:56:50who know how to build.
3:56:52And this is very important for data
3:56:53engineers as well. Why? Because
3:56:54sometimes you just want to build a quick
3:56:56POC pipeline just to provide it to
3:56:58stakeholders that's it.
3:57:00So it is a very good thing for data
3:57:02engineers as well, not just like for non
3:57:03data engineers or non technical people.
3:57:05This is very important for data
3:57:06engineers as well.
3:57:07Makes sense? So this component is like
3:57:09very recently added in Databricks very
3:57:11very recently and I hope it makes sense.
3:57:14Very good. Let's look at the last topic
3:57:16for today. It's called Databricks
3:57:19Lakehouse which is actually one of the
3:57:23hottest topics right now. Why?
3:57:25Because this has changed the dynamics
3:57:28how people use databases nowadays. So
3:57:31what is this Lakehouse?
3:57:32First of all, let me just tell you
3:57:33Lakehouse is nothing but
3:57:35your PostgreSQL database. Really? Yeah.
3:57:39PostgreSQL database but managed by
3:57:42Databricks.
3:57:43Really? Just managed? Not really much
3:57:45more than that.
3:57:46So what actually happened is
3:57:48Databricks Lakehouse is actually a very
3:57:51interesting concept and I created like
3:57:52dedicated video on Databricks Lakehouse.
3:57:55You can just search Databricks Lakehouse
3:57:56and you will just find the video for
3:57:57that. But I will still tell you. Let me
3:57:59just show you first of all.
3:58:01Databricks
3:58:03Lakehouse
3:58:07Yeah, this is the video Databricks
3:58:09Lakehouse. I just created I think one
3:58:10hour tutorial on it because it is like
3:58:12very I created like I think three months
3:58:14back when it was like newly newly
3:58:17launched. So like not launched, it
3:58:20gained popularity during that time.
3:58:21Obviously it was just like like like
3:58:23launched way back but yeah.
3:58:25So what is this thing? So Databricks is
3:58:28using PostgreSQL database under the hood
3:58:29for sure, but that is not the only
3:58:31thing.
3:58:33They have used something called as you
3:58:36can say decoupled architecture.
3:58:38What does it mean? So let's say
3:58:40in databases
3:58:42we get two major things.
3:58:44Two major things. I can also show you
3:58:46the official documentation. Not official
3:58:47documentation. They are just written
3:58:48like a very good
3:58:50blog on this. Oh, let's find it. Let's
3:58:52try to find it. If I just find it
3:58:54Databricks Lakehouse blog.
3:58:58I think it was created by um
3:59:03their senior member and I really like
3:59:04that blog. Let me just find it.
3:59:06So I just found this particular blog
3:59:08finally because there are multiple
3:59:10blogs, but I really like this blog by
3:59:12Reynoldsen and this blog actually
3:59:15explained the things very very well. So
3:59:16basically
3:59:18the traditional database has two major
3:59:20things first of all wall and data bytes.
3:59:23Wall write ahead logs and data bytes.
3:59:26Wall is just for the metadata. Data
3:59:28bytes are the real bytes that we just
3:59:29write and that is why it is also called
3:59:31as data and your logs.
3:59:35Now this wall and data files were
3:59:40you can say
3:59:41hardly connected with this particular
3:59:43database engine
3:59:45in the traditional database.
3:59:47But now in this lake base this is
3:59:50decoupled. That means this engine will
3:59:53be in a separate machine.
3:59:55This wall
3:59:56which they now call it as safe keeper is
3:59:59created in a separate machine. Page
4:00:02server
4:00:03which is a real data
4:00:05is also created in a separate machine.
4:00:08Now this is decoupled.
4:00:10Now this is not the only thing because
4:00:12obviously decoupled architectures are
4:00:14there, but the major thing that they
4:00:15have done is this one.
4:00:18Object storage.
4:00:19They have internally or you can say they
4:00:23are internally converting your page
4:00:25server basically real pages into the
4:00:28object storage in the form of parquet
4:00:31files so that you can use let say Spark
4:00:35or any engine to directly access the
4:00:37data.
4:00:39And this way
4:00:41your data lives in the cloud object
4:00:43storage.
4:00:45You do not even need to create multiple
4:00:47you can say replicas of the data.
4:00:49You do not even need to move the data.
4:00:51You just need to change the engine. If
4:00:53you want to use the database database
4:00:55engine you just use Postgres. If you
4:00:57just want to use Spark you can just use
4:00:58Spark. Simple.
4:01:00There'll be just one copy of the data.
4:01:02This concept is very very very very like
4:01:06you can say important and it is changing
4:01:08the dynamics of the databases nowadays.
4:01:10Especially for data engineering,
4:01:12especially for data engineering because
4:01:14now what you can do, you can literally
4:01:16access these databases via the Unity
4:01:18catalog. You can even access the you can
4:01:21say
4:01:23you can even convert these tables or you
4:01:25can say also sync these tables to the
4:01:27Unity catalog tables or basically your
4:01:30traditional Delta tables. You can sync
4:01:32these things.
4:01:33Really yeah. You can even sync these
4:01:35things. There are like built-in
4:01:36pipelines that you can just use.
4:01:39Oh yes.
4:01:41Everything is there. Everything is
4:01:43there.
4:01:46This is actually very good.
4:01:47And you can also just read this article
4:01:49if you want to, but the major thing that
4:01:51is discussed in this particular entire
4:01:53entire article is the one that I just
4:01:55told you like the decoupled architecture
4:01:56and that
4:01:57you can say
4:02:00the this particular thing object
4:02:01storage. And this gave rise to a new
4:02:03concept
4:02:05called
4:02:06Delta
4:02:07Lake Transactional Analytical Processing
4:02:09because before that
4:02:11we were using OLTP
4:02:13or OLAP.
4:02:16And yes, there was like one more concept
4:02:17that came in the middle which was
4:02:20HTAP, Hybrid Transactional Analytical
4:02:23Processing.
4:02:24But yeah, it was like not very very
4:02:25popular
4:02:26because in there you would need to
4:02:27create the replica still, but here you
4:02:29do not need to create the replica.
4:02:32This is the concept of Data Bricks Lake
4:02:34Base and yes, in the free edition you
4:02:36get database Lake Base as well. Wow.
4:02:38They have literally invested so much
4:02:41to provide these features for free. And
4:02:44to be honest, that's how people learn,
4:02:45that's how people use more. So it is
4:02:47like directly related to their sales as
4:02:49well, but that is fine. So if you just
4:02:50want to create this thing, you can click
4:02:52on Compute and here you will see Lake
4:02:54Base and then you can just click on Lake
4:02:56Base.
4:02:56Postgre, they have created dedicated
4:02:58space as well. Now it like it was there
4:03:00indicated here before, but now they have
4:03:03just created a dedicated space. It's
4:03:04called Lake Base Postgre.
4:03:06And there you can just go, explore, and
4:03:08just create. I already have one this
4:03:10particular thing. Lake base demo. I just
4:03:12created this thing three months back.
4:03:14And if you just want to learn Databricks
4:03:16Lake base alone in detail, you can just
4:03:18watch that video that I just showed. And
4:03:20that's how you can just
4:03:22managed Databricks Lake base, database,
4:03:25branches, everything.
4:03:27Everything.
4:03:29Make sense? Make sense? Make sense?
4:03:31This is all about our Databricks
4:03:34concepts that we have just covered in
4:03:37this particular video. And I think we
4:03:39have covered 20 plus concepts. I think
4:03:4020 21 concepts. And I think that is
4:03:42good.
4:03:43And we learned a lot of a lot a lot of
4:03:46things. And I hope that you also learned
4:03:48and understood so many things. And I
4:03:49hope now you can just become a better
4:03:51Databricks data engineer, Databricks
4:03:53developer, or any developer who wants to
4:03:55learn Databricks, right?
4:03:57So that's all for today's video. If you
4:03:59loved the video, and I know you loved
4:04:01the video, simply hit the subscribe
4:04:02button and share your thoughts in the
4:04:04comment section. And yes, share this
4:04:05video with your friends so that they can
4:04:06also learn and grow. And happy learning.

About this transcript

This page contains the full transcript of Databricks Was HARD Until I Learned These 21 CONCEPTS (Databricks Ansh Lamba) by Ansh Lamba, generated from the public captions YouTube serves with the video. The transcript has 42,215 words across 6,967 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.