Delta Lake Architecture Explained | How the Lakehouse Revolutionizes Data Engineering" — Transcript

Full transcript

1. 0:01[Music]
2. 0:08Well, we all know what is a data lake
3. 0:10and actually we have used data lake in
4. 0:12this particular course couple of time
5. 0:14which is actually our Azure data lake
6. 0:16storage gen 2. So we know that we can
7. 0:19store a huge amount of data inside a
8. 0:21data lake. Um but we don't know what is
9. 0:23a delta lake and now that's the time to
10. 0:26understand uh a delta lake actually and
11. 0:28we also need to understand delta lake
12. 0:30architecture in this. So let's focus on
13. 0:32this first. Uh so what is a data lake?
14. 0:35The data lake stores a large volume of
15. 0:37structured semistructured or
16. 0:39unstructured data in short all kind of
17. 0:41data in its native format. Uh mostly
18. 0:43data lake is like welcoming all kind of
19. 0:45data from no matter which kind of source
20. 0:47it is. Data lake architecture has
21. 0:50evolved in recent years to better meet
22. 0:52the demands of increasingly datadriven
23. 0:55enterprises as the data volume continues
24. 0:57to rise. We are living in a world where
25. 0:59data is increasing every day and then
26. 1:01when we have huge amount of data the
27. 1:04data lake which was invented before few
28. 1:06years has actually some evolution inside
29. 1:09that and then this evolution has changed
30. 1:11the data lake structure also with that
31. 1:14and from that only we got something
32. 1:16which is known as delta lake before I
33. 1:19explain what is a delta lake let me tell
34. 1:21you one thing guys delta lake is not one
35. 1:23of the new service of Azure cloud it's
36. 1:25not it's not a service it's actually a
37. 1:27conceptual architecture ure and you have
38. 1:29to implement this conceptual
39. 1:31architecture with your data lake only.
40. 1:33So when you're going to use your data
41. 1:34lake gen 2, you are going to associate
42. 1:37that in such a way that it is going to
43. 1:38be treated like a delta lake. So what is
44. 1:41this? Well, the delta lake is an
45. 1:43open-source storage layer that brings
46. 1:46reliability and performance to your
47. 1:47existing data links. So why and how this
48. 1:51is going to brings reliability and
49. 1:53performance actually and why it is
50. 1:54required that is something which we're
51. 1:55going to focus right now. They are
52. 1:57saying delta lake provides asset
53. 1:59transactions, scalable metadata handling
54. 2:02and unifi streaming and batch data
55. 2:05processing. Now all the things are
56. 2:08obviously missing in the normal data
57. 2:09lake. Most of the time data is just
58. 2:12going to store the huge amount of data
59. 2:14inside the data lake kind of services
60. 2:16but it's not associating with asset
61. 2:18transaction. I hope you heard about
62. 2:20asset transactions, atomicity,
63. 2:22consistency, whatever. And uh when you
64. 2:25want scalable metadata handling or you
65. 2:27want some unified or structured
66. 2:29streaming kind of things, all these
67. 2:30things are where actually uh missing
68. 2:32from the data lake architecture where
69. 2:35delta lake is actually focusing on this
70. 2:37kind of things. The delta lake runs on
71. 2:39top of your existing data lake. So it's
72. 2:42not a new service as I said on your
73. 2:44existing data lake only. this is going
74. 2:45to be running and it is fully compatible
75. 2:48with Apache Spark APIs. That's one of
76. 2:50the reason when you're going to use your
77. 2:52Synapse notebooks or you're going to use
78. 2:54your datab bricks notebooks, your Apache
79. 2:56Spark APIs will directly associate with
80. 2:58your data lake and then you can
81. 3:00implement Delta Lake with that. In order
82. 3:03to understand this concept properly,
83. 3:04let's focus on this next slide. This is
84. 3:07actually something which is a Delta Lake
85. 3:09architecture. Now I have a two
86. 3:11variations of this architecture. This is
87. 3:13a simpler one. If you focus on this
88. 3:15architecture left side, you have a data
89. 3:18which is maybe a batch data or streaming
90. 3:20data which is going to be ingested into
91. 3:22this delta lake architecture.
92. 3:24The bottom layer of this delta lake
93. 3:26architecture you can see is nothing but
94. 3:27your same existing data lake. On top of
95. 3:30this data lake, you're going to
96. 3:31implement delta lake where we have three
97. 3:33different variations of our data. We
98. 3:36have something which is known as bronze
99. 3:38table, silver table and gold table. Some
100. 3:40people call this thing layers also
101. 3:42bronze layer, silver layer and gold
102. 3:44layer of your data. And these layers are
103. 3:46actually making it special. These layers
104. 3:48are actually going to have uh different
105. 3:50kind of processed unprocessed data
106. 3:52inside that which are going to make this
107. 3:54thing special. Now most of the time when
108. 3:56you have a data in bronze, silver and
109. 3:59gold each layer is having some
110. 4:01characteristics of the data. The final
111. 4:04layer of this particular data is gold
112. 4:05layer or gold tables. This is actually
113. 4:07the one which is going to be used with
114. 4:09your Azure machine learning and some
115. 4:11other further processing. But then the
116. 4:13process actually going to start from the
117. 4:15bronze. Most of the time your bronze
118. 4:18layer or bronze tables are going to have
119. 4:20your streaming data batch data which is
120. 4:22coming from various sources. Once the
121. 4:24data is stored inside the bronze layer,
122. 4:26we will pick the data and we are going
123. 4:28to refine those data with some kind of
124. 4:31additional joins or some other
125. 4:32meaningful associations with that. This
126. 4:35meaningful refined tables are going to
127. 4:36be treated as a silver layer or silver
128. 4:38tables with that. But after that also
129. 4:41silver tables are not ready to be
130. 4:42processed or not something which is
131. 4:44exactly looking for the business
132. 4:46requirement and that's why on the silver
133. 4:48tables we are going to apply some kind
134. 4:49of aggregates. These aggregates are
135. 4:52going to be applied and then final data
136. 4:53layer is going to be created which is
137. 4:55your gold layer which is going to be
138. 4:56further useful in the processing of
139. 4:58PowerBI or maybe machine learning or
140. 5:01maybe some artificial intelligence kind
141. 5:03of things. These three layers are
142. 5:05actually making this Delta Lake
143. 5:06architecture special and that's why
144. 5:08let's deep dive into these words which
145. 5:10are bronze, silver and gold.
146. 5:14Hey guys, sorry for interruption. My
147. 5:15name is Marauti and I'm here to make an
148. 5:18very important announcement. I hope you
149. 5:20are liking our videos and you're doing a
150. 5:22continuous learning with us on an Azure
151. 5:24cloud and AI related topics. If you are
152. 5:27enjoying this thing, I'm going to
153. 5:28announce skilltech.club club which is
154. 5:30our upcoming website which is going to
155. 5:32be launched very soon. We are here to
156. 5:34tell you one thing that everyone who is
157. 5:37a subscriber of this particular channel
158. 5:39will get Azure cloud and Azure AI
159. 5:41related certification courses free of
160. 5:43cost in skilltech.club. So you will be a
161. 5:47part of the skilltech.club kind of a
162. 5:48membership automatically free of cost
163. 5:51and everyone who's a subscriber of this
164. 5:53particular channel will get those
165. 5:55benefits which are available in that. So
166. 5:57what are you waiting for? I request you
167. 5:59to please subscribe to this channel and
168. 6:01share it with your friends and families
169. 6:02if they are also interested in Azure
170. 6:05cloud and AI learning. That's it from my
171. 6:07side. Now you can carry on with your
172. 6:09learning. Thank you. If I focus on this,
173. 6:13the answer says that the bronze table
174. 6:15contains raw data ingested from its
175. 6:18various sources. It can be JSON, RDBMS,
176. 6:21IoT data or maybe some data coming from
177. 6:23the event hub kind of live streaming.
178. 6:25This bronze data is going to be further
179. 6:27processed into a silver. So silver
180. 6:29tables will provide more refined view of
181. 6:31our data. Most of the time in this case
182. 6:33you can join the fields from various
183. 6:35bronze tables to enrich your streaming
184. 6:37records or maybe you can update your
185. 6:39account statuses based on the recent
186. 6:41activities. It's something which is a
187. 6:43further processing or data preparation
188. 6:45kind of things which you are applying on
189. 6:47top of your bronze layer. Once the
190. 6:49silver layer is processed, you are going
191. 6:51to have the final layer process from
192. 6:53this which is gold tables. These gold
193. 6:55tables are going to provide business
194. 6:57level aggregates and often used for
195. 6:59reporting and dashboarding or maybe for
196. 7:01machine learning and model creations.
197. 7:04This would include aggregations such as
198. 7:06maybe it's going to be daily active
199. 7:07website users you want to see or maybe
200. 7:09you want to get a sales per store uh or
201. 7:12maybe you want to associate with the
202. 7:14gross revenue per quarter for any
203. 7:16particular company and department. This
204. 7:18kind of more meaningful data is going to
205. 7:20be used in my gold layer. And then this
206. 7:23transition of data from bronze to silver
207. 7:25to gold is the heart of your delta lake
208. 7:27architecture. The end outputs are
209. 7:30actionable insights, dashboards and
210. 7:32reports which maybe you're going to
211. 7:34associate with any existing service.
212. 7:36This delta lake architecture if you
213. 7:38understand with this bronze, silver and
214. 7:40gold, the detailed view of this
215. 7:42particular architecture is going to be
216. 7:43somehow going to look like this. Now if
217. 7:46you see right now my delta lake
218. 7:48architecture is actually focusing only
219. 7:50on this box but I want you to see the
220. 7:52full picture of the delta lake
221. 7:53architecture and that's the reason I
222. 7:55have associated this delta lake
223. 7:57architecture with the other services
224. 7:58which are associated with that you can
225. 8:00see I'm taking a scenario of data bricks
226. 8:02right now because exactly after this
227. 8:04video I'm going to show you how you can
228. 8:06implement delta lake architecture with
229. 8:08data bricks actually in this case
230. 8:10obviously the flow is going to start
231. 8:12from the streaming data or maybe a batch
232. 8:15data which is coming from various
233. 8:16sources. All the sources of data which
234. 8:18are coming into this will be ingested
235. 8:20into this particular data bricks and
236. 8:22then that is going to be stored inside
237. 8:24my data lake as a delta lake and that is
238. 8:28going to be my bronze layer of my data.
239. 8:30Once blown layer of data is there we are
240. 8:32going to take that data and we are going
241. 8:34to do some data preparation ETL kind of
242. 8:36things on that and from bronze we are
243. 8:38going to get another layer of the data
244. 8:40which is going to be silver. These are
245. 8:42going to be much more refined tables as
246. 8:43we discuss but this is not something
247. 8:46which is having some business related
248. 8:47aggregates on that. So we'll take the
249. 8:49silver tables and then we are going to
250. 8:51apply some extraction some aggregators
251. 8:53on that and then we are going to have a
252. 8:55final layer of the data which is going
253. 8:57to be my gold layer. Technically guys
254. 8:59this gold layer is actually nothing but
255. 9:01your data m. This is the one which is
256. 9:03focusing on some business features and
257. 9:05capabilities with that. And that data is
258. 9:07a final produced data of this process
259. 9:10which is going to be further stored into
260. 9:12a maybe a cloud data warehouse like
261. 9:14Synapse or maybe it can be directly
262. 9:16associated with the PowerBI or maybe
263. 9:19Tableau or maybe some other related
264. 9:21product where we can either generate
265. 9:23reports or we can use this thing for
266. 9:25machine learning model training or some
267. 9:27other purpose which can be there. This
268. 9:30full process, this full architecture is
269. 9:32actually showing you that how Delta Lake
270. 9:34is going to be an important part for
271. 9:36your end to-end data processing which is
272. 9:38happening with this kind of data bricks
273. 9:40or Azure Synapse kind of services. This
274. 9:43is going to be an complex architecture
275. 9:45to understand but I'll be giving you one
276. 9:48guarantee that if you understand this
277. 9:49architecture and if you know how to
278. 9:51implement this thing this way in most
279. 9:53organizations in your projects when you
280. 9:55are doing data analytics this is going
281. 9:57to be the heart of that architecture.
282. 10:01Now if you understood this architecture
283. 10:03I want you to take a screenshot of this
284. 10:05particular one and then just make sure
285. 10:07that you are going to refer this thing
286. 10:09in our next sample. Next two videos are
287. 10:12going to show you how you can implement
288. 10:13Delta Lake architecture with data bras
289. 10:16actually. So let's have a look at that
290. 10:18one. I hope you understood the
291. 10:20architecture and it will be much more
292. 10:22clear when you see this thing
293. 10:23practically how it is going to be
294. 10:25implemented in data bricks. Thank you.
295. 10:29[Music]