Databricks Data Engineer Professional Practice Test Questions - Part 32 — Transcript

Full transcript

1. 0:01All right, let's tackle this scenario
2. 0:04here where
3. 0:05fine-grained access controls are
4. 0:07required.
5. 0:09And we want to minimize
6. 0:11maintenance overhead ensuring
7. 0:13scalability.
8. 0:15Option A,
9. 0:16external tables require manual
10. 0:19management of storage paths
11. 0:22and adds complexity.
12. 0:25Hence, incorrect.
13. 0:26Now, option B,
14. 0:28scheduling optimize and vacuum increases
15. 0:32operational overhead.
16. 0:34We'll delete it. If you want the PDF
17. 0:36version of this course, please enroll in
18. 0:38diamond membership or above by clicking
19. 0:41the join button now.
20. 0:43Once enrolled, please connect and inbox
21. 0:45me on LinkedIn at the rate of Cloud Guru
22. 0:48Amit or Instagram at the rate of Amit
23. 0:50Fizi. I'll be glad to help you out with
24. 0:53the PDF version of this course
25. 0:55and hands-on files if available for this
26. 0:57course. Option C, Hive metastore
27. 1:01lacks
28. 1:03Unity Catalog's fine-grained governance
29. 1:06and modern optimization features. Wrong
30. 1:09answer.
31. 1:10Option D,
32. 1:12says create a managed table in Unity
33. 1:14Catalog,
34. 1:16configure Unity Catalog permissions, and
35. 1:19rely on predictive optimization to
36. 1:21enhance performance and simplify
37. 1:24maintenance.
38. 1:25Managed Unity Catalog tables with
39. 1:28predictive optimization
40. 1:31strong governance,
41. 1:33reduced
42. 1:34maintenance,
43. 1:36and improved query performance.
44. 1:38We'll lock option D as the right answer.
45. 1:41Let's bring the heat to the snow.
46. 1:44We
47. 1:46need
48. 1:47to manage an order
49. 1:49Delta table with deletion vectors
50. 1:51enabled to improve efficiency here.
51. 1:55We need to ensure both performance and
52. 1:57data integrity.
53. 1:58Option A,
54. 2:01physical rewrite of files adds overhead
55. 2:05and is
56. 2:06not
57. 2:08how deletion vectors operate.
58. 2:10Incorrect. Now, let's look at option B.
59. 2:14Deletion vectors do not alter data
60. 2:17files, only metadata tracks deleted
61. 2:20rows. Incorrect.
62. 2:22Option C
63. 2:23says rows are flagged as deleted in
64. 2:25metadata only, not in data files.
65. 2:29Deletion vectors
66. 2:32mark rows as deleted in metadata,
67. 2:36leaving files unchanged for efficiency.
68. 2:39We'll keep it.
69. 2:40Option D,
70. 2:41Delta doesn't
71. 2:44automatically purge rows.
72. 2:47Permanent removal requires vacuum.
73. 2:51Hence, incorrect. We'll lock option D as
74. 2:54the right answer. Let's now tackle this.
75. 2:58We need to make sure the rules are
76. 2:59applied dynamically to the claims table.
77. 3:03We have the JSON
78. 3:04code looks something like this.
79. 3:08Option A, SQL constraint blocks
80. 3:12cannot dynamically parse JSON metadata.
81. 3:14Incorrect.
82. 3:16Now, option B says load the JSON
83. 3:18metadata, iterate through its entries,
84. 3:22and apply expectations using DLT
85. 3:25expectation all.
86. 3:27DLT expectation all allows dynamic
87. 3:31application of JSON defined rules,
88. 3:35ensuring flexibility and scalability.
89. 3:37We'll keep it.
90. 3:39Option C, external API calls add latency
91. 3:45and complexity without native
92. 3:47integration. Wrong answer. Option D,
93. 3:50@dlt.expect
94. 3:54decorators require hardcoding each rule,
95. 3:58reducing adaptability. We'll delete it.
96. 4:01Option B is the right answer.
97. 4:03Let's bring the heat to the snow.
98. 4:06We
99. 4:07need automatically propagated to the
100. 4:11customer silver
101. 4:13during pipeline execution here.
102. 4:16So, there's a silver table named
103. 4:18customer silver.
104. 4:20We need to make sure
105. 4:22deletions in the bronze table are
106. 4:24propagated in the silver table.
107. 4:27Option A, apply changes
108. 4:30without CDF
109. 4:33only filters soft deleted rows, not true
110. 4:37deletions.
111. 4:38Incorrect.
112. 4:39Now, option B,
113. 4:41CDF on target tables
114. 4:44doesn't capture source deletions for
115. 4:47propagation. Wrong answer.
116. 4:49Option C says enable CDF on customer
117. 4:53bronze,
118. 4:54read its CDF stream, and use apply
119. 4:57changes with apply as deletes for
120. 5:00customer silver.
121. 5:02CDF on the source with apply changes and
122. 5:08apply as deletes ensures deletions flow
123. 5:11to the target table. We'll keep it.
124. 5:13Let's move to option D.
125. 5:16Option D, vacuum removes files but
126. 5:18requires full rebuild,
127. 5:21adding unnecessary overhead. We'll
128. 5:23delete it. Option C is the right answer.
129. 5:27Let's now tackle this.
130. 5:29We need to make sure cust key
131. 5:32is not null,
132. 5:34and order amount is greater than zero.
133. 5:38Option A,
134. 5:40expect
135. 5:42or drop ensures invalid rows are
136. 5:46dropped, enforcing data quality. This
137. 5:49looks good.
138. 5:51We'll uh keep option A for now. Option
139. 5:53B,
140. 5:55chained uh chained expect or drop is not
141. 6:01supported
142. 6:02in the syntax itself. Syntax are
143. 6:05incorrect.
144. 6:06Now, option C,
145. 6:08expect
146. 6:09only
147. 6:10flags invalid rows, but doesn't drop
148. 6:14them.
149. 6:15Wrong answer.
150. 6:17Option D,
151. 6:18expect in
152. 6:20chained form
153. 6:22still retains invalid rows not meeting
154. 6:25the requirement of the question.
155. 6:27Incorrect. Option A is the right answer.
156. 6:30Let's now look at this scenario here. We
157. 6:32need to
158. 6:33a way that is automatically available on
159. 6:36every cluster provisioned in the
160. 6:38workspace.
161. 6:39Let's look at option A.
162. 6:41Uploading to a
163. 6:44Unity Catalog volumes
164. 6:46with an init script ensures
165. 6:49PyYAML is installed consistently across
166. 6:53all new clusters. We'll keep it.
167. 6:56Option B,
168. 6:57get repo
169. 6:59do not automatically install
170. 7:01wheel files
171. 7:03on clusters.
172. 7:05Incorrect. Now, let's move to option C.
173. 7:08Option C, installing from a user home
174. 7:12directory limits availability and
175. 7:15doesn't scale across all clusters. Wrong
176. 7:17answer.
177. 7:19Option D, a private PyPI repository
178. 7:23requires external connectivity,
179. 7:25which is not possible in an air-gapped
180. 7:28workspace.
181. 7:29Wrong answer. Option A is the right
182. 7:31choice.
183. 7:32Let's now look at this. We need to
184. 7:35invoke a job named my project job.
185. 7:39The execution should respect the prod
186. 7:41target context. so the job runs with the
187. 7:44correct target specific configuration.
188. 7:47Option A, Databricks job run doesn't
189. 7:50support the environment flag for bundle
190. 7:54target execution, incorrect.
191. 7:56Now, option B,
192. 7:57Databricks execute is not a valid CLI
193. 8:02syntax, wrong answer.
194. 8:04Option C,
195. 8:06Databricks run ignores bundle target
196. 8:09context
197. 8:10and cannot enforce
198. 8:13flag T prod, we'll delete it.
199. 8:16Option D,
200. 8:17Databricks bundle run my project job/t
201. 8:23prod correctly executes the job in the
202. 8:26specified bundle target context, we'll
203. 8:28lock option D as the right answer.
204. 8:31Let's now tackle this.
205. 8:34We want a modular and testable way to
206. 8:37apply data frame transform logic. Option
207. 8:40A,
208. 8:41pipeline class mixes object design with
209. 8:45ETL logic, reducing modularity for
210. 8:47testing, incorrect. Now, option B,
211. 8:51transform data function isolates
212. 8:54transformation logic and enables unit
213. 8:57testing with assert data frame equal,
214. 9:00we'll keep it. Now, option C,
215. 9:03DF transform upper value fails
216. 9:07since the function signature doesn't
217. 9:10align with data frame expectation, we'll
218. 9:12delete it. Option B is the right answer.
219. 9:16Let's now look at this scenario here.
220. 9:19The patient records across several Delta
221. 9:22Lake tables.
222. 9:24We want to know how the results are
223. 9:26generated each time the dashboard
224. 9:28refreshes.
225. 9:29Option A, scanning all Delta files in is
226. 9:33inefficient
227. 9:35and not how
228. 9:37count star is optimized. Incorrect.
229. 9:41Option B
230. 9:43cached results depend on a refresh
231. 9:47and doesn't guarantee accurate row
232. 9:50counts. Incorrect.
233. 9:52Option C says the row count is derived
234. 9:54from the Delta transaction logs.
235. 9:56Delta transaction logs track row counts
236. 10:00and enable efficient count star queries.
237. 10:04Let's keep it. Option D
238. 10:07parquet metadata contains
239. 10:09file
240. 10:11statistics but
241. 10:13Delta relies on transaction logs for
242. 10:16consistent counts. We'll delete it.
243. 10:18Option C is the right answer. Let's
244. 10:21bring the heat to this snow.
245. 10:23The team needs to report daily resource
246. 10:26consumption by SKU tier across all
247. 10:29workspace. We
248. 10:31need to
249. 10:33we have decided to query the billing a
250. 10:35system.billing.usage
251. 10:37system table in Databricks to generate
252. 10:39accurate usage metrics. So which SQL
253. 10:42query will correctly return the daily
254. 10:44usage by product?
255. 10:46Option A
256. 10:48basically uses
257. 10:50sum dbus
258. 10:52which is not the correct field for DBU
259. 10:56aggregation.
260. 10:57Incorrect. Now option B basically uses
261. 11:02count usage quantity
262. 11:05which misinterprets the consumption by
263. 11:08counting rows instead of summing usage.
264. 11:13Wrong choice. We are left out with
265. 11:15option C.
266. 11:17It basically uses sum usage quantity
267. 11:20group by
268. 11:22day and
269. 11:25SKU
270. 11:26which correctly calculates daily DBU
271. 11:30usage by product. Looks good. Let's lock
272. 11:33option C as the right choice.
273. 11:36So, please, please, please don't go
274. 11:37away. Let's meet in next part of this
275. 11:39series.
276. 11:40If you want the PDF version of this
277. 11:42course, please enroll in diamond
278. 11:43membership or above by clicking the join
279. 11:47button now.
280. 11:48Once enrolled, please connect and inbox
281. 11:50me on LinkedIn at the rate of Cloud Guru
282. 11:53Amit or Instagram at the rate of Amit
283. 11:55Physique. I'll be glad to help you out
284. 11:57with the PDF version of this course
285. 11:59and hands-on files if available for this
286. 12:02course. Thank you so much for watching
287. 12:03this video.