YT YouTube2Text

YouTube2Text → Transcripts → Delta Lake Interview Questions

Delta Lake Interview Questions — Transcript

by DataBeli · 3,575 words · 498 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Hi, in this video I'm going to cover the
0:03real interview questions about Delta
0:05Lake that I ask when I take interviews
0:08of data engineers and I have included
0:11the theoretical questions that confirms
0:13the in-depth knowledge of Delta Lake
0:15concepts. Also, I have included many
0:18practical questions that confirms that
0:20the person has actually worked on Delta
0:23Lake. And talking about myself, I'm
0:25Narendra Kumar. I have 12 years of
0:28experience in IT and data engineering. I
0:30have spent eight years now. I have
0:32completed these certifications also and
0:35taking interviews for data engineers. It
0:37is part of my daily job. So let's get
0:40started.
0:42So the first question that I ask it is
0:44uh let's say if we have created a data
0:46table or if we have stored the delta
0:49data in some particular location. If we
0:52go to that location, so what kind of
0:54files and folders do we see? So the
0:57answer to that is let's say if this is a
0:59sample table we have created. So there
1:02we will see some parquet files and a
1:05folder with the name delta log. This
1:08park file they really contain the actual
1:12data of the table and delta log these
1:15contain the transaction history and the
1:18transaction history it is contained in
1:19terms of these JSON log file and CRC
1:23these are the check sum files which
1:25ensure the consistency or completeness
1:28of these JSON files and after we keep
1:32writing multiple operation on delta log.
1:35So after some time this JSON will be
1:38combined and here also a park file will
1:41be created. So that is how the kind of
1:44files and folders we see. And in case
1:47our table if it is partitioned then what
1:50we see is we will see the data log in
1:53the similar way but instead of directly
1:55those uh park file we will see that
1:58partitioned folders. So here country is
2:01the partition column. These are the
2:03values and then again if there is
2:05further two-level partitioning then
2:08category is the partition column and
2:10clothing it is the actual partition or
2:12the value of that clothing category
2:14column and then we will see the actual
2:17park files. So that is how the folder
2:20structure is there for delta league and
2:23if the candidate is able to answer this
2:25correctly so that confirms that the
2:28candidate has actually gone to the
2:29location they know what kind of files
2:31and all are there.
2:34Now moving on to the next question there
2:36we ask like within the data log what
2:40kind of information is stored. So if I
2:43open any of the JSON file. So here we
2:46can see or what we shall tell is like
2:48each of the JSON it contains detailed
2:52information like what operation was
2:54performed. In this case it is the right
2:56operation and while performing the
2:58operation what other parameters were
3:00used which notebook was used which
3:03cluster was used everything in every
3:06information is stored and also how many
3:08files were impacted how many rows were
3:11written what was the size of the data
3:14that was written and along with that for
3:18each and every file. So in this case
3:20this was a new file that was written. So
3:23it is add otherwise if something is
3:25deleted it will show as a remove and for
3:28each file it contain like partition
3:31values, size and also it contains
3:34statistics. This states part it is very
3:37important because for each and every
3:40column it contains what is minimum value
3:43and what is maximum value. So those
3:46detail it contain for each and every
3:48column and this information it is very
3:51helpful when we retrieve the data.
3:54Now coming to the next question which is
3:57again a another practical question that
3:59I ask. Let's say if I'm fetching some
4:02data from a table and I apply a filter
4:05in in this case let's say if I mention
4:08select star from table where ID is
4:10greater than five then how does it
4:14utilize that filter internally to fetch
4:17the data efficiently.
4:19So the answer to that is uh because in
4:22data log the data is stored in terms of
4:25actual park file but it will not read
4:28all the file. What it will do is first
4:30it will come to these delta logs and
4:33from these logs it has this statistics
4:36for each and every file. For example in
4:38this case it knows that the uh in this
4:41particular file the value of ID is
4:44minimum is one maximum is three. And if
4:47I'm putting the condition that id is
4:49greater than five. So it knows from here
4:52that
4:53id greater than five values will not be
4:56there in this file. So it will not even
4:58check out this particular park file. So
5:01that is how based on this data logs it
5:04is able to identify which all files it
5:08shall go through to pick up that
5:09information. So then once it knows which
5:13files has the required columns based on
5:16that filter then it will go to those
5:18park file and within park also it will
5:20efficiently pick up only the required
5:23columns. So that is how internally
5:25things work in park. So that is how the
5:28filter works very efficiently in case of
5:31data tables.
5:33Now the next comes a theoretical
5:35question which is a generic question
5:37like tell me some of the features of
5:39Delta Lake. Then the features are that
5:42Delta Lake tables they support acid
5:45transactions. They support schema
5:47enforcement and evolution. We can even
5:50like automatically handle new column or
5:52deleted columns. It supports time
5:55travel. We can go back in time and fetch
5:57any previous version of the table. And
6:00it has very optimized metadata
6:02management which we just saw. And it
6:05supports unified batch and stream
6:08processing. Both kind of operations we
6:10can do on a single table. And it makes
6:13upsert, delete and other complex
6:15operations more efficient and uh those
6:19there we get to save lot of cost. So
6:22these are some of the features of delta
6:24league. Now the next follow-up question
6:27comes about these features. So let's say
6:30I ask like uh how does delta link it
6:33performs asset transaction or these
6:35absurd delete operation efficiently. So
6:38the answer how we can explain explain is
6:41let's say we have a table we want to
6:44delete some record and insert new data.
6:47So for new inserted data obviously it
6:49will create new file and for the data
6:51that we want to delete it will not
6:54rewrite those all those files or it will
6:57not delete those files immediately but
6:59it will create new file and in its log
7:03it will mention that the previous file
7:05shall be considered as removed and new
7:08file shall be considered as added. So
7:10without rewriting the entire data it
7:13will create new file and in the logs it
7:16maintain which file it shall refer and
7:18which file it shall not refer and using
7:21that log itself it can uh perform time
7:24travel that we can go back in time and
7:27then we can check previous version of
7:29the data. Now the next question comes
7:33about time travel like how exactly we
7:36can do time travel and what are the
7:38different options we can use while doing
7:41time travel
7:43and the answer to that is that we can
7:45use describe history to check what all
7:48versions are there for the given table
7:51and to perform the actual time travel
7:54what we can do is we get two option
7:56either using this version number column
7:59we can use like version is of and
8:03whatever version we specify from that
8:05previous version it will fetch the data
8:07in this case it is a select query second
8:10option we get let's say if we don't know
8:12the version and simply we want like I
8:15want the
8:16table where the uh date was like 18th of
8:21November so that till that time whatever
8:25data was there in the table that data I
8:28want in those cases we use time stamp is
8:31of. So whatever time stamp we specify
8:33that time whatever was the data in the
8:36table that will be returned. And here we
8:39can check out the data using the select
8:41query. And if we want to make this
8:44version as the latest version then
8:46simply we can do restore table operation
8:49and then the data will be restored from
8:51the time travel. So here is a s simple
8:55command. So this way we can restore the
8:57table.
8:59Now the next question comes like what is
9:02upsert operation and what is insert only
9:05merge operation. So what is upsert is
9:08like when we doing the merge operation
9:12when we compare based on some column if
9:15it is matching then we update the data
9:17if it is not matching it means it is new
9:20data then we insert the data. This is
9:23called as upsert operation. And if we
9:26simply remove this when matched let's
9:28say we don't want to update anything or
9:30we are expecting that the data that is
9:33incoming it will never be updated it
9:35will always be new only. So then we what
9:38we do is we do only when note match then
9:41we insert the data. If same data is
9:43coming in we are not expecting it to be
9:46updated. So we will not update it. So
9:49that is called as insert only merge.
9:53Now the next question comes about
9:55optimize and zorder. So what is
9:58optimize? It compacts small files and
10:01improve query performance. So let's say
10:04wherever the data is stored for our
10:06table, if the files are less than 1 GB.
10:09So whenever we run the optimize
10:11operation, it will combine the small
10:13files and it will create the files
10:15around size of 1 GB. So the size
10:18question can also be there like what is
10:20the size that it tries to create those
10:22table it is 1 GB and what is Z order. So
10:27when it is combining those small file or
10:30even if those are big file what it
10:33actually does it it colllocate related
10:35data. So in zorder command we provide a
10:39column name as well. So for example this
10:43is how we run a simple optimize command
10:46which combine the small uh files and it
10:50does it don't care about collocating any
10:52data but along with optimize if you
10:56specify any column let's say ID L is Z
11:00order then it what it will do is it will
11:04put the similar ids into same file for
11:08example let's say uh 0 to 100 ids it
11:11will put into one file then 100 to 200
11:13into another file. That's how it will
11:16put it. Some people they are confused.
11:18They think like think that for each ID
11:21it will create separate separate file.
11:23That is not the case. Uh it will put
11:26similar ids into same file and in zorder
11:30we can even specify more than one column
11:33as well. So in that case those based on
11:36those two column it will put similar
11:39data in in same file that is how it
11:42works.
11:44Now the next questions comes like as the
11:47data lake supports time travel so until
11:51when it keeps the history. So most of
11:54the people think that uh by default like
11:57it is 7 days. So after 7 days it is
11:59cleared out automatically.
12:02But in reality the data in our delta
12:05lake table it is never cleared out
12:07automatically unless we run any vacuum
12:10command. So if we don't specify any or
12:13specify or run any vacuum command then
12:16data is never cleared. But in recent uh
12:20datab bricks release what they have done
12:22is they are doing maintenance
12:24automatically. So in datab bricks
12:27environment specifically they run the
12:29vacuum command automatically after some
12:32certain duration then the data is
12:34cleared out but it is only when the
12:37command is run otherwise the data is not
12:39cleared out and to actually clear out
12:43the data we have this vacuum command and
12:46how much data it will clear uh so that
12:49that is controlled how that is also
12:52another question so that is are
12:55controlled using these two properties.
12:58Let's say log retention duration, how
13:01much of delta logs we want to retain and
13:05deleted file retention duration. So the
13:08actual data how much duration we want to
13:10retain. So in this case the park file
13:13the actual data it will retain let's say
13:15last 7 days and uh the other log file
13:20the delta logs that it stored those it
13:23will retain for 30 days. So for example
13:27if I run on a run a vacuum command on a
13:30table with these configurations. So what
13:33it will do is it doesn't mean that the
13:35data loaded earlier than 7 days will be
13:38deleted. What it means is in the table
13:42after writing the data if we do any
13:45delete operation or if we do any upsert
13:48operation or merge operation that time
13:50if some file are internally considered
13:53as removed in data log only those file
13:57it will delete. So that is again like
14:00some people I find they are confused
14:01they think they think like any data
14:04older than 7 days will be deleted that
14:06is not the case. If the files are
14:08required for the latest version of the
14:10table whether those are loaded years ago
14:14those will not be deleted when we run
14:16vacuum but if the file is considered
14:19deleted in latest version of data log
14:22then only it will be deleted. So that is
14:25where like uh this vacuum comes into
14:28picture and how we run vacuum it is
14:31simple like vacuum and then the table
14:33name and with vacuum there comes another
14:36option like we can do dry run also. So
14:40let's say this is also a question like
14:42how can we check what files will vacuum
14:45delete without actually running vacuum
14:47or deleting them. Then we specify this
14:50dry run operation. It will what it will
14:53do is it will actually not delete
14:55anything but it will simply put the list
14:57of files that it is it is going to
15:00delete if we run actual vacuum. So it is
15:03kind of a dry run with just provide the
15:05list.
15:07Now the next questions come about
15:09cloning. So what are the two types of
15:11cloning that we can use with Delta Lake?
15:14So the answer to that it is a shallow
15:18clone and deep clone. And the question
15:20again it is like what is difference
15:22between these two. So what cello clone
15:25does it it creates a quick copy that
15:28does not copy the actual data file. Uh
15:31it only created its own metadata but
15:34underlying data files it will refer to
15:37the original table wherein deep clone
15:39what it does it it create a totally
15:42separate metadata and it also copies
15:44over all the data files as well. So that
15:47it that it was what deep clone does.
15:51The next question about the same comes
15:53like when shall we use each of these. So
15:57if we want to do some quick testing or
16:00like try out some basic operation for a
16:03quick analysis then we shall create a
16:05shallow clone. But if we really want to
16:08create a copy of the table then we shall
16:10use deep clone. And uh also like we can
16:16I also ask like in case of shallow clone
16:18and deep clone if after cloning I delete
16:21the actual table then what happens? So
16:24then in case of shallow clone it will
16:26fail because the underlying data would
16:28have been deleted and deep clone it will
16:31work fine because the underlying data
16:33was copied fully and the same happens if
16:36you run vacuum as well. If you run
16:38vacuum then shalloc clone may fail if
16:41the underlying files are deleted but
16:43deep clone it will work fine. The next
16:46question it is like how exactly do we
16:49use these shallow and deep clone. So
16:51answer to that it is very simple. Let's
16:53say we do create or select table then
16:57shallow clone and then the original
16:59table. So it will take this table and
17:01create this new shallow clone of the
17:04table. In case of deep clone, we specify
17:07deep clone. That is how simple it is.
17:10Now the next question comes about
17:12partitioning and liquid clustering. So
17:15we can we normally ask like what is
17:17liquid clustering or what is the
17:19difference between partitioning and
17:21liquid clustering. So in case of
17:24partitioning it actually creates the
17:26folder or directories based on the
17:28provided column. And in case of liquid
17:31clustering what it does is it is similar
17:33to optimize and zorder. So it uh
17:37colllocates the similar data into same
17:40file but it don't create the actual
17:43directory and actual folder structure
17:45there. Now the next question comes how
17:48exactly do we use these two? So here is
17:51one example. So partitioning we specify
17:55partitioned by while creating the table
17:57or while writing the data. And in case
18:00of liquid clustering in instead of
18:03partitioned by we simply specify cluster
18:06by and the advantage of cluster by is
18:09that even after creating the table later
18:11on we can change the cluster columns but
18:14in case of partitioning we cannot change
18:17it. And why we can't change it? Because
18:21in case of partitioned it has actually
18:23created folders for each and every
18:26column. So it will not it will not be
18:28able to create the folders again. But in
18:31case of liquid clustering it simply
18:33rewrite these files into the same
18:35folder. So that is how it is able to do
18:38it. Now the follow-up question on liquid
18:40clustering is that does the clustering
18:44happens automatically while writing the
18:46data into table and the answer to that
18:49seems like yes but it is no while
18:52writing the data automatically it will
18:54not create those clusters. it will
18:57create or uh write those sim similar
19:00data into similar files only when we
19:02will run optimize command. That is when
19:05it will uh rewrite the data based on
19:08those clustering columns and that is
19:10where there will be some extra cost
19:12involved for when we use liquid
19:14clustering wherein in case of
19:17partitioning because the target data
19:19itself it is written in terms of
19:21folders. So while writing itself it will
19:24automatically write it write it into
19:26different different folders and no
19:28separate setup or step is required. So
19:31both has their pros and cons. And
19:34similarly
19:35partitioning we use when we have low
19:38cardality of the data. Low cardinality
19:41means like low or like less number of
19:43distinct values for the columns like
19:46year, country like that. And for
19:49clustering we normally use the high
19:51cardality columns like ID and other
19:54types of such columns.
19:57Now the next question or topic comes
19:59about change data feed. So we ask like
20:02what is change data feed? When shall we
20:05use it and how exactly we can use it. So
20:09what change data feed is? uh if we
20:11enable change data feed on any table
20:14then it provide a very detailed
20:17information about
20:19record level operation that are being
20:21performed. For example, this is our
20:25original table and on top of this table
20:27if one record we are updating one we are
20:30deleting and one um new record that we
20:34are inserting. So in change data feed uh
20:37it will say that this record is deleted
20:40this record is inserted and this record
20:43this is the data before changing it and
20:46this is the data after changing it. So
20:49detailed level change type and all the
20:52data and all the operation everything it
20:55provides in a very detailed logs manner.
20:59And when we enable change data feed then
21:03we ask like where exactly does it fetch
21:05that data or where exactly is this data
21:07stored. So along with delta log folder
21:11in the delta table it create another
21:14folder change data and that is when that
21:18is where in this park file it will store
21:21this information that is how internally
21:23it works. And the next question comes
21:26like how exactly we can use change data
21:29feed. So one option is that while
21:31creating the table itself we can specify
21:35this property delta dot enable change
21:38data feed equal to true then it will be
21:40enabled for this table or after creating
21:43the table later on also we can do alter
21:46table and then alter this property then
21:49also we can set it up. Now the next
21:52question comes let's say if we have
21:54created a table we inserted or did some
21:57operation then later on if we enable it
22:00then will it still uh showcase those
22:03changes in the CDF which was there
22:06before enabling the change data feed the
22:09answer to that it is no uh it start
22:12tracking those changes only after we
22:14enable this property so previous
22:16operations we can't fetch.
22:19So that covers all the questions on
22:21Delta Lake and I have created similar
22:24video on spark as well. You can go
22:26through that as well if you want and if
22:29you still have any doubts let me know
22:30and if you want to plan a mock interview
22:32with me the link is there in the
22:34description. That's all for now. Thank
22:37you. Bye-bye. Happy learning.

About this transcript

This page contains the full transcript of Delta Lake Interview Questions by DataBeli, generated from the public captions YouTube serves with the video. The transcript has 3,575 words across 498 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.