YT YouTube2Text

YouTube2Text → Transcripts → Azure + Databricks Interview Questions from Real Project Scenarios

Azure + Databricks Interview Questions from Real Project Scenarios — Transcript

by DataBeli · 7,882 words · 1,103 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Hi, I'm Narendra Kumar. I'm a senior
0:03data architect with 12 years of
0:04experience in IT. I'm a certified
0:07Microsoft solutions architect and a
0:09datab bricks MVP as well. Uh taking
0:12interviews of data engineers and data
0:14architects. It is part of my day-to-day
0:16work. And in this 40-minute video, we
0:19are going to cover 50 plus questions
0:21related to Azure and datab brick
0:23services as well as lot of project
0:26related scenarios. We will cover
0:28questions related to data injection,
0:31metadatriven framework, data bricks,
0:33data transformation, CI/CD, governance,
0:37email notification, security and data
0:39consumption as well. Uh after going
0:42through this video, you will be able to
0:44answer any questions in any of the
0:45interview related to Azure and data
0:48engineering projects. Uh before this
0:51video, I have created two other videos.
0:54one about a detailed course on Azure
0:56data factory and all the integrations
0:58with it and another one about a
1:01healthcare end toend Azure data
1:03engineering project. So I strongly
1:05recommend that you go through those
1:07videos so that these questions will seem
1:09more relatable and you will be able to
1:11understand them properly. And if you are
1:14already familiar with Azure services and
1:16you have worked on some hands-on project
1:19then you can go through this video
1:20directly as well. And in this video I'm
1:24going to show lot of ADF related real
1:26pipelines, real data factory notebooks
1:30and other real Azure related elements.
1:32All those artifacts are taken from this
1:35project and all those artifacts are
1:37available on GitHub for free. And to
1:40take it a step further, I have created a
1:43PDF document as well for all the
1:45questions that I'm going to cover in
1:47this video along with screenshots and
1:50relevant answers so that you can use
1:52this PDF as a revision document before
1:55going to any of the interviews.
1:58So the link for this PDF is also
2:00available in the description of this
2:01video. And on my channel you will find
2:04lot of latest stuff related to data
2:07engineering. Lot of projects are there.
2:09Other interview related videos are
2:11there. So feel free to check them out if
2:13you are interested and if you are
2:16interested to stay upto date what is
2:17going on in the market about data
2:19engineering and my videos. So don't
2:21forget to hit the subscribe button. So
2:24without any delay let's get started.
2:28Let's start with some Azure data factory
2:30related questions. So the first question
2:33can be like what is the difference
2:34between a linked service and a data set
2:36in data factory. So the answer is linked
2:39service it provides the ability to
2:41connect with any external service
2:43outside data factory. It can be a source
2:46or any other Azure service. Here we
2:48mainly provide connection details like
2:50their URL or server name and
2:53authentication related details so that
2:55we get the ability to connect with
2:57anything external outside data factory
3:00wherein data set it is the next step
3:03after linked service where
3:06we specify the exact uh database or
3:09table name or in case of files we
3:11specify the exact file name folder name
3:13etc.
3:15Now the next question it can be like
3:16what is the difference between debugging
3:19a pipeline and clicking on trigger now.
3:22So we can say debug as the name suggest
3:24it is about like debugging uh any
3:27pipeline while doing the development
3:29where the pipeline runs end to end uh in
3:32the current branch whatever we have
3:34selected wherein trigger now it actually
3:37triggers the published uh copy of the
3:39pipeline and another difference is that
3:42in debug if we are using a for each kind
3:45of loop so there things will run
3:47sequentially only wherein in trigger now
3:50from the published pipeline it will be
3:52able to actually trigger multiple things
3:54in parallel and next jumping to some
3:57active activity related questions. So
4:01first one it can be let's say uh in a
4:03lookup activity if you are reading some
4:05data from a SQL DB then how do you use
4:09like first row only option when do you
4:11enable it when do you disable it? This
4:13question we check so that we know that
4:15actually the person has really used this
4:17lookup or not. So the answer to that is
4:21if we enable the first row only then it
4:24will always pick up the first row itself
4:26directly as a uh JSON object note as an
4:30array. So that way whenever we have to
4:33access any element from this then we can
4:36directly access it like the activity dot
4:39output dot first row then the the column
4:42name. So we don't have to parse any
4:44array. But uh in any case if we are
4:47expecting a multiple rows as output and
4:50if you want to use the array actually uh
4:53then we keep it disabled so that in the
4:55next activity like we can directly read
4:58the entire array itself just by using
5:01output dot value and that entire array
5:04can be like used for any further
5:05processing. And the next question it
5:08could it could be about for each
5:09activity that how do we pass any element
5:13to the for each and how do we use it
5:15inside the for each loop for multiple
5:18elements. So for passing part we can say
5:20that in the setting section uh in the
5:23items we pass the output of any previous
5:25activity or we can pass any other array
5:28as well that we can pass and inside the
5:31for each if we want to access the
5:33element that we have passed. So that is
5:36directly available is at the rate item.
5:39So at the rate item dot whatever we want
5:41to use. So in the array whatever we had
5:44passed we we are able to use it inside.
5:47Now the next question it could be about
5:49fail activity like what is the fail
5:51activity and when we may need it. So
5:54fail activity it is a simple activity
5:56which uh just set the overall status of
5:59the pipeline as failure and there we can
6:02pass any failure message and error code
6:05as well. So normally we use this
6:07activity when we want to do something
6:10based on the failure of the previous
6:12activities or pipelines. So but if we
6:15normally do any operation like adding
6:18information into audit table or
6:20something then overall status will be
6:22considered as success because the last
6:24script was successful even though it was
6:26based on failure of previous script or
6:29previous activity. So to set overall
6:32status as failure we can manually call
6:35this activity so that specifically the
6:37overall status of the pipeline is set as
6:39failed.
6:41Uh the next question it can be about
6:43switch activity like what is the switch
6:45activity and how practically you have
6:47used it. So we can say like switch in
6:50the switch activity we can pass an
6:52expression or any variable. So depending
6:55on its value it can have multiple case
6:58statements. So depending on the value we
7:01can call different set of activities and
7:04we can say we have used it for the
7:07deciding on the source systems. So let's
7:09say if we are passing source system as
7:11input then for postress we can call a
7:14separate copy data for ADS we can call a
7:17separate one and in future if we have
7:19more sources we can directly add in the
7:21switch statement itself because we need
7:23separate copy data for each and every
7:25source. So that is how we have used it.
7:28And the next question it is very
7:30important. So it can be like how can we
7:33pass any parameter value from a adf
7:36pipeline to the datab bricks notebook
7:38and then use it in the notebook. So for
7:41that what we do is in our notebook
7:43activity uh we get this base parameters
7:46option. So there we can pass any number
7:48of parameters from ADF and then on datab
7:52brick side we create widgets to receive
7:54the values of those parameters and after
7:57reading the values from widgets then we
7:59can use them inside the notebook.
8:01however we want. And the next question,
8:04this one I would say like almost more
8:06than 50% people don't know the answer.
8:09So that is uh how can we pass any output
8:12parameter or any return value from datab
8:15bricks notebook back to the pipeline and
8:17then use it there. So the answer to that
8:20is like we have this command dbutil
8:23notebook.exit exit there whatever value
8:26we pass as a string or as a JSON. So
8:28that is what is returned as a output
8:31from the notebook and then inside the
8:33pipeline we can use that value as the
8:36output value from the notebook. So in
8:39the next activities if we want to use it
8:41we can use like that activity dot output
8:44dot run output that way we receive the
8:47value back. Now the next question it can
8:50be like what is the difference between a
8:52pipeline parameter and a global
8:54parameter. So pipeline parameter those
8:57that is something we create for a
9:00particular pipeline and their values we
9:02can pass through another pipeline or
9:05even through trigger as well and the
9:07scope of those parameters are within
9:09that pipeline. We can use them within
9:11that pipeline however we want not
9:14outside that pipeline. Whereas global
9:17parameters so that is something we
9:19create in the manage section separately
9:21which are independent of the any
9:24pipeline. So there whatever parameter we
9:27create those we can use them in any of
9:29the pipeline and these parameters are
9:32kind of global. So we define them once
9:34in ADF and use them everywhere.
9:37And the next question could be about the
9:39same thing that how do we use pipeline
9:42parameter or global parameter in a
9:44pipeline. So the answer is pipeline
9:47parameter we can directly access them as
9:49pipeline dotparameters dot the parameter
9:51name. And for global parameters we use
9:55like pipeline dot global parameters dot
9:58parameter name. And then the next
10:00question it can be about dynamic
10:02expressions that have you used any
10:04dynamic expression in ADF and uh what
10:07kind of things you have used or what all
10:09functions you have used. So there
10:11depending on your real experience and
10:13what things you have tried out you can
10:15answer like we can say like these are
10:17some of the common things that we can
10:19mention that I have used if statement a
10:22greater statement uh that checks like
10:25anything values greater than any any
10:28value or not then we can mention we I
10:30have used concat function to concat
10:33multiple strings and we can say we have
10:35used coales function to like pick up
10:39anything is null or not else like
10:41picking up a blank value or some other
10:43value. So these are some of the things
10:46you can mention and depending on your
10:47own hands-on you can mention uh some
10:50dynamic expressions that you have
10:52created.
10:53The next question it can be like what is
10:55the difference between a cancel and
10:57cancel recursive when you are debugging
11:00something. So these kind of questions
11:02they really confirm whether you have
11:04really hands-on experience or not. So
11:07cancel it will cancel only the current
11:09pipeline. If there is a inner pipeline
11:11triggered by this that will still keep
11:13running at the back end wherein in
11:16cancel recursive it will cancel any
11:18inner pipelines as well. Whatever has
11:21been triggered by this pipeline run. The
11:24next one it can be like what is a
11:25pipeline run ID and how can we access it
11:28dynamically within the expressions or
11:31parameters. So pipeline run ID it is the
11:34unique ID given to each run of the
11:36pipeline. Every time we run it, a new
11:39pipeline run ID is generated and within
11:42our parameters or dynamic expressions
11:45directly we can access it using either
11:47pipeline.tr run id and the next question
11:51it could be a small scenario based one.
11:53Let's say if we have multiple activities
11:55in the pipeline and let's say third or
11:57fourth activity failed then we have
12:00fixed the issue and if you want to rerun
12:02from that failed activity again then how
12:04can we do that? So for that we can say
12:07that in the monitor section we get uh in
12:10the run of the pipeline we get these
12:12options like either we can run uh rerun
12:14the entire pipeline or directly we can
12:17run uh from the failed activity or any
12:20selected activity as well and then next
12:23comes the very standard question like
12:25what are the types of triggers that we
12:27can create in ADF. So there we can say
12:30that we can have four types of triggers.
12:33So first one is schedulebased trigger
12:35where we specify that at what time we
12:38want to trigger something on a with a
12:40particular window. Let's say if we want
12:42to trigger something every day, every
12:44week, every month at some fixed time,
12:47then we use this as schedule trigger. If
12:50we want to something run something over
12:53a fixed window of time without
12:55specifying the exact time. Let's say if
12:57we want to run something every 15
13:00minutes, every 2 3 hours, every 1 2
13:03months without worrying what time it
13:05shall run, then we use the tumbling
13:07window. And if we want to trigger it
13:10based on the events from a storage like
13:12whenever some file is created or if a
13:16file is deleted uh then we use the uh
13:19storage event based trigger and then the
13:22last one is custom events. So if we want
13:25more decoupled scenarios wherein if we
13:28create an external uh like event grid
13:30there if we get a new event and based on
13:34that if you want to trigger something uh
13:36then like we can use this custom events
13:39trigger as well and next one also is a
13:42very standard question like what are the
13:44types of integration runtimes that we
13:46can use in ADF. So in total there are
13:49three types. So one is self-hosted
13:51integration runtime. So we use this one
13:54when we want to connect something which
13:56is on premise or if it if uh the service
13:59is on private network then we set up our
14:02own self-hosted integration runtime to
14:04connect with that and if the service we
14:07want to connect it is on Azure directly
14:10then we use Azure integration runtime
14:13and then we have a autoresolve
14:14integration runtime as well which when
14:17we use so it will automatically decide
14:19like which integration runtime it can
14:21use so that we don't have to uh select
14:23and choose. So these were all the core
14:26ADF related questions. Now let's jump to
14:29some scenario- based questions related
14:31to metadatriven framework. So the first
14:34question can be the very generic that
14:36have you worked on any metadatriven
14:38framework? Can you explain it at a very
14:40high level? Or in other words, it can be
14:43like uh have you like ingested multiple
14:46tables in let's say hundreds. If you
14:48have to ingest hundreds of tables from
14:50different sources then how will you
14:52build a framework for that. So for that
14:55we can say that we have some uh control
14:58tables in our se separate SQL DB and
15:01based on those tables we have created
15:03some reusable pipelines and notebooks
15:06which can be used to ingest the data
15:08from any data from any number of tables
15:11from the same sources or similar type of
15:14sources. There we have separate tables.
15:17Let's say one table with the table
15:19related configurations like uh the
15:22source configuration source schema or
15:24path table name or folder and target
15:27related target table related information
15:30everything the type of load we want
15:33watermark columns merge keys these kind
15:36of columns we have and then we have a
15:39watermark table where we control like
15:42which where we mention the watermark
15:44value for incremental load and we have
15:46audit table where we log each and
15:49everything in detail. For each table we
15:51me we log the run ID, table ID, stage,
15:55status, number of records, everything we
15:57log in a audit table and using these
16:01three tables we drive or we have built
16:04reusable
16:06pipelines in ADF where we read the
16:09configurations using lookup activities.
16:11Then we have prepared our copy data
16:14activity and the notebook activity. Copy
16:17data for source to landing notebooks for
16:19landing to bronze and silver. All of
16:21them are parameterized so that these can
16:24be used for each and every uh table that
16:27that is coming in and source to landing
16:30piece because that is uh we need to
16:32control it for different sources. So we
16:35have a clone of that one. So one copy
16:38data can be used for any table or any
16:40file injection from single source. We
16:43have multiple such copy data. But the
16:45notebook part that is like completely
16:48reusable. So whether it is a file-based
16:51source or database based source. So for
16:54everything we have a single notebook
16:56that is able to ingest every type of
16:58data load and our pipelines they are
17:01driven by our metadata tables and they
17:05log everything. they update the
17:07watermarks end to end. So that is how
17:09like this framework works end to end.
17:12And the next question it can be a
17:14follow-up or something specific. Let's
17:16say if you are doing incremental load uh
17:19then how do you make sure that in in
17:22every injection it is reading only
17:24incremental data. So the answer to that
17:26is like we have a watermark table where
17:29we track the track the
17:33current injection timestamp until when
17:35the data has been loaded and that value
17:38we read inside our pipelines using
17:41lookup and then we specify that in our
17:44queries while reading the data. So if
17:46the load type is incremental then we use
17:49the watermark column and watermark value
17:51to apply a wear close so that it read
17:54only incremental data every time. And
17:56the next question it could be similar
17:58about file based source that in your
18:01source if in the same folder there are
18:03new files keep coming so how do you make
18:06sure to read only the new files from the
18:08source. So there again we can say that
18:10we have a watermark table where we store
18:14like until what time the files has been
18:16loaded and whenever we are bringing new
18:19data that time in our copy data activity
18:23we have this start time for the last
18:26modified. So there we specify the value
18:29so that whatever files has been modified
18:31after that time step only those files
18:34are picked up. And the next scenario
18:36based question can be let's say if your
18:38pipeline is running so there are three
18:40phases landing bronze silver if your
18:43pipeline fails in between in any of
18:45these two stages let's say landing or
18:47bronze in case of incremental load then
18:50how do you make sure that in the next
18:52run we don't lose any data we are
18:56reading that incremental data again so
18:59uh the answer to that is so in case of
19:01incremental data we update the watermark
19:05value only after the data has been
19:07written successfully into our silver
19:09layer. If it fails before any of these
19:13steps like in silver or before that we
19:15don't update the watermark value. So if
19:18we don't update it in the next run
19:20automatically because the value was not
19:22updated it will pick up the data from
19:25previous value only. It will not skip
19:27the data which was loaded previously and
19:30only that new data it will read and then
19:32load end to end again and also to make
19:35sure that uh whatever data was landed as
19:39part of previous run it is not loaded
19:41again we are keeping track of a landing
19:44time stamp that we pass uh throughout
19:47the layer like landing branch and
19:49silver. So whatever data we land in our
19:52landing folder we create a timestamp
19:55folder there and only that folder we
19:57read in bronze only that data we read in
20:00silver. So that is how our pipelines are
20:03errorprone uh sorry error proof. Now the
20:07next question can be like how are you
20:09creating your notebook so that they can
20:12be like same notebook can be used for
20:14file based sources and any RDBMS or
20:17database related sources. So there we
20:19can say that for database related
20:22sources we land the data always in the
20:24park format because it is very compact.
20:27For file based sources we land the data
20:29in original format CSV or JSON whatever
20:32it is coming in. And then in our landing
20:35to branch notebook we uh we have a
20:37parameter as well source file format
20:40which comes up from our metadata tables.
20:42Based on that file format we uh provide
20:45the spark read options accordingly and
20:48then rest of the steps are common. So
20:51that is how the same notebook works for
20:53any file format the data is coming in.
20:56And the next question can be like what
20:57kind of audit columns you are
20:59maintaining into your bronze, silver and
21:01gold tables. So there we can say that we
21:04are we mainly maintain three types of
21:07columns. One is landing timestamp so
21:09that we can track the folder where the
21:12data is landed and when it is landing. A
21:15second one is insert timestamp so that
21:18we can know when each and every row is
21:20inserted into our tables. And third one
21:23for silver layer we can say we have a
21:25update timestamp as well so that if same
21:28record is updated then like we update
21:30this time stamp so that we can know when
21:32the record has been updated. The next
21:35question it can be like how do you use a
21:37merge statement or how do you make sure
21:39that while writing the data into silver
21:42layer there are no duplicates. So the
21:44answer to that is we use a merge
21:46statement based on some primary keys
21:49which are defined in the table. So we
21:51merge into our target table by reading
21:53the data from source. First we check our
21:56condition based on the primary keys. If
21:59the data is already there in the target
22:01then we update all the values except
22:04insert timestamp because we don't we
22:06want to retain original all the other
22:08columns we update and if based on those
22:11primary keys the data is not present
22:13that means that is new data. So then
22:16then only we insert that as a new data
22:18into our target table. So that is how we
22:21use a merge statement. So to this way we
22:24don't end up creating any duplicates
22:26into our silver layer. Uh the next
22:29question it can be let's say if you have
22:31built your network uh if you have built
22:33your framework sorry then uh if you have
22:37to ingest one more table from the same
22:39source then what are the steps that you
22:42follow? So then we can say for that one
22:44we just have to make entry into our
22:46metadata tables. So from the same source
22:50we just add another row into our tables
22:53for the table configuration and
22:55watermark values if required and that's
22:58it. So just by adding that metadata
23:00entry the table is picked up
23:02automatically and ingested end to end
23:04and we can mention that we have a is
23:06active column as well using which we can
23:09control as well which table we want to
23:11run and which table we can skip while
23:13running. And the next question can be
23:15like uh if in your framework if you have
23:18to bring in a new source then what are
23:21the steps that you will follow. So we
23:24can say if the new source if it is of of
23:26a different type than currently whatever
23:29we have in the system let's say if we
23:30have a postgrace and adls and if you
23:33want to ingest the data from Oracle then
23:35we need a separate linked service
23:37separate data set and copy data activity
23:40which we need to add in our switch
23:41statement so that it can land the data
23:44from source to landing from for that
23:46source. So all these connections we have
23:49to configure in linked service. Then
23:51create a new data set, create a new copy
23:53data and add in this switch statement
23:56for source to landing injection. Apart
23:59from this uh like nothing else is
24:01required because our notebooks are
24:03reusable. So those can be used to ingest
24:06the data from any of the source. And
24:09then if the source is of same type let's
24:12say we are ingesting the data from
24:14postgress already we want to connect
24:17another postgress and bring the data
24:19from there then we have two options
24:21either we can create a cloned linked
24:24service with the different connection
24:26details or in the same linked service we
24:28can parameterize even the connection
24:31details as well like the URL username
24:34password those things also we can
24:35parameterize and put in a config table
24:38and then even use the same copy data,
24:40same data set linked service for a new
24:43source of same type.
24:46Now let's cover some of the questions
24:48related to gold layer. So there the
24:51first question it can be like have you
24:52worked on or used any STM documents and
24:56what kind of details are there in an STM
24:58document. So there we can say that for
25:01every table like we have got the
25:04documents from the client side or we can
25:06say like we prepared them after working
25:08with the end users where we captured the
25:11details like each and every column their
25:14data type from which source table it is
25:16coming. What is the transformation logic
25:19that is required? Which joins and
25:21filters are required? What will be the
25:24exact table name? Uh so what will be the
25:27load type? these with all these kind of
25:29detailed information we prepare in a STM
25:32document and the next question can be
25:35like uh have you generated any notebooks
25:38in datab bricks using AI. So we can say
25:41yes for gold layer specifically we have
25:43generated lot of notebooks using AI
25:46wherein first we go or we prepared the
25:48HTTM documents then we provided those
25:52documents directly to Genie code and
25:54asked it to create asked it to create
25:57notebook for us. So it created the very
25:59initial version of the notebooks with
26:01really good details. So those notebooks
26:04were created automatically and then we
26:07reviewed the notebooks. We went through
26:09the logic again to confirm and then
26:11refine the notebooks as and when
26:13required. And then the next question can
26:15be like how are you using a framework to
26:19like run the multiple notebooks in the
26:21gold layer. So there we can say like we
26:24have the same metadata tables that we
26:27can use for both silver and gold layer.
26:30So for gold layer as well we specify our
26:32table names and other details into our
26:34metadata DB itself. From there we'll
26:38read the list using lookup activity
26:40iterate over it in a for each and then
26:43call the notebook using a notebook
26:45activity based on the table name
26:47dynamically. So whatever notebook is
26:49present in the standard folder that is
26:51called automatically when we pass the
26:54name of the notebook dynamically uh
26:57through an expression
26:59and whatever notebook we are calling. So
27:02before and after that we are even
27:04logging our status into our OD table as
27:06well so that we can track the progress
27:08and status
27:10and then the next question it can be
27:12like are you doing full load or
27:14incremental load for your goal tables.
27:16So there we can say we have for small
27:19tables we always do full load so that we
27:22don't have to worry about incremental
27:24logic. So full load is easy and faster.
27:27For big tables, we can say that we have
27:29a driving table. From that table, we
27:32read the data based on some incremental
27:34logic which we put in the watermark
27:37table. Read the watermark. We read the
27:39incremental data. So that is the driving
27:41table and the big table. For other
27:43tables, we we read the data completely
27:46and join the tables. That is how we do
27:48like incremental load for some of the
27:50tables. So now next let's cover some of
27:53the questions related to failure
27:55handling. So the question it can be like
27:59what kind of options are you using in
28:00your pipelines to handle failures and
28:04make your pipelines more robust. So
28:06there we can say like we are using
28:08various options like for each and every
28:11activity we are configuring the retry
28:13options that if there is some temporary
28:16or a a quick failure that so it can
28:19retry the same activity again. And from
28:22overall design perspective we have kept
28:25all the uh data related operations into
28:27a separate pipeline so that even if a
28:30single activity fails in the pipeline.
28:33Uh then in the outer pipeline we are
28:35able to capture it and log it into
28:37properly. And then in terms of auditing
28:39and logging, we are logging the
28:41information in very detail including the
28:44ADF run ID so that we can trace back the
28:47things and table ID, status, start time,
28:50end time, everything we are logging and
28:53we have set up a daily summary status
28:55email as well so that directly we can
28:57know if something has been success,
28:59everything is success or something has
29:01failed. For the failed entries directly
29:04we can check out the table ID and their
29:06run ID directly go to ADF and check out
29:09the failure. And here the next question
29:11can be like what kind of options do we
29:13get at the activity level to handle
29:15failures. So there we can say we have
29:18three options for timeout for very long
29:21running we can specify a timeout so that
29:24we don't have to wait indefinitely and
29:27then we have a number of retries in case
29:29of failure how many times we want to
29:31retry and retry interval as well how
29:34much gap we want to provide between
29:37different runs and the next question can
29:39be like how exactly you can find the
29:42error message or what is wrong with the
29:44pipeline. So there we can say that in
29:47our notifications we get the run id of
29:49the pipeline. Using that run id directly
29:52we can go to monitor section find that
29:55particular run ID and then in the
29:57activity output we can directly check
30:00out the error message. From there we can
30:02understand what could be wrong and then
30:04the next question can be like after
30:06fixing any problem do you rerun your
30:09pipeline entirely or do you rerun from
30:11the failed activity. So the answer to
30:14that can be like it depends on the
30:16scenario. If some ADF activity related
30:19failure is there for example copy data
30:21or something then if you're applying a
30:24fix we need to publish the fix or save
30:26it and then again run it from the start.
30:29But if the failure is related to some
30:31notebook then like we can fix the issue
30:34inside that notebook and we can rerun
30:36from the failed activity as well. And it
30:39also depends on like how much f how much
30:42time it will take to rerun from start or
30:44from failed and what kind of impact it
30:47can have like if it can create duplicate
30:49data by rerunning entire pipeline then
30:52we avoid it and if it is just a quick
30:54run which uh doesn't create any
30:56duplicate data or something like that
30:58then we can run from the failed activity
31:00as well. Uh next moving on to some
31:03monitoring related questions and the
31:05question can be like uh have you
31:08monitored the uh copy data activity and
31:11there what kind of details we can figure
31:13out from the monitoring section. So
31:15there we can say that for moni for copy
31:17data we get a specific details section
31:20in the monitoring section where we can
31:23monitor how many uh rows has been read
31:25and written how how much data volume has
31:28been read and written how many parallel
31:30connections were used what was the speed
31:32of data duration of speed so all these
31:36details we get in the output of the copy
31:38data and the next question it can be
31:40let's say for lookup kind of activities
31:43or for script kind of activities it is
31:45if we are using dynamic expressions to
31:47prepare those queries. How can we know
31:50that at runtime what exact query was
31:52used. So there we can say in the
31:55monitoring section we can check out the
31:57input of that particular activity. There
32:00we can check the exact query that has
32:02been prepared using the dynamic
32:04expression. So these kind of statements
32:06are available for lookup uh script kind
32:09of activities. And next let's cover some
32:12of the questions related to email
32:14notification. So the generic question
32:16can be like which service are you using
32:18for email notification. So there we can
32:21say that we have built our own logic
32:23apps workflow and that we are calling it
32:26in the ADF using a web activity. And
32:29then the next question can be let's say
32:31if you have to send a consolidated email
32:34for all your silver and gold uh table
32:37injections then how can you send it? So
32:40there we can say that we have an audit
32:42table wherein we insert the status and
32:45record count for each and every table
32:47for silver and gold and there we use the
32:49same ADF run ID for logging. So using
32:52that run ID we pick up all the rows and
32:55then include that in a single email that
32:58is how we prep we prepare and send a
33:00consolidated email. Then the next
33:02question can be like can you explain the
33:04design of your email notification
33:06pipeline or it can be like uh how how
33:10have you created a reusable pipeline for
33:12email notification. There we can say
33:14that our pipeline it takes just af run
33:17id based on that run ID it fetches all
33:20the information from our audit table and
33:23also it check overall status whether
33:25there is a single at least one failure
33:27or everything is successful. Based on
33:30that we prepare email subject, header,
33:33footer and a separate status table, a
33:36separate a set variable in separate set
33:39variable activities and then we prepare
33:42a consolidated
33:44body that we can send to logic apps and
33:47a follow-up question can be like what
33:49kind of parameterization you have done
33:51for logic apps. So there we can say that
33:54we have parameterized the subject email
33:57recipient and body part we are sending
34:00it in three different parameters. So we
34:03have a header where we send the highle
34:06information. Then we have a status table
34:09and then we have a footer. So using
34:12these variables we are able to
34:14parameterize everything while sending
34:16the data to logic apps. And then the
34:19next question can be like have you
34:20worked on the logic app site and there
34:22what kind of activities you are using.
34:26So there we can say yes I have worked
34:28and uh there mainly like we are using
34:30four activities. We are first one is
34:33about a trigger. We are using a HTTP
34:35based trigger so that it can be
34:37triggered from a web activity from ADF.
34:40Then the table we are passing that is
34:43available as a JSON array. So we convert
34:46the table uh using a using a create HTML
34:50activity. Then we add border using a
34:53compose activity and then we send the
34:56email which is parameterized using a
34:58send email activity. And next let's
35:01cover some security related questions.
35:03So we can say like how are you storing
35:06your credentials securely in your
35:08environment. So there we can say that we
35:10have configured Azure key volt and
35:13whatever passwords or credentials we
35:16need to store we store them in the key
35:18volt as secrets and then we have created
35:21a linked service for key volt in our ADF
35:24and then those secrets we read them
35:27through the key volt here by selecting
35:29the linked service then the secret name
35:31we are able to read the secret and use
35:33directly in the uh linked services
35:36without without exposing the actual
35:40password or actual credential in the
35:42linked service or on ADF site. The next
35:45question it can be about connecting
35:47ADLS. So it can be like uh what kind of
35:50authentication you are using to connect
35:53with ADLS and how can we set up a
35:55connection with ADLS in Azure data
35:57factory. So there we can say that we are
35:59using system assigned managed identity
36:02and for the managed identity of Azure
36:05data factory we have given the store
36:08storage blob data contributor access on
36:11that storage account and if you provide
36:13that access then we don't have to
36:15provide any credential for connecting
36:17the connection is automatically using
36:20that managed identity and the next
36:23question which is very very important
36:25question that if you have a new storage
36:27account then how can you connect that
36:30storage account in datab bricks uh and
36:33what all steps you need to follow how
36:35exactly you can connect it and read
36:37write some data. So there we can say
36:39that for that connection we are using
36:41the unity catalog access connector. So
36:45using that access connector step one is
36:47that we have created a credential in
36:49unity catalog using that same managed
36:53identity of that connector.
36:55Then is the next step we can say that
36:58for that connector managed identity we
37:00have given the storage blob data
37:03contributor access on the ads side. Then
37:06as a step three of using that
37:09credential, we shall create an external
37:12location in the Unity catalog which uses
37:15that uses that credential and then
37:18connect to our ADLS account. And as a
37:22final steps to actually read and write
37:24files from the ADLS we can say we have
37:27created a volume using that external
37:30location
37:32and in that volume directly also we can
37:34browse all the files and folders and
37:36using that path we are able to read and
37:38write any data to the ADLS account. Then
37:42now let's cover some of the questions
37:44related to CI/CD process. So the
37:47question can be on Azure data factory
37:49side how do you follow CI/CD process. So
37:52we can say whenever we want to work on
37:55the main work on any new feature we
37:58create a new branch from the main branch
38:01and in that branch we make all our
38:03changes whatever we want to make the
38:05changes we apply those changes we test
38:08them and once we are ready then we
38:11create a pull request using which we can
38:14take our changes to the main branch that
38:16pull request it is reviewed by a senior
38:19architect or any other senior your data
38:22engineer after review the pull request
38:24is merged into the main branch and then
38:28the changes are published into the final
38:30published branch. So that is how we
38:32follow the CI/CD process on ADF side.
38:36And now the next question can be like
38:38how do you follow CI/CD process on Azure
38:40datab brick side. So there we can say
38:43that our final notebooks those reside in
38:45the main workspace folder which is
38:47outside all the user related folders. In
38:51that folder we have connected it through
38:53g and created a g folder. Whenever we
38:56want to work on any new feature then in
39:00our individual users folder we clone the
39:03g folder. Uh there we create a new
39:06branch out of the main branch. In that
39:09branch we work on all the changes. Then
39:12we uh push our changes to our feature
39:15branch. Create a pull request. That pull
39:18request is again reviewed by another
39:20architect level person. And then the
39:22changes are merged to the main branch.
39:25After the changes are merged to the main
39:27branch on the git side or devops side.
39:30Then in the final location again like we
39:34go to the g folder. there we pull all
39:37the latest changes which are available
39:40on the g side and then those changes are
39:43available for to be used by any
39:45pipeline. So that is how we follow this
39:47CI/CD process from development
39:49perspective
39:51and then next question can be like how
39:53the how your changes are deployed from
39:56one environment to another and what what
39:59kind of things are on your side that you
40:01take care. So there we can say that for
40:03CI/CD process we have a DevOps engineer
40:06in the team. They have built the
40:08deployment pipeline which can take the
40:10changes from one workspace and then
40:12deploy them in another workspace. On our
40:15side the way we are creating our
40:17pipelines and notebooks are uh so that
40:20those can be used in any of the
40:22environment without any changes. There
40:24we can say that we are using global kind
40:26of parameters
40:29where we need to specify the value at
40:31one place only what kind of environment
40:33it is and then everything is
40:35parameterized so that whatever
40:38environment it is the same code and same
40:40same pipeline work in both the
40:42environments.
40:44Now next let's cover some of the
40:46questions related to data consumption
40:48side. So there we can ask like uh what
40:52are the data consumption related
40:54services that you have worked on. So we
40:56can say that I have worked on uh datab
40:59bricks dashboard which is called as AIBI
41:01dashboard as well. I have created genie
41:04agent as well and datab bricks one also
41:06it is used by default for any data
41:08consumption.
41:10Uh then the next question specific to
41:12dashboard can be like how are you
41:14creating a dashboard in your existing
41:16system. So there we can say that the
41:18initial draft of the dashboard we are
41:20creating using Genie code. There we are
41:23specifying the tables that we are using
41:25and the main KPIs that it need to build
41:28at a very high level. After that it is
41:31able to provide a draft version of the
41:33dashboard which is really good and that
41:36dashboard we can change any details that
41:38we want. For example whatever widgets
41:41are there. So we can change their uh
41:44visualization type, their axis, their
41:47details. We can change their size, even
41:50delete them. In the data section, we can
41:52go ahead and make changes to any queries
41:55which are being used. So that way we can
41:57customize our dashboard depending on the
42:00need. And then finally publish the
42:02dashboard. So that is how like we are
42:04working on a dashboard.
42:07And then next question it can be about
42:09genie agent. So normally this is
42:10something actually I would say very few
42:12people know nowadays but it is very easy
42:15option. So if somebody ask you have you
42:17created any genie agent or how do you
42:19create an agent. So it is very simple we
42:22can say that we go to genie agent
42:24section and we click on new. There we
42:27select the tables that we want to
42:29connect or metric views whatever
42:30artifacts we want to build on it and
42:33then we click on create.
42:36So just like that using only those steps
42:39directly we can create a very initial
42:41version of the uh genie agent and the
42:45next question it can also be like uh
42:47something basic like like what exactly
42:50is a genie agent and how it is used. So
42:53there we can say that in Genie agent we
42:56provide some artifacts like tables,
42:59metric views, materialized views all
43:02those things we can provide as sources
43:05and then it has ready to be used
43:07capability to provide any answer based
43:10on natural language question. So the
43:12customers they can ask their question
43:14directly in plain English instead of a
43:16SQL and they will get their answer
43:18directly and a nice visualization as
43:21well in the answer ready to be used.
43:24And the next question related to Genie
43:26agent can be like what kind of things
43:29you can customize in a genie agent. So
43:32there we can say that we can provide its
43:34name, we can change its description, we
43:37can provide list of common questions so
43:40that those appear for any new user. We
43:42can even provide a thumbnail or a logo
43:45for that. And we can uh if you want to
43:48customize the responses more then we can
43:51set up some list of instructions for
43:53that. We can give it some example
43:55queries
43:57and to validate its response we can add
43:59some benchmarks as well. and then
44:02confirm the output whether it is meeting
44:04our expectations or not and then the
44:07next question can be like are you aware
44:09about datab bricks gen1
44:11like what it can do or what are its
44:13capabilities so we can say yes gen1 it
44:17is a dedicated separate UI from datab
44:20brick side which is specially designed
44:23for business users where they don't need
44:25to run any query or see any technical
44:27information they can directly ask any
44:30question about any of the data wherever
44:32the data it is genie will figure it out
44:35and just directly provide the response
44:37in natural language along with a good
44:39visualization
44:41and then next question it can be like in
44:44datab bricks genie related to genie what
44:46all services you are aware of and what
44:49all services you have used so there we
44:52can say that I have mainly used three
44:54services related to genie so genie code
44:58for writing any code or modifying any
45:00notebook books. I have also created
45:02genie agents uh to like provide any
45:05natural language based queries which
45:08users can ask based on the curated set
45:10of data sets or tables that we are
45:13providing. And then third one is Genie 1
45:16that is a dedicated UI for business
45:18users which is configured automatically
45:21and it provide good response based on
45:23wherever our data is landing. So these
45:27were all the questions that I wanted to
45:29cover here and here we have covered all
45:31the questions related to data injection
45:34metadatriven framework transformation
45:38CI/CD governance email notification
45:41security and data consumption as well.
45:45So I hope now you are more clear how to
45:47answer all these questions and you are
45:49more confident to present this or any
45:52other project in any of the interviews.
45:54So if you really like these questions uh
45:57please put a like and comment on this
45:59video. You can download the PDF as well
46:01which I have provided uh so that you can
46:04revise it anytime you want you are going
46:06for any interview and please share it
46:09with others also so that they are also
46:11aware they also feel more confident and
46:14if you have any other questions in mind
46:16which are asked to you you don't know
46:18how to reply put them also in the
46:20comment section so that I can have a
46:22look I can probably create another video
46:25as well so that's all for now thank you
46:29bye-bye Say happy learning.

About this transcript

This page contains the full transcript of Azure + Databricks Interview Questions from Real Project Scenarios by DataBeli, generated from the public captions YouTube serves with the video. The transcript has 7,882 words across 1,103 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.