YT YouTube2Text

YouTube2Text → Transcripts → Azure Data Engineer Mock Interview - ADF Scenario Based Special

Azure Data Engineer Mock Interview - ADF Scenario Based Special — Transcript

by Azurelib Academy · 3,783 words · 533 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:02yeah hi urashi thank you for coming in
0:05for this interview so can you please
0:07brief us about your profile what you
0:12do okay thank you deak so my name isashi
0:15p I have a total six years of experience
0:18in IT industry in banking and financial
0:21sector domain BFS So currently I'm
0:23working with infosis uh as a data
0:27engineer okay great uh
0:31so can you just tell me a little bit
0:33about your
0:35project uh okay so uh we like in our
0:39project We are following this uh
0:41Medallion architecture so where we have
0:44this uh ra raw layer bronze layer uh
0:48sorry bronze layer silver layer and gold
0:49layer so in first layer like we are uh
0:53like in this bronze layer we are moving
0:56our data from on Prim to Cloud using
0:59this ADF and then like in a uh uh ADLs
1:03like in ADLs we cannot get everything so
1:05that so that further we need to use data
1:08braks for the pre-processing like for
1:11filtering the records and then finally
1:13once data is available in silver layer
1:15then we have to uh like uh like use some
1:20business logic like we have to use this
1:22aggregation functions and all and then
1:24we move the data to gold layer and from
1:26gold layer like uh we just connect this
1:29gold to powerbi so that our business and
1:32all can get the data from there okay um
1:38have you worked on the Azure data
1:39Factory as
1:41well yeah you worked on I worked on it
1:45okay have you created some of the
1:47pipelines or you just aware of it you
1:49created on your own yeah I have I have
1:51created some of the pipelines
1:55okay now let's imagine that there is a
1:58situation okay so
2:02will there is a one adless storage
2:05account okay okay you are getting a data
2:09there is some third party which is
2:10dropping the data into this ADLs account
2:13okay you have to create a pipeline which
2:17is going to pull the data I mean pull
2:19the file from this ads location and dump
2:23this information into a SQL
2:26database okay how would you do that
2:30okay so our source is in ADLs
2:33only and we have to move in the ADLs
2:36location only right source is ADLs and
2:39destination is a SQL
2:42database okay so SQL database like from
2:47uh so like we so we have to create uh
2:51the pipeline so for that we need to
2:53create the data sets two data sets two
2:55link service and one Pipeline and there
2:58we will use this copy activity so that
3:01we can uh copy the data from this ADLs
3:05to SQL
3:07DB okay great but imagine that there is
3:11another twist okay you want to run this
3:16pipeline only when a file arrived okay
3:21imagine as as soon as a file arrived I
3:23want to execute this pipeline how we can
3:26do that so for that we can use this uh
3:31uh storage event trigger uh so that what
3:35it does actually uh this uh trigger is
3:38used whenever any file get arrives in
3:41our location so at that time it triggers
3:44the uh like the pipeline uh get started
3:48so there we can there we have an option
3:50like start with like what like the file
3:54like so we can mention there if we have
3:55do CSV file so we can mention in the
3:57trigger and our pipeline getss started
4:01okay understood this is also understood
4:03so imagine a file comes you your
4:07pipeline triggers because of your
4:08storage event and it get executed and
4:12then finally uh you know it data get
4:15dumb but I want that as soon as you
4:18process the file you know you move this
4:21file to a different folder as well right
4:25so that it should not get I mean uh it
4:27should get it defined as okay this file
4:29is already processed right so you should
4:32have a one extra folder somewhere in
4:34ADLs and there once you have datag get
4:37dump into the DAT database I want you to
4:40move this file to a different folder how
4:42would you do that so we want to like for
4:46in particular folder we want this file
4:49right
4:50yes so actually we in in the for you can
4:54use this 4 activity and there we can uh
4:58like Define uh this folder location like
5:02we we use this string interpolation so
5:05that where we can Define uh like for
5:07example for today's date we want uh to
5:11get the file so we can mention that uh
5:15in
5:18there
5:20okay
5:21okay okay you talked about the for
5:25activity okay what for activity
5:30do so it actually it trates like if we
5:34are moving the data from for for example
5:37400 for 10 tables so we can mention all
5:41these 10 tables there and it will uh
5:44like move the data for all those 10
5:46tables okay imagine another situation
5:50okay another scenario based
5:52question so you said like there are 10
5:55tables which data you want to pull okay
5:58you say that okay I'm going to do it
6:00through a forage right out of those 10
6:05tables I need to have a situation that
6:10if this table belongs to let's say a
6:14critical table table or let's say I
6:18let's say the data size let's go by a
6:19data size so if there is a more than you
6:24know 100K rows for this table only then
6:28only you should pull it otherwise you
6:29should not pull the data for this table
6:32okay so imagine again I'll repeat it to
6:34avoid any confusion you have a 10 tables
6:37you are pulling the data from all those
6:3810 tables using your for each but while
6:42you're doing it you need to ensure that
6:45it pull only the data in case when that
6:48specific table have more than 100K
6:51records how would you solve this
6:55problem so uh first uh we can get get
6:59the count actually from a table so for
7:02that we can use lookup activity so that
7:04there we can get the count of the table
7:07and then we can use the if if then this
7:10if activity if he activity so there we
7:13can mention like if um this activity
7:16records is greater than uh like is equal
7:19to 100 greater than 100K records then
7:22then only we will uh pull the records
7:25for uh that
7:26table okay you will write this logic
7:30within the for each activity or outside
7:32the for each activity where are you
7:33going to write
7:35this uh so we will uh write this uh in
7:43in if if
7:47activity see again okay like in for each
7:51activity only there we will use this
7:54yeah so this if and this lookup goes
7:57within the forage or outside the forage
8:00lookup is uh outside the forage but this
8:03if we can uh this uh in this uh if a
8:08condition we can write inside the forage
8:10are you
8:13sure
8:16yes
8:19okay okay uh that's
8:23fine now you have uh probably have uh
8:29this pipeline okay you added the trigger
8:32to this pipeline as well but imagine
8:35that because of some certain reason when
8:38this pipeline was working for all those
8:3910 tables for five table it worked
8:43perfectly well okay but when it tries to
8:47copy the data for the sixth table this
8:50got break because of some certain reason
8:53so pipeline got failed okay now I want
8:58you to to copy only the data from the
9:02sixth table onwards because till five we
9:05already copied
9:06it question number one is it
9:09possible and if it is possible how you
9:12would do
9:14that uh yes it is possible so um here
9:19like if uh we have this this kind of
9:22scenario so we can use this pipeline
9:24parameters actually so there we as in
9:27pipeline parameters we all like 10
9:31tables we Define at the run like at the
9:33run time it will uh show as like for
9:37example this Six Tables already got
9:39successor and we want to run from Seven
9:42tables 7 8 9 10 so at the run time we
9:45can remove those six tables and we can
9:48mention only the failed uh uh pipeline
9:51table
9:52name okay great so it means that you are
9:56saying that in the pipeline itself when
9:58you're making the pipeline you're not
9:59hardcoding the tables you're passing it
10:01as a parameters
10:04only okay makes
10:06sense and how you get to know that only
10:09these first six got passed I
10:12mean what how you get to know that okay
10:14out of 10 only first six get executed
10:17because probably it will run in parallel
10:19right so you're not very sure which one
10:21got
10:22failed so then so this we can check in
10:24our manage tab like there we have this
10:27history right so there we can get all
10:30the details of our successful pipelines
10:34and failed
10:35pipelines okay makes sense makes sense
10:39okay great now you done this now imagine
10:44that all this looks perfectly well your
10:47data get copied and
10:49all now I want you that as soon as any
10:54of my pipeline get failed you should get
10:57an email
10:59right to get to know that okay something
11:02got failed how would you do
11:05that
11:07okay uh email notification
11:11right you tell
11:14me yeah
11:16I there is just one activity where we
11:20can added this um notifications and all
11:23and we can get the
11:27notification but right now no I'm not
11:30getting it okay no worries no
11:33worries okay so imagine that email also
11:36came okay now this pretty much looks
11:41okay you okay now imagine that this
11:44email is also come I want you to run
11:47this
11:48pipeline once every day okay at
11:5312:00 central
11:55time okay Central Time us
11:59my this network was disconnected so I
12:04okay so I'm saying that
12:07imagine um you created this pipeline it
12:11is email notification you added it now
12:14this pipeline you want to execute every
12:17day
12:18one and um every day once at 12:00
12:23Central Time Zone not India time zone
12:25Central Time Zone how would you do that
12:34yeah I think there is some
12:36hiccup
12:38urashi urashi is
12:41back yes yes you heard the question or
12:44should I repeat no actually I didn't
12:47have it okay so I'll repeat the question
12:49one more time uh you did this
12:52notification everything pipeline is fine
12:55now I want to execute this pipeline once
12:57every day and 12:00 midnight Central
13:02Time Zone not India time zone Central
13:04Time Zone okay how you will configure
13:07your pipeline so that it will run in
13:09this
13:10manner okay once in a day right at
13:14Central Time
13:16Zone 12:00 Central Time Zone how would
13:20you do that
13:21okay okay so we can
13:26uh I use this um
13:30tumbling window trigger there and we can
13:33mention this time and like instead of
13:37ISD we can dare use this CSD time
13:41zone okay tumbling window is fine why
13:45you not using the schedule uh trigger
13:49why you are using tumbling window
13:50trigger any specific reason behind
13:54it
13:56uh for uh like in uming window like if
14:01tumbling window like if we want to set
14:03any dependency so that we can use this
14:06tumbling window trigger and uh
14:14uh yeah we can also use this schedule
14:18but U
14:23okay okay no worries no worries okay so
14:27imagine your
14:29triggers also got added up now it's time
14:32to move your code from your Dev
14:35environment to the uat environment for
14:38the testing how would you do
14:40that so uh we use this uh Azure devops
14:45there so we can uh like we uh we just uh
14:50move our code like we move our arm
14:54templates from da environment to our
14:56higher environment so there we have to
14:58create the PIP line and we can move from
15:01one environment to another environment
15:04okay so you're going to use the Azure
15:05devops for doing this yeah okay
15:11great then
15:13[Music]
15:15um have you ever heard about the global
15:21parameters in as you data Factory Global
15:26parameters uh yes global
15:29means we just need to create it once and
15:33then we can use it anywhere like but I
15:37haven't created it actually
15:41okay okay you were doing this deployment
15:44coming back again to deployment do you
15:46have the same workspace or you have a
15:48different workspace in a different
15:50environment how is your project
15:51structure looks like so same ADF um no
15:55no we for D environment we have one ADF
15:59like uh ADLs account and for another
16:02environment we have another account
16:06and okay for ADLs or for ADF what you're
16:11talking confirm me again for ADLs and
16:14what about the
16:16ADF ADF
16:20uh ADF yes ADF for ADF also for ADF also
16:25okay great so thank you thank you Rashi
16:29let's get to the second part of it where
16:32we do an assessment what went right what
16:34went wrong okay great so starting with
16:38is uh your project description was okay
16:43I mean I feel that it has to be a little
16:45more extensive so okay you know couple
16:48of things but you sounds a little less
16:50confident so you know a lot of things
16:52it's not like that you don't know so
16:54please you know you have a confidence so
16:59if you little low on confidence you
17:01don't see uh I mean that energy in your
17:04voice and I mean if even if your answer
17:07is right now it seems like it's wrong
17:09right so be confident be confident you
17:11know a lot of things it's not like that
17:12you don't know okay great so when you
17:16talk about your
17:17project not starting with just that we
17:20have a three layer architecture or a
17:22medallion architecture it should start
17:24with something like our project is some
17:26let's say for XY Z domain so maybe I
17:29have a banking insurance project and we
17:31are trying to build their a data
17:32platform or we are trying to build a a
17:35data system which for which we are using
17:37the three layer architecture something
17:39like that and then we we clearly talk
17:42about what is the source what is the
17:44destination and what you are doing in
17:46middle that is what has to be crystal
17:48clear so overall this is a source this
17:51is the U this what we are doing so you
17:54could be say that we are getting a data
17:56from these three different sources or
17:57four different sources and we are using
18:00ADF to pull the data then maybe we are
18:02using three layer architecture in the
18:04spark or data braks and then uh this
18:07data will be you know provided to the
18:09downstream system something like that so
18:11it's like you know end to end okay
18:14somehow and it will be easy for the
18:16interviewer also you know
18:18to uh assume it like what is going on in
18:22the project okay then uh you talked
18:26about the aure data Factory
18:30so that initial discussion was fine like
18:33how you pull the data from you know from
18:36one location to another one from you
18:39were right that two data set to link
18:41service needed to get create but try to
18:43be little bit more elaborative there
18:45that we need a link service we need a
18:48two link service one for making
18:50connection to your ADLs location and one
18:53for database so just little bit little
18:56bit elaborative that why you need this
18:58link service and you data set you said
19:01it like two data set is needed copy
19:02activity needed it's a perfectly fine
19:04answer if you want you can little bit
19:06elaborate remember interview is more
19:10about killing the time okay we try to
19:13kill the time as much as possible okay
19:15so you can be little bit elaborate okay
19:18so that can cut down extra question
19:19which might come on your way okay yes
19:23then uh I give you another scenario like
19:25how you automatically get pops up your
19:28answer was almost almost right that
19:31storage even trigger is going to be used
19:33so that it can automatically picks up
19:35the file and take that condition that
19:37was right not a problem but again you
19:40when you answering this question you
19:42were not very confident see confidence
19:45is important okay that's that's ensure
19:50that you are not putting it as a fluke
19:53sometimes people put it as like fluke
19:55you're not confident so you're not sure
19:56of it so whatever you are you know be
19:59sure of it so at least even in your
20:02heart you are not sure of it but you
20:05know pretend like that you are sure of
20:07it okay so confidence is
20:11important then uh so so far everything
20:13was right I mean it was like you were
20:15answering the question so that's why I'm
20:17saying that you have to be more
20:18confident then we talked about uh the
20:21for activity that for multiple table for
20:24each activity was fine but you got
20:26little bit fumble like where the if
20:29activity comes within the forage or
20:31outside the forage you have to whether I
20:34will ask you have to clearly picture out
20:37that okay this is how the logic is going
20:39to work okay so you have to be crystal
20:41clear so there little bit you get in
20:44doubt then you said it wrong that lookup
20:46activity has to be outside the forage so
20:50lookup activity in this case need to be
20:53inside the forage because you might be
20:55checking for every table so you are
20:58checking the the count for each and
20:59every table right any table which is
21:01having 100K plus record only then you
21:03are copying it right so this lookup
21:05activity has to be within the forage not
21:08outside the forage if it is outside
21:10forage it can check for only one table
21:13but you want to check for every table so
21:14it has to be within it okay so I'm not
21:18sure we have Twisted the some kind of
21:20you know uh yeah the scenario so be sure
21:26of it that's one thing which didn't went
21:29well okay then uh then we talked about
21:34the email uh one thing like how you know
21:38if this get uh fail this one of the
21:41table got failed and some of get passed
21:43some of didn't got pass so how you would
21:45do your answer is right that
21:47parameterization code is going to help
21:49you so I mean I would love to you know
21:54uh put it in a little different way your
21:57answer was right answer was not wrong
22:00but saying something like that that yes
22:03we know that this kind of a situation
22:05happens and when we are doing a fall
22:07Loop kind of thing that if any of the
22:09iteration got failed eventually my
22:11activity might got failed so what we can
22:13do is like we parameterize this entire
22:16pipeline okay so that rather than
22:19hardcoding passing hardcoded value is in
22:21the forage we passing it as a pipeline
22:24parameter so that and that for will take
22:27the value from the pipeline parameter
22:29and uh dynamically run it so next time
22:33if out of 10 tables we have to run it
22:34for a six tables only or a four four
22:36tables only I can rerun the pipeline
22:39just providing those only the four
22:41tables so automatically everything will
22:43get dynamically get executed again okay
22:47so answer was on the on the right side
22:51only but again full marks only will be
22:54given if you you know confidence second
22:57elaborated it properly
23:00then email notification you got off the
23:02card so email notification you are not
23:04aware of it so work on that email
23:07notification maybe like you using a
23:08logic app one way if you categorically
23:11want to send an something email or else
23:13you can send the uh set the email
23:16notification at the Azure data Factory
23:18level itself so you can go to the alerts
23:20and metrics and there you can set it uh
23:23condition that if any of the specific
23:25pipeline Go fail automatically send an
23:27email through an uh Microsoft itself
23:31okay so one way is that another one
23:33could be through a logic app so both the
23:36answers any one of them at least you can
23:38give it or you can you know U at least
23:40talk about it then when I said like how
23:44you going to schedule the pipeline you
23:46talked about the tumbling window so
23:49tumbling window trigger I mean you said
23:52it it could be good but then you added
23:54like why you using tumbling window
23:57trigger then you said that okay we have
23:59a dependency on so probably you can say
24:01that we can use a schedule event Trigger
24:04or tumbling window trigger both of them
24:05can do the work here it's not like that
24:08you can pick only one of them anyone can
24:10solve the problem uh tumbling window
24:12trigger is oneon-one so it if you have
24:14multiple pipelines eventually you end up
24:16with creating multiple tumbling window
24:17triggers um but if you have a schedu
24:20event trigger you can reuse them okay so
24:23you get take a call in tuming window
24:24trigger is not wrong but when I ask why
24:27you need it probably you need to ensure
24:29that rather than saying that dependency
24:31and all you should have said it like
24:33that I can also use the schedule event
24:35trigger also that also going to work but
24:38like we want to create a separate one
24:40that's why probably and if retry needed
24:42to be done tumbling window trigger can
24:44do the rres your answer for that setting
24:47up that time zone is also fine so that
24:50is Right Way Global parameters you're
24:52not aware of it so work on that Global
24:55parameters lastly uh
24:59that uh cicd thing so cicd thing is okay
25:03so your answer is right that we are
25:04going to use the devops and all so that
25:06is also going to work so I think overall
25:09it's fine but devops answer also you can
25:11little bit elaborative in a little
25:13better manner so overall looks like
25:15couple of things you know I mean I would
25:18say that 60 70% of the things you know
25:20you have to be more confident and just
25:21little bit when you practice little bit
25:24more so the formation of your answers
25:28would be much better that's the only
25:31thing so I think you are on the right
25:32track thank you so much okay thank you

About this transcript

This page contains the full transcript of Azure Data Engineer Mock Interview - ADF Scenario Based Special by Azurelib Academy, generated from the public captions YouTube serves with the video. The transcript has 3,783 words across 533 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.