How I’d Learn Data Engineering in 2026 From Zero to Senior (Built by a Data Engineer) — Transcript
Full transcript
- 0:00Hey friends. So now last year I shared
- 0:02with you a road map on how to become a
- 0:04data engineer and honestly I didn't
- 0:06expect the reaction it got. Many of you
- 0:08followed the road map step by step and
- 0:10even some shared with us that they
- 0:12managed to get a job. So congrats that
- 0:15was really amazing to see. So now what I
- 0:17have done I went back to that road map
- 0:19and I reviewed every part of it. And
- 0:22after that not only I updated the
- 0:24content based on the market but also I
- 0:27extended the road map. This time it
- 0:29doesn't stop at getting your first job.
- 0:32It goes further in what actually happens
- 0:34after you get hired. So that means how
- 0:37to grow inside a company and what are
- 0:39the things that you have to learn there.
- 0:41So my friend in this video I'm going to
- 0:42walk you through the full journey from
- 0:45the absolute beginner to become a senior
- 0:47data engineer. And by the way if you are
- 0:49new here then welcome. My name is Bar
- 0:51and I have 17 years industry experience
- 0:54as a data engineer. I led one of the
- 0:56biggest data engineering and AI project
- 0:58at Mercedes-Benz and in this channel I'm
- 1:01sharing everything that I know from the
- 1:03industry. So now let's deep dive into
- 1:05the road map.
- 1:06>> Wow, you just dive right in.
- 1:08>> Okay. So now like the previous road map
- 1:10I have built everything using notion. It
- 1:12is like a template for you. You can go
- 1:14and download it or duplicate it in order
- 1:16to add your own notes and as well you
- 1:19can go and build your own timeline
- 1:20around it. And as well it is a living
- 1:22template. So each time I find something
- 1:24new, I'm going to go and update it. Of
- 1:26course, you can find the link in the
- 1:27description. Okay. So now I have
- 1:29splitted the whole thing into two
- 1:31journeys and inside each journey there
- 1:33are phases and inside the faces there
- 1:35are steps. So now the first one is the
- 1:37hiring journey. This part focuses on
- 1:39learning just enough skills to get you
- 1:42your first data engineering job. So
- 1:44we're going to be focused and learn only
- 1:45the things that we need without wasting
- 1:47a lot of time. The second one is the
- 1:50growth journey. This is what happens
- 1:52after you got hired. So how to grow on
- 1:54the company? What are the things that
- 1:57you have to add and learn as you are
- 1:59working? Because my friends, we never
- 2:01stop learning even after we get a job.
- 2:03So this is the full big picture and now
- 2:05we're going to deep dive into each
- 2:07journey into each skill. So let's start
- 2:09with the first one, the hiring journey.
- 2:11So we're going to scroll down over here.
- 2:12So now again the hiring journey, it's
- 2:14all about learning all the skills that
- 2:16we need to take you from zero until you
- 2:19get hired. Now we start with the phase
- 2:21one I call it the absolute beginner. So
- 2:23here it's not about learning any tool or
- 2:25platform. It's all about understanding
- 2:28what is data engineering and make a
- 2:30decision at the end. And by the way if
- 2:32you go and click on this you're going to
- 2:33find more details inside the road map.
- 2:36I'm going to put more context and guides
- 2:38inside each skill and as well at the end
- 2:40you're going to find resources. So now
- 2:41what I want you to do at the start is to
- 2:44understand what is data engineering. So
- 2:46what the role looks like inside a
- 2:48company and what kind of problems and
- 2:50tasks we solve every day and then you
- 2:53have to start comparing and
- 2:54understanding what are the differences
- 2:55between data engineers, data analyst,
- 2:58data scientist just have some basic
- 3:00understanding what are those different
- 3:02data roles and of course I cannot now
- 3:04deep dive into each skill and what this
- 3:06means because this might take ages. This
- 3:08is your job to do that for each skill.
- 3:11But as a data engineers we write a code
- 3:13to move the data from the sources. Then
- 3:15we do some transformations,
- 3:17preparations, clean up and prepare
- 3:19everything in order to load it in the
- 3:21target so that we can do analytics. And
- 3:23by the way, I have here some nice
- 3:25resources for you. I went deep dive into
- 3:27this role as I explained how to learn
- 3:29Python for data engineers. And in this
- 3:32data movie, I have explained all
- 3:33different rules. All what you have to do
- 3:35is to understand what data engineering
- 3:37means. So this is the first step. Then
- 3:39the second step which is the most
- 3:41important one. After you understood what
- 3:43we do, you have to make an honest
- 3:45decision and ask yourself, do I actually
- 3:48want and enjoy doing such activities as
- 3:52a data engineer every day, every week?
- 3:54If it's no, then it's totally fine. You
- 3:56can just close the video now and that's
- 3:58it. But if you say this is the right job
- 4:00for me, then you have to commit to this
- 4:03path. So this decision, my friend, is
- 4:05the most important thing. It's going to
- 4:06save years for you. You have to forget
- 4:08all the hype and the salaries about the
- 4:10data engineering. this something that
- 4:12you want and you love to do. If you say
- 4:14yes, then welcome to the club and we're
- 4:16going to go to the next phases. Okay,
- 4:18let's go to the face one. Here the real
- 4:20work starts. On this face, it's all
- 4:22about building your foundations. And the
- 4:24goal here is very simple. I want it to
- 4:27feel very normal working with data and
- 4:29writing a code. That means we will not
- 4:31deep dive into any data engineering
- 4:34topics yet. It's going to be very
- 4:35overwhelming. Instead, reduce the noise
- 4:38and just focus on only those three
- 4:40skills. And there is like one rule that
- 4:42I want you to follow. You have always to
- 4:44practice way more than consuming. And I
- 4:46call it the 80/20 rule. 80% practicing
- 4:50and only 20% consuming. If you just sit
- 4:52down and start watching a course without
- 4:54doing anything, then you are not
- 4:56learning anything. And the second rule
- 4:58here is that you have to learn one skill
- 5:00at a time. So never learn two things
- 5:03together. Don't go and learn SQL
- 5:04together with Python. Only one thing at
- 5:07a time. And another thing that I want
- 5:09you to focus in this phase try to
- 5:10understand more than memorizing the
- 5:12syntax. I want you to understand how
- 5:14things works behind the scenes and how
- 5:16the logic works because who needs syntax
- 5:19we have now AI right? And of course
- 5:20speaking about AI try to avoid it for
- 5:23now. You can use it in order to explain
- 5:25something for you but not to write a
- 5:27code for you. Only with a real struggle
- 5:29you're going to learn. So now about the
- 5:30skills the first thing that you have to
- 5:32learn is SQL. It is the language that
- 5:35you use in order to talk to the data.
- 5:38This is the number one skill in any data
- 5:40role. So you can use it as a data
- 5:42engineer in order to do a lot of data
- 5:44transformations inside data lakeouses
- 5:47and you can use it as well to build a
- 5:48business logics and stuff like that. I
- 5:50have here for you a list of all the
- 5:52things that you have to learn. So learn
- 5:54the basics about the select statements,
- 5:56how to filter the data, how to do data
- 5:58transformations and cleaning. You have
- 5:59to learn a lot of scale functions, how
- 6:01to do data aggregations and how to join
- 6:04tables. Then comes advanced topics like
- 6:06the window functions, the CTE,
- 6:08subqueries, views. Now, of course, I can
- 6:10totally recommend my SQL course. It got
- 6:122 million views in just few months. So,
- 6:15the first thing is SQL. Then you're
- 6:17going to go to the next one is the
- 6:19Python. This is the only and the core
- 6:21programming language that you have to
- 6:23learn as a data engineer. And my friend,
- 6:25you have to master it. If you cannot
- 6:26write a good clean Python code then I
- 6:29really don't think you can do any data
- 6:31engineering and here it is really tricky
- 6:33because there are a lot of things to
- 6:35learn about Python that's why I made a
- 6:37very compacted list for you to learn it
- 6:39it's not a lot actually so you have to
- 6:41learn about the variables and the data
- 6:42types the data structures very important
- 6:45the list and the dictionaries the
- 6:47control flow the if else loop how to
- 6:49write a functions and now comes
- 6:51something really important how to work
- 6:53with files especially the CSV and the
- 6:55JSON how to do error handling, do
- 6:58logging, connect to different systems
- 7:00like databases, APIs and Kafka. Of
- 7:02course, I explained more in details why
- 7:05we have to learn all those stuff. You
- 7:06can check the video how to learn Python
- 7:08as a data engineer. And as well, many of
- 7:10those topics are already covered in my
- 7:12boot camp. You can check as well the
- 7:13link over here. So, you have to master
- 7:15this language. And now comes the third
- 7:17skill which is really nice one. You have
- 7:19to learn about Git and GitHub. Since
- 7:21your job is to write code, then this
- 7:24comes as automatic thing. You have to
- 7:26learn as well how to work with the
- 7:27GitHub. So it is the way where we save
- 7:29our code, track our changes, we can
- 7:32collaborate with others and without this
- 7:34of course you cannot do any real
- 7:35projects at companies. So you just have
- 7:37to learn the basics like how to create a
- 7:39repository, how the workflow works, add,
- 7:42commit, push, pull. You can learn it
- 7:44really fast. Those are the foundations,
- 7:46the three core skills that you have to
- 7:47learn and again one after another. And
- 7:50my friends, if you want to break into
- 7:51data engineering and learn all the
- 7:53skills that I mentioned in one place,
- 7:55then I always recommend data camp. When
- 7:57learning key data engineering skills,
- 7:59it's hand down one of the most
- 8:01interactive handson platform out there.
- 8:04The associate data engineering in SQL
- 8:06track is perfect starting point. You
- 8:09will go from understanding the basics of
- 8:11data engineering and ETL ELT workflows
- 8:14to master posress SQL SQL joins and data
- 8:17warehouse principles with snowflake. You
- 8:20will also dive into database designs,
- 8:22normalization and even build real world
- 8:25projects like analyzing the students
- 8:27mental health and London travel network
- 8:29in order to start building your
- 8:30portfolio. And once you completed the
- 8:32courses, you can try to pass an industry
- 8:35recognized data engineering career
- 8:37certification which is a huge boost when
- 8:40you are searching for job. So if you are
- 8:41serious about becoming a data engineer,
- 8:43check the link below. I even can get you
- 8:4525% discounts. Now moving on to the
- 8:48second phase, the data engineering core.
- 8:50It is where you're going to get
- 8:51specialized and start feeling like a
- 8:53real data engineer. So the goal here is
- 8:56not to learn how to code. It's all about
- 8:58how to think like a data engineer and to
- 9:01learn how data systems are built end to
- 9:03end. You have to experience the whole
- 9:05process. And I recommend you here to
- 9:07focus on the concepts more than the
- 9:09tools, the platforms and the UIs. So
- 9:11don't rush on learning how to do things.
- 9:14Instead learn the why concepts behind
- 9:16those stuff and that's why the first
- 9:18thing that you have to do in this phase
- 9:19is to learn some theory. Here we are not
- 9:22learning any tool. We are not learning
- 9:24any like programming language. We just
- 9:26want to research, read and understand
- 9:28the concept. Like for example here, what
- 9:30is a data pipeline? What are the
- 9:32differences between batch processing and
- 9:35stream processing? What is an ETL, an
- 9:38ELT? What are the differences between
- 9:40them? And some basics about the data
- 9:42architectures like the data lake, data
- 9:44warehouse and the most important of
- 9:45course the data lakehouse. try to
- 9:47understand as well what is data
- 9:49moduling. What is the theory behind the
- 9:51Hadoop and the Spark, how the Apache
- 9:54Cafka work. It is just theory. We are
- 9:56not practicing or anything. And of
- 9:58course, many of those terms I already
- 10:00covered in for example the data movie.
- 10:02So go over there and start checking the
- 10:04data engineering part. Other than that,
- 10:06I have as well an article about 200
- 10:09terms that we use in data worlds and the
- 10:12part about data processing and data
- 10:13engineering I just posted here in the
- 10:15notion. So get yourself familiar with
- 10:18the data engineering terminology. Now
- 10:20the next step we have to learn about
- 10:22something called data bricks. This is
- 10:24not like a tool or let's say programming
- 10:27language. It is a platform that you can
- 10:29do many things inside it. Data
- 10:31analytics, data engineering and data
- 10:33science totally based on the cloud using
- 10:35Azure and AWS. But it is a modern web
- 10:38but it is a modern platform that allows
- 10:41big companies to scale. So it can really
- 10:43process huge amount of data and it is
- 10:45getting a lot of attention. My last five
- 10:48years I built completely projects using
- 10:50only this one platform. If you learn it,
- 10:52it's going to be big plus on your
- 10:54resume. And here in this step, I just
- 10:55want you to read as well about what is
- 10:58data bricks, why the company exist, what
- 11:00are the main components inside the
- 11:02platform and start learning the basic
- 11:04terminology that we use in data bricks
- 11:06like what is a notebook, what a cluster.
- 11:08So just a theory and exploring the
- 11:11platform and the tool. That's because
- 11:13the deep dive going to happen in the
- 11:14next step where we're going to learn
- 11:16about spark and pi spark. This is the
- 11:18main engine that we use in data bricks
- 11:21in order to process massive amount of
- 11:23data. It help us to scale by using the
- 11:26distributed processing. So start by
- 11:28understanding what is spark and why we
- 11:30need distributed processing. What are
- 11:32the data frames and how they work and
- 11:34then start to learn how to use it inside
- 11:36for example data bricks. Just learn here
- 11:38the basics. Don't go deep dive into like
- 11:40each detail about the bicep spark
- 11:42because this going to take a lot of
- 11:43time. Just learn how to read data, how
- 11:45to process it and then how to write it
- 11:47back. Then the other stuff you're going
- 11:48to learn it as you become a senior data
- 11:50engineer. So now my friend I can tell
- 11:52you actually this is the last skill that
- 11:55you need to learn in this journey. This
- 11:57is very simple. So SQL, Python, Git and
- 11:59GitHub, data bricks and pispark. With
- 12:02that you can understand exactly why we
- 12:04have learned all those skills. You don't
- 12:05have to go and build 20 projects. Just
- 12:08make one clean and strong project. So
- 12:10one idea that I really recommend you to
- 12:12do is to go and build a data lakehouse.
- 12:15You're going to use data bricks,
- 12:16pispark, python, sql, everything in
- 12:18order to move the data into like three
- 12:21layers. So you're going to enter the
- 12:22data as very messy raw data. Then you're
- 12:25going to start transforming it, cleaning
- 12:27it until it goes to the last layer where
- 12:30everything looks ready. Once you do
- 12:31this, you're going to really experience
- 12:33the whole process end to end. I have a
- 12:35very similar project but using an SQL to
- 12:38build a data warehouse. It is very
- 12:40similar project where you're going to go
- 12:41and process the data in different layers
- 12:43but you can adapt it and use completely
- 12:45pispar and data bricks in order to do
- 12:47the project. Now again as I told you you
- 12:49have to do only one project because
- 12:51there's a lot of things to do. You have
- 12:53to do it right. You have to document it
- 12:55correctly in the GitHub. You have to
- 12:56learn how to explain it as well in the
- 12:58interviews. Make very solid one and
- 13:01everything completed and you still have
- 13:02time. You can start maybe a second
- 13:04project. So that's it actually for this
- 13:06phase. If you get it right, then you are
- 13:08not longer just learning data
- 13:10engineering. Actually, you can do the
- 13:12job at companies. All right, so that's
- 13:13all for the phase two. And now we're
- 13:15going to go to a scary phase where I
- 13:17want you to get hired. This is to be
- 13:19honest where most of the people like
- 13:21stuck or slow down or stop. It's all
- 13:23because they don't feel confident enough
- 13:25to start applying for jobs. They always
- 13:27going to feel like there is something
- 13:28missing. There are few skills and tools
- 13:30I didn't learn yet. I see someone in
- 13:32LinkedIn like listing a lot of skills
- 13:34that I don't know about it. So they are
- 13:37always stuck in this loop where they go
- 13:39from one course to another course and
- 13:41they will never start applying for jobs.
- 13:43And this is totally wrong because my
- 13:44friend you will never feel 100% ready to
- 13:48start applying. If you feel like 60 or
- 13:5070% then actually you are ready. So in
- 13:52this phase you have to start preparing
- 13:54yourself to get hired. You have to
- 13:56search for jobs and of course keep
- 13:58learning. Now let's talk about the first
- 13:59thing that you might think about is to
- 14:01get certified. Now the thing is if it is
- 14:03your first job the recruiters might look
- 14:06whether you have a certificate or not
- 14:07and if they see one maybe they get
- 14:09closer but I can tell you in many
- 14:12scenarios it is not the number one
- 14:14reason why someone talks to you. So in
- 14:16most cases the project has more weights
- 14:19than certificates because certificates
- 14:21does not say that you are a data
- 14:23engineer but if I check your project I
- 14:25can see exactly how you went through the
- 14:27process and I can trust it more that you
- 14:29can do the job. So projects has more
- 14:32weight than certificates and another
- 14:34mistakes that you might do is that you
- 14:36start collecting and chasing a lot of
- 14:38certificates and this is really huge
- 14:40waste of time and money. So now what I
- 14:42recommend you is to pick one relevant
- 14:45certificate like for example in this
- 14:46road map as you can see we learned about
- 14:48the pi spark the data bricks. If you
- 14:50have the money and the time then I
- 14:52recommend you to take certificates from
- 14:54data bricks. It is a huge plus to your
- 14:56story. So you have learned all the
- 14:58topics about the pi spark and the data
- 15:00bricks. You have done a project and as
- 15:01well you have a certificate. Now the
- 15:03thing is with all thoseations they are
- 15:05always connected to specific platform
- 15:08like if you want to get certified from
- 15:10IBM you have to learn a lot of things
- 15:12about the IPM platform. The same thing
- 15:14goes for Google or Azure or AWS. You
- 15:17cannot go and start get certified by all
- 15:19those different platforms. Then you have
- 15:21to learn all of them. And actually this
- 15:23is something that we never have to do.
- 15:25So my friend, don't go and chase all
- 15:27those different certificates. It's
- 15:29really going to waste your time, your
- 15:30money, your energy. So instead, do one
- 15:33solid data engineering project and maybe
- 15:35get only one certification that is
- 15:38relevant to your story, to your path. So
- 15:40that's all about the certification.
- 15:42Don't overthink it. Now let's go to the
- 15:44next step. Here you have to build one
- 15:46clean resume. So here there are like a
- 15:48lot of best practices about how to build
- 15:50a resume. Just from my experience, I can
- 15:53tell you make it easy to read. Make it
- 15:55clean. Don't over complicate it. Don't
- 15:57put the skills that you wish to learn.
- 16:00Just be honest about what you actually
- 16:02have done. And make it just easy to
- 16:04read. So we have to spend only few
- 16:06seconds reading it. Instead of like
- 16:08minute, just make the first version. And
- 16:10of course keep improving it as you start
- 16:12doing interviews. You start getting like
- 16:14feedbacks and you start adjusting it.
- 16:16And again like the certificate, a
- 16:18perfect resume will not get you hired.
- 16:20It is just the start of the conversation
- 16:22between us. Now the third thing that you
- 16:23have to do is to optimize your LinkedIn
- 16:25profile. So LinkedIn is not like social
- 16:28media. It's not your Facebook. And as
- 16:29well I'm not asking you to be an
- 16:31influencer on LinkedIn. Just makes it
- 16:33looks professional. So professional
- 16:35photo, some nice header statement, the
- 16:37about section, the skills that you have
- 16:39learned. Just start connecting with
- 16:41other people like maybe me LinkedIn as
- 16:43well. Maybe start posting about your
- 16:45progress. As you learn new skills, you
- 16:47do new projects. Just make few
- 16:49celebration about what you are doing.
- 16:51And the most important thing as well in
- 16:52LinkedIn, you have to feature your
- 16:55portfolio project. And this is exactly
- 16:57the next step. We have to build a
- 16:59portfolio. And of course, there are like
- 17:00multiple ways on how to build a
- 17:01portfolio. But for us as data engineers,
- 17:03things are easy. The best way is to have
- 17:06a solid profile in GitHub. So put the
- 17:09whole project in the GitHub. And here
- 17:11you have to make sure that the code is
- 17:13clean, it has comments, you are
- 17:15following the standards on styling and
- 17:17formatting your code. And by using the
- 17:19readme, you explain maybe who are you
- 17:21and what is the projects all about, what
- 17:24are the use case, the outcome. Think
- 17:26about it like your resume, your
- 17:28certificate. This is really important.
- 17:30Make clean profile in GitHub and really
- 17:32you're going to stand out of the crowds.
- 17:34Of course, you can go like step deeper
- 17:36where you build a personal website which
- 17:38is I'm going to say it is optional and
- 17:39of course if you don't know how to build
- 17:41a website and you don't want to invest
- 17:43money or time on it, then it's enough to
- 17:45use in notion. So you can build your
- 17:47whole portfolio in the notion and like
- 17:49introduce yourself as your personal
- 17:51website. It is totally for free but
- 17:53again this is optional. Way more
- 17:55important is the GitHub profile. And
- 17:57once you have all those stuff you start
- 17:59applying for jobs.
- 18:00>> I'd like to apply for a job.
- 18:01>> Okay. So now I understand totally this
- 18:03is an emotional topic and I can tell you
- 18:06the job market is horrible but it is
- 18:08totally doable. So you have to have the
- 18:11right mindset about it. Now the first
- 18:13thing is that it is your first job. So
- 18:15it must not be perfect. So what I'm
- 18:17trying to say here that don't be picky.
- 18:19Don't go and say you know what I'm going
- 18:21to apply only for those two three
- 18:23companies or only for very specific
- 18:25role. My friend this is something that
- 18:27you do if you are a senior data engineer
- 18:30if you are like maybe someone like me.
- 18:31So don't compare yourself with a senior
- 18:34data engineers because the game there is
- 18:36totally different as a first job. And I
- 18:39can tell you story from me as I wanted
- 18:41to change my job from my first job to
- 18:43the second job. I applied over hund
- 18:45times and the job market was way better
- 18:47than now. But I don't want to demotivate
- 18:50you. It is totally doable. It's just the
- 18:52game changed and you have to have the
- 18:54right mindset. Previously I didn't build
- 18:56like portfolio projects and get
- 18:58certified and all those stuff. Now you
- 18:59have to do more things in order to get
- 19:01hired. So now the process going to be
- 19:03like this. You have to prepare for the
- 19:05interview. So learn about the classical
- 19:07questions that's going to be asked for
- 19:09data engineers and as well very
- 19:10important learn how to explain your
- 19:12project. Explain the rough plan, the
- 19:15faces of your project, what you have
- 19:16done and what are the outcomes of this
- 19:19project. So learn how to tell it as a
- 19:21story end to end and start applying for
- 19:23job. See 100 200 times. Don't get
- 19:25emotional about it. It is not about you,
- 19:28nothing personal. And once you got
- 19:30invited to an interview, it is very
- 19:32important to understand and learn what
- 19:34happens during the interview. This is
- 19:36really a learning process. So you're
- 19:38going to learn how to answer questions,
- 19:40which questions you didn't answer
- 19:42correctly. And there is like one thing
- 19:43that I really like to see in the
- 19:45interviews that I can trust you with the
- 19:47job. I have to get the feeling that you
- 19:49are a builder, someone that's going to
- 19:51go and build things. So this is their
- 19:53process. You keep improving, you keep
- 19:55repeating and fix the weak spots until
- 19:57I'm sure you're going to get hired. So
- 19:59now that's all for this phase. And now
- 20:01let's say that after pushing a lot, you
- 20:03got accepted for your first data
- 20:05engineering job. And of course if this
- 20:06happens, then congrats. And I can tell
- 20:08you, you're going to go and enter a new
- 20:10journey. It is where you're going to
- 20:12grow on the job. So now I'm going to
- 20:13call this phase the junior data engineer
- 20:16phase. So now as well I want you to get
- 20:18the right mindset for this. Now you are
- 20:20in new project in your team and if you
- 20:22are lucky enough you might get an
- 20:24onboarding process. So someone can
- 20:26explain for you the project and the
- 20:27tools and then you can start getting
- 20:29your first tasks and start writing your
- 20:32first code of course simple tasks and
- 20:34then you might fix some issues from
- 20:36incident. And here I want you to
- 20:37understand at this phase it is totally
- 20:40okay to make mistakes. Actually this is
- 20:42the part of the process and there is a
- 20:44big difference between making mistakes
- 20:46and repeating the same mistake over and
- 20:49over. So my friend if you make mistake
- 20:51at the work it is totally fine but you
- 20:53have to learn from it. Try to not keep
- 20:55repeating the same mistake over and
- 20:57over. Doing mistakes it is totally okay.
- 20:59So don't be afraid of this. And another
- 21:01mistake that I found a lot from the
- 21:03juniors that they are afraid of asking.
- 21:05they are like shy. Don't be afraid of
- 21:07that. Keep asking. It doesn't mean that
- 21:09you don't know anything. If I go now
- 21:11into a new company and I get hired in
- 21:14projects, the first thing that I do, I
- 21:16keep asking them things. So, what are
- 21:18the source systems? Explain for me this
- 21:21business process. Explain for me this
- 21:23source system. What actually the data
- 21:25means in this specific table? So, keep
- 21:27asking questions at the start. And the
- 21:29third thing that might happen at that,
- 21:31yes, you're going to learn things from
- 21:32the job, but not everything. So this
- 21:34could happens a lot that you are working
- 21:36on projects that are using some outdated
- 21:39tools and you as data engineers of
- 21:41course you can learn those tools in the
- 21:43job but the thing is you are not
- 21:44learning the right skills for the
- 21:46future. Of course you have to learn from
- 21:48the job and as well to keep your eye
- 21:50open on how the market is moving. For
- 21:52example, if I just focused on my first
- 21:54job, I would be now a talent expert. It
- 21:57is an ETL tool and as well a database or
- 22:00expert. And I'm sure this will not be
- 22:02enough to me moving to any other
- 22:04company. So yes, learn things from
- 22:07project and as well learn new things.
- 22:09And this is what I'm going to show you
- 22:10now. All right. Now the first thing that
- 22:12I want you to learn is about the data
- 22:14security basics. Now the thing is as a
- 22:16data engineer is totally fine if you
- 22:18break a code or a pipeline. It is not
- 22:21that catastrophic for the company. But
- 22:22if there is like security issue, it's
- 22:24going to be a big headache for you. Even
- 22:26if you are a junior, I don't know
- 22:27whether you follow the news or something
- 22:28like that, but from time to time you're
- 22:30going to see some headlines about code
- 22:32of a specific company was in public or
- 22:35the data leak. So a small security
- 22:37mistake that you might do can bring a
- 22:39big issue for the company and you can
- 22:40prevent that by having some security
- 22:43awareness like for example don't put
- 22:44credentials like password access tokens
- 22:47in the code. Don't push your code in the
- 22:49public GitHub only like small things in
- 22:52order to work safely and to write a
- 22:54secure code. Now another thing that you
- 22:56can learn at this phase is the Apache
- 22:58Cafka. It is a streaming technology that
- 23:00moves the data in real time. Instead of
- 23:02moving the data in big chunks and big
- 23:05batches once a day or twice, instead
- 23:07we're going to have like events,
- 23:09messages, stream. So learn the theory
- 23:11about it and as well how to use code in
- 23:14order to interact with Kafka and consume
- 23:16data from it. Now moving to the next
- 23:18skill, it is all about learning the
- 23:20cloud. So here we have different options
- 23:22either learn Azure or AWS of course if
- 23:25your project is using RD cloud then
- 23:27learn exactly that platform otherwise I
- 23:29might recommend you to learn the
- 23:31Microsoft Azure so again you don't have
- 23:33to master it just learn the theory and
- 23:35learn some basics like how the data is
- 23:37stored in the cloud what are the
- 23:39different services that we can use in
- 23:41the Azure how to build a pipeline and as
- 23:43well learn about how to be cost
- 23:46efficient in the cloud now the first one
- 23:47I thought about it a lot you have to
- 23:49start learning about AI engineering. The
- 23:52thing is recently the game changed for
- 23:53data engineers. We were just providing
- 23:56data for analytical use cases maybe for
- 23:58PowerBI or like classical machine
- 24:01learning but now there is like a huge
- 24:03new use case for the data. So now we are
- 24:05providing data for AI topic and now here
- 24:08things might get mixed between data
- 24:10engineering and AI engineering. Now
- 24:12companies are expecting over the time
- 24:14that you can do both. I'm going to say
- 24:16it's going to be amazing for your career
- 24:18as a data engineer to start learning
- 24:20those topics. So learn the theory, what
- 24:22are the tools are there. I have as well
- 24:24a full road map on AI engineering. You
- 24:26can follow it. So as you can see in this
- 24:28phase, we are expanding your skills and
- 24:31the tools that you are learning. So
- 24:33again keep learning from your mistakes,
- 24:35from your job, from your projects. But
- 24:37again keep your eye open on the new
- 24:39technology and the new skills that might
- 24:41be required from you as a data engineer.
- 24:43And it is really simple. just learn the
- 24:45theory, the basics and you don't have to
- 24:47get certified or whatever. So that's all
- 24:49for the junior data engineer phase and
- 24:51now we're going to go to the next one,
- 24:53the senior data engineer. And now my
- 24:55friend, in this stage your role going to
- 24:57completely change. You are not longer
- 24:59like required to execute tasks that are
- 25:02defined from someone else. Now we're
- 25:04going to trust you with complex
- 25:05problems. So now you have to make
- 25:07decisions. You have to take
- 25:09responsibilities that goes beyond like
- 25:12implementing a ticket. So now the
- 25:13mindset is very simple but yet very
- 25:16powerful. You have to start delegating
- 25:18simple tasks for the other junior data
- 25:20engineers but at the same time you start
- 25:23taking ownership of complex problems and
- 25:26you have to help the other junior data
- 25:28engineers in order to grow. So you have
- 25:30to work as a mentor for them. So now let
- 25:32me show you the skills that are required
- 25:34at this phase. Now the first one is
- 25:37reviewing the code. So you have to work
- 25:39as investigator. It's not all about like
- 25:41catching a syntax mistake. It's all
- 25:43about improving the overall quality of
- 25:46the code and to start writing maybe
- 25:48guidelines and standards that the others
- 25:50must follow. Then you're going to review
- 25:52whether they are following the rules and
- 25:54the standards. So not only you have to
- 25:56learn how to review the code as well,
- 25:58you have to learn how to give feedback
- 26:00to the whole team in order for them as
- 26:02well to write better code over the time.
- 26:04Now the next skill or let's say the role
- 26:06that you have to do is mentoring and
- 26:08technical leadership. So as I said you
- 26:10have to go and guide different data
- 26:12engineers. You have to learn as well how
- 26:14to quickly solve any blockers at the
- 26:17project and start like answering
- 26:19questions from multiple data engineers
- 26:21at the same time. So you're going to
- 26:22find yourself at this phase that you are
- 26:24helping others doing their job more than
- 26:27like doing your own stuff. So here going
- 26:29to be a little bit complicated where you
- 26:31have to balance between helping others
- 26:33doing the job and at the same time you
- 26:35have as well to solve your complex
- 26:37tasks. But of course don't underestimate
- 26:39helping the others because you are
- 26:41multiplier. Now the others going to move
- 26:43the project faster safer with your help.
- 26:45Now the next skill really you have now
- 26:47to go advanced in mastering how to
- 26:50optimize the performance and the cost.
- 26:52So you have to start caring about how
- 26:54much things cost and how fast they run.
- 26:57You need to balance between the speed
- 26:58and the budget in order to design a
- 27:00system that scales without wasting a lot
- 27:03of money. Now another skill that you
- 27:04have to go now deeper and learn the
- 27:07advanced things about it, it is how to
- 27:09do data modeling because you design
- 27:10models that should scale. So you have to
- 27:13learn how to design models that are
- 27:15scalable and at the same time still easy
- 27:18to use. This is something that many
- 27:20junior developers cannot do. And we're
- 27:22going to count on you the senior
- 27:23developer to solve this dilemma by
- 27:25building a solid data model. And
- 27:27finally, I have here something called
- 27:28system design and architectural
- 27:30thinking. So here you have always to
- 27:32think about the whole system end to end.
- 27:34And if you decide to change something in
- 27:36the system, we going to count on you to
- 27:38understand the impact of this change
- 27:40because you are one of the few people
- 27:42that can understand the whole process
- 27:44end to end. So your opinion going to be
- 27:46asked many times about decisions that
- 27:49might affect the whole system in the
- 27:51future. So as you can see in this phase
- 27:52as a senior data engineer it's not all
- 27:54about like learning every tool. It's all
- 27:57about like judgment about ownership and
- 27:59helping your team building better
- 28:01systems. By the way as I was speaking
- 28:03about like searching for job as a junior
- 28:05data engineer here the game is totally
- 28:08changed because now you have enough
- 28:09skills in order to start picking the
- 28:12right company with the right role. So if
- 28:14you are working at company and you are
- 28:16like more than two years keep your eye
- 28:18open on your dream job on your dream
- 28:21position and once you find the right
- 28:23moment then apply for the job and switch
- 28:25but if you are totally happy you don't
- 28:26have to do that of course the rule says
- 28:28don't stay at the same project more than
- 28:315 years try to find like another project
- 28:33at the same company. So now let's go to
- 28:36the last phase of data engineering. Now
- 28:38the thing is there is like not only one
- 28:40path after like senior data engineer you
- 28:42can go as a manager maybe as a data
- 28:45product owner or project lead but one of
- 28:48the coolest role that you can take over
- 28:50is the data architect. So now the
- 28:52perspective really changes because up
- 28:54until now most of your work was all
- 28:56about like building things. You go in a
- 28:59project and you build a data systems but
- 29:01as a data architect the question totally
- 29:03change. So you are no longer thinking
- 29:05about how to build this project. Instead
- 29:08you think about how should this system
- 29:09be designed in the first place. The main
- 29:11goal here is really interesting is
- 29:13seeing the whole big picture not only
- 29:16like one pipeline, one project, one
- 29:18team. You have to see the entire data
- 29:21platform across the whole company or
- 29:23maybe only one sector. So that means
- 29:26your scope is not anymore narrowed to
- 29:28the implementation of one project. So
- 29:30you have to see the whole thing end to
- 29:32end and how everything is interconnected
- 29:34to each others and you have to start
- 29:36thinking about the long-term direction
- 29:38>> spin. I'm trying to confuse your sense
- 29:40of direction.
- 29:41>> So your job here is not anymore like
- 29:43sitting behind the screen and writing a
- 29:45code every day. Instead it's all about
- 29:47discussions, decisions and designs. So
- 29:50now there is a lot of things about this
- 29:51role but I'm going to show you few
- 29:53skills that are required as fundamentals
- 29:55for this role. Now the first thing is
- 29:57about decision making and influence. You
- 29:59as a data architect, you make long-term
- 30:01decisions. You have to explain why you
- 30:04decided for this direction and you have
- 30:06as well to influence other teams and
- 30:08departments to go in the direction that
- 30:10you have decided on. It's not about how
- 30:12to execute it exactly, it's about the
- 30:14direction. Now, another skill that you
- 30:16have to develop is to do the alignment
- 30:18between the business and the technology.
- 30:20So, this is really a big thing because
- 30:22you have to have understanding about a
- 30:23lot of technologies and as well deep
- 30:25understanding about your company, about
- 30:27the business. and they're going to ask
- 30:29you to do this mapping, this alignment.
- 30:31Now to the next skill, it's all about
- 30:33the data governance and the ownership.
- 30:35So part of your job is to define the
- 30:37rules, the responsibilities, the
- 30:39standard because you're going to have
- 30:40like different teams working on
- 30:41different types of data and shapes. And
- 30:43you're going to define for the company
- 30:45how the shape of the data must look like
- 30:47to make the data trusted across multiple
- 30:49teams. Now moving on to the last one,
- 30:51the platform level architecture. So
- 30:52you're going to design like let's say a
- 30:54blueprint or a data platform that many
- 30:57different teams depends on like for
- 30:59example you're going to say you know
- 31:00what we're going to build a data
- 31:01lakehouse for the entire company or
- 31:04maybe you're going to adopt the data
- 31:06mesh where you're going to say you know
- 31:07what for each domain we should produce
- 31:09data product so you have to put the
- 31:11blueprint and the other teams must
- 31:13follow it. So as you can see as a data
- 31:14architect your job is no longer learning
- 31:17how to code or how to build a pipeline.
- 31:19Now you are discussing tradeoffs. You
- 31:22are making decisions and you are like a
- 31:24bridge between multiple teams and
- 31:26multiple departments where you define
- 31:28standards and blueprints for maybe the
- 31:30whole company and of course if the
- 31:31company very big then only for that
- 31:33department. Of course there are other
- 31:35passes as I said manager data product
- 31:37owner or maybe a data officer but from
- 31:40technical perspective this is the last
- 31:42thing that you can grow. Maybe the last
- 31:44role is that to leave the job and be
- 31:46content creator like me. All right, my
- 31:48friends. So, that was a long one. If
- 31:50you're still here, then I really
- 31:51apologize. But I really wanted to show
- 31:53the full big picture of data engineering
- 31:55career. Absolute beginner until the last
- 31:58shape of the data architects. And now if
- 32:00you are total beginner, I totally
- 32:02understand you if I lost you at the ends
- 32:04because learning all those stuff take
- 32:06years. I don't want to scare you. The
- 32:08first journey it could done really fast
- 32:10in one year or less. So it should not
- 32:12take years to get hired as a junior data
- 32:15engineer. But I'm talking about the next
- 32:17phases. This could take time. So I just
- 32:19wanted to show you the full road map. So
- 32:21maybe you can start using only the first
- 32:23part and after a while after you get a
- 32:25job then you go back to the road map and
- 32:27understand I have to do the next step.
- 32:29So it can sticks with you in different
- 32:31phases at your career as a data
- 32:33engineer. Now, if you like this type of
- 32:35content where I'm sharing my experience
- 32:36as a data engineer with you, then
- 32:38support the channel by subscribing,
- 32:40liking, and committing in order to reach
- 32:42nice people like you. And if you're
- 32:44still here, then thank you so much for
- 32:46watching. I'm going to see you in the
- 32:48next video. So, bye-bye. Bye-bye.
- 32:50>> Bye.
About this transcript
This page contains the full transcript of How I’d Learn Data Engineering in 2026 From Zero to Senior (Built by a Data Engineer) by Data with Baraa, generated from the public captions YouTube serves with the video. The transcript has 6,930 words across 952 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.