YouTube2Text

How I’d Learn Data Engineering in 2026 From Zero to Senior (Built by a Data Engineer) — Transcript

by Data with Baraa · 6,930 words · 952 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Hey friends. So now last year I shared
  2. 0:02with you a road map on how to become a
  3. 0:04data engineer and honestly I didn't
  4. 0:06expect the reaction it got. Many of you
  5. 0:08followed the road map step by step and
  6. 0:10even some shared with us that they
  7. 0:12managed to get a job. So congrats that
  8. 0:15was really amazing to see. So now what I
  9. 0:17have done I went back to that road map
  10. 0:19and I reviewed every part of it. And
  11. 0:22after that not only I updated the
  12. 0:24content based on the market but also I
  13. 0:27extended the road map. This time it
  14. 0:29doesn't stop at getting your first job.
  15. 0:32It goes further in what actually happens
  16. 0:34after you get hired. So that means how
  17. 0:37to grow inside a company and what are
  18. 0:39the things that you have to learn there.
  19. 0:41So my friend in this video I'm going to
  20. 0:42walk you through the full journey from
  21. 0:45the absolute beginner to become a senior
  22. 0:47data engineer. And by the way if you are
  23. 0:49new here then welcome. My name is Bar
  24. 0:51and I have 17 years industry experience
  25. 0:54as a data engineer. I led one of the
  26. 0:56biggest data engineering and AI project
  27. 0:58at Mercedes-Benz and in this channel I'm
  28. 1:01sharing everything that I know from the
  29. 1:03industry. So now let's deep dive into
  30. 1:05the road map.
  31. 1:06>> Wow, you just dive right in.
  32. 1:08>> Okay. So now like the previous road map
  33. 1:10I have built everything using notion. It
  34. 1:12is like a template for you. You can go
  35. 1:14and download it or duplicate it in order
  36. 1:16to add your own notes and as well you
  37. 1:19can go and build your own timeline
  38. 1:20around it. And as well it is a living
  39. 1:22template. So each time I find something
  40. 1:24new, I'm going to go and update it. Of
  41. 1:26course, you can find the link in the
  42. 1:27description. Okay. So now I have
  43. 1:29splitted the whole thing into two
  44. 1:31journeys and inside each journey there
  45. 1:33are phases and inside the faces there
  46. 1:35are steps. So now the first one is the
  47. 1:37hiring journey. This part focuses on
  48. 1:39learning just enough skills to get you
  49. 1:42your first data engineering job. So
  50. 1:44we're going to be focused and learn only
  51. 1:45the things that we need without wasting
  52. 1:47a lot of time. The second one is the
  53. 1:50growth journey. This is what happens
  54. 1:52after you got hired. So how to grow on
  55. 1:54the company? What are the things that
  56. 1:57you have to add and learn as you are
  57. 1:59working? Because my friends, we never
  58. 2:01stop learning even after we get a job.
  59. 2:03So this is the full big picture and now
  60. 2:05we're going to deep dive into each
  61. 2:07journey into each skill. So let's start
  62. 2:09with the first one, the hiring journey.
  63. 2:11So we're going to scroll down over here.
  64. 2:12So now again the hiring journey, it's
  65. 2:14all about learning all the skills that
  66. 2:16we need to take you from zero until you
  67. 2:19get hired. Now we start with the phase
  68. 2:21one I call it the absolute beginner. So
  69. 2:23here it's not about learning any tool or
  70. 2:25platform. It's all about understanding
  71. 2:28what is data engineering and make a
  72. 2:30decision at the end. And by the way if
  73. 2:32you go and click on this you're going to
  74. 2:33find more details inside the road map.
  75. 2:36I'm going to put more context and guides
  76. 2:38inside each skill and as well at the end
  77. 2:40you're going to find resources. So now
  78. 2:41what I want you to do at the start is to
  79. 2:44understand what is data engineering. So
  80. 2:46what the role looks like inside a
  81. 2:48company and what kind of problems and
  82. 2:50tasks we solve every day and then you
  83. 2:53have to start comparing and
  84. 2:54understanding what are the differences
  85. 2:55between data engineers, data analyst,
  86. 2:58data scientist just have some basic
  87. 3:00understanding what are those different
  88. 3:02data roles and of course I cannot now
  89. 3:04deep dive into each skill and what this
  90. 3:06means because this might take ages. This
  91. 3:08is your job to do that for each skill.
  92. 3:11But as a data engineers we write a code
  93. 3:13to move the data from the sources. Then
  94. 3:15we do some transformations,
  95. 3:17preparations, clean up and prepare
  96. 3:19everything in order to load it in the
  97. 3:21target so that we can do analytics. And
  98. 3:23by the way, I have here some nice
  99. 3:25resources for you. I went deep dive into
  100. 3:27this role as I explained how to learn
  101. 3:29Python for data engineers. And in this
  102. 3:32data movie, I have explained all
  103. 3:33different rules. All what you have to do
  104. 3:35is to understand what data engineering
  105. 3:37means. So this is the first step. Then
  106. 3:39the second step which is the most
  107. 3:41important one. After you understood what
  108. 3:43we do, you have to make an honest
  109. 3:45decision and ask yourself, do I actually
  110. 3:48want and enjoy doing such activities as
  111. 3:52a data engineer every day, every week?
  112. 3:54If it's no, then it's totally fine. You
  113. 3:56can just close the video now and that's
  114. 3:58it. But if you say this is the right job
  115. 4:00for me, then you have to commit to this
  116. 4:03path. So this decision, my friend, is
  117. 4:05the most important thing. It's going to
  118. 4:06save years for you. You have to forget
  119. 4:08all the hype and the salaries about the
  120. 4:10data engineering. this something that
  121. 4:12you want and you love to do. If you say
  122. 4:14yes, then welcome to the club and we're
  123. 4:16going to go to the next phases. Okay,
  124. 4:18let's go to the face one. Here the real
  125. 4:20work starts. On this face, it's all
  126. 4:22about building your foundations. And the
  127. 4:24goal here is very simple. I want it to
  128. 4:27feel very normal working with data and
  129. 4:29writing a code. That means we will not
  130. 4:31deep dive into any data engineering
  131. 4:34topics yet. It's going to be very
  132. 4:35overwhelming. Instead, reduce the noise
  133. 4:38and just focus on only those three
  134. 4:40skills. And there is like one rule that
  135. 4:42I want you to follow. You have always to
  136. 4:44practice way more than consuming. And I
  137. 4:46call it the 80/20 rule. 80% practicing
  138. 4:50and only 20% consuming. If you just sit
  139. 4:52down and start watching a course without
  140. 4:54doing anything, then you are not
  141. 4:56learning anything. And the second rule
  142. 4:58here is that you have to learn one skill
  143. 5:00at a time. So never learn two things
  144. 5:03together. Don't go and learn SQL
  145. 5:04together with Python. Only one thing at
  146. 5:07a time. And another thing that I want
  147. 5:09you to focus in this phase try to
  148. 5:10understand more than memorizing the
  149. 5:12syntax. I want you to understand how
  150. 5:14things works behind the scenes and how
  151. 5:16the logic works because who needs syntax
  152. 5:19we have now AI right? And of course
  153. 5:20speaking about AI try to avoid it for
  154. 5:23now. You can use it in order to explain
  155. 5:25something for you but not to write a
  156. 5:27code for you. Only with a real struggle
  157. 5:29you're going to learn. So now about the
  158. 5:30skills the first thing that you have to
  159. 5:32learn is SQL. It is the language that
  160. 5:35you use in order to talk to the data.
  161. 5:38This is the number one skill in any data
  162. 5:40role. So you can use it as a data
  163. 5:42engineer in order to do a lot of data
  164. 5:44transformations inside data lakeouses
  165. 5:47and you can use it as well to build a
  166. 5:48business logics and stuff like that. I
  167. 5:50have here for you a list of all the
  168. 5:52things that you have to learn. So learn
  169. 5:54the basics about the select statements,
  170. 5:56how to filter the data, how to do data
  171. 5:58transformations and cleaning. You have
  172. 5:59to learn a lot of scale functions, how
  173. 6:01to do data aggregations and how to join
  174. 6:04tables. Then comes advanced topics like
  175. 6:06the window functions, the CTE,
  176. 6:08subqueries, views. Now, of course, I can
  177. 6:10totally recommend my SQL course. It got
  178. 6:122 million views in just few months. So,
  179. 6:15the first thing is SQL. Then you're
  180. 6:17going to go to the next one is the
  181. 6:19Python. This is the only and the core
  182. 6:21programming language that you have to
  183. 6:23learn as a data engineer. And my friend,
  184. 6:25you have to master it. If you cannot
  185. 6:26write a good clean Python code then I
  186. 6:29really don't think you can do any data
  187. 6:31engineering and here it is really tricky
  188. 6:33because there are a lot of things to
  189. 6:35learn about Python that's why I made a
  190. 6:37very compacted list for you to learn it
  191. 6:39it's not a lot actually so you have to
  192. 6:41learn about the variables and the data
  193. 6:42types the data structures very important
  194. 6:45the list and the dictionaries the
  195. 6:47control flow the if else loop how to
  196. 6:49write a functions and now comes
  197. 6:51something really important how to work
  198. 6:53with files especially the CSV and the
  199. 6:55JSON how to do error handling, do
  200. 6:58logging, connect to different systems
  201. 7:00like databases, APIs and Kafka. Of
  202. 7:02course, I explained more in details why
  203. 7:05we have to learn all those stuff. You
  204. 7:06can check the video how to learn Python
  205. 7:08as a data engineer. And as well, many of
  206. 7:10those topics are already covered in my
  207. 7:12boot camp. You can check as well the
  208. 7:13link over here. So, you have to master
  209. 7:15this language. And now comes the third
  210. 7:17skill which is really nice one. You have
  211. 7:19to learn about Git and GitHub. Since
  212. 7:21your job is to write code, then this
  213. 7:24comes as automatic thing. You have to
  214. 7:26learn as well how to work with the
  215. 7:27GitHub. So it is the way where we save
  216. 7:29our code, track our changes, we can
  217. 7:32collaborate with others and without this
  218. 7:34of course you cannot do any real
  219. 7:35projects at companies. So you just have
  220. 7:37to learn the basics like how to create a
  221. 7:39repository, how the workflow works, add,
  222. 7:42commit, push, pull. You can learn it
  223. 7:44really fast. Those are the foundations,
  224. 7:46the three core skills that you have to
  225. 7:47learn and again one after another. And
  226. 7:50my friends, if you want to break into
  227. 7:51data engineering and learn all the
  228. 7:53skills that I mentioned in one place,
  229. 7:55then I always recommend data camp. When
  230. 7:57learning key data engineering skills,
  231. 7:59it's hand down one of the most
  232. 8:01interactive handson platform out there.
  233. 8:04The associate data engineering in SQL
  234. 8:06track is perfect starting point. You
  235. 8:09will go from understanding the basics of
  236. 8:11data engineering and ETL ELT workflows
  237. 8:14to master posress SQL SQL joins and data
  238. 8:17warehouse principles with snowflake. You
  239. 8:20will also dive into database designs,
  240. 8:22normalization and even build real world
  241. 8:25projects like analyzing the students
  242. 8:27mental health and London travel network
  243. 8:29in order to start building your
  244. 8:30portfolio. And once you completed the
  245. 8:32courses, you can try to pass an industry
  246. 8:35recognized data engineering career
  247. 8:37certification which is a huge boost when
  248. 8:40you are searching for job. So if you are
  249. 8:41serious about becoming a data engineer,
  250. 8:43check the link below. I even can get you
  251. 8:4525% discounts. Now moving on to the
  252. 8:48second phase, the data engineering core.
  253. 8:50It is where you're going to get
  254. 8:51specialized and start feeling like a
  255. 8:53real data engineer. So the goal here is
  256. 8:56not to learn how to code. It's all about
  257. 8:58how to think like a data engineer and to
  258. 9:01learn how data systems are built end to
  259. 9:03end. You have to experience the whole
  260. 9:05process. And I recommend you here to
  261. 9:07focus on the concepts more than the
  262. 9:09tools, the platforms and the UIs. So
  263. 9:11don't rush on learning how to do things.
  264. 9:14Instead learn the why concepts behind
  265. 9:16those stuff and that's why the first
  266. 9:18thing that you have to do in this phase
  267. 9:19is to learn some theory. Here we are not
  268. 9:22learning any tool. We are not learning
  269. 9:24any like programming language. We just
  270. 9:26want to research, read and understand
  271. 9:28the concept. Like for example here, what
  272. 9:30is a data pipeline? What are the
  273. 9:32differences between batch processing and
  274. 9:35stream processing? What is an ETL, an
  275. 9:38ELT? What are the differences between
  276. 9:40them? And some basics about the data
  277. 9:42architectures like the data lake, data
  278. 9:44warehouse and the most important of
  279. 9:45course the data lakehouse. try to
  280. 9:47understand as well what is data
  281. 9:49moduling. What is the theory behind the
  282. 9:51Hadoop and the Spark, how the Apache
  283. 9:54Cafka work. It is just theory. We are
  284. 9:56not practicing or anything. And of
  285. 9:58course, many of those terms I already
  286. 10:00covered in for example the data movie.
  287. 10:02So go over there and start checking the
  288. 10:04data engineering part. Other than that,
  289. 10:06I have as well an article about 200
  290. 10:09terms that we use in data worlds and the
  291. 10:12part about data processing and data
  292. 10:13engineering I just posted here in the
  293. 10:15notion. So get yourself familiar with
  294. 10:18the data engineering terminology. Now
  295. 10:20the next step we have to learn about
  296. 10:22something called data bricks. This is
  297. 10:24not like a tool or let's say programming
  298. 10:27language. It is a platform that you can
  299. 10:29do many things inside it. Data
  300. 10:31analytics, data engineering and data
  301. 10:33science totally based on the cloud using
  302. 10:35Azure and AWS. But it is a modern web
  303. 10:38but it is a modern platform that allows
  304. 10:41big companies to scale. So it can really
  305. 10:43process huge amount of data and it is
  306. 10:45getting a lot of attention. My last five
  307. 10:48years I built completely projects using
  308. 10:50only this one platform. If you learn it,
  309. 10:52it's going to be big plus on your
  310. 10:54resume. And here in this step, I just
  311. 10:55want you to read as well about what is
  312. 10:58data bricks, why the company exist, what
  313. 11:00are the main components inside the
  314. 11:02platform and start learning the basic
  315. 11:04terminology that we use in data bricks
  316. 11:06like what is a notebook, what a cluster.
  317. 11:08So just a theory and exploring the
  318. 11:11platform and the tool. That's because
  319. 11:13the deep dive going to happen in the
  320. 11:14next step where we're going to learn
  321. 11:16about spark and pi spark. This is the
  322. 11:18main engine that we use in data bricks
  323. 11:21in order to process massive amount of
  324. 11:23data. It help us to scale by using the
  325. 11:26distributed processing. So start by
  326. 11:28understanding what is spark and why we
  327. 11:30need distributed processing. What are
  328. 11:32the data frames and how they work and
  329. 11:34then start to learn how to use it inside
  330. 11:36for example data bricks. Just learn here
  331. 11:38the basics. Don't go deep dive into like
  332. 11:40each detail about the bicep spark
  333. 11:42because this going to take a lot of
  334. 11:43time. Just learn how to read data, how
  335. 11:45to process it and then how to write it
  336. 11:47back. Then the other stuff you're going
  337. 11:48to learn it as you become a senior data
  338. 11:50engineer. So now my friend I can tell
  339. 11:52you actually this is the last skill that
  340. 11:55you need to learn in this journey. This
  341. 11:57is very simple. So SQL, Python, Git and
  342. 11:59GitHub, data bricks and pispark. With
  343. 12:02that you can understand exactly why we
  344. 12:04have learned all those skills. You don't
  345. 12:05have to go and build 20 projects. Just
  346. 12:08make one clean and strong project. So
  347. 12:10one idea that I really recommend you to
  348. 12:12do is to go and build a data lakehouse.
  349. 12:15You're going to use data bricks,
  350. 12:16pispark, python, sql, everything in
  351. 12:18order to move the data into like three
  352. 12:21layers. So you're going to enter the
  353. 12:22data as very messy raw data. Then you're
  354. 12:25going to start transforming it, cleaning
  355. 12:27it until it goes to the last layer where
  356. 12:30everything looks ready. Once you do
  357. 12:31this, you're going to really experience
  358. 12:33the whole process end to end. I have a
  359. 12:35very similar project but using an SQL to
  360. 12:38build a data warehouse. It is very
  361. 12:40similar project where you're going to go
  362. 12:41and process the data in different layers
  363. 12:43but you can adapt it and use completely
  364. 12:45pispar and data bricks in order to do
  365. 12:47the project. Now again as I told you you
  366. 12:49have to do only one project because
  367. 12:51there's a lot of things to do. You have
  368. 12:53to do it right. You have to document it
  369. 12:55correctly in the GitHub. You have to
  370. 12:56learn how to explain it as well in the
  371. 12:58interviews. Make very solid one and
  372. 13:01everything completed and you still have
  373. 13:02time. You can start maybe a second
  374. 13:04project. So that's it actually for this
  375. 13:06phase. If you get it right, then you are
  376. 13:08not longer just learning data
  377. 13:10engineering. Actually, you can do the
  378. 13:12job at companies. All right, so that's
  379. 13:13all for the phase two. And now we're
  380. 13:15going to go to a scary phase where I
  381. 13:17want you to get hired. This is to be
  382. 13:19honest where most of the people like
  383. 13:21stuck or slow down or stop. It's all
  384. 13:23because they don't feel confident enough
  385. 13:25to start applying for jobs. They always
  386. 13:27going to feel like there is something
  387. 13:28missing. There are few skills and tools
  388. 13:30I didn't learn yet. I see someone in
  389. 13:32LinkedIn like listing a lot of skills
  390. 13:34that I don't know about it. So they are
  391. 13:37always stuck in this loop where they go
  392. 13:39from one course to another course and
  393. 13:41they will never start applying for jobs.
  394. 13:43And this is totally wrong because my
  395. 13:44friend you will never feel 100% ready to
  396. 13:48start applying. If you feel like 60 or
  397. 13:5070% then actually you are ready. So in
  398. 13:52this phase you have to start preparing
  399. 13:54yourself to get hired. You have to
  400. 13:56search for jobs and of course keep
  401. 13:58learning. Now let's talk about the first
  402. 13:59thing that you might think about is to
  403. 14:01get certified. Now the thing is if it is
  404. 14:03your first job the recruiters might look
  405. 14:06whether you have a certificate or not
  406. 14:07and if they see one maybe they get
  407. 14:09closer but I can tell you in many
  408. 14:12scenarios it is not the number one
  409. 14:14reason why someone talks to you. So in
  410. 14:16most cases the project has more weights
  411. 14:19than certificates because certificates
  412. 14:21does not say that you are a data
  413. 14:23engineer but if I check your project I
  414. 14:25can see exactly how you went through the
  415. 14:27process and I can trust it more that you
  416. 14:29can do the job. So projects has more
  417. 14:32weight than certificates and another
  418. 14:34mistakes that you might do is that you
  419. 14:36start collecting and chasing a lot of
  420. 14:38certificates and this is really huge
  421. 14:40waste of time and money. So now what I
  422. 14:42recommend you is to pick one relevant
  423. 14:45certificate like for example in this
  424. 14:46road map as you can see we learned about
  425. 14:48the pi spark the data bricks. If you
  426. 14:50have the money and the time then I
  427. 14:52recommend you to take certificates from
  428. 14:54data bricks. It is a huge plus to your
  429. 14:56story. So you have learned all the
  430. 14:58topics about the pi spark and the data
  431. 15:00bricks. You have done a project and as
  432. 15:01well you have a certificate. Now the
  433. 15:03thing is with all thoseations they are
  434. 15:05always connected to specific platform
  435. 15:08like if you want to get certified from
  436. 15:10IBM you have to learn a lot of things
  437. 15:12about the IPM platform. The same thing
  438. 15:14goes for Google or Azure or AWS. You
  439. 15:17cannot go and start get certified by all
  440. 15:19those different platforms. Then you have
  441. 15:21to learn all of them. And actually this
  442. 15:23is something that we never have to do.
  443. 15:25So my friend, don't go and chase all
  444. 15:27those different certificates. It's
  445. 15:29really going to waste your time, your
  446. 15:30money, your energy. So instead, do one
  447. 15:33solid data engineering project and maybe
  448. 15:35get only one certification that is
  449. 15:38relevant to your story, to your path. So
  450. 15:40that's all about the certification.
  451. 15:42Don't overthink it. Now let's go to the
  452. 15:44next step. Here you have to build one
  453. 15:46clean resume. So here there are like a
  454. 15:48lot of best practices about how to build
  455. 15:50a resume. Just from my experience, I can
  456. 15:53tell you make it easy to read. Make it
  457. 15:55clean. Don't over complicate it. Don't
  458. 15:57put the skills that you wish to learn.
  459. 16:00Just be honest about what you actually
  460. 16:02have done. And make it just easy to
  461. 16:04read. So we have to spend only few
  462. 16:06seconds reading it. Instead of like
  463. 16:08minute, just make the first version. And
  464. 16:10of course keep improving it as you start
  465. 16:12doing interviews. You start getting like
  466. 16:14feedbacks and you start adjusting it.
  467. 16:16And again like the certificate, a
  468. 16:18perfect resume will not get you hired.
  469. 16:20It is just the start of the conversation
  470. 16:22between us. Now the third thing that you
  471. 16:23have to do is to optimize your LinkedIn
  472. 16:25profile. So LinkedIn is not like social
  473. 16:28media. It's not your Facebook. And as
  474. 16:29well I'm not asking you to be an
  475. 16:31influencer on LinkedIn. Just makes it
  476. 16:33looks professional. So professional
  477. 16:35photo, some nice header statement, the
  478. 16:37about section, the skills that you have
  479. 16:39learned. Just start connecting with
  480. 16:41other people like maybe me LinkedIn as
  481. 16:43well. Maybe start posting about your
  482. 16:45progress. As you learn new skills, you
  483. 16:47do new projects. Just make few
  484. 16:49celebration about what you are doing.
  485. 16:51And the most important thing as well in
  486. 16:52LinkedIn, you have to feature your
  487. 16:55portfolio project. And this is exactly
  488. 16:57the next step. We have to build a
  489. 16:59portfolio. And of course, there are like
  490. 17:00multiple ways on how to build a
  491. 17:01portfolio. But for us as data engineers,
  492. 17:03things are easy. The best way is to have
  493. 17:06a solid profile in GitHub. So put the
  494. 17:09whole project in the GitHub. And here
  495. 17:11you have to make sure that the code is
  496. 17:13clean, it has comments, you are
  497. 17:15following the standards on styling and
  498. 17:17formatting your code. And by using the
  499. 17:19readme, you explain maybe who are you
  500. 17:21and what is the projects all about, what
  501. 17:24are the use case, the outcome. Think
  502. 17:26about it like your resume, your
  503. 17:28certificate. This is really important.
  504. 17:30Make clean profile in GitHub and really
  505. 17:32you're going to stand out of the crowds.
  506. 17:34Of course, you can go like step deeper
  507. 17:36where you build a personal website which
  508. 17:38is I'm going to say it is optional and
  509. 17:39of course if you don't know how to build
  510. 17:41a website and you don't want to invest
  511. 17:43money or time on it, then it's enough to
  512. 17:45use in notion. So you can build your
  513. 17:47whole portfolio in the notion and like
  514. 17:49introduce yourself as your personal
  515. 17:51website. It is totally for free but
  516. 17:53again this is optional. Way more
  517. 17:55important is the GitHub profile. And
  518. 17:57once you have all those stuff you start
  519. 17:59applying for jobs.
  520. 18:00>> I'd like to apply for a job.
  521. 18:01>> Okay. So now I understand totally this
  522. 18:03is an emotional topic and I can tell you
  523. 18:06the job market is horrible but it is
  524. 18:08totally doable. So you have to have the
  525. 18:11right mindset about it. Now the first
  526. 18:13thing is that it is your first job. So
  527. 18:15it must not be perfect. So what I'm
  528. 18:17trying to say here that don't be picky.
  529. 18:19Don't go and say you know what I'm going
  530. 18:21to apply only for those two three
  531. 18:23companies or only for very specific
  532. 18:25role. My friend this is something that
  533. 18:27you do if you are a senior data engineer
  534. 18:30if you are like maybe someone like me.
  535. 18:31So don't compare yourself with a senior
  536. 18:34data engineers because the game there is
  537. 18:36totally different as a first job. And I
  538. 18:39can tell you story from me as I wanted
  539. 18:41to change my job from my first job to
  540. 18:43the second job. I applied over hund
  541. 18:45times and the job market was way better
  542. 18:47than now. But I don't want to demotivate
  543. 18:50you. It is totally doable. It's just the
  544. 18:52game changed and you have to have the
  545. 18:54right mindset. Previously I didn't build
  546. 18:56like portfolio projects and get
  547. 18:58certified and all those stuff. Now you
  548. 18:59have to do more things in order to get
  549. 19:01hired. So now the process going to be
  550. 19:03like this. You have to prepare for the
  551. 19:05interview. So learn about the classical
  552. 19:07questions that's going to be asked for
  553. 19:09data engineers and as well very
  554. 19:10important learn how to explain your
  555. 19:12project. Explain the rough plan, the
  556. 19:15faces of your project, what you have
  557. 19:16done and what are the outcomes of this
  558. 19:19project. So learn how to tell it as a
  559. 19:21story end to end and start applying for
  560. 19:23job. See 100 200 times. Don't get
  561. 19:25emotional about it. It is not about you,
  562. 19:28nothing personal. And once you got
  563. 19:30invited to an interview, it is very
  564. 19:32important to understand and learn what
  565. 19:34happens during the interview. This is
  566. 19:36really a learning process. So you're
  567. 19:38going to learn how to answer questions,
  568. 19:40which questions you didn't answer
  569. 19:42correctly. And there is like one thing
  570. 19:43that I really like to see in the
  571. 19:45interviews that I can trust you with the
  572. 19:47job. I have to get the feeling that you
  573. 19:49are a builder, someone that's going to
  574. 19:51go and build things. So this is their
  575. 19:53process. You keep improving, you keep
  576. 19:55repeating and fix the weak spots until
  577. 19:57I'm sure you're going to get hired. So
  578. 19:59now that's all for this phase. And now
  579. 20:01let's say that after pushing a lot, you
  580. 20:03got accepted for your first data
  581. 20:05engineering job. And of course if this
  582. 20:06happens, then congrats. And I can tell
  583. 20:08you, you're going to go and enter a new
  584. 20:10journey. It is where you're going to
  585. 20:12grow on the job. So now I'm going to
  586. 20:13call this phase the junior data engineer
  587. 20:16phase. So now as well I want you to get
  588. 20:18the right mindset for this. Now you are
  589. 20:20in new project in your team and if you
  590. 20:22are lucky enough you might get an
  591. 20:24onboarding process. So someone can
  592. 20:26explain for you the project and the
  593. 20:27tools and then you can start getting
  594. 20:29your first tasks and start writing your
  595. 20:32first code of course simple tasks and
  596. 20:34then you might fix some issues from
  597. 20:36incident. And here I want you to
  598. 20:37understand at this phase it is totally
  599. 20:40okay to make mistakes. Actually this is
  600. 20:42the part of the process and there is a
  601. 20:44big difference between making mistakes
  602. 20:46and repeating the same mistake over and
  603. 20:49over. So my friend if you make mistake
  604. 20:51at the work it is totally fine but you
  605. 20:53have to learn from it. Try to not keep
  606. 20:55repeating the same mistake over and
  607. 20:57over. Doing mistakes it is totally okay.
  608. 20:59So don't be afraid of this. And another
  609. 21:01mistake that I found a lot from the
  610. 21:03juniors that they are afraid of asking.
  611. 21:05they are like shy. Don't be afraid of
  612. 21:07that. Keep asking. It doesn't mean that
  613. 21:09you don't know anything. If I go now
  614. 21:11into a new company and I get hired in
  615. 21:14projects, the first thing that I do, I
  616. 21:16keep asking them things. So, what are
  617. 21:18the source systems? Explain for me this
  618. 21:21business process. Explain for me this
  619. 21:23source system. What actually the data
  620. 21:25means in this specific table? So, keep
  621. 21:27asking questions at the start. And the
  622. 21:29third thing that might happen at that,
  623. 21:31yes, you're going to learn things from
  624. 21:32the job, but not everything. So this
  625. 21:34could happens a lot that you are working
  626. 21:36on projects that are using some outdated
  627. 21:39tools and you as data engineers of
  628. 21:41course you can learn those tools in the
  629. 21:43job but the thing is you are not
  630. 21:44learning the right skills for the
  631. 21:46future. Of course you have to learn from
  632. 21:48the job and as well to keep your eye
  633. 21:50open on how the market is moving. For
  634. 21:52example, if I just focused on my first
  635. 21:54job, I would be now a talent expert. It
  636. 21:57is an ETL tool and as well a database or
  637. 22:00expert. And I'm sure this will not be
  638. 22:02enough to me moving to any other
  639. 22:04company. So yes, learn things from
  640. 22:07project and as well learn new things.
  641. 22:09And this is what I'm going to show you
  642. 22:10now. All right. Now the first thing that
  643. 22:12I want you to learn is about the data
  644. 22:14security basics. Now the thing is as a
  645. 22:16data engineer is totally fine if you
  646. 22:18break a code or a pipeline. It is not
  647. 22:21that catastrophic for the company. But
  648. 22:22if there is like security issue, it's
  649. 22:24going to be a big headache for you. Even
  650. 22:26if you are a junior, I don't know
  651. 22:27whether you follow the news or something
  652. 22:28like that, but from time to time you're
  653. 22:30going to see some headlines about code
  654. 22:32of a specific company was in public or
  655. 22:35the data leak. So a small security
  656. 22:37mistake that you might do can bring a
  657. 22:39big issue for the company and you can
  658. 22:40prevent that by having some security
  659. 22:43awareness like for example don't put
  660. 22:44credentials like password access tokens
  661. 22:47in the code. Don't push your code in the
  662. 22:49public GitHub only like small things in
  663. 22:52order to work safely and to write a
  664. 22:54secure code. Now another thing that you
  665. 22:56can learn at this phase is the Apache
  666. 22:58Cafka. It is a streaming technology that
  667. 23:00moves the data in real time. Instead of
  668. 23:02moving the data in big chunks and big
  669. 23:05batches once a day or twice, instead
  670. 23:07we're going to have like events,
  671. 23:09messages, stream. So learn the theory
  672. 23:11about it and as well how to use code in
  673. 23:14order to interact with Kafka and consume
  674. 23:16data from it. Now moving to the next
  675. 23:18skill, it is all about learning the
  676. 23:20cloud. So here we have different options
  677. 23:22either learn Azure or AWS of course if
  678. 23:25your project is using RD cloud then
  679. 23:27learn exactly that platform otherwise I
  680. 23:29might recommend you to learn the
  681. 23:31Microsoft Azure so again you don't have
  682. 23:33to master it just learn the theory and
  683. 23:35learn some basics like how the data is
  684. 23:37stored in the cloud what are the
  685. 23:39different services that we can use in
  686. 23:41the Azure how to build a pipeline and as
  687. 23:43well learn about how to be cost
  688. 23:46efficient in the cloud now the first one
  689. 23:47I thought about it a lot you have to
  690. 23:49start learning about AI engineering. The
  691. 23:52thing is recently the game changed for
  692. 23:53data engineers. We were just providing
  693. 23:56data for analytical use cases maybe for
  694. 23:58PowerBI or like classical machine
  695. 24:01learning but now there is like a huge
  696. 24:03new use case for the data. So now we are
  697. 24:05providing data for AI topic and now here
  698. 24:08things might get mixed between data
  699. 24:10engineering and AI engineering. Now
  700. 24:12companies are expecting over the time
  701. 24:14that you can do both. I'm going to say
  702. 24:16it's going to be amazing for your career
  703. 24:18as a data engineer to start learning
  704. 24:20those topics. So learn the theory, what
  705. 24:22are the tools are there. I have as well
  706. 24:24a full road map on AI engineering. You
  707. 24:26can follow it. So as you can see in this
  708. 24:28phase, we are expanding your skills and
  709. 24:31the tools that you are learning. So
  710. 24:33again keep learning from your mistakes,
  711. 24:35from your job, from your projects. But
  712. 24:37again keep your eye open on the new
  713. 24:39technology and the new skills that might
  714. 24:41be required from you as a data engineer.
  715. 24:43And it is really simple. just learn the
  716. 24:45theory, the basics and you don't have to
  717. 24:47get certified or whatever. So that's all
  718. 24:49for the junior data engineer phase and
  719. 24:51now we're going to go to the next one,
  720. 24:53the senior data engineer. And now my
  721. 24:55friend, in this stage your role going to
  722. 24:57completely change. You are not longer
  723. 24:59like required to execute tasks that are
  724. 25:02defined from someone else. Now we're
  725. 25:04going to trust you with complex
  726. 25:05problems. So now you have to make
  727. 25:07decisions. You have to take
  728. 25:09responsibilities that goes beyond like
  729. 25:12implementing a ticket. So now the
  730. 25:13mindset is very simple but yet very
  731. 25:16powerful. You have to start delegating
  732. 25:18simple tasks for the other junior data
  733. 25:20engineers but at the same time you start
  734. 25:23taking ownership of complex problems and
  735. 25:26you have to help the other junior data
  736. 25:28engineers in order to grow. So you have
  737. 25:30to work as a mentor for them. So now let
  738. 25:32me show you the skills that are required
  739. 25:34at this phase. Now the first one is
  740. 25:37reviewing the code. So you have to work
  741. 25:39as investigator. It's not all about like
  742. 25:41catching a syntax mistake. It's all
  743. 25:43about improving the overall quality of
  744. 25:46the code and to start writing maybe
  745. 25:48guidelines and standards that the others
  746. 25:50must follow. Then you're going to review
  747. 25:52whether they are following the rules and
  748. 25:54the standards. So not only you have to
  749. 25:56learn how to review the code as well,
  750. 25:58you have to learn how to give feedback
  751. 26:00to the whole team in order for them as
  752. 26:02well to write better code over the time.
  753. 26:04Now the next skill or let's say the role
  754. 26:06that you have to do is mentoring and
  755. 26:08technical leadership. So as I said you
  756. 26:10have to go and guide different data
  757. 26:12engineers. You have to learn as well how
  758. 26:14to quickly solve any blockers at the
  759. 26:17project and start like answering
  760. 26:19questions from multiple data engineers
  761. 26:21at the same time. So you're going to
  762. 26:22find yourself at this phase that you are
  763. 26:24helping others doing their job more than
  764. 26:27like doing your own stuff. So here going
  765. 26:29to be a little bit complicated where you
  766. 26:31have to balance between helping others
  767. 26:33doing the job and at the same time you
  768. 26:35have as well to solve your complex
  769. 26:37tasks. But of course don't underestimate
  770. 26:39helping the others because you are
  771. 26:41multiplier. Now the others going to move
  772. 26:43the project faster safer with your help.
  773. 26:45Now the next skill really you have now
  774. 26:47to go advanced in mastering how to
  775. 26:50optimize the performance and the cost.
  776. 26:52So you have to start caring about how
  777. 26:54much things cost and how fast they run.
  778. 26:57You need to balance between the speed
  779. 26:58and the budget in order to design a
  780. 27:00system that scales without wasting a lot
  781. 27:03of money. Now another skill that you
  782. 27:04have to go now deeper and learn the
  783. 27:07advanced things about it, it is how to
  784. 27:09do data modeling because you design
  785. 27:10models that should scale. So you have to
  786. 27:13learn how to design models that are
  787. 27:15scalable and at the same time still easy
  788. 27:18to use. This is something that many
  789. 27:20junior developers cannot do. And we're
  790. 27:22going to count on you the senior
  791. 27:23developer to solve this dilemma by
  792. 27:25building a solid data model. And
  793. 27:27finally, I have here something called
  794. 27:28system design and architectural
  795. 27:30thinking. So here you have always to
  796. 27:32think about the whole system end to end.
  797. 27:34And if you decide to change something in
  798. 27:36the system, we going to count on you to
  799. 27:38understand the impact of this change
  800. 27:40because you are one of the few people
  801. 27:42that can understand the whole process
  802. 27:44end to end. So your opinion going to be
  803. 27:46asked many times about decisions that
  804. 27:49might affect the whole system in the
  805. 27:51future. So as you can see in this phase
  806. 27:52as a senior data engineer it's not all
  807. 27:54about like learning every tool. It's all
  808. 27:57about like judgment about ownership and
  809. 27:59helping your team building better
  810. 28:01systems. By the way as I was speaking
  811. 28:03about like searching for job as a junior
  812. 28:05data engineer here the game is totally
  813. 28:08changed because now you have enough
  814. 28:09skills in order to start picking the
  815. 28:12right company with the right role. So if
  816. 28:14you are working at company and you are
  817. 28:16like more than two years keep your eye
  818. 28:18open on your dream job on your dream
  819. 28:21position and once you find the right
  820. 28:23moment then apply for the job and switch
  821. 28:25but if you are totally happy you don't
  822. 28:26have to do that of course the rule says
  823. 28:28don't stay at the same project more than
  824. 28:315 years try to find like another project
  825. 28:33at the same company. So now let's go to
  826. 28:36the last phase of data engineering. Now
  827. 28:38the thing is there is like not only one
  828. 28:40path after like senior data engineer you
  829. 28:42can go as a manager maybe as a data
  830. 28:45product owner or project lead but one of
  831. 28:48the coolest role that you can take over
  832. 28:50is the data architect. So now the
  833. 28:52perspective really changes because up
  834. 28:54until now most of your work was all
  835. 28:56about like building things. You go in a
  836. 28:59project and you build a data systems but
  837. 29:01as a data architect the question totally
  838. 29:03change. So you are no longer thinking
  839. 29:05about how to build this project. Instead
  840. 29:08you think about how should this system
  841. 29:09be designed in the first place. The main
  842. 29:11goal here is really interesting is
  843. 29:13seeing the whole big picture not only
  844. 29:16like one pipeline, one project, one
  845. 29:18team. You have to see the entire data
  846. 29:21platform across the whole company or
  847. 29:23maybe only one sector. So that means
  848. 29:26your scope is not anymore narrowed to
  849. 29:28the implementation of one project. So
  850. 29:30you have to see the whole thing end to
  851. 29:32end and how everything is interconnected
  852. 29:34to each others and you have to start
  853. 29:36thinking about the long-term direction
  854. 29:38>> spin. I'm trying to confuse your sense
  855. 29:40of direction.
  856. 29:41>> So your job here is not anymore like
  857. 29:43sitting behind the screen and writing a
  858. 29:45code every day. Instead it's all about
  859. 29:47discussions, decisions and designs. So
  860. 29:50now there is a lot of things about this
  861. 29:51role but I'm going to show you few
  862. 29:53skills that are required as fundamentals
  863. 29:55for this role. Now the first thing is
  864. 29:57about decision making and influence. You
  865. 29:59as a data architect, you make long-term
  866. 30:01decisions. You have to explain why you
  867. 30:04decided for this direction and you have
  868. 30:06as well to influence other teams and
  869. 30:08departments to go in the direction that
  870. 30:10you have decided on. It's not about how
  871. 30:12to execute it exactly, it's about the
  872. 30:14direction. Now, another skill that you
  873. 30:16have to develop is to do the alignment
  874. 30:18between the business and the technology.
  875. 30:20So, this is really a big thing because
  876. 30:22you have to have understanding about a
  877. 30:23lot of technologies and as well deep
  878. 30:25understanding about your company, about
  879. 30:27the business. and they're going to ask
  880. 30:29you to do this mapping, this alignment.
  881. 30:31Now to the next skill, it's all about
  882. 30:33the data governance and the ownership.
  883. 30:35So part of your job is to define the
  884. 30:37rules, the responsibilities, the
  885. 30:39standard because you're going to have
  886. 30:40like different teams working on
  887. 30:41different types of data and shapes. And
  888. 30:43you're going to define for the company
  889. 30:45how the shape of the data must look like
  890. 30:47to make the data trusted across multiple
  891. 30:49teams. Now moving on to the last one,
  892. 30:51the platform level architecture. So
  893. 30:52you're going to design like let's say a
  894. 30:54blueprint or a data platform that many
  895. 30:57different teams depends on like for
  896. 30:59example you're going to say you know
  897. 31:00what we're going to build a data
  898. 31:01lakehouse for the entire company or
  899. 31:04maybe you're going to adopt the data
  900. 31:06mesh where you're going to say you know
  901. 31:07what for each domain we should produce
  902. 31:09data product so you have to put the
  903. 31:11blueprint and the other teams must
  904. 31:13follow it. So as you can see as a data
  905. 31:14architect your job is no longer learning
  906. 31:17how to code or how to build a pipeline.
  907. 31:19Now you are discussing tradeoffs. You
  908. 31:22are making decisions and you are like a
  909. 31:24bridge between multiple teams and
  910. 31:26multiple departments where you define
  911. 31:28standards and blueprints for maybe the
  912. 31:30whole company and of course if the
  913. 31:31company very big then only for that
  914. 31:33department. Of course there are other
  915. 31:35passes as I said manager data product
  916. 31:37owner or maybe a data officer but from
  917. 31:40technical perspective this is the last
  918. 31:42thing that you can grow. Maybe the last
  919. 31:44role is that to leave the job and be
  920. 31:46content creator like me. All right, my
  921. 31:48friends. So, that was a long one. If
  922. 31:50you're still here, then I really
  923. 31:51apologize. But I really wanted to show
  924. 31:53the full big picture of data engineering
  925. 31:55career. Absolute beginner until the last
  926. 31:58shape of the data architects. And now if
  927. 32:00you are total beginner, I totally
  928. 32:02understand you if I lost you at the ends
  929. 32:04because learning all those stuff take
  930. 32:06years. I don't want to scare you. The
  931. 32:08first journey it could done really fast
  932. 32:10in one year or less. So it should not
  933. 32:12take years to get hired as a junior data
  934. 32:15engineer. But I'm talking about the next
  935. 32:17phases. This could take time. So I just
  936. 32:19wanted to show you the full road map. So
  937. 32:21maybe you can start using only the first
  938. 32:23part and after a while after you get a
  939. 32:25job then you go back to the road map and
  940. 32:27understand I have to do the next step.
  941. 32:29So it can sticks with you in different
  942. 32:31phases at your career as a data
  943. 32:33engineer. Now, if you like this type of
  944. 32:35content where I'm sharing my experience
  945. 32:36as a data engineer with you, then
  946. 32:38support the channel by subscribing,
  947. 32:40liking, and committing in order to reach
  948. 32:42nice people like you. And if you're
  949. 32:44still here, then thank you so much for
  950. 32:46watching. I'm going to see you in the
  951. 32:48next video. So, bye-bye. Bye-bye.
  952. 32:50>> Bye.

About this transcript

This page contains the full transcript of How I’d Learn Data Engineering in 2026 From Zero to Senior (Built by a Data Engineer) by Data with Baraa, generated from the public captions YouTube serves with the video. The transcript has 6,930 words across 952 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.