YT YouTube2Text

YouTube2Text → Transcripts → From QA Engineer to AI-Powered Product Engineer - Dimitar Topuzov

From QA Engineer to AI-Powered Product Engineer - Dimitar Topuzov — Transcript

by Challenge Accepted Events · 4,503 words · 669 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:01Challenge accepted. [music] Challenge
0:03accepted.
0:06Challenge accepted. Challenge [music]
0:09accepted. Challenge accepted. P2 is
0:13always [music] challenge accepted.
0:20Our
0:26next speaker is well known to the QA
0:28challenge accepted audience. He is a QA
0:31architect at progress last year's QA of
0:34the year and the person many of you know
0:36as Chicho Mitkco or in Bulgarian Chich
0:41Mitkco
0:43topos works on scalable automation and
0:45engineering practices. Today he explores
0:48the move from QA engineer to AI powered
0:52product engineer. Please welcome Deita.
0:58Challenge accepted. Challenge [music]
1:01accepted.
1:03is always trying to accept.
1:20>> Hi everyone. Uh I'm Deita and today we
1:23will talk how AI is changing the way how
1:26we build software. how lines between
1:30testing, development and product are
1:32becoming less clear, what skills matter
1:35most and why QA mindset can be a great
1:38advantage in the age of AI. So when we
1:41talk about QA and AI, usually we have
1:44two out camps and one of them is telling
1:46us AI writes the code, it can write the
1:50test, it can even use agents to drive
1:52browsers and it can explore our
1:54application as real user. So we no
1:57longer need QAs. And the other camp is
2:00telling us AI produce more code. So more
2:03code need to be verified. We need more
2:05QA and QA will be more important than
2:07ever.
2:09Here is Mitkco in the middle. He refused
2:11to take take sight in this conflict
2:14because he believes that both camps are
2:16kind of right and kind of wrong in the
2:18same time. [snorts] He believes that uh
2:22we still have the need for high quality
2:24software. What's different is how we
2:27produce it. To understand our future,
2:30probably we should look into our past
2:33and compare traditional hand womb with
2:35modern womb machine. We we have the same
2:39need for high quality fabric. What's
2:41different is tools we use to produce it
2:44and skill we need to operate with these
2:46tools. With the traditional hand womb,
2:49we need operator who is master of
2:52movements with their hands. And for the
2:55modern machine, we need operator who
2:57knows how to set up the machine
2:59correctly and use it. In both cases,
3:03after we produce uh the thing we are
3:06doing, we inspect it and we need to do
3:09some judgment to check if the quality is
3:12good or not and then take some
3:14decisions. With software, it's not
3:17different. This picture move us to the
3:19early days of Unix at Bellwaps where way
3:23before uh ASQB started to certify
3:27software testers they had the need for
3:29high quality software. They had those
3:32big machines that were big in size but
3:34very weak as uh compute power. They also
3:38had the assembly language in which was
3:41uh very hard to write software. But this
3:44software was able to run fast on
3:46hardware. They had they had the B
3:48language in which was in which it was
3:51easy to write software but this software
3:53was too slow to run on hardware they
3:55had. So they needed a high quality
3:57solution that allow them to write
3:59programs that run fast and u
4:04be easy to write. So this is how Dennis
4:07Rich and his friends created the C
4:10language to solve that problem and how
4:13they tested it and how they made the C
4:15language highquality product. When they
4:18wrote it, they they used themselves
4:20[snorts] at Bellaps to write Unix
4:24programs to write Unix itself. And in
4:27this way they had direct feedback uh if
4:30the product is okay, how to improve it
4:33and this is how they ensured quality in
4:35early ' 70s.
4:37About 20 years later we saw this person
4:41Linuvolts
4:43he wanted to have a Unix like system on
4:46his own machine. So no PMS, no QA, no
4:50scrum masters, no jer epics, just a
4:53curious engineer who wanted to build
4:55something, learn something and this is
4:58how he released the first version of
5:01Linux. He opened the court, shared with
5:03the community, invited the community to
5:05to to step in and then the community
5:09bring its own problems, its own
5:11expertise. And nowadays, Linux is huge
5:14collaborative project that runs almost
5:16on every server out there. So, uh how
5:21they ensure quality at such a big scale?
5:23They have contributors. Contributors are
5:26people who are doing the change. No
5:28matter if it's bug fix or feature,
5:30contributor is the person doing the
5:32change and contributors uh also the
5:37person who need to test the change and
5:39ensure it works. But most important is
5:42that contributor must provide evidence
5:46for every claim he do. He must also
5:50explain very clearly to reviewers and
5:52maintainers what is the change about and
5:56then uh reviewers review the code and
5:58maintainers who are domain experts in
6:00specific uh subsystems review how the
6:04proposed change affect the entire
6:06subsystem because they are domain expert
6:09in this subsystem.
6:11So what are the lessons learned from
6:13this story? You don't need to have a PM
6:15title to have a great idea and you don't
6:18need to have quality something title to
6:21have great software. You need discipline
6:24evidences and do not merge changes
6:28without some clear evidence that it's
6:31for good.
6:34About the same time mid '90s Microsoft
6:37had different challenges. They were huge
6:39corporation which want to make money and
6:42how they wanted to make money. Uh they
6:44wanted to ship operating system for the
6:47masses. They were preparing Windows 95
6:50release. They had huge marketing
6:52campaigns. CDs should be available in
6:55stores at some date and then everyone
6:59should install the Windows and it should
7:01work and do not crash. So when you have
7:04strict deadlines you put a lot of
7:06marketing effort in in this thing it
7:09sounds starts sound like a project
7:12that's why they had separate program
7:15management discipline at that time they
7:17also had testing and development as
7:20separate disciplines and they put huge
7:23effort in testing almost the same amount
7:26of people developing the product and the
7:29same amount of people testing the
7:33It was very expensive but failures would
7:36be far more expensive. Imagine you go to
7:39the store buy the cities and it crash it
7:41will be catastrophic failure. So uh
7:45probably when you have strict deadlines
7:47a lot of marketing uh doing this makes
7:50sense because it made them successful.
7:53It's fact.
7:55What happened in next decades? In next
7:58decades, Microsoft still invested uh a
8:01lot in testing but they saw that
8:04one-on-one ratio do not scale and they
8:07decided to invest in testing but make
8:10testing more efficient.
8:12And how they decided to do it? They they
8:15they invested heavily in automation.
8:19They moved from traditional software
8:21test engineer role to software developer
8:24engineer in test role which is engineer
8:27who is uh supposed to write code just in
8:31this case the code are automated tests.
8:35They also kept all the good practices
8:37from '9s like external dog food forging
8:40and they were very serious uh
8:43organization. They wrote a lot of books
8:46and they were great. So they did in this
8:50way they released Visual Studio Team
8:52Foundation Server which is the father of
8:54current Asia DevOps.
8:59Almost the same time Linux had a
9:02problem. Linux community was growing.
9:05There were a lot of contributors and to
9:07manage source code they used BitKeeper
9:10but they lost access free access to uh
9:13Bitkeeper and they were in trouble. They
9:16needed source control system that can uh
9:20manage contributions at such scale and
9:23this is how Linux Linux came with his
9:26idea for decentralized source control
9:28system and he invented git
9:32what happened in next years few years
9:34later git got popular because people saw
9:38it's good then Microsoft
9:41embraced the g protocol and they made it
9:43available in TFS
9:45when they moved to Azure DevOps uh the
9:49Git protocol became the default default
9:51protocol. You were able to create TFS
9:53based repository or Git based
9:55repository.
9:56A few years later, Microsoft themselves
9:59moved their code bases from TFS
10:01repositories to Git repositories and
10:03they fa phased out the TFS protocol.
10:07So lessons from this story, great ideas
10:11beat the structures, beat all the good
10:14practices in testing and huge
10:16investments.
10:18Why? Because quality is not just working
10:21without bugs, without problems. The
10:23quality is solving a real problem for
10:26users in the right way. And it also
10:28required a lot of product thinking.
10:32So then what we saw? We saw that AI
10:36arrived and we saw
10:38toothbrush with AI AI washing machine by
10:42Samsung. We even saw tires with uh AI
10:46technology label. And why we saw all of
10:50this? Because every company and every
10:53CEO they were afraid that if they uh do
10:56not jump jump on the AI train fast
10:59enough, they will be out of business. So
11:02we saw situation where CEOs collect all
11:05the company and tell everyone should use
11:07AI and then we saw directors telling
11:10every team should use AI and then we saw
11:12managers asking engineers give me your
11:15your AI usage report by Friday because
11:17they always in hurry and everything is
11:19urgent and then we saw uh people uh
11:23engineers reporting oh we adopted 25% uh
11:27uh uh we use AI in 25% % of our tasks
11:33and next month they report higher
11:35percent and everything goes up to the
11:37chain to the co and everyone is happy
11:39but the question here is what improved
11:42for the users so I don't know uh
11:45sometimes something improves not always
11:48but what I saw uh is that sometimes we
11:53uh uh use AI as expensive camouflage and
11:58we try to do something with it uh but we
12:02forget the good engineering practices.
12:05So I saw people trying to use AI to
12:09generate uh release notes from messy
12:11commit messages and J items
12:15uh instead of just have a good
12:17engineering higen and use semantic
12:19commit messages that then
12:22generate release notes based on semantic
12:24commit messages. I saw uh people with a
12:29lot of manual test cases in test
12:31management systems using AI to select
12:34which of those test to run because they
12:37cannot run all of them. Here I would ask
12:41why not use AI to automate those tests
12:43and then you'll have no problem which
12:45test to select.
12:48Uh also what I saw is people with
12:53unreadable reports from their Jenkins
12:55jobs and they were not understanding
12:58their reports themselves. they were
12:59losing a lot of time to understand if
13:02it's flaky test or real problem in the
13:05in the product and they decide oh we use
13:08AI to summarize our reports and AI to
13:11tell if it's flaky test or book in the
13:13product and this is the wrong thing
13:14because if you don't understand your
13:16reports probably you don't have enough
13:18data in them so here the the good
13:21engineering practice says use more clear
13:24assertions collect more walks collect
13:26traces now we have modern tools like
13:29playright web driver io. You can collect
13:32traces while your test automation
13:34automated test runs. You can improve
13:37your error messages and then your
13:39reports will be more clear and
13:42understandable both for humans and AI.
13:46Uh last thing I saw it's uh a bit scary.
13:50I saw some testing frameworks and
13:52libraries using so-called smart AI steps
13:56inside end to end tests. So how it
13:59looks? It looks like we have browser act
14:04boot to shopping cart. It's plain
14:06English. You run it under the hood. It
14:10uses AI to analyze the dome and do some
14:14actions. And when I show this to uh a
14:18manager,
14:19so that's great. We can take all of our
14:23manual test cases in the test case
14:26management systems. We will put this put
14:29it in such statements and we are done.
14:31Automation is done. Uh but what's the
14:35problem? The problem is that this is
14:37even worse than uh record uh and replay
14:43uh test automation which was popular
14:45something like 10 or 15 years ago. Why?
14:49Because at least record recorded test
14:52automation was hard to maintain as well.
14:56But at least recorded test automation
14:58was deterministic and it was running
15:00fast. This one runs slow because AI need
15:04to think a lot and need reasoning.
15:07It's expensive because because it burns
15:10tokens and also it's probabilistic and
15:14can lead to flaky test because AI is not
15:17deterministic.
15:19So uh if AI help you to tolerate bad
15:23engineering, it's not helping you. It's
15:25just making the mess scalable
15:29and we saw more theater. Uh let me tell
15:33you a story once I talked with a manager
15:37and he said uh in our company we are
15:41doing uh agent that helps PMS to uh
15:46define better road map. uh these agents
15:50fetch data from girather sources and
15:53tell what's the high priority and then
15:56he said they have developer agents that
15:58help their developers to write code and
16:01he said now we want to create automation
16:04Q agent and manual Q agent and I told
16:07myself here's something very wrong with
16:09this uh why because I understand why
16:14some companies have manual QS and
16:16automation QAS they want to first test
16:18it manually, ship it fast and then uh
16:21automate later. Okay, makes sense for
16:24them. But when we talk about agents to
16:28create the manual agent that explore the
16:30page and take some actions, you need to
16:33do the same things you need for the
16:35automated Q agent. In both cases, you
16:39need to start the browser, inspect the
16:41dome, then uh having this context, it's
16:45doesn't matter if AI will write code or
16:48AI will do click. It's all the same. So,
16:52no point to do uh two separate agents.
16:56Uh and it's always hard to tell a person
16:59that there is a problem. So, I decided
17:01to to ask him a question and joke in the
17:03same time. And I asked him, "So uh
17:08manager agent Kuga uh uh
17:13uh probably you need something [snorts]
17:16to approve vocations of your automation
17:19QA agent and uh your manual Q agent. Uh
17:23I think uh he didn't got my joke or
17:25decided to joke and said ah we can do it
17:28because workday has the APIs and we can
17:30do such agent." Uh so Jo joke on the
17:34side if you if you do agents that are
17:38mapped to your current roles uh you have
17:41more problems than that because imagine
17:44you do agents that PMS are using to
17:47define the road map and they prepare uh
17:49product requirement document. Then uh
17:52they give this document to UX people.
17:55You wait UX people to work and then uh
17:58when product requirement document and UX
18:00are ready developers start
18:02implementation and they then when they
18:05start implementation they find some
18:07technical issues that simply do not
18:09allow them to continue and they need to
18:11have those long meetings with product
18:14people again to discuss the requirements
18:17and all that stuff. So basically you
18:21have the same hand handoffs and those
18:24handoffs and this communication back and
18:26forth uh it's it's it's taking time. So
18:30using agents it's it's help but uh you
18:35you need somehow to to work on uh your
18:38boundaries and handoffs and procedures
18:41as well to get benefits from using AI.
18:44So
18:46uh when uh usually when I uh try to
18:52decide how it's the best way to shape
18:55the things I look at the best
18:56organizations out there so I decided to
18:59look at entropic and here is Boris
19:02Churnney who uh after they created a
19:06coordin
19:13uh we could have shipped this if we
19:16started from static mock in Figma. This
19:19is the design or we have started with uh
19:22PRD the product requirement
19:24documentation.
19:27So how they did it? They are all
19:30engineers. They have no PMs, no
19:32developers, no Q, all engineers. They
19:36didn't start with static mocks in Figma.
19:38They just added engineers who understand
19:40UX to the project. And they started to
19:43work uh on the real code base and shape
19:46it to look visually okay uh uh directly
19:51in the codebase.
19:54uh and requirements were also shaped in
19:57the process of creating the product and
19:59this is how they created it so fast.
20:03So you know what's the difference
20:06between boss and a leader I guess uh in
20:09both cases you have the structure you
20:11have boss you have the leader uh but
20:14imagine what happens you if you have uh
20:17ninja someone like Lenos and give him an
20:22AI if uh you give people who have both
20:26product thinking and are good engineers
20:28and understand the architectures if you
20:31give them AI they are like unstoppable.
20:35So uh this is a chart uh from survey
20:40from Wankchain uh very good company and
20:44uh this survey show a chart with two X's
20:47and one of them is how good you are at
20:51technical skills and architectures and
20:53writing code and the other uh X is uh
20:57how well you understand the product the
21:00users and basically this chart says that
21:05if you are bad at engineering and do not
21:09understand product good enough, you were
21:11always in trouble even before AI. What
21:14happens with AI is um nowadays AI can
21:19build something but we need people to
21:21review it to review the architecture to
21:25analyze the technical tradeoffs also we
21:28need to review it from product
21:30perspective if if this is good for the
21:33product or not and ultimately if we have
21:37people who are good both at engineering
21:40and have a good understanding of the
21:43product and the problem they are like
21:46unicorns. So basically your goal no
21:49matter if you are developer, QA, PM,
21:52whatever your goal is to to get there in
21:55the unicorn rectangle.
21:58So what skills matter and what skills
22:01can get you in this rectangle?
22:05First you need to have deep
22:06understanding for the product, for the
22:09user, for the domain, why you're doing
22:11the things. Then you need critical
22:13thinking. You need to question
22:16everything requirements, output,
22:18assumptions.
22:19You need to have uh enough technical
22:22background to do uh the hard technical
22:26judgments like architecture and
22:28trade-offs. You need to collect
22:30evidences, not just VIP code. You need
22:34test benchmarks, evals that show and
22:38prove that your change is for good. And
22:42last but not least, you need to keep
22:44your engineering discipline. Uh insist
22:47on reviews, have the guard rails,
22:50approvals because otherwise we have
22:54incidents.
22:56To illustrate this, the deep
22:59understanding uh Linux and G is great
23:02example of deep understanding. You have
23:06a struggle. you very well understand the
23:10problem because you struggle and you you
23:12leave it the hard way. Then when you
23:16leave it the hard way probably you you
23:18very well know what's the real problem
23:20and what to fix what to address.
23:24Also a good example for this is in some
23:26companies developers are also
23:29responsible to uh to to do support and
23:33answer on tickets. And in this way if a
23:37developer do something and then should
23:39answer the support for the same thing it
23:42helps him have the good understanding
23:44for the problem. Uh and then it allow
23:47him uh to to design good solutions
23:51about the critical thinking. Uh there
23:54was uh interesting experiment a few
23:59months ago and some people
24:03had a codebase they had a book but they
24:07committed the fix for this book in the
24:09codebase. the book was fixed and they
24:12they told uh some agents and models, hey
24:16fix this book and 70% of the cases those
24:20agents and models modified the code
24:23pretending they they fixed something but
24:25the fix was already there.
24:28So uh you need the critical thinking
24:30here is what we have uh in teams where I
24:33work. We have agents they are not
24:35coupled with roles they are coupled with
24:37type of work they are doing. So we have
24:39bug fixer agent and this bug fixer bug
24:42fixer agent what it does is first u try
24:46to reproduce the issue and if it cannot
24:49reproduce it it do not try to fix it so
24:53uh this about critical thinking about
24:55technical judgment I have a hobby
24:58project it's called craft driver uh and
25:01I'm pretending that I'm doing playright
25:04really right and there uh there zero
25:09lines of code written by me by human. Uh
25:13what I do is uh I decide architecture
25:16myself. I decide I need real browsers. I
25:19need playright APIs but I need real
25:22browsers.
25:24So I do the judgment.
25:27A help me implement the code then I
25:29review the code and again do the
25:31judgments. When I was about to uh
25:35implement visual testing feature in
25:38craft driver, I told myself I will
25:42experiment with five ideas. I will tell
25:44AI to benchmark five ideas how you can
25:47compare images isolated.
25:50Then I
25:52did all those uh five approaches
25:55benchmarked which one is the fastest and
25:59uh I didn't choose the fastest one
26:01because it was with a lot of
26:03dependencies and that was huge
26:06trade-off. So I decided something else.
26:08So I did the judgments
26:13evidences not wipes. This time I will
26:16give uh example not from hobby projects
26:18but again what we do at work. At work
26:21I'm working in team uh where we are
26:24doing
26:26AI agent that generate web applications
26:30and uh our agent is generating
26:33applications.
26:34Uh we have developers trying to improve
26:37the product. They go to conferences like
26:40uh previous day we had dev challenge
26:41accepted people explaining how you
26:45should write agents our developer listen
26:47to this say hey this is for good we'll
26:50do it and they do it but sometimes it
26:52works sometimes not uh so this is why uh
26:57I created evals where I'm evaluating if
27:01their change is really working then
27:03inspect the result if there there are
27:05some gaps instead walk an issue. What I
27:08do is I have the evolves, I have the
27:11results and I analyze them. I asked uh
27:14agent to to address the gaps and then I
27:18run evals again and I have evidence if
27:21chain my fix is working or not and then
27:23I directly open a pull request with fix
27:25and with evidences that this this fix
27:28works and only things that's left is a
27:31developer need to review it.
27:35So I didn't stop testing. I stopped
27:38treating testing as a boundary.
27:42About discipline. Uh again few months
27:46ago uh some AWS services were down. Why?
27:50Why they they were down? They have their
27:53own coding agent called Kira. In
27:56Bulgaria we have a lot of Kirros.
27:59Uh so their coding agent decided that uh
28:05it will delete and rebuild production
28:08database because this is the easiest way
28:10to address something and this is how AWS
28:14go down. The same thing can happen with
28:16the Bulgarian Kiru developer if he got
28:19drunk and decide that he'll fix
28:21something on production and if and you
28:24don't have guard rails. So basically you
28:27you should keep uh keep good engineering
28:30practices like uh mandatory approvements
28:34uh for deployments reviews. No one no
28:38one there as person or as AI or as agent
28:42should be a wow to do something that's
28:45not reviewed.
28:47So uh those are the skills that matter
28:51most. We saw them and now remove the
28:55title skills that matter most and on the
28:57same same slide you can put the title QA
29:01mindset. In all the QA books they teach
29:04us that QA should have the product
29:08understanding. They teach us on critical
29:11thinking.
29:13They teach us how to test and collect
29:16evidences.
29:18Also the more senior QAs they don't just
29:21test the more senior QAs also take care
29:24how to fix the entire process how entire
29:27engineering process and those senior QA
29:31suggest improvement in the process
29:32suggest uh uh things like reviews guard
29:36rails approval good engineering
29:38practices
29:41so uh QA are almost there
29:45the maybe probably The the only one
29:48that's missing is in most QAs nowadays
29:52is uh most of them still they are not
29:56technical enough to understand the code
29:59I mean not the test automation code but
30:01to understand the product code and
30:04architecture and be able to take uh hard
30:07decisions having in mind tra tradeoffs
30:11that's maybe probably the only missing
30:13piece but they have most of the things
30:16already and just a little bit to be
30:20unicorns.
30:21So, uh how QA can stay useful?
30:26Do not stop testing. Stop treating
30:28testing as a boundary and focus how you
30:31can deliver more value for your company
30:33because believe me, companies care about
30:36value. They don't care about test cases.
30:39So, that's all from me. Do not defend
30:42the title, defend the value and try to
30:44provide it.
30:47>> [applause]
30:48>> Thank you debitter.
30:50He has some answers. Do you have
30:52questions?
30:54Please raise your head if uh hand if you
30:57have question and wait for the mic.
31:01There's a question by Nikolai.
31:07>> Yeah, thanks. Great talk. Uh wonder how
31:10how you handle in your team uh the PR
31:12reviews developers do. I believe you can
31:15produce lots of them based on these vows
31:18and all that. So do you still do this by
31:20hand?
31:22>> Mostly yes and uh
31:26yeah I can generate a lot of code but uh
31:32you should be able to understand it
31:34because if you need to support it in
31:36future are you sure you can support it
31:38if you don't understand it? Yeah, you
31:40understand it. But still there are 1,000
31:43code reviews the developers need to do.
31:47>> So now the role moved to reviewer not
31:50builder. We saw this in this diagram.
31:53>> Yeah, it's changed.
31:55>> Yeah, thanks.
31:56>> Thank you. The funny thing is they both
31:58will be at the speaker together
32:00>> during next break. So if you have a
32:04emerging question now, you can ask it.
32:06>> Okay. uh about the QA mindset uh slide I
32:10would say that this is the developer
32:12mindset also and at least that's what I
32:15expect from my colleagues okay uh and do
32:18you think that the roles of the
32:19developers and the QAs are like blurring
32:21and merging together now more and more
32:24with the help of AI
32:25>> yes absolutely uh that was kind of
32:28purpose for my talk so uh if you you're
32:32absolutely right you can remove the QA
32:34mindset you can put there the good
32:37engineer mindset and it will be still
32:39true because uh no matter if you are QA
32:43or developer you are either good
32:45engineer or bad engineer and nowadays
32:48it's important how good engineer you are
32:51>> thank you
32:52>> and how close you are to unicorn
32:54>> we have to conf [laughter]
32:58uh we have some announcements about the
33:00conference but later today now uh we
33:03will have a coffee break and you can can
33:06Meet Mitco and Nikki at the speakers
33:09corner. Make some noise for Deo
33:15TO

About this transcript

This page contains the full transcript of From QA Engineer to AI-Powered Product Engineer - Dimitar Topuzov by Challenge Accepted Events, generated from the public captions YouTube serves with the video. The transcript has 4,503 words across 669 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.