Agents Made Code Free. Maintaining It Didn't Get Cheaper. - Dmitry Buykin, Maersk — Transcript
Full transcript
- 0:00Hi, I'm Dmitry.
- 0:02In this session, I will show how I work
- 0:04on maintaining large AI coding projects.
- 0:08Coding agents made code cheap.
- 0:10They made entropy even cheaper.
- 0:13In the Marvel movie, Thanos watched his
- 0:16home planet Titan
- 0:17collapse under its own growth. So when
- 0:20the universe got crowded, he erased half
- 0:23of everything.
- 0:25This talk is about the one time I did
- 0:27the same in one project.
- 0:29Our code was growing the way Titan did.
- 0:32When I found what my agents had done,
- 0:36I deleted tests and boilerplate code
- 0:40nobody used. I had to rewrite the whole
- 0:43project and I lost some useful pieces.
- 0:46And that's the dilemma.
- 0:48Drown in chaos or break what people use.
- 0:51In here,
- 0:52we'll build the third option.
- 0:55For that project, the rewrite didn't
- 0:57hold. The agents kept writing and the
- 1:01mess grew back.
- 1:02So I drew a line above it.
- 1:05There's more change than anyone can
- 1:07check. The third option is to stay under
- 1:11it with small sweeps every week and
- 1:13feedback loops built into the system.
- 1:16Each chapter of this talk is one thing I
- 1:18had to learn to get there.
- 1:21Thanos needed six stones for one snap.
- 1:24Ours are six questions and they light up
- 1:27in the corner as we go. The first one is
- 1:30time.
- 1:31How do we stay under the line?
- 1:34Chapter one. To stay under the line, I
- 1:37first had to know what right means here.
- 1:40I used to think the tests decide that.
- 1:43So what does right actually depend on?
- 1:48Let's keep one task in mind. Someone
- 1:50wants to change the amount
- 1:53on an order we already approved. Two
- 1:55kinds of agents meet here.
- 1:57One wrote the instructions and other
- 2:00follows them in production.
- 2:04We run this in 110 countries with about
- 2:0710,000 scenarios of prose instructions.
- 2:11Every task lands in a slice, one process
- 2:14in one country. That's the second stone,
- 2:17space.
- 2:19It asks where a change is allowed to go.
- 2:22By default, it stays inside its slice.
- 2:27Most of this system is prose, about
- 2:29three quarters by my estimate. So prose
- 2:31needs engineering too.
- 2:34At start, we borrow it mutation testing.
- 2:36You break one sentence on purpose and
- 2:38you watch for red test.
- 2:41Here, almost none turned red. And if
- 2:44your wrapper is all code, your agent
- 2:47instructions and your tests are your
- 2:49prose.
- 2:51With agents, a rewrite is easy. The hard
- 2:54part is knowing what has to survive it.
- 2:57For code, tests can tell you. For prose,
- 3:00only people can. Agents can't build a
- 3:03complex system on their own.
- 3:05They fix what is easy to see and break
- 3:08consistency on the way. And people also
- 3:11get tired, so we spend their judgment
- 3:14only where it's needed.
- 3:17When business asks check with the team,
- 3:19it hides a real question. Is this a
- 3:22human step or can we automate it?
- 3:25Experts can't answer that. They don't
- 3:27see how the systems connect inside. So
- 3:30developers dig through the traces and
- 3:32logs with their coding agents and they
- 3:35decide which it is.
- 3:37In our task, the team leads decision
- 3:40stays human. The limit was data we could
- 3:43connect.
- 3:44Business experts see the rules.
- 3:47Developers see what's under the water.
- 3:50And that's where most bugs live.
- 3:52They have their coding agents write
- 3:55those links down so their on time can
- 3:57check them.
- 3:58Our guide for experts says it simply
- 4:01business tell us what we work out how.
- 4:06One document to readers. Agents read the
- 4:09blocks so the blocks have to be exact.
- 4:12Experts read the text around them and
- 4:15they sign it. We keep one copy, a second
- 4:18one would drift and works both ways. Our
- 4:20code needs a plain description for
- 4:22experts too.
- 4:24Experts can only fix what they can see.
- 4:27Each of them reviews a task exactly the
- 4:30way the agent saw it. The node becomes
- 4:33the next change made by a coding agent
- 4:36and tests it on that same task. So right
- 4:40depends on people. No single person
- 4:43knows it. A group of experts decides and
- 4:46that's why we bring them in from the
- 4:49start.
- 4:50You'll see the soul stone here a lot.
- 4:53We take it in chapter four.
- 4:56Chapter two.
- 4:57Experts correct one task at a time while
- 5:00agents run the same pros all day.
- 5:03We track every outcome accepted or
- 5:06rejected. The accuracy shows us where
- 5:10people still need to look.
- 5:12I used to think the goal was to move
- 5:14everything into code.
- 5:16So where do we still need judgment?
- 5:19Here's our task again. The pros says the
- 5:22limit is 1,500. The code says 1,000. The
- 5:26same order is fine in one place
- 5:29and too high in the other and nothing
- 5:32throws an error.
- 5:33Even a line like the not call X teaches
- 5:36the model that X exists.
- 5:39That's the third stone, reality. Is it
- 5:42true or does it only look true for the
- 5:45model?
- 5:46Pros has no compiler, so we built one.
- 5:50Then the author lints every changed file
- 5:53into ledger
- 5:54just like code.
- 5:58Lint can't stop copies drifting apart,
- 6:01so we watched the walks. A walk is the
- 6:05one agent takes through the pros.
- 6:07When agents keep working in the same
- 6:09path, we turn it into one workflow tool
- 6:13and we delete the pros that walked it.
- 6:16Our first workflow tools made things
- 6:19worse. The pros still named the steps
- 6:22inside them, so agents run those steps
- 6:25again. It worked once the pros kept only
- 6:28the decisions.
- 6:29Now a walk becomes code
- 6:32only when it repeats and agrees across
- 6:35countries. Otherwise, it stays pros.
- 6:39What still varies stays in pros inside
- 6:41the frame.
- 6:43Code prepares the facts and carries out
- 6:45the decision.
- 6:47Experts own the decision table and the
- 6:49model only picks a row.
- 6:52If no row fits,
- 6:54a person takes over. We haven't built
- 6:56the next step yet. The model will take
- 6:58the small decisions
- 7:00and workflows will connect them.
- 7:03This boundary moves both ways. We moved
- 7:05hundreds of lines of expressions out of
- 7:08code and into pros
- 7:11because experts couldn't read them.
- 7:14A regex is still fine for spotting a
- 7:15critical signal.
- 7:17When there are many, they move into
- 7:19conflict.
- 7:20Judgment stays wherever the answer
- 7:22varies in a place an expert can read.
- 7:26Chapter three.
- 7:28Now the model decides and its decision
- 7:30becomes an action.
- 7:32I used to think control meant
- 7:35permissions on the agent.
- 7:37So what does it mean to stay in control?
- 7:41Control starts with what the model sees.
- 7:44That's what the fourth stone mined. What
- 7:47does the model know?
- 7:49When we let it choose what to load, it
- 7:51loaded everything.
- 7:53Then it invented a pricing check just
- 7:55because prices were visible.
- 7:58Whatever sits in its context becomes a
- 8:00reason to act.
- 8:03The runtime decides what the model sees.
- 8:05For our task, that's three facts from
- 8:08the order. The full record stays with
- 8:11the runtime and the model makes fewer
- 8:13wrong moves.
- 8:17Here how we minimize context bloat. For
- 8:20a big set, the model passes a handle, a
- 8:22short name for data the runtime already
- 8:24holds.
- 8:26The runtime checks it before any write.
- 8:28It keeps the context small
- 8:30and still enough for the decision. At
- 8:32the same time, it validates context for
- 8:36consistency.
- 8:38A model with the right facts can still
- 8:40do damage. That's the fifth stone,
- 8:43power. What made change? We control a
- 8:46change twice during development and
- 8:49again at the write at runtime.
- 8:52Only code opens a write path, so we can
- 8:55change pros many times a day.
- 8:57The open rings are the parts that still
- 8:59depend on people.
- 9:02Let's zoom in and run write. Five walls
- 9:05stand in front of it, and the sixth
- 9:08checks that it landed.
- 9:09If a write looks like it failed, an
- 9:12agent retries, and the journal makes
- 9:15sure it runs only once. Control means
- 9:17much more than permissions on the agent.
- 9:24Chapter four.
- 9:25Most days I glide over the system. From
- 9:28up here, everything looks fine, so I
- 9:31don't go down into every change my
- 9:33agents made.
- 9:35Then the surface gets rough. How do we
- 9:38know we're losing control?
- 9:41All those controls protect the system.
- 9:43Nothing protected my attention. At the
- 9:46start, I read every change, and that's
- 9:48how the system picked up my habits.
- 9:51Then the volume grew past what one
- 9:54person can read.
- 9:56I was burning out and I stopped reading
- 9:58every change.
- 10:00That was the price of the last stone,
- 10:02soul. Who judges and what does it cost?
- 10:05With it, the gauntlet is complete.
- 10:08Every tech lead I know has been here.
- 10:11The one person every change waits on.
- 10:14Agents just get you there in weeks
- 10:17instead of years.
- 10:19I needed a way of working that doesn't
- 10:21run through one head.
- 10:23And the first gauge was me. When I'm
- 10:25tired and I can't explain what changed,
- 10:28that's the signal.
- 10:29The dashboard can be green while that
- 10:32gauge is firing. So I stop
- 10:35and fix the process first. I don't wait
- 10:38for a number.
- 10:39I couldn't read every diff anymore, so I
- 10:42built instruments that watch how things
- 10:45change and their derivatives.
- 10:47One of them is blast radius.
- 10:50Before we built one new check, we
- 10:52replayed its rule over several thousand
- 10:55past sessions. It would have fired on
- 10:57less than 1%.
- 10:59Where the numbers jump, I go down and
- 11:02read.
- 11:03When something looks wrong, we wait. One
- 11:06signal waits until it comes back, even
- 11:08when it comes from an expert.
- 11:10The cluster across slices is different.
- 11:13Then we fix the concept behind it once.
- 11:17Left alone, an agent keeps adding
- 11:19conditions until today's tasks pass. The
- 11:22eval score drops when you take them out
- 11:25and you get an honest list of gaps.
- 11:31One red slice is rarely alone.
- 11:33The expert's node becomes an eval, and
- 11:36an agent searches the siblings for the
- 11:38same failure.
- 11:40Siblings differ on purpose
- 11:42by market and by local law.
- 11:44So we sweep them, and the fix goes only
- 11:48where the failure is confirmed.
- 11:53If anytime 10 to 20 of these loops are
- 11:57open.
- 11:57At one point, I had become the message
- 12:00bus between users and my agents. At this
- 12:03scale, every user has to be an evaluator
- 12:07with a screen to give feedback without
- 12:09me.
- 12:10My job as the architect is to build
- 12:12those loops into the system.
- 12:15Each loop writes to a ledger, so nothing
- 12:17waits on my memory and nothing waits on
- 12:20me.
- 12:24Agents forget everything when a session
- 12:26ends, so the memory has to live in
- 12:29files.
- 12:30We keep two kinds, what happened and
- 12:33what is true now.
- 12:34The next session starts from what is
- 12:37true now.
- 12:40Every issue has to connect to its root
- 12:43cause. Without that, the agent decides
- 12:46on wrong facts and the check has nothing
- 12:49to count. So we don't act without
- 12:51evidence.
- 12:54Reality comes back.
- 12:57In chapter two, it asked if the process
- 12:59is true. Now it asks if the change is.
- 13:03I used to trust green.
- 13:05So what would make us change our mind?
- 13:09Green was tempting because checking is
- 13:11where the times goes. My agents spend
- 13:14almost 9/10 of their time on a change,
- 13:17just checking it. And checking took a
- 13:19bigger share of my own week two.
- 13:22Verification speed is the limit.
- 13:27We rank evidence by what it proves. It
- 13:30will play on real tasks seats at the
- 13:33top. Before the data arrives,
- 13:35we write down what would prove us wrong.
- 13:38And every new check has to fail once on
- 13:40purpose before we trust it.
- 13:46AI refactors fail quietly. Every
- 13:50condition and every decision table row
- 13:53is a branch. And after a refactor, each
- 13:56one has to land exactly once.
- 13:59Here the branch for amounts over the
- 14:01limit got lost.
- 14:02Our [snorts] order would have gone
- 14:04through
- 14:05without the team lead.
- 14:07Real tasks decide what we test. We
- 14:10freeze them as a set and replay the old
- 14:13version against the new one.
- 14:16If no real task takes a branch, we call
- 14:18that no evidence.
- 14:21You can't trust one agent to judge its
- 14:24own work. A second agent checks it with
- 14:27a checklist and without the author's
- 14:30reasoning.
- 14:31It can be talked into agreeing. Over six
- 14:34months,
- 14:35it grew to most of my checking.
- 14:39Coding agents finish most of a batch and
- 14:42quietly skip the rest, especially when
- 14:45their context is full.
- 14:47The ledger shows what was kept.
- 14:50In one backlog, most items looks
- 14:52covered, but only about a quarter could
- 14:55really be closed. So now, evidence we
- 14:59can count is what changes our mind.
- 15:03This is the third option from the start
- 15:06of the talk. I used to think there were
- 15:09two choices, keep everything or snap. So
- 15:13what is worth carrying forward?
- 15:16This one process over a year. Some
- 15:19branches merged back, some still drift,
- 15:22and some are dead but still look alive.
- 15:25Which ones deserve to survive?
- 15:30A fix is only a moment. These forces
- 15:33keep pushing right after it. You've met
- 15:36most of them already. This chapter deals
- 15:39with regrowth and branching.
- 15:44The first gauge was me back in chapter
- 15:46four. These are the gauges for the code.
- 15:50Time gets its answers here. The bridge
- 15:53line comes back as a number.
- 15:55A ratchet means the count can only go
- 15:57down. Two gauges run today and six more
- 16:01are next.
- 16:04Every week, real use decides what stays.
- 16:07Anything no real task reaches gets a
- 16:10name and its evidence before it goes.
- 16:13That's the evidence I didn't have when I
- 16:16snapped.
- 16:17Tests kept that variance alive. Agents
- 16:20kept every option that they still
- 16:23mentioned.
- 16:24Now retired variant takes its test with
- 16:27it.
- 16:29Here's the whole method as one week.
- 16:32This is the third option
- 16:34as a process.
- 16:39Together that's the full gauntlet.
- 16:42Thanos used his for one snap. We use
- 16:45ours every week in small doses with
- 16:48evidence.
- 16:49I haven't had to snap again.
- 16:55Soul had a second price and I only saw
- 16:58it later. After we delivered to
- 17:00production, I found my own engineering
- 17:03principles built into the system's
- 17:06design.
- 17:07Its skills and its memory carry my
- 17:09habits and my biases.
- 17:12Conway said the system copies the org
- 17:14chart.
- 17:15Ours copies whoever corrects it.
- 17:18Here are 18 questions, three for each
- 17:21stone.
- 17:22I'm using them when working on my
- 17:24projects.
- 17:26Pause here and pick one your own
- 17:29repository can't answer yet.
- 17:32That's where to start.
- 17:37I learned to use this process when I was
- 17:40already burning out. If you're the one
- 17:43your team's changes wait on today,
- 17:46this is for you. Use your stones while
- 17:49you can still read every change. In the
- 17:52end, Thanos retires to a garden. Maybe
- 17:55he should start using them daily.
- 17:59Thank you.
About this transcript
This page contains the full transcript of Agents Made Code Free. Maintaining It Didn't Get Cheaper. - Dmitry Buykin, Maersk by AI Engineer, generated from the public captions YouTube serves with the video. The transcript has 2,428 words across 402 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.