YouTube2Text

Agents Made Code Free. Maintaining It Didn't Get Cheaper. - Dmitry Buykin, Maersk — Transcript

by AI Engineer · 2,428 words · 402 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Hi, I'm Dmitry.
  2. 0:02In this session, I will show how I work
  3. 0:04on maintaining large AI coding projects.
  4. 0:08Coding agents made code cheap.
  5. 0:10They made entropy even cheaper.
  6. 0:13In the Marvel movie, Thanos watched his
  7. 0:16home planet Titan
  8. 0:17collapse under its own growth. So when
  9. 0:20the universe got crowded, he erased half
  10. 0:23of everything.
  11. 0:25This talk is about the one time I did
  12. 0:27the same in one project.
  13. 0:29Our code was growing the way Titan did.
  14. 0:32When I found what my agents had done,
  15. 0:36I deleted tests and boilerplate code
  16. 0:40nobody used. I had to rewrite the whole
  17. 0:43project and I lost some useful pieces.
  18. 0:46And that's the dilemma.
  19. 0:48Drown in chaos or break what people use.
  20. 0:51In here,
  21. 0:52we'll build the third option.
  22. 0:55For that project, the rewrite didn't
  23. 0:57hold. The agents kept writing and the
  24. 1:01mess grew back.
  25. 1:02So I drew a line above it.
  26. 1:05There's more change than anyone can
  27. 1:07check. The third option is to stay under
  28. 1:11it with small sweeps every week and
  29. 1:13feedback loops built into the system.
  30. 1:16Each chapter of this talk is one thing I
  31. 1:18had to learn to get there.
  32. 1:21Thanos needed six stones for one snap.
  33. 1:24Ours are six questions and they light up
  34. 1:27in the corner as we go. The first one is
  35. 1:30time.
  36. 1:31How do we stay under the line?
  37. 1:34Chapter one. To stay under the line, I
  38. 1:37first had to know what right means here.
  39. 1:40I used to think the tests decide that.
  40. 1:43So what does right actually depend on?
  41. 1:48Let's keep one task in mind. Someone
  42. 1:50wants to change the amount
  43. 1:53on an order we already approved. Two
  44. 1:55kinds of agents meet here.
  45. 1:57One wrote the instructions and other
  46. 2:00follows them in production.
  47. 2:04We run this in 110 countries with about
  48. 2:0710,000 scenarios of prose instructions.
  49. 2:11Every task lands in a slice, one process
  50. 2:14in one country. That's the second stone,
  51. 2:17space.
  52. 2:19It asks where a change is allowed to go.
  53. 2:22By default, it stays inside its slice.
  54. 2:27Most of this system is prose, about
  55. 2:29three quarters by my estimate. So prose
  56. 2:31needs engineering too.
  57. 2:34At start, we borrow it mutation testing.
  58. 2:36You break one sentence on purpose and
  59. 2:38you watch for red test.
  60. 2:41Here, almost none turned red. And if
  61. 2:44your wrapper is all code, your agent
  62. 2:47instructions and your tests are your
  63. 2:49prose.
  64. 2:51With agents, a rewrite is easy. The hard
  65. 2:54part is knowing what has to survive it.
  66. 2:57For code, tests can tell you. For prose,
  67. 3:00only people can. Agents can't build a
  68. 3:03complex system on their own.
  69. 3:05They fix what is easy to see and break
  70. 3:08consistency on the way. And people also
  71. 3:11get tired, so we spend their judgment
  72. 3:14only where it's needed.
  73. 3:17When business asks check with the team,
  74. 3:19it hides a real question. Is this a
  75. 3:22human step or can we automate it?
  76. 3:25Experts can't answer that. They don't
  77. 3:27see how the systems connect inside. So
  78. 3:30developers dig through the traces and
  79. 3:32logs with their coding agents and they
  80. 3:35decide which it is.
  81. 3:37In our task, the team leads decision
  82. 3:40stays human. The limit was data we could
  83. 3:43connect.
  84. 3:44Business experts see the rules.
  85. 3:47Developers see what's under the water.
  86. 3:50And that's where most bugs live.
  87. 3:52They have their coding agents write
  88. 3:55those links down so their on time can
  89. 3:57check them.
  90. 3:58Our guide for experts says it simply
  91. 4:01business tell us what we work out how.
  92. 4:06One document to readers. Agents read the
  93. 4:09blocks so the blocks have to be exact.
  94. 4:12Experts read the text around them and
  95. 4:15they sign it. We keep one copy, a second
  96. 4:18one would drift and works both ways. Our
  97. 4:20code needs a plain description for
  98. 4:22experts too.
  99. 4:24Experts can only fix what they can see.
  100. 4:27Each of them reviews a task exactly the
  101. 4:30way the agent saw it. The node becomes
  102. 4:33the next change made by a coding agent
  103. 4:36and tests it on that same task. So right
  104. 4:40depends on people. No single person
  105. 4:43knows it. A group of experts decides and
  106. 4:46that's why we bring them in from the
  107. 4:49start.
  108. 4:50You'll see the soul stone here a lot.
  109. 4:53We take it in chapter four.
  110. 4:56Chapter two.
  111. 4:57Experts correct one task at a time while
  112. 5:00agents run the same pros all day.
  113. 5:03We track every outcome accepted or
  114. 5:06rejected. The accuracy shows us where
  115. 5:10people still need to look.
  116. 5:12I used to think the goal was to move
  117. 5:14everything into code.
  118. 5:16So where do we still need judgment?
  119. 5:19Here's our task again. The pros says the
  120. 5:22limit is 1,500. The code says 1,000. The
  121. 5:26same order is fine in one place
  122. 5:29and too high in the other and nothing
  123. 5:32throws an error.
  124. 5:33Even a line like the not call X teaches
  125. 5:36the model that X exists.
  126. 5:39That's the third stone, reality. Is it
  127. 5:42true or does it only look true for the
  128. 5:45model?
  129. 5:46Pros has no compiler, so we built one.
  130. 5:50Then the author lints every changed file
  131. 5:53into ledger
  132. 5:54just like code.
  133. 5:58Lint can't stop copies drifting apart,
  134. 6:01so we watched the walks. A walk is the
  135. 6:05one agent takes through the pros.
  136. 6:07When agents keep working in the same
  137. 6:09path, we turn it into one workflow tool
  138. 6:13and we delete the pros that walked it.
  139. 6:16Our first workflow tools made things
  140. 6:19worse. The pros still named the steps
  141. 6:22inside them, so agents run those steps
  142. 6:25again. It worked once the pros kept only
  143. 6:28the decisions.
  144. 6:29Now a walk becomes code
  145. 6:32only when it repeats and agrees across
  146. 6:35countries. Otherwise, it stays pros.
  147. 6:39What still varies stays in pros inside
  148. 6:41the frame.
  149. 6:43Code prepares the facts and carries out
  150. 6:45the decision.
  151. 6:47Experts own the decision table and the
  152. 6:49model only picks a row.
  153. 6:52If no row fits,
  154. 6:54a person takes over. We haven't built
  155. 6:56the next step yet. The model will take
  156. 6:58the small decisions
  157. 7:00and workflows will connect them.
  158. 7:03This boundary moves both ways. We moved
  159. 7:05hundreds of lines of expressions out of
  160. 7:08code and into pros
  161. 7:11because experts couldn't read them.
  162. 7:14A regex is still fine for spotting a
  163. 7:15critical signal.
  164. 7:17When there are many, they move into
  165. 7:19conflict.
  166. 7:20Judgment stays wherever the answer
  167. 7:22varies in a place an expert can read.
  168. 7:26Chapter three.
  169. 7:28Now the model decides and its decision
  170. 7:30becomes an action.
  171. 7:32I used to think control meant
  172. 7:35permissions on the agent.
  173. 7:37So what does it mean to stay in control?
  174. 7:41Control starts with what the model sees.
  175. 7:44That's what the fourth stone mined. What
  176. 7:47does the model know?
  177. 7:49When we let it choose what to load, it
  178. 7:51loaded everything.
  179. 7:53Then it invented a pricing check just
  180. 7:55because prices were visible.
  181. 7:58Whatever sits in its context becomes a
  182. 8:00reason to act.
  183. 8:03The runtime decides what the model sees.
  184. 8:05For our task, that's three facts from
  185. 8:08the order. The full record stays with
  186. 8:11the runtime and the model makes fewer
  187. 8:13wrong moves.
  188. 8:17Here how we minimize context bloat. For
  189. 8:20a big set, the model passes a handle, a
  190. 8:22short name for data the runtime already
  191. 8:24holds.
  192. 8:26The runtime checks it before any write.
  193. 8:28It keeps the context small
  194. 8:30and still enough for the decision. At
  195. 8:32the same time, it validates context for
  196. 8:36consistency.
  197. 8:38A model with the right facts can still
  198. 8:40do damage. That's the fifth stone,
  199. 8:43power. What made change? We control a
  200. 8:46change twice during development and
  201. 8:49again at the write at runtime.
  202. 8:52Only code opens a write path, so we can
  203. 8:55change pros many times a day.
  204. 8:57The open rings are the parts that still
  205. 8:59depend on people.
  206. 9:02Let's zoom in and run write. Five walls
  207. 9:05stand in front of it, and the sixth
  208. 9:08checks that it landed.
  209. 9:09If a write looks like it failed, an
  210. 9:12agent retries, and the journal makes
  211. 9:15sure it runs only once. Control means
  212. 9:17much more than permissions on the agent.
  213. 9:24Chapter four.
  214. 9:25Most days I glide over the system. From
  215. 9:28up here, everything looks fine, so I
  216. 9:31don't go down into every change my
  217. 9:33agents made.
  218. 9:35Then the surface gets rough. How do we
  219. 9:38know we're losing control?
  220. 9:41All those controls protect the system.
  221. 9:43Nothing protected my attention. At the
  222. 9:46start, I read every change, and that's
  223. 9:48how the system picked up my habits.
  224. 9:51Then the volume grew past what one
  225. 9:54person can read.
  226. 9:56I was burning out and I stopped reading
  227. 9:58every change.
  228. 10:00That was the price of the last stone,
  229. 10:02soul. Who judges and what does it cost?
  230. 10:05With it, the gauntlet is complete.
  231. 10:08Every tech lead I know has been here.
  232. 10:11The one person every change waits on.
  233. 10:14Agents just get you there in weeks
  234. 10:17instead of years.
  235. 10:19I needed a way of working that doesn't
  236. 10:21run through one head.
  237. 10:23And the first gauge was me. When I'm
  238. 10:25tired and I can't explain what changed,
  239. 10:28that's the signal.
  240. 10:29The dashboard can be green while that
  241. 10:32gauge is firing. So I stop
  242. 10:35and fix the process first. I don't wait
  243. 10:38for a number.
  244. 10:39I couldn't read every diff anymore, so I
  245. 10:42built instruments that watch how things
  246. 10:45change and their derivatives.
  247. 10:47One of them is blast radius.
  248. 10:50Before we built one new check, we
  249. 10:52replayed its rule over several thousand
  250. 10:55past sessions. It would have fired on
  251. 10:57less than 1%.
  252. 10:59Where the numbers jump, I go down and
  253. 11:02read.
  254. 11:03When something looks wrong, we wait. One
  255. 11:06signal waits until it comes back, even
  256. 11:08when it comes from an expert.
  257. 11:10The cluster across slices is different.
  258. 11:13Then we fix the concept behind it once.
  259. 11:17Left alone, an agent keeps adding
  260. 11:19conditions until today's tasks pass. The
  261. 11:22eval score drops when you take them out
  262. 11:25and you get an honest list of gaps.
  263. 11:31One red slice is rarely alone.
  264. 11:33The expert's node becomes an eval, and
  265. 11:36an agent searches the siblings for the
  266. 11:38same failure.
  267. 11:40Siblings differ on purpose
  268. 11:42by market and by local law.
  269. 11:44So we sweep them, and the fix goes only
  270. 11:48where the failure is confirmed.
  271. 11:53If anytime 10 to 20 of these loops are
  272. 11:57open.
  273. 11:57At one point, I had become the message
  274. 12:00bus between users and my agents. At this
  275. 12:03scale, every user has to be an evaluator
  276. 12:07with a screen to give feedback without
  277. 12:09me.
  278. 12:10My job as the architect is to build
  279. 12:12those loops into the system.
  280. 12:15Each loop writes to a ledger, so nothing
  281. 12:17waits on my memory and nothing waits on
  282. 12:20me.
  283. 12:24Agents forget everything when a session
  284. 12:26ends, so the memory has to live in
  285. 12:29files.
  286. 12:30We keep two kinds, what happened and
  287. 12:33what is true now.
  288. 12:34The next session starts from what is
  289. 12:37true now.
  290. 12:40Every issue has to connect to its root
  291. 12:43cause. Without that, the agent decides
  292. 12:46on wrong facts and the check has nothing
  293. 12:49to count. So we don't act without
  294. 12:51evidence.
  295. 12:54Reality comes back.
  296. 12:57In chapter two, it asked if the process
  297. 12:59is true. Now it asks if the change is.
  298. 13:03I used to trust green.
  299. 13:05So what would make us change our mind?
  300. 13:09Green was tempting because checking is
  301. 13:11where the times goes. My agents spend
  302. 13:14almost 9/10 of their time on a change,
  303. 13:17just checking it. And checking took a
  304. 13:19bigger share of my own week two.
  305. 13:22Verification speed is the limit.
  306. 13:27We rank evidence by what it proves. It
  307. 13:30will play on real tasks seats at the
  308. 13:33top. Before the data arrives,
  309. 13:35we write down what would prove us wrong.
  310. 13:38And every new check has to fail once on
  311. 13:40purpose before we trust it.
  312. 13:46AI refactors fail quietly. Every
  313. 13:50condition and every decision table row
  314. 13:53is a branch. And after a refactor, each
  315. 13:56one has to land exactly once.
  316. 13:59Here the branch for amounts over the
  317. 14:01limit got lost.
  318. 14:02Our [snorts] order would have gone
  319. 14:04through
  320. 14:05without the team lead.
  321. 14:07Real tasks decide what we test. We
  322. 14:10freeze them as a set and replay the old
  323. 14:13version against the new one.
  324. 14:16If no real task takes a branch, we call
  325. 14:18that no evidence.
  326. 14:21You can't trust one agent to judge its
  327. 14:24own work. A second agent checks it with
  328. 14:27a checklist and without the author's
  329. 14:30reasoning.
  330. 14:31It can be talked into agreeing. Over six
  331. 14:34months,
  332. 14:35it grew to most of my checking.
  333. 14:39Coding agents finish most of a batch and
  334. 14:42quietly skip the rest, especially when
  335. 14:45their context is full.
  336. 14:47The ledger shows what was kept.
  337. 14:50In one backlog, most items looks
  338. 14:52covered, but only about a quarter could
  339. 14:55really be closed. So now, evidence we
  340. 14:59can count is what changes our mind.
  341. 15:03This is the third option from the start
  342. 15:06of the talk. I used to think there were
  343. 15:09two choices, keep everything or snap. So
  344. 15:13what is worth carrying forward?
  345. 15:16This one process over a year. Some
  346. 15:19branches merged back, some still drift,
  347. 15:22and some are dead but still look alive.
  348. 15:25Which ones deserve to survive?
  349. 15:30A fix is only a moment. These forces
  350. 15:33keep pushing right after it. You've met
  351. 15:36most of them already. This chapter deals
  352. 15:39with regrowth and branching.
  353. 15:44The first gauge was me back in chapter
  354. 15:46four. These are the gauges for the code.
  355. 15:50Time gets its answers here. The bridge
  356. 15:53line comes back as a number.
  357. 15:55A ratchet means the count can only go
  358. 15:57down. Two gauges run today and six more
  359. 16:01are next.
  360. 16:04Every week, real use decides what stays.
  361. 16:07Anything no real task reaches gets a
  362. 16:10name and its evidence before it goes.
  363. 16:13That's the evidence I didn't have when I
  364. 16:16snapped.
  365. 16:17Tests kept that variance alive. Agents
  366. 16:20kept every option that they still
  367. 16:23mentioned.
  368. 16:24Now retired variant takes its test with
  369. 16:27it.
  370. 16:29Here's the whole method as one week.
  371. 16:32This is the third option
  372. 16:34as a process.
  373. 16:39Together that's the full gauntlet.
  374. 16:42Thanos used his for one snap. We use
  375. 16:45ours every week in small doses with
  376. 16:48evidence.
  377. 16:49I haven't had to snap again.
  378. 16:55Soul had a second price and I only saw
  379. 16:58it later. After we delivered to
  380. 17:00production, I found my own engineering
  381. 17:03principles built into the system's
  382. 17:06design.
  383. 17:07Its skills and its memory carry my
  384. 17:09habits and my biases.
  385. 17:12Conway said the system copies the org
  386. 17:14chart.
  387. 17:15Ours copies whoever corrects it.
  388. 17:18Here are 18 questions, three for each
  389. 17:21stone.
  390. 17:22I'm using them when working on my
  391. 17:24projects.
  392. 17:26Pause here and pick one your own
  393. 17:29repository can't answer yet.
  394. 17:32That's where to start.
  395. 17:37I learned to use this process when I was
  396. 17:40already burning out. If you're the one
  397. 17:43your team's changes wait on today,
  398. 17:46this is for you. Use your stones while
  399. 17:49you can still read every change. In the
  400. 17:52end, Thanos retires to a garden. Maybe
  401. 17:55he should start using them daily.
  402. 17:59Thank you.

About this transcript

This page contains the full transcript of Agents Made Code Free. Maintaining It Didn't Get Cheaper. - Dmitry Buykin, Maersk by AI Engineer, generated from the public captions YouTube serves with the video. The transcript has 2,428 words across 402 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.