YouTube2Text

The Agents Were Fine. Everything Around Them Broke. - Joey Wang, Harvey — Transcript

by AI Engineer · 1,797 words · 271 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Let me start with one number from last
  2. 0:02month. On average, more than one agent
  3. 0:05has started on our platform every single
  4. 0:08minute, day and night.
  5. 0:11And honestly, the agents were the easy
  6. 0:13part.
  7. 0:15Hi, I'm Joey. I bu the cloud agent
  8. 0:19platform we built inside Harvey.
  9. 0:21After 10,000 PRs, the biggest lesson I
  10. 0:25can share is this. The thing that broke
  11. 0:28was an agent. It was everything around
  12. 0:31them.
  13. 0:33So, what does one submit add up to?
  14. 0:3610,000 PRs that an agent wrote. And it
  15. 0:40isn't just Tavac team playing with a
  16. 0:43toy. Last month, about 460 different
  17. 0:47people at Harvey use it. And over the
  18. 0:51summer, it stopped being only about
  19. 0:53code. People started asking a data
  20. 0:56question in Slack and getting answers
  21. 0:58back. My favorite recently someone on
  22. 1:02our marketing team dropped a prototype
  23. 1:04and a talk track into Slack and ask for
  24. 1:07a story book for a product video.
  25. 1:10Once the harness exists, people bring
  26. 1:12the work that you never planned for.
  27. 1:16So a promise up front. This is not a
  28. 1:19talk about models, evos or leaderboard.
  29. 1:22This is talk about hardness,
  30. 1:25odd things around the model,
  31. 1:29checking out the ripple, holding the
  32. 1:30key, opening the PR, getting the merged.
  33. 1:34When we first wrote about Spectre back
  34. 1:36in April, we put it like this. At some
  35. 1:40point, a harness become the product.
  36. 1:43Now, whenever I tell people this, I get
  37. 1:46the same question. Why on earth would
  38. 1:48you build this when clock or cursor
  39. 1:51already exist?
  40. 1:53My answer, we don't compete with them.
  41. 1:56We run them. And here's how I see it as
  42. 1:59my own view. Agents on your laptop are
  43. 2:02powerful, but they don't scale. You can
  44. 2:05realistically run five to 10 of them.
  45. 2:08Pass that you need a cloud. And cloud
  46. 2:11needs our secrets, our real stack, our
  47. 2:14alerts and tickets, one set of rules and
  48. 2:18enough capacity that nobody gets
  49. 2:20starved. Those single two owns that for
  50. 2:23us. So we build layer and that layer
  51. 2:26exactly what broke. So I want to tell
  52. 2:29three stories today. How we stopped
  53. 2:32competing on writing code. how we let
  54. 2:34any agent drive and were broke once they
  55. 2:38did. The first story start with the
  56. 2:41question we keep hearing.
  57. 2:44More of our engineers were using cursor
  58. 2:47and they ask how do I get our secret
  59. 2:50there? How do I test against our real
  60. 2:52stack? We could have rebuilt all that
  61. 2:56inside cursor and then again for next to
  62. 2:59and the one after that.
  63. 3:01Instead, the answer became one sentence.
  64. 3:04Just ask cursor to use vector.
  65. 3:08And that's really the whole idea. People
  66. 3:10will always have a favorite agent and
  67. 3:13the favorite keeps changing. So pick
  68. 3:15yours. Cloud code, codeex, cursor, trag
  69. 3:19your own hands.
  70. 3:21Start on your laptop, push the session
  71. 3:23to the cloud, put it back. That's how
  72. 3:27Spectre sits underneath all of that.
  73. 3:30Because once the code was written, the
  74. 3:32real group starts
  75. 3:34picture appears day. Alert files in
  76. 3:37Slack at 2 in the morning. An agent
  77. 3:40picks up nobody typing. It writes a fix,
  78. 3:44spins up real copy of our stack and post
  79. 3:48the proof on the PR. When CI fails, the
  80. 3:52follow-up loop reads the failure and
  81. 3:54pushes another fix while everyone still
  82. 3:56sleep. By morning, the PR sits in one
  83. 4:00review queue under the same rule as
  84. 4:02other PR whichever agents wrote it. The
  85. 4:07owner gets a notch and only once your
  86. 4:10working day starts and once approved a
  87. 4:13agent can automerge it.
  88. 4:17If you only build one of those stages
  89. 4:20that I described, build a follow-up
  90. 4:22loop. We switched on for everyone in
  91. 4:25early October and that months. Four out
  92. 4:28of five agent PRs merged into our
  93. 4:30backend PR had been through it.
  94. 4:35That's the difference between agent that
  95. 4:36write code and an agent that finished
  96. 4:39the job.
  97. 4:41And then there's the knowledge. Every
  98. 4:43team has that one person who knows
  99. 4:45exactly how to test the tricky service.
  100. 4:48Uh which flag you never touch. And that
  101. 4:51knowledge leaves in their head. of their
  102. 4:54laptop.
  103. 4:55So we gave it a hope projects
  104. 4:59people write down what matters. Adrian
  105. 5:01write down what they learn. Team pings
  106. 5:04the skills you trust and there's a
  107. 5:06record of who rent what.
  108. 5:09Any agent working in the project with
  109. 5:11the same page. My personal view
  110. 5:14whichever agent you pick the audit trail
  111. 5:16and the checks should be the same.
  112. 5:19The second story is about rule. Back in
  113. 5:23May, if you want a spectctor to do
  114. 5:25something, you open a browser tab or
  115. 5:28tacking Slack. The whole month, exactly
  116. 5:32one task came out from terminal.
  117. 5:35So, we made a rule. If a person can
  118. 5:37click it, an agent can call it.
  119. 5:41Start a task. Turn on the follow-up loop
  120. 5:44on automerge, publish a page, all from
  121. 5:47CLI,
  122. 5:49which means your laptop agent can call
  123. 5:51it. A agent inside the sandbox can also
  124. 5:54call it. Even CI can call it too. And
  125. 5:57then people ran with it. By October, the
  126. 6:01command line was starting about 6,500 ts
  127. 6:04a month. The web app didn't shrink.
  128. 6:08Everyone else just grew around it.
  129. 6:11Task started from the terminal were
  130. 6:13merging mops and task started in browser
  131. 6:16and the new kind of users showed up.
  132. 6:19People who start hundreds of tasks a
  133. 6:21month and never open the app at all.
  134. 6:25And this is where it gets fun. In
  135. 6:27October, I had a big task. So I bought
  136. 6:31code session and asked to be the
  137. 6:32coordinator.
  138. 6:34It split the tickets into 10 different
  139. 6:37milestones. K one for me and the rest to
  140. 6:41Spectre with hundreds of tasks. Each has
  141. 6:44its own sandbox to validate end to end.
  142. 6:48Then at the end of the October, our it
  143. 6:50give everyone trag
  144. 6:53now watch Spectre for me. It list the
  145. 6:56question Spectre is waiting on draft my
  146. 6:58answers and once I say yes, send them
  147. 7:01back.
  148. 7:03I actually DM to my friend like that. I
  149. 7:06just use cursor clock code or whatever
  150. 7:08to spam spectre. Now,
  151. 7:11well, that's the dream, but it's also
  152. 7:13the problem.
  153. 7:15A quick detour. When we look at where
  154. 7:18our task time actually went, the slowest
  155. 7:20step wasn't model syncing. It was get
  156. 7:24just checking out every ripple we own
  157. 7:26took over a minute on average.
  158. 7:30The fix keep a copy of every ripple on
  159. 7:33every machine and let each task borrow
  160. 7:36it.
  161. 7:38The third story is the one where it
  162. 7:40broke in mid August. A release goes out
  163. 7:44with a bug. That bug triggers alert that
  164. 7:47fires once per workspace. So one bug
  165. 7:50become hundreds of alerts all landing in
  166. 7:53one single S channel.
  167. 7:55Someone has set up workflow that tag our
  168. 7:58agent on every alert and every tag
  169. 8:01started task. On normal day those task
  170. 8:04started about 50 task. That day in about
  171. 8:08four hours they started around 900.
  172. 8:12900 agents are chasing the same bug
  173. 8:14until that wasn't room left for a single
  174. 8:17note.
  175. 8:18The fix was almost boring. A repeat
  176. 8:21alert now joins the investigation that's
  177. 8:23already open.
  178. 8:25Two months later, it happened again. And
  179. 8:28this time, it wasn't a bot. On October,
  180. 8:32the engineer did exactly what we've been
  181. 8:35asking everyone to do. Hand work to
  182. 8:38Spectre.
  183. 8:40About 950 tasks from the command line.
  184. 8:43In about 3 hours, nothing in our
  185. 8:46platform can work. The cluster filled up
  186. 8:50and everyone else work sat in line
  187. 8:53behind one laptop. That's not a user
  188. 8:55problem. That's our problem.
  189. 8:594 days later, we had a real fix. A fair
  190. 9:03launch queue. Now starting a task gives
  191. 9:06you a place in line and the dispatcher
  192. 9:09started when there's room. People
  193. 9:12sitting at keyboard goes first. Everyone
  194. 9:15gets a fair share and can borrow more
  195. 9:17than the cluster is quite.
  196. 9:20And whenever you start a task, you can
  197. 9:22see a placing line. When we hit it with
  198. 9:24a 300 task burst in low test, the task
  199. 9:28lit went from almost 2 seconds to 70
  200. 9:30milliseconds.
  201. 9:32So did it work?
  202. 9:35When it's the busiest day of every week
  203. 9:38back in August, two certain task and
  204. 9:41about one in 10 didn't finish.
  205. 9:45Then in November 4th, more than twice
  206. 9:47that and much better in the finish line,
  207. 9:51but we still have some room to grow.
  208. 9:55I will be modest, we change several
  209. 9:56things at once, so I cannot tell you
  210. 9:59which fix mattered the most, but the
  211. 10:01birds kept getting bigger and the
  212. 10:03platform kept up.
  213. 10:06And just when that was on the crunch
  214. 10:08show, the model neck moves.
  215. 10:10At the end of the October, our dev
  216. 10:13platform team raised an alarm. PRS
  217. 10:16across the company had jumped around 60%
  218. 10:20in a single month. A merge that used to
  219. 10:23take 15 to 30 minutes was taking over an
  220. 10:26hour now. And merge Q kept having to be
  221. 10:29switched off. So now a PR rebase onto
  222. 10:32main rerun CI and merge if it's green.
  223. 10:37If main breaks everyone CI breaks with
  224. 10:39it. Alterately
  225. 10:42we haven't cracked this one yet.
  226. 10:45But before I go there's one more thing.
  227. 10:49While we were busy fixing all it
  228. 10:52something else was quite growing. Most
  229. 10:55of what I've been showing you start with
  230. 10:57a person asking.
  231. 10:59The fastest growing source of the work
  232. 11:02on Spectre doesn't these are automations
  233. 11:06agent that wake up on schedule on an
  234. 11:08event and just do the work tri alert
  235. 11:13read the PR write the post.
  236. 11:16Last month about 110 people had
  237. 11:19automation running and we learned to
  238. 11:21give them a guardrail on day one.
  239. 11:24If one fails five times in a row, it
  240. 11:26switch itself off and it can carry a
  241. 11:30budget or expiry date because when no
  242. 11:33one's watching, an agent can start
  243. 11:35thousands of tasks, too.
  244. 11:38So, here's what I'm chasing next, and I
  245. 11:41don't have an answer yet, honestly.
  246. 11:44What if every Slack channel had its own
  247. 11:46agent that never locks off? One that
  248. 11:49follows competition and start work on
  249. 11:51its own. What if every pure prove it
  250. 11:54works before a human even look at it?
  251. 11:58Today only one merge in 20 carries that
  252. 12:01kind of proof.
  253. 12:03And how do you merge when agents never
  254. 12:06stop opening PRs?
  255. 12:08Ask me again 6 months. I will tell you
  256. 12:11what broke. If you're building something
  257. 12:14like this when you work, here's three
  258. 12:16things I do again. Oh, what happens
  259. 12:19after code is written. Make Eric action
  260. 12:23something an agent can call and cap what
  261. 12:26an agent, human or alert can start
  262. 12:29before they find out what they can do
  263. 12:31with it.
  264. 12:33And one last thought, that's my own
  265. 12:36view. Engineers can write code with
  266. 12:39whichever agent they like. The work
  267. 12:42that's lost is the harness that runs it,
  268. 12:44checks it, and get it merged.
  269. 12:48Thank you so much for watching.
  270. 12:51Come find me on linking if you are
  271. 12:53building similar things.

About this transcript

This page contains the full transcript of The Agents Were Fine. Everything Around Them Broke. - Joey Wang, Harvey by AI Engineer, generated from the public captions YouTube serves with the video. The transcript has 1,797 words across 271 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.