The Agents Were Fine. Everything Around Them Broke. - Joey Wang, Harvey — Transcript
Full transcript
- 0:00Let me start with one number from last
- 0:02month. On average, more than one agent
- 0:05has started on our platform every single
- 0:08minute, day and night.
- 0:11And honestly, the agents were the easy
- 0:13part.
- 0:15Hi, I'm Joey. I bu the cloud agent
- 0:19platform we built inside Harvey.
- 0:21After 10,000 PRs, the biggest lesson I
- 0:25can share is this. The thing that broke
- 0:28was an agent. It was everything around
- 0:31them.
- 0:33So, what does one submit add up to?
- 0:3610,000 PRs that an agent wrote. And it
- 0:40isn't just Tavac team playing with a
- 0:43toy. Last month, about 460 different
- 0:47people at Harvey use it. And over the
- 0:51summer, it stopped being only about
- 0:53code. People started asking a data
- 0:56question in Slack and getting answers
- 0:58back. My favorite recently someone on
- 1:02our marketing team dropped a prototype
- 1:04and a talk track into Slack and ask for
- 1:07a story book for a product video.
- 1:10Once the harness exists, people bring
- 1:12the work that you never planned for.
- 1:16So a promise up front. This is not a
- 1:19talk about models, evos or leaderboard.
- 1:22This is talk about hardness,
- 1:25odd things around the model,
- 1:29checking out the ripple, holding the
- 1:30key, opening the PR, getting the merged.
- 1:34When we first wrote about Spectre back
- 1:36in April, we put it like this. At some
- 1:40point, a harness become the product.
- 1:43Now, whenever I tell people this, I get
- 1:46the same question. Why on earth would
- 1:48you build this when clock or cursor
- 1:51already exist?
- 1:53My answer, we don't compete with them.
- 1:56We run them. And here's how I see it as
- 1:59my own view. Agents on your laptop are
- 2:02powerful, but they don't scale. You can
- 2:05realistically run five to 10 of them.
- 2:08Pass that you need a cloud. And cloud
- 2:11needs our secrets, our real stack, our
- 2:14alerts and tickets, one set of rules and
- 2:18enough capacity that nobody gets
- 2:20starved. Those single two owns that for
- 2:23us. So we build layer and that layer
- 2:26exactly what broke. So I want to tell
- 2:29three stories today. How we stopped
- 2:32competing on writing code. how we let
- 2:34any agent drive and were broke once they
- 2:38did. The first story start with the
- 2:41question we keep hearing.
- 2:44More of our engineers were using cursor
- 2:47and they ask how do I get our secret
- 2:50there? How do I test against our real
- 2:52stack? We could have rebuilt all that
- 2:56inside cursor and then again for next to
- 2:59and the one after that.
- 3:01Instead, the answer became one sentence.
- 3:04Just ask cursor to use vector.
- 3:08And that's really the whole idea. People
- 3:10will always have a favorite agent and
- 3:13the favorite keeps changing. So pick
- 3:15yours. Cloud code, codeex, cursor, trag
- 3:19your own hands.
- 3:21Start on your laptop, push the session
- 3:23to the cloud, put it back. That's how
- 3:27Spectre sits underneath all of that.
- 3:30Because once the code was written, the
- 3:32real group starts
- 3:34picture appears day. Alert files in
- 3:37Slack at 2 in the morning. An agent
- 3:40picks up nobody typing. It writes a fix,
- 3:44spins up real copy of our stack and post
- 3:48the proof on the PR. When CI fails, the
- 3:52follow-up loop reads the failure and
- 3:54pushes another fix while everyone still
- 3:56sleep. By morning, the PR sits in one
- 4:00review queue under the same rule as
- 4:02other PR whichever agents wrote it. The
- 4:07owner gets a notch and only once your
- 4:10working day starts and once approved a
- 4:13agent can automerge it.
- 4:17If you only build one of those stages
- 4:20that I described, build a follow-up
- 4:22loop. We switched on for everyone in
- 4:25early October and that months. Four out
- 4:28of five agent PRs merged into our
- 4:30backend PR had been through it.
- 4:35That's the difference between agent that
- 4:36write code and an agent that finished
- 4:39the job.
- 4:41And then there's the knowledge. Every
- 4:43team has that one person who knows
- 4:45exactly how to test the tricky service.
- 4:48Uh which flag you never touch. And that
- 4:51knowledge leaves in their head. of their
- 4:54laptop.
- 4:55So we gave it a hope projects
- 4:59people write down what matters. Adrian
- 5:01write down what they learn. Team pings
- 5:04the skills you trust and there's a
- 5:06record of who rent what.
- 5:09Any agent working in the project with
- 5:11the same page. My personal view
- 5:14whichever agent you pick the audit trail
- 5:16and the checks should be the same.
- 5:19The second story is about rule. Back in
- 5:23May, if you want a spectctor to do
- 5:25something, you open a browser tab or
- 5:28tacking Slack. The whole month, exactly
- 5:32one task came out from terminal.
- 5:35So, we made a rule. If a person can
- 5:37click it, an agent can call it.
- 5:41Start a task. Turn on the follow-up loop
- 5:44on automerge, publish a page, all from
- 5:47CLI,
- 5:49which means your laptop agent can call
- 5:51it. A agent inside the sandbox can also
- 5:54call it. Even CI can call it too. And
- 5:57then people ran with it. By October, the
- 6:01command line was starting about 6,500 ts
- 6:04a month. The web app didn't shrink.
- 6:08Everyone else just grew around it.
- 6:11Task started from the terminal were
- 6:13merging mops and task started in browser
- 6:16and the new kind of users showed up.
- 6:19People who start hundreds of tasks a
- 6:21month and never open the app at all.
- 6:25And this is where it gets fun. In
- 6:27October, I had a big task. So I bought
- 6:31code session and asked to be the
- 6:32coordinator.
- 6:34It split the tickets into 10 different
- 6:37milestones. K one for me and the rest to
- 6:41Spectre with hundreds of tasks. Each has
- 6:44its own sandbox to validate end to end.
- 6:48Then at the end of the October, our it
- 6:50give everyone trag
- 6:53now watch Spectre for me. It list the
- 6:56question Spectre is waiting on draft my
- 6:58answers and once I say yes, send them
- 7:01back.
- 7:03I actually DM to my friend like that. I
- 7:06just use cursor clock code or whatever
- 7:08to spam spectre. Now,
- 7:11well, that's the dream, but it's also
- 7:13the problem.
- 7:15A quick detour. When we look at where
- 7:18our task time actually went, the slowest
- 7:20step wasn't model syncing. It was get
- 7:24just checking out every ripple we own
- 7:26took over a minute on average.
- 7:30The fix keep a copy of every ripple on
- 7:33every machine and let each task borrow
- 7:36it.
- 7:38The third story is the one where it
- 7:40broke in mid August. A release goes out
- 7:44with a bug. That bug triggers alert that
- 7:47fires once per workspace. So one bug
- 7:50become hundreds of alerts all landing in
- 7:53one single S channel.
- 7:55Someone has set up workflow that tag our
- 7:58agent on every alert and every tag
- 8:01started task. On normal day those task
- 8:04started about 50 task. That day in about
- 8:08four hours they started around 900.
- 8:12900 agents are chasing the same bug
- 8:14until that wasn't room left for a single
- 8:17note.
- 8:18The fix was almost boring. A repeat
- 8:21alert now joins the investigation that's
- 8:23already open.
- 8:25Two months later, it happened again. And
- 8:28this time, it wasn't a bot. On October,
- 8:32the engineer did exactly what we've been
- 8:35asking everyone to do. Hand work to
- 8:38Spectre.
- 8:40About 950 tasks from the command line.
- 8:43In about 3 hours, nothing in our
- 8:46platform can work. The cluster filled up
- 8:50and everyone else work sat in line
- 8:53behind one laptop. That's not a user
- 8:55problem. That's our problem.
- 8:594 days later, we had a real fix. A fair
- 9:03launch queue. Now starting a task gives
- 9:06you a place in line and the dispatcher
- 9:09started when there's room. People
- 9:12sitting at keyboard goes first. Everyone
- 9:15gets a fair share and can borrow more
- 9:17than the cluster is quite.
- 9:20And whenever you start a task, you can
- 9:22see a placing line. When we hit it with
- 9:24a 300 task burst in low test, the task
- 9:28lit went from almost 2 seconds to 70
- 9:30milliseconds.
- 9:32So did it work?
- 9:35When it's the busiest day of every week
- 9:38back in August, two certain task and
- 9:41about one in 10 didn't finish.
- 9:45Then in November 4th, more than twice
- 9:47that and much better in the finish line,
- 9:51but we still have some room to grow.
- 9:55I will be modest, we change several
- 9:56things at once, so I cannot tell you
- 9:59which fix mattered the most, but the
- 10:01birds kept getting bigger and the
- 10:03platform kept up.
- 10:06And just when that was on the crunch
- 10:08show, the model neck moves.
- 10:10At the end of the October, our dev
- 10:13platform team raised an alarm. PRS
- 10:16across the company had jumped around 60%
- 10:20in a single month. A merge that used to
- 10:23take 15 to 30 minutes was taking over an
- 10:26hour now. And merge Q kept having to be
- 10:29switched off. So now a PR rebase onto
- 10:32main rerun CI and merge if it's green.
- 10:37If main breaks everyone CI breaks with
- 10:39it. Alterately
- 10:42we haven't cracked this one yet.
- 10:45But before I go there's one more thing.
- 10:49While we were busy fixing all it
- 10:52something else was quite growing. Most
- 10:55of what I've been showing you start with
- 10:57a person asking.
- 10:59The fastest growing source of the work
- 11:02on Spectre doesn't these are automations
- 11:06agent that wake up on schedule on an
- 11:08event and just do the work tri alert
- 11:13read the PR write the post.
- 11:16Last month about 110 people had
- 11:19automation running and we learned to
- 11:21give them a guardrail on day one.
- 11:24If one fails five times in a row, it
- 11:26switch itself off and it can carry a
- 11:30budget or expiry date because when no
- 11:33one's watching, an agent can start
- 11:35thousands of tasks, too.
- 11:38So, here's what I'm chasing next, and I
- 11:41don't have an answer yet, honestly.
- 11:44What if every Slack channel had its own
- 11:46agent that never locks off? One that
- 11:49follows competition and start work on
- 11:51its own. What if every pure prove it
- 11:54works before a human even look at it?
- 11:58Today only one merge in 20 carries that
- 12:01kind of proof.
- 12:03And how do you merge when agents never
- 12:06stop opening PRs?
- 12:08Ask me again 6 months. I will tell you
- 12:11what broke. If you're building something
- 12:14like this when you work, here's three
- 12:16things I do again. Oh, what happens
- 12:19after code is written. Make Eric action
- 12:23something an agent can call and cap what
- 12:26an agent, human or alert can start
- 12:29before they find out what they can do
- 12:31with it.
- 12:33And one last thought, that's my own
- 12:36view. Engineers can write code with
- 12:39whichever agent they like. The work
- 12:42that's lost is the harness that runs it,
- 12:44checks it, and get it merged.
- 12:48Thank you so much for watching.
- 12:51Come find me on linking if you are
- 12:53building similar things.
About this transcript
This page contains the full transcript of The Agents Were Fine. Everything Around Them Broke. - Joey Wang, Harvey by AI Engineer, generated from the public captions YouTube serves with the video. The transcript has 1,797 words across 271 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.