YT YouTube2Text

YouTube2Text → Transcripts → NEW DeepSeek Harness - Super Agent!

NEW DeepSeek Harness - Super Agent! — Transcript

by Sam Witteveen · 3,332 words · 484 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Okay, so this past weekend DeepSeq
0:02released something that a lot of people
0:04out there are calling the answer to
0:05Claude code. It's an agent that sits on
0:08a GitHub repo. It watches the issues and
0:10pull requests. And when someone on your
0:12team asks it to do something, it then
0:15goes off into its own session, does the
0:17work, and comes back with a PR. Now,
0:20while that's cool in itself, that's not
0:22really the part that I want to talk
0:24about here. The part that I find really
0:26interesting is how they built it. The
0:28whole thing is just a set of plugins
0:31dropped into their agent harness. And to
0:34understand why that matters, we need to
0:36go back and actually look at the harness
0:38itself. So, when the DeepSeq harness
0:40came out back in August, I didn't
0:42actually cover it. And honestly, the
0:43reason for that was I just got so busy
0:45playing with it and trying out new ideas
0:47with it all the time. I kept putting off
0:49making the video because I felt like
0:51there were so many things that I wanted
0:52to show. For me, the whole plugin
0:55architecture of that harness is really
0:57cool, and it just allows you to do a
1:00whole bunch of things like this new
1:01super agent enables. Now, obviously in
1:04the meantime, a lot of people ended up
1:05covering it. It's turned out to be one
1:07of the most starred projects on GitHub
1:09this year. So, in this video, I'm going
1:10to do a sort of catch-up on that and
1:13look at some of the new stuff that
1:14they've actually built on top of this.
1:16All right, so the new thing that DeepSeq
1:17has launched is called super agent. And
1:19along with the repos that they launched,
1:21they basically written an article on X
1:23called exploring super agent design
1:26towards a more autonomous agent system.
1:29And in that article and the repo,
1:30they're very up front about that this is
1:33a research prototype. It's not an
1:35official DeepSeq product. And they even
1:37mention that it grew out of their own
1:39day-to-day development work. And that a
1:41lot of the ideas and the engineering
1:43here were actually written by coding
1:45agents themselves. So, the basic idea is
1:48this. Instead of you opening a session
1:51and talking to an agent, the agent
1:53listens to where your team already
1:55works. And the example that they've
1:56launched now, that happens to be GitHub.
1:59It also happens to be an app called Face
2:01Shu, which is ByteDance's sort of Slack
2:03equivalent. Now, when someone who's
2:05allowed to asks it for something, it
2:08opens up a work item, it spins up its
2:11own session in its own directory, does
2:14all of the work, replies back to the
2:16original conversation, and then usually
2:19in that reply, it's going to have a pull
2:21request in there. And in many ways, you
2:23can kind of think of this as building on
2:25the whole ideas of software factories,
2:28of automated software generation and
2:30stuff that we've seen a lot of people
2:31being talking about over the past couple
2:33of months. Now, Deep Seek themselves say
2:35that they've been running an internal
2:37version of this continuously. It's been
2:40reproducing and fixing bug issues,
2:42responding to code review comments, and
2:44it's actually been making improvements
2:46to the Deep Seek harness itself. So,
2:49before I go into showing you the how of
2:51how it actually all works, I want to
2:53show you why the harness layer matters
2:55so much. Recently on the Venture B
2:57podcast Beyond the Pilot, which I
2:59co-host, we've interviewed a number of
3:01interesting people, including the CTO of
3:03Instacart and the CTO of Box. Now, both
3:06of those episodes are coming up over the
3:08next couple of weeks or so, but one of
3:10the questions that I put to both of them
3:13was basically, what's more important
3:15now, the model or the harness? Both of
3:18them came back on the side of the
3:19harness, and I'll leave it for you to
3:21watch the full interviews cuz it was
3:23fascinating conversations. But clearly
3:25now, a lot of the innovation has moved
3:27away from just the model itself to the
3:30actual harnesses. And maybe it will
3:32swing back in the future as we get
3:34better models, but currently you need to
3:36be focused on where the harnesses are
3:38going. All right, so one of the things
3:39that my team and I are trying to do
3:41right now is increase the coverage and
3:43the number of videos that we're doing.
3:45It would really help if you just support
3:46the channel by clicking that subscribe
3:48button. We know that most of you are
3:50watching a lot of the videos and still
3:52not subscribed. Anyway, back to why the
3:55harness is so important. So, this is the
3:57Frontier harness chart and result table.
4:00So, these aren't my numbers. This comes
4:02from a benchmark called Frontier Harness
4:04Eval. And what they did was take one
4:07model, the Kim e K3, and run it through
4:09nine different coding agent harnesses,
4:1212 configurations in total. Codex,
4:15Claude Code, Pi, Open Code, DeepSeek
4:18Harness, etc. They then gave it the same
4:2130 tasks, same model, same runtime. The
4:24only thing that changed here was the
4:26harness. Now, you could make the
4:28argument that it's not a totally fair
4:29comparison, that Claude Code is
4:31optimized for its models, that Codex is
4:34optimized for the models, but it's
4:35really interesting to see the results
4:37that they got here. Codex came out on
4:40top with 66.7 pass rate. But if you look
4:43at that next row, DeepSeek Harness in
4:46its creator configuration and Claude
4:49Code both scored 63.3%.
4:53That's the same pass rate, same model,
4:56but the DeepSeek Harness run cost only
4:58about $3.28 per successful task, and
5:02Claude Code cost about $18.34.
5:05Now, across the whole table, the gap
5:07between the cheapest and most expensive
5:10configurations was actually about 17.5x.
5:14Now, I'd be careful reading too much
5:15into the exact configurations here. It's
5:1830 tasks, so one task is worth about
5:20three and a third points. The difference
5:22between Codex and DeepSeek is literally
5:25just one task here. But I do think the
5:27big picture holds true here, that the
5:29harness isn't just a wrapper. It changes
5:32the cost, the caching, the speed, and
5:35the success rate, even when the model is
5:38held constant. So, if you're evaluating
5:40a model, I'd try and do it in two or
5:43three harnesses before deciding whether
5:46the model is any good. Otherwise, you
5:48might be just blaming a model for what
5:49actually is the harness's
5:51responsibility. All right, so what is
5:53DeepSeek Harness, and why does it do
5:55well here? So, DeepSeek Harness, or DSH,
5:58came out on August 13th, alongside the
6:01initial release of DeepSeek version 4
6:04Pro. It's MIT licensed and there's web
6:06UI. So, you can start with just one NPX
6:09command and a desktop app. It's not
6:12locked to the Deep Seek models. I think
6:14that's really important to stress here.
6:16It works with other providers and even
6:18with OpenAI compatible endpoints. The
6:21key tagline here though that sort of
6:23sums this up is that everything is a
6:25plugin and they mean that very
6:28literally. In most agent frameworks,
6:30you've got a core, the agent loop, the
6:33model calls, the tool handling, and then
6:36plugins sort of hang off on the side. In
6:39DSH, the router itself is a plugin, the
6:43tool registry, a plugin, the session
6:45log, a plugin, even the agent loop
6:48itself is a plugin. Underneath all of
6:50this is a framework called Cordis and
6:53there's a paper behind that from three
6:55researchers at the Beijing University.
6:57And this framework, Deep Seek called a
6:59programming paradigm for spatio-temporal
7:02composability. Now,
7:04that sounds like a lot, but the two
7:06ideas here are actually pretty simple.
7:08The first is what they call temporal
7:09composability. When you remove a
7:11component, everything it did gets
7:13completely undone. If a plugin
7:16registered a tool or a hook into event,
7:18unloading it reverses all of that. The
7:21simplest way to think about this is that
7:23you can unplug something really cleanly.
7:26The second element here is spatial
7:27composability. Components declare what
7:30they depend on and the runtime activates
7:33or deactivates them as those
7:35dependencies come and go. So, you can
7:37basically plug in things safely and
7:39unplug them safely as well. Now, why
7:41does an agent harness need this? Because
7:43systems are changing constantly. You're
7:45swapping models, you're adding tools,
7:47you're turning things on and off per
7:49permission. And if every one of those
7:51changes sort of leaves junk behind,
7:54things get messy really quickly. We saw
7:57this even when people sort of reverse
7:58engineered how Claude code was working.
8:01Everyone that's making agent harnesses
8:03at the moment is moving at such a fast
8:05pace. It's just a simple fact that often
8:07code gets left behind. Anyway, so how
8:10does this look in practice? Well,
8:11running DSH is actually a tree of
8:13plugins which gets composed when it
8:16boots up from layers. First come bundles
8:19which are package sets of plugins plus
8:21their configs, then your profiles patch
8:23a YAML file called cordis.patch, then
8:26you've got a home level patch, and then
8:28anything you pass in on the command
8:30line. So, each layer here can target any
8:33plugin by its ID and replace its config
8:36or insert new ones. They've even got a
8:39command in there for dump config that
8:41just prints out the whole tree that your
8:43machine is booting from, and every row
8:45in that can be overwritten. Now, in the
8:47UI, they've got four modes and they work
8:50in the same way. You've got standard,
8:52PGC, minimal, and creator. And those
8:54themselves are just different sets of
8:57plugins. The creator mode is sort of the
8:59logical end point for this. You can
9:01describe a plugin in chat and the agent
9:04will write it, install it, and then
9:07check that it's actually live. So, you
9:09can kind of think of this as while
9:11everyone's been focused on just
9:12continual learning for models, this is
9:15bringing continual learning into the
9:17harness. Another thing that's worth
9:19knowing here is that the core repo here
9:21isn't taking outside pull requests now.
9:23They're pointing people to discussions
9:26and building plugins instead, which
9:29actually makes total sense for this
9:31design, right? If there's no core to
9:33patch, then plugins are how you
9:36contribute. And that's what brings us
9:38back to the whole super agent. So, if
9:40you open the super agent repo, one of
9:42the first files you'll see is the
9:43cordis.patch.yaml file. And that file is
9:46basically the whole integration. Inserts
9:48a core super agent plugin, a GitHub
9:51watcher plugin, and a facial watcher
9:53plugin by default. Now, obviously if
9:55you're not using those, you could just
9:56turn those off. But the way super agent
9:58works is that it defines a separate
10:00preset for the workers. You can think of
10:03it as DSH's standard preset with a bunch
10:06of things stripped out. No asking the
10:08person questions, no plan mode, no web
10:11fetch, no spinning off sub agents, etc.
10:14And the shell is set so that any command
10:16running longer than 5 seconds gets
10:18pushed to a background job. That way a
10:20long build doesn't stop the worker from
10:22noticing that someone just left a new
10:24command as it's going through.
10:26Obviously, that can be one of the
10:27frustrating things with Claude code and
10:29Codex is once it starts going, you're
10:32sitting there waiting to basically
10:33update it and you perhaps want to inject
10:35a prompt in while it's still going as
10:38you realize it's going in the wrong
10:39direction. So, what you're looking at in
10:41this agent is a fairly different system
10:43entirely. It's built from configurations
10:46and plugins. There's no forking of the
10:48harness and really that's the whole
10:50payoff of the DSH design that we've just
10:53talked about. Now, DSH itself did
10:55actually have an experimental agent
10:57teams feature with a roster, a task
11:00board, a mailbox. So, why build this?
11:03Well, you can think about agent teams is
11:05about coordinating agents inside one
11:08job. The super agent is about handling
11:11lots of independent jobs that arrive
11:14from the outside world over perhaps a
11:16long period of time and also from lots
11:18of different people posting here. It's
11:20actually addressing a different problem
11:22than what the teams feature did. And one
11:24cool practical detail here is that the
11:26router and the workers can each be given
11:28a different model. So, I'll come back to
11:30that in a second. Now, if we look at the
11:32architecture here, this basically has
11:35two layers to it. On top there's one
11:37long running session that they call the
11:39router. It handles all the human
11:41interaction, keeps the big picture
11:43context, hands out work to perhaps
11:45different plugins or different agents.
11:48Underneath are those worker agents, each
11:50running in their own isolated session.
11:53Workers never have to talk to people
11:54directly. They report back to the
11:56router. The router does all the
11:58communication. Everything going in from
12:00the outside is a trigger. A new issue, a
12:03review comment, a chat message, etc.
12:06Each trigger first goes through a gate,
12:08which is plain code and not even a
12:10model. And that gives you a
12:11deterministic feature to basically
12:14check, is this person even on the
12:16allowed list to be able to comment or to
12:19be able to file a pull request, etc. It
12:21also helps it to answer things like, did
12:23they actually address the agent? Now,
12:25only when those checks are passed does
12:27the model actually get to look at it and
12:29decide, you know, whether it's real
12:30work. If it is, it becomes a ticket with
12:33what they call a mandate. That's a
12:35record of who asked for it, under which
12:37rules, how far the work is allowed to
12:39go, etc. That ticket and the mandate
12:41then get passed to a worker. Now, here's
12:44a really cool detail that they've got in
12:46there. Once a worker owns a ticket, it
12:48subscribes to that conversation. So,
12:51when someone comments on the PR that it
12:53opened, that comment goes straight to
12:55the worker. It skips the router
12:57entirely. And you can think about that
12:59skipping of that orchestration layer and
13:01going straight through just saves you a
13:03huge amount on tokens, etc. In fact,
13:06when they turned off this in their
13:07experiments, router decisions actually
13:10went up by 30% and obviously the amount
13:13of tokens went up with it. For me
13:15though, the part that I find most
13:16interesting here though is about the
13:18router as the bottleneck. So, we would
13:20assume that one central session that's
13:22doing the orchestration, handing
13:23everything out, would just get
13:25overwhelmed as things get busy. It
13:28actually turns out here that it kind of
13:30does the opposite. When triggers arrive
13:32while the router is in mid-inference,
13:35they pile up and they get handled
13:37together in the next call. And because
13:39the way that they've structured that
13:40router's job to be sort of long input
13:43and short on output, it actually allows
13:46it to batch these things really cheaply.
13:48Now, in Deep Six stress test, they
13:50pushed the trigger rate from about one a
13:52minute to about 40 a minute. Average
13:55batch size went from one to 2.8. And
13:58sure, each call did get a bit slower
14:00from about 2.5 seconds to 3.9, but the
14:04time per decision, because it's being
14:06done now in batches, actually drops from
14:086.4 seconds down to 1.6 seconds. So, it
14:12got more efficient under load. They also
14:15say that the router job is often simpler
14:17than you'd expect and a lighter model
14:20usually handles it fine. That kind of
14:22goes against the grain for a lot of
14:24thinking of where people have been
14:25thinking that your orchestrator needs to
14:27have the best model and that your sub
14:29agents can get away with simpler models.
14:31It turns out that that router doesn't
14:33even need a bash tool. So overall that
14:35router can use a fast, cheap model. You
14:38can actually use the stronger model for
14:40some of your workers. And this is
14:41something that I definitely want to test
14:43more with open models going forward.
14:45Anyway, on top of this they have this
14:47philosophy, which I think is a really
14:49useful part. The principle is take as
14:52little human attention as possible and
14:55give as much value as possible. And I
14:57kind of feel as we're building agents,
14:59it's really hard to argue against that.
15:01You want an agent that only acts when
15:03someone allowed asks for it or when a
15:05rule the team wrote actually allows it.
15:08Interestingly, even in there, when work
15:09nobody asked for starts, it actually
15:12starts in a mode where it just records
15:15what it would have done. It then
15:16confirms itself whether or that would
15:18have actually worked and then only
15:20speaks up when there's a result or a
15:22decision that somebody needs to make. I
15:24really think there are a lot of lessons
15:26in here that all of us can take away for
15:28how we build our own agents. Okay, so
15:31what's missing here? Let me go through a
15:33few caveats here. First, this is a
15:35prototype and they say that really
15:37clearly. The published code isn't
15:39identical to their internal version.
15:41Some designs in the article are actually
15:43future directions that aren't in the
15:45repo yet. And from what I can see,
15:47things like the watchdog session perhaps
15:49isn't fully there yet and I'm sure there
15:51are others that are missing as well.
15:53Second, this is all kind of narrow. It's
15:55one GitHub repo. They haven't set this
15:57up to work with Slack or with email. But
16:00actually I'm not really worried about
16:01that because everything is basically a
16:03plugin. They've also included a cookbook
16:06for adding your own source code in
16:07there. So my guess is very quickly we'll
16:09see forks of this with lots of other
16:11integrations in there. Third up, the
16:13thing that we should be a little bit
16:14concerned about is that the super agent
16:17experiments that they ran on and
16:19reported their benchmarks on are
16:21actually run on synthetic workloads in a
16:23simplified environment. And they say
16:25themselves that this is basically just a
16:27preliminary reference. And on things
16:29like the 30 tasks, this is just a small
16:31sample. So, just treat those rankings as
16:33a rough guide. Finally, the big thing to
16:35be a little bit concerned about here is
16:37security. They do list the problems of
16:40prompt injection, credentials leaking,
16:42permissions across devices, etc. as
16:44things that they haven't fully solved
16:46yet. So, naturally, you probably want to
16:48be a bit careful about how you use this.
16:51But, if you are using it with local
16:52models in a totally locked-down
16:53environment, you probably don't have a
16:55lot to really worry about here. All
16:57right. So, overall, while I think super
16:59agent is a really interesting product
17:01here, what is even more interesting here
17:04is just basically the proof of design.
17:06Deep Seek has built something which is
17:08quite a different agent system on top of
17:10their own harness with just a few patch
17:12files and plugins without touching the
17:15core of the harness at all. And for me,
17:17the bigger point here is just showing
17:18you how much harnesses are evolving at
17:21such a quick pace at the moment. Anyway,
17:23as always, I've put the links to the
17:25repos, the paper, the benchmark, all of
17:27that's in the description. Let me know
17:29in the comments what you think. Do you
17:31see the future of the harnesses of being
17:34these composable plugins where different
17:36people create different ideas in
17:39different ways, and we're not just
17:41reliant on the same coding agent
17:43interfaces that we've seen up until now?
17:46As always, if you found the video
17:47useful, please click like and subscribe,
17:49and I'll talk to you in the next video.
17:50Bye for now.

About this transcript

This page contains the full transcript of NEW DeepSeek Harness - Super Agent! by Sam Witteveen, generated from the public captions YouTube serves with the video. The transcript has 3,332 words across 484 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.