YT YouTube2Text

YouTube2Text → Transcripts → Agentic Engineering Benchmarks: How I RANK Astra, Fable 5.1, and Open-Weights

Agentic Engineering Benchmarks: How I RANK Astra, Fable 5.1, and Open-Weights — Transcript

by IndyDevDan · 7,941 words · 1,177 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00What's up engineers? Any Dev Dan here.
0:02As the intelligence explosion continues
0:04and new models like Claude Fable 5.1 and
0:07Open AI's GPT-6 Astra continue to push
0:11the frontier, the signal that benchmarks
0:13are giving us is getting blurry. The
0:16Artificial Analysis Index is where most
0:18engineers go to get their information.
0:20It's recently been updated to the 4.3
0:22version and this benchmark itself is a
0:24bit overloaded. There's a lot of
0:26compressed information here in an
0:28attempt to create a broad sweeping
0:30statement around what models are the
0:32best. The problem with indexes is that
0:35not all benchmarks are created equally.
0:37Some are vastly more important than
0:40others, but that depends on what you're
0:43trying to accomplish as an engineer. Do
0:45you know what benchmarks matter most for
0:48your work and why? Part of the problem
0:50is that benchmarks are moving target.
0:52The way we used to use agents just 2
0:55months ago is not the way we should use
0:57agents now or in 2 months. On this
0:59channel, we move where the ball is
1:01going, not where it is. I've been
1:03building with language models since it
1:04was first possible way back in 2023 and
1:07as you've likely noticed, no single
1:10benchmark is perfect, but each benchmark
1:12does tell a story about what each model
1:15can do for you. Here's a question I
1:16think every agentic engineer should be
1:18able to answer. If you could only pick
1:20five benchmarks, which five [music]
1:23would you pick? In this video, that's
1:25exactly what we're going to do. I'll
1:26share which benchmarks I would pick if I
1:28could only choose five. This forces
1:31decision-making criteria and it forces
1:33you to prioritize and push away [music]
1:35from these catch-all indices that are
1:37valuable, but only to a point. By the
1:40end of this video, you'll have a deeper
1:41understanding of how to select and
1:43[music] analyze benchmarks for your
1:45agentic engineering. Let's create a top
1:48five benchmark for agentic
1:50>> [music]
1:50>> engineering.
1:55If I only had access to five benchmarks,
1:58here's what I would pick. We'll start
2:00roughly with the most important one, but
2:02they all matter. What I want to do here
2:04is show that although the AA index is
2:06very powerful, you can quickly
2:08outperform it and pick better models at
2:10better prices by just picking benchmarks
2:13that align with the work you
2:14specifically want to accomplish. Let's
2:16start with the number one and
2:18potentially most important pick.
2:19Terminal Bench is the clearest pure
2:22agentic coding benchmark. Here's what it
2:24looks like. You send it a task prompt,
2:26an agent runs inside of a prepared
2:28container with code, data, and system
2:30state. It runs your usual harness loop,
2:33looping over commands and results, and
2:35then a verifier validates the resulting
2:38state for a pass or fail. It has over 60
2:40tasks that span software engineering,
2:42machine learning, science, operations,
2:44security, hardware, and media. Choosing
2:47a model is a three-dimensional problem.
2:49It's about performance, cost, and speed
2:52together as a single unit. This is the
2:53tradeoff triangle, and you have to
2:55consider all three when you're choosing
2:56models. Terminal Bench tells us that
2:58Astra is in the lead on performance.
3:01Cloud fable 5.1 trailing, and
3:03interestingly, there's a big drop-off in
3:06models at that 40 to 42% level. When I'm
3:10choosing benchmarks, I'm constantly
3:12looking for actual variance. It's
3:14important that I see models falling off
3:17a curve because that means there's some
3:18alpha in using the top state-of-the-art
3:21models. Interestingly here, you can see
3:23Kimik 3 is not great here. Why is that?
3:27Did all that distillation miss
3:28something? On the flip side, we can see
3:30GLM 5.3 performing a lot better, and
3:33this creates a serious gap in my own
3:36model ranking. I like to separate models
3:37into three chunks: state-of-the-art,
3:40workhorse, and lightweight. And you can
3:42see here there's a huge gap and a
3:43misplaced model where Kimik 3 is
3:46technically above GLM 5.3. So, how can
3:49we reconcile this? We do this by looking
3:51at multiple benchmarks. One benchmark is
3:53not enough, but of course score isn't
3:55all that matters. How many tokens did it
3:57take you to get there? How long did it
3:59take? How much did it cost? If we search
4:00for Astra, the like very clear
4:03state-of-the-art winner right now, we
4:04can see something incredible. Astra Max
4:06is using 2.7x less than Cloud Fable 5
4:10and about 2.5x less than Cloud Fable
4:135.1. So again, performance, speed, and
4:16cost all matter together. A lot of these
4:19models burn a ton of tokens to get to
4:21the result. As you can see here, GLM 5.3
4:24is an example of one of those models.
4:26Same with Gemini 3.8 Flash or GLM 5.3
4:30Flash. A lot of these powerful workhorse
4:32models in that A and B tier, very
4:34powerful, and they're a lot more
4:35affordable, trade-off token usage to get
4:38you there. As we look at Terminal Bench,
4:39we can see that. And of course, when we
4:41look at score versus output tokens, we
4:43can very, very clearly see that Astra is
4:46the winner. It's not just about
4:47performance, it's about what it costs to
4:49get the performance. The big equation I
4:51like to use now is not tokens versus
4:54result or output tokens per task, it's
4:56useful agent output per hour, which is a
4:59little bit harder to measure, as you can
5:01imagine, right? It's what are the actual
5:04valuable outputs I'm getting per hour of
5:06token spend that's going into this
5:08system. Every benchmark we're going to
5:09look at here is going to lead up to that
5:11big idea. What you're trying to
5:12accomplish directly dictates the
5:14benchmarks you should pay attention to.
5:16So you can see Astra here is absolutely
5:18smashing. And we can even pull in a
5:21couple of the other Astra levels, right?
5:22Medium and low, and you can see they're
5:24all near or right at the frontier for
5:27the cost. Now, let's move on to cost.
5:29This is where things get really wacky.
5:30We can use Astra as our primary model.
5:32As I'm looking through these benchmarks,
5:34I like to use one model as my kind of
5:35control group for the rest of the
5:37benchmark, and you can see here once
5:39again, we have an insane split in actual
5:42cost. This is input and output tokens
5:44put together, GPT-6 and cache tokens, of
5:46course, but GPT-6 Max, look at that
5:49price difference from Fable. And so, if
5:51you were just looking at performance,
5:52you would look at this benchmark, or you
5:54would look at the headlining artificial
5:55analysis index, and you might just go,
5:57"These two models are tied. It doesn't
5:59matter which one I use." That's not the
6:00case. If you care about in terminal
6:02agent coding, Astra is a clear, clear
6:05winner because performance might be your
6:07top priority, but then you have a second
6:09priority. Is it speed or cost? For most
6:11of us, it's going to be cost. Look at
6:12this huge gap here. This is more like
6:15four times. 8, 16, 24, 32. Yeah, about
6:17four times cheaper to run Astra than it
6:20is Fable, and about 2.4 times better
6:23than running Claude Fable 5.1. Astra,
6:25clear winner here on Terminal Bench, and
6:27you know, we can see that in the chart
6:28as well. It's in that sweet spot. Now,
6:30speed is the kind of third variable that
6:33I always look at across benchmarks,
6:35although this is the one that I'm
6:36willing to give up the most. You can see
6:39Astra, not great, but not terrible in
6:41terms of speed. Very, you know,
6:42workable. This is a time per task. This
6:45is more aligned with that key equation
6:47that I'm looking for. It's useful agent
6:49output per hour. Useful agent work per
6:51hour. And then it scales all the way up.
6:53You can see, unfortunately, a lot of our
6:55great open weights models spending a lot
6:58of time thinking in order to get that
7:00result. That means that it's going to
7:01slow down the model. The more you have
7:02to think to solve a problem, the slower
7:04you'll be able to solve the problem.
7:06Simple, but important to emphasize.
7:07Terminal Bench is a really important one
7:09because it puts together those three key
7:11variables against a very, very simple
7:14benchmark. And of course, you can always
7:15dive into the details here. I'm not
7:16going to over fixate on every single
7:18detail, but if you're interested,
7:19there's a great breakdown of every
7:21single task that Terminal Bench V4 goes
7:23through. You can see for this specific
7:25task, the task metadata, the type of
7:27machine the agent will be working on,
7:29agent timeout for this sandbox, the
7:31difficulty explanation, and then we have
7:33the instruction.md for the actual task.
7:36Reverse engineer the layout and write
7:38output.config.json with the following
7:40format. So, for this specific task, the
7:42agent has to read this image, understand
7:44the components, and then reverse
7:46engineer and write this layout in this
7:48specific format. Image, component, text,
7:50yada yada yada. So, it has to output all
7:52the details of every single component
7:54here. It has constraints, do not cheat,
7:56here's a time limit, here's the output.
7:58So, you can imagine a bunch of tasks
7:59like this that make up the terminal
8:01bench. So, this is a pure agentic coding
8:03benchmark for AI agents with a clear
8:05focus on software engineering tasks. Our
8:08next benchmark that I paid close
8:10attention to for agentic engineering
8:12work is Apex Agents.
8:18So, Apex Agents is interesting because
8:20it's a measure of agent performance over
8:22three data-rich professions. The
8:24professions are investment banking
8:26analysis, management consulting, and
8:28corporate lawyers. So, three concrete
8:31roles, three non-trivial knowledge work
8:34professions. And I use the Apex Agents
8:36benchmark as a proxy for other digital
8:40domains, further knowledge worker
8:42domains. These are very hard domains.
8:44All the tasks have been vetted from
8:45experts at McKinsey, BCG, Deloitte,
8:47Goldman Sachs, Morgan Stanley, JP
8:49Morgan. We have experts creating these
8:51tasks that our agents are working
8:53against. And you can see here they're
8:54getting closer to that saturation point.
8:56I think benchmarks get saturated at
8:58about that 85 and of course 90 plus
9:00percentage. But we can see a nice
9:02breakdown here. It's nice and simple.
9:04Who's winning, who's losing? We are
9:05missing information from this benchmark,
9:07specifically the cost and the time taken
9:10to accomplish this work. That's
9:11important information, but nonetheless,
9:13I like this benchmark because we're
9:14looking at real knowledge work tasks.
9:16Let's dial into the management
9:17consultant. As you would expect, we have
9:20Astra, Fable, and then surprise, MuSpark
9:221.1. Imagine where MuSpark 1.3 is going
9:25to place. Okay, Kimik 3 is up here, Grok
9:284.6, Fable 5 a lot worse for some reason
9:31on this benchmark. And the whole idea
9:33here is we have agents taking in inputs
9:37in their workspace. They're reading
9:39documents and they're doing real digital
9:41knowledge work outside of the software
9:43engineering domain. And this is
9:45important to not over fixate on software
9:47engineering. We want to look at real
9:49tasks, real domains that you and your
9:51company might be building around. Real
9:53domains that you're deploying software
9:55engineering and deploying your AI agents
9:57against. And so, the best way to do that
9:59is just to look for hard domains that
10:01your agents have to work really hard for
10:03and then treat that as a proxy for your
10:05domain. Every benchmark is a proxy, but
10:07by looking at this benchmark, we get a
10:09clearer idea of how specific models will
10:12perform in other tangential knowledge
10:14worker domains. So, for example, one of
10:16the management consultant tasks looks
10:17like this. This is a sample task. Use
10:19the estimated market share chart and
10:21Brightpath customer segmentation. Please
10:23calculate the potential revenue for the
10:24SMB accounting segment if it achieved
10:26the target share. A nice high-level
10:29prompt. And another great part about
10:31this benchmark is that it better
10:32resembles knowledge worker prompts. So,
10:35you and I as engineers, we can use and
10:37manipulate our agents in really powerful
10:39ways to create powerful skills, powerful
10:41system prompts as we did a couple weeks
10:43ago to fix Opus 5's smart-ass output. We
10:47can do a lot of different things. But
10:48when you're having internal users or
10:50customers use your product, you're going
10:51to have to engineer your system to take
10:53in their types of prompts. And that's
10:55why this is important, right? They're
10:57not going to be writing full-on
10:58engineering plans. You can imagine some
11:00deeper level of understanding from a
11:02knowledge worker, from a management
11:04consultant, but nothing like what you
11:06can do as a software engineer. So, it's
11:07important to have a benchmark that
11:08reflects those types of inputs. And then
11:10we have, of course, the environments.
11:12So, give them access to specific tools,
11:14show the actual file structure. This
11:16sample task here shows down the actual
11:18breakdown, the actual trajectory of the
11:19agent, which is great to see. So, Apex
11:21agent is an important benchmark. I pay
11:23attention to this. They also just have
11:24their Apex Suite, which is a little bit
11:26more broad. They have one for
11:27accounting, they have one for software
11:28engineering. But specifically in trying
11:30to diversify my five benchmarks, if I
11:33had to pick five, this would be my
11:35number two because it pulls away from
11:37our big but small world of software
11:39engineering. There are hundreds and
11:41thousands of other domains you can apply
11:43agents into and that you probably are
11:44inside of your business and this gives
11:46us a proxy for understanding how agents
11:48will perform in hard domains. [music]
11:51All right, so next up we have a really
11:52important benchmark. We have automation
11:55bench.
12:00So before we dive into automation bench
12:02and the importance of it, I just want to
12:03take a quick break here to shout out
12:06everyone that's watching, everyone
12:08that's been viewing the channel,
12:09everyone that's been returning week
12:10after week. And if you're new, you know,
12:12big shout out to you. We've been on a
12:14nice smooth growth curve for the past
12:16year here. We're almost at that 150,000
12:19mark and every time this channel moves
12:22closer to a big milestone, I always
12:24think we were never supposed to get this
12:26big. I started this channel to share
12:28engineering ideas that I thought were
12:30missing in the market surrounding
12:31language models, surrounding prompt
12:33engineering, surrounding keeping that
12:35core of what software engineering is as
12:38we move into this new age of AI. It's
12:40been really incredible to have engineers
12:41that consistently join the journey and
12:43follow and learn. The industry has
12:45changed a lot, but we have continued
12:48throughout to evolve throughout it. So I
12:49just want to say, you know, a big thank
12:50you to everyone that's subscribed, to
12:52everyone that's been watching videos,
12:54everyone that likes and shares and all
12:55that. I really, really appreciate you
12:57being here. It feels so great to have
12:59other engineers who are interested in
13:01pushing what you can do with brand new
13:03technology. It feels great to have
13:04optimists that are commenting, that are
13:06paying attention, that are watching week
13:08after week. Even if you don't leave any
13:09comments, right? Most engineers watching
13:11this, you don't leave any comments, but
13:12you show up to get information to help
13:14you decide how you want to build and how
13:16you want to leverage this technology. So
13:18I just want to thank you and of course I
13:19want to shout out every
13:21agenticengineer.com
13:22member. The phase three product is on
13:24the way, so stay tuned for that.
13:26Everyone that's a member is going to
13:28receive a discount on what's coming
13:29next. I'm so excited to share the ideas
13:31there. The things we can do with this
13:32technology is not well understood, and
13:35week after week, I'm going to keep
13:37sharing so that you can make that career
13:39leap, so that you can start that
13:40business, so that you can move up in the
13:42company you're already working at.
13:43That's the goal here. It's real tangible
13:45engineering values. What a single
13:46engineer can do is no longer limited,
13:49and on this channel and on a
13:50geneticengineer.com, we're going to
13:52prove it. Huge thanks to you. Smash that
13:54subscribe button. Let's push past 150K,
13:57and let's jump back into automation
13:59bench. So, automation bench is more of a
14:01classical software automation across
14:05different applications. We're still in
14:07that realm of knowledge work, right?
14:09We're doing digital work with agents,
14:11but here agents start acting more like
14:14employees. Okay? And so, this is the
14:16interesting part about this benchmark.
14:17We have over 600 tasks across six
14:20business domains: finance, HR,
14:21marketing, operations, sales, support,
14:23and a bunch of common tools that you and
14:26I use every single day, right?
14:27Communication tools. So, our agents are
14:29not just doing the work, they're pulling
14:31information from common applications and
14:33resources, and they're getting work done
14:35across applications. Okay? So, this is
14:38super important, and a key key idea here
14:40is that the model has to complete
14:41objectives without triggering guardrail
14:44violations. Okay? This is a big big big
14:48really important piece that a lot of
14:50benchmarks don't capture. They capture
14:51pass or fail. This says, "Here's the
14:53goal. You need to do this to accomplish,
14:55and if you bump against a guardrail, you
14:57also fail. You also lose points."
14:59Guardrails are critically important.
15:01What is this? Is it what it looks like
15:02to be aligned at a low level? What does
15:04alignment really mean? It means the
15:06model is doing things you asked it to
15:08do, and it's not doing things you did
15:09not want it to do. And so, again, that
15:11comes down to being great at prompt,
15:13context, harness engineering, but also,
15:15this is just what the core four is,
15:18right? And part of the core four is the
15:19model. Some models are just going to
15:22break guidelines. They're going to break
15:24your guardrails more often than other
15:25agents. So, I really like Automation
15:27Bench for this specific reason. It is
15:29doing common knowledge work tasks, and
15:32it's asking, "Did you bust something on
15:34your way up?" Right? Did you break
15:36something on your way to completion?
15:38Right? Imagine a co-worker where they
15:40accomplished the task, but they broke
15:42something else along the way. That's not
15:44as valuable. That's very clearly not as
15:46valuable. So, you can see a couple
15:47interesting things here. Where is that
15:49orange Claude line? Instead, we have
15:51Grok 4.6 up here. We have GLM 5.3. And
15:55tracing it, we have GLM 5.3 Flash.
15:57Fascinating. And below that, we have GPT
15:595.6 Soul. So, again, if you were just
16:01looking at one benchmark, you would miss
16:04this. If you were looking at an index,
16:05you would miss this. It's important to
16:07understand what you're trying to
16:09accomplish. And so, again, as I'm
16:11working through this, try to really
16:12think about, "What am I optimizing for?
16:14Why did I pick these five benchmarks?
16:16What story am I trying to complete? What
16:19proxy am I building for the capabilities
16:21I'm looking for in language models?"
16:23Make a good guess as we work through
16:25this. And so, all the way down here, we
16:26have Claude 5.1. Okay? And by all the
16:28way down here, we're really only down,
16:30you know, let's say, 8%. Okay? Or 9%.
16:33So, it's not a massive amount, but for a
16:35state-of-the-art model that just came
16:37out, you would expect this to be a bit
16:38better. And then, we just go downhill
16:40from here. A decent amount of variance
16:41here on the score of Automation Bench.
16:44And again, the key here is what makes
16:45this trickier and very important and
16:47valuable is that there are no guardrail
16:49violations. So, that's key. You have to
16:51accomplish the task while not breaking
16:54guardrail violations. Very important.
16:56You can see most agents completed the
16:58work, but that's not what this is about.
17:00objectives completed regardless of
17:02guardrail violations. So, all of a
17:04sudden, you can see Opus jumped up here.
17:05Fable jumped up here. But again, this is
17:07taking out the guardrail violations. So,
17:09now you're looking at this, right? You
17:11have more information. Holy If I
17:13need a very strict performant model
17:16that's going to really follow the
17:17instructions to the T and not break
17:19guardrails, maybe I shouldn't use Fable.
17:21Maybe I shouldn't use Opus. It's all
17:23about the use case you're looking for.
17:25Again, Astra, big winner here. Now, of
17:27course, as you know, it's about
17:28performance, speed, and cost. What about
17:30tokens? So, here's another good view of
17:33the violations. Higher is better. Did
17:35you accomplish the task while not
17:37triggering a bunch of violations? It's
17:39another great one you can take a look at
17:40here. Right, here's a bunch of domains,
17:42finance, HR, marketing, operations,
17:44sales. You can see here, for some
17:45reason, Fable having a hard time with
17:47finance, while Astra looks a lot more
17:50aligned. And again, I'm not just trying
17:52to sit here and glaze Astra. This is
17:54just what's showing up across domains,
17:56right? Across benchmarks. Same kind of
17:59trend here. Looking at performance
18:00across all these applications, check out
18:02Astra. You can see where it's strong,
18:03where it's weak. For instance, you know,
18:05digging into the benchmark just a little
18:07bit, going a depth two, depth three in
18:09some of the work you're doing, some of
18:11the research you're doing on benchmarks
18:12is going to help you a lot. For example,
18:14if you're using Gmail a lot, you're
18:16building a Gmail software factory, or
18:18Gmail AI developer workflow, what
18:21options do you have outside of the state
18:23of the art? Well, take a look at this.
18:24Rock 4.6, not looking too bad. And look
18:27at this, crazy option, Quinn 3.8, not
18:29too bad. We also have GLM 5.3, not
18:31looking bad at all. But, if we take a
18:33look at DeepSeek here, maybe you want to
18:34stay away from DeepSeek, right? You
18:36don't just want to default to a cheap
18:38model that's in the workhorse level,
18:40A-tier, B-tier. It's not that simple to
18:42just pick a model and stick with it.
18:43This is why I always advocate you want
18:45to model stack. It's too small-brained
18:48to just think that one model can do
18:49everything you need across every domain,
18:51right? Especially if you're doing and
18:53scaling real product agents, agents
18:55working in your product that you build
18:57via SDKs, one model is not the answer.
19:01Combine compute, don't select compute.
19:03There's different slots here. If you're
19:05willing to take a performance hit in a
19:06specific domain, on a specific tool,
19:08that's important to know about.
19:09Automation Bench gives you a little
19:11insight into that. Returning to time and
19:14token, how fast, how much does it cost?
19:16And token usage is a piece of that
19:18puzzle. Once again, let's just look at
19:19our control model, look at how low Astra
19:22is. It's using basically no tokens when
19:24you compare it to the rest. Well, really
19:27GPT-OSS 120 billion is using no tokens,
19:29but it's also on the result.
19:31But you can see the rest of these models
19:32are really, really consuming more and
19:34more tokens. And actually we have to
19:36give Fable a little bit of credit here.
19:37It's actually performing in a nice range
19:39of token usage here. Of course, we know
19:42that this picture changes a bit when we
19:43look at costs. Astra here on the higher
19:46range of our costs, but still almost
19:48half as cheap as these Fable 5.1 and
19:51Fable 5 models. A lot of value here in
19:53understanding your performance, speed,
19:56cost trade-off. So I really like
19:58automation bench. Feel free to dive in,
20:00obviously, to the details of any one of
20:02these benchmarks. All of them all linked
20:03in the description for you. So this is
20:05my number three pick. Now the next two
20:07are very important. I'm getting five
20:09picks here. The next two really, really
20:11complete the picture. It's not just
20:12about performance, it's not just about
20:14breaking guardrails. The next benchmark
20:16tells us if our agent is telling the
20:18truth. [music]
20:23My fourth pick for benchmarks, if I
20:25could only pick five, the Omniscience
20:27benchmark. The Artificial Analysis
20:28Omniscience benchmark is basically the
20:31hallucination benchmark, right? That's
20:33the simplest way to put this. It gives
20:34us information into one of the most
20:36important and criticized aspects of
20:39language models. It's the hallucination
20:40rate. And specifically, we can make this
20:42more real, right? It's really answering
20:44what does it cost for your agent to be
20:46honest? What's the cost of reducing
20:48hallucination? And it's really answering
20:50which model can you actually trust as
20:52you scale up your tasks? That's what AI
20:55Omniscience is really telling you. Now
20:57you probably stop noticing that
20:58hallucinations are occurring.
21:01As people and engineers stop looking at
21:03their code, stop looking at all the
21:04details of their plan, do less and less
21:06review, we're going to miss these
21:07things. And so, one of the best ways to
21:10continue missing it is to pick a model
21:11that doesn't do it at all. Pick a model
21:13that bails out when it doesn't have an
21:15answer. That's the important part of
21:17this. One of the key parts about this
21:18benchmark is this. Each answer is graded
21:21as correct, incorrect, partial, or not
21:24attempted. The agents can opt to not
21:27answer at all. This is super key. As we
21:29talked about in last week's agent swarm
21:31video where I showcased a V1 version of
21:33my simple swarm system inspired and
21:36unlocked, honestly, by OpenAI's GPT-6
21:39Astra Swarms where they hacked
21:41themselves and they hacked Hugging Face.
21:42The prototype agent swarm, which of
21:44course I'll link in the description for
21:45you. That video absolutely exploded for
21:47a good reason. One of the things that
21:49they failed to do in one of their
21:50benchmark tests that caused their agents
21:53to hack themselves and hack Hugging Face
21:55is they didn't give it away to say I
21:56don't know, right? They didn't give it
21:57away to say I can't answer this, so I'm
22:00going to stop. Now, obviously this is a
22:01little tricky cuz you want your agent to
22:03keep trying, to keep working toward that
22:05goal even in the face of difficulty.
22:07But, you need to give your agent a way
22:09to say I can't do this. I'm done. I
22:12can't, right? I don't know. And the nice
22:15part about Omniscience, the way that
22:16this is graded, is that they have no
22:18score change for saying that they don't
22:20know. And so, you can see that right
22:21here, right? This is super, super
22:22important. The value of these benchmarks
22:24is in the details, right? It rewards
22:26correct answers, penalizes
22:27hallucinations, and has no penalty for
22:29refusing to answer. Very powerful. The
22:32agent can say I don't know. A lot of
22:34benchmarks miss this and it's a nice
22:36thing to add to your benchmark. There's
22:38pass, there's fail, and there's I don't
22:40know. All right? There's an honest, I'm
22:42not sure, and this gives your agent a
22:44way to back out. So, why would this be
22:46important? Keep your mind on that as we
22:48work through our last two benchmarks
22:49here. Why would it be important for your
22:51agent to be able to say not sure?
22:52Looking at the actual raw numbers, our
22:54most trustworthy models are the ones you
22:56would expect, the ones you have to pay
22:57the most for. You have to pay for
22:59honesty and consistency, but we do have
23:00a couple great runner-ups. Looks like
23:02these other models are actually more
23:04truthy than Grok 4.6. We have Gemini 3.8
23:07Flash. Love to see that. That's one of
23:09my favorite workhorse models right now.
23:12We have the surprising Muse Spark 1.3.
23:14And then, of course, Soul. But things
23:16really kind of fall off a cliff after
23:18this point. So, you can bet that a lot
23:20of the safety and alignment research and
23:22training that's going on in our top two
23:24labs, Anthropic and OpenAI, is paying
23:26off. You can see that in this benchmark.
23:28So, again, this is why the specific
23:30benchmark matters. You lose information
23:32in indices. The index is a quick proxy
23:35of a proxy. You're getting pretty far
23:37away from what you're really looking at,
23:38okay? And I say that as we're building
23:40up my top five index as well, right? So,
23:42that looks good. If you're looking for
23:44honest models, you can see which models
23:46you should stay away from, right? You
23:47got to be careful. Or not stay away
23:49from, but just models to be more aware
23:51of. 5.3 Flash, mhm, might hallucinate on
23:53you. GLM 5.3, a little bit better, or
23:55not a little bit, twice as good. So, if
23:57you're looking for consistent,
23:59non-hallucinating models, you're going
24:00to want to move up the scale. And zero
24:03here means as many correct answers as
24:06incorrect answers. All right, so this
24:08zero point is basically a 50/50 shot on
24:10a factual answer. Okay, so very risky,
24:13and you can see it just falls off a
24:14cliff here. And so, you know, looking at
24:16Qwen 3.8, 27 billion, and Luna, small
24:18models uh having a hard time finding
24:21that factual information for you, right?
24:22So, super powerful one. Here's the
24:24overall accuracy. Uh you can see there's
24:26a lot of value here in choosing
24:28state-of-the-art models, but also Gemini
24:30Flash, really good. DeepSeek V2 Pro,
24:32pretty good. Grok, Kimmy, not bad,
24:34right? Unfortunately, we have GLM 5.3
24:36falling off a cliff here. So, again,
24:38just digging into more of the details,
24:40understanding where models are strong
24:41and weak across performance, speed, and
24:44cost across different benchmarks. That
24:46looks good. Interesting thing that they
24:47have here is a language performance. Um
24:50it looks like Muse Spark, although it
24:52performs pretty well, starts getting
24:53kind of rough on software engineering QA
24:56tasks and truthiness across software
24:58engineering tasks. I definitely would
25:00not use any of these models down in this
25:02range if you need reliable, consistent
25:05outputs. This range here, all usable
25:07models, right? These are all shippable,
25:09usable models. And that continues our
25:10story here of you want to use a model
25:12stack, not an individual model. Combine
25:14compute, don't select compute.
25:16Truthiness, right? We're looking at how
25:18honest our model is and we're not
25:20penalizing our model for saying, "I
25:21don't know." Which is going to be
25:23increasingly important as we scale up
25:25our compute to scale up our impact. Our
25:27last benchmark is one you would expect
25:30with one key focus. We're looking at
25:32Deep SWE.
25:37Deep SWE is a relatively new benchmark
25:39that pushes on SWE-bench Pro and other
25:43variants by forcing the agent to do
25:45longer-running tasks. Okay, so here's
25:47the old version, here's the updated 1.1.
25:49Um, this is missing FABLE 5.1, so just
25:52to call that out. This is something that
25:54I also look for when I'm looking for
25:56benchmark providers. You want to see
25:58that they're actively maintaining the
25:59models and rolling out the fresh updates
26:02as soon as possible. I imagine they'll
26:03have FABLE 5.1 in here pretty soon. One
26:05thing I hate to see is model provider
26:08bias. I immediately just disregard
26:10benchmarks that are not including
26:12specific models on purpose because as
26:14engineers with our boots on the ground,
26:15we just have to be real about where the
26:17leverage is. But you can see here, we
26:19have a nice surprise. Deep SWE, right?
26:21Performing long-running software
26:23engineering tasks. The key here is, let
26:25me emphasize this, right? It's
26:26long-horizon work. This is one of the
26:28big, big things I'm focused on more and
26:31more and more is long-horizon work. One
26:33of the things I also liked about this
26:35benchmark is that they use realistic
26:37short prompts. So, why would I like
26:39that? If you've been following the
26:40channel, you know that I'm a big
26:41advocate of planning your work. What is
26:43a plan? It's a prompt scaled up. It's a
26:46big prompt. What I like about this is
26:48this emphasizes that idea even more. If
26:50you can hand a shitty, short, high-level
26:53prompt into one of these agents and they
26:55can accomplish a large amount of
26:57engineering work on your behalf, guess
26:59what they can do with a ton of details
27:01and a detailed plan you've written out
27:03and you've invested effort into. I know
27:05that there's a large cohort of engineers
27:07that are more agent take the wheel, more
27:09less input, more do less, try to get
27:12more out. I think that's veering towards
27:13the floor of what's possible, which is
27:16tending toward live coding. I'm pushing
27:18toward agentic engineering. This is the
27:20maximum capability. This is what you
27:21could do if you invested. That's what
27:23I'm interested in, not the floor, the
27:25ceiling. The short prompts tell us that
27:26if you made it a big prompt, a detailed
27:28prompt, the agents will do even better.
27:30Okay, so what do we have here? We have a
27:31couple nice surprises. We have Astra
27:33coming in first place once again with a
27:36killer 30K tokens out in only 29 steps
27:39and a low average cost. But then right
27:41below that, we have Gemini 3.8 Flash, my
27:43favorite workhorse model right now,
27:45right? In my model stack, I can throw on
27:46my favorites here and you can see all
27:48the models I'm actively using. So, of
27:50course, all the S-tier models I'm using
27:53pretty rapidly, pretty aggressively. But
27:54then, you know, right in that workhorse
27:56level where we're trying to push down
27:57our costs by literally an order of
27:59magnitude by pushing away from these
28:00insane state-of-the-art models, right?
28:02We can go down 10x in cost and with
28:04insane models like GLM 5.3 Flash, we can
28:06go down 20x in cost. We can get a lot of
28:10work done with the workhorse level
28:12models. Gemini 3.8 Flash, I'm having a
28:14really great time with this model. It's
28:16really nice to see that that far up
28:17because as you can see, you get paid to
28:20use this model. That's another way to
28:21look at cost savings. You get paid to
28:23pick the right model for the right tasks
28:25and you can learn what the right tasks
28:26are by picking the right benchmarks that
28:28best align with the work you're trying
28:30to accomplish. So, I hope that all makes
28:31sense, right? Um on the flip side, you
28:33can see here once again, we can see our
28:34workhorse models using a lot of alpha
28:36tokens, using a lot of steps. It takes
28:38longer for Gemini 3.8 Flash to get
28:40there, but it can get there. So, if you
28:42can sacrifice speed for performance and
28:45cost, pick Gemini 3.8 Flash for your
28:47long horizon deep software engineering
28:49tasks. Again, this is only information
28:51you'll find if you dig a little bit
28:53deeper, right? Two levels deep, three
28:54levels deep. Don't just look at the
28:56index, look at what you need to do and
28:58then pick the benchmarks that align most
28:59closely to what you're trying to
29:01accomplish. No real surprises after
29:02that. Fable coming in here. So, you can
29:04imagine, you know, Fable is going to be
29:06up here, maybe in the top spot, maybe
29:07right below. It doesn't really matter
29:09because we know it's going to be far
29:11more expensive than Astro. Couple of
29:13nice mentionables here. GLM 5.3 looking
29:16really good. Kimmy looking nice. And
29:18then it just kind of falls off from
29:19here. If you want to go really cheap,
29:21the best option is going to be your Luna
29:24and your of course GLM 5.3 flash. And
29:26then down here, if you're willing to
29:27really sack performance on software
29:29engineering or maybe do some sub-agent
29:31delegation with some really, really
29:33well-designed handoff prompts, maybe you
29:35can get value out of Deep Seek before
29:37flash. I personally wouldn't risk it
29:38though. So, Deep Sweep, longer horizon
29:40software engineering tasks, you can
29:41imagine what this is like. Public
29:43benchmarks sourced from GitHub issues
29:45and pull requests that carry more
29:46details, blah blah blah blah blah. So,
29:48again, feel free to dive into the
29:49details of all these.
29:54Now, what I want to do is up-level all
29:56this. Why did I pick these five
29:58benchmarks? What is the common pattern
30:00here? What am I working toward? Why
30:02didn't I pick a long context benchmark?
30:05Why didn't I pick more software
30:06engineering focused stuff, right? Why
30:08isn't the artificial analysis clump of
30:1110 different benchmarks enough? Let's
30:13break it down. I'm focused on one big
30:15idea in my agentic engineering. I'm
30:18working toward agentic systems that run
30:20autonomously with no oversight. So,
30:23what's the common theme between all the
30:25benchmarks I've mentioned here? We have
30:27Terminal Bench before, Apex Agents,
30:30Automation Bench, Amnesiance, and Deep
30:32Sweep. What picture am I painting here?
30:34What am I trying to accomplish? You
30:35know, viewers of the channel, this is
30:36not going to be a surprise for you at
30:38all. I am working toward since December,
30:40even before that. Really, it was like
30:41the midway point of 2025. Everything I
30:44started to do veered toward one bigger
30:46concept, Outloop agentic coding. Going
30:49even further, Outloop agentic
30:50engineering. So, we're building systems
30:52that build systems. We're building a
30:54team of agents plus code that operate
30:56better than they could alone. And the
30:58key kicker here is that they operate
31:00without you and I, or with much less
31:03oversight than most engineers are used
31:04to right now. Let's first off talk about
31:06a really simple idea for every benchmark
31:08I look at, which is variance in
31:10performance. Okay, so what do I mean by
31:11that? Here's a benchmark that I don't
31:13pay any attention to. It doesn't matter
31:15what this benchmark is saying. If I see
31:17this pattern, and you know, see if you
31:19can spot the pattern here. If I see this
31:20pattern, I walk away from the benchmark.
31:23AA long context retrieval V1.1 for
31:26measuring the agent's ability to pull
31:27information from a long context window.
31:30If I see this, I don't care.
31:32Right? This benchmark is dead to me.
31:33Why? This is what saturation looks like.
31:35It's a flat line. This is the tell. If
31:37you see a benchmark that has a flat
31:39line, just don't spend your time. Don't
31:41focus on this. The information isn't
31:43valuable. Now, again, if the work you're
31:45doing is constantly about maxing out a
31:47context window and getting that 3% on
31:50Kimi K3 to recall something in the 900k
31:53context. If that's an advantage for you,
31:54of course this benchmark matters. But
31:56for me, this is the signal of a
31:58saturated benchmark. It's a flat line.
32:00Everyone can do it. There's no value
32:02here. And so, you know, as you saw,
32:04there is a decent amount of variation in
32:06the other benchmarks that I showed here.
32:09Deep Swee, I'm fissients. And it's not
32:11always what you would expect. Obviously,
32:12we see our state-of-the-art models near
32:14the top, but sometimes we don't.
32:15Automation bench is a great example of
32:17that. That's one of the big things I
32:18look for is variance in performance.
32:20Where's the alpha? Where's the
32:22information gain? You know, why did I
32:23choose these five benchmarks? There are
32:25themes we've been building up on the
32:27channel week after week. Let's just like
32:28walk through some of these, right?
32:30Alignment, aka guardrail following,
32:32right? Not breaking the guardrails you
32:34set up. And guardrails is everything
32:35down to like instructions in your
32:38prompt. And what you want to do with
32:39those like cannot fail guardrails or
32:42warning-triggering guardrails is harness
32:44engineer them into your agent harness,
32:46right? You want those to be part of the
32:47law or build them into your AI developer
32:50workflow as a code step, right? A
32:52validation step, which then turns into
32:53your software factory. So, another big
32:55idea, as you know, is long-horizon
32:58tasks. That's what Deep Seek is all
32:59about. There are a couple other like the
33:01meter-long running benchmark is really
33:04important, but a lot of those aren't
33:05getting updated or they're only held
33:07privately or like But that's a big theme
33:09I'm looking for. It's long-horizon
33:11engineering work. I want to know that my
33:13agents can run off for a long time and
33:15accomplish a nice set of work. Now, the
33:17other kind of big theme you saw here
33:19throughout is performance and cost being
33:22the most important thing because the
33:24OpenAI engineers are doing a great job
33:26here. AGI you can't pay for is
33:28irrelevant. That is just a factual
33:30statement for almost every engineer. AGI
33:32you can't pay for doesn't matter at all.
33:34GPT-6 Astra is an incredible
33:36state-of-the-art model. It is, of
33:38course, very expensive still, but as you
33:40can see here, cost per task is a cliff
33:43compared to Opus and Fable, right? It's
33:45a cliff. They're doing a great job here.
33:46Obviously, we want these prices to come
33:48down. Obviously, they will. It'll take
33:49time. But, of course, then we have great
33:51models right behind them. We're talking
33:53about Kimmy, we're talking about GLM,
33:55Gemini doing a really, really great job
33:57here. One of the best models available
33:58right now, frankly. I'm looking for that
34:00performance and cost balance. Why?
34:02Again, long-running agentic engineering
34:05work that's happening without me. And
34:07this is going to be hard. Most engineers
34:09won't get here. Very, very important to
34:11have low deception, to not hallucinate
34:13because guess what one hallucination is
34:15going to cause in your long-running
34:17agent pipeline, your long-running
34:19software factory, your long-running
34:20chain of agents plus code. What happens
34:23in your AI developer workflows then?
34:24Every subsequent agent gets a messed up
34:26result because one of them hallucinated.
34:28So, omniscients is really important to
34:30pay attention to for long-running agent
34:32tasks. Once again, just calling out some
34:34models, really happy about Gemini 3.8
34:36Flash hitting that performance cost
34:38sweet spot. Grok not bad, they really
34:40kicked things up with the Grok 4.6
34:42series. But then we have things like, of
34:44course, Astra absolutely killing it. The
34:46nice part is all of these
34:47state-of-the-art models are basically
34:49the same here. So, if you use one of
34:51them, you'll get all of it for
34:52long-running tasks in a affordable way
34:54that's not going to break your bank as
34:55you're running and scaling up agents.
34:57Again, last week we talked about agent
34:58swarms. Paying for a GPT-6 Astra swarm
35:02>> [laughter]
35:03>> or Fable swarm is going to break the
35:05bank, but I've run many of the Gemini
35:073.8 Flash swarms or the GLM swarms, much
35:10more affordable. Again, I'll link last
35:12week's video, that one went absolutely
35:13viral for a good reason. Check that one
35:14out. I showcased my V1 agent swarm
35:17system and really dug into the
35:19implications of what OpenAI's Astra
35:22swarm hugging face incident really told
35:25us, right? Really unlocked for us. And
35:26the answer is, I'll spoil it for you a
35:28little bit, is swarms. Agent swarms are
35:30viable. They're real. I ran a swarm the
35:32other day, it did a ton of work for
35:34me in an unstructured way. Definitely
35:36check that one out. What else am I
35:37looking for, right? Low deception is
35:38key, but also, as you saw with
35:41automation bench and Apex agents, it's
35:44not good enough to just have your agents
35:46do software work. We know that that is
35:48one domain, and if you're building real
35:51products, you need to be able to operate
35:53across finance, HR, marketing,
35:54operations, sales, support, and then
35:56your specific domain, whatever you're
35:58actually in. So, that's where proxies
36:01for domain-specific work that's very
36:03challenging comes into play, right? Some
36:05of the greatest examples you can pull
36:07from is investment banking analysis,
36:09management consulting, and of course,
36:11legal research, client advice,
36:13regulatory, you know, litigation,
36:15mergers, acquisitions, compliance,
36:17high-stakes, high-reward domain-specific
36:20problems. If agents can perform well
36:22here, they'll probably be able to
36:24perform well in your domain. So, you
36:26want to look for proxies for that. And
36:27then, of course, raw engineering skills,
36:29right? Deep Swede or others. You could
36:30have pulled Cursor Bench. We looked at
36:32Terminal Bench as well. Any of these
36:34other proxies for software engineering
36:36also going to be valuable. You just
36:38don't want to fixate on software
36:39engineering. You know, the big theme is
36:41long horizon, no human in the loop,
36:44honest agents we can trust shipping on
36:46our behalf. That's what it takes to
36:48build a system that builds a system. AI
36:50agents, AI developer workflows, software
36:52factories, everything we discussed last
36:54week surrounding this new multi-agent
36:56paradigm of agent swarms, and eventually
36:58dark factories all require these ideas,
37:01and that's why I'm focused on these five
37:03benchmarks. So, of course, you know,
37:04these five benchmarks are also limited
37:06in nature. I'm looking at a ton of other
37:08ones. These are just the top five if I
37:10had to pick these five. Here are a
37:12couple other ideas I'm looking for in my
37:13benchmarks: delegation, small agent
37:15teams, agent handoffs, of course, bigger
37:17ideas like agent swarms. But, I want to
37:18know things like, can this model
37:20coordinate with a several other versions
37:21of itself or versions of itself plus a
37:24few other agents? Right, small agent
37:25teams, or I like to call SATs. I want to
37:27know if the agent can recover from
37:28failure. Can it work without me coming
37:31in and repairing the process? You know,
37:32that all kind of fits under the theme of
37:34self-healing, self-validation, and
37:37agent-to-agent communication, right? I
37:39would pay a ton for benchmarks like
37:40that. I have a whole list of benchmarks
37:42I wish I had. That's a video for another
37:44day. And these all are just big ideas
37:46that we keep tapping into week after
37:48week here on the channel. Next
37:49generation agentic engineering is about
37:51investing into systems that work on your
37:52behalf. So, check out last week's video
37:55on the simple swarm system as a
37:57practical continuation for this. You can
37:59see the exact models I picked in that
38:00video. As you now know, this video has
38:03helped you understand why I've chosen
38:05those models. Comment down below. Let me
38:06know what your favorite benchmarks are.
38:09Don't say the index. You know, give me a
38:11specific benchmark. I'm super curious
38:13what you pay attention to. Have you
38:15dialed your benchmarking process into
38:18more specific benchmarks? Fixating on
38:20one isn't the move, but fixating on a
38:22few that align with your domain is
38:24clearly where there is serious alpha in
38:26choosing a model stack, not a single
38:29individual model. As you can see, I've
38:31arranged my model stack in that order
38:33with the models that make the most sense
38:36for the work I'm doing. Always keep one
38:38eye on the index, but then always keep
38:40an eye on your own personal index for
38:42the best results. That's what I do. I
38:44recommend you do the same. It's not
38:45about performance, it's about
38:46performance, cost, [music] and speed.
38:48And the big takeaway is of course you
38:50want to make sure your benchmarks are
38:51aligned with the work you do, not some
38:53global [music] index. If you got value
38:55out of this video, you know what to do.
38:57Like, comment, subscribe. Huge shout out
38:59to everyone that's been along the ride.
39:01This is going to be a great [music] end
39:03of the year. We have so many big ideas
39:05to cover and discuss and share. So,
39:06thanks to every engineer that's been
39:08along with the journey for years now,
39:10and for every engineer that's going to
39:12come next. Let's blast past that 150k
39:14mark. You know where to find me every
39:16single Monday. Stay focused and keep
39:20building.

About this transcript

This page contains the full transcript of Agentic Engineering Benchmarks: How I RANK Astra, Fable 5.1, and Open-Weights by IndyDevDan, generated from the public captions YouTube serves with the video. The transcript has 7,941 words across 1,177 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.