YT YouTube2Text

YouTube2Text → Transcripts → Meta just caught up with Anthropic

Meta just caught up with Anthropic — Transcript

by Theo - t3․gg · 7,848 words · 1,098 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00It finally happened. A lab that isn't
0:02named OpenAI or Anthropic shipped a
0:04model that is in the same tier as Fable
0:06and Astra. I know this sounds insane,
0:08but it really did happen. Meta's back in
0:10a way I never expected to see. I know I
0:12was pretty hyped about Muse Spark 1.2. I
0:15didn't really cover 1.3, but it was a
0:16decent model. I did not expect Muse
0:19Wonder to blow me away the ways that it
0:21has. I have been using this model for a
0:22few days now and I really am impressed
0:25with it. And when you see the
0:26benchmarks, you'll understand. For
0:28example, here wait.
0:30That is a sinfully ugly vibe coded UI
0:33that Muse made when I was testing it.
0:35That That can't be right. Okay, let's go
0:38to the benchmark. Oh, that's a pile of
0:40horrible things people have been saying
0:42about me because I like the model. Theo
0:44got super paid off, zero credibility. Uh
0:47Kind of difficult to believe that Meta
0:49didn't pay you to say that. Why is this
0:51guy tweeting like Meta paid him? How
0:53much did they pay you? From the the Tibo
0:55not not sure who that is, but uh the
0:57real numbers are looking pretty good.
0:59Not only has Muse Wonder in Wonderful
1:01been a huge jump over where Spark 1.3
1:03was, the Wonderful Max run appears to
1:06have beaten all OpenAI models on
1:09artificial analysis, putting them in
1:11third place overall. Which is pretty
1:13nuts. So, what's up with this model? Has
1:15it really dethroned OpenAI for the
1:17second place lab? Is it really as good
1:19as these benches seem to indicate? Am I
1:21full of crap? Am I just paid off to say
1:23these things? Well, I'll let you guys
1:25know. Meta has not paid me a cent. I am
1:27lucky to have early tested this, but the
1:29only people paying me for this video are
1:30today's sponsor. You probably noticed
1:32the recent surge of personal assistant
1:33type tools. Things like the Meta Muse
1:35product Rockbot, Dots from OpenAI, and
1:38more. They all have something in common.
1:40They're not relying on the knowledge the
1:41model already has. If the model could
1:43only respond with what it already knew,
1:45these tools would be useless. They need
1:47detailed up-to-date context on the
1:48things that they are working on and
1:50thinking about. The way they all do that
1:51is the same. They do it through the web.
1:53If they did it through today's sponsor
1:54Browserbase, they'd probably be much
1:55better. I can't even begin to tell you
1:57how many issues I've run into with tools
1:58like Rockbot. Where sure they can
2:00navigate the web and find things, but as
2:02soon as you need them to sign into any
2:03of your accounts, you're kind of
2:04screwed. And you often end up having to
2:06do crazy things like spinning up a full
2:08virtual machine or giving them a
2:09dedicated Mac mini.
2:11Browser-based is far ahead with all of
2:12this to the point where I honestly hope
2:14that XAI or OpenAI buys them because
2:16their credential system is great and
2:18integrates directly with one password
2:20allowing your agents to sign into your
2:22things in their browser. And if you
2:24don't need proper navigation, you just
2:26want to find some things on the web, the
2:27search API is great. Trivial for both
2:29you and your agents to get the data they
2:31need from the web. And if they want to
2:33get more detail from a web page, the
2:34fetch API is here and ready to go, too.
2:36It lets you put in a URL and get back
2:38HTML, JSON, or markdown, whatever your
2:40agent prefers. Considering the fact that
2:4285% of the APIs on the web can't be
2:45accessed through traditional curls, this
2:47is essential to making your agents more
2:49capable. Give your agents the whole web
2:51at soda.link/browserbase.
2:53Welcome back. I am going to do something
2:56a bit different here and start with the
2:57negative because I just want to get it
3:00out of the way. The first thing I hate
3:02about this model release is that it's
3:03not one model, it's two. There is Muse
3:06Wonder and Muse Wonder Full. Yes, Wonder
3:09Full like a pun, but it's Wonder space
3:12full. Normally, this wouldn't be too big
3:14a deal, just cringe naming, but the gap
3:16between these models is insane. Muse
3:19Wonder runs at around 120 TPS, the full
3:22version runs at around 50. So, it's
3:24clearly a different, bigger model. This
3:27also wouldn't be that cringe of a thing
3:28outside of the naming except for the
3:30fact that only Muse Wonder is available
3:33over API and has a price. The Wonder
3:35Full version is only available through a
3:38Muse code subscription. This makes it
3:40really hard to measure, in particular,
3:42to do price versus performance because
3:44there is no way to measure the full
3:46version of the model in terms of price
3:48of performance because they never gave
3:50us a price for it. Another kind of
3:51cringe thing is they didn't put pricing
3:53info anywhere in their announcement
3:55post. They did show off how good the
3:57numbers are for the full max runs they
4:00did, showing that it beat out Fable,
4:02Astra, and Opus 5.5 in a large number of
4:05benches. Memory seems to be one of the
4:07main focuses in this model. They were
4:09talking about how they trained it to
4:11know how memory works during post
4:13training to really get these ideas baked
4:15into the model, which is cool and novel
4:17and interesting. I am curious what this
4:19ends up looking like long-term. The
4:21terminal bench scores are nuts,
4:22absolutely destroying Fable 5.1's
4:25numbers, and getting close to Astra and
4:27Opus 5.5's. They got a state of the art
4:29in deep sweep and sweep marathon with
4:31the full version, but again, we don't
4:34know the price. They just didn't put out
4:36the number. Apparently, they're
4:37massively constrained in compute, but it
4:38would be nice for them to at least say
4:40what it would cost so that we have those
4:41numbers to reference. It is what it is.
4:44We do have pricing for the regular
4:45Mixtral 8x7B model, thankfully. It's $4
4:47per million, 20 per mill out, and 20
4:50cent cash reads. That puts it at a
4:51little cheaper than Opus, which kind of
4:54sucks because this isn't the full
4:56version that they showed off in a lot of
4:58the benchmarks, and the one that I am
5:00particularly enjoying. This is the much
5:03smaller version. This would indicate
5:05that Mixtral full version is going to be
5:07priced way higher, possibly higher than
5:09Opus 5.5. That is not a thing that I
5:11think would be worth paying for at all.
5:14We have more negatives to cover, and I
5:15promise this will be worth it because
5:17once I get all of these out of the way,
5:18I can go into glaze mode and explain all
5:20of the things I love about this model
5:21because I have been really genuinely
5:23enjoying it. The other problem is this
5:25model's 3D capabilities. I'll be frank,
5:28they're not really good. You can see
5:31from this tractor demo comparing Opus
5:335.5 to Mixtral full
5:35that it's not very good.
5:38I will also skip to the thing I normally
5:40save until the end, which is the fish
5:42slop run, because it is also not very
5:44good.
5:45Is it as bad as Grok 4.7?
5:48No, not at all. In fact, the movement
5:50and core mechanics actually feel decent.
5:53It also has animations that uh
5:55I've definitely seen before. A lot of
5:58models seem to have gotten a similar
5:59data set and harness for doing the 3D
6:02training.
6:03If they did get that harness, they did
6:06some of the worst work in using it that
6:08I've seen because this model is really
6:10bad, ironically enough, at models. The
6:12things it made in Blender are absolutely
6:15garbage. The way it handled the like
6:17coloring and lighting on the sides is
6:18bad, but at least like the game
6:20functions and the movement feels good,
6:22which uh Grok did not come close to.
6:25That all said, this is better than the
6:27like Haiku version, but both Sonnet and
6:30Opus destroyed it. Fable obviously
6:32destroyed it, but Fable's 3D capability
6:34isn't quite as high. For what it's
6:35worth, Meta even called this out. They
6:37have a section about limitations of the
6:39model, and they specifically call out
6:40that visual and spatial reasoning,
6:41especially 3D tasks, remain a known
6:44limitation. I posted about this on
6:45Twitter already, and people were upset,
6:47so I'll double down on it here. The
6:49smartest coding model in the world would
6:50not be good at 3D stuff unless it was
6:52trained specifically for it.
6:54I really want to emphasize this point.
6:56The recent jump we've seen in the 3D
6:57capabilities in models is not because
6:59they suddenly got so good that they can
7:01do 3D. Things like 3D, things like
7:03computer use, a lot of these
7:04capabilities that we really like and are
7:06excited by aren't things that emerge
7:08from the models getting smarter. They
7:10are specifically trained into the models
7:12in post-training using environments that
7:14were created for these particular tasks.
7:17It appears as though there is a good 3D
7:19training environment that some external
7:21company made that they've been selling
7:22to the labs. I noticed this initially
7:24with Kimi K3 when I commented on that
7:25model being surprisingly good at 3D in
7:27Blender. Since then, the models from the
7:29frontier labs have had a similar jump.
7:31The best place to see this, though, is
7:33with Fable 5.1 to Opus 5.5. Obviously,
7:36Opus 5.5 is better for a lot of stuff
7:38and way, way cheaper, and most
7:40importantly here, newer. Fable 5.1 is
7:43smarter sometimes, slightly dumber other
7:45times, way more expensive, but nobody
7:47would disagree that Fable 5.1 is a
7:49bigger and potentially more capable
7:51model, which is why, despite the like 5%
7:54plus or minus gap between those two
7:55models, the massive jump for Opus 5.5's
7:593D capability is so impressive, but it
8:01also showcases that the smartness and
8:03intelligence of the model does not map
8:05directly to its 3D capabilities. What
8:07I'm trying to say is, if a model is bad
8:08overall, but good at 3D, that's kind of
8:10impressive. If a model is good overall
8:12and bad at 3D, that's still impressive.
8:14But if a model is bad overall and bad at
8:163D,
8:17Groq 4.7. Sorry, I got a cough there.
8:21Then that model probably sucks. So, call
8:23me contradictory all you want here. I
8:25don't think this model should be used
8:27for 3D anything, and if 3D use cases are
8:30something that you care about, go
8:31subscribe to your Claude plan and don't
8:33complain. It's an unbelievable value for
8:35the money. And if you're here trying to
8:37find the absolute best way to spend your
8:39dollars on an AI right now, it's not
8:40going to be local GPUs, it's not going
8:42to be some weird budget plan from some
8:44Chinese lab, and it's certainly not
8:45going to be paying for your tokens on
8:47OpenRouter. The $200 Claude Go plan is
8:49unbelievably good of a value, and I
8:52guess I should address the final
8:53question, which is how much usage can
8:55you get on the $50 and $200 Muse Go
8:57plans? I don't know. I haven't
8:59subscribed for either. I'm still using
9:00the API key that they sent me. That's a
9:02thing I hope to try in the future, but I
9:04do not have time, and I wanted to get a
9:05video out to jump on top of a whole of
9:07the things I've seen people saying on
9:08the about this model,
9:10because a lot of it's kind of dumb, and
9:11I want to show why I like it so much.
9:14And now I think it's time for the
9:15positivity to begin a bit. This is a
9:17silly one to call out, but I benched it
9:19against SkateBench, which I haven't run
9:21in my videos for a bit, because usually
9:22it shows weird things. For example, Opus
9:255 Max got an 84% and 5.5 Max got a 79%.
9:29So, RL tends to nerf models on this,
9:32I've noticed. It's a weird bench, but
9:34it's fun to see how models behave on it.
9:36The only model line to ever break 90%
9:38has been Gemini, and Gemini 3.1 Pro got
9:40a 96 as well as 3.8 Flash High. Nothing
9:43else has come close to those numbers,
9:45except for now, Muse Wonder Full,
9:47getting a 95%.
9:49This might be concerning if you know how
9:51I feel about Gemini models, but you hear
9:53me out. Gemini models have this
9:55unbelievable capability to understand
9:57complex grammars, 3D reasoning, not in
10:00the like making a 3D game sense, but in
10:03the understanding 3D transformations in
10:05its like brain, so to speak. It's also
10:08good at labeling things in vision, like
10:11you give it a photo, say label these
10:12things, and it can pick the right pixels
10:14and do that type of task. For some
10:16reason, all of this stuff combined turns
10:18out to make the Gemini models good at
10:19naming skateboard tricks.
10:21Muse seems to have some of that same
10:23weirdness baked in, and this got me
10:26scared if I'm not lying. Like I was
10:28fearful that if they had done whatever
10:31Gemini did to have this capability, then
10:33it might have also nerfed the model
10:34other places. So, let's take a look at
10:36some of those other places.
10:38Okay, I I lied. One more negative thing
10:40before we dive in on all of this. This
10:42model sucks at UI. It isn't just bad, it
10:45is atrocious.
10:47It's not making UIs that are like wrong
10:49the same way that Grok does, but it's
10:51making UIs that are slop the same way
10:54like Sonnet 3.7 did. So, if you want a
10:57model that is an all-rounder that can do
10:59anything from debugging kernels to
11:01making beautiful UIs to building games,
11:04go back to Claude. That's where you'll
11:06find that. You're not going to find that
11:07here.
11:08So, what the hell do I like about this
11:10model if I just spent the last 10
11:11minutes railing on it?
11:13Ah, let me show you.
11:15I'm going to start in a kind of weird
11:17place. I've been playing a game called
11:19Reskate lately. It's an open-source
11:22wrapper for the version of EA Skate on
11:24Steam right now that is peer-to-peer and
11:26allows you to mod the game and play old
11:28maps and things. I've been enjoying it a
11:29lot. But, the store for it, specifically
11:31Thunderstore, I noticed is very very
11:34slow to index. It's open source and I
11:37thought it'd be interesting to throw
11:38this new model at the task of auditing
11:40it to find what might be causing the
11:43slowness.
11:44So what was I impressed with? Is it how
11:46thoughtful the response was? Did it make
11:48a really good set of changes? Did it
11:50file a PR? Like no, none of that. Let's
11:53just read what it said. The audit is
11:55done. The full report is in file. Top
11:58findings, the CDN setup is fine. The
12:00slowness comes from data that does not
12:02cache well. Rescale itself is small. Its
12:05index is only 0.48 megabytes and it
12:08loads in 0.78 seconds on a warm edge.
12:10The problems hit on cold paths and at
12:13scale. I don't know if you guys noticed
12:15what I've noticed here. I don't hate
12:17reading this. In fact, I almost enjoy
12:20it. To be clear, I'm not saying this
12:23isn't slop. I'm saying it's a different
12:25flavor of slop. It's a flavor I haven't
12:28tasted yet and it doesn't taste like yet
12:30another Claude distillation, which is
12:32rare and very nice, especially when you
12:34consider the absurd volume of Claude
12:37data that Anthropic's been collecting
12:38with their absurd bill with Anthropic. I
12:41expected whole models from Meta going
12:43forward to just talk and act like Claude
12:46because they had so much data from
12:48Claude. This doesn't. I have also seen
12:51from others very different experiences
12:53talking with this model. So I did a real
12:56brief test where I removed some of my
12:57call outs in my agent MD around I have
13:00ADHD, make your things short and easy to
13:03read, vary sentence length a bit, and
13:06use simplified technical English where
13:07it makes sense to. And it does get a
13:10good bit more verbose. I do think the
13:11defaults with this model aren't
13:13necessarily the most pleasant to
13:14interact with, but when I spun it up on
13:17my machine, didn't change anything, I
13:18just used my normal agent MD, ooh, it
13:21was my favorite model to work with
13:23immediately just from how it talked. But
13:26am I taking a huge hit in the
13:28intelligence when I do that?
13:30Depends on the task, but generally
13:32speaking, no.
13:34One of my fun new side projects is named
13:36Lust. It is a subset of Rust that should
13:38compile 100 times faster. Since I was
13:41able to rewrite the TypeScript compiler
13:42in Rust, I figured, why not try to write
13:44my own compiler for my own language and
13:46then recompile TS Rust in that instead.
13:49Trying my hardest to find things that
13:51the models can't do, you know?
13:53This thread's been going for a while. It
13:55has spun up over 115
13:58agents throughout its runs in order to
14:01split up the work. I asked it at one
14:03point to start spinning up every task on
14:06Muse and on Opus, and then set up a
14:09judge panel to review the changes from
14:11both so that I could have them to
14:12compare. And here, I asked for an update
14:14on this. Give me an update on how Muse
14:16versus Opus is going for you here. Short
14:18version, Opus does the more correct and
14:20deeper work. Muse does more work and
14:22faster. Muse is also a surprisingly good
14:25reviewer.
14:26All three blind judges picked Opus. Mean
14:28score is a pretty brutal difference
14:30here, 41 versus 30 out of 50. But, the
14:32Opus build is about 2.5 times faster.
14:35Its changes made no behavior Oh, sorry,
14:37this is just for one specific task. Uh
14:40the Opus version was faster. The Muse
14:42version had a bug. Pretty rough.
14:45So, overall, let's see how it feels.
14:47Muse in daily work is producing most of
14:49the landings and some of the results are
14:50excellent. It fixed a bunch of things
14:52that were causing accepts to pass when
14:55they shouldn't have. It made a bunch of
14:57changes to the driver layer. Uh fixed
14:59name bugs. As a reviewer, it's thorough
15:01and correct. It wrote 209 probes against
15:03the Opus fixes and gave clear fixed
15:05groups. As a judge, it was fair. It
15:07would rank its own entry second. I did
15:09have multiple times where I had both
15:10Opus and Muse report on something, dig
15:14into some code, give me a path forward
15:16to fix some issue, and was surprised how
15:19often Opus preferred Muse's plan. To be
15:21fair, in a lot of those scenarios, Muse
15:23preferred Opus's plan, but when I set
15:25them up to talk to each other, which uh
15:26not to self-plug, is a really cool
15:28feature in T3 Code now, when they would
15:30do their work separately and then
15:31interface with each other, they would
15:33often end up agreeing to use Muse's plan
15:35as the base with some feedback from
15:36Opus, which really surprised me. After a
15:39lot of back and forth, Opus, which is
15:41running this thread, concluded that Muse
15:43would make sense for most of the work,
15:45fixers, standard library work, fuzzing,
15:47and reviews. Opus does the hardcore
15:50pieces where wrong design is costly,
15:52like this T2 speed merge, the trait
15:54system fixes. And each of them reviews
15:56the others' work, and every review gets
15:58turned into fixes. Opus rated Muse at a
16:00B+ and rated itself at an A- saying that
16:04it is higher quality at a lower speed
16:07and higher cost. That said, we don't
16:09actually know what the cost is with Muse
16:11yet, so take that as you will. Let's
16:13talk a bit about how this model
16:15delegates and talks to other models,
16:17because it's been very, very impressive
16:18to me. I just did a quick test here
16:20where I told it to spin up three
16:21delegates to explore and analyze various
16:23parts of the T3 Code code base using the
16:25current slug for the alpha model and
16:27Haiku 55. And now we can hop in and see
16:30how it prompts itself. Act as the
16:32research sub-agent for this task.
16:33Explore and analyze the T3 Code
16:35contracts and provider and agent
16:36integration layer. Scope, directly
16:39describe the scope and how providers and
16:41MCP or agents connect to the server.
16:42Requirements, read actual implementation
16:44bodies for every claim you site. Search
16:46results only locate candidates. Do not
16:48cite a file you did not open. Map, yada
16:50yada yada. Do not edit any files as
16:51announced only. This is a really good
16:55thing for a model to send to a
16:56sub-agent, which makes this model really
16:59good at orchestration both of itself and
17:01of other models. I have found it to be a
17:04pretty good commander. The model that is
17:06passing messages to and from other
17:08things handles steering pretty well. It
17:10is generally just good at operating with
17:13other models. I've been surprised about
17:16this because this is one of the things
17:17that labs that aren't named in Tropicana
17:19Open AI tend to struggle with. Not the
17:21case here at all. This model is quite
17:24good, actually, at getting other models
17:26to do things for it. It also handles a
17:28very large numbers of orchestrated
17:30things going at once, which is also,
17:32again, a rare skill. All of this is why
17:34I think this model really does feel
17:36frontier-ish. It has its quirks. It's
17:38not quite as capable in certain places,
17:40which I promise we'll get to a bit more
17:41of, but it is the first model not from
17:44one of those labs that I don't question
17:46the ability to run for long tasks on.
17:48It's not It's the first model from
17:50another lab that I see orchestrating
17:52sub-agents and workflows this well. What
17:55I'm trying to say is this model is a
17:57third flavor of frontier, which has its
18:00benefits, has its negatives, will have a
18:02cost that probably won't be great, but
18:03it's different, and I'm really enjoying
18:06the different aspect of it. This is
18:08where you should account for my bias. As
18:09somebody who does a lot of reviewing of
18:12models, something being different is
18:14inherently just going to be nice to me.
18:17I can't help but think back to my
18:18history in the audio world, where
18:20somebody who reviews a lot of speakers
18:22might like a new speaker more than they
18:24should, simply cuz it's different. Maybe
18:26even cuz it's wrong in some ways,
18:28because they're so used to all the other
18:29speakers trying to be the same, but
18:31slightly louder or slightly marginally
18:33better in certain areas, that a speaker
18:35that just sounds entirely different is
18:36exciting to them. And I know that is
18:38part of why I'm enjoying this model so
18:40much, cuz it just feels and talks and
18:42acts a little bit different without
18:44being stupid. And that's the other thing
18:47I really like about this model. The
18:48stupid, well, I guess the lack of. This
18:51model has not done any egregious dumb
18:53It's kind of strange. I have had
18:57fewer dumb experiences with this model
19:00in the four to five days I've been
19:01pushing its limits than I have in the 30
19:04minutes I spent prepping for this video
19:06with Saul and Astra. In one given
19:08thread, I have more weird with
19:10Astra than I've had with this model
19:13total. Not exaggerating. It just doesn't
19:15do stupid I would go as far as to
19:17say it's slightly less dumb than even
19:20Opus is. And if I was to rank dumbness,
19:23I would say Astra's 10 out of 10. It is
19:25so dumb. That doesn't mean it's not
19:26smart. I consider smart and dumb
19:28different traits. You can have both.
19:29Astra is really smart and really dumb.
19:31Opus is even more smart and way less
19:34dumb. I would say Opus is like three out
19:36of 10 dumb and 10 out of 10 smart feels
19:38right now. This model takes a hit on the
19:40smart. The ceiling's a bit lower. It's
19:41like an eight or a nine out of 10. But
19:43the dumb is even lower somehow. It's
19:45like a one or two out of 10. It just
19:47doesn't do dumb It does ugly
19:49Like it sucks at front end. It sucks at
19:513D. But it doesn't do dumb things. I
19:53would not expect this model to ever do
19:55the silly mistakes like deleting a
19:57folder or directory that it's not
19:59supposed to, or putting itself in a
20:01broken state like I've seen every other
20:03model do at some point. It just behaves
20:05well. I'm going to give a silly example
20:08that I think will really showcase why
20:09I've been enjoying this model so much. I
20:11recently added a feature in T3 Code
20:13where you can remotely update all of
20:14your machines and all of their providers
20:16with one click. Technically two clicks.
20:18You have to hit the button on the bottom
20:19and then hit confirm. And I noticed that
20:20it wasn't updating the T3 Code runtime
20:23itself on all the remote machines. The
20:25cause of this ended up being a pretty
20:26weird niche bug. I'll show you from my
20:29run with Opus. First off, I noticed it
20:31being confused because it was checking
20:33the history and saw that the instance
20:36was updated. So I responded when it was
20:38running with a quick steer saying I
20:40manually ran the update after so that it
20:43knew the update had been run. That's why
20:45it was seeing the update. It wasn't like
20:46it was late. It had run just by hand
20:49after. It didn't run the way it was
20:51supposed to.
20:52And from here it did things correctly. I
20:54found the bug and fixed it. The fix is
20:56uncommitted in the work tree. Cause yada
20:58yada yada with the log show. Too much
21:00text. This particular paragraph is just
21:03too much in my opinion. So what do the
21:05logs show? It showed all the things that
21:07happened. It showed the fix. And then it
21:10showed the checks it did. The checks
21:12still failed silently. If NPM's
21:14unreachable, yada yada yada. Should I
21:16open a PR for this change? I said full
21:18stop to pull in my skill for this. It
21:20had Soul review it. It put up the PR.
21:23All was looking pretty good. Let's
21:24compare that to how Soul handled the
21:26task. I sent the same exact prompt here.
21:29It ran way too much Said,
21:31"Settings in the sidebar use different
21:32rules to select remote updates."
21:35It did not seem to notice that we had
21:36just changed the thing. I even said in
21:38the prompt, "The new update everything
21:40flow does not update remote
21:41environments." And it didn't seem to
21:43notice the the new part there. And this
21:45is part of what I'm not liking with
21:46opening AI models. It almost feels like
21:47it drops important words in its own
21:49brain sometimes and lost track of the
21:51fact that the new change is what matters
21:53here. I then hit an error because their
21:55async questions tool did not work
21:57properly with my proxy. Partially my
21:59fault, partially their fault in my
22:01opinion, especially cuz I had Soul make
22:03these changes to the proxy to support
22:04this at all. Regardless, I told it to
22:06continue to get out of the broken state.
22:08It told me it fixed it locally. The
22:10sidebar excludes remote servers while
22:12its NPM check was pending or failed.
22:14Removed the check. That's not the
22:16solution here. The solution here is
22:19making sure you know which version of
22:21the app to update to the remote to. What
22:24this would have done is updated your
22:25local app, updated the Muse, cursor,
22:29Claude, and Codexes of the remote
22:30machines, and then made you, after
22:32updating your app, run it again to
22:34update all the remote servers. It would
22:35just push the current version of your
22:37app to those servers. Not good.
22:39Actually, quite bad. So, I asked it,
22:41"How does this compare to the other PRs
22:42fixing it?" I asked it what happens if
22:44the host is behind, as well. And then it
22:45just kind of spit Okay, that's
22:48a lot about how Opus handled this and
22:49how Soul handled this. How did Muse
22:52handle it? Top finding, the flow is
22:54broken by one failed web request. A fix
22:56is already open and ready. I made no
22:58code changes in this work tree. For the
23:00sidebar flow updated a remote server, it
23:02adds npm, is this version published?
23:04That check is in here. The check uses
23:06effects HTTP client. It adds two trace
23:08headers to the request. I ran the
23:09current code with a test fetch and saw
23:11both headers in the request. Do you see
23:13how much more readable this is? Do you
23:15understand? But that's not the best
23:17part. Let's scroll a little bit. Fix
23:20status. Two open PRs fix this with the
23:23same one-line change. Turn off trace
23:25headers for this one request. There are
23:27two PRs here. Are those the ones from
23:30Opus and Soul? No. The first one is from
23:34somebody else. The second one is the
23:37Opus one. So, not only was this model's
23:39response much much much more easy to
23:42understand, it also found a much
23:45smaller, cleaner PR somebody else had
23:47already filed that fixes the problem.
23:50None of the other models found a PR that
23:52fixed it. None of the other models had a
23:54readable output on what the problem was,
23:57and none of the other models suggested
23:59just merging this other thing that
24:01already fixes it. I hope you can agree
24:04with me that without question, this is
24:06the best response of the three. I don't
24:09think any benchmark's going to measure
24:10this properly. I'm sorry, it's pretty
24:12hard to measure this type of thing. God
24:14damn, this model is much more pleasant
24:17to work with for a lot of real world
24:19stuff, and I will be throwing it a lot
24:21of bug reports going forward just in
24:23case I want to understand what's wrong,
24:25and maybe somebody else already has a
24:27PR. That all said, I have to complain a
24:29bit more. I set up some benches, benches
24:33are very very loose term here, where I
24:35had different models go through a lot of
24:38my giant code bases and audit them for
24:41things that are good, bad, ugly, and
24:42help me prioritize what we should work
24:44on next. Here is an example where I had
24:46the models all go through T3 code's
24:48architecture and make comments on it.
24:50Opus smoked here. Opus 5.5 had an insane
24:54score when I ran it through all of this.
24:57Sonnet 5.5 did surprisingly good as
24:59well. Then we have Soul and Astra, and
25:01then Muse right under. This is
25:02interesting cuz Muse and Astra ended up
25:04being relatively close in their scores.
25:07But then the normal Muse Wonder, the
25:09non-full version, dropped to almost half
25:13of Opus's score. Yeah, that is a over
25:1630% drop in the quality of the findings
25:20and analysis that normal Muse Wonder
25:22could do versus Muse Wonder full. I
25:24don't think Muse Wonder makes any sense
25:26at all for the price. It does not seem
25:28particularly compelling, and I think a
25:30lot of the demos people have been
25:31showing online are using Muse Wonder cuz
25:33they're taking the API, they're throwing
25:34it into whatever tools they normally
25:36use, and then they see bad performance.
25:37I get it. Muse Wonder appears to be a
25:40bad model. Muse Wonder full is neck and
25:43neck with Astra and Fable in lots of
25:45real-world stuff, and is really pleasant
25:47to work with. I ran a similar audit on
25:50Lakebed, my cloud product. This one says
25:536.1 Soul was the best analysis by far
25:56with a nine out of 10 across all the
25:58judges, and Muse Wonder fell quite a bit
26:00here, getting a 5.75.
26:03Opus 5.5, GPT-6 Astra, and Sonnet were
26:06all relatively close in scores. I would
26:08be a bit sus here cuz Sonnet 5.5 scoring
26:10higher than Opus suggests this might not
26:12be the best bench. But again, I whipped
26:15these together just to see. We are going
26:17to see another fun trend with Muse
26:19Wonder full. The amount of time it takes
26:21and the amount of tokens it takes to get
26:23its results. It took 16 minutes to get
26:26this result, which puts it at nearly
26:27half the time for 6.1 Soul, and way less
26:31than half the time for Opus. Sonnet took
26:3380 minutes to run this one. Not great. A
26:36similar run I did with TS Rust, which is
26:38quite the code base, saw similar gaps in
26:41the time to complete. Muse Wonder full
26:44scored nearly identically to Fable 5.1.
26:47It took 17 minutes and got not a great
26:51score compared to these other things,
26:52but considering that it took 10 plus
26:54minutes less than some of these other
26:56scores, and it took comically less than
26:58Sonnet and Opus, nearly half the amount
27:00of time. Then Muse Wonder, again, not a
27:02great score. 5 minutes and 9 seconds to
27:05complete this, though. What I'm trying
27:06to say is these models are fast and
27:07relatively token efficient. You can see
27:09this on Slop Politics. Again, we don't
27:11have cost for Muse Wonder Full, so we
27:12can't do cost versus intelligence here.
27:15All we can do is token output. And we
27:17can see it uses fewer tokens than Opus
27:195.5 on X High in order to get nearly
27:22identical scores. This means if we
27:24exclude Max from this, which sadly does
27:27kick out the Muse Wonder models because
27:28they're tagged as Max, even though
27:29they're technically X High, if I recall,
27:31it is the best scoring model if you
27:33exclude the Max models, other than of
27:35course 5.5 X High being just barely
27:37above. But again, that means this model
27:40is scoring absurdly well. So, if you go
27:42over to the bar chart, you can see here
27:44that Gemini 4 Argon uses slightly more
27:46tokens, but everything other than OpenAI
27:50models that matter right now are using
27:52quite a bit more tokens, including Opus
27:54and Fable. And of course, if I turn on
27:57Sonnet, all-time high token usage.
28:00Impressive. So, even beating out Grok in
28:02token efficiency. I would not have
28:03expected this level of token efficiency
28:05with this release. Also want to call out
28:08that both Wonder and Wonder Full had
28:10very similar token usage. 53.8K per task
28:13for Wonder Full, and 54.2K
28:16tokens per task for the standard Muse
28:18Wonder. Weird that these models behave
28:20so similarly despite Muse Wonder being
28:22so much dumber than the full version.
28:24God, I I hate these names so much.
28:27Context on what I did here to make this
28:28all make a little more sense. I had Opus
28:31go find eight random PRs of varying
28:34complexity that hadn't merged yet in T3
28:36Code, and then have all three of these
28:38models being Soul 6.1, Wonder Full, and
28:42Opus 5.5 review all of them. Compare
28:45Here are findings, verify the findings,
28:47and see which models had the most novel
28:49and accurate results. Here are those
28:51results. Opus 5.5 found 23 things across
28:55those PRs, 21 of which were valid. One
28:58was possible and one was wrong. Eight of
29:01them were nits. GPT-61 Soul only found
29:04nine things. Massive gap from what Opus
29:06found and what Soul found there. Seven
29:09of them were valid. It had two invalid
29:11findings. So, its novel score here is
29:13four, two valid and plausible findings.
29:17Then we have Muse, 16 findings.
29:19Smack-dab in the middle between Opus and
29:22Soul. Of those, it found five novel
29:24things. So, yeah, of course, Opus is
29:26still finding the most novel stuff, but
29:28it's also throwing a shitload of nits
29:30into its findings. Soul is not throwing
29:33a bunch of nits in, but it's also not
29:34finding anywhere near as much stuff. So,
29:36if Soul is your only reviewer, you
29:38should pull other things in. Muse was a
29:40weirdly good middle ground here. This
29:42does also call out that Muse didn't find
29:44as many high severity findings as Soul
29:47and Opus did, but I thought this was
29:49very useful to see.
29:51Summary here is that Opus found the most
29:52problems no one else found, including
29:54four medium severity issues. It writes
29:55about one nit per PR, and in one PR it
29:58flagged intended behavior as a bug. Oof,
30:00yeah, that this is very Opus-y. 61 Soul
30:03never wrote a nit, so every finding
30:04mattered. It found nothing on three of
30:06the four small PRs, even where real nits
30:08existed. Its unique findings came from
30:10going deep, like reading the upstream
30:12agent device source in the hyperdrive
30:14docs. Muse had the most overlap with
30:16finding bugs the others found, so if
30:18you're going to use just one, it seems
30:19to be good there. It found a busy reopen
30:21token revoke that Opus missed. Most of
30:23its unique findings are low severity,
30:25and once it asked for changes over a
30:26gap, the PR description it already
30:28listed. Not great, but also
30:31quite solid overall. This comparison
30:33might be my favorite of the bunch. I
30:35asked both Opus and Muse waterfall to do
30:38an incredibly, quote, deep dive on the
30:40T3 code code base, as well as the
30:42feedback that we get through channels
30:44like GitHub issues, and PRs,
30:45discussions, etc. I want your help
30:47prioritizing what we should be working
30:48on next. It spun up a ton of agents,
30:51wrote a bunch of stuff, and okay, I sent
30:54that at 10:33, and I got my response
30:57back at 11:04. So, 30 minutes later, it
31:00wrote this big old report. T3 code, what
31:03should we work on next? 10 agents did
31:04the audit, nine read every open issue,
31:07every PR, and every discussion, plus the
31:08code base. A 10th like I'm sorry, I'm
31:11doing my best, but this is just so much
31:13text for things that don't actually
31:14matter.
31:16The short version has four sentences per
31:18bullet point.
31:20Says to ship V2 to stable safely and
31:21soon. Bottleneck is merging, not writing
31:24code about half October, yada yada yada.
31:27Users want reliability, not features.
31:29The biggest pain is run stuck on
31:30running. Server resource killing, yada
31:32yada yada. Week one, merge and small
31:35fixes. It does an okay job categorizing
31:38all these things, making rows, but it's
31:40just so much text. Then there's the
31:42section for ship gate for V2 stable.
31:46This is not the most readable thing. It
31:49wrote a lot of text. And now, let's
31:51compare to what we got out of Muse.
31:54I I sent this at 9:19. I noticed that it
31:57got like frozen. I think it's going to
31:59like do a reset around the same time. I
32:01actually don't know how long it took
32:03overall cuz of all of that, sadly, so I
32:04can't get you realistic timestamps. I
32:06sent that at 10:24, and then it finished
32:08at 10:31. So, if we combine the 7
32:11minutes plus that, it ended up being
32:14about 17-ish minutes. About half the
32:16time, and that lines up for my
32:17experience overall. It's quite fast and
32:19efficient, and doesn't burn as many
32:21tokens, or check as many dumb places
32:23before it goes the right place. The best
32:25way I can describe this is it has good
32:26intuition. It knows where to look for
32:28things, and it finds the things it's
32:29looking for relatively fast. I don't
32:31know how the hell they did this. I don't
32:32know if they just trained them in the
32:33giant meta code base and got a lot of
32:35good data, but it's really good at
32:37knowing where to check and checking the
32:38right places first. It does have a silly
32:40call at the start saying, "This is a
32:42plan, not a special mode. I haven't
32:44started implementation." So, it seems
32:46like it was trained on this idea of plan
32:47mode, and it calls us out. I've seen
32:48this a few times when I ask it for
32:49plans. But, let's read the actual info
32:52here.
32:52Goal: Pick the next work for T3 code
32:55from real evidence. 10 research agents
32:56audited GitHub issues, PRs, discussions,
32:58and the code. One critic tried to
32:59falsify the top claims. I then re-ran
33:01the key GitHub counts and code checks
33:03myself. That is a lot more readable than
33:05the top of the last one.
33:07State in numbers: Open issues, bugs,
33:10speed, review. This is still a bit much
33:13for my taste. Once we go down here, we
33:15see what I like. Key decisions: One,
33:17freeze new providers and new services
33:19until the P0 below ships. Provider
33:21breath already drives the top issue
33:22clusters. Claude 118 and code 76 title
33:25hits. Kill the legacy sidebar. Two
33:27sidebars double every fix. Do not keep
33:29both. Treat discussions as untrusted
33:31demand until you check shipped statuses.
33:33Discussions never auto close. Sample the
33:35PR backlog before you judge it. Kanban
33:38does not prove review starvation.
33:40Yada yada. Not the best, but very
33:43readable. Now we get into the things it
33:45thinks I should do. I find these to be
33:46very, very clear and easy to read.
33:49There's a P0 section for what it thinks
33:50I should do for the next 2 weeks. It
33:52starts with Fix the two server stalls.
33:54Size M plus M. Here is an issue. It says
33:57what matters about it. Here's another
33:59issue that shows one stuck provider
34:01holds all four effect workers. The pool
34:03size is four in this file. Add per
34:06isolation on a stuck worker time out.
34:08Next, stop the PR quota and payload
34:10burns. Size small to medium. This issue
34:13reports 2,800 Bitbucket calls in 100
34:15minutes and 22,423
34:17GitLab calls in 12.5 hours. Stop
34:20re-fetch on focus. Cache per PR. Back
34:22off on quota errors. Another one showing
34:24a 17 megabyte thread snapshot. I don't
34:27know if I'm just tired of Claude slop. I
34:29find this 100 times more readable and
34:31clearer. The way it's using bold at the
34:33start of these say what matters. Fix
34:35provider first use. Open code shows only
34:3729 built-in models. This is an issue
34:39with certain providers not sending the
34:42right models to our app in time, so we
34:44show a wrong set. Very good to see. Fix
34:46scope stranding at the root, not per
34:49scope. Size small for the bug, medium
34:51for the system. I'll be real, this one's
34:53not as readable or clear as the others.
34:55I would probably go look at the issue to
34:56see what's wrong. In fact, we can here.
34:58Let's see how clear the issue is. iOS
35:00provider accounts fail when missing
35:02providers manage on fresh T3 connect
35:04tokens. Okay, so in certain cases
35:06connecting over the iOS app is failing
35:08with T3 connect. Good to see.
35:10Fifth, run a one-week data loss and
35:12billing screen. Recent titles named
35:14destroyed environment files, settings
35:16edits that revert sub-ages to keep
35:17billing after cancel. Counts are low,
35:19severity is high, evidence is thin, so
35:22repro each one on the current nightly.
35:24Fair callouts. It specifically said the
35:26counts are low, but severity is high,
35:27but there's very little evidence, so we
35:29should repro it. Then it has a priority
35:30one section with simpler things.
35:33Priority two saying later or not now.
35:35It's so much less taxed and it's so much
35:37more readable. So, what I'm trying to
35:39say with this model is it might not be
35:41the smartest, it might not find things
35:43the best, but it is way over my bar for
35:45what is usable and useful, far past what
35:48I would expect from a Gemini model or a
35:50Grok model. It's honestly the first
35:52model since Fable that I've used outside
35:54of Anthropic and OpenAI that I could see
35:56myself daily driving and I'm reaching
35:58for it a lot. I still bring in Opus for
36:00the hairy stuff. I'll often have this
36:02model call Opus to double-check on its
36:04work. I'm accepting slightly less
36:06capability because I like the higher
36:09floor for stupid, where it doesn't do as
36:10dumb of things, and I really enjoy the
36:13new flavor in its responses. The way it
36:15talks to me, I personally find way more
36:18readable and nice to interact with.
36:20Should you drop your quad code sub for
36:22this? Absolutely not. I don't even know
36:24how good the subsidies are on their
36:25subscription plans. Not that we could
36:27know, because again, they're not
36:29releasing the price for the model I
36:30like. There is so much cringe about this
36:32drop, and you can tell that Meta is not
36:34used to having a model good enough to
36:35care, but I care. I like this model. I
36:38will continue using it until they cut
36:39off my API key, then I'll probably go
36:41try the subscription, find that it's too
36:42limiting, and then turn and go back to
36:44Opus. But, if the price is right and the
36:46usage is right, I could see myself using
36:48this model a lot. Congratulations to the
36:50team at Meta for pulling this one off. I
36:52really didn't see it coming. I did not
36:54think that after the small model
36:56releases like the Spark line, that I
36:58would find myself actually reaching for
37:00this against other frontier stuff, but
37:03they did it. They kind of cooked.
37:05Even if this model isn't that exciting
37:06to you, the fact that Meta's caught up
37:08should be. This makes the future way
37:10more competitive, and if you've seen how
37:12much better the models from both
37:13Anthropic and OpenAI have been both in
37:15intelligence and in cost since they got
37:17more competitive, you know how powerful
37:19a third player could be. So, I guess all
37:21that's left is figure out what the hell
37:23is going on at DeepMind. Seriously,
37:24guys, you've been teasing Gemini 4 Argon
37:26for how long now? Where is it? When are
37:29we going to get it? When can I talk
37:30about it? When can I show the strengths
37:32and weaknesses?
37:33What Seriously, where is this model?
37:35Ugh.
37:36I
37:37The list of things that we've gotten
37:39before Gemini 4 Argon is going to be
37:41longer than the list of things we got
37:42before GTA 6 at this point. I don't know
37:44what's going on at Google, but I do know
37:46what's going on at Meta, cuz it seems to
37:47be pretty dang good. I'm hyped about
37:49this release. I'm curious how y'all
37:51feel. Am I overhyping this model? Am I
37:53being paid off by my friends at Meta who
37:55made React and now made a model I like?
37:57What do you guys think? Let me know in
37:59the comments, and until next time,
38:01peace and out.

About this transcript

This page contains the full transcript of Meta just caught up with Anthropic by Theo - t3․gg, generated from the public captions YouTube serves with the video. The transcript has 7,848 words across 1,098 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.