YT YouTube2Text

YouTube2Text → Transcripts → Meta's New Models & Agent

Meta's New Models & Agent — Transcript

by Sam Witteveen · 4,199 words · 588 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Okay, so last week Meta released the
0:02weights for Muse Spark open an 816
0:07billion parameter model that they're
0:08calling the most capable US open weights
0:12model to date. And I've got to say based
0:14on the numbers that they've released,
0:15this claim seems to actually hold up.
0:18But this isn't the only thing that
0:19they've been releasing. Meta has been on
0:22a absolute tear lately with what they've
0:25been shipping. So don't forget it's not
0:27much more than a year that the Meta
0:29superintelligence lab was actually
0:31created. And they shipped an open model
0:33Muse Glimmer early in the year, which at
0:35the time I talked about as being very
0:37interesting because of the fact that it
0:39was the first model from that new team.
0:42Clearly that set a bar. After that they
0:44had other people leave from Google to
0:47join that team, and I've got to say that
0:49they've really been doing a lot of
0:51interesting stuff since. So in the same
0:53week as them shipping this Muse Spark
0:55open, they shipped two closed frontier
0:59models. One of which you can't even get
1:01through their own API or through open
1:03router, etc. And it's important to
1:05distinguish that what they've actually
1:07released here is not the model behind
1:09their personal agent app, which is doing
1:11really well here in the US. And it turns
1:14out that model might actually be an
1:15entirely different model altogether. So
1:18in this video I want to try and untangle
1:19what Meta actually released, and then
1:21actually show some demos with the open
1:23model. Okay, so let's look at the actual
1:25model. So Muse Spark open is a mixture
1:29of experts model. It's 816 billion
1:31parameters total with 42 billion
1:35parameters active per token. It's got
1:38what now has become kind of the standard
1:401 million token context window. It's
1:43also natively multimodal, which is
1:46really cool to see, and it's not just
1:48text and images that's making it
1:49multimodal. It's video and audio in as
1:52well, but as to be expected it's just
1:55returning text. Now Meta is pitching
1:57this as a sort of general purpose model
1:59for coding, agentic work, a variety of
2:02different things. And they've made this
2:05claim that it's the most capable US
2:07model. It's very specific claim, right?
2:10So, it's kind of funny that they're not
2:12saying it's the best model out there,
2:13and I guess that's clearly pointing at
2:15certain countries perhaps having better
2:16models. And that kind of makes sense,
2:19right? If you look on the Artificial
2:20Analysis Intelligence Index, this is
2:23scoring 39. So, the top open models
2:26right now are all Chinese. We've got the
2:28Mimo 2.6 Pro at 46, we've got GLM 5.3 at
2:3445, Kimmy K3 at 44, and even the Qwen
2:373.8 is just above a Spark open at 40.
2:41So, while I do think this is clearly a
2:43solid model, it's not the leading model
2:45in the open field. And I do feel that
2:48shouldn't be something that stops us
2:50from sort of trying this model out.
2:52Definitely, organizations like Meta tend
2:55to make models and train on data that is
2:59quite different than the Chinese models.
3:02So, perhaps while there's no
3:03benchmarking going on or anything like
3:05that here for benchmarks, it's
3:07interesting to see how this actually
3:09generalizes to a variety of different
3:11tasks. The other big release that just
3:13came out just after this, which was also
3:16a US open release, was Reflections Beam.
3:20So, Beam's weights literally just went
3:22up on Hugging Face last Friday. Even
3:26though they announced the model earlier
3:27on in the month, and actually at the
3:28time I wanted to make a video about it,
3:30but there were just no weights to even
3:32try it. So, Meta's weights have come out
3:34first, and that's why I'm focused on
3:35this. Now, clearly Spark open is beating
3:39the Beam model, right? The Beam model
3:41scores 30 on that same index. So, Spark
3:44open is clearly a lot higher than that
3:46with 39. And when you look at the
3:48benchmarks that both companies
3:50published, Spark open is winning the
3:51majority of them. For things like tool
3:54use on MCP Atlas, it's 89 versus 79. For
3:59Deep Seek, it's 60 versus 44. Although
4:02Beam does edge it on the terminal bench
4:05with 80 versus 76. But here's the thing,
4:08in many ways it's not a fair comparison.
4:10Beam is a much smaller model. Beam is
4:13only 501 billion parameters in total
4:16with 23 billion active. And on top of
4:19that, they've really focused with that
4:20model to make it very token efficient.
4:23And something that I do think we have to
4:25take into account is that model is fully
4:27Apache 2 licensed. Spark Open is under
4:31Meta's own Muse license. Now, for most
4:34people that's going to be fine, but you
4:36do have to display a built with Muse
4:38Spark Open in your product. And if you
4:41run a model as a service business with
4:43more than $20 million a year in revenue,
4:46you need a separate agreement with Meta
4:48to be able to do that. So, I guess I
4:51would say that Spark Open is the more
4:52capable model, but it's much bigger.
4:55Beam is the cheaper, cleaner licensed,
4:58smaller one. Now, as for running this
5:00thing yourself, Meta worked with Apple
5:02on this. The headline setup is two M5
5:05Ultra Mac Studios with 512 gigs each.
5:09And apparently Apple measured that at
5:11about 25 tokens per second. Now, I don't
5:14know off the top of my head how much
5:15that would actually cost, but from what
5:17I'm hearing, not only would it cost a
5:19fortune, but you're probably going to be
5:21waiting until January or February for
5:24that hardware to actually arrive. Now,
5:26if you're not buying Macs, they've got
5:27an NVFP 4 checkpoint. It's around 530
5:31gigs. And Unsloth themselves have made a
5:342-bit checkpoint. That gets it down to
5:36around 300 gigs. But clearly for most
5:38people though, the realistic way to try
5:40this is going to be one of the hosted
5:42service providers. And in that sense,
5:44it's been on Open Router since day one.
5:47So, I'll come back to the demos for the
5:48Spark open in a second, but let's just
5:51talk about what else Meta has actually
5:53been releasing here. Spark open is
5:55clearly not their best model. It's not
5:57even really close to their best model.
5:59Meta clearly is back in the game of
6:02being a frontier model creator here. 3
6:05days before Spark open, they actually
6:07launched Muse wonder and Muse wonderful,
6:11with wonder and full being two words
6:14there. So, these are closed models, and
6:16Meta has been benchmarking those against
6:18GPT-6 Astra, Claude Opus 5.5, and on the
6:22Artificial Analysis Index, wonder scores
6:2552 and wonderful scores 56, which is
6:30pretty good compared to Opus 5, which is
6:31only two points ahead at 58. Now,
6:34obviously sitting under those is a bunch
6:36of different models, but also includes
6:38the Muse Spark 1.3,
6:41which scores 48. And I think a lot of
6:43people, myself in- included, were hoping
6:46that the Muse Spark 1.3 was going to be
6:49the model that was opened up for
6:50everyone. But, that really wasn't the
6:52case, I guess. The open model is
6:54actually nine points behind that sort of
6:57mid-tier closed model and 17 points
7:00behind their best model. Now, one of the
7:02issues you're going to have with getting
7:03access to these is that they're just not
7:06all out there. So, Spark 1.3 is on both
7:09Meta's API and on open router at a $1.25
7:14in $4.25 out. They also have a version
7:17of the model which is much, much cheaper
7:20at 10 cents in and 20 cents out. But,
7:23obviously the catch there is that Meta
7:26gets to train on your prompts and your
7:28outputs. So, just as we've seen the
7:30Chinese labs basically give sweet deals
7:33if they can train on your data, Meta's
7:35doing basically the same thing here. And
7:37I guess that's not a huge surprise. The
7:40Muse wonder model is on the Meta API and
7:44open router and that's $4 in and $20
7:47out, which turns out to be exactly the
7:49same cost of what Opus 5.5 is, but
7:52wonderful, remember that's two words,
7:55wonderful, isn't on any API. It's only
7:59available inside Muse code on the $50 or
8:02the $200 a month plans. So, I do think
8:05it is a bit rich for them to basically
8:07make all these claims about this model
8:10and yet nobody can sort of independently
8:12test them cuz there is no API available
8:15for people to actually try this out.
8:17Interestingly, on LM Arena, they put the
8:20Wonder model through more than 5,000
8:23real agent sessions and it only came
8:26back at number 12 in their agent arena,
8:30which is kind of interesting because it
8:31came in at number two on their text
8:33leaderboards. But, clearly it's a strong
8:35model. Perhaps on agent work, it's
8:37actually not leading. Honestly, I'm not
8:39sure what to make of the model coming in
8:4112th for agents because one of the areas
8:44that Meta is clearly actually winning a
8:47lot at the moment is not benchmarks,
8:50it's personal agents. So, as I'm
8:52recording this, I'm in San Francisco at
8:54the moment and personal agents seem to
8:56be the thing that everyone is talking
8:58about. Meta's Muse app is clearly the
9:01personal agent with the most amount of
9:03momentum at the moment. It launched in
9:05the US on September 8th. Within a couple
9:07of weeks, it's been downloaded more than
9:092.5 million times and it's actually been
9:12the number one app on the Apple App
9:14Store, just showing that they are
9:16getting real traction, not just with the
9:18AI developer community. So, the big
9:21question there is what can the app
9:22actually do? Well, it does things like
9:24reads your email, books things for you,
9:26fills in forms, it can pay on your
9:29behalf, it can negotiate for you. It's
9:31interesting because they're saying that
9:32each user gets their own isolated
9:35virtual machine with the agent and their
9:38data in it. They've got a free tier, a
9:40$20 per month tier, and a $100 per month
9:43tier in there. And while apps like
9:45ChatGPT and Gemini are still far more
9:48widely used overall, in the specific
9:51sort of category of personal agents, it
9:53does seem like Muse is the one people
9:55are talking about. Now, like I said
9:57before, unfortunately Muse doesn't run
9:59on Spark Open, right? It's on a
10:02customized version of Spark, and we
10:05don't really know too much about exactly
10:06which versions it's running, if it's
10:08running any specific fine-tunes of it,
10:11that kind of thing. But, there has been
10:12some really interesting info that has
10:14come out about how they actually built
10:16this. So, Alexander Wang actually went
10:18through this on the David Senra's
10:20podcast, and I've got to say it's one of
10:23the more useful explanations I've heard
10:25about how a lab actually ships an agent
10:28product, and I've put the link to that
10:30in the description. So, basically it
10:32started back in February with Alexander
10:34Wang and Nat Friedman spent weeks living
10:36with sort of open-claw agents, and then
10:39a whole bunch of that team trying out
10:41these open-claw agents. And their
10:43conclusion was basically that the magic
10:45was real, but these things just didn't
10:48really work reliably. So, over the next
10:50few weeks they worked on prototyping
10:53both the agent system, but also the
10:56models to be able to support that. And
10:59it's interesting that it looks like the
11:00Muse personal agent really has come from
11:027 months of grinding with what they're
11:05saying is a team of under 200 people
11:07across both the model and the product
11:10for it. I found it really interesting
11:11how they described that they actually
11:13did this. So, what they did was they
11:15listed more than 100 specific behaviors
11:18and capabilities
11:20that the ideal personal agent needed to
11:22have. And they put all those into a
11:24spreadsheet, and every row got its own
11:27set of evals. They then trained the
11:30model against those evals. And they
11:33basically had sort of like a green
11:35light, red light system of knowing,
11:37"Okay, with this particular model, do
11:40these evals pass or not?" And very
11:42quickly they could see what percentages
11:44were actually working and what
11:45percentages did it. And you can imagine
11:47that they were then basically using lots
11:49of reinforcement learning environments
11:51to do post training, etc. to get these
11:53things to work. They kept looking at the
11:55use cases that didn't work and then
11:57updating for them. And I thought it was
11:59interesting that he talked about each
12:01row had a launch blocking threshold and
12:04a good threshold. So there were lots of
12:06checkpoints where they were green on
12:08about 80% of the rows and red on about
12:1020. And they literally decided that
12:12wasn't good enough. He mentioned that
12:14the first checkpoint that was green on
12:16every row was that specifically trained
12:18version of Spark 1.3. And I kind of feel
12:21what he described in there is really a
12:24fantastic way of how to build any
12:27agentic product. If you're building your
12:29own agent, you probably don't have 100
12:31behaviors written down with evals for
12:33each of them. But you could start with
12:3520 and then grow it from there. Write
12:37down what the agent has to do, build
12:39some evals for each one, and then just
12:41don't ship until they're passing all of
12:44the evals. So like I said, the link for
12:46that is in the description. All right,
12:48let's jump in and have a look at the
12:49demos of how this model actually goes.
12:51But before that, I have an ask for you.
12:53If you're watching this and my other
12:54videos and you haven't subscribed,
12:56please subscribe. I'm really trying to
12:58get the number up by the end of the
13:00year. This helps a lot in making these
13:02videos get out there and let more people
13:04see them. Okay, so jumping into the demo
13:07here, what I'm going to do is compare
13:09both the Muse Spark Open and the Muse
13:12Spark Wonder. So both these models I'm
13:15running on Open Router. So you can see
13:18Open Router here for Muse Spark Open,
13:21there are actually quite a number of
13:22providers already. We've got Modal doing
13:24quite quickly, Together AI, Nebius,
13:27Fireworks, Base Ten, etc. It is
13:30interesting to see in here that Meta
13:31themselves are not actually running a
13:33version of this. All of them here have
13:35the same pricing, $1.25 in, $4.25 out
13:39for the open model. If we look at Muse
13:41Wonder, it's quite a different story.
13:43There's only one provider, being Meta
13:46themselves. This one is the $4 in, $20
13:50out. Again, in this day and age, that's
13:51not really something that's very cheap
13:53there. And the other thing I'll point
13:55out, too, is that Meta is actually
13:57retaining prompts, even though they're
13:59saying that they don't use them for
14:01training. All right, so I've basically
14:03just set these up here where I can use
14:06them in my playground. And I kicked them
14:08off basically doing the SVG test. And
14:10the reason why it was that people were
14:11sort of criticizing the Meta models for
14:14graphics and for doing websites and
14:16stuff like that. I'll got to say, for
14:18the Wonder model, it's not bad here. If
14:21we sort of look at the pelican SVG, it's
14:24only okay. It's definitely not as good
14:26as some of the other ones. Although,
14:28more and more I'm starting to feel that
14:29this is probably not that useful of a
14:31test, because the model companies
14:32themselves are sort of over fitting on
14:34it. What I do find really interesting in
14:36here is just how little thinking Muse
14:39Wonder does. So, you can see that
14:41basically it's just got 55 tokens of
14:44thinking in here, which is kind of just
14:47rephrasing
14:48what I've put in the prompt for
14:51basically putting it together. And then
14:52it goes straight into generating the SVG
14:55in here. If we look at Muse Spark open,
14:58it's a very different story here. So,
15:00you can see for a start, pelican not
15:02looking so great. Bicycle perhaps not
15:05okay. And actually, the feet on the
15:06pedals is pretty good. But
15:08fundamentally, it seems like there's a
15:09bunch of things going on there. And more
15:12importantly, I think that the thing here
15:13is just how much thinking this does
15:17before it actually goes to generate out.
15:21So, you can see that it generated a lot
15:23more tokens than the Wanda version in
15:26there. And really, I think that's a sign
15:28of basically how they've been doing the
15:30RL here. The Wanda version, while being
15:33a different pre-train and almost
15:35certainly a much bigger model, etc. It's
15:36also had just a lot better reasoning and
15:40long chain of thought done on it as
15:42opposed to the open one. I guess here's
15:44an old kickoff just the long prompt for
15:46asking them to basically do a long
15:47essay. Meta clearly doesn't want to give
15:50away how they're doing the long chain of
15:53thought, right? So, if we look at
15:55something like this, where we've got the
15:57open model, we can see that this is got
16:001,000 tokens of thinking, which is
16:04actually open thinking that we're
16:06getting there. And if we compare that,
16:08you can see that's come to a stop at the
16:10end there. In this case, we actually
16:11have gotten quite a substantial essay
16:13out. That's actually not always the
16:15case. So, you can see that if I come
16:17back and run this again, this time we're
16:19getting less thinking tokens. But, quite
16:21a number of times when I've run the
16:22test, while the quality coming out is
16:25quite good, the amounts just vary widely
16:28of where sometimes it's only giving sort
16:31of 3,000 to 4,000 tokens back, sometimes
16:34it's giving 7,000 tokens back. So, this
16:36is with the open model. If we look at
16:38Muse Wanda, we see again we're getting
16:41just really low amounts of thinking in
16:44here, but pretty good quality output
16:47going on here. Now, going back to Muse
16:49Spark open, you can see here this time
16:51we only got 1,800 words. Last time we
16:53got 4,900 words. It's just very erratic,
16:57and while of course I expect that it's
16:58going to be different each time, I
17:00wouldn't expect it to have such a huge
17:03range of difference. The variation there
17:05is really high. Where you can see from
17:07Muse Wanda, it basically went through
17:09and gave us a full essay back there.
17:11Now, I don't know what the details are
17:14of what they're actually giving us back
17:16of reasoning. I suspect that we're
17:18getting perhaps like a summarization of
17:21the reasoning tokens back. This is
17:23something that's probably worth looking
17:24into a lot more if you're going to
17:26actually use the paid Muse Wonder model
17:29in here. That's it for things like the
17:31rag Q&A, this does a decent job going
17:33through it. It's just using a lot of
17:36thinking tokens. If I try turning
17:38thinking down to low, I can actually get
17:40it to use a lot less tokens, but that
17:43certainly does seem to affect the output
17:46of going through this. All right, so
17:48running the sort of basic agentic tests
17:51of going through it, you'll see that the
17:53Spark open actually does a really good
17:55job at the function calling and dealing
17:57with different kinds of things going on
18:00in the function calling here. So, the
18:02basic tests, it's passing 10 out of 10
18:05for this. Not surprisingly, the Muse
18:07Wonder also scores perfect 10 out of 10
18:10for these particular ones. The story
18:12does change slightly when we look at the
18:15agentic sort of tests in here. The
18:17model's still doing very well, but it
18:19tends to basically end up using a lot of
18:22tool calls I found. So, for this one,
18:24where plenty of other models would have
18:26done this very quickly, this is doing
18:28lots and lots of tool calls to actually
18:30do this. And this could actually explain
18:33why it was not doing great on the agent
18:36arena Elo scores there is that it was
18:39just probably taking a long time. That
18:41said, it does pretty well on most of the
18:43other things dealing with broken tools,
18:45it did did very well with that. It was
18:48able to recover and just keep going and
18:50stuff like that. All the things with
18:51distractors and stuff like that in the
18:53prompts, it did pretty well. The open
18:55model scored seven out of eight. If we
18:57come across to the Muse Wonder, it has
18:59no problems with these and it scores an
19:02eight out of eight here. And you'll
19:04notice that like with the other one, it
19:06just kept going and going and going for
19:08this targeted edit, but this one
19:10basically just a handful of tool calls
19:13and it was done. So, that sort of
19:15separates the quality of the open model
19:19versus their frontier model. The last
19:21one that I'll look at is actually the
19:22HTML one. I don't think this model is
19:25very good for these kind of tasks. I
19:27don't think either of these models is
19:28necessarily great for this kind of task.
19:31The Muse wonder, very good at logic,
19:34very good at those kind of things,
19:35perhaps less so at coding, but that
19:38doesn't surprise me. I don't think
19:39that's what Meta is sort of going for.
19:41If we look at the websites here, this
19:43one has actually turned out not bad. So,
19:45this is the Spark open version in there.
19:49If you look at the one for Muse Spark
19:51wonder, it's pretty plain, I got to say,
19:54right? I don't think it's that great
19:56compared to some of the other models
19:58that I tested in here. Looking at this,
20:00both of them are using sort of 6 and 1/2
20:02thousand tokens to actually get here.
20:04Overall, I think the model's pretty
20:05good. My guess is that the people who
20:08are going to end up using this, cuz
20:10think, honestly, I would switch away
20:12from something like Qwen 3.8 next or
20:15even the Qwen 3.8 27B. Don't forget this
20:18model is so big that very few people are
20:21going to be able to run this locally or
20:24actually do anything with it locally.
20:26Most people are going to be running it
20:28off an API, and I kind of suspect the
20:30main usage of the open model is really
20:34going to be US companies, perhaps some
20:37companies in Europe, et cetera, who
20:40perhaps for political reasons don't want
20:42to use a Chinese model or certainly want
20:44to be able to have plausible
20:45deniability. And this is something that
20:47I'm definitely seeing a lot of companies
20:50not want to admit that they're using the
20:51Qwen models in production, purely for
20:54what it seems to me to be political
20:56reasons. All right, so where does this
20:58leave things? Overall, I think this is a
21:00really good sign. Meta is definitely in
21:02the open models game again. They're not
21:04doing the whole sort of trick that
21:06OpenAI did of just put out one model and
21:09then abandon it and not do anything
21:11more. They've now got back in. They've
21:13put out multiple models. Hopefully,
21:15they'll keep going and update those
21:16models. The Glimmer model, their 30B
21:19Apache license model, came out in
21:21August. And when it launched, Mark
21:23Zuckerberg said that they'd open the
21:24weights to Muse Spark. And 2 months
21:26later, here we are, right? It's a
21:28different version from perhaps what we
21:30would like and that we would like to
21:31have that 1.3 version, but at least it's
21:34a good sign here. And with this past
21:36week being a big week for US open models
21:39generally with both the Spark open and
21:42Beam both landing weights in Hugging
21:44Face. It's great to see these companies
21:46entering the game and certainly good for
21:49a lot of other languages, et cetera,
21:51because Meta's got a lot of multilingual
21:53data and they've got certain countries
21:55and regions that they prioritize,
21:57perhaps like European languages, certain
21:59Southeast Asian languages, et cetera,
22:02that are often just not very strong in
22:04the Chinese models. So, while they're
22:06landing this model quite a bit behind
22:08the Chinese open weight models, this
22:10model does make sense for a lot of
22:11people to actually try out here. I'd
22:14love to see their next open release at
22:16least get to the Spark 1.3 level. If we
22:19could see an open model coming out from
22:21Meta every 3 months, you've got to think
22:23that that's going to be great for the
22:24whole ecosystem. Anyway, I've put the
22:26links to the Spark open, to Beam, to the
22:29David Senra interview that I talked
22:31about in the description. Let me know in
22:33the comments what you think about Meta's
22:35split between the sort of open models
22:37and the closed models. And as always, if
22:40you found the video useful, please click
22:41like and subscribe, and I will talk to
22:43you in the next video. Bye for now.

About this transcript

This page contains the full transcript of Meta's New Models & Agent by Sam Witteveen, generated from the public captions YouTube serves with the video. The transcript has 4,199 words across 588 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.