YT YouTube2Text

YouTube2Text → Transcripts → So I've been testing Gemini 4...

So I've been testing Gemini 4... — Transcript

by Theo - t3․gg · 7,925 words · 1,129 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00It finally happened. Gemini 4 Argon is
0:02here. Before we get started, I got a
0:03quick question. How long do you think it
0:05has been since we got a normal-sized
0:07large Gemini model release, like the Pro
0:09line? How many months do you think it
0:11has been? The gap between a model
0:12release like 5 6 Soul and 6 Astra was 1
0:15to 2 months. The gap between 3 1 Pro and
0:18Gemini 4 Argon is 8 months. It's been
0:22since February. We have not gotten
0:24anything other than flash models from
0:26Google. And now, after teasing Gemini 4
0:28Argon over a week ago, they have finally
0:30put it out for all of us to use. With a
0:32drop like this, there's a lot we have to
0:34cover, from how it's priced to how good
0:36it is at front end, to all the 3D
0:37capabilities, the weird things that make
0:39Gemini models Gemini models, the unique
0:42capabilities in those models, and of
0:43course, the comparison with recent
0:45releases like Opus 5.5, GPT 6 1 Soul,
0:49and Muse Wonderful, which if you've seen
0:50my video on that, I am quite enjoying. A
0:52handful of my friends at Google actually
0:54stuck their necks out a bit to get me
0:56early access to this model. I'm very
0:57thankful for them, and I sincerely hope
0:59they don't get in trouble for what I'm
1:00going to say. There are things that are
1:02good about this model, and there are
1:03things that are less good about this
1:05model. Normally, this is where I'd make
1:06a joke about how I'm not a paid shill
1:08and the company isn't paying me, but in
1:09this particular case, I can't because
1:11Google does pay me quite a bit because
1:13I'm a YouTuber, as you might know,
1:14watching this video likely on YouTube.
1:16All of that said, they are not paying me
1:18for my coverage for anti-gravity and
1:19Gemini stuff. In fact, my previous
1:21Gemini videos almost got me demonetized
1:23on YouTube, so this comes at a cost to
1:25me, if anything, which is why I hope you
1:27can forgive us for a quick break for
1:28today's sponsor. I got an interesting
1:30question for you. You want people like
1:31me to use your product because if you
1:33do, you probably shouldn't do a normal
1:35sign-in flow because I'm not going to be
1:37clicking those buttons myself nowadays.
1:38I do most of my work through my agents
1:40now, which means any platform that my
1:42agents choose needs to be usable by the
1:44agent. In particular, it needs to be
1:45able to be signed up for by your agents.
1:48Today's sponsor is WorkOS, and that
1:49might confuse you because they are a
1:50platform for user signing in, right?
1:53Well, they are. They are the best way to
1:54have humans and businesses signing into
1:56your app. So, if you're trying to get
1:57enterprise customers, then you should
1:59probably be using WorkOS. That's why
2:01everyone from OpenAI and Anthropic to
2:02Fowl, Cursor, Bolt, Vanta, Carta, and
2:05more are all using them. But, I want to
2:06talk about AuthMD because this is a
2:07super, super cool standard. The goal of
2:09AuthMD is to make it easier for agents
2:11to sign up for your apps. It's a new
2:13standard that's been designed by WorkOS
2:15in partnership with companies like Fire
2:16Crawl and Cloudflare. And the results
2:18speak for themselves. There's already a
2:19ton of companies adopting these
2:20standards like Neon, Monday, Parallel,
2:23and more. What makes it so cool is that
2:24agents can now register on behalf of a
2:26user. So, if you're trying to get
2:27business customers and you want their
2:29agents signing up, this is a great place
2:31to start. Get Auth Right for all your
2:32users at swyd.link/workos.
2:35Now that that's out of the way, let's
2:36dive into the newest model from Google,
2:38Gemini 4 Argon. Flash banging warning
2:40for their website because it is not dark
2:42mode for some reason. Argon is now
2:44globally available today for developers
2:46and enterprises via the API and studio
2:48and Gemini platform, Google AI Ultra
2:50subscribers and anti-gravity, and Google
2:52AI Ultra and Pro subs in the Gemini app
2:54and AI mode in search. This is the God,
2:57I did not realize how bad Google's like
2:59products line had gotten until this.
3:01It's a mess. Model excels at deep work
3:04in complex real-world tasks across
3:06domains, including coding, finance,
3:08legal, cybersecurity, and science. Its
3:10depth and precision across multi-step
3:12tasks lead Argon to outperform other
3:14frontier models on benchmarks across
3:16domains, ranking number one on the VALS
3:19index, which measures model performance
3:21across critical economic sectors like
3:22finance, coding, legal, and tax work.
3:25And yes, indeed, Gemini 4 Argon does
3:27appear to be one of the best values
3:29here. Somehow they have a cost for
3:31wonderful,
3:32though, which is strange cuz I was never
3:35given token pricing. Okay, they're
3:36saying it's $4.00 in and $20.00 out.
3:38That's wrong. We don't have a cost for
3:40wonderful, and it won't be the same
3:42price as standard wonder, obviously. So,
3:45uh that makes me trust this a lot less.
3:47I'll be real. Regardless, I have my own
3:49benches. I've I've run a lot of benches
3:51on this model. I'll start with some
3:53benches that aren't mine. With Slot
3:54Politics, I already shared the numbers
3:56for Gemini here. You can see that its
3:58cost is not actually that great when you
4:02factor in how many tokens it uses on
4:04something like the Artificial Analysis
4:05Intelligence Index, but its score is
4:07putting it above Astra and also above 61
4:10Soul, but meaningfully lower than Opus
4:125.5 on high as well as obviously X high
4:15and max. And it is also more expensive
4:18than Opus 5.5 on high despite getting a
4:19lower score. Of course, Artificial
4:21Analysis Intelligence Index is not a
4:23perfect measure of model capabilities.
4:25It is not the best index for things like
4:27code, but it is a good general idea of
4:30where intelligence sets. I chose to show
4:32you these numbers first because I'm
4:33scared people's brains fall out whenever
4:35I show them the model price. Because if
4:37you see this $2 per mil in and $10 per
4:39mil out, your immediate assumption's
4:41going to be this model is cheap when you
4:43compare it to things like the 5.25 rates
4:46for Opus 5.5. This seems like a really
4:48good deal. It's not. It's a combination
4:51of reasons. The first one is that this
4:52model is it's token hungry. It's not
4:54particularly efficient, which is part of
4:56what puts it in the higher price range
4:58like we just saw on Artificial Analysis.
5:00The other really, really egregious
5:02problem is that it sucks at hitting
5:04cash. I have had the worst rate of cash
5:07reads from any model I've used recently
5:10with this release. I'm talking like 70
5:13to 80% cash hit versus other models
5:16where I see 99% plus. It means that that
5:1895% discount on cash reads is not going
5:22to happen as often and you're going to
5:23actually eat a lot of input cost on this
5:25model. This might be a bug in
5:26Anti-Gravity either in how it's actually
5:28sending things or how it's reporting
5:30data to me. It is slightly possible this
5:31is a bug in the MCP implementation that
5:33we use in T3 Code because you will never
5:35catch me dead in their awful CLI, but I
5:38am not particularly confident in any of
5:40this and I am not interested in doing
5:41more work to figure it out because
5:43Google should be the ones doing this,
5:45It's me. They have a section about using
5:46it to solve complex engineering tasks
5:49saying that uh anti-gravity has helped
5:51teams increase agentic code submissions
5:52by more than 35% while reducing code
5:56rollbacks by nearly a third.
5:59Those are not great numbers considering
6:01that what they were using before was
6:03also not particularly good. The more
6:06interesting thing and I won't air out
6:07who this was, but I was talking to a
6:09Google employee in not one of the ones
6:11that Google knows I talked to, so don't
6:12go after the people who normally work
6:14with me. The way they described Argon to
6:16me wasn't it's a revolution or this
6:19model has changed how we build. The way
6:20it was described to me is I find myself
6:22reaching for Opus 5 less. Not 5.5, Opus
6:265. The one that I notoriously panned
6:28after realizing I was wrong in my first
6:30video. It is what it is. Let's see what
6:32this can do at least according to
6:34Google. Apparently they use it to
6:35untangle browser feature code in Chrome.
6:38Piping multiple clean code changes at a
6:40time, Argon helped the team finish an
6:43architectural refactor across hundreds
6:45of files completing months of work in
6:47weeks. This feels like it was written in
6:502024. It optimized a production Python
6:53code base in just a few hours. The
6:56Here they have some fun demos of the
6:57cool 3D stuff they've been doing with
6:58it.
6:59God, why is it making sounds? That's
7:01horrible.
7:02This isn't a generated video. This is
7:04generated 3D models that they then went
7:07through with some type of like software
7:09rotoscope type thing to show the models
7:12in video form.
7:14Very cool. Animations are still a bit
7:16cringe.
7:18This YouTube player embed is broken and
7:19the overlay won't disappear.
7:22Classic Google-isms. I I've joked for a
7:24bit now, you'll know when Google has a
7:26good model when their software finally
7:27starts working. They got a quote from
7:29someone at Deloitte. Early impressions
7:32from Gemini 4 are very strong. For
7:33long-running tasks across these
7:35workflows, it completed successfully
7:36100% of the time in our e-vals,
7:38maintaining context and running through
7:40multiple tools across impressive spans.
7:43Instead of the reasoning delivers
7:44significantly higher, clearer, yada
7:46yada, the text clips.
7:48>> [sighs]
7:49>> Google being Google. I shouldn't be as
7:51annoyed as I am by this completed
7:53successfully 100% of the time in our
7:55evals thing because this depends
7:57entirely on what's being evaluated. For
7:59example, I have an eval that Gemini
8:02completed 100% of the time. It's
8:03Skatebench. I still actually can't
8:05believe this. I think it's cool as hell.
8:07Gemini for Argon got a perfect score on
8:09my private version of my benchmark for
8:12seeing how well models name skateboard
8:14tricks. For reference, Opus 5.5 on Max
8:17gets a 79%. This bench has proven to be
8:19weirdly difficult for most models to do.
8:23In particular, some of the tricks that
8:24are like switch and fakie stance cuz
8:25skateboarding is a whole grammar around
8:27the way you're standing on the board.
8:28Google has always had a weird lead here
8:30with models like 3.8 flash getting a 96%
8:33but seeing the first perfect score was
8:36kind of trippy for me. Especially when I
8:38looked into it more and saw just how
8:40many errors it was hitting when it ran
8:42because the API was quite unstable
8:44during my early testing. I don't know if
8:46that's improved since but I was getting
8:48pretty intense error rates and I had to
8:49overhaul a bunch of my benchmark just to
8:51get around the fact that the endpoints
8:53error as often as they did. It was also
8:55relatively impressive in terms of its
8:56token utilization in the bench like it
8:58got answers fast and didn't overthink
9:00which is rare for a Gemini model and
9:02nice to see. So, yeah, if your task is
9:05somehow reminiscent to naming skateboard
9:07tricks, there's a decent chance you're
9:08going to have a good experience with
9:09Argon both in its like speed, token
9:11efficiency, but also its accuracy in
9:13doing the thing. But again, when you say
9:15a model has 100% success in our evals,
9:18the eval and its relation to the work
9:20you're doing matters and I don't think a
9:22lot of people are making a living naming
9:24skateboard tricks. I wish they were. I
9:25wish I could have that be my job but uh
9:27sadly it is not. They also say that the
9:29model has advanced reasoning and
9:31multimodal capabilities while
9:32maintaining strict grounding, yada yada
9:34yada. It's good at looking at documents,
9:36PDFs, videos, audio, things like that. I
9:39have historically seen a meaningful lead
9:40in Gemini models' ability to like do
9:42things with vision. Like giving it a
9:44picture of a motherboard and asking it
9:46to label the parts and draw that in like
9:48Python on top of it. The Google models
9:49have historically been very impressive
9:51in this space. I did not benchmark any
9:53of that because I was just using it for
9:54code. But if you have complex vision
9:56tasks, it is definitely worth running
9:58them against Gemini 4 just to see if you
10:01have a real benefit. You might be
10:02surprised. Do note though, as we start
10:04to complain more about Googleisms, that
10:06the current price of $2 per mil in and
10:08$10 per mil out is an introductory
10:11price. This is Google's new favorite
10:13thing to do when they drop a model. They
10:14charge a reasonable price up until a
10:17certain date, and then they double the
10:19price after. So this model's actually
10:21going to be $4 per mil in and $20 per
10:23mil out starting December 1st. It's
10:25almost like they don't want you to start
10:27building heavily on top of the model.
10:29It's It's rough. They've been doing this
10:31for most of their model drops recently
10:32and it makes pricing weird. I I don't
10:34think it's as bad as what Meta did by
10:36not putting out a price at all for
10:37wonderful and only having it in the sub,
10:39but I still hate this. Just give us a
10:41price that you can actually serve it at.
10:43Lower the price in the future. Don't
10:44raise it. The only lab that regularly
10:47raises prices for models is our friends
10:49over at Google. And then they tell us to
10:51get started. So I guess we should.
10:52First, we'll start with one of the
10:53things Google models have historically
10:55been pretty strong at. The thing is of
10:57course front end. I have used Google
10:59models for front end stuff pretty often
11:01when they are the new thing.
11:03Specifically when 3 Pro and 3 1 Pro
11:04dropped, I found a lot of good uses. I
11:06built a lot of the UI and UX around
11:09Lawn, my alternative to Frame.io for
11:11video stuff. It's what we actually use
11:13for all of our video management for my
11:15company. Every one of my YouTube videos
11:17gets uploaded on this platform and
11:19managed by my team on Lawn, and a lot of
11:21the UI and UX here was built by Gemini 3
11:241 Pro. I obviously did a lot more work
11:26on top of that using the Opus models at
11:28the time, but I found that the back and
11:30forth between those models allowed me to
11:32make actually unique-ish homepages at
11:35the time. So, let's see how that
11:37capability has carried through. I think
11:39this is quite good. I don't necessarily
11:42think a homepage should have this much
11:43text on it,
11:45but it is very nice-looking. I would not
11:48have guessed that this was AI-generated
11:50if I just stumbled on a page like this
11:51like browsing the web. It also made a
11:53nice little interactive section here
11:55where you can do things that is actually
11:57functional and not cringe. Like, this is
12:00decent. Believe me, I would talk a lot
12:02of crap if I could. This is fine. Hop
12:04into the other designs. That one had a
12:05bit of like font popping, that might
12:07just be the way that WhichAI is set up.
12:08By the way, WhichAI created by our
12:10friend Dara, really cool kid. Check it
12:12out whichai.dev. Very useful to see how
12:14the front-end capabilities vary across
12:16different models. All of the versions
12:18I'm showing now are using the official
12:20Anthropic design skill. So, if you
12:22aren't getting good designs out of the
12:24model, definitely try turning that on
12:26and see if you have better results. If I
12:28turn it off, you get a lot more cringe,
12:30a lot more of this like unnecessary text
12:33everywhere, a lot of fonts that are kind
12:35of gross, just not the best quality.
12:39Like, all of them have that like awful
12:41top nav, like every single one of them
12:43does. God, this retro one is so cringe.
12:45And this is without any skills on. So,
12:47if you are using this for design, highly
12:50recommend Anthropic design skill. I
12:51don't recommend the taste skill though.
12:53That gets it back to where it was with
12:55the weird cringe top navs and as well as
12:57like the really consistent gross layout.
13:00The design skill seems to be a good
13:01balance that actually forces the model
13:03to make things that look decent and
13:04unique. So, for front-end, seems pretty
13:06good. Is it state-of-the-art? I still
13:08think Fable 5.1 is the best I've seen.
13:11Opus has its moments, but Fable overall
13:13makes the coolest,
13:15most unique front-ends and like
13:17homepages, marketing sites, that type of
13:19thing.
13:20I still don't think any model is good at
13:21just like doing things right the first
13:23try, and I prefer ones where I can and
13:25back and forth. For example, with
13:27Slapolitics, I used OpenAI models for
13:29this entire site. I did everything with
13:32Astra on ultrafast. Not because
13:34ultrafast is magically better at design,
13:36all OpenAI models kind of suck at
13:37design, but since I was using ultrafast,
13:39I could tell it what I wanted and ask
13:41for different versions and go back and
13:42forth until I got it just how I wanted
13:44without having to wait constantly
13:45throughout. So, I think that speed is
13:47really important when you're iterating
13:49on real UX, not like a marketing site,
13:52but like the buttons people press on the
13:53website and how they interface with it.
13:55Google models are fast, but that doesn't
13:57necessarily mean they get work done
13:59quickly. Anecdotally, having tested this
14:01a bunch myself, it's not a super fast
14:03model. It might generate tokens quickly,
14:05but it doesn't solve problems fast.
14:07Next, we should talk a bit about the
14:08unique capabilities of Google models.
14:11This is strange because every lab kind
14:14of has something like this. For example,
14:17with OpenAI models, you can generate
14:19images and it will use their API to call
14:21the image gen tools and models that they
14:22have, where with Claude models, you
14:24can't do that. You have to call some
14:25other external service. Grok models have
14:27the ability to do cool things with the
14:29Twitter API. Like if I want to get good
14:31data out of Twitter, I can't do it
14:33through the API, it's way too expensive.
14:35So, I'll just tell Codex or Claude to
14:37call Grok, and by calling Grok, it gets
14:39access to the XAI APIs. A lot of
14:42different models have these types of
14:43capabilities. If they have access to
14:44unique data, you can get it through that
14:47model. So, why am I bringing this up?
14:48Gemini models have some unique
14:50capabilities, many of which are very
14:52useful to me. Why are they useful to me?
14:54I'm a YouTuber. That means I have to do
14:56a lot of stuff on YouTube, and Google
14:57gives very generous access to the
14:59YouTube API and capabilities to the
15:01Gemini models. If you want to get a
15:03transcript from a YouTube video, the
15:04Gemini API is one of the best ways to do
15:06it right now. And we've been using the
15:07Google models heavily for all of those
15:09types of things throughout my
15:10businesses. So, I will be using this
15:13model a lot. Not because I necessarily
15:15want to or I'm choosing to cuz it's so
15:17uniquely intelligent and such a good
15:18value. Just because Google has unique
15:21capabilities that they're providing
15:22their models through their APIs, through
15:24their data centers, through the things
15:26that they uniquely have access to. That
15:28said, I've actually found the Gemini
15:30models not that good at most Google
15:32services. I have a much better
15:34experience dealing with my email inbox
15:36through tools like Grok and Chat GPT
15:39than I do with anything Gemini. In fact,
15:42every time the Gemini button appears in
15:43some app, it it scares me and is often
15:47not functional or just outright cringe.
15:49I think if you're a YouTuber or you work
15:51with YouTubers, Gemini will be a very
15:53useful thing to have in your pipeline.
15:55If you're doing a lot of unique vision
15:57work, historically the Gemini models
15:59have been good at that as well, so worth
16:01benchmarking. I don't think there is as
16:03much here as there used to be though,
16:04cuz a lot of the things that Google's
16:06models were uniquely good at have been
16:09maintained and achieved by others
16:11externally. There is some secret sauce
16:14in this model. There's something that
16:15makes it different. You can see it in
16:17SkateBench. It's the only model to get a
16:19perfect score and the only models to
16:21break the 90 range are Gemini and
16:24somehow Muse wonderful. Everything else
16:26is in the 80s, 70s, even 50s and 60s.
16:29So, there might be more secret hidden
16:32capabilities I'm not seeing in the
16:33model. I've heard people say that
16:34they're really good for
16:35internationalization, localization,
16:37multilingual stuff. I haven't seen too
16:39much of this unique capability in my own
16:40testing. I just wanted to call it out
16:42because it's not that common nowadays.
16:45Usually, since all the models are just
16:47throw all the data in pre-training and
16:49then RL it on code, we don't get as much
16:51unique capability in things anymore, but
16:53Google still has some and I thought it
16:55was worth noting. Now, we get into the
16:57meat, the things that y'all are most
16:59hyped about, things like the real
17:02day-to-day code work, the things I hate
17:04about Gemini itself and the Gemini
17:06ecosystem, Opus, Soul, and Muse and how
17:08they compare, and of course, the thing
17:10everyone is hyped about, weirdly so,
17:13Fish Slap, my game that I make with
17:14every model to see how good they are at
17:163D. So, uh without further ado, I think
17:19it's time for fish slop. And no, the
17:21video is not encoding incorrectly. It
17:24did actually screw up the top of the
17:25water that badly. The UI is uh
17:29it's Gemini, to say the least. It did
17:31make the whole game loop where there's
17:33five tanks to get through, which is
17:36solid. Submarine models,
17:38not particularly great. It's not as bad
17:40as things like Muse and Grok, but it's
17:42not great. Considering how crazy all the
17:453D demos I saw other people posting
17:47were,
17:48I expected a lot more. I gave it the
17:50exact same prompt, exact same setup,
17:52exact same tools, and everything as the
17:53others had.
17:55It does animation decently well. Like,
17:56the way the fish moves is reasonable.
17:58The eyes are pointed in the right
17:59direction, which might not sound like
18:01much, but just a few months ago that was
18:03a rare skill.
18:04But, there are some uniquely
18:07Google-esque things here.
18:09Let me show you one of the funnier
18:11examples.
18:13So, games have to have physics and
18:15mechanics. My game has a mechanic where
18:17the fish eat the food.
18:19There's a problem.
18:21The food can fall to the bottom of the
18:23tank, and when it does, it doesn't
18:24expire.
18:26The fish can go to the bottom of the
18:27tank to eat it, but they can't figure
18:29out how.
18:31So, the fish will come over cuz they're
18:32hungry. They'll try to eat the food
18:34that's on the floor, so they can't, and
18:36then they'll just sit there staring at
18:37the food trying to get to it until they
18:39die. It wouldn't be too bad if I could
18:40just like feed them more, but the game
18:42has a mechanic where you can only put so
18:44much food in the tank at a time, and it
18:47forgot to make these expire.
18:49So, it will actually just sit here
18:51forever. I also just realized the sound
18:53is not routing, so let me fix that
18:55quick. There we go. Sorry for the lack
18:57of sound for the first bit. As I said,
18:59you weren't missing out on too much.
19:00It's not particularly great-sounding
19:02game.
19:03But, this version was bad enough and
19:05broken enough that I wanted to give it
19:08one more shot.
19:09So, I did. I told about some of the
19:11things that were broken, like the
19:12shimmering on the ceiling of the tank,
19:14the food mechanic being broken where the
19:16food wouldn't expire and they couldn't
19:17get at the bottom, and that I was
19:19disappointed in the graphics and I
19:20wanted it to improve. So, it ran for a
19:22while. It came back already looking a
19:26hell of a lot better.
19:28Didn't touch the model for the sub,
19:31sadly,
19:32but it did fix a lot of the things
19:35involving like the feeding of the fish.
19:37And I was able to actually like play
19:39this version for a bit. But then you
19:40start to notice some of the
19:44the Gemini-isms. And the best I can put
19:46it is like
19:47it doesn't have big model smell. Big
19:50model smell is really hard to describe.
19:52It's a a type of thoughtfulness and
19:54generalization that models do when they
19:56have like a better understanding of the
19:58world that comes from the model being
20:01bigger. I talked about this a bit in
20:03terms of 3D in my previous video about
20:06Muse, where I talked about how the
20:08model's 3D capabilities don't reflect
20:10how smart or capable the model is. 3D is
20:12a thing that they train in separately,
20:14like they add it on.
20:16This model clearly got more time with
20:18that 3D stuff, but it also has some
20:21stupid mistakes that I haven't seen
20:23other models do. Monsters spawn every
20:25minute. When you finish the wave, they
20:27drop a power up, and the power up lasts
20:30for 30 seconds. Why is there a power up
20:33that helps with my attack? I can't shoot
20:36anything after I get it, and they expire
20:38before I can use it. So, one of the
20:40power ups is legitimately useless
20:42because there will never be a monster in
20:44the tank when you have it on because it
20:46will disappear before the monsters come
20:48in. It's just like the type of silly
20:50thing that a human would not have done,
20:53that a smart model also would not have
20:55done, that a model that's goal is to
20:58just write code that seems good without
21:00thinking about it would absolutely do.
21:02Imagine that you were a part-time
21:04contributor to a game like this, and
21:06there was three power-ups made, you
21:07might throw out the idea of a power-up
21:09that does more damage and not even think
21:11about it and then add it. A smart model
21:13should have a good enough understanding
21:14of the system to know that that is
21:15stupid. Gemini doesn't. I think now is a
21:17better time than ever to start talking
21:19about the way this model behaves in
21:21real-world tasks. In particular,
21:24hallucination. The thing I just showed
21:26you wasn't hallucination. It's not like
21:27the model imagined this thing that
21:29wasn't the case. It saw a power-up
21:31system, it came up with an idea for the
21:33power-up system, and it put it in the
21:35game. It might be a stupid power-up,
21:36doesn't make sense, but it didn't
21:37hallucinate anything. According to
21:39Artificial Analysis, the model actually
21:41doesn't hallucinate much. On their AI
21:44Omniscience benchmark, which is their
21:45attempt to measure hallucination, Gemini
21:474 Argon had a 15% hallucination rate,
21:50the lowest of any model scoring over a
21:5245 on the intelligence index. And the S
21:54and D, you can see here in the official
21:56charts, Gemini 4 Argon has an
21:58impressively low hallucination rate. But
22:01the AI Omniscience bench is not just
22:03hallucinations. It's both incorrect
22:05answers as well as partial answers and
22:07refusals. So, if it gets the questions
22:10correctly, it gets points. If it gets
22:11them incorrectly, it loses points. If it
22:13says, "I can't answer this," it's
22:15neutral. When you look at the overall
22:17score, it's no longer first place, it's
22:19now in fourth because it gets few enough
22:22things right that its refusal to get
22:24things wrong does not really matter as
22:26much. It's just like, "Sorry, can't
22:28help." more often than others. Cool.
22:30Nice to see Google making their models
22:32hallucinate less. Still not a
22:34particularly good score. Sadly, from my
22:37experience, this benchmark is not that
22:40realistic because I have seen this model
22:43just be wrong a lot. Here is where we
22:46get into the real code work and where my
22:50actual breakdown of the model begins. I
22:52started by throwing it at TS Rust, a
22:55project I built that I talked about in
22:56other videos where I rewrote the entire
22:58chess compiler in Rust using Opus 5.5. I
23:01had a bunch of different models audit
23:03the code base and describe it, things
23:05that could be improved, what is it worth
23:07knowing in this code base. I would take
23:09all the claims it made, validate them,
23:11and then put them through a panel of
23:12judges to rate how good or bad the model
23:15did. GPT-61 Soul came out on the top
23:17making 38 claims that were all true. It
23:19had zero wrong claims, and the claims
23:21were informative, accurate, and useful.
23:23So, all the judges gave it a pretty good
23:25score, averaged an 8.84. Opus 5.5
23:27slightly lower, 8.78. It took way
23:30longer, 49 minutes versus 28 minutes,
23:32and it cost $24 instead of $4, and it
23:35got one of the claims wrong. Still a
23:37very good score, just made some mistakes
23:39and took too long and was too expensive.
23:406 Astra also had no wrong claims, was
23:44way faster at 20 minutes, still a bit
23:45too expensive. 61 Soul on X High, things
23:48improved. Also, funny, uh 6 Astra on Max
23:51scored worse than 6 Astra on X High.
23:54Once again, as I've been saying for a
23:56while, you should probably avoid the max
23:57reasoning levels. They do more damage
23:59than good. You might notice something's
24:01missing, though. Where is Gemini?
24:03Well, we hadn't scrolled down far
24:05enough.
24:07Yeah. Gemini for Argon, in my audit the
24:10code base bench, scored a 4.38,
24:13less than half of Opus's score and 61
24:16Soul's score. It also took 43 minutes
24:19long, almost double the time 61 Soul
24:22took, and more than double it than it
24:24took on X High. On the bright side, it
24:26would have only cost $3, but on the less
24:28bright side, of its 36 claims, 24 were
24:31false. The majority of the things it
24:34claimed about the code base were wrong.
24:36Do you understand how egregious that is?
24:39Duh. It means that when this model's
24:41doing real work in a real code base,
24:43it's as likely, if not more likely, to
24:45be wrong than right. And this has lined
24:47up with my experience using it. I I hate
24:50to be that guy, especially cuz I've
24:52likely burned some credibility with a
24:54lot of y'all with my Muse Wonder video.
24:56I don't care. Muse Wonder is a unique
24:58balance here where it is getting similar
25:00scores to Fable, way faster, and gets
25:03things wrong, but eight out of 33 is
25:05comparable to Fable 5.1's 11 of 33, and
25:08a whole different world than 24 of 36.
25:12I'm going to be so real, and we'll
25:13you'll see what I mean as we go deeper.
25:15I don't think anybody who finds this
25:17model on the same frontier level
25:19capability as things like Opus 5.5,
25:21Fable 5.1, 6 Astra, and 6.1 Soul, or
25:24even Muse Wonder. I don't think anyone
25:25who finds these models comparable is
25:27actually working on that real of stuff.
25:29Show me demos if you disagree. I would
25:31like to be wrong here. I want to have
25:33more faith in Google as a competitor,
25:35but the numbers aren't going to get
25:37better as we keep going. The next round
25:39was one where I audited a real bug in T3
25:42code related to how the bulk remote
25:44update workflows were functioning. Opus
25:465.5 got the best score with an 822. It
25:49had five claims that were wrong of its
25:5046. It was $26 to run, and took 20
25:53minutes, but these are like good, useful
25:55numbers. 6.1 Soul got a lower score. It
25:58missed some of the high severity things
26:00Opus 5.5 found, but it didn't say
26:02anything incorrect. It took 10 minutes,
26:03and it cost a dollar. Insane. This is
26:05why I like 6.1 Soul so much as a review
26:07model. Muse Wonder scored quite a bit
26:09lower. I think it missed a handful of
26:10claims, and it still had the same
26:12roughly error rate as Opus did, but it
26:14only took 6 minutes, and most
26:16importantly, in my opinion, it was super
26:18readable. So again, I like Muse Wonder
26:20cuz it's a different flavor, and finds
26:22useful stuff. Gemini for Argon is
26:24another flavor. It's not just a Claude
26:26distill like a lot of the Open Weight
26:28Labs are doing. So like, if you talk to
26:30something like GLM 5.3 Flash, Kimmy K3,
26:32or any of those other models, they all
26:34talk and act relatively similarly. All
26:36four of these models behave entirely
26:37differently. Gemini is too dumb to
26:40justify the flavor change. Muse Wonder
26:43Full is probably too expensive to
26:44justify it. We don't know cuz again, we
26:45don't have the price. But at least it's
26:47not lying 14 out of 40 times. Yeah. Muse
26:52has taken whatever I was hoping Gemini
26:54would do.
26:56Hopefully you're starting to see why I
26:57was nicer to Muse than people expected.
26:59Testing both the Muse model and the
27:01Gemini model at the same time really
27:03deeply reminded me of how much I don't
27:05like working with Gemini. Sadly, it gets
27:08worse. I didn't do a fancy visual for
27:10this one. This is an audit of my cloud
27:12product Lakebed. 61 Soul got a nine out
27:15of 10. Sonic 55 got an 8 and 1/2 out of
27:1710, mostly cuz Sonic 5 on Mac's will
27:20just burn and burn and burn. So,
27:22it finds everything. Muse took a pretty
27:25big hit here going from the 825 from
27:27Opus 55 down to 5.75. It's not as big of
27:30a hit as Gemini gets, 2.25.
27:33These are scores out of 10. Muse gave
27:35its findings a three. Opus, Fable, and
27:37Astra gave its findings a two. Of its 78
27:41findings, 56 were wrong. That is a 70%
27:45failure [snorts] rate. This model is
27:48more likely to gaslight your agents than
27:50it is to give useful feedback in
27:52reviews. If you do decide to use this
27:55model for real work for some reason, for
27:57the love of God, please put the things
28:00it does through Soul or Opus or some
28:03model that doesn't have this egregious
28:06rate of lying. I can't fathom how any
28:09bench measured this model is not
28:11hallucinating a lot when I don't I've
28:13never seen it not hallucinate in my
28:15day-to-day real-world code work. In my
28:17final run where I had it go through the
28:19T3 code architecture, it got a 3.95 out
28:22of 10.
28:24And you can also see again Muse
28:25wonderful being not like as good if not
28:27better than things like Fable and Astra,
28:29but neck and neck with Astra while also
28:32being much more pleasant to work with is
28:34worth noting. Again, to compare to Muse,
28:36it got 29 true claims, eight partially
28:39true, four false. Argon got 31 true, 14
28:44partially true, 47 false. Gemini for
28:47Argon had more false findings than Muse
28:50wonderful had true findings. What the
28:53It's not just like kind of bad or
28:57a little more likely to do something
28:58wrong. No, it's like an order of
28:59magnitude gap here. It is lying 10 times
29:03more than any of the other models I'm
29:05benching other than Muse wonder the
29:06non-full version. That one's just
29:07garbage. Throw it out. Don't touch it.
29:09But like the only bad models in this
29:11bench were Muse wonder standard and
29:13Gemini 4 Argon and they're both garbage.
29:16As you saw in my Muse video, I'm a
29:18little bit more forgiving if the model's
29:20pleasant to interact with. No points
29:22here either, Google.
29:24Walls upon walls of text full of things
29:27that don't really matter that much. Not
29:31very pleasant to read. But if the code
29:33works, that's fine then, right? Well, I
29:35had it do some work around the SQL light
29:38connection layer in T3 code in hopes of
29:40stabilizing it for heavy users on
29:42platforms like Windows where we're
29:43having some connectivity issues. I
29:45already knew a lot of the fixes. I was
29:46just curious if Gemini could find them.
29:49So, there's lots of users reporting
29:50issues with instability in connections.
29:51A lot appear to be SQL light related.
29:53What should we prioritize to fix this
29:55issue? I even gave it a hint with async
29:56SQL light. This is not the right
29:58solution. The right solution is pooling
30:00and having read pools at a single right.
30:02It did a hunt. It did a lot of the
30:05classic Gemini-isms of just reading
30:08every single file synchronously. It just
30:13kind of went on and on with that.
30:17It does correctly identify the split
30:18read pool and a single right connection
30:20piece.
30:22I tell it to make a PR for priority one,
30:24which is the first section it had in
30:26here.
30:27Uh 30 minutes later, way longer than it
30:30should have taken, I've been told by
30:32some people that apparently Gemini is
30:34way slower in T3 code. You might think
30:37that's our fault. It literally cannot
30:38be. We're using their ACP bindings that
30:41they provide on their servers that we
30:42embed in our app. We actually did a
30:44bunch of weird to make it work
30:46at all because Google's a special
30:48snowflake. We'll talk more about the
30:49Google special snowflake
30:50Regardless, 30 minutes is an
30:52unreasonable amount of time to make this
30:54PR which when we open, we will see
30:58is closed.
30:59Because this change should not have been
31:011,200 lines of code. This change
31:03should have been 30. At this point, I
31:05was done. Especially when I read the PR
31:07which by the way, we have skills in the
31:09repo for how to format the PR that are
31:11really simple. It's supposed to open
31:13with a simple two to three sentence
31:15description of the exact problem the
31:17user had that they're trying to fix. We
31:19don't get that. We get a summary. We
31:20split the SQLite read accesses and write
31:22accesses in this particular file in the
31:24orchestration projection and query
31:25services that read snapshots and
31:27standalone select queries. Is this a
31:29run-on sentence? It is. It's unreadable
31:32slop that didn't follow any of the repo
31:35rules. Just garbage. So, I was already
31:37pretty ready to close it. Then I saw how
31:38big it was. Then I saw my favorite
31:41thing. CI failed. It just didn't write
31:45the code right. The check and the lint
31:47both failed and it never bothered to fix
31:49it or take care of any of the high or
31:52higher level comments that it got from
31:55our review bots. So, I gave up. I opened
31:58up Opus 5.5. I pasted the thread ID in
32:02the PR and said, "This is opened by
32:03Gemini for Argon. I think the PR is
32:05garbage and doesn't really fix the
32:06problems that it barely stated. Go
32:08through the T3 code thread, figure out
32:10where it went wrong, give me your honest
32:11thoughts on the work that Gemini did.
32:12You agree it's slop, close the PR." I
32:14agree it's slop and I closed it with a
32:16comment that gives the reasons. Where it
32:17went wrong. Turn one, the diagnosis was
32:19mostly correct, but it had no evidence.
32:21Gemini found the real mechanism, the
32:23synchronous database sync runs on the
32:24main thread. Yada yada yada.
32:27Turn two, it fixed the wrong half of its
32:29own diagnosis. The reader pool still
32:31opens database sync on the main thread.
32:33A slow read on a reader connection
32:35blocks heartbeats the same way it does
32:36on the writer. The only effect is less
32:39waiting on this one semaphore call and
32:41that matters only when a fiber yields
32:43inside a transaction.
32:44What it's saying is it didn't work. It
32:46also called out that it didn't check
32:47prior art. There was already two PRs
32:50that fixed these problems. You might
32:52remember the previous video with Muse
32:54where I pointed out how much I liked
32:55that Muse said, "I'm not going to write
32:57code. Here's a PR that fixes it. You
32:58should merge this instead." This model
33:00doesn't do that. In fact, this model
33:02will hallucinate its own incorrect
33:03solution instead. The correct answer was
33:05in the PR tab on our GitHub and not only
33:08did it not find that, it invented its
33:10own wrong answer. It also solved it with
33:12a regex to route requests. I wish I was
33:16kidding. It's so bad. It was so garbage.
33:19According to Opus, Gemini can read code
33:21and explain how it works. It does not
33:23test its own theory. It never asked
33:25whether a reader pool on the main thread
33:27stops the event loop from blocking. It
33:30answered your loose question with a
33:32confident roadmap and a percentage that
33:34it invented. When I told it to do
33:35priority one, it built that literally
33:37and at maximum size without checking
33:39whether priority one was even correct.
33:41So, I told it to go make the changes and
33:43it was 21 lines of code.
33:45They are now merged. They'll be in the
33:46next nightly and people have a better
33:48experience with SQLite. Real quick crash
33:50out about the instability of this model
33:52and the anti-gravity stuff. If I get my
33:55Anthropic account banned, that sucks,
33:57but I can open a new one with a new
33:58email address and I don't lose too much.
34:00If my OpenAI account gets banned,
34:01slightly more annoying, but again, I
34:03open a new one, I'm good. I might sign
34:04in to both of those with my Google
34:05account, but if I'm banned, it's just my
34:08sign-in to Anthropic or OpenAI that's
34:09banned. If my Gemini account gets
34:11banned, I might lose my YouTube channel
34:13cuz my Gemini account is my Google
34:14account. So, I could lose my email, I
34:16could lose my billing system for my
34:18credit cards, I could lose my saved
34:20passwords in Chrome, I could lose my
34:22access to my YouTube channel, I will
34:24lose my access to everything else I sign
34:25in to with Google. There's a real risk.
34:27They have promised that they will only
34:29ban you from Gemini and anti-gravity.
34:31They won't ban you from Google anymore.
34:33But, I've seen people getting their
34:34Google accounts banned still, so it's a
34:36risk.
34:37As such, integrating their official ECP
34:40into T3 code felt risky. We chose to do
34:43it because I was told it should be fine,
34:45but I can also tell they're not putting
34:46a whole lot of effort into this, which
34:48you can see in places like here, where
34:50the model just failed outright to
34:52connect to the gateway in the middle of
34:54a run after reading endless files for an
34:57hour and 2 minutes.
35:00And then it just failed and refused to
35:02continue, and I had to restart this
35:04entire thread with none of its existing
35:06history. Horrible. And then there's a
35:08bunch of silly little Googleisms, like
35:11they don't report the model and then let
35:13you pick effort in the ECP. All of the
35:16model slugs are paired with the effort
35:20levels. So, you have to pick for Argon
35:22high, medium, or low as a model. Every
35:26other lab gets these things right. ECPs
35:30are pretty simple standard to implement
35:32and integrate. Google is Google, and
35:35they got all of this wrong. It is
35:37embarrassing how miserable it was to
35:39integrate anti-gravity and how high a
35:41percentage of the bugs in T3 code are
35:44just for anti-gravity. It has
35:45consistently been a thorn in my side
35:47since we first integrated it, and even
35:49having Gemini 4 did not make it better.
35:52It was a relief to have a reasonably
35:54usable UI for a Gemini model for the
35:56first time ever. I honestly thought it
35:58would make me like it more. But, now
35:59that I see it next to models that are
36:01competent, like Muse, Opus, even like
36:04Sonnet, honestly,
36:05it just hurts more. And now we get to
36:08the end, comparing it to the other
36:10models. This feels unfair.
36:12Feels unfair to the other models. This
36:14is just garbage. Don't ask me about how
36:16good the usage is on the $200 plan or
36:17anything. I don't care. I'm not going to
36:19push the limits enough to know.
36:21It's probably practically unlimited
36:22because you're not going to do a whole
36:23lot with it cuz you're going to get
36:24frustrated and give up. It's just It's
36:26just bad. I'm going to put my money
36:28where my mouth is here and show you just
36:29how committed I am to this. I don't
36:31trust any Gemini models nor Muse Wonder.
36:38To be clear, Muse Wonder full is fine,
36:41but the standard Muse Wonder as well as
36:43Gemini 3, 4, etc. They do not contribute
36:48useful code the T3 code code base. I
36:51would like to find some way to auto
36:53close all pull requests that are filed
36:56using models like Gemini 4 Argon or the
37:00normal Muse Wonder
37:02when they are filed to T3 code. I am not
37:05joking.
37:06I will find a way to get this in later
37:07this week. I have wasted more time in
37:10tokens trying to vet code that was
37:13written by Gemini 4 than I would have
37:15spent writing the code with a smarter
37:16model in the first place. Gemini is not
37:18a useful model for anything in the code
37:20world other than maybe as like a first
37:22or second pass on a nice looking
37:24marketing page. As soon as I'm done with
37:26this video, I'm just not going to use
37:28Gemini for anything other than like
37:29fetching data for YouTube. This is one
37:31of the worst releases of the year. I
37:33don't care what the benchmarks say. I
37:35have been testing it constantly in real
37:37world work and I have not merged a thing
37:39that this model has shipped. This feels
37:41like a throwback to 2024 with some of
37:44the hallucinations I see this model
37:46making. I was played up to this point
37:48because I'm thankful for the Google
37:49employees I've talked to. They have
37:51generally been one of the more receptive
37:52labs to my feedback and I tried to share
37:55this feedback with them early. I've also
37:56heard rumors that there is a better
37:58snapshot like carbon or something that's
37:59coming soon. I don't know how much
38:01better it's going to be, but it needs to
38:02be a lot because this model is garbage
38:04and I have no intention to ever use it
38:06again other than for fetching data on
38:08YouTube. Tell me what y'all think. Am I
38:10overreacting massively or is this model
38:12actually just garbage? They told me it
38:13was good at code and it's not good at
38:15code, so I'm I'm pretty done with it.
38:17Maybe you can find some cool uses. Let
38:19me know in the comments if you do. Until
38:21next time,
38:22I'm sticking with Muse.
38:24Okay, seriously, I'm probably just going
38:25to stick with Opus 5.5. You get the idea
38:26though. I just I'm out. Bye.

About this transcript

This page contains the full transcript of So I've been testing Gemini 4... by Theo - t3․gg, generated from the public captions YouTube serves with the video. The transcript has 7,925 words across 1,129 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.