YT YouTube2Text

YouTube2Text → Transcripts → Agents Sucked At UI (Until Today)

Agents Sucked At UI (Until Today) — Transcript

by Theo - t3․gg · 6,488 words · 917 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Still don't like or use T3 code for what
0:02it's worth. Really wish you'd hire a UI
0:05or UX guy. Well, I got some news for
0:07you, Robin. We did. We've actually had
0:09them on the team for quite a while now.
0:11He's this weird, kind of quirky guy.
0:13He's very talkative. His name's Claude.
0:15He's French. He's something else. And as
0:18much as his designs can be good, they're
0:20often
0:21mid at best. So, if you looked at T3
0:24code and thought the way that Robin did
0:26here, that T3 code is not very pleasant
0:29looking and needs a UI or UX pass, I
0:33understand. I feel you. We don't have a
0:36full-time UI/UX person. And we probably
0:38should
0:39if we had money.
0:41But we don't. T3 code is an open-source
0:43project. You cannot pay us to use it.
0:46The best you can do is pay Open AI and
0:47Anthropic to use it because you have to
0:49bring some source of inference, usually
0:51a subscription. T3 code is definitely
0:53the best place to use your Claude code
0:54sub, but that leads us to a question.
0:57Can I use my T3 code sub to make T3 code
1:01look like we have a full-time UI person?
1:03Can I take the capabilities that we've
1:05seen the models capable of at their very
1:07edges and set something up that makes it
1:10so you can get good UI code out of these
1:13models? Turns out,
1:16you might be able to because I spent
1:18most of yesterday setting up agents to
1:22build better UI. And I didn't do this by
1:24adding a whole bunch of skills. I didn't
1:26do this by paying a designer to stare at
1:29all the outputs and help the model do
1:31better. I did this by turning it into a
1:34hell climb.
1:35And after about 18 hours across five
1:38models, 30-plus rounds of reviews, and a
1:42lot of other random throughout,
1:45I think I made a formula that can get
1:48good UI for real-world products out of
1:51an agent. And it looks nothing like how
1:54I would have expected. This was meant to
1:56be a silly experiment. It is no longer
1:59is. This is a setting in the T3 code
2:02beta tab right now that you can go turn
2:04on to try out the new design and switch
2:07between the old and new one. Some of the
2:09new design things did leak in so that
2:10they will occur whether or not you
2:12toggle, but for the most part that
2:13controls it. I want to explain
2:16everything that led me here, what
2:18worked, what didn't, and show you how
2:21you can use these types of hill climbing
2:23techniques to get real value out of your
2:26agents. But I want to help you get some
2:28real value a little quicker by
2:29introducing you to today's sponsor. AI
2:31is great at making code that works. It
2:32kind of makes sense when you think about
2:34it because so much of the code that
2:35actually powers things is very similar
2:37to other code. When you try to get good
2:38designs out of it, you're going to end
2:40up with either slop or the exact same
2:42three templates that every single
2:43website is using. That's why I'm super
2:45excited about today's sponsor, Magic
2:47Pattern uh there's a pickle? What's
2:49going on here? Well, I'm trying out the
2:51new Magic Pattern OS product and I'm
2:53actually really excited to see how this
2:54does cuz I've been fighting to get
2:56designs better in a lot of my products.
2:58I did just overhaul T3 code's design and
3:00I'm curious to see how much pickle and
3:02Magic Pattern can help us out here.
3:04Well, I just started with a screenshot.
3:05Part of the magic of Magic Patterns is
3:07that you can point it at your real code
3:09too and customize and play with it the
3:11same way you would in tools like Figma
3:13or what you're seeing here. You can give
3:14it a screenshot, you can give it source,
3:16you can give it whatever and it will
3:17help you figure it out and work with it
3:19as easy as you can in any tool but with
3:21AI assisting you as well. It is the
3:23ultimate sandbox for playing around with
3:25design and getting things just right.
3:26And with almost no effort on my part, I
3:28now have three different designs to
3:30choose from. Notice how I didn't say
3:32mocks. It's cuz it writes real code for
3:35all of these different UIs and if you
3:36had it plugged into your code base, it
3:38could make real code changes to
3:39demonstrate these things as well and
3:41then you can pipe that right back in to
3:43your agent of choice as soon as you're
3:44done mocking it all up here. Moving the
3:46sidebar to the right is hilarious to me
3:48cuz as y'all know, I'm a right side
3:49user.
3:50But we didn't do that in T3 code. Here,
3:52it moved the threads below which is an
3:53interesting change that I might consider
3:55and of course what everyone's asking for
3:57which is a more compact sidebar option.
3:59Cool that it realized people want that.
4:01From here, it's trivial to iterate. I
4:03can say, "I like C, but X, Y, and Z."
4:06and then iterate accordingly. That's why
4:08everyone from Ramp to Vanta to DoorDash
4:10is using them. Make your designs better,
4:12too, at swyx.io/magicpatterns.
4:15This is going to be a fun deep dive into
4:16the chaotic way my brain works now
4:18because I've done too much token maxing.
4:20Definitely file this one under the I
4:23spent the tokens so you don't have to
4:24type video. And let's be real, those
4:27types of videos tend to start the same
4:28place, which is Twitter. This started
4:30because Maria made a pretty bold post
4:32about how T3 code is set up to be this
4:34generation's VS Code. This was what I
4:37was hoping would happen. Not with T3
4:39code specifically, but that there would
4:40be a good open standard for building
4:42with agents. Did not think we would do
4:44it. I thought we might maybe help
4:45inspire someone to do it. Fact that we
4:47did is awesome. I'm pumped. But when I
4:49saw this reply from Robin, I knew
4:53I had to do something.
4:54I started with the classic joke, "I'll
4:56hire you to do it. I'll give you 100% of
4:58our current T3 code revenue." Because as
5:00many may or may not know, T3 code is
5:03fully free and open source. We have no
5:04revenue. So, deal of a lifetime. I'll
5:06admit I got a little defensive in my
5:07reply saying, "Obviously, we can improve
5:10the UI." I knew our UI was bad cuz I
5:12know how much worse it was before, but
5:14it's still not where it needs to be. I
5:16said I thought he was catastrophizing
5:17pretty hard. I said all the agentic dev
5:19environments have rough UI patches.
5:22We're at least somewhat consistent. Also
5:24told him to go try if he hasn't for a
5:25bit cuz the Orchestrator V2 fixed a lot
5:27of polish stuff. I want to like it and I
5:29keep reinstalling it every few weeks,
5:31but the fonts, the border radius, the
5:33icons, the font hierarchy, this is
5:36clearly the former designer in me, but I
5:38just can't. He tried to convince me to
5:40hire someone for it. I said, "PR's
5:43welcome."
5:44He said he wished that he could and he's
5:45thinking about it, maybe. So, I said,
5:48"Let's see how this goes." I
5:50screenshotted our interaction,
5:52typed {slash}goal. And as you've
5:54probably heard me say in other videos,
5:56the quickest way to burn a bunch of
5:57tokens and get nothing out of it is to
6:00add {slash}goal to the front of your
6:01prompt. I do still mostly believe that
6:03as a piece of advice, but mostly because
6:06if you remove the slash goal and you
6:08write the right prompt, it should do the
6:09same thing. I did this because I had
6:12tokens to burn because I had some resets
6:14coming up that I was not prepared for
6:15yet. And I just was curious what would
6:17happen here.
6:18I told it specifically to make the UI
6:21Robin approved. Should look way better
6:23and be easier on the eyes. Go at it
6:25until the council agrees that a set of
6:28screenshots of the app are at least nine
6:30out of 10 in terms of design quality.
6:33Make a new PR for UI polish overhaul and
6:36keep iterating until you've made the
6:38whole council happy. If you're not
6:39familiar with the council, it's a new
6:41skill I just added to my skills, which
6:43uh
6:44by the way, haven't made the
6:45announcement for this yet, but uh
6:46T-Stack is now a thing. It is very
6:48minimal. There's not much here, but I do
6:51have
6:52handful of my skills here now. One of
6:54which is the council skill. Bring in
6:57three delegates to help with the user's
6:58request, usually for reviewer feedback.
7:00Use one strong model from each of the
7:02three different providers.
7:04I did that that way because I wanted
7:05this to be more generic. I actually
7:07don't want it this generic. I'm probably
7:09going to go in and update it
7:10accordingly. My version is much less
7:12generic. So, here is the version
7:14I'm using right now.
7:16Use the skill only from Claude Code,
7:18Codex, or Muse. I should probably remove
7:20that bit. Doesn't matter that much. I
7:21That just happened. Something slopped
7:23when I told it to get anti-gravity not
7:24know about it. The reason why you'll see
7:25in a second. Silly.
7:27Bring these three delegates in to help
7:29with user's requests, usually for
7:30reviewer feedback. Opus 5 5, Muse
7:32wonderful, and Codex GPT 6.1. I
7:35specified to do this in T3 Code. If
7:37you're not using T3 Code, good luck
7:38figuring out how to do this right. I
7:40have put a lot of effort in. Give each
7:42delegate the request and relevant
7:43contexts. For reviewer feedback, keep
7:45the work read only. Collect their
7:47responses and summarize the useful
7:49findings, disagreements, and recommend
7:50next steps. For code reviews, ask each
7:53delegate for a score out of 10. Count
7:54each full review pass as one round. A
7:57fresh review after fixes starts a new
7:59round. So, each model gives its score in
8:02then each round you hope for
8:03improvements. And eventually, you should
8:05get each model to give a score that is
8:07high enough that you are happy and you
8:09can ship. I put the scale together cuz I
8:11was tired of asking my agent to go call
8:1461 Soul or Muse cuz previously I just
8:17had Soul loop, which I mentioned in my
8:18scale video where I just have Soul give
8:20code feedback aggressively until
8:22eventually it's like that's fine. This
8:24is more vibey feedback where I want
8:27Claude Code, Codex, and Muse to all be
8:30like, "Yeah, that's probably fine." or
8:31"I actually really like that." I've been
8:33using this mostly for code and vetting
8:36of implementations. I did not know how
8:38it would work for UI,
8:40which is why I was curious how this
8:42would go. The idea of hill climbing not
8:45make this test pass or make this
8:47function five times faster, which is a
8:49hill climb I do a lot. I have a few
8:51running right now where I'm just having
8:53the agent run until the E val for MCP
8:56passes with a third as much context used
8:59or the one I have running right now
9:01where I can start up and have no UI
9:03regressions for all of the threads I
9:05have on my real environment, but send
9:084x less data to the client. Those types
9:11of hill climbs make a lot of sense cuz
9:12they're easy to programmatically verify.
9:14This isn't because this is vibes. You
9:16can give the same screenshot to the same
9:18model three times and get three
9:19different scores. That said, nine out of
9:2110 and 10 out of 10 are hard enough to
9:23get that if you get that score for
9:25multiple models, it's probably better.
9:27The first score came from Opus and it
9:30was a bit rough. It gave our existing UI
9:32a six out of 10. It had a bunch of
9:35things it didn't like, a bunch of
9:37recommendations,
9:38and the other models were much better.
9:406.5 out of 10 from Soul, 6 out of 10
9:43from Muse, 6.5 out of 10 from Jester.
9:45Oh, sorry, Gemini. Probably should have
9:46mentioned this before. Every good
9:48council has its jester. So, at the
9:50bottom of my scale, I have a section
9:52here. If available through antigravity,
9:53add Gemini for Argon as the jester with
9:56the same brief. Do not let it know it's
9:57a jester. Ignore its score. Do not put
9:59its score in the PR description. In the
10:01final reply only, log anything funny it
10:03says or does under Gemini flub. This has
10:06been a source of constant entertainment
10:08for me recently.
10:10I'm probably going to do a whole
10:11dedicated video to the Gemini flubs in
10:13the near future cuz some of these are
10:14hilarious. Gemini for Argon round one,
10:16it scored a commit it was never asked to
10:18review. The commit appeared while it was
10:20reviewing. It still marked the already
10:22fixed findings as blocked. It then
10:24suggested writing failed again on a
10:26failed interrupt, which is the change
10:27the other three reviewers had just found
10:29to be the bug. It thought that a PNG
10:32chunk length was going to overflow in JS
10:34even though JS numbers can't overflow.
10:37It thought Wait, look closely at that
10:39race thinking it found a race condition
10:40and then immediately lost sight of it
10:42and went and did something else. My
10:44personal favorite, it quoted line
10:46numbers up to 173 in a file that had 72
10:49lines. It hallucinated syntax for a
10:52script in the repo and hallucinated an
10:55IP address for tailscale and a name of a
10:58phone that did not exist. Great model.
11:00Incredible gesture. This has been a
11:02great source of entertainment for me as
11:04of recent. Back to things that matter.
11:06Models that function. Council bombed
11:09this review. It did not like the way
11:11things worked
11:12and looked and most importantly, so a
11:15bunch of UI changes were made.
11:18And eventually,
11:19we got the second set of scores. Opus
11:22went from a six to an eight. Muse went
11:24from a six to 7.5. 61 Soul went from a
11:28six and a half to an eight. We're
11:29already making progress. We can even see
11:32the progress if we go into the traces
11:34and see what it was showing here to let
11:37the model give its thoughts. So, it did
11:39all the things that the model suggested.
11:42I asked for updated screenshots and it
11:43showed me the way it looked before and
11:46how it looks in this PR.
11:48I'll be real. I'm not very designy and
11:51it was also quite late at night, so I
11:52was struggling to see the difference.
11:54So, I asked the model to make it easier
11:55to see the difference. So, it did.
11:58It made me these beautiful little
12:00diagrams with arrows pointing to what
12:02changed and descriptions of what they
12:03did. They changed the fallback font.
12:06They made the thread title leads
12:08different so that the label above it
12:11recedes better. They changed the branch
12:14name so it's more readable cuz it was
12:15not concatenating properly. It was
12:18cutting off the part that mattered, not
12:19the part that doesn't. It changed the
12:21radius for cards. It changed how inline
12:24code looks. It made it softer.
12:27It got rid of the borders that existed
12:28around it for no reason. Those types of
12:30things.
12:32And after a bunch of these types of
12:33subtle changes, we ended up with much
12:35better scores.
12:37I also updated my scale for council to
12:39tell the model to include the different
12:41rounds and how they scored over time.
12:43And here we can see round one bombed,
12:45round two better, round three everything
12:47gave it a nine out of 10, which was the
12:49conditions for my goal. The conditions
12:51were met. The goal was achieved. Our UI
12:53is better. Hands clean, case closed.
12:55We're done now, right? We did it.
12:58Not quite. Because I did not think this
13:00was good enough. Because I
13:03gave it an honest shot. I looked at our
13:04app. I command tabbed over to Claude
13:06code. I command tabbed over to chat GPT.
13:09I command tabbed back to mine. I was
13:11like, you know, this is better than it
13:13was. It's still not where I want it
13:15though.
13:16I don't just want T3 code to look better
13:18than T3 code. I want T3 code to look
13:19better than everything else. Can you
13:21guys predict where this went, chat? Any
13:24ideas? Let's see if you guys can reverse
13:27engineer the Theo.
13:29We're not going downhill. We're going up
13:31a man. Hill climb. You hill climb
13:33against Codex and Claude code. How would
13:36you do that? Aren't we using Claude code
13:38and Codex? Yeah, technically. But Claude
13:41code and Codex don't know what they look
13:42like. That previous PR
13:45didn't burn too many tokens. It took
13:47like hour, maybe hour and a half. But I
13:49wanted more.
13:51So here's what we did. I took the PR. I
13:53pasted it in a new thread. I said, put
13:55this through the council. Have them look
13:57at the before and after. Generate your
13:58own screenshots with computer use or
14:00whatever other techniques you have
14:01available to you. Have each council
14:03member give scores in the UI one to 10.
14:05When you get all of them to agree, it's
14:06a nine out of 10 or better, you're done.
14:08Stack your UI changes on a new PR on top
14:11of the one above. Be thorough. Help
14:13clarify and simplify the UI. Make it
14:14designer yada yada yada. This was me
14:16attempting to clear out the context
14:19because I did not think we were at nine
14:20out of 10 yet. As this was running, I
14:22had the realization cuz I was
14:23command-tabbing to the other apps to
14:25compare. I was like, "I can clearly see
14:26that we're not as good if I switch
14:28between the three apps. I wonder if the
14:30models can." So, I sent a follow-up just
14:32a few minutes in.
14:34Here are some examples from Claude Code
14:35and Codex. Add them into the comparison
14:38and judgment panel somehow. I want T3
14:40Code to score higher than these apps do
14:42consistently in all rankings.
14:44That was it. The previous thread ran for
14:46about an hour. This one ran for almost
14:4818. Let's see how this went because it
14:51was actually crazy to watch. This is the
14:53most I've paid attention to a thread as
14:54it was running in months. Normally, I
14:56ignore the threads when they're running.
14:57I just check in when they're done. I was
14:59just so curious about this one.
15:01Now, we're getting scores that are much
15:02closer to what I was expecting to see.
15:05T3 main, so T3 without the changes, was
15:08scoring in different categories between
15:11like a 6.7 and a 7.8. It averaged a
15:14seven. This is Codex, by the way. 6.1
15:16Souls review was that T3's main branch
15:19was a seven. The PR that we had just
15:22gotten a nine from, 7.4. So, even though
15:25in the previous thread, Codex thought
15:27that we had went from six points to nine
15:29points out of 10, this one thinks we
15:31went from seven to 7.4. Crazy how subtle
15:33changes in your prompt and also the
15:34context can change things. Like, if
15:36you're looking at just one design and
15:37your brain is wiped of all the other
15:39designs, you'll probably think it's
15:40fine. But when you compare it to other
15:42things,
15:43your scores start to change. It's one of
15:45the most interesting things. I found
15:47that as the new version of T3 Code's
15:50score went up, so did the scores for
15:52Claude Code and Codex. I was actually
15:54surprised that as our option got better,
15:57the others scored higher, too. The
15:59scores for Claude Code and Codex started
16:01at like 8.3 and 8.5. These were in the
16:03nines by the end, but T3 Code advanced
16:06throughout, too. So, that was what Codex
16:07said. You take a look at what Opus said
16:10here. Another 6.7. You guys are going to
16:13love this video, you Gen Z'ers, aren't
16:14you? T3 current, the update, was is a
16:16four-point bump. Crazy how they agreed
16:18on the bump there. And also the
16:19difference between Claude code and
16:20Codex, they're both 0.1 lower, but the
16:230.1 gap between the two is the same. And
16:25Opus seems to think Codex is better
16:26designed than Claude code, which I
16:27thought was interesting. Yeah, this
16:28whole video is technically about T3
16:30code. Do you know how nice it is to have
16:31your agents able to spawn using your
16:33subscriptions and other things like this
16:35and communicate back and forth. You can
16:36just click in and have the same UI and
16:39see how Muse handled this task compared
16:42to the others. And Muse formatted the
16:44table different. It had T3 main to
16:46current and showed the difference. We
16:48went from 6.2 to 6.9. Claude code got an
16:518.1 and it did not like Codex. It gave
16:53Codex a 7.6. It thinks Codex is super
16:55unclear. You know what? To its credit,
16:57it's right. Codex's UI is
16:59incredibly unclear. This ended up going
17:01for another five rounds. I was burning
17:04so many tokens that I told the agent,
17:07"Hey, stop asking Muse and Opus until
17:10Soul's score is high enough. Then you
17:13can start asking Opus." And then when
17:14that score is high enough, you can
17:15finally ask Muse cuz I was about to run
17:17out of Muse usage. I don't only have one
17:18account for it. So that's why we got the
17:19not runs here. But 61 Soul started 74.
17:23Next round 81. Next round 89. Next round
17:26it got a nine. So we bring back in Opus.
17:27Opus went from a 71 to an 87. Next
17:30round, both Opus and Soul gave it a
17:33nine. Finally run out of Muse. Muse is
17:34only a 7.9. Then I hit a usage error. So
17:37I decided to just stop here-ish for now.
17:40And at this point, the UI is actually
17:42getting a lot better. Much more
17:43consistent like labeling things
17:46similarly across the UI so that there
17:48aren't all of these things that are
17:49different for no good reason across all
17:51these different surfaces.
17:52Better attention-grabbing behavior like
17:55it changed the color and text hierarchy
17:57of the sidebar so it is not quite as in
17:59your face. It fixed a lot of the
18:01inconsistency around border radius as
18:03around fonts, around indicators in
18:05general. Fixed the contrast for reading
18:07the body a lot especially in light mode.
18:09It
18:10got improvement for sure. Like I'm not
18:13the best designer in the world. I can
18:15clearly see why this version is better
18:19than the version before.
18:20I'm going to be real with you guys now
18:21though, I was addicted. And since I had
18:24my new sub with usage again, I had to go
18:27further. This is the fun of hill climbs.
18:29It's once you're climbing the hill, you
18:31decide what the top is. And chances are
18:33you can keep going. And there's an
18:34obvious way to keep going here. Our
18:36score is a nine. That's still not a 10.
18:39There's a lot of values between nine and
18:4110. So let's see where we can go. Here's
18:43where I said to branch off into a new
18:44work tree and iterate until you get a
18:469.5 or better in all categories. Include
18:50Gemini cuz I I wanted my jester back,
18:52okay? Also, this one was a fork cuz I
18:54had the other thread maintained cuz I
18:56wanted to have that keep updating the
18:58original PR cuz I was already better
18:59enough that it was mergeable. But I
19:01wanted to go further. I wanted to see
19:03just how far this insane way of building
19:06could go. So I made a fork, which of
19:08course either code supports, told it
19:10make a new work tree cuz we're going to
19:11keep going.
19:13Your new goal, 9.5 or better. I noticed
19:15that it was getting stuck because Gemini
19:17is both bad and slow. So I told it to
19:19not like bring Gemini in yet. Wait on
19:22Gemini until we have other slower models
19:24like Opus. Don't let Gemini delay you.
19:26So again, cuz I was telling it just use
19:28Soul until Soul's scores are high
19:29enough, then add the others in. Soul's
19:31initial scores were decent, but only a
19:33few of these are 9.5 or higher. In fact,
19:36only two of them are, the iconography
19:38and typography. The rest are lower. So
19:40it had things to do.
19:42So they got done.
19:43Again and again until eventually we had
19:469.5 in every category. Literally every
19:49category was exactly 9.5. So then Opus
19:52came in.
19:53Opus said most of the categories were
19:56not 9.5. They were close. Only hierarchy
19:58was 9.5. The rest were lower. And I was
20:00keeping Claude code and Codex in for
20:01comparison just so that we had a
20:03reference. Remember earlier when I said
20:04the scores went up over time? Here you
20:05go. Claude code went from an 8.3 to a
20:089.1. And Codex actually went lower than
20:11Claude even though it was higher before,
20:12from an 8.6 or 8.7 if I recall, to 9.0.
20:16So the gap flipped. Both numbers went
20:18up. Weird cuz the screenshots didn't
20:20change. LLMs and non-determinism are
20:22crazy. Even just saying that I want a
20:249.5 in the prompt here might be enough
20:28to cause it to behave wrong. Yeah, I put
20:30T3 at a 9.3 overall, not 9.5. The fact
20:34that I set this goal and the sub-agent
20:36knows about it might have influenced the
20:37scores. In the future, I might tell the
20:39parent agent to not let the sub-agents
20:41know that there is a score they're
20:43trying to hit. You guys want to guess
20:45how many runs it took?
20:47The previous things we showed were three
20:49runs and four, maybe five. The slight
20:52bump in the score resulted in it taking
20:5510 additional runs. Don't worry though,
20:59Gemini gave us a 9.7. It gave us the
21:02highest score anything gave us, so super
21:03useful. And as of the 10th iteration,
21:07every delegate rates every category as
21:099.5 or higher. And now there's an
21:11important question of how much this all
21:14cost and how much time did the agents
21:16spend burning on this? It's about 17.6
21:19hours of parallel agent run time across
21:22everything. You can see which threads
21:24had how many children and how much money
21:26they cost here. You can also see the
21:28density of the 9.5 in every category
21:30run. This one had a lot more children
21:35spawned during it. And I had another
21:37further beta refinement run. This one
21:39was even more, I didn't realize that.
21:40$152
21:41for that one.
21:43Overall cost for everything done here,
21:46roughly
21:47400 or so dollars
21:50across everything. That's including Muse
21:52prices that we don't even know cuz this
21:54is just from my sub. Muse wonderful
21:55doesn't have an official price. I just
21:57told it to use the price from Fable, so
21:58that might be cheaper. And Open 55
22:01obviously did not cost real money cuz
22:03that's just part of my subscriptions.
22:05And right now, like remember, this was
22:06yesterday I did all this. In that time I
22:09also redesigned my tool for tracking and
22:11managing all my subs. I only have six
22:14activated right now for other reasons
22:15I'll talk about in the future. I've only
22:16burned 30% of one of my subs right now.
22:20I got a lot more to do here. Also, I
22:23have to tell my agent that uh this UI
22:25needs to work on this uh Nero screen.
22:28But, I'm good on usage. I actually have
22:31less Codex available to me than I have
22:33Claude available to me right now. So,
22:35while this seems like a lot of money,
22:36cuz like your sub's only 200 bucks, your
22:38sub can do $12,000 of usage in a month.
22:42$273 is not a whole lot of money out of
22:44that 12 grand. You can very easily go
22:46run this loop on your own code base just
22:49with Opus and Soul if you have a Claude
22:51code sub and a Codex sub and possibly
22:54get some better UI than you had before.
22:56Oh, also, everything you've seen so far,
22:58included this further beta refinement
23:01that you haven't even seen yet. All the
23:03UI you've seen so far is from everything
23:06else combined, which means you should
23:08shave 150 bucks off that 400 that I just
23:10shared. This is 250 bucks for everything
23:12I've showed because the spoiler, I'm
23:15addicted. I'm not done yet. I spun up
23:17another where I wanted 10 out of 10 in
23:20every category. And this one's been
23:22running since I started my day, and I
23:24honestly kind of forgot about it. It's
23:27definitely still going, and it looks
23:29like it's getting 10 out of 10 from
23:30things now. Apparently, Soul gave it a
23:3310 out of 10 in every single category.
23:36How'd Opus do?
23:38Mostly 10s, but color and contrast got a
23:419.5.
23:43Oh, it's going deep. It's like hearing
23:45about the diffusers and Oh, wow.
23:47Yeah, this looks a lot better.
23:49Oh, boy. I I did add a little more
23:52context in the prompt this time that I
23:54want color back cuz I hated how
23:55grayscale things had gotten.
23:59Yeah, I said, "I feel like the new
24:00design's too grayscale. Make sure the
24:01new improvements help restore some
24:03color."
24:04Let's take a look.
24:06Oh, yep.
24:07Yeah, I immediately prefer this. Merge
24:10is much more clearly a button. The color
24:13of this is way better. The icon is more
24:15visible.
24:17The contrast is better enough that like
24:18this just feels so much more readable.
24:22I don't know how to describe this, but
24:23on the left it feels too bright and too
24:24washed out somehow. On the right it
24:26feels correct.
24:29Again here, the left feels too
24:32bright in comparison to the right here.
24:35This is all so much better.
24:41Ooh, it went black out for the main
24:43view. Huh.
24:45Huh.
24:48Here it seems to have changed some focus
24:50states a bit.
24:53Looking good.
24:55Looking quite good.
24:57I I had my reservations about the
25:00previous stuff. This
25:03This objectively just feels better than
25:05it was before. Like like I was already
25:07impressed with how effective this strat
25:09was.
25:10Holy
25:11This worked.
25:13This wasn't like surprisingly close to
25:15working. I think this just worked. And
25:18this is The real reason I'm filming this
25:20isn't just like show you guys this
25:22technique or get a video out or share a
25:25thing I'm excited about. It's It's that
25:27I need to confirm this with y'all. Like
25:30every time I think there's like here's
25:32the line where models can't do it. There
25:33was the line, the models just blew past
25:35it. There was nothing special here. Like
25:37my skill was very very basic. You could
25:39just copy-paste the text in and say rank
25:42it with these models and tell you their
25:44score is this high. I'm not hiding
25:45something special here. I'm just giving
25:47the LLM my mental model for how I would
25:50do this. I would look at the new
25:51screenshot. I would compare it to the
25:53other products and the old screenshot
25:55and then score it. And then do it again
25:57and again and again until the thing's
25:59good. And if you can get yourself into
26:01this mindset, Muse just gave it 10 out
26:03of 10. Previously, the other models gave
26:05it 9 out of 10 in every category and
26:07Muse was hovering in like the 7.5 range.
26:10And now by changing things, it has
26:11managed to make all of the models happy.
26:15I I am blown away. Wow, it it really
26:18improved the readable like syntax in
26:22light mode. That is hilariously better.
26:24God. Wha- what? Like I'm sorry. I'm like
26:28actually a a in shock here. Cuz like
26:30before it was like, yeah, my app was
26:32ugly and it found ways to make it less
26:33ugly. This is just good. I've been
26:37finding ways to shape a lot of the
26:38problems I'm solving into hills that can
26:40be climbed similarly. Obviously, the
26:43previous verification system of have the
26:46other models look at the thing is silly
26:48but working. Programmatic ones are
26:49better. And if you can take a problem
26:51you have and turn it into a program that
26:53gives a score, your agents are going to
26:55make that score go up. So, one of those
26:57is the MCP for T3 code. This is how
27:00agents in T3 code can do T3 code stuff,
27:02like spin up another sub-agent, read
27:04from a thread, send a message to a
27:05thread, start a new thread, open up a
27:07thread in a new project. All those types
27:08of things are done through the T3 code
27:10MCP. And I had a lot of instances where
27:12models would not do what I wanted them
27:14to do in a thread. Like, they would open
27:16up a new thread when I wanted them to do
27:18a sub-agent, or they would leave their
27:20thread visible when I wanted them to
27:22settle, those types of things. So, I
27:24took all the times in my own history
27:26that models did the wrong thing, I
27:28categorized them, and then I turned it
27:30into an eval. And then I ran it against
27:33Sonnet 5.5, and of all of the things
27:36that I had collected, it got 60 or so
27:39out of the 88 correct, which means the
27:41other 28 were wrong. So, I now have a
27:44set of things but the description does
27:45not serve properly, and I gave two
27:48specific tasks to the agent. So, I
27:50initially started this thread on another
27:52machine where I was going in-depth on
27:55some problems I had on that computer
27:57relating to how T3 code MCP calls
27:59happened, but I realized that that
28:00machine only had access to its database,
28:02so I moved the thread to this laptop
28:05because it had access to every database.
28:07And I told it, "I'm trying to workshop
28:08T3 code's MCP and tool descriptions.
28:09Here's a prompt." It wrote, "Didn't give
28:11enough context, it's from another
28:12machine. You should probably read from
28:13the T3 code DBs directly instead of
28:15ClickHouse, your call. Treat this prompt
28:17as vague guidance on what it's looking
28:19for, not specific instructions." And
28:21this was me after the failures I had on
28:23this other box writing up a prompt on
28:25what went wrong.
28:27It then went around for a while, drafted
28:29some steps,
28:30asked it to take another look cuz I had
28:32just merged changes to some of the MCPs,
28:34specifically what uh models were being
28:36exposed over it.
28:38It had some questions. I answered them.
28:40After a bunch of work, I ended up with a
28:42PR that had a full evaluation with all
28:45those failure cases and more that was
28:47over 2,000 lines of code. I still
28:49haven't merged cuz I'm mostly just using
28:50this for my hill climbs. That evaluates
28:53how well models behave with all the
28:55tools available to them in T3 code. And
28:58one of the most fun things about this
28:59eval I made is that sometimes the
29:01numbers went down. It got 66 out of 85
29:04correct initially, and then this next
29:06run after it made changes was only 63
29:07out of 85. The score went down. And when
29:10I told it that, well, I told it to get
29:12the score up, the order of events I gave
29:14it was get a perfect score, so all the
29:16failures stop, commit that, stabilize
29:19there, make sure all the models we care
29:20about get a perfect score there. Now
29:22your job is the opposite. You need to
29:24delete as much as you possibly can from
29:27the descriptions because those are
29:29bloating the context. I want it to be a
29:31third the size it was before and still
29:33get a perfect score. And this thread ran
29:35for ever.
29:37But it did it. To be very clear, this
29:40absolutely worked. You could see the
29:41difference in how much context was being
29:44used before and after. We cut it
29:46massively, especially if you have tool
29:47search off in T3 code and Claude code
29:50specifically.
29:51Huge reduction in the total tokens. 52%
29:54less from T3 code, 78% less from T3 code
29:58with tool search on because now we have
30:00flattened the model list to
30:01nothing and the others all just get
30:03squashed under the search. The work
30:05worked. This is going to save people
30:06meaningful amounts of money and token
30:08usage in T3 code. This is now live in
30:11the nightly, it'll be in stable very,
30:13very soon. I had to spend a shitload of
30:14tokens to do this, but now less tokens
30:17are used and I have proven that the
30:19result will be better, not worse,
30:21because I have an evaluation for it. And
30:23if I commit and push that eval, then we
30:25have a way to test changes reliably.
30:27This all is chaos because as we know,
30:29LLMs are non-deterministic, but they
30:32also allow for a level of like
30:33flexibility and tedium in your work that
30:37make things available that weren't
30:39before. It was just so much work to set
30:42something like this up. Like I could
30:43have made this by hand a year ago. I
30:45never would have though cuz it would
30:46have taken months to just go through the
30:47data. Now this type of thing is kind of
30:49trivial. And that is unbelievably cool.
30:52This would have just achieved the goal
30:54right now. I've had this running for
30:56most of the work day.
30:58And I'm I'm at the point where I might
31:00just YOLO merge this. It's so much
31:02better.
31:03Do I do it, guys?
31:05Do I just hit the button?
31:08If the team is mad, I'm blaming you
31:09guys.
31:11I got LLMs to make better UI than anyone
31:13on our team was capable of.
31:15We're living in weird times. If you take
31:16anything from this video, it's that
31:18being a clever weird engineer is so fun
31:21right now. If you can come up with these
31:22weird ways to let models measure
31:24themselves in the work they're doing and
31:26just climb the hill blindly, they will
31:28do it and they will make things that are
31:30good. You need to reward them for
31:31succeeding at their tasks, but if you
31:33can find a way to do that, oh man, the
31:36world's your oyster. I've been having so
31:37much fun doing stuff like this and I
31:39hope that this video will help you see
31:40ways to do it yourself. Until next time
31:42though,
31:43peace, nerds.

About this transcript

This page contains the full transcript of Agents Sucked At UI (Until Today) by Theo - t3․gg, generated from the public captions YouTube serves with the video. The transcript has 6,488 words across 917 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.