YT YouTube2Text

YouTube2Text → Transcripts → Most Valuable Skill of 2026: Managing AI Agents

Most Valuable Skill of 2026: Managing AI Agents — Transcript

by Greg Isenberg · 8,264 words · 1,137 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00I don't think it's crazy to say that the
0:01people who know how to run teams of AI
0:03agents are going to be able to
0:05outperform everyone else in this AI age
0:07we're in. So, how do you actually run
0:10teams of AI agents? What does this mean?
0:12What's the stepbystep way to actually do
0:14this? And what are the tools you need to
0:17know like Devon AI to actually go and do
0:19this? Well, on today's episode, I
0:21brought on Ryan Carson, and he clearly
0:23explains all of this for you. even gives
0:26you a tour of his insane setup for how
0:30he manages uh his AI agent. So, by the
0:34end of this episode, you're going to
0:35understand what cloud agents are, how
0:37you can set up automations, how you can
0:39create a software factory that creates
0:41software 24/7, and you're going to be
0:43able to understand it just really
0:45clearly because he does such a good job
0:47at explaining all this. So, enjoy the
0:49episode. I'll see you at the end, and
0:51have a creative [music] day.
1:00Ryan Carson back on the pod by the end
1:02of this episode. Ryan, what are people
1:05going to get out of this?
1:06>> So, you're going to do be able to do
1:09three things really well. You're going
1:10to be able to run cloud agents. You're
1:12going to be able to do automations. Uh,
1:15and you're going to be able to ship
1:18faster. So whether you're a founder, a
1:20soloreneur, a stay a stay-at-home
1:22parent, you're just going to become an
1:24agent pro by the time we're done.
1:27>> Okay? Cuz I think a lot of people
1:29listening to this might have they've
1:31played with agents, cloud agents. Um,
1:34but you're going to help them just get
1:36the confidence basically to be world
1:38class. You know, I think that's what I'm
1:40trying to that's what I've been thinking
1:41a lot about. It's like okay there's you
1:44know a bunch of people who have played
1:46with agents but how do we become
1:48worldclass agent operators?
1:50>> Yeah essentially you are a manager of
1:52agents now. So what no matter what you
1:55used to do, whether it was a people
1:56manager, an IC, you are going to become
1:59a manager of agents now and you need to
2:02be the best in the world. And it doesn't
2:04matter whether you're a VC, a founder,
2:06uh an IC, a mom, a dad, like a college
2:10student, you should manage agents and be
2:13the best at it. And we're going to show
2:14you some tips for that.
2:16>> Okay. So where do we start?
2:18>> Okay, so let me give some context,
2:20right? So people understand where is
2:22this coming from. So, I've been a
2:24founder CEO for like 25 years. Um, uh,
2:26I'm working on my fourth company now.
2:28Um, at my, uh, last company, Treehouse,
2:31which was acquired, we had, you know, I
2:34think up to like 110 full-time
2:35employees, taught a million people how
2:37to code, you know, so I've done the
2:39whole build organization thing and
2:40manage organization thing. Um, and you
2:43learn a lot about managing people on
2:44that. Um, and now this startup which I'm
2:48running it, I am one employee right now.
2:51Um, so we've raised a seed round. We're
2:54scaling really fast. Our revenue
2:55probably is going to 4x this month. Um,
2:58and I'm probably going to be hiring my
2:59first employees. Um, we are essentially
3:02an AI divorce agent for divorce firms.
3:05So, uh, we we we do the the dirty topic
3:08that no one seems to want to want to
3:10tackle, which is family law. So, I've
3:11learned a lot about how to scale me,
3:13right? Um, and so I'm going to help
3:17folks understand how do you manage a lot
3:18of agents well? Um, how do you check the
3:21work? How do you automate things? How do
3:23you work from your phone? Um, you know,
3:25I do almost 50% probably more of my work
3:28from my phone. Um, and uh, it's pretty
3:32wild. So, that's where we're going to
3:34take you. And and and I will say as
3:36well, like this is going to change every
3:393 to 6 months. Um, so, uh, we're going
3:43to share a lot with you. Get your hands
3:44dirty. Try it. Um, and, uh, the best way
3:47by to learn is to do. Cool. Yeah, I'm
3:50actually I'm particularly interested in
3:52the phone stuff because I actually do
3:54zero% of my stuff on the phone.
3:56>> What? Yeah. Okay, let's [laughter] talk.
3:59>> So, uh let me start by just showing you
4:01well when I'm at my desk, what is the
4:04world like? Right. So, right now I'm in
4:05my home office. Um I I I I work from
4:09Connecticut. Um and I do all my desk
4:12work here and I've got this big 52-in
4:15Dell monitor. Um, I used to have like a
4:16couple monitors and I was like, screw
4:18it. I just want a huge monitor. So, uh,
4:21let me share my screen and, uh, all
4:24right. So, what I do is I set up eight
4:28screens on my monitor and, uh, I also,
4:32uh, enjoy food, uh, and snacks as as I
4:35code. Um, but the key is to have eight
4:38things up at once because I have to
4:40multitask a lot of agents at once. Um, I
4:44also use a paper system to keep track of
4:46some of my to-dos, which is a little bit
4:47weird. I use the UG Monk system, which
4:50is kind of fun. And I've got a killer
4:52Razor mouse. Um, I was getting a lot of
4:55wrist pain, actually. Um, so I decided
4:57to go uh vertical mouse on that. Um, I
4:59also uh have a button uh which actually
5:03is connected to Whisper Flow. So, I
5:05almost uh use Whisper Flow for all of uh
5:08all of my stuff. So, I've got Slack in
5:10the top left. That's where I get
5:12notifications and things about what my
5:14agents are doing or what is happening.
5:16Um that that's the primary thing I use
5:18Slack for. Uh and then what I do is I
5:22lay out uh various other screens. So
5:23I've got a one password always set up.
5:26It's very important to keep your keys uh
5:29safe, secure, and separated from the
5:33agent. Right? So I've got all my prod
5:35write keys in one password. I do not
5:38give my agents uh prod keys. Um, so
5:41production writing is dangerous, right?
5:43And your agents will do something bad.
5:45Um, and so you need to set up a system
5:47where they uh ask you for the key when
5:50they need to write to prod and then you
5:51go into one password, you copy and paste
5:53it into the session and you're both very
5:56clear about what you're doing. So uh
5:58that's one tip. Um, keep your keys
6:00secure. And then I use Devon. So uh
6:04Devon is is created by Cognition. I
6:06think it's one of the best software
6:08factories in the industry. It's not
6:10cheap um but it's good. Um the things
6:13I'm going to talk about today are are
6:14not Devon specific. Uh but they do work
6:17very well in Devon. And one of the keys
6:19is cloud agents. Um then I usually have
6:22two screens set up where I'm testing um
6:26my app. It's just a lot of testing. And
6:28then of course I've got X in the top
6:30left and top right. And then on the
6:32right I've got codeex um which we'll
6:35talk a little bit more how I do most my
6:37work in in cloud in Devon. Uh, and a lot
6:40of that work, like we talked about, 50%
6:42is probably on my phone. Um, and then
6:44whenever I do desktop or whenever I'm
6:46doing, uh, things locally, I'll do it in
6:48codecs. Uh, usually that's in the bottom
6:51right. So, that's kind of my when I'm at
6:54my desk, uh, my setup, um, and what I do
6:58when I'm here. But the truth is I do a
7:01lot of work, you know, from this my
7:04iPhone. This is my beautiful wife. Um,
7:06uh, it's just I get a lot of stuff done.
7:09You know, I I'll be in the shower and
7:10I'm thinking, okay, I need to check on
7:12that PR or maybe I want to land that PR,
7:14right? You know, grab it, talk to Devon,
7:17which is just done in a browser on my on
7:19my mobile and I get a lot of stuff done
7:20that way. So, that's my setup.
7:24So I I keep hearing this that Devon is
7:28is almost you know it's the most
7:30powerful coding agent and you know can
7:33you just for folks who haven't heard of
7:35Devon or or who haven't played with it
7:37like why is that what is going on? So,
7:40um, so about 2 years ago, maybe two and
7:43a half years ago, Devon came out and
7:44they pitched it as this idea of like
7:46it's a software engineer, right? And
7:49honestly, it didn't work very well,
7:50right? The the models were just not good
7:53enough. And no matter how good the
7:54harness was, the models just weren't
7:56there. Um, and then fast forward to, you
7:59know, 2026, um, the models are
8:02definitely good enough. And when you
8:04have a good harness that is cloud-based,
8:08um, it's really good. So, think of it as
8:10you've got your choices, right? So, if
8:12you're using agents to build things,
8:14you're probably using codecs, you know,
8:15from OpenAI or using Cloud Code from
8:18Anthropic or you're using one of the
8:21indie, I call them the the indie uh,
8:23agent harnesses. You've got AMP, you've
8:25got Cursor, which is not independent
8:27anymore, but sort of own owned by Elon.
8:31Um, and then you've got cognition which
8:33makes Devon. You've got factory uh as
8:36another one as well. So you've got these
8:37choices, right? And so I thought, okay,
8:41I want to scale myself horizontally.
8:46Like I want to be able to run as many
8:47agents as I possibly can and I don't
8:50want to be working on my laptop and
8:52trying to figure out
8:55which code is colliding, you know, which
8:58work tree am I on? all a lot of
9:00technical stuff that really
9:01[clears throat] it's it keeps you from
9:03shipping. Um, and you know, I know
9:06there's a lot of different folks that
9:07listen to the show. You might have
9:09soloreneurs who are are are newer to
9:12coding. They don't have computer science
9:13degrees. They're not super technical,
9:15but they're building things. Then you
9:16may have, you know, software engineers
9:18who are multiplying themselves 10 to
9:20100x. They they they have deep
9:22understanding of software and
9:23architecture. And then you have founders
9:26who are coding but are abstracting
9:28themselves. and you have VCs who are
9:29coding, you know. Um, but, uh,
9:33everybody's going to learn a couple
9:35skills. Um, and what's interesting is
9:37I'll sort of, uh, zoom out for a second.
9:40I think we all thought that engineering
9:43and being technical was going away, but
9:46actually what's happening is the more
9:48you become a better agent manager, um,
9:50actually the more technical you become.
9:53uh and I I think it a good analogy is
9:56that you are basically an engineering
9:59manager right and so to be a good
10:02manager of many agents you do need to
10:05become technical right there's this
10:07reality of you have to understand what
10:09Postgres is you have to understand a
10:11production environment versus a dev
10:13environment you have to understand
10:14migrations like but the truth is you'll
10:16learn all that by using agents right and
10:20so I am way more technical than I have
10:23ever been in my whole life. Um, and so I
10:26think the idea is become more and more
10:28technical. Therefore, okay, so let's
10:29back up. Why Devon? What is it? It's a
10:32really good agent. They have uh their
10:35cloud environment really nailed down. So
10:37that's the first takeaway I want to talk
10:38about today is cloud. Okay. Um, how can
10:42you run more agents, get more done? Uh,
10:46and the answer is you work in the cloud,
10:48not on your local machine. Now,
10:52for the folks that aren't super
10:53technical, I'll sort of explain what
10:54does that mean. So, when you're writing
10:56code or you're building things, you are
10:58editing code, right? That code has to be
11:00edited and then hosted somewhere, right?
11:03Uh and and usually what we used to do is
11:06we used to all work on our Macs and
11:08that's called local development. And you
11:10have to set up your whole local
11:11development environment. You have to
11:12have your database, you have to have
11:13your authentication, you have to have
11:15everything local, you have to have your
11:17dev server set up. Um, and then what
11:20happens if you want to do two things at
11:22once? You have to either use get work
11:25trees, which is pretty technical and
11:27hard to understand, or you have to
11:29literally make another copy of your code
11:32in another directory. And then, so then
11:34to work on two things at once, you have
11:35to have two copies of your code, which
11:36is hard. And then what about three or
11:38four? What if you want to do five or 10
11:39things? It just doesn't work. So what
11:43Devon does and and I think a lot of good
11:45uh independent agent harnesses like
11:47cursor is doing this now um I think
11:50codeex is starting to do this okay is
11:54they create what's called a VM or a
11:56virtual machine in the cloud and it's
11:59it's basically a little server and it is
12:02your development environment and you
12:04click a button and it spins up and all
12:06of a sudden you can code in there and
12:09the beautiful thing is if you can you
12:11can create infinite numbers of these
12:13things. So I often have, you know, at
12:16least five uh uh cloud agents working at
12:20once, often 10. And there's no risk that
12:23the code is going to collide. So what it
12:25does is it takes away all of the mental
12:28overhead of trying to coordinate these
12:30things. Like I want to work in this part
12:32of the code, but what if it overlaps
12:34this part of the code or what if, you
12:36know, I want to do that? Have I
12:38synchronized and pulled the latest? like
12:41I don't know. And when I work in the
12:43cloud, I literally just open up my
12:45browser and I click new session and a
12:48new and a new VM spins up and I never
12:50ever have to think about it. So I think
12:53if you are working locally, I honestly
12:55think you are a caveman. Like I I think
12:58you're you are holding yourself back and
13:01you are shipping 10x less than you could
13:04be. And it is not smart. No matter if
13:09you think the cool people work locally
13:10and that all the smart ends, it's not
13:13true anymore. You work in the cloud.
13:15>> Yeah. I mean, there are a lot of people
13:18like on X who are sort of like local
13:20maxis who look at people
13:24who spin up virtual machines and cloud
13:27agents as sort of like amateur-ish. What
13:30do you say to people like that?
13:32>> They they are not they are not doing
13:35real work.
13:36>> [laughter]
13:37>> Um, here's the deal, right? So, as soon
13:40as we hit PMF with Untangle and we and
13:43we started acquiring real customers and
13:44having to ship real fixes, real
13:46features, like I had to
13:49probably 50x my output, right? Um, and
13:53you just can't do it if you're working
13:55locally. So, I think it's it's nice to
13:58work locally and sometimes you have to
14:00if you're doing heavy front-end stuff.
14:01So, say say you're shipping a brand new
14:04feature with a lot of UI, like a lot of
14:06new UI, right? Then yes, of course,
14:10you're probably going to spin up a local
14:12agent and you're going to do some light
14:14front end, you know, probably
14:16wireframing. Um, but as quickly as you
14:19can, you want to move that into the
14:20cloud and then let a cloud agent take
14:22over. Um, so I think this is happening
14:25to a lot of people. I think a lot of the
14:27traditional
14:30knowledge of how to work is is is out of
14:34date. Um that's where we're at.
14:37>> Okay. So if you do start using cloud
14:40agents and you do 50x output, you know,
14:43in some ways that is a little stressful
14:45because you have so much stuff going on,
14:48right? And now all of a sudden you have
14:49to manage more. So how do you how do you
14:53structure a system so you don't go
14:55absolutely crazy?
14:56>> So it's interesting I think first of all
15:00you have to shift your expectation of
15:03what your work is and what your work is
15:07now is making high stakes decisions
15:09almost all day. So in the past like we
15:12all had the luxury of making probably
15:15you know at most maybe two to three
15:17highstakes decisions per day right you'd
15:19have an important meeting with your key
15:21team or with a customer and you'd make a
15:24couple important decisions and you would
15:26communicate those decisions and then
15:27they would get done over you know two
15:29weeks. Um, you know, I think you have to
15:33sort of mentally pace yourself to say
15:35probably by lunch you're going to have
15:38made, you know, 10 to 20 high stakes
15:42decisions.
15:44Um, and so you have to learn how to do
15:47that. So, which is your question, how do
15:49you do that? Um, and I think there's a
15:52couple simple things, right? Um, so when
15:55I'm working with my agents, I actually
15:56have to organize my threads by and I pin
16:00the ones that are the most important
16:02work we're getting done today, right?
16:04Cuz there's a lot of quick buck bug
16:06fixes or, you know, small things you're
16:08shipping and you kind of learn to
16:10separate two into two buckets buckets.
16:13What are the big big important things
16:14I'm getting done today?
16:17Pin those. And and then the other things
16:19just let them rip and get back to them
16:21when you can. Um, and then set yourself
16:24this kind of uh almost timer, right,
16:26where you're like, "Okay, I'm going to
16:29go crazy if I just click through my
16:31threads like this all day." You just get
16:35completely wiped out, right? And so,
16:38it's almost this discipline of saying,
16:40"I'm actually, you know, that the
16:42phrasing smooth is uh slow is smooth and
16:45and smooth is fast."
16:46>> Yeah. I I think there's a little bit of
16:49of that where you realize I can only
16:51check on my agents and make highstake
16:54decisions probably every 25 minutes.
16:58Um and so what you might do is settle
17:00into this cadence where you check on
17:02your high stakes threads like every 25
17:04minutes just so you can like literally
17:05mentally rest. Um uh and then the other
17:10side is that is just understand that
17:12this is the world we live in and it is
17:14kind of exhausting. Um uh and and that
17:17is kind of where we're at. And in in
17:20order to survive in this new world,
17:21you're you're going to work a lot more,
17:23not a lot less. And on that note, I'll
17:25share my screen because there is
17:28essentially
17:30uh a world we live in now where you're
17:32going to ship probably,
17:34you know, so this is
17:37my PRs, right? And we're looking, these
17:40are days, right? And you can see, you
17:42know, the average here is is sort of
17:45this 22 to 25 PRs a day, right? And and
17:50sometimes, you know, 40 a day. Um most
17:54of these are being, you know, merged,
17:56right? A couple are being closed. And
17:57what's funny is what happened this day.
18:00Um I went uh hiking with my son.
18:03>> Which day is that? I can't see your
18:04cursor.
18:05>> Oh, it's uh the was the 18th.
18:07>> 18th. Um, and so we went to climb Mount
18:09Washington and I literally couldn't
18:11access my my phone. Um, but notice
18:14immediately the morning of, right? You
18:16know, while he was sleeping, I I shipped
18:18what is that, you know, eight PRs.
18:21[laughter]
18:22Um, so this is kind of, you know, the
18:24world we live in now. Um,
18:27and we could talk about that. Um, but
18:30let's zoom out to the the lessons here.
18:32You've got to go cloud, right? You have
18:35to understand in order to compete in
18:37this new world, you're going to be
18:38working half the time from your phone.
18:41And it is tiring. Um but but but it's
18:46like going to the gym. Okay, I have this
18:47new muscle like I never had to exercise
18:49before and it turns out this weird back
18:51muscle. It turns out like that's the
18:52most important muscle in the world.
18:54[laughter]
18:55And if you don't do it, you're not going
18:56to win. So the way to do that is then
18:59set some sort of cadence for yourself.
19:01Pin your most important threads. Write
19:04it down. So, this, you know, what I'm
19:05going to sort of flash up on the screen
19:07here is my written to-do list. Um, every
19:10day, you know, I use this hilarious
19:13analog system to like to remind myself
19:16what are the most important things we're
19:18going to ship today because there's
19:20going to be 10 probably PRs that we ship
19:22that are fire, you know, putting out
19:24fires. Um, and and you can get
19:27distracted and exhausted. So, so cloud
19:29and then exercise this new weird back
19:32muscle which is making high stakes
19:34decisions all day. Um,
19:37>> including on your phone, right? What
19:38you're basically saying is like
19:40>> you're going to because of the velocity
19:42of
19:43uh
19:45of you know do knowledge work and you're
19:50you're just you'll have [sighs]
19:52>> stuff going on and you will need to be
19:55there to say yes. know actually this,
19:58you know, giving feedback in real time
20:00and
20:00>> you know why your point is why wait
20:04until you're in front of your Mac to
20:06give feedback.
20:08um
20:08>> it makes no sense
20:10>> if you're trying to hit product market
20:11fit and if you're trying to 4x revenue
20:14in you know in a month or whatever it is
20:17um you know why
20:18>> got to be available
20:19>> got to be available
20:20>> it and you know I think it just it's
20:22easy to think of these things as humans
20:24right like you have a team now that is
20:27infinitely scalable they're going to do
20:29work very fast and they're going to be
20:31blocked by you making important
20:32decisions right um and you can't
20:36delegate the decisions Yet I think we're
20:38still very much even if you use Fable
20:40like this is a trick is is use a Fable
20:43thread as kind of the manager and then
20:46ask it to spin up children and I do this
20:48a lot. So, you know, I'll spin up a
20:49Fable thread and I'll say, "Okay, um,
20:52you know, we want to accomplish these
20:53five things. You know, spin up five, you
20:55know, child Devon sessions. Don't use
20:57Fable. I use Fusion for that, which is a
20:59cheaper model or it it's cool. It's like
21:01their their new model where they've got
21:04um Fable is kind of the parent and
21:06they've got a sidekick.
21:08Point is, you're not going to it's not
21:09viable to run Fable or any super premium
21:12model all the time, but have a a sort of
21:15a parent, you know, very intelligent
21:17model run children. You can do these
21:19things, but in the end, you're going to
21:22have to pay attention. Um, and it's just
21:25like if you ran a big team and you were
21:27out to lunch on your boat all the time,
21:30you got to get in the office, and the
21:32office is your phone. Um, and so get it
21:36done. And so, Greg, this is my homework
21:37for you. Next time we talk, I want you
21:39to be doing more than 50% of your work
21:41on your phone.
21:42>> Yeah. I mean, that's why that's why
21:44you're here, right? Like, you're a lot
21:46of people ask me like, "How do you
21:47choose guests?" And and why do you
21:50explore topics, certain topics over
21:52other topics? And it's it's really it's
21:55for everyone. Like, I love arming people
21:58with the information, the tactics, the
22:00systems, but I'm also everyone. Like,
22:02I'm a part of that, right? Like a part
22:03of that is like I know that you're
22:05you're doing this and I want to
22:07understand why you're doing it. And I
22:09also want to understand like you know
22:11one of the takeaways I have is your
22:13system for dealing with tasks is not is
22:16not too dissimilar to a a very organized
22:19person pre- AI right
22:21>> like the idea of like pinning you know
22:24your most important things like it's on
22:27one hand it's pretty obvious but on the
22:28other hand if no one tells you like
22:31>> hey you know the way to to do this so
22:33you stay sane is via that system you're
22:37you it's overwhelming. Right.
22:39>> Yeah, it is funny. It's like back to the
22:41basics, right? It's just in an agent
22:44age. Um, and it turns out like being a
22:47very very good engineering manager is is
22:49probably one of the most valuable skills
22:51you can you can teach yourself. And you
22:54know, I'm uh have the honor of sort of
22:56mentoring uh my nephew right now and and
22:59he's coming to me and asking me how do I
23:01win? You know, he's just come out of
23:02college soon. How do I win in this new
23:05world? Right? And you know, I'm
23:07basically teaching him what we're
23:08talking about in the show. It's like,
23:10okay, your job is going to be managing
23:11as many agents as you possibly can. Get
23:14good at that, right? Which means, you
23:17know, you need to start to adopt these
23:19these these behaviors, right? Work in
23:21the cloud, prioritize what you're doing,
23:24learn how to make high stakes decisions
23:26quickly, get technical. Um, you know, it
23:30it is a farce that engineering is going
23:32away. Like if anybody tells you
23:34engineers are going away or becoming
23:36less technical, they're just not doing
23:38the work, right?
23:39>> Um and it's like being a carpenter. Like
23:42say you had robot carpenters around. If
23:44you were a real carpenter, you'd be a
23:46much better manager of those robot
23:48carpenters because you understand how to
23:50make the cut. You know where to put the
23:52nail. Um and so and so it's important.
23:56So, to back up and if you're listening
23:58to the show, like the thing I want you
24:01to walk away is is cloud agents. Get
24:04your reps in, figure out how that works.
24:07Try it a couple times. Um, do it from
24:10your phone and get comfortable, you
24:13know, looking at your small phone
24:14interface and and learning how to work
24:17that way. Um, so I think that's that's
24:20thing one. Uh, it's very important. The
24:22second thing I want to dive into is
24:23automations.
24:25Okay, so say you're working in the
24:26cloud. Well done. Um, you know, you're
24:29starting to separate yourself from the
24:30crowd. The second thing is automations.
24:33Um,
24:34you need to sort of think about things
24:36that you would have normally had a
24:38meeting with, uh, you know, once a week
24:40with your team, uh, where you check on
24:42something or daily you check on
24:44something and build an automation. What
24:47does that mean? Um,
24:49so the first thing to do is go back to
24:52your agent, start a session, and say,
24:54"Okay, I want you to automate this task
24:58every x amount of days." And so, one of
25:01them that I do, it's really simple, is
25:03that I want to automate the test of
25:06signing up for Untangle, um, creating a
25:09case, onboarding a client, having that
25:12client go through discovery, you know,
25:14basically our user experience. But I
25:16want that to be automated in a browser.
25:18Like this is not rocket science. This is
25:20basic user testing, right? Um but it
25:24turns out that good agents can do this.
25:26And so the big unlock is uh set up your
25:31agents so that they can make themselves
25:34better um uh instead of you trying to
25:37build these systems. Um and so I set
25:40that up and then in Devon it's really
25:42cool. It's called a playbook. But the
25:44idea is it's it it's it's sort of like
25:46here's how to do this thing. It's
25:48different than a skill. It's more of a
25:50list of things to do, how to do them
25:52correctly. Um, and I have a uh it's
25:56called endtoend
25:58uh signup test and that runs three times
26:00a week. Um, because it is expensive.
26:03It's it's probably 60 bucks in tokens.
26:06Um, because it's doing a lot of it's
26:08doing a lot of browser testing. Um, and
26:12uh, I think that's sort of a side point
26:14I want to explain here is it's it's now
26:18vital that your agent can can properly
26:21brow browser test. Um, and it really
26:24should work out of the box, right? This
26:26is part of the reason I used Devon is
26:29because they've been using cloud agents
26:32that can do browser testing for 2 years
26:34now. It's so good. It records a video.
26:38It annotates the video. it it then looks
26:41at its own video and it fixes bugs that
26:43it sees, right? So you have this agentic
26:46loop that is specifically around browser
26:48testing that then is automated three
26:50times a week, right? And then what what
26:54what do you do with that? And then you
26:55need to trigger a triage based off of
26:58that. So say that it's like, oh wow,
27:00during this the the third test failed in
27:04this automation, I'm going to spin up a
27:07Devon session or an agent session to
27:08then fix it. Um, and then you have to
27:12figure out where do you know that that
27:13failed? How do you know? Did the agent
27:16just kind of f, you know, silently fail
27:18and then fix itself? What about the PR
27:20that was created? So you have to start
27:22thinking about how do you see these
27:24things when they happen, right? as the
27:27manager of all these agents, how do you
27:30know uh what's failing, what's not? And
27:32and that's again where you should talk
27:34to your agent and say, "Okay, we've got
27:35this end to end test. You know, I
27:37understand it runs Monday, Wednesday,
27:39Friday. Um I understand it, you know, it
27:42it triggers a child triage session if
27:45anything goes wrong. How do I know about
27:47it?" And it and then your agent might
27:48say, "Well, you know, why don't I post
27:50in Slack?" You know, and and you can
27:52say, "Well, okay, how are you going to
27:53do that?" Well, we need to connect the
27:55MCP. you know, how am I going to know to
27:57look in that channel? There's just all
27:59this machinery that you need to to to
28:01automate and figure out and and work
28:03through um to build that system um of
28:07automation so that you are doing less of
28:09the work.
28:10>> So this specific automation is this more
28:13of a like bug testing QA automation or
28:17is it more of a um
28:22you know UX QA perspective? Th this one
28:26is more about um uh finding bugs that
28:29have been introduced to prod that
28:31somehow uh we missed in our in our user
28:35tests um
28:36>> in our automated test suite.
28:38>> So that one in particular is like hey we
28:40just can't have signups go down and not
28:42know about it. Um, but you know,
28:45sometimes your your automated test suite
28:47just it it doesn't catch everything. Um,
28:50but a good oldfashioned clicking through
28:52the site in a browser catches that.
28:55>> Um, it is not really about the UX. Um, I
28:59think that there's there's no one thing
29:02I will say is there's no substitute and
29:06we probably won't have a substitute for
29:08a while of you using your own app. there
29:12just isn't a good substitute right now.
29:15So, right now, every every so often, I
29:18literally go through our entire app
29:20myself. Um, and then I obviously quickly
29:23spin up, you know, sessions to fix bugs
29:25I find. But, um, we're we're just so far
29:28from agents still. you the the smartest
29:31models in the world still lack this
29:33sense of obvious
29:36you know uh intelligence where you're
29:40like why did you think that was okay?
29:42Well, yeah. I think the bigger question
29:44is, you know, if if there's a division
29:46of labor between human beings and
29:48agents, what are the tasks that you as
29:50the human being are going to be doing?
29:52What are the sets of tasks? And what are
29:54the sets of tasks that the agents are
29:57going to be doing that you're okay with
29:58them doing and keeping you in the loop?
30:00So, my question for you around
30:01automations is beyond beyond QA, what
30:04are some other automations that you know
30:08a founder could be implementing? So um
30:11let me pull up my list for you. So the
30:15kind of automations that I have are uh
30:19clearly uh uh QA bug fixine. Um the
30:23other is what I call my production
30:25watchdog. Um so we've got a lot of real
30:30law firms using Untangle. got a lot of
30:32real clients, you know, going through
30:34discovery and we've got just a huge
30:37amount of activity and and honestly I
30:39can't keep track of it uh you know
30:41mentally by just looking through logs.
30:43And so what I do is every day at 9:00
30:45a.m. I have what's called a production
30:46watchdog automation which what it does
30:49is it goes through all of the events in
30:52the database um that happened for our
30:55paid customers and it summarizes that in
30:59uh a JSON file and then that JSON file
31:03is in our admin and then I I you know
31:05know every morning I go in and I click
31:07on uh the production watchdog and I read
31:09through the summary of what the
31:11customers did um and it's so valuable.
31:16Um, so it's sort of like you can
31:17imagine, you know, your chief of staff
31:19showing up and saying, you know, all
31:20right, Greg, here's the important stuff
31:22that happened yesterday for our
31:23customers, right? Um, and I'm going to
31:26roll it up and explain, you know, the
31:28things that went well and a couple bugs
31:30that we saw. Um, so that's an
31:32automation. Um, essentially what
31:35happened yesterday that was important.
31:37Um, so that one
31:38>> that's a big one.
31:39>> Yeah,
31:39>> that's honestly a big one. like it's
31:40it's like one of those ones that sounds
31:42small but is a big one because you're
31:43going you think you know what's going on
31:46but you don't really
31:48>> you don't I mean and it's it's so
31:50shocking to me like yeah the amount of
31:52times I'm like oh uh and this is the
31:55other trick man so use a production
31:58watchdog automation pick your most
32:00important customers and then make sure
32:02that in that report it's like the
32:04customer did X here's a link to view
32:07that right and so I actually have a
32:10production instance where you click it
32:11and you actually see what the customer
32:14was doing and then I'll be like whoa
32:17that's weird like and we have this
32:20amazing ability that you know over you
32:23know uh you know millions of years of
32:26evolution in very quickly picking up
32:28things that are off. [laughter]
32:30And so there's a couple times I'm like,
32:32"That's weird." And I click into it and
32:34it turns out, oh, the user experience um
32:37that the agent built on that wasn't was
32:39wasn't quite what I thought it was. Um
32:42and so yeah, production watchdog
32:44automation that links out to real UI um
32:49is just man, it's a game changer. So
32:52that's a big automation. Um this is a
32:54really important a self-improvement
32:56loop.
32:57How does this work? Okay, everyone's
32:58talking about self-improvement, right?
33:00>> Um, so in Untangle, we have an agent
33:03called Grace, and she is essentially a
33:05parallegal. Um, and she has a lot of
33:09chats with both uh our customers who are
33:11attorneys, our customers who are
33:13parallegals, and our customers and their
33:15customers who are clients that are
33:17getting divorced. Um, so how do I
33:20actually grade those chats and then
33:22improve them uh without managing the
33:26fine details? So what I do is I every
33:28day I have an automation that looks at
33:31these chats and then grades them on a
33:33rubric. And so you and again just talk
33:36to your agent about this. You pick the
33:38most important part of your app that you
33:39want to self-improve and say here's how
33:42you agent judge whether this thing that
33:45happens is good or bad. Um and then
33:48every day I want this automation to kick
33:50off. Look at those things for us.
33:53They're conversations that Grace is
33:54having. and then I want you to grade
33:56them. And then if anything is bad or
33:59below this score, I want you to kick off
34:00a child session and fix it. And it is
34:04shocking to me the number of small, you
34:07know, fine UX details that have been
34:10picked up by this loop and have been
34:12fixed. And honestly, there are things
34:14that I just wouldn't bother doing.
34:16Either I didn't know about them or I'd
34:18be like, me, you know, it's a paper cut.
34:20Like, are we really going to, you know,
34:22ship a fix to that? But because the
34:24agent already identified it, spun up a
34:26PR, and it's ready to ship, I just go
34:29ship it. [laughter]
34:31And you know, we probably ship I would
34:33say probably three of these a day.
34:35>> Wow.
34:36>> That's that's huge.
34:38>> And you know, a lot of a lot of the
34:41backlash on on loops is is just that
34:44it's expensive. Um, how how how have you
34:48with with Devon specifically, has it
34:50been like absurdly expensive? Like what
34:52are we talking? No. So he Well, so
34:54here's the thing. So last month I spent
34:57probably 20 grand in in tokens, uh,
34:59which is just too much. Like it's not
35:01viable. I I think all of us are in a
35:03place where we we're getting to the spot
35:06where it's realizing, okay, for real
35:08engineering work per employee, you're
35:09looking at probably 5 grand a month.
35:12Like that's probably where we're going
35:13to shake out here. And anything more
35:15than that, you you really need to figure
35:17out model routing. Um and so thankfully
35:21Cognition knows this and every
35:23everyone's figuring this out like yeah
35:25whether it's cursor cognition
35:28>> I would say factory AMP all of the
35:30independent agent labs understand like
35:32[laughter] we our customers have a limit
35:35like they're not going to pay infinite
35:37token budgets
35:38and then you have the this is me ranting
35:40a bit but it's like you have cloud code
35:42and codeex who are just severely you
35:44know fund they're they're funding tokens
35:48and making it seem cheap, but this is
35:49not going to work long term. And so what
35:52I do is basically I have learned how to
35:55use the right model for the right task.
35:57And so these reinforcement loops, they
36:00use what's called SUI 1.7. It's a
36:03fine-tuned model that Cognition has
36:05specifically built for coding. It's
36:06super cheap um compared to, you know,
36:10Opus 48, you know, GPT56. And so a lot
36:13of these loops happen with this uh you
36:15know cheaper you know finely tuned model
36:18and we're just going to see more of
36:20that. So you're talking I don't know it
36:22you know actually I could probably I
36:24could probably look and dig it out but
36:25you're t probably talking like five
36:26bucks a session on that.
36:27>> Okay that's not bad.
36:28>> I mean and if you're not willing to pay
36:3015 bucks a day
36:31>> to improve one of the core you know
36:33feature sets of your like what are you
36:35doing? Like either it's not a real
36:36company or you don't care.
36:38>> Yeah.
36:40>> Totally. spend the money like and and
36:43then you know figure out the the system
36:47uh to reduce your token cost. I will say
36:49this is why I don't understand why
36:52anybody would build their engineering on
36:55a frontier lab stack. Like I'm just
36:58going to call it out like if you are if
37:00your whole engineering uh you know
37:02motion is happening inside of cloud code
37:04or inside of codeex what are you doing
37:07like because they are not incentivized
37:10to make it reasonable for you long term
37:14right they're going to lock you in to
37:15their models to their process whereas if
37:18you use an independent agent lab you
37:21know like an AMP like a Devon like a
37:23factory like sort of a cursor but
37:25they're kind of weird now those systems
37:27are going to optimize for affordable
37:29engineering long term. Um, and uh, I
37:34don't think you want to be locked in to
37:37anthropic or or open AI, you know,
37:40solely.
37:41>> I mean, to be clear, you can still use
37:43their models within some of these
37:46products, right? So, you're not
37:47abandoning ship.
37:48>> No, it's actually so it's it's yes and
37:51right. So the beautiful thing is if you
37:52pay you know an independent agent lab
37:55like a factory like a cursor like a
37:57cognition like an amp you know etc
37:59they're incentivized to figure out how
38:01do I give you the best results for the
38:04lowest price. So they'll they'll say,
38:05"Okay, we're going to use GPT56 soul on
38:09extra high for this type of task, and
38:11then we're going to route, you know, to
38:12a uh you know, to a SWE 1.7, you know,
38:16and then we're going to have it checked
38:18with a Fable." Like, they're going to
38:19figure out that hard stuff.
38:21>> Um, and the other thing I will say is if
38:23you're trying to build a software
38:25factory for yourself, stop. like it, you
38:28know, the reason why you see, you know,
38:31Ramp launch, Inspect, which is their
38:33custom in in-house agent, is because
38:35they got to a size where they had to
38:36build their their software factory
38:38in-house. And if you get to a certain
38:41size, you will do that. But now, if
38:44you're a oneperson shop and you're just
38:45building something that's just you, you
38:47can probably get away with this $200 a
38:49month uh, you know, angle on a pick your
38:53frontier lab. But as soon as you
38:55graduate into, okay, we're building a
38:56real product. We have product market
38:58fit. We're going to start hiring people.
38:59You got to build a software factory. And
39:01at that point, you want to be on an
39:03independent agent lab. Like otherwise,
39:08you're going to have to build it
39:09yourself, which is stupid, or you're
39:10going to pay and be locked into, you
39:13know, a walled garden. Um, and you can't
39:16do that. The the analogy I use for it
39:18is, you know, working with an
39:20independent lab is sort of like a uh
39:24like working with like a mortgage broker
39:26or travel agent. Like they're going to
39:28make a lot of calls to make sure to
39:29[laughter] get you the best price and
39:31stuff like that. Um
39:34>> that's an Yeah, to me I use both. You
39:37know, I I'm in cloud code, but I'm also
39:40I use cursor personally. Um, although
39:43Devon I'm interested in I might if
39:45people are interested I might just do
39:46like a whole
39:48breakdown on Devon if people if people
39:50want to want to do that.
39:51>> Devon's so good now. Um, you know
39:54>> I just it's so good. It it's amazing.
39:56Um, but I will say I do use codecs,
39:59right? The the the the Mac app is so
40:02beautiful like
40:04>> and there's and they are subsidizing the
40:06tokens so much that if you want an agent
40:10on your machine just to do really cool
40:12helpful stuff, open browser tabs and
40:14create Google Docs and check your email
40:16and and do all this information work and
40:20and basically get it for free. I mean,
40:22you're paying 200 bucks and you get
40:23almost infinite tokens. like awesome,
40:26but don't build your company software
40:27factory on that. Like it I just think
40:31it's bonkers to do that.
40:32>> Mhm. We've covered a lot today. Is there
40:34anything else you wanted to uh
40:37[laughter] cover?
40:38>> I feel like I'm just vomiting, you know,
40:40all all the stuff in my head that
40:42>> Well, no, I it's important that like
40:46this is what I wanted. I wanted you just
40:47to vomit everywhere because like it's
40:50sort of like advice that you you you
40:53would give to your to your nephew I
40:55think um or 22-year-old um
40:58>> you know a 22-year-old fresh out of
41:00college. It's like what are the minds I
41:02need to know in terms of
41:04>> managing agents? So is there anything
41:07else that people need to know? Cloud
41:08agents, automations.
41:11Yeah, cloud
41:13automations like th those are the two
41:16that I want people to walk away from. I
41:18guess the third is is really you will be
41:21using a software factory like and what I
41:23mean is the agents are going to be
41:24writing 100% of your code, reviewing
41:26100% of your code, shipping 100% of your
41:28code. Like that's where we're going. Um
41:30and so the the more you can get there,
41:34the the better. thing one uh thing two I
41:37think you know thing three is um it it
41:40is important to build um uh a credible
41:44reputation for yourself right and I will
41:47say you know the work I put in over you
41:50know the last 20 years on X like it
41:54really pays dividends and what I mean by
41:56that is if you if you share what you're
41:59learning publicly and you generally give
42:01like write very helpful articles on X.
42:05Um, and obviously the ALGO really does
42:07reward articles still. So, you should be
42:10writing articles on X. Um, you should
42:13absolutely be on X, no matter how you
42:14feel about Elon. Um, uh, because sharing
42:18your knowledge really does build up
42:19credibility, which will pay dividends.
42:22And I don't mean just like getting paid
42:23to post. I mean the the the the network
42:26and the relationships. I mean, the
42:27reason why I'm on the show is because we
42:29built a relationship over X. Um, and so
42:32I would encourage people take the time
42:34to share what you're learning. It
42:36doesn't have to be super polished. Um,
42:38but get out there and be useful to
42:40people. Um, and it will open up doors.
42:43Um, so, uh, you know, more than you
42:45think. Just get out there and share it.
42:47Um, and, uh, and you'll reap the
42:49rewards. Not immediately, but
42:51eventually.
42:51>> I mean, even if you don't know, you can
42:53just say, "I don't know." And that's
42:54your and that's your strategy, right?
42:56And that's interesting, too.
42:58>> Amen. And if you don't know, then ask an
43:00agent to help you.
43:01>> Yeah. So,
43:02>> I mean, my original co-host of this show
43:05was Sahel Bloom. Sahil Bloom worked in
43:08private equity and during COVID he just
43:12started writing about topics he was
43:14interested in like he would be like just
43:17finance topics or you know particular
43:20individual who was interesting and it
43:22like it it it read as like Wikipedia
43:25articles but really focused on Twitter
43:28specifically like optimized for Twitter
43:30and and that was his thing and he wasn't
43:32saying I know everything he was just
43:34saying like I'm learning this thing.
43:36Here's the thing I'm learning. He grew
43:38his audience over a million Twitter
43:40followers, wrote a bestselling New York
43:41Times bestselling book, started
43:43companies on top of it, raised a fund on
43:45top of it. So, anything is possible for
43:48sure. Uh on on X.
43:52>> Yep. Just just get out there and do do
43:55and learn. Um
43:56>> uh you know, there there's no try.
43:59There's only do as Yoda says, right? So,
44:02>> totally. Uh Ryan Carson, I feel smarter
44:06and that's that was my goal. Um
44:10and uh speaking of X, you're a wonderful
44:13follow. I'll include links where you can
44:16follow Ryan across the internet,
44:18including X. Um he's a mustf follow, so
44:21please do that. Thank you for coming on,
44:23sharing your generous uh thoughts,
44:27incredible setup, and uh is there
44:30anything you want to leave people with?
44:32No, thanks Greg. Thanks for having the
44:34show. I appreciate you putting out the
44:35constant knowled uh constant knowledge
44:37and sharing. Um it really makes the
44:39internet a better place. So happy to be
44:41here and thanks for having me.
44:43>> All right, I'll see you next time.
44:45Take care.

About this transcript

This page contains the full transcript of Most Valuable Skill of 2026: Managing AI Agents by Greg Isenberg, generated from the public captions YouTube serves with the video. The transcript has 8,264 words across 1,137 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.