YT YouTube2Text

YouTube2Text → Transcripts → Building Great Agent Skills: The Missing Manual

Building Great Agent Skills: The Missing Manual — Transcript

by AI Engineer · 4,204 words · 597 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Hello friends, I was dearly hoping to be
0:01able to come to the Air Engineer World's
0:03Fair, but family matters have intruded
0:06and I'm not able to make it. However, I
0:08will not be leaving you empty-handed.
0:09I'm going to give you the talk that I
0:11would have given in San Francisco. This
0:13talk is called The Missing Manual, How
0:15to Write Great Skills, and I think that
0:18the ability to distinguish good skills
0:20from bad skills is only getting more
0:22important. As developers, we seem to be
0:24pretty talented at finding different
0:25forms of hell for us to go to. In like a
0:29few years ago, we had tutorial hell,
0:31which is where you would go into a bunch
0:32of tutorials trying to learn something,
0:34not be able to piece it together, and
0:36sort of just get into this cycle you
0:38couldn't get out of. We had framework
0:41hell, where every other 10 minutes there
0:43was a JavaScript framework being
0:44announced, and you know, you had to
0:45learn the hot new thing all the time.
0:47And now, I think we have another version
0:49of hell, which is skill hell. Skill hell
0:52is where you have all of these skills
0:54available, freely available, that you
0:55can download, contribute to, you can
0:57figure out on your own, but you don't
0:59really know how the pieces all work
1:01together. You can't tell a good skill
1:03from a bad skill. And this means that
1:04people are trying to piece together
1:06these frameworks, trying to try
1:07everything that's out there all at once.
1:10And they sort of can't, or rather, they
1:12don't get the results that the skills
1:14themselves promise. This is true at an
1:15individual level, but it's also true at
1:17an organization level, too.
1:19Organizations have no way or no
1:21understanding on how to build good
1:23skills, how to take their operating
1:25procedures and turn them into things
1:27that an agent can do. you don't do that,
1:29then it's hard to get the bounty that
1:31skills can offer. Just one more skill,
1:34bro. That's kind of seems like what
1:36we're saying. And I feel a bit of guilt
1:38here, too, because we have Matt Pocock
1:40skills, which is my skills repo, which
1:42is one of the most popular engineering
1:44skill sets out there. And so, I feel
1:46like I want to help the people who use
1:48my skills get out of skill hell. So, how
1:50do we do it? How do we get out of this?
1:51Well, what is actually missing here?
1:54Well, in my opinion, the thing that
1:55we're missing is we don't know what
1:57makes a skill great. We can't yet look
1:59at a skill and go, "Okay, this skill is
2:01doing these good things and these bad
2:03things." There's no shared rubric, no
2:06framework for looking at a skill and
2:07making it better. And so, that's what
2:09I'm going to give you in this talk. I'm
2:10going to give you a skill checklist, a
2:12checklist of things you can look at
2:14inside the skill to make sure that it's
2:16doing what it says it's doing and ways
2:18you can improve it, ways you can write
2:20skills. This checklist looks like this.
2:22We start with the trigger of the skill,
2:24how the skill is invoked, and the
2:26decisions that you need to design there.
2:29Then the internal structure of the
2:31skill, how the skill is actually
2:32composed and laid out internally. Then
2:35number three is how do you actually
2:36steer using the skill? How do you get
2:39the skill to tell the agent what to do?
2:42Then four, how do you make the skill as
2:44small as possible? Because once we've
2:46got a working skill, we then need to
2:49basically maximize it, prune out all of
2:51the irrelevant stuff, prune out all of
2:52the no ops. There's one handy advantage
2:54of me not being in the room with you,
2:56which is you can immediately go and try
2:57this out because I've encoded all of
2:59this into a new skill in my repo called
3:01writing great skills. So, if you've got
3:03an immediate use case for this, then
3:05just go to this my skills repo, you
3:06know, just close this browser, get out
3:08of here, and go and use this skill to
3:11either improve your skills or write
3:13great new ones. But, let's now go
3:14through the checklist then. We have
3:16number one, the trigger, the way the
3:18skill is invoked. And in order to talk
3:19about this, I'm actually going to do a
3:21bit of comparison here, which is that my
3:23skills are often compared to another set
3:25of extremely popular engineering skills
3:27called superpowers. And I'm really often
3:29asked the question, "How do your skills
3:31compare to superpowers? What's the
3:33difference between them?" To understand
3:34that, we need to understand the
3:35difference between user invoked and
3:37model invoked skills. Anytime you have a
3:40skill, you can always invoke it
3:42manually. So, the skill sits on your
3:44file system, the agent will just be able
3:46to pull up the skill and understand
3:48what's in there. And you can always do
3:50that by communicating that to the agent.
3:52Doesn't always look like this forward
3:53slash depending on the harness, but that
3:55you can always use or invoke your
3:57skills. Another way that skills can be
3:59invoked is by the agent itself. These
4:01are called model invocable skills or
4:03model invoked skills. You can take a
4:05description, so the description of the
4:08skill always ends up in the agent's
4:10context, and the agent can look in that
4:13and go, "Okay, based on that
4:14description, I'm going to invoke the
4:16skill and I end up reading the skill.md
4:19file, which is where the meat of the
4:21skill is, into my context window. That's
4:23how you invoke a skill. That's what
4:24happens when a skill is invoked. So,
4:26this description serves as a kind of
4:28context pointer. It sits in the agent's
4:30context pointing to another file where
4:33the agent can go if it wants more
4:35context. But, that context pointer, you
4:36don't need to put it into the agent's
4:39context. It can just be invisible from
4:42the agent, and that is what we call a
4:43user invocable skill. So, some skills
4:46can only be invoked by the user because
4:48they don't have this context pointer.
4:50It's optional. For instance, we can see
4:52in my code base design here, this is a
4:54model invocable skill. It has a
4:56description that ends up in the agent's
4:57context window. But, if we look at my
4:59grill me skill instead, we can see it
5:01has disable model invocation true. This
5:04means that this little description here
5:05will only show to the user. It won't be
5:08visible to the agent. So, this then is
5:09tip number one. Decide if your skill is
5:12user invoked or model invoked. Now, you
5:14might think that model invoked skills
5:16are better, right? Because either the
5:17model can invoke it itself or the user
5:20can invoke it. It's more flexible. But,
5:22every time you add a model invoked skill
5:24into your agent's environment, it
5:27increases what I'm going to call the
5:28context load on that agent. It adds a
5:31new description, which is costing you
5:34tokens on every request, but also adding
5:36a different thing for the agent to think
5:39about. So, if you have a hundred model
5:41invoked skills, that's going to be a
5:42hundred descriptions inside the context
5:45for your agent. So, it seems to make
5:46sense then to either tamp down the
5:48number of model invoked skills or to
5:50just use all user invoked skills. But,
5:53user invoked skills have a different
5:54load, which is the more user invoked
5:57skills you have, the higher cognitive
5:59load on the user. In other words, the
6:00more things the user needs to keep in
6:03their head, the more skill you require
6:05from the pilot. And so, if we compare
6:07Matt Percot skills to Superpowers,
6:10Superpowers is primarily model invoked
6:12skills. It gives the agent superpowers.
6:16Whereas my skills, I much prefer to be
6:18in full control. That means I get to
6:20keep the context load on the agent as
6:22small as possible, but it does impose
6:25more of a cognitive load on me. So, I
6:27need to understand the skills really
6:29deeply in order to get the most use out.
6:31So, why have I done this? Why did I
6:33prefer user invoked skills? Well, every
6:36time you have a model invoked skill, it
6:38basically you get a cost in
6:40unpredictability. Because every time you
6:42have a context pointer pointing from one
6:44resource to another, the model may just
6:46choose not to follow it, you know, even
6:49if it's absolutely perfect for the task,
6:51it may just choose not to invoke the
6:54skill. I much prefer removing that level
6:57of unpredictability, imposing a bit more
6:59cognitive load on the user, and what you
7:01get is just you're removing a class of
7:04problem from even being a problem.
7:06Because this unpredictability leaves
7:07people to need to eval their skills to
7:10make sure they're being called at the
7:11right time, which is really nasty and
7:14it's a problem I prefer to avoid. But,
7:16what I'm hoping to show you here is that
7:17model invoked skills and user invoked
7:19skills both have their same costs. So,
7:22it's not an easy decision which one you
7:24choose. So, that then is the trigger,
7:26how the skill gets invoked. Now, let's
7:28talk about the structure, the internal
7:31layout of the skill. I think of there as
7:32being two main units that you need to
7:34put into most skills. These two units
7:37are the steps and the reference. The
7:40steps are the step-by-step procedure
7:42that the skill is going to walk through
7:45and the reference is any supporting
7:46information that helps it walk through
7:48those steps. You can have skills that
7:51have no steps and are only reference and
7:53you can have skills that are no
7:55reference and only a set of simple steps
7:57to walk through. But if you start
7:58thinking of skills as composed of these
8:00two units, it really helps just break
8:02them down a lot more. If we look at an
8:04example, one of my skills called 2 PRD
8:06creates a product requirements document
8:09out of the current context window. It's
8:10got three steps in it. So it finds the
8:13relevant context, it confirms the test
8:16seams with the user. So there's like a
8:18little human in the loop checkpoint
8:19there just to make sure we're not doing
8:21anything weird with the testing, which I
8:22find really important. And then we write
8:25the product requirements document. To
8:26handle those three steps, we've got two
8:29bits of reference material. We've got a
8:31little bit of reference on what is a
8:33test seam and then we've got a product
8:36requirements document template. So just
8:38a literal markdown template which is
8:41used to write the PRD. So this is a
8:42great way to write a skill from scratch.
8:45You work out if you need some steps,
8:47then you write those steps and you work
8:49out what reference material those steps
8:50need and you put it in a separate little
8:52spot in the skill, which is for
8:54reference material. However, there's a
8:55really important constraint that we need
8:57to think about, which is tip number
8:59three, we want to make the main skill.md
9:02file as small as possible. Every skill
9:05is composed of its description and then
9:07a skill.md file and then any reference
9:09material that branches off that. And
9:11this skill.md file, if we make it small,
9:14then we're saving in a bunch of
9:15different ways. Smaller skills are just
9:17easier to maintain, easier to audit,
9:19fewer words to think about. And every
9:22time you shave off a word, that is a
9:24token shaved, that multiple tokens
9:26shaved from your skills cost. So I do
9:28believe that small skills are really
9:31important both for maintainers and for
9:33users. One really useful way you can
9:35make your skill smaller is by thinking
9:37about the different branches of the
9:39skill, the different ways the skill can
9:41be used. Because if you have reference
9:43material that's only used in one branch,
9:45then that's a candidate for being
9:46removed from the main skill.md. For
9:49instance, if we look at my 2PRD here, we
9:51have two pieces of reference material,
9:53what is a test seam and the PRD
9:55template. Well, we need the PRD template
9:57every single time because we are always
10:00creating a PRD and we probably also need
10:02the what is a test seam information
10:04every time because we're always asking
10:06about the test seams. So, 2PRD, there's
10:08only one branch and all the reference
10:10material belongs on that branch, so it
10:12probably also belongs in the skill.md
10:15file. However, if we look at a different
10:16skill of mine, which is domain modeling,
10:19domain modeling does two things. It
10:21updates a local glossary called
10:23context.md and then it also creates
10:25architectural decision records. In other
10:27words, it's doing two different things
10:30or it might actually choose to do
10:31neither of these, in which case it
10:33doesn't need the template and it doesn't
10:34need the ADR template, either. So, in
10:37other words, domain modeling has two or
10:39maybe three branches and this means that
10:41we don't need to include the ADR
10:43template or the context.md template into
10:46the main skill. They can be moved into
10:49separate zones. The way you do that is
10:51you have the skill.md file, then you put
10:54it behind a context pointer and you
10:56point the context template to a separate
10:59markdown file inside the skills folder.
11:01That context pointer literally just
11:03says, "If you need the template or if
11:04you need to update the context.md file,
11:07go to this file." And I call that an
11:09external reference. It's a reference
11:12that's external to the skill.md that you
11:14can just easily reference the agent can
11:16pull in very easily because it's bundled
11:18along with the skill. So, this is a
11:19technique you can use for making the
11:22skill.md as small as possible, which is
11:24has so many benefits.
11:26Hide branching reference material behind
11:28context pointers. In other words, if you
11:30feel like your skill is going to be used
11:32in lots of different ways, then take the
11:34reference material that's relevant for
11:35those branches and hide them behind
11:37context pointers. So, that is structure.
11:39We need to think about making the
11:41skill.md super duper small. We need to
11:43think about the branches in our skill
11:45moving material out behind context
11:47pointers. And we need to think about
11:49steps and reference, which are the two
11:51main units inside a skill. Let's go next
11:54to steering, the actual ways we get the
11:57agent to do what we want it to do. And
11:58for me, steering comes down to one
12:01really cool technique, which is the kind
12:03of main thing I want you to get from
12:05this talk. This technique fixes this
12:07issue, which is the agent doesn't do
12:10what I want. In other words, I specify
12:12something in the skill, I think that
12:14I've been clear, and then it just
12:15doesn't do the thing. Now, I think the
12:17main reason this happens is because
12:19you're not using a technique called
12:21leading words. The idea of leading
12:23words, or light vert if you like
12:26literary theory, I suppose, is that
12:28there are certain words that pack in a
12:30bunch of meaning into a very small
12:32space. These leading words are really
12:35powerful with agents because you put the
12:37leading word in the skill itself in the
12:40text, and then the agent will repeat the
12:42leading word back to itself as part of
12:44its operations, as part of its thinking
12:46tokens, and as part of its output to
12:48you. And then, because it's
12:50re-emphasizing that word and that word
12:52hopefully describes what you want from
12:54the agent, that then goes and changes
12:56its behavior. Let's make this more
12:58concrete with an example. So, let's
13:00imagine that we have a problem, which is
13:02a classic problem with agents, which is
13:03that they code layer by layer. In other
13:06words, if you give them a big tranche of
13:07work to do, they will generally code up
13:09all of the database layer, then all of
13:11the schemas, then all of the API
13:13endpoints, then all of the front end.
13:15They don't do the sort of typical um
13:18human thing, which is to seek feedback
13:20early on, get something small working,
13:23and then expand out from there. Now, we
13:25can try to encourage the agent to do
13:26that by just saying, you know, don't
13:28code layer by layer, Um make sure that
13:30you create a small slice first and then
13:32go from there. But what if instead we
13:34use the leading word? We said, "vertical
13:36slice" is our leading word. We want to
13:39slice up the work instead of horizontal
13:41slices into vertical slices. A vertical
13:44slice is a pretty well-known terminology
13:46in development, and so this will
13:48hopefully trigger the agent's priors and
13:50it will understand what we mean. We
13:52don't just have to like have a two-word
13:54skill where it just says vertical slice.
13:56What we're doing is we're packing lots
13:57of meaning into a relatively short
14:00phrase that we then repeat throughout
14:02the skill. The cool thing about this
14:03technique is you can know if it's worked
14:05because you say vertical slice in your
14:07skill, and then you'll notice in the
14:09reasoning traces that it's saying,
14:10"Okay, we're going to do this as a thin
14:12vertical slice." Then you should get
14:14better implementation plans. Everyone
14:15I've explained this technique to sort of
14:17feels like, "Oh yeah, I've been doing
14:19that for a while. I've been using these
14:21little phrases to try to encourage the
14:23agent to do what I want." All I'm asking
14:26you now is to use those consistently
14:28within your skills and watch in the
14:31thinking traces as the agent adopts your
14:33way of doing it. So often if the agent
14:35isn't doing what you want, you need to
14:37make your leading words more consistent,
14:40more powerful, and look for others
14:42because, you know, English is a pretty
14:45wide API in terms of different functions
14:47you can call, different things you can
14:49experiment with, and there are many
14:50leading word candidates out there. And
14:52agents are actually pretty good at
14:53helping you think of them. Another
14:55little lever you can use with agents is
14:58sometimes the agent just doesn't do
15:00enough legwork. What I mean by this is
15:02that, okay, we're on a step, let's say,
15:05and maybe the step is to ask clarifying
15:07questions or to explore the code base,
15:09and the agent just doesn't do enough of
15:12it. It doesn't put enough effort into
15:14that particular step. A real classic
15:16case of this and something that I have
15:18found almost everywhere it
15:20is plan mode. Because in plan mode, we
15:23have two steps. We have ask clarifying
15:25questions and then create a plan. And
15:28what I have found in every single
15:29implementation of plan mode I've tried
15:31is that ask clarifying questions just,
15:34you know, it doesn't ever do enough
15:36legwork. It sees that its ultimate goal
15:38is to create a plan and so it just does
15:40a small amount of legwork with ask
15:42clarifying questions, ask you a couple
15:43of things, and then eagerly creates the
15:46plan. So, what was my solution here?
15:48Instead of doing plan mode, I instead
15:50have a skill called grill with docs,
15:52which is kind of my ask clarifying
15:54questions phase. And then, I split that
15:57up into a separate skill. So, I split
15:59the planning into its own skill. So,
16:02grill with docs now is its own skill
16:04where the agent only sees that part of
16:07the process. And then, after grill with
16:09docs completes, we then go and do 2 PRD.
16:12In other words, we have step one and
16:14step two, but the agent only sees one
16:16step at a time. So, this is a really
16:18cool technique for increasing legwork on
16:21the step that you're on by hiding the
16:23future goal, hiding the future steps.
16:25It's not always necessary to split
16:27skills into individual steps, but in
16:31particular cases where you really want
16:33an extra chunk of legwork, it really
16:36there's no technique like it. It works
16:37very, very well. So, that is steering
16:39using leading words to capture what you
16:41want in small reusable tokens and then
16:44making sure that it's doing the right
16:46amount of legwork per step. So, let's
16:48head now into pruning. Now, pruning
16:50really is just a quick fire set of
16:52failure modes, different things that you
16:54can get wrong. And the first is fairly
16:56obvious is we do not want massive
16:59skills. Massive skills are usually a
17:02kind of symptom of something else going
17:04wrong. So, a symptom of one of these
17:05other failure modes. And the first one
17:07is pretty simple. Don't repeat yourself.
17:10You need to make sure you're watching
17:12out for duplication. And in general, I
17:14like to have every part of the skill to
17:17have a single source of truth. In other
17:19words, if you have a piece of reference
17:21material like the PRD template, let's
17:22say, or something even smaller like what
17:25is a test seam, you make sure that you
17:27don't repeat that in several places or
17:29like cover multiple steps in multiple
17:31places. Just make sure each part has a
17:34single source of truth and you're not
17:35repeating yourself even across reference
17:38material, too. The next way that skills
17:39get big is via sediment. And sediment is
17:43just a classic thing when people are
17:46working on the same set of docs, really,
17:48which is that everyone starts
17:50contributing to a shared markdown file.
17:52People add their own stuff. They don't
17:54feel brave enough to delete and modify
17:56anyone else's. And so you just end up
17:57with this huge amount of sediment with
18:00often irrelevant material for the skill,
18:02especially stuff that hasn't been laid
18:04out properly. With a skill with a lot of
18:06sediments, you really need to look at
18:07structure. That's the first thing you
18:09need to do. You need to make sure that
18:10the stuff that's been added is relevant
18:12for all branches. If it's not, then move
18:15it into the correct branches. Or if it's
18:16just totally irrelevant, maybe just
18:18remove it or kill it. Or maybe there's
18:21stuff in there that's totally stale, in
18:22which case you just need to kill it
18:24dead. The next failure mode is really
18:25common when an agent writes your skills,
18:28which are no-ops. So things inside the
18:31skill that appear to do something but
18:34don't actually influence the agent's
18:36behavior inside the context of the
18:37skill. Let's imagine we have an
18:38implement skill and we have an entire
18:40paragraph of the skill that tells the
18:42agent to write a long detailed commit
18:44message. What would happen if you just
18:46deleted that paragraph? Well, the agent
18:49would probably still write a decent like
18:51long commit message. People ask me a lot
18:53how I get my skills so small, and it's
18:56just using these techniques, using
18:58deletion tests, using um making sure
19:00that I compact things into leading
19:02words, I don't have anything irrelevant
19:04in there, and I don't have any sediment.
19:06And that finally brings us to the full
19:08sweep of things. Number one, we check
19:10the trigger. We make sure that it's
19:12firing at the right times. We check
19:14whether we're imposing context load or
19:16cognitive load. With structure, we think
19:18about branches. We think about
19:21structuring things into steps and
19:23reference. And we make sure that
19:25material that's only relevant for one
19:26branch is outside of the main skill.md.
19:29With steering, we're thinking about
19:30condensing text down into leading words
19:33and watching those leading words appear
19:35in the reasoning traces. And we're also
19:37thinking about legwork. Should we break
19:39this skill down further to increase its
19:42focus on the current phase by hiding the
19:44future phase phase from it. And with
19:46pruning, we're doing a final pruning
19:48pass over the entire skill, watching out
19:50for sediments, watching out for crud,
19:52and watching out especially for no-ops.
19:54Now, all of this stuff, the best way to
19:57get started with this framework is
19:58inside this skill, inside the writing
20:00great skills skill. You can check it out
20:03from my papago skills, download it, use
20:05it to improve your own skills, and maybe
20:07even use it to run over some community
20:10authored skills so you can check that
20:12the skills that you're actually pulling
20:14in are any good. If you want to follow
20:15along with my stuff, then I have a
20:17newsletter up on aihero.dev. And my
20:20plans for the next few months are to
20:21release an AI coding crash course, which
20:23is an intro to a lot of the stuff I've
20:25been talking about and how you get off
20:27the ground working with engineering and
20:29AI. I hope that what I've given you is
20:32enough to help you escape from skill
20:34hell or at least try to make the bitter
20:36journey out of there. I'm so sorry not
20:38to be able to attend in person, but
20:40thanks for watching. I'll see you very
20:42soon.

About this transcript

This page contains the full transcript of Building Great Agent Skills: The Missing Manual by AI Engineer, generated from the public captions YouTube serves with the video. The transcript has 4,204 words across 597 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.