Your SKILL.md Is 70% Stuff the Model Already Knows - Nikolay Petrov, Intel — Transcript
Full transcript
- 0:00Hello. My name is Nikolai Petrov and we
- 0:02will be talking about skills and agents
- 0:04knowledge of those skills.
- 0:08We'll start it with the simple prompts
- 0:10where we've been
- 0:12playing with the new stuff, new work
- 0:14being done by the agents, repeating
- 0:16those prompts over and over again, and
- 0:18then we start packing those into the
- 0:20skills,
- 0:22creating repeatable, reproducible,
- 0:24portable type of their
- 0:27functionality
- 0:28or capabilities that we can
- 0:31teach agents to work with.
- 0:33And some of those skills are very
- 0:34important, hundreds of stars on GitHub.
- 0:40There are different types of their
- 0:41skills. Some
- 0:43teach models something new, bringing new
- 0:45knowledge or capabilities to the model,
- 0:48but
- 0:50they're also type of their just steering
- 0:52or process skills where we
- 0:55do not bring any additional information
- 0:57but just getting repeat
- 0:59table
- 1:00flow of the actions that we want agents
- 1:02to perform.
- 1:04Both are useful. In this session we will
- 1:07be talking about new knowledge type of
- 1:09the skills
- 1:10and
- 1:13we'll see what we can
- 1:15get from the model's understanding of
- 1:17those skills.
- 1:21So we get our first skills packed, start
- 1:24using them
- 1:26basically
- 1:28starting with their the feel of
- 1:30does it work or not.
- 1:32But there is also their more technical
- 1:34way and more deterministic way for the
- 1:36checking if skills are working or not
- 1:38and what's the
- 1:39impacts of those skills
- 1:41are evolves.
- 1:43There are different formats where
- 1:44evolves can be defined,
- 1:46but at the sense this is task definition
- 1:48that you want your agent to be able to
- 1:51solve.
- 1:53You put it in the place, ideally more
- 1:56deterministic, so we can validate it.
- 1:58And we can two arms. One where no skill
- 2:03loaded and their second one where we're
- 2:05loading the skill and providing this
- 2:06additional information to the context.
- 2:09And we compare how they runs. There are
- 2:12multiple things. One is how many
- 2:16evolves been solved.
- 2:17In this case with no skill we're solving
- 2:19three out of five.
- 2:21With skill we're solving all five,
- 2:23meaning that we get
- 2:24two more tasks.
- 2:26There also would be implications on the
- 2:28tokens usage or time that it took
- 2:31those tasks to be completed and other
- 2:33factors.
- 2:35But at end we are comparing those and
- 2:37seeing what makes sense what not.
- 2:39And this is one of the main tools for
- 2:42evaluating the skills and
- 2:44saying that they not just
- 2:47nice ideas but actually working pieces
- 2:49of instructions and context for the
- 2:51agents.
- 2:53But what next? We get to the stage there
- 2:57we have their working skill. It solved
- 2:59some evolves.
- 3:01But we know that
- 3:03there should be a better way. We have
- 3:06models evolving. We have
- 3:08some
- 3:10maybe cases there we want
- 3:14to get skill more efficient, meaning
- 3:16consuming less tokens. So what's their
- 3:19their options there?
- 3:21One is just change line by line and try
- 3:25brute force
- 3:26all the different combinations.
- 3:28Another would be to think through what
- 3:31makes sense so maybe use something
- 3:35as the user feedback on the skills to
- 3:37see where skills perform well there it's
- 3:40not.
- 3:41And with new models just running skill
- 3:45and seeing how it works, what's the
- 3:48options there?
- 3:49And
- 3:50how large those modification should be?
- 3:52Just
- 3:54lines, sentences, ideas.
- 4:00So what
- 4:01we found to be useful is instead of
- 4:04relying on their luck, we can put more
- 4:08deterministic process here
- 4:10using idea of their items from their
- 4:12fact score.
- 4:15We can split their skill and
- 4:18instructions into those items
- 4:21and
- 4:23for each of those have
- 4:25pair of question and answer that we
- 4:27expect.
- 4:29This could be either general ideas like
- 4:32proposition in this case or contracts
- 4:35there we put more
- 4:37definitive expectation on their
- 4:40action.
- 4:44And in general, we will be getting their
- 4:50some predefined amount of the items per
- 4:53skill and we will have let say
- 4:5725 or 30
- 4:59items per skill.
- 5:01Then for each of the items, we will
- 5:04again run two arms, one without skill
- 5:07and one with skill.
- 5:10And we will have judge
- 5:13or panel of judges that will be
- 5:15evaluating model answers. So we asking
- 5:17those questions to the model that we
- 5:19interested in.
- 5:21We evaluating those questions with their
- 5:24judge, ideally different family of the
- 5:27models or
- 5:29panel of judges.
- 5:30We will ensure that the question do not
- 5:33leak answers.
- 5:36We
- 5:38depending on the case can compare with
- 5:40no tools or additional
- 5:42information to check their internal
- 5:44model understanding or provided with
- 5:47their some tools like web access. So
- 5:50it would be more practical.
- 5:52And we
- 5:55can get their simple forward counts
- 5:57matrix.
- 6:02Basically where we are checking that the
- 6:04model
- 6:06uh
- 6:07knows or do not know something
- 6:09and how this maps to their
- 6:12skill impact on this. So the simplest
- 6:15option is where model was able to answer
- 6:18this particular
- 6:20questions even without skill or with
- 6:23skill. This means that skill have no
- 6:25impact on this and this was already part
- 6:27of the model's knowledge.
- 6:29The second one is where without skill
- 6:32actually model was able to answer this
- 6:34correctly.
- 6:35And by adding skill
- 6:38we saw degradation in performance and
- 6:40model capability to
- 6:42talk or explain particular aspects.
- 6:47The ideal case is where without skill
- 6:50model wasn't able to answer something
- 6:52and with skill we start seeing that it
- 6:54can answer those new
- 6:56type of the questions and scenarios and
- 6:57this is what we were aiming for teach
- 7:00model something new.
- 7:02But there also can be there fourth
- 7:04option where
- 7:06skills were written not
- 7:08ideally or in some
- 7:12might be not clean way
- 7:14where model wasn't able to answer it
- 7:16either without skill or with skill and
- 7:19while this topic might be relevant and
- 7:21important for the model it still sort of
- 7:23misses some of those concepts.
- 7:28The simplest thing
- 7:29to fix are
- 7:33instructions or items that actually
- 7:35hurts performance and
- 7:38for popular skills those are pretty
- 7:40rare.
- 7:41So in their
- 7:44uh
- 7:45box below you see that it's just one
- 7:48case out of the mostly 400
- 7:51items in entropic skills and others
- 7:55don't have those.
- 7:56But for their smaller skills that are
- 7:59less popular you probably having more of
- 8:01those. This is what we are seeing with
- 8:03some of our um experiments.
- 8:06And
- 8:08this is just straightforward the
- 8:10simplest thing to fix
- 8:12to ensure that your skill do not reduce
- 8:15model capabilities and performance and
- 8:17bring only prevalent
- 8:19context there.
- 8:22And we also can actually color this in
- 8:24this scenario this is done by lines but
- 8:27actually going with items you might have
- 8:29there
- 8:30different coloring for a single line or
- 8:32a single sentence.
- 8:35And this just gives your example there
- 8:39visualization where
- 8:40this would be helpful
- 8:42details and instructions in terms
- 8:45what new stuff you bring to the model
- 8:48and what
- 8:49is
- 8:52something that model already knows.
- 8:55Cost wise
- 8:56we are running all those questions and
- 8:58asking model
- 9:00but still relatively this is cheap for
- 9:0329 skills 14 bucks of there
- 9:06sonnet and haiku
- 9:07calls with their sonnet used as a judge.
- 9:12And
- 9:16there
- 9:17couple ways how we also can check their
- 9:21strictness of their some knowledge.
- 9:25We have
- 9:26lenient and strict ways of the checking
- 9:28this where first checks the general idea
- 9:32while the second one is checking for
- 9:35exact details and you can think about
- 9:38the difference between general knowledge
- 9:40and understanding of the topic to the
- 9:43understanding of the exact
- 9:45keys parameters and exact values or
- 9:48steps.
- 9:50And what we see across the models is
- 9:53the difference is large
- 9:55especially with their
- 9:59larger frontier models where we see opus
- 10:01dropping from 70% understanding of their
- 10:03skill in case of the general
- 10:05understanding
- 10:06to
- 10:07just 22% of the specific
- 10:10details.
- 10:12And
- 10:13this is much our expectations. We see
- 10:15that new models are
- 10:17pretty well generalized and understand
- 10:18concepts.
- 10:19But it's still helpful to provide their
- 10:22with exact details and data that will be
- 10:24useful for their solving those problems.
- 10:30Here is outlook of their
- 10:34comparing those three popular rappers
- 10:36with the skills
- 10:38across different models. So one of their
- 10:41interesting observations here is that
- 10:43smaller model have
- 10:45in general lower understanding of their
- 10:50skill
- 10:52with their larger sonnet having better
- 10:54understanding, which is
- 10:57why we have bigger models.
- 10:59But also we see that overall their
- 11:01models already understand a lot of their
- 11:08details and context of those skills. And
- 11:10you would expect that with the newer
- 11:12models and more advances in their
- 11:16better training capabilities, we will
- 11:18have
- 11:19more and more
- 11:22in terms of the model's understanding.
- 11:24But also we see that this pretty
- 11:26significantly differs skill to skill.
- 11:29And
- 11:31it's not bad. Some skills are intended
- 11:32to be directional.
- 11:35But it's also useful piece of
- 11:37information to see how those
- 11:40skills bring new knowledge
- 11:43to the different model and different
- 11:45families and this should be run ideally
- 11:47in the broader
- 11:48set of the models.
- 11:55But
- 11:56just having their
- 11:59this novelty piece of information is
- 12:03just one of the signals. It's basically
- 12:04initial vector for optimization.
- 12:06At the end, we still have to run evolves
- 12:08and validate that
- 12:11all all changes that we
- 12:13try to make with their skill actually
- 12:15works.
- 12:17So the one of their simpler scenarios
- 12:20would be we can look on their skill and
- 12:23try to remove some or all of their
- 12:25already
- 12:27known pieces of information from the
- 12:29skill and run evaluation.
- 12:33It might come up with their same
- 12:34scoring. Basically we will be getting
- 12:37their same results model still capable
- 12:39of solving those problems. Then great we
- 12:41can reduce
- 12:42skill size, save tokens, save context.
- 12:46And instead of their
- 12:48brute forcing and guessing what we can
- 12:50or should remove
- 12:52or add or how to fix their uh
- 12:56degradations we can have a guidance.
- 13:01So all of those measurements experiments
- 13:04been done with the docume which is a
- 13:06framework for their comparing
- 13:10everything related to the agents and
- 13:11context
- 13:13where it's is skill, agents MD, MCP
- 13:17tools, whatever. And novelty is one of
- 13:20their pieces of the functionality where
- 13:23you can get this initial vector for
- 13:26their skill optimization and evaluation
- 13:28but
- 13:29there are also uh evals. There is
- 13:32assessment of the context impact,
- 13:33routing, comparing their
- 13:36uh descriptions, naming, bunch of the
- 13:38things that related to their
- 13:42skills where it would be helpful to
- 13:45actually get their more deterministic
- 13:46evaluations to make
- 13:49conscious decisions in terms of the what
- 13:50should be done with the skills.
- 13:54So thank you for your time and please
- 13:57go ahead and write better skills.
About this transcript
This page contains the full transcript of Your SKILL.md Is 70% Stuff the Model Already Knows - Nikolay Petrov, Intel by AI Engineer, generated from the public captions YouTube serves with the video. The transcript has 1,723 words across 344 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.