YouTube2Text

Your SKILL.md Is 70% Stuff the Model Already Knows - Nikolay Petrov, Intel — Transcript

by AI Engineer · 1,723 words · 344 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Hello. My name is Nikolai Petrov and we
  2. 0:02will be talking about skills and agents
  3. 0:04knowledge of those skills.
  4. 0:08We'll start it with the simple prompts
  5. 0:10where we've been
  6. 0:12playing with the new stuff, new work
  7. 0:14being done by the agents, repeating
  8. 0:16those prompts over and over again, and
  9. 0:18then we start packing those into the
  10. 0:20skills,
  11. 0:22creating repeatable, reproducible,
  12. 0:24portable type of their
  13. 0:27functionality
  14. 0:28or capabilities that we can
  15. 0:31teach agents to work with.
  16. 0:33And some of those skills are very
  17. 0:34important, hundreds of stars on GitHub.
  18. 0:40There are different types of their
  19. 0:41skills. Some
  20. 0:43teach models something new, bringing new
  21. 0:45knowledge or capabilities to the model,
  22. 0:48but
  23. 0:50they're also type of their just steering
  24. 0:52or process skills where we
  25. 0:55do not bring any additional information
  26. 0:57but just getting repeat
  27. 0:59table
  28. 1:00flow of the actions that we want agents
  29. 1:02to perform.
  30. 1:04Both are useful. In this session we will
  31. 1:07be talking about new knowledge type of
  32. 1:09the skills
  33. 1:10and
  34. 1:13we'll see what we can
  35. 1:15get from the model's understanding of
  36. 1:17those skills.
  37. 1:21So we get our first skills packed, start
  38. 1:24using them
  39. 1:26basically
  40. 1:28starting with their the feel of
  41. 1:30does it work or not.
  42. 1:32But there is also their more technical
  43. 1:34way and more deterministic way for the
  44. 1:36checking if skills are working or not
  45. 1:38and what's the
  46. 1:39impacts of those skills
  47. 1:41are evolves.
  48. 1:43There are different formats where
  49. 1:44evolves can be defined,
  50. 1:46but at the sense this is task definition
  51. 1:48that you want your agent to be able to
  52. 1:51solve.
  53. 1:53You put it in the place, ideally more
  54. 1:56deterministic, so we can validate it.
  55. 1:58And we can two arms. One where no skill
  56. 2:03loaded and their second one where we're
  57. 2:05loading the skill and providing this
  58. 2:06additional information to the context.
  59. 2:09And we compare how they runs. There are
  60. 2:12multiple things. One is how many
  61. 2:16evolves been solved.
  62. 2:17In this case with no skill we're solving
  63. 2:19three out of five.
  64. 2:21With skill we're solving all five,
  65. 2:23meaning that we get
  66. 2:24two more tasks.
  67. 2:26There also would be implications on the
  68. 2:28tokens usage or time that it took
  69. 2:31those tasks to be completed and other
  70. 2:33factors.
  71. 2:35But at end we are comparing those and
  72. 2:37seeing what makes sense what not.
  73. 2:39And this is one of the main tools for
  74. 2:42evaluating the skills and
  75. 2:44saying that they not just
  76. 2:47nice ideas but actually working pieces
  77. 2:49of instructions and context for the
  78. 2:51agents.
  79. 2:53But what next? We get to the stage there
  80. 2:57we have their working skill. It solved
  81. 2:59some evolves.
  82. 3:01But we know that
  83. 3:03there should be a better way. We have
  84. 3:06models evolving. We have
  85. 3:08some
  86. 3:10maybe cases there we want
  87. 3:14to get skill more efficient, meaning
  88. 3:16consuming less tokens. So what's their
  89. 3:19their options there?
  90. 3:21One is just change line by line and try
  91. 3:25brute force
  92. 3:26all the different combinations.
  93. 3:28Another would be to think through what
  94. 3:31makes sense so maybe use something
  95. 3:35as the user feedback on the skills to
  96. 3:37see where skills perform well there it's
  97. 3:40not.
  98. 3:41And with new models just running skill
  99. 3:45and seeing how it works, what's the
  100. 3:48options there?
  101. 3:49And
  102. 3:50how large those modification should be?
  103. 3:52Just
  104. 3:54lines, sentences, ideas.
  105. 4:00So what
  106. 4:01we found to be useful is instead of
  107. 4:04relying on their luck, we can put more
  108. 4:08deterministic process here
  109. 4:10using idea of their items from their
  110. 4:12fact score.
  111. 4:15We can split their skill and
  112. 4:18instructions into those items
  113. 4:21and
  114. 4:23for each of those have
  115. 4:25pair of question and answer that we
  116. 4:27expect.
  117. 4:29This could be either general ideas like
  118. 4:32proposition in this case or contracts
  119. 4:35there we put more
  120. 4:37definitive expectation on their
  121. 4:40action.
  122. 4:44And in general, we will be getting their
  123. 4:50some predefined amount of the items per
  124. 4:53skill and we will have let say
  125. 4:5725 or 30
  126. 4:59items per skill.
  127. 5:01Then for each of the items, we will
  128. 5:04again run two arms, one without skill
  129. 5:07and one with skill.
  130. 5:10And we will have judge
  131. 5:13or panel of judges that will be
  132. 5:15evaluating model answers. So we asking
  133. 5:17those questions to the model that we
  134. 5:19interested in.
  135. 5:21We evaluating those questions with their
  136. 5:24judge, ideally different family of the
  137. 5:27models or
  138. 5:29panel of judges.
  139. 5:30We will ensure that the question do not
  140. 5:33leak answers.
  141. 5:36We
  142. 5:38depending on the case can compare with
  143. 5:40no tools or additional
  144. 5:42information to check their internal
  145. 5:44model understanding or provided with
  146. 5:47their some tools like web access. So
  147. 5:50it would be more practical.
  148. 5:52And we
  149. 5:55can get their simple forward counts
  150. 5:57matrix.
  151. 6:02Basically where we are checking that the
  152. 6:04model
  153. 6:06uh
  154. 6:07knows or do not know something
  155. 6:09and how this maps to their
  156. 6:12skill impact on this. So the simplest
  157. 6:15option is where model was able to answer
  158. 6:18this particular
  159. 6:20questions even without skill or with
  160. 6:23skill. This means that skill have no
  161. 6:25impact on this and this was already part
  162. 6:27of the model's knowledge.
  163. 6:29The second one is where without skill
  164. 6:32actually model was able to answer this
  165. 6:34correctly.
  166. 6:35And by adding skill
  167. 6:38we saw degradation in performance and
  168. 6:40model capability to
  169. 6:42talk or explain particular aspects.
  170. 6:47The ideal case is where without skill
  171. 6:50model wasn't able to answer something
  172. 6:52and with skill we start seeing that it
  173. 6:54can answer those new
  174. 6:56type of the questions and scenarios and
  175. 6:57this is what we were aiming for teach
  176. 7:00model something new.
  177. 7:02But there also can be there fourth
  178. 7:04option where
  179. 7:06skills were written not
  180. 7:08ideally or in some
  181. 7:12might be not clean way
  182. 7:14where model wasn't able to answer it
  183. 7:16either without skill or with skill and
  184. 7:19while this topic might be relevant and
  185. 7:21important for the model it still sort of
  186. 7:23misses some of those concepts.
  187. 7:28The simplest thing
  188. 7:29to fix are
  189. 7:33instructions or items that actually
  190. 7:35hurts performance and
  191. 7:38for popular skills those are pretty
  192. 7:40rare.
  193. 7:41So in their
  194. 7:44uh
  195. 7:45box below you see that it's just one
  196. 7:48case out of the mostly 400
  197. 7:51items in entropic skills and others
  198. 7:55don't have those.
  199. 7:56But for their smaller skills that are
  200. 7:59less popular you probably having more of
  201. 8:01those. This is what we are seeing with
  202. 8:03some of our um experiments.
  203. 8:06And
  204. 8:08this is just straightforward the
  205. 8:10simplest thing to fix
  206. 8:12to ensure that your skill do not reduce
  207. 8:15model capabilities and performance and
  208. 8:17bring only prevalent
  209. 8:19context there.
  210. 8:22And we also can actually color this in
  211. 8:24this scenario this is done by lines but
  212. 8:27actually going with items you might have
  213. 8:29there
  214. 8:30different coloring for a single line or
  215. 8:32a single sentence.
  216. 8:35And this just gives your example there
  217. 8:39visualization where
  218. 8:40this would be helpful
  219. 8:42details and instructions in terms
  220. 8:45what new stuff you bring to the model
  221. 8:48and what
  222. 8:49is
  223. 8:52something that model already knows.
  224. 8:55Cost wise
  225. 8:56we are running all those questions and
  226. 8:58asking model
  227. 9:00but still relatively this is cheap for
  228. 9:0329 skills 14 bucks of there
  229. 9:06sonnet and haiku
  230. 9:07calls with their sonnet used as a judge.
  231. 9:12And
  232. 9:16there
  233. 9:17couple ways how we also can check their
  234. 9:21strictness of their some knowledge.
  235. 9:25We have
  236. 9:26lenient and strict ways of the checking
  237. 9:28this where first checks the general idea
  238. 9:32while the second one is checking for
  239. 9:35exact details and you can think about
  240. 9:38the difference between general knowledge
  241. 9:40and understanding of the topic to the
  242. 9:43understanding of the exact
  243. 9:45keys parameters and exact values or
  244. 9:48steps.
  245. 9:50And what we see across the models is
  246. 9:53the difference is large
  247. 9:55especially with their
  248. 9:59larger frontier models where we see opus
  249. 10:01dropping from 70% understanding of their
  250. 10:03skill in case of the general
  251. 10:05understanding
  252. 10:06to
  253. 10:07just 22% of the specific
  254. 10:10details.
  255. 10:12And
  256. 10:13this is much our expectations. We see
  257. 10:15that new models are
  258. 10:17pretty well generalized and understand
  259. 10:18concepts.
  260. 10:19But it's still helpful to provide their
  261. 10:22with exact details and data that will be
  262. 10:24useful for their solving those problems.
  263. 10:30Here is outlook of their
  264. 10:34comparing those three popular rappers
  265. 10:36with the skills
  266. 10:38across different models. So one of their
  267. 10:41interesting observations here is that
  268. 10:43smaller model have
  269. 10:45in general lower understanding of their
  270. 10:50skill
  271. 10:52with their larger sonnet having better
  272. 10:54understanding, which is
  273. 10:57why we have bigger models.
  274. 10:59But also we see that overall their
  275. 11:01models already understand a lot of their
  276. 11:08details and context of those skills. And
  277. 11:10you would expect that with the newer
  278. 11:12models and more advances in their
  279. 11:16better training capabilities, we will
  280. 11:18have
  281. 11:19more and more
  282. 11:22in terms of the model's understanding.
  283. 11:24But also we see that this pretty
  284. 11:26significantly differs skill to skill.
  285. 11:29And
  286. 11:31it's not bad. Some skills are intended
  287. 11:32to be directional.
  288. 11:35But it's also useful piece of
  289. 11:37information to see how those
  290. 11:40skills bring new knowledge
  291. 11:43to the different model and different
  292. 11:45families and this should be run ideally
  293. 11:47in the broader
  294. 11:48set of the models.
  295. 11:55But
  296. 11:56just having their
  297. 11:59this novelty piece of information is
  298. 12:03just one of the signals. It's basically
  299. 12:04initial vector for optimization.
  300. 12:06At the end, we still have to run evolves
  301. 12:08and validate that
  302. 12:11all all changes that we
  303. 12:13try to make with their skill actually
  304. 12:15works.
  305. 12:17So the one of their simpler scenarios
  306. 12:20would be we can look on their skill and
  307. 12:23try to remove some or all of their
  308. 12:25already
  309. 12:27known pieces of information from the
  310. 12:29skill and run evaluation.
  311. 12:33It might come up with their same
  312. 12:34scoring. Basically we will be getting
  313. 12:37their same results model still capable
  314. 12:39of solving those problems. Then great we
  315. 12:41can reduce
  316. 12:42skill size, save tokens, save context.
  317. 12:46And instead of their
  318. 12:48brute forcing and guessing what we can
  319. 12:50or should remove
  320. 12:52or add or how to fix their uh
  321. 12:56degradations we can have a guidance.
  322. 13:01So all of those measurements experiments
  323. 13:04been done with the docume which is a
  324. 13:06framework for their comparing
  325. 13:10everything related to the agents and
  326. 13:11context
  327. 13:13where it's is skill, agents MD, MCP
  328. 13:17tools, whatever. And novelty is one of
  329. 13:20their pieces of the functionality where
  330. 13:23you can get this initial vector for
  331. 13:26their skill optimization and evaluation
  332. 13:28but
  333. 13:29there are also uh evals. There is
  334. 13:32assessment of the context impact,
  335. 13:33routing, comparing their
  336. 13:36uh descriptions, naming, bunch of the
  337. 13:38things that related to their
  338. 13:42skills where it would be helpful to
  339. 13:45actually get their more deterministic
  340. 13:46evaluations to make
  341. 13:49conscious decisions in terms of the what
  342. 13:50should be done with the skills.
  343. 13:54So thank you for your time and please
  344. 13:57go ahead and write better skills.

About this transcript

This page contains the full transcript of Your SKILL.md Is 70% Stuff the Model Already Knows - Nikolay Petrov, Intel by AI Engineer, generated from the public captions YouTube serves with the video. The transcript has 1,723 words across 344 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.