Google Harness2 Explained: How AI Agents Improve Their Own Workflow — Transcript

Full transcript

1. 0:00What if your AI agent needs a better
2. 0:02workflow before it needs a better model?
3. 0:04In Google's Harness 2 paper, GLM-5
4. 0:07running inside Open Code went from 64.2
5. 0:10to 75.7 on Work Buddy Bench. That's 11.5
6. 0:15points with the model's weights
7. 0:16unchanged. Here's what actually changed.
8. 0:20An AI model generates responses. An
9. 0:22agent also needs instructions, tools,
10. 0:25memory, and a process for getting work
11. 0:27done.
12. 0:28That surrounding system is called a
13. 0:30harness. Think of a capable employee.
14. 0:33Their ability matters, but so do their
15. 0:35checklist, filing system, and hand-offs.
16. 0:37Harness 2 works on that surrounding
17. 0:39setup.
18. 0:40The release exposes eight editable
19. 0:42components, including prompts,
20. 0:44guardrails, skills, scripts, and
21. 0:47sub-agents. These are different ways to
22. 0:49shape how the agent works. A new
23. 0:52checklist might change its sequence. A
24. 0:54helper script might handle a repeated
25. 0:56calculation. The model stays fixed while
26. 0:58the operating setup can be revised.
27. 1:01Imagine an agent preparing an expense
28. 1:03spreadsheet from receipts. This is an
29. 1:06illustration, not a test we ran. It
30. 1:08produces a neat-looking file, but misses
31. 1:11a duplicate receipt, and never reopens
32. 1:13the spreadsheet. The question becomes,
33. 1:16what change to its workflow could help
34. 1:18it catch those kinds of mistakes?
35. 1:20Back to our expense example. First, run
36. 1:23the task once and inspect the record of
37. 1:25what the agent did. Then propose two
38. 1:28adjustments. A general habit, like
39. 1:30checking finished output, and a
40. 1:32task-specific procedure, like
41. 1:34reconciling receipt identifiers before
42. 1:37adding totals.
43. 1:38These are candidate edits. They're
44. 1:40hypotheses, not proven improvements.
45. 1:43Next, run both candidate harnesses on
46. 1:46the same task. Compare their actions and
47. 1:49outputs with that first run. Did the new
48. 1:51checking step happen? And did it address
49. 1:54the observed problem? There's no ground
50. 1:56truth answer key guiding refinement. The
51. 1:59system judges behavioral evidence, which
52. 2:01can still mislead it. Benchmark scoring
53. 2:04happens separately afterward.
54. 2:06Then comes compose. The system uses
55. 2:09those execution comparisons to build a
56. 2:11revised harness. It can combine
57. 2:13supported edits, adjust conflicts,
58. 2:16choose one candidate, or keep the
59. 2:18current setup.
60. 2:19It runs the resulting harness, too.
61. 2:21So, the loop is propose, probe, compose.
62. 2:25Change the workflow, observe the
63. 2:26consequences, and decide what to retain.
64. 2:29There's also learning across tasks.
65. 2:32Afterward, a library can retain the
66. 2:34adopted harness and evidence-supported
67. 2:36lessons. But, the next task starts from
68. 2:38the base harness again.
69. 2:40Earlier components are selectively
70. 2:42reused or adapted during refinement.
71. 2:45Think of a growing playbook that helps
72. 2:47choose the next experiment, rather than
73. 2:49a model being retrained after every job.
74. 2:52The authors report gains across three
75. 2:54professional benchmarks and four model
76. 2:56and harness configurations.
77. 2:58That is encouraging evidence for this
78. 2:59approach.
79. 3:00It is still research evidence from
80. 3:02particular setups.
81. 3:04The headline gain isn't a promise for
82. 3:06your spreadsheet, your code base, or
83. 3:08every future task. Model judgment and
84. 3:11noisy executions can still steer edits
85. 3:13badly.
86. 3:14Refinement isn't free. Every candidate
87. 3:17run and editor session adds model
88. 3:19inference.
89. 3:20You can explore alternatives in
90. 3:21parallel, or refine through successive
91. 3:23steps, but both spend extra compute.
92. 3:27When judging efficiency, include the
93. 3:29reference runs, the experiments, the
94. 3:31editing, and the final execution.
95. 3:33Compare deliverable quality with that
96. 3:35complete cost, not just the last run.
97. 3:38One important evaluation detail, the
98. 3:40release documentation records a Codex
99. 3:42validation run that fetched a public
100. 3:44grading rubric without being asked. The
101. 3:47authors recommend network restrictions
102. 3:48before evaluation.
103. 3:50That's a warning about keeping tests
104. 3:52clean. It doesn't establish that every
105. 3:54published result is contaminated. Any
106. 3:56reproduction needs to control what the
107. 3:58agent can access.
108. 3:59If you try this, start with a few
109. 4:01representative tasks, compare the
110. 4:03unchanged baseline, and measure
111. 4:05deliverable quality alongside total
112. 4:07cost. Keep evaluation criteria separate
113. 4:10from refinement.
114. 4:12The useful question is whether the
115. 4:13improved workflow earns its extra work.
116. 4:16If this helped, like it, share it with
117. 4:18someone building AI agents, and comment
118. 4:20with the task you'd test first.