SummaryThe Harness2 framework from Northwestern and Google enables an agent's skeleton (prompts, tools, skills, memory) to self-improve on new tasks without answer keys. The system relies on a regulator model proposing changes, testing them, and retaining only those that yield measurable benefits. A single iteration boosted GLM-5's WorkBuddy-Bench score by 11.5 points, surpassing Gemini 3.5 Flash operating with its default skeleton. |
Why it mattersManual skeleton tuning is slow and becomes ineffective when facing new tasks from the real world that lack the correct labels for optimization. Harness2's test-and-keep cycle offers a practical, plug-and-play path for task-based adaptation without requiring human intervention for re-engineering. The negative control experiment—where blind editing caused coding scores to drop from 73.4 to 63.2—emphasizes that the validation run is not optional but a fundamental mechanism of the system. |
Highlights- The regulator model proposes skeleton corrections; only edits that improve behavior in actual execution are retained
- Task-level lessons accumulate in a library that guides subsequent edits
- Skipping the test run step caused coding scores to drop by approximately 10 points, confirming the necessity of the cycle
|
Critical takeThe evidence relies on a single comparison (WorkBuddy-Bench), a single model (GLM-5), and a single improvement step, as summarized in a single X post. Therefore, the generalizability of the results to different tasks, models, and consecutive iterations has not yet been verified. The comparison baseline, Gemini 3.5 Flash running with a 'default' skeleton, is a deliberately weak reference point that exaggerates the gains claimed in the headline. |
This critique was written by the local AI model
Qwen3.8-27B. I chose the story; I do not review each critique before it is published.