No. 120
2026-11-16
the AI pulse — brewed, not raw
 AI CRITIQUE

Northwestern and Google Enable Agents to Self-Improve with Harness2

Read the article    English (automatic translation)

Summary

The Harness2 framework from Northwestern and Google enables an agent's skeleton (prompts, tools, skills, memory) to self-improve on new tasks without answer keys. The system relies on a regulator model proposing changes, testing them, and retaining only those that yield measurable benefits. A single iteration boosted GLM-5's WorkBuddy-Bench score by 11.5 points, surpassing Gemini 3.5 Flash operating with its default skeleton.

Why it matters

Manual skeleton tuning is slow and becomes ineffective when facing new tasks from the real world that lack the correct labels for optimization. Harness2's test-and-keep cycle offers a practical, plug-and-play path for task-based adaptation without requiring human intervention for re-engineering. The negative control experiment—where blind editing caused coding scores to drop from 73.4 to 63.2—emphasizes that the validation run is not optional but a fundamental mechanism of the system.

Highlights

  • The regulator model proposes skeleton corrections; only edits that improve behavior in actual execution are retained
  • Task-level lessons accumulate in a library that guides subsequent edits
  • Skipping the test run step caused coding scores to drop by approximately 10 points, confirming the necessity of the cycle

Critical take

The evidence relies on a single comparison (WorkBuddy-Bench), a single model (GLM-5), and a single improvement step, as summarized in a single X post. Therefore, the generalizability of the results to different tasks, models, and consecutive iterations has not yet been verified. The comparison baseline, Gemini 3.5 Flash running with a 'default' skeleton, is a deliberately weak reference point that exaggerates the gains claimed in the headline.

This critique was written by the local AI model Qwen3.8-27B. I chose the story; I do not review each critique before it is published.
Back to the issue