NVIDIA Introduces Mid-Harness Method to Boost Success Rates in Terminal Agents

Summary

NVIDIA's Mid-Harness paper proposes adding a verification step between a large language model (LLM) and a terminal tool (harness). The system samples multiple candidate shell commands, a verifier selects the best one, and then the command is executed. On TerminalBench-Lite, a GPT-5.6 Sol verifier selecting from 8 candidates increases the Pass@1 rate from 50% to 68%.

Why it matters

Terminal agents remain a weak point for LLMs. This study offers a directly applicable test-time compute technique that requires no changes to the model or the tool. The core empirical finding is that verifier quality is far more important than the number of samples. This directs engineering effort toward better verification rather than brute-force sampling. This approach carries direct cost implications for production agent stacks.

Highlights

• On TerminalBench-Lite, using a strong verifier (GPT-5.6 Sol) to select from 8 sampled actions raises the Pass@1 rate from 50% to 68%.
• When a weak verifier is used, increasing the number of samples yields almost no gain. Verification quality is the bottleneck.
• Pairwise comparison and distillation methods enable a small TMAX-9B model to serve as its own verifier. Combining action sampling with trajectory sampling outperforms full trajectory sampling at a lower token cost.

Critical take

The 18-point increase is real, but a 32% error rate remains. Moreover, this result relies on a frontier-class verifier like GPT-5.6 Sol. The benefit quickly disappears when a weaker model is used. This limits the practicality of the method for teams that cannot afford to run a high-end model as a judge at every step. Additionally, labeling this approach a 'mid-harness' architectural contribution may overstate the novelty. The method is fundamentally a sample-verify-select pattern, which is already common in code generation pipelines and is merely repackaged here for shell commands.