Microsoft and Hugging Face Release ThinkingBox Benchmark to Measure Agent Consistency

Summary

Microsoft and Hugging Face released a benchmark called ThinkingBox. This benchmark scores AI agents not by the tool calls they generate or their final messages, but by the actual database state they leave behind in a 507-state workflow (each run 20 times). A key finding is that 67% of 'clean-looking' attempts still wrote incorrect values, created unwanted side effects, or skipped necessary operations.

Why it matters

Most agent evaluations treat a formally correct tool call or a polite final response as proof of success. However, this benchmark shows that this gap is massive. Agents routinely close tickets as 'resolved' when the required final state is 'pending.' By repeating each task 20 times and reporting pass@1, pass@20, and the strict 20/20 count, it distinguishes what a model can do from what it does reliably. This distinction is critical in production environments, where a 90% success rate means one in ten customers receives a broken outcome.

Highlights

• 67.24% of failed attempts ended cleanly without any tool errors. However, 77.61% of these attempts had incorrect field values in the backend.
• The real differentiator is consistency: GPT-6 Astra retains 78% of its pass@1 performance after 20 repetitions, while GLM-5.1, Kimi-K2.6, and DeepSeek-V4-Pro retain only about 8%.
• Kimi-K3 (open-weight) solves at least one attempt for 93.89% of tasks. However, it passes all 20 attempts in only 13.41% of cases. This reverses the coverage-reliability imbalance seen in proprietary models.

Critical take

The benchmark is limited to five business domains and 507 tasks. Therefore, its rankings say little about coding, research, or multi-agent orchestration workloads. Additionally, the 'observed 20/20' metric is a raw count without a confidence interval. This is an extremely conservative (and arguably binary) measure of reliability. A model with a 95% success rate per attempt will almost never reach this threshold over 20 runs.