IBM: Agent scaffolding on the same model boosted task completion from 30% to 81%

Summary

IBM compared three agent scaffolds—CUGA, Hermes, and Deep Agents—on the same GLM-5.3-flash model across 90 AppWorld tasks. Task completion rates were reported as 81%, 47%, and 30%, respectively. IBM argues that scaffold architecture is as critical as the model itself.

Why it matters

The ecosystem now hosts approximately 200 agent scaffolds. Teams need a principled method to match scaffolds to workloads rather than blindly trusting trending models. The fact that scaffold components like task decomposition, pre-execution validation, and pagination control alone create a 51-point gap indicates that the 'agent reliability' gap is largely an engineering issue, not a model capability one. For enterprises deploying agents in systems with high sensitivity to payments, CRM, or PII, the difference between prompt-level rules and runtime-enforced policies is the difference between a guideline and a guarantee.

Highlights

• Same model, three scaffolds: 81% vs. 47% vs. 30% task completion on AppWorld; the scenario-level consistency (SGC) gap widens further at 20 vs. 8 vs. 2
• CUGA's superiority stems not from a stronger model, but from a cycle of decomposition, CodeAct in a sandbox, pre-execution validation, automatic reflection, and pagination control
• The policy layer (Intent Guard, Tool Approval, ToolGuard, Output Formatter) enforces rules at runtime rather than in the system prompt, ensuring that model hallucinations or prompt injections cannot bypass these controls

Critical take

This is IBM testing its own product in its own lab. The 'controlled experiment' is not entirely controlled. While CUGA utilized task decomposition, validation, reflection, and pagination control, the other two scaffolds used simpler MCP tool configurations. Therefore, part of the 51-point difference likely comes from enabling more features rather than a fundamental architectural superiority. A single benchmark of 90 tasks, the absence of policy enforcement in the scored runs, and the publication venue—a Hugging Face community blog—limit the weight a skeptical reader should place on the headline numbers.