SummaryHugging Face has released an open-source LoRA adapter (Apache 2.0) that fine-tunes Alibaba's Qwen 3.8-27B model using reinforcement learning. Specifically designed for the 'pi' coding agent, this adapter increased the success rate on difficult SWE-Bench Pro tasks from 39% to 65% while reducing the average number of tool calls from 59 to 23. |
Why it mattersThis demonstrates a practical model of maintaining a single base model and swapping in lightweight, agent-specific adapters rather than training a new model for every use case. This approach could lower the cost of deploying coding agents at scale. Reinforcement learning-based training that rewards accuracy and tool-call efficiency points to a more economical path for agentic workflows where token and API costs are dominant. |
Highlights- SWE-Bench Pro (51 difficult tasks): Success rate rose from 39.2% to 64.7%; Terminal-Bench 2.1: Increased from 39.5% to 55.8%.
- Average tool calls dropped sharply: From 59 to 23 on SWE-Bench Pro, and from 18 to 8 on Terminal-Bench 2.1. The number of tokens generated per task decreased by approximately 4.5 times.
- Released under the Apache 2.0 license; runnable via vLLM/llama.cpp. Hugging Face plans to release adapters for other agents (opencode, Hermes) on the same base model.
|
Critical takeAll headline figures come from a single-run evaluation by the development team within the pi test framework. There are no independent repetitions or cross-agent comparisons. While the 25-point increase on SWE-Bench is impressive, the generalizability of this result to other agent frameworks or multi-trial scenarios has not yet been verified. |
This critique was written by the local AI model
Qwen3.8-27B. I chose the story; I do not review each critique before it is published.