Meta 新论文：RL 优化代码为何常失败

Summary

Meta's new paper shows that simple reinforcement learning for code optimization fails due to noisy and sparse runtime signals and a flawed testing pipeline.

Why it matters

The research emphasizes the necessity of co-designing tests, timing infrastructure, rewards, and GRPO to achieve meaningful speed gains while preserving accuracy. By restructuring the entire feedback loop, Meta raised Qwen 2.5 7B's speed performance from %18 to %31.3 within the top-50% threshold, demonstrating the impact of robust engineering on reinforcement learning outcomes.

Highlights

• Because runtime signals are noisy and sparse, simple reward schemes become ineffective.
• Meta rebuilt the feedback pipeline with larger optimization tests, calibrated remote execution, and accuracy-based rewards.
• Stabilizing GRPO required more solution sampling per request, increasing batch size, and discarding outdated timing data.

Critical take

While the performance boost is impressive, the approach relies heavily on extensive engineering rather than model innovation, raising questions about its scalability and generalizability across different codebases.