Probing the Origins of Reasoning Performance: Representational Quality for Mathematical Problem-Solving in RL vs. SFT Fine-Tuned Models

Summary

The study investigates why models fine-tuned with RL outperform SFT models in mathematical reasoning. Findings indicate that RL makes internal representations more linearly separable and establishes a depth-dependent hierarchical layer structure. In contrast, SFT distributes layer importance equally across layers.

Why it matters

The performance gap between RL and SFT in reasoning is widely observed, yet the underlying mechanism remains unclear. Demonstrating that RL does more than calibrate outputs by reshaping how information is represented directly impacts training pipeline design and interpretability. This finding also contributes to the debate over whether RL teaches structured reasoning or merely rewards superficial response patterns.

Highlights

• Linear probes on layer-based hidden states predict answer accuracy more accurately in RL models, indicating the presence of more structured and separable representations.
• Average ablation reveals that RL models develop a hierarchical architecture where deeper layers become increasingly critical, whereas in SFT models, importance is distributed equally across layers.
• Variability in token counts during repeated sampling is inconsistent among RL models. This suggests that adaptive computation allocation is driven not only by the difference between RL and SFT, but by the entire training pipeline.

Critical take

The claim that "RL fundamentally restructures representations" goes beyond what linear probing and average ablation can prove. This is particularly notable for a workshop-length study. The token distribution results are also explicitly mixed, which undermines the paper's claim of a "clean RL advantage."