SHAPE analyzes the mathematical reasoning of large language models

Summary

SHAPE decomposes the chain-of-thought (CoT) of large language models (LLMs) into mathematical 'meaning spaces' (algebraic and geometric) and 'heuristics' (such as simplification and working backward). The study demonstrates that heuristic diversity is the best predictor of answer correctness, proving more effective than CoT length. It also reveals that reinforcement learning (RL) fine-tuning narrows the model's heuristic repertoire.

Why it matters

This work provides a theoretically grounded diagnostic tool for understanding why models succeed or fail in mathematics, moving beyond simple accuracy metrics. The finding that RL induces a 'mode-seeking' tendency in heuristic usage is practically significant, suggesting that current fine-tuning may make models less flexible in problem-solving. The intervention method demonstrated, which encourages heuristic diversity, offers a concrete and interpretable lever for improvement.

Highlights

• Mathematical heuristics (e.g., working backward, simplification) predict final answer correctness better than traditional CoT features like token length.
• RL-based fine-tuning leads to a 'mode-seeking' effect: models get stuck in a narrow set of strategies rather than discovering diverse ones.
• Concentration of reasoning effort in a few meaning spaces correlates with correct answers. The authors note that this pattern mirrors human problem-solving processes.

Critical take

This study was only accepted at a workshop (ICML 2026 AI for Math Workshop) and not in the main conference sessions. The meaning space and heuristic classifications are transferred from mathematical education without rigorous validation that they cleanly align with token-level LLM reasoning. The result that 'diverse heuristics increase accuracy' was demonstrated on a limited model set. Therefore, generalizability across different architectures and difficulty levels has not yet been proven.