Large language models think by approach, not topic, in mathematics

Summary

The article argues that large language models (LLMs) internally organize mathematical reasoning around reusable 'approaches' (e.g., algebraic transformation, geometric configuration) rather than around specific benchmark topics. Researchers used a generation-replay protocol to extract and cluster activation-importance signatures across eight open models and five math sources.

Why it matters

If the true axis of organization is the reasoning approach, the field's standard practice of stratifying benchmarks by topic and adjusting training rules based on topic balance may amount to optimizing the wrong variable. This necessitates reframing how we diagnose model weaknesses, design curricula for fine-tuning, and create evaluation sets that measure genuine generalization ability rather than superficial topic coverage.

Highlights

• Generation-replay protocol: Replay the full prompt and generation trajectory; cluster unsupervised activation-importance signatures on reasoning tokens.
• Approach-level consistency was confirmed in 77–82% of clusters according to two boundary LLM judges; in-source audits showed this rate remained between 6–11%.
• Changing the desired reasoning approach altered cluster assignment in 7 out of 8 model conditions; mere rephrasing largely preserved this assignment.

Critical take

The validation loop is somewhat circular: whether clusters possess 'approach consistency' is asked of two boundary LLM judges. However, the discrete concept of an 'approach' is never rigorously defined or independently grounded. This leaves the 77–82% rate dependent on the judges' own hidden biases. Additionally, the generation-replay protocol applies a specific interpretive lens to activations. Alternative extraction methods (e.g., probing, causal tracing) could in principle produce different cluster geometries, yet the article does not stress-test this sensitivity.