Fundamental models' geometry drawing ability is being tested

Summary

The article presents an open-source benchmark consisting of 954 olympiad geometry problems (including a difficult subset of 297) that tests whether foundational models can not only arrive at the correct answer but also generate the geometric figures required to solve the problem. A striking gap is observed in current models: strong problem-solving capabilities coexist with an average Asymptote compilation success rate of only 36.14% for the generated figures.

Why it matters

Existing benchmarks like MathVista and MathVerse measure only answer correctness, leaving figure-generation capability unmeasured. This study isolates this competency, which may represent a hidden bottleneck in geometric reasoning. By pairing each problem with a human-created, high-fidelity Asymptote figure and a multimodal metric set (text, code, visual, VLM, constraint-based), it provides a concrete diagnostic tool showing where claims of 'mathematical reasoning' break down. Its open-source release makes the tool immediately available for model developers who want to rigorously test spatial-geometric fidelity beyond multiple-choice accuracy.

Highlights

• 954 problems + 297 hard subset; each paired with a human-created Asymptote figure accepted as correct
• Only a 36.14% average compilation success rate in current foundational models
• A five-metric evaluation set covering text, code, visual, VLM, and constraint-based checks

Critical take

The benchmark conflates geometric understanding with proficiency in a specific DSL by tying correct figure representation solely to Asymptote. A model that excels in spatial reasoning but struggles to write Asymptote code is penalized in the same way as a model that directly misidentifies the solution. Furthermore, the 36% compilation rate is a coarse average; without problem-level error classification, it is difficult to distinguish whether errors stem from incorrect structure, syntax mistakes, or missing auxiliary lines. This limits the diagnostic precision claimed by the paper.