No. 119
2026-11-15
the AI pulse — brewed, not raw
 AI CRITIQUE

AI Skipped Critical Questions in Tax Test

Read the article    English (automatic translation)

Summary

Surge AI's 'DAYJOB: Tax' benchmark simulated 50 real-world US tax tasks involving up to 1,789 documents. The test revealed that even the best-performing model (Claude Sonnet 5.5) passed only 10.8% of 250 attempts. This result indicates that while current AI systems are reliable in numerical calculations, they consistently fail to determine which questions should be asked before advising a client.

Why it matters

Tax work is a high-risk, document-intensive profession. Missing a single cross-reference can cost clients tens of thousands of dollars in penalties or lost deductions. The benchmark's core finding is that AI excels at the 'answer' part but falls short on the 'what should I verify first?' question. This gap is critical for deploying AI in regulated advisory roles, from law to financial planning. Additionally, the benchmark provides a concrete, repeatable metric (a 95% threshold per item, 5-fold repetition) that the industry can track over time.

Highlights

  • The best model (Claude Sonnet 5.5) passed only 27 out of 250 attempts; all top 10 models remained below 11%
  • In the home sale scenario, GPT-6 Astra failed to flag the 2-year residency deadline in any of 5 runs; Claude Opus 5.5 caught this detail every time
  • In the Roth IRA split task, Claude Opus 5.5 assumed the couple was legally unmarried in all 5 attempts; GPT-6 Astra requested verification of marital status in 4 out of 5 attempts

Critical take

The 10.8% overall success rate is striking. However, the benchmark's design sets an extremely high bar by requiring 95% of items to be passed in a single attempt. This may exaggerate practical error rates. A model that answers 90% of items correctly for a return could be considered 'close' in ways hidden by the binary pass/fail framework. Furthermore, with only 50 tasks and 5 repetitions, confidence intervals for these single-digit percentages are quite wide. Therefore, small ranking differences between models (e.g., 9.6% vs. 9.2%) should not be read as meaningful distinctions.

This critique was written by the local AI model Qwen3.8-27B. I chose the story; I do not review each critique before it is published.
Back to the issue