AI CRITIQUE

Görsel, 30 sabit çift içeren bir kelime ekleme görevinde Qwen3.8-27B'nin farklı rakam uzunlukları ve rakam sayılarındaki performansını gösteren bir çubuk grafik sunmaktadır; genel doğruluk oranı %23,57'dir.

Qwen3.8-27B shows low performance in word-based addition test

Read the article English (automatic translation)

Summary

Simon Willison conducted a controlled local benchmark on the Q4_K_M quantized Qwen3.8-27B model. The model was asked to sum positive integers and provide the result in text only. Without a thinking process, the model achieved only 23.57% numerical accuracy. Accuracy was 97% for small numbers but dropped to 6.44% for 10-13 digit numbers. When the thinking process was enabled, accuracy was achieved in 167 out of 169 attempts in a single run.

Why it matters

This test isolates a reproducible failure mode not visible in standard MMLU-style benchmarks: multi-digit arithmetic under strict output format constraints. The sharp contrast between format compliance (96.17%) and numerical accuracy shows that the model 'knows the task' but cannot perform the calculation. This distinction is crucial for anyone developing agents that rely on Large Language Models (LLMs) for precise calculations. Running the test on a consumer-grade DGX Spark makes the findings directly relevant to local LLM implementers rather than just cloud API users.

Highlights

23.57% accuracy without a thinking process versus 167/169 accuracy with it: the chain-of-thought step performs nearly all of the arithmetic work.
Accuracy drops from 97.04% for 1-3 digit numbers to 6.44% for 10-13 digit numbers. This drop of over 90% correlates more with operand length than with task difficulty.
96.17% format compliance indicates the model reliably followed the 'text only' instruction but still calculated the number incorrectly. Format compliance does not imply calculation accuracy.

Critical take

The Q4_K_M quantization is a significant confounding variable: attributing the 6.44% failure rate to the 'Qwen3.8-27B' model rather than the heavily compressed 4-bit checkpoint exaggerates the finding. Additionally, the single-run thinking test (167/169) is not directly comparable to the 30-attempt non-thinking grid. Therefore, the narrative that 'the thinking process solves the problem' is weaker than the headline implies.

Bu kritik, yerel yapay zeka modeli Qwen3.8-27B tarafından yazılmıştır. Haberi ben seçtim; kritik metnini yayımlanmadan önce tek tek denetlemiyorum.This critique was written by the local AI model Qwen3.8-27B. I chose the story; I do not review each critique before it is published.

Back to the issue