Qwen Releases E-Commerce Bench for Long-Term Autonomous Business Benchmarking

Summary

Qwen (Alibaba) released the E-Commerce Bench benchmark. In this evaluation, AI agents manage a virtual online store for 365 days in a market built from real e-commerce data. Agents start with ¥100,000 in capital and manage procurement, pricing, inventory, and cash flow. The key finding is that almost none of the tested models improved their purchasing or strategic decisions over the course of the year.

Why it matters

Long-term, multi-step business operations are the area where current large language model (LLM) agents struggle most visibly. A 365-day simulation including scammers, storage fees, and reputation mechanisms moves evaluation far beyond single-turn question-answering benchmarks. Scoring across seven axes, rather than a single profit figure, reveals trade-offs such as aggressive pricing versus inventory risk. While a single-metric leaderboard hides these nuances, this approach offers researchers a more honest picture of where autonomous trading agents still fall short.

Highlights

• The simulated market features 6,886 products, 576 suppliers (152 of which are scammers), 600-minute workdays, storage fees, returns, and a reputation mechanism.
• Almost no model learned to buy cheaper or improve its strategy over the full 365-day period.
• The seven-axis evaluation shows that no single model dominates across all dimensions, highlighting fragmented strengths rather than a clear winner.

Critical take

The benchmark was self-published by Qwen/Alibaba. The 57/100 editorial score on the source site indicates limited independent verification. Without a third-party re-test or a clear evaluation rubric for the six secondary axes, the conclusion that 'no model dominates' may partly stem from benchmark design choices that favor the authors' own model. Additionally, even if seeded with real data, a simulated market cannot fully reflect the adversarial, noisy, and legally constrained nature of real e-commerce. Therefore, the 365-day results should be read as a stress test, not proof of real-world applicability.