Qwen and Taobao launch store-running agent benchmark
Qwen and Taobao & Tmall Group have released E-Commerce Bench, which tests how well AI agents run online stores for a simulated year, a task with no natural stopping point.
Qwen announced E-Commerce Bench, built with Taobao & Tmall Group. Most agent benchmarks score a deliverable produced within a fixed number of turns. This one instead tests ongoing operation: an agent starts with ¥100,000, can run several stores, and spends a simulated year researching categories, negotiating with suppliers, setting prices, and managing inventory and cash flow.
The environment includes 6,886 products across 60 categories, 576 suppliers, 12 store types, 10 market events and 8 promotions. Category sales, return rates and the holiday calendar are desensitized from real platform data. Each simulated day has a 600-minute budget, and every tool call uses some of it. Checking the balance costs 10 minutes and messaging a supplier costs 30. Revenue reaches the bank slowly: it passes through escrow, waits nine days, and then needs a manual withdrawal. Storage fees, return rates and reputation penalties also affect results.
To make runs reproducible, customer demand is computed deterministically from named factors rather than sampled. Suppliers are split into two parts: a deterministic negotiation kernel decides every quote and concession, and an LLM only turns those decisions into dialogue. Qwen says this stops agents from jailbreaking prices below cost.
Final scores cover profit, cash-flow management, negotiation, fraud avoidance, efficiency, execution and learning over the year.
Qwen ran five episodes for each of 18 models, and results differed by orders of magnitude. GPT-5.6 Sol finished at ¥1.43 million, 14.31 times its starting capital. Qwen3.5-Plus averaged ¥1,100. The best open-weight model was Qwen3.8-Max-Preview at 4.16. The available text cuts off there, so the full leaderboard and per-dimension scores are not yet available.