Artificial Analysis: 24 of 103 inference endpoints underperform

Artificial Analysis released version 1.1 of its Endpoint Accuracy Index. The index measures how much of a model's accuracy each inference provider keeps compared with a reference deployment that the firm hosts itself. It tested 103 serverless endpoints between 8 October and 4 November 2026. Of those, 79 were statistically in line with the reference and 24 scored below it.

The index gives equal weight to three evaluations: BFCL-500 for tool calling, HLE-250 for scientific reasoning and AA-LCR-25 for long-context recall at about 100k input tokens. In v1.1, AA-LCR-25 now runs 25 repeats instead of 10. It also fixes two answer keys and uses an updated LLM grader. The average confidence interval on that test narrows from ±4.7 to ±2.6 points, and the interval on the composite index narrows from ±5.5 to ±4.3. Raw AA-LCR-25 scores from v1.0 and v1.1 are not directly comparable.

Coverage adds GLM-5.3 Flash and Muse Spark Open. DeepSeek V4.1 Flash 1017 replaces DeepSeek V4 Pro, and GLM-5.2 is dropped. Results by model:

• DeepSeek V4.1 Flash 1017: all 22 endpoints in line with the reference
• GLM-5.3: 10 of 26 below, with scores as low as 67.2
• GLM-5.3 Flash: 5 of 22 below
• gpt-oss-120b: 5 of 16 below
• Kimi K3: 2 of 12 below
• Muse Spark Open: 2 of 5 below

Artificial Analysis grouped the causes into four types:

• Output caps of 16K to 32K tokens that cut off HLE-250 reasoning before the model answers
• Fewer reasoning tokens at the same requested settings
• Mishandled tool calls, such as returning object arguments as JSON-encoded text
• Request timeouts and streams that end early

Quantization alone did not explain the gaps. All three GLM-5.3 Flash endpoints serving FP4 weights scored 97.2 to 104.2 against an FP8 reference.