Artificial Analysis launches Speech Agent Arena
Artificial Analysis has launched a human-judged arena for speech-to-speech agents, and its first results show the model users most prefer talking to is not the one that most reliably completes tasks.
Artificial Analysis announced the Speech Agent Arena, a benchmark that tests native and cascaded speech-to-speech systems while humans complete real-world tasks with them. It measures two things: which model participants prefer to talk to, and whether the model completes the requested action.
Paid, screened third-party participants hold separate live conversations with two hidden models on the same scenario. There are 15 agentic scenarios that require tool calls, such as ordering takeout, and 20 non-agentic ones, such as asking about opening hours. Pairwise votes produce a Preference Elo score. For agentic scenarios, Task Success Rate is the share of eligible conversations in which the model made the correct final tool call or calls. Except for one dental-booking example, the prompts, tool schemas and participant instructions are private to reduce overfitting.
Google's Gemini 3.1 Flash Live Preview (Minimal) leads on preference at 1,046 Elo. It is followed by the same model on High at 1,014, OpenAI's GPT-Realtime-1.5 at 1,000, GPT Realtime (Aug '25) at 944 and ElevenLabs Agents at 937. Gemini's task success rate, however, is 74.6%. Artificial Analysis notes that some conversations can sound as if a task was completed when the required tool call failed.
On task success, SpaceXAI's Grok Voice Think Fast 2.0 High leads at 94.7%, followed by GPT-Realtime-2.1 High at 91.5% and ElevenLabs Agents at 90.5%.
Preference generally rose as time to first audio fell. Gemini's was 0.96 seconds, while Qwen Audio 3.0 Realtime Plus took 1.54 seconds and scored 699 Elo. Prices per hour of input audio vary widely: $1.50 for Gemini, $4.80 for Grok and $10.75 for GPT-Realtime-2.1 High.