Liquid AI releases Pipette on-device benchmark suite

Liquid AI announced Pipette, an open-source platform for benchmarking foundation models on edge devices, built with Artificial Analysis. The premise is that on-device behavior belongs to the whole deployed system, meaning model, quantization, runtime and device together, and not to the model on its own.

The release has five parts:

• A public dataset of lab-verified results covering more than 1,000 configurations. It spans 30+ models, several quantization formats, llama.cpp builds for macOS, iOS, Windows and Android, and context lengths from 256 to 8,192 tokens.
• Device coverage on MacBook Pro (M5 Max), iPhone 17 Pro and Galaxy S26 Ultra so far. AMD Ryzen AI Max+ 395 and Radeon 8060S results are listed as coming soon.
• Open-source benchmark clients for all four platforms, including native iOS and Android apps. Community-submitted results are in beta.
• An interactive dashboard that pairs on-device throughput, latency, context scaling and memory figures with quality scores. The quality scores come from IFBench, GPQA Diamond and MATH-500, run on the quantized model files using H100 reference systems.
• Apache 2.0 infrastructure in three repositories: pipette-mgmt, pipette-clients and pipette-scores.

Liquid points to several tradeoffs in the data. On the Galaxy S26 Ultra at Q4_K_M, Granite-4.0-H-350M keeps 78.4% of its decode throughput when input grows from 256 to 4,096 tokens. Granite-4.0-350M keeps only 33.8%. Liquid's own mixture-of-experts model, LFM2.5-8B-A1B, decodes 2.4x faster than Qwen3.5-4B, but its memory use still peaks at 5.29 GiB. On the iPhone, MiniCPM5-1B finishes the test workload 15.8% faster than LFM2.5-1.2B-Instruct, while LFM scores 9.0 points higher on MATH-500.

Pipette shows these tradeoffs side by side and does not reduce them to a single ranking. Liquid's models appear among those benchmarked.