Fastino gets 44x LoRA throughput on one L4

Fastino detailed how it redesigned adapter serving for Pioneer, its platform for fine-tuned small models. The company says the change brought a 44x throughput gain on the workload that broke its old setup. It also says a single Nvidia L4 now serves a fleet of 1,024 adapters at about 97 requests per second, with p95 latency of 177 ms.

Pioneer deployments are rarely one model. They usually run tens to hundreds of LoRA adapters on top of GLiNER2, a 205M-parameter bidirectional encoder for entity extraction, classification and relation extraction. All the adapters share one backbone on one GPU. Fastino's previous serving path used PEFT hot-swapping: a 25-adapter least-recently-used (LRU) cache behind a single process-level lock, with one adapter active per forward pass. The post by Matthew Thomas says that once the set of adapters in flight outgrows the cache, every request turns into a disk read. In Fastino's benchmarks, that cut throughput about 44x and raised p95 latency by nearly two orders of magnitude, with no warning beforehand.

The new path uses Punica's Segmented Gather-Matrix-Vector (SGMV) kernel. It groups a batch's tokens by adapter, runs the base matmul once, then applies each adapter's low-rank update in a single fused kernel. Each request no longer has to swap adapters in; instead each token looks up its adapter's weights. The new setup keeps 32 adapters resident on the GPU and 128 more pinned in host memory. On the same hardware, throughput went from 1.83 to 81.3 req/s.

The work runs as a plugin for vllm-factory, a plugin host built on vLLM. It was delivered through three stacked PRs, all now merged, which add the full GLiNER2 task schema and per-request LoRA routing for encoders. The excerpt available here doesn't include the full benchmark setup.