Qwen open-sources FlashQLA linear attention kernels

Qwen has open-sourced FlashQLA, a library of fused linear attention kernels built on TileLang. It targets Gated Delta Network (GDN) chunked prefill. GDN has been the main attention layer across the Qwen family since Qwen3-Next, from Qwen3-Next-80B-A3B through the Qwen3.5 and Qwen3.6 series.

The team says GDN's overhead in training and inference is no longer negligible now that models run at 397A17B, 122A10B, 35B and 27B and context windows go past 256K. On NVIDIA Hopper, FlashQLA reports 2–3× forward speedups and 2× backward speedups over the FLA Triton kernel across several scenarios. Qwen says the gains are largest in pretraining and in on-device agentic inference.

The design addresses two competing problems in FLA's implementation. First, FLA splits the work into separate memory-bound kernels, so intermediates such as W, U and S get written to and read back from HBM. Second, the recurrent state step can launch only batch_size × num_heads thread blocks, which leaves GPUs underused at small batch sizes or under tensor parallelism. Fusing everything into one kernel fixes the first problem but makes the second worse.

FlashQLA takes a middle path. It uses two fused kernels with context-parallel preprocessing between them. It also turns on context parallelism within a single GPU automatically, based on the exponential decay of the GDN gate, under TP, long-sequence and small-head-count settings. Qwen also describes rewriting the math to cut Tensor Core, CUDA Core and SFU overhead "without sacrificing numerical precision." The kernels manually split work across warpgroups so that data movement and computation overlap.

The code and benchmarks are on GitHub at QwenLM/FlashQLA. The source material does not say whether the library supports GPUs other than Hopper.