Pruna AI shrinks SmolLM2 sevenfold with quantization
Pruna AI says 4-bit quantization of Hugging Face's SmolLM2 cut memory use about sevenfold with modest perplexity loss, showing how much further already-compact language models can be squeezed for constrained deployment.
Pruna AI reported results from a collaboration with Hugging Face to compress SmolLM2, the compact model family released in 135M, 360M and 1.7B parameter sizes and trained on 11 trillion tokens.
Using its own library, Pruna tested several quantization methods (GPTQ, AWQ, HQQ, Quanto and BitsAndBytes' LLM-Int8) on the 135M-Instruct and 1.7B-Instruct models. The 135M model was run at 2-, 4- and 8-bit precision across all configurations. Testing on the 1.7B model focused mainly on 4-bit, with some 2-bit and 8-bit runs. Evaluation used 1,000 samples from the smol-smoltalk dataset, and methods that need calibration were given 100 training samples.
The company says 4-bit is the best balance. AWQ kept perplexity at 3.82, against 3.38 for the base model, while cutting memory to 106 MB, a reduction of about 7x. Asynchronous latency fell as low as 57.84 ms. At 8-bit, HQQ improved perplexity to 3.25 and brought memory down to 230 MB, about 3x smaller, with synchronous latency of 72.47 ms. At 2-bit, GPTQ reached 94 MB, more than 8x smaller, but perplexity "suffers significantly." Pruna suggests parameter-efficient fine-tuning might recover some of that loss in future work.
The post also says the unquantized base model still had the best asynchronous inference time. The available excerpt is cut off before the full speed comparison, so it does not show where the headline "2x faster" figure comes from. Pruna also said a dedicated DataModule for the evaluation dataset is coming but has not yet been released.