qat-suite

A lightweight quantization suite built to optimize large language models for resource-constrained edge environments.

Developed by SwissAI, qat-suite is an open-source quantization toolkit designed to compress large language models (LLMs) for efficient deployment on hardware-limited devices. It provides native support for popular inference formats like vLLM and Apple MLX, utilizing advanced quantization-aware distillation (QAD) algorithms to generate ultra-low-bit formats (including INT2, INT3, INT4, and INT6). By maintaining high model accuracy while drastically reducing memory footprints, the suite serves as a critical bridge for running complex multilingual models on mobile and edge platforms.

https://github.com/swiss-ai/qat-suite