PUNICA

Punica is an open source LLM serving framework that runs multiple LoRA fine-tuned models on a single GPU cluster at the cost of serving just one.

Punica optimizes multi-tenant LLM deployment by consolidating multiple Low-Rank Adaptation (LoRA) models under a single shared base model on a GPU cluster. Using a specialized CUDA kernel design (Segmented Gather Matrix-Vector Multiplication, or SGMV), the system batches GPU operations across different LoRA adapters. This approach eliminates the need to load separate base model weights for every user request, boosting throughput by up to 12x compared to standard serving systems while adding only 2ms of latency per token.

https://github.com/punica-ai/punica