# cuBLASLt Projects at AI Tinkerers

> Canonical HTML: https://aitinkerers.org/technologies/cublaslt
> Markdown URL: https://aitinkerers.org/technologies/cublaslt.md
> Technology record last updated: 2026-03-19T06:02:52Z
> Generated: 2026-09-22T23:36:47Z

cuBLASLt is NVIDIA's lightweight, high-performance library for highly-optimized General Matrix-to-matrix Multiply (GEMM) operations.

cuBLASLt (cuBLAS Light) delivers maximum throughput for critical deep learning and HPC workloads by focusing exclusively on advanced GEMM operations (Level 3 BLAS). It provides a flexible, multi-stage API that enables developers to programmatically select optimal algorithms and heuristics for specific GPU architectures, such as leveraging Tensor Cores on NVIDIA A100 or H100 GPUs. The library is engineered for low-latency kernel launches and supports mixed-precision compute (FP16, BF16, TF32, INT8), often employing kernel fusion to combine multiple operations and minimize overhead. This targeted optimization makes it the go-to tool for accelerating large-scale AI training and inference models.

- Official technology site: https://docs.nvidia.com/cuda/cublas/index.html
- Public AI Tinkerers demos and talks: 1
- Result page: 1 of 1

## Recent Public Talks and Demos

### [Slashing GPU Kernel Launch Overhead in LLM Inference](https://berlin.aitinkerers.org/talks/rsvp_ghQq_D2zfys)

LLM inference often involves millions of kernel launches. The token-by-token nature of decode workloads bombards the GPU with smaller, sequential operations, creating a hidden bottleneck in driver overhead, not compute. We'll demonstrate a multi-process Rust application that decouples inference logic from GPU execution using a lock-free shared memory queue. This architecture enables intelligent, on-the-fly batching of identical operations into a single, efficient cuBLASLt call. The results are significant: we'll show a 90%+ reduction in kernel launches and a 22% speedup on a realistic FP16 decode workload. We'll also explore why this same technique results in a slight slowdown for compute-bound prefill workloads, providing a nuanced, first-principles look at a core optimization used by all major inference engines. Update: GitHub repo added

- Event context: AI Tinkerers Meetup August 13 — 2025-08-13 — Berlin
- Public talk page: https://berlin.aitinkerers.org/talks/rsvp_ghQq_D2zfys

## Related Technologies

- [Boost](https://aitinkerers.org/technologies/boost) ([Markdown](https://aitinkerers.org/technologies/boost.md)) — 1 public demo
- [CUDA](https://aitinkerers.org/technologies/cuda) ([Markdown](https://aitinkerers.org/technologies/cuda.md)) — 15 public demos
- [Folly](https://aitinkerers.org/technologies/folly) ([Markdown](https://aitinkerers.org/technologies/folly.md)) — 1 public demo
- [LMAX Disruptor](https://aitinkerers.org/technologies/lmax-disruptor) ([Markdown](https://aitinkerers.org/technologies/lmax-disruptor.md)) — 1 public demo
- [Rust](https://aitinkerers.org/technologies/rust) ([Markdown](https://aitinkerers.org/technologies/rust.md)) — 56 public demos
- [Shared memory](https://aitinkerers.org/technologies/shared-memory) ([Markdown](https://aitinkerers.org/technologies/shared-memory.md)) — 1 public demo