# vLLM Projects at AI Tinkerers

> Canonical HTML: https://aitinkerers.org/technologies/vllm
> Markdown URL: https://aitinkerers.org/technologies/vllm.md
> Technology record last updated: 2026-09-18T15:13:59Z
> Generated: 2026-09-20T15:44:03Z

vLLM is the high-throughput, memory-efficient LLM inference engine: it leverages PagedAttention to maximize GPU utilization and cut serving costs.

This is the engine for scaling LLM inference: vLLM (Virtual Large Language Model) is an open-source library engineered for high-throughput and low-latency serving. Its core innovation is PagedAttention, a memory management technique inspired by OS virtual memory, which efficiently handles the Key-Value (KV) cache. This optimization drastically reduces memory overhead—up to 90% in some reported cases—and allows for continuous batching of requests. The result: significantly higher request capacity on the same hardware, lower GPU usage, and a production-ready, cost-effective serving system that supports popular models like Llama and Mistral, complete with an OpenAI-compatible API server.

- Official technology site: https://vllm.ai/
- Public AI Tinkerers demos and talks: 33
- Result page: 1 of 2

## Recent Public Talks and Demos

### [LMStack - Development stack using local LLMs](https://seattle.aitinkerers.org/talks/rsvp_A8GvI03RSK8)

Writing real code with local LLMs is difficult. lmstack is a local stack for real development which puts your GPU to work on the right kind of tasks. LMStack introduces an infrastructure manager and a task classifier. Both of them work together to make sure local models are always at work. I've been using this stack for a month to build features on an open source project - Clawrium. LMStack show why the entire stack needs to be different when using local llms and what're the pitfalls of using standard tooling. What works and what to avoid.

- Event context: AI Dev Tools Track - Seattle - August 4 — 2026-08-05 — Seattle
- Public talk page: https://seattle.aitinkerers.org/talks/rsvp_A8GvI03RSK8

### [Meet North Mini Code: Cohere's first model for developers.](https://montreal.aitinkerers.org/talks/rsvp_RFfBetDE0zk)

North Mini Code is a 30B parameter MoE coding model (3B active) that Cohere released June 9 under Apache 2.0, trained specifically for agentic software engineering. We will demo it live in OpenCode so you can watch it work through an agentic coding task end to end. Alongside the live agent session, we'll walk through the architecture and the post-training pipeline that got it there: two stages of SFT followed by async RLVR across terminal and SWE environments.

- Event context: AI Tinkerers Montreal - June Demo Meetup — 2026-06-17 — Montreal
- Public talk page: https://montreal.aitinkerers.org/talks/rsvp_RFfBetDE0zk

### [Pop Goes The Weasel - 'WeazlChat - Incognito Mode for AI'](https://manchester-nh.aitinkerers.org/talks/rsvp_IUzHf7B639g)

Weazlchat is a private, local app that keeps context sessions private. I'll start with an overview of the app, how you can get it and build it yourself, and then how - architecturally - the app can be used. Weazlchat has session history, context compaction, Realtime Stats, Pretty markdown rendering (with Charm Glamour) encrypted session history using bcrypt, Model Agnostic.

- Event context: AI Tinkerers Manchester (Bedford), NH - May 2026 Meetup — 2026-05-20 — Manchester NH
- Public talk page: https://manchester-nh.aitinkerers.org/talks/rsvp_IUzHf7B639g

### [VLMs for Electronics Design](https://raleigh.aitinkerers.org/talks/rsvp_PDqUleDH6rI)

An electronics design tool (think KiCAD or Altium) that leverages VLMs to turn vendor datasheet PDFs into live component models extracted directly from manufacturer reference documentation. The AI Tinkerers demo shows how I built the extraction pipeline using small open weight VLMs deployed on Modal, and a UI built on top of Zed's GPUI framework. The code and benchmark results discussed during the talk are now available on Github as the Parselab project.

- Event context: AI Tinkerers Raleigh Meetup — May 6, 2026 — 2026-05-06 — Raleigh
- Public talk page: https://raleigh.aitinkerers.org/talks/rsvp_PDqUleDH6rI

### [MARSYS: Multi-Agent Workflows Without the Plumbing](https://lausanne.aitinkerers.org/talks/rsvp_eigQG2pH8qI)

MARSYS is an open-source Python framework for building multi-agent workflows. Define your agents and who talks to who -- the framework handles parallel execution, branch isolation, convergence, context management, and routing automatically. It works with 7+ LLM providers out of the box (OpenAI, Anthropic, Google, local models), ships with ready-to-use agents (browser, file operations, code executor, data analyst), and lets you fine-tune local models directly from execution traces. `pip install marsys` and you're running.

- Event context: AI Tinkerers Lausanne April 2026 Meetup — 2026-04-30 — Lausanne
- Public talk page: https://lausanne.aitinkerers.org/talks/rsvp_eigQG2pH8qI

### [Nixus: Orchestrating Agentic Infrastructure](https://hong-kong.aitinkerers.org/talks/rsvp_v6WPm4wWO9M)

Nixus is a set of recipes for orchestrating the building blocks of modern agentic tasks. The talk shows a quick overview of how Nixus serves only the necessary files needed for running a particular agentic flow in an easily composable and more importantly easily swappable system. This is the first time I'll be revealing the Nixus approach and releasing sample recipes.

- Event context: AI Tinkerers Hong Kong at AWS: Agentic AI in Action (April) — 2026-04-29 — Hong Kong
- Public talk page: https://hong-kong.aitinkerers.org/talks/rsvp_v6WPm4wWO9M

### [Scaling RAG: Hybrid Search and Hierarchical Chunking for 780k Pages](https://poland.aitinkerers.org/talks/rsvp_BCaEvuBCHLM)

I built a custom desktop-server search engine designed to help me instantly find and manage documents within my 40GB PDF library. Technical Overview: - The Interface: A Windows application where I can search and browse through the results easily. - The Search Brain: A backend powered by FastAPI that uses "hybrid search" - Data Processing: Python and Bash scripts that handle the heavy lifting, such as pulling Markdown and generating page thumbnails from every file. - Annotation AI: vLLM based LLM server that extract metadata. - The Future: I am currently adding a RAG (Retrieval-Augmented Generation) feature so I can ask the AI questions directly about the content of my documents.

- Event context: AI Tinkerers Poland - Meetup in Gdańsk #1 — 2026-04-23 — Poland
- Public talk page: https://poland.aitinkerers.org/talks/rsvp_BCaEvuBCHLM

### [On-premise AI solution for Cloud PBX provider](https://valencia.aitinkerers.org/talks/rsvp_zguJ0GbdGVs)

On-premise AI transcription. I rebuild faster-whisper lib to make it more efficient for dual-channel transcription. I've built GPU-servers infrastructure for CloudPBX providers with transcirption and analysys.

- Event context: AI Tinkerers Valencia April Meetup — 2026-04-21 — Valencia
- Public talk page: https://valencia.aitinkerers.org/talks/rsvp_zguJ0GbdGVs

### [Virtual model endpoints: fast mode, long context, & unlimited context](https://seattle.aitinkerers.org/talks/rsvp_jFDQjMA0UtY)

We optimize GPU compute for inference. We've stood up several clusters of capacity serving common/popular models. This has pushed us to providing unique variants that solve common context/memory issues and we've provided these in a way that's a simple as changing to another OpenAI compliant model. For development or agentic scenarios, these can enable multi-million context as well as unlimited model context all behind a simple model endpoint.

- Event context: AI Dev Tool Track — Seattle Meetup - April 13th, 2026 — 2026-04-14 — Seattle
- Public talk page: https://seattle.aitinkerers.org/talks/rsvp_jFDQjMA0UtY

### [What If Your OS Was Built for Agents, Not Humans?](https://la.aitinkerers.org/talks/rsvp_7OKp4tR0VHs)

AitherOS is a full operating system kernel for autonomous AI agents. Where traditional frameworks treat agents as stateless function calls, AitherOS gives every agent a persistent identity, a five-tier memory hierarchy, cryptographically signed capability tokens, and native inter-agent communication — the same primitives an OS kernel provides to processes, elevated to the agent abstraction. The system orchestrates 29 specialized agents across 12 architectural layers, from bare infrastructure through cognition, perception, memory, security, and training. Every request passes through intent classification and effort-based model routing, dynamically selecting the right LLM weight class for the task — from a 1B parameter model for simple lookups to a 14B reasoning model for complex planning — reducing inference costs by up to 90% without sacrificing capability. When a problem demands it, the swarm coding engine marshals 11 agents through a four-phase pipeline — architect, build, review, judge — producing tested, reviewed code autonomously. Security is not bolted on. Every agent capability is gated by HMAC-SHA256 signed tokens under a default-deny policy. Agents cannot acquire permissions they were not explicitly granted. The system continuously improves itself through a pain-driven learning loop: execution failures are captured, analyzed, and converted into corrective training examples that feed back into model fine-tuning automatically. AitherOS ships with an open Agent Development Kit supporting the MCP protocol, enabling third-party agents to plug into the full OS stack. The system running at demo.aitherium.com is not a prototype — it is the production deployment.

- Event context: AI Tinkerers LA: March Builder Meetup & Live Demos — 2026-03-20 — Los Angeles
- Public talk page: https://la.aitinkerers.org/talks/rsvp_7OKp4tR0VHs

### [VLLM and Qdrant - GPU goes Brrrr!](https://manchester-nh.aitinkerers.org/talks/rsvp_RGPw96tcjiA)

This demo goes over the fundamentals of VLLM and the QDrant vector database. We'll spin up some Docker containers with the LLM, Database and Embedding model, and then run some interesting benchmarks. I'll demonstrate just how much more powerful VLLM can be on hardware when compared to sequential model runners.

- Event context: AI Tinkerers Manchester (Bedford), NH - March 2026 Meetup — 2026-03-18 — Manchester NH
- Public talk page: https://manchester-nh.aitinkerers.org/talks/rsvp_RGPw96tcjiA

### [Your Brand Translator](https://paris.aitinkerers.org/talks/rsvp__5FOk0bFKFc)

A personal branch is a must for today's professionals and entrepreneurs. But these same people don't have time to do it themselves and services cost thousands. So we build an agent, - based on OpenClaw for easy interaction via WhatsApp, Telegram and Signal - with a ton of image and video editing tool: IG filters, automatic clipping, deep image generation, caption generation, video from image generation, video modifications, music generation, application of any HF-based model - and the ability to schedule the posts via buffer Share your expertise and experience effortlessly.

- Event context: High-Performance Local AI Development: Kick-off ThinkStation PGX — 2026-03-17 — Paris
- Public talk page: https://paris.aitinkerers.org/talks/rsvp__5FOk0bFKFc

### [From Local Prototyping to Distributed Clusters: An Open Source Platform for ML Research Teams](https://toronto.aitinkerers.org/talks/rsvp_nes4M_7YhNc)

Our CEO, Co-Founder Ali Asaria will be there to present. He is the main technical SME on this topic and is best positioned to share and answer questions. We assure you this will not be a pitch and the audience will get technical insights/how-tos/best practices we've learned working with top research labs around the world. We'll demo the use of the tool we built to scale from a local Jupyter notebook to a distributed training run across a cluster of GPUs. We’ll cover how we handled the "boring but critical" parts of the training workflow: automatic checkpoint recovery for spot instances, one-line hyperparameter sweeps, and unified experiment tracking that works across AMD, NVIDIA, and Apple Silicon.

- Event context: AI Tinkerers Toronto - January 2026 Meetup at Google! — 2026-01-29 — Toronto
- Public talk page: https://toronto.aitinkerers.org/talks/rsvp_nes4M_7YhNc

### [Neural Probing LLMs for 'Immediate Inference'](https://montreal.aitinkerers.org/talks/rsvp_qy_wNEb75lc)

'Training' is an ongoing challenge but 'Inference' will be the dominant performance challenge of AI going forward, signalled by Groq acquisition by Nvidia. Autogeneration is slow and expensive, and it's now the dominant 'bottleneck'. 'Probing' or adding novel architectures onto LLMs (Heads) can accelerated inference by measuring the 'state' of an LLM side-stepping the requirement to generate tokens. I don't have a super fancy presentation or clean GitHub yet, it's just lab notes and demo. It actually works, this is serious, not just a toy.

- Event context: AI Tinkerers Montreal - January 2026 Meetup at Ateko — 2026-01-21 — Montreal
- Public talk page: https://montreal.aitinkerers.org/talks/rsvp_qy_wNEb75lc

### [Local hosting - sometimes joy can come in small packages](https://hong-kong.aitinkerers.org/talks/rsvp_cW9hxrT3uxI)

Often people consider AI inferencing as something that can only be done on cloud. And if it's to be done locally, it needs to be done in a large computer case - culminating in a hot and loud system that's sure to keep everyone at home in an annoyed state. I'll be presenting on how local AI inferencing doesn't need to be full of compromises, but also some of the different approaches to building out a system for local hosting. As somebody with a passion for building small form-factor PCs (SFFPCs), I've had a lot of experience building small and space-efficient desktops. Unlike building a SFFPC that can run games quietly and with good temperatures, I'll be showing how an AI-capable SFFPC is actually a simpler affair than building a gaming system. I'll also go through the different ways to host locally, with a particular focus on Windows applications, as well as Ubuntu via WSL. To meet the requirement of applied AI in finance, I will also be showing a new version of the Counsel Intelligence model that runs locally on my desktop.

- Event context: AI Tinkerers Hong Kong & GBA: Applied AI in Finance Demos — 2026-01-20 — Hong Kong
- Public talk page: https://hong-kong.aitinkerers.org/talks/rsvp_cW9hxrT3uxI

### [Document Analytics with Open weight models for Sovereign Data](https://berlin.aitinkerers.org/talks/rsvp_5W86mvFdiH0)

get started with Document analytics in < 100 lines of code. PDF to Image -> VLM for OCR with Image -> Structured JSON Response -> Text to SQL with Tool Call -> Uses Qwen-3-VL-4B-Instruct

- Event context: AI Tinkerers Berlin Meetup - November 12th, 2025 — 2025-11-12 — Berlin
- Public talk page: https://berlin.aitinkerers.org/talks/rsvp_5W86mvFdiH0

### [LLMs for retrieval and recommendation](https://toronto.aitinkerers.org/talks/rsvp_fnoZbMMW_ao)

I will show how to use vLLM and an openweight model to make a simple recommendation engine and use guided decoding to limit the output of the llm to the allowed items only. No finetuning needed and it will work on google colab so basically no hardware needed either. the code i shared will be a bit more, that is just a draft.

- Event context: AI Tinkerers Toronto - October 2025 Meetup at CIBC | Simplii — 2025-10-30 — Toronto
- Public talk page: https://toronto.aitinkerers.org/talks/rsvp_fnoZbMMW_ao

### [Confidential Computing on Multi-GPUs](https://sf.aitinkerers.org/talks/rsvp_ID1Y9xB9IF4)

What was required to run large models such as Deepseek R1 and Qwen3 Coder inside secure hardware enclaves on NVIDIA GPUs and how this wasn’t possible until very recently

- Event context: AI Infra Jam: Demos That Weren’t Possible Two Months Ago — 2025-09-25 — San Francisco
- Public talk page: https://sf.aitinkerers.org/talks/rsvp_ID1Y9xB9IF4

### [Hexagone: Auto Anonymize for AI](https://paris.aitinkerers.org/talks/rsvp_NKaulXDzh9Q)

The session demonstrates Hexagone AI’s tool that automatically redacts personal data in text, images, and audio within PDFs, preserving utility for AI training.

- Event context: Epic Demo night 0.2.1 - Sep 18th, 2025 — 2025-09-18 — Paris
- Public talk page: https://paris.aitinkerers.org/talks/rsvp_NKaulXDzh9Q

### [Claude Coach](https://sf.aitinkerers.org/talks/rsvp_tGa2XRHWhjo)

Better prompts and OSS models for clauding

- Event context: AI HyperEngineering: Claude/Amp Maxxing, Background Agents, CI/CD — 2025-08-22 — San Francisco
- Public talk page: https://sf.aitinkerers.org/talks/rsvp_tGa2XRHWhjo

### [A Hacker's Guide to Slashing LLM Bills](https://orange-county.aitinkerers.org/talks/rsvp_qoTnl7ZATgw)

Generic benchmarks are useless. The performance of an LLM stack depends entirely on your specific traffic patterns. How to decode your workload patterns from production data How to sort through mess of config optimizations including reasoning length, cache length, draft model length, quantization level, input/output length caps, multi-turn aggregation limits, context length limits, quantization value (32 to 1 bit), throughput/latency targets, batching, parallelism settings, TTFT sensitivity, whole sets of caching and other settings, model slug etc. from ex-Intel optimization lead

- Event context: AI Tinkerers - Orange County Meetup- Thursday July 31st 2025 — 2025-08-01 — Orange County
- Public talk page: https://orange-county.aitinkerers.org/talks/rsvp_qoTnl7ZATgw

### [Assembly of Experts: Linear-time construction of ultra-large language model variants](https://munich.aitinkerers.org/talks/rsvp_5nHjZ4TDvtA)

Requiring 10^13-10^15 FLOPs to calculate one 8 bit weight in an LLM is inefficient and not beautiful. Therefore, we developed the new "Assembly-of-Experts" construction method. It allows to create meaningful child variants of existing Mixture-of-Experts parent models in linear time. For this, parts of the parent models are recombined, guided by structural and semantic features. Using AoE, we constructed the DeepSeek R1T "Chimera", a 671B open-weights hybrid model combining DeepSeek's V3-0324 and R1 model variants. The child is achieving about R1-level intelligence, while using about 40% fewer output tokens, close to V3. Constructed without any fine-tuning or distillation, the Chimera exhibits surprisingly compact, orderly reasoning compared to its parent models. Moreover, it showed behaviour outside of the distribution, i.e. it is not just an interpolation of the parent models. Community response has been encouraging. The model has 8,200 downloads on Hugging Face and, at peak at the end of May, reached 1.35 billion processed tokens per day on OpenRouter and almost 5 billion tokens per day on chutes.ai. Since then, we have developed the second generation, DeepSeek-TNG R1T2 Chimera. It is a Tri-Mind that also leverages the new R1-0528, and uses a more refined construction method. It appears to be 20% faster than R1, while being almost always more intelligent. It is not quite as intelligent as R1-0528, but typically more than twices as fast, and also with a beautiful chain of thought like R1T. Since R1T2s release on OpenRouter on July 7th, it reached 1 billion tokens per day already on its first full day of operation. It broke 2 B/d on July 17th and grew to become the 5th most popular LLM on Chutes on July 19th. It currently runs on 30 Chutes instances of 8xH200 each, totalling 240 H200 GPUs.

- Event context: AI Tinkerers Munich - July 25 — 2025-07-25 — Munich
- Public talk page: https://munich.aitinkerers.org/talks/rsvp_5nHjZ4TDvtA

### [Sistema Agéntico – Experto Wi-Fi](https://medellin.aitinkerers.org/talks/rsvp_AOo7EbsKuT0)

Presentaremos una arquitectura de inteligencia artificial agéntica, que convierte un modelo de lenguaje pequeño (Mistral 7B) en un agente experto en Wi-Fi. El objetivo es brindar respuestas técnicas precisas en contextos reales de red, con baja latencia y sin depender de modelos grandes en la nube. La solución se basa en destilación agéntica, transfiriendo el conocimiento desde un modelo de gran capacidad (Gemini 2.5) hacia un modelo más eficiente que opera on-premise, habilitado para razonamiento paso a paso (Chain of Thought - CoT) y validado con expertos humanos y herramientas automáticas como RagaAI. Durante la presentación mostraremos el flujo completo de entrenamiento, despliegue en vLLM, integración vía API y ejemplos reales de preguntas respondidas por el agente.

- Event context: AI Tinkerers Medellín #14 - Wizeline - 26 de Junio, 2025 — 2025-06-26 — Medellín
- Public talk page: https://medellin.aitinkerers.org/talks/rsvp_AOo7EbsKuT0

### [Put your idle devices to work and become your own AI cloud](https://liverpool.aitinkerers.org/talks/rsvp_FBkKgRje2gE)

I am the founder of Kalavai, a tool that turns any device into a scalable platform for GenAI. It helps developers aggregate compute from any source (cloud, on prem, laptops) in a unified layer, and manages one-click distributed deployment of AI models. What I'd love to present is our new LLM queuing system, which is an asynchronous batch processing queue that helps developers optimise their workloads at scale, much faster and cost effective than real time inference. Behind the scenes, the queue handles batch requests, whilst workers (any computer) pick inference jobs and run them locally. Workers do just-in-time model deployment, which means there is no model idle time (costly!). And because inference jobs are batched, we can optimise model throughput by configuring batch size. During the presentation, I plan to show how easy it is to run a batch evaluation of multiple models on a test dataset on my computer. This will demonstrate: 1) the power of personal computing devices 2) how queuing maximises hardware utilisation 3) how easy it is to auto deploy models

- Event context: AI Tinkerers - Liverpool Inaugural Meetup (June) — 2025-06-26 — Liverpool
- Public talk page: https://liverpool.aitinkerers.org/talks/rsvp_FBkKgRje2gE

## Related Technologies

- [Python](https://aitinkerers.org/technologies/python) ([Markdown](https://aitinkerers.org/technologies/python.md)) — 662 public demos
- [llama](https://aitinkerers.org/technologies/llama) ([Markdown](https://aitinkerers.org/technologies/llama.md)) — 48 public demos
- [Hugging Face](https://aitinkerers.org/technologies/hugging-face) ([Markdown](https://aitinkerers.org/technologies/hugging-face.md)) — 41 public demos
- [Docker](https://aitinkerers.org/technologies/docker) ([Markdown](https://aitinkerers.org/technologies/docker.md)) — 147 public demos
- [FastAPI](https://aitinkerers.org/technologies/fastapi) ([Markdown](https://aitinkerers.org/technologies/fastapi.md)) — 181 public demos
- [Kubernetes](https://aitinkerers.org/technologies/kubernetes) ([Markdown](https://aitinkerers.org/technologies/kubernetes.md)) — 35 public demos
- [Modal](https://aitinkerers.org/technologies/modal) ([Markdown](https://aitinkerers.org/technologies/modal.md)) — 14 public demos
- [PyTorch](https://aitinkerers.org/technologies/pytorch) ([Markdown](https://aitinkerers.org/technologies/pytorch.md)) — 273 public demos
- [Qwen](https://aitinkerers.org/technologies/qwen) ([Markdown](https://aitinkerers.org/technologies/qwen.md)) — 18 public demos
- [Transformers](https://aitinkerers.org/technologies/transformers) ([Markdown](https://aitinkerers.org/technologies/transformers.md)) — 148 public demos
- [Anthropic API](https://aitinkerers.org/technologies/anthropic-api) ([Markdown](https://aitinkerers.org/technologies/anthropic-api.md)) — 66 public demos
- [asyncio](https://aitinkerers.org/technologies/asyncio) ([Markdown](https://aitinkerers.org/technologies/asyncio.md)) — 3 public demos
- [CUDA](https://aitinkerers.org/technologies/cuda) ([Markdown](https://aitinkerers.org/technologies/cuda.md)) — 15 public demos
- [Gemma](https://aitinkerers.org/technologies/gemma) ([Markdown](https://aitinkerers.org/technologies/gemma.md)) — 4 public demos
- [IBM Granite](https://aitinkerers.org/technologies/ibm-granite) ([Markdown](https://aitinkerers.org/technologies/ibm-granite.md)) — 4 public demos
- [Ollama](https://aitinkerers.org/technologies/ollama) ([Markdown](https://aitinkerers.org/technologies/ollama.md)) — 77 public demos
- [OpenAI API](https://aitinkerers.org/technologies/openai-api) ([Markdown](https://aitinkerers.org/technologies/openai-api.md)) — 520 public demos
- [Qwen3](https://aitinkerers.org/technologies/qwen3) ([Markdown](https://aitinkerers.org/technologies/qwen3.md)) — 9 public demos

## More Results

- Next: https://aitinkerers.org/technologies/vllm.md?page=2