# vLLM Projects at AI Tinkerers

> Canonical HTML: https://aitinkerers.org/technologies/vllm
> Markdown URL: https://aitinkerers.org/technologies/vllm.md
> Technology record last updated: 2026-09-18T15:13:59Z
> Generated: 2026-10-06T12:21:46Z

vLLM is the high-throughput, memory-efficient LLM inference engine: it leverages PagedAttention to maximize GPU utilization and cut serving costs.

This is the engine for scaling LLM inference: vLLM (Virtual Large Language Model) is an open-source library engineered for high-throughput and low-latency serving. Its core innovation is PagedAttention, a memory management technique inspired by OS virtual memory, which efficiently handles the Key-Value (KV) cache. This optimization drastically reduces memory overhead—up to 90% in some reported cases—and allows for continuous batching of requests. The result: significantly higher request capacity on the same hardware, lower GPU usage, and a production-ready, cost-effective serving system that supports popular models like Llama and Mistral, complete with an OpenAI-compatible API server.

- Official technology site: https://vllm.ai/
- Public AI Tinkerers demos and talks: 36
- Result page: 1 of 2

## Recent Public Talks and Demos

### [Shisa DE-1: Building "Jev at home" (over a weekend)](https://tokyo.aitinkerers.org/talks/rsvp_5S61x1HvEsg)

TypeSafe released Jev a universal classifier/"decision making" model to much Internet hype and not a lot of architectural details earlier this month. After diving in and playing around with both the closed service and the various open implementations popping up, I built and open-sourced my own over a weekend: Shisa DE-1 (Decision Engine)

- Event context: AI Tinkerers Tokyo — October 5 Meetup — 2026-10-05 — Tokyo
- Public talk page: https://tokyo.aitinkerers.org/talks/rsvp_5S61x1HvEsg

### [Shop Radio - Using Astra to create a physical device](https://manchester-nh.aitinkerers.org/talks/rsvp_iFlFkC-eEXU)

I built an internet radio streaming device using Golang, Raspberry PI4 and GPT Astra. The main application was built at https://github.com/bprendie/subweazl and was ported to raspi using Codex, and the physical design was entirely modeled in FreeCAD using GPT6-Astra, and product redering were done in Blender.

- Event context: AI Tinkerers Manchester (Bedford), NH - September 2026 Meetup — 2026-09-23 — Manchester NH
- Public talk page: https://manchester-nh.aitinkerers.org/talks/rsvp_iFlFkC-eEXU

### [Ubiklaw: A Voice-to-Dashboard Home Agent Running Locally on NVIDIA DGX Spark](https://seattle.aitinkerers.org/talks/rsvp_E55Z1RZPQhw)

Most smart-home dashboards are designed once and slowly go stale. Ubiklaw treats the dashboard itself as something a family can change in the moment. Say, “Put the Front Door and Patio cameras on the left, our ETFs tickers in the middle and Seattle weather on the right.” The tablet transcribes the request, lets you correct it, then publishes a new persistent screen without opening an editor or deploying an app. The model never writes HTML, JavaScript, or provider code. It selects from reviewed layouts, data sources, views and opaque device profiles, then returns a small JSON document. An immutable renderer owns the interface, credentials, network access and failure behavior. A typed publisher validates the complete state against the tablet’s exact viewport before committing a revision. The Whisper voice model, OpenClaw service, Muse Glimmer 30B LLM and the publishing pipeline run locally on one NVIDIA DGX Spark. The resulting displays can combine weather, market data, family lists, Sensibo climate controls and receive-only Ring camera views.

- Event context: AI Tinkerers Seattle: August Demo Night w/ NVIDIA — 2026-08-20 — Seattle
- Public talk page: https://seattle.aitinkerers.org/talks/rsvp_E55Z1RZPQhw

### [LMStack - Development stack using local LLMs](https://seattle.aitinkerers.org/talks/rsvp_A8GvI03RSK8)

Writing real code with local LLMs is difficult. lmstack is a local stack for real development which puts your GPU to work on the right kind of tasks. LMStack introduces an infrastructure manager and a task classifier. Both of them work together to make sure local models are always at work. I've been using this stack for a month to build features on an open source project - Clawrium. LMStack show why the entire stack needs to be different when using local llms and what're the pitfalls of using standard tooling. What works and what to avoid.

- Event context: AI Dev Tools Track - Seattle - August 4 — 2026-08-05 — Seattle
- Public talk page: https://seattle.aitinkerers.org/talks/rsvp_A8GvI03RSK8

### [Meet North Mini Code: Cohere's first model for developers.](https://montreal.aitinkerers.org/talks/rsvp_RFfBetDE0zk)

North Mini Code is a 30B parameter MoE coding model (3B active) that Cohere released June 9 under Apache 2.0, trained specifically for agentic software engineering. We will demo it live in OpenCode so you can watch it work through an agentic coding task end to end. Alongside the live agent session, we'll walk through the architecture and the post-training pipeline that got it there: two stages of SFT followed by async RLVR across terminal and SWE environments.

- Event context: AI Tinkerers Montreal - June Demo Meetup — 2026-06-17 — Montreal
- Public talk page: https://montreal.aitinkerers.org/talks/rsvp_RFfBetDE0zk

### [Pop Goes The Weasel - 'WeazlChat - Incognito Mode for AI'](https://manchester-nh.aitinkerers.org/talks/rsvp_IUzHf7B639g)

Weazlchat is a private, local app that keeps context sessions private. I'll start with an overview of the app, how you can get it and build it yourself, and then how - architecturally - the app can be used. Weazlchat has session history, context compaction, Realtime Stats, Pretty markdown rendering (with Charm Glamour) encrypted session history using bcrypt, Model Agnostic.

- Event context: AI Tinkerers Manchester (Bedford), NH - May 2026 Meetup — 2026-05-20 — Manchester NH
- Public talk page: https://manchester-nh.aitinkerers.org/talks/rsvp_IUzHf7B639g

### [VLMs for Electronics Design](https://raleigh.aitinkerers.org/talks/rsvp_PDqUleDH6rI)

An electronics design tool (think KiCAD or Altium) that leverages VLMs to turn vendor datasheet PDFs into live component models extracted directly from manufacturer reference documentation. The AI Tinkerers demo shows how I built the extraction pipeline using small open weight VLMs deployed on Modal, and a UI built on top of Zed's GPUI framework. The code and benchmark results discussed during the talk are now available on Github as the Parselab project.

- Event context: AI Tinkerers Raleigh Meetup — May 6, 2026 — 2026-05-06 — Raleigh
- Public talk page: https://raleigh.aitinkerers.org/talks/rsvp_PDqUleDH6rI

### [MARSYS: Multi-Agent Workflows Without the Plumbing](https://lausanne.aitinkerers.org/talks/rsvp_eigQG2pH8qI)

MARSYS is an open-source Python framework for building multi-agent workflows. Define your agents and who talks to who -- the framework handles parallel execution, branch isolation, convergence, context management, and routing automatically. It works with 7+ LLM providers out of the box (OpenAI, Anthropic, Google, local models), ships with ready-to-use agents (browser, file operations, code executor, data analyst), and lets you fine-tune local models directly from execution traces. `pip install marsys` and you're running.

- Event context: AI Tinkerers Lausanne April 2026 Meetup — 2026-04-30 — Lausanne
- Public talk page: https://lausanne.aitinkerers.org/talks/rsvp_eigQG2pH8qI

### [Nixus: Orchestrating Agentic Infrastructure](https://hong-kong.aitinkerers.org/talks/rsvp_v6WPm4wWO9M)

Nixus is a set of recipes for orchestrating the building blocks of modern agentic tasks. The talk shows a quick overview of how Nixus serves only the necessary files needed for running a particular agentic flow in an easily composable and more importantly easily swappable system. This is the first time I'll be revealing the Nixus approach and releasing sample recipes.

- Event context: AI Tinkerers Hong Kong at AWS: Agentic AI in Action (April) — 2026-04-29 — Hong Kong
- Public talk page: https://hong-kong.aitinkerers.org/talks/rsvp_v6WPm4wWO9M

### [Scaling RAG: Hybrid Search and Hierarchical Chunking for 780k Pages](https://poland.aitinkerers.org/talks/rsvp_BCaEvuBCHLM)

I built a custom desktop-server search engine designed to help me instantly find and manage documents within my 40GB PDF library. Technical Overview: - The Interface: A Windows application where I can search and browse through the results easily. - The Search Brain: A backend powered by FastAPI that uses "hybrid search" - Data Processing: Python and Bash scripts that handle the heavy lifting, such as pulling Markdown and generating page thumbnails from every file. - Annotation AI: vLLM based LLM server that extract metadata. - The Future: I am currently adding a RAG (Retrieval-Augmented Generation) feature so I can ask the AI questions directly about the content of my documents.

- Event context: AI Tinkerers Poland - Meetup in Gdańsk #1 — 2026-04-23 — Poland
- Public talk page: https://poland.aitinkerers.org/talks/rsvp_BCaEvuBCHLM

### [On-premise AI solution for Cloud PBX provider](https://valencia.aitinkerers.org/talks/rsvp_zguJ0GbdGVs)

On-premise AI transcription. I rebuild faster-whisper lib to make it more efficient for dual-channel transcription. I've built GPU-servers infrastructure for CloudPBX providers with transcirption and analysys.

- Event context: AI Tinkerers Valencia April Meetup — 2026-04-21 — Valencia
- Public talk page: https://valencia.aitinkerers.org/talks/rsvp_zguJ0GbdGVs

### [Virtual model endpoints: fast mode, long context, & unlimited context](https://seattle.aitinkerers.org/talks/rsvp_jFDQjMA0UtY)

We optimize GPU compute for inference. We've stood up several clusters of capacity serving common/popular models. This has pushed us to providing unique variants that solve common context/memory issues and we've provided these in a way that's a simple as changing to another OpenAI compliant model. For development or agentic scenarios, these can enable multi-million context as well as unlimited model context all behind a simple model endpoint.

- Event context: AI Dev Tool Track — Seattle Meetup - April 13th, 2026 — 2026-04-14 — Seattle
- Public talk page: https://seattle.aitinkerers.org/talks/rsvp_jFDQjMA0UtY

### [What If Your OS Was Built for Agents, Not Humans?](https://la.aitinkerers.org/talks/rsvp_7OKp4tR0VHs)

AitherOS is a full operating system kernel for autonomous AI agents. Where traditional frameworks treat agents as stateless function calls, AitherOS gives every agent a persistent identity, a five-tier memory hierarchy, cryptographically signed capability tokens, and native inter-agent communication — the same primitives an OS kernel provides to processes, elevated to the agent abstraction. The system orchestrates 29 specialized agents across 12 architectural layers, from bare infrastructure through cognition, perception, memory, security, and training. Every request passes through intent classification and effort-based model routing, dynamically selecting the right LLM weight class for the task — from a 1B parameter model for simple lookups to a 14B reasoning model for complex planning — reducing inference costs by up to 90% without sacrificing capability. When a problem demands it, the swarm coding engine marshals 11 agents through a four-phase pipeline — architect, build, review, judge — producing tested, reviewed code autonomously. Security is not bolted on. Every agent capability is gated by HMAC-SHA256 signed tokens under a default-deny policy. Agents cannot acquire permissions they were not explicitly granted. The system continuously improves itself through a pain-driven learning loop: execution failures are captured, analyzed, and converted into corrective training examples that feed back into model fine-tuning automatically. AitherOS ships with an open Agent Development Kit supporting the MCP protocol, enabling third-party agents to plug into the full OS stack. The system running at demo.aitherium.com is not a prototype — it is the production deployment.

- Event context: AI Tinkerers LA: March Builder Meetup & Live Demos — 2026-03-20 — Los Angeles
- Public talk page: https://la.aitinkerers.org/talks/rsvp_7OKp4tR0VHs

### [VLLM and Qdrant - GPU goes Brrrr!](https://manchester-nh.aitinkerers.org/talks/rsvp_RGPw96tcjiA)

This demo goes over the fundamentals of VLLM and the QDrant vector database. We'll spin up some Docker containers with the LLM, Database and Embedding model, and then run some interesting benchmarks. I'll demonstrate just how much more powerful VLLM can be on hardware when compared to sequential model runners.

- Event context: AI Tinkerers Manchester (Bedford), NH - March 2026 Meetup — 2026-03-18 — Manchester NH
- Public talk page: https://manchester-nh.aitinkerers.org/talks/rsvp_RGPw96tcjiA

### [Your Brand Translator](https://paris.aitinkerers.org/talks/rsvp__5FOk0bFKFc)

A personal branch is a must for today's professionals and entrepreneurs. But these same people don't have time to do it themselves and services cost thousands. So we build an agent, - based on OpenClaw for easy interaction via WhatsApp, Telegram and Signal - with a ton of image and video editing tool: IG filters, automatic clipping, deep image generation, caption generation, video from image generation, video modifications, music generation, application of any HF-based model - and the ability to schedule the posts via buffer Share your expertise and experience effortlessly.

- Event context: High-Performance Local AI Development: Kick-off ThinkStation PGX — 2026-03-17 — Paris
- Public talk page: https://paris.aitinkerers.org/talks/rsvp__5FOk0bFKFc

### [From Local Prototyping to Distributed Clusters: An Open Source Platform for ML Research Teams](https://toronto.aitinkerers.org/talks/rsvp_nes4M_7YhNc)

Our CEO, Co-Founder Ali Asaria will be there to present. He is the main technical SME on this topic and is best positioned to share and answer questions. We assure you this will not be a pitch and the audience will get technical insights/how-tos/best practices we've learned working with top research labs around the world. We'll demo the use of the tool we built to scale from a local Jupyter notebook to a distributed training run across a cluster of GPUs. We’ll cover how we handled the "boring but critical" parts of the training workflow: automatic checkpoint recovery for spot instances, one-line hyperparameter sweeps, and unified experiment tracking that works across AMD, NVIDIA, and Apple Silicon.

- Event context: AI Tinkerers Toronto - January 2026 Meetup at Google! — 2026-01-29 — Toronto
- Public talk page: https://toronto.aitinkerers.org/talks/rsvp_nes4M_7YhNc

### [Neural Probing LLMs for 'Immediate Inference'](https://montreal.aitinkerers.org/talks/rsvp_qy_wNEb75lc)

'Training' is an ongoing challenge but 'Inference' will be the dominant performance challenge of AI going forward, signalled by Groq acquisition by Nvidia. Autogeneration is slow and expensive, and it's now the dominant 'bottleneck'. 'Probing' or adding novel architectures onto LLMs (Heads) can accelerated inference by measuring the 'state' of an LLM side-stepping the requirement to generate tokens. I don't have a super fancy presentation or clean GitHub yet, it's just lab notes and demo. It actually works, this is serious, not just a toy.

- Event context: AI Tinkerers Montreal - January 2026 Meetup at Ateko — 2026-01-21 — Montreal
- Public talk page: https://montreal.aitinkerers.org/talks/rsvp_qy_wNEb75lc

### [Local hosting - sometimes joy can come in small packages](https://hong-kong.aitinkerers.org/talks/rsvp_cW9hxrT3uxI)

Often people consider AI inferencing as something that can only be done on cloud. And if it's to be done locally, it needs to be done in a large computer case - culminating in a hot and loud system that's sure to keep everyone at home in an annoyed state. I'll be presenting on how local AI inferencing doesn't need to be full of compromises, but also some of the different approaches to building out a system for local hosting. As somebody with a passion for building small form-factor PCs (SFFPCs), I've had a lot of experience building small and space-efficient desktops. Unlike building a SFFPC that can run games quietly and with good temperatures, I'll be showing how an AI-capable SFFPC is actually a simpler affair than building a gaming system. I'll also go through the different ways to host locally, with a particular focus on Windows applications, as well as Ubuntu via WSL. To meet the requirement of applied AI in finance, I will also be showing a new version of the Counsel Intelligence model that runs locally on my desktop.

- Event context: AI Tinkerers Hong Kong & GBA: Applied AI in Finance Demos — 2026-01-20 — Hong Kong
- Public talk page: https://hong-kong.aitinkerers.org/talks/rsvp_cW9hxrT3uxI

### [Document Analytics with Open weight models for Sovereign Data](https://berlin.aitinkerers.org/talks/rsvp_5W86mvFdiH0)

get started with Document analytics in < 100 lines of code. PDF to Image -> VLM for OCR with Image -> Structured JSON Response -> Text to SQL with Tool Call -> Uses Qwen-3-VL-4B-Instruct

- Event context: AI Tinkerers Berlin Meetup - November 12th, 2025 — 2025-11-12 — Berlin
- Public talk page: https://berlin.aitinkerers.org/talks/rsvp_5W86mvFdiH0

### [LLMs for retrieval and recommendation](https://toronto.aitinkerers.org/talks/rsvp_fnoZbMMW_ao)

I will show how to use vLLM and an openweight model to make a simple recommendation engine and use guided decoding to limit the output of the llm to the allowed items only. No finetuning needed and it will work on google colab so basically no hardware needed either. the code i shared will be a bit more, that is just a draft.

- Event context: AI Tinkerers Toronto - October 2025 Meetup at CIBC | Simplii — 2025-10-30 — Toronto
- Public talk page: https://toronto.aitinkerers.org/talks/rsvp_fnoZbMMW_ao

### [Confidential Computing on Multi-GPUs](https://sf.aitinkerers.org/talks/rsvp_ID1Y9xB9IF4)

What was required to run large models such as Deepseek R1 and Qwen3 Coder inside secure hardware enclaves on NVIDIA GPUs and how this wasn’t possible until very recently

- Event context: AI Infra Jam: Demos That Weren’t Possible Two Months Ago — 2025-09-25 — San Francisco
- Public talk page: https://sf.aitinkerers.org/talks/rsvp_ID1Y9xB9IF4

### [Hexagone: Auto Anonymize for AI](https://paris.aitinkerers.org/talks/rsvp_NKaulXDzh9Q)

The session demonstrates Hexagone AI’s tool that automatically redacts personal data in text, images, and audio within PDFs, preserving utility for AI training.

- Event context: Epic Demo night 0.2.1 - Sep 18th, 2025 — 2025-09-18 — Paris
- Public talk page: https://paris.aitinkerers.org/talks/rsvp_NKaulXDzh9Q

### [Claude Coach](https://sf.aitinkerers.org/talks/rsvp_tGa2XRHWhjo)

Better prompts and OSS models for clauding

- Event context: AI HyperEngineering: Claude/Amp Maxxing, Background Agents, CI/CD — 2025-08-22 — San Francisco
- Public talk page: https://sf.aitinkerers.org/talks/rsvp_tGa2XRHWhjo

### [A Hacker's Guide to Slashing LLM Bills](https://orange-county.aitinkerers.org/talks/rsvp_qoTnl7ZATgw)

Generic benchmarks are useless. The performance of an LLM stack depends entirely on your specific traffic patterns. How to decode your workload patterns from production data How to sort through mess of config optimizations including reasoning length, cache length, draft model length, quantization level, input/output length caps, multi-turn aggregation limits, context length limits, quantization value (32 to 1 bit), throughput/latency targets, batching, parallelism settings, TTFT sensitivity, whole sets of caching and other settings, model slug etc. from ex-Intel optimization lead

- Event context: AI Tinkerers - Orange County Meetup- Thursday July 31st 2025 — 2025-08-01 — Orange County
- Public talk page: https://orange-county.aitinkerers.org/talks/rsvp_qoTnl7ZATgw

## Related Technologies

- [Python](https://aitinkerers.org/technologies/python) ([Markdown](https://aitinkerers.org/technologies/python.md)) — 673 public demos
- [Hugging Face](https://aitinkerers.org/technologies/hugging-face) ([Markdown](https://aitinkerers.org/technologies/hugging-face.md)) — 42 public demos
- [llama](https://aitinkerers.org/technologies/llama) ([Markdown](https://aitinkerers.org/technologies/llama.md)) — 48 public demos
- [Docker](https://aitinkerers.org/technologies/docker) ([Markdown](https://aitinkerers.org/technologies/docker.md)) — 150 public demos
- [FastAPI](https://aitinkerers.org/technologies/fastapi) ([Markdown](https://aitinkerers.org/technologies/fastapi.md)) — 183 public demos
- [Transformers](https://aitinkerers.org/technologies/transformers) ([Markdown](https://aitinkerers.org/technologies/transformers.md)) — 149 public demos
- [Kubernetes](https://aitinkerers.org/technologies/kubernetes) ([Markdown](https://aitinkerers.org/technologies/kubernetes.md)) — 35 public demos
- [Modal](https://aitinkerers.org/technologies/modal) ([Markdown](https://aitinkerers.org/technologies/modal.md)) — 14 public demos
- [Ollama](https://aitinkerers.org/technologies/ollama) ([Markdown](https://aitinkerers.org/technologies/ollama.md)) — 78 public demos
- [PyTorch](https://aitinkerers.org/technologies/pytorch) ([Markdown](https://aitinkerers.org/technologies/pytorch.md)) — 273 public demos
- [Qwen](https://aitinkerers.org/technologies/qwen) ([Markdown](https://aitinkerers.org/technologies/qwen.md)) — 18 public demos
- [Anthropic API](https://aitinkerers.org/technologies/anthropic-api) ([Markdown](https://aitinkerers.org/technologies/anthropic-api.md)) — 66 public demos
- [asyncio](https://aitinkerers.org/technologies/asyncio) ([Markdown](https://aitinkerers.org/technologies/asyncio.md)) — 3 public demos
- [CUDA](https://aitinkerers.org/technologies/cuda) ([Markdown](https://aitinkerers.org/technologies/cuda.md)) — 15 public demos
- [Gemma](https://aitinkerers.org/technologies/gemma) ([Markdown](https://aitinkerers.org/technologies/gemma.md)) — 5 public demos
- [Golang](https://aitinkerers.org/technologies/golang) ([Markdown](https://aitinkerers.org/technologies/golang.md)) — 7 public demos
- [IBM Granite](https://aitinkerers.org/technologies/ibm-granite) ([Markdown](https://aitinkerers.org/technologies/ibm-granite.md)) — 4 public demos
- [NVIDIA DGX Spark](https://aitinkerers.org/technologies/nvidia-dgx-spark) ([Markdown](https://aitinkerers.org/technologies/nvidia-dgx-spark.md)) — 5 public demos

## More Results

- Next: https://aitinkerers.org/technologies/vllm.md?page=2