vLLM for serving rollouts during async RL

vLLM decouples generation and training in RLHF pipelines, using an AsyncLLMEngine to serve rollouts continuously while trainers update weights mid-flight.

Standard reinforcement learning pipelines waste massive amounts of compute because training accelerators sit idle during rollout generation, and vice versa. vLLM solves this bottleneck by running generation and training as parallel coroutines. Using the AsyncLLMEngine alongside native weight-syncing APIs (utilizing NCCL or IPC), vLLM continuously streams rollout data to a shared buffer while the trainer updates the model. When new weights are ready, vLLM pauses generation with a specialized 'keep' mode, swaps the weights mid-flight, and resumes the active generation stream without losing in-flight requests. This asynchronous execution eliminates pipeline bubbles, drastically boosting GPU utilization and training throughput.

https://docs.vllm.ai/en/stable/features/async_rl.html