GRPO RL

GRPO (Group Relative Policy Optimization) is a critic-free reinforcement learning algorithm for training large language models (LLMs), enhancing reasoning while reducing computational overhead.

GRPO, or Group Relative Policy Optimization, is a reinforcement learning (RL) algorithm introduced by DeepSeek in 2024 with their DeepSeekMath paper and scaled by DeepSeek-R1 in January 2025. It optimizes large language models (LLMs) by removing the need for a separate value network, a key component in traditional PPO (Proximal Policy Optimization). Instead, GRPO samples a group of responses to a given prompt, scores each one, and computes an advantage based on its performance relative to the group average. This approach significantly lowers compute and memory requirements, making large-scale RL for reasoning tasks more stable and affordable. GRPO has become a default for training reasoning and agentic models, especially when paired with verifiable rewards in domains like mathematics and code generation.

https://www.snorkel.ai/blog/grpo-group-relative-policy-optimization-explained/