ML inference

ML inference is the execution phase where a trained machine learning model processes live, unseen inputs to calculate real-time predictions or generate content.

While model training builds the brain, ML inference is that brain at work in production. It is the operational phase where a finalized algorithm processes live data (such as a user prompt, a camera feed, or a financial transaction) to output immediate results like text generation, object detection, or fraud scores. Because production workloads demand low latency and high throughput, optimizing inference involves specialized hardware like GPUs and TPUs alongside software techniques like quantization and pruning to minimize compute costs without sacrificing accuracy.

https://mlcommons.org/en/inference-datacenter-30/