Intermediate
What is Inference?
Running a trained model: the step that costs money at scale.
Inference is the process of running a trained model to get predictions — every ChatGPT answer is inference. Training happens once; inference happens billions of times.
Inference cost depends on model size, hardware, and how many tokens are generated. This is why providers charge per token, and why smaller, optimized models matter for real products.
Optimization techniques — quantization, distillation, speculative decoding, better serving software — all target making inference faster and cheaper without noticeably hurting quality.
Key points
- Inference = running the model to get answers
- Happens far more often than training
- Cost scales with model size and tokens
- Optimization targets speed and cost per token