Inference
The process of running a trained AI model to generate predictions or outputs from new inputs.
Definition
Inference is when a trained AI model processes new input data to produce outputs—whether that's generating text, classifying images, or making predictions. It's the "using" phase as opposed to the "training" phase of AI.
Inference is what happens every time you send a prompt to ChatGPT or ask an AI to analyze an image. The model applies what it learned during training to your specific input.
Why It Matters
Inference costs—measured in compute time and money—are the ongoing expense of using AI. Understanding inference helps optimize prompts, choose appropriate models, and manage AI operating costs.
Faster inference means more responsive applications; cheaper inference means AI can be applied to more use cases cost-effectively.
Examples in Practice
A company runs inference on GPT-4 for complex analysis but uses a smaller, cheaper model for simple classification to manage costs.
An image recognition system runs inference on security camera feeds in real-time to detect anomalies.
A developer optimizes their prompts to reduce token count, cutting inference costs by 40% while maintaining output quality.