AI Inference

Inference is the moment an already trained AI model gets to work: it receives a new input (a text, a question, an image, a row of data) and produces an output (an answer, a classification, a prediction). It is the phase in which the model is used, as opposed to the training phase, where the model learns by adjusting its parameters from examples.

Put simply: training is teaching the model; inference is asking it. When an AI agent answers a question, drafts an email or decides which action to take, it is performing inference. Every answer from a conversational assistant, every classified document and every lead scored by a model is an act of inference.

It matters because inference is what the business actually consumes and what determines cost and speed in production. A model is trained once (or every so often), but it runs inference millions of times: that is why latency, spend per query and the quality of each answer depend on how this phase is managed.

How inference works

An AI model, especially an LLM, is essentially a mathematical function with millions or billions of parameters that were locked in during training. During inference that function does not change: it only runs. The system takes the input, converts it into tokens, processes it through the model's layers and generates the output token by token, predicting each new word from the previous one and from the context available within the context window.

Two parameters govern this stage. Temperature controls how deterministic or creative the output is: low for precise tasks, high for more varied generation. And the prompt, along with any data injected through RAG, defines the context the model reasons over. None of this modifies the model: they are inputs that condition the same fixed function.

Why it matters for the business

Inference is the part of AI that scales with usage, and that is why it concentrates the operating cost. Training a large model is expensive but happens rarely; inference happens in every customer interaction, in every incoming document and in every automated decision. For an Argentine company that puts a customer service assistant on WhatsApp, what it pays month after month is inference: every conversation consumes input and output tokens. Optimizing this phase (choosing a model of the right size, limiting the context, caching frequent answers) has a direct impact on margin.

Speed is also decided here. Inference latency is the time the model takes to start responding and to complete the response. In a conversational experience, a delay of several seconds breaks the sense of flow; in a batch process (classifying 10,000 invoices from one day), what matters is total throughput, not the instant response.

A concrete example in Argentina

A consumer goods distributor licenses an already trained model to classify the orders that come in by email and WhatsApp. Once in production, every new email triggers an inference: the model reads the text, identifies product, quantity and customer, and creates the order in the system. The team never retrains the model day to day; it just uses it, thousands of times a week. The monthly cost and the speed of each order are, in practice, inference metrics.

Inference vs training

Common mistakes

• Confusing inference with training. Asking a model during inference to "learn" from what you tell it in the chat is a frequent misunderstanding: it retains nothing between sessions except what comes in through its context. For it to truly learn, you need fine-tuning or retraining.
• Underestimating the accumulated cost. A cheap pilot can become expensive at scale, because inference multiplies with every interaction.
• Ignoring latency. A larger model usually gives better answers but runs inference more slowly; in conversational use cases, a smaller, faster one is sometimes the better choice.
• Forgetting data governance. What gets sent in each inference (customer data, sensitive documents) travels to the model: that is why where and how it runs matters.