LLMOps in Practice

Getting an AI feature working is the easy part. A prompt, an API call, a demo that impresses people in a meeting. The hard part starts when it has to run every day, for real users, without anyone watching it constantly.

That gap is what LLMOps covers. It is not a new discipline so much as ordinary operations applied to a component with an awkward property: it can fail without erroring. Your dashboards stay green while the answers quietly get worse. Everything below follows from that one fact.

Evaluation comes first

Before deployment, before monitoring, before anything else. If you cannot tell a good output from a bad one automatically, you cannot safely change anything, and you will not know when something has degraded.

Start with a dataset. Fifty to two hundred real inputs with known good outputs, drawn from actual usage rather than invented. Include the awkward cases: the ambiguous question, the one where the right answer is "I do not know", the one that previously went wrong.

Then pick metrics that suit the task.

For anything with a correct answer, such as classification, extraction or routing, use accuracy and inspect the confusion matrix. This is the easy case and worth engineering towards where you can.

For retrieval, measure whether the right document appeared and where it ranked. Keep this separate from answer quality; conflating them makes failures impossible to diagnose.

For open ended generation, use assertions where possible and a model as judge where not. Assertions are cheap and reliable: is it valid JSON, does it contain a citation, is it under the length limit, does it avoid the phrases you banned. A judging model handles the subjective remainder, but calibrate it against human ratings on a sample before trusting it, and be aware that it costs money on every evaluation run.

Run the suite in CI on every change to a prompt, a model or retrieval logic. A prompt edit is a code change and deserves the same gate.

Version everything that shapes the output

When quality drops, the first question is always what changed. You need to be able to answer it.

Prompts belong in version control, not in a database field somebody edited through an admin panel at four in the afternoon. Treat them as source: reviewed, diffed, and tagged with the release that shipped them.

prompts/
  support_triage/
    v1.txt
    v2.txt        # added explicit "unknown" category
    current -> v2.txt

Model versions need pinning too. Providers release new models regularly, and an alias that silently points somewhere new will change behaviour under you. Pin the exact model identifier, and treat an upgrade as a deliberate change: run the evaluation suite against the new model, compare, then move.

Log the full triple with every request: prompt version, model identifier, and retrieval configuration. When someone reports a bad answer three weeks later, that record is the only thing that lets you reproduce it.

Observability

Standard application monitoring tells you the request succeeded. It does not tell you the answer was wrong. You need a second layer.

Log per request, at minimum:

• Input and output, subject to your data policy

• Model, prompt version, and any retrieval parameters

• Token counts in and out

• Latency, broken down by stage where there is more than one

• Cost, computed at the time rather than reconstructed later

• A trace identifier linking the stages of a multi step request

For anything with more than one model call, tracing stops being optional. A request that retrieves, reranks, generates and validates has four places to be slow and four places to be wrong, and an aggregate latency number tells you nothing about which. Tools such as Langfuse, LangSmith or plain OpenTelemetry spans all work; the discipline of instrumenting each stage matters more than the choice.

Feedback as a signal

Instrument the interface so users can flag a bad answer with one click. Those flags are the most valuable data you will collect, because they identify failure cases you did not anticipate. Route them into a review queue, and feed the confirmed ones back into the evaluation dataset. That loop is what makes the system improve rather than drift.

Cost

AI features have a per request cost that scales with usage, which is unlike most of the software around them. A feature that is cheap in testing can be expensive at production volume, and the discovery usually happens on an invoice.

Attribute cost per request and per feature from day one. Aggregate spend is not actionable. Cost per feature tells you which one to optimise; cost per customer tells you whether the pricing works.

The levers, roughly in order of effectiveness:

Cache aggressively. Identical requests should not be paid for twice. Beyond exact match caching, most providers support prompt caching, where a large stable prefix such as a system prompt or a document is cached server side and charged at a fraction of the normal input rate on subsequent requests. For any workload with a big fixed preamble this is the single largest saving available, and it needs the stable content placed first in the prompt to work.

Match the model to the task. Routing, classification and extraction rarely need the most capable model available. Reserve the expensive one for the work that genuinely benefits, and validate the cheaper model against your evaluation suite rather than assuming. A tiered approach, where a small model handles the clear cases and escalates the ambiguous ones, often gets most of the quality at a fraction of the cost.

Batch what is not interactive. Overnight classification, bulk enrichment and backfills do not need a synchronous response. Most providers offer a batch endpoint at roughly half price for work that can wait.

Trim the prompt. Input tokens are usually the bulk of the bill in a retrieval system. Retrieving ten chunks where five would do is a permanent tax on every request. This is where a reranker pays for itself twice: better answers and a shorter prompt.

Latency

Model calls are slow by the standards of the rest of your stack, and users notice. Three things help.

Stream. Time to first token is what perceived speed depends on, not total duration. A response that begins in half a second and takes eight seconds to finish feels faster than one that appears complete after four.

Parallelise what is independent. Retrieval and any other lookups can run concurrently. Only genuinely sequential steps should be sequential.

Set timeouts and know what happens after them. A model call that hangs must not hang the request. Decide in advance whether the fallback is a cached answer, a degraded response, or an honest error, and make that path as well tested as the happy one.

Reliability

Provider APIs rate limit, occasionally return errors, and are sometimes slow. Handle it the way you would any other external dependency.

Retry with exponential backoff and jitter on transient failures, respecting whatever retry hint the response carries. Most official client libraries do this by default; know what yours does before adding your own layer on top.

Decide what happens when the model is unavailable. For a drafting feature, degrading to a blank editor is fine. For something in a critical path, you need either a fallback provider or a queue, and both should be exercised rather than theoretical.

Validate structured output before acting on it. If you expect JSON matching a schema, check it. Most providers now support constrained output that guarantees a schema, which is better than parsing and hoping. Where you cannot use it, validate and retry once with the error included in the request; if it fails twice, fail loudly.

Deployment

Prompt and model changes are behaviour changes and deserve gradual rollout. Ship them behind a flag, send a small share of traffic to the new version, compare the metrics, and expand.

This matters more than for ordinary code because the failure is subtle. A broken deployment errors and you roll back within minutes. A prompt change that makes answers slightly worse can run for weeks before someone raises it. Comparing quality metrics across variants at rollout is what catches it.

Keep rollback to a single step. If a prompt version is a file reference and a model is a pinned identifier, reverting is a configuration change rather than a redeploy.

Security and data handling

Two things that belong in the operational picture rather than being left to a separate review.

Know what leaves your infrastructure. Anything in a prompt goes to the provider. Establish what may and may not be sent, and enforce it in code rather than in a policy document. Check the provider's retention terms and whether zero retention is available and required for your case.

Treat model output as untrusted input. If it reaches a database, a shell, a browser or another system, it needs the same validation you would apply to anything a user typed. The model may have been influenced by content it retrieved, and that content may not have been written by someone friendly.

What a mature setup looks like

Prompts in git with review. Models pinned. An evaluation suite in CI that gates changes. Per request tracing with cost attribution. A feedback mechanism that feeds the evaluation set. Gradual rollout with quality comparison. A tested fallback for provider failure. Alerts on cost, latency and quality, not just errors.

None of that is exotic. It is the same operational maturity you would expect around any other production dependency, applied to one that happens to fail quietly. Teams that build it early move quickly afterwards, because they can change things without fear. Teams that skip it end up afraid to touch a prompt that works.

If you are moving an AI feature from prototype to production, or you have one in production that nobody can safely change, talk to Eight Mile. AI assistants, cloud infrastructure, CI/CD, system architecture and technical consultancy are the work we do.