LLM monitoring tells you that something changed; LLM observability lets you work out why. For an application built on a language model, you need both, because the usual signals (uptime, latency, error rate) can all look healthy while the answers get worse.
This article is about what to instrument and what to alert on: the metrics, traces and evaluation loops that make an LLM application operable. It does not cover where the data lives or how to keep it inside a region; for that, read self-hosting the LLM stack for data residency. For building and running such systems, see our LLMOps services.
Why classic monitoring is not enough
A request to a web service either succeeds or fails. A request to an LLM application can return HTTP 200 with a fluent, wrong answer. The failure is semantic, and no status code encodes it. Add that a single user request may fan out into a retrieval query, several model calls, tool invocations and retries, and a latency graph tells you almost nothing about what happened.
So the unit of observation changes. It is not the request; it is the trace of everything that happened to produce one response.
Traces are the foundation
A trace for an LLM request should be a tree of spans: the incoming request, retrieval, each model call, each tool call, post-processing. On each model-call span, record:
- the model name and version, and the provider or endpoint,
- the prompt template identifier and version (not just the rendered text),
- key parameters such as temperature and maximum output tokens,
- input and output token counts,
- time to first token and total duration for streaming responses,
- finish reason (completed, truncated, filtered, tool call),
- errors and retries.
You do not need to invent the field names. The OpenTelemetry project maintains semantic conventions for generative AI covering spans, metrics and events for model and agent calls. The page notes that these conventions now live in a dedicated repository, so check there for current attribute names, and expect them to keep evolving. Instrumenting to a shared convention means you can change backends without re-instrumenting your code.
Capture the prompt, but decide on purpose
Storing full prompts and completions is the most useful debugging data you have and also the riskiest, since they often contain personal or confidential information. Decide per environment whether to store content, redact before storing, or store only identifiers and hashes. The privacy side is covered in personal data in LLM prompts and traces.
The metrics that matter
| Category | Metrics | Why it matters |
|---|---|---|
| Reliability | Error rate and timeouts by model and endpoint; retry rate; rate-limit responses | Provider incidents and quota limits show up here first |
| Latency | End-to-end, time to first token, per-step latency percentiles (p50, p95, p99) | Users feel the tail, and one slow tool call can dominate |
| Cost | Tokens in and out per request, cost per request, cost by feature, tenant and model | Spend scales with prompt length and traffic, and a prompt change can double it overnight |
| Output quality | Scores from automated evaluators, user feedback rates, refusal and fallback rates | The signal that catches fluent wrong answers |
| Safety | Guardrail trigger rates, blocked inputs and outputs, policy violations | Shows abuse attempts and regressions in filtering |
| Retrieval (RAG) | Number of chunks retrieved, retrieval score distribution, empty-result rate, share of answers citing sources | Most RAG failures are retrieval failures |
| Agents | Steps per task, tool error rate, loops and step-limit hits, task completion | Agents fail by wandering, not by crashing |
Slice every one of these by prompt version, model version, tenant or customer segment, and feature. An average over everything hides the one workflow that broke.
Measuring quality without ground truth
Quality is the hard part. Labels are scarce, and "good" is often subjective. There are four practical approaches, and mature systems combine them.
- Programmatic checks. Cheap and deterministic. Does the output parse as the required JSON? Does it stay within length limits? Does it contain a forbidden phrase? Does every cited source ID exist in the retrieved set? Run these on every response.
- Model-graded evaluation. Use a second model as a judge against a rubric: relevance, faithfulness to the retrieved context, tone. It scales, but judges have biases and can be wrong, so calibrate against human labels on a sample and re-check when you change the judge model. Treat the scores as a trend indicator, not as truth.
- User signals. Thumbs up or down, edits to a draft, copy actions, escalations to a human, repeated rephrasing of the same question. These are noisy but reflect reality and cost nothing to collect.
- Human review of samples. Sample production traces, weighting toward low evaluator scores and negative feedback, and have people label them. This is how you build and refresh your evaluation set.
Open-source libraries can help. Evidently is an Apache 2.0 framework for evaluating, testing and monitoring data and AI systems, including LLM outputs. Pick a tool for the checks and storage you need rather than for its dashboard.
Close the loop: production traces become test cases
The most valuable habit in LLM operations is turning failures into regression tests. When a bad trace is found, add it, with its expected behavior, to a versioned evaluation set. Run that set in CI whenever you change a prompt, a model, a retrieval setting or a tool definition. Then production teaches the test suite, and the test suite protects production.
This also makes model upgrades manageable. Providers retire and replace model versions; with an evaluation set you can run the candidate against the same cases and see what moved before you switch.
Monitoring retrieval separately
If your application uses retrieval-augmented generation, evaluate retrieval on its own, apart from generation. For a sample of queries, check whether the passages needed to answer were in the retrieved set at all. If they were not, no prompt change will fix the answer. Watch for index freshness (documents updated at the source but not re-embedded), unusually low similarity scores, and queries that return nothing. For image-heavy corpora, see multimodal RAG.
Alerting that does not cry wolf
LLM metrics are noisy, so alert design matters more than in conventional services.
- Page on hard failures: error rate, timeouts, provider outages, cost per hour above a ceiling, a spike in guardrail blocks.
- Ticket on slow drift: evaluator scores trending down over days, rising fallback rates, growing retrieval-miss rate.
- Alert on change events: any shift correlated with a new prompt version, model version or deployment. Tag every trace with those versions so you can see this immediately.
- Set cost guards early. A runaway agent loop or a prompt that suddenly includes a whole document can burn budget quickly. Hard per-request token limits and per-tenant budgets are cheap insurance.
A starting setup
You can get most of the value with a small footprint:
- Instrument every model, retrieval and tool call with spans using the OpenTelemetry conventions, with version tags on each.
- Export traces to a backend you control or trust, and decide the content-storage policy first.
- Dashboard latency, error rate and token cost by feature and version.
- Add programmatic checks to every response and a small judge-based evaluator on a sample.
- Collect at least one user feedback signal.
- Build the evaluation set from real failures, and run it in CI.
This is the same discipline described in what is MLOps, adapted to systems where the artifacts are prompts, indexes and evaluation sets. We build and operate these stacks for teams, from tracing to evaluation pipelines, within our LLMOps and NLP and LLM application work. To scope yours, request a quote.