How to Instrument and Monitor AI Agents in Production

The OpenTelemetry GenAI semantic conventions stabilized enough this year that most agent frameworks now emit compatible span attributes out of the box. That standardization matters. AI agent observability monitoring production workloads stopped being a bespoke exercise per vendor, but the work of actually running these systems, catching regressions before customers noticed, and keeping token bills defensible only got harder as agents took on more of the workflow.

We instrumented agents for three deployments: a textile ERP query bot on 20-year-old MS SQL, a WhatsApp order-taking system for an FMCG distributor in Hinglish, and a daily cylinder rotation report for an industrial gas fleet. All three ran on legacy backends. All three talked to end users on WhatsApp with a 15-second Twilio webhook window. All three had budget ceilings small enough that a runaway retry loop showed up on the P&L within a day.

Standard OpenTelemetry got us traces. Getting to useful signal took another six months of work on top.

What AI agent observability actually measures

Traditional web app observability tracks four numbers: request rate, error rate, latency, saturation. An agent workflow needs more, because a single user turn fans out into planning, retrieval, one or more tool calls, and synthesis. The ReAct agent behind the textile ERP emitted between 4 and 18 spans per WhatsApp message, depending on how many SQL queries the model needed to answer the question.

The MIT NANDA study put the failure rate at 95% for generative AI pilots, and follow-up analysis showed that 88% of enterprise agents that worked in controlled demos failed once deployed to real workflows. The reasons were rarely dramatic. The pilot worked on the four inputs the demo used. Production sent 4,000 inputs. Nobody was watching.

We settled on measuring three layers.

Per LLM call. Model version, prompt tokens, completion tokens, cache hit rate, latency at P50/P95/P99, finish reason. The OpenTelemetry GenAI conventions define these as gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.response.finish_reasons. Any modern SDK will emit them if you set OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental.

Per tool call. Which tool was invoked, arguments, latency, success or error, retries. This is where most operational damage happens in AI agent observability monitoring production workloads. A hallucinated SQL query does not break the LLM span; it breaks the tool span underneath.

Per session. Total cost, number of turns, whether the user got the answer they asked for, whether a human had to intervene. Cost per successful task is the number that eventually matters to the business. Cost per LLM call is not.

Those three layers form the working definition of AI agent observability for the deployments we run. Anything less and the traces are noise. Anything more and the storage bill outruns the inference bill.

Structuring spans around the ReAct loop

The default OpenTelemetry instrumentation from most SDKs wraps each LLM call in a span and calls it done. That is not enough for AI agent observability in production, because a single user turn fans out into a tree of decisions and you cannot debug a failure without seeing the tree.

We used the pattern Greptime documented for MCP tool tracing: a root span for the user turn, an agent.plan span for reasoning, one child span per tool call, and one final agent.synthesize span for the response. Every span carried the session ID, tenant ID, and cost so far as attributes.

// otel/instrumentation.ts
import { trace, SpanStatusCode } from '@opentelemetry/api';
 
const tracer = trace.getTracer('whatsapp-agent', '1.4.0');
 
export async function runAgentTurn(session: Session, message: string) {
  return tracer.startActiveSpan('agent.turn', async (root) => {
    root.setAttribute('session.id', session.id);
    root.setAttribute('tenant.id', session.tenantId);
    root.setAttribute('message.length', message.length);
    try {
      const plan = await tracer.startActiveSpan('agent.plan', (s) =>
        planNextStep(session, message).finally(() => s.end())
      );
      const result = await executePlan(plan);       // emits tool spans
      root.setAttribute('agent.turn.cost_usd', session.costUsd);
      root.setAttribute('agent.turn.tool_calls', session.toolCallCount);
      return result;
    } catch (err) {
      root.recordException(err as Error);
      root.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      root.end();
    }
  });
}

The value of this structure showed up the first time an agent misbehaved in the FMCG deployment. A salesperson placed an order for 5 cartons; the agent charged for 50.

The LLM span looked normal. The tool span for apply_pricing_rules looked normal. The plan span showed the model had extracted quantity 50 because a downstream regex had appended the price code 0 to the quantity string. We fixed it in an hour. Without the layered spans, we would have spent a day arguing with logs.

The AI agent observability vendors we tested

We picked platforms based on what mattered per deployment: cost tracking for the FMCG bot, schema-aware evaluation for the ERP query bot, and long-running trace debugging for the cylinder report agent. No single vendor won across all three for AI agent observability monitoring production traffic, and pricing at real production volume was different enough from entry-tier pricing that we ended up self-hosting two of them.

Entry-tier pricing on every vendor looked reasonable. At one million traces per month, the same vendor was ten times the price of self-hosted Langfuse. Do the math at your actual expected volume before you commit.

We ran Langfuse self-hosted on Railway alongside our backend for the FMCG deployment because the client cared about data residency and we wanted per-tenant cost dashboards. Langfuse was acquired by Clickhouse in January 2026, the open-source code is still MIT-licensed, and the community stayed active. For the textile ERP, we sent traces to Arize Phoenix specifically because we wanted retrieval evaluation: the ReAct agent generated SQL, and Phoenix let us score how often the generated query actually returned the rows the user was asking about. A medium comparison from May summed up our own reading of the field: Langfuse for operations, Phoenix for RAG evaluation, LangSmith if you're all-in on LangChain, Datadog if you already run APM on it.

A note on Helicone. It was our first choice for the FMCG bot because the gateway pattern is genuinely elegant: change one URL, get logging with sub-5ms overhead. Following Mintlify's acquisition in March 2026, the platform moved to maintenance mode. New features slowed. We migrated to Langfuse. This is why we now default to self-hostable options for anything a client cares about long term.

Sampling strategies for AI agent observability at scale

At the volumes an FMCG bot handles during monsoon reorder season, capturing every prompt and completion at full fidelity is not affordable. Structural span storage is cheap. Content storage is not, and content is the thing regulators care about.

We settled on a hybrid pattern documented in the OpenTelemetry Collector sampling docs: 100% of structural spans (attributes only, no prompt or completion text), 100% of errors and slow requests with full content, and 5-10% of normal traces with full content. Head-based sampling at the SDK level was too coarse; a naive 10% rate meant we lost most of the interesting long-tail failures. Tail-based sampling at the Collector let us keep the useful ones.

# otel-collector-config.yaml
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 5000 }
      - name: keep-guardrail-hits
        type: string_attribute
        string_attribute: { key: guardrail.blocked, values: ["true"] }
      - name: sample-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 8 }

The keep-guardrail-hits policy caught the case we cared about most: any turn where a safety filter blocked the response. Those are rare in absolute terms and always worth investigating. Losing them to random sampling would defeat the point of running the filter.

For the cylinder rotation report agent, which ran once a day per tenant, we sampled at 100%. The volume was low enough that content storage did not matter and the traces were the primary debugging surface. Different systems, different sampling budgets. There is no single answer.

Cost observability and per-tenant budgets for agent monitoring in production

Multi-tenant agent systems fail unusually when one tenant sends 20,000 messages in a day. This happened to us on the FMCG deployment in April. A distributor's WhatsApp Business account was compromised. An automated script sent thousands of malformed messages. Our agent classified each one and tried to place an order. The token bill for that tenant would have been about $340 for one day, more than the entire monthly retainer.

We caught it because we had a budget guardrail that halted inference for any tenant crossing $10 in a 24-hour rolling window. The guardrail was crude. It worked.

// services/budgets.ts
const DAILY_LIMIT_USD = 10;
 
export async function assertBudget(tenantId: string, projectedUsd: number) {
  const spent = await redis.get(`cost:${tenantId}:${todayIST()}`);
  const total = Number(spent || 0) + projectedUsd;
  if (total > DAILY_LIMIT_USD) {
    metrics.counter('agent.budget.blocked').add(1, { tenant: tenantId });
    throw new BudgetExceededError(tenantId, total, DAILY_LIMIT_USD);
  }
  await redis.incrbyfloat(`cost:${tenantId}:${todayIST()}`, projectedUsd);
}

Redis TTL rolled the counter over at midnight IST. Anthropic's prompt caching reduced our baseline cost per turn by about 78% on the textile ERP because the 200-line schema prompt got re-sent every turn and was mostly cached. Without caching, the same workload would have hit the $10 daily limit under normal use, not just during an incident. This is worth stating clearly: cost observability is only useful if you have levers to pull when the number goes wrong. Caching is one lever. Budgets are another. Model routing between Haiku and Sonnet is a third.

The pattern we built for retry safety, described in more detail in Idempotent AI Agents, also mattered here. A retry storm doubles or triples cost per user without any additional user-visible progress. Retry counts became a first-class metric.

Alerts that caught real degradation

The alerts we started with fired on latency and error rate, and they never caught anything useful. Latency and errors are already lagging indicators for LLM systems. By the time P95 latency crosses a threshold, a segment of customers has already been receiving slow responses for hours. Good AI agent observability catches problems before customers report them; bad AI agent observability confirms what the customer already told you.

The alerts that actually fired on real problems, in order of usefulness:

1. Tool error rate by tool name, above 2% over 15 minutes. Caught schema drift twice. The ERP added a column, the SQL translator hadn't been updated, and the check_stock tool started returning empty results. Alert fired within 20 minutes. Fix took 40.

2. Prompt cache hit rate below 70%. On a stable system prompt, cache hit rate should be at least 90%. A drop meant somebody had edited the prompt, or a whitespace change had invalidated the cache prefix. Cheap and diagnostic.

3. Cost per successful task, 24-hour rolling. Not cost per request. Cost per resolved user session, where success was defined by whether the user's follow-up message indicated they got what they needed. This metric drifted upward when the model started using more tool calls per turn, which was itself a sign the ReAct loop was oscillating.

4. Guardrail hit rate above baseline. A sudden increase in safety filter blocks meant either an adversarial user or, more commonly, a data pipeline change that started injecting content the guardrails didn't like. Both worth investigating.

5. Retry count P95 above 2. Any tool with retry P95 above 2 was silently retrying itself into a budget hole. See the idempotency post for how we made retries safe first, then observable.

Multi-agent handoffs and the correlation ID that fixed everything

The daily cylinder rotation report system, described in detail in Real-Time Cylinder Rotation Analytics, used four agents in sequence: a data-fetching agent, an anomaly detector, a narrative generator, and a WhatsApp formatter. Each ran as a separate Google ADK LlmAgent in its own process. When something went wrong, the trace stopped at the process boundary.

The fix was one line: propagate the OpenTelemetry trace context through every inter-agent message as a header, and continue the trace in the receiving process. @opentelemetry/context-async-hooks handled the propagation. Every agent now emitted spans under the same trace ID as the original scheduled trigger. A failed report at 7:03 AM IST could be walked backwards through the anomaly detector, the data fetch, and the underlying MongoDB query in one view.

This sounds obvious. We didn't do it for the first two months and paid for it in every incident review.

Semantic drift and what we still cannot measure

The observability stack we built handles almost everything except one category: silent semantic drift. The model is still producing outputs, the tool calls still succeed, the cost is stable, but the answers have quietly gotten worse. This happens when the underlying data distribution shifts (new SKU categories, new payment methods), or when a model provider updates a snapshot without changing the version string.

We have not solved this. Arize Phoenix has drift detection features built on embedding distances between production traces and a labeled reference set, and we ran it in shadow mode for two months on the ERP query bot. It caught one real regression (a schema change that broke a class of stock queries) and produced three false positives (all correlated with legitimate business seasonality). The signal-to-noise ratio was not good enough to alert on.

The current answer is a weekly manual review of sampled traces by the engineer who owns the deployment. That does not scale. It is the biggest open problem in AI agent observability monitoring production workloads at our size, and the MLflow 2026 monitoring guide suggests the industry does not have a good answer yet either. Semantic drift will probably require a second tier of monitoring on top of the OpenTelemetry-based AI agent observability layer we already run.

What we would do differently

Instrument on day one, not on day thirty. Every deployment where we added AI agent observability after the fact took at least two weeks longer to stabilize than the deployments where it was there from the start. There is no cheap retrofit for monitoring production traffic once it has already broken.

Standardize on OpenTelemetry GenAI conventions from the beginning. We started with vendor-native SDKs on two projects. Migrating later cost time. The OpenTelemetry GenAI conventions are stable enough now that vendor lock is a choice, not a constraint.

Budget guardrails before dashboards. A dashboard tells you a tenant is spending too much. A budget guardrail stops the spending. Build the guardrail first. Dashboards are for tuning the guardrail, not for reacting to the incident.

Cost per successful task, not cost per token. Cost per token is a vanity metric. Cost per resolved user session is a business metric. The gap between them is the number to optimize. The harness patterns in Building Harnesses for Operational AI Agents exist mostly to shrink that gap.

Do not pay enterprise-tier prices for features you can self-host. For the FMCG and textile deployments, Langfuse self-hosted on the same Railway project as the backend gave us tracing, cost dashboards, and per-tenant analytics for the cost of the Postgres instance. The equivalent hosted tier would have doubled our monthly infrastructure bill.

References

1. OpenTelemetry, "Inside the LLM Call: GenAI Observability with OpenTelemetry," 2026. opentelemetry.io/blog/2026/genai-observability
2. OpenTelemetry, "Semantic Conventions for Generative AI Systems." opentelemetry.io/docs/specs/semconv/gen-ai
3. Greptime, "How OpenTelemetry Traces LLM Calls, Agent Reasoning, and MCP Tools," May 2026. greptime.com/blogs
4. Datadog, "LLM Observability Cost Estimation." docs.datadoghq.com/llm_observability
5. MIT NANDA Initiative, as reported by Fortune, "95% of generative AI pilots at companies are failing," Aug 2025. fortune.com
6. Fiddler AI, "AI Agent Failure Rate: Why 70-95% Fail in Production," 2026. fiddler.ai/blog
7. Langfuse, "LLM Observability Platform Documentation." langfuse.com
8. Arize, "Phoenix Open-Source LLM Observability." phoenix.arize.com
9. Anthropic, "Prompt Caching." anthropic.com/news/prompt-caching
10. MLflow, "Monitoring Agentic AI in Production: 2026 Guide." mlflow.org/articles
11. Kanerika, "LLMOps Observability: LangSmith vs Arize vs Langfuse vs W&B," May 2026. medium.com
12. Gartner, "Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027," June 2025. gartner.com

The observability stack described here was deployed across three production AI agent systems in India. Traces flowed through the OpenTelemetry Collector into self-hosted Langfuse and Arize Phoenix instances, running on Railway alongside the agent backends. Instrumentation was TypeScript for the WhatsApp and ERP bots, Python for the Google ADK report agents. Inference ran on Gemini 3.1 Flash Lite with prompt caching where the provider supported it.