In this article
Key takeaways
- Agent observability means being able to replay any run: every model call, tool call, retrieval, guardrail and hand-off, with its outcome and cost. Uptime and latency dashboards cannot tell you whether the agent did the right thing.
- Trace the run, not just the model. One root span per task, with the agent version, prompt version and outcome, and a child span for each step. The OpenTelemetry GenAI conventions give you standard names for most of it.
- Page people for harm, not for noise: failed-run spikes, tool outages, runaway loops and blocked write actions. Slow drift in latency, cost, escalations and evaluation scores belongs in tickets and reviews.
- The weekly trace review is the habit that makes agents better. Sample real runs, label them, turn failures into evaluation cases, and gate every model or prompt change on those cases.
- Decide what content you log before launch. Prompts and tool results often hold personal or regulated data, so redact by default, sample, restrict access and set retention.
What is agent observability?
Agent observability is the ability to see what an AI agent did on any run and why: the request, each model and tool call, what came back, which guardrails fired, whether a person stepped in, how the task ended and what it cost. It joins traces, metrics, logs and evaluation scores so you can debug one bad answer and also spot patterns across thousands of runs.
Traditional monitoring asks whether a service is up and fast. That question still matters, but an agent can be up, fast and wrong. It can pick the wrong tool, pass a made-up ID into a real API, loop through the same search four times, or give a confident answer built on a stale document. None of that shows up as a 500 error.
LLM observability moved the focus to the model call: prompt, completion, tokens, latency. Agents need one more step, because the unit that matters is the task. A single run might involve six model calls, three tools, a retrieval step and a hand-off to a person. You need to see the whole chain, in order, with its result.
| Application monitoring (APM) | LLM observability | Agent observability | |
|---|---|---|---|
| Unit of work | Request or transaction | Model call | Task: a full multi-step run |
| Main question | Is it up and fast? | What did the model see and say? | Did the agent do the right thing, the right way, at an acceptable cost? |
| Key signals | Errors, latency, throughput, saturation | Prompts, completions, tokens, latency, model version | Outcome, steps, tool calls and errors, guardrail actions, hand-offs, cost per task, evaluation scores |
| Typical blind spot | Wrong but successful responses | What happened between model calls | Only as good as your outcome definitions |
You need all three. Agent observability does not replace your APM; the tool calls an agent makes are ordinary API calls, and their latency and errors still belong in the same place as the rest of your services. What it adds is the agent-shaped view on top. If you are still deciding what an agent is in your context, our explainer on what agentic AI is covers the basics.
Why AI agents in production fail quietly
Most agent failures return a 200 status and a polite sentence. That is what makes them expensive. These six patterns cover most of what we see after launch, and each one leaves a trace you can look for.
Wrong tool, or wrong arguments
The agent calls a search when it should call a lookup, or passes a plausible but invented ID to a real API.
Shows up asTool errors, empty results, or a tool sequence that differs from the usual pathLoops and runaway runs
The agent repeats a step because the result never satisfies it. Each loop adds tokens to the context.
Shows up asSteps per run in the long tail, cost per run spikes, repeated identical tool callsConfident, unsupported answers
The answer reads well but is not backed by what the tools or documents returned. The classic hallucination, now with actions attached.
Shows up asLow groundedness scores in online evals, reviewer overrides, user complaintsSilent drift
A provider updates a model, a payer changes a rule, a document set goes stale. Nothing breaks; results just get a little worse.
Shows up asA new response model version, a slow fall in task success, new failure clustersInstructions hidden in data
A web page, email or document read by a tool contains text that tries to redirect the agent: prompt injection.
Shows up asGuardrail blocks, unexpected tool calls after a retrieval, actions outside the normal scopeCost creep
Longer contexts, extra retries and a more capable model on every step. Each change is small and reasonable on its own.
Shows up asCost per successful task rising week over week while volume stays flatLook at the third column of each card. Every failure mode is visible, but only if the trace holds the right fields and someone is looking at the right aggregate. That is the design problem the rest of this guide works through.
Anatomy of an agent trace
A trace is a tree of spans. In agent tracing, the root span is the run, and each child is a step: a model call, a tool call, a retrieval, a guardrail check. Here is one run of a fictitious claim-status agent answering "where is claim 44817?". Select a span to see what it records.
- gen_ai.operation.name
- "invoke_agent"
- gen_ai.agent.name
- "claim_status_agent"
- gen_ai.conversation.id
- "conv_19c2"
- app.agent.version
- "2026.11.3"
- app.prompt.version
- "cs-prompt-v14"
- app.outcome
- "answered"
The root span. It records who asked, which version of the agent answered and how the run ended. The outcome attribute is the one most teams forget, and every dashboard further down depends on it.
- gen_ai.operation.name
- "chat"
- gen_ai.request.model
- "model-a"
- gen_ai.response.model
- "model-a-2026-09"
- gen_ai.usage.input_tokens
- 3,412
- gen_ai.usage.output_tokens
- 186
- gen_ai.response.finish_reasons
- ["tool_calls"]
The model decides it needs to look the claim up. Note that the response model carries a dated version the request did not ask for. When that value changes, behaviour can change with it.
- gen_ai.operation.name
- "execute_tool"
- gen_ai.tool.name
- "lookup_claim"
- gen_ai.tool.call.id
- "call_01"
- error.type
- "timeout"
- app.tool.target
- "payer_api"
The payer API timed out at three seconds. In the chat window this looked like a slow answer, not a failure. Only the trace shows it, and only a tool error rate metric shows that it happened to 6% of runs this morning.
- gen_ai.operation.name
- "execute_tool"
- gen_ai.tool.name
- "lookup_claim"
- gen_ai.tool.call.id
- "call_02"
- app.retry.attempt
- 2
- app.tool.side_effect
- "read"
The retry worked. Retries are normal. A retry rate that doubles overnight is not, and it is usually the first sign of a dependency problem.
- gen_ai.operation.name
- "chat"
- gen_ai.request.model
- "model-a"
- gen_ai.usage.input_tokens
- 4,980
- gen_ai.usage.output_tokens
- 241
- gen_ai.response.finish_reasons
- ["stop"]
The answer draft. Input tokens grew by almost half because the tool result went into the context. Across a long run, context growth is where agent cost quietly comes from.
- app.guardrail.name
- "phi_redaction"
- app.guardrail.action
- "redacted"
- app.guardrail.hits
- 1
A custom span; the GenAI conventions do not define guardrails. Give every check its own span so you can count blocks, redactions and near misses by agent version.
app.* attribute (example)Illustrative trace. The agent, model names, IDs and values are fictitious.
Use the OpenTelemetry GenAI conventions, with your eyes open
OpenTelemetry now has semantic conventions for generative AI, maintained in their own repository. They define operations such as create_agent, invoke_agent, execute_tool and chat, and attributes such as gen_ai.agent.name, gen_ai.tool.name, gen_ai.usage.input_tokens and gen_ai.response.finish_reasons. Using them means any OpenTelemetry-compatible backend can read your agent traces, and you can switch backends later without re-instrumenting.
Two caveats. The conventions are still marked Development, so attribute names can change between versions; pin the version you build against. And they do not cover everything an agent needs, such as guardrail results or business outcomes, so keep your own fields in a separate namespace like app.*.
// span name: "execute_tool lookup_claim"
{
"gen_ai.operation.name": "execute_tool",
"gen_ai.tool.name": "lookup_claim",
"gen_ai.tool.call.id": "call_01",
"error.type": "timeout",
// your own fields: keep them out of the gen_ai.* namespace
"app.agent.version": "2026.11.3",
"app.tool.target": "payer_api",
"app.tool.side_effect": "read",
"app.retry.attempt": 1
}Content is opt-in for a reasonIn the GenAI conventions, capturing message content (prompts, outputs, tool arguments and results) is opt-in and off by default. That is the right default for most teams, especially in healthcare and finance. Metadata on every span, redacted content on a sample, is a sensible place to start.
What to trace in every agent run
Think in six layers. The first three are mostly mechanical and come with good instrumentation libraries. The last three are where teams cut corners, and they are the ones that let you answer "was this run any good?".
- 1
Run and identity
Which agent, which version, for whom, through which channel.
- trace / run ID
- conversation ID
- agent name + version
- prompt version
- tenant or user (pseudonymous)
- channel
- 2
Model calls
What was asked of the model and what it cost.
- request vs response model
- input / output tokens
- latency, time to first token
- finish reason
- parameters
- cached tokens
- 3
Tool calls
What the agent did to other systems, and whether it worked.
- tool name + call ID
- status, error type
- retries
- latency
- read or write
- arguments (redacted)
- 4
Retrieval and context
What the agent knew when it decided.
- document IDs, not text
- retrieval scores
- context size
- truncation
- memory reads / writes
- source freshness
- 5
Decisions and guardrails
Where the agent was checked, stopped or handed off.
- steps taken
- guardrail checks + action
- policy denials
- confidence score
- hand-off to human, with reason
- agent-to-agent hand-offs
- 6
Outcome and feedback
How the task ended and what happened next.
- outcome: resolved, escalated, abandoned, failed
- reviewer edits or overrides
- user rating
- business result
- cost of the run
The outcome field deserves a meeting of its own. "Resolved" means something different for a scheduling agent, a coding assistant and a claims agent. Agree the definitions with the business owner before launch, and write them down. Every success rate, escalation rate and cost-per-task number on your dashboard depends on them.
What not to logRaw patient or customer data in span attributes, API keys passed through tool arguments, and full documents copied into traces. Store references and IDs, redact content before it leaves the process, and treat your trace store like any other system that holds sensitive data: access control, encryption, retention. For healthcare agents, our guide to HIPAA-compliant AI covers the logging and audit requirements in more depth.
What to alert on: fewer pages, better tickets
The fastest way to make a team ignore AI agent monitoring is to page them for every odd answer. Split signals by urgency. Page for harm that is happening now. Ticket the slower problems. Leave the rest for the weekly review.
- Failed-run spikeFailed or abandoned runs above 3x the 7-day baseline for 10 minutes
- Tool or dependency outageError rate on a critical tool above 10% for 5 minutes
- Runaway runAny run past a hard step cap or cost cap; kill it, then page
- Blocked write actionThe agent tried an action its policy forbids, such as a refund above its limit
- Latency regressionp95 run latency 50% above the 7-day baseline
- Escalation creepHand-offs to humans up a quarter week over week
- Cost per successful taskUp 30% week over week at flat volume
- Online eval dropSampled groundedness or correctness score below its floor
- Model version changeThe response model reported by the provider changed
- Retry rate doublingOften the early sign of a dependency problem
- Slow driftTask success sliding a point or two a week
- New failure clustersGroups of similar failed or overridden runs
- Most expensive runsThe top 20 by cost, read end to end
- Guardrail near missesRedactions and blocks, by agent version
- User feedback themesRatings and comments, joined to traces
Thresholds are starting points for illustration. Take a baseline first and tune them to your own volume and risk.
Two rules make alerts useful. First, alert on symptoms users feel, such as failed tasks, slowness and wrong actions, and use traces to find causes. Second, every alert should carry a link to example traces and a short runbook: who owns the agent, how to roll back a prompt or model, and how to switch the agent to a safe mode such as "draft only, human sends".
We review your current logging and alerting, list the gaps against the six layers above, and propose the smallest set of alerts worth having.
What to review, and how often
Alerts catch incidents. Reviews catch everything else. Agents change behaviour without anyone deploying code, so the review routine is part of running them, not an optional extra.
- Daily15 min · agent owner
Triage yesterday's failures
Failed, abandoned and escalated runs from the last 24 hours, open tickets from alerts, and any blocked write actions. The goal is to spot anything new, not to fix it on the spot.
- Weekly60 min · engineer + domain expert
Read real traces
Sample 30 to 50 runs across outcomes: successes, escalations, failures and low eval scores. Label each one, update the failure taxonomy, add the interesting ones to the evaluation set, and pick one fix for the week. This is the meeting that improves the agent.
- Monthlygovernance · business owner
Accuracy, drift, cost and access
Evaluation scores against human labels, drift in rules or source data, cost trends, who can see raw trace content, and whether retention still matches policy. Decide whether the agent's autonomy should grow, stay or shrink.
- On changerelease gate · CI
Evaluate before you ship
Any change to a model, prompt, tool or retrieval source runs against the golden set and a replay of recent traces before rollout, then goes out as a canary with its metrics compared side by side.
Put a domain expert in the weekly reviewAn engineer can see that a tool call failed. Only a billing specialist, nurse or underwriter can tell you that a "successful" answer was wrong in a way that matters. Their labels are what make your evaluation set worth trusting.
Agent evaluation in production: closing the loop
Observability tells you what happened. Evaluation tells you whether it was good. In production, agent evaluation runs on three tracks, and the traces feed all three.
Deterministic checks
Rules you can compute: did the agent call the required tool, is the output valid JSON, does the cited claim ID exist, did it stay under the refund limit. Cheap enough to run on every trace.
Model-graded checks
An LLM judge scores groundedness, correctness or tone on a sample of live traces. Useful at scale, but only after you have checked it against human labels. Our guide to LLM-as-a-judge evaluation covers how.
Human labels
Reviewer verdicts from the weekly session and overrides from human-in-the-loop steps. Slow and expensive, and the ground truth everything else is checked against.
The loop that matters: a failure found in a trace becomes a labelled example, the example joins the golden set, and the golden set gates the next change. After a few months the evaluation set reflects your real traffic instead of the examples someone wrote before launch. That is the difference between LLM evaluation as a one-off benchmark and evaluation as an operating habit.
The agent metrics that matter on one screen
Eight numbers cover most of what a business owner and an engineer need to agree on. Show each one by agent version, so a regression is visible the day a change ships.
Illustrative dashboard. Values are fictitious.
Note the denominator in cost per successful task. Cost per run can fall while cost per success rises, if the agent gets cheaper by giving up sooner. Pair every efficiency metric with a quality metric, and read them together.
Choosing AI agent observability tools
The market for LLM observability tools moves quickly, and features overlap. It is more useful to choose a shape first and a product second. Instrument with OpenTelemetry wherever you can, so the shape can change later without rewriting your code.
| Approach | Examples | Good fit when | Watch for |
|---|---|---|---|
| Open standard instrumentation | OpenTelemetry with the GenAI semantic conventions | Always, as the base layer | Conventions still in Development; pin versions |
| Open-source LLM tracing and eval platforms | Langfuse, Arize Phoenix | You want self-hosting, data residency or no per-seat cost | You operate, scale and secure it |
| Framework-linked platforms | LangSmith | Your agents are built on LangChain or LangGraph | Tighter coupling to one framework's way of working |
| APM vendors with LLM features | Datadog LLM Observability, and similar | You want agents next to the rest of your services and on-call | Pricing at agent span volumes; depth of evaluation features varies |
| Your own data platform | Trace tables in your warehouse or lakehouse, plus BI | Long retention for audit, joins to business outcomes | You build the trace viewer and review workflow |
Not a ranking or an endorsement. Product capabilities change often; check current vendor documentation.
Questions that usually decide it: where may trace content legally live, how many spans a day you will produce at full volume, who runs the weekly review and in which tool, and whether you need to join traces to business data such as claims, tickets or invoices. Many teams end up with two destinations: a tracing platform for engineers, and summarised trace tables in the data warehouse for governance and reporting.
Agent observability in the first 30 days
You do not need a platform project to start. This sequence works for a single agent and scales to a portfolio.
Define outcomes per agent
Write down what counts as resolved, escalated, abandoned and failed, with the business owner. Add the business result you can join to later.
Instrument the run
A root span per run with agent and prompt versions, child spans for model calls, tools, retrieval and guardrails. Use the GenAI conventions where they exist and
app.*fields where they do not.Set the content policy
What gets redacted, what share of runs keeps content, who can read it and for how long. Agree it with security and compliance before the first trace leaves the process.
Run in shadow mode and take a baseline
Let the agent work alongside people for two weeks without acting on its own. Record success, escalation, latency and cost so later thresholds are based on your data.
Switch on a small alert set
Three or four paging alerts and five or six ticket alerts, each with a runbook and a safe mode. Add more only when a review shows something they would have caught.
Start the weekly review and the release gate
Book the hour, pick the sample, label the traces, and make the golden set a required check for any model, prompt or tool change.
What this looked like in a Bonami deployment
The clearest example in our published work is a prior authorization agent for a regional health system handling about 4,500 authorization requests a month. The case study does not name the observability tooling, so we will not either. What it does describe is the operating routine, and it maps closely onto this guide.
A five-phase rollout began in shadow mode, with specialists reviewing every AI output before anything was submitted.
High-confidence cases were submitted after a brief review. Lower-confidence cases went to human review.
Cycle time, approvals and returns were tracked every week during the rollout.
Accuracy, denials and payer rule drift were reviewed monthly, and ownership moved to operations once the pilot held.
median authorization cycle, down from 11 days
Results as published in Bonami's prior authorization case study. Client anonymised.
The same pattern appears in our 3PL invoice reconciliation agent, where configurable value and confidence thresholds keep unusual charges in front of people and every match, exception and dispute is logged with its evidence. In both cases, what made the business comfortable giving the agent more autonomy was not the model. It was being able to see, and show, why the agent did what it did.
How Bonami helps teams run agents in production
Bonami builds and operates AI agents through Bonami X AI, our AI agent development practice. We design the tracing, alerting and review loop alongside the agent, and we can add them to agents you already run.
Observability review
For an agent already live: a gap list against the six trace layers, an alert set with runbooks, and a content and retention policy.
Instrumentation and dashboards
OpenTelemetry tracing to the backend you choose, outcome definitions, and the dashboard above built from your data. Platform work sits with our DevOps team.
Evaluation and review loop
Golden sets from real traces, a calibrated LLM judge, release gates in CI, and a weekly review your domain experts can run without us.
Frequently asked questions about agent observability
What is agent observability?
Agent observability is the ability to see, for any run of an AI agent, what it was asked, which model calls and tool calls it made, what each one returned, which guardrails fired, how the run ended and what it cost. It combines traces, metrics, logs and evaluation results so a team can explain a single bad answer and also spot patterns across thousands of runs.
How is AI agent observability different from LLM observability?
LLM observability usually looks at single model calls: the prompt, the response, latency and tokens. Agent observability has to follow a whole run, where the agent plans, calls several tools, retries, reads documents, hands off to a person or another agent, and takes actions in other systems. The unit you care about is the task, not the model call, so outcome, steps, tool errors and cost per task matter more than any one response.
What should you trace in an AI agent?
At minimum: one root span per run with the agent name, agent version, prompt version and final outcome; a child span for every model call with the model, token counts, latency and finish reason; a span for every tool call with the tool name, status, error type and retries; spans for retrieval and guardrail checks; and any hand-off to a person. Add the cost of the run and a link to the business result, such as a ticket closed or a claim paid.
What alerts should you set for AI agents in production?
Keep paging alerts few and tied to harm: a spike in failed runs, a tool or dependency outage, runaway loops or cost per run above a hard cap, and any blocked attempt at a disallowed write action. Slower problems such as rising latency, more escalations to humans, higher cost per successful task or a drop in online evaluation scores should open tickets instead of waking someone up.
Is OpenTelemetry ready for tracing AI agents?
It is usable today. The OpenTelemetry GenAI semantic conventions define operations such as invoke_agent, execute_tool and chat, and attributes such as gen_ai.agent.name, gen_ai.tool.name and gen_ai.usage.input_tokens. The conventions are still marked as Development, so names can change between versions. Pin the version you instrument against and keep your own attributes in a separate namespace.
How do you monitor AI agents for hallucinations and drift?
Run evaluations on a sample of live traces, using rules where you can and an LLM judge where you cannot, and check the judge against human labels regularly. Track the scores over time and by agent version. Drift shows up as a slow fall in task success or evaluation scores, a change in the response model reported by the provider, or new clusters of failures in the weekly review. Re-run a fixed golden set whenever a model, prompt or tool changes.
Should you log prompts and responses from AI agents?
Only with a policy. Prompts, tool arguments and outputs often contain personal or regulated data. The OpenTelemetry GenAI conventions treat message content as opt-in for that reason. A common approach is to keep metadata on every span, store redacted content for a sample of runs, restrict who can open raw content, and set a retention period that matches your compliance obligations.
Which AI agent observability tools should we use?
Start from OpenTelemetry instrumentation so you are not locked in, then choose where to send the data. The options are open-source LLM tracing and evaluation platforms such as Langfuse or Arize Phoenix, framework-linked platforms such as LangSmith, the LLM features in APM vendors such as Datadog, or tables in your own data warehouse. The right choice depends on data residency, volume, the frameworks you use and who will run the review.
How often should a team review AI agent traces?
A short daily look at failed and escalated runs, a weekly review of a sampled set of traces with an engineer and a domain expert, a monthly governance review of accuracy, drift, cost and access, and an evaluation gate before every change to a model, prompt, tool or retrieval source. The weekly review is the one most teams skip, and it is where most improvements come from.
The takeaway: if you cannot replay a run, you cannot run the agent
Launching an agent is the easy part. Keeping it right while models update, data changes and users find new ways to ask is the actual job. That needs three things working together: traces detailed enough to replay any run, a short list of alerts that only fire when something is really wrong, and a weekly habit of reading real runs and turning failures into tests.
None of this needs an exotic stack. Standard OpenTelemetry spans, clear outcome definitions, a sensible content policy and an hour a week with a domain expert will put you ahead of most teams running AI agents in production today.
Want a second pair of eyes on an agent in production?
Tell us what the agent does, how many runs it handles a day and what you log today. We will tell you what we would trace, alert on and review first.
This article is general guidance. The hero illustration, sample trace, alert thresholds and dashboard values are fictitious and illustrative. OpenTelemetry names are taken from the OpenTelemetry GenAI semantic conventions, which are in Development status and may change. Product names are given as examples only. Case-study results are quoted from Bonami's published case studies; clients are anonymised and results will differ for other organizations.