AI Agent Evaluation and LLM Observability: How to Know Your Agent Actually Works

AI agent evaluation answers one question: does this agent complete the real task, correctly and consistently, at a cost and speed you can live with? You answer it in two places. Before launch, you run offline evals: a fixed set of realistic tasks with known correct outcomes, scored automatically and spot-checked by people. After launch, you use LLM observability: traces of every run, cost and latency metrics, quality scores on live traffic, and the rate at which humans approve or correct what the agent does. Then you connect the two, so every production failure becomes a new test case.

Most teams have half of this. The LangChain State of Agent Engineering survey of 1,340 practitioners, fielded in late 2025, found 89% had some form of observability for their agents but only 52.4% ran offline evaluations on test sets. The same survey named quality as the top barrier to production, cited by 32% of respondents. Watching an agent is not the same as knowing whether it is right.

Why evaluating AI agents is harder than testing normal software

Traditional code is deterministic: same input, same output, so a unit test passes or fails. An agent is different in three ways.

• Outputs vary between runs. The same ticket can produce a different reply, or a different sequence of tool calls, on each attempt.
• There are many valid answers. Two refund emails can both be correct while sharing almost no words, so exact string matching fails.
• The path matters, not just the answer. An agent can reach the right final message after calling the wrong API, leaking data into a log, or burning ten times the expected tokens.

Reliability across runs is the part people underestimate. The τ-bench paper (Yao et al., 2024) tested tool-using agents in simulated retail and airline customer service and found that even state-of-the-art function-calling agents at the time, like GPT-4o, succeeded on under 50% of tasks, with pass^8 (succeeding on all eight repeated attempts) under 25% in retail. Models have improved since, but the lesson holds: a single successful demo tells you very little.

Offline evals: build a golden dataset first

A golden dataset is a set of realistic inputs paired with what "correct" looks like. For a support agent, that is real (anonymized) tickets plus the right resolution. For an invoice extraction pipeline, it is real PDFs plus the verified field values. Some practical rules:

• Start small and real. Anthropic's engineering team, in Demystifying evals for AI agents (January 2026), recommends starting with 20 to 50 simple tasks drawn from real failures rather than waiting until you have hundreds.
• Cover the ugly cases on purpose. Blurry scans, angry customers, requests in the wrong language, prompts that try to override instructions. The easy 80% will pass; you are measuring the rest.
• Grade outcomes, not wording. Where possible, check the end state: was the CRM field updated, was the right refund amount drafted, did the ticket get the right tag. τ-bench does exactly this by comparing the final database state with the annotated goal.
• Run each task several times. Report both pass@k (at least one of k attempts succeeded) and pass^k (all k succeeded). For a customer-facing agent, pass^k is the number that predicts complaints.
• Separate capability evals from regression evals. Anthropic's guide describes capability evals as targeting things the agent can't yet do well, and regression evals as a suite that should stay near 100% so nothing that used to work silently breaks.

Graders: code first, LLM-as-judge second, humans for calibration

Every task needs a grader: the logic that decides pass or fail. There are three kinds, and good eval suites mix them.

1. 1Code-based graders. Exact match on extracted fields, JSON schema validation, checking that a tool was called with the right arguments, verifying a database record exists. Fast, cheap and objective. Use them wherever the answer is checkable.
2. 2Model-based graders (LLM-as-judge). A second model scores the output against a rubric, such as whether a reply is grounded in the retrieved policy or whether its tone matches brand guidelines. Flexible and scalable, but not deterministic.
3. 3Human graders. A domain expert reviews a sample. Slow and expensive, and still the standard you calibrate everything else against.

LLM-as-judge is useful, and it has known biases. The original MT-Bench and Chatbot Arena study (Zheng et al., 2023) found strong judges like GPT-4 reached over 80% agreement with human preferences, about the same as agreement between humans. The same paper documented position bias, verbosity bias and self-enhancement bias. Other research sharpens the point:

• Order changes the verdict. In Wang et al. (2023), simply swapping the order of two answers let Vicuna-13B beat ChatGPT on 66 of 80 queries when ChatGPT was the judge.
• Judges favor their own outputs. Panickssery, Bowman and Feng (2024) showed LLM evaluators score their own generations higher than others' even when human annotators rate them as equal, and linked this to the model's ability to recognize its own text.
• Your criteria will drift. Shankar et al. (2024) found that people refine their grading criteria while grading outputs, so a rubric written up front is rarely final.

Online monitoring: LLM observability in production

Offline evals tell you the agent was good on the day you tested it. Observability tells you what it is doing now. The core unit is the trace: a record of one run, broken into spans for each model call, retrieval step and tool call, with inputs, outputs, timing, token counts and errors.

The emerging standard for this is OpenTelemetry's GenAI semantic conventions. As described on the OpenTelemetry blog, they define span types such as invoke_agent for the top-level agent run, chat for model calls and execute_tool for tool invocations, attributes like gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, and metrics including gen_ai.client.operation.duration and gen_ai.client.token.usage. The attribute registry also includes gen_ai.evaluation.* attributes for attaching eval scores. Two cautions: the conventions have moved to a dedicated repository and are still under active development, so expect names to change; and prompt and response content is not captured by default. You opt in, which matters when traces would otherwise contain customer data.

Using the standard is worth it even if you only ever use one tool, because it keeps your telemetry portable. You can switch observability vendors without re-instrumenting the agent.

Cost, latency and drift: the metrics that sneak up on you

Cost should be tracked per completed task, not per API call. An agent that retries three times and makes six tool calls can cost many times what the prototype suggested. Break it down by step so you can see which prompt or tool loop is expensive.

Latency needs percentiles, not averages. A p50 of 4 seconds with a p95 of 45 seconds means one in twenty customers waits almost a minute. Watch time to first token for chat interfaces and total time to completion for background jobs.

Drift has two sources. Your inputs change: new products, new document layouts, a new customer segment. And the model can change underneath you. Chen, Zaharia and Zou (2023) found GPT-4's accuracy on identifying prime numbers fell from 84% in March 2023 to 51% in June 2023, noting that the behavior of the "same" LLM service can change substantially in a short time. Pin model versions where your provider allows it, and rerun your eval suite on a schedule even when you have changed nothing.

Human review and approval rates as a metric

If your agent drafts actions that a person approves, you already have one of the best quality signals available. The LangChain survey found human review (59.8%) was still the most common evaluation method, ahead of LLM-as-judge (53.3%). Instrument the approval step and track:

• Approval rate without edits. The share of drafts a reviewer accepts as-is. This is your most honest measure of usefulness.
• Edit distance on approved drafts. Small tweaks versus near rewrites tell very different stories.
• Rejection reasons. A one-click reason code (wrong data, wrong tone, should have escalated) turns rejections into a labeled dataset.
• Time to approve. If reviewers take as long as doing the task by hand, the agent is not saving time yet.
• Escalation rate. How often the agent correctly hands off to a human. Too low can be as bad as too high.

Rejected and heavily edited drafts are the best source of new golden-dataset cases, because they are real failures from real traffic. Feed them back weekly.

The AI agent evaluation metrics table

Regression testing before every prompt or model change

Prompts are code. A one-line change to fix one customer's complaint can break ten other cases. Treat it like any other deploy:

1. 1Version prompts, tool definitions and model identifiers in source control.
2. 2On every change, run the regression suite automatically in CI, several trials per task.
3. 3Compare against the current production baseline on success rate, pass^k, cost and latency, not just the headline score.
4. 4Block the release if any regression-suite task that used to pass now fails, unless someone signs off on the trade-off.
5. 5For model upgrades, also run a shadow period: send a copy of live traffic to the new version, score both, and switch only when the numbers hold.
6. 6Read a sample of the transcripts. Anthropic's guide is blunt about this: scores only mean something if you have checked that failures are fair and graders measure what matters.

Tooling categories for LLM evaluation and observability

You don't need all of these, and the right choice depends on your stack and data rules. Listed neutrally as examples, not recommendations:

• Eval frameworks and test runners (define datasets, graders and CI checks): open-source options include promptfoo, DeepEval, Ragas for retrieval-heavy systems, OpenAI Evals and Inspect.
• Tracing and LLM observability platforms (collect traces, costs and scores): open-source or self-hostable options include Langfuse, Arize Phoenix and OpenLLMetry; commercial platforms include LangSmith, Braintrust, Weights & Biases Weave and Datadog LLM Observability.
• General observability backends that accept OpenTelemetry data, useful if you want GenAI traces next to the rest of your application telemetry.
• Annotation and review queues for human grading, often built into the platforms above or into your own approval UI.

If your data can't leave your cloud, a self-hosted, OpenTelemetry-based setup is usually the simplest path through security review.

How Flowrest Labs approaches agent evaluation

We build evaluation in from the first sprint, because "it looked fine in the demo" is how agents fail quietly in production. For each custom AI agent or workflow automation, we agree on what success means with your team, build a test set from your real examples, and put human approval gates in front of consequential actions so every approval and rejection becomes a measurable signal. Every build also ships with system health monitoring, automated error alerts and a 30-day post-launch support window, and you own the code, prompts and eval suite outright.

For more on the failure modes evals are meant to catch, see how to prevent AI hallucinations in production and why agentic AI projects fail. If you already run an agent and aren't sure how well it performs, AI consulting and workflow audits are a good place to start.

Want to know whether your current agent actually works? Book a free 30-minute workflow audit.