An AI agent without metrics is a risk, not a tool
You have deployed an AI agent — support chatbot, sales assistant, extraction pipeline. It seems to work. But how do you know it works well? That its responses are correct, that its performance is not degrading, that users are satisfied? Without metrics and tests, you are flying blind. And an AI agent that hallucinates in production can cost far more than the problem it is supposed to solve.
The problem
Evaluating AI agents is the poor relation of enterprise AI projects:
"It seems to work" is not enough
The majority of AI agents in SMBs are evaluated anecdotally: you test a few queries by hand, it looks fine, you deploy. But the LLM is probabilistic — the same question can give a good answer 9 times out of 10 and a hallucination the 10th. Without systematic evaluation on a sufficient volume, you do not know your real error rate. On 1,000 interactions per month, a 5% hallucination rate means 50 false answers sent to your customers.
Silent degradation
An AI agent can degrade without anyone noticing: the provider updates the model (a subtle change in behavior), the knowledge base becomes obsolete, a new type of request appears that the agent cannot handle. Without continuous monitoring, quality drops from 95% to 80% over 3 months — and you only discover it when an important customer complains. The cost of this silent degradation is invisible but real.
What to measure, how to measure?
Technical teams know how to evaluate a classification model (accuracy, F1-score). But a conversational AI agent is more complex: how do you score the quality of a natural-language response? How do you measure whether the agent understood the intent correctly? How do you detect a subtle hallucination? Classic ML metrics do not apply directly, and the new metrics (faithfulness, relevance, coherence) are little known outside research teams.
The AI solution
The evaluation matrix covers three dimensions with concrete metrics and actionable thresholds:
Dimension 1: Response quality
Four key metrics: Faithfulness (is the response faithful to the sources? Threshold > 95%), Relevance (does the response answer the question asked? Threshold > 90%), Coherence (is the response logical and well structured? Threshold > 90%) and Hallucination rate (% of responses containing invented information, threshold < 2%). These metrics are calculated automatically via LLM-as-judge and validated by human sampling.
Dimension 2: Operational performance
Five metrics: Response time (P50, P95, P99 — threshold P95 < 5 s), Resolution rate (% of requests resolved without a human — target 60-70%), Escalation rate (% transferred to a human — target 15-25%), Cost per interaction (in euros, including tokens + infrastructure) and Availability (uptime > 99.5%). These metrics are collected automatically by the observability pipeline.
Dimension 3: Security and compliance
Three metrics: Blocked prompt injection rate (% of attacks intercepted — target > 95%), Data leaks (number of responses containing unauthorized sensitive data — target 0) and Scope compliance (% of responses staying within the agent's authorized scope — target > 99%). These metrics are measured by automated adversarial tests and production monitoring.
Implementation
Setting up evaluation in three phases:
Phase 1 — Create the test dataset (weeks 1-2)
Build a test set of 200 to 500 cases representative of your real interactions. For each case: the user query, the expected response (ground truth), and the scoring criteria. Include the edge cases, ambiguous questions and injection attempts. This dataset is your reference for all future tests. Put it under versioning (Git) and enrich it every month with interesting real cases.
Phase 2 — Automate the evaluations (weeks 3-4)
Configure an automated evaluation pipeline with promptfoo, Ragas or DeepEval. The pipeline runs the agent on the test dataset, calculates the metrics via LLM-as-judge, and produces a report. Integrate this pipeline into your CI/CD: it runs automatically on every prompt or configuration change. Define the passing thresholds: if faithfulness < 95%, the deployment is blocked.
Phase 3 — Continuous monitoring in production (weeks 5-6)
Deploy monitoring that collects metrics in real time on each production interaction. Configure alerts on degradations: if response time P95 > 8 s, if the escalation rate > 35%, if the average quality score drops by more than 5 points. Plan a weekly human audit on 50 random interactions. Produce a dashboard accessible to the whole team with 30-day trends.
Results
Frequently asked questions
How do you measure the quality of an AI agent's responses?
Three complementary approaches: automatic evaluation by an LLM-as-judge (a second model scores the first one's responses against defined criteria), human evaluation on a sample (50-100 cases per week), and technical metrics (faithfulness, relevance, coherence). Combining the three gives a reliable view of quality in production.
How often should an AI agent be tested?
Regression tests should run automatically on every modification of the prompt, configuration or model. In stable production, a weekly audit on a sample of 50 to 100 interactions is enough. During the deployment phase, evaluation should be daily. Real-time monitoring complements the planned tests.
What is the minimum set of metrics to track for an SMB?
To start, 5 metrics are enough: resolution rate (% of requests handled without a human), average response time, satisfaction rate (user feedback), hallucination rate (factually incorrect responses) and escalation rate. These 5 indicators cover quality, performance and security. Add finer metrics when volume justifies the investment.
How do you detect an AI agent's hallucinations?
Three methods: cross-checking against the source base (does the agent cite information that actually exists in the knowledge base?), the self-consistency check (does the model give the same answer when you rephrase the question?) and the LLM-as-judge (a second model evaluates the factual fidelity of the response). In production, the acceptable hallucination rate is < 2% for customer support and < 0.5% for regulated domains.
For technical profiles
Complete quality matrix of an AI agent
| Metric | Recommended threshold | Measurement tool | Frequency |
|---|---|---|---|
| Faithfulness | > 95% | Ragas, DeepEval, LLM-judge | Each deployment + weekly |
| Relevance | > 90% | Ragas, promptfoo | Each deployment + weekly |
| Hallucination rate | < 2% | LLM-judge + human audit | Continuous + weekly |
| Response time P95 | < 5 s | LangSmith, Datadog | Continuous |
| Resolution rate | 60-70% | Custom dashboard | Daily |
| Prompt injection blocked | > 95% | Garak, promptfoo | Each deployment |
| Cost per interaction | < EUR 0.15 | Token counter + infra | Weekly |