Every modification of your AI must pass tests
An innocuous prompt change that halves answer quality. A provider model update that breaks the output format. An addition of documents to the index that pollutes existing results. These regressions are frequent and silent in LLM applications. The solution: a non-regression testing protocol adapted to the specificities of LLMs — semantic evaluation rather than exact equality, configurable tolerance thresholds, and CI/CD automation. Our clients who test before each deployment detect the vast majority of regressions before going into production.
The problem
LLM applications are fragile by nature. Unlike classic software where inputs and outputs are deterministic, an AI assistant is sensitive to dozens of interconnected parameters: the prompt content, the model version, the composition of the vector index, the retrieval parameters, the temperature, and even the order of the chunks in the context.
The most common causes of regression:
- Prompt modification — A developer adds an instruction to the system prompt to fix an edge case. The fix works, but degrades 15% of the answers on other question categories. Without a test suite, the regression goes unnoticed for weeks.
- Provider model update — OpenAI, Anthropic and Google update their models regularly (new versions, deprecations). Each update can change the model's behavior on your specific use cases. A prompt optimized for GPT-4o-2025-05 may produce different results on GPT-4o-2025-08.
- Document index evolution — Adding 500 new documents to your vector base can pollute the retrieval for existing queries. The new chunks compete with the old ones in the top-K. The retriever surfaces less relevant documents for questions that worked well before.
- Configuration change — Modifying the number of retrieved chunks (from 5 to 10), the embedding model, or the reranking parameters impacts the whole chain in cascade. Each parameter interacts with the others in a non-linear way.
The AI solution
A non-regression testing protocol for LLMs is structured around three components: a reference test set, semantic assertions, and a CI/CD integration.
Reference test suite (Golden Set)
Build a set of 200 to 500 test cases covering your query categories: factual (40%), comparative (20%), procedural (20%), multi-document (10%), out of scope (10%). Each case includes: the question, the expected answer, the relevant sources, and the specific evaluation criteria. Version the Golden Set in Git and update it quarterly.
Semantic assertions
Replace assertEqual with semantic assertions: does the answer contain the expected key points (semantic recall)? Is it faithful to the sources (faithfulness > 0.85)? Does it respect the expected format (valid JSON, citations present)? Use DeepEval or Ragas as the assertion framework. Define thresholds per test category: stricter for regulatory questions, more tolerant for summaries.
CI/CD automation
Integrate the test suite into your CI/CD pipeline. Automatic trigger on each PR that modifies a prompt, a pipeline parameter or the model configuration. Execution on a staging environment with a stable test index. The pipeline blocks the merge if a metric drops below the threshold. Detailed report with the failed test cases and the comparative before/after scores.
Implementation
Setting up an LLM non-regression suite takes 3 to 5 weeks. Here are the three key phases.
Building the Golden Set (weeks 1-2)
Collect the 200 most representative questions from your users (production logs). For each question, generate an answer with your current pipeline and have it validated by a business expert. Annotate: is the answer correct, complete, well-sourced? Store everything in a versioned JSON file. Categorize by query type and by criticality (blocking, major, minor). Plan for 20% trick questions to test the refusal.
Assertion framework (weeks 3-4)
Implement the assertions with DeepEval: assert_test() for each case, with faithfulness, relevance and format metrics. Configure the thresholds per category: regulatory (faithfulness > 0.90), informational (> 0.80), conversational (> 0.70). Add format assertions: presence of citations, min/max length, no personal data. Run the complete suite and fix the false positives (poorly formulated test cases).
CI/CD integration and maintenance (week 5+)
Integrate into GitHub Actions or GitLab CI: automatic execution on each PR touching the prompts/, config/ or models/ folder. Parallelize the test execution (5 workers, 200 tests in ~10 minutes). Configure a before/after comparison report visible in the PR. Maintain the Golden Set: add 10 new cases per month, remove obsolete cases, recalibrate the thresholds quarterly.
Results
Results measured after setting up non-regression tests on AI assistants in production.
FAQ
Why aren't classic tests enough for an LLM?
Classic unit tests verify deterministic outputs (assertEqual). An LLM produces non-deterministic outputs: the same question can generate two different answers, both correct. LLM non-regression tests must evaluate semantic quality (is the answer faithful, relevant, complete?) rather than exact equality. This requires specific metrics (embeddings similarity, LLM-as-Judge) and tolerance thresholds.
What should you test before changing the LLM model?
Run your complete non-regression suite (200+ test cases) on the new model and compare the faithfulness, relevance and format compliance scores with the old model. Recommended thresholds: no metric should drop by more than 3 points. Also test the latency and cost per query. Plan a canary deployment (10% of traffic) for 1 week before the full switch.
How do you automate non-regression tests in a CI/CD pipeline?
Integrate the tests into your GitHub Actions, GitLab CI or Jenkins pipeline. On each pull request that modifies a prompt, a model or the RAG configuration, the pipeline automatically runs the test suite on a staging environment. Use DeepEval or Ragas in pytest mode for the assertions. The pipeline blocks the merge if a metric drops below the threshold. Typical execution time: 5 to 15 minutes for 200 test cases.
For technical profiles
Technical implementation of LLM non-regression tests
The implementation rests on DeepEval (pytest framework for LLMs) or Ragas with a pytest wrapper. Each test case is defined as a tuple (input, expected_output, context) and evaluated by semantic metrics rather than by exact equality.
Golden Set structure (JSON):
Each entry contains: id, category (factual/comparative/procedural/out_of_scope), question, expected_answer, expected_sources (array of doc_ids), evaluation_criteria (specific metrics and thresholds), severity (critical/major/minor).
Recommended assertion types:
- Faithfulness — Is the answer faithful to the context? Threshold configurable per category. Implementation: Ragas faithfulness or DeepEval FaithfulnessMetric.
- Semantic similarity — Is the answer semantically close to the expected answer? Cosine on embeddings, threshold > 0.75. Useful for checking completeness.
- Format compliance — Does the answer respect the expected format? Valid JSON, citations present, length within bounds. Classic assertions (assertEqual, assertContains).
- Toxicity / Safety — Does the answer contain no dangerous content? Guardrails AI or a toxicity classification model. Threshold: 0 tolerance.
Comparison of LLM testing frameworks
| Criterion | DeepEval | Ragas + pytest | Promptfoo | Custom (in-house) |
|---|---|---|---|---|
| Ease of CI/CD integration | Excellent (native pytest) | Good | Good (CLI) | Variable |
| Number of metrics | 14 metrics | 8 metrics | 5 metrics | Custom |
| Comparison reports | Confident AI | Basic | Web UI | To be developed |
| Cost | Open source | Open source | Open source | Dev time |
| Learning curve | Low | Medium | Low | High |