Continuous Evaluation: The Engineering Foundation for Reliable LLM Applications

By Michael Mifsud·22 July 2026

The Probabilistic Trap

In traditional software development, code is deterministic. For a given input, you expect an identical, predictable output. If you write a sorting algorithm or a tax-calculating function, unit tests provide complete confidence: the system either works, or it is broken.

But when we build applications powered by Large Language Models (LLMs), this deterministic paradigm collapses.

LLMs are probabilistic. They do not operate on fixed logical branches; they operate on word-token probabilities. This introducing a unique engineering challenge:

• The Regression Ripple: A minor tweak to a system prompt to fix a specific edge case can silently degrade performance on five other previously solved cases.
• The Illusion of Stability: A prompt that appears to work perfectly during five manual trial runs on a developer’s machine can fail spectacularly when subjected to the diversity of live production traffic.
• The Non-Scalability of Tweaking: Treating prompt engineering as an ad-hoc cycle of trial, error, and manual inspection is not software engineering. It is guess-work.

If we want to build AI systems that are sufficiently robust to automate core business workflows, we must abandon blind prompt-tweaking. We need an evaluation-driven development workflow—a disciplined framework that treats every model interaction, prompt change, and retrieval adjustment as a testable hypothesis.

The Continuous Improvement Cycle: Analyse, Measure, Improve

Before writing a single test case, we must establish a repeating lifecycle for our LLM applications. This cycle consists of three continuous, non-negotiable stages: Analyse, Measure, and Improve.

Too many teams try to skip straight to the “Improve” stage, spending days rewriting prompts in a vacuum. This is a waste of engineering time.

1. Analyse: Capture and Inspect Traces

You cannot improve what you do not observe. Every meaningful interaction in your system must be traced and recorded. A complete trace is not just the final user query and the model’s response; it must include:

• The exact system prompts and template variables used.
• The raw retrieved document chunks (if using RAG).
• The model version, temperature, and top-p parameters.
• The complete raw output, including reasoning tokens and metadata.

Once this data is collected, developers must perform manual trace inspection. This is not a task you can instantly automate away. You must sit down and read the logs. Look for patterns in the failures:

• Are retrievals pulling irrelevant context?
• Is the model hallucinating details not present in the retrieved chunks?
• Is the tone misaligned with the brand?
• Are structured workflows breaking because of JSON formatting errors?

Understanding real user behaviors and failure modes is the absolute prerequisite to engineering a solution.

2. Measure: Define Actionable Metrics

Once you identify a recurring failure pattern, you must convert your observations into measurable metrics.

The golden rule of LLM metrics is to prefer simple, actionable, binary assertions over vague scores.

Avoid generic, multi-point qualitative metrics such as:

• Helpfulness: 4.2 / 5
• Empathy: 7 / 10

These scores are virtually useless to an engineer. If the helpfulness rating drops from 4.2 to 3.9 after a prompt change, what code needs to change? What line in the prompt broke? You have no way of knowing.

Instead, design highly specific, binary (Pass/Fail) metrics that directly target the engineering problem:

• Did the classifier accurately match the user query to the “billing” category? (Pass / Fail)
• Is the output a valid JSON document conforming to our Pydantic schema? (Pass / Fail)
• Did the response contain any prohibited competitor name? (Pass / Fail)
• Does the summary contain the three mandatory data fields extracted from the source document? (Pass / Fail)

These binary checks are clear, objective, and immediately point to the root cause of a regression.

3. Improve: Treat Changes as Hypotheses

Once your failures are measurable, you can safely begin the work of optimization. You might:

• Refine a system prompt to emphasize formatting constraints.
• Tune your RAG pipeline by adjusting chunk sizes or re-ranking retrieved documents.
• Swap out a general-purpose model for a fine-tuned, task-specific variant.

Every single change you make must be treated as an experimental hypothesis. Before you merge any prompt update or model migration, re-run your entire evaluation suite against your historical examples.

• Did the target metric improve?
• Did previously green assertions turn red (regress)?

Only when the evaluation suite demonstrates an overall net improvement should the change be allowed into production.

The Three Tiers of LLM Evaluations

A mature LLM application does not rely on a single type of test. To balance cost, speed, and qualitative depth, you must implement a tiered evaluation strategy spanning three distinct layers.

Level 1 — Programmatic Unit Evals

Unit evaluations are your first line of defense. They are deterministic, programmatic assertions that do not require calling another LLM. They are designed to run in seconds inside your continuous integration (CI/CD) pipeline on every single code commit.

Typical programmatic assertions include:

• Schema Validation: Ensuring the output matches a precise JSON schema or Pydantic model.
• Constraint Compliance: Verifying that a classification output belongs exclusively to an allowed list of strings (e.g., ["billing", "technical_support", "sales"]).
• Format Checks: Ensuring numeric outputs contain valid numbers, dates match standard formats, or generated links are valid URLs.
• Safety Assertions: Programmatically blocking common injection strings or verifying that prohibited keywords are entirely absent.

Because Level 1 evals do not require an LLM, they are virtually free and highly parallelizable. If a developer accidentally breaks a core JSON contract, Level 1 evals will catch it in CI before a single dollar is spent on downstream API calls.

Level 2 — Human + LLM Judge

Programmatic assertions are excellent for structural integrity, but they cannot evaluate subjective quality. They cannot tell you if an assistant’s response sounds empathetic, whether a technical explanation is actually clear, or if a summary contains subtle hallucinations.

For subjective quality, you need Level 2. The transition to Level 2 is achieved through a structured, multi-step alignment loop:

Step 1: Curate the Golden Dataset

Collect approximately 100 highly representative examples of real-world inputs and historical conversations. If your application has not launched yet, generate synthetic inputs using a strong model, but ensure they are carefully reviewed for realism.

Step 2: Establish the Expert Verdict

Create a centralized, shared spreadsheet. For each of your 100 representative inputs, run your current application and record the model output.

Have a domain expert—not the software developer—review each output. This reviewer could be a customer support lead, a product manager, or a legal counsel. For each row, the expert must record:

1. A binary verdict: Pass or Fail.
2. Detailed Notes explaining exactly why a response failed (e.g., “The tone was defensive when addressing the billing error,” or “The explanation of the API endpoint omitted the rate-limit constraint.”).

This spreadsheet is your ground truth. It defines exactly what “good” means for your specific business.

Step 3: Introduce the LLM Judge

Now, we automate the expert. Prompt a powerful, frontier model to act as an evaluator. Provide the LLM Judge with:

• The original user input and any retrieved context.
• The application output.
• The precise evaluation criteria established by the domain expert’s notes.

Your evaluator prompt must enforce a strict format:

1. Reasoning: Force the judge to write out its analytical reasoning step-by-step first. (This significantly improves evaluation accuracy).
2. Explanation: A brief summary of its decision.
3. Verdict: A strict binary PASS or FAIL.

Step 4: Measure and Tune Agreement

Run the LLM Judge across your 100 golden examples and compare its verdicts directly to the human expert’s verdicts. Calculate your Agreement Rate:

$$\text{Agreement Rate} = \frac{\text{Direct Agreements}}{\text{Total Examples}} \times 100$$

On your first run, you might see an agreement rate of 70%. Identify the 30% of cases where the human and the LLM Judge disagreed. Feed these disagreements into your development loop:

• If the human marked a response as a Fail but the LLM Judge marked it as a Pass, analyze why the judge missed the failure.
• Update your evaluator prompt to make the criteria more explicit. Add concrete examples of what constitutes a failure.
• Re-run the comparison.

Repeat this alignment cycle until your LLM Judge achieves a consistent 85% to 90% agreement rate with your human expert. At this point, you have successfully automated a highly reliable proxy for your human expert that can evaluate thousands of outputs in minutes.

Step 5: Prevent Evaluator Drift

An automated judge is not a “set-and-forget” utility. Regularly sample a small percentage of production traces (e.g., 2% of weekly traffic) and have your human experts manually audit the LLM Judge’s decisions. If you notice the agreement rate dropping over time—due to shifting user behavior, new product features, or model updates—re-tune your evaluator prompt.

Level 3 — Production A/B Testing

Level 3 is the final tier. It is only introduced after your application has achieved maturity in Levels 1 and 2.

Level 3 involves deploying two variants of your application (Variant A and Variant B) to separate cohorts of live users and measuring real business outcomes:

• Did Variant B reduce support ticket escalation rates?
• Did Variant B increase user task completion or conversion rates?
• Did Variant B improve user retention over a 30-day window?

It is critical to understand that Level 3 is for optimizing business outcomes, not for catching bugs. Both Variant A and Variant B must already be thoroughly vetted by your Level 1 and Level 2 evaluations before being exposed to live production users.

Common Evaluation Pitfalls

When teams begin implementing evaluations, they frequently fall into three classic traps:

1. Blind Prompt Tweaking

This is the most common pattern of failure. A developer sees a single bad output in production, immediately opens the system prompt, adds a highly specific instruction to block that specific failure, and pushes it to production.

This is the LLM equivalent of patching code without running your regression tests. You might fix the immediate bug, but you have no idea what else you just broke. Prompt tweaks should only be merged after they have been programmatically validated against your entire golden dataset.

2. Over-reliance on Generic Metrics

Many development teams waste weeks attempting to use generic, academic evaluations like “Truthfulness,” “Toxicity,” or “Helpfulness” from standard benchmark suites.

These metrics are designed for evaluating base foundation models, not specific domain-driven applications. Your users do not need a general-purpose helpful assistant; they need an assistant that can accurately process insurance claims, resolve API issues, or draft legally compliant contracts. Build your evaluations around your specific business constraints, not generic academic definitions.

3. Purchasing Complex Tooling Too Early

The enterprise AI landscape is flooded with expensive, complex LLM observability and evaluation platforms.

Do not purchase these tools before you know what you are measuring. An external platform cannot define what “quality” means for your business. First, build a simple, internal workflow to understand your application’s unique failure modes. Once you have a mature, working evaluation process, you can evaluate third-party tools to determine if they offer any meaningful operational efficiencies.

Bootstrapping a Practical Eval Suite

You do not need a massive budget or complex infrastructure to start building reliable LLM applications today. You can bootstrap a robust, production-grade evaluation system using standard engineering tools already present in your stack:

1. The evals/ Directory: Create a dedicated directory in your repository.
2. A Structured JSON Dataset: Store your 100 golden examples in a simple JSON file:

   [
     {
       "id": "case_001",
       "user_input": "My billing statement is showing a charge from July 15th that I don't recognize.",
       "expected_category": "billing",
       "required_keywords": ["charge", "billing", "statement"]
     }
   ]
   

3. Simple Python Assertions with pytest: Write standard programmatic tests to run in your local shell and CI pipeline:

   import pytest
   from my_app import process_query
   
   @pytest.mark.parametrize("test_case", load_golden_dataset())
   def test_classification_and_constraints(test_case):
       output = process_query(test_case["user_input"])
       
       # Assert valid structure
       assert output.get("category") == test_case["expected_category"]
       
       # Assert keyword inclusion
       for keyword in test_case["required_keywords"]:
           assert keyword in output["response"].lower()
   

4. A Strong Judge Prompt: Call a frontier LLM using a clear, highly structured markdown template to act as your Level 2 judge, dumping results back into a CSV file for manual audit.

By starting small, focusing on binary, business-aligned metrics, and automating only after establishing human agreement, you build more than just a better prompt. You build a predictable, repeatable, and scalable engineering process that allows your AI systems to improve safely and continuously over time.