An AI feature can be wrong in a way a normal feature cannot. A validation error is caught by a schema. A wrong calculation is caught by a test. An AI feature can produce output that is well-formed, confident, plausible and still wrong, and no amount of looking at it tells you whether this particular answer is one a customer would be embarrassed by.
That gap is why evaluation tends to arrive late. The prompt gets tweaked because output looked off in a demo, the model gets swapped, three weeks later nobody can say whether the feature is better or whether the examples they remember were the easy ones.
Why “feels better” is not an evaluation
Subjective assessment is the default because it is the cheapest thing available, and it genuinely catches things an assertion cannot: a tone that is technically compliant but reads as though the model was being polite, an explanation that is correct but buries the answer.
Its failure mode is that it is not repeatable. Two reviewers judging the same output disagree. The same reviewer judging it a week later disagrees with their earlier self. Because there is no recorded set of inputs, a regression looks like a change in mood rather than a change in behaviour, and the response is another round of prompt tweaking with no way to tell whether it helped.
The practical symptom is that prompt changes accumulate without evidence. Not because anyone is careless, but because the loop has no measurement in it. An evaluation is what turns "this looks better" into "this fixes cases 7, 12 and 19 and broke case 23", which is a claim you can argue with.
The security controls for AI features need the same discipline for a different reason, covered in building AI-powered business applications without sacrificing security.
Build the evaluation set before you iterate
The dataset is the asset. Prompts are cheap to write and cheap to change; a set of representative inputs with recorded expected behaviour is expensive to build and gets more valuable over time.
A useful set is small and specific rather than large and generic. Start with a small representative set drawn from the feature's real inputs: ordinary requests, edge cases your support inbox knows about, known failure modes, and deliberate refusal cases. No size is universally right, so expand it when production or user-review failures reveal a case it missed. Keep each case in a versioned file so it diffs like code.
One case is worth describing in full, because the shape does the work. A lead qualification case carries an identifier, the input, what to expect, and a pair of tags. The input is a company, its approximate size, and a message about replacing spreadsheets, with no budget mentioned. What to expect is a named schema, the fields the application depends on with the budget band recorded as unknown, a flag saying no refusal is expected, and a note in prose explaining why: the budget was never stated, so a correct answer records that rather than inferring a figure.
The schema and fields are machine-checkable. The note is for a person. Keeping both in one record is the point: the automated checks stay cheap and fast, and the judgement calls are written down where the next person can read them.
The case worth including from the start is the one where the honest answer is that the system does not know. Features get tuned toward confident output, and a model that infers a figure nobody stated is a defect that looks like competence.
Deterministic checks come first
Run every check that needs no judgement before any check that does. Deterministic assertions are fast, free, reproducible and they do not get tired, so they should gate every run including the local one.
- Schema validity. Does the output parse against the shape the application expects?
- Required fields. Is every mandatory field present and non-empty?
- Value domains. Is each field inside the set the application allows?
- Grounding. Does every asserted fact appear in the supplied input?
- Refusal behaviour. When the case expects a refusal, did it refuse, and did it do so without inventing a partial answer on the way?
- Length and format. Are there the output size and formatting constraints the UI depends on?
Grounding is the check teams most often skip and the one that catches the most. Extract the factual claims from the output, check each against the input, and fail the case when one is unsupported. A model that invents a company size is easy to catch this way and hard to spot in a manual reading.
Keep these checks in the ordinary test suite. A schema assertion belongs next to the other unit tests, not in an evaluation script only someone remembers to run.
Model-graded evaluation and its limits
Some things are not decidable by assertion: whether the summary is faithful, whether the tone is appropriate, whether the explanation actually answers the question. Model-graded evaluation uses a model to judge output against a rubric, which extends coverage to those cases.
A rubric has to be written down, because the judge is otherwise measuring its own preferences. Three to five criteria, each with a clear threshold, each about something observable: does the response address what was asked, is every claim traceable to the input, does it state uncertainty where the input is silent, is it free of invented specifics.
The limits are real. A judge model has its own failure modes, and on a task similar to the one being judged it tends to agree with itself. Rubrics drift toward whatever the current output looks like if they are not pinned. And the judge is not ground truth: its scores are a proxy that needs periodic checking against human review.
The workable arrangement is a narrow role for the judge. Deterministic checks gate correctness. The judge ranks the cases that passed, so human review looks at the plausible ones instead of the whole set. And a sample of human judgement is compared against the judge regularly, because a judge that has quietly drifted is worse than no judge: it still produces confident numbers.
Latency and cost budgets
Both are quality attributes of the feature, and both are cheaper to hold as numbers before launch than as complaints after it. Neither budget is a target for the model; both are constraints the application is designed around.
A budget record has three parts, and the numbers in them are illustrative rather than recommended. Latency carries a median and a tail: the wait a typical user perceives, and what the interface has to tolerate. Cost carries a ceiling for a fixed number of runs, set from your own provider pricing. Reliability carries a ceiling on the rate of retries the feature is allowed to need.
A median alone is not enough. Feature changes tend to leave the median roughly where it was while the tail grows, because the change adds a conditional path that only some inputs take. Track the tail, and design the interface around it: a streaming or partial-progress surface for a long operation is usually worth more than a faster model.
Cost needs the same treatment. Per-run cost is not the number that matters; cost multiplied by volume is, and it changes when the feature gets a wider audience. Long prompts and large retrieved contexts are the first things to cut when a budget is missed, and a per-user cap on an expensive path is a normal product decision.
What belongs in CI, and what belongs with a person
The split is by cost and by judgement, not by importance.
On every pull request, the deterministic tier: schema and value-domain checks, deterministic refusal and contract cases, a cheap stable regression subset, and integration checks run against mocked or stubbed dependencies rather than a live provider. This tier must be fast enough that nobody is tempted to skip it, because a slow gate gets disabled, and a disabled gate is worse than none.
On a schedule, or before a release, everything that depends on a live model or a provider: the full evaluation set, model-graded scoring, provider latency and cost, and repeated measurements wherever variance matters, with scores recorded per run. Storing that history matters more than any single threshold.
With a person, because a machine cannot settle it: a representative sample, any case a reviewer disputes with the judge, and every case where a real user reported a problem. Those three feeds keep the evaluation set honest about what the feature does.
Latency and cost are release budgets, and a prompt edit that doubles either is a real regression even when every case still passes. Keep them separate from quality scoring, and measure them on that scheduled tier rather than gating every pull request on a live provider call.
A minimal evaluation harness
The harness does not need to be clever. It needs to run, produce a diff against the previous run, and fail the build.
In about twenty lines in any language it does four things: it takes the cases and the feature under test, runs the feature once per case, collects the failures from the schema check, the field check, the grounding check and the refusal check, and returns one result per case carrying its identifier, its tags and its failures.
Then it runs twice. The first run is against the configuration in production, the second against what the pull request proposes, and the comparison is the output. A case that passed before and fails now is a regression, and the build fails listing those case identifiers.
That comparison is the part worth keeping. Evaluating a proposed configuration alone produces a score with no reference point, and a score without a baseline cannot distinguish a change that fixed three cases from one that fixed three and broke two.
Human review as part of the system
Review is not a stage before launch. In a feature whose output varies, human judgement is part of how the system knows what it is doing, and it has to be designed rather than bolted on.
Give the queue a shape: the case, the input, the output, the deterministic results, the judge score, and a free-text verdict. Ask for a decision rather than a score, because a reviewer asked to rate quality will rate everything around seven. Offer three outcomes and accept nothing: fine, usable after an edit, not shippable.
Route the third category somewhere specific. A case a reviewer marks unusable is either a gap in the evaluation set, a defect in the prompt, or a limitation the feature does not have. Those need three different responses, and the distinction is lost the moment the verdict is a number in a table.
Sampled review also does something a total review cannot: it gives evidence from a defined, representative sample rather than from the cases somebody happened to look at. Read the figure it produces as an observed rate in that sample, not a population estimate: precision depends on sample size and composition.
The AI Sales Platform concept Superwebtech has in development is designed to use AI across discovery, enrichment, qualification and outreach, which is the shape where output quality varies and nobody can read every result. An evaluation set per stage, with the stage carrying the most risk getting the largest sample, is how that stays measurable. It is in development, so none of its numbers exist yet and none are claimed here.
Keep the evaluation set current
An evaluation set decays. Cases reflect the feature as it was, the feature changes, and the gap grows quietly. Three habits keep it aligned.
- Add a case every time a real user reports a bad output, before fixing the cause. The reported case is the one your set was missing.
- Retire cases the feature has genuinely stopped producing rather than leaving them passing for the wrong reason, so the set does not drift towards measuring obsolete behaviour.
- Re-read expectations when the prompt or model changes in a way you did not intend. Most unexpected regressions are a changed answer to a case whose expectation was too vague to catch it.
Track the composition as well as the total. A set that is ninety per cent ordinary requests will report ninety per cent success while saying nothing about refusals, edge cases or adversarial input, because that is all it contains. Tags exist so you can ask what a score is made of.
The measure of a working evaluation process is not the score. It is that when someone proposes a change, the answer to "is this better" arrives in minutes and names the cases that moved. Teams that get there make fewer, larger prompt changes.
If you are working out where an evaluation process belongs in your own delivery, the services page covers how we scope AI work, and the AI solutions page sets out the application areas we build against. Latency and cost budgets are treated as architecture in modern website performance, and getting that wrong is a common reason a fast feature turns slow.
