How to test an LLM system

An LLM system is not one thing. It is a chain: interpret the question, choose a tool, produce the arguments for that tool, retrieve, then write. Teams reach for the end of that chain when they think about testing, because that is what the user sees. It is the worst place to start.

Assert the arguments, not the prose

On a conversational analytics platform and again on a RAG assistant, we built integration suites that do something simpler than they sound: given a prompt, the test asserts that the model produced the correct retrieval filters. Not the right answer — the right query.

That single move changes the economics of testing. The assertion is deterministic: a set of filters either matches or it does not. It is fast, because you are comparing structured values, not judging text. It runs on every release like any other unit test, in CI, with no human in the loop and no scoring model.

The test checks the query, and never runs it. Filter regressions are caught without paying for retrieval.

It also stops the bill from growing

There is a second benefit that nobody puts in the eval literature. Because the suite never executes the query, it does not touch the search index. On one project the retrieval cost had become material at production volume — enough that running the full end-to-end suite on every commit would have been a line item somebody noticed. Testing the tool-call layer alone removed that entirely.

Quality and cost usually pull against each other in these systems. This is one of the few places where the same decision improves both, which is why it is the first thing I set up now.

What this does not cover

It does not tell you whether the final answer is good. Nothing cheap does. For that you need the slower half: a set of reference cases with expected outputs, a rubric, sometimes with a model applying it where that earns its place, and above all a validation programme run with the people who are accountable for the output.

Each of those three later got an article of its own, and they are worth reading in that order: first the yardstick, then who holds it, finally who answers for what comes out.

On the guideline work, that meant a long, iterative programme with the specialists who sign the document. Their feedback is what made the pipeline good, not the model choice. It took months, and it was the most valuable part of the project.

A practical order

Start with the tool-call assertions, because they are cheap and they catch the regressions that actually happen. Add reference cases for the generated output, run against them before each release rather than each commit. Then, in production, capture structured feedback from the people using it — a rating and a line of text on each output is enough to see where quality drifts.

If it isn't measured, it isn't finished. But measuring does not have to mean an expensive end-to-end harness nobody runs twice.

The series that came out of it

The three things this piece leaves open each got their own article, and they are worth reading in this order.

Golden dataset and test set is the yardstick. Without a set of examples carrying the expected answer, "it works better" stays an opinion, and every other technical decision rests on nothing.

LLM-as-judge is who holds the yardstick once the cases become too many for one person. It works, but only after being calibrated against real human judgement — and with its own documented biases kept in check.

Human-in-the-loop is the person who stays in the loop regardless. It is not a substitute for the measurement: it is how you survive while you build it, and by the third month you can see whether it was designed well.