The useful unit of evaluation for an LLM product is usually not the completion. It is the task.
A response can read perfectly and still come from a failed system. The retriever may have returned the wrong document. The agent may have selected the wrong tool. The answer may not be supported by its evidence. A valid JSON object may contain the wrong values. The user may eventually get the right result after twelve seconds and several unnecessary model calls.
So I prefer to separate evaluation by layer.
Offline evaluation starts with representative tasks and explicit definitions of success: correct answer, correct evidence, correct tool, valid arguments, appropriate refusal, successful final state, or whatever the workflow actually requires. Components should also be evaluated separately when possible. An end-to-end failure tells you that something went wrong; it does not tell you where.
Online evaluation answers a different question: what happens when real users interact with the system? Task completion, corrections, escalation, latency, failures, cost per successful task and repeated human overrides often tell you more than another model benchmark.
LLM-as-judge is useful for open-ended outputs, especially for comparison and triage, but the judge is another model with its own biases. I want it calibrated against examples humans have actually labelled.
The point is not to accumulate metrics. It is to make a failure attributable.