Prompt Evaluation
The process of testing a prompt across representative inputs using criteria such as correctness, format compliance, safety, consistency, latency, and cost.
Choosing between prompts by reading one attractive answer is weak evidence. Prompt evaluation measures behavior over a representative set of inputs.
A useful evaluation set includes more than routine examples. Depending on the system, it can include boundary cases, missing context, contradictory evidence, and adversarial or policy-sensitive inputs.
Possible measurements depend on the task:
Evaluation can combine several task-dependent dimensions. Correctness can be measured against a reference response; format compliance through schema or JSON validation; and consistency through variation across repeated runs. Safety can be assessed through prohibited actions or data exposure, while cost includes token consumption and latency. These dimensions measure different properties and can be reported separately according to the task's requirements.
A single aggregate score is not always sufficient. In a critical workflow, a rare high-impact failure can outweigh a small improvement in average quality.
See Prompt Engineering.