Prompt Evaluation
The process of testing a prompt across representative inputs using criteria such as correctness, format compliance, safety, consistency, latency, and cost.
Choosing between prompts by reading one attractive answer is weak evidence. Prompt evaluation measures behavior over a representative set of inputs.
A useful evaluation set includes more than routine examples. Depending on the system, it can include boundary cases, missing context, contradictory evidence, and adversarial or policy-sensitive inputs.
Possible measurements depend on the task:
| Dimension | Example | |---|---| | Correctness | agreement with a reference | | Format | schema or JSON validation | | Consistency | variation across repeated runs | | Safety | prohibited action or data exposure | | Cost | token use and latency |
A single aggregate score is not always sufficient. In a critical workflow, a rare high-impact failure can outweigh a small improvement in average quality.
See Prompt Engineering.