Evals

Evals are sets of prompts and expected responses that measure how well the agent answers questions that matter to your business. They are easy to underinvest in for AI analytics, and they are one of the main ways the data team stays proactive about quality instead of waiting for business users to hit bad answers.

Where to find it

Open Evals in the product to see defined evals and how the agent has performed on them over time.

What evals cover

Evals can be simple or complex:

• Lookup questions: for example, a metric value or breakdown that should always resolve through the semantic layer.
• Deep analysis questions: playbook-backed analyses where structure and methodology matter as much as the number.

Over time, teams often grow to dozens or hundreds of evals covering the facets of the business they care about most.

When to run them

Run evals:

• On a fixed schedule (for example daily or weekly).
• When you upgrade models.
• When you change context available to the agent (playbooks, AI context, or semantic definitions).

The result tells you whether quality improved or regressed before those changes reach every business user.

Who creates evals

• Your team can add custom evals for questions you know will be asked repeatedly.
• Sundial provides curated and auto-generated evals for common cross-customer question patterns.

Together with Observability and the Context Engine, evals complete the governance loop: watch real usage, improve context, and verify that quality holds.