Observability

Observability is where the data team keeps a pulse on how the agents are used: which questions people ask, how sessions go, and how trustworthy the answers look. The goal is to spot problems early (usually missing or incomplete context) and improve the system before bad answers spread.

Where to find it

Open Observability in the product. From there you can review aggregate usage and drill into individual sessions.

What you can see

• Volume and mix: how many questions are being asked and what types they fall into.
• Sessions: each conversation, with the ability to open the full session.
• Trust signals: an assessment of each answer, high, medium, low, or unverified.
• Playbook usage: whether a playbook was used for the question.
• Categories: how questions are classified so you can find patterns.

How trust signals are assessed

Trust generally depends on factors such as:

• Whether the answer came from the semantic layer (verified metrics) or required custom SQL.
• Whether a playbook guided the analysis or the model had to improvise a path.

High trust usually means the agent stayed on governed definitions and known playbooks. Lower or unverified signals are a cue to inspect the session and decide what context is missing.

Closing the loop

Typical issues you will find:

• Someone asked about a metric that is not defined in the semantic layer yet.
• The agent answered with a methodology your team does not prefer.

Use those findings to update the Context Engine: add semantic definitions, AI context, or playbooks, and confirm quality with Evals. Observability turns reactive firefighting into a deliberate feedback loop between real usage and better context.