ConceptsConceptsdashboard
Concepts
Every concept of variA/Bly evaluation in one place: what it is, why it matters, how it relates to the others, where you see it, with an example.
What you'll have
A clear picture of how a project, its evaluation policy and every score, check and loop fit together.
Everything in evaluation hangs off one object: the project's evaluation policy. A project is one AI application; creating it creates policy v1, so the first answer you send is scored with nothing to configure. Every answer records the policy version and the scoring rule that scored it, and a later change never rewrites an earlier verdict.
| Page | Concepts |
|---|---|
| Organisation, projects and access | Organisation, project, project domain, environment, API keys and scopes, roles, sign-in |
| Answers, sessions and agents | Answer (event), session and steps, agents and hand-offs |
| Policy, versions and re-scoring | Evaluation policy, versions, publishing, comparing, policy as YAML, re-score, verdict history |
| How answers are judged | Answer types, scoring rules, sample %, Max-Accuracy |
| Scores, checks and statuses | Dimensions, categories, weights, checks, per-dimension bars, statuses, Safety |
| Claims, grounding and where to correct | Claim verdicts, explanations, grounding and documents, SQL rows, where to correct |
| Data definitions | Code lists, document scope, hand-off contracts, SQL checks, posting rules, expected answers, requirement templates |
| Improve, review and release | Issues, human review, datasets, collection rules, runs, release gate, alerts, experiments |
| Integrations, data and usage | Integrations, deleting a project, data retention, usage and plans |
How the pieces connect
- You create a project and choose its domain. Policy v1 is created for you.
- Your app sends answers with an API key; steps of one conversation share a session.
- Each answer is pinned to the policy version in force and the first scoring rule that matches it.
- It is scored on dimensions in six categories, its claims are checked against its sources, and checks compare exact values against your data definitions.
- Failing checks become issues, with where to correct. People can review answers; alerts watch the trend; datasets, runs and the release gate test a release; experiments compare variants.
- You change the policy as a draft, publish it with a note, and can re-score past answers.