HelpGlossarydashboard
Glossary
Every term variA/Bly uses, in plain words: the same definitions you see when you hover a label.
Every term variA/Bly uses, in plain words. Most are also shown when you hover over the label in the product. Terms are in alphabetical order.
| Term | Meaning |
|---|---|
| Advanced settings ▸ | Every setting: dimensions and weights, scoring rules, data definitions, versions and the policy as YAML. |
| Agent | The agent or component of your app that produced this answer. |
| Agent flow | The path through the agents, with the result of each hand-off. |
| Agent path | The agents the conversation passed through, in order. |
| Agreement | How often reviewers and the scorer reach the same verdict, overall, per check and per claim. |
| Alerts | Rules that watch a metric of this project's answers over a time window and notify a webhook or email when it crosses a threshold. |
| All passed | Every check that applies to this answer passed. |
| Answer type | How claims are judged. Answers from sources: every factual claim must be supported. Document analysis: the same, but introductions aren't counted and conclusions drawn from verified facts pass. |
| Answers from sources | Every factual claim in the answer must be supported by the documents or query results it was given. For Q&A, chat, RAG, text-to-SQL and agents. |
| Answers what was asked | Does the answer give the kind of thing the question asked for, for example an amount when asked “how much”? |
| API key | A secret your app sends with each answer to prove it belongs to your project. Created under Project configuration › Integration & keys; each key has the permissions you chose. |
| API keys | Keys your apps and CI use to send answers or read results. Each has a scope. |
| Applies to | Which answers this rule covers, built from fields, operators and values. |
| Audit log | Every policy change in this project, who made it and when. |
| Breaks a business rule | Shown on the sentence that states a value breaking one of your business rules, with the reason. |
| Business rules | Your own rules an answer must obey, such as a benefit limit or a deadline, checked in code. Their check is Follows business rules. |
| Change note | What changed in this version and why. |
| Change the evaluation policy | The API key permission needed to push or publish the policy from code, and for the SDK settings calls. |
| Chart of accounts | The ledger accounts your company books to. An entry to an account not on it is proven wrong. |
| Checks | Yes/no checks that compare exact values: document version, hand-off fields, SQL rules, expected answers, posting rules and more. |
| Checks & weights | Every dimension the scorer has, by category, with each category's weight in the overall score. |
| Claim | One checkable statement in an answer, such as “the deductible is $1,500”. Each claim gets a verdict. |
| Code lists | Your lists of valid codes and their spellings, for example leave types or states, so values match even when written differently. |
| Cohen's kappa | Agreement corrected for the agreement two raters would reach by chance. 1 is perfect agreement, 0 is no better than chance. Shown only once there are enough pairs. |
| Coherence | Is the answer logically consistent, within itself and across the steps of a session? |
| Collection rule | A rule that copies live answers into a dataset after they are scored. |
| Collection rules | Rules that copy live answers into a dataset after they are scored, for example 10% of production answers scoring below 70. |
| Contradicted | A source says something different, for example a different figure or date. Needs attention. |
| Create a project | Set up a new project. Evaluation starts with the first answer you send; there is nothing else to configure first. |
| Data definitions | Your code lists, document scope, hand-off contracts, SQL schema and rules, posting rules, expected answers and requirement templates. |
| Datasets | Fixed sets of questions, optionally with expected answers. |
| Datasets & runs | Optional. Fixed sets of questions you send through your app for releases, and comparisons between runs (for example release 4.2 vs 4.3). |
| Decided | A winner was chosen. |
| Decision metric | In an experiment: the one metric, chosen in advance, that decides the winner. |
| Delete project | Permanently delete this project and everything in it. |
| Depth | Reasoning, use of evidence, and for agents: tool use, step order and efficiency. |
| Deterministic | Scored by the standard checks only. The same answer always gets the same score. |
| Dimensions at their bar | The check that an answer’s dimensions are at or above the bars set in Checks & weights. |
| Document analysis | For reviews and reports. Facts must still be supported, but introductions aren't counted, conclusions drawn from verified facts pass, and a claim naming a document is checked against that document. |
| Document scope | What makes a document apply: dates, plan or academic years, regions, products, versions. Lets variA/Bly catch answers built from the wrong version. |
| Domain | The project's subject area. It picks the grounding models. |
| Draft | A version being edited. It scores nothing until you publish it. |
| End a session | Telling variA/Bly a conversation is over (client.end_session), so the checks over the whole conversation run. |
| Environment | The label the answer was logged with (environment= in the SDK), for example production. Empty when none was sent. It is never a policy scope. |
| Evaluation policy | The project's rules for scoring answers: the answer type per scoring rule, Max-Accuracy, category weights and the data the checks compare against. One version is in force for the whole project; every change is a new version. |
| Event | One answer your app sent: the question, the answer and everything sent with it. |
| Expected answers | Reference answers you send with an event, compared by value. |
| Experiment variant | The experiment variant (prompt or model version) that produced this answer, if it was part of an experiment. |
| Experiments | Split live traffic between prompt or model versions and compare each with the control on the scores and checks of real users' answers. |
| Fail | The check ran and found a problem. Open the answer to see the evidence. |
| Faithfulness | How well the answer's claims are supported by its sources, from 0 to 1. |
| Field | A field declared in the hand-off contract. |
| Follows business rules | Does the answer obey your business rules? A broken rule is confirmed by the reasoning model before it counts; it never changes grounding. |
| Follows from verified findings | A conclusion worked out from claims that were themselves verified, for example a difference between two checked figures. Document analysis only. |
| Framing (not counted) | An introduction or connecting sentence with nothing to check, like “Here is my review”. Shown, but not counted. Document analysis only. |
| Grounding | Are the answer's claims backed by its sources? Includes faithfulness, contradictions, figures and citations. |
| Guardrail | In an experiment: a limit the winning variant must not break, such as Safety failing on at most 1% of answers. |
| Hand-off contracts | The fields that must pass unchanged from one agent to the next, with their types and rules. |
| Hand-off fields unchanged | When one agent passes work to another, did the declared fields (for example onset date or leave type) arrive unchanged? |
| Hand-offs carry the facts | When one agent passes work to another, did the fields of its hand-off contract arrive unchanged? The check's name in the Checks tab. |
| Hand-offs | Whether the fields passed between agents arrived unchanged. |
| Healthy | Every check that applies passed. |
| History | Every time this answer was scored: on arrival and by each re-score. The most recent one is what the product shows. |
| How answers are judged | The default scoring rule's answer type: every factual claim must be supported (Answers from sources) or the answer is an analysis (Document analysis). |
| How it is scored | How the scorer measures this dimension. |
| Human review | People check answers the scorer judged, record their own verdict, and see how often they agree with the scorer. |
| In force | The one version scoring new answers for the whole project. |
| Integration & keys | How to send answers to variA/Bly, and the API keys to do it. |
| Issues | Answers that need attention, grouped by cause, with where the fix belongs: retrieval, a tool call, routing, a hand-off, the answer itself or setup. |
| Journal entries follow the posting rules | Do the journal entries use the accounts your posting rules allow for that kind of document? |
| Latency | How long your app took to produce the answer. |
| Leading | The variant most likely to be best so far, with its probability. |
| Live events | Every answer your AI produced, scored as it arrived. Filter, sort and open any row to see its scores, claims, checks and evidence. |
| Live experiment | The experiment currently splitting live traffic, and which variant is leading. |
| Live traffic split | How live traffic is divided between the variants. |
| LLM cost | What your app's own model call cost, from the tokens you reported. |
| Match preview | How many stored answers this rule would have scored over a past window, counted before you save. An estimate from history. |
| Matches the expected answer | When you send an expected answer, do the answer's amounts, numbers, percentages and dates match it? A separate verdict that never changes faithfulness. |
| Max-Accuracy | A reasoning model double-checks the claims the standard checks can't settle. More accurate on hard cases; verdicts can occasionally differ between identical runs on close calls. |
| Max-Accuracy share | The share of matching answers whose hardest claims go to the reasoning model. |
| Missing qualifier | The figure exists in the sources, but only under a condition the answer left out, for example “in-network” when out-of-network is different. |
| Mode | Deterministic only, or Max-Accuracy for a share of the matching answers. |
| Needs attention | At least one check failed. Open the answer to see which, and why. |
| Needs attention below | A dimension bar for live answers: an answer below it needs attention. |
| No safety regression | The safety check's pass rate did not fall against the control. |
| Not applicable | This check doesn't apply to this answer, for example a SQL check on an answer with no query. |
| Not configured | A check needs a data definition you haven’t given. Never counted as a pass. |
| Not in sources | Nothing in the sources supports this claim. It may be invented, or the right source wasn't retrieved. Needs attention. |
| Not supported | Nothing in the sources supports this claim. Needs attention. |
| On arrival | In an answer’s History: the verdict given when the answer came in. |
| Organization | Usage and billing, all projects, and the policy audit log. |
| Overall | A weighted combination of the six categories, from 0 to 1. The weights are set in the evaluation policy. |
| Overview | A one-page summary of this project: how many answers need attention, top issues, the live experiment and recent policy changes. |
| P(best) | The probability that this variant is the best one on the overall score, given the answers so far. |
| Pass | The check ran and the answer met it. |
| Policy audit log | Every policy change in the current project, who made it and when. |
| Posting rules | For finance agents: which accounts a journal entry may debit or credit for each kind of document. |
| Project configuration | Project name, domain, environments, API keys and deleting the project. |
| Project name | A name for this project. |
| Project role override | A different role for one member in the current project only. |
| Projected invoice | The expected invoice for this month at the current rate. |
| Quality | Readability, tone, clarity and following instructions. |
| Queue rule | A rule that sends a sample of newly scored answers to human review. |
| Queue rules | Rules that send a sample of newly scored answers to human review, for example 10% of production answers failing grounding. |
| Re-score | Score stored answers again with a published policy version. A new verdict is added; the earlier ones are kept. |
| Recent policy changes | The latest versions of the evaluation policy. |
| Release gate | Passes when every item has a scored answer, the pass rate meets the minimum and regressions against a baseline run stay within the limit. |
| Release minimum | A dimension bar for release runs: a run whose mean is below it fails the release gate. |
| Relied on | The sources (or tool calls, or computed values) a passing check relied on. |
| Requirement coverage | Does the answer include everything a template requires, for example the network tier and plan year next to any amount? |
| Requirement templates | Lists of what an answer must include, for example the amount, network tier and plan year. |
| Right document version | Was the answer built from the version of a document that applies, such as the right plan year, state or effective date, and not an outdated or other-region copy? |
| Run | One pass of your app over a dataset, logged with run= on each answer. The release gate judges a run. |
| Running | Splitting live traffic now. |
| Runs & comparison | Compare tagged groups of answers, variants or time windows item by item: what improved, what broke and what stayed the same. |
| Runs & release gate | Runs of a dataset logged from your code, compared item by item, and the gate a CI job asks before shipping. |
| Safety | Freedom from harmful content, bias, personal data leaks and prompt injection, from 0 to 1. Higher is safer. |
| Sample % | The share of a scoring rule’s matching answers that are scored. Answers outside it are stored, not scored. |
| Sample reached | How much of the required sample has been collected. |
| Sample size | Whether enough answers have been scored to trust the result. |
| Scoring rule | The policy rule that scored this answer. |
| Scoring rules | How each answer is scored: conditions on environment, agent, tag or metadata, the answer type and the Max-Accuracy share. The first matching rule wins; the last scores everything else. |
| Semantic | Does the answer address the question, completely and specifically? |
| Sender had | In a hand-off: the value the sending agent had in its own inputs (its question, documents, tool results, or what it received in an earlier hand-off). |
| Sent | In a hand-off: the value the sending agent put in the hand-off to the next agent. |
| Session | The conversation or workflow this answer belongs to. |
| Sessions | Conversations or workflows with several steps or agents. Shows the path between agents and whether information passed between them arrived unchanged. |
| Ship readiness | Whether the leading variant is safe to roll out: it is the statistical winner, nothing got less safe, no check regressed, and the sample is large enough. |
| SQL checks | Your database schema (names and types only) and rules for the queries your agents write. |
| SQL follows the rules | Does the SQL query your agent ran follow your schema and rules, for example “always filter on plan year” or “never select SSN”? variA/Bly checks the query text; it never connects to your database. |
| Statistical winner | One variant has at least the required probability of being best (usually 95%). |
| Steps | The number of steps in the session. |
| Suggested next variant | A change suggested from the weaknesses seen in this experiment's answers. |
| Superseded | An earlier version, kept for history. Answers scored with it keep its verdicts. |
| Supported | The sources back this claim. |
| Top issues | The most frequent causes of answers needing attention. |
| Turn | In a conversation: one customer message and the agents’ work up to the reply the customer receives. |
| Usage & billing | Evaluations used this month against your plan. |
| Variant | The experiment variant (prompt or model version) that produced this answer. |
| Versions & audit | Every policy version, who changed what and when, a comparison of any two versions, and putting an earlier version back in force. |
| Warn | Something worth a look, but not proven wrong. |
| Weakest step | The lowest overall score of any step in the session. |
| What does your AI do? | The first of three questions when you create a project (with One agent or several? and What is it built with?). Tick every use case that applies: your choices set how answers are judged (the answer type) and what the Connect step asks your app to send. |
| Where to correct | Where the fix belongs: retrieval (the right source wasn’t found), a tool call, routing, a hand-off between agents, the answer itself, or setup. |
| Whole conversation | In an answer’s History: a verdict added because a rule over the whole conversation found this turn broke it. |
| Why it isn't counted | Why this sentence is shown but not scored. |
| Your data | Optional definitions of your data (code lists, document scope, contracts, SQL schema) so checks can compare exact values. |
| Δ vs control (95% CI) | How much better or worse than the control this variant scores, with the range the true difference is 95% likely to fall in. |