All docs
HelpGlossarydashboard

Glossary

Every term variA/Bly uses, in plain words: the same definitions you see when you hover a label.

Every term variA/Bly uses, in plain words. Most are also shown when you hover over the label in the product. Terms are in alphabetical order.

TermMeaning
Advanced settings ▸Every setting: dimensions and weights, scoring rules, data definitions, versions and the policy as YAML.
AgentThe agent or component of your app that produced this answer.
Agent flowThe path through the agents, with the result of each hand-off.
Agent pathThe agents the conversation passed through, in order.
AgreementHow often reviewers and the scorer reach the same verdict, overall, per check and per claim.
AlertsRules that watch a metric of this project's answers over a time window and notify a webhook or email when it crosses a threshold.
All passedEvery check that applies to this answer passed.
Answer typeHow claims are judged. Answers from sources: every factual claim must be supported. Document analysis: the same, but introductions aren't counted and conclusions drawn from verified facts pass.
Answers from sourcesEvery factual claim in the answer must be supported by the documents or query results it was given. For Q&A, chat, RAG, text-to-SQL and agents.
Answers what was askedDoes the answer give the kind of thing the question asked for, for example an amount when asked “how much”?
API keyA secret your app sends with each answer to prove it belongs to your project. Created under Project configuration › Integration & keys; each key has the permissions you chose.
API keysKeys your apps and CI use to send answers or read results. Each has a scope.
Applies toWhich answers this rule covers, built from fields, operators and values.
Audit logEvery policy change in this project, who made it and when.
Breaks a business ruleShown on the sentence that states a value breaking one of your business rules, with the reason.
Business rulesYour own rules an answer must obey, such as a benefit limit or a deadline, checked in code. Their check is Follows business rules.
Change noteWhat changed in this version and why.
Change the evaluation policyThe API key permission needed to push or publish the policy from code, and for the SDK settings calls.
Chart of accountsThe ledger accounts your company books to. An entry to an account not on it is proven wrong.
ChecksYes/no checks that compare exact values: document version, hand-off fields, SQL rules, expected answers, posting rules and more.
Checks & weightsEvery dimension the scorer has, by category, with each category's weight in the overall score.
ClaimOne checkable statement in an answer, such as “the deductible is $1,500”. Each claim gets a verdict.
Code listsYour lists of valid codes and their spellings, for example leave types or states, so values match even when written differently.
Cohen's kappaAgreement corrected for the agreement two raters would reach by chance. 1 is perfect agreement, 0 is no better than chance. Shown only once there are enough pairs.
CoherenceIs the answer logically consistent, within itself and across the steps of a session?
Collection ruleA rule that copies live answers into a dataset after they are scored.
Collection rulesRules that copy live answers into a dataset after they are scored, for example 10% of production answers scoring below 70.
ContradictedA source says something different, for example a different figure or date. Needs attention.
Create a projectSet up a new project. Evaluation starts with the first answer you send; there is nothing else to configure first.
Data definitionsYour code lists, document scope, hand-off contracts, SQL schema and rules, posting rules, expected answers and requirement templates.
DatasetsFixed sets of questions, optionally with expected answers.
Datasets & runsOptional. Fixed sets of questions you send through your app for releases, and comparisons between runs (for example release 4.2 vs 4.3).
DecidedA winner was chosen.
Decision metricIn an experiment: the one metric, chosen in advance, that decides the winner.
Delete projectPermanently delete this project and everything in it.
DepthReasoning, use of evidence, and for agents: tool use, step order and efficiency.
DeterministicScored by the standard checks only. The same answer always gets the same score.
Dimensions at their barThe check that an answer’s dimensions are at or above the bars set in Checks & weights.
Document analysisFor reviews and reports. Facts must still be supported, but introductions aren't counted, conclusions drawn from verified facts pass, and a claim naming a document is checked against that document.
Document scopeWhat makes a document apply: dates, plan or academic years, regions, products, versions. Lets variA/Bly catch answers built from the wrong version.
DomainThe project's subject area. It picks the grounding models.
DraftA version being edited. It scores nothing until you publish it.
End a sessionTelling variA/Bly a conversation is over (client.end_session), so the checks over the whole conversation run.
EnvironmentThe label the answer was logged with (environment= in the SDK), for example production. Empty when none was sent. It is never a policy scope.
Evaluation policyThe project's rules for scoring answers: the answer type per scoring rule, Max-Accuracy, category weights and the data the checks compare against. One version is in force for the whole project; every change is a new version.
EventOne answer your app sent: the question, the answer and everything sent with it.
Expected answersReference answers you send with an event, compared by value.
Experiment variantThe experiment variant (prompt or model version) that produced this answer, if it was part of an experiment.
ExperimentsSplit live traffic between prompt or model versions and compare each with the control on the scores and checks of real users' answers.
FailThe check ran and found a problem. Open the answer to see the evidence.
FaithfulnessHow well the answer's claims are supported by its sources, from 0 to 1.
FieldA field declared in the hand-off contract.
Follows business rulesDoes the answer obey your business rules? A broken rule is confirmed by the reasoning model before it counts; it never changes grounding.
Follows from verified findingsA conclusion worked out from claims that were themselves verified, for example a difference between two checked figures. Document analysis only.
Framing (not counted)An introduction or connecting sentence with nothing to check, like “Here is my review”. Shown, but not counted. Document analysis only.
GroundingAre the answer's claims backed by its sources? Includes faithfulness, contradictions, figures and citations.
GuardrailIn an experiment: a limit the winning variant must not break, such as Safety failing on at most 1% of answers.
Hand-off contractsThe fields that must pass unchanged from one agent to the next, with their types and rules.
Hand-off fields unchangedWhen one agent passes work to another, did the declared fields (for example onset date or leave type) arrive unchanged?
Hand-offs carry the factsWhen one agent passes work to another, did the fields of its hand-off contract arrive unchanged? The check's name in the Checks tab.
Hand-offsWhether the fields passed between agents arrived unchanged.
HealthyEvery check that applies passed.
HistoryEvery time this answer was scored: on arrival and by each re-score. The most recent one is what the product shows.
How answers are judgedThe default scoring rule's answer type: every factual claim must be supported (Answers from sources) or the answer is an analysis (Document analysis).
How it is scoredHow the scorer measures this dimension.
Human reviewPeople check answers the scorer judged, record their own verdict, and see how often they agree with the scorer.
In forceThe one version scoring new answers for the whole project.
Integration & keysHow to send answers to variA/Bly, and the API keys to do it.
IssuesAnswers that need attention, grouped by cause, with where the fix belongs: retrieval, a tool call, routing, a hand-off, the answer itself or setup.
Journal entries follow the posting rulesDo the journal entries use the accounts your posting rules allow for that kind of document?
LatencyHow long your app took to produce the answer.
LeadingThe variant most likely to be best so far, with its probability.
Live eventsEvery answer your AI produced, scored as it arrived. Filter, sort and open any row to see its scores, claims, checks and evidence.
Live experimentThe experiment currently splitting live traffic, and which variant is leading.
Live traffic splitHow live traffic is divided between the variants.
LLM costWhat your app's own model call cost, from the tokens you reported.
Match previewHow many stored answers this rule would have scored over a past window, counted before you save. An estimate from history.
Matches the expected answerWhen you send an expected answer, do the answer's amounts, numbers, percentages and dates match it? A separate verdict that never changes faithfulness.
Max-AccuracyA reasoning model double-checks the claims the standard checks can't settle. More accurate on hard cases; verdicts can occasionally differ between identical runs on close calls.
Max-Accuracy shareThe share of matching answers whose hardest claims go to the reasoning model.
Missing qualifierThe figure exists in the sources, but only under a condition the answer left out, for example “in-network” when out-of-network is different.
ModeDeterministic only, or Max-Accuracy for a share of the matching answers.
Needs attentionAt least one check failed. Open the answer to see which, and why.
Needs attention belowA dimension bar for live answers: an answer below it needs attention.
No safety regressionThe safety check's pass rate did not fall against the control.
Not applicableThis check doesn't apply to this answer, for example a SQL check on an answer with no query.
Not configuredA check needs a data definition you haven’t given. Never counted as a pass.
Not in sourcesNothing in the sources supports this claim. It may be invented, or the right source wasn't retrieved. Needs attention.
Not supportedNothing in the sources supports this claim. Needs attention.
On arrivalIn an answer’s History: the verdict given when the answer came in.
OrganizationUsage and billing, all projects, and the policy audit log.
OverallA weighted combination of the six categories, from 0 to 1. The weights are set in the evaluation policy.
OverviewA one-page summary of this project: how many answers need attention, top issues, the live experiment and recent policy changes.
P(best)The probability that this variant is the best one on the overall score, given the answers so far.
PassThe check ran and the answer met it.
Policy audit logEvery policy change in the current project, who made it and when.
Posting rulesFor finance agents: which accounts a journal entry may debit or credit for each kind of document.
Project configurationProject name, domain, environments, API keys and deleting the project.
Project nameA name for this project.
Project role overrideA different role for one member in the current project only.
Projected invoiceThe expected invoice for this month at the current rate.
QualityReadability, tone, clarity and following instructions.
Queue ruleA rule that sends a sample of newly scored answers to human review.
Queue rulesRules that send a sample of newly scored answers to human review, for example 10% of production answers failing grounding.
Re-scoreScore stored answers again with a published policy version. A new verdict is added; the earlier ones are kept.
Recent policy changesThe latest versions of the evaluation policy.
Release gatePasses when every item has a scored answer, the pass rate meets the minimum and regressions against a baseline run stay within the limit.
Release minimumA dimension bar for release runs: a run whose mean is below it fails the release gate.
Relied onThe sources (or tool calls, or computed values) a passing check relied on.
Requirement coverageDoes the answer include everything a template requires, for example the network tier and plan year next to any amount?
Requirement templatesLists of what an answer must include, for example the amount, network tier and plan year.
Right document versionWas the answer built from the version of a document that applies, such as the right plan year, state or effective date, and not an outdated or other-region copy?
RunOne pass of your app over a dataset, logged with run= on each answer. The release gate judges a run.
RunningSplitting live traffic now.
Runs & comparisonCompare tagged groups of answers, variants or time windows item by item: what improved, what broke and what stayed the same.
Runs & release gateRuns of a dataset logged from your code, compared item by item, and the gate a CI job asks before shipping.
SafetyFreedom from harmful content, bias, personal data leaks and prompt injection, from 0 to 1. Higher is safer.
Sample %The share of a scoring rule’s matching answers that are scored. Answers outside it are stored, not scored.
Sample reachedHow much of the required sample has been collected.
Sample sizeWhether enough answers have been scored to trust the result.
Scoring ruleThe policy rule that scored this answer.
Scoring rulesHow each answer is scored: conditions on environment, agent, tag or metadata, the answer type and the Max-Accuracy share. The first matching rule wins; the last scores everything else.
SemanticDoes the answer address the question, completely and specifically?
Sender hadIn a hand-off: the value the sending agent had in its own inputs (its question, documents, tool results, or what it received in an earlier hand-off).
SentIn a hand-off: the value the sending agent put in the hand-off to the next agent.
SessionThe conversation or workflow this answer belongs to.
SessionsConversations or workflows with several steps or agents. Shows the path between agents and whether information passed between them arrived unchanged.
Ship readinessWhether the leading variant is safe to roll out: it is the statistical winner, nothing got less safe, no check regressed, and the sample is large enough.
SQL checksYour database schema (names and types only) and rules for the queries your agents write.
SQL follows the rulesDoes the SQL query your agent ran follow your schema and rules, for example “always filter on plan year” or “never select SSN”? variA/Bly checks the query text; it never connects to your database.
Statistical winnerOne variant has at least the required probability of being best (usually 95%).
StepsThe number of steps in the session.
Suggested next variantA change suggested from the weaknesses seen in this experiment's answers.
SupersededAn earlier version, kept for history. Answers scored with it keep its verdicts.
SupportedThe sources back this claim.
Top issuesThe most frequent causes of answers needing attention.
TurnIn a conversation: one customer message and the agents’ work up to the reply the customer receives.
Usage & billingEvaluations used this month against your plan.
VariantThe experiment variant (prompt or model version) that produced this answer.
Versions & auditEvery policy version, who changed what and when, a comparison of any two versions, and putting an earlier version back in force.
WarnSomething worth a look, but not proven wrong.
Weakest stepThe lowest overall score of any step in the session.
What does your AI do?The first of three questions when you create a project (with One agent or several? and What is it built with?). Tick every use case that applies: your choices set how answers are judged (the answer type) and what the Connect step asks your app to send.
Where to correctWhere the fix belongs: retrieval (the right source wasn’t found), a tool call, routing, a hand-off between agents, the answer itself, or setup.
Whole conversationIn an answer’s History: a verdict added because a rule over the whole conversation found this turn broke it.
Why it isn't countedWhy this sentence is shown but not scored.
Your dataOptional definitions of your data (code lists, document scope, contracts, SQL schema) so checks can compare exact values.
Δ vs control (95% CI)How much better or worse than the control this variant scores, with the range the true difference is 95% likely to fall in.