All docs
EvaluationCompare versionsKeep a set of examplesdashboard

Keep a set of examples

Collect real prompts and the answers you would want, so prompt generation is shaped by your domain.

What you'll have

A named set of your own examples that the prompt optimiser uses as golden examples when it writes a variant.

You'll need
  • A project
  • A handful of real prompts and the answers you would have wanted

A dataset is a named set of prompt-and-answer examples from your own domain, kept in the project so it can be used somewhere other than your head.

What datasets are for here

Today a dataset feeds prompt generation. When you ask for an optimised variant of a prompt, the examples you have collected are given to the generator as golden examples, so the prompt it writes is shaped by how your domain actually reads rather than by a generic idea of a good answer.

That is the whole current use. Scoring runs on your live traffic, not on a stored set, so a dataset is not a test set and there is no run-the-dataset button — worth knowing before you build one and wait for a score.

Create one

Datasets in the sidebar → New dataset. Give it a name that says what the examples have in common, then add samples: a prompt, the answer you would want, and any context that answer depends on.

What makes a set worth keeping

  • Real examples, not invented ones. Take them out of production traffic. Invented examples encode what you imagine your users ask.
  • The hard ones. Examples where a plausible answer is wrong are worth ten where any answer is fine.
  • Enough to disagree with each other. A set where every example wants the same thing produces a prompt that only knows how to do that thing.

What is fixed once written

Samples can be added and the whole dataset can be deleted; individual rows cannot currently be edited or removed. Get an example right before you add it, or rebuild the set — it is a small enough thing to redo.