Build a retrieval benchmark before publishing
Specify the intended outcome
For each named case, write the question, expected source IDs, forbidden source IDs and abstention expectation. Use identifiers from Knowledge. A positive case needs an expected source; an abstention case cannot have one. Include customer wording, ambiguous requests and unrelated topics. A small suite with clear reasons is easier to maintain than many unexplained keyword checks.
Read recall and ranking separately
Expected-source recall is the fraction of listed expected IDs retrieved. Mean reciprocal rank, or MRR, measures how early the first expected source occurs among distinct results. Rank one contributes 1 and rank two 0.5. A query can retrieve one expected document first while missing another, so a high MRR does not imply complete recall.
Understand the publication gate
A case passes when abstention matches, no forbidden source appears and a positive case has at least one expected hit. Publishing requires a positive case, all cases passing and the configured minimum MRR. Complete recall of every expected ID is not required. If each document matters, inspect recall or write separate cases. These assertions do not grade semantic answer correctness.
Treat failures as feedback
A failed case may reveal a weak title, duplicate procedure, missing synonym or incomplete source. Inspect the debugger before increasing weights. After configuration, corpus or suite changes, rerun the benchmark and obtain a new signed publication preview. Comparisons are meaningful only for unchanged cases in the same profile. Document why any assertion changes rather than quietly making a failing suite easier.
Product and guides
Explore creekX Agents · Quick start · Documentation. Written for v0.10 early access, 23 October 2026.
Frequently asked questions
Where should I start?
Read the separate Agents quick start and evaluate v0.10 in a private local workspace before a customer pilot.