Test and Fairly Compare Search Systems That Give AI Agents Memory

## Claim Only What the Test Results Actually Show Distinguish whole-system claims from feature claims. A statement that an entire retrieval system is best requires evidence across retrieval quality, long-context memory, calibration, reliability, cost, and operational behavior. No single feature or benchmark can establish it. When evaluating a feature, ask whether that feature delivers meaningful value to users of the system. Do not spend the evaluation proving that one proprietary algorithm beats another when the product decision depends on user outcomes. Before running an experiment, write the claim, unit of analysis, expected effect, decision threshold, and conditions under which the result would not generalize. Reject conclusions whose scope is broader than the tested population, workload, engine, or configuration. ## Choose the Database and the Source Inside It Separately Model knowledge routing on two independent axes: - the corpus or database that owns storage and access policy; - the source or collection inside that corpus. Resolve both axes through an explicit precedence chain such as per-call input, authenticated allowance, session context, repository marker, user configuration, then default. Record the resolved values with each query so silent misrouting can be diagnosed. Treat remote or agent-facing callers as untrusted unless trust is explicitly established. Apply source scope to every read path through one shared helper; do not hand-build filters in individual queries. Validate identifiers before using them in joins or key construction, and define uniqueness on the full source-scoped key. ## Compare Search Modes with All Their Settings Together Define named modes as coherent bundles of cache policy, token budget, query expansion, relational retrieval, intent weighting, and result limit. Resolve configuration through one deterministic chain, with per-call overrides above deployment configuration and a documented fallback. Compare modes as complete user-facing choices rather than cherry-picking one favorable knob. Report the active value and its source for every knob. Apply mode resolution in the lowest shared search path so production queries and evaluation replays exercise the same behavior. When a new retrieval arm applies only to a recognized query type, make it deterministic and a no-op elsewhere. Provide a per-query override for controlled experiments without changing the system-wide default. ## Estimate Cost at Real Levels of Use Calculate downstream cost from query volume, delivered context tokens, model input price, expansion calls, and cache-hit rate. Show at least light, typical, and heavy usage levels so readers can scale the result without extrapolating from one query. Separate isolated per-query cost from real agent-loop spend. Prompt caching, retrieval-cache hits, prefix churn, and repeated tool calls can materially change the realized total. State which savings are measured, assumed, or excluded. Evaluate mode and downstream model as a pair. Oversized context can overwhelm a low-capacity model, while an expensive model may be starved by an overly restrictive budget. Prefer natural pairings and call out mismatches as distinct failure modes rather than ranking cost alone. ## Decide What to Measure and How to Judge It Before Testing For every reported metric, publish a plain-language definition alongside the machine-readable output. Use one canonical glossary for terms such as precision at a cutoff, discounted gain, reciprocal rank, overlap, calibration, latency, and cost. Before collecting results, specify datasets, sample selection, expected direction, primary metrics, threats to validity, and exclusion rules. For paired comparisons, use a paired resampling method and correct for multiple comparisons when testing several metrics or variants. Preserve both human-readable summaries and structured metadata. Do not add ad hoc glossary fields beside individual values; attach one consistent metadata block per response so downstream tools can parse it reliably. ## Make Question-Routing Tests Easy for Anyone to Repeat Separate training fixtures from held-out fixtures. Score with both strict and explicitly documented lenient criteria when formatting variation can obscure semantically correct routing, but do not change criteria after seeing held-out results. For each cross-model run, store a receipt that binds the model identifier, prompt-template hash, fixture hash, harness revision, timestamp, and scoring version. Keep raw predictions so scoring logic can be improved or audited without paying to rerun the model. Use ablations to identify load-bearing signals. Remove one routing clause, hierarchy layer, or context field at a time, then compare against the same held-out set. A performance drop supports a causal design claim; an unpaired before-and-after anecdote does not. ## Keep Saved Search Results (Cache) from Mixing Across Settings Include every behavior-changing input in the cache key: source scope, mode knobs, result limit, expansion state, retrieval arms and depth, embedding provider, embedding column or model, and any feature flags that alter ranking. Version the cache-key contract. When semantics change, increment the version so old rows become unreachable through a normal miss rather than being interpreted under the new configuration. Test cross-mode, cross-source, cross-provider, and relational-on versus relational-off isolation. Run parity tests against every supported storage engine because an embedded engine may hide encoding or type behavior that appears only in a production database. ## Protect Private Data and Explain Security Fixes Without Revealing Weak Spots Store evaluation records in the project’s audit area, never in a user’s live private corpus. Use synthetic names and organizations in fixtures, examples, reports, release notes, and public discussions. Replace relationship-revealing queries with generic placeholders. Describe security fixes functionally in public artifacts. Explain the protective mechanism and operator action without publishing a curated list of sensitive records, exposure duration, or the easiest probe path for unpatched systems. Keep detailed exposure analysis in restricted review evidence. Before publishing, ask whether a reader with no prior context could infer private relationships or directly target an older deployment; if so, generalize the wording. ## Keep Research Documents Current and Short with Automatic Checks Use a layered documentation structure: - an always-loaded orientation containing the research goal, routing model, architecture, and cross-cutting invariants; - current-state reference documents loaded only for the relevant subsystem; - a changelog and version history for past behavior. Do not append release-by-release narration to current-state references. Update the existing entry to the new truth and keep historical detail in version control and the changelog. Enforce size limits, forbidden historical markers, generated glossary freshness, and documentation-bundle freshness in continuous integration. A prose rule without an automated guard is not a durable constraint. ## Save the Full Output and Settings of Every Run Write progress to a channel separate from data output. Keep phase names stable, make structured progress an explicit option, and ensure long-running operations emit heartbeats without corrupting machine-readable results. For tests and evaluations, capture complete output to a file before displaying a tail or filtered summary. Record the original command’s exit code before running any viewer. Never pipe a gate directly through a truncating command, because the pipeline can report the viewer’s success while discarding the actual failure details. Each run should retain configuration, resolved mode, dataset identity, hashes, metrics, cost, warnings, exit status, and timestamps. A result is decision-grade only when another reviewer can reconstruct what ran and distinguish measured facts from assumptions.