Build an AI Agent That Investigates with Tools and Test It So It Can't Just Guess

## Keep the Investigation Engine Separate from the Plugins for Each Service Organize the system into explicit layers: user-facing entry points, configuration loading, an investigation engine, a tool manager, and pluggable sources, toolsets, prompts, and destinations. Entry points should route requests and establish logging; configuration should validate settings and construct integrations; the engine should select tools, gather evidence, analyze results, and optionally return findings to an authorized destination. Keep provider-specific behavior behind plugins so the core loop can reason over a stable tool contract. Load integrations dynamically from validated configuration, and keep source ingestion, investigation, and destination writes distinct so read-only diagnosis does not silently become an external mutation. ## Make Each Tool a Thin Wrapper That Explains Failures Exactly Prefer a thin wrapper around one external endpoint or operation. Give each integration a small validated configuration object, an explicit prerequisite or health check, and a tool function whose parameters correspond directly to the underlying request. Avoid heavy client abstractions when a transparent transport call makes behavior and errors easier to inspect. Return enough failure context for the caller to correct itself: the exact query or command, time range, filters, relevant parameters, response status, and underlying error message. A no-data result must state what was searched and where. Preserve access controls, validate executable commands, keep tools read-only by default, and never expose or commit secrets. ## Limit How Much Data a Tool Returns Before the AI Reads It Never expose an unbounded collection through a tool. Require server-side filters for endpoints that can return large result sets, and design the schema so callers cannot omit the most important scope parameter. Token or memory overflow caused by an unrestricted list operation is a tool-contract defect, not a prompt problem. When the remote service cannot filter adequately, add explicit client-side controls such as maximum depth, result limits, field selection, or a structured query expression. Report truncation and the applied bounds. Keep the default response compact while allowing targeted expansion when the investigation needs more evidence. ## Keep Old Config Names Working and Put Shared Behavior at the Right Class Level When renaming a configuration field, accept deprecated names as extra input, map them to the current field, and emit a deprecation warning. Keep only current names in the public schema and serialized output so compatibility does not pollute generated configuration or downstream logic. Before adding a field, method, timeout, or retry policy to a named integration, inspect its class hierarchy and sibling implementations. Place shared behavior at the most general level that truly needs it. Use a proven retry facility with explicit attempt and delay policy; write a custom loop only when the standard mechanism cannot express the requirement, and document that reason. ## Test in Three Layers, from Unit Tests to Full Agent Evaluations Use three complementary layers. Unit tests isolate local behavior, integration tests exercise tool and service boundaries, and agent evaluations verify end-to-end investigation behavior with realistic prompts and fixtures. New local behavior needs unit coverage, new integrations need boundary coverage, and complex investigations need evaluation cases. Keep source structure and test structure aligned. Use transport-level HTTP mocks when testing request behavior so headers, URLs, parameters, and adapters are exercised instead of replacing a high-level function. Run live tools only in an intentionally configured environment, and distinguish ordinary deterministic tests from evaluations that require model credentials or external services. ## Write Evaluations That Catch Made-Up Answers (Hallucinations) Make each prompt request the exact facts the evaluation checks. Prefer expected values that can only be discovered by querying, such as a unique injected identifier, exact count, title, or error code. The expected answer must remain invisible to the agent. If the answer is too generic to prove discovery, require evidence that the correct tool was called. Test discovery rather than recognition. Use neutral resource names, realistic scenarios, and prompts that do not reveal the implementation or likely solution. Avoid generic assertions that a model could guess from domain knowledge. Use only registered test tags, unique resource names, pinned dependency or image versions, and explicit failure exits when setup cannot prove the service is functional. ## Let Several Test Runs Share Cloud Resources Without Breaking Each Other Assume cloud-backed evaluations from multiple branches can overlap against the same account. Setup must be idempotent: create or reuse resources without failing when they already exist. Cleanup must never delete a static shared resource that another run may still use; omit cleanup or limit it to a unique run-scoped identifier. Verify functionality through a real API response rather than readiness alone. Poll with bounded retries to handle creation races, fail early when the expected content never appears, and capture diagnostics on timeout. Reuse standardized shared infrastructure when multiple tests need the same service, but keep each test’s data and identifiers isolated. ## Trace Terminal Display Bugs and Evaluation Drops Back to Real Run Records For interactive terminal defects, reproduce through a pseudo-terminal and capture raw escape sequences. Compare cursor movement and erase operations with the rendered frame height, then test the renderer through a forced-terminal in-memory stream. Fix the frame or refresh contract that creates drift; do not patch cursor movement blindly when behavior differs at the bottom of the terminal. For agent-evaluation regressions, compare one baseline trace and one current trace for the same test and model before relying on source diffs or aggregate statistics. Inspect rendered prompts, tool-call order, per-call metrics, and returned evidence. A template may grow in source while shrinking at runtime, and aggregate variance can hide a deterministic per-run change. Read the full change set of a suspected revision before reverting only the most visible file. ## Format and Check Only Files in Scope and Never Push Without Permission Follow repository-specific style, typing, test, and version-control policy, but do not run broad auto-fix, formatting, or repository-wide checks when they are known to modify unrelated files unless the user or governing workflow explicitly requests them. Limit automatic formatting and linting to files in scope, and keep reviews focused with the corresponding tests. Place imports and shared utilities according to the project convention, preserve reversible history, and avoid forceful history rewriting unless explicitly authorized. A local task does not itself authorize committing, pushing, publishing, or changing external state. Report which verification ran and which checks were intentionally excluded. ## Update Every Reference When a Doc Page or Heading Changes Use repository documentation as the primary source for project behavior before consulting the public website. When a page moves, a filename changes, or a heading changes its anchor, search the entire repository for the old path and anchor, including documentation, source strings, top-level guides, and comments, then update every affected reference. Update the documentation system’s real navigation source rather than a similarly named but inactive configuration. Preserve renderer-specific structure, including required blank lines and valid heading hierarchy. Avoid stale capability lists and repetitive security filler; prefer concise examples and consolidated troubleshooting blocks. When an integration is added, update its overview, index, dedicated page, navigation entry, and visual asset together.