Build Async Rust Services That Survive Cancellation and Failure

## Finish the Real Code Instead of Mocks or Stubs Finish the requested behavior end to end. Do not substitute mocks, fallback implementations, stubs, or test-only shortcuts for missing production logic. - Use real dependencies and the actual integration path. - Do not manipulate tests to conceal a broken implementation. - Avoid environment workarounds that bypass required setup. - Record any unavoidable incomplete work explicitly in the resulting plan or specification. - Never claim that a feature works without evidence from the path users will exercise. Autonomous execution means completing the rigorous path, not reducing the definition of done. ## Apply the Same Design Choices to Every Related Data Type When a system handles several related signal or data families, design shared behavior for all of them even if one family is the current priority. - Identify semantics common to ingestion, storage, search, analytics, and lifecycle management. - Keep family-specific optimizations behind explicit boundaries. - Test that shared decisions do not privilege the first implemented family accidentally. - Preserve consistent error, configuration, and observability behavior across paths. - Document intentional differences rather than letting implementations drift. Prioritization may control delivery order, but it must not silently narrow the architecture. ## Put Safety First, Then Speed, Then Convenience Use this engineering priority: safety first, performance second, developer convenience third. Optimize only after invariants and failure behavior are understood. - Reject invalid input at public boundaries. - Define explicit limits as named constants and return structured errors when they are exceeded. - Do not mask dependency, storage, or network failure with a successful fallback. - Treat unexpected state as an error or violated invariant, not as a successful no-op. - Allow missing optional data only when the contract explicitly defines it as optional. Performance work must preserve correctness during failure, retry, cancellation, and partial progress. ## Use Debug Assertions for Internal Rules and Errors for Bad Input Use debug-only assertions to document internal invariants that reviewers can reason about. Use recoverable error results for invalid user input, external data, and runtime conditions because debug assertions may disappear in optimized builds. - Propagate library errors with `Result` and the `?` operator. - Avoid unconditional extraction methods in reusable library code. - Preserve meaningful error types instead of flattening every failure. - Use fallible filesystem and metadata operations when errors carry information. - Add context without discarding the original cause. - Never silently ignore an error with an unused assignment. An impossible state should fail visibly enough to support diagnosis. ## Keep Async Code Safe When Tasks Are Cancelled Assume that asynchronous work can be cancelled at any suspension point. Structure state transitions so cancellation cannot expose partially updated state or lose data. - Prefer actor-style ownership and message passing for cancellation-sensitive mutable state. - Do not hold a lock across an await point. - Use synchronous critical sections when mutation must be atomic. - Cancel cooperatively through a cancellation signal rather than aborting tasks arbitrarily. - In selection loops, pin and resume an in-flight future instead of recreating and dropping it. - Make shutdown and cleanup idempotent. Choose simpler cloning or shared ownership in actor and async code when it makes ownership and cancellation behavior safer. ## Keep Types and Ownership Clear, and Split Large Files Use established domain identifiers and type aliases instead of passing raw strings for new concepts. Let the type system distinguish values that should not be mixed. - Avoid shadowing names within a function when it obscures state transitions. - Return references when ownership remains local and cloning is unnecessary. - Use explicit shared ownership when values cross task boundaries. - Prefer procedural loops when iterator chains make fallible control flow hard to read. - Split oversized files and functions by coherent responsibility. - Keep public interfaces separated from implementation details. Readability is a reliability tool when code coordinates asynchronous state. ## Test Through the Real Service Interface Users Reach Do not declare a feature complete until it has been exercised through the real public stack. If users reach behavior through a network protocol, gateway, or service boundary, test through that boundary rather than calling an internal function directly. 1. Start the real service with representative configuration. 2. Send input through the supported ingestion or command interface. 3. Retrieve the result through the supported query or response interface. 4. Verify persistence, error mapping, and observable side effects. 5. Confirm that internal and public behavior agree. Internal tests are useful, but they do not replace proof through the production path. ## Choose Test Types by Purpose, Including Failure Tests Use fast unit tests for isolated logic, property-based tests for broad state spaces, failure-injection tests for resilience, and integration tests for complete service behavior. - Narrow property-test inputs enough to produce useful counterexamples. - Use real service fixtures when storage or protocol behavior matters. - Keep retries limited to suites that intentionally tolerate nondeterministic scheduling. - Run failure-point coverage when changes affect recovery or state transitions. - Add an integration test for each new public endpoint. - Test the error path as deliberately as the success path. Do not let repeated retries turn a deterministic defect into an apparent pass. ## Run Checks With the Repository's Pinned Toolchain and Commands Run builds, formatting, lints, tests, documentation checks, license checks, dependency audits, and spell checks through repository-defined commands. - Use the pinned toolchain and required formatter channel. - Start declared local services before integration suites and tear them down afterward. - Pass test configuration through documented environment variables without embedding real credentials. - Run the narrow affected package first, then the all-feature workspace suite. - Treat every lint warning as work to resolve or justify with the smallest targeted allowance. - Verify that each change compiles independently, not only as part of a later stacked change. Direct commands are acceptable only when they preserve the setup performed by the canonical workflow. ## Track Architecture Changes in Three Kinds of Records Track architectural change through three complementary records: - **characteristics:** proactive requirements the architecture must satisfy; - **gaps:** limitations discovered through incidents, testing, or production behavior; - **deviations:** intentional departures from the preferred architecture, including rationale and constraints. Link code changes to the relevant record when they alter system guarantees. Keep formal models, simulation workflows, decision records, and implementation status distinct so an aspirational design is not mistaken for completed behavior. Update the record when evidence changes the architecture, not only when a new feature is announced. ## Reply to Review Comments in Their Existing Threads Make changes easy to review by keeping diffs focused and explanations attached to the decision they address. - Reply to a review comment in its existing thread instead of creating a separate comment at the same location. - Verify through the hosting interface that the reply is linked to the intended parent. - Explain why a non-obvious invariant, allowance, clone, or deviation is necessary. - Separate unrelated refactoring from functional changes. - Update living guidance when a repeated failure pattern reveals a durable rule. Review hygiene preserves context and prevents the same issue from being resolved in parallel conversations. ## Pass Every Check in the Final Diff Before Reporting Done Before reporting completion, inspect the final diff and confirm: - all applicable lints pass without unexplained warnings; - formatting checks pass for every changed source and test file; - no unconditional extraction, silent error suppression, or unjustified cloning remains; - production-interface and affected integration tests pass; - required headers, log conventions, documentation, and dependency audits pass; - new files and functions remain focused and maintainable; - no credential or sensitive value is present. Fix failures at their cause. Do not disable checks, weaken tests, or bypass the production path to obtain a green result.