Add Backend Code and the Right Tests to a Multi-Service App

## Set Up the Environment and Confirm the Services Are Running - Backend dependencies live in a lockfile-managed virtual environment at the repository root. If it does not exist yet, create it from the frozen lockfile and activate it before running anything. - Credentials and keys needed by tests live in the repository's local environment file. Read them from there instead of inventing values or requesting new ones. - Assume all application services are already running, but verify rather than trust: check the log directory and confirm the relevant service is still emitting entries. - Reach the database through the running container using the project's documented client invocation, not through an improvised connection of your own. - When exercising the web application by browser automation, sign in with the account the end-to-end setup creates. If it does not exist, register it through the signup page, since the first registered account becomes the administrator. ## Keep Database Queries in One Layer and Send Test Calls Through the Frontend - Put every database operation inside the designated data-access directories. Do not run queries from anywhere else, so schema knowledge stays in one layer and can be reviewed in one place. - When calling the backend during manual or scripted testing, always go through the frontend origin rather than the backend port directly. A request that skips the frontend also skips the proxy, session, and authentication behaviour that real traffic passes through, so the result is not representative. ## Put New Work in the Matching Background Worker Pool Asynchronous work is split across specialized worker pools rather than one general queue. Place new work in the pool whose profile matches it. - A primary pool coordinates system-wide operations: resource deletion, synchronization, pruning, and periodic checks. - Fetch pools pull data from external sources and spawn a processing task per batch, with watchdog monitoring so a stalled source is detected. - Processing pools run the ingestion pipeline: persist records, split them into chunks, enrich, embed, write to the search index, and update metadata. - A light pool handles short operations at high concurrency. A heavy pool handles resource-intensive operations at low concurrency. - A monitoring pool, single-threaded because monitoring needs no parallelism, collects queue depth, memory, and health metrics. - A scheduler pool owns periodic triggers, from interval checks measured in seconds to hourly cleanups, and resolves the tenant for each scheduled task. Shared properties across pools: thread pools rather than processes for stability, per-tenant isolation, priority queues, heartbeat and liveness checking, automatic retry and failure recovery, a fast shared store for coordination between processes, and the relational database for durable task state. ## Give Every Background Task an Expiration and Its Own Timeout - Declare tasks with the framework's application-independent decorator rather than binding them to a concrete application instance, so importing a task never drags in application construction. - Keep task modules under the designated task directories for their edition or tier. - Never enqueue a task without an expiration. Supply one whether the task is sent from the schedule or from another task. Submitting code that enqueues without an expiration is not acceptable, because it lets the queue grow without bound. - Because tasks execute in thread pools, the framework's own time limit features are silently disabled. Implement timeout logic inside the task itself. - Workers do not reload on code change. After changing worker code, ask the maintainer to restart the worker before testing, and do not assume the new code is live. ## Add Endpoints and Raise Errors with the Project's Own Error Type - When adding an endpoint, type the handler function directly instead of declaring a separate response model field. - Raise the project's own error type instead of the framework's generic HTTP exception, and never hardcode status codes or import status constants from the framework. - A global handler converts the project error into one standard response body carrying an error code and a detail. This removes boilerplate and keeps error handling consistent across the backend. - Error codes come from a central registry. If a new category is needed, add it to the registry first; never invent a code at the call site. - When forwarding an upstream failure whose status comes from the upstream response, pass that status through the error type's status override argument rather than abandoning the error type. ## Keep Strict Types and Write Only Comments That Stay Useful - Install and run the repository's pre-commit hooks across all files before proposing a change. - Keep everything strictly typed on both the backend and the frontend. Untyped escape hatches are not an acceptable shortcut. - Keep comments brief and limited to information that stays relevant long-term. Do not write comments that narrate the change in progress, such as what was just added, removed, or refactored; they become noise the moment the change lands. ## Write Database Migrations by Hand for Default and Private Schemas - Generate the revision file with the migration tool, then write the migration body manually inside the file it creates. Do not rely on autogeneration to express intent. - Multi-tenant deployments need the migration applied against the private schema configuration in addition to the default one, and the same distinction applies when creating the revision. - Treat the migration as part of the change under review, not as an afterthought applied afterwards. ## Pick the One Kind of Test That Proves the Change, with Little Mocking Four levels exist. Prefer the higher-fidelity levels over heavy mocking, and pick the one that actually proves the behaviour. - Unit tests assume no service is reachable and mock every interaction with the outside world. Write them only for complex, isolated modules whose logic is worth pinning down on its own. - Dependency-backed unit tests assume external dependencies are running — database, cache, object storage, search index, outbound network — while the application containers are not. Functions are called directly and selected components may still be mocked. Use them to minimize mocking while keeping the ability to control flaky behaviour or to verify that an internal call happened with specific arguments, which a pure integration test cannot do. - Integration tests run against a real deployment and mock nothing. Prefer these, falling back to dependency-backed unit tests when mocking or internal verification is genuinely required. - Browser-driven tests add the web server on top of the full deployment. Use them for behaviour that needs substantial coordination between frontend and backend. Do not overtest. A given change usually needs one kind of test, not all four. ## Reuse Existing Fixtures and Helpers When Writing Integration Tests - Before writing an integration test, read the root test configuration for available fixtures and the shared utilities directory for helpers. - Prefer the manager helpers in those utilities over calling endpoints directly with a raw HTTP library, and prefer fixtures over calling the helpers by hand. Creating an administrator manually when an administrator fixture exists duplicates setup and drifts from the rest of the suite. - Tests are parallelized at the directory level, so files within a directory must not depend on shared mutable state. - When a test makes a real model call, use each provider's cheap and fast tier rather than a flagship model. Test cost scales with the suite, and what is being verified does not depend on model strength. - Copy the shape of an existing exemplary test in the same category rather than inventing a new structure. ## Read Each Service's Logs Before Guessing Why Something Failed - When writing integration tests or driving the running system by hand, read the per-service debug log files in the log directory. Every service tails its output there. - Read the logs before forming a theory about a failure. An unrelated cause, such as a service that is not running, looks identical to a real defect until the log is read. ## Keep API Keys Out of the Repository and Protect New Endpoints - Never commit API keys or secrets to the repository. - Store integration credentials through the encrypted credential storage, never in plain configuration. - Follow the established role-based access control patterns when adding a feature. A new endpoint inherits no protection by default. - Validate input with schema models at the boundary. - Use parameterized queries so that injection is impossible by construction, and never assemble a query by string concatenation. ## Tag Every AI Model Call with a Registered Operation Name for Tracing Every call to a language model, embedding model, reranker, image generator, speech component, or intent classifier must open a generation span tagged with a value from the central flow registry. Use the wrapper matching the call path: one for calls made through the model abstraction, another for direct provider or HTTP calls that bypass it. - Add a new registry value before instrumenting a new operation. Never pass a raw string as the tag. - Tags name the operation, such as reranking or image editing, not the provider. The provider belongs in the model configuration metadata. - The automatic fallback emits untagged sentinel values for calls that reach the abstraction without an explicit span. Those sentinels are visible in dashboards and mean instrumentation is missing. Fix the call site instead of relying on the fallback. ## Research the Code, Then Write a Plan Without Timelines or Code Research before writing. Explore the relevant parts of the codebase first; a plan written from assumptions is worse than no plan. A plan includes at least these elements: - Issues to address: what the change is meant to do. - Important notes: things found during research that matter for the implementation. - Implementation strategy: the high-level approach for making the change happen. - Tests: which tests will verify the correct behaviour, chosen deliberately and kept minimal. Do not include a timeline section, and do not include a treatment of rollback steps. Do not write code as part of the plan; keep it high level, though referring to specific files or functions is useful. This is a minimum list, so add more when it helps. ## Read a Sub-Project's Own Rules Before Working in It - Root guidance covers shared concerns. Each sub-project keeps its own standards document next to its code, and that document governs its own tree. - Read the scoped document before working in that area. Do not assume the rules of one client apply to another: a mobile client without a document object model, with styling primitives whose spacing values are absolute rather than a relative step scale, and with its own routing and view primitives, follows different rules from a web client. - Broader engineering practices live in the repository's contribution guide. Read it and follow it in addition to this document.