Change a Distributed Job Platform Monorepo and Test It With Real Services

## Run Workspace Commands From the Root and Pick Checks by Package Type Run workspace commands from the repository root through the pinned package-manager wrapper. Build required dependencies before starting an application or command-line tool, and use workspace filters to limit work to the affected package graph. Follow the repository's dependency-installation policy exactly. If it requires editing package declarations before installation, do that instead of invoking an add command that bypasses reviewable version choices. Choose verification by package role: - Applications and private packages use static type checking when their build does not prove correctness. - Published packages use their real package build so exports and generated artifacts are exercised. - Major changes, new files, schema work, and broad refactors receive local type or build verification. - Tiny edits may rely on focused checks and continuous integration when the project explicitly permits it. Do not run a broad build merely because it is familiar. Use the check that can actually fail for the changed layer. ## Test Queues and Databases With Real Service Containers, Not Mocks Colocate tests with source and use the repository's single supported test runner. Start with the affected package or one test file, building its dependencies first only when the workspace requires it. Do not replace databases, queues, or caches with mocks when the project provides disposable service containers. Use isolated test helpers that supply real connection options and clean state for: - Queue and cache behavior. - Relational transactions and migrations. - Combined workflows that cross storage boundaries. Reserve a lightweight unit boundary for pure functions. For infrastructure behavior, prefer a controlled real service so serialization, timing, retries, and transaction semantics remain visible. Run formatting and automatic lint fixes before the final read-only lint check. Prefer static imports; use dynamic loading only for an unavoidable cycle, measured code-splitting benefit, or genuine runtime condition. ## Record Published Package Changes and Keep Shared Dependencies on One Version Add a versioned change record whenever a published package or integration changes. Default to a patch release for fixes and small compatible changes, confirm a minor release with maintainers, and never select a major release without explicit approval. For server-only behavior with no published-package change, use the server change-note mechanism instead of a package version record. Write release descriptions for users: - Lead with the behavior that changed. - Use one plain sentence when possible. - Omit internal tool, queue, storage, and deployment names. - Do not describe the implementation as the benefit. Keep intentionally shared dependencies on one exact version across the workspace when runtime type identity depends on it. Before adding such a dependency, find the existing pin and match it exactly; ranges or parallel versions can make schemas from one package incompatible with consumers in another. ## Trace the Job Request Path Before Moving Work Between Layers Understand the request path before editing it: 1. An external request enters the API and dashboard application. 2. Services validate intent and call the run-lifecycle engine. 3. The engine persists state and enqueues work. 4. A supervisor claims work and manages isolated execution. 5. Results return through the lifecycle engine. 6. Transactional state and analytical events flow to their separate stores. Keep public SDKs, command-line tooling, shared types, build extensions, client hooks, queue workers, databases, analytics, scheduling, and execution engines in their declared packages. Import public shared types through supported subpaths rather than broad package roots when the export contract requires it. Do not move responsibilities across these layers as a convenience. A change to scheduling, queuing, execution, or persistence must account for the full state transition and failure path. ## Keep the Removed Execution Engine Gone and Reject Old Clients Cleanly When an old engine has been removed, historical directory names or compatibility enums do not prove that it still runs. Treat the current architecture documentation and active package graph as authoritative. Never reintroduce the retired path to satisfy an old branch or request. Compatibility handling may recognize an obsolete version only to reject it or finalize it gracefully with a client error; it must not route work into nonexistent execution code or turn an expected rejection into a server failure. Before changing compatibility behavior: - Trace the active engine selection. - Confirm how obsolete clients are identified. - Preserve clean error semantics. - Add tests for both the current path and retired-version rejection. Keep versioned SDK guidance and generated agent rules in their dedicated maintenance pass. Do not update distributed instruction bundles unless the task explicitly includes them. ## Write Tasks With the Current Public SDK and Update Docs When It Changes Define durable tasks through the current public SDK entry point. Do not import obsolete version subpaths or revive a deprecated job-definition API. A task definition should have: - A stable identifier. - A typed input contract. - One asynchronous run function. - Explicit errors for invalid input. - Idempotent behavior or a documented retry boundary for side effects. Treat code bundled into customer execution images as public runtime code even when it originates in the command-line package. Verify package exports, bundled dependencies, and container behavior together. Update public documentation and examples when the SDK contract changes. Keep product documentation in its designated site and follow that site's local formatting and navigation rules. ## Run a Real Task End to End on a Local Build Before Saying It Works Use a maintained reference project to test SDK and platform changes against a local build. Link the reference project to the workspace packages rather than testing a released package by mistake. Run the flow in order: 1. Start required storage and queue services. 2. Apply database migrations and seed the reference organization and project. 3. Build the SDK, command-line tool, and affected packages. 4. Start the API application in the background and verify its health endpoint. 5. Start the local task worker from the linked reference project. 6. Wait for an explicit worker-ready signal. 7. Trigger a representative task through the supported control interface. 8. Poll or list runs until the expected terminal state appears. 9. Inspect the returned result and persisted run state. Do not report end-to-end success from compilation alone. Preserve command output or run identifiers as evidence, and stop background processes when verification finishes. ## Pin Multi-Platform Container Images and Strip Debug Markers Before Merge Pin container images to multi-platform index digests, not architecture-specific image digests. Select the digest from the image's platform-neutral release page so development and continuous integration resolve the correct architecture. Use structured debug crumbs while developing distributed flows: - Add markers at state transitions, queue boundaries, and external calls. - Use only registered application namespaces; do not invent a new namespace without approval. - Clear existing traces before reproducing a bug. - Query with limits and cursor pagination. - Keep temporary markers on the development branch while the investigation is active. - Strip markers and imports before merge. Reviewers should not flag recognized temporary instrumentation merely because it exists on the branch. They should verify that it cannot reach production and that the strip step is part of completion. ## Run Only the Checks That Fit the Change and Report What Was Skipped Before handoff, inspect the full changed-file set and run only the applicable gates: - Formatter, lint fix, and read-only lint. - Type checking for applications and private packages. - Real builds for published packages. - Focused tests with disposable service containers. - A package change record or server note at the correct boundary. - Exact shared dependency pins. - Current-engine and retired-version tests when routing changed. - A multi-platform image digest when image references changed. - A linked reference-project run when SDK or execution behavior changed. - Removal of temporary debug instrumentation before merge. Report what ran, what passed, and any gate intentionally omitted with its reason. Do not claim runtime, package, or container correctness from a weaker check that cannot observe it.