Edit a Plugin-Based Data Pipeline That Runs on Two Runtimes Without Breaking It

## Keep Versions in One File and Build From the Same Root Keep versions and toolchain coordinates canonical, and always build from the same place. - Store every version number and toolchain coordinate in one canonical file; all consumers (build scripts, helpers, tests, package descriptors) read from it, and you update only that file. - When a value is intentionally hardcoded or duplicated in several places, such as a runtime-compatibility path segment, do not treat it as a bug or compute it dynamically; find the current value in the canonical spot and match it. - Run all build and test commands from one consistent root so relative wiring resolves identically every time. - Prefer a vendored, pinned toolchain provisioned by a bootstrap step over a globally installed language or version manager, so every environment builds the same way. ## Point Test Filters at the Task That Actually Owns the Tests Understand how the test tasks are wired before running a filtered subset. - An umbrella test task often excludes every class itself and only delegates to sub-tasks, so a name filter applied to it silently matches nothing and looks like it passed. Apply filters to the concrete task that actually owns the tests. - Suites are split on purpose: some tests are moved out of their obvious suite, for example tests that boot a second runtime run in the suite that provides that runtime. If a filter finds nothing, the target most likely lives in the other suite. - Know where unit, integration, and acceptance tests each live so you invoke the right entry point. ## Restructure Code So a New Case Looks Planned From the Start When you modify code, do not simply append to what is there. Evaluate whether the surrounding code still makes sense with your change. If an abstraction was designed for one case and you are adding a second, restructure so the result reads as if both cases were intended from the start. Give each distinct concept its own clearly named unit, and compose those units in higher-level code so the intent is obvious without reading each implementation. ## Build Plugins on the Framework's Base Classes and Registry Build extensions on the framework's own extension points, not around them. - Inherit from the type-specific base class and override its lifecycle hooks: a setup or register hook plus the type's entry point. - Declare configuration through the framework's declarative option DSL instead of reading raw parameters, so validation, coercion, and documentation generation are handled for you. - Set concurrency at the class level: shared when multiple worker threads may call the instance at once, single when calls must be serialized so only one thread runs it at a time. - Route all plugin lookup through the central registry, register new built-ins there, and let external plugins be discovered as packaged units. ## Put Metrics and Shared Behavior in a Delegator That Wraps Each Plugin Keep cross-cutting concerns in a wrapping layer and take collaborators from the framework. - Run every plugin behind a delegator that adds cross-cutting concerns such as metrics and thread-safety; when changing behavior, decide whether it belongs in the delegator or in the plugin itself. - Let metrics follow a namespace hierarchy, creating children through a namespace call. - Provide a null-object metric that implements the same interface but discards data, for when metrics are disabled or under test. - Never instantiate metric objects directly; accept them from the framework. Standard mixins such as logging and configuration are included automatically, while heavier capabilities stay opt-in. ## Keep the Worker Loop Fast and Start and Stop in the Right Order Protect the hot path and respect start and stop ordering. - Treat the worker loop as the hot path: it pulls batches, runs them through the compiled pipeline, and signals completion. Do not add blocking calls or heavy allocations inside it. - Follow the strict lifecycle order: outputs and filters register first, then workers start, then inputs start last; on shutdown, inputs stop first, workers drain, then filters and outputs close. Violating this order causes data loss or deadlock. - Use atomic primitives for state shared across threads, not plain boolean fields, because cross-thread memory visibility is not otherwise guaranteed. ## Compile Pipeline Configuration Into Code and Dump It for Debugging Some engines compile declarative configuration into executable classes at runtime through a staged pipeline: parse the configuration text with a grammar into a syntax tree; lower that tree into an intermediate representation, a directed acyclic graph of nodes and conditionals; generate source for each node and compile it into classes at runtime inside a dedicated namespace; then wire the compiled units into the execution loop. Conditionals compile into predicate functions wired into branch nodes. Provide a debug flag that dumps the generated source to a directory so you can inspect what actually runs. ## Handle Errors, Nulls, and Logging the Same Way Across the Code Apply consistent structure, error, null, and logging conventions across the codebase. - Organize modules by functional domain, not architectural layer, and place each new type in the domain it belongs to. - Keep naming consistent: reserved suffixes and prefixes for extension, delegator, and abstract types; mark classes final unless subclassing is intended. - Prefer unchecked runtime exceptions; use checked ones only when unavoidable, always with a descriptive message and a preserved cause chain. - For nulls, validate constructor arguments with an explicit non-null check, annotate intentionally nullable contracts, and use an optional type for lookups that may fail rather than as a general null replacement. - Declare one static, final logger per class and choose levels deliberately: fine diagnostics low, recoverable issues as warnings, uncaught failures as errors. - Favor immutable final fields, use concurrent map types for shared caches, and document thread-safety where a type is shared. ## Send API Responses Through One Helper and Keep Logic in Command Classes Layer the service so routing, responses, and logic each have one home. - Have API modules inherit from a shared base that extends the web framework's base class and receives common dependencies through its constructor. - Mount endpoints at path prefixes declared in one central namespace map; add new namespaces there, not through ad-hoc routing. - Send every response through a single response helper that wraps payloads with default metadata; do not write raw responses by hand. - Represent errors with a small set of typed error subclasses such as not-found, bad-request, and timeout; raise them from handlers and let the base format one consistent envelope. - Keep business logic in dedicated command classes, not route handlers; a handler builds a command and passes its result to the response helper. ## Bridge Two Runtimes and Test the Round Trip Through Both When code bridges two runtimes, such as a host virtual machine driving a second language, register bridged classes at the framework's designated registration point and follow the patterns already there. Obtain per-thread context objects at the point of use, never storing them at construction time, because a stored context fails when reused from another thread. Test the round trip through both layers rather than each side alone, so you catch type-coercion and registration bugs that single-layer unit tests miss. Tests that load the second runtime must point their module lookup paths at the vendored location; that path is intentional, so do not change it while editing those tests. ## Test With Real Pipeline Helpers and Clean Up State Afterward Exercise real behavior with purpose-built helpers, and leave no state behind. - Prefer provided pipeline helpers that spin up a real pipeline and capture results over hand-constructing pipeline objects. - For isolation, define throwaway plugin classes inside the spec file and register them with the real registry. - Use shared examples for interface contracts and shared context for common setup; drive integration tests with a fixture-and-service pattern that bootstraps from fixture files. - Confirm that existing rescue or catch blocks actually cover newly added code paths. - Clean up global or virtual-machine state in teardown: every property you set needs a matching clear so state does not leak into other tests. - Prefer concrete, purpose-built doubles that inherit real base classes over generic mocks; reserve mocks for infrastructure boundaries, and let the real method run when a stub is only for setup. ## Check That Each New Test Fails Without Its Change Make sure each test actually depends on the change it guards. - Verify every new test fails without its corresponding change; a test that passes with or without the fix proves nothing. - Assert on the specific behavior the change introduces, not on pre-existing conditions: pin the assertion to an observable difference such as a new field, a changed value, or a previously missing log line. - For a bug fix, reproduce the bug in the test first; watch it fail for the same reason the bug manifests, then apply the fix and watch it pass. - Beware tautological mocks: stubbing a method to return the value you then assert is circular, and if deleting the implementation while keeping only the mock still passes, the test only exercises the mock. ## Roll Out Breaking Changes in Stages and Mask Secrets in Logs Change behavior on a staged ladder, guard sensitive values, and keep docs in step. - Stage breaking changes: introduce new behavior as opt-in and add deprecation warnings in minor releases, then change the default while keeping an opt-out and finally remove the old behavior in major releases. - For configuration that carries sensitive values, apply the framework's dedicated secret type or wrapper and override the string representation so the value is masked; never write such values to logs at any level by default. - Treat documentation as part of the change: update the relevant reference and contributor docs whenever behavior changes, follow each format's conventions, and require docs before merging user-facing features.