Build a Web Page Content Extractor and Debug Missing Content

## Make the Extractor Behave the Same in Every Supported Environment Design the extractor around a stable document interface rather than one browser implementation. List every supported environment and its constraints before changing shared parsing behavior. - Browser builds may use the native document model. - Server builds may accept documents created by different DOM implementations and may expose an asynchronous API. - Command-line builds must preserve the same extraction semantics while offering machine-readable and text-oriented modes. - Edge runtimes may provide the most constrained document implementation and should define the compatibility floor. - Bundle entry points must expose the same public behavior even when their module formats differ. Keep environment adapters thin. Place parsing, scoring, normalization, sanitization, and element-specific behavior in shared layers unless a runtime limitation requires an explicit exception. ## Keep the Cleanup Steps in Order and Do Not Reorder Them for One Site Treat the extraction pipeline as an ordered algorithm, not a collection of interchangeable cleanup functions. Each stage may create assumptions consumed by later stages. Use this sequence as the default: 1. Flatten encapsulated document trees. 2. Resolve streamed or deferred content. 3. Select the main content automatically or through an explicit selector. 4. Normalize structures that removal rules could otherwise hide or discard. 5. Convert recognized callouts into a stable semantic representation. 6. Remove undersized media and hidden elements. 7. Remove low-scoring blocks. 8. Apply exact and partial clutter selectors. 9. Remove repeated boilerplate and other content patterns. 10. Normalize the remaining document. 11. Resolve relative links and resource locations. Document why a stage must precede another whenever the dependency is not obvious. Do not reorder shared stages to fix one site until a focused test proves the new order preserves every required intermediate representation. ## Use a Debug Log of Removals to Find Where Content Disappeared Provide a debug mode that reports the selected content root and every removal with its stage, selector, reason, and representative text. Debug mode may preserve diagnostic attributes and avoid structure-flattening operations that would erase evidence. When expected content disappears: 1. Inspect the removal log for the missing content. 2. Disable one stage at a time to identify the first stage that causes the loss. 3. If a selector is responsible, distinguish exact selectors from partial-match selectors. 4. Bypass automatic content detection with an explicit selector when the wrong root was chosen. 5. Confirm that protected code or preformatted regions are excluded from generic removal. 6. Re-enable all unrelated stages before accepting the fix. Prefer evidence from the earliest incorrect transition over guesses based on the final document. Do not disable broad cleanup behavior permanently when a narrow selector, ordering, or element rule can solve the defect. ## Treat Page HTML as Hostile and Block Script Injection Parse markup through a safe document-construction helper that does not execute scripts or load resources. Never assign untrusted markup directly through a browser parsing sink. - Strip script-bearing URL schemes from link and media attributes. - Remove embedded-document source content from frames. - Remove event-handler attributes. - Escape every page-derived value before interpolating it into generated markup. - Prefer node creation and attribute setters when they make escaping requirements explicit. - Sanitize the completed output as a backstop, not as a substitute for safe construction. Assume attacker-controlled quotes in media URLs, alternate text, metadata images, and descriptions can terminate an attribute and inject another one. Every extractor that generates markup must have an adversarial attribute-injection test. ## Prove Each Fix with a Small Test Page That Fails First For extraction defects, create the smallest document fixture that still reproduces the failure and pair it with the exact expected output. Keep fixtures deterministic and free of personal or source-specific identifiers. - Replace real names, email addresses, URLs, account identifiers, and recognizable text with generic placeholders. - Include only the structure needed to exercise the failing pipeline stage. - For security fixes, include a poisoned attribute or equivalent adversarial payload. - Confirm the fixture fails before applying the fix. - Apply the smallest behavior change that makes the fixture pass. - Run the focused fixture and the shared extraction suite across supported environments. A fixture that passes before and after the implementation change does not prove a regression was fixed. If reverting the fix is impractical, construct the expected result or controlled toggle so the old behavior is still demonstrated to fail. ## Test DOM and Bundle Differences in Each Environment Test edge behavior where document implementations and bundle formats differ, even when the shared algorithm is unchanged. - Convert live element collections to static arrays before mutating the document. - Verify that named utilities remain reachable when a bundle exposes only a default export. - Exercise asynchronous server adapters independently from synchronous browser adapters. - Test edge-runtime behavior through its actual request boundary rather than assuming browser parity. - Verify command-line text and structured modes against the same fixture corpus. - Keep environment-only workarounds in adapters and cover each one with a focused compatibility test. When an environment-specific fix changes shared output, treat it as a cross-environment behavior change and validate every supported adapter. Do not let a packaging workaround silently redefine the extractor's public contract.