Build a System That Reads Uploaded Documents and Lets AI Agents Search Them

## Name Branches Clearly and Keep API, Worker, and Shared Code Apart Make ownership visible before changing code so each behavior stays in the layer that can test and maintain it. ### Use traceable branches - Name branches as `type/user/description pattern`. - Keep `type` lowercase and choose a recognized change class such as `feat`, `fix`, `refactor`, `chore`, `docs`, `test`, `perf`, `ci`, `build`, or `revert`. - Use the human work owner for `user`; never substitute a generic agent or tool name. - Keep `description` short, lowercase, and kebab-case. ### Separate runtime responsibilities - Keep synchronous API handling, asynchronous document work, shared models and services, and deployment concerns in distinct packages. - Place endpoint orchestration, business rules, persistence, parsing, retrieval, and generic utilities in modules with explicit ownership; treat independently distributed SDKs as separate products. - When behavior crosses packages, define the shared contract first and test each owning implementation. ## Send Each Uploaded File to a Parser That Fits Its Format Use a staged pipeline so every document follows a visible path from upload to cited retrieval evidence. 1. Accept the upload, create a job, persist the source object, and enqueue asynchronous work. 2. Profile the document once and route it to a parser by file type and structural evidence. 3. Convert parser output into one typed tabular contract. 4. Convert rows into typed chunk records. 5. Publish document, section, chunk, and graph state before retrieval. ### Keep parser routing explicit - Use one typed parser entry point for all formats and return task-local artifact paths. - Route PDF, DOCX, PPTX, spreadsheet, Markdown, text, image, and raw-fragment inputs through format-specific strategies, converting legacy office formats before the modern parser. - For PDFs, separate open semantic classification from routing-only categories such as atlas, scanned, slides, and generic. - Run structural anatomy once at the entry point for oversized documents, then pass the resulting shard plan downstream. - After parsing, remove unreferenced images and apply deterministic image compression. ## Find the Heading Hierarchy and Keep Tables and Images While Converting Determine structure from the strongest available evidence and retain tables, images, and heading relationships during conversion. ### Resolve headings in evidence order 1. Use an embedded table of contents as ground truth when available. 2. Match explicit numbering patterns and language-specific chapter markers. 3. Use an LLM hierarchy model only for unresolved candidate headings. 4. For layout-rich PDFs, cluster font geometry into a bounded set of heading tiers. ### Preserve format-specific behavior - Iterate DOCX body elements in source order, including paragraphs, tables, images, and table-of-contents regions. - Maintain a heading stack that closes descendants when a new heading returns to a parent level. - Convert tables to HTML with accurate `rowspan` and `colspan`, and deduplicate images with a perceptual hash. - Route standard PDFs through extraction and Markdown parsing; route atlas-like or oversized documents through their specialized paths. - Keep summarization, hierarchy recognition, image description, OCR, and coarse classification behind separate model configuration keys. ## Save Chunks, Navigation, and Manifest Files in a Fixed Format Treat the persisted corpus as authoritative; intermediate DataFrames are transient implementation details. ### Store corpus and document state - Keep corpus-wide graph metadata and retrieval-hit statistics beside one directory per ingested document. - Persist ordered chunks, hierarchical navigation, parse metadata, the full heading tree, archived source assets, extracted images and tables, and diagnostic outputs. - Use exactly three core chunk types: `text`, `image`, and `table`. - Give every chunk a deterministic `chunk_id`, typed content, a hierarchical path, and metadata for length, summaries, page numbers, tokens, keywords, file paths, and cross-chunk references. ### Preserve references and navigation - Represent inline relationships through `connect_to` entries containing `target`, `relation`, `ref`, and character-offset `position`. - Use `embeds` for inline assets and `related` for non-inline associations. - Build `doc_nav.json` as a section tree with summaries, chunk counts, children, and image or table resources. - Build `manifest.json` with source metadata, statistics, and a nested `HIERARCHY`. - Convert transient parser rows to typed chunk payloads before writing `chunks.json`. ## Publish Database Records, Search Indexes, and Graph Links Together Publish one coherent revision so relational records, search channels, and graph edges describe the same chunk set. ### Maintain relational ownership - Store document identity, owner, namespace, status, active revision, and source filename at document level. - Store parent links, paths, titles, levels, summaries, and sort order in a revision-aware section tree. - Store content hashes, chunk types, searchable text, source paths, asset paths, metadata, and sort order per chunk. - Keep graph nodes document-scoped and graph edges undirected when they represent symmetric relatedness. ### Publish in a fixed sequence 1. Deduplicate chunks across documents by deterministic content hash. 2. Build ancestor sections from chunk paths in top-down order. 3. Generate separate content, path, and raw-term search channels for every chunk. 4. Create document graph nodes and keyword-overlap edges only when at least 3 keywords are shared and the score is at least 0.8. Use tokenized content plus summaries for content search, tokenized filenames and section paths for path search, and raw content plus paths for substring search. ## Support Classic Ranked Search and Agent Search Within a Set Budget Separate request shaping, lexical search, hydration, graph access, hit statistics, workflow execution, navigation, and evidence rendering by ownership. ### Preserve both retrieval modes - In legacy mode, combine path, content, and term channels with reciprocal-rank fusion. - Use default channel weights of `path=1.0`, `content=2.0`, and `term=1.5`, with `score = weight / (k + rank + 1)` and `k=60`. - In agentic mode, let a planner decompose complex queries into a DAG of retrieval steps; do not plan answer-synthesis steps inside retrieval. - Enforce one token budget across the complete DAG. When exhausted, stop safely and return the best evidence collected so far. - Execute discovery, constrained breadth-first section navigation, and evidence rendering as distinct phases. - Expose `evidence_text` as the primary response and retain deprecated `answer_text` only as an empty compatibility field. ### Hydrate and rank deliberately - Render unopened sections as title-and-summary outlines and hydrate raw `text`, `image`, and `table` chunks only for selected leaves. - Resolve connected assets natively and embed them inline without brittle string-replacement placeholders. - Rank agent results by `agent_score`, then `discovery_score`; demote rows with `agent_score=0` to a fallback pool. - Filter excluded documents, excluded sections, and allowed chunk types before assembly. - Attach citations containing `document_id`, `chunk_id`, `source_file_name`, and `section_path`. ## Keep Chunk IDs Stable and Cache Only Successful Results Make shortcuts, identifiers, mutation order, and telemetry explicit so optimizations do not change retrieval semantics. ### Keep results reproducible - Derive chunk IDs from a stable hash of text content alone, excluding image and table references, so identical content deduplicates across uploads. - Plan DOM mutations first, then apply them in reverse order to avoid index shifts. - Deduplicate equal images within a document by perceptual hash while reusing cached metadata. - Give repeated images at different positions deterministic but unique IDs, and scan past binary identifiers when finding nearby textual context. ### Bound shortcuts and caches - When `total_chunks <= top_k`, return all chunks directly through an explicit small-corpus route. - Cache by user, namespace, query, `top_k`, and filters; verify the cache version before execution and write only after success. - Record per-chunk and per-document hits for ranking boosts without mutating source content. - Keep append-only run and step records for agent actions, inputs, outputs, latency, and token usage. ## Handle LLM API Quirks and Check Locally Before Reporting Done Keep model constraints and local verification commands explicit so failures remain diagnosable. ### Harden model integrations - Include the word `json` when requesting DeepSeek `json_object` mode. - During streaming, concatenate `delta.content` only when it is not `None`. - Rotate API tokens within per-token rate limits, apply cooldowns, and retry inline with the next available token. - Keep retrieval evidence-only; downstream agents decide whether evidence is sufficient and synthesize answers outside the retrieval service. ### Verify the system locally - Install from the pinned workspace, copy environment templates locally, and start required database, cache, object-storage, API, and worker services. - Provide focused parser, profiler, hierarchy, and agentic end-to-end debug commands, then run lint, safe auto-fixes, and type checks across API, worker, and shared packages. - Verify local API documentation and backing-service endpoints before diagnosing higher-level pipeline failures. - Do not report completion until the affected parser, publication path, retrieval mode, and quality checks have direct results.