Palimnote

Beyond packages/db, the monorepo has grown, phase by phase, to include the following shared packages, each documented in full later in this report:

A cornerstone of the whole design is a closed, fixed vocabulary of ten relationship categories that every classified citation, annotation, and graph edge is expressed in terms of: explicit_reference, secondary_scholarly_recommendation, historical_context, prerequisite, conceptual_influence, disagreement_polemical_target, interpretive_aid, parallel_comparison, optional_extension, and ai_inferred. This exact list appears, verbatim and in the same order, in the Postgres relationship_category enum, in the packages/ai-adapters TypeScript types, in packages/roadmap’s tier-mapping table, and in packages/curriculum’s stage-mapping table — the single vocabulary threading through the whole system, from extraction through classification through ranking through pedagogical sequencing.

The project’s original design document specified persisted reading_roadmaps and roadmap_items tables — a stored snapshot of each user’s computed roadmap. The shipped schema deliberately does not have these tables. Instead, the roadmap is computed fresh on every request from the underlying citation graph (graph_edges) plus the durable, user-authored inputs that actually need to persist: reading_records, understanding_ratings, and roadmap_overrides. The documented rationale is that a stored snapshot would drift the instant a re-analysis runs or a user changes a rating, whereas computing on demand makes “recalculation respects overrides” free rather than requiring separate reconciliation logic. The same on-demand philosophy is later extended to the Curriculum view, the Roadmap-annotated 3D graph projection, and the Library’s cross-run “resource role” view.

No AI vendor SDK is imported anywhere in the codebase. Both packages/ai-adapters’ OpenAI and Anthropic providers, and packages/bibliographic’s four external bibliographic sources, are implemented over plain fetch calls against each provider’s HTTP API. This was a deliberate choice recorded in the project’s own design-decision log: no SDK dependency to install or build, and less code and less supply-chain surface than pulling in two vendor SDKs — business logic anywhere in the codebase only ever depends on the small LLMProvider/BibliographicSource interfaces, never on a concrete vendor import.

Every AI task defaults to the cheapest viable model tier (gpt-5.4-nano / claude-haiku-4-5), with a pricier tier reserved only for specific, deliberately narrower uses (a research-synthesis route, a competency-designation route) and never silently promoted — a promotion is a documented, explicit configuration change, gated on eval-harness evidence. Every AI call, including a $0-cost heuristic fallback, is logged to an ai_usage_log table, which is what powers the admin cost dashboard.

parseDocument(buffer, mimeType) dispatches by MIME type to parsePdf, parseText, or parseEpub, then passes the result through an internal sanitizeParsedDocument step before returning. This sanitization step recursively applies sanitizeExtractedText (see §4.9) to every string field of the parsed document — the merged text, the detected title/author, and every page’s and block’s text — so that every parser’s output passes through the same defensive choke point before it can ever reach the database or a job payload. This exists specifically because of a real production incident: a malformed-Unicode fragment from a damaged or OCR’d source once crashed the entire worker process with “unsupported Unicode escape sequence,” rather than failing just the one job.

This is the structurally central file of the package. It defines the shared ParsedDocument/ParsedPage/ParsedBlock type contract the whole package is built around, and orchestrates pdf.js extraction, OCR fallback, and optional GROBID enrichment.

processedTextFromPages(pages) builds the reader/analysis transcript by joining only prose-bearing block kinds (title, header, body, caption) — footnotes, endnotes, and bibliography blocks stay persisted and individually addressable but are deliberately never folded into the body text, preventing the same source material from being shown twice, once as apparatus and once as prose.

mergePageTexts(pageTexts) trims, drops empty pages, and joins the rest — rebuilt deliberately from the final per-page texts (including any OCR results), fixing a prior bug where a scanned document whose text layer started empty stayed empty in the merged document text even after OCR successfully recovered content.

**metadataConfidenceFor(titleSource, hasFallbackTitle)** is a small decision table converting a title's provenance into a numeric confidence consumed downstream by the worker's auto-ready gate: a GROBID header-sourced title scores0.95(near-certain); a *recovered* title (the header title was distrusted and a body heading was substituted instead) scores only0.7, deliberately kept below the auto-ready threshold so the document routes to manual review rather than asserting a second guess with the same false certainty that caused the original defect; a plain PDF-metadata or first-line fallback scores0.65if present, else0`.

parsePdf(buffer) runs, in order: (1) pdf.js text-layer extraction, once merged and once per-page; (2) PDF metadata reading, tolerating failure as “no metadata available” rather than a parse failure; (3) per-page block construction, flagging any page whose text is under 40 characters as low-confidence (possibly scanned); (4) OCR fallback for sparse pages, folding results into both the per-page record and the merged document text; (5) a title fallback from the first non-empty short line of the merged text if no PDF-metadata title exists; (6) optional GROBID enrichment.

Where GROBID runs and returns at least one body block, its entire structured block set replaces — never appends to — each page’s block array; the code’s own comment is explicit that appending would be “a structural lie,” since the pdf.js fallback body already contains footnote and bibliography glyphs inline, and appending structured notes on top would show the same material twice. The raw per-page pdf.js/OCR text is left untouched even after replacement, which is what lets a later recovery pass (§4.3) still see the original text. structureState is set to "full" only if GROBID actually ran, produced non-empty text, and produced at least one genuinely body-kind block — a syntactically valid but content-empty TEI response is explicitly not enough to claim structural fidelity.

The most intricate file in the package, holding extensive fixture-driven “contamination guard” logic addressing a specific, measured GROBID model failure: its citation-segmentation model (trained on itemized numbered reference lists, not the continuous-prose, multi-citation-per-footnote style common in humanities scholarship) sometimes “bleeds” one bibliography entry’s author into a structurally nearby sibling that has no author of its own. A fixed adjacency window (three positions in document order) and a whole-document pre-pass mapping every confidently-owned surname to its true owning entry together decide whether a bare surname on a different entry is contamination or a genuinely different person who happens to share a name — a document-wide (unbounded) version of this check was proven, by an adversarial two-distinct-people-same-surname test, to be too broad.

walkBody(node, ctx) recursively walks the TEI <text> subtree in document order, emitting one block per structurally meaningful element: headings become header blocks; <note> elements become footnote or endnote blocks (carrying their n attribute as the marker, emitted separately from body text so a paragraph’s footnotes are never duplicated into its own prose); paragraphs become body blocks with their nested notes excluded from the paragraph’s own text but still recursed into as their own blocks; bibliography structures become reference blocks using the contamination-guarded text-builder; figures and tables become caption blocks.

headerAuthorsMissing(header) and recoverTitleFromHeadings(blocks) implement a Phase-20.67 fix for a real, observed defect: on at least one real fixture, GROBID’s header-segmentation model locked onto the wrong page region entirely (extracting a journal’s copublisher line as the “title,” with zero author names also extracted). Because there is no reliable text blocklist for every possible publisher string, the fix instead uses “the extracted header region contains zero person names at all” as the structural tell, and recovers a replacement title from GROBID’s own separate body-segmentation pass — finding the earliest page carrying any heading with bounding-box coordinates and picking the heading with the largest bounding-box height, reasoning that a running venue line prints smaller than the real title on the same page.

processWithGrobid(buffer) reads the configured GROBID service URL from the environment and returns null immediately if it is unset — a missing configuration means “disabled,” never a silent transfer of a private document to an unintended third-party service. A per-process concurrency semaphore (default limit of 1) serializes calls to the GROBID service, since the underlying model service is memory-bound and can be OOM-killed under parallel full-text requests. A companion module, gcpIdToken.ts, mints a self-signed-JWT-bearer Google ID token so the worker (which runs outside GCP and has no metadata server available to it) can call an authenticated, private Cloud Run GROBID instance — falling back to null (unauthenticated) if no service-account key is configured, which then naturally degrades to the standard “GROBID unavailable” fallback path.

ocrLowTextPages(pdf, pageTexts) selects pages whose text is under 40 characters, capped at a configurable maximum (default 12 pages), creates one Tesseract worker for the whole batch to bound memory, and processes pages sequentially rather than in parallel — an explicit memory-management choice. Any failure (a missing native renderer, a missing language model) is caught and the function returns whatever results were already collected, since OCR is explicitly treated as an enhancement that must never discard page boundaries or degrade an otherwise-readable PDF.

This addresses a separate, confirmed GROBID gap, documented as “D-20-89”: re-running the same fixture with and without requesting positional coordinates produced byte-identical missing-endnote counts, ruling out a request-parameter fix. recoverTruncatedEndnotes({pageTexts, structuredMarkers}) scans raw per-page text for a “Notes”/“Endnotes” heading, then expects a strictly sequential run of numbered entries (1, 2, 3, …), filtering out running-header/footer boilerplate lines (detected by their repetition rate across the document) along the way. A trust gate requires at least three sequential entries to be found before any are accepted at all — “not a real endnotes list” is the safe default otherwise — and any recovered marker already present in GROBID’s own structural output is dropped, so recovery can only add coverage, never duplicate it. Recovered entries are tagged recovered: true and, because they are endnote-kind, are automatically excluded from the reader’s body transcript by processedTextFromPages — recovery can add addressable apparatus, but can never leak into prose.

Explicitly documented as “not a parser” — a pattern-matching, always-on, zero-cost baseline stage that runs ahead of any AI-assisted extraction. Handles reference/bibliography-section splitting, inline author-year forms ((Kant 1781), Kant (1781)), a deliberately narrow allowlist of direct classical-work mentions (so it never over-matches generic title-cased phrases), and — the module’s most extensively justified addition — two “note-style” citation patterns added after a concrete production observation: on a real 2001 philosophy article that cites entirely via numbered footnotes with no bibliography section at all, both the reference-section pass and the author-year pass extracted zero citations, and GROBID itself only recovered four of roughly forty notes since it had no bibliography section of its own to anchor against. Without these two added patterns, the pipeline would have seen none of that article’s citations at all.

splitNoteEntries(text) un-bundles a single footnote block that actually contains two independent citations joined by connective prose with no semicolon at all — but only commits to a split when every resulting segment independently looks like a real citation, an explicitly conservative “when in doubt, do not split” discipline recorded as a direct fix for an earlier over-eager split that had mis-divided ordinary scholarly-discourse sentences.

extractCitationMentions(sources, max) is the provenance-preserving sibling of the simpler discovery-oriented extractCitations: a bibliography-type source is trusted verbatim as a single citation (specifically avoiding the case where the extractor misreads a title’s own words, such as one containing the phrase “Nicomachean Ethics,” as a second spurious inline citation); a footnote or endnote source is first split via splitNoteEntries, then each resulting segment is independently re-run through the extractor, and even a segment that yields zero regex matches is still preserved verbatim as a low-confidence lookup candidate rather than silently dropped — “a structural footnote or endnote entry is itself evidence even when it lacks a year or uses an unrecognized catalog style.”

detectFootnotes(text) is a much simpler, legacy-scoped heuristic for plain-text/Markdown documents only: it finds a trailing run of consecutive numbered lines and cross-checks each candidate number against an in-body [N]/(N) marker earlier in the document, which is the real false-positive guard (not the length of the run) — this is what filters out an unrelated numbered list, like a table of contents, sitting near the end of a document.

parseText(buffer, mimeType) produces a single-page document with a conservative, line-pattern-based structural split (a Markdown heading, a recognized “notes”/“bibliography” section heading, a caption line), explicitly documented as still “structure-limited,” since line patterns cannot assert real page-layout truth the way GROBID’s structural pass can.

parseEpub(buffer) parses via epub2, producing one page per non-empty chapter (no finer within-chapter structural split), converting each chapter’s HTML to plain text. Its own documentation is explicit that an encrypted or malformed EPUB produces a normal processing failure, never a DRM bypass attempt.

extractAuthorApparatus({blocks, text, maxCitationBlocks}) runs a structural pass over already-classified blocks (tracking an “in endnotes” state that flips on at a recognized notes heading and off at the next section heading, catching the common case where a structural parser classified a note-section paragraph as ordinary body text) plus a heuristic citation pass over the merged text, de-duplicating by normalized text across both.

sanitizeExtractedText(text) replaces every unpaired UTF-16 surrogate with the Unicode replacement character and strips NUL bytes — the ingestion-boundary fix for the production crash described in §4.1, applied universally to every parser’s output.

storage.ts is the shared Supabase Storage access layer (server-only, service-role key), offering uploadDocumentFile (never silently overwrites, upsert: false), createSignedUploadUrl (lets a large file be PUT directly to Storage from the browser, bypassing Vercel’s request-body size limit), downloadDocumentFile, deleteDocumentFile, getDocumentFileSize (the authoritative Storage-reported byte count, used specifically to close a signed-upload quota-bypass path where a client-declared size could otherwise be trusted without verification), and getSignedDocumentUrl (a short-lived signed URL for the reader UI, whose caller is responsible for checking ownership before ever calling it, since the returned URL itself grants access to anyone holding it for its TTL window).

validation.ts performs format- and archive-level validation — explicitly not a malware-detection claim, as the module’s own comment states directly — checking PDF magic bytes, EPUB ZIP/manifest structure with a zip-bomb-style entry-count guard, and (for plain text) the absence of embedded binary data, plus an optional private ClamAV-compatible scan that is genuinely a no-op (returning “valid” without pretending to scan) whenever no scanner endpoint is configured.

@ice/roadmap is a single, dependency-free module implementing the pure ranking function behind the personalized reading roadmap. It performs no I/O, no database access, and no randomness — which is exactly what lets its own test suite assert the project’s plan-derived Heidegger and Vico acceptance cases deterministically.

CATEGORY_TIER is a total, hardcoded mapping from each of the ten relationship categories to one of seven priority tiers (essential, high, strongly_recommended, contextual, interpretive_aid, comparative, optional) — prerequisite maps to essential; conceptual_influence and disagreement_polemical_target map to high; and so on. KNOWN_THRESHOLD = 60 is defined once here and imported everywhere else in the system that needs the “known” cutoff.

matchesReaderLevel(materialLevel, selectedLevel, mode) decides whether one piece of material belongs in a given reader-level view: universal (untagged) material and the "all" selection are always shown; a "cumulative" mode shows everything at or below the selected level (a superset accumulation); an "exact" mode shows only material tagged at exactly that level (plus universal material, deliberately still included).

collapseDuplicateCandidates(candidates) implements duplicate-edition collapse: because the graph traversal can reach several distinct bibliographic records for the same real-world work (the book itself, a review of it, a second edition), this function groups candidates by a normalized title and merges each group into one deterministic “primary” survivor — chosen first by whether the item is already in the reader’s library, then by shallowest graph depth, then by highest centrality, then by a stable id tiebreak — while unioning categories, taking the maximum confidence, and recording every folded-away id so a later graph-node lookup can still try any of them.

rankRoadmap(candidates, profile, overrides, options) is the package’s central export. For each candidate it resolves the strongest category present (by tier rank), applies any manual tier override, filters by the requested depth mode (concise keeps only high-tier-and-above) and reader level (a manual override always bypasses the level filter, so a pinned item is never silently hidden by a level change), determines “known” status from the profile, and generates a deterministic, template-based “why this, here” explanation via reasonFor(category, centrality, known) — never a model call. The final sort chains: any manual position pin first; then known items sink below unknown ones; then priority tier; then descending centrality; then descending confidence; then an alphabetical title tiebreak. If a time budget is supplied, items are never dropped for exceeding it, only flagged overBudget: true.

mergeRoadmapsAcrossRoots(perRoot, profile, options) (Phase 22.7) merges several selected root works’ roadmaps into one sequence, reusing — never re-implementing — the same collapseDuplicateCandidates and rankRoadmap pipeline over the unioned candidate set, with a documented composition rule for multi-root overrides: an item is hidden in the merged view only if every root that reaches it individually marks it hidden.

@ice/curriculum’s only dependency is @ice/roadmap itself. stageForRelationship(category) is a total, compile-time-exhaustive lookup mapping each of the ten relationship categories into one of five fixed pedagogical stages, in order: prerequisites, formative_context, core_engagement, interpretation_context, extension. checkpointFor(category) is likewise a total, hardcoded lookup of deterministic, template-based self-check questions keyed by category — the module’s own documentation frames this explicitly as matching the project’s anti-hallucination posture, since a self-check question “is cheap to fabricate plausibly and expensive to verify,” so it is generated the same deterministic way reasonFor() is, with no model call at all.

hasCycle(items) is a general-purpose, textbook three-color depth-first cycle detector, deliberately more general than the fixed five-stage structure strictly needs — and assertAcyclicStages(items) uses it as a genuine runtime proof, not merely an assumption, that the fixed stage ordering can never produce a cyclic dependency graph, functioning as the pure-function equivalent of a database CHECK constraint.

@ice/consistency implements Phase 20.7’s data-integrity layer: nine deterministic checks, each taking an already-fetched, flat, whole-database snapshot and returning a list of typed mismatches, plus a matching repair action where one can be safely derived. Its central anti-hallucination discipline, stated in its own type definitions, is that a repair is only ever emitted when the correct value is derivable from an already-existing canonical fact — never guessed — and every emitted repair’s reason field must name the canonical source it was derived from.

The nine checks are: citation↔︎Library-item consistency (a resolved citation with no matching Library link, or one pointing at the wrong link); Library-item↔︎canonical-work consistency (a stale pointer into an identity that has since been merged away, correctable to the resolved winner via the merge-chain resolver); graph-node↔︎canonical-entity consistency (report-only — deliberately never repairs the same fact the prior check already owns, to avoid two code paths applying the same patch twice); graph-edge endpoint validity (a dangling edge, since graph_edges has no real foreign key — repaired by deletion, since there is no canonical replacement endpoint to guess); annotation↔︎related-work consistency (a stale cached title-label resynced from its own resolved foreign-key target); roadmap-item target canonicality (a roadmap override, reading record, or understanding rating still pointing at a non-canonical duplicate record); reader-source↔︎citation consistency (a citation’s cached processing-run id resynced from its own real foreign-key chain); RAG-citation anchor integrity (the one security-sensitive check — verifying a cited chunk’s owner matches the citing conversation’s owner, with no repair path other than deletion, since substituting any other chunk for a privacy-boundary violation would itself be a guess); and title/author/year agreement (a broad, mostly report-only check between a work and its canonical identity, deliberately never auto-repaired in the work-title direction, since a work’s own uploaded-document title is legitimately allowed to differ from an aggregated canonical title — for instance across translations — and overwriting either side “would risk destroying real, user-supplied data to chase an aggregate that isn’t necessarily more correct for this row”).

runAllConsistencyChecks(snapshot) runs all nine checks in a fixed order and concatenates their findings, and this exact function is what both a plain report run and the actual repair-mode runner both call — guaranteeing “what would be reported” and “what a repair pass acts on” can never diverge into two separately-maintained code paths. The worker-side runner (apps/worker/src/consistency/run.ts, described in §9) applies any repairs inside a single database transaction, so a partial failure mid-batch can never leave the database in a state worse than either fully pre- or fully post-repair, and it re-runs the full check pass after applying repairs to print a genuine, empirically verified before/after comparison rather than merely asserting success.

@ice/deletion was built in Phase 20.3 to replace an earlier “fire-and-forget” purge implementation that could report success while Storage bytes were, in fact, still present. The package contains no I/O at all — every side effect (database reads/writes, Storage deletion, job cancellation) is injected via an effects interface, which is what makes its ordering, failure, and idempotency guarantees testable as deterministic unit tests without a real database or Storage bucket.

The core design invariant, stated in the module’s own header comment, is that deletion is deliberately ordered Storage first, database last: a durable cleanup record is persisted before any destructive step; queued jobs are cancelled so no worker can re-touch the work mid-delete; every private Storage object is deleted; and only then is the work row hard-deleted, letting Postgres cascade the rest. This ordering is what guarantees the specific failure mode the project’s honesty requirement targets — “database rows gone but bytes left behind with no record of it” — cannot happen silently: bytes are only ever removed before the rows that track them, and any Storage failure halts the run in a persisted, retryable storage_failed state rather than being reported as success.

executeWorkDeletion(effects, input) short-circuits immediately if a cleanup record already shows completed (idempotent on repeat calls); otherwise it collects the work’s Storage paths (unioning with any already-recorded pending paths from a prior partial attempt) and persists the collected state before any destructive step, which is the crash-safety anchor — if everything below crashes, the pending paths already survive on disk. It cancels queued jobs; deletes every pending Storage path (attempting every path even after an early failure, retaining only the first error message, and removing successfully-deleted paths from the pending set as it goes, so a retry only re-attempts what’s still outstanding); and only once every path is confirmed gone does it call the database-delete effect. A crash between the database delete and the final “completed” write converges cleanly on retry, since the machine re-runs against an empty pending set and the work-row delete becomes a safe no-op.

@ice/phase-lifecycle is not consumed by the product at runtime at all; it is meta-tooling governing the AI-agent development process that built the project. It was built specifically because a prior agent session had, in the project’s own words, “failed closed” without a real way to safely and provably terminate its active model context and hand off to a fresh one at a phase boundary — the package’s header comment states its rationale precisely: “a shell command cannot prove that an agent’s active context was destroyed,” so the package refuses to fake compaction and instead requires a host-supplied adapter that can atomically replace the active session.

PhaseLifecycleController.closePhase(input) enforces, in order: that a phase may only dispatch its immediate successor (never skip or reorder phases); that the handoff prompt seeding the next phase is non-blank; that the phase’s own verification actually passed (the tracker is never advanced on a failed test); and, critically, a host preflight check — if the host cannot prove it can replace the active session, the controller throws before claiming any dishonest success, and it writes its durable bookkeeping (the compact handoff record and the status tracker) before attempting the risky, one-shot context-replacement call, mirroring the same “durable write before destructive step” discipline seen in @ice/deletion.

@ice/config’s pipeline.ts centralizes which analysis pipeline version a deployment runs, replacing what its own header comment describes as a real production incident: two call sites once independently compared the pipeline-version environment variable for exact string equality, so any unrecognized value — including a genuinely new, valid version — silently fell back to the oldest pipeline and silently disabled edition reprocessing, with no diagnostic anywhere. isEditionPipeline(version) reframes the question from an equality check to an ordered one — “is this pipeline at least the edition pipeline?” — which is the question the code actually needs answered, and parsePipelineVersion logs a one-time warning (not a silent fallback) whenever it encounters a genuinely unrecognized value.

stages.ts is the single source of truth for the ordered, real processing-stage sequences of each pipeline version, shared between the worker (which sets these exact string labels on a processing run’s stage column as it executes) and the web application (which renders them as an honest, step-by-step progress indicator) — the module’s own comment stresses that neither pipeline version is credited with a stage it doesn’t actually set.

phase12.ts, phase18.ts, and phase22.ts each define independently addressable release flags, all following the identical pattern and all defaulting off except a Phase-12 “foundation” flag — and all sharing one explicit design principle stated in each module’s own comment: these are release controls, not authorization controls. Every protected route still performs its own auth and ownership check regardless of flag state; a flag only controls whether a feature is exposed at all, deliberately so a feature’s mere presence in a deployed build can never itself become a security boundary.

@ice/observability is the single, dependency-light error-and-event reporting seam shared by both applications, following the same “adapter plus fallback” pattern used elsewhere in the codebase (the mail provider, the AI classifier): if a real monitoring backend is configured, errors are forwarded there, but regardless, every error is always logged locally in a structured, parseable form — nothing is ever silently swallowed. reportError(error, context) normalizes any thrown value to a real error, logs a structured JSON record, and forwards to an optionally-registered external reporter. reportEvent(event, context) logs an operational metric as a structured, content-free record — explicitly the safe channel for cost, usage, and audit events, distinct from the failure channel, and specifically documented as standing in for a dedicated audit table in places (such as the consistency-repair applier) where building new schema for an audit trail was judged unnecessary given this existing structured-event log.

The package has no dependency on any monitoring vendor’s SDK itself — the actual Sentry wiring lives in the consuming application (apps/worker/src/sentry.ts), which registers itself as the external reporter only if a DSN is actually configured, keeping the worker (and any future consumer) free of that dependency otherwise.

Beyond the schema described in §3, packages/db/src/index.ts establishes the single shared Postgres connection with {prepare: false} — a deliberate choice tied directly to production infrastructure, since Supabase’s transaction-pooling connection mode (used by the Render-deployed worker) does not support prepared statements. It also bridges Drizzle-managed tables with pg-boss’s separate, Drizzle-unaware internal schema: cancelQueuedJobsForDocuments(documentIds) deletes any non-terminal queued job whose payload references one of the given documents, catching and treating a “relation does not exist” Postgres error as “nothing was queued” rather than propagating it — because pg-boss lazily creates its own internal schema only on first start, so a fresh test database genuinely has no such schema to query yet. This function exists because Postgres’s own cascading delete on a work/document has no visibility into pg-boss’s separate job table, and without it a job left non-terminal at delete time would survive forever and eventually fire a spurious “document not found” error when a worker later dequeued it — a real batch of hours-old stale jobs was discovered draining on local worker boot during a Phase-19 audit before this fix.

queue.ts defines the four named pg-boss queues (extract-text, analyze-work, resolve-citation-metadata, expand-cross-library-graph) and, as of Phase 20.5, the pure decision core for the single idempotent reprocess command. buildPgBossConfig(connectionString) parses the database URL into discrete host/port/user/password fields rather than passing a connection string directly to pg-boss’s underlying driver — a documented fix for a real production incident where the driver’s own connection-parameter merging logic silently overwrote an explicitly-set TLS option whenever a connection string was also supplied, breaking worker startup against the database’s pooler after a credential rotation. planReprocess(input) is a pure function (no database access) that classifies the current queue/run state into one of four actions: reuse an already-queued identical attempt (a double-click protection), recover an orphaned active job left by a dead worker (only when the job is stale and there is no fresher heartbeat proving the run is still genuinely alive), enqueue fresh when nothing is pending, or report a conflict when a live attempt is genuinely running and starting another would duplicate paid AI work.

buildGraph(userId, rootWorkId?) is the single canonical graph-construction function behind both the global and work-scoped Visualization. It assembles works, cited bibliographic records and concepts (with missing/read/unread/reading node state derived from the user’s own reading records and understanding ratings), discovered research sources with provenance, durable resource-role recommendations, and structurally-anchored passage annotations into one typed node/link payload — recomputed fresh on every request, never persisted.

A central, carefully-documented step is the Phase 20.6 canonical collapse: research_resource rows sharing a derived work key (restricted to primary/edition roles) are grouped, and all but one representative are aliased to it, so a cited book that happened to resolve to two separate catalogue records renders as one visual node rather than two — deliberately not applied to reviews, which are instead attached via relation edges rather than collapsed, since a review is not the same work as the thing it reviews. A separate, explicit mergeExternal() upsert function unions provider/provenance arrays across data sources, keeps the better of two conflicting authority scores, and is written specifically to never let a source-type label overwrite a properly-typed bibliographic-record node.

Edges are assembled from six independent sources (explicit citation edges, concept-presupposition edges, synthetic never-persisted document-outline edges for the work-scoped view, discovered-source edges, source-to-source relation edges, and — for the global graph only — the paid, evidence-hashed Cross-Library Graph judgments, filtered to keep only the single latest judgment per directed pair) and then deduplicated by a stable source|edgeType|target key, merging evidence and provenance arrays and keeping the maximum confidence across duplicates. A final pass computes each node’s real in-app destination link, each link’s stable id and directed-vs-undirected flag, and propagates an associatedWorkIds set outward one hop from every work node, so the accessible table and the 3D scene can filter consistently by “which of my own works does this relate to.”

computeRoadmapCandidates(userId, rootWorkId) runs a single recursive Common Table Expression over the citation graph, rooted at the primary work: the base case is direct work-to-bibliographic-record edges, and the recursive case re-enters the graph by matching a reached record’s normalized title against another owned, non-deleted work’s title, then continues from that work’s own edges — capped at a traversal depth of four and cycle-guarded via an accumulated work-id path. This is the literal implementation of the project’s documented, honest limitation that transitivity (“Kant leads to Hume”) only surfaces when the intermediate work is itself in the reader’s own uploaded library, never a claim about the wider field’s citation graph.

roadmapGraph.ts’s buildRoadmapGraph(userId, rootWorkIds, options) composes the canonical buildGraph() payload with per-root roadmap candidates and the pure, package-level mergeRoadmapsAcrossRoots, then joins the resulting ranked items back onto graph node ids via a documented precedence chain: an exact bibliographic-record match first, then any record collapsed into that item during duplicate-merging, then the item’s own matched owned work, and only as a last, explicitly-flagged fallback, a normalized-title match against a precomputed label index — the roadmap’s own “why this, here” explanation surfaces this fallback basis so a rare homonym mis-merge stays inspectable rather than silent. An item matching no graph node at all is simply skipped from the annotated graph (it remains in the plain roadmap list) — an accepted, honest gap rather than a forced, potentially wrong match.

getLibrary(userId, options) reads from the durable, cross-run learning_resource/resource_role tables (populated only once a modern-pipeline analysis run has established a work’s canonical identity), returning an honestly empty shelf — not an error — for a user whose works have no identity yet. Credibility for a (work, resource) pair is joined by matching a resource’s normalized key scoped specifically to the user’s own work ids, deliberately avoiding a raw string comparison that could compare against an unrelated identity. The same Phase-20.6 canonical-collapse logic seen in graph.ts is applied again here, at the Library-list level, so a book, its review, and its second edition present as one entry with the others attached beneath it. librarySearch.ts exists as an entirely separate, dependency-free module purely so a client component can safely import real (not merely typed) search-matching logic without dragging the database driver into the browser bundle — a bundle-boundary problem documented explicitly in the module’s own header comment. Its hasReaderLevelSignal(items) function encodes a candid, dated observation: as of the writing of that module, essentially every resource_role row in production carries a null reader level, so a level filter over entirely-null data would be a mathematically correct no-op, not a defect — the Library UI accordingly only renders the level filter when this function returns true, replacing it with an honest inline note otherwise, and the filter will reappear automatically the moment any future write path actually sets a real level, with no code change required.

getPublishedEdition(documentId) assembles the entire published-edition reader payload in one batched, largely Promise.all-parallelized read: pages and blocks, authorial and AI-generated notes with claim-level evidence, research resources with credibility, canonically-grouped work display, resource relations, the three passage-annotation buckets (anchored, whole-work, and reader-hidden), term variants, and apparatus. A documented de-duplication step computes a footprint of every modern, block-anchored apparatus entry and filters the legacy footnote table down to only entries with no modern equivalent, so a v4-run document never shows the same authorial footnote twice under two different representations.

answerRagConversation(input) loads the owned conversation, persists the user’s question, retrieves owner-scoped chunks, generates an answer (gated behind an explicit provider-enabled flag, a per-response cost cap, a shared daily spend cap, and a latency cap, degrading to the deterministic fallback on any failure), persists the assistant message and its citations, and — deliberately after the answer has already been generated, so the shared cost pool reflects the Socratic answer’s own spend first — kicks off (without awaiting) the competency-inference pipeline as a background promise the caller races against a short timeout. currentRagSpend(userId) is exported specifically so competencyData.ts can gate its own spend against the same combined daily total — the two chat sub-features deliberately share one pool rather than each getting an independent budget.

processCompetencySignals(input) in competencyData.ts wraps everything in a try/catch that always resolves rather than throws (a competency-inference failure must never fail the underlying chat answer); merges the always-on deterministic detector’s findings with the gated model’s findings (the detector’s result for a given target always wins over the model’s, since it is free and deterministic — the model only fills gaps the detector didn’t find); suppresses any signal the user has already explicitly undone earlier in the same conversation; enforces a daily cap on applied writes, not raw detections, silently dropping signals beyond the remaining allowance rather than queuing them; and, for each surviving signal, applies it transactionally alongside its own audit-ledger row, checking the mastery-source precedence chain first and recording a skipped_precedence ledger entry (never overwriting a stronger existing signal) when it should not apply.

deletionEffects(userId) supplies the real, database- and Storage-backed implementation of the pure @ice/deletion state machine’s injected effects interface described in §8.1 — including the explicit deletion of polymorphic graph_edges referencing the work, since Postgres cannot cascade a relation that carries no real foreign key. retryPendingCleanups(userId) defends against a subtle correctness trap: before resuming a persisted, retryable cleanup record, it checks whether the underlying work has, in the meantime, actually been restored by the user — if the work row still exists and is no longer trashed, the stale cleanup record is deleted rather than resumed, so a restored work is never accidentally re-deleted by a lingering retry. purgeExpiredTrash(userId) runs opportunistically at the top of the trash page’s own load, rather than as a scheduled job — the project’s own documentation states this was a deliberate choice, since trash is pure web CRUD with no worker or AI involvement that would otherwise justify standing up new scheduling infrastructure.

writer.ts is pure, framework-free domain logic with no database import at all: a strict Zod schema for the ProseMirror document shape (bounded to 10,000 blocks and one million total characters); CSL-JSON normalization tolerant of many real-world field-name aliases; MLA in-text and Works Cited formatting; and hand-rolled BibTeX and RIS parsers.

writerExport.ts builds real binary exports without any external PDF or DOCX library dependency: createWriterPdf hand-assembles a minimal, spec-valid PDF from scratch (escaping special characters, wrapping long lines, building the content-stream operators, and manually writing the numbered objects, cross-reference table, and trailer with correct byte offsets); createWriterDocx builds a minimal, valid Office Open XML document by hand-implementing an uncompressed ZIP container — including its own from-scratch CRC-32 implementation — the module’s own comment noting that “uncompressed ZIP is enough for a portable, standards-compliant DOCX.”

auth.ts configures Auth.js v5 with a Credentials provider and JWT sessions, a choice forced by the fact that Auth.js only auto-wires database sessions for OAuth-adapter flows, not Credentials — session revocability is instead achieved through the users.sessionVersion counter checked on every request inside the jwt callback: if the currently-stored session version no longer matches what the token was stamped with (for instance, after a password reset incremented it), the callback returns null, invalidating that JWT immediately despite JWT sessions having no natural server-side revocation mechanism of their own. auth.d.ts is a pure type-only file fixing a genuinely subtle TypeScript footgun: augmenting the wrong re-exported module (rather than the module that actually declares the Session interface) silently creates an unrelated interface that never merges with the real one, leaving session.user.id typed as possibly undefined everywhere with no compiler error to catch it.

auth-service.ts implements registration (silently no-op on an already-registered email, so registration can never be used to enumerate existing accounts), email verification, and password reset — the reset path is where sessionVersion is actually incremented, in the same database update as the new password hash, which is exactly the mechanism auth.ts’s jwt callback checks.

actions.ts implements the corresponding Next.js Server Actions, each applying the same pre-authentication rate limits as their sibling API routes and redirecting to a deliberately generic, indistinguishable error state on either a validation failure or a rate-limit rejection — anti-enumeration by construction, not merely by convention.

apiRateLimit.ts’s enforceUserRateLimit is a database-backed, cluster-wide, fixed-window counter for authenticated routes, with a deliberate fallback: if the upsert throws an error matching a “relation does not exist” pattern (the additive rate-limit migration has not yet reached this particular database), it degrades to an in-process, per-instance counter rather than turning every rate-limited route into a server error during a rolling deployment — logging a one-time warning per scope so the fallback state is still visible without spamming.

preAuthRateLimit.ts is a separate, purely in-memory limiter for the handful of routes that have no authenticated user id to key a limit by (registration, password-reset request, password reset itself), keyed instead by a platform-set client IP header that cannot be spoofed by the requester, with an explicit, documented eviction strategy once its tracked-key count crosses ten thousand — first evicting anything provably stale (older than two hours), and only if that alone is insufficient, evicting the oldest entries down to ninety percent of the cap, an accepted best-effort trade-off the module’s own comment records candidly as something a real, durable, cluster-wide table would eventually need to replace.

citationResolver.ts provides a lighter, Writer-mode-specific DOI/ISBN/title lookup against Crossref and Open Library, independent of the worker’s own bibliographic pipeline. annotations.ts loads a legacy document’s annotations joined to their resolved bibliographic target, computing an isHeuristic flag from the stored prompt version so the UI can honestly distinguish a real model verdict from the deterministic fallback. graphEdgeCategory.ts’s deriveEdgeCategory fills in a missing relationship category for exactly two, and only two, edge-writing code paths that are provably unambiguous by construction — the module’s own comment stresses this is an exhaustively verified mapping, not a keyword heuristic, and warns explicitly against extending it by guessing. graphExpansion.ts computes the same cost-estimation function the worker’s execution path independently re-derives, so a client-shown preview and the server’s actual guardrail can never silently disagree. ragConversationClient.ts’s loadOrCreateRagConversation fixes a real, documented production defect where a localStorage-persisted conversation pointer that stopped resolving (for instance, because the owning test account was deleted during routine cleanup) permanently disabled the chat panel for that browser; the fix treats a 404 on a stored id exactly like “no id was ever stored,” transparently creating a fresh conversation, while still allowing any other kind of error (a genuine network failure, an authentication failure) to propagate untouched rather than being silently treated as “gone.” v4Backfill.ts computes a deliberately read-only, dry-run cost forecast for backfilling existing documents through the newer v4 pipeline, explicitly never enqueuing any real work itself. cost.ts groups a run’s existing AI-usage log rows by stage and task for the reader’s itemized cost breakdown, adding no new tracking of its own. mail.ts, like the AI-provider layer, follows the adapter-plus-honest-fallback pattern — falling back to console logging, never silent failure, when no real email provider key is configured. works.ts’s two ownership-resolution helpers, getOwnedDocument and getOwnedWork, are the single most repeated pattern in the entire API layer, discussed further in §12.

Ownership and IDOR posture. Almost every protected route resolves ownership through getOwnedDocument(workId, userId) or getOwnedWork(workId, userId) (§10.10), both of which return null — never a distinguishable “forbidden” value — for a resource that exists but belongs to someone else. Every caller uniformly turns a null into a 404, so a request can never distinguish “this does not exist” from “this exists but is not yours.” The same posture governs the admin area (requireAdmin() returns a plain 404 for a non-admin, not a 403, so the very existence of the admin surface is not revealed) and every phase-gated feature (a disabled feature 404s rather than 403s, so its existence is not revealed either).

Rate limiting. Two independent mechanisms are used depending on whether a route has an authenticated user to key on: enforceUserRateLimit() (database-backed, cluster-wide) for authenticated, cost-bearing, or abuse-prone routes, and preAuthRateLimit() (in-memory, IP- or email-keyed) for the handful of pre-session routes. Composed guards — requireWriterApiUser(), requireRagApiUser() — bundle a feature-flag check, an authentication check, and a rate-limit check into one reusable call.

Feature flags from @ice/config gate whole route families (Writer, RAG, cross-library graph expansion, conversational competency) to a 404, not a 401 or 403, when the corresponding release flag is off.

The Auth.js catch-all route re-exports the framework’s own handlers directly. register, request-reset, and reset-password each apply the pre-authentication rate limits described in §10.9, and each is deliberately engineered so its response shape is identical regardless of whether the underlying account exists — registration and reset-request both always return a generic success acknowledgment, closing the account-enumeration channel that a differing response would otherwise open. verify-email is a browser-navigated redirect endpoint, not a JSON API — it always redirects, either to a login-success state or a verification-error state, never returning a raw JSON error to a clicked email link.

GET /api/works lists the caller’s own ready works. DELETE /api/works/[workId] soft-deletes (sets deletedAt, no cascade, no Storage touch — a reversible, idempotent action). GET .../status is the reader’s polling endpoint, computing an honest stalled flag on read (never via a scheduled sweep) by comparing the latest run’s heartbeat age against a documented threshold. POST .../analyze and POST .../confirm gate concurrent double-runs and, in confirm’s case, contain a documented D-23-6 fix: it checks whether a processing_runs row already exists for the document (a data-driven, not environment-variable-driven, signal that the modern pipeline already ran its own analysis) before deciding whether to enqueue the legacy analyzer at all — enqueuing it unconditionally would silently wipe an edition document’s richer, run-scoped citation set. POST .../reprocess is gated to edition-pipeline documents only and delegates its entire concurrency/idempotency decision to the pure planReprocess() function described in §8.5, translating its four possible outcomes (reuse, recover, enqueue, conflict) directly into the HTTP response. POST .../purge and POST .../restore implement the honest, retryable permanent-deletion contract from §8.1/§10.6 — a purge response is only ever reported successful when every Storage byte is actually confirmed gone.

The reader/* sub-routes (highlights, bookmarks, notes, annotations, passage-annotations, position, terms) implement the full CRUD surface for the annotated reading experience, each independently ownership-scoped; passage-annotations in particular resolves ownership through a run-to-document-to-work chain rather than a direct owner column, since passage annotations belong to one document, not one user, by their own schema shape. The upload/* sub-routes (init, complete, proxy, and a legacy single-request upload) implement the multi-phase, security-hardened upload pipeline: init reserves quota and creates a signed direct-to-Storage upload URL; complete re-verifies the actual Storage-reported byte size rather than trusting the client’s own declared size — closing a documented, real quota-bypass path — before atomically flipping the document into the processing queue, guarded so a duplicate completion call cannot double-enqueue; proxy is a same-origin fallback for browser environments that block the direct signed-URL PUT, itself bounded to a small byte cap consistent with the platform’s own request-body limit.

The roadmap and curriculum routes are thin wrappers around the pure computation packages described in §6 and §10.2 — nothing here persists anything beyond the durable user-authored profile inputs. POST .../roadmap/item performs a partial, field-by-field upsert (understanding score, reading status, and/or a manual override are each independently optional) with no underlying database unique constraint, resolved instead at the application level as a find-then-update-or-insert. GET .../diagnostic opportunistically infers a weak mastery score from other already-completed prerequisite works, but only ever fills a genuinely empty slot — an existing rating, of any source, is never overwritten by this inference. Both api/graph and api/works/[workId]/graph support an identical ?layout=roadmap opt-in that switches the response to the roadmap-annotated projection from §10.2 without changing the default, unannotated response shape at all — a deliberate backward-compatibility guarantee for existing clients and tests.

GET /api/library performs the server-authoritative search described in §10.3 — search is never merely a client-side filter over an already-downloaded list. POST /api/library/[resourceId]/status upserts reading state against the shared, un-owned resource catalog, checking only that the resource genuinely exists (not that the caller “owns” it, since the catalog itself has no owner column).

POST /api/rag/conversations/[id] streams its response as Server-Sent Events: a user event echoing the persisted question, tokenized delta events building the answer incrementally, citation events per resolved source, up to several seconds’ worth of competency events (raced against a timeout so a slow or failing competency pass can never delay or break the underlying chat answer), and a final done event. POST /api/rag/competency-signals/[id]/undo is engineered to be genuinely idempotent (undoing an already-undone signal is a safe no-op) and fails closed with a conflict response, rather than silently overwriting, if the signal’s current state is neither “applied” nor “already undone” — restoring a stale prior value over a newer one would itself be a data-integrity bug.

Every route resolves ownership through project- and document-scoped helpers that require both a project and (where relevant) a document id to match the caller, so no route ever grants access by a bare document identifier alone. The export route streams a genuine binary DOCX or PDF produced by the hand-rolled generators in §10.7, with Cache-Control: no-store so a generated export is never inadvertently cached. Citation import supports four distinct input shapes (an owned Library resource, a DOI/ISBN/title lookup, or pasted BibTeX/RIS text) behind one discriminated-union request body.

The one route here, pipeline-v4/backfill-forecast, is deliberately read-only: it returns a cost/scope projection for a hypothetical v4 backfill and never itself enqueues any real reprocessing — a genuine dry run, not a trigger with a misleading name.

Structural PDF extraction degrades silently to a flatter fallback when GROBID is unavailable. The GROBID adapter returns a bare null — no error, no distinguishing signal beyond a downstream structure-limited label — on an unset service URL, an authentication failure, a timeout, or any non-success HTTP response, and no code path retries a failed call, alerts an operator, or surfaces the degradation to the reader beyond that one internal flag. The labeling discipline itself is genuine and honest — a structure-limited block is never presented as a page-anchored GROBID block — but the practical consequence is that citation and footnote anchoring quality for any given document depends on the availability of one external service at the moment of ingestion.

Roadmap “transitivity” is capped at the reader’s own uploaded library, not the wider scholarly graph — already discussed in §6.1/§10.2, and worth restating plainly as a limitation: the roadmap is a materially smaller object than “the citation graph of the field,” and any claim about the system surfacing a reader’s implicit intellectual context inherits this cap directly.

A discrepancy exists between the application’s privacy-page copy and its actual code. The privacy page states that deleting an account removes all uploaded files, extracted text, annotations, and derived data, and that a user can export their notes, roadmap, and bibliography. A targeted search of the codebase for a self-service, account-wide deletion route or a bulk personal-data export endpoint found no matching implementation — what is actually implemented is per-work soft-delete/restore and a permanent-delete state machine scoped to one work at a time, described in §8.1 and §10.6, not an account-wide action, and no export feature exists outside Writer’s own per-project document export. This is a genuine documentation-versus-implementation mismatch in user-facing legal copy, worth naming directly rather than silently assuming the copy is accurate.

The automated CI test suite runs against an older pipeline version than production actually serves. The continuous-integration configuration pins the analysis pipeline to its second version for its automated end-to-end run, while the project’s own log states that production has since been promoted to the fourth version as its default. This is recorded as a deliberate, documented trade-off, not an oversight, with newer-pipeline-specific behavior instead covered by point-in-time manual canary runs and unit tests — but it means a green automated test run is evidence for the deterministic subset of behavior it actually exercises, not for every version-specific feature production currently runs.

Only a minority of the full end-to-end test suite runs automatically on every push; the remainder require a live background worker process, live Supabase Storage, and/or live external bibliographic and AI APIs, and are documented as run manually rather than gated in continuous integration — an explicit, reasoned trade-off around cost and external-service availability, not an oversight, but one the project’s own log states plainly rather than implying full automated coverage.

Several research-discovery source providers are architecturally present in the codebase but not confirmed configured in the current production deployment. Five keyless scholarly sources are always enabled; four further providers (a web-search provider, a video platform, and two social-media adapters) each require an API key or access token that, as of the project’s most recent recorded audit, had not yet been rotated and configured in production — reported honestly as disabled, not silently skipped or faked, whenever the corresponding credential is absent.

Explicit-citation resolution has a measured, non-trivial miss rate, concentrated in older or book-form citations. The project’s own log reports a specific, measured recall figure on its private gold-evaluation fixture (improved substantially over the course of several targeted fixes, including the form-aware provider-ordering and Open Library trailing-year fixes described in §5.2), with the remaining misses attributed to genuine source-coverage gaps for older monographs at specific external providers, not an extraction-logic defect — this bounds any claim about citation-extraction completeness to the specific fixture it was measured against, not a general accuracy guarantee.

Authorization is enforced per-route rather than by a single centralized middleware layer. Every protected page and API route calls its own ownership/session-check helper individually rather than passing through one framework-level gate; the project’s own audit confirmed, by manually reviewing every route, that each one does in fact perform its own check, so this is not currently an exploitable gap — but it is a structural pattern where a newly added route that forgot to call the shared helpers would not be caught by any framework-level guarantee, only by code review discipline.

Feature-flagged capabilities default off, and their production-live status should never be assumed from code presence alone. Writer mode, the paid cross-library graph expansion, the interactive reader, canonical Library identity, the Ask Library chat, and conversational competency designation are each gated behind an independently addressable release flag defaulting to off — the project’s own log records at least one real incident where a local development environment had never set any of these flags, silently disabling an entire feature for every local end-to-end test run without anyone noticing until the gap was specifically investigated.

bibliographic_records is an unscoped, append-only shared catalog that orphans on deletion — an accepted trade-off at the project’s current single-user cost scale, discussed already in §3.5, with a periodic orphan-row cleanup recorded as future, unbuilt work rather than a currently scheduled task.

No shared packages/ui was ever scaffolded — reader, roadmap, and graph components live directly inside the web application rather than in an independently reusable package, a deliberate scoping decision (there is currently exactly one consuming application) rather than a defect, revisited only if a second application ever needs to share these components.

A repository-wide search for literal TODO/FIXME/HACK markers in application source found none of substance — every “stub” reference that did turn up refers to the intentionally-labeled deterministic heuristic classifier fallback, not an unfinished implementation. This is worth stating for methodological transparency: the genuine gaps recorded here come entirely from the project’s own structured status log and from direct code-versus-documentation cross-checking, not from code comments admitting incompleteness.

The project maintains: a substantial unit-test surface for every pure package described in this report (packages/roadmap’s ranking logic is tested against the plan’s own worked Heidegger and Vico acceptance cases; the heuristic AI classifier is tested with no network or key required, by construction; packages/consistency’s nine checks and packages/deletion’s state machine are both tested as pure functions against injected effects); a large integration-test surface at the worker layer covering extraction, passage annotations, citation integrity, RAG indexing, queue recovery, canonical-identity merging, and consistency-repair application; and an end-to-end browser test suite (over fifty specs at the time of the project’s most recent audit) split explicitly into a smaller continuous-integration-safe subset and a larger, manually-run full-stack subset requiring live external services. Manual VoiceOver (screen-reader) verification is recorded, repeatedly and explicitly, as never having been completed — an acknowledged, standing accessibility-verification gap distinct from the project’s substantial automated axe-core accessibility scanning, which the project’s log reports finding zero WCAG 2A/2AA violations across every route swept as of its most recent accessibility pass.

Across its whole recorded history, the project exhibits a consistent pattern worth naming explicitly, since it recurs in nearly every package documented in this report: a defect is found (often via a real production incident, a load test, or an adversarial fixture), the root cause is diagnosed and named precisely (a specific race condition, a specific library bug, a specific silent-fallback path), a fix is implemented that is deliberately narrow and evidenced rather than broadly defensive, and the incident and its fix are both recorded in the project’s own durable log rather than only in a commit message. This pattern is visible in the GROBID contamination guards (§4.3), the pg-boss connection-string TLS fix (§8.5), the citation-role foreign-key race recovery (§9.4), the camera-framing trigonometry fix (§13.2), and dozens of smaller instances throughout — a project culture of diagnosing before patching, and of recording the diagnosis alongside the patch, rather than merely making a symptom go away.

The Introduction follows a six-part structure. It opens with the general problem of background-knowledge dependency in reading: drawing on four held secondary sources (a matched-pairs study of expert and novice teachers, a national consensus synthesis on learning science, a physics-education monograph reporting Ericsson, Krampe, and Tesch-Römer’s deliberate-practice findings, and Willingham’s review of critical-thinking research), it establishes that organized background knowledge measurably differs from disorganized information, and — citing Willingham’s own transfer-study data directly — that merely supplying background structure does not, by itself, equalize performance between a reader who has it organized and one who does not; only nineteen percent of subjects in the reviewed transfer studies spontaneously recognized an earlier problem’s relevance to a new one, and even being told outright raised the rate only to thirty-five percent. The paper is explicit that none of these four sources studies reading, annotation, or an AI-assisted tool at all — any connection to a reading application is an analogy the manuscript itself constructs, not a claim the sources make.

Palimnote is then introduced directly, in its second paragraph, described at the level of its actually-implemented mechanisms (provenance records, real-lookup-only citation resolution, the ten relationship categories, on-demand roadmap computation, retrieval-grounded chat) — with an explicit caveat that none of this is evidence the system adapts to a reader’s true competence or supplies the map an expert brings; that stronger claim is named as “a stated design aspiration for a future version of the product, not a claim made here about the built system.”

A third paragraph introduces the generation-reliability problem — the fact that a system generating content for a reader (whether a tutoring question or an attached annotation) introduces a difficulty a reader cannot independently check at the moment of generation — drawing on both a comprehensive hallucination survey and, as one named instance rather than the section’s organizing subject, the SocraticLM paper’s own framing of a “Question-Answering” paradigm that leaves students passively provided with answers. A fourth paragraph previews Palimnote’s response — the built mechanisms already described — and a fifth states the paper’s thesis, explicitly marked as provisional pending ratification of the underlying thesis-selection process (discussed in §17.4 below). The section closes by previewing the remainder of the manuscript’s structure.

Background is the manuscript’s literature-review section, organized into four subsections corresponding to the four literatures the paper draws on, each drawing sources only for the specific construct it establishes, with an explicit disclaimer at the top that none of this literature studies a reading-and-annotation tool of the kind Palimnote is.

Trust calibration and the limits of disclosure. Anchored on a comprehensive hallucination-in-natural-language-generation survey, which divides hallucination into content that contradicts a source and content that cannot be verified from it, and states that reliable expression of a model’s own uncertainty remains, in the survey’s own words, “a recognized and still largely unsolved research problem.” The section’s sharpest complication — introduced here and returned to at length in the Theoretical Argument — is Kizilcec’s field experiment on an algorithmic peer-assessment interface, which found trust in the system was not monotonic with the amount of disclosed detail: a procedural explanation restored trust after an expectation-violating outcome, but adding the underlying raw data on top of that explanation did not sustain the improvement — it reopened the original trust gap. A second study, on AI-assisted credibility judgments, found that attaching any natural-language explanation to an automated verdict substantially increased reliance on that verdict regardless of whether the verdict was actually correct — users, in the study’s own words, “could not discern whether the AI directed them towards the truth.” A third source, a comparative study of how historians, professional fact-checkers, and undergraduates evaluate unfamiliar websites, found that the fact-checkers — who consistently outperformed the other two groups — did so chiefly by leaving a site to check independent corroborating sources rather than by scrutinizing the site’s own internal features, in real tension with any design (including Palimnote’s own) that relies on internal, per-source metadata display as its primary route to calibrated trust. A practitioner-validated set of eighteen human-AI interaction guidelines is cited for two directly relevant recommendations — “make clear what the system can do” and “make clear why the system did what it did” — while also reporting that the second of these was, across the very products the guidelines were validated against, one of the most frequently violated despite active research interest in explanation generally.

Cognitive load and the case for guided sequencing. Anchored on Sweller’s own cognitive load theory, whose supporting studies are drawn almost entirely from mathematics, geometry, and technical instruction — the paper is explicit that this literature neither tests nor discusses sustained narrative or humanistic reading, which is Palimnote’s actual use case. A crucial distinction the paper insists on: cognitive load theory’s own ordering criterion is a passage’s intrinsic element interactivity, not a dependency relation between separate works — a materially different basis from what Palimnote’s roadmap actually orders by. A separate strand establishes the scaffolding construct’s defining feature: Wood, Bruner, and Ross’s originating observational study of tutors assisting children describes support that is withdrawn gradually and non-automatically, tracked to the child’s own demonstrated competence — and later practitioner synthesis states the requirement directly: assistance “must gradually be reduced (or faded) so that the student ultimately learns to perform well independently.” The formal expertise-reversal effect — guidance calibrated for a novice becoming redundant or actively harmful once a learner’s own prior knowledge is sufficient — is documented both in Kirschner, Sweller, and Clark’s extension of cognitive load theory and, at its own primary source, in Kalyuga, Ayres, Chandler, and Sweller’s naming of the phenomenon.

Extended cognition as a bounded lens. Anchored on Clark and Chalmers’s classic argument, quoted at its own most consequential line — “there is nothing sacred about skull and skin” — and its four jointly-necessary criteria for treating an external resource as constitutive of a standing belief: constancy of use, easy accessibility, automatic endorsement of retrieved information without further scrutiny, and past conscious endorsement at the moment of storage (a fourth criterion the argument’s own authors flag as the most contestable). The paper is careful to frame this as a philosophical individuation condition for the concept of belief, not an empirically validated design instrument, and notes the argument itself predates retrieval-augmented generation and every AI system the manuscript describes entirely — applied here only as an analogy, never as a framework the source itself endorses.

Metaphor as a methodological guardrail. Drawing on a single edited volume containing an unresolved, internal three-way disagreement — Boyd arguing that computational vocabulary like “information processing” did genuine, theory-constitutive work in early cognitive psychology rather than serving as mere expository shorthand; Pylyshyn, in the same volume, denying this and holding that such vocabulary describes mental activity literally; and Kuhn, writing in direct dialogue with Boyd, arguing that repeated exposure to a metaphor can manufacture a reader’s sense of resemblance rather than merely report one already there. The manuscript states its own resulting discipline directly: any description of Palimnote’s retrieval pipeline in memory-adjacent language is treated throughout as an expository analogy, named as such wherever it appears, never as an unhedged equivalence between what a retrieval system does and what remembering is.

Methods opens not with a generic methodology statement but with the paper’s own origin narrative: the project began from an observation about SocraticLM, a large-language-model tutoring system, whose own paper defines Socratic teaching in a narrow, two-part way — dialogic exchange and probing questions — citing only critical-thinking-pedagogy literature, not classical or philosophy-of-education scholarship. Peer-reviewed classics scholarship on the Socratic elenchos, the paper reports, indicates the method is not reducible to this surface mechanics — it combines a repeatable question-and-answer form with a further strategic aim (testing an interlocutor’s doxastic coherence, not the truth of any single answer) and a substantive epistemological commitment to definitional knowledge, per Benson’s chapter in The Cambridge Companion to Socrates — and the same scholarship documents a genuine, unresolved scholarly debate over whether the elenchos is best read as capable of constructive results or purely refutative, on which the manuscript itself takes no position. This scope gap, the paper states, motivated a broader survey of the pedagogical, cognitive-science, and human-computer-interaction literatures the manuscript actually draws on.

The section then states the paper’s own citability discipline directly: every candidate source was logged with bibliographic metadata confirmed against the document itself, classified against a four-part citability rule (formal peer review, primary-source status, acceptance-for-publication, or scholarly synthesis meeting comparable provenance standards), with every candidate quotation independently, mechanically checked as a verbatim substring against the source’s extracted text before any claim resting on it could be treated as usable. Literature search proceeded across nine disciplinary areas, each independently reviewed by a second reviewer before its findings were treated as usable — this entire discipline, and the governed multi-agent process behind it, is the subject of §17–§19 below.

A distinctive subsection, “Process Architecture as a Methodological Contribution,” states that the manuscript’s own development instantiates, at manuscript scale, the same design principles Palimnote applies to reading support: verifiable offloading, multiple-agent parallel processing, and user-directed iteration. It discloses that this research project operated as one half of a supervised two-agent structure — two parallel instances of the same underlying model, one building the Palimnote application itself in a separate repository, the other conducting this research program — feeding each other through standing loops: research findings informing the application’s own refinement, and the application’s evolving, real state informing this manuscript’s implementation-status claims, so that a claim here about what the system does tracks the live application rather than an aspirational design. Every output from either side, the section states, passed through the author’s own review-and-correction gate before adoption. A further subsection frames the AI role in this process explicitly as an indexing mechanism — drawing a documented lineage from Otlet’s proposal to externalize and interlink recorded knowledge, through Garfield’s demonstration that citation indexing surfaces structure a subject index misses, de Solla Price’s finding that citation networks form a discoverable non-random structure, and Small’s operationalization of citation relatedness as a computable measure — rather than as an autonomous reasoner, with every such output mechanically verified (an exact-match substring check, or resolution against an external authoritative source) before being retained.

A “Conservative Novelty Statement” subsection is unusually candid about what the paper does not claim as new: multi-agent LLM pipelines for structured cognitive tasks, hierarchical prompt-chaining for long-form co-writing, and recursive-decomposition multi-source reasoning are each named as already-published, established practices this project does not claim to originate. What is offered as new is the specific combination and discipline — a verification ledger checked claim-by-claim and quote-by-quote before any material is treated as citable — not any one constituent mechanism alone. A final subsection, “A Single-Operator Capability Argument,” frames the process as a structured division of labor (mechanical, verifiable work offloaded to AI agents; judgment — acquisition direction, interpretive choices, final argument decisions, and verification of every quoted string — retained by a single author), analogized explicitly to Engelbart’s 1962 vision of machinery that extends human capability without displacing human direction and Licklider’s 1960 account of task allocation by comparative advantage — while stating plainly that this capability argument assumes, rather than removes, a demanding precondition: it describes what a user with sufficient domain competency can direct and verify, not a method that functions independent of that competency.

This section is the manuscript’s direct description of the built system, organized by subsystem, and is written under a strict implementation-status discipline: every capability is described as implemented, partially implemented, planned, proposed, or explicitly unknown where the project’s own records conflict — the same five-value vocabulary used throughout the project-analysis documents summarized in Part I of this report. It covers, in order: ingestion and structural extraction (including the corrected, corroborated account of the GROBID service actually running as a private, authenticated cloud instance in production, rather than strictly locally as an earlier document had stated, with a footnote explaining the discrepancy was resolved in favor of the more recent, corroborated record rather than silently picking a side); citation extraction, resolution, and classification (with explicit numeric ceilings on lookups, candidates, and inspected resources, and an explicit statement that the pipeline is single-hop from an uploaded work to its own citations’ metadata, never a multi-level recursive traversal into a cited work’s own bibliography — a distinct and narrower claim than the reading roadmap’s own, separately bounded, four-level-deep recursive traversal); the provenance and confidence discipline; independent-dimension credibility scoring; the annotated reader and its contestability controls; rule-based roadmap sequencing (explicitly non-adaptive, with the one mechanism that could carry a genuine competency signal — the conversational competency feature — noted as built into the schema, live in production, but sitting disabled behind two feature flags and invisible to any current user); the knowledge graph; retrieval-grounded chat (described precisely as Socratic-method-informed and locally authored, with the manuscript stating directly that no SocraticLM training data, weights, or weight-diff artifacts are used); and Writer mode, closing with a broader observation the section elevates to a design property in its own right — that a feature’s presence in the codebase is not evidence of its presence for any current user, since several substantial capabilities ship behind independently addressable release flags defaulting to off.

This is the paper’s analytical core, developing the mechanism-to-construct correspondence in full. It opens by restating the argument’s evidential discipline precisely: every mechanism examined is one that is actually built and inspectable in the codebase, read against constructs an already-established literature associates with real effects on trust, reliance, verification behavior, and learning — never assessed by any measurement this project itself performed.

The “Provenance, Verifiable Offloading, and the Constructs of Calibrated Reliance” subsection argues that Palimnote’s central design commitment is a cluster of mechanisms that together offload mechanical work from the reader while structurally withholding automatic acceptance of what is offloaded — and connects this directly to the hallucination survey’s own framing of the general problem these mechanisms answer to, and to the human-AI interaction guidelines’ own normative recommendations. It is explicit, immediately afterward, about what this correspondence does not license: no source studies an annotation tool’s provenance display specifically, so the claim licensed is narrower — that the mechanisms instantiate the design elements this literature associates with calibrated reliance, argued at the level of what the code does, not at the level of what any reader has been shown to experience.

“Why Palimnote, and What the Mapping Adds” is the section’s most self-critical passage. It concedes, without qualification, that individual mechanisms of this kind are not novel to this project alone — a reference manager extended with a grounded-retrieval plugin, a literature-search assistant, an established citation apparatus, or a document-grounded notebook product could each already implement one or more of them — and states directly that being fully built and inspectable is not, on its own, a property that distinguishes Palimnote from those comparable tools, since their own codebases and documentation are equally inspectable. What the combined mapping adds, the section argues, is a genuine internal tension a single-literature survey would not by itself reveal: the very mechanisms that make the trust-calibration reading strongest are the same mechanisms that make the extended-mind reading weakest, discussed at length in the Synthesis subsection.

“The Strongest Objection: Non-Monotonic Transparency” treats Kizilcec’s field-experiment finding as primary, peer-reviewed evidence — not merely a caveat — that the naive intuition “more disclosure straightforwardly builds more trust” is false in at least one controlled, high-stakes setting, while carefully distinguishing Kizilcec’s comparative, outcome-linked, felt-discrepancy-triggered disclosure manipulation from Palimnote’s own constant, low-stakes, uniformly-attached provenance record — the two are not the same design object, and the manuscript states plainly that it does not import the “amount of transparency” finding as though it transferred directly.

“The Sequencing Co-Pillar: Order Without Fading” maps the roadmap and curriculum onto the guided-instruction literature’s constructs while insisting on a precise distinction: the roadmap’s ordering basis is bibliographic prerequisite dependency, not the intrinsic element-interactivity load basis cognitive-load-theory sequencing actually orders by — this is, the section states plainly, “a mapping onto the sequencing sense of the construct, not a claim that Palimnote’s ordering criterion is the ordering criterion the cited literature uses.” The construct the pillar borrows carries a defining feature Palimnote’s mechanism structurally lacks — fading — and the section names the expertise-reversal effect directly as a documented risk the fixed, non-adaptive tiers have no mechanism to detect.

“Extended Mind as a Criterion-by-Criterion Lens” is where the paper’s self-reported partial failure lives. Of the four Clark-and-Chalmers criteria, two carry the whole of the analysis: automatic endorsement and past conscious endorsement. On automatic endorsement, the section argues that every trust-calibration mechanism already described exists precisely to interrupt unexamined acceptance of generated output, not to smooth it into unreflective use — meaning the design withholds automatic endorsement by deliberate structural choice, which under the extended-mind criteria registers as non-satisfaction of the one criterion that would make the stronger extended-mind reading available at all. On past conscious endorsement, by contrast, the paper finds “the cleanest fit anywhere in this analysis”: the system’s own verification-status field and its approve/dispute/reject/hide/edit actions operationalize a discrete, timestamped act of endorsement rather than merely assuming one. The section is explicit that these two readings — the trust-calibration reading and the extended-mind non-satisfaction — are not evidence for each other; they are two re-descriptions of the identical mechanisms from two different theoretical vantage points, and neither is offered as an empirical test of anything.

“The Metaphor Guardrail: Retrieval Described, Not Memory Claimed” applies the Background section’s Boyd/Pylyshyn/Kuhn disagreement concretely, citing the technical description of retrieval-augmented generation itself (Lewis et al.) to state that the manuscript accordingly never describes Palimnote’s retrieval pipeline as working like human memory or as remembering — treating it strictly as a concrete object for discussing analogies and disanalogies, never as evidence the pipeline shares a cognitive property with human recall.

The closing “Synthesis” subsection states the paper’s central analytical payoff directly: read together, a system that offloads mechanical lookup while withholding automatic endorsement of what is offloaded is, on Clark and Chalmers’s own terms, simultaneously a design-level instantiation of verifiable-offloading constructs and a design that declines the one criterion that would make it a stronger extended-mind reading of the reader’s own cognition — two co-descriptive readings of the same mechanisms, neither one corroborating the other.

Limitations is organized around several distinct kinds of bound on the paper’s own argument, each stated without softening. The roadmap and missing-link detection are bounded to the user’s own uploaded corpus. The rule-based sequencing carries the documented expertise-reversal design cost, with no mechanism to detect it, and no mechanism anywhere in the system measures or targets cognitive load at all, so any load-related framing in the paper is stated as “a hypothesis the design invites rather than a quantity the system tracks.” More fundamentally, every mechanism-to-construct correspondence in the paper is a design-level analysis, not a measured effect — no component of Palimnote has undergone empirical evaluation with readers, and production usage to date is the project’s own verification runs, not an established user base. The literature the paper itself uses to describe the provenance mechanisms is candidly reported as complicating the question those mechanisms address more than resolving it — Kizilcec’s and Pareek’s findings, both introduced in Background, are jointly evidence that more disclosed detail does not straightforwardly produce better-calibrated reliance, which the paper states is “the seam this paper’s central argument must be read against.” The extended-mind lens carries the identical limitation from the opposite direction: no study located for this project tests whether real users of a digital annotation tool actually withhold or grant automatic endorsement in practice; that the design structurally declines to invite it is an affordance the paper can describe, not a behavior it has observed. The section also names, candidly, two literatures the paper’s own motivating frame depends on but has not itself acquired and verified in full: the primary expertise-canon studies the Introduction’s secondary sources themselves rest on, and the classics scholarship that would confirm or disconfirm the paper’s own comparison between SocraticLM’s operationalization and a fuller reading of Socratic pedagogy. The section closes by stating, for every one of its central mechanism-to-construct mappings, the specific empirical study that would falsify it — a controlled comparison finding no verification-behavior difference with and without the provenance display; a finding that fixed priority tiers perform no better than an unordered list for novices, or worse than no order at all once a reader’s prior knowledge is high; and evidence that readers treat the system’s stored library with the same unexamined reliance the extended-mind criteria describe.

Future Work names three categories of study the paper’s argument invites but does not itself design: a trust-calibration study testing whether the provenance display actually changes how a reader assesses reliability; a verification-behavior study testing whether a reader given a dispute affordance actually uses it; and an expertise-transfer study testing whether the roadmap and knowledge profile narrow any measurable novice-expert performance gap. It records specific, named further acquisitions that would strengthen particular claims (a still-unacquired study on whether AI explanations improve human-AI team performance beyond accuracy alone; two established credibility-research frameworks that would ground the independent-dimension credibility design in the field’s own vocabulary; a companion empirical paper to an already-held philosophical chapter on AI systems impersonating philosophers). It poses, without answering, two open design questions already discussed in Limitations and the Theoretical Argument — whether the roadmap should evolve toward genuinely contingent, competence-responsive fading, and whether the dormant conversational competency feature should be activated, explicitly framed as questions for empirical validation to settle, not implementation alone. It closes with a single, carefully bounded speculative gesture toward scale: extending the roadmap and knowledge-profile structure from one reader’s own library toward a larger, shared map linking many readers’ libraries would be, the paper states, “one small, bounded instance” of a decades-long documentation-science and citation-network-analysis research program — and immediately imports that program’s own strongest available caution against the gesture, via Bowker and Star’s argument that any classification scheme embeds contingent choices about what counts as related and whose material is well served, extended here by analogy, not asserted as Bowker and Star’s own claim, to any future, larger version of Palimnote’s own graph.

The Conclusion restates the argument’s four contributions and their explicit boundaries without introducing any new citation or claim beyond what the preceding sections already established — a discipline the section’s own construction notes record as checked directly against the paper’s own claim-tracking record. It adds one further, separately-argued claim concerning not Palimnote but the manuscript’s own production: the structured division of labor documented in Methods is offered as a design claim about concentrating a researcher’s own expert judgment in acquisition, verification, and interpretation while offloading mechanical work to constrained AI agents — itself a design claim, not a measured effect, and independent of anything argued about Palimnote’s own mechanisms. The paper’s final paragraph states its governing discipline as directly as the manuscript ever does: “the distinction this paper has held throughout is between a mechanism that exists and an effect that has been shown… Until such work is done, what stands is a mapping from mechanism to construct, offered as a way of describing, precisely and without inflation, what a specific built system does and does not warrant a reader in concluding from it.”