Architecture

Dependency direction

pangenome-range-cli ──> pangenome-range-build ──> pangenome-range-query
     │                         │                         │
     └────> upstream gbz       └────────> pangenome-range-format
                               └ - - -> gbz-base (research/oracle only)

pangenome-range-format owns the normative v1 header/root/directory and regional codecs, corruption checks, reader primitives, structural validation, storage seam, and network cost model. It has no GBZ dependency. pangenome-range-query owns canonical graph/tile semantics, comparison, and hashing. pangenome-range-build owns GBZ source access, reference anchoring, tile selection, the bounded encoder pipeline, build metrics, and candidate-layout experiments. gbz-base remains only in research baselines and independent source-oracle checks; it no longer defines or constructs production payloads. The current object is normatively specified in File Format v1 and summarized in Fixed-window archive v1.

Direct archive construction

The normal v1 encoder writes a temporary sibling of the requested .pngr object. It emits the provisional 64-byte header and root, reserves the exact fixed directory-page span, and then appends accepted payloads in deterministic reference/coordinate order. Compact per-reference/per-bucket descriptors are retained for directory backfill; there is no payload spool, second full-file copy, occurrence table, or global pending-entry sort.

The normal regional path uses the project-owned LocalSubgraph selector. It copies exact packed GBWT records, forward node sequences, and canonical topology edges directly from PangenomeSource; it does not call gbz-base, enumerate, or sort anonymous paths while encoding. Exact record lengths provide the adaptive-split preflight, so an oversized parent is rejected before a second payload corpus is copied. The real reference is retained as a GBWT occurrence anchor with fragment and node offsets. Weighted anonymous paths are reconstructed only by a reader for the tiles selected by a query.

Tile selection/materialization and compression share one bounded persistent worker pool controlled by --threads and --max-queued-bytes; the CLI defaults to available parallelism capped at eight and a 256 MiB queue cap. Construction uses a rolling window rather than per-batch thread creation. Results are consumed in input order, and adaptive children precede later completed work, so worker completion order cannot change archive offsets or bytes. The completed temporary object is flushed, then the standard gate validates directory/range structure, BLAKE3-128 over every unique encoded payload, exact decompression, and regional structural decoding with bounded workers. Only then is it atomically renamed. validate --mode full additionally reconstructs every physical tile traversal; verify remains the independent selected-query source oracle. Failed temporary objects are removed unless --keep-partial is set.

An opt-in production path can append path-members-v1- pages to the same temporary object. --path-membership parses the four Simple-SDS document-array structures embedded in GBWT, builds the catalog from source metadata, and locates one completed tile's traversal starts at a time. Each LF round groups positions by GBWT record, reads and decodes that record once, then releases it. The catalog is front-coded in 1,024-ID pages. The extension descriptor records the identity-source implementation, authenticated GBZ SHA-256, catalog/group/occurrence totals, the sum of each group's distinct-path count, and codec distribution. Fixed 4 KiB membership-directory pages align one-for-one with graph directory pages; each entry addresses one multiplicity- and orientation-bearing tile page. Its traversal digest includes manifest identity, core bounds, the regional payload BLAKE3-128, and canonical oriented nodes. Before rename, validation requires exact digest, occurrence-weight, unique-count, multiplicity, catalog-bound, and regional-integrity reconciliation. The paired prepared summary/catalog mode remains an independent byte-for-byte oracle; normal archives are unaffected.

The encoder computes exact total selected reference bases before payload work. Progress snapshots report current reference/coordinate, accepted and physical chunks, processed/total bases, global and per-reference percentage, observed bp/s and chunks/s, a rate-derived ETA, build/processing elapsed time, and the current temp-final byte length. JSON and plain progress contain the same measurements; --progress-interval-seconds controls their cadence. Percent is coordinate-based rather than inferred from compressed bytes.

Progress covers every potentially long CLI phase. Input and output checksum workers report bytes, percentage, transfer rate, elapsed time, and ETA. For the default disk source, input SHA-256 is computed by the same sequential reads that construct the source cache, eliminating a redundant whole-source pass. GBZ load/cache construction and compact path-index construction emit elapsed-time heartbeats because those parsers do not expose reliable partial-work counts. The final structural validation reports directory entries/pages and unique physical payloads, compressed bytes reread, percentage, rate, elapsed time, and ETA. Interactive terminals select readable plain progress by default; JSON mode emits stable newline-delimited events, while redirected commands remain quiet unless a mode is selected explicitly.

Critical-path wall phases in encoder reports are non-overlapping through the output SHA-256 pass. Source checksum and source preparation are explicitly overlapping worker-wall measurements with a separate combined wall. Aggregate selection, encoding, compression, decode, and reconstruction worker milliseconds are labeled separately and must not be added to the wall critical path.

Source access seam

PangenomeSource isolates reference discovery, owned active node/record access, and reference-position lookup from the encoder pipeline. DiskGbzSource is the default encoder adapter. It parses GBZ/simple-sds sections directly, streams the packed-record body and decoded concatenated node sequences into four ephemeral files, writes arithmetic offset tables, and reads them through four 16 MiB block caches. Each cache has 16 fixed shards so bounded workers do not serialize on one cache lock; the total explicit read-cache limit remains 64 MiB. The source cache is removed on exit and is independent of the atomic .pngr temporary sibling.

An explicit persistent source cache stores the same four indexed data files plus a deterministic serialized SourcePathIndex, GBWT DA locate support, a source path catalog, and an atomic versioned JSON manifest. Its source byte length/SHA-256, GBZ serialization versions, component lengths, reference metadata digest, index interval, counts, bytes, and checksum are validated before reuse. Each raw component also has BLAKE3-128 per 256 KiB block, verified lazily before a block enters the bounded memory cache, so warm open does not require rereading 11.9 GB merely to detect later touched-block corruption. Builds use a temporary sibling and interprocess lock; warm encodes authenticate the input GBZ, open the cache read-only, and deserialize the sparse real-reference index instead of repeating cache creation and reference traversal. Persistent caches require explicit pruning; ephemeral caches retain cleanup-on-drop behavior.

GBZ v1 stores 425,853,421 record offsets and 212,926,710 sequence offsets for the retained HPRC source. The cache therefore intentionally trades 11.92 GB of ephemeral disk, or 12.06 GB for the reusable cache including integrity sidecars and the sparse reference index, for a measured 621,808 KiB populated whole-encode process peak instead of fully loading the 5.49 GB GBZ at an 8,775,928 KiB peak. The compact simple-sds indices used while constructing the cache still scale with record/sequence count; “bounded” means the source body and active reads are not retained in RAM, not constant memory independent of graph metadata.

SourcePathIndex is project-owned and samples only real reference paths roughly every 1,000 bp using length-only sequence lookup. LocalSubgraph performs interval walking and bidirectional context expansion from packed records. Neither is a global haplotype occurrence index. LoadedGbzSource remains available via --source-access loaded as the byte-correctness baseline.

pangenome-range-build::gbwt_locate owns the small construction-only parser for document-array samples already embedded in GBWT. DiskGbzSource performs bounded, batched LF locate without constructing vg's .ri, loading a complete GBWT, or adding the local fork as a dependency. The format and decoder crates remain independent of GBWT. Persistent source-cache v2 serializes the parsed DA structures and canonical catalog in authenticated files, so named-path encoding works identically through ephemeral and reusable cache paths.

Some population GBWTs contain only named sample haplotypes and omit the reference_samples tag. For an explicitly coordinate-relative research archive, --sample NAME --reference-haplotype N may select that exact real named haplotype as the archive anchor. This does not synthesize a sample or claim standard-reference coordinates. The override is limited to the bounded disk source and ephemeral cache path; persistent caches remain bound to the reference identities recorded when they were constructed.

Primary npm product boundary

packages/browser is the primary npm package. It publishes isolated reader, viewer, and Node library entries plus a separate Node executable shim. The root and /reader entries contain no DOM, Node built-ins, launcher logic, or native code; /viewer owns the framework-neutral rendering contract, /node owns Node-only range sources, and bin/ owns native process selection and launch. Adding the executable therefore does not change the reader/viewer dependency graph or bundle-size budgets. packages/benchmark remains private and owns Node and real-browser measurements.

The shim maps the runtime operating system, architecture, and Linux libc to one exact-version optional @pangenome-range/cli-* package. Those platform packages are generated only during release staging and contain no JavaScript runtime, installer, downloader, or compiler. The shim validates package/binary metadata, then spawns the Rust CLI with inherited standard streams and forwarded arguments, signals, and exit status. Missing or unsupported native packages do not prevent ordinary library imports.

The public contracts, strict HTTP/Blob/memory/file range sources, v1 bootstrap/root and arithmetic directory reader, byte-bounded caches, pure-JavaScript zstd decompressor, record-preserving regional decoder, and canonical graph assembly exist. Rust and TypeScript decode the same current fixture into typed-array-oriented nodes, sequences, topology, real reference traversal, and weighted tile-local paths, then produce identical canonical hashes. A bounded progressive Canvas 2D viewer consumes only queryTiles() and the query trace contract. A public-network browser benchmark corpus remains explicitly unimplemented.

Named identity is a separate lazy reader capability. pathCatalogInfo(), pathById(), pathsByIds(), and searchPaths() page the catalog without loading it wholesale. tilePathMemberships() reconciles one already-decoded graph tile, while queryWithPathMembership() returns graph, membership, and catalog traces separately. The compatibility pathMembership() performs graph-directory lookup, reads the aligned membership-directory pages, fetches only selected tile pages, and then fetches only catalog pages containing referenced path IDs. It returns tile-local group membership and real source-path records without changing query() results or loading GBWT/native code. The documentation tube-map inspector loads named paths on demand, filters and copies names, and highlights only the same canonical path ID in currently loaded tiles.

Named identity fails closed unless archive-meta-v1- is present and its source GBZ SHA-256 exactly matches the identity-source SHA-256 in path-members-v1-. Browser decoding separately caps materialized membership records at 250,000 per group and per tile; large multiplicities remain scalar weights rather than expanded objects.

The benchmark package wraps the public reader with a versioned workload, Node and Playwright runners, a strict/fault-injectable local range origin, immutable raw result retention, an optional WASM decoder, and a remote-origin validator. The browser page resolves pangenome-range/reader through an import map to the built public ESM entry; it does not reach into reader source or substitute Node file reads. A separate module origin serves page/reader/optional WASM assets, while the archive origin records exact cross-origin HEAD and range traffic with stable connection identifiers. Planned reader ranges stay separate from requests observed at the origin so an HTTP-cache hit cannot be mistaken for a library-cache hit.

Range sources and traces

Both languages expose a 64-bit source length and exact offset/length reads. Rust FileRangeSource uses positioned file reads, so it does not depend on or mutate a shared seek cursor. TypeScript keeps offsets as bigint and implements HttpRangeSource, BlobRangeSource, MemoryRangeSource, TracingRangeSource, and the Node-only FileRangeSource.

Trace summaries retain raw call order, offsets, lengths, and success, then derive:

• request and successful-request counts;
• total, unique, and duplicate requested bytes;
• smallest and largest request;
• the number of disjoint ranges after overlap/adjacency coalescing;
• how many nonempty reads could be merged with another read.

Unique-byte accounting is the union of half-open requested intervals. Failed reads are still requests and remain in the trace. Zero-length reads count as operations but cover no bytes.

HttpRangeSource lazily discovers size with usable HEAD metadata and falls back to GET Range: bytes=0-0. Exact reads require 206, matching exposed Content-Range and Content-Length, Accept-Ranges: bytes, and a stable exposed ETag; later reads use If-Range by default. A 200 whole-object body is rejected without reading it unless a caller explicitly configures a small maxFullResponseBytes cap. Normal reads leave the browser HTTP cache enabled; benchmarks opt into cache: "no-store" when measuring cold transport. The integration origin also supplies CORS, immutable no-transform cache policy, and raw request evidence. Browser/library and HTTP cache effects remain reported separately.

The archive reader retains the bounded bootstrap/root, uses separate byte-bounded LRU caches for directory pages, compressed graph payloads, encoded extension descriptors/pages, and decoded feature pages, fetches missing pages as contiguous spans, and coalesces selected payloads into one parallel dependency round. Optional query traces report exact ranges and layer bytes, dependency rounds, cache hits, decode/decompression/merge timings, selected counts, and the canonical BLAKE3 result. When tracing is disabled, the merge remains required but trace accounting and hashing are skipped.

The optional named-loci-v1--- extension is a sorted fence descriptor plus independently compressed leaves. It is emitted only with an exact GFF3 input and real reference sample binding are supplied. The default summary-pyr-v1-- extension is a fixed-grid, factor-four pyramid built from accepted core-tile counters. Both are lazy range APIs and remain optional extension entries so an unknown reader can still decode graph regions. Neither introduces sample identity or merges anonymous traversals across tiles.

Viewer pipeline

The /viewer entry is a one-way consumer of the reader API:

PangenomeArchive.queryTiles()
  -> deterministic tube-map adapter
  -> reference-anchored layout snapshot
  -> caller-owned SVG renderer

The viewer never opens URLs, reads byte ranges, parses archive bytes, or owns an archive. buildTubeMapModel() receives decoded tiles from its caller, sorts them independently of arrival order, selects a deterministic top-N set of anonymous local patterns, and collapses bounded reference and alternate structural segments. It refuses models above 400 displayed groups or 800 topology edges instead of truncating them. The bounded main-thread layout did not justify worker-transfer complexity.

Reference topology can merge by node identity. Weighted anonymous traversals remain attached to their source tile, are ranked by exact integer weight, and are never stitched across tile boundaries. The SVG shows orientation, reference-anchored alternate branches, curved topology, tile dividers, and local weights. Narrow labels disappear at locus-fit scale and return on zoom. renderTubeMapSvg() returns explicit cleanup for its listeners and DOM.

The VitePress demo is a one-screen application shell over that public boundary. It owns archive selection, native locus search, optional thin summaries, exact planRegion() limits, browser history, cancellation, URL state, and overlay inspectors. SVG pan and zoom stay local and do not create a request storm. Old visual content stays visible until the first replacement tile arrives.

The application normalizes wheel delta modes into a bounded continuous zoom, keeps the pointer's graph coordinate fixed, and coalesces SVG repaints to one animation frame. Pattern-label lanes use deterministic collision-free spacing; node labels are width-gated. Selecting a collapsed node opens the bounded inspector without changing topology, while expansion is a separate explicit action.

The public viewer exports the adapter, layout, SVG renderer, location parser, and region-policy helpers used by the application. It remains framework-neutral and cannot import Vue, VitePress, benchmark code, archive transport, or Node built-ins.

Cost simulation

NetworkProfile estimates idealized latency as:

ceil(requests / max_parallel) * (RTT + per-request overhead)
  + requested bits / bandwidth

The aggregate estimator is optimistic: it assumes all requests are known up front and perfectly parallelizable. Candidate experiments use a second method that accepts explicit dependency groups, so the bootstrap/index round precedes the data round. Both models still exclude connection setup, congestion, server variance, browser scheduling, decompression, parsing, and computation. They are useful for ranking early layouts but cannot replace real HTTP/browser benchmarks.

Correctness boundary

CanonicalSubgraph represents node IDs/sequences, oriented edge topology, the real reference traversal, and reference intervals. CanonicalHaplotypeTile separately represents anonymous local traversals, exact weights, and tile provenance. Maps and sets remove irrelevant node/edge ordering; path collections are sorted while retaining duplicate multiplicity; traversal order and orientation remain significant. A stable, domain-separated BLAKE3 digest makes oracle/candidate comparisons cheap to store.

The experiment now emits CanonicalSubgraph from:

1. an upstream GBZ/GBZ-base reference query, and
2. the fixed-window candidate reader over RangeSource.

Every current-format query has two gates. The assembled graph is compared with an independent local source extraction for node sequences, oriented edges, the reference traversal, and coordinates. Every selected tile is then freshly extracted from the source with the exact core interval and construction halo and compared for oriented traversal, weight, total weight, semantics, and provenance. Anonymous paths are never stitched across tiles. The oracle remains a full local source load and is not presented as a remotely efficient reader.

Upstream API boundary discovered

The gbz 0.7.0 crate safely exposes graph nodes, paths, metadata, tags, translation presence, sample/contig names, haplotype count, and reference-sample tags after full deserialization. The inspection CLI uses those APIs. It does not claim the upstream load is range-efficient. The experiment oracle also uses those APIs: simple_sds::serialize::load_from opens and deserializes the whole GBZ. That makes it a source/oracle path, not the candidate remote reader.