Architecture

System Overview

PrismDB is a Rust workspace composed of focused crates, each owning a single responsibility.

Crate Responsibilities

Data Model

A record is a set of independent dimensions, each holding a short textual fact. Records do not store heavy artifacts directly; they carry artifact fields that point to external systems (object stores, CDNs, other databases).

Each dimension is typed:

A lens is the retrieval strategy active on a dimension: exact, bm25, vector, or a combination. Semantic dimensions are indexed for BM25 sparse and dense retrieval immediately on insert; both legs run in parallel at query time.

Unified Storage Engine

PrismDB uses qdrant-edge as a unified, in-process storage engine. Both retrieval modes (BM25 sparse and dense 1024d) run within the same embedded engine per dimension. There is no separate BM25 or vector service to coordinate.

Why Unified

The original design planned separate Tantivy (BM25) and Qdrant (vector) engines with a hot/cold tiering model. The actual implementation unified everything into qdrant-edge: BM25 sparse vectors and dense embeddings live as payloads on the same points. This eliminates cross-engine coordination, simplifies deployment (one data directory, one process), and makes every record searchable via all lenses immediately on insert.

Graduation Lifecycle

Graduation is a logical event, not a physical data migration. A SQLite segments registry tracks segment state. When POST /collections/{name}/graduate is called:

1. Seal the current segment
2. Learn: the cost ladder of vocabulary expansion tiers runs on the new data. Only novel (dimension, value) pairs are NLI-judged; previously seen vocabulary hits the pair cache instantly. This makes re-graduation effectively free.
3. Reindex: BM25 sparse vectors are rebuilt to incorporate learned expansions
4. Entity documents (temporal collections only): compile per-stream deduplicated fact sets for entity-level retrieval
5. Open the next segment

The graduation endpoint is score-identical to the manual flow (load → learn → delete → reload). Incremental cost is proportional to novel vocabulary only.

Query Flow

Score Fusion

When multiple lenses return results for the same record, scores are incomparable across engines (BM25 produces unbounded scores; vector cosine similarity is 0–1). PrismDB uses Reciprocal Rank Fusion (RRF):

RRF_score(record) = Σ 1 / (k + rank_in_source)

Where k = 60 (standard constant). RRF is rank-based, not score-based: no normalization needed, no weight tuning, agnostic to scale.

Cross-Dimension Coherence

After retrieval and RRF merge, a cross-dimension filter enforces that results hold across the dimensions the query actually touched. This is a precision mechanism that monolithic vector search cannot express: it prevents single-dimension matches from dominating when the query spans multiple fields.

Model Backends

All embedding and NLI inference routes through a configurable backend (prismdb.yaml: embedding_backend / nli_backend, values auto|mlx|gpu|cpu):

• cpu: ONNX Runtime in-process. Always available.
• gpu: NVIDIA, auto-detected.
• mlx: a unified sidecar (spawned by the binary) serves embedding + NLI judging on the Apple GPU. Measured: ~1,465 NLI pairs/s; lexicon embedding ~250× faster than CPU (8,666 lemmas in 4.9s vs ~20min). Under mlx, no CPU model remains in the live pipeline.
• auto: picks the best available.

prismdb prefill --backend mlx pre-fills the NLI pair cache from the GPU before POST /learn; the engine then hits 100% cache and learn completes in milliseconds.

Stats & Adaptive Optimization (design-stage)

The stats collector (SQLite-backed) records lightweight per-query events: dimensions searched, lens used, hit/miss, latency. The prismdb-optimizer crate analyzes these offline.

Automatic lens promotion driven by these stats is design-stage: the optimizer is not wired into the serving path today, and the auto_optimize flag exists as a schema field only. A learned routing model trained on query logs measured net-zero end-to-end in its single-feature form and ships disabled (routing.learned_enabled: false).

Temporal Axis (Optional)

Collections can enable temporal mode. When enabled, each dimension becomes a semantic stream: its facts evolve over time rather than being a single static value.

Track T-001, dimension "activity":

  t=09:00  "entering through main door"
  t=09:02  "talking to receptionist"
  t=09:05  "walking toward elevators"

The record at any point in time is a snapshot: the latest fact from each dimension’s stream as of that timestamp.

Temporal Query Types

Temporal mode is additive: non-temporal collections are byte-identical. Each point carries a _ts numeric payload; the TrackIndex caches per-stream presence intervals for efficient as-of queries.

Temporal Storage

One qdrant-edge point per chunk, sharing a stream_key (default: track_id). Fact timestamps persist as numeric _ts payload. Trajectory queries run two fully-gated pipeline legs joined on the stream key with strict ts_A < ts_B ordering.

Configuration (Mapping)

PrismDB uses a mapping system with two modes:

Explicit Mapping

The user defines dimensions, types, and lenses upfront:

{
  "mappings": {
    "dimensions": {
      "category": { "type": "exact", "values": ["footwear", "outerwear"] },
      "material": { "type": "semantic", "lens": ["exact", "bm25", "vector"] }
    },
    "artifact_fields": ["product_id", "image_url", "price"]
  }
}

Dynamic Mapping

Unmapped dimensions are inferred on first encounter:

• Few distinct values → exact
• Free-form text → semantic with exact lens only, auto_optimize: true

dynamic: "strict" rejects unmapped fields entirely.

Both modes coexist: explicit for known dimensions, dynamic for emergent ones.

Vocabulary Expansion (Cost Ladder)

At write time (POST /learn or during graduation), PrismDB runs a cost ladder of expansion tiers (ConceptNet lineage, NLI sense resolution, enum aliases, synonyms, lexicon projection), each vocabulary gap closed by the cheapest tier that closes it, every candidate gated by the NLI judge, every verdict cached.

Full tier-by-tier detail, caps, and measured boundaries: Vocabulary Expansion.

Backup and Restore

Backup

• Data directory: the qdrant-edge data, SQLite databases, and segment registry. Snapshot tooling (tools/smoke/snapshot.sh save|restore <label>) captures a prepared data dir.
• NLI pair cache: global ~/.cache/prismdb/cache.sqlite; survives data-dir wipes, contains all NLI verdicts. This is the checkpoint-resume mechanism.

Restore

The persistent collection registry rehydrates on server restart. A restored data directory + pair cache means warm re-learn (all pairs cached → millisecond learn).

Schema Evolution

Adding a Dimension

New records receive the dimension immediately. Existing records return null for the new dimension, which is excluded from matching (not treated as a mismatch).

Removing a Dimension

Mark as deprecated → planner ignores it in query plans → future operations drop the dimension’s data.

Null Values

Engine-level null_values (default ["NA", "N/A", ""]): sentinel facts are skipped at insert (absence-native sparsity).