Design Decisions

Each decision below records the choice made, the alternatives considered, and the rationale.

1. Retrieval intelligence in the storage layer, not the application

Decision: Semantic matching, query planning, and adaptive optimization live inside PrismDB, not in application code.

Rationale: When retrieval logic lives in the application layer, every consumer (chat interface, REST API, dashboard, background worker) must independently implement synonym handling, embedding generation, multi-strategy search, and score fusion. This leads to inconsistent behavior and duplicated effort.

By placing retrieval intelligence in the storage layer, any application issues a simple Search("blue leather shoes") call and receives semantically matched results. The database owns the optimization; applications are consumers, not implementors.

This follows the same principle that led relational databases to internalize query optimization rather than leaving join strategies to application developers.

2. Per-dimension retrieval strategy

Decision: Each dimension has its own independent retrieval strategy (lens), rather than a single embedding per record.

Rationale: Concatenating all field values into a single embedding vector loses dimensional independence. If a record has material="genuine Italian leather", color="midnight navy", and style="minimalist low-top sneaker", a combined embedding places all three concepts in one vector. A query for "blue leather" now competes with "sneaker" in the same embedding space: dimensions dilute each other.

Per-dimension lenses keep each axis independent. The query "blue leather shoes" routes "blue" to the color dimension and "leather" to the material dimension. Each match is evaluated in its own semantic space, with no dilution.

This is the core structural insight that distinguishes PrismDB from document-level vector search.

3. Index everything on insert; adaptive promotion deferred

Decision (evolved): The original design had dimensions start with minimal indexing (exact only) and get progressively promoted to richer strategies based on observed query patterns. The implemented system indexes semantic dimensions with BM25 sparse + dense from the moment of insert; the unified qdrant-edge engine made both cheap enough that lazy promotion solved a problem that no longer existed.

Rationale: Adaptive promotion existed to avoid paying vector-index cost on fields that might never be searched semantically. With both indices living as payloads on the same point in one embedded engine, the marginal cost of indexing everything upfront is small, and every record is fully searchable immediately: no cold-start synonym misses, no promotion lag.

Stats-driven optimization survives as an offline analysis path (prismdb-optimizer reads the SQLite stats store) and as the research line behind learned routing. It is not in the serving path today.

4. Graduation lifecycle

Decision: Data is organized into logical segments tracked in a SQLite registry. Graduation (triggered via POST /collections/{name}/graduate) seals a segment, learns vocabulary from the new data, reindexes, and opens the next segment.

Rationale: Vocabulary expansion (the cost ladder of NLI-judged tiers) is an ingestion-time process that enriches the BM25 index. Running it in batch at graduation boundaries amortizes the cost: only novel (dimension, value) pairs are NLI-judged; previously seen vocabulary hits the pair cache instantly, making re-graduation effectively free.

The graduation model is inspired by:

• LSM-trees (memtable → SSTable): immutable segments after compaction
• Append-only log systems: accumulate records, then seal the batch

All records are searchable immediately on insert via all lenses (BM25 sparse + dense). Graduation adds learned vocabulary to the BM25 index and compiles entity documents for temporal collections.

5. Write-time expansion only: queries are pure retrieval

Decision (evolved): All vocabulary expansion (ConceptNet, NLI sense resolution, enum aliases, synonyms, lexicon projection) happens at write time (POST /learn / graduation). The original design kept a query-time expansion fallback for cold starts; the implemented system dropped it entirely. At query time, the only models involved are the embedding encoder (for the query itself): no synonym generation, no LLM, no expansion.

Rationale: Processing at write time amortizes cost across all future reads: an expansion judged once serves every future query, cached forever in the global pair cache. Query-time expansion multiplies cost by query volume and makes latency and results non-deterministic.

The cold-start problem the online fallback was meant to solve dissolved for the same reason as decision #3: indexing and learning are cheap enough to run upfront. The result is a hard property the system can promise: queries are ~5ms, deterministic, and their cost never depends on vocabulary difficulty.

6. Reciprocal Rank Fusion for score normalization

Decision: When multiple lenses return results for the same dimension (e.g., BM25 race with vector), scores are combined using Reciprocal Rank Fusion (RRF) rather than weighted averages or score normalization.

Rationale: BM25 produces unbounded scores (e.g., 12.7, 8.3, 5.1) while vector cosine similarity produces bounded scores (e.g., 0.96, 0.87, 0.72). These scales are incomparable; normalizing them requires knowing the score distribution, which varies per query and per corpus.

RRF sidesteps the problem entirely by using only rank position:

RRF_score(doc) = Σ 1 / (k + rank_in_source)

No normalization needed. No weight tuning. Robust across heterogeneous scoring systems. It is the standard approach used by Elasticsearch, Qdrant, and most hybrid search implementations.

7. SQLite for metadata and statistics

Decision: Collection schemas, tenant configuration, optimizer statistics, and standing query definitions are stored in embedded SQLite.

Rationale: These datasets are small (hundreds to low thousands of records), read-heavy, and require transactional consistency. SQLite handles this workload trivially with WAL mode.

The optimizer’s statistical needs are deliberately simple: per-dimension query frequency counters, lens hit/miss rates, and moving averages. The operations are UPDATE ... SET count = count + 1 and SELECT ... ORDER BY count DESC: counters and sorted reads, not analytical queries. Even at 100k queries/day, SQLite with WAL mode handles the write throughput comfortably.

Analytical engines like DuckDB excel at complex aggregations over millions of rows (OLAP workloads), but the optimizer doesn’t need that; it needs fast counters that inform promotion decisions. Adding a pluggable stats engine (SQLite vs DuckDB vs ClickHouse) would introduce abstraction complexity without solving a real problem. SQLite is sufficient, and sufficiency is the design goal.

SQLite also eliminates an external dependency: no separate database server to deploy, configure, or maintain. The metadata store is part of the PrismDB binary.

8. Rust core with unified embedded engine

Decision: PrismDB is implemented in Rust, using qdrant-edge as a unified in-process engine (BM25 sparse + dense) with SQLite for metadata, caches, and segment registry.

Rationale: A single unified engine eliminates cross-engine coordination. BM25 sparse vectors and dense embeddings live as payloads on the same qdrant-edge points per dimension. (ColBERT multivectors are also produced and stored, but no retrieval path consumes them; they are dormant.) Combined with:

• rusqlite: SQLite for stats, NLI pair caches, segment registry
• fastembed: BGE-M3 dense (1024d) embeddings in-process via ONNX Runtime
• ort: NLI cross-encoder (DeBERTa-v3 MNLI) for write-time sense resolution

The result is a single, self-contained binary with no external service dependencies. Deployment is ./prismdb serve: no Docker Compose, no coordination. When backend=mlx (Apple GPU), the binary spawns a local sidecar that serves all embedding + NLI inference; it is managed by PrismDB, not a separately deployed service.

9. Segments as immutable units

Decision: The segment is the fundamental storage primitive. Once a segment closes (graduates), it is never modified.

Rationale: Immutability at the segment level dramatically simplifies several operational concerns:

For the rare case of individual record deletion (e.g., GDPR compliance), a tombstone file is maintained per segment. Periodic compaction rewrites segments with accumulated tombstones removed. This keeps the immutability guarantee intact for normal operations while supporting legally mandated deletion.

10. Standing queries as reverse index

Decision: PrismDB supports registering persistent queries that evaluate every incoming record in real time, emitting events on match.

Rationale: The traditional search pattern is “query finds records.” Standing queries invert this: “records find queries.” When a new record is inserted, it is evaluated against all registered matchers before being indexed.

This enables notification and alerting use cases without polling:

1. A standing query is registered with a search expression and delivery target (webhook, server stream, WebSocket)
2. The query is decomposed by the planner into a compiled matcher: exact filters + pre-computed embedding
3. Every incoming record is checked against active matchers: exact filters first (cheap, short-circuit on miss), then embedding similarity (one cosine distance computation, not a full index search)
4. On match above threshold, an event is emitted to the delivery target

The cost per record is proportional to the number of active standing queries, not the size of the dataset. Exact filter short-circuiting ensures that most records are rejected cheaply before any embedding computation occurs.

This unifies search and monitoring in a single system: the same semantic matching logic serves both ad-hoc queries and real-time alerts.

11. Single node now, clustering-ready architecture

Decision: PrismDB launches as a single-node system. Architectural decisions are made to not impede future clustering, but no distributed coordination is implemented in v1.

Rationale: Distributed systems introduce consensus protocols, partition tolerance trade-offs, and operational complexity that slow down iteration on the core value proposition (semantic multi-dimensional search). Getting the storage engine, query planner, and adaptive optimizer right matters more than horizontal scalability at this stage.

The architecture is future-proofed through deliberate constraints:

• Segments are atomic units with globally unique IDs: replicable by shipping segment files
• Hot tier is single-writer per collection: no concurrent write conflicts to resolve
• Standing query matchers are stateless (compiled filter + pre-computed embedding): trivially replicable across nodes
• APIs address records by ID, not by memory pointer: no locality assumptions

The natural future topology is sharding by collection + replication by segment. Collections are independent (no cross-collection joins), so sharding introduces no coordination overhead. Segment replication is file-level copy of immutable data.

12. License strategy

Decision: Proprietary during development, with planned transition to Apache 2.0 upon public release.

Alternatives considered:

Rationale: The primary goal is professional projection and community adoption. Apache 2.0 maximizes both: enterprises can use PrismDB without legal friction, and contributors can fork and extend freely. The patent grant (absent from MIT) provides additional clarity for commercial users.

If traction justifies a business model, an open-core approach adds enterprise features (advanced multi-tenancy, SSO, audit logging, SLA guarantees) on top of the Apache 2.0 core. Managed PrismDB Cloud is another revenue path that doesn’t require relicensing.

The BSL/SSPL path remains available as a fallback if cloud provider free-riding becomes a real concern, but premature protective licensing signals distrust and suppresses early adoption.

13. Embedding library: fastembed over raw candle/ort

Decision: Use the fastembed crate for embedding generation rather than raw candle or ort.

Rationale: fastembed provides pre-built sentence transformer models (AllMiniLM-L6-v2, BGE variants) with automatic model download, tokenization, and inference in a single API. Raw candle requires manual model weight loading, tokenizer pipeline construction, and inference orchestration: substantial implementation effort for identical results. The abstraction cost is near-zero (fastembed wraps ONNX Runtime internally), and swapping to raw ort/candle remains possible if custom model loading is needed later.

14. Query decomposition over full-query broadcast

Decision: Decompose multi-term queries into per-dimension sub-queries rather than broadcasting the full query to all dimensions.

Rationale: Broadcasting “blue leather shoes” to every dimension causes each dimension to match on the full query’s semantics. The material dimension matches on “leather” overlap regardless of whether color or style match. This produces high recall but low precision: records matching in one dimension appear even when they fail in others.

Decomposition routes “blue” to color, “leather” to material, “shoes” to style. Each dimension searches only its relevant terms. Combined with cross-dimension AND enforcement (post-filter), this ensures results match across all query aspects, not just one.

The decomposition is hybrid: IDF (document frequency statistics from the BM25 index) handles lexically present terms with semantic-first routing; embedding similarity (term vs dimension centroid) handles terms not in the index. The routing uses the same BGE-M3 embedding model used for search; no separate model needed.