Benchmarks & Positioning
All numbers below are from frozen regression datasets and a public benchmark, measured end-to-end against the running engine. Dev and heldout splits are kept separate; nothing is tuned against heldout.
Against a Production Baseline
The Argus-faithful dataset (200 production-shaped surveillance records: telegraphic enums, NA sparsity) compares PrismDB against the production system’s substring comparator under a protocol deliberately generous to the baseline (free keyword decomposition, precision-rewarding scoring, corpus-aware queries):
| System | Score |
|---|---|
| PrismDB | 37/50 |
| Production substring baseline | 34/50 |
The natural-language suite is where the gap opens (15/25, with age-vocabulary queries won via automatic enum projection: 60-plus → elderly, 20s → young). Zero manual aliases were declared; the automatic tiers covered everything. The baseline’s failures are a structural ceiling (substring matching cannot bridge vocabulary); PrismDB’s remaining failures have identified mechanisms and roadmaps.
ESCI (public e-commerce benchmark)
Amazon’s ESCI product-search benchmark, ~7.6M NLI pairs at full scale:
| Metric | Before | After corpus-derived query understanding |
|---|---|---|
| nDCG@10 (all-200) | 0.504 | 0.557 |
| nDCG@10 (frozen test half) | n/a | 0.568 (+0.066 vs round baseline; test gain exceeded dev, no overfitting) |
| Recall@50 | 0.638 | 0.804 |
| Warm latency | n/a | ~5ms (release build) |
The gap to a monolithic hybrid retriever (0.663) closed ~33%, entirely via corpus-derived query understanding: zero domain hardcodes.
Regression Suites
| Dataset | Shape | Score |
|---|---|---|
| Catalog | 30 records, 19 queries | 19/19 |
| Large catalog | 250 records, 56 queries (dev/heldout) | 52/56 (26/28 + 26/28) |
| Surveillance proxy | 200 vision-LLM records, 12 dims | 30/50 |
| Argus-faithful | 200 production-shaped records | 37/50 vs baseline 34/50 |
| Workspace temporal | 15 tracks, 51 timestamped chunks | 11/16 |
Remaining workspace failures are documented honest boundaries: NLI correctly refusing weak-evidence matches, one measured dense-prefilter limit, one cross-track re-identification query that is out of scope.
Honest Positioning
Where PrismDB wins: attribute-decomposed queries over structured records, meaning queries whose terms map to independent record fields ("blue leather shoes", "elderly person carrying a red bag"). Per-dimension routing plus write-time vocabulary expansion is structurally better there than substring matching or one monolithic embedding.
Known structural boundaries:
- Title-phrase decomposition: queries that are one long noun phrase against one long title field don’t decompose into dimensions; monolithic retrieval keeps an edge.
- Reranker ceiling: a cross-encoder reranker over the same candidates reaches 0.694 nDCG@10 on ESCI’s test half; closing that gap requires a rerank stage, which is on the cost ladder to be weighed like every other tier.
- Interpretive leaps invisible to dense retrieval (
portfolio → document): beyond the current lexicon-projection tier’s reach; the query-log learning tier is the identified path.
Next public benchmark, STaRK-Amazon (NeurIPS 2024, planned): multi-attribute conversational product queries over structured entities (1.03M products), exactly the attribute-decomposed shape PrismDB is built for. Dense baseline is 39.16 Hit@1 with LLM-based SOTA at 61.73. ESCI shows where PrismDB loses (title-phrase); STaRK-Amazon is expected to show where it wins. Together, the honest story.
Every score above is reproducible from the frozen datasets in the repository (testdata/, tools/smoke, tools/benchmark).