Benchmarks & Positioning
All numbers below are from frozen regression datasets, measured end-to-end against the running engine (re-baseline 2026-09-02, release build). Dev and heldout splits are kept separate; nothing is tuned against heldout.
Against a Production Baseline
The Argus-faithful dataset (200 production-shaped surveillance records: telegraphic enums, NA sparsity) compares PrismDB against the production system’s substring comparator under a protocol deliberately generous to the baseline (free keyword decomposition, precision-rewarding scoring, corpus-aware queries):
| System | Score |
|---|---|
| PrismDB | 37/50 |
| Production substring baseline | 34/50 |
The natural-language suite is where the gap opens (15/25, with age-vocabulary queries won via automatic enum projection: 60-plus → elderly, 20s → young). Zero manual aliases were declared; the automatic tiers covered everything. The baseline’s failures are a structural ceiling (substring matching cannot bridge vocabulary); PrismDB’s remaining failures have identified mechanisms and roadmaps.
Regression Suites
| Dataset | Shape | Score |
|---|---|---|
| Catalog | 30 records, 19 queries | 19/19 |
| Large catalog | 250 records, 56 queries (dev/heldout) | 52/56 (26/28 + 26/28) |
| Surveillance proxy | 200 vision-LLM records, 12 dims | 30/50 |
| Argus-faithful | 200 production-shaped records | 37/50 vs baseline 34/50 |
| Argus v2 (mixed entity) | 250 records, 50 queries (25 dims) | 34/50 vs baseline 42/50 |
| Workspace temporal | 15 tracks, 51 timestamped chunks | 11/16 |
Remaining workspace failures are documented honest boundaries: NLI correctly refusing weak-evidence matches, one measured dense-prefilter limit, one cross-track re-identification query that is out of scope.
Honest Positioning
Where PrismDB wins: attribute-decomposed queries over structured records, meaning queries whose terms map to independent record fields ("blue leather shoes", "elderly person carrying a red bag"). Per-dimension routing plus write-time vocabulary expansion is structurally better there than substring matching or one monolithic embedding.
Known structural boundaries:
- Many-dimension corpora: on Argus v2 (25 dims, mixed person/vehicle records) the cross-dimension gate drops true matches that the substring baseline keeps (34/50 vs 42/50). This is the current engine round.
- Reranker: a cross-encoder reranker over the same candidates is a rung on the cost ladder, weighed like every other tier; not yet measured on a fit judge.
- Interpretive leaps invisible to dense retrieval (
portfolio → document): beyond the current lexicon-projection tier’s reach; the query-log learning tier is the identified path.
Next public benchmark, STaRK-Amazon (NeurIPS 2024, planned): multi-attribute conversational product queries over structured entities (1.03M products), exactly the attribute-decomposed shape PrismDB is built for. Published dense baseline is 39.16 Hit@1 with LLM-based SOTA at 61.73 on the full corpus; PrismDB will be measured on a small subset with local baselines (substring, BM25, dense) on the same subset, so its numbers are not comparable to the published ones. A benchmark only counts here if it fits: the query shape has to be attribute-decomposed over structured records, the baselines have to run on the same subset, and the split has to be fixed before anything is measured. A suite that fails that test is not published as a result.
Every score above comes from a frozen dataset and a fixed protocol, re-measured end to end against the running engine. The datasets and the harness travel with the source, so the runs are reproducible for anyone holding it.