Query Engine

Overview

The query engine decomposes a search request into per-dimension operations, routes each term to the dimension where it carries signal, executes BM25 sparse and dense legs in parallel for every routed dimension, and merges results into a single ranked list via Reciprocal Rank Fusion.

Queries are pure retrieval: every expansion tier runs at write time. Nothing here calls a model larger than the embedding encoder, and warm latency is ~5ms.

Query Planner

Input

Output: Prism

A Prism is the query plan, named after the core metaphor. It specifies which dimensions the query touches, which terms route to each, and which extra channels (whole-query dense, residual boost) are active.

Query Rewrites (corpus-derived)

Before routing, zero-hit terms are repaired against the collection’s own vocabulary (no external dictionary):

• Spell correction: banded Levenshtein against the corpus vocabulary, guarded by a ConceptNet real-word gate plus digit and first-letter checks ("leathr" → "leather", but real words are never rewritten).
• Compound splitting: a token splits when its halves form a real corpus bigram ("redcell" → "red cell").
• Adjacent merging: neighboring tokens merge when the compound spelling dominates the spaced bigram in the corpus ("deco art" → "DecoArt", both spellings kept in the BM25 query).

Term Routing

Each term routes to the dimension where it carries the most signal:

• IDF routing, semantic-first: a term with a semantic home routes there over an exact home. Exact dimensions are routable via their derived enum tails (male, elderly).
• DF guard: a term whose document frequency exceeds 2/3 of a dimension’s points produces no routing or constraint from that dimension (it carries no signal there).
• Embedding-routing fallback: zero-IDF queries route to the dimension with the highest top-1 dense similarity, gated on that dimension’s dense top score.
• Noise handling: terms still zero-hit after rewrites and unknown to ConceptNet are treated as noise; their fraction weights extra whole-query dense channels (zero noise → zero extra channels).

Each dimension’s BM25 and dense search then runs with only its routed terms, not the full query.

Lens Execution

Every dimension type has a fixed execution shape; there is no per-query strategy selection:

BM25 prefetch depth is IDF-weighted; dense prefetch is query-coverage-weighted. Dense legs are scoped to routed dimensions only.

Scoring: Reciprocal Rank Fusion (RRF)

Different lenses produce incomparable scores: BM25 returns unbounded term-frequency scores, dense search returns cosine similarity (0–1), exact returns binary match. RRF normalizes by using rank position instead of raw scores.

Formula

RRF_score(doc) = Σ  1 / (k + rank_in_source_i)

Where:

• k = 60 (standard constant, dampens the contribution of low-ranked results)
• The sum is over all sources (lenses) that returned the document

Example

Source A (BM25):   doc_X rank 1,  doc_Y rank 2,  doc_Z rank 3
Source B (dense):  doc_Z rank 1,  doc_X rank 2,  doc_W rank 3

doc_X: 1/(60+1) + 1/(60+2) = 0.01639 + 0.01613 = 0.03252
doc_Z: 1/(60+3) + 1/(60+1) = 0.01587 + 0.01639 = 0.03227
doc_Y: 1/(60+2) + 0         = 0.01613
doc_W: 0         + 1/(60+3) = 0.01587

Final ranking: X > Z > Y > W

Documents appearing in multiple sources rank higher. Position matters more than raw score. No normalization or tuning required.

Cross-Dimension Coherence

After RRF merge, a post-filter enforces coherence across the dimensions the query actually touched:

• A candidate passes a dimension if it matched via BM25 or sits in that dimension’s dense top-15%.
• Two routed dimensions → strict AND.
• Three or more → N−1 (one weak dimension is forgiven, but emphatic dimensions can never be the dropped one).
• Terms with multiple dimension homes get forgiveness across their homes.

Relaxation: strict AND yielding 0 results falls back to N−1. Opt-in backfill: true pads the tail with gate rejects ordered by evidence; zero survivors stays zero, so the precision contract stays intact.

Mixed-query semantic-residual boost: when a routed core is joined by a dropped real-word phrase (≥2 ConceptNet-known zero-hit terms), survivors are reordered by rrf × (1 + 4·sim_norm), where the residual phrase is embedded alone and compared against each survivor’s stored per-dimension vectors. Negation markers disable the boost.

Search Diagnostics

The search response carries lightweight diagnostics alongside results:

Roadmap

Learned routing: replacing the IDF heuristic with a trained per-field weighting model. A single-feature variant measured net-zero end-to-end and ships disabled (routing.learned_enabled: false); a multi-feature variant is under evaluation. Aggregations (GROUP BY over exact dimensions, semantic clustering over embedding space) are design-stage, not in the current API.