Vocabulary Expansion

The Cost Ladder

The vocabulary gap (your corpus says "midnight navy", your users search "blue") is PrismDB’s central problem. The system closes it entirely at write time, through a ladder of expansion tiers ordered by cost. The doctrine: every gap is closed by the cheapest tier that closes it, chosen by comparison. Expensive tiers (trained models, LLMs) are genuinely weighed, never dismissed by habit. “LLM-free today” is a comparison outcome, not a constraint.

Every candidate expansion, whatever tier proposed it, passes the same gate: the NLI cross-encoder (DeBERTa-v3 MNLI) judges whether the expansion is actually entailed. All verdicts are cached globally per unique pair; a warm re-learn is milliseconds, and re-graduation of known vocabulary judges zero pairs.

Tier 1: ConceptNet Lineage

Leaf-to-parent graph expansion over the ConceptNet knowledge graph, gated by the NLI judge:

  • cowhide → leather: kept (NLI confirms entailment)
  • oat → cereal, boot → torture: rejected (graph noise the judge filters out)

Cheapest tier: a graph walk plus one cached judgment per candidate.

Tier 2: NLI Sense Resolution

For each (dimension, value), corpus-derived candidates are judged directly: navy → blue, runner → shoe, daypack → bag. A modifier tier catches attributes embedded in packed values: "navy nylon windbreaker" gains blue.

Capped selection: candidates are capped at M = min(500, dimension_vocabulary). Below 500 unique values, everything is judged. Above (e.g. e-commerce title fields with ~5,000 values), a hybrid dense∪PMI selector picks the top-500: PMI top-250 (co-occurring interpretive candidates, walking → pace) ∪ dense top-250 (taxonomic/attribute, charcoal → black). Measured: PMI alone has 45% no-signal on single-token values; dense alone buries interpretive accepts. This keeps learn cost O(novel vocabulary).

Tier 3: Enum Alias Tiers (exact dimensions)

Exact dimensions get searchable vocabulary automatically:

  1. Prefix stripping: the value set’s common prefix is derived and stripped: PERSON_GENDER_MALE → male.
  2. Tail expansion: the stripped tails expand via ConceptNet synonyms, NLI-gated: man ↔ male.

Manual alias declaration (values: [{value, aliases}]) exists as an override, unused by every regression dataset. Zero curated aliases needed in practice.

Tier 4: Synonym Tier (semantic dimensions)

1-hop Synonym/SimilarTo edges from ConceptNet per token, judged with the full value as premise (copular pair), capped at 4 accepted synonyms per source token with a document-frequency ceiling of 5%. Closes the gap where lineage alone was blind: lingering → loiter. Includes a lemma fallback (lingering → linger → loiter) because ConceptNet stores lemmas.

Tier 5: Lexicon Projection (the generative rung)

The tiers above only propose words connected to the corpus by a graph. Some query vocabulary is invisible to any graph: fidgeting → nervous, briskly → hurry, angrily → aggressive, 60-plus → elderly.

The lexicon-projection tier closes this with a general-English interpretive lexicon: top-20k frequency words ∩ WordNet classes (adjectives, motion/emotion/social/communication verbs, person/feeling nouns) = 8,666 lemmas, shipped as a compile-time asset. Per value, the lexicon is dense-prefiltered to the top-500 and NLI-judged with the full-value premise.

Jurisdiction guard: candidates already present in the collection vocabulary are excluded; in-corpus words belong to the anchored sense-resolution tier. (The alternative own-dimension-only guard was measured and refuted: it produced cross-dimension aliasing like mahogany → wooden leaking into color.)

Measured boundary: interpretive leaps invisible to dense retrieval at any granularity (portfolio → document ranks ~2,236 in the lexicon by dense similarity) remain open; that class needs a stronger generative rung.

What Was Measured and Refuted

The ladder’s shape is the product of measurement, not preference. Notable refuted alternatives:

  • Query-time expansion of any kind: never runs; queries stay ~5ms and deterministic.
  • Capped/small lexicon (~980 words by class caps): real targets sit at class ranks 977–2,237; any honest general recipe is the full classes.
  • Token-level dense prefilter: measurably worse than full-value premise.
  • NLI threshold 0.90 as junk filter: junk survives 0.9 while true targets die at 0.85.
  • Supplementary dense gate at query time: every tested threshold broke precision.

Roadmap: Query-Log Learning

The next tier (qlog::) learns from real usage: the engine queries its own index over logged queries, and an NLI + LLM-judge cascade gates candidate expansions derived from behavioral splits. Lab-validated on hard pairs; engine integration is the next round. This is the cost ladder’s comparison doctrine in action: the LLM enters exactly where cheaper tiers were refuted.