The Ecosystem · 2026–present
PrismDB
A record is not one thing, so searching it with one strategy throws away most of what it knows.
An embedded search engine that stops treating a record as one thing: every field searched in its own terms, and the vocabulary gap closed long before anyone types a query.
PrismDB's site, a beam of white light entering a prism and leaving as three labelled rays (a learned NLI match on colour, a BM25 match on material, a dense match on style), each carrying the field it resolved.
The flattening
A record is composite. Every index it lands in insists otherwise.
Product catalogues, surveillance logs, legal documents, sensor histories: a great deal of the world's data arrives as records whose fields are short pieces of writing. Colour, material, behaviour, condition, clause. Each field is a separate axis holding a separate small fact. Then the record is handed to an index, and the first thing the index does is forget that.
-
Flattened into characters
Substring and keyword matching treat a field as an opaque run of characters. Someone who types blue never reaches a record that says midnight navy, because nothing in the index knows those are the same claim about the same axis.
-
Averaged into a point
A single embedding over the whole record blends every field into one vector. Colour, material and behaviour compete for the same dimensions, and afterwards not one of them can be asked about on its own.
-
One strategy for every kind of field
An enumerated category and a free-text description are different questions wearing the same shape. Almost nothing adapts per field, so the strict field ends up searched loosely and the loose one strictly.
-
The vocabulary gap is nobody's fault
The pipeline that writes the record, increasingly a model, has its own vocabulary. The person searching has another. Full-grain cowhide and leather are both correct, and neither side is going to give way.
-
The usual repair is charged per query
Bridge the gap at query time and it is paid for on every search forever: synonym lists that rot as the corpus moves, or a model call that makes each query slower, costlier, and no longer the same twice.
-
Precision and meaning, choose one
Loosen the match and leatherette starts passing as leather. Tighten it and the record that said navy is unreachable. While the whole record shares one setting, the two failures trade against each other and neither can be fixed for good.
The choice the record is forced into
The decomposition
Not one strategy for the database. One strategy per field.
The fields were never uniform, so the retrieval should not be either. Colour wants an equivalence that has been learned: navy, cobalt and indigo all answer to blue. Material wants lexical precision, because leatherette is not leather and no amount of semantic generosity should blur that. Style wants proximity, because a sneaker is a shoe. Give each axis the strategy its own question implies, then fuse what comes back.
One strategy per database is a guess about the data. One strategy per field is a reading of it.
The second reframe: the gap is closed before the query exists
The vocabulary problem does not have to be solved while someone is waiting. Every distinct value a corpus holds is known at the moment it is written, and that is when the work can be done: the value is expanded into the words people will actually reach for, each candidate expansion is judged rather than assumed, and the verdict is cached against the value itself. Once per unique value, not once per search.
What is left at query time is pure retrieval: routing, matching, fusing. No rewriting, no model in the path, no network call, and the same query returns the same answer tomorrow. Moving the intelligence earlier is what buys that determinism. It is not a performance trick applied afterwards; it is where the design starts.
The expansions themselves are found on a ladder ordered by cost, and the doctrine is comparison rather than prohibition. A graph walk is tried before a trained judge, and a judge before a generative model, and each gap is closed by the cheapest rung that genuinely closes it. Nothing is forbidden for being expensive. It simply has to beat something cheaper that was measured first.
What the engine keeps
Six mechanisms, each one a refusal to flatten something
None of these is a feature in the sense of an addition. Each marks a place where the ordinary design discards a distinction the data had already made, and where this one declines to.
A lens for each field
Every field carries its own retrieval strategy: equality for an enumeration, lexical scoring where the precise word matters, dense proximity where it does not. On a free-text field the lexical and dense legs run at once and their rankings are fused, so neither has to be right alone.
Vocabulary learned, never curated
The words a corpus is missing are worked out from the corpus itself: graph lineage, sense resolution against the value the candidate came from, prefix stripping on enumerations, and a general-English lexicon for the interpretive leaps no graph contains. Every candidate faces the same entailment judge before it is kept, and not one synonym is written by hand.
The query is decomposed too
A query is no more a single thing than a record is. Its terms are split and routed to the fields where they carry signal, and the messiness of real input (misspellings, compounds, terms that should have been one) is repaired before routing rather than after failing.
Agreement across the fields the query touched
A record that matches on colour and fails on material is not a close call, it is wrong. Because the fields stayed separate, that constraint can be stated at all. It is exactly the precision a single blended vector has no way to express.
Time as a field that moves
When a record describes something that changes, a field becomes a stream of timestamped facts rather than one settled value. What was said at a moment, across an interval, or in a particular order becomes something you can ask for, without leaving the engine that holds everything else.
Learning that stays incremental
Growth happens in segments: seal what has arrived, learn only the vocabulary that is genuinely new, reindex, open the next. Because every judgement is cached against the value that provoked it, re-learning a corpus that has not moved judges nothing at all.
Architecture
Two times, not two services
The interesting division in this system is not between components. It is between when work is allowed to be expensive and when it is not. Most of the design follows from putting that line in a different place than usual.
Where the expensive work is allowed to happen
Most systems that close a vocabulary gap close it inside the request. That is the natural place for it: the query is right there, the gap is visible, and a model can be asked. It also means every user pays for the same inference repeatedly, that the answer drifts as the model underneath is replaced, and that a search has quietly become a distributed system with a dependency that can be down.
Inverting it costs something honest. Writes are slower, and a corpus carrying genuinely new vocabulary has to be worked through before its new words are searchable. That is the trade, made deliberately: ingestion is a batch, and a batch is allowed to take its time, whereas a search is a person waiting.
One query, end to end
Routing is the part carrying the most judgement. Deciding that blue belongs to colour and leather to material is what makes per-field retrieval possible at all, and it is decided from the statistics of the corpus rather than from a configuration file describing this particular domain. A domain file would work, and would have to be written again for every corpus that ever arrives.
The fusion step is deliberately unclever. Rankings from different fields and different strategies are combined by position rather than by score, because a lexical score and a vector distance are not on a common scale and pretending they are is where a great deal of hybrid search goes quietly wrong.
The substrate
It is one binary. The index, the embedding and judging models, and the accumulated statistics all ship inside it and run in the process that already holds the data, the arrangement SQLite made ordinary, rather than the arrangement a cluster requires. Nothing to stand up, no queue between the parts, no two versions to keep in agreement.
Underneath, a small set of crates divided by responsibility rather than by layer.
- corethe vocabulary of the model itself: collections, fields, facts, lenses
- storagethe unified index, and the statistics kept beside it
- plannerwhich terms reach which field, and how the rankings fuse
- semanticthe embeddings, the entailment judge, the knowledge graph
- optimizerwhat the accumulated statistics suggest changing
- serverthe surface everything outside talks to
- cliserving, and the batch work that has to precede it
What accumulates
The index gets denser. The query does not get slower.
Because the expensive work is anchored to values rather than to requests, what the system learns is a property of the corpus and not of the traffic. It thickens as the data grows, while the thing a person actually waits for stays exactly where it was.
What accumulates is a vocabulary. Not a synonym file that somebody maintains, but a record of every equivalence the corpus turned out to need, with a judgement attached to each: this expansion held, this one was proposed and refused. It can be read. When a search fails, the question of why is answerable by looking at what the index learned about the value in question, which is a different situation from a similarity score that came out low for reasons nobody can recover.
The same discipline produces a list of things that did not work, and that list is kept. Query-time expansion of every kind, refused for what it does to determinism. A small curated lexicon, refused because the words actually needed turned out to sit outside it. A confidence threshold raised high enough to filter junk, refused because the junk survived it while the true matches did not. Frozen suites and a public benchmark, dev and held-out kept apart, nothing tuned against the half that decides. The engine outperforms a production substring matcher on a dataset built to that system’s own shape, with no synonym declared by hand. And it loses, structurally, on queries that are one long noun phrase against one long title, because such a query has no fields to decompose into. Both are worth saying. A boundary that has been located is worth more than a benchmark that has been won.
The numbers themselves live on the engine’s own site, where the tables, the splits and the instructions for reproducing them belong. This page is about the argument that produced them.
PrismDB was extracted from Prisma, which needed to search reflections along independent dimensions and found nothing that would do it. The two now share an intuition and very little else: that anything worth understanding is composite, and that the way to understand it is to hold its parts apart long enough to hear them separately. Prisma applies that to a question and eight lenses. PrismDB applies it to a record and its fields.
This is the studio’s clarity principle below the interface, where it is hardest to hold. Structure the data already has should be preserved and used, not flattened for the convenience of the index. A system that knows a record has fields and then searches it as though it does not has thrown away the only thing that made the record legible in the first place.