Why Claude Still Answers From the Obsidian Note You Replaced

Hello builders,

You walk out of this with a number you can defend to a client or drop into a thread, and nobody can argue with it, because it is about your data and theirs is not. Inside one thirty-day window I watched four credible practitioners give four incompatible answers about retrieval. So I stopped reading and went to test RAG retrieval on your own data, which in my case meant twenty questions I’d actually asked, and the result ranked our options backwards from what I expected.

Four answers, none measured

A data engineer, top comment on his thread: “BM25 text retrieval is great… just text an timestamps in a database.” Somebody agreeing, more bluntly: “All roads lead to bm25.” Somebody arguing the exact opposite, that a pure markdown brain becomes an absolute mess within days and that a temporal graph solved it. And a fourth, watching the cycle: “Funny how everyone reinvents the knowledge graph, hits the friction wall, then quietly crawls back to markdown files.”

All four are competent and all four are describing real experience. None of them measured the same corpus, which is the detail that ends the argument. We cannot settle this by reading. We can only settle it by measuring.

The golden set is the instrument

Twenty to forty questions you already ask, each with an answer you know is right and the page that holds it. That’s the whole thing, and building it is the highest-value hour we will spend here.

Pull the questions out of our own history. Invented ones are unconsciously written to match the pages we remember writing, and a set like that scores high on everything and discriminates nothing. Then we add the two categories people leave out:

- id: q01
  question: "How does this project deploy?"
  expect_page: pages/deployment-github-action.md
  must_not_return: pages/deployment.md   # superseded; returning it is a fail
  class: superseded-trap

- id: q19
  question: "What did we decide about rate limiting in the mobile client?"
  expect_page: null                      # genuinely not in the vault
  class: should-fail

Every fact we have ever marked superseded is a golden question, because the old answer must not come back. And three or four questions whose answers are genuinely absent catch the failure no recall metric will ever show us: a confident, fluent, entirely invented answer assembled from adjacent pages.

The sort that inverts your whole bake-off

Before we trust any lexical arm, here’s the thing that cost me a day and a half. From the SQLite FTS5 documentation:

The “-1” term at the start of the formula is not found in most implementations of the BM25 algorithm. Without it, a better match is assigned a numerically higher BM25 score. Since the default sorting order is “ascending”, this means that appending “ORDER BY bm25(ft)” to a query would cause results to be returned in order from worst to best. The “DESC” keyword would be required in order to return the best matches first. In order to avoid this pitfall, the FTS5 implementation of BM25 multiplies the result by -1 before returning it, ensuring that better matches are assigned numerically lower scores.

Better matches score more negative. So ORDER BY bm25(ft) returns best-first with no DESC, and adding DESC because it looks right gives us the worst matches first, ranked confidently, with no error. Let’s prove it on two rows before trusting a single score. Paste this into sqlite3 :memory::

CREATE VIRTUAL TABLE ft USING fts5(body);
INSERT INTO ft(body) VALUES ('vault rot vault rot vault rot');  -- strongest match
INSERT INTO ft(body) VALUES ('vault rot');                      -- weaker match
SELECT rowid, bm25(ft), rank FROM ft WHERE ft MATCH 'vault rot' ORDER BY bm25(ft);

My embedding arm was beating my lexical arm by a mile, which fit my expectations perfectly, and that is exactly why I did not check it. The lexical arm was returning its worst matches. Two more things from the same page while we are in there: k1 and b are hard-coded at 1.2 and 0.75 and we cannot tune them, and ORDER BY rank is the faster equivalent of a bare bm25() call.

Read the trap columns, not recall

Here’s what came back on my own vault, at 214 pages and roughly 78k tokens, scored at k equals 5.

The plain index with targeted reads took 17/20 on recall, 3/3 on the superseded traps and 2/3 on the should-fails, at $0.011 a query. FTS5 lexical, sorted on rank with the hook column weighted, took 18/20, 1/3 and 1/3, at $0.004. Embeddings over pgvector at top-5 took 16/20, 0/3 and 0/3, at $0.009.

Read the shape before the values. The three arms land within two of each other on raw recall and separate completely on the trap columns, and the trap columns are the ones that tell us whether the rest of our system can be trusted. Raw recall is the number everybody quotes and it is the least useful column on the board.

There’s a structural reason the index wins those columns and it is not about quality. State lives on the index line, so an index-based arm filters a retired page before it reads it, while a search-based arm has to retrieve the page before it can discover the page is dead. Marking a fact superseded doesn’t change its embedding.

Those illustrative numbers are from my vault and they are not your answer. Look at the shape rather than the values. Run the twenty questions, write down the decision, and write down underneath it what would reverse the decision, because a decision with no reversal condition isn’t a decision, it’s a preference.

Now go build something this weekend!

John Cook