LLM Wiki Maintenance: One Wrong Summary, 16 Errors

Summary:

1. The 1 to 4 to 16 cascade: how one wrong summary becomes structural error in six weeks.
2. The five lint checks (Karpathy’s three plus two production add-ons) that catch drift early.
3. The cost ladder at 50, 500, and 5000 entries; when delta-lint becomes mandatory.
4. The slash command template you ship as .claude/commands/wiki-lint.md.

Llm wiki maintenance is the part of the Karpathy pattern nobody warned you about. The wiki compounds. So do the bugs in it. One slightly-wrong summary on a Sunday-night ingest becomes a wrong claim in a paid client proposal six weeks later, and you do not notice because the cascade is structural, not cosmetic. This article walks the failure mechanism, the lint pass that cuts it at level one, and the cost math at every wiki scale.

What is the biggest risk in llm wiki maintenance?

Hallucination compounding. The mechanism is not abstract. Walk it once.

You ingest a research paper at ten on Sunday night. The paper says “chunking with 20% overlap reduced retrieval noise by ~30% on the authors’ benchmark.” The agent summarizes it as “chunking with overlap reduces retrieval noise by 30%, generally.” The word “generally” is the wrong summary. The paper made a measured claim about one benchmark; the wiki entry promoted it to a universal.

Four other wiki entries link to that research entry. account-acme.md references it when recommending a chunking strategy. concept-chunking.md cites it. module-search-pipeline.md references the same claim. A new project page on a Tuesday in week three writes “per concept-chunking, overlap reduces noise by 30%”, citing your wiki rather than the paper.

Six weeks later you write a proposal to Acme citing your wiki. The proposal claims the customer’s pipeline will reduce retrieval noise by 30%. Acme’s engineer reads the paper, finds the claim is benchmark-specific, and asks you about it on a call.

One wrong summary on ingest. Four linked entries. Sixteen downstream references. The numbers are illustrative; the mechanism is real. The exact branching factor depends on how interlinked your wiki is. The key word from the original Reddit threads on this is “structural.” Once a wrong summary has been cited in four places, fixing it means fixing four. Once those four have been cited from, fixing it means fixing twenty.

Why the cost is non-linear

The longer the drift sits, the more it costs to clean up, and the cost does not grow in a straight line. A lint pass that runs weekly catches the drift while the cost is still bounded. A lint pass that never runs lets the cost grow until you cite a wrong claim in a paid client proposal.

ultrathink-art on r/learnmachinelearning (363 upvotes on the “hardest part” thread) said it best, quoted in Files Over Prompts chapter 5:

Stale entries compound just as fast as accurate ones.

That sentence is the entire economic argument for the lint pass. The wiki without lint is a knowledge base that gets worse over time at compound interest. The wiki with lint is one that converges on truth.

What does a wiki lint pass actually check?

Karpathy’s gist names lint as one of three named operations on the wiki (the others are Ingest and Query). Three lint targets, each a one-line test the agent runs against the wiki.

Contradictions between pages. Does account-acme.md say the renewal date is November 14 while q4-pipeline.md says November 21? Both cannot be right. The agent flags the mismatch and you reconcile manually.

Stale claims. Does any wiki entry’s source-path raw file have an mtime newer than the entry’s ingested-at timestamp? Flag for re-ingestion. The human approves which stale entries are worth re-running.

Orphan pages. Are there entries no other entry links to? Orphans are not necessarily wrong; they are often artifacts of an old ingest the agent never integrated. The lint surfaces; the human triages.

Two more checks are worth adding for production use:

Frontmatter completeness. Every entry has type, source-path, and ingested-at. Without these, the other three checks degrade silently.

Cross-reference validity. Every [[wiki/something.md]] link points at a file that exists in wiki/. Broken wikilinks compound the same way bad summaries do.

Five checks. All automatable. None of them fix the wiki; they surface drift for human resolution. That separation is load-bearing. The agent should never silently rewrite a wiki entry the lint flagged as stale; the human decides which fix is correct.

How do you ship the lint pass?

Two paths. Both valid. One is cost-bounded and human-triggered; the other is automatic and burns tokens on every save.

Path A: a slash command. Create .claude/commands/wiki-lint.md in your wiki root. Invoke it manually when you finish a writing session or once a week, whichever comes first. The cost is bounded because you decide when it fires. Recommended for the first month of running your wiki.

Path B: a hook. Configure a hook in .claude/settings.json that fires automatically on every Write tool call targeting a file in wiki/. The cost is unbounded; every save burns tokens. Ship this once you have absorbed the cost math below and decided the automation is worth the bill.

The slash command is the recommended ship. Save the following to .claude/commands/wiki-lint.md:

---
description: Health-check the wiki. Surface contradictions, stale claims, and orphan pages.
---

You are the wiki lint pass. Your job is to surface drift, not fix it.

Read every file in wiki/*.md. For each file, check:

1. Frontmatter completeness. Every entry has `type`, `source-path`, `ingested-at`.
2. Cross-reference validity. Every [[wiki/<filename>]] link points at a file that exists.
3. Stale claims. If the entry's source-path raw file's mtime is newer than the entry's ingested-at timestamp, flag for re-ingestion.
4. Contradictions. Compare the entry against every other entry that shares a [[link]] or named entity. Flag any factual contradiction.
5. Orphan pages. List any entry with zero inbound [[links]] from other entries.

Output a structured report:

# Wiki Lint Report
## Frontmatter issues
- [list, one per file]
## Stale claims
- [list, one per file]
## Contradictions
- [list, one per pair]
## Orphan pages
- [list, one per file]

DO NOT modify any wiki entry. Surface findings; the human decides on resolutions.

In Claude Code, type /wiki-lint. The agent reads the command, walks the five checks, and produces the report. Read the report. Resolve the items it names. Repeat weekly.

There is a third framing on Reddit and YouTube that conflates lint with skills. Some posts describe the lint as “a Claude Code skill that fires on save.” That phrasing is wrong. Skills are user-invocable; they live at .claude/skills/<name>/SKILL.md and you call them by name. Hooks are event-triggered; they fire on tool events automatically. There is no primitive called “a skill that fires on save.” If a post says to build one, what they mean is a hook. Use the hook config.

What the cost math actually looks like

The numbers below assume Anthropic’s published pricing for claude-sonnet-4-6 (~$3 per million input tokens, ~$15 per million output tokens). A reasonable model for lint passes that do not need Opus’s full reasoning depth.

For a 50-entry wiki, each entry averaging 500 input tokens (reading) and 50 output tokens (flagged items), one full lint pass:

Input:  50 entries × 500 tokens = 25,000 tokens = $0.075
Output: 50 entries ×  50 tokens =  2,500 tokens = $0.038
Total per pass:                                    ~$0.11

A daily lint is about $3.30 per month. A weekly lint is about $0.50 per month. Both are in the noise of an Anthropic bill.

The cost ladder at different scales:

The 5,000-entry cliff is what the OP of the r/AI_Agents “Spent a weekend” thread named verbatim (quoted in Files Over Prompts chapter 12):

Ingest is expensive. Great for curated personal small scale knowledge, painful for an enterprise doc dump.

He is right. The fix at scale is delta-lint: only re-check entries whose ingested-at timestamp has changed since the last pass, plus any entries that link to a changed entry. For a wiki where a handful of entries change per week, delta-lint runs against five entries instead of five thousand, and the cost drops by about 80%.

The honest threshold: around 100 entries, switch from full-lint to delta-lint. The total bill stays under $5 a month. The cost stays bounded. The substrate stays trustworthy.

What broke the first time I ran it

The first full lint pass against my real wiki surfaced three findings in five minutes. One contradiction between two entries about the same Acme account (different next-action dates). One stale claim where a paper had been updated and my entry had not. One orphan from an old ingest I had never linked into anything. Three findings. About five minutes to resolve. The “would have missed by hand” payoff is the catch.

The drift-laden run a couple of weeks later looked different:

# Wiki Lint Report
## Frontmatter issues
- account-globex.md: missing `ingested-at`.
## Stale claims
- concept-rag-evaluation.md: source-path arxiv-2410-12345.md mtime is 6 weeks newer.
- module-auth-service.md: source-path pr-12389-token-rotation.md was deleted.
## Contradictions
- account-acme.md vs q4-pipeline.md: renewal date is "November 14" in one, "November 21" in the other.
## Orphan pages
- concept-prompt-caching.md
- person-thariq.md

That run took about twenty minutes to resolve: re-ingested the stale paper, archived the deleted PR’s module entry, picked the correct Acme renewal date, linked the two orphans into the existing structure. The cost of those twenty minutes is the cost of NOT citing the wrong Acme renewal date in next week’s proposal. That is the trade.

What should you actually do?

• If your wiki has fewer than 50 entries → ship the slash command, run it once weekly. Skip the hook.
• If your wiki is between 50 and 100 entries → still full-lint weekly. The cost is under $1 a month.
• If your wiki has crossed 100 entries → switch to delta-lint before tracagnotto’s “fucking nightmare” cost becomes yours.
• If you have not run a lint pass yet → run one now against your current wiki. Expect at least one finding. Resolve it. The five minutes of cleanup pays back the first time the agent does not give a wrong answer in a real meeting.

bottom_line

• The wiki without lint is a knowledge base that decays at compound interest. The fix is non-negotiable.
• Five checks, ship as a slash command, run weekly. About 50 cents per month at personal scale.
• The cascade is real. One wrong summary becomes sixteen downstream references in six weeks. The lint pass cuts the tree at level one. That is the entire game.