llmwiki

LLM-powered knowledge base from your Claude Code, Codex CLI, Cursor, Gemini CLI, and Obsidian sessions. Built on Andrej Karpathy's LLM Wiki pattern.

Rebuilt on every master push from the synthetic sessions in examples/demo-sessions/. No personal data. Shows every feature of the real tool (activity heatmap, tool charts, token usage, model info cards, vs-comparisons, project topics) running against safe reference data.

Every Claude Code, Codex CLI, Copilot, Cursor, and Gemini CLI session writes a full transcript to disk. You already have hundreds of them and never look at them again.

llmwiki turns that dormant history into a beautiful, searchable, interlinked knowledge base — locally, in two commands. Plus, it produces AI-consumable exports (llms.txt, llms-full.txt, JSON-LD graph, per-page .txt + .json siblings) so other AI agents can query your wiki directly.

./setup.sh                         # one-time install
./build.sh && ./serve.sh           # build + serve at http://127.0.0.1:8765

Contributing in one line: read CONTRIBUTING.md, keep PRs focused (one concern each), use feat: / fix: / docs: / chore: / test: commit prefixes, never commit real session data (raw/ is gitignored), no new runtime deps. CI must be green to merge.

All screenshots below are from the public demo site which is built on every master push from the dummy example sessions. Your own wiki will look identical — just with your real work.

• All your sessions, converted from .jsonl to clean, redacted markdown
• A Karpathy-style wiki — sources/, entities/, concepts/, syntheses/, comparisons/, questions/ linked with [[wikilinks]]
• A beautiful static site you can browse locally or deploy to GitHub Pages
  • Global search (Cmd+K command palette with fuzzy match over pre-built index)
  • highlight.js client-side syntax highlighting (light + dark themes)
  • Dark mode (system-aware + manual toggle with data-theme)
  • Keyboard shortcuts: / search · g h/p/s nav · j/k rows · ? help
  • Collapsible tool-result sections (auto-expand > 500 chars)
  • Copy-as-markdown + copy-code buttons
  • Breadcrumbs + reading progress bar
  • Filter bar on sessions table (project/model/date/text)
  • Reading time estimates (X min read)
  • Related pages panel at the bottom of every session
  • Activity heatmap on the home page
  • Model info cards with structured schema (provider, pricing, benchmarks)
  • Auto-generated vs-comparison pages between AI models
  • Append-only changelog timeline with pricing sparkline
  • Project topic chips (GitHub-style tags on project cards)
  • Agent labels (colored badges: Claude/Codex/Copilot/Cursor/Gemini)
  • Recently-updated card on the home page
  • Dataview-style structured queries in the command palette
  • Hover-to-preview wikilinks
  • Deep-link icons next to every heading
  • Mobile-responsive + print-friendly

Every HTML page has sibling machine-readable files at the same URL:

• <page>.html — human HTML with schema.org microdata
• <page>.txt — plain text version (no HTML tags)
• <page>.json — structured metadata + body

Site-level AI-agent entry points:

Every page also includes an <!-- llmwiki:metadata --> HTML comment that AI agents can parse without fetching the separate .json sibling.

• 4-factor confidence scoring — source count, source quality, recency, cross-references; with Ebbinghaus-inspired decay per content-type
• 5-state lifecycle machine — draft → reviewed → verified → stale → archived with 90-day auto-stale
• 15 lint rules — 8 structural (frontmatter, link integrity, orphans, freshness, duplicates, index sync…) + 3 LLM-powered (contradictions, claim verification, summary accuracy) + stale_candidates (#51) + cache_tier_consistency (#52) + tags_topics_convention (#302) + stale_reference_detection (#303)
• Auto Dream — MEMORY.md consolidation after 24h + 5 sessions: resolve relative dates, prune outdated, 200-line cap
• 9 navigation files — CLAUDE.md, AGENTS.md, MEMORY.md, SOUL.md, CRITICAL_FACTS.md, hints.md, hot.md + per-project hot caches

• link-obsidian CLI — symlinks the whole project into an Obsidian vault; graph view + backlinks + full-text search just work
• Dataview dashboard — 10 ready-to-use queries (recently updated, by confidence, by lifecycle, by project, by entity type, open questions, stale pages)
• Templater templates — 4 templates for source/entity/concept/synthesis pages, seeded with confidence + lifecycle + today's date
• Category pages — tag-based index pages in both Dataview (Obsidian) and static markdown (HTML) modes
• Integration guide — docs/obsidian-integration.md covers 6 recommended plugins with per-plugin configs

• SessionStart hook — auto-syncs new sessions in the background on every Claude Code launch
• File watcher — llmwiki watch polls agent stores with debounce and runs sync on change
• Auto-build on sync — /wiki-sync triggers /wiki-build (configurable; default on)
• Configurable scheduled sync — llmwiki schedule generates OS-specific task files (launchd/systemd/Task Scheduler)
• MCP server — 12 production tools (query, search, list, read, lint, sync, export, + confidence, lifecycle, dashboard, entity search, category browse) queryable from any MCP client (Claude Desktop, Cline, Cursor, ChatGPT desktop)
• Multi-agent skill mirror — llmwiki install-skills mirrors .claude/skills/ to .codex/skills/ and .agents/skills/
• Pending ingest queue — SessionStart hook converts + queues; /wiki-sync processes queue
• No servers, no database, no npm — Python stdlib + markdown. Syntax highlighting loads from a highlight.js CDN at view time.

┌─────────────────────────────────────┐
│  ~/.claude/projects/*/*.jsonl       │  ← Claude Code sessions
│  ~/.codex/sessions/**/*.jsonl       │  ← Codex CLI sessions
│  ~/Library/.../Cursor/workspaceS…   │  ← Cursor
│  ~/Documents/Obsidian Vault/        │  ← Obsidian
│  ~/.gemini/                         │  ← Gemini CLI
└──────────────┬──────────────────────┘
               │
               ▼   python3 -m llmwiki sync
┌─────────────────────────────────────┐
│  raw/sessions/<project>/            │  ← immutable markdown (Karpathy layer 1)
│     2026-04-08-<slug>.md            │
└──────────────┬──────────────────────┘
               │
               ▼   /wiki-ingest  (your coding agent)
┌─────────────────────────────────────┐
│  wiki/sources/<slug>.md             │  ← LLM-generated wiki (Karpathy layer 2)
│  wiki/entities/<Name>.md            │
│  wiki/concepts/<Name>.md            │
│  wiki/syntheses/<Name>.md           │
│  wiki/comparisons/<Name>.md         │
│  wiki/questions/<Name>.md           │
│  wiki/index.md, overview.md, log.md │
└──────────────┬──────────────────────┘
               │
               ▼   python3 -m llmwiki build
┌─────────────────────────────────────┐
│  site/                              │  ← static HTML + AI exports
│  ├── index.html, style.css, ...     │
│  ├── sessions/<project>/<slug>.html │
│  ├── sessions/<project>/<slug>.txt  │  (AI sibling)
│  ├── sessions/<project>/<slug>.json │  (AI sibling)
│  ├── llms.txt, llms-full.txt        │
│  ├── graph.jsonld                   │
│  ├── sitemap.xml, rss.xml           │
│  ├── robots.txt, ai-readme.md       │
│  ├── manifest.json                  │
│  └── search-index.json              │
└─────────────────────────────────────┘

See docs/architecture.md for the full 3-layer Karpathy + 8-layer build breakdown.

Full production documentation lives under docs/. The editorial hub is docs/index.md — tutorials, per-agent guides, reference, and deployment, all in one place.

Start here:

Contributing to docs? See the style guide.

git clone https://github.com/Pratiyush/llm-wiki.git
cd llm-wiki
./setup.sh

git clone https://github.com/Pratiyush/llm-wiki.git
cd llm-wiki
setup.bat

pip install -e .                # basic — everything you need
pip install -e '.[pdf]'         # + PDF ingestion
pip install -e '.[dev]'         # + pytest + ruff
pip install -e '.[all]'         # all of the above

Syntax highlighting is now powered by highlight.js, loaded from a CDN at view time — no optional deps required.

1. Creates raw/, wiki/, site/ data directories
2. Installs the llmwiki Python package in-place
3. Detects your coding agents and enables matching adapters
4. Optionally offers to install the SessionStart hook into ~/.claude/settings.json for auto-sync
5. Runs a first sync so you see output immediately

Running the project? The governance scaffold lives under docs/maintainers/ and is loaded by a dedicated skill:

Four Claude Code slash commands automate the common ops:

• /review-pr <N> — apply the REVIEW_CHECKLIST to a PR and post findings
• /triage-issue <N> — label + milestone + priority a new issue
• /release <version> — walk the release process step by step
• /maintainer — meta-skill that loads every governance doc as context

The unit suite (pytest tests/ — 472 tests) runs in milliseconds and covers every module. The end-to-end suite under tests/e2e/ is separate: it builds a minimal demo site, serves it on a random port, drives a real browser via Playwright, and runs scenarios written in Gherkin via pytest-bdd.

Why both? Unit tests lock the contract at the module boundary; E2E locks the contract at the user's browser. A diff that passes unit tests but breaks the Cmd+K palette will fail E2E.

Install the extras (one-time, ~300 MB for Chromium):

pip install -e '.[e2e]'
python -m playwright install chromium

Run the suite:

pytest tests/e2e/ --browser=chromium

Run a single feature:

pytest tests/e2e/test_command_palette.py --browser=chromium -v

The E2E suite is excluded from the default pytest tests/ run (see the --ignore=tests/e2e addopt in pyproject.toml) so you can iterate on the unit suite without waiting for browser installs. CI runs the E2E job as a separate workflow (.github/workflows/e2e.yml) that only fires on PRs touching build.py, the viz modules, or tests/e2e/**.

Feature files live under tests/e2e/features/ — one per UI area (homepage, session page, command palette, keyboard nav, mobile nav, theme toggle, copy-as-markdown, responsive breakpoints, edge cases, accessibility, visual regression). Step definitions are all in tests/e2e/steps/ui_steps.py. Adding a new scenario is usually a 2-line change to a .feature file plus maybe one new step.

Run locally with an HTML report:

pytest tests/e2e/ --browser=chromium \
  --html=e2e-report/index.html --self-contained-html
open e2e-report/index.html     # macOS — opens the browseable report

Where to see test reports:

Locally, the HTML report is one file (e2e-report/index.html) that you can open in any browser — pass/fail per scenario, duration, stdout/stderr, screenshot on failure.

Run llmwiki schedule to generate the right scheduled task file for your OS from your config (cadence, time, paths). Or copy a static template:

Cadence (daily / weekly / hourly), hour/minute, and paths are all configurable in examples/sessions_config.json. See docs/scheduled-sync.md for full instructions.

llmwiki init                    # scaffold raw/ wiki/ site/ + seed nav files
llmwiki sync                    # convert .jsonl → markdown (auto-build + auto-lint if configured)
llmwiki build                   # compile static HTML + AI exports
llmwiki serve                   # local HTTP server on 127.0.0.1:8765
llmwiki adapters                # list available adapters + configured state (v1.0)
llmwiki graph                   # build knowledge graph (v0.2)
llmwiki watch                   # file watcher with debounce (v0.2)
llmwiki export-obsidian         # write wiki to Obsidian vault (v0.2)
llmwiki export-qmd              # export wiki as a qmd collection (v0.6)
llmwiki export-marp             # export Marp slide deck from wiki (v0.7)
llmwiki eval                    # 7-check structural quality score /100 (v0.3)
llmwiki lint                    # 11-rule wiki lint (8 basic + 3 LLM-powered, v1.0)
llmwiki check-links             # verify internal links in site/ (v0.4)
llmwiki export <format>         # AI-consumable exports (v0.4)
llmwiki synthesize              # auto-ingest synthesis pipeline (v0.5)
llmwiki manifest                # build site manifest + perf budget (v0.4)
llmwiki link-obsidian           # symlink project into Obsidian vault (v1.0)
llmwiki install-skills          # mirror .claude/skills to .codex/ and .agents/ (v1.0)
llmwiki schedule                # generate OS-specific scheduled sync task (v1.0)
llmwiki version

Each subcommand has its own --help. All commands are also wrapped in one-click shell/batch scripts: sync.sh/.bat, build.sh/.bat, serve.sh/.bat, upgrade.sh/.bat.

Adding a new agent is one small file — subclass BaseAdapter, declare SUPPORTED_SCHEMA_VERSIONS, ship a fixture + snapshot test.

llmwiki ships its own MCP server (stdio transport, no SDK dependency) so any MCP client can query your wiki directly.

python3 -m llmwiki.mcp   # runs on stdin/stdout

Twelve production tools (7 core + 5 added in v1.0 #159):

Register in your MCP client's config — e.g. for Claude Desktop, add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "llmwiki": {
      "command": "python3",
      "args": ["-m", "llmwiki.mcp"]
    }
  }
}

Single JSON config at examples/sessions_config.json. Copy to config.json and edit:

{
  "filters": {
    "live_session_minutes": 60,
    "exclude_projects": []
  },
  "redaction": {
    "real_username": "YOUR_USERNAME",
    "replacement_username": "USER",
    "extra_patterns": [
      "(?i)(api[_-]?key|secret|token|bearer|password)...",
      "sk-[A-Za-z0-9]{20,}"
    ]
  },
  "truncation": {
    "tool_result_chars": 500,
    "bash_stdout_lines": 5
  },
  "adapters": {
    "obsidian": {
      "vault_paths": ["~/Documents/Obsidian Vault"]
    }
  }
}

All paths, regexes, truncation limits, and per-adapter settings are tunable. See docs/configuration.md.

Gitignore-style pattern file at the repo root. Skip entire projects, dates, or specific sessions without touching config:

# Skip a whole project
confidential-client/
# Skip anything before a date
*2025-*
# Keep exception
!confidential-client/public-*

This project follows the three-layer structure described in Karpathy's gist:

1. Raw sources (raw/) — immutable. Session transcripts converted from .jsonl.
2. The wiki (wiki/) — LLM-generated. One page per entity, concept, source. Interlinked via [[wikilinks]].
3. The schema (CLAUDE.md, AGENTS.md) — tells your agent how to ingest and query.

See docs/architecture.md for the full breakdown and how it maps to the file tree.

• Stdlib first — only mandatory runtime dep is markdown. pypdf is an optional extra for PDF ingestion.
• Works offline — no Google fonts, no external CSS. Syntax highlighting loads from a highlight.js CDN but degrades gracefully without it.
• Redact by default — username, API keys, tokens, emails all get redacted before entering the wiki.
• Idempotent everything — re-running any command is safe and cheap.
• Agent-agnostic core — the converter doesn't know which agent produced the .jsonl; adapters translate.
• Privacy by default — localhost-only binding, no telemetry, no cloud calls.
• Dual-format output (v0.4) — every page ships both for humans (HTML) and AI agents (TXT + JSON + JSON-LD + sitemap + llms.txt).

• Getting started — 5-minute quickstart
• Setup guide — 15-minute end-to-end tutorial: local setup → deploy to GitHub Pages → customization (v1.0)
• Obsidian integration — 5-minute setup, 6 recommended plugins, config recipes (v1.0)
• Architecture — Karpathy 3-layer + 8-layer build breakdown
• Configuration — every tuning knob
• Privacy — redaction rules + .llmwikiignore + localhost binding
• Scheduled sync — daily/weekly/hourly task setup per OS
• Windows setup — Windows-specific gotchas
• Framework — Open Source Framework v4.1 adapted for agent-native dev tools
• Research — Phase 1.25 analysis of 15 prior LLM Wiki implementations
• Feature matrix — all 161 features across 16 categories
• Roadmap — Phase × Layer × Item MoSCoW table
• v0.4 roadmap — AI & Human Dual-Format plan
• Translations: i18n/zh-CN, i18n/ja, i18n/es

Per-adapter docs:

• Claude Code adapter
• Codex CLI adapter
• Cursor adapter
• Gemini CLI adapter
• Obsidian adapter
• PDF adapter
• Copilot adapter (Chat + CLI)

Shipped milestones:

• v0.5.0 — Folder-level _context.md, auto-ingest, adapter graduations, lazy search index, scheduled sync, WCAG, E2E tests (milestone)
• v0.6.0 — qmd export, GitLab Pages CI, PyPI release automation, maintainer governance scaffold (milestone)
• v0.7.0 — Structured model-profile schema, vs-comparison pages, append-only changelog timeline (milestone)
• v0.8.0 — 365-day activity heatmap, tool-calling bar chart, token usage card, session metrics frontmatter (milestone)
• v0.9.0 — Project topics, agent labels, Copilot adapters, image pipeline, highlight.js, public demo deployment
• v0.9.x — Sprint 1-4 foundation for v1.0.0 Obsidian integration: confidence scoring, lifecycle state machine, 9 navigation files, 11 lint rules, Auto Dream, Dataview dashboard, multi-agent skills, 12-tool MCP server, meeting + Jira adapters

Active milestones:

• GitHub Pages — shipped in v0.1 via .github/workflows/pages.yml (triggers on push to master). See docs/deploy/github-pages.md.
• Docker / GHCR — pull and run: docker compose pull && docker compose up -d. Image published to ghcr.io/pratiyush/llm-wiki on every tag push. See docs/deploy/docker.md.
• GitLab Pages — copy .gitlab-ci.yml.example → .gitlab-ci.yml. See docs/deploy/gitlab-pages.md.
• Vercel / Netlify — static deploy after llmwiki build. See docs/deploy/vercel-netlify.md.
• Any static host — llmwiki build writes to site/, which you can rsync/scp anywhere.

• Andrej Karpathy for the LLM Wiki idea
• SamurAIGPT/llm-wiki-agent, lucasastorian/llmwiki, xoai/sage-wiki, and bashiraziz/llm-wiki-template — prior art that shaped this.
• Python Markdown for the rendering pipeline, and highlight.js for client-side syntax highlighting.
• llmstxt.org for the llms.txt spec used in v0.4.

MIT © Pratiyush