Internals¶

How esbi-cli works inside, and why it is built this way. For using it, read Getting started, How do I...? and the command reference. For what is unfinished, read Known limits below.

The idea in one paragraph¶

The pattern is Karpathy's LLM Wiki: raw/ holds the originals and is never edited; wiki/ holds notes the worker writes and maintains; SCHEMA.md holds the conventions the model is given. A Python worker does the file work. A language model only ever returns JSON, which the worker validates and then applies. The model never touches a file and never calls a tool, so it works with small local models.

Data flow¶

flowchart LR
  subgraph capture
    A[inbox/ clips and PDFs] --> Q
    B[email via IMAP] --> A
    C[sb add / legacy import] --> Q
  end
  Q[(queue, SQLite)] --> R[run_queue]
  R --> X[extract: web, PDF, clip]
  X --> S[save_raw -> raw/]
  X --> CH[split_chunks]
  CH --> RD[read_chunks: notes per chunk]
  RD --> PL[make_plan: core plan]
  PL --> DG[make_digest: abstract and ideas]
  RD --> AG[aggregate: terms, quotes, relations]
  PL --> CN[connect: links to existing pages]
  DG --> AP[apply_plan]
  AG --> AP
  CN --> AP
  AP --> W[wiki/ notes + attachments/]
  W --> G[git commit in the vault]
  W --> IDX[index.md, log.md, daily note]

One source, step by step (ingest/pipeline.py):

Fence (fence.py): every prompt that carries untrusted text (the source, a chunk, a title, the existing pages, the question) passes it through fence_safe first: a look-alike of the tags the prompt builders open (<source>, <chunk>, <chunk_notes>, <section_notes>, <existing_pages>...) or a chat marker (<|...|>, <<SYS>>, [INST]) is turned into something inert, and titles and URLs are quoted with json.dumps so they cannot add lines to the instructions. It stops a source from forging structure; it does not make a model ignore what the source asks.
Extract (extract/): a URL becomes text with trafilatura (and up to five article image links); a PDF becomes plain Markdown from PDFium's characters (extract/pdf_text.py: paragraphs, headings by font size, weight and numbering, table rows one per line) plus up to eight figures, images and captioned vector drawings (extract/pdf_figures.py); a Web Clipper note is read from its own body, after the known page chrome of five sites (GitHub, LinkedIn, X, Reddit, YouTube) has been removed whole line by whole line (extract/noise.py; the count goes to stripped_lines). Every URL the worker fetches itself goes through netguard.safe_get: http(s) only, public addresses only, every redirect re-checked.
Deduplicate: by URL and by a hash of the text. A duplicate is skipped with no model call.
Read (ingest/chunks.py, ingest/read.py): a source longer than max_source_chars is cut into chunks at paragraph boundaries and the references section is dropped. The reading model takes notes on every chunk: points, terms, verbatim quotes, relations. A source of more than max_chunks chunks then has those notes merged into section notes (ingest/mapreduce.py, see Long sources). A chunk it cannot read twice is read again in two halves; only when both halves fail too is it skipped with a warning (Only half of chunk 2 of 5 could be read. when one of them works).
Retrieve (ingest/retrieve.py, index.py): the worker finds existing pages that may relate to this source with SQLite full-text search (FTS5) over a persistent index that is kept in step with the files by their size and modification time. Page lookups by name and source lookups by URL or hash use the same index.
Plan (ingest/plan.py): the writing model gets the notes (not the raw text) and the titles of candidate pages, and returns a core plan: title, one-liner, summary, key points, tags, concepts, entities. A source that fits in max_source_chars is not chunked: the model gets its text, and this one call also writes the detailed summary, key ideas, terms, quotes and relations.
Digest (ingest/digest.py, chunked sources only): a second, small call writes the detailed summary, key ideas and open questions. The detailed summary is a list of 3 to 4 paragraphs (paragraphs) and the key ideas are 4 to 6 short ones, because a list is something a small model's grammar can bound and one long string was not (the first try failed about one time in four with llama3.2; it is now about one in twenty-five). The retry names the problem and asks for less. Terms, quotes and diagram relations are not asked of the model again: code merges what the chunk notes already contain.
Connect (ingest/connect.py): a third call is shown the existing pages with their one-liners and says how the new source relates to them, with the reason.
Scrub, then apply (scrub.py, ingest/apply.py): apply_plan first walks the whole plan and passes every string the model wrote through scrub (plain prose: no HTML, no image syntax, no control characters; a Markdown link or address only if the source contains it; a [[link]] only to an existing page that this side of the email line may point at), counting the addresses it cut (ApplyResult.stripped_addresses). Quotes and glossary terms are not scrubbed: they are checked word for word against the source instead. apply_plan is the only module that writes wiki pages. It checks every claim before writing (see "Trust model") and then writes the source note, creates or extends concept and entity pages, writes figures into attachments/, and records what it did. It reports what it dropped in ApplyResult (unsupported_terms, trivial_terms, dropped_edges, unsupported_entities). The writing is done inside interrupts.deferred(), so a signal waits for the note to be whole.
Consolidate (ingest/consolidate.py, optional): for the concept and entity pages this source touched that now have enough sources, a ## Summary is written on top of their per-source sections, up to [run].max_consolidations_per_run (0 by default). sb consolidate does the same on demand.
Report (report/): index.md, log.md, and the daily note are regenerated. The vault is committed to git.

Long sources¶

A source of at most max_chunks chunks (16 of 8,000 characters by default) is read one call per chunk and goes to the plan as it is, exactly as always. A longer one (a book, a 60,000-word document) used to be sampled, so the note was about its first third. Now it is read by a bounded map-reduce (ingest/mapreduce.py):

Map. Every chunk gets its own notes (points, terms, quotes, relations), with the same retries and the same split into halves as before.
Reduce. The notes of each group of 5 chunks are merged by the model into one section note, level by level, until at most max_chunks notes remain. A group of one passes up unchanged. The plan, the detailed summary and the connections are written from those notes, never from the raw text, so no prompt outgrows the model's context window (a merge prompt carries at most 12,000 characters of notes).
The finale reserves 7 calls. Before each chunk the worker checks that the calls left cover the merges the notes read so far will need plus the 7 calls the plan (2 attempts), the detailed summary (3) and the connections (2) can take. When they do not, reading stops there.

What the model is allowed to do in a merge is small. It rewrites only the points of a section (3 to 8 sentences). The terms, quotes and relations of every chunk are carried over by code (a spread of them per section), never by the model, so nothing it could invent reaches the note, and they are still checked against the source at the end like any other. If a merge fails (invalid twice, too slow, no calls left), code merges the points and the run warns Section 2 was merged without the model (...).

The call cap. [run].max_calls_per_source (60, minimum 12) counts every model call of one source, retries and halves included; a call that fails was still paid for. When it stops the reading early, the source is not dropped: the note starts, under its title, with a warning callout saying how many parts and what share of the text were not read and where the unread part starts (> [!warning] Not read: 12 of 52 parts of the source (the last 23% of the text, starting at "...") because the limit of 60 model calls per source ([run].max_calls_per_source) was reached., No leído: ... in a Spanish note), and the run prints the same as a warning. Nothing is dropped silently. A source that needs more than about 45 chunks (roughly 350,000 characters at the default chunk size, fewer when chunks need retries) is read only as far as the cap allows: raise the cap for books. A source of 40 chunks takes about 55 calls, which is two to three times the model calls (so, roughly, the tokens and the time) of the old sampling, which made 16 chunk calls and the finale.

The progress lines and sb run --json show the new step as merging section 2 of 8 ({"event":"step","name":"merging section","index":2,"total":8}), like the chunk step.

State: where everything lives¶

What	Where	Owner
Originals	`vault/raw/` (PDFs and Markdown snapshots)	Written once, never edited
Notes	`vault/wiki/{sources,concepts,entities,syntheses}/`	The worker; you may edit
Figures	`vault/attachments/<source>/`	The worker
Daily index	`vault/wiki/daily/YYYY-MM-DD.md`	Regenerated; your ticks are the only input
Home page	`vault/Home.md`, between the two marker comments	The block is the worker's; the rest is yours
Review items	`vault/wiki/review/`	The worker puts things here; you resolve them
Queue	`vault/.esbi/queue.sqlite3`	The worker
Page index (names, URLs, hashes, full text)	`vault/.esbi/index.sqlite3`	The worker; disposable, rebuilt from the vault if deleted
Run history	`vault/.esbi/runs.jsonl`	The worker. An interrupted scheduled run does not count as the day's nightly run
Lock	`vault/.esbi/run.lock` (`flock`)	A crashed run cannot leave it stuck
Mail state	`vault/.esbi/mail-state.json`	The worker: the mailbox generation (UIDVALIDITY) and the highest UID taken, a hint that lets the mail search find mail labelled after it arrived (see Which mail is taken); unreadable means none
Mail password	macOS Keychain, service `esbi-cli-imap` (the system keyring off macOS)	Never in a file
Update check and notices	`~/.cache/esbi-cli/update.json` and `notices.json` (`$XDG_CACHE_HOME/esbi-cli`)	The worker: when the update check last ran, and when the `timeout` renamed-key notice was last printed for each config (once a day); disposable
Settings	`~/.config/esbi-cli/config.toml`, or the file `--config` or `$ESBI_CONFIG` names (`sb info` prints the path)	You

The vault is its own git repository, separate from the code. Each ingested source is one commit there, so any change can be reviewed or reverted.

Design decisions, and why¶

The model returns JSON, the worker writes files. Small local models cannot be trusted with file tools, but they can fill a schema. The schema (llm/schemas.py, pydantic) is the contract, and it is also what the model is given as a JSON schema, so the model's decoder is constrained to it. Every list in the schema but a concept's aliases has a maxItems: measured on a 3B model, without it a chunk's notes ran away in 3 of 4 cases.

Reading is separate from writing. A shallow summary was a reading problem: the model saw only the first 12k characters of sources whose median is 47k. Long sources are read in full, in chunks, by a cheap model; the writing model works from the notes. They can be two different models ([llm.summarize] reads, [llm.synthesize] writes) or the same one with different limits.

One job per model call. One giant plan left the abstract empty. The core plan, the digest and the connections are three small calls; code does the merging that needs no model. Connections are a separate call because showing a small model existing pages while it summarizes makes it copy their wording into the new note.

Trust model: the worker checks what the model claims.

Claim	Check
A quote	Must appear verbatim in the source (ignoring case and accents)
A glossary term	Must appear in the source text (markup and spaces around it are stripped). Dropped as trivia: a duplicate (ignoring case, accents and punctuation, and an acronym in either letter order such as `IA` and `AI`), a generic word (`data`, `system`: a per-language list, `generic_terms`), a one-letter term (a formula symbol), a definition that says it has none (`not defined in the text`: a per-language pattern, `disclaimers`)
An entity (person, tool, company)	Its name or an alias must appear in the source text
A link to a page	Must resolve to a real page, or it is dropped
An address (Markdown link or bare URL) or an image	Kept only if the exact address is in the source; otherwise removed and counted. Image syntax and HTML are never kept
A name for a concept, entity or alias	Not `Home`, `index`, `log`, `lint`, `SCHEMA` or a date, and an alias must not already name another page
A contradiction	Only on a page on the same side of the email line as the source
A concept named like a source	Skipped: Obsidian resolves `[[links]]` by filename across folders, so it would be ambiguous
The language	The notes' language (`[notes].language`), with one retry; a second failure is accepted and reported. The check is a count of common words, so a language without a word list is never judged
A one-line summary that is a URL, a label or the title	Replaced by the first sentence of the summary
The concept diagram	Drawn by code from the extracted relations, so the Mermaid syntax is always valid. A relation whose end is neither in the source nor a concept, entity, term or alias of the note (translated or inflected is fine) is dropped, and the run says how many (`Dropped 2 diagram edges: an end was not in the source or the note.`). The prompts ask for relation ends as names of one to four words, not sentences; with fewer than two relations left the map is left out
A consolidated summary	Built only from the page's own sections. 80 to 900 characters, no links, URLs or HTML, the notes' language, and no quote, number or capitalised name that the sections do not contain. One retry; a second failure goes to `wiki/review/` and the page stays as it was

Risky changes go to review, never silently. Contradictions and lint findings become notes under wiki/review/. The worker deletes nothing of yours. Contradiction flagging is off by default because small models flag tenuous ones.

A model timeout is not "the model is down". LLMTimeout fails one chunk or one source and the run goes on. Only a real connection failure stops a run, and then nothing is counted against the sources.

An interruption is not a failure either. Ctrl-C or SIGTERM puts the source being read back in the queue with no attempt counted, and a note is never cut in half (deferred() makes a signal wait until the note and its pages are written). Before this, three interruptions parked a perfectly good source. The run exits 130 or 143, and the nightly job does not count an interrupted run as that night's run.

The daily note is a view, not a document. It is regenerated from state (source frontmatter plus the queue) on every run. The only thing read back from it is your - [x] ticks, before each regeneration. The worker never depends on text it wrote earlier in that note.

Idempotent by default. The same link twice is one queue item (tracking parameters, case and trailing slash are normalized). The same PDF twice is recognized by its bytes. A different PDF with the same file name is kept as name (2).pdf. raw/ is never overwritten.

Scheduling uses the calendar, not an interval. The nightly job is a launchd StartCalendarInterval at [run].nightly_time (default 03:00), which launchd runs at the next wake if the Mac was asleep. An hourly interval and run-at-load are only a retry net, because launchd drops interval ticks that fall while the Mac sleeps. --if-due makes the job do real work once per day, plus one more batch per hourly tick while sources are queued or readable files wait in inbox/, whatever ended the last batch (a source, token or USD limit, or an empty queue that filled again; not an outage or an interruption, which wait for the next nightly boundary), up to [run].max_batches_per_day scheduled batches since the nightly boundary (runlog.is_due: runs stopped by an outage or an interruption, and manual runs, are not counted). The run is wrapped in caffeinate -i.

Social posts and YouTube are captured by hand. Through the Obsidian Web Clipper; the worker does not scrape. The clipped text is the source.

`sb reingest`: how a rebuild stays safe¶

Commit the vault and tag it pre-reingest-<date> (an existing tag means a resumed run, and the first tag is the true "before").
For each note not yet at format: 2 (or matching the titles given): read its original from raw/ (a PDF is re-extracted, so figures come back).
Run the normal pipeline with the old title (so existing [[links]] stay valid) and the old raw path.
Only once the new plan exists, before_write removes what the old note added elsewhere: its ## From [[note]] sections in concept pages (in any language the vault was written in), and any concept page nobody else cites. A model failure before that point leaves the old note untouched.
Restore status, read, captured, content_hash; merge tags; mark format: 2; commit.

Because the marker is the note's own frontmatter, an interrupted rebuild resumes by running the command again.

Code map¶

src/esbi_cli/
  cli.py            Typer commands; the menu re-invokes them (no logic of its own)
  config.py         config.toml -> Config; find_config() is the single answer to "which config"
  doctor.py         sb doctor
  vault.py          Page/Vault: frontmatter, safe titles, find/resolve page, free_path
  gitops.py         commit only worker-managed paths
  queue.py          SQLite queue: claim, release, fail, complete; migrates old schemas
  run.py            drain the queue within limits, isolate failures per source; on_event reports progress
  interrupts.py     SIGINT/SIGTERM as an exception, raised only where it is safe (deferred() protects writes)
  jsonout.py        every --json shape (contract 1) and the JSON Lines events of `sb run --json`
  ocr_models.py     the known vision models, their memory and Ollama needs, pulling a model
  runlock.py runlog.py schedule.py   flock, run history and due-ness, launchd plist
  reingest.py       rebuild notes from raw/
  netguard.py       safe fetching: public addresses only, redirects re-checked
  capture/          inbox scan, one-time import of an older folder of links and PDFs
  fence.py scrub.py  untrusted text in prompts (made inert) and model text out of them (plain prose only)
  extract/          html (trafilatura), pdf (the PDF reader: pdf.py, pdf_text.py, pdf_figures.py, on PDFium), clip (Web Clipper notes), noise (page chrome of five sites), image (OCR)
  mail/             IMAP client (peek only), mail -> clip note (images, links), fetch with dedupe
  ingest/           chunks, read, mapreduce (long sources), plan, digest, connect, retrieve (FTS5), apply, pipeline, consolidate
  llm/              adapter (ollama, lmstudio, openai-compatible, anthropic, claude-cli, codex-cli), schemas (the contract)
  report/           daily index, index.md, read-state sync
  lint/             deterministic checks and the Lint.md report
  ask/              sb ask: retrieval, grounded answer, citation checks
  bench/            sb bench: compare models on your own material

Testing¶

uv run pytest and uv run ruff check .. Tests sit at a few seams, not at every function:

the queue (Queue), capture (scan_inbox, import_legacy, clip extraction), run_queue (which takes an injected ingest_fn), the ingest pipeline, and the CLI wiring, where only the model is faked (cli.make_llm);
FakeLLM returns queued JSON payloads in order and records the prompts it was given; there are also fakes for IMAP, the Keychain and launchctl, so no test touches the real mailbox, Keychain or launchd;
the guards (quotes, terms, entities, language, one-liners) have tests that fail when the guard is removed, which was checked by breaking each one on purpose.

A test must never run against your real vault. To try real models, use a scratch copy of the vault and a config pointing at it.

How to extend it¶

A new kind of source. Add an extractor that returns an ExtractedDoc (title, text, kind, url, optional pdf_bytes, figures, image_links), and route to it from extract_source or from the clip reader. Everything after extraction is shared. Keep the rule that the worker only fetches public http(s) addresses through netguard.

A new model provider. Implement complete_json(system, user, schema) -> str, tokens_used and sends_text_out (true unless the model runs on this machine: the privacy rules read it, and a provider without it counts as local), and add the provider to make_llm in llm/adapter.py. Nothing else knows which provider is in use.

A change to the note layout. Edit apply_plan, bump NOTE_FORMAT in ingest/apply.py, and run sb reingest to bring older notes to the new layout. The bump is what marks old notes as needing a rebuild.

A change to what the model is asked. Edit the prompt in the module that owns it and the schema in llm/schemas.py. Add the field's bound (maxItems) and a check in apply_plan if the field makes a claim about the source.

Known limits¶

Open work is in the issue tracker. The main ones: quality is bounded by the model (a 3B model writes useful but unpolished notes); retrieval is by keyword unless you turn on search by meaning; a source that needs more than about 45 chunks is read only as far as max_calls_per_source allows, and the unread tail is said in the note (raise the cap for books: Long sources); in PDFs, tables are not Markdown tables, text inside images and charts is dropped, the thresholds that tell chart text from prose and the panels of a figure were tuned on about ten papers, and the line-end hyphen rule is a heuristic (a compound such as gpt-researcher can be joined, and a broken word with a long first half keeps its hyphen); scanned PDFs and images are read only with an OCR model configured (small vision model: it can miss or swap words in dense small print, reads no handwriting reliably, and photos with no text are rejected, not described).