Architecture

esbi-cli is a modular monolith (one Python package, one process) organised as ports and adapters (hexagonal): the logic that decides what a note says sits in the middle, and everything that touches the outside world (a model, the network, a mailbox, the Keychain, launchd, the disk) sits behind a small seam that tests replace with a fake. This page says what that means in this code, and why; for the data flow and the file layout see Internals.

The one principle that shapes everything

The model never touches files and never needs tool calling. The worker retrieves candidate pages, asks the model for a JSON edit plan, validates it against a schema, checks what it claims against the source, and only then writes. So the model is an adapter that turns text into validated data, and nothing else. That is what lets small local models work: they only have to produce JSON, not drive tools.

Core, ports, adapters

	What it is	Where
Core	Ingest (read, plan, apply), consolidated summaries, the retrieval and ask logic, the queue and run loop, the daily index, lint, privacy rules, the machine-readable output, signal handling	ingest/ (including consolidate.py), ask/, run.py, queue.py, report/, lint/, privacy.py, vault.py, index.py, jsonout.py, interrupts.py
Ports	The few things the core needs from outside, written as a small protocol or as a callable passed in	LLM (complete_json, tokens_used, sends_text_out), MailClient, the extractor callable, ingest_fn and on_event in run_queue, the launchctl runner, the DNS resolver, the Keychain backend
Adapters	The real implementations	llm/adapter.py (Ollama, LM Studio, OpenAI-compatible, Anthropic, the claude tool, the codex tool, embeddings, OCR), ocr_models.py (which vision models exist, what each needs, and whether the local Ollama has one), mail/imap.py, extract/ (web, PDF, image, clip), netguard.py, mail/credentials.py (Keychain, or the system keyring off macOS), schedule.py (launchd), gitops.py

The dependency rule: the core depends on the ports (today the LLM protocol and its errors live in llm/adapter.py, next to the adapters) and never builds or imports a concrete provider. The CLI (cli.py) is the composition root that builds the adapters from the config (make_llm, make_embedder, make_ocr, make_mail_client) and hands them in. One known leak: the ingest pipeline asks mail/fetch.py whether a file came from email (is_mail_file, called is_mail_pdf before images were kept too), because the privacy rules need that fact. The vault is the one place the core touches the disk directly, because the vault is the product: plain Markdown files you own.

The ports are plain Python protocols and injected callables, not abstract base classes: nothing in this project needs a class hierarchy to express "a thing that returns JSON for a prompt".

Modules that carry a rule

Four modules exist for one rule each, and the rest of the code goes through them.

• interrupts.py: a signal stops a run only where that is safe. SIGINT and SIGTERM are turned into an exception in the main thread, but only inside handling(), and a signal raises (Interrupted) only inside an interruptible() block. Anything that writes more than one file wraps the writing in deferred(): a signal that arrives there waits until the block ends, so a source note, the concept pages it touched and the log line are written whole or not at all. run_queue makes the call to the model interruptible() and puts the item back in the queue without counting an attempt; sb ingest and sb reingest use exit_when_interrupted(), which exits with 128 plus the signal number. Interrupted derives from BaseException, like KeyboardInterrupt, so an except Exception meant for a failed source cannot swallow it.
• jsonout.py: the one place that knows a machine-readable shape. Commands with --json call a function in it (jsonout.status(...), jsonout.event(...), jsonout.fail(...)) and never build a dict themselves, so the contract can be read, and tested, in one file. It also owns the contract number (additive change: same number; breaking change: bump) and only_events(), which drops the human lines from stdout during sb run --json so stdout carries events and nothing else. run_queue(on_event=...) is its port: the loop reports started, source_started, source_failed through a callback and the CLI adds the rest, so the loop has no idea what JSON is.
• ingest/consolidate.py: summaries from evidence, checked before they are written. It builds the ## Summary of a concept page from the page's own From sections only (never the sources), checks the answer against those sections (size, markup, language, nothing stated that they do not say), retries once, and otherwise writes a note to wiki/review/ and leaves the page as it was. It is also where the privacy rule applies: a page that email helped write is summarised only by a model that runs on this machine. sb consolidate and the automatic pass (consolidate_due, capped by [run].max_consolidations_per_run) both call it.
• The mail-link origin and ingest(from_email=...). A link found in a mail is queued with the origin mail-link; run passes from_email=True to the ingest of any queue item with that origin, and the pipeline then marks the source kind: email, so everything downstream (the private model, the hidden-from-cloud pages) treats a page reached through a mail as email. The same reason makes .esbi/mail-pdfs.txt hold the hash of every attachment saved from a mail, images included.

Why every external effect is behind a seam

Because tests never touch the network, a real model, the Keychain or launchd. They fake at the seams and nowhere else: a FakeLLM that returns queued JSON, a fake mail client, a fake launchctl, a resolver that returns chosen addresses, a fake Keychain backend. A new adapter or a new effect gets its seam first and its fake second; code that reaches the outside world without a seam cannot be tested and does not get merged.

Patterns that earn their place

• Adapter: one per model provider and per source type, chosen by a prefix in the config (ollama/…, claude-cli/…).
• Facade: Vault hides the layout of the folders and the page index behind a few methods.
• Repository: Queue (SQLite) is the only owner of the work queue state.
• Composition root: cli.py wires the parts; no module builds its own collaborators.

Patterns are not added for their own sake: a layer or an interface that has one implementation and no seam to fake is not worth having.

Rules the design holds to

• KISS and YAGNI: the smallest change that works; a setting needs a reason to exist.
• Separation of concerns: extracting text, planning, applying and reporting are separate steps with separate tests.
• DRY, without forcing it: one source of truth for a fact (for example the language labels live in lang.py; the privacy decision lives in privacy.py).
• Safe by default: email never reaches a model that sends text away; fetching goes through netguard.safe_get only; secrets live in the Keychain.

Gotchas for developers

• iCloud and .venv. In an iCloud-synced folder (~/Documents), macOS flags files inside dot-folders as hidden and Python 3.13 then skips hidden .pth files, so import esbi_cli fails. Keep the environment outside, with UV_PROJECT_ENVIRONMENT.
• One writer at a time. RunLock is an flock, so a crashed run can never leave a stuck lock. The order in a run is: lock, then the "is it due" check, then build the model.
• One way to fetch. All URL fetching goes through netguard.safe_get (public addresses only, every redirect hop checked, the connection pinned to the address that was checked, size and time capped). Do not call httpx directly.
• The nightly time is in the plist. The launchd agent is static: changing [run].nightly_time needs sb schedule install again (sb doctor warns on a mismatch). Launchd agents cannot read ~/Documents until macOS privacy allows that binary, which only the user can grant.
• The daily note is a view. It is regenerated from state (source notes and the queue); the only input channel is the user's ticked checkboxes, read before each regeneration. Never make the worker depend on text it wrote earlier in that note. Home.md is partly the user's: only the block between the esbi markers is rewritten.
• State lives in the vault. .esbi/ holds the queue, the page index and logs. Anything that changes the queue table needs a migration (Queue.__init__ migrates old databases), because the real database lives in users' vaults.
• Injected defaults bind at definition time. A function that takes an injectable runner as a default argument (launchctl=run_launchctl) must be given it explicitly at the call site, or a test's patch is silently bypassed and the test talks to the real system.
• A signal raises only where it is safe. Handlers are installed only inside interrupts.handling() (and nothing is installed off the main thread). New code that writes several files must do it inside deferred(), or a Ctrl-C can leave half a note; new code that waits on a model or the network can be wrapped in interruptible(). A try/except Exception is not enough to clean up after an interruption, since Interrupted is a BaseException: use finally, or let run_queue release the item.
• jsonout.event(kind, /, **fields) takes its kind positionally. A step event has a field called name, and a keyword called name would collide with a parameter of the same name. Keep the /, and never name a parameter after a field an event can carry.
• Every --json command goes through jsonout. A new field is additive and keeps the contract number; a renamed or removed one is a breaking change that bumps it (and tests/test_json_output.py holds the golden copy of each shape, checked strictly: a key that a command emits and the golden shape does not list fails the test, so a new field has to be written down in the shape, and in JSON output, in the same change). Printing to stdout from a command that has --json breaks the "stdout is JSON only" rule: use stderr for anything human.
• The proxy setting is per command. netguard.use_environment_proxy is module state that load_config sets from [network] and reset_loaded() puts back to False at the start of each invocation, so the menu or a test that runs several commands in one process cannot carry a proxy setting from one to the next.
• A clip's doc.text is the stripped text. extract/clip.py removes known page chrome (extract/noise.py) before it builds the ExtractedDoc, so everything after extraction (the content hash, the raw/ snapshot, the chunks, quote checks) sees the stripped text, never the clip as it arrived; that original lives only in raw/inbox/. ExtractedDoc.stripped_lines carries the count into the note's frontmatter. A rule that works on the original clip text has to read raw/inbox/, not doc.text.
• PDFs: PDFium is not thread-safe, and its y axis points up. The PDF reader (extract/pdf.py, pdf_text.py, pdf_figures.py) opens a document, reads it and closes it in one call, in one thread; PDFium (pypdfium2's engine) does not support concurrent use, so never read two PDFs from parallel threads. Coordinates are PDF points with the origin at the bottom left (Rect is left, bottom, right, top), so a line lower on the page has a smaller baseline, and "below" means a smaller y. PDFium gives characters with a font size and weight and nothing else: paragraphs, headings and table rows are rebuilt by pdf_text.py, and a scan has no characters at all (OCR takes over below 200 characters).
• PDF tests read committed files. The fixtures are in tests/fixtures/pdf/ (a scan, a locked file, a rotated page, charts, drawn figures, a figure repeated 100 times, a huge image...), written once with a throwaway script, so no PDF library is needed to build them and the reader is tested on files it did not make. Add a case by committing a small PDF, not by generating one in the test.
• Heading names are data. Note headings come from the language catalogue (lang.py), and readers recognise the headings of every catalogued language. Do not compare against a hard-coded heading.