Sinain

Ambient intelligence that sees what you see, hears what you hear, and acts on your behalf.

Quick Start · Docs · Privacy · Configuration · Contributing

Sinain captures your screen and audio continuously, runs OCR and transcription, and feeds a rolling context window to your agent. The agent analyzes what's happening, surfaces advice on a private HUD overlay, and can act on its own — fixing code, running commands, or spawning background tasks.

• Screen capture → OCR → context digest, updated every few seconds.
• System audio → transcription (local whisper.cpp or cloud) → real-time awareness.
• Private overlay: only you see it. Never in screenshots, recordings, or screen shares.

Sinain feeds the same screen and audio context to any MCP-compatible agent. Switch agents without losing context. Add new ones without reconfiguring.

• Tested with Claude Code, Codex, Goose, Junie, and Aider. Any MCP-compatible agent works.
• Knowledge modules travel with you — export from one machine, import on another.
• Run with an OpenClaw gateway, or use the shell harness (sinain-agent/run.sh) to connect your own agent.

By default, sinain uses cloud APIs (OpenRouter) for transcription and analysis. When you need tighter control, switch privacy modes — no code changes, one env var.

• off → standard → strict → paranoid — four modes in ~/.sinain/.env.
• paranoid mode: Ollama + whisper.cpp, fully offline. No network calls.
• HUD overlay is invisible to screen capture (NSWindow.sharingType = .none).

npx @geravant/sinain start

That's it. On first run, sinain will:

1. Run an interactive setup wizard — transcription backend, API key, agent, privacy mode
2. Auto-download the overlay app, sck-capture binary, and Python dependencies
3. Start all services — sinain-core, sense_client, overlay, and agent

Re-run the wizard anytime: npx @geravant/sinain start --setup

• Node.js 18+ — nodejs.org (LTS recommended)
• Python 3.10+ — brew install python3 (macOS) or python.org
• OpenRouter API key (optional for local-only mode) — openrouter.ai

Fully local? No API key needed. Ollama + whisper-cli = zero cloud. See Running Fully Local.

1. System Settings → Privacy & Security → Screen Recording — add your Terminal
2. System Settings → Privacy & Security → Microphone — add your Terminal

npx @geravant/sinain stop             # stop all services
npx @geravant/sinain status           # check what's running
npx @geravant/sinain start --setup    # re-run setup wizard
npx @geravant/sinain start --no-sense # skip screen capture
npx @geravant/sinain start --no-overlay  # headless mode

┌─── Your Device ─────────────────────────────────────────────────────┐
│                                                                     │
│  sck-capture (Swift)                                                │
│    ├─ system audio (PCM) ──► sinain-core :9500                      │
│    └─ screen frames (JPEG) ──► sense_client ─── POST /sense ──►    │
│                                                      │              │
│                              ┌────────────────────────┘              │
│                              │                                      │
│                         sinain-core                                 │
│                           ├─ audio pipeline → transcription         │
│                           ├─ agent loop → digest + HUD text         │
│                           ├─ escalation ──► OpenClaw Gateway (WS)   │
│                           │                  or sinain-agent (poll)  │
│                           └─ WebSocket feed                         │
│                                  │                                  │
│                                  ▼                                  │
│                           overlay (Flutter)                         │
│                           private, invisible to screen capture      │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
                                   │
                          ┌────────┴─────────┐
                          ▼                  ▼
                   OpenClaw Gateway    sinain-agent
                   (server or local)   (bare agent, no gateway)
                     ├─ sinain-hud plugin
                     │   └─ sinain-knowledge (curation, playbook, eval)
                     └─ SITUATION.md, Telegram alerts

All config via .env at project root (created by the setup wizard or cp .env.example .env).

The context analysis loop runs every 3–30 seconds, sending recent audio/screen context to an LLM. It produces a digest used for escalation scoring — when the score threshold is met (or always in rich mode), the digest is forwarded to the escalation agent for a full response.

See docs/CONFIGURATION.md for the full reference.

See Privacy Threat Model for the full design.

Global hotkeys use Cmd+Shift (macOS) or Ctrl+Shift (Windows):

See docs/HOTKEYS.md for all 15 shortcuts.

No cloud APIs needed. Local models handle everything:

# 1. Install local transcription
./setup-local-stt.sh

# 2. Install Ollama + vision model
brew install ollama && ollama pull llava

# 3. Start in local mode
./start-local.sh

Sinain builds a persistent knowledge graph from everything it captures — audio transcriptions, screen OCR, and agent interactions. Facts are distilled incrementally (on buffer full and session end), stored in an EAV triplestore with graph relationships, and retrieved via hybrid search (FTS5 + tag-based + entity graph backrefs with RRF fusion).

The integration step is fully deterministic — no LLM decides what to store. Every extracted fact is preserved.

npx @geravant/sinain export-knowledge   # export playbook, modules, graph
npx @geravant/sinain import-knowledge ~/sinain-knowledge-export.tar.gz

See Knowledge System docs for architecture details.

See CONTRIBUTING.md.

MIT