JRVS — Local-First AI Agent Framework

JRVS is a RAG-powered CLI AI assistant built for developers who want explicit control, predictable behavior, and extensible architecture when working with local language models.

Designed for privacy-sensitive, offline, or resource-constrained environments — not cloud-scale SaaS.

Most agent frameworks optimize for hosted APIs and rapid abstraction. JRVS is optimized for a different set of constraints:

• Local inference — CPU/GPU, quantized models, limited memory
• Offline or privacy-sensitive workflows — no data leaves your machine
• Explicit control — over tools, memory, retrieval, and context
• Extensibility without tight coupling — via MCP and UTCP protocols

# 1. Clone the repo
git clone https://github.com/Xthebuilder/JRVS.git
cd JRVS

# 2. Create a virtual environment (recommended)
python3 -m venv venv
source venv/bin/activate        # macOS/Linux
# venv\Scripts\activate         # Windows

# 3. Install Python dependencies
pip install -r requirements.txt

# 4. Pull a model (Ollama)
ollama pull llama3.1

# 5. Run JRVS
python main.py

• macOS: chmod +x setup_mac.sh && ./setup_mac.sh
• Windows: setup_windows.bat

pip install -r requirements.txt
ollama serve
ollama pull llama3.1
python main.py

# Install Homebrew if needed
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

brew install python@3.11 ollama node
pip3 install -r requirements.txt
ollama serve
ollama pull llama3.1
python3 main.py

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
ollama serve &
ollama pull llama3.1
python main.py

python main.py                          # Default (Ollama)
python main.py --use-lmstudio           # Use LM Studio
python main.py --theme cyberpunk        # Set theme
python main.py --model llama3.1:8b      # Set model
python main.py --debug                  # Debug mode

All options: python main.py --help

User query
    │
    ▼
Hybrid Search ──── FAISS (semantic) + FTS5 (keyword) ──── RRF fusion
    │
    ▼
Cross-Encoder Reranker (ms-marco-MiniLM-L-6-v2)
    │
    ▼
MMR Diversity Filter
    │
    ▼
Context injected into LLM prompt → Response

1. Ingestion: URLs/files → chunked → BGE embeddings → FAISS + FTS5
2. Retrieval: hybrid FAISS + keyword search → reranked → diverse context
3. Memory: every conversation turn is embedded and stored for cross-session recall

# Set your API key (one time)
export BRAVE_API_KEY="your_key_here"

# Or set at runtime
/brave-key your_key_here

# Search and auto-ingest results
/websearch what is FAISS used for

Results are automatically scraped (full page, not just snippets) and added to your knowledge base.

Drop any text-based file into the uploads/ folder:

cp my_notes.txt /path/to/JRVS/uploads/

Then ingest it:

jarvis❯ /upload ingest-all

Supports 30+ extensions: .txt, .md, .py, .js, .json, .csv, .yaml, .pdf, and more.

All variables can be set in your shell or a .env file.

Example — remote Ollama:

export OLLAMA_BASE_URL="http://192.168.1.100:11434"
export OLLAMA_DEFAULT_MODEL="llama3.1:8b"
python main.py

Configure servers in mcp_gateway/client_config.json:

{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/home/user"]
    },
    "memory": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-memory"]
    }
  }
}

JRVS auto-connects on startup. Use /mcp-servers and /mcp-tools to inspect.

Available MCP servers: filesystem, github, postgres, brave-search, memory, slack, and more.

python mcp_gateway/server.py

17 tools exposed: RAG search, web scraping, calendar, model switching, and more.

See docs/MCP_SETUP.md for full configuration.

JRVS implements UTCP — a lightweight protocol for AI agents to discover and call tools directly via their native protocols (HTTP, WebSocket, CLI).

# Start the API server
python api/server.py

# Discover available tools
curl http://localhost:8000/utcp

# Call a tool directly
curl -X POST http://localhost:8000/api/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "Hello JRVS!"}'

See docs/UTCP_GUIDE.md for details.

JRVS/
├── main.py                  # Entry point
├── config.py                # All configuration (env vars)
├── requirements.txt
│
├── rag/
│   ├── embeddings.py        # BGE-base-en-v1.5 embeddings (768-dim)
│   ├── vector_store.py      # FAISS index (HNSW auto-upgrade at 200k vectors)
│   ├── retriever.py         # Hybrid search + MMR pipeline
│   └── reranker.py          # Cross-encoder reranker
│
├── core/
│   ├── database.py          # SQLite (aiosqlite) + FTS5
│   ├── file_handler.py      # File upload ingestion
│   └── calendar.py          # Calendar parsing
│
├── llm/
│   ├── ollama_client.py     # Ollama /api/chat integration
│   └── lmstudio_client.py   # LM Studio integration
│
├── cli/
│   ├── interface.py         # JarvisCLI main loop
│   ├── commands.py          # Command routing
│   └── themes.py            # Matrix / Cyberpunk / Minimal themes
│
├── scraper/
│   ├── web_scraper.py       # BeautifulSoup scraper
│   └── brave_search.py      # Brave Search API client
│
├── api/
│   └── server.py            # FastAPI server + UTCP endpoint
│
├── mcp_gateway/
│   ├── server.py            # MCP server (17 tools)
│   ├── client.py            # MCP client
│   └── client_config.json   # MCP server configuration
│
├── uploads/                 # Drop files here for ingestion
│
└── data/                    # Auto-generated
    ├── jarvis.db            # SQLite database
    └── faiss_index.*        # Vector index

from rag.retriever import rag_retriever
from llm.ollama_client import ollama_client

# Add a document to the knowledge base
doc_id = await rag_retriever.add_document(content, title, url)

# Retrieve relevant context
context = await rag_retriever.retrieve_context(query)

# Generate a response
response = await ollama_client.generate(query, context=context)

"Cannot connect to Ollama"

ollama serve          # Start Ollama
ollama list           # Verify models are installed
ollama pull llama3.1  # Pull a model if list is empty

Or switch to LM Studio: python main.py --use-lmstudio

"Cannot connect to LM Studio"

• Open LM Studio → enable the local server → load a model

Import errors / missing packages

pip install -r requirements.txt
python --version   # Must be 3.8+

Performance issues

• Use a smaller model: ollama pull llama3.1:8b
• Reduce MAX_CONTEXT_LENGTH in config.py
• Clear the vector cache: rm data/faiss_index.*

FAISS dimension mismatch after upgrading JRVS auto-detects and rebuilds the index if the embedding model changed. Old .map files are auto-migrated to SQLite.

MCP servers not connecting

• Install Node.js: node --version and npm --version
• Check paths in mcp_gateway/client_config.json

Contributions are welcome, especially:

• Tool protocol extensions (MCP/UTCP)
• Performance improvements to the RAG pipeline
• Documentation and design feedback

Please open an issue before large changes to align on direction.

Educational and personal use. Respect website terms of service when scraping.

Ollama · FAISS · BGE Embeddings · Sentence Transformers · Rich · BeautifulSoup · Brave Search