Vectorizer

High-performance vector database and search engine in Rust for semantic search, document indexing, and AI applications. Ships as a Cargo workspace (5 crates) with binary RPC + HTTP transports, a React dashboard, and native SDKs for Rust, Python, TypeScript, Go, and C#.

• VectorizerRPC (default, port 15503) — binary MessagePack over TCP, multiplexed connection pool. See wire spec.
• REST API (port 15002) — universal HTTP fallback, powers the dashboard and any caller that doesn't speak raw TCP.
• gRPC — Qdrant-compatible service.
• GraphQL — full REST parity with async-graphql + GraphiQL playground.
• MCP — 31 focused tools for AI model integration (Cursor, Claude Desktop, etc.).
• UMICP Protocol — native JSON types + tool discovery endpoint.

• SIMD acceleration — AVX2-optimized vector ops with runtime CPU detection (5-10x faster).
• Metal GPU — macOS Apple Silicon via hive-gpu 0.2; logs render real device name, driver, VRAM.
• Sub-3ms search (CPU) / <1ms (GPU) via HNSW indexing.
• 4-5x faster than Qdrant in head-to-head benchmarks (0.16-0.23ms vs 0.80-0.87ms avg latency).

• .vecdb unified format — 20-30% space savings, automatic snapshots.
• Memory-mapped storage — datasets larger than RAM, efficient OS paging.
• Product Quantization — 64x memory reduction with minimal accuracy loss.
• Scalar Quantization + cache hit ratio metrics.

• Raft consensus via openraft (pinned =0.10.0-alpha.17) — automatic leader election in 1-5s, write-redirect via HTTP 307, WAL-backed durable replication, DNS discovery for Kubernetes headless services.
• Master-Replica — TCP streaming replication with full/partial sync, exponential reconnect backoff (5s→60s).
• Distributed sharding — horizontal scaling with automatic routing; distributed hybrid search via RemoteHybridSearch RPC with dense-only fallback for mixed-version clusters.
• HiveHub cluster mode — multi-tenant with quotas, usage tracking, tenant isolation, mandatory MMap storage, 1GB cache cap.

• Semantic similarity — Cosine, Euclidean, Dot Product.
• Hybrid search — Dense + Sparse with Reciprocal Rank Fusion (RRF).
• Intelligent search — query expansion, semantic reranking.
• Multi-collection search across projects.
• Graph relationships — automatic edge discovery, neighbor exploration, shortest-path finding.

• Built-in providers — TF-IDF, BM25, FastEmbed, BERT, MiniLM, custom models.
• Document conversion — PDF, DOCX, XLSX, PPTX, HTML, XML, images (14 formats).
• Qdrant API compatibility — Snapshots, Sharding, Cluster Management, Query (with prefetch), Search Groups, Matrix, Named Vectors (partial), PQ/Binary quantization config.
• Summarization — extractive, keyword, sentence, abstractive (OpenAI GPT).

• JWT + API Key authentication with RBAC.
• JWT secret is mandatory — boot refuses to start with empty / default / <32 char secrets when auth is enabled.
• First-run root credentials written to {data_dir}/.root_credentials (0o600), never logged.
• Payload encryption — optional ECC-P256 + AES-256-GCM, zero-knowledge, per-collection policies (docs).
• TLS 1.2/1.3 with mTLS, configurable cipher suites, ALPN.
• Per-API-key rate limiting with tiers + overrides.
• Path-traversal guard on file discovery; canonicalized base, symlink-escape refusal.

• Web Dashboard — React + TypeScript; JWT login, graph CRUD (edges, neighbors, paths), collection management, API sandbox, setup wizard with glassmorphism design. Embedded in the binary (~26MB, no external assets needed).
• Desktop GUI — Electron + vis-network for visual database management.

Highlights — see CHANGELOG.md for the full breakdown.

Breaking

• RPC is default transport (rpc.enabled: true, port 15503). REST stays on 15002. Migration guide: docs/migration/rpc-default.md. Opt out with rpc.enabled: false.
• gRPC SearchResult.score narrowed double → float. Clients on the pre-v3 proto must regenerate.
• JWT secret must be explicitly configured — no more insecure default. Generate via openssl rand -hex 64 and inject via VECTORIZER_JWT_SECRET.
• Configs moved under config/ — config.yml → config/config.yml, presets under config/presets/. Legacy ./config.yml still works with a deprecation warning (removed in v3.1).
• Cargo workspace split — vectorizer-core, vectorizer-protocol, vectorizer, vectorizer-server, vectorizer-cli. Callers reaching into the server layer need to switch from vectorizer::{server,api,grpc,logging,umicp}::* to vectorizer_server::*.

Removed

• Standalone JavaScript SDK dropped — TypeScript SDK ships compiled CJS + ESM, usable from plain JS. Migrate @hivehub/vectorizer-sdk-js → @hivehub/vectorizer-sdk.
• TypeScript SDK scope is @hivehub, not @hivellm (docs corrected).
• Framework integration packages dropped — langchain, langchain-js, langflow, n8n, tensorflow, pytorch adapters. Published versions stay installable; integrate against native SDKs directly.

Added

• Layered config loader — VECTORIZER_MODE=dev|production merges config/modes/<mode>.yml over base. Deep YAML merge with null-clear semantics. See docs/deployment/configuration.md.
• Docker collapsed to one compose with profiles — docker compose --profile <default|dev|ha|hub> up -d.
• C# SDK RPC transport (Vectorizer.Sdk.Rpc 3.0.0) — TCP + MessagePack framing, connection pool, ASP.NET Core DI.
• #![deny(missing_docs)] + cargo doc -D warnings CI gate — cleared 2,219 missing-docs warnings to 0.
• unwrap_used / expect_used denied workspace-wide — every production .unwrap() either returns Result or sits behind a documented #[allow].

Changed

• rmcp 0.10 → 1.5 — MCP SDK major rewrite; builder-based construction across every handler.
• Second-pass dep migrations — reqwest 0.13, arrow/parquet 58, zip 8, tantivy 0.26, hmac 0.13 + sha2 0.11, hf-hub 0.5, sysinfo 0.38, candle 0.10.2, bcrypt 0.19, openraft pinned =0.10.0-alpha.17.
• Frontend majors — React 19, react-router 7, TypeScript 6 (dashboard), vitest 4, eslint 10, Electron 41, Vue-router 5 (GUI).
• parking_lot migration complete — all std::sync::{Mutex,RwLock} off the hot path; CI grep gate prevents regression.
• Hot-path rand / hmac / tonic 0.14 / prost 0.14 / bincode 2.0 upgraded.

curl -fsSL https://raw.githubusercontent.com/hivellm/vectorizer/main/scripts/install.sh | bash

Installs CLI + systemd service. Commands: sudo systemctl {status|restart|stop} vectorizer, sudo journalctl -u vectorizer -f.

powershell -c "irm https://raw.githubusercontent.com/hivellm/vectorizer/main/scripts/install.ps1 | iex"

Installs CLI + Windows Service (requires Admin). Commands: Get-Service Vectorizer, {Start|Stop|Restart}-Service Vectorizer.

docker run -d \
  --name vectorizer \
  -p 15002:15002 -p 15503:15503 \
  -v $(pwd)/vectorizer-data:/vectorizer/data \
  -e VECTORIZER_AUTH_ENABLED=true \
  -e VECTORIZER_ADMIN_USERNAME=admin \
  -e VECTORIZER_ADMIN_PASSWORD=your-secure-password \
  -e VECTORIZER_JWT_SECRET=$(openssl rand -hex 64) \
  --restart unless-stopped \
  hivehub/vectorizer:latest

Docker Compose with profiles:

cp .env.example .env
# Edit .env with your credentials
docker compose --profile default up -d          # standalone
docker compose --profile dev up -d              # dev overlay
docker compose --profile ha up -d               # Raft cluster
docker compose --profile hub up -d              # multi-tenant

Profiles are mutually exclusive on host port 15002.

Images: Docker Hub · GHCR

git clone https://github.com/hivellm/vectorizer.git
cd vectorizer

cargo build --release                          # Basic
cargo build --release --features hive-gpu      # macOS Metal
cargo build --release --features full          # All features
./target/release/vectorizer

Upgrading from v2.x? RPC is now on by default on port 15503. REST is unchanged. If you can't expose the new port, set rpc.enabled: false. See v3.x migration guide.

Configs live under config/:

config/
├── config.yml             # Base config (your deployment)
├── config.example.yml     # Reference
├── modes/
│   ├── dev.yml            # Layered override: verbose logs, loopback, watcher on
│   └── production.yml     # Layered override: warn logs, larger threads/cache, zstd, scheduled snapshots
└── presets/               # Standalone full configs (legacy style)
    ├── production.yml
    ├── cluster.yml
    ├── hub.yml
    └── development.yml

Layered loader (recommended):

VECTORIZER_MODE=production ./target/release/vectorizer

Merges config/modes/production.yml over config/config.yml. Typos in the mode override fail fast at boot.

Auth is enabled by default in Docker. Default creds — change in production.

# Login
curl -X POST http://localhost:15002/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"admin"}'

# JWT in requests
curl http://localhost:15002/collections \
  -H "Authorization: Bearer YOUR_JWT_TOKEN"

# Create API key (JWT required)
curl -X POST http://localhost:15002/auth/keys \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_JWT_TOKEN" \
  -d '{"name":"Production","permissions":["read","write"],"expires_in_days":90}'

# API key in requests (NO Bearer prefix)
curl http://localhost:15002/collections \
  -H "Authorization: YOUR_API_KEY"

Production must set:

• VECTORIZER_JWT_SECRET — ≥32 chars, not the historical default. Boot aborts otherwise.
• VECTORIZER_ADMIN_PASSWORD — strong, ≥32 chars.

First-run root credentials are written to {data_dir}/.root_credentials (0o600), never printed to stdout. Read and delete after first login.

See Docker Authentication Guide and Security Policy.

• Search: 4-5x faster (0.16-0.23ms vs 0.80-0.87ms avg latency).
• Insert: Fire-and-forget pattern, configurable batch / body limits, background processing.
• Scenarios: Small (1K) / Medium (5K) / Large (10K) vectors × dimensions 384 / 512 / 768.

See Benchmark Documentation.

• MCP integration (31 tools) — native AI-agent protocol.
• Graph relationships — auto-discovery + full GUI (edges, path-finding, neighbor exploration).
• GraphQL — full REST parity + GraphiQL.
• Document processing — 14 formats built in.
• Qdrant compatibility — full API + migration tools.
• Performance — 4-5x faster than Qdrant in benchmarks.
• Binary RPC default — MessagePack over TCP on port 15503 for low-overhead client traffic.
• Complete SDK coverage — Rust, Python, TypeScript (+JS), Go, C# — all on v3.0.0.

Best fit: AI apps needing MCP, document ingestion, graph relationships, and sub-ms search with an embedded dashboard.

• RAG systems — semantic search with automatic document conversion.
• Document search — PDFs, Office, web content.
• Code analysis — semantic code navigation.
• Knowledge bases — enterprise multi-format search.

Cursor / Claude Desktop config:

{
  "mcpServers": {
    "vectorizer": {
      "url": "http://localhost:15002/mcp",
      "type": "streamablehttp"
    }
  }
}

Core operations (9) list_collections · create_collection · get_collection_info · insert_text · get_vector · update_vector · delete_vector · search · multi_collection_search

Advanced search (4) search_intelligent (query expansion) · search_semantic (reranking) · search_extra (combined) · search_hybrid (dense + sparse RRF)

Discovery & files (7) filter_collections · expand_queries · get_file_content · list_files · get_file_chunks · get_project_outline · get_related_files

Graph (8) graph_list_nodes · graph_get_neighbors · graph_find_related · graph_find_path · graph_create_edge · graph_delete_edge · graph_discover_edges · graph_discover_status

Maintenance (3) list_empty_collections · cleanup_empty_collections · get_collection_stats

Cluster-management operations are REST-only for security.

All SDKs synchronized at v3.0.0. The TypeScript SDK ships compiled CJS + ESM — usable from plain JavaScript, no separate JS package needed.

Every SDK accepts both vectorizer://host[:port] (RPC, default port 15503) and http(s)://host[:port] (REST) URLs through the same endpoint parser.

• Config migration — parse Qdrant YAML/JSON → Vectorizer format.
• Data migration — export from Qdrant, import into Vectorizer.
• Validation — integrity + compatibility checks.
• REST compatibility — full Qdrant API at /qdrant/*.

use vectorizer::migration::qdrant::{QdrantDataExporter, QdrantDataImporter};

let exported = QdrantDataExporter::export_collection(
    "http://localhost:6333",
    "my_collection"
).await?;

let result = QdrantDataImporter::import_collection(&store, &exported).await?;

See Qdrant Migration Guide.

Multi-tenant cluster mode integration with HiveHub.Cloud.

• Tenant isolation — owner-scoped collections.
• Quota enforcement — collections / vectors / storage per tenant.
• Usage tracking — automatic reporting.
• User-scoped backups.

hub:
  enabled: true
  api_url: "https://api.hivehub.cloud"
  tenant_isolation: "collection"
  usage_report_interval: 300

export HIVEHUB_SERVICE_API_KEY="your-service-api-key"

Cluster-mode requirements (enforced at boot):

cluster:
  enabled: true
  node_id: "node-1"
  memory:
    max_cache_memory_bytes: 1073741824
    enforce_mmap_storage: true
    disable_file_watcher: true
    strict_validation: true

See HiveHub Integration and Cluster Memory Limits.

crates/
├── vectorizer-core/       # Foundation: error, codec, quantization, simd, compression, paths
├── vectorizer-protocol/   # RPC wire types + tonic-generated gRPC
├── vectorizer/            # Engine (umbrella): db, embedding, models, cache, persistence, search, ...
├── vectorizer-server/     # Transport: HTTP / gRPC / MCP / RPC + binary
└── vectorizer-cli/        # CLI binaries
sdks/rust/                 # Rust SDK — re-exports vectorizer-protocol wire types

Runtime directories resolve to platform-standard locations (~/.local/share/vectorizer/ on Linux, ~/Library/Application Support/vectorizer/ on macOS, %APPDATA%\vectorizer\ on Windows), overridable via VECTORIZER_DATA_DIR / VECTORIZER_LOGS_DIR.

• User Documentation — install + tutorials
• API Reference — REST
• VectorizerRPC Spec — wire protocol
• RPC Operator Guide
• Configuration — layered loader
• v3.x Migration — RPC-default rollout
• Dashboard Integration
• Qdrant Compatibility
• HiveHub Integration
• Cluster Memory Limits
• MCP Guide
• Encryption
• Technical Specs — architecture, performance, implementation

Apache License 2.0 — see LICENSE.

See CONTRIBUTING.md.