freshcrate — #inference

Home > #inference

Tag: #inference

11 packages • ⭐ 107,710 total stars

vllmv0.22.1🏛️ Flagship⭐77,587

A high-throughput and memory-efficient inference and serving engine for LLMs

amd blackwell cuda deepseek deepseek-v3 gpt gpt-oss inference pythonby vllm-project

faster-whisper1.2.1🏛️ Flagship⭐22,327

Faster Whisper transcription with CTranslate2

ctranslate2 inference openai pypi quantization speech transformer whisperby Guillaume Klein

ctranslate2v4.7.2🌳 Mature⭐4,444

Fast inference engine for Transformer models

cuda inference machine mkl neural nmt opennmt pypi translationby OpenNMT

xgrammarv0.2.1🌳 Mature⭐1,637

Efficient, Flexible and Portable Structured Generation

inference learning machine pypiby MLC Team

vllm-mlxv0.3.0🌳 Mature⭐917

OpenAI and Anthropic compatible server for Apple Silicon. Run LLMs and vision-language models (Llama, Qwen-VL, LLaVA) with continuous batching, MCP tool calling, and multimodal support. Native MLX bac

anthropic apple-silicon audio-processing claude-code computer-vision image-understanding inference llm pythonby waybarrios

apache-tvm-ffiv0.1.11🌿 Growing⭐377

tvm ffi

inference learning machine pypiby TVM FFI team

oramacorev1.2.38🌿 Growing⭐249

OramaCore is the complete runtime you need for your projects, answer engines, copilots, and search. It includes a fully-fledged full-text search engine, vector database, LLM interface, and many more u

fulltext-search inference llms rust vector-database vector-searchby oramasearch

llm7.iomain@2026-06-01🌿 Growing⭐142

LLM7.io offers a single API gateway that connects you to a wide array of leading AI models from various providers.

ai api artificial-intelligence inference large-language-models llm models typescriptby chigwell

rasputin-memoryv0.9.1🌱 Seedling⭐30

The memory system your AI agent deserves. 4-stage hybrid retrieval — Vector + BM25 + Knowledge Graph + Neural Reranker — in <150ms. Self-hosted, $0/query, built for agents that need to actually rememb

agent-memory ai ai-memory bm25 embeddings falkordb hybrid-search inference python ragby jcartu

tritonclient2.67.0🌱 Seedling

Python client library and utilities for communicating with Triton Inference Server

client grpc http inference pypi server service tensorrt tritonby NVIDIA Inc.

foundation-ai-agent1.0.1🌱 Seedling

12 native protocol layers for AI-agent systems. No wrappers. No SDK dependencies.

ai-agent consent crdt inference memory merkle npm protocol webrtcby Dagobert

Tag: #inference

Trending in #inference