Open source repositories tagged with #inference, ranked by health score.
Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.
A native .NET LLM inference engine for GGUF models. TensorSharp provides a console application, a web-based chatbot interface, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows/MacOS/Linux with full GPU capability
OpenVINO™ is an open source toolkit for optimizing and deploying AI inference
A unified inference and post-training framework for accelerated video generation.
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
Local LLM Testing & Benchmarking for Apple Silicon
A framework for efficient model inference with omni-modality models
Kubernetes enhancements for Network Topology Aware Gang Scheduling & Autoscaling
On-device Speech AI for Apple Silicon
Platform for deploying and routing GPU-accelerated inference, streaming, and batch workloads at scale.
