AIAIBrix
AIBrix is a cloud-native infrastructure framework for large-scale LLM inference, providing scalable and cost-efficient inference components.
Inference & Runtime
Model serving, execution runtimes, acceleration, and routing for inference workloads.
TrackedUnavailableArchivedInactive
Production model serving frameworks and stacks.
AI
BEBentoML
BentoML is an open-source framework for packaging, containerizing, and deploying machine learning models into production-ready services.
BEBeta9
An open-source serverless runtime for AI workloads providing ultrafast container startup, GPU support, and scale-to-zero capabilities.
DYDynamo
Explore Dynamo by NVIDIA, an open-source framework for efficient multi-GPU inference, optimizing throughput and latency for large-scale deployments.
GOGolem
An open source durable computing platform that simplifies building and deploying highly reliable distributed systems.
GPgpustack
Open-source GPU cluster manager for efficient model training and high-performance inference orchestration.
HAHAMi
Discover HAMi, the middleware that simplifies AI resource management across diverse hardware, enhancing performance and cluster utilization in cloud-native environments.
KAKaito
Kaito is a Kubernetes AI Toolchain Operator that automates deployment and management of large-model inference and tuning workflows, with built-in RAG support and node auto-provisioning.
KSKServe
KServe: a Kubernetes-native model inference platform for scalable predictive and generative AI deployments.
KVkvcached
Virtualized elastic KV cache that brings OS-style virtual memory to LLM systems, enabling demand-driven KV allocation and improved GPU utilization.
LMLMCache
A high-performance KV cache layer for LLM serving that reduces time-to-first-token and increases throughput, especially for long-context and multi-turn scenarios.
MOModular Platform
An open, production-grade AI platform including the MAX inference server and Mojo libraries to accelerate model deployment across hardware.
NVNVIDIA Cloud Functions
NVIDIA Cloud Functions (NVCF) is a platform for deploying, managing, and running GPU-accelerated inference and streaming workloads at scale, powering build.nvidia.com.
OMoMLX
Local LLM server for Apple Silicon with continuous batching and SSD caching. Native macOS menu bar app, OpenAI-compatible API, optimized for Claude Code. Supports multiple model types including text LLMs, VLMs, and embedding models.
RORoboflow Inference
Roboflow Inference is a computer-vision inference and workflow platform that supports local and cloud deployment, video stream workflows, and rich model integrations.
SGsglang-omni
A multi-stage serving runtime for omni, speech, and TTS models from the SGLang project, designed for multi-stage decoding across heterogeneous stages and exposed through OpenAI-compatible endpoints.
SISIE
Open-source inference server and production cluster that serves all the models your agent needs with shared GPU resources.
TETensorRT-LLM
NVIDIA's open-source toolbox for optimized LLM inference, designed for efficient GPU serving and enterprise deployment.
TRTriton Inference Server
Triton Inference Server: NVIDIA's high-performance inference server supporting multiple model formats and deployment options.
VLvLLM
High-throughput, memory-efficient inference and serving engine for large language models.
Runtime engines and low-level inference kernels.
AMAmplifier
Microsoft's tooling for development and deployment assistance, aimed at performance analysis, model deployment and pipeline support for AI projects.
CACandle
Candle by Hugging Face: a minimalist, high-performance ML framework in Rust designed for serverless inference and lightweight deployments.
CHChitu
A production-focused inference framework for large language models, offering high performance, multi-hardware support, and scalable deployment.
COCoral NPU
Coral NPU is an energy-efficient machine learning accelerator core for edge devices provided by Google Coral.
DEDeepGEMM
Clean and efficient FP8 GEMM kernels with fine-grained scaling to support accurate and performant low-precision matrix computations.
EXexo
exo: Run your own AI cluster at home using everyday devices, supporting distributed inference and a ChatGPT-compatible API.
FLFlash Attention
Fast and memory-efficient exact attention implementation optimized for large Transformer training and inference.
FLFlash Linear Attention (fla)
A Triton-based, PyTorch library of efficient linear-attention kernels and models for scalable sequence modeling.
FLFlashInfer
FlashInfer is a kernel library and JIT toolset for LLM serving that implements efficient attention and sampling kernels to improve GPU throughput and latency for inference serving.
GEGenesis
Universal physics simulation and generative data platform for robotics and embodied AI, open-source physics engine.
GPgpt-oss
gpt-oss is an open-weight model series released by OpenAI, designed for high-reasoning and customizable developer use cases.
KAKAI Scheduler
A Kubernetes-native scheduler for large-scale AI workloads, providing efficient resource orchestration and optimization for containerized AI training and inference workflows.
KTKTransformers
A flexible framework for LLM inference optimizations, offering kernel injection, prefix caching and multi-level acceleration strategies.
KUKubeAI
An AI inferencing operator for Kubernetes that simplifies deploying LLMs, embeddings, and speech-to-text services.
KUKubeRay
KubeRay is the Ray Project's open-source Kubernetes operator for deploying and managing Ray applications on Kubernetes.
LILiteRT
A high-performance, scalable lightweight deep learning inference runtime for edge devices.
LLllama.cpp
llama.cpp is a lightweight LLM inference library in C/C++, designed for efficient local and cloud inference across diverse hardware.
LLllm-d
A Kubernetes-native distributed inference stack providing well‑lit paths for high-performance LLM serving across diverse accelerators.
MAMachine Learning Systems (MLSysBook)
An open-source textbook on engineering real-world AI systems, covering system design from edge devices to cloud deployment.
MIMini-SGLang
A lightweight, high-performance inference framework for large language models that balances engineering practicality with readability.
MImistral.rs
mistral.rs is a lightweight, high-performance Rust inference library for running Mistral-family models in resource-constrained environments.
MLMLX LM
A Python toolkit for running and fine-tuning LLMs on Apple Silicon, with support for quantization, distributed inference and Hugging Face integration.
MOMooncake
Mooncake is a KVCache-centric disaggregated architecture for LLM serving, providing a high-performance Transfer Engine and distributed KVCache storage.
NCNCCL
High-performance collective communication primitives for GPUs, optimized for PCIe, NVLink, NVSwitch and RDMA networks.
OLOllama
Local large language model runner that enables users to easily run and manage various open-source LLM models in local environments.
ONONNX
ONNX is an open model exchange format and ecosystem that improves interoperability between frameworks, tools, and hardware.
ONONNX Runtime
ONNX Runtime is a cross-platform, high-performance machine learning inference and training accelerator that runs models exported from PyTorch, TensorFlow/Keras and traditional ML libraries across diverse hardware.
OPOpenVINO
OpenVINO is an open-source toolkit from Intel for optimizing and deploying deep learning models for inference.
OUOutlines
A library for structured generation that simplifies producing and validating JSON/Pydantic outputs directly from LLMs.
RARamaLama
RamaLama simplifies running and serving AI models by packaging them as OCI container images and choosing hardware-optimized images for the host automatically.
SGSGLang
High-performance open-source framework for LLM and VLM inference, supporting multimodal, extreme concurrency, and flexible frontend programming.
SPSpice.ai
An open-source accelerated engine for time-series and data-grounded AI, offering SQL queries, full-text search, and LLM inference.
TItinygrad
tinygrad is a minimalist deep learning library that implements tensor operations and autodiff with very little code, making it ideal for learning and experimentation.
TRTriton
Triton is a language and toolchain for high-performance deep learning kernels and compiler development, designed to simplify GPU kernel programming while delivering strong performance.
VLvLLM Production Stack
A reference system for Kubernetes-native cluster deployment and community-driven performance optimization for vLLM.
XGXGrammar
An efficient, flexible and portable structured generation engine that enforces syntactic correctness via constrained decoding.
XIXinference (Xorbits Inference)
A model serving and inference framework that supports multiple backends, distributed deployments, and OpenAI-compatible APIs.
Sandboxes and isolated execution environments for AI workloads.
AGAgent Executor (AX)
Google's open source distributed agent runtime that coordinates agentic loops, manages executions with event logging, and provides native recovery and resumption for reliable agent deployments.
AGAgent Sandbox
An experimental sandbox project by Kubernetes SIGs aiming to provide a Kubernetes-native environment for running, orchestrating, and managing agent workloads securely and at scale.
AGAgentENV
Distributed platform for running agent environments at scale, powering agentic RL training for Kimi K3.
AIAIO Sandbox
All-in-one sandbox environment for AI agents that combines Browser, Shell, File, MCP and VSCode Server into a single containerized runtime.
BOBoxLite
A lightweight runtime and container tooling for embedding, sandboxing, and shipping AI agents.
CUCubeSandbox
A high-performance, hardware-isolated sandbox service for AI agents built on RustVMM and KVM, with E2B SDK compatibility and sub-60ms cold starts.
DADaytona
Secure and elastic infrastructure for running AI-generated code with isolated sandboxes, concurrency and persistent sandbox state.
E2E2B
Secure open source cloud runtime for AI apps & AI agents.
FLFlox
A Nix-powered, reproducible and shareable development environment and package manager.
HEherdr
The runtime coding agents live on — a Rust terminal multiplexer that keeps agent terminals running in a background server, marks every pane working, blocked, or idle, and lets agents drive it through a CLI and socket API.
KCKCoral
A lightweight benchmark server from MLC AI that executes GPU programs remotely for agentic GPU programming. It manages GPU and node resources behind a unified endpoint, with sandboxed workers and a Rust router for multi-node scheduling.
LILiteBox
A security-focused library OS that minimizes host interfaces and supports kernel- and user-mode constrained execution.
MOMonty
A minimal, secure Python interpreter written in Rust for safely executing LLM-generated Python code.
OMOM1
OpenMind's modular AI runtime for deploying multimodal agents across digital environments and physical robots
OPOpenSandbox
A universal sandbox platform for AI application scenarios, providing multi-language SDKs, a unified sandbox protocol, and extensible runtimes.
OPOpenShell
NVIDIA OpenShell is a safe, private runtime for autonomous AI agents, providing sandboxed execution environments with declarative YAML policies to protect data, credentials, and infrastructure from unauthorized access.
SASandBase Harness
A local-first, self-hosted runtime for building and operating AI agents, providing persistent sessions, sandboxed execution backends, credential vaults, approval gates, audit trails, and replay.
SASandbox Runtime
A lightweight sandboxing tool for enforcing filesystem and network restrictions on arbitrary processes at the OS level, without requiring a container.
STStrands Box
An open source sandbox engine for AI agents that combines OS-level isolation with default-deny Dogwood policies and credential injection, keeping secrets outside the agent process. Written in Rust, currently supporting macOS on Apple silicon.
GPU-oriented acceleration and performance optimization.
CUCUTLASS
CUDA Templates for Linear Algebra Subroutines (CUTLASS), a high-performance matrix computation template library provided by NVIDIA.
FLFlashMLA
Efficient multi-head latent attention kernels designed to accelerate large-scale Transformer training and inference with reduced memory footprint.
LULupine
A GPU over IP bridge that lets CPU-only machines attach and use GPUs hosted on remote machines over the network.
TITileLang
TileLang is a domain-specific language for high-performance AI kernels that simplifies writing GPU/CPU/accelerator operators.
TRTransformer Engine
Transformer Engine is an NVIDIA library focused on low-precision training and inference optimizations for Transformer models, supporting formats like FP8 to improve speed and memory efficiency.
XLXLA
XLA (Accelerated Linear Algebra) is a compiler for machine learning that generates optimized code for CPUs, GPUs, and accelerators to improve model execution performance.
On-device and local inference systems.
CACactus
An energy-efficient inference engine and numerical computing framework for phones, optimized for ARM CPUs to run large models with low power and memory footprint.
FRFreeToken
An edge-native Mixture-of-Experts serving engine that runs 290B+ frontier open-weight models locally on consumer hardware, treating GPUs, CPUs, host memory, and interconnects as one elastic inference platform.
GGggml
ggml is a lightweight tensor library for machine learning optimized for efficient model inference across hardware.
LAlaya-mlx
Native MLX runtime for Laya typed decision models: answers constrained questions (choice, rubric score, yes/no) with calibrated probabilities from a single bidirectional forward pass — 7–14 ms on M3 Max, no text generation, PyTorch, or cloud API.
TRTransformers.js
Transformers.js: a JavaScript implementation of Hugging Face Transformers for the browser and Node, with WASM/ONNX backends for optimized on-device inference.
WEWebLLM
High-performance in-browser LLM inference engine that leverages WebGPU for hardware-accelerated, privacy-preserving inference in the browser.
Routing, policy, and gateway controls for model access.
9R9router
A free AI router and token saver that connects Claude Code, Codex, Cursor, Cline, Copilot and other coding tools to 40+ providers and 100+ models, with tiered auto-fallback from subscriptions to cheap and free tiers and 20-40% token savings.
AGAgent Router
An open source control plane for AI and agent traffic built on Envoy Gateway, providing a unified OpenAI-compatible API with centralized credentials, routing, quotas, and failover. Formerly Envoy AI Gateway.
AGagentgateway
A high-performance proxy data plane for agents, providing security, observability, and governance capabilities for agent-to-agent and agent-to-tool communication.
AIAI Gateway (Portkey)
Portkey's AI Gateway is a high-performance, enterprise-ready LLM routing and governance platform that supports many model providers and rich guardrail policies.
ARArchGW
ArchGW is a model-native proxy server for agents that provides routing, guardrails, tool calling and end-to-end observability.
CLClaude Code Router
An intelligent code routing tool that optimizes Claude AI request distribution and response handling in code development, enhancing development efficiency.
CLClawRouter
ClawRouter is an agent-native LLM router empowering OpenClaw with smart routing, cost optimization, and micropayments support.
CLCloudBase AI ToolKit
CloudBase AI ToolKit provides out-of-the-box AI IDE, frontend and backend templates, and deployment pipelines to help developers quickly generate, deploy and host full-stack AI applications.
FRfreelm
freelm is an LLM router that pools the official free tiers of Gemini, Groq, OpenRouter, Cerebras, Mistral, NVIDIA NIM, Cloudflare Workers AI and other providers behind one OpenAI-compatible API, using the user's own keys. It runs in-process as a Python or TypeScript library or as a local server (`freelm serve`), routing each request by measured latency and per-model quota, with classified failure handling and racing of slow providers — free-only by design.
GAGateway API Inference Extension
Combines Gateway API with Envoy External Processing to provide a Kubernetes-native inference gateway for optimizing GenAI inference deployments.
HAHarnessRouter
Self-hosted unified interface for agent harnesses that runs Codex, Claude Code, and 26 harness bases through one OpenAI Responses-compatible API, implementing the open Unified Harness Protocol (UHP).
HIHigress
A cloud-native API gateway based on Istio and Envoy that supports Wasm plugins and AI Gateway features including MCP hosting and multi-model integrations.
KIKinference
An inference control plane for Kubernetes and vLLM that selects among inference endpoints while applying policies, routing, and load balancing. It monitors vLLM KV-cache events and engine metrics to make placement decisions.
LILiteLLM
LiteLLM is a lightweight LLM gateway and proxy framework providing a unified OpenAI-format API, routing, rate-limits, and pluggable provider integrations for production deployments.
LLllms.py
Lightweight multi-provider LLM client with an OpenAI-compatible server API and optional chat UI.
MAmagpie
A local gateway and menu-bar app that puts every coding agent's model in one place — Claude Code on Kimi, Codex on DeepSeek — switching models per agent, reusing subscriptions everywhere, and failing over across accounts and providers.
NENeMo Switchyard
Rust proxy and library for LLM traffic that routes requests across providers and translates between OpenAI and Anthropic APIs.
OMOmniRoute
A free MIT-licensed AI gateway that fronts 380 providers (150+ with free tiers) and 1,400+ models behind one endpoint for Claude Code, Codex, Cursor, Cline, and Copilot, with quota-aware auto-fallback and stacked token compression.
PLPlano
Plano is an open-source AI gateway and policy runtime for routing, securing, and observing production LLM/API traffic.
SUSub2API
A self-hosted gateway that converts Claude, OpenAI, Gemini, and Grok subscriptions into OpenAI-compatible API endpoints, with multi-account scheduling, token-level billing, and key distribution.
SUSuperagent
A secure proxy between apps, models and tools that enforces runtime protections and validates tool calls.
TOTokenHub
TokenHub is a private enterprise AI gateway that unifies multi-provider model access behind OpenAI- and Anthropic-compatible APIs, with role-based workspaces for users, team leaders, and administrators.
UNUnified AI System
An Apache-2.0 self-hosted AI gateway and MCP server that fronts OpenAI, Anthropic, and Gemini APIs with virtual keys, budgets, caching, guardrails, reverse MCP governance, and provider-free deterministic prompt enhancement.
VLvLLM Semantic Router
An intelligent Mixture-of-Models router that directs requests to the most suitable models to improve inference accuracy and efficiency.
No projects match the current filters.