AeroLLM

High-performance intelligent LLM routing and proxy server

Key Facts & Overview

AeroLLM is a high-performance LLM routing and proxy server written in Go. It routes between multiple LLM providers with strategies like round-robin, latency-based, cost-based, fallback, and circuit breaker. It includes agentic tool execution, Redis caching, rate limiting, OpenTelemetry observability, guardrails, FinOps, HITL approvals, hybrid RAG, WASM sandboxing, and a Kubernetes operator.

›Request path covers recovery, logging, auth, rate limiting, injection shield, PII redaction, budget check, cache lookup, provider routing, agent loop, AIOps self-optimization, and usage recording.
›DAG-based graph orchestrator with dependency-aware concurrency.
›Hybrid RAG with dense + keyword retrieval and Reciprocal Rank Fusion.
›Immutable audit ledger, shadow traffic, and GitOps prompt versioning.

Core Architectural Features

Multi-provider routing: OpenAI, Anthropic, Gemini, Bedrock, Azure, Groq, Cohere, DeepSeek

Routing strategies: round-robin, latency, cost, fallback, circuit breaker

Agentic tool execution with hooks, retry, and error recovery

Exact-match and semantic caching with Redis

Rate limiting per API key

OpenTelemetry distributed tracing

PII redaction and prompt injection shield

Per-model cost tracking, budgets, and webhook alerts

WASM sandbox for zero-trust tool execution

Kubernetes operator for routes, budgets, and agent pipelines

Installation

git clone https://github.com/ayoubzulfiqar/aerollm.git
cd aerollm
go mod download
go build -o aerollm ./cmd/server

# Or Docker Compose
docker-compose up -d

Usage

./aerollm

# Chat completions
curl -X POST http://localhost:8080/v1/chat/completions -H "Authorization: Bearer $API_KEY" -H "Content-Type: application/json" -d '{"model":"gpt-4","messages":[{"role":"user","content":"hi"}]}'

# Health
curl http://localhost:8080/healthz

API Reference

POST /v1/chat/completions

OpenAI-compatible chat completions with provider routing.

GET /healthz

Liveness health check.

GET /readyz

Readiness health check.

POST /v1/shadow

Shadow traffic for provider comparison.

POST /mcp

Model Context Protocol server for external tools.

Resources & Repositories

GitHub ↗Docs ↗AeroLLM case study