Production AI infrastructure rarely fails because the model is bad. It fails because the path from the client to the model is brittle. Rate limits change. Providers degrade. Context windows grow. Logs disappear. Secrets leak. A routing layer that only forwards requests isn't enough when you need budget controls, guardrails, and operational visibility.
I built AeroLLM around the idea that an LLM proxy should be as reliable as the rest of your control plane. It's written in Go, which matters for throughput and latency in a long-lived service. It uses Redis for caching and state, OpenTelemetry for observability, and a modular internal package structure that keeps provider logic, guardrails, and orchestration separate.
What is an LLM router and why do you need one?
An LLM router sits between your application and model providers. Instead of hardcoding provider credentials and model names across services, you send requests to one endpoint and the router decides where they go. That decision can be based on cost, latency, model capabilities, availability, or custom business rules.
The immediate benefit is portability. Swap providers without changing application code. The longer-term benefit is resilience. If one provider returns 429s or elevated latency, the router can shift traffic without a deploy or a config push to every client.
AeroLLM expands that model with agentic execution. The router doesn't just forward chat completions. It can run tool calls, orchestrate multi-step reasoning, enforce budgets, and emit telemetry for every hop. That makes it closer to an inference control plane than a simple reverse proxy.
How does routing work in AeroLLM?
AeroLLM uses a universal provider registry. Each provider implements a common adapter interface, and the router normalizes streaming and non-streaming responses before they reach the client. That means the application sees a consistent API even when the backend provider changes.
Routing strategies are pluggable. Round-robin is the default. Latency-based routing prefers the fastest healthy provider. Cost-based routing prefers the cheapest model that satisfies the request. Fallback routing sends traffic to a secondary provider when the primary fails. Circuit-breaker logic stops sending traffic to providers that are repeatedly failing, then probes them for recovery.
Session stickiness, URI hash routing, and header-based routing are available for workloads that need affinity or tenant isolation. The graph orchestrator can express more complex dependencies with a DAG, running concurrent branches when requests are independent and serializing when later steps depend on earlier results.
What does the agentic runtime do?
The agentic runtime turns a chat completion request into a tool-using workflow. The model can request function calls. AeroLLM validates the request, executes the tool in an isolated runtime, captures the result, and feeds it back into the conversation loop. Retries and error recovery are built in, so a transient tool failure doesn't kill the whole agent run.
Tool execution is sandboxed. The WASM runtime executes untrusted tool code with zero-trust isolation. That matters when tools are provided by third parties or generated dynamically. The MCP server exposes those tools to external agents through a standard Model Context Protocol interface.
Memory is split between short-term message memory and long-term vector memory. Short-term memory preserves conversation state for the active session. Long-term vector memory supports retrieval across sessions, which is useful for agents that need to remember user preferences or prior decisions without stuffing everything into the context window.
What guardrails and safety controls are included?
PII redaction runs before requests leave the proxy. Prompt injection shielding inspects user content for known manipulation patterns. API key scoping binds keys to specific models, providers, and budgets so a leaked key can't pivot across the entire fleet.
FinOps controls track token usage and cost per model. Budget enforcement blocks requests that would exceed configured limits. Webhook alerts fire when spend crosses thresholds or when a provider's cost profile changes unexpectedly.
Human-in-the-loop approvals add a Redis-backed approval queue for high-risk tool calls or expensive model requests. That gives operators a chance to inspect agent behavior before it executes, without adding latency to the happy path.
The immutable audit ledger records request and response hashes in a chain. That creates tamper-evident traceability for compliance and incident review.
How do you operate and observe AeroLLM?
OpenTelemetry is first-class. Every request emits traces that cover auth, rate limiting, cache lookup, provider routing, agent loops, and usage recording. Export to any OTLP-compatible backend without changing application code.
Health and readiness endpoints support Kubernetes liveness and probes. Resilience status, SLO budgets, backpressure state, quota enforcement, and incident management are exposed as REST endpoints for automation.
Shadow traffic routes a sample of requests to secondary providers without affecting the user-facing response. That's useful for A/B testing models, validating new providers, or collecting comparison data before cutover.
Structured JSON logging, metrics, and evaluation endpoints round out the operational surface. The evaluation engine can judge responses, detect regressions, and run benchmarks against datasets.
How do you deploy AeroLLM?
The standard path is Docker Compose. One command starts the proxy with Redis for caching and state. The Kubernetes operator extends that to control-plane reconciliation for routes, budgets, agent pipelines, and policy rules.
Configuration uses Viper with config.yaml or AEROLLM_-prefixed environment variables. That makes containerized deployment straightforward and secret management compatible with most orchestration platforms.
The CLI exposes common operations: init config, build and publish WASM plugins, sync prompt templates from Git, check health, inspect resilience status, manage shadow traffic, evaluate responses, and administer policies, incidents, secrets, regions, and scheduled tasks.
FAQ
What is AeroLLM?
AeroLLM is a high-performance LLM routing and proxy server written in Go. It routes requests across multiple LLM providers, executes agentic tools, caches responses in Redis, and exposes OpenTelemetry observability.
Which LLM providers does AeroLLM support?
It supports OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI, Groq, Cohere, and DeepSeek through a universal provider registry.
What routing strategies are available?
Round-robin, latency-based, cost-based, fallback, and circuit-breaker routing. It also supports intent classification, model selection, and bandit-based optimization.
Does AeroLLM include guardrails and safety controls?
Yes. It includes PII redaction, prompt injection shielding, API key scoping, budget enforcement, and human-in-the-loop approval flows backed by Redis.
What observability does AeroLLM provide?
OpenTelemetry distributed tracing, structured logging, metrics, shadow traffic, resilience status, SLO budgets, and HAProxy-style operational endpoints.
Can AeroLLM execute external tools?
Yes. It includes an agentic tool execution runtime with hooks, retry, error recovery, WASM sandboxing, and a Model Context Protocol server for external tool integration.
Does AeroLLM support caching?
Yes. It offers exact-match caching and semantic caching via Redis, plus short-term message memory and long-term vector memory interfaces.
Is AeroLLM open source?
The repository is public at github.com/ayoubzulfiqar/aerollm, but it carries a proprietary license requiring explicit written permission for use.