API Gateway Design and Edge Computing

6.1 Distributed Rate-Limiting Algorithms

A rate limiter answers one question: has this caller exceeded its allowance? The four classical algorithms answer it differently, and the differences matter under burst traffic and at distributed scale.

The four algorithms

A fixed window counts requests in a clock-aligned interval and resets at the boundary. It costs one counter per caller and it has one flaw. A caller that sends its full allowance at the end of one window and again at the start of the next achieves twice the rate. With a limit of 100 per minute, a caller sends 200 requests in two seconds.

A sliding window log stores a timestamp per request and counts those inside the trailing window. It is exact and it costs memory proportional to the limit, which is unacceptable for a high limit across many callers.

A sliding window counter approximates the log. It keeps the current window's count and the previous window's count, and it weights the previous one by how much of it still falls inside the trailing window. The error is a few percent and the cost is two integers.

A token bucket holds tokens that refill at a constant rate up to a capacity. A request consumes a token, and a request with no token available is rejected or delayed. The capacity sets the allowed burst and the refill rate sets the sustained rate, which is the property that makes it the usual choice: it permits a burst without permitting a sustained overload.

A leaky bucket enforces a constant output rate by queueing. It smooths bursts completely rather than permitting them, which suits an outbound rate limit against a fragile third party and which adds latency.

  FIXED WINDOW, limit 100 per minute
    minute 1                    | minute 2
    ..................[100 reqs]|[100 reqs]..................
                       11:59:59 | 12:00:01
                       200 requests in 2 seconds, both windows legal

  SLIDING WINDOW COUNTER, same limit
    weight = fraction of the previous window still in range
    at 12:00:01, 98% of minute 1 is still inside the trailing minute
    effective count = 100 * 0.98 + 100 = 198 -> rejected

  TOKEN BUCKET, rate 100/min, capacity 100
    bucket refills at 1.67 tokens per second
    after spending 100 at 11:59:59, the bucket is empty
    at 12:00:01 it holds 3 tokens -> 3 requests allowed, not 100

The generic cell rate algorithm in the last row deserves a mention. It is a token bucket expressed as one "theoretical arrival time" value, so it costs a single timestamp per caller. It updates with arithmetic rather than with a refill loop. It is what several high-throughput limiters use internally.

Making it work across many nodes

A limiter on one node counts only that node's traffic. With 20 gateway nodes and a per-caller limit of 1,000 per minute, a naive per-node limit of 50 rejects a caller whose traffic happens to land unevenly, and a per-node limit of 1,000 permits 20,000.

Three designs solve it, and they trade accuracy against latency.

A centralized counter in Redis is exact. Every request performs one round trip to the same shard, which adds 0.2 to 1 millisecond and makes that shard a single point of failure for the limit. Sharding by caller key spreads the load, because rate limiting is naturally partitioned by the key it limits.

The operation must be atomic, and that means a script rather than a read followed by a write. Two requests that read the same count and both write back one more allow one extra request.

-- Token bucket in one atomic Redis call.
-- KEYS[1] bucket key
-- ARGV[1] capacity, ARGV[2] refill per second,
-- ARGV[3] now in ms,  ARGV[4] cost
local capacity   = tonumber(ARGV[1])
local rate       = tonumber(ARGV[2])
local now        = tonumber(ARGV[3])
local cost       = tonumber(ARGV[4])

local state   = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens  = tonumber(state[1])
local last    = tonumber(state[2])

if tokens == nil then
  tokens = capacity
  last   = now
end

-- Refill for the elapsed time, capped at capacity.
local elapsed = math.max(0, now - last) / 1000.0
tokens = math.min(capacity, tokens + elapsed * rate)

local allowed = 0
if tokens >= cost then
  tokens  = tokens - cost
  allowed = 1
end

redis.call('HMSET', KEYS[1], 'tokens', tokens, 'ts', now)
-- Expire well after the bucket would refill completely, so idle
-- callers do not accumulate keys forever.
redis.call('PEXPIRE', KEYS[1], math.ceil(capacity / rate * 2000))

return {allowed, math.floor(tokens)}

A local counter with periodic synchronization removes the round trip. Each node limits locally against its share and exchanges counts every few hundred milliseconds. The limit is approximate, with the error bounded by the traffic that arrives inside one sync interval, and the request path never waits.

A distributed token bucket with lease-based allocation sits between the two. Each node leases a block of tokens from the central store and spends them locally, returning unused tokens when the lease expires. One round trip covers many requests, and the accuracy is exact in aggregate.

Choosing the key

The limit key decides what the limiter actually protects. A key of the client IP address limits one network, which punishes an office behind one address and does nothing against a distributed source. A key of the API key or the authenticated subject limits one customer, which is the usual intent. A key of the endpoint plus the subject limits one expensive operation without restricting cheap ones.

Layered limits are the practical answer, and each layer protects a different thing. A global limit protects the origin. A per-customer limit enforces the contract. A per-endpoint-per-customer limit protects one expensive path. A per-IP limit on unauthenticated endpoints slows credential stuffing. Evaluate them cheapest first and reject at the first violation.

Cost-based limiting handles the case where requests are not equal. Charge tokens proportional to the work: one token for a point lookup, fifty for a report, two hundred for an export. GraphQL endpoints need this, because one query can be a thousand times more expensive than another at the same URL, and a request-count limit cannot express the difference.

6.2 Edge Runtime Execution

Edge computing runs application code in the points of presence that already terminate TLS and serve cached content. The motivation is latency: the edge is 10 to 30 milliseconds from a user where the origin can be 150.

Two isolation models

A traditional serverless function runs in a container or a microVM. Isolation is strong and start-up costs 100 to 500 milliseconds, which is unacceptable when the whole latency budget is 30.

V8 isolates take a different approach. One process runs many isolates, each with its own heap and its own global object, sharing one JavaScript engine and one JIT compiler. Starting an isolate costs under 5 milliseconds and a few megabytes, because the engine is already warm. Cloudflare Workers popularized this model.

WebAssembly gives the other option. A module is a sandboxed linear memory plus a validated instruction set, with no ambient authority: a module can touch only what the host explicitly imports. Instantiation costs under a millisecond from a pre-compiled module, and the model is language-agnostic, so Rust, Go, C++, and others compile to it.

The isolation column carries the risk. A V8 isolate boundary is enforced by the engine, so an engine vulnerability crosses it, which is why isolate-based platforms invest heavily in exploit mitigation and in restricting timers that enable side-channel measurement. A WebAssembly boundary is simpler to reason about, because linear memory is bounds-checked and the module has no capability it was not given.

The Bytecode Alliance ratified WASI 0.3 in June 2026, rebasing the interface on the component model's native asynchronous primitives with stream and future types and full socket support. That removed the impedance mismatch that made asynchronous work awkward. WebAssembly is production-ready for edge functions and serverless in 2026, and general-purpose backend microservices remain a stretch, largely because threading support is unfinished.

What the edge cannot do

Edge runtimes carry hard constraints, and a design that ignores them fails late.

Execution time is bounded, often to tens or hundreds of milliseconds of CPU. Memory is bounded to low tens of megabytes. There is usually no local disk. There is no persistent connection pool, because the isolate for one request does not necessarily exist for the next. And a database connection from the edge is a connection from hundreds of locations, which exhausts a connection limit designed for a handful of application servers.

The last one is the most common failure. An edge function that opens a PostgreSQL connection per request against a database with a 500 connection limit fails as soon as traffic spreads across regions. The answers are a connection pooler that the edge talks to over HTTP, a data API rather than a wire protocol, or a datastore designed for it.

Caching at the edge

The cache is the reason the edge exists, and the headers that control it are more expressive than most deployments use.

max-age sets the browser's freshness lifetime. s-maxage sets the shared cache's lifetime independently, so a response can be fresh for 60 seconds at the edge and 0 seconds in the browser. stale-while-revalidate lets the cache serve a stale response immediately while it fetches a fresh one in the background, which removes the latency cliff at expiry. stale-if-error lets it serve stale content when the origin is down, which converts an outage into staleness.

  Cache-Control: s-maxage=60, stale-while-revalidate=600

  t=0        response cached, fresh
  t=0..60    HIT, served from edge                    ~5 ms
  t=60..660  STALE HIT, served from edge immediately  ~5 ms
             and revalidated in the background
  t>660      MISS, must fetch from origin             ~150 ms

  WITHOUT stale-while-revalidate:
  t=60       every concurrent request misses at once  ~150 ms
             and they all hit the origin: a thundering herd

  Request coalescing is the companion control: the edge sends ONE
  request to the origin for a given key and fans the response out to
  every waiting client. Without it, a popular key at expiry sends
  thousands of identical requests to the origin.

Cache key design decides the hit ratio. The default key is the full URL, so a tracking parameter such as utm_source creates a separate entry per campaign and destroys the ratio for one resource. Normalize the key: strip parameters that do not affect the response, sort the ones that do, and lowercase the host.

The Vary header is the other half. Vary: Accept-Encoding is necessary and cheap. Vary: User-Agent is catastrophic, because it creates one cache entry per user agent string and there are millions. Where content genuinely differs by device class, normalize into two or three buckets at the edge and vary on the normalized value.

Purging needs a design from the start. Purge by URL is exact and unusable at scale, because one product change touches dozens of URLs. Surrogate keys, sometimes called cache tags, attach labels to a response and purge by label: tag every response that includes product 4471 with product-4471, and one purge invalidates all of them. That is the mechanism that makes a long time to live safe, because correctness comes from the purge rather than from expiry.

6.3 Edge Security and Zero Trust Authentication

The edge is the trust boundary. Everything arriving is untrusted, and the decisions made here determine what the rest of the platform has to defend against.

Validating tokens without a central bottleneck

A JSON Web Token carries claims and a signature. Validation needs the issuer's public key, the signature check, the expiry check, the audience check, and the issuer check. With an asymmetric algorithm, the public key is public, so any edge node validates locally with no call to the identity provider.

That property is what makes JWT viable at the edge. A thousand edge locations each validating locally scale linearly. The same design with opaque tokens needs an introspection call per request to a central endpoint, which adds a round trip and creates a global dependency.

The cost is revocation. A JWT is valid until it expires and nothing consults a server, so a logout or a compromised account cannot be enforced before expiry. The standard compromise is short-lived access tokens of five to fifteen minutes plus long-lived refresh tokens that are revocable, which bounds the exposure to one token lifetime.

Token translation

The edge is the right place to convert an external credential into an internal one. The external token is issued by an identity provider, it has a long audience list, it carries claims the internal system does not need, and it can arrive in a format that the internal system cannot parse.

The translation pattern validates the external token once at the edge, then mints a short-lived internal token scoped to the specific call, and forwards that. The internal token carries a narrow audience, a short expiry measured in seconds, and only the claims the downstream needs.

Three properties follow. Internal services never see the external credential, so a compromised internal service cannot replay it against the identity provider. The internal token cannot be used outside the call it was minted for. And the internal format stays stable while external identity providers change.

A gateway that forwards the original bearer token has none of those properties, and it makes every internal service a place where a customer credential can leak into a log.

API keys and hashing

An API key is a bearer credential with no expiry, which is why it needs careful handling.

Store a hash, never the key. Because an API key is high-entropy and generated rather than chosen, a slow password hash is unnecessary: a single SHA-256 is sufficient and fast. That matters, because the verification runs on every request, and Argon2 at password-grade parameters costs hundreds of milliseconds per call.

Give the key a structure that makes it operationally useful. A prefix that identifies the environment and the key type lets a secret scanner recognize it in a repository. A separate non-secret identifier lets a lookup happen by index rather than by scanning every hash. And a checksum lets the gateway reject a malformed key without a database query.

  sk_live_4f9c2a1e_7Kq3vNxPzR8mLwY2dHtJbFsA9gEuCnXi
  |  |    |         |
  |  |    |         +-- 32 chars of secret, 192 bits of entropy
  |  |    +------------ key id, non-secret, indexed lookup
  |  +----------------- environment: live or test
  +-------------------- secret key, as opposed to a publishable key

  Storage: (key_id, sha256(full_key), scopes, created_at, last_used_at,
            expires_at, revoked_at)
  Lookup:  index on key_id, then constant-time compare of the hash.

Show the full key exactly once at creation. Record last_used_at so that unused keys can be found and revoked. Support two active keys per customer so that rotation does not require downtime. And scan your own repositories and logs for the prefix, because that is where keys actually leak.

Web application firewall rules at the edge

A web application firewall inspects requests against rules and blocks matches. Running it at the edge means an attack is dropped before it consumes origin capacity.

Two rule models exist. Negative security blocks known-bad patterns, which the OWASP Core Rule Set provides for injection, cross-site scripting, and path traversal. It catches known attacks and misses novel ones. Positive security allows only what matches a schema, which catches everything else and requires a maintained schema per endpoint. An API with an OpenAPI or Protobuf definition already has that schema, which makes positive security far more achievable for an API than for a website.

The operational problem is false positives. A rule set deployed in blocking mode on day one will block legitimate traffic, and the report will be a customer complaint. Deploy in detection mode, collect matches for a full business cycle including month-end and any seasonal pattern, tune the rules that fire on legitimate traffic, and only then enable blocking. Keep a per-rule metric so that a newly noisy rule is visible before it is blocking.

Rule evaluation costs CPU, and it is proportional to the body size for rules that inspect bodies. Set a body inspection limit and decide explicitly what happens to a request that exceeds it, because "fail open" and "fail closed" are both defensible and the default is rarely the one you want.

Bot management sits alongside. Rate limiting handles volume, and it does not distinguish a scraper at a modest rate from a user. Signals that do include TLS fingerprinting, HTTP/2 frame ordering fingerprints, the coherence of headers against the claimed user agent, and behavioral analysis over a session. The correct response is usually not a block but a challenge or a tarpit, since a block tells the operator exactly what to change.

6.4 Dynamic API Aggregation and Federation

A client that needs data from six services makes six calls, each with its own round trip, and it must know about all six. Aggregation at the edge collapses that into one call over the slow link and six fast calls inside the datacenter.

The backend for frontend pattern

One aggregation layer serving every client is a compromise that serves none well. A mobile client needs small payloads and few round trips. A web client needs richer data. A partner integration needs a stable versioned contract.

A backend for frontend gives each client type its own aggregation layer, owned by the team that owns that client. The mobile layer returns exactly what the mobile screen renders. The duplication between layers is real and it is cheaper than the coupling that one shared layer creates, because each layer changes at the pace of its own client.

GraphQL federation

GraphQL lets a client specify exactly the fields it needs, which removes over-fetching and under-fetching. Federation distributes one schema across several services.

Each subgraph owns part of the schema and declares which types it can resolve and which key fields identify them. A router composes the subgraph schemas into one supergraph, plans a query across them, executes the parts, and assembles the result. Apollo Federation v2 is the established implementation and its composition was rewritten in Rust for speed and error quality. The GraphQL Foundation's Composite Schemas working group standardizes the same ideas with explicit directives, and several routers implement that instead.

  query { order(id:"4471") { total  customer { name tier }
                             lines { sku  product { title price } } } }

  SUPERGRAPH PLAN
  step 1  orders subgraph:    order(id) { total customer { __typename id }
                                          lines { sku product {__typename id} } }
  step 2  (parallel)
          customers subgraph:  _entities([{__typename:"Customer", id:"c-88"}])
                                 { name tier }
          catalog subgraph:    _entities([{__typename:"Product", id:"p-1"},
                                          {__typename:"Product", id:"p-2"}])
                                 { title price }
  step 3  router merges the entity results into the response shape

  The router batches every Product into ONE _entities call. Without
  that batching the plan would issue one call per line: the N+1 problem
  moved from the database into the query plan.

Three costs come with federation. The router is on the critical path of every request and it is a new component to operate and to scale. The query plan can be pathological, so a query depth limit, a complexity budget, and a persisted query allowlist are all necessary in production. And the N+1 problem moves into the plan, which entity batching and per-request data loaders solve.

Field-level caching is the compensating advantage. Because the router knows which subgraph produced which field, it caches per field with per-field time to live. A product title changes rarely and caches for an hour. A price changes often and caches for ten seconds. A stock level does not cache at all. One response then mixes three freshness policies, which a REST endpoint cannot express.

Protocol translation

An edge that speaks HTTP and JSON in front of gRPC services needs a translation layer. gRPC-Gateway and Envoy's gRPC-JSON transcoder both generate it from the service definition. The mapping is therefore derived rather than written.

The cost is a full decode and re-encode per request. Section 1.3 priced JSON parsing at several times Protobuf parsing, and the transcoder pays both sides. For a large payload at a high rate this becomes the gateway's dominant CPU cost, and it is worth measuring rather than assuming.

Two things do not translate cleanly and need explicit design. Streaming maps awkwardly onto plain HTTP, so server streaming becomes server-sent events or chunked JSON, and bidirectional streaming needs a WebSocket. And gRPC status codes map onto HTTP status codes imperfectly, so the mapping deserves a documented table rather than a default that surprises clients.

Response shaping and its overhead

Field selection lets a client ask for a subset, whether through a GraphQL selection set or a fields parameter on a REST endpoint. It cuts payload and it moves work to the gateway.

Three overheads are worth measuring before adopting it broadly. Transformation costs CPU proportional to the response size. Caching gets harder, because every distinct field set is a distinct cache entry, which fragments the cache exactly as a tracking parameter does. And a partial failure needs a defined semantic: if four of six upstream calls succeed, the response is partial, and the client needs to distinguish a missing field from a null value.

GraphQL answers the last point with a top-level errors array alongside partial data, which is a good design that most clients ignore. Whatever the protocol, decide the partial-failure semantic explicitly, document it, and make the client handle it, because under load partial failure is the normal case rather than the exception.

Sources and further reading

The IETF RateLimit header fields draft for the response header conventions. The Redis documentation on scripting and on atomicity. The Cloudflare Workers and Fastly Compute documentation for the isolate and WebAssembly models. The WASI 0.3 and component model specifications from the Bytecode Alliance. RFC 9111 for HTTP caching and RFC 5861 for stale-while-revalidate and stale-if-error. RFC 7519 for JSON Web Tokens and the OAuth 2.0 Security Best Current Practice. The OWASP Core Rule Set and the OWASP API Security Top 10. The Apollo Federation documentation and the GraphQL Composite Schemas working group materials.