The Model Gateway Is a Policy Boundary, Not a Proxy -Production AI Systems - Part 2

Part 2 of 5 in Production AI Systems. Part 1 established the responsibility map around a probabilistic model.

An AI product often begins with one provider SDK imported directly into application code. That is a sensible experiment. It becomes an architectural liability when each service develops its own model names, credentials, retry rules, token accounting, streaming format, timeouts, and fallback behavior.

The common response is to add a model gateway. The dangerous response is to make that gateway responsible for every concern involving AI.

A model gateway is a policy and compatibility boundary for model access, not the owner of product behavior.

That distinction determines whether the gateway removes coupling or merely concentrates it into a new monolith.

The coupling appears before the migration

Provider coupling is often described as a future migration problem. Its immediate cost is inconsistent behavior.

Suppose three product teams generate support-case summaries, extract issue signals, and draft troubleshooting questions. Each calls a provider directly. One retries timeouts twice; another retries every error; the third does not retry. One records token usage by tenant; another records only a global total. Two send confidential data to an approved regional endpoint; the third uses a default endpoint. All three work in the happy path.

The system has no single answer to basic operational questions: Which requests are retryable? Which providers may process which data classes? How is spend attributed? What happens during quota exhaustion? Which model capabilities are required by each task?

A gateway earns its place when these policies need one enforceable boundary.

flowchart LR
    A[Product Services] --> G[Model Gateway]
    G --> P1[Provider A]
    G --> P2[Provider B]
    G --> L[Private Runtime]

The stable interface should express intent and constraints rather than reproduce one provider's request format. A request can identify the task, required capabilities, data classification, latency and cost budgets, context size, output contract, and whether streaming is required. The gateway then resolves those requirements against approved backends.

This contract should be versioned like any other internal API. Provider evolution is unavoidable; unreviewed semantic drift in the internal contract is not.

What the gateway owns

The gateway sits between product workflows and model providers. Its natural responsibilities are those that remain meaningful across products but differ across providers:

• authentication to provider endpoints;
• request, response, error, and stream normalization;
• capability and model metadata;
• routing under data, latency, cost, and availability policy;
• provider-facing rate limits and quotas;
• timeout and retry classification;
• tested fallback policy;
• token, latency, and cost attribution;
• data-region and provider allowlists;
• model-access traces.

Recent infrastructure work reflects this distinction. The Kubernetes Gateway API Inference Extension separates an inference gateway from endpoint-selection logic and uses model-serving metrics and capabilities for routing. Its details target self-hosted inference on Kubernetes, but the architectural signal is broader: inference routing requires information beyond an HTTP path or round-robin backend list.

The gateway should still expose backend limitations. If one provider supports native schema constraints and another supports only JSON mode, normalization must not claim equivalent guarantees. A stable interface should hide incidental syntax, not erase meaningful differences.

What the gateway must not own

The gateway should not decide which customer and case records a support agent may read, how a support-case summary is assembled, whether a claim is supported by evidence, or when a human must approve an outcome. Those are product and domain decisions.

flowchart TB
    subgraph Product Boundary
        A[Authorization]
        O[Workflow Orchestration]
        R[Domain Retrieval]
        V[Business Validation]
    end
    subgraph Gateway Boundary
        C[Normalized Contract]
        RT[Routing Policy]
        Q[Quotas and Retry Budgets]
        N[Provider Normalization]
    end
    subgraph Provider Boundary
        P1[Hosted Provider]
        P2[Alternative Provider]
        P3[Private Runtime]
    end
    A --> O
    R --> O
    O --> C
    C --> RT --> Q --> N
    N --> P1
    N --> P2
    N --> P3
    N --> O
    O --> V

This separation prevents two recurring failures. First, domain behavior does not become coupled to gateway releases. Second, gateway operators do not need access to business data and permissions merely to manage model traffic.

Route by requirements, not brand names

A product request should declare what it needs. A useful conceptual contract might include:

{
  "task": "support_case_summary",
  "requirements": {
    "capabilities": ["structured-output"],
    "dataClass": "confidential",
    "region": "approved-india",
    "maxLatencyMs": 5000,
    "maxCostUsd": 0.03
  },
  "generation": {
    "maxOutputTokens": 500,
    "temperature": 0.1
  },
  "schemaVersion": "support_case_summary_v3"
}

The gateway can compare these requirements with a capability registry. The registry is operational configuration, not marketing metadata. It needs model context limits, supported modalities, structured-output behavior, regional availability, approved data classes, retention policy, cost tier, and health state.

Routing should begin deterministic. A policy such as “confidential data uses approved private backends; interactive summaries require structured output and a five-second latency budget” is auditable. Quality-based or learned routing becomes defensible only after the evaluation system can show that routing decisions improve task outcomes without violating policy.

Retries and fallbacks change semantics

Retries are not a generic resilience switch. A timeout or closed connection does not prove that the provider did no work. Retrying can increase cost, produce a different answer, or duplicate a downstream effect if generation is coupled to action.

The gateway should classify failures before retrying:

Fallbacks are also compatibility decisions. A secondary model may interpret tools differently, support a smaller context, apply different safety behavior, or produce lower task quality. A configured fallback is not ready until it has passed the same contract and evaluation checks as the primary route.

Quotas are part of routing

Provider quotas, tenant budgets, and workload priority meet at the gateway. Without workload separation, a batch evaluation can consume capacity needed by an interactive product, or one tenant can exhaust a shared token budget.

The gateway can enforce requests per interval, tokens per interval, concurrent requests, daily spend, and per-workflow budgets. It should preserve distinct capacity for interactive, batch, and evaluation workloads where their service objectives differ.

This is also where cost becomes attributable. “Provider spend increased” is less actionable than “fallback retries doubled cost per accepted support-case summary for workflow version 12.” The gateway supplies call-level usage; orchestration and validation supply the final business disposition. Both are required to calculate cost per successful task.

A gateway service is not always necessary

The architectural boundary does not require a network service.

A single application using one provider with one data policy may be better served by a small internal adapter. A shared library may be enough when teams release together and centralized enforcement is not yet required. A service becomes valuable when independent products need consistent policy, multiple providers or runtimes must be supported, credentials and quotas need centralized control, or model traffic requires separate operational ownership.

The progression should follow demonstrated coupling:

direct provider client
→ application-owned adapter
→ shared contract and policy library
→ gateway service when centralized enforcement is justified

Creating the service early adds a network hop, a shared failure domain, deployment work, and an organization-wide API that will be difficult to change. The smallest boundary that provides consistent behavior is the right starting point.

Design packet: responsibility and routing

The routing decision can then be reviewed in a fixed order:

1. Is the provider allowed for this data class, tenant, and region?
2. Does the backend satisfy required capabilities and context size?
3. Is healthy quota available for this workload class?
4. Which eligible route best satisfies declared latency and cost objectives?
5. Has the route passed the task's evaluation and fallback gates?

The gateway now gives the support-case-summary workflow a stable, policy-aware path to generation. It still cannot decide whether the returned summary is trustworthy. Part 3 examines the output-trust boundary: the controls that convert generated structures into application data.

Sources

• Kubernetes Gateway API Inference Extension
• AWS Builders' Library: Timeouts, Retries, and Backoff with Jitter
• NIST: Artificial Intelligence Risk Management Framework—Generative AI Profile