A Production AI System Is More Than a Model Call - Production AI Systems - Part 1

Part 1 of 5 in Production AI Systems, a series about the deterministic engineering boundaries around probabilistic models.

A support-case-summary prototype can look complete with three boxes: an application receives case notes and attachments, sends text to a model, and displays the answer. The diagram is honest about the experiment. It is dangerously incomplete as a production design.

The first real user introduces questions that the model call cannot answer. Which support-case records may this user access? What happens when document parsing loses a column? Which prompt produced this summary? Is every claim supported by the case evidence? Can the request be retried safely? How much did a successful summary cost? What should happen when the provider times out after completing the request?

These are not peripheral concerns. They are the system.

A production AI system is a conventional distributed system containing probabilistic components.

The practical consequence is that the model should not own responsibilities that deterministic software can enforce more reliably.

The prototype boundary is too small

The direct integration encourages the application to treat generation as an ordinary remote procedure call:

flowchart LR
    U[User] --> A[Application]
    A --> M[Model API]
    M --> A
    A --> U

That boundary assumes the provider is available, the input is authorized, the prompt remains effective, the response follows its requested shape, every assertion is grounded, and a retry is harmless. None of those assumptions is stable.

Consider a plausible failure. A support agent requests a summary for a customer case. The application fetches the case notes and attachments, embeds them in a prompt, and asks for structured output. The provider returns valid JSON containing an incident detail that appeared in another customer's case record. The response is syntactically correct, fast, and inexpensive. Every model-call dashboard is green, yet the product has failed.

The defect could have entered through tenant scoping, retrieval, context construction, generation, evidence validation, or presentation. Treating the model call as the system makes those layers invisible.

Expand the system around the uncertainty

A useful production boundary separates responsibilities according to what must be deterministic and what may remain probabilistic.

flowchart TB
    U[Client] --> B[Request Boundary]
    B --> O[Workflow Orchestrator]
    O --> R[Authorized Retrieval]
    O --> G[Model Gateway]
    O --> V[Output Validation]
    O --> J[Durable Job System]
    O --> E[Evaluation Hooks]
    R --> D[(Domain Data)]
    G --> M1[Hosted Model]
    G --> M2[Private Runtime]
    V --> S[(Result and Provenance Store)]
    J --> W[Workers]
    E --> T[(Trace and Evaluation Store)]

This is not a requirement to deploy nine services. It is a map of responsibilities. A small product may implement several boxes in one process. The boundary matters before the deployment topology does.

The request boundary establishes identity, tenant, authorization, input limits, rate limits, and correlation identifiers. The orchestrator owns the product workflow: which evidence to retrieve, which prompt and schema versions to select, whether work is synchronous or durable, and what disposition follows validation. The model gateway translates product intent into provider calls and applies provider-facing policy. The validator decides whether generated output is usable. The evaluation system measures whether changes improve the task rather than merely changing the prose.

The model performs generation inside those controls. It does not replace them.

Logical boundaries before service boundaries

Separating responsibilities does not justify creating a distributed platform on day one. A modular monolith can enforce the same boundaries with ordinary functions, database tables, and explicit interfaces. Retrieval, validation, and orchestration can share one deployment while retaining different ownership in the design.

Split a responsibility into a service only when an operational reason appears: independent scaling, a distinct security boundary, separate release cadence, centralized policy across products, or a failure domain that must be isolated. Otherwise, the network adds latency and new failure modes without improving correctness.

This distinction is particularly useful in architecture reviews. The question is first, “Where is this decision enforced?” Only then should the team ask, “Which process runs it?” A clear logical boundary can move later. A missing boundary is harder to recover after product code, prompts, and provider behavior have become entangled.

Authorization ends before context begins

An especially important boundary is authorization. The model must never decide which records the caller may see. Authorization belongs in deterministic application code, and filtering must happen before content enters the model context.

flowchart LR
    Q[Support Case Request] --> A{Authorized?}
    A -->|No| X[Reject]
    A -->|Yes| F[Tenant-Scoped Fetch]
    F --> C[Authorized Context]
    C --> M[Model]

This remains true even if the prompt says, “Only use records the user may access.” Prompts express desired behavior; they are not access-control mechanisms. OWASP's current guidance similarly recommends enforcing privilege controls outside the model and treating external content as untrusted.

The same rule applies after generation. A model may propose a tool call, but application code must validate its arguments, authorize the action, check tenant ownership, and require approval where consequences justify it. The proposal is untrusted input until those checks pass.

Orchestration makes the workflow explicit

The orchestrator is often misunderstood as a place to hide a large prompt or an unconstrained agent loop. Its more valuable role is explicit coordination.

For the support-case summary, it might:

1. Load the requested support case only after authorization.
2. Select a versioned summary workflow.
3. Fetch the approved prompt and output contract.
4. Build context from authorized evidence.
5. Send generation requirements to the model gateway.
6. Validate structure, evidence, and policy.
7. Route uncertain results to review.
8. Persist the accepted output and its provenance.
9. Emit a complete trace and evaluation signals.

Explicit stages allow a failure to be assigned to the correct owner. A parsing problem is not a model-quality problem. A retrieval miss is not repaired by raising temperature. An unauthorized record is not made safe by asking the model to ignore it.

This separation also permits different execution strategies. A short, predictable classification may remain synchronous. OCR, multi-document extraction, or a workflow that waits for human review should become a durable job. The business workflow stays the same even when its execution topology changes.

Provenance is part of the result

A generated summary is not fully described by its text. Its behavior can depend on:

• application and workflow versions;
• prompt and output-schema versions;
• model and provider configuration;
• retrieval index, embedding, and reranker versions;
• source-document versions;
• sampling configuration;
• tool versions and responses.

Without these identifiers, two outputs cannot be compared or reproduced meaningfully. “The model changed” is not a diagnosis when the prompt, retrieval corpus, schema, and routing policy also changed.

The stored result should therefore include both the accepted output and enough provenance to explain its production. For sensitive workloads, that does not mean logging every raw prompt indefinitely. It means defining retention, redaction, and access policies while preserving the identifiers needed for audit and regression analysis.

NIST's Generative AI Profile frames risk management across the AI lifecycle through governance, mapping, measurement, and management. That lifecycle perspective is a useful corrective to model-centric architecture: operational trust depends on the surrounding decisions and evidence, not solely on a model evaluation performed before deployment.

Design packet: the responsibility map

The following map is the first reusable artifact in the series.

This is a design-review tool, not an organizational chart. Several responsibilities may live in one deployable unit. A problem appears when one responsibility is absent or assigned to a component incapable of enforcing it.

A complete request has one accountable path

The architecture can now describe a support-case-summary request without pretending uncertainty disappears:

sequenceDiagram
    actor U as Support Agent
    participant A as Application
    participant O as Orchestrator
    participant R as Retrieval
    participant G as Model Gateway
    participant V as Validator
    participant S as Result Store

    U->>A: Request support-case summary
    A->>A: Authenticate and authorize
    A->>O: Start versioned workflow
    O->>R: Fetch authorized evidence
    R-->>O: Case and account context
    O->>G: Generate under declared constraints
    G-->>O: Output plus usage metadata
    O->>V: Validate schema, evidence, and policy
    V-->>O: Accept, review, repair, or reject
    O->>S: Persist result and provenance
    O-->>A: Return disposition
    A-->>U: Display accepted result or status

The most important property is not the number of components. It is that every transition has an owner, a contract, and an observable disposition.

The next article narrows this architecture to the model gateway: the boundary that keeps provider-specific behavior and access policy from spreading through product code.

Sources

• NIST: Artificial Intelligence Risk Management Framework—Generative AI Profile
• OWASP: LLM Prompt Injection Prevention Cheat Sheet
• OpenTelemetry: Semantic Conventions