A serviceable enterprise AI reference architecture has seven layers. Most organisations have built two or three of them well and are discovering the rest through incidents.

The model gateway. A single, controlled path to every model your organisation uses — commercial, open-weight, self-hosted. It handles routing, fallback, rate limiting, cost attribution, prompt and response logging, and access control. Without it, model choice is scattered across dozens of codebases, spend is unattributable, and swapping a model becomes a migration project. With it, model portability becomes a configuration decision, which is exactly what it should be.

The gateway is also where policy gets enforced cheaply. Data residency rules, PII redaction, blocked model versions and per-team budgets are all far easier to implement at a chokepoint than in every application.

The context layer: This is retrieval, but broader than the vector-database story. Enterprise context comes from documents, structured records, APIs, knowledge graphs, ticket histories and conversation state, and it must arrive with permissions intact. The hard problems here are not embedding quality; they are chunking strategy for real document structures, hybrid retrieval that combines semantic and keyword matching, freshness, and — the one that derails production launches — enforcing the user’s own access rights at query time rather than indexing everything into a flat, over-permissioned corpus.

Orchestration and control flow: The logic that decides what happens in what order: routing between prompts, calling tools, looping, retrying, escalating. The architectural principle worth holding onto is that non-determinism should be wrapped in deterministic scaffolding. Let the model make judgements; do not let it own control flow that a state machine could own more reliably. Systems that give the model responsibility for everything are impressive in demos and unmaintainable in production.

The action layer: Where AI systems stop reading and start doing: writing to a CRM, issuing a refund, updating a record, triggering a workflow. This layer needs the discipline of an API platform — versioned contracts, idempotency, least-privilege credentials, dry-run modes, rate limits and audit trails. Most organisations underinvest here because it looks like ordinary integration work. It is ordinary integration work, exposed to a non-deterministic caller, which raises the bar rather than lowering it.

Memory and state: Session context, user preferences, task state, and durable knowledge accumulated across interactions. This is where naive designs quietly break: unbounded context growth, stale memory that contradicts current data, no expiry policy, and no clear rule about what should persist versus what should be re-derived. Treat memory as a storage system with a schema and a retention policy, not as an ever-growing transcript.

Evaluation: The layer that separates systems you can improve from systems you can only hope about. It comprises curated test sets drawn from real usage, automated scoring, regression suites that run on every prompt or model change, and structured human review for the judgements automation cannot make. Evaluation is infrastructure, not a phase. If it is not automated, it will not survive contact with a release schedule.

Observability and guardrails: Tracing at the workflow level — not just individual model calls but the whole chain, with inputs, retrieved context, tool calls and outcomes attached. Plus the runtime controls: input validation, output filtering, prompt injection defences, sensitive-data checks and cost circuit breakers. When something goes wrong in production, the question is always “what context did it see and what did it do,” and only tracing answers it.