Enterprise AI Architecture — RAG, Agents & Governance

Start from the system, not the model

In enterprise programmes, the hard part of AI is rarely model selection. It is grounding answers in approved knowledge, keeping citations auditable, controlling cost under concurrency, and making sure a fluent wrong answer cannot bypass existing security or change-management controls. The same architecture habits that keep transaction platforms honest — explicit boundaries, idempotency, observability — apply to LLM-backed workflows.

That framing comes from applying RAG and agentic tooling to architecture and SDLC corpora inside regulated delivery, as described in the applied RAG case study and the deeper RAG architecture notes. The stack there is familiar enterprise Java territory: Spring Boot services integrating retrieval and LLM APIs on GCP (including Vertex AI paths), not a separate “AI island”.

Reference shape for an enterprise AI path

A workable shape for internal assistants and design accelerators usually has five layers:

Corpus & access — HLD/LLD, ADRs, API contracts, runbooks; ACL-aware ingestion.
Retrieval — chunking, embeddings, hybrid search, re-ranking; freshness and tombstones.
Generation — prompts with tool/policy constraints; structured outputs where possible.
Orchestration — agents only where multi-step work is justified; human approval for side effects.
Control plane — evals, tracing, cost budgets, red-team tests, audit logs.

The RAG, LLM gateway and agents diagrams show how those layers sit relative to each other without inventing a greenfield product stack.

LLM gateways and application boundaries

Do not scatter raw model SDK calls across every microservice. A thin gateway (or shared library owned by a platform team) centralises provider credentials, model allow-lists, prompt-template versions, rate limits, timeout defaults and request/response logging. Downstream Java services then call a stable internal contract — similar to how API façades front legacy cores in microservices migrations.

That boundary is also where you enforce “no training on tenant prompts”, data-residency routing to private or VPC-scoped endpoints, and per-workload quotas. Treat the gateway as a control plane hop, not a second business domain: keep it boring, versioned and observable. Pair with cloud habits for private networking and identity.

Agents: earn the extra moving parts

A single grounded retrieve-then-generate pass is enough for many architecture Q&A and review assistants. Agents (plan → tool calls → re-plan) help when the task needs multiple retrievals, structured comparisons, or draft artefacts across several sources. They also multiply failure modes: loop storms, tool misuse and runaway token spend.

Default to the simplest flow that meets the job. Add agent loops only with hard step caps, allow-listed tools, deterministic timeouts and a human gate before any write-action (tickets, config changes, merge requests). Tool results should be typed and validated before they re-enter the prompt; free-form tool output is another hallucination surface.

Security and data boundaries

Treat prompts and retrieved chunks as sensitive as the source systems. Prefer private model endpoints or VPC-scoped APIs for regulated content. Strip secrets before indexing. Enforce the same authorisation model on retrieval that the source wiki or repository already uses — retrieval that ignores ACLs is a data leak with nicer UX.

For write-actions, require explicit human confirmation and record who approved what. Prompt injection via retrieved documents is a real threat: isolate system instructions from corpus text, constrain tools, and refuse when retrieved content attempts to override policy. Red-team both the retrieval path and the tool path, not only the chat UI.

Tenant isolation

Multi-team or multi-client assistants need isolation at every layer: separate indexes or hard partition keys, query-time ACL filters, per-tenant gateway credentials or impersonation scopes, and audit trails that never mix corpora in a single cached response. Shared embedding infrastructure is fine; shared answer caches across tenants are not.

In delivery programmes this often maps to engagement or LOB boundaries rather than classic SaaS tenants — the principle is the same. See also the enterprise SaaS diagram for how control planes and data planes stay separated.

Caching and token management

Every query pays retrieve + generate. Cache stable retrieval slices (by corpus version and query fingerprint) and reuse prompt prefixes where the provider supports it. Cap max input and output tokens per workflow; pack context by relevance, not by stuffing the window. Drop low-ranking chunks before generation when they only inflate cost.

Token budgets belong next to latency and error budgets: publish them per assistant, alert when a workflow approaches the cap, and kill agent loops that exceed step or token limits. Cost is an architecture concern — an unbounded loop is a denial-of-wallet risk as much as a reliability one.

Observability and evaluation

Log traces across retrieval, prompt version, model, latency and token spend. Propagate correlation IDs from the edge through the gateway into provider calls — the same discipline used on event-driven paths. Evaluate answer quality with golden questions that include “should refuse” cases. Track empty-retrieval rate, citation click-through and human override rate as first-class SLIs.

Treat eval harnesses as part of the delivery pipeline, not a quarterly slide. When prompt or index versions change, re-run the suite before promoting. Pair offline evals with sampled human review on production traffic where risk warrants it.

Governance without freezing delivery

Governance should answer: which corpora are in scope, which models are approved, what must be cited, and when a human must sign off. Publish those as ADRs. Keep the model out of the authority path for production change — assistants accelerate design and review; they do not become the system of record.

That principle is also how I describe my own practice on the profile and resume. Model allow-lists, data-handling clauses and residency constraints belong in architecture review checklists alongside the usual API and resilience questions.

Enterprise AI architecture that stays governable