RAG architecture that can be cited and audited
Grounding LLMs in HLD/LLD, ADRs and API contracts so design answers stay checkable. Part of the AI architecture pillar.
Why RAG in enterprise delivery
Architecture knowledge already exists in large programmes; it is just hard to find under time pressure. Retrieval-augmented generation indexes that corpus so an assistant can answer with pointers back to source documents. The value is faster design and review cycles with citations — not unchecked generation. See the enterprise RAG case study for how that played out in delivery practice on GCP with Java/Spring Boot integration, vector search and LLM APIs (including Vertex AI paths).
The RAG architecture diagram summarises the ingest → retrieve → cite → generate path used in that framing.
Prefer retrieval over fine-tuning for living corpora
Enterprise architecture knowledge changes weekly. Fine-tuning bakes a snapshot into weights and gives you weak citation and weak retractability when an ADR is superseded. Retrieval keeps knowledge in systems you can update, permission and audit — re-indexing a corrected decision takes minutes. The tradeoff is real: retrieval quality becomes the ceiling on answer quality, so engineering effort moves into chunking, embeddings, ranking and evaluation rather than training jobs.
Ingestion, chunking and metadata
Prefer structured sources (ADRs, OpenAPI, runbooks, HLD/LLD) over undifferentiated PDF dumps when you can. Chunk along semantic boundaries — an ADR’s decision and consequences, an API contract’s endpoint — rather than fixed token windows alone. Store document IDs, version, owner, status and ACL metadata with every chunk so answers can deep-link and access checks can run at query time. Tombstone or version-filter superseded documents so the index does not keep citing reversed decisions.
Ingestion pipelines should be idempotent and replayable: a bad parse must be fixable without rebuilding the world. Strip secrets and PII before embedding. Keep a lineage record from chunk back to source URI and commit/version so auditors can reconstruct what the assistant saw.
Retrieval quality
Hybrid search (keyword + dense vectors) usually beats either alone on technical corpora full of acronyms and class names. Re-rank top candidates before prompting. Measure recall on a fixed question set that includes ambiguous and out-of-scope prompts. When retrieval confidence is low, refuse or ask for clarification instead of improvising.
Latency budgets matter: every query pays retrieve + generate. Keep context windows intentional — pack by score and diversity, not by filling the model. For multi-hop questions, prefer a controlled second retrieval over hoping a larger window will invent structure.
Caching, freshness and token budgets
Cache embedding lookups for unchanged chunks and cache retrieval results keyed by query fingerprint plus corpus version. Invalidate on re-index. Do not cache final answers across users or tenants when ACLs differ — cached citations that ignore permissions are a quiet breach.
Token management is part of RAG design: budget tokens for system instructions, retrieved evidence and the user question separately; truncate or drop the lowest-ranking chunks first. Cap output tokens for review assistants so they draft crisply. Route generation through the LLM gateway so quotas and model allow-lists stay centralised.
Tenant and ACL isolation
Filter at query time with the caller’s identity — never rely on “we only indexed public docs” as the control. For multi-engagement corpora, partition indexes or enforce hard metadata filters so one client’s ADRs cannot appear in another’s context window. Shared platform services (embedders, re-rankers) are acceptable; shared answer memory is not.
This matches the broader tenant isolation notes on the AI pillar and the trust-boundary thinking used on cloud perimeters.
Prompting, citations and refusal
Require the model to quote or cite retrieved chunk IDs. Ban answers that invent APIs or decisions not present in context. For regulated domains, “I do not have a sourced answer” is a successful outcome. Keep prompt templates versioned alongside application code.
Structure the prompt so corpus text cannot override system policy. Prefer structured outputs (decision summary + citation list) when the consumer is a review checklist rather than free chat. Humans remain accountable for accepting a design — citations make that review possible.
Operations, evaluation and human accountability
Rebuild or incrementally update indexes when source docs change. Monitor empty-retrieval rates, citation click-through and human override rates. Run golden-question evals on every prompt or index promotion. The architect (or owning engineer) remains answerable for the design — assistance compresses the work; it does not own the decision.
Pair with security, cost caps and evaluation harnesses on the parent AI pillar, and with cloud controls for residency and private endpoints. When agents wrap RAG for multi-step work, inherit the same step caps and human gates described under agents.