A Reference RAG Architecture on OCI

Every RAG system is the same five jobs in a trench coat: get the documents in, turn them into searchable meaning, store that meaning, find the right pieces at question time, and write an answer grounded in what was found. The jobs are stable; the choices inside each job are where architectures succeed or quietly rot. What follows is the reference design we deploy on OCI, part of our complete guide to AI on OCI, written for the team that has to operate the thing in month twelve, not just demo it in week two.

The five layers

Ingestion

Documents arrive in Object Storage, the natural staging ground: cheap, durable, event capable, and policy controlled. An event driven pipeline, Functions for light parsing or containers for heavy lifting, normalizes formats, strips boilerplate, and attaches the metadata that the rest of the system will live on: source system, document identity, version, sensitivity class, and access scope. Skimping on metadata at this layer is the original sin of RAG builds; everything downstream, filtering, citation, entitlement, depends on it.

Chunking and embedding

Documents are split into chunks, the retrievable atoms, and each chunk is embedded into a vector. Chunking is a quality decision disguised as plumbing: chunks that respect document structure, sections, headings, tables kept whole, outperform fixed windows cut mid sentence. Embedding is a commitment decision disguised as a dropdown: the model you choose here writes itself into every stored vector, and changing it later means re embedding the corpus. The selection method, and the OCI options for running it, get their own article in choosing embedding models for vector workloads on OCI.

Storage and retrieval

The vectors land in a store that can answer nearest neighbor queries fast, filtered by metadata. On OCI this is a real choice among three paths, compared below, and it is the most consequential decision in the architecture.

Generation

Retrieved chunks plus the question go to a language model, typically through the hosted models of the OCI Generative AI service, with instructions that bind the answer to the provided context and require citations. Model choice here is swappable if the architecture treats it that way; put a thin gateway in front of generation so models change by configuration, not migration.

Orchestration and observability

Something has to own the flow: query rewriting, retrieval, optional reranking, prompt assembly, response streaming, and the trace that records what was retrieved and why. Whether that something is the managed agents layer or your own service, the trace is non negotiable, because a RAG system without retrieval traces cannot be debugged, only vibed about.

The decision that shapes everything: where the vectors live

The selection logic compresses well. If the corpus is enterprise data already governed in Oracle, or answers need joins against business tables, take the in database path described in Oracle 23ai Vector Search. If the requirement is a grounded internal assistant over documents and speed matters, start with the managed path described in OCI Generative AI Agents. If retrieval itself is the product and you need keyword plus vector hybrid scoring, OpenSearch earns its operational keep. Mixed estates legitimately run two of the three; what they should not do is run one of each per team by accident.

The retrieval layer deserves engineering, not defaults

Four practices separate production retrieval from demo retrieval. Metadata filtering comes first: real questions carry structure, this region, this product, after this date, and a retriever that cannot filter on metadata answers from the wrong slice with full confidence. Hybrid retrieval comes second: vector similarity misses exact identifiers, part numbers, error codes, names, that keyword search catches trivially, so the strongest systems blend both signals. Reranking comes third: retrieve generously, then let a lightweight scoring pass pick the handful of chunks that actually answer, which costs milliseconds and buys accuracy. And query transformation comes fourth: user questions are often poor search queries, and rewriting them, expanding follow ups with conversation context, retrieves what the user meant rather than what they typed.

Security boundaries that survive review

Draw four boundaries before the first document is indexed. The corpus boundary: a knowledge store carries one sensitivity class, because retrieval does not re check document entitlements per user unless you build that check; mixed sensitivity corpora are how the intern reads the board minutes. The network boundary: every hop, ingestion, embedding, retrieval, generation, stays on private endpoints inside the VCN, with no model traffic leaving the tenancy unintended. The identity boundary: the application authenticates users and maps them to corpora; services authenticate to each other with resource principals, not embedded keys. And the audit boundary: queries, retrieved sources, and answers are logged into the same observability stack as everything else. The wider program, including data residency and model governance, is covered in governing AI workloads on OCI.

The build sequence

Evaluation: the harness is the product

The golden set from step one of the build sequence deserves its own treatment, because it quietly becomes the most valuable artifact in the system. A RAG deployment has two separable qualities, and the harness must measure them separately. Retrieval quality asks whether the right chunks surfaced, measured as recall against the known correct sources for each golden question, and it is the number to fix first because nothing downstream can compensate for it. Answer quality asks whether the generated response is faithful to the retrieved context, complete, and properly cited, scored by human graders at first and by a grading model once its agreement with the humans is established. Run the harness on every meaningful change, a new embedding model, a chunking revision, a generation model swap, a corpus refresh, and the upgrade decisions that paralyze committees become afternoon experiments with numbers attached. The same harness doubles as a regression net in production: a weekly automated run against the live system catches the slow degradations, corpus drift, index staleness, prompt rot, that no alert threshold was ever going to name. Teams that skip the harness do not actually skip evaluation; they outsource it to their users, who conduct it less systematically and report the results to the wrong people.

Cost shape

RAG costs concentrate in two places: embedding at ingestion, a large one time cost per corpus revision, and generation at query time, the continuous spend that scales with adoption and context length. Retrieval itself is comparatively cheap on all three paths. The levers that matter are context discipline, passing five good chunks beats passing twenty mediocre ones on both quality and tokens, model tiering at the generation layer, and not re embedding the corpus on a whim. Metering per application from the first week, the practice detailed in controlling AI workload costs on OCI, keeps the bill decomposable when adoption succeeds.

The independent view

RAG is where enterprise AI either earns trust or teaches the organization to ignore it, and the difference is almost never the model. It is corpus discipline, retrieval engineering, and boundaries drawn before launch, which is to say architecture and operations, the work we deliver through our OCI implementation practice on a fixed Project fee and keep healthy under Managed Monthly retainers. The reference design above is opinionated because the failure modes are predictable; where your estate justifies a different opinion, that is exactly the conversation an assessment is for.