RAG, RINSE and REPEAT: Guidance on Building RAG Systems

We have all heard of Retrieval-Augmented Generation (RAG) systems and organizations are building Bots and Agents within their environments quickly. I remember when I built my first CMMC Agent to help my colleagues understand CMMC deadlines and individual controls. However, I worked in an environment where we operated with Zero Trust and I knew my source data did not come from any internal documents which were all classified and labeled with DLP policies built in. And I’ll  admit I pseudo manually populated it with some PowerShell. RAG systems can help businesses jump with solutions that are powerful but building them requires navigating complex engineering and security challenges. To build a resilient RAG solution, organizations must adopt secure-by-design architecture, understand the inherent vulnerabilities of AI pipelines, and implement robust guardrails at every layer.

Every RAG system needs to follow the process and a specific governed pattern to understand the risks along the way, as well as a general understanding of the systems that support individual processes involved.  A secure architecture has the following steps, as shown in the Microsoft Copilot generated image below:

1. Knowledge Sources and Indexing Pipeline

The foundation of any RAG system lies in its knowledge sources and indexing pipeline. This critical phase demands collaboration among all stakeholders to design the system, determine its usage, and define the roles involved. Before integrating any data source, it is essential to thoroughly review and approve it. Each source must be classified and labeled according to its sensitivity level. Ideally, sensitive fields and information types should be excluded from the system. However, if such elements are present, they must be redacted or masked prior to indexing or processing in the embedding pipeline.

The pipeline should also incorporate sanitization steps to identify and mitigate malicious instructions and other risks, as explored in the AI Technical Threats blog. AI advancements have introduced the vector database—a system that uses vectors to represent mathematical data instead of traditional rows and columns. To safeguard this database, it must be secured through encryption at rest, strict Identity and Access Management (IAM) controls, and network isolation.

2. Identity, Access, and Policy

Modern organizations are shifting their Identity and Access Management (IAM) strategies toward a “Zero Trust” model, which requires verifying every connection to a system and enforcing the principle of least privilege. Identity and access serve as the entry point to the system. Authorization (AuthZ) determines who is permitted to access the system, while authentication (AuthN) specifies which applications, infrastructure, or data users can access. All access requests must be evaluated by a policy and authorization layer.

There is a growing trend toward more granular access control with Attribute-Based Access Control (ABAC), where entities are tagged with attributes such as location, department, or custom labels like “SECRETPROJECT.” Workload attributes are similarly tagged to ensure alignment. For further insights, refer to the blog post on “Enhancing AI Security with Attribute Based Access.” AI models must adhere to Zero Trust principles and best practices, following standards such as NIST SP-800-63, DoDICAM, and the Identity and Access controls outlined in NIST SP-800-53. Ultimately, organizations should tailor their risk approach to their unique business requirements.

3. Application and Orchestration

This layer orchestrates the application workflow using tools like LangChain or LlamaIndex. Maintaining strict separation between system, developer, and user layers is crucial for preserving security and clarity. Each tier serves its designated purpose, and messages must be directed only to their intended recipients. In a RAG system, messages are tailored for users as well as the system itself. Developer messages can establish functions and modules, contributing to the application layer context, which must be guarded alongside system messages, especially those containing immutable rules.

A clear distinction must exist regarding the role each message serves, eliminating opportunities for prompt override or policy bypass. The challenge of jailbreaking underscores the importance of robust separation of duties. In the realm of AI, this principle is elevated, and maintaining separation between developer and end-user messages remains the most effective defense against jailbreaking.

4. The Retrieval Layer

The retrieval layer is one of the most vulnerable aspects of a RAG system because it interfaces directly with the data tier. Without authorization controls for each user, role, and request, the agent may acquire super-user privileges. It is essential for RAG systems to filter out documents that exceed the user's clearance level and implement query constraints to limit the scope of searches. Unsecured retrieval layers can essentially serve as a “natural language DLP bypass.” In practice, users do not directly request data; instead, documents are retrieved in response to their queries. If the retrieval layer is not properly secured, conversational interactions can circumvent document boundaries and data loss prevention rules, effectively turning the LLM into a summarization engine that overlooks DLP restrictions.

5. The LLM and Generation Layer Once the relevant context is retrieved, it is passed to the LLM for generation. This layer must be hardened through the use of system prompts with non‑overridable rules that clearly establish instruction precedence and enforce security boundaries. Retrieved content must be explicitly tagged and labeled as retrieved context, ensuring the model treats it strictly as reference information rather than as executable instructions. Retrieved text should be injected using context tagging, explicitly marking it as “untrusted retrieved content” so the model interprets it as data to analyze, summarize, or cite—not directives to follow. Without this explicit contextual separation, the LLM cannot reliably distinguish between system rules and policies, developer instructions, user prompts, and retrieved document text. When these boundaries blur, the result is not hallucination but instruction confusion, where untrusted content can override or interfere with intended model behavior.

6. Output Filtering and Response Before the generated response is presented to the user, it passes through an output filtering and response layer. This layer applies to Data Loss Prevention (DLP) and safety controls to inspect the model’s output for sensitive data, secrets, or policy violations before anything is disclosed. For example, if a response would expose personally identifiable information (PII), credentials, internal identifiers, or content restricted by classification or regulatory policy, the output filter can block, redact, or safely rewrite the response. This layer also enforces data minimization, ensuring the model returns only the minimal excerpts or summaries necessary to answer the user’s question, rather than reproducing or paraphrasing entire documents. By acting as a final guardrail, the output filtering layer helps prevent accidental data leakage, enforces compliance obligations, and ensures that even when retrieval or generation behaves unexpectedly, sensitive content is not exposed to the end user.

7. Observability, Output Filtering, Security, and Governance The lifecycle of a secure RAG system is continuously monitored to ensure transparency, accountability, and resilience against misuse or abuse. A well‑designed system logs prompts, retrieval events, and generated responses—using privacy‑aware redaction where appropriate—so that every interaction is explainable and auditable. These logs enable security and compliance teams to understand who accessed the system, what information was retrieved, and how the model responded. In addition to auditing, the system actively monitors for anomalous behavior, such as repeated or patterned prompt injection attempts, unusually broad or high‑volume retrieval queries, or access patterns that deviate from normal usage. For example, systematic querying across multiple sensitive topics or iterative re‑phrasing of similar questions may indicate an attempted data exfiltration or policy bypass. By detecting and responding to these signals early, organizations can intervene before meaningful impact occurs. This continuous monitoring and governance approach aligns with established risk frameworks such as the NIST AI Risk Management Framework (AI RMF), ensuring that RAG systems remain secure, compliant, and trustworthy throughout their operational lifecycle.

A resilient RAG system depends on disciplined, defense‑in‑depth risk mitigation across every layer of the pipeline. The most effective protections begin with rigorous data governance—classifying sources, excluding or redacting sensitive fields, sanitizing content before indexing, and securing vector stores with encryption and strict IAM. Zero Trust identity and policy enforcement further reduce exposure by validating every request, applying least‑privilege access, and using granular ABAC controls to prevent unauthorized retrieval. Strong orchestration boundaries ensure system, developer, and user messages remain isolated, minimizing jailbreak and override risks. At the retrieval layer, per‑request authorization, clearance‑aware filtering, and query constraints prevent the system from becoming a natural‑language DLP bypass. The generation layer must treat retrieved text as untrusted data through explicit tagging and non‑overridable system rules that preserve instruction hierarchy. Finally, output filtering, continuous monitoring, anomaly detection, and governance aligned to frameworks like NIST AI RMF ensure that even unexpected model behavior cannot leak sensitive information. Together, these techniques create a secure‑by‑design RAG architecture that is resilient, auditable, and trustworthy.