Your RAG Post-Filter Is a Leak: Permission-Aware Retrieval in Django (Part 4)

Short version

• Post-filtering RAG is a leak pattern. Retrieve top-k by similarity, then drop what the user can't see, and you get empty answers for narrow users, restricted text in logs and rerankers, and a filter every new code path has to remember.
• Pre-filter permissions into the vector query. Every chunk carries tenant_id and allowed_principals. At query time you resolve the caller's principals and rank only over the intersection. Forbidden text never becomes a candidate.
• Use the same door as tools. Part 2's scoped grant already knows who the agent acts for. Retrieval takes that user (or the grant) and applies the same visible set. No service-account index scan.
• Keep a fail-closed ORM re-check before anything hits the prompt. Pre-filter is the primary mechanism; the re-check is a safety net for ACL lag (Pavel Duglas, Warden).
• 12 pytest tests in the demo prove: Alice never sees Bob's canary, shared docs appear for both, revoked shares disappear, and an agent grant returns the same chunks as the human.

Most RAG tutorials stop at "embed the docs, search by cosine, stuff the top-k into the prompt." That is fine for a personal notes app. It is wrong the moment two users in the same tenant have different document access.

The failure mode is quiet. Tenant isolation still passes. Customer A never sees Customer B. Inside the tenant, a junior sales rep asks the assistant about "the reorg," and the answer quotes an HR memo only three directors can open. The vector index never asked who was asking. It only asked what was similar. Pavel Duglas has written the cleanest public version of this audit finding I have seen: the core app has a careful permission model, then retrieval flattens it into "everything in the tenant" (Permission-Aware RAG for Multi-Tenant SaaS).

This is Part 4 of Shipping AI Agents in Django. Part 1 put a human approval gate in front of destructive actions. Part 2 gave each agent run a scoped, 15-minute grant checked against the human's own Django permissions. Part 3 exposed that same door over MCP so a second surface could not invent a second rule set. Retrieval is the next surface. If tools go through visible_to(user) and RAG scans the whole tenant index, you have rebuilt the Instatic shape from Part 3: one API that enforces object rules, and another path that does not.

Opinion, labelled as such: if your assistant can quote a document the signed-in user cannot open in the UI, that is a bug, not a clever agent.

The leak, in one diagram

Post-filter (common, wrong as the primary control)
  Query ──► ANN over tenant index ──► top-k texts ──► drop ACL ──► prompt
                 ▲                         │
                 │                    logs / reranker / traces
                 └──────── already saw restricted chunks ────────┘

Pre-filter (what to ship)
  Query + user ──► principals ──► ANN WHERE tenant AND ACL overlap ──► top-k ──► prompt
                                      ▲
                                      └── forbidden chunks never candidates

Three reasons post-filter fails as the primary control, following Duglas:

1. Empty results. Fetch 10, keep 1, answer quality collapses for every narrow user.
2. Things see data before the filter. Rerankers, debug middleware, and GenAI traces often touch the full top-k. Restricted text sits in logs and sometimes in a third-party call.
3. It is easy to forget. One new code path that skips the drop and you have a leak. Security that depends on every developer remembering a step is not security.

Microsoft's secure multitenant RAG guidance makes the same architectural point: every retrieval request must include a tenant discriminator and user-level authorization filters, applied before grounding data reaches the model (Secure Multitenant RAG). Supabase's docs show the database version of the same idea: RLS on document_sections so a similarity query only returns rows the caller may read (RAG with Permissions).

Same door as tools and MCP

Parts 2 and 3 already argued that every agent action should go through one function: check the grant, check the human's Django permission, apply the row-level queryset, write an audit row. Retrieval is an action. If list_open_invoices uses Invoice.objects.visible_to(user) and retrieve() scans every chunk in the tenant, the assistant has a second, quieter read path.

Wire it the boring way:

Agent runtime
   │  Bearer grant (Part 2) or signed-in user
   ▼
retrieve(grant_or_user, tenant_id, query, k)
   │  resolve grant → on_behalf_of
   │  docs.view_document still held?
   │  principals_for(user, tenant)
   ▼
visible_chunks(...)  →  rank  →  recheck_against_orm  →  prompt

Do not hand the model a tenant_id tool argument and trust it. Bind the principal server-side to the session or the grant, the same way Part 3's MCP tools refuse to run without a verified bearer. The model may influence the query string. It must not influence who is asking.

Step 1: put ACL metadata on every chunk

When you chunk a document, copy a flat principal list onto the chunk. Do not store nested ACL trees in the index. Vector filters are good at arrays and bad at nested logic.

class Document(models.Model):
    tenant = models.ForeignKey(Tenant, on_delete=models.CASCADE)
    title = models.CharField(max_length=200)
    owner = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.PROTECT)
    allowed_principals = models.JSONField(default=list)  # ["user:12", "group:sales"]
    acl_version = models.PositiveIntegerField(default=1)


class Chunk(models.Model):
    document = models.ForeignKey(Document, on_delete=models.CASCADE, related_name="chunks")
    tenant = models.ForeignKey(Tenant, on_delete=models.CASCADE)
    text = models.TextField()
    allowed_principals = models.JSONField(default=list)  # denormalized
    acl_version = models.PositiveIntegerField(default=1)
    embedding = models.JSONField(default=list)  # stub floats in the demo

Principals are a single namespace: user:…, group:…, role:…. Share with a team via group:sales, not a list of every member. When someone leaves the team, their principal set shrinks at query time and you do not re-embed.

The demo stores embeddings as JSON float lists and ranks with cosine similarity so the tests run without pgvector or a hosted index. In production you would push the same filters into pgvector (or your vector store's metadata filter) the way Supabase does with RLS on the sections table.

Step 2: resolve principals at query time, never "tenant_id only"

def principals_for(user, tenant_id: int) -> list[str]:
    membership = Membership.objects.filter(user=user, tenant_id=tenant_id).first()
    if membership is None:
        return []  # fail closed
    principals = [f"user:{user.pk}"]
    principals += [f"group:{g}" for g in membership.groups]
    principals += [f"role:{g.name}" for g in user.groups.all()]
    return principals

The search function takes a user (or a Part 2 grant), not a bare tenant_id. Duglas's rule: make it impossible to call search without an identity. Background jobs that truly need a system scan get a separately named function that makes reviewers nervous.

Step 3: retrieve() pre-filters, then ranks

def visible_chunks(user, tenant_id: int):
    principals = principals_for(user, tenant_id)
    if not principals:
        return Chunk.objects.none()
    qs = Chunk.objects.filter(tenant_id=tenant_id).select_related("document")
    allowed = [c.pk for c in qs if set(c.allowed_principals) & set(principals)]
    return Chunk.objects.filter(pk__in=allowed).select_related("document")


def retrieve(user_or_grant, *, tenant_id: int, query: str, k: int = 5):
    user = _require_user(user_or_grant)  # User or usable AgentGrant
    if not user.has_perm("docs.view_document"):
        raise RetrievalDenied("missing docs.view_document")
    candidates = list(visible_chunks(user, tenant_id))
    return _rank(candidates, embed(query), k)

On SQLite the principal overlap happens in process after the tenant cut. That is still pre-filter: ranking never sees forbidden text. On Postgres you would express the overlap in SQL (jsonb / array overlap, or RLS) so the ANN index itself never returns those rows.

An agent call uses the same function:

def retrieve_for_token(raw_token: str, *, tenant_id: int, query: str, k: int = 5):
    grant = resolve_grant(raw_token)  # Part 2: expired / revoked / agent off → None
    if grant is None:
        raise RetrievalDenied("invalid or expired grant")
    return retrieve(grant, tenant_id=tenant_id, query=query, k=k)

_require_user unwraps the grant to on_behalf_of and checks docs:read is in the grant scopes. The agent cannot retrieve what the human cannot read, and cannot retrieve without the Django permission either. Same door as call_tool() from Parts 2 and 3.

Step 4: contrast — what post-filter leaks

The demo ships a deliberate retrieve_postfilter() so the tests can show the failure:

def retrieve_postfilter(user_or_grant, *, tenant_id: int, query: str, k: int = 5):
    user = _require_user(user_or_grant)
    all_tenant = list(Chunk.objects.filter(tenant_id=tenant_id))
    ranked = _rank(all_tenant, embed(query), k=max(k * 3, 10))
    intermediate_texts = [h.text for h in ranked]  # ← restricted text is here
    principals = set(principals_for(user, tenant_id))
    allowed_ids = {
        c.pk for c in all_tenant if set(c.allowed_principals) & principals
    }
    final = [h for h in ranked if h.chunk_id in allowed_ids][:k]
    return {"hits": final, "intermediate_texts": intermediate_texts, ...}

Seed Bob's private chunk with a canary string CANARY-7Q2X. Alice's post-filter path drops it from the final hits — and still holds it in intermediate_texts. That is the log line, the reranker payload, the span attribute you forgot to redact. Alice's pre-filter path never has the string at all.

Warden's README frames a useful refinement: treat the vector pre-filter as a performance optimization that must be a permissive superset, and keep an authoritative check (their Gate 2) as the security boundary before context (geminimir/warden). That matches Duglas's "re-check against the source of truth before answering." In Django terms:

def recheck_against_orm(user, hits):
    """Safety net before the prompt. Drops stale ACL hits if the index lagged."""
    docs = {d.pk: d for d in Document.objects.filter(pk__in={h.document_id for h in hits})}
    kept = []
    for h in hits:
        doc = docs.get(h.document_id)
        if doc and set(principals_for(user, doc.tenant_id)) & set(doc.allowed_principals):
            kept.append(h)
    return kept

Pre-filter is how you get useful top-k. The ORM re-check is how you survive a delayed share-revoke worker. Revocations may propagate lazily; grants should propagate eagerly. Stale grants quietly destroy recall. Stale revokes must not leak — Gate 2 / recheck_against_orm exists so they do not.

Step 5: shares and revokes without re-embedding

ACL changes are metadata patches:

def share_document(doc, principal: str):
    principals = list(dict.fromkeys([*doc.allowed_principals, principal]))
    doc.allowed_principals = principals
    doc.acl_version += 1
    doc.save(update_fields=["allowed_principals", "acl_version", "is_shared"])
    doc.chunks.update(allowed_principals=principals, acl_version=doc.acl_version)


def revoke_share(doc, principal: str):
    principals = [p for p in doc.allowed_principals if p != principal]
    doc.allowed_principals = principals
    doc.acl_version += 1
    doc.save(update_fields=["allowed_principals", "acl_version", "is_shared"])
    doc.chunks.update(allowed_principals=principals, acl_version=doc.acl_version)

No re-embed. The next retrieve() for the revoked user simply does not match. The test suite shares Alice's private plan with Bob, confirms he retrieves it, revokes, and asserts it is gone. A second test feeds the stale hits into recheck_against_orm after revoke and asserts the safety net drops them even if the index lagged.

Step 6: what to put in traces (and what not to)

If you emit OpenTelemetry GenAI spans, prefer recording retrieval metadata (operation name, top-k, document ids after the ACL filter) over raw chunk text. The May 2026 OpenTelemetry GenAI observability walkthrough is explicit that content capture is opt-in because prompts and tool results can hold sensitive data (Inside the LLM Call). Opinion: treat pre-filter intermediate sets the same way you would treat a password field — they should not exist, and if a debug flag materializes them, keep them out of exporters by default. Pipeline item #9 in this series is the deeper tracing post; here the rule is simpler: never log chunk text from before the permission filter.

The test suite that goes looking for leaks

Happy-path demos do not catch this. The demo under /workspace/hashnode/demo05 seeds one tenant with Alice-only, Bob-only (canary), shared sales handbook, directors-only HR, plus a second tenant as bait. Twelve pytest tests, all green on Django 5.2.19 without an external vector DB:

• Alice never receives CANARY-7Q2X; Bob never receives Alice's private plan.
• Shared handbook appears for both sales users; HR reorg appears for directors only.
• Cross-tenant chunks never return; querying another tenant without membership is empty.
• Post-filter intermediate holds the canary; pre-filter does not.
• Revoke removes the doc on the next retrieve; ORM re-check drops stale hits.
• Agent grant via retrieve(grant) / retrieve_for_token returns the same chunk ids as the human.
• Missing docs.view_document raises RetrievalDenied.

def test_postfilter_exposes_canary_in_intermediate(alice, corpus):
    result = retrieve_postfilter(
        alice, tenant_id=corpus["tenant"].pk, query="payroll salary secret bonus", k=3
    )
    assert "CANARY-7Q2X" not in " ".join(h.text for h in result["hits"])
    assert any("CANARY-7Q2X" in t for t in result["intermediate_texts"])

def test_agent_grant_uses_same_visible_to_as_human(alice, agent, corpus):
    grant, token = issue_grant(agent=agent, user=alice, scopes=["docs:read"])
    human = retrieve(alice, tenant_id=corpus["tenant"].pk, query="payroll salary", k=5)
    via_token = retrieve_for_token(token, tenant_id=corpus["tenant"].pk, query="payroll salary", k=5)
    assert {h.chunk_id for h in via_token} == {h.chunk_id for h in human}

Copy the canary pattern into your own CI. Duglas reports it has caught real leaks — both times in logging, not in the answers users saw.

What this doesn't do

• No real ANN index. The demo ranks in process over float lists. Production should push tenant + principal filters into pgvector / your hosted vector DB so the index never materializes forbidden rows.
• No ReBAC graph. Flat principals cover owner / group / role. Relationship-based schemes (folders, inherited project ACLs) need a label materialization step; Warden's capability-label pushdown is one design for that scale.
• Derived artifacts. Summaries, cached answers, and conversation memory inherit the intersection of their sources' permissions, or they become a new leak. Out of scope for this post; do not store a tenant-wide "weekly digest" built from mixed-ACL docs.
• Prompt injection still works inside the readable set. Scopes and ACL shrink what is reachable. They do not stop a junk query over every doc the user can already open.
• RLS is complementary, not automatic in Django+SQLite. Supabase's pattern is the right instinct on Postgres. Keep the ORM visible_chunks / recheck_against_orm path even when RLS is on — defence in depth, same as Parts 2–3 keeping call_tool() as the single door.

Checklist for your PR template

### Permission-aware RAG checklist
- [ ] Every chunk has tenant_id, source/document id, allowed_principals, acl_version
- [ ] retrieve() requires a User or Part 2 grant — no tenant-only search helper on the hot path
- [ ] Principal filter is part of the candidate query (pre-filter), not a drop after top-k
- [ ] Agent grants resolve to on_behalf_of; same visible set as the human
- [ ] Share/revoke patches chunk metadata without re-embedding; revoke tested
- [ ] ORM (or authz graph) re-check runs before chunks enter the prompt
- [ ] Canary document test in CI; string never appears in answers, logs, or traces
- [ ] Post-filter / over-fetch path is not used as the primary control
- [ ] GenAI spans do not export pre-filter chunk text by default
- [ ] Derived caches/summaries either per-user or intersection-ACL

Open question for readers

Pre-filter with flat principals is simple and testable. At large ACL graphs the alternative is label pushdown plus an authoritative re-check (Warden's Gates 1–2), or expanding ID lists from OpenFGA/SpiceDB into the ANN filter — which preserves recall but can blow up for principals authorized on millions of docs.

Which do you ship in a mid-size Django SaaS, and where did it break? Flat allowed_principals on chunks until it hurts? Always an external authz service? Or RLS-only and hope every query path sets the session variable? Reply with what you run and what failed in production. I read every comment and will fold the good answers into the next part.

Shipping AI Agents in Django

• Part 1: Human-in-the-loop approval gates
• Part 2: Least-privilege agent identity
• Part 3: MCP server without skipping row-level checks
• Part 4: permission-aware retrieval (this post)
• Planned next: cost per successful workflow — measuring LLM spend that actually matters (routing, prompt caching, budgets)

If you're wiring agents into a Django or Python product and want a second pair of eyes on the retrieval permission model, that's the kind of work we do at Lycore.