# Don't Put the Prompt on the Span: Tracing Django AI Agents with OpenTelemetry (Part 6)

Picture the ticket. A user thumbs-downs an answer from your support agent: it quoted a refund window that isn't in your docs. Your Django logs say `POST /ask 200 4.1s`. That's all.

Which knowledge-base chunks did the model see? Which prompt version ran? Did the order lookup fail and the model improvise? How many turns did it take, and what did they cost? If answering that takes a day of guesswork, you have uptime monitoring, not observability.

This is Part 6 of **Shipping AI Agents in Django**. [Part 5](https://aidevpro.hashnode.dev/llm-cost-per-successful-workflow-django) put a price on every workflow. This part makes every run explainable with OpenTelemetry, without turning your trace backend into a second copy of your users' conversations.

> **Short version**
>
> - **One span per step:** `invoke_agent` → `retrieval` → `chat {model}` → `execute_tool {name}`. These names come from the OpenTelemetry GenAI semantic conventions, which are still marked *Development* in November 2026.
> - **Spans carry shape, not content.** Put ids, token counts, cost, finish reasons and error *types* on spans. Prompts, outputs and tool arguments go to an access-controlled archive, and the span gets a `*_ref` URI.
> - **Enforce an attribute allowlist in code** and run a PII canary test in CI. Turn off automatic exception events: exception messages echo user input.
> - **Sample on the tail, not the head.** Keep every error, every expensive run (Part 5's cost attribute) and every slow run, plus a 10% baseline.
> - **Keep a DB flight recorder** that survives sampling, because thumbs-downs arrive after the sampler has already made its decision.
> - Demo: Django 5.2.19, opentelemetry-sdk 1.45.2, **6 pytest tests passing**, and a Collector config validated and smoke-tested on otelcol-contrib 0.165.0.

## Where the conventions stand (November 2026)

The GenAI semantic conventions moved out of the main semconv repo into their own: [open-telemetry/semantic-conventions-genai](https://github.com/open-telemetry/semantic-conventions-genai). Every GenAI document there is still labelled **Development**, and the [stabilization tracking issue (#511)](https://github.com/open-telemetry/semantic-conventions-genai/issues/511), last updated 6 November, describes itself as a "working draft". Names can still change. Pin what you emit and keep it in one module.

Here's what I'm building on, all from the current docs:

| Step | Span name | Kind | Key attributes |
|---|---|---|---|
| Agent run (in-process) | `invoke_agent {gen_ai.agent.name}` | INTERNAL | `gen_ai.operation.name=invoke_agent`, `gen_ai.agent.name`, `gen_ai.conversation.id` |
| Retrieval in your process | `retrieval` | INTERNAL | `gen_ai.retrieval.top_k` |
| Model call | `{operation} {model}`, e.g. `chat claude-sonnet-5-5` | CLIENT | `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.usage.*`, `gen_ai.response.finish_reasons` |
| Tool call | `execute_tool {gen_ai.tool.name}` | INTERNAL | `gen_ai.tool.name`, `gen_ai.tool.call.id` (only if the model supplied one) |

Sources: [agent spans](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md), [retrieval](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/retrieval.md), [client inference](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/client-inference.md), [execute tool](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/execute-tool.md).

Three details matter more than they look:

1. **`invoke_agent` vs `invoke_workflow`.** The agent-spans doc says to use `invoke_agent` when an LLM drives the control flow through tool calls, and `invoke_workflow` when your application defines it (LangGraph's `StateGraph`, CrewAI's `Crew`). A Django view with a fixed `for step in range(4)` loop sits in between. I use `invoke_agent` because the model chooses the tools. If your code picks every step, use `invoke_workflow`.
2. **Sampling attributes go in at span creation.** The spec says `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.request.model` and friends SHOULD be provided at creation time, because samplers only see what exists when the span starts.
3. **`gen_ai.agent.name` must be low-cardinality.** It's a name like `support`, not a UUID and not a framework type name like `StateGraph`. Per-run ids belong somewhere else.

For Python libraries there's official auto-instrumentation in progress. On the contrib repo's main branch, `opentelemetry-instrumentation-openai-v2` is marked deprecated (from 2.5b0) in favour of `opentelemetry-instrumentation-genai-openai` (1.3b0 on PyPI, 23 Oct 2026), which is built on `opentelemetry-util-genai`. Use it if it covers your SDK. You still need to hand-instrument your own agent loop, retrieval and tools. The execute-tool doc says so directly: application developers are "encouraged to follow this semantic convention for tools invoked by their own code".

## What goes on a span, and what never does

The spec is blunt about content. Model instructions, inputs and outputs "are considered sensitive", instrumentations "SHOULD NOT capture them by default", and for production it recommends storing content externally and putting references on spans ([gen-ai-spans, "Capturing instructions, inputs, and outputs"](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md#full-buffered-content)). The content attributes (`gen_ai.input.messages`, `gen_ai.output.messages`, `gen_ai.tool.call.arguments`, `gen_ai.tool.call.result`, `gen_ai.retrieval.query.text`, `gen_ai.prompt.variable.*`) are all **Opt-In**, and each is flagged as likely to contain PII.

My split:

| On the span | In the archive (ref on span) | Nowhere |
|---|---|---|
| model, provider, max_tokens | system prompt + messages (redacted) | raw email / phone / card numbers |
| token counts incl. cache buckets | model output (redacted) | `str(exception)` |
| cost in micro-USD | tool arguments and results (redacted) | auth headers, API keys |
| response id, finish reasons | | retrieved chunk *text* |
| retrieved **doc ids**, top_k | | |
| tool name, call id, **argument names** | | |
| `error.type` (class name) | | |
| prompt version, run id | | |

One trap worth knowing about: env var names haven't settled. The spec says that if content capture is configured through an environment variable, it MUST be called `OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT`. But `opentelemetry-util-genai` 1.3b0 reads `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` (I checked the wheel's source). If you rely on "content is off by default", assert it in a test. Don't trust an env var you set once.

## The tracer module

Everything lives in one module, `agentrace/genai.py`, that owns the attribute names. Two helpers do the safety work:

```python
ALLOWED = frozenset({
    "gen_ai.operation.name", "gen_ai.provider.name", "gen_ai.agent.name",
    "gen_ai.conversation.id", "gen_ai.request.model", "gen_ai.request.max_tokens",
    "gen_ai.response.model", "gen_ai.response.id", "gen_ai.response.finish_reasons",
    "gen_ai.usage.input_tokens", "gen_ai.usage.output_tokens",
    "gen_ai.usage.cache_read.input_tokens", "gen_ai.usage.cache_write.input_tokens",
    "gen_ai.input.messages_ref", "gen_ai.output.messages_ref",
    "gen_ai.tool.name", "gen_ai.tool.type", "gen_ai.tool.call.id",
    "gen_ai.retrieval.top_k", "server.address", "error.type",
    # app-specific, prefixed per OTel naming guidance
    "helpdesk.run.id", "helpdesk.cost.micro_usd", "helpdesk.retrieval.doc_ids",
    "helpdesk.tool.arg_names", "helpdesk.prompt.version", "helpdesk.feedback",
})


def _set(span, attrs: dict):
    unknown = set(attrs) - ALLOWED
    if unknown:
        raise ValueError(f"span attribute(s) not on allowlist: {sorted(unknown)}")
    span.set_attributes({k: v for k, v in attrs.items() if v is not None})


def _fail(span, exc: BaseException) -> str:
    # error.type = class name. NOT str(exc): exception messages love to echo
    # user input ("no order for jane@example.com").
    etype = type(exc).__qualname__
    span.set_attribute("error.type", etype)
    span.set_status(Status(StatusCode.ERROR))
    return etype
```

The allowlist fails loudly in dev and CI. A teammate who adds `span.set_attribute("user.email", ...)` gets a red test before the attribute ever reaches a backend. Custom attributes use an application prefix (`helpdesk.*`), which is what the [OTel naming guidance](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/general/naming.md#recommendations-for-application-developers) recommends for internal-only names.

`_fail` breaks a rule on purpose. The general [recording-errors](https://github.com/open-telemetry/semantic-conventions/blob/v1.45.0/docs/general/recording-errors.md) convention says the span status description SHOULD be set to the exception message, and the GenAI exceptions doc notes that `exception.message` "may contain sensitive information". For agent tools, whose exceptions are built from user-supplied arguments, I choose the class name and drop the message. The full exception still goes to your error tracker, which has its own access controls. The Python SDK's `start_as_current_span` records an exception event and sets the status by default, so every span below passes `record_exception=False, set_status_on_exception=False` and calls `_fail` itself.

## The model-call span

```python
def chat(self, provider: str, model: str, messages: list[dict], call, max_tokens=1024):
    with _tracer().start_as_current_span(
        f"chat {model}", kind=SpanKind.CLIENT,
        # sampling-relevant attributes go in at creation time (spec SHOULD)
        attributes={"gen_ai.operation.name": "chat", "gen_ai.provider.name": provider,
                    "gen_ai.request.model": model, "gen_ai.request.max_tokens": max_tokens},
        record_exception=False, set_status_on_exception=False,
    ) as span:
        in_ref = self.archive(scrub_messages(messages))
        _set(span, {"gen_ai.input.messages_ref": in_ref})
        try:
            resp = call()
        except Exception as exc:
            self._step(span, kind="chat", name=model, input_ref=in_ref,
                       error_type=_fail(span, exc))
            raise
        u = resp["usage"]
        cost = cost_micro_usd(model, u)
        out_ref = self.archive(scrub_messages(resp["messages"]))
        _set(span, {
            "gen_ai.response.id": resp["id"], "gen_ai.response.model": resp["model"],
            "gen_ai.response.finish_reasons": resp["finish_reasons"],
            # input_tokens INCLUDES cache reads/writes (semconv-genai)
            "gen_ai.usage.input_tokens": u["uncached_input"] + u["cache_read"] + u["cache_write"],
            "gen_ai.usage.output_tokens": u["output"],
            "gen_ai.usage.cache_read.input_tokens": u["cache_read"] or None,
            "gen_ai.usage.cache_write.input_tokens": u["cache_write"] or None,
            "gen_ai.output.messages_ref": out_ref,
            "helpdesk.cost.micro_usd": cost,
        })
        self.run.cost_micro_usd += cost
        self._step(span, kind="chat", name=model, cost_micro_usd=cost,
                   input_ref=in_ref, output_ref=out_ref)
        return resp
```

Notes:

- **Token semantics match Part 5.** `gen_ai.usage.input_tokens` SHOULD include cache reads and writes, and the cache attributes are subsets that are "Recommended when available and non-zero". That's why zeros become `None` and get dropped.
- **Cost is an integer, in micro-USD.** With prices quoted per million tokens, micro-USD per token is the same number as USD per MTok, so the maths stays exact. The integer matters later: the Collector's `numeric_attribute` sampling policy compares `int64` values ([config.go](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/config.go)).
- **The archive write happens before the call,** so a timeout still leaves you the exact prompt that hung.
- `archive()` stores a redacted body in a `PromptArchive` table with an `expires_at` (14 days in the demo; set your retention with whoever owns privacy) and returns `https://ops.example.com/llm-archive/<uuid>`. The spec's `_ref` attributes take an absolute URI, and the spec warns against signed-URL credentials in it. Access control belongs on the archive, not in the link.

Redaction before archiving is a regex floor for emails, card-like digit runs and phone numbers. It isn't a guarantee: names, addresses and free-text health details get through. That's why the archive is a separate table with separate permissions, and why it expires.

## Tool spans: argument names yes, values no

```python
def tool(self, name: str, call_id: str | None, args: dict, fn):
    with _tracer().start_as_current_span(
        f"execute_tool {name}", kind=SpanKind.INTERNAL,
        attributes={"gen_ai.operation.name": "execute_tool", "gen_ai.tool.name": name,
                    "gen_ai.tool.type": "function"},
        record_exception=False, set_status_on_exception=False,
    ) as span:
        # arg NAMES are schema; arg VALUES are user data
        _set(span, {"gen_ai.tool.call.id": call_id, "helpdesk.tool.arg_names": sorted(args)})
        in_ref = self.archive(scrub_messages([{"role": "tool_args", "content": repr(args)}]))
        try:
            result = fn(**args)
        except Exception as exc:
            self._step(span, kind="execute_tool", name=name, input_ref=in_ref,
                       error_type=_fail(span, exc))
            raise
        ...
```

Argument names tell you a lot ("the model called `lookup_order` without `order_id`") and carry no user data. `gen_ai.tool.call.id` is conditionally required only when the model supplied one, so `None` is dropped by `_set` rather than faked.

The retrieval span records `gen_ai.retrieval.top_k` and the **ids** of the chunks that came back, never their text. If you followed [Part 4](https://aidevpro.hashnode.dev/permission-aware-rag-django-prefilter), those ids already passed a permission pre-filter, and putting chunk text on a span would leak exactly what Part 4 kept out of the prompt.

## Wiring it into Django

```python
# otel_setup.py: call once per process (manage.py, wsgi.py, or gunicorn post_fork)
def configure(exporter: SpanExporter, *, ratio: float | None = None) -> TracerProvider:
    if ratio is None:
        # 1.0 when a Collector does tail sampling; lower only if it doesn't.
        ratio = float(os.environ.get("HELPDESK_TRACE_RATIO", "1.0"))
    provider = TracerProvider(
        resource=Resource.create({"service.name": "helpdesk"}),
        sampler=ParentBased(TraceIdRatioBased(ratio)),
    )
    provider.add_span_processor(BatchSpanProcessor(exporter))
    trace.set_tracer_provider(provider)
    return provider
```

Then add `DjangoInstrumentor().instrument()` ([docs](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation/opentelemetry-instrumentation-django)) and your `invoke_agent` span nests under the request's SERVER span. The demo test asserts that parent link. Under gunicorn or uWSGI, set up the provider in a post-fork hook: the OTel Python docs warn that `BatchSpanProcessor` "is not fork-safe" ([fork-process-model](https://github.com/open-telemetry/opentelemetry-python/tree/main/docs/examples/fork-process-model)). Exclude health checks with `OTEL_PYTHON_DJANGO_EXCLUDED_URLS`.

## Sampling: decide at the tail, record at the source

Head sampling at 10% means 90% of your bad answers have no trace. And "bad" isn't known when the trace starts. So I export everything from the app (`ratio=1.0`) and let the Collector decide after the trace finishes:

```yaml
processors:
  # Second line of defence. The app-side allowlist is the first.
  redaction:
    allow_all_keys: true
    blocked_key_patterns:
      - "gen_ai\\.(input|output)\\.messages$"
      - "gen_ai\\.system_instructions$"
      - "gen_ai\\.tool\\.call\\.(arguments|result)$"
      - "gen_ai\\.prompt\\.variable\\..*"
    blocked_values:
      - "[\\w.+-]+@[\\w-]+\\.[\\w.-]+"   # email addresses
    summary: info
  tail_sampling:
    decision_wait: 30s
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: expensive-runs      # >= $0.02, the Part 5 cost attribute
        type: numeric_attribute
        numeric_attribute: {key: helpdesk.cost.micro_usd, min_value: 20000}
      - name: slow-runs
        type: latency
        latency: {threshold_ms: 20000}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 10}
```

I ran `otelcol-contrib validate` on the full file (receivers and pipeline included in the demo). I also ran a smoke test against a live Collector with the baseline set to 0%. A cheap successful span was dropped. A span with `helpdesk.cost.micro_usd=45000` was kept, its `gen_ai.input.messages` came out as `****` and an email inside another attribute became `mail ****`. A span with ERROR status was kept.

Two caveats from the [tail sampling README](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md). All spans of a trace must reach the same Collector instance, so at scale you need a load-balancing exporter tier in front. And spans are held in memory for `decision_wait`, so an agent run longer than that gets decided on partial data. The [redaction processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/redactionprocessor/README.md) has a disclaimer of its own: it's "one line of defence rather than the only compliance measure". That's why the allowlist sits in your code.

## The flight recorder: why the DB keeps its own copy

Tail sampling still can't see the future. The thumbs-down arrives minutes after `decision_wait` expired, and by then a cheap, fast, error-free bad answer has been dropped from the 90%.

So each step also writes a `RunStep` row: kind, name, span id, cost, archive refs, doc ids and error type. `AgentRun` stores the `trace_id`, the agent span's `span_id` and whether the trace was sampled. The archive gets written whatever the sampling decision. That matches the spec's guidance that upload hooks SHOULD be invoked "regardless of the span sampling decision".

Feedback comes in on a different request, so it lands in a different trace. A [span link](https://opentelemetry.io/docs/concepts/signals/traces/#span-links) connects it back:

```python
def record_feedback(run: AgentRun, verdict: str) -> None:
    run.feedback = verdict
    run.save(update_fields=["feedback"])
    links = []
    if run.trace_id:
        ctx = SpanContext(trace_id=int(run.trace_id, 16), span_id=int(run.span_id, 16),
                          is_remote=True, trace_flags=TraceFlags(int(run.sampled)))
        links.append(Link(ctx))
    with trace.get_tracer("agentrace").start_as_current_span(
        "helpdesk.feedback", links=links,
        attributes={"helpdesk.run.id": str(run.id), "helpdesk.feedback": verdict},
    ):
        pass
```

## Debugging the bad answer

Back to the ticket. `GET /runs/<id>/explain` (staff-only in real life) returns this (abridged):

```json
{"run": "…", "trace_id": "4bf9…", "trace_exported": true, "status": "ok",
 "feedback": "bad", "cost_usd": 0.0058,
 "steps": [
  {"seq": 1, "kind": "retrieval", "name": "retrieval", "docs": ["kb-12", "kb-40"]},
  {"seq": 2, "kind": "chat", "name": "claude-sonnet-5-5", "cost_usd": 0.0029,
   "input": "https://ops.example.com/llm-archive/…", "output": "…"},
  {"seq": 3, "kind": "execute_tool", "name": "lookup_order", "error": ""},
  {"seq": 4, "kind": "chat", "name": "claude-sonnet-5-5", "cost_usd": 0.0029, "…": "…"}
 ]}
```

The triage order I'd follow:

1. **Retrieval:** were the right doc ids returned? If `kb-12` (refunds) wasn't there, it's a retrieval bug and the model never had a chance.
2. **Tool:** did `lookup_order` fail? An `error.type` followed by a confident answer means the model improvised around a failure. Make tool errors terminal or explicit in the prompt.
3. **Prompt version:** `helpdesk.prompt.version` on the agent span. Did the regression start with `support-v7`?
4. **Finish reason:** a `max_tokens`-style stop on the final turn means a truncated answer, not a wrong one.
5. Only then open the archived input and output (an audited access) to read what the model actually saw.

If `trace_exported` is true, the `trace_id` takes you to the waterfall in your backend. If it's false, the steps above still work from the database alone.

## What the tests prove

Six tests, no network, all green on Python 3.13 / Django 5.2.19 / opentelemetry-sdk 1.45.2 with an `InMemorySpanExporter`:

- The span tree is `retrieval`, `chat`, `execute_tool`, `chat` under `invoke_agent support`, with semconv kinds and names.
- Usage: input tokens include cache reads (6,400 = 400 + 6,000), and cost is 2,900 micro-USD per call and 5,800 per run.
- **PII canary:** run with `canary-7731@example.com` in the question and the tool args, with a failing tool. The canary appears in no span name, attribute, status, event or link. The archive contains `<email>` instead. I mutated `_fail` to use `str(exc)` and this test went red.
- The allowlist rejects `user.email`.
- With a sampler at ratio 0, nothing is exported, yet `explain()` still returns all four steps and the archive refs.
- Django round trip: the server span is the parent of the agent span, the feedback span links back to it, and the explain endpoint reports `cost_usd == 0.0058`.

The canary test is the one to copy:

```python
def test_no_pii_on_any_span_even_on_tool_error(spans):
    run = run_support_agent(f"Refund? my email is {CANARY}", order_id="Z-9", email=CANARY)
    assert run.status == "error"
    for span in spans.get_finished_spans():
        assert CANARY not in everything_in(span), span.name
    tool = by_name(spans)["execute_tool lookup_order"]
    assert tool.status.status_code == StatusCode.ERROR
    assert tool.attributes["error.type"] == "OrderNotFound"
    assert not tool.events  # no exception event carrying str(exc)
```

## What this doesn't do

- **Streaming.** No `gen_ai.response.time_to_first_chunk`. The spec defines it, and the demo doesn't stream.
- **Metrics.** The spec has token and duration histograms. Spans plus the DB cover debugging, but dashboards want metrics.
- **Celery.** If the agent runs in a task, propagate context into it, or the trace breaks at the queue.
- **Spec churn.** Names are Development status. One module owns them, so a rename is a one-file change plus the allowlist test.

## Checklist for your PR template

```markdown
### Agent tracing checklist
- [ ] invoke_agent / retrieval / chat / execute_tool spans; sampling attrs at creation
- [ ] Attribute allowlist enforced in code; PII canary test in CI
- [ ] record_exception=False on agent spans; error.type = class name
- [ ] Content only in the archive (redacted, TTL, separate permissions); *_ref on spans
- [ ] usage.input_tokens includes cache tokens; integer cost attribute
- [ ] App exports 100%; Collector tail-samples errors + expensive + slow + baseline
- [ ] trace_id and steps stored in DB regardless of sampling; feedback linked
```

## Open question for readers

Here's the trade-off I haven't settled. **Where does the prompt live when you debug?** This design keeps content out of traces completely and puts it in a redacted, expiring, audited archive. It's safer, but reading a bad run means a second click and an access request, and the regex redaction can scrub away the exact detail that caused the bad answer. The alternative is the spec's Opt-In content attributes, on a separate trace pipeline with a short retention and restricted access. Debugging gets faster, and the blast radius gets bigger.

**Which do you run in production, and who's allowed to read raw prompts?** And on sampling: would you add a policy that keeps *every* run for a user's first week, so onboarding failures are always traceable, or is that a privacy cost you wouldn't take?

## The series so far

1. [Human-in-the-loop AI agents in Django: approval gates](https://aidevpro.hashnode.dev/human-in-the-loop-ai-agents-django-approval-gates)
2. [Least-privilege agent identity in Django](https://aidevpro.hashnode.dev/ai-agent-least-privilege-identity-django)
3. [Your MCP server is a second API](https://aidevpro.hashnode.dev/mcp-server-django-row-level-permissions-agents)
4. [Permission-aware RAG without post-filters](https://aidevpro.hashnode.dev/permission-aware-rag-django-prefilter)
5. [Cost per successful workflow](https://aidevpro.hashnode.dev/llm-cost-per-successful-workflow-django)
6. Tracing agent runs with OpenTelemetry (this post)

**Next, Part 7:** structured outputs that don't break. Pydantic schemas for LLM responses, validation failures as first-class span events, and retry rules that don't double your bill.

The demo is a plain Django 5.2 app: `agentrace/` (genai spans, redaction, flight recorder, explain), `otelcol.yaml`, and one test file. More of what we build is at [lycore.com](https://lycore.com).