Don't Put the Prompt on the Span: Tracing Django AI Agents with OpenTelemetry (Part 6)

Picture the ticket. A user thumbs-downs an answer from your support agent: it quoted a refund window that isn't in your docs. Your Django logs say POST /ask 200 4.1s. That's all.

Which knowledge-base chunks did the model see? Which prompt version ran? Did the order lookup fail and the model improvise? How many turns did it take, and what did they cost? If answering that takes a day of guesswork, you have uptime monitoring, not observability.

This is Part 6 of Shipping AI Agents in Django. Part 5 put a price on every workflow. This part makes every run explainable with OpenTelemetry, without turning your trace backend into a second copy of your users' conversations.

Short version

• One span per step: invoke_agent → retrieval → chat {model} → execute_tool {name}. These names come from the OpenTelemetry GenAI semantic conventions, which are still marked Development in November 2026.
• Spans carry shape, not content. Put ids, token counts, cost, finish reasons and error types on spans. Prompts, outputs and tool arguments go to an access-controlled archive, and the span gets a *_ref URI.
• Enforce an attribute allowlist in code and run a PII canary test in CI. Turn off automatic exception events: exception messages echo user input.
• Sample on the tail, not the head. Keep every error, every expensive run (Part 5's cost attribute) and every slow run, plus a 10% baseline.
• Keep a DB flight recorder that survives sampling, because thumbs-downs arrive after the sampler has already made its decision.
• Demo: Django 5.2.19, opentelemetry-sdk 1.45.2, 6 pytest tests passing, and a Collector config validated and smoke-tested on otelcol-contrib 0.165.0.

Where the conventions stand (November 2026)

The GenAI semantic conventions moved out of the main semconv repo into their own: open-telemetry/semantic-conventions-genai. Every GenAI document there is still labelled Development, and the stabilization tracking issue (#511), last updated 6 November, describes itself as a "working draft". Names can still change. Pin what you emit and keep it in one module.

Here's what I'm building on, all from the current docs:

Sources: agent spans, retrieval, client inference, execute tool.

Three details matter more than they look:

1. invoke_agent vs invoke_workflow. The agent-spans doc says to use invoke_agent when an LLM drives the control flow through tool calls, and invoke_workflow when your application defines it (LangGraph's StateGraph, CrewAI's Crew). A Django view with a fixed for step in range(4) loop sits in between. I use invoke_agent because the model chooses the tools. If your code picks every step, use invoke_workflow.
2. Sampling attributes go in at span creation. The spec says gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model and friends SHOULD be provided at creation time, because samplers only see what exists when the span starts.
3. gen_ai.agent.name must be low-cardinality. It's a name like support, not a UUID and not a framework type name like StateGraph. Per-run ids belong somewhere else.

For Python libraries there's official auto-instrumentation in progress. On the contrib repo's main branch, opentelemetry-instrumentation-openai-v2 is marked deprecated (from 2.5b0) in favour of opentelemetry-instrumentation-genai-openai (1.3b0 on PyPI, 23 Oct 2026), which is built on opentelemetry-util-genai. Use it if it covers your SDK. You still need to hand-instrument your own agent loop, retrieval and tools. The execute-tool doc says so directly: application developers are "encouraged to follow this semantic convention for tools invoked by their own code".

What goes on a span, and what never does

The spec is blunt about content. Model instructions, inputs and outputs "are considered sensitive", instrumentations "SHOULD NOT capture them by default", and for production it recommends storing content externally and putting references on spans (gen-ai-spans, "Capturing instructions, inputs, and outputs"). The content attributes (gen_ai.input.messages, gen_ai.output.messages, gen_ai.tool.call.arguments, gen_ai.tool.call.result, gen_ai.retrieval.query.text, gen_ai.prompt.variable.*) are all Opt-In, and each is flagged as likely to contain PII.

My split:

One trap worth knowing about: env var names haven't settled. The spec says that if content capture is configured through an environment variable, it MUST be called OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT. But opentelemetry-util-genai 1.3b0 reads OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT (I checked the wheel's source). If you rely on "content is off by default", assert it in a test. Don't trust an env var you set once.

The tracer module

Everything lives in one module, agentrace/genai.py, that owns the attribute names. Two helpers do the safety work:

ALLOWED = frozenset({
    "gen_ai.operation.name", "gen_ai.provider.name", "gen_ai.agent.name",
    "gen_ai.conversation.id", "gen_ai.request.model", "gen_ai.request.max_tokens",
    "gen_ai.response.model", "gen_ai.response.id", "gen_ai.response.finish_reasons",
    "gen_ai.usage.input_tokens", "gen_ai.usage.output_tokens",
    "gen_ai.usage.cache_read.input_tokens", "gen_ai.usage.cache_write.input_tokens",
    "gen_ai.input.messages_ref", "gen_ai.output.messages_ref",
    "gen_ai.tool.name", "gen_ai.tool.type", "gen_ai.tool.call.id",
    "gen_ai.retrieval.top_k", "server.address", "error.type",
    # app-specific, prefixed per OTel naming guidance
    "helpdesk.run.id", "helpdesk.cost.micro_usd", "helpdesk.retrieval.doc_ids",
    "helpdesk.tool.arg_names", "helpdesk.prompt.version", "helpdesk.feedback",
})


def _set(span, attrs: dict):
    unknown = set(attrs) - ALLOWED
    if unknown:
        raise ValueError(f"span attribute(s) not on allowlist: {sorted(unknown)}")
    span.set_attributes({k: v for k, v in attrs.items() if v is not None})


def _fail(span, exc: BaseException) -> str:
    # error.type = class name. NOT str(exc): exception messages love to echo
    # user input ("no order for jane@example.com").
    etype = type(exc).__qualname__
    span.set_attribute("error.type", etype)
    span.set_status(Status(StatusCode.ERROR))
    return etype

The allowlist fails loudly in dev and CI. A teammate who adds span.set_attribute("user.email", ...) gets a red test before the attribute ever reaches a backend. Custom attributes use an application prefix (helpdesk.*), which is what the OTel naming guidance recommends for internal-only names.

_fail breaks a rule on purpose. The general recording-errors convention says the span status description SHOULD be set to the exception message, and the GenAI exceptions doc notes that exception.message "may contain sensitive information". For agent tools, whose exceptions are built from user-supplied arguments, I choose the class name and drop the message. The full exception still goes to your error tracker, which has its own access controls. The Python SDK's start_as_current_span records an exception event and sets the status by default, so every span below passes record_exception=False, set_status_on_exception=False and calls _fail itself.

The model-call span

def chat(self, provider: str, model: str, messages: list[dict], call, max_tokens=1024):
    with _tracer().start_as_current_span(
        f"chat {model}", kind=SpanKind.CLIENT,
        # sampling-relevant attributes go in at creation time (spec SHOULD)
        attributes={"gen_ai.operation.name": "chat", "gen_ai.provider.name": provider,
                    "gen_ai.request.model": model, "gen_ai.request.max_tokens": max_tokens},
        record_exception=False, set_status_on_exception=False,
    ) as span:
        in_ref = self.archive(scrub_messages(messages))
        _set(span, {"gen_ai.input.messages_ref": in_ref})
        try:
            resp = call()
        except Exception as exc:
            self._step(span, kind="chat", name=model, input_ref=in_ref,
                       error_type=_fail(span, exc))
            raise
        u = resp["usage"]
        cost = cost_micro_usd(model, u)
        out_ref = self.archive(scrub_messages(resp["messages"]))
        _set(span, {
            "gen_ai.response.id": resp["id"], "gen_ai.response.model": resp["model"],
            "gen_ai.response.finish_reasons": resp["finish_reasons"],
            # input_tokens INCLUDES cache reads/writes (semconv-genai)
            "gen_ai.usage.input_tokens": u["uncached_input"] + u["cache_read"] + u["cache_write"],
            "gen_ai.usage.output_tokens": u["output"],
            "gen_ai.usage.cache_read.input_tokens": u["cache_read"] or None,
            "gen_ai.usage.cache_write.input_tokens": u["cache_write"] or None,
            "gen_ai.output.messages_ref": out_ref,
            "helpdesk.cost.micro_usd": cost,
        })
        self.run.cost_micro_usd += cost
        self._step(span, kind="chat", name=model, cost_micro_usd=cost,
                   input_ref=in_ref, output_ref=out_ref)
        return resp

Notes:

• Token semantics match Part 5. gen_ai.usage.input_tokens SHOULD include cache reads and writes, and the cache attributes are subsets that are "Recommended when available and non-zero". That's why zeros become None and get dropped.
• Cost is an integer, in micro-USD. With prices quoted per million tokens, micro-USD per token is the same number as USD per MTok, so the maths stays exact. The integer matters later: the Collector's numeric_attribute sampling policy compares int64 values (config.go).
• The archive write happens before the call, so a timeout still leaves you the exact prompt that hung.
• archive() stores a redacted body in a PromptArchive table with an expires_at (14 days in the demo; set your retention with whoever owns privacy) and returns https://ops.example.com/llm-archive/<uuid>. The spec's _ref attributes take an absolute URI, and the spec warns against signed-URL credentials in it. Access control belongs on the archive, not in the link.

Redaction before archiving is a regex floor for emails, card-like digit runs and phone numbers. It isn't a guarantee: names, addresses and free-text health details get through. That's why the archive is a separate table with separate permissions, and why it expires.

Tool spans: argument names yes, values no

def tool(self, name: str, call_id: str | None, args: dict, fn):
    with _tracer().start_as_current_span(
        f"execute_tool {name}", kind=SpanKind.INTERNAL,
        attributes={"gen_ai.operation.name": "execute_tool", "gen_ai.tool.name": name,
                    "gen_ai.tool.type": "function"},
        record_exception=False, set_status_on_exception=False,
    ) as span:
        # arg NAMES are schema; arg VALUES are user data
        _set(span, {"gen_ai.tool.call.id": call_id, "helpdesk.tool.arg_names": sorted(args)})
        in_ref = self.archive(scrub_messages([{"role": "tool_args", "content": repr(args)}]))
        try:
            result = fn(**args)
        except Exception as exc:
            self._step(span, kind="execute_tool", name=name, input_ref=in_ref,
                       error_type=_fail(span, exc))
            raise
        ...

Argument names tell you a lot ("the model called lookup_order without order_id") and carry no user data. gen_ai.tool.call.id is conditionally required only when the model supplied one, so None is dropped by _set rather than faked.

The retrieval span records gen_ai.retrieval.top_k and the ids of the chunks that came back, never their text. If you followed Part 4, those ids already passed a permission pre-filter, and putting chunk text on a span would leak exactly what Part 4 kept out of the prompt.

Wiring it into Django

# otel_setup.py: call once per process (manage.py, wsgi.py, or gunicorn post_fork)
def configure(exporter: SpanExporter, *, ratio: float | None = None) -> TracerProvider:
    if ratio is None:
        # 1.0 when a Collector does tail sampling; lower only if it doesn't.
        ratio = float(os.environ.get("HELPDESK_TRACE_RATIO", "1.0"))
    provider = TracerProvider(
        resource=Resource.create({"service.name": "helpdesk"}),
        sampler=ParentBased(TraceIdRatioBased(ratio)),
    )
    provider.add_span_processor(BatchSpanProcessor(exporter))
    trace.set_tracer_provider(provider)
    return provider

Then add DjangoInstrumentor().instrument() (docs) and your invoke_agent span nests under the request's SERVER span. The demo test asserts that parent link. Under gunicorn or uWSGI, set up the provider in a post-fork hook: the OTel Python docs warn that BatchSpanProcessor "is not fork-safe" (fork-process-model). Exclude health checks with OTEL_PYTHON_DJANGO_EXCLUDED_URLS.

Sampling: decide at the tail, record at the source

Head sampling at 10% means 90% of your bad answers have no trace. And "bad" isn't known when the trace starts. So I export everything from the app (ratio=1.0) and let the Collector decide after the trace finishes:

processors:
  # Second line of defence. The app-side allowlist is the first.
  redaction:
    allow_all_keys: true
    blocked_key_patterns:
      - "gen_ai\\.(input|output)\\.messages$"
      - "gen_ai\\.system_instructions$"
      - "gen_ai\\.tool\\.call\\.(arguments|result)$"
      - "gen_ai\\.prompt\\.variable\\..*"
    blocked_values:
      - "[\\w.+-]+@[\\w-]+\\.[\\w.-]+"   # email addresses
    summary: info
  tail_sampling:
    decision_wait: 30s
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: expensive-runs      # >= $0.02, the Part 5 cost attribute
        type: numeric_attribute
        numeric_attribute: {key: helpdesk.cost.micro_usd, min_value: 20000}
      - name: slow-runs
        type: latency
        latency: {threshold_ms: 20000}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 10}

I ran otelcol-contrib validate on the full file (receivers and pipeline included in the demo). I also ran a smoke test against a live Collector with the baseline set to 0%. A cheap successful span was dropped. A span with helpdesk.cost.micro_usd=45000 was kept, its gen_ai.input.messages came out as **** and an email inside another attribute became mail ****. A span with ERROR status was kept.

Two caveats from the tail sampling README. All spans of a trace must reach the same Collector instance, so at scale you need a load-balancing exporter tier in front. And spans are held in memory for decision_wait, so an agent run longer than that gets decided on partial data. The redaction processor has a disclaimer of its own: it's "one line of defence rather than the only compliance measure". That's why the allowlist sits in your code.

The flight recorder: why the DB keeps its own copy

Tail sampling still can't see the future. The thumbs-down arrives minutes after decision_wait expired, and by then a cheap, fast, error-free bad answer has been dropped from the 90%.

So each step also writes a RunStep row: kind, name, span id, cost, archive refs, doc ids and error type. AgentRun stores the trace_id, the agent span's span_id and whether the trace was sampled. The archive gets written whatever the sampling decision. That matches the spec's guidance that upload hooks SHOULD be invoked "regardless of the span sampling decision".

Feedback comes in on a different request, so it lands in a different trace. A span link connects it back:

def record_feedback(run: AgentRun, verdict: str) -> None:
    run.feedback = verdict
    run.save(update_fields=["feedback"])
    links = []
    if run.trace_id:
        ctx = SpanContext(trace_id=int(run.trace_id, 16), span_id=int(run.span_id, 16),
                          is_remote=True, trace_flags=TraceFlags(int(run.sampled)))
        links.append(Link(ctx))
    with trace.get_tracer("agentrace").start_as_current_span(
        "helpdesk.feedback", links=links,
        attributes={"helpdesk.run.id": str(run.id), "helpdesk.feedback": verdict},
    ):
        pass

Debugging the bad answer

Back to the ticket. GET /runs/<id>/explain (staff-only in real life) returns this (abridged):

{"run": "…", "trace_id": "4bf9…", "trace_exported": true, "status": "ok",
 "feedback": "bad", "cost_usd": 0.0058,
 "steps": [
  {"seq": 1, "kind": "retrieval", "name": "retrieval", "docs": ["kb-12", "kb-40"]},
  {"seq": 2, "kind": "chat", "name": "claude-sonnet-5-5", "cost_usd": 0.0029,
   "input": "https://ops.example.com/llm-archive/…", "output": "…"},
  {"seq": 3, "kind": "execute_tool", "name": "lookup_order", "error": ""},
  {"seq": 4, "kind": "chat", "name": "claude-sonnet-5-5", "cost_usd": 0.0029, "…": "…"}
 ]}

The triage order I'd follow:

1. Retrieval: were the right doc ids returned? If kb-12 (refunds) wasn't there, it's a retrieval bug and the model never had a chance.
2. Tool: did lookup_order fail? An error.type followed by a confident answer means the model improvised around a failure. Make tool errors terminal or explicit in the prompt.
3. Prompt version: helpdesk.prompt.version on the agent span. Did the regression start with support-v7?
4. Finish reason: a max_tokens-style stop on the final turn means a truncated answer, not a wrong one.
5. Only then open the archived input and output (an audited access) to read what the model actually saw.

If trace_exported is true, the trace_id takes you to the waterfall in your backend. If it's false, the steps above still work from the database alone.

What the tests prove

Six tests, no network, all green on Python 3.13 / Django 5.2.19 / opentelemetry-sdk 1.45.2 with an InMemorySpanExporter:

• The span tree is retrieval, chat, execute_tool, chat under invoke_agent support, with semconv kinds and names.
• Usage: input tokens include cache reads (6,400 = 400 + 6,000), and cost is 2,900 micro-USD per call and 5,800 per run.
• PII canary: run with canary-7731@example.com in the question and the tool args, with a failing tool. The canary appears in no span name, attribute, status, event or link. The archive contains <email> instead. I mutated _fail to use str(exc) and this test went red.
• The allowlist rejects user.email.
• With a sampler at ratio 0, nothing is exported, yet explain() still returns all four steps and the archive refs.
• Django round trip: the server span is the parent of the agent span, the feedback span links back to it, and the explain endpoint reports cost_usd == 0.0058.

The canary test is the one to copy:

def test_no_pii_on_any_span_even_on_tool_error(spans):
    run = run_support_agent(f"Refund? my email is {CANARY}", order_id="Z-9", email=CANARY)
    assert run.status == "error"
    for span in spans.get_finished_spans():
        assert CANARY not in everything_in(span), span.name
    tool = by_name(spans)["execute_tool lookup_order"]
    assert tool.status.status_code == StatusCode.ERROR
    assert tool.attributes["error.type"] == "OrderNotFound"
    assert not tool.events  # no exception event carrying str(exc)

What this doesn't do

• Streaming. No gen_ai.response.time_to_first_chunk. The spec defines it, and the demo doesn't stream.
• Metrics. The spec has token and duration histograms. Spans plus the DB cover debugging, but dashboards want metrics.
• Celery. If the agent runs in a task, propagate context into it, or the trace breaks at the queue.
• Spec churn. Names are Development status. One module owns them, so a rename is a one-file change plus the allowlist test.

Checklist for your PR template

### Agent tracing checklist
- [ ] invoke_agent / retrieval / chat / execute_tool spans; sampling attrs at creation
- [ ] Attribute allowlist enforced in code; PII canary test in CI
- [ ] record_exception=False on agent spans; error.type = class name
- [ ] Content only in the archive (redacted, TTL, separate permissions); *_ref on spans
- [ ] usage.input_tokens includes cache tokens; integer cost attribute
- [ ] App exports 100%; Collector tail-samples errors + expensive + slow + baseline
- [ ] trace_id and steps stored in DB regardless of sampling; feedback linked

Open question for readers

Here's the trade-off I haven't settled. Where does the prompt live when you debug? This design keeps content out of traces completely and puts it in a redacted, expiring, audited archive. It's safer, but reading a bad run means a second click and an access request, and the regex redaction can scrub away the exact detail that caused the bad answer. The alternative is the spec's Opt-In content attributes, on a separate trace pipeline with a short retention and restricted access. Debugging gets faster, and the blast radius gets bigger.

Which do you run in production, and who's allowed to read raw prompts? And on sampling: would you add a policy that keeps every run for a user's first week, so onboarding failures are always traceable, or is that a privacy cost you wouldn't take?

The series so far

1. Human-in-the-loop AI agents in Django: approval gates
2. Least-privilege agent identity in Django
3. Your MCP server is a second API
4. Permission-aware RAG without post-filters
5. Cost per successful workflow
6. Tracing agent runs with OpenTelemetry (this post)

Next, Part 7: structured outputs that don't break. Pydantic schemas for LLM responses, validation failures as first-class span events, and retry rules that don't double your bill.

The demo is a plain Django 5.2 app: agentrace/ (genai spans, redaction, flight recorder, explain), otelcol.yaml, and one test file. More of what we build is at lycore.com.