Structured Spans: The Trace Every Agent Run Should Emit

Module 19: Fleet Observability · Expansion Guide

Back to Module 19

The Problem

Sunday, 2:14 AM. PagerDuty pings: a migration agent that's been running since Friday afternoon just “completed.” Production database integrity check is failing. Sixteen hours of work, your weekend on the line, and the only thing you have to debug with is the last chat-completion's response — one chunk of free-form text saying “Migration complete. All rows transferred.”

You start trying to reconstruct what the agent actually did. The Anthropic console shows token usage by hour, not by tool call. The application log has print statements like [migrator] processing batch 47 — but no batch IDs, no timing, no idea which of the 12 sub-agents wrote that line. The audit table has schema_change rows but no link back to which agent run caused them.

You spend the next four hours doing forensic work that should have been a single query against your traces. You don't have traces.

This isn't a logging problem. This is the difference between hope and observability.

The Core Insight

If you can't replay an agent run by reading its trace, you don't have observability — you have hope.

Print-statement logging works for human-driven services because the human is loading context into their head as they read. For agents, the “human” is the next debug session at 2 AM, with no context, with the full run already finished, with the production damage already done. You have one shot to reconstruct what happened and you'd better have the data.

The pattern that gives you that data is OpenTelemetry-style structured spans, emitted at every tool-call boundary, correlated by a single task_id that propagates across retries, sub-agents, queue boundaries, and human-handoffs. Every span carries enough metadata to answer the three questions every postmortem starts with: what was the agent told to do, what did it actually do, and what did each step return.

// across actors

TEAM: One trace per agent run, queryable across the fleet so you can find the bad one.

SOLO: One trace per agent run; same query surface, just fewer rows.

BOT: One trace per inference, sampled at 1–10% for cost; rare bad inferences still surface in the tail.