Disclosure, I work on an agent in this space, operatex.dev.
The tracing versus evaluation split is exactly the difference that people mess up. A green dashboard with no scoring layer just means nobody checked yet, not that anything's actually fine.
Green dashboard means nobody checked yet is exactly it. Tracing shows what happened, evaluation shows whether it was right, and most teams ship the first and skip the second.
The expensive green light line is the right warning. Tracing is necessary, but it is not judgement. A trace can show the route an answer took while still leaving open whether the route was acceptable, grounded and owned. The production mistake is buying telemetry and quietly treating it as evaluation. The dashboard can be green while the semantic failure is still moving through the workflow.
Congrats Paolo. That’s a really important topic. I’d love to have a feedback from you on this tool I made purely to understand my iterations with AI. Still not Otel compliant but able to work with all the local agents like cc or cowork.
I've had great luck tracing using the built-in Claude transcripts of my agents. Once I knew they were there, it was easy enough to work with another agent to parse the json and do forensic analysis. In one case, I found out that Claude's web search is a web search tool built by Anthropic which easily explained why my research agents were missing an association that was easily looked up on Google. In another case after watching a cascade of behavior among a team of agents, I was able to isolate the cause to a mechanism where each agent was first investigating portions of the repo it lived in before deciding what to do via an ls command. Tracing has been very useful to understand what's going on in my agent-enabled applicaitons.
Parsing the transcripts with a second agent for forensic analysis is a sharp move. That ls-before-deciding cascade is the kind of emergent behavior only tracing catches. Underused surface.
Disclosure, I work on an agent in this space, operatex.dev.
The tracing versus evaluation split is exactly the difference that people mess up. A green dashboard with no scoring layer just means nobody checked yet, not that anything's actually fine.
Green dashboard means nobody checked yet is exactly it. Tracing shows what happened, evaluation shows whether it was right, and most teams ship the first and skip the second.
The expensive green light line is the right warning. Tracing is necessary, but it is not judgement. A trace can show the route an answer took while still leaving open whether the route was acceptable, grounded and owned. The production mistake is buying telemetry and quietly treating it as evaluation. The dashboard can be green while the semantic failure is still moving through the workflow.
Right, the green dashboard is the trap. A trace shows what happened and stays silent on whether it was correct. Someone still owns that call.
Congrats Paolo. That’s a really important topic. I’d love to have a feedback from you on this tool I made purely to understand my iterations with AI. Still not Otel compliant but able to work with all the local agents like cc or cowork.
https://github.com/maxturazzini/local-agent-viewer
I would really appreciate any contribution. Thanks
Max
Hey Paolo! Thanks for this detailed explainer.
I've had great luck tracing using the built-in Claude transcripts of my agents. Once I knew they were there, it was easy enough to work with another agent to parse the json and do forensic analysis. In one case, I found out that Claude's web search is a web search tool built by Anthropic which easily explained why my research agents were missing an association that was easily looked up on Google. In another case after watching a cascade of behavior among a team of agents, I was able to isolate the cause to a mechanism where each agent was first investigating portions of the repo it lived in before deciding what to do via an ls command. Tracing has been very useful to understand what's going on in my agent-enabled applicaitons.
Parsing the transcripts with a second agent for forensic analysis is a sharp move. That ls-before-deciding cascade is the kind of emergent behavior only tracing catches. Underused surface.