r/LLMDevs • u/Sensitive-Parsnip-12 • 1d ago
Help Wanted does anyone else feel like debugging multi step LLM apps turns into detective work pretty quickly?
i’m curious how common this actually is.
you build an LLM workflow with tools, retrieval, state, maybe some evaluators or retries.
then something weird happens.
nothing necessarily crashed. the workflow may have completed perfectly fine.
but now you’re going through logs trying to figure out where the behavior first started getting weird.
i’ve heard people say they trace call order, compare step inputs and outputs, inspect state, replay the run, or compare it against something that worked.
what are you guys actually doing?
i’m working on Traser because of this problem. it tries to take a big execution and narrow it down to a few differences worth investigating.
but the last thing I want to do is build a giant feature list based on assumptions.
I’d rather have LLM developers tell me what’s actually annoying, what’s already solved, and what I’m misunderstanding.
even if you think the whole premise is wrong, I want to hear why.
and if anyone has an ugly sanitized run they’d let me look at with them, that would honestly be more useful than a signup right now.
2
u/conifer_v11 1d ago
what i actually do is log the full serialized input to every model call, not just the step name. 90% of the weird runs are a retrieval chunk that shifted or a tool result that came back empty and nobody noticed.
diffing two runs only helps if the runs are actually comparable. temp 0 and a frozen seed on the retriever first, otherwise every diff is noise and you'll chase the wrong step.
1
u/Sensitive-Parsnip-12 1d ago
hmm the comparability point is probably the part i haven’t thought through enough yetif the runs are already drifting because of retrieval or sampling, the diff itself can send you down the wrong path. how do you usually decide when two runs are actually comparable enough to diff, and what do you use today to line up those full serialized inputs
1
u/conifer_v11 1d ago
comparable enough is when the prompt hash and the tool-result hashes match before the first sampled token. i dump every model call as one jsonl line — actual model id not "latest", messages, tools, tool_results, temp, seed, max_tokens — canonical json, sha256 the blob. if retrieval isn't frozen i don't start the diff at all, pin the corpus snapshot / retriever seed first or you're just diffing two different inputs and calling it a bug. lining them up is the request id on that line plus a normal json diff; first line whose hash disagrees is the split, everything after is fallout. temp 0 is necessary and not sufficient if the retriever or a tool is still rolling dice.
1
u/Sensitive-Parsnip-12 1d ago
super helpful primarily that part about not even starting the diff unless retrieval is frozen. sounds like for you the comparison is only useful once you’ve basically proven the runs had the same setup up until the split. do you usually do all of that hashing/comparison with your own script, or is there a tool in your stack handling part of it already?
1
u/conifer_v11 1d ago
own script. one jsonl line per call, canonical dump, sha256 the blob. i don't want a product wrapping it because then the serializer is another moving part. python hashlib +
jq -Sis the whole stack. the request id is just a field on that line so a normal diff finds the first hash that disagrees.1
u/Sensitive-Parsnip-12 1d ago
fair i can see why you’d want to keep that part simple if the whole point is having something you can trust when everything else is acting weird once you find that first hash mismatch tho, does the investigation usually get pretty obvious from there or is there still a bunch of manual work figuring out why that split actually mattered?
1
u/Wonderful_Day_8811 1d ago
i do this all the time and it feels less like debugging and more like being a detective in a show where the clues make no sense until the last 5 minutes. usually i start by comparing the state at key steps between a broken run and a working run, but the tricky part is when the failure is subtle like the output looks fine but some tool call happened in wrong order. logs get messy fast especially with retries
half the time i end up building my own little diff tool for the traces just to survive, so i see why you are building Traser. the narrowing down part is what matters, nobody got time to scroll through 200 steps
1
u/Sensitive-Parsnip-12 1d ago
yep, sounds all too familiar. i’m actually really curious about the little diff tool you built, especially since it sounds like you made it because the normal logs weren’t enough. what does it actually compare or surface for you when you’re looking at a broken run vs a working one? like is it mostly state changes, tool order, inputs/outputs, retries, or something else. and what part of that usually ends up saving you the most time when you’re trying to figure out what actually mattered?
1
u/eddzsh 1d ago
Worth logging which retry attempt produced the final tool args. A successful run on attempt 4 is a different beast from attempt 1, and plain step diffs hide that.
1
u/Sensitive-Parsnip-12 1d ago
good point especially if the final successful attempt hides everything that happened before it. how do you usually keep track of that now? do you look at every retry separately, or mostly care about which attempt produced the args that actually got used? and when you’re debuggingwhat about the retry history usually tells you something is worth digging into?
1
u/Training_Isopod3722 1d ago
The exact assembled prompt and tool args are what I'd keep first. A trace can look clean while one optional field quietly disappears between steps, then every retry just makes the story harder to read.
1
u/Sensitive-Parsnip-12 1d ago
yeah, the optional field disappearing between steps feels like the kind of thing that’d be really easy to miss once retries start stacking up. when you catch something like that, are you usually diffing the exact prompt and tool args step by step, or do you have some other way of spotting where the field first disappeared?
1
u/Training_Isopod3722 18h ago
Step-by-step, but only after pinning the inputs that can move on their own. I’d store canonical request records for every model call: model version, assembled messages, tool schema, tool results, retrieval snapshot, retry number. Then diff from the start and stop at the first changed field. Looking backward from the bad answer is where it turns into detective work.
1
u/Sensitive-Parsnip-12 18h ago
yeah that tracks starting from the beginning and stopping at the first changed field seems way cleaner than trying to reason backward from a bad final answer. what made you settle on that approach instead of starting from the output and tracing back?
1
u/RealSharpNinja 1d ago
Simply have a competing model debug. They love finding each others failures, it is informative to their own creators.
2
u/Sensitive-Parsnip-12 1d ago
i’ve heard people do this too. what do you usually give the second model when you ask it to debug the run, the full trace or just the parts that look suspicious? and how often does it actually point you to the right issue versus giving you a plausible sounding explanation?
1
u/RealSharpNinja 1d ago
I use a shared logging system that allows detailed access to actions taken by each, making audits easy.
1
u/Sensitive-Parsnip-12 19h ago
when you say audits are easy do you mean the logging usually makes the actual failure obvious too, or mostly that you can see everything each model did and then still have to reason through it
1
u/RealSharpNinja 17h ago
It's both. I have a complete dev lab ecosystem designed to capture not just what was done, but the context around it. I have plugins for claude, codex, grok, copilot, cline and opencode that not only capture the data, but apply enforcement. This makes it easy to bring a new agent into in-flight work to perform adversarial validation of the work done and the claims made by other agents.
1
u/Sensitive-Parsnip-12 17h ago
I see, that makes sense sounds like you’ve built a pretty serious validation layer around the agents. if you’re comfortable sharing any of it, even just screenshots or a rough overview of how the pieces fit together i’d love to see what you’ve built.
1
u/RealSharpNinja 16h ago
1
1
u/Sensitive-Parsnip-12 16h ago
took a look through the repo and what you built is pretty cool I liked how much context you preserve around each agent action instead of just logging the final event, especially model/provider metadata, tool calls / results and the normalization warnings when something doesn’t line up cleanly. that’s the kind of stuff that makes the history actually trustworthy to investigate. if you got any execution histories from a weird failure you’d be comfortable sanitizing and sharing I’d love to take alook at one. even if it’s just a single run where you already know what went wrong would be really useful for me to study let me know my discord user is maas_dorian
1
u/RealSharpNinja 15h ago
Just merged a lot of good stuff to main.
1
u/Sensitive-Parsnip-12 15h ago
appreciate you sharing the updated repochecked out the new goodies on main and the session logging changes stood out. keeping more provider context and preserving structured tool inputs and results gave me a few things to think about for Traser around provenance and normalization thanks a ton for helping me out 🙏 gave you a star feel free to check us out nothing out yet but working on it traser.dev
1
u/Dalius-Gabryelle 1d ago
Running the same input a few times and lining the steps up side by side helps a lot since the first place the runs split usually tells you more than the final bad answer
1
u/Sensitive-Parsnip-12 1d ago
when you line them up side by side, what usually jumps out first? tool order, state, prompts, args, outputs, or something else. and are you comparing against one run you already know is good, or looking across a few runs to see where one starts behaving differently?
1
u/veithIO 1d ago
Multi-step extraction pipelines here (documents in, JSON out). The unglamorous finding: when a completed run looks weird, the first divergence is usually at the input boundary, not in the model. The doc-to-text conversion emits a table in a different order, a header lands in a different chunk, and every step downstream looks slightly off while doing its job correctly. Diffing step inputs top-down from the start of the run got me further than reasoning backwards from the bad output.
So for Traser, what I'd actually want is the earliest step whose input differs from a known-good run, rather than a list of output diffs. Everything after that point is mostly consequence.
1
u/Sensitive-Parsnip-12 1d ago
interesting..specifically the input boundary part. that’s pretty similar to something to what someone else told me about looking at what each step actually received instead of starting from the final output. if you have one of those doc extraction failures you could sanitize, would you be open to letting me work through a good and bad run with you? i’d really like to see if Traser actually surfaces the same thing you find manually.
1
u/toothpastespiders 1d ago
I have a love hate relationship with it. The bad is that it's exactly what you're describing. Convoluted and tedious to track down. The good is that the underlying issue is typically that I'd been lazy about some fundamental design elements that 'should' have made it easy to track down. So in a sense it's forcing me to fix up some code that'd been on an eternal todo list. Because of that I usually do just track everything step by step.
1
u/Sensitive-Parsnip-12 1d ago
yeah makes sense too sometimes the debugging pain is probably telling you the system itself needs to be easier to reason about.. when you track everything step by step, what are you usually trying to find first? bad state, a weird tool call, unexpected input, or is it different every time?
1
u/frost_hearth_gift 1d ago
diffing logs is useless noise unless you freeze temperature and seed. I only trust traces that capture the exact byte sequence of every model input because anything else hides the real failure point
1
u/Sensitive-Parsnip-12 1d ago
so you’re basically saying the trace is only as trustworthy as the exact input snapshot behind it. how are you capturing that byte for byte today, just raw serialized payloads before each model call or something more structured?
1
u/mastra_ai 1d ago
Just last will we launched multi-turn evals for Mastra. Let us know if it fits your use case.
You can use gates to assert deterministic actions, and LLM-as-judge to grade the conversational context for accuracy.
You can then using Mastra's built-in tracing to examine both the primary workflow output and the automated evals.
1
u/IndieGoHacker 1d ago
my experience is that the weirdness almost always starts at the first step that touches something outside the model — a retrieval chunk that shifted, a tool returning empty or in a different order, a header landing in a different chunk. once i freeze those boundary inputs and log the exact assembled prompt per step, the model itself is usually just doing what it was given and the real bug is upstream of it. diffing a good run against a bad one only works if you're comparing the same inputs, which means pinning the retriever snapshot and tool results first or you're just chasing noise.
1
u/Sensitive-Parsnip-12 1d ago
this keeps pointing upstream more than at the model itself when you pin the retriever snapshot and tool results, what are you actually using to compare the runs after that? just a normal json diff, or have you built something around finding the first boundary where the inputs stop matching?
1
2
u/Physical_Economy_340 1d ago
yeah totally detective work. what saved me was logging the exact assembled prompt plus tool args at every step and diffing a good run against a bad one, you spot an order flip or a missing field in seconds instead of scrolling 200 steps. if traser can auto surface just the few steps where inputs diverge that is the real win.