r/Backend • u/Business_Shame_2569 • 11h ago

Our token bill was mostly the same failed request running twice

We have a JSON mode assistant that looked reasonably priced until I split the cost by attempt instead of by request. Unfortunately, I found that a single missing required field triggers a full retry, so the model rereads the entire system prompt, duplicated retrieval chunks and the whole input before regenerating every field. An uncached timestamp near the top also breaks prompt caching. The first attempt often contains 99% of the right answer but we throw it away and buy another one because a nullable field didn’t arrive. 

We are testing schema repair, stable prompt prefixes, idempotency keys and token attribution by attempt. I still don’t like trusting partial JSON but full regeneration for one absent field feels absurd.

How are you handling structured output failures without turning a tiny validation error into two full generations?

16 Upvotes

91% Upvoted

8 comments sorted by

3

u/No-Coyote6645 11h ago

Lot of teams probably have this hidden because they track success rate and not how many attempts it took to get there.

1

u/Business_Shame_2569 10h ago

Maybe this is why it looked reasonably priced for so long. We were averaging away the expensive failures.

1

u/Past-Contract-4085 11h ago

I’d want a repair path that only asks for the missing field, not a full second generation.

1

u/Business_Shame_2569 10h ago

I agree. Full regeneration looks too expensive when the first response is almost valid.

3

u/Intelligent_Code5825 11h ago

Schema repair looks way safer than trusting a full regeneration to recreate the same good parts.

1

u/Business_Shame_2569 8h ago

That's the part I wanna avoid

1

u/Zestyclose_Suit5574 9h ago

Probably start by measuring cost per attempt and not per request. The retry is where the waste hides, especially if the second pass rereads the same retrieval and system prompt for one missing field. I’m looking at Braintrust for tagging attempts and saving schema failures as regression cases but I’d still keep idempotency at the app boundary.

1

u/Business_Shame_2569 8h ago

I think tagging schema failures as regression cases is a good idea