How do you catch it when a model silently changes under you?

[u/pedroassumpcao]

We run prompts against a few different providers (OpenAI, Anthropic, some stuff through OpenRouter). Every so often something quietly gets worse, the output quality drops, a prompt that worked starts returning junk, or a model gets deprecated and the replacement behaves differently.

Right now we mostly catch it by accident: someone notices, or a customer complains. That feels bad on us, a lot.

How do you all handle this? Do you re-run some kind of fixed eval set on a schedule? Just eyeball it? Have something that alerts you?

Any insights I could use?

Thanks.

[u/Mundane_Incident_853]

I'm confused - if it's local, don't you have absolute control over which model you're running?

Are you on the wrong sub?

[u/whatsupnorton]

Well if you were running local like this subreddit then you might be able to get some answers, but you’re not so I have no idea

[u/Weird-Consequence366]

sha256sum is a good indication

[u/Equivalent_Bit_461]

Local how?

[u/EvolvingDior]

r/lostredditors