In regards to benchmaxxing...

[u/Nevermore1215]

With benchmaxxing being a high status concern amongst many users, it's reasonable to assume that most open bench harnesses have been trained for. Whether or not that is the case, we'll never truly know.

I wanted to toss in a suggestion because I think this would reasonably nullify a good portion of the concerns that come from models being trained to complete a bench.

Why doesn't everyone simply ask their agent to create a bench that hammers the subjects and topics of what YOU regularly do? that way, the bench metrics are unique to your use case and you can identify whether or not a model fulfills your needs whether it be different quants, different fine tunes, different models, or even KV weights.

it might be a bit tedious but think of it as a "one time" pain to create it and then have your newly downloaded models or configs run the gauntlet?

---------

this almost certainly obliterates the believed compromise that a model was trained to have good benchmark scores because I doubt any company is going to have training access to a harness you had your agent create... post release.

I'm curious what others think, what other ideas there are to get accurate tests, etc!

[u/No-Fuel-9202]

For LLM Labs, Benchmaxxing is like Prisoner's Dilemma, if they don't do it, they know that others will...

[u/nickm_27]

I think lots of users who use models for something serious have their own benchmarks, I have 3 different primary use cases each with a decent set of benchmarks that work quite well at least as an initial litmus test on if a model is even worth bothering with.

The wisdom has always been "use the model that is best for your use case, not the one that scores highest on XYZ" A lot of the talk around benchmaxxing comes from those that use benchmark scores as the direct decider on which models are best.

[u/Viktri1 → u/nickm_27]

With something like Hermes it's become quite easy to run the benchmarks too so that even non tech people like me can run our own benchmarks and analytics.

[u/whichsideisup]

Software development coming full circle.

[u/Defiant-Lettuce-9156]

As someone who makes benchmarks for my local models, it’s not easy. It’s a whole rabbit hole to go down if you get into benchmarking. If you want to do it accurately that is.
A bad benchmark is worse than no benchmark, and it’s easier than you think to make an inaccurate benchmark

[u/Lesser-than]

if it floats you boat sure, its not a measurement worth sharing with anyone though as its on your use case, or how you like to prompt a model. I usually have a few small tasks I ask a new model, they are simple enough most new models should be able to handle it, but the way they get to the answer is more important than if they can answer it to me.

[u/gjr23]

Completely agree. I have done this and have found it quite useful given its personalized nature. I was also tired of reading tok/s at 4k context which was fast but arguably not very useful.

[u/ea_man]

I have a dozen of those, some ran like 250 times to evaluate similar quants and finetunes.

[u/simrankoulsm]

Public benchmarks are useful for narrowing the field, but a small private regression suite based on your actual prompts is probably what determines whether a model is worth keeping. I would score more than final-answer accuracy too like format compliance, hallucination rate, consistency across seeds, tool use, and how well it handles ambiguous requirements.

[u/En-tro-py]

Why doesn't everyone simply ask their agent to create a bench that hammers the subjects and topics of what YOU regularly do?

That's generally the advice around here - everyone's workload, expectation, and hardware is different so you kinda need to to go DIY or just go on posted benchmarks and vibes.