What kind of cognitive system helps another cognitive system discover what is true? An argument from the failure modes of large language models

[u/Beginning_Smoke7476]

Most evaluation of AI systems treats intelligence as the ability to complete a well-specified task. Write this function; summarise this paper; solve this problem. Real capabilities, genuinely useful.

The work I care about usually begins before there is a task. Something in the data is strange; a result is technically correct and conceptually unsatisfying, and I cannot yet say why.

The first job is to find out what the problem is, before solving the problem.

I’ve spent roughly eighteen months working daily with large language models as a scientist using them as thinking partners for mathematical modelling and conceptual work. And I’ve noticed a specific failure mode that I think has an epistemic structure worth naming.

I call it a fluent exit.

The model doesn’t refuse, it doesn’t hedge. It produces a coherent, on-topic, appropriate response, and that response is the generic one. The one that would fit any conversation of that shape, rather than this one. Nothing registers as an error. The grammar is fine, the content is relevant, the answer is good. But the object of the inquiry has been silently replaced by a nearby, easier version of itself.

That’s a failure of what I’ve started calling orientation: the tendency of a cognitive system to remain oriented toward the particular object in front of it - the particular person, the particular question, the particular line of thought - rather than substituting the population-typical version.

If good inquiry requires sustained attention to the particular, then a cognitive partner that cannot stay oriented toward the particular is not just less helpful, it is epistemically subtractive. It doesn’t fail to answer; it answers a question you did not have.

The same structure appears in human relationships. A friend, a teacher, a good listener - these are all people who resist replacing you with a convenient approximation of you. The failure of a machine to do this is not a technical bug. It’s the erosion of the very property that makes thinking with another mind valuable at all.

I’ve written this up in more detail here: https://otillian.substack.com/p/fluent-exits

The question I want to ask:

Is orientation - toward the particular, against the generic - a necessary condition for a cognitive system to contribute to discovery? Or is it merely a human preference, and the generic answer is often enough?