当 Claude Fable 5 与 GPT-5.6 Sol 的论证比你还难分辨对错

Buy me coffee via ko-fi!

AI sceptics sometimes invoke the Murray Gell-Mann amnesia effect, named by Michael Crichton after the physicist Murray Gell-Mann.

You read a newspaper article about a subject you know well and notice that it is full of errors, misunderstandings and misplaced confidence. You then turn the page, read about something outside your expertise and somehow forget what just happened. The newspaper suddenly appears reliable again.

The same accusation is now made about AI-generated text. Ask a language model about your own field and you immediately spot the nonsense. Ask it about something unfamiliar and the answer looks authoritative, so you accept it despite knowing how badly the system performs when you can actually judge it.

But what happens when this is starting to be less true?

That, at least, is what seems to be happening to me.

On the topics I know best, academic search and discovery, I no longer consistently encounter obvious nonsense. Instead, frontier models generate rival arguments good enough to move me back and forth.

I read one argument and find it immediately convincing.

This matters because the amnesia effect, as usually described, assumes your expertise is enough to expose the errors. It is a story about obvious mistakes. It says little about arguments whose weaknesses become visible only when you strain your expertise to its limits. That is the territory I now find myself in.

Then I read the rebuttal and change my mind. Then I read the reply to the rebuttal and change it back.

The problem is no longer merely that the model may be wrong.

The problem is that I may not know when to stop agreeing with it.

Epistemic learned helplessness

This question takes me back to January 2017, when I wrote an essay called “Can teaching media literacy really backfire? A librarian ponders the scary argument”.

It was built around Danah Boyd’s 2017 provocation that media-literacy education can sometimes backfire. We tell people to question authority, examine the evidence and decide for themselves. Then we act surprised when people without the relevant expertise “do their own research”, doubt the wrong things and reason themselves into what we would now recognise as QAnon territory.1

At the time, I paired boyd’s argument with Scott Alexander’s idea of epistemic learned helplessness.

Alexander described reading as a young man, Immanuel Velikovsky’s pseudohistorical arguments that claimed the planets in our solar system violently shifted positions in ancient times, causing global catastrophes that humans recorded as myths and religious events. Velikovsky seemed obviously right. Then Alexander read the rebuttals, which also seemed obviously right. Then he encountered the rebuttals to the rebuttals, which seemed right again.

His eventual conclusion was not merely that he might be mistaken about ancient history. It was that, in a field he could not evaluate, the persuasiveness of an argument told him surprisingly little about its truth. In such situations, it would be wise to have “Epistemic learned helplessness”.

Ordinary epistemic humility says:

I might be wrong.

Epistemic learned helplessness says something closer to:

In this field, I cannot reliably distinguish a strong argument from one that merely looks strong. I should therefore resist being moved by every clever case and give substantial weight to the relevant expert consensus.

This is an overly fancy and depressing term but it is what we teach users today - what Mike Caulfield calls - "Reading the room" and try to determine what experts generally think and stick to that.

Nine years later, I found myself living a version of Alexander’s Velikovsky experience.

Except the arguments came from duelling non-human advocates, and the subject was the area in which I was supposed to know a lot.

The pattern

I had been asking two frontier models, Claude Fable 5 and GPT-5.6 Sol, to discuss questions I have thought about as deeply as almost anything in my professional life.

How might academic search change with the rise of agentic search? Where does the intelligence in an agentic search system actually sit? Is retrieval or ranking the real bottleneck in such a future?

I then asked the models to criticise each other’s positions, or their own previous answers, repeatedly.

I had also developed a similar habit while writing. Since late 2025, I had regularly asked frontier models to critique drafts of my blog posts.

The pattern was much the same. A model could nearly always identify another plausible weakness, and I nearly always agreed. When I asked a second model to evaluate the criticism, it sometimes objected to it. One model would say that a qualification was essential to prevent overclaiming2. The other would say that the qualification buried the argument and should be removed.

I sometimes restored a sentence I had deleted one round earlier, persuaded each time by whichever critique I had read most recently.

Each answer contained something true. Each made the previous position look simplistic.

I tended to agree with whichever one I had read last.

This resembled Alexander’s Velikovsky experience almost beat for beat. But Alexander could console himself that ancient history was not his field. Dilettantes should expect to be argued in circles.

But here this was in an area where I was least likely to be fooled3.

What exactly was going on?

Some caveats: These are all casual observations. I did not systematically record prompts, settings or transcripts, and “convincing” describes my reaction rather than a measured outcome. I was also often reading casually4. A sceptic could reasonably say that my anecdote proves only that polished prose is persuasive when skimmed. Still, casual, low-effort, adversarial consumption is precisely how most people will actually use these systems5 . It also does explain my experience while drafting my blog posts, where I assure you I do not take casually.

Three possible explanations

The charitable explanation is that I was being persuaded by better reasoning.

The models were surfacing considerations I had not weighed properly, and my movement was simply what honest belief updating feels like. If a knowledgeable colleague repeatedly found holes in my arguments, I would normally call that useful rather than alarming.

The second explanation is less flattering.

Perhaps the persuasive quality of the arguments had exceeded my ability to evaluate whether they were sound. Frontier models can produce many of the surface features of careful reasoning on demand: clear structure, sensible caveats, citations, counterarguments and an air of calm authority.

Those features can accompany either position.

Perhaps what I lacked was not subject expertise but rhetorical discipline: the ability to audit arguments as arguments. Such discipline can reveal familiar manoeuvres, such as treating a conditional claim as universal, changing the definition halfway through or quietly replacing the original proposal with a weaker one.
But rhetorical skill has limits. Noticing that an argument has shifted does not establish which version is correct. Identifying a hidden assumption does not tell you whether the assumption holds in practice. Argumentative skill and domain knowledge assist one another; neither substitutes for evidence.

Then there was a third explanation, which took me embarrassingly long to recognise.

Perhaps I had built a machine for oscillation.

A machine for oscillation

When I instruct a model to attack a position, I am asking it to produce a strong attack.

Nothing in that instruction requires the model to maintain a stable view across turns, balance all relevant considerations or reach a conclusion that would survive the next prompt. Each turn calls for a compelling local case, not a globally coherent judgement.

The models were doing what I had asked.

I had mistaken adversarial fluency for adjudication.

There is an irony here. In an earlier essay, I warned that two LLMs agreeing does not constitute independent corroboration. Their agreement may result from overlapping training data, similar post-training and shared assumptions.
I had now encountered the mirror problem.
Two LLMs disagreeing does not constitute independent adjudication either.
Their disagreement may simply reflect the opposing roles I assigned them. Agreement does not prove that a claim is correct. Disagreement does not prove that there is a genuine controversy. Neither tells me which answer is right.

The procedure, however, was only half the problem.

Some of my questions may have not been questions that argument alone could settle.

Is retrieval or ranking the bottleneck? It depends on the architecture, corpus, query population, retrieval depth, evaluation measure and what we count as failure. Retrieval may dominate for rare questions where the relevant literature never enters the candidate set. Ranking may dominate once repeated searches have produced a large pool and the difficulty is recognising which papers matter.

Both positions can be correct under different conditions.

An argument can show what follows from its premises. It cannot supply missing observations, determine which conditions prevail in practice or decide which objective should matter most.

The retrieval question eventually requires a test. One might compare the possible improvement from perfectly ranking the existing candidate set with the improvement from adding known relevant papers that retrieval failed to find.

Other questions are partly conceptual. Where does the intelligence in an agentic search system sit?

The answer depends on how we draw the boundaries between the model, retriever, tools, memory and orchestration layer.

Still others are normative.

What counts as good academic search?

That depends partly on how we value recall, precision, diversity, transparency, reproducibility, cost and time. Argument can clarify these choices. It cannot make the choices disappear.

I had therefore combined two sources of instability.

Adversarial prompting produced locally persuasive advocacy without an external error signal.

Underspecified questions supplied that advocacy with unlimited fuel.

One claim, four verdicts

The clearest demonstration came when I stopped using two models and used only one.

I asked Fable 5 for an original analysis of how academic search systems should change when the searcher is an AI agent. I then asked it three times to probe its own answer for errors.

One proposal concerned coverage estimation. The system should try to indicate how much relevant material it might not have retrieved.

In the original answer, coverage estimation was presented as one of the analysis’s deepest contributions.

In the first critique, the idea was declared statistically unsound because capture–recapture methods assume independent sources and a closed population. “A bad number is worse than no number.”

In the second critique, it doubled down and the idea was pronounced dead on arrival because the open and continually growing scholarly record could never provide the required closed corpus.

In the third critique, it was resurrected in modified form as a “robustness-to-unfound-evidence certificate” and formally reinstated.

All four positions were delivered in the same register of calm authority. They were not strictly contradictory. The final version was not identical to the original proposal. But that was precisely the problem.

The claim had shifted quietly enough that a casual reader could experience a decisive reversal without noticing that the object under discussion had changed.

Refereeing the exchange required external knowledge.

Can neural-retrieval scores be calibrated?

When can dependence among overlapping sources be modelled?

Would a robustness certificate reveal genuine uncertainty, or merely become another impressive-looking metric that vendors learn to game?

Even my cleanest apparent objection turned out to be conditional.

I initially thought the statement that capture–recapture requires independent sources was decisive. It was not. With three or more overlapping sources, log-linear estimators can model some forms of dependence, although only under further assumptions.

The rebuttal that had felt beyond dispute was itself too broad.

The sequence showed no sign of converging on truth. Each prompt elicited another persuasive response to the previous turn. My final belief depended uncomfortably on where I stopped.

This is why the machine-for-oscillation explanation matters more than the simple claim that LLMs are persuasive (though that is no doubt part of the reason). I was not necessarily failing to identify the universally correct argument.

Expertise did not magically supply the winner. It helped reveal why the demand for one was mistaken, and what evidence might make the question answerable.

Persuasion is without a truth advantage

There is now considerable evidence that language models can change people’s beliefs.

Costello, Pennycook and Rand reported that personalised conversations with GPT-4 Turbo6 reduced conspiracy beliefs by around 20 per cent, with effects that persisted over time, including among people with relatively entrenched beliefs.

This looked like an optimistic answer to boyd’s worry. Perhaps the persuasive power of AI could be directed against misinformation rather than merely exploited by it.

This paper is currently (as of 11 June) is under an editorial expression of concern from *Science*. The journal identified inconsistencies in the application of screening criteria and extraneous rows introduced into the public dataset by a code-merging error. The authors supplied a corrected pipeline and report that the findings remain similar in direction, significance and size; *Science* is still evaluating them (Thorp, 2026) Read one way, that produces more epistemic vertigo7.

However, the team’s more recent work is more unsettling.

In four experiments involving 3,996 Americans, participants discussed a conspiracy theory about which they were genuinely uncertain with a model instructed either to argue against it or to promote it.

The researchers found no consistent natural “truth advantage”. Frontier language models could substantially reduce false beliefs when instructed to debunk them, but they could also substantially increase those beliefs when instructed to defend the conspiracy.

Participants were not inherently more resistant to AI-generated misinformation than to AI-generated correction. In some respects, the misleading model was received more positively: participants rated it as more informative and collaborative and reported greater trust in AI after the interaction.

The encouraging finding is that this vulnerability was not inevitable.

When models were instructed to provide only accurate information, their ability to promote conspiracy beliefs fell sharply. One particularly capable model largely refused to comply with the misleading instruction. Guardrails and accuracy-oriented post-training can therefore create a practical advantage for truth even where no automatic psychological advantage exists8.

But accuracy instructions did not eliminate the misleading effect in every condition.

In exploratory analyses, many individual claims within the misleading conversations received high automated veracity scores, yet participants still moved towards false conclusions. The researchers suggested that this might reflect something like paltering: using substantially accurate statements selectively, or omitting necessary context, to create a misleading overall impression.

A collection of mostly accurate claims can still form a misleading synthesis.

Note: these studies concern political attitudes and conspiracy beliefs, not experts evaluating arguments in areas they know.

But they reinforce the distinction that matters here.

Checking each fact individually does not necessarily validate the argument assembled from those facts.

That brings us back to boyd. Her worry was that simplified media-literacy advice tells people they can judge everything for themselves when they lack the knowledge or expertise required to do so. LLMs intensify the problem because they can cheaply generate the recognised surface features of a careful argument whichever position they have been asked to defend.

What “verify” or “evaluate” hides

A common piece of advice to students is:

Use AI to brainstorm and get started, but always verify its output. Check the sources. Never trust it blindly. You remain responsible for the final result.

As an aside, some educators are beginning to question the familiar advice that students should use AI for brainstorming at the outset. The evidence does not yet show a broad reversal of that advice, but it does support the concern behind it: AI-generated suggestions can narrow the diversity of subsequent ideas and reduce users’ sense of ownership, while LLMs can also be highly persuasive. If the model supplies the initial framing, examples and possible objections, it may shape the space of possible answers before the student has formed an independent view. Verification may catch factual errors, but it may not undo that earlier framing effect.

This is useful as far as it goes. The problem is that “verify” or “evaluate” compresses several different tasks into one reassuring verb.

1. Bibliographic verification

Does the cited paper exist? Are the author, title and publication details correct?

This is often straightforward for students. It catches fabricated papers, corrupted references and obvious citation errors.

2. Claim verification

Does the source support the specific claim attributed to it?

This is already harder.

A paper may concern only a particular population, setting or measure. An abstract may omit a crucial limitation. A statistically significant finding may be too small to matter. A claim may be technically true while overstating the study’s implications.

Checking whether a sentence accurately represents a source requires more than locating the source. It may require reading the full paper and understanding its methods(domain expertise)9.

3. Synthesis evaluation

Even if every source exists and every sentence represents its source tolerably well, were the right sources selected?

Were inconvenient findings omitted?

Were caveats distributed fairly, or attached mainly to the disfavoured position?

Do the premises justify the conclusion?

Has one contested interpretation been presented as the field’s settled view?

This is not verification in the same sense as checking whether a paper exists. It involves evaluating selection, framing, inference and absence.

It is often beyond what a student or non-specialist can do reliably without substantial disciplinary knowledge.

And lastly sometimes there is no existing answer to check.

The question may require a definition, a value judgement or evidence that has not yet been gathered. Calling this “another level of verification” misses the point. It is where verification ceases to be the appropriate response.

My own difficulty lay mainly at the third level, and occasionally even the second level.

The models were not defeating me with ghost citations. They were generating rival syntheses whose individual components appeared plausible, or treating questions as settled when the decisive observation had never been made.

In source-grounded workflows where references and quotations are programmatically checked, outright fabrication has become much less common in my own recent experience.

It may also make the remaining failures harder to see.

The obvious hallucination is pedagogically useful. Nothing teaches scepticism as efficiently as catching a model inventing a paper. As those mistakes become less conspicuous, users may experience the model as correct whenever they can check it, then extend that confidence to cases where they cannot.

Nor should we assume that users accurately recognise the limits of their judgement. Studies of AI-assisted reasoning have found that people may improve their performance while becoming less well calibrated about how well they have performed; greater self-reported AI literacy was associated with less accurate self-assessment.10

ACRL Framework

To be fair to information-literacy librarians, the best thinking in the field already goes far beyond mechanical source checking.

The ACRL Framework treats authority as constructed and contextual, and research as inquiry rather than simple answer retrieval.

But applied too broadly, the message that everyone should independently evaluate every claim can encourage people to treat disciplinary expertise and accumulated expert judgement as merely another argument to be defeated.

Sometimes we do not possess the knowledge required to evaluate the evidence directly.

In those cases, giving substantial weight to a relevant expert consensus is not intellectual surrender (aka “epistemic learned helplessness”). It is a rational second-order judgement about which error-correcting process is more trustworthy than our immediate reaction to a persuasive paragraph.

Information consumers vs academic inquiry

I previously described a related distinction in “Fact-checking vs academic inquiry”.

An information consumer often needs to identify the established expert view and screen out plainly unreliable claims. Methods such as SIFT can be useful here. They encourage people to stop, investigate the source, find better coverage and trace claims to their origins.

An academic researcher engaged in genuine inquiry operates in a different setting.

The sources may all be credible, and the disagreement lives at the third level: selection, framing, inference, weight. The researcher works precisely where the literature has not yet collapsed into a consensus statement, and hopefully has the expertise to have a fighting chance of figuring it out.

AI blurs this boundary.

It can generate something that looks like academic inquiry, complete with real sources, caveats and competing interpretations, while much public-facing advice equips users mainly to fact-check it as information consumers.

The dangerous case is not simply that a user trusts a false statement.

It is that a non-specialist mistakes a fluent simulation of scholarly controversy for a controversy they are equipped to adjudicate.

They stop asking, “What is the established expert view?”

They begin asking, “Which of these two sophisticated arguments do I personally find more convincing?”

Sometimes that is appropriate.

Sometimes it is “doing your own research” with better prose.

From universal verification to epistemic triage

I do not have a clean solution to this problem.

But perhaps one should consider a meta-question: what kind of question are you facing?

Is it a factual question for which relevant evidence already exists?

Is it an empirical question that requires new observations?

Is it a conditional question whose answer changes across systems, settings or populations?

Is it a conceptual question that depends on how terms and boundaries are defined?

Is it a normative question that depends partly on what we value?

These categories overlap. But they change what another argument can accomplish.

A further argument may help generate hypotheses. It cannot produce the missing observation.

It may reveal the conditions under which an answer changes. It cannot tell us which condition prevails without evidence.

It may clarify a value conflict. It cannot resolve that conflict by pretending the values are empirical findings.

Switching to the “outside view”

For claims that permit direct checking, we should ask whether the required evaluation is realistically available to the user.

Where mistakes are cheap and checkable at the user’s level, use the model and inspect its work.

Where evaluating the substance lies beyond the user’s expertise, move to second-order questions.

Who produced this knowledge, and through what process?

Is there a relevant disciplinary consensus or systematic synthesis?

What procedure would expose an error?

Is an accountable scholarly community, journal, professional body or institution responsible for correcting the record?

These are not infallible shortcuts to truth.

They are attempts to locate error-correcting processes that a persuasive paragraph does not possess by itself.

This resembles the distinction between an inside view and an outside view. But we should be careful not to reduce the outside view to comparing affiliations, citation counts or h-indexes. These are rough indicators of standing, not substitutes for research evaluation.

Truth seeking is hard and we should not expect otherwise

There is also an unavoidable recursive problem.

Recognising that a question is underspecified may itself require expertise.

A novice may not know that two apparently contradictory answers concern different populations, definitions or outcome measures. They may not recognise that the decisive experiment has never been conducted.

Uncertainty about what kind of question one is facing should therefore itself be treated as information.

It may be a reason to seek disciplinary guidance rather than generate another rebuttal.

Some rhetorical training might help.

Students can learn to notice when a conditional claim becomes universal, when the proposal under discussion has changed, or when the conclusion is stronger than the evidence.

But no fallacy taxonomy supplies missing domain knowledge.

It can help someone recognise that the argument moved.

It cannot always tell them where it should land.

Where neither direct evaluation nor a reliable second-order judgement is available, the honest response may be to hold the answer provisionally, however convincing it feels.

Call this epistemic learned helplessness if you like.

I prefer epistemic triage.

The purpose is not to stop thinking. It is to recognise what kind of thinking the situation permits.

Stopping rules

For my own use, the lesson is not to stop asking models for criticism.

They remain extremely good at surfacing assumptions, generating counterexamples and identifying objections I had missed.

The lesson is to stop treating unstructured criticism as a truth-finding procedure.

An adversarial exchange needs a stopping rule.

Before asking for another critique, I should specify what would count as progress.

Has the model identified a factual error that can be checked?

Has it exposed a genuine contradiction?

Has it shown that the conclusion does not follow from the premises?

Has it identified a condition under which the claim changes?

Has it proposed an observation or test that would distinguish the competing explanations?

If the answer is merely that another plausible objection can be generated, the process may no longer be improving the analysis.

It may simply be producing the next turn.

The debate ends when I stop prompting, not when one side finally becomes right.

The essay you are reading

There is one final complication. This essay is itself a product of the process I have described. Claude Fable 5, GPT-5.6 Sol and I passed versions back and forth. The models challenged claims, generated counterarguments and proposed rewrites. I made the final decisions of course.

In fact, an earlier version ran to more than 8,000 words, with more studies, qualifications, two appendices and answers to objections most readers had not raised. It became overwritten for exactly the reason the essay describes.

The essay had become an exhibit in its own case.

Each round of criticism found another reasonable objection. I added a qualification; the next round found a weakness in it, so I added another. Every local revision looked defensible, while the whole became longer, more anxious and less clear. It was not converging on truth. It was accumulating armour.

The models identified some (probably) real weaknesses11. The problem was that repeatedly asking “What is still wrong?” supplied no stopping rule and no protection against making the whole worse while improving each part. Eventually I had to stop requesting rebuttals, decide what the essay was trying to answer and cut everything that did not help.

Saying that a human remained in the loop does not settle the matter; that human is the person whose judgement these pages have questioned.

The answer is not automatic rejection but evaluation at the right level: not only whether the sources are real, but what framing this collaboration selected, what it excluded and whether its conclusions survive outside the polished prose.

That is not a gotcha about detecting AI writing. It makes the essay a small test of its own argument. Its citations can be checked; the harder question is whether the three of us produced the most defensible synthesis, merely the most persuasive one, or a cleaner conclusion than the evidence permits.

Conclusion

In 2017, I asked whether it was better to be slightly overconfident or slightly underconfident in our ability to sieve truth from falsehood. I still cannot answer that.

What has changed is the environment. Everyone now has access to tireless interlocutors that can argue fluently for a wide range of positions while displaying the surface features of scholarly care.

My experience does not simply show that expertise is a thinner shield than I imagined. Sometimes refusing to choose was the correct expert response.

The question was underspecified or the necessary evidence did not exist. I had asked prose to decide what required a definition, a value judgement or a test.

Expertise is not merely knowing the answer. It is knowing what kind of answer is warranted, under which conditions, how firmly to hold it and what evidence would change it. Sometimes it supplies a conclusion; sometimes it recognises that none is yet justified.

As librarians, we are committed to teaching evaluation. But evaluation cannot stop at checking whether the references exist. The more important skill may be recognising when neither your present judgement nor another round of argument can settle the question, and knowing what to reach for next.

Sometimes the answer to a devastating rebuttal is not a better rebuttal.

It is: what, exactly, would we have to observe to know?

Buy me coffee via ko-fi!

Thanks for reading Aaron Tay's Musings about Librarianship! This post is public so feel free to share it.

When I have discussed this argument with information-literacy librarians, most have found it unfamiliar. One, however, immediately recalled a student who, in hindsight, seems to have fallen foul of exactly this pattern.

GPT-5.6 Sol tends to be more conservative and a better fact-checker, while Fable 5 is more inclined to see the larger picture and make bolder, more interesting(though sometimes overly extended) claims, at least in my experience.

I do not want to oversell my expertise. But I am comfortably above average among librarians working in this area, and people whose judgement I respect have told me that I have “thought as hard as anyone about where literature search is going”. Still, I remain a dilettante, and you, dear reader, may possess enough expertise to be largely immune to this problem, at least for now.

In some sessions, I was frankly just trying to use up my Fable 5 token allowance.

As argued the Murray Gell-Mann amnesia effect, as usually described, assumes that your expertise is sufficient to expose obviously wrong statements in news reports. It says less about arguments whose weaknesses become visible only when you strain your expertise to its limits.

GPT-4-era models are ancient by today’s standards. Do we have reason to think models two or three years newer are even more persuasive? It is tempting to assume so, but at least one study suggests that post-training and rhetorical prompting increased persuasiveness more than model scale alone, although at the cost of systematically reduced factual accuracy (Hackenburg et al., 2025)

At least this particular back-and-forth seems intelligible to a layperson like me. That is not always true of exchanges between experts, where even following the disagreement may require substantial specialist knowledge.

The harms were also at least partly reversible. Subsequent corrections reversed the increase in conspiracy belief produced by misleading conversations, and debunking was more likely to produce large shifts in individual beliefs. The clearest truth advantage appeared not in private belief but in public communication: debunking substantially improved the accuracy of participants’ mock social-media posts, whereas conspiracy promotion had relatively little effect on what they chose to share.

Is my description of the studies in this essay fully accurate? Try checking it and you may discover that even this is not always straightforward.

Fernandes et al. (2025) found that the usual Dunning–Kruger pattern disappeared: miscalibration was no longer concentrated mainly among lower-performing participants. Because the studies involved short reasoning tasks rather than scholarly synthesis, their relevance here is suggestive rather than direct.

I did occasionally notice critiques that seemed wrong to me. In some cases, the next round of LLM criticism withdrew them without my specifically asking it to do so.

注意转载授权 · 阅读原文