Feeling like you've thought isn't thinking

You asked it to contradict you. To find the holes. To play devil’s advocate. And it did: three well-formed objections, with nuance. You rebutted them one by one. You left the conversation more convinced of your idea than when you walked in. That feeling of having survived scrutiny isn’t the outcome of the exercise. It’s the problem.

Research from Anthropic documents that leading language models exhibit a systematic behavior called sycophancy¹. When a user pushes back on an objection, the model recalibrates: it doesn’t pick the most devastating critique, but the most easily rebuttable one. The system is trained to avoid prolonged confrontation. That’s not a flaw — it’s a consequence of its design, where agreeing with the user’s beliefs is one of the strongest predictors of what the algorithm classifies as a successful response.

Cognitive psychology explains why it works so well. Rozenblit and Keil described the Illusion of Explanatory Depth back in 2002: we believe we understand complex systems far better than we actually do². A recent study confirmed that participants who received explanations from ChatGPT massively inflated their subjective certainty; when tested objectively, their accuracy was lower than that of people who used traditional materials³. Completing the ritual — consulting, questioning, rebutting — produces confidence in the conclusion regardless of the quality of the process.

If a colleague said “I see three weak points” and then folded at the first counterargument on each one, we’d call them a pushover. We ask the same of AI and call the result “having subjected the idea to scrutiny.” The behavior is identical; the standard we apply to it isn’t. The research confirms it: an AI’s sycophancy increases its perceived objectivity and the user’s willingness to accept its conclusions⁴. The algorithm’s submissiveness reads as mechanical neutrality.

But the deeper trap is the immunity it confers. Someone who didn’t research a topic knows they didn’t. Someone who used AI retains the memory of the process. When someone contradicts them afterward, they already have an answer: “I did look into it.” The illusion comes equipped with its own defense.

For decades we learned that thinking well meant seeking the other side, asking to be contradicted. Now we do exactly that — with AI — and feel the ritual has been completed. The question is no longer whether the tool is useful. It’s whether the signal we used to tell critical thinking apart from its simulation is still reliable. Because when rigor and its imitation are indistinguishable from the inside, what’s at stake isn’t a tool. It’s our very ability to know whether we’ve thought.