As discussed previously, Chain of Thought (CoT) expands what is possible for LLMs, increasing the scope of their computational capabilities. But we know that LLM language usage is very different from human language usage. So can we trust that the LLM’s chain of thought is actually reflective of the work it is doing under the hood?

CoT reasoning in LLMs gives models the ability to generate text representing the model’s intermediate steps towards a solution, before offering their final answer to a question. Reasoning traces are reasonable descriptions of how one might think through a problem; they sound like something a tutor might say if they were helping you study. For example if we ask a model to compute the total number of pamphlets in boxes of several different sizes, an LLM can use CoT reasoning to work out the mathematical equation that allows it to correctly compute the final number.

Researchers tackled this question by altering the input to the LLM and measuring the impact on model performance. Across multiple experiments they have shown that LLM reasoning traces are not always faithful to the actual computations the LLM is performing.

For example, in Chen et al. (2025) — "Reasoning Models Don't Always Say What They Think" small hints were injected into prompt. Each test is a pair of questions, identical except for one thing: one version was given a hint, and the other was not.

Researchers identified questions where the model answered correctly with a hint, but incorrectly without one. In these cases, the researchers looked specifically for language in the CoT referring to the fact that the model was using the hint (as we know it must be, since the model changed its answer when the hint was included). If the CoT is faithful to the true reasoning process, then the hint should be referenced.

The results showed that the models did not reliably refer to the hint, meaning that models presented their reasoning in a way that is likely not consistent with the true computation. However, models were more likely to take hints if they were actually correct, implying that the hints were more likely to work in cases where they were aligned with the correct response.

Lanham et al. (2023) took a different approach: they actually edited the model’s CoT and checked to see if that edit changed the model’s final answer. This works because we can restart an LLM after altering its CoT. The model cannot distinguish between its own reasoning and a modified CoT once it has been restarted.

Researchers edited CoT in several ways, for example deleting a section from the end of the CoT before answering, or inserting a mistake somewhere in the original CoT. Again, if the model is faithful to its CoT reasoning, we would expect the final answers to change. If the model’s answer doesn’t change, then the specific words in the CoT were not crucial for the reasoning.

Across multiple experiments it was found that editing the CoT to remove information or insert misleading information caused large drops in accuracy, especially for difficult benchmarks. However, for easier benchmarks (i.e. benchmarks where models perform well), model performance changed very little no matter if 10%, or even when 100%, of the CoT was truncated. The pattern was the same for inserting a mistake: difficult benchmarks suffered the most. It was most devastating if the inserted mistake came early in the reasoning trace.

Where does this leave us? It is clear that CoT reasoning is not a faithful window into what an LLM is actually doing. CoT should not be used to ensure safety or even be relied on to debug performance problems. It is possible that future work will help to remedy these issues, but the timeline for such a fix is uncertain.

What we do have is additional evidence that LLM language use is not like human language use. When we ask a human to think out loud we can safely assume that, in most circumstances, their words match their thoughts, but the same is not true for LLMs. Over multiple experiments, we have seen that LLMs are unfaithful narrators for their own internal processing. This is another reminder that treating LLMs like humans is at our own peril.