Max Planck research: Reasoning makes LLM errors more consistent

A Max Planck study finds that enabling reasoning increases consistency across different large language models (LLMs), even making their mistakes more aligned. The authors tested nine Qwen and GPT models on three prediction tasks based on US Census data. After enabling reasoning, the proportion of cases where all nine models answered incorrectly rose from 2.0 to 2.4 percentage points. For groups with the longest reasoning chains, corrected consistency dropped to approximately 0.31 to 0.39. The authors suggest checking whether any model in a set can provide the correct answer when decisions involving humans are made, because higher consistency may also indicate more shared errors.