Can AI’s random preferences sway experts? Keep the reasoning behind your decisions
A collected report describes an experiment with 50 causal-inference experts, with adoption of an AI agent’s random preference reaching 86% on some analytical decisions. Keep the study’s scope, advice and evidence distinct; the maximum is not a result for everyone.
What to consider
Use AI for options and objections while retaining an independent basis for the final choice. The collected report attributes an experiment involving 50 causal-inference experts to Anthropic. It reports that preferences randomly assigned to an AI agent influenced analytical decisions, with adoption reaching 86% on one decision. That warrants examining how advice changes judgment; it is neither a model error rate nor a population-wide influence rate. Record criteria, evidence and reasons for revising a decision, not just the final recommendation.
Keep the maximum adoption figure within its experimental scope
The summary refers to four analytical decisions. When citing the maximum, retain the participant count, specialty and task scope rather than describing an 86% chance that any AI will mislead any expert. Adoption of a random preference, answer correctness and participant satisfaction are separate questions.
The summary also says participants with prior views could change them after contrary advice, with no reliable protection associated with experience and other characteristics. Writing an initial view creates a comparison record, not a proven defense against influence. The full report and study design were not reviewed here, so no further statistical interpretation is added.
Turn a recommendation into reasons you can question
State the objective and non-negotiable constraints before asking for arguments for and against options. For each reason that could change the choice, identify whether it rests on available material, an unconfirmed assumption or a forecast. A complete-sounding explanation is not necessarily a complete evidential case.
Before reading the AI’s advice, write down the criteria used to compare two proposals. If the choice changes, note whether new material changed the assessment or the explanation simply sounded more persuasive. This supports later review without establishing that influence has been eliminated.
Keep a responsible person accountable for the tradeoff
For a reversible decision, define a limited trial and conditions for review. For choices affecting a team, a budget or a longer commitment, have a responsible person explain the tradeoff and missing information. AI can organize the discussion record without making its recommendation the team’s accountability mechanism.
These workflow suggestions are based on the collected report, not an experiment or a measurement of team decision quality. The reported figures illustrate the question raised; they do not rank model safety, identify a more gullible profession or quantify the benefit of a proposed process.
Source links
The links below were supplied with the collected records. This article offers independent analysis of their summaries; we have not individually checked the original sources or tested the products.