Algorithm-based advice taking and clinical judgement: impact of advice distance and algorithm information

Evidence-based algorithms can improve both lay and professional judgements and decisions, yet they remain underutilised. Research on advice taking established that humans tend to discount advice – especially when it contradicts their own judgement (“egocentric advice discounting”) – but this can be mitigated by knowledge about the advisor’s past performance. Advice discounting has typically been investigated using tasks with outcomes of low importance (e.g., general knowledge questions), and students as participants. Using the judge-advisor framework, we tested whether the principles of advice discounting apply in the clinical domain. We used realistic patient scenarios, algorithmic advice from a validated cancer risk calculator, and General Practitioners (GPs) as participants. GPs could update their risk estimates after receiving algorithmic advice. Half of them received information about the algorithm’s derivation, validation, and accuracy. We measured Weight of Advice and found that, on average, GPs weighed their estimates and the algorithm equally – but not always: they retained their initial estimates 29% of the time, and fully updated them 27% of the time. Updating did not depend on whether GPs were informed about the algorithm. We found a weak negative quadratic relationship between estimate updating and advice distance: although GPs integrate algorithmic advice on average, they may somewhat discount it, if it is very different from their own estimate. These results present a more complex picture than simple egocentric discounting of advice. They cast a more optimistic view of advice taking, where experts weigh algorithmic advice and their own judgement equally and move towards the advice even when it contradicts their own initial estimates.

The Bad Thing About Good Advice: Understanding When and How Advice Exacerbates Overconfidence

Much research on advice taking examines how people revise point estimates given input from others. This work has established that people often egocentrically discount advice. If they were to place more weight on advice, their point estimates would be more accurate. Yet the focus on point estimates and accuracy has resulted in a narrow conception of what it means to heed advice. We distinguish between revisions of point estimates and revisions of attendant probability distributions. Point estimates represent a single best guess; distributions represent the probabilities that people assign to all possible answers. A more complete picture of advice taking is provided by considering revisions of distributions, which reflect changes in both confidence and best guesses. We capture this using a new measure of advice utilization: the influence of advice. We observe that, when input from a high-quality advisor largely agrees with a person’s initial opinion, it engenders little change in one’s point estimate and, hence, little change in accuracy yet significantly increases confidence. This pattern suggests more advice taking than generally suspected. However, it is not necessarily beneficial. Because people are typically overconfident to begin with, receiving advice that agrees with their initial opinion can exacerbate overconfidence. In several experiments, we manipulate advisor quality and measure the extent to which advice agrees with a person’s initial opinion. The results allow us to pinpoint circumstances in which heeding advice is beneficial, improving accuracy or reducing overconfidence, as well as circumstances in which it is harmful, hurting accuracy or exacerbating overconfidence. This paper was accepted by Yuval Rottenstreich, judgment and decision making.

The Effect of Consumer-Activated Mind-Set and Product Involvement on the Compliance With Recommender System Advice

Whereas the research gauging the effectiveness of e-commerce recommender systems (RS) has depended on their design factors, recent work proposes a key role for consumer’s psychological factors. Involvement should reduce the compliance with RS advice because a consumer highly involved with the product perceives high choice risk and assigns low value to the advice. However, a consumer’s activated mind-set captured by implicit theory (fixed vs. growth mind-set) should also shape compliance. It is hypothesized that the two factors interact to jointly mitigate advice taking. Specifically, consumers whose fixed mind-set is primed comply with the RS advice less often when involvement is high. This and other anticipated effects (i.e., consumer’s importance of social approval, positive affect, and need for cognition) on advice compliance are tested in an experiment on 251 Canadian adults. In the experiment, compliance occurred when the participant follows the RS advice, and product involvement was initially measured. The results show that priming a fixed mind-set, which orients shoppers toward a performance goal, motivates them to comply with the RS advice when involvement is low. Priming a growth mind-set, which orients shoppers toward a learning goal, nullifies such effect. Positive affect and the importance of social approval had no significant impact on advice taking. Therefore, the effect of involvement on RS effectiveness is contingent on the shopper’s accessible mind-set.