Do low-confidence individuals decrease group judgments’ accuracy? Investigations in terms of the wisdom of crowds framework

In group judgments in a binary choice task, the judgments of individuals with low confidence (i.e., they feel that the judgment was not correct) may be regarded as unreliable. Previous studies have shown that aggregating individuals’ diverse judgments can lead to high accuracy in group judgments, a phenomenon known as the wisdom of crowds. Therefore, if low-confidence individuals make diverse judgments between individuals and the mean of accuracy of their judgments is above the chance level (.50), it is likely that they will not always decrease the accuracy of group judgments. To investigate this issue, the present study conducted behavioral experiments using binary choice inferential tasks, and computer simulations of group judgments by manipulating group sizes and individuals’ confidence levels. Results revealed that (I) judgment patterns were highly similar between individuals regardless of their confidence levels; (II) the low-confidence group could make judgments as accurate as the high-confidence group, as the group size increased; and (III) even if there were low-confidence individuals in a group, they generally did not inhibit group judgment accuracy. The results suggest the usefulness of low-confidence individuals’ judgments in a group and provide practical implications for real-world group judgments.

Do low-confidence individuals decrease group judgments’ accuracy? Investigations in terms of the wisdom of crowds framework

In group judgments in a binary choice task, the judgments of individuals with low confidence (i.e., they feel that the judgment was not correct) may be regarded as unreliable. Previous studies have shown that aggregating individuals’ diverse judgments can lead to high accuracy in group judgments, a phenomenon known as the wisdom of crowds. Therefore, if low-confidence individuals make diverse judgments between individuals and the mean of accuracy of their judgments is above the chance level (.50), it is likely that they will not always decrease the accuracy of group judgments. To investigate this issue, the present study conducted behavioral experiments using binary choice inferential tasks, and computer simulations of group judgments by manipulating group sizes and individuals’ confidence levels. Results revealed that (I) judgment patterns were highly similar between individuals regardless of their confidence levels; (II) the low-confidence group could make judgments as accurate as the high-confidence group, as the group size increased; and (III) even if there were low-confidence individuals in a group, they generally did not inhibit group judgment accuracy. The results suggest the usefulness of low-confidence individuals’ judgments in a group and provide practical implications for real-world group judgments.

Mathematically aggregating experts’ predictions of possible futures

Structured protocols offer a transparent and systematic way to elicit and combine/aggregate, probabilistic predictions from multiple experts. These judgements can be aggregated behaviourally or mathematically to derive a final group prediction. Mathematical rules (e.g., weighted linear combinations of judgments) provide an objective approach to aggregation. The quality of this aggregation can be defined in terms of accuracy, calibration and informativeness. These measures can be used to compare different aggregation approaches and help decide on which aggregation produces the “best” final prediction. When experts’ performance can be scored on similar questions ahead of time, these scores can be translated into performance-based weights, and a performance-based weighted aggregation can then be used. When this is not possible though, several other aggregation methods, informed by measurable proxies for good performance, can be formulated and compared. Here, we develop a suite of aggregation methods, informed by previous experience and the available literature. We differentially weight our experts’ estimates by measures of reasoning, engagement, openness to changing their mind, informativeness, prior knowledge, and extremity, asymmetry or granularity of estimates. Next, we investigate the relative performance of these aggregation methods using three datasets. The main goal of this research is to explore how measures of knowledge and behaviour of individuals can be leveraged to produce a better performing combined group judgment. Although the accuracy, calibration, and informativeness of the majority of methods are very similar, a couple of the aggregation methods consistently distinguish themselves as among the best or worst. Moreover, the majority of methods outperform the usual benchmarks provided by the simple average or the median of estimates.

Group Judgment and Decision Making in Auditing: Research in the Time of COVID-19 and Beyond

The COVID-19 pandemic has fundamentally changed the ways auditors work and interact with team members and others in the financial reporting process. In particular, there has been a move away from face-to-face interactions to the use of virtual teams, with strong indications many of these changes will remain post-pandemic. We examine the impacts of the pandemic on group judgment and decision making (JDM) research in auditing by reviewing research on auditor interactions with respect to the review process (including coaching), fraud brainstorming, consultations within audit firms, and parties outside the audit firm such as client management and the audit committee. Through the pandemic lens and for each auditor interaction, we consider new research questions for audit JDM researchers to investigate and new ways of addressing existing research questions given these fundamental changes. We also identify potential impacts on research methods used to address these questions during the pandemic and beyond.

Opinion Aggregation and Individual Expertise

Group judgments are often influenced by their members’ individual expertise. It is less clear, though, how individual expertise should affect the group judgments. This chapter surveys a wide range of models of opinion aggregation and group judgment: models where all group members have the same impact on the group judgment, models that take into account differences in individual accuracy, and models where group members revise their beliefs as a function of their mutual respect. The scope of these models covers the aggregation of propositional attitudes, probability functions, and numerical estimates. By comparing these different kinds of models and contrasting them with findings in psychology, management science, and the expert judgment literature, the chapter provides a better understanding of the role of expertise in group agency, both from a theoretical and from an empirical perspective.

The Editorial Meeting at a Little Magazine: An Ethnography of Group Judgment

Judgment and distinction have been topics of persistent interest for cultural sociologists. Recent theory has particularly emphasized social interaction and cognition as key sites for understanding judgment, and a number of studies examine gatekeeping practices as a means of understanding the interactive and perceptual determinants of judgment. This article builds upon previous work by presenting the results of an eighteen-month ethnography at a long-running little magazine based in a large American city. In addition to providing an empirical description of an important but understudied domain of cultural production, this article has two findings of theoretical interest. First, without strong external constraints on processes of group evaluation, editors’ judgments became markedly negative, and their deliberations were often inconclusive. Second, negative and positive evaluations were not symmetric, but were produced by two differing sets of evaluative practices. These findings are broadly consistent with developing field theoretic descriptions of social life, but also raise many empirical and theoretical questions.Citation: Merriman, Ben. 2017. “The Editorial Meeting at a Little Magazine: An Ethnography of Group Judgment.” Journal of Contemporary Ethnography 46(4): 440-463.

Multi-Criteria Decision Making Approach Regarding the Choice of University for Postgraduate Studies

<p>The problem set in this paper regards how to make the choice of University for postgraduate studies. The multi-criteria decision making approach is suggested to be used for solving this problem. The idea is to decompose the problem into the following elements: goal-choice of University for postgraduate studies; criteria that contribute to achieving the goal and that can be of quantitative or qualitative nature; and alternatives-Universities that the choice will be made from. For such problem a hierarchical model can be built, and it can be solved by using the analytic hierarchy process (AHP). The objective of this research, which will be conducted in the form of a questionnaire among the best fourth-year students from Ss. Cyril and Methodius University in Skopje, Faculty of Economics-Skopje, is to determine how many of them will continue their education in postgraduate studies, and where, as well as to identify the criteria that are important in their opinion when choosing a University. Once the participants’ answers are received, two groups of criteria that are important regarding the choice of University (in Macedonia and abroad) will be created, and thus two groups of participants will be formed. The choice of criteria will be made according to the arithmetic mean, and if the number is high then factor analysis will be used for their reduction. Afterwards, the participants will be introduced to the AHP method and for combining the individual judgments in group judgment; the geometric mean will be used. The University that is the best choice for each of the participants will come as a result of the ranking of the overall priorities of the alternatives.</p>

The Editorial Meeting at a Little Magazine: An Ethnography of Group Judgment

Judgment and distinction have been topics of persistent interest for cultural sociologists. Recent theory has particularly emphasized social interaction and cognition as key sites for understanding judgment, and a number of studies examine gatekeeping practices as a means of understanding the interactive and perceptual determinants of judgment. This article builds upon previous work by presenting the results of an eighteen-month ethnography at a long-running little magazine based in a large American city. In addition to providing an empirical description of an important but understudied domain of cultural production, this article has two findings of theoretical interest. First, without strong external constraints on processes of group evaluation, editors’ judgments became markedly negative, and their deliberations were often inconclusive. Second, negative and positive evaluations were not symmetric, but were produced by two differing sets of evaluative practices. These findings are broadly consistent with developing field theoretic descriptions of social life, but also raise many empirical and theoretical questions.