scholarly journals Mathematically aggregating experts’ predictions of possible futures

PLoS ONE ◽  
2021 ◽  
Vol 16 (9) ◽  
pp. e0256919
Author(s):  
A. M. Hanea ◽  
D. P. Wilkinson ◽  
M. McBride ◽  
A. Lyon ◽  
D. van Ravenzwaaij ◽  
...  

Structured protocols offer a transparent and systematic way to elicit and combine/aggregate, probabilistic predictions from multiple experts. These judgements can be aggregated behaviourally or mathematically to derive a final group prediction. Mathematical rules (e.g., weighted linear combinations of judgments) provide an objective approach to aggregation. The quality of this aggregation can be defined in terms of accuracy, calibration and informativeness. These measures can be used to compare different aggregation approaches and help decide on which aggregation produces the “best” final prediction. When experts’ performance can be scored on similar questions ahead of time, these scores can be translated into performance-based weights, and a performance-based weighted aggregation can then be used. When this is not possible though, several other aggregation methods, informed by measurable proxies for good performance, can be formulated and compared. Here, we develop a suite of aggregation methods, informed by previous experience and the available literature. We differentially weight our experts’ estimates by measures of reasoning, engagement, openness to changing their mind, informativeness, prior knowledge, and extremity, asymmetry or granularity of estimates. Next, we investigate the relative performance of these aggregation methods using three datasets. The main goal of this research is to explore how measures of knowledge and behaviour of individuals can be leveraged to produce a better performing combined group judgment. Although the accuracy, calibration, and informativeness of the majority of methods are very similar, a couple of the aggregation methods consistently distinguish themselves as among the best or worst. Moreover, the majority of methods outperform the usual benchmarks provided by the simple average or the median of estimates.

2021 ◽  
Author(s):  
Anca Hanea ◽  
David Peter Wilkinson ◽  
Marissa McBride ◽  
Aidan Lyon ◽  
Don van Ravenzwaaij ◽  
...  

Experts are often asked to represent their uncertainty as a subjective probability. Structured protocols offer a transparent and systematic way to elicit and combine probability judgements from multiple experts. As part of this process, experts are asked to individually estimate a probability (e.g., of a future event) which needs to be combined/aggregated into a final group prediction. The experts' judgements can be aggregated behaviourally (by striving for consensus), or mathematically (by using a mathematical rule to combine individual estimates). Mathematical rules (e.g., weighted linear combinations of judgments) provide an objective approach to aggregation. However, the choice of a rule is not straightforward, and the aggregated group probability judgement's quality depends on it. The quality of an aggregation can be defined in terms of accuracy, calibration and informativeness. These measures can be used to compare different aggregation approaches and help decide on which aggregation produces the "best" final prediction.In the ideal case, individual experts' performance (as probability assessors) is scored, these scores are translated into performance-based weights, and a performance-based weighted aggregation is used. When this is not possible though, several other aggregation methods, informed by measurable proxies for good performance, can be formulated and compared. We use several data sets to investigate the relative performance of multiple aggregation methods informed by previous experience and the available literature. Even though the accuracy, calibration, and informativeness of the majority of methods are very similar, two of the aggregation methods distinguish themselves as the best and worst.


Author(s):  
Victoria Chen

The purpose of this study is to examine whether Multimedia learning theory (Mayer, 1997; Schnotz & Kürschner, 2007) holds true when images are the primary source of information and text information is secondary. I will test how temporal arrangement of audio and image presentations affects quality of learning in this situation. I hypothesize that when audio is played before or after the image participants will require increased cognitive processing to mentally integrate the two sources of information resulting in deeper learning and transfer of learning. On the other hand when audio is played while the image is shown, I hypothesize that participants with high prior knowledge of the subject will score lower than participants with low prior knowledge, because prior knowledge will interfere with knowledge from the two sources causing a redundancy effect. This experiment will lead to greater understanding of multimedia teaching and learning in classrooms as well as how it affects deeper learning.


2013 ◽  
Vol 43 (2) ◽  
pp. 184-196 ◽  
Author(s):  
Ingrid Hemmer ◽  
Michael Hemmer ◽  
Eva Neidhardt ◽  
Gabriele Obermaier ◽  
Rainer Uphues ◽  
...  

2005 ◽  
Vol 14 (01) ◽  
pp. 45-71 ◽  
Author(s):  
GHASSAN BEYDOUN ◽  
ACHIM HOFFMANN ◽  
JESUALDO TOMÁS FERNÁNDEZ BREIS ◽  
RODRIGO MARTÍNEZ BÉJAR ◽  
RAFAEL VALENCIA-GARCIA ◽  
...  

In any modelling activity, a framework to determine the maturity of a developed model before its use is highly advantageous. Such a framework would save modellers expensive time in many areas of information systems. It would also lower the risk of users relying on an incomplete or inaccurate model. In this paper, we develop a framework which uses internal inconsistencies as a quantitative indicator for estimating the completeness and correctness of a model as it is cooperatively evolved. Whilst internal inconsistencies are due to bad fit between different parts of a model, we argue that they are also correlated with how the evolved model fits with the "world". This argument underpins our framework to evaluate integrated models. Contributions of this paper are three folds: firstly, it presents a theoretically grounded framework for integrating models. We extend an existing incremental modelling framework, NRDR, which represents multiple hierarchical restricted domains (MHRD), with automatic concept integration to allow NRDR to deal with multiple experts. Secondly, we couple this integration framework with a theoretically grounded monitoring process to assess the quality of the cooperatively developed model. Thirdly, we illustrate an initial empirical study of our evaluation and integration framework in a computer hardware administration domain. We capture and integrate computer hardware models from several experts and we use our modelling evaluation framework to evaluate the resultant cooperative model.


1977 ◽  
Vol 84 (1) ◽  
pp. 158-172 ◽  
Author(s):  
Hillel J. Einhorn ◽  
Robin M. Hogarth ◽  
Eric Klempner
Keyword(s):  

2014 ◽  
Vol 50 (64) ◽  
pp. 8947-8950 ◽  
Author(s):  
M. Mayzel ◽  
K. Kazimierczuk ◽  
V. Yu. Orekhov

Improved quality of NMR spectra reconstruction from non-uniformly sampled data by harnessing the causality principle and prior knowledge about the signal phase.


2021 ◽  
Author(s):  
◽  
Margaret Franken

<p>Research on second language teaching and learning has to date focused primarily on the major skills of reading, writing, listening, and speaking but has treated them as relatively separate areas of investigation. By contrast this research investigates the role of one skill, speaking, in the performance of another, writing. The study investigates the effect of an instructional sequence that aimed to prepare upper high school students (Form 6/Grade 12) to write better argument essays. The sequence was experienced by the students in two ways. One way was for students to engage in talk with a peer before and during writing. The other way was for students to work in a solitary way. Qualitative data analysis compared the writing scores gained by students on two sets of variables: one to indicate the quality of text in general terms (Hamp-Lyons, 1986) and the other to indicate quality of text in terms of specific features of argument: claims, elaboration of claims, grounds and elaboration of grounds (Toulmin, 1958; Toulmin, Rieke and Janki, 1984). The results of the qualitative analysis indicate that opportunity to work with a peer before and during writing had a limited and specific effect on the texts that students wrote. Positive effects for opportunity to talk were seen in the quantity of grounds-related material, but only when students wrote texts that appeared to require more content and domain-specific knowledge (Alexander, Schallert and Hare, 1991). It appeared that talk could operate to help students access relevant prior knowledge (Alexander, Schallert and. Hare, 1991) to support the claims made in their argument texts. Working in a solitary way resulted in significantly better mean scores for linguistic accuracy and complexity. This finding is not consistent with claims made in the output hypothesis (Swain, 1985; Swain, 1995; Swain and Lapkin, 1995). One explanation is that students working together may not have been 'pushed' to consider aspects of language form in attempts to communicate meaning. Another is that this did occur but consideration of form did not transfer to subsequent writing. Another variable that appeared to influence writing quality was the level of resourcing provided for the writing task. Access to textual resources (input in the form of cohesive and linear text) appeared significantly to affect all three of the general measures of text quality, suggesting that textual input is a valuable linguistic and rhetorical resource for writers. When students' texts were analysed specifically for frequency of features of argument, different effects were found for levels of resourcing. Claims and elaboration of claims were most affected by the semi-resourced form of input represented by fact sheets (lists of propositions). Students appeared to make use of input in the form of fact sheets for meeting claim-like requirements in their texts. This might have been because the fact sheets represented information in a way that required the least amount of transformation to be accessible and useful. Analysis of transcript data was carried out on three selected pairs of students to explore the nature of the talk which produced significant and positive results. The type of talk associated with the pair that showed the greatest scores was qualitatively different in terms of the amount and topic of substantive talk and the frequency of responses to initiations. The talk also operated to push each participant, particularly the weaker of the two, to respond, explain and elaborate. The fact that the weaker student in the most productive pair made use of what he articulated suggested that, for him, the talk appeared to set the discourse parameters of the writing task. In addition, the results of the study pointed to the fact that speaking with a partner, particularly a more expert partner, before and during writing can bring positive effects particularly for drawing on relevant prior knowledge thereby enhancing content and domain-specific knowledge. A proficient and interactionally expert partner can promote discussion of relevant prior knowledge useful for supporting claims made in argument texts. The analysis of transcript data indicates that few students show interactional proficiency and that this may prove a worthwhile focus for pedagogy. The present study supports the line of research in collaborative learning (Cohen, 1994) as it has explored the conditions under which positive effects on writing are likely to occur. Research may profitably continue to explore the features of successful interaction and the conditions that successful interaction creates, particularly as it enables better writing. Not only are conditions worthy of further research, so too are effects, as they are likely to operate on different aspects of writing and in different genres. Constraints operating particularly in the area of argument need to continue to be explored empirically. The present study has concluded with the belief that there is still much to know in the relationship between speaking and writing. For this reason, teachers may do well to pay careful consideration to the way in which pair and group tasks are managed in the classroom. This entails the provision of guidance and support for the participants so that purposeful interaction occurs.</p>


2021 ◽  
Vol 23 (3) ◽  
pp. 443-453
Author(s):  
Maysa Alshraideh ◽  
Shereen Ababneh ◽  
Elif Elcin Gunay ◽  
Omar Al-Araidah

The paper provides a multiple-experts Fuzzy-TOPSIS decision-making model for the selection among maintenance contractors based on the quality of tendering documents. The study introduces a set of selection criteria utilizing benefit and cost criteria from literature. The proposed model aggregates subjective linguistic assessments of multiple experts that express their opinions on the degree of importance of criteria and allows multiple decisionmakers to evaluate the compliance of contractors’ documents. For a case study, the model is applied to select among contractors tendering to maintain the heavy-duty cranes of an international steel company from literature. Several decision-making scenarios are investigated, and major changes in the final decision are observed. The changes in obtained results illustrate the need to better address uncertainties in rating and tendering an overqualified contractor at a higher cost.


2020 ◽  
Vol 2020 (28) ◽  
pp. 19-24
Author(s):  
Altynay Kadyrova ◽  
Vlado Kitanovski ◽  
Marius Pedersen

Quality assessment is performed through the use of variety of quality attributes. It is crucial to identify relevant attributes for quality assessment. We focus on 2.5D print quality assessment and its quality attributes. An experiment with observers showed the most frequently used attributes to judge quality of 2.5D prints with and without reference images. Colour, sharpness, elevation, lightness, and naturalness are the top five the most frequently used attributes for both with and without reference cases. We observed that content, previous experience and knowledge, and aesthetic appearance may impact quality judgement.


Author(s):  
Eduardo Bezerra ◽  
Geraldo Xexéo ◽  
Marta Mattoso

In this chapter, we consider the problem of constrained clustering of documents. We focus on documents that present some form of structural information, in which prior knowledge is provided. Such structured data can guide the algorithm to a better clustering model. We consider the existence of a particular form of information to be clustered: textual documents that present a logical structure represented in XML format. Based on this consideration, we present algorithms that take advantage of XML metadata (structural information), thus improving the quality of the generated clustering models. This chapter also addresses the problem of inconsistent constraints and defines algorithms that eliminate inconsistencies, also based on the existence of structural information associated to the XML document collection.


Sign in / Sign up

Export Citation Format

Share Document