Expert ratings of computer capabilities to answer PIAAC literacy questions, comparing average using Maybe=50% and 3-expert minimum, by level of question difficulty

Abstract:Bayesian inferencing as a machine learning technique was evaluated for identifying pre-crash activity and crash type from accident narratives describing 3,686 motor vehicle crashes. It was hypothesized that a Bayesian model could learn from a computer search for 63 keywords related to accident categories. Learning was described in terms of the ability to accurately classify previously unclassifiable narratives not containing the original keywords. When narratives contained keywords, the results obtained using both the Bayesian model and keyword search corresponded closely to expert ratings (P(detection)≥0.9, and P(false positive)≤0.05). For narratives not containing keywords, when the threshold used by the Bayesian model was varied between p>0.5 and p>0.9, the overall probability of detecting a category assigned by the expert varied between 67% and 12%. False positives correspondingly varied between 32% and 3%. These latter results demonstrated that the Bayesian system learned from the results of the keyword searches.

Download Full-text

A Comparison of Expert Ratings and Marker-Less Hand Tracking Along OSATS-Derived Motion Scales

IEEE Transactions on Human-Machine Systems ◽

10.1109/thms.2020.3035763 ◽

2020 ◽

pp. 1-10

Author(s):

David P. Azari ◽

Brady L. Miller ◽

Brian V. Le ◽

Jacob A. Greenberg ◽

Reginald C. Bruskewitz ◽

...

Keyword(s):

Hand Tracking ◽

Expert Ratings

Download Full-text

Erring Experts? A Critique of Wine Ratings as Hedonic Scaling

Journal of Wine Economics ◽

10.1017/jwe.2020.42 ◽

2020 ◽

Vol 15 (4) ◽

pp. 386-393

Author(s):

Denton Marks

Keyword(s):

External Validity ◽

Rating Scales ◽

Food Science ◽

Sensory Sensitivity ◽

Behavioral Sciences ◽

Transaction Prices ◽

Magnitude Scales ◽

Expert Ratings ◽

Jel Classifications

AbstractConsumers use expert ratings to help choose wine, and economists find correlations between ratings and transaction prices. Rating scales resemble hedonic scales in the behavioral sciences, which suffer from an “intersubjectivity” problem. Taste is a private sensation; people taste differently (an external validity problem), so ratings are often unreliable hedonic markers of enjoyment. But why? Hedonic measurements from food science (“general Labeled Magnitude Scales”) attempt to adjust for differences in perceived sensory sensitivity and offer clues. Resulting insights illustrate wine ratings’ shortcomings as reliable guides to enjoyment. (JEL Classifications: C14, D12, D91, L15, L66)

Download Full-text