scholarly journals Doubly Robust Crowdsourcing

2022 ◽  
Vol 73 ◽  
pp. 209-229
Author(s):  
Chong Liu ◽  
Yu-Xiang Wang
Keyword(s):  
Large Scale ◽  
Amazon Mechanical Turk ◽  
Test Time ◽  
Fair Price ◽  
Data Points ◽  
Label Aggregation ◽  
Doubly Robust Estimation ◽  
Sheer Size ◽  
Statistical Estimation Problem ◽  
Doubly Robust

Large-scale labeled dataset is the indispensable fuel that ignites the AI revolution as we see today. Most such datasets are constructed using crowdsourcing services such as Amazon Mechanical Turk which provides noisy labels from non-experts at a fair price. The sheer size of such datasets mandates that it is only feasible to collect a few labels per data point. We formulate the problem of test-time label aggregation as a statistical estimation problem of inferring the expected voting score. By imitating workers with supervised learners and using them in a doubly robust estimation framework, we prove that the variance of estimation can be substantially reduced, even if the learner is a poor approximation. Synthetic and real-world experiments show that by combining the doubly robust approach with adaptive worker/item selection rules, we often need much lower label cost to achieve nearly the same accuracy as in the ideal world where all workers label all data points.

scholarly journals Farmland lease, high-rent threat and contract instability: evidence from China

2021 ◽  
Vol ahead-of-print (ahead-of-print) ◽  
Author(s):  
Wenjue Zhu ◽  
Krishna P. Paudel ◽  
Sean Inoue ◽  
Biliang Luo
Keyword(s):  
Theoretical Model ◽  
Large Scale ◽  
Estimation Method ◽  
Rent Control ◽  
Content Type ◽  
Land Rental ◽  
High Rent ◽  
Doubly Robust Estimation ◽  
The Stability ◽  
Doubly Robust

PurposeThe purpose is to understand why contract instability occurs when small landowners lease their land to large landholders.Design/methodology/approachThe authors develop a contract theoretical model to understand the stability problem in the farmland lease contract in China, where most landowners are small landholders.FindingsResults from the doubly robust estimation method used on randomly selected interview data from 552 households in nine provinces of China indicate that contract instability can arise endogenously when large landholders sign a contract. The authors conclude that a suitable rent control regime or contract enforcement may be necessary to promote a large-scale farmland transfer in China.Originality/valueThe authors develop a contract theoretical model and apply it to the land rental market in China. Data used are original and collected from farmers located in nine provinces of China.

scholarly journals ON GENERATING DIGITAL ELEVATION MODELS FROM LIDAR DATA – RESOLUTION VERSUS ACCURACY AND TOPOGRAPHIC WETNESS INDEX INDICES IN NORTHERN PEATLANDS

2012 ◽  
Vol 38 (2) ◽  
pp. 57-69 ◽  
Author(s):  
Abdulghani Hasan ◽  
Petter Pilesjö ◽  
Andreas Persson
Keyword(s):  
Large Scale ◽  
Drainage Area ◽  
Data Sets ◽  
Topographic Wetness Index ◽  
Absolute Deviation ◽  
Digital Elevation ◽  
Elevation Data ◽  
Scale Modelling ◽  
Data Points ◽  
Emission Modelling

Global change and GHG emission modelling are dependent on accurate wetness estimations for predictions of e.g. methane emissions. This study aims to quantify how the slope, drainage area and the TWI vary with the resolution of DEMs for a flat peatland area. Six DEMs with spatial resolutions from 0.5 to 90 m were interpolated with four different search radiuses. The relationship between accuracy of the DEM and the slope was tested. The LiDAR elevation data was divided into two data sets. The number of data points facilitated an evaluation dataset with data points not more than 10 mm away from the cell centre points in the interpolation dataset. The DEM was evaluated using a quantile-quantile test and the normalized median absolute deviation. It showed independence of the resolution when using the same search radius. The accuracy of the estimated elevation for different slopes was tested using the 0.5 meter DEM and it showed a higher deviation from evaluation data for steep areas. The slope estimations between resolutions showed differences with values that exceeded 50%. Drainage areas were tested for three resolutions, with coinciding evaluation points. The model ability to generate drainage area at each resolution was tested by pair wise comparison of three data subsets and showed differences of more than 50% in 25% of the evaluated points. The results show that consideration of DEM resolution is a necessity for the use of slope, drainage area and TWI data in large scale modelling.

scholarly journals Adjustment for Missing Data in Complex Surveys Using Doubly Robust Estimation

Epidemiology ◽  
2010 ◽  
Vol 21 (6) ◽  
pp. 863-871 ◽  
Author(s):  
Kathleen E. Wirth ◽  
Eric J. Tchetgen Tchetgen ◽  
Megan Murray
Keyword(s):  
Missing Data ◽  
Robust Estimation ◽  
Complex Surveys ◽  
Doubly Robust Estimation ◽  
Doubly Robust

Assessing Triangulation Across Methodologies, Methods, and Stakeholder Groups: The Joys, Woes, and Politics of Interpreting Convergent and Divergent Data

2018 ◽  
Vol 41 (1) ◽  
pp. 125-144 ◽  
Author(s):  
Rebecca Campbell ◽  
Rachael Goodman-Williams ◽  
Hannah Feeney ◽  
Giannina Fehler-Cabral
Keyword(s):  
Large Scale ◽  
Contextual Information ◽  
Policy Makers ◽  
Specific Data ◽  
Qualitative And Quantitative ◽  
Stakeholder Groups ◽  
Data Points ◽  
Evaluation Project ◽  
Collection Methods

The purpose of this study was to develop triangulation coding methods for a large-scale action research and evaluation project and to examine how practitioners and policy makers interpreted both convergent and divergent data. We created a color-coded system that evaluated the extent of triangulation across methodologies (qualitative and quantitative), data collection methods (observations, interviews, and archival records), and stakeholder groups (five distinct disciplines/organizations). Triangulation was assessed for both specific data points (e.g., a piece of historical/contextual information or qualitative theme) and substantive findings that emanated from further analysis of those data points (e.g., a statistical model or a mechanistic qualitative assertion that links themes). We present five case study examples that explore the complexities of interpreting triangulation data and determining whether data are deemed credible and actionable if not convergent.

scholarly journals Towards Accurate and Efficient Chinese Part-of-Speech Tagging

2016 ◽  
Vol 42 (3) ◽  
pp. 391-419 ◽  
Author(s):  
Weiwei Sun ◽  
Xiaojun Wan
Keyword(s):  
Hybrid Systems ◽  
Language Processing ◽  
Large Scale ◽  
Unlabeled Data ◽  
Training Data ◽  
Test Time ◽  
System Combination ◽  
Pos Tagging ◽  
Hybrid Approaches ◽  
Lexical Relations

From the perspective of structural linguistics, we explore paradigmatic and syntagmatic lexical relations for Chinese POS tagging, an important and challenging task for Chinese language processing. Paradigmatic lexical relations are explicitly captured by word clustering on large-scale unlabeled data and are used to design new features to enhance a discriminative tagger. Syntagmatic lexical relations are implicitly captured by syntactic parsing in the constituency formalism, and are utilized via system combination. Experiments on the Penn Chinese Treebank demonstrate the importance of both paradigmatic and syntagmatic relations. Our linguistically motivated, hybrid approaches yield a relative error reduction of 18% in total over state-of-the-art baselines. Despite the effectiveness to boost accuracy, computationally expensive parsers make hybrid systems inappropriate for many realistic NLP applications. In this article, we are also concerned with improving tagging efficiency at test time. In particular, we explore unlabeled data to transfer the predictive power of hybrid models to simple sequence models. Specifically, hybrid systems are utilized to create large-scale pseudo training data for cheap models. Experimental results illustrate that the re-compiled models not only achieve high accuracy with respect to per token classification, but also serve as a front-end to a parser well.

scholarly journals Testing the Waters: Behavior across Participant Pools

2021 ◽  
Vol 111 (2) ◽  
pp. 687-719
Author(s):  
Erik Snowberg ◽  
Leeat Yariv
Keyword(s):  
External Validity ◽  
Large Scale ◽  
Student Population ◽  
University Student ◽  
Mechanical Turk ◽  
Amazon Mechanical Turk ◽  
Lab Experiments ◽  
The Us ◽  
Small Set ◽  
Observer Effects

We leverage a large-scale incentivized survey eliciting behaviors from (almost) an entire undergraduate university student population, a representative sample of the US population, and Amazon Mechanical Turk (MTurk) to address concerns about the external validity of experiments with student participants. Behavior in the student population offers bounds on behaviors in other populations, and correlations between behaviors are similar across samples. Furthermore, non-student samples exhibit higher levels of noise. Adding historical lab participation data, we find a small set of attributes over which lab participants differ from non-lab participants. An additional set of lab experiments shows no evidence of observer effects. (JEL C83, D90, D91)

scholarly journals Learning to be selfish? A large-scale longitudinal analysis of Dictator games played on Amazon Mechanical Turk

2019 ◽  
Author(s):  
Antonio Alonso Arechar ◽  
David Gertler Rand
Keyword(s):  
Longitudinal Analysis ◽  
Large Scale ◽  
Dictator Game ◽  
Mechanical Turk ◽  
Amazon Mechanical Turk ◽  
Social Setting ◽  
Negative Effects ◽  
Dictator Games

We investigate whether experience playing the Dictator Game (DG) affects prosociality by aggregating data from 37 experiments run on Amazon Mechanical Turk over a six-year period. While prior evidence has shown a correlation between experience on Amazon Mechanical Turk and selfishness, it is unclear to what extent this is the result of selection versus learning. Examining a total of 27,266 decisions made by 17,791 unique individuals, our data shows evidence of significant negative effects of both selection and learning. First, people who participated in a greater total number of our experiments were more selfish, even in their first game – indicating that people who are more likely to select into our experiments are more selfish. Second, a given individual tends to transfer less money over successive experiments – indicating that experience with the DG leads to greater selfishness. These results provide clear evidence of learning even in this non-strategic social setting.

Large-Scale Sensor Network Analysis

2013 ◽  
pp. 314-347 ◽  
Author(s):  
Joaquin Vanschoren ◽  
Ugo Vespier ◽  
Shengfa Miao ◽  
Marvin Meeng ◽  
Ricardo Cachucho ◽  
...  
Keyword(s):  
Big Data ◽  
Data Analysis ◽  
Large Scale ◽  
Vital Signs ◽  
Sensor Data ◽  
Atmospheric Conditions ◽  
Big Data Applications ◽  
The World ◽  
Sheer Size ◽  
Effective Use

Sensors are increasingly being used to monitor the world around us. They measure movements of structures such as bridges, windmills, and plane wings, human’s vital signs, atmospheric conditions, and fluctuations in power and water networks. In many cases, this results in large networks with different types of sensors, generating impressive amounts of data. As the volume and complexity of data increases, their effective use becomes more challenging, and novel solutions are needed both on a technical as well as a scientific level. Founded on several real-world applications, this chapter discusses the challenges involved in large-scale sensor data analysis and describes practical solutions to address them. Due to the sheer size of the data and the large amount of computation involved, these are clearly “Big Data” applications.

scholarly journals Doubly Robust Estimation of Causal Effect

2020 ◽  
Vol 13 (1) ◽  
Author(s):  
Xiaochun Li ◽  
Changyu Shen
Keyword(s):  
Propensity Score ◽  
Causal Effect ◽  
Model Specification ◽  
Robust Estimator ◽  
Data Set ◽  
Multiple Regressions ◽  
Doubly Robust Estimation ◽  
Doubly Robust ◽  
Confounding Adjustment ◽  
The Relationship

Propensity score–based methods or multiple regressions of the outcome are often used for confounding adjustment in analysis of observational studies. In either approach, a model is needed: A model describing the relationship between the treatment assignment and covariates in the propensity score–based method or a model for the outcome and covariates in the multiple regressions. The 2 models are usually unknown to the investigators and must be estimated. The correct model specification, therefore, is essential for the validity of the final causal estimate. We describe in this article a doubly robust estimator which combines both models propitiously to offer analysts 2 chances for obtaining a valid causal estimate and demonstrate its use through a data set from the Lindner Center Study.

Sign in / Sign up

Export Citation Format

Share Document