Clarifying Agreement Calculations and Analysis for End-User Elicitation Studies

We clarify fundamental aspects of end-user elicitation, enabling such studies to be run and analyzed with confidence, correctness, and scientific rigor. To this end, our contributions are multifold. We introduce a formal model of end-user elicitation in HCI and identify three types of agreement analysis: expert , codebook , and computer . We show that agreement is a mathematical tolerance relation generating a tolerance space over the set of elicited proposals. We review current measures of agreement and show that all can be computed from an agreement graph . In response to recent criticisms, we show that chance agreement represents an issue solely for inter-rater reliability studies and not for end-user elicitation, where it is opposed by chance disagreement . We conduct extensive simulations of 16 statistical tests for agreement rates, and report Type I errors and power. Based on our findings, we provide recommendations for practitioners and introduce a five-level hierarchy for elicitation studies.

Reflections on Popular Culture and Philosophy

Contributions to the philosophical genre of popular culture and philosophy aim to popularize philosophical ideas with the help of references to the products of popular (mass) culture with TV series like The Simpsons, Hollywood blockbusters like The Matrix and Jurassic Park, or popular music groups like Metallica. While being commercially successful, books in this comparatively new genre are often criticized for lacking scientific rigor, providing a shallow cultural commentary, and having little didactic value to foster philosophical understanding. This paper discusses some of these methodological and didactic objections and seeks to encourage a constructive discussion of concerns with the genre. It shows how the genre similar to previous attempts to foster public understanding of philosophy and that it is a methodologically viable approach to reach a broad range of readers with diverse informational preferences and educational backgrounds. Considering what makes this approach to the popularization of philosophical thinking successful will shed light on some of the criteria for popularization of philosophy in general.

Epistemic Oppression, Construct Validity, & Scientific Rigor: Commentary on ‘Bias, Fairness, & Validity in Graduate Admissions: A Psychometric Perspective’ (Accepted Manuscript Version)

In this commentary, I highlight flaws in the article by Woo and colleagues (in press, this issue) that undermine its credibility and utility as rigorous science that contributes to the field by discussing: 1) the concept of epistemic oppression (e.g., Dotson, 2014) regarding the glaring exclusion of multiple germane bodies of research; and 2) the importance of including construct validity within a psychometric article regarding the GRE (ETS, 2019). I conclude with a plea to the authors that the matter of anti-Black violence and murder, for which they reference, is one to be taken respectfully, seriously, and somberly.

Improving Taxonomic Practices and Enhancing Its Extensibility—An Example from Araneology

Planetary extinction of biodiversity underscores the need for taxonomy. Here, we scrutinize spider taxonomy over the last decade (2008–2018), compiling 2083 published accounts of newly described species. We evaluated what type of data were used to delineate species, whether data were made freely available, whether an explicit species hypothesis was stated, what types of media were used, the sample sizes, and the degree to which species constructs were integrative. The findings we report reveal that taxonomy remains largely descriptive, not integrative, and provides no explicit conceptual framework. Less than 4% of accounts explicitly stated a species concept and over one-third of all new species described were based on 1–2 specimens or only one sex. Only ~5% of studies made data freely available, and only ~14% of all newly described species employed more than one line of evidence, with molecular data used in ~6% of the studies. These same trends have been discovered in other animal groups, and therefore we find it logical that taxonomists face an uphill challenge when justifying the scientific rigor of their field and securing the needed resources. To move taxonomy forward, we make recommendations that, if implemented, will enhance its rigor, repeatability, and scientific standards.

The Singularity Myth

Ray Kurzweil’s book "The Singularity Is Near" constitutes a most exhaustive compilation of “singularitarian” arguments and one of the most serious publications on the subject, but the work lacks scientific rigor. Kurzweil and the singularitarians are indulging in somesort of para-science, which differs from real science in matters of methodology and rigor. They tend to overlook rigorous scientific practices such as focusing on natural laws,giving precise definitions, verifying the data meticulously, and estimating theuncertainties. I list a number of scientific wrongdoings in this book. I try to rectify some of them in order to properly present my critique of the Singularity concept.

Self-criticisms toward a socially responsible science in the field of management

Management scholarship should be placed in a unique position to develop relevant scientific knowledge because business and management organizations are deeply involved in most global challenges. However, different critical voices have recently been raised in essays and editorials, and reports have questioned research in the management field, identifying multiple deficiencies that can limit the growth of a relatively young field. Based on an analysis of published criticisms of management research, we would like to shed light on the current state of management research and identify some limitations that should be considered and should guide the growth of this field of knowledge. This work offers guidance on the main problems of the discipline that should be addressed to encourage the transformation of management research to meet both scientific rigor and social relevance. The article ends with a discussion and a call to action for directing research toward the possibility and necessity of reinforcing “responsible research” in the management field. JEL CLASSIFICATION: M00, M10

Best Practices for Monitoring and Assessing the Ecological Response to River Restoration

Nature-based solutions are widely advocated for freshwater ecosystem conservation and restoration. As increasing amounts of river restoration are undertaken, the need to understand the ecological response to different measures and where measures are best applied becomes more pressing. It is essential that appraisal methods follow a sound scientific approach. Here, experienced restoration appraisal experts review current best practice and academic knowledge to make recommendations and provide guidance that will enable practitioners to gather and analyse meaningful data, using scientific rigor to appraise restoration success. What should be monitored depends on the river type and the type and scale of intervention. By understanding how habitats are likely to change we can anticipate what species, life stages, and communities are likely to be affected. Monitoring should therefore be integrated and include both environmental/habitat and biota assessments. A robust scientific approach to monitoring and appraisal is resource intensive. We recommend that appraisal efforts be directed to where they will provide the greatest evidence, including ‘flagship’ restoration schemes for detailed long-term monitoring. Such an approach will provide the evidence needed to understand which restoration measures work where and ensure that they can be applied with confidence elsewhere.

Defining a Path Toward the Use of Fast-Scan Cyclic Voltammetry in Human Studies

Fast Scan Cyclic Voltammetry (FSCV) has been used for decades as a neurochemical tool for in vivo detection of phasic changes in electroactive neurotransmitters in animal models. Recently, multiple research groups have initiated human neurochemical studies using FSCV or demonstrated interest in bringing FSCV into clinical use. However, there remain technical challenges that limit clinical implementation of FSCV by creating barriers to appropriate scientific rigor and patient safety. In order to progress with clinical FSCV, these limitations must be first addressed through (1) appropriate pre-clinical studies to ensure accurate measurement of neurotransmitters and (2) the application of a risk management framework to assess patient safety. The intent of this work is to bring awareness of the current issues associated with FSCV to the scientific, engineering, and clinical communities and encourage them to seek solutions or alternatives that ensure data accuracy, rigor and reproducibility, and patient safety.

An updated insect exclosure design for pollination ecology

Exclosures are a common method for quantifying the effects of animal pollinators on flowering plant species. However, a lack of standardized designs or clear descriptions of previously implemented exclosure designs decreases replicability in pollination studies and reduces scientific rigor. We summarized previous descriptions of pollination exclosure designs, and developed/tested a novel exclosure design in alpine environments on the Beartooth Plateau in northern Wyoming, USA. This exclosure design consists of a cylindrical internal wire frame, integrated ground stakes, and various mesh materials attached to the exterior. Exclosures on the plateau showed high efficacy in inhibiting insects from pollinating flowering plants, and nearly all of these exclosures remained functional throughout the time they were in place. Our updated exclosure design is effective, inexpensive, easy to produce, and widely applicable across differing ecosystems and experimental design types.

The Role of Applied Mechanics in Bridging the Gaps in Prior Learning for Aspirants of Engineering Education

Building a technology-driven world appears to be the main motivational force behind students choosing to undertake engineering studies. The first year of engineering education plays a significant role in demonstrating sufficient mathematical and scientific rigor to satisfy these motivational factors. The common applied mechanics courses play a central role in achieving this. At the same time, a vast majority of students suffer from a lack of the necessary mathematical skills and analytical orientation for various reasons. Due to different educational philosophies and teaching pedagogies, a lack of proper integration between mathematics and applied mechanics is common. Several efforts were made to build better curriculum, teaching, and learning systems, resulting in widely varied solutions, but most of them require drastically different implementation approaches. With sufficient rigor in teaching and assessment, the first-year applied mechanics (common) courses designed for engineering students can solve students’ mathematical and motivational lapses and help bridge the gaps between pre-university and university education endeavours. This paper presents evidence supporting this argument. In particular, datasets collected from the direct experiences delivering the first-year static and dynamics courses to many students over the past decade and a half are analysed to establish the proposition.