A Conceptual Framework for Investigating and Mitigating Machine Learning Measurement Bias (MLMB) in Psychological Assessment

Given significant concerns about fairness and bias in the use of artificial intelligence (AI) and machine learning (ML) for assessing psychological constructs, we provide a conceptual framework for investigating and mitigating machine learning measurement bias (MLMB) from a psychometric perspective. MLMB is defined as differential functioning of the trained ML model between subgroups. MLMB can empirically manifest when a trained ML model produces different predicted score levels for individuals belonging to different subgroups (e.g., race, gender) despite them having the same ground truth level for the underlying construct of interest (e.g., personality), and/or when the model yields differential predictive accuracies across the subgroups. Because the development of ML models involves both data and algorithms, both biased data and algorithm training bias are potential sources of MLMB. Data bias can occur in the form of nonequivalence between subgroups in the ground truth, platform-based construct, behavioral expression, and/or feature computing. Algorithm training bias can occur when algorithms are developed with nonequivalence in the relation between extracted features and ground truth (i.e., algorithm features are differentially used, weighted, or transformed between subgroups). We explain how these potential sources of bias may manifest during ML model development and share initial ideas on how to mitigate them, recognizing that the development of new statistical and algorithmic procedures will need to follow. We also discuss how this framework brings clarity to MLMB but does not reduce the complexity of the issue.

Download Full-text

A machine learning-based approach for the inference of immunotherapy biomarker status in lung adenocarcinoma from hematoxylin and eosin (H&E) histopathology images.

Journal of Clinical Oncology ◽

10.1200/jco.2020.38.15_suppl.3122 ◽

2020 ◽

Vol 38 (15_suppl) ◽

pp. 3122-3122

Author(s):

Cory Batenchuk ◽

Huan-Wei Chang ◽

Peter Cimermancic ◽

Eunhee S. Yi ◽

Apaar Sadhwani ◽

...

Keyword(s):

Machine Learning ◽

Model Development ◽

Treatment Decision ◽

Ground Truth ◽

The Cancer Genome Atlas ◽

Classification Model ◽

Feature Maps ◽

Current Standard ◽

Gene Signatures ◽

Biological Features

3122 Background: The current standard work-up for both diagnosis and predictive biomarker testing in metastatic non-small cell lung cancer (NSCLC), can exhaust an entire tumor specimen. Notably, gene mutation panels or tumor mutation burden (TMB) testing currently requires 10 tissue slides and ranges from 10 days to 3 weeks from sample acquisition to test result. As more companion diagnostic (CDx)-restricted drugs are developed for NSCLC, rapid, tissue-sparing tests are sorely needed. We investigated whether TMB, T-effector (TEFF) gene signatures and PD-L1 status can be inferred from H&E images alone using a machine learning approach. Methods: Algorithm development included two steps: First, a neural network was trained to segment hand-annotated, pathologist-confirmed biological features from H&E images, such as tumor architecture and cell types. Second, these feature maps were fed into a classification model to predict the biomarker status. Ground truth biomarker status of the H&E-associated tumor samples came from whole exome sequencing (WES) for TMB, RNAseq for the TEFF gene signatures or reverse-phase protein array for PD-L1. Digital H&E images of NSCLC adenocarcinoma for model development were obtained from the cancer genome atlas (TCGA) and commercial sources. Results: This approach achieves > 75% accuracy in predicting TMB, TEFF and PD-L1 status, offers a way to interpret the model, and provides biological insights into the tumor-host microenvironment. Conclusions: These findings suggest that biomarker inference from H&E images is feasible, and may be sufficiently accurate to supplement or replace current tissue-based tests in a clinical setting. Our approach utilizes biological features for inference, and is thus robust, interpretable, and readily verifiable by pathologists. Finally, biomarker status inference from a single H&E image may enable testing in patients whose tumor tissue has been exhausted, spare further tissue use, and return test results within hours to enable rapid treatment decision-making to maximize patient benefit.

Download Full-text

A Conceptual Framework to Aid Attribute Selection in Machine Learning Student Performance Prediction Models

International Journal of Interactive Mobile Technologies (iJIM) ◽

10.3991/ijim.v15i15.20019 ◽

2021 ◽

Vol 15 (15) ◽

pp. 4

Author(s):

Ijaz Muhammad Khan ◽

Abdul Rahim Ahmad ◽

Nafaa Jabeur ◽

Mohammed Najah Mahdi

Keyword(s):

Machine Learning ◽

Conceptual Framework ◽

Student Performance ◽

Learning Analytics ◽

Prediction Models ◽

Model Development ◽

Machine Learning Algorithms ◽

Attribute Selection ◽

Building Models ◽

Dynamic Attributes

One of the important key applications of learning analytics is offering an opportunity to the institutions to track the student’s academic activities and provide them with real-time adaptive consultations if the student academic performance diverts towards the inadequate outcome. Still, numerous barriers exist while developing and implementing such kind of learning analytics applications. Machine learning algorithms emerge as useful tools to endorse learning analytics by building models capable of forecasting the final outcome of students based on their available attributes. The machine learning algorithm’s performance demotes with using the entire attributes and thus a vigilant selection of predicting attributes boosts the performance of the produced model. Though, several constructive techniques facilitate to identify the subset of productive attributes, however, the challenging task is to evaluate if the prediction attributes are meaningful, explicit, and controllable by the students. This paper reviews the existing literature to come up with the student’s attributes used in developing prediction models. We propose a conceptual framework which demonstrates the classification of attributes as either latent or dynamic. The latent attributes may appear significant but the student is not able to control these attribute, on the other hand, the student has command to restrain the dynamic attributes. Each of the major class is further categorized to present an opportunity to the researchers to pick constructive attributes for model development.

Download Full-text

Nanosecond Photodynamics Simulations of a Cis-Trans Isomerization Are Enabled by Machine Learning

10.26434/chemrxiv.13047863 ◽

2020 ◽

Author(s):

Jingbai Li ◽

Patrick Reiser ◽

André Eberhard ◽

Pascal Friederich ◽

Steven Lopez

Keyword(s):

Machine Learning ◽

Neural Networks ◽

Excited State ◽

Adaptive Sampling ◽

Computational Cost ◽

Ground Truth ◽

Absolute Error ◽

Photochemical Reactions ◽

Computational Techniques ◽

Full Potential

Photochemical reactions are being increasingly used to construct complex molecular architectures with mild and straightforward reaction conditions. Computational techniques are increasingly important to understand the reactivities and chemoselectivities of photochemical isomerization reactions because they offer molecular bonding information along the excited-state(s) of photodynamics. These photodynamics simulations are resource-intensive and are typically limited to 1–10 picoseconds and 1,000 trajectories due to high computational cost. Most organic photochemical reactions have excited-state lifetimes exceeding 1 picosecond, which places them outside possible computational studies. Westermeyr et al. demonstrated that a machine learning approach could significantly lengthen photodynamics simulation times for a model system, methylenimmonium cation (CH2NH2+).We have developed a Python-based code, Python Rapid Artificial Intelligence Ab Initio Molecular Dynamics (PyRAI2MD), to accomplish the unprecedented 10 ns cis-trans photodynamics of trans-hexafluoro-2-butene (CF3–CH=CH–CF3) in 3.5 days. The same simulation would take approximately 58 years with ground-truth multiconfigurational dynamics. We proposed an innovative scheme combining Wigner sampling, geometrical interpolations, and short-time quantum chemical trajectories to effectively sample the initial data, facilitating the adaptive sampling to generate an informative and data-efficient training set with 6,232 data points. Our neural networks achieved chemical accuracy (mean absolute error of 0.032 eV). Our 4,814 trajectories reproduced the S1 half-life (60.5 fs), the photochemical product ratio (trans: cis = 2.3: 1), and autonomously discovered a pathway towards a carbene. The neural networks have also shown the capability of generalizing the full potential energy surface with chemically incomplete data (trans → cis but not cis → trans pathways) that may offer future automated photochemical reaction discoveries.

Download Full-text

A NOVEL EXTENSIVE EX-VIVO OCT DATABASE FROM MURINE MODELS OF COLORECTAL CANCER

British Journal of Surgery ◽

10.1093/bjs/znab160.030 ◽

2021 ◽

Vol 108 (Supplement_3) ◽

Author(s):

J Bote ◽

J F Ortega-Morán ◽

C L Saratxaga ◽

B Pagador ◽

A Picón ◽

...

Keyword(s):

Colorectal Cancer ◽

Machine Learning ◽

Structural Information ◽

Ex Vivo ◽

Ground Truth ◽

Colon Polyps ◽

Learning Methods ◽

Non Invasive ◽

Machine Learning Methods ◽

In Situ Methods

Abstract INTRODUCTION New non-invasive technologies for improving early diagnosis of colorectal cancer (CRC) are demanded by clinicians. Optical Coherence Tomography (OCT) provides sub-surface structural information and offers diagnosis capabilities of colon polyps, further improved by machine learning methods. Databases of OCT images are necessary to facilitate algorithms development and testing. MATERIALS AND METHODS A database has been acquired from rat colonic samples with a Thorlabs OCT system with 930nm centre wavelength that provides 1.2KHz A-scan rate, 7μm axial resolution in air, 4μm lateral resolution, 1.7mm imaging depth in air, 6mm x 6mm FOV, and 107dB sensitivity. The colon from anaesthetised animals has been excised and samples have been extracted and preserved for ex-vivo analysis with the OCT equipment. RESULTS This database consists of OCT 3D volumes (C-scans) and 2D images (B-scans) of murine samples from: 1) healthy tissue, for ground-truth comparison (18 samples; 66 C-scans; 17,478 B-scans); 2) hyperplastic polyps, obtained from an induced colorectal hyperplastic murine model (47 samples; 153 C-scans; 42,450 B-scans); 3) neoplastic polyps (adenomatous and adenocarcinomatous), obtained from clinically validated Pirc F344/NTac-Apcam1137 rat model (232 samples; 564 C-scans; 158,557 B-scans); and 4) unknown tissue (polyp adjacent, presumably healthy) (98 samples; 157 C-scans; 42,070 B-scans). CONCLUSIONS A novel extensive ex-vivo OCT database of murine CRC model has been obtained and will be openly published for the research community. It can be used for classification/segmentation machine learning methods, for correlation between OCT features and histopathological structures, and for developing new non-invasive in-situ methods of diagnosis of colorectal cancer.

Download Full-text

Experimental Evaluation of Computer Vision and Machine Learning-Based UAV Detection and Ranging

Drones ◽

10.3390/drones5020037 ◽

2021 ◽

Vol 5 (2) ◽

pp. 37

Author(s):

Bingsheng Wei ◽

Martin Barczyk

Keyword(s):

Machine Learning ◽

Mean Squared Error ◽

Tracking System ◽

Ground Truth ◽

White Background ◽

Cascade Classifier ◽

Detection Algorithms ◽

Squared Error ◽

Test Conditions ◽

Video Feed

We consider the problem of vision-based detection and ranging of a target UAV using the video feed from a monocular camera onboard a pursuer UAV. Our previously published work in this area employed a cascade classifier algorithm to locate the target UAV, which was found to perform poorly in complex background scenes. We thus study the replacement of the cascade classifier algorithm with newer machine learning-based object detection algorithms. Five candidate algorithms are implemented and quantitatively tested in terms of their efficiency (measured as frames per second processing rate), accuracy (measured as the root mean squared error between ground truth and detected location), and consistency (measured as mean average precision) in a variety of flight patterns, backgrounds, and test conditions. Assigning relative weights of 20%, 40% and 40% to these three criteria, we find that when flying over a white background, the top three performers are YOLO v2 (76.73 out of 100), Faster RCNN v2 (63.65 out of 100), and Tiny YOLO (59.50 out of 100), while over a realistic background, the top three performers are Faster RCNN v2 (54.35 out of 100, SSD MobileNet v1 (51.68 out of 100) and SSD Inception v2 (50.72 out of 100), leading us to recommend Faster RCNN v2 as the recommended solution. We then provide a roadmap for further work in integrating the object detector into our vision-based UAV tracking system.

Download Full-text

An encryption–decryption framework to validating single-particle imaging

Scientific Reports ◽

10.1038/s41598-020-79589-0 ◽

2021 ◽

Vol 11 (1) ◽

Cited By ~ 1

Author(s):

Zhou Shen ◽

Colin Zhi Wei Teo ◽

Kartik Ayyer ◽

N. Duane Loh

Keyword(s):

Conceptual Framework ◽

Single Particle ◽

Latent Variables ◽

Ground Truth ◽

Resolving Power ◽

Validation Measure ◽

Encryption Decryption ◽

Diffraction Patterns ◽

Particle Imaging ◽

Single Particle Imaging

AbstractWe propose an encryption–decryption framework for validating diffraction intensity volumes reconstructed using single-particle imaging (SPI) with X-ray free-electron lasers (XFELs) when the ground truth volume is absent. This conceptual framework exploits each reconstructed volumes’ ability to decipher latent variables (e.g. orientations) of unseen sentinel diffraction patterns. Using this framework, we quantify novel measures of orientation disconcurrence, inconsistency, and disagreement between the decryptions by two independently reconstructed volumes. We also study how these measures can be used to define data sufficiency and its relation to spatial resolution, and the practical consequences of focusing XFEL pulses to smaller foci. This conceptual framework overcomes critical ambiguities in using Fourier Shell Correlation (FSC) as a validation measure for SPI. Finally, we show how this encryption-decryption framework naturally leads to an information-theoretic reformulation of the resolving power of XFEL-SPI, which we hope will lead to principled frameworks for experiment and instrument design.

Download Full-text

Assessment of Machine Learning-Based Audiovisual Quality Predictors

ACM Transactions on Multimedia Computing Communications and Applications ◽

10.1145/3430376 ◽

2021 ◽

Vol 17 (2) ◽

pp. 1-22

Author(s):

Mythili K. ◽

Manish Narwaria

Keyword(s):

Machine Learning ◽

Signal Processing ◽

Quality Assessment ◽

Computational Models ◽

Model Development ◽

Multimedia Communication ◽

Current Approach ◽

Learning Ability ◽

Data Uncertainty ◽

Attractive Alternative

Quality assessment of audiovisual (AV) signals is important from the perspective of system design, optimization, and management of a modern multimedia communication system. However, automatic prediction of AV quality via the use of computational models remains challenging. In this context, machine learning (ML) appears to be an attractive alternative to the traditional approaches. This is especially when such assessment needs to be made in no-reference (i.e., the original signal is unavailable) fashion. While development of ML-based quality predictors is desirable, we argue that proper assessment and validation of such predictors is also crucial before they can be deployed in practice. To this end, we raise some fundamental questions about the current approach of ML-based model development for AV quality assessment and signal processing for multimedia communication in general. We also identify specific limitations associated with the current validation strategy which have implications on analysis and comparison of ML-based quality predictors. These include a lack of consideration of: (a) data uncertainty, (b) domain knowledge, (c) explicit learning ability of the trained model, and (d) interpretability of the resultant model. Therefore, the primary goal of this article is to shed some light into mentioned factors. Our analysis and proposed recommendations are of particular importance in the light of significant interests in ML methods for multimedia signal processing (specifically in cases where human-labeled data is used), and a lack of discussion of mentioned issues in existing literature.

Download Full-text

Unraveling the deep learning gearbox in optical coherence tomography image segmentation towards explainable artificial intelligence

Communications Biology ◽

10.1038/s42003-021-01697-y ◽

2021 ◽

Vol 4 (1) ◽

Author(s):

Peter M. Maloca ◽

Philipp L. Müller ◽

Aaron Y. Lee ◽

Adnan Tufail ◽

Konstantinos Balaskas ◽

...

Keyword(s):

Neural Network ◽

Machine Learning ◽

Optical Coherence Tomography ◽

Image Segmentation ◽

Convolutional Neural Network ◽

Learning Algorithm ◽

Ground Truth ◽

Optical Coherence Tomography Image ◽

Optical Coherence ◽

Tomography Image

AbstractMachine learning has greatly facilitated the analysis of medical data, while the internal operations usually remain intransparent. To better comprehend these opaque procedures, a convolutional neural network for optical coherence tomography image segmentation was enhanced with a Traceable Relevance Explainability (T-REX) technique. The proposed application was based on three components: ground truth generation by multiple graders, calculation of Hamming distances among graders and the machine learning algorithm, as well as a smart data visualization (‘neural recording’). An overall average variability of 1.75% between the human graders and the algorithm was found, slightly minor to 2.02% among human graders. The ambiguity in ground truth had noteworthy impact on machine learning results, which could be visualized. The convolutional neural network balanced between graders and allowed for modifiable predictions dependent on the compartment. Using the proposed T-REX setup, machine learning processes could be rendered more transparent and understandable, possibly leading to optimized applications.

Download Full-text

Predictive Maintenance in Building Facilities: A Machine Learning-Based Approach

Sensors ◽

10.3390/s21041044 ◽

2021 ◽

Vol 21 (4) ◽

pp. 1044

Author(s):

Yassine Bouabdallaoui ◽

Zoubeir Lafhaj ◽

Pascal Yim ◽

Laure Ducoulombier ◽

Belkacem Bennadji

Keyword(s):

Machine Learning ◽

Model Development ◽

Predictive Maintenance ◽

Data Availability ◽

Machine Learning Techniques ◽

Automation System ◽

Building Maintenance ◽

Collection Data ◽

Iot Devices

The operation and maintenance of buildings has seen several advances in recent years. Multiple information and communication technology (ICT) solutions have been introduced to better manage building maintenance. However, maintenance practices in buildings remain less efficient and lead to significant energy waste. In this paper, a predictive maintenance framework based on machine learning techniques is proposed. This framework aims to provide guidelines to implement predictive maintenance for building installations. The framework is organised into five steps: data collection, data processing, model development, fault notification and model improvement. A sport facility was selected as a case study in this work to demonstrate the framework. Data were collected from different heating ventilation and air conditioning (HVAC) installations using Internet of Things (IoT) devices and a building automation system (BAS). Then, a deep learning model was used to predict failures. The case study showed the potential of this framework to predict failures. However, multiple obstacles and barriers were observed related to data availability and feedback collection. The overall results of this paper can help to provide guidelines for scientists and practitioners to implement predictive maintenance approaches in buildings.

Download Full-text

Assessing biases, relaxing moralism: On ground-truthing practices in machine learning design and application

Big Data & Society ◽

10.1177/20539517211013569 ◽

2021 ◽

Vol 8 (1) ◽

pp. 205395172110135

Author(s):

Florian Jaton

Keyword(s):

Machine Learning ◽

William James ◽

A Priori ◽

Learning Algorithms ◽

Three Dimensional ◽

Ground Truth ◽

Machine Learning Algorithms ◽

Ground Truthing ◽

Set Up ◽

The Moment

This theoretical paper considers the morality of machine learning algorithms and systems in the light of the biases that ground their correctness. It begins by presenting biases not as a priori negative entities but as contingent external referents—often gathered in benchmarked repositories called ground-truth datasets—that define what needs to be learned and allow for performance measures. I then argue that ground-truth datasets and their concomitant practices—that fundamentally involve establishing biases to enable learning procedures—can be described by their respective morality, here defined as the more or less accounted experience of hesitation when faced with what pragmatist philosopher William James called “genuine options”—that is, choices to be made in the heat of the moment that engage different possible futures. I then stress three constitutive dimensions of this pragmatist morality, as far as ground-truthing practices are concerned: (I) the definition of the problem to be solved (problematization), (II) the identification of the data to be collected and set up (databasing), and (III) the qualification of the targets to be learned (labeling). I finally suggest that this three-dimensional conceptual space can be used to map machine learning algorithmic projects in terms of the morality of their respective and constitutive ground-truthing practices. Such techno-moral graphs may, in turn, serve as equipment for greater governance of machine learning algorithms and systems.

Download Full-text