Evaluation of cell segmentation methods without reference segmentations

Cell segmentation is a cornerstone of many bioimage informatics studies. Inaccurate segmentation introduces computational error in downstream cellular analysis. Evaluating the segmentation results is thus a necessary step for developing the segmentation methods as well as choosing the most appropriate one for a certain kind of tissue or image. The evaluation process has typically involved comparison of segmentations to those generated by humans, which can be expensive and subject to unknown bias. We present here an approach that seeks to evaluate cell segmentation methods without relying upon comparison to results from humans. For this, we defined a number of segmentation quality metrics that can be applied to multichannel fluorescence images. We calculated these metrics for 11 previously-described segmentation methods applied to datasets from 5 multiplexed microscope modalities covering 5 tissues. Using principal component analysis to combine the metrics we defined an overall cell segmentation quality score and ranked the segmentation methods. A Reproducible Research Archive containing all data and code will be made available upon publication at http://hubmap.scs.cmu.edu.

Algorithmic Bias in Education

Draft Preprint. In this paper, we review algorithmic bias in education, discussing the causes of that bias and reviewing the empirical literature on the specific ways that algorithmic bias is known to have manifested in education. While other recent work has reviewed mathematical definitions of fairness and expanded algorithmic approaches to reducing bias, our review focuses instead on solidifying the current understanding of the concrete impacts of algorithmic bias in education—which groups are known to be impacted and which stages and agents in the development and deployment of educational algorithms are implicated. We discuss theoretical and formal perspectives on algorithmic bias, connect those perspectives to the machine learning pipeline, and review metrics for assessing bias. Next, we review the evidence around algorithmic bias in education, beginning with the most heavily-studied categories of race/ethnicity, gender, and nationality, and moving to the available evidence of bias for less-studied categories, such as socioeconomic status, disability, and military-connected status. Acknowledging the gaps in what has been studied, we propose a framework for moving from unknown bias to known bias and from fairness to equity. We discuss obstacles to addressing these challenges and propose four areas of effort for mitigating and resolving the problems of algorithmic bias in AIED systems and other educational technology.

Designing experiments informed by observational studies

The increasing availability of passively observed data has yielded a growing interest in “data fusion” methods, which involve merging data from observational and experimental sources to draw causal conclusions. Such methods often require a precarious tradeoff between the unknown bias in the observational dataset and the often-large variance in the experimental dataset. We propose an alternative approach, which avoids this tradeoff: rather than using observational data for inference, we use it to design a more efficient experiment. We consider the case of a stratified experiment with a binary outcome and suppose pilot estimates for the stratum potential outcome variances can be obtained from the observational study. We extend existing results to generate confidence sets for these variances, while accounting for the possibility of unmeasured confounding. Then, we pose the experimental design problem as a regret minimization problem subject to the constraints imposed by our confidence sets. We show that this problem can be converted into a concave maximization and solved using conventional methods. Finally, we demonstrate the practical utility of our methods using data from the Women’s Health Initiative.

Asynchronous Adaptive Fault-Tolerant Control for Networked Stochastic Unmanned Surface Vehicles with Multiple Types of Actuator Faults

This paper aims to design an asynchronous adaptive fault-tolerant controller for the networked stochastic unmanned surface vehicles (NSUSVs) subject to multiple types of actuator faults and external disturbance. The partial fault and bias fault of the actuator are taken into consideration simultaneously. By estimating online the unknown bias fault of the actuator and the external disturbances, the proposed adaptive fault-tolerant controller can automatically compensate for these impacts produced by actuator faults and external perturbation while preserving the uniformly ultimate boundedness of the solutions. Both the faulty actuator and the designed controller are asynchronous with the NSUSVs. Moreover, a mode-dependent adaptive event-triggered mechanism (AETM) is introduced in order to facilitate network resources utilization. Finally, the effectiveness and correctness of the proposed design scheme are verified by a numerical example.

Estimation of Impedance and Susceptance Parameters of a 3-Phase Cable System Using PMU Data

This paper proposes a new regression-based method to estimate resistance, reactance, and susceptance parameters of a 3-phase cable segment using phasor measurement unit (PMU) data. The novelty of this method is that it gives accurate parameter estimates in the presence of unknown bias errors in the measurements. Bias errors are fixed errors present in the measurement equipment and have been neglected in previous such attempts of estimating parameters of a 3-phase line or cable segment. In power system networks, the sensors used for current and voltage measurements have inherent magnitude and phase errors whose measurements need to be corrected using calibrated correction coefficients. Neglecting or using wrong error correction coefficients causes fixed bias errors in the measured current and voltage signals. Measured current and voltage signals at different time instances are the variables in the regression model used to estimate the cable parameters. Thus, the bias errors in the sensors become fixed errors in the variables. This error in variables leads to inaccuracy in the estimated parameters. To avoid this, the proposed method uses a new regression model using extra parameters which facilitate the modeling of present but unknown bias errors in the measurement system. These added parameters account for the errors present in the non- or wrongly calibrated sensors. Apart from the measurement bias, random measurement errors also contribute to the total uncertainty of the estimated parameters. This paper also presents and compares methods to estimate the total uncertainty in the estimated parameters caused by the bias and random errors present in the measurement system. Results from simulation-based and laboratory experiments are presented to show the efficacy of the proposed method. A discussion about analyzing the obtained results is also presented.

Comparisons of echo-integration performance from two multiplexed echosounders

A time-series of acoustically derived aquatic biomass estimates relies on the acoustic equipment maintaining the same performance throughout the time-series. This is normally achieved through a regular calibration process. When the acoustic equipment changes it is necessary to verify that the new equipment produces a similar result to the old equipment, otherwise an unknown bias can be introduced into the time-series. The commonly used Simrad EK60 echosounder has been superseded by the Simrad EK80 echosounder and the performance of these two scientific echosounder systems was compared using interleaved pinging through the same transducer. This was repeated for multiple transducer frequencies (18, 38, 70, 120, and 200 kHz) and from two vessels (Norway’s G.O. Sars in the North Sea and The Netherlands’ Tridens in the Northeast Atlantic Ocean). The broadband facility of the EK80 was not used. Regressions of the grid-integrated backscatter from the two systems were highly linear. The difference in area backscattering coefficients in typical survey conditions was less than 0.6 dB (12%) at the main survey frequency of 38 kHz. In most conventional fish acoustic surveys, the observed differences are less than other sources of survey bias and uncertainty.

Multiple Adaptive Fading Schmidt-Kalman Filter for Unknown Bias

Unknown biases in dynamic and measurement models of the dynamic systems can bring greatly negative effects to the state estimates when using a conventional Kalman filter algorithm. Schmidt introduces the “consider” analysis to account for errors in both the dynamic and measurement models due to the unknown biases. Although the Schmidt-Kalman filter “considers” the biases, the uncertain initial values and incorrect covariance matrices of the unknown biases still are not considered. To solve this problem, a multiple adaptive fading Schmidt-Kalman filter (MAFSKF) is designed by using the proposed multiple adaptive fading Kalman filter to mitigate the negative effects of the unknown biases in dynamic or measurement model. The performance of the MAFSKF algorithm is verified by simulation.

Convenience Sample of On-Farm Research Cooperators Representative of Wisconsin Farmers

Researchers interested in describing or understanding agroecological systems have many reasons to consider on-farm research. Yet, despite the inherent realism and pedagogical value of on-farm studies, recruiting cooperators can be difficult and this difficulty can result in so-called “convenience samples” containing a potentially large and unknown bias. There is often no formal justification for claiming that on-farm research results can be extrapolated to farms beyond those participating in the study. In some sufficiently well-understood research areas, models may be able to correct for potential bias; however, no theoretical argument is as persuasive as a direct comparison between a randomized and a convenience sample. In a 30-cooperator on-farm study investigating weed community dynamics across the state of Wisconsin, we distributed a written survey probing farmer weed management behaviors and attitudes. The survey contained 59 questions that overlapped a large, randomized survey of farmer corn pest management behavior. We compared 187 respondents from the larger survey with the 18 respondents from our on-farm study. For dichotomous response questions, we found no difference in response rate for 80% of the questions (α = 0.2, β > 0.5). Differences between the two groups were logically connected to the selection criteria used to recruit cooperators in the on-farm study. Similarly, comparisons of nondichotomous response questions did not differ for 80% of the questions (α = 0.05, β > 0.9). Exploratory multivariate analyses failed to reveal differences that might have been hidden from the marginal analyses. We argue that our findings support the notion that the convenience samples often associated with on-farm research may be representative of the more general class of farms, despite lack of bias protection provided by truly randomized designs.