Opportunistic Use of Crowdsourced Workers for Online Relabeling of Potential Adversarial Examples

<div>Deep Neural Networks (DDNs) have achieved tremendous success in handling various Machine Learning (ML) tasks, such as speech recognition, Natural Language Processing, and image classification. However, they have shown vulnerability to well-designed inputs called adversarial examples. Researchers in industry and academia have proposed many adversarial example defense techniques. However, none can provide complete robustness. The cutting-edge defense techniques offer partial reliability. Thus, complementing them with another layer of protection is a must, especially for mission-critical applications. This paper proposes a novel Online Selection and Relabeling Algorithm (OSRA) that opportunistically utilizes a limited number of crowdsourced workers (budget-constraint crowdsourcing) to maximize the ML system’s robustness. OSRA strives to use crowdsourced workers effectively by selecting the most suspicious inputs (the potential adversarial examples) and moving them to the crowdsourced workers to be validated and corrected (relabeled). As a result, the impact of adversarial examples gets reduced, and accordingly, the ML system becomes more robust. We also proposed a heuristic threshold selection method that contributes to enhancing the prediction system’s reliability. We empirically validated our proposed algorithm and found that it can efficiently and optimally utilize the allocated budget for crowdsourcing. It is also effectively integrated with a state-ofthe- art black-box (transfer-based) defense technique, resulting in a more robust system. Simulation results show that OSRA can outperform a random selection algorithm by 60% and achieve comparable performance to an optimal offline selection benchmark. They also show that OSRA’s performance has a positive correlation with system robustness.<br></div>

Opportunistic Use of Crowdsourced Workers for Online Relabeling of Potential Adversarial Examples

<div>Deep Neural Networks (DDNs) have achieved tremendous success in handling various Machine Learning (ML) tasks, such as speech recognition, Natural Language Processing, and image classification. However, they have shown vulnerability to well-designed inputs called adversarial examples. Researchers in industry and academia have proposed many adversarial example defense techniques. However, none can provide complete robustness. The cutting-edge defense techniques offer partial reliability. Thus, complementing them with another layer of protection is a must, especially for mission-critical applications. This paper proposes a novel Online Selection and Relabeling Algorithm (OSRA) that opportunistically utilizes a limited number of crowdsourced workers (budget-constraint crowdsourcing) to maximize the ML system’s robustness. OSRA strives to use crowdsourced workers effectively by selecting the most suspicious inputs (the potential adversarial examples) and moving them to the crowdsourced workers to be validated and corrected (relabeled). As a result, the impact of adversarial examples gets reduced, and accordingly, the ML system becomes more robust. We also proposed a heuristic threshold selection method that contributes to enhancing the prediction system’s reliability. We empirically validated our proposed algorithm and found that it can efficiently and optimally utilize the allocated budget for crowdsourcing. It is also effectively integrated with a state-ofthe- art black-box (transfer-based) defense technique, resulting in a more robust system. Simulation results show that OSRA can outperform a random selection algorithm by 60% and achieve comparable performance to an optimal offline selection benchmark. They also show that OSRA’s performance has a positive correlation with system robustness.<br></div>

Evaluating Network Threshold Selection for Structural and Functional Brain Connectomes

Structural and functional brain connectomes represent macroscale neurophysical data collected through methods such as magnetic resonance imaging (MRI). Such data may contain noise that contribute to false positive edges, which motivates the use of edge-wise thresholding. Thresholding procedures are useful for reducing network density in graphs to retain only the most informative, non-noisy edges. Nevertheless, limited consensus exists on selecting appropriate threshold levels. We compare existing thresholding methods and introduce a novel thresholding approach in the context of MRI-derived and simulated brain connectomes. Performance is measured using normalized mutual information (NMI), a quantity robust to arbitrary changes in partition labeling, and describes the similarity of community structure between two node-matched networks. We found that the percolation-based threshold and our newly proposed objective function-based threshold exhibited the best performance in terms of NMI. We show an application of these two thresholding methods to real data that showed that both percolation-based and objective function-based thresholding provide statistically similar NMI values between real world FC networks and structural connectivity (SC) counterparts, where shared modular structure is assumed. The two thresholding methods, however, achieve these NMI values at significantly different thresholds (p<0.0001) in both simulated and real networks. Moreover, the threshold obtained from the objective function gives a more accurate estimate of the number of modules present in the network and includes more flexibility in threshold selection, suggesting that this method may represent a useful option for graph thresholding.

DenVar: Density-based Variation analysis of multiplex imaging data

Multiplex immunohistochemistry (mIHC) and multiplexed ion beam imaging (MIBI) platforms have become increasingly popular for studying complex single-cell biology in cancer patients. In such studies, researchers test for association between functional markers and survival in a two-step process. First, they count the number of positive cells, defined as the number of cells where a functional marker is significantly expressed. Then, they partition the patients into two groups and test for association between the group label with survival. Consequently, the approach suffers from subjectivity and lack of robustness. In this paper, we propose a threshold-free distance metric between patients solely based on their marker probability densities. Using the proposed distance, we have developed two association tests, one based on hierarchical clustering and the other based on linear mixed model. Our method obviates the need for the arduous step of threshold selection, getting rid of the subjectivity bias. The method also intuitively generalizes to joint analysis of multiple markers. We assessed the performance of our method through extensive simulation studies and also used it to analyze two multiplex imaging datasets.