Learn, Generate, Rank, Explain: A Case Study of Visual Explanation by Generative Machine Learning

While the computer vision problem of searching for activities in videos is usually addressed by using discriminative models, their decisions tend to be opaque and difficult for people to understand. We propose a case study of a novel machine learning approach for generative searching and ranking of motion capture activities with visual explanation. Instead of directly ranking videos in the database given a text query, our approach uses a variant of Generative Adversarial Networks (GANs) to generate exemplars based on the query and uses them to search for the activity of interest in a large database. Our model is able to achieve comparable results to its discriminative counterpart, while being able to dynamically generate visual explanations. In addition to our searching and ranking method, we present an explanation interface that enables the user to successfully explore the model’s explanations and its confidence by revealing query-based, model-generated motion capture clips that contributed to the model’s decision. Finally, we conducted a user study with 44 participants to show that by using our model and interface, participants benefit from a deeper understanding of the model’s conceptualization of the search query. We discovered that the XAI system yielded a comparable level of efficiency, accuracy, and user-machine synchronization as its black-box counterpart, if the user exhibited a high level of trust for AI explanation.

Flesh of pumpkin from ecological farming as part of fruit suitable for non-destructive cultivar classification using computer vision

The aim of this study was to build the discriminative models for distinguishing the different cultivars of flesh of pumpkin ‘Bambino’, ‘Butternut’, ‘Uchiki Kuri’ and ‘Orange’ based on selected textures of the outer surface of images of cubes. The novelty of research involved the use of about 2000 different textures for one image. The highest total accuracy (98%) of discrimination of pumpkin ‘Bambino’, ‘Butternut’, ‘Uchiki Kuri’ and ‘Orange’ was determined for models built based on textures selected from the color space Lab and the IBk classifier and some of the individual cultivars were classified with the correctness of 100%. The total accuracy of up to 96% was observed for color space RGB and 97.5% for color space XYZ. In the case of color channels, the total accuracies reached 91% for channel b, 89.5% for channel X, 89% for channel Z.

Discriminative Power of Geometric Parameters of Different Cultivars of Sour Cherry Pits Determined Using Machine Learning

The aim of this study was to develop models based on linear dimensions or shape factors, and the sets of combined linear dimensions and shape factors for discrimination of sour cherry pits of different cultivars (‘Debreceni botermo’, ‘Łutówka’, ‘Nefris’, ‘Kelleris’). The geometric parameters were calculated using image processing. The pits of different sour cherry cultivars statistically significantly differed in terms of selected dimensions and shape factors. The discriminative models built based on linear dimensions produced average accuracies of up to 95% for distinguishing the pit cultivars in the case of ‘Nefris’ vs. ‘Kelleris’ and 72% for all four cultivars. The average accuracies for the discriminative models built based on shape factors were up to 95% for the ‘Nefris’ and ‘Kelleris’ pits and 73% for four cultivars. The models combining the linear dimensions and shape factors produced accuracies reaching 96% for the ‘Nefris’ vs. ‘Kelleris’ pits and 75% for all cultivars. The geometric parameters with high discriminative power may be used for distinguishing different cultivars of sour cherry pits. It can be of great importance for practical applications. It may allow avoiding the adulteration and mixing of different cultivars.

Discrimination of tomato seeds belonging to different cultivars using machine learning

This study was aimed at developing the discriminant models for distinguishing the tomato seeds based on texture parameters of the outer surface of seeds calculated from the images (scans) converted to individual color channels R, G, B, L, a, b, X, Y, Z. The seeds of tomatoes ‘Green Zebra’, ‘Ożarowski’, ‘Pineapple’, Sacher F1 and Sandoline F1 were discriminated in pairs. The highest results were observed for models built based on sets of textures selected individually from color channels R, L and X and sets of textures selected from all color channels. In all cases, the tomato seeds ‘Green Zebra’ and ‘Ożarowski’ were discriminated with the highest average accuracy equal to 97% for the Multilayer Perceptron classifier and 96.25% for Random Forest for color channel R, 95.25% (Multilayer Perceptron) and 95% (Random Forest) for color channel L, 93% (Multilayer Perceptron) and 95% (Random Forest) for color channel X, 99.75% (Multilayer Perceptron) and 99.5% (Random Forest) for a set of textures selected from all color channels (R, G, B, L, a, b, X, Y, X). The highest average accuracies for other pairs of cultivars reached 98.25% for ‘Ożarowski’ vs. Sacher F1, 95.75% for ‘Pineapple’ vs. Sandoline F1, 97.5% for ‘Green Zebra’ vs. Sandoline F1, 97.25% for Sacher F1 vs. Sandoline F1 for models built based on textures selected from all color channels. The obtained results may be used in practice for the identification of cultivar of tomato seeds. The developed models allow to distinguish the tomato seed cultivars in an objective and fast way using digital image processing. The results confirmed the usefulness of texture parameters of the outer surface of tomato seeds for classification purposes. The discriminative models allow to obtain a very high probability and may be applied to authenticate and detect seed adulteration.

Deep representation features from DreamDIAXMBD improve the analysis of data-independent acquisition proteomics

We developed DreamDIAXMBD (denoted as DreamDIA), a software suite based on a deep representation model for data-independent acquisition (DIA) data analysis. DreamDIA adopts a data-driven strategy to capture comprehensive information from elution patterns of peptides in DIA data and achieves considerable improvements on both identification and quantification performance compared with other state-of-the-art methods such as OpenSWATH, Skyline and DIA-NN. Specifically, in contrast to existing methods which use only 6 to 10 selected fragment ions from spectral libraries, DreamDIA extracts additional features from hundreds of theoretical elution profiles originated from different ions of each precursor using a deep representation network. To achieve higher coverage of target peptides without sacrificing specificity, the extracted features are further processed by nonlinear discriminative models under the framework of positive-unlabeled learning with decoy peptides as affirmative negative controls. DreamDIA is publicly available at https://github.com/xmuyulab/DreamDIA-XMBD for high coverage and accuracy DIA data analysis.

Non-Mass Enhancements on DCE-MRI: Development and Validation of a Radiomics-Based Signature for Breast Cancer Diagnoses

PurposeWe aimed to assess the additional value of a radiomics-based signature for distinguishing between benign and malignant non-mass enhancement lesions (NMEs) on dynamic contrast-enhanced breast magnetic resonance imaging (breast DCE-MRI).MethodsIn this retrospective study, 232 patients with 247 histopathologically confirmed NMEs (malignant: 191; benign: 56) were enrolled from December 2017 to October 2020 as a primary cohort to develop the discriminative models. Radiomic features were extracted from one post-contrast phase (around 90s after contrast injection) of breast DCE-MRI images. The least absolute shrinkage and selection operator (LASSO) regression model was adapted to select features and construct the radiomics-based signature. Based on clinical and routine MR features, radiomics features, and combined information, three discriminative models were built using multivariable logistic regression analyses. In addition, an independent cohort of 72 patients with 72 NMEs (malignant: 50; benign: 22) was collected from November 2020 to April 2021 for the validation of the three discriminative models. Finally, the combined model was assessed using nomogram and decision curve analyses.ResultsThe routine MR model with two selected features of the time-intensity curve (TIC) type and MR-reported axillary lymph node (ALN) status showed a high sensitivity of 0.942 (95%CI, 0.906 - 0.974) and low specificity of 0.589 (95%CI, 0.464 - 0.714). The radiomics model with six selected features was significantly correlated with malignancy (P<0.001 for both primary and validation cohorts). Finally, the individual combined model, which contained factors including TIC types and radiomics signatures, showed good discrimination, with an acceptable sensitivity of 0.869 (95%CI, 0.816 to 0.916), improved specificity of 0.839 (95%CI, 0.750 to 0.929). The nomogram was applied to the validation cohort, reaching good discrimination, with a sensitivity of 0.820 (95%CI, 0.700 to 0.920), specificity of 0.864 (95%CI,0.682 to 1.000). The combined model was clinically helpful, as demonstrated by decision curve analysis.ConclusionsOur study added radiomics signatures into a conventional clinical model and developed a radiomics nomogram including radiomics signatures and TIC types. This radiomics model could be used to differentiate benign from malignant NMEs in patients with suspicious lesions on breast MRI.

An Intelligent Code Search Approach Using Hybrid Encoders

The intelligent code search with natural language queries has become an important researching area in software engineering. In this paper, we propose a novel deep learning framework At-CodeSM for source code search. The powerful code encoder in At-CodeSM, which is implemented with an abstract syntax tree parsing algorithm (Tree-LSTM) and token-level encoders, maintains both the lexical and structural features of source code in the process of code vectorizing. Both the representative and discriminative models are implemented with deep neural networks. Our experiments on the CodeSearchNet dataset show that At-CodeSM yields better performance in the task of intelligent code searching than previous approaches.

DreamDIA-XMBD: deep representation features improve the analysis of data-independent acquisition proteomics

We developed DreamDIA-XMBD, a software suite for data-independent acquisition (DIA) data analysis. DreamDIA-XMBD adopts a data-driven strategy to capture comprehensive information from elution patterns of target peptides in DIA data and achieves considerable improvements on both identification and quantification performance compared with other state-of-the-art methods such as OpenSWATH, Skyline and DIA-NN. More specifically, in contrast to existing methods which use only 6 to 10 selected transitions from spectral library, DreamDIA-XMBD extracts additional features from dozens of theoretical elution profiles originated from different ions of each precursor using a deep representation network. To achieve higher coverage of target peptides without sacrificing specificity, the extracted features are further processed by non-linear discriminative models under the framework of positive-unlabeled learning with decoy peptides as affirmative negative controls. DreamDIA-XMBD is written in Python, and is publicly available at https://github.com/xmuyulab/Dream-DIA-XMBD for high coverage and precision DIA data analysis.