Using Convolutional Neural Networks to Efficiently Extract Immense Phenological Data From Community Science Images

Community science image libraries offer a massive, but largely untapped, source of observational data for phenological research. The iNaturalist platform offers a particularly rich archive, containing more than 49 million verifiable, georeferenced, open access images, encompassing seven continents and over 278,000 species. A critical limitation preventing scientists from taking full advantage of this rich data source is labor. Each image must be manually inspected and categorized by phenophase, which is both time-intensive and costly. Consequently, researchers may only be able to use a subset of the total number of images available in the database. While iNaturalist has the potential to yield enough data for high-resolution and spatially extensive studies, it requires more efficient tools for phenological data extraction. A promising solution is automation of the image annotation process using deep learning. Recent innovations in deep learning have made these open-source tools accessible to a general research audience. However, it is unknown whether deep learning tools can accurately and efficiently annotate phenophases in community science images. Here, we train a convolutional neural network (CNN) to annotate images of Alliaria petiolata into distinct phenophases from iNaturalist and compare the performance of the model with non-expert human annotators. We demonstrate that researchers can successfully employ deep learning techniques to extract phenological information from community science images. A CNN classified two-stage phenology (flowering and non-flowering) with 95.9% accuracy and classified four-stage phenology (vegetative, budding, flowering, and fruiting) with 86.4% accuracy. The overall accuracy of the CNN did not differ from humans (p = 0.383), although performance varied across phenophases. We found that a primary challenge of using deep learning for image annotation was not related to the model itself, but instead in the quality of the community science images. Up to 4% of A. petiolata images in iNaturalist were taken from an improper distance, were physically manipulated, or were digitally altered, which limited both human and machine annotators in accurately classifying phenology. Thus, we provide a list of photography guidelines that could be included in community science platforms to inform community scientists in the best practices for creating images that facilitate phenological analysis.

Design Computer-Aided Diagnosis System Based on Chest CT Evaluation of Pulmonary Nodules

At present, the diagnosis and treatment of lung cancer have always been one of the research hotspots in the medical field. Early diagnosis and treatment of this disease are necessary means to improve the survival rate of lung cancer patients and reduce their mortality. The introduction of computer-aided diagnosis technology can easily, quickly, and accurately identify the lung nodule area as an imaging feature of early lung cancer for the clinical diagnosis of lung cancer and is helpful for the quantitative analysis of the characteristics of lung nodules and is useful for distinguishing benign and malignant lung nodules. Growth provides an objective diagnostic reference standard. This paper studies ITK and VTK toolkits and builds a system platform with MFC. By studying the process of doctors diagnosing lung nodules, the whole system is divided into seven modules: suspected lung shadow detection, image display and image annotation, and interaction. The system passes through the entire lung nodule auxiliary diagnosis process and obtains the number of nodules, the number of malignant nodules, and the number of false positives in each set of lung CT images to analyze the performance of the auxiliary diagnosis system. In this paper, a lung region segmentation method is proposed, which makes use of the obvious differences between the lung parenchyma and other human tissues connected with it, as well as the position relationship and shape characteristics of each human tissue in the image. Experiments are carried out to solve the problems of lung boundary, inaccurate segmentation of lung wall, and depression caused by noise and pleural nodule adhesion. Experiments show that there are 2316 CT images in 8 sets of images of different patients, and the number of nodules is 56. A total of 49 nodules were detected by the system, 7 were missed, and the detection rate was 87.5%. A total of 64 false-positive nodules were detected, with an average of 8 per set of images. This shows that the system is effective for CT images of different devices, pixel pitch, and slice pitch and has high sensitivity, which can provide doctors with good advice.

The Image Annotation Refinement in Embedding Feature Space based on Mutual Information

The ever-increasing size of images has made automatic image annotation one of the most important tasks in the fields of machine learning and computer vision. Despite continuous efforts in inventing new annotation algorithms and new models, results of the state-of-the-art image annotation methods are often unsatisfactory. In this paper, to further improve annotation refinement performance, a novel approach based on weighted mutual information to automatically refine the original annotations of images is proposed. Unlike the traditional refinement model using only visual feature, the proposed model use semantic embedding to properly map labels and visual features to a meaningful semantic space. To accurately measure the relevance between the particular image and its original annotations, the proposed model utilize all available information including image-to-image, label-to-label and image-to-label. Experimental results conducted on three typical datasets show not only the validity of the refinement, but also the superiority of the proposed algorithm over existing ones. The improvement largely benefits from our proposed mutual information method and utilizing all available information.

CAPTCHA for crowdsourced image annotation: directions and efficiency analysis

Purpose Image annotation plays an important role in image retrieval process, especially when it comes to content-based image retrieval. In order to compensate the intrinsic weakness of machines in performing cognitive task of (human-like) image annotation, leveraging humans’ knowledge and abilities in the form of crowdsourcing-based annotation have gained momentum. Among various approaches for this purpose, an innovative one is integrating the annotation process into the CAPTCHA workflow. In this paper, the current state of the research works in the field and experimental efficiency analysis of this approach are investigated.Design/methodology/approach At first, and with the aim of presenting a current state report of research studies in the field, a comprehensive literature review is provided. Then, several experiments and statistical analyses are conducted to investigate how CAPTCHA-based image annotation is reliable, accurate and efficient.Findings In addition to study of current trends and best practices for CAPTCHA-based image annotation, the experimental results demonstrated that despite some intrinsic limitations on leveraging the CAPTCHA as a crowdsourcing platform, when the challenge, i.e. annotation task, is selected and designed appropriately, the efficiency of CAPTCHA-based image annotation can outperform traditional approaches. Nonetheless, there are several design considerations that should be taken into account when the CAPTCHA is used as an image annotation platform.Originality/value To the best of the authors’ knowledge, this is the first study to analyze different aspects of the titular topic through exploration of the literature and experimental investigation. Therefore, it is anticipated that the outcomes of this study can draw a roadmap for not only CAPTCHA-based image annotation but also CAPTCHA-mediated crowdsourcing and even image annotation.

Automated Image Analysis to Map the Extent of Deep-Sea Mining Plumes Reveals a Larger Impacted Area Compared to Manual Analysis

Deep sea mining for poly-metallic nodules impacts the environment in many ways. A key potential hazard is the creation of a sediment plume from resuspending sediment during seabed mining. The resuspended matter disperses with currents but eventually resettles on the seabed. Resettling causes a blanketing of the seafloor environment, potentially causing harm to in-, epi- and hyperbenthic communities with possible cascading effects into food webs of deep sea habitats. Mapping the extent of such blanketing is thus an important factor in quantifying potential impacts of deep-sea mining.One technology that can assess seabed blanketing is optical imaging with cameras at square-kilometre scale. To efficiently analyse the resulting Terabytes of image data with minimized bias, automated image analysis is required. Moreover, effective quantitative monitoring of the blanketing requires ground truthing of the image data. Here, we present results from a camera-based monitoring of a deep-sea mining simulation combined with automated image analysis using the CoMoNoD method and low-cost seabed sediment traps for quantification of the blanketing thickness. We found that the impacted area was about 50 percent larger than previously determined by manual image annotation.

Single-trial detection of EEG error-related potentials in serial visual presentation paradigm

When the outcome of an event is not the same as expected, the cognitive state that monitors performance elicits a time-locked brain response termed as Error-Related Potential (ErrP). Objective – In the existing work, ErrP is not recorded when there is a disassociation between an object and its description. The objective of this work is to propose a Serial Visual Presentation (SVP) experimental paradigm to record ErrP when an image and its label are disassociated. Additionally, this work aims to propose a novel method for detecting ErrP on a single-trial basis. Method – The method followed in this work includes designing of SVP paradigm in which labeled images from six categories (bike, car, flower, fruit, cat, and dog) are presented serially. In this work, a text (visual) or an audio clip describing the image in one word is presented as the label. Further, the ErrP is detected on a single-trial basis using novel electrode-averaged features. Results - The ErrP data recorded from 11 subjects’ have consistent characteristics compared to existing ErrP literature. Detection of ErrP on a single-trial basis is carried out using a novel feature extraction method on two type labeling types separately. The best average classification accuracy achieved is 69.09±4.70% and 63.33±4.56% for the audio and visual type of labeling the image, respectively. The proposed feature extraction method achieved higher classification accuracy when compared with two existing feature extraction methods. Significance - The significance of this work is that it can be used as a Brain-Computer Interface (BCI) system for quantitative evaluation and treatment of mild cognitive impairment. This work can also find non-clinical BCI applications such as image annotation.

A localization strategy combined with transfer learning for image annotation

This study aims to solve the overfitting problem caused by insufficient labeled images in the automatic image annotation field. We propose a transfer learning model called CNN-2L that incorporates the label localization strategy described in this study. The model consists of an InceptionV3 network pretrained on the ImageNet dataset and a label localization algorithm. First, the pretrained InceptionV3 network extracts features from the target dataset that are used to train a specific classifier and fine-tune the entire network to obtain an optimal model. Then, the obtained model is used to derive the probabilities of the predicted labels. For this purpose, we introduce a squeeze and excitation (SE) module into the network architecture that augments the useful feature information, inhibits useless feature information, and conducts feature reweighting. Next, we perform label localization to obtain the label probabilities and determine the final label set for each image. During this process, the number of labels must be determined. The optimal K value is obtained experimentally and used to determine the number of predicted labels, thereby solving the empty label set problem that occurs when the predicted label values of images are below a fixed threshold. Experiments on the Corel5k multilabel image dataset verify that CNN-2L improves the labeling precision by 18% and 15% compared with the traditional multiple-Bernoulli relevance model (MBRM) and joint equal contribution (JEC) algorithms, respectively, and it improves the recall by 6% compared with JEC. Additionally, it improves the precision by 20% and 11% compared with the deep learning methods Weight-KNN and adaptive hypergraph learning (AHL), respectively. Although CNN-2L fails to improve the recall compared with the semantic extension model (SEM), it improves the comprehensive index of the F1 value by 1%. The experimental results reveal that the proposed transfer learning model based on a label localization strategy is effective for automatic image annotation and substantially boosts the multilabel image annotation performance.