Do You Ever Get Off Track in a Conversation? The Conversational System’s Anatomy and Evaluation Metrics

Conversational systems are now applicable to almost every business domain. Evaluation is an important step in the creation of dialog systems so that they may be readily tested and prototyped. There is no universally agreed upon metric for evaluating all dialog systems. Human evaluation, which is not computerized, is now the most effective and complete evaluation approach. Data gathering and analysis are evaluation activities that need human intervention. In this work, we address the many types of dialog systems and the assessment methods that may be used with them. The benefits and drawbacks of each sort of evaluation approach are also explored, which could better help us understand the expectations associated with developing an automated evaluation system. The objective of this study is to investigate conversational agents, their design approaches and evaluation metrics. This approach can help us to better understand the overall process of dialog system development, and future possibilities to enhance user experience. Because human assessment is costly and time consuming, we emphasize the need of having a generally recognized and automated evaluation model for conversational systems, which may significantly minimize the amount of time required for analysis.

Evaluating the Impact of Integrating Similar Translations into Neural Machine Translation

Previous research has shown that simple methods of augmenting machine translation training data and input sentences with translations of similar sentences (or fuzzy matches), retrieved from a translation memory or bilingual corpus, lead to considerable improvements in translation quality, as assessed by a limited set of automatic evaluation metrics. In this study, we extend this evaluation by calculating a wider range of automated quality metrics that tap into different aspects of translation quality and by performing manual MT error analysis. Moreover, we investigate in more detail how fuzzy matches influence translations and where potential quality improvements could still be made by carrying out a series of quantitative analyses that focus on different characteristics of the retrieved fuzzy matches. The automated evaluation shows that the quality of NFR translations is higher than the NMT baseline in terms of all metrics. However, the manual error analysis did not reveal a difference between the two systems in terms of total number of translation errors; yet, different profiles emerged when considering the types of errors made. Finally, in our analysis of how fuzzy matches influence NFR translations, we identified a number of features that could be used to improve the selection of fuzzy matches for NFR data augmentation.

Unified Website Accessibility Assessment Framework

In this chapter, a unified web accessibility assessment (UWAA) framework and its software has been proposed. UWAA framework was developed by considering Web Content Accessibility Guideline 2.0 to evaluate accessibility of web sites by integrating more than one evaluation approach. Achecker tool as an automated evaluation approach and barrier walkthrough (BW) as an expert-based evaluation approach were integrated in the UWAA framework. The framework also provides suggestions to recover from the problems determined to the evaluators. The websites of three universities were evaluated to determine the framework's accuracy and consistency. It was revealed that the results obtained from automated and expert-based evaluation methods were consistent and complementary with each other. Furthermore, it has been demonstrated that problems which cannot be determined by an automated tool but which can be detected by an expert can be identified by BW method.

A Study on Text Rank Algorithm for Automatic Text Summarization and Question Generation

This paper proposes a new rule-based approach to automated question generation. The proposed approach focuses on the analysis of both sentence syntax and semantic structure. The design and implementation of the proposed approach is also described in detail. Although the primary purpose of a design system is to generate query from sentences, automated evaluation results show that it can also perform great when reading comprehension datasets that focus on question output from paragraphs. With regard to human evaluation, the designed system performs better than all other systems and generates the most natural (human-like) questions. We present a fresh approach to automatic question generation that significantly increases the percentage of acceptable questions compared to prior state-of-the-art systems. In our system, we will take data from various sources for a particular topic and summarize it for the convenience of the people, so that they don't have to go through so multiple sites for relevant data.

Diagnostic method and device for evaluating and forecasting the technical condition of farm machinery in operation

The paper discusses methods and ways to diagnose the technical condition of agricultural machines and harvesters, existing practices, and approaches to get reliable data on the current health of the machinery used. The device for assessing and predicting machines’ technical condition includes software and technical means developed with virtual technologies to measure diagnostic parameters of the machinery. The main device elements are digital sensors with physical modifiers (pressure, temperature, medium composition and motion sensors, a-d converters with signal amplifiers), software to configure data gathering, and output to conduct analyses and produce recommendations. The core of the present approach is the technology of virtual prediction of breakdowns by changes in the technical condition parameters. It is based on modular devices, software with an interface that collects and processes data and provides a complete set of failure diagnostics and forecasting. The given method based on a device operating in the information and communication network increases farm machinery’s performance. Furthermore, it reduces operating costs due to the prevention of expensive breakdowns, individual forecasting, and scheduled maintenance of machines in operation. The approach under consideration was applied in the laboratory of digital engineering technologies of the Bashkir State Agrarian University Republic of Bashkortostan of the Russian Federation. The given work is aimed to boost the efficiency of the farm machinery diagnostics and maintenance system by applying a virtual breakdown prediction technology to conduct an automated evaluation, registration, and analysis of a machine’s condition. It can be achieved by developing software and technical means to register data and their structure systematization.

Autonomous Directional Drilling Planning and Execution Using an Industry 4.0 Platform

Today, directional drilling is considered a mix between art and science only performed by experts in the field. In this paper, we present an autonomous directional drilling framework using an industry 4.0 platform that is built on intelligent planning and execution capabilities and is supported by surface and downhole automation technologies to achieve consistently performing directional drilling operations accessible for easy remote operations. Intelligent planning builds on standard planning activities that are needed for directional drilling applications and advances them with rich data pipelines that feed predictive and prescriptive machine-learning (ML) models; this enables more accurate BHA tendencies, operating parameters, and trajectory plans that ultimately reduce executional risk and uncertainty. Intelligent execution provides technologies that facilitate decision-making activities, whether they be from the wellsite or town, by leveraging the digital-drilling program that is generated from the intelligent planning activities. The program connects planning expectations, real-time execution data from the surface and downhole equipment, and generates insights from data analytics, physics-based simulations, and offset analysis to achieve consistent directional drilling performance that is transparent to all stakeholders. This new framework enables a self-steering BHA for directional drilling operations. The workflow involves an automated evaluation of the current bit position with respect to the initial plan, automated evaluation of the maximum dogleg capability of the BHA, and the capability to examine the health of the BHA tools and, if needed, an automated re-planning of an optimized working plan. This is accomplished on a system level with interdependencies on the different elements that make up the complete workflow. This new autonomous directional drilling framework will minimize operational risk and cost-per-foot drilled; maximize performance, procedural adherence, and establish consistent results across fields, rigs, and trajectories while enabling modern remote operations.

SE-CapsNet: Automated evaluation of plant disease severity based on feature extraction through Squeeze and Excitation (SE) networks and Capsule networks

Diseases in plants harm the quantity of the overall food production as well as the quality of the yield. Early detection, diagnosis and treatment can greatly reduce losses, both economic and ecological. Intuitively, reduction in the use of agrochemicals due to timely detection of the disease, would greatly help in mitigating the environmental impact. In this paper, the authors have proposed an improved feature computation approach based on Squeeze and Excitation (SE) Networks, before processing by the original Capsule networks (CapsNet) for classification, for estimating the disease severity in plants. Two SE networks, one based on AlexNet and another on ResNet have been combined with Capsule networks. Leaf images for the devastating Late Blight disease occurring in the Tomato crop have been utilized from the PlantVillage dataset. The images, divided into four severity stages i.e. healthy, early, middle and end, are downscaled, enhanced and given as input to the SE networks. The feature maps generated from the two networks are separately given as input to the Capsule Network for classification and their performances are compared with the original CapsNet, on two image sizes 32X32 and 64X64. SE-Alex-CapsNet achieves the highest accuracy of 92.1% and SE-Res CapsNet achieves the highest accuracy of 93.75% with 64X64 image size, as compared to CapsNet that results in 85.53% accuracy. The classification accuracies of six state-of-the-art CNN models namely AlexNet, SqueezeNet, ResNet50, VGG16, VGG19 and Inception V3 are also presented for comparison purposes. Accuracy as well as precision, recall, F1-score, validation loss etc. measures have been recorded and compared. The findings have been validated by implementing the proposed approaches with another dataset, achieving similar resultant accuracy measures. The implementation was also accomplished with datasets after noise addition in six different variations, to verify the robustness of the proposed model. Based on the performances, the proposed techniques can be exploited for disease severity assessment in other crops as well and can be extended to other areas of applications such as plant species classification, weed identification etc. In addition to improved performance, with reduced image size, the proposed methodology can be utilized to create a mobile application requiring low processing capabilities, to be installed on reasonably priced smartphones for practical usage by farmers.

Highly accurate differentiation of bone marrow cell morphologies using deep neural networks on a large image data set

Biomedical applications of deep learning algorithms rely on large expert annotated data sets. The classification of bone marrow (BM) cell cytomorphology, an important cornerstone of hematological diagnosis, is still done manually thousands of times every day because of a lack of data sets and trained models. We applied convolutional neural networks (CNNs) to a large data set of 171 374 microscopic cytological images taken from BM smears from 945 patients diagnosed with a variety of hematological diseases. The data set is the largest expert-annotated pool of BM cytology images available in the literature. It allows us to train high-quality classifiers of leukocyte cytomorphology that identify a wide range of diagnostically relevant cell species with high precision and recall. Our CNNs outcompete previous feature-based approaches and provide a proof-of-concept for the classification problem of single BM cells. This study is a step toward automated evaluation of BM cell morphology using state-of-the-art image-classification algorithms. The underlying data set represents an educational resource, as well as a reference for future artificial intelligence–based approaches to BM cytomorphology.