Expected Reciprocal Rank

2017 ◽  
pp. 1-2
Author(s):  
Tetsuya Sakai
Keyword(s):  
Reciprocal Rank

Recommending Relevant Tutorial Fragments for API-Related Natural Language Questions

2021 ◽  
Vol 31 (09) ◽  
pp. 1251-1275
Author(s):  
Di Wu ◽  
Xiao-Yuan Jing ◽  
Haowen Chen ◽  
Xiaohui Kong ◽  
Jifeng Xuan
Keyword(s):  
Natural Language ◽  
State Of The Art ◽  
Metric Learning ◽  
Application Programming Interface ◽  
Manual Annotation ◽  
Candidate List ◽  
Novel Approach ◽  
Application Programming ◽  
Programming Interface ◽  
Reciprocal Rank

Application Programming Interface (API) tutorial is an important API learning resource. To help developers learn APIs, an API tutorial is often split into a number of consecutive units that describe the same topic (i.e. tutorial fragment). We regard a tutorial fragment explaining an API as a relevant fragment of the API. Automatically recommending relevant tutorial fragments can help developers learn how to use an API. However, existing approaches often employ supervised or unsupervised manner to recommend relevant fragments, which suffers from much manual annotation effort or inaccurate recommended results. Furthermore, these approaches only support developers to input exact API names. In practice, developers often do not know which APIs to use so that they are more likely to use natural language to describe API-related questions. In this paper, we propose a novel approach, called Tutorial Fragment Recommendation (TuFraRec), to effectively recommend relevant tutorial fragments for API-related natural language questions, without much manual annotation effort. For an API tutorial, we split it into fragments and extract APIs from each fragment to build API-fragment pairs. Given a question, TuFraRec first generates several clarification APIs that are related to the question. We use clarification APIs and API-fragment pairs to construct candidate API-fragment pairs. Then, we design a semi-supervised metric learning (SML)-based model to find relevant API-fragment pairs from the candidate list, which can work well with a few labeled API-fragment pairs and a large number of unlabeled API-fragment pairs. In this way, the manual effort for labeling the relevance of API-fragment pairs can be reduced. Finally, we sort and recommend relevant API-fragment pairs based on the recommended strategy. We evaluate TuFraRec on 200 API-related natural language questions and two public tutorial datasets (Java and Android). The results demonstrate that on average TuFraRec improves NDCG@5 by 0.06 and 0.09, and improves Mean Reciprocal Rank (MRR) by 0.07 and 0.09 on two tutorial datasets as compared with the state-of-the-art approach.

Deep Generative Recommendation with Maximizing Reciprocal Rank

2020 ◽  
pp. 123-130
Author(s):  
Xiaoyi Sun ◽  
Huafeng Liu ◽  
Liping Jing ◽  
Jian Yu
Keyword(s):  
Reciprocal Rank

scholarly journals Analyzing Linguistic Features for Answer Re-Ranking of Why-Questions

2022 ◽  
Vol 24 (3) ◽  
pp. 1-16
Author(s):  
Manvi Breja ◽  
Sanjay Kumar Jain
Keyword(s):  
Deep Learning ◽  
Linguistic Features ◽  
Answer Extraction ◽  
Feature Similarity ◽  
Question And Answer ◽  
Why Questions ◽  
Learning Frameworks ◽  
Reciprocal Rank ◽  
Features Importance

Why-type non-factoid questions are ambiguous and involve variations in their answers. A challenge in returning one appropriate answer to user requires the process of appropriate answer extraction, re-ranking and validation. There are cases where the need is to understand the meaning and context of a document rather than finding exact words involved in question. The paper addresses this problem by exploring lexico-syntactic, semantic and contextual query-dependent features, some of which are based on deep learning frameworks to depict the probability of answer candidate being relevant for the question. The features are weighted by the score returned by ensemble ExtraTreesClassifier according to features importance. An answer re-ranker model is implemented that finds the highest ranked answer comprising largest value of feature similarity between question and answer candidate and thus achieving 0.64 Mean Reciprocal Rank (MRR). Further, answer is validated by matching the answer type of answer candidate and returns the highest ranked answer candidate with matched answer type to a user.

scholarly journals Consumer health information and question answering: helping consumers find answers to their health-related information needs

2019 ◽  
Vol 27 (2) ◽  
pp. 194-201 ◽  
Author(s):  
Dina Demner-Fushman ◽  
Yassine Mrabet ◽  
Asma Ben Abacha
Keyword(s):  
Information Retrieval ◽  
Deep Learning ◽  
Real Life ◽  
Mean Average Precision ◽  
Consumer Health ◽  
National Library ◽  
Average Precision ◽  
Related Information ◽  
Health Related ◽  
Reciprocal Rank

Abstract Objective Consumers increasingly turn to the internet in search of health-related information; and they want their questions answered with short and precise passages, rather than needing to analyze lists of relevant documents returned by search engines and reading each document to find an answer. We aim to answer consumer health questions with information from reliable sources. Materials and Methods We combine knowledge-based, traditional machine and deep learning approaches to understand consumers’ questions and select the best answers from consumer-oriented sources. We evaluate the end-to-end system and its components on simple questions generated in a pilot development of MedlinePlus Alexa skill, as well as the short and long real-life questions submitted to the National Library of Medicine by consumers. Results Our system achieves 78.7% mean average precision and 87.9% mean reciprocal rank on simple Alexa questions, and 44.5% mean average precision and 51.6% mean reciprocal rank on real-life questions submitted by National Library of Medicine consumers. Discussion The ensemble of deep learning, domain knowledge, and traditional approaches recognizes question type and focus well in the simple questions, but it leaves room for improvement on the real-life consumers’ questions. Information retrieval approaches alone are sufficient for finding answers to simple Alexa questions. Answering real-life questions, however, benefits from a combination of information retrieval and inference approaches. Conclusion A pilot practical implementation of research needed to help consumers find reliable answers to their health-related questions demonstrates that for most questions the reliable answers exist and can be found automatically with acceptable accuracy.

scholarly journals LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts

2019 ◽  
Vol 33 ◽  
pp. 6810-6817 ◽  
Author(s):  
Shuming Ma ◽  
Lei Cui ◽  
Damai Dai ◽  
Furu Wei ◽  
Xu Sun
Keyword(s):  
Large Scale ◽  
Online Video ◽  
Neural Models ◽  
Evaluation Protocol ◽  
Likelihood Score ◽  
Dynamic Vision ◽  
Log Likelihood ◽  
The Right ◽  
Vision And Language ◽  
Reciprocal Rank

We introduce the task of automatic live commenting. Live commenting, which is also called “video barrage”, is an emerging feature on online video sites that allows real-time comments from viewers to fly across the screen like bullets or roll at the right side of the screen. The live comments are a mixture of opinions for the video and the chit chats with other comments. Automatic live commenting requires AI agents to comprehend the videos and interact with human viewers who also make the comments, so it is a good testbed of an AI agent’s ability to deal with both dynamic vision and language. In this work, we construct a large-scale live comment dataset with 2,361 videos and 895,929 live comments. Then, we introduce two neural models to generate live comments based on the visual and textual contexts, which achieve better performance than previous neural baselines such as the sequence-to-sequence model. Finally, we provide a retrieval-based evaluation protocol for automatic live commenting where the model is asked to sort a set of candidate comments based on the log-likelihood score, and evaluated on metrics such as mean-reciprocal-rank. Putting it all together, we demonstrate the first “LiveBot”. The datasets and the codes can be found at https://github.com/lancopku/livebot.

scholarly journals RankEval: Open Tool for Evaluation of Machine-Learned Ranking

2013 ◽  
Vol 100 (1) ◽  
pp. 63-72 ◽  
Author(s):  
Eleftherios Avramidis
Keyword(s):  
Information Retrieval ◽  
Open Source ◽  
Machine Translation ◽  
Research Field ◽  
Reward Systems ◽  
Quality Estimation ◽  
Open Source Tool ◽  
Statistical Measures ◽  
The One ◽  
Reciprocal Rank

Abstract Recent research and applications for evaluation and quality estimation of Machine Translation require statistical measures for comparing machine-predicted ranking against gold sets annotated by humans. Additional to the existing practice of measuring segment-level correlation with Kendall tau, we propose using ranking metrics from the research field of Information Retrieval such as Mean Reciprocal Rank, Normalized Discounted Cumulative Gain and Expected Reciprocal Rank. These reward systems that predict correctly the highest ranked items than the one of lower ones. We present an open source tool ”RANKEVAL” providing implementation of these metrics. It can be either run independently as a script supporting common formats or can be imported to any Python application.

Sign in / Sign up

Export Citation Format

Share Document