Cheap IR evaluation

2020 ◽  
Vol 54 (2) ◽  
pp. 1-2
Author(s):  
Kevin Roitero
Keyword(s):  
Statistical Power ◽  
Information Needs ◽  
Web Search ◽  
State Of The Art ◽  
Extensive Study ◽  
Test Collection ◽  
Test Collections ◽  
Fine Grained ◽  
Retrieval Systems ◽  
Ranked List

To evaluate Information Retrieval (IR) effectiveness, a possible approach is to use test collections, which are composed of a collection of documents, a set of description of information needs (called topics), and a set of relevant documents to each topic. Test collections are modelled in a competition scenario: for example, in the well known TREC initiative, participants run their own retrieval systems over a set of topics and they provide a ranked list of retrieved documents; some of the retrieved documents (usually the first ranked) constitute the so called pool, and their relevance is evaluated by human assessors; the document list is then used to compute effectiveness metrics and rank the participant systems. Private Web Search companies also run their in-house evaluation exercises; although the details are mostly unknown, and the aims are somehow different, the overall approach shares several issues with the test collection approach. The aim of this work is to: (i) develop and improve some state-of-the-art work on the evaluation of IR effectiveness while saving resources, and (ii) propose a novel, more principled and engineered, overall approach to test collection based effectiveness evaluation. In this thesis we focus on three main directions: the first part details the usage of few topics (i.e., information needs) in retrieval evaluation and shows an extensive study detailing the effect of using fewer topics for retrieval evaluation in terms of number of topics, topics subsets, and statistical power. The second part of this thesis discusses the evaluation without relevance judgements, reproducing, extending, and generalizing state-of-the-art methods and investigating their combinations by means of data fusion techniques and machine learning. Finally, the third part uses crowdsourcing to gather relevance labels, and in particular shows the effect of using fine grained judgement scales; furthermore, explores methods to transform judgements between different relevance scales. Awarded by: University of Udine, Udine, Italy on 19 March 2020. Supervised by: Professor Stefano Mizzaro. Available at: https://kevinroitero.com/resources/kr-phd-thesis.pdf.

scholarly journals A Test Collection for Dataset Retrieval in Biodiversity Research

2021 ◽  
Vol 7 ◽  
Author(s):  
Felicitas Löffler ◽  
Andreas Schuldt ◽  
Birgitta König-Ries ◽  
Helge Bruelheide ◽  
Friederike Klan
Keyword(s):  
Information Needs ◽  
Research Field ◽  
Test Collection ◽  
Test Collections ◽  
Biodiversity Research ◽  
Retrieval Systems ◽  
Binary Relevance ◽  
Increasing Demand ◽  
World Information ◽  
Bef China

Searching for scientific datasets is a prominent task in scholars' daily research practice. A variety of data publishers, archives and data portals offer search applications that allow the discovery of datasets. The evaluation of such dataset retrieval systems requires proper test collections, including questions that reflect real world information needs of scholars, a set of datasets and human judgements assessing the relevance of the datasets to the questions in the benchmark corpus. Unfortunately, only very few test collections exist for a dataset search. In this paper, we introduce the BEF-China test collection, the very first test collection for dataset retrieval in biodiversity research, a research field with an increasing demand in data discovery services. The test collection consists of 14 questions, a corpus of 372 datasets from the BEF-China project and binary relevance judgements provided by a biodiversity expert.

scholarly journals State-of-the-Art Review on Relevance of Genetic Algorithm to Internet Web Search

2012 ◽  
Vol 2012 ◽  
pp. 1-7 ◽  
Author(s):  
Kehinde Agbele ◽  
Ademola Adesina ◽  
Daniel Ekong ◽  
Oluwafemi Ayangbekun
Keyword(s):  
Genetic Algorithm ◽  
Genetic Algorithms ◽  
Information Retrieval ◽  
Information Needs ◽  
Web Search ◽  
State Of The Art ◽  
Retrieval Systems ◽  
Online Databases ◽  
Information Retrieval Systems ◽  
Survival Of The Fittest

People use search engines to find information they desire with the aim that their information needs will be met. Information retrieval (IR) is a field that is concerned primarily with the searching and retrieving of information in the documents and also searching the search engine, online databases, and Internet. Genetic algorithms (GAs) are robust, efficient, and optimizated methods in a wide area of search problems motivated by Darwin’s principles of natural selection and survival of the fittest. This paper describes information retrieval systems (IRS) components. This paper looks at how GAs can be applied in the field of IR and specifically the relevance of genetic algorithms to internet web search. Finally, from the proposals surveyed it turns out that GA is applied to diverse problem fields of internet web search.

scholarly journals Effective collection construction for information retrieval evaluation and optimization

2020 ◽  
Vol 54 (2) ◽  
pp. 1-2
Author(s):  
Dan Li
Keyword(s):  
Information Retrieval ◽  
Configuration Space ◽  
Sampling Method ◽  
Bayesian Optimization ◽  
Test Collection ◽  
Test Collections ◽  
Information Retrieval Evaluation ◽  
Retrieval Systems ◽  
Ranking Model ◽  
Relevance Assessments

The availability of test collections in Cranfield paradigm has significantly benefited the development of models, methods and tools in information retrieval. Such test collections typically consist of a set of topics, a document collection and a set of relevance assessments. Constructing these test collections requires effort of various perspectives such as topic selection, document selection, relevance assessment, and relevance label aggregation etc. The work in the thesis provides a fundamental way of constructing and utilizing test collections in information retrieval in an effective, efficient and reliable manner. To that end, we have focused on four aspects. We first study the document selection issue when building test collections. We devise an active sampling method for efficient large-scale evaluation [Li and Kanoulas, 2017]. Different from past sampling-based approaches, we account for the fact that some systems are of higher quality than others, and we design the sampling distribution to over-sample documents from these systems. At the same time, the estimated evaluation measures are unbiased, and assessments can be used to evaluate new, novel systems without introducing any systematic error. Then a natural further step is determining when to stop the document selection and assessment procedure. This is an important but understudied problem in the construction of test collections. We consider both the gain of identifying relevant documents and the cost of assessing documents as the optimization goals. We handle the problem under the continuous active learning framework by jointly training a ranking model to rank documents, and estimating the total number of relevant documents in the collection using a "greedy" sampling method [Li and Kanoulas, 2020]. The next stage of constructing a test collection is assessing relevance. We study how to denoise relevance assessments by aggregating from multiple crowd annotation sources to obtain high-quality relevance assessments. This helps to boost the quality of relevance assessments acquired in a crowdsourcing manner. We assume a Gaussian process prior on query-document pairs to model their correlation. The proposed model shows good performance in terms of interring true relevance labels. Besides, it allows predicting relevance labels for new tasks that has no crowd annotations, which is a new functionality of CrowdGP. Ablation studies demonstrate that the effectiveness is attributed to the modelling of task correlation based on the axillary information of tasks and the prior relevance information of documents to queries. After a test collection is constructed, it can be used to either evaluate retrieval systems or train a ranking model. We propose to use it to optimize the configuration of retrieval systems. We use Bayesian optimization approach to model the effect of a δ -step in the configuration space to the effectiveness of the retrieval system, by suggesting to use different similarity functions (covariance functions) for continuous and categorical values, and examine their ability to effectively and efficiently guide the search in the configuration space [Li and Kanoulas, 2018]. Beyond the algorithmic and empirical contributions, work done as part of this thesis also contributed to the research community as the CLEF Technology Assisted Reviews in Empirical Medicine Tracks in 2017, 2018, and 2019 [Kanoulas et al., 2017, 2018, 2019]. Awarded by: University of Amsterdam, Amsterdam, The Netherlands. Supervised by: Evangelos Kanoulas. Available at: https://dare.uva.nl/search?identifier=3438a2b6-9271-4f2c-add5-3c811cc48d42.

TIPS: Time-aware Personalised Semantic-based query auto-completion

2020 ◽  
pp. 016555152096869
Author(s):  
Saedeh Tahery ◽  
Saeed Farzi
Keyword(s):  
Search Engines ◽  
Information Needs ◽  
Semantic Information ◽  
State Of The Art ◽  
Language Model ◽  
Experimental Studies ◽  
Short Length ◽  
Context Aware ◽  
Ranked List ◽  
Time Aware

With the rapid growth of the Internet, search engines play vital roles in meeting the users’ information needs. However, formulating information needs to simple queries for canonical users is a problem yet. Therefore, query auto-completion, which is one of the most important characteristics of the search engines, is leveraged to provide a ranked list of queries matching the user’s entered prefix. Although query auto-completion utilises useful information provided by search engine logs, time-, semantic- and context-aware features are still important resources of extra knowledge. Specifically, in this study, a hybrid query auto-completion system called TIPS ( Time-aware Personalised Semantic-based query auto-completion) is introduced to combine the well-known systems performing based on popularity and neural language model. Furthermore, this system is supplemented by time-aware features that blend both context and semantic information in a collaborative manner. Experimental studies on the standard AOL dataset are conducted to compare our proposed system with state-of-the-art methods, that is, FactorCell, ConcatCell and Unadapted. The results illustrate the significant superiorities of TIPS in terms of mean reciprocal rank (MRR), especially for short-length prefixes.

scholarly journals Improving the Retrieval of Arabic Web Search Results Using Enhanced k-Means Clustering Algorithm

Entropy ◽  
2021 ◽  
Vol 23 (4) ◽  
pp. 449
Author(s):  
Amjad F. Alsuhaim ◽  
Aqil M. Azmi ◽  
Muhammad Hussain
Keyword(s):  
Information Retrieval ◽  
Execution Time ◽  
Clustering Algorithm ◽  
Web Search ◽  
Writing Style ◽  
Search Query ◽  
Search Results ◽  
Retrieval Systems ◽  
Information Retrieval Systems ◽  
Ranked List

Traditional information retrieval systems return a ranked list of results to a user’s query. This list is often long, and the user cannot explore all the results retrieved. It is also ineffective for a highly ambiguous language such as Arabic. The modern writing style of Arabic excludes the diacritical marking, without which Arabic words become ambiguous. For a search query, the user has to skim over the document to infer if the word has the same meaning they are after, which is a time-consuming task. It is hoped that clustering the retrieved documents will collate documents into clear and meaningful groups. In this paper, we use an enhanced k-means clustering algorithm, which yields a faster clustering time than the regular k-means. The algorithm uses the distance calculated from previous iterations to minimize the number of distance calculations. We propose a system to cluster Arabic search results using the enhanced k-means algorithm, labeling each cluster with the most frequent word in the cluster. This system will help Arabic web users identify each cluster’s topic and go directly to the required cluster. Experimentally, the enhanced k-means algorithm reduced the execution time by 60% for the stemmed dataset and 47% for the non-stemmed dataset when compared to the regular k-means, while slightly improving the purity.

scholarly journals Semantic Table Retrieval Using Keyword and Table Queries

2021 ◽  
Vol 15 (3) ◽  
pp. 1-33
Author(s):  
Shuo Zhang ◽  
Krisztian Balog
Keyword(s):  
Information Needs ◽  
State Of The Art ◽  
Similarity Measures ◽  
Semantic Representations ◽  
Information Need ◽  
Keyword Query ◽  
Semantic Spaces ◽  
Amount Of Knowledge ◽  
Vector Representations ◽  
Ranked List

Tables on the Web contain a vast amount of knowledge in a structured form. To tap into this valuable resource, we address the problem of table retrieval: answering an information need with a ranked list of tables. We investigate this problem in two different variants, based on how the information need is expressed: as a keyword query or as an existing table (“query-by-table”). The main novel contribution of this work is a semantic table retrieval framework for matching information needs (keyword or table queries) against tables. Specifically, we (i) represent queries and tables in multiple semantic spaces (both discrete sparse and continuous dense vector representations) and (ii) introduce various similarity measures for matching those semantic representations. We consider all possible combinations of semantic representations and similarity measures and use these as features in a supervised learning model. Using two purpose-built test collections based on Wikipedia tables, we demonstrate significant and substantial improvements over state-of-the-art baselines.

scholarly journals Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders

2021 ◽  
Vol 17 (4) ◽  
pp. 1-23
Author(s):  
Nicola Messina ◽  
Giuseppe Amato ◽  
Andrea Esuli ◽  
Fabrizio Falchi ◽  
Claudio Gennaro ◽  
...  
Keyword(s):  
Information Retrieval ◽  
Large Scale ◽  
State Of The Art ◽  
Retrieval Task ◽  
Fine Grained ◽  
Retrieval Systems ◽  
Sentence Retrieval ◽  
Novel Approach ◽  
Sentence Level ◽  
Sentence Matching

Despite the evolution of deep-learning-based visual-textual processing systems, precise multi-modal matching remains a challenging task. In this work, we tackle the task of cross-modal retrieval through image-sentence matching based on word-region alignments, using supervision only at the global image-sentence level. Specifically, we present a novel approach called Transformer Encoder Reasoning and Alignment Network (TERAN). TERAN enforces a fine-grained match between the underlying components of images and sentences (i.e., image regions and words, respectively) to preserve the informative richness of both modalities. TERAN obtains state-of-the-art results on the image retrieval task on both MS-COCO and Flickr30k datasets. Moreover, on MS-COCO, it also outperforms current approaches on the sentence retrieval task. Focusing on scalable cross-modal information retrieval, TERAN is designed to keep the visual and textual data pipelines well separated. Cross-attention links invalidate any chance to separately extract visual and textual features needed for the online search and the offline indexing steps in large-scale retrieval systems. In this respect, TERAN merges the information from the two domains only during the final alignment phase, immediately before the loss computation. We argue that the fine-grained alignments produced by TERAN pave the way toward the research for effective and efficient methods for large-scale cross-modal information retrieval. We compare the effectiveness of our approach against relevant state-of-the-art methods. On the MS-COCO 1K test set, we obtain an improvement of 5.7% and 3.5% respectively on the image and the sentence retrieval tasks on the Recall@1 metric. The code used for the experiments is publicly available on GitHub at https://github.com/mesnico/TERAN .

Behavioral Genetics: Concepts for Research and Practice in Language Development and Disorders

1995 ◽  
Vol 38 (5) ◽  
pp. 1126-1142 ◽  
Author(s):  
Jeffrey W. Gilger
Keyword(s):  
Language Development ◽  
Behavioral Genetics ◽  
State Of The Art ◽  
Genetic Research ◽  
Great Promise ◽  
Behavioral Genetic ◽  
Fine Grained ◽  
Future Goals ◽  
Current State ◽  
Research Designs

This paper is an introduction to behavioral genetics for researchers and practioners in language development and disorders. The specific aims are to illustrate some essential concepts and to show how behavioral genetic research can be applied to the language sciences. Past genetic research on language-related traits has tended to focus on simple etiology (i.e., the heritability or familiality of language skills). The current state of the art, however, suggests that great promise lies in addressing more complex questions through behavioral genetic paradigms. In terms of future goals it is suggested that: (a) more behavioral genetic work of all types should be done—including replications and expansions of preliminary studies already in print; (b) work should focus on fine-grained, theory-based phenotypes with research designs that can address complex questions in language development; and (c) work in this area should utilize a variety of samples and methods (e.g., twin and family samples, heritability and segregation analyses, linkage and association tests, etc.).

Classification of means and methods of the Web semantic retrieval

2017 ◽  
pp. 030-050
Author(s):  
J.V. Rogushina ◽  
Keyword(s):  
Search Engines ◽  
Domain Knowledge ◽  
Information Needs ◽  
Web Search ◽  
User Interaction ◽  
Query Languages ◽  
Semantic Search ◽  
Semantic Retrieval ◽  
The Web

Problems associated with the improve ment of information retrieval for open environment are considered and the need for it’s semantization is grounded. Thecurrent state and prospects of development of semantic search engines that are focused on the Web information resources processing are analysed, the criteria for the classification of such systems are reviewed. In this analysis the significant attention is paid to the semantic search use of ontologies that contain knowledge about the subject area and the search users. The sources of ontological knowledge and methods of their processing for the improvement of the search procedures are considered. Examples of semantic search systems that use structured query languages (eg, SPARQL), lists of keywords and queries in natural language are proposed. Such criteria for the classification of semantic search engines like architecture, coupling, transparency, user context, modification requests, ontology structure, etc. are considered. Different ways of support of semantic and otology based modification of user queries that improve the completeness and accuracy of the search are analyzed. On base of analysis of the properties of existing semantic search engines in terms of these criteria, the areas for further improvement of these systems are selected: the development of metasearch systems, semantic modification of user requests, the determination of an user-acceptable transparency level of the search procedures, flexibility of domain knowledge management tools, increasing productivity and scalability. In addition, the development of means of semantic Web search needs in use of some external knowledge base which contains knowledge about the domain of user information needs, and in providing the users with the ability to independent selection of knowledge that is used in the search process. There is necessary to take into account the history of user interaction with the retrieval system and the search context for personalization of the query results and their ordering in accordance with the user information needs. All these aspects were taken into account in the design and implementation of semantic search engine "MAIPS" that is based on an ontological model of users and resources cooperation into the Web.

The dilemma of the direct answer

2020 ◽  
Vol 54 (1) ◽  
pp. 1-12
Author(s):  
Martin Potthast ◽  
Matthias Hagen ◽  
Benno Stein
Keyword(s):  
Paradigm Shift ◽  
Information Needs ◽  
Web Search ◽  
Information Sources ◽  
Information Access ◽  
User Preferences ◽  
Information System Design ◽  
Trade Offs ◽  
Remarkable Progress ◽  
Direct Answer

No Web technology has undergone such an impressive evolution as Web search engines did and still do. Starting with the promise of "Bringing order to the Web" 1 by compiling information sources matching a query, retrieval technology has been evolving to a kind of "oracle machinery", being able to recommend a single source, and even to provide direct answers extracted from that source. Notwithstanding the remarkable progress made and the apparent user preferences for direct answers, this paradigm shift comes at a price which is higher than one might expect at first sight, affecting both users and search engine developers in their own way. We call this tradeoff "the dilemma of the direct answer"; it deserves an analysis which has to go beyond system-oriented aspects but scrutinize the way our society deals with both their information needs and means to information access. The paper in hand contributes to this analysis by putting the evolution of retrieval technology and the expectations at it in the context of information retrieval history. Moreover, we discuss the trade offs in information behavior and information system design that users and developers may face in the future.

Sign in / Sign up

Export Citation Format

Share Document