Author(s):  
Riina Vuorikari ◽  
Nikos Manouselis ◽  
Erik Duval

Social information retrieval systems, such as recommender systems, can benefit greatly from sharable and reusable evaluations of online resources. For example, in distributed repositories with rich collections of learning resources, users can benefit from evaluations, ratings, reviews, annotations, etc. that previous users have provided. Furthermore, sharing these evaluations and annotations can help attain the critical mass of data required for social information retrieval systems to be effective and efficient. This kind of interoperability requires a common framework that can be used to describe in a reusable manner the evaluation approach, as well as the results of the evaluation. This chapter discusses this concept, focusing on the rationale for a reusable and interoperable framework, that can be used to facilitate the representation, management and reuse of results from the evaluation of learning resources. For this purpose, we review a variety of evaluation approaches for learning resources, and study ways in which evaluation results may be characterised, so as to draw requirements for sharable and reusable evaluation metadata. Usage scenarios illustrate how evaluation metadata can be useful in the context of recommender systems for learning resources.


1967 ◽  
Vol 06 (02) ◽  
pp. 45-51 ◽  
Author(s):  
A. Kent ◽  
J. Belzer ◽  
M. Kuhfeerst ◽  
E. D. Dym ◽  
D. L. Shirey ◽  
...  

An experiment is described which attempts to derive quantitative indicators regarding the potential relevance predictability of the intermediate stimuli used to represent documents in information retrieval systems. In effect, since the decision to peruse an entire document is often predicated upon the examination of one »level of processing« of the document (e.g., the citation and/or abstract), it became interesting to analyze the properties of what constitutes »relevance«. However, prior to such an analysis, an even more elementary step had to be made, namely, to determine what portions of a document should be examined.An evaluation of the ability of intermediate response products (IRPs), functioning as cues to the information content of full documents, to predict the relevance determination that would be subsequently made on these documents by motivated users of information retrieval systems, was made under controlled experimental conditions. The hypothesis that there might be other intermediate response products (selected extracts from the document, i.e., first paragraph, last paragraph, and the combination of first and last paragraph), that would be as representative of the full document as the traditional IRPs (citation and abstract) was tested systematically. The results showed that:1. there is no significant difference among the several IRP treatment groups on the number of cue evaluations of relevancy which match the subsequent user relevancy decision on the document;2. first and last paragraph combinations have consistently predicted relevancy to a higher degree than the other IRPs;3. abstracts were undistinguished as predictors; and4. the apparent high predictability rating for citations was not substantive.Some of these results are quite different than would be expected from previous work with unmotivated subjects.


2005 ◽  
Vol 14 (5) ◽  
pp. 335-346
Author(s):  
Por Carlos Benito Amat ◽  
Por Carlos Benito Amat

Libri ◽  
2020 ◽  
Vol 70 (3) ◽  
pp. 227-237
Author(s):  
Mahdi Zeynali-Tazehkandi ◽  
Mohsen Nowkarizi

AbstractEvaluation of information retrieval systems is a fundamental topic in Library and Information Science. The aim of this paper is to connect the system-oriented and the user-oriented approaches to relevant philosophical schools. By reviewing the related literature, it was found that the evaluation of information retrieval systems is successful if it benefits from both system-oriented and user-oriented approaches (composite). The system-oriented approach is rooted in Parmenides’ philosophy of stability (immovable) which Plato accepts and attributes to the world of forms; the user-oriented approach is rooted in Heraclitus’ flux philosophy (motion) which Plato defers and attributes to the tangible world. Thus, using Plato’s theory is a comprehensive approach for recognizing the concept of relevance. The theoretical and philosophical foundations determine the type of research methods and techniques. Therefore, Plato’s dialectical method is an appropriate composite method for evaluating information retrieval systems.


Sign in / Sign up

Export Citation Format

Share Document