A review of:
Kwon, Nahyun. "Public Library Patrons' Use of Collaborative Chat Reference Service: The Effectiveness of Question Answering by Question Type." Library & Information Science Research 29.1 (Mar. 2007): 70-91.
Objective – To assess the effectiveness of a collaborative chat reference service in answering different types of question. Specifically, the study compares the degree of answer completion and the level of user satisfaction for simple factual questions vs. more in-depth subject-based reference questions, and for ‘local’ (pertaining to a particular library) and non-local questions.
Design – Content analysis of 415 transcripts of reference transactions, which were also compared to corresponding user satisfaction survey results.
Setting – An online collaborative reference service offered by a large public library system (33 branch and regional locations). This service is part of the Metropolitan Co-operative Library System: a virtual reference consortium of U.S. libraries (public, academic, special, and corporate) that provides 24/7 service.
Subjects – Reference librarians from around the U.S. (49 different libraries), and users logging into the service via the public library system’s portal (primarily patrons of the 49 libraries).
Method – Content analysis was used to evaluate virtual reference transcripts recorded between January and June, 2004. Reliability was enhanced through triangulation, with researchers comparing the content analysis of each transcript against the results of a voluntary exit survey. Of 1,387 transactions that occurred during the period of study, 420 users completed the survey and these formed the basis of the study, apart from 5 transactions that were omitted because the questions were incomprehensible. Questions were examined and assigned to five categories: “simple, factual questions; subject-based research questions; resource access questions; circulation-related questions; and local library information inquiries” (80-81). Answers were classed as either “completely answered, partially answered or unanswered, referred, and problematic endings” (82). Lastly, user satisfaction was surveyed on three measures: satisfaction with the answer, perceived staff quality, and willingness to return. In general, the methods used were clearly described and appeared reliable.
Main results – Distribution of question types: By far the largest group of questions were circulation-related (48.9%), with subject-based research questions coming next (25.8%), then simple factual questions (9.6%), resource access questions (8.9%), and local library information inquiries (6.8%).
Effectiveness of chat reference service by question type: No statistically significant difference was found between simple factual questions and subject-based research questions in terms of answer completeness and user satisfaction. However, a statistically significant difference was found when comparing ‘local’ (circulation and local library information questions) and ‘non-local’ (simple factual and subject-based research questions), with both satisfaction and answer completeness being lower for local questions.
Conclusions – The suggestion that chat reference may not be as appropriate for in-depth, subject-based research questions as it is for simple factual questions is not supported by this research. In fact, the author notes that “subject-based research questions, when answered, were answered as completely as factual questions and found to be the question type that gives the greatest satisfaction to the patrons among all question types” (86).
Lower satisfaction and answer completion were found among local vs. non-local queries. Additionally, there appeared to be some confusion among patrons about the nature of the collaborative service – they often assumed that the librarian answering their question was from their local library. The author suggests some form of triage to direct local questions to the appropriate venue from the outset, thus avoiding confusion and unnecessary referrals. The emergence of repetitive questions also signalled the need for the development of FAQs for chat reference staff and the incorporation of such questions into chat reference training.
Teaching and User Satisfaction in an Academic Chat Reference Consortium
