Information Systems Evaluation

Due to the prevalent use of Information Systems (IS) in modern organisations, evaluation research in this field is becoming more and more important. In light of this, a set of rigorous methodologies were developed and used by IS researchers and practitioners to evaluate the increasingly complex IS implementation used. Moreover, different types of IS and different focusing perspectives of the evaluation require the selection and use of different evaluation approaches and methodologies. This chapter aims to identify, explore, investigate, and discuss the various key methodologies that can be used in IS evaluation from different perspectives, namely in nature (e.g. summative vs. formative evaluation) and in strategy (e.g. goal-based, goal-free, and criteria-based evaluation). Six case studies are also presented and discussed in this chapter to illustrate how the different IS evaluation methodologies can be applied in practices. The chapter concludes that evaluation methodologies should be selected depending on the nature of the IS and the specific goals and objectives of the evaluation. Nonetheless, it is also proposed that formative criteria-based evaluation and summative criteria-based evaluation are currently among the more widely used in IS research. The authors suggest that the combined used of one or more of these approaches can be applied at different stages of the IS life cycle in order to generate more rigorous and reliable evaluation outcomes. Moreover, results and outcomes of IS evaluation research will not just be useful in practically guiding actions to improve the current system, but can also be used to generate new knowledge and theory to be adopted by future IS research.

Creating Interpretive Space for Engaged Scholarship

The purpose of this chapter is to argue that the approach of engaged scholarship provides interpretive space for practitioners who are introducing change in their organization. In this case, the change involved implementation of process innovations, which continue to be an important challenge for business and public sector bodies. The research domain was a subsidiary of APC by Schneider Electric located in Ireland and involved a two-year study where the principal researcher had the status of a temporary employee. A new form of Action Research (AR) called dialogical AR was tested in this study. Key finding from an analysis of the interviews showed that the approach was both helpful and stimulating for the practitioner.

Taxonomy of IT Intangible Assets for Public Administration Based on the Electronic Government Maturity Model in Uruguay

This chapter presents a taxonomy of IT intangible asset indicators for Public Administration, relating the indicators to the Electronic Government Maturity Model proposed by the Uruguayan Agency for Electronic Government and Information Society. Indicators are categorized according to a consolidated intellectual capital model. The Taxonomy is mapped at the indicator level against the EGMM subareas covering all of the relevant aspects associated with the intangible IT assets of the Public Administration in Uruguay. The main challenges and future lines of work for building a consolidated maturity model of IT intangible assets in Public Administration are also presented.

Examining Web 2.0 E-Learning Tools

With so many fields using new technologies in e-learning, we are all challenged with selecting and effectively implementing new Web 2.0 tools. This chapter provides a mixed method research approach to quickly evaluate available Web 2.0 tools and instructional implementation. Class observations and pilot study surveys were used to determine students’ levels of satisfaction after using various numbers of Web 2.0 tools and varying student work group sizes. The pilot studies were designed to model initial classroom examinations when integrating emerging Web 2.0 technologies. Use of this type of pilot study approach is necessitated as many individual class sizes are too small for a full research study, and the time needed to conduct a full study using multiple classes could cause the results to quickly be out of date, thus not providing the needed immediate classroom data for just in time learning. Fast emerging technologies pose a unique challenge to traditional research methodology. Where immediate specific classroom data is needed, a needs analysis with a pilot study is the best option. Note, with emerging technologies, it is difficult to find appropriate literature to determine its effectiveness in the classroom. If desired, compiling the results from many small pilot studies offers an additional benefit of fleshing out key issues to be examined later in greater detail using a full research study for extending theory or scientific practices.

Combining Research Paradigms to Improve Poor Student Performance

The aim of this chapter is to demonstrate the advantages of combining methods from different research paradigms. Positivism, interpretivism, and critical social theory are presented as major paradigms in Information Systems research. The chapter demonstrates the use of methods representative of these three research paradigms in a single research setting. The main problem in the research setting is the poor performance of students in a specific module of their academic programme. This problem is addressed by initiating an action research project using methods representing different research paradigms in the different phases of the project. The argument for using mixed methods is presented by providing information on research paradigms, discussing the problem environment, describing the research process, and finally, reflecting on research paradigms and their application in this environment.

Mapping Participatory Design Methods to the Cognitive Process of Creativity to Facilitate Requirements Engineering

This chapter describes the User-Driven Creativity Framework: a framework that links several Participatory Design (PD) activities into one combined method. This framework, designed to be accordant with the mental process model of creativity, aims to integrate user involvement and creativity in the early stages of application requirements, gathering, and concept development. This chapter aims to contribute to recent discussions on how user-centered or participatory design methods can contribute to information systems development methodologies. The authors describe a mobile language learning case study that demonstrates how an application of the framework resulted in system (paper) prototypes and unveiled perceptions of learners and teachers, effectively yielding the necessary in-depth user knowledge and involvement to establish a strong foundation for further agile development activities. This chapter provides engineers or end-user representatives with a hands-on guide to elicit user requirements and envision possible future application information architectures.

Exploring Higher Education Students’ Technological Identities using Critical Discourse Analysis

This chapter applies a critical theory lens to understanding how South African university students construct meaning about the role of ICTs in their lives. Critical Discourse Analysis (CDA) has been used as a theoretical and analytical device drawing on theorists Fairclough and Gee to examine the key concepts of meaning, identity, context, and power. The specific concepts that inform this study are Fairclough’s three-level framework that enables the situating of texts within the socio-historical conditions and context that govern their process, and Gee’s notion of D(d)iscourses and conceptualization of grand societal “Big C” Conversations. This approach provides insights into students’ educational and social identities and the position of globalisation and the information society in both facilitating and constraining students’ participation and future opportunities. The research confirms that the majority of students regard ICTs as necessary, important, and valuable to life. However, it reveals that some students perceive themselves as not being able to participate in the opportunities technology could offer them. In contrast to government rhetoric, ICTs are not the answer but should be viewed as part of the problem. Drawing on Foucault’s understanding of power as a choice under constraint, this methodological approach also enables examination of how students are empowered or disempowered through their Discourses about ICTs.

Subject Recommended Samples

In a research project, the selection of the sample method is crucial, since it has repercussions throughout the entirety of the study. It determines how the population under scrutiny will be represented and with what accuracy. Hence, it has an important impact in terms of the reliability and validity of the research in general, and consequently, its conclusions. This chapter aims to explore snowball sampling as a chain-referral sampling method. An introductory review of the relevant literature highlights its main characteristics, benefits, and shortcomings, and provides a broader insight to circumstances where it can be successfully applied. This theoretical prologue is followed by the analysis of its employment in an online questionnaire and the presentation of the lessons learned from this sampling decision.

eCommerce Trust Beliefs

Many studies have raised awareness of the importance of trust in the online commercial environment, and it is widely acknowledged. Due to the international nature of eCommerce, it is likely that the influence of culture may extend to online consumers’ trust responses. In this chapter, a trust measurement instrument that had been previously validated in Hong Kong was applied in both the United States and in Ireland—countries that differ in terms of individualism, uncertainty avoidance, and power-distance. Survey methodology was used to collect data. The results provide a refined understanding as to the influence of national culture on the generation of online consumers’ trust beliefs. In doing so, they advance the understanding of information systems and diffusion researchers as well as contributing to the understanding of online vendors who seek to gain insight into the factors that can engender consumer trust in their websites.

Experiences in Applying Mixed-Methods Approach in Information Systems Research

Mixed-methods research, which comprises both quantitative and qualitative components, is widely perceived as a means to resolve the inherent limitations of traditional single method designs and is thus expected to yield richer and more holistic findings. Despite such distinctive benefits and continuous advocacy from Information Systems (IS) researchers, the use of mixed-methods approaches in the IS field has not been high. This chapter discusses some of the key reasons that led to this low application rate of mixed-methods design in the IS field, ranging from misunderstanding the term with multiple-methods research to practical difficulties for design and implementation. Two previous IS studies are used as examples to illustrate the discussion. The chapter concludes by recommending that in order to apply mixed-methods design successfully, IS researchers need to plan and consider thoroughly how the quantitative and qualitative components (i.e. from data collection to data analysis to reporting of findings) can be genuinely integrated together and supplement one another, in relation to the predefined research questions and the specific research contexts.