Integrating the Balanced Scorecard and Software Measurement Frameworks

This chapter describes the integration of the Capability Maturity Model (CMM) and ISO-9126 software measurement frameworks with the National Aeronautics and Space Administration Independent Verification and Validation Facility (NASA IV&V). Balanced Scorecard IV&V is a unique aspect of software development practice as it provides a service of independent and objective lifecycle evaluation of the software product and processes used for development. To accomplish this rigorous task a sophisticated measurement program is desirable. This chapter describes the application and integration of strategic measurement (BSC) with organizational measurement (CMM) and product measurement (ISO-9126). The CMM is a measurement model of ordinal ranking of an organization’s software process variability and repeatability. As an organization’s process becomes more mature it may traverse the scale from a level one to a level five organization. The CMM provides a basis for collecting accurate and timely measures of process performance. The international standard ISO/IEC-9126 focuses on information technology and software product evaluation through measurement of software quality characteristics. The development of a core set of metrics for implementing the Balanced Scorecard is the most difficult aspect of the approach. Developing metrics that create the necessary linkages of the operational directives with the strategic mission prove to be fundamentally difficult as it is typical to view organizational performance in terms of outcomes or results. The metrics must address performance drivers or the measures that provide feedback concerning day to day progress.

Management of Large Balanced Scorecard Implementations

In this chapter, established information resource management theory is applied to improve the development and maintenance of large balanced scorecard implementations. The balanced scorecard has proved to be an effective tool for measuring business performance. Maintaining a business-wide balanced scorecard measurement system over a longer period implies, however, many risks. An example of such a risk is the excessive growth of scorecards as well as scorecard metrics, resulting in massive data warehouses and difficulties with the interpretation of data. This is particularly the case in large organisations. This chapter proposes balanced scorecard management framework that is illustrated with the experience gathered from the company-wide balanced scorecard implementation in the insurance company Nationale-Nederlanden in the Netherlands.

Using a Balanced Scorecard Framework to Leverage the Value Delivered by IS

The pace of business and technological change continues to accelerate and the gap between business requirements and the capability of Information Systems (IS) groups to deliver, is getting wider. Very often both business and IS executives fail to understand the value of their IS investments and the factors that underpin IS performance. What is required to address these issues is a broad range of metrics to gauge both IS value and performance. The fundamental principle underlying the approach described in this chapter is that measurement should be used as a catalyst for change. “You cannot fully understand a subject until it is measured,” is a cliche from a famous physicist. The subject matter under review is that of the overall effectiveness and efficiency of an IS group in meeting the business needs. The emphasis is more on the information systems that provide business functionality than on technology processes and technical infrastructure issues. IS management is particularly challenging as it usually lacks mature measurement. This is paradoxical as the IS industry promotes the use of information systems to more effectively manage businesses.

Methodologies for Investment Evaluation

The evaluation of information technology (IT) investments has been a recognised problem area for the last four decades, but has recently been fuelled by rising IT budgets, intangible benefits and considerable risks and gained renewed interest of both management and academics. IT investments already constitute a large and increasing portion of the capital expenditures of many organizations, and are bound to absorb a large part of future funding of new business initiatives. However, for virtually all firms, it is difficult to evaluate the business contribution of an IT investment to current operations or corporate strategy. Consequently, there is a great call for methods and techniques that can be of help in evaluating IT investments, preferably at the proposal and decision-making stages. The contribution of this chapter to the problem area is twofold. First, the different concepts, which are used in evaluation are discussed and more narrowly defined. When speaking about IT investments, concepts are used that originate from different disciplines. In many cases there is not much agreement on the precise meaning of the different concepts used. However, a common language is a prerequisite for the successful communication between the different organizational stakeholders in evaluation. In addition to this, the chapter reviews the current methods for IT investment evaluation and puts them into a frame of reference. All too often new methods and guidelines for investment evaluation are introduced, without building on the extensive body of knowledge that is already incorporated in the available methods. Four basic approaches are discerned: the financial approach, the multi-criteria approach, the ratio approach and the portfolio approach. These approaches are subsequently compared on a number of characteristics on the basis of methods that serve as examples for the different approaches. The chapter concludes with a review of key limitations of evaluations, suggestions on how to improve evaluation practice and recommendations for future research. This chapter draws on earlier work as published in Renkema and Berghout (1997), Berghout (1997), and Renkema (1996; 2000).

Using Cost Benefit Analysis for Enterprise Resource Planning Project Evaluation

The goal of this chapter is to demonstrate how cost benefit analysis can be applied to large-scale ERP projects, and that these methods can incorporate the intangible benefits, e.g., user satisfaction. Detailed information on the business case utilized by a large computer manufacturer in their decision to implement the SAP system R/3 is presented. We illustrate how this organization utilized techniques to include intangibles in the implementation project’s cost benefit analysis. The chapter concludes with a discussion on the state of valuing ERP projects and questions to be answered in the future.

The IT Evaluation and Benefits Management Life-Cycle

It appears that somewhat of a dichotomy exists in many contemporary organisations with respect to the question of investment in information and particularly in information technology (IT). On the one hand, discussions of the new information-based economy and the promise of the new e-business domain leads inevitably to enormous faith being placed in IT, or perhaps more accurately, on the critical, appropriate utilisation of IT to deliver business benefits. Such faith is illustrated by quotes such as: “Across all industries, information and the technology that delivers it have become critical, strategic assets for business firms and their managers” (Laudon and Laudon, 2000). But such enthusiasm is tempered by another view or concern that IT is not delivering on its promises, that it is “oversold and undelivered” (Earl, 1994), and that demonstrating the business value of IT investment is difficult in many instances. This concern that managers do not perceive that they are deriving value for money when it comes to IT investments is troubling when information and IT are often presented as the very backbone of the new economy. Such cynicism is reflected in quotes such as: “There are many different ways to ruin a company. Speculation is the fastest, IT is the most reliable” (Kempis et al., 1999).

A Comparative Analysis of the Balanced Scorecard as Applied in Government and Industry Organizations

Organizations have become increasingly dependent on information technologies to conduct daily operations, achieve competitive advantage and to create and penetrate new markets. This dependence has come at a high price, in 1990 U.S. companies spent over $154 billion on information technologies. However, organizations have found it difficult to measure the value added from these investments. Survey results found four significant barriers to measuring financial performance related to information technologies including: • Difficulty of measuring economic benefits • Inability to determine returns • Lack of good metrics • Incomplete records/accounting of investments The Balanced Scorecard framework provides part of the structure required to overcome these barriers. How organizations can overcome these barriers and successfully measure performance with respect to achieving strategic plans is the focus of this chapter. This chapter provides a comparison of results of two case studies regarding the use of the Balanced Scorecard measurement framework. The application of the Balanced Scorecard (BSC) is evaluated for a Fortune 500 information technology organization and a government organization. Both organizations have a business focus of software development. The BSC framework is applied and reviewed in both contexts to provide insight into unique organizational characteristics for government and contract software environments. A specific focus is to inform the use of financial measures such as Return On Investment (ROI) in the government context. The BSC framework provides the necessary structure to evaluate quantitative and qualitative information and identify the critical linkages between financial measures of past performance and key measures of future performance.

Evaluating the Management of Enterprise Systems with the Balanced Scorecard

The management of Enterprise Systems (ES) software consists of two main tasks: the implementation and the use, stabilisation and change of this comprehensive software. The Balanced Scorecard, a framework originally developed in order to structure the performance measurement for an enterprise or a department, can also be used for the evaluation of ES software. Adapting the Balanced Scorecard and adding a new fifth project perspective allows the comprehensive evaluation of Enterprise Systems and represents an alternative IT evaluation approach. It supports the time consuming implementation of enterprise systems as well and the benefits realization stage. Furthermore, the application of the Balanced Scorecard for IT evaluation represents a novel application area for this strategic management concept.

A Framework to Evaluate the Informatization Level

As information and communication technologies have rapidly developed in the 1990s, enormous changes have taken place everywhere. At work environment, these have been newer tools for increasing organizational productivity, and these are transforming organizations to the degree that Taylorism once did (Davenport, 1998). These trends have spread over various fields of society, and have over countries caused economical and cultural innovation and reformation. These phenomena can be summarized as informatization. Informatization is defined as “converting the main goods and energy of a social economy to information through the revolution of high data communication technology and utilizing information produced by gathering, processing and distributing data within the vast fields of the society” (National Computerization Agency [NCA], 1997). Since The United States’ NII project has been evaluated as one of the important success factors for economical growth, most countries have considered informatization as one of the most effective means for improving a nation’s competitiveness. Similarly, many organizations have considered informatization as a strategy to improve quality of public service and productivity. They have tried to implement informatization and extensive investments are often budgeted and expanded to acquire information technology (IT). An Information Strategy Plan (ISP) is needed at first to implement informatization of an organization. ISP usually includes business strategy, information technology strategy, project priorities, and an organization’s structure strategy. Thus, when an ISP is set up, it describes whether the business or organization’s strategic goals and objectives can be achieved through IT, in which field further IT investment will be needed, and whether efficient investment in IT will be made. In order to discuss these topics, the current organization’s informatization level first must be known. Moreover, since the middle of 1990, many countries have put emphasis on performance based management, in which the government has to set up investment plans according to its performance. For example, to budget IT, it is required to first evaluate its performance and results. In this respect, evaluation of an organization’s informatization level in order to review how much organization informatization it achieves is an important managerial concern. However, this is not a simple problem because informatization includes many intangible factors such as the quality of information and an organization’s culture. In this chapter, framework and metrics are introduced to evaluate the organization’s informatization level. This framework is designed to provide reasonable information by gathering and analyzing various IT metrics for determining whether organizations have made efficient and effective use of IT and have achieved the organizational strategic goals and objectives through IT. Therefore, the evaluation results can be used to improve the organization’s informatization level. The remainder of this paper is organized as follows: in the following section, some case studies and background information are presented. The next section introduces a framework, and then future trends are discussed in the next section. Finally, the summary and conclusion are presented.

Evaluating IS Quality

IT Evaluation is essential, given that the value of investment in the IT industry is currently almost $2 trillion US. There is no doubt that an effective organisation will try to evaluate IT effectiveness, by linking performance measures with a financial perspective (i.e. a shareholders’ view); an internal business perspective (i.e. company planning for excellence); a customer perspective; and the innovation and learning perspective (i.e. the means to improve and create value), in order to move consistently forward. The last three perspectives are at times derived by using the same measures/instruments, via an interpretive approach based upon views of different tiers of stakeholders. Such an approach reflects a movement away from the more technical measures like benchmarking. Instead, IT effectiveness is evaluated in terms of the use of IT, or success of IT outcomes, through seeking to understand the effectiveness of the delivered IT application to the job performance of stakeholders. The merit of this interpretive approach is increasingly applicable to sectors like ecommerce, where it is very apparent that customers are concerned with the effectiveness of such IT applications. With regard to IT research, the interpretive approach was initially crystallised in the Success Model formulated by DeLone and McLean (1992). Their evaluative tools were Use and User Satisfaction. However, if research in related industries is considered, it rapidly becomes apparent that evaluation of quality is a more highly regarded approach. In seeking to adapt this approach to IT, it is important to consider the key components of an IT system, for which effectiveness would be measured in terms of quality; what quality means in an IT context; and how stakeholders internally derive an evaluation of such quality. In summary, this chapter reports on research which has produced a redefined IS Success Model, in which quality is the key to effectiveness. It also reports results of a related empirical study, which reaffirmed this IS Success Model and then investigated whether quality was better measured in terms of stakeholders’ expectations for IS performance and their perceptions of actual performance, or whether measurement of perceptions alone provided sufficient understanding of IS quality/effectiveness.