MIPS and BIPS are Megaflops: Limits of Unidimensional Assessments

We believe that a failure to incorporate human performance measures into system test protocols will result in imprecise and incomplete data when attempting to estimate field test performance from a total systems perspective. Traditional methods of evaluating local area network performance generally refer to the network's throughput, time delays, data rate (BIPS), or media access protocol efficiency. These measures are quite acceptable when determining point-to-point benchmark network performance but do not take into account the more global man-machine performance issues associated with people using network systems to perform tasks and execute functions concurrently within a “total systems” context. This paper experimentally compares differences in human productivity/efficiency while using: 1) an existing data gathering system consisting of several geographically distributed, unconnected, and disparate mainframes; and 2) a prototype Intelligent Gateway (Local Area Network) connecting mainframes and offering the user less complexify in procedure execution and an easy to use interface. Tests were conducted with volunteer users in a repeated measures experimental design. Each test subject was randomly assigned to each of two conditions and required to execute routine tasks with each of two systems. ANOVA results revealed significant differences in task completion times and human error rates between the two systems. An increase in human productivity/efficiency was observed using the gateway LAN. We propose to extend the traditional computer performance measurement (monitoring) boundaries, which now encompass only the network hardware, to include an overall “input-to-output” local area network performance measure, combining both measures of user productivity and network performance. A discussion of trade-offs between unidimensional assessment methods using large sample sizes and multiple methods with small sample sizes is also presented. We further believe that without the addition of specific behavioral measures, MIPS and BIPS are truly Megaflops.