Applying Design of Experiments (DOE) to Performance Evaluation of Commercial Cloud Services

2013 ◽  
Vol 5 (3) ◽  
pp. 75-93
Author(s):  
Zheng Li ◽  
Liam O’Brien ◽  
He Zhang ◽  
Rajiv Ranjan
Keyword(s):  
Performance Evaluation ◽  
Experimental Design ◽  
Design Of Experiments ◽  
Evaluation Studies ◽  
Cloud Services ◽  
Experimental Results ◽  
Performance Evaluations ◽  
Experimental Sample ◽  
Practical Evaluation ◽  
Services Evaluation

Appropriate performance evaluations of commercial Cloud services are crucial and beneficial for both customers and providers to understand the service runtime, while suitable experimental design and analysis would be vital for practical evaluation implementations. However, there seems to be a lack of effective methods for Cloud services performance evaluation. For example, in most of the existing evaluation studies, experimental factors (also called parameters or variables) were considered randomly and intuitively, experimental sample sizes were determined on the fly, and few experimental results were comprehensively analyzed. To address these issues, the authors suggest applying Design of Experiments (DOE) to Cloud services evaluation. To facilitate applying DOE techniques, this paper introduces an experimental factor framework and a set of DOE application scenarios. As such, new evaluators can explore and conveniently adapt our work to their own experiments for performance evaluation of commercial Cloud services.

scholarly journals A Dockers Storage Performance Evaluation: Impact of Backing File Systems

2021 ◽  
pp. 8-17
Author(s):  
Amer Ramadan ◽  
Keyword(s):  
Performance Evaluation ◽  
Experimental Design ◽  
File Systems ◽  
Experimental Results ◽  
File Server ◽  
Storage Performance ◽  
File Access ◽  
Mail Server ◽  
The Impact ◽  
E Mail

This paper reports on an in-depth examination of the impact of the backing filesystems to Docker performance in the context of Linux container-based virtualization. The experimental design was a 3x3x4 arrangement, i.e., we considered three different numbers of Docker containers, three filesystems (Ext4, XFS and Btrfs), and four application workloads related to Web server I/O activity, e-mail server I/O activity, file server I/O activity and random file access I/O activity, respectively. The experimental results indicate that Ext4 is the most optimal filesystem, among the considered filesystems, for the considered experimental settings. In addition, the XFS filesystem is not suitable for workloads that are dominated by synchronous random write components (e.g., characteristical for mail workload), while the Btrfs filesystem is not suitable for workloads dominated by random write and sequential write components (e.g., file server workload).

scholarly journals Commentary on Kopiez, Wolf, and Platz: The impact of playing from memory on performance evaluation

2017 ◽  
Vol 12 (1-2) ◽  
pp. 15
Author(s):  
Bruno Gingras
Keyword(s):  
Performance Evaluation ◽  
Experimental Design ◽  
Performance Evaluations ◽  
Future Research ◽  
Distractor Effect ◽  
Visual Distractor ◽  
Significant Negative Effect ◽  
Negative Effect ◽  
The Impact

Kopiez et al. (this volume) empirically investigated the effect of playing from memory (more specifically, playing with a conspicuously visible music stand) on the evaluation of audiovisual recordings of solo instrumental performances. Their study, a replication of Williamon (1999), corroborates the finding that the presence of a music stand has a small but significant negative effect on performance evaluations. In this commentary, I present some possible explanations for this effect and discuss relevant implications for future research and for professional performers. I also provide some suggestions for improving the experimental design, such as using a better measure of musical sophistication, controlling for the potential visual distractor effect of the music stand, and developing a more comprehensive and statistically robust scale for evaluating performances.

Public administration approach

2014 ◽  
Vol 35 (6/7) ◽  
pp. 495-507 ◽  
Author(s):  
Jurgita Rudžionienė ◽  
Jaroslav Dvorak
Keyword(s):  
Performance Evaluation ◽  
Public Administration ◽  
Information Services ◽  
Evidence Based ◽  
Economic Aspects ◽  
Content Type ◽  
Evaluation Functions ◽  
Services Evaluation ◽  
Library Evaluation ◽  
Evaluation Measurement

Purpose – The purpose of this paper is to define the problem and to initiate discussion on library evaluation as significant part of institutional evidence-based management from public administration approach. Design/methodology/approach – In order to fulfilling the purpose, special attention to present the concepts of valuing information, library performance evaluation, measurement, etc. is drawn, main evaluation functions are analysed. Economic aspects of information services vs intellectual ones are discussed. Consistent patterns and principles of public administration as well as possibilities of public administration influence in creation of systematic base of library performance evaluation as well as of information services impact to the user are analysed. Findings – The paper provides insights about different aspects of information services evaluation. Results of analysis of economic aspects of information services vs intellectual ones are presented, consistent patterns and principles of public administration, possibilities of public administration influence in creation of systematic base of library performance evaluation as well as of information services impact to the user possibilities are presented. Originality/value – The paper fulfills need to study how public administration could involve library evaluation as tool for evidence-based decision making.

scholarly journals Developing An Evaluation Protocol For The Toronto Public Library (TPL) System Through The Application Of A Building Performance Evaluation (BPE) Of A Branch Library To Inform Its Retrofit Strategy

2021 ◽  
Author(s):  
Rosemary Martin
Keyword(s):  
Performance Evaluation ◽  
Thermal Comfort ◽  
Water Consumption ◽  
User Satisfaction ◽  
Public Library ◽  
Performance Evaluations ◽  
Building Performance ◽  
Design Intent ◽  
Evaluation Protocol ◽  
Operational Costs

Buildings play a significant role in our economy and society. Substantial capital is invested in buildings, and they are the locales where a large portion (e.g., work, cultural, religious, social and personal activities) of our lives are conducted. Despite the significant monetary and temporal investments in buildings, building performance evaluations (BPEs) are not standard practice. From BPEs that have been conducted, important findings have been identified. Significant gaps frequently exist between the design intent of buildings and their measured performance (e.g., energy and water consumption) and user satisfaction (e.g., thermal comfort, lighting, noise). Environmental (e.g., resource consumption) and economic drivers (e.g., productivity, operational costs) are spurring the growth of BPEs. A BPE was conducted of the Weston Public Library (WPL) with the intent of informing a retrofit strategy and developing a protocol for conducting BPEs in the Toronto Public Library (TPL) system.

Asbestos in Drinking Water Performance Evaluation Studies

2009 ◽  
pp. 273-273-15
Author(s):  
Gary B. Collins ◽  
Paul W. Britton ◽  
Patrick J. Clark ◽  
Kim A. Brackett ◽  
Eric J. Chatfield
Keyword(s):  
Drinking Water ◽  
Performance Evaluation ◽  
Evaluation Studies

Sorting Through Performance Evaluations: The Influence of Performance Evaluation Reform on Teacher Attrition and Mobility

2020 ◽  
Vol 57 (6) ◽  
pp. 2339-2377 ◽  
Author(s):  
Luis A. Rodriguez ◽  
Walker A. Swain ◽  
Matthew G. Springer
Keyword(s):  
Performance Evaluation ◽  
Teacher Attrition ◽  
Effective Teachers ◽  
Evaluation System ◽  
Race To The Top ◽  
Performance Evaluations ◽  
American Education ◽  
Mobility Patterns ◽  
Low Performing Schools ◽  
School Settings

The federal Race to the Top initiative signified a shift in American education policy whereby accountability efforts moved from the school to the teacher level. Using administrative data from Tennessee, we explore whether evaluation reforms differentially influenced mobility patterns for teachers of varying effectiveness. We find that the rollout of a statewide evaluation system, even without punitive consequences, was associated with increased turnover; however, there was comparably greater retention of more effective teachers, with larger differences in turnover between highly and minimally effective teachers confined to urban districts and low-performing schools. These results imply that states and districts can increase exit rates of low-performing instructors in the absence of automatic dismissals, which is a pattern that our analyses suggest may not generalize beyond urban school settings.

scholarly journals WIT: A Wireless Integrated Traffic Model

2008 ◽  
Vol 4 (3) ◽  
pp. 219-235 ◽  
Author(s):  
Arta Doci ◽  
Fatos Xhafa
Keyword(s):  
Performance Evaluation ◽  
Ad Hoc Network ◽  
Network Performance ◽  
Ad Hoc ◽  
Performance Metrics ◽  
Building Blocks ◽  
Performance Evaluations ◽  
Network Applications ◽  
Protocol Performance ◽  
Synthetic Models

Simulation is a common approach for designing ad hoc network applications, due to the slow deployment of these networks. The main building blocks of ad hoc network applications are the routing protocols, mobility, and traffic models. Several studies, which use synthetic models, show that mobility and traffic have a significant effect on protocol performance. Synthetic models do not realistically reflect the environment where the ad hoc networks will be deployed. In addition, mobility and traffic tools are designed independently of each other, however real trace data challenge that assumption. Indeed, recent protocol performance evaluation using real testbeds show that performance evaluations under real testbeds and simulations that use synthetic models differ significantly. In this paper we consider jointly both real mobility and traffic for protocol performance evaluation. The contributions of this work are as follows: (1) demonstrates that real mobility and traffic are interconnected; (2) announces the design and implementation of WIT –Wireless Integrated Traffic–, which includes the design of a real traffic generator; (3) shows that under real mobility and integrated traffic the performance metrics need to be re-thought, thus we propose availability as a new ad hoc network protocol performance metric; and, finally, (4) evaluates protocol performance under synthetic and real mobility models with integrated traffic. We believe that the results of our work constitute a step forward toward benchmarking of ad hoc network performance evaluations.

Sign in / Sign up

Export Citation Format

Share Document