scholarly journals Recommendations for the Empirical Assessment of Human-AI Work Systems: A Contribution to AI Measurement Science

2021 ◽  
Author(s):  
Robert Hoffman ◽  
Gary Klein ◽  
William Clancey ◽  
Shane T. Mueller
Keyword(s):  
Research Methods ◽  
Study Design ◽  
Design Research ◽  
Statistical Analyses ◽  
Work Systems ◽  
Empirical Assessment ◽  
Measurement Science ◽  
Technical Details ◽  
Online Experimentation

This Report is a companion to the Report titled "Requirements for the Evaluation of Human-AI Work Systems." Whereas that Report focused on the minimum necessary empirical requirements for the assessment of AI systems, this Report provides additional recommendations and technical details to assist the developers of AI systems. Recommendations are presented covering study design, research methods, measurement, statistical analyses, and online experimentation. This guidance should be applicable to all research intended to evaluate the effectivity of AI systems.

Requirements for the Empirical Assessment of Human-AI Work Systems: A Contribution to AI Measurement Science

2021 ◽  
Author(s):  
Gary Klein ◽  
Robert Hoffman ◽  
Shane T. Mueller ◽  
William Clancey
Keyword(s):  
Design Research ◽  
Empirical Evaluation ◽  
Laboratory Research ◽  
Investment Performance ◽  
Work Systems ◽  
Measurement Science ◽  
Series Of Experiments ◽  
Online Experimentation ◽  
Bench Testing ◽  
Work Method

The development of AI systems represents a significant investment. But to realize the promise of that investment, performance assessment is necessary. Empirical evaluation of Human-AI work systems must adduce convincing empirical evidence that the work method and its AI technology are learnable, usable, and useful. The theme to this Report is the notion that AI assessment must be effective but must also be efficient. Bench testing of a prototype of an AI system cannot require extensive series of experiments with complex designs. Thus, the empirical requirements that are presented in this Report involve escaping some of the constraints that are imposed in traditional laboratory research. Also, there is a recognition of new constraints that are unique to AI evaluation contexts. Empirical requirements are presented covering study design, research methods, statistical analyses, and online experimentation. The 15 requirements presented in this Report should be applicable to all research intended to evaluate the effectivity of AI systems.

scholarly journals Urban Sub-Center Design Framework Based on the Walkability Evaluation Method: Taking Coomera Town Sub-Center as an Example

2021 ◽  
Vol 13 (11) ◽  
pp. 6259
Author(s):  
Jizhong Shao ◽  
Minge Yang ◽  
Guan Liu ◽  
Ye Li ◽  
Dan Luo ◽  
...  
Keyword(s):  
Research Methods ◽  
Quantitative Research ◽  
Design Research ◽  
Evaluation Method ◽  
Rapid Development ◽  
Research Area ◽  
Low Carbon ◽  
Design Strategies ◽  
Healthy City ◽  
Quantitative Research Methods

As current society’s reflection on the rapid development of motorization and increasing emphasis on the ecological environment, the study of walkable cities has become one of the key points of urban sustainable design. Creating a walkable city is an effective way to build a low-carbon and healthy city. With the development of cities, walkability concepts and theories are constantly being given new life, and research methods and design strategies continue to be updated. A city’s walkability and walkability index have become current research hotspots. Based on prior research on walkability and related urban policies, this study selects Coomera Town on the Gold Coast of Queensland, Australia, as the research area because of Coomera Town policy regulations and environmental requirements. This study utilizes traditional qualitative and quantitative research methods, machine mining technology, and the deep learning big data analysis technology to conduct thematic design research in a real place. Its combines walkability evaluation with walkability design to construct a walkable city in a targeted manner. This provides a reference for related city design in the future.

Identifying Design Interventions in Cities for Urban Sustainability

2018 ◽  
Author(s):  
Mohit Arora ◽  
Felix Raspall ◽  
Arlindo Silva
Keyword(s):  
Built Environment ◽  
Research Methods ◽  
Urban Growth ◽  
Design Research ◽  
Urban Sustainability ◽  
Descriptive Study ◽  
Urban Systems ◽  
Demolition Waste ◽  
Material Reuse

Cities have been the focus of recent sustainability and climate change mitigation efforts primarily because of unprecedented urban growth and ever-increasing resources consumption. A worrying trend has been the ever-decreasing life of buildings in cities because of premature building obsolescence. Premature building obsolescence has been cited as the major driver of demolition waste which accounts for more than 40% of total waste generated annually. This waste stream poses a bigger challenge as the pressure on natural resources increases with urban growth. A traditional way of looking at the urban sustainability has been from the perspective of the environmental sciences and waste management methods. Analyzing urban areas with design science perspectives could provide novel insights to improve existing resource consumption patterns and transform sustainability growth in cities. This study focuses on the problem of demolition waste arising from the premature building obsolescence in cities. It applies a design research methodology framework for identifying existing problems associated with demolition waste and generating strategies to transform cities into more sustainable urban systems. In the problem clarification phase, a detailed literature review was supported with stakeholder’s interviews to identify the state-of-art for building demolition process and demolition waste. Research was further extended to descriptive study-I phase to carry out a demolition case study and generate support tools to enable transformation in the existing scenario for achieving a desired state. Singapore, a dense city state of South-East Asia has been taken as a case study in this research. Results show that applying design research methods could help open-up a new dimension to solve urban sustainability challenge for built environment. It highlights that material reuse could lead to significant improvement in the built environment sustainability but the challenge associated with realization of material reuse practice needs to be addressed. Descriptive study-I concludes with the strategies on creating a reuse market through entrepreneurial innovation and an alternative material supply chain of secondary materials for regional housing demand. These results highlight the role of design research methods for tackling complex systems level problems in cities.

scholarly journals THE IMPLICATIONS OF SOCIAL RESPONSIBILITY IN REDUCING YOUTH UNEMPLOYMENT IN ROMANIA

2019 ◽  
Vol 3 (2) ◽  
pp. 33
Author(s):  
Sorina-Geanina Stanescu ◽  
Ana-Maria Comandaru(Andrei)
Keyword(s):  
Sustainable Development ◽  
Social Responsibility ◽  
Research Methods ◽  
Empirical Research ◽  
Design Research ◽  
National Level ◽  
Youth Unemployment ◽  
Business Environment ◽  
Statistical Interpretation ◽  
Main Research

Aim: Lately, there is more and more discussion about sustainable development and social responsibility within organizations, thus increasing the pressure on organizations and managers to act ethically and responsibly. The main purpose of this study is to present how Romanian businesses have integrated aspects of social responsibility into the decision-making and management systems of organizations. The study focuses on identifying the main factors behind the development of the concept of social responsibility and presenting the link between economic activity and sustainable development. A major focus of this research is the study on the implications of corporate social responsibility in reducing the number of youth unemployment at national level. Design/Research methods: As far as the methodology of research is concerned, we will start from the theoretical documentation and we will continue with empirical research, using descriptive analysis and statistical interpretation of data as the main research methods. Conclusions/findings: Empirical research conducted in this paper allowed us to observe the involvement of organizations in adopting support measures for the social problems of young people unemployed. Originality/value of the article: Social responsibility is a vast concept that is growing in Romania and the implication of this concept in solving unemployment has led us to realize the present research that we want to add value to both the business environment in Romania and young researchers interested in this field.

Using compatibility charts to optimize research into fire retardants

2021 ◽  
Vol 29 (6) ◽  
pp. 40-49
Author(s):  
T. Yu. Eremina ◽  
D. A. Korolchenko ◽  
F. A. Portnov
Keyword(s):  
Research Methods ◽  
Interdisciplinary Research ◽  
Building Materials ◽  
Design Research ◽  
Methodological Approach ◽  
Experimental Studies ◽  
Assessment Method ◽  
Material Research ◽  
Fire Retardants ◽  
Interdisciplinary Methods

Introduction. Interdisciplinary research is a most relevant issue in science and education. The integration of intellectual resources with research and production infrastructure is acknowledged as the main goal of interdisciplinary research in the international practice. The main (analytical) part. The authors propose the following methodological approach to the study, based on the distribution of interdisciplinary methods into groups by the scale of the research subject (material). In this case, the studies have the following levels: microlevel, supramolecular level, material research, design research. The paper presents research methods used at each of these levels. The co-authors propose to optimize the study of performance characteristics of building materials and fire retardants through the use of a compatibility chart with regard for the study levels and the analysis of methods of experimental research at each level. Using a compatibility chart. The methodology of the study. A research into the fire retardant efficiency of esters of phosphoric acid, used to modify wood, was selected as a practical example for compatibility diagrams. The project encompasses a number of methods applicable to compatibility charts: the method of elemental analysis, the Gibbs energy assessment method, the sample surface assessment method, the electron microscopy method, methods of assessing fire-hazardous characteristics of wood, the water sorption method, strength and biosecurity assessment methods. Conclusions. The co-authors first proposed an algorithm for generalizing the empirical data on mechanochemical characteristics of materials using interdisciplinary methods in the form of a compatibility chart. This methodology optimizes research into any composite materials though it preserves targeted research methods and eliminates impractical and concomitant experimental studies, thus, reducing labour costs and environmental impacts.

No lab, no problem

2019 ◽  
Vol 14 (1) ◽  
pp. 152-168
Author(s):  
Jordan Gallant ◽  
Gary Libben
Keyword(s):  
Ecological Validity ◽  
Psycholinguistic Research ◽  
Online Experiments ◽  
The Future ◽  
Overall Evaluation ◽  
Technical Details ◽  
Online Experimentation ◽  
Behavioural Experiments ◽  
Millisecond Timing ◽  
The Web

Abstract We present new opportunities for psycholinguistic research that are made available by presenting experiments online over the web. We focus on PsychoPy3, which is a new version of a system for the development and delivery of behavioural experiments. Crucially, it allows for both these functions to be performed online. We note that experiments delivered over the web have significant efficiency advantages. They also open up new opportunities to increase the ecological validity of experiments and to facilitate the participation of members of populations that have thus far been less studied in the psycholinguistic literature. We discuss the crucial matter of millisecond timing in online experiments. The technical details of implementation of a behavioural psycholinguistic experiment are presented, along with listings of additional technical resources and support. Our overall evaluation is that although online experimentation still has technical challenges and improvements are ongoing, it may well represent the future of behavioural psycholinguistic research.

scholarly journals Responsibility for employees as a challenge for contemporary organizations

2019 ◽  
Vol 3 (1) ◽  
pp. 207
Author(s):  
Magdalena Wróbel
Keyword(s):  
Human Capital ◽  
Competitive Advantage ◽  
Research Methods ◽  
Design Research ◽  
Statistical Data ◽  
Literature Study ◽  
Good Practices ◽  
Iso 26000 ◽  
Labor Practices ◽  
The Impact

Aim: The paper considers the issue of responsibility on the part of contemporary organizations for their employees. In her paper, the author described ISO 26000 guidelines for organizations concerning labor practices and the impact of these guidelines on competitiveness of organizations. The paper also analyzed labor practices implemented by firms in Poland in the years 2013-2015. Design / Research methods: Based on statistical data and literature study as well as Responsible Business in Poland. Good practices reports., the period 2013-2015 is considered for analysis. Conclusions / findings: Entrepreneurs in Poland seem to take more responsibility for employees, while recognizing the key role played by human capital in building competitive advantage of an organization. This also supports the company’s image and competitiveness, while improving employee loyalty.

scholarly journals Systematic Review and Quality Evaluation of Pharmacoeconomic Studies on Traditional Chinese Medicines

2021 ◽  
Vol 9 ◽  
Author(s):  
Nan Yang ◽  
Huihui Zhang ◽  
Taoyi Deng ◽  
Jeff Jianfei Guo ◽  
Ming Hu
Keyword(s):  
Sample Size ◽  
Study Design ◽  
Design Research ◽  
Cost Minimization ◽  
Pharmacoeconomic Evaluation ◽  
Cost Utility ◽  
Quality Score ◽  
Future Research ◽  
Economic Evaluations

Objectives: This study was aimed to find and appraise the available published pharmacoeconomic research on Traditional Chinese Medicine (TCM), to identify related issues and make suggestions for improvement in future research.Methods: After developing a search strategy and establishing inclusion and exclusion criteria, pharmacoeconomic studies on TCM were sourced from seven Chinese and English databases from inception to April 2020. Basic information about the studies and key pharmacoeconomic items of each study were extracted. The quality of each study was evaluated by using the British Medical Journal economic submissions checklist for authors and peer reviewers, focusing on factors such as study design, research time horizon, sample size, perspective, and evaluation methods.Results: A total of 431 published pharmacoeconomic articles with 434 studies on topics including cost-effectiveness, cost-benefit, cost-minimization, cost-utility, or combination analyses were identified and included in this review. Of these, 424 were published in Chinese and 7 in English. These studies conducted economic evaluations of 264 Chinese patent medicines and 70 types of TCM prescriptions for 143 diseases, including those of the central nervous, cardiovascular, respiratory, gynecologyical, and other systems. The studied TCMs included blood-activating agents (such as Xuesaitong tablet, Fufant Danshen tablet, and Danhong Injection), blood circulation promoting agents (such as Shuxuetong injection, Rupixiao tablet, and Fufang Danshen injection), and other therapeutic agents. The overall quality score of the studies was 0.62 (range 0.38 to 0.85). The mean quality score of studies in English was 0.72, which was higher than that of studies in Chinese with 0.62.Conclusions: The quality of pharmacoeconomic studies on TCM was relatively, generally low. Major concerns included study design, inappropriate pharmacoeconomic evaluation, insufficient sample size, or non-scientific assessment. Enhanced methodological training and cooperation, the development of a targeted pharmacoeconomic evaluation guideline, and proposal of a reasonable health outcome index are warranted to improve quality of future studies.

Sign in / Sign up

Export Citation Format

Share Document