scholarly journals Unit testing, model validation, and biological simulation

2016 ◽  
Author(s):  
Gopal Sarma ◽  
Travis W Jacobs ◽  
Mark D Watts ◽  
S. Vahid Ghayoomie ◽  
Stephen D Larson ◽  
...  
Keyword(s):  
Model Validation ◽  
Computational Models ◽  
Cultural Practices ◽  
Open Science ◽  
Heterogeneous Data ◽  
Scientific Models ◽  
Research Quality ◽  
Unit Testing ◽  
Test Driven Development ◽  
Validation Tests

The growth of the software industry has gone hand in hand with the development of tools and cultural practices for ensuring the reliability of complex pieces of software. These tools and practices are now acknowledged to be essential to the management of modern software. As computational models and methods have become increasingly common in the biological sciences, it is important to examine how these practices can accelerate biological software development and improve research quality. In this article, we give a focused case study of our experience with the practices of unit testing and test-driven development in OpenWorm, an open-science project aimed at modeling Caenorhabditis elegans. We identify and discuss the challenges of incorporating test-driven development into a heterogeneous, data-driven project, as well as the role of model validation tests, a category of tests unique to software which expresses scientific models.

scholarly journals Unit testing, model validation, and biological simulation

2016 ◽  
Author(s):  
Gopal Sarma ◽  
Travis W Jacobs ◽  
Mark D Watts ◽  
S. Vahid Ghayoomie ◽  
Stephen D Larson ◽  
...  
Keyword(s):  
Model Validation ◽  
Computational Models ◽  
Cultural Practices ◽  
Open Science ◽  
Heterogeneous Data ◽  
Scientific Models ◽  
Research Quality ◽  
Unit Testing ◽  
Test Driven Development ◽  
Validation Tests

The growth of the software industry has gone hand in hand with the development of tools and cultural practices for ensuring the reliability of complex pieces of software. These tools and practices are now acknowledged to be essential to the management of modern software. As computational models and methods have become increasingly common in the biological sciences, it is important to examine how these practices can accelerate biological software development and improve research quality. In this article, we give a focused case study of our experience with the practices of unit testing and test-driven development in OpenWorm, an open-science project aimed at modeling Caenorhabditis elegans. We identify and discuss the challenges of incorporating test-driven development into a heterogeneous, data-driven project, as well as the role of model validation tests, a category of tests unique to software which expresses scientific models.

scholarly journals Unit testing, model validation, and biological simulation

2015 ◽  
Author(s):  
Gopal Sarma ◽  
Travis W Jacobs ◽  
Mark D Watts ◽  
Vahid Ghayoomi ◽  
Richard C Gerkin ◽  
...  
Keyword(s):  
Model Validation ◽  
Computational Models ◽  
Cultural Practices ◽  
Open Science ◽  
Heterogeneous Data ◽  
Scientific Models ◽  
Research Quality ◽  
Unit Testing ◽  
Test Driven Development ◽  
Validation Tests

The growth of the software industry has gone hand in hand with the development of tools and cultural practices for ensuring the reliability of complex pieces of software. These tools and practices are now acknowledged to be essential to the management of modern software. As computational models and methods have become increasingly common in the biological sciences, it is important to examine how these practices can accelerate biological software development and improve research quality. In this article, we give a focused case study of our experience with the practices of unit testing and test-driven development in OpenWorm, an open-science project aimed at modeling Caenorhabditis elegans. We identify and discuss the challenges of incorporating test-driven development into a heterogeneous, data-driven project, as well as the role of model validation tests, a category of tests unique to software which expresses scientific models.

scholarly journals Unit testing, model validation, and biological simulation

F1000Research ◽  
2016 ◽  
Vol 5 ◽  
pp. 1946 ◽  
Author(s):  
Gopal P. Sarma ◽  
Travis W. Jacobs ◽  
Mark D. Watts ◽  
S. Vahid Ghayoomie ◽  
Stephen D. Larson ◽  
...  
Keyword(s):  
Model Validation ◽  
Computational Models ◽  
Cultural Practices ◽  
Open Science ◽  
Heterogeneous Data ◽  
Scientific Models ◽  
Research Quality ◽  
Unit Testing ◽  
Test Driven Development ◽  
Validation Tests

The growth of the software industry has gone hand in hand with the development of tools and cultural practices for ensuring the reliability of complex pieces of software. These tools and practices are now acknowledged to be essential to the management of modern software. As computational models and methods have become increasingly common in the biological sciences, it is important to examine how these practices can accelerate biological software development and improve research quality. In this article, we give a focused case study of our experience with the practices of unit testing and test-driven development in OpenWorm, an open-science project aimed at modeling Caenorhabditis elegans. We identify and discuss the challenges of incorporating test-driven development into a heterogeneous, data-driven project, as well as the role of model validation tests, a category of tests unique to software which expresses scientific models.

A Call to Develop Standards for Those Delivering ‘Research Practice’ Training

2021 ◽  
Vol 03 ◽  
Author(s):  
Danny Kingsley
Keyword(s):  
Information Science ◽  
Task Force ◽  
Core Competencies ◽  
Scholarly Communication ◽  
Academic Research ◽  
Open Science ◽  
Early Career ◽  
Research Quality ◽  
Research Practice ◽  
The North

The nature of the research endeavour is changing rapidly and requires a wide set of skills beyond the research focus. The delivery of aspects of researcher training ‘beyond the bench’ is met by different sections of an institution, including the research office, the media office and the library. In Australia researcher training in open access, research data management and other aspects of open science is primarily offered by librarians. But what training do librarians receive in scholarly communication within their librarianship degrees? For a degree to be offered in librarianship and information science, it must be accredited by the Australian Library and Information Association (ALIA), with a curriculum that is based on ALIA’s lists of skills and attributes. However, these lists do not contain any reference to key open research terms and are almost mutually exclusive with core competencies in scholarly communication as identified by the North American Serials Interest Group and an international Joint Task Force. Over the past decade teaching by academics in universities has been professionalised with courses and qualifications. Those responsible for researcher training within universities and the material that is being offered should also meet an agreed accreditation. This paper is arguing that there is a clear need to develop parallel standards around ‘research practice’ training for PhD students and Early Career Researchers, and those delivering this training should be able to demonstrate their skills against these standards. Models to begin developing accreditation standards are starting to emerge, with the recent launch of the Centre for Academic Research Quality and Improvement in the UK. There are multiple organisations, both grassroots and long-established that would be able to contribute to this project.

scholarly journals Falsifiability Is Not Optional

2017 ◽  
Author(s):  
Etienne P. LeBel ◽  
Derek Michael Berger ◽  
Lorne Campbell ◽  
Timothy Loving
Keyword(s):  
Statistical Power ◽  
A Priori ◽  
Open Science ◽  
Research Quality ◽  
Original Research ◽  
Fine Grained ◽  
Psychological Phenomena ◽  
Trade Offs ◽  
Cumulative Knowledge ◽  
Improved Model

Finkel, Eastwick, and Reis (2016; FER2016) argued the post-2011 methodological reform movement has focused narrowly on replicability, neglecting other essential goals of research. We agree multiple scientific goals are essential, but argue, however, a more fine-grained language, conceptualization, and approach to replication is needed to accomplish these goals. Replication is the general empirical mechanism for testing and falsifying theory. Sufficiently methodologically similar replications, also known as direct replications, test the basic existence of phenomena and ensure cumulative progress is possible a priori. In contrast, increasingly methodologically dissimilar replications, also known as conceptual replications, test the relevance of auxiliary hypotheses (e.g., manipulation and measurement issues, contextual factors) required to productively investigate validity and generalizability. Without prioritizing replicability, a field is not empirically falsifiable. We also disagree with FER2016’s position that “bigger samples are generally better, but … that very large samples could have the downside of commandeering resources that would have been better invested in other studies” (abstract). We identify problematic assumptions involved in FER2016’s modifications of our original research-economic model, and present an improved model that quantifies when (and whether) it is reasonable to worry that increasing statistical power will engender potential trade-offs. Sufficiently-powering studies (i.e., >80%) maximizes both research efficiency and confidence in the literature (research quality). Given we are in agreement with FER2016 on all key open science points, we are eager to start seeing the accelerated rate of cumulative knowledge development of social psychological phenomena such a sufficiently transparent, powered, and falsifiable approach will generate.

scholarly journals Assessing the macroeconomic impacts of individual behavioral changes on carbon emissions

2019 ◽  
Vol 158 (2) ◽  
pp. 141-160 ◽  
Author(s):  
Leila Niamir ◽  
Gregor Kiesewetter ◽  
Fabian Wagner ◽  
Wolfgang Schöpp ◽  
Tatiana Filatova ◽  
...  
Keyword(s):  
Social Norms ◽  
Carbon Emissions ◽  
Energy Demand ◽  
Computational Models ◽  
Social Dynamics ◽  
Cultural Practices ◽  
Carbon Price ◽  
Computational Social Science ◽  
Green Energy ◽  
Green Technologies

Abstract In the last decade, instigated by the Paris agreement and United Nations Climate Change Conferences (COP22 and COP23), the efforts to limit temperature increase to 1.5 °C above pre-industrial levels are expanding. The required reductions in greenhouse gas emissions imply a massive decarbonization worldwide with much involvement of regions, cities, businesses, and individuals in addition to the commitments at the national levels. Improving end-use efficiency is emphasized in previous IPCC reports (IPCC 2014). Serving as the primary ‘agents of change’ in the transformative process towards green economies, households have a key role in global emission reduction. Individual actions, especially when amplified through social dynamics, shape green energy demand and affect investments in new energy technologies that collectively can curb regional and national emissions. However, most energy-economics models—usually based on equilibrium and optimization assumptions—have a very limited representation of household heterogeneity and treat households as purely rational economic actors. This paper illustrates how computational social science models can complement traditional models by addressing this limitation. We demonstrate the usefulness of behaviorally rich agent-based computational models by simulating various behavioral and climate scenarios for residential electricity demand and compare them with the business as usual (SSP2) scenario. Our results show that residential energy demand is strongly linked to personal and social norms. Empirical evidence from surveys reveals that social norms have an essential role in shaping personal norms. When assessing the cumulative impacts of these behavioral processes, we quantify individual and combined effects of social dynamics and of carbon pricing on individual energy efficiency and on the aggregated regional energy demand and emissions. The intensity of social interactions and learning plays an equally important role for the uptake of green technologies as economic considerations, and therefore in addition to carbon-price policies (top-down approach), implementing policies on education, social and cultural practices can significantly reduce residential carbon emissions.

Model Validation Methods for Multiple Correlated Responses via Covariance-Overlap Based Distance

2019 ◽  
Vol 142 (4) ◽  
Author(s):  
Jiexiang Hu ◽  
Ping Jiang ◽  
Qi Zhou ◽  
Austin McKeand ◽  
Seung-Kyum Choi
Keyword(s):  
Engineering Design ◽  
Model Validation ◽  
Computational Models ◽  
Engineering Problem ◽  
Cumulative Probability ◽  
Global Evaluation ◽  
Correlated Responses ◽  
Validation Methods ◽  
Engineering Design Problems ◽  
Practical Engineering

Abstract Model validation methods have been widely used in engineering design to provide a quantified assessment of the agreement between simulation predictions and experimental observations. For the validation of simulation models with multiple correlated outputs, not only the uncertainty of the responses but also the correlation between them needs to be considered. Most of the existing validation methods for multiple correlated responses focus on the area metric, which only compares the overall area difference between the two cumulative probability distribution curves. The differences in the distributions of the data sets are not fully utilized. In this paper, two covariance-overlap based model validation (COMV) methods are proposed for the validation of multiple correlated responses. The COMV method is used for a single validation site, while the covariance-overlap pooling based model validation (COPMV) method can pool the evidence from different validation sites into a scalar measure to give a global evaluation about the candidate model. The effectiveness and merits of the proposed methods are demonstrated by comparing with three different existing validation methods on three numerical examples and a practical engineering problem of a turbine blade validation example. The influence of sample size and the number of partitions in the proposed methods are also discussed. Results show that the proposed method shows better performance on the uncertainty estimation of different computational models, which is useful for practical engineering design problems with multiple correlated responses.

Sign in / Sign up

Export Citation Format

Share Document