Innovations for Evaluation Research: Multiform Protocols, Visual Analog Scaling, and the Retrospective Pretest–Posttest Design

2018 ◽  
Vol 41 (2) ◽  
pp. 246-269 ◽  
Author(s):  
Rong Chang ◽  
Todd D. Little
Keyword(s):  
Data Collection ◽  
Real World ◽  
Empirical Studies ◽  
Evaluation Research ◽  
Innovative Methods ◽  
Research Techniques ◽  
Retrospective Pretest

In this article, we review three innovative methods: multiform protocols, visual analog scaling, and the retrospective pretest–posttest design that can be used in evaluation research. These three techniques have been proposed for decades, but unfortunately, they are still not utilized readily in evaluation research. Our goal is to familiarize researchers with these underutilized research techniques that could reduce personnel effort and costs for data collection while producing better inferences for a study. We begin by discussing their applications and special unique features. We then discuss each technique’s strengths and limitations and offer practical tips on how to better implement these methods in evaluation research. We then showcase two recent empirical studies that implement these methods in real-world evaluation research applications.

scholarly journals Friendship paradox in growth networks: analytical and empirical analysis

2021 ◽  
Vol 6 (1) ◽  
Author(s):  
Sergei P. Sidorov ◽  
Sergei V. Mironov ◽  
Alexey A. Grigoriev
Keyword(s):  
Real World ◽  
Preferential Attachment ◽  
Empirical Studies ◽  
Mean Field ◽  
Network Models ◽  
Average Degree ◽  
Closure Model ◽  
Triadic Closure ◽  
Attachment Mechanism ◽  
Over Time

AbstractMany empirical studies have shown that in social, citation, collaboration, and other types of networks in real world, the degree of almost every node is less than the average degree of its neighbors. This imbalance is well known in sociology as the friendship paradox and states that your friends are more popular than you on average. If we introduce a value equal to the ratio of the average degree of the neighbors for a certain node to the degree of this node (which is called the ‘friendship index’, FI), then the FI value of more than 1 for most nodes indicates the presence of the friendship paradox in the network. In this paper, we study the behavior of the FI over time for networks generated by growth network models. We will focus our analysis on two models based on the use of the preferential attachment mechanism: the Barabási–Albert model and the triadic closure model. Using the mean-field approach, we obtain differential equations describing the dynamics of changes in the FI over time, and accordingly, after obtaining their solutions, we find the expected values of this index over iterations. The results show that the values of FI are decreasing over time for all nodes in both models. However, for networks constructed in accordance with the triadic closure model, this decrease occurs at a much slower rate than for the Barabási–Albert graphs. In addition, we analyze several real-world networks and show that their FI distributions follow a power law. We show that both the Barabási–Albert and the triadic closure networks exhibit the same behavior. However, for networks based on the triadic closure model, the distributions of FI are more heavy-tailed and, in this sense, are closer to the distributions for real networks.

Validation of a Musculoskeletal Digital Assessment Routing Tool (DART): Protocol for a Pilot Randomised Crossover Non-Inferiority Trial (Preprint)

2021 ◽  
Author(s):  
Cabella Lowe ◽  
Harry Hanuman Sing ◽  
William Marsh ◽  
Dylan Morrissey
Keyword(s):  
Clinical Trials ◽  
Data Collection ◽  
Real World ◽  
User Satisfaction ◽  
Allocation Concealment ◽  
Trial Methodology ◽  
Primary Analysis ◽  
Digital Assessment ◽  
System Usability Scale ◽  
The Impact

BACKGROUND Musculoskeletal conditions account for 16% of global disability, resulting in a negative effect on millions of patients and an increasing burden on healthcare utilization. Digital technologies to improve health care outcomes and efficiency are considered a priority; however, innovations are rarely tested with sufficient rigor in clinical trials, the gold standard for clinical proof of safety and efficacy. We have developed a new musculoskeletal Digital Assessment Routing Tool (DART) that allows users to self-assess and be directed to the right care. DART requires validation in a real-world setting prior to implementation. OBJECTIVE This pilot study will assess the feasibility of a future trial by exploring key aspects of trial methodology, assess the procedures and collect exploratory data to inform the design of a definitive, randomized, crossover, non-inferiority trial to assess DART safety and effectiveness. METHODS We will collect data from 76 adult participants presenting to an NHS England GP practice with a musculoskeletal condition. Participants will complete both a DART assessment and a physiotherapist-led triage with the order determined by randomization. The primary analysis will involve an absolute agreement ICC (A,1) estimate with 95% confidence intervals between DART and the clinician for assessment outcomes sign-posting to condition management pathways. Data will be collected to allow analysis of participant recruitment and retention, randomization, allocation concealment, blinding, data collection process and bias. In addition, the impact of trial burden and potential barriers to intervention delivery will be considered. DART user satisfaction will be measured using the System Usability Scale. RESULTS A UK NHS ethics submission will be submitted during June 2021 and pending approval, recruitment will commence during August 2021 with data collection anticipated to last for 3 months. Results will be reported in a follow-up paper later in 2021. CONCLUSIONS This study will inform the design of a randomized controlled crossover non-inferiority study that will provide evidence concerning mHealth DART system clinical sign posting in an NHS setting prior to real-world implementation. Success should produce evidence of a safe, effective system with excellent usability, facilitating quicker and easier patient access to appropriate care while reducing the burden on primary and secondary care musculoskeletal services. This rigorous approach to mHealth system testing could be used as a guide for other developers of similar applications. CLINICALTRIAL This trial is registered with Clinical Trials number NCT04904029

scholarly journals A State-of-the-Art Review on Empirical Data Collection for External Governed Pedestrians Complex Movement

2018 ◽  
Vol 2018 ◽  
pp. 1-42 ◽  
Author(s):  
Xiaomeng Shi ◽  
Zhirui Ye ◽  
Nirajan Shiwakoti ◽  
Offer Grembek
Keyword(s):  
Data Collection ◽  
Empirical Data ◽  
Empirical Studies ◽  
Movement Patterns ◽  
Simulation Models ◽  
Straight Line ◽  
Challenges And Opportunities ◽  
Pedestrian Traffic ◽  
Random Flows ◽  
Complex Movement

Complex movement patterns of pedestrian traffic, ranging from unidirectional to multidirectional flows, are frequently observed in major public infrastructure such as transport hubs. These multidirectional movements can result in increased number of conflicts, thereby influencing the mobility and safety of pedestrian facilities. Therefore, empirical data collection on pedestrians’ complex movement has been on the rise in the past two decades. Although there are several reviews of mathematical simulation models for pedestrian traffic in the existing literature, a detailed review examining the challenges and opportunities on empirical studies on the pedestrians complex movements is limited in the literature. The overall aim of this study is to present a systematic review on the empirical data collection for uni- and multidirectional crowd complex movements. We first categorized the complex movements of pedestrian crowd into two general categories, namely, external governed movements and internal driven movements based on the interactions with the infrastructure and among pedestrians, respectively. Further, considering the hierarchy of movement complexity, we decomposed the externally governed movements of pedestrian traffic into several unique movement patterns including straight line, turning, egress and ingress, opposing, weaving, merging, diverging, and random flows. Analysis of the literature showed that empirical data were highly rich in straight line and egress flow while medium rich in turning, merging, weaving, and opposing flows, but poor in ingress, diverging, and random flows. We put emphasis on the need for the future global collaborative efforts on data sharing for the complex crowd movements.

scholarly journals Alternative Strategies to Avoid Layoff in Airlines Industry During the Covid-19 Pandemic

2021 ◽  
Vol 4 (2) ◽  
pp. 69-83
Author(s):  
Padmalini Singh ◽  
Dilip D ◽  
Nuthan Jeevraj P ◽  
Yip Wei Hung ◽  
Daisy Mui Hung Kee ◽  
...  
Keyword(s):  
Data Analysis ◽  
Data Collection ◽  
Airline Industry ◽  
Empirical Studies ◽  
Secondary Data ◽  
Electronic Media ◽  
Primary Data ◽  
Online Questionnaire ◽  
Alternative Strategies ◽  
Airline Company

This research was carried out to explore the layoff situation among the Airline industry during the COVID-19 pandemic and suggest the alternative strategies that would consider by the Airline company. The method used in this research is an online questionnaire distributed to 100 respondents as the primary data collection and the secondary data analysis from the empirical studies that can obtain from electronic and non-electronic media. The findings of this research indicated that one of the suggested alternative strategies, Furlough, is the most recommended one. However, Attrition and Hiring Freeze strategy has less recommended because of the low effectiveness of this strategy on downsizing companies. The study enables the airline company to consider the alternative strategies on their downsizing policy in order to minimize the employees and company loss.

scholarly journals Looking Inside the Black Box: The Importance of Causal Mechanism and Treatment Effect Heterogeneity in Experimentally Evaluated Criminal Justice Interventions

2016 ◽  
Vol 31 (1) ◽  
pp. 89-112
Author(s):  
Na Chong Min
Keyword(s):  
Treatment Effect ◽  
Empirical Studies ◽  
Evaluation Research ◽  
Black Box ◽  
Average Treatment Effect ◽  
Causal Mechanism ◽  
Population Average ◽  
Treatment Effect Heterogeneity ◽  
Policies And Programs ◽  
Effect Heterogeneity

This paper discusses limitations of the ???black-box??? experimental archetype by highlighting the narrowness of outcome-focused approaches. For a more complete understanding of the nuanced implications of policies and programs, this study calls for an investigation of causal mechanism and treatment effect heterogeneity in experimentally evaluated interventions. This study draws on two distinct but closely related empirical studies, one undertaken by Na and Paternoster (2012) and the other by Na, Loughran, and Paternoster (2015), that go beyond the estimation of a population average treatment effect by adopting more recent methodological advancements that are still underappreciated and underutilized in evaluation research.

scholarly journals Virtual Physics Laboratory with Real World Problem Based on Ngada Local Wisdom in Basic Physics Practicum

2021 ◽  
Vol 5 (4) ◽  
pp. 520
Author(s):  
Maria Yuliana Kua ◽  
Ni Wayan Suparmi ◽  
Dek Ngurah Laba Laksana
Keyword(s):  
Data Collection ◽  
Real World ◽  
Average Score ◽  
Study Program ◽  
Face To Face ◽  
Physics Laboratory ◽  
Real World Problem ◽  
Validation Score ◽  
Basic Physics ◽  
The Media

This research is based on the problem where practical activities in the Basic Physics Practicum course can no longer be carried out optimally due to changes in the learning model from face-to-face (offline) to online (online) during the COVID-19 pandemic. The purpose of this study was to develop a virtual physics laboratory as a medium in carrying out practical activities and to analyze the feasibility of the product through the validation results of experts and the results of product trials on prospective users. This type of research is Research & Development with ADDIE development model. The subjects of this study were 12 lecturers and 47 students of the STKIP Citra Bakti science education study program. Data collection techniques using validation sheets and questionnaires. The data collection instruments are in the form of validation assessment sheets and response questionnaires of prospective users. The data from this study were analyzed qualitatively descriptive to decide the feasibility of the product being developed. The results of the research showed that the average validation score of the material expert was 4.63, the media expert was 4.41, the learning design expert was 4.30, and the linguist was 4.51. The validation results of the four validators are in the very good category. Meanwhile, the results of product trials to lecturers and students as potential users are in the very good category with an average score of 4.53 and 4.57, respectively. Based on these data, this virtual physics laboratory product with real world problems based on Ngada local wisdom is recommended to be applied to the Basic Physics Practicum course and to help students in their independent practicum activities.

scholarly journals 1464Application of Webinars for popularizing Occupational Health Epidemiology during the Covid-19 Pandemic in India

2021 ◽  
Vol 50 (Supplement_1) ◽  
Author(s):  
Rajiv Kumar Jain
Keyword(s):  
Data Analysis ◽  
Data Collection ◽  
Occupational Health ◽  
Average Duration ◽  
Health Practice ◽  
Innovative Methods ◽  
Innovative Method ◽  
Rightful Place ◽  
Health Practitioners ◽  
Research Initiatives

Abstract Focus of Presentation The process and results of popularizing Occupational Health Epidemiology amongst Occupational Health practitioners in India during the covid-19 Pandemic in India through Webinars Findings The 25 webinars of average duration of 90 minutes, with contents relating to Covid-19 Epidemiology in India, generated immense interest amongst Occupational Health Practitioners with reference to innovative methods of data collection, analysis of data, results dissemination and integration of results in occupational Health practice during pandemic of Covid-19 in India. Conclusions/Implications Occupational Health Epidemiology is a neglected discipline in India. Innovative method of use of webinars amongst Occupational Health practitioners can be used for popularizing the methods, data analysis and results dissemination etc. It is expected that this interest shall be sustained in Post Pandemic period and the discipline of Occupational Health Epidemiology will get its rightful place amongst Occupational Health practitioners in India leading to research initiatives and application of results in the practice of Occupational Health in India.

OP117 Digital Real-World Evidence In Times Of General Data Protection Regulation

2021 ◽  
Vol 37 (S1) ◽  
pp. 1-1
Author(s):  
Rhodri Saunders ◽  
Rafael Torrejon Torres ◽  
Maximilian Blüher
Keyword(s):  
Data Collection ◽  
Real World ◽  
Data Protection ◽  
Safety Data ◽  
Design Approach ◽  
Data Set ◽  
General Data Protection Regulation ◽  
Real World Evidence ◽  
Quality Markers ◽  
General Data

IntroductionReal-world evidence (RWE) is a useful supplement to a product's evidence base especially for medical devices, which are often unsuitable for randomized controlled trials. Generally, RWE is analyzed retrospectively (for example, healthcare records), which lack granularity for health-economic analysis. Prospective collection of RWE in hospitals can promote device-specific endpoint assessment. The advent of the General Data Protection Regulation (GDPR) requires a privacy-by-design approach. This work describes a workflow for a GDPR-compliant device-specific RWE collection as part of quality improvement initiatives (QII).MethodsA literature review identifies relevant clinical and quality markers as endpoints to the investigated technology. A panel of experts grade these endpoints on their clinical significance, privacy sensitivity, analytic value, and feasibility for collection. Endpoints meeting a predefined cut-off are considered quality markers for the QII. Finally, an RWE data collection app is designed to collect the quality markers using either longitudinal, pseudonymized data or single time-point anonymized data to ensure data protection by design.ResultsUsing this approach relevant clinical markers were identified in a GDPR-compliant manner. The data collection app design ensured that patient data were protected, while maintaining minimum requirements on patient information and consent. The pilot QII collected data on over 5,000 procedures, which represents the largest single data set available for the tested technology. Due to its prospective nature this programme was the first to collect patient outcomes in sufficient quantity for analysis, while previous studies only recorded adverse events.ConclusionsGDPR and RWE can co-exist in harmony. A design approach, which has data protection in mind from the start can combine high quality RWE collection of efficacy and safety data with maximum patient privacy.

Glean

2021 ◽  
Vol 14 (6) ◽  
pp. 997-1005
Author(s):  
Sandeep Tata ◽  
Navneet Potti ◽  
James B. Wendt ◽  
Lauro Beltrão Costa ◽  
Marc Najork ◽  
...  
Keyword(s):  
Machine Learning ◽  
Data Management ◽  
Real World ◽  
Empirical Studies ◽  
Ground Truth ◽  
Training Data ◽  
Ground Truth Data ◽  
Document Type ◽  
Machine Learning Model ◽  
Structured Information

Extracting structured information from templatic documents is an important problem with the potential to automate many real-world business workflows such as payment, procurement, and payroll. The core challenge is that such documents can be laid out in virtually infinitely different ways. A good solution to this problem is one that generalizes well not only to known templates such as invoices from a known vendor, but also to unseen ones. We developed a system called Glean to tackle this problem. Given a target schema for a document type and some labeled documents of that type, Glean uses machine learning to automatically extract structured information from other documents of that type. In this paper, we describe the overall architecture of Glean, and discuss three key data management challenges : 1) managing the quality of ground truth data, 2) generating training data for the machine learning model using labeled documents, and 3) building tools that help a developer rapidly build and improve a model for a given document type. Through empirical studies on a real-world dataset, we show that these data management techniques allow us to train a model that is over 5 F1 points better than the exact same model architecture without the techniques we describe. We argue that for such information-extraction problems, designing abstractions that carefully manage the training data is at least as important as choosing a good model architecture.

Real-world outcomes and clinical characteristics of patients with brain metastases from EGFR mutated non-small cell lung cancer: Data from a large retrospective study (REFLECT).

2021 ◽  
Vol 39 (15_suppl) ◽  
pp. 9086-9086
Author(s):  
Urska Janzic ◽  
Alfredo Addeo ◽  
Elizabeth Dudnik ◽  
Andriani Charpidou ◽  
Adam Pluzanski ◽  
...  
Keyword(s):  
Lung Cancer ◽  
Data Collection ◽  
Brain Metastases ◽  
Small Cell Lung Cancer ◽  
Cell Lung Cancer ◽  
Real World ◽  
Clinical Characteristics ◽  
Small Cell ◽  
Small Cell Lung ◽  
Cancer Data

9086 Background: Brain metastases (BM) frequently occur in patients (pts) with epidermal growth factor receptor mutated non-small cell lung cancer (EGFRm NSCLC) and represent a poor prognostic marker. This study aimed to describe the clinical characteristics, treatment patterns and survival outcomes in EGFRm NSCLC pts treated with 1st or 2nd generation tyrosine-kinases inhibitors (TKIs) in first-line (1L). Methods: The retrospective real-world study REFLECT (NCT04031898) collected data from 896 pts initiating 1L TKI between 1 January 2015-30 June 2018 in Europe and Israel. Descriptive statistics were used to assess demographic and clinical characteristics in subgroups of patients with and without BM. Kaplan-Meier methods were used to estimate median real world progression free survival (mPFS) and overall survival (mOS) from start of 1L. Results: Out of 896 pts, 198 (22.1%) had BM at start of 1L, 134 (15%) developed BM later (any time), and 564 (62.9%) had no sign of BM at the time of data collection. Among pts who later developed BM the median time between the start of 1L and first diagnosis of BM was 13.5 months. Median duration of follow-up was 21.5 months. Of 332 pts with BM at any time 64.2% were female, similar to the ratio in pts without BM (64.0%). At diagnosis, median age was 65 years in pts with BM vs. 70 in those who never developed BM. Of pts with BM at any time, 50.9% had exon 19 deletion, 30.4% L858R point mutation and 18.7 % uncommon EGFR mutations at baseline, compared to 56.6%, 31.7% and 11.7% in pts without BM, respectively. At data collection, 94.9% of the pts with BM at diagnosis had progressed compared to 79.8% among those with no BM. Overall, whole brain radiation was the most frequently used treatment for BM (31.0%) followed by stereotactic radiosurgery (18.1%) and targeted therapies (13.3%). T790M testing rates were highest among pts developing BM later (85.7%) and lowest among those with BM from start (66.1%). The T790M positivity rate was highest in pts developing BM later (65.7%) and lowest among those with BM from start (50.4%). More pts received osimertinib in later lines among those with BM at any time compared to those without BM (51.3% vs 43.8%). Median real world PFS and OS (95% CI) were shorter among pts with BM at baseline compared to those never developing BM: 10.2 (8.8, 11.5) vs 15.2 (13.7, 16.1) months, and 19.4 (17.1, 22.1) vs 30.3 (27.1, 33.8) months, respectively. At the time of data collection, 77.3% of pts with BM at baseline were deceased compared to 52.5% pts with no BM. Conclusions: More than one third of pts included in REFLECT had BM at any time. Uncommon EGFR variants at baseline were observed more frequently in pts with BM. mPFS and mOS were shorter in pts with BM at baseline compared to those never developing BM. These data highlight the need for improved treatment and CNS control in pts with EGFRm NSCLC. Clinical trial information: NCT04031898.

Sign in / Sign up

Export Citation Format

Share Document