scholarly journals Assessing the performance of methodological search filters to improve the efficiency of evidence information retrieval: five literature reviews and a qualitative study

2017 ◽  
Vol 21 (69) ◽  
pp. 1-148 ◽  
Author(s):  
Carol Lefebvre ◽  
Julie Glanville ◽  
Sophie Beale ◽  
Charles Boachie ◽  
Steven Duffy ◽  
...  
Keyword(s):  
Decision Making ◽  
Performance Measures ◽  
Diagnostic Tests ◽  
Reference Standard ◽  
Performance Information ◽  
Search Filter ◽  
Filter Performance ◽  
Literature Reviews ◽  
Search Filters ◽  
Study Designs

BackgroundEffective study identification is essential for conducting health research, developing clinical guidance and health policy and supporting health-care decision-making. Methodological search filters (combinations of search terms to capture a specific study design) can assist in searching to achieve this.ObjectivesThis project investigated the methods used to assess the performance of methodological search filters, the information that searchers require when choosing search filters and how that information could be better provided.MethodsFive literature reviews were undertaken in 2010/11: search filter development and testing; comparison of search filters; decision-making in choosing search filters; diagnostic test accuracy (DTA) study methods; and decision-making in choosing diagnostic tests. We conducted interviews and a questionnaire with experienced searchers to learn what information assists in the choice of search filters and how filters are used. These investigations informed the development of various approaches to gathering and reporting search filter performance data. We acknowledge that there has been a regrettable delay between carrying out the project, including the searches, and the publication of this report, because of serious illness of the principal investigator.ResultsThe development of filters most frequently involved using a reference standard derived from hand-searching journals. Most filters were validated internally only. Reporting of methods was generally poor. Sensitivity, precision and specificity were the most commonly reported performance measures and were presented in tables. Aspects of DTA study methods are applicable to search filters, particularly in the development of the reference standard. There is limited evidence on how clinicians choose between diagnostic tests. No published literature was found on how searchers select filters. Interviewing and questioning searchers via a questionnaire found that filters were not appropriate for all tasks but were predominantly used to reduce large numbers of retrieved records and to introduce focus. The Inter Technology Appraisal Support Collaboration (InterTASC) Information Specialists’ Sub-Group (ISSG) Search Filters Resource was most frequently mentioned by both groups as the resource consulted to select a filter. Randomised controlled trial (RCT) and systematic review filters, in particular the Cochrane RCT and the McMaster Hedges filters, were most frequently mentioned. The majority indicated that they used different filters depending on the requirement for sensitivity or precision. Over half of the respondents used the filters available in databases. Interviewees used various approaches when using and adapting search filters. Respondents suggested that the main factors that would make choosing a filter easier were the availability of critical appraisals and more detailed performance information. Provenance and having the filter available in a central storage location were also important.LimitationsThe questionnaire could have been shorter and could have included more multiple choice questions, and the reviews of filter performance focused on only four study designs.ConclusionsSearch filter studies should use a representative reference standard and explicitly report methods and results. Performance measures should be presented systematically and clearly. Searchers find filters useful in certain circumstances but expressed a need for more user-friendly performance information to aid filter choice. We suggest approaches to use, adapt and report search filter performance. Future work could include research around search filters and performance measures for study designs not addressed here, exploration of alternative methods of displaying performance results and numerical synthesis of performance comparison results.FundingThe National Institute for Health Research (NIHR) Health Technology Assessment programme and Medical Research Council–NIHR Methodology Research Programme (grant number G0901496).

scholarly journals Translation and validation of PubMed and Embase search filters for identification of systematic reviews, intervention studies, and observational studies in the field of first aid

2021 ◽  
Vol 109 (4) ◽  
Author(s):  
Bert Avau ◽  
Hans Van Remoortel ◽  
Emmy De Buck
Keyword(s):  
Systematic Reviews ◽  
Observational Studies ◽  
Intervention Studies ◽  
First Aid ◽  
Search Filter ◽  
Filter Performance ◽  
Ovid Medline ◽  
Search Filters ◽  
Study Type ◽  
Evidence Summaries

Objective: The aim of this project was to validate search filters for systematic reviews, intervention studies, and observational studies translated from Ovid MEDLINE and Embase syntax and used for searches in PubMed and Embase.com during the development of evidence summaries supporting first aid guidelines. We aimed to achieve a balance among recall, specificity, precision, and number needed to read (NNR).Methods: Reference gold standards were constructed per study type derived from existing evidence summaries. Search filter performance was assessed through retrospective searches and measurement of relative recall, specificity, precision, and NNR when using the translated search filters. Where necessary, search filters were optimized. Adapted filters were validated in separate validation gold standards.Results: Search filters for systematic reviews and observational studies reached recall of ≥85% in both PubMed and Embase. Corresponding specificities for systematic review filters were ≥96% in both databases, with a precision of 9.7% (NNR 10) in PubMed and 5.4% (NNR 19) in Embase. For observational study filters, specificity, precision, and NNR were 68%, 2%, and 51 in PubMed and 47%, 0.8%, and 123 in Embase, respectively. These filters were considered sufficiently effective. Search filters for intervention studies reached a recall of 85% and 83% in PubMed and Embase, respectively. Optimization led to recall of ≥95% with specificity, precision, and NNR of 49%, 1.3%, and 79 in PubMed and 56%, 0.74%, and 136 in Embase, respectively.Conclusions: We report validated filters to search for systematic reviews, observational studies, and intervention studies in guideline projects in PubMed and Embase.com.

scholarly journals Integrated Care Search: Development and validation of a PubMed search filter for retrieving the integrated care research evidence

2020 ◽  
Author(s):  
Raechel A. Damarell ◽  
Suzanne Lewis ◽  
Camilla Trenerry ◽  
Jennifer J. Tieman
Keyword(s):  
Integrated Care ◽  
Gold Standard ◽  
Research Evidence ◽  
Evidence Base ◽  
Standard Test ◽  
Advisory Group ◽  
Search Filter ◽  
Filter Performance ◽  
Search Filters ◽  
Development And Validation

Abstract Background Integrated care is an increasingly important principle for organising healthcare. Integrated care models show promise in reducing resource wastage and service fragmentation whilst improving the accessibility, patient-centredness and quality of care for patients. Those needing reliable access to the growing research evidence base for integrated care can be frustrated by search challenges reflective of the topic's complexity. The aim of this study is to report the empirical development and validation of two search filters for rapid and effective retrieval of integrated care evidence in PubMed. One filter is optimised for recall and the other for precision.Methods An Expert Advisory Group comprising international integrated care experts guided the study. A gold standard test set of citations was formed from screening Handbook Integrated Care chapter references for relevance. This set was divided into a Term Identification Set (20%) for determining candidate terms using frequency analysis; a Filter Development Set (40%) for testing performance of term combinations; and a Filter Validation Set (40%) reserved for confirming final filter performance. In developing the high recall filter, recall was steadily increased while maintaining precision at ≥ 50%. Similarly, the high precision filter sought to maximise precision while keeping recall ≥ 50%. For each term combination tested, an approximation of precision was obtained by reviewing the first 100 citations retrieved in Medline for relevance.Results The gold standard set comprised 534 citations. The search filter optimised for recall ('Broad Integrated Care Search') achieved 86.0%-88.3% recall with corresponding low precision (47%-53%). The search filter optimised for precise searching ('Narrow Integrated Care Search') demonstrated precision of 73%-95% with recall reduced to between 55.9% and 59.8%. These filters are now available as one-click URL hyperlinks in the website of International Foundation for Integrated Care.Conclusions The Broad and Narrow Integrated Care Search filters provide potential users, such as policy makers and researchers, seamless, reliable and ongoing access to integrated care evidence for decision making. These filters were developed according to a rigorous and transparent methodology designed to circumvent the challenges of information retrieval posed by this complex, multifaceted topic.

Does Performance Measurement Improve Strategic Decision Making? Findings From a National Survey of Nonprofit Social Service Agencies

2010 ◽  
Vol 39 (4) ◽  
pp. 571-587 ◽  
Author(s):  
Kelly LeRoux ◽  
Nathaniel S. Wright
Keyword(s):  
Decision Making ◽  
Performance Measurement ◽  
Nonprofit Organizations ◽  
Performance Measures ◽  
Social Service ◽  
Nonprofit Sector ◽  
The United States ◽  
Strategic Decision ◽  
Strategic Decision Making ◽  
Performance Information

Nonprofits have encountered increased pressures for accountability and performance in recent years, both from their funding entities as well as the public. The adoption of performance measurement systems assumes that managers will use performance information to make better decisions. However, little research has focused on performance information use in the nonprofit sector. This study seeks to address this gap in the literature. Using survey data from several hundred nonprofit social service organizations in the United States, this article examines the extent to which reliance on various performance measures improves strategic decision making within nonprofit organizations. Authors find a positive relationship between the range of performance measures used by nonprofits and their level of effectiveness in strategic decision making. Other factors that also contribute to strategic decision making within nonprofits include effective governance, funding diversity, and education level of the executive director.

scholarly journals Integrated Care Search: Development and validation of a PubMed search filter for retrieving the integrated care research evidence

2019 ◽  
Author(s):  
Raechel A. Damarell ◽  
Suzanne Lewis ◽  
Camilla Trenerry ◽  
Jennifer J. Tieman
Keyword(s):  
Integrated Care ◽  
Gold Standard ◽  
Research Evidence ◽  
Evidence Base ◽  
Standard Test ◽  
Advisory Group ◽  
Search Filter ◽  
Filter Performance ◽  
Search Filters ◽  
Development And Validation

Abstract Background Integrated care is an increasingly important principle for organising healthcare. Integrated care models show promise in reducing resource wastage and service fragmentation whilst improving the accessibility, patient-centredness and quality of care for patients. Those needing reliable access to the growing research evidence base for integrated care can be frustrated by search challenges reflective of the topic's complexity. The aim of this study is to report the empirical development and validation of two search filters for rapid and effective retrieval of integrated care evidence in PubMed. One filter is optimised for recall and the other for precision.Methods An Expert Advisory Group comprising international integrated care experts guided the study. A gold standard test set of citations was formed from screening Handbook Integrated Care chapter references for relevance. This set was divided into a Term Identification Set (20%) for determining candidate terms using frequency analysis; a Filter Development Set (40%) for testing performance of term combinations; and a Filter Validation Set (40%) reserved for confirming final filter performance. In developing the high recall filter, recall was steadily increased while maintaining precision at ≥ 50%. Similarly, the high precision filter sought to maximise precision while keeping recall ≥ 50%. For each term combination tested, an approximation of precision was obtained by reviewing the first 100 citations retrieved in Medline for relevance.Results The gold standard set comprised 534 citations. The search filter optimised for recall ('Broad Integrated Care Search') achieved 86.0%-88.3% recall with corresponding low precision (47%-53%). The search filter optimised for precise searching ('Narrow Integrated Care Search') demonstrated precision of 73%-95% with recall reduced to between 55.9% and 59.8%. These filters are now available as one-click URL hyperlinks in the website of International Foundation for Integrated Care.Conclusions The Broad and Narrow Integrated Care Search filters provide potential users, such as policy makers and researchers, seamless, reliable and ongoing access to integrated care evidence for decision making. These filters were developed according to a rigorous and transparent methodology designed to circumvent the challenges of information retrieval posed by this complex, multifaceted topic.

Information visualization to support management decisions

2014 ◽  
Vol 13 (02) ◽  
pp. 407-428 ◽  
Author(s):  
Jasser Al-Kassab ◽  
Zied M. Ouertani ◽  
Giovanni Schiuma ◽  
Andy Neely
Keyword(s):  
Decision Making ◽  
Performance Measures ◽  
Information Visualization ◽  
Knowledge Creation ◽  
Communication Channel ◽  
Critical Role ◽  
Performance Information ◽  
Business Decisions ◽  
Business Decision ◽  
And Control

Information visualization can accelerate perception, provide insight and control, and harness this flood of valuable data to gain a competitive advantage in making business decisions. Although such a statement seems to be obvious, there is a lack in the literature of practical evidence of the benefit of information visualization. The main contribution of this paper is to illustrate how, for a major European apparel retailer, the visualization of performance information plays a critical role in improving business decisions and in extracting insights from Redio Frequency Idetification (RFID)-based performance measures. In this paper, we identify — based on a literature review — three fundamental managerial functions of information visualization, namely as: a communication medium, a knowledge management means, and a decision-support instrument. Then, we provide — based on real industrial case evidence — how information visualization supports business decision-making. Several examples are provided to evidence the benefit of information visualization through its three identified managerial functions. We find that — depending on the way performance information is shaped, communicated, and made interactive — it not only helps decision making, but also offers a means of knowledge creation, as well as an appropriate communication channel.

scholarly journals Integrated Care Search: Development and validation of a PubMed search filter for retrieving the integrated care research evidence

2019 ◽  
Author(s):  
Raechel A. Damarell ◽  
Suzanne Lewis ◽  
Camilla Trenerry ◽  
Jennifer J. Tieman
Keyword(s):  
Integrated Care ◽  
Gold Standard ◽  
Research Evidence ◽  
Evidence Base ◽  
Standard Test ◽  
Advisory Group ◽  
Search Filter ◽  
Filter Performance ◽  
Search Filters ◽  
Development And Validation

Abstract Background Integrated care is an increasingly important principle for organising healthcare. Integrated care models show promise in reducing resource wastage and service fragmentation whilst improving the accessibility, patient-centredness and quality of care for patients. Those needing reliable access to the growing research evidence base for integrated care can be frustrated by search challenges reflective of the topic's complexity. The aim of this study is to report the empirical development and validation of two search filters for rapid and effective retrieval of integrated care evidence in PubMed. One filter is optimised for recall and the other for precision.Methods An Expert Advisory Group comprising international integrated care experts guided the study. A gold standard test set of citations was formed from screening Handbook Integrated Care chapter references for relevance. This set was divided into a Term Identification Set (20%) for determining candidate terms using frequency analysis; a Filter Development Set (40%) for testing performance of term combinations; and a Filter Validation Set (40%) reserved for confirming final filter performance. In developing the high recall filter, recall was steadily increased while maintaining precision at ≥ 50%. Similarly, the high precision filter sought to maximise precision while keeping recall ≥ 50%. For each term combination tested, an approximation of precision was obtained by reviewing the first 100 citations retrieved in Medline for relevance.Results The gold standard set comprised 534 citations. The search filter optimised for recall ('Broad Integrated Care Search') achieved 86.0%-88.3% recall with corresponding low precision (47%-53%). The search filter optimised for precise searching ('Narrow Integrated Care Search') demonstrated precision of 73%-95% with recall reduced to between 55.9% and 59.8%. These filters are now available as one-click URL hyperlinks in the website of International Foundation for Integrated Care.Conclusions The Broad and Narrow Integrated Care Search filters provide potential users, such as policy makers and researchers, seamless, reliable and ongoing access to integrated care evidence for decision making. These filters were developed according to a rigorous and transparent methodology designed to circumvent the challenges of information retrieval posed by this complex, multifaceted topic.

scholarly journals Developing an approach for decision making units integration using multi objective particle swarm optimization

2014 ◽  
Vol 4 (1) ◽  
pp. 48 ◽  
Author(s):  
Abdorrahman Haeri ◽  
Kamran Rezaie ◽  
Seyed Morteza Hatefi
Keyword(s):  
Decision Making ◽  
Particle Swarm Optimization ◽  
Performance Measures ◽  
Particle Swarm ◽  
Final Decision ◽  
Swarm Optimization ◽  
Multi Objective ◽  
Decision Making Unit ◽  
The Difference ◽  
Decision Making Units

In recent years, integration between companies, suppliers or organizational departments attracted much attention. Decision making about integration encounters with major concerns. One of these concerns is which units should be integrated and what is the effect of integration on performance measures. In this paper the problem of decision making unit (DMU) integration is considered. It is tried to integrate DMUs so that the considered criteria are satisfied. In this research two criteria are considered that are mean of efficiencies of DMUs and the difference between DMUs that have largest and smallest efficiencies. For this purpose multi objective particle swarm optimization (MOPSO) is applied. A case with 17 DMUs is considered. The results show that integration has increased both considered criteria effectively.  Additionally this approach can presents different alternatives for decision maker (DM) that enables DM to select the final decision for integration.

A Stochastic Approach to Risk Management and Decision Making in Defined Benefit Pension Schemes

2003 ◽  
Vol 9 (3) ◽  
pp. 493-586 ◽  
Author(s):  
S. Haberman ◽  
C. Day ◽  
D. Fogarty ◽  
M. Z. Khorasanee ◽  
M. McWhirter ◽  
...  
Keyword(s):  
Decision Making ◽  
Performance Measures ◽  
Asset Allocation ◽  
Stochastic Approach ◽  
Free Variable ◽  
Defined Benefit ◽  
Contribution Rate ◽  
And Performance ◽  
Allocation Decisions ◽  
Made In

ABSTRACTThe trustees and sponsors of defined benefit schemes rely on the advice of the Scheme Actuary to make important decisions concerning the funding of the scheme, the investment of its assets, and the use of surplus assets to improve benefits. These decisions have to be made in the face of considerable uncertainty about financial and demographic factors that will affect the future experience of the scheme and its success in meeting various objectives.The traditional actuarial valuation combined with actuarial judgement has played an important role in guiding decision making; but we argue that stochastic methods can add value in certain crucial areas, in particular the financial risk management of defined benefit schemes. Rather than dealing with risk by incorporating margins in the valuation basis, a stochastic approach allows the actuary to evaluate specific and quantifiable risk and performance measures for alternative funding and investment strategies.This paper recommends a framework that, when combined with a suitable stochastic model, measures the risks inherent in contribution rate and asset allocation decisions, allowing better decisions to be made. In doing this, we suggest and apply various risk and performance measures that may be thought appropriate, although our intention is to illustrate their use rather than prescribe them as objective standards. The framework provides the means to explore the trade-offs involved in possible contribution and asset allocation decisions, and points to decision strategies expected to give improved outcomes for the same level of risk. A feature of the approach that marks it out from current asset/liability techniques is that it examines the funding and investment decisions together. It does not derive a contribution rate in the traditional way, but leaves this as free variable, in the same way that the investment decision is taken to be a free variable. Another distinctive feature of our framework is that it is based on projection rather than on valuation, involving stochastic simulation of the experience of the scheme over a time horizon reflecting the concerns of the trustees and the sponsoring employer.The paper provides a case study (based on a model final salary pension scheme) showing the advantages of the framework, and goes on to explain how the results may practically be communicated to trustees and scheme sponsors.

EVIDENCE, VALUES, AND DECISION MAKING

2014 ◽  
Vol 30 (2) ◽  
pp. 233-238 ◽  
Author(s):  
Michael D. Rawlins
Keyword(s):  
Decision Making ◽  
Cost Effectiveness ◽  
Observational Study ◽  
Social Values ◽  
Clinical Excellence ◽  
Health Technology ◽  
Evidence Base ◽  
Social Value ◽  
Value Judgments ◽  
Study Designs

Background: The evidence supporting the use of new, or established, interventions may be derived from either (or both) experimental or observational study designs. Although a rigorous examination of the evidence base for clinical and cost-effectiveness is essential, it is never sufficient, and those undertaking a health technology assessment (HTA) also have to exercise judgments.Methods: The basis for this discussion is largely from the author's experience as chairman of the national Institute for Health and Clinical Excellence (NICE).Results: The judgments necessary for HTA to make are twofold. Scientific judgments relate to the interpretation of the science. Social value judgments are concerned with the ethical principles, preferences, culture, and aspirations of society.Conclusions: How scientific and social value judgments might be most appropriately captured is a challenge for all HTA agencies. Although competent HTA bodies should be able to exercise scientific judgments they have no legitimacy to impose their own social values. These must ultimately be informed by the general public.

scholarly journals Development of an efficient search filter to retrieve systematic reviews from PubMed

2021 ◽  
Vol 109 (4) ◽  
Author(s):  
José Antonio Salvador-Oliván ◽  
Gonzalo Marco-Cuenca ◽  
Rosario Arquero-Avilés
Keyword(s):  
Systematic Review ◽  
Decision Making ◽  
Systematic Reviews ◽  
Search Strategy ◽  
Average Precision ◽  
Search Filter ◽  
Search Terms ◽  
Mesh Terms ◽  
Cochrane Database ◽  
Relevant Publication

Objective: Locating systematic reviews is essential for clinicians and researchers when creating or updating reviews and for decision-making in health care. This study aimed to develop a search filter for retrieving systematic reviews that improves upon the performance of the PubMed systematic review search filter.Methods: Search terms were identified from abstracts of reviews published in Cochrane Database of Systematic Reviews and the titles of articles indexed as systematic reviews in PubMed. Both the precision of the candidate terms and the number of systematic reviews retrieved from PubMed were evaluated after excluding the subset of articles retrieved by the PubMed systematic review filter. Terms that achieved a precision greater than 70% and relevant publication types indexed with MeSH terms were included in the filter search strategy.Results: The search strategy used in our filter added specific terms not included in PubMed’s systematic review filter and achieved a 61.3% increase in the number of retrieved articles that are potential systematic reviews. Moreover, it achieved an average precision that is likely greater than 80%.Conclusions: The developed search filter will enable users to identify more systematic reviews from PubMed than the PubMed systematic review filter with high precision.

Sign in / Sign up

Export Citation Format

Share Document