Multi-Institutional FASTQ File Exchange as a Means of Proficiency Testing for Next-Generation Sequencing Bioinformatics and Variant Interpretation

Context.—Most current proficiency testing challenges for next-generation sequencing assays are methods-based proficiency testing surveys that use DNA from characterized reference samples to test both the wet-bench and bioinformatics/dry-bench aspects of the tests. Methods-based proficiency testing surveys are limited by the number and types of mutations that either are naturally present or can be introduced into a single DNA sample. Objective.—To address these limitations by exploring a model of in silico proficiency testing in which sequence data from a single well-characterized specimen are manipulated electronically. Design.—DNA from the College of American Pathologists reference genome was enriched using the Illumina TruSeq and Life Technologies AmpliSeq panels and sequenced on the MiSeq and Ion Torrent platforms, respectively. The resulting data were mutagenized in silico and 26 variants, including single-nucleotide variants, deletions, and dinucleotide substitutions, were added at variant allele fractions (VAFs) from 10% to 50%. Participating clinical laboratories downloaded these files and analyzed them using their clinical bioinformatics pipelines. Results.—Laboratories using the AmpliSeq/Ion Torrent and/or the TruSeq/MiSeq participated in the 2 surveys. On average, laboratories identified 24.6 of 26 variants (95%) overall and 21.4 of 22 variants (97%) with VAFs greater than 15%. No false-positive calls were reported. The most frequently missed variants were single-nucleotide variants with VAFs less than 15%. Across both challenges, reported VAF concordance was excellent, with less than 1% median absolute difference between the simulated VAF and mean reported VAF. Conclusions.—The results indicate that in silico proficiency testing is a feasible approach for methods-based proficiency testing, and demonstrate that the sensitivity and specificity of current next-generation sequencing bioinformatics across clinical laboratories are high.

Download Full-text

Performance Comparison of Different Analytic Methods in Proficiency Testing for Mutations in the BRAF, EGFR, and KRAS Genes: A Study of the College of American Pathologists Molecular Oncology Committee

Archives of Pathology & Laboratory Medicine ◽

10.5858/arpa.2018-0396-cp ◽

2019 ◽

Vol 143 (10) ◽

pp. 1203-1211 ◽

Cited By ~ 2

Author(s):

Joel T. Moncur ◽

Angela N. Bartley ◽

Julia A. Bridge ◽

Suzanne Kamel-Reid ◽

Alexander J. Lazar ◽

...

Keyword(s):

Next Generation Sequencing ◽

Proficiency Testing ◽

Critical Issue ◽

Performance Comparison ◽

Test Method ◽

Next Generation ◽

Molecular Oncology ◽

Assay Methods ◽

Commercial Kits ◽

Generation Sequencing

Context.— The performance of laboratory testing has recently come under increased scrutiny as part of important and ongoing debates on regulation and reimbursement. To address this critical issue, this study compares the performance of assay methods, using either commercial kits or assays designed and implemented by single laboratories (“home brews”), including next-generation sequencing methods, on proficiency testing provided by the College of American Pathologists Molecular Oncology Committee. Objective.— To compare the performance of different assay methods on College of American Pathologists proficiency testing for variant analysis of 3 common oncology analytes: BRAF, EGFR, and KRAS. Design.— There were 6897 total responses across 35 different proficiency testing samples interrogating 13 different variants as well as wild-type sequences for BRAF, EGFR, and KRAS. Performance was analyzed by test method, kit manufacturer, variants tested, and preanalytic and postanalytic practices. Results.— Of 26 reported commercial kits, 23 achieved greater than 95% accuracy. Laboratory-developed tests with no kit specified demonstrated 96.8% or greater accuracy across all 3 analytes (1123 [96.8%] acceptable of 1160 total responses for BRAF; 848 [97.5%] acceptable of 870 total responses for EGFR; 942 [97.0%] acceptable of 971 total responses for KRAS). Next-generation sequencing platforms (summed across all analytes and 2 platforms) demonstrated 99.4% accuracy for these analytes (165 [99.4%] acceptable of 166 total next-generation sequencing responses). Slight differences in performance were noted among select commercial assays, dependent upon the particular design and specificity of the assay. Wide differences were noted in the lower limits of neoplastic cellularity laboratories accepted for testing. Conclusions.— These data demonstrate the high degree of accuracy and comparable performance across all laboratories, regardless of methodology. However, care must be taken in understanding the diagnostic specificity and reported analytic sensitivity of individual methods.

Download Full-text

Proficiency Testing of Standardized Samples Shows Very High Interlaboratory Agreement for Clinical Next-Generation Sequencing–Based Oncology Assays

Archives of Pathology & Laboratory Medicine ◽

10.5858/arpa.2018-0336-cp ◽

2018 ◽

Vol 143 (4) ◽

pp. 463-471 ◽

Cited By ~ 9

Author(s):

Jason D. Merker ◽

Kelly Devereaux ◽

A. John Iafrate ◽

Suzanne Kamel-Reid ◽

Annette S. Kim ◽

...

Keyword(s):

Next Generation Sequencing ◽

Proficiency Testing ◽

False Negative ◽

High Specificity ◽

Variant Allele ◽

Next Generation ◽

Single Nucleotide Variants ◽

Sequencing Platform ◽

Negative Results ◽

Generation Sequencing

Context.— Next-generation sequencing–based assays are being increasingly used in the clinical setting for the detection of somatic variants in solid tumors, but limited data are available regarding the interlaboratory performance of these assays. Objective.— To examine proficiency testing data from the initial College of American Pathologists (CAP) Next-Generation Sequencing Solid Tumor survey to report on laboratory performance. Design.— CAP proficiency testing results from 111 laboratories were analyzed for accuracy and associated assay performance characteristics. Results.— The overall accuracy observed for all variants was 98.3%. Rare false-negative results could not be attributed to sequencing platform, selection method, or other assay characteristics. The median and average of the variant allele fractions reported by the laboratories were within 10% of those orthogonally determined by digital polymerase chain reaction for each variant. The median coverage reported at the variant sites ranged from 1922 to 3297. Conclusions.— Laboratories demonstrated an overall accuracy of greater than 98% with high specificity when examining 10 clinically relevant somatic single-nucleotide variants with a variant allele fraction of 15% or greater. These initial data suggest excellent performance, but further ongoing studies are needed to evaluate the performance of lower variant allele fractions and additional variant types.

Download Full-text

Proficiency Testing of Standardized Samples Shows High Interlaboratory Agreement for Clinical Next-Generation Sequencing–Based Hematologic Malignancy Assays With Survey Material–Specific Differences in Variant Frequencies

Archives of Pathology & Laboratory Medicine ◽

10.5858/arpa.2019-0352-cp ◽

2020 ◽

Vol 144 (8) ◽

pp. 959-966 ◽

Cited By ~ 1

Author(s):

Alissa Keegan ◽

Julia A. Bridge ◽

Neal I. Lindeman ◽

Thomas A. Long ◽

Jason D. Merker ◽

...

Keyword(s):

Next Generation Sequencing ◽

Proficiency Testing ◽

Hematologic Malignancy ◽

False Negative ◽

False Negative Rate ◽

Hematologic Malignancies ◽

Variant Allele ◽

Next Generation ◽

Single Nucleotide Variants ◽

Generation Sequencing

Context.— As laboratories increasingly turn from single-analyte testing in hematologic malignancies to next-generation sequencing–based panel testing, there is a corresponding need for proficiency testing to ensure adequate performance of these next-generation sequencing assays for optimal patient care. Objective.— To report the performance of laboratories on proficiency testing from the first 4 College of American Pathologists Next-Generation Sequencing Hematologic Malignancy surveys. Design.— College of American Pathologists proficiency testing results for 36 different engineered variants and/or allele fractions as well as a sample with no pathogenic variants were analyzed for accuracy and associated assay performance characteristics. Results.— The overall sensitivity observed for all variants was 93.5% (2190 of 2341) with 99.8% specificity (22 800 of 22 840). The false-negative rate was 6.5% (151 of 2341), and the largest single cause of these errors was difficulty in identifying variants in the sequence of CEBPA that is rich in cytosines and guanines. False-positive results (0.18%; 40 of 22 840) were most likely the result of preanalytic or postanalytic errors. Interestingly, the variant allele fractions were almost uniformly lower than the engineered fraction (as measured by digital polymerase chain reaction). Extensive troubleshooting identified a multifactorial cause for the low variant allele fractions, a result of an interaction between the linearized nature of the plasmid and the Illumina TruSeq chemistry. Conclusions.— Laboratories demonstrated an overall accuracy of 99.2% (24 990 of 25 181) with 99.8% specificity and 93.5% sensitivity when examining 36 clinically relevant somatic single-nucleotide variants with a variant allele fraction of 10% or greater. The data also highlight an issue with artificial linearized plasmids as survey material for next-generation sequencing.

Download Full-text

Multi-laboratory proficiency testing of clinical cancer genomic profiling by next-generation sequencing

Pathology - Research and Practice ◽

10.1016/j.prp.2018.05.020 ◽

2018 ◽

Vol 214 (7) ◽

pp. 957-963 ◽

Cited By ~ 4

Author(s):

Qing Zhong ◽

Ulrich Wagner ◽

Henriette Kurt ◽

Francesca Molinari ◽

Gieri Cathomas ◽

...

Keyword(s):

Next Generation Sequencing ◽

Proficiency Testing ◽

Genomic Profiling ◽

Next Generation ◽

Clinical Cancer ◽

Generation Sequencing

Download Full-text

Three-Year Experience of an External Proficiency Testing Survey for Next-Generation Sequencing-Based Testing for Germline Mutation

Journal of Laboratory Medicine and Quality Assurance ◽

10.15263/jlmqa.2020.42.1.48 ◽

2020 ◽

Vol 42 (1) ◽

pp. 48-53

Author(s):

Moon-Woo Seong ◽

Manjin Kim ◽

Ho-Seob Shin ◽

Sung Im Cho ◽

Sung Sup Park

Keyword(s):

Next Generation Sequencing ◽

Proficiency Testing ◽

Germline Mutation ◽

Next Generation ◽

Generation Sequencing

Download Full-text

Next-Generation Sequencing Somatic and Germline Assay Troubleshooting Guide Derived From Proficiency Testing Data

Archives of Pathology & Laboratory Medicine ◽

10.5858/arpa.2020-0842-cp ◽

2021 ◽

Author(s):

Valentina Nardi ◽

Karen D. Tsuchiya ◽

Annette S. Kim ◽

Lora J. H. Bean ◽

Jaimie G. Halley ◽

...

Keyword(s):

Next Generation Sequencing ◽

Proficiency Testing ◽

False Negative ◽

Gc Content ◽

Next Generation ◽

Improve Performance ◽

Common Causes ◽

Testing Data ◽

Survey Results ◽

Generation Sequencing

Context.— Next-generation sequencing–based assays are increasingly used in clinical molecular laboratories to detect somatic variants in solid tumors and hematologic malignancies and to detect constitutional variants. Proficiency testing data are potential sources of information about challenges in performing these assays. Objective.— To examine the most common sources of unacceptable results from the College of American Pathologists Next-Generation Sequencing Bioinformatics, Hematological Malignancies, Solid Tumor, and Germline surveys, and provide recommendations on how to avoid these pitfalls and improve performance. Design.— The College of American Pathologists next-generation sequencing somatic and germline proficiency testing survey results from 2016 to 2019 were analyzed to identify the most common causes of unacceptable results. Results.— On somatic and germline proficiency testing surveys, 95.9% (18 815/19 623) and 97.8% (33 890/34 641) of all variants were correctly identified, respectively. The most common causes of unacceptable results related to sequencing were false-negative errors in genomic regions that were difficult to sequence because of high GC content. False-positive errors occurred in the context of homopolymers and pseudogenes. Recurrent errors in variant annotation were seen for dinucleotide and duplication variants and included unacceptable transcript selection and outdated variant nomenclature. A small percentage of preanalytic or postanalytic errors were attributed to specimen swaps and transcription errors. Conclusions.— Laboratories demonstrate overall excellent performance for detecting variants in both somatic and germline proficiency testing surveys. Proficiency testing survey results highlight infrequent, but recurrent, analytic and nonanalytic challenges in performing next- generation sequencing–based assays and point to remedies to help laboratories improve performance.

Download Full-text

Assembling and Validating Bioinformatic Pipelines for Next-Generation Sequencing Clinical Assays

Archives of Pathology & Laboratory Medicine ◽

10.5858/arpa.2019-0476-ra ◽

2020 ◽

Vol 144 (9) ◽

pp. 1118-1130 ◽

Cited By ~ 2

Author(s):

Jeffrey A SoRelle ◽

Megan Wachsmann ◽

Brandi L. Cantarel

Keyword(s):

Genetic Variation ◽

Next Generation Sequencing ◽

Proficiency Testing ◽

Real World ◽

Data Exchange ◽

Genetic Alterations ◽

Data Sets ◽

Next Generation ◽

Clinical Laboratories ◽

Generation Sequencing

Context.— Clinical next-generation sequencing (NGS) is being rapidly adopted, but analysis and interpretation of large data sets prompt new challenges for a clinical laboratory setting. Clinical NGS results rely heavily on the bioinformatics pipeline for identifying genetic variation in complex samples. The choice of bioinformatics algorithms, genome assembly, and genetic annotation databases are important for determining genetic alterations associated with disease. The analysis methods are often tuned to the assay to maximize accuracy. Once a pipeline has been developed, it must be validated to determine accuracy and reproducibility for samples similar to real-world cases. In silico proficiency testing or institutional data exchange will ensure consistency among clinical laboratories. Objective.— To provide molecular pathologists a step-by-step guide to bioinformatics analysis and validation design in order to navigate the regulatory and validation standards of implementing a bioinformatic pipeline as a part of a new clinical NGS assay. Data Sources.— This guide uses published studies on genomic analysis, bioinformatics methods, and methods comparison studies to inform the reader on what resources, including open source software tools and databases, are available for genetic variant detection and interpretation. Conclusions.— This review covers 4 key concepts: (1) bioinformatic analysis design for detecting genetic variation, (2) the resources for assessing genetic effects, (3) analysis validation assessment experiments and data sets, including a diverse set of samples to mimic real-world challenges that assess accuracy and reproducibility, and (4) if concordance between clinical laboratories will be improved by proficiency testing designed to test bioinformatic pipelines.

Download Full-text

Next-Generation Sequencing (NGS) Methods Show Superior or Equivalent Performance to Non-NGS Methods on BRAF, EGFR, and KRAS Proficiency Testing Samples

Archives of Pathology & Laboratory Medicine ◽

10.5858/arpa.2018-0394-cp ◽

2019 ◽

Vol 143 (8) ◽

pp. 980-984 ◽

Cited By ~ 2

Author(s):

Lea F. Surrey ◽

Fredrick D. Oakley ◽

Jason D. Merker ◽

Thomas A. Long ◽

Patricia Vasalos ◽

...

Keyword(s):

Next Generation Sequencing ◽

Proficiency Testing ◽

Rapid Expansion ◽

Next Generation ◽

Limited Data ◽

Comparative Performance ◽

Panel Surveys ◽

Molecular Oncology ◽

Next Generation Sequencing Ngs ◽

Generation Sequencing

Context.— There has been a rapid expansion of next-generation sequencing (NGS)–based assays for the detection of somatic variants in solid tumors. However, limited data are available regarding the comparative performance of NGS and non-NGS assays using standardized samples across a large number of laboratories. Objective.— To compare the performance of NGS and non-NGS assays using well-characterized proficiency testing samples provided by the College of American Pathologists (CAP) Molecular Oncology Committee. A secondary goal was to compare the use of preanalytic and postanalytic practices. Design.— A total of 17 343 responses were obtained from participants in the BRAF, EGFR, KRAS, and the Multigene Tumor Panel surveys across 84 different proficiency testing samples interrogating 16 variants and 3 wild-type sequences. Performance and preanalytic/postanalytic practices were analyzed by method. Results.— While both NGS and non-NGS achieved an acceptable response rate of greater than 95%, the overall performance of NGS methods was significantly better than that of non-NGS methods for the identification of variants in BRAF (overall 97.8% versus 95.6% acceptable responses, P = .001) and EGFR (overall 98.5% versus 97.3%, P = .01) and was similar for KRAS (overall 98.8% and 97.6%, P = .10). There were specific variant differences, but in all discrepant cases, NGS methods outperformed non-NGS methods. NGS laboratories also more consistently used preanalytic and postanalytic practices suggested by the CAP checklist requirements than non-NGS laboratories. Conclusions.— The overall analytic performance of both methods was excellent. For specific BRAF and EGFR variants, NGS outperformed non-NGS methods and NGS laboratories report superior adherence to suggested laboratory practices.

Download Full-text