Formative Evaluation of Ontology Learning Methods for Entity Discovery by Using Existing Ontologies as Reference Standards

2013 ◽  
Vol 52 (04) ◽  
pp. 308-316 ◽  
Author(s):  
K. J. Mitchell ◽  
W. W. Chapman ◽  
G. K. Savova ◽  
N. Sioutos ◽  
D. L. Rubin ◽  
...  
Keyword(s):  
Formative Evaluation ◽  
Performance Metrics ◽  
Recognition System ◽  
Reference Standards ◽  
Entity Recognition ◽  
Ontology Development ◽  
Reference Standard ◽  
Ontology Learning ◽  
Step Method ◽  
Optimum Parameters

SummaryObjective: Developing a two-step method for formative evaluation of statistical Ontology Learning (OL) algorithms that leverages existing biomedical ontologies as reference standards.Methods: In the first step optimum parameters are established. A ‘gap list’ of entities is generated by finding the set of entities present in a later version of the ontology that are not present in an earlier version of the ontology. A named entity recognition system is used to identify entities in a corpus of biomedical documents that are present in the ‘gap list’, generating a reference standard. The output of the algorithm (new entity candidates), produced by statistical methods, is subsequently compared against this reference standard. An OL method that performs perfectly will be able to learn all of the terms in this reference stand ard. Using evaluation metrics and precision-recall curves for different thresholds and parameters, we compute the optimum parameters for each method. In the second step, human judges with exper tise in ontology development evaluate each candidate suggested by the algorithm con figured with the optimum parameters previously established. These judgments are used to compute two performance metrics developed from our previous work: Entity Suggestion Rate (ESR) and Entity Acceptance Rate (EAR).Results: Using this method, we evaluated two statistical OL methods for OL in two medical domains. For the pathology domain, we obtained 49% ESR, 28% EAR with the Lin method and 52% ESR, 39% EAR with the Church method. For the radiology domain, we obtain 87% ESA, 9% EAR using Lin method and 96% ESR, 16% EAR using Church method.Conclusion: This method is sufficiently general and flexible enough to permit comparison of any OL method for a specific corpus and ontology of interest.

scholarly journals Development and Formative Evaluation of a Low-Fidelity Equine Castration Model for Veterinary Education

2021 ◽  
Vol 8 ◽  
Author(s):  
M. Katie Sheats ◽  
Megan J. Burke ◽  
James B. Robertson ◽  
Katherine E. Fiebrandt ◽  
Callie A. Fogle
Keyword(s):  
Student Performance ◽  
Student Attitudes ◽  
Formative Evaluation ◽  
Performance Metrics ◽  
Early Stage ◽  
Future Research ◽  
Student Survey ◽  
Significant Difference ◽  
Veterinary Faculty ◽  
Fidelity Model

Entrustable Professional Activities (EPAs) are units of activity that early-stage professionals perform in the workplace that necessitate simultaneous integration of multiple competencies. EPA #6 requires students to perform a common surgical procedure on a stable patient, including pre-operative and post-operative management. Castration is one of the most common surgeries performed by equine primary care practitioners and is considered an “entry-level competency” for veterinary graduates entering equine private practice, however, to our knowledge there are no equine castration models available for veterinary student education. Therefore, we developed an inexpensive, low-fidelity model of equine field castration and evaluated it using a mixed-methods approach. Two different groups of students, with or without model experience, completed surveys before and after live horse castration. Students who used the model also completed model specific surveys. Videos of the students completing the model were evaluated by at least two different equine veterinary faculty using a 15-point rubric, and inter-rater reliability of the rubric was determined. After completing the model, students reflected on strengths and weaknesses of their performance. From our student survey results, we determined that student attitudes toward the model were mostly positive. Interestingly, there were several student attitudes toward the model that became significantly more favorable after live horse castration. Prior to live horse castration, there was no significant difference in confidence in model vs. no-model groups. Following live horse castration, students who used the model had higher confidence in procedure preparation and hand-ties than students who did not use the model, but they had lower scores for confidence during patient recovery. When reflecting on model castration, students most commonly cited preparation and surgical description as strengths, and ligature placement and hand-ties as weaknesses. Experts provided several suggestions to improve the model, including incorporation of emasculators and the need for better model stabilization. Our findings suggest that both students and veterinary educators feel that this low-fidelity model has educational value. Rubric performance metrics were favorable, but additional steps are needed to improve grading consistency among educators. Future research will determine whether student performance on the model is predictive of competence score during live-horse castration.

scholarly journals Integration of digital maxillary dental casts with 3D facial images in orthodontic patients:

2019 ◽  
Vol 90 (3) ◽  
pp. 397-404 ◽  
Author(s):  
Zhuoxing Xiao ◽  
Zijin Liu ◽  
Yan Gu
Keyword(s):  
Intraclass Correlation ◽  
Three Dimensional ◽  
Correlation Coefficients ◽  
Reference Standards ◽  
Clinical Diagnostics ◽  
Step Method ◽  
Intraclass Correlation Coefficients ◽  
Orthodontic Patients ◽  
Facial Images ◽  
Dental Casts

ABSTRACT Objective To evaluate three-dimensional (3D) accuracy and reliability of nonradiographic dentofacial images integrated with a two-step method. Methods 3D facial images, cone-beam computed tomography (CBCT) images and digital maxillary dental casts were obtained from 20 pre-orthodontic subjects. Digital dental casts were integrated into 3D facial images using a two-step method based on the anterior tooth area. 3D coordinate values of five dental landmarks were identified in both dentofacial images and CBCT images. The accuracy of the integration method was assessed with paired t-tests between dentofacial images and CBCT-based reference standards. Intraclass correlation coefficients (ICCs) were assessed for the reliability of dentofacial images and CBCT-based images. Analysis of variance and Kruskal-Wallis tests evaluated the accuracy of the method in different dimensions. Results There was no statistical difference between dentofacial images and CBCT reference standards in both translational and rotational dimensions (P > .05). Translational mean absolute errors for full dentitions were within 0.42 mm and ICCs were over 0.998 in x, y, and z directions. Rotational mean absolute errors for full dentitions were within 0.92° and ICCs over 0.734 in pitch, yaw, and roll orientations. Integration errors were significantly greater in the first molar, z-translation, and pitch rotation (P < .05). Conclusions Integrating 3D dentofacial images with the two-step method is precise and acceptable for clinical diagnostics and scientific purposes. Errors were greater in the molar region, z-translation, and pitch rotation.

Assessment of Protein Quality Methodology for Infant Formulas

1985 ◽  
Vol 68 (4) ◽  
pp. 680-683
Author(s):  
Geraldine Vaughan Mitchell ◽  
Mamie Young Jenkins
Keyword(s):  
Quality Evaluation ◽  
Protein Quality ◽  
Reference Standards ◽  
Infant Formulas ◽  
Casein Diet ◽  
Reference Standard ◽  
Powdered Milk ◽  
The Third ◽  
Net Protein

Abstract Rat bioassay was used to assess the protein quality of powdered infant formulas and to evaluate the feasibility of using modified casein diets (containing the same source and level of fat and carbohydrate contributed by the infant formulas) as reference standards. Modification of the casein diet to match the milk-based formulas caused a significant reduction in weekly protein efficiency ratios (PER) and net protein ratios (NPR) at the third and fourth weeks. Modification of the casein diet to simulate the soy-based formulas had no significant effect on NPR values; PER values were more varied. When PER and NPR values of the powdered milk-based formulas were expressed relative to the unmodified reference standard, the relative values were lower than when each matched reference was used. With few exceptions, the relative weekly PER values of the soy-based formulas were similar regardless of the standard used. The relative NPR values of the formulas had a pattern similar to the relative PER values. The data indicate that protein quality evaluation of infant formulas using rat bioassay warrants the use of matched casein reference diets for each type of formula.

Sign in / Sign up

Export Citation Format

Share Document