Statement validity assessment: Inter-rater reliability of criteria-based content analysis in the mock-crime paradigm

Studies that are based on content analyses of portions of the Financial Accounting Standards Board Public Record have appeared regularly in accounting and business literature since 1978. Inter-rater reliability is a crucial determinant of the validity of content analyses, yet none of the studies based on content analysis of the Public Record report any measure of inter-rater reliability. This study provides some evidence of the degree of inter-rater reliability of these studies. Krippendorffs coefficient of agreement, a measure of inter-rater reliability is derived for each of eight issues from four raters performing a content analysis of respondent letters in the Public Record volume, Exposure Draft: Accounting for Certain Acquisitions of Banking or Thrift Institutions. In general, the coefficients indicated that extreme caution should be exercised in making inferences from studies based on content analyses of the Financial Accounting Standards Board Public Record.

Download Full-text

Verbal content analysis for the study of Verisimilitude (V1) in the System for Analysis of Validity in Evaluation (SAVE)

South Florida Journal of Development ◽

10.46932/sfjdv2n2-204 ◽

2021 ◽

Vol 2 (2) ◽

pp. 3685-3688

Author(s):

Marta Grau Olivares

Keyword(s):

Content Analysis ◽

External Source ◽

Reality Monitoring ◽

True Statement ◽

The Unconscious ◽

Verbal Content ◽

Verbal Message ◽

Scope Of Application ◽

Internal Coherence ◽

Statement Validity

In the SAVE Metaprotocol (López, et al, 2018), Verisimilitude (V1) refers to that which has the appearance of truth, possibility of being believed, and with "absence of subjective incredibility". Corresponding to this section is everything related to the analysis of the content of the verbal message and which gives internal coherence to the subject's statement. This phase is obtained directly through the interview with the subject, which we record in order to be able to carry out the content analysis later. It can be using a criterial, structural, or linguistic analysis, or simply employing a study of the communicative pattern. Some examples of content analysis are Statement Validity Analysis (SVA) with its Criteria-Based Content Analysis or CBCA and its validity list. The CBCA is composed of 19 criteria used to assess the content of the statements and which to date has been admitted as court evidence in cases of possible child sexual offense. It is administered together with a validity checklist and its value as evidence depends on the scope of application (Köhnken, Manzanero & Scott 2015). The instrument called Reality Monitoring (RM), like the previous one is a tool used to assess the credibility of the testimony and that focuses on the semantic aspects that differentiate a true statement from a false or not very credible one (Raye & Johnson, 1980). Although in this aspect, we should not make the mistake of confusing credibility with veracity, since when assessing a testimony the truth as such does not exist, and depends on the functioning of our memory, which is not like a tape recorder and is not static, but is constantly exposed to changes and distortions. According to this model, the origin of our memories comes from two basic sources: External Memories, generated from the perception of real situations; and Internal Memories, the product of our imagination. The unconscious reasoning process that our mind follows to decide which information comes from an external source and which forms an internal one is called Reality Monitoring. Four qualitative attributes allow us to differentiate between these two types of memories:

Download Full-text

Is This Testimony Truthful, Fabricated, or Based on False Memory?

European Psychologist ◽

10.1027/1016-9040/a000200 ◽

2014 ◽

Vol 19 (3) ◽

pp. 207-220 ◽

Cited By ~ 38

Author(s):

Renate Volbert ◽

Max Steller

Keyword(s):

False Memory ◽

False Memories ◽

True Statement ◽

Content Quality ◽

Validity Assessment ◽

Statement Validity Assessment ◽

Truth Teller ◽

The Individual ◽

Underlying Processes ◽

Content Characteristics

In 1989, Steller and Köhnken presented a systematic compilation of content characteristics for distinguishing between truthful and fabricated testimonies (criteria-based content analysis – CBCA) designed to be applied within a more comprehensive overall diagnostic procedure known as statement validity assessment (SVA; Steller, 1989 ). The subsequent 25 years have seen a marked increase in knowledge about the distinction between experience-based and non-experience-based statements. This supports the SVA approach and permits a better explanation of the underlying processes. The rationale of CBCA is that a true statement differs in content quality from a fabricated account because (a) a truth teller can draw on an episodic autobiographical representation containing a multitude of details, whereas a liar has to relate to scripts containing only general details of an event; and (b) a liar is busier with strategic self-presentation than a truth teller. The present article proposes a modified model of content characteristics that pays greater attention than before to these underlying processes. SVA takes into account that content quality is influenced not only by the veracity of a statement but also by other (personal and contextual) variables that need to be considered in the individual credibility assessment. Theoretical analyses and empirical research do not indicate comparable qualitative differences between true statements and those based on false memories. Witnesses giving testimony based on false memories do not fabricate false statements actively, and they make no effort to conceal a deception; they are not deceiving but mistaken. In these cases, a noncritical application of content criteria can lead to false results. To examine the hypothesis that a statement is based on a false memory, it is necessary to focus on the way in which the statement has emerged and evolved.

Download Full-text

Inter-rater Reliability of Criteria-Based Content Analysis of Children's Statements of Abuse

Journal of Forensic Sciences ◽

10.1111/1556-4029.12816 ◽

2015 ◽

Vol 60 (5) ◽

pp. 1247-1252 ◽

Cited By ~ 3

Author(s):

Gérard Niveau ◽

Marie-Josée Lacasa ◽

Michel Berclaz ◽

Michèle Germond

Keyword(s):

Content Analysis ◽

Rater Reliability

Download Full-text

Staging achilles tendinopathy using ultrasound imaging: the development and investigation of a new ultrasound imaging criteria based on the continuum model of tendon pathology

BMJ Open Sport & Exercise Medicine ◽

10.1136/bmjsem-2019-000699 ◽

2020 ◽

Vol 6 (1) ◽

pp. e000699 ◽

Cited By ~ 3

Author(s):

Wesley Matthews ◽

Richard Ellis ◽

James W Furness ◽

Evelyne Rathbone ◽

Wayne Hing

Keyword(s):

Ultrasound Imaging ◽

Continuum Model ◽

Weighted Kappa ◽

Achilles Tendinopathy ◽

Face Validity ◽

Ultrasound Images ◽

Rater Reliability ◽

Validity Assessment ◽

Tendon Pathology ◽

The Continuum

AimTo develop a standardised ultrasound imaging (USI)-based criteria for the diagnosis of tendinopathy that aligns with the continuum model of tendon pathology. Secondary aims were to assess both the intra-rater and inter-rater reliability of the criteria.MethodsA criteria was developed following a face validity assessment and a total of 31 Achilles tendon ultrasound images were analysed. Intra-rater and inter-rater reliability were assessed for overall tendinopathy stage (normal, reactive/early dysrepair or late dysrepair/degenerative) as well as for individual parameters (thickness, echogenicity and vascularity). Quadratic weighted kappa (kw) was used to report on reliability.ResultsIntra-rater reliability was ‘substantial’ for overall tendinopathy staging (kw rater A; 0.77, 95% CI 0.59 to 0.94, rater B; 0.70, 95% CI 0.52 to 0.89) and ranged from ‘substantial’ to ‘almost perfect’ for thickness (kw rater A; 0.75, 95% CI 0.59 to 0.90, rater B; 0.84, 95% CI 0.71 to 0.98), echogenicity (kw rater A; 0.78, 95% CI 0.62 to 0.95, rater B; 0.73, 95% CI 0.58 to 0.89) and vascularity (kw rater A; 0.86, 95% CI 0.74 to 0.98, rater B; 0.89, 95% CI 0.79 to 0.99). Inter-rater reliability ranged from ‘substantial’ to ‘almost perfect’ for overall tendinopathy staging (kw round 1; 0.75, 95% CI 0.58 to 0.91, round 2; 0.81, 95% CI 0.63 to 0.99), thickness (kw round 1; 0.65, 95% CI 0.48 to 0.83, round 2; 0.77, 95% CI 0.60 to 0.93), echogenicity (kw round 1; 0.70, 95% CI 0.54 to 0.85, round 2; 0.76, 95% CI 0.58 to 0.94) and vascularity (kw round 1; 0.89, 95% CI 0.79 to 0.99, round 2; 0.86, 95% CI 0.74 to 0.98). Inter-rater reliability increased from ‘substantial’ in round 1 (kw 0.75, 95% CI 0.58 to 0.91) to ‘almost perfect’ in round 2 (0.81, 95% CI 0.63 to 0.99).ConclusionIntra-rater and inter-rater reliability were ‘substantial’ to ‘almost perfect’ when utilising an USI-based criteria to diagnose Achilles tendinopathy. This is the first study to use the continuum model of tendon pathology to develop an USI-based criteria to diagnose tendinopathy.

Download Full-text

The Reliability of Environmental Measures of the College Alcohol Environment

Journal of Drug Education ◽

10.2190/m66g-t1kf-6gxe-y1yq ◽

2002 ◽

Vol 32 (4) ◽

pp. 287-301 ◽

Cited By ~ 2

Author(s):

John D. Clapp ◽

Mike Whitney ◽

Audrey M. Shillington

Keyword(s):

Content Analysis ◽

College Campuses ◽

Future Research ◽

Environmental Scanning ◽

Alcohol Prevention ◽

Drinking Patterns ◽

Community Settings ◽

Rater Reliability ◽

Environmental Measures ◽

Alcohol Related Problems

Much of what we know about students' drinking patterns and problems related to alcohol use is based on survey research. Although local and national survey data are important to alcohol-prevention projects, they do not sufficiently capture the complexity of the alcohol environment. Environmental prevention approaches to alcohol-related problems have been shown to be effective in community settings and researchers have begun to study and adapt such approaches for use on college campuses. Many environmental approaches require systematic scanning of the campus alcohol environment. This study assessed the inter-rater reliability of two environmental scanning tools (a newspaper content analysis form and a bulletin analysis form) designed to identify alcohol-related advertisements targeting college students. Inter-rater reliability for these forms varied across different rating categories and ranged from poor to excellent. Suggestions for future research are addressed.

Download Full-text

Of Frames and Cultures - A Cross-Cultural Comparison of TV Newscasts

Zeitschrift für Medienpsychologie ◽

10.1026/1617-6383.18.4.160 ◽

2006 ◽

Vol 18 (4) ◽

pp. 160-173 ◽

Cited By ~ 3

Author(s):

Maria Senokozlieva ◽

Oliver Fischer ◽

Gary Bente ◽

Nicole Krämer

Keyword(s):

Content Analysis ◽

Arab World ◽

Methodological Approach ◽

Cross Cultural ◽

Cultural Comparison ◽

Explorative Study ◽

Cross Cultural Comparison ◽

Tv News ◽

The Usa ◽

The Relationship

Abstract. TV news are essentially cultural phenomena. Previous research suggests that the often-overlooked formal and implicit characteristics of newscasts may be systematically related to culture-specific characteristics. Investigating these characteristics by means of a frame-by-frame content analysis is identified as a particularly promising methodological approach. To examine the relationship between culture and selected formal characteristics of newscasts, we present an explorative study that compares material from the USA, the Arab world, and Germany. Results indicate that there are many significant differences, some of which are in line with expectations derived from cultural specifics. Specifically, we argue that the number of persons presented as well as the context in which they are presented can be interpreted as indicators of Individualism/Collectivism. The conclusions underline the validity of the chosen methodological approach, but also demonstrate the need for more comprehensive and theory-driven category schemes.

Download Full-text

Searching Online for Methods of Suicide

Crisis ◽

10.1027/0227-5910/a000701 ◽

2020 ◽

pp. 1-8

Author(s):

Christian Ulrich Eriksen ◽

Flemming Konradsen ◽

Thilde Vildekilde

Keyword(s):

Content Analysis ◽

English Language ◽

Significant Proportion ◽

Background Information ◽

Access To Information ◽

Online Content ◽

Negative Effects ◽

Search Terms ◽

Methods Of Suicide

Abstract. Background: Information on methods of suicide is available online, and access to information on methods of suicide appears to contribute to a small but significant proportion of suicides. There is limited documentation of how methods of suicide are being profiled, as well as what content exists in other languages than English. Aim: We aimed to analyze and compare how methods of suicide are profiled on Danish and English-language websites. Method: We applied a categorization and content analysis of websites describing methods of suicide. Sites were retrieved by applying widely used Danish and English-language search terms. Results: A total of 136 English-language websites and 106 Danish-language websites were included for analysis. Websites were more often categorized as prevention or support sites, academic or policy sites, and against suicide sites than dedicated suicide sites (i.e., pro-suicide sites), or information sites. However, information on methods of suicide was available, and 20.1% and 8.9% of the English and Danish-language sites, respectively, suggested that a particular method of suicide was quick, easy, painless, or certain to result in death. Limitations: Only one author coded and analyzed all websites. A further operationalization of the content analysis checklist is warranted to increase reliability. Conclusion: The websites primarily had a prevention or anti-suicide focus, but information on methods of suicide was available, requiring an increased focus on how to diminish the negative effects of harmful online content.

Download Full-text