Reliability and Validity in Automated Content Analysis

In light of the research in other chapters in this volume, this chapter considers some of the important and as-yet-unresolved methodological issues in automated content analysis. The chapter focuses on DICTION in particular, but the concerns raised here also apply to automated content analytic techniques more generally. Those concerns are twofold. First, the chapter considers the importance of aggregation for the reliability of content analyses, both human- and computer-coded. Second, the chapter reviews some of the difficulties associated with testing the validity of the kinds of complex (latent) variables on which DICTION is focused. On the whole, the chapter argues that this (and its companion) volume reflect just some of the many possibilities for DICTION-based analyses, but researchers must proceed with a certain amount of caution as well.

Environmental Disclosures and Impression Management

A significant stream of social and environmental accounting research investigates the relationship between a corporation’s self-reported disclosures of its own social responsibility and environmental activities and third-party evaluations of that corporation’s actual social responsibility and environmental performance. Generally, researchers have utilized one of two theories to motivate and test this relationship. One theory—signaling or voluntary disclosure theory—argues that corporations with superior corporate social responsibility or environmental performance use disclosure to signal to interested parties a level of performance that poorer corporate performers cannot disclose. A second theory—legitimacy or impression management theory—argues that corporations use disclosures to manage impressions, often masking their actual social responsibility and environmental performance. In this chapter, the authors seek to comment on how DICTION has been and can be utilized to advance this stream of social and environmental accounting research.

Optimal Portfolio Construction Using Qualitative and Quantitative Signals

Finance and accounting research has recently focused on extracting the tone or sentiment of a document (such as an earnings press release, cover story about a company, or management’s presentations to analysts) by using positive or negative words/phrases in the document. This chapter shows that signals based on tone or sentiment (extracted from qualitative data) can achieve abnormal returns, and in some studies, incremental abnormal returns beyond quantitative signals. In this chapter, the authors exploit the information content of qualitative data in addition to quantitative signals in selecting optimal portfolios. Using optimization techniques developed by Brandt, Santa-Clara, and Valkonov (2009), and later extended by Hand and Green (2011), the authors show that significantly higher returns can be obtained by combining quantitative and qualitative data obtained from firms’ Management Discussion and Analysis (MD&A) sections of their Form 10-Q (10-K) SEC filings than using quantitative signals.

A Review of Textual Analysis in Economics and Finance

This chapter provides an overview of studies in finance and economics that use automated textual analysis algorithms to analyze the informational content of a wide variety of texts, including journalist’s coverage of news events, management-issued statements, and Internet stock message boards. In these studies, researchers quantify qualitative information with one or more of the following textual tone variables: textual negativity, positivity, and uncertainty. The studies show that textual negativity and positivity conveyed by managers and journalists helps predict future firm level and aggregate economic activity. Textual negativity and positivity, in turn, affect asset prices, although the information is sometimes incorporated with some delay. Textual uncertainty of management-issued information is associated with future cash flow volatility and asset price volatility. In contrast, the textual tone of stock market message board postings is, on average, not very informative in explaining asset prices. The use of automated textual analysis algorithms in finance and economics is a relatively new phenomenon and research in this area is expected to continue to grow.

The Impact of CEO Charismatic Rhetoric on Firm Performance

Leaders perceived as charismatic tend to have transformational effects on both individuals and organizations. Building on strategic leadership and charismatic leadership theories, this study explores the degree to which one type of charismatic leadership behavior, CEO charismatic rhetoric, influences firm performance. To do so, this chapter examines the charismatic rhetoric of CEOs found in their annual letter to shareholders for large firms listed on the S&P 500 stock index over the years 2001 to 2005. In examining shareholder letters, DICTION’s predefined dictionaries were combined to create measures of charismatic rhetoric dimensions consistent with charismatic leadership theory. Results reveal that, contrary to expectations, charismatic rhetoric dimensions display a significant negative relationship to measures of firm performance. Further, outsider CEOs were found to express greater levels of charismatic rhetoric than insider CEOs.

Fraud Detection and Corporate Filings

This chapter is focused on detection of fraud in organizations by using content-based analysis on the annual reports issued by firms. Unlike a variety of previous work on fraud detection that have used quantitative financial information, this research examines qualitative textual content in annual reports to decipher evidence of fraud embedded in these reports through careful examination of the tone, content, and emphasis across reports. The basic premise of this research is that organizations tend to camouflage negative findings to sound less damaging. The real intent of the writer is hidden in content but can be revealed through structured content analysis. Using a corpus of annual reports of companies where fraud has occurred and juxtaposed with companies where fraud has not been detected, this study systematically examines the differences in the use of language. The results of this study reveal that fraudulent annual reports exhibit themes of optimism, variety, complexity, activity, and passivity. On the other hand, nonfraudulent annual reports exhibit themes of certainty and realism.

Disclosure Alignment and Transparency Signaling in CSR Reports

This chapter investigates companies' disclosure alignment and transparency signaling within the 2011 CSR annual reports of 36 U.S. firms in the Global Forbes 2000. DICTION 6.0 was used for the text analysis. The study found that CSR reports are fairly similar to corporate financial annual reports but can be classified more accurately as a hybrid discourse with normative elements matching genres emanating from science, business, government, religion, and social movements. Despite the relatively short time that CSR reports have been in existence, this chapter provides evidence that CSR reporting has become institutionalized quickly. The measures of transparency signaling and disclosure alignment reveal that companies know the rules for reporting and are following them. CSR reporting on societal and environmental impacts and performance receive the most focused discussion, while human rights, labor, and product responsibility discussions are at minimal levels. The study ends with future research directions.

Executives’ Use of Emotional Language and Investor Reactions

Drawing on theories from social judgment and persuasion, this chapter proposes that investors will be sensitive to the use of emotional language by top executives in their communication. In addition, the effect of emotional language on investors will depend on investors’ prior beliefs about the firm. Using a sample of 8,990 transcripts of presentations by executives of 632 organizations at investor conferences between 2004 and 2010, results support the prediction that executives’ use of emotional language has a significant influence on investors and that different ways of expressing emotion have differential effects on investor judgment. Moreover, the effect of emotional language is stronger when the firm’s uncertainty level is high and growth prospect is low. The results provide insights into how organizations can use language strategically to manage their relationships with stock investors, thereby contributing to a growing literature on symbolic management of organization.

Exploring Signs of Hubris in CEO Language

This chapter focuses on the potential for DICTION to identify inaptly hubristic language of Chief Executive Officers. CEO hubris is examined as a syndrome possessing identifiable symptoms that have possible links to CEO language and DICTION measures. The authors make some exploratory predictions regarding the nature of these links and assess them using the text of speeches of the former long-serving CEO of British Petroleum, John Browne. In a quest for validation, they then apply the results of that assessment to some oral and written examples of the discourse of News Corporation’s CEO, Rupert Murdoch. The results, although mixed, show some promise regarding the usefulness of DICTION in identifying hubristic CEO-speak. One interesting finding is that DICTION’s calculated variable, Variety, is associated strongly and consistently with the language use of Browne and Murdoch, evidencing a high Type Token Ratio. The authors attribute this result to Browne and Murdoch possibly experiencing low anxiety as they strived to manage impressions of themselves by inducing the outside world to “know” what they were seemingly utterly convinced about - their own superiority. The chapter concludes by suggesting some refinements and extensions of the study.

Measuring the Tone of Accounting and Financial Narrative

Research in accounting and finance has measured the tone of financial narrative using word frequency counts based mainly on four different wordlists: 1) a wordlist developed in Henry’s (2006, 2008) analysis of earnings announcements (Henry Wordlist); 2) a wordlist developed in Loughran and McDonald’s (2011) analysis of 10-K filings (LM Wordlist); 3) a wordlist from DICTION (DICTION Wordlist) software developed by Roderick Hart; and 4) a wordlist from the General Inquirer program (GI Wordlist) developed by social psychologist Philip Stone. This chapter examines alternative measures of the tone of narrative in earnings press releases based on these word lists, explores the statistical relations among the alternative measures, and tests whether those relations vary depending on aspects of the earnings news being announced and other factors.