Evaluating Public Deliberation: Including the Audience Perspective

I argue that in evaluating public deliberation, the basic criterion should be how deliberating citizens’ need for usable input is met, rather than how the debaters embody Habermasian consensus-oriented ideals, and I question assessment of “deliberative quality” on that basis, such as the “Discourse Quality Index.” Studies of public deliberation should instead build on an Aristotelian notion of deliberation, on Rawls’s idea of “reasonable disagreement” and on the deliberating audience’s needs. To explore these, we need real-time studies of audience reception of public deliberation. I place the studies I call for in a typology of studies, present a study with novel methodological features and discuss its implications for criteria for public deliberation.

Clarity (User Comments/Interactions)

The ‘clarity’ of user comments is an indicator for the quality of statements made in online discussions. User comments below news content on websites or in social media can be examined to determine whether they are clearly written, that is comprehensible to the reader in terms of form, style, and content. Clarity of user comments is essential for the contribution of a comment to a discussion and the exchange between commenters. Field of application/theoretical foundation: As a complement to various further criteria (e.g. coherence, occurrence of arguments), the variable ‘clarity’ of a user comment indicates the comment’s contribution to the deliberative quality of online discussions. Normative approaches to discourse ethics (e.g. Habermas, 1992) assume that contributions to discussions should be phrased understandably to be more valuable for the success of a discussion. Example studies: Medium Measure Unit of analysis Studies Online; online discussions below news posts Clarity Individual user comment Naab & Küchler (work in progress) Ziegele & Quiring (2015) Info about variables Variable name/definition: Verständlichkeit eines Nutzerkommentars Operationalization/coding instructions: Es wird kodiert, wie leicht sich Leser*innen der Sinn eines Nutzerkommentars erschließt, wie gut er nachvollziehbar ist. Hierbei zählt der Gesamteindruck. Indikatoren für eine hohe Verständlichkeit sind: Ein verständlicher Sprachstil überwiegend Standardsprache ohne akademische Begriffe bzw. Fremdwörter Vermeidung von Reimen oder literarischen Schreibformen Vermeidung von übertriebenem Cyberslang oder Umgangssprache Keine Verwendung von Fremdsprachen Eine klare, wenig verschachtelte Satzstruktur Keine auffälligen Rechtschreib- und Grammatikfehler Eine eindeutige rhetorische Gestaltung durch den Verzicht auf Ironie, Metaphern und abstrakte Bilder Eine hohe Prägnanz der Aussagen im Sinne der Verbindung von „Bedeutungsreichtum mit einem hohen Maß an Klarheit, Angemessenheit, Anschaulichkeit und Einfachheit“ Verzicht auf das Voraussetzen von speziellem Hintergrundwissen, das beim Durchschnittsleser nicht vorausgesetzt werden kann. Der Kodierer kann sich die Frage stellen: Wie leicht erschließt sich mir der Sinn des Kommentars, wie nachvollziehbar ist er? „Baseline“ ist der Code „0“. Level of analysis: einzelner Nutzerkommentar Values: 0/ normal verständlich, 1/ schwer verständlich, 2/ überhaupt nicht verständlich, 99/ nicht eindeutig zuzuordnen Intercoder reliability: The variable showed good performance in tests for intercoder agreement (percentage agreement = 86%; Krippendorf’s alpha = .72) in the study by Ziegele, Breiner, & Quiring (2014). References Habermas, J. (1992). Faktizität und Geltung: Beiträge zur Diskurstheorie des Rechts und des demokratischen Rechtsstaates. Suhrkamp. Naab, T.K. & Küchler, C. (work in progress). Unveröffentlichtes Codebuch aus dem DFG-Projekt „Gegenseitige Sanktionierung unter NutzerInnen von Kommentarbereichen auf Nachrichtenwebseiten und auf Facebook“. Augsburg. Steenbergen, M. R., Bächtiger, A., Spörndli, M., & Steiner, J. (2003). Measuring political deliberation: A discourse quality index. Comparative European Politics, 1(1), 21–48. doi:10.1057/palgrave.cep.6110002 Ziegele, M., Breiner, T., & Quiring, O. (2014). What creates interactivity in online news discussions? An exploratory analysis of discussion factors in user comments on news items. Journal of Communication, 64(6), 1111–1138. doi:10.1111/jcom.12123 Ziegele, M. & Quiring, O. (2015). Codebuch: Der Diskussionswert von Online-Nachrichten. Unveröffentlichtes Codebuch aus dem DFG-Projekt „Vom Nachrichtenwert zum Diskussionswert“. Mainz.

Arguments (User Comments/Interactions)

The occurrence of ‘arguments’ indicates the quality of statements made in online discussions. User comments below news content on websites or in social media can be examined whether they give reasons for or against positions or assertions. The occurrence of arguments in user comments or entire online discussions can be investigated in various ways depending on the exact research focus. Thus, the occurrence of arguments, their number, quality, or content (what reasons are raised?) can be examined. To determine the occurrence and number of arguments, arguments must first be identified. To ascertain the quality of arguments an evaluation standard is necessary (what constitutes high- or low-quality arguments?). Field of application/theoretical foundation: As a complement to various further criteria (e.g. coherence, clarity), the occurrence of arguments in user comments indicates the comments’ contribution to the deliberative quality of online discussions. Normative approaches to discourse ethics (e.g. Habermas, 1992; Steenberger et al., 2003) assume that contributions to a discussion that provide arguments for a position are more valuable to the success of a discussion than contributions that do not provide arguments. In addition, studies with various other theoretical backgrounds analyze arguments in user comments. For instance, the occurrence of arguments in user comments is interpreted as an indicator for factuality. Factuality, in turn, is regarded as a discussion factor comparable to news factors in journalism research (Ziegele, Breiner & Quiring, 2014). Example studies: Medium Measure Unit of analysis Studies Online; online discussions below news posts Occurence of arguments Individual user comment Naab & Küchler (work in progress) Ziegele & Quiring (2015) Info about variables Variable name/definition: Vorhandensein von Argument(en) in einem Nutzerkommentar Operationalization/coding instructions: Argumente sind Aussagen, die dazu dienen sollen, Behauptungen zu begründen oder zu widerlegen. Es wird kodiert, ob ein Nutzerkommentar Argumente verwendet, um eine oder mehrere geäußerte(n) Meinung(en) zu begründen. Von Interesse ist hier, ob die Autor*innen von Nutzerkommentaren ihre eigenen Aussagen mit Argumenten unterstützen oder ob diese unbegründet bleiben. Die Leser*innen müssen sich bei zusammenhängenden Aussagen im Kommentar fragen, ob die Frage nach dem „Warum“ beantwortet wird. Anders gesagt: Gut erkennbar ist ein Argument, wenn man es problemlos mit einem Kausalsatz (weil...) an eine Behauptung anfügen kann oder wenn man eine begründende „Wenn-Dann-Beziehung“ zwischen den Sätzen herstellen kann. Eine Aneinanderreihung von Aussagen zählt nicht als Argumentation. Es geht hier nicht um die Qualität der Argumente. Es wird nur kodiert, ob Begründungen für Aussagen/Behauptungen/Positionen angeführt werden. Level of analysis: einzelner Nutzerkommentar Values: 0/ keine Argumente, 1/ mind. ein Argument, 99/ nicht eindeutig zuzuordnen References Naab, T.K. & Küchler, C. (work in progress). Unveröffentlichtes Codebuch aus dem DFG-Projekt „Gegenseitige Sanktionierung unter NutzerInnen von Kommentarbereichen auf Nachrichtenwebseiten und auf Facebook“. Augsburg. Habermas, J. (1992). Faktizität und Geltung: Beiträge zur Diskurstheorie des Rechts und des demokratischen Rechtsstaates. Suhrkamp. Steenbergen, M. R., Bächtiger, A., Spörndli, M., & Steiner, J. (2003). Measuring political deliberation: A discourse quality index. Comparative European Politics, 1(1), 21–48. doi:10.1057/palgrave.cep.6110002 Ziegele, M., Breiner, T., & Quiring, O. (2014). What creates interactivity in online news discussions? An exploratory analysis of discussion factors in user comments on news items. Journal of Communication, 64(6), 1111–1138. doi:10.1111/jcom.12123 Ziegele, M. & Quiring, O. (2015). Codebuch: Der Diskussionswert von Online-Nachrichten. Unveröffentlichtes Codebuch aus dem DFG-Projekt „Vom Nachrichtenwert zum Diskussionswert“. Mainz.

Europeanisation through Conditionality and Deliberation

This paper analyses the Europeanisation of Kosovo by focusing on conditionality and deliberation as EU enlargement tools. Despite the high presence of EU institutions in Kosovo, the EU does not recognize Kosovo as a country, but offers the possibility of integration as it currently does for all other Western Balkan countries. This paper analyses the way in which the EU applies conditionality towards Kosovo in the pre-accession period and the extent to which this is combined with deliberation. In order to measure conditionality and deliberation, I take as a case study the Stabilization Association Process Dialogue Meeting between EU and Kosovo. For conditionality I present my own methodology, whereas for deliberation I apply Discourse Quality Index (DQI). I conclude, that EU uses deliberationin the process of conditionality, whereas Kosovo representatives use conditionality very rarely.

Deliberation level of constitutional debates in Georgian Parliament

In the given paper a discourse quality index (DQI) tool, rooted to criteria elaborated by Jurgen Habermas’s in his discourse ethics, will be modified and used to assess the deliberation level of the 1995 Georgian Parliament. The methodology will accurately represent important principles of deliberation. Due to its focus on observable behavior and its detailed coding instructions, a discourse quality index can be a reliable measurement of the quality of political debates. The DQI for a parliamentary debate in the example of the 1995 parliament of Georgia will be illustrated in the given article. The parliamentarian debates concerning the adoption of the 1995 constitution of Georgia, according to its importance, will be taken as the specific case to be analysed. In the framework of the research, scholars will study and analyse over 200 pages of stenographic recordings of the parliamentary debates connected to the mentioned topic. The DQI score will be evaluated based on the analyses of the stenographic records.

The Role of Precision in Spatial Narratives: Using a Modified Discourse Quality Index to Measure the Quality of Deliberative Spatial Data

The importance of spatial precision in geographic information science is not limited to quantitative data. As spatial data can also exist in qualitative form, modifying a discourse quality index from the field of discourse ethics can help understand whether mentioning specific spatial locations changes the quality of spatial narratives. The discourse quality index was modified by incorporating an item into the index that detected the presence and magnitude of a spatial precision construct. The spatial narratives analyzed with this modified index were 151 public comments submitted during a public policy revision process, for a national forest plan revision at the Chugach National Forest in Alaska, USA. Analysis showed when discourse quality values underwent a comparison of means test, the values were significantly changed between comments with no spatial precision versus those considered to have spatial precision. The results suggest, preliminarily, that employing spatial precision in narratives changes discourse quality during deliberative activities. Further, the way in which people use spatial precision to communicate during a policy revision process can impact how spatial narratives are understood and valued.

Deliberation about the 2014 Umbrella Movement in a Hong Kong elite newspaper: A quantitative study of the Hong Kong Economic Journal

Deliberative democratic theorists have argued that effective deliberation is central to democracy. Does Hong Kong possess a viable public sphere for deliberating important public issues, as the city has been striving for a full democracy since the 1980s? This article addresses this significant question by examining the quality of deliberation on the 2014 Umbrella Movement by the editorials and commentaries in an elite print Chinese newspaper – the Hong Kong Economic Journal – based on the “Discourse Quality Index” and other criteria used by major works on mediated deliberation. This article argues that the Journal has served as a viable public sphere for deliberating important public concerns in Hong Kong. The Journal’s editorials and commentaries performed well in terms of offering reasoned arguments and engaging in dialogue with opposing viewpoints. Regarding respect for the actors which were involved in the Umbrella Movement, the Journal as a whole had maintained a civilized tone. However, there was an indication that commentary authors had less tolerance toward actors from the opposite camps.

Debates electorales presidenciales bajo la mirada deliberativa. Caso México, 2012 y 2018

<p>Los procesos electorales y sus prácticas asociadas de comunicación política representan elementos esenciales en regímenes democráticos. Propongo enriquecer la comprensión de campañas electorales a partir de la perspectiva deliberativa. La ventaja clave de una alta calidad deliberativa en campañas consiste en la posibilidad de contribuir al incremento de la racionalidad del voto, factor crucial en democracias incipientes. Este artículo expone el análisis y la evaluación del comportamiento discursivo y deliberativo de candidatos a la presidencia en seis debates en las elecciones federales mexicanas de 2012 y 2018.</p><p>Para evaluar la calidad deliberativa en los debates, empleé elementos del Discourse Quality Index(DQI). Los resultados en la elección de 2012 muestran que, en contraste con los primeros dos debates efectuados bajo formatos tradicionales, el diseño del tercero (#YoSoy132) “forzó” a los candidatos a justificar sus posturas y relacionarse dialógicamente. Por su parte, en la elección de 2018, cuyos moderadores, auditorio y candidatos interactuaron de forma directa, se incrementó la calidad deliberativa en ciertos aspectos, aunque no necesariamente en el nivel argumentativo.</p><div><br clear="all" /><br /></div>

DelibAnalysis: Understanding the quality of online political discourse with machine learning

This article proposes an automated methodology for the analysis of online political discourse. Drawing from the discourse quality index (DQI) by Steenbergen et al., it applies a machine learning–based quantitative approach to measuring the discourse quality of political discussions online. The DelibAnalysis framework aims to provide an accessible, replicable methodology for the measurement of discourse quality that is both platform and language agnostic. The framework uses a simplified version of the DQI to train a classifier, which can then be used to predict the discourse quality of any non-coded comment in a given political discussion online. The objective of this research is to provide a systematic framework for the automated discourse quality analysis of large datasets and, in applying this framework, to yield insight into the structure and features of political discussions online.