A Self-Supervised Representation Learning of Sentence Structure for Authorship Attribution

The syntactic structure of sentences in a document substantially informs about its authorial writing style. Sentence representation learning has been widely explored in recent years and it has been shown that it improves the generalization of different downstream tasks across many domains. Even though utilizing probing methods in several studies suggests that these learned contextual representations implicitly encode some amount of syntax, explicit syntactic information further improves the performance of deep neural models in the domain of authorship attribution. These observations have motivated us to investigate the explicit representation learning of syntactic structure of sentences. In this article, we propose a self-supervised framework for learning structural representations of sentences. The self-supervised network contains two components; a lexical sub-network and a syntactic sub-network which take the sequence of words and their corresponding structural labels as the input, respectively. Due to the n -to-1 mapping of words to their structural labels, each word will be embedded into a vector representation which mainly carries structural information. We evaluate the learned structural representations of sentences using different probing tasks, and subsequently utilize them in the authorship attribution task. Our experimental results indicate that the structural embeddings significantly improve the classification tasks when concatenated with the existing pre-trained word embeddings.

The non-anthropocentric informational agents: Codes, software, and the logic of emergence in cybersecurity

Many theoretical approaches to cybersecurity adopt an anthropocentric conceptualisation of agency; that is, tying the capacity to act to human subjectivity and disregarding the role of the non-human in co-constructing its own (in)security. This article argues that such approaches are insufficient in capturing the complexities of cyber incidents, particularly those that involve self-perpetuating malware and autonomous cyber attacks that can produce unintentional and unpredictable consequences. Using interdisciplinary insights from the philosophy of information and software studies, the article counters the anthropocentrism in the cybersecurity literature by investigating the agency of syntactic information (that is, codes/software) in co-producing the logics and politics of cybersecurity. It specifically studies the complexities of codes/software as informational agents, their self-organising capacities, and their autonomous properties to develop an understanding of cybersecurity as emergent security. Emergence is introduced in the article as a non-linear security logic that captures the peculiar agential capacities of codes/software and the ways in which they challenge human control and intentionality by co-constructing enmity and by co-producing the subjects and objects of cybersecurity.

Testing the automaticity of syntax using masked visual priming

Language comprehension proceeds at a very fast pace. It is argued that context influences the speed of language comprehension by providing informative cues for the correct processing of the incoming linguistic input. Priming studies investigating the role of context in language processing have shown that humans quickly recognise target words that share orthographic, morphological, or semantic information with their preceding primes. How syntactic information influences the processing of incoming words is however less known. Early syntactic priming studies reported faster recognition for noun and verb targets (e.g., apple or sing) following primes with which they form grammatical phrases or sentences (the apple, he sings). The studies however leave open a number of questions about the reported effect, including the degree of automaticity of syntactic priming, the facilitative versus inhibitory nature, and the specific mechanism underlying the priming effect—that is, the type of syntactic information primed on the target word. Here we employed a masked syntactic priming paradigm in four behavioural experiments in German language to test whether masked primes automatically facilitate the categorization of nouns and verbs presented as flashing visual words. Overall, we found robust syntactic priming effects with masked primes—thus suggesting high automaticity of the process—but only when verbs were morpho-syntactically marked (er kau-t; he chew-s). Furthermore, we found that, compared to baseline, primes slow down target categorisation when the relationship between prime and target is syntactically incorrect, rather than speeding it up when the prime-target relationship is syntactically correct. This argues in favour of an inhibitory nature of syntactic priming. Overall, the data indicate that humans automatically extract abstract syntactic features from word categories as flashing visual words, which has an impact on the speed of successful language processing during language comprehension.

Aspect-level sentiment analysis merged with knowledge graph and graph convolutional neural network

Aspect level sentiment analysis belongs to fine-grained sentiment analysis, w hich has caused extensive research in academic circles in recent years. For this task, th e recurrent neural network (RNN) model is usually used for feature extraction, but the model cannot effectively obtain the structural information of the text. Recent studies h ave begun to use the graph convolutional network (GCN) to model the syntactic depen dency tree of the text to solve this problem. For short text data, the text information is not enough to accurately determine the emotional polarity of the aspect words, and the knowledge graph is not effectively used as external knowledge that can enrich the sem antic information. In order to solve the above problems, this paper proposes a graph co nvolutional neural network (GCN) model that can process syntactic information, know ledge graphs and text semantic information. The model works on the “syntax-knowled ge” graph to extract syntactic information and common sense information at the same t ime. Compared with the latest model, the model in this paper can effectively improve t he accuracy of aspect-level sentiment classification on two datasets.

Sentence Compression Using BERT and Graph Convolutional Networks

Sentence compression is a natural language-processing task that produces a short paraphrase of an input sentence by deleting words from the input sentence while ensuring grammatical correctness and preserving meaningful core information. This study introduces a graph convolutional network (GCN) into a sentence compression task to encode syntactic information, such as dependency trees. As we upgrade the GCN to activate a directed edge, the compression model with the GCN layers can distinguish between parent and child nodes in a dependency tree when aggregating adjacent nodes. Furthermore, by increasing the number of GCN layers, the model can gradually collect high-order information of a dependency tree when propagating node information through the layers. We implement a sentence compression model for Korean and English, respectively. This model consists of three components: pre-trained BERT model, GCN layers, and a scoring layer. The scoring layer can determine whether a word should remain in a compressed sentence by relying on the word vector containing contextual and syntactic information encoded by BERT and GCN layers. To train and evaluate the proposed model, we used the Google sentence compression dataset for English and a Korean sentence compression corpus containing about 140,000 sentence pairs for Korean. The experimental results demonstrate that the proposed model achieves state-of-the-art performance for English. To the best of our knowledge, this sentence compression model based on the deep learning model trained with a large-scale corpus is the first attempt for Korean.

Lexical attention and aspect-oriented graph convolutional networks for aspect-based sentiment analysis

The purpose of aspect-based sentiment analysis is to predict the sentiment polarity of different aspects in a text. In previous work, while attention has been paid to the use of Graph Convolutional Networks (GCN) to encode syntactic dependencies in order to exploit syntactic information, previous models have tended to confuse opinion words from different aspects due to the complexity of language and the diversity of aspects. On the other hand, the effect of word lexicality on aspects’ sentiment polarity judgments has not been considered in previous studies. In this paper, we propose lexical attention and aspect-oriented GCN to solve the above problems. First, we construct an aspect-oriented dependency-parsed tree by analyzing and pruning the dependency-parsed tree of the sentence, then use the lexical attention mechanism to focus on the features of the lexical properties that play a key role in determining the sentiment polarity, and finally extract the aspect-oriented lexical weighted features by a GCN.Extensive experimental results on three benchmark datasets demonstrate the effectiveness of our approach.

EXPRESS: Reliance on semantic and structural heuristics in sentence comprehension across the lifespan

People sometimes misinterpret the sentences that they read. One possible reason suggested in the literature is a race between slow bottom-up algorithmic processing and “fast and frugal” top-down heuristic processing that serves to support fast-paced communication but sometimes results in incorrect representations. Heuristic processing can be both semantic, relying on world knowledge and semantic relations between words, and structural, relying on structural economy. Scattered experimental evidence suggests that reliance on heuristics may change from greater reliance on syntactic information in younger people to greater reliance on semantic information in older people. We tested whether the reliance on structural and semantic heuristics changes with age in 137 Russian-speaking adolescents, 135 young adults, and 77 older adults. In a self-paced reading task with comprehension questions, participants read unambiguous high- vs. low-attachment sentences that were either semantically plausible or implausible: i.e., the syntactic structure either matched or contradicted the semantic relations between words. We found that the use of top-down heuristics in comprehension increased across the lifespan. Adolescents did not rely on structural heuristics, in contrast to young and older adults. At the same time, older adults relied on semantic heuristics more than young adults and adolescents. Importantly, we found that top-down heuristic processing was faster than bottom-up algorithmic processing: slower reading times were associated with greater accuracy specifically in implausible sentences.

Processing similarities between native speakers and non-balanced bilinguals

Aims and objectives: The study investigates human sentence processing and argues that information from multiple sources is considered equally in native and non-native languages. Non-syntactic information does not overrule the parsing decisions prompted by syntactic cues. Methodology: The experiment used ambiguous relative clauses (RC) in a self-paced reading task with 20 native and 45 non-native adult speakers of English and Russian. The software Linger recorded participants’ answers to comprehension questions and the time they spent reading each word. Data and analysis: Mixed linear analysis performed in R checked for the effect of a matrix verb, RC length, social conventions, the native language and the language of testing on RC processing and interpretation. Findings: Both native and non-native speakers followed social conventions in deciding on the interpretation of the RC. However, this information never overruled the attachment decision prompted by the matrix predicate or by the length of the RC which entails certain sentence prosody. Originality: The study is innovative in investigating the extent to which each factor affected RC processing. It shows that social conventions enhance processing when they conspire with the structural parse prompted by linguistic cues. When they do not, syntactic information governs sentence parsing in both L1 and L2. Significance/implications: The study provides evidence that sentence processing uses linguistic structure as a first parsing hypothesis, which can then be adjusted to incorporate the incoming information from multiple sources. Limitations: The findings need further support from testing L2 learners of Russian in various socio-cultural contexts.

Syntax Role for Neural Semantic Role Labeling

Semantic role labeling (SRL) is dedicated to recognizing the semantic predicate-argument structure of a sentence. Previous studies in terms of traditional models have shown syntactic information can make remarkable contributions to SRL performance; however, the necessity of syntactic information was challenged by a few recent neural SRL studies that demonstrate impressive performance without syntactic backbones and suggest that syntax information becomes much less important for neural semantic role labeling, especially when paired with recent deep neural network and large-scale pre-trained language models. Despite this notion, the neural SRL field still lacks a systematic and full investigation on the relevance of syntactic information in SRL, for both dependency and both monolingual and multilingual settings. This paper intends to quantify the importance of syntactic information for neural SRL in the deep learning framework. We introduce three typical SRL frameworks (baselines), sequence-based, tree-based, and graph-based, which are accompanied by two categories of exploiting syntactic information: syntax pruningbased and syntax feature-based. Experiments are conducted on the CoNLL-2005, 2009, and 2012 benchmarks for all languages available, and results show that neural SRL models can still benefit from syntactic information under certain conditions. Furthermore, we show the quantitative significance of syntax to neural SRL models together with a thorough empirical survey using existing models.