Associative Diffusion and the Pitfalls of Structural Reductionism

In their insightful comment, DellaPosta and Davoodi argue that our finding (Goldberg and Stein 2018) that segmented networks inhibit cultural differentiation does not generalize to large networks. However, their demonstration rests on an incorrect implementation of the preference updating process in the associative diffusion model. We show that once this discrepancy is corrected, cultural differentiation is more pronounced in fully connected networks, irrespective of network size and even under extreme assumptions about cognitive decay. We use this as an opportunity to discuss the associative diffusion model’s assumptions and scope conditions, as well as to critically reassess prevailing contagion-based diffusion models.

Binary and Multiclass Text Classification by Means of Separable Convolutional Neural Network

In this paper, the structure of a separable convolutional neural network that consists of an embedding layer, separable convolutional layers, convolutional layer and global average pooling is represented for binary and multiclass text classifications. The advantage of the proposed structure is the absence of multiple fully connected layers, which is used to increase the classification accuracy but raises the computational cost. The combination of low-cost separable convolutional layers and a convolutional layer is proposed to gain high accuracy and, simultaneously, to reduce the complexity of neural classifiers. Advantages are demonstrated at binary and multiclass classifications of written texts by means of the proposed networks under the sigmoid and Softmax activation functions in convolutional layer. At binary and multiclass classifications, the accuracy obtained by separable convolutional neural networks is higher in comparison with some investigated types of recurrent neural networks and fully connected networks.

Periodic modes of group dominance in fully coupled neural networks

Nonlinear systems of differential equations with delay, which are mathematical models of fully connected networks of impulse neurons, are considered. Purpose of this work is to study the dynamic properties of one special class of solutions to these systems. Large parameter methods are used to study the existence and stability in сonsidered models of special periodic motions – the so-called group dominance or k-dominance modes, where k ∈ N. Results. It is shown that each such regime is a relaxation cycle, exactly k components of which perform synchronous impulse oscillations, and all other components are asymptotically small. The maximum number of stable coexisting group dominance cycles in the system with an appropriate choice of parameters is 2m − 1, where m is the number of network elements. Conclusion. Considered model with maximum possible number of couplings allows us to describe the most complex and diverse behavior that may be observed in biological neural associations. A feature of the k-dominance modes we have considered is that some of the network neurons are in a non-working (refractory) state. Each periodic k-dominance mode can be associated with a binary vector (α1, α2, . . . , αm), where αj = 1 if the j-th neuron is active and αj = 0 otherwise. Taking this into account, we come to the conclusion that these modes can be used to build devices with associative memory based on artificial neural networks.

On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization

The prevailing thinking is that orthogonal weights are crucial to enforcing dynamical isometry and speeding up training. The increase in learning speed that results from orthogonal initialization in linear networks has been well-proven. However, while the same is believed to also hold for nonlinear networks when the dynamical isometry condition is satisfied, the training dynamics behind this contention have not been thoroughly explored. In this work, we study the dynamics of ultra-wide networks across a range of architectures, including Fully Connected Networks (FCNs) and Convolutional Neural Networks (CNNs) with orthogonal initialization via neural tangent kernel (NTK). Through a series of propositions and lemmas, we prove that two NTKs, one corresponding to Gaussian weights and one to orthogonal weights, are equal when the network width is infinite. Further, during training, the NTK of an orthogonally-initialized infinite-width network should theoretically remain constant. This suggests that the orthogonal initialization cannot speed up training in the NTK (lazy training) regime, contrary to the prevailing thoughts. In order to explore under what circumstances can orthogonality accelerate training, we conduct a thorough empirical investigation outside the NTK regime. We find that when the hyper-parameters are set to achieve a linear regime in nonlinear activation, orthogonal initialization can improve the learning speed with a large learning rate or large depth.

Towards Scalable Complete Verification of Relu Neural Networks via Dependency-based Branching

We introduce an efficient method for the complete verification of ReLU-based feed-forward neural networks. The method implements branching on the ReLU states on the basis of a notion of dependency between the nodes. This results in dividing the original verification problem into a set of sub-problems whose MILP formulations require fewer integrality constraints. We evaluate the method on all of the ReLU-based fully connected networks from the first competition for neural network verification. The experimental results obtained show 145% performance gains over the present state-of-the-art in complete verification.

Topological measurement of deep neural networks using persistent homology

The inner representation of deep neural networks (DNNs) is indecipherable, which makes it difficult to tune DNN models, control their training process, and interpret their outputs. In this paper, we propose a novel approach to investigate the inner representation of DNNs through topological data analysis (TDA). Persistent homology (PH), one of the outstanding methods in TDA, was employed for investigating the complexities of trained DNNs. We constructed clique complexes on trained DNNs and calculated the one-dimensional PH of DNNs. The PH reveals the combinational effects of multiple neurons in DNNs at different resolutions, which is difficult to be captured without using PH. Evaluations were conducted using fully connected networks (FCNs) and networks combining FCNs and convolutional neural networks (CNNs) trained on the MNIST and CIFAR-10 data sets. Evaluation results demonstrate that the PH of DNNs reflects both the excess of neurons and problem difficulty, making PH one of the prominent methods for investigating the inner representation of DNNs.

Higher-order simplicial synchronization of coupled topological signals

Simplicial complexes capture the underlying network topology and geometry of complex systems ranging from the brain to social networks. Here we show that algebraic topology is a fundamental tool to capture the higher-order dynamics of simplicial complexes. In particular we consider topological signals, i.e., dynamical signals defined on simplices of different dimension, here taken to be nodes and links for simplicity. We show that coupling between signals defined on nodes and links leads to explosive topological synchronization in which phases defined on nodes synchronize simultaneously to phases defined on links at a discontinuous phase transition. We study the model on real connectomes and on simplicial complexes and network models. Finally, we provide a comprehensive theoretical approach that captures this transition on fully connected networks and on random networks treated within the annealed approximation, establishing the conditions for observing a closed hysteresis loop in the large network limit.

HCMSL: Hybrid Cross-modal Similarity Learning for Cross-modal Retrieval

The purpose of cross-modal retrieval is to find the relationship between different modal samples and to retrieve other modal samples with similar semantics by using a certain modal sample. As the data of different modalities presents heterogeneous low-level feature and semantic-related high-level features, the main problem of cross-modal retrieval is how to measure the similarity between different modalities. In this article, we present a novel cross-modal retrieval method, named Hybrid Cross-Modal Similarity Learning model (HCMSL for short). It aims to capture sufficient semantic information from both labeled and unlabeled cross-modal pairs and intra-modal pairs with same classification label. Specifically, a coupled deep fully connected networks are used to map cross-modal feature representations into a common subspace. Weight-sharing strategy is utilized between two branches of networks to diminish cross-modal heterogeneity. Furthermore, two Siamese CNN models are employed to learn intra-modal similarity from samples of same modality. Comprehensive experiments on real datasets clearly demonstrate that our proposed technique achieves substantial improvements over the state-of-the-art cross-modal retrieval techniques.