Emergence of a compositional neural code for written words: Recycling of a convolutional neural network for reading

The visual word form area (VWFA) is a region of human inferotemporal cortex that emerges at a fixed location in the occipitotemporal cortex during reading acquisition and systematically responds to written words in literate individuals. According to the neuronal recycling hypothesis, this region arises through the repurposing, for letter recognition, of a subpart of the ventral visual pathway initially involved in face and object recognition. Furthermore, according to the biased connectivity hypothesis, its reproducible localization is due to preexisting connections from this subregion to areas involved in spoken-language processing. Here, we evaluate those hypotheses in an explicit computational model. We trained a deep convolutional neural network of the ventral visual pathway, first to categorize pictures and then to recognize written words invariantly for case, font, and size. We show that the model can account for many properties of the VWFA, particularly when a subset of units possesses a biased connectivity to word output units. The network develops a sparse, invariant representation of written words, based on a restricted set of reading-selective units. Their activation mimics several properties of the VWFA, and their lesioning causes a reading-specific deficit. The model predicts that, in literate brains, written words are encoded by a compositional neural code with neurons tuned either to individual letters and their ordinal position relative to word start or word ending or to pairs of letters (bigrams).

Comparing Object Recognition in Humans and Deep Convolutional Neural Networks—An Eye Tracking Study

Deep convolutional neural networks (DCNNs) and the ventral visual pathway share vast architectural and functional similarities in visual challenges such as object recognition. Recent insights have demonstrated that both hierarchical cascades can be compared in terms of both exerted behavior and underlying activation. However, these approaches ignore key differences in spatial priorities of information processing. In this proof-of-concept study, we demonstrate a comparison of human observers (N = 45) and three feedforward DCNNs through eye tracking and saliency maps. The results reveal fundamentally different resolutions in both visualization methods that need to be considered for an insightful comparison. Moreover, we provide evidence that a DCNN with biologically plausible receptive field sizes called vNet reveals higher agreement with human viewing behavior as contrasted with a standard ResNet architecture. We find that image-specific factors such as category, animacy, arousal, and valence have a direct link to the agreement of spatial object recognition priorities in humans and DCNNs, while other measures such as difficulty and general image properties do not. With this approach, we try to open up new perspectives at the intersection of biological and computer vision research.

Evidence from imaging resilience genetics for a protective mechanism against schizophrenia in the ventral visual pathway

Recently, the first genetic variants conferring resilience to schizophrenia have been identified. However, the neurobiological mechanisms underlying their protective effect remain unknown. Current models implicate adaptive neuroplastic changes in the visual system and their pro-cognitive effects in schizophrenia resilience. Here, we test the hypothesis that comparable changes can emerge from schizophrenia resilience genes. To this end, we used structural magnetic resonance imaging to investigate the effects of a schizophrenia polygenic resilience score (PRSResilience) on cortical morphology (discovery sample: n=101; UK Biobank replication sample: n=33,224). We observed positive correlations between PRSResilience and cortical volume in the fusiform gyrus, a central hub within the ventral visual pathway. Our findings indicate that resilience to schizophrenia arises partly from genetically mediated enhancements of visual processing capacities for social and non-social object information. This implies an important role of visual information processing for mitigating schizophrenia risk, which might also be exploitable for early intervention studies.

Clustered functional domains for curves and corners in cortical area V4

The ventral visual pathway is crucially involved in integrating low-level visual features into complex representations for objects and scenes. At an intermediate stage of the ventral visual pathway, V4 plays a crucial role in supporting this transformation. Many V4 neurons are selective for shape segments like curves and corners, however it remains unclear whether these neurons are organized into clustered functional domains, a structural motif common across other visual cortices. Using two-photon calcium imaging in awake macaques, we confirmed and localized cortical domains selective for curves or corners in V4. Single-cell resolution imaging confirmed that curve or corner selective neurons were spatially clustered into such domains. When tested with hexagonal-segment stimuli, we find that stimulus smoothness is the cardinal difference between curve and corner selectivity in V4. Combining cortical population responses with single neuron analysis, our results reveal that curves and corners are encoded by neurons clustered into functional domains in V4. This functionally-specific population architecture bridges the gap between the early and late cortices of the ventral pathway and may serve to facilitate complex object recognition.

CORSegNet: Deep Neural Network for Core Object Segmentation on Medical Images

In recent decades, convolutional neural networks (CNNs) have delivered promising results in vision-related tasks across different domains. Previous studies have introduced deeper network architectures to further improve the performances of object classification, localization, and segmentation. However, this induces the complexity in mapping network’s layer to the processing elements in the ventral visual pathway. Although CORnet models are not precisely biomimetic, they are closer approximations to the anatomy of ventral visual pathway compared with other deep neural networks. The uniqueness of this architecture inspires us to extend it into a core object segmentation network, CORSegnet-Z. This architecture utilizes CORnet-Z building blocks as the encoding elements. We train and evaluate the proposed model using two large datasets. Our proposed model shows significant improvements on the segmentation metrics in delineating cartilage tissues from knee magnetic resonance (MR) images and segmenting lesion boundary from dermoscopic images.