Visual world eyetracking using WebGazer.js

The visual world paradigm is one of the most influential paradigms to study real-time language processing. The present study tested whether visual world studies can be moved online, using PCIbex software (Zehr & Schwarz, 2018) and the WebGazer.js algorithm (Papoutsaki et al., 2016) to collect eye-movement data. Experiment 1 was a fixation task in which the participants looked at a fixation cross in multiple positions on the computer screen. Experiment 2 was a web-based replication of a visual world experiment by Dijkgraaf, Hartsuiker and Duyck (2017). Firstly, both experiments revealed that the spatial accuracy of the data allowed us to distinguish looks across the four quadrants of the computer screen. This suggest that the spatial resolution of WebGazer.js is fine-grained enough for most visual world experiments (which typically involve a four-by-four quadrant-based set-up of the visual display). Secondly, both experiments revealed a delay of roughly 300 ms in the time course of the eye movements, most likely caused by the internal processing speed of the browser or WebGazer.js. This delay can be problematic in studying questions that require a fine-grained temporal resolution and requires further investigation.

Right for the wrong reason; wrong for the right reason: Gibson and Arnheim on picture perception

Although J J Gibson’s theory of picture perception was often crude and biased toward naturalism, its fundamental division between the visual world and the visual field made it a semiotic theory. Contrariwise, although Arnheim wrote sensitively on pictures, he never seemed to admit that they were signs. This paper reviews both Gibson’s and Arnheim’s theories of picture perception, and explains where Arnheim’s biases caused him to lose the possibility of framing his approach in the most basic semiotic terms. Nevertheless, using the phenomenological semiotics of Sonesson and his theory of the Lifeworld Hierarchy, I demonstrate latent semiotic elements in Arnheim’s theory, due perhaps to Alfred Schutz’s influence. Hoping to argue against the brute theory of denotation, Arnheim instead sought to delay invocation of (conventional) signs as long as possible, and his idea of iconic pictorialization assumes but does not name signification. Nevertheless, I propose that Arnheim has a kind of theory of the Lifeworld Hierarchy inside the picture. Thus, he (wrongly) does not see the picture as overtly signifying but interestingly gives hints about how to treat the objects of the virtual world of the picture based on their relationship to the overall style of the work.

Mapping the visual world to the human brain

The visual maps measured non-invasively in the brain of human and non-human primates reliably reflect the underlying neuronal responses recorded with invasive electrodes.

Black, White, Colourful, Gray: Visual Effects in the Children’s Book Mimi & Liza

The study presents the reader with an intermedial interpretation of the storybook Mimi & Liza written by Katarína Kerekesová – Katarína Moláková – Alexandra Salmela (2013). The storybook follows the story of the friendship of two little girls, Mimi, who sees the world proliferating in mad colours, and the blind Liza, who is immersed in inner seeing. The two girls are presented as each other’s opposites through the semiotics of two counterpointing colour schemes. The analysis is based on Mitchell’s conception of media (Mitchell, 1994), that is, it sets out by acknowledging the intermedial state of the culture of children’s books, and then it follows the unfolding of the visual elements up through the investigation of expressive visual effects created by the text’s rhetoric. The visualization happening with the help of language is the condition of the common worldview of the blind and seeing characters as well as the guiding principle and goal of the volume; therefore besides the visual representation characteristic of children’s books, an emphasized role is given to the validation of the ekphrastic perspective in the analyzed work. The ekphrases of the text are presented as intermedial references (Rajewsky, 2010) based on Irina O. Rajewsky’s interpretation of intermediality. A unique feature of the interpretation is that the ekphrases of the volume read as sort of imaginary/imagination ekphrases which create the special, children’s book version of ekphrasis. It is characteristic for this imagination ekphrases that the order of the imaginary image and its linguistic description create an undecidable symbiosis. These images, however, can also be interpreted as inverted ekphrases, since they function not merely as descriptions of imagination ekphrases, but also as the visual world representations of linguistic imagination. Through several examples the study introduces and analyzes the mechanisms of the visualization happening with the help of language as well as the scenery painted with words.

Unsaid Thoughts: Thinking in the absence of (some) verbal logical connectives

Combining two thoughts into a compound mental representation is a central feature of our verbal and non-verbal logical abilities. We here approach this issue by focusing on the contingency that while natural languages typically verbalise only two of the sixteen connectives from formal logic to express compound thoughts —"and" and "or"— the remainder appear to be entertainable as non-verbal, conceptual representations and this suggests a way to probe how linguistic and non-linguistic thinking processes relate. In a visual world experiment aimed at tracking both comprehension-related and reasoning-related aspects of the capacity to represent compound thoughts, we found that participants are capable of learning and interpreting a made-up word for logic’s NAND operator, indicating that unlexicalised logical connectives are nonetheless conceptually available.

A sensory memory to preserve visual representations across eye movements

Saccadic eye movements (saccades) disrupt the continuous flow of visual information, yet our perception of the visual world remains uninterrupted. Here we assess the representation of the visual scene across saccades from single-trial spike trains of extrastriate visual areas, using a combined electrophysiology and statistical modeling approach. Using a model-based decoder we generate a high temporal resolution readout of visual information, and identify the specific changes in neurons’ spatiotemporal sensitivity that underly an integrated perisaccadic representation of visual space. Our results show that by maintaining a memory of the visual scene, extrastriate neurons produce an uninterrupted representation of the visual world. Extrastriate neurons exhibit a late response enhancement close to the time of saccade onset, which preserves the latest pre-saccadic information until the post-saccadic flow of retinal information resumes. These results show how our brain exploits available information to maintain a representation of the scene while visual inputs are disrupted.