Social Context-aware Person Search in Videos via Multi-modal Cues

Person search has long been treated as a crucial and challenging task to support deeper insight in personalized summarization and personality discovery. Traditional methods, e.g., person re-identification and face recognition techniques, which profile video characters based on visual information, are often limited by relatively fixed poses or small variation of viewpoints and suffer from more realistic scenes with high motion complexity (e.g., movies). At the same time, long videos such as movies often have logical story lines and are composed of continuously developmental plots. In this situation, different persons usually meet on a specific occasion, in which informative social cues are performed. We notice that these social cues could semantically profile their personality and benefit person search task in two aspects. First, persons with certain relationships usually co-occur in short intervals; in case one of them is easier to be identified, the social relation cues extracted from their co-occurrences could further benefit the identification for the harder ones. Second, social relations could reveal the association between certain scenes and characters (e.g., classmate relationship may only exist among students), which could narrow down candidates into certain persons with a specific relationship. In this way, high-level social relation cues could improve the effectiveness of person search. Along this line, in this article, we propose a social context-aware framework, which fuses visual and social contexts to profile persons in more semantic perspectives and better deal with person search task in complex scenarios. Specifically, we first segment videos into several independent scene units and abstract out social contexts within these scene units. Then, we construct inner-personal links through a graph formulation operation for each scene unit, in which both visual cues and relation cues are considered. Finally, we perform a relation-aware label propagation to identify characters’ occurrences, combining low-level semantic cues (i.e., visual cues) and high-level semantic cues (i.e., relation cues) to further enhance the accuracy. Experiments on real-world datasets validate that our solution outperforms several competitive baselines.

On the Study of Transformers for Query Suggestion

When conducting a search task, users may find it difficult to articulate their need, even more so when the task is complex. To help them complete their search, search engine usually provide query suggestions. A good query suggestion system requires to model user behavior during the search session. In this article, we study multiple Transformer architectures applied to the query suggestion task and compare them with recurrent neural network (RNN)-based models. We experiment Transformer models with different tokenizers, with different Encoders (large pretrained models or fully trained ones), and with two kinds of architectures (flat or hierarchic). We study the performance and the behaviors of these various models, and observe that Transformer-based models outperform RNN-based ones. We show that while the hierarchical architectures exhibit very good performances for query suggestion, the flat models are more suitable for complex and long search tasks. Finally, we investigate the flat models behavior and demonstrate that they indeed learn to recover the hierarchy of a search session.

Visual segmentation of complex naturalistic structures in an infant eye-tracking search task

An infant's everyday visual environment is composed of a complex array of entities, some of which are well integrated into their surroundings. Although infants are already sensitive to some categories in their first year of life, it is not clear which visual information supports their detection of meaningful elements within naturalistic scenes. Here we investigated the impact of image characteristics on 8-month-olds' search performance using a gaze contingent eye-tracking search task. Infants had to detect a target patch on a background image. The stimuli consisted of images taken from three categories: vegetation, non-living natural elements (e.g., stones), and manmade artifacts, for which we also assessed target background differences in lower- and higher-level visual properties. Our results showed that larger target-background differences in the statistical properties scaling invariance and entropy, and also stimulus backgrounds including low pictorial depth, predicted better detection performance. Furthermore, category membership only affected search performance if supported by luminance contrast. Data from an adult comparison group also indicated that infants' search performance relied more on lower-order visual properties than adults. Taken together, these results suggest that infants use a combination of property- and category-related information to parse complex visual stimuli.

The forest, the trees, and the leaves across adulthood: Age-related changes on a visual search task containing three-level hierarchical stimuli

Selecting relevant visual information in complex scenes by processing either global information or local parts helps us act efficiently within our environment and achieve goals. A global advantage (faster global than local processing) and global interference (global processing interferes with local processing) comprise an evidentiary global precedence phenomenon in early adulthood. However, the impact of healthy aging on this phenomenon remains unclear. As such, we collected behavioral data during a visual search task, including three-levels hierarchical stimuli (i.e., global, intermediate, and local levels) with several hierarchical distractors, in 50 healthy adults (26 younger (mean age: 26 years) and 24 older (mean age: 62 years)). Results revealed that processing information presented at the global and intermediate levels was independent of age. Conversely, older adults were slower for local processing compared to the younger adults, suggesting lower efficiency to deal with visual distractors during detail-oriented visual search. Although healthy older adults continued exhibiting a global precedence phenomenon, they were disproportionately less efficient during local aspects of information processing, especially when multiple visual information was displayed. Our results could have important implications for many life situations by suggesting that visual information processing is impacted by healthy aging, even with similar visual stimuli objectively presented.

Designing Combinations of Pictogram and Text Size for Icons: Effects of Text Size, Pictogram Size, and Familiarity on Older Adults’ Visual Search Performance

Objective This study aimed to determine suitable combinations of text and pictogram sizes for older adults and investigated the visual prioritization of pictogram versus text. Background Icons have become an indispensable part of application (app) design. Pictogram size and text size of icons influence the usability of apps, especially by aged users. However, few studies have investigated the influences of different pictogram and text size combinations on readability, legibility, and visual search performance for older adults. Method This study used eye-tracking technology to investigate the effects of different pictogram and text size combinations as well as familiarity on readability, legibility, and visual search performance for older adults. A 3 (pictogram size) × 3 (text size) × 2 (familiarity) repeated-measures experimental design was used. Results The results of this study suggest that pictogram size and text size significantly affect visual search performance and that familiarity moderates the effect of text size on distribution of fixation duration proportion for text and pictograms. Conclusion Large pictogram and text sizes improved the readability and legibility of icons for older adults. Furthermore, the older adults fixated the area of text prior to pictograms when the pictogram size was larger than 72 × 72 px (1.38° × 1.38°) in the visual search task. Application The results of this study suggest using different combinations of pictogram and text sizes for older adults under different scenarios. The findings of this study act as practical support for designers and developers of mobile apps for older adults.

Belief in Control: Voluntary Choice Enhances Subsequent Task Performance under Undefeated Choice-Outcome Causation

Humans are believed to have volition through which they act upon and change the external environment. As an exercise of volition, making a voluntary choice facilitates the subsequent behavioral performance relative to a forced choice. However, it is unclear how this facilitation is constrained by the perceived relationship between a choice and its outcome. In a series of experiments, participants were free or forced to choose one of two presented pictures. The outcome of the choice was then revealed, which could be always the chosen picture or always the unchosen picture (i.e., a confirmed choice-outcome causation), a blank screen with no picture at all (i.e., an unrevealed choice-outcome relation), the chosen or unchosen picture with equal probability (i.e., a defeated choice-outcome causation), or a third picture different from the two preceding options (again, a defeated choice-outcome causation). Participants then complete a visual search task with the task-irrelevant picture (or the blank screen) serving as a background. Results showed that the search performance was improved after a voluntary choice under both the confirmed causation and the unrevealed relation, but not under the defeated causation. Over individuals, the improved performance due to voluntary choice under confirmed causation positively correlated with the improved performance under the unrevealed relation, and with the reported belief in controlling the outcome of the choice. Our findings suggest that the exercise of volition motivates subsequent behavior, and this motivation is restricted to an “undefeated” choice-outcome causation which affords a belief in controlling the outcome by exerting volition.

Alternation emerges as a multi-modal strategy for turbulent odor navigation

Foraging mammals exhibit a familiar yet poorly characterized phenomenon, "alternation", a momentary pause to sniff in the air often preceded by the animal rearing on its hind legs or raising its head. Intriguingly, rodents executing an olfactory search task spontaneously exhibit alternation in the presence of airflow, suggesting that alternation may serve an important role during turbulent plume-tracking. To test this hypothesis, we combine fully-resolved numerical simulations of turbulent odor transport and Bellman optimization methods for decision-making under partial observability. We show that an agent trained to minimize search time in a realistic odor plume exhibits extensive alternation together with the characteristic cast-and-surge behavior commonly observed in flying insects. Alternation is tightly linked with casting and occurs more frequently when the agent is far downwind of the source, where the likelihood of detecting airborne cues is higher relative to cues close to the ground. Casting and alternation emerge as complementary tools for effective exploration when cues are sparse. We develop a model based on marginal value theory to capture the interplay between casting, surging and alternation. More generally, we show how multiple sensorimotor modalities can be fruitfully integrated during complex goal-directed behavior.

Voice Assistant vs. Chatbot – Examining the Fit Between Conversational Agents’ Interaction Modalities and Information Search Tasks

Owing to technological advancements in artificial intelligence, voice assistants (VAs) offer speech as a new interaction modality. Compared to text-based interaction, speech is natural and intuitive, which is why companies use VAs in customer service. However, we do not yet know for which kinds of tasks speech is beneficial. Drawing on task-technology fit theory, we present a research model to examine the applicability of VAs to different tasks. To test this model, we conducted a laboratory experiment with 116 participants who had to complete an information search task with a VA or a chatbot. The results show that speech exhibits higher perceived efficiency, lower cognitive effort, higher enjoyment, and higher service satisfaction than text-based interaction. We also find that these effects depend on the task’s goal-directedness. These findings extend task-technology fit theory to customers’ choice of interaction modalities and inform practitioners about the use of VAs for information search tasks.

Sincere praise and flattery: Reward value and association with the praise-seeking trait

Sincere praise reliably conveys positive or negative feedback, while flattery always conveys positive but unreliable feedback. These two praise types have not been compared in terms of communication effectiveness and individual preferences using neuroimaging. Through functional magnetic resonance imaging, we measured brain activity when healthy young participants received sincere praise or flattery after performing a visual search task. Higher activation was observed in the right nucleus accumbens during sincere praise than during flattery, and praise reliability correlated with posterior cingulate cortex activity, implying a motivational effect of sincere praise. In line with this, sincere praise uniquely activated several cortical areas potentially involved in concern regarding others' evaluations. A high praise-seeking tendency was associated with lower activation of the inferior parietal sulcus during sincere praise compared to flattery after poor task performance, potentially reflecting suppression of negative feedback to maintain self-esteem. In summary, the neural dynamics of the motivational and socio-emotional effects of praise differed.