scholarly journals Detecting Human-Object Interactions via Functional Generalization

2020 ◽  
Vol 34 (07) ◽  
pp. 10460-10469 ◽  
Author(s):  
Ankan Bansal ◽  
Sai Saketh Rambhatla ◽  
Abhinav Shrivastava ◽  
Rama Chellappa
Keyword(s):  
Experimental Validation ◽  
State Of The Art ◽  
Visual Features ◽  
Average Precision ◽  
Proposed Model ◽  
Human Object ◽  
Significant Performance ◽  
Unseen Objects ◽  
Performance Gains ◽  
Object Interactions

We present an approach for detecting human-object interactions (HOIs) in images, based on the idea that humans interact with functionally similar objects in a similar manner. The proposed model is simple and efficiently uses the data, visual features of the human, relative spatial orientation of the human and the object, and the knowledge that functionally similar objects take part in similar interactions with humans. We provide extensive experimental validation for our approach and demonstrate state-of-the-art results for HOI detection. On the HICO-Det dataset our method achieves a gain of over 2.5% absolute points in mean average precision (mAP) over state-of-the-art. We also show that our approach leads to significant performance gains for zero-shot HOI detection in the seen object setting. We further demonstrate that using a generic object detector, our model can generalize to interactions involving previously unseen objects.

scholarly journals The Image Annotation Refinement in Embedding Feature Space based on Mutual Information

2022 ◽  
Vol 16 ◽  
pp. 191-201
Author(s):  
Wei Li ◽  
Haiyu Song ◽  
Hongda Zhang ◽  
Houjie Li ◽  
Pengjie Wang
Keyword(s):  
Mutual Information ◽  
Image Annotation ◽  
State Of The Art ◽  
Feature Space ◽  
Semantic Space ◽  
Visual Features ◽  
Novel Approach ◽  
Proposed Model ◽  
Information Method ◽  
Available Information

The ever-increasing size of images has made automatic image annotation one of the most important tasks in the fields of machine learning and computer vision. Despite continuous efforts in inventing new annotation algorithms and new models, results of the state-of-the-art image annotation methods are often unsatisfactory. In this paper, to further improve annotation refinement performance, a novel approach based on weighted mutual information to automatically refine the original annotations of images is proposed. Unlike the traditional refinement model using only visual feature, the proposed model use semantic embedding to properly map labels and visual features to a meaningful semantic space. To accurately measure the relevance between the particular image and its original annotations, the proposed model utilize all available information including image-to-image, label-to-label and image-to-label. Experimental results conducted on three typical datasets show not only the validity of the refinement, but also the superiority of the proposed algorithm over existing ones. The improvement largely benefits from our proposed mutual information method and utilizing all available information.

scholarly journals Turbo Learning Framework for Human-Object Interactions Recognition and Human Pose Estimation

2019 ◽  
Vol 33 ◽  
pp. 898-905 ◽  
Author(s):  
Wei Feng ◽  
Wentao Liu ◽  
Tong Li ◽  
Jing Peng ◽  
Chen Qian ◽  
...  
Keyword(s):  
Pose Estimation ◽  
Message Passing ◽  
Closed Loop ◽  
State Of The Art ◽  
Human Action ◽  
Complementary Information ◽  
Learning Framework ◽  
Human Object ◽  
Human Pose ◽  
Object Interactions

Human-object interactions (HOI) recognition and pose estimation are two closely related tasks. Human pose is an essential cue for recognizing actions and localizing the interacted objects. Meanwhile, human action and their interacted objects’ localizations provide guidance for pose estimation. In this paper, we propose a turbo learning framework to perform HOI recognition and pose estimation simultaneously. First, two modules are designed to enforce message passing between the tasks, i.e. pose aware HOI recognition module and HOI guided pose estimation module. Then, these two modules form a closed loop to utilize the complementary information iteratively, which can be trained in an end-to-end manner. The proposed method achieves the state-of-the-art performance on two public benchmarks including Verbs in COCO (V-COCO) and HICO-DET datasets.

scholarly journals Detecting human—object interaction with multi-level pairwise feature network

2020 ◽  
Author(s):  
Hanchao Liu ◽  
Tai-Jiang Mu ◽  
Xiaolei Huang
Keyword(s):  
Spatial Configuration ◽  
State Of The Art ◽  
Body Part ◽  
Human Action ◽  
Image Understanding ◽  
Semantic Level ◽  
Object Interaction ◽  
Human Object ◽  
Multi Level ◽  
Object Interactions

Abstract Human–object interaction (HOI) detection is crucial for human-centric image understanding which aims to infer ⟨human, action, object⟩ triplets within an image. Recent studies often exploit visual features and the spatial configuration of a human–object pair in order to learn the action linking the human and object in the pair. We argue that such a paradigm of pairwise feature extraction and action inference can be applied not only at the whole human and object instance level, but also at the part level at which a body part interacts with an object, and at the semantic level by considering the semantic label of an object along with human appearance and human–object spatial configuration, to infer the action. We thus propose a multi-level pairwise feature network (PFNet) for detecting human–object interactions. The network consists of three parallel streams to characterize HOI utilizing pairwise features at the above three levels; the three streams are finally fused to give the action prediction. Extensive experiments show that our proposed PFNet outperforms other state-of-the-art methods on the V-COCO dataset and achieves comparable results to the state-of-the-art on the HICO-DET dataset.

scholarly journals Action-Guided Attention Mining and Relation Reasoning Network for Human-Object Interaction Detection

2020 ◽  
Author(s):  
Xue Lin ◽  
Qi Zou ◽  
Xixia Xu
Keyword(s):  
State Of The Art ◽  
Subtle Difference ◽  
Fine Grained ◽  
Interaction Detection ◽  
Object Interaction ◽  
Proposed Model ◽  
Human Object ◽  
Domination Problem ◽  
Combination Space ◽  
Activation Map

Human-object interaction (HOI) detection is important to understand human-centric scenes and is challenging due to subtle difference between fine-grained actions, and multiple co-occurring interactions. Most approaches tackle the problems by considering the multi-stream information and even introducing extra knowledge, which suffer from a huge combination space and the non-interactive pair domination problem. In this paper, we propose an Action-Guided attention mining and Relation Reasoning (AGRR) network to solve the problems. Relation reasoning on human-object pairs is performed by exploiting contextual compatibility consistency among pairs to filter out the non-interactive combinations. To better discriminate the subtle difference between fine-grained actions, an action-aware attention based on class activation map is proposed to mine the most relevant features for recognizing HOIs. Extensive experiments on V-COCO and HICO-DET datasets demonstrate the effectiveness of the proposed model compared with the state-of-the-art approaches.

scholarly journals Posture Detection of Individual Pigs Based on Lightweight Convolution Neural Networks and Efficient Channel-Wise Attention

Sensors ◽  
2021 ◽  
Vol 21 (24) ◽  
pp. 8369
Author(s):  
Yizhi Luo ◽  
Zhixiong Zeng ◽  
Huazhong Lu ◽  
Enli Lv
Keyword(s):  
State Of The Art ◽  
Early Morning ◽  
Model Parameters ◽  
Model Structure ◽  
Computation Complexity ◽  
Average Precision ◽  
Symmetrical Structure ◽  
Attention Model ◽  
Proposed Model ◽  
Statistical Results

In this paper, a lightweight channel-wise attention model is proposed for the real-time detection of five representative pig postures: standing, lying on the belly, lying on the side, sitting, and mounting. An optimized compressed block with symmetrical structure is proposed based on model structure and parameter statistics, and the efficient channel attention modules are considered as a channel-wise mechanism to improve the model architecture.The results show that the algorithm’s average precision in detecting standing, lying on the belly, lying on the side, sitting, and mounting is 97.7%, 95.2%, 95.7%, 87.5%, and 84.1%, respectively, and the speed of inference is around 63 ms (CPU = i7, RAM = 8G) per postures image. Compared with state-of-the-art models (ResNet50, Darknet53, CSPDarknet53, MobileNetV3-Large, and MobileNetV3-Small), the proposed model has fewer model parameters and lower computation complexity. The statistical results of the postures (with continuous 24 h monitoring) show that some pigs will eat in the early morning, and the peak of the pig’s feeding appears after the input of new feed, which reflects the health of the pig herd for farmers.

scholarly journals Multi-Task Classification and Segmentation for Explicable Capsule Endoscopy Diagnostics

2021 ◽  
Vol 8 ◽  
Author(s):  
Zishang Kong ◽  
Min He ◽  
Qianjiang Luo ◽  
Xiansong Huang ◽  
Pengxu Wei ◽  
...  
Keyword(s):  
Capsule Endoscopy ◽  
State Of The Art ◽  
Semantic Segmentation ◽  
Motion Blur ◽  
Joint Learning ◽  
Task Classification ◽  
Significant Performance ◽  
Degree Classification ◽  
Performance Gains ◽  
First Time

Capsule endoscopy is a leading diagnostic tool for small bowel lesions which faces certain challenges such as time-consuming interpretation and harsh optical environment inside the small intestine. Specialists unavoidably waste lots of time on searching for a high clearness degree image for accurate diagnostics. However, current clearness degree classification methods are based on either traditional attributes or an unexplainable deep neural network. In this paper, we propose a multi-task framework, called the multi-task classification and segmentation network (MTCSN), to achieve joint learning of clearness degree (CD) and tissue semantic segmentation (TSS) for the first time. In the MTCSN, the CD helps to generate better refined TSS, while TSS provides an explicable semantic map to better classify the CD. In addition, we present a new benchmark, named the Capsule-Endoscopy Crohn’s Disease dataset, which introduces the challenges faced in the real world including motion blur, excreta occlusion, reflection, and various complex alimentary scenes that are widely acknowledged in endoscopy examination. Extensive experiments and ablation studies report the significant performance gains of the MTCSN over state-of-the-art methods.

scholarly journals Fusion of multi-view ultrasonic data for increased detection performance in non-destructive evaluation

2020 ◽  
Vol 476 (2243) ◽  
pp. 20200086
Author(s):  
Paul D. Wilcox ◽  
Anthony J. Croxford ◽  
Nicolas Budyn ◽  
Rhodri L. T. Bevan ◽  
Jie Zhang ◽  
...  
Keyword(s):  
State Of The Art ◽  
False Alarms ◽  
Single View ◽  
Non Destructive Evaluation ◽  
Skilled Operator ◽  
Fusion Methods ◽  
Significant Performance ◽  
Test Position ◽  
Performance Gains ◽  
Non Destructive

State-of-the-art ultrasonic non-destructive evaluation (NDE) uses an array to rapidly generate multiple, information-rich views at each test position on a safety-critical component. However, the information for detecting potential defects is dispersed across views, and a typical inspection may involve thousands of test positions. Interpretation requires painstaking analysis by a skilled operator. In this paper, various methods for fusing multi-view data are developed. Compared with any one single view, all methods are shown to yield significant performance gains, which may be related to the general and edge cases for NDE. In the general case, a defect is clearly detectable in at least one individual view, but the view(s) depends on the defect location and orientation. Here, the performance gain from data fusion is mainly the result of the selective use of information from the most appropriate view(s) and fusion provides a means to substantially reduce operator burden. The edge cases are defects that cannot be reliably detected in any one individual view without false alarms. Here, certain fusion methods are shown to enable detection with reduced false alarms. In this context, fusion allows NDE capability to be extended with potential implications for the design and operation of engineering assets.

scholarly journals Solving Exist-Random Quantified Stochastic Boolean Satisfiability via Clause Selection

2018 ◽  
Author(s):  
Nian-Ze Lee ◽  
Yen-Shi Wang ◽  
Jie-Hong R. Jiang
Keyword(s):  
Approximate Solution ◽  
State Of The Art ◽  
Expressive Language ◽  
Boolean Satisfiability ◽  
Decision Problems ◽  
Exact Answer ◽  
Quantified Boolean Formulas ◽  
Boolean Formulas ◽  
Significant Performance ◽  
Performance Gains

Stochastic Boolean satisfiability (SSAT) is an expressive language to formulate decision problems with randomness. Solving SSAT formulas has the same PSPACE-complete computational complexity as solving quantified Boolean formulas (QBFs). Despite its broad applications and profound theoretical values, SSAT has received relatively little attention compared to QBF. In this paper, we focus on exist-random quantified SSAT formulas, also known as E-MAJSAT, which is a special fragment of SSAT commonly applied in probabilistic conformant planning, posteriori hypothesis, and maximum expected utility. Based on clause selection, a recently proposed QBF technique, we propose an algorithm to solve E-MAJSAT. Moreover, our method can provide an approximate solution to E-MAJSAT with a lower bound when an exact answer is too expensive to compute. Experiments show that the proposed algorithm achieves significant performance gains and memory savings over the state-of-the-art SSAT solvers on a number of benchmark formulas, and provides useful lower bounds for cases where prior methods fail to compute exact answers.

scholarly journals Discretizing Continuous Action Space for On-Policy Optimization

2020 ◽  
Vol 34 (04) ◽  
pp. 5981-5988
Author(s):  
Yunhao Tang ◽  
Shipra Agrawal
Keyword(s):  
Complex Dynamics ◽  
State Of The Art ◽  
Discrete Distribution ◽  
Action Space ◽  
High Dimensional ◽  
Continuous Control ◽  
Continuous Action ◽  
Significant Performance ◽  
Policy Optimization ◽  
Performance Gains

In this work, we show that discretizing action space for continuous control is a simple yet powerful technique for on-policy optimization. The explosion in the number of discrete actions can be efficiently addressed by a policy with factorized distribution across action dimensions. We show that the discrete policy achieves significant performance gains with state-of-the-art on-policy optimization algorithms (PPO, TRPO, ACKTR) especially on high-dimensional tasks with complex dynamics. Additionally, we show that an ordinal parameterization of the discrete distribution can introduce the inductive bias that encodes the natural ordering between discrete actions. This ordinal architecture further significantly improves the performance of PPO/TRPO.

Online Object Tracking via Novel Adaptive Multicue Based Particle Filter Framework for Video Surveillance

2018 ◽  
Vol 27 (06) ◽  
pp. 1850023 ◽  
Author(s):  
Gurjit Singh Walia ◽  
Rajiv Kapoor
Keyword(s):  
Particle Filter ◽  
Object Tracking ◽  
Experimental Validation ◽  
State Of The Art ◽  
Context Sensitive ◽  
Proposed Model ◽  
Power Set ◽  
Particle Level ◽  
Adaptive Fusion ◽  
Level Fusion

Multicue based object tracking frameworks have been extensively explored due to their numerous applications in the field of computer vision. However, the online adaptive fusion of multicue under scale and illumination variations, partial or full occlusion, background clutters and object deformation remains an open challenge problem. In order to address this, we propose an online visual tracking algorithm using adaptive integration of multicue in a particle filter framework. The particle level fusion process is modelled as Shafer’s model with a power set defined over two focal elements. Partial conflicting masses and conjunctive consensus among three cues are estimated for each evaluated particle. Partial conflicts among cues are redistributed using Dezert-Smarandache Theory (DSmT) based proportional conflict redistribution rules (PCR-6). Additionally, context sensitive transductive cues reliabilities are used for discounting the particle likelihoods for quick adaptation of tracker. In the proposed model, automatic boosting of good particles and suppression of low performing particles not only improves resampling process but also enhances tracker accuracy. Experimental validation over benchmarked video sequences reveals that the proposed multicue tracking framework outperforms state-of-the-art trackers under various dynamic environmental challenges.

Sign in / Sign up

Export Citation Format

Share Document