scholarly journals Rethinking the Bottom-Up Framework for Query-Based Video Localization

2020 ◽  
Vol 34 (07) ◽  
pp. 10551-10558 ◽  
Author(s):  
Long Chen ◽  
Chujie Lu ◽  
Siliang Tang ◽  
Jun Xiao ◽  
Dong Zhang ◽  
...  
Keyword(s):  
State Of The Art ◽  
Ground Truth ◽  
Video Frame ◽  
Top Down ◽  
Bottom Up ◽  
Multi Scale ◽  
Feature Pyramid ◽  
Multi Level ◽  
Classification And Regression ◽  
Model Graph

In this paper, we focus on the task query-based video localization, i.e., localizing a query in a long and untrimmed video. The prevailing solutions for this problem can be grouped into two categories: i) Top-down approach: It pre-cuts the video into a set of moment candidates, then it does classification and regression for each candidate; ii) Bottom-up approach: It injects the whole query content into each video frame, then it predicts the probabilities of each frame as a ground truth segment boundary (i.e., start or end). Both two frameworks have respective shortcomings: the top-down models suffer from heavy computations and they are sensitive to the heuristic rules, while the performance of bottom-up models is behind the performance of top-down counterpart thus far. However, we argue that the performance of bottom-up framework is severely underestimated by current unreasonable designs, including both the backbone and head network. To this end, we design a novel bottom-up model: Graph-FPN with Dense Predictions (GDP). For the backbone, GDP firstly generates a frame feature pyramid to capture multi-level semantics, then it utilizes graph convolution to encode the plentiful scene relationships, which incidentally mitigates the semantic gaps in the multi-scale feature pyramid. For the head network, GDP regards all frames falling in the ground truth segment as the foreground, and each foreground frame regresses the unique distances from its location to bi-directional boundaries. Extensive experiments on two challenging query-based video localization tasks (natural language video localization and video relocalization), involving four challenging benchmarks (TACoS, Charades-STA, ActivityNet Captions, and Activity-VRL), have shown that GDP surpasses the state-of-the-art top-down models.

scholarly journals Adaptive Feature Pyramid Network to Predict Crisp Boundaries via NMS Layer and ODS F-Measure Loss Function

Information ◽  
2022 ◽  
Vol 13 (1) ◽  
pp. 32
Author(s):  
Gang Sun ◽  
Hancheng Yu ◽  
Xiangtao Jiang ◽  
Mingkui Feng
Keyword(s):  
Edge Detection ◽  
Loss Function ◽  
State Of The Art ◽  
Cross Entropy ◽  
Post Processing ◽  
Multi Scale ◽  
Feature Pyramid ◽  
Multi Level ◽  
Different Levels ◽  
F Measure

Edge detection is one of the fundamental computer vision tasks. Recent methods for edge detection based on a convolutional neural network (CNN) typically employ the weighted cross-entropy loss. Their predicted results being thick and needing post-processing before calculating the optimal dataset scale (ODS) F-measure for evaluation. To achieve end-to-end training, we propose a non-maximum suppression layer (NMS) to obtain sharp boundaries without the need for post-processing. The ODS F-measure can be calculated based on these sharp boundaries. So, the ODS F-measure loss function is proposed to train the network. Besides, we propose an adaptive multi-level feature pyramid network (AFPN) to better fuse different levels of features. Furthermore, to enrich multi-scale features learned by AFPN, we introduce a pyramid context module (PCM) that includes dilated convolution to extract multi-scale features. Experimental results indicate that the proposed AFPN achieves state-of-the-art performance on the BSDS500 dataset (ODS F-score of 0.837) and the NYUDv2 dataset (ODS F-score of 0.780).

scholarly journals M2Det: A Single-Shot Object Detector Based on Multi-Level Feature Pyramid Network

2019 ◽  
Vol 33 ◽  
pp. 9259-9266 ◽  
Author(s):  
Qijie Zhao ◽  
Tao Sheng ◽  
Yongtao Wang ◽  
Zhi Tang ◽  
Ying Chen ◽  
...  
Keyword(s):  
Feature Fusion ◽  
State Of The Art ◽  
Single Shot ◽  
Multi Scale ◽  
One Stage ◽  
Single Scale ◽  
Feature Pyramid ◽  
Multi Level ◽  
Multiple Levels ◽  
Inference Strategy

Feature pyramids are widely exploited by both the state-of-the-art one-stage object detectors (e.g., DSSD, RetinaNet, RefineDet) and the two-stage object detectors (e.g., Mask RCNN, DetNet) to alleviate the problem arising from scale variation across object instances. Although these object detectors with feature pyramids achieve encouraging results, they have some limitations due to that they only simply construct the feature pyramid according to the inherent multiscale, pyramidal architecture of the backbones which are originally designed for object classification task. Newly, in this work, we present Multi-Level Feature Pyramid Network (MLFPN) to construct more effective feature pyramids for detecting objects of different scales. First, we fuse multi-level features (i.e. multiple layers) extracted by backbone as the base feature. Second, we feed the base feature into a block of alternating joint Thinned U-shape Modules and Feature Fusion Modules and exploit the decoder layers of each Ushape module as the features for detecting objects. Finally, we gather up the decoder layers with equivalent scales (sizes) to construct a feature pyramid for object detection, in which every feature map consists of the layers (features) from multiple levels. To evaluate the effectiveness of the proposed MLFPN, we design and train a powerful end-to-end one-stage object detector we call M2Det by integrating it into the architecture of SSD, and achieve better detection performance than state-of-the-art one-stage detectors. Specifically, on MSCOCO benchmark, M2Det achieves AP of 41.0 at speed of 11.8 FPS with single-scale inference strategy and AP of 44.2 with multi-scale inference strategy, which are the new stateof-the-art results among one-stage detectors. The code will be made available on https://github.com/qijiezhao/M2Det.

scholarly journals Bottom-up and Layerwise Domain Adaptation for Pedestrian Detection in Thermal Images

2021 ◽  
Vol 17 (1) ◽  
pp. 1-19
Author(s):  
My Kieu ◽  
Andrew D. Bagdanov ◽  
Marco Bertini
Keyword(s):  
Domain Adaptation ◽  
State Of The Art ◽  
Pedestrian Detection ◽  
Challenging Problem ◽  
Top Down ◽  
Bottom Up ◽  
Security Applications ◽  
Lighting Conditions ◽  
Initial Layers ◽  
Single Modality

Pedestrian detection is a canonical problem for safety and security applications, and it remains a challenging problem due to the highly variable lighting conditions in which pedestrians must be detected. This article investigates several domain adaptation approaches to adapt RGB-trained detectors to the thermal domain. Building on our earlier work on domain adaptation for privacy-preserving pedestrian detection, we conducted an extensive experimental evaluation comparing top-down and bottom-up domain adaptation and also propose two new bottom-up domain adaptation strategies. For top-down domain adaptation, we leverage a detector pre-trained on RGB imagery and efficiently adapt it to perform pedestrian detection in the thermal domain. Our bottom-up domain adaptation approaches include two steps: first, training an adapter segment corresponding to initial layers of the RGB-trained detector adapts to the new input distribution; then, we reconnect the adapter segment to the original RGB-trained detector for final adaptation with a top-down loss. To the best of our knowledge, our bottom-up domain adaptation approaches outperform the best-performing single-modality pedestrian detection results on KAIST and outperform the state of the art on FLIR.

scholarly journals Bottom-up and Top-down: Bidirectional Additive Net for Edge Detection

2020 ◽  
Author(s):  
Lianli Gao ◽  
Zhilong Zhou ◽  
Heng Tao Shen ◽  
Jingkuan Song
Keyword(s):  
Edge Detection ◽  
Spatial Information ◽  
New Records ◽  
State Of The Art ◽  
Top Down ◽  
Bottom Up ◽  
Image Edge Detection ◽  
Universal Network ◽  
Image Edge ◽  
Hierarchical Representations

Image edge detection is considered as a cornerstone task in computer vision. Due to the nature of hierarchical representations learned in CNN, it is intuitive to design side networks utilizing the richer convolutional features to improve the edge detection. However, there is no consensus way to integrate the hierarchical information. In this paper, we propose an effective and end-to-end framework, named Bidirectional Additive Net (BAN), for image edge detection. In the proposed framework, we focus on two main problems: 1) how to design a universal network for incorporating hierarchical information sufficiently; and 2) how to achieve effective information flow between different stages and gradually improve the edge map stage by stage. To tackle these problems, we design a consecutive bottom-up and top-down architecture, where a bottom-up branch can gradually remove detailed or sharp boundaries to enable accurate edge detection and a top-down branch offers a chance of error-correcting by revisiting the low-level features that contain rich textual and spatial information. And attended additive module (AAM) is designed to cumulatively refine edges by selecting pivotal features in each stage. Experimental results show that our proposed methods can improve the edge detection performance to new records and achieve state-of-the-art results on two public benchmarks: BSDS500 and NYUDv2.

scholarly journals Complete Bottom-Up Predicate Invention in Meta-Interpretive Learning

2020 ◽  
Author(s):  
Céline Hocquette ◽  
Stephen H. Muggleton
Keyword(s):  
State Of The Art ◽  
Order Logic ◽  
Learning Performance ◽  
Sample Complexity ◽  
Logic Programs ◽  
Top Down ◽  
Bottom Up ◽  
Predicate Invention ◽  
Feature Discovery ◽  
Second Order Logic

Predicate Invention in Meta-Interpretive Learning (MIL) is generally based on a top-down approach, and the search for a consistent hypothesis is carried out starting from the positive examples as goals. We consider augmenting top-down MIL systems with a bottom-up step during which the background knowledge is generalised with an extension of the immediate consequence operator for second-order logic programs. This new method provides a way to perform extensive predicate invention useful for feature discovery. We demonstrate this method is complete with respect to a fragment of dyadic datalog. We theoretically prove this method reduces the number of clauses to be learned for the top-down learner, which in turn can reduce the sample complexity. We formalise an equivalence relation for predicates which is used to eliminate redundant predicates. Our experimental results suggest pairing the state-of-the-art MIL system Metagol with an initial bottom-up step can significantly improve learning performance.

scholarly journals DGCN: Dynamic Graph Convolutional Network for Efficient Multi-Person Pose Estimation

2020 ◽  
Vol 34 (07) ◽  
pp. 11924-11931
Author(s):  
Zhongwei Qiu ◽  
Kai Qiu ◽  
Jianlong Fu ◽  
Dongmei Fu
Keyword(s):  
Pose Estimation ◽  
State Of The Art ◽  
Semantic Relations ◽  
Dynamic Graphs ◽  
Dynamic Graph ◽  
Convolutional Network ◽  
Bottom Up ◽  
Multi Level ◽  
Human Pose ◽  
Relative Gains

Multi-person pose estimation aims to detect human keypoints from images with multiple persons. Bottom-up methods for multi-person pose estimation have attracted extensive attention, owing to the good balance between efficiency and accuracy. Recent bottom-up methods usually follow the principle of keypoints localization and grouping, where relations between keypoints are the keys to group keypoints. These relations spontaneously construct a graph of keypoints, where the edges represent the relations between two nodes (i.e., keypoints). Existing bottom-up methods mainly define relations by empirically picking out edges from this graph, while omitting edges that may contain useful semantic relations. In this paper, we propose a novel Dynamic Graph Convolutional Module (DGCM) to model rich relations in the keypoints graph. Specifically, we take into account all relations (all edges of the graph) and construct dynamic graphs to tolerate large variations of human pose. The DGCM is quite lightweight, which allows it to be stacked like a pyramid architecture and learn structural relations from multi-level features. Our network with single DGCM based on ResNet-50 achieves relative gains of 3.2% and 4.8% over state-of-the-art bottom-up methods on COCO keypoints and MPII dataset, respectively.

scholarly journals A Fast 4K Video Frame Interpolation Using a Multi-Scale Optical Flow Reconstruction Network

Symmetry ◽  
2019 ◽  
Vol 11 (10) ◽  
pp. 1251 ◽  
Author(s):  
Ahn ◽  
Jeong ◽  
Kim ◽  
Kwon ◽  
Yoo
Keyword(s):  
High Resolution ◽  
Optical Flow ◽  
State Of The Art ◽  
Interpolation Method ◽  
Video Frame ◽  
Frame Interpolation ◽  
Multi Scale ◽  
Reconstruction Scheme ◽  
Flow Reconstruction

Recently, video frame interpolation research developed with a convolutional neural network has shown remarkable results. However, these methods demand huge amounts of memory and run time for high-resolution videos, and are unable to process a 4K frame in a single pass. In this paper, we propose a fast 4K video frame interpolation method, based upon a multi-scale optical flow reconstruction scheme. The proposed method predicts low resolution bi-directional optical flow, and reconstructs it into high resolution. We also proposed consistency and multi-scale smoothness loss to enhance the quality of the predicted optical flow. Furthermore, we use adversarial loss to make the interpolated frame more seamless and natural. We demonstrated that the proposed method outperforms the existing state-of-the-art methods in quantitative evaluation, while it runs up to 4.39× faster than those methods for 4K videos.

Sign in / Sign up

Export Citation Format

Share Document