A Hardware–Software Co-Design Framework for Real-Time Video Stabilization

2019 ◽  
Vol 29 (02) ◽  
pp. 2050027
Author(s):  
Hassan Javed ◽  
Muhammad Bilal ◽  
Shahid Masud
Keyword(s):  
Real Time ◽  
Video Stream ◽  
Video Stabilization ◽  
Design Framework ◽  
Local Motion ◽  
Video Frames ◽  
Key Points ◽  
Implementation Scheme ◽  
Computationally Intensive ◽  
High Level

Live digital video is a valuable source of information in security, broadcast and industrial quality control applications. Motion jitter due to camera and platform instability is a common artefact found in captured video which renders it less effective for subsequent computer vision tasks such as detection and tracking of objects, background modeling, mosaicking, etc. The process of algorithmically compensating for the motion jitter is hence a mandatory pre-processing step in many applications. This process, called video stabilization, requires estimation of global motion from consecutive video frames and is constrainted by additional challenges such as preservation of intentional motion and native frame resolution. The problem is exacerbated in the presence of local motion of foreground objects and requires robust compensation of the same. As such achieving real-time performance for this computationally intensive operation is a difficult task for embedded processors with limited computational and memory resources. In this work, development of an optimized hardware–software co-design framework for video stabilization has been investigated. Efficient video stabilization depends on the identification of key points in the frame which in turn requires dense feature calculation at the pixel level. This task has been identified to be most suitable for offloading the pipelined hardware implemented in the FPGA fabric due to the involvement of complex memory and computation operations. Subsequent tasks to be performed for the overall stabilization algorithm utilize these sparse key points and have been found to be efficiently handled in the software. The proposed Hardware–Software (HW–SW) co-design framework has been implemented on Zedboard FPGA platform which houses Xilinx Zynq SOC equipped with ARM A9 processor. The proposed implementation scheme can process real-time video stream input at 28 frames per second and is at least twice faster than the corresponding software-only approach. Two different hardware accelerator designs have been implemented using different high-level synthesis tools using rapid prototyping principle and consume less than 50% of logic resources available on the host FPGA while being at least 30% faster than contemporary designs.

scholarly journals Deep Reinforcement Learning for End-to-End Local Motion Planning of Autonomous Aerial Robots in Unknown Outdoor Environments: Real-Time Flight Experiments

Sensors ◽  
2021 ◽  
Vol 21 (7) ◽  
pp. 2534
Author(s):  
Oualid Doukhi ◽  
Deok-Jin Lee
Keyword(s):  
Reinforcement Learning ◽  
Real Time ◽  
Autonomous Navigation ◽  
Control Technique ◽  
Local Motion ◽  
Aerial Robots ◽  
Novel Approach ◽  
Outdoor Environments ◽  
Aerial Vehicle ◽  
High Level

Autonomous navigation and collision avoidance missions represent a significant challenge for robotics systems as they generally operate in dynamic environments that require a high level of autonomy and flexible decision-making capabilities. This challenge becomes more applicable in micro aerial vehicles (MAVs) due to their limited size and computational power. This paper presents a novel approach for enabling a micro aerial vehicle system equipped with a laser range finder to autonomously navigate among obstacles and achieve a user-specified goal location in a GPS-denied environment, without the need for mapping or path planning. The proposed system uses an actor–critic-based reinforcement learning technique to train the aerial robot in a Gazebo simulator to perform a point-goal navigation task by directly mapping the noisy MAV’s state and laser scan measurements to continuous motion control. The obtained policy can perform collision-free flight in the real world while being trained entirely on a 3D simulator. Intensive simulations and real-time experiments were conducted and compared with a nonlinear model predictive control technique to show the generalization capabilities to new unseen environments, and robustness against localization noise. The obtained results demonstrate our system’s effectiveness in flying safely and reaching the desired points by planning smooth forward linear velocity and heading rates.

scholarly journals Efficient Real-Time Video-in-Video Insertion into a Pre-Encoded Video Stream

2011 ◽  
Vol 2011 ◽  
pp. 1-11 ◽  
Author(s):  
Dan Grois ◽  
Evgeny Kaminsky ◽  
Ofer Hadar
Keyword(s):  
Real Time ◽  
Efficient Method ◽  
Video Sequence ◽  
Video Stream ◽  
Experimental Results ◽  
Second Step ◽  
Visual Content ◽  
Low Bit Rate ◽  
Computationally Intensive ◽  
Insertion Process

This work relates to the developing and implementing of an efficient method and system for the fast real-time Video-in-Video (ViV) insertion, thereby enabling efficiently inserting a video sequence into a predefined location within a pre-encoded video stream. The proposed method and system are based on dividing the video insertion process into two steps. The first step (i.e., the Video-in-Video Constrained Format (ViVCF) encoder) includes the modification of the conventional H.264/AVC video encoder to support the visual content insertion Constrained Format (CF), including generation of isolated regions without using the Frequent Macroblock Ordering (FMO) slicing, and to support the fast real-time insertion of overlays. Although, the first step is computationally intensive, it should to be performed only once even if different overlays have to be modified (e.g., for different users). The second step for performing the ViV insertion (i.e., the ViVCF inserter) is relatively simple (operating mostly in a bit-domain), and is performed separately for each different overlay. The performance of the presented method and system is demonstrated and compared with the H.264/AVC reference software (JM 12); according to our experimental results, there is a significantly low bit-rate overhead, while there is substantially no degradation in the PSNR quality.

scholarly journals Deep Learning-Based Real-Time Multiple-Person Action Recognition System

Sensors ◽  
2020 ◽  
Vol 20 (17) ◽  
pp. 4758
Author(s):  
Jen-Kai Tsai ◽  
Chen-Chien Hsu ◽  
Wei-Yen Wang ◽  
Shao-Kang Huang
Keyword(s):  
Deep Learning ◽  
Real Time ◽  
Action Recognition ◽  
Recognition System ◽  
Video Stream ◽  
Term Care ◽  
Sliding Windows ◽  
Video Frames ◽  
Single Person ◽  
Smart Surveillance

Action recognition has gained great attention in automatic video analysis, greatly reducing the cost of human resources for smart surveillance. Most methods, however, focus on the detection of only one action event for a single person in a well-segmented video, rather than the recognition of multiple actions performed by more than one person at the same time for an untrimmed video. In this paper, we propose a deep learning-based multiple-person action recognition system for use in various real-time smart surveillance applications. By capturing a video stream of the scene, the proposed system can detect and track multiple people appearing in the scene and subsequently recognize their actions. Thanks to high resolution of the video frames, we establish a zoom-in function to obtain more satisfactory action recognition results when people in the scene become too far from the camera. To further improve the accuracy, recognition results from inflated 3D ConvNet (I3D) with multiple sliding windows are processed by a nonmaximum suppression (NMS) approach to obtain a more robust decision. Experimental results show that the proposed method can perform multiple-person action recognition in real time suitable for applications such as long-term care environments.

Between Jerusalem and Athens

2018 ◽  
Author(s):  
Nurit Yaari
Keyword(s):  
Real Time ◽  
Social Conditions ◽  
Cultural Events ◽  
Greek Drama ◽  
The Past ◽  
Key Points ◽  
Mise En Scene ◽  
Dramatic Writing ◽  
Artistic Performance

How does a theatrical tradition emerge in the fields of dramatic writing and artistic performance? Can a culture, in which theatre played no part in the past, create a theatrical tradition in real time—and how? What was the contribution of classical Greek drama to the evolution of Israeli theatre? How do political and social conditions affect the encounter between cultures—and what role do they play in creating a theatre with a distinctive identity? This book, the first of its kind, attempts to answer these and other questions, by examining the reception of classical Greek drama in the Israeli theatre over the last seventy years. It deals with dramatic and aesthetic issues while analysing translations, adaptations, new writing, mise-en-scène, and ‘post dramatic’ performances of classical Greek drama that were created and staged at key points of the development of Israeli culture amidst fateful political, social, and cultural events in the country’s history.

scholarly journals Graph-Based Construction of 3D Korean Giwa House Models

Buildings ◽  
2019 ◽  
Vol 9 (3) ◽  
pp. 68
Author(s):  
Mankyu Sung
Keyword(s):  
Data Structure ◽  
Computer Graphics ◽  
Real Time ◽  
Texture Image ◽  
Time Rate ◽  
Commercial Software ◽  
Spline Curve ◽  
High Level ◽  
Complicated Part ◽  
Different Parts

This paper proposes a graph-based algorithm for constructing 3D Korean traditional houses automatically using a computer graphics technique. In particular, we target designing the most popular traditional house type, a giwa house, whose roof is covered with a set of Korean traditional roof tiles called giwa. In our approach, we divided the whole design processes into two different parts. At a high level, we propose a special data structure called ‘modeling graphs’. A modeling graph consists of a set of nodes and edges. A node represents a particular component of the house and an edge represents the connection between two components with all associated parameters, including an offset vector between components. Users can easily add/ delete nodes and make them connect by an edge through a few mouse clicks. Once a modeling graph is built, then it is interpreted and rendered on a component-by-component basis by traversing nodes in a procedural way. At a low level, we came up with all the required parameters for constructing the components. Among all the components, the most beautiful but complicated part is the gently curved roof structures. In order to represent the sophisticated roof style, we introduce a spline curve-based modeling technique that is able to create curvy silhouettes of three different roof styles. In this process, rather than just applying a simple texture image onto the roof, which is widely used in commercial software, we actually laid out 3D giwa tiles on the roof seamlessly, which generated more realistic looks. Through many experiments, we verified that the proposed algorithm can model and render the giwa house at a real time rate.

scholarly journals Real-Time High-Level Acute Pain Detection Using a Smartphone and a Wrist-Worn Electrodermal Activity Sensor

Sensors ◽  
2021 ◽  
Vol 21 (12) ◽  
pp. 3956
Author(s):  
Youngsun Kong ◽  
Hugo F. Posada-Quintero ◽  
Ki H. Chon
Keyword(s):  
Real Time ◽  
Electrodermal Activity ◽  
Smartphone Application ◽  
Real Time Processing ◽  
Pain Stimulation ◽  
Significant Difference ◽  
Before And After ◽  
Electrical Pulses ◽  
Pain Detection ◽  
High Level

The subjectiveness of pain can lead to inaccurate prescribing of pain medication, which can exacerbate drug addiction and overdose. Given that pain is often experienced in patients’ homes, there is an urgent need for ambulatory devices that can quantify pain in real-time. We implemented three time- and frequency-domain electrodermal activity (EDA) indices in our smartphone application that collects EDA signals using a wrist-worn device. We then evaluated our computational algorithms using thermal grill data from ten subjects. The thermal grill delivered a level of pain that was calibrated for each subject to be 8 out of 10 on a visual analog scale (VAS). Furthermore, we simulated the real-time processing of the smartphone application using a dataset pre-collected from another group of fifteen subjects who underwent pain stimulation using electrical pulses, which elicited a VAS pain score level 7 out of 10. All EDA features showed significant difference between painless and pain segments, termed for the 5-s segments before and after each pain stimulus. Random forest showed the highest accuracy in detecting pain, 81.5%, with 78.9% sensitivity and 84.2% specificity with leave-one-subject-out cross-validation approach. Our results show the potential of a smartphone application to provide near real-time objective pain detection.

scholarly journals A Modified KNN Algorithm for High-Performance Computing on FPGA of Real-Time m-QAM Demodulators

Electronics ◽  
2021 ◽  
Vol 10 (5) ◽  
pp. 627
Author(s):  
David Marquez-Viloria ◽  
Luis Castano-Londono ◽  
Neil Guerrero-Gonzalez
Keyword(s):  
Real Time ◽  
High Performance ◽  
Interference Mitigation ◽  
Parallel Implementation ◽  
Computational Time ◽  
Successful Implementation ◽  
Interchannel Interference ◽  
The Difference ◽  
High Level ◽  
Performance Computing

A methodology for scalable and concurrent real-time implementation of highly recurrent algorithms is presented and experimentally validated using the AWS-FPGA. This paper presents a parallel implementation of a KNN algorithm focused on the m-QAM demodulators using high-level synthesis for fast prototyping, parameterization, and scalability of the design. The proposed design shows the successful implementation of the KNN algorithm for interchannel interference mitigation in a 3 × 16 Gbaud 16-QAM Nyquist WDM system. Additionally, we present a modified version of the KNN algorithm in which comparisons among data symbols are reduced by identifying the closest neighbor using the rule of the 8-connected clusters used for image processing. Real-time implementation of the modified KNN on a Xilinx Virtex UltraScale+ VU9P AWS-FPGA board was compared with the results obtained in previous work using the same data from the same experimental setup but offline DSP using Matlab. The results show that the difference is negligible below FEC limit. Additionally, the modified KNN shows a reduction of operations from 43 percent to 75 percent, depending on the symbol’s position in the constellation, achieving a reduction 47.25% reduction in total computational time for 100 K input symbols processed on 20 parallel cores compared to the KNN algorithm.

scholarly journals The Relationship between Symptom Flare of Atopic Dermatitis and Airborne Japanese Cedar and Cypress Pollen Counts: A Self-Scoring Diary Study

2012 ◽  
Vol 2012 ◽  
pp. 1-5 ◽  
Author(s):  
Haruko Nishie ◽  
Mariko Kato ◽  
Shiori Kato ◽  
Hiroshi Odajima ◽  
Rumiko Shibata ◽  
...  
Keyword(s):  
Atopic Dermatitis ◽  
Real Time ◽  
Pollen Season ◽  
Airborne Pollen ◽  
Japanese Cedar ◽  
Diary Study ◽  
Pollen Counts ◽  
High Level ◽  
The Relationship

Background. With an increase in Japanese cedar and cypress (JC) pollinosis, the relationship between JC pollen and atopic dermatitis (AD) has been studied. Some reports suggest that JC pollen can be one exacerbating factor for AD, but there has been no report that discusses JC pollen counts relating to AD symptom flare although actual airborne JC pollen counts can widely fluctuate throughout the pollen season. Objective. The relationship between symptom flare of AD and airborne JC pollen counts was examined. Methods. We monitored JC pollen counts in real time and divided the counts into low and high level. We then analyzed self-scored “itch intensity” recorded by 14 AD patients through a self-scoring diary. Results. Among the 14 patients, 7 had significantly higher itch intensity while the pollen counts were high. Conclusion. Even during the pollen season, actual airborne pollen counts can widely fluctuate. Our study suggested that symptom flare of AD could be influenced by the actual pollen counts.

Sign in / Sign up

Export Citation Format

Share Document