Real-time, low-latency closed-loop feedback using markerless posture tracking

The ability to control a behavioral task or stimulate neural activity based on animal behavior in real-time is an important tool for experimental neuroscientists. Ideally, such tools are noninvasive, low-latency, and provide interfaces to trigger external hardware based on posture. Recent advances in pose estimation with deep learning allows researchers to train deep neural networks to accurately quantify a wide variety of animal behaviors. Here, we provide a new <monospace>DeepLabCut-Live!</monospace> package that achieves low-latency real-time pose estimation (within 15 ms, >100 FPS), with an additional forward-prediction module that achieves zero-latency feedback, and a dynamic-cropping mode that allows for higher inference speeds. We also provide three options for using this tool with ease: (1) a stand-alone GUI (called <monospace>DLC-Live! GUI</monospace>), and integration into (2) <monospace>Bonsai,</monospace> and (3) <monospace>AutoPilot</monospace>. Lastly, we benchmarked performance on a wide range of systems so that experimentalists can easily decide what hardware is required for their needs.

Download Full-text

Real-time, low-latency closed-loop feedback using markerless posture tracking

10.1101/2020.08.04.236422 ◽

2020 ◽

Author(s):

Gary Kane ◽

Gonçalo Lopes ◽

Jonny L. Saunders ◽

Alexander Mathis ◽

Mackenzie W. Mathis

Keyword(s):

Real Time ◽

Pose Estimation ◽

Animal Behavior ◽

Deep Neural Networks ◽

Low Latency ◽

Forward Prediction ◽

Prediction Module ◽

Wide Range ◽

Loop Feedback ◽

Closed Loop Feedback

AbstractThe ability to control a behavioral task or stimulate neural activity based on animal behavior in real-time is an important tool for experimental neuroscientists. Ideally, such tools are noninvasive, low-latency, and provide interfaces to trigger external hardware based on posture. Recent advances in pose estimation with deep learning allows researchers to train deep neural networks to accurately quantify a wide variety of animal behaviors. Here we provide a new DeepLabCut-Live! package that achieves low-latency real-time pose estimation (within 15 ms, >100 FPS), with an additional forward-prediction module that achieves zero-latency feedback, and a dynamic-cropping mode that allows for higher inference speeds. We also provide three options for using this tool with ease: (1) a stand-alone GUI (called DLC-Live! GUI), and integration into (2) Bonsai and (3) AutoPilot. Lastly, we benchmarked performance on a wide range of systems so that experimentalists can easily decide what hardware is required for their needs.

Download Full-text

Decision letter: Real-time, low-latency closed-loop feedback using markerless posture tracking

10.7554/elife.61909.sa1 ◽

2020 ◽

Author(s):

Gordon J Berman ◽

Tiago Branco

Keyword(s):

Real Time ◽

Closed Loop ◽

Low Latency ◽

Loop Feedback ◽

Closed Loop Feedback ◽

Posture Tracking

Download Full-text

Author response: Real-time, low-latency closed-loop feedback using markerless posture tracking

10.7554/elife.61909.sa2 ◽

2020 ◽

Author(s):

Gary A Kane ◽

Gonçalo Lopes ◽

Jonny L Saunders ◽

Alexander Mathis ◽

Mackenzie W Mathis

Keyword(s):

Real Time ◽

Closed Loop ◽

Author Response ◽

Low Latency ◽

Loop Feedback ◽

Closed Loop Feedback ◽

Posture Tracking

Download Full-text

Integrated Prognostics Observer for Condition Monitoring of an Automated Manual Transmission Dry Clutch System

International Journal of Prognostics and Health Management ◽

10.36001/ijphm.2017.v8i2.2638 ◽

2020 ◽

Vol 8 (2) ◽

Author(s):

Sivakumar Ramalingam ◽

Hanumath VV Prasad ◽

Srinivasa Prakash Regalla

Keyword(s):

Real Time ◽

Condition Monitoring ◽

Closed Loop ◽

Manual Transmission ◽

The Real ◽

Clutch System ◽

Loop Feedback ◽

Automated Manual Transmission ◽

Closed Loop Feedback ◽

Dry Clutch

The closed loop feedback control system of an Automated Manual Transmission (AMT) electro-pneumatic clutch actuator is used for intelligent real time condition monitoring, enhanced diagnostics and prognostic health management of the dry clutch system, by integrating with the existing gearbox prognostics observer. The real-time sensor data of the clutch actuator piston position is analyzed for monitoring the condition of the clutch system. Original parameters of the new clutch are stored in the Electrically Erasable Programmable Read-only Memory (EEPROM) of the AMT controller and the real-time data is used by the observer for assessing the degradation/wear of the frictional clutch parts. Also, clutch slip during torque transmission is monitored, using the engine speed and the gearbox input shaft speed from Controller Area Network (CAN). Condition monitoring of clutch system provides enhanced prognostic functionality for AMT system which ensures consistent clutch performance, gear shift quality and timely warning for recalibration, repair and/or replacement of the critical wear and tear parts. Also, systematic analysis of the monitored data provides an accurate diagnosis of a developing fault. Thus, with the advanced control systems in place for AMT, a closed loop feedback based condition monitoring system is modelled for improved diagnostics and prognostics of AMT clutch system.

Download Full-text

Toward Coordination Control of Multiple Fish-Like Robots: Real-Time Vision-Based Pose Estimation and Tracking via Deep Neural Networks

IEEE/CAA Journal of Automatica Sinica ◽

10.1109/jas.2021.1004228 ◽

2021 ◽

Vol 8 (12) ◽

pp. 1964-1976

Author(s):

Tianhao Zhang ◽

Jiuhong Xiao ◽

Liang Li ◽

Chen Wang ◽

Guangming Xie

Keyword(s):

Neural Networks ◽

Real Time ◽

Pose Estimation ◽

Deep Neural Networks ◽

Coordination Control

Download Full-text

Multi-Scale, Multi-Object and Real-Time Face Detection and Head Pose Estimation Using Deep Neural Networks

The Journal of Korea Robotics Society ◽

10.7746/jkros.2017.12.3.313 ◽

2017 ◽

Vol 12 (3) ◽

pp. 313-321 ◽

Cited By ~ 1

Author(s):

Byungtae Ahn ◽

◽

Dong-Geol Choi ◽

In So Kweon

Keyword(s):

Neural Networks ◽

Real Time ◽

Face Detection ◽

Pose Estimation ◽

Deep Neural Networks ◽

Head Pose Estimation ◽

Head Pose ◽

Multi Scale

Download Full-text

Closed-loop feedback control for microfluidic systems through automated capacitive fluid height sensing

10.1101/221002 ◽

2017 ◽

Author(s):

L. R. Soenksen ◽

T. Kassis ◽

M. Noh ◽

L.G. Griffith ◽

D.L. Trumper

Keyword(s):

Feedback Control ◽

Closed Loop ◽

Cost Effective ◽

Capacitive Sensor ◽

Open Loop ◽

Small Scale ◽

Microfluidic Systems ◽

Wide Range ◽

Loop Feedback ◽

Closed Loop Feedback

AbstractPrecise fluid height sensing in open-channel microfluidics has long been a desirable feature for a wide range of applications. However, performing accurate measurements of the fluid level in small-scale reservoirs (<1mL) has proven to be an elusive goal, especially if direct fluid-sensor contact needs to be avoided. In particular, gravity-driven systems used in several microfluidic applications to establish pressure gradients and impose flow remain open-loop and largely unmonitored due to these sensing limitations. Here we present an optimized self-shielded coplanar capacitive sensor design and automated control system to provide submillimeter fluid-height resolution (~250 μm) and control of small-scale open reservoirs without the need for direct fluid contact. Results from testing and validation of our optimized sensor and system also suggest that accurate fluid height information can be used to robustly characterize, calibrate and dynamically control a range of microfluidic systems with complex pumping mechanisms, even in cell culture conditions. Capacitive sensing technology provides a scalable and cost-effective way to enable continuous monitoring and closed-loop feedback control of fluid volumes in small-scale gravity-dominated wells in a variety of microfluidic applications.

Download Full-text

Real-Time WebRTC based Mobile Surveillance System

International Journal of Engineering and Management Research ◽

10.31033/ijemr.11.3.4 ◽

2021 ◽

Vol 11 (3) ◽

Author(s):

Alistair Baretto ◽

Noel Pudussery ◽

Veerasai Subramaniam ◽

Amroz Siddiqui

Keyword(s):

Object Detection ◽

Real Time ◽

Deep Neural Networks ◽

Ease Of Use ◽

Problem Statement ◽

Wide Range ◽

Technological Growth ◽

Computationally Intensive ◽

Processing Techniques ◽

Opening Up

The rapid growth that has taken place in Computer Vision has been instrumental in driving the advancement of Image processing techniques and drawing inferences from them. Combined with the enormous capabilities that Deep Neural networks bring to the table, computers can be efficiently trained to automate the tasks and yield accurate and robust results quickly thus optimizing the process. Technological growth has enabled us to bring such computationally intensive tasks to lighter and lower-end mobile devices thus opening up a wide range of possibilities. WebRTC-the open-source web standard enables us to send multimedia-based data from peer to peer paving the way for Real-time Communication over the Web. With this project, we aim to build on one such opportunity that can enable us to perform custom object detection through an android based application installed on our mobile phones. Therefore, our problem statement is to be able to capture real-time feeds, perform custom object detection, generate inference results, and appropriately send intruder alerts when needed. To implement this, we propose a mobile-based over-the-cloud solution that can capitalize on the enormous and encouraging features of the YOLO algorithm and incorporate the functionalities of OpenCV’s DNN module for providing us with fast and correct inferences. Coupled with a good and intuitive UI, we can ensure ease of use of our application.

Download Full-text