Real-time grasping strategies using event camera

AbstractRobotic vision plays a key role for perceiving the environment in grasping applications. However, the conventional framed-based robotic vision, suffering from motion blur and low sampling rate, may not meet the automation needs of evolving industrial requirements. This paper, for the first time, proposes an event-based robotic grasping framework for multiple known and unknown objects in a cluttered scene. With advantages of microsecond-level sampling rate and no motion blur of event camera, the model-based and model-free approaches are developed for known and unknown objects’ grasping respectively. The event-based multi-view approach is used to localize the objects in the scene in the model-based approach, and then point cloud processing is utilized to cluster and register the objects. The proposed model-free approach, on the other hand, utilizes the developed event-based object segmentation, visual servoing and grasp planning to localize, align to, and grasp the targeting object. Using a UR10 robot with an eye-in-hand neuromorphic camera and a Barrett hand gripper, the proposed approaches are experimentally validated with objects of different sizes. Furthermore, it demonstrates robustness and a significant advantage over grasping with a traditional frame-based camera in low-light conditions.

Download Full-text

An Unified Approach to Model-Based and Model-Free Visual Servoing

Computer Vision — ECCV 2002 - Lecture Notes in Computer Science ◽

10.1007/3-540-47979-1_29 ◽

2002 ◽

pp. 433-447 ◽

Cited By ~ 4

Author(s):

Ezio Malis

Keyword(s):

Visual Servoing ◽

Unified Approach ◽

Model Based ◽

Model Free

Download Full-text

Event-based tracking of human hands

Sensor Review ◽

10.1108/sr-03-2021-0095 ◽

2021 ◽

Vol 41 (4) ◽

pp. 382-389

Author(s):

Laura Duarte ◽

Mohammad Safeea ◽

Pedro Neto

Keyword(s):

Dynamic Range ◽

Estimation Error ◽

Computational Cost ◽

Motion Blur ◽

Hand Position ◽

Single Event ◽

Content Type ◽

Event Camera ◽

Event Based ◽

Human Hands

Purpose This paper proposes a novel method for human hands tracking using data from an event camera. The event camera detects changes in brightness, measuring motion, with low latency, no motion blur, low power consumption and high dynamic range. Captured frames are analysed using lightweight algorithms reporting three-dimensional (3D) hand position data. The chosen pick-and-place scenario serves as an example input for collaborative human–robot interactions and in obstacle avoidance for human–robot safety applications. Design/methodology/approach Events data are pre-processed into intensity frames. The regions of interest (ROI) are defined through object edge event activity, reducing noise. ROI features are extracted for use in-depth perception. Findings Event-based tracking of human hand demonstrated feasible, in real time and at a low computational cost. The proposed ROI-finding method reduces noise from intensity images, achieving up to 89% of data reduction in relation to the original, while preserving the features. The depth estimation error in relation to ground truth (measured with wearables), measured using dynamic time warping and using a single event camera, is from 15 to 30 millimetres, depending on the plane it is measured. Originality/value Tracking of human hands in 3 D space using a single event camera data and lightweight algorithms to define ROI features (hands tracking in space).

Download Full-text

Model-based and model-free reinforcement learning for visual servoing

2009 IEEE International Conference on Robotics and Automation ◽

10.1109/robot.2009.5152834 ◽

2009 ◽

Cited By ~ 8

Author(s):

A.M. Farahmand ◽

A. Shademan ◽

M. Jagersand ◽

C. Szepesvari

Keyword(s):

Reinforcement Learning ◽

Visual Servoing ◽

Model Based ◽

Model Free

Download Full-text

Representation, abstraction, and simple-minded sophisticates

Behavioral and Brain Sciences ◽

10.1017/s0140525x19002942 ◽

2020 ◽

Vol 43 ◽

Author(s):

Peter Dayan

Keyword(s):

Decision Theory ◽

Predictive Coding ◽

Bayesian Decision Theory ◽

Bayesian Decision ◽

Model Based ◽

Model Free

Abstract Bayesian decision theory provides a simple formal elucidation of some of the ways that representation and representational abstraction are involved with, and exploit, both prediction and its rather distant cousin, predictive coding. Both model-free and model-based methods are involved.

Download Full-text

Shaping Model-Free Reinforcement-Learning with Model-Based Pseudorewards

10.32470/ccn.2018.1191-0 ◽

2018 ◽

Author(s):

Paul Krueger ◽

Thomas Griffiths

Keyword(s):

Reinforcement Learning ◽

Model Based ◽

Model Free

Download Full-text

Model-Based and Model-Free Social Cognition

10.31234/osf.io/ue6j2 ◽

2019 ◽

Author(s):

Leor M Hackel ◽

Jeffrey Jordan Berg ◽

Björn Lindström ◽

David Amodio

Keyword(s):

Reinforcement Learning ◽

Social Cognition ◽

Learning Strategies ◽

Memory Systems ◽

Learning Task ◽

Financial Advisors ◽

Model Based ◽

Model Free ◽

Systems Model ◽

Task Assessment

Do habits play a role in our social impressions? To investigate the contribution of habits to the formation of social attitudes, we examined the roles of model-free and model-based reinforcement learning in social interactions—computations linked in past work to habit and planning, respectively. Participants in this study learned about novel individuals in a sequential reinforcement learning paradigm, choosing financial advisors who led them to high- or low-paying stocks. Results indicated that participants relied on both model-based and model-free learning, such that each independently predicted choice during the learning task and self-reported liking in a post-task assessment. Specifically, participants liked advisors who could provide large future rewards as well as advisors who had provided them with large rewards in the past. Moreover, participants varied in their use of model-based and model-free learning strategies, and this individual difference influenced the way in which learning related to self-reported attitudes: among participants who relied more on model-free learning, model-free social learning related more to post-task attitudes. We discuss implications for attitudes, trait impressions, and social behavior, as well as the role of habits in a memory systems model of social cognition.

Download Full-text