Exploiting Serialized Fine-Grained Action Recognition Using WiFi Sensing

Gestures serve an important role in enabling natural interactions with computing devices, and they form an important part of everyday nonverbal communication. In increasingly many application scenarios of gesture interaction, such as gesture-based authentication, calligraphy, sketching, and even artistic expression, not only are the underlying gestures complex and consist of multiple strokes but also the correctness of the gestures depends on the order at which the strokes are performed. In this paper, we present WiCG, an innovative and novel WiFi sensing approach for capturing and providing feedback on stroke order. Our approach tracks the user’s hand movement during writing and exploits this information in combination with statistical methods and machine learning techniques to infer what characters have been written and at which stroke order. We consider Chinese calligraphy as our use case as the resulting gestures are highly complex, and their assessment depends on the correct stroke order. We develop a set of analyses and algorithms to overcome many issues of this challenging task. We have conducted extensive experiments and user studies to evaluate our approach. Experimental results show that our approach is highly effective in identifying the written characters and their written stroke order. We show that our approach can adapt to different deployment environments and user patterns.

Algorithm for Predicting the Speed of Gesture Interaction with the User Interface of a Mobile Software Application

In this article, Keystroke Level Model and Touchless Hand Gesture Level Model were adapted, as well as Fitts’s law, to work with touch screens of the mobile devices. The article takes into account typical gestures that users most often perform in the process of interaction with touch screens of mobile devices (Tap, Pinch, Swipe, Pan, Drag-and-Drop), as well as time components of gestures (time for direct execution of a gesture, time for mental preparation for the gesture, time for the user to select the necessary grip on the mobile device, time to select the ‘dominant’ hand to perform gestures, the time to select the finger to perform the gesture, time to move the hand away from the touch screen). A method for determining the speed of user gesture interaction has been developed. The problem was formulated and an algorithm was developed. The details of the operation and use of the algorithm are considered. A list of arrays and variables used as initial data for determining the speed of the sequence execution of operations using gesture interaction has been developed. The preparation of the initial data for the operation of the algorithm provides for a research experiment to obtain average values of the temporal components of gestures. The developed algorithm is intended for use at the design stage of mobile software applications that will run on mobile devices with touch screens.

Magnetic Array Assisted Triboelectric Nanogenerator Sensor for Real-Time Gesture Interaction

In human-machine interaction, robotic hands are useful in many scenarios. To operate robotic hands via gestures instead of handles will greatly improve the convenience and intuition of human-machine interaction. Here, we present a magnetic array assisted sliding triboelectric sensor for achieving a real-time gesture interaction between a human hand and robotic hand. With a finger’s traction movement of flexion or extension, the sensor can induce positive/negative pulse signals. Through counting the pulses in unit time, the degree, speed, and direction of finger motion can be judged in real-time. The magnetic array plays an important role in generating the quantifiable pulses. The designed two parts of magnetic array can transform sliding motion into contact-separation and constrain the sliding pathway, respectively, thus improve the durability, low speed signal amplitude, and stability of the system. This direct quantization approach and optimization of wearable gesture sensor provide a new strategy for achieving a natural, intuitive, and real-time human-robotic interaction.