Vision-based Pakistani Sign Language Recognition Using Bag-of-Words and Support Vector Machines

In order to perform their daily activities, a person is required to communicating with others. This can be a major obstacle for the deaf population of the world, who communicate using sign languages (SL). Pakistani Sign Language (PSL) is used by more than 250,000 deaf Pakistanis. Developing a SL recognition system would greatly facilitate these people. This study aimed to collect data of static and dynamic PSL alphabets and to develop a vision-based system for their recognition using Bag-of-Words (BoW) and Support Vector Machine (SVM) techniques. A total of 5,120 images for 36 static PSL alphabet signs and 353 videos with 45,224 frames for 3 dynamic PSL alphabet signs were collected from 10 native signers of PSL. The developed system used the collected data as input, resized the data to various scales and converted the RGB images into grayscale. The resized grayscale images were segmented using Thresholding technique and features were extracted using Speeded Up Robust Feature (SURF). The obtained SURF descriptors were clustered using K-means clustering. A BoW was obtained by computing the Euclidean distance between the SURF descriptors and the clustered data. The codebooks were divided into training and testing using 5-fold cross validation. The highest overall classification accuracy for static PSL signs was 97.80% at 750×750 image dimensions and 500 Bags. For dynamic PSL signs a 96.53% accuracy was obtained at 480×270 video resolution and 200 Bags.

The Cortical Organization of Syntactic Processing Is Supramodal: Evidence from American Sign Language

Areas within the left-lateralized neural network for language have been found to be sensitive to syntactic complexity in spoken and written language. Previous research has revealed that these areas are active for sign language as well, but whether these areas are specifically responsive to syntactic complexity in sign language independent of lexical processing has yet to be found. To investigate the question, we used fMRI to neuroimage deaf native signers' comprehension of 180 sign strings in American Sign Language (ASL) with a picture-probe recognition task. The ASL strings were all six signs in length but varied at three levels of syntactic complexity: sign lists, two-word sentences, and complex sentences. Syntactic complexity significantly affected comprehension and memory, both behaviorally and neurally, by facilitating accuracy and response time on the picture-probe recognition task and eliciting a left lateralized activation response pattern in anterior and posterior superior temporal sulcus (aSTS and pSTS). Minimal or absent syntactic structure reduced picture-probe recognition and elicited activation in bilateral pSTS and occipital-temporal cortex. These results provide evidence from a sign language, ASL, that the combinatorial processing of anterior STS and pSTS is supramodal in nature. The results further suggest that the neurolinguistic processing of ASL is characterized by overlapping and separable neural systems for syntactic and lexical processing.

Age of exposure and subject/object asymmetries when wh-movement goes rightward

We report an experiment addressing the comprehension of LIS interrogatives in three adult populations with different times of exposure to sign language: native signers, early signers, and late signers. We investigate whether delayed exposure to language affects comprehension of interrogatives and whether there is an advantage for subject dependencies over object dependencies, as systematically reported for spoken languages. The answer to the first question is positive: there is evidence that natives outperform non-native signers, confirming permanent effects of delayed exposure to sign language even decades after childhood. However, the performance in subject interrogatives was lower than in object interrogatives in all groups of participants. We discuss several possible reasons for this unexpected finding.

Upper extremity biomechanics in native and non-native signers

BACKGROUND: Individuals fluent in sign language (signers) born to non-signing, non-deaf parents (non-natives) may have a greater injury risk than signers born to signing, deaf parents (natives). A comprehensive analysis of movement while signing in natives and non-natives has not been completed and could provide insight into the greater injury prevalence of non-natives. OBJECTIVE: The objective of this study was to determine differences in upper extremity biomechanics between non-natives and natives. METHODS: Strength, ‘micro’ rests, muscle activation, ballistic signing, joint angle, and work envelope were captured across groups. RESULTS: Non-natives had fewer rests (p = 0.002) and greater activation (p = 0.008) in non-dominant upper trapezius. For ballistic signing, natives had greater anterior-posterior jerk (p = 0.033) and for joint angle, natives demonstrated greater wrist flexion-extension range of motion (p = 0.040). Natives also demonstrated greater maximum medial-lateral (p = 0.015), and greater minimum medial-lateral (p = 0.019) and superior-inferior (p = 0.027) positions. CONCLUSIONS: We observed that natives presented with more rests and less activation, but greater ballistic tendencies, joint angle, and envelope compared to non-natives. Additional work should explore potential links between these outcomes and injury risk in signers.

Language modality does not change how our brain distinguishes different mathematical operations

In this study, we investigate the impact of experience with a signed language on the neurocognitive processes recruited by adults solving single-digit arithmetic problems. We use event-related potentials (ERPs) to identify the components that are modulated by problem size and operation type in Deaf American Sign Language (ASL) native signers as well as in hearing English-speaking participants. Participants were presented with subtraction and multiplication problems in a delayed verification task. Problem size was manipulated in small and large with an additional extra-large subtraction condition to equate the overall magnitude with large multiplication problems. Results show overall comparable behavioral results across groups and similar ERP dissociations between operation types. First, an early operation type effect is observed between 180ms and 210ms post problem onset, suggesting that both groups have a similar attentional differentiation for processing subtraction and multiplication problems. Second, on the posterior-occipital component between 240ms and 300ms, similarly for both groups only subtraction problems show modulation with problem size suggesting that only this category recruit quantity-related processes. Control analyses exclude this effect as being perceptual and magnitude related. These results are the first evidence that the two operations rely on distinct cognitive processes within the ASL native signing population and this distinction is equivalent to the one observed in the English-speaking population.

The unexplored role of handshape similarity in processing numbers on the hands

With two simple experiments we investigate the overlooked influence of handshape similarity for processing numerical information conveyed on the hands. In most finger-counting sequences there is a tight relationship between the number of fingers raised and the numerical value represented. This creates a possible confound where numbers closer to each other are also represented by handshapes that are more similar. By using the American Sign Language (ASL) number signs we are able to dissociate between the two variables orthogonally. First, we test the effect of handshape similarity in a same/different judgment task in a group of hearing non-signers and then test the interference of handshape in a number judgment task in a group of native ASL signers. Our results show an effect of handshape similarity and its interaction with numerical value even in the group of native signers for whom these handshapes are linguistic symbols and not a learning tool for acquiring numerical concepts. Because prior studies have never considered handshape similarity, these results open new directions for understanding the relationship between finger-based counting, internal hand representations and numerical proficiency.

The unexplored role of handshape similarity in processing numbers on the hands

With two simple experiments we investigate the overlooked influence of handshape similarity for processing numerical information conveyed on the hands. In most finger-counting sequences there is a tight relationship between the number of fingers raised and the numerical value represented. This creates a possible confound where numbers closer to each other are also represented by handshapes that are more similar. By using the ASL number signs we are able to dissociate between the two variables orthogonally. First, we test the effect of handshape similarity in a same/different judgment task in a group of hearing non-signers and then test the interference of handshape in a number judgment task in a group of native ASL signers. Our results show an effect of handshape similarity and its interaction with numerical value even in the group of native signers for whom these handshapes are linguistic symbols and not a learning tool for acquiring numerical concepts. Because prior studies have never considered handshape similarity, these results open new directions for understanding the relationship between finger-based counting, internal hand representations and numerical proficiency.

Theory of Mind (ToM) Development in Thai Deaf Children

This study explored theory of mind (ToM) development in school-aged deaf children. To address new questions, we gave a standard, well-controlled false-belief test to a large (n = 200) sample of severely-to-profoundly deaf children aged 8–15 years in a non-Western culture (Thailand). There were 190 deaf children of hearing parents and 10 deaf native signers with signing deaf parents, consistent with overall population ratios. Comparing our Thai sample’s ToM performance on standard tests of false-belief understanding with that reported in past studies, our results showed a 67% ToM success rate for Thai severely-to-profoundly deaf children of hearing parents similar to collective findings from past research on smaller samples in Australia, Estonia, France, Great Britain, Sweden, and the United States. Our Thai deaf native signers likewise performed equivalently to native signers of similar age studied in past research in Australia and the United States. Collectively, the detailed findings of our study suggest promising new directions for future studies to pursue in order to build upon this novel and theoretically provocative evidence about how ToM development and ToM delay unfold for school-aged deaf children growing up in varied cultures, school settings, and family circumstances.

Reading skills of D/deaf students – native signers

Reading skills of D/deaf students fall behind their hearing peers. The difference in reading skills between D/deaf and hearing children has not decreased for over past three decades. Low level of reading skills in D/deaf students has been associated with their language delay, which is mainly observed in D/deaf children using spoken language that is not fully accessible to “D/deaf individuals” instead of “ppl with hearing impairment”. D/deaf children immersed in sign language since their birth usually do not encounter language problems and they have a potential to become highly-skilled readers. In the present studies we have investigated reading skills of D/deaf students who are native signers of Polish Sign Language. The results have indicated that D/deaf students showed lower level of reading skills than their hearing peers. The present studies call in question Polish education system dedicated to D/deaf students who are native signers. The obtained results suggest that reading classes are probably not adapted to the needs and abilities of highly competent signers.