Algorithm for decoding visual gestures for an assistive virtual keyboard

AbstrakSejak pertama kali komputer ditemukan, keyboard selalu menjadi alat utama yang menjadi penghubung interaksi antara manusia dan komputer. Saat ini banyak komputer yang menerapkan teknologi pengolahan citra untuk menjadikannya perantara interaksi antara komputer dan manusia.Dalam penelitian ini, penulis mencoba untuk menerapkan teknologi pengolahan citra yang digunakan untuk keyboard virtual pada aplikasi web. Digunakan webcam untuk menangkap citra ujung jari telunjuk. Hasil capture citra akan dikirimkan ke server localhost untuk diproses dengan image processing. Untuk mendeteksi ujung jari telunjuk, digunakan metode Haar Cascade Classifier. Proses pendeteksian tersebut menghasilkan koordinat yang akan dikirimkan ke aplikasi web yang selanjutnya dijadikan acuan untuk menentukan posisi tombol pada keyboard virtual. Sehingga keyboard virtual akan menampilan karakter sesuai dengan yang ditunjuk oleh ujung jari telunjuk.Dari hasil pengujian yang dilakukan, jarak optimal ujung jari telunjuk dengan webcam adalah 20 – 35 cm. Derajat kemiringan ujung jari telunjuk untuk dapat terdeteksi antara 0° – 10°. Sistem mampu mengenali ujung jari telunjuk pada ruangan berlatar belakang putih polos dan terdapat sedikit perabot. Waktu respon untuk menampilkan karakter keyboard virtual rata-rata 5,156 detik. Sehingga keyboard virtual pada sistem ini belum mampu dijadikan antarmuka pada aplikasi web, dikarenakan masih sulit digunakan dalam mengarahkan ujung jari telunjuk ke tombol karakter yang diinginkan.Kata kunci—aplikasi web, Haar Cascade Classifier, keyboard virtual, pengolahan citra AbstractSince the first computer was founded, keyboard is always been a primary tool for interaction between humans and computers. Today, many computers use image processing technology to make interaction between computers and humans.The author try to apply image processing technology that implemented to virtual keyboard on web application. Using a webcam to capture the tip of index finger and the results will be sent to the localhost server for processing with image processing. Using Haar Cascade Classifier method to detect the tip of index finger, it will produce coordinates that sent to the web application and it used as a reference for determining button positions on virtual keyboard. Virtual keyboard characters will display after appointed by the tip of index finger.From testing results, optimal distance from index finger to webcam is 20 – 35 cm. System can recognize the tip of index finger on white background and room with few furnitures. Average response time for displaying virtual keyboard sentences is 3 minutes and 28.838 seconds. So the virtual keyboard on this system was not able to be used as interface on web application, because it difficult to use in directing the tip of index finger to the character keys.Keywords—web application, Haar Cascade Classifier, virtual keyboard, image processing

Download Full-text

Exploring a Bilingual Next Word Predictor for a Federated Learning Mobile Application

10.36227/techrxiv.18058682.v1 ◽

2022 ◽

Author(s):

Natali Alfonso Burgos ◽

Karol Kiš ◽

Peter Bakarac ◽

Michal Kvasnica ◽

Giovanni Licitra

Keyword(s):

Learning Environment ◽

Mobile Application ◽

Homomorphic Encryption ◽

Aggregation Method ◽

Performance Gap ◽

Parallel Corpora ◽

Client Server ◽

Virtual Keyboard ◽

Distributed Training ◽

Target Performance

We explore a bilingual next-word predictor (NWP) under federated optimization for a mobile application. A character-based LSTM is server-trained on English and Dutch texts from a custom parallel corpora. This is used as the target performance. We simulate a federated learning environment to assess the feasibility of distributed training for the same model. The popular Federated Averaging (FedAvg) algorithm is used as the aggregation method. We show that the federated LSTM achieves decent performance, yet it is still sub-optimal. We suggest possible next steps to bridge this performance gap. Furthermore, we explore the effects of language imbalance varying the ratio of English and Dutch training texts (or clients). We show the model upholds performance (of the balanced case) up and until a 80/20 imbalance before decaying rapidly. Lastly, we describe the implementation of local client training, word prediction and client-server communication in a custom virtual keyboard for Android platforms. Additionally, homomorphic encryption is applied to provide with secure aggregation guarding the user from malicious servers.

Download Full-text