Advances in Low-Resource and Endangered Languages

This paper reports on the approaches and results for the collection, analysis, and processing of low-resource and endangered languages carried out under the Low-Resource Languages for Emergent Incidents (LORELEI) Program1. LORELEI was a multi-year research and development program designed to discover new methods of quickly ramping up human language technology capabilities for low-resource languages, grounded in situations such as humanitarian and disaster relief use cases. The goal was to advance human language technology methods to better enable rapid, low-cost development of capabilities, with a focus on developing methods that apply to languages of any type from any language family, thus eliminating the need to tailor specific technologies to a narrow set of input languages with specific typological characteristics. We report in detail on evaluation scenarios developed for the program.

Human Language Technology

Human language technology encompasses a wide array of speech and text processing capabilities. The Defense Advanced Research Projects Agency’s pioneering research on automatic transcription, translation, and content analysis were major artificial intelligence success stories that changed science fiction into social fact. During a 40-year period, 10 seminal DARPA programs produced breakthrough capabilities that were further improved and widely deployed in popular consumer products, as well as in many commercial, industrial, and governmental applications. The Defense Advanced Research Projects Agency produced the core enabling technologies by setting crisp, aggressive, and quantitative technical objectives; by providing strong multiyear funding; and by using the Defense Advanced Research Projects Agency’s Common Task Method, which was powerful, efficient, and easy to administer. To achieve these breakthroughs, multidisciplinary academic and industrial research teams working in parallel took advantage of increasingly large and diverse sets of linguistic data and rapidly increasing computational power to develop and use increasingly sophisticated forms of machine learning. This article describes the progression of technical advances underlying key successes and the seminal programs that produced them.

Managing mining project documentation using human language technology

Purpose This paper aims to develop a system, which would enable efficient management and exploitation of documentation in electronic form, related to mining projects, with information retrieval and information extraction (IE) features, using various language resources and natural language processing. Design/methodology/approach The system is designed to integrate textual, lexical, semantic and terminological resources, enabling advanced document search and extraction of information. These resources are integrated with a set of Web services and applications, for different user profiles and use-cases. Findings The use of the system is illustrated by examples demonstrating keyword search supported by Web query expansion services, search based on regular expressions, corpus search based on local grammars, followed by extraction of information based on this search and finally, search with lexical masks using domain and semantic markers. Originality/value The presented system is the first software solution for implementation of human language technology in management of documentation from the mining engineering domain, but it is also applicable to other engineering and non-engineering domains. The system is independent of the type of alphabet (Cyrillic and Latin), which makes it applicable to other languages of the Balkan region related to Serbian, and its support for morphological dictionaries can be applied in most morphologically complex languages, such as Slavic languages. Significant search improvements and the efficiency of IE are based on semantic networks and terminology dictionaries, with the support of local grammars.