Mining open source text documents for intelligence gathering

In this paper we investigate the possibility of improving the efficiency of data compression, and thus reducing storage requirements, for seven widely used text document formats. We propose an open-source text compression software library, featuring an advanced word-substitution scheme with static and semidynamic word dictionaries. The empirical results show an average storage space reduction as high as 78 percent compared to uncompressed documents, and as high as 30 percent compared to documents compressed with the free compression software gzip.

Download Full-text

TACIT: An open-source text analysis, crawling, and interpretation tool

Behavior Research Methods ◽

10.3758/s13428-016-0722-4 ◽

2016 ◽

Vol 49 (2) ◽

pp. 538-547 ◽

Cited By ~ 6

Author(s):

Morteza Dehghani ◽

Kate M. Johnson ◽

Justin Garten ◽

Reihane Boghrati ◽

Joe Hoover ◽

...

Keyword(s):

Open Source ◽

Text Analysis ◽

Source Text

Download Full-text

Open- Source Intelligence Gathering and Open- Analysis Intelligence for Biosecurity

Invasive Species ◽

10.1017/9781139019606.005 ◽

2017 ◽

pp. 84-92 ◽

Cited By ~ 1

Author(s):

Geoff Grossel ◽

Aidan Lyon ◽

Mike Nunn

Keyword(s):

Open Source ◽

Intelligence Gathering ◽

Open Source Intelligence

Download Full-text

Conclusion

Information Hunters ◽

10.1093/oso/9780190944612.003.0010 ◽

2019 ◽

pp. 208-213

Author(s):

Kathy Peiss

Keyword(s):

World War Ii ◽

Open Source ◽

Information Science ◽

Library And Information Science ◽

Unfinished Business ◽

Intelligence Gathering ◽

Open Source Intelligence ◽

The Holocaust ◽

Late Twentieth ◽

Late Twentieth Century

The collecting missions made an imprint on the postwar world of books and information. The OSS and military efforts to acquire open-source intelligence propelled advances in library and information science already underway. A number of those involved in wartime acquisitions became pioneers in this field. The program of acquisition offered a prototype for open-source intelligence gathering after the war. These missions also contributed to a growing orientation among American libraries toward internationalism, in which collecting foreign holdings was deemed essential to American power. For the most part, however, the collections themselves attracted little notice. With the Holocaust awareness of the late twentieth century, the acquisition of looted Jewish books was investigated by the Justice Department and President Bill Clinton’s Commission on Holocaust Assets. Looted and displaced books remain part of the unfinished business of World War II.

Download Full-text

TACIT: An Open-Source Text Analysis, Crawling and Interpretation Tool

SSRN Electronic Journal ◽

10.2139/ssrn.2660651 ◽

2015 ◽

Author(s):

Morteza Dehghani ◽

Kate M. Johnson ◽

Justin Garten ◽

Vijayan Balasubramanian ◽

Anurag Singh ◽

...

Keyword(s):

Open Source ◽

Text Analysis ◽

Source Text

Download Full-text

Text Retrieval for XML-Encoded Corpora: A Lexical Approach

Proceedings of Balisage: The Markup Conference 2008 ◽

10.4242/balisagevol1.quin01 ◽

2009 ◽

Cited By ~ 1

Author(s):

Liam R. E. Quin

Keyword(s):

World Wide Web ◽

Open Source ◽

Full Text ◽

World Wide ◽

Text Retrieval ◽

Source Text ◽

Xml Documents ◽

Text Corpora ◽

Lexical Approach

This paper describes some modifications done to an open source text retrieval package to make it XML-aware, and contrasts this lexical approach, in which XML documents are primarily treated as sequences of characters rather than trees, with the W3C XPath 1.0 and XQuery 2.0 Full-Text facility. Specific usage scenarios are taken into consideration, including World Wide Web publication and the searching and analysis of text corpora for research purposes.

Download Full-text