Compression and full-text indexing for Digital Libraries

Searching for contents in present digital libraries is still very primitive; most websites provide a search field where users can enter information such as book title, author name, or terms they expect to be found in the book. Some platforms provide advanced search options, which allow the users to narrow the search results by specific parameters such as year, author name, publisher, and similar. Currently, when users find a book which might be of interest to them, this search process ends; only a full-text search or references at the end of the book may provide some additional pointers. In this chapter, the author is going to give an example of how a user could permanently get recommendations for additional contents even while reading the article, using present machine learning and artificial intelligence techniques.

Download Full-text

Full-text indexing for optimizing selection operations in large-scale data analytics

Proceedings of the second international workshop on MapReduce and its applications - MapReduce '11 ◽

10.1145/1996092.1996105 ◽

2011 ◽

Cited By ~ 13

Author(s):

Jimmy Lin ◽

Dmitriy Ryaboy ◽

Kevin Weil

Keyword(s):

Full Text ◽

Data Analytics ◽

Large Scale ◽

Text Indexing ◽

Large Scale Data ◽

Scale Data

Download Full-text

Compact and hash based variants of the suffix array

Bulletin of the Polish Academy of Sciences Technical Sciences ◽

10.1515/bpasts-2017-0046 ◽

2017 ◽

Vol 65 (4) ◽

pp. 407-418

Author(s):

S. Grabowski ◽

M. Raniszewski

Keyword(s):

Data Structure ◽

Full Text ◽

Hash Table ◽

Suffix Array ◽

Fixed Size ◽

Text Indexing ◽

Text Length ◽

Speed Up ◽

Pattern Searches ◽

Search Interval

AbstractFull-text indexing aims at building a data structure over a given text capable of efficiently finding arbitrary text patterns, and possibly requiring little space. We propose two suffix array inspired full-text indexes. One, called SA-hash, augments the suffix array with a hash table to speed up pattern searches due to significantly narrowed search interval before the binary search phase. The other, called FBCSA, is a compact data structure, similar to Mäkinen’s compact suffix array (MakCSA), but working on fixed size blocks. Experiments on the widely used Pizza & Chili datasets show that SA-hash is about 2–3 times faster in pattern searches (counts) than the standard suffix array, for the price of requiring 0.2n–1.1nbytes of extra space, wherenis the text length. FBCSA, in one of the presented variants, reduces the suffix array size by a factor of about 1.5–2, while it gets close in search times, winning in speed with its competitors known from the literature, MakCSA and LCSA.

Download Full-text