Bregman Hyperplane Trees for Fast Approximate Nearest Neighbor Search

2012 ◽  
Vol 3 (4) ◽  
pp. 75-87
Author(s):  
Bilegsaikhan Naidan ◽  
Magnus Lie Hetland
Keyword(s):  
Query Processing ◽  
Nearest Neighbor ◽  
Nearest Neighbor Search ◽  
Index Structure ◽  
Data Sets ◽  
Search Performance ◽  
Space Partitioning ◽  
Neighbor Search ◽  
Order Of Magnitude ◽  
Exact Nearest Neighbor

This article presents a new approximate index structure, the Bregman hyperplane tree, for indexing the Bregman divergence, aiming to decrease the number of distance computations required at query processing time, by sacrificing some accuracy in the result. The experimental results on various high-dimensional data sets demonstrate that the proposed index structure performs comparably to the state-of-the-art Bregman ball tree in terms of search performance and result quality. Moreover, this method results in a speedup of well over an order of magnitude for index construction. The authors also apply their space partitioning principle to the Bregman ball tree and obtain a new index structure for exact nearest neighbor search that is faster to build and a slightly slower at query processing than the original.

scholarly journals The Chemfp Project

2019 ◽  
Author(s):  
Andrew Dalke
Keyword(s):  
Similarity Search ◽  
High Performance ◽  
Nearest Neighbor ◽  
Main Memory ◽  
Nearest Neighbor Search ◽  
Search Performance ◽  
Command Line ◽  
Data Set ◽  
Neighbor Search ◽  
Order Of Magnitude

<div>This paper describes the 10 years of work and research results of the chemfp project, available from http://chemfp.com/ . The project started as a way to promote the FPS format for cheminformatics fingerprint exchange. This is a line-oriented text format meant to be easy to read and write. It supports metadata such as the fingerprint type and data provenance.The chemfp package for Python was developed to provide the basic command-line tools and Python API for working with fingerprint data, because a format without useful tools will not be used. The similarity search performance improved by an order of magnitude over the decade, due to careful implementation and effective use of CPU hardware, including AVX2 support for faster popcount calculations than the built-in POPCNT instruction. The implementation details for high-performance search have rarely been discussed in the literature. As a result, many tools and published papers use implementations which are not close to the machine's capabilities. This paper describes those details to help with future optimization efforts. The most advanced version of chemfp evaluates about 130 million 1024-bit fingerprint Tanimotos per second on a single core of a standard x86-64 server machine. When combined with the BitBound algorithm, a k=1000 nearest-neighbor search of the 1.8 million 2048-bit Morgan fingerprints of ChEMBL 24 averages 27 ms/query and the same search of the 970 million PubChem fingerprints averages 220 ms/query, making chemfp one of the fastest similarity search tools available for CPUs. This appears to be several times faster than previously published work in the field, including in papers which use much more sophisticated data structures. A close analysis shows that nearly all earlier work assumes that the intersection popcount was the limiting performance factor, while on modern hardware uncompressed search is effectively memory bandwidth limited. For example, AVX2 search is 10% faster when memory prefetching, and the popcount evaluation time is far faster than fetching a random location in main memory. It proved difficult to evaluate existing tool performance because in the few cases where the tools were available, each used its own format, data sets, and search tasks. This paper introduces the chemfp benchmark data set to help make head-to-head comparisons easier in the future, and to help promote the FPS format. The FPS format is slow for tasks like web server reloads and command-line scripting. This paper also describes the FPB format, which is a binary application format for fast loads. </div>

scholarly journals The chemfp Project

2019 ◽  
Author(s):  
Andrew Dalke
Keyword(s):  
Similarity Search ◽  
High Performance ◽  
Nearest Neighbor ◽  
Main Memory ◽  
Nearest Neighbor Search ◽  
Search Performance ◽  
Command Line ◽  
Data Set ◽  
Neighbor Search ◽  
Order Of Magnitude

<div>This paper describes the 10 years of work and research results of the chemfp project, available from http://chemfp.com/ . The project started as a way to promote the FPS format for cheminformatics fingerprint exchange. This is a line-oriented text format meant to be easy to read and write. It supports metadata such as the fingerprint type and data provenance.The chemfp package for Python was developed to provide the basic command-line tools and Python API for working with fingerprint data, because a format without useful tools will not be used. <br></div><div><br></div><div>The similarity search performance improved by an order of magnitude over the decade, due to careful implementation and effective use of CPU hardware, including AVX2 support for faster popcount calculations than the built-in POPCNT instruction. The implementation details for high-performance search have rarely been discussed in the literature. As a result, many tools and published papers use implementations which are not close to the machine's capabilities.</div><div><br></div><div>This paper describes those details to help with future optimization efforts.</div><div><br></div><div>The most advanced version of chemfp evaluates about 130 million 1024-bit fingerprint Tanimotos per second on a single core of a standard x86-64 server machine. When combined with the BitBound algorithm, a k=1000 nearest-neighbor search of the 1.8 million 2048-bit Morgan fingerprints of ChEMBL 24 averages 27 ms/query and the same search of the 970 million PubChem fingerprints averages 220 ms/query, making chemfp one of the fastest similarity search tools available for CPUs. This appears to be several times faster than previously published work in the field, including in papers which use much more sophisticated data structures.</div><div><br></div><div>A close analysis shows that nearly all earlier work assumes that the intersection popcount was the limiting performance factor, while on modern hardware uncompressed search is effectively memory bandwidth limited. For example, AVX2 search is 10% faster when memory prefetching, and the popcount evaluation time is far faster than fetching a random location in main memory. It proved difficult to evaluate existing tool performance because in the few cases where the tools were available, each used its own format, data sets, and search tasks.</div><div><br></div><div>This paper introduces the chemfp benchmark data set to help make head-to-head comparisons easier in the future, and to help promote the FPS format. The FPS format is slow for tasks like web server reloads and command-line scripting. This paper also describes the FPB format, which is a binary application format for fast loads. </div>

scholarly journals The chemfp Project

2019 ◽  
Author(s):  
Andrew Dalke
Keyword(s):  
Similarity Search ◽  
High Performance ◽  
Nearest Neighbor ◽  
Main Memory ◽  
Nearest Neighbor Search ◽  
Search Performance ◽  
Command Line ◽  
Data Set ◽  
Neighbor Search ◽  
Order Of Magnitude

<div>This paper describes the 10 years of work and research results of the chemfp project, available from http://chemfp.com/ . The project started as a way to promote the FPS format for cheminformatics fingerprint exchange. This is a line-oriented text format meant to be easy to read and write. It supports metadata such as the fingerprint type and data provenance.The chemfp package for Python was developed to provide the basic command-line tools and Python API for working with fingerprint data, because a format without useful tools will not be used. <br></div><div><br></div><div>The similarity search performance improved by an order of magnitude over the decade, due to careful implementation and effective use of CPU hardware, including AVX2 support for faster popcount calculations than the built-in POPCNT instruction. The implementation details for high-performance search have rarely been discussed in the literature. As a result, many tools and published papers use implementations which are not close to the machine's capabilities.</div><div><br></div><div>This paper describes those details to help with future optimization efforts.</div><div><br></div><div>The most advanced version of chemfp evaluates about 130 million 1024-bit fingerprint Tanimotos per second on a single core of a standard x86-64 server machine. When combined with the BitBound algorithm, a k=1000 nearest-neighbor search of the 1.8 million 2048-bit Morgan fingerprints of ChEMBL 24 averages 27 ms/query and the same search of the 970 million PubChem fingerprints averages 220 ms/query, making chemfp one of the fastest similarity search tools available for CPUs. This appears to be several times faster than previously published work in the field, including in papers which use much more sophisticated data structures.</div><div><br></div><div>A close analysis shows that nearly all earlier work assumes that the intersection popcount was the limiting performance factor, while on modern hardware uncompressed search is effectively memory bandwidth limited. For example, AVX2 search is 10% faster when memory prefetching, and the popcount evaluation time is far faster than fetching a random location in main memory. It proved difficult to evaluate existing tool performance because in the few cases where the tools were available, each used its own format, data sets, and search tasks.</div><div><br></div><div>This paper introduces the chemfp benchmark data set to help make head-to-head comparisons easier in the future, and to help promote the FPS format. The FPS format is slow for tasks like web server reloads and command-line scripting. This paper also describes the FPB format, which is a binary application format for fast loads. </div>

scholarly journals A Hybrid Spatial Indexing Structure of Massive Point Cloud Based on Octree and 3D R*-Tree

2021 ◽  
Vol 11 (20) ◽  
pp. 9581
Author(s):  
Wei Wang ◽  
Yi Zhang ◽  
Genyu Ge ◽  
Qin Jiang ◽  
Yang Wang ◽  
...  
Keyword(s):  
Point Cloud ◽  
Nearest Neighbor ◽  
Nearest Neighbor Search ◽  
Index Structure ◽  
Spatial Indexing ◽  
Cloud Data ◽  
Neighbor Search ◽  
Nearest Neighbor Searching ◽  
Indexing Structure ◽  
Balanced Tree

The spatial index structure is one of the most important research topics for organizing and managing massive 3D Point Cloud. As a point in Point Cloud consists of Cartesian coordinates (x,y,z), the common method to explore geometric information and features is nearest neighbor searching. An efficient spatial indexing structure directly affects the speed of the nearest neighbor search. Octree and kd-tree are the most used for Point Cloud data. However, Octree or KD-tree do not perform best in nearest neighbor searching. A highly balanced tree, 3D R*-tree is considered the most effective method so far. So, a hybrid spatial indexing structure is proposed based on Octree and 3D R*-tree. In this paper, we discussed how thresholds influence the performance of nearest neighbor searching and constructing the tree. Finally, an adaptive way method adopted to set thresholds. Furthermore, we obtained a better performance in tree construction and nearest neighbor searching than Octree and 3D R*-tree.

scholarly journals k-Nearest Neighbor Search based on Node Density in MANETs

2014 ◽  
Vol 10 (4) ◽  
pp. 385-405 ◽  
Author(s):  
Yuka Komai ◽  
Yuya Sasaki ◽  
Takahiro Hara ◽  
Shojiro Nishio
Keyword(s):  
Query Processing ◽  
Nearest Neighbor ◽  
Nearest Neighbor Search ◽  
K Nearest Neighbor ◽  
Node Density ◽  
Neighbor Search ◽  
Query Result ◽  
Knn Query ◽  
The One ◽  
K Nearest Neighbor Search

In a kNN query processing method, it is important to appropriately estimate the range that includes kNNs. While the range could be estimated based on the node density in the entire network, it is not always appropriate because the density of nodes in the network is not uniform. In this paper, we propose two kNN query processing methods in MANETs where the density of nodes is ununiform; the One-Hop (OH) method and the Query Log (QL) method. In the OH method, the nearest node from the point specified by the query acquires its neighbors' location and then determines the size of a circle region (the estimated kNN circle) which includes kNNs with high probability. In the QL method, a node which relays a reply of a kNN query stores the information on the query result for future queries.

scholarly journals Mobile Location Indexing Based On Synthetic Moving Objects

2019 ◽  
Vol 9 (4) ◽  
pp. 2556
Author(s):  
Thu Thu Zan ◽  
Sabai Phyu
Keyword(s):  
Performance Evaluation ◽  
Location Privacy ◽  
Moving Objects ◽  
Nearest Neighbor ◽  
Data Access ◽  
Nearest Neighbor Search ◽  
Index Structure ◽  
Mobile Objects ◽  
Neighbor Search ◽  
Privacy And Confidentiality

Today, the number of researches based on the data they move known as mobile objects indexing came out from the traditional static one. There are some indexing approaches to handle the complicated moving positions. One of the suitable ideas is pre-ordering these objects before building index structure. In this paper, a structure, a presorted-nearest index tree algorithm is proposed that allowed maintaining, updating, and range querying mobile objects within the desired period. Besides, it gives the advantage of an index structure to easy data access and fast query along with the retrieving nearest locations from a location point in the index structure. A synthetic mobile position dataset is also proposed for performance evaluation so that it is free from location privacy and confidentiality. The detail experimental results are discussed together with the performance evaluation of KDtree-based index structure. Both approaches are similarly efficient in range searching. However, the proposed approach is especially much more save time for the nearest neighbor search within a range than KD tree-based calculation.

scholarly journals Tree Index Nearest Neighbor Search of Moving Objects along a Road Network

2021 ◽  
Vol 2021 ◽  
pp. 1-18
Author(s):  
Wei Jiang ◽  
Fangliang Wei ◽  
Guanyu Li ◽  
Mei Bai ◽  
Yongqiang Ren ◽  
...  
Keyword(s):  
Road Network ◽  
Moving Objects ◽  
Nearest Neighbor ◽  
Search Algorithm ◽  
Urban Transportation ◽  
Road Networks ◽  
Nearest Neighbor Search ◽  
Index Structure ◽  
Widespread Application ◽  
Neighbor Search

With the widespread application of location-based service (LBS) technology in the urban Internet of Things, urban transportation has become a research hotspot. One key issue of urban transportation is the nearest neighbor search of moving objects along a road network. The fast-updating operations of moving objects along a road network suppress the query response time of urban services. Thus, a tree-indexed searching method is proposed to quickly find the answers to user-defined queries on frequently updating road networks. First, a novel index structure, called the double tree-hash index, is designed to reorganize the corresponding relationships of moving objects and road networks. Second, an index-enhanced search algorithm is proposed to quickly find the k -nearest neighbors of moving objects along the road network. Finally, an experiment shows that compared with state-of-the-art algorithms, our algorithm shows a significant improvement in search efficiency on frequently updating road networks.

Sign in / Sign up

Export Citation Format

Share Document