L2AP: Fast Cosine Similarity Search With Prefix L-2 Norm Bounds
| David C. Anastasiu and George Karypis |
| 30th IEEE International Conference on Data Engineering (ICDE), 2014 |
Download Paper  |
| Abstract The All-Pairs similarity search, or self-similarity join problem, finds all pairs of vectors in a high dimensional sparse dataset with a similarity value higher than a given threshold. The problem has been classically solved using a dynamically built inverted index. The search time is reduced by early pruning of candidates using size and value-based bounds on the similarity. In the context of cosine similarity and weighted vectors, leveraging the Cauchy-Schwarz inequality, we propose new l2-norm bounds for reducing the inverted index size, candidate pool size, and the number of full dot-product computations. We tighten previous candidate generation and verification bounds and introduce sev- eral new ones to further improve our algorithm’s performance. Our new pruning strategies enable significant speedups over baseline approaches, most times outperforming even approximate solutions. We perform an extensive evaluation of our algorithm, L2AP, and compare against state-of-the-art exact and approximate methods, AllPairs, MMJoin, and BayesLSH, across a variety of real-world datasets and similarity thresholds. |
| Comments L2AP's code is available here. |
| Research topics: Data mining | L2AP |