The Long and Short of It: Proportion-Based Relevance to Capture Document Semantics End-to-End | by Anthony Alcaraz | Nov, 2023

Dominant search methods today typically rely on keywords matching or vector space similarity to estimate relevance between a query and documents. However, these techniques struggle when it comes to searching corpora using entire files, papers or even books as search queries.

Keyword-based Retrieval

While keywords searches excel for short look up, they fail to capture semantics critical for long-form content. A document correctly discussing “cloud platforms” may be completely missed by a query seeking expertise in “AWS”. Exact term matches face vocabulary mismatch issues frequently in lengthy texts.

Vector Similarity Search

Modern vector embedding models like BERT condensed meaning into hundreds of numerical dimensions accurately estimating semantic similarity. However, transformer architectures with self-attention don’t scale beyond 512–1024 tokens due to exploding computation.

Without the capacity to fully ingest documents, the resulting “bag-of-words” partial embeddings lose the nuances of meaning interspersed across sections. The context gets lost in abstraction.

The prohibitive compute complexity also restricts fine-tuning on most real-world corpora limiting accuracy. Unsupervised learning provides one alternative but solid techniques are lacking.

In a recent paper, researchers address exactly these pitfalls by re-imagining relevance for ultra-long queries and documents. Their innovations unlock new potential for AI document search.

Dominant search paradigms today are ineffective for queries that run into thousands of words as input text. Key issues faced include:

• Transformers like BERT have quadratic self-attention complexity, making them infeasible for sequences beyond 512–1024 tokens. Their sparse attention alternatives compromise on accuracy.
• Lexical models matching based on exact term overlaps cannot infer semantic similarity critical for long-form text.
• Lack of labelled training data for most domain collections necessitates…