The team at Remedy Legal examined their document retrieval pipeline to understand how effectively their legal agents could surface relevant precedents from an expanding collection of legislation, case law, tribunal decisions, and housing guidance. They recognized that reliable retrieval directly impacts downstream agent performance across legal analysis, case building, and user interactions.
The baseline
The evaluation focused on 550 case law documents—challenging to search due to their density, structural inconsistency, and overlapping legal coverage. The initial approach embedded each document as a single vector and ranked results by cosine similarity.
The team measured performance through self-retrieval tests: using each document's description as a query and checking whether the original document appeared in the top 10 results. This baseline achieved a "hit rate came back at 82.4%." Missing one in five relevant documents posed problems for a system designed to surface the correct cases.
Why whole-document embeddings fall short
When searching the baseline system, identical case law documents appeared repeatedly across completely different queries. Long tribunal decisions covering distinct legal areas were treated as interchangeable. The embedding vectors became diluted—a 768-dimensional representation attempted to capture an entire document's semantic content, including procedural history, multiple legal arguments, statutory interpretation, and final rulings.
Geometrically, each topic pulls the embedding vector in different directions. A focused statutory instrument produces an embedding positioned cleanly in one region of vector space. However, a lengthy tribunal decision touching housing disrepair, procedural fairness, costs, and multiple statutes "ends up as an average of all those directions; close-ish to lots of things but not particularly close to any of them."
Chunking
The team implemented document segmentation into approximately 250-token chunks with sentence and paragraph-aware splitting, guided by research on chunk size optimization for dense retrieval. This approach ensures each embedding represents a focused concept rather than an entire document.
Each chunk receives its own embedding, and searches identify the most relevant chunk before tracing back to the parent document—a pattern inspired by parent-document retrieval approaches. The embedding now "represents one coherent idea rather than a centroid of everything."
Where embeddings stop helping
Embeddings operate within continuous vector space, excelling at semantic proximity—matching "quiet enjoyment" with "peaceful occupation of premises" or "retaliatory eviction" with "revenge eviction by landlord." However, they lack mechanisms for exact symbolic matching.
Legal text contains numerous identifiers existing outside semantic space: statute references like "section 21(1)(b)," legislation titles such as "Housing Act 2004," and case citations. These arbitrary symbols lack semantic relationships—"section 21" and "section 20" maintain equal distance in embedding space regardless of actual legal relationships.
Analysis of retrieval failures revealed that "almost every miss was a query anchored to a specific legal identifier, like a statute reference, a case citation, or a specific procedural rule." Embeddings could not distinguish between documents discussing similar themes while referencing different legislation.
Adding the keyword signal
The solution combined embeddings with BM25-style keyword ranking through hybrid search. The team selected this approach because their retrieval system handled semantic matching well but remained structurally blind to exact identifiers—precisely what keyword search addresses.
They implemented Reciprocal Rank Fusion (RRF) to combine rankings, sidesteping normalization issues between cosine similarity and BM25 scores operating on completely different scales. RRF converts each ranking to a positional score using the formula 1/(k + rank), where k equals 60. Documents ranked first by vector search receive a score of 1/61; ranked third receive 1/63. Both rankings calculate independently, and scores are summed. Documents appearing in both rankings receive natural boosts while documents found by only one system still surface through rank arithmetic.
The results
Running the same 550 self-retrieval queries against the hybrid system produced dramatically different results: "99.8%: one miss out of 550."
The improvement from 82.4% to 99.8% stemmed almost entirely from the keyword signal capturing exact queries that vector search structurally could not handle. The single remaining miss represents an edge case—a particularly unusual tribunal decision whose description lacks sufficient distinctive terms matching its content for either signal to surface confidently.
Performance metrics showed hybrid search achieving approximately 13ms P50 latency for queries themselves, with full round-trip latency of roughly 88ms P50, all running on an existing Postgres instance.
When rolled out across the complete corpus, staging results demonstrated pass@k performance reaching 99.9% across the full 31,000-document collection, with only the same edge case from testing remaining unresolved.
The team acknowledged self-retrieval testing limitations. These controlled tests use queries derived from documents themselves, creating inherent advantages. Subsequent evaluation with agent-generated queries not previously encountered by the system showed results "holding up very well."
Ongoing testing expanded to cross-register queries expecting legislative results rather than case law, multi-hop queries requiring document chains, and adversarial queries designed to challenge the ranking system. The focus remains creating legal support grounded in genuine authority rather than speculation.
