Hybrid Search for RAG: Semantic Chunking, BM25 and Vector Fusion

Most RAG quality problems are retrieval problems. The model reasons perfectly well over the wrong passage. Two decisions control what gets retrieved: how you cut documents into chunks, and how you rank those chunks for a query. This article covers both in depth. It deliberately stays below the level of the whole agent; for how retrieval fits alongside orchestration, tools and guardrails, see building autonomous AI agents with Next.js and vector databases.

Why one retriever is not enough

Dense embeddings and BM25 fail in opposite directions, which is exactly why combining them works.

Technical users search with identifiers far more often than people expect. They paste stack traces and type exact config keys. A pure vector index returns something “about” the error, which is often worse than returning nothing.

Chunking: structure first, semantics second

A chunk is the unit of retrieval. If a chunk does not make sense on its own, no ranking algorithm will rescue it. Fixed-size windows (for example, 512 tokens with overlap) are a reasonable baseline, but they cut tables in half and separate headings from the paragraphs they describe.

What “semantic chunking” actually means

Semantic chunking embeds each sentence and starts a new chunk wherever the similarity between neighbouring sentences drops sharply. The idea is to break where the topic changes rather than at an arbitrary token count.

The evidence for doing it everywhere is weak. Qu, Tu and Bao (2024) compared semantic and fixed-size chunking across retrieval tasks and concluded that the computational cost of semantic chunking “is not justified by consistent performance gains”. That matches the practical experience behind most guidance: structure does more of the work than similarity does.

The order I recommend is:

1. Split on structure. Headings, list items, table rows, code blocks, FAQ pairs. Your source format (Markdown, HTML, CMS blocks) usually tells you where these are.
2. Split long sections semantically. Only sections over your size budget need further splitting, and that is where embedding breakpoints beat cutting mid-sentence.
3. Prefix every chunk with its heading path. “Billing > Refunds > Partial refunds” turns an orphaned paragraph into something both retrievers can match.

Anthropic’s Contextual Retrieval write-up takes step 3 further. It uses a model to generate 50–100 tokens of chunk-specific context, then indexes the enriched chunk in both the embedding index and BM25. On their benchmark, the top-20 retrieval failure rate fell by 49% with contextual embeddings plus contextual BM25, and by 67% once reranking was added. Those are their numbers on their datasets. Treat them as a reason to test the technique, not as a guarantee for your corpus.

A minimal semantic splitter for long sections

This Python sketch assumes you have already split by heading. embed is any function that returns one vector per input string, from whichever embedding model you use.

import re
import numpy as np

def split_sentences(text: str) -> list[str]:
    # Naive splitter: swap in a proper sentence segmenter for real corpora.
    return [s.strip() for s in re.split(r"(?<=[.!?])\s+", text) if s.strip()]

def semantic_chunks(heading_path: str, text: str, embed,
                    percentile: float = 10, max_chars: int = 2000) -> list[str]:
    sentences = split_sentences(text)
    if len(sentences) < 3:
        return [f"{heading_path}\n\n{text}"]

    vecs = np.asarray(embed(sentences), dtype=float)
    vecs /= np.linalg.norm(vecs, axis=1, keepdims=True)
    sims = np.sum(vecs[:-1] * vecs[1:], axis=1)  # sentence i vs sentence i+1
    cutoff = np.percentile(sims, percentile)     # lowest-similarity gaps become breaks

    chunks, current = [], [sentences[0]]
    for sentence, sim in zip(sentences[1:], sims):
        too_long = sum(len(s) for s in current) + len(sentence) > max_chars
        if sim <= cutoff or too_long:
            chunks.append(current)
            current = []
        current.append(sentence)
    chunks.append(current)

    # The heading path travels with every chunk, for both BM25 and embeddings.
    return [f"{heading_path}\n\n{' '.join(c)}" for c in chunks]

The percentile and the size cap are tuning knobs, not constants. Set them by measuring retrieval on your own questions (see the evaluation section below).

Fusing keyword and vector results

BM25 scores are unbounded and depend on the corpus. Cosine or dot-product scores live on a different scale. Adding them together is not meaningful unless you normalise first. There are two common ways to handle this.

Reciprocal rank fusion (the default)

RRF ignores raw scores and uses only rank positions. Each document scores the sum of 1 / (k + rank) across every list it appears in. The constant k is conventionally 60, the value used in the original paper by Cormack, Clarke and Buettcher and the default in Pinecone’s RRF guide. A larger k flattens the advantage of the top positions.

def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    """Fuse ranked lists of document IDs (best first) into one ranking."""
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda item: item[1], reverse=True)

fused = reciprocal_rank_fusion([bm25_ids, dense_ids])[:50]

RRF needs no training data and does not break when you swap an embedding model. That robustness is why it is the right first choice.

Weighted score fusion (alpha)

The alternative blends normalised scores: alpha * dense + (1 - alpha) * sparse. It can beat RRF when you have labelled relevance data to tune alpha, but there is no universal best value. Pinecone’s own guidance suggests trying 0.75, 0.5 and 0.25 against your data. Without an evaluation set, you are guessing.

Hybrid search in Pinecone today

The URL of this article mentions Pinecone, so here is where its hybrid support stands as of October 2026, based on Pinecone’s hybrid search documentation. There are now three patterns:

Pinecone recommends the Documents API for new text workloads. A full-text query on that API looks like this (Python SDK v10 or later):

text_response = index.documents.search(
    namespace="docs",
    top_k=50,
    score_by=[{"type": "text", "fields": ["body"], "query": user_query}],
    include_fields=["title", "body", "heading_path"],
)
# Run the dense search the same way with a "dense_vector" score_by entry,
# collect each response's _id values in rank order, then fuse with RRF.

One detail catches people out: multi-word text queries match documents containing any of the terms. For exact phrases, use a $match_phrase filter or a query_string query. The same architecture works on any engine with BM25 and vectors, such as OpenSearch, Elasticsearch, Qdrant, Weaviate or Postgres with pgvector and full-text search. RRF is a few lines of client code wherever you run it.

Rerank after you fuse

Fusion decides which candidates survive; a cross-encoder reranker decides their final order. A reranker scores the query and passage together, which neither BM25 nor a bi-encoder embedding does. The usual shape is:

1. Retrieve 50–150 candidates from each retriever.
2. Fuse with RRF and keep the top 50 or so.
3. Rerank and pass the top 5–20 to the model.

Reranking adds latency and per-query cost, so tune the candidate count against your latency budget rather than copying someone else’s.

Measure before you tune anything

Every knob above (chunk size, breakpoint percentile, k, alpha, candidate counts) should be set against a fixed set of real questions with known relevant passages. Retrieval metrics such as context precision and recall tell you whether a change helped. I cover that loop in evaluating production RAG pipelines with RAGAS. Without it, a chunking change that “feels better” on three test questions can quietly make fifty others worse.

Implementation checklist

• Chunk on structure first; semantic-split only sections over your size budget.
• Prefix each chunk with its heading path (or generated context), and index that same text in both BM25 and the vector store.
• Store a stable chunk ID shared by both indexes so fusion can match results.
• Start with RRF at k = 60; move to weighted fusion only once you have labelled data to tune alpha.
• Rerank the fused candidates before they reach the model.
• Re-run the retrieval evaluation whenever you change the embedding model, chunker or fusion settings.

Questions people actually ask

Is semantic chunking worth the extra cost?

Usually not as a blanket strategy. Published comparisons show inconsistent gains over fixed-size chunking. Structure-aware splitting plus heading prefixes is cheaper and more predictable. Use embedding breakpoints for long, unstructured sections, and confirm the benefit on your own evaluation set.

Should I use RRF or a weighted alpha blend?

Start with RRF. It needs no tuning and is robust to differences in score scale. A weighted blend can do better once you have labelled queries to tune alpha against, but a badly chosen alpha does worse than RRF.

Does Pinecone support BM25?

Yes. Full-text search on Pinecone’s Documents API uses BM25 scoring. On the older Vectors API, keyword signal comes from sparse vectors, for example from the hosted pinecone-sparse-english-v0 model, stored alongside dense vectors in a dotproduct index or in a separate sparse index.