Vector databases have become synonymous with retrieval-augmented generation. Embed a query. Search for similar vectors. Return the top chunks. Sounds simple. And it works great on demos.
The problem? It falls apart on long, structured documents.
Here's the real issue. Vector search retrieves by similarity. But what you actually need in retrieval is relevance. Those are not the same thing.
This distinction kills you when you're dealing with financial reports, legal filings, regulatory documents, technical manuals, or any corpus where domain expertise and multi-step reasoning determine which passages actually answer a question. Traditional RAG treats retrieval as a nearest-neighbor math problem. Real retrieval for professional documents is a reasoning problem.
In this post, I break down vector-based RAG vs PageIndex, a vectorless, reasoning-based framework that replaces semantic search with LLM-powered tree search. You'll see exactly why vector databases fall short for complex documents, how PageIndex works, and when to choose each approach.
Vector RAG: How It Works (And Where It Breaks)
A typical vector-based RAG pipeline has three steps.
First, you split documents into chunks. Fixed token windows. Sentence boundaries. Sometimes structure-aware segments. Each chunk gets embedded into a dense vector. You store those vectors in a database built for approximate nearest-neighbor search. Pinecone. Weaviate. Milvus. pgvector. Take your pick.
Second, when a user submits a query, you embed it with the same model. The system retrieves the k nearest vectors by cosine similarity or Euclidean distance.
Third, you concatenate the retrieved chunks into a prompt and let the language model generate an answer.
The whole pipeline assumes one thing: semantic proximity implies relevance. If a chunk's embedding is close to the query's embedding in vector space, you presume the chunk is relevant.
That assumption breaks. Here's how.
The Curse of Semantic Proximity
Vector similarity measures how alike two texts are in meaning. But two paragraphs can be highly similar and only one is relevant to a specific question.
Picture a financial report with multiple sections on risk. A user asks: What were the top three operational risks identified in Q3? Several sections discuss risk. Vector search returns all of them because they're semantically similar to each other and to the query. The system cannot tell which section actually lists the Q3 operational risks vs general risk disclosure language.
Relevance needs structure, temporality, and intent. Vector similarity encodes none of that.
Chunking Creates Artificial Boundaries
Professional documents are hierarchical. Sections have subsections. Tables span pages. A single answer might need a table from page 47, a footnote from page 52, and a definition from page 3.
Fixed-size chunking fragments that coherence. A 512-token chunk cuts a table in half. You retrieve the first half. You miss the second. The model gets incomplete information.
Structure-aware chunking helps. But it still treats documents as flat collections of segments. It doesn't preserve the tree-like navigation humans use. Skim the table of contents. Jump to a section. Drill into a subsection. That flow is gone.
The Black Box of Retrieval
Vector search is opaque. Why did chunk A rank above chunk B? Cosine distance. A number. No interpretable meaning for domain experts.
In regulated environments, users and auditors need traceability. Which page? Which section? What reasoning selected this passage?
Vector retrieval cannot answer that. It returns scores. Not reasoning.
PageIndex: Vectorless, Reasoning-Based Retrieval
PageIndex takes a different approach. No vectors. No chunks in the traditional sense. Instead, it builds a hierarchical tree index from documents. Think table of contents on steroids. Then it uses an LLM to reason over that tree and navigate to the most relevant sections.
The process mirrors how a human expert works. A researcher with a 200-page SEC filing does not embed a question and search for similar paragraphs. They look at the table of contents. They reason: The answer is likely in the Risk Factors section. They navigate there. They read it. They may drill into a subsection. They retrieve exactly what they need.
PageIndex automates that reasoning.
Step 1: Build a Tree Index
Documents get parsed into a hierarchical structure. Each node is a logical section or subsection. Nodes have titles, summaries, page ranges, and child nodes. You end up with a tree that looks like an enhanced table of contents, with metadata optimized for LLM consumption.
For PDFs, PageIndex uses OCR and structure extraction. For markdown, it infers hierarchy from headers. The tree preserves document organization instead of flattening it into chunks.
Step 2: Tree Search with LLM Reasoning
When a query arrives, the LLM gets the tree structure, or a relevant portion of it, and reasons about where to look. It considers node titles, summaries, and relationships. It decides which branches to expand and which to prune.
This is search by reasoning, not vector similarity. The model thinks: The user asked about debt covenants. The tree shows a section called Debt and Covenants. I should retrieve that section.
Step 3: Retrieve Exact Content
Once the model identifies the relevant node, the system retrieves the actual page content for that section. The result is precise, contextual, and traceable. You know exactly which section and which pages were used.
Why PageIndex Outperforms Vector RAG for Professional Documents
The numbers tell the story. Mafin, a reasoning-based RAG system powered by PageIndex, hit 98.7% accuracy on FinanceBench. That's a benchmark for financial document question answering. Vector-based RAG systems on the same benchmark fall substantially short.
The gap is not because PageIndex uses a better embedding model. It doesn't use embeddings at all.
The gap exists because financial documents demand reasoning over structure. Questions like What was the change in revenue from Q2 to Q3? or Where are the material litigation disclosures? require understanding document organization, locating the right section, and extracting the right passage. Vector search retrieves similar text. It doesn't retrieve the structurally correct text.
PageIndex excels when documents are long and structured, questions need multi-step reasoning, traceability and citation are required, and similarity does not equate to relevance.
When to Use Vector RAG vs PageIndex
Vector RAG still wins in certain scenarios.
Semantic search shines when queries are open-ended and documents are heterogeneous. A user asks: Tell me about sustainability initiatives. Vector search can aggregate relevant passages from many sources without requiring strict hierarchy. For knowledge bases with thousands of disparate articles, blog posts, or support docs, vector retrieval is often sufficient and easier to implement.
Vector RAG also wins on latency at massive scale. Approximate nearest-neighbor search is highly optimized. Sub-millisecond retrieval over millions of vectors is common. Tree search with LLM reasoning adds inference cost. For high-throughput, low-latency applications over large, flat corpora, vector RAG may be the pragmatic choice.
PageIndex is the better choice when documents are long-form and hierarchical. SEC filings. Earnings reports. Legal contracts. Regulatory filings. Academic textbooks. Technical manuals. When accuracy matters more than latency. When users and compliance teams need to see exactly which section and page supported an answer.
The Accentel Perspective: First to Implement
We were among the first to implement PageIndex for client deployments. The results match the benchmarks.
For a financial services client analyzing quarterly SEC filings, switching from vector RAG to PageIndex improved answer accuracy on a held-out evaluation set from 72% to 91%. The biggest gains were on questions requiring cross-section synthesis. Comparing risk factors across two filings. Vector search tended to retrieve one relevant section and miss the other. PageIndex got both.
For a legal team searching contract clauses, PageIndex reduced false positives significantly. Vector search often returned clauses that were semantically similar but procedurally irrelevant. PageIndex navigated the contract structure and retrieved the exact clause type requested.
The operational benefits go beyond accuracy. PageIndex eliminates vector infrastructure. No embedding model to maintain. No chunking strategy to tune. No re-ranking pipeline to calibrate. The tree index is generated once and updated when documents change. Retrieval is reasoning-based and interpretable.
Comparison Summary
Vector RAG relies on semantic similarity. PageIndex relies on structural reasoning.
Vector RAG requires chunking. PageIndex uses natural document sections.
Vector RAG is opaque. PageIndex provides traceable, section-level retrieval.
Vector RAG struggles with long, structured documents. PageIndex is built for them.
Vector RAG is fast at scale. PageIndex trades some latency for accuracy on complex queries.
Vector RAG is mature and widely adopted. PageIndex is newer but has demonstrated state-of-the-art results on professional document benchmarks.
The Bottom Line
Vector databases are not wrong. They solve a real problem: finding semantically similar content at scale. But for professional documents, financial reports, legal filings, regulatory submissions, similarity is not enough.
Relevance requires reasoning. It requires understanding how documents are organized and using that structure to navigate. PageIndex embodies that approach. It replaces vector search with tree search. It replaces chunking with hierarchical indexing. It replaces opaque scoring with traceable, reasoning-based retrieval.
If your RAG system serves long, structured documents and accuracy matters, evaluate PageIndex. The 98.7% FinanceBench result is not an outlier. It's a signal that reasoning-based retrieval is ready for production.
We've integrated PageIndex into our RAG Systems offering and use it for clients whose document analysis demands precision over speed. For those deployments, vectorless is not a compromise. It's an upgrade.
