Vector databases, a key technology in building retrieval augmented generation or RAG applications, has a scaling problem that few are talking about. 

According to new research by EyeLevel.ai, an AI tools company, the precision of vector similarity search degrades in as few as 10,000 pages, reaching a 12% performance hit by the 100,000-page mark.

The research also tested EyeLevel’s enterprise-grade RAG platform which does not use vectors. EyeLevel lost only 2% accuracy at scale.

The findings suggest that while vector databases have become highly popular tools to build RAG and LLM-based applications, developers may face unexpected challenges as they shift from testing to production and attempt to scale their applications.  

The work was performed by Daniel Warfield, a data scientist and RAG engineer and Dr. Benjamin Fletcher, PhD, a computer scientist and former senior engineer at IBM Watson. Both men work for EyeLevel.ai. The data, code and methods of this test will beopen sourced and available shortly. Others are invited to run the data and corroborate or challenge these findings. 

What’s Inside

In this report, we will review how the test was constructed, the detailed findings, our theories on why vector similarity search experienced challenges and suggested approaches to scale RAG without the performance hit. We also encourage you to read our prior research in which EyeLevel’s GroundX APIs bested LangChain, Pinecone and Llamaindex based RAG systems by 50-120% on accuracy over 1,000 pages of content.  

Defining RAG 

Feel free to skip this section if you’re familiar with RAG.  

RAG stands for “Retrieval Augmented Generation”. When you ask a RAG system a query, RAG does the following steps: 

1. Retrieval: Based on the query from the user, the RAG system retrieves relevant knowledge from a set of documents. 

2. Augmentation: The RAG system combines the retrieved information with the user query to construct a prompt. 

3. Generation: The augmented prompt is passed to a large language model, generating the final output. 

The implementation of these three steps can vary wildly between RAG approaches. However, the objective is the same: to make a language model more useful by feeding it information from real-world, relevant documents. 

Beyond The Tech Demo 

When most developers begin experimenting with RAG they might grab a few documents, stick them into a RAG document store and be blown away by the results. Like magic, many RAG systems can allow a language model to understand books, company documents, emails, and more. 

However, as one continues experimenting with RAG, some difficulties begin to emerge. 

1. Many documents are not purely textual. They might have images, tables, or complex formatting. While many RAG systems can parse complex documents, the quality of parsing varies widely between RAG approaches. We explore the realities of parsing in another article. 

2. As a RAG system is exposed to more documents, it has more opportunities to retrieve the wrong document, potentially causing a degradation in performance   

3. Because of technical complexity, the underlying non-determinism of language models, and the difficulty of profiling the performance of LLM applications in real world settings, it can be difficult to predict the cost and level of effort of developing RAG applications. 

In this article we’ll focus on the second and third problems listed above; performance degradation of RAG at scale and difficulties of implementation 

The Test 

To test how much larger document sets degrade the performance of RAG systems, we first defined a set of 92 questions based on real-world documents.  

We then constructed four document sets to apply RAG to. All four of these document sets contain the same 310 pages of documents which answer our 92 test questions. However, each document set also contains a different number of irrelevant pages from miscellaneous documents. We started with 1,000 pages and scaled up to 100,000 in our largest test. 
 

An ideal RAG system would, in theory, behave identically across all document sets, as all document sets contain the same answers to the same questions. In practice, however, added information in a docstore can trick a RAG system into retrieving the wrong context for a given query. The more documents there are, the more likely this is to happen. Therefore, RAG performance tends to degrade as the number of documents increases. 

In this test we applied each of these three popular RAG approaches to the four document sets mentioned above:

• LangChain: a popular python library designed to abstract certain LLM workflows. 
• LlamaIndex: a popular python library which has advanced vector embedding capability, and advanced RAG functionality. 
• EyeLevel’s GroundX: a feature complete retrieval engine built for RAG. 

By applying each of these RAG approaches to the four document sets, we can study the relative performance of each RAG approach at scale. 

For both LangChain and LlamaIndex we employed Pinecone as our vector store and OpenAI’s text-embedding-ada-002 for embedding. GroundX, being an all-in-one solution, was used in isolation up to the point of generation. All approaches used OpenAI's gpt-4-1106-preview for the final generation of results. Results for each approach were evaluated as being true or false via human evaluation. 

The Effect of Scale on RAG 

We ran the test as defined in the previous section and got the following results. 

As can be seen in the figure above, the rate at which RAG degrades in performance varies widely between RAG approaches. Based on these results one might expect GroundX to degrade in performance by 2% per 100,000 documents, while LCPC and LI might degrade 10-12% per 100,000 documents. The reason for this difference in robustness to larger document sets, likely, has to do with the realities of using vector search as the bedrock of a RAG system.  

In theory a high dimensional vector space can hold a vast amount of information. 100,000 in binary is 17 values long (11000011010100000). So, if we only use binary vectors with unit components in a high dimensional vector space, we could store each page in our 100,000 page set with only a 17 dimensional space. Text-embedding-ada-002, which is the encoder used in this experiment, outputs a 1536-dimension vector. If one calculates 2^1536 (effectively calculating how many things one could describe using only binary vectors in this space) the result would be a number that’s significantly greater than the number of atoms in the known universe. Of course, actual embeddings are not restricted to binary numbers; they can be expressed in decimal numbers of very high precision. Even relatively small vector spaces can hold a vast amount of information. 

The trick is, how do you get information into a vector space meaningfully? RAG needs content to be placed in a vector space such that similar things can be searched, thus the encoder has to practically organize information into useful regions. It’s our theory that modern encoders don’t have what it takes to organize large sets of documents in these vector spaces, even if the vector spaces can theoretically fit a near infinite amount of information. The encoder can only put so much information into a vector space before the vector space gets so cluttered that distance-based search is rendered non-performant. 

EyeLevel’s GroundX doesn’t use vector similarity as its core search strategy, but rather a tuned comparison based on the similarity of semantic objects. There are no vectors used in this approach. This is likely why GroundX exhibits superior performance in larger document sets. 

In this test we employed what is commonly referred to as “naive” RAG. LlamaIndex and LangChain allow for many advanced RAG approaches, but they had little impact on performance and were harder to employ at larger scales. We cover that in another article which will be released shortly.

The Surprising Technical Difficulty of Scale 

While 100,000 pages seems like a lot, it’s actually a fairly small amount of information for industries like engineering, law, and healthcare. Initially we imagined testing on much larger document sets, but while conducting this test we were surprised by the practical difficulty of getting LangChain to work at scale; forcing us to reduce the scope of our test. 

To get RAG up and running for a set of PDF documents, the first step is to parse the content of those PDFs into some sort of textual representation. LangChain uses libraries from Unstructured.io to perform parsing on complex PDFs, which works seamlessly for small document sets. 

Surprisingly, though, the speed of LangChain parsing is incredibly slow. Based on our analysis it appears that Unstructured uses a variety of models to detect and parse out key elements within a PDF. These models should employ GPU acceleration, but they don’t. That results in LangChain taking days to parse a modestly sized set of documents, even on very large (and expensive) compute instances. To get LangChain working we needed to reverse engineer portions of Unstructured and inject code to enable GPU utilization of these models. 

It appears that this is a known issue in Unstructured, as seen in the notes below. As it stands, it presents significant difficulty in scaling LangChain to larger document sets, given LangChain abstracts away fine grain control of Unstructured. 

Source: Github

We only made improvements to LangChain parsing up to the point where this test became feasible. If you want to modify LangChain for faster parsing, here are some resources: 

• The default directory loader of LangChain is Unstructured (source1, source2). 
• Unstructured uses “hi res” for the PDFs by default if text extraction cannot be performed on the document (source1 , source2 ). Other options are available like “fast” and “OCR only”, which have different processing intensities 
• “Hi Res” involves: 
  • Converting the pdf into images (source) 
  • Running a layout detection model to understand the layout of the documents (source). This model benefits greatly from GPU utilization, but does not leverage the GPU unless ONNX is installed (source) 
  • OCR extraction using tesseract (by default) (source) which is a very compute intensive process (source) 
  • Running the page through a table layout model (source) 

While our configuration efforts resulted in faster processing times, it was still too slow to be feasible for larger document sets. To reduce time, we did “hi res” parsing on the relevant documents and “fast” parsing on documents which were irrelevant to our questions. With this configuration, parsing 100,000 pages of documents took 8 hours. If we had applied “hi res” to all documents, we imagine that parsing would have taken 31 days (at around 30 seconds per page). 

At the end of the day, this test took two senior engineers (one who’s worked at a directorial level at several AI companies, and a multi company CTO with decades of applied experience of AI at scale) several weeks to do the development necessary to write this article, largely because of the difficulty of applying LangChain to a modestly sized document set.  To get LangChain working in a production setting, we estimate that the following efforts would be required: 

• Tesseract would need to be interfaced with in a way that is more compute and time efficient. This would likely require a high-performance CPU instance, and modifications to the LangChain source code. 
• The layout and table models would need to be made to run on a GPU instance 
• To do both tasks in a cost-efficient manner, these tasks should probably be decoupled. However, this is not possible with the current abstraction of LangChain. 

On top of using a unique technology which is highly performant, GroundX also abstracts virtually all of these technical difficulties behind an API. You upload your documents, then search the results. That’s it. 

If you want RAG to be even easier, one of the things that makes Eyelevel so compelling is the service aspect they provide to GroundX. You can work with Eyelevel as a partner to get GroundX working quickly and performantly for large scale applications. 

Conclusion 

When choosing a platform to build RAG applications, engineers must balance a variety of key metrics. The robustness of a system to maintain performance at scale is one of those critical metrics. In this head-to-head test on real-world documents, EyeLevel’s GroundX exhibited a heightened level of performance at scale, beating LangChain and LlamaIndex. 

Another key metric is efficiency at scale. As it turns out, LangChain has significant implementation difficulties which can make the large-scale distribution of LangChain powered RAG difficult and costly. 

Is this the last word? Certainly not. In future research, we will test various advanced RAG techniques, additional RAG frameworks such as Amazon Q and GPTs and increasingly complex and multimodal data types. So stay tuned. 

If you’re curious about running these results yourself, please reach out to us at info@eyelevel.ai.