On the latest episode of RAG Masters, CEO of Unify AI Dan Lenton explored the technology behind LLM Routing along with hosts Neil Katz and Daniel Warfield.
Understanding LLM Routing: The Core Concept
At its essence, LLM routing is about sending prompts to the most suitable model based on a combination of factors. Dan Lenton, CEO of Unify AI, breaks it down: "Basically what routing means is that the prompt comes in and we send the prompt to the most appropriate model. By balancing user preferences for quality, speed, and cost is what the routing does."
This isn't a simple if-then decision tree. It's powered by a sophisticated neural scoring function that predicts which model will perform best for a given prompt. Lenton elaborates: "We predict the quality that each model will have on the prompt without having to actually query the LLM. This is with a much smaller LLM that predicts how good would GPT-4 be with this prompt? How good would Llama be at this prompt? How good would Mistral be?"
The router is trained using a BERT-like encoder model and an LLM, which acts as a judge to evaluate the quality of responses from various models. This process generates a labeled training dataset that enables supervised training of the BERT model to predict the performance of each model for a given prompt.
The Power of Optimization: Why LLM Routing Matters
There are many benefits of LLM Routing, with optimization at the forefront. Not every task requires the computational muscle of a top-tier model like GPT-4. Lenton points out: "For many questions that are simple, you don't need such a powerful model. So particularly when you're scaling out your application and you start to care about the speed and the cost margins, then a lot of companies are wasting a lot of money and seeing very slow responses on these large models, which are overkill for a lot of the easy prompts."
By routing simpler queries to more lightweight models, companies can slash costs and boost response times without sacrificing quality. This is crucial for businesses operating at scale, where even minor inefficiencies can snowball into significant expenses.
Lenton explains some of the common pitfalls that developers run into when trying to build and optimize these types of large-scale prompts without utilizing LLM Routing in the below clip:
LLM Routing in Retrieval Augmented Generation (RAG) Systems
While LLM routing shines in straightforward AI applications, its true potential emerges in more complex systems, particularly with Retrieval-Augmented Generation (RAG). Lenton notes: "The biggest use case of routing and RAG comes more when you have an agentic hierarchical RAG system and there are different LLMs doing different subtasks and superset tasks."
In these advanced RAG systems, different components of the process can leverage different models. A simple task like pulling an email address from a thread might use a smaller, faster model, while complex reasoning could tap into a more powerful LLM. This granular approach allows developers to build highly efficient, responsive systems that allocate resources intelligently based on the specific demands of each subtask.
The Future of LLM Routing: System Prompt Routing
As impressive as current LLM routing capabilities are, the technology is still evolving. On the horizon is what Lenton calls system prompt routing: "What we are seeing, and what we're now working on is a router that not only routes across different models and providers, but actually you can route across different instantiations of the system prompt that work well with different subtasks and so on."
This approach tackles one of the toughest challenges in AI development: crafting system prompts that handle all edge cases without compromising performance on simpler tasks. By dynamically selecting not just the model, but the specific prompt configuration best suited to each task, developers can build more robust and reliable AI systems that maintain high performance across diverse use cases.
Lenton explains some of the common pitfalls that developers run into when trying to build and optimize these types of large-scale prompts without utilizing advanced LLM Routing in the below clip:
Benefits of LLM Routing in Scaled Applications
LLM routing is particularly valuable when deploying applications at scale, where cost and performance are critical factors. "Particularly when you're scaling out your application and you start to care about the speed and you start to care about the margins, the cost margins, that is, then a lot of companies are wasting a lot of money and, you know, very slow responses on these very large models, which are overkill for a lot of the easy prompts," Lenton points out.
By intelligently routing simple prompts to cheaper, faster models and reserving more powerful models for complex queries, LLM routing helps optimize the cost-effectiveness and performance of language model deployments. Unify AI's technology is highly customizable, allowing users to define their own evaluation metrics, comparison models, and training data to tailor the router to their specific use case and requirements.
Conclusion
LLM routing represents a major leap forward in AI optimization, paving the way for more efficient, cost-effective, and responsive AI systems. As the tech evolves, with developments like system prompt routing on the horizon, there is exciting potential for future applications.
The key to success lies in striking the right balance between innovation and practical implementation. By starting with clear use cases, focusing on measurable improvements, and gradually expanding capabilities, businesses can harness LLM routing to build AI systems that aren't just smarter, but more efficient and effective at solving real-world problems.
As we look to the future of AI applications, it's clear that technologies like LLM routing will be crucial in shaping how we build and deploy intelligent systems. The challenge – and the opportunity – lies in bridging the gap between cutting-edge research and practical, value-driven applications.
You can watch the full episode of RAG Masters here:
