Agent Orchestration at Scale: Lessons From Superhuman Go

By Kellen Gillespie, Applied Research Scientist, Superhuman. 

We recently launched a new version of Superhuman Go, an AI assistant that works across your apps and tabs, understands context, and jumps in to help at the right moment. Under the hood, Go routes user requests to the right tool from a growing library of 100+ agents (e.g., calendar, email, and project management agents) to help users get more work done. And if users don’t find the right agent, they can also add their own. 

In building and scaling Go, one of the main challenges was ensuring that orchestration and routing worked as we added more agents. This is a known risk: Existing research shows that large language model (LLM) tool-calling performance degrades as more agents are added, with a 7%–85% performance drop as the number of tools scales from 49 to 741. Most related research, however, uses curated toolsets with uniform definitions in its analysis. There isn’t much research that explores how routing breaks at scale in production enterprise settings, where tools span diverse domains with overlapping scopes.

This risk directly impacts user experience—if Go picks the wrong tool when given too many choices, the user’s request fails or requires manual correction. At Superhuman’s scale, even a 1% failure rate can affect hundreds of thousands of customers. To ensure we could deliver an optimal user experience at scale, we needed to understand exactly when and why routing breaks and explore different solutions to fix it. 

In this blog post, we’ll take a closer look at our team’s research into Go’s routing performance degradation, compare different solutions to improve performance, and do a deep dive into the most promising fix: embedded shortlisting. 

Identifying the root cause of scaling performance degradation

Go's catalog consists of over 100 agents and 800 individual tools, ranging from single-purpose agents (Weather, Google Translate) to multi-tool suites (Gmail with 15+ actions, Jira with 20+). To successfully complete a user’s query, Go must route it at two levels: first, to the relevant agent or product, then to the specific tool within that agent or product that can complete the task. For example, if a user asks Go to “Send a follow-up email to Larry about rescheduling the meeting,” Go must first recognize that we want to use the right agent (e.g., the Email Assistant agent) and then find the right tool within the agent (e.g., use the Gmail agent if the user’s email provider is Gmail). 

Failure can occur at any point in this journey. If a user’s query is ambiguous, Go might map it to multiple possible candidates and could end up picking the wrong one. If Go fails the first step, then it will always return the wrong tool. However, even if Go finds the right agent, there’s a high degree of semantic overlap across tools. There are multiple email clients, writing tools, project management software, and document editors that sound alike (e.g., Google Docs versus Coda Docs). As a result, even if Go finds the right agent, it might pick the wrong tool for a task. To solve this problem, we need to understand how much each of these issues contribute to Go’s overall performance degradation. 

To answer this question, we developed an evaluation framework to measure routing performance across three frontier models: GPT-5.1, GPT-5.4, and Claude Sonnet 4.5. To get an appropriate baseline, we configured these models to use flat routing (i.e., the model chooses directly among all available tools, without hierarchical routing). We quantified performance using F1 (i.e., the harmonic mean of precision and recall). 

We tested these models on a custom-built query set, primarily composed of synthetic queries and a small sample of anonymized production Go queries. The synthetic queries span various difficulty levels: easier explicit queries that name the tool (e.g., “send a Gmail email”) and more difficult implicit queries that require reasoning to identify the need (e.g., “send a note to the team about Monday’s deadline”). 

Through this testing, we found that F1 decreases as you add more tools across both synthetic and production queries—an unsurprising finding given the existing research. Specifically, F1 drops by 16%–23% as we scale from 10 to 110 agents (or 51 to 584 tools). Interestingly, the type of model doesn’t really change the results; stronger models perform slightly better, but the overall degradation trend remains the same.

When we break it down, we find something interesting: The performance degradation is recall-driven. Specifically, precision drops moderately (68%→60%) while recall drops more than twice as fast (55%→37%). This means that as the catalog grows, the model increasingly misses valid tools more often than it selects invalid ones. This suggests that tool retrieval (e.g., the model surfacing the right tool) might be the bigger problem for us to solve.

Figure 1: A breakdown of precision, recall and F1 for model performance across various sizes of tool catalogs. 

Solving the tool retrieval problem

There are many different ways to improve tool retrieval. Here are just some of the solutions we considered: 

• Richer tool descriptions: Enriching tool metadata to improve retrieval
• Model provider–based tool search: Common examples include OpenAI’s namespace-based tool search 
• Hierarchical LLM routing: A technique where an LLM first picks the category, then routes within that category
• BM25 lexical retrieval: A simple search algorithm for retrieving the most relevant document given a user’s query
• Embedding-based shortlisting: Narrowing down a list of candidates to a smaller number by using an embedding model 

To determine the best solution for our use case (e.g., improving tool retrieval for Go), we tested all these techniques across all three frontier models, measuring the retriever’s performance on our synthetic query set (via the F1 metric) as we added more tools to Go’s catalog. We also added an oracle shortlister that models perfect retrieval, since it always has the right tool in its shortlist. This helps us see how much of the degradation can be recovered with shortlisting quality and how much is due to other confusion. 

The testing helped us quickly eliminate options. For instance, rich metadata provided a benefit at the agent level but a near-zero effect at the tool level. Similarly, with BM25, we saw that it performed worse than our baseline flat routing (i.e., just using the model’s built-in routing mechanism); we believe this is because enterprise tools share too much vocabulary for lexical matching to be effective. Interestingly, we saw that hierarchical LLM routing underperformed all other approaches. Our hypothesis is that while hierarchical routing narrows the search space at the category level, it loses some of that advantage when the selected category expands into a large set of tools for the router to sift through—especially when some of those tools might be irrelevant. As a result, the overall performance suffers. 

Two options seemed promising: the model provider–based tool search and embedding-based shortlisting. However, while the tool search performs well when it’s used for fewer than 180 tools, it plateaus at larger scales. The other option, shortlisting, outperformed flat routing at every scale point (by 6%–11%) and matched or exceeded platform tool search as we added more tools. These results are model agnostic, with all three models improving performance by ~10%. We further validated this on our production datasets, where we saw a 10%–17% improvement across models with shortlisting.

Figure 2a: F1 performance across some of the techniques we explored. As mentioned above, shortlisting outperformed flat routing and platform tool search even as the number of tools expanded. 

Figure 2b: Shortlisting outperforms flat routing on synthetic (left) and production (right) datasets across various models and tool catalog sizes.   

A natural next question is, “What makes embedding shortlisting so effective?” We’ll take a closer look at this question in our next section. 

Why is shortlisting so effective?

To answer this question, we conducted a deeper analysis by examining the query-level outcomes across flat routing and embedding shortlisting. The full results are shown in Figure 3. 

At the top level, these results confirm that shortlisting outperforms flat routing, as 80% of queries receive at least one correct tool, compared with 69% under flat routing. But it’s interesting to understand why—there’s more green (i.e., correct predictions) and fewer red (i.e., router misses), suggesting shortlisting converts routing misses into correct predictions. We think this is likely because the model now has fewer options to choose from (less bloat), making it easier to pick the right one. 

Figure 3: Query-level outcomes analysis between flat routing and embedded shortlisting. 

We also analyzed individual tool predictions across flat routing and shortlisting to further understand this trend. The full results are below, but we see that, similar to query-level outcomes, there are more correct predictions and fewer errors. In this case, it’s driven by a 15% reduction in “catch-all absorption,” which happens when the model can’t find the answer and defaults to a broad tool or agent. When shortlisting, it’s able to find the right specialized solution. 

Figure 4: Prediction-level accuracy analysis between flat routing and embedded shortlisting. 

We think that the model’s positional bias also plays a role here, as it favors the first candidate tools listed, leading to ranked retrieval methods achieving an even greater accuracy boost. Our analysis shows that this can result in a 2%–4% boost in accuracy, as shown in the table in Figure 5. 

Figure 5: Analysis of the impact of the model’s position bias. 

In summary, shortlisting works by reducing the number of options, increasing the likelihood that it picks the right tool, and reducing the probability that the model chooses the generic catch-all option. 

Shortcomings of shortlisting 

Even though shortlisting is incredibly effective at improving tool retrieval, it’s not perfect. Specifically, there are two major gaps: shortlister miss and router confusion. Let’s take a closer look at each. 

Shortlister miss

If the correct tool never makes the shortlist, then the router cannot identify it, resulting in an overall miss. In Figure 3, we see that this affects 9% of the queries. Interestingly enough, flat routing doesn’t have this issue, as it considers the entire tool catalog rather than narrowing it. Thankfully, the rate remains stable across scales, so the net benefit of shortlisting continues to improve as catalogs grow (and some initial results show that better embedding models can help close this gap, too). 

Router confusion

In this case, even when the right tools are on the shortlist, the model still doesn't return the right one because it gets confused among them. This happens 10% of the time, as represented in Figure 2 by the decline in the oracle’s performance as we added more tools. We also see this in the prediction-level accuracy illustrated in Figure 4, where shortlisting doesn’t change the amount of cross-cluster confusion error (i.e., when the model picks the tool from the wrong functional group). This type of error is especially pernicious for enterprise use cases, where there are often semantically similar tools (e.g., Jira and Asana for task management).

The confusion gap requires catalog-level changes, such as tool deduplication, clearer descriptions, or multi-turn clarification for ambiguous intent—something shortlisting isn't designed to address (and we’ll see this finding again).

Optimizing shortlisting

Despite the shortcomings, we’re still really excited about shortlisting’s ability to improve routing and overall agent orchestration. To further optimize this solution, we compared the performance of various shortlisting methods (keyword-based, embedding-based, LLM-based) and structures (hierarchical, flat). 

We found a few interesting things. First, tool level (i.e., flattened retrieval) outperforms all pack level (i.e., choose a category and then the tool within it) by 2%–4%, as shown in Figure 6. We think this is because pack expansion loads irrelevant sibling tools, which bloats the router. 

Second, there’s promise in using smaller embedding models. We tested a few different retriever models, including a simple BM25 model, E5-large (~300-million-parameter model), OpenAI’s large text embedding model, and ToolRet (a refined version of E5 using query-and-tool pairs in a non-enterprise domain). Interestingly, we’re seeing similar performance between ToolRet and OpenAI’s embedding model, even with this domain mismatch. This suggests there may be performance gains from testing a smaller, more finely tuned, locally run embedding model, which would also reduce latency. This is something we’re definitely excited to explore in the future. 

Figure 6: A breakdown of tool versus pack-level performance across various embedding models. 

The bottom line

As Go launches and scales to more users, we’re excited to use this work to ensure that the quality of the user experience continues to meet our high bar, even as we add more agents and tools. If you want further details about our research, you can read the full preprint paper. And if you want to help us further improve agent orchestration and build performant agentic systems, apply to the open roles on our Research and Engineering teams.