How do MCPs compare against a dedicated company search agent?

There's a recurring assumption in the current wave of AI tooling discourse: if you have a sufficiently capable model and connect it to your company tools' MCPs, the retrieval problem is solved. The model will figure out where to look, what to pull, and how to synthesize it. The reasoning layer, in this view, is the bottleneck. The retrieval layer is just plumbing.

We tested this assumption.

At Super, we build a retrieval engine for enterprise knowledge, the kind of system that connects to a company's Slack, Linear, CRM, support tools, and documents, then answers questions across all of them.

With the Model Context Protocol becoming the universal standard for connecting AI models to external tools (97 million monthly SDK downloads, backed by Anthropic, OpenAI, Google, and Microsoft as of late 2025), a straightforward question kept surfacing

Could a team skip a purpose-built retrieval layer entirely and just wire Claude to their tools via MCP? Has Anthropic killed our startup?

We designed a benchmark to find out.

The methodology

We compared three configurations, all querying the same company data (ours):

1. Super - our product, connected to Slack, Linear, Attio, Intercom, and Slite for this test.
2. Claude Code with MCPs - Claude with five MCP integrations: Linear, Attio, Slack, Intercom, and Slite. The same data sources a team would realistically wire up.
3. Claude Code with Super - Claude using Super's MCP as its single data source. Think of this as a hybrid: Claude's reasoning on top of our search layer, instead of individual tool connections.

Same questions. Same data. Same person rating the answers.

One thing to flag upfront: we ran this on our own company data, and a Super team member did the rating blindfolded.

What we asked

We ran the kinds of queries a real team member would ask on any given Tuesday:

1. Simple lookups: "What's our Stripe discount for annual plans?", "What's Slite's registered address?", "What are our SLAs?"
2. People-related queries: "What has Jason achieved in July 2025?", "List the main features Charley worked on, one per month"
3. Complex synthesis: "Generate a changelog from tickets closed in Linear and merged PRs last week, only about Super"
4. Broad knowledge: "Give a 360 overview of Frontify" (a customer account)

Some of these need a single fact pulled from one source. Others need the AI to stitch together information from three or four tools and make sense of it. That range matters, because the gap between these setups changes dramatically depending on what you're asking.

How we rated

Every answer got a score from 0 to 4:

• 0 - Given for completely incorrect answers that misinterpreted the question, gave the wrong information, or hallucinated.
• 1 - For answers that were based on mixed data from correct sources and outdated sources
• 2 - For answers that were partially correct, but could be risky for data-related queries.
• 3 - For nearly perfect answers in terms of facts.
• 4 - For answers that were not only factually correct, but well-formatted, and added related context.

The results

Upfront, here is what we observed

• Our dedicated company search answered 2.6x faster
• It gives stellar answers (exhaustive and fully complete) 83% of the times, versus 25% with Claude.
• Simple connectors gave wrong or poor answers in 31% of the cases (vs 9% for Super)

Speed

For simple lookups, the gap was modest and ignorable.

But for anything requiring multiple sources, Super was 1.5-8x faster. Across all 41 questions, Claude with MCPs averaged 2.6x slower than Super. Claude with Super averaged 1.7x slower.

The widest gap came up on a 12-month feature breakdown, where Super took 35 seconds and Claude with MCPs took 282 seconds.

The reason is architectural. Claude with MCPs calls each tool sequentially, one after another. A question like "What did Charley work on across 12 months?" triggers eight or more separate searches across four integrations. Each call adds latency. When one tool throws an error (Intercom cascade errors alone added roughly 30 seconds in some runs), the entire chain stalls.

This is not a problem unique to our test. Research on parallel versus serial tool calling consistently shows the latency difference is structural. Relace documented a 4x reduction in end-to-end latency by parallelizing tool calls while retaining accuracy. The fundamental issue is that serial tool calling scales linearly with the number of sources queried, while parallel retrieval keeps latency roughly constant regardless of source count.

The MCP ecosystem compounds this. A DEV Community analysis described the current state as a "Wild West of broken implementations,"documenting endpoint discovery failures and retry logic entering infinite loops.

Super queries all sources in parallel with a single call. The difference purely comes down to how the retrieval layer is architected.

Accuracy

For simple factual queries, Super still led but the gap was smaller. The gap appeared when complexity increased.

On a question about a customer's 360-degree overview, Super returned the deal size, key contacts, usage metrics, and strategic context. Claude with MCPs found half the facts and ran into Attio search errors.

For a question on Jason's achievements question, Claude with MCPs retrieved 10 unrelated Jason contacts from Attio. The signal-to-noise ratio degraded as the number of sources increased, which is the opposite of what you want.

This tracks with what enterprise search benchmarks show more broadly. Scale AI's Agentic Tool Use leaderboard and Aisera's CLASSic framework both evaluate AI agents on multi-tool tasks, and both find that accuracy degrades as tool count increases.

The agent must decide tooling, query formulation, and reconciling conflicting results.

A purpose-built retrieval system doesn't face this issue because the orchestration logic is pre-existing rather than improvised by the model at inference time on every query.

An interesting finding from the hybrid: Claude + Super MCP

This was the most interesting configuration to us. Claude using Super as its single data source instead of five separate MCPs.

It landed in the middle on speed, faster than Claude with individual MCPs, slower than Super on its own. Which makes sense: you still have the overhead of Claude's reasoning loop, but you skip the serial tool-calling across multiple integrations.

Where it got interesting was accuracy. On some queries, Claude's reasoning on top of Super's retrieval produced answers that were structured differently, sometimes more analytical, than Super alone. The changelog task was a good example. Claude added context and formatting that made the output more immediately useful.

This hybrid setup was the most thought-provoking result of the whole test. If Super handles retrieval and Claude handles reasoning, you get the reliability of purpose-built search with the analytical depth of a general-purpose model. We're biased, obviously, but we think this is where things are heading.

Where each approach shines

Claude with MCPs has a few strengths:

• Reasoning depth: Once Claude had the right information, its analysis was sharp. It didn't just find data, it interpreted it. The changelog task produced well-structured, contextualized output.
• Conversational flexibility: You can ask follow-ups, change the angle, have it reformat. It's a conversation, not a search query. That matters for exploratory work where you don't know exactly what you're looking for yet.
• No additional cost: If you already pay for Claude, the MCP route costs nothing extra beyond setup time. For a small team, that's a real consideration.

Super's advantages showed up most clearly when complexity increased:

• Speed under complexity: When questions required pulling from 3-4 sources, the integrated architecture meant no serial tool-calling overhead. One query, all sources, one answer.
• Reliability: No cascade errors, no retries, no "Intercom MCP timed out" moments. It either finds it or tells you it can't.
• Signal-to-noise: Claude with MCPs sometimes returned everything it could find, relevant or not (those 10 unrelated Jason contacts from Attio, for instance)

Why we're sharing this

We use Claude ourselves. Every day. We built an MCP integration so Claude can use Super as a data source. We're publishing this because teams are making real decisions about how to set up their AI knowledge stack right now, and there's very little honest data out there to help them.

For the specific job of finding and synthesizing company knowledge across scattered tools, a purpose-built engine was faster and more reliable than wiring up individual MCPs. For open-ended reasoning and analysis, Claude's general intelligence is hard to beat. And for some use cases, the combination of both was the most interesting result of all.