We Gave Agents IDE-Native Search Tools. They Got Faster and Cheaper.

Denis Shiryaev

A correction
Updated July 29, 2026

We got an important part of this experiment wrong.

When we published this article, the search skill was loaded, but the IDE MCP tools behind it were not actually called. The agents used ordinary shell tools instead, and AIA Chat displayed some of those calls as “Search.” We read that as IDE-native search.

The quality, latency, and cost numbers are real, and the statistical calculations were correct. Our explanation was not. The experiment compared a plain agent with an agent that had extra search instructions. It did not show that IDE-native search made agents faster or cheaper.

Please do not treat the tool-adoption chart or the causal explanation below as evidence for the tools. We have kept the original article intact so the correction has context.

Read what happened when we fixed the setup and checked the raw traces.

We ran the same coding tasks with and without prebundled tooling, across multiple models and languages. Here’s what changed.

Why We Built This

When coding agents search code, they default to shell tools. grep and find work, but they’re blind to project structure, symbol boundaries, and language semantics. The agent burns tokens sifting through noisy output and making follow-up calls to narrow things down.

So we tried something obvious: what if the agent could use the IDE’s own search instead?

We built a prebundled skill that pairs a search prompt with a unified MCP tool. One tool, four modes: file search, text search, regex, and symbol lookup. A universal router dispatches calls to the right backend.

Methodology

The eval pipeline spins up an MCP server alongside the IDE so the agent has access to the configured tools and skills. We run identical coding tasks with and without tooling, then compare with paired delta analysis.

We track four things: quality, latency, cost, and budget discipline. Quality asks whether all tests passed. Latency tracks median and P95 task time. Cost converts token consumption into dollars. Budget discipline tracks how often a single task exceeds the USD 0.50 budget cap.

We report improvement deltas only when they pass our significance threshold: p < 0.05, paired test with 95% confidence intervals. Metrics without a significant change are either omitted from the charts or called out explicitly. We tried four configuration variants, selected the one with the best latency and cost tradeoff, then re-ran it on different models and languages to check that the results held.

Results

The selected configuration was a prebundled search skill plus a unified IDE-native tool and universal router. Compared with the no-tooling baseline, it reduced latency and cost without producing a statistically significant quality change.

Configuration Explorer

We tested four tool configurations before choosing the final shape. Lower latency and lower total cost are better, so the lower-left corner of the plot is the target.

Cross-Model Validation

We re-ran the experiment with GPT 5.4 on Java and Kotlin codebases. The pattern holds: latency and cost both drop. Kotlin saw the biggest cost improvement, with total cost falling 13.48%.

How Models Adopt Tooling

Codex sends 91% of its search calls through the new IDE-native tool. Claude is a different story: Opus uses it for about half its searches, and Haiku only 28%, preferring grep and find instead.

This makes sense. Claude already has strong built-in code search, so it leans on what it knows. Codex doesn’t, so it grabs the better tool when one is available. The takeaway: prebundled tooling fills gaps. Where the model already has good search, it adds less. Where search is weak, it makes a real difference.

What’s Next

The eval pipeline works. Now we’re using it.

We’re running the same experiment on smaller models next. Our hunch is that they’ll benefit even more, since they have less built-in search capability to fall back on.

The current results are strongest on Java and Kotlin. We’re expanding to Python, .NET, and TypeScript with bigger sample sizes.

Meanwhile, the winning configuration is being prepared for the integrated IntelliJ IDEA MCP Server, so agent sessions can use IDE-native tooling when the server is enabled.

The next step is to turn this feature on by default in upcoming AI Assistant plugin updates.

Want to try it before the default rollout?

1. Set these registry keys to true: llm.chat.agent.codex.mcp.idea, llm.chat.agent.skills.settings.enabled, and llm.agents.contrib.bundled.skills.sync.enabled.
2. In AI Assistant, choose Codex for the best results.
3. Ask the agent to find something across the current project.

What happened when we ran it again

We fixed the setup and inspected the raw traces instead of relying on tool labels in the UI. The follow-up covered 2,700 trajectories: 225 tasks, three runs per task, and four Codex configurations.

This was not an exact repeat of the original experiment. The agents had direct access to search_symbol, search_class, and search_usages, rather than the original skill-and-router bundle. We therefore treat it as a new test, not as retroactive proof of the original claim.

GPT-5.4-mini and GPT-5.5 did not call IDE search in any of their 675 trajectories. GPT-5.5 was still around 6% cheaper. Whatever caused that difference, it was not an IDE search call.

GPT-5.6 did use the tools. We found IDE search calls in 616 of 675 low-reasoning trajectories and 660 of 675 medium-reasoning trajectories. The cost went the other way. Median cost rose by about 6% at low reasoning and 7% at medium reasoning. In the medium-reasoning run, P95 cost rose by about 15% and total cost by 10%.

Our conclusion now is simple: this setup has not earned a default-on rollout. It may help on particular tasks, but we did not find a stable benefit across models.