Five AI Search Platforms Collectively Missed 12% of Benchmark Studies

Five publicly accessible retrieval-augmented and large language model–based evidence-search platforms returned incomplete and uneven sets of relevant studies in response to realistic queries, according to an evaluation published as an article in press in npj Digital Medicine.

Investigators tested two academic search tools, Ai2 Paper Finder (Allen Institute for AI) and Consensus (Consensus Technologies), and three conversational platforms: ChatGPT-4.1 (OpenAI), Gemini-2.5-Pro (Google DeepMind), and Claude-Sonnet-4.0 (Anthropic). To avoid benchmark contamination, they used a prospectively assembled, non-public gold-standard corpus of 83 studies on natural language processing tools for identifying substance use disorders in electronic health records.

Each platform received 15 query formulations, each executed in triplicate, between June 16 and 22, 2025, using free browser-based tiers. Median formulation-level recall, a primary outcome, ranged from 7.2% for Gemini-2.5-Pro to 42.2% for Ai2 Paper Finder. When all formulations and executions were pooled, recall ranged from 45.8% for Claude-Sonnet-4.0 to 72.3% for Ai2 Paper Finder. Overall, 12.0% of gold-standard studies were not retrieved by any platform.

Gaps varied by evidence category. For tobacco-related studies, the largest category, the probability that a single formulation returned no relevant study ranged from 47% for Ai2 Paper Finder to 80% for ChatGPT-4.1. Consensus retrieved no study published before 2016, and never-retrieval was more common for conference proceedings than journal articles (38.9% vs 4.6%; P<0.001). Combining Ai2 Paper Finder with Claude-Sonnet-4.0 covered 83.1% of the benchmark.

The investigators noted that the study evaluated one research question and evidence corpus, that platforms were tested through free browser-based interfaces during a defined period, and that internal retrieval mechanisms were not observable. They framed the findings as a warning rather than a platform ranking, stating that a single AI-assisted query should not be treated as a sufficient representation of the available evidence.

Source: Farrag AN, Soliman A, Azubuike C, et al. Blind spots in AI-assisted healthcare evidence search: multiplatform evaluation of clinical retrieval gaps and risk-of-bias. NPJ Digit Med. Published online September 29, 2026. doi:10.1038/s41746-026-03277-y