Authors: Ritik Sharma, Balvir Singh Thakur
Abstract: General-purpose large language models (LLMs) are increasingly deployed as autonomous research agents: given a web-search tool and a task-specific instruction, they are expected to compile structured, factually complete documents from public sources with little human oversight. We report an empirical case study, spanning three successive system architectures of a deployed research-automation module, evaluating whether a GPT-5-class model can reproduce the completeness and accuracy of a curated reference document. We show that a completeness gap recurs at every layer of the pipeline, not just at the LLM layer intuition would blame first. At the LLM layer, a self-directed search-stopping criterion does not scale with true task size, producing near-complete coverage on a low-activity subject but only 9 of 16 known events (56%) on a high-activity subject under comparable search effort. On a large structured disclosure table, the model correctly reports the verified aggregate figure but declines to enumerate individual line items it cannot verify from search snippets, capturing 0% of line items in the case measured. Replacing the model's self-directed search with a deterministic, non-LLM pre-fetch of an authoritative source, combined with a mandatory reconciliation checklist, more than triples measured coverage (9 to 28 events) — but building that deterministic stage itself surfaced two further completeness bugs, both caught and fixed before production through direct re-verification against live source data: a naive entity-name query missing 4 of 5 known related records until query construction was corrected, and a silent 1,000-record pagination cap that would have discarded 69% of one subject's true history had it gone undetected. We then report the most severe instance of the same underlying pattern: in a substantially more deterministic third-generation architecture, purpose-built to eliminate LLM-driven table extraction entirely, a "most recently filed wins" version-selection heuristic silently selected an incomplete partial-amendment filing over a complete original filing for the same reporting period, capturing only 1 of 10 real holdings (3.8% of true portfolio value) — a bug caught only via comparison against a second, independent reference for a different subject, not by the team's own prior testing, and which in turn silently disabled a downstream data-quality safety check that depended on the same broken data. We report these as an empirical case study rather than a statistically powered benchmark, and specify the exact follow-up measurement needed to convert the paper's central open finding into a confirmed before/after result.
