Research Preprints Document AI Weaknesses in Idea Selection, Memory Fidelity, and Multimodal Evidence Use

Oxford Study Measures Filtering Difficulties in Engineering and Strategy

At the University of Oxford, researchers William Bolton and Philip Torr evaluated AI models across technical domains where expert solutions were established only after model training took place. The authors examined system performance in Formula 1 vehicle design under 2026 aerodynamic rules and in competitive Magic: The Gathering deck construction.

In the Formula 1 evaluation, the top-performing model, OpenAI's GPT-5.2, matched 10 of 40 verified human engineering innovations. However, the system required 166 separate proposals across multiple runs to reach those matches. In the card deck evaluations, models identified several effective interactions but routinely omitted the strategic context that made tournament combinations workable.

Selection Demands Shift Labor Rather Than Removing It

Bolton and Torr concluded that the central difficulty was filtering and prioritisation rather than creative generation. The evaluated models proved capable of proposing plausible concepts, but they struggled to rank the most effective choices without human intervention.

This dynamic carries direct operational consequences for enterprise deployment. An automated system that produces twenty plausible marketing proposals, software changes, or clinical summaries may appear productive. Yet if human specialists must spend substantial time sifting through seventeen unviable concepts to identify three workable answers, the deployment shifts labor rather than eliminating it. The authors suggest measuring precision among top recommendations rather than crediting a system simply because an accurate answer appears somewhere in a lengthy catalog.

Memory Compression Can Strip Source Uncertainty

In a separate preprint, independent researcher Alex Kwon documented an operational vulnerability termed "factwashing." The issue arises when an AI system condenses or rewrites information for long-term storage, keeping the primary assertion while shedding crucial details about who made the claim, their level of confidence, or its timeframe. Through this mechanism, unverified hearsay can enter an agent's persistent memory as an established truth without generating an explicit factual fabrication.

Kwon developed an open-source write-time inspection mechanism that contrasts proposed memory entries against the original source text. During an evaluation using an unmodified installation of mem0 2.0.7, the inspection tool flagged five out of eight hedged hearsay statements.

The author emphasized that this evaluation represented a single configuration and a single sampled run, with a wide uncertainty interval and one recorded false positive across five control samples. Even with those qualifications, the risk remains concrete for enterprise workflows that automatically convert customer calls, electronic mail, and message threads into records for downstream decisions.

Multimodal Systems Bypass Video Analysis for Text Shortcuts

Research from the Video-DeepResearch team revealed that multimodal agents frequently bypass necessary evidence. When tasked with answering questions requiring simultaneous investigation of continuous video footage and open-web sources, systems exhibited a pronounced preference for text-based web searches, often neglecting required visual inspection.

The authors observed that general frontier models regularly relied on internal training data rather than performing direct examinations of video frames. Even when accurate answers required inspecting specific moments in a visual recording, models frequently defaulted to external text commentary or broad background associations to formulate their responses.

Specialised Architectures Outperform Frontier Models on Visual Benchmarks

To address these shortcuts, the Video-DeepResearch researchers trained specialised 30-billion and 35-billion parameter models configured with staged tool permissions for visual and text analysis. Across the authors' custom 200-question benchmark, their strongest model variant achieved a 64.0 percent average accuracy score.

By comparison, general frontier models delivered lower accuracy under the authors' assessment conditions: Claude 4.5 Sonnet recorded 59.0 percent, Gemini 2.5 Pro reached 57.5 percent, and GPT-5 achieved 52.5 percent. The authors noted that these figures stem from a newly constructed benchmark evaluated partly through an automated model, meaning the conclusions require independent external replication.

The technical takeaway remains relevant for operational systems. When an automated workflow requires reviewing a legal document, engineering schematic, or video file, an agent must provide verifiable logs demonstrating that it retrieved the required file, activated the correct tool, and tied its conclusions to explicit evidence.