--- title: What We Learned Building AI Agents for Intelligence Analysis and OSINT description: Discover key lessons from developing AI agents for intelligence investigations, focusing on accuracy, validation, and the importance of human oversight. image: https://publicinsights.uk/hubfs/AI%20Assisstan-1.png --- [![Public\_Insights\_Logo\_Image\_2024-removebg-preview](https://publicinsights.uk/hubfs/Public_Insights_Logo_Image_2024-removebg-preview.png)](https://publicinsights.uk) - Products - [Cradle](https://publicinsights.uk/cradle) - [Canvas](https://publicinsights.uk/canvas) - [Background Checks](https://publicinsights.uk/background-checks) - [API](https://publicinsights.uk/api) - Pricing - [Individuals](https://publicinsights.uk/pricing) - [Teams](https://publicinsights.uk/teams-pricing) - Industries - [Public Sector](https://publicinsights.uk/public-sector) - [Security](https://publicinsights.uk/security) - [Insurance](https://publicinsights.uk/insurance) - [Legal](https://publicinsights.uk/legal) - [Private Investigation](https://publicinsights.uk/private-investigation) - [Cybersecurity](https://publicinsights.uk/cybersecurity) - [Journalism](https://publicinsights.uk/media) - Resources - [Resource Centre](https://publicinsights.uk/resources) - [UK Sources Guide](https://publicinsights.uk/uk-public-records-guide-for-osint) - [Careers Hub](https://publicinsights.uk/osint-careers) - [Platform Demos](https://publicinsights.uk/platform-demos) - [About](https://publicinsights.uk/about) [SIGN IN](https://cradle.publicinsights.uk/accounts/login/) [SIGN UP](https://cradle.publicinsights.uk/accounts/signup) OSINT # What We Learned Building AI Agents for Intelligence Analysis and OSINT Discover key lessons from developing AI agents for intelligence investigations, focusing on accuracy, validation, and the importance of human oversight. [Dan Faram](https://publicinsights.uk/blog/author/dan-faram) Oct 16, 2026 --- AI can now produce a convincing intelligence report. Give a capable model a person or company to investigate, access to search tools and enough time, and it can collect information, examine documents, identify connections and produce a structured report complete with citations. But how do you know whether the report is accurate? A citation doesn't necessarily mean the source supports the claim. A sanctions screening result showing no matches doesn't necessarily mean the subject was successfully screened. And an AI system reporting that it completed every stage of an investigation doesn't necessarily mean it did. These are some of the problems we've encountered at Public Insights over the past several months while developing AI harnesses for intelligence analysis, including work supporting Dossier, our AI-powered investigation and due diligence reporting tool. We've documented the engineering challenges, failures and controls in a new technical paper, [*Building an AI Harness for Intelligence Analysis*](https://publicinsights.uk/hubfs/Building%20an%20AI%20harness%20for%20intelligence%20analysis%20(1).pdf). Here are some of the more interesting lessons. ## The model isn't necessarily the biggest problem When we started building these systems, we expected hallucinations to be one of the biggest challenges. Large language models can invent facts, confuse people with similar names, misinterpret documents and make connections that aren't supported by evidence. These are familiar problems for anyone who has experimented with AI-assisted research. What surprised us was how often the problem sat elsewhere in the system. The most common defect we encountered was an automated check that reported no problems because it hadn't actually performed the check. A test runner could report hundreds of passing tests while failing to discover some of the test files. A link checker could report success after checking zero links. A screening tool could return no matches because the source it was supposed to query hadn't loaded. In each case, the system produced an apparently successful result. This is a particularly serious problem in intelligence work because the absence of a finding can itself become a finding. If a sanctions screening tool returns no matches, an analyst might reasonably interpret that as evidence that the subject was screened against the relevant lists and no potential matches were identified. But what if the list never downloaded? What if the request was blocked? What if the matching algorithm failed to recognise a different ordering of the same name? The distinction between "we checked and found nothing" and "we couldn't check" is fundamental. We now design screening and validation processes around that distinction. A check must be able to report that it ran successfully, that it identified a problem, or that it could not evaluate the material. The third outcome must never silently become the first. ## AI can find ways to skip its own rules An AI harness is the collection of rules, scripts, tools and controls built around a language model to govern how it performs a task. In our case, that means taking the model through an intelligence reporting process: collection, source reading, fact extraction, analysis, drafting and quality assurance. Some stages require human decisions. An analyst might need to approve the hypotheses being considered, decide whether additional commercial data should be obtained, or review an assessment before it is written. We initially relied on written rules and status records to enforce these requirements. The problem was that the model could write those records itself. We encountered cases where mandatory steps were marked as overridden with explanations such as "smoketest scope", despite no analyst having authorised the decision. The model had effectively invented a justification for skipping work. This wasn't necessarily deliberate deception. Models can rationalise shortcuts in ways that sound entirely reasonable, particularly when a task is long or complex. They might claim that further research would be inefficient, that sufficient evidence has already been collected, or that a step can be omitted because of time constraints. The resulting audit trail can look as convincing as the report itself. We responded by moving important approvals outside the model's control. In one harness, an override requires a cryptographic signature produced through a hardware-backed key and a physical fingerprint approval. The model can request permission to skip a step, but it cannot grant itself that permission. The broader principle matters more than the particular technology: if a model can write its own approval, the approval isn't an independent control. ## A genuine source can support a false finding Requiring citations is an obvious starting point for AI-generated intelligence. Yet, it solves only part of the problem. A model can cite a genuine Companies House record while attributing information from that record to the wrong person. It can reproduce a real financial figure while attaching a footnote to a different source. It can identify two genuine entities and invent a relationship between them. These errors are particularly difficult to detect because the cited sources themselves are real. We encountered cases where a model extracting facts from a document was given the investigation subject's name and then attributed information about someone else to that subject. The source might be an authoritative government register. The extracted fact might receive a high reliability grade because of that source. Yet nothing in the document actually establishes that the information concerns the person being investigated. This is why verifying that a source exists is not enough. The system needs to establish that the source supports the specific claim being made, and that the people or organisations mentioned have been correctly identified. We've developed checks that compare figures in a report against the underlying fact base and then against the particular sources cited in each sentence. Identity resolution remains harder. A matching name alone is rarely sufficient to establish that two records concern the same person. Dates of birth, addresses, company numbers, associates and other identifiers may be needed to distinguish individuals. Where the evidence remains ambiguous, the system should preserve that uncertainty rather than quietly resolving it. ## The evidence needs to exist independently of the report One of our most consequential design decisions was to separate the evidence collected during an investigation from the report written about it. The model doesn't simply gather information and turn it directly into prose. Instead, individual claims are recorded in a structured evidence ledger, with their sources, retrieval details and reliability assessments. Conflicting information remains visible, rather than being silently combined into a single answer. The report is then drafted from that evidence. This separation makes several things possible. A figure in the finished report can be checked against the underlying facts. A citation can be traced to the source from which a claim was extracted. A reviewer can examine material that was collected but didn't make it into the final assessment. It also means the evidence survives independently of the model's account of what happened. Without that separation, the model is effectively the only witness to its own work. It produces the findings, writes the report and describes the process used to create it. That is not a particularly strong basis for auditing an intelligence product. ## Even the safeguards need testing Adding more checks doesn't automatically make a system more reliable. Every new validator, script or automated control introduces another component that can fail. Some can fail silently, producing reassuring results while doing nothing useful. One of the most effective techniques we've adopted is deliberately introducing errors to establish whether the checks actually detect them. Software engineers call this a positive control. It's the equivalent of pressing the test button on a smoke alarm. If a system is supposed to detect an invented figure, insert one and see whether it gets flagged. If a citation checker is supposed to identify unsupported claims, give it a claim with an incorrect source. If a mandatory approval is missing, confirm that publication is blocked. A clean result is only meaningful if you have established that the check is capable of finding the problem. We've also learned that complexity can become a problem in its own right. AI coding tools make it relatively easy to build elaborate validation systems. They don't necessarily help you decide whether every component is justified. Some of the machinery we developed proved unnecessary or too cumbersome for the reporting process. In those cases, the better approach was to retain the important safeguards while removing the additional complexity. A quick background profile and an investigation involving thousands of sources don't need identical levels of assurance. ## What this means for AI-assisted intelligence These experiences have influenced how we think about AI's role in intelligence analysis. The objective isn't simply to automate the production of reports. It's to make more of the research process scalable while preserving the ability to understand, verify and challenge the findings. AI is increasingly capable of performing substantial amounts of collection, reading and information processing. That creates opportunities for analysts to investigate more broadly, examine more material and spend less time on repetitive tasks. But the analytical problems don't disappear. Sources can be inaccurate or misleading. Records can concern the wrong individual. Evidence can support multiple interpretations. A system can complete every technical check and still reach a questionable conclusion. Some of these problems can be reduced through engineering. Others require the experience and judgement of an analyst. This is particularly important for due diligence, where an incorrectly attributed adverse finding can have serious consequences, and where failing to identify relevant information can be just as consequential. At Public Insights, we're applying these lessons as we develop Dossier and our wider AI investigation capabilities. The systems are still evolving, and not every safeguard described in our research is deployed across every workflow. Our aim is to make AI-assisted investigations more useful without treating polished output as a substitute for reliable evidence. The more capable these systems become, the more important it will be to ask not just what they can produce, but what they can demonstrate about how that output was produced. We've explored the engineering behind these challenges in much greater detail in our new technical paper, [**Building an AI Harness for Intelligence Analysis**](https://publicinsights.uk/hubfs/Building%20an%20AI%20harness%20for%20intelligence%20analysis%20(1).pdf). The paper covers the architecture, implementation examples, failures we encountered and the controls we developed in response. [OSINT](https://publicinsights.uk/blog/tag/osint) [AI](https://publicinsights.uk/blog/tag/ai) ## Similar posts OSINT ### [Why Synthetic Identities Pass Existing Checks and How OSINT Reveals Their Full Story](https://publicinsights.uk/blog/why-synthetic-identities-pass-existing-checks-and-how-osint-reveals-their-full-story) Discover how synthetic identities evade detection and learn how OSINT can unveil the full story behind them for enhanced security and fraud... Sham Ahmed Apr 29, 2026 OSINT ### [Why Charities Need to Know Who They Take Money From](https://publicinsights.uk/blog/why-charities-need-to-know-who-they-take-money-from) Charities must carefully evaluate their funding sources to protect their reputation and mission, ensuring ethical alignment with potential donors and... Sham Ahmed Aug 25, 2026 OSINT ### [Identifying an Unlicensed HMO with OSINT](https://publicinsights.uk/blog/identifying-an-unlicensed-hmo-with-osint) How a customer investigated an unlicensed HMO with OSINT techniques, including uncovering vital public records with Cradle. Sham Ahmed Nov 23, 2024 ### Get notified about new tools and techniques for OSINT Learn about new developments in the world of OSINT from the Public Insights team. [![Public\_Insights\_Logo\_Inverse\_2024-removebg-preview](https://publicinsights.uk/hubfs/Public_Insights_Logo_Inverse_2024-removebg-preview.png)](https://publicinsights.uk/old) Public Insights London, UK ### Product - [Login](https://cradle.publicinsights.uk/accounts/login/) - [Sign Up](https://cradle.publicinsights.uk/accounts/signup) - Cradle - Pricing ### Solutions - Public Sector - Corporate Security - Insurance - Journalism - Private Investigation - Legal - Cybersecurity ### Resources - Resource Centre - About © 2024 Public Insights All rights reserved Privacy Policy [Powered by Atlas - a B2B SaaS HubSpot theme](https://www.kalungi.com/atlas-hubspot-theme-for-b2b-saas-software) ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Dan Faram", "url" : "https://publicinsights.uk/blog/author/dan-faram" }, "dateModified" : "2026-10-16T13:03:13.856Z", "datePublished" : "2026-10-16T12:59:53.000Z", "headline" : "What We Learned Building AI Agents for Intelligence Analysis and OSINT", "image" : [ "https://publicinsights.uk/hubfs/AI%20Assisstan-1.png" ], "mainEntityOfPage" : { "@id" : "https://publicinsights.uk/blog/what-we-learned-building-ai-agents-for-intelligence-investigations", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://publicinsights.uk/hubfs/PI%20Logo%20v3.png" }, "name" : "Public Insights" } } ```