Having spent time in intelligence, both in government and in the private sector, I have developed a fairly ingrained scepticism towards anything that claims to produce answers without showing its working. That scepticism has only sharpened over the last couple of years, as the market has filled up with tools that promise to automate investigative work using AI. Some of them are genuinely useful. A lot of them are not, and the difficult part, for anyone trying to buy or rely on one, is that it is almost impossible to tell the two apart from the outside.
The reason it is difficult is worth dwelling on, because it is not obvious. When someone demonstrates an AI investigation tool, they show you a case where it worked. The tool may take an entity name, produce a network, and the network looks plausible. You have no way of knowing whether it looked plausible because the tool did the right thing, or because it did something roughly in the vicinity of the right thing and you were not in a position to check. A plausible-looking answer and a correct answer are not the same, and the gap between them is exactly where a tool either earns trust or quietly fails to.
This article is about how we try to close that gap for our own tool, AI Assistant, which conducts automated corporate network investigations from UK public records data. My aim is to show the kind of validation we think this class of tool actually requires, partly because I think it is interesting, and partly because I think the standard is higher than most people realise, and that is worth saying out loud.
Before the validation, a short description of the thing being validated. Autoinvestigate takes a seed entity (or small group of entities), usually a company or person, and expands outwards through public data we have aggregated at Public Insights. It follows the connections that a human investigator would follow: officers, shared addresses, corporate shareholders, beneficial owners, and builds a network from them. The intended use is the early, laborious stage of a corporate investigation, the part where an analyst would otherwise spend hours clicking through filings to work out who is connected to whom and how.
The important constraint is that it works blind. It is given a starting point and public data and nothing else. It does not get told what the answer is, and for the validation to mean anything, it must not be able to find out. That constraint turns out to be central, and I will come back to it.
The question I started with was simple. How would I know whether an AI Assistant is any good? Not "does it produce a network", but "does it produce the right network". And the answer we settled on is that there is already a large body of work against which the tool can be measured: published investigations by journalists and researchers into UK corporate structures.
Over the last decade, a considerable amount of skilled investigative work has gone into unpicking how UK companies and partnerships are used to obscure ownership and control. There is reporting on nominee directors and corporate officers used to front large numbers of shell companies. There is work on Scottish Limited Partnerships and their role in moving money. There is analysis of beneficial ownership concealed behind offshore entities, and of mass-registration addresses where hundreds or thousands of companies share a single mailbox. These investigations are public, they are detailed, and crucially, the people who produced them showed their working. They represent, in effect, a set of worked answers.
So the benchmark is this. Take a published investigation. Identify the entity a human investigator started from. Give that same starting point to AI Assistant, blind, and see whether the tool, working only from public register data, independently reaches the same structural conclusions that the human investigators reached. Does it surface the same network? Does it identify the same pattern? Does it recognise, for example, that a cluster of companies is fronted by the same corporate nominee, or that control sits with an offshore entity rather than the named director?
If it does, that is meaningful evidence the tool works, because it got there on its own, against a known answer produced by skilled people. If it does not, the gap tells us precisely where the tool needs to be sharpened, and that is at least as valuable.
Here is where the exercise gets more involved than it first appears, and where most of the real work lives.
The entire benchmark depends on the tool working blind. If an AI Assistant can, in any way, learn the answer rather than derive it, the test is worthless. It is the difference between a student solving a problem and a student who has seen the mark scheme. And it turns out there are surprisingly many ways for the answer to leak in, several of which are not obvious until you go looking.
The most instructive example concerns the use of web search. Modern AI systems can search the web, and that is often useful. But these investigations are published. If the tool can search the web while working a case, it does not need to derive anything, it can simply read the article that describes the answer and hand it back. The person marking the test would see a correct result and conclude the tool is brilliant, when in fact it has done no investigative work at all. It has looked up the answer.
Preventing this is harder than it sounds, and the reason is a genuinely subtle technical point worth explaining plainly. The instinct, if you want to stop a system reaching the internet, is to cut off its network access. But the web search in a modern AI system does not necessarily run on your own machine. It can run on the AI provider’s servers, out of your reach. From where you are sitting, there is no network connection to block, because it is not your connection to block. Isolation, in this setting, cannot be a matter of pulling a cable. It has to be built deliberately into the tool’s own logic, and you have to verify that it holds, because if you get it wrong, you will run a test you believe is blind while the tool quietly reads the answer sheet.
There is a second, quieter version of the same problem. Even with web search switched off, a capable AI system carries a great deal of general knowledge. Some of these investigations are well known. Ask the system about a famous case and it may simply recognise it and recite what it knows, again without doing any actual work on the data in front of it. Guarding against that means being careful not to prompt the tool in a way that invites it to recall, and accepting that a well-known case is a weaker test than an obscure one for exactly this reason.
I labour this point because it is the heart of why validation is hard, and why a benchmark score on its own is close to worthless. A number is only as trustworthy as the process that produced it, and a process that has quietly let the answer leak in produces a number that means nothing at all. Most of the effort in this work is not in running the tool. It is in making absolutely sure the conditions under which you run it are what you think they are.
Running these benchmarks taught me a discipline that now governs how we develop the tool, and it is simple to state but easy to neglect. You verify what the tool actually did, on the real system, with evidence. You do not infer that it worked from the fact that nothing obviously went wrong.
This sounds pedantic until you have been caught by it. Suppose the tool is supposed to recognise a particular kind of corporate structure and handle it in a particular way. You run a case, the output looks fine, no errors appear, and you move on satisfied. But "looks fine and no errors" is not the same as "did the right thing". The structure you were testing for might not have appeared in that particular case at all, so the tool was never actually tested. Or the tool might have produced the right-looking output for the wrong reason. The absence of a visible problem is not evidence that the mechanism you care about did its job. It is only evidence that you did not see it fail, which is a much weaker thing.
So the standard we hold to is that an improvement is only real when we can point to the tool doing the specific thing we intended, on the live system, on real data, and show the evidence of it. Not the code that should cause it. Not a test against a simplified stand-in for the real data. The actual behaviour, observed. It is a slower way to work, and it is the only way I trust.
Much of this development is done in partnership with an AI assistant. This is worth being candid about, because it is central to how we work and because it is exactly the kind of thing that tends to be either oversold or hand-waved.
The assistant is genuinely capable. It can reason about corporate structures, work through the logic of the tool, run investigations and analyse the results at a pace no human matches. It is, in the proper sense, a collaborator on this work rather than a feature bolted onto it.
But the discipline I described above applies to the assistant as much as to the tool. An AI assistant, like any capable system, will sometimes conclude that something works because it looks as though it should. The value is not in delegating to it and accepting the result. The value is in a way of working where every conclusion is checked against what actually happened in the real system, where confident assertions are treated as things to be verified rather than facts to be accepted, and where the human remains responsible for the judgement that matters. Used that way, the combination is powerful. The point is not that the AI does the work. The point is that a rigorous process, with a capable AI inside it and a human holding the standard, produces results you can actually stand behind.
I think this is worth saying because there is a great deal of noise in this space, a lot of tools and claims that do not survive contact with a hard question. The way we try to distinguish ourselves is not by claiming our tool is flawless. It is by being able to show, in detail, the process by which we make it trustworthy. A tool that has been genuinely tested against skilled human work, under conditions you have taken real care to make honest, is a different proposition from one that merely demonstrates well.
Running investigations against real benchmarks is now a continuous part of how we develop AI Assistant. Each case is, in effect, a probe. It exercises a particular kind of reasoning, handling a particular type of entity, reading a particular part of the public record, and it shows us where the tool already performs and where it can be sharpened further.
That process has driven real, specific improvements. It has extended how completely the tool represents partnership structures, where control is often held through corporate members rather than named individuals. It has improved how it handles ownership held through offshore entities, and how it reads the filing history of a company, where a great deal of the interesting information, such as a company’s previous names, lives out of sight of the current record. Each of these came from watching the tool work a real case, seeing precisely where more could be done, and doing it, then confirming on the real system that the change did what we intended.
The result is a tool that gets steadily better in ways we can actually point to, against a standard set by skilled investigators rather than by our own marketing. That, to me, is the only honest way to build something in this space.
Some concrete examples make this less abstract, and each one traces back to a specific piece of published investigative work that set the standard we tested against.
If the standard of validation I have described is the standard you would want from an intelligence tool, then AI Assistant is built to it, and the best way to judge it is to use it. You can sign up and try it directly, or get in touch and we'll walk you through a live investigation using your own cases.
I would rather you tested it than took my word for it. That is rather the whole point.