BACKGROUND
A government team running an application often needs to know how data is collected and where it ends up getting geographically stored. For an application that already exists, that’s a hard question to answer.
The data can move through application code, third-party SDKs, configuration, infrastructure, databases, queues, logging systems, and external services. Some destinations never show up in the repository at all, because endpoints and credentials are set at deployment time.
Doing that reconstruction by hand, across a large portfolio of applications, is slow, and it’s easy to miss things.
We wanted to see whether an AI coding agent could take on that investigation. We built an experiment, Sovereignty Trace, to find out.
HYPOTHESIS
Could a coding agent understand an unfamiliar repository well enough to find where sensitive data is stored, processed, or sent, and back each finding with evidence from the code?
If it could, most of the custom static-analysis machinery becomes unnecessary. What’s left is the method around the agent: what it looks for, the trusted provider and policy knowledge it can consult, the evidence it has to produce, and the checks applied to its result.
APPROACH
We started with static analysis
The first version of Sovereignty Trace treated the problem as a data-flow scanner.
We built language-specific analysis for Python and TypeScript, added Joern, an open-source static code analysis platform, to follow data across function boundaries, read Terraform for deployment information, created a registry of known service providers, and added policy rules for BC Government privacy and public-cloud requirements.
The goal was to turn repository evidence into a chain of increasingly specific facts, working outward from a piece of sensitive data to the policy that governs it.
On a synthetic benchmark, the approach looked promising. Joern solved one of the hardest parts of our own implementation: following data across functions. Combining our native analysis with Joern reached high precision and recall on the test cases we had built.
Those benchmarks reflected the patterns we already knew to test.
A real repository exposed the gaps
We then tested Sovereignty Trace against a production-style government application with multiple external integrations, messaging channels, data stores, and infrastructure components.
The scanner found several outbound data flows and produced several policy findings. Its LLM-assisted provider resolver even inferred some services where the endpoint was supplied through runtime configuration.
A manual repository review then found major parts of the application the scanner did not pick up.
SMS delivery used vendor-specific SDK methods. The app sent email through a mail library. PostgreSQL, Redis-backed queues, and centralized logging were all present, but none matched the destination categories we had defined.
One object-storage flow we meant to support was missed too, because the client was created in one part of the application and injected into another through the framework’s dependency injection. Tracing it meant following the application’s structure across several files.
These were normal architectural patterns in a modern application.
We could keep adding signatures for each SDK, framework, and infrastructure component we ran into. That path had a problem: to keep up, we were slowly rebuilding a tool that could understand any codebase on its own.
We compared it with a coding agent
A coding agent reviewing the same repository worked differently.
The agent started from the application itself. It built an understanding of the modules, dependencies, adapters, configuration, infrastructure, and main integration points before looking for specific destinations.
From there it could investigate questions like:
- What data does this application handle?
- Which components persist or queue that data?
- Which external services receive it?
- Which logging or observability systems might see it?
- Where are service clients created, and how are they passed through the application?
- Which destinations are known only through deployment configuration?
That broader context let it find integrations, stores, and data movement without us having to guess the exact library or framework API in advance.
The difference showed up as two ways of asking the same question:
For this problem, the top-down approach fits better.
We changed the architecture
The current version of Sovereignty Trace is smaller.
Sovereignty Trace now delegates the repository work. It launches an existing coding agent, at first through the Claude Agent SDK, and runs the process around it.
The agent works to a consistent set of instructions. It inspects the repository the way an engineer would: reading source files, searching dependencies, following configuration, inspecting infrastructure, understanding framework wiring, and revisiting earlier assumptions as it learns more about the system.
Its job is to identify:
- potentially sensitive data the application handles
- databases and other persistent stores
- external APIs and SaaS integrations
- messaging such as email and SMS
- queues and background processing
- logging and observability destinations
- object and file storage
- AI or model-provider integrations
- infrastructure and deployment configuration
- unknown or runtime-supplied destinations
For each finding, the agent points at the evidence rather than quoting it. It reports a file and a line range, and code reads the actual text out of the repository. Because the agent never types the quoted lines itself, it can’t fabricate a quote. A citation that doesn’t resolve to real lines is dropped, and a finding that loses all of its citations drops with it.
We let the agent observe, and let code do the math
The first version let the agent fill in the whole assessment, including the risk levels, the jurisdictions, and the lines it quoted. Run five times against the same commit, it produced 0, 4, 1, 5, and 4 high-risk findings, and four of those five runs failed our own checks.
The unreliable part was the derived work. A model is good at reading a repository and noticing that a piece of data heads toward a particular service. It is much less steady at applying a scoring rule the same way every time.
So we narrowed what the agent decides. Per finding, it now makes a handful of judgments: whether there is a finding at all, which provider it matches from a list we show it, how to classify the data, whether the code path is the live one or an alternate, which region the repository pins, and which category the finding belongs under. It also writes the plain-language notes.
Everything derived from those judgments moved into ordinary code: reading the cited lines, working out the jurisdiction, scoring the risk, and rolling up how much of the assessment got covered. The scoring rule used to be a paragraph in the agent’s instructions, applied by feel. It is now a plain function with a test for every case, so the same inputs always land on the same score.
What the agent is shown, and what it isn’t
Some facts don’t come from the repository, and we still need a way to ground those.
We first tried to keep the agent’s prompt lean by letting it fetch policy and provider details through tools, only when it needed them. Across five runs on the same repository, it answered 2, then 4, then 3, then 5, then 4 of the same six policy questions. Judging whether something meets a policy means reading the policy, so the full policy text now goes into every session up front.
Provider knowledge is split in two. The agent sees only enough to recognize a service: its name, a short description, and the domains and package names that give it away. Where that service stores and processes data is held back, and read by code after the agent has named the provider. Handing the agent those residency facts would invite it to recite them from memory instead of the record. The agent decides which provider a finding points to; code decides what that provider means for sovereignty.
If a provider fact, a location, or a policy implication can’t be established from the evidence, the result stays UNKNOWN.
The knowledge has to come from people who understand the domain
The provider and policy knowledge needs human review before anyone treats it as authoritative.
Provider records need review by people who understand the service, including what is known about storage, processing, legal entities, regional options, and other sovereignty-relevant details. Policy records need similar care from people who understand the applicable privacy, security, and government requirements.
For this experiment we’re starting with BC Government, because it gives us a concrete jurisdiction and policy context to build against. The structure is meant to extend past it. Another public body, province, or country could maintain its own policy pack and provider knowledge without changing how the coding agent investigates the repository.
So the reusable part is the method itself: a way to combine repository analysis with expert-maintained knowledge for whatever jurisdiction the software runs in.
We keep a small validation layer
Because the agent no longer writes the quoted evidence or the derived fields, most of what an earlier validation step used to check has gone away. There is no snippet to compare against the file, and no provider fact typed from memory to confirm.
What’s left is lighter. Code checks that the draft fits the expected shape, and that the assessment covered the ground: all ten component categories reported on, all six policy questions answered. A draft that doesn’t parse is the one real error, and the only case where the system asks the agent to try again. The rest are recorded as gaps in the finished document rather than failures, so the report still stands and says where it is thin.
The coding agent is replaceable
Claude Code was the first runtime, running through the Claude Agent SDK. We’ve since added a second, Swival, wired in through an open agent protocol. Building it was the real test of whether the method leans on Claude Code, and it doesn’t: the same instructions and the same structured output drive both. Adapters for other agents such as Codex and Copilot sit behind the same interface, ready to be filled in.
Swival is there for one reason in particular: running fully offline. Pointed at a model on the organization’s own hardware, it can assess a repository without the source code or the model ever leaving that environment. For sovereignty work, that is the point. We’ve been testing whether a local model can carry the investigation as well as a hosted one, and the early results are close: on the same repositories, a local Qwen 3.6 model has been coming out on par with Claude Sonnet 5.
TAKEAWAYS
01
A strong benchmark score can still hide real gaps. Our controlled tests showed high precision and recall, and the real repository still exposed major gaps. The benchmark proved the scanner could find patterns we had already anticipated. It told us nothing about the patterns we had never thought to encode. A passing score only confirmed that the scanner matched our own assumptions.
02
Where data goes is usually an architectural relationship you trace through the application. We expected domains, URLs, and known destinations to be strong signals for where data goes. In practice, destinations were often hidden behind runtime configuration, dependency injection, SDK abstractions, and framework wiring. A repository can show that an application uses a service without ever naming the endpoint. That shifts the task from extracting destinations to understanding how the application is put together.
03
An LLM placed after the scanner can only reason about what the scanner already found. Our first LLM-assisted resolver improved some provider identification. It could still only work with the observations the scanner handed it. If the scanner never found an integration, database, queue, or logging system, the LLM never saw it either. The fix was to move the coding agent upstream, so it runs the repository investigation itself instead of enriching an incomplete set of scanner results.
04
Let the model judge; let plain code do the math. Our first version let the agent fill in the whole assessment, including the risk scores. Five runs on the same commit gave 0, 4, 1, 5, and 4 high-risk findings. The model was good at reading the repository and spotting where data went, and shaky at applying a scoring rule the same way twice. So the agent now makes a few judgments per finding and writes the notes, while ordinary code does everything derived from them: reading the cited lines, working out the jurisdiction, and scoring the risk. The rule is a function with a test for every case, so the same inputs always land on the same score.
05
Show the model what it needs to recognize something, and hold back the facts it would otherwise guess. The agent sees enough to identify a service: its name, and the domains and packages that give it away. It doesn’t see where that service stores data, because it would tend to recite that from memory instead of the record, so code reads the residency off the matched record afterward. Policy runs the other way. Judging whether something meets a rule means reading the rule, so the full policy text goes into every session, after we watched an on-demand version answer a different subset of the policy questions on every run.
THE CODE
Sovereignty Trace is being developed as an open-source experiment.
The full source code is on GitHub.