Approaches to automated name matching

BACKGROUND

A common challenge when aggregating data from multiple sources is that the same real-world entity often appears under slightly different names. One source might record “Acme Solutions Inc” while another has “Acme Solutions Corp” or simply “Acme Solutions.” Location references, abbreviations, typos, and inconsistent use of legal suffixes all compound the problem. At scale, this kind of duplication quietly degrades any analysis built on top of the data. Figures get split across ghost duplicates. Trends become unreliable.

We needed a way to automatically detect when two differently spelled strings referred to the same real-world organization.

HYPOTHESIS

We believed that a combination of established string-matching algorithms and modern AI could reliably identify duplicate names without requiring human review. The variability in naming conventions seemed like a solvable pattern recognition problem.

APPROACH

We worked through four methods, roughly in order of sophistication.

Algorithmic exploration. We evaluated three standard fuzzy matching libraries: FuzzyWuzzy, Jaro-Winkler, and Levenshtein distance. Levenshtein counts the minimum number of single-character edits needed to turn one string into another. Jaro-Winkler does something slightly different: it gives more weight to characters at the front of a string, which turns out to matter a lot when names share a common root word. That made it the better performer for our use case. We settled on a 96% similarity threshold, which sounds precise but is really a judgment call. Drop it lower and you start matching things that shouldn’t match. Push it higher and you miss real duplicates that differ by a single character. Even at 96%, the error rate was high enough that we couldn’t rely on it alone.

Algorithmic tuning. We tried to make the matching smarter by making it more context-aware. This meant switching between Jaro-Winkler and Levenshtein depending on string length, and adjusting thresholds dynamically when common legal suffixes like “Inc,” “Corp,” or “LLC” were detected. The idea was that a name ending in “Corp” versus “Inc” shouldn’t tank the similarity score if everything else matches. This helped in specific cases but introduced new failure modes elsewhere. Tuning one parameter to fix one class of errors tends to break something adjacent.

AI integration. We brought in a large language model API to evaluate semantic similarity between names, hoping it could handle cases where character-level approaches failed. It did reduce false positives. But it overcorrected in the other direction: names that were related but organizationally distinct got collapsed together. Something like “Northgate Tech” and “Northgate Tech Canada” look semantically identical to a language model, but depending on context they may need to be treated as separate entities. The AI had no way to know which interpretation was correct without more context than we were giving it.

Normalization. We built preprocessing routines to clean strings before any comparison ran. This included uppercasing everything, stripping parenthetical content, running unicode normalization to handle accented or non-standard characters, and standardizing regional abbreviations so that “BC,” “B.C.,” and “British Columbia” all resolved to the same token. Normalization helped reduce noise but it couldn’t handle the deeper ambiguity in how organizations name themselves across different contexts and jurisdictions.

Because no single method was reliable enough on its own, we landed on a three-tier structure as the most practical way to manage the problem within our constraints. A static lookup table handles known duplicates identified and mapped manually. Algorithmic matching runs at import time to catch new duplicates as data comes in. And for everything the first two tiers miss, a manual review path lets a human explicitly resolve ambiguous cases. The automated tiers handled the clear majority of cases, meaningfully reducing manual effort, though performance at significantly higher data volumes would warrant revisiting the algorithmic tier.

TAKEAWAYS