Entity Resolution

What is entity resolution?

Real data rarely comes with perfect shared identifiers. The same physician appears as "Jonathan Smith MD" in one file and "Smith, Jon" in another, the same hospital carries different names in claims and accreditation records, and the same drug appears under several product codes. Entity resolution decides which of these records belong together.

The problem has two sides. Deduplication finds repeated records inside one dataset, and record linkage connects records across datasets. Both produce clusters of records that each represent one real entity.

How entity resolution works

Most pipelines follow the same stages:

• Standardize: normalize names, credentials, addresses and phone numbers into consistent formats.
• Block: group records into candidate sets, for example by state and last name, so the system does not compare every record with every other record.
• Compare: score candidate pairs on fields such as name similarity, address distance and matching identifiers.
• Decide: accept, reject or send pairs to review. Deterministic rules require exact agreement on chosen fields, while probabilistic methods, rooted in the Fellegi-Sunter model published in 1969, weigh how strongly each agreement or disagreement suggests a match.
• Cluster: merge accepted pairs into entities and give each a stable internal identifier.

Entity resolution in healthcare data

Healthcare has strong identifiers for some entities and none for others. Providers have the National Provider Identifier (NPI), Medicare-certified facilities have the CMS Certification Number (CCN), and drug products have the National Drug Code (NDC), which RxNorm groups into clinical drug concepts. These identifiers make many joins exact, but many sources, such as recall notices, clinical trial site lists and state license files, lack them.

Organizations are especially hard. One health system may bill under many NPIs and Taxpayer Identification Numbers, and one CCN can cover several campuses. The United States has never adopted a national patient identifier, and Congress has repeatedly blocked funding to develop one, so patient matching relies on names, birth dates and addresses.

Why entity resolution matters

Every cross-dataset analysis depends on it. Linking prescribers to their procedure volume, facilities to their ownership records, or drug products to recalls and shortages all fails quietly when matches are wrong.

Errors run in both directions. False matches merge different entities and inflate counts, while missed matches split one entity into several and hide its true size. Reporting match rates and reviewing a sample of matches is the practical safeguard for targeting lists, market sizing and compliance screening.