Text and Data Mining: why entity resolution breaks silently when volume grows
Text and Data Mining: why entity resolution breaks silently when volume grows
Entity resolution is where most TDM pipelines quietly degrade. Not with an exception, not with an alert — just with noise that accumulates until the analysis stops being useful.
You launch a pipeline that extracts mentions of companies, people, or products from public sources. At 10,000 records a day, it works. At 500,000, you start seeing duplicates, ghosts, and collisions. "Apple" refers to the company, the fruit, and three startups. "J. García" resolves to four different individuals depending on context. Your aggregated signal is now polluted, and no monitoring dashboard told you it happened.
This is not a hypothetical. It is a structural problem in Text and Data Mining at scale, and it deserves a specific engineering response — not just a tuning pass on your NER model.
What entity resolution actually involves in a TDM context
Named Entity Recognition (NER) identifies candidate entities in text. Entity resolution is the step after: deciding whether "Microsoft Corp.", "Microsoft Corporation", "MSFT", and "the Redmond company" all point to the same node in your graph or dataset.
At low volume, a basic fuzzy match plus a curated dictionary covers most cases. At high volume, three things change simultaneously:
- Surface form diversity explodes. Abbreviations, misspellings, regional naming conventions, and informal references multiply faster than your dictionary can absorb them.
- Ambiguity compounds. The same surface form maps to different entities depending on the document's temporal, geographical, and topical context — context that your matcher may not have access to.
- Throughput pressure forces shortcuts. When you need to process millions of records within a latency window, you cut the expensive disambiguation steps first.
The result is a pipeline that technically completes — metrics green, no 5xx errors — but produces entity graphs riddled with silent errors.
The three failure modes you need to instrument
1. Fragmentation (one entity split into many)
"Amazon Web Services", "AWS", "Amazon's cloud unit" and "Amazon Cloud" end up as four separate nodes. Downstream, any aggregation query underestimates signal for each fragment. Trend analysis misses a spike because it is distributed across variants.
Detection signal: track the ratio of unique surface forms per canonical entity ID. A sudden increase in that ratio without a corresponding increase in legitimate new references is fragmentation.
2. Conflation (multiple entities merged into one)
"Santander" absorbs mentions of the city, the bank, and a lesser-known tech subsidiary. Your sentiment score for the financial entity is now contaminated by unrelated content.
Detection signal: measure intra-entity semantic coherence over time. If the average cosine distance between document embeddings attributed to a single entity ID climbs above your baseline, you have conflation.
3. Drift (resolution rules that were correct last month are wrong today)
A company rebrands. A public figure changes roles. A product name is retired and reassigned to a different SKU. Your resolution dictionary was built at T₀ and has not been updated since.
Detection signal: version your entity dictionaries explicitly. Log resolution decisions with the dictionary version that made them. When a new version is deployed, run a diff on resolution outcomes for the previous 48 hours of data. Any entity with more than a threshold percentage of reclassified mentions warrants manual review.
Architectural choices that prevent silent degradation
Separate extraction from resolution. Run NER and store raw surface forms with their source context. Apply entity resolution as a second, independently versioned pass. This lets you reprocess historical data when your resolution logic improves without re-ingesting from the source API.
Build entity resolution as a service, not a library. When resolution logic is embedded in your main processing code, updating it requires a full deployment. As a separate service with a clean contract, you can hot-swap resolution models, roll back independently, and expose a dedicated metrics endpoint.
Use context vectors, not just string matching. The surrounding text of a mention — publication domain, date, co-occurring entities — is the signal that breaks ambiguity. A mention of "Mercury" in a document co-occurring with "NASA" and "orbital trajectory" resolves differently than one co-occurring with "Queen" and "Freddie". Encode that context. Even a lightweight document embedding passed to your resolver dramatically reduces conflation rates.
Design for reprocessing from day one. Data pipelines that consume from APIs like FeedScale have the advantage of working with structured, consistently formatted signals. That structure is only valuable if your resolution layer can consume it deterministically and you can replay it. Idempotent processing is not a nice-to-have — it is what makes entity resolution fixable after the fact.
Measuring resolution quality without ground truth
Ground truth labeling at scale is expensive. You usually do not have it. There are three proxy metrics worth tracking continuously:
- Resolution rate: percentage of extracted entities that resolve to a known canonical ID versus falling into an "unknown" bucket. A sudden drop means your dictionary is lagging behind new surface forms.
- Singleton ratio: entities that appear only once, ever. Some singletons are legitimate (obscure references). A rising singleton ratio usually means fragmentation is accumulating.
- Cross-source agreement: if the same entity appears in two independent source streams and resolves to different canonical IDs, flag it. Disagreement across sources is a cheap signal of resolution inconsistency.
None of these require labeled data. They are computable from your own pipeline metadata.
Before you scale, ask this
Most teams hit entity resolution problems at volume because they validated accuracy on a small, clean sample and assumed it would hold. It does not. The relationship between volume and resolution accuracy is not linear — it degrades non-linearly as surface form diversity grows.
The right time to instrument these signals is before you scale the data intake, not after the analysis starts looking wrong. Fix the measurement layer first. Then scale the volume.
If your TDM pipeline processes signals from public sources at any meaningful throughput, entity resolution is not a post-processing concern. It is a first-class architectural component — and it needs the same rigor you would apply to any other stateful service in your system.