Data Architectures: how to handle late-arriving data without poisoning your analytics pipeline
Data Architectures: how to handle late-arriving data without poisoning your analytics pipeline
Late-arriving data is one of the most underestimated failure modes in production pipelines. You ship a working system. Metrics look stable. Then, three days later, a batch of signals from 48 hours ago lands in your ingestion layer — and your aggregations are now wrong. Not broken. Wrong. That distinction matters enormously when downstream consumers are making decisions based on those numbers.
The problem is not rare. It is structural. Public data sources — feeds, APIs, distributed collectors — do not deliver events in the order they happen. Network delays, source-side buffering, retry queues, and indexing lag all introduce temporal drift between the moment something occurs and the moment your system sees it. If your architecture assumes arrival time equals event time, you are building on a false premise.
This post is about how to fix that — architecturally, not as a patch.
Why arrival time and event time are not the same thing
Every event has two timestamps: when it happened, and when your system received it. In tightly controlled internal systems, the gap between the two is milliseconds. In pipelines consuming external data APIs — especially those ingesting signals from the public internet — that gap can stretch to hours or days.
Sources go temporarily unavailable. Retry logic fills queues that flush later. Upstream indexing pipelines have their own lag. A signal published at 14:00 UTC might not reach your aggregation layer until 02:00 UTC the next day.
If your pipeline uses arrival time as a proxy for event time — because it is simpler — every aggregation window you compute becomes an approximation at best, a lie at worst. Dashboards show a quiet Monday when Monday was actually noisy. Trend analysis misses spikes. Sentiment curves lag real-world events by 24 hours without any visible indicator that they are stale.
The fix starts with storing both timestamps on every record from the moment of ingestion. This is not optional. Without both, you cannot reason about drift after the fact.
The watermark model: defining how late is too late
Stream processing frameworks introduced the concept of watermarks precisely to solve this problem. A watermark is a threshold that declares: "events older than T are considered complete; anything arriving after this threshold is late."
In practice, you configure a maximum tolerated lateness — say, 6 hours — and your windowing logic holds aggregation results open until that window expires. Signals arriving within the tolerance get folded into the correct window. Signals arriving after it are routed to a side output for separate handling.
This sounds clean in theory. The hard part is calibrating T. Set it too tight and you drop legitimate late arrivals. Set it too loose and your pipeline holds state indefinitely, memory pressure climbs, and downstream consumers wait too long for results.
Calibrating watermark tolerance requires empirical data about your specific sources. You need to measure the 95th and 99th percentile of arrival delay per source, per time-of-day. This is not a one-time exercise — source behavior changes, and your watermarks should adapt.
A practical starting point: instrument your ingestion layer to log the delta between event_time and ingestion_time for every record. After two weeks of production traffic, you have a real distribution. Set your watermarks based on that, not on a guess.
Reprocessing windows: the safety net you will need
Even with well-calibrated watermarks, some events will arrive genuinely late — beyond your tolerance threshold. Your architecture needs a defined answer to the question: what happens to them?
There are three patterns, each with real trade-offs.
Discard and log. Late arrivals beyond the tolerance are dropped. A dead-letter log captures them for audit. Simple to implement, but your aggregations have known gaps. Acceptable when late-arriving data volume is small and the use case tolerates approximation.
Recompute on trigger. When a late record arrives, trigger a recomputation of the affected window. The updated result overwrites the previous one. This is correct but expensive if late arrivals are frequent and windows are large. It also requires downstream consumers to handle result updates — which most systems are not designed for by default.
Immutable append with correction records. Never overwrite. Instead, emit a correction record that carries the delta between the old aggregation and the new one. Downstream consumers apply corrections incrementally. This is the most operationally complex pattern, but it preserves full audit history and avoids the consistency problems of overwriting.
For most B2B analytics pipelines consuming external data APIs, the recompute-on-trigger pattern is the pragmatic choice — provided you gate it with a secondary watermark that defines a hard cutoff beyond which no recomputation happens, regardless of arrivals.
Schema design decisions that make late data tractable
Architecture decisions at the schema level either make late-data handling feasible or turn it into a maintenance nightmare.
Store event_time, ingestion_time, and processing_time as separate, indexed columns on every fact table. Never conflate them into a single timestamp field. This seems obvious but is routinely skipped in early iterations and later becomes a migration project.
Partition your fact tables by event_time, not ingestion_time. This makes recomputation queries scan only the relevant partition rather than the entire table. On large datasets, the difference between a 30-second query and a 3-minute query is often just this one partitioning decision.
If you are consuming signals through an API like FeedScale, check whether the API exposes both the original publication timestamp and the indexing timestamp in the response payload. If it does, use both. If it only exposes one, treat every aggregation as an estimate and document that assumption explicitly in your data catalog.
Monitoring: the metric your team is probably not tracking
Most pipeline monitoring covers throughput, error rates, and latency of processing. Few teams track event-time lag distribution as a first-class metric.
Add a gauge that continuously measures the 50th, 95th, and 99th percentile of (ingestion_time - event_time) across all active sources. Alert when the 95th percentile exceeds your watermark tolerance. That alert means your watermark is no longer calibrated to reality — and your aggregations are silently wrong.
This metric costs almost nothing to compute. It is already derivable from the timestamps you are storing. The only reason most teams do not have it is that they did not think about late-arriving data when they designed the pipeline. Now you have no excuse.
Build for the data you will actually receive, not the data you wish you had
External data pipelines never deliver perfect, ordered, low-latency streams. The public internet does not work that way. Sources buffer, retry, and lag. Your architecture needs to treat temporal disorder as the baseline condition, not an edge case to handle later.
The teams that get this right early — separating event time from arrival time, calibrating watermarks empirically, designing schemas that make recomputation cheap — spend their time iterating on analysis. The teams that skip it spend their time debugging why last Tuesday's numbers changed on Friday.
Design for the data you will actually receive. The rest follows.