Late Data Is Not an Edge Case
Treating late-arriving data as an exception is how you get metrics that silently restate themselves for days after the fact. Design for lateness upfront or debug it forever.
Someone files a ticket. The numbers in yesterday’s report changed. Not dramatically — just a few percent. You check the pipeline, nothing failed. You check the source, nothing was rerun. What happened is that late data arrived, got processed, and quietly updated your “final” metrics. Nobody told you. Nothing broke. The pipeline did exactly what it was built to do.
That’s the problem.
Late-arriving data is not an anomaly in production data systems — it’s a baseline condition. Mobile events buffer during connectivity gaps. Third-party platforms batch their exports. IoT devices sync hours after the fact. Distributed systems don’t guarantee delivery order. If your pipeline was designed assuming data arrives promptly and in order, you have a correctness problem dressed up as normal operation.
Why it’s hard to take seriously
Late data doesn’t cause errors. It doesn’t fail your CI checks. It doesn’t alert. It just arrives quietly, gets processed if your system accepts late records, and depending on how you’ve built things, either gets dropped silently or triggers a restatement you may or may not notice.
Most teams treat it as an edge case because most of the time, the lateness is small and the downstream impact is marginal. So it never gets properly designed for. Then the edge case becomes the norm — a vendor slips, a network blip stretches the latency window, a backfill drops events out of order — and you’re debugging a metric drift problem with no tooling to understand it.
The design decisions that matter
Define your lateness tolerance explicitly. For every data source, decide: what’s the maximum time between event occurrence and expected ingestion? This isn’t a guess — it’s a contract. Events outside that window are late. Events inside are on time. If you can’t define this, you can’t build reliable logic around it.
Use event time, not processing time, for business metrics. Processing time is when your pipeline saw the record. Event time is when the thing actually happened. For any metric that maps to real-world behavior — conversions, sessions, errors, revenue — you want event time. Using processing time is faster and simpler; it’s also wrong in ways that compound as lateness increases.
Decide what late records do. There are two honest choices: drop them with a clear policy (records older than N hours are excluded, documented and known), or restate the affected windows (accept the late record, mark the downstream metrics as updated). Both are valid. The invalid choice is to neither drop cleanly nor restate explicitly — letting late data quietly influence “final” numbers with no audit trail.
Build watermarks or equivalent mechanisms. In streaming systems, a watermark is a claim: “all events up to time T have arrived.” It lets the system close a window and emit results without waiting forever. In batch systems, the equivalent is a known cutoff with explicit late-data handling logic. Either way, you need a mechanism for declaring completeness. Without one, every window is always provisional and you’ll never know when to trust it.
Instrument lateness. Measure the distribution of event-time lag for each source. Alert when it shifts. A source that’s typically 30 seconds late becoming 4 hours late is signal — something changed upstream. You want to know that before your users notice the gap in the dashboard.
Late data isn’t a failure mode. It’s the default mode. Build for it intentionally and your metrics mean something. Ignore it and you’ll spend your time explaining to stakeholders why last Tuesday’s numbers keep changing.
Design the lateness window before you design the pipeline. Everything else follows from that.