# Why Your Batch Checks Are Too Late

> Waiting for overnight batch jobs to catch silent schema drift or corrupted data is a recipe for broken downstream AI models. You need validation at the gate.

Published 2026-07-17 by Sovont. Tags: Data Engineering, AI Production.
Web page: https://sovont.com/blog/why-your-batch-checks-are-too-late/

The default pattern for data quality is passive. You ingest raw events, dump them into a lake, and run a scheduled dbt test or Great Expectations suite overnight. If something breaks, an on-call engineer gets paged at 9:00 AM the next morning. 

In a traditional BI dashboard world, this is annoying but survivable. In a production AI world, it's a catastrophic failure mode. 

By the time your batch checks run, corrupted, missing, or drifted data has already poisoned your vector database, triggered invalid real-time model inferences, and directly degraded user experience. You don't just have a data cleanup task; you have a poisoned system state, broken agent workflows, and frustrated clients.

Batch checks are a coroner's report. You need a bouncer.

## Validating at the Gate

Real-time validation means testing your data *before* it hits your primary storage or downstream consumers. If a stream payload violates your schema, schema-on-read isn't going to save you. You must validate inline.

Here is what that looks like in practice:

1. **Leverage the Message Broker:** If you are using Kafka, Redpanda, or Pulsar, enforce schema registries. Reject malformed payloads at the broker level. Do not let non-compliant producers write to the stream.
2. **Micro-validations in Stream Processing:** For lightweight transformations, execute structured schema and sanity checks (e.g., checking nulls, value ranges, or state consistency) using tools like faust-streaming, Flink, or simple serverless edge functions.
3. **The Dead Letter Queue (DLQ):** Do not raise a generic exception and crash your consumer. Route the invalid records directly to a DLQ, alert the source owner immediately, and keep the healthy data flowing. 

## What to Validate

You don't need complex statistical checks inline. Leave distribution drift detection for the batch process. At the gate, you check for structural integrity:
- **Missing or null required keys:** Especially user IDs, timestamps, or system triggers.
- **Type mismatch:** Expected floats receiving stringified garbage.
- **Payload size anomalies:** An embedding vector field that is suddenly empty or a chunk of text that is five orders of magnitude larger than normal.

Stop treating data validation as an overnight maintenance chore. In the era of live, agentic systems, data quality is an operational runtime requirement. If you aren't checking it in real-time, you are simply waiting to get burned.

Put a bouncer at the door.