Most batch jobs are written for the night when everything works. That night takes care of itself. The design work is for the other one, when the source database falls over at 2 a.m., comes back at 3 with half its rows, and nobody is awake to notice.
Streaming gets the conference talks, but a lot of the data a business runs on still lands in scheduled batches, run by Azkaban or Airflow or something newer. Teams spend a surprising amount of time arguing about which scheduler. Most of that time is wasted. A badly designed job hurts you the same way on any of them.
I led Azkaban batch pipelines at LinkedIn, and this is the checklist I took from that work.
Give every stage its own step
I split a pipeline into five steps. Extract reads a bounded slice of the source, usually one time window. Transform cleans and joins it into the model the destination wants. Check tests the output before anyone can read it. Load writes it. Publish tells downstream jobs and people that new data is ready.
Each one is a separate node in the flow, with its own retries and its own log. A single script that does all five is quicker to write. It’s also the one you end up reading at 7 a.m., trying to work out whether the load ran before the crash or after.
The date is an argument
A job should never ask the clock what to process. It should be told. Pass the window in as a parameter and you get two things for free: rerunning yesterday’s job tomorrow produces yesterday’s data, and a backfill is the same job with a different date.
Jobs that work out “yesterday” from the current time look fine until one gets stuck in a queue past midnight. Then it quietly processes the wrong day and reports success.
Running it twice should change nothing
Any step can fail halfway and get run again, by a retry or by a person. So the load has to be idempotent: running it twice leaves the destination exactly as running it once would. Overwrite the window’s partition, or merge on a key. Blind appends are how duplicate rows end up in a revenue report.
Pair that with writing somewhere private first. Build the new data in staging, check it there, and only then make it visible in one atomic move. Readers see the old complete data or the new complete data. Never half of each.
And check the data itself. An exit code of zero tells you the process didn’t crash, and nothing about whether the output is any good. Before publishing, I compare row counts with recent runs, make sure key columns aren’t null, and reconcile totals against the source system’s own figures. A failed check stops the flow and leaves yesterday’s data where it was.
Here’s a nightly job that loads one day of orders into a reporting table. The date goes in on the command line, so a rerun is the same command:
run_orders_load --date 2026-06-02
The load replaces that one day (the job fills in the date; it’s written out here for readability):
-- 1. Build the day in a private staging table
DROP TABLE IF EXISTS orders_staging_20260602;
CREATE TABLE orders_staging_20260602 AS
SELECT * FROM source_orders WHERE order_date = '2026-06-02';
-- 2. Check it before anyone can see it
-- Fail the job if any of these are true:
-- row count is under 50% of the 7-day average
-- any order_id is NULL
-- SUM(amount) differs by more than 0.1% from an independent total,
-- such as the source system's own daily figure
-- 3. Swap it in, as one step
BEGIN;
DELETE FROM orders_report WHERE order_date = '2026-06-02';
INSERT INTO orders_report SELECT * FROM orders_staging_20260602;
COMMIT;
The DROP at the top is what makes a rerun safe, since a staging table left behind by a failed run would otherwise stop the next attempt cold. The staging table is named after its date, so two days backfilling at once can’t drop each other’s work. The transaction is what makes the swap one step. Without it, a dashboard that refreshes between the DELETE and the INSERT shows a day with no orders. If your warehouse can’t do multi-statement transactions, overwrite the partition instead.
A bad night, played out
Say the source database goes down at 2 a.m. Extract fails and retries. At 3 a.m. the database comes back, but replication is behind and it has less than half of yesterday’s rows. Extract and transform both succeed, and the check sees a row count far below the weekly average and stops the flow.
The report still shows complete data through the day before. Whoever is on call gets one alert naming the check that failed. In the morning, once the source has caught up, someone runs the same command again instead of patching a table by hand in a SQL console.
The failures that don’t page you
That scenario is the easy kind, because something broke loudly. The quiet problems do more damage.
Timing standing in for dependencies is one. “Run at 3 a.m. because the upstream job usually finishes by 2” works until the night it doesn’t, and then you process a partial input with full confidence. Depend on the upstream job’s completion signal, or on its output actually existing.
Late data is another. Records arrive after their day has closed, always. Decide up front how far back each run will look, and reprocess that whole range every time. Otherwise the number for a given day depends on when you happened to run. Since the load above replaces whole days, that’s just the same job over more dates.
Then there’s the job that hangs. It never fails, so nothing alerts. Give every flow a deadline for when its data should be ready, and page when it passes, failure or not. I’d rather be woken for a late job than hear about it from whoever opens the dashboard first.
Where batch fits next to events
People sometimes talk about events as the thing that finally retires batch. I don’t buy it. Events keep operational systems in step within seconds. Batch does the heavy, set-based work, like full reconciliations and history rebuilds, that would be silly to do one event at a time.
Events also go missing now and then. A consumer stays down longer than the topic keeps messages, or a retry gives up and drops one, and the stream has no way of knowing. The nightly reconciliation does. That’s reason enough to keep the boring scheduled job around long after the streaming system is live.