A data engineering team optimized a Python batch job that processes daily transaction records. The job previously read all records into memory at once; it now uses a streaming/chunked approach (reading 1000 records at a time). The job runs nightly and the team wants a regression scout before the next scheduled run. Write findings to report.md.

=============== FILE: inputs/batch-job.md ===============

Transaction Batch Job Overview

Change

• jobs/process_transactions.py — main job file (CHANGED: added chunked reading)
• Reads 1000 rows at a time using OFFSET/LIMIT instead of SELECT *
• Downstream: writes results to reports/daily_summary.json and inserts to processed_transactions table

Job Pipeline

1. Read transactions from transactions table
2. Apply enrichment from merchant_lookup table (JOIN)
3. Calculate per-merchant summaries
4. Write summary to reports/daily_summary.json
5. Insert enriched records to processed_transactions (upsert on transaction_id)

Known Edge Cases

• Duplicate transaction_id values exist in source (job uses upsert)
• Some transactions have NULL merchant_id (handled by left join, produces "unknown" merchant)
• Report file is overwritten atomically at job end
• Database uses READ COMMITTED isolation; other processes may write during job run

Performance Notes

• Old approach: loaded ~500k rows into memory, occasionally hit 8GB limit
• New approach: processes in chunks, but OFFSET performance degrades for large offsets
• Nightly window: job must complete within 2-hour maintenance window =============== END FILE ===============