ETL: Build the data pipeline that runs every night, survives a bad source, and hands the business a number it can trust at 2 a.m.
ETL: Build the data pipeline that runs every night, survives a bad source, and hands the business a number it can trust at 2 a.m. - Image 2

ETL: Build the data pipeline that runs every night, survives a bad source, and hands the business a number it can trust at 2 a.m.

$ 151.409,00

This guide shows you how to build data pipelines that a business can rely on, the kind that run every night, survive a bad source, and leave the warehouse correct instead of half-updated. It starts from the shape of the problem (extract from somewhere, reshape it, load it somewhere queryable) and builds the practices that separate a pile of cron scripts from a real pipeline: how to extract incrementally instead of reloading everything, how to transform in a way you can test and that fails loudly on bad data, the ELT-versus-ETL choice and when to push the work into the warehouse, idempotency and backfills so a rerun is safe and a gap is fillable, orchestration with Airflow (DAGs, schedules, retries, dependencies, and the gotchas), data quality checks that catch the broken load before the dashboard does, and the monitoring that tells you a pipeline silently stopped. The book does not tour every tool. It teaches you to design a pipeline that is incremental, idempotent, tested, and observable, so the numbers downstream are trustworthy and a 2 a.m. failure is a retry, not an outage. For engineers and analysts who move data and want it to arrive correct, on time, every time.

SKU: ETL-EN Category: Tags: , , ,

Description

Moving data sounds simple until it is your job: the source schema changes overnight, a job half-finishes and leaves the warehouse inconsistent, a silent type coercion corrupts a column, and nobody notices until a dashboard is wrong in a meeting. ETL is less about the extract, transform, and load verbs and more about doing them idempotently, on a schedule, with the failures handled, so the data a business decides on is actually correct.

This guide shows you how to build data pipelines that a business can rely on, the kind that run every night, survive a bad source, and leave the warehouse correct instead of half-updated. It starts from the shape of the problem (extract from somewhere, reshape it, load it somewhere queryable) and builds the practices that separate a pile of cron scripts from a real pipeline: how to extract incrementally instead of reloading everything, how to transform in a way you can test and that fails loudly on bad data, the ELT-versus-ETL choice and when to push the work into the warehouse, idempotency and backfills so a rerun is safe and a gap is fillable, orchestration with Airflow (DAGs, schedules, retries, dependencies, and the gotchas), data quality checks that catch the broken load before the dashboard does, and the monitoring that tells you a pipeline silently stopped. The book does not tour every tool. It teaches you to design a pipeline that is incremental, idempotent, tested, and observable, so the numbers downstream are trustworthy and a 2 a.m. failure is a retry, not an outage. For engineers and analysts who move data and want it to arrive correct, on time, every time.

Written for you if

This guide is for: engineers and analysts who move data between systems with brittle scripts and cron jobs, and want to build pipelines that run on a schedule, recover from failure, and produce numbers people can trust.

The full table of contents

  • Chapter 1: My dashboard was wrong in the meeting, and everyone saw it
  • Chapter 2: Everyone says “build a pipeline.” What am I actually building?
  • Chapter 3: Do I have to re-read the whole database every single night?
  • Chapter 4: The pipeline goes green, but my numbers come out wrong
  • Chapter 5: Should I clean the data before it lands, or after?
  • Chapter 6: I reran the job and now every row is in there twice
  • Chapter 7: Someone queried my table mid-load and got the wrong number
  • Chapter 8: I’m running every step by hand, and one night I’ll forget
  • Chapter 9: A team upstream changed a column and my pipeline never noticed
  • Chapter 10: Every task passed. So why are the numbers still wrong?
  • Chapter 11: How would I even know if my pipeline quietly stopped?
  • Chapter 12: The cloud bill keeps climbing, and finance is asking me why
  • Chapter 13: The night my pipeline survived without me, and I saw how far I’d come

Related products