What Is a Data Pipeline? Components, Examples and How They Fail

Every dashboard you trust depends on something you never see. Somewhere, a process pulls orders from your store, payments from your processor and leads from your CRM, cleans them up and lands them in one place. When that process breaks at 2 a.m., Monday’s sales meeting starts with the wrong numbers.

That process is a data pipeline. This guide explains what a data pipeline is, walks through each stage with a concrete example and covers the failure-handling habits that separate reliable pipelines from fragile ones.

What a data pipeline is

AWS defines a data pipeline as “a series of processing steps to prepare enterprise data for analysis.” In plain terms, it’s an automated conveyor belt for data: it collects data from sources, changes it into a useful shape and delivers it somewhere people or software can use it.

A pipeline might be as simple as a nightly job copying a spreadsheet into a database, or as complex as a system processing millions of events per minute from apps and devices.

Don’t confuse data pipelines with CI/CD pipelines. Both automate a series of steps, but CI/CD pipelines move code to production, while data pipelines move data to analytics.

The 3 core stages: ingestion, transformation and delivery

1. Ingestion

Ingestion pulls data in from sources. AWS notes that sources can push data in or the pipeline can pull it, through APIs, webhooks or database replication, either in real time or on a schedule.

Common sources:

• Business apps: CRM, ecommerce platform, POS, accounting software
• Databases behind your own applications
• Files: CSV exports, spreadsheets, PDFs
• Event streams: website clicks, app events, IoT sensors

2. Transformation

Transformation turns raw data into something trustworthy and useful. According to AWS, this can include cleaning, reformatting, de-duplicating, validating, filtering and summarizing. Typical steps:

• Standardize dates, currencies and time zones
• Remove duplicates and test records
• Join data, such as matching orders to customers
• Apply business rules, like how you define an “active customer”
• Aggregate into daily or weekly summaries

Our guide on how to clean data covers many of these steps in detail.

3. Delivery

Delivery writes the result to a destination, sometimes called a sink: a data warehouse, a data lakehouse, a BI tool, a machine learning model or even back into an operational app.

The supporting layers: orchestration and monitoring

Two more pieces hold everything together:

• Orchestration schedules steps, runs them in the right order and handles dependencies, such as “don’t build the revenue table until both orders and refunds have loaded.”
• Monitoring checks that each run finished on time and that the data looks right.

Key takeaway: A pipeline isn’t finished when data flows once. It’s finished when you’ll know within minutes if it stops flowing or starts flowing wrong.

Batch vs streaming pipelines

AWS describes batch pipelines as running periodically, often off-peak, for high-volume work, and streaming pipelines as processing small, incremental packets continuously while tolerating lost or out-of-order data.

Most businesses should start with batch. Move a specific use case to streaming only when someone can act on fresher data and the value justifies the extra complexity.

ETL, ELT and other pipeline patterns

ETL is one kind of data pipeline, not a synonym for it. In ETL, data is transformed before loading. In ELT, raw data is loaded first and transformed inside the warehouse, a pattern that cloud warehouses made popular. Some pipelines don’t transform at all; they just replicate data. Our ETL vs ELT guide compares the 2 in depth.

Other patterns you’ll hear about:

• Reverse ETL: sends modeled data from the warehouse back into business tools, such as pushing a “high-value customer” flag into your CRM.
• Change data capture (CDC): copies only the rows that changed in a database, rather than reloading everything.

A real-world example: an online store’s sales pipeline

Imagine a store selling on its own website and a marketplace. The owner wants a daily dashboard of profit by product.

1. Ingest: at 1 a.m., connectors pull new orders from the ecommerce platform, payouts and fees from the payment processor, ad spend from ad platforms and product costs from a spreadsheet.
2. Land raw: everything is loaded as-is into raw tables in a cloud warehouse.
3. Transform: SQL models standardize currencies, match refunds to the original orders, attach product costs and allocate ad spend by product.
4. Test: checks confirm order totals match the platform’s own report and that no order is missing a cost.
5. Deliver: a daily_profit_by_product table feeds the BI dashboard. See how to build a business dashboard.
6. Alert: if any step fails or totals don’t match, the owner gets a message before 7 a.m.

Why data pipelines fail, and how to handle it

Pipelines fail for predictable reasons. Planning for them is what makes a pipeline production-ready.

Failure-handling tips that pay off

• Retry, but not forever. Orchestrators such as Apache Airflow retry failed tasks with a set count and delay by default. Set limits and alert when they’re exhausted.
• Set timeouts. Airflow, for example, lets you set an execution timeout so a stuck task fails instead of hanging indefinitely.
• Use a dead-letter destination. Google Cloud Pub/Sub can forward messages that repeatedly fail to a dead-letter topic after a configured number of delivery attempts, so 1 bad message doesn’t block the rest. The same idea works in batch with a quarantine table.
• Design for reruns. Use upserts or partition overwrites so you can safely replay a day without creating duplicates.
• Keep raw data. If you store what arrived, you can fix logic and reprocess history.
• Test the data, not just the code. A job that runs without errors can still deliver wrong numbers. For more on watching systems, see observability vs monitoring.
• Give every pipeline an owner. Alerts that go to everyone are ignored by everyone.

Where to start building your first pipeline

1. Pick 1 question worth answering, like daily profit or weekly churn.
2. List the sources needed and check how each one exports data: API, webhook, file or database access.
3. Choose a destination, usually a managed cloud warehouse for a first project.
4. Use managed connectors for common SaaS sources before writing custom code.
5. Load raw, then transform in SQL, keeping business rules in version control.
6. Add 3 basic tests: freshness, row counts and a total that reconciles with the source system.
7. Schedule and alert with your platform’s scheduler or an orchestrator, and send failures to a named owner.
8. Document what each table means and where it comes from.

As pipelines multiply, this becomes a full-time discipline. Our guide on how to become a data engineer explains the role.

Conclusion

A data pipeline is the automated path from raw data to useful answers: ingest, transform, deliver, with orchestration and monitoring around it. Building one that runs once is easy. Building one you can trust means planning for outages, schema changes, duplicates and silent errors from day one. Start with a single business question, use batch and managed tools, and add tests and alerts before you add more sources.

For related terms, browse our glossary.

Sources

• AWS: What is a data pipeline? — definition, components, batch vs streaming and how ETL relates to pipelines
• Apache Airflow documentation: Tasks — task retries and execution timeouts
• Google Cloud Pub/Sub: Handle message failures — dead-letter topics and maximum delivery attempts