Learn
What Is a Data Pipeline?
Any automated flow of data from a source to a destination, with some processing in between. Here's what that covers, and doesn't.
The basic shape
A data pipeline moves data from one or more sources, through some processing steps, to a destination — usually repeatedly, on a schedule or trigger, without someone running it by hand each time.
Pipeline, workflow, or job?
These terms overlap. "Workflow" often emphasizes the logical steps — what transforms what. "Pipeline" often emphasizes the end-to-end, typically automated flow. "Job" usually refers to a single scheduled execution. In practice, most tools use them loosely and interchangeably.
Building the steps vs. automating the run
DataQloo's Visual Designer lets you build the steps of a pipeline — connect a source, filter and summarize it, save the result — on a visual canvas today. Running that workflow automatically, on a recurring schedule, is on DataQloo's roadmap rather than available now; today, workflows run on demand.
See also: What Is ETL?, ETL vs. ELT
Frequently asked questions
ETL is one common pattern for a data pipeline, but "pipeline" is the broader term — it can describe any automated movement of data, including real-time streaming, not just batch extract-transform-load jobs.
See DataQloo’s Visual Designer for yourself.