Data engineering
The open-source scheduler that runs data pipelines in the right order, every time.
Airflow runs workflows defined as DAGs (directed acyclic graphs): extract from here, load there, run dbt, refresh the dashboard, and alert someone if a step fails.
It is the most widely used orchestrator and connects to almost every data tool. Alternatives include Dagster, Prefect, Azure Data Factory and the schedulers built into Databricks and Fabric.
Python files that define the steps and how they depend on each other.
Run tasks on time, in order and in parallel.
Prebuilt tasks for databases, clouds and APIs.
See runs, retry failures and read the logs.
Typical ranges, not quotes. Scope, data and team size move them most.
Transformations written as tested, version-controlled SQL: software engineering for analytics.
A durable, high-speed log of events that many systems can write to and read from in real time.
The lakehouse platform from the creators of Apache Spark: data engineering, analytics and AI in one place.
Compare IT service companies with sourced, labelled facts, or post your project and get matched.
