Knowledge hub · 5 guides

Data engineering

Building the pipes and platforms that collect, clean and store data so analytics and AI can use it.

What it is

Data engineering moves data from where it is created (ERP, CRM, apps, sensors) into one platform, cleans it, and shapes it into tables people and models can trust. It is the foundation under every dashboard and every AI project.

The modern pattern is the cloud lakehouse: cheap storage for raw data, a fast query engine on top, and transformations written as code that is tested and version-controlled like any other software.

Explainer

How data flows through a modern platform

Sources
- ERP & CRM
- Apps & databases
- Files & SaaS
- Events & IoT
Ingest
Batch loads or real-time streams, scheduled and monitored.
Lakehouse
Bronze · Raw, as it arrived
Silver · Cleaned and joined
Gold · Business-ready
Serve
- Dashboards
- AI & ML
- Apps & APIs

The “medallion” pattern: land data as it arrives (bronze), clean and join it (silver), then publish business-ready tables (gold) for dashboards, AI and apps.

Guides

Guides in Data engineering

Databricks

The lakehouse platform from the creators of Apache Spark: data engineering, analytics and AI in one place.

High data volumes and complex transformations
Streaming and machine learning alongside BI
Keeping data in open formats in your own storage

Typical project: 8–16 weeksSee vendors

Snowflake

The cloud data platform that made warehousing simple: storage and compute apart, almost no tuning.

SQL-first analytics teams
Fast setup with little administration
Sharing data with customers and partners

Typical project: 6–12 weeksSee vendors

Google BigQuery

Google's serverless data warehouse: no clusters to manage, just SQL.

Organisations on Google Cloud
Digital, marketing and web analytics
Teams that want zero infrastructure

Typical project: 6–14 weeksSee vendors

dbt

Transformations written as tested, version-controlled SQL: software engineering for analytics.

Teams replacing messy SQL scripts and stored procedures
Warehouses with many analysts contributing
Organisations that want tested, documented data

Typical project: 4–8 weeksSee vendors

Apache Airflow

The open-source scheduler that runs data pipelines in the right order, every time.

Many pipelines across different tools
Teams comfortable with Python
Complex dependencies and backfills

Typical project: 3–8 weeksSee vendors

At a glance

Guide	Best fit	Pricing	Typical project
Databricks	Large-scale data engineering, streaming and ML on one platform	Pay per use in Databricks Units (DBUs), plus your cloud's compute and storage	8–16 weeks
Snowflake	SQL-first analytics with little administration, and data sharing	Pay per use: credits for compute (billed per second) plus storage per terabyte	6–12 weeks
BigQuery	Serverless analytics on Google Cloud, especially marketing and web data	On-demand per terabyte scanned, or reserved capacity (slots); storage billed separately	6–14 weeks
dbt	A reliable, documented transformation layer on any warehouse	dbt Core is free and open source; dbt Labs' hosted platform is priced per seat and usage	4–8 weeks
Airflow	Orchestrating many pipelines across many tools	Free software; managed services charge per environment and size	3–8 weeks

Glossary

Key terms

Data warehouse: A database designed for analysis, holding structured, modelled tables.
Data lake: Cheap storage for raw files of any type.
Lakehouse: Lake storage with warehouse features: tables, transactions and fast SQL.
ETL / ELT: Extract, transform, load. ELT loads first and transforms inside the platform.
Orchestration: Scheduling pipeline steps in the right order, with retries and alerts.
Data governance: Who owns data, what it means, who can see it, and how good it is.

Also worth reading

Other areas

Find Data engineering partners

IT service companies with these skills on their profile, every fact labelled with its source.

See vendors Post a project