Free Data Engineer Roadmap: Build Reliable Pipelines from Linux to Spark Quantum Institute of Analytics (QIA) — original free learning checklist Reviewed 27 October 2026 Learning time is flexible. Software, API use and examinations may cost extra. [ ] 1. Understand the product: dependable data, not moving files Data engineering creates systems that make data available, understandable, and reliable for other people and applications. A pipeline that copies records successfully can still be wrong if it duplicates orders, loses updates, or changes the meaning of a reporting day. Begin by imagining a fictional distributor receiving sales files from branches in Pakistan and the Gulf. Analysts need consistent daily totals, finance needs traceability, and operations needs late files identified. These needs define the system more clearly than a list of fashionable technologies. Checkpoint: build a small example, explain its limitations, and record evidence. [ ] 2. Choose a local-first setup and realistic learning rhythm Linux, Python, PostgreSQL, dbt Core, Apache Airflow, and Apache Spark have freely accessible documentation and open-source components. Their surrounding services are not automatically free. Managed databases, hosted orchestration, storage, networking, and commercial development platforms may charge separately. Docker Desktop also has licensing terms that differ from simply using open-source container components; verify whether your situation qualifies before adopting it. Free documentation is a learning resource, not a guarantee of free software access, hosting, or unrestricted organizational use. Checkpoint: build a small example, explain its limitations, and record evidence. [ ] 3. Learn Linux and the command line as operational tools Study directories, relative and absolute paths, permissions, environment variables, processes, pipes, and exit codes. Practice listing files, inspecting text, searching logs, redirecting output, and checking disk usage. Understand standard input, standard output, and standard error before writing automation that depends on them. Learn how file ownership affects services and why granting broad write permissions is not a sensible universal fix. Practice on disposable directories so an incorrect deletion command does not damage valuable files. Checkpoint: build a small example, explain its limitations, and record evidence. [ ] 4. Write Python ingestion code that survives imperfect inputs Learn Python functions, modules, exceptions, context managers, iterators, datetime handling, and structured logging. Practice reading CSV and JSON, making HTTP requests, parsing responses, and writing files safely. Distinguish an HTTP failure from a valid response containing no records. Understand timeouts and bounded retries: endlessly retrying a bad request can overload a source without improving the outcome. Respect published API limits and terms, and never scrape personal information simply because a page is publicly reachable. Checkpoint: build a small example, explain its limitations, and record evidence. [ ] 5. Make SQL and PostgreSQL your strongest foundation Learn table creation, data types, constraints, joins, aggregation, common table expressions, window functions, and transactions. Use PostgreSQL to understand primary keys, foreign keys, unique constraints, and indexes in a real database. Choose numeric types deliberately for money and preserve currency as a separate field. Understand timestamp types and document how local times become reporting dates. Read query plans with EXPLAIN, and use execution analysis carefully because actually running a query can consume resources or modify data when combined with certain statements. Checkpoint: build a small example, explain its limitations, and record evidence. [ ] 6. Model analytical data around grain and history Study the distinction between operational normalization and analytical dimensional modeling. Define the grain of a fact table before adding dimensions. An order-level fact and an order-item fact answer different questions; mixing them creates duplicated measures. Learn surrogate keys, natural keys, date dimensions, and slowly changing dimensions. A current customer address may be unsuitable for reporting historical sales by the location that applied when the sale occurred, so decide whether the analysis needs historical attributes. Checkpoint: build a small example, explain its limitations, and record evidence. [ ] 7. Build batch ETL with idempotency and explicit state Learn the differences between extracting, transforming, and loading before debating whether ETL or ELT is universally better. Choose the order that fits your data, infrastructure, and controls. Start with full refreshes on tiny tables, then add incremental ingestion using a documented watermark or source change indicator. A maximum timestamp alone can miss late-arriving records, updates with old timestamps, or deletions. Decide how overlapping extraction windows and deduplication will handle those cases. Checkpoint: build a small example, explain its limitations, and record evidence. [ ] 8. Use Docker to make environments reproducible Learn images, containers, volumes, networks, ports, and image tags. Build a small image for your ingestion application and use a multi-service configuration only after each service works independently. Understand that a container is not a full virtual machine and that removing a container can remove data unless persistence is configured correctly. Use maintained base images, avoid unnecessary packages, and do not assume that a successful image build proves the application is secure. Checkpoint: build a small example, explain its limitations, and record evidence. [ ] 9. Learn dbt for tested, documented SQL transformations Introduce dbt after you can already express transformations in SQL. Use dbt Core and a compatible PostgreSQL adapter, checking current installation instructions because package and version requirements evolve. Learn sources, models, references, materializations, tests, documentation, and dependency graphs. Separate lightly cleaned staging models from business-facing marts. Keep business logic in readable models rather than hiding every decision inside an elaborate macro. A dependency graph is useful only when the meaning of its nodes is clear. Checkpoint: build a small example, explain its limitations, and record evidence. [ ] 10. Add Airflow only when you have a workflow to orchestrate Apache Airflow schedules and coordinates tasks; it is not a substitute for well-designed ingestion code or a tool for processing every row itself. Learn DAGs, tasks, dependencies, scheduling, logical dates, data intervals, retries, and backfills using the documentation for your installed version. Separate task definitions from heavy work at parse time. A scheduler needs to inspect workflow files repeatedly, so expensive network calls during parsing can make the system fragile before a task even starts. Checkpoint: build a small example, explain its limitations, and record evidence. [ ] 11. Learn Spark through the problems it actually solves Spark becomes useful when the data volume or computation requires capabilities beyond a straightforward local workflow. Do not introduce it merely to make a small project sound large. Begin with local-mode PySpark and learn DataFrames, explicit schemas, transformations, actions, lazy evaluation, partitions, joins, and aggregations. Understand why shuffling data across partitions can dominate runtime and why collecting a large distributed result into the driver can exhaust memory. Avoid claiming distributed performance from a tiny laptop demonstration. Checkpoint: build a small example, explain its limitations, and record evidence. [ ] 12. Explore cloud concepts without creating an uncontrolled bill Study object storage, identity and access management, compute, managed databases, regions, networking, encryption, and observability at a conceptual level first. Map your local components onto these services without deploying everything. Free tiers and trial credits change, differ by account and region, and can exclude important charges. Budget alerts are usually notifications rather than guaranteed hard spending caps. Data transfer, retained storage, snapshots, public addresses, and forgotten resources can continue to cost money after a demonstration stops running. Checkpoint: build a small example, explain its limitations, and record evidence. [ ] 13. Build reliability, privacy, and an incident response habit Observe row counts, freshness, rejected records, duplicate rates, task durations, and reconciliation differences. Logs should identify the run and failure stage without exposing customer details or credentials. Define alerts that lead to an action rather than sending every minor event to a human. Learn backups and restore testing: a backup file that has never been restored is weak evidence of recoverability. Document acceptable data delay and the limits of your small demonstration instead of inventing enterprise service guarantees. Checkpoint: build a small example, explain its limitations, and record evidence. [ ] 14. Assemble the capstone and assess operational independence Build one coherent capstone rather than disconnected tool demonstrations. Ingest synthetic branch orders, preserve raw files, load PostgreSQL, transform with dbt, orchestrate with Airflow, and publish a documented analytical mart. Use Docker where it improves reproducibility and include Spark as a separate justified experiment if the main workload is small. Provide architecture, setup instructions, data contracts, tests, sample outputs, recovery steps, privacy notes, and known limitations. Show a late file, duplicate input, schema failure, and successful rerun, not only the ideal path. Checkpoint: build a small example, explain its limitations, and record evidence. FREE RESOURCES Linux command-line foundations: https://ubuntu.com/tutorials/command-line-for-beginners PostgreSQL tutorial: https://www.postgresql.org/docs/current/tutorial.html Docker getting started: https://docs.docker.com/get-started/ dbt introduction: https://docs.getdbt.com/docs/introduction Airflow concepts and operation: https://airflow.apache.org/docs/ Spark SQL and DataFrames: https://spark.apache.org/docs/latest/sql-programming-guide.html AWS cost-management guidance: https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-managing-costs.html Data Engineering Zoomcamp: https://github.com/DataTalksClub/data-engineering-zoomcamp NYC taxi trip record downloads: https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page dbt Learn: https://learn.getdbt.com/ DIRECT VIDEO LESSONS Data Engineer Bootcamp (FREE 27+ Hour Course) - SQL, Python, Cloud, Bash, AI, Git & GitHub: https://www.youtube.com/watch?v=ol9_NnC9-cc Data Engineering Course for Beginners: https://www.youtube.com/watch?v=PHsC_t0j1dU How I would learn Data Engineering (if I could start over) – Built by a Data Engineer: https://www.youtube.com/watch?v=odZHmYgebbw dbt core & Airflow 101: Building data pipelines demystified: https://www.youtube.com/watch?v=d0wXnimEMUk SQL Tutorial - Full Database Course for Beginners: https://www.youtube.com/watch?v=HXV3zeQKqGY Learn Python - Full Course for Beginners [Tutorial]: https://www.youtube.com/watch?v=rfscVS0vtbw Docker Crash Course for Absolute Beginners [NEW]: https://www.youtube.com/watch?v=pg19Z8LL06w Docker Tutorial for Beginners [FULL COURSE in 3 Hours]: https://www.youtube.com/watch?v=3c-iBn73dDE Git and GitHub for Beginners - Crash Course: https://www.youtube.com/watch?v=RGOj5yH7evk How to write data pipelines with Apache Airflow® 3.0 - Timothy Davis: https://www.youtube.com/watch?v=v_jbmhVWBvM Apache Spark/PySpark Tutorial: What is Apache Spark? | Lecture 1: https://www.youtube.com/watch?v=XhjIJGsAMsQ Explore dbt's huge library of training courses to level up your data workflows: https://www.youtube.com/watch?v=I0iRPqNgPBI