Data Engineering with AWS

Define data modeling and learn how to structure data effectively to create knowledge, including polyglot persistence and when to combine relational, document, and graph databases for different needs

Apply relational database concepts and techniques, including ACID properties, normalization, star schemas, and JSONB columns, to organize and query structured and unstructured data

Explore NoSQL data modeling, covering document databases, storage formats, OLTP vs. OLAP, hot vs. cold data, and how models connect logical design to physical storage

Explore graph databases, emphasizing relationships as first-class data using nodes and edges, including NoSQL concepts, Cypher/GQL querying, modeling, and why graphs excel at connected data

Explore how data warehouses unify scattered operational systems into a single source of truth, and compare OLTP vs. OLAP, dimensional modeling basics, and star and snowflake schemas.

Define fact grain, build fact and dimension tables with surrogate keys, and write Redshift DDL that encodes distribution styles, sort keys, and compression for performance.

Build ETL pipelines in Python that pull data from PostgreSQL, Cassandra, and Neo4j, then clean, conform, and derive dimensions for a consistent warehouse-ready dataset.

Stage data through S3, load it with the COPY command, optimize tables for parallel query performance, create materialized views, and run data quality validation checks.

Differentiate between the data warehouse, data lake, and data lakehouse paradigms for structured and unstructured data, and learn medallion architecture principles

Set up AWS S3 as a data lake foundation, ingest PostgreSQL CDC via DMS, catalog with Glue, query with Athena, and set up Lake Formation governance

Transform bronze (raw) data to silver (clean/enriched) data with PySpark, then aggregate to gold KPIs (business metrics). Optimize your Spark queries with partitioning and caching.

Create Iceberg tables and S3 Tables with AWS Glue, implement ACID MERGE for CDC upserts, and query historical snapshots (time travel)

Discover how Apache Airflow orchestrates data pipelines as code. Author DAGs and tasks, pass data with XCom, configure runtime parameters, apply Jinja templating, and build branching dependencies.

Orchestrate production pipelines with schedules, data intervals, and catchup. Apply single-responsibility design, debug with flatfile snapshots, trigger DAGs on asset events, and map tasks dynamically

Build end-to-end ETL and ELT pipelines that move data through S3 into Redshift. Use SQL operators, Jinja template inheritance, and data constraint checks, then deploy to production with Amazon MWAA.

Build a lakehouse on AWS with S3, Glue, Iceberg, and Athena. Automate ingestion, handle schema evolution with crawlers, and promote data through bronze, silver, and gold layers.