Skills you'll learn:

Automate Data Pipelines
Course
Schedule, monitor, and manage data workflows efficiently using tools like Apache Airflow. Build data pipelines by leveraging Airflow DAGs to organize tasks and utilize AWS resources such as S3 and Redshift to process and move data effectively between systems. Engage in hands-on projects to automate and maintain complex data pipelines, streamlining operations and improving data reliability. Gain expertise in workflow automation, data integration, and error handling, enabling you to construct efficient and scalable data pipelines in production environments. Ideal for data engineers and professionals aiming to advance their skills in managing and automating data workflows.
Schedule, monitor, and manage data workflows efficiently using tools like Apache Airflow. Build data pipelines by leveraging Airflow DAGs to organize tasks and utilize AWS resources such as S3 and Redshift to process and move data effectively between systems. Engage in hands-on projects to automate and maintain complex data pipelines, streamlining operations and improving data reliability. Gain expertise in workflow automation, data integration, and error handling, enabling you to construct efficient and scalable data pipelines in production environments. Ideal for data engineers and professionals aiming to advance their skills in managing and automating data workflows.
Intermediate
2 weeks
Last Updated October 31, 2024
Prerequisites:
Intermediate
2 weeks
Last Updated October 31, 2024
Skills you'll learn:
Prerequisites:
Course Lessons
Lesson 1
Introduction to Automating Data Pipelines
Welcome to Automating Data Pipelines. In this lesson, you'll be introduced to the topic, prerequisites for the course, and the environment and tools you'll be using to build data pipelines.
Lesson 2
Data Pipelines
In this lesson, you'll learn about the components of a data pipeline including Directed Acyclic Graphs (DAGs). You'll practice creating data pipelines with DAGs and Apache Airflow
Lesson 3
Airflow and AWS
This lesson creates connections between Airflow and AWS first by creating credentials, then copying S3 data, leveraging connections and hooks, and building S3 data to the Redshift DAG.
Lesson 4
Data Quality
Students will learn how to track data lineage and set up data pipeline schedules, partition data to optimize pipelines, investigating Data Quality issues, and write tests to ensure data quality.
Lesson 5
Production Data Pipelines
In this last lesson, students will learn how to build Pipelines with maintainability and reusability in mind. They will also learn about pipeline monitoring.
Lesson 6 • Project
Data Pipelines
Students work on a music streaming company’s data infrastructure by creating and automating a set of data pipelines with Airflow, monitoring and debugging production pipelines
Taught By The Best

Sean Murdock
Professor at Brigham Young University Idaho
Sean currently teaches cybersecurity and DevOps courses at Brigham Young University Idaho. He has been a software engineer for over 16 years. Some of the most exciting projects he has worked on involved data pipelines for DNA processing and vehicle telematics.
The Udacity Difference
Combine technology training for employees with industry experts, mentors, and projects, for critical thinking that pushes innovation. Our proven upskilling system goes after success—relentlessly.

Demonstrate proficiency with practical projects
Projects are based on real-world scenarios and challenges, allowing you to apply the skills you learn to practical situations, while giving you real hands-on experience.
Gain proven experience
Retain knowledge longer
Apply new skills immediately

Top-tier services to ensure learner success
Reviewers provide timely and constructive feedback on your project submissions, highlighting areas of improvement and offering practical tips to enhance your work.
Get help from subject matter experts
Learn industry best practices
Gain valuable insights and improve your skills

Enroll in Automate Data Pipelines. Choose the plan that works for you
All Access monthly
Cancel Anytime
Unlimited access to our top-rated courses
Hands-on projects with expert feedback
Personalized career coaching and interview prep
Program Certificates
Best Value
All Access bundle1
All the same great benefits as our monthly plan
The most cost-effective way to develop the skills you want
- 1Discount applies to the first 4 months of membership, after which plans are converted to month-to-month.
Your subscription also includes:
Your subscription also includes:

4 weeks
Intermediate

3 weeks
Intermediate

4 weeks
Intermediate

3 hours
Intermediate

3 weeks
Intermediate

4 weeks
Advanced

4 weeks
Intermediate

2 weeks
Advanced

4 weeks
Advanced

4 weeks
Advanced

1 week
Fluency

2 weeks
Intermediate

(4)
2 months
Advanced

4 weeks
Advanced

3 weeks
Intermediate

2 months
Beginner