Udacity part of Accenture logo

Automate Data Pipelines

Course

Schedule, monitor, and manage data workflows efficiently using tools like Apache Airflow. Build data pipelines by leveraging Airflow DAGs to organize tasks and utilize AWS resources such as S3 and Redshift to process and move data effectively between systems. Engage in hands-on projects to automate and maintain complex data pipelines, streamlining operations and improving data reliability. Gain expertise in workflow automation, data integration, and error handling, enabling you to construct efficient and scalable data pipelines in production environments. Ideal for data engineers and professionals aiming to advance their skills in managing and automating data workflows.

Schedule, monitor, and manage data workflows efficiently using tools like Apache Airflow. Build data pipelines by leveraging Airflow DAGs to organize tasks and utilize AWS resources such as S3 and Redshift to process and move data effectively between systems. Engage in hands-on projects to automate and maintain complex data pipelines, streamlining operations and improving data reliability. Gain expertise in workflow automation, data integration, and error handling, enabling you to construct efficient and scalable data pipelines in production environments. Ideal for data engineers and professionals aiming to advance their skills in managing and automating data workflows.

  • Intermediate

  • 2 weeks

  • Last Updated October 31, 2024

Skills you'll learn:

Apache AirflowData pipeline dags

Prerequisites:

Data modeling basicsIntermediate PythonDatabase fundamentalsIntermediate SQLAmazon web services basics

Intermediate

2 weeks

Last Updated October 31, 2024

Skills you'll learn:

Apache Airflow • Data pipeline dags • Data pipeline partitioning • Amazon s3

Prerequisites:

Data modeling basics • Intermediate Python • Database fundamentals

Course Lessons

Lesson 1

Introduction to Automating Data Pipelines

Welcome to Automating Data Pipelines. In this lesson, you'll be introduced to the topic, prerequisites for the course, and the environment and tools you'll be using to build data pipelines.

Lesson 2

Data Pipelines

In this lesson, you'll learn about the components of a data pipeline including Directed Acyclic Graphs (DAGs). You'll practice creating data pipelines with DAGs and Apache Airflow

Lesson 3

Airflow and AWS

This lesson creates connections between Airflow and AWS first by creating credentials, then copying S3 data, leveraging connections and hooks, and building S3 data to the Redshift DAG.

Lesson 4

Data Quality

Students will learn how to track data lineage and set up data pipeline schedules, partition data to optimize pipelines, investigating Data Quality issues, and write tests to ensure data quality.

Lesson 5

Production Data Pipelines

In this last lesson, students will learn how to build Pipelines with maintainability and reusability in mind. They will also learn about pipeline monitoring.

Lesson 6 • Project

Data Pipelines

Students work on a music streaming company’s data infrastructure by creating and automating a set of data pipelines with Airflow, monitoring and debugging production pipelines

Taught By The Best

Photo of Sean Murdock

Sean Murdock

Professor at Brigham Young University Idaho

Sean currently teaches cybersecurity and DevOps courses at Brigham Young University Idaho. He has been a software engineer for over 16 years. Some of the most exciting projects he has worked on involved data pipelines for DNA processing and vehicle telematics.

The Udacity Difference

Combine technology training for employees with industry experts, mentors, and projects, for critical thinking that pushes innovation. Our proven upskilling system goes after success—relentlessly.

Demonstrate proficiency with practical projects

Projects are based on real-world scenarios and challenges, allowing you to apply the skills you learn to practical situations, while giving you real hands-on experience.

  • Gain proven experience

  • Retain knowledge longer

  • Apply new skills immediately

Top-tier services to ensure learner success

Reviewers provide timely and constructive feedback on your project submissions, highlighting areas of improvement and offering practical tips to enhance your work.

  • Get help from subject matter experts

  • Learn industry best practices

  • Gain valuable insights and improve your skills

Enroll in Automate Data Pipelines. Choose the plan that works for you

All Access monthly

  • Cancel Anytime

  • Unlimited access to our top-rated courses

  • Hands-on projects with expert feedback

  • Personalized career coaching and interview prep

  • Program Certificates

Best Value

All Access bundle1

  • All the same great benefits as our monthly plan

  • The most cost-effective way to develop the skills you want

  1. 1Discount applies to the first 4 months of membership, after which plans are converted to month-to-month.

Your subscription also includes:

Udacity Accenture logo

Company

  • Facebook
  • Twitter
  • LinkedIn
  • Instagram

© 2011-2025 Udacity, Inc. "Nanodegree" is a registered trademark of Udacity. © 2011-2025 Udacity, Inc.
We use cookies and other data collection technologies to provide the best experience for our customers.