Real-world projects from industry experts
With real-world projects and immersive content built in partnership with top-tier companies, you’ll master the tech skills companies want.
Learn Spark & Data Lakes
Course
Learn more about the big data ecosystem and how to use Spark to work with massive datasets.
Estimated time
1 month
Enroll by
June 7, 2023
Get access to classroom immediately on enrollment
Skills acquired
AWS Glue, AWS Data Lakes, Apache Spark, Data Transformation
What You Will Learn
Spark and Data Lakes
1 month to completeBuild a data lake on AWS and a data catalog following the principles of data lakehouse architecture. Learn about the big data ecosystem and the power of Apache Spark for data wrangling and transformation. Work with AWS data tools and services to extract, load, process, query, and transform semi-structured data in data lakes.
Prerequisite knowledge
Intermediate Python, Intermediate SQL
Big Data Ecosystem, Data Lakes, & Spark
Identify what constitutes the big data ecosystem for data engineering. Explain the purpose and evolution of data lakes in the big data ecosystem. Compare the Spark framework with Hadoop framework. Identify when to use Spark and when not to use it and describe the features of lakehouse architecture.
Spark Essentials
Wrangle data with Spark and functional programming to scale across distributed systems. Process data with Spark DataFrames and Spark SQL. Process data in common formats such as CSV and JSON. Use the Spark RDDs API to wrangle data and transform and filter data with Spark.
Using Spark & Data Lakes in the AWS Cloud
Use distributed data storage with Amazon S3 and identify properties of AWS S3 data lakes. Identify service options for using Spark in AWS and configure AWS Glue. Create and run Spark Jobs with AWS Glue.
Ingesting & Organizing Data in Lakehouse Architecture on AWS
Use Spark with AWS Glue to run ELT processes on data of diverse sources, structures, and vintages in lakehouse architecture. Create a Glue Data Catalog and Glue Tables. Use AWS Athena for ad-hoc queries in a lakehouse. Leverage Glue for SQL AWS S3 queries and ELT. Ingest data into lakehouse zones. Transform and filter data into curated lakehouse zones with Spark and AWS Glue. Join and process data into lakehouse zones with Spark and AWS Glue.
Course Project: STEDI Human Balance Analytics
Act as a data engineer for the STEDI team to build a data lakehouse solution for sensor data that trains a machine learning model. Build an ELT (Extract, Load, Transform) pipeline for lakehouse architecture, load data from an AWS S3 data lake, process the data into analytics tables using Spark and AWS Glue, and load them back into lakehouse architecture.
All Our Courses Include
Real-time support
On demand help. Receive instant help with your learning directly in the classroom. Stay on track and get unstuck.
Workspaces
Validate your understanding of concepts learned by checking the output and quality of your code in real-time.
Flexible learning program
Tailor a learning plan that fits your busy life. Learn at your own pace and reach your personal goals on the schedule that works best for you.
Course offerings
Class content
- Real-world projects
- Project reviews
- Project feedback from experienced reviewers
Student services
- Student community
- Real-time support
Succeed with personalized services.
We provide services customized for your needs at every step of your learning journey to ensure your success.
- Experienced Project Reviewers
- Real-Time Support
Get timely feedback on your projects.
- Personalized feedback
- Unlimited submissions and feedback loops
- Practical tips and industry best practices
- Additional suggested resources to improve
1,400+ project reviewers
2.7M projects reviewed
88/100 reviewer rating
1.1 hours avg project review turnaround time
Learn with the best.
Learn with the best.
Sean Murdock
Professor at Brigham Young University IdahoSean currently teaches cybersecurity and DevOps courses at Brigham Young University Idaho. He has been a software engineer for over 16 years. Some of the most exciting projects he has worked on involved data pipelines for DNA processing and vehicle telematics.
Spark and Data Lakes
Get started today
Learn
Learn about the big data ecosystem and the power of Apache Spark for data wrangling and transformation.
Average Time
On average, successful students take 1 month to complete this program.
Benefits include
- Real-world projects from industry experts
- Real-time support
Program Details
Do I need to apply? What are the admission criteria?
No. This Course accepts all applicants regardless of experience and specific background.
What are the prerequisites for enrollment?
A well-prepared learner has experience in Relational database design, SQL, Basic dimensional modeling, Data Modeling Basics, Amazon Web Services Basics, and Python.
How is this course structured?
This course is comprised of content and curriculum to support one project. We estimate that students can complete the program in one month.
The project will be reviewed by the Udacity reviewer network and platform. Feedback will be provided and if you do not pass the project, you will be asked to resubmit the project until it passes.
How long is this course?
Access to this course runs for the length of time specified in the payment card above. If you do not graduate within that time period, you will continue learning with month to month payments. See the Terms of Use and FAQs for other policies regarding the terms of access to our programs.
Can I switch my start date? Can I get a refund?
Please see the Udacity Program Terms of Use and FAQs for policies on enrollment in our programs.
What software and versions will I need in this course?
There are no software and version requirements to complete this course. All coursework and projects can be completed via Student Workspaces in the Udacity online classroom. Udacity’s full technical requirements are listed here.