Sensor Fusion

Name: Sensor Fusion Nanodegree Program
Rating: 4.8 (175 reviews)

Nanodegree Program

The Sensor Fusion Engineer Nanodegree program consists of four courses that teach the fundamentals of sensor fusion and perception for self-driving cars. The program covers lidar, radar, camera, and Kalman filters, and includes lessons on working with real-world data, filtering, segmentation, clustering, and object tracking. In addition, students will work on a capstone project that involves building a complete sensor fusion pipeline for autonomous vehicles. Upon completing the program, graduates will have the skills and knowledge necessary to design and implement sensor fusion systems for self-driving cars.

Built in collaboration with

Mercedes-Benz

Advanced

3 months

Real-world Projects

Completion Certificate

Last Updated August 15, 2024

Skills you'll learn:

Radar • Sensor fusion • Digital cameras • Kalman filters

Prerequisites:

Basic calculus • Basic probability • Intermediate C++

Courses In This Program

Course 1 • 2 days

Welcome

Lesson 1

Welcome to the Sensor Fusion Engineer Nanodegree Program

Welcome to the Sensor Fusion Engineer Nanodegree Program! In this lesson, you will learn more about the structure of the program and meet the team.

Lesson 2

Introduction to Sensor Fusion

Lesson 3

Get Help with Your Account

What to do if you have questions about your account or general questions about the program.

Lesson 4

Getting Help

You are starting a challenging but rewarding journey! Take 5 minutes to read how to get help with projects and content.

Lesson 1

Welcome to the Sensor Fusion Engineer Nanodegree Program

Welcome to the Sensor Fusion Engineer Nanodegree Program! In this lesson, you will learn more about the structure of the program and meet the team.

Lesson 2

Introduction to Sensor Fusion

Lesson 3

Get Help with Your Account

What to do if you have questions about your account or general questions about the program.

Lesson 4

Getting Help

You are starting a challenging but rewarding journey! Take 5 minutes to read how to get help with projects and content.

Course 2 • 1 month

Lidar Obstacle Detection

Lesson 1

Introduction to Lidar and Point Clouds

Learn about lidar and point clouds. Use a simulation highway environment to explore lidar sensing and generate point clouds.

Lesson 2

Point Cloud Segmentation

In this lesson, you will be using Ransac with a plane model to segment point cloud data and separate it into points that are part of the road and points that are not.

Lesson 3

Clustering Obstacles

Perform Euclidean clustering, and learn how to build KD-Trees to use them to do efficient nearest neighbor search for clustering.

Lesson 4

Working with Real PCD

Take what you have learned in the previous lessons and apply it to real pcd being played back in a video.

Lesson 5 • Project

Lidar Obstacle Detection Project

In this lesson, students will submit the project that they have developed over the previous lessons.

Lesson 1

Introduction to Lidar and Point Clouds

Learn about lidar and point clouds. Use a simulation highway environment to explore lidar sensing and generate point clouds.

Lesson 2

Point Cloud Segmentation

In this lesson, you will be using Ransac with a plane model to segment point cloud data and separate it into points that are part of the road and points that are not.

Lesson 3

Clustering Obstacles

Perform Euclidean clustering, and learn how to build KD-Trees to use them to do efficient nearest neighbor search for clustering.

Lesson 4

Working with Real PCD

Take what you have learned in the previous lessons and apply it to real pcd being played back in a video.

Lesson 5 • Project

Lidar Obstacle Detection Project

In this lesson, students will submit the project that they have developed over the previous lessons.

Course 3 • 2 weeks

Camera

Lesson 1

Introduction

Introduction to the instructor, and the guest Timo Rehfeld, Mercedes-Benz R&D North America, and the course overview.

Lesson 2

Autonomous Vehicles and Computer Vision

Learn about the various levels of autonomy, some typical sensor sets, basics of camera technology, and an introduction into the OpenCV computer vision library.

Lesson 3

Engineering a Collision Detection System

Learn the collision detection basics, and estimating the TTC with Lidar and Camera.

Lesson 4

Tracking Image Features

Learn about the intensity gradient and filtering techniques; extract corners, infer features of an image, and track an object across multiple images.

Lesson 5 • Project

Camera Based 2D Feature Tracking

Lesson 6

Combining Camera and Lidar

Learn to improve the tracking process results by combining the Camera and Lidar output

Lesson 7 • Project

Track an Object in 3D Space

Lesson 8

Final Thoughts from Timo

Lesson 1

Introduction

Introduction to the instructor, and the guest Timo Rehfeld, Mercedes-Benz R&D North America, and the course overview.

Lesson 2

Autonomous Vehicles and Computer Vision

Learn about the various levels of autonomy, some typical sensor sets, basics of camera technology, and an introduction into the OpenCV computer vision library.

Lesson 3

Engineering a Collision Detection System

Learn the collision detection basics, and estimating the TTC with Lidar and Camera.

Lesson 4

Tracking Image Features

Learn about the intensity gradient and filtering techniques; extract corners, infer features of an image, and track an object across multiple images.

Lesson 5 • Project

Camera Based 2D Feature Tracking

Lesson 6

Combining Camera and Lidar

Learn to improve the tracking process results by combining the Camera and Lidar output

Lesson 7 • Project

Track an Object in 3D Space

Lesson 8

Final Thoughts from Timo

Course 4 • 1 week

Radar

Lesson 1

Introduction

Lesson 2

Radar Principles

Review Radar functionality, FMCW waveform, Radar Hardware, Schematic and the Radar Equation

Lesson 3

Range-Doppler Estimation

Estimate the range and velocity of the target using the FMCW radar

Lesson 4

Clutter, CFAR, AoA, and Clustering

Discuss - Clutter formation and then its removal using CFAR technique. After that

Lesson 5 • Project

Radar Target Generation and Detection

Lesson 1

Introduction

Lesson 2

Radar Principles

Review Radar functionality, FMCW waveform, Radar Hardware, Schematic and the Radar Equation

Lesson 3

Range-Doppler Estimation

Estimate the range and velocity of the target using the FMCW radar

Lesson 4

Clutter, CFAR, AoA, and Clustering

Discuss - Clutter formation and then its removal using CFAR technique. After that

Lesson 5 • Project

Radar Target Generation and Detection

Taught By The Best

David Silver

Staff Software Engineer at Kodiak Robotics

David works on Self-driving trucks at Kodiak. As a Staff Software Engineer, David writes planning, control, simulation and mapping software for self-driving trucks.

Stephen Welch

Instructor

Stephen is a Content Developer at Udacity and has worked on the C++ and Self-Driving Car Engineer Nanodegree programs. He started teaching and coding while completing a Ph.D. in mathematics, and has been passionate about engineering education ever since.

Abdullah Zaidi

Instructor

Abdullah holds his M.S from the University of Maryland and is an expert in the field of Radio Frequency Design and Digital Signal processing. After spending several years at Qualcomm, Abdullah joined Metawave, where he now leads Radar development for autonomous driving.

Andreas Haja

Instructor

Andreas Haja is an engineer, educator, and autonomous vehicle enthusiast. Andreas now works as an engineering professor in Germany. Previously, he developed computer vision algorithms and autonomous vehicle prototypes using C++.

Aaron Brown

Senior Software Engineer

Aaron has over 7 years in the autonomous vehicle field, starting as a Udacity content developer then instructor, focusing on Lidar in the Sensor Fusion and Self-Driving Car Nanodegree Program. For the last 5 years, he's been with Mercedes-Benz R&D North America, working in functional testing, Sensor Fusion, and Machine Learning driven by a passion for technological evolution and problem-solving.

Ratings & Reviews

Average Rating: 4.8 Stars

175 Reviews

Nikhil N.

March 26, 2023

The program is well thought out, the content is very close to the real world. The projects and examples provide a good balance of hands-on learning and theory.

Alejandro A.

November 22, 2022

Really Cool! The project was exciting! Doing it from scratch.

Andrei M.

October 14, 2022

The only in depth course on LiDAR, and it's in C++, so no more playing. Udacity has done a great job teaming up with Mercedes for this one.

Adan Yusseff D.

April 10, 2022

Very well structured, the tutorials are very good to learn. It's being a long time since i haven't code on C++ and this program guides you really well when you have some gaps in the knowledge like me. It is a hard program, but with a lot of interesting information from real life.

Rajnish B.

April 7, 2022

It's going great. I absolutely love the project and quizzes.

The Udacity Difference

Combine technology training for employees with industry experts, mentors, and projects, for critical thinking that pushes innovation. Our proven upskilling system goes after success—relentlessly.

Demonstrate proficiency with practical projects

Projects are based on real-world scenarios and challenges, allowing you to apply the skills you learn to practical situations, while giving you real hands-on experience.

Gain proven experience
Retain knowledge longer
Apply new skills immediately

Top-tier services to ensure learner success

Reviewers provide timely and constructive feedback on your project submissions, highlighting areas of improvement and offering practical tips to enhance your work.