Most recent update: July 15, 2021

You may have read about some of the recent impressive advances in deep-learning applications, and are now eager to build your own neural networks. But which of the many libraries should you use? In this article, we present PyTorch and TensorFlow, the two most commonly used frameworks for deep-learning.

Read on to find out which one offers the best conditions for you to implement your projects.

Recap: Defining deep-learning

Let’s briefly recall what we mean by the term deep-learning. It refers to a group of algorithms from the large family of machine-learning models. Its name comes from the deep interconnected networks, called neural nets, which are the building blocks for deep-learning models.

The recent advances in computational resources and access to large data collections have facilitated a surge in deep-learning applications in many areas, from machine translation to automated driving. If you have a complex task that conventional machine-learning algorithms find hard to solve, chances are that a neural network will improve the performance — provided that you have the data to train it.

PyTorch vs. TensorFlow: The Key Facts

PyTorch and TensorFlow lead the list of the most popular frameworks in deep-learning. Let’s look at some key facts about the two libraries. 

PyTorch was released in 2016 by Facebook’s AI Research lab. As the name implies, it is primarily meant to be used in Python, but it has a C++ interface, too (so it’s available to other languages). Python enthusiasts love it for its imperative programming style, which they see as more “pythonic” than that of other, more declarative frameworks. Given its pythonic nature, PyTorch fits smoothly into the Python machine learning ecosystem.

TensorFlow, on the other hand, has interfaces in many programming languages. But the high-level Keras API for TensorFlow in Python has proven so successful with deep-learning practitioners that the newest TensorFlow version integrates it by default.

The Keras interface offers ready-made building blocks which significantly improve the speed at which even newcomers can implement deep-learning architectures. TensorFlow, which is named after the high-dimensional data frames that “flow” through a neural network, was developed at Google Brain and has been around a little bit longer than PyTorch, since 2015.

A Deeper Dive

Below, we get into the nitty-gritty of PyTorch and TensorFlow and cover their main differences.

Deployment

Let’s say you have successfully trained your neural network. How do you make it available to other people? The migration of your model to production is referred to as deployment. 

For years, TensorFlow has been clearly superior in this regard, as it offers native systems for deploying your models. TensorFlow Serving makes it easy to offer and update your trained models on the server-side. TensorFlow Lite, on the other hand, allows you to compress your trained model so that it can be used on mobile devices. TensorFlow has even gone a step further with the release of Tensorflow.js, which allows deep-learning models to be trained and deployed in JavaScript and Node.js.

Until recently, PyTorch did not have a comparable set of features. But in March 2020, Facebook announced the release of TorchServe, a PyTorch model-serving library. Much like its competitor TensorFlow Serving, TorchServe offers various features such as support for deploying multiple models and exposing RESTful endpoints for integration with your apps. In so doing, PyTorch has now eliminated the need for developers to write custom code.

Today, PyTorch boasts a rich selection of fully-fledged deployment tools. Apart from TorchServe, you can use TorchScript to enable models to be serialized and used in non-Python environments, and PyTorch Mobile to deploy deep-learning models on mobile devices. 

Domain

We have seen that both PyTorch and TensorFlow are increasingly looking alike, as each framework integrates new functionality implemented by the other. But when it comes to the libraries’ user bases, the divide between the two is still very much alive. 

PyTorch has long been the preferred deep-learning library for researchers, while TensorFlow is much more widely used in production.

PyTorch’s ease of use makes it convenient for fast, hacky solutions and smaller-scale models. But TensorFlow’s extensions for deployment on both servers and mobile devices, combined with the lack of Python overhead, make this the preferred option for companies that work with deep-learning models.

Distributed Training

Another distinguishing feature between PyTorch and TensorFlow lies in how readily PyTorch facilitates parallelism. In PyTorch, all it takes to enable training across multiple GPUs is to add a single line of code that wraps a model inside PyTorch’s DataParallel class. Of course, TensorFlow also allows for distributed training, but the solution requires writing more code.

So… Which Library Should You Use?

So, for your first deep-learning model, should you use PyTorch or TensorFlow? There is no definitive answer to this question. As a rule of thumb, if you’re already a Python programmer, or want to build a model just for yourself or for use in research, you can follow the general recommendation and use PyTorch. 

On the other hand, TensorFlow has long been geared towards production and many developers prefer it for this exact reason. PyTorch is certainly catching up in this regard, and a few years down the line we can expect PyTorch and TensorFlow to continue becoming increasingly more similar to each other. 

Developers for both libraries have continually been integrating popular features from their competitor, resulting in a process of gradual convergence. For instance, TensorBoard, the tool that allows you to observe the behavior of your training parameters over time, used to be a feature exclusive only to TensorFlow. Now, PyTorch also offers native support for TensorBoard. Additionally, PyTorch recently released PyTorch Lightning, a high-level interface to PyTorch — just like Keras is to TensorFlow.

In either case, we encourage you to try and understand as much as possible about your neural networks regardless of which framework you choose. In the end, it will just be a tool to help you build your models. Happy learning!

Conclusion

Both TensorFlow and PyTorch have their advantages as starting platforms to get into neural network programming. Traditionally, researchers and Python enthusiasts have preferred PyTorch, while TensorFlow has long been the favored option for building large scale deep-learning models for use in production.

However, the latest releases have seen the two libraries converge towards a more similar profile. As long as you stick to either TensorFlow or PyTorch as your deep-learning framework, you can’t go wrong.

Looking to get started with deep-learning? Enroll in our AI Nanodegree.

Stephen Welch
Stephen Welch
Stephen is a Content Developer at Udacity and has built the C++ and Self-Driving Car Engineer Nanodegree programs. He started teaching and coding while completing a Ph.D. in mathematics, and has been passionate about engineering education ever since.