No
Yes
View More
View Less
Working...
Close
OK
Cancel
Confirm
System Message
Delete
Schedule
An unknown error has occurred and your request could not be completed. Please contact support.
Scheduled
Wait Listed
Personal Calendar
Speaking
Conference Event
Meeting
Interest
Schedule TBD
Conflict Found
This session is already scheduled at another time. Would you like to...
Loading...
Please enter a maximum of {0} characters.
Please enter a maximum of {0} words.
must be 50 characters or less.
must be 40 characters or less.
Session Summary
We were unable to load the map image.
This has not yet been assigned to a map.
Search Catalog
Reply
Replies ()
Search
New Post
Microblog
Microblog Thread
Post Reply
Post
Your session timed out.
This web page is not optimized for viewing on a mobile device. Visit this site in a desktop browser to access the full set of features.
2017 GTC San Jose

S7724 - Tofu: Parallelizing Deep Learning Using Automatic Tensor Tiling

Session Speakers
Session Description

We'll cover how to automatically select the best parallelism for a deep learning algorithm. Current deep learning systems like Tensorflow and MXNet focus on only one specific parallelization strategy, data parallelism, which requires large training batch sizes to scale. An alternative approach, model parallelism, does not have such requirements but is inefficient when model parameter size is large. Choosing the right parallelism is tedious for users because it requires extensive analysis of the whole program. Therefore, we propose Tofu that could automatically parallelize a deep learning program. We first cast the problem of finding the best parallelization strategy as the problem of finding the best tiling to partition the computation with the least overall communication. We propose an algorithm that is provably optimal and the resulting optimal parallelization solution is a hybrid of data and model parallelism. Our system, called Tofu, can automatically transform the dataflow graph captured by an existing deep learning system frontend into a parallel dataflow graph based on the optimal tiling it has found.


Additional Session Information
Advanced
Talk
Deep Learning and AI HPC and Supercomputing
Higher Education / Research
25 minutes
Session Schedule