Compiling Stochastic Gradient Descent on to Distributed GPUs

Project contacts:

• Gurbinder Gill (gill@cs.utexas.edu)
• Roshan Dathathri (roshan@cs.utexas.edu)

Project description:

The trend towards processor heterogeneity and distributed-memory
has significantly increased the complexity of parallel programming. In addition,
the mix of applications that need to run on parallel platforms today is very diverse,
and includes graph applications that typically have irregular memory accesses and
unpredictable control-flow. To simplify the programming of graph applications
on such platforms,
a compiler called Abelian [1] takes single-source, high-level specifications of graph analytics algorithms and automatically translates them into distributed,
heterogeneous implementations.
Input programs are written in Galois.
The Abelian compiler then generates IrGL [2] code and inserts the required
communication call to the Gluon [3] substrate. The generated code can then be run on
distributed CPUs or GPUs.

In typical graph applications like connected components,
the labels on nodes are scalars.
However, in some applications like
matrix completion using stochastic gradient descent (SGD),
the labels on nodes are vectors of a fixed-width (they represent an embedding of the node).
SGD is an important application
that is representative of several machine learning tasks. For example, it may be used
to recommend movies in streaming websites.
The Abelian compiler supports vector node labels only for
distributed CPU code generation.
In this project, you will add support in the Abelian compiler to generate
GPU code for vector node labels.
You will also need to add support in the IrGL runtime for vector node labels.
The layout of the vector impacts performance,
so your implementation needs to be optimized for GPUs.

Hardware:

• Bridges cluster P100 nodes: 4 machines each with 2 NVIDIA P100 GPUs.
• Bridges cluster V100 nodes: 2 machines each with 8 NVIDIA Volta V100 GPUs.
• Internal Machine located in UT called Tuxedo: 1 machine with 4 K80 GPUs and 2 GTX 1080 GPUs.

Project deliverables and deadlines:

• (Nov 1) A clear statement in English describing your project proposal.
• (Nov 8) An understanding of the project and source code for implementing your ideas,
  and a set up of the existing Abelian compiler (including llvm).
• (Dec 6) An extension to the Abelian compiler that compiles SGD implemented in Galois
  to the D-IrGL system, so that it can run on distributed GPUs.
• (Dec 6) A project report, written like an ACM conference paper, that summarizes the work you did.

Papers:

[1] Abelian: a compiler for graph analytics on distributed, heterogeneous platforms, Euro-Par 2018

https://www.cs.utexas.edu/~roshan/Abelian.pdf

[2] A compiler for throughput optimization of graph algorithms on GPUs, OOPSLA 2016

https://www.cs.rochester.edu/~sree/papers/sree-oopsla2016.pdf

[3] Gluon: a communication-optimizing substrate for distributed heterogeneous graph analytics, PLDI 2018

https://www.cs.utexas.edu/~roshan/Gluon.pdf