We'll discuss the Bayesian statistical paradigm and Markov Chain Monte Carlo (MCMC) algorithms - the cornerstone of modern Bayesian computation. Scalable MCMC for big datasets and complex models is currently an open research question. Using GPUs provides a promising and largely unexplored avenue for accelerating these algorithms, but is nontrivial, because MCMC is inherently sequential and has traditionally been considered difficult to parallelize. We'll show how Gibbs sampling, a widely used MCMC algorithm, can be effectively parallelized on GPUs for a large class of exchangeable hierarchical Bayesian models. Participants will learn the mathematical and hardware/software challenges in bringing GPUs to the Bayesian community. Background in Bayesian statistics or MCMC is not assumed.