Recurrent neural networks are widely used to solve a variety of problems. As the quantity of data and the amount of available compute have increased, model sizes have also grown. We'll describe an approach to reduce the parameter count of RNNs using a simple pruning schedule without increasing the training time. The reduction in parameters achieves two goals. It helps reduce the size of the neural network, allowing it to be deployed on mobile and embedded devices. It also helps speed up evaluation time for inference. We'll demonstrate how this technique works for vanilla RNNs and the more complex gated recurrent units.