Machine learning and deep learning have been applied to medical images to predict tumor type, genomics, and therapy effects. They can also be used to segment images, such as to define a tumor. While some traditional machine learning work has been multi-dimensional and multi-parametric, very few deep learning applications have gone beyond applying photographic networks to medical image problems. As such, they ignore some of the rich information available in other dimensions (3D and time) as well as parameter space (other types of images). We'll discuss some of the challenges and early results in extending traditional 2D convolutional neural networks to n-dimensional images, including space, time, and other parametric image types. Challenges include representational issues as well as computational (for example, memory constraints). Applications we'll show include multi-dimensional image segmentation of brain tumors as well as prediction of tumor genomics and therapy response.