We'll present the use of deep learning for autonomous condition assessment of civil infrastructure systems. Regular inspection of civil infrastructure systems is crucial for safe operations. Manual inspection is currently the predominant method of inspection and is time-consuming, tedious, and subjective. A less time-consuming and inexpensive alternative is the use of optical instrumentation (for example, digital cameras), where the feasibility of using image processing techniques to detect deterioration in structures has been acknowledged by leading experts in the field. Due to the recent advances in using CNNs, the vision-based classification performance of computers has been improved significantly. A CNN learns the appropriate classification features that in traditional algorithms were hand-engineered. Eliminating the need for dependence on prior knowledge and human effort in designing features is a major advantage of CNNs. We'll discuss CNN-based approaches for condition assessment of infrastructure systems, including a new framework that combines deep convolutional neural network and Naive Bayes classifier to detect cracks in videos. The crack patches are spatially and temporally clustered and the posterior probabilities of being real cracks are derived. Experimental tests have been carried out to evaluate the performance of the proposed system.