Primary porosity and permeability are critical inputs to reliable static and dynamic earth models. In many clastic sedimentary systems, these rock properties have a strong relationship to grain size. Having an automated and consistent core grain-size interpretation free from interpreter bias brings consistency and improved model accuracy while also potentially helping to reduce interpretation cost and time. In this paper, a method is presented to characterize grain size rapidly and consistently from core photographs using deep convolutional neural networks (CNNs). In this proof-of-concept study, several deep CNNs were trained with different network architectures to predict discrete grain size from core photographs and to achieve satisfactory accuracy on blind-test data. The training data includes 300-pixels-per-inch photographs of more than 100 m of cores from one well and associated grain-size description. To generate training and validation data for supervised training, “postage-stamp” 512 × 512 pixel images (4.3 cm × 4.3 cm) were sampled from the core photographs and discrete grain-size labels were assigned to each image. In this way, tens of thousands of labeled examples were generated for training, validating, and testing the CNNs. A blind test on two different wells was conducted by comparing the CNN-predicted against the manually interpreted grain size. Numerical error analysis and visual inspection of the prediction showed the CNNs produced satisfactory accuracy of a continuous grain-size log and even picked up some heterogeneities that were “upscaled” by the manual core interpretation process. This was an important improvement over the manual core interpretation process as these heterogeneities are important to capture when modeling fluid flow.