In view of the low accuracy of functional Magnetic Resonance Imaging (fMRI) brain age prediction and the lack of research on the combination of this problem and deep learning, an fMRI brain age prediction model with Lightweight Multi-scale Convolutional Network (LMCN) was proposed. Firstly, the Pearson correlation coefficient (R) of the Region Of Interest (ROI) in fMRI was calculated to obtain the Functional Connectivity (FC) matrix of the ROI as input. Secondly, the number of FC channels was increased to ensure the number of features and the size of the feature map was reduced simultaneously. At the same time, the multi-scale dilated convolution module RFB (Receptive Field Block) with the characteristics of human visual attention was used to extract age features. Finally, the predicted brain age was output by the fully connected layer, and the ablation prediction results of each brain region were calculated to explore the key brain regions influencing the brain age prediction results. Evaluation was carried out on two public datasets, E-NKI and Cam-CAN. It can be seen that the memory required for LMCN parameters is 2.30 MB, which is 60.3% and 52.0% less than those of MobileNetV3 and ShuffleNetV2 respectively. In terms of prediction results, on E-NKI dataset, LMCN has the Mean Absolute Error (MAE) of 5.16, the R of 0.947, and the Root Mean Square Error (RMSE) of 6.40. Compared to the model that combines network-based feature selection with the least angle regression, LMCN has the MAE decreased by 1.34 and the R increased by 0.037; on Cam-CAN dataset, LMCN has the MAE of 5.97, the R of 0.904, and the RMSE of 7.93. Compared to the connectome-based machine learning model, LMCN has the R increased by 0.019, and the RMSE decreased by 0.64. The results show that while LMCN has small number of parameters and is easy to deploy, it can improve the accuracy of fMRI brain age prediction effectively and provide clues for assessing the brain status of healthy adults.