Crack segmentation is an important prerequisite for evaluating the damage degree of pavement diseases. In order to balance the effectiveness and real-time of deep neural network segmentation, a lightweight asphalt pavement crack segmentation neural network based on U?Net encoder-decoder structure was proposed, namely PIPNet (Parallel dilated convolution of Inverted Pyramid Network). The encoding part was an inverted pyramid structure. Multi-branch parallel dilated convolution module with different dilatation rates was proposed to extract multi-scale information from the top, middle and bottom features and reduce model complexity, which combined deep separable convolutions with ordinary convolutions and gradually reduced the number of parallel convolutions. Drawing on the characteristics of GhostNet, an inverse residual lightweight module was designed, which was embedded with parallel dual pooling attention. Test results on GAPs384 dataset show that, compared with ResNet50 encoding method, PIPNet has mIoU (mean Intersection over Union) 1.10 percentage points higher with only about one-sixth of parameter quantity and MFLOPs (Million FLOating Point operations), and its mIoU is 4.14 and 9.95 percentage points higher than those of lightweight GhostNet and SegNet, respectively. Experimental results show that PIPNet has high crack segmentation performance while reducing the model complexity, and has good adaptability to segmentation of different pavement crack images.