The problem of network traffic classification has always been a challenge of iterative methods with the development of network communication， and many solutions have been developed. At present， most network data classification methods focus on the balanced dataset to facilitate experiment and calculation. To solve the problem that most real network datasets are still unbalanced， a network traffic classification model VAE-ABC （Variational AutoEncoder- Adaptive Boosting-Convolutional neural network） was proposed by integrating Variational AutoEncoder （VAE） and Adaptive Boosting Convolutional Neural Network （AdaBoost-CNN）. Firstly， at the data level， VAE was used to partially enhance the unbalanced dataset， and shorten the learning time with the VAE’s characteristics of learning data potential distribution. Then， in order to improve classification effect at the algorithm level， combining with the idea of ensemble learning， AdaBoost-CNN algorithm was designed on the basis of Adaptive Boosting （AdaBoost） algorithm with using an improved Convolutional Neural Network （CNN） as a weak classifier， thereby improving the accuracy of learning and training. Finally， the fully connected layer was used to complete feature mapping， and then the final classification results were obtained through an activation function Sigmoid. After multiple comparisons， experimental results show that the proposed model achieves an accuracy of 94.31% on the unbalanced sub-dataset of partitioned classification dataset ISCX VPN-nonVPN. Compared with AdaBoost-SVM， using Support Vector Machine （SVM） as a weak classifier， SMOTE-SVM， combining SMOTE （Synthetic Minority Oversampling TEchnique） and SVM， and SMOTE-AB-D-T， with Decision Tree （D-T） as a weak classifier and combined with SMOTE algorithm， the proposed model has the accuracy increased by 1.34， 0.63 and 0.24 percentage points， respectively. It can be seen that the classification effect of this model is better than those of other models on this dataset.