In order to solve the Multi-Objective Tracking (MOT) algorithms’ problems such as ID Switch (IDS) caused by fuzzy pedestrian features and verify the importance of pedestrian appearance in the tracking process, an Attention Self-Correlation Network (ASCN) based on center point detection model was proposed. Firstly, the original image was learned by channel and spatial attention networks to obtain two different feature maps, and the deep information was decoupled. Then, more accurate pedestrian appearance features and pedestrian orientation information were obtained through the autocorrelation learning between the feature maps, and this information was used to track association process. In addition, a tracking dataset of videos at low frame rate conditions was produced to verify the performance of the improved algorithm. When the video frame rate conditions were not ideal, the pedestrian appearance information was obtained by the improved algorithm through ASCN, and the algorithm had better accuracy and robustness than the algorithms only using pedestrian orientation information. Finally, the improved algorithm was tested on the MOT17 dataset of MOT Challenge. Experimental results show that compared with the FairMOT (Fairness in MOT) without adding ASCN, the improved algorithm has the Multiple Object Tracking Accuracy (MOTA) and Identification F-Score (IDF1) increased by 0.5 percentage points and 1.1 percentage points respectively, the number of IDS decreased by 32.2%, and the running speed on a single NVIDIA Tesla V100 card reached 21.2 frames per second. The above proves that the improved algorithm not only reduces the errors in the tracking process, but also improves the overall tracking performance, and can meet the real-time requirements.