Currently, in the field of video tracking, the typical Siamese network based algorithms only locate the center point of target, which results in poor locating performance on fast-deformation objects. Therefore, a real-time video tracking algorithm based on Siamese detection network called Siamese-FC Region-convolutional neural network (SiamRFC) was proposed. SiamRFC can directly predict the center position of the target, thus dealing with the rapid deformation. Firstly, the position of the center point of the target was obtained by judging the similarity. Then, the idea of object detection was used to return the optimal position by selecting a series of candidate boxes. Experimental results show that SiamRFC has good performance on the VOT2015|16|17 test sets.