For the low tag identification rate problem caused by that reader can not identify multiple tags simultaneously in single-antenna Radio Frequency Identification (RFID) system, combined with multi-antenna technology and the grouping of binary tree slots based on tags ID sequence, an adaptive tree grouping and blind separation anti-collision algorithm for RFID system was proposed. In the presented algorithm, by adjusting query code length of reader according to the number of antennas in RFID system and sending a query signal, the eligible response tags were assigned to the appropriate slots so that the number of tags in each slot was less than or equalled to the number of antennas and met Blind Source Separation (BSS) system conditions which could identify tags, so as to achieve the purpose of identifying tags simultaneously and quickly. Compared with the Blind Separation and Dynamic Bit-slot Grouping (BSDBG) algorithm using the same multi-antenna technology, the simulation results show that the tag identification speed of the proposed algorithm increases from 20% to 69% and the tag identification rate improves from 60% to 88% when the number of antennas is from 4 to 32, while it has low complexity, low hardware overhead and it is relatively simple to implement and conducive to the promotion and use.