In order to make full use of Adjacent Reference Point (ARP) information in radio-map, a new method of establishing both location fingerprint database based on Received Signal Strength (RSS) and physical neighbor information database for each Reference Point (RP) in the off-line phase was proposed to improve the accuracy of fingerprinting-based probabilistic localization. In the on-line phase, based on the probability distribution of RSS, the system first used Bayesian inference to calculate the most adjacent points for each test point. Then, by using physical neighbor information database, the system found the physical adjacent points with respect to the most adjacent points. In the set of most adjacent and physical adjacent points, the system selected feature points for second Bayesian inference. Finally, the system estimated the position of each test point at the center of the group of feature points which had the Maximum A Posterior (MAP) probability. The simulation results show that, compared with the traditional method without physical neighbor information database, the proposed method can improve the localization accuracy by nearly 10%, which enhances the reliability of location determination.