In three-dimensional sound reproduction with two speakers, Crosstalk Cancellation System (CCS) performance optimization often pay more attention to the effect independently by the factors such as inverse filter parameters design and loudspeaker configuration. A frequency-domain Least-Squares (LS) estimation approximation was proposed to use for the performance optimization. The relationship between these factors and their effect on CCS performance was evaluated systematically. To achieve the tradeoff of computing efficiency and system performance of crosstalk cancellation algorithm, this method obtained the optimization parameters. The effect of crosstalk cancellation was evaluated with Channel Separation (CS) and Performance Error (PE) index, and the simulation results indicate that these parameters can obtain good crosstalk cancellation effect.