Abstract: The equivalent channel matrix dimension of Generalized Frequency Division Multiplexing with Multiple Input Multiple Output (MIMO-GFDM) system is very large, and the traditional Multiple Input Multiple Output (MIMO) detection algorithm has high complexity and poor performance. Aiming at those problems, Reactive Tabu Search (RTS) detection algorithm in massive MIMO systems was applied to MIMO-GFDM system, and the high complexity problem of the initial value in RTS algorithm was also solved. Firstly, by using the positive definite symmetry of the matrix used in Minimum Mean Squared Error (MMSE) detection algorithm, Cholesky decomposition was applied to the matrix, and Sherman-Morrison formula was combined to iteratively calculate the initial value, reducing high complexity of the initial value inversion. Then, with the result of the improved MMSE detection as the initial value of RTS algorithm, the optimum solution was searched globally from the initial value. Finally, the iteration numbers and Bit Error Rate (BER) performance were researched through simulations. Theoretical analysis and simulation results show that, in MIMO-GFDM, the improved RTS signal detection algorithm has much lower BER than traditional signal detection algorithms. In 4 Quadrature Amplitude Modulation (4QAM), the RTS algorithm has approximately 6 dB lower signal-to-noise performance gain than MMSE detection (when BER is 10^-3). In 16QAM, the RTS algorithm has approximately 4 dB lower signal-to-noise performance gain than MMSE detection (when BER is 10^-2). Compared with the traditional RTS algorithm, the proposed algorithm has lower complexity without affecting the BER performance.

Key words: Generalized Frequency Division Multiplexing (GFDM), Multiple Input Multiple Output (MIMO), Reactive Tabu Search (RTS), signal detection, Cholesky decomposition, Sherman-Morrison formula

摘要： 针对多输入多输出的广义频分复用（MIMO-GFDM）系统的等效信道矩阵维度极大，传统的MIMO检测算法复杂度高且性能不佳的问题，将大规模MIMO系统中的动态禁忌搜索（RTS）检测算法运用到MIMO-GFDM系统中，并解决了RTS算法初始值的求解复杂度高的问题。首先利用最小均方误差（MMSE）检测算法所用到矩阵的正定对称性将矩阵Cholesky分解，并结合Sherman-Morrison公式迭代计算初始值，降低了初始值求逆的高复杂度；然后以改进的MMSE检测结果作为RTS算法的初始值，从初始值逐步全局搜索最优解；最后通过仿真，对不同算法的迭代次数和误码率（BER）性能进行了研究。理论分析与仿真结果表明：在MIMO-GFDM中，所提改进RTS信号检测算法误码率远低于传统信号检测算法。在4QAM时，RTS算法大约有低于MMSE检测6 dB的信噪比性能增益（误码率在10^-3时）；在16QAM时，RTS算法大约有低于MMSE检测4 dB的信噪比性能增益（误码率在10^-2时）。与传统RTS算法性相比，所提改进算法在不影响误码率性能的同时降低了算法复杂度。

关键词: 广义频分复用, 多输入多输出, 动态禁忌搜索, 信号检测, Cholesky分解, Sherman-Morrison公式