Abstract:

As the smallest semantic unit， sememe is crucial for headline generation task. Although Sememe-Driven Language Model （SDLM） is one of the mainstream models， it has limited encoding capability when dealing with long text sequences， does not fully consider positional relationships， and is prone to introduce noisy knowledge to affect the quality of generated headlines. To address the above problems， a Transformer-based generative headline model was proposed， namely Tran-A-SDLM （Transformer Adaption based Sememe-Driven Language Model with positional embedding and knowledge reasoning）， which fully combined the advantages of adaptive position embedding and knowledge reasoning mechanism. Firstly， Transformer model was introduced to enhance the model’s encoding capability for text sequences. Secondly， the adaptive positional embedding mechanism was utilized to enhance the model’s positional awareness capability， thereby improving the learning of contextual sememe knowledge. In addition， a knowledge reasoning module was introduced for representing the sememe knowledge and guiding the model to generate accurate headlines. Finally， to demonstrate the superiority of Tran-A-SDLM， experiments were conducted on Large scale Chinese Short Text Summarization （LCSTS） dataset. Experimental results show that Tran-A-SDLM achieves improvements of 0.2， 0.7 and 0.5 percentage points respectively in ROUGE-1， ROUGE-2 and ROUGE-L scores， compared to RNN-context-SDLM. Results of the ablation study further validate the effectiveness of the proposed model.

Key words: generative headline, adaptive position embedding, Transformer, knowledge reasoning, natural language processing

摘要：

义原作为最小的语义单位对于标题生成任务至关重要。尽管义原驱动的神经语言模型（SDLM）是主流模型之一，但它在处理长文本序列时编码能力有限，未充分考虑位置关系，易引入噪声知识进而影响生成标题的质量。针对上述问题，提出一种基于Transformer的生成式标题模型Tran-A-SDLM（Transformer Adaption based Sememe-Driven Language Model with positional embedding and knowledge reasoning）。该模型充分结合自适应位置编码和知识推理机制的优势。首先，引入Transformer模型以增强模型对文本序列的编码能力；其次，利用自适应位置编码机制增强模型的位置感知能力，从而增强对上下文义原知识的学习；此外，引入知识推理模块，用于表示义原知识，并指导模型生成准确标题；最后，为验证Tran-A-SDLM的优越性，在大规模中文短文本摘要（LCSTS）数据集上进行实验。实验结果表明，与RNN-context-SDLM相比，Tran-A-SDLM在ROUGE-1、ROUGE-2和ROUGE-L值上分别提升了0.2、0.7和0.5个百分点。消融实验结果进一步验证了所提模型的有效性。

关键词: 生成式标题, 自适应位置编码, Transformer, 知识推理, 自然语言处理