Abstract:

To address poor performance of the existing pre-trained model in Tibetan segmentation tasks， a method was proposed to establish a tokenization reconstruction standard to regulate the constraint text， and subsequently reconstruct the tokenization of the Tibetan pre-trained model to perform Tibetan segmentation tasks. Firstly， the standardization operation was performed on the original text to solve the incorrect cuts due to language mixing and so on. Secondly， reconstruction of the tokenization at syllable granularity was performed on the pre-trained model to make the cut-off units parallel to the labeled units. Finally， after completing the sticky cuts using the improved sliding window restoration method， the Re-TiBERT-BiLSTM-CRF model was established by the use of the “Begin， Middle， End and Single” （BMES） four element annotation method， so as to obtain the Tibetan word segmentation system. Experimental results show that the pre-trained model after reconstructing the tokenization is significantly better than the original pre-trained model in the segmentation tasks. The obtained system has a high Tibetan word segmentation precision， and its F1 value can reach up to 97.15%， so it can complete Tibetan segmentation tasks well.

Key words: Tibetan language information processing, Tibetan word segmentation model, pre-trained model, natural language processing, tokenization reconstruction

摘要：

针对现有的预训练模型在藏文分词任务中表现不佳的问题，提出一种建立重构标记器规范约束文本，随后重构藏文预训练模型的标记器以进行藏文分词任务的方法。首先，对原始文本进行规范化操作，以解决因语言混用等导致的错误切分的问题；其次，对预训练模型进行音节粒度的标记器重构，使得切分单元与标注单元平行；最后，在利用改进的滑动窗口还原法完成黏着切分后，利用“词首、词中、词尾、孤立”（BMES）四元标注法建立Re-TiBERT-BiLSTM-CRF模型，从而得到藏文分词系统。实验结果表明，重构标记器后的预训练模型在分词任务中明显优于原始预训练模型，而得到的系统拥有较高的藏文分词精确率，F1值最高可达97.15%，能够较好地完成藏文分词任务。

关键词: 藏语信息处理, 藏文分词模型, 预训练模型, 自然语言处理, 标记器重构