关键词: 检索增强生成, 大型语言模型, 长文档处理, 基准数据集, 自动问答构建

Abstract:

With the development of Pretrained Language Model （PLM）， Retrieval-Augmented Generation （RAG） is widely concerned as an emerging task. A comprehensive and objective evaluation of RAG is considered essential to reveal the limitations of the existing methods and to indicate future research directions. However， a lack of systematic evaluation benchmarks for RAG is observed， especially in the context of long documents. To address this issue， an automatic question-answering construction strategy based on focused fragments was proposed， aiming to build large-scale QA datasets efficiently and accurately. Based on this strategy， the first bilingual RAG evaluation benchmark dataset for long documents， named LoRAG， was constructed， covering English-Chinese bilingual documents from multiple domains such as law， finance， and literature， with an average document length of 57 000 tokens in English and 76 000 tokens in Chinese. Systematic experiments on the two key stages of RAG — retrieval and generation were conducted using the LoRAG dataset. In the retrieval stage， multiple mainstream embedding models， including text-embedding-ada-002， the bge-large series， bge-m3， and Multilingual-E5-large-instruct， were evaluated， and the reranking model bge-reranker-v2-m3 was introduced for performance optimization and comparison. In the generation stage， representative Large Language Models （LLM）， including Vicuna-13B， ChatGLM2-6B， Llama2-7B， and Claude2， were tested comprehensively. Experimental results show that the constructed dataset LoRAG reveals the positioning challenges faced by current embedding methods in long-document retrieval， as well as the limitations of LLM in balancing relevance and conciseness during the generation process， providing clear research directions for future method improvements.

Key words: Retrieval-Augmented Generation (RAG), Large Language Model (LLM), long-document processing, benchmark dataset, automatic question-answer construction