Abstract:

As a critical way of higher education quality assurance， the scientific and rational implementation of undergraduate teaching audit and evaluation impacts the level of talent cultivation in universities directly. However， traditional manual review modes are inefficient and subjective when faced with massive heterogeneous data， making it difficult to meet the demands for accuracy and standardization in undergraduate teaching evaluation. Therefore， an intelligent undergraduate teaching evaluation system based on Large Language Models （LLMs） and multi-agent architecture — SmartEval — was proposed. In the system， input contents were parsed through a semantic understanding module， the tasks were decomposed and scheduled using a planner， and a Retrieval-Augmented Generation （RAG） module was integrated with three types of agents： question-answering， summarization， and diagnostics to realize end-to-end automation of the entire process of “data collection-metric analysis-decision support”. Experimental results based on the “1+3+3” series reports on 2023 undergraduate teaching evaluation of selected universities demonstrate that SmartEval outperforms the existing mainstream LLMs， such as GLM-4 and Qwen2.5， in metrics such as question-answering accuracy， ROUGE-L （Recall-Oriented Understudy for Gisting Evaluation L） score for summarization， and F1-value for diagnostics significantly. Furthermore， consistency tests with expert groups validate the reliability of SmartEval results.

Key words: undergraduate teaching evaluation, Large Language Model (LLM), agent, digital intelligence, educational informatization

摘要：

本科教学审核评估作为高等教育质量保障的重要手段，它的科学合理实施直接影响高校人才培养水平。然而，传统的人工审阅模式在面对海量异构数据时效率低下且主观性强，难以满足本科教学评估对精准性和标准化的需求。因此，提出一种基于大语言模型（LLM）和多智能体架构的本科教学评估系统——智评宝（SmartEval）。该系统通过语义理解模块解析输入内容，并利用计划器进行任务分解与调度，同时结合检索增强生成（RAG）模块及问答、摘要与诊断三类智能体实现对“数据采集-指标分析-决策支持”全流程的自动化处理。在2023年度部分高校本科教学评估的“1+3+3”系列报告基础上的实验结果表明，与GLM-4和Qwen2.5等主流LLM相比，SmartEval在问答准确率、摘要ROUGE-L（Recall-Oriented Understudy for Gisting Evaluation L）值以及诊断F1值等指标上均表现出显著优势。进一步地，通过与专家组的一致性检验比对，验证了SmartEval结果的可靠性。

关键词: 本科教学评估, 大语言模型, 智能体, 数智化, 教育信息化