Agricultural user identification method integrating image-text and large language model

doi:10.11772/j.issn.1001-9081.2025091082

Journal of Computer Applications ›› 2026, Vol. 46 ›› Issue (10): 3143-3152.DOI: 10.11772/j.issn.1001-9081.2025091082

• Artificial intelligence • Previous Articles

Agricultural user identification method integrating image-text and large language model

Yimeng SHANG¹, Zenglin CHI¹, Hongming ZHANG¹, Bin LIU¹, Guoqiang HU², Dangdang NIU¹()

^1.College of Information Engineering，Northwest A&F University，Yangling Shaanxi 712100，China
^2.Network & Education Technology Centre，Northwest A&F University，Yangling Shaanxi 712100，China

Received:2025-09-19 Revised:2025-12-24 Accepted:2026-01-04 Online:2026-01-21 Published:2026-10-10
Contact: Dangdang NIU
About author:SHANG Yimeng， born in 2001， M. S. candidate. His research interests include deep learning， multimodal learning.
CHI Zenglin， born in 1997， M. S. candidate. His research interests include combinatorial optimization.
ZHANG Hongming， born in 1979， Ph. D.， professor. His research interests include artificial intelligence， smart agriculture.
LIU Bin， born in 1981， Ph. D.， professor. His research interests include computer vision， parallel algorithms for deep learning.
HU Guoqiang， born in 1981， M. S.， senior engineer. His research interests include information network technology application， education informatization.
Supported by:
National Natural Science Foundation of China(62206222);National Key Research and Development Program of China(2020YFD1100601)

融合图文与大语言模型的农业用户身份识别方法

尚毅蒙¹, 迟增林¹, 张宏鸣¹, 刘斌¹, 胡国强², 牛当当¹()

^1.西北农林科技大学信息工程学院，陕西杨凌 712100
^2.西北农林科技大学网络与教育技术中心，陕西杨凌 712100

通讯作者: 牛当当
作者简介:尚毅蒙（2001—），男，甘肃庆阳人，硕士研究生，主要研究方向：深度学习、多模态学习
迟增林（1997—），男，山东青岛人，硕士研究生，主要研究方向：组合优化
张宏鸣（1979—），男，内蒙古赤峰人，教授，博士，CCF会员，主要研究方向：人工智能、智慧农业
刘斌（1981—），男，陕西渭南人，教授，博士，CCF会员，主要研究方向：计算机视觉、深度学习并行算法
胡国强（1981—）：男，陕西周至人，高级工程师，硕士，主要研究方向：信息网络技术应用、教育信息化
基金资助:
国家自然科学基金资助项目(62206222);国家重点研发计划项目(2020YFD1100601)

Abstract

Abstract:

The difficulty of relying on historical interaction data for effective modelling in the cold-start stage of recommender systems constrains the quality and precision of personalized services severely. To address this issue， user profile pictures and nicknames obtained from login recommender systems were utilized， and a multimodal identification model combining user profile pictures and nicknames， ResBERT-MMNet （ResNet-BERT MultiModal Network）， was proposed. In the model， a multi-layer stacked bidirectional residual cross-modal attention mechanism was designed to achieve image-text semantic interaction， thereby enhancing the understanding of user identities. Furthermore， to address the limitation of incorrect predictions produced by multimodal models under low-confidence conditions， a Large Language Model （LLM）?based prediction of agricultural user identity types was introduced， and the DeepSeek-R1-Distill-Qwen-7B model was used for Low-Rank Adaptation （LoRA） fine-tuning， so as to infer user identity types from user nicknames， and the calculation results of LLM were employed as a supplementary decision-making module to improve overall system robustness. Experimental results on a self-built dataset demonstrate that ResBERT-MMNet outperforms other models in multi-type agricultural user identification tasks with the identification precision and F1-score reaching 67.47% and 68.05%， respectively. After integrating with the fine-tuned LLM， the model achieves an identification precision of 69.73% and an F1-score of 70.06%， indicating strong practical applicability. It can be seen that the proposed model provides the prior data in the cold-start stage of agricultural recommender systems and offers a technical solution for user profiling and precision services.

Key words: recommender system, cold-start, multimodal learning, cross-modal attention, residual structure, Low-Rank Adaptation (LoRA)

摘要：

推荐系统冷启动阶段难以依赖历史交互数据进行有效建模，严重制约了个性化服务的质量与精准度。针对这一问题，以用户登录推荐系统即可获取到的头像和昵称作为研究对象，提出一种结合用户头像与昵称信息的多模态身份识别模型ResBERT-MMNet（ResNet-BERT MultiModal Network），其中设计多层堆叠的双向残差跨模态注意力机制实现图文语义交互，增强对用户身份的理解能力。进一步地，针对多模态模型在低置信度场景下可能产生错误预测的局限性，引入基于大语言模型（LLM）的农业用户身份类型预测，使用DeepSeek-R1-Distill-Qwen-7B模型进行低秩适配（LoRA）微调，从而能够根据用户昵称推理出用户的身份类型，并将LLM的计算结果作为补充决策模块提升系统整体鲁棒性。在自建数据集上的实验结果表明，ResBERT-MMNet在多个农业用户类别的识别任务中的识别精确率和F1值分别达到了67.47%和68.05%，相较于其他模型取得了更优异的表现；结合微调的LLM，识别精确率和F1值分别提升到69.73%和70.06%，表现出较强的实际适用性，为农业领域推荐系统在冷启动阶段提供先验数据，同时也为用户画像构建和精准服务贡献了技术解决思路。

CLC Number:

TP391.3

Yimeng SHANG, Zenglin CHI, Hongming ZHANG, Bin LIU, Guoqiang HU, Dangdang NIU. Agricultural user identification method integrating image-text and large language model[J]. Journal of Computer Applications, 2026, 46(10): 3143-3152.

尚毅蒙, 迟增林, 张宏鸣, 刘斌, 胡国强, 牛当当. 融合图文与大语言模型的农业用户身份识别方法[J]. 《计算机应用》唯一官方网站, 2026, 46(10): 3143-3152.

Figures/Tables 15

References 36

[1]	Bouni M， Hssina B， Douzi K， et al. Towards an efficient recommender systems in smart agriculture： a deep reinforcement learning approach［J］. Procedia Computer Science， 2022， 203： 825-830.
[2]	Srinivasu P N， Ijaz M F， Woźniak M. XAI‐driven model for crop recommender system for use in precision agriculture［J］. Computational Intelligence， 2024， 40（1）： No.e12629.
[3]	Baraian I， Erdei R， Tamaian R， et al. Trend-enabled recommender system with diversity enhancer for crop recommendation［J］. Agriculture， 2025， 15（15）： No.1614.
[4]	Wang J， Lu H， Caverlee J， et al. Large language models as data augmenters for cold-start item recommendation［C］// WWW 2024. New York： ACM， 2024： 726-729.
[5]	毛骞，谢维成，乔逸天，等. 推荐系统冷启动问题解决方法研究综述［J］. 计算机科学与探索， 2024， 18（5）： 1197-1210.
	Mao Qian， Xie Weicheng， Qiao Yitian， et al. Survey on solving cold start problem in recommendation systems［J］. Journal of Frontiers of Computer Science and Technology， 2024， 18（5）： 1197-1210.
[6]	Yuan H， Hernandez A A. User cold start problem in recommendation systems： a systematic review［J］. IEEE Access， 2023， 11： 136958-136977.
[7]	Panteli A， Boutsinas B. Addressing the cold-start problem in recommender systems based on frequent patterns［J］. Algorithms， 2023， 16（4）： No.182.
[8]	Hao B， Yin H， Zhang J， et al. A multi-strategy-based pre-training method for cold-start recommendation［J］. ACM Transactions on Information Systems， 2023， 41（2）： No.31.
[9]	Wang B， Shi Y， Mu W， et al. Modeling of farmers’ production performance portrait based on GASA-WFCM clustering［J］. Computers and Electronics in Agriculture， 2023， 214： No.108320.
[10]	da Silveira F， de Faria Corrêa R G， Baierle I C， et al. Behavioral profile of farmers in the adoption of agriculture 4.0 technologies in the agri-food system： a case study in Brazil［J］. Frontiers in Sustainable Food Systems， 2025， 9： No.1624753.
[11]	Boyera S， Addison C， Msengezi C. Farmer profiling： making data work for smallholder farmers［EB/OL］. （2017-11）［2025-08-10］..
[12]	冯建英，王博，吴丹丹，等. 用户画像技术与其在农业领域应用研究进展［J］. 农业机械学报， 2021， 52（S1）： 385-395.
	Feng Jianying， Wang Bo， Wu Dandan， et al. User profiling technology and its application in agriculture［J］. Transactions of the Chinese Society for Agricultural Machinery， 2021， 52（S1）： 385-395.
[13]	Qiang M， Wang Z， Li S， et al. Exploring unified training framework for multimodal user profiling［C］// COLING 2025. Stroudsburg： ACL， 2025： 1699-1710.
[14]	Liu F， Cheng Z， Sun C， et al. User diverse preference modeling by multimodal attentive metric learning［C］// MM 2019. New York： ACM， 2019： 1526-1534.
[15]	Estruch C P， Palacios R P， Rosso P. Learning multimodal gender profile using neural networks［C］// RANLP 2017. ［S.l.］： INCOMA Ltd.， 2017： 577-582.
[16]	Prottasha N J， Kowsher M， Raman H， et al. User profile with large language models： construction， updating， and benchmarking［PP/OL］. V2. arXiv （2025-03-16）［2025-08-10］..
[17]	David S， Meidan Y， Hersko I， et al. ProfiLLM： an LLM-based framework for implicit profiling of chatbot users［PP/OL］. arXiv （2025-06-16）［2025-08-10］..
[18]	Woźniak S， Duszenko J， Kocoń J， et al. Improving LLM-based recommender systems with user-controllable profiles［C］// WWW 2025. New York： ACM， 2025： 2102-2111.
[19]	He K， Zhang X， Ren S， et al. Deep residual learning for image recognition［C］// CVPR 2016. Piscataway： IEEE， 2016： 770-778.
[20]	Devlin J， Chang M W， Lee K， et al. BERT： pre-training of deep bidirectional Transformers for language understanding［C］// NAACL-HLT 2019， Volume 1 （Long and Short Papers）. Stroudsburg： ACL， 2019： 4171-4186.
[21]	Vaswani A， Shazeer N， Parmar N， et al. Attention is all you need［C］// NeurIPS 2017. Red Hook： Curran Associates Inc.， 2017： 6000-6010.
[22]	Guo D， Yang D， Zhang H， et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning［J］. Nature， 2025， 645（8081）： 633-638.
[23]	Hu E J， Shen Y， Wallis P， et al. LoRA： low-rank adaptation of large language models［PP/OL］. V2. arXiv （2021-10-16）［2025-08-10］..
[24]	Cui Y， Che W， Liu T， et al. Pre-training with whole word masking for Chinese BERT［J］. IEEE/ACM Transactions on Audio， Speech， and Language Processing， 2021， 29： 3504-3514.
[25]	Brown T B， Mann B， Ryder N， et al. Language models are few-shot learners［C］// NeurIPS 2020. Red Hook： Curran Associates Inc.， 2020： 1877-1901.
[26]	Zhang Y， Sidibé D， Morel O， et al. Deep multimodal fusion for semantic image segmentation： a survey［J］. Image and Vision Computing， 2021， 105： No.104042.
[27]	Qiu C， Wu W， Zhang X F， et al. Late fusion with triplet margin objective for multimodal ideology prediction and analysis［C］// EMNLP 2022. Stroudsburg： ACL， 2022： 9720-9736.
[28]	Jiao T， Guo C， Feng X， et al. A comprehensive survey on deep learning multi-modal fusion： methods， technologies and applications［J］. Computers， Materials and Continua， 2024， 80（1）： 1-35.
[29]	Shannon C E. A mathematical theory of communication［J］. The Bell System Technical Journal， 1948， 27（3）： 379-423.
[30]	Zheng J， Zhang S， Wang X， et al. Multimodal representations learning based on mutual information maximization and minimization and identity embedding for multimodal sentiment analysis［PP/OL］. V2. arXiv （2022-07-04）［2025-08-10］..
[31]	Tian Y， Krishnan D， Isola P. Contrastive multiview coding［C］// ECCV 2020， LNCS 12356. Cham： Springer， 2020： 776-794.
[32]	Alayrac J B， Donahue J， Luc P， et al. Flamingo： a visual language model for few-shot learning［C］// NeurIPS 2022. Red Hook： Curran Associates Inc.， 2022： 23716-23736.
[33]	Li J， Li D， Xiong C， et al. BLIP： bootstrapping language-image pre-training for unified vision-language understanding and generation［J］. Proceedings of Machine Learning Research， 2022， 162： 12888-12900.
[34]	Radford A， Kim J W， Hallacy C， et al. Learning transferable visual models from natural language supervision［J］. Proceedings of Machine Learning Research， 2021， 139： 8748-8763.
[35]	Geifman Y， El-Yaniv R. Selective classification for deep neural networks［C］// NeurIPS 2017. Red Hook： Curran Associates Inc.， 2017： 4885-4894.
[36]	Traub J， Bungert T J， Lüth C T， et al. Overcoming common flaws in the evaluation of selective classification systems［C］// NeurIPS 2024. Red Hook： Curran Associates Inc.， 2024： 2323-2347.

身份类别	样本数	占比/%
农民	665	18.96
农资和农产品经销商	668	19.02
农技推广人员	355	10.13
农业社会化服务组织	368	10.45
农业技术研究员	442	12.58
农业政策研究员	396	11.28
非农业相关人员	617	17.58

身份类别	样本数	占比/%
农民	665	18.96
农资和农产品经销商	668	19.02
农技推广人员	355	10.13
农业社会化服务组织	368	10.45
农业技术研究员	442	12.58
农业政策研究员	396	11.28
非农业相关人员	617	17.58

模型	A₁	A₂	P	R	F₁
ResNet	47.75	65.15	44.88	45.46	45.17
BERT	49.27	64.59	45.38	46.21	45.79
CNN+RNN	47.80	61.49	42.62	42.95	42.78
CNN+LSTM	50.84	67.70	48.53	47.16	47.84
CNN+BiLSTM	51.87	64.25	46.14	46.41	46.27
Flamingo-3B	60.42	70.12	52.35	50.91	51.62
BLIP-VQA-base	53.61	67.89	49.36	50.10	49.72
CLIP-VIT-base-patch32	61.29	71.75	56.92	55.18	56.03
ResBERT-MMNet	69.17	77.90	67.47	68.64	68.05

模型	A₁	A₂	P	R	F₁
ResNet	47.75	65.15	44.88	45.46	45.17
BERT	49.27	64.59	45.38	46.21	45.79
CNN+RNN	47.80	61.49	42.62	42.95	42.78
CNN+LSTM	50.84	67.70	48.53	47.16	47.84
CNN+BiLSTM	51.87	64.25	46.14	46.41	46.27
Flamingo-3B	60.42	70.12	52.35	50.91	51.62
BLIP-VQA-base	53.61	67.89	49.36	50.10	49.72
CLIP-VIT-base-patch32	61.29	71.75	56.92	55.18	56.03
ResBERT-MMNet	69.17	77.90	67.47	68.64	68.05

融合策略	跨模态注意力	多层堆叠	残差结构	特征融合方式
Baseline-Concat	—	—	—	直接拼接
MHA-NoResidual	（1×1）	—	—	交互后拼接
StackedMHA-NoResidual	（5×2）	√	—	交互后拼接
StackedMHA+Residual	（5×2）	√	√	交互后拼接

Agricultural user identification method integrating image-text and large language model

融合图文与大语言模型的农业用户身份识别方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 15

References 36

Related Articles 15

Recommended Articles

Metrics

融合策略	A₁	A₂	P	R	F₁
Baseline-Concat	51.84	64.17	40.62	42.45	41.48
MHA-NoResidual	61.28	73.36	52.79	51.55	52.18
StackedMHA-NoResidual	65.68	74.47	59.90	59.22	59.63
StackedMHA+Residual	69.17	77.90	67.47	68.64	68.05

策略	P-value
Baseline-Concat vs MHA-NoResidual	2.253 5×10^-7
Baseline-Concat vs StackedMHA-NoResidual	1.819 3×10^-5
Baseline-Concat vs StackedMHA+Residual	2.835 2×10^-7
MHA-NoResidual vs StackedMHA-NoResidual	7.298 5×10^-3
MHA-NoResidual vs StackedMHA+Residual	4.169 2×10^-4
StackedMHA-NoResidual vs StackedMHA+Residual	3.997 2×10^-2

评价指标	F-value	P-value
Top-1准确率	190.256 3	1.012 6×10^-12
Top-2准确率	144.345 3	8.537 9×10^-12
Precision	108.281 7	7.705 4×10^-11
Recall	49.396 3	2.594 6×10^-8
F1-Score	82.662 0	5.913 3×10^-10

模型	A₁	A₂	P	R	F₁
Qwen-7B	53.42	61.87	51.33	44.64	47.74
Qwen-7Bs	58.40	69.93	59.26	56.52	57.86
ResBERT-MMNet	69.17	77.90	67.47	68.64	68.05
ResBERT-MMNet+Qwen-7Bs	72.56	79.02	69.73	70.46	70.06

模型	参数量/10⁹	显存占用/GB	推理延迟/ms
Qwen-7B	7.62	14.26	2 709.98
Qwen-7Bs	7.75	14.38	3 078.28
ResBERT-MMNet	0.13	0.41	476.10
ResBERT-MMNet+Qwen-7Bs	7.89	14.92	1 344.05

头像	昵称	真实身份	ResBERT-MMNet预测	Qwen-7Bs预测	最终预测身份
图6（a）	“茶韵清香”	农资和农产品经销商	农资和农产品经销商置信度0.971	农资和农产品经销商	农资和农产品经销商
图6（b）	“平安是福”	农民	农民置信度0.833	农民	农民
图6（c）	“人间清醒”	农业社会化服务组织	农技推广人员置信度0.462	农业社会化服务组织	农业社会化服务组织
图6（d）	“顺其自然”	农民	农民置信度0.905	农业技术研究员	农民
图6（e）	“顺其自然”	农技推广人员	农技推广人员置信度0.836	农业技术研究员	农技推广人员
图6（f）	“西峡县香菇批发”	农资和农产品经销商	农资和农产品经销商置信度0.942 8	农资和农产品经销商	农资和农产品经销商

[1]	Zhenhui GONG, Xiaoyu SHI, Yun LU, Yangcheng LIU, Mingsheng SHANG. Fair recommendation framework for large language models with sensitive attribute absence [J]. Journal of Computer Applications, 2026, 46(9): 2838-2846.
[2]	Lin HUANG, Yingshun LI, Weiyan TONG, Shuyuan ZHANG, Zihan WANG. Remote sensing image object detection algorithm based on improved real-time detection Transformer [J]. Journal of Computer Applications, 2026, 46(8): 2652-2659.
[3]	Didi HOU, Shaodong HONG, Yujie FU, Yunhe CUI, Guowei SHEN. Cybersecurity named entity recognition method based on large language model linking [J]. Journal of Computer Applications, 2026, 46(7): 2064-2073.
[4]	Yunping HE, Leichun WANG, Ruirui SONG, Xiangfeng LU, Jinxiang WEI, Xiaomeng LIU. Dual-channel multimodal sentiment analysis model based on contrast invariance and reinforcement specificity [J]. Journal of Computer Applications, 2026, 46(6): 1767-1775.
[5]	Yi DU, Mingjin XU, Jiayi KONG, Liyao WANG, Chen ZHAO. Low-rank adaptive parameter-efficient fine-tuning algorithm based on YOLOv11 [J]. Journal of Computer Applications, 2026, 46(6): 1738-1745.
[6]	Haihua ZHAO, Yijun HU, Rui TANG, Xian MO. Multimodal recommendation method based on semantic fusion and contrast enhancement [J]. Journal of Computer Applications, 2026, 46(4): 1058-1068.
[7]	Haoqian JIANG, Dong ZHANG, Guanyu LI, Heng CHEN. SetaCRS： Conversational recommender system with structure-enhanced hierarchical task-oriented prompting strategy [J]. Journal of Computer Applications, 2026, 46(2): 368-377.
[8]	Long CHEN, Xinmin HOU. Graph convolutional recommender system integrating contrastive learning and global graph data augmentation [J]. Journal of Computer Applications, 2026, 46(10): 3076-3082.
[9]	Yancui SHI, Haozhe QIN. Recommendation method integrating user behaviors and improved long-tail algorithm [J]. Journal of Computer Applications, 2026, 46(1): 95-103.
[10]	Chao LIU, Yanhua YU. Knowledge-aware recommendation model combining denoising strategy and multi-view contrastive learning [J]. Journal of Computer Applications, 2025, 45(9): 2827-2837.
[11]	Renjie TIAN, Mingli JING, Long JIAO, Fei WANG. Recommendation algorithm of graph contrastive learning based on hybrid negative sampling [J]. Journal of Computer Applications, 2025, 45(4): 1053-1060.
[12]	Tingjie TANG, Jiajin HUANG, Jin QIN. Session-based recommendation with graph auxiliary learning [J]. Journal of Computer Applications, 2024, 44(9): 2711-2718.
[13]	Xin YANG, Xueni CHEN, Chunjiang WU, Shijie ZHOU. Short-term traffic flow prediction of urban highway based on variant residual model and Transformer [J]. Journal of Computer Applications, 2024, 44(9): 2947-2951.
[14]	Yusheng LIU, Xuezhong XIAO. High-fidelity image editing based on fine-tuning of diffusion model [J]. Journal of Computer Applications, 2024, 44(11): 3574-3580.
[15]	Li ZENG, Jingru YANG, Gang HUANG, Xiang JING, Chaoran LUO. Survey on hypergraph application methods： issues， advances， and challenges [J]. Journal of Computer Applications, 2024, 44(11): 3315-3326.