Industrial process control method based on local policy interaction exploration-based deep deterministic policy gradient

doi:10.11772/j.issn.1001-9081.2021050716

Journal of Computer Applications ›› 2022, Vol. 42 ›› Issue (5): 1642-1648.DOI: 10.11772/j.issn.1001-9081.2021050716

Special Issue: 前沿与综合应用

• Frontier and comprehensive applications • Previous Articles

Industrial process control method based on local policy interaction exploration-based deep deterministic policy gradient

Shaobin DENG¹^,²^,³^,⁴, Jun ZHU¹^,²^,³, Xiaofeng ZHOU¹^,²^,³(), Shuai LI¹^,²^,³^,⁴, Shurui LIU¹^,²^,³

^1.Key Laboratory of Networked Control System，Chinese Academy of Sciences，Shenyang Liaoning 110016，China
^2.Shenyang Institute of Automation，Chinese Academy of Sciences，Shenyang Liaoning 110169，China
^3.Institutes for Robotics and Intelligent Manufacturing Innovation，Chinese Academy of Sciences，Shenyang Liaoning 110169，China
^4.University of Chinese Academy of Sciences，Beijing 100049，China

Received:2021-05-07 Revised:2021-09-27 Accepted:2021-11-26 Online:2022-03-08 Published:2022-05-10
Contact: Xiaofeng ZHOU
About author:DENG Shaobin， born in 1997，M. S. candidate. His researchinterests include reinforcement learning，industrial process control.
ZHU Jun， born in 1964，M. S.，research follow. His researchinterests include automatic control，industrial automation.
ZHOU Xiaofeng， born in 1978，Ph. D.，associate research fellow.Her research interests include machine learning， industrial process optimization.
LI Shuai， born in 1988，Ph. D. candidate，associate researchfellow. His research interests include machine learning，data mining.
LIU Shurui， born in 1993，M. S.，assistant research fellow. Hisresearch interests include industrial process modeling and control，machine learning.
Supported by:
Program of Liaoning Province “Xingliao Talents Plan”(XLYC1808009)

基于局部策略交互探索的深度确定性策略梯度的工业过程控制方法

邓绍斌¹^,²^,³^,⁴, 朱军¹^,²^,³, 周晓锋¹^,²^,³(), 李帅¹^,²^,³^,⁴, 刘舒锐¹^,²^,³

^1.中国科学院网络化控制系统重点实验室, 沈阳 110016
^2.中国科学院沈阳自动化研究所, 沈阳 110169
^3.中国科学院机器人与智能制造创新研究院, 沈阳 110169
^4.中国科学院大学, 北京 100049

通讯作者: 周晓锋
作者简介:邓绍斌（1997—），男，江西赣州人，硕士研究生，主要研究方向：强化学习、工业过程控制
朱军（1964—），男，辽宁沈阳人，研究员，硕士，主要研究方向：自动控制、工业自动化
周晓锋（1978—），女，辽宁本溪人，副研究员，博士，主要研究方向：机器学习、工业过程优化 zhouxf@sia.cn
李帅（1988—）男，辽宁锦州人，副研究员，博士研究生，主要研究方向：机器学习、数据挖掘
刘舒锐（1993—）男，湖北襄阳人，助理研究员，硕士，主要研究方向：工业过程建模与控制、机器学习。
基金资助:
辽宁省“兴辽英才计划”项目(XLYC1808009)

Abstract

Abstract:

In order to achieve the stable and precise control of industrial processes with non-linearity， hysteresis， and strong coupling， a new control method based on Local Policy Interaction Exploration-based Deep Deterministic Policy Gradient （LPIE-DDPG） was proposed for the continuous control of deep reinforcement learning. Firstly， the Deep Deterministic Policy Gradient （DDPG） algorithm was used as the control strategy to greatly reduce the phenomena of overshoot and oscillation in the control process. At the same time， the control strategy of original controller was used as the local strategy for searching， and interactive exploration was used as the rule for learning， thereby improving the learning efficiency and stability. Finally， a penicillin fermentation process simulation platform was built under the framework of Gym and the experiments were carried out. Simulation results show that， compared with DDPG， the proposed LPIE-DDPG improves the convergence efficiency by 27.3%； compared with Proportion-Integration-Differentiation （PID）， the proposed LPIE-DDPG has fewer overshoot and oscillation phenomena on temperature control effect， and has the penicillin concentration increased by 3.8% in yield. In conclusion， the proposed method can effectively improve the training efficiency and improve the stability of industrial process control.

Key words: industrial process control, deep reinforcement learning, Deep Deterministic Policy Gradient (DDPG), Local Policy Interaction Exploration (LPIE), penicillin fermentation process

摘要：

为了实现对非线性、滞后性和强耦合的工业过程稳定精确的控制，提出了一种基于局部策略交互探索的深度确定性策略梯度（LPIE-DDPG）的控制方法用于深度强化学习的连续控制。首先，使用深度确定性策略梯度（DDPG）算法作为控制策略，从而极大地减小控制过程中的超调和振荡现象；同时，使用原控制器的控制策略作为局部策略进行搜索，并以交互探索规则进行学习，提高了学习效率和学习稳定性；最后，在Gym框架下搭建青霉素发酵过程仿真平台并进行实验。仿真结果表明，相较于DDPG，LPIE-DDPG在收敛效率上提升了27.3%；相较于比例-积分-微分（PID），LPIE-DDPG在温度控制效果上有更少的超调和振荡现象，在产量上青霉素浓度提高了3.8%。可见所提方法能有效提升训练效率，同时提高工业过程控制的稳定性。

关键词: 工业过程控制, 深度强化学习, 深度确定性策略梯度, 局部策略交互探索, 青霉素发酵过程

CLC Number:

TP273.2

Shaobin DENG, Jun ZHU, Xiaofeng ZHOU, Shuai LI, Shurui LIU. Industrial process control method based on local policy interaction exploration-based deep deterministic policy gradient[J]. Journal of Computer Applications, 2022, 42(5): 1642-1648.

邓绍斌, 朱军, 周晓锋, 李帅, 刘舒锐. 基于局部策略交互探索的深度确定性策略梯度的工业过程控制方法[J]. 《计算机应用》唯一官方网站, 2022, 42(5): 1642-1648.

Figures/Tables 11

Fig. 1 Structure of LPIE-DDPG

Fig. 2 Penicillin fermentation process

Fig. 3 Markov model of penicillin fermentation

Tab. 1 Neural network parameter setting

网络	输入神经元数	隐含层神经元数	输出神经元数	学习率
Actor	7	32，16	1	1E $- 4$
Critic	7	32，16	1	1E $- 3$

Tab. 1 Neural network parameter setting

网络	输入神经元数	隐含层神经元数	输出神经元数	学习率
Actor	7	32，16	1	1E $- 4$
Critic	7	32，16	1	1E $- 3$

Tab. 2 Initial values of penicillin fermentation process variables

编号	初始过程状态	数值
1	底物浓度/（ $g · L - 1$ ）	0.050
2	氧气浓度/（ $g · L - 1$ ）	1.110
3	菌体浓度/（ $g · L - 1$ ）	13.000
4	青霉素浓度/（ $g · L - 1$ ）	0.920
5	培养基体积/L	102.220
6	二氧化碳/（ $m m o l · L - 1$ ）	2.372
7	H+浓度/（ $m o l · L - 1$ ）	1E $- 5$
8	发酵器反应温度/K	299.000
9	热量/cal	77 092.970
10	通风率/（ $L · h - 1$ ）	30.000
11	搅拌功率/W	8.600
12	底物添加/（ $L · h - 1$ ）	0.042
13	酸流流速/（ $L · h - 1$ ）	0.007 5
14	碱流流速/（ $L · h - 1$ ）	0.022 5
15	冷水流速/（ $L · h - 1$ ）	40.000
16	热水流速/（ $L · h - 1$ ）	0.000

Tab. 2 Initial values of penicillin fermentation process variables

编号	初始过程状态	数值
1	底物浓度/（ $g · L - 1$ ）	0.050
2	氧气浓度/（ $g · L - 1$ ）	1.110
3	菌体浓度/（ $g · L - 1$ ）	13.000
4	青霉素浓度/（ $g · L - 1$ ）	0.920
5	培养基体积/L	102.220
6	二氧化碳/（ $m m o l · L - 1$ ）	2.372
7	H+浓度/（ $m o l · L - 1$ ）	1E $- 5$
8	发酵器反应温度/K	299.000
9	热量/cal	77 092.970
10	通风率/（ $L · h - 1$ ）	30.000
11	搅拌功率/W	8.600
12	底物添加/（ $L · h - 1$ ）	0.042
13	酸流流速/（ $L · h - 1$ ）	0.007 5
14	碱流流速/（ $L · h - 1$ ）	0.022 5
15	冷水流速/（ $L · h - 1$ ）	40.000
16	热水流速/（ $L · h - 1$ ）	0.000

Tab. 3 Algorithm hyperparameter setting

超参数	数值
折扣因子	0.95
经验池容量	100 000
更新间隔	50
噪声强度	0.2
批处理量	64

Fig. 4 Reward values of different algorithms changing with number of iterations

Tab. 4 Comparison of learning efficiency of different methods

控制方法	局部收敛集数	局部收敛平均奖励	收敛集数	收敛平均奖励
LPIE-DDPG	100	15 000	800	18 000
DDPG	750	14 000	1 100	18 000
LPIE-DQN	100	12 000	500	14 000
DQN	250	10 000	1 200	14 000

Fig. 5 Comparison of temperature control effect of LPIE-DDPG， LPIE-DQN and PID

Fig. 6 Comparison of penicillin yield under LPIE-DDPG，LPIE-DQN and PID control

Fig. 7 Comparison of interference resistance ofLPIE-DDPG， LPIE-DQN and PID

References 16

1	BANSAL V， PERKINS J D， PISTIKOPOULOS E N. A case study in simultaneous design and control using rigorous， mixed-integer dynamic optimization models ［J］. Industrial and Engineering Chemistry Research， 2002， 41（4）： 760-778. 10.1021/ie010156n
2	ASTEASUAIN M， BANDONI A， SARMORIA C， et al. Simultaneous process and control system design for grade transition in styrene polymerization ［J］. Chemical Engineering Science， 2006， 61（10）： 3362-3378. 10.1016/j.ces.2005.12.012
3	赵海丞，邹应全，刘睿佳，等.温控系统中变调节周期PID算法［J］.计算机应用，2016，36（S2）：116-119.
	ZHAO H C， ZOU Y Q， LIU R J， et al. PID algorithm of variable adjustment period based on temperature control system ［J］. Journal of Computer Applications， 2016， 36（S2）： 116-119.
4	包元兴，丁炯，杨遂军，等.强耦合双通道热分析炉温度跟随控制策略研究［J］.测控技术，2016，35（5）：70-74. 10.3969/j.issn.1000-8829.2016.05.018
	BAO Y X， DING J， YANG S J， et al. Study on temperature following control strategy for strong-coupled dual-channel thermal analysis furnace ［J］. Measurement and Control Technology， 2016， 35（5）： 70-74. 10.3969/j.issn.1000-8829.2016.05.018
5	吴鹏松，吴朝野，周东华.大纯滞后信号解耦内模控制系统研究［J］.化工自动化及仪表，2012，39（9）：1115-1117，1176. 10.3969/j.issn.1000-3932.2012.09.002
	WU P S， WU C Y， ZHOU D H. Research on signal-decoupling internal mode control system with big time lag ［J］. Control and Instruments in Chemical Industry， 2012， 39（9）： 1115-1117， 1176. 10.3969/j.issn.1000-3932.2012.09.002
6	张惠琳，李醒飞，杨少波，等.深海自持式智能浮标双闭环模糊PID定深控制［J］.信息与控制，2019，48（2）：202-208，216.
	ZHANG H L， LI X F， YANG S B， et al. Dual closed-loop fuzzy PID depth control for deep-sea self-holding intelligent buoy ［J］. Information and Control， 2019， 48（2）： 202-208， 216.
7	庄绪君，李宏光.基于遗传算法与迭代动态规划混合策略的青霉素发酵过程优化控制［J］.计算机与应用化学，2013，30（9）：1051-1054. 10.3969/j.issn.1001-4160.2013.09.021
	ZHUANG X J， LI H G. Optimization control strategies combined genetic algorithms and iterative dynamic programming for penicillin fermentation processes ［J］. Computers and Applied Chemistry， 2013， 30（9）： 1051-1054. 10.3969/j.issn.1001-4160.2013.09.021
8	郝鹃，余建军，周文慧.基于平均强化学习的订单生产方式企业订单接受策略［J］.计算机应用，2013，33（4）：976-979. 10.3724/sp.j.1087.2013.00976
	HAO J， YU J J， ZHOU W H. Order acceptance policy in make-to-order manufacturing based on average-reward reinforcement learning ［J］. Journal of Computer Applications， 2013， 33（4）： 976-979. 10.3724/sp.j.1087.2013.00976
9	王建平，王刚，毛晓彬，等.基于深度强化学习的二连杆机械臂运动控制方法［J］.计算机应用，2021，41（6）：1799-1804.
	WANG J P， WANG G， MAO X B， et al. Motion control method of two-link manipulator based on deep reinforcement learning ［J］. Journal of Computer Applications， 2021， 41（6）： 1799-1804.
10	多南讯，吕强，林辉灿，等.迈进高维连续空间：深度强化学习在机器人领域中的应用［J］.机器人，2019，41（2）：276-288.
	DUO N X， LYU Q， LIN H C， et al. Step into high-dimensional and continuous action space： a survey on applications of deep reinforcement learning to robotics ［J］. Robot， 2019， 41（2）： 276-288.
11	刘洋，李建军.深度确定性策略梯度算法优化［J］.辽宁工程技术大学学报（自然科学版），2020，39（6）：545-549. 10.1016/j.cie.2021.107621
	LIU Y， LI J J. Optimization of deep deterministic policy gradient algorithm ［J］. Journal of Liaoning Technical University （Natural Science）， 2020， 39（6）：545-549. 10.1016/j.cie.2021.107621
12	HOU Y N， LIU L F， WEI Q， et al. A novel DDPG method with prioritized experience replay ［C］// Proceedings of the 2017 IEEE International Conference on Systems， Man， and Cybernetics. Piscataway： IEEE， 2017： 316-321. 10.1109/smc.2017.8122622
13	NIAN R， LIU J F， HUANG B. A review on reinforcement learning： Introduction and applications in industrial process control ［J］. Computers and Chemical Engineering， 2020， 139： Article No.106886. 10.1016/j.compchemeng.2020.106886
14	李云龙，唐文俊，白成海，等.青霉素生产工艺优化及代谢分析提高产量［J］.中国抗生素杂志，2019，44（6）：679-686. 10.3969/j.issn.1001-8689.2019.06.006
	LI Y L， TANG W J， BAI C H， et al. Optimization of the feeding process and metabolism analysis to improve the yield of penicillin ［J］. Chinese Journal of Antibiotics， 2019， 44（6）： 679-686. 10.3969/j.issn.1001-8689.2019.06.006
15	王蕾，陈进东，潘丰.引力搜索算法在青霉素发酵模型参数估计中的应用［J］.计算机应用，2013，33（11）：3296-3299，3304. 10.3724/sp.j.1087.2013.03296
	WANG L， CHEN J D， PAN F. Applications of gravitational search algorithm in parameters estimation of penicillin fermentation process model ［J］. Journal of Computer Applications， 2013， 33（11）： 3296-3299， 3304. 10.3724/sp.j.1087.2013.03296
16	叶凌箭，程江华.基于Matlab/Simulink的青霉素发酵过程仿真平台［J］.系统仿真学报，2015，27（3）：515-520.
	YE L J， CHENG J H. Simulation platform of penicillin fermentation process based on Matlab/Simulink ［J］. Journal of System Simulation， 2015， 27（3）： 515-520.

[1]	Yi ZHOU, Hua GAO, Yongshen TIAN. Proximal policy optimization algorithm based on clipping optimization and policy guidance [J]. Journal of Computer Applications, 2024, 44(8): 2334-2341.
[2]	Tian MA, Runtao XI, Jiahao LYU, Yijie ZENG, Jiayi YANG, Jiehui ZHANG. Mobile robot 3D space path planning method based on deep reinforcement learning [J]. Journal of Computer Applications, 2024, 44(7): 2055-2064.
[3]	Xiaoyan ZHAO, Wei HAN, Junna ZHANG, Peiyan YUAN. Collaborative offloading strategy in internet of vehicles based on asynchronous deep reinforcement learning [J]. Journal of Computer Applications, 2024, 44(5): 1501-1510.
[4]	Rui TANG, Chuanlin PANG, Ruizhi ZHANG, Chuan LIU, Shibo YUE. DDPG-based resource allocation in D2D communication-empowered cellular network [J]. Journal of Computer Applications, 2024, 44(5): 1562-1569.
[5]	Xintong QIN, Zhengyu SONG, Tianwei HOU, Feiyue WANG, Xin SUN, Wei LI. Channel access and resource allocation algorithm for adaptive p-persistent mobile ad hoc network [J]. Journal of Computer Applications, 2024, 44(3): 863-868.
[6]	Yuanchao LI, Chongben TAO, Chen WANG. Gait control method based on maximum entropy deep reinforcement learning for biped robot [J]. Journal of Computer Applications, 2024, 44(2): 445-451.
[7]	Fuqin DENG, Huifeng GUAN, Chaoen TAN, Lanhui FU, Hongmin WANG, Tinlun LAM, Jianmin ZHANG. Multi-robot reinforcement learning path planning method based on request-response communication mechanism and local attention mechanism [J]. Journal of Computer Applications, 2024, 44(2): 432-438.
[8]	Jiachen YU, Ye YANG. Irregular object grasping by soft robotic arm based on clipped proximal policy optimization algorithm [J]. Journal of Computer Applications, 2024, 44(11): 3629-3638.
[9]	Yu WANG, Zhihui GUAN, Yuanpeng LI. Distributed UAV cluster pursuit decision-making based on trajectory prediction and MADDPG [J]. Journal of Computer Applications, 2024, 44(11): 3623-3628.
[10]	Jie LONG, Liang XIE, Haijiao XU. Integrated deep reinforcement learning portfolio model [J]. Journal of Computer Applications, 2024, 44(1): 300-310.
[11]	Yu WANG, Tianjun REN, Zilin FAN. Air combat maneuver decision-making of unmanned aerial vehicle based on guided Minimax-DDQN [J]. Journal of Computer Applications, 2023, 43(8): 2636-2643.
[12]	Ziteng WANG, Yaxin YU, Zifang XIA, Jiaqi QIAO. Sparse reward exploration mechanism fusing curiosity and policy distillation [J]. Journal of Computer Applications, 2023, 43(7): 2082-2090.
[13]	Xiaolin LI, Yusang JIANG. Task offloading algorithm for UAV-assisted mobile edge computing [J]. Journal of Computer Applications, 2023, 43(6): 1893-1899.
[14]	Heping FANG, Shuguang LIU, Yongyi RAN, Kunhua ZHONG. Integrated scheduling optimization of multiple data centers based on deep reinforcement learning [J]. Journal of Computer Applications, 2023, 43(6): 1884-1892.
[15]	Xiaohui HUANG, Kaiming YANG, Jiahao LING. Order dispatching by multi-agent reinforcement learning based on shared attention [J]. Journal of Computer Applications, 2023, 43(5): 1620-1624.

Industrial process control method based on local policy interaction exploration-based deep deterministic policy gradient

基于局部策略交互探索的深度确定性策略梯度的工业过程控制方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 16

Related Articles 15

Recommended Articles

Metrics